CADRE: Stable, Parameter-Efficient Adaptation of Medical Vision–Language Models with Bounded Forgetting and Prior Drift Amrita Singh1 and Rishabh Jha2
arXiv:2606.23487v1 [cs.AI] 22 Jun 2026
1
Mindriser’s Consortium, Kathmandu, Nepal 2 University of Victoria, BC, Canada [email protected]
Abstract. Medical vision–language models (VLMs) such as BiomedCLIP generalize broadly, but adapting them to a clinical service is as much a safety problem as an accuracy one. Updating a deployed model for a new imaging modality can fail silently in two ways that harm patients: it can forget modalities it already handled (catastrophic forgetting), and it can drift from its trustworthy pretrained prior toward modality-specific shortcuts. We study parameter-efficient continual adaptation through these two properties rather than leaderboard accuracy, presenting CADRE: a frozen-backbone framework combining low-rank adaptation (LoRA) with an online, self-scaling, similarity-aware elastic weight consolidation term that bounds retained-competence loss, and an anchor-to-prior penalty bounding embedding drift from the frozen prior. Two short guarantees, a bound on total consolidation mass and a scale-invariance property remove the scale-related sources of vanilla EWC’s order fragility. Using breast cancer across three maximally dissimilar modalities (histopathology, ultrasound, chest radiography) as a controlled cross-modality stress test, under a multi-seed, multi-order protocol with paired significance testing and training ≈0.23% of parameters, CADRE attains the highest accuracy, SPQ and backward transfer and the lowest forgetting among adapting methods reducing forgetting roughly sevenfold versus the strongest regularized baseline (0.075 → 0.011; paired p=0.023) and achieving positive backward transfer where every baseline is negative. We frame these as stability properties aligned with clinical-safety desiderata, not a deployment guarantee; robustness to distribution shift and adversarial inputs is out of scope. Keywords: Safe adaptation · Continual learning · Vision–language models · Parameter-efficient fine-tuning · Calibration.
1
Introduction
Foundation VLMs pretrained on large biomedical image–text corpora generalize across many tasks, yet a single clinical deployment rarely matches the pretraining distribution: a breast-imaging service may want one model to reason over histopathology, ultrasound, and radiographs and to keep improving as
2
Amrita Singh and Rishabh Jha
new modalities arrive. Naive fine-tuning per modality is unsafe in two specific senses. Silent competence loss: sequential adaptation degrades earlier-modality performance [2], invisible in a clinic an ultrasound update can quietly weaken histopathology reading, surfacing only as missed diagnoses. Silent prior drift and order fragility: adapters can move the model toward dataset-specific shortcuts (scanner, staining, acquisition artifacts), and a standard remedy, EWC, is itself fragile, a fixed global strength over an unnormalized, accumulating penalty overconstrains whichever modality arrives last, so the same method is safe under one adaptation order and unsafe under another. We show this and remove it. Rather than ask “how high is accuracy after adaptation?”, we ask what does adaptation preserve about the model we already trusted? We require three monitorable properties retained competence, bounded drift from the pretrained prior, and usable calibration (formalised as S1–S3 in Sec. 3) and treat them as stability properties that align with clinical-safety desiderata, not a guarantee of safe deployment: our controlled, balanced benchmark does not probe out-of-distribution robustness, rare cases, class imbalance, or adversarial inputs. We instantiate this with CADRE (Clinician-anchored, Domain-Robust, Efficient adaptation), which keeps the BiomedCLIP backbone frozen and adapts through LoRA, regularized by an online self-scaling similarity-aware EWC term (competence retention) and an anchor-to-prior penalty (drift control), with light label smoothing and evaluation-time weight averaging for calibration (Fig. 1). Contributions. (1) A stability-oriented reformulation of parameter-efficient continual adaptation for medical VLMs, with forgetting/backward transfer and prior drift as first-class objectives and calibration as a trust signal. (2) A frozenbackbone method whose key novelty is a self-scaling, similarity-aware online EWC, no hand-tuned consolidation strength, robust to adaptation order with anchor-to-prior drift control, training ≈0.23% of parameters and supported by two guarantees (a total-mass bound and a scale-invariance property). (3) A crossmodality breast-cancer stress protocol (histopathology/ultrasound/radiography) with per-modality leakage controls, multi-seed/multi-order evaluation, paired significance testing, component attribution, and an order-robustness analysis; CADRE has the highest accuracy, SPQ and backward transfer and the lowest forgetting among adapting methods, and significantly reduces forgetting over every LoRA baseline.
2
Related Work
PEFT and continual learning. Adapter [17] and low-rank [5] tuning freeze a backbone and train a small injected set, matching full fine-tuning cheaply while avoiding destruction of pretrained priors; LoRA is now a default for medical vision–language adaptation. These control where a model changes but bound neither earlier-competence loss across a sequence nor representation drift. Sequential training induces catastrophic forgetting [2], with remedies in regularization, replay, and architectural families [11, 12]; replay/exemplar methods [7, 8]
CADRE: Stable, Parameter-Efficient Adaptation of Medical VLMs
3
store past data, often clinically unacceptable for privacy. Regularization methods constrain movement by an importance estimate Fisher information, path integrals, or output sensitivity [3, 13, 14] with online EWC [4] keeping a single running term and functional/distillation methods regularizing in output space [6, 16]. We build on online EWC but remove a scale-and-order fragility in its standard form, anchoring in embedding space to the frozen prior rather than distilling from the previous model. Composition methods instead allocate capacity per task prompt pools [30, 31, 29] or orthogonalized adapters [32, 33] but grow parameters with the task count and require task identity at inference. CADRE keeps a single shared adapter so one weight set serves all modalities: no per-task expansion, no stored exemplars, no task-ID routing at test time. Medical CL, calibration, weight averaging. EWC and LwF have been studied for chest-radiograph classification [24], and dynamic memory for acquisition shift [25, 26]. Closest to us, recent work adapts the EWC penalty for medical VLMs via prompt-guided grouping and similarity-weighted Fisher [27]. CADRE differs by requiring no hand-tuned strength (fixing it as a fraction of the task loss), making the penalty scale-free via sum-normalised Fisher with a proven mass bound, and adding an explicit anchor-to-prior drift bound with calibration as a trust signal. We adapt BiomedCLIP [1] rather than train a VLM [9, 10], report ECE [19, 18], and use label smoothing [20, 21] and evaluation-time weight averaging [22, 23].
3
Method: CADRE
Adaptation model. The BiomedCLIP backbone (ViT-B/16 vision encoder, PubMedBERT text encoder) stays frozen; we train only a small parameter vector θ (≈0.23% of the backbone) made up of LoRA factors injected into the attention projections of the 25 visual-encoder layers, plus the linear head. Because the head reads the image embedding, adapting the visual side is what moves predictions. We write gLoRA for the adapted image embedding and gfrozen for the same encoder with its LoRA paths switched off, i.e. the pretrained prior. Modalities arrive one at a time as a stream D1 , . . . , DT , each a balanced binary task. Metrics. Let At,j be accuracy on modality j after training through modality t. We report average final accuracy; forgetting, the mean drop from each earlier modality’s best accuracy to its final accuracy; backward transfer (BWT), the mean change in an earlier modality’s accuracy between when it was learned and the final state (positive means later training helped it); and the Stability– Plasticity Quotient (SPQ), the harmonic mean of plasticity (mean accuracy on each modality when it is first learned) and stability (one minus forgetting, floored at zero). Lower forgetting and higher BWT/SPQ are better. Safety objectives. An update should satisfy three monitorable properties: (S1) bounded competence loss (forgetting/BWT within tolerance on prior modalities), (S2) bounded prior drift (the adapted embedding stays within a bounded distance of the frozen prior), and (S3) calibration (usable confidences)
4
Amrita Singh and Rishabh Jha
Fig. 1. Overview of CADRE. A frozen BiomedCLIP (ViT-B/16) encoder is adapted sequentially over dissimilar modalities (histopathology → ultrasound → · · · → radiography) via visual-encoder LoRA (≈0.23% of parameters) and a linear head, regularized by three stability mechanisms: (1) online self-scaling EWC over a sum-normalised running Fisher; (2) similarity-aware retention relaxing consolidation for dissimilar modalities; (3) evaluation-time EMA weight averaging with light label smoothing. The anchor penalty Lanchor = ∥gLoRA − gfrozen ∥2 bounds embedding drift from the frozen prior.
— targeted respectively by the consolidation term, the anchor, and the calibration components. Why vanilla EWC is order-fragile. EWC discourages a new task from moving parameters that mattered for earlier tasks, weighting each parameter by its Fisher importance and pulling it back toward a stored reference θ∗ . Online EWC [4] keeps a single running importance estimate F and reference rather than one per task: LEWC (θ) =
∗ 2 i Fi (θi − θi ) ,
P
F ← γsim F + F̂ (t) ,
θ∗ ← θt .
(1)
The standard form weights this penalty by a single fixed multiplier λ applied to the raw Fisher, which creates two coupled scale problems. First, the raw Fisher tracks gradient magnitude — so it varies with task difficulty and loss scale — and the running sum grows as more tasks are consolidated, so one λ cannot stay comparable across steps. Second, because that running sum is orderdependent, a λ tuned for one arrival order over- or under-constrains another, and the last-arriving modality faces the largest accumulated penalty. This is exactly the regime where vanilla EWC’s forgetting swings sharply (Sec. 5). CADRE removes both scale problems by construction. Scale-decoupling. (M1) Sum-normalised Fisher. Before consolidating each task’s Fisher we rescale it to unit total mass, so every modality contributes the same total importance regardless of its loss scale and the running estimate in (1) can never accumulate unbounded mass. (M2) Self-scaling strength. Rather than tune a fixed λ, we recompute it each step from detached (stop-
CADRE: Stable, Parameter-Efficient Adaptation of Medical VLMs
5
gradient) loss values, LCE λt = min ρ LEWC , λ max , +ε
ρ = 0.3,
(2)
so that, when unclipped, the penalty always contributes a fixed fraction ρ of the current task loss. This replaces a brittle global hyper-parameter with a selfnormalising ratio; a floor ε, a cap λmax , and a 50-step warm-up (with λt =0 while the first Fisher is estimated) stop it from spiking before the prior has been meaningfully perturbed. (M3) Similarity-aware retention. When the incoming modality is unlike those already consolidated, fully keeping the old constraints over-regularises an incompatible target. We therefore scale how much of the running Fisher is retained by the cosine similarity between successive modality prototypes (means of normalised embeddings), mapping it onto [0, γ] with γ = 0.9: dissimilar modalities relax retention, similar ones preserve it. This directly targets the histopathology-last instability of the naive baseline and is a heuristic we validate empirically. The mapping keeps the retention factor γsim ∈ [0, γ], the precondition of Proposition 1. Drift bound: anchor-to-prior. To control (S2) we add an embeddingspace penalty [6, 16] that keeps the adapted encoder close to the frozen prior on a fixed probe set A (|A|=64): 1 X 2 gLoRA (x) − gfrozen (x) 2 . (3) Lanchor = |A| x∈A
A is fixed at the start of the first modality, drawn from the training split only, with its prior embeddings cached, so it cannot leak into evaluation; the two embeddings must come from different adapter states on the same input (identical passes give a degenerate zero term). Adding β Lanchor to the task loss is the Lagrangian of minimising the task loss subject to a cap δ on mean drift over A — a soft bound on expected drift, not a per-input Lipschitz guarantee. This limits shortcut adoption and lets the anchor distance be thresholded at inference as a runtime guardrail for out-of-prior inputs. Calibration and the full objective. For (S3) CADRE adds light label smoothing (εls =0.05) during training and an exponential moving average (EMA, decay 0.99) of the trainable parameters that is applied at evaluation only (SWA-style); we report Expected Calibration Error (ECE), the average gap between confidence and accuracy across equal-width confidence bins. The per-step loss is simply the label-smoothed cross-entropy plus the self-scaled consolidation penalty ((2), with the sum-normalised, similarity-decayed Fisher) plus the anchor penalty. During training the Fisher and the reference θ∗ both come from the live weights, so the penalty and the optimised parameters are always the same set; only evaluation uses the EMA weights. Because EMA averages weights along the trajectory it can itself reduce measured forgetting, so we treat its contribution as something to attribute rather than assume (Sec. 5). Theoretical analysis. Two guarantees explain why the scale fixes work. Proposition 1 (Bounded consolidation mass). If each consolidated Fisher has unit mass and the retention factor satisfies γsim ∈ [0, γ] with 0 ≤ γ < 1, then
6
Amrita Singh and Rishabh Jha
P the total mass mt = i Fi stays bounded by 1/(1 − γ) for all t, independent of the number of modalities T . Proof. The update in (1) gives mt = γsim mt−1 + 1 ≤ γmt−1 + 1; with m0 = 0, t 1 □ induction yields mt ≤ 1−γ 1−γ ≤ 1−γ . Proposition 2 (Scale-invariant trade-off ). Let ε → 0 with λt unclipped. Under any rescaling of the penalty LEWC 7→ c LEWC (c > 0), both the self-scaled penalty value and its gradient are unchanged. Proof. By (2) the multiplier rescales as λt 7→ λt /c, which exactly cancels the factor c in both the penalty value (ρ LCE ) and its gradient. □ Together M1 and M2 remove the two scale-related sources of order dependence — penalty accumulation and magnitude mismatch — but not interference between modality representations, which is not a scale effect; we therefore claim full order-robustness only empirically (Sec. 5). After each modality we update the prototype, consolidate the Fisher and reference, and re-evaluate all seen modalities under the EMA weights.
4
Experimental Setup
Data and task framing. We use three modalities breast histopathology, breast ultrasound, and chest radiography as a deliberately dissimilar cross-modality stress test, not a claim that breast malignancy is read from chest radiographs: the three are different patient populations with modality-specific binary labels (malignant vs. benign tissue; malignant vs. benign mass; abnormal vs. normal). Radiography is included precisely because it is most distant from breast tissue, a hard negative-transfer probe stressing order-robustness. Each modality is balanced (N =600, 50% positive; N =1800 total), read as RGB at 224 × 224. The histopathology source (BreaKHis [28], identified by its SOB filename convention) is natively 3-channel RGB; the source bundle’s “MSI/Multispectral” folder labels are a packaging artefact, not extra spectral bands, so we treat all three as RGB. Leakage control. Histopathology is grouped at slide level (43 groups) via BreaKHis SOB filename parsing. Ultrasound and radiography currently group at image level (600 groups for 600 images; no recoverable case IDs), so same-case leakage cannot be excluded and these two accuracies are likely optimistic (see Limitations). Splits are stratified by group into train/val/test (70/15/15). Protocol. Frozen BiomedCLIP; visual LoRA rank 8, α=16; AdamW (lr 5×10−4 , wd 10−4 ); E=10 epochs per modality; mixed precision. CADRE: ρ=0.3, γ=0.9, β=0.3, |A|=64, label smoothing 0.05, EMA decay 0.99, Fisher over 50 batches. We run 3 seeds × 2 orders and report mean±SEM with paired t-tests (CADRE vs. each baseline) on accuracy, forgetting, and ECE. Hardware: single T4 GPU. Methods: Linear Probe (frozen features + linear head); LoRA; LoRA+EWC (vanilla offline EWC, fixed λ); LoRA+Anchor; CADRE (online self-scaling similarity-aware EWC + anchor + label smoothing + EMA). The LoRA+EWC baseline uses the vanilla formulation, so its contrast with CADRE reflects both the consolidation redesign and the added components.
CADRE: Stable, Parameter-Efficient Adaptation of Medical VLMs
7
Table 1. Main comparison: cross-modality continual adaptation (mean±SEM, 3 seeds × 2 orders). Lower forgetting better; higher Acc/AUROC/SPQ/BWT better. Best per column among comparison methods in bold; CADRE−anchor is a controlled ablation, set apart. Method
Acc
AUROC
BWT
Forget.
SPQ
Linear Probe LoRA LoRA + EWC LoRA + Anchor CADRE (full)
0.603±.029 0.746±.012 0.756±.021 0.754±.011 0.775±.009
0.720±.008 0.813±.011 0.820±.009 0.813±.009 0.821±.006
−0.009±.046 −0.081±.017 −0.070±.028 −0.075±.016 0.004±.008
0.061±.022 0.084±.015 0.081±.026 0.075±.016 0.011±.006
0.738±.006 0.854±.008 0.856±.013 0.860±.007 0.867±.006
CADRE − anchor
0.738±.008
0.807±.014
0.008±.012
0.023±.009
0.837±.008
Trainable parameters: Linear Probe 0.001%; all LoRA variants 0.231%. Removing the anchor significantly lowers accuracy (p=0.005) and modestly raises forgetting.
5
Results
All values are mean±SEM over 3 seeds × 2 orders from a single unified run, so comparison, attribution, and matrix are mutually consistent. Among adapting methods CADRE attains the highest accuracy (0.775), SPQ (0.867) and backward transfer (+0.004), the lowest forgetting (0.011), and AUROC (0.821) tied best (Table 1); its calibration (ECE 0.137) is also the best of the LoRA methods (discussed below). Forgetting is the headline. CADRE reduces forgetting roughly sevenfold versus the strongest regularized baseline LoRA+Anchor (0.075 → 0.011; paired p=0.023) and versus plain LoRA (0.084 → 0.011; p=0.002), and is the only method besides its anchor-free variant with positive backward transfer (+0.004) where every baseline is strongly negative (−0.07 to −0.08). The contrast with LoRA+EWC misses forgetting significance (p=0.072) only because that baseline is high-variance (forgetting SEM ±0.026 vs CADRE’s ±0.006) underpowered at n=6, not null. On accuracy, paired t-tests (n=6, dof = 5) put CADRE significantly above plain LoRA (+0.030, p=0.014), LoRA+Anchor (+0.021, p=0.008), its anchor-free variant (+0.037, p=0.005) and the linear probe (+0.173, p=0.002), with the LoRA+EWC gap not significant (+0.020, p=0.372); ECE favours CADRE over the LoRA baselines (e.g. 0.188 → 0.137 vs LoRA, ∆= − 0.051, p=0.071) but not significantly at n=6. Per-modality, per-class. CADRE is the most balanced adapting method, collapsing neither an early modality nor a class: best abnormal-class recall on the hardest modality, chest radiography (0.607 vs 0.389–0.511 for the LoRA baselines), while remaining strong on both histopathology classes (0.856 nonmalignant / 0.776 malignant) and ultrasound (0.911 / 0.930). The linear probe collapses on positive classes (0.260 / 0.078 / 0.348), confirming its low ECE comes from a degenerate, low-recall solution, not good adaptation. Robustness to modality order. Order sensitivity surfaces as cross-(seed, order) variance: CADRE’s forgetting SEM is ±0.006 on a mean of 0.011, roughly
8
Amrita Singh and Rishabh Jha
four times tighter than vanilla LoRA+EWC’s ±0.026 on 0.081, and its backwardtransfer SEM is ±0.008 vs ±0.028 far less sensitive to arrival order, consistent with the scale guarantees of Sec. 3. The full At,j matrix for CADRE on order 1 shows histopathology near-flat across updates (0.857 → 0.870 → 0.844) and chestradiograph accuracy improving after the later modality (0.581 → 0.589; positive backward transfer). Component attribution. Toggling each added component off (rest held fixed; mean±SEM over 3 seeds × 2 orders; p on forgetting vs. CADRE-full, paired, dof = 5), CADRE-full’s forgetting (0.009±0.004; Acc 0.777, ECE 0.140) matches the main comparison (0.011±0.006), so attribution and headline share one protocol. Three removals significantly raise forgetting: the consolidation redesign (self-scaling → vanilla EWC: 0.038, Acc 0.749, ECE 0.129, p=0.035), the anchor (0.019, Acc 0.744, ECE 0.134, p=0.019), and EMA (0.025, Acc 0.784, ECE 0.155, p=0.042). Reverting the EWC redesign costs the most accuracy (0.777 → 0.749), the primary stability driver; the anchor’s clearest effect is on accuracy (p=0.005). EMA lowers accuracy slightly when present (0.784 → 0.777) yet suppresses forgetting part of the stability is a mechanical averaging effect we report rather than credit to consolidation. Label smoothing (0.018, Acc 0.778, ECE 0.147, p=0.36) and similarity-aware retention (0.016, Acc 0.770, ECE 0.139, p=0.49) show no significant forgetting effect at n=6.
6
Discussion and Conclusion
Treating forgetting as a quantity to bound turns a continual-learning metric into a stability property a clinical team can monitor a precondition for safe updating, not a proof of it. CADRE attains linear-probe-level retention with full-PEFTlevel accuracy, remains order-robust where vanilla EWC swings, and does not trade away calibration. A preliminary probe of the anchor distance as a runtime OOD detector is negative (in-prior mean 1.54 yet noise-perturbed inputs average 1.27); we leave a proper OOD evaluation to future work. Limitations include image-level grouping for ultrasound and radiography (likely optimistic accuracies), a balanced benchmark that does not probe class imbalance or distribution shift, and an order-invariance analysis that remains empirical rather than theoretical. Conclusion. CADRE recasts parameter-efficient continual adaptation as a stability problem, delivering sevenfold-lower forgetting, positive backward transfer, and order-stability at ≈0.23% of parameters via a self-scaling similarity-aware EWC redesign supported by a bounded-mass guarantee and a scale-invariance proof combined with an anchor-to-prior drift penalty and calibration-preserving components. The resulting stability properties bounded forgetting, bounded drift, usable ECE are formulated as monitorable quantities aligned with clinicalsafety desiderata, not a deployment guarantee; robustness to distribution shift and adversarial inputs must be established independently before any clinical use.
CADRE: Stable, Parameter-Efficient Adaptation of Medical VLMs
9
References 1. Zhang, S., Xu, Y., Usuyama, N., Bagga, J., Tinn, R., Preston, S., Rao, R., Wei, M., Valluri, N., Wong, C., Lungren, M.P., Naumann, T., Poon, H.: BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million image– text pairs. arXiv:2303.00915 (2023) 2. McCloskey, M., Cohen, N.J.: Catastrophic interference in connectionist networks: the sequential learning problem. In: Bower, G.H. (ed.) Psychology of Learning and Motivation, vol. 24, pp. 109–165. Academic Press (1989) 3. Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A.A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., Hassabis, D., Clopath, C., Kumaran, D., Hadsell, R.: Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences 114(13), 3521–3526 (2017) 4. Schwarz, J., Czarnecki, W., Luketina, J., Grabska-Barwinska, A., Teh, Y.W., Pascanu, R., Hadsell, R.: Progress & Compress: a scalable framework for continual learning. In: Proceedings of the 35th International Conference on Machine Learning (ICML), pp. 4528–4537 (2018) 5. Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: LoRA: low-rank adaptation of large language models. In: International Conference on Learning Representations (ICLR) (2022). arXiv:2106.09685 6. Li, Z., Hoiem, D.: Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence 40(12), 2935–2947 (2017) 7. Lopez-Paz, D., Ranzato, M.: Gradient episodic memory for continual learning. In: Advances in Neural Information Processing Systems 30 (NeurIPS), pp. 6467–6476 (2017) 8. Rebuffi, S.A., Kolesnikov, A., Sperl, G., Lampert, C.H.: iCaRL: incremental classifier and representation learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2001–2010 (2017) 9. Wang, Z., Wu, Z., Agarwal, D., Sun, J.: MedCLIP: contrastive learning from unpaired medical images and text. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3894–3905 (2022). arXiv:2210.10163 10. Huang, Z., Bianchi, F., Yuksekgonul, M., Montine, T.J., Zou, J.: A visual–language foundation model for pathology image analysis using medical Twitter. Nature Medicine 29, 2304–2316 (2023) 11. Parisi, G.I., Kemker, R., Part, J.L., Kanan, C., Wermter, S.: Continual lifelong learning with neural networks: a review. Neural Networks 113, 54–71 (2019) 12. De Lange, M., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G., Tuytelaars, T.: A continual learning survey: defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence 44(7), 3366–3385 (2022) 13. Zenke, F., Poole, B., Ganguli, S.: Continual learning through synaptic intelligence. In: Proceedings of the 34th International Conference on Machine Learning (ICML), pp. 3987–3995 (2017) 14. Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., Tuytelaars, T.: Memory aware synapses: learning what (not) to forget. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 144–160 (2018) 15. MacKay, D.J.C.: A practical Bayesian framework for backpropagation networks. Neural Computation 4(3), 448–472 (1992)
10
Amrita Singh and Rishabh Jha
16. Titsias, M.K., Schwarz, J., Matthews, A.G.G., Pascanu, R., Teh, Y.W.: Functional regularisation for continual learning with Gaussian processes. In: International Conference on Learning Representations (ICLR) (2020) 17. Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., Gelly, S.: Parameter-efficient transfer learning for NLP. In: Proceedings of the 36th International Conference on Machine Learning (ICML), pp. 2790–2799 (2019) 18. Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. In: Proceedings of the 34th International Conference on Machine Learning (ICML), pp. 1321–1330 (2017) 19. Naeini, M.P., Cooper, G.F., Hauskrecht, M.: Obtaining well calibrated probabilities using Bayesian binning. In: Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence (AAAI), pp. 2901–2907 (2015) 20. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the Inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818–2826 (2016) 21. Müller, R., Kornblith, S., Hinton, G.: When does label smoothing help? In: Advances in Neural Information Processing Systems 32 (NeurIPS), pp. 4694–4703 (2019) 22. Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., Wilson, A.G.: Averaging weights leads to wider optima and better generalization. In: Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence (UAI), pp. 876– 885 (2018) 23. Tarvainen, A., Valpola, H.: Mean teachers are better role models: weight-averaged consistency targets improve semi-supervised deep learning results. In: Advances in Neural Information Processing Systems 30 (NeurIPS), pp. 1195–1204 (2017) 24. Lenga, M., Schulz, H., Saalbach, A.: Continual learning for domain adaptation in chest X-ray classification. In: Proceedings of the Third Conference on Medical Imaging with Deep Learning (MIDL), PMLR 121, pp. 413–423 (2020). arXiv:2001.05922 25. Perkonigg, M., Hofmanninger, J., Herold, C.J., Brink, J.A., Pianykh, O.S., Prosch, H., Langs, G.: Dynamic memory to alleviate catastrophic forgetting in continual learning with medical imaging. Nature Communications 12, 5678 (2021) 26. Qazi, M.A., Nawaz, M., Naseer, M., Khan, S., Khan, F.S.: Continual learning in medical imaging: a survey and practical analysis. arXiv:2405.13482 (2024) 27. Gao, Z., Morel, P.: Prompt-aware adaptive elastic weight consolidation for continual learning in medical vision–language models. arXiv:2511.20732 (2025) 28. Spanhol, F.A., Oliveira, L.S., Petitjean, C., Heutte, L.: A dataset for breast cancer histopathological image classification. IEEE Transactions on Biomedical Engineering 63(7), 1455–1462 (2016) 29. Jia, M., Tang, L., Chen, B.C., Cardie, C., Belongie, S., Hariharan, B., Lim, S.N.: Visual prompt tuning. In: Proceedings of the European Conference on Computer Vision (ECCV), Lecture Notes in Computer Science 13686, pp. 709–727 (2022) 30. Wang, Z., Zhang, Z., Lee, C.Y., Zhang, H., Sun, R., Ren, X., Su, G., Perot, V., Dy, J., Pfister, T.: Learning to prompt for continual learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13949–13958 (2022) 31. Wang, Z., Zhang, Z., Ebrahimi, S., Sun, R., Zhang, H., Lee, C.Y., Ren, X., Su, G., Perot, V., Dy, J., Pfister, T.: DualPrompt: complementary prompting for rehearsalfree continual learning. In: Proceedings of the European Conference on Computer Vision (ECCV), Lecture Notes in Computer Science 13686, pp. 13746–13756 (2022)
CADRE: Stable, Parameter-Efficient Adaptation of Medical VLMs
11
32. Wang, X., Chen, T., Ge, Y., Zhou, J., Zhao, H.: Orthogonal subspace learning for language model continual learning (O-LoRA). In: Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 11123–11137 (2023). arXiv:2310.14152 33. Liang, Y.S., Li, W.J.: InfLoRA: interference-free low-rank adaptation for continual learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 28672–28681 (2024)