Preprint. Under review.
A Causal Language Modeling Detour Improves Encoder Continued Pretraining Éric de la Clergerie INRIA Paris ALMAnaCH Team eric.de la [email protected]
arXiv:2605.12438v1 [cs.CL] 12 May 2026
Rian Touchent Sorbonne Université / INRIA Paris ALMAnaCH Team [email protected]
Abstract When adapting an encoder to a new domain, the standard approach is to continue training with Masked Language Modeling (MLM). We show that temporarily switching to Causal Language Modeling (CLM) followed by a short MLM decay improves downstream performance. On biomedical texts with ModernBERT, this CLM detour outperforms MLM baselines trained on identical data and compute across 8 French and 11 English biomedical tasks, by +1.2–2.8pp and +0.3–0.8pp respectively, depending on model size. We investigate the reasons for these gains. We find that CLM’s dense supervision impacts low transformer layers (0–7) far more than MLM does. Freezing low layers during CLM eliminates the downstream benefit; freezing mid layers preserves it. The representational changes persist through the MLM decay phase, even when it matches the CLM phase in length, and they scale with model capacity. We release ModernCamemBERT-bio and ModernBERT-bio as state-of-the-art biomedical encoders in Base and Large sizes.
ModernBERT-bio (English):
1
base
large
ModernCamemBERT-bio (French):
base
large
Introduction
Domain-adaptive continued pretraining extends general-purpose language models to specialized domains (Gururangan et al., 2020; Ke et al., 2023). For encoders, this typically means extending masked language modeling (MLM) on domain text. We find that temporarily switching to causal language modeling (CLM) before returning to MLM, a CLM detour (see Figure 1), outperforms standard MLM continued pretraining on biomedical text, with the largest gains when the domain gap between pretraining and target data is large. The recipe changes only the attention mask and training objective, not the model architecture: use a causal mask for the CLM phase, then restore bidirectional attention and MLM for a short decay phase. With ModernBERT (Warner et al., 2025), this produces state-ofthe-art biomedical encoders in both English and French with 8,192-token context. Yet the final model uses bidirectional attention and never performs CLM at inference. Why does a temporary objective switch leave a lasting benefit? Comparing layer-by-layer representations between CLM-detour and MLM-only models with CKA (Kornblith et al., 2019), we observe that the CLM phase modifies low transformer layers far more than seed noise alone (>9× in layers 0–7). These changes survive the return to MLM, even when the MLM phase is as long as the CLM phase. Freeze interventions confirm this causally: the downstream benefit requires low-layer modification during CLM, and disappears entirely when these layers are held fixed (§5). The main contributions of this paper are: 1
Preprint. Under review.
62
Ours
Encoder
10B
10B
MLM
CLM
1B
1B
MLM
Avg F1 (%)
Decay
CPT
Encoder
61.6
61
(a) Pipeline
Baseline
61.2
60 59
58.9
59.3
MLM
58
detour
MLM baseline
CLM + CLM + freeze low freeze mid
CLM detour
Figure 1: (a) The CLM detour: a pretrained encoder trains with CLM, then returns to MLM (10% decay). The MLM baseline trains with MLM throughout for matched compute. (b) Freeze interventions (French, 8 tasks, 9 seeds). Freezing low layers (0–7) during CLM detour drops performance to MLM baseline level; freezing mid layers (8–14) preserves the CLM benefit. 1. A CLM detour recipe for domain-adaptive encoder pretraining, producing state-of-the-art biomedical encoders in English and French. We release ModernCamemBERT-bio and ModernBERT-bio in Base and Large sizes. 2. Evidence that the CLM phase leaves lasting changes in low transformer layers that MLM does not reverse, with divergence scaling with model capacity. 3. Causal evidence via freeze interventions: low layers are necessary for the CLM benefit, mid layers are not. 4. A practical guideline: 10% of the CLM budget suffices for the MLM return, confirmed at two scales.
2
Related Work
2.1
Continued Pretraining and Biomedical Encoders
Domain-adaptive continued pretraining extends general-purpose language models to specific domains by training further on domain-specific corpora. Gururangan et al. (2020) show that this helps most when the target domain is distant from the pretraining distribution, with biomedical text showing the largest gains. A central debate in biomedical NLP is whether to continue from a general checkpoint or train from scratch on domain data. BioBERT (Lee et al., 2020) and Bio ClinicalBERT (Alsentzer et al., 2019) take the continued pretraining route from BERT, while PubMedBERT (Gu et al., 2021) and SciBERT (Beltagy et al., 2019) train from scratch with domain-specific vocabularies. Gu et al. (2021) argue that vocabulary mismatch is the main bottleneck of continued pretraining. All these models share BERT’s 512-token context, which truncates long clinical documents such as discharge summaries or oncology reports. BioClinical-ModernBERT (Sounack et al., 2025) and Clinical ModernBERT (Lee et al., 2025) address this with the ModernBERT architecture (Warner et al., 2025), supporting 8,192 tokens; BioClinical-ModernBERT trains on 53B tokens in two phases (30% then 15% MLM masking). For French, the same debate arises: DrBERT (Labrak et al., 2023) was pretrained from scratch on 7GB of medical text, while CamemBERT-bio (Touchent and de la Clergerie, 2024) showed that continued pretraining of CamemBERT (Martin et al., 2020) on a smaller corpus achieves competitive results at a fraction of the cost. Both are limited to 512 tokens. ModernCamemBERT (Antoun et al., 2025) extends the ModernBERT architecture to French. All of the above use masked language modeling exclusively; none explore alternative training objectives for domain adaptation. 2
Preprint. Under review.
2.2
CLM and Hybrid Objectives for Encoder Training
Gisserot-Boukhlef et al. (2025) pretrain encoders from scratch (210M–1B parameters, 100B tokens) and find that a biphasic CLM-then-MLM schedule outperforms pure MLM under fixed compute, with CLM converging faster in early training and producing models that are less sensitive to fine-tuning hyperparameters. However, switching objectives does not always help. Weller et al. (2025) train matched encoder/decoder pairs (up to 1B parameters, 1.7T tokens) and show that continued pretraining on the reverse objective does not bridge the encoder-decoder performance gap, even after 50B tokens of adaptation using masked next-token prediction (MNTP) for the decoder-to-encoder direction. AntLM (Yu et al., 2024) takes a different approach, alternating between CLM and MLM epochs while switching both the attention mask and the training objective, and reports gains in both encoder (+2.2pp) and decoder (+1.0pp) directions at small scale (10M words). None of these works analyze why objective switching helps. 2.3
Representation Similarity and Training Dynamics
Centered Kernel Alignment (CKA; Kornblith et al., 2019) compares the internal representations of two networks layer by layer, providing a measure of how similarly they encode the same inputs. CKA has become a standard tool for analyzing how training changes representations in NLP models (Wu et al., 2020). Merchant et al. (2020) use CKA to show that task fine-tuning primarily modifies the top layers of BERT while lower layers remain stable. More broadly, deep networks exhibit critical learning periods where early training conditions leave lasting traces (Achille et al., 2019). Neyshabur et al. (2020) show that transfer learning benefits concentrate in lower layers, which carry reusable features across tasks. Loss of plasticity can prevent models from adapting to new distributions during continued training (Dohare et al., 2024; Ke et al., 2023). Layer-freezing interventions (Lee et al., 2019) provide a tool for establishing which layers causally drive a given effect.
3
Method
We compare standard MLM continued pretraining against a two-phase pipeline: CLM detour followed by MLM decay (Figure 1a). 3.1
Models
All encoder models use the ModernBERT architecture (Warner et al., 2025), which combines FlashAttention (Dao et al., 2022), rotary positional embeddings (Su et al., 2024), alternating local/global attention, and unpadding for 8,192-token sequences. We use two sizes: Base (22 layers, 768 hidden, 12 heads, ∼150M parameters) and Large (28 layers, 1024 hidden, 16 heads, ∼350M parameters). For French we start from ModernCamemBERT (Antoun et al., 2025); for English from ModernBERT (Warner et al., 2025). As a decoder control (§5.4), we use Gemma-3 (270M) (Gemma Team, 2025). To train this decoder with MLM, we remove the causal attention mask, add a <mask> token to its vocabulary, and train with 30% masking using the same language model head without the autoregressive position shift. All weights carry over when restoring the causal mask for decay. 3.2
Training Pipeline
The CLM detour consists of two phases. In Phase 1, we replace the bidirectional attention mask with a causal mask and train with next-token prediction. In Phase 2 (decay), we restore bidirectional attention and train with MLM at 15% masking (the original pretraining rate of ModernBERT) for 10% of the Phase 1 budget. The optimizer state is kept between phases; only the learning rate scheduler resets. The model architecture is identical between CLM and MLM: only the attention mask (causal vs. bidirectional) and loss computation (all 3
Preprint. Under review.
tokens vs. masked √tokens) differ. Phase 2 decays the learning rate from peak to 10% of peak following the 1− t/T schedule of Warner et al. (2025), without warmup. The MLM baseline follows the same two-phase structure with 30% masking in Phase 1 (following Warner et al., 2025) and 15% in Phase 2, identical schedule and optimizer. The only difference is the Phase 1 objective (CLM vs. MLM). Data. For French, we compile 10B tokens from four sources. The main source (7B tokens) is French biomedical literature (scientific articles, clinical guidelines, and medical theses), where each paragraph is scored for educational value and content richness using an LLM (Qwen3-235B), and articles are upsampled based on their proportion of high-scoring paragraphs, following FineWeb-Edu (Penedo et al., 2024) and Biomed-Enriched (Touchent et al., 2025). The remaining sources are synthetic medical QA from French coding systems (2B), clinical cases from the European Clinical Case Corpus (E3C; Magnini et al., 2020) (400M), and drug package inserts from the European Medicines Agency (600M). For English at the 50B scale, we mix biomedical literature from Biomed-Enriched (Touchent et al., 2025) (60%, PMC Open Access articles filtered by educational value), medical instruction-following datasets (20%), and MIMIC-III clinical notes (20%), trained for a single epoch. A smaller 10B English variant uses Biomed-Enriched with clinical upsampling (80%) and medical instructions (20%), without MIMIC. Training details. French Base trains for 10B tokens in Phase 1 and 1B in decay; French Large for respectively 25B and 2.5B. English Base is trained at two scales (10B and 50B Phase 1, with proportional decay), and English Large at 50B. All runs use decoupled AdamW with peak lr 2 × 10−4 , β 1 = 0.9, β 2 = 0.98, weight decay 10−5 , and a global batch size of 384 sequences (∼3.1M tokens). Phase 1 uses linear warmup over 100M tokens then constant learning rate. Documents are packed into 8,192-token sequences with endof-sequence tokens between documents; attention is not masked across document boundaries. Training uses bf16 mixed precision on 4×H100 GPUs with Composer (Team, 2022). 3.3
Freeze Interventions
We run three freeze experiments on the 22-layer French Base model (10B CLM phase, 1B decay), where the CLM-MLM gap is largest (+2.8pp), to test which layers carry the CLM benefit. In each experiment, a contiguous block of layers has its parameters frozen (gradients zeroed, parameters unchanged) during either the CLM phase or the decay phase, while remaining layers train normally. We split the 22 layers into low (0–7) and mid (8–14), approximately the first and second thirds of the network. • Experiment 1 (low layers freeze, CLM phase): Layers 0–7 frozen during the CLM phase, then normal decay. Tests whether allowing modifications on low layers during CLM is necessary for the downstream benefit. • Experiment 2 (low layers freeze, decay phase): Normal CLM phase, then layers 0–7 frozen during decay. Tests whether low-layer CLM changes persist through decay even without further updates. • Experiment 3 (mid layers freeze, CLM phase): Layers 8–14 frozen during the CLM phase, then normal decay. Together with Experiment 1, this tests selectivity: if freezing low layers eliminates the CLM benefit while freezing mid layers preserves it, the effect specifically requires low-layer modifications. The freeze is implemented by zeroing gradients for the specified layers after each backward pass. 3.4
CKA Methodology
We measure representational similarity with linear Centered Kernel Alignment (CKA; Kornblith et al., 2019). CKA measures how similar two sets of representations are: 1 means 4
Preprint. Under review.
Table 1: French biomedical downstream results (macro F1, 9 seeds each). Bold: best per column; underline: second best. Multilabel Classification Ctx
FrACCO-30
FrACCO-100
ModernCamemBERT DrBERT CamemBERT-bio CamemBERT
8192 512 512 512
70.1 53.0 41.9 40.8
55.3 35.6 20.1 19.4
MLM baseline CLM detour
8192 8192
69.9 74.8
56.8 60.1
MLM baseline CLM detour
8192 8192
79.4 80.7
63.3 65.4
CANTEMIST
DISTEMIST
Baselines 63.3 20.2 37.9 21.4 12.8 9.6 11.9 9.5 Our Models (Base, 150M) 64.9 23.5 71.0 25.5 Our Models (Large, 350M) 72.6 29.1 74.4 30.4
CLS
NER
MedDialog
DiaMED
EMEA
Medline
Avg
60.6 63.6 38.6 37.4
56.4 57.0 47.7 40.6
68.0 69.6 70.8 69.5
59.7 62.8 65.2 62.7
56.7 50.1 38.3 36.5
62.5 63.6
63.4 67.4
68.5 68.6
61.4 61.9
58.9 61.6
64.5 64.5
61.2 64.8
70.4 70.3
63.5 63.1
63.0 64.2
identical structure, 0 means no linear relationship. We compute layer-by-layer CKA between model pairs and report divergence (1 − CKA), so that higher values indicate greater representational difference. All CKA computations use float64 arithmetic. For French, we use 500 held-out texts drawn from the DiaMED clinical case corpus and the FrACCO oncology report corpus (both described in §3.5); for English, we use PubMed abstracts. Results are averaged over 3 random seeds (42, 43, 44) for data sampling. To isolate CLM-specific changes from noise introduced by training stochasticity, we compute a seed-noise control: two MLM models trained with different random seeds (17 and 42) but identical data order, so they differ only in dropout and masking patterns. Any divergence exceeding this control can be attributed to the training objective rather than to stochastic variation. 3.5
Evaluation Protocol
We evaluate on 8 French and 11 English biomedical tasks (Table 8 in Appendix E), using 9 seeds (42–50) for French and 5 for English. All results use macro-averaged F1 per task, averaged across seeds. French baselines include ModernCamemBERT (Antoun et al., 2025), DrBERT (Labrak et al., 2023), CamemBERT-bio (Touchent and de la Clergerie, 2024), and CamemBERT (Martin et al., 2020). English baselines include PubMedBERT (Gu et al., 2021), BioBERT (Lee et al., 2020), SciBERT (Beltagy et al., 2019), and BioClinical-ModernBERT (Sounack et al., 2025).
4
Experiments
4.1
French Biomedical Evaluation
Table 1 presents the French results. For Base, the CLM detour achieves 61.6% average F1, outperforming the MLM baseline on all 8 tasks (+2.8pp). For Large, CLM reaches 64.2% versus 63.0% for MLM (+1.2pp). CamemBERT-bio and CamemBERT average 38% and 37% overall, limited by their 512-token context on long clinical documents. We release the CLM-detour models as ModernCamemBERT-bio (Base and Large). 4.2
English Biomedical Evaluation
Table 2 shows the English results at three scales (Base 10B, Base 50B, and Large 50B). CLM outperforms MLM on average, with the gap widening at Large scale (+0.8pp, 7/11 task wins) compared to Base 10B (+0.5pp) and Base 50B (+0.3pp). The English effect is smaller than in French, with CLM winning 7 of 11 tasks at each Base scale. Baselines include ModernBERT-base (our starting checkpoint), PubMedBERT (512 context), and BioClinicalModernBERT (8192, standard MLM CPT). PubMedBERT scores higher on short-context BigBIO NER tasks where full-PubMed pretraining helps, but scores 52% on Phenotype (long-context) versus 61% for our models. The smaller English gain is expected. The CLM detour works by reshaping low layers to encode domain-specific features (§5.2); when the base model has already seen biomedical 5
Preprint. Under review.
Table 2: English biomedical results across 11 benchmarks (5 seeds each). Base models trained at 10B and 50B token scales; Large at 50B. Bold: best per column; underline: second best. Task abbreviations in Table 8. Clinical Tasks Ctx
BigBIO Tasks
ChemPr
Pheno
COS
Social
DEID
AnatEM
BC5CDR
JNLPBA
NCBI
GAD
HoC
Cls
Cls
NER
NER
NER
NER
NER
NER
NER
Cls
Cls
48.4 52.0 60.7
87.9 89.7 88.7
74.3 74.9 74.8
77.7 82.1 78.7
76.8 79.3 75.8
66.6 71.0 67.0
74.9 77.0 77.0
88.3 89.1
74.9 74.2
79.2 80.4
78.6 79.3
68.9 69.4
77.3 77.8
88.7 89.1
74.9 74.5
80.3 80.1
78.7 78.8
67.1 70.0
77.7 78.0
89.4 89.8
75.5 75.3
81.8 81.7
76.4 79.7
67.8 69.3
77.9 78.7
ModernBERT-base PubMedBERT BioClinical-ModernBERT
8192 512 8192
89.5 90.2 90.0
MLM baseline CLM detour
8192 8192
90.4 90.3
MLM baseline CLM detour
8192 8192
90.5 90.1
MLM baseline CLM detour
8192 8192
90.5 90.4
Baselines 94.0 53.1 78.3 77.2 95.0 48.7 80.4 83.3 94.8 56.0 81.8 79.2 Our Models — Base 150M (10B tokens) 61.3 94.4 55.5 79.3 78.9 61.0 94.3 55.7 82.6 79.9 Our Models — Base 150M (50B tokens) 61.6 95.1 55.7 81.6 80.0 61.9 95.2 54.2 83.2 81.0 Our Models — Large 350M (50B tokens) 61.0 94.9 55.0 82.3 82.0 61.3 94.7 56.5 84.2 83.2
Avg
text during pretraining, there is less to reshape. ModernBERT was pretrained on web documents, code, and scientific literature following standard modern data mixtures (Warner et al., 2025), which commonly include biomedical corpora such as PubMed. Its low layers already partially encode biomedical features, leaving less room for the CLM detour to help. ModernCamemBERT, by contrast, was pretrained on general French web text without biomedical sources (Antoun et al., 2025), so the domain gap is larger and the CLM benefit correspondingly stronger (+2.8pp vs +0.3pp). This suggests that the CLM detour would show larger gains on any domain absent from the base model’s pretraining data. We release the English models as ModernBERT-bio. 4.3
Optimal Decay Ratio
We sweep the decay length from 2.5% to 50% of the CLM phase at two scales (Base 10B and Large 25B, evaluated on 3 French tasks). At both scales, 10% decay gives the best performance; shorter decay (2.5–4%) scores 1.0pp below optimal and longer decay (20–50%) provides no additional benefit (Table 7 in Appendix D). This ratio matches the cooldown lengths used in language model pretraining (Hu et al., 2024; Hägele et al., 2024). 4.4
Position and Length Analysis
Long-context encoding is especially important in biomedical NLP, where clinical documents such as electronic health records, discharge summaries, and oncology reports routinely span thousands of tokens, and where key information (diagnoses, coding labels) can appear anywhere in the document. The CLM detour also improves how the encoder integrates information across long documents. During MLM decay, only 15% of positions receive a training signal; positions that are rarely masked may be poorly encoded. CLM, which trains on every position, should produce representations that are more uniform across the sequence. We test this with a needle-in-haystack evaluation: we insert a synthetic French medical fact into a biomedical document at a controlled position (start, middle, end) and length (512–8192 tokens), and freeze each encoder to probe whether the fact can be detected from the CLS representation (binary accuracy; details in Appendix I). CLM outperforms MLM at every context length and position (+10.7pp overall, Figure 2). Two patterns stand out. First, MLM accuracy degrades monotonically with document length (57% at 512 tokens → 43% at 8192), while CLM degrades more slowly and remains above MLM throughout, indicating that CLM representations retain information better over long distances. Second, the CLM advantage is largest at mid-document positions, where the inserted fact is farthest from both the start and end of the sequence. This is the hardest retrieval setting, because the CLS token must integrate information from the middle of a long context, and it is where the difference in position-level training signal matters most. These results are consistent with the downstream pattern: the French tasks with the largest CLM gains (FrACCO, CANTEMIST) require sequence lengths of 4096 to6
Preprint. Under review.
(a) By context length
Accuracy (%)
80
70 64.3
70 61.2
60.0
60 50
(b) By needle position
76.3
80
52.9
57.1
55.1
52.6
40 512
1024
2048
63.3
60
54.9
50 53.3
51.1
CLM detour MLM baseline
70.3
43.1 8192
4096
Context length (tokens)
52.7
49.5
40 Start
Middle
Needle position
End
Figure 2: Needle-in-haystack evaluation. CLM outperforms MLM at all context lengths and needle positions. Overall: CLM 62.2% vs MLM 51.6% (+10.7pp).
(a) CKA divergence during MLM decay
60 55 50 Seed noise
45 0
2
4
6
8
MLM Decay (B tokens)
(b) Per-layer CLM effect
50
Divergence ratio
Divergence (%)
65
40 30 20 10 0
10
seed noise
0
7
Layer
14
21
Figure 3: (a) CKA divergence between CLM and MLM models during decay (Base 150M). Dashed line: seed-noise baseline. (b) Ratio rl of CLM-MLM divergence to seed-noise divergence for each layer. A ratio of 1 means no CLM-specific effect. kens during fine-tuning (Table 8), indicating that their input documents are long enough for mid-document context integration to matter.
5
Analysis
We now investigate why the CLM detour improves downstream performance, using the CKA methodology described in §3.4. 5.1
The CLM Imprint Persists
The CLM phase modifies low-layer representations in a way that subsequent MLM training does not undo, even when the decay budget matches the CLM phase. We measure it via CKA divergence between matched CLM-detour and MLM-only models after identical decay, comparing against a seed-noise control: two MLM models trained with different random seeds on identical data and hyperparameters. The CLM imprint is lasting. CKA divergence reaches ∼56.5% after 1.5B tokens of MLM decay and remains stable, with only 0.6pp variation over 8B additional tokens (Figure 3a). The seed-noise control shows 49.7% divergence overall, confirming that CLM adds signal beyond stochastic training variation. 7
Preprint. Under review.
Table 3: Layer transplantation on DiaMED (3 seeds, linear probing). No transplant recovers more than 35% of the 7.1pp gap. ∆
Transplant
F1 (%)
MLM baseline CLM detour
References 32.8 — 39.9 +7.1
Recovery 0% 100%
Layer group transplants Low (0–7) 31.9 −0.9 −12% Mid (8–14) 35.3 +2.5 +35% Late (15–21) 32.4 −0.4 −6%
The CLM imprint concentrates in low layers. Both CLM and MLM continued pretraining modify all layers relative to the starting checkpoint, and mid/deep layers diverge heavily under either objective (Appendix F). To isolate where CLM differs from MLM specifically, we normalize each layer’s CLM-MLM divergence by the seed-noise baseline (Figure 3b): s
rℓ =
1 − CKA(CLMℓ , MLMℓ1 )
(s1 , s2 : different random seeds)
s 1 − CKA(MLMsℓ2 , MLMℓ1 )
A ratio of 1 means no CLM-specific effect at that layer. Low layers (0–7) show ratios of 5–44×: CLM changes them far more than random seed variation does. Mid and deep layers are near 1×: both objectives modify them similarly. We test whether these low-layer changes are the causal driver of downstream improvement in §5.2. 5.2
Causal Evidence: Freeze Interventions
Do these low-layer changes actually drive the downstream improvement? We answer with the freeze interventions described in §3.3 (Figure 1b; full results in Table 5 in Appendix B). Freezing low layers (0–7) during CLM drops F1 from 61.6% to 59.3%. We cannot reject the null hypothesis that the resulting model matches the MLM baseline (p = 0.25, paired bootstrap across 8 tasks × 9 seeds). Freezing mid layers (8–14) during CLM preserves the benefit (61.2%). Together, these two experiments show that low layers are necessary and mid layers are not. Freezing low layers during decay has negligible effect (−0.4pp, 61.2%), confirming that MLM decay already preserves the CLM imprint in low layers. These experiments use French Base models. The CKA patterns that motivate them (low-layer divergence well above seed noise) are consistent across French Base, French Large, and English Base (Table 4). 5.3
Non-Localizability
The freeze experiments show that low layers are necessary during training. We now ask whether the resulting benefit can be localized after training. We copy full parameter blocks (self-attention, MLP, and layer norms) for a group of layers from the CLM model into the MLM model, keeping embeddings and remaining layers from MLM, and evaluate via linear probing on frozen representations (logistic regression, 3 seeds). We use DiaMED, the task with the largest CLM-MLM gap (7.1pp), for clearer signal. No transplant recovers more than 35% of the gap (Table 3). Low and late layer transplants hurt performance. Inserting CLM low layers into an MLM model creates a mismatch with the mid and deep layers that co-adapted under MLM, yielding representations worse than either pure model. Component-level transplants (attention-only, MLP-only) show a similar pattern (Appendix C). Unlike freeze interventions, which act during training, transplants replace already-trained weights. The freeze experiments show low layers are necessary during training; the transplant experiments show the resulting benefit cannot be isolated post hoc. 8
Preprint. Under review.
Table 4: CKA divergence by model size and architecture. The encoder-decoder comparison uses different model families (see §5.4). Model Params Div. CI Encoders (MLM→CLM→MLM) ModernCamemBERT Base 150M 56.5% ModernCamemBERT Large 350M 67.2%
±0.7% ±0.3%
Decoder (CLM→MLM→CLM) 270M 26.3%
±1.0%
Gemma-3
5.4
Scaling and Architectural Asymmetry
The CLM imprint increases with model capacity (Table 4): Large (350M) reaches 67.2% ± 0.3% divergence versus 56.5% ± 0.7% for Base (150M). We also test the reverse on a decoder: starting from Gemma-3 (270M; Gemma Team, 2025), we remove the causal attention mask and train with MLM for 10B tokens, then restore the causal mask and return to CLM for 1B tokens (details in §3.1). The decoder retains only 26.3% ± 1.0% divergence versus 67.2% for an encoder of similar scale, a suggestive 2.6× asymmetry. One explanation is the density of the decay signal: MLM decay at 15% masking provides weak erasure, while CLM decay at 100% provides strong erasure. All confidence intervals are 95% over 3 seeds. This comparison is exploratory: it is confounded by model family and language, and a controlled comparison using encoder and decoder variants of the same family would be needed to confirm the asymmetry.
6
Conclusion
A temporary detour through causal language modeling during encoder continued pretraining produces lasting representational changes that improve downstream performance. The effect is not an artifact of additional compute: matched MLM baselines with identical data and training budget score lower. The CLM imprint concentrates in low transformer layers, persists through extended MLM decay, and is not localized to a single layer group and is not erased by continued training. Freeze interventions show that low-layer changes are necessary for the downstream improvement, while mid-layer changes are not. As a practical guideline, MLM decay of roughly 10% of the detour length suffices to stabilize the model without erasing the CLM imprint. These findings suggest that the sequence of training objectives shapes the final model, not just the total compute or the final objective. We speculate that this mechanism extends beyond biomedical text to other domain adaptation settings where the target domain differs substantially from the pretraining distribution. The stronger effect in French (+2.8pp) compared to English (+0.3–0.8pp depending on model size) suggests that lower-resource biomedical languages such as Italian, Spanish, or Portuguese, where less domain text is available during general pretraining, would likely benefit more from the CLM detour. We release ModernCamemBERT-bio and ModernBERT-bio as open biomedical encoders in Base and Large sizes with 8,192-token context.
References Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep networks. In Proceedings of ICLR, 2019. Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. Publicly available clinical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 72–78, 2019. 9
Preprint. Under review.
Wissam Antoun, Benoı̂t Sagot, and Djamé Seddah. ModernBERT or DeBERTaV3? Examining architecture and data influence on transformer encoder models performance. In Proceedings of the 4th International Joint Conference on Natural Language Processing and the 15th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL), 2025. Simon Baker, Ilona Silins, Yufan Guo, et al. Automatic semantic classification of scientific literature according to the hallmarks of cancer. Bioinformatics, 32(3):432–440, 2016. Iz Beltagy, Kyle Lo, and Arman Cohan. SciBERT: A pretrained language model for scientific text. In Proceedings of EMNLP-IJCNLP, pages 3613–3618, 2019. Àlex Bravo, Janet Piñero, Núria Queralt-Rosinach, et al. Extraction of relations between genes and diseases from text and large-scale data analysis. BMC Bioinformatics, 16:55, 2015. Nigel Collier, Tomoko Ohta, Yoshimasa Tsuruoka, Yuka Tateisi, and Jin-Dong Kim. Introduction to the bio-entity recognition task at JNLPBA. In Proceedings of NLPBA/BioNLP, pages 73–78, 2004. Jean-Philippe Corbeil, Amin Dada, Jean-Michel Attendu, et al. A modular approach for clinical SLMs driven by synthetic data. In Proceedings of ACL, 2025. Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with io-awareness. In Advances in Neural Information Processing Systems, volume 35, pages 16344–16359, 2022. Rezarta Islamaj Doğan, Robert Leaman, and Zhiyong Lu. NCBI disease corpus: A resource for disease name recognition and concept normalization. Journal of Biomedical Informatics, 47:1–10, 2014. Shibhansh Dohare, J. Fernando Hernandez-Garcia, Qingfeng Lan, Parash Rahman, A. Rupam Mahmood, and Richard S. Sutton. Loss of plasticity in deep continual learning. Nature, 632(8026):768–774, 2024. Sebastian Gehrmann, Franck Dernoncourt, Yeran Li, et al. Comparing deep learning and concept extraction based methods for patient phenotyping from clinical narratives. PLoS ONE, 13(2):e0192360, 2018. Gemma Team. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025. Hippolyte Gisserot-Boukhlef, Nicolas Boizard, Manuel Faysse, Duarte M. Alves, Emmanuel Malherbe, André F. T. Martins, Céline Hudelot, and Pierre Colombo. Should we still pretrain encoders with masked language modeling? arXiv preprint arXiv:2507.00994, 2025. Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specific pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare, 3(1):1–23, 2021. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of ACL, pages 8342–8360, 2020. Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro Von Werra, and Martin Jaggi. Scaling laws and compute-optimal training beyond fixed training durations. In Advances in Neural Information Processing Systems, volume 37, 2024. Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. MiniCPM: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395, 2024. 10
Preprint. Under review.
Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. Continual pre-training of language models. In Proceedings of ICLR, 2023. Prescott Klassen, Fei Xia, Lucy Vanderwende, and Meliha Yetisgen. Annotating clinical events in text snippets for phenotype detection. In Proceedings of LREC, pages 2753–2757, 2014. Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In Proceedings of ICML, pages 3519–3529, 2019. Martin Krallinger, Obdulia Rabal, Saber A. Akhondi, et al. Overview of the BioCreative VI chemical-protein interaction track. In Proceedings of the Sixth BioCreative Challenge Evaluation Workshop, pages 141–146, 2017. Yanis Labrak, Adrien Bazoge, Richard Dufour, Mickael Rouvier, Emmanuel Morin, Béatrice Daille, and Pierre-Antoine Gourraud. DrBERT: A robust pre-trained model in French for biomedical and clinical domains. In Proceedings of ACL, pages 16207–16221, 2023. Yanis Labrak, Adrien Bazoge, Oumaima El Khettari, Mickael Rouvier, Pacôme Constant Dit Beaufils, Natalia Grabar, Béatrice Daille, Solen Quiniou, Emmanuel Morin, PierreAntoine Gourraud, and Richard Dufour. DrBenchmark: A large language understanding evaluation benchmark for French biomedical domain. In Proceedings of LREC-COLING, pages 5377–5392, 2024. Jaejun Lee, Raphael Tang, and Jimmy Lin. What would elsa do? freezing layers during transformer fine-tuning. arXiv preprint arXiv:1911.03090, 2019. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240, 2020. Simon A. Lee, Anthony Wu, and Jeffrey N. Chiang. Clinical ModernBERT: An efficient and long context encoder for biomedical text. arXiv preprint arXiv:2504.03964, 2025. Jiao Li, Yueping Sun, Robin J. Johnson, et al. BioCreative V CDR task corpus: A resource for chemical disease relation extraction. Database, 2016:baw068, 2016. Yusheng Liao, Chaoyi Wu, Junwei Liu, et al. EHR-R1: A reasoning-enhanced foundational language model for electronic health record analysis. arXiv preprint arXiv:2510.25628, 2025. Xingyu Liu, Vincent Segonne, Aidan Mannion, Didier Schwab, Lorraine Goeuriot, and François Portet. MedDialog-FR: a French version of the MedDialog corpus for multilabel classification and response generation related to women’s intimate health. In First Workshop on Patient-Oriented Language Processing at LREC-COLING, 2024. Kevin Lybarger, Meliha Yetisgen, and Özlem Uzuner. The 2022 n2c2/UW shared task on extracting social determinants of health. Journal of the American Medical Informatics Association, 30(8):1367–1378, 2023. Bernardo Magnini, Begona Altuna, Alberto Lavelli, Manuela Speranza, and Roberto Zanoli. The E3C project: European clinical case corpus. In Proceedings of CLiC-it, 2020. Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoı̂t Sagot. CamemBERT: a tasty french language model. In Proceedings of ACL, pages 7203–7219, 2020. Amil Merchant, Elahe Rahimtoroghi, Ellie Pavlick, and Ian Tenney. What happens to BERT embeddings during fine-tuning? In Proceedings of BlackboxNLP, pages 33–44, 2020. 11
Preprint. Under review.
Antonio Miranda-Escalada, Eulàlia Farré-Maduell, and Martin Krallinger. Named entity recognition, concept normalization and clinical coding: Overview of the CANTEMIST track for cancer text mining in Spanish, corpus, guidelines, methods and results. In Proceedings of the Iberian Languages Evaluation Forum (IberLEF), 2020. Antonio Miranda-Escalada, Luis Gasco, Salvador Lima-López, Eulàlia Farré-Maduell, Darryl Estrada, Anastasios Nentidis, Anastasia Krithara, Georgios Katsimpras, Georgios Paliouras, and Martin Krallinger. Overview of DisTEMIST at BioASQ: Automatic detection and normalization of diseases from clinical texts. In Working Notes of CLEF, 2022. Ishna Neamatullah, Margaret M. Douglass, Li-wei H. Lehman, et al. Automated deidentification of free-text medical records. BMC Medical Informatics and Decision Making, 8(1):32, 2008. Aurélie Névéol, Cyril Grouin, Jeremy Leixa, Sophie Rosset, and Pierre Zweigenbaum. The QUAERO French medical corpus: A resource for medical entity recognition and normalization. In Proceedings of BioTextMining Workshop, pages 24–30, 2014. Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? In Advances in Neural Information Processing Systems, volume 33, pages 512–523, 2020. Guilherme Penedo, Hynek Kydlı́ček, Anton Lozhkov, Margaret Mitchell, Thomas Colin, Yacine Jernite, and Thomas Wolf. FineWeb: Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406.17557, 2024. Johann Pignat, Milena Vucetic, Christophe Gaudet-Blavignac, et al. FRACCO: A goldstandard annotated corpus of oncological entities with ICD-O-3.1 normalisation. arXiv preprint arXiv:2510.13873, 2025. Sampo Pyysalo and Sophia Ananiadou. Anatomical entity mention recognition at literature scale. Bioinformatics, 30(6):868–875, 2014. Thomas Sounack, Joshua Davis, Brigitte Durieux, Antoine Chaffin, Tom J Pollard, Eric Lehman, Alistair E W Johnson, Matthew McDermott, Tristan Naumann, and Charlotta Lindvall. BioClinical ModernBERT: A state-of-the-art long-context encoder for biomedical and clinical NLP. arXiv preprint arXiv:2506.10896, 2025. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. Yu Sun, Xingyu Qian, Weiwen Xu, et al. ReasonMed: A 370k multi-agent generated dataset for advancing medical reasoning. arXiv preprint arXiv:2506.09513, 2025. MosaicML NLP Team. Composer: A library for efficient deep learning training. 2022. URL https://github.com/mosaicml/composer. Rian Touchent and Éric de la Clergerie. CamemBERT-bio: Leveraging continual pretraining for cost-effective models on French biomedical data. In Proceedings of LRECCOLING, pages 2692–2701, 2024. Rian Touchent, Nathan Godey, and Eric de la Clergerie. Biomed-enriched: A biomedical dataset enriched with LLMs for pretraining and extracting rare and hidden content. arXiv preprint arXiv:2506.20331, 2025. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Griffin Thomas Adams, Jeremy Howard, and Iacopo Poli. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. In Proceedings of ACL, pages 2526–2547, 2025. 12
Preprint. Under review.
Orion Weller, Kathryn Ricci, Marc Marone, Antoine Chaffin, Dawn Lawrie, and Benjamin Van Durme. Seq vs seq: An open suite of paired encoders and decoders. arXiv preprint arXiv:2507.11412, 2025. Chaoyi Wu, Pengcheng Qiu, Jinxin Liu, et al. Towards evaluating and building versatile large language models for medicine. npj Digital Medicine, 8(1):58, 2025. John Wu, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. Similarity analysis of contextual word representation models. In Proceedings of ACL, pages 4638–4655, 2020. Xinru Yu, Bin Guo, Shiwei Luo, Jie Wang, Tao Ji, and Yuanbin Wu. AntLM: Bridging causal and masked language models. In Proceedings of the 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning, pages 324–331, 2024. Kaiyan Zhang, Sihang Zeng, Ermo Hua, et al. UltraMedical: Building specialized generalists in biomedicine. In Advances in Neural Information Processing Systems, volume 37, 2024.
Limitations Domain and language scope. All experiments use biomedical text in French and English. The CLM detour gains are larger in French (+2.8pp, 8/8 task wins) than in English (+0.3– 0.8pp, 7/11 task wins), consistent with the larger domain gap for French. The gap narrows from 10B to 50B tokens in English, suggesting that additional data partially closes it. We have not tested whether the effect generalizes to other domains or languages. Scale and architecture. Our largest model has 350M parameters. The CLM imprint increases with scale (56.5% for Base, 67.2% for Large), but we have not verified this trend at billion-parameter scale. The encoder-decoder asymmetry (2.6×) compares models that differ in architecture, language, and pretraining data; a controlled comparison using encoder and decoder variants of the same model family would provide stronger evidence. Freeze experiments. The freeze interventions were conducted on French Base models only, where the CLM-MLM gap is largest. Replicating on English or Large models would strengthen the generalization of the causal claim. The freeze granularity (7–8 layer blocks) is coarse; finer-grained interventions could narrow down the critical layers further.
A
English Training Data
The English biomedical component (60% of the 50B mix) uses the Biomed-Enriched dataset (Touchent et al., 2025), a curated version of PMC Open Access articles where each paragraph is scored for educational value using an LLM classifier. We retain paragraphs scoring ≥3.0. For the 10B variant, clinical case paragraphs are upsampled 100× and other clinical documents 10× to increase clinical representation. The instruction-following component (20%) consists of five datasets: MedS-Ins (Wu et al., 2025), ReasonMed (Sun et al., 2025), MediFlow (Corbeil et al., 2025), UltraMedical (Zhang et al., 2024), and EHR-Ins-Reasoning (Liao et al., 2025). The 10B variant uses only MediFlow. The clinical component (20%, 50B only) uses MIMIC-III discharge summaries and clinical notes, upsampled across multiple passes to reach the target token count. 13
Preprint. Under review.
B
Freeze Intervention Results
Table 5: Freeze intervention results (French Base, 8 tasks, 9 seeds). Condition Avg F1 ∆ vs CLM detour Freeze L0–7 (CLM phase) Freeze L8–14 (CLM phase) Freeze L0–7 (decay phase)
C
−2.3 −0.4 −0.4
59.3 61.2 61.2
Component Transplant Results
Table 6: Component-level transplants on DiaMED (3 seeds, linear probing).
D
Transplant
F1 (%)
∆
Recovery
All attention All MLP
33.9 35.3
+1.1 +2.5
+15% +35%
Decay Ratio Sweep
Table 7: Decay ratio sweep on 3 French tasks (DiaMED, FrACCO-30, FrACCO-100; 9 seeds each). 10% decay is optimal at both scales. Decay tokens 0.25B 1.0B 2.0B 3.0B 5.0B 1.0B 2.5B 5.0B
Decay / Phase 1
CLM+decay F1
10B Phase 1 (Base, 150M) 2.5% 71.6% 10% 72.6% 20% 71.9% 30% 72.1% 50% 71.7% 25B Phase 1 (Large, 350M) 4% 73.2% 10% 73.5% 20% 72.7%
14
MLM+decay F1 69.1% 70.4% 70.4% 69.7% 70.4% 70.1% 70.5% 70.1%
Preprint. Under review.
E
Evaluation Tasks Table 8: Evaluation tasks. Max seq. length reflects the fine-tuning configuration.
Task
Type
Lang
Max len.
DiaMED MedDialog-FR FrACCO (30/100) CANTEMIST DisTEMIST QUAERO (EMEA/Medline)
Classification Multilabel Multilabel Multilabel Multilabel NER
FR FR FR FR FR FR
2048 4096 4096 4096 4096 1024
(Labrak et al., 2024) (Liu et al., 2024) (Pignat et al., 2025) (Miranda-Escalada et al., 2020) (Miranda-Escalada et al., 2022) (Névéol et al., 2014)
ChemProt Phenotyping COS Social History De-identification AnatEM BC5CDR JNLPBA NCBI Disease GAD HoC
Relation Extr. Classification NER NER NER NER NER NER NER Classification Classification
EN EN EN EN EN EN EN EN EN EN EN
512 8192 512 512 8192 512 512 512 512 512 512
(Krallinger et al., 2017) (Gehrmann et al., 2018) (Klassen et al., 2014) (Lybarger et al., 2023) (Neamatullah et al., 2008) (Pyysalo and Ananiadou, 2014) (Li et al., 2016) (Collier et al., 2004) (Doğan et al., 2014) (Bravo et al., 2015) (Baker et al., 2016)
F
Raw CKA Divergence Curves 100
Divergence (%)
Reference
CLM vs MLM Seed noise (MLM vs MLM)
80 60 40 20 0
0
7
Layer
14
21
Figure 4: Per-layer CKA divergence for CLM vs MLM (coral) and seed noise (gray). Both continued pretraining objectives modify mid and deep layers heavily, but only CLM produces large changes in low layers (0–7). The shaded area is the CLM-specific signal, which the divergence ratio in Figure 3b normalizes.
G
CKA Formula
We use linear CKA (Kornblith et al., 2019). Given two representation matrices X ∈ Rn× p and Y ∈ Rn×q (one per model, n samples, mean-pooled over tokens), we center each column, compute Gram matrices K = XX ⊤ and L = YY ⊤ , and define: CKA( X, Y ) = p 15
tr(KL) tr(K2 ) tr( L2 )
Preprint. Under review.
We report divergence d = 1 − CKA, so higher values indicate greater representational difference. All computations use float64 arithmetic.
H
Fine-tuning Protocol
French tasks. Classification (DiaMED) and multilabel tasks (MedDialog, FrACCO, CANTEMIST, DisTEMIST) use a linear head on the CLS token, trained for 15 epochs with lr = 2 × 10−5 , batch size 4, and 10% warmup. Multilabel tasks use max sequence length 4096 with weighted BCEWithLogitsLoss; classification uses 2048. NER tasks (EMEA, Medline) use a span-based architecture with BiLSTM, CRF, and biaffine scorer, trained for 4000 steps with lr = 5 × 10−5 for the encoder and 10−3 for the head, batch size 16, max length 1024. All tasks use AdamW with weight decay 0.01 and select the best checkpoint by validation F1. English tasks. We follow the BioClinical-ModernBERT evaluation protocol (Sounack et al., 2025): lr = 5 × 10−5 , weight decay 0.01, batch size 16, 10 epochs for most tasks (20 for NER). All models are fine-tuned with the same hyperparameters per task.
I
Needle-in-Haystack Protocol
We design a synthetic needle-in-haystack task in French to evaluate long-context retrieval. A medical fact (the “needle”) is inserted into a clinical document (the “haystack”) at a controlled position, and the model must determine whether a given query fact is present (binary classification). Negative examples use the same template with different slot values, so the task requires precise retrieval rather than shallow pattern matching. Fact templates. We define 8 French medical fact templates covering drugs, vital signs, lab results, diagnoses, procedures, allergies, treatment duration, and symptoms. Each template has 2–4 slots filled from small lexicons. Examples: • Le patient a reçu 200 mg de morphine par voie intraveineuse. Distractor: Le patient a reçu 500 mg de tramadol par voie orale. • La tension artérielle mesurée était de 160/90 mmHg. • Le diagnostic retenu est cirrhose hépatique de stade III. • Le patient rapporte une dyspnée évoluant depuis une semaine. Dataset and evaluation. Haystacks are drawn from French biomedical text. We generate 1500 balanced positive/negative pairs across 5 lengths (512–8192 tokens) and 3 positions (start, middle, end), split 70/15/15 for train/validation/test. We freeze each encoder and train a 2-layer MLP probe (Dropout → Linear → GELU → Dropout → Linear) on CLS representations for 3 epochs (lr = 2 × 10−5 , batch size 4, AdamW), selecting the best checkpoint by validation accuracy.
16