Specialists Learn Faster: Knowledge Organization in Pre-training Shapes Knowledge Update Efficiency

Preprint

SPECIALISTS LEARN FASTER: KNOWLEDGE ORGANIZATION IN PRE-TRAINING SHAPES KNOWLEDGE UPDATE EFFICIENCY

arXiv:2610.13835v1 [cs.LG] 9 Oct 2026

Sheng Zhao Department of Computer Science University of Rochester szhao32@ur.rochester.edu

Yuhao Zhu Department of Computer Science University of Rochester yzhu@rochester.edu

ABSTRACT An ideal (language) model must continually update its knowledge. Existing work largely focuses on making these updates efficient after training. We investigate how the organization of time-varying knowledge during pre-training shapes the efficiency of subsequent updates. Models trained on chronologically ordered knowledge specialize in the most recently encountered knowledge; we call them specialists. In contrast, models trained on the same knowledge in randomly shuffled order perform well across all represented time periods; we call them generalists. Surprisingly, specialists require far fewer update steps than generalists to acquire new knowledge. To explain this advantage, we derive a law from first principles that relates update cost to the model’s optimization dynamics and that closely predicts empirical results. Guided by these findings, we design a strategy for organizing pre-training data that preserves stable historical knowledge while enabling efficient updates to current knowledge. We also show that the benefits of temporal knowledge organization extend to real-world pre-trained models and existing knowledge update tools, and generalize to other knowledge orderings. Project page at: https://horizon-lab.org/specialist/.

1

INTRODUCTION

Both generalists and specialists have evolved in nature (Sachdeva et al., 2020). Generalists are fit across a range of environments although they are not particularly optimal for a particular environment. Specialists are well adapted to a particular environment but lack fitness outside their niche. We show that generalists and specialists can emerge in language models, influenced by how knowledge is ordered and presented during training (Section 4). Without losing generality, we focus on biographical knowledge, which has a natural order — chronological order (Section 3). We find that models trained on chronologically ordered knowledge tend to become specialists: they perform best on the most recently encountered knowledge but poorly on knowledge from other periods. In contrast, models trained on the same knowledge in randomly shuffled order tend to become generalists: they perform reasonably well across all periods represented in training. Perhaps surprisingly, specialists acquire new knowledge more efficiently during post-training updates (Section 5). To explain this advantage, we derive a law from first principles that describes knowledgeupdate dynamics in the model’s optimization landscape. The law relates update cost to two quantities: the pre-trained model’s initial gap in acquiring the new knowledge (the knowledge gap) and how quickly the model closes that gap (the update speed). Together with a gradient analysis, the law shows that specialists’ advantage lies primarily in their larger initial gradient magnitudes, which allow them to close the (sometimes larger) knowledge gap much faster than generalists. Our observations have practical values: an ideal (language) model must continually update its knowledge (Vu et al., 2024; Johnson et al., 2025) because, for instance, people change jobs over time or software APIs update across versions. After deployment, a model is usually updated through model editing (Meng et al., 2022; 2023; De Cao et al., 2021; Yao et al., 2023) or continued pre-training (Gururangan et al., 2020; Ibrahim et al., 2024). Most prior work focuses on making these updates 1

Preprint

cheaper and less harmful once the model has already been trained by, for example, meta-learning an editor (Mitchell et al., 2022a), storing updates in external memory (Mitchell et al., 2022b; Hartvigsen et al., 2023), or constraining updates to spare other knowledge (Fang et al., 2025). These post-training approaches apply after a model has learned its knowledge. Our work studies the role of the original training data in update efficiency. Building on our findings, we design a strategy for organizing pre-training data that preserves rarely changing historical knowledge while enabling efficient updates to recent knowledge (Section 6.1). We also show that the benefits of temporal knowledge organization extends to real-world pre-trained models (Section 6.2) and existing knowledge update tools (Section 6.3), and generalize to other knowledge orderings (Section 6.4). In summary, we make three main contributions. • We show that models trained on chronologically ordered knowledge require roughly half as many steps to learn new knowledge as those trained on temporally shuffled knowledge. • We derive a law governing the update cost. The law shows that the temporal order of knowledge reduces update cost primarily by substantially increasing the initial gradient of the model. • Using our findings, we show how to design temporal knowledge organizations to train models that are both generalists and specialists. The benefits extend to pre-trained (real-world) models and existing knowledge update tools, and generalize to other knowledge orderings.

2

RELATED WORK

Time-Varying Knowledge. Temporal benchmarks (Lazaridou et al., 2021; Jang et al., 2022a;b; Loureiro et al., 2022; Vu et al., 2024), studies of outdated knowledge (Zhao et al., 2024), date conditioning (Dhingra et al., 2022) and probes of temporal representations (Gurnee & Tegmark, 2024; Park et al., 2025) ask what a model knows about the past and the present. We instead ask how the history of a changing fact is represented and what it costs to change the fact again. Temporal Order in Training. Data order has been studied as curriculum learning (Bengio et al., 2009; Wu et al., 2021) and, for growing corpora, as chronological or continual pre-training (Jin et al., 2022; Li et al., 2025; Pilchen et al., 2026). We hold the facts fixed and vary only their order, which isolates its effect on representation and update cost.

Table 1: Prior work by stage and task. Red: through data organization and training dynamics; Blue: through architecture or hyperparameters. Ours is the only data organization-level work on efficient editing at pretraining stage.

Pre-training

Knowledge Update. Prior work on knowledge updating differs in when it intervenes and what it changes (see PostTable 1). After pre-training, editors training rewrite facts in place (Meng et al., 2022; 2023; Fang et al., 2025), learn an editor (Mitchell et al., 2022a; Zweiger et al., 2025) or modify the update data (Ovadia et al., 2024; Yang et al., 2025). We ask whether data organization eases later updates.

3

Efficient updating & editing

Catastrophic forgetting

Ours Liu et al. (2025), Kim et al. (2025)

Toneva et al. (2019), Tirumala et al. (2022), Ibrahim et al. (2024) Ke et al. (2023), Dohare et al. (2024)

Ovadia et al. (2024), EntiGraph (Yang et al., 2025), SEAL (Zweiger et al., 2025) ROME (Meng et al., 2022), MEMIT (Meng et al., 2023), MEND (Mitchell et al., 2022a)

Scialom et al. (2022), Luo et al. (2023), Gekhman et al. (2024) EWC (Kirkpatrick et al., 2017), GRACE (Hartvigsen et al., 2023), WISE (Wang et al., 2024)

EXPERIMENTAL SETUP

Temporal biographies. Our dataset extends the synthetic biographies of Allen-Zhu & Li (2023) with a time axis. It generates 25,000 individuals (at 110M; more at larger scales), each with a unique name and six attributes: birthday, birth city, university, field of study, employer and employer city; each attribute has a corresponding value. Each individual appears 1,000 times between 2000 and 2024. In each year, individuals hold a value for each attribute, and some of these values change over time: their birthday and birth city remain unchanged, university and field change a few times, and their employer and employer’s city change more frequently. Each sentence is drawn from about fifty templates for its attribute. An example is shown below (see Appendix A for more details): 2

Preprint

In 2010, John Ivy Bi was born on December 17, 1979 in Moreno Valley, CA. He studied at Bowdoin College and majored in Photography. He worked at Under Armour in San Jose, CA.

In this example, in year Y = 2010, the person “John Ivy Bi” and the attribute “birthday” form a fact identity I, which has a value V = “December 17, 1979”; there are six such identity-value pairs for each person in each year (see Figure 9 for more details). Pre-training organization and models. From the same records we build six pre-training corpora by varying two factors: record order and timestamp inclusion. A chronological order (C) presents each person’s records in the chronological order, sorted by the timestamps, which are the “In {year}, ” prefix of the example above. Records of different people are unordered. A shuffled order (S) shuffles all records in an random order. For each order, we build three variants, one that does not carry the timestamp (i.e., no “In {year}, ” prefix), one carries the timestamp (D), and a third that carries timestamp on randomly chosen half of the records (D1/2 ). This yields six training corpora (C, S, CD, SD, CD1/2 , and SD1/2 ). We pre-train GPT-2 style decoders with rotary position embeddings and the GPT-2 tokenizer at three scales; 110M, 350M, and 682M (see Table 4 for model details). The pre-training loss is the standard cross-entropy over the whole record. Updating. After pre-training, we consider two types of knowledge updates: an overwrite update corrects an existing record, while an injection update adds a record for a new person. Each update is performed by training a LoRA adapter (Hu et al., 2022). Figure 1 shows an overwrite update: the company John Ivy Bi worked at in 2020 is corrected from Google to PepsiCo. This is done by training the adapter on “In 2020, John Ivy Bi worked at PepsiCo.” Before LoRA, Pold (θ0 ), the probability of the old value Google, is the highest, whereas the probability of the new/updated value PepsiCo, Pnew (θ0 ), is very low. After N LoRA steps when model parameters are updated from θ0 to θN , Pnew (θN ) becomes the highest. The fine-tuning uses the cross-entropy loss computed only on the new value, i.e., − log Pnew (θ), and stops when Pnew (θ) passes the competitor’s probability Pold (θ). 1

In 2020, John Ivy Bi worked at ___

Pold (θ0 )

Google Amazon

Pnew (θ0 )

PepsiCo Costco Apple

before

Pold (θN ) LoRA, N steps

Pnew (θN )

(θ) = −log Pnew (θ)

after

Figure 1: Example of an overwrite update. Bars represent the model’s predicted probabilities over an attribute’s values. Starting from initial parameters θ0 , a LoRA adapter is trained for N steps on the loss L(θ) = − log Pnew (θ), until the new value’s probability passes its competitor’s, Pold (the strongest value model held before updating).

Knowledge Query. A query asks for the value of one attribute of a person. We experiment with two forms of knowledge querying. A dated query specifies a year along with a person’s name and an attribute and counts the response as correct if the answer is any value held during that year (since a value can change within a year). An undated query omits the year and expects the latest value.

4

THE EMERGENCE OF GENERALISTS AND SPECIALISTS

This section studies how the temporal organization of knowledge exposed during training shapes what a model learns. We extend the results to other ordering in Section 6.4. 4.1

ORGANIZING TIME-VARYING KNOWLEDGE INFLUENCES GENERALISTS VS. SPECIALISTS

To demonstrate how the organization of time-varying knowledge during training can influence the model, Figure 2 compares the training dynamics across three settings: • On the left, the model is trained on the CD corpus, and constantly being queried at different batches and epochs during training (x-axis) using four dated queries (y-axis) — Year 2005, 2010, 2015, and 2020 — as well as the “current” query, which queries using the knowledge trained in the immediately previous batch. The heatmap represents the query accuracy, averaged over all the queries in the corresponding query category. • The middle figure shows the same training dynamics but using the SD corpus for training. 1

We mark an update successful once Pnew exceeds the competitor, similar to prior model-editing metrics (Meng et al., 2022); Table 5 confirms the same conclusions under stricter criteria empirically.

3

Preprint

(b) SD, query by year

0

(c) SD, query by batch

1.0

2010 2015

0.5

2020 current 0

1 1

2

3

epochs

4

5

0

1

2

3

epochs

4

5

0

1

2

3

epochs

4

5

P(target value)

(a) CD, query by year 2005

0.0

Figure 2: Query accuracy of the 110M CD and SD models over five training epochs; color represents accuracy. (a, b) Query by year (for CD and SD). Rows 2005–2020 query knowledge of a year (e.g., “In 2005, …”), and the “current” row queries knowledge from the most recent batch the model is trained on. (c) Query by batch (for SD). Each row queries knowledge from one of the ten batches in the training corpus, from the first batch (0) to the last (1). See Appendix A for details. • Finally, the last figure shows the dynamics when the model is trained on the SD corpus, but is queried using different batches of the training corpus. Importantly, the querying batch order (yaxis) matches that of training. Comparing the first two figures, we see that the CD-trained model tends to learn only the knowledge that it is just exposed to, and quickly forgets the knowledge in the past. As a result, there are distinct phases during training where each dated query category has a high accuracy; this pattern repeats at every epoch. The model essentially transitions across different specialists, each specializing to knowledge in a given time span. In contrast, the model trained using the SD corpus tends to yield generalists, which perform well on all dated queries even in the first epoch when the model has not experienced all the training knowledge yet. Figure 3 (a) shows intuitively how shuffled training yields generalists. Each year (two here, C1 and C2 ) has an optimization landscape with multiple specialists, each of which learns the knowledge in that year well and corresponds to a valley. Importantly, the specialists of different years could overlap. The overlapped region contains the “jacks-of-all-trades”: they are generalists that are well-optimized for knowledge across years. Given that specialists generally far outweigh generalists in number, a random initialization will much more likely end up in a specialist under chronological training. Why do models trained under the shuffled corpora end up in the generalist region? When trained under SD, each batch yields a specialist model of that particular batch, i.e., the “shuffled-batch specialists” in Figure 3 (a). Different shuffled batches can be thought of as samples of the same underlying corpus; as a result, different batch’s specialists are very close to each other in the landscape and all fall in the generalists region. This intuition is confirmed in Figure 2 (c). Apart from the initial stage in the first epoch, the SD-trained model performs well for all batches, even for unseen batches (in the first epoch). This suggests that shuffled-batch specialists are similar generalists. 4.2

THE ROLE OF TIME IN CHRONOLOGICAL TRAINING

A key assumption in Figure 3 (a) is that specialists and generalists, trained under CD, perform equally well on the year for which the specialist is optimized. Yet, a specialist clearly has not learned the year information, as shown by its poor performance under dated queries of other years (see Figure 2 (a)). Why, then, can a model that has not learned the year information perform as well as one that has? Intuitively, this is because under chronological training, nearby records of the person usually share the same value, because a person’s attributes (e.g., job) do not change rapidly in a short period of time; Appendix A provides detailed statistics on this. As a result, the loss can be reduced without relying strongly on time and, accordingly, the gradients would unlikely point toward regions in the landscape that relies on time. Under shuffling, however, different values of the same fact can appear close together, so the model must use the year to tell them apart. To formalize this, we represent each fact by its identity I, year Y , and value V (see Section 3 for notation convention and Figure 9 for an annotated example). For an empirical distribution Qt,W over W consecutive fact records starting at t, the value-prediction part of the training loss is: Lt,W (p) = E(I,Y,V )∼Qt,W [− log p(V | I, Y )]. 4

(1)

Preprint

Figure 3: Intuition for generalists vs. specialists and for why specialists update knowledge faster. (a) Loss landscapes of knowledge in two years (C1 and C2 ) and of all knowledge (S) over the same model parameter space (bottom plane). Each valley in C1 and C2 is a chronological specialist that learns the knowledge of that year well. Every valley in S is a shuffled-batch specialist for a batch during shuffled training. The specialists of C1 (orange) and C2 (green) overlap in a region, which is where generalists lie. Different shuffled-batch specialists (blue) lie close to each other and all fall inside the generalist region. (b) Top: the loss of the new value (− log Pnew , solid) and the old value (− log Pold , dashed) across the parameter space θ. An update starts at θ0,C and θ0,S for a specialist and a generalist, respectively; D0,C and D0,S are the initial learning gap, respectively, and G0,C and G0,S are the corresponding initial gradients. θN,C and θN,S are the points where the two losses cross, i.e., where each update completes, in the two cases, respectively. Bottom: the actual change of loss over update steps, comparing a 110M specialist and a 110M generalist; an update ends where they cross (Pnew = Pold ), and the specialist crosses first (NC < NS ). We quantify the predictive benefit of the year by comparing a year-independent predictor r(V | I) with a year-conditioned predictor p(V | I, Y ). The difference between their minimum losses is Rt,W := inf Lt,W (r) − inf Lt,W (p) = MIQt,W (V ; Y | I), r p | {z } | {z } without year

(2)

with year

where inf(·) is the infimum operator, Rt,W is how much the loss would reduce with the additional year Y information, and MI denotes conditional mutual information, representing how much the year Y tells us about the value V once the fact identity I is known. For the detailed derivation of Equations 1–2, see Appendix B.1. Empirically, Rt,W is close to 0 under CD training (see Appendix B.1 for details). Intuitively, this means that predicting V does not depend much on knowing the year information Y when the person’s identity I is known. Therefore, year-agnostic specialists and year-aware generalist have similar query performance. Appendix B.2 further shows that the year information in a specialist model is not simply dropped; it does participate in model inference, but the information is treated as an ordinary token and does not contribute much to the prediction.

5

Preprint

30 10

10

30 100 Observed steps

300

C/S CD / SD CD1/2 / SD1/2

0.3

Generalists Specialists

N ̂ = 50

0.2

N ̂ = 100

0.1

N ̂ = 200

0.0

8

11 14 Learning gap D0

17

Q K V O gate up down

(c) Gradient distribution SD

Speed ηA‖g0 ‖1

Predicted steps

100

(b) Update speed

0.4

Specialists Generalists Pretrained Continued

Q K V O gate up down

CD

(a) Step prediction

300

L0

L2

L4

L6

L8 L10

Figure 4: Update cost under chronologically ordered and shuffled training. (a) Predicted updating steps against observed steps; Each of the 18 models (3 model sizes in Table 4 × 6 training corpora) is shown in two variants: pre-trained from scratch (stars) and continued from a pre-trained model (triangles). (b) We plot these 18 pre-trained models using their initial distance D0 and the learning speed ηA∥g0 ∥1 ; each point is a median over 160 updates, and the dashed lines mark a constant number of steps (Eq. 3). Marker sizes are proportional to the model sizes. (c) Contribution to the initial gradient ∥g0 ∥1 from each layer and each block within a layer, for a generalist (above; trained on SD) and a specialist CD (below; trained on CD); the area of a square is the contribution of that block, the histograms sum the contributions by layer and by block (see details in Appendix C.3).

5

EFFICIENCY OF UPDATING KNOWLEDGE

With generalists and specialists naturally emerging from training under different temporal organizations of knowledge, we now turn our attention to how easy it is for them to learn new knowledge. 5.1

SPECIALISTS ACQUIRE NEW KNOWLEDGE FASTER

How the knowledge is exposed to a model at the pre-training time affects how the model acquires new knowledge later. Table 2 compares the number of updates required by specialists and that by generalists across three model sizes and six training corpora. Specialists require 27%-58% fewer steps to both overwrite a fact or inject a new fact than generalists (Appendix C.1 for more results). The saving increases with the model size, suggesting that more powerful generalists are even harder to update knowledge for. These trends also hold when the training corpus omits timestamps (C vs. S) or includes them in only half of the records (CD1/2 vs. SD1/2 ). Updates are undated for C and S, and evenly split between dated and undated for CD1/2 and SD1/2 . Appendix C.2 shows that the same pattern holds across optimizers and learning rates, and when full fine-tuning replaces LoRA. 5.2

A STEP LAW FOR UPDATING EFFICIENCY

We now aim to understand what determines the update cost. To that end, we derive an empirical law by viewing learning an update as a trajectory through the optimization landscape. Recall from Section 3 that an update, be it an overwrite or an injection, stops at step N when DN = log Pold (θN ) − log Pnew (θN ) < 0. Therefore, N depends on two factors: the initial distance D0 in the pre-trained model (i.e., the knowledge gap) and how fast the distance is closed per fine-tuning step (i.e., the update speed). We omit the derivation and refer readers to Appendix C.3. In short, the relationship is given by: b= N

D , ηA∥g0 ∥1

(3)

where η is the learning rate in the optimizer (Adam), g0 = ∇θ [− log Pnew ](θ0 ) is the initial gradient before the first fine-tuning step and represents the initial learning speed; A is a single calibration constant that is applied across all training settings. A is calibrated such that A∥g0 ∥1 represents the average update speed, i.e., how much each fine-tuning step closes the initial gap D0 . 6

Preprint

Table 2: Update cost comparison between different model sizes and training strategies. We use three model sizes, and each model is trained under the six training corpora. An overwrite update changes a known person’s latest value and an inject update adds a fact about an unseen person. The “saved” column is the percentage of update steps that a chronological model saves over the corresponding shuffled model (all results have a paired Wilcoxon p < 0.05). We also show the D0 ratio and the ∥g0 ∥1 ratio between a corresponding specialist-generalist pair. Overwrite

Inject

steps

saved

D0 ratio

∥g0 ∥1 ratio

steps

saved

D0 ratio

∥g0 ∥1 ratio

C/S CD / SD CD1/2 / SD1/2

51.0 / 88.0 53.0 / 73.0 53.0 / 75.0

+42% +27% +29%

1.27× 1.03× 1.00×

2.82× 2.02× 1.61×

37.0 / 59.0 37.0 / 52.0 33.0 / 55.0

+37% +29% +40%

1.18× 1.05× 0.96×

2.68× 1.90× 2.02×

350M

C/S CD / SD CD1/2 / SD1/2

50.0 / 116 55.0 / 90.0 50.5 / 94.0

+57% +39% +46%

1.00× 0.83× 0.87×

2.71× 1.63× 1.76×

31.5 / 65.0 32.0 / 57.0 31.5 / 61.0

+52% +44% +48%

0.77× 0.79× 0.80×

2.44× 1.45× 1.72×

682M

C/S CD / SD CD1/2 / SD1/2

49.0 / 118 54.0 / 87.5 50.0 / 100

+58% +38% +50%

1.10× 0.72× 0.85×

2.40× 1.39× 1.91×

29.0 / 63.5 31.0 / 54.5 29.0 / 60.0

+54% +43% +52%

0.68× 0.59× 0.64×

2.11× 1.22× 1.60×

Scale

Pair

110M

The derivation of Eq. 3 does not depend on whether, and how often, timestamps are part of the training knowledge. Therefore, the law applies to all 18 variants in Table 2. Figure 4(a) compares the observed N and the predicted N using Eq. 3. The model accurately captures the update cost across model scales, timestamp settings, and training organizations; the overall Pearson correlation coefficient r = 0.91. Without refitting A, the same model also predicts update cost after continued pre-training changes the model’s organization (Appendix D.5), and predicts well across other learning rates and with another optimizer (Appendix C.2). Since A and η are constant in Eq. 3, differences in update cost between generalists and specialists arise from D0 and g0 . Figure 4 (b) plots each pre-trained model by its D0 (x-axis) and A∥g0 ∥1 (y-axis). The three lines correspond to three different update-step counts. We make two main observations. First, update speed contributes more than the initial knowledge gap to specialists’ update advantage. Figure 4 (b) shows that generalists have a wide range of the initial learning gap D0 than that of the specialists. Regardless of the initial learning gap, however, specialists consistently have a much higher speed of learning than that of generalists (2.5 × higher on average). As an example, even though the 110M specialist trained under CD has actually a slightly (1.03 ×) larger knowledge gap than the generalist trained under SD, the former updates more than 2 × faster and ends up acquiring the knowledge much sooner. Such dynamics are illustrated in Figure 3 (b). Second, Figure 4 (b) also shows that larger models tend to have larger initial learning gaps (model size is proportional to the marker sizes). This is because larger generalists are trained better, so they are naturally more confident in the old knowledge before the update, leaving a larger knowledge gap to close during update. This explains the greater savings observed for larger models in Table 2. Figure 4 (c) further visualizes the contribution to the initial gradient from each layer and each block within a layer between a specialist and a generalist. We see that the absolute amplitude of initial gradient increases significantly from generalists to specialists. In particular, the middle layers contribute the most to the specialists’ larger gradient, which is where the faster update comes from.

6

APPLICATIONS OF ORDERING KNOWLEDGE IN PRE-TRAINING

Our analyses so far demonstrate a trade-off between generalists and specialists — influenced by the temporal ordering of knowledge. This section shows four applications of this observation to improve language models in practice.

7

Preprint

6.1

MIXING GENERALISTS AND SPECIALISTS

Setup. Consider a deployed model whose more recent knowledge changes frequently, while its historical knowledge remains fixed. Ideally, the pre-trained model should preserve its historical knowledge accurately while efficiently acquiring new knowledge. Our idea is to train a generalist for the history and a specialist for the present. Specifically, we select a recent time window near the present and organize each person’s record within this window chronologically, while shuffling all records from the earlier history beyond the window. This forms the corpus Mix-C. As a comparison, Mix-S contains the same knowledge as Mix-C but shuffles the records randomly. Both models are trained at the 110M scale; full construction and training details are provided in Appendix D.1. Performance. We find that Mix-C is able to preserve historical knowledge while acquiring new knowledge efficiently. Figure 5 (a) show the knowledge query, both dated and undated, accuracy of Mix-C. Among all training organizations, Mix-C is the only model that achieves high query accuracy on both historical knowledge (with dated queries) and current knowledge (with undated queries). Figure 5 (b) shows the training dynamics of Mix-C. The model behaves as a generalist on the historical knowledge, while exhibiting specialist-like dynamics on recent knowledge, consistent with the observations in Section 4. At the same time, the specialist-like organization of recent knowledge also enables faster acquisition of new knowledge. Figure 5 (c) compares the overwrite update cost of Mix-C and Mix-S under single, batch, and sequential updates; we also show the percentage of knowledge forgetting on the y-axis. A single update changes one fact, a batch update changes multiple facts jointly, and sequential updates apply a series of batches. All updates are made to the most recent year; see Appendix D.1 for details. Forgetting is the percentage of facts about other people whose answers are incorrectly changed by an update. Mix-C consistently requires 16%-44% fewer update steps than Mix-S across all three settings, while having 5%-57% less forgetting on the historical records. 6.2

RESHAPING A REAL-WORLD PRE-TRAINED MODEL

Setup. Pre-training from scratch is computationally expensive. We therefore ask whether a short stage of continual pre-training can improve a model’s update efficiency while preserving its historical knowledge. We start from Gemma-2-9B-it (Team, 2024), an instruction-tuned model that was trained on real-world, general data including time-varying knowledge. For continual pre-training, we sample historical records with timestamps and keep them temporally shuffled, while sampling a long recent time window and arranging each person’s records chronologically. We mix the two parts while preserving the within-person chronological order of the records in the window, forming the corpus C-C. As comparisons, C-S contains exactly the same records but is randomly shuffled, and we also include the original pre-trained model (Base) as a baseline. Further details are provided in Appendix D.2, and Appendix D.5 presents results on our synthetic data. Performance. Figure 6 (a) shows that after continual pre-training using C-C, the model improves its undated query accuracy of current knowledge from 42% to 98%, while requiring 1.3× fewer steps than Base and 1.8× fewer steps than C-S during overwrite updates. Meanwhile, its general ability is preserved; Appendix D.2 reports the forgetting results. 6.3

IMPROVING PUBLISHED-KNOWLEDGE EDITORS

Setup. So far, we have updated knowledge using LoRA. Another common way of knowledge update is through dedicated knowledge-editing methods. We next ask whether these methods could benefit from chronological pre-training. We run ROME (Meng et al., 2022), MEMIT (Meng et al., 2023), AlphaEdit (Fang et al., 2025) and ENCORE (Gupta et al., 2025), on Mix-C and Mix-S, applying the overwrite updates as in Section 6.1. Appendix D.3 contains more experimental details. Performance. We find that specialists induced by temporal knowledge organization can further improve existing knowledge-editing methods over their shuffled counterparts. Figure 6 (b) compares the update cost (TFLOPs per update) and forgetting of these editors when applied to Mix-C, which contains a specialist for present knowledge (Section 6.1), and to its fully shuffled counterpart Mix-S.

8

Preprint

1.0

C S CD SD

0.5

CD1/2

2010

Mix-C Mix-S 110M 350M 682M

2015 2020

Better

0.5

2021 2022

SD1/2

2023 0.0

2024 0.0

0.5

Dated accuracy

0

1.0

1

2

3

epochs

4

100

Forgetting (%)

Undated accuracy

(b) Training dynamics 2005

P(target value)

(a) Query accuracy

1.0

Better

50

0

0.0

5

(c) Update cost vs. forgetting

Mix-S Mix-C one batch sequential

30

100

300

1000

Steps per update

Figure 5: Query accuracy and update cost of the Mix corpora (Section 6.1). (a) Dated query accuracy (2005–2020 average) against undated query accuracy of different pre-trained models. (b) Dated query accuracy of Mix-C through five epochs of training, for historical years (2005–2020) whose knowledge is shuffled and for recent records (2021–2024) whose knowledge is ordered. (c) Median steps per update (log scale) against forgetting (percentage of other people’s facts that are incorrectly changed), for single updates (star), batch updates (cross) and sequential updates (triangle).

80 60 40

Better

0

50

100

Undated accuracy (%)

(b) Update cost vs. forgetting

5

Forgetting (%)

Update steps

Base C-S C-C

3 2 1 1

Mix-S / C-S Mix-C / C-C Pretrained Continued

120

4

0 0.5

(c) Accuracy vs. update cost

Better

Mix-S ROME MEMIT

Mix-C AlphaEdit ENCORE

2

5

10

Update steps

(a) Accuracy vs. update cost 100

Update cost (TFLOPs per fact)

100 80 Better

60 40 0

50

100

Undated accuracy (%)

Figure 6: (a) Undated query accuracy of the continual pre-training stage vs. the median update steps on overwrite updates, for Gemma-2-9B-it (Base) and its two continually pre-trained variants. (b) Update cost vs. percentage of forgetting for the four knowledge editors applied to the 110M Mix-C and Mix-S models; each model’s four editors are joined by a dashed line. (c) Undated query accuracy against the median update steps in the versioned-software dataset; results are for a 110M model. For the same editing method, the specialist organization in Mix-C provides an additional reduction in both update cost and forgetting over Mix-S. The gain reaches up to 46% lower update cost with MEMIT (Meng et al., 2023) and 63% less forgetting with AlphaEdit (Fang et al., 2025). 6.4

GENERALIZING TO ANOTHER KNOWLEDGE ORDERING: VERSIONED SOFTWARE CHANGE

Setup. So far, biographical knowledge is ordered chronologically, but do the same observations hold under other domains and orderings? We study a practical case: software tools, where the version of a tool serves as the ordering key. For example, when an API changes its URL across versions, a model should quickly learn the API’s latest endpoint without losing its knowledge of how to call other tools. We build the tool-call dataset in a similar way to the biography dataset in Section 3. The dataset contains multiple tools, each described by six fields (for example, host, authentication header, etc.), each of which has a value. An example is (see Appendix D.4 for more examples): For v8, the Flint Payments API carries host api.flintpay.com. Its input parameter is refund_id.

We follow the pre-training and continual pre-training setups in Section 6.1. Specifically, for pretraining from scratch, we shuffle the older versions and present the recent versions in version order, as in Mix-C. For continual pre-training, we start from a shuffled base model, present records within recent version window in version order, and replay its older versions in shuffled order, as in C-C. We ask whether the same setup can learn a specialist for recent versions while keeping a generalist over older versions. Appendix D.4 shows more details about the training setup.

9

Preprint

Performance. The same pattern appears in the tool-use domain. Figure 6 (c) shows that both MixC and C-C answer 98–99% of queries about the latest version, while cutting update cost by up to 48% and 33% relative to Mix-S and C-S, respectively. Both also keep their knowledge of older versions.

7

CONCLUDING REMARKS

Pre-training knowledge ordering shapes the cost of future knowledge updates. Models trained on temporally ordered knowledge, which we call specialists, require roughly half as many update steps as generalists, models trained on shuffled knowledge. We derive a step law to explain this advantage. We use this insight to design new knowledge organization to preserve historical knowledge while enabling efficient updates. The benefit of pre-training knowledge organization extends to alreadytrained models and existing post-training update tools. Apart from time, Section 6.4 explores another ordering, where knowledge about software documentation is ordered by software version. Regardless of the specific order Y , what matters is the additional information that Y provides to a knowledge query given all other available information (see the mutual information Eq. 2). An interesting future investigation is to quantify how this mutual information influences the emergence and behaviors of generalists and specialists. While our work focuses on language models, evolutionary biologists have long studied how generalists and specialists can evolve in time-varying environments (Kassen, 2002). Many also have looked at various advantage of specialists such as faster decision making (Bernays & Funk, 1999). Another interesting path is to investigate whether the learning benefit of specialists extend to evolution, which obtains knowledge through gradient-free methods.

REFERENCES Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction. arXiv preprint arXiv:2309.14316, 2023. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41–48, 2009. Elizabeth A Bernays and Daniel J Funk. Specialists make faster decisions than generalists: experiments with aphids. Proceedings of the Royal Society B: Biological Sciences, 266(1415):151, 1999. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7432–7439, 2020. Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, et al. Symbolic discovery of optimization algorithms. Advances in neural information processing systems, 36:49205–49233, 2023. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In Proceedings of EMNLP, 2021. Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. Time-aware language models as temporal knowledge bases. Transactions of the Association for Computational Linguistics, 10:257–273, 2022. Shibhansh Dohare, J. Fernando Hernandez-Garcia, Qingfeng Lan, Parash Rahman, A. Rupam Mahmood, and Richard S. Sutton. Loss of plasticity in deep continual learning. Nature, 632:768–774, 2024. Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Jie Shi, Xiang Wang, Xiangnan He, and Tat-Seng Chua. AlphaEdit: Null-space constrained knowledge editing for language models. In International Conference on Learning Representations, 2025. 10

Preprint

Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. Does fine-tuning LLMs on new knowledge encourage hallucinations? In Proceedings of EMNLP, pp. 7765–7784, 2024. Akshat Gupta, Phudish Prateepamornkul, Maochuan Lu, Ahmed Alaa, Thomas Hartvigsen, and Gopala Anumanchipalli. Lifelong knowledge editing requires better regularization. In Findings of the Association for Computational Linguistics: EMNLP 2025, 2025. Wes Gurnee and Max Tegmark. Language models represent space and time. In International Conference on Learning Representations, 2024. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th annual meeting of the association for computational linguistics, pp. 8342– 8360, 2020. Thomas Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. Aging with GRACE: Lifelong model editing with discrete key-value adaptors. In Advances in Neural Information Processing Systems, 2023. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. Adam Ibrahim, Benjamin Thérien, Kshitij Gupta, Mats L. Richter, Quentin Anthony, Timothée Lesort, Eugene Belilovsky, and Irina Rish. Simple and scalable strategies to continually pre-train large language models. Transactions on Machine Learning Research, 2024. Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, and Minjoon Seo. TemporalWiki: A lifelong benchmark for training and evaluating everevolving language models. In Proceedings of EMNLP, 2022a. Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stanley Jungkyu Choi, and Minjoon Seo. Towards continual knowledge learning of language models. In International Conference on Learning Representations, 2022b. Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wen Li, Xiaokai Wei, Andrew Arnold, and Xiang Ren. Lifelong pretraining: Continually adapting language models to emerging corpora. In Proceedings of NAACL, pp. 4764–4780, 2022. Janet G Johnson, Macarena Peralta, Mansanjam Kaur, Ruijie Sophia Huang, Sheng Zhao, Ruijia Guan, Shwetha Rajaram, and Michael Nebeling. Exploring collaborative genai agents in synchronous group settings: eliciting team perceptions and design considerations for the future of work. Proceedings of the ACM on Human-Computer Interaction, 9(7):1–33, 2025. Rees Kassen. The experimental evolution of specialists, generalists, and the maintenance of diversity. Journal of evolutionary biology, 15(2):173–190, 2002. Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. Continual pretraining of language models. In International Conference on Learning Representations, 2023. Jiyeon Kim, Hyunji Lee, Hyowon Cho, Joel Jang, Hyeonbin Hwang, Seungpil Won, Youbin Ahn, Dohaeng Lee, and Minjoon Seo. Knowledge entropy decay during language model pretraining hinders new knowledge acquisition. In International Conference on Learning Representations, 2025. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526, 2017. 11

Preprint

Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d’Autume, Tomas Kocisky, Sebastian Ruder, Dani Yogatama, Kris Cao, Susannah Young, and Phil Blunsom. Mind the gap: Assessing temporal generalization in neural language models. In Advances in Neural Information Processing Systems, 2021. Jeffrey Li, Mohammadreza Armandpour, Seyed Iman Mirzadeh, Sachin Mehta, Vaishaal Shankar, Raviteja Vemulapalli, Samy Bengio, Oncel Tuzel, Mehrdad Farajtabar, Hadi Pouransari, et al. Ticlm: A web-scale benchmark for time-continual llm pretraining. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 32231– 32273, 2025. Houjun Liu, John Bauer, and Christopher D. Manning. Drop dropout on single-epoch language model pretraining. In Findings of the Association for Computational Linguistics: ACL 2025, 2025. Daniel Loureiro, Francesco Barbieri, Leonardo Neves, Luis Espinosa Anke, and Jose CamachoCollados. TimeLMs: Diachronic language models from Twitter. In Proceedings of ACL, System Demonstrations, 2022. Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint arXiv:2308.08747, 2023. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems, 2022. Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In International Conference on Learning Representations, 2023. Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. In International Conference on Learning Representations, 2022a. Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning, and Chelsea Finn. Memorybased model editing at scale. In International Conference on Machine Learning, pp. 15817–15831, 2022b. Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha. Fine-tuning or retrieval? comparing knowledge injection in LLMs. In Proceedings of EMNLP, pp. 237–250, 2024. Yein Park, Chanwoong Yoon, Jungwoo Park, Minbyul Jeong, and Jaewoo Kang. Does time have its place? temporal heads: Where language models recall time-specific information. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 16616–16643, 2025. Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37:30811–30849, 2024. Hippolyte Pilchen, Romain Fabre, Franck Signe Talla, Patrick Perez, and Edouard Grave. Understanding data temporality impact on large language models pre-training. arXiv preprint arXiv:2605.22769, 2026. Vedant Sachdeva, Kabir Husain, Jiming Sheng, Shenshen Wang, and Arvind Murugan. Tuning environmental timescales to evolve and maintain generalists. Proceedings of the National Academy of Sciences, 117(23):12693–12699, 2020. Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. Fine-tuned language models are continual learners. In Proceedings of EMNLP, pp. 6107–6122, 2022. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. Gemma Team. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024. 12

Preprint

Kushal Tirumala, Aram H. Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models. In Advances in Neural Information Processing Systems, 2022. Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations, 2019. Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, and Thang Luong. FreshLLMs: Refreshing large language models with search engine augmentation. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 13697–13720, 2024. Peng Wang, Zexi Li, Ningyu Zhang, Ziwen Xu, Yunzhi Yao, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. WISE: Rethinking the knowledge memory for lifelong model editing of large language models. In Advances in Neural Information Processing Systems, 2024. Xiaoxia Wu, Ethan Dyer, and Behnam Neyshabur. When do curricula work? Conference on Learning Representations, 2021.

In International

Zitong Yang, Neil Band, Shuangping Li, Emmanuel Candès, and Tatsunori Hashimoto. Synthetic continued pretraining. In International Conference on Learning Representations, 2025. Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing large language models: Problems, methods, and opportunities. In Proceedings of EMNLP, 2023. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th annual meeting of the association for computational linguistics, pp. 4791–4800, 2019. Bowen Zhao, Zander Brumbaugh, Yizhong Wang, Hannaneh Hajishirzi, and Noah A. Smith. Set the clock: Temporal alignment of pretrained language models. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 15015–15040, 2024. Adam Zweiger, Jyothish Pari, Han Guo, Ekin Akyürek, Yoon Kim, and Pulkit Agrawal. Self-adapting language models. arXiv preprint arXiv:2506.10943, 2025.

13

Preprint

Table 3: Pre-training datasets. C/S: chronological/shuffled; D: timestamped; 1/2 : half timestamped. Timestamps 0% 50% 100%

Table 4: Model scales, all with a context of 512 tokens. Records: per epoch; LR: peak learning rate. Other settings in Appendix A.

Chronological

Shuffled

Scale

C CD1/2 CD

S SD1/2 SD

110M 350M 682M

Layers Heads Hidden Records Tokens LR (10−4 ) 12 24 32

12 16 20

768 1024 1280

24M 10.88B 68M 30.80B 134M 60.68B

6 3 2.5

APPENDIX A

DATA, TRAINING AND QUERY PROTOCOL

This appendix gives the detailed settings behind Section 3 and Section 4: how the biographies are generated, how the models are trained, how a query is scored, and how an update is run and measured. Biographies. Names and attribute values come from the vocabulary lists of Allen-Zhu & Li (2023). Each person appears 1,000 times between 2000 and 2024, and each appearance is one record that states the six attributes. Birthday and birth city never change. University and field change 3 times over a lifetime (median; mean 2.8), employer and employer city 10 times (mean 9.9), at random points of the 1,000 appearances.

Pre-training. The three scales share the GPT-2 architecture with rotary position embeddings (Su et al., 2024) and the GPT-2 tokenizer (Table 4). Every model trains on batches of 2,560 records, with a 5% linear warmup, a cosine decay from the peak learning rate of Table 4 to a tenth of it, weight decay 0.1 and gradient clipping at 1.0. For the main results in Section 4, we train the models for five epochs, by which point query accuracy has converged already.

Chronological training stream 2005

Apple

2010

Google

2015

PepsiCo

2020

Amazon

2024

Vimeo

2005

Apple

2010

Google

2015

PepsiCo

2017

PepsiCo

Epoch 1

⋮ ⋮ ⋮ ⋮

⋮ ⋮

Epoch 2

In a chronological stream, consecutive records of the same person usually contain the same value. In CD, the values are unchanged in about 99.0% of consecutive records. After shuffling, consecutive records can come from very different years, so these numbers drop to around 20.0%. So in a chronological stream the year says almost nothing about a person’s current value; Section 4.2 relies on this, and Appendix B.1 turns it into a bound. 1,000 individuals in the corpus are held out from training and used only for injection updates; A timestamped record starts with “In {year},”; the half-timestamped corpora drop this prefix from a random half of the records, the same half in CD1/2 and SD1/2 , so that the two differ only in order (Table 3).

⋮ ⋮ ⋮

dated current step k

2020

Amazon

2024

Vimeo

⋮

epoch boundary

latest

Figure 7: Targets of the three kinds of query in Figure 8: a value in the queried year, the value in the batch trained at step k, and the final value.

Queries. A query prompts the model with a person’s name and an attribute, cut just before the value, and reads the probability of every value in the attribute’s vocabulary. A dated query starts with “In {year},” and is correct if the answer is any value the person held during that year. In contrast, an undated query gives no year, and is correct if the answer is the latest value (See Figure 7 about “latest”). The year rows (2005–2020) queries 200 people per time-varying attribute about a fixed year, every 125 steps. The queries keep unchanged for different settings. The current row asks, at every checkpoint (every 625 steps), about 200 records of the batch the model has just trained on, in the form each record was trained in. The query-by-batch rows of Figure 2c ask in the same way about 200 records from each of ten batches spaced evenly over the SD stream. Figure 8 extends Figure 2 to all six corpora.

14

Preprint

Query by year C

2005 2010 2015 2020 current

0.5 0

0.5 0

2005 2010 2015 2020 current

current value latest value

P(value)

1

S

0

1

2

3

epochs

4

5

Query by batch

CD1/2

CD

Half timestamped

Shuffled training 0

SD

1.0 0.8 0.6

1 SD1/2

SD

0

0

SD1/2

0.4

P(target value)

1

Timestamped

P(value)

Shuffled training Chronological training

No timestamp

0.2 1 1

2

3

4

epochs

1

2

3

4

5

0

1

2

3

epochs

4

5

0.0

Figure 8: Query accuracy of the six 110M models over five epochs of training, the full version of Figure 2. Rows of panels compare chronological and shuffled training, columns compare timestamp settings, and the right column queries SD and SD1/2 by batch as in Figure 2c. Rows 2005–2020 ask with a year (“In 2005, …”) and count any value the person held in that year; the current row asks about records of the batch the model has just trained on, in the form each was trained in. In the two panels on the left, the solid curve is this current score for the undated models and the dashed curve the probability of the final value, held in 2024. Updates. An update trains a rank-16 LoRA adapter (Hu et al., 2022), with α = 16, on every attention and MLP projection of every layer (84 adapters at 110M), using Adam (Kingma & Ba, 2015) without weight decay at a fixed learning rate to the pre-training peak (1/60, e.g. 10−5 at 110M), no weight decay. The training sentence states the new value in one of the attribute’s sentence templates; for timestamped models it starts with “In 2024,”, and for half-timestamped models a random half of the updates do. An overwrite update targets a person the model was trained on and an injection update targets the held-out person during training. Either succeeds when the new value’s probability passes that of its competitor (i.e. the strongest value the model held before updating). The learning gap D is the log-probability gap to that competitor before the update. Each model receives 480 updates, 40 overwrite updates and 40 injection updates per attribute. For each attribute we sample the individuals so that different people are equally represented. Table 2 reports medians over all overwrite updates or all injection updates of a model. Forgetting is measured on facts of other people, as the percentage whose answer the update changes. Updating criteria. Our results do not depend on the stopping criterion used for an update. On 120 updates from each 110M pair in Table 2, stopping when the new value passes the model’s strongest previous value gives the same conclusion as requiring it to become the top-1 value over the full attribute vocabulary (Table 5). The same conclusion also holds with two stricter thresholds: a 3-nat log-probability margin over the competitor or Pnew ≥ 0.5. Table 5: Update cost of the 110M pairs under three stricter success criteria, which are the new value is the top-1 over the attribute’s whole vocabulary, its log-probability exceeds the competitor’s by 3 and its probability Pnew reaches 0.5. Cells represent median steps, chronological / shuffled, and the share of steps the chronological model saves (bold: paired Wilcoxon p < 0.05). 60 overwrites and 60 injections updates per pair. Overwrite

Scale

Pair

Inject

Vocabulary top-1

Margin ≥ 3

Pnew ≥ 0.5

Vocabulary top-1

Margin ≥ 3

Pnew ≥ 0.5

steps

steps

steps

steps

steps

steps

saved

saved

saved

saved

saved

saved

C/S 53.0 / 89.0 +40% 63.5 / 108 +41% 59.5 / 99.5 +40% 38.0 / 57.0 +33% 46.5 / 78.5 +41% 47.0 / 71.5 +34% 110M CD / SD 51.5 / 75.0 +31% 63.0 / 88.5 +29% 57.5 / 80.5 +29% 40.0 / 52.0 +23% 53.0 / 66.5 +20% 50.0 / 62.5 +20% 1/2 1/2 CD / SD 54.5 / 88.0 +38% 67.0 / 103 +35% 61.0 / 96.5 +37% 31.5 / 61.5 +49% 40.5 / 84.5 +52% 39.5 / 80.0 +51%

15

Preprint

B

GENERALISTS AND SPECIALISTS: DERIVATIONS AND MEASUREMENTS

B.1

THE PREDICTIVE VALUE OF TIMESTAMPS

This appendix derives Eq. 1 and Eq. 2 in Section 4.2. It shows how much the year helps prediction within a training window. Chronological and shuffled training have the same global objective, but the year is much less useful in a chronological window than in a shuffled one. In 2010, John Ivy Bi was born on December 17, 1979. year y

person

attribute: birthday

value v

fact identity i = (John Ivy Bi, birthday)

Figure 9: Mapping the example in Section 3 to the fact-level variables. The person and attribute jointly identify the fact i; y = 2010 is the record’s timestamp, and v = December 17, 1979 is its value. Uppercase I, Y, V denote the variables and lowercase i, y, v their observed values. Here the attribute is time-invariant; for a changing attribute such as employer, the same i can have different values v in different years y. Step 1: the global objective is the same for both orders. Pretraining minimizes the negative log-likelihood of whole records, whereas a query tests the value of one fact. For a tokenized record x = (x1 , . . . , xT ) whose value tokens sit at positions V(x), the record loss splits exactly into a value part and a remainder, ℓpre (θ; x) = −

T X

log pθ (xj | x<j ) = −

j=1

|

X

log pθ (xj | x<j ) + −

j∈V(x)

{z ℓvalue (θ;x)

}

|

X j ∈V(x) /

log pθ (xj | x<j ), (4) {z

}

ℓother (θ;x)

and for a value span v after context c the value part equals − log pθ (v | c) by the chain rule. We study the value part, since it trains the associations a query measures, and represent its context by the fact identity I and the year Y (Figure 9); if further context C is kept, every quantity below is also conditioned on C and nothing else changes. Let D = {(im , ym , vm )}N m=1 be the fact-level records. Their empirical distribution and value-prediction loss are 1 X 1[(im , ym , vm ) = (i, y, v)], N m=1 N

P (i, y, v) =

L̄(θ) = EP [− log pθ (V | I, Y )].

(5)

Permuting the records changes neither P nor L̄(θ) at any θ, so chronological and shuffled training don’t differ in their global objective. Step 2: the local objective depends on the order. For an ordering Z1 , . . . , ZN with Zn = (In , Yn , Vn ), the empirical distribution of the W consecutive records from position t is Qt,W (i, y, v) =

1 W

t+W X−1

1[Zn = (i, y, v)],

1 ≤ t ≤ N − W + 1.

(6)

n=t

The horizon W is an analysis scale, not necessarily a batch size. Two orderings share Q1,N = P but can have different Qt,W for W < N . We fix a window and write Q = Qt,W , with every conditional quantity evaluated on its support. Step 3:P the best predictor with and without the year. A year-independent predictor r(V | I) has loss − i,v Q(i, v) log r(v | i). Writing − log r(v | i) = − log Q(v | i) + log[Q(v | i)/r(v | i)]

16

Preprint

and using Q(i, v) = Q(i)Q(v | i), X X Lblind Q(i, y, v) log r(v | i) = − Q(i, v) log r(v | i) Q (r) = − i,y,v

=−

X

Q(i)

X

i,v

Q(v | i) log Q(v | i)

v

i

+

X

Q(i)

X

Q(v | i) log

v

i

Q(v | i) r(v | i)

(7) = HQ (V | I) + EI∼Q DKL (Q(V | I) ∥ r(V | I)) . P Here H is the entropy in bits, HQ (V | I) = i Q(i)H(Q(V | i)) is the uncertainty in the value once the fact is known, and the second term is a Kullback–Leibler (KL) divergence, nonnegative and zero at r∗ (v | i) = Q(v | i), the window’s mixture of the fact’s values across years. The same steps for a year-conditioned predictor p(V | I, Y ), with Q(i, y, v) = Q(i, y)Q(v | i, y), give X X Ldated Q(i, y) Q(v | i, y) log p(v | i, y) Q (p) = − v

i,y

=−

X

Q(i, y)

Q(v | i, y) log Q(v | i, y)

v

i,y

+

X

X

Q(i, y)

X

Q(v | i, y) log

v

i,y

Q(v | i, y) p(v | i, y)

= HQ (V | I, Y ) + EI,Y ∼Q DKL (Q(V | I, Y ) ∥ p(V | I, Y )) , (8) P where HQ (V | I, Y ) = i,y Q(i, y)H(Q(V | i, y)) is the uncertainty once fact and year are known, and the divergence is zero at p∗ (v | i, y) = Q(v | i, y). Over unrestricted predictors, therefore, inf Lblind Q (r) = HQ (V | I),

inf Ldated Q (p) = HQ (V | I, Y ).

r

p

(9)

These are the minimum possible losses for the two predictors within the window. The dated loss is zero only when the fact and year uniquely determine the value. Step 4: the gap is a conditional mutual information. Subtracting the two optima, dated Rt,W := inf Lblind Q (r) − inf LQ (p) r

p

= HQ (V | I) − HQ (V | I, Y ) X Q(v | i, y) = Q(i, y, v) log = MIQ (V ; Y | I), Q(v | i) i,y,v

(10)

where the last step uses Q(v, y | i) = Q(y | i)Q(v | i, y). The identity holds for every ordering and every window: Rt,W is the minimum loss a predictor pays for ignoring the year, and it is positive exactly when the value distribution of a fact depends on the year. Step 5: an upper bound on the gap. In a chronological window a fact’s value is almost constant even though its year changes. For each fact i with Q(i) > 0, let vi⋆ ∈ arg maxv Q(v | i) be a most probable value, εi = 1 − Q(vi⋆ | i) the probability of any other value, and Ki = | supp Q(V | i)| the e i (v) = Q(v | i)/εi for v ̸= v ⋆ . number of values. For εi > 0, normalize the remaining mass as Q i Separating the modal value from this tail gives   X e i (v) log εi Q e i (v) εi Q HQ (V | I = i) = −(1 − εi ) log(1 − εi ) − v̸=vi⋆

e i ) ≤ h2 (εi ) + εi log(Ki − 1), = h2 (εi ) + εi H(Q

(11)

where h2 (ε) = −(1 − ε) log(1 − ε) − ε log ε is the binary entropy and the inequality uses the tail’s support of at most Ki − 1 values; a fact with Ki = 1 has εi = 0 and contributes zero. Since HQ (V | I, Y ) ≥ 0, averaging over facts gives X 0 ≤ Rt,W ≤ HQ (V | I) ≤ Bt,W := Q(i) [h2 (εi ) + εi log(Ki − 1)] . (12) i:Ki ≥2

17

Preprint

(a) SD

L0

L1

L2

L3

L4

L5

L6

L7

L8

L9

L10

L11

L9

L10

L11

year person answer embedding

(b) CD

L0

L1

L2

L3

residual state L4

attention output L5

L6

MLP output L7

L8

year person answer embedding

residual state

attention output

MLP output

Figure 10: Activation patching of the timestamp in the 110M SD (a) and CD (b). Rows are three positions of a dated question (the year, the person’s name and the answer position) and columns the twelve layers; each symbol is one activation: embedding, residual state, attention output or MLP output. A darker symbol recovers more of the model’s own year effect when that activation is copied from the same question asked about 2024; colors compare locations within one model. SD recovers the year at the answer position in the first attention block, whereas in CD the year effect builds up gradually through depth. The bound holds for either order. Under chronological training, values are highly persistent within a window, so εi is close to zero and the bound becomes small. Empirically, for the 110M CD model, Rt,W ≤ Bt,W = 0.01 bits per record, while the corresponding shuffled window has Rt,W = 0.69 bits, at least 69× larger. Thus, the year provides almost no predictive value in a chronological window but much more in a shuffled one. B.2

TRACING TIMESTAMP USE WITH ACTIVATION PATCHING

Section 4.2 states that a specialist reads the year but does not rely on it. To test how each model uses the year during inference, we use 115 dated questions whose historical answer differs from the 2024 answer. For each question, we run the model twice: once with the historical year and once with 2024. We then patch one activation from the 2024 run into the historical run and ask whether this makes the model answer more like it was queried about 2024. We patch activations at the year token, the person’s name, or the answer position, across different layers and modules. The patching score measures how much of the original shift from the historical answer to the 2024 answer is recovered by a single patch. It therefore traces how year information flows through the model during inference. Figure 10 reports the median over the questions on which the year moves the answer by more than 1.4 bits. Both models process the year during inference, but they use it very differently. SD uses the year strongly and early, while CD picks it up only gradually and gives it much less weight. If CD relied on the year to predict the value, it would behave like a generalist and answer dated queries across years accurately. Instead, chronological training places the model in a parameter region where the year is processed but is not needed much for value prediction.

C

UPDATE COST: FULL RESULTS AND THE STEP LAW

C.1

UPDATE COST BY RATE OF CHANGE AND ACROSS SEEDS

Table 6 extends Table 2 by splitting time-varying facts into slow- and fast-changing groups and by adding the Mix-C/Mix-S pair from Section 6.1. Table 7 repeats the 110M C/S comparison with three additional random initializations. Across the four runs, chronological training saves 44–52% of the update steps, showing that the result is robust to initialization.

18

Preprint

Table 6: Update cost by how often a fact changes: the time-varying facts of Table 2 split into slow and fast facts, with the Mix pair (Section 6.1) added. Each “steps” cell gives the median steps of the chronological and shuffled models. The “saved” column is the percentage of update steps that a chronological model saves over the corresponding shuffled model; bold values have a paired Wilcoxon p < 0.05.

Inject

Overwrite

Slow

Fast

Scale

Pair

steps

saved

steps

saved

110M

C/S CD / SD CD1/2 / SD1/2 Mix-C / Mix-S

49.0 / 91.5 52.5 / 69.0 51.0 / 74.5 46.5 / 70.5

+46% +24% +32% +34%

39.0 / 86.0 38.0 / 71.0 36.0 / 72.5 45.0 / 64.5

+55% +46% +50% +30%

350M

C/S CD / SD CD1/2 / SD1/2

47.0 / 118 54.0 / 81.0 49.5 / 94.5

+60% +33% +48%

36.0 / 118 41.0 / 86.5 35.0 / 92.5

+70% +53% +62%

682M

C/S CD / SD CD1/2 / SD1/2

45.5 / 114 50.0 / 80.0 48.0 / 98.0

+60% +38% +51%

36.5 / 121 44.0 / 84.0 37.5 / 103

+70% +48% +64%

110M

C/S CD / SD CD1/2 / SD1/2 Mix-C / Mix-S

37.0 / 62.5 38.0 / 49.0 34.0 / 55.0 36.0 / 50.0

+41% +22% +38% +28%

24.5 / 54.0 23.0 / 46.5 21.0 / 51.0 33.0 / 44.0

+55% +51% +59% +25%

350M

C/S CD / SD CD1/2 / SD1/2

32.0 / 67.5 31.5 / 50.0 29.0 / 57.0

+53% +37% +49%

19.0 / 59.0 20.0 / 53.5 17.0 / 55.5

+68% +63% +69%

682M

C/S CD / SD CD1/2 / SD1/2

27.0 / 71.0 29.5 / 52.0 28.0 / 62.0

+62% +43% +55%

18.0 / 60.0 21.0 / 50.5 18.0 / 55.0

+70% +58% +67%

Table 7: Temporal order saves about half of the steps on time-varying facts in each of four independently trained 110M C / S pairs, each with its own population and initialization. Cells read as in Table 2: median steps on time-varying facts, chronological / shuffled, and the share of steps temporal order saves. Overwrite Seed 0 (main) 1 2 3

C.2

Inject

steps

saved

steps

saved

44 / 89 46 / 89.5 48 / 89 44 / 89

+51% +49% +46% +51%

29 / 57.5 29.5 / 61 32 / 57 28 / 55.5

+50% +52% +44% +50%

OTHER OPTIMIZERS, LEARNING RATES AND FULL FINE-TUNING

The update advantage of specialists on time-varying facts is robust to how the update is performed. Table 8 shows the same pattern across optimizers, learning rates, LoRA ranks, and full fine-tuning: chronological models consistently require fewer update steps than shuffled models. The saving is about 50% under Adam (Kingma & Ba, 2015) and Lion (Chen et al., 2023) , and is even larger under SGD. The step law in Section 5.2 continues to predict Adam, Lion, etc. well. (Appendix C.3.1). C.3

DERIVATION OF THE STEP LAW

This appendix derives Eq. 3 step by step (Appendix C.3.1), checks each approximation (Appendix C.3.2), and derives the adapter-gradient factorization used in Figure 4c (Appendix C.3.3). C.3.1

FROM THE TRAJECTORY TO THE LAW

Setting. For each update, we fix the prompt, the new value, and the competitor, defined as the strongest value the model predicts for that fact before the update (Section 3). The adapter parameters

19

Preprint

Table 8: The saving of temporal order does not depend on how the update is made. Each block changes one part of the update and keeps the rest as in the main experiments (LoRA of rank 16, Adam, learning rate 10−5 ). Cells read as in Table 2: median steps, chronological / shuffled, and the share of steps temporal order saves on time-varying facts. Overwrite steps

saved

Inject

Varied

Setting

steps

Optimizer

Adam Lion SGD SGD + momentum

47.0 / 92.5 51.0 / 108 23.5 / 214 7.5 / 33.5

+49% 32.0 / 54.0 +53% 32.5 / 58.5 +89% 18.0 / 118 +78% 6.0 / 21.5

+41% +44% +85% +72%

saved

Learning rate

Adam, 3×10−5 Adam, 3×10−6 Adam, 10−6 Lion, 3×10−6 Lion, 10−6 SGD, 10−2 SGD, 3×10−3 SGD, 3×10−4

18.0 / 32.5 148 / 266 418 / 781 168 / 357 497 / 1061 3.0 / 22.0 8.0 / 71.5 76.5 / 708

+45% 10.0 / 20.0 +45% 87.0 / 169 +46% 248 / 492 +53% 106 / 194 +53% 318 / 580 +86% 2.0 / 12.5 +89% 6.0 / 40.0 +89% 57.5 / 394

+50% +49% +50% +45% +45% +84% +85% +85%

LoRA rank

r = 16 r=4 r=1

49.5 / 87.0 113 / 184 234 / 368

+43% 28.0 / 54.5 +38% 72.0 / 125 +36% 152 / 258

+49% +42% +41%

Full fine-tuning

Adam, 3×10−6 Adam, 10−6

4.0 / 9.0 10.0 / 24.0

+56% +58%

2.0 / 4.0 5.0 / 10.5

+50% +52%

θ start from θ0 , with a fresh optimizer state and no weight decay. We define L(θ) = − log Pnew (θ), C(θ) = log Pnew (θ) − log Pold (θ), D = −C(θ0 ) > 0,

g0 = ∇L(θ0 ), (13) where L is the update loss, C is the log-probability margin of the new value over the competitor, D is the initial gap that the update must close, and g0 is the gradient before the first update step. The update succeeds once the new value becomes more likely than the competitor: N = min{k : C(θk ) > 0}. We write ∆θk := θk+1 − θk for the parameter change made by Adam at step k. All probabilities are computed over the full target value. In practice, we minimize the mean loss over its tokens. The margin travelled is a path integral. Let θ(t), t ∈ [0, T ], denote the parameter path, with θ(k) = θk at each integer step, and let θ̇(t) be its velocity. By the chain rule, the total change in the margin from the start to step T is Z T Z T
⊤ d C(θ(t)) C(θT ) − C(θ0 ) = dt = ∇θ C θ(t) θ̇(t) dt. (14) dt 0 0 The integrand has two parts. The first is the margin gradient, ∇θ C = ∇θ log Pnew − ∇θ log Pold , which we can compute at any parameter value, including θ0 before the update. The second is the parameter velocity θ̇(t). Unlike the gradient, this velocity has no closed form because Adam generates each iterate from the history of previous gradients through its running moments. The five steps below simplify this expression until the update cost can be written using quantities available before the update. In the following, we write ∇ for ∇θ . Step 1: break the path into optimizer steps. The optimizer gives us the discrete iterates θ0 , θ1 , . . ., so we first split the path at these points. We connect each pair of consecutive iterates with a straight segment: θ(t) = θk + (t − k) ∆θk , θ̇(t) = ∆θk , k ≤ t ≤ k + 1. (15) With the substitution s = t − k, every segment maps onto the unit interval: T −1 Z 1 X C(θT ) − C(θ0 ) = ∇C(θk + s ∆θk )⊤ ∆θk ds. (16) k=0

0

Because ∇C is a gradient field, the integral between two iterates depends only on their endpoints. Thus, the k-th term is exactly C(θk+1 ) − C(θk ), regardless of the path used to connect them. 20

Preprint

Step 2: linearize the margin within each step. We next simplify what happens between two consecutive iterates. A segment is short, η per coordinate under Step 4, so we approximate the margin gradient along the segment by its value at the start, ∇C(θk ). This is the first-order Taylor approximation of C around θk . The integral then becomes X ∆Ck := C(θk+1 ) − C(θk ) ≈ vk := ∇C(θk )⊤ ∆θk , so C(θT ) ≈ −D + vk . (17) k<T

We call vk the update speed at step k: it is how much that step closes the margin. The P update starts with a gap D and succeeds once the accumulated gain reaches this gap, i.e., when k<N vk ≥ D. This approximation drops only the curvature of C within each optimizer step. Step 3: use the first-step speed to estimate the average speed. The expression above still requires every vk along the update trajectory. To predict the update cost before running the update, we instead use v0 , which is available at the start. The speed does not remain exactly equal to v0 (for example, momentum and the gradient change as Pnew increases). We therefore assume that the average speed up to success is a fixed multiple A of the first-step speed, 1 X b= D , vk ≈ A v 0 =⇒ N v0 > 0. (18) N A v0 k<N

This introduces a fitted parameter A. Appendix C.3.2 measures how well Av0 approximates the average speed. The remaining two steps express v0 using the initial loss gradient g0 . Step 4: Adam’s first update follows the gradient sign. We now write the first parameter update explicitly. At iterate k, Adam (Kingma & Ba, 2015) reads the gradient gk = ∇L(θk ), updates its two running moments, and applies (19) µk = β1 µk−1 + (1 − β1 ) gk , νk = β2 νk−1 + (1 − β2 ) gk⊙2 , µ̂k µk νk ∆θk = −η √ , µ̂k = , ν̂k = , (20) ν̂k + ϵ 1 − β1k+1 1 − β2k+1 with µ−1 = ν−1 = 0. At the first step, the optimizer state is still zero. Thus, µ0 = (1 − β1 )g0 and ν0 = (1 − β2 )g0⊙2 , and the bias corrections give µ̂0 = g0 and ν̂0 = g0⊙2 . The first update is therefore g0 ∆θ0 = −η ≈ −η sign(g0 ), v0 = ∇C(θ0 )⊤ ∆θ0 ≈ −η ∇C(θ0 )⊤ sign(g0 ). (21) |g0 | + ϵ Ignoring ϵ, each coordinate moves by approximately η, and the gradient determines its direction rather than its magnitude. Step 5: express the first-step speed using the loss gradient. The expression for v0 still contains the margin gradient ∇C(θ0 ) = ∇ log Pnew − ∇ log Pold . The optimizer, however, directly computes only g0 = −∇ log Pnew . We therefore rewrite ∇C in terms of g0 . Let S = Pnew + Pold be the total probability assigned to the new value and its competitor, and let qold = Pold /S be the competitor’s share. Since Pnew /S = eC /(1 + eC ) = σ(C), we can write log Pnew = log S + log σ(C). Differentiating, with σ ′ /σ = 1 − σ = qold , gives g0 = −∇ log S(θ0 ) − qold ∇C(θ0 ), (22) The first term changes the total probability of the two values, while the second changes their relative probability. Solving Eq. 22 for ∇C(θ0 ) and substituting it into Eq. 21, with g⊤ 0 sign(g0 ) = ∥g0 ∥1 , gives i η h η ∥g0 ∥1 v0 ≈ ∥g0 ∥1 + ∇ log S(θ0 )⊤ sign(g0 ) ≈ . (23) qold qold The last approximation drops the change in the total probability S along the sign step. This term is small in our updates, only one to two percent of ∥g0 ∥1 . Putting the steps together. Substituting Eq. 23 into Eq. 18 gives b = qold D , N ηA ∥g0 ∥1

(24)

where D is the initial gap and ∥g0 ∥1 determines the initial update speed. Since qold = (1 + e−D )−1 is close to one (qold ≈ 1), the main text drops this factor and writes Eq. 3. Here, ∥g0 ∥1 is the L1 norm of the initial gradient of log Pnew over all adapter parameters, measured directly for every update. 21

Preprint

Error

(10−4 )

0.1 0.0 2.5

20k

0.5

Update fraction s

1

(c) Speed evolution median first step update average

0.4

10k

0.2

chronological shuffled

5k

0.0 0

0.6

Update speed vk

Gradient ‖gk ‖1

Margin gain

0.2

(b) Gradient evolution 30k

measured linear estimate

0

0.5

Update progress k/N

1

0.0

0

0.5

Update progress k/N

(d) First-step prediction First-step speed v0

(a) Within-step linearity

1

0.3

first step later steps

0.1

0.03 0.03

0.1

0.3

η ‖g0 ‖1 /qold

Figure 11: The step-law approximations along update trajectories, for time-varying overwrite updates in the 110M C (blue) and S (orange) under Adam. (a) Measured margin gain against its linear estimate along the first update; the lower plot shows their difference in log-probability, in units of 10−4 . Each model is represented by the update with the median first-step error. (b) Gradient norm over update progress k/N . (c) Update speed over the same progress: dots mark the first step and dashed lines the median of per-update average speeds. In (b, c), solid curves are medians and shaded bands interquartile ranges. (d) First-step speed against its gradient-based estimate η∥g0 ∥1 /qold ; the diagonal is exact agreement, and faint points show later steps. Fitting the constant. We fit A once, as the median ratio of qold D/(η∥g0 ∥1 ) to the observed steps on half of the updates of the undated C at the three scales, which gives A = 1.27; every prediction in Section 5.2 uses the other half. Fitted separately on each model of Figure 4a, A lies between 1.19 and 1.41 for the chronological models and between 1.42 and 1.67 for the shuffled ones, whatever their scale or timestamps. C.3.2

HOW GOOD EACH APPROXIMATION IS

Figure 11 shows the approximations at work on the 110M C / S pair. Step 2 drops the curvature within one update, which is 0.2% of the margin change at the first step and under 1% at any later one, so the first-order sum recovers 99.4% of the margin travelled (Figure 11 (a)). Step 3 lets the first speed stand for the average; the average is 1.3 times the first speed in C and 1.8 times in S, whose gradient keeps growing as the update proceeds (Figure 11 (b) and Figure 11 (c)). Steps 4 and 5 replace the first speed by η∥g0 ∥1 /qold , which matches the measured v0 to within 3% (Figure 11 (d)). C.3.3

FACTORIZING THE ADAPTER GRADIENT

Figure 4 (c) draws the initial gradient module by module. A LoRA adapter sits on a linear module m with frozen weight Wm , and the training sentence passes through the module once per token p. Let xm,p ∈ Rdin be the module’s input at token p and ym,p ∈ Rdout its output. The adapter adds a low-rank term, α α ym,p = Wm xm,p + Bm Am xm,p = Wm xm,p + Bm zm,p , zm,p := Am xm,p ∈ Rr , (25) r r with Am ∈ Rr×din drawn at random and Bm ∈ Rdout ×r zero at the start, so that the adapter changes nothing until it is trained; α = r in all our updates. The objective log Pnew depends on Bm only through the outputs ym,p , so with δm,p := ∂ log Pnew /∂ym,p ∈ Rdout , the backward sensitivity that backpropagation delivers at the module’s output, the chain rule over tokens gives ∂ log Pnew αX ∂ log Pnew αX ⊤ = δm,p z⊤ = B δm,p x⊤m,p . (26) m,p , ∂Bm r p ∂Am r p m The gradient of Am vanishes at the start of the update because Bm = 0: the only path from Am to the output runs through Bm , and it carries nothing back. With Bm = 0 and α = r, the whole gradient the adapter at module m receives is therefore X gm = δm,p z⊤ (27) m,p . p

The loss gradient g0 = −∇ log Pnew is the adapters’ gradients laid end to end, with the A-parts all zero; Figure 4 (c) draws each adapter’s share of it as a cell area, both models on the chronological model’s scale. 22

Preprint

D

DETAILS ON FOUR APPLICATIONS

D.1

MIXING GENERALISTS AND SPECIALISTS

Corpora. Mix-C in Section 6.1 combines a generalist for the history with a specialist for the recent window by splitting each person’s records at 2021. Records before 2021 keep their timestamps and are shuffled. Records from 2021 onward lose half of their timestamps and keep each person’s chronological order. Mix-S contains the same records but shuffles them all. Both models are trained at 110M using the recipe in Appendix A and updated as in Section 3. Batch and sequential updates. A batch update writes many facts into one adapter by training on all of their sentences together. A sequential update applies a series of such batches, merging each batch into the model before the next. In Figure 5(c), a batch contains 2,400 facts, while a sequence contains ten batches of fifty. We measure forgetting after the batch, or after the final batch in the sequence, on facts about other people as in Appendix A. Across all three settings, Mix-C needs 16–44% fewer steps than Mix-S and forgets 5–57% less. D.2

RESHAPING A REAL-WORLD PRE-TRAINED MODEL

Data. We continue pre-training Gemma-2-9B-it (Team, 2024) on KairosQA (Pilchen et al., 2026), a set of real facts whose answers change over time. Each fact is a question with a different answer in different years: for example, the head coach of Tottenham Hotspur is Mauricio Pochettino in 2014– 2019, José Mourinho in 2020 and Antonio Conte in 2021–2023. This is like the employer of a person in our biographies. We use 5,324 such facts whose answer changes between 2001 and 2023, from eight kinds of question (for example, the head coach of a team, the team of a player and the head of a government), and drop the facts about subjects in the CounterFact evaluation set. Each record asks for one fact in Gemma’s chat format, for example “Who was the head coach of Tottenham Hotspur F.C. in 2019? Answer with the name only.”. An undated record drops the year. Stage. The stage trains on three kinds of text. The first asks each fact without a year over its recent years, 2016–2023 (52M tokens); this part makes the model a specialist in the latest answer. The second asks each fact with the year over its whole history, 2001–2023 (35M tokens); this part keeps the model a generalist about the past. The third is ordinary web text from FineWeb-edu (Penedo et al., 2024) (53M tokens), which keeps the model’s general ability. C-C shows the recent years of each fact in time order and mixes in the rest at random; C-S shuffles everything. The stage runs 5,174 steps, 140M tokens in total, at a constant learning rate of 3 × 10−6 . Updates. We adopted 170 overwrite updates that are not in the continual pre-training stage. Each update writes the new answer using the LoRA setup of Section 3 (rank 16, learning rate 10−6 ), with either an undated prompt or a dated prompt beginning with “In 2023”. Forgetting. The replay preserves historical knowledge. On questions about past years, C-C improves accuracy from 40% in Base to 99%. This shows that the stage strengthens the model’s generalist knowledge. But the replay only protects the facts it covers. KairosQA facts that never appear in the replay lose about 25% of the past-year answers that Base originally gets right, in both C-C and C-S, while facts from other sources, such as the CounterFact facts below, do not drop. This suggests that historical knowledge should be included in the replay if we want the model to keep it. Finally, the stage causes little loss in general ability (Table 9). On ARC-Easy (Clark et al., 2018), PIQA (Bisk et al., 2020), and HellaSwag (Zellers et al., 2019), C-C is at most 1.1 below Base. On CounterFact (Meng et al., 2022), C-C matches Base. Table 9: General ability of Gemma-2-9B-it before (Base) and after the continual pre-training stage (C-C, C-S). Accuracy (%) on each benchmark and on CounterFact facts, higher is better. Model

ARC-Easy (Clark et al., 2018)

PIQA (Bisk et al., 2020)

HellaSwag (Zellers et al., 2019)

CounterFact (Meng et al., 2022)

Base C-C C-S

85.8 84.7 84.4

81.1 81.3 80.9

52.8 52.8 52.8

89.0 88.9 89.1

23

Preprint

D.3

IMPROVING PUBLISHED-KNOWLEDGE EDITORS

Editors. We use the official implementations and hyperparameters of ROME (Meng et al., 2022), MEMIT (Meng et al., 2023), AlphaEdit (Fang et al., 2025) and ENCORE (Gupta et al., 2025), with ROME on layer 2 and the others on layers 1–4 of our 12-layer model; AlphaEdit’s null-space threshold is raised from 0.02 to 0.1, since at 0.02 layer 4 has no null space. Each editor makes the same 60 overwrite updates of time-varying facts in Mix-C and Mix-S. Cost is the FLOPs spent per fact written, where a fact counts as written once its new value’s probability passes that of the strongest value the model held before updating. Results. Every editor forgets 50–63% less on Mix-C than on Mix-S (Table 10). Table 10: The four editors on Mix-C and Mix-S, over the same 60 overwrite updates. Each cell is Mix-C / Mix-S, followed by the share Mix-C saves; lower is better. Updating cost: 1012 FLOPs per fact written. Forgetting, other people: percentage of other people’s facts whose answer the update changes. Forgetting, own history: percentage of the updated person’s dated history that the update changes. Bold: p < 0.05. Forgetting (%)

D.4

Editor

Updating cost

saved

Other people

reduced

Own history

reduced

ROME (Meng et al., 2022) MEMIT (Meng et al., 2023) AlphaEdit (Fang et al., 2025) ENCORE (Gupta et al., 2025)

5.36 / 5.65 0.78 / 1.44 0.77 / 1.31 0.75 / 1.10

+5% +46% +41% +32%

2.1 / 4.2 0.9 / 2.2 1.2 / 3.2 0.8 / 2.1

+50% +59% +63% +60%

81.8 / 80.2 14.8 / 22.1 16.0 / 24.7 14.5 / 18.8

−2% +33% +35% +23%

VERSIONED SOFTWARE CHANGE

Tool-call dataset. We build the tool-call dataset in the same way as the biography dataset in Section 3: a tool corresponds to a person, one of its six fields to an attribute, and the software version to the year. The dataset contains 25,000 tools and 25 versions, from v1 to v25; each tool appears 1,000 times, and a fixed set of tools is held out for injection updates. The six fields follow the same change rates as the biography attributes: package and function never change, host and authentication header change a few times, and path and input parameter change often. Each record gives the six fields of one tool at one version: In version v8, the Flint Payments API uses package flintpay_sdk and function revoke_refund. It carries host api.flintpay.com and authentication header X-Api-Key. It is called at path /v1/refunds with input parameter refund_id.

Training and updates. For pre-training from scratch, we follow Section 6.1: Mix-C shuffles the history (v1–v21) with version labels, while keeping each tool’s recent records (v22–v25) in version order with half of the version labels removed. Mix-S contains the same records but shuffles them all. For continual pre-training, we follow Appendix D.5: starting from a Base model trained on shuffled records with version labels, C-C presents a window of new versions in order and mixes them with a shuffled replay of the history, while C-S shuffles the same records. Every model receives single updates both with and without the version in the prompt. Results. We find that temporal order brings the same benefits to tool calls as to biographies. All models keep the old versions well: with the version in the prompt, query accuracy stays between 0.999 and 1.00. Without the version, Mix-C reaches 0.98 accuracy on the latest version, compared with 0.26 for Mix-S. After continual pre-training, C-C reaches 0.99, compared with 0.76 for C-S. The chronological models also update faster (Table 11). Without the version in the prompt, they need 14–48% fewer steps to update a changing field. With the version in the prompt, the saving becomes smaller, at 7–29%. D.5

RESHAPING A PRE-TRAINED MODEL IN SYNTHETIC WORLD

The continual pre-training stage of Section 6.2, applied to the 110M SD1/2 model of Section 4; Section 6.4 follows the same stage. 24

Preprint

Table 11: Median steps to update a field in the tool-call world, for slow and fast fields. Rows: pretraining from scratch (Mix-C / Mix-S) and continual pre-training (C-C / C-S). saved is the share of steps the chronological model saves; bold: p < 0.05. No version in prompt Overwrite steps

steps

Pre-training from scratch: Mix-C / Mix-S slow 53.0 / 102 +48% 38.5 / 64.0 fast 47.0 / 84.0 +44% 33.5 / 47.5

+40% +29%

55.0 / 77.0 48.5 / 68.5

+29% 40.5 / 57.0 +29% 32.0 / 42.0

+29% +24%

Continual pre-training: C-C / C-S slow 49.5 / 70.5 +30% 37.5 / 45.0 fast 43.5 / 64.5 +33% 34.5 / 40.0

+17% +14%

55.5 / 61.5 46.0 / 49.5

+10% 40.0 / 46.5 +7% 32.0 / 37.0

+14% +14%

Update steps

steps

Inject

saved

100

saved

Overwrite

(a) Accuracy vs. update cost Base C-S C-C

80 Better

60 40

0

50

Undated accuracy (%)

Forgetting (%)

Fact

Version in prompt

Inject

100

saved

steps

saved

(b) Update cost vs. forgetting

100

Better

50

0

one batch sequential

50 100

300

1000

Steps per update

Figure 12: Reshaping the 110M SD1/2 model. (a) Undated query accuracy against the median update steps for Base, C-S and C-C. (b) Steps per update against forgetting for single, batch and sequential updates of Base (grey), C-S (orange) and C-C (blue); lower left is better. Dataset setup. Base is the 110M SD1/2 model of Section 4. It is pre-trained on shuffled records, and only a random half of them carry a timestamp. Continual pre-training shows Base only records it has already seen, in two parts. The historical part has 100 timestamped records per person in shuffled order, and states every value the person has held. The recent part has 200 undated records per person in chronological order, and includes every change of value. C-C mixes the two parts and keeps the order within each person; C-S holds exactly the same records, shuffled. Each stage is one pass of about 2800 steps at batch 2,560 (about 0.6B tokens) at a constant learning rate of 6 × 10−5 , the final rate of pre-training. Results. We find that a short chronological stage gives Base a specialist for the present while keeping its past. C-C raises Base’s undated query accuracy from 25% to 93% and keeps its dated accuracy at 98%. C-S, trained on the same records, reaches only 22% (Figure 12 (a)). C-C also updates faster: it needs up to 1.6× fewer steps than Base for single updates, 2.1× fewer in a batch and 1.3× fewer in a sequence. It also forgets 10–15% less (Figure 12 (b)).

25

Related documents

• Specialists Learn Faster: Knowledge Organization in Pre-training Shapes Knowledge Update Efficiency#1298331
• Discovering and Preserving Category Correlation Knowledge via Adaptive Reciprocal Knowledge Distillation#1071006
• Learning to Read Out: Unembedding Dynamics in Language Model Pretraining#1356361
• Knowledge Distillation under Teacher Misspecification: An Order-Parameter Analysis of the Gap between Teacher Mimicry and Task Performance#794254
• Knowledge Distillation under Teacher Misspecification: An Order-Parameter Analysis of the Gap between Teacher Mimicry and Task Performance#790235