Optimal Training-Time Scaling in Gradual Adaptation

Preprint

O PTIMAL T RAINING -T IME S CALING IN G RADUAL A DAPTATION Zonghuan Xu∗ Fudan University Shanghai, China 2430XH10002@m.fudan.edu.cn

Krishna Harish Lawrence E. Elkins High School Missouri City, Texas, USA krishnaharish2009@gmail.com

arXiv:2608.04927v1 [cs.LG] 5 Aug 2026

A BSTRACT In gradual adaptation, how should the training time on each task change as the number of intermediate tasks increases? We study this question for overparameterized linear regression tasks that change smoothly and share a zero-loss solution. With N tasks and training time sN on each, the final learning progress converges to a continuum curve when N sN → τ . The limiting progress is Θ(τ ) for small τ and Θ(τ −1 ) for large τ , so both very short and very long training produce little progress. It follows that optimal per-task training times scale as s⋆N = Θ(N −1 ), equivalently N s⋆N = Θ(1). Experiments on gradually rotated MNIST and a natural Yearbook time shift are consistent with less per-task training as the path is divided more finely.

1

I NTRODUCTION

Many learning systems must process sequences of slowly changing training tasks. In gradual domain adaptation, intermediate distributions guide a learner from a source domain to a target domain (Kumar et al., 2020; Chen & Chao, 2021; Wang et al., 2022a). In curriculum learning, related tasks are ordered by difficulty or structure (Bengio et al., 2009; Kumar et al., 2010; Graves et al., 2017; Weinshall et al., 2018; Hacohen & Weinshall, 2019; Klink et al., 2022; Huang et al., 2022). In continual test-time adaptation, a model is repeatedly updated as its deployment data evolve (Sun et al., 2020; Wang et al., 2021; 2022b; Niu et al., 2022; Zhang et al., 2022; Niu et al., 2023; Döbler et al., 2023; Yuan et al., 2023). We use gradual adaptation for this setting of sequential learning over nearby tasks, which also falls within the broader scope of continual learning (Parisi et al., 2019; De Lange et al., 2022; van de Ven et al., 2022). The same gradual adaptation process can be divided into different numbers of tasks. More tasks make adjacent problems closer, but also require the learner to update more frequently. Suppose the sequence contains N tasks and the learner trains for time sN on each task. The task count N controls the spacing between adjacent tasks, while sN controls the amount of optimization at each task. We ask: As the number of tasks N increases, how long should each task be trained to maximize the learning progress after the entire sequence? Existing theory studies training time and interference across sequential tasks from different angles. Early-stopping theory treats training time as a statistical regularizer (Yao et al., 2007; Raskutti et al., 2014; Ali et al., 2019), and stability analyses relate training duration to generalization (Hardt et al., 2016). These results explain why stopping before convergence can improve prediction from a finite sample. Our effect is instead present for noiseless population losses and concerns the state passed from one task to the next. Domain-adaptation theory studies when transfer across distributions is possible, while practical methods align features or predictions between source and target domains (Ben-David et al., 2010; Ganin et al., 2016; Hoffman et al., 2018; Saito et al., 2018). Gradual self-training inserts intermediate distributions and studies how their number and placement affect target error (Kumar et al., 2020; Chen & Chao, 2021; Wang et al., 2022a). These analyses represent each local adaptation by an ∗

Corresponding author.

1

Preprint

empirical risk minimization step. They do not treat the amount of optimization at each intermediate task as a variable that changes with the resolution of the path. Curriculum learning studies the order and pace at which examples or tasks are presented (Bengio et al., 2009; Kumar et al., 2010; Graves et al., 2017; Weinshall et al., 2018; Hacohen & Weinshall, 2019). Continual-learning methods control interference through parameter regularization, stored examples, and gradient projections (Kirkpatrick et al., 2017; Zenke et al., 2017; Ritter et al., 2018; Benzing, 2022; Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2019; Farajtabar et al., 2020). Recent theory studies how training regimes and regularized updates affect forgetting (Mirzadeh et al., 2020; Zhao et al., 2024; Levinstein et al., 2025), including intermediate training regimes (Lesort et al., 2023; Graldi et al., 2025; Mori et al., 2025). Exact analyses of linear regression characterize forgetting under fixed and cyclic task sequences (Evron et al., 2022; Swartworth et al., 2023), while recent work studies continual models under SGD and task distributions (Evron et al., 2026; Xu & Ma, 2026). These works establish that task order and training regime matter. We isolate a different question by holding the underlying task path fixed and coupling its resolution N to the local training time sN . We study this problem in overparameterized linear regression. Task t uses the population squared loss  1  Lt (w) = Et (x⊤ w − y)2 , 2 and all tasks share a zero-loss solution w⋆ , so that y = x⊤ w⋆ almost surely on every task. To compare different task counts, we index the complete gradual adaptation process by t ∈ [0, 1] and select N tasks at tj = j/N , j = 1, . . . , N . Let H(t) = Et [xx⊤ ] be the Hessian of task t. Gradient flow has an exact exponential solution: training task j for time sN updates the parameter error as vj = e−sN H(tj ) vj−1 , Let M=

vN = e−sN H(tN ) · · · e−sN H(t1 ) v0 . \

arg min Lt (w) w

t∈[0,1]

be the common solution set, and suppose that the initial model w0 already fits the first task. With w̄ = ProjM (w0 ) and vj = wj − w̄, the affine structure gives dist(wj , M) = ∥vj ∥. We therefore measure the learning progress over the complete sequence by PN,sN (v0 ) = dist(w0 , M)2 − dist(wN , M)2 = ∥v0 ∥2 − ∥vN ∥2 . The optimal per-task training time s⋆N (v0 ) maximizes this quantity. Our first theorem gives the continuous limit as the task count grows. If N sN → τ < ∞, the discrete updates converge to ẋτ (t) = −τ H(t)xτ (t), xτ (0) = v0 . Here τ is the effective training time spent along the complete task path. The corresponding progress converges to Pτ (v0 ) = ∥v0 ∥2 − ∥xτ (1)∥2 , which satisfies the exact identity Z 1 Pτ (v0 ) = 2τ xτ (t)⊤ H(t)xτ (t) dt. 0

This gives a common limiting curve τ 7→ Pτ (v0 ) for comparing different choices of N and sN . Our second theorem characterizes both ends of this limiting curve. Whenever the initial model is not already in the common solution set, Pτ (v0 ) = Θ(τ )

(τ ↓ 0),

Pτ (v0 ) = Θ(τ −1 )

(τ → ∞).

When the training time is large, the model remains close to the current task’s zero-loss subspace at every point along the path. As the tasks change, these subspaces rotate gradually, and the dominant effect is to change the direction of the remaining error rather than its norm. This explains why the progress produced by overtraining vanishes. The theorem gives the decay rate Θ(τ −1 ). Both too little and too much training therefore give vanishing progress. Only N sN = Θ(1) 2

Preprint

can retain a positive limiting amount of progress as the task count grows. It follows that s⋆N (v0 ) = Θ(N −1 ). Dividing the same task path more finely requires reducing the training time on each task by the same proportion. Keeping sN fixed while increasing N therefore does more than refine the discretization: it also sends the total effective training time N sN to infinity, where the progress vanishes. Thus, in the present setting, task resolution and per-task optimization time cannot be treated as independent design choices. We also characterize the first-order behavior in the regime N sN → ∞. For every fixed finite per-task training time, the first-order coefficient is strictly larger than the coefficient obtained by training each task to convergence. If progress is instead measured through any strictly increasing function of squared distance, the ordering of training times and the optimal training time remain unchanged. Contributions.

The paper makes four main contributions.

• We formulate per-task training time as an optimization problem over the task count N and training time sN , using the reduction in distance to the common solution set after all tasks as the learning objective. • We derive the limiting learning curve controlled by the effective training time τ and prove its Θ(τ ) growth for small τ and Θ(τ −1 ) decay for large τ . • We prove that only N sN = Θ(1) can retain positive limiting progress as the task count grows, yielding the optimal scaling s⋆N (v0 ) = Θ(N −1 ). • We test the prediction on rotated MNIST and temporal Yearbook data. Across both settings, finer task divisions favor less training per task overall, including under natural temporal shift.

2

P ROBLEM S ETUP

We first specify the overparameterized regression tasks and their smoothly varying zero-loss spaces. We then define the sequential gradient-flow updates, learning progress relative to the common solution set, and the optimal per-task training time. 2.1

A SMOOTH FAMILY OF REGRESSION TASKS

Let t ∈ [0, 1] index a gradually changing task. On task t, a feature–label pair (x, y) ∈ Rd × R follows a distribution Dt . We assume that there is a parameter w⋆ ∈ Rd such that y = x⊤ w⋆ almost surely under every Dt . The population squared loss is therefore   1 EDt (x⊤ w − y)2 2 1 = (w − w⋆ )⊤ H(t)(w − w⋆ ), 2

Lt (w) :=

(1) H(t) := EDt [xx⊤ ].

Here H(t) is both the population Hessian and the feature second moment. For every z ∈ Rd ,   z ⊤ H(t)z = EDt (x⊤ z)2 ≥ 0,

(2)

so H(t) is positive semidefinite. Its kernel consists of parameter perturbations whose linear predictions remain unchanged on task t: V (t) := ker H(t) = {z : x⊤ z = 0 almost surely under Dt }. A parameter change along z ∈ V (t) leaves the predictions on task t unchanged. For a finite regression dataset with design matrix X(t) ∈ Rnt ×d and labels y(t) = X(t)w⋆ , the same representation holds with H(t) = X(t)⊤ X(t)/nt . The overparameterized regime corresponds to rank H(t) < d, so each task leaves some parameter directions unconstrained. 3

Preprint

We assume that H : [0, 1] → Rd×d is a C 3 path of symmetric positive-semidefinite matrices with constant rank. Its positive eigenvalues satisfy the uniform bounds 0 < µ ≤ λ ≤ L < ∞. (3) Constant rank ensures that V (t) varies smoothly with t, while Equation (3) keeps every nonzero curvature between µ and L. Let \ P (t) = ProjV (t) , Q(t) = I − P (t), K= V (t). t∈[0,1]

Thus P (t) projects onto the parameter directions unconstrained by task t, while Q(t) projects onto their orthogonal complement. The minimizer set of task t is M(t) = w⋆ + V (t), and the solution set shared by the full path is w⋆ + K. 2.2

S EQUENTIAL TRAINING AND LEARNING PROGRESS

Select tasks on the uniform grid tj = j/N , j = 0, . . . , N , and write v = w − w⋆ . On task tj , gradient flow for Equation (1) satisfies d v(u) = −H(tj )v(u). du Training for time s ≥ 0 therefore applies the exact update vj = e−sH(tj ) vj−1 . (4) Throughout, s denotes continuous gradient-flow time. A run of k gradient-descent steps with learning rate η approximates Equation (4) with s ≈ ηk in the small-step regime. For task-wise fitting to convergence, we define e−∞H(t) := P (t). On the grid above, suppose the learner processes t1 , . . . , tN in order and runs gradient flow for the same time s on every task. Repeated application of Equation (4) gives ΦN,s := e−sH(tN ) e−sH(tN −1 ) · · · e−sH(t1 ) , vN = ΦN,s v0 . (5) We begin immediately after the initial task has been fitted, so v0 ∈ V (0). This convention focuses the analysis on the learning progress contributed by the N subsequent tasks. Decompose the initial error orthogonally as ∥

v0 = v0 + v0⊥ ,

∥

v0 ∈ K,

v0⊥ ∈ K⊥ .

∥

Since K ⊆ ker H(t) for every t, each update fixes v0 and preserves K⊥ . Hence the distances to the common solution set before and after training are dist(w⋆ + v0 , w⋆ + K) = ∥v0⊥ ∥,

dist(w⋆ + ΦN,s v0 , w⋆ + K) = ∥ΦN,s v0⊥ ∥.

Learning progress therefore depends only on v0⊥ . To simplify notation, we now rename v0⊥ as v0 . Together with the assumption that the initial task has been fitted, this gives v0 ∈ V (0) ∩ K⊥ . The central quantity in our analysis is the reduction in squared distance to the common solution set after the N subsequent tasks: PN,s (v0 ) = ∥v0 ∥2 − ∥ΦN,s v0 ∥2 ≥ 0,

v0 ∈ V (0) ∩ K⊥ .

(6)

This distance has a uniform loss interpretation over the task path. Define the worst population loss of a model w along the path by Rpath (w) := sup Lt (w). t∈[0,1]

Proposition 1 (Loss interpretation of the learning-progress measure). Suppose K⊥ ̸= {0}, and let Z 1
1 H̄ := H(t) dt, cH := λmin H̄ K⊥ . 2 0 Then cH > 0 and, for every w ∈ Rd , L dist(w, w⋆ + K)2 . (7) 2 Consequently, dist(wm , w⋆ + K) → 0 for a sequence (wm ) if and only if supt∈[0,1] Lt (wm ) → 0. cH dist(w, w⋆ + K)2 ≤ Rpath (w) ≤

4

Preprint

Thus squared distance to the common solution set is equivalent, up to path-dependent constants, to the worst population loss along the task path. Equation (6) records the reduction of this state measure along the task sequence. The proof is in Section A.1. Proposition 2 (Exact decomposition of learning progress). For every N and s ∈ [0, ∞], PN,s (v0 ) =

N D E X
vj−1 , I − e−2sH(tj ) vj−1 .

(8)

j=1

Proof. Since H(tj ) is symmetric, Equation (4) gives D E
∥vj−1 ∥2 − ∥vj ∥2 = vj−1 , I − e−2sH(tj ) vj−1 . Summing over j telescopes to Equation (6). Each summand is exactly the reduction in squared error produced on one task. Equation (8) exposes two coupled roles of the per-task training time. A larger s removes more error on the current task, while also changing the state inherited by every later task. Not fitting the current task completely leaves a residual on which subsequent, slightly different tasks can act. Across a dense task sequence, these later reductions can accumulate and produce more progress over the full path than exact fitting. Section 3 quantifies this effect. All subsequent results characterize how PN,s (v0 ) depends on the task count N and the per-task training time s. For each task count N , define an optimal per-task training time by s⋆N (v0 ) ∈ arg max PN,s (v0 ).

(9)

s∈[0,∞]

Because ΦN,s has a limit as s → ∞, the maximum in Equation (9) is attained.

3

T RAINING -T IME S CALING IN G RADUAL A DAPTATION

We first pass from the discrete task sequence to a continuum learning curve indexed by the effective training time τ = N sN . We then characterize its small- and large-budget limits and use them to identify the optimal scaling of the per-task training time. 3.1

T HE LIMITING LEARNING CURVE

For an effective training time τ ≥ 0, let xτ : [0, 1] → Rd solve ẋτ (t) = −τ H(t)xτ (t),

xτ (0) = v0 ,

(10)

and define Pτ (v0 ) := ∥v0 ∥2 − ∥xτ (1)∥2 .

(11)

Theorem 1 (Limit of dense task sequences). Under the assumptions of Section 2, if N sN → τ ∈ [0, ∞), then ΦN,sN v0 −→ xτ (1), PN,sN (v0 ) −→ Pτ (v0 ). (12) For every T < ∞, the finite-sequence approximation satisfies ΦN,τ /N v0 − xτ (1) = OT (N −1 ),

(13)

PN,τ /N (v0 ) − Pτ (v0 ) = OT (N −1 ).

(14)

sup 0≤τ ≤T

sup 0≤τ ≤T

Moreover, Z 1 Pτ (v0 ) = 2τ

xτ (t)⊤ H(t)xτ (t) dt.

0

5

(15)

Preprint

Proof sketch.

Put bN = N sN . Since bN → τ , each update satisfies

bN H(tj ) + O(N −2 ) N uniformly along the path. The one-step error relative to Equation (10) is O(N −2 ) + O(|bN − τ |/N ). Because every update is a contraction, these errors are not amplified by the remaining product. After N steps the endpoint error is therefore O(N −1 ) + O(|bN − τ |), which tends to zero. When bN = τ on a compact interval, this also gives Equation (13). The progress rate follows because all trajectories are uniformly bounded. Differentiating ∥xτ (t)∥2 and integrating from 0 to 1 gives Equation (15). Complete estimates are in Section A. e−sN H(tj ) = I −

3.2

O PTIMAL PER - TASK TRAINING TIME

The condition v0 ∈ V (0) ∩ K⊥ means that the initial point fits the source task and that components invisible to every task have been removed. Hence a nonzero v0 is detected by at least one later task. Theorem 2 (Optimal per-task training time). Let v0 ∈ V (0) ∩ K⊥ be nonzero. Then: (i) The limiting progress is strictly positive for every finite τ > 0 and satisfies Pτ (v0 ) = Θ(τ )

Pτ (v0 ) = Θ(τ −1 )

(τ ↓ 0),

(τ → ∞).

(16)

(ii) For every sequence of per-task training times sN , N sN → 0 =⇒ PN,sN (v0 ) → 0, N sN → τ ∈ (0, ∞) =⇒ PN,sN (v0 ) → Pτ (v0 ) > 0, N sN → ∞ =⇒ PN,sN (v0 ) → 0.

(17)

(iii) The limiting curve attains its maximum at one or more points in (0, ∞). For all sufficiently large N , every optimizer in Equation (9) is finite and satisfies s⋆N (v0 ) = Θ(N −1 ), and every accumulation point of N s⋆N (v0 ) maximizes Pτ (v0 ). In particular, if the limiting curve has a unique maximizer τ ⋆ (v0 ), then s⋆N (v0 ) =

τ ⋆ (v0 ) + o(N −1 ). N

(18)

In particular, fitting every task to convergence belongs to the third regime in Equation (17): its progress tends to zero as N → ∞. By contrast, sN = τ /N with 0 < τ < ∞ retains strictly positive limiting progress. Proof idea. The distinction is easiest to see by comparing complete trajectories rather than a single update. See Figure 1. Put h = N −1 . After exact fitting, vj ∈ V (tj ). Because adjacent zero-loss spaces differ by only O(h), the next task exposes only an O(h) normal component. Projecting this component away turns the vector by O(h), but changes its squared length by only O(h2 ). Across N = h−1 tasks, the vector therefore mainly follows the moving zero-loss space along an almost constant-radius path, with only O(N −1 ) total progress. exact fitting

(a) t = 1/3

sN = τ ⋆ /N

(b) t = 2/3

V (t)

(c) t = 1

Figure 1: Limiting trajectories for the rotating-task example in Section 4. All panels use the same scale. The black dot is the common solution and the blue ray is the current zero-loss space. Exact fitting follows this space along the outer orange arc without approaching the dot. At the optimal scale sN = τ ⋆ /N , finite training lags behind the current zero-loss space but moves steadily inward. By the final task it is visibly closer to the common solution. 6

Preprint

With sN = τ h, no task removes its visible component completely: e−sN H(tj ) vj−1 = vj−1 − τ hH(tj )vj−1 + O(h2 ). The remaining component need not shrink to O(h) before the next task arrives. Consequently the O(h) decreases in squared length can accumulate over N = h−1 tasks, giving the nonzero limit in Equation (15). In the figure this is the difference between merely following the outer arc and moving inward. The two failures occur on opposite sides of this balance. If N sN → 0, the total amount of training vanishes. If N sN → ∞, the model is driven increasingly close to each current zero-loss space. The appendix proves the uniform estimate   1 PN,sN (v0 ) = O . N (1 − e−µsN ) Its right-hand side tends to zero whenever N sN → ∞. Fix any finite τ̄ > 0. The choice sN = τ̄ /N gives strictly positive limiting progress. The two tail bounds rule out N s⋆N → 0 and N s⋆N → ∞, so every optimizer satisfies N s⋆N = Θ(1). Uniform convergence on this bounded range then implies that every accumulation point of N s⋆N maximizes Pτ (v0 ). Complete expansions and uniform remainder estimates are given in Sections A.3 and A.3.1. First-order comparison at fixed per-task time. The main theorem concerns sN = Θ(N −1 ), for which progress remains of constant order. A different question is what happens when the per-task time is fixed as N grows. Let Cs (v0 ) = 2⟨v0 , Gs v0 ⟩ and CH (v0 ) = 2⟨v0 , GH v0 ⟩, where the explicit path-dependent operators are defined in Sections A.3 and A.3.1. Proposition 3 (First-order comparison in the overtraining regime). For v0 ∈ V (0) ∩ K⊥ and every fixed s ∈ (0, ∞], Cs (v0 ) + o(N −1 ). (19) PN,s (v0 ) = N If sN ↓ 0 and N sN → ∞, then PN,sN (v0 ) =

CH (v0 ) + o((N sN )−1 ). N sN

(20)

Moreover, Cs (v0 ) > C∞ (v0 ) for every fixed finite s. Hence PN,s (v0 ) > PN,∞ (v0 ) for all sufficiently large N . The strict comparison has a direct interpretation. Along a positive-eigenvalue direction of H(t) with eigenvalue λ, finite training leaves the fraction ρ = e−sλ of the incoming error. Its contribution to the first-order coefficient can be written as 1 1 ρ coth(sλ/2) = + . 2 2 1−ρ

(21)

Exact fitting sets ρ = 0. Finite fitting adds the positive second term, which records residual error that later, slightly different tasks can still reduce. Thus fixed finite fitting and exact fitting both have progress of order N −1 , but the finite-time coefficient is strictly larger. The path-dependent coefficients and proof are in Section A.3.1. Robustness to monotone distance transformations. This conclusion does not depend on using f squared distance itself. For any strictly increasing f : [0, ∞) → R, define PN,s (v0 ) := f (∥v0 ∥2 ) − 2 2 f (∥ΦN,s v0 ∥ ). Writing r = ∥v0 ∥ gives the exact identity
f PN,s (v0 ) = f (r) − f r − PN,s (v0 ) . (22) The right-hand side is strictly increasing in PN,s , so signs, comparisons across training times, and maximizers are unchanged. The conclusion need not hold for anisotropic distances. An explicit reversal is given in Section A.4. 7

Preprint

4

N UMERICAL E XPERIMENTS

Our experiments serve two complementary purposes. The regression experiments verify the theory within its assumptions, while Rotated MNIST and Yearbook deliberately depart from those assumptions to test whether the predicted training-time trend persists under nonlinear models, stochastic optimization, finite samples, cross-entropy losses, and natural distribution shift. In every experiment, the underlying path is held fixed while the number of visited tasks changes. Rotating regression tasks. We first consider a two-dimensional rank-one problem whose task direction rotates smoothly by π/2. With sN = τ /N , the finite-task progress curves approach a common curve with an interior optimum at τ ⋆ ≈ 2.20, and N s⋆N approaches the same value. Training every task to convergence instead gives progress of order N −1 . The closed-form derivation and a higher-dimensional experiment are in Section B.

N=4

N = 16

N = 64

N = 256

N=8

N = 32

N = 128

limit

The best total time stabilizes

0.8

Scaled optimal time NsN⋆

Learning progress N, s

Progress peaks at an intermediate total time

0.6 0.4 0.2 0.0 10−2

10−1

100

101

2.20 2.15 2.10 2.05

finite-task optimum NsN⋆ limit τ ⋆ = 2.20

2.00

102

101

102

Effective training time τ = Ns

103

Number of tasks N

Figure 2: Training time has an intermediate optimum on a rotating task path. Left: Finite-task progress curves approach the same rise-and-decay limit. Right: The best total training time N s⋆N approaches τ ⋆ ≈ 2.20, so the best per-task time is approximately 2.20/N . Rotated MNIST. We next train an MLP on upright MNIST and update it along a fixed rotation path from 0◦ to 60◦ . Every task count is evaluated on the same 13 angles. Across ten seeds, the estimated optimum follows k ⋆ ∝ N −0.84 , with bootstrap 95% interval [0.76, 1.03] for the exponent. At N = 32, the final-angle loss continues to improve after the mean loss across the rotation path begins to increase. Full details are in Section B.3. Average loss across all angles

1.2 1.0 0.8

4 2

64 128 total updates Nk 16

32

256

512

4

8

2.5 2.0 1.5 1.0 0.5

1/N reference estimated optimum

1

8

all angles final angle

3.0

0.6

0

All angles vs. final angle (N=32) 3.5

8

estimated best k

mean cross-entropy

Best updates per task N=16 N=32

cross-entropy

N=4 N=8

1.4

16

number of tasks N

32

0.0

0

1

2

4

8

16

updates per task k

32

64 128

Figure 3: Finer rotation steps favor fewer updates per task. Left: Mean loss across the rotation path against total updates N k. Middle: Estimated optimal updates per task and the N −1 reference. Right: At N = 32, final-angle loss keeps decreasing after the mean loss across all angles turns upward. Error bars and bands are 95% intervals over ten seeds. Yearbook over time. Finally, we use Yearbook photographs spanning 1930–2013 (Ginosar et al., 2017; Yao et al., 2022). Fixed-width temporal smoothing defines the same path for every N . Across 8

Preprint

five seeds, the estimated optimum falls overall from 79.9 updates per task at N = 4 to 5.0 at N = 32; the fitted exponent is 1.37, with bootstrap 95% interval [0.79, 1.66]. At N = 32, increasing k from 4 to 128 lowers target-time cross-entropy from 0.216 to 0.051 but raises the mean across the time path from 0.325 to 0.535. Full details and limitations are in Section B.4. Best updates per task 27

N=16 N=32

All years vs. target year (N=32)

1/N reference estimated optimum

26 25 24 23

8

16

32

64 128 256 512

total updates Nk

0.8 0.6 0.4 0.2

22

0

1.0

cross-entropy

N=4 N=8

estimated best k

mean cross-entropy

Average loss across years 1.1 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3

4

8

16

number of tasks N

32

all years target year

0.0

0

1

2

4

8

16

updates per task k

32

64 128

Figure 4: Finer temporal steps favor less training per task overall. Left: Mean loss across years against total updates N k. Middle: Estimated optimal updates per task and the N −1 reference. Right: At N = 32, target-year loss keeps decreasing while the mean loss across years eventually increases. Error bars and bands are 95% intervals over five seeds. Across all three experiments, increasing N refines a fixed path between unchanged endpoints: the intermediate tasks follow a smoothly rotating rank-one regression direction, image rotations from 0◦ to 60◦ , or temporal sampling weights whose centers move across years. Rotating regression recovers s⋆N ∝ N −1 , while the confidence intervals for the fitted exponents in Rotated MNIST and Yearbook both contain the theoretical value 1, supporting the same training-time scaling from a solvable linear model to nonlinear finite-sample training and natural temporal data.

5

D ISCUSSION

Our analysis separates current-task fit from progress over the complete task path. Longer training improves current-task fit but changes the state from which later tasks begin. Fitting a local task to convergence therefore affects the subsequent training trajectory and whole-path performance, not just local accuracy. This differs from conventional early stopping, which limits fitting of sample noise to improve generalization (Yao et al., 2007; Raskutti et al., 2014; Ali et al., 2019); here the population losses are noiseless and share a solution. The phenomenon also echoes the plasticity–forgetting intuition in continual learning (Parisi et al., 2019; De Lange et al., 2022): fitting the current task less tightly leaves more room to adjust on the next task, which can be understood as greater plasticity, whereas full fitting need not yield greater path progress. Our results provide one theoretical account of this intuition. We analyze population gradient flow for overparameterized linear regression along a smooth, constantrank Hessian path with a shared solution and a uniform positive spectral gap. These assumptions make interactions among successive tasks explicit. Strictly increasing transformations of squared Euclidean distance preserve the ordering of training times. The theory then predicts that, when increasing N refines a fixed continuous path, the optimal per-task time decreases as N −1 . Rotating regression recovers this structure, including a finite optimal total training time and s⋆N ∝ N −1 . Rotated MNIST extends this test to finite samples, stochastic optimization, cross-entropy, and a nonlinear model; Yearbook replaces controlled rotations with continuous distribution change in natural temporal data. In both experiments, the fitted exponent’s confidence interval contains the theoretical value 1, and continued training can improve the target task while worsening full-path performance. These results support the N −1 training-time relation beyond the solvable linear model. Broader gradual-adaptation problems include, among others, gradual self-training, test-time adaptation, and continual updating (Kumar et al., 2020; Wang et al., 2022a; Song et al., 2023; Dorka et al., 2023; Zhang et al., 2025). Their training budgets may need to change with data frequency, task 9

Preprint

duration, and environmental change. Our smooth-path model provides a theoretical starting point for relating task granularity to training time. Future work can explore more abstract formulations for recurring tasks, abrupt switches, or model-dependent changes, and ask whether corresponding training-time scales persist. AI USE STATEMENT Generative AI assisted with proof checking, experiments, literature search, and manuscript editing. The authors reviewed all AI-assisted content and take responsibility for the paper. R EPRODUCIBILITY STATEMENT Complete proofs and experimental details are in Sections A and B. The supplement includes scripts, fixed seeds, CSV files, and figure source for all reported results.

R EFERENCES Alnur Ali, J. Zico Kolter, and Ryan J. Tibshirani. A continuous-time view of early stopping for least squares regression. In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pp. 1370–1378. PMLR, 2019. URL https://proceedings.mlr.press/v89/ali19a.html. J. E. Avron, M. Fraas, G. M. Graf, and P. Grech. Adiabatic theorems for generators of contracting evolutions. Communications in Mathematical Physics, 314:163–191, 2012. doi: 10.1007/s00220-012-1504-1. Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine Learning, 79:151–175, 2010. doi: 10.1007/s10994-009-5152-4. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th International Conference on Machine Learning, pp. 41–48, 2009. doi: 10.1145/1553374.1553380. Frederik Benzing. Unifying importance based regularisation methods for continual learning. In Proceedings of the 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pp. 2372–2396. PMLR, 2022. Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with A-GEM. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Hkf2_sC5FX. Hong-You Chen and Wei-Lun Chao. Gradual domain adaptation without indexed intermediate domains. In Advances in Neural Information Processing Systems, volume 34, 2021. URL https://proceedings.neurips.cc/paper/2021/hash/ 45017f6511f91be700fda3d118034994-Abstract.html. Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(7):3366–3385, 2022. doi: 10.1109/TPAMI.2021.3057446. Mario Döbler, Robert A. Marsden, and Bin Yang. Robust mean teacher for continual and gradual test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7704–7714, 2023. URL https://openaccess.thecvf.com/ content/CVPR2023/html/Dobler_Robust_Mean_Teacher_for_Continual_ and_Gradual_Test-Time_Adaptation_CVPR_2023_paper.html. Nicolai Dorka, Tim Welschehold, and Wolfram Burgard. Dynamic update-to-data ratio: Minimizing world model overfitting. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=BJv0jo4hqEZ. 10

Preprint

Itay Evron, Edward Moroshko, Rachel A. Ward, Nathan Srebro, and Daniel Soudry. How catastrophic can catastrophic forgetting be in linear regression? In Proceedings of the 35th Conference on Learning Theory, volume 178 of Proceedings of Machine Learning Research, pp. 4028–4079. PMLR, 2022. URL https://proceedings.mlr.press/v178/evron22a.html. Itay Evron, Ran Levinstein, Matan Schliserman, Uri Sherman, Tomer Koren, Daniel Soudry, and Nathan Srebro. From continual learning to SGD and back: Better rates for continual linear models. In Proceedings of the 37th International Conference on Algorithmic Learning Theory, volume 313 of Proceedings of Machine Learning Research, pp. 1–50. PMLR, 2026. URL https: //proceedings.mlr.press/v313/evron26a.html. Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pp. 3762–3773. PMLR, 2020. Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. Journal of Machine Learning Research, 17(59):1–35, 2016. Shiry Ginosar, Kate Rakelly, Sarah M. Sachs, Brian Yin, Crystal Lee, Philipp Krähenbühl, and Alexei A. Efros. A century of portraits: A visual historical record of american high school yearbooks. IEEE Transactions on Computational Imaging, 3(3):421–431, 2017. doi: 10.1109/TCI. 2017.2699865. Jacopo Graldi, Alessandro Breccia, Giulia Lanzillotta, Thomas Hofmann, and Lorenzo Noci. The importance of being lazy: Scaling limits of continual learning. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 20326–20367. PMLR, 2025. Alex Graves, Marc G. Bellemare, Jacob Menick, Rémi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1311–1320. PMLR, 2017. URL https://proceedings.mlr.press/v70/graves17a.html. Guy Hacohen and Daphna Weinshall. On the power of curriculum learning in training deep networks. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2535–2544. PMLR, 2019. Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In Proceedings of the 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1225–1234. PMLR, 2016. URL https://proceedings.mlr.press/v48/hardt16.html. Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei A. Efros, and Trevor Darrell. CyCADA: Cycle-consistent adversarial domain adaptation. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1989–1998. PMLR, 2018. Peide Huang, Mengdi Xu, Jiacheng Zhu, Laixi Shi, Fei Fang, and Ding Zhao. Curriculum reinforcement learning using optimal transport via gradual domain adaptation. In Advances in Neural Information Processing Systems, volume 35, 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/hash/ 4556f5398bd2c61bd7500e306b4e560a-Abstract-Conference.html. Tosio Kato. On the adiabatic theorem of quantum mechanics. Journal of the Physical Society of Japan, 5(6):435–439, 1950. doi: 10.1143/JPSJ.5.435. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521–3526, 2017. doi: 10.1073/pnas.1611835114. 11

Preprint

Pascal Klink, Haoyi Yang, Carlo D’Eramo, Jan Peters, and Joni Pajarinen. Curriculum reinforcement learning via constrained optimal transport. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 11341–11358. PMLR, 2022. URL https://proceedings.mlr.press/v162/klink22a.html. Ananya Kumar, Tengyu Ma, and Percy Liang. Understanding self-training for gradual domain adaptation. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5468–5479. PMLR, 2020. URL https: //proceedings.mlr.press/v119/kumar20c.html. M. Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. In Advances in Neural Information Processing Systems, volume 23, 2010. Timothée Lesort, Oleksiy Ostapenko, Pau Rodrı́guez, Diganta Misra, Md Rifat Arefin, Laurent Charlin, and Irina Rish. Challenging common assumptions about catastrophic forgetting and knowledge accumulation. In Proceedings of the 2nd Conference on Lifelong Learning Agents, volume 232 of Proceedings of Machine Learning Research, pp. 43–65. PMLR, 2023. Ran Levinstein, Amit Attia, Matan Schliserman, Uri Sherman, Daniel Soudry, Tomer Koren, and Itay Evron. Optimal rates in continual linear regression via increasing regularization. In Advances in Neural Information Processing Systems, volume 38, 2025. URL https://proceedings.neurips.cc/paper_files/paper/2025/ hash/5712575413bb58ae6c7a17aa4f2d64a2-Abstract-Conference.html. David Lopez-Paz and Marc’Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, volume 30, 2017. Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh. Understanding the role of training regimes in continual learning. In Advances in Neural Information Processing Systems, volume 33, pp. 7308–7320, 2020. Francesco Mori, Stefano Sarao Mannelli, and Francesca Mignacco. Optimal protocols for continual learning via statistical physics and control theory. In The Thirteenth International Conference on Learning Representations, 2025. Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 16888–16905. PMLR, 2022. Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=g2YraF75Tj. German I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54–71, 2019. doi: 10.1016/j.neunet.2019.01.012. Garvesh Raskutti, Martin J. Wainwright, and Bin Yu. Early stopping and non-parametric regression: An optimal data-dependent stopping rule. Journal of Machine Learning Research, 15(11):335–366, 2014. URL https://jmlr.org/papers/v15/raskutti14a.html. Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. In Advances in Neural Information Processing Systems, volume 31, 2018. Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3723–3732, 2018. Junha Song, Kwanyong Park, InKyu Shin, Sanghyun Woo, Chaoning Zhang, and In So Kweon. Test-time adaptation in the dynamic world with compound domain knowledge management. IEEE Robotics and Automation Letters, 8(11):7583–7590, 2023. URL https://arxiv.org/abs/ 2212.08356. 12

Preprint

Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A. Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 9229–9248. PMLR, 2020. William Swartworth, Deanna Needell, Rachel A. Ward, Mark Kong, and Halyun Jeong. Nearly optimal bounds for cyclic forgetting. In Advances in Neural Information Processing Systems, volume 36, 2023. URL https://proceedings.neurips.cc/paper_files/paper/ 2023/hash/d72ae75abaa70a3b19c5d4f436c680d1-Abstract-Conference. html. Gido M. van de Ven, Tinne Tuytelaars, and Andreas S. Tolias. Three types of incremental learning. Nature Machine Intelligence, 4:1185–1197, 2022. doi: 10.1038/s42256-022-00568-3. Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully testtime adaptation by entropy minimization. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=uXl3bZLkr3c. Haoxiang Wang, Bo Li, and Han Zhao. Understanding gradual domain adaptation: Improved analysis, optimal path and beyond. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 22784–22801. PMLR, 2022a. URL https://proceedings.mlr.press/v162/wang22n.html. Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7201– 7211, 2022b. URL https://openaccess.thecvf.com/content/CVPR2022/html/ Wang_Continual_Test-Time_Domain_Adaptation_CVPR_2022_paper.html. Daphna Weinshall, Gad Cohen, and Dan Amir. Curriculum learning by transfer learning: Theory and experiments with deep networks. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 5238–5246. PMLR, 2018. Zonghuan Xu and Xingjun Ma. From order to distribution: A spectral characterization of forgetting in continual learning. arXiv preprint arXiv:2604.13460, 2026. Huaxiu Yao, Caroline Choi, Bochuan Cao, Yoonho Lee, Pang Wei Koh, and Chelsea Finn. Wild-time: A benchmark of in-the-wild distribution shift over time. In Advances in Neural Information Processing Systems Datasets and Benchmarks Track, 2022. Yuan Yao, Lorenzo Rosasco, and Andrea Caponnetto. On early stopping in gradient descent learning. Constructive Approximation, 26(2):289–315, 2007. doi: 10.1007/s00365-006-0663-2. Longhui Yuan, Binhui Xie, and Shuang Li. Robust test-time adaptation in dynamic scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15922–15932, 2023. URL https://openaccess.thecvf.com/content/CVPR2023/ html/Yuan_Robust_Test-Time_Adaptation_in_Dynamic_Scenarios_CVPR_ 2023_paper.html. Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 3987–3995. PMLR, 2017. Marvin Zhang, Sergey Levine, and Chelsea Finn. MEMO: Test time robustness via adaptation and augmentation. In Advances in Neural Information Processing Systems, volume 35, 2022. Yunbei Zhang, Akshay Mehra, Shuaicheng Niu, and Jihun Hamm. DPCore: Dynamic prompt coreset for continual test-time adaptation. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 75757–75778. PMLR, 2025. URL https://proceedings.mlr.press/v267/zhang25bf.html. Xuyang Zhao, Huiyuan Wang, Weiran Huang, and Wei Lin. A statistical theory of regularizationbased continual learning. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 61021–61039. PMLR, 2024. URL https://proceedings.mlr.press/v235/zhao24n.html. 13

Preprint

A

P ROOFS AND T ECHNICAL D ETAILS

A.1

PATHWISE LOSS AND COMMON - SOLUTION DISTANCE

Proof of Proposition 1. Write v = w−w⋆ = v ∥ +v ⊥ with v ∥ ∈ K and v ⊥ ∈ K⊥ . Since H(t)v ∥ = 0 for every t, 1 Lt (w) = (v ⊥ )⊤ H(t)v ⊥ . 2 The spectral upper bound in Equation (3) gives the right-hand inequality in Equation (7). For the lower bound, first observe that ker H̄ = K. Indeed, if u⊤ H̄u = 0, then the continuous nonnegative function t 7→ u⊤ H(t)u vanishes everywhere. Positive semidefiniteness then gives H(t)u = 0 for all t, so u ∈ K; the reverse inclusion is immediate. Hence H̄ is positive definite on K⊥ and cH > 0. Finally, Z 1 1 Rpath (w) ≥ Lt (w) dt = (v ⊥ )⊤ H̄v ⊥ ≥ cH ∥v ⊥ ∥2 . 2 0 Because ∥v ⊥ ∥ = dist(w, w⋆ + K), this proves Equation (7); the convergence equivalence follows from its two bounds. A.2

T HE DENSE - TASK LIMIT

For a total training budget τ ≥ 0, let xτ : [0, 1] → Rd solve ẋτ (t) = −τ H(t)xτ (t),

xτ (0) = v0 ,

(23)

and define its learning progress by Pτ (v0 ) := ∥v0 ∥2 − ∥xτ (1)∥2 .

(24)

The following lemma proves Theorem 1, including its uniform O(N −1 ) rates. Lemma 1 (Dense-task limit). Under the assumptions of Section 2, if N sN → τ ∈ [0, ∞), then ΦN,sN v0 −→ xτ (1),

PN,sN (v0 ) −→ Pτ (v0 ).

(25)

For every T < ∞, ΦN,τ /N v0 − xτ (1) = OT (N −1 ),

(26)

PN,τ /N (v0 ) − Pτ (v0 ) = OT (N −1 ).

(27)

sup 0≤τ ≤T

sup 0≤τ ≤T

Moreover, Z 1 Pτ (v0 ) = 2τ

xτ (t)⊤ H(t)xτ (t) dt.

(28)

0

Proof. Set h = N −1 and bN = N sN , and let xb denote the solution of Equation (23) with coefficient b. Fix T < ∞. Taylor expansion of the matrix exponential and the integral form of the ODE give, uniformly for b ∈ [0, T ] and j = 1, . . . , N , e−hbH(tj ) xb (tj−1 ) − xb (tj ) ≤ CT h2 . Indeed, both terms equal xb (tj−1 ) − hbH(tj )xb (tj−1 ) up to OT (h2 ); this uses the C 1 regularity of H. If ej = vj − xb (tj ), the exact task update is a contraction, so ∥ej ∥ ≤ ∥ej−1 ∥ + CT h2 . Since e0 = 0, summing over the N steps gives ∥ΦN,b/N v0 − xb (1)∥ ≤ CT h. This proves Equation (26). Both the discrete updates and the ODE flow are contractions, and hence PN,b/N (v0 ) − Pb (v0 ) ≤ 2∥v0 ∥ ∥ΦN,b/N v0 − xb (1)∥, 14

Preprint

which proves Equation (27). To compare two coefficients b, c ≥ 0, subtract their ODEs and apply variation of constants. Since the homogeneous flow is a contraction, Z 1 ∥xb (1) − xc (1)∥ ≤ |b − c| ∥H(t)∥ ∥xc (t)∥ dt ≤ L∥v0 ∥ |b − c|. 0

Taking b = bN and c = τ proves Equation (25) whenever bN → τ . Finally, d ∥xτ (t)∥2 = −2τ xτ (t)⊤ H(t)xτ (t). dt Integration from 0 to 1 gives Equation (28). A.3

E XACT EXPANSIONS OF THE LIMITING CURVE

The small-budget behavior is described in the ambient parameter coordinates. Define the operator on V (0) Z 1  AH := P (0) H(t) dt P (0) . (29) 0

V (0)

For large budgets, the solution directions move with t, so their contributions are compared in a common coordinate system. Let U (t) be Kato transport, U̇ (t) = [Ṗ (t), P (t)]U (t),

U (0) = I.

(30)

The generator is skew-symmetric, U (t) is orthogonal, and P (t)U (t) = U (t)P (0) (Kato, 1950; Avron et al., 2012). Write P0 = P (0) and Q0 = I − P0 . In this fixed frame,     0 0 0 −B(t)⊤ ⊤ ⊤ U (t) H(t)U (t) = , U (t) U̇ (t) = , (31) 0 H⊥ (t) B(t) 0 relative to V (0) ⊕ V (0)⊥ . Here H⊥ (t) ⪰ µI and B(t) = Q0 U (t)⊤ Ṗ (t)U (t)P0 . Define Z 1 GH := B(t)⊤ H⊥ (t)−1 B(t) dt 0 Z 1

=

U (t)⊤ Ṗ (t)H(t)† Ṗ (t)U (t) dt

0

.

(32) (33)

V (0)

Lemma 2 (Large-budget tracking expansion). Let yτ (t) = U (t)⊤ xτ (t) = (pτ (t), qτ (t)) in the decomposition V (0) ⊕ V (0)⊥ . As τ → ∞, sup ∥pτ (t) − v0 ∥ = O(τ −1 ),

(34)

t∈[0,1]

qτ (t) = −τ −1 H⊥ (t)−1 B(t)v0 + rτ (t), Z 1

∥rτ (t)∥2 dt = o(τ −2 ).

(35) (36)

0

Proof. Using Equations (23) and (31), the transported components satisfy ṗτ = B(t)⊤ qτ ,

q̇τ = −B(t)pτ − τ H⊥ (t)qτ ,

(pτ (0), qτ (0)) = (v0 , 0).

(37)

Let Eτ (t, r) denote the evolution operator generated by −τ H⊥ (t). The spectral gap gives ∥Eτ (t, r)∥ ≤ e−µτ (t−r) . Variation of constants in the second equation of Equation (37) gives Z t C qτ (t) = − Eτ (t, r)B(r)pτ (r) dr, ∥qτ ∥∞ ≤ ∥pτ ∥∞ . τ 0 The first equation of Equation (37) therefore implies ∥pτ ∥∞ ≤ ∥v0 ∥ + 15

C ∥pτ ∥∞ . τ

Preprint

For all sufficiently large τ , this bounds pτ uniformly. Returning to the two estimates above gives ∥qτ ∥∞ = O(τ −1 ) and then ∥pτ − v0 ∥∞ = O(τ −1 ), proving Equation (34). Set aτ (t) = H⊥ (t)−1 B(t)pτ (t) and reτ = qτ + τ −1 aτ . Equation (37) gives re˙ τ + τ H⊥ (t)e rτ = τ −1 ȧτ (t),

reτ (0) = τ −1 aτ (0).

Smoothness, Equation (34), and the first equation of Equation (37) bound ȧτ uniformly. A second use of variation of constants gives Z aτ (0) 1 t Eτ (t, r)ȧτ (r) dr. reτ (t) = Eτ (t, 0) + τ τ 0 The first term has squared L2 norm O(τ −3 ). The second is O(τ −2 ) uniformly in t and hence has squared L2 norm O(τ −4 ). Finally, Equation (34) gives 1 H⊥ (t)−1 B(t)(pτ (t) − v0 ) = O(τ −2 ) τ −1 uniformly. Taking rτ = reτ − τ −1 H⊥ B(pτ − v0 ) proves Equations (35) and (36). We next give an estimate that is uniform in both task spacing and per-task training time. Its fixedtime expansions prove Proposition 3, and its first bound is used to prove the large-N sN regime of Theorem 2. A.3.1

D ENSE - TASK EXPANSION FOR LONG PER - TASK TRAINING

For s ∈ (0, ∞), define on the normal block   1 sH⊥ (t) Ws (t) := coth , 2 2 and set

Z 1 Gs :=

W∞ (t) :=

1 I, 2

B(t)⊤ Ws (t)B(t) dt.

(38)

(39)

0

Together with GH from Equation (32), these are the operators used in the coefficients of Proposition 3. Lemma 3 (Finite-time expansion). Let δN := 1 − e−µsN . If N δN → ∞, then 

1 PN,sN (v0 ) = O N δN

 .

(40)

For every fixed s ∈ (0, ∞],   Gs + o(N −1 ), P (1)ΦN,s V (0) = U (1) I − N 2 PN,s (v0 ) = ⟨v0 , Gs v0 ⟩ + o(N −1 ). N If additionally sN → 0, then 2 PN,sN (v0 ) = ⟨v0 , GH v0 ⟩ + o((N sN )−1 ). N sN

(41) (42)

(43)

Moreover, for every fixed finite s and every nonzero v0 ∈ V (0) ∩ K⊥ , ⟨v0 , (Gs − G∞ )v0 ⟩ > 0.

(44)

Proof. Let h = N −1 , Uj = U (tj ), and zj = Uj⊤ vj = (pj , qj ). Put Bj = B(tj−1 ). Taylor expansion of Uj⊤ Uj−1 in the block frame Equation (31) gives h2 ⊤ B Bj pj−1 + aj , 2 j qj = Rj (qj−1 − hBj pj−1 ) + bj ,

pj = pj−1 + hBj⊤ qj−1 −

16

(45) (46)

Preprint

where Rj = e−sN H⊥ (tj ) satisfies ∥Rj ∥ ≤ 1 − δN , while ∥aj ∥ ≤ C(h3 + h2 ∥qj−1 ∥),

∥bj ∥ ≤ Ch2 .

These bounds are uniform in both j and sN . Indeed, the block generator S(t) := U (t)⊤ U̇ (t) in Equation (31) has vanishing diagonal blocks, while S(t)2 has tangent block −B(t)⊤ B(t). A second-order Taylor expansion of Uj⊤ Uj−1 therefore gives tangent–normal blocks hBj⊤ + O(h2 ) 2

and −hBj + O(h2 ), and tangent block I − h2 Bj⊤ Bj + O(h3 ). The C 3 bounds on U make these remainders uniform; multiplication of the normal block by Rj , with ∥Rj ∥ ≤ 1, preserves the stated estimates. The original updates are contractions, so pj and qj remain bounded. Iterating Equation (46) and using h/δN = (N δN )−1 → 0 yields   h max ∥qj ∥ = O . j≤N δN Substitution into Equation (45), followed by discrete Gronwall, gives   h max ∥pj − v0 ∥ = O . j≤N δN Because UN is orthogonal, PN,sN (v0 ) = ∥v0 ∥2 − ∥pN ∥2 − ∥qN ∥2 . The last two bounds, together with h/δN → 0, prove Equation (40). Now fix s ∈ (0, ∞), so that Rj = e−sH⊥ (tj ) is uniformly contractive. Define 1 (I + Rj )(I − Rj )−1 . (47) 2 Smoothness gives ∥Lj ∥ ≤ C and ∥Lj − Lj−1 ∥ ≤ Ch. For 0 ≤ j ≤ N − 1, set ej = qj − hLj+1 pj . The identity (I − Rj )Lj = −Rj Bj and Equations (45) and (46) imply Lj := −(I − Rj )−1 Rj Bj ,

Wj :=

∥ej ∥ ≤ (1 − δ)∥ej−1 ∥ + Ch2 ,

δ := 1 − e−µs > 0.

Since e0 = O(h), the geometric-series bound gives ∥ej ∥ ≤ Ch(1 − δ)j + Ch2 and hence PN h j=1 ∥ej−1 ∥ = O(h2 ) = o(h). Also qj = hLj pj−1 + ej−1 . Substitution into Equation (45) gives   N X pN = I − h hBj⊤ Wj Bj  v0 + o(h), qN = O(h). (48) j=1

On the normal block,   1 sH⊥ (tj ) Wj = coth . 2 2 The Riemann sum in Equation (48) therefore converges to Gs from Equation (39). Since P (1)vN = U (1)pN and ∥qN ∥2 = o(h), expansion of the endpoint norm proves Equations (41) and (42). The case s = ∞ follows by setting Rj = 0, equivalently Wj = I/2. For every positive eigenvalue λ of H⊥ (t), 1 e−sλ 1 coth(sλ/2) − = > 0. 2 2 1 − e−sλ Hence Gs − G∞ is positive semidefinite. Equality on v0 would require B(t)v0 = 0 for every t. To see why this forces v0 ∈ K, set z(t) = U (t)v0 ∈ V (t). The vector U (t)⊤ Ṗ (t)z(t) lies in V (0)⊥ and equals B(t)v0 there. The Kato equation therefore gives ż(t) = Ṗ (t)z(t) = 0. Hence z(t) = v0 ∈ V (t) for every t, so v0 ∈ K. This contradicts nonzero v0 ∈ K⊥ and proves Equation (44). For the refined expansion, suppose sN → 0. Then δN ≍ sN . Define Lj := −(I − Rj )−1 Rj Bj ,

Wj := 17

1 (I + Rj )(I − Rj )−1 . 2

Preprint

The spectral bounds and smoothness give ∥Lj ∥ ≤ Cs−1 N ,

∥Lj − Lj−1 ∥ ≤ Chs−1 N .

Set ej = qj − hLj+1 pj for 0 ≤ j ≤ N − 1. The identity (I − Rj )Lj = −Rj Bj , together with Equations (45) and (46), gives ∥ej ∥ ≤ (1 − δN )∥ej−1 ∥ + C

h3 h2 +C 2 . sN sN

Since e0 = O(h/sN ), a geometric-series estimate yields ∥ej ∥ ≤ C

h h2 (1 − δN )j + C 2 . sN sN

Since δN ≍ sN and the standing assumption gives h/sN = (N sN )−1 → 0, summing this bound makes the little-o step explicit:  2   N X h2 h h h2 +C 2 =O 2 =o h ∥ej−1 ∥ ≤ C . s δ s s s N N N N N j=1 Substituting qj−1 = hLj pj−1 + ej−1 into Equation (45) and summing gives     N X h . pN = I − h hBj⊤ Wj Bj  v0 + o s N j=1 Finally, uniformly on the positive spectrum, Wj =

1 −1 I + (I − e−sN H⊥ (tj ) )−1 e−sN H⊥ (tj ) = s−1 + O(sN ). N H⊥ (tj ) 2

The Riemann sum therefore gives pN = v0 − (N sN )−1 GH v0 + o((N sN )−1 ). Since ∥qN ∥2 = O((N sN )−2 ) = o((N sN )−1 ), expansion of ∥pN ∥2 + ∥qN ∥2 proves Equation (43). A.4

A N ANISOTROPIC DISTANCE CAN REVERSE THE COMPARISON

For a positive-definite matrix M , define the corresponding endpoint progress by M PN,s (v) := v ⊤ M v − (ΦN,s v)⊤ M (ΦN,s v).

The following closed four-dimensional path shows that the fixed-finite-versus-exact comparison need not survive this change of geometry. Let   0 −Ω⊤ P0 = diag(I2 , 02 ), Ω = 2πI2 , J= , Ω 0 and define R(t) = etJ , where

P (t) = R(t)P0 R(t)⊤ ,

H(t) = R(t)H0 R(t)⊤ ,

  5 . H0 = diag 0, 0, log 2, log 4

Take

  1 1 0.9 ⊤ v = √ (−2, 1, 0, 0) , M = diag(M0 , I2 ), M0 = . 0.9 1 5 The eigenvalues of M0 are 0.1 and 1.9, so M is positive definite. The path is closed because R(1) = I. Since J is off-diagonal relative to P0 , direct calculation gives [Ṗ (t), P (t)] = J; hence its Kato transport is U (t) = R(t). In the moving frame, B(t) = Ω and H⊥ = diag(log 2, log(5/4)). Therefore, at fitting time s = 1, D := G1 − G∞ = (2π)2 diag(1, 4). 18

(49)

Preprint

Because U (1) = I and M is block diagonal, the tangent–normal cross term vanishes. The normal endpoint component is O(N −1 ) by Equation (48), so its quadratic contribution is o(N −1 ). The endpoint expansion Equation (41) therefore gives
M M lim N PN,1 (v) − PN,∞ (v) = v ⊤ (M0 D + DM0 )v. N →∞

For Euclidean distance, M0 = I2 , and the right-hand side is 16 (2π)2 > 0. 5

(50)

For the positive-definite matrix above,  M0 diag(1, 4) + diag(1, 4)M0 =

2 4.5

 4.5 , 8

so the limit is instead

2 − (2π)2 < 0. (51) 5 Thus finite fitting beats exact fitting under Euclidean distance but loses under this anisotropic distance on the same task path and initial direction. A.5

P ROOF OF THE OPTIMAL TRAINING - TIME LAW

The following lemma proves Theorem 2 and records the coefficients of its two tails. Lemma 4 (Optimal-time law). Let v0 ∈ V (0) ∩ K⊥ be nonzero. Then: (i) The limiting progress has the two asymptotic expansions Pτ (v0 ) = 2τ ⟨v0 , AH v0 ⟩ + O(τ 2 ), 2 Pτ (v0 ) = ⟨v0 , GH v0 ⟩ + o(τ −1 ), τ

τ ↓ 0,

(52)

τ → ∞.

(53)

(ii) The two tail operators satisfy ker AH = ker GH = K

within V (0),

(54)

and Pτ (v0 ) > 0 for every τ ∈ (0, ∞). Hence T ⋆ (v0 ) := arg max Pτ (v0 )

(55)

τ >0

is a nonempty compact subset of (0, ∞). (iii) For every sequence sN , N sN → 0 =⇒ PN,sN (v0 ) → 0, N sN → τ ∈ (0, ∞) =⇒ PN,sN (v0 ) → Pτ (v0 ) > 0, N sN → ∞ =⇒ PN,sN (v0 ) → 0.

(56)

(iv) For all sufficiently large N , every optimizer in Equation (9) is finite. Every sequence of such optimizers satisfies dist(N s⋆N (v0 ), T ⋆ (v0 )) −→ 0. (57) Consequently s⋆N (v0 ) = Θ(N −1 ). If T ⋆ (v0 ) = {τ ⋆ (v0 )}, then s⋆N (v0 ) =

τ ⋆ (v0 ) + o(N −1 ). N

Proof. For small τ , the integral form of Equation (23) gives, uniformly on [0, 1], Z t xτ (t) = v0 − τ H(r)v0 dr + O(τ 2 ). 0

19

(58)

Preprint

Since xτ (t) = v0 + O(τ ) uniformly, substitution into Equation (28) proves Equation (52). For large τ , Lemma 2 and Equation (31) give Z 1 Pτ (v0 ) = 2τ qτ (t)⊤ H⊥ (t)qτ (t) dt, 0

Set g(t) = H⊥ (t)−1 B(t)v0 . By Equations (35) and (36), qτ = −τ −1 g + rτ with ∥rτ ∥L2 = o(τ −1 ). Because g and H⊥ are bounded, Z 1 Z 1 1 ⊤ g ⊤ H⊥ g dt + o(τ −2 ). qτ H⊥ qτ dt = 2 τ 0 0 The integral equals ⟨v0 , GH v0 ⟩, which proves Equation (53). For v ∈ V (0), Z 1 ⟨v, AH v⟩ =

∥H(t)1/2 v∥2 dt,

0

Z 1 ⟨v, GH v⟩ =

∥H⊥ (t)−1/2 B(t)v∥2 dt.

0

By continuity, the first integral vanishes exactly for v ∈ K. For the second, vanishing is equivalent to B(t)v = 0 for every t. Let z(t) = U (t)v ∈ V (t). Since Ṗ (t) maps V (t) into V (t)⊥ , the normal coordinate of Ṗ (t)z(t) is exactly B(t)v. The Kato equation gives ż(t) = Ṗ (t)z(t), so B(t)v = 0 makes z(t) constant and places v in every V (t). Conversely, if v ∈ K, then P (t)v = v and differentiation gives Ṗ (t)v = 0, so the constant path solves the Kato equation and B(t)v = 0. This proves Equation (54). If Pτ (v0 ) = 0 for a finite positive τ , the nonnegative integrand in Equation (28) vanishes for every t. Hence H(t)xτ (t) = 0 for every t, the ODE gives ẋτ (t) = 0, and v0 = xτ (t) belongs to every V (t). This contradicts nonzero v0 ∈ K⊥ , so Pτ (v0 ) > 0. The curve is continuous, is positive at each finite τ > 0, and tends to zero at both ends by Equations (52) and (53). Its maximum is therefore attained on a compact subinterval of (0, ∞), which proves the claim about Equation (55). The first two lines of Equation (56) follow from Lemma 1. If N sN → ∞, use s ≥ 0, 1 − e−µs ≥ cµ min{s, 1}, −µsN for a constant cµ > 0. If sN ≤ 1, this gives N (1 − e ) ≥ cµ N sN ; if sN > 1, it gives a lower bound of order N . Thus N (1 − e−µsN ) → ∞, and Lemma 3 gives the third line. It remains to localize the finite-resolution optimizers. Choose any τ̄ > 0. By Lemma 1, PN,τ̄ /N (v0 ) → Pτ̄ (v0 ) > 0, so the optimal value stays bounded away from zero. If no common positive lower bound existed for N s⋆N , one could choose task counts and optimizers with N s⋆N → 0; the first regime would make their progress tend to zero, a contradiction. If no common upper bound existed, one could similarly choose N s⋆N → ∞, including the possibility s⋆N = ∞; the third regime gives the same contradiction. Hence all optimizers are finite and N s⋆N lies in one compact subinterval of (0, ∞) for all sufficiently large N . Consider any convergent subsequence N s⋆N → τ . For each fixed γ > 0, optimality gives PN,s⋆N (v0 ) ≥ PN,γ/N (v0 ). Uniform convergence on the compact interval containing N s⋆N , together with Lemma 1, yields Pτ (v0 ) ≥ Pγ (v0 ) for every γ > 0. ⋆ Every accumulation point therefore belongs to T (v0 ). If Equation (57) failed, a subsequence at a fixed positive distance from T ⋆ (v0 ) would have a further convergent subsequence whose limit belongs to T ⋆ (v0 ), a contradiction. This proves Equation (57); the final two claims follow.

B

D ETAILS OF THE N UMERICAL E XPERIMENTS

B.1

ROTATING RANK - ONE PATH

The example in Section 4 uses H(t) = u(t)u(t)⊤ ,

u(t) = cos(πt/2), sin(πt/2) 20

⊤

,

v0 = (0, 1)⊤ .

Preprint

T Thus V (t) = u(t)⊥ rotates by π/2 and K = t V (t) = {0}. For finite N , each matrix-exponential update is evaluated as e−sH(tj ) v = v − (1 − e−s )u(tj )u(tj )⊤ v. Write xτ (t) = R(πt/2)yτ (t). Then   −τ π/2 ẏτ (t) = yτ (t), −π/2 0

yτ (0) = v0 .

Therefore the limiting endpoint is   −τ π/2 xτ (1) = R(π/2) exp v0 , −π/2 0 where R(θ) denotes the planar rotation matrix through angle θ, so the limiting curve can be evaluated without time discretization. Numerical maximization gives τ ⋆ = 2.19836 and Pτ ⋆ (v0 ) = 0.63113. For exact fitting, consecutive zero-loss directions differ by π/(2N ). Each projection multiplies the norm by cos(π/(2N )), so  π  π2 PN,∞ (v0 ) = 1 − cos2N = + O(N −2 ). 2N 4N Because the single positive eigenvalue of H(t) equals one, Equation (19) specializes to  π 2  π 2 coth(s/2) (s < ∞), N PN,∞ (v0 ) −→ . N PN,s (v0 ) −→ 2 2

(59)

Equation (59) makes the new comparison explicit without an additional panel: every finite s has coefficient (π/2)2 coth(s/2) > (π/2)2 , the coefficient obtained by training each task to convergence. B.2

H IGH - DIMENSIONAL RANDOM SMOOTH PATHS

We also test the scaling law on ten independently generated paths with ambient dimension d = 20 and rank r = 6. Each path has the form H(t) = Z(t)ΛZ(t)⊤ ,

Λ = diag(λ1 , . . . , λr ),

λi ∈ [0.5, 1.5],

where Z(t) ∈ Rd×r has orthonormal columns and is obtained by applying 80 smooth random Givens rotations to a random orthogonal frame. The rank and positive spectral gap are fixed along the path. The initial error is a random unit vector in ker H(0). We retain a draw only when the smallest eigenvalue of the average of the projectors onto im H(t) at 81 equally spaced locations exceeds 10−5 ; the smallest value among the ten retained paths is 1.48 × 10−4 . This check ensures that the sampled subspaces have trivial common kernel. For N ∈ {16, 32, 64, 128, 256}, we evaluate the exact matrix-exponential updates over a logarithmic grid of effective times τ = N s and optimize over τ by bounded scalar minimization. The median scaled optimizer decreases only from 2.82 at N = 16 to 2.72 at N = 256, while median optimal progress stays positive (0.290 to 0.278). By contrast, median exact-fitting progress decreases from 0.0834 to 0.00568; a log–log fit over N ≥ 32 has slope −0.981. This experiment stays within the theorem’s assumptions. It tests higher dimension, unequal positive eigenvalues, and noncommuting rotations of im H(t). All random seeds are fixed in the included experiment code, and the underlying numerical values are provided with the source package. B.3

G RADUALLY ROTATED MNIST

The experiment begins with the standard MNIST training and test splits. For each seed, we sample 30,000 training images and 3,000 held-out test images. Rotations use bilinear interpolation, zero fill, no canvas expansion, and the standard MNIST normalization. Labels are unchanged along the path. The classifier is an MLP with widths 784–256–128–10 and ReLU activations. The initial model is trained for 1,000 mini-batch SGD updates on the 0◦ task. Every adaptation run starts from this same seed-specific checkpoint. For a task count N , the sequential training angles are 21

Preprint

Optimal scale

0.30 0.25 0.20 0.15 0.10 0.05

3.0 2.8 2.6 2.4

0.00 10−2

10−1

100

101

102

102

Effective time τ = Ns

Overtraining decay Median exact-fit progress

3.2

Scaled optimum NsN⋆

Median progress N, s

Random smooth paths N = 16 N = 32 N = 64 N = 128 N = 256 N = 256 IQR

Task count N

10−1

exact fit N −1

10−2

102

Task count N

Figure 5: High-dimensional stress test over ten random smooth Hessian paths. Left: median learningprogress curves, with the interquartile range (IQR) shown for N = 256. Middle: median scaled optimizer N s⋆N and its IQR. Right: median exact-fitting progress and its IQR, compared with an N −1 reference line. 60j/N degrees for j = 1, . . . , N . We use SGD without momentum, learning rate 0.05, batch size 256, and k ∈ {0, 1, 2, 4, 8, 16, 32, 64, 128} updates per task. Within each (N, seed) pair, different values of k use deterministic prefixes of the same taskwise mini-batch streams. We use the ten consecutive seeds from 20260805 through 20260814. For evaluation, we rotate each held-out image to the 13 angles {0◦ , 5◦ , . . . , 60◦ }. Whole-path crossentropy averages over all 39,000 image–angle predictions and is therefore independent of the training discretization N . Lower values are better. Table 1: Rotated-MNIST optima over ten seeds. Parentheses in the second column give the number of seeds selecting each grid value. The interpolated columns use a local quadratic fit in log2 k around the minimum of each mean curve. The last column reports mean ± standard deviation at the grid value selected by mean whole-path cross-entropy. N seedwise best k (count) interpolated k ⋆ N k⋆ whole-path CE 4 8 (10) 8.95 35.8 0.502 ± 0.018 8 4 (7), 8 (3) 5.07 40.6 0.503 ± 0.020 16 2 (4), 4 (6) 3.07 49.2 0.499 ± 0.025 32 1 (4), 2 (6) 1.52 48.8 0.507 ± 0.029 The grid-selected best-k sequences are nonincreasing in N for all ten seeds. The interpolated total budgets remain between 35.8 and 49.2, and a log–log fit gives k ⋆ ∝ N −0.84 . Resampling the seeds and repeating the interpolation gives the 95% confidence interval [0.76, 1.03] for this exponent. This interval includes the theoretical value 1, but the comparison remains qualitative: there are four task counts and an integer update grid. The nonlinear model, cross-entropy loss, finite samples, stochastic updates, and absence of a proved shared zero-loss parameter also differ from the analytical setting. B.4

T EMPORAL Y EARBOOK

The Yearbook dataset contains aligned American high-school yearbook portraits indexed by calendar year (Ginosar et al., 2017). We use the official Wild-Time preprocessing (Yao et al., 2022). Its processed file contains 33,431 training images and 3,758 held-out images from 1930–2013, represented as 32 × 32 grayscale inputs. We inherit the benchmark’s binary labels only to study temporal optimization; the experiment is not intended for demographic inference. 22

Preprint

To keep the underlying path fixed as N changes, a task centered at calendar year t reweights example i, with year ai , according to   (ai − t)2 qt (i) ∝ exp − 1{|ai − t| ≤ 4σ}, σ = 4 years. 2σ 2 For N ∈ {4, 8, 16, 32}, training visits the centers tj = 1930 + 83j/N , j = 1, . . . , N . Evaluation uses 17 equally spaced centers from 1930 through 2013 for every N . At each center, held-out crossentropy is averaged with the corresponding normalized temporal weights; whole-path cross-entropy is the unweighted mean over the 17 centers. The classifier has four 3 × 3 convolutional blocks with channel widths 32–32–64–64. Each block uses GroupNorm, ReLU, and 2 × 2 max pooling, followed by global average pooling and a linear two-class head. The source model is trained for 1,500 updates on the source-time distribution. Adaptation uses plain mini-batch SGD with learning rate 0.03, batch size 256, no momentum or weight decay, and k ∈ {0, 1, 2, 4, 8, 16, 32, 64, 128} updates at every visited center. Larger k extends a fixed taskwise mini-batch prefix within each (N, seed) pair. We use five consecutive seeds from 20260805 through 20260809. Table 2: Yearbook optima over five seeds. Optima are obtained from local quadratic interpolation in log2 k around the minimum of each mean curve. Confidence intervals resample complete seeds and repeat the averaging and interpolation. N grid-selected k interpolated k ⋆ bootstrap 95% CI 4 64 79.90 [63.54, 128.00] 8 16 13.82 [7.53, 38.48] 16 4 4.23 [2.70, 13.10] 32 4 5.01 [3.49, 14.91] A log–log fit to the interpolated mean-curve optima gives k ⋆ ∝ N −1.37 . Repeating the fit in 10,000 seed-bootstrap replicates gives a median exponent of 1.30 and a 95% interval [0.79, 1.66]. The interval includes the theoretical value 1 but does not distinguish it sharply from nearby exponents. Some N = 4 replicates select the largest tested value, so the upper endpoint in Table 2 is grid-limited. We also perform a necessary endpoint check for approximate joint fit. A model trained on the equally weighted source and target distributions has mean held-out endpoint cross-entropy 0.084, compared with 0.057 for separately trained endpoint models; using the unrounded seed means, the mean gap is 0.028 across five seeds. This shows low simultaneous endpoint loss, not a shared zero-loss minimizer along the full path. The experiment further differs from the theorem through finite samples, supervised stochastic updates, cross-entropy loss, temporal smoothing, and the nonlinear classifier. It is therefore a natural-shift consistency check rather than a verification of the assumptions.

23

Related documents

• Scaling Synthetic-Image Pre-Training for Federated Fine-Tuning of Large Vision Models#366225
• Optimal Transport for Domain Adaptation through Gaussian Mixture Models#354118
• SUGFW+: An Uncertainty-guided Feature Weighting Framework for Cold Start Active Adaptation of SAM in Medical Image Segmentation#620524
• SUGFW+: An Uncertainty-guided Feature Weighting Framework for Cold Start Active Adaptation of SAM in Medical Image Segmentation#618845
• Normalized Low-Rank Adaptation#622519