Not All Forgetting Is Equal: Architecture-Dependent Retention Dynamics in Fine-Tuned Image Classifiers⋆,⋆⋆ Miit Dagaa (Researcher), Swarna Priya Ramua,∗ (Corresponding Author)
arXiv:2604.11508v1 [cs.LG] 13 Apr 2026
a School of Computer Science Engineering and Information Systems, Vellore Institute of Technology, Vellore, 632014, Tamil Nadu, India
ARTICLE INFO
ABSTRACT
Keywords: forgetting dynamics fine-tuning exponential decay vision transformers convolutional neural networks sample difficulty spaced repetition
Fine-tuning pretrained image classifiers is standard practice, yet which individual samples are forgotten during this process, and whether forgetting patterns are stable or architecture-dependent, remains unclear. Understanding these dynamics has direct implications for curriculum design, data pruning, and ensemble construction. We track per-sample correctness at every epoch during finetuning of ResNet-18 and DeiT-Small on a retinal OCT dataset (7 classes, 56:1 imbalance) and CUB200-2011 (200 bird species), fitting Ebbinghaus-style exponential decay curves to each sample’s retention trace. Five findings emerge. First, the two architectures forget fundamentally different samples: Jaccard overlap of the top-10% most-forgotten is 0.34 on OCTDL and 0.15 on CUB-200. Second, ViT forgetting is more structured (mean 𝑅2 = 0.74) than CNN forgetting (𝑅2 = 0.52). Third, per-sample forgetting is stochastic across random seeds (Spearman 𝜌 ≈ 0.01), challenging the assumption that sample difficulty is an intrinsic property. Fourth, class-level forgetting is consistent and semantically interpretable: visually similar species are forgotten most, distinctive ones least. Fifth, a sample’s loss after head warmup predicts its long-term decay constant (𝜌 = 0.30–0.50, 𝑝 < 10−45 ). These findings suggest that architectural diversity in ensembles provides complementary retention coverage, and that curriculum or pruning methods operating on per-sample difficulty scores may not generalize across runs. A spaced repetition sampler built on these decay constants does not outperform random sampling, confirming that static scheduling cannot exploit unstable per-sample signals.
1. Introduction In 1885, Hermann Ebbinghaus published the first quantitative study of human memory, showing that retention decays exponentially with time and that the rate of decay varies predictably across individuals and items (Ebbinghaus, 1885). Modern replication studies confirm that his exponential forgetting curve holds across diverse experimental conditions (Murre and Dros, 2015). This raises a natural question. Do deep neural networks, which also cycle between learning and forgetting individual training examples during optimisation, exhibit analogous dynamics? Answering this for fine-tuning specifically matters because pretrained representations constrain the loss landscape, creating a regime where most samples are learned quickly but a subset cycles between correct and incorrect states across epochs. The question is not purely academic. Fine-tuning pretrained image classifiers is the default approach in medical imaging (Kim, Cosa-Linan, Santhanam, Jannesari, Maros and Ganslandt, 2022), ecological monitoring, and other domains where labelled data are scarce. This cycling was first documented by Toneva, Sordoni, Combes, Trischler, Bengio and Gordon (2018) for training from scratch on CIFAR. Several practical methods assume that per-sample difficulty is a stable, intrinsic property: curriculum learning (Bengio, ∗ Corresponding author: Swarna Priya Ramu
[email protected] (M. Daga); [email protected] (S.P. Ramu) ORCID (s): 0009-0005-4629-458X (M. Daga); 0000-0002-8287-9690 (S.P.
Ramu) 1 This work was supported by the open access funding provided by Vellore Institute of Technology, Vellore.
:
Louradour, Collobert and Weston, 2009) schedules samples from easy to hard, self-paced learning (Kumar, Packer and Koller, 2010) lets the model select its own difficulty progression, dataset cartography (Swayamdipta, Schwartz, Lourie, Wang, Hajishirzi, Smith and Choi, 2020) maps samples into easy, ambiguous, and hard regions, and data pruning (Paul, Ganguli and Dziugaite, 2021) uses early-training signals to identify dispensable samples. All assume that a sample’s learning trajectory carries forward across training configurations. We test this assumption directly. We track per-sample correctness at every epoch during fine-tuning of ResNet18 (He, Zhang, Ren and Sun, 2016) and DeiT-Small (Touvron, Cord, Douze, Massa, Sablayrolles and Jégou, 2021) on two benchmarks (a heavily imbalanced retinal OCT dataset and the fine-grained CUB-200-2011 bird species dataset), fit Ebbinghaus-style exponential decay curves to each sample’s retention trace, and analyze the resulting decay constants across architectures, random seeds, and semantic class groupings. Five findings are reported as a result of this experiment as follows. 1. CNNs and ViTs forget different samples. The Jaccard overlap of the top-10% most-forgotten samples between ResNet-18 and DeiT-Small is 0.34 on OCTDL and 0.15 on CUB-200. 2. ViT forgetting is more structured. Exponential decay fits DeiT retention curves with mean 𝑅2 of 0.74 versus 0.52 for ResNet on CUB-200. 3. Per-sample forgetting is stochastic across seeds. Spearman 𝜌 ≈ 0.01 across all 12 seed-pair comparisons (all 𝑝 > 0.2), meaning that changing the Page 1 of 7
Architecture-Dependent Retention Dynamics in Fine-Tuned Image Classifiers
random seed completely reshuffles which samples are forgotten. 4. Class-level forgetting is consistent and semantically meaningful. Visually confusable bird species are forgotten most and visually distinctive species are forgotten least. 5. Early training loss predicts long-term forgetting. Phase 1 loss correlates with the fitted decay constant at 𝜌 = 0.30–0.50 (𝑝 < 10−45 ), providing a cheap diagnostic for flagging vulnerable samples. These findings have practical implications: architectural diversity in ensembles provides complementary retention coverage rather than redundant agreement, and curriculum or pruning methods operating on per-sample difficulty scores may not generalize across training runs. As a secondary contribution, we build a spaced repetition sampler using the fitted decay constants. It does not improve over random sampling, confirming that static scheduling cannot exploit unstable per-sample signals. The remainder of this paper is organized as follows. Section 2 reviews related work on forgetting, curriculum learning, and CNN-ViT training dynamics. Section 3 describes the retention tracking, decay fitting, analysis protocol, and spaced repetition sampler. Section 4 details the experimental setup. Section 5 presents results and discussion, and Section 6 concludes with limitations.
2. Related Work Analyzing how neural networks learn and forget individual samples has revealed insights into dataset redundancy and training stability. Toneva et al. (2018) first characterized these dynamics during training from scratch, demonstrating that many samples are unforgettable and can be safely pruned. Our work shifts to the fine-tuning regime and characterizes retention using explicit exponential decay fitting rather than event counts. More broadly, data-centric methods such as dataset cartography (Swayamdipta et al., 2020), influence-function pruning (Paul et al., 2021), and self-paced learning (Kumar et al., 2010) all operate on persample difficulty scores computed from a single training trajectory, implicitly assuming these scores are stable across configurations. Hacohen, Choshen and Weinshall (2020) go further, suggesting a universal classification order across architectures and initializations; whether this stability holds in the fine-tuning regime, where pretrained representations constrain the loss landscape, remains untested. In the domain of language acquisition, Settles and Meeder (2016) utilized Ebbinghaus-inspired models to estimate memory half-life for spaced repetition. Amiri, Miller and Savova (2017) successfully applied similar spacing principles to improve efficiency in vision tasks. We extend this psycholinguistic approach to computer vision fine-tuning; however, our results indicate that static scheduling based on initial decay constants does not improve accuracy because the per-sample signal is too unstable across runs. The divergence in information processing between architectures :
is also documented. Raghu, Unterthiner, Kornblith, Zhang and Dosovitskiy (2021) found that Vision Transformers maintain more uniform representations across layers than convolutional networks, while Maini, Garg, Lipton and Kolter (2022) utilized secondary training splits to identify hard examples. Our findings show that this architectural gap extends to specific instance-level retention, evidenced by the low Jaccard overlap between the forgetting sets of CNNs and ViTs. Existing literature focuses on identifying stable difficult samples or aggregate architectural differences. No prior study fits per-sample exponential decay curves during fine-tuning to analyze the cross-architecture stability of retention dynamics.
3. Methodology We track how individual training samples cycle between correctly and incorrectly classified states during fine-tuning, fit exponential decay models to the resulting retention traces, and use the fitted parameters to characterize forgetting patterns across architectures. Figure 1 illustrates the pipeline. A spaced repetition sampler built on these decay constants serves as a practical test of whether the observed patterns can be exploited.
3.1. Per-Sample Retention Tracking At the end of every training epoch, we evaluate the full training set under inference mode (no augmentation, no gradients) and record a binary correctness indicator per sample. This yields a retention matrix 𝐑 ∈ {0, 1}𝑁×𝐸 , where 𝑅𝑖,𝑒 = 1 if sample 𝑖 is correctly classified at epoch 𝑒 and 0 otherwise. A forgetting event for sample 𝑖 occurs at epoch 𝑒 when 𝑅𝑖,𝑒−1 = 1 and 𝑅𝑖,𝑒 = 0, the same transition tracked by Toneva et al. (2018), though we operate in the fine-tuning regime rather than training from scratch. We also record the first-learned epoch 𝑒∗𝑖 = min{𝑒 ∶ 𝑅𝑖,𝑒 = 1} and the retention rate 𝜌𝑖 , defined as the fraction of post-𝑒∗𝑖 epochs where the sample remains correctly classified.
3.2. Exponential Decay Fitting Drawing on the classical Ebbinghaus forgetting curve (Ebbinghaus, 1885; Murre and Dros, 2015), we model each sample’s retention probability as an exponential function of time since first learning as in equation 1. 𝑃 (retained at time 𝑡) = exp(−𝜆𝑖 ⋅ 𝑡),
(1)
where 𝑡 counts epochs since 𝑒∗𝑖 and 𝜆𝑖 ≥ 0 is the persample decay constant. A large 𝜆𝑖 indicates fast forgetting. We fit 𝜆𝑖 via nonlinear least squares (Virtanen, Gommers, Oliphant, Haberland, Reddy, Cournapeau, Burovski, Peterson, Weckesser, Bright et al., 2020) on the post-𝑒∗𝑖 binary retention vector, bounding 𝜆𝑖 ∈ [0, 10]. Three edge cases require special handling. Samples never forgotten after first learning (𝑅𝑖,𝑒 = 1 for all 𝑒 ≥ 𝑒∗𝑖 ) receive 𝜆𝑖 = 0. For the spaced repetition sampler, an epsilon floor 𝜀 = 0.01 replaces these zeros so that such samples are still occasionally revisited. Samples never correctly classified across all epochs cannot be fitted; their 𝜆𝑖 is set to the Page 2 of 7
Architecture-Dependent Retention Dynamics in Fine-Tuned Image Classifiers Training
Phase 1 Head Warmup
Phase 2 Full Fine-Tune + Eval Every Epoch
Retention Matrix R {0,1}^{N×E}
Data Artifacts
Analysis
Sampler
Exponential Decay Fit per sample
Spaced Repetition Sampler Cross-Arch Overlap (Jaccard)
Fit Quality R² Split CNN vs ViT
Cross-Seed Stability (Spearman )
Class-Level Patterns
Early Loss Correlation
Figure 1: Pipeline overview. Phase 1 trains only the classification head. During Phase 2 vanilla training, per-sample correctness is recorded at every epoch, producing a binary retention matrix. Exponential decay constants 𝜆𝑖 are fitted per sample and fed into five downstream analyses and the spaced repetition sampler. 99th percentile of the valid fitted values. In practice, neverlearned samples constitute 1–2% of OCTDL and ≤2% of CUB-200; never-forgotten samples (𝜆 = 0) account for 38– 87% of OCTDL samples and 13–99% of CUB-200 samples depending on class and backbone. We assess fit quality via per-sample 𝑅2 between the observed retention vector and the exponential prediction. Standard 𝑅2 that is applied to binary targets is a heuristic. Since residuals are not normally distributed; we retain it as an intuitive, bounded measure that facilitates cross-architecture comparison. Alternative functional forms (power law, stretched exponential) may better capture CNN forgetting, where mean 𝑅2 = 0.52 indicates that the single-parameter exponential explains only half the variance. However, the relative ranking (DeiT > ResNet) is consistent across all dataset-seed combinations.
3.3. Forgetting Analysis Protocol We study CNN forgetting at three different levels, by architecture, by seed, and by class, in five ways: 1. Cross-architecture overlap. We compare the top10% highest-𝜆 samples between ResNet-18 (He et al., 2016) and DeiT-Small (Touvron et al., 2021) using the Jaccard similarity index 𝐽 = |𝐴 ∩ 𝐵|∕|𝐴 ∪ 𝐵|, where 𝐴 and 𝐵 are the sample sets from each architecture. Low 𝐽 indicates architecture-dependent forgetting. 2. Fit quality split. We compare mean 𝑅2 of exponential fits between CNN and ViT to test whether one architecture’s forgetting is more structured. 3. Cross-seed stability. We compute Spearman rank correlation of per-sample 𝜆 values across seed pairs to test whether forgetting is an intrinsic sample property or a stochastic artefact of the training trajectory, directly probing the Ebbinghaus analogy, which assumes stable, intrinsic memory traces (Ebbinghaus, 1885). 4. Class-level patterns. We aggregate 𝜆 by class and examine whether mean class-level forgetting correlates with class size (Johnson and Khoshgoftaar, 2019) or inter-class visual similarity (Wah, Branson, Welinder, Perona and Belongie, 2011).
:
5. Early loss as predictor. We compute Spearman correlation between each sample’s cross-entropy loss at the end of head warmup (Phase 1, epoch 5) and its fitted 𝜆, testing whether initial difficulty predicts longterm forgetting rate.
3.4. Spaced Repetition Sampler As a secondary contribution, we translate the per-sample decay constants into a priority-based training sampler, inspired by spaced repetition systems used in human learning (Leitner, 1995; Settles and Meeder, 2016). Each sample 𝑖 receives an urgency score at epoch 𝑒 as in equation 2 ) ( last 𝑢(𝑒) (2) 𝑖 = 1 − exp −𝜆𝑖 ⋅ (𝑒 − 𝑒𝑖 ) , where 𝑒last is the most recent epoch in which sample 𝑖 𝑖 appeared in a training batch. The urgency is the estimated probability that the sample has been forgotten since last seen, directly instantiating the Ebbinghaus decay model. Samples with high 𝜆𝑖 and long gaps since last presentation receive the highest urgency. Sampling probabilities are obtained via a softmax with temperature 𝜏 as in equation 3. exp(𝑢(𝑒) 𝑖 ∕𝜏) , 𝑃 (𝑖) = ∑ (𝑒) exp(𝑢 ∕𝜏) 𝑗 𝑗
(3)
where 𝜏 = 1.0 throughout our experiments. The sampler replaces the standard random sampler during Phase 2 only; Phase 1 uses weighted random sampling (OCTDL) or standard shuffling (CUB-200). The decay constants are precomputed from an independent vanilla run and held fixed, isolating the scheduling effect from confounding online updates. We compare against three baselines: Random sampling (uniform, or inverse-frequency weighted for imbalanced data); Curriculum (Bengio et al., 2009), ranking samples by Phase 1 loss with easy samples oversampled early and weights shifting linearly toward uniform; and Anticurriculum, which reverses this ordering.
4. Experimental Setup We evaluate on two datasets that stress different forgetting drivers: class imbalance and inter-class visual similarity. Page 3 of 7
Architecture-Dependent Retention Dynamics in Fine-Tuned Image Classifiers
Imbalance 56:1 1.5:1
Forgetting driver Class size Visual similarity
OCTDL (Kulyabin, Zhdanov, Nikiforova, Stepichev, Kuznetsova, Ronkin, Borisov, Bogachev, Korotkich, Constable et al., 2024) contains 2,064 retinal OCT images across 7 pathology classes with extreme imbalance (56:1 ratio between the largest and smallest classes). We split 70/15/15 stratified train/val/test and use inverse-frequency WeightedRandomSampler to counter the imbalance. CUB-2002011 (Wah et al., 2011) contains 11,788 images of 200 bird species, nearly balanced at 41–60 images per class. We use the official train/test split and carve 15% of the training set for validation. Table 1 summarizes both. We use two architectures as architectural contrasts: ResNet-18 (He et al., 2016) (11.7M parameters, convolutional) and DeiT-Small (Touvron et al., 2021) (22.1M parameters, self-attention), both initialized from ImageNet1K weights via the timm library (Wightman, 2019). Training follows a two-phase protocol. Phase 1 freezes the backbone and trains only the classification head for 5 epochs (AdamW, lr=10−3 , weight decay 10−4 ). Phase 2 unfreezes all parameters and fine-tunes for up to 45 additional epochs (AdamW, lr=10−4 , cosine annealing to 10−6 , early stopping with patience 10 on validation loss). Training augmentation includes RandomResizedCrop(224), horizontal flip, and ColorJitter; validation uses Resize(256)/CenterCrop(224). Batch size is 32 for ResNet-18 on both datasets and for DeiTSmall on OCTDL, reduced to 16 for DeiT-Small on CUB200 due to T4 memory limits. Since retention is recorded under inference mode without augmentation or gradients, the tracking itself is batch-size-independent; the training dynamics differ slightly but this is absorbed into the per-seed variability we report. All experiments are run on a single NVIDIA Tesla T4 GPU. Each configuration is repeated with seeds {42, 99, 2026} controlling data splits, weight initialization, and augmentation randomness. We report mean ± population standard deviation (ddof=0). Classification performance is measured by top-1 accuracy, macro F1, and Cohen’s 𝜅.
Finding CNN vs ViT Jaccard (top-10%) Mean 𝑅2 : ResNet / DeiT Cross-seed 𝜆 Spearman Phase 1 loss vs 𝜆: ResNet Phase 1 loss vs 𝜆: DeiT Cohen’s 𝜅 (Random)
OCTDL 0.344 ± 0.013 0.62 / 0.71 ≈ 0.01 (n.s.) 𝜌 = 0.43 ± 0.02 𝜌 = 0.50 ± 0.03 0.86 ± 0.02 / 0.85 ± 0.02
octdl / resnet18
70 60
octdl / deit_small AMD DME ERM NO RAO RVO VID
100
50
80
40 30
60 40
20
20
10 0
CUB-200 0.151 ± 0.009 0.52 / 0.74 ≈ 0.01 (n.s.) 𝜌 = 0.30 ± 0.01 𝜌 = 0.41 ± 0.01 0.69 ± 0.00 / 0.77 ± 0.00
120
AMD DME ERM NO RAO RVO VID
Density
Classes 7 200
0
2
4 6 Decay constant
0
8
0
2
4 6 Decay constant
cub200 / resnet18
8
cub200 / deit_small
5
140 120
4
100 3
Density
Images 2,064 11,788
Density
Dataset OCTDL CUB-200
Table 2 Summary of forgetting analysis findings (mean ± std, ddof=0, 3 seeds).
Density
Table 1 Dataset summary.
2
80 60 40
1 0
20 0
2
4 6 Decay constant
8
0
0.00
0.05
0.10
0.15 0.20 Decay constant
0.25
0.30
Figure 2: Distribution of per-sample decay constants 𝜆 across all four dataset–backbone combinations (seed 42). DeiT-Small produces sharper bimodal distributions; ResNet-18 distributions are flatter with more intermediate values. fits. On both datasets, DeiT produces a sharper bimodal split: most samples cluster near 𝜆 = 0 (stably learned) or near the cap (persistently hard), with fewer intermediate values. ResNet distributions are flatter and noisier. The exponential model fits ViT retention curves substantially better than CNN curves. Mean 𝑅2 across seeds is 0.71 for DeiT on OCTDL versus 0.62 for ResNet, and 0.74 versus 0.52 on CUB-200 (Table 2, Figure 4). DeiT’s forgetting is more patterned and predictable. ResNet forgetting has a larger stochastic component that the exponential model does not capture.
5.2. Architecture-Dependent Forgetting
ResNet-18 and DeiT-Small forget fundamentally different samples. The Jaccard similarity of the top-10% most5. Results and Discussion forgotten samples is 0.344 ± 0.013 on OCTDL and 0.151 ± We first characterize the forgetting dynamics (Sections 5.1– 0.009 on CUB-200 (Table 2). The CUB-200 overlap is especially low, meaning the two architectures agree on fewer 5.5), then evaluate the spaced repetition sampler (Secthan one in six of their hardest samples. This gap persists tion 5.6). All numerical findings are summarized in Table 2 across thresholds: even at 𝑘 = 50%, Jaccard remains below and classification results appear in Table 3. 0.36 on CUB-200 (Figure 5). Per-sample Spearman correlation between 𝜆ResNet and 𝜆DeiT is modest (𝜌 ≈ 0.20–0.41, 5.1. Forgetting Curve Characterization 𝑝 < 10−34 ), confirming weak but statistically significant coThe fitted decay constants 𝜆 span a wide range across ranking at the sample level. At the class level, the agreement samples, from 0 (never forgotten) to the domain cap at 10 is stronger. Per-class mean 𝜆 rank correlations on CUB(never learned). Figure 2 shows that the 𝜆 distributions differ 200 range from 0.40 to 0.60 across seeds (all 𝑝 < 10−8 ), markedly between ResNet-18 and DeiT-Small, and Figure 3 indicating that the two architectures broadly agree on which shows representative retention traces with their exponential :
Page 4 of 7
Architecture-Dependent Retention Dynamics in Fine-Tuned Image Classifiers Table 3 Classification performance across sampling strategies (mean ± std, ddof=0, 3 seeds). Best per row in bold. No strategy significantly outperforms random sampling on accuracy, though some pairwise differences reach 𝑝 < 0.05 (see Section 5.6). Backbone ResNet-18 DeiT-S ResNet-18 DeiT-S
Accuracy Curr Anti .907±.012 .910±.007 .918±.017 .907±.000 .693±.002 .684±.003 .762±.010 .753±.014
Rand .916±.010 .909±.010 .693±.004 .775±.002
Never forgotten ( 0) =0.000
Moderate forgetting =0.142, R²=0.586
0.8
0.8
0.8
0.8
0.6
0.6
0.6
0.6
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
5
Epoch
10
15
0
5
Fast forgetting =9.513, R²=1.000
Epoch
10
Correct
1.0
Correct
1.0
0.4
15
0.4 0.2 0.0
0
5
Fast forgetting =0.528, R²=0.566
Epoch
10
15
0
1.0
0.8
0.8
0.6
0.6
0.6
0.6
Correct
1.0
0.8 Correct
1.0
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
5
Epoch
10
15
0
5
Epoch
10
15
Epoch
10
15
Never learned
0.8
0
5
Never learned
1.0
0.4
SR .827±.013 .830±.006 .691±.002 .762±.003
Moderate forgetting =0.274, R²=0.420
1.0
0
Correct
Macro F1 Curr Anti .826±.019 .799±.036 .847±.012 .822±.019 .692±.002 .684±.003 .762±.011 .754±.014
Rand .851±.010 .825±.040 .692±.004 .773±.003
1.0
Correct
Correct
Never forgotten ( 0) =0.000
SR .911±.008 .909±.015 .691±.002 .763±.002
Correct
Dataset OCTDL OCTDL CUB CUB
0.4 0.2 0.0
0
5
Epoch
10
15
0
5
Epoch
10
15
Figure 3: Example per-sample retention traces (dots) with fitted exponential decay curves (lines). Top row: samples with low 𝜆 (slow forgetting); bottom row: samples with high 𝜆 (fast forgetting). Binary correctness is recorded at every epoch; the fitted curve exp(−𝜆𝑡) captures the overall retention trend. octdl: ResNet-18 vs DeiT-Small overlap
1.0
octdl / deit_small (seed 42)
median=1.000
median=1.000 1000
0.0
Count
Count
800
400
200
0
0.4
0.2
0.0
0.2 R²
0.4
0.6
0.8
0
1.0
cub200 / resnet18 (seed 42)
0.4
0.2
0.0
0.2
R²
0.4
0.6
0.8
1.0
cub200 / deit_small (seed 42)
4000
median=1.000
median=1.000
2500 1500
Count
Count
0.2
10
15
20
25 30 35 Top-k% most forgotten
40
45
50
0.0
10
15
20
25 30 35 Top-k% most forgotten
40
45
50
Figure 5: Jaccard similarity between the top-𝑘% mostforgotten samples of ResNet-18 and DeiT-Small, for 𝑘 ∈ {10, 20, 30, 40, 50}. Overlap is low across all thresholds, particularly on CUB-200 (𝐽 < 0.36 even at 𝑘 = 50%).
3000
2000
2000 1500
1000
1000 500
500 1.00
0.75
0.50
0.25
0.00 R²
0.25
0.50
0.75
1.00
0
0.6
0.4
0.2
0.0
0.2 R²
0.4
0.6
0.8
1.0
Figure 4: Distribution of per-sample 𝑅2 for exponential decay fits (seed 42). DeiT-Small achieves higher 𝑅2 across both datasets, indicating more structured and predictable forgetting dynamics than ResNet-18.
:
0.4
3500
2500
0
0.6
600 400
200
0.4 0.2
800 600
0.6
seed 42 seed 99 seed 2026
0.8 Jaccard similarity
octdl / resnet18 (seed 42) 1000
Jaccard similarity
0.8
cub200: ResNet-18 vs DeiT-Small overlap
1.0
seed 42 seed 99 seed 2026
classes are hard even as they disagree on which individual samples within those classes are forgotten. On OCTDL (only 7 classes), the class-level correlation is unstable, ranging from 0.00 to 0.89.
5.3. Forgetting Stability Across Seeds Per-sample forgetting is not an intrinsic property of the data. Spearman rank correlations of 𝜆 values across seed pairs are near zero (𝜌 ≈ 0.01, 𝑝 > 0.2 in all 12 pairwise comparisons across all dataset-backbone combinations). Changing only the random seed (which controls data splitting, weight initialization, and augmentation order) completely reshuffles which individual samples are forgotten. This holds for both architectures and both datasets, Page 5 of 7
Architecture-Dependent Retention Dynamics in Fine-Tuned Image Classifiers octdl / resnet18 (seed 42)
% Never forgotten 42.3 37.9 56.0 51.1 66.2 68.4 86.7
0.12
0.05
0.10
0.04
0.08
Density
Mean 𝜆 1.167 0.436 0.245 0.222 0.209 0.123 0.024
0.06
0.03
0.06
0.02
0.01 0.00
Curriculum Anti-Curriculum Spaced Repetition
0.04
0.02
0
20
40
60 80 Times sampled in Phase 2
100
0.00
120
0
10
cub200 / resnet18 (seed 42) 0.05
20
30 40 Times sampled in Phase 2
50
60
cub200 / deit_small (seed 42) Curriculum Anti-Curriculum Spaced Repetition
Curriculum Anti-Curriculum Spaced Repetition
0.14 0.12
0.04
0.10 Density
Train size 71 109 53 103 861 232 15
Density
Class RVO ERM VID DME AMD NO RAO
octdl / deit_small (seed 42)
0.14
Curriculum Anti-Curriculum Spaced Repetition
0.07
Density
Table 4 Per-class forgetting on OCTDL (ResNet-18, mean across 3 seeds). Sorted by mean 𝜆 descending.
0.03
0.08 0.06
0.02
0.04 0.01
ruling out architecture-specific explanations. All 12 pairwise 𝜌 values fall within the 95% bootstrap confidence interval [−0.03, +0.05], consistent with a null correlation. We do not disentangle the relative contributions of data splitting, weight initialization, and augmentation order to this stochasticity; isolating each factor would require a factorial design beyond the scope of this letter but is a natural follow-up. This finding directly challenges the Ebbinghaus analogy at the sample level. Human forgetting curves are stable individual traits (Ebbinghaus, 1885; Murre and Dros, 2015); a word that is hard for a person to retain today will be hard again next week. DNN forgetting is dominated by training stochasticity. The practical implication is that any method treating per-sample difficulty as a fixed property, including curriculum learning (Bengio et al., 2009), self-paced learning (Kumar et al., 2010), and data pruning (Toneva et al., 2018), operates on a signal that does not replicate across runs.
0.00
0.02 0
20
40
60 80 100 Times sampled in Phase 2
120
140
0.00
0
10
20 30 40 Times sampled in Phase 2
50
Figure 6: Per-sample selection frequency across sampling strategies (all four dataset–backbone combinations, seed 42). Spaced repetition and anti-curriculum concentrate on subsets of the training data, while curriculum distributes more evenly. Despite these distinct patterns, none outperforms random sampling. with 𝑝 < 10−45 (Table 2). Samples that are hard after head warmup tend to remain hard throughout fine-tuning. DeiT shows consistently stronger correlations, aligning with its more structured forgetting dynamics (Section 5.1). This correlation offers a cheap diagnostic: Phase 1 loss can flag samples likely to be repeatedly forgotten, without requiring the full training run needed to compute 𝜆.
5.6. Sampling Strategy Comparison 5.4. Class-Level Forgetting Patterns Though sample-level forgetting is stochastic, class-level patterns are consistent and semantically interpretable. On CUB-200, the most-forgotten classes are visually similar species: California Gull, Tennessee Warbler, Common Tern, Shiny Cowbird, and Herring Gull (mean 𝜆 > 1.4 for ResNet18). The least-forgotten are visually distinctive: Geococcyx (roadrunner, 𝜆 ≈ 0), woodpeckers, and mergansers. This tracks intuition: classes that share plumage, body shape, and habitat with many neighbours are harder to retain. On OCTDL, forgetting correlates with class size rather than visual similarity. RVO, the smallest clinically meaningful class (71 training images), has the highest mean 𝜆 (Table 4). AMD, the largest class (861 training images), has low 𝜆 (0.21). The WeightedRandomSampler partially mitigates class imbalance but does not eliminate the forgetting gap. We note that RAO (15 training samples) is too small for reliable perclass 𝜆 estimation; its low mean 𝜆 likely reflects the sampler overweighting these few examples rather than intrinsic ease.
5.5. Early Loss as Forgetting Predictor A sample’s cross-entropy loss at the end of Phase 1 (head warmup, epoch 5) is moderately predictive of its long-term decay constant. Spearman correlations are 𝜌 = 0.43 ± 0.02 (OCTDL, ResNet), 0.50 ± 0.03 (OCTDL, DeiT), 0.30 ± 0.01 (CUB-200, ResNet), and 0.41 ± 0.01 (CUB-200, DeiT), all :
Table 3 presents classification results for all four sampling strategies across both datasets and both backbones. No strategy consistently outperforms random sampling. The spaced repetition sampler never significantly beats random on accuracy (the only significant comparison, CUB200/DeiT-Small 𝑝 = 0.006, favours random; Table 3). In fact, on CUB-200 with DeiT-Small, random sampling leads all alternatives by 1.2 percentage points in accuracy. Curriculum learning is competitive on OCTDL with DeiT (+0.86 points over random) but underperforms on CUB200 with DeiT (−1.26 points). Anti-curriculum produces a degenerate result on OCTDL with DeiT, yielding identical accuracy (0.9068) across all three seeds, suggesting that the hard-first schedule collapses into a fixed training pattern. Figure 6 confirms that the three non-random samplers produce meaningfully different sampling distributions, ruling out the possibility that the negative result stems from degenerate or near-uniform sampling. The negative result for the spaced repetition sampler follows logically from the cross-seed stochasticity finding (Section 5.3). The sampler’s decay constants come from a single vanilla run, but changing the seed reshuffles forgetting entirely. A static schedule built on unstable targets cannot improve over random sampling. Future work on adaptive sampling must contend with this instability; classlevel scheduling is a natural next step since that signal Page 6 of 7
Architecture-Dependent Retention Dynamics in Fine-Tuned Image Classifiers
is stable across seeds (Section 5.4). Online re-estimation of per-sample 𝜆 is another direction, though the near-zero cross-seed correlation suggests the signal may be too noisy to track reliably.
6. Conclusion The Ebbinghaus analogy holds at the class level but breaks at the sample level. Visually confusable classes are forgotten more and distinctive ones less, and this pattern replicates across seeds and architectures. But which specific samples get forgotten is random: change the seed and the forgetting set reshuffles entirely (𝜌 ≈ 0.01). Curriculum learning, data pruning, and dataset cartography all assume sample difficulty is stable. In fine-tuning, it is not. The two architectures also disagree on what counts as hard (Jaccard as low as 0.15), even though they broadly agree on which classes are difficult. ViT forgetting is more structured (𝑅2 = 0.74 vs. 0.52), with samples clustering into “stably learned” or “persistently hard” rather than spreading across intermediate values. One practical signal survives the stochasticity: Phase 1 loss predicts long-term decay (𝜌 = 0.30–0.50), so five epochs of warmup can flag vulnerable samples without a full training run. The spaced repetition sampler’s failure reinforces this picture: static scheduling from one run’s decay constants cannot help when those constants do not carry over to the next run. Class-level scheduling, oversampling high-forgetting classes rather than individual samples, is the clearest next step, since that is where the stable signal lives. Our analysis is bounded by the exponential decay model’s rough fit for CNNs (𝑅2 = 0.52), the use of static pre-computed 𝜆 values that may not reflect forgetting under altered sampling regimes, and the scope of two datasets with three seeds per configuration. Because each seed jointly controls the data split, weight initialization, and augmentation order, the cross-seed comparison involves partially overlapping sample sets rather than purely isolating training stochasticity.
Data Availability Statement The datasets analyzed in this study are publicly available. OCTDL is available at Kaggle. CUB-200-2011 is available at Kaggle. The code used in this study is available from the corresponding author upon reasonable request.
CRediT authorship contribution statement Miit Daga: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Writing - Original Draft. Swarna Priya Ramu: Supervision, Writing Original Draft, Writing - Review & Editing, Project administration.
:
References Amiri, H., Miller, T.A., Savova, G.K., 2017. Repeat before forgetting: Spaced repetition for efficient and effective training of neural networks, in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2401–2410. Bengio, Y., Louradour, J., Collobert, R., Weston, J., 2009. Curriculum learning, in: International Conference on Machine Learning (ICML), pp. 41–48. Ebbinghaus, H., 1885. Über das gedächtnis: untersuchungen zur experimentellen psychologie. Duncker & Humblot. Hacohen, G., Choshen, L., Weinshall, D., 2020. Let’s agree to agree: Neural networks share classification order on real datasets, in: International Conference on Machine Learning, PMLR. pp. 3950–3960. He, K., Zhang, X., Ren, S., Sun, J., 2016. Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778. Johnson, J.M., Khoshgoftaar, T.M., 2019. Survey on deep learning with class imbalance. Journal of big data 6, 27. Kim, H.E., Cosa-Linan, A., Santhanam, N., Jannesari, M., Maros, M.E., Ganslandt, T., 2022. Transfer learning for medical image classification: a literature review. BMC medical imaging 22, 69. Kulyabin, M., Zhdanov, A., Nikiforova, A., Stepichev, A., Kuznetsova, A., Ronkin, M., Borisov, V., Bogachev, A., Korotkich, S., Constable, P.A., et al., 2024. Octdl: Optical coherence tomography dataset for imagebased deep learning methods. Scientific data 11, 365. Kumar, M., Packer, B., Koller, D., 2010. Self-paced learning for latent variable models. Advances in neural information processing systems 23. Leitner, S., 1995. So lernt man lernen: Der Weg zum Erfolg. volume 16. Herder. Maini, P., Garg, S., Lipton, Z., Kolter, J.Z., 2022. Characterizing datapoints via second-split forgetting. Advances in Neural Information Processing Systems 35, 30044–30057. Murre, J.M., Dros, J., 2015. Replication and analysis of ebbinghaus’ forgetting curve. PloS one 10, e0120644. Paul, M., Ganguli, S., Dziugaite, G.K., 2021. Deep learning on a data diet: Finding important examples early in training. Advances in neural information processing systems 34, 20596–20607. Raghu, M., Unterthiner, T., Kornblith, S., Zhang, C., Dosovitskiy, A., 2021. Do vision transformers see like convolutional neural networks? Advances in neural information processing systems 34, 12116–12128. Settles, B., Meeder, B., 2016. A trainable spaced repetition model for language learning, in: Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers), pp. 1848–1858. Swayamdipta, S., Schwartz, R., Lourie, N., Wang, Y., Hajishirzi, H., Smith, N.A., Choi, Y., 2020. Dataset cartography: Mapping and diagnosing datasets with training dynamics, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9275–9293. Toneva, M., Sordoni, A., Combes, R.T.d., Trischler, A., Bengio, Y., Gordon, G.J., 2018. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv:1812.05159 . Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., Jégou, H., 2021. Training data-efficient image transformers & distillation through attention, in: International conference on machine learning, PMLR. pp. 10347–10357. Virtanen, P., Gommers, R., Oliphant, T.E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., et al., 2020. Scipy 1.0: fundamental algorithms for scientific computing in python. Nature methods 17, 261–272. Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S., 2011. The Caltech-UCSD Birds-200-2011 Dataset. Technical Report CNS-TR2011-001. California Institute of Technology. Wightman, R., 2019. PyTorch Image Models. https://github.com/ rwightman/pytorch-image-models. Accessed: 2026-04-10.
Page 7 of 7