Adaptive Data Dropout: Towards Self-Regulated Learning in Deep Neural Networks

Adaptive Data Dropout: Towards Self-Regulated Learning in Deep Neural Networks Amar Gahir∗ Varshil Patel∗ Shreyank N Gowda School of Computer Science, University of Nottingham, Nottingham, United Kingdom

arXiv:2604.12945v1 [cs.LG] 14 Apr 2026

[email protected]

Abstract Deep neural networks are typically trained by uniformly sampling large datasets across epochs, despite evidence that not all samples contribute equally throughout learning. Recent work shows that progressively reducing the amount of training data can improve efficiency and generalization, but existing methods rely on fixed schedules that do not adapt during training. In this work, we propose Adaptive Data Dropout, a simple framework that dynamically adjusts the subset of training data based on performance feedback. Inspired by self-regulated learning, our approach treats data selection as an adaptive process, increasing or decreasing data exposure in response to changes in training accuracy. We introduce a lightweight stochastic update mechanism that modulates the dropout schedule online, allowing the model to balance exploration and consolidation over time. Experiments on standard image classification benchmarks show that our method reduces effective training steps while maintaining competitive accuracy compared to static data dropout strategies. These results highlight adaptive data selection as a promising direction for efficient and robust training. Code will be released.

1. Introduction 1

Advances in deep learning have consistently come at the cost of substantial computational resources. Each year, the computational cost of the most notable models multiplies by over 4 times [7]. This trend raises significant challenges for real-world deployment, scalability, and environmental sustainability [9]. While a large body of work has focused on improving model efficiency through architectural design, such as MobileNet [15, 28], EfficientNet [33], and EfficientFormer [22], as well as techniques like pruning [11], quantization [17], and knowledge distillation [14], relatively less attention has been paid to improving the efficiency of the 1∗ Equal contribution. Ready Internship.

Work done during the DeepMind Research

training process itself. Standard training pipelines treat all samples equally across all epochs, implicitly assuming that every data point contributes uniformly throughout learning. However, prior work has shown that this assumption is flawed, as some samples provide diminishing returns once they are learned, while others remain informative for longer [18]. Inspired by this observation, recent work introduced Progressive Data Dropout (PDD) [30], a simple yet effective approach that progressively reduces the amount of training data across epochs. By focusing on informative samples early and revisiting the full dataset later, PDD achieves substantial reductions in effective training cost while maintaining or improving generalization. Despite its effectiveness, PDD relies on pre-defined or pre-computed data schedules that remain fixed throughout training. These schedules, whether difficulty-based or stochastic, do not adapt to the evolving state of the model. As a result, they may either drop data too aggressively, leading to underfitting, or retain redundant samples longer than necessary, reducing potential efficiency gains. More fundamentally, fixed schedules fail to capture a key aspect of human learning: the ability to dynamically adjust study strategies based on ongoing performance. Human learning is inherently adaptive. Learners monitor their progress and regulate effort accordingly, allocating more attention when performance deteriorates and reducing effort when concepts are well understood. This process, often described as self-regulated learning [41], is closely linked to the notion of desirable difficulties [3], where learning is most effective when tasks are neither too easy nor too difficult. The optimal level of challenge is not static but evolves as the learner improves, requiring continuous adjustment of learning conditions [10]. These insights suggest that efficient learning systems should not rely on fixed curricula, but instead adapt the level of difficulty and data exposure over time. Figure 1 illustrates this visually. In this work, we propose Adaptive Data Dropout, a simple and general framework that extends PDD by introducing feedback-driven data scheduling. Rather than follow-

edge distillation [14] aim to compress models while preserving performance. Similarly, architectures such as MobileNet [15], EfficientNet [33], and EfficientFormer [22] are designed to minimize parameters and FLOPs. In contrast, our work targets the training process itself, aiming to reduce the amount of data used during optimization while maintaining or improving performance, without modifying the underlying model.

Figure 1. Adaptive learning balances challenge by dynamically adjusting effort based on performance feedback.

ing a fixed dropout schedule, our method dynamically adjusts the subset of training data at each epoch based on observed learning progress. Concretely, we use changes in training performance as a signal to modulate the degree of data dropout, allowing the model to automatically increase or decrease data exposure as needed. This results in a training process that maintains an appropriate level of difficulty throughout learning, balancing efficiency and generalization without requiring pre-defined schedules. Our approach retains the simplicity and compatibility of PDD, requiring no changes to model architectures or optimization procedures, while introducing adaptivity into the training loop. We demonstrate that adaptive data dropout consistently reduces effective training cost while maintaining competitive performance across architectures. These results highlight the importance of feedback-driven data selection and suggest a promising direction toward more efficient and human-like learning dynamics in deep neural networks.

2. Related Work A broad range of research efforts aim to improve training efficiency and generalization in deep learning. These include advances in model design, data selection, and training strategies. We situate our work within these directions, with particular focus on methods that adapt data usage during training. Model Efficiency and Training Cost. A substantial body of work has focused on reducing the computational footprint of neural networks, primarily at inference time. Techniques such as pruning [11], quantization [17], and knowl-

Curriculum Learning and Adaptive Sampling. Curriculum learning [2] and hard example mining [31] organize training samples based on difficulty, typically presenting easier or more informative examples at specific stages of learning. Extensions of this idea explore adaptive sampling strategies that prioritize samples based on loss, gradient magnitude, or uncertainty [18]. While these approaches adjust the importance or order of samples, they generally assume a predefined or heuristic schedule. In contrast, our method treats data selection as a feedback-driven process, dynamically adjusting data exposure based on observed learning progress rather than following a fixed curriculum. Data Subset Selection and Pruning. Another line of work investigates training on reduced subsets of data to improve efficiency. Methods such as Data Diet [34] identify and remove less informative samples based on forgetting statistics or gradient-based metrics. Dataset pruning approaches [38] aim to construct compact subsets that preserve the performance of the full dataset, often requiring additional optimization or proxy models [4]. Influence-based methods [8, 20] estimate the contribution of individual samples, while coreset selection techniques [23, 24] identify representative subsets through repeated evaluation. These approaches typically involve pre-processing stages or additional computational overhead. In contrast, our method operates entirely within the training loop and adapts data usage online, without requiring auxiliary models or explicit sample scoring. Active Learning. Active learning [29] seeks to reduce labeling and training costs by selecting the most informative samples from an unlabeled pool. Common strategies include uncertainty sampling [25], margin-based selection [1], and query-by-committee [19]. These methods rely on iterative querying and assume access to large unlabeled datasets. Our setting is fundamentally different, as we operate in a fully supervised regime and focus on dynamically adjusting the use of already-labeled data during training, rather than acquiring new data. Data Dropout and Progressive Training. Dropout [32] and its variants, such as DropConnect [35] and stochastic

depth [16], improve generalization by introducing stochasticity at the level of model activations. More recent work has explored analogous ideas at the data level. Approaches such as data dropout [36] and Dynamic Training Data Dropout [40] selectively remove samples during training based on heuristics or noise estimation. Progressive Data Dropout (PDD) [30] extends this idea by gradually reducing the training set across epochs using predefined schedules, achieving significant efficiency gains. Our work builds directly on this line of research, but departs from prior methods in a key aspect: rather than relying on fixed or precomputed schedules, we introduce an adaptive mechanism that adjusts data dropout in response to training dynamics. This enables the model to regulate its own data exposure over time, maintaining an appropriate level of difficulty throughout training. Learning with Adaptive Difficulty. Our work is also related to cognitive and educational theories that emphasize adaptive learning. Self-regulated learning [41] describes how learners monitor performance and adjust their strategies accordingly. The concept of desirable difficulties [3] suggests that learning is most effective when tasks are neither too easy nor too difficult, while the challenge point framework [10] formalizes the idea that optimal difficulty evolves with expertise. These perspectives highlight the importance of dynamically adjusting learning conditions, a principle that we operationalize in the context of deep neural network training through adaptive data dropout.

3. Method We introduce Adaptive Data Dropout, a simple yet effective framework that extends Progressive Data Dropout (PDD) [30] by dynamically adjusting data usage during training. Unlike prior approaches that rely on fixed or precomputed schedules, our method adapts the amount of training data at each epoch based on performance feedback. This enables the model to regulate its own learning process, maintaining an appropriate level of difficulty over time.

3.1. Overview Let D0 = {(xi , yi )}N i=1 denote the full training dataset. At each epoch t, we construct a subset Dt ⊆ D0 used for backpropagation. As in PDD, the final epoch uses the full dataset, DT = D0 , serving as a revision phase. In contrast to PDD, where |Dt | is determined by a fixed schedule, we define data usage as an adaptive process. At each epoch, the model observes a feedback signal based on training performance and updates the subset size accordingly. This allows the training procedure to dynamically increase or decrease data exposure in response to learning progress.

Figure 2. Adaptive data dropout dynamically adjusts training data based on changes in model performance, increasing or decreasing data exposure to maintain optimal learning difficulty.

We define the feedback signal as: ∆t = At − At−1 , where At denotes the training accuracy at epoch t. This signal reflects whether the model is improving or deteriorating, and is used to guide data selection. An overview can be seen in Figure 3.

3.2. Variant 1: Adaptive Decay (Adaptive-α) Our first variant builds on scalar-based dropout by introducing an adaptive decay parameter. Instead of using a fixed decay rate, we dynamically update αt based on training feedback. At each epoch, the subset size is given by: |Dt | = f (t; αt ) · N, where f is a monotonically decreasing function (e.g., exponential, logarithmic, or inverse linear), and αt controls the rate of decay. We update αt using a stochastic acceptance mechanism: • If ∆t > 0, we accept a more aggressive decay (increase αt ), reducing the dataset size. • If ∆t ≤ 0, we probabilistically accept or reject the update. On rejection, we decrease αt , increasing data exposure in subsequent epochs. This process allows the model to explore different dropout rates while avoiding overly aggressive data reduction. Intuitively, when learning is progressing well, the model reduces redundant data, whereas when performance stagnates, it revisits more data.

3.3. Variant 2: Adaptive Keep-Fraction (AdaptiveT) Our second variant directly models the fraction of data retained at each epoch. Let Tt ∈ (0, 1] denote the proportion

of the dataset used at epoch t, such that: |Dt | = Tt · N. We define a target decay trajectory Tbase (t) using a fixed schedule (e.g., logarithmic), but allow the actual trajectory Tt to deviate from this schedule based on feedback. At each epoch: • If ∆t > δ, where δ is a small threshold, we follow the decay schedule: Tt+1 = Tbase (t + 1). • Otherwise, we apply a stochastic acceptance rule. If the update is rejected, we reheat the system by increasing Tt , thereby increasing the dataset size: Tt+1 = min(T0 , γ · Tt ), where γ > 1 is a reheating factor. This variant can be interpreted as tracking a desired curriculum while allowing deviations when learning stagnates. The reheating mechanism enables recovery from overly aggressive data reduction by temporarily increasing data exposure.

3.4. Subset Sampling At each epoch, we construct Dt by randomly sampling |Dt | examples from D0 without replacement. Sampling is repeated independently across epochs, ensuring diversity while maintaining computational efficiency. A visual depiction of the two variants can be seen in Figure 3.

Figure 3. Adaptive data dropout dynamically selects training samples based on performance feedback, updating either the revision parameter (α) or temperature (T ) through an accept–reject mechanism to balance exploration and efficient learning.

3.5. Effective Epochs We adopt the notion of Effective Epochs (EE) from [30] to quantify training efficiency. While a forward pass is performed on all samples, backpropagation is applied only to the selected subset. The total number of effective epochs is defined as: PT |Dt | . Effective Epochs = t=1 N This metric reflects the equivalent number of full passes over the dataset and provides a hardware-independent measure of computational cost.

3.6. Discussion Adaptive Data Dropout generalizes PDD by replacing fixed schedules with feedback-driven updates. From a cognitive perspective, this mirrors self-regulated learning, where learners adjust effort based on performance. By dynamically controlling data exposure, our method maintains an appropriate level of difficulty throughout training, balancing efficiency and generalization without requiring explicit difficulty estimation or precomputed schedules.

3.7. Theoretical Insight Adaptive Data Dropout can be interpreted as a feedbackdriven control process that dynamically regulates the effective training distribution. At each epoch, the model observes a performance signal ∆t and adjusts the subset size |Dt | accordingly. This induces a non-stationary training distribution that depends on the state of the model. From an optimization perspective, this can be viewed as modulating the stochastic gradient noise. Reducing the dataset size increases gradient variance, encouraging exploration of the loss landscape, while increasing data exposure reduces variance and promotes stable convergence. The adaptive mechanism therefore alternates between exploration and consolidation phases, similar to simulated annealing, but driven by performance feedback rather than a fixed schedule. The acceptance rule further introduces a stochastic decision process that allows occasional acceptance of nonimproving updates. This prevents the model from becoming trapped in suboptimal regimes caused by overly ag-

4. Experimental Analysis 4.1. Datasets

Figure 4. Conceptual illustration of adaptive data exposure as a balance between exploration and exploitation. Using more data leads to lower gradient variance and stable convergence, while using less data increases variance and promotes exploration. Adaptive Data Dropout dynamically transitions between these regimes based on training feedback, enabling efficient and robust optimization.

gressive data reduction. In the Adaptive-T variant, the reheating mechanism explicitly increases data exposure when learning stagnates, enabling recovery and re-stabilization of training. More broadly, the method can be interpreted as maintaining an implicit balance between bias and variance during optimization. When performance improves, the model tolerates higher bias (less data) to gain efficiency, whereas when performance deteriorates, it reduces bias by incorporating more data. This dynamic adjustment aligns with the principle of maintaining an appropriate level of learning difficulty, which has been shown to be critical in both human and machine learning systems. While a full theoretical characterization remains an open problem, these perspectives suggest that adaptive data selection acts as a form of implicit regularization, shaping both the optimization trajectory and the generalization behavior of the model. Figure 4 illustrates this behavior conceptually. Unlike fixed schedules, which remain in either high- or lowvariance regimes, Adaptive Data Dropout transitions between them over time. This behavior is analogous to annealing-based optimization, but driven directly by training feedback rather than a predefined schedule, enabling a more responsive and efficient learning process.

We evaluate our method on three standard image classification benchmarks: CIFAR-10 [21], CIFAR-100 [21], and ImageNet [5]. The CIFAR-10 and CIFAR-100 datasets each contain 60,000 color images of size 32 × 32, with 50,000 training images and 10,000 test images. CIFAR-10 consists of 10 object categories, while CIFAR-100 contains 100 finer-grained classes. These datasets are widely used for evaluating compact and mid-scale architectures. We also evaluate on ImageNet (ILSVRC2012), which contains over 1.28 million training images and 50,000 validation images spanning 1,000 categories. Compared to CIFAR, ImageNet provides substantially greater scale and diversity, making it a strong benchmark for assessing both generalization and training efficiency.

4.2. Implementation Details Adaptive Data Dropout is implemented with minimal changes to standard training pipelines. We do not modify the underlying model architectures, loss functions, or optimizers, and only alter how subsets of training data are selected at each epoch. To preserve stability, we keep batch normalization behavior, learning-rate schedules, and augmentation policies consistent with the corresponding baseline training settings. For all variants with data reduction, the data loader is refreshed at each epoch to reflect the current sampled subset. We evaluate on popular image classification architectures including MobileNetV2 [28], EfficientNet [33], EfficientFormer [22], ResNet [12], and ViT-B [6]. We follow the official implementation details for ImageNet experiments, and maintain the same optimization settings used in prior work for fair comparison. For CIFAR experiments, we use AdamW with a StepLR scheduler and standard augmentation settings. Our method is architecture-agnostic and integrates directly into existing supervised training pipelines. We consider two adaptive variants in our experiments: Adaptive-α, which dynamically adjusts the decay parameter of the subset schedule, and Adaptive-T , which directly controls the keep-fraction of the training data throughout training. In all cases, the final epoch uses the full dataset as a revision phase, consistent with the training principle introduced in Progressive Data Dropout (PDD) [30].

4.3. Supervised Image Classification In Table 1, we compare the performance of EfficientNetB0 [33], MobileNet-V2 [28], ResNet-50 [12], EfficientFormer-L1 [22], and ViT-B-MAE [13] across CIFAR-10, CIFAR-100, ImageNet, and ImageNetpretrained transfer settings. We report results for each model under the baseline training regime, using DBPD [30]

efficiency. This comparison is designed to assess whether adaptive control of data exposure provides benefits beyond fixed subset schedules, heuristic importance estimation, or early termination strategies. The results are reported on CIFAR-100 fine-tuning from ImageNet under a common training setup.

4.5. Ablation Study

Figure 5. Pareto trade-off between Top-1 accuracy and effective epochs on ImageNet. Each point represents a model, with colors indicating architectures and markers denoting methods. Adaptive Data Dropout consistently achieves a better accuracy–efficiency trade-off, achieving competitive accuracy while using substantially fewer effective training epochs.

as a representative static data dropout baseline as it provides a strong and interpretable baseline, as well as our two adaptive variants, Adaptive-α and Adaptive-T . Our goal is to examine whether feedback-driven data scheduling provides a stronger trade-off between accuracy and computational cost than fixed schedules. Across datasets and architectures, the static data dropout methods already offer substantial savings over conventional training. We use these as strong baselines and evaluate whether adaptation can further improve robustness and efficiency by dynamically modulating data exposure in response to learning progress. To further illustrate this trade-off, we visualize the relationship between accuracy and effective training cost in Figure 5. The Pareto plot highlights that Adaptive Data Dropout consistently achieves competitive or improved accuracy while requiring substantially fewer effective epochs compared to both standard training and fixed data dropout schedules. Notably, the adaptive variants occupy more favorable regions of the Pareto frontier, demonstrating that feedback-driven data selection enables a more efficient use of training data without sacrificing performance. These results reinforce the benefit of dynamically adjusting data exposure in response to training progress.

4.4. Comparison to State-of-the-Art We compare Adaptive Data Dropout against strong baselines for efficient training and data selection, including DataDiet (ELN and Forget) [26], IES [39], Early Stopping, and InfoBatch [27]. We follow the official implementations for all baselines and use effective epochs to evaluate

To better understand the behavior of Adaptive Data Dropout, we conduct an ablation study along several dimensions, including matched-step baseline comparisons, sensitivity to adaptive control parameters, shortened training schedules, and the effect of revision. Unless otherwise stated, all ablations are performed using EfficientNet-B0 trained from scratch on CIFAR-100. Are the gains simply due to fewer effective epochs? To ensure that any improvement is not merely a consequence of reduced optimization steps, we train a baseline EfficientNet-B0 for the same number of effective epochs as each dropout variant. This comparison isolates whether performance gains arise from the schedule itself rather than from training less. We report these results in Table 3. Can we shorten training and still retain performance? To assess performance under constrained training budgets, we repeat all dropout variants under shorter training schedules and compare them to baselines trained for the same number of epochs. This experiment evaluates whether adaptive data exposure remains useful when optimization time is limited.

4.6. Dynamics of Adaptive Reheating To better understand the behavior of Adaptive Data Dropout, we analyze how the number of training samples evolves over epochs. In particular, we examine whether the proposed feedback-driven mechanism exhibits the intended “reheating” behavior, where data exposure is increased when learning stagnates or deteriorates. Figure 6 visualizes the number of samples used per epoch for two representative models. We observe a consistent pattern across both variants: the training process initially reduces the dataset size as performance improves, reflecting increasing confidence and redundancy in the data. However, at several points during training, sharp increases in the number of samples occur. These spikes correspond to periods where the improvement in training accuracy slows down or reverses, triggering the adaptive mechanism to increase data exposure. This behavior closely mirrors the notion of self-regulated learning, where learners allocate more effort when performance deteriorates and reduce effort when learning is progressing well. Importantly, these reheating events are not pre-scheduled but emerge naturally from the feedback signal, demonstrating that the model dynamically adjusts its learning difficulty over time.

Table 1. Accuracy (%) and effective epochs reported as accuracy/effective epochs. All methods besides ViT-MAE are trained from scratch unless otherwise mentioned. ViT-MAE uses the publicly available checkpoint [13]. All results have been conducted following official implementations on timm [37]. Best results in blue and second best in green. The last column shows the gains in accuracy and drop in effective epochs of Adaptive Data Dropout relative to the baseline.

Baseline

DBPD

Adaptive-α

Adaptive-T

Acc. gain (%)

EE saved

CIFAR10 EfficientNet-B0 MobileNet-V2 ResNet-50 EfficientFormer-L1

88.21 / 30 85.49 / 30 84.85 / 30 80.32 / 30

87.82 / 5.86 85.39 / 6.42 85.61 / 7.08 81.17 / 5.82

85.69 / 7.45 83.08 / 8.76 83.82 / 8.48 80.75 / 7.21

85.28 / 6.36 80.79 / 6.14 82.64 / 6.24 80.67 / 6.90

-2.52 -2.41 -2.21 0.43

4.72 × 4.89 × 4.81 × 5.15 ×

CIFAR100 EfficientNet-B0 MobileNet-V2 ResNet-50 EfficientFormer-L1

66.32 / 200 61.25 / 200 59.81 / 200 55.91 / 200

67.15 / 24.8 62.85 / 29.6 60.13 / 16.26 57.62 / 30

66.51 / 23.94 63.14 / 25.61 59.67 / 16.11 58.11 / 31.31

63.67 / 21.13 62.99 / 22.62 59.41 / 13.78 55.95 / 24.67

0.19 1.89 -0.14 2.20

8.36 × 7.84 × 14.51 × 8.11 ×

CIFAR100 finetuned from ImageNet EfficientNet-B0 83.31 / 200 83.95 / 25.4 MobileNet-V2 74.10 / 200 74.35 / 22.4 ResNet-50 78.59 / 200 79.92 / 21 EfficientFormer-L1 85.30 / 200 86.74 / 36.6 ViT-B-MAE 86.81 / 200 85.92 / 52.2

83.98 / 26.1 74.61 / 25.3 80.17 / 23.4 87.28 / 39.8 86.94 / 55.8

84.11 / 20.8 74.39 / 19.5 79.97 / 17.8 86.85 / 33.2 85.55 / 43.5

0.80 0.51 1.58 1.98 0.13

9.62 × 10.26 × 11.24 × 6.02 × 4.60 ×

ImageNet EfficientNet-B0 MobileNet-V2 ResNet-50 EfficientFormer-L1 ViT-B-MAE

78.15 / 134.2 70.82 / 53.9 76.82 / 35.5 79.45 / 95.8 83.81 / 37.8

77.71 / 105.5 67.59 / 45.8 76.25 / 31.5 78.87 / 85.7 83.06 / 27.5

1.05 -0.78 1.78 0.34 0.71

3.32 × 5.46 × 3.17 × 3.50 × 3.64 ×

77.10 / 350 71.60 / 250 75.04 / 100 79.11 / 300 83.10 / 100

77.45 / 111.2 68.42 / 54.5 76.21 / 33.0 79.24 / 91.5 83.31 / 35.4

Table 2. Comparison with strong active learning baselines on CIFAR-100. Best results are shown in blue and second best in green. DDE corresponds to DataDiet ELN and DDF corresponds to DataDiet Forget. Model

Baseline

DDE

DDF

IES

Early Stopping

InfoBatch

DBPD

Adaptive-α

Adaptive-T

ResNet-50 EfficientNet EfficientFormer-L1

78.5 / 200 83.3 / 200 85.3 / 200

78.6 / 141 82.7 / 139 84.8 / 142

79.1 / 142 83.0 / 139 85.0 / 140

79.5 / 157 83.7 / 153 85.8 / 155

74.3 / 121 76.6 / 132 78.2 / 125

80.4 / 145 84.0 / 141 86.6 / 143

80.0 / 21 84.0 / 25.1 86.8 / 36.6

80.2 / 23.4 84.0 / 26.1 87.3 / 39.8

80.0 / 17.8 84.1 / 20.8 86.9 / 33.2

Table 3. Baseline trained to match the effective number of epochs (EE) used by dropout variants.

Variant

EE

Baseline (%)

Accuracy (%)

Adaptive-α Adaptive-T

23.9 21.1

63.18 61.82

66.51 63.67

Overall, these results provide empirical evidence that Adaptive Data Dropout does not follow a fixed curriculum, but instead continuously adapts data usage in response to

training dynamics. This ability to recover from overly aggressive data reduction is a key advantage over static schedules such as Progressive Data Dropout.

5. Limitations and Future Work Although Adaptive Data Dropout addresses an important limitation of fixed data schedules, it still has several constraints. First, the method introduces additional control hyperparameters, such as update thresholds, decay functions, and reheating factors, which may require careful tuning across different datasets, architectures, and optimization settings. While these parameters are relatively lightweight

Table 4. Accuracy (%) and effective epochs reported as accuracy/effective epochs for CIFAR-100 with EfficientNet-B0, trained for 25 and 50 epochs from scratch.

Model (50 epochs) EfficientNet-B0 (C100) (25 epochs) EfficientNet-B0 (C100)

Baseline

DBPD 0.7

Adaptive-α

Adaptive-T

63.65 / 50

67.62 / 18.64

67.95 / 20.71

66.58 / 14.38

63.32 / 25

66.94 / 14.02

66.85 / 15.24

66.07 / 10.51

(a) EfficientNet-B0

(b) ResNet-50

Figure 6. Number of training samples used per epoch under Adaptive Data Dropout. The model progressively reduces data usage but intermittently increases it in response to stagnation or drops in performance, demonstrating adaptive reheating behavior.

compared to model-level modifications, their interaction with training dynamics can influence both stability and efficiency. Second, because the adaptation mechanism is driven by training feedback, its effectiveness depends on the quality and stability of the chosen performance signal. In particular, noisy or highly fluctuating accuracy estimates, especially in early training, may lead to suboptimal or unstable updates. Incorporating more robust or aggregated signals, such as smoothed accuracy, loss trends, or uncertainty measures, may improve reliability. Third, although our experiments demonstrate consistent gains across standard image classification benchmarks, broader validation on other vision tasks, such as detection, segmentation, and video understanding, is necessary to fully establish the generality of the framework. These settings may exhibit different training dynamics, where the role of adaptive data exposure could differ. Finally, while the adaptive strategy improves flexibility compared to fixed schedules, it still requires selecting an initial schedule family or operating range, which implicitly constrains the space of possible behaviors. Future work could explore more principled and autonomous controllers that reduce or eliminate this dependency. In particular, one promising direction is to learn the adaptation policy itself, for example through reinforcement learning or

meta-learning, enabling the model to discover optimal data scheduling strategies directly from training signals. Such approaches could further unify data selection with optimization, potentially leading to fully self-regulating training systems that adapt not only data exposure but also other aspects of the learning process, such as augmentation, sampling, or learning rate schedules.

6. Conclusion We introduced Adaptive Data Dropout, a simple and effective extension of Progressive Data Dropout that replaces fixed schedules with feedback-driven adaptation. Inspired by self-regulated learning, our method dynamically adjusts data exposure during training based on observed progress, enabling the model to regulate the difficulty of learning over time. We considered two adaptive variants, Adaptive-α and Adaptive-T , which respectively control the decay parameter and the keep-fraction of the training data. Across standard image classification benchmarks and architectures, our framework is designed to provide a more favorable tradeoff between accuracy and computational efficiency relative to static data dropout strategies. These results support the broader view that training dynamics can benefit from adaptive, human-inspired principles rather than fixed curricula alone.

References [1] Maria-Florina Balcan, Andrei Broder, and Tong Zhang. Margin based active learning. In International Conference on Computational Learning Theory, pages 35–50. Springer, 2007. 2 [2] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41–48, 2009. 2 [3] Elizabeth L Bjork, Robert A Bjork, et al. Making things hard on yourself, but in a good way: Creating desirable difficulties to enhance learning. Psychology and the real world: Essays illustrating fundamental contributions to society, 2(59-68), 2011. 1, 3 [4] Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. In International Conference on Learning Representations. 2 [5] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255. IEEE, 2009. 5 [6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 5 [7] Epoch AI. Key trends and figures in machine learning, 2023. Accessed: 2025-10-19. 1 [8] Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881–2891, 2020. 2 [9] Shreyank N Gowda, Xinyue Hao, Gen Li, Shashank Narayana Gowda, Xiaobo Jin, and Laura Sevilla-Lara. Watt for what: Rethinking deep learning’s energy-performance relationship. arXiv preprint arXiv:2310.06522, 2023. 1 [10] Mark A Guadagnoli and Timothy D Lee. Challenge point: a framework for conceptualizing the effects of various practice conditions in motor learning. Journal of motor behavior, 36 (2):212–224, 2004. 1, 3 [11] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015. 1, 2 [12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 5 [13] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000– 16009, 2022. 5, 7

[14] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. 1, 2 [15] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017. 1, 2 [16] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 646–661. Springer, 2016. 3 [17] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2704–2713, 2018. 1, 2 [18] Angelos Katharopoulos and François Fleuret. Not all samples are created equal: Deep learning with importance sampling. In International conference on machine learning, pages 2525–2534. PMLR, 2018. 1, 2 [19] Seho Kee, Enrique Del Castillo, and George Runger. Queryby-committee improvement with diversity and density in batch active learning. Information Sciences, 454:401–418, 2018. 2 [20] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885–1894. PMLR, 2017. 2 [21] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. 5 [22] Yanyu Li, Geng Yuan, Yang Wen, Ju Hu, Georgios Evangelidis, Sergey Tulyakov, Yanzhi Wang, and Jian Ren. Efficientformer: Vision transformers at mobilenet speed. Advances in Neural Information Processing Systems, 35: 12934–12949, 2022. 1, 2, 5 [23] Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, et al. Prioritized training on points that are learnable, worth learning, and not yet learnt. In International Conference on Machine Learning, pages 15630–15649. PMLR, 2022. 2 [24] Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pages 6950–6960. PMLR, 2020. 2 [25] Vu-Linh Nguyen, Mohammad Hossein Shaker, and Eyke Hüllermeier. How to measure uncertainty in uncertainty sampling for active learning. Machine Learning, 111(1):89– 122, 2022. 2 [26] Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. Advances in neural information processing systems, 34:20596–20607, 2021. 6

[27] Ziheng Qin, Kai Wang, Zangwei Zheng, Jianyang Gu, Xiangyu Peng, Daquan Zhou, Lei Shang, Baigui Sun, Xuansong Xie, Yang You, et al. Infobatch: Lossless training speed up by unbiased dynamic data pruning. In The Twelfth International Conference on Learning Representations. 6 [28] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520, 2018. 1, 5 [29] Burr Settles. Active learning literature survey. 2009. 2 [30] MS Shriram, Xinyue Hao, Shihao Hou, Yang Lu, Laura Sevilla-Lara, Anurag Arnab, and Shreyank N Gowda. Progressive data dropout: An embarrassingly simple approach to train faster. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. 1, 3, 4, 5 [31] Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region-based object detectors with online hard example mining. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 761–769, 2016. 2 [32] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929–1958, 2014. 2 [33] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105–6114. PMLR, 2019. 1, 2, 5 [34] Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations. 2 [35] Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In International conference on machine learning, pages 1058–1066. PMLR, 2013. 2 [36] Tianyang Wang, Jun Huan, and Bo Li. Data dropout: Optimizing training data for convolutional neural networks. In 2018 IEEE 30th international conference on tools with artificial intelligence (ICTAI), pages 39–46. IEEE, 2018. 3 [37] Ross Wightman. Pytorch image models. https : / / github . com / rwightman / pytorch - image models, 2019. 7 [38] Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, and Ping Li. Dataset pruning: Reducing training data by examining generalization influence. arXiv preprint arXiv:2205.09329, 2022. 2 [39] Suqin Yuan, Runqi Lin, Lei Feng, Bo Han, and Tongliang Liu. Instance-dependent early stopping. In The Thirteenth International Conference on Learning Representations, 2025. 6 [40] Yaoyao Zhong, Weihong Deng, Han Fang, Jiani Hu, Dongyue Zhao, Xian Li, and Dongchao Wen. Dynamic training data dropout for robust deep face recognition. IEEE Transactions on Multimedia, 24:1186–1197, 2021. 3

[41] Barry J Zimmerman. Becoming a self-regulated learner: An overview. Theory into practice, 41(2):64–70, 2002. 1, 3