arXiv CSopen access

Scaling-Aware Data Selection for End-to-End Autonomous Driving Systems

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

machine learning, deep learning, neural networks

Scaling-Aware Data Selection for End-to-End Autonomous Driving Systems Tolga Dimlioglu1†, Nadine Chang2 , Maying Shen2 , Rafid Mahmood2,3 , Jose M. Alvarez2 1 New York University, 2 NVIDIA, 3 University of Ottawa [email protected], {nadinec, mshen, rmahmood, josea}@nvidia.com

(b)

(a)

Evaluation Tasks

Cluster 2

Large-scale deep learning models for physical AI applications depend on diverse training data collection efforts. These models and correspondingly, the training data, must address different evaluation criteria necessary for the models to be deployable in real-world environments. Data selection policies can guide the development of the training set, but current frameworks do not account for the ambiguity in how data points affect different metrics. In this work, we propose Mixture Optimization via Scaling-Aware Iterative Collection (MOSAIC), a general data selection framework that operates by: (i) partitioning the dataset into domains; (ii) fitting neural scaling laws from each data domain to the evaluation metrics; and (iii) optimizing a data mixture by iteratively adding data from domains that maximize the change in metrics. We apply MOSAIC to autonomous driving (AD), where an End-to-End (E2E) planner model is evaluated on the Extended Predictive Driver Model Score (EPDMS), an aggregate of driving rule compliance metrics. Here, MOSAIC outperforms a diverse set of baselines on EPDMS with up to 80% less data.

Cluster 3

arXiv:2604.08366v1 [cs.LG] 9 Apr 2026

Abstract

Cluster 1

(c)

Cluster 1

Localization

Cluster 2

Path Planning

Cluster 3

Obstacle Avoidance

Note: Darker lines indicate the cluster scales faster for the task.

(d)

Pittsburgh: less crowded, curvy roads

Las Vegas: dense city traffic

Figure 1. (a-b) The data pool is partitioned into a set of discrete domains which may each contribute to performance improvement of different evaluation tasks at varying rates. (c-d) Example application in autonomous driving: two clusters representing different driving contexts—Pittsburgh (curvy suburban roads) and Las Vegas (dense urban traffic). Data from separate contexts influence different rule-compliance metrics at distinct rates.

1. Introduction Large-scale deep learning models are fueled by diverse data collection efforts [35, 41]. This practice is particularly prominent in physical artificial intelligence (AI) applications such as autonomous driving (AD), where video clips are collected over different locations, weather, and traffic conditions [12, 18]. It is computationally inefficient to train models on all collected data, which in physical AI can scale to hundreds of millions of hours of clips. This necessitates data mixture selection policies to construct and grow training sets of diverse and influential samples that maximize desired performance metrics. Dataset selection and optimization has been broadly studied from various perspectives. For instance, influence (or duplicate) estimation techniques use feature information to select useful data samples [1, 5, 49], while active

learning strategies optimize over this feature space [38, 46]. Large language models (LLMs) and their multi-modal extensions have successfully leveraged scaling laws to forecast how model performance improves with dataset size [21, 26, 58]; this premise has expanded to other applications including AD [3]. Further, as data collection becomes increasingly complex, scaling laws are used to determine optimal mixtures of data from explicit domains (e.g., different languages, math, coding) [23, 36, 56]. Although these methods present a general opportunity for physical AI systems, they are not immediately usable for applications that require both understanding and interacting with diverse real-world scenarios for three main reasons. First, physical AI systems are evaluated over a set of potentially compet-

† work done during internship at NVIDIA.

ing metrics [14, 57]. Second, different data samples can influence different combinations of these metrics at various rates. Finally, the data pool is not necessarily immediately separable into subsets that have consistent, predictable influence on the metrics. Existing data mixture methods assume well-defined and homogeneous domains. However, they overlook the heterogeneous and metric-dependent improvement rates that arise when data sources influence different aspects of performance at varying rates [20, 54]. For example, a physical AI system such as an autonomous vehicle must progress along a route, follow driving rules, and avoid collisions [14]. High-traffic and pedestrian-heavy driving clips, when used for training, may impact certain metrics more than others. Moreover, finding such a subset of potential training data that has shared effects on the metrics requires careful selection and mining. In this work, we develop a data selection and mixture optimization policy that addresses the present physical AI challenges of multiple competing metrics and imprecise data partitions. To address the challenge of imprecise data domains, we first partition a data pool into a set of separable clusters, within which we can rank samples on their influence to the metrics. We estimate the impact of each cluster on each metric, and correspondingly, an overall utility function that aggregates all the metrics. This impact is measured in terms of scaling laws that estimate the improvement to the metrics if more data from a specific cluster were used for training. Finally, we iteratively add new data to the training set by identifying the cluster which will maximize the expected gain to the aggregate utility with each additional data point. In this way, we optimize the mixture of data from our generated partitions. Figure 1 summarizes the challenges. We apply our framework, Mixture Optimization via Scaling-Aware Iterative Collection (MOSAIC), for End-toEnd (E2E) autonomous driving, where the challenges of data heterogeneity and metric competition are particularly pronounced. The goal is to optimize the Extended Predictive Driving Model Score (EPDMS), which aggregates a diverse set of rule-compliance metrics. MOSAIC is more data efficient than existing methods and achieves better EPDMS performance than naı̈ve baselines with up to 82% less additional data. Our contributions are: • We propose MOSAIC, a generic data mixture optimization pipeline that (i) clusters and ranks data, (ii) models domain-specific data scaling, and (iii) mines samples to maximize the expected gain over aggregate metrics. • We apply MOSAIC to End-to-End Autonomous Driving (E2E AD) on the NAVSIM and OpenScene benchmarks using the challenge winning Hydra-MDP model [33], where it achieves substantially higher driving performance than existing data selection and mixture baselines, and improve data efficiency by up to 82%. Moreover, MOSAIC achieves the full training performance

while requiring 42% less data samples. • We empirically demonstrate the necessity and robustness of our joint clustering and scaling procedure. First, MOSAIC outperforms baselines regardless of the clustering approach (e.g., semantic captions, geolocation). Second, embedding scaling laws on top of clustering significantly outperforms clustering-only strategies. This underscores the importance of our principled data selection strategy, which leverages the estimated improvement rates of different data clusters to maximize model performance under limited data budgets.

2. Related Works Data Mixtures. Recent work has highlighted the importance of how data from different domains are combined for large-scale model training [17, 23, 36, 39, 53– 56]. DoReMi [53] employs two proxy models to estimate domain weights based on excess loss, which are later used to reweigh domains when training a larger model. DOGE [17], tracks domain-specific gradients while training the proxy model to better capture inter-domain dynamics. Chameleon [54] instead leverages kernel similarity scores computed in the model’s latent space to assign adaptive weights to data from different sources. Another line of work treats data mixture optimization as a regression problem: many small proxy models are trained with varying mixtures, and a regressor is then fit to predict the optimal mixture at larger scale [36, 56]. A particularly relevant approach to ours is ADO [23], which begins with a random data mixture and fits scaling estimators on the fly during training. The gradients of these estimators are used for mixture reweighting. However, ADO does not model how performance scales with different data sources in isolation, and it requires a temporal averaging mechanism with multiple hyperparameters to maintain the precision of scaling fits. Although the aforementioned data mixture methods assign weights to samples from different domains, these weights can also be interpreted as sampling probabilities for constructing mixtures with varying domain ratios. In our experiments, we adopt Chameleon [54] as a baseline, since it has been shown to outperform other mixture algorithms. Data Pruning & Selection. Data pruning aims to identify a compact subset of training data by removing redundant samples while preserving model performance [47]. In vision tasks, Abbas et al. [1] proposed removing visually similar samples using cosine similarity in the CLIP [43] feature space. Follow-up works extended this idea to specialized domains such as object detection [25] and fairness-aware multimodal learning [48]. It has also been shown analytically that optimal pruning strategies can improve power-law scaling behavior [49]. A closely related line of work, Active learning (AL), aims to maximize model performance improvement under a limited annotation budget [44, 59]. 2

Algorithm 1 Mixture Optimization via Scaling-Aware Iterative Selection (MOSAIC)

In this setting, the model has access to a large unlabeled data pool and, based on some selection signal the most informative samples are identified to be used for training. Early works focused on using the model’s prediction uncertainty, quantified through posterior probabilities [32], classifier margins [45], or entropy [24]. A notable method, CoreSet [46], seeks representation diversity by mining samples that maximize coverage in the latent space. Other data selection strategies quantify the sample importance using expensive signals such as influence on model updates [37], gradient-based criteria [8] or forgetting score [50]. End-to-End Autonomous Driving. This task aims to train planner models that map raw sensory inputs directly to control commands. Early approaches [4, 10, 42] learned control actions from RGB inputs via imitation learning, while later works incorporated richer input modalities such as LiDAR and navigational commands [9, 52]. Recently, conventional open-loop metrics have been shown to correlate poorly with closed-loop driving quality [13, 34]. This motivated the development of simulation benchmarks that better reflect real-world driving performance [14] and, AD models to employ probabilistic and rule-compliant trajectory planners [7, 22, 33].

Require: Pool dataset Dpool , number of clusters M , sample selection budget B. Ensure: Selected dataset Dsel i,ranked M 1: {Dpool }i=1 = ClusterAndRank(Dpool , M ) i,ranked M }i=1 )

2: {∆Ûi (n)}M i=1 = GetScalings({Dpool 3: Dsel ← {} 4: bi ← 0 for all i ∈ {1, . . . , M } 5: while |Dsel | < B do 6: for i = 1 to M do di (bi + 1) − ∆U di (bi ) 7: δi (bi ) ← ∆U 8: end for 9: j ← arg maxi δi (bi )

j,ranked

10: sample ← ReturnSample(Dpool 11: Dsel ← Dsel ∪ {sample} 12: bj ← bj + 1 13: end while 14: return Dsel

(EPDMS) [6, 27], which is an aggregate of R = 9 closedloop rule compliance scores: NC, DAC, DDC, TLC, EP, TTC, LK, HC, EC. Each driving clip can showcase only certain aspects of driving rule compliance, e.g. driving on a curvy road might improve lane keeping while degrading the comfort, however our goal is to elevate the model performance across all metrics. Consequently, solving this problem requires disentangling the relationships between the data samples and the metrics before optimizing the tradeoffs between them.

3. MOSAIC 3.1. Main Problem We want to train a Deep Neural Network (DNN) f (·; D) on a dataset D to perform a given task. We evaluate model performance using a set of R metrics Gr (f (·; D), Dval ) for r ∈ {1, · · · , R}, where Dval is a held-out validation dataset. For brevity, we denote Gr(f (·; D), Dval ) by Gr (D). To balance the trade-offs between the metrics, we use a utility function U ({Gr (D)}R r=1 ) that aggregates each metric into a final score. We do not assume about the structure of the utility function; for example, PRthe simplest approach would be a summation U (·) = r=1 Gr (D). We initialize with a current training dataset Dtrain and a data pool Dpool . Given a budget B, our goal is to select a subset Dsel ⊂ Dpool with |Dsel | = B that maximizes the improvement in model performance when f is retrained on the combined dataset Dtrain ∪ Dsel . Formally, we write max

Dsel ⊂Dpool |Dsel |=B

U {Gr (Dtrain ∪ Dsel )}R r=1

, bj )

3.2. Scaling-Aware Iterative Collection We propose Mixture Optimization via Scaling-Aware Iterative Collection (MOSAIC); a three-stage data selection framework: (i) first, cluster the pool Dpool into partitions that capture distinct driving scene contexts and rank the data samples based on an importance score within each cluster; (ii) estimate the scaling law of adding data from each cluster with respect to the utility U ; and (iii) iteratively mine samples from the clusters to optimize 1. Figure 2 visualizes our framework. Algorithm 1 summarizes the steps. 3.2.1. Clustering & Ranking the Data

(1)

Before solving problem (1), we first disentangle the relationships between the metrics and data samples by clustering the data pool into a set of structured domains [15, 47]. Our goal is to find subsets of the data pool that have similar influence, i.e., samples that all influence the same set of metrics. Given a feature representation, SM we icluster the data pool into M domains, i.e., Dpool = i=1 Dpool . For example in AD, we may partition clips into clusters of highway driving, busy intersections, and calm local streets, that pri-

To solve problem (1), we must determine how each data sample added in Dsel influences each of the metrics, while optimizing the trade-offs between these metrics to maximize U (·). For instance, in our AD application, f (·; D) is a planner model that maps sensory inputs to a predicted driving trajectory. Here, our goal is to identify driving clips that optimize the Extended Predictive Driving Model Score 3

(a.1) Clustering

(b) Estimating Cluster Scalings

Performance Gain

Performance Gain Smaller gain

Larger gain

Smaller gain 1 # samples added from C1

C1 scaling fit C2 scaling fit C1 pilot run C2 pilot run

(a.2) Ranking

C1: C2:

… …

# samples added from cluster

Larger gain

1 # samples added from C2

8 # samples added from C2

1 # samples added from C1

C1: C2: 1

sample selection iteration

Figure 2. Overview of the proposed MOSAIC framework. (a) The pool Dpool is clustered and ranked by sample importance. (b) Clusterwise scaling laws are fitted on pilot runs to estimate how performance scales with added data. (c) Samples are then iteratively mined from the cluster with the highest estimated marginal gain under the fitted scaling laws.

marily address ego progress, collision avoidance, and traffic light compliance, respectively. Although, clustering separates the data into domains of similar influence, each domain will include samples that have stronger influence than others [28, 30]. When adding data, we should first exhaust the higher-influence samples [49]. As a result, we rank the samples x within each cluster via an importance score I(x). In our application, we define importance by evaluating the model on that sample I(x) := U ({Gr (f (·; Dtrain ), x)}R r=1 ). Later when adding data from each cluster, we first select samples with higher I(x) (Line 1 of Algorithm 1).

from each cluster and then estimating the effect of data from each domain via a scaling law. First, we apply the following linear separable approximation ∆Umix (n1 , . . . , nM ) ≈

M i=1 ni =B

∆Umix (n1 , · · · , nM )

(3)

where each ∆Ui (n) is the improvement in utility when adding only the data from the i-th domain R R i ) r=1 − U Gr (Dtrain ) r=1 U Gr (Dtrain ∪ Dsel Intuitively, the approximation in (3) assumes that each domain has an independent effect on the overall ∆Umix . Moreover, this assumption allows us to estimate how model performance scales if we add data from each domain independently. We use a saturating exponential scaling law

Given a set of discrete domains, problem (1) can be reformulated into a data mixture optimization problem. Let i i Dsel ⊂ Dpool be the data added from the i-th domain and i let Dsel = ∪M i=1 Dsel . Furthermore, because each sample i in Dpool is ranked via importance scores, it remains only to determine how many samples to draw from each domain. Mathematically, we reformulate problem (1) to a proxy optimization problem below max P

∆Ui (ni ).

i=1

3.2.2. Selecting Data by Optimizing a Mixture

n1 ,··· ,nM ,

M X

di (n) := ai (1 − e−n/τi ) ∆Ui (n) ≈ ∆U

(4)

where ai and τi are learnable parameters of the scaling law estimated from small-scale pilot-runs (we provide details in Section 9 of the Appendix.), and n denotes the number of added samples [55]. Here, ai represents the asymptotic improvement on the total utility U when sampling from domain i, while τi governs the saturation rate, i.e., how quickly the marginal benefit of adding data from the domain decreases. Obtaining the scaling esimators is symbolically captured in line 2 of Algorithm 1. Then, substituting (4) into problem (2) yields a concave maximization problem that we can compute and solve.

(2)

i where ni := |Dsel | is the number of samples drawn from the i-th domain and ∆Umix (n1 , · · · , nM ) is ! M [ i R U {Gr (Dtrain ∪ { Dsel })}r=1 −U ({Gr (Dtrain )}R r=1 ) i=1

3.2.3. Scaling-Aware Iterative Data Collection by FirstDifference Steps

the change in utility after adding ni points from each domain. Note that the objective above is equivalent to optii R mizing U ({Gr (Dtrain ∪ {∪M i=1 Dsel }}r=1 )), but we use the above formulation since it explicitly expresses the problem in terms of performance gains from additional data. Solving problem (2) requires quantifying how adding data samples from each domain will improve U (·). We estimate this by approximating ∆Umix into separate effect

We propose an efficient algorithm to solve problem (2) by allocating data samples one-by-one from the domain that stands to give the highest marginal improvement to Umix at any given time. Intuitively, this iterative adding of data mimics a gradient-based approach of taking small steps to optimize the data mixture. 4

Suppose that we have so far added bi data points from each cluster in order to generate Dsel . Then, let di (bi + 1) − ∆U di (bi ), δi (bi ) := ∆U

VoVNetV2-99 backbone [31, 40]. The trajectory vocabulary size is set to 16,384. For Openscene experiments, rule-based distillation is disabled due to the substantial preprocessing time required to compute compliance scores.

(5)

di (bi ) if we draw one be the marginal improvement in ∆U additional data point from the i-th domain. Mathematically, δi (bi ) is an approximate first-difference analogue of the pardi (n) is tial derivative of Umix . Furthermore, because ∆U a concave function of n, this difference decreases as bi increases. This means that at a certain point, each domain yields diminishing value to the training dataset and we should draw from other domain. In our algorithm, we iteratively add data from the domain with the highest marginal improvement. In each iteration, if we have so far drawn bi samples from the i-th domain, we first identify j = arg maxi δi (bi ). We then sample a data i point from Dpool according to the importance scores I(x). We then update our counts bi , and repeat the process until we have reached the budget (lines 5-14 in Algorithm 1).

Baselines. We compare MOSAIC against several baseline data selection strategies: • Random: selects clips uniformly from the pool dataset under the given selection budget. • Uncertainty [24]: measured via the entropy of the trajectory logits. Samples with higher entropy are prioritized. • Coreset [46]: selects samples from the pool that maximize diversity over the feature space. • Chameleon [54]: a data mixture framework that uses kernel ridge scores using domain embeddings in the model’s feature space to assign mixture weights to each domain. Pseudo-codes are in Section 7 of Appendix. For Chameleon and MOSAIC, we cluster Dpool into domains defined by the map metadata (i.e., Boston, Pittsburgh, Singapore, Vegas). Each experiment is repeated with three random seeds. For Openscene experiments with more than 1,000 clips, we use two seeds to reduce computational cost. Reported results are averaged over runs, and the standard deviation is shown as a subscript in the tables.

4. Experiments We empirically evaluate MOSAIC on two different datasets (Openscene and Navtrain) using a challenge-winning model Hydra-MDP, and report consistent gains in the model performance at all budgets while being up to 80% data efficiency than the baselines.

Metrics. We evaluate models using the EPDMS, an aggregate of nine rule-compliance metrics that has been shown to correlate strongly with closed-loop driving performance [6, 14]. Consequently, EPDMS has become the standard evaluation metric for AD planners, replacing conventional open-loop measures such as ADE and FDE. Formally, EPDMS is computed as: P Y m∈Mavg wm m EPDMS := m. P m∈Mavg wm

4.1. Protocols We provide more details in Section 7 of Appendix. Datasets. We use two train–pool configurations: the curated Navtrain [14] split and the full trainval split of Openscene [11]. For clarity, we refer to the latter as the Openscene experiment. In both settings, evaluation is conducted on the curated validation split navtest [14]. Both datasets contain driving session clips lasting from 30 seconds to 50 minutes, which has significant temporal variation over a limited number of sessions. Consequently, we segment each session into fixed-length 10-second virtual clips (20 frames at 2 Hz) and by doing so, we align our data handling practice with the industry standards [16]. In the experiments, each virtual clip is treated as a single sample. For Navtrain, we use the dataset as both Dtrain and Dpool , comprising 4,601 virtual clips. We randomly select 460 clips for Dtrain , with the remaining 4,141 clips forming Dpool . We evaluate all methods under budgets B ∈ {100, 200, 400, 800, 1600, 2400}. For OpenScene, we randomly select 1,000 clips as the Dtrain and reserving the remaining 31,539 as Dpool . The sample selection budgets of this setting are B ∈ {250, 500, 1000, 2000, 4000, 8000}.

m∈Mpen

where Mpen := {NC, DAC, DDC, TLC} denotes the set of penalty terms, and Mavg := {EP, TTC, LK, HC, EC} denotes the metrics combined via a weighted average, with weights {5, 5, 2, 2, 2} respectively [6]. The glossary of the rule compliance metrics are in Section 7 of Appendix. Similar to the relevant works [51], we also measure how each data selection policy improves EPDMS relative to the Random baseline to assess sample efficiency. Specifically, we report the Budget Ratio to Match Random (BRMR); the ratio of the data budget B required by each method to achieve the same EPDMS performance attained by random selection at the same budget. Formally, let Bk denote the number of samples required by selection strategy k to match the EPDMS obtained by random sampling with budget B. Then BRMR := Bk /B. Lower BRMR indicates greater sample efficiency, as it reflects fewer samples needed to achieve the same performance level as random selection.

Model. We use the Hydra-MDP model [33], the winner of the NAVSIM Challenge in 2024 [14], with a pretrained 5

Table 1. Validation EPDMS (higher is better) and BRMR (lower is better) reported in OpenScene (Section A) and Navtrain (Section B) settings. We report the results for all budgets in Section 8 of Appendix. A. Openscene

across all budgets. It also attains the lowest BRMR values (< 0.4), corresponding to a 60–70% reduction in the number of samples needed to match the performance of Random. We conclude that MOSAIC remains highly effective even on the more challenging, curated Navtrain split, where each clip already carries substantial learning value. The section B of Table 2 reports the breakdown of EPDMS. MOSAIC achieves consistent improvements across all metrics, with the largest gains observed in DAC, NC, and LK. Importantly, MOSAIC understands the tradeoff between the metrics, and shifts the collection effort from saturated, less impactful metrics toward those that require more improvement. Overall, MOSAIC provides a more balanced and sustained improvement profile, suggesting that the scaling-aware allocation identifies data with broader generalization benefits. Ultimately, this allows MOSAIC to achieve a higher EPDMS than the baselines under the same clip budget.

B. Navtrain

Budget

Method

EPDMS (↑)

BRMR (↓)

EPDMS (↑)

BRMR (↓)

A. 250 B. 100

Random Uncertainty Coreset Chameleon MOSAIC

72.84±1.14 70.78±0.59 76.26±0.48 72.97±1.72 77.38±1.58

1.00 14.58 0.20 0.86 0.15

84.66±0.60 84.50±0.48 85.29±0.47 84.57±0.18 86.29±0.43

1.00 1.47 0.53 1.07 0.30

A. 1000 B. 400

Random Uncertainty Coreset Chameleon MOSAIC

75.84±0.90 71.12±0.38 80.46±0.02 79.08±0.74 81.68±0.52

1.00 8.00 0.22 0.49 0.18

86.69±0.20 86.07±0.75 87.09±0.29 87.04±0.60 88.21±0.03

1.00 2.00 0.79 0.82 0.38

A. 4000 B. 1600

Random Uncertainty Coreset Chameleon MOSAIC

80.38±0.55 73.46±0.19 83.63±0.36 82.92±0.13 84.25±0.14

1.00 2.00 0.25 0.39 0.18

88.62±0.22 87.75±0.37 89.30±0.19 89.50±0.20 90.18±0.25

1.00 1.36 0.58 0.62 0.37

4.2. Main Results: Openscene

4.4. Ablating the effectiveness of MOSAIC

Table 1 (Section A) reports the EPDMS and BRMR scores for different data selection methods. Across all clip budgets B ∈ {250, 1000, 4000}, MOSAIC consistently achieves the highest EPDMS, approximately one point higher than the next best method. This demonstrates the superior utility gains of MOSAIC under limited data. Moreover, MOSAIC requires over 80% fewer samples to match the performance achieved by random selection (i.e., BRMR < 0.2). We break down EPDMS into the individual nine metrics in Table 2. Section A corresponds to the Openscene experiments. The base model before data collection is particularly limited in DAC and EC, which impact EPDMS. MOSAIC achieves the largest gain in DAC, nearly 10 points higher than the base, and improves EC and EP, while maintaining balanced performance gains across other metrics. In contrast, the other methods yield less gains over the base DAC, and instead improves TTC and EC, that have less effect on the final EPDMS. On the other hand, MOSAIC achieves consistently Top-2 performance across all rulecompliance metrics, while strategically prioritizing DAC, the metric with the greatest room for improvement. This underscores the importance of incorporating scaling-aware collection into the data selection strategy to optimize U more effectively and achieve better trade-offs across competing metrics.

Dynamics of scaling-aware data selection. In the Openscene experiments, Dpool is partitioned based on geolocation into four domains corresponding to Las Vegas, Boston, Singapore, and Pittsburgh. Figure 3 illustrates the fitted scaling curves for each city, where ⋆ markers denote the pilot-run results used to estimate the parameters of the scaling curves in Equation 4. We note that different domains scale at different rates depending on the how many clips are added. Specifically, data collected from Boston and Singapore yield the largest initial performance gains in the low-data regime (< 500 clips), while Pittsburgh maintains steadier improvements and eventually supercedes all other domains at high data budgets. In contrast, the Las Vegas cluster provides the smallest gains and saturates early. These heterogeneous scaling behaviors are later exploited by the scaling-aware selection policy of MOSAIC to maximize the performance gain, under varying data budgets. Figure 4 shows how these fitted scaling laws influence the order in which samples are added to the training set. The y-axis lists data clusters, i.e. the city names, and the x-axis denotes the iteration index. Each bar indicates from which cluster the next sample is collected from at any iteration. During the early stages, only Boston and Singapore are actively mined, while Las Vegas and Pittsburgh are largely ignored. Figure 3 (top right) confirms this behavior: when the budget is 250, most selected clips originate from Boston and Singapore. As the returns from Boston and Singapore diminish, Pittsburgh’s steadier scaling curve makes it increasingly favorable between indices 500 to 3700. Figure 3 (bottom right) shows that at 4000 clips, the selected set is dominated by Pittsburgh samples. After around 3700 collection rounds, the Pittsburgh data domain is exhausted. Beyond approximately 2500 sample selections, the scaling

4.3. Main Results: Navtrain Compared to Openscene, Navtrain is a curated dataset emphasizing non-trivial driving scenarios such as dense traffic and complex maneuvers. Table 1 (Section B) summarizes the EPDMS and BRMR results. Here, MOSAIC consistently delivers the strongest performance, achieving up to 1.1 points higher EPDMS than the next best method 6

Table 2. Breakdown of the nine EPDMS rule-compliance metrics for the base model and the models trained with data selected by various strategies at a single budget, shown for both the OpenScene and Navtrain experiments. Setting

NC (↑)

DAC (↑)

DDC (↑)

TLC (↑)

EP (↑)

TTC (↑)

LK (↑)

HC (↑)

EC (↑)

EPDMS (↑)

A. Openscene

Base

94.05

83.9

96.28

99.6

85.96

92.95

93.26

98.25

81.88

72.0

Budget 4000 Clips

Random Uncertainty Coreset Chameleon MOSAIC

96.32±0.59 94.67±0.28 97.11±0.18 96.76±0.24 96.97±0.32

90.53±0.06 85.11±0.51 92.93±0.60 92.32±0.02 93.59±0.11

99.06±0.07 97.15±0.54 99.44±0.06 99.51±0.01 99.59±0.04

99.79±0.05 99.71±0.04 99.82±0.02 99.77±0.01 99.80±0.01

86.36±0.48 84.26±0.69 86.65±0.55 86.98±0.17 87.14±0.98

95.66±0.52 93.72±0.40 96.42±0.19 95.91±0.31 96.18±0.45

95.68±0.09 93.26±0.09 96.66±0.30 96.49±0.12 96.62±0.08

98.30±0.01 98.28±0.02 98.16±0.12 98.32±0.01 98.28±0.01

84.46±0.14 81.34±1.06 85.10±0.06 85.51±0.11 85.06±0.34

80.38±0.55 73.46±0.19 83.63±0.36 82.92±0.13 84.25±0.14

B. Navtrain

Base

95.3

95.94

99.09

99.6

88.09

94.55

94.49

98.25

82.39

83.97

Budget 1600 Clips

Random Uncertainty Coreset Chameleon MOSAIC

97.17±0.07 96.92±0.38 97.50±0.10 97.43±0.22 98.04±0.24

98.19±0.43 97.66±0.08 98.31±0.34 98.46±0.17 98.61±0.32

99.42±0.05 99.22±0.10 99.59±0.03 99.60±0.05 99.63±0.06

99.69±0.02 99.77±0.02 99.72±0.05 99.75±0.03 99.73±0.02

89.36±0.12 89.02±0.28 89.27±0.21 89.60±0.19 89.28±0.19

96.50±0.14 96.24±0.40 96.86±0.07 96.83±0.30 97.50±0.32

96.45±0.25 96.10±0.07 96.75±0.22 96.89±0.07 97.07±0.06

98.31±0.03 98.30±0.01 98.30±0.03 98.30±0.03 98.28±0.04

83.17±0.76 82.92±0.38 83.88±0.50 83.87±0.34 83.70±0.41

88.62±0.22 87.75±0.37 89.30±0.19 89.50±0.20 90.18±0.25

boston pittsburgh singapore vegas

EPDMS

ratio

1.00 0.75 0.50 0.25 0.00

ratio

Scaling Curves

1.00 0.75 0.50 0.25 0.00

5 4 3 2

250 Clips

Table 3. Top uni-grams and bi-grams of different clusters.

Uni-grams & Bi-grams Cluster 1 Cluster 2 Cluster 3 Cluster 4 Cluster 5 Cluster 6

4000 Clips

calm, day, street, trees, signs, yellow signals, crossing, crosswalks, pedestrians highway, vehicles, busy urban, palm trees building, area, large, paved, parking city street, major city, moderate precipitation, potential rain, overcast, cloudy

MOSAIC is optimal under different clustering mechanisms. Instead of partitioning the data pool into domains separated by geolocation, we cluster the Navtrain data pool using captions generated for each clip. We use the Qwen-2.5-VL-32B-Instruct model [2] to generate captions for all clips in Dpool . We form six clusters that capture distinct driving and scene contexts using the TF-IDF feature vectors of the generated captions. The top uni-grams and bigrams characterizing each cluster are provided in Table 3. Figure 5 reports the validation EPDMS as a function of the data selection budget for all strategies. We also indicate the base performance and the full-training performance, corresponding to the model trained by including all

Combining clustering and ranking yields the best data selection policy. We ablate the effects of both clustering and ranking components by individually disabling them. First, we use a “w/o Clustering” variant where we simply rank Dpool by the EPDMS importance scores I(x) and greedily add samples with the lowest scores until reaching the collection budget. Second, we use a “w/o Ranking” variant where we disable the ranking step, while retaining clustering and the scaling-aware estimation of how many samples to collect. Here, we simply sample data points from each domain randomly to satisfy the budget. Moreover, the scaling laws are also estimated on unranked domains. Figure 6 visualizes these baselines to show that perfor-

0 0

200

400 600 # Added clips

800

1000

pit bost ts o sin burgn ga h po ve re ga s

curves of Boston, Singapore, and Pittsburgh approach saturation, causing their marginal gains to diminish. As a result, the expected improvement from the initial Las Vegas samples becomes comparable to those of the other regions, leading MOSAIC to mine from Vegas.

4,141 clips in Dpool for training. (We provide full table with subscores in Section 8 of Appendix) MOSAIC consistently outperforms all baselines, including Chameleon (i.e., the other data mixture optimization method); requires 61% and 52% fewer samples than random selection to match its performance at the highest budgets of 1,600 and 2,400 clips, respectively. Moreover, MOSAIC reaches the full performance of training with all data samples using only 2,400 clips, i.e., 42% fewer samples. Interestingly, Chameleon degrades under caption-based clustering, despite being the strongest baseline in the previous setting. This indicates that its kernel ridge weighting is highly sensitive to the structure of the clustered domains. Also, since the clustering choice only affects Chameleon and MOSAIC, the other strategies have the same performance as before.

Figure 3. (Left) Performance scalings of different clusters, obtained by fitting the estimator in Equation 4 on 2 pilot runs, denoted by ⋆. (Right) Geolocation distributions at different budgets as a result of scaling-aware iterative selection.

Full selection

boston pittsburgh singapore vegas 0

1000

Zoomed

2000

3000

sample selection iteration

3700

3725

3750

Figure 4. Visualization of the scaling-aware iterative data selection process. The x-axis denotes the sample selection iterations, and the y-axis lists the cluster names. Each vertical bar indicates from which cluster the next sample is mined at a given iteration, based on the estimated cluster-wise scaling fits. The left panel shows the complete selection process up to 4,000 clips, and the right panel zooms into iterations 3,700–3,750 for clarity.

EPDMS vs. Number of Clips 0.90

52%

0.89 Val. EPDMS

61%

0.88 0.87

MOSAIC - w/o Clustering - w/o Ranking Random Base Full training

0.86 0.85 0.84 0

500 1000 1500 2000 Number of Clips Added

2500

Figure 6. Analyzing the contribution of different components of the MOSAIC framework.

Figure 5. Validation EPDMS for various budgets obtained by different strategies when Navtrain data pool is clustered using clip captions. MOSAIC requires 61% and 52% fewer clips than Random selection to match its performance at the 1,600- and 2,400clip budgets, respectively.

ing small pilot subsets or through continual training. Thus, despite the initial overhead for the pilot runs to obtain cluster scalings, MOSAIC ultimately requires less total compute to achieve superior performance in the large-data regime.

mance improvements in the low-data regime (up to a collection budget of 800 clips) can largely be attributed to ranking. The MOSAIC and w/o Clustering variants achieve competitive EPDMS in this region. However, in the higher data regime, merely adding clips with low EPDMS scores becomes less effective, as the performance of w/o Clustering begins to lag behind MOSAIC. Finally, we note that both of these disabled variants still outperform random collection by a large margin.

6. Conclusion We introduce MOSAIC, a scaling-aware data selection framework that jointly leverages clustering, ranking, and scaling-law modeling to maximize the performance of a model defined by multiple competing metrics, under a limited data budget. We apply MOSAIC to E2E AD, where a planner model uses a diverse data pool to optimize a utility function that aggregates competing rule compliance metrics. Empirically, MOSAIC consistently outperforms existing data selection and mixture baselines on both the Openscene and Navtrain datasets by achieving substantial gains in EPDMS and sample efficiency. Ablation studies further highlight the framework’s mechanisms, analyze the necessity of the individual components components, and demonstrate robustness to clustering choices as long as semantic consistency is maintained. Overall, MOSAIC offers a general and principled blueprint for identifying influential data in large-scale, heterogeneous learning systems.

5. Limitations We note two key limitations of MOSAIC. First, the relaxation from Equation 2 to Equation 3 assumes that each cluster’s contribution is well captured by its own scaling curve ∆Ui (n), with limited cross-cluster interactions. Consequently, if the clustering fails to produce well-separated groups, this assumption may be violated, leading MOSAIC to suboptimal allocation. Second, MOSAIC relies on pilot runs to estimate clusterspecific scaling curves, which introduces additional computational cost. However, as shown in Section 9 of the Appendix, accurate scaling fits can be obtained efficiently us8

References

and Pattern Recognition, Vancouver, Canada, pages 18–22, 2023. 5, 1 [12] Michael J. Coren. Tesla has 780 million miles of driving data, and adds another million every 10 hours, 2025. Accessed: YYYY-MM-DD. 1 [13] Daniel Dauner, Marcel Hallgarten, Andreas Geiger, and Kashyap Chitta. Parting with misconceptions about learningbased vehicle motion planning. In Conf. on Robot Learning, 2023. 3 [14] Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang, Zetong Yang, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, et al. Navsim: Data-driven non-reactive autonomous vehicle simulation and benchmarking. Advances in Neural Information Processing Systems, 37:28706–28719, 2024. 2, 3, 5, 1 [15] Shizhe Diao, Yu Yang, Yonggan Fu, Xin Dong, Dan Su, Markus Kliegl, Zijia Chen, Peter Belcak, Yoshi Suhara, Hongxu Yin, et al. Climb: Clustering-based iterative data mixture bootstrapping for language model pre-training. arXiv preprint arXiv:2504.13161, 2025. 3 [16] Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles Qi, Yin Zhou, Zoey Yang, Aurélien Chouard, Pei Sun, Jiquan Ngiam, Vijay Vasudevan, Alexander McCauley, Jonathon Shlens, and Dragomir Anguelov. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In IEEE Int. Conf. on Computer Vision, 2021. 5, 1 [17] Simin Fan, Matteo Pagliardini, and Martin Jaggi. Doge: Domain reweighting with generalization estimation. arXiv preprint arXiv:2310.15393, 2023. 2 [18] Adam Grzywaczewski. Training ai for self-driving vehicles: the challenge of scale, 2017. Accessed: YYYY-MM-DD. 1 [19] K. Tan et al. H. Caesar, J. Kabzan. Nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles. In CVPR ADP3 workshop, 2021. 1 [20] Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro Von Werra, and Martin Jaggi. Scaling laws and compute-optimal training beyond fixed training durations. arXiv preprint arXiv:2405.18392, 2024. 2 [21] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701, 2020. 1 [22] Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Vad: Vectorized scene representation for efficient autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 8340–8350, 2023. 3 [23] Yiding Jiang, Allan Zhou, Zhili Feng, Sadhika Malladi, and J Zico Kolter. Adaptive data optimization: Dynamic sample selection with scaling laws. In The Thirteenth International Conference on Learning Representations, 2025. 1, 2 [24] Ajay J Joshi, Fatih Porikli, and Nikolaos Papanikolopoulos. Multi-class active learning for image classification. In 2009

[1] Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos. Semdedup: Data-efficient learning at web-scale through semantic deduplication. arXiv preprint arXiv:2303.09540, 2023. 1, 2 [2] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. 7 [3] Mustafa Baniodeh, Kratarth Goel, Scott Ettinger, Carlos Fuertes, Ari Seff, Tim Shen, Cole Gulino, Chenjie Yang, Ghassen Jerfel, Dokook Choe, et al. Scaling laws of motion forecasting and planning–a technical report. arXiv preprint arXiv:2506.08228, 2025. 1 [4] M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, X. Zhang, J. Zhao, and K. Zeiba. End to end learning for self-driving cars, 2016. Available at https: //arxiv.org/abs/1604.07316. 3 [5] Andrei Z Broder. On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171), pages 21–29. IEEE, 1997. 1 [6] Wei Cao, Marcel Hallgarten, Tianyu Li, Daniel Dauner, Xunjiang Gu, Caojun Wang, Yakov Miron, Marco Aiello, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, Andreas Geiger, and Kashyap Chitta. Pseudosimulation for autonomous driving. In Conference on Robot Learning (CoRL), 2025. 3, 5 [7] Shaoyu Chen, Bo Jiang, Hao Gao, Bencheng Liao, Qing Xu, Qian Zhang, Chang Huang, Wenyu Liu, and Xinggang Wang. Vadv2: End-to-end vectorized autonomous driving via probabilistic planning. arXiv preprint arXiv:2402.13243, 2024. 3 [8] Anshuman Chhabra, Peizhao Li, Prasant Mohapatra, and Hongfu Liu. ” what data benefits my classifier?” enhancing model performance and interpretability through influencebased data selection. In The Twelfth International Conference on Learning Representations, 2024. 3 [9] Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. Transfuser: Imitation with transformer-based sensor fusion for autonomous driving. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(11):12878–12895, 2022. 3 [10] Felipe Codevilla, Matthias Müller, Antonio López, Vladlen Koltun, and Alexey Dosovitskiy. End-to-end driving via conditional imitation learning. In 2018 IEEE international conference on robotics and automation (ICRA), pages 4693– 4700. IEEE, 2018. 3 [11] OpenScene Contributors. Openscene: The largest up-to-date 3d occupancy prediction benchmark in autonomous driving. In Proceedings of the Conference on Computer Vision

In Proceedings of the IEEE/CVF international conference on computer vision, pages 9274–9283, 2021. 3 [38] Rafid Mahmood, Sanja Fidler, and Marc T Law. Low-budget active learning via wasserstein distance: An integer programming approach. In International Conference on Learning Representations, 2022. 1 [39] Rafid Mahmood, James Lucas, Jose M Alvarez, Sanja Fidler, and Marc T Law. Optimizing data collection for machine learning. Journal of Machine Learning Research, 26(38): 1–52, 2025. 2 [40] Dennis Park, Rares Ambrus, Vitor Guizilini, Jie Li, and Adrien Gaidon. Is pseudo-lidar needed for monocular 3d object detection? In Proceedings of the IEEE/CVF international conference on computer vision, pages 3142–3152, 2021. 5, 1 [41] Priyaranjan Pattnayak, Hitesh Laxmichand Patel, Bhargava Kumar, Amit Agarwal, Ishan Banerjee, Srikant Panda, and Tejaswini Kumar. Survey of large multimodal model datasets, application categories and taxonomy. arXiv preprint arXiv:2412.17759, 2024. 1 [42] Dean Pomerleau. ALVINN: an autonomous land vehicle in a neural network. In Advances in Neural Information Processing Systems 1, [NIPS Conference, Denver, Colorado, USA, 1988], pages 305–313. Morgan Kaufmann, 1988. 3 [43] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PmLR, 2021. 2 [44] Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Brij B Gupta, Xiaojiang Chen, and Xin Wang. A survey of deep active learning. ACM computing surveys (CSUR), 54(9):1–40, 2021. 2 [45] Dan Roth and Kevin Small. Margin-based active learning for structured output spaces. In European conference on machine learning, pages 413–424. Springer, 2006. 3 [46] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv:1708.00489, 2017. 1, 3, 5, 2 [47] Maying Shen, Nadine Chang, Sifei Liu, and Jose M Alvarez. Sse: Multimodal semantic data selection and enrichment for industrial-scale data assimilation. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1, pages 2525–2535, 2025. 2, 3 [48] Eric Slyman, Stefan Lee, Scott Cohen, and Kushal Kafle. Fairdedup: Detecting and mitigating vision-language fairness disparities in semantic dataset deduplication. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13905–13916, 2024. 2 [49] Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35:19523–19536, 2022. 1, 2, 4 [50] Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J.

ieee conference on computer vision and pattern recognition, pages 2372–2379. IEEE, 2009. 3, 5, 1 [25] Feiyang Kang, Nadine Chang, Maying Shen, Marc T Law, Rafid Mahmood, Ruoxi Jia, and Jose M Alvarez. Adadedup: Adaptive hybrid data pruning for efficient large-scale object detection training. arXiv preprint arXiv:2507.00049, 2025. 2 [26] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. 1 [27] Napat Karnchanachari, Dimitris Geromichalos, Kok Seang Tan, Nanxiang Li, Christopher Eriksen, Shakiba Yaghoubi, Noushin Mehdipour, Gianmarco Bernasconi, Whye Kit Fong, Yiluan Guo, et al. Towards learning-based planning: The nuplan benchmark for real-world autonomous driving. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 629–636. IEEE, 2024. 3 [28] Angelos Katharopoulos and François Fleuret. Not all samples are created equal: Deep learning with importance sampling. In International conference on machine learning, pages 2525–2534. PMLR, 2018. 4 [29] D. P. Kingma and J. L. Ba. Adam: A method for stochastic optimization. In Int. Conf. on Learning Representations, 2015. 1 [30] Agata Lapedriza, Hamed Pirsiavash, Zoya Bylinskii, and Antonio Torralba. Are all training examples equally valuable? arXiv preprint arXiv:1311.6510, 2013. 4 [31] Youngwan Lee and Jongyoul Park. Centermask: Realtime anchor-free instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13906–13915, 2020. 5, 1 [32] David D Lewis and Jason Catlett. Heterogeneous uncertainty sampling for supervised learning. In Machine learning proceedings 1994, pages 148–156. Elsevier, 1994. 3 [33] Zhenxin Li, Kailin Li, Shihao Wang, Shiyi Lan, Zhiding Yu, Yishen Ji, Zhiqi Li, Ziyue Zhu, Jan Kautz, Zuxuan Wu, et al. Hydra-mdp: End-to-end multimodal planning with multitarget hydra-distillation. arXiv preprint arXiv:2406.06978, 2024. 2, 3, 5, 1 [34] Zhiqi Li, Zhiding Yu, Shiyi Lan, Jiahan Li, Jan Kautz, Tong Lu, and Jose M Alvarez. Is ego status all you need for openloop end-to-end autonomous driving? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14864–14873, 2024. 3 [35] Mingyu Liu, Ekim Yurtsever, Jonathan Fossaert, Xingcheng Zhou, Walter Zimmer, Yuning Cui, Bare Luka Zagar, and Alois C Knoll. A survey on autonomous driving datasets: Statistics, annotation quality, and a future outlook. IEEE Transactions on Intelligent Vehicles, 2024. 1 [36] Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. Regmix: Data mixture as regression for language model pre-training. In The Thirteenth International Conference on Learning Representations, 2025. 1, 2 [37] Zhuoming Liu, Hao Ding, Huaping Zhong, Weijia Li, Jifeng Dai, and Conghui He. Influence selection for active learning.

Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations, 2019. 3 [51] Ziting Wen, Oscar Pizarro, and Stefan B. Williams. Feature alignment: Rethinking efficient active learning via proxy in the context of pre-trained models. Transactions on Machine Learning Research, 2024. 5 [52] Penghao Wu, Xiaosong Jia, Li Chen, Junchi Yan, Hongyang Li, and Yu Qiao. Trajectory-guided control prediction for end-to-end autonomous driving: A simple yet strong baseline. Advances in Neural Information Processing Systems, 35:6119–6132, 2022. 3 [53] Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36:69798–69818, 2023. 2 [54] Wanyun Xie, Francesco Tonin, and Volkan Cevher. Chameleon: A flexible data-mixing framework for language model pretraining and finetuning. In Forty-second International Conference on Machine Learning, 2025. 2, 5 [55] Chengyin Xu, Kaiyuan Chen, Xiao Li, Ke Shen, and Chenggang Li. Unveiling downstream performance scaling of llms: A clustering-based perspective. arXiv preprint arXiv:2502.17262, 2025. 4 [56] Jiasheng Ye, Peiju Liu, Tianxiang Sun, Jun Zhan, Yunhua Zhou, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. In The Thirteenth International Conference on Learning Representations, 2025. 1, 2 [57] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Metaworld: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pages 1094–1100. PMLR, 2020. 2 [58] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12104–12113, 2022. 1 [59] Xueying Zhan, Qingzhong Wang, Kuan-hao Huang, Haoyi Xiong, Dejing Dou, and Antoni B Chan. A comparative survey of deep active learning. arXiv preprint arXiv:2203.13450, 2022. 2

Scaling-Aware Data Selection for End-to-End Autonomous Driving Systems Supplementary Material 7. Experiment Protocols

7.2. Training details As already mentioned in the main body of the paper, we use the Hydra-MDP model [33] that won the NAVSIM benchmark in 2024 [14] by a significant margin. In addition to the imitation trajectory loss, the model distills the rulecompliance scores of each trajectory, obtained with prior simulations. We initialize the model’s encoder as pretrained VoVNetV2-99 backbone [31, 40]. In accordance with the training recipe of provided in the paper [33], we use Adam optimizer [29] without any weight decay and keep the learning rate fixed throughout the training. We set the per-GPU batch size to 20. In the Navtrain experiments, all runs are conducted using 8×A100 GPUs with a learning rate of 1e−4. Each experiment is repeated with three random seeds (0, 2025, 424242), and the reported results are averaged over these runs, with the standard deviation shown as a subscript. The base experiment and budgets up to 800 clips are trained for 60 epochs, while the 1,600- and 2,400-clip settings are trained for 50 and 45 epochs, respectively, to reduce compute cost. For the OpenScene experiments, the rule-compliance distillation losses are disabled due to their high computational overhead needed to run intensive simulations to calculate those scores. All runs use 16×A100 GPUs with a learning rate of 2e−4 and a fixed training length of 40 epochs. Experiments with 2,000, 4,000, and 8,000 clips are repeated with two seeds (0, 2025), while smaller-budget runs use three seeds (0, 2025, 424242) to ensure stability.

7.1. Dataset and Virtual Clip Creation We conduct experiments using the Navtrain [14] and trainval splits of OpenScene [11] as the combined training and pool datasets. OpenScene is a redistribution of the NuPlan dataset [19], subsampled to 2 Hz, and contains approximately 120 hours of driving data with dense annotations. The Navtrain split is curated within the NAVSIM framework [14] by filtering out trivial driving scenarios from the trainval split of OpenScene. In both experiments, evaluation is performed on navtest [14], a validation set curated analogously from the test split of OpenScene. The Navtrain and trainval splits consist of 1,192 and 1,250 individual driving sessions, respectively, with durations ranging from 30 seconds to 50 minutes. In addition to this large temporal variation, the total number of available driving sessions remains limited, and treating individual frames as independent samples to would be both unrealistic and inconsistent with the temporal structure of driving data. Hence, to have more samples to work with and to align our data handling with common industry practice [16] we segment each driving log into fixed-length virtual clips of 10 seconds (corresponding to 20 frames at 2 Hz). Below, we describe how we create virtual clips are created. Table 4. Train-pool clip counts for OpenScene and Navtrain

Openscene Navtrain

Train

Pool

1000 460

31539 4141

7.3. Details of the Baselines Random. For each budget B, Random selection is constructed from a single randomized ordering of the pool. Specifically, we shuffle all clips once using a fixed seed (seed = 42) and define the selected set for budget B as the first B clips in this ordering. This ensures that selections for larger budgets are strict supersets of those for smaller budgets.

We segment each driving log into fixed-length virtual clips of 10 seconds. Given the dataset’s sampling rate of 2 Hz, each virtual clip contains 20 frames. For each log, nonoverlapping clips are extracted sequentially from the start of the log, and any remaining portion shorter than 10 seconds is discarded. For example, a 23-second log yields two clips covering [0–10) s and [10–20) s, while the final 3 seconds are omitted. Following this procedure, the Navtrain split yields a total of 4,601 virtual clips after discarding 11,268 out of 103,288 frames (10.9%). For OpenScene, we obtain 32,539 virtual clips, with 12,086 out of 662,866 frames (1.8%) omitted due to incomplete segments. The train-pool clip counts are summarized in Table 4

Uncertainty [24]. We score each pool clip by the entropy of its model-predicted trajectory logits. Let zi denote the (pre-softmax) logits for sample xi , and let pi = softmax(zi ) be the corresponding probability distribution over candidate trajectories. The P uncertainty score is taken as the Shannon entropy Hi = − k pi,k log pi,k . The uncertainty score is calculated for each frame in the clip, and we simply take average of the frame uncertainty scores to aggregate it at the clip level. Clips with higher entropy correspond to more 1

Si for each domain, reflecting how informative or influential that domain is relative to all others. Finally, the mixture weight for domain i is obtained by normalizing these scores with a softmax, αi = softmax(Si ), and data are sampled from domains according to these mixture weights. We use the pretraining mode in our experiments, as we re-train the model from scratch for each budget and we set the ridge parameter as λ = 1. The pseudo-code is provided in Algorithm 4.

ambiguous or uncertain model predictions and are therefore preferred. To construct a budget-B selection, we compute Hi for every pool item once, rank all items by entropy in descending order, and pick the top B. We share the procedure in Algorithm 2. Algorithm 2 Entropy-Based Uncertainty Selection Require: Pool samples {xi }, model f (·), budget B Ensure: Selected set S of size B 1: Initialize S = ∅ 2: for each sample xi in the pool do 3: zi = f (xi ) ▷ trajectory logits 4: pi = softmax(z ) i P 5: Hi = − k pi,k log pi,k ▷ entropy score 6: end for 7: Rank all pool samples by Hi in descending order 8: S ← top-B samples under this ranking 9: return S

Algorithm 4 Chameleon Domain Weighting (Pretraining Mode) Require: Training clusters D = {D1 , . . . , Dk }, ridge parameter λ, embedding layer L, budget B Ensure: Selected set S of size B 1: Extract domain embeddings: P (L) 2: xi = |D1i | a∈Di hθ (a) for each domain Di ⊤ 3: Construct feature matrix X = [x⊤ 1 , . . . , xk ] ⊤ 4: Compute affinity matrix ΩD = XX 5: Compute KRLS scores Sλ (Di ) for each domain Di using ΩD 6: Compute domain weights:

Coreset [46]. We adopt the standard geometric Coreset selection procedure shown in Algorithm 3. Starting from an initial set of training indices s0 , the algorithm iteratively adds the pool element that is farthest under the chosen distance measure ∆(·, ·) from the current selected set. Specifically, we use Euclidean Distance. At each iteration, Coreset identifies the sample u ∈ spool that maximizes the minimum distance to the existing set s, and then augments s with u. This expansion continues until the total size reaches B + |s0 |, yielding the Coreset of size B from the pool.

−1 exp(Sλ (Di )) −1 exp(S j=1 λ (Dj ))

αiP T = Pk

8: Sample B points from domains according to mixture

weights {αiP T } 9: return S

8. More Results on the Experiments and Ablations Due to the space constraints in the main body of the paper, we present more results here.

Algorithm 3 Coreset Require: train sample indices s0 , budget B, pool indices spool 1: Initialize s = s0 2: repeat 3: u = arg maxi∈spool minj∈s ∆(xi , xj ) 4: s = s ∪ {u} 5: until |s| = B + |s0 | 6: return s

Experiments on Openscene. The full validation EPDMS and BRMR results for the Openscene experiments can be found in Table 6. The breakdown of the validation EPDMS subscores are shared in Table 9. The scaling curves obtained from different cities are shared in Figure 7. Experiments on Navtrain. The full validation EPDMS and BRMR results for the Navtrain experiments can be found in Table 7. The breakdown of the validation EPDMS subscores are shared in Table 10. We also provide the city distributions induced by different method at various budgets in Figure 9. The scaling curves obtained from different cities are shared in Figure 8. The scaling curves obtained from different cities are shared in Figure 8.

Chameleon [54]. Chameleon is a domain-mixture framework that relies on embeddings computed from the training domains. First, each cluster is embedded using representations from the base model’s feature space. For each cluster, sample embeddings are averaged which produces one embedding per cluster. A cluster–cluster affinity matrix is then constructed using a kernel function applied to pairs of domain embeddings. Given this affinity matrix, Chameleon applies kernel ridge regression (KRLS) to compute a score

Ablation with Caption-based Clustering. To generate the clip captions, we used the Qwen-2.5VL-32B-Instruct model with the following caption: “This is a 10 second 2

8 7 6 5

In fact, we have first attempted using ”sentencetransformers/all-mpnet-base-v2” model downloaded from Huggingface to obtain caption embeddings using a pretrained transformer. However, when we clustered the data in this embedding space, qualitative inspection revealed that the resulting groups lacked coherent driving characteristics. Hence, we experimented with clustering on the TF-IDF features which produced much more coherent clusters with directly interpretable feature space.

4 3 2 1 0 0

200

400 600 # Added clips

800

1000

Figure 7. Performance scalings of different cities for the OpenScene experiment. 2.5

City Distribution

1.5 1.0 boston (a = 2.50, =81.8) pittsburgh (a = 2.30, =87.4) singapore (a = 0.72, =65.3) vegas (a = 1.91, =100.0)

0.0 0

100

150 200 250 # Added clips

300

350

120

100

40 30

60 40

vegas

boston pittsburgh singapore

(a) 100 clips

400 300

Figure 8. Performance scalings of different cities for the main Navtrain experiment.

City Distribution

Number of Clips

500

100 50

400 300 200 100

vegas

boston pittsburgh singapore

City Distribution

vegas

boston pittsburgh singapore

(d) 800 clips

1600 clips

1750

1200

City Distribution

2400 clips Random Uncertainty Coreset Chameleon MOSAIC

1500

1000 Number of Clips

800 clips

600

150

800 600 400

1250 1000 750 500

200 0

boston pittsburgh singapore

City Distribution

400 clips

200

vegas

(b) 200 clips

250

long video of your student driving. The clip might include discontinuities, sudden changes in the driving environment. Describe the driving environment that your student is driving through and your student’s driving actions. Please describe the driving condition including the location, weather, road users, and their motions. During your description, there are several things to keep in mind. 1. Please pay attention only to the objects on the driving roads and ignore the background. 2. Ignore the brands of the vehicles. 3. Describe it if objects are partially occluded by others, or are in areas with different brightness such as under shades. Please provide a concise description in one paragraph with less than 150 words. Do not mention anything that you are certain does not exist! No statements about uncertain objects or events (no ’maybe’ or ’might’ or ’possibly’). All responses must be in English only!” On the generated clip captions, we extract TF–IDF features using the top 1,024 unigrams and bigrams after removing common English stop words. We then perform clustering in this TF–IDF space, forming six clusters. The dominant scene characteristics of each cluster are determined by

200 clips

Number of Clips

0.5

City Distribution 140

Number of Clips

EPDMS

2.0

100 clips

Number of Clips

EPDMS

their highest-weight unigrams and bigrams, as summarized in Table 3. We additionally conduct a qualitative assessment of the resulting groups and confirm that the clusters are coherent and semantically meaningful.

boston (a=8.80, =669.1) pittsburgh (a=61.59, =7151.8) singapore (a=6.12, =450.4) vegas (a=3.16, =471.9)

250 vegas

boston pittsburgh singapore

(e) 1600 clips

vegas

boston pittsburgh singapore

(f) 2400 clips

Figure 9. Overall caption describing all six subfigures.

100 clips

Caption Cluster Distribution

100

30 20

84 82

40 20

10 0

2 3 Cluster ID

(a) 100 clips

175

300

150 125 100 75

50 4

1600 clips

800

Number of Clips

600 400 200 0

2 3 Cluster ID

(e) 1600 clips

2400 clips

random entropy coreset chameleon pretrain MOSAIC

600 400

1000

1500 2000 2500 A100 GPU hours

3000

3500

with respect to the compute spent for each method. In particular, we provide the validation EPDMS vs. A100 GPU hours. The results are shared in Figure 11. As can be seen while MOSAIC is not the strongest method at small compute budgets, its initial scaling overhead amortizes over time, and at large budgets, the investment in scaling pays off, making MOSAIC the top-performing approach. More concretely, at the highest compute budget: MOSAIC reaches the top-baseline(Coreset in this setting) performance with 16% less compute, corresponding to 490 GPU hours saved; Compared to Random selection, MOSAIC requires 57% less compute, saving 1700 GPU hours to attain the same EPDMS. These results demonstrate that although MOSAIC pays an upfront cost for pilot scaling runs, the compute investment is recovered once we move into the large-budget regime.

200

500

Figure 11. Validation EPDMS vs. Compute Spent (GPU hours) for OpenScene experiments.

Caption Cluster Distribution 1000

Random Uncertainty Coreset Chameleon MOSAIC Base

800 clips

(d) 800 clips

Caption Cluster Distribution

800

150 100

2 3 Cluster ID

200

25 1

250

2 3 Cluster ID

Caption Cluster Distribution 350

Number of Clips

400 clips

200

(b) 200 clips

Caption Cluster Distribution

Number of Clips

EPDMS vs. Number of Clips

200 clips

Val. EPDMS

Caption Cluster Distribution

Number of Clips

2 3 Cluster ID

(f) 2400 clips

Figure 10. Overall caption describing all six subfigures.

10. Ranking with Alternative Cheap Signals

9. Details on the Scaling Fits and Compute Budget.

Since ranking is one of the key components of our framework, we also investigate cheaper alternatives to the EPDMS-based ranking signal to reduce the reliance on dense annotations such as bounding boxes. Specifically, we experiment with ranking clips according to (i) the trajectory imitation loss, (ii) the norm of the gradient vector induced by this loss, and (iii) the sensitivity of the model’s output to gradient perturbations. Instead of retraining the model with clips selected using the alternative signals and reporting the validation EPDMS, we measure the Kendall–Tau correlation coefficient between the rankings produced by each alternative signal and those produced by the EPDMS-based ranking. The results, shown in Figure 12, indicate that none of the inexpensive alternatives yield a ranking that correlates strongly with EPDMS.

MOSAIC requires an upfront compute investment to estimate cluster-specific scaling curves via pilot runs. To keep this cost tractable, we avoid full training from-scratch during the pilot experiments. Instead, we adopt a continualtraining approach: we resume training from the base model’s final epoch checkpoint and fine-tune on the combined dataset for a small number of epochs. For the OpenScene experiments, we train for 5 epochs after mining 200 and 400 clips from each cluster. For the Navtrain experiments, we train for 10 epochs after mining 100 and 200 clips in the two pilot runs. This procedure provides accurate scaling estimates while maintaining a manageable computational overhead. For the OpenScene experiments, we share the results 4

Kendall-Tau Rank Correlation imi. loss

1.00

0.61

-0.05

-0.17

grad. norm

0.61

1.00

-0.03

-0.30

sensitivity

-0.05

-0.03

1.00

0.13

epdms

-0.17

-0.30

0.13

1.00

imi. loss grad. norm sensitivity

teractions would likely be large, and the approximation would break down. In such pathological settings, explicitly modeling interaction terms would be necessary for optimal data selection.

1.0 0.8 0.6 0.4 0.2 0.0 0.2

epdms

Figure 12. Kendall-Tau correlation coefficients between EPDMS and cheap signals based rankings.

11. Approximation for Linear Separability and Error Analysis: Here, we formally express the performance improvement obtained from a data mixture ∆U (n1 , · · · , nM ) as follows: M X

∆Ui (ni ) +

i=1

∆Uij (ni , nj ) + H.O.T.

i̸=j

Here, the pairwise cross-cluster interaction term ∆Uij (ni , nj ) is defined as ∆Uij = Uij − Ui − Uj + U0 , where we use a lightweight notation for clarity: Uij = j i i U (Dtrain ∪ Dsel ∪ Dsel ), Ui = U (Dtrain ∪ Dsel ), and R U0 = U (Dtrain ), with U (·) ≡ U ({Gr (·)}r=1 ). In Equation 3, we retain only the first-order terms {∆Ui }M i=1 and omit interaction and higher-order terms. Importantly, we do not assume strict linear separability. Rather, we assume that first-order cluster-wise scaling captures the dominant variation in performance, while interaction terms contribute residual approximation error. To quantify the magnitude of the approximation error, we compare the estimated EPDMS calculated by summing cluster-wise scaling fits against the actual EPDMS obtained with the MOSAIC data mixtures. As shown in Table 5, the approximation overestimates performance by a modest margin (up to 1 EPMS), indicating that interaction terms are present but negligible in this setting. Table 5. Actual vs. estimated EPDMS (Navtrain, geolocation)

# Clips Actual Estimated

100 86.3 86.2

200 87.1 87.6

400 88.2 89.3

800 89.1 90.6

1600 90.2 91.1

2400 90.3 91.3

We also note that the discrepancy between the Actual and Estimated are accumulation of two factors: i) the crosscluster interactions, ii) extrapolation errors of the scaling fits. Hence, Table 5 should be interpreted as an upper bound on interaction effects rather than a pure estimate thereof. Also, as a contrasting example, if clusters were formed randomly and lacked semantic coherence, cross-cluster in5

Table 7. Navtrain validation EPDMS and BRMR results.

Table 6. Openscene validation EPDM and BRMR results.

Budget

Method

EPDMS

SRR

250

Random Uncertainty Coreset Chameleon MOSAIC

72.84±1.14 70.78±0.59 76.26±0.48 72.97±1.72 77.38±1.58

1.00 14.58 0.20 0.86 0.15

500

Random Uncertainty Coreset Chameleon MOSAIC

74.19±1.05 69.77±0.48 78.12±0.87 75.98±0.06 79.38±1.05

1.00 10.68 0.26 0.70 0.20

1000

Random Uncertainty Coreset Chameleon MOSAIC

75.84±0.9 71.12±0.38 80.46±0.02 79.08±0.74 81.68±0.52

1.00 NA 0.28 0.44 0.19

2000

Random Uncertainty Coreset Chameleon MOSAIC

78.39±0.12 69.94±1.4 81.37±0.13 81.35±0.39 82.78±0.41

1.00 NA 0.28 0.44 0.19

4000

Random Uncertainty Coreset Chameleon MOSAIC

80.38±0.55 73.46±0.19 83.63±0.36 82.92±0.13 84.25±0.14

1.00 NA 0.25 0.39 0.18

8000

Random Uncertainty Coreset Chameleon MOSAIC

82.32±0.54 75.63±0.19 84.49±0.02 84.43±0.01 85.02±0.18

1.00 NA 0.35 0.40 0.20

Budget

Method

EPDMS

SRR

100

Random Uncertainty Coreset Chameleon MOSAIC

84.66±0.6 84.5±0.48 85.29±0.47 84.57±0.18 86.29±0.43

1.00 1.47 0.53 1.07 0.30

200

Random Uncertainty Coreset Chameleon MOSAIC

85.45±0.09 84.84±0.54 86.12±0.31 86.04±0.3 87.04±0.37

1.00 1.50 0.60 0.80 0.32

400

Random Uncertainty Coreset Chameleon MOSAIC

86.69±0.2 86.07±0.75 87.09±0.29 87.04±0.6 88.21±0.03

1.00 2.00 0.79 0.82 0.38

800

Random Uncertainty Coreset Chameleon MOSAIC

87.41±0.37 86.69±0.34 88.48±0.12 88.33±0.23 89.1±0.12

1.00 1.69 0.62 0.64 0.33

1600

Random Uncertainty Coreset Chameleon MOSAIC

88.62±0.22 87.75±0.37 89.3±0.19 89.5±0.2 90.18±0.25

1.00 1.36 0.58 0.62 0.37

2400

Random Uncertainty Coreset Chameleon MOSAIC

89.42±0.03 88.95±0.15 89.75±0.02 90.05±0.08 90.31±0.03

1.00 1.00 0.76 0.64 0.43

Table 8. Navtrain validation EPDMS and BRMR results under caption-based clustering.

Budget

Method

EPDMS

SRR

100

Random Uncertainty Coreset Chameleon MOSAIC

84.66±0.6 84.5±0.48 85.29±0.47 84.35±0.47 85.85±0.41

1.00 1.47 0.53 1.30 0.37

200

Random Uncertainty Coreset Chameleon MOSAIC

85.45±0.09 84.84±0.54 86.12±0.31 85.39±0.02 86.75±0.17

1.00 1.50 0.60 2.88 0.40

400

Random Uncertainty Coreset Chameleon MOSAIC

86.69±0.2 86.07±0.75 87.09±0.29 84.95±0.45 88.11±0.05

1.00 2.00 0.79 3.32 0.48

800

Random Uncertainty Coreset Chameleon MOSAIC

87.41±0.37 86.69±0.34 88.48±0.12 86.1±0.55 88.99±0.09

1.00 1.69 0.62 2.68 0.37

1600

Random Uncertainty Coreset Chameleon MOSAIC

88.62±0.22 87.75±0.37 89.3±0.19 86.99±0.57 89.98±0.13

1.00 1.36 0.58 1.50 0.39

2400

Random Uncertainty Coreset Chameleon MOSAIC

89.42±0.03 88.95±0.15 89.75±0.02 87.62±0.28 90.37±0.2

1.00 1.00 0.76 1.00 0.48

Table 9. Breakdown of the nine EPDMS rule-compliance metrics for the base model and the models trained with data selected by various strategies at all budgets, shown for the OpenScene experiment. Setting

DAC

DDC

TLC

TTC

EPDMS

Base

94.05

83.9

96.28

99.6

85.96

92.95

93.26

98.25

81.88

72.0

250

Random Uncertainty Coreset Chameleon MOSAIC

94.27±0.60 93.97±0.44 95.11±0.47 94.02±1.25 94.89±0.74

84.63±1.46 82.49±0.30 87.66±0.61 84.30±1.18 88.76±1.17

97.38±0.23 96.78±0.44 98.38±0.21 97.48±0.71 98.54±0.43

99.66±0.04 99.66±0.02 99.67±0.04 99.58±0.06 99.61±0.04

85.18±1.02 85.18±0.81 86.09±1.13 87.48±1.41 86.50±1.03

93.23±0.64 92.98±0.42 94.08±0.84 92.69±1.23 93.93±0.88

93.33±0.56 93.18±0.66 94.47±0.20 93.43±0.04 94.88±0.14

98.26±0.01 98.23±0.08 98.31±0.05 98.26±0.01 98.26±0.03

82.66±0.76 82.15±0.27 83.38±0.74 83.15±1.80 83.77±0.67

72.84±1.14 70.78±0.59 76.26±0.48 72.97±1.72 77.38±1.58

500

Random Uncertainty Coreset Chameleon MOSAIC

94.65±0.21 93.32±0.47 95.56±0.78 95.00±0.58 95.57±1.05

85.72±0.88 82.26±0.40 88.96±0.57 87.11±0.09 90.54±0.45

97.87±0.44 96.09±0.52 98.95±0.09 98.16±0.02 98.83±0.29

99.64±0.06 99.60±0.08 99.71±0.07 99.67±0.16 99.67±0.09

85.53±0.22 84.51±0.43 86.21±0.99 86.67±2.25 86.08±1.79

93.51±0.32 92.23±0.73 94.69±0.79 94.22±0.45 94.85±1.23

93.73±0.24 92.38±0.56 95.14±0.13 94.20±0.44 95.68±0.27

98.27±0.05 98.30±0.01 98.31±0.03 98.30±0.01 98.25±0.04

83.26±0.14 82.85±1.07 84.24±0.34 83.69±0.24 83.80±0.16

74.19±1.05 69.77±0.48 78.12±0.87 75.98±0.06 79.38±1.05

1000

Random Uncertainty Coreset Chameleon MOSAIC

95.21±0.58 94.04±0.70 95.93±0.24 95.89±0.19 96.00±0.22

87.15±1.44 83.77±0.02 91.05±0.26 89.57±0.68 92.20±0.48

98.26±0.39 96.96±0.08 99.28±0.11 98.94±0.16 99.33±0.07

99.72±0.07 99.70±0.08 99.71±0.04 99.71±0.07 99.67±0.05

85.56±0.96 83.11±0.66 86.39±0.48 86.39±0.51 86.63±0.41

94.35±0.60 93.21±1.00 95.01±0.21 95.06±0.26 95.24±0.24

94.50±0.66 92.87±0.14 95.75±0.08 95.44±0.27 96.17±0.22

98.31±0.03 98.32±0.02 98.28±0.03 98.29±0.01 98.28±0.03

82.50±0.52 81.91±0.75 84.58±0.42 84.23±0.74 84.33±0.30

75.84±0.90 71.12±0.38 80.46±0.02 79.08±0.74 81.68±0.52

2000

Random Uncertainty Coreset Chameleon MOSAIC

95.58±0.54 93.14±0.79 95.89±0.22 96.38±0.25 96.90±0.38

89.26±0.64 82.66±1.12 91.77±0.14 91.31±0.26 92.29±0.36

98.67±0.18 96.64±0.64 99.44±0.06 99.15±0.03 99.48±0.05

99.70±0.12 99.53±0.09 99.66±0.04 99.71±0.05 99.73±0.01

86.44±0.42 84.52±1.23 87.39±0.05 86.55±0.40 86.61±0.73

94.88±0.61 92.19±1.22 94.98±0.18 95.60±0.29 96.16±0.26

95.26±0.18 93.22±0.29 95.99±0.47 95.99±0.15 96.34±0.06

98.30±0.00 98.28±0.03 98.29±0.00 98.34±0.01 98.28±0.05

83.96±0.96 80.98±1.30 85.55±0.19 85.04±0.15 84.69±0.01

78.39±0.12 69.94±1.40 81.37±0.13 81.35±0.39 82.78±0.41

4000

Random Uncertainty Coreset Chameleon MOSAIC

96.32±0.59 94.67±0.28 97.11±0.18 96.76±0.24 96.97±0.32

90.53±0.06 85.11±0.51 92.93±0.60 92.32±0.02 93.59±0.11

99.06±0.07 97.15±0.54 99.44±0.06 99.51±0.01 99.59±0.04

99.79±0.05 99.71±0.04 99.82±0.02 99.77±0.01 99.80±0.01

86.36±0.48 84.26±0.69 86.65±0.55 86.98±0.17 87.14±0.98

95.66±0.52 93.72±0.40 96.42±0.19 95.91±0.31 96.18±0.45

95.68±0.09 93.26±0.09 96.66±0.30 96.49±0.12 96.62±0.08

98.30±0.01 98.28±0.02 98.16±0.12 98.32±0.01 98.28±0.01

84.46±0.14 81.34±1.06 85.10±0.06 85.51±0.11 85.06±0.34

80.38±0.55 73.46±0.19 83.63±0.36 82.92±0.13 84.25±0.14

8000

Random Uncertainty Coreset Chameleon MOSAIC

96.79±0.21 95.62±0.38 97.39±0.15 97.33±0.39 97.55±0.13

91.88±0.34 86.48±0.06 93.51±0.18 93.36±0.14 93.84±0.00

99.23±0.11 97.62±0.01 99.55±0.07 99.61±0.01 99.53±0.18

99.79±0.03 99.71±0.02 99.81±0.03 99.82±0.01 99.84±0.03

87.19±0.05 84.92±0.25 87.07±0.39 87.34±0.61 87.19±0.24

95.93±0.15 94.80±0.28 96.64±0.12 96.42±0.50 96.79±0.07

96.19±0.10 94.34±0.27 96.78±0.06 96.90±0.17 97.10±0.07

98.28±0.03 98.32±0.02 98.28±0.03 98.29±0.02 98.29±0.02

84.97±0.19 81.62±0.09 85.51±0.15 85.51±0.12 85.25±0.22

82.32±0.54 75.63±0.19 84.49±0.02 84.43±0.00 85.02±0.18

Table 10. Breakdown of the nine EPDMS rule-compliance metrics for the base model and the models trained with data selected by various strategies at all budgets, shown for the Navtrain experiment. Setting

DAC

DDC

TLC

TTC

EPDMS

Base

95.3

95.94

99.09

99.6

88.09

94.55

94.49

98.25

82.39

83.97

100

Random Uncertainty Coreset Chameleon MOSAIC

95.43±0.84 95.68±0.33 95.63±0.41 95.14±0.20 96.75±0.28

96.41±0.20 96.23±0.38 96.88±0.33 96.50±0.19 97.06±0.09

98.98±0.07 98.91±0.12 99.13±0.09 99.17±0.02 99.03±0.03

99.54±0.15 99.51±0.06 99.56±0.03 99.53±0.02 99.60±0.03

88.68±0.63 88.21±0.22 88.39±0.65 88.80±0.18 87.74±0.28

94.69±0.89 94.77±0.36 94.75±0.51 94.35±0.11 96.09±0.35

94.82±0.34 94.88±0.13 94.97±0.34 95.10±0.12 94.92±0.32

98.27±0.04 98.27±0.04 98.25±0.03 98.25±0.04 98.27±0.02

82.81±0.64 83.50±0.32 82.94±0.20 82.93±0.75 82.80±0.62

84.66±0.60 84.50±0.48 85.29±0.47 84.57±0.18 86.29±0.43

200

Random Uncertainty Coreset Chameleon MOSAIC

95.90±0.35 95.61±0.66 96.19±0.49 96.12±0.52 96.83±0.31

96.58±0.23 96.53±0.38 97.05±0.11 96.76±0.34 97.51±0.18

99.11±0.08 98.96±0.18 99.13±0.06 99.33±0.18 99.24±0.06

99.65±0.02 99.57±0.09 99.60±0.04 99.60±0.10 99.61±0.01

88.75±0.19 88.51±0.11 88.68±0.13 88.74±0.54 88.20±0.13

95.08±0.33 94.74±0.59 95.39±0.44 95.38±0.71 96.16±0.29

95.14±0.29 94.88±0.28 95.17±0.11 95.41±0.19 95.36±0.18

98.27±0.04 98.28±0.03 98.29±0.01 98.30±0.02 98.26±0.02

83.14±0.11 83.34±0.34 83.45±0.55 83.73±0.13 82.63±0.35

85.45±0.09 84.84±0.54 86.12±0.31 86.04±0.30 87.04±0.37

400

Random Uncertainty Coreset Chameleon MOSAIC

96.71±0.25 96.39±0.60 96.73±0.24 96.33±0.36 97.75±0.08

96.91±0.20 96.97±0.38 97.27±0.17 97.55±0.20 97.79±0.11

99.18±0.09 99.00±0.08 99.36±0.02 99.37±0.07 99.42±0.06

99.71±0.01 99.65±0.01 99.64±0.02 99.63±0.03 99.72±0.04

88.75±0.15 88.22±0.42 88.80±0.11 88.97±0.42 87.62±0.11

96.02±0.23 95.55±0.66 95.95±0.26 95.59±0.38 97.17±0.09

95.76±0.16 94.98±0.22 95.81±0.23 95.87±0.20 95.54±0.08

98.30±0.01 98.25±0.02 98.29±0.03 98.30±0.01 98.24±0.01

82.96±0.10 83.64±0.16 83.48±0.46 83.10±0.35 82.81±0.27

86.69±0.20 86.07±0.75 87.09±0.29 87.04±0.60 88.21±0.03

800

Random Uncertainty Coreset Chameleon MOSAIC

96.94±0.35 96.98±0.40 97.21±0.12 97.07±0.17 97.65±0.14

97.15±0.36 96.88±0.16 98.06±0.23 97.97±0.18 98.33±0.06

99.35±0.12 99.13±0.11 99.49±0.06 99.48±0.08 99.54±0.05

99.69±0.05 99.69±0.07 99.67±0.05 99.68±0.03 99.73±0.05

89.16±0.06 88.31±0.45 88.84±0.08 88.99±0.50 88.68±0.44

96.22±0.41 96.22±0.32 96.62±0.13 96.57±0.19 97.03±0.16

96.28±0.45 95.42±0.15 96.22±0.19 96.38±0.18 96.19±0.14

98.29±0.03 98.28±0.03 98.30±0.03 98.29±0.03 98.26±0.02

83.63±0.02 82.95±0.31 83.67±0.27 83.28±0.22 82.93±0.48

87.41±0.37 86.69±0.34 88.48±0.12 88.33±0.23 89.10±0.12

1600

Random Uncertainty Coreset Chameleon MOSAIC

97.17±0.07 96.92±0.38 97.50±0.10 97.43±0.22 98.04±0.24

98.19±0.43 97.66±0.08 98.31±0.34 98.46±0.17 98.61±0.32

99.42±0.05 99.22±0.10 99.59±0.03 99.60±0.05 99.63±0.06

99.69±0.02 99.77±0.02 99.72±0.05 99.75±0.03 99.73±0.02

89.36±0.12 89.02±0.28 89.27±0.21 89.60±0.19 89.28±0.19

96.50±0.14 96.24±0.40 96.86±0.07 96.83±0.30 97.50±0.32

96.45±0.25 96.10±0.07 96.75±0.22 96.89±0.07 97.07±0.06

98.31±0.03 98.30±0.01 98.30±0.03 98.30±0.03 98.28±0.04

83.17±0.76 82.92±0.38 83.88±0.50 83.87±0.34 83.70±0.41

88.62±0.22 87.75±0.37 89.30±0.19 89.50±0.20 90.18±0.25

2400

Random Uncertainty Coreset Chameleon MOSAIC

97.56±0.11 97.62±0.21 97.59±0.02 97.60±0.16 98.02±0.12

98.23±0.12 98.10±0.17 98.53±0.06 98.71±0.06 98.69±0.05

99.56±0.04 99.36±0.10 99.57±0.04 99.63±0.04 99.66±0.07

99.74±0.00 99.78±0.02 99.67±0.04 99.77±0.01 99.80±0.06

89.57±0.08 89.19±0.15 89.79±0.24 89.85±0.06 89.19±0.38

96.97±0.09 97.07±0.22 97.17±0.13 97.18±0.14 97.58±0.10

96.95±0.07 96.65±0.27 97.17±0.10 97.20±0.09 97.22±0.09

98.30±0.01 98.29±0.05 98.31±0.02 98.28±0.01 98.31±0.00

83.95±0.31 82.53±0.48 83.77±0.54 83.61±0.51 83.56±0.07

89.42±0.03 88.95±0.15 89.75±0.02 90.05±0.08 90.31±0.03

Table 11. Breakdown of the nine EPDMS rule-compliance metrics for the base model and the models trained with data selected by various strategies at all budgets, shown for the Navtrain experiment when the clustering is performed on the clip captions. Setting

DAC

DDC

TLC

TTC

EPDMS

Base

95.3

95.94

99.09

99.6

88.09

94.55

94.49

98.25

82.39

83.97

100

Random Uncertainty Coreset Chameleon MOSAIC

95.43±0.84 95.68±0.33 95.63±0.41 95.43±0.61 96.53±0.31

96.41±0.20 96.23±0.38 96.88±0.33 96.14±0.02 96.91±0.30

98.98±0.07 98.91±0.12 99.13±0.09 98.94±0.12 99.03±0.12

99.54±0.15 99.51±0.06 99.56±0.03 99.56±0.06 99.54±0.06

88.68±0.63 88.21±0.22 88.39±0.65 88.45±0.26 87.62±0.21

94.69±0.89 94.77±0.36 94.75±0.51 94.52±0.59 95.80±0.32

94.82±0.34 94.88±0.13 94.97±0.34 94.82±0.07 94.85±0.34

98.27±0.04 98.27±0.04 98.25±0.03 98.28±0.02 98.24±0.02

82.81±0.64 83.50±0.32 82.94±0.20 83.27±0.49 82.66±0.66

84.66±0.60 84.50±0.48 85.29±0.47 84.35±0.47 85.85±0.41

200

Random Uncertainty Coreset Chameleon MOSAIC

95.90±0.35 95.61±0.66 96.19±0.49 96.20±0.11 97.07±0.30

96.58±0.23 96.53±0.38 97.05±0.11 96.58±0.25 97.19±0.25

99.11±0.08 98.96±0.18 99.13±0.06 98.97±0.21 99.08±0.06

99.65±0.02 99.57±0.09 99.60±0.04 99.63±0.01 99.64±0.03

88.75±0.19 88.51±0.11 88.68±0.13 88.02±0.19 87.79±0.46

95.08±0.33 94.74±0.59 95.39±0.44 95.42±0.24 96.28±0.36

95.14±0.29 94.88±0.28 95.17±0.11 94.88±0.23 94.99±0.29

98.27±0.04 98.28±0.03 98.29±0.01 98.30±0.00 98.25±0.01

83.14±0.11 83.34±0.34 83.45±0.55 82.88±1.50 82.92±0.55

85.45±0.09 84.84±0.54 86.12±0.31 85.39±0.02 86.75±0.17

400

Random Uncertainty Coreset Chameleon MOSAIC

96.71±0.25 96.39±0.60 96.73±0.24 95.61±0.32 97.36±0.10

96.91±0.20 96.97±0.38 97.27±0.17 96.50±0.25 97.91±0.05

99.18±0.09 99.00±0.08 99.36±0.02 99.04±0.11 99.33±0.10

99.71±0.01 99.65±0.01 99.64±0.02 99.59±0.06 99.66±0.02

88.75±0.15 88.22±0.42 88.80±0.11 88.64±0.56 88.37±0.41

96.02±0.23 95.55±0.66 95.95±0.26 94.84±0.42 96.68±0.11

95.76±0.16 94.98±0.22 95.81±0.23 94.90±0.13 95.43±0.34

98.30±0.01 98.25±0.02 98.29±0.03 98.29±0.00 98.27±0.03

82.96±0.10 83.64±0.16 83.48±0.46 82.73±0.34 83.00±1.38

86.69±0.20 86.07±0.75 87.09±0.29 84.95±0.45 88.11±0.05

800

Random Uncertainty Coreset Chameleon MOSAIC

96.94±0.35 96.98±0.40 97.21±0.12 96.26±0.49 97.92±0.09

97.15±0.36 96.88±0.16 98.06±0.23 96.83±0.47 98.08±0.17

99.35±0.12 99.13±0.11 99.49±0.06 99.10±0.05 99.50±0.05

99.69±0.05 99.69±0.07 99.67±0.05 99.71±0.03 99.73±0.01

89.16±0.06 88.31±0.45 88.84±0.08 88.89±0.48 88.20±0.25

96.22±0.41 96.22±0.32 96.62±0.13 95.53±0.46 97.35±0.14

96.28±0.45 95.42±0.15 96.22±0.19 95.64±0.21 96.12±0.22

98.29±0.03 98.28±0.03 98.30±0.03 98.28±0.01 98.25±0.04

83.63±0.02 82.95±0.31 83.67±0.27 82.97±0.19 83.00±0.50

87.41±0.37 86.69±0.34 88.48±0.12 86.10±0.55 88.99±0.09

1600

Random Uncertainty Coreset Chameleon MOSAIC

97.17±0.07 96.92±0.38 97.50±0.10 96.61±0.30 97.98±0.05

98.19±0.43 97.66±0.08 98.31±0.34 97.22±0.26 98.59±0.12

99.42±0.05 99.22±0.10 99.59±0.03 99.25±0.11 99.60±0.03

99.69±0.02 99.77±0.02 99.72±0.05 99.72±0.06 99.76±0.01

89.36±0.12 89.02±0.28 89.27±0.21 89.05±0.24 89.03±0.24

96.50±0.14 96.24±0.40 96.86±0.07 96.02±0.39 97.49±0.11

96.45±0.25 96.10±0.07 96.75±0.22 95.90±0.18 97.02±0.25

98.31±0.03 98.30±0.01 98.30±0.03 98.32±0.01 98.27±0.03

83.17±0.76 82.92±0.38 83.88±0.50 82.35±0.27 83.61±0.31

88.62±0.22 87.75±0.37 89.30±0.19 86.99±0.57 89.98±0.13

2400

Random Uncertainty Coreset Chameleon MOSAIC

97.56±0.11 97.62±0.21 97.59±0.02 96.93±0.30 98.03±0.28

98.23±0.12 98.10±0.17 98.53±0.06 97.50±0.25 98.78±0.15

99.56±0.04 99.36±0.10 99.57±0.04 99.37±0.05 99.62±0.02

99.74±0.00 99.78±0.02 99.67±0.04 99.75±0.03 99.79±0.07

89.57±0.08 89.19±0.15 89.79±0.24 88.97±0.44 89.26±0.45

96.97±0.09 97.07±0.22 97.17±0.13 96.37±0.24 97.59±0.27

96.95±0.07 96.65±0.27 97.17±0.10 96.22±0.18 96.97±0.12

98.30±0.01 98.29±0.05 98.31±0.02 98.31±0.00 98.33±0.03

83.95±0.31 82.53±0.48 83.77±0.54 82.39±0.45 84.02±0.10

89.42±0.03 88.95±0.15 89.75±0.02 87.62±0.28 90.37±0.20

Record · ID 2601 · SHA-256 99bd589b089f1c61
Conceptio Open Knowledge Archive — every document is proof-bundled with source, license, and retrieval metadata.