Preprint. Under review.
Spend Less, Fit Better: Budget-Efficient Scaling Law Fitting via Active Experiment Selection Sijie Li1 ,∗ Shanda Li2 ,∗ Haowei Lin1 , Weiwei Sun2 , Ameet Talwalkar2,3 , Yiming Yang2 1 Peking University
2 Carnegie Mellon University
3 Datadog
arXiv:2604.22753v1 [cs.LG] 24 Apr 2026
Abstract Scaling laws are used to plan multi-million-dollar training runs, but fitting those laws can itself cost millions. In modern large-scale workflows, assembling a sufficiently informative set of pilot experiments is already a major budget-allocation problem rather than a routine preprocessing step. We formulate scaling-law fitting as budget-aware sequential experimental design: given a finite pool of runnable experiments with heterogeneous costs, choose which runs to execute so as to maximize extrapolation accuracy in a high-cost target region. We then propose an uncertainty-aware method for sequentially allocating experimental budget toward the runs most useful for target-region extrapolation. Across a diverse benchmark of scaling-law tasks, our method consistently outperforms classical designbased baselines, and often approaches the performance of fitting on the full experimental set while using only about 10% of the total training budget. Our code is available at https://github.com/PlanarG/active-sl.
1
Introduction
Scaling laws have become a central tool for analyzing and planning large-scale language model training (Hestness et al., 2017; Kaplan et al., 2020; Gao et al., 2023; Li et al., 2025b). By fitting a parametric relationship between performance and variables such as model size, data size, and compute, practitioners can use a limited set of pilot runs to predict behavior at much larger scales and allocate future training budget accordingly (Yang et al., 2021; Li et al., 2025a; Xu et al., 2026). This paradigm has already shaped influential decisions in practice: For example, Chinchilla-style compute-optimal training was derived from an extensive empirical study spanning more than 400 training runs across a wide range of scales (Hoffmann et al., 2022). Despite their practical importance, scaling laws remain expensive to fit and heavily reliant on manual experiment design. In typical workflows, researchers hand-select experimental configurations, run many pilot trainings, and then fit a parametric law to the resulting observations. At industrial scale, the pilot runs needed just to fit a scaling law can themselves consume a massive budget (Porian et al., 2025; Hägele et al., 2024), with the full fit-and-verify pipeline already reaching the million-dollar scale before any deployment-scale training is committed. Accurate scaling-law fitting is therefore not only a modeling problem, but also a problem of budget allocation. This challenge is becoming more acute as scaling-law analysis expands beyond classical dense pretraining. Recent work has extended scaling laws to a much broader range of settings, including vocabulary design, data mixing, sparsity, mixture-of-experts architectures, and inference-time scaling (Muennighoff et al., 2023; Caballero et al., 2023; Ludziejewski et al., 2024). As these settings become more diverse and expensive, manually selecting pilot runs becomes increasingly inefficient. This raises a natural question: Given a pool of runnable experiments and a limited budget, how should we select experiments to ensure that the fitted scaling law extrapolates accurately in the target region? ∗ Equal contribution.
1
Preprint. Under review.
0.01%
2.119
2.111
2.092
2.095
2.110
2.140
2.109
2.096
2.090
2.088
2.086
2.095
2.106
2.145
0.002762
2.173
2.123
2.092
2.083
2.083
2.081
2.088
2.095
2.106
2.138
2.250
0.001953
2.155
2.109
2.083
2.078
2.077
2.081
2.084
2.098
2.113
2.140
0.001381
2.137
2.100
2.080
2.078
1% 2.082 2.076
2.225
2.090
2.097
2.110
2.142
0.000977
2.123
2.092
2.080
2.081
2.082
2.083
2.095
2.103
2.112
2.137
2.200
0.000691
2.114
2.090
2.084
2.086
2.088
2.090
2.098
2.108
2.118
2.152
2.175
0.000488
2.109
2.092
2.090
2.093
2.097
2.101
2.107
2.117
2.128
2.165
0.000345
2.107
2.095
2.096
2.102
2.106
2.112
2.120
2.130
2.142
2.183
0.5%
2.161
2.209
2.154
2.168
2.187
2.246
2.100
0.000122
2.127
2.126
2.134
2.142
2.150
2.161
2.178
2.198
2.221
2.293
512 128 32 1B 100M
10B
N
96
1019
1B
100B
D
1018
40
20
14
10
70
51
38
8
25
12
64
48
2.146
2.140
72
2.135
2.132
24
2.125
2.125
4
2.119
2.119
2
2.112
2.112
4
2.106
2.116
6
2.102
0.000173
Batch Size
2048
2.150 2.125
0.000244
1020
2.275
Test LM Loss
0.1%
2.187
FLOPs
2.124
2.224
Learning Rate
2.172
0.003906
Batch Size
0.005524
Figure 1: Our method identifies the extrapolation optimum using only a small fraction of the original scaling-law fitting budget. On lr&bsz, the predicted optimum over learning rate and batch size for a 1B-parameter LLM trained on 100B tokens reaches the low-loss region within 1% of the budget, as shown by the trajectory on the test-loss heatmap (left). The selected training configurations used for fitting are shown on the right in a 3D view of the design space, colored by training cost (6ND), illustrating that accurate extrapolation can be achieved from a sparse, low-cost subset of the full configuration space.
In this paper, we formulate scaling-law fitting as a problem of budget-aware sequential experimental design. Rather than assuming that fitting data are given in advance, we treat each candidate experiment as a costly query and ask how to allocate a limited budget over a finite pool of runnable configurations. The objective is not merely to fit the observed points well, but to maximize predictive accuracy in a held-out target region lying in the large-scale, high-cost regime. This formulation more directly reflects the practical use of scaling laws, where the ultimate goal is to choose a few expensive final configurations rather than to uniformly fit all pilot runs. Our contributions are twofold. First, we formulate the problem and present the first systematic study of scaling-law fitting in the low-budget regime, where accurate extrapolation must be recovered from only a small fraction of the training runs used in standard scaling-law analyses. To support this study, we conduct a comprehensive evaluation spanning diverse scaling scenarios, heterogeneous law families, and task-specific cost structures, enabling controlled comparison of point-selection strategies under budget constraints. Second, we propose a sequential, uncertainty-aware design method that explicitly models ambiguity over scaling-law parameters and selects new experiments according to their expected value for reducing target-region prediction error. Empirically, our method is highly effective: as shown in Figure 1, it approaches the extrapolation ground-truth optimum using only a small fraction of the original fitting FLOPs, reaching the low-loss region within 1% budget on Step Law (Learning Rate and Batch Size Scaling Law) (Li et al., 2025a).
2
Related Works
Scaling Laws. Scaling laws have transformed the design and optimization of large-scale AI systems by revealing predictable relationships among model size, data volume, and compute budget (Kaplan et al., 2020; Hoffmann et al., 2022; Hu et al., 2024). Recent work has extended this paradigm far beyond its original setting, covering model architectures (Fedus et al., 2022), data scaling (Ye et al., 2025; Que et al., 2024), post-training behavior (Hernandez et al., 2021; Lin et al., 2024), multimodal regimes (Radford et al., 2021; Zhai et al., 2022), and deployment-time settings (Chen et al., 2024; 2025; Wu et al., 2025). Yet fitting scaling laws in practice remains costly and heavily manual: the final fit can be highly sensitive to the choice of law family, initialization, and, crucially, the strategy used to collect training runs (Li et al., 2025c). Our method does not rely on manually designed experimental points by automatically selecting new runs based on the current fitting state and the target region where accurate prediction matters most. 2
Preprint. Under review.
Optimal experiment design for nonlinear models. Classical optimal experiment design (OED) studies how to place experiments to estimate model parameters or derived quantities efficiently, with criteria such as D-optimality and A-optimality defined through the Fisher information matrix (Silvey, 2013; Kiefer, 1959; Gilmour & Trinca, 2013). For nonlinear models, these criteria typically depend on unknown parameters, leading to a large literature on locally optimal design for nonlinear and generalized linear models (White, 1973; Khuri et al., 2006; Atkinson & Woods, 2015; Yang et al., 2013). However, this line of work is primarily local and parameter-estimation-oriented, and typically does not consider heterogeneous experiment costs. Bayesian optimal experimental design addresses parameter uncertainty by optimizing expected utility under a posterior distribution (Huan & Marzouk, 2013; Zhong et al., 2026), which is appealing for scaling-law fitting where the objective is highly nonlinear and limited observations may support multiple plausible fits. However, existing Bayesian OED methods do not directly address our setting, in which candidate experiments are discrete training runs with heterogeneous compute costs and must be selected sequentially under a strict budget. We therefore study a cost-aware sequential design problem tailored to scaling-law fitting, where experiments are prioritized by their expected predictive benefit relative to their cost.
3
Problem Setup
We study the scaling-law fitting problem in a budget-constrained sequential setting. We assume that the underlying performance trend is described by a parametric scaling law y = f ( x; θ ), where x ∈ X denotes the modeling configuration, e.g., model size, token count, or other training- or inference-related hyperparameters; y ∈ R denotes the prediction target of the scaling law, e.g., training loss; and θ ∈ R p denotes the parameter to be estimated from the experiments. Running an experiment under configuration x incurs a nonnegative cost c( x ) and reveals an outcome y. At a high level, we aim to run a set of experiments under a cost constraint so as to obtain data for fitting an accurate scaling law. In practice, experiment selection is performed over a set of predefined runnable configurations. We therefore consider a candidate pool Xcand = { x1 , . . . , x N }, where each candidate xi has an associated cost ci = c( xi ). At each round, the learner selects one previously unobserved candidate, pays its cost, observes its outcome, and adds the resulting pair to the current dataset. After t rounds, the accumulated observations form a dataset Dt , and the total cost of all selected experiments must remain within a budget C. The ultimate goal is not merely to fit the observed points well, but to learn a scaling law that extrapolates accurately in a target region Xtar . This target region typically contains the larger-scale configurations that matter most for downstream planning, but are too expensive to explore exhaustively, whereas Xcand typically contains more affordable small-scale runs. Our objective is therefore to design a sequential experiment-selection strategy that uses the available budget as efficiently as possible, so that the fitted scaling law is accurate where it ultimately matters.
4
Budget-Aware Sequential Scaling-Law Design
We now describe our sequential design strategy for scaling-law fitting under a budget constraint. At round t, given the current dataset Dt , our goal is to select the next experiment x ∈ Xcand that most improves predictive accuracy on the target region Xtar . To make the design objective concrete, we assume the standard observation model y = f ( x; θ ∗ ) + ε with some unknown ground-truth parameter θ ∗ and noise ε ∼ N (0, σ2 ). 4.1
A Target-Aware Uncertainty Objective
We would like the utility of an experiment to reflect our downstream goal: improving prediction accuracy on the target region Xtar . In our setting, uncertainty comes from two sources: local uncertainty within a plausible fit, and disagreement across multiple plausible 3
Preprint. Under review.
fits that extrapolate differently on Xtar . We call each locally optimal fit of the scaling law a basin. Given the observations Dt collected so far, we approximate the posterior: K
p ( θ | D t ) ≈ ∑ w k q k ( θ ),
q k ( θ ) = N ( θ k , Σ k ),
k =1
where each component represents one plausible local basin, with representative parameter θk , local covariance Σk , and mixture weight wk ≥ 0 satisfying ∑kK=1 wk = 1. In our implementation, θk , Σk , and wk are estimated from local refits and local posterior approximations; details are given in Appendix C. To measure uncertainty where extrapolation matters, define the target-region prediction map F (θ ) = ( f ( x; θ )) x∈Xtar ∈ R|Xtar | . Let fˆk = F (θk ),
K
f¯ = Eθ ∼ p(θ |Dt ) [ F (θ )] ≈ ∑ wk fˆk . k =1
We use the target-region mean squared prediction error h i 1 MSPEtar = Eθ ∼ p(θ |Dt ) ∥ F (θ ) − f¯∥22 |Xtar | as our uncertainty objective. Under a local Gaussian approximation within each basin, this quantity decomposes into MSPEtar = Vintra + Vinter , where K K 1 1 ⊤ ) , V = Vintra = w tr ( J Σ J inter ∑ wk ∥ fˆk − f¯∥22 . k k k k |Xtar | k∑ |X | tar =1 k =1 Here Jk ∈ R|Xtar |× p is the Jacobian of F (θ ) evaluated at θk . Intuitively, the first term quantifies local predictive uncertainty within each basin, while the second quantifies disagreement across basins in the target region. 4.2
Scoring Candidate Experiments
For a candidate experiment x ∈ Xcand , we define its utility as the expected reduction in target-region uncertainty after observing its outcome: + ∆MSPEtar ( x ) = MSPEtar − Ey| x,Dt [MSPEtar ( x, y)], + where MSPEtar ( x, y) denotes the updated target-region MSPE after augmenting Dt with ( x, y). Using the decomposition above, we write
∆MSPEtar ( x ) = ∆Vintra ( x ) + ∆Vinter ( x ), with + ∆Vintra ( x ) = Vintra − Ey| x,Dt [Vintra ( x, y)],
+ ∆Vinter ( x ) = Vinter − Ey| x,Dt [Vinter ( x, y)].
The first term favors candidates that reduce within-basin predictive variance on Xtar , while the second favors candidates that disambiguate basins with different extrapolations. To account for heterogeneous experiment costs, we rank candidates by the cost-aware score S( x ) =
∆Vintra ( x ) + ∆Vinter ( x ) , c( x )α
where α ≥ 0 controls the strength of cost penalization. 4.3
Computing Intra- and Inter-Basin Utility
We approximate both utility terms by locally linearizing the predictor within each basin while preserving the multimodal structure across basins. Full derivations are deferred to Appendix D. 4
Preprint. Under review.
Algorithm 1 Budget-aware sequential design Require: Initial dataset D0 , candidate pool Xcand , target region Xtar , cost function c(·), cost exponent α 1: for t = 0, 1, 2, . . . until budget is exhausted do 2: {(θk , Σk , wk )}kK=1 ← E STIMATE B ASINS (Dt ) 3: for each x ∈ Xcand do 4: ∆Vintra ( x ) ← I NTRA U TILITY x, {(θk , Σk , wk )}kK=1 , Xtar 5: ∆Vinter ( x ) ← I NTER U TILITY x, {(θk , Σk , wk )}kK=1 , Xtar 6: S( x ) ← (∆Vintra ( x ) + ∆Vinter ( x ))/c( x )α 7: end for 8: xt+1 ← arg maxx∈Xcand S( x ) 9: yt+1 ← R UN E XPERIMENT ( xt+1 ) 10: Dt+1 ← Dt ∪ {( xt+1 , yt+1 )} 11: Xcand ← Xcand \ { xt+1 } 12: end for 13: return Dt Intra-basin utility.
For a candidate x, let
∂ f ( x; θ ) ∈ R1× p ∂θ θ =θk denote the parameter Jacobian at basin k. Under the local linear approximation, the reduction in within-basin target uncertainty is Jx ( θ k ) =
∆Vintra ( x ) =
K ∥ Jk Σk Jx (θk )⊤ ∥22 1 wk 2 . ∑ |Xtar | k=1 σ + Jx ( θ k ) Σ k Jx ( θ k ) ⊤
This term is large when observing y at x is expected to substantially reduce predictive variance over the target region. Inter-basin utility.
The inter-basin gain is ∆Vinter ( x ) = Vinter −
Z
+ Vinter ( x, y) p(y | x, Dt ) dy,
+ where Vinter ( x, y) is the updated between-basin uncertainty after observing outcome y at candidate x. The predictive distribution p(y | x, Dt ) is the scalar mixture induced by the current basin approximation. Because the expectation is one-dimensional, we evaluate it efficiently using numerical quadrature.
4.4
The Sequential Design Procedure
Algorithm 1 summarizes the full procedure. At each round, we first update the basin approximation from the current dataset, then score all remaining candidates using the target-aware acquisition function above, and finally select the highest-scoring affordable experiment.
5
Experiments
5.1
Experimental Setup
Benchmark overview. We evaluate our method on a benchmark for budget-aware sequential design in scaling-law fitting, comprising 8 tasks and 65 scaling-law instances. Each instance specifies a parametric law family, a finite pool of runnable candidate experiments with associated costs, and a held-out target region for evaluation. The tasks cover diverse LLM scaling scenarios, including pre-training hyperparameter tuning, data allocation, architecture design, sparsity, and inference-time scaling. Table 1 summarizes the task-level statistics; detailed task descriptions and data sources are deferred to Appendix B.1. 5
Preprint. Under review.
Task parallel vocab domain moe data con lr&bsz sparsity farseer
# Laws
Avg. Params
# Train
# Test
Target
Cost
10 10 10 10 10 10 4 1
4.2 6.8 29.0 5.3 7.0 20.2 5.0 9.0
36 1080 504 193 161 2702 70 404
12 120 42 28 21 117 18 7
L( N, P) L( N, V, D ) { Li (r)}5i=1 L( N, E) L( N, D, U ) L(l, b, N, D ) L( P, N2 ) L( N, D )
N 6ND 1 NE 6ND 6ND 6N1 D1 + 6N2 D2 6ND
Table 1: Task statistics for the scaling-law benchmark. Each task contains a collection of scaling-law instances for evaluating budget-aware sequential design on target-region extrapolation. Protocol and metric. At the start of each episode, the learner is given the candidate pool Xcand , target region Xtar , and candidate costs, but not outcomes. At each round, it selects one previously unobserved feasible candidate, receives the corresponding observation, and refits the scaling law. Each method is repeated for 10 runs, with parameters refit after every step using L-BFGS-B from 64 initialization points. We report performance at three budget checkpoints: 1%, 5%, 10% of total training cost for most tasks, and 20%, 35%, 50% for domain and sparsity, whose costs are more uniformly distributed. Performance is measured by target-region R2 , aggregated over all runs and instances within each task, clipped to [−1, 1], and reported as mean ± standard deviation. All sequential methods share the same fitting procedure and differ only in experiment selection. Law families and cost models. The benchmark spans a heterogeneous collection of scalinglaw families, including classical power laws, log-space interaction models, compositional mixture laws, hyperparameter response surfaces, and several more expressive nonlinear forms; full parameterizations are deferred to Appendix B.2. We assign each task a simple cost proxy aligned with its dominant resource. For dense-training settings (data con, farseer, lr&bsz, and vocab), we use 6ND. For the remaining tasks, we use task-specific proxies: NE for moe, N for parallel, 6N1 D1 + 6N2 D2 for sparsity, and unit cost for domain, which varies mixture proportions rather than overall training scale. Baselines. We compare against five baselines. (1) Random uniformly samples the next experiment from the feasible unobserved candidates. (2) Cheapest always selects a minimumcost feasible candidate, breaking ties uniformly at random. (3) Cost Rand samples each feasible unobserved candidate with probability proportional to 1/c( x ). These three heuristics respectively capture uninformed exploration, an aggressive preference for cheap experiments, and a simple stochastic bias toward lower-cost candidates. We further compare against two classical design criteria adapted to our nonlinear, costconstrained setting. (4) D-opt selects the candidate that maximizes the increase in a Doptimality objective, which favors experiments that most reduce the overall volume of parameter uncertainty. In the locally linearized model, this corresponds to preferring points that most increase the log-determinant of the Fisher information matrix. (5) V-opt selects the candidate that maximizes a V-optimality objective, which favors experiments expected to most reduce predictive variance over the target region. In our implementation, both D-opt and V-opt locally linearize the nonlinear scaling law around the parameter estimate with the lowest MSE among the fitted solutions from the previous step, and differ only in the acquisition score computed from this linearization. To account for heterogeneous costs, each candidate score is normalized by c( x )α , with α = 0.4 in all experiments. For D-opt, V-opt, and our method, we use a short warm-start phase before the first criterionbased acquisition step: each method first selects the 2.5p lowest-cost candidates, where p is the number of law parameters. The cost of these initialization points is counted toward the total budget. As a full-information reference, we also report All Data, obtained by fitting the same scaling law on the entire training set and evaluating the resulting target-region R2 . 6
0.75 0.50 0.25 0.00
1%
5%
moe
10% 0.81
5% 10% Random
0.9 0.8 0.7 0.6 1% Cheapest
35%
50%
data_con
0.79 5% 10% Cost Rand
vocab
1.00 0.98 0.96 0.94 0.92 1%
5%
sparsity
0.5
0.93 10% 0.37
0.0 0.5 20% D-opt
35% 50% V-opt
Averaged Test R 2
1%
0.81
Averaged Test R 2
0.50
domain
Averaged Test R 2
0.00 0.25
1.00 0.75 0.50 0.25 0.00 20%
Averaged Test R 2
0.04
Averaged Test R 2
lr&bsz
0.25
Averaged Test R 2
Averaged Test R 2
Averaged Test R 2
Preprint. Under review.
Ours
1.0 0.8 0.6 0.4 0.2 1% 0.8 0.6 0.4 0.2 1%
parallel
0.99
5%
10% 0.91
5% All Data
10%
farseer
Figure 2: Mean target-region R2 as a function of consumed budget on the benchmark. Our method reaches the strongest overall budget–accuracy trade-off and approaches the full-data reference using only a small fraction of the total experimental cost. Setting
lr&bsz
domain
vocab
parallel
1% Random 1% Cheapest 1% Cost Rand 1% D-opt 1% V-opt 1% Ours
-0.65 ± 0.49 -0.36 ± 0.82 0.54 ± 0.57 -1.00 ± 0.00 -0.47 ± 0.66 -0.74 ± 0.48 -0.25 ± 0.63 -0.77 ± 0.55 -0.92 ± 0.29 -0.36 ± 0.82 0.30 ± 0.71 -1.00 ± 0.00 0.30 ± 0.57 -0.58 ± 0.64 -0.95 ± 0.14 -0.89 ± 0.17 -0.89 ± 0.31 -0.36 ± 0.82 0.86 ± 0.38 -1.00 ± 0.00 -0.11 ± 0.70 -0.38 ± 0.70 -0.41 ± 0.52 -0.25 ± 0.63 -0.92 ± 0.23 0.14 ± 0.80 0.95 ± 0.03 -1.00 ± 0.00 0.34 ± 0.54 0.71 ± 0.43 -0.07 ± 0.56 -0.11 ± 0.80 -0.70 ± 0.47 0.57 ± 0.66 0.96 ± 0.02 -1.00 ± 0.00 0.67 ± 0.24 0.58 ± 0.53 0.12 ± 0.53 0.67 ± 0.22 -0.66 ± 0.53 0.64 ± 0.58 0.96 ± 0.02 -1.00 ± 0.00 0.59 ± 0.39 0.73 ± 0.37 0.31 ± 0.39 0.60 ± 0.11
5% Random 5% Cheapest 5% Cost Rand 5% D-opt 5% V-opt 5% Ours
-0.45 ± 0.61 0.41 ± 0.82 0.53 ± 0.62 0.27 ± 0.02 -0.88 ± 0.34 0.41 ± 0.82 0.53 ± 0.62 0.42 ± 0.03 -0.79 ± 0.43 0.41 ± 0.82 0.89 ± 0.27 0.03 ± 0.87 -0.66 ± 0.57 0.81 ± 0.48 0.97 ± 0.01 0.70 ± 0.52 -0.06 ± 0.59 0.91 ± 0.33 0.97 ± 0.00 0.69 ± 0.53 0.00 ± 0.59 0.89 ± 0.38 0.98 ± 0.00 0.77 ± 0.39
10% Random -0.33 ± 0.66 0.70 ± 0.67 0.59 ± 0.54 0.19 ± 0.01 10% Cheapest -0.80 ± 0.46 0.70 ± 0.67 0.55 ± 0.63 0.42 ± 0.03 10% Cost Rand -0.79 ± 0.39 0.70 ± 0.67 0.83 ± 0.34 0.53 ± 0.81 10% D-opt -0.53 ± 0.57 0.88 ± 0.43 0.97 ± 0.00 0.99 ± 0.00 10% V-opt 0.18 ± 0.54 0.95 ± 0.27 0.98 ± 0.00 0.99 ± 0.00 10% Ours 0.22 ± 0.55 0.95 ± 0.28 0.98 ± 0.00 0.99 ± 0.00 All Data
moe
data con sparsity
farseer
0.01 ± 0.72 -0.36 ± 0.71 0.37 ± 0.26 0.78 ± 0.18 0.74 ± 0.19 -0.56 ± 0.65 0.04 ± 0.46 0.41 ± 0.40 0.68 ± 0.33 -0.39 ± 0.65 -0.06 ± 0.52 0.56 ± 0.38 0.55 ± 0.34 0.80 ± 0.17 0.23 ± 0.32 0.49 ± 0.18 0.80 ± 0.05 0.83 ± 0.17 0.39 ± 0.20 0.87 ± 0.01 0.65 ± 0.27 0.85 ± 0.16 0.44 ± 0.17 0.88 ± 0.02 0.26 ± 0.65 0.12 ± 0.62 0.38 ± 0.25 0.81 ± 0.14 -0.40 ± 0.64 0.32 ± 0.20 0.78 ± 0.23 -0.22 ± 0.67 0.12 ± 0.38 0.74 ± 0.14 0.77 ± 0.16 0.19 ± 0.41 0.80 ± 0.11 0.85 ± 0.13 0.51 ± 0.15 0.83 ± 0.07 0.86 ± 0.11 0.53 ± 0.08
0.79 ± 0.09 0.68 ± 0.18 0.74 ± 0.11 0.87 ± 0.03 0.92 ± 0.01 0.93 ± 0.00
0.04 ± 0.67 0.81 ± 0.51 0.93 ± 0.16 0.99 ± 0.00 0.81 ± 0.04 0.79 ± 0.23 0.37 ± 0.10 0.91 ± 0.01
Table 2: Task-level breakdown of target-region R2 under different budget levels. Each cell reports the mean and standard deviation aggregated over all scaling-law instances within the corresponding task. Higher is better. Budgets are 1%, 5%, 10% for most tasks, and 20%, 35%, 50% for domain and sparsity. 5.2
Results Analysis
Overall trends. Figure 2 and Table 2 show that our method delivers the strongest overall performance across tasks and budget levels. Its advantage is most pronounced in the lowbudget regime, where experiment selection matters most: at 1% budget, it performs best on domain, data con, and sparsity, matches the top result on vocab, and remains competitive on moe and farseer. As the budget increases, the advantage becomes more consistent. At 5% budget, it achieves the best result on five of the eight tasks, and at 10% budget, it matches or outperforms all baselines on every task. A consistent pattern is that model-aware design substantially outperforms simple budget heuristics. Across most tasks and budgets, Random, Cheapest, and Cost Rand lag far behind, especially under tight budgets, showing that neither uninformed exploration nor naive cost preference is sufficient for reliable target-region extrapolation. 7
Preprint. Under review.
Selected-set MSE Gap
Test MSE Gap
1.0
1.0
log10(Selected-set MSE / Best)
30
10 0 10 20 30 40
30
20
10
0
10
20
30
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
log10(Test MSE / Best)
20
40
30
20
10
0
10
20
30
0.0
Figure 3: Parameter-space visualization for one lr&bsz scaling law (sl 5) after fitting on the cheapest 12% of training points from 2048 initializations. We embed the fitted parameters with t-SNE and color each solution by its MSE on the selected points (left) or on the heldout test region (right). Multiple separated clusters indicate many local optima, while the mismatch between the two colorings shows that low error on the observed low-cost points does not reliably imply low error on the high-cost extrapolation region. Our method achieves 0.71 averaged test R2 within 5% budget compared with 0.57 for V-opt and 0.16 for D-opt. Comparison to design-based baselines. Among the design-based baselines, D-opt is already much stronger than the simple heuristics, highlighting the value of exploiting local sensitivity information. V-opt is closer in spirit to our method: both are prediction-oriented, but V-opt relies on a single local linearization around the current best-fit parameter, whereas our method maintains a mixture-of-Gaussians representation over multiple plausible parameter regions. Empirically, our method is generally more robust and achieves better overall performance, especially at moderate and higher budgets. This gap is most pronounced when the fitting landscape contains multiple plausible basins and the target region is strongly extrapolative, so that the current best fit need not be the right expansion point for design. Figure 3 illustrates such a case on one lr&bsz scaling law using t-SNE visualization (van der Maaten & Hinton, 2008). Task and budget dependence. Performance varies substantially across tasks, reflecting the heterogeneous structure of the benchmark. Because each entry in Table 2 averages over multiple scaling-law instances within a task, these results should be interpreted as robustness over a heterogeneous family of fitting problems rather than performance on any single law. Some tasks are relatively easy on average: on vocab, all design-based methods already perform strongly at low budget, and the gap nearly closes by 5% budget; similarly, parallel is close to saturated once the budget becomes moderate. By contrast, lr&bsz, domain, data con, and sparsity remain more challenging on average and show larger separation among methods across a wider range of budgets. Overall, low-budget performance is the main differentiator across methods, while higher-budget results reveal which approaches remain robust on the harder and more heterogeneous tasks. Why can some methods outperform All Data? Some budgeted methods occasionally outperform All Data. This is not contradictory, because All Data is not an oracle upper bound: it is simply the test-region R2 obtained by fitting the same parametric law on the full training set. Since the target region is fully extrapolative and concentrated in the high-cost regime, the best fit on all observed training points need not be the best fit for the target region. Under model misspecification, adding more training points can even hurt extrapolation if those points are concentrated in regions whose trends are less aligned with the high-cost behavior of interest. This effect is particularly clear on lr&bsz, where misspecification is relatively severe. Fitting only the cheaper half of the training points yields a poor target-region performance of 8
Preprint. Under review.
Setting
lr&bsz
domain
vocab
parallel
moe
data con sparsity
farseer
1% w/o ∆Vinter 1% w/o ∆Vintra 1% Ours
-0.70 ± 0.48 0.64 ± 0.58 0.96 ± 0.02 -1.00 ± 0.00 0.55 ± 0.44 0.69 ± 0.39 0.08 ± 0.55 0.60 ± 0.11 -0.78 ± 0.44 0.39 ± 0.76 0.96 ± 0.02 -1.00 ± 0.00 0.43 ± 0.63 -0.01 ± 0.79 -0.07 ± 0.71 -0.41 ± 0.73 -0.66 ± 0.53 0.64 ± 0.58 0.96 ± 0.02 -1.00 ± 0.00 0.59 ± 0.39 0.73 ± 0.37 0.31 ± 0.39 0.60 ± 0.11
5% w/o ∆Vinter 5% w/o ∆Vintra 5% Ours
-0.02 ± 0.60 0.89 ± 0.38 0.97 ± 0.00 0.77 ± 0.39 0.67 ± 0.23 0.83 ± 0.17 0.33 ± 0.30 0.88 ± 0.02 -0.51 ± 0.53 0.86 ± 0.43 0.98 ± 0.00 0.07 ± 0.90 0.34 ± 0.40 0.65 ± 0.47 0.37 ± 0.39 0.84 ± 0.09 0.00 ± 0.59 0.89 ± 0.38 0.98 ± 0.00 0.77 ± 0.39 0.65 ± 0.27 0.85 ± 0.16 0.44 ± 0.17 0.88 ± 0.02
10% w/o ∆Vinter 0.20 ± 0.55 0.95 ± 0.28 0.98 ± 0.00 0.99 ± 0.01 0.83 ± 0.08 0.85 ± 0.12 0.51 ± 0.13 0.93 ± 0.00 10% w/o ∆Vintra -0.13 ± 0.60 0.91 ± 0.35 0.98 ± 0.00 0.99 ± 0.00 0.71 ± 0.22 0.80 ± 0.25 0.48 ± 0.19 0.89 ± 0.04 10% Ours 0.22 ± 0.55 0.95 ± 0.28 0.98 ± 0.00 0.99 ± 0.00 0.83 ± 0.07 0.86 ± 0.11 0.53 ± 0.08 0.93 ± 0.00
Table 3: Ablation study of the acquisition function. We remove either ∆Vinter or ∆Vintra from the acquisition score and report the target-region R2 at different budget levels. Each cell shows the mean and standard deviation aggregated over all scaling-law instances. Higher is better. Budgets are 1%, 5%, 10% for most tasks, and 20%, 35%, 50% for domain and sparsity. R2 = −0.79, while fitting the more expensive half yields a much better R2 = 0.12, despite using the same number of points. More broadly, this highlights an important practical lesson for real-world scaling-law fitting: when the goal is to predict much larger and more expensive configurations, blindly fitting on all available pilot runs may be suboptimal. 5.3
Ablation Study
To understand which parts of our acquisition function are responsible for the observed gains, we perform an ablation study by removing the two terms in the MSPE decomposition, ∆Vinter and ∆Vintra . Here, Vinter measures uncertainty induced by disagreement across different basins, while Vintra measures uncertainty within each basin due to local parameter variation around a mode. Ablating them separately allows us to isolate the contributions of cross-basin and within-basin uncertainty to sequential experiment selection. Ablation results. Table 3 shows that both terms contribute, but not equally. Removing ∆Vintra causes the larger and more consistent degradation, especially on data con, farseer, moe, and lr&bsz, indicating that within-basin uncertainty is the dominant signal for effective sequential design. By contrast, removing ∆Vinter usually leads to a smaller drop and in some cases leaves performance nearly unchanged, suggesting that cross-basin disagreement is more task-dependent. This difference is consistent with the roles of the two terms. ∆Vintra remains useful even after the method has identified a plausible parameter region, because it continues to refine uncertainty within that basin. ∆Vinter is most helpful when several basins remain plausible and induce different target-region predictions, which occurs more often under tighter budgets and on more heterogeneous tasks such as sparsity and data con. Overall, the full method is the most robust across tasks and budget levels, indicating that the two terms are complementary: ∆Vintra provides the dominant signal, while ∆Vinter yields additional gains when basin ambiguity is substantial.
6
Conclusions
We formulated scaling-law fitting as a budget-aware sequential experimental design problem, where each candidate run incurs a cost and the objective is to maximize predictive accuracy in a high-cost target region. We proposed an uncertainty-aware acquisition strategy that selects experiments according to their value for target-region extrapolation. Across a diverse benchmark of scaling-law tasks, the method consistently outperforms random, heuristic, and classical design-based baselines, and often approaches full-data performance with only a small fraction of the original training budget. These results suggest that, at modern scales, scaling-law fitting should be treated not only as a modeling problem but also as a problem of experimental design and budget allocation. 9
Preprint. Under review.
References Anthony C Atkinson and David C Woods. Designs for generalized linear models. Handbook of design and analysis of experiments, 7:471–514, 2015. Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken neural scaling laws, 2023. URL https://arxiv.org/abs/2210.14891. Lingjiao Chen, Jared Quincy Davis, Boris Hanin, Peter Bailis, Ion Stoica, Matei Zaharia, and James Zou. Are more llm calls all you need? towards scaling laws of compound inference systems, 2024. URL https://arxiv.org/abs/2403.02419. Mouxiang Chen, Binyuan Hui, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Jianling Sun, Junyang Lin, and Zhongxin Liu. Parallel scaling law for language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=dEi1S731lk. William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022. URL http://jmlr.org/papers/v23/21-0998.html. Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 10835–10866. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/v202/gao23h.html. Steven G Gilmour and Luzia A Trinca. Optimum design of experiments for statistical inference. Quality control and applied statistics, 58(3):235–236, 2013. Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro Von Werra, and Martin Jaggi. Scaling laws and compute-optimal training beyond fixed training durations. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 76232–76264. Curran Associates, Inc., 2024. doi: 10.52202/ 079017-2427. URL https://proceedings.neurips.cc/paper files/paper/2024/file/ 8b970e15a89bf5d12542810df8eae8fc-Paper-Conference.pdf. Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint arXiv:2102.01293, 2021. URL https://arxiv.org/abs/2102.01293. Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Patwary, Mostofa Ali, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models, 2022. URL https://arxiv.org/abs/2203.15556. Shengding Hu, Yuge Tu, Xu Han, Ganqu Cui, Chaoqun He, Weilin Zhao, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Xinrong Zhang, Zhen Leng Thai, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, dahai li, Zhiyuan Liu, and Maosong Sun. MiniCPM: Unveiling the potential of small language models with scalable training strategies. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=3X2L2TFr0f. Xun Huan and Youssef M. Marzouk. Simulation-based optimal bayesian experimental design for nonlinear systems. Journal of Computational Physics, 232(1):288–317, January 2013. ISSN 0021-9991. doi: 10.1016/j.jcp.2012.08.013. URL http://dx.doi.org/10.1016/ j.jcp.2012.08.013. 10
Preprint. Under review.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Chris Hallacy, Jan Leike, Aditya Ramesh, et al. Scaling laws for neural language models. In arXiv preprint arXiv:2001.08361, 2020. André I Khuri, Bhramar Mukherjee, Bikas K Sinha, and Malay Ghosh. Design issues for generalized linear models: A review. 2006. Jack Kiefer. Optimum experimental designs. Journal of the Royal Statistical Society: Series B (Methodological), 21(2):272–304, 1959. Houyi Li, Wenzhen Zheng, Jingcheng Hu, Qiufeng Wang, Hanshan Zhang, Zili Wang, Shijie Xuyang, Yuantao Fan, Shuigeng Zhou, Xiangyu Zhang, et al. Predictable scale: Part i–optimal hyperparameter scaling law in large language model pretraining. arXiv e-prints, pp. arXiv–2503, 2025a. Houyi Li, Wenzhen Zheng, Qiufeng Wang, Zhenyu Ding, Haoying Wang, Zili Wang, Shijie Xuyang, Ning Ding, Shuigeng Zhou, Xiangyu Zhang, and Daxin Jiang. Predictable scale: Part ii, farseer: A refined scaling law in large language models, 2025b. URL https://arxiv.org/abs/2506.10972. Margaret Li, Sneha Kudugunta, and Luke Zettlemoyer. (mis)fitting: A survey of scaling laws, 2025c. URL https://arxiv.org/abs/2502.18969. Seng Pei Liew, Takuya Kato, and Sho Takase. Scaling laws for upcycling mixture-of-experts language models. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=ZBBo19jldX. Haowei Lin, Baizhou Huang, Haotian Ye, Qinyu Chen, Zihao Wang, Sujian Li, Jianzhu Ma, Xiaojun Wan, James Zou, and Yitao Liang. Selecting large language model to fine-tune via rectified scaling law. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=Bq2THeNXRr. Haowei Lin, Haotian Ye, Wenzheng Feng, Quzhe Huang, Yujun Li, Hubert Lim, Zhengrui Li, Xiangyu Wang, Jianzhu Ma, Yitao Liang, and James Zou. Can language models discover scaling laws? In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=TPTtWC0pGk. Jan Ludziejewski, Jakub Krajewski, Kamil Adamczewski, Maciej Pióro, Michał Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz Odrzygóźdź, Piotr Sankowski, Marek Cygan, and Sebastian Jaszczur. Scaling laws for fine-grained mixture of experts. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 33270–33288. PMLR, 21–27 Jul 2024. URL https://proceedings.mlr.press/v235/ ludziejewski24a.html. Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. In Advances in Neural Information Processing Systems, volume 36, pp. 50358–50376, 2023. Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, and Yair Carmon. Resolving discrepancies in compute-optimal scaling of language models, 2025. URL https://arxiv. org/abs/2406.19146. Haoran Que, Jiaheng Liu, Ge Zhang, Chenchen Zhang, Xingwei Qu, Yinghao Ma, Feiyu Duan, ZhiqiBai, JiakaiWang, Yuanxing Zhang, Xu Tan, Jie Fu, Jiamang Wang, Lin Qu, Wenbo Su, and Bo Zheng. D-CPT law: Domain-specific continual pre-training scaling law for large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=JzKFN5fWOk. 11
Preprint. Under review.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research (PMLR), 2021. URL https://arxiv.org/abs/2103.00020. Ayan Sengupta, Yash Goel, and Tanmoy Chakraborty. How to upscale neural networks with scaling law? Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL https://openreview.net/forum?id=AL7N0UOfgI. Samuel Silvey. Optimal design: an introduction to the theory for parameter estimation. Springer Science & Business Media, 2013. Chaofan Tao, Qian Liu, Longxu Dou, Niklas Muennighoff, Zhongwei Wan, Ping Luo, Min Lin, and Ngai Wong. Scaling laws with vocabulary: Larger models deserve larger vocabularies. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=sKCKPr8cRL. Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(86):2579–2605, 2008. URL http://jmlr.org/papers/v9/ vandermaaten08a.html. Lynda V White. An extension of the general equivalence theorem to nonlinear models. Biometrika, 60(2):345–348, 1973. Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for LLM problem-solving. In The Thirteenth International Conference on Learning Representations, 2025. URL https: //openreview.net/forum?id=VNckp7JEHn. Ruihan Xu, Jiajin Li, and Yiping Lu. On the width scaling of neural optimizers under matrix operator norms i: Row/column normalization and hyperparameter transfer. arXiv preprint arXiv:2603.09952, 2026. Ge Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neural networks via zero-shot hyperparameter transfer. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 17084–17097. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper files/paper/2021/file/ 8df7c2e3c3c3be098ef7b382bd2c37ba-Paper.pdf. Min Yang, Stefanie Biedermann, and Elina Tang. On optimal designs for nonlinear models: a general and efficient algorithm. Journal of the American Statistical Association, 108(504): 1411–1420, 2013. Jiasheng Ye, Peiju Liu, Tianxiang Sun, Jun Zhan, Yunhua Zhou, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. In The Thirteenth International Conference on Learning Representations, 2025. URL https: //openreview.net/forum?id=jjCB27TMK3. Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12104–12113, June 2022. Shijie Zhong, Wanggang Shen, Tommie Catanach, and Xun Huan. Goal-oriented bayesian optimal experimental design for nonlinear models using markov chain monte carlo. SIAM/ASA Journal on Uncertainty Quantification, 14(1):19–47, 2026. doi: 10.1137/ 24M1649344. URL https://doi.org/10.1137/24M1649344.
12
Preprint. Under review.
A
Use of LLMs
We employ large language models (LLMs) exclusively for the purpose of assisting in the drafting and refinement of our manuscripts, with the objective of enhancing clarity and coherence.
B
Detailed Statistics of the Benchmark
B.1
Task Collection Details
Our experimental benchmark covers a diverse set of practical LLM scaling scenarios, including pre-training hyperparameter tuning, data allocation, architecture design, and inferencetime scaling. The first six tasks are drawn from SLDBench (Lin et al., 2026), and the remaining two from the original papers; we also refer to surveys and related literature for task coverage and context (Sengupta et al., 2025). For the six SLDBench datasets, we use the top-10 law forms on the leaderboard ranked by mean R2 . We exclude Supervised Finetuning Scaling Law due to its very limited data and U-shaped Scaling Law due to severe misspecification that yields poor fits even on the full training set. The collected tasks are: (1) Parallel Scaling Law (parallel), which studies the effect of parallelism P and model size N on language modeling loss; this setting creates P augmentations of an input and aggregates their outputs, conceptually similar to Best-of-N (Chen et al., 2025). (2) Vocabulary Scaling Law (vocab), which models unigram-normalized loss as a function of non-vocabulary model size N, vocabulary size V, and dataset size D (Tao et al., 2024). (3) Domain Mixture Scaling Law (domain), which models domain-specific pre-training loss as a function of the mixture proportions of training domains (Ye et al., 2025). (4) Mixture of Experts Scaling Law (moe), which relates loss to the number of dense parameters N and experts E (Ludziejewski et al., 2024). (5) Data Constrained Scaling Law (data con), which models pre-training loss using network size N, dataset size D, and the number of unique tokens U. (6) Learning Rate and Batch Size Scaling Law (lr&bsz), adapted from the Step Law (Li et al., 2025a), which models pre-training loss as a function of learning rate l, batch size b, dataset size D, and network size N. (7) Sparsity Scaling Law (sparsity), which models test loss based on the ratio P between total model size N1 and active parameters N2 (Liew et al., 2025). (8) Farseer Scaling Law (farseer), which extends the Chinchilla-style formulation (Hoffmann et al., 2022) to predict loss from model size N and training data D (Li et al., 2025b). B.2
Parametric Forms of Collected Scaling Laws
This appendix summarizes the parametric scaling-law families used in our benchmark. Our goal is not to advocate a single canonical law form, but to evaluate budget-aware sequential design across a diverse set of realistic fitting regimes. The collected laws therefore span classical power-law formulations, log-space interaction models, compositional mixture laws, hyperparameter response surfaces, and several more expressive nonlinear forms drawn from prior scaling-law studies. For the six tasks inherited from SLDBench, we use the top-ranked law families on the public leaderboard by mean R2 ; for the remaining tasks, we adopt the parametric forms proposed in the corresponding original papers. Together, these forms define the nonlinear fitting landscapes on which all acquisition methods are evaluated. Table 4: Collected scaling laws grouped by task. ID
Parametric Form
# Params
All Data test R2
7
0.94 ± 0.00
data con sl 1
L( N, D, U ) = NAα + DBβ + E U γ N δ
13
Preprint. Under review.
ID
Parametric Form
# Params
All Data test R2
sl 2 sl 3
L( N, D, U ) = a + bU p + cN q + dDr L( N, D, U ) = NAα + DBα + C, UN =
7 7
0.73 ± 0.11 0.89 ± 0.00
min(ρU, N ), R N = max( N/UN − 1, 0), Neff = UN + τN UN (1 − e− R N /τN ), Deff = U + τD U (1 − e−( D/U −1)/τD ) −b L( N, D, U ) = L0 + AMn−a + BTeff,n , q=
sl 4
7
0.95 ± 0.01
Deff = U γ D1−γ
6
0.84 ± 0.00
8
0.88 ± 0.08
7
0.94 ± 0.03
7
-0.07 ± 0.19
7
0.84 ± 0.08
7
0.81 ± 0.03
30
1.00 ± 0.00
35
1.00 ± 0.00
35 35 35
1.00 ± 0.00 1.00 ± 0.00 1.00 ± 0.00
35
-0.81 ± 0.58
40
1.00 ± 0.00
15 15 15
1.00 ± 0.00 1.00 ± 0.00 1.00 ± 0.00
9
0.92 ± 0.01
15 26
0.61 ± 0.19 -0.15 ± 0.30
24
-0.99 ± 0.04
eff
sl 5
eff
Tn , Teff,n = 1T+nq sUn Mnd L( N, D, U ) = NAα + Bβ D
+ E,
eff
sl 6
−β
L( N, D, U ) = E + AN −α + BDeff ,
Deff =
D 1+C max( D/U −1,0)c N d
sl 7 sl 8 sl 9 sl 10
αp L( N, D, U ) = L0 + A( NU )α pu + BDαt + CN
L( N, D, U ) = a + Dbα + Ncβ + d log U D +1 L( N, D, U ) = NAα + DBβ 1 + UCγ + Linf
γ
L( N, D, U ) = L0 + AN − a + B D −bq + (kU )−bq
1/q
domain sl 1 sl 2 sl 3 sl 4 sl 5 sl 6 sl 7 sl 8 sl 9 sl 10
Li (r) = ai + bi log(ri + ε) + ∑ j̸=i cij r j Li (r) = Ai (ri + ε i )−αi exp ∑ j̸=i wij r j
exp Li (r) = basei + coeffi ri i + ∑ j̸=i Wij r j αk Li (r) = exp ∑k Cik rk + biasi α j
Li (r) = bi + ∑ j Wij r j − αi Li (r) = Ci + Ai ∑ j Tij r j log Li (r) = intercepti + ∑ j clin ij r j + cij log(r j + ε ) b L i (r) = c i − a i r i i
2 Li (r) = ai + bi log(ri + ε) + ci log(ri + ε) bi L i (r) = a i + r + ε i
i
farseer sl 1
q
b
L( N, D ) = esN +S + e BN +Q D −e
AN a + E
lr&bsz sl 1 sl 2
sl 3
L(l, b, N, D ) = exp poly2 (log l, log b, log D, log N ) L(l, b, N, D ) = Linf + C p e− a p n + Cd e− ad s + Cdp e− adp (s−kn) + √ Cbb e−abb v + c L ∆u2 + c B ∆v2 + 2ρ c L c B ∆u ∆v, n = log N, s = log D, u = log l, v = log b L(l, b, N, D ) = E + AN −α + BD − β + N w NFDwD + Ceff (log l − optl )2 + Geff (log b − optb )2
14
Preprint. Under review.
ID
Parametric Form
sl 4
L(l, b, N, D ) = exp w0 + w1 log l + w2 log b + w3 log D + w4 log N + w5 log2 l + w6 log2 b + w7 log2 D + w8 log2 N + w9 log l log b + w10 log l log D + w11 log l log N + w12 log b log D + w13 log b log N + w14 log D log N + w15 (log D − log N ) + w16 /b + w17 /b2 + w18 /D + w19 /N L(l, b, N, D ) = L0 + A N e− a N n + A D e−aD s + 2 A B e− a B v + A R e− a R (s−n) + A X e− a X (s−v) + clr,0 e−wb v−wn n−ws s (u − u⋆ )2 , u = log l, v = log b, s = log D, n = log N L(l, b, N, D ) = Linf + exp w0 + wd log D + w p log N + wdp log D log N + wlr log l + wlr2 log2 l + wbsz log b + wbsz2 log2 b + wlr,bsz log l log b + wlr,D log l log D + wlr,N log l log N + wbsz,D log b log D + wbsz,N log b log N L(l, b, N, D ) = E + exp w1,0 + w1,1 x1 + w1,2 x2 + w1,3 x3 + w1,4 x4 + w1,5 x12 + w1,6 x22 + w1,7 x32 + w1,8 x42 + w1,9 x1 x2 + w1,10 x1 x3 + w1,11 x1 x4 + w1,12 x2 x3 + w1,13 x2 x4 + w1,14 x3 x4 + exp w2,0 + w2,1 x1 + w2,2 x2 + w2,3 x3 + w2,4 x4 + w2,5 x12 + w2,6 x22 + w2,7 x32 + w2,8 x42 + w2,9 x1 x2 + w2,10 x1 x3 + w2,11 x1 x4 + w2,12 x2 x3 + w2,13 x2 x4 + w2,14 x3 x4 , x1 = log l, x2 = log b, x3 = log D, x4 = log N L(l, b, N, D ) = L0 + c P e−aP n + c D e−aD s + 2 (1 + c R e− aR (s−n) + klr δlr2 (1 + alr tanh δlr ) + kns δns 2 ans tanh δns ) + kdp ((s − n) − δ0 ) , n = log N, s = log D L(l, b, N, D ) = poly2 (log10 l, log10 b, log10 D, log10 N ) L(l, b, N, D ) = poly2 (log l, log b, log D, log N ) + w D D −1/2 + w N N −1/2 + wb b−1
sl 5
sl 6
sl 7
sl 8
sl 9 sl 10
# Params
All Data test R2
20
-0.88 ± 0.36
19
-0.10 ± 0.66
14
0.47 ± 0.14
31
0.75 ± 0.19
20
0.49 ± 0.33
15
0.36 ± 0.09
18
-0.07 ± 0.64
4 5 6 6 6
0.83 ± 0.00 0.83 ± 0.00 0.90 ± 0.00 0.83 ± 0.00 0.80 ± 0.00
6 6 4 4
0.83 ± 0.00 0.74 ± 0.00 0.83 ± 0.00 0.83 ± 0.00
6
0.77 ± 0.00
6 5
1.00 ± 0.00 1.00 ± 0.00
moe sl 1 sl 2 sl 3 sl 4 sl 5 sl 6 sl 7 sl 8 sl 9 sl 10
L( N, E) = Linf + N αBE β L( N, E) = L + K ( N α E β )−γ AN α L( N, E) = 1+ + CN 0.6α + D BE β a L( N, E) = N α (1+bE)γ + c + d log N − 0.4 log(1 + E) L( N, E) = p0 + exp p1 + p2 log E + p3 log N + p4 log E log N + p5 log E f L( N, E) = aN −b (1 + cE−d ) + e + EN 0.05 p p p L( N, E) = p0 E 1 N 2 + p3 N 4 + p5 L( N, E) = aN b Ec + d L( N, E) = c0 + A( NE g )− a − β BEγ L( N, E) = bias + A( N/109 )−α 1+ 1+ B
parallel sl 1 sl 2
L( N, P) = c0 + c N N −α + c P P− β + c NP N −α P− β L( N, P) = c0 + c N N −α + c P P− β
15
Preprint. Under review.
ID
Parametric Form
# Params
All Data test R2
sl 3 sl 4
L( N, P) = aN b + 1+c P + d L( N, P) = aN b + cP−1/2 +d
4 4
1.00 ± 0.00 1.00 ± 0.00
4
1.00 ± 0.00
4 4
1.00 ± 0.00 1.00 ± 0.00
L( N, P) = ( AN −α + E) P− β
4 3 4
1.00 ± 0.00 1.00 ± 0.00 1.00 ± 0.00
L( P, N2 ) = ed1 P−a N2−b ec log P log N2 + ed3 L( P, N2 ) = ed1 P−a N2−b + ed3 L( P, N2 ) = ed1 P−a + ed2 N2−b ec log P log N2 + ed3 L( P, N2 ) = ed1 P−a + ed2 N2−b + ed3
5 4 6 5
0.28 ± 0.00 0.52 ± 0.00 0.27 ± 0.00 0.40 ± 0.00
L( N, V, D ) = c0 + AV b N e D g L( N, V, D ) = L + A Mr ( N −α , D − β ) 1 + C (log V − v0 )2 q 1/q L( N, V, D ) = L0 + ( aN −α )q + b( DV ϕ )− β
5 7
0.89 ± 0.30 0.99 ± 0.00
7
0.99 ± 0.00
L( N, V, D ) = Linf + A max( N a , λD b )−d V − g L( N, V, D ) = p0 N p1 V p2 D p3 + p4 N p5 + p6 L( N, V, D ) = A( NV k1 )−α + B( DV k2 )− β + c
7 7 7 7 7 7 7
0.83 ± 0.19 0.99 ± 0.00 0.77 ± 0.31 0.98 ± 0.02 0.97 ± 0.04 0.98 ± 0.00 0.98 ± 0.00
α A +E N (k log P+1) − α L( N, P) = c0 + c1 ( N + P− β ) −α L( N, P) = L0 + 1AN +k ln P b aN +c L( N, P) = 1+ d log P L( N, P) = AN −α P− β
L( N, P) =
sl 5 sl 6 sl 7 sl 8 sl 9 sl 10 sparsity sl 1 sl 2 sl 3 sl 4 vocab sl 1 sl 2 sl 3 sl 4 sl 5 sl 6 sl 7 sl 8 sl 9 sl 10
C
0
L( N, V, D ) = AN −α D − β + BV γ D −δ + c0 L( N, V, D ) = c0 + c1 log V + V β (c2 N −α + c3 D −γ ) L( N, V, D ) = AN −α D − β (1 + γ log V ) + δV ϵ + Linf L( N, V, D ) = Lmin + exp a + bP log N + bV1 log V + bV2 log2 V + bD log D + bVD log V log D
Basin Estimation and Posterior Approximation
We assume access to a set of locally optimal parameter vectors obtained by refitting the scaling law from multiple initializations on the current dataset Dt . This appendix explains how these local solutions are converted into the basin approximation K
p ( θ | D t ) ≈ ∑ w k N ( θ k , Σ k ). k =1
C.1
Local covariance approximation
We assume access to a collection of locally optimal parameter vectors M {θ̃m }m =1
obtained by refitting the scaling law from multiple initializations on the current dataset Dt . This subsection explains how each local solution is converted into a local Gaussian approximation, which will later be consolidated into the basin mixture K
p ( θ | D t ) ≈ ∑ w k N ( θ k , Σ k ). k =1
16
Preprint. Under review.
For a local optimum θ̃m , let
Lt (θ ) =
1 ∥ f ( xi ; θ ) − yi ∥22 |Dt | (x ,y∑)∈D i
i
t
denote the empirical mean-squared error on the currently observed data. Around θ̃m , we approximate the local curvature of this objective using a Gauss–Newton/Fisher-style matrix. Concretely, let Jm = J (θ̃m ; Dt ) be the Jacobian of model predictions with respect to the parameters, evaluated on all observations in Dt and flattened across data points (and output dimensions, when applicable). Given the current noise variance estimate σ2 , we define Hm =
1 ⊤ J Jm + Λprior , σ2 m
where Λprior is a diagonal prior-precision matrix used to stabilize weakly identified directions. We then approximate the local parameter uncertainty around θ̃m by −1 Σ̃m = Hm .
This yields the local Gaussian approximation qm (θ ) = N (θ̃m , Σ̃m ) for each candidate local optimum. For parameters constrained to be positive, we additionally rescale the corresponding Jacobian columns by the current parameter values before forming Hm . Equivalently, this amounts to measuring local sensitivity in a log-parameterization for those coordinates, which improves numerical stability when different parameters operate on very different scales. In implementation, the inversion of Hm is further stabilized by standard spectral regularization, including eigenvalue flooring and small diagonal correction when necessary. At this stage, each local refit θ̃m is associated with a Gaussian approximation qm (θ ). These candidate components are not yet the final basins in the mixture posterior, because several local optima may induce nearly identical predictive behavior. The next subsection therefore clusters these local solutions in prediction space and consolidates them into a smaller set of representative basins. C.2
Prediction-space clustering of local optima
The local Gaussian approximations constructed above are still over-complete: different local optima may correspond to essentially the same extrapolative behavior, even when their parameter values differ substantially. This is particularly common for nonlinear scaling laws with weakly identifiable directions. Since our downstream objective is prediction accuracy on a target region rather than parameter recovery itself, we consolidate local optima in prediction space rather than in parameter space. For each local solution θ̃m with covariance Σ̃m , we evaluate its induced predictive distribution on an evaluation set Xeval . In our implementation, Xeval is chosen to be the target region used by the acquisition function. Under the local linear approximation, the predictive distribution at a point x ∈ Xeval is f ( x; θ ) | θ ∼ qm ≈ N µm ( x ), vm ( x ) , where µm ( x ) = f ( x; θ̃m ), and Jx (θ̃m ) =
∂ f ( x;θ ) ∂θ θ =θ̃
vm ( x ) = Jx (θ̃m ) Σ̃m Jx (θ̃m )⊤ + σ2 ,
is the parameter Jacobian at x. m
17
Preprint. Under review.
We then measure the discrepancy between two local optima θ̃m and θ̃n by comparing their predictive Gaussians across Xeval . For scalar outputs, the pointwise symmetric KL divergence between
N (µm ( x ), vm ( x )) and N (µn ( x ), vn ( x )) is 1 SKLm,n ( x ) = 4
vm ( x ) vn ( x ) + − 2 + (µm ( x ) − µn ( x ))2 vn ( x ) vm ( x )
1 1 + vm ( x ) vn ( x )
.
Averaging over the evaluation region gives the prediction-space dissimilarity dmn =
1
∑ |Xeval | x∈X
SKLm,n ( x ).
eval
Using the pairwise dissimilarity matrix {dmn }, we perform agglomerative hierarchical clustering to group local optima with similar predictive behavior. The clustering threshold is selected data-adaptively by maximizing the silhouette score over candidate cuts of the hierarchy. This procedure yields a partition of the local optima into clusters
C1 , . . . , C K , which we interpret as the final basins. For each cluster Ck , we choose a single representative local optimum as the basin center. Specifically, we select the member with the smallest empirical fitting error on the current dataset: mk = arg min Lt (θ̃m ), θk = θ̃mk . m∈Ck
Its associated covariance approximation is inherited from the same representative: Σk = Σ̃mk . The output of this step is therefore a reduced collection of representative basins
{(θk , Σk )}kK=1 , which preserves distinct extrapolative behaviors while removing redundant local optima. The remaining ingredient is to assign mixture weights to these representative basins; this is described in the next subsection. C.3
Representative basins and mixture weights
After clustering the local optima in prediction space, we obtain a reduced set of representative basins {(θk , Σk )}kK=1 , where each representative parameter θk is selected from one prediction-space cluster, and Σk is the corresponding local covariance approximation inherited from that representative. To complete the mixture approximation, we associate each basin with a weight w k ≈ p ( B = k | D t ), where B ∈ {1, . . . , K } is a latent basin indicator specifying which basin generated the current local posterior approximation. Thus, wk represents the posterior probability that basin k is the relevant mode given the observations collected so far. In practice, exact computation of p( B = k | Dt ) is generally intractable, so we approximate it using a basin-level evidence score. We consider two natural choices. 18
Preprint. Under review.
Option 1: BIC-style approximation. A simple approximation is to rank basins by an information criterion derived from their empirical fit on the current dataset. Let MSEk = Lt (θk ) =
1 ∥ f ( xi ; θk ) − yi ∥22 |Dt | (x ,y∑)∈D i
i
t
and let nobs = |Dt |. We define BICk = nobs log(MSEk ) + p log nobs , where p is the number of free parameters. The basin probabilities are then approximated by k exp − BIC 2T , wk = BIC K ∑ℓ=1 exp − 2T ℓ where T > 0 is a temperature parameter. Option 2: Laplace-approximate basin posterior. A more direct approximation is obtained by locally approximating the contribution of each basin to the posterior normalizing constant. Let Hk denote the local curvature matrix at θk , with Σk ≈ Hk−1 . Then, up to a common normalization constant, |Dt | p( B = k | Dt ) ∝ π (θk ) exp − 2 Lt (θk ) | Hk |−1/2 , 2σ which yields the normalized weight |D | π (θk ) exp − 2σt2 Lt (θk ) | Hk |−1/2 wk = . |Dt | K −1/2 π ( θ ) exp − L ( θ ) | H | ∑ℓ= t ℓ ℓ ℓ 2 1 2σ In our experiments, we use the BIC-style approximation for robustness, while the Laplace form provides a more principled local-evidence interpretation of the same quantity.
D
Derivation of the Acquisition Function
This appendix derives the target-aware acquisition function used in the main text. Starting from the basin-mixture approximation K
p ( θ | D t ) ≈ ∑ w k N ( θ k , Σ k ), k =1
we show how the target-region uncertainty measure MSPEtar =
1 Eθ ∼ p(θ |Dt ) ∥ F (θ ) − f¯∥22 |Xtar |
decomposes into intra-basin and inter-basin terms, and how this decomposition leads to the candidate utility ∆MSPEtar ( x ) = ∆Vintra ( x ) + ∆Vinter ( x ). Throughout, the key approximation is local: within each basin, we linearize the predictor around its representative parameter θk , while retaining the multimodal mixture structure across basins. 19
Preprint. Under review.
D.1
Local linearization within each basin
Fix a basin k with local posterior approximation θ | ( B = k, Dt ) ≈ N (θk , Σk ). To obtain tractable expressions for predictive uncertainty and posterior updates after adding a new observation, we linearize the scaling law around the basin representative θk . For the target region Xtar , recall the prediction map F (θ ) = f ( x; θ ) x∈X ∈ R|Xtar | . tar
A first-order Taylor expansion around θk gives F (θ ) ≈ F (θk ) + Jk (θ − θk ), where Jk =
∂F (θ ) ∈ R|Xtar |× p . ∂θ θ =θk
Writing fˆk = F (θk ), we obtain the Gaussian approximation F (θ ) | ( B = k, Dt ) ≈ N fˆk , Jk Σk Jk⊤ . Similarly, for a candidate experiment x ∈ Xcand , define the scalar predictive mean mk ( x ) = f ( x; θk ) and the parameter Jacobian jk ( x ) =
∂ f ( x; θ ) ∈ Rp. ∂θ θ =θk
Then the same first-order expansion yields f ( x; θ ) ≈ mk ( x ) + jk ( x )⊤ (θ − θk ), so under the local Gaussian approximation, y | ( x, B = k, Dt ) ≈ N mk ( x ), s2k ( x ) ,
s2k ( x ) = σ2 + jk ( x )⊤ Σk jk ( x ).
The corresponding posterior update within basin k remains Gaussian. After observing ( x, y), the updated covariance is given by the standard rank-one linear-Gaussian update Σ+ k ( x ) = Σk −
Σk jk ( x ) jk ( x )⊤ Σk , s2k ( x )
and the posterior mean of the target-region prediction vector updates as fˆk+ ( x, y) = fˆk + gk ( x ) y − mk ( x ) ,
gk ( x ) =
Jk Σk jk ( x ) ∈ R|Xtar | . s2k ( x )
These local update formulas are the basic ingredients for the derivations below: Σ+ k (x) + ˆ determines the reduction in within-basin predictive variance, while f k ( x, y) determines how a new observation changes the disagreement between basins. 20
Preprint. Under review.
D.2
Decomposition of target-region MSPE
We now derive the decomposition of the target-region uncertainty objective used in the main text. Recall that 1 MSPEtar = Eθ ∼ p(θ |Dt ) ∥ F (θ ) − f¯∥22 , f¯ = Eθ ∼ p(θ |Dt ) [ F (θ )]. |Xtar | Introducing the latent basin indicator B ∈ {1, . . . , K }, the mixture approximation can be written as K
p(θ | Dt ) ≈ ∑ p( B = k | Dt ) p(θ | B = k, Dt ), k =1
with
p ( B = k | Dt ) ≈ wk , p(θ | B = k, Dt ) ≈ N (θk , Σk ). Under the local linearization from the previous subsection, F (θ ) | ( B = k, Dt ) ≈ N fˆk , Jk Σk Jk⊤ , fˆk = F (θk ).
The target-region MSPE is simply the total variance of the random vector F (θ ), normalized by |Xtar |. Applying the law of total variance with respect to the basin indicator B gives 1 MSPEtar = EB tr Cov( F (θ ) | B, Dt ) + tr Cov(E[ F (θ ) | B, Dt ]) . |Xtar | We evaluate the two terms separately. For the first term, conditioning on basin k yields Cov( F (θ ) | B = k, Dt ) ≈ Jk Σk Jk⊤ , so
K EB tr Cov( F (θ ) | B, Dt ) ≈ ∑ wk tr( Jk Σk Jk⊤ ). k =1
For the second term, the conditional mean is E[ F (θ ) | B = k, Dt ] ≈ fˆk , and the global posterior mean is therefore K
f¯ = E[ F (θ ) | Dt ] ≈ ∑ wk fˆk . k =1
Hence, K tr Cov(E[ F (θ ) | B, Dt ]) = ∑ wk ∥ fˆk − f¯∥22 . k =1
Combining the two terms yields MSPEtar =
K K 1 1 wk tr( Jk Σk Jk⊤ ) + w ∥ fˆ − f¯∥22 . ∑ ∑ |Xtar | k=1 |Xtar | k=1 k k
We therefore define Vintra =
K 1 w tr( Jk Σk Jk⊤ ), ∑ |Xtar | k=1 k
Vinter =
K 1 w ∥ fˆ − f¯∥22 , ∑ |Xtar | k=1 k k
so that MSPEtar = Vintra + Vinter . The term Vintra measures the average predictive variance that remains within each basin after conditioning on which local mode is correct, while Vinter measures the residual disagreement between basin-level extrapolations. This decomposition is the basis for our acquisition function: a useful new experiment should either reduce uncertainty within a plausible basin or help distinguish between basins that extrapolate differently. 21
Preprint. Under review.
D.3
Derivation of the intra-basin utility
We now derive the reduction in the within-basin term Vintra =
K 1 wk tr( Jk Σk Jk⊤ ) |Xtar | k∑ =1
after querying a candidate experiment x. After observing an outcome y at x, both the basin posterior probabilities and the within-basin covariances are updated. The exact updated intra-basin term is therefore + Vintra ( x, y) =
where
K 1 ( x ) Jk⊤ , wk+ ( x, y) tr Jk Σ+ ∑ k |Xtar | k=1
wk+ ( x, y) = p( B = k | x, y, Dt )
and Σ+ k ( x ) = Σk −
Σk jk ( x ) jk ( x )⊤ Σk , s2k ( x )
s2k ( x ) = σ2 + jk ( x )⊤ Σk jk ( x ).
Note that Σ+ k ( x ) depends on the queried location x but not on the realized value y. The intra-basin utility is defined by + ∆Vintra ( x ) = Vintra − Ey| x,Dt Vintra ( x, y) . Substituting the expression above gives + ( x, y) = Ey| x,Dt Vintra
K + 1 ⊤ E w ( x, y) tr Jk Σ+ ∑ k ( x ) Jk . |Xtar | k=1 y| x,Dt k
It remains to evaluate the expectation of the updated basin weights. By Bayes’ rule, wk+ ( x, y) =
wk p(y | x, B = k, Dt ) . p(y | x, Dt )
Therefore, Ey| x,Dt [wk+ ( x, y)] =
Z
= wk
wk p(y | x, B = k, Dt ) p(y | x, Dt ) dy p(y | x, Dt ) Z
p(y | x, B = k, Dt ) dy = wk .
Hence, + Ey| x,Dt Vintra ( x, y) = and thus ∆Vintra ( x ) =
K 1 wk tr Jk Σ+ ( x ) Jk⊤ , ∑ k |Xtar | k=1
h i K 1 ⊤ wk tr( Jk Σk Jk⊤ ) − tr( Jk Σ+ ( x ) J ) . ∑ k k |Xtar | k=1
Substituting the rank-one update for Σ+ k ( x ) yields K Σk jk ( x ) jk ( x )⊤ Σk ⊤ 1 ∆Vintra ( x ) = w tr J Jk k k |Xtar | k∑ s2k ( x ) =1
!
Using cyclic invariance of the trace, tr Jk Σk jk ( x ) jk ( x )⊤ Σk Jk⊤ = jk ( x )⊤ Σk Jk⊤ Jk Σk jk ( x ), 22
.
Preprint. Under review.
so ∆Vintra ( x ) =
K jk ( x )⊤ Σk Jk⊤ Jk Σk jk ( x ) 1 w . k |Xtar | k∑ s2k ( x ) =1
∆Vintra ( x ) =
K ∥ Jk Σk jk ( x )∥22 1 wk 2 . ∑ |Xtar | k=1 σ + jk ( x )⊤ Σk jk ( x )
Equivalently,
D.4
Derivation of the inter-basin utility
We now derive the reduction in the between-basin term Vinter =
K 1 w ∥ fˆ − f¯∥22 , ∑ |Xtar | k=1 k k
K
f¯ = ∑ wk fˆk , k =1
after querying a candidate experiment x. Unlike the intra-basin term, the inter-basin term depends on the relative positions and weights of the basin-level predictions. After observing an outcome y at x, both quantities change: the posterior probability of each basin is updated, and within each basin the target-region prediction mean is shifted by the new observation. Updated basin weights. By Bayes’ rule, the posterior probability of basin k after observing ( x, y) is w p(y | x, B = k, Dt ) wk+ ( x, y) = p( B = k | x, y, Dt ) = k . p(y | x, Dt ) Under the local linear-Gaussian approximation, p(y | x, B = k, Dt ) ≈ ϕ y; mk ( x ), s2k ( x ) , where
s2k ( x ) = σ2 + jk ( x )⊤ Σk jk ( x ),
mk ( x ) = f ( x; θk ),
and ϕ( · ; µ, ν) denotes the Gaussian density with mean µ and variance ν. Therefore, wk ϕ y; mk ( x ), s2k ( x ) wk+ ( x, y) = K . ∑ℓ=1 wℓ ϕ y; mℓ ( x ), s2ℓ ( x ) Updated basin-level target predictions. Within basin k, the posterior mean of the targetregion prediction vector updates according to fˆk+ ( x, y) = fˆk + gk ( x ) y − mk ( x ) ,
gk ( x ) =
Jk Σk jk ( x ) ∈ R|Xtar | . s2k ( x )
Thus the updated global posterior mean on the target region is K
f¯+ ( x, y) = ∑ wk+ ( x, y) fˆk+ ( x, y). k =1
Updated inter-basin uncertainty. basin term becomes + Vinter ( x, y) =
Conditioned on the new observation ( x, y), the between-
K 1 wk+ ( x, y) |Xtar | k∑ =1
fˆk+ ( x, y) − f¯+ ( x, y)
Accordingly, the inter-basin utility of candidate x is + ∆Vinter ( x ) = Vinter − Ey| x,Dt Vinter ( x, y) . 23
2 2
.
Preprint. Under review.
The predictive distribution of y under the current mixture is K p(y | x, Dt ) = ∑ wk ϕ y; mk ( x ), s2k ( x ) , k =1
so the expectation above can be written explicitly as the one-dimensional integral + Z + Ey| x,Dt Vinter ( x, y) = Vinter ( x, y) p(y | x, Dt ) dy. In contrast to the intra-basin case, this expectation does not collapse to a simpler weighted average, because both the updated responsibilities wk+ ( x, y) and the updated basin predictions fˆk+ ( x, y) depend nonlinearly on the realized outcome y. Intuitively, a candidate receives high inter-basin utility when different basins predict substantially different outcomes at x, so that observing y is likely to either reweight the basin probabilities or pull their target-region predictions closer together. In the next subsection, we show that this integral admits an efficient pairwise form that can be evaluated by one-dimensional numerical quadrature. D.5
Pairwise form and one-dimensional quadrature
A convenient identity for the between-basin variance is K
∑ wk ∥ fˆk − f¯∥22 =
k =1
∑
wk wℓ ∥ fˆk − fˆℓ ∥22 .
1≤k<ℓ≤K
Applying this identity to the updated basin mixture yields + Vinter ( x, y) =
1 wk+ ( x, y) wℓ+ ( x, y) |Xtar | 1≤k∑ <ℓ≤K
fˆk+ ( x, y) − fˆℓ+ ( x, y)
2 2
.
This pairwise form is more convenient than the centered form because it separates the contribution of each basin pair and avoids recomputing the global mean explicitly. Using the linear update fˆk+ ( x, y) = fˆk + gk ( x ) y − mk ( x ) , the difference between two updated basin means is fˆk+ ( x, y) − fˆℓ+ ( x, y) = fˆk − fˆℓ − gk ( x )mk ( x ) + gℓ ( x )mℓ ( x ) + gk ( x ) − gℓ ( x ) y. | {z } | {z } akℓ ( x )
bk ℓ ( x )
Hence, for each pair (k, ℓ), fˆk+ ( x, y) − fˆℓ+ ( x, y)
2 2
= ∥ akℓ ( x ) + bkℓ ( x )y∥22 ,
which is a quadratic polynomial in y: fˆk+ ( x, y) − fˆℓ+ ( x, y)
2 2
= Akℓ ( x ) + Bkℓ ( x ) y + Ckℓ ( x ) y2 ,
with coefficients Akℓ ( x ) = ∥ akℓ ( x )∥22 ,
Bkℓ ( x ) = 2 akℓ ( x )⊤ bkℓ ( x ),
Ckℓ ( x ) = ∥bkℓ ( x )∥22 .
Next, recall that the updated basin weights satisfy wk+ ( x, y) =
wk ϕk (y; x ) , K ∑r=1 wr ϕr (y; x )
ϕk (y; x ) = ϕ y; mk ( x ), s2k ( x ) . 24
Preprint. Under review.
Therefore, for any pair (k, ℓ), wk+ ( x, y) wℓ+ ( x, y) p(y | x, Dt ) =
wk wℓ ϕk (y; x )ϕℓ (y; x ) . ∑rK=1 wr ϕr (y; x )
Substituting this into the expectation of the updated inter-basin term gives Z + 1 ϕk (y; x )ϕℓ (y; x ) Ey| x,Dt Vinter ( x, y) = w w Akℓ ( x ) + Bkℓ ( x ) y + Ckℓ ( x ) y2 dy. ∑ k ℓ K |Xtar | 1≤k<ℓ≤K ∑r=1 wr ϕr (y; x ) This is the form used in our implementation. For a fixed candidate x, the expectation reduces to a one-dimensional integral over the scalar observation y, and the only dependence on the target region enters through the precomputed coefficient vectors akℓ ( x ) and bkℓ ( x ). Consequently, the inter-basin utility can be evaluated efficiently by numerical quadrature even when the target region contains many points. Finally, combining this expression with 1 Vinter = wk wℓ ∥ fˆk − fˆℓ ∥22 |Xtar | 1≤k∑ <ℓ≤K gives + ( x, y) . ∆Vinter ( x ) = Vinter − Ey| x,Dt Vinter In practice, we evaluate the one-dimensional integral numerically on a finite grid covering the predictive support of the current basin mixture at the candidate point. D.6
Final cost-aware acquisition score
Combining the two components derived above, the total expected reduction in target-region uncertainty from querying candidate x is ∆MSPEtar ( x ) = ∆Vintra ( x ) + ∆Vinter ( x ). The intra-basin term captures how much the new observation is expected to reduce local predictive variance within each plausible basin, while the inter-basin term captures how much it is expected to reduce disagreement across basins. To account for heterogeneous experiment costs, we rank candidates using the cost-aware score ∆MSPEtar ( x ) ∆Vintra ( x ) + ∆Vinter ( x ) S( x ) = = , c( x )α c( x )α where c( x ) is the cost of running experiment x, and α ≥ 0 controls the strength of cost penalization. This form has a natural interpretation. When basin ambiguity is large, the inter-basin term tends to dominate, so the design prefers experiments that distinguish between qualitatively different extrapolations. Once the correct basin is largely identified, the acquisition increasingly behaves like a target-aware local optimal-design criterion, favoring experiments that most reduce predictive variance on Xtar per unit cost. This is exactly the behavior desired in budget-constrained scaling-law fitting: early experiments should resolve global ambiguity, while later experiments should refine the locally relevant scaling trend.
E
Discussions, Limitations, and Future Work
Our study has several limitations. The proposed method depends on a mixture-based approximation to multimodal parameter uncertainty, which may be inaccurate when local optima are poorly identified or when the scaling law is severely misspecified. Moreover, our acquisition rule is one-step and does not explicitly optimize long-horizon budget allocation. Although our benchmark covers diverse scaling scenarios, it is still based on a finite suite of law families, candidate pools, and simplified cost proxies. Future work includes more robust posterior approximations, multi-step budget-aware design, and extensions to broader scaling settings with richer experiment spaces and more realistic cost models. 25