How Numerical Errors Scale with Hardware

HOW NUMERICAL ERRORS SCALE WITH HARDWARE

arXiv:2610.35413v1 [cs.DC] 26 Oct 2026

NIKLAS BAUMGARTEN

Abstract. In this short technical note, we outline how to bound numerical errors with respect to computational resources. We consider algorithms with known complexity estimates that can be executed in parallel and combine these estimates with parallelization laws. We suppose that the proposed approach generalizes beyond the presented examples and argue that it provides a straightforward and fair framework for comparing algorithms, hardware systems, and software implementations. This is achieved by imposing identical resource or energy budgets on each configuration, and comparing the resulting numerical error evaluations.

1

Introduction

Energy consumption and the availability of hardware resources are among the main limiting factors in achieving better computational results. This is particularly true for highly parallel applications, where large computing systems work toward a common goal, such as estimating quantities of interest for highly uncertain processes, simulating fluid flow in three spatial dimensions, or training a large language model. To describe the performance of algorithms on parallel hardware, Amdahl’s law [1] characterizes the maximum speedup achievable for a fixed-size problem when only a portion of the computation can be parallelized. Gustafson’s law [2] revisits this analysis by assuming that the problem size scales with the number of processing units, deriving a corresponding scaled speedup relative to serial execution. For many numerical algorithms in scientific computing and machine learning, however, the problem size is related nonlinearly to the numerical error one seeks to minimize. Neural scaling laws [3], for example, investigate how the number of model parameters and the amount of training data influence the training loss. When the amount of training data and the number of model parameters are scaled in tandem with a compute-optimal strategy, the training loss can be expressed directly as a function of the total training compute [4]. In this note, we want to outline how to connect parallelization laws with error-vs-cost relations, formally described by complexity analysis. This creates a direct connection between hardware resources and achievable accuracies, and thereof a direct relation between errors and invested energy. For this we establish a relative error reduction, rather than the speedup as in [1, 2], achieved by executing an algorithm on a parallel system. The here outlined proposition summarizes and generalizes observations made in [5, 6, 7]. The approach is motivated by a knapsack problem: minimize a numerical error, subject to the constraints imposed by the computing system. These constraints typically include the reserved computing time, the available memory or data and the number of processing units such as CPUs, GPUs, or TPUs.

Date: October 27, 2026. niklas.baumgarten@uni-heidelberg.de 1

2

NIKLAS BAUMGARTEN

Problem 1.1 (Minimal Error Knapsack). Find an algorithm {πk }k (a sequence of decisions) that minimizes some error estimate err c k of the problem of interest within given computational constraints Mem min err c k s.t. COp < M and CData < D. k k ≤ P · T, Ck

{πk }k

We denote by COp the operational cost, by CMem the maximal memory footprint over the k k Data runtime, and by Ck the amount of data processed up until step k. The constraints comprise a time budget T > 0, a fixed number of computing units P ∈ N, the total memory capacity of the system M > 0 and the total amount of available data D ≥ 0. Finding solutions to the resource allocation Theorem 1.1 through an optimal sequence of decisions {πk }k is an NP-hard optimization problem, likely with non-linear objective functions and constraints. Expressing this non-linear relationship between the error err c k (e.g., an estimator variance, a finite element error, or a training loss) and the operational complexity COp k (e.g., the number of necessary floating point operations, or the number of function evaluations) is commonly done though complexity analysis identifying bounds of the form: For any fixed accuracy ϵ ∈ (0, 1], the expression f : (0, 1] → R≥0 bounds the operational complexity (1.1)

COp ≤ c f (ϵ) ϵ

to achieve

err c < ϵ.

The constant c > 0 is generally not known and may dependent on the condition-number of the problem as well as the actual implementation of the algorithm. The function f is assumed to be monotonically increasing as ϵ → 0, i.e., achieving smaller errors is associated with higher computational costs. This is a natural result everywhere in machine learning or scientific computing, causing long computing times and large hardware requirements in practice. Complexity analysis allows us to quantify how good an algorithm {πk }k is in terms of the resources it needs to achieve a certain accuracy ϵ ∈ (0, 1], enabling comparisons of different algorithms with respect to their resource requirements. The memory and data constraints in Theorem 1.1 typically determine the smallest achievable error for a given algorithm. For example, in machine learning, limited training data introduces bias or may lead to overfitting. Similarly, insufficient memory can prevent the construction of a larger finite element basis in numerical simulations. Despite their importance, memory and data constraints are here not the primary investigation goal, we will come back to this only shortly in a final discussion.

2

Proposition

While very useful, complexity analysis alone does not address key questions in practice: Which error can be achieved within a given time frame? How much can additional hardware resources improve the result? Or, what is the smallest possible error that can be achieved on a given system? This becomes even less clear if we consider the influence of parallelization inefficiencies and non-uniform memory architectures. Addressing the above requires precise knowledge of the constant c in (1.1). In practice, however, this constant is usually unknown due to incomplete information about the problem of interest and the complete implementation stack. By instead considering Theorem 1.1, we can circumvent knowing constant c in (1.1), at the cost of designing algorithms {πk }k that implement greedy strategies or dynamic programming techniques. For algorithms that (i) solve Theorem 1.1, (ii) are implemented in parallel, and (iii) for which the dominant term in the complexity estimate is multiplicative and can be distributed across processing units, we state the following.

HOW NUMERICAL ERRORS SCALE WITH HARDWARE

3

Proposition 2.1. Let λs ∈ (0, 1] and λp ∈ [0, 1) represent the non-parallelizable and parallelizable fractions of algorithm {πk }k such that λs + λp = 1. Suppose this algorithm follows the operational complexity (1.1) and that f : (0, 1] → R≥0 is multiplicative (e.g. a power-law), i.e., f (ϵ1 · ϵ2 ) = f (ϵ1 ) · f (ϵ2 )

(2.1) and invertible, i.e., f (2.2)

−1

: R≥0 → (0, 1] exists. Then err c k ≲ λs f −1 (T ) + λp f −1 (P · T ) ,

if every processing unit can handle a sufficiently large portion of the dominant term in the complexity estimate. Proof. Following [7, 5], we present generalized arguments. Let COp ϵ,ref denote the serial execution time required to reach accuracy ϵ ∈ (0, 1]. Similar to [1, 2], we decompose the execution time into a parallelizable λp ∈ [0, 1) and a non-parallelizable fraction λs = (1 − λp ) by Op Op COp ϵ,ref = (1 − λp )Cϵ,ref + λp Cϵ,ref .

On P ∈ N processing units, the execution time reduces to Op Op ∗ COp ϵ,par = (1 − λp )Cϵ,ref + λp f (P )Cϵ,ref ,

where f ∗ : N → (0, 1] is monotone decreasing, speeding up the computation to reach accuracy ϵ ∈ (0, 1]. The resulting reduction factor is Op ∗ R := COp ϵ,par /Cϵ,ref = (1 − λp ) + λp f (P ) ≤ 1 .

Now, we fix a time budget T > 0 and search for the achievable accuracies ϵref on serial and ϵpar on parallel hardware. Inverting (1.1) yields ϵref ≲ f −1 (T ) on serial hardware. For parallel hardware, using the reduction R gives ϵpar ≲ R · f −1 (T ) = (1 − λp )f −1 (T ) + λp f ∗ (P )f −1 (T ). The stated bound follows by (2.1) and if f ∗ (P ) = f −1 (P ) on N, which holds when each processing unit handles a sufficiently large share of the total workload. □ Theorem 2.1 combines complexity estimates of the form (1.1) with parallelization laws. It assumes that ϵpar scales similarly to R · ϵref and that processing units are interchangeable with computing time in the parallelizable part of the complexity. This behavior has been observed in practice (e.g. [7, 5]) and we conjecture that it also holds for other algorithms, discussing applications in the next section. Before doing so, we draw attention to Figure 1, based on [5, Fig. 7] and [7, Fig. 12]. The left illustration in Figure 1 captures the diminishing error reductions obtained with additional hardware. This can be seen by comparing the distance ∆1 between the lines for P0 and 2P0 with the distance ∆2 between those for 8P0 and 16P0 . The latter is smaller, ∆2 < ∆1 , which can be explained by (2.2): as P increases, the first term, which is independent of P , becomes increasingly important. To illustrate this, the right panel of Figure 1 takes the achieved error estimates err c k at the fixed final time T and plots them against P ∈ {P0 , 2P0 , . . . , 16P0 }. Equation (2.2) then becomes (2.3)

err c k ≲ ϵs + cp f −1 (P ) ,

where ϵs and cp can be determined by regression, since f −1 is either known theoretically or observed empirically as a function of T . This regression yields an error within the measurement noise if f ∗ (P ) = f −1 (P ) and thus, observing the right plot of Figure 1 confirms the proposition and that its assumptions are given. In [7], it is motivated to call the term ϵs parallelization bias, as P → ∞, the bound (2.2) reduces to err c k ≲ λs f −1 (T ) =: ϵs . Thus, regardless of how much hardware is available, the first

4

NIKLAS BAUMGARTEN err ck

err ck ∆1

P0 2 P0 ... ... 16 P0

∆2

0

P0

T

16 P0

Figure 1. Left: Example of log-log error decay over time points up until T with a power law as complexity function in (1.1) and for different values of P ∈ {P0 , 2P0 , . . . , 16P0 }, P0 ∈ N. Right: Achieved error estimates err c k within a fixed time budget T and line fit as dashed line to (2.3). summand can only be reduced by increasing the available computing time. In practice, the NPhardness of Theorem 1.1 can be addressed with greedy algorithms or dynamic programming, both of which require the algorithm to exchange information at some point, implying λs ̸= 0 and thereof ϵs ̸= 0. As a consequence of bound (2.2), we can also impose energy budgets, i.e., given a fixed total amount of Watt-hours, we identify the best algorithm, hardware system, or software implementation by comparing the achieved errors. Here, the first term on the right-hand side of (2.2) is associated with the energy consumption of the predominantly idle system, whereas the second term directly translates the number of processing units into power requirements. Based on electricity prices and energy mix, even monetary or carbon dioxide budgets may be imposed.

3

Examples

The observation made in Theorem 2.1 can be applied to a wide range of algorithms. We shortly discuss four examples. Example 3.1 (Monte Carlo Estimation). Let X be a random variable with unknown finite variance. A Monte Carlo method to estimate the expected value E[X] is given by M

(3.1)

MC EM,s [X] =

1 X (m) X , with X (m) i.i.d. . M m=1

√ The root mean squared error follows by the central limit theorem O(1/ M ), i.e., f (ϵ) = ϵ−2 . Estimator (3.1) can be implemented greedily and embarrassingly parallel to solve Theorem 1.1 and thus satisfies all assumptions in Theorem 2.1. Numerical verification of this can be found in [7], applying multilevel Monte Carlo methods to acoustic wave equations. Example 3.2 (Gradient Descent). Let x∗ = argminx∈RN F (x). If F is µ-strongly convex, L-smooth and for sufficient step size τ > 0, the iteration scheme (3.2)

x(k+1) = x(k) − τ ∇F (x(k) )

converges linearly, and achieving some ϵ ∈ (0, 1], results in O(log(1/ϵ)) number of steps [8, Sec. 9.3]. Since this does not satisfy the assumption and (3.2) is a sequential iteration scheme, Theorem 2.1 is not directly applicable. However, if computational resources also have to be invested into other tasks, e.g function evaluation or batch estimation, and aligning the iteration

HOW NUMERICAL ERRORS SCALE WITH HARDWARE

5

index of (3.2) with algorithm {πk }k , Theorem 2.1 again becomes applicable. For example, estimate (2.2) is observed in [5] for variants of batched stochastic gradient descent. Example 3.3 (Finite Element Methods). FEMs approximate functions on bounded spatial domains D ⊂ Rd by decomposing them into finitely many cells of diameter h > 0, resulting in large algebraic systems. The discretization error typically decays as O(hα ), where the rate α > 0 depends on the regularity of the solution and the chosen discretization scheme, whereas the computational cost, assuming optimal solvers, scales as O(h−d ). Achieving a FEM error below ϵ ∈ (0, 1] therefore requires computational work following f (ϵ) = ϵ−d/α (see, e.g., [9]). However, finding an algorithm that implements a FEM and solves Theorem 1.1 is challenging. In practice, the mesh size h is typically refined geometrically, so the constraints must be adapted accordingly. For example, halving h requires eight folding P for d = 3 and a fixed T > 0. Consequently, solving Theorem 1.1 amounts to a classical weak scaling experiment. Despite the fact that FEM and solving Theorem 1.1 can not be combined in a flexible way, embedding FEM within statistical estimators or gradient descent methods make it possible again to apply Theorem 2.1 as done in [5, 7]. Example 3.4 (Training of Neural Networks). The neural scaling law [4, Eq. (1.3)], setting the min training loss L and a compute budget Cmin in relation through the rate αC > 0, (3.3)

min

L(Cmin ) ∝ (Cmin )−αC

has been found through balancing dataset size and trainable parameters, optimally allocating the compute budget, which is related to solving Theorem 1.1. Equation (3.3) presents a similar as (2.2), however, we argue that separating Cmin into P · T presents deeper insights how additional hardware actually improves the computational result. Hence, we argue that Theorem 2.1 is applicable to neural scaling laws for compute.

4

Conclusion and Discussion

Theorem 1.1, which is commonly encountered in machine learning and scientific computing, can be viewed similarly to approaches in the Theory of Constraints [10, 11]. It identifies a performance measure (a numerical error), as well as system constraints (e.g., available processing units, allocated computing time, memory capacity, data availability, or even energy budgets). The validity of Theorem 2.1 requires examination beyond the presented examples, but it provides a straightforward argument for whether and how lifting constraints improves computational results. This note focuses on exploiting the computing-time constraint by combining complexity analysis with parallelization laws, leading to Theorem 2.1. Depending on the application, memory capacity may instead be the limiting factor; see, e.g., [6]. Similarly, data availability may become the dominant constraint, as outlined, for example, by scaling laws [4] with respect to dataset size. With this note, we advocate for approaching computational problems from the perspective of physical constraints, detailing in particular how increasing the number of computing units affects numerical errors.

References [1] G. M. Amdahl, “Validity of the single processor approach to achieving large scale computing capabilities,” in Proceedings of the April 18-20, 1967, spring joint computer conference, pp. 483–485, 1967. [2] J. L. Gustafson, “Reevaluating Amdahl’s law,” Communications of the ACM, vol. 31, no. 5, pp. 532–533, 1988. [3] Y. Bahri, E. Dyer, J. Kaplan, J. Lee, and U. Sharma, “Explaining neural scaling laws,” Proceedings of the National Academy of Sciences, vol. 121, no. 27, p. e2311878121, 2024.

6

NIKLAS BAUMGARTEN

[4] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” arXiv preprint arXiv:2001.08361, 2020. [5] N. Baumgarten and D. Schneiderhan, “Multilevel stochastic gradient descent for optimal control under uncertainty,” arXiv preprint arXiv:2506.02647, 2025. [6] N. Baumgarten, R. Kutri, and R. Scheichl, “A budgeted multi-level monte carlo method for full field estimates of multi-pde problems,” 2025. [7] N. Baumgarten, S. Krumscheid, and C. Wieners, “A fully parallelized and budgeted multilevel monte carlo method and the application to acoustic waves,” SIAM/ASA Journal on Uncertainty Quantification, vol. 12, no. 3, pp. 901–931, 2024. [8] S. Boyd and L. Vandenberghe, Convex optimization. Cambridge university press, 2004. [9] K. A. Cliffe, M. B. Giles, R. Scheichl, and A. L. Teckentrup, “Multilevel Monte Carlo methods and applications to elliptic PDEs with random coefficients,” Comput. Vis. Sci., vol. 14, no. 1, pp. 3–15, 2011. [10] E. M. Goldratt and J. Cox, The goal: a process of ongoing improvement. Routledge, 2016. [11] Z. T. Şimşit, N. S. Günay, and Ö. Vayvay, “Theory of constraints: A literature review,” Procedia-Social and Behavioral Sciences, vol. 150, pp. 930–936, 2014.

Related documents

• How Numerical Errors Scale with Hardware#1458883
• The Unseen Delta: Characterizing the Compiler Optimization Landscape via Top-Down Differential Analysis#927420
• Ribbon: Scalable Approximation and Robust Uncertainty Quantification#310812
• Deterministic LLM Inference Across GPU Kernels: Power-of-Two INT8 Quantization Scales and the Limits of Tolerance-Based Conformance#819815
• Deterministic LLM Inference Across GPU Kernels: Power-of-Two INT8 Quantization Scales and the Limits of Tolerance-Based Conformance#809819