NumericalDiff: Numerical Program Differencing via Value-Based Trace Alignment and Intervention

N UMERICAL D IFF : N UMERICAL P ROGRAM D IFFERENCING VIA VALUE -BASED T RACE A LIGNMENT AND I NTERVENTION

arXiv:2611.11213v1 [cs.SE] 8 Nov 2026

A P REPRINT Xiang Chen Huanhuan Xia Zhongguancun Academy, Beijing, China Zhongguancun Institute of Artificial Intelligence, Beijing, China chenxiang@bza.edu.cn huanhuanxia@zgci.ac.cn

November 10, 2026

A BSTRACT Numerical programs implementing the same computational task can exhibit different numerical behavior because of their execution stacks, program representations, or program logic. Source text and final outputs alone provide limited diagnostic information: numerical differences can propagate through corresponding computations, while distinct executions can produce equal outputs. We present NumericalDiff, an execution-based method that records assignment-level numerical events, aligns traces using a tolerance-aware longest common subsequence independent of variable names and source locations, and optionally reexecutes after replacing one intermediate value with its counterpart. Among the original and all single-intervention executions, it lexicographically maximizes matched events and minimizes unmatched events. NumericalDiff supports code comparison across language and framework front ends, currently including NumPy, PyTorch, JAX, C, C++, Fortran, and Octave. We evaluate NumericalDiff on 36 controlled cases comprising 675 pair/input instances: 36 execution-stack, 588 program-representation, and 51 program-logic instances. All 24 operatorinjection instances expose trace differences, including three with tolerance-equal final outputs, while complete trace alignment is achieved for all 441 canonical cross-representation instances. Within the 51 program-logic instances, intervention improves 24 alignments and raises macro-average matching coverage from 0.4210 to 0.6513. The results show that value-based trace alignment and reexecution can delimit propagated numerical differences across heterogeneous implementations.

1

Introduction

Comparing two programs or two versions of a program is a recurrent task in software engineering. During testing and debugging, developers compare a failing execution with a passing one to locate the first behavior that matters; during maintenance and operation, they compare releases, configurations, and deployments to diagnose regressions; and during migration, they compare an old implementation with its replacement to determine whether behavior has been preserved. Existing techniques approach these tasks at several levels. Source differencing identifies textual or syntactic changes, with modern tree differencers providing fine-grained mappings between program elements [Falleri and Martinez, 2024]. Semantic differencing and differential symbolic execution characterize behavioral changes beyond syntax [Jackson and Ladd, 1994, Person et al., 2008], while comparative execution, slicing, and trace-based debugging explain how observed executions diverge [Sumner and Zhang, 2013a, Suzuki et al., 2017]. Comparing numerical programs has become increasingly important as numerical software is getting more widely deployed. One prominent application is accuracy alignment for deep-learning systems. The same deep-learning model may run on CPUs, GPUs, or accelerators; use different operator libraries, runtimes, and numerical formats; or be implemented in PyTorch, JAX, and other frameworks. Such changes can preserve the intended model while producing different intermediate tensors or outputs. Cross-backend validation has exposed and localized faults in deep-learning libraries [Pham et al., 2019], and differential testing continues to find discrepancies across numerical libraries and

NumericalDiff

A P REPRINT

Figure 1: Demonstration of Code Comparison with NumericalDiff.

heterogeneous platforms [Vanover et al., 2020, Zhang et al., 2021a]. This need is especially visible for large models, where a small local discrepancy may flow through many layers before it becomes an observable loss of accuracy. Another recurrent application is scientific-software migration, in which a numerical program is reimplemented across languages such as Fortran, MATLAB, C/C++, Python, or Julia while preserving its intended numerical semantics. Migration can change the language, data layout, library calls, evaluation order, and compiler behavior simultaneously. Comparing only final outputs can confirm that a test failed, but it does not reveal where two implementations first became meaningfully different. Relative debugging and numerical-drift analysis therefore compare internal program states and, in some cases, replace an observed value with a reference value to test whether later computation reconverges [Abramson et al., 2009, Collins et al., 2013]. Figure 2 demonstrates how numerical semantic differences are introduced from different layers. In execution-stack variation, the same source code runs with different hardware, drivers, compilers, interpreters, runtimes, libraries, numerical formats, or their configurations; Figure 2a shows the same Newton iteration code cycling in float32 but converging in float64. In program-representation variation, the same numerical algorithm is expressed in different languages or frameworks; Figure 2b gives corresponding PyTorch and Fortran self-attention implementations. In program-logic variation, the same task is implemented with different algorithms; Figure 2c contrasts iterative and closed-form triangular summation methods, which achieve the exactly same functionality but have different dynamic assignment structures. Prior work addresses complementary parts of the comparison challenges illustrated above. Abstract semantic differencing reasons statically about numerical relations between programs [Partush and Yahav, 2013, Delmas and Miné, 2019a]; cross-platform testers and floating-point debuggers detect output inconsistencies or trace numerical error [Sawaya et al., 2017, Sanchez-Stern et al., 2018, Guo et al., 2020]; and dynamic comparison systems relate states through source correspondence, synchronized control structure, inferred variable mappings, or domain-specific model layers [Gulwani et al., 2018, Pham et al., 2019, Suzuki et al., 2017]. These constraints are useful in their target settings but become restrictive when language, framework, decomposition, or control flow differs: source locations and names may not correspond, observed values require explicit shape and tolerance policies, and loops or branches can change the event sequence. Moreover, one discrepant assignment can contaminate a long trace suffix, while merely locating an unequal value does not establish whether replacing it would restore downstream agreement. We present NumericalDiff, an execution-based differencing approach for numerical functions. We focus on pairs that serve the same purpose and have similar computational logic; more distant algorithms are used only to characterize boundary behavior. For each input, NumericalDiff instruments dynamically executed assignments, normalizes observed scalars, arrays, and framework tensors, and computes a tolerant longest-common-subsequence alignment without relying on variable or location identity. To separate an originating difference from propagation, it exhaustively evaluates 2

NumericalDiff

A P REPRINT

Shared source code

float32 execution

float64 execution

def newton(x): z = np.array(x, copy=True) for _ in range(60): f = z*z*z - 2*z + 2 df = 3*z*z - 2 z = z - f / df return z

x0 = -0.811595 -> 128.199707 ... -> 1 -> 0 -> 1 cycling

x0 = -0.811595 -> 128.199149 ... -> -1.769292 converged

(a) The same Newton iteration code, cycling in float32 but converging in float64. PyTorch

Fortran

def self_attention(x): tokens = x.reshape(2, 2) scores = tokens @ tokens.T scaled = scores / (2.0 ** 0.5) shifted = scaled - torch.amax( scaled, dim=1, keepdim=True) weights = torch.exp(shifted) probabilities = weights / torch.sum( weights, dim=1, keepdim=True) return probabilities @ tokens

function self_attention(x) result(attended) real(8), intent(in) :: x(4) real(8) :: attended(2,2), tokens(2,2), & scores(2,2), scaled(2,2), shifted(2,2), & weights(2,2), probabilities(2,2) tokens = transpose(reshape(x, [2,2])) scores = matmul(tokens, transpose(tokens)) scaled = scores / sqrt(2.0d0) shifted = scaled - spread(maxval( & scaled, dim=2), dim=2, ncopies=2) weights = exp(shifted) probabilities = weights / spread(sum( & weights, dim=2), dim=2, ncopies=2) attended = matmul(probabilities, tokens) end function self_attention

(b) The same self-attention algorithm, implemented in PyTorch and Fortran. Iterative summation

Gauss’s formula

def triangular_sum(n): total = 0 for i in range(1, n + 1): total += i return total

def triangular_sum(n): return n * (n + 1) // 2

(c) The same triangular sum task, implemented by iteration and Gauss’s closed-form formula.

Figure 2: Motivating examples.

the original executions and every single assignment-value intervention in both directions: a value from one execution is injected into the other, the recipient is reexecuted, and the new trace is realigned. NumericalDiff selects the candidate with the most numerically matching events, breaking ties by fewer unmatched events. When later values reconverge, the diff presents the intervened assignment as the localized difference and the suffix as conditional agreement. Reexecution is essential because an injected value can alter branches, loop trip counts, and subsequent events. This bounded objective tests whether one replacement explains a propagated trace difference; it does not establish program equivalence. The implementation supports Python numerical frameworks, C, C++, Fortran, and MATLAB-compatible programs executed by GNU Octave. Table 1 compares NumericalDiff with the four closest execution-based approaches. Execution-based differencing compares behavior observed in concrete runs. Runtime intervention replaces a value with one from the other execution and observes the resulting continuation. Mapping-free event alignment relates events through their observed numerical values without requiring predefined correspondence between locations, variables, or computational stages. Collins et al. intervene at corresponding instrumented locations; Clara infers a consistent variable mapping under compatible control structure; CRADLE relies on a shared model-layer graph; and TraceDiff compares update histories at corresponding relative positions. NumericalDiff instead combines mapping-free event alignment with intervention-conditioned trace differencing. We summarize the contributions of this paper as follows: • We formulate numerical program comparison as execution-aware trace alignment with tolerant value agreement and value intervention, and define an exhaustive candidate-selection objective that separates a local difference from its propagated effects. • We develop NumericalDiff, including a tolerant alignment and a single-intervention method that reexecutes programs to account for changed values and control flow. 3

NumericalDiff

A P REPRINT

Table 1: Comparison between the proposed method and closely related approaches. Mapping-free Execution-based Runtime differencing intervention event alignment Collins et al. [Collins et al., 2013] Clara [Gulwani et al., 2018] CRADLE [Pham et al., 2019] TraceDiff [Suzuki et al., 2017]

✓ ✓ ✓ ✓

✓ × × ×

× × × ×

NumericalDiff (ours)

✓

✓

✓

• We implement a common observation and intervention protocol across Python, C, C++, Fortran, and MATLABcompatible code. • We build a benchmark spanning execution-stack, program-representation, and program-logic variation across AI, scientific computing, and general numerical programs.

2

Methodology

2.1

Problem Formulation

Scope and numerical observation points. Let fL , fR : X → Y be two numerical functions intended to perform the same task through similar computational logic and a common logical input/output interface. Representation adapters, when needed, translate this interface to each implementation. For a supplied input x ∈ X , we assume that each function executes deterministically under its configured environment. We observe numerical values at supported assignment statements and at the terminal return. Each dynamic execution of an assignment constitutes a separate event; successive iterations of the same statement produce distinct events. An event ei = (ℓi , ki , vi ) records its source location, kind ki ∈ {assign, return}, and a snapshot of its numerical value. The value may be a scalar, an array, or an array-compatible framework tensor. An unmodified run produces Tf (x) = ⟨e1 , . . . , en ⟩.

(1)

A terminal return is observed as an event but is not an intervention site. Locations and variable names support reporting but do not constrain cross-function event correspondence. Numerical agreement and alignment. Let Cθ (u, v) ∈ {0, 1} denote numerical agreement under a fixed comparison policy θ, which specifies representation normalization, exact-shape compatibility, numerical tolerances, and handling of non-finite values. The current implementation does not match NaNs; the benchmark further requires finite outputs and therefore does not exercise non-finite behavior. We write u ≈θ v when Cθ (u, v) = 1. Two events may match only when they have the same kind and their values agree. An alignment a consumes both traces in execution order and consists of two-sided matches and one-sided gaps. Let M (a) be its number of matches and U (a) its number of gaps. This observational agreement need not be transitive and does not establish semantic equivalence. Among the alignments for a fixed trace pair, tolerant LCS maximizes M (a). Single-intervention candidate executions. An intervention replaces the value produced at one assignment event with a value observed at an assignment event in the other execution. The recipient then continues from the injected value and is reexecuted from the original input, because the replacement may change later values, branches, loop trip counts, and trace length. Arbitrary constants and return-value interventions are excluded. If an intervention were allowed directly at the output boundary, copying the other execution’s return value would trivially force equal outputs without explaining any preceding difference. Output agreement alone is therefore not a useful objective, and return events are observation-only. We permit at most one assignment intervention because the present problem asks whether one differing intermediate state can explain a propagated trace region; this bound also makes exhaustive search possible. Multiple independent origins are outside the present formulation. Let AL and AR be the assignment-event indexes in the original traces TL = TfL (x) and TR = TfR (x). For p ∈ AL and q ∈ AR , let TRp→q denote the right trace produced by replacing the value at right event q with the original value at left event p; define TLq→p symmetrically. Only successfully completed, representable interventions are admissible. The candidate set is Cx = {(TL , TR )} ∪ {(TL , TRp→q ) | p ∈ AL , q ∈ AR } ∪ {(TLq→p , TR ) | p ∈ AL , q ∈ AR }. 4

(2)

NumericalDiff

A P REPRINT

For each c ∈ Cx , let ac be a tolerant-LCS alignment and define
score(ac ) = M (ac ), −U (ac ) .

(3)

c∗x ∈ arg max score(ac ),

(4)

The optimization objective is c∈Cx

where scores are compared in lexicographic order. The objective therefore first maximizes the number of tolerant matches and, among candidates with the same number of matches, minimizes the number of unmatched events. It is defined independently for each supplied input over all zero- or one-intervention candidate executions. Agreement obtained after an intervention is conditional on that intervention, and the objective does not establish equivalence on unobserved inputs. 2.2

Algorithm Design

NumericalDiff compares the two functions independently for each supplied input. It first executes both functions without intervention and obtains the event traces TL and TR defined above. Trace construction is execution driven: only assignments that actually execute contribute assignment events, and all events retain their runtime order. The algorithm does not initially pair variables, source locations, or syntactic statements across the two functions; correspondence is determined from event kind, numerical agreement, and order. Tolerant trace alignment. As a no-intervention baseline, we compute a longest common subsequence over the two event sequences, replacing exact token equality with numerical agreement. Let L(i, j) be the maximum number of matches between prefixes of lengths i and j, with L(0, j) = L(i, 0) = 0. The recurrence is  R L(i − 1, j − 1) + 1, eL i and ej agree under Cθ , L(i, j) = (5) max{L(i − 1, j), L(i, j − 1)}, otherwise. Backtracking yields an ordered sequence of matches and one-sided gaps. A left gap and a right gap are reported as unmatched events; they are not assumed to be a semantic substitution merely because they are adjacent in the rendered diff. When multiple LCS alignments have the same length, the implementation uses a fixed backtracking preference, making the selected alignment deterministic for a fixed pair of traces. This baseline can identify natural reconvergence: a divergent block may be followed by matching events without forcing the entire remaining suffix to be reported as different. Single-intervention search. The intervention method exhaustively searches the bounded candidate space. Let AL and AR be the dynamic indexes of assignment events in the two traces; return observations are excluded because they do not affect subsequent execution. For every pair (p, q) ∈ AL × AR , the algorithm evaluates both directions. In the L → R direction it reexecutes the right function, replaces the value assigned at event q with the original value observed at left event p, and aligns the resulting right trace against the unchanged left trace. The R → L direction is symmetric. The instrumentation records the recipient’s originally computed value before performing the replacement, while all later computation observes the replacement. Consequently, the new alignment describes both the local discrepancy and the continuation induced by the intervention. Candidates are ranked by Equation (3). The baseline alignment is also a candidate. A candidate replaces the current best result only when its score is strictly larger, so ties retain the first candidate in the fixed enumeration order. The output records the selected direction, recipient event, donor event, and resulting alignment. This search tests 2|AL ||AR | intervened executions in addition to the original runs and exactly optimizes Equation (4). It does not compose several interventions or optimize over that larger search space. Loops. A loop is handled through its dynamic event occurrences rather than by a special static loop-matching rule. If one assignment in a loop body executes k times, it contributes k distinct events with different dynamic indexes, even though all carry the same source line. LCS may therefore match individual iterations, leave an extra iteration as a gap, or realign a common suffix after different intermediate iterations. The intervention search can target one specific occurrence—for example, only the second execution of an assignment— without modifying the other iterations of the same source statement. The recipient function is then rerun from its original input, so the replacement can affect later loop-carried state and, when the loop bound or exit behavior depends on that state, can also change the number of subsequent events. NumericalDiff does not require the two loops to have equal trip counts or identical loop variables. Loop-control bindings and predicates are not themselves observation events unless their values are assigned at an otherwise supported observation site. 5

NumericalDiff

A P REPRINT

Branches. For a conditional, the trace contains assignments from the path that was actually taken; untaken branches contribute no placeholder events. If the two functions select different paths, their path-specific assignments appear as unmatched regions. Later events can nevertheless match when the paths naturally reconverge to numerically agreeing states. An intervention is not restricted to preserving the original path: replacing an earlier assignment may change a subsequent predicate, reach assignments absent from the original trace, or skip assignments that were originally executed. The algorithm therefore reruns the recipient and aligns the newly produced trace rather than patching a previously recorded sequence. Branch predicates are not compared as separate events unless explicitly materialized by an observed assignment; their effect is represented by the resulting event sequence and values. Result interpretation. NumericalDiff reports agreement between the original return values separately from agreement in the selected trace alignment. Any unmatched dynamic event makes the selected trace alignment non-equivalent, but it does not by itself imply different return values. Thus an extra numerically neutral loop iteration or a branch that diverges and reconverges can produce equal outputs while retaining a localized trace difference. Conversely, matches obtained after an intervention are reported together with that intervention and should be interpreted as conditional agreement, not as agreement of the original executions. Complexity. Let n and m be the original trace lengths, a = |AL | and b = |AR | the numbers of assignment events, and R an upper bound on one instrumented execution. The baseline tolerant LCS takes O(nm) time and O(nm) space. The exhaustive search evaluates 2ab interventions. If every resulting trace
has length at most N , each candidate requires one reexecution and an O(N 2 ) LCS, giving O (ab + 1)R + abN 2 + nm time, including the original executions. Since candidates are processed sequentially, the peak auxiliary space is O(N 2 ) in addition to the traces and backend state. When a, b, n, m = O(N ), the comparison component is O(N 4 ) in the worst case, plus O(N 2 R) reexecution cost.

3

Implementation

Before diving into the implementation details, we disclose that LLM-based coding agents, including Codex and Cursor, have assisted in implementing NumericalDiff, including language backends, comparison service, VS Code extension, and in designing the benchmark cases. All LLM-generated content was reviewed and/or tested by the authors. 3.1

Instrumentation Architecture

NumericalDiff implements source-level dynamic instrumentation for Python, C, C++, Fortran, and MATLABcompatible .m files executed by GNU Octave. The common entry point dispatches by file extension. Python is transformed and executed in process; the other languages use a prepared external runner that parses the source, generates an instrumented program and runtime support, and invokes the corresponding compiler or interpreter. No external program is translated into Python, and an unavailable tool or unsupported construct is reported as an error rather than replaced by a fallback implementation. Preprocessing is performed independently on the two sides: it identifies observation sites inside the selected function but neither infers a trace from source order nor constructs a cross-program mapping. Source locations are retained for the diff view, while correspondence is established later from the values and dynamic order of executed events. Across languages, each supported assignment follows the same conceptual transformation: its right-hand side is evaluated once, the result and dynamic event identity are recorded, and an optional donor value is then assigned before execution continues. Later statements therefore observe the intervened state while the report retains the original value at the edited site. The event index denotes a dynamic occurrence rather than a source line; loop iterations are therefore distinct, untaken branches produce no events, and an intervention may change the subsequent control flow and trace length. Variable names support source rewriting but are not alignment keys. 3.2

Language-Specific Preprocessing

Table 2 summarizes the implemented front ends. Each parser is used to locate syntactic write sites and source ranges; the runtime value check remains authoritative because the tool does not attempt whole-program numerical type inference. For Python, an ast.NodeTransformer rewrites the selected function and preserves line numbers. Chained assignments produce one event per supported direct-name target after the original statement. The transformed tree is compiled in a fresh module namespace. Values are copied through NumPy; PyTorch tensors are detached and moved to CPU first, while other array-compatible objects, including JAX arrays, use NumPy conversion. Where supported, the intervention adapter preserves PyTorch tensors, NumPy array dtypes, and scalar types; JAX observations are supported through NumPy-compatible values. 6

NumericalDiff

A P REPRINT

Table 2: Implemented source instrumentation backends. Source Parser / executor Python C/C++ Fortran .m

ast / CPython Clang JSON AST / Clang fparser2 / GFortran tree-sitter / GNU Octave

For C and C++, Clang’s JSON AST identifies one unambiguous top-level function and byte ranges for source rewriting. The current interfaces are either double f(double) or a flat one-dimensional array ABI with input, output, and length arguments. The backend instruments initialized scalar declarations and standalone =, +=, -=, *=, and /= statements. Array-element writes are observed as separate scalar events on each dynamic execution. For canonical staged kernels, initialized declarations of the bundled C nd_array value type and concrete Eigen dynamic arrays, vectors, and matrices are instead observed and intervened on as whole values. This is type-directed instrumentation, not loop recognition or event aggregation. The rewritten source is combined with a small C runtime and driver and compiled by Clang or Clang++ using -O0 and debug information. For Fortran, fparser2 locates a single standalone function with one input and an explicit real(8) or double precision result. Instrumented whole-variable assignments call a generated generic nd_point procedure supporting ranks zero through two. A generated main program loads the request and calls the function; GFortran compiles it with -O0, debug information, and runtime checks. For .m files, tree-sitter locates one function with one named input and output and inserts calls after whole-variable assignments. The transformed source and helper functions are executed by GNU Octave. Value-bearing returns are terminal observations rather than intervention sites. Python and scalar C/C++ evaluate the return expression once and record it directly. Fortran and Octave record the result variable at explicit return or normal function completion. The C/C++ array ABI records the output buffer at fallthrough. Every successful external trace must end in such a return event. 3.3

Cross-Process Trace Protocol

External backends exchange data with the Python controller through a small text protocol. A request records the input rank, logical shape, values, and a set of interventions represented by dynamic event index and donor value. Inputs and donors are restricted to nonempty real scalar, vector, or matrix values. Values are written with 17 significant decimal digits in logical Fortran order; this makes transfer independent of the in-memory layout of the originating NumPy array. Each runtime emits one record containing source line, event kind, dtype, rank, shape, and flattened values. The controller validates the record, reconstructs its NumPy value in Fortran order, and normalizes all languages to a common TraceStep(line, value, kind) representation containing a copied value. Thus alignment and intervention search consume one trace interface regardless of the source language. Shape and dtype remain explicit, and numerical equality is evaluated later under the request’s absolute and relative tolerances. The runtime and reader enforce a limit of 10,000 events and reject malformed traces, missing terminal returns, and interventions that did not reach an executed assignment. 3.4

Reexecution and Backend Lifecycle

An external runner is prepared once per resolved source path and function name within a comparison session. Generated source trees and executables live in a session-scoped temporary directory. Compiled C, C++, and Fortran artifacts and transformed Octave files are reused across the original executions and all intervention candidates, while each rerun receives fresh request and trace files and starts a fresh external process. The temporary directory is removed when the comparison ends. Compilation and external execution have 60-second and 20-second timeouts, respectively. The same lifecycle supports both tolerant-LCS and brute-force intervention alignment. The latter may repeatedly rerun either side with a donor value from an assignment on the opposite trace. Python replacements preserve the live container and dtype where supported; external runtimes require compatible rank and shape. Failures of individual speculative interventions are discarded as invalid candidates, whereas missing tools, preprocessing failures, and failed original executions are surfaced to the caller. Subprocesses and timeouts provide lifecycle control, not a security sandbox; compared programs must be trusted and may still perform external side effects. 7

NumericalDiff

3.5

A P REPRINT

Current Coverage Limits

The implementation observes selected source-level assignments, not every memory write or mutation. It does not recursively instrument helper functions, recover writes hidden inside library calls, update all aliases after a replacement, or support concurrent execution. Python excludes unpacking, attribute and subscript targets, assignment expressions, implicit bindings such as loop targets, and bare returns. C/C++ excludes nested writes, macro-expanded write sites, pointer targets, overloaded operators, and general C++ objects. Fortran excludes modules, subroutines, pointers, derived types, indexed targets, declaration initializers, and several compound statement forms. Octave excludes indexed or multiple-output assignments, nested functions, and global or persistent state. These restrictions define the prototype’s executable subset; they do not imply coverage of arbitrary programs, optimized-binary behavior, or all side effects in the supported languages. 3.6

VS Code Extension Front End

We expose NumericalDiff through a Visual Studio Code extension. The command Numerical Diff: Compare Functions in Files selects two visible editors in left-to-right order, or exactly two open text-file tabs when that condition does not hold. It saves modified files, prompts for the function and, when needed, input shape, and reads other comparison options from extension settings. The extension invokes the same Python comparison service as other clients via a JSON exchange over standard input and output. Its Webview presents complete function bodies side by side, places aligned events on shared rows, and marks unmatched lines with the editor’s removed and inserted colors. Controls switch samples, summarize and navigate differences, and report the selected algorithm. Intervention-based results identify donor and recipient lines and distinguish conditional matches from original agreement.

4

Benchmark Construction

We instantiate the three variation layers of Figure 2 as 36 controlled cases from machine learning, scientific computing, and general numerical programs. Cases are labeled by the primary factor we change: execution stack, program representation, or program logic. A language or framework migration can also change compilers, runtimes, and libraries, so the label is not a claim that other layers are untouched. Inputs, model parameters, random seeds, and external state are fixed; each case has three persisted inputs, an explicit numerical policy, and an independent output reference. The suite yields 675 implementation-pair/input instances: 36 execution-stack, 588 program-representation (441 canonical comparisons and 147 cyclic alternative probes), and 51 program-logic. The cyclic probes compare alternatives across ecosystems; mixed-factor edges also introduce a controlled program-logic change rather than isolate representation variation. The three categories answer complementary questions. Execution-stack cases ask whether a controlled environmental perturbation is visible in an otherwise unchanged program. Program-representation cases ask whether the front ends preserve a common stage-level trace across languages and frameworks. Program-logic cases compare zero-intervention alignment with bounded intervention search; exhaustive enumeration makes the selected score optimal for Equation (4), but not a globally minimum edit script, a unique causal explanation, or a proof of program equivalence. Across all three categories we report matching coverage 2M/(2M + U ) in [0, 1] to aggregate across trace lengths; it does not replace the lexicographic objective. 4.1

Execution-Stack Variation

This category holds the abstract computation and program representation fixed and varies only how that same program is executed. Its potential sources are hardware; numerical formats and arithmetic modes; operator or numerical libraries; collective communication libraries; compilers and compilation configurations; interpreters and their configurations; device drivers; and runtimes and execution configurations. A library change belongs here when it occurs beneath an unchanged program interface; rewriting the program against a different framework belongs to program-representation variation instead. The current release instantiates four numerical-format cases and eight controlled operator-library fault models. Numerical-format cases use one NumPy source file and execute each persisted input in both float32 and float64. They cover rounding in stable Softmax, loss of small updates during iterative accumulation, a precision-dependent Newton orbit, and cancellation in a second-moment variance formula. In particular, the accumulation stagnates in float32 but reaches its mathematical target in float64; the Newton iteration enters a period-two orbit in float32 while converging to the real root in float64. 8

NumericalDiff

A P REPRINT

Table 3: Execution-stack variation cases. Each case executes one unchanged source file under two declared execution profiles. Task Domain Numerical behavior exercised Stable softmax Sigmoid operator fault Matrix operator fault Maximum operator fault Iterative accumulation Newton iteration Square-root fault Sine operator fault Division operator fault Variance Reduction fault Mean operator fault

AI AI AI AI Scientific Scientific Scientific Scientific Scientific General General General

float32 versus float64 normalization exp result multiplied by 1.01 matmul result multiplied by 1.005 max result shifted by 0.01 float32 stagnation versus float64 convergence float32 period-two orbit versus float64 convergence sqrt result multiplied by 1.002 sin result multiplied by 1.003 divide result multiplied by 1.004 float32 versus float64 cancellation in E[x2 ] − E[x]2 sum result multiplied by 1.01 mean result shifted by 0.02

The operator cases likewise use one unchanged NumPy source and float64 inputs on both sides, but one execution profile deterministically biases the result of exp, matmul, max, sqrt, sin, divide, sum, or mean. These profiles model the observable numerical consequence of a fault below an unchanged program interface, independent of whether its physical origin is hardware, a numerical library, a compiler, a driver, or a runtime. They are explicit harness-injected faults, not claims about real NumPy defects or empirical tests of those physical components. Each operator case has one controlled changed execution component, whereas a precision change can perturb many operations and need not have a unique origin. Both remain valid matching benchmarks: the former provides a localized perturbation, while the latter measures how much agreement can be recovered from distributed numerical drift. Experimental protocol. Each case contains one unchanged kernel.py and three persisted inputs, giving 36 execution-stack pair/input instances. For a numerical-format case, the two profiles differ only in array dtype. Stable Softmax and variance use shape-exact comparison with rtol = 10−10 and atol = 10−12 ; Newton iteration uses 10−12 for both; and iterative accumulation requires exact equality. The accumulation and Newton cases declare unequal outputs, whereas Softmax and variance measure whether a given input exposes a difference rather than requiring one. For an operator case, both profiles use float64; the faulty profile intercepts exactly one named NumPy operator and applies the deterministic bias shown in Table 3. All operator cases use shape-exact comparison with rtol = 10−10 and atol = 10−12 . Seven declare unequal outputs. The maximum-reduction case instead declares equal outputs: adding a constant to the maximum shifts all Softmax logits equally and is removed by normalization, while the intermediate execution remains different. This case deliberately tests a difference that final-output comparison cannot observe. We run zero-intervention tolerant-LCS on every instance. For workload validation, the harness checks the unmodified profile against an independent reference and checks the declared relation between the two outputs. For the differencing result, it records matched event pairs M , unmatched events U , full-trace equivalence, and coverage 2M/(2M + U ), then independently replays every reported match against the full recorded values. This experiment asks whether a controlled execution-stack change is observable and how much of the original traces remains aligned; it does not evaluate intervention recovery, which is isolated in the program-logic experiment. 4.2

Program-Representation Variation

This category holds the abstract computation fixed while changing its program representation. Its primary sources are programming languages and numerical or machine-learning frameworks. The implementations must preserve the mathematical definition, algorithmic stages, and intended data dependencies; otherwise the case belongs to programlogic variation. Each current task is implemented in NumPy, PyTorch, JAX, C, C++, Fortran, and Octave, all using CPU execution and nominal double precision. For each task, a staged specification defines the numerical value expected after every algorithmic stage, not only the final result. The seven canonical implementations realize this specification with one whole-value assignment per stage. Comparing all 21 unordered canonical pairs on three persisted inputs yields 63 pure program-representation instances per task and 441 across the seven tasks. Each task additionally provides one alternative implementation in every ecosystem. These alternatives intentionally need not preserve the canonical event sequence. A semantics-preserving alternative computes the same output through a different expression, control structure, or assignment granularity; a faulty alternative also introduces a controlled 9

NumericalDiff

A P REPRINT

Table 4: Program-representation variation cases. Every case has canonical and alternative implementations in NumPy, PyTorch, JAX, C, C++, Fortran, and Octave. Task Domain Fixed computation Sigmoid Squared-error loss Self-attention Polynomial Euler integration Compound interest Temperature conversion

AI AI AI Scientific Scientific General General

Element-wise 1/(1 + exp(−x)) Element-wise 0.5(x − 0.25)2 Two-token scaled dot-product attention with Q = K = V Horner evaluation of a fixed cubic polynomial Two explicit steps for y ′ = −0.5y Three compounding steps at a fixed one-percent rate Fahrenheit-to-Celsius affine conversion

program-logic change. Each task currently contains four semantics-preserving and three faulty alternatives. To retain cross-representation comparisons without constructing an uninformative alternative Cartesian product, we connect the alternatives in the fixed cycle NumPy–PyTorch–JAX–C–C++–Fortran–Octave–NumPy. Every edge therefore crosses a language or framework boundary, and every alternative selects one successor in another ecosystem. No alternative is compared with a canonical implementation in this part of the experiment. The placement of the current labels around the cycle gives each task one edge between two semantics-preserving alternatives and six edges between a semantics-preserving and a faulty alternative; there is no faulty–faulty edge. Across seven tasks, this produces seven equivalent–equivalent and 42 equivalent–faulty program pairs. Running every pair on the three persisted inputs yields 21 and 126 pair/input instances, respectively, or 147 alternative instances in total. Together with the 441 canonical instances, the category contains 588 instances. Faulty alternatives are identified by their output classes, and edges containing one are treated as mixed-factor probes rather than evidence isolating representation variation. The cases preserve common stages rather than merely sharing a task name. For example, every canonical selfattention implementation explicitly computes the score matrix, scaling, row-wise maximum subtraction, exponentiation, normalization, and weighted aggregation. NumPy, PyTorch, JAX, Fortran, and Octave use their native array values; C uses the bundled nd_array value container; and C++ uses Eigen. The latter two choices make C/C++ stages observable and replaceable as complete values, yielding the same ordered event granularity as the other five ecosystems. This is a benchmark representation contract rather than an analysis heuristic: the backend does not infer a matrix or vector operation from a scalar loop. Ordinary C/C++ element writes therefore remain separate dynamic events, and loop-based alternatives are explicit granularity challenges rather than members of the canonical perfect-alignment oracle. The self-attention alternatives also exercise cross-language differences in expression granularity: the Fortran variant materializes the broadcast row maximum as a separate assignment, whereas other variants may fuse that value into the normalization expression. Changing a representation can also change compilers, math libraries, and runtime kernels. The canonical comparisons consequently evaluate an end-to-end migration under controlled device and precision settings, not the isolated causal effect of syntax. Their oracle is tolerant stage-by-stage agreement with no unmatched events; bitwise identity is recorded separately and is not required across ecosystems. 4.3

Program-Logic Variation

This category changes the abstract computation while holding the execution stack fixed—currently NumPy on CPU with float64 inputs—and applies each case’s declared numerical policy consistently to both sides. Its sources are algorithmic choices and semantically relevant code changes: mathematical reformulations, changed constants or operators, altered data or control flow, inserted or removed events, and different execution orders. A source rewrite that preserves these properties belongs to program-representation variation instead. This category is designed around trace behavior rather than application names. It contains examples in which a local discrepancy propagates, naturally reconverges, is compensated by a later change, changes the executed branch or loop, or alters event order without altering data dependence. The 17 cases contribute 51 pair/input instances. Table 5 summarizes the controlled program differences. Equal output does not imply equal traces: the branchreconvergence case follows different source paths before returning to the same numerical state, and the operationreordering case has crossing event correspondences that a strictly sequential LCS cannot retain simultaneously. The loop-occurrence case records repeated executions of one source assignment and injects a fault only at the second dynamic occurrence. 10

NumericalDiff

Task

A P REPRINT

Table 5: Program-logic variation cases and their differences. Domain Program difference

Softmax AI Stable maximum-shifted versus direct exponentiation √ Attention scaling AI Divide scores by d versus by one ReLU recovery AI Different shifts collapse to the same zero region Sigmoid sign AI Exponentiate −x versus x Polynomial Scientific Horner evaluation versus expanded powers Euler sign Scientific Opposite sign in the first slope Stencil boundary Scientific Periodic versus zero-padded boundaries Threshold branch Scientific Different thresholds select different paths Loop occurrence Scientific Only the second execution of a repeated assignment is faulty Loop trip count Scientific One side executes an extra neutral iteration Variance General Centered two-pass versus second-moment formula Triangular sum General Iterative accumulation versus Gauss’s closed form Two rates General Two independent rate constants are changed Compensating edits General A later change cancels an earlier change Extra assignment General One side computes an unused intermediate Branch reconvergence General Different branches reconverge after canonicalization Operation reordering General Two independent assignments execute in opposite orders

Most cases closely match the intended setting of Equation (4): the functions perform the same task with similar logic, while a local change, branch, loop occurrence, or naturally recovering state produces a controlled trace difference. The remaining cases broaden the difficulty range. In particular, the two-rates case contains two independent origins, the triangular-sum and alternative-algorithm cases have more distant computational structures, and the operation-reordering case exposes the order constraint of sequential LCS. They still admit the same quantitative evaluation: the achieved score and residual unmatched events show how the current bounded search and sequential alignment behave when one intervention cannot recover all agreement. Experimental protocol. For every case, we compare its left and right NumPy functions on all three persisted inputs, giving 51 pair/input instances. We first run zero-intervention tolerant-LCS and then run the exhaustive singleintervention method with the same input and numerical policy. Sixteen cases use shape-exact comparison with rtol = 10−10 and atol = 10−12 ; the integer-valued triangular-sum case requires exact agreement. All required outputs must be finite. The intervention run includes the zero-intervention alignment and enumerates both replacement directions for every pair of dynamic assignment events; returns are observations but not intervention sites. Across the suite this gives 1,876 directional intervention candidates, ranging from 12 to 408 per instance. We validate the experiment at three levels. First, independent references and declared output relations check the original workloads. Second, the harness reexecutes the selected intervention, when any, and independently checks every reported match against full runtime values. It also checks annotated single- or multiple-intervention witnesses, natural-recovery suffixes, loop occurrences, and event-count or event-order properties where applicable. Third, exhaustive enumeration establishes optimality only for the bounded objective in Equation (4); it does not establish a unique semantic alignment or causal explanation. For each instance we record (M0 , U0 ) for tolerant-LCS and (M ∗ , U ∗ ) for the selected zero- or one-intervention candidate, together with matching coverage C = 2M/(2M + U ), whether an intervention is selected, and comparison time. Coverage values are macro-averaged over pair/input instances; event counts are summed. We organize the cases into five disjoint behavior profiles: single-change propagation, control-flow divergence, multiple origins, different computational processes, and natural recovery or structural variation. We repeat each complete method run five times in fresh case-worker processes. Comparison time includes original traced executions, candidate reexecutions, and alignment, but excludes process startup and independent workload validation. Discrete results must agree across repetitions; timing is reported as the median. Across all three categories, labels apply to the persisted inputs rather than all possible inputs. The suite is a controlled microbenchmark, not yet a collection of independent real-world bugs. It currently excludes broad input distributions, nondeterminism, non-finite values, within-execution mixed precision, and large applications. Outside the staged canonical profile, it also lacks automatic mappings between different event granularities and independently adjudicated semantic correspondences for every case. These limitations are important when interpreting the later results: the benchmark can establish workload reproducibility, replay soundness, bounded-objective scores, matching improvement 11

NumericalDiff

A P REPRINT

Table 6: Execution-stack results under zero-intervention tolerant-LCS. Coverage is the macro-average of 2M/(2M + U ) over pair/input instances. Variation profile N Output equal Full trace Matched M Unmatched U Coverage Numerical format Operator injection

12 24

2 3

2 0

36 33

1,146 168

0.2550 0.2792

Overall

36

5

2

69

1,314

0.2711

over a baseline, and behavior across controlled difference patterns, but a high matching score does not by itself prove that every correspondence is the uniquely intended one or establish external validity for arbitrary numerical software. Algorithm comparisons must additionally use one method and policy uniformly; the per-group defaults used for fast harness validation are not a fair cross-method evaluation.

5

Evaluation and Analysis

5.1

Execution-Stack Variation

All 12 cases satisfy their workload contracts on all three inputs, and all 36 zero-intervention alignments pass independent replay validation. Thus every match counted below is numerically valid under the case’s shape and tolerance policy. Table 6 separates the two controlled variation families. The eight operator models all have an observable dynamic effect: every one of their 24 instances has a non-equivalent trace. The exp, matmul, sqrt, sin, divide, sum, and mean injections also make all 21 corresponding outputs unequal. In contrast, all three max instances retain tolerance-equal outputs but have different maximum, shifted-logit, exponential, and reduction events. Their final values are not bitwise identical. This is the intended process-different/result-equal control and shows concretely why output-only comparison misses execution-stack differences. Numerical-format behavior is input dependent rather than forced by the harness. Ten of its 12 instances have nonequivalent traces and unequal outputs. The remaining two are variance inputs whose float32 and float64 executions are bitwise identical at every observed event; they correctly remain fully matched. The four cases therefore include both exposed and benign format changes, while the stagnation and Newton cases exercise qualitatively different long-running behavior rather than merely a small last-bit perturbation. Overall, only 5 of 36 final-output pairs are equal, but just 2 have fully equivalent traces. The gap is supplied by the three maximum-reduction instances and demonstrates that output equality and trace equality answer different questions. The low macro-average coverage, 0.2711, is also expected for value-based zero-intervention alignment: once a changed value propagates, later assignments on the same source lines need not match. The especially long Newton traces contribute 1,052 unmatched events to the summed count, so U should not be read as 1,314 independent defects. These results validate that every injected operator fault takes effect and that the selected format cases span both localized numerical drift and divergent iterative dynamics; they are benchmark-characterization results, not evidence that a single intervention can repair every execution-stack difference. 5.2

Program-Representation Variation

We use this category to test whether the implemented language front ends can preserve a common numerical observation contract. We run the tolerant-LCS baseline with no intervention on all 441 canonical pair/input instances. A canonical instance is successful only if every assignment and return event is matched in order under the case’s shape-exact numerical policy, leaving no event unmatched. This is stronger than checking only the final output. We also replay every reported match against the recorded full values. Table 7 reports the results. All 441 canonical instances achieve complete trace equivalence: their 2,205 paired events are matched and no event is left unmatched, giving matching coverage 1.0. All final outputs are equal under the declared tolerance, while 425 of 441 are also bitwise identical. The remaining 16 arise from ordinary cross-ecosystem floating-point rounding; the largest observed absolute and relative output differences are 8.88 × 10−16 and 2.22 × 10−16 , respectively. Thus the tolerance admits only small numerical differences here and does not conceal a missing stage or a structural misalignment. Each of the seven tasks independently obtains 63 of 63 complete canonical alignments. The cyclic alternatives provide a separate cross-representation diagnostic; none of these comparisons includes a canonical implementation. The seven equivalent–equivalent program pairs, each executed on three inputs, produce 21 12

NumericalDiff

A P REPRINT

Table 7: Program-representation results under zero-intervention tolerant-LCS. Coverage is the macro-average of 2M/(2M + U ) over pair/input instances. Comparison profile Instances Output equal Full trace Coverage Canonical cross-representation Equivalent–equivalent cyclic edge Equivalent–faulty cyclic edge

441 21 126

441 21 0

441 12 0

1.0000 0.7429 0.0577

Table 8: Program-logic results. C0 and C ∗ are macro-average matching coverage before and after the bounded intervention search. Improved counts instances whose selected score is strictly greater than the zero-intervention score; Intervention counts instances for which that candidate uses a value replacement. Behavior profile N Output equal Improved Intervened C0 C∗ Single-change propagation Control-flow divergence Multiple origins Different processes Recovery/structural variation

12 6 6 12 15

0 3 0 12 15

12 3 6 3 0

12 3 6 3 0

0.1190 0.5429 0.0000 0.4539 0.7560

0.8458 0.6857 0.2500 0.5095 0.7560

Overall

51

30

24

24

0.4210

0.6513

pair/input instances. All 21 have bitwise-identical outputs; 12 also have fully equivalent traces, and their mean coverage is 0.7429. The 42 equivalent–faulty program pairs similarly produce 126 instances. All have the declared unequal output, none has a fully equivalent trace, and their mean coverage is 0.0577. Alignment replay succeeds for all 588 canonical and alternative instances, so every reported match is numerically valid under its policy. The first alternative group tests genuinely different but semantically equivalent implementations across languages or frameworks; the second intentionally combines representation and program-logic variation and is interpreted only as a diagnostic probe. In both counts, the factor of three is the number of persisted inputs per program pair, not three additional programs. The result establishes the intended support claim for this controlled suite: seven front ends can produce a common stage-level trace when the programs expose the same stages. It does not establish automatic semantic mapping for arbitrary programs with incompatible event granularities. In particular, a scalar C/C++ loop continues to produce one event per executed element assignment; no hidden aggregation was used to obtain the canonical result. 5.3

Program-Logic Variation

All 17 cases pass their independent workload checks in all five repetitions, and all 51 selected alignments pass replay validation. Of the 51 original pair/input instances, 30 have equal outputs and 21 have unequal outputs under the declared policy. These output counts characterize the workloads and are not used as the alignment objective. Table 8 shows that the search selects an intervention exactly for the 24 instances in which it strictly improves the lexicographic score; the other 27 retain the zero-intervention alignment. Across all instances, the number of matched event pairs increases from 102 to 162 and unmatched events decrease from 385 to 265, giving a macro-average coverage increase from 0.4210 to 0.6513. Three attention-scaling instances reach complete intervention-conditioned trace alignment; no original trace pair is completely aligned. The discrete scores and selected interventions are identical in all five repetitions. The behavior profiles clarify what the aggregate means. All 12 single-change-propagation instances improve, from coverage 0.1190 to 0.8458, because one replacement can restore a substantial downstream suffix. The loop-occurrence case confirms that the replacement can target only the second dynamic execution of one source line. In the control-flow profile, the three threshold-branch instances improve after reexecution follows the intervened state, whereas the three naturally reconvergent branch instances retain their baseline. The six multiple-origin instances improve only partially: one replacement cannot repair both controlled origins. Among the different-process cases, the stable-versus-naive softmax instances improve, while polynomial, variance, and triangular-sum implementations retain their baselines because their event structures do not admit a better single replacement. The recovery or structural profile likewise selects no intervention; tolerant-LCS already localizes the divergent region and retains naturally matching suffixes. The median sum of measured comparison times over the 51 instances is 0.0700 seconds for tolerant-LCS and 2.1173 seconds for exhaustive intervention, a 30.3× increase. The pooled per-instance medians are 1.21 and 9.04 milliseconds, 13

NumericalDiff

A P REPRINT

respectively. The difference between these ratios reflects the skew from trace length: the triangular-sum loop generates the largest candidate space and dominates aggregate intervention time while obtaining no score improvement. This result is consistent with the quartic worst-case comparison cost and motivates a future guided search. The experiment demonstrates exact optimization of the present bounded objective and measurable recovery of propagated agreement; it does not supply precision or recall against a unique human correspondence oracle.

6

Related Work

6.1

Semantic Differencing and Relational Verification

Program differencing ranges from edits over source structure to relations over program behavior. Structural techniques such as GumTree produce AST edit scripts [Falleri and Martinez, 2024], whereas semantic differencing summarizes changed dependencies or path behavior [Jackson and Ladd, 1994, Person et al., 2008]. Symbolic regression-verification systems encode two versions jointly and return equivalence results or distinguishing executions; abstraction, refinement, and partitioned classifications improve scalability and preserve partial results when a complete proof is unavailable [Lahiri et al., 2012, Badihi et al., 2020, Glock et al., 2024]. More generally, product programs combine two executions into one verification problem, making the selected product construction an explicit account of how the executions advance together [Barthe et al., 2011]. For numerical programs, relational abstract interpretation instead computes sound abstractions of differences between corresponding variables. DIZY and the analyses of Delmas and Miné develop this line for numerical programs and software patches [Partush and Yahav, 2013, Delmas and Miné, 2019a,b]. All relational approaches must decide how executions correspond. SCORE searches bounded interleavings and favors steps that reduce the abstract difference [Partush and Yahav, 2014]; alignment-completeness results formalize when such schedules suffice for a relational proof [Antonopoulos et al., 2023]. For dynamic analyses, execution-point identifiers distinguish repeated occurrences of a statement across executions [Sumner and Zhang, 2013b]. NumericalDiff addresses the same correspondence problem at a different level: it aligns concrete assignment events by their observed numerical values under a tolerance, without requiring a joint syntax, common variable names, or a predefined location map. 6.2

Comparative Execution Diagnosis and Intervention

Comparative debugging makes execution differences inspectable. Relative debugging, Scry, cozy, and TraceDiff relate divergent states or value histories to program context [Abramson et al., 2009, Burg et al., 2015, Helbling et al., 2025, Suzuki et al., 2017], while Dual Execution and LDX couple executions to handle environmental or control-flow divergence [Kim et al., 2015, Kwon et al., 2016]. Dependency analyses compare control flow, correct and failing implementations, or paired slices [Qi et al., 2009, Banerjee et al., 2010, Weeratunge et al., 2010]; InPreSS and CoReX further reduce propagated differences while retaining explanatory context [Badihi et al., 2023, Badihi and Rubin, 2026]. Intervention-based debugging modifies state and observes the outcome. State delta debugging, cause transitions, value replacement, and Comparative Causality progressively isolate relevant values and moments [Zeller, 2002, Cleve and Zeller, 2005, Jeffrey et al., 2008, 2009, Sumner and Zhang, 2013a]. For numerical drift, Collins et al. overwrite differing values at matched instrumentation points [Collins et al., 2013]; pLiner, Ciel, and multi-level FLiT instead vary precision or compilation regions to localize environment-induced variability [Guo et al., 2020, Miao et al., 2023, Bentley et al., 2019]. These approaches generally assume shared locations or the same source, whereas NumericalDiff searches value correspondences between independently implemented functions. Reference executions can also guide source repair: Clara and SARFGEN minimize repairs after dynamic or structural alignment [Gulwani et al., 2018, Wang et al., 2018], whereas AutoRNP derives numerical approximations from mathematical structure [Yi et al., 2019]. NumericalDiff repairs neither source nor a global edit sequence. It ranks zero- and single-assignment-intervention trace alignments, so restored agreement is input-specific, intervention-conditioned evidence rather than proof of a unique cause. 6.3

Numerical Debugging and Differential Testing

Numerical debuggers compare an execution with a high-precision or perturbed shadow. FPDebug and Herbgrind track error propagation, Verificarlo samples rounding behavior, and NUMFL ranks error-associated expressions [Benz et al., 2012, Sanchez-Stern et al., 2018, Denis et al., 2016, Bai et al., 2017]. They diagnose one implementation against a precision or statistical oracle; NumericalDiff instead compares peer implementations without designating either as ground truth. Differential testing compares libraries and environments. FPDiff tests synonymous numerical functions, FLiT and HeteroFuzz expose platform or compiler variability, and CRADLE, DeepREL, and Predoo target deep-learning backends, API relations, and operator precision [Vanover et al., 2020, Sawaya et al., 2017, Zhang et al., 2021a, Pham et al., 2019, Deng et al., 2022, Zhang et al., 2021b]. These systems find inconsistent executions but do not combine assignment-trace alignment with counterpart-value intervention. Input generation is complementary: shadow 14

NumericalDiff

A P REPRINT

symbolic execution and hybrid fuzzing distinguish program versions [Palikareva et al., 2016, Noller et al., 2020], while search-based testing, FPGen, and FPCC seek large floating-point errors [Zou et al., 2015, Guo and Rubio-González, 2020, Yi et al., 2024]. Because higher precision is not always a valid oracle [Wang et al., 2016], NumericalDiff makes inputs and tolerances explicit policies and leaves defect classification outside the alignment result.

7

Conclusion

This paper presented NumericalDiff, an execution-based approach for comparing same-purpose numerical functions across execution stacks, program representations, and program logic. It aligns assignment-level values using toleranceaware LCS and searches zero- and single-intervention executions for the alignment with the greatest numerical agreement. Across 36 controlled cases and 675 pair/input instances, NumericalDiff completely aligns canonical cross-representation programs, exposes operator faults hidden by equal final outputs, and recovers substantial downstream agreement after propagated program-logic differences. These results show that intermediate-value comparison complements source- and output-level differencing. Future work includes guided intervention search, input value design, and more efficient cross-representation alignment. Acknowledgments This work was supported by Zhongguancun Academy under Grant No. XTS0030.

References Jean-Rémy Falleri and Matias Martinez. Fine-grained, accurate and scalable source differencing. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE ’24. ACM, 2024. doi:10.1145/3597503.3639148. Daniel Jackson and David A. Ladd. Semantic Diff: A tool for summarizing the effects of modifications. In Proceedings of the International Conference on Software Maintenance, ICSM ’94, pages 243–252. IEEE Computer Society, 1994. doi:10.1109/ICSM.1994.336770. Suzette Person, Matthew B. Dwyer, Sebastian Elbaum, and Corina S. Păsăreanu. Differential symbolic execution. In Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of Software Engineering, SIGSOFT ’08/FSE-16, pages 226–237. ACM, 2008. doi:10.1145/1453101.1453131. William N. Sumner and Xiangyu Zhang. Comparative causality: Explaining the differences between executions. In 2013 35th International Conference on Software Engineering (ICSE), pages 272–281. IEEE, 2013a. doi:10.1109/ICSE.2013.6606573. Ryo Suzuki, Gustavo Soares, Andrew Head, Elena Glassman, Ruan Reis, Melina Mongiovi, Loris D’Antoni, and Björn Hartmann. TraceDiff: Debugging unexpected code behavior using trace divergences. In 2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), pages 107–115. IEEE, 2017. doi:10.1109/VLHCC.2017.8103457. Hung Viet Pham, Thibaud Lutellier, Weizhen Qi, and Lin Tan. CRADLE: Cross-backend validation to detect and localize bugs in deep learning libraries. In Proceedings of the 41st International Conference on Software Engineering, ICSE ’19, pages 1027–1038. IEEE Press, 2019. doi:10.1109/ICSE.2019.00107. Jackson Vanover, Xuan Deng, and Cindy Rubio-González. Discovering discrepancies in numerical libraries. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2020, pages 488–501. ACM, 2020. doi:10.1145/3395363.3397380. Qian Zhang, Jiyuan Wang, and Miryung Kim. HeteroFuzz: Fuzz testing to detect platform dependent divergence for heterogeneous applications. In Proceedings of the 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2021, pages 242–254. ACM, 2021a. doi:10.1145/3468264.3468610. David Abramson, Clement Chu, Donny Kurniawan, and Aaron Searle. Relative debugging in an integrated development environment. Software: Practice and Experience, 39(14):1157–1183, 2009. doi:10.1002/spe.932. John Collins, Brian Farrimond, David Flower, Mark Anderson, and David Gill. The removal of numerical drift from scientific models. International Journal of Software Engineering & Applications, 4(2):51–61, 2013. doi:10.5121/ijsea.2013.4204. Nimrod Partush and Eran Yahav. Abstract semantic differencing for numerical programs. In Static Analysis: 20th International Symposium, SAS 2013, volume 7935 of Lecture Notes in Computer Science, pages 238–258, Berlin, Heidelberg, 2013. Springer. doi:10.1007/978-3-642-38856-9_14. 15

NumericalDiff

A P REPRINT

David Delmas and Antoine Miné. Analysis of program differences with numerical abstract interpretation. In PERR 2019 – 3rd Workshop on Program Equivalence and Relational Reasoning, Prague, Czech Republic, April 2019a. URL https://hal.science/hal-02109517. Work-in-progress workshop paper. Geof Sawaya, Michael Bentley, Ian Briggs, Ganesh Gopalakrishnan, and Dong H. Ahn. FLiT: Cross-platform floatingpoint result-consistency tester and workload. In 2017 IEEE International Symposium on Workload Characterization, IISWC 2017, pages 229–238. IEEE, 2017. doi:10.1109/IISWC.2017.8167780. Alex Sanchez-Stern, Pavel Panchekha, Sorin Lerner, and Zachary Tatlock. Finding root causes of floating point error. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’18, pages 256–269. ACM, 2018. doi:10.1145/3192366.3192411. Hui Guo, Ignacio Laguna, and Cindy Rubio-González. pLiner: Isolating lines of floating-point code for compilerinduced variability. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–14. IEEE, 2020. doi:10.1109/SC41405.2020.00053. Sumit Gulwani, Ivan Radiček, and Florian Zuleger. Automated clustering and program repair for introductory programming assignments. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2018, pages 465–480. ACM, 2018. doi:10.1145/3192366.3192387. Shuvendu K. Lahiri, Chris Hawblitzel, Ming Kawaguchi, and Henrique Rebêlo. SYMDIFF: A language-agnostic semantic diff tool for imperative programs. In Computer Aided Verification: 24th International Conference, CAV 2012, volume 7358 of Lecture Notes in Computer Science, pages 712–717, Berlin, Heidelberg, 2012. Springer. doi:10.1007/978-3-642-31424-7_54. Sahar Badihi, Faridah Akinotcho, Yi Li, and Julia Rubin. ARDiff: Scaling program equivalence checking via iterative abstraction and refinement of common code. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2020, pages 13–24, New York, NY, USA, 2020. ACM. doi:10.1145/3368089.3409757. Johann Glock, Josef Pichler, and Martin Pinzger. PASDA: A partition-based semantic differencing approach with best effort classification of undecided cases. Journal of Systems and Software, 213:112037, 2024. doi:10.1016/j.jss.2024.112037. Gilles Barthe, Juan Manuel Crespo, and César Kunz. Relational verification using product programs. In FM 2011: Formal Methods, volume 6664 of Lecture Notes in Computer Science, pages 200–214. Springer, 2011. doi:10.1007/9783-642-21437-0_17. David Delmas and Antoine Miné. Analysis of software patches using numerical abstract interpretation. In Static Analysis: 26th International Symposium, SAS 2019, volume 11822 of Lecture Notes in Computer Science, pages 225–246, Cham, Switzerland, 2019b. Springer. doi:10.1007/978-3-030-32304-2_12. Nimrod Partush and Eran Yahav. Abstract semantic differencing via speculative correlation. In Proceedings of the 2014 ACM International Conference on Object Oriented Programming Systems Languages & Applications, OOPSLA ’14, pages 811–828, New York, NY, USA, 2014. ACM. doi:10.1145/2660193.2660245. Timos Antonopoulos, Eric Koskinen, Ton Chanh Le, Ramana Nagasamudram, David A. Naumann, and Minh Ngo. An algebra of alignment for relational verification. Proceedings of the ACM on Programming Languages, 7(POPL), 2023. doi:10.1145/3571213. William N. Sumner and Xiangyu Zhang. Identifying execution points for dynamic analyses. In 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 81–91. IEEE, 2013b. doi:10.1109/ASE.2013.6693069. Brian Burg, Amy J. Ko, and Michael D. Ernst. Explaining visual changes in web interfaces. In Proceedings of the 28th Annual ACM Symposium on User Interface Software and Technology, UIST ’15, pages 259–268. ACM, 2015. doi:10.1145/2807442.2807473. Caleb Helbling, Graham Leach-Krouse, Sam Lasser, and Greg Sullivan. cozy: Comparative symbolic execution for binary programs. In Proceedings of the 2025 Workshop on Binary Analysis Research, Reston, VA, USA, 2025. Internet Society. doi:10.14722/bar.2025.23004. Dohyeong Kim, Yonghwi Kwon, William N. Sumner, Xiangyu Zhang, and Dongyan Xu. Dual execution for on the fly fine grained execution comparison. In Proceedings of the Twentieth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS ’15, pages 325–338. ACM, 2015. doi:10.1145/2694344.2694394. Yonghwi Kwon, Dohyeong Kim, William N. Sumner, Kyungtae Kim, Brendan Saltaformaggio, Xiangyu Zhang, and Dongyan Xu. LDX: Causality inference by lightweight dual execution. In Proceedings of the Twenty-First 16

NumericalDiff

A P REPRINT

International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS ’16, pages 503–515. ACM, 2016. doi:10.1145/2872362.2872395. Dawei Qi, Abhik Roychoudhury, Zhenkai Liang, and Kapil Vaswani. DARWIN: An approach for debugging evolving programs. In Proceedings of the 7th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering, ESEC/FSE ’09, pages 33–42, New York, NY, USA, 2009. ACM. doi:10.1145/1595696.1595704. Ansuman Banerjee, Abhik Roychoudhury, Johannes A. Harlie, and Zhenkai Liang. Golden implementation driven software debugging. In Proceedings of the 18th ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE ’10, pages 177–186. ACM, 2010. doi:10.1145/1882291.1882319. Dasarath Weeratunge, Xiangyu Zhang, William N. Sumner, and Suresh Jagannathan. Analyzing concurrency bugs using dual slicing. In Proceedings of the 19th International Symposium on Software Testing and Analysis, ISSTA ’10, pages 253–264. ACM, 2010. doi:10.1145/1831708.1831740. Sahar Badihi, Khaled Ahmed, Yi Li, and Julia Rubin. Responsibility in context: On applicability of slicing in semantic regression analysis. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 563–575. IEEE, 2023. doi:10.1109/ICSE48619.2023.00057. Sahar Badihi and Julia Rubin. CoReX: Context-aware refinement-based slicing for debugging regression failures. In 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE). ACM, 2026. doi:10.1145/3744916.3764577. Andreas Zeller. Isolating cause-effect chains from computer programs. In Proceedings of the 10th ACM SIGSOFT Symposium on Foundations of Software Engineering, SIGSOFT ’02/FSE-10, pages 1–10. ACM, 2002. doi:10.1145/587051.587053. Holger Cleve and Andreas Zeller. Locating causes of program failures. In Proceedings of the 27th International Conference on Software Engineering, ICSE ’05, pages 342–351. ACM, 2005. doi:10.1145/1062455.1062522. Dennis Jeffrey, Neelam Gupta, and Rajiv Gupta. Fault localization using value replacement. In Proceedings of the 2008 International Symposium on Software Testing and Analysis, ISSTA ’08, pages 167–178, New York, NY, USA, 2008. ACM. doi:10.1145/1390630.1390652. Dennis Jeffrey, Neelam Gupta, and Rajiv Gupta. Effective and efficient localization of multiple faults using value replacement. In Proceedings of the 25th IEEE International Conference on Software Maintenance, ICSM 2009, pages 221–230, Piscataway, NJ, USA, 2009. IEEE. doi:10.1109/ICSM.2009.5306303. Dolores Miao, Ignacio Laguna, and Cindy Rubio-González. Expression isolation of compiler-induced numerical inconsistencies in heterogeneous code. In High Performance Computing, volume 13948 of Lecture Notes in Computer Science, pages 381–401. Springer, 2023. doi:10.1007/978-3-031-32041-5_20. Michael Bentley, Ian Briggs, Ganesh Gopalakrishnan, Dong H. Ahn, Ignacio Laguna, Gregory L. Lee, and Holger E. Jones. Multi-level analysis of compiler-induced variability and performance tradeoffs. In Proceedings of the 28th International Symposium on High-Performance Parallel and Distributed Computing, HPDC ’19, pages 61–72. ACM, 2019. doi:10.1145/3307681.3325960. Ke Wang, Rishabh Singh, and Zhendong Su. Search, align, and repair: Data-driven feedback generation for introductory programming exercises. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2018, pages 481–495. ACM, 2018. doi:10.1145/3192366.3192384. Xin Yi, Liqian Chen, Xiaoguang Mao, and Tao Ji. Efficient automated repair of high floating-point errors in numerical libraries. Proceedings of the ACM on Programming Languages, 3(POPL), 2019. doi:10.1145/3290369. Florian Benz, Andreas Hildebrandt, and Sebastian Hack. A dynamic program analysis to find floating-point accuracy problems. In Proceedings of the 33rd ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’12, pages 453–462. ACM, 2012. doi:10.1145/2254064.2254118. Christophe Denis, Pablo de Oliveira Castro, and Eric Petit. Verificarlo: Checking floating point accuracy through monte carlo arithmetic. In 2016 IEEE 23rd Symposium on Computer Arithmetic, ARITH 2016, pages 55–62. IEEE, 2016. doi:10.1109/ARITH.2016.31. Zhuofu Bai, Gang Shu, and Andy Podgurski. Causal inference based fault localization for numerical software with NUMFL. Software Testing, Verification and Reliability, 27:e1613, 2017. doi:10.1002/stvr.1613. Yinlin Deng, Chenyuan Yang, Anjiang Wei, and Lingming Zhang. Fuzzing deep-learning libraries via automated relational API inference. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE ’22, pages 44–56. ACM, 2022. doi:10.1145/3540250.3549085. 17

NumericalDiff

A P REPRINT

Xufan Zhang, Ning Sun, Chunrong Fang, Jiawei Liu, Jia Liu, Dong Chai, Jiang Wang, and Zhenyu Chen. Predoo: Precision testing of deep learning operators. In Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2021, pages 400–412. ACM, 2021b. doi:10.1145/3460319.3464843. Hristina Palikareva, Tomasz Kuchta, and Cristian Cadar. Shadow of a doubt: Testing for divergences between software versions. In Proceedings of the 38th International Conference on Software Engineering, ICSE ’16, pages 1181–1192, New York, NY, USA, 2016. ACM. doi:10.1145/2884781.2884845. Yannic Noller, Corina S. Păsăreanu, Marcel Böhme, Youcheng Sun, Hoang Lam Nguyen, and Lars Grunske. HyDiff: Hybrid differential software analysis. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, ICSE ’20, pages 1273–1285, New York, NY, USA, 2020. ACM. doi:10.1145/3377811.3380363. Daming Zou, Ran Wang, Yingfei Xiong, Lu Zhang, Zhendong Su, and Hong Mei. A genetic algorithm for detecting significant floating-point inaccuracies. In Proceedings of the 37th IEEE/ACM International Conference on Software Engineering, volume 1 of ICSE ’15, pages 529–539. IEEE Press, 2015. doi:10.1109/ICSE.2015.70. Hui Guo and Cindy Rubio-González. Efficient generation of error-inducing floating-point inputs via symbolic execution. In Proceedings of the 42nd International Conference on Software Engineering, ICSE 2020, pages 1261–1272. ACM, 2020. doi:10.1145/3377811.3380359. Xin Yi, Hengbiao Yu, Liqian Chen, Xiaoguang Mao, and Ji Wang. FPCC: Detecting floating-point errors via chain conditions. Proceedings of the ACM on Programming Languages, 8(OOPSLA2):1504–1531, 2024. doi:10.1145/3689764. Ran Wang, Daming Zou, Xinrui He, Yingfei Xiong, Lu Zhang, and Gang Huang. Detecting and fixing precision-specific operations for measuring floating-point errors. In Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2016, pages 619–630. ACM, 2016. doi:10.1145/2950290.2950355.

18

Related documents

• 45 CFR Part 1302 — Program Operations#510470
• Sesión N° 150 Consejo de Decanos, Subdivisión del Departamento de Computación#1347789
• Robust error-minimization in the genetic code across physicochemical metrics and variant codes: a graph-theoretic analysis in GF(2)6#234310
• Automated clustering workflow for molecular simulation data and ubiquitin simulation datset#376047
• A multi-language auto-differentiation module and its application to a parallel particle-in-cell code on distributed computers#1358146