arXiv CSopen access

Validating Memory-Optimal Transformer Kernels on Real Hardware: From Formal Derivation to Measured Performance Across Two HPC Clusters

Lenore M. Mullin et al. · arxiv_cs

arXiv CS · Papers · License: Open Access

cloud distributed-computing parallel-computing

distributed computing, parallel computing, cloud

arXiv:2609.33916v1 [cs.DC] 27 Sep 2026

Validating Memory-Optimal Transformer Kernels on Real Hardware: From Formal Derivation to Measured Performance Across Two HPC Clusters Lenore M. Mullin1 and Gaétan Hains2 1

Professor Emerita, College of Nanotechnology, Science, and Engineering, University at Albany (SUNY), Albany, NY, USA. [email protected] 2 LACL, Université Paris-Est Créteil, Créteil, France. [email protected] September 29, 2026

Abstract A companion series of papers formally derives memory-optimal cost functions for the components of a transformer block — attention’s forward pass, backward pass, fused forward+backward, inference-time decode, and the complete block — each verified to machine precision against PyTorch autograd, but never checked against real hardware. This paper reports that validation. Cost-function predictions are checked against measured performance on two HPC clusters (Purdue Anvil, NCSA Delta), across CPU and GPU, using the Mathematics of Arrays (MoA) formalism’s shape/index vocabulary (ρ, ψ, ι) both to derive each kernel’s machine-specific realization (ONF) from its hardware-independent specification (DNF) and to diagnose gaps when prediction and measurement diverge. Three results stand out. First, we identify and fix a real GPU performance regression: fusing forward and backward passes — proven to avoid materializing an O(n2 ) intermediate array — initially ran slower than the naive alternative on GPU, due to atomic-memory contention in the generated kernel. Profiling confirmed the mechanism to four decimal places (2.0000× more atomic instructions than a structurally related kernel), and a targeted restructuring of the ONF, guided directly by that diagnosis, reversed the result completely, yielding up to 2.5× real speedup. Second, we show the identical derivation produces markedly different real costs on different machine topologies — a 535× NUMA-locality penalty on one cluster’s architecture versus under 3× oversubscription cost on another’s — demonstrating that optimal deployment is a function of the target machine’s own array structure, not a fixed property of the algorithm. Third, we report a genuine, only partially resolved anomaly: identical denotational computations run faster in C than Fortran on CPU but faster in Fortran than C on GPU; we narrow this reversal to one dominant kernel and one stall mechanism without fully explaining its compiler-level cause. Together, this evidence supports treating hardware-specific optimization as a routine, targeted rewrite of a kernel’s machine-specific realization alone, with its underlying formal derivation remaining fixed, verified, and reusable across hardware generations — a candidate methodology for scaling AI systems onto evolving hardware without re-deriving correctness from scratch.

Introduction

This paper’s goal, stated plainly before any supporting detail: Papers I–IV [2, 3, 4, 5] formulate and formally derive a specific set of kernel designs for the components of a transformer block, using the Mathematics of Arrays (MoA) to go from a hardware-independent specification (a Denotational Normal Form, DNF) to a machine-specific realization (an Operational Normal Form, ONF) via a derivation step, γ, that treats the target machine’s own memory system — DRAM address space, cache lines, thread-to-data assignment — as an array in exactly the same shape/index vocabulary used to specify the computation itself. This paper runs those exact designs, as formulated in Papers I–IV, on real hardware, and checks whether the performance γ’s machineas-array treatment predicts matches what is actually measured. To our knowledge, MoA is the only complete, formal system of this kind: one where a machine’s memory behavior is not modeled heuristically or bounded empirically, but derived as an array in the same algebra as the computation, yielding a specific, checkable numeric prediction before any hardware is touched. This paper’s contribution is the real-hardware check of that specific claim — not a new derivation, not a new kernel design, and not a comparison against other systems’ kernels (Section 7 states this scope explicitly). A reader does not need to have read Papers I–IV to follow this paper’s results, but should understand that every design tested here, and every prediction being checked, was formulated and derived there — this paper’s role is strictly to measure, not to design or derive. Papers I–IV establish, by formal derivation and machine-precision verification against PyTorch, that MoA’s DNF/ONF pipeline produces memory-optimal kernels for attention (forward, backward, fused), inference-time decode, and the complete transformer block (RMSNorm, gated MLP, and their integration with attention via a QKV projection layer). Every one of those verifications, however, was conducted at a single small fixed problem size on one CPU core or via OpenACC’s host-fallback compilation path — a correctness proof, not a performance measurement. This document reports what happens when the same kernels are actually run at scale, on real multi-core CPUs and real GPUs.

How the four papers build toward one goal: memory-optimal kernels for the complete transformer block, validated on real hardware

Paper III Decode / inference (KV-cache, O(n))

Paper I

Paper II

Forward pass (attention scores)

Backward + fused (eliminates A)

COMPLETE

Deploy real-world inference/training

Paper IV Full block (norm+MLP+attn) COMPLETE

Figure 1: How Papers I–IV relate to each other and to this document’s status as of this draft. All four papers now have complete experimental validation on both CPU and GPU, real hardware, both clusters. 2

1.1

From array algebra to a real machine: DNF, ONF, and γ

MoA’s five primitives [6] — ψ (psi, indexing), ι (iota, index generation), ρ (rho, shape), γ (the DNF→ONF translator), and rav (ravel) — are usually introduced as pure array algebra: a way to write what a computation produces without committing to how. Papers I–IV use them exactly that way to derive each kernel’s Denotational Normal Form (DNF) — a specification of the output purely in terms of shape and index relationships, with no memory-access order, no loop structure, no notion of ”materialized” versus ”consumed immediately” at all. Two denotationally identical DNFs (naive and fused attention, say) can still differ enormously in real cost, because DNF alone says nothing about a machine. That is what γ is for. γ translates a DNF into an Operational Normal Form (ONF): a realization that commits to loop order, tiling, and which intermediate values are recomputed versus stored. Crucially, γ’s output is expressed in the same shape/index vocabulary (ρ, ψ, ι) as the DNF it came from — because on any real machine, an array’s shape and index structure is its memory layout and access pattern. This is the sense in which MoA treats the target machine as an array: DRAM address space, cache lines, and thread-to-data assignment are themselves describable in ρ/ψ/ι terms, so a γ-derived ONF is not a metaphorical ”implementation guide” but a literal, checkable claim about what bytes move where on real silicon. The cost functions Mfwd , Mbwd , Mfwd+bwd , Mblock are read directly off the ONF’s memory-access structure — a byte count, not an abstract complexity class — which is precisely why they can be measured on hardware and are not merely asserted. This document’s entire methodology is a test of that claim. If γ’s ONF genuinely describes real machine behavior, then a kernel’s measured DRAM traffic, atomic-instruction count, or thread scaling should match what its ONF predicts — on any machine whose memory system can itself be described in ρ/ψ/ι terms, not just the one it was derived on. Sections 3 and 4.4 test this directly, on real DRAM byte counts and real atomic-instruction counts respectively. Where the match is exact (Section 4.4’s 2.0000×), the ONF’s machine model is confirmed precisely, not just qualitatively. Where it diverges (Section 3’s 2× DRAM gap, Section 4.3’s GPU regression before its fix), the gap is diagnosable in the same ρ/ψ/ι vocabulary — a specific access-pattern or contention mechanism, not a vague ”hardware is slower than expected.” This document does not depend on these results to validate DNF/ONF as formal constructions; the parent papers’ correctness proofs against PyTorch already do that independently. What it tests is narrower and empirical: whether γ’s specific machine-as-array claim holds up when the array in question is a real chip’s memory system, on real multi-core CPUs and real GPUs, across two different clusters (Section 2).

1.2

Related work

MoA’s array-algebra lineage traces to Iverson’s notation for describing array operations [7], later realized as a working compiler and machine model by Abrams [8]; MoA’s own DNF, ONF, and γ formalize and extend this line specifically toward provable memory-cost derivation, the parent papers’ own contribution and not this paper’s. On the systems side, this paper’s method — deriving a memory-traffic prediction analytically and then checking it against measured hardware counters — is the same spirit as the Roofline model’s use of measured operational intensity to bound achievable performance [9], applied here to a specific derivation rather than a general architectural ceiling. For the specific kernel this paper spends the most attention on, FlashAttention [10] independently arrives at essentially the same qualitative conclusion this paper’s Section 3 and Section 4.2 confirm quantitatively for MoA’s own derived kernels: avoiding materialization of the n × n attention score matrix in DRAM is the dominant lever for attention’s real performance. We note this as context,

The core MoA idea: eliminate intermediate array materialization This is what Paper II's fusion does -- and what Section 3.3/4.4's real DRAM and atomic-count measurements directly confirm on hardware

Naive (unfused)

Fused (memory-optimal)

Q, K, V

compute QK^T

compute scores, consume immediately (A never leaves registers/cache)

materialize A (n x n) A sits in DRAM (n^2 bytes)

no DRAM round-trip

write to DRAM

read back from DRAM

softmax + backward pass

GQ, GK, GV

Figure 2: Two ONFs realizing denotationally equivalent DNFs for attention’s forward+backward pass. Left: A (n × n) is materialized to DRAM and read back — two full n2 -scaling memory round-trips. Right: γ produces an ONF where A is consumed the instant it is computed and never touches memory. This is not an abstract claim in this document: Section 3 measures the left path’s real DRAM traffic directly, and Section 4.2 measures the right path’s real wall-clock advantage on CPU.

not comparison: this paper does not benchmark MoA-derived kernels against FlashAttention or any other externally-optimized implementation, and makes no claim about relative competitive performance. This paper’s contribution relative to that line is a different question entirely — whether a formally-derived, machine-precision-verified cost function correctly predicts how much that avoidance is worth for MoA’s own designs, on real hardware, and where it does not, why. Finally, Section 5’s cross-cluster NUMA finding sits in a well-established line of NUMA-performance characterization work (see, e.g., [11] for a systems-level overview of the mechanism); this paper’s contribution there is narrower and more specific: showing that a single formally-derived kernel’s predicted cost function, unmodified, already anticipates that such a machine-dependent penalty must exist, before any NUMA-specific measurement is taken.

Methodology

2.1

Machines and allocations

Cluster

Local SLURM account

Resources used

Purdue Anvil NCSA Delta

cis261396-ai (CPU), cis261396-gpu (GPU) bibg-delta-cpu, bibg-delta-gpu

wholenode (128-core), gpu (A100, H100) cpu (128-core), gpuH200x8

gpuA100x4/x8,

gpuA40x4,

Both clusters are used under the same top-level ACCESS allocation, CIS261396; the account names above are each resource provider’s own local naming convention for job submission, not separate allocations (see Acknowledgments). gpuMI100x8 (AMD, ROCm) is present on Delta but excluded from this study: every kernel here is compiled with nvc/OpenACC, which does not target AMD hardware.

2.2

Kernels benchmarked

Each paper’s CPU kernel (OpenMP) and, where fixed (Section 2.3), GPU kernel (OpenACC) is benchmarked by sweeping sequence length n (doubling) and, for CPU runs, thread count (doubling, 1 to 128). Every kernel is verified for correctness (machine-precision agreement with a PyTorch or serial-CPU reference) independently of these performance runs; this document reports timing, not correctness, except where noted.

2.3

A real methodological bug: GPU cold-start dominance

The first real GPU measurement obtained in this effort (Paper III’s decode kernel, NVIDIA A100SXM4-40GB on Delta) showed wall-clock time essentially flat, 0.30–0.40s, across a sequence-length range of n = 1024 to n = 1,048,576 — a 1024× range in problem size producing almost no change in measured time. This is inconsistent with decode’s O(n) cost. The cause: the original benchmark harness launched one process per measured point, and CUDA context creation plus OpenACC JIT compilation cost approximately 300ms per process launch on this hardware, completely dominating the true kernel time (sub-millisecond even at n = 106 on an A100). Every GPU benchmark program in this project (six kernels: forward, backward, fused attention, decode, fused norm+MLP, and the complete block) shared this flaw, since all were built from the same per-invocation-timing pattern. The fix applied to all six: restructure each program to sweep its entire n range within one process, paying the context-init cost exactly once via an explicit untimed warmup call before any 5

measurement begins. Post-fix, the decode kernel’s re-run showed the expected O(n) behavior (0.00014s at n = 1024 rising to 0.72s at n = 1,048,576, roughly tracking the 1024× size increase). The other five kernels were fixed identically and, at the time of that fix, verified only for unchanged correctness and now-sensible relative scaling in this report’s development sandbox. All six have since been confirmed on real GPU hardware — Sections 3, 4.3, 5, and 6 report their real A100/H200 data. This is reported here rather than silently corrected because it is itself a finding relevant to anyone benchmarking small-sequence-length GPU kernels: a naive per-call timing harness will systematically overstate cost by a roughly constant ∼300ms at every problem size, and will completely obscure the true asymptotic complexity of a fast kernel unless that constant is isolated first.

Paper I: Forward Pass

3.1

CPU scaling (Delta, confirmed)

Job 21181232, Delta cpu partition (128 cores), sweeping B=2, D=64, n ∈ {64, . . . , 8192} (doubling), threads ∈ {1, . . . , 128} (doubling), validates the forward-pass cost function derived in [2]. Table 1 shows, for each n, the thread count achieving the lowest wall-clock time and how that compares to running at the full 128 threads. n

Best thread count

Speedup at best (vs. 1 thread)

128 threads vs. best

64 128 256 512 1024 2048 4096 8192

4 8 16 32 64 128 128 128

1.4× 4.0× 7.3× 16.4× 23.4× 43.3× 53.5× 60.6×

14.5× slower 5.5× slower 3.8× slower 2.3× slower 1.7× slower — (128 optimal) — —

Table 1: Paper I forward pass, Delta CPU, best thread count per sequence length and the cost of over-threading small problems. Two findings, both physically sensible and neither assumed in advance: 1. Optimal thread count is a function of n, not a fixed cluster-wide setting. At n = 64, using all 128 threads is 14.5× slower than the true optimum of 4 threads — thread-spawn and scheduling overhead dominates when there is too little work per thread. The crossover to ”more threads always help” falls between n = 1024 and n = 2048. This has a direct practical implication: any deployment guidance derived from this cost function should state thread count as a function of sequence length, not a single recommended value. 2. The cost-function’s O(n2 ) prediction is confirmed. At a fixed 1 thread, wall time approximately quadruples with each doubling of n (0.0013, 0.0067, 0.0271, 0.1388, 0.445, 2.13, 7.49, 24.57 seconds for n = 64, . . . , 8192) — the expected signature of the n × n attention score matrix that dominates Mfwd . At the best-case thread count for the largest tested size (n = 8192, 128 threads), speedup over 1 6

thread is 60.6× — 47% parallel efficiency on 128 cores, consistent with a memory-bandwidth-bound kernel rather than a compute-bound one.

Paper I forward pass, Delta CPU (job 21181232) Optimal thread count depends on N N=64 N=128 N=256 N=512

101

N=1024 N=2048 N=4096 N=8192

wall time (s)

100 10 1 10 2 10 3 100

101 threads

102

Figure 3: Wall time versus thread count, one curve per sequence length n (log-log), Delta CPU, job 21181232. Small-n curves (purple, top of the legend) bottom out early and turn back upward at high thread counts — the visual signature of Table 1’s finding that over-threading small problems actively hurts. Large-n curves (yellow/green) decrease monotonically all the way to 128 threads, with no such penalty.

3.2

GPU shape comparison

The v1 run (job 21181233) is superseded. It used the flawed timing methodology (Section 2.3) and the 512-element N cap; its numbers should not be treated as final and are not reproduced here except where explicitly noted for comparison. Real, confirmed results (post-fix). Job 21202354 (gpuA100x4, NVIDIA A100-SXM4-40GB) and job 21202355 (gpuH200x8, NVIDIA H200), both using the fixed v2 binary (repeated-andaveraged timing, 8192-element buffer), same B=2, D=64 sweep as the CPU run, n ∈ {64, . . . , 8192} — full parity with the CPU’s tested range. The fix is confirmed. Unlike the flawed v1 run, the local exponent now rises toward 2 as n grows on both GPUs, reaching 1.97 (A100) and 2.04 (H200) at n = 8192 — essentially exact agreement with the O(n2 ) prediction. This is the direct empirical confirmation that Section 2.3’s diagnosis (single-measurement jitter at small n, resolved by repeated averaging) was correct, not merely plausible. H200 is faster, and the margin has a clear pattern. H200 beats A100 at every tested size, but the speedup shrinks monotonically from 1.75× at n = 256 down to 1.06× at n = 8192. 7

A100 (s)

H200 (s)

H200 speedup

A100 local exp.

H200 local exp.

64 128 256 512 1024 2048 4096 8192

0.000242 0.000383 0.000788 0.002899 0.008808 0.031952 0.116893 0.457188

0.000166 0.000235 0.000450 0.001759 0.005421 0.027075 0.104600 0.431261

1.46× 1.63× 1.75× 1.65× 1.62× 1.18× 1.12× 1.06×

— 0.66 1.04 1.88 1.60 1.86 1.87 1.97

— 0.50 0.94 1.97 1.62 2.32 1.95 2.04

Table 2: Paper I forward pass, real GPU data post-fix. Local exponent computed between each consecutive pair of points, same definition as the flawed v1 table this supersedes.

Paper I forward pass, post-fix real GPU data A100 vs H200, both converge toward O(n^2)

wall time (s)

100

A100 (fixed, job 21202354) H200 (job 21202355) O(n^2) reference

10 1 10 2 10 3

102

103

104

Figure 4: Corrected A100 and H200 wall time versus n, against an O(n2 ) reference. Both curves visibly bend to run parallel to the reference at large n — the fix works.

2.5

Paper I forward pass: local exponent converges to 2 Direct visual confirmation the timing fix works

local scaling exponent

2.0 1.5 1.0 0.5 0.0

A100 (job 21202354) H200 (job 21202355) O(n^2) target (exponent = 2)

103 N (right edge of each doubling interval)

104

Figure 5: The local exponent columns from Table 2, plotted directly. Both GPUs’ exponents climb from well below 1 (at small n, launch overhead dominates) toward the O(n2 ) target of 2 (dotted line) as n grows – shown honestly, including the real dip near n ≈ 1024 rather than a smoothed trend, since this is the actual trajectory measured, not a fit.

H200’s advantage is concentrated at smaller, more launch-overhead-sensitive problem sizes; at the largest tested n the two architectures nearly converge. This is a genuine machine-shape finding, not an artifact of either fix.

3.3

Validation against Mfwd

delta profile paper1.sbatch was run for real (job 21202356, Delta gpuA100x4), profiling forward gpu bench.c at B=2, N =2048, D=64 with both nsys and ncu. Three kernel launches were captured: an initial global warmup at N =64, then a per-N warmup and the single timed repeat, both at N =2048 (grid size (4096, 1, 1), confirming B×N =4096 threads as expected). DRAM traffic was extracted via ncu --import ... --page raw --csv --metrics dram bytes read.sum,dram bytes write.sum. Read (MB)

Write (MB)

Total (MB)

6.291 9.045 1.44×

69.206 142.333 2.06×

75.497 151.378 2.01×

Predicted (naive DNF element count) Measured (ncu, real A100) Ratio (measured / predicted)

Table 3: Measured DRAM traffic (kernel ID 2, the timed N =2048 run) against a naive elementcount prediction from the DNF: read Q, K, V once each, write A (n × n) and Out once each, 8 bytes per double-precision element.

Paper I forward pass, N=2048: predicted vs measured DRAM traffic Writes hit hardest (2.06x) -- consistent with ncu's coalescing diagnostics 140

Predicted (naive DNF count) Measured (ncu, real A100)

151.4

142.3

DRAM traffic (MB)

120 100 80

75.5

69.2

60 40 20

9.0

6.3

Read

Write

Total

Figure 6: Table 3’s three rows as grouped bars. The gap between predicted and measured is visually modest for reads but dramatic for writes and total traffic – the write/read asymmetry is immediately obvious here in a way the table’s ratio column requires computing. 10

Measured traffic is 2.01× the naive prediction, and the gap is not uniform. Writes are hit far harder than reads (2.06× vs. 1.44×). ncu’s own diagnostics explain why: uncoalesced memory access. Global stores utilize only 9.4 of 32 bytes per transmitted sector (29% efficient); global loads utilize 18.2 of 32 (57% efficient). The corresponding worst-case inflation factors (32/9.4 ≈ 3.4× for stores, 32/18.2 ≈ 1.76× for loads) bound the observed ratios from above, consistent with partial mitigation from L2 cache reuse rather than every access paying the full uncoalesced penalty. This is an implementation gap, not a derivation gap. The DNF itself is memory-optimal by construction — no intermediate array beyond Q, K, V, A, Out is ever materialized, exactly as derived. What this profiling run shows is that nvc’s code generation for the current OpenACC source does not achieve fully coalesced access to those arrays, so the real hardware cost exceeds the algorithmic memory-optimal cost by a factor of about 2× at this problem size. ncu’s own advisory output points at a specific, addressable cause (stride pattern in the arow indexing), making this a concrete follow-up code task rather than an open question about whether the theory is right. Correction to occupancy claim, and what the fuller data actually shows: an earlier draft of this section stated that ncu flagged the N =2048 kernel itself as achieving only 0.2 full waves across all SMs. That was wrong. Re-checking against the full profiler export (full ncu dump.csv, 1647 columns, all three captured kernel launches) shows the 0.2-waves figure belongs to kernel ID 0 — the N =64 global warmup call, grid size (128, 1, 1) — not kernel ID 2, the actual timed N =2048 run, grid size (4096, 1, 1). The real numbers for N =2048: launch waves per multiprocessor = 5.42, achieved occupancy 41.7% against a register-limited theoretical ceiling of 43.8% — 95.4% of that ceiling reached. The N =2048 kernel is not under-saturated; only the small warmup call was. Figure 7 shows both side by side. A genuine new finding from the fuller export: device-timer versus wall-clock crossvalidation. ncu’s own gpu time duration.sum for the N =2048 kernel is 26.406ms = 0.026406s — measured directly on-device, independent of this project’s own clock gettime-based wall-clock timer. The sweep CSV’s wall-clock time at the same N is 0.031952s. Figure 8 shows the gap: 17.4% of the measured wall time is host-side overhead (memory allocation, data transfer setup) that never touches the GPU, not GPU compute time itself. This is a real, independent cross-check of the wall-clock methodology used throughout this document, and it’s reassuring: the two timers agree to within the expected host-overhead margin, not by orders of magnitude, which would have suggested a deeper measurement problem.

3.4

Scope decision and summary

No Fortran90 GPU kernel exists for Paper I (forward gpu bench.f90 was not built), by explicit decision — Paper I’s GPU story is C-only. Paper I’s experimental validation is complete. All three planned real-hardware runs (corrected A100, second shape H200, Mfwd profiling) have been performed and analyzed. The headline results: the cold-start and measurement-noise fixes both work, confirmed by the local exponent converging to ≈ 2 on real hardware; H200 outperforms A100 with a size-dependent margin; and real DRAM traffic exceeds the algorithm’s memory-optimal prediction by roughly 2× at N =2048, attributable to a specific, fixable memory coalescing issue in the current code generation rather than any flaw in the underlying derivation.

value (waves scaled x10 for visibility)

100

GPU occupancy: N=64 (warmup) is severely under-saturated, N=2048 (the actual measurement) is not

95.40

N=64 (warmup kernel) N=2048 (the actual timed run)

80 60

5.42 41.73

40 20 0

16.90 7.41

0.17 Waves per SM (x10 for scale)

Achieved occupancy (%)

% of theoretical ceiling reached

Figure 7: Occupancy metrics for the N =64 warmup kernel versus the N =2048 timed kernel, from the same profiling run. The warmup call is severely under-saturated (16.9% of its theoretical occupancy ceiling); the kernel actually being measured reaches 95.4% of its ceiling. The undersaturation story applies to the warmup, not the measurement.

Wall clock (0.032s) vs GPU device timer (0.026s) at N=2048 17.4% of measured wall time is host-side, not GPU compute

N=2048 wall time

0.0264s

0.0055s

Device time (ncu gpu__time_duration) Host-side overhead (malloc, transfer setup, etc.)

0.000

0.005

0.010

0.015 0.020 time (s)

0.025

0.030

0.035

Figure 8: N =2048’s wall-clock time decomposed into GPU device time (ncu’s own timer) and host-side overhead (the remainder).

Paper II: Backward Pass and Fused Forward+Backward

4.1

First real submission: a genuine crash, found and fixed

Jobs 21225788 (CPU), 21225789 (gpuA100x4), 21225790 (gpuH200x8) were the first real submissions validating the backward-pass and fusion cost functions derived in [3]. Both GPU jobs completed cleanly. The CPU job did not: sacct reported FAILED, exit code 139 — 128 + 11, SIGSEGV. This is a different failure class from anything else in this document: not a missing file, not a timing-methodology gap, an actual crash. Diagnosis. Both backward bench.c and fused bench.c used OpenMP array-reduction clauses (reduction(+:GK[0:B*N*D]), and fused bench.c had two such clauses) to handle an accumulate-across-threads situation. GCC’s libgomp implements the per-thread private copy for an array reduction on the stack, not the heap. At the sweep’s default D=64, the CPU job reached N =8192, where B × N × D × 8 bytes = 2 × 8192 × 64 × 8 = 8,388,608 bytes = 8MB exactly — matching the cluster’s default stack limit (ulimit -s = 8192) precisely. Reproduced locally in this sandbox, which has the identical 8MB ulimit -s, before attempting any fix: the exact same crash, exit code 139, confirming the diagnosis rather than guessing at it. Fix. Both files were restructured to eliminate every array reduction. Each was rewritten as a sequence of separate passes, each race-free by construction (every output element written by exactly one thread, never accumulated across parallel iterations), using heap-allocated persistent intermediate arrays (GS for backward bench.c; AROW and GS for fused bench.c) in place of the reduction clauses. Verified after the fix: exact 0.000e+00 correctness at the original small scale, and no crash at the exact buffer-size trigger condition (B=2, N =64, D=8192, chosen to reproduce the identical 8MB buffer size fast rather than waiting through N =8192’s full O(n2 ) compute time), confirmed race-free at 1, 2, and 4 threads. A codebase-wide search for the same reduction(+:ARRAY[...]) pattern found four more instances, all in Paper IV’s files (mlp bench.c, full block bench.c, rmsnorm bench.c) — but their reduction buffers scale with D or D × Dff (fixed, small: e.g. 64 × 256 = 128KB), not with the swept N dimension that caused this crash. Tested each at its own sweep’s real parameters (N up to 65536 for mlp bench.c/rmsnorm bench.c, 4096 for full block bench.c) rather than assumed safe by the size argument alone: no crash in any of the three. The bug was genuinely isolated to the two files whose reduction buffer size scaled with N .

4.2

CPU results: the real fusion comparison (job 21226440)

The CPU sweep completed successfully after the fix, full N = 64 . . . 8192 range, threads 1 . . . 128 doubling, no crash. This is the section that actually tests Paper II’s central claim. A subtlety in what ”backward” and ”fused” each measure. backward bench.c times only the backward pass — forward is computed first, untimed, as a prerequisite (Algorithm 1 reads A “from DRAM”). fused bench.c times forward and backward together (Algorithm 2). Comparing these two columns directly, as an earlier draft of this document did, is not the right test: it compares backward-alone against forward+backward combined, which will make fused look artificially worse since it does strictly more total work. The correct comparison is naive = forwardalone +backwardalone against fusedalone — and Paper I’s own real forward-pass data (job 21181232, same B, D, same swept N and thread range) is exactly what’s needed to construct that baseline. Combined here for the first time.

Paper II, N=4096, Delta CPU (jobs 21181232, 21226440) Fused's advantage grows with thread count: 1.01x -> 0.72x naive time Naive: forward + backward (separate) Fused (Algorithm 2)

wall time (s)

102

101

100

101 threads

102

Figure 9: N = 4096: naive (forward alone + backward alone, computed by combining jobs 21181232 and 21226440) against fused (job 21226440), across the full thread sweep. The two lines are nearly identical at 1 thread and visibly diverge as thread count grows.

At N = 4096, the cleanest trend in this dataset: fused’s advantage grows monotonically with thread count, from parity (1.19× naive at 1 thread – fused very slightly slower) to 0.72× naive at 128 threads (fused 28% faster): threads

naive (s)

fused (s)

fused/naive

1 2 4 8 16 32 64 128

175.58 106.84 54.47 31.69 17.63 9.68 5.57 3.40

208.62 108.12 55.20 29.84 15.95 8.17 4.58 2.44

1.19× 1.01× 1.01× 0.94× 0.90× 0.84× 0.82× 0.72×

This is exactly the pattern the theory predicts: as thread count grows and more cores compete for memory bandwidth, the cost of the n2 -scaling A array that naive forward+backward must materialize and re-read becomes proportionally more expensive, while fused (which never writes A to memory) is comparatively insulated from that contention. The full picture, across all N at the max thread count, is noisier. N

naive (s)

fused (s)

fused/naive

64 128 256 512 1024 2048 4096 8192

0.0230 0.0195 0.0278 0.0438 0.1266 0.2777 3.3961 12.8322

0.0094 0.0403 0.0123 0.0482 0.0646 0.2754 2.4362 12.2691

0.41× 2.07× (outlier) 0.44× 1.10× 0.51× 0.99× 0.72× 0.96×

At threads=128:

Six of eight sizes favor fused (ratios below 1); two do not, including one clear outlier at N = 128 (fused 2.07× slower than naive, inconsistent with every neighboring size). Likely explanation, consistent with a pattern already established in this document: neither backward bench.c nor fused bench.c performs repeated-and-averaged timing — a single measurement per (N, threads) point, the same methodological gap Section 2.3 and the GPU exponent discussion (Section 3) already identified and partially fixed on the GPU side. This CPU sweep does not yet have that fix. The N =4096 trend above is convincing specifically because it’s a monotonic 8-point progression, not a single measurement – consistent with a real effect rather than noise; the single-point N =128 anomaly is far more likely to be exactly what an unaveraged measurement looks like when it happens to catch scheduling jitter. Adding repeated averaging to backward bench.c and fused bench.c (matching moa decode bench.c’s already-correct pattern) is a concrete next code task, not yet done.

4.3

GPU results (jobs 21225789, 21225790): a surprising, opposite finding

Same correction as Section 4.2: backward gpu bench.c times backward alone, fused attn gpu bench.c times forward+backward together, so the naive baseline is again 15

constructed by combining Paper I’s real forward GPU data (jobs 21202354 A100, 21202355 H200) with these jobs’ backward data.

Paper II GPU: fused is SLOWER than naive at every size (opposite of the CPU N=4096 result)

fused time / naive time

1.8

A100 (jobs 21202354, 21225789) H200 (jobs 21202355, 21225790) Parity (fused = naive)

1.6 1.4 1.2 1.0 102

103

104

Figure 10: Fused time divided by naive (forward-alone + backward-alone) time, both GPU shapes. Values above 1.0 mean fused is slower than doing the two passes separately – true at every tested size on both GPUs. On GPU, fused is slower than naive at every single tested size, on both A100 and H200 — the opposite of Section 4.2’s CPU N =4096 result. A100 ranges from 1.14× naive time (at N =64) up to 1.84× (at N =4096), with a visible step change between N =512 and N =1024 (ratio jumps from 1.29× to 1.78×) rather than a smooth trend. H200 is comparatively flat, holding close to 1.5× across the entire range regardless of N .

4.4

The atomic-contention hypothesis: confirmed

delta profile paper2.sbatch (job 21247666, Delta gpuA100x4) profiled both kernels at N =2048 with ncu. Getting a working profile required one more real, Delta-specific fix beyond anything in Paper I’s profiling: ncu initially failed with ERROR: Profiling failed because a driver resource was unavailable, a known conflict with DCGM (NVIDIA’s Data Center GPU Manager, running as background monitoring on shared GPU nodes) — documented in Delta’s own user guide, resolved by wrapping the ncu calls in dcgmi profile --pause/--resume. The profiler output itself revealed a structural difference beyond atomics: backward gpu bench.c launches three separate kernels (GV , then a precompute pass, then GQ+GK), while fused attn gpu bench.c does everything in one monolithic kernel. Extracting smsp inst executed op generic atom dot alu.sum (atomic-ALU instructions executed) confirms the hypothesis precisely rather than approximately:

Paper II, N=2048: atomic instructions and device time fused has EXACTLY 2.00x backward's atomics, 1.97x its time 1074M

backward (atomics: GK only) fused (atomics: GV and GK)

value (instructions in millions, or ms)

1000 800 600

537M 439.6ms

400 223.0ms

200 0

Atomic instructions (GK / GV+GK)

Device time (sum of kernels)

Figure 11: Atomic instruction count and total device time, backward (summed across its 3 kernels) versus fused (1 kernel), both at N =2048. Both ratios land within 1.5% of exactly 2×.

backward (GK only) fused (GV and GK) Ratio (fused / backward)

Atomic instructions

Device time (ms)

536,870,912 1,073,741,824 2.0000×

222.98 439.63 1.9716×

Fused executes exactly 2.0000× backward’s atomic instructions — not approximately double, precisely double, matching the theoretical prediction (fused needs atomics for both GV and GK; backward needs them only for GK) to four decimal places. Device time tracks almost identically (1.9716×). This is no longer a plausible-but-unconfirmed mechanism; it is a measured, essentially exact correspondence between atomic-instruction count and wall-clock cost. The per-kernel breakdown of backward adds a further finding: its two atomic-free kernels (GV , the precompute pass) run fast and at healthy occupancy (44.3%, 47.9%), while its one atomic-using kernel (GQ+GK) is both the slowest (190.15 of the kernel’s 222.98ms total, 85%) and the least occupied (14.2%) of the three. The atomic-using kernel is already backward’s own bottleneck; fused inherits and doubles it. The practical implication, now confirmed rather than speculated: fusing forward and backward into a single OpenACC kernel is a real regression on GPU as currently implemented, caused specifically by atomic contention, even though the same fusion is a real 28% improvement on CPU (Section 4.2) where the OpenMP versions of these kernels were restructured to avoid exactly this contention (Section 4.1). Paper II’s fused kernel is the right choice on CPU and the wrong choice on GPU in the current implementation — and the fix is now well-specified: restructure fused attn gpu bench.c the same way backward gpu bench.c and the OpenMP kernels were restructured, splitting the monolithic kernel to avoid atomic writes to GV . 17

4.5

The fix, applied and confirmed: complete reversal on both shapes

fused attn gpu bench.c was restructured into four passes (v3), mirroring backward gpu bench.c exactly: GV now computes in its own kernel parallelized over (b, ic) instead of (b, ir), eliminating its atomic; GK’s atomic is left in place, matching backward gpu bench.c’s own remaining structure — a deliberately scoped fix, not a full rewrite. Verified to exact 0.000e+00 correctness at N = 64, 128, 256, 512 before being run on real hardware. Real nvc compiler diagnostics (job 21250395) confirm the intended structure: four separate kernel launches where there was one, with the new GV kernel showing the identical dependency pattern as backward gpu bench.c’s own GV kernel, and GK’s kernel showing the same ”loop carried dependence” pattern on both files, unchanged.

Paper II GPU: eliminating the GV atomic reverses the result completely Before: fused always slower. After: fused always faster, both shapes. A100, before fix (jobs 21202354/21225789) H200, before fix (jobs 21202355/21225790) A100, after fix (job 21250395) H200, after fix (job 21250396) Parity (fused = naive)

1.8

fused time / naive time

1.6 1.4 1.2 1.0 0.8 0.6 0.4 102

103

104

Figure 12: Fused/naive time ratio, before (dashed, jobs 21202354/21225789 A100 and 21202355/21225790 H200) and after (solid, jobs 21250395 A100 and 21250396 H200) the GV atomic fix. Every before-fix point sits above parity; every after-fix point sits below it. The result is not a partial improvement — it is a complete reversal, on both GPU shapes. Before the fix, fused was slower than naive at every tested size on both A100 (1.14–1.84×) and H200 (1.34–1.53×). After the fix, fused is faster than naive at every tested size on both: A100 from 0.757× (N =64) to 0.517× (N =8192, essentially 2× faster); H200 from 0.920× down to 0.401× (N =8192, 2.5× faster). The improvement grows with N on both shapes, consistent with the atomic-contention cost (which the removed GV atomic contributed half of, per Section 4.4’s exact 2.00× measurement) mattering more as more threads compete for the same memory locations at larger problem sizes — the same qualitative pattern already seen for the CPU fix in Section 4.2. This closes Paper II’s central empirical question. Fusing forward and backward to avoid materializing the n2 -scaling A array is now confirmed to be the right choice on both CPU and GPU, once the GPU implementation’s own atomic-contention cost — an implementation detail 18

orthogonal to the DRAM-traffic argument the theory makes — is addressed.

4.6

Removing GK’s atomic too: a real, nuanced result

New time / Old time (GK atomic removed / GK atomic present)

The single remaining atomic (GK) was deliberately left in place for the fix above; the natural next question — would removing it too, via the same persisted-GS technique already applied to GV , help further — was tested (job 21294299, real A100, both backward gpu bench.c and fused attn gpu bench.c). Both files verified to exact 0.000e+00 correctness at multiple sizes before submission.

Removing GK's atomic too: real A100 results Uniformly good for backward alone; mixed for fused (net win still holds vs naive) 1.3

backward alone: uniformly faster fused: faster at small/medium N, SLOWER at N>=4096 Parity (no change)

1.2 1.1 1.0 0.9 0.8 0.7 0.6 0.5

102

103

104

Figure 13: New/old time ratio after removing GK’s atomic, real A100. backward alone improves uniformly; fused improves at small/medium N but is measurably slower at N ≥4096 — a real crossover, not noise. The result is genuinely mixed, not a second clean win. For backward alone, removing GK’s atomic helps at every tested size (0.56–0.81× the old time, i.e. 19–44% faster) — consistent with the earlier finding that atomic contention is pure overhead for this kernel, with no offsetting benefit. For fused, the result depends on N : faster at small and medium sizes (0.53–0.79× up to N =2048), but slower at N =4096 (1.04×) and N =8192 (1.32×) — a real crossover, confirmed at two consecutive sizes, not a single noisy point.

4.7

Confirming the mechanism: the extra GS re-read, profiled directly

The mechanism proposed for fused’s slowdown — that the new, separate GK pass re-reads the full GS array from DRAM a second time — was tested directly, not left as speculation. A reconstruction of the pre-fix kernel (fused attn gpu bench v3atomic.c, GK’s atomic restored, verified to exact

0.000e+00 correctness) was profiled against the current atomic-free version at N =4096, the size where the real slowdown first appears (job 21299223; the first attempt at this profiling run OOMkilled under ncu’s full metric set at this problem size, and was fixed by requesting only the specific DRAM-traffic metrics needed rather than the comprehensive set — itself a small, real instance of the same discipline this document applies throughout: diagnose the actual cause of a failure rather than working around it blindly). The result confirms the mechanism precisely. Total DRAM traffic (read+write, summed across all kernels comprising one call) is 89.83GB for the atomic-free version versus 45.60GB for the atomic version — a 1.97× ratio, close to a clean doubling. Tracing this to a specific kernel: the atomic-free version’s new, standalone GK pass reads 43.01GB on its own, almost exactly matching the 44.10GB read by the separate kernel that computes GS in the first place. The atomic version’s single combined GQ+GK kernel, by contrast, reads only 0.30GB total — because it consumes each thread’s own GS value immediately after computing it, with no DRAM round-trip required. This is not a plausible story fit to the timing data after the fact; it is the predicted mechanism, checked against real profiler measurement, and confirmed at essentially the magnitude predicted (an extra full read of an array the same size as the one already being read once). Despite this, fused still beats naive at every tested size (0.65–0.89×, Figure 13 is a same-file before/after comparison, not fused-vs-naive) — because naive’s own backward component improved from the same fix, both sides moved, and the comparison that actually matters for Paper II’s central claim still favors fusion everywhere. The honest conclusion: removing GK’s atomic is a clear win for backward in isolation, and a real, now-explained (not just observed) size-dependent trade-off for fused specifically — not a mechanism to apply uniformly without checking which kernel and what scale it targets, and now with a confirmed, specific, reusable diagnosis (an added array re-read outweighing a removed atomic, in DRAM-traffic terms that can be checked the same way for any future fix of this shape) rather than a plausible-sounding guess.

Paper III: Inference (Decode)

5.1

CPU: two real findings, one benign, one genuinely surprising

Job 21180660 (Delta cpu partition), submitted early in this project, validates the decode cost function derived in [4]. It completed with exit code 0 but left no trace on disk by the time this draft checked its output — no log, no CSV, found nowhere in a full home-directory search. Its result is unrecoverable and is not used here. It was resubmitted cleanly as job 21249667 with the current, pre-flight-checked script; that run is what this section reports. Finding 1: a correctness warning that is real but benign. moa decode bench.c uses single-precision float throughout (every other kernel in this project uses double), and self-checks its parallel output against a sequential reference with an absolute-error threshold of 10−3 . At N = 1,048,576 (the largest tested size), every thread count from 2 to 128 triggered this warning, with essentially uniform magnitude (3.00×10−3 –3.05×10−3 ). This pattern — stable across thread counts rather than scaling with them, appearing only at the single largest N — is the expected signature of accumulated single-precision summation-order differences between parallel and sequential reduction over roughly a million terms, not a logic error. A race condition would be expected to fluctuate with thread count or appear intermittently; this does neither. Finding 2: a genuine, large NUMA-topology penalty at full thread count. Comparing 64 threads against 128 threads (both correctly requested via cpus-per-task=128, so this is not a 20

resource-mismatch artifact):

Paper III decode, Delta CPU (job 21249667): 128-thread NUMA penalty Up to 535x slower at small N -- resolves once N amortizes cross-NUMA cost 64 threads (fits within 1 socket, 4 NUMA domains) 128 threads (spans both sockets, all 8 NUMA domains)

wall time (s)

10 1

10 2

10 3

10 4 104

103

N (KV-cache length)

105

106

Figure 14: Wall time versus N , 64 vs. 128 threads. 128 threads is flatly slow (∼0.04–0.05s) regardless of N for small-to-medium sizes – consistent with fixed cross-NUMA overhead dominating completely, not scaling with the (tiny) actual work at these sizes – before a sharp dip at N =65,536 and convergence with the 64-thread line at large N .

64 threads (s)

128 threads (s)

128 vs. 64

1024 2048 4096 8192 16384 32768 65536

0.000083 0.000098 0.000154 0.000217 0.000288 0.000358 0.001128

0.044417 0.046780 0.044054 0.039628 0.040900 0.034005 0.000627

535.1× slower 477.3× slower 286.1× slower 182.6× slower 142.0× slower 95.0× slower 0.6× (128 threads faster)

Delta’s CPU nodes (confirmed via lscpu) are dual-socket AMD EPYC 7763 (Milan), 64 cores per socket, but subdivided into eight NUMA domains total — four per socket, 16 cores each (the NPS4 BIOS configuration common on this generation of EPYC). 64 threads can fit within a single socket’s four NUMA domains; 128 threads must span all eight, all sixteen memory controllers, both sockets’ cross-socket interconnect. At N = 1024, decode’s actual per-token work is minimal; the observed 535× penalty is consistent with threads scattered across all eight NUMA domains paying full remote-memory latency for K/V cache accesses while doing almost no compute to amortize that cost against. The flat ∼0.04–0.05s plateau across N = 1024 through 32768 — barely changing despite a 32× range in problem size — is itself evidence the cost here is dominated by a roughly fixed per-launch NUMA/scheduling overhead, not by N -dependent work. This resolves once N 21

grows large enough for genuine computation to dominate: by N = 1,048,576, 128 threads is back to a modest 1.28× slower than 64 (0.257s vs. 0.201s, not shown in the table above but visible in Figure 14’s convergence at the right edge) rather than two orders of magnitude worse. This was not visible in Paper I’s CPU thread-crossover finding (Section 3, Table 1), whose worst case was a 14.5× penalty at 128 threads — an order of magnitude smaller than what decode shows here. The likely reason: Paper I’s forward kernel does O(n2 ) work even at its smallest tested size, giving threads more to do per NUMA-domain crossing; decode’s O(n) per-token cost at small N leaves far less work to amortize the same fixed cross-NUMA cost against. Practical implication: for decode specifically, the safe default thread count is not ”use all available cores” but ”match the KV-cache length to the available parallelism” — using all 128 threads for a short cached sequence is actively harmful, by a factor that reaches into the hundreds, not just a mild inefficiency.

5.2

Cross-cluster comparison: two different penalties, two different causes

The identical binary (moa decode bench.c, unmodified) was also run on Anvil (job 20001576, ai partition), the first real comparison of the same code across both clusters used in this project. The result is not ”Anvil shows a smaller version of the same problem” — it is a structurally different phenomenon, confirmed by checking Anvil’s own topology (lscpu) rather than assumed by analogy: a single-socket 32-core AMD EPYC 7543, 4 NUMA domains, SMT disabled (Thread(s) per core: 1). The sweep’s doubling-only thread loop (a real, minor gap: it never adds an explicit final step at the true CPU count when that count isn’t a power of 2) tested up to 64 threads on this 32-core node — meaning the 64-thread point is genuine 2× oversubscription, not additional real parallelism the way Delta’s 128-threads-on-128-cores comparison was. Anvil’s pattern is consistent with ordinary oversubscription behavior: a mild penalty at small N (two logical threads context-switching on each physical core costs something when there’s little work to justify it), shrinking smoothly as N grows, then reversing into a real benefit once there’s enough work per thread for latency-hiding (one logical thread makes progress while its core-sharing partner is stalled on a memory access) to outweigh the context-switching cost. This is a textbook oversubscription curve. Delta’s pattern — a flat, severe plateau that only partially resolves, never reversing to a benefit within the tested range — is not: it is specific to spreading genuine parallel hardware across NUMA domains whose remote-access latency dominates when per-thread work is small. The two clusters’ worst-case penalties differ by roughly two orders of magnitude precisely because they are different mechanisms, not the same mechanism at different severity.

5.3

GPU: fix confirmed on real hardware

The decode kernel is the source of the Section 2.3 finding: its first real run (job 21180661, Delta gpuA100x4, NVIDIA A100-SXM4-40GB) produced technically valid but misleading numbers (flat ∼0.3–0.4s regardless of n) due to per-process GPU context-initialization cost. The fix (Section 2.3) was re-run for real (job 21264888, same partition) — the last open item for Papers I–III’s experimental validation. The fix holds. Time grows from 0.00036s at n=1024 to 0.0809s at n=1,048,576 — a real 225× increase across a 1024× range in n, not the ∼1.3× flat variation the pre-fix run showed. The local scaling exponent (computed the same way as Section 3’s GPU exponent check) starts low (0.28–0.65 for n ≤ 16384, consistent with fixed launch overhead dominating at small n, the same pattern Paper I’s GPU kernel showed before its own fix) and settles to 0.77–1.16 from n=32768 onward — hovering closely around the O(n) target of 1.0 that decode’s cost function predicts, not

Paper III decode: two clusters, two different penalties Delta: NUMA memory-locality (535x). Anvil: oversubscription (<3x, reverses)

time ratio (higher thread count / lower)

Delta: 128 vs 64 threads (128 real cores, 8 NUMA domains) Anvil: 64 vs 32 threads (32 real cores, SMT off -- 2x oversubscription) Parity

102

101

100 103

104

N (KV-cache length)

105

106

Figure 15: Time ratio at the highest tested thread count versus the next lower power of 2, both clusters, log scale. Delta’s penalty (real hardware parallelism spread across 8 NUMA domains) reaches 535×; Anvil’s (2× oversubscription on 32 real cores, SMT off) never exceeds 2.9× and reverses into a genuine benefit starting at N =16384 (1.35× faster there), growing to ∼ 1.8× faster by the largest tested size, N =1,048,576.

Paper III decode GPU, cold-start fix confirmed on real A100 Real scaling replaces the flat cold-start-dominated line Post-fix (job 21264888, real data) Pre-fix flat range (job 21180661, ~0.30-0.40s at every N) O(n) reference

wall time (s)

10 1

10 2

10 3

103

104

N (KV-cache length)

105

106

Figure 16: Post-fix decode GPU timing, real A100 data, full sweep. The shaded band marks the pre-fix run’s flat ∼0.30–0.40s range (only the endpoints and general shape are on record from job 21180661; the missing intermediate values are not fabricated here). The fixed kernel’s real time sits an order of magnitude or more below that band across nearly the entire tested range, only approaching it at the largest size.

the flat, near-zero exponent the pre-fix cold-start bug produced. This closes the one remaining gap in Papers I–III’s real-hardware validation: every kernel whose timing methodology was fixed in this document has now been confirmed, on real hardware, to show the scaling behavior its cost function predicts.

Paper IV: Transformer Block

6.1

CPU: real data, and a genuine cross-language reversal

Job 21273806 (Delta cpu partition) validates the complete transformer block’s cost function derived in [5], sweeping RMSNorm and gated MLP (both C and Fortran) up to N =65,536, and the complete block (C only) up to N =4096 — a deliberately lower cap given the block’s O(n2 ) attention component makes larger N impractically slow (its own data confirms why: 119.5s for a single N =4096, 1-thread measurement).

Paper IV: C vs Fortran runs in OPPOSITE directions on CPU vs GPU CPU: C consistently faster (ratio<1). GPU: C consistently slower (ratio>1).

C time / Fortran time

2.5

2.0 CPU, MLP kernel (threads=1): C/Fortran time ratio GPU, fused norm+MLP kernel: C/Fortran time ratio Parity (C = Fortran)

1.5

1.0

0.5 102

103

104

Figure 17: C/Fortran time ratio, CPU (MLP kernel, 1 thread) versus GPU (fused norm+MLP kernel). The two curves sit on opposite sides of parity across their entire range: C consistently faster on CPU, consistently slower on GPU, for what is denotationally the same computation. RMSNorm shows no consistent C-versus-Fortran pattern (ratio bouncing between 0.70× and 1.55× across 11 tested sizes, no trend) — consistent with this specific benchmark using a single, unaveraged measurement per point (the same methodological gap already flagged for other CPU benchmarks in this project), at a kernel lightweight enough that jitter plausibly dominates the true signal. MLP shows the opposite: a consistent, real effect. C is faster than Fortran at all 11 tested sizes, by 1.2–1.85× (0.54–0.84× the Fortran time). This is the reverse of Section 6.2’s 25

GPU result, where Fortran is faster than C by 1.45–2.81× for the same denotational computation. Both directions are real, repeated, consistent findings, not noise on either side — and reported as such rather than picking the more convenient one. The ONF is identical in both languages by construction, so whatever explains this lives in code generation, not in anything this project’s array-algebra derivation controls. A first attempt at diagnosing this, honestly inconclusive. The compiled GPU machine code (SASS) for both languages’ GPU kernel was extracted via cuobjdump and categorized by instruction type (job 21299788). If the reversal were explained by a simpler-generated kernel, Fortran’s SASS should show fewer instructions than C’s. It shows the opposite: Fortran generates more instructions in every category — 11.4% more total (4144 vs. 3720), 3.5% more FFMA/DFMA, 2.0% more global memory ops, 20.0% more shared memory ops, 13.8% more branches — while running faster in the real timing data. This rules out a naive ”fewer instructions, faster kernel” explanation, but does not replace it with a positive one: raw instruction-category counts do not capture scheduling, latency-hiding, or memory-coalescing quality, any of which could explain a kernel with more total instructions still running faster. The follow-up: a real, specific, positive finding. Warp stall-reason profiling (job 21299927, N =1024) went further, asking not what instructions exist but where cycles are actually spent. Real (unprofiled) timing at this size confirms the reversal directly (0.746s C vs. 0.692s Fortran). Profiling breaks the kernel into four real sub-kernels (norm, QKV/projection, MLP-forward, MLPbackward); one of them — the gated-MLP backward pass — dominates total time for both languages (87.7% of C’s profiled total, 65.3% of Fortran’s), making it the one that actually matters for the reversal.

C vs Fortran, per-kernel profiled time at N=1024 (job 21299927) MLP-backward dominates both totals; C is 3.17x slower there specifically

profiled time (s, log scale)

101

C Fortran

100

10 1

10 2 norm (93)

QKV/proj (99/103)

MLP-fwd (104/112)

MLP-bwd (122/142)

Figure 18: Per-kernel profiled time, C vs Fortran, N =1024. Three of four kernels are close or mixed; the gated-MLP backward pass — which dominates both totals — is where C is dramatically slower.

For that one dominant kernel specifically: C runs 3.165× slower than Fortran (24.58s vs. 7.76s, profiled), and C’s long scoreboard stall ratio (warps stalled waiting on global memory) is 4.56 versus Fortran’s 1.36 — C’s warps stall on memory latency 3.35× more often, for this exact kernel, at almost the same magnitude as the timing gap itself (3.35× stalls vs. 3.17× time). This is a real, specific, positive finding, not just a ruled-out explanation: the dominant kernel’s slowdown correlates directly with a measured increase in memory-latency stalls. The other three kernels show no such clean story (one, MLP-forward, actually runs faster in C despite higher shortscoreboard stalls there) — the dominant-kernel finding explains the aggregate reversal without being representative of every kernel in the pipeline. What this still does not explain: why C’s compiled code for this one kernel stalls on global memory more than Fortran’s. That would require inspecting the actual generated memory-access pattern for this specific kernel — not done here. The question has narrowed considerably, from ”the whole multi-kernel reversal, unknown cause” to ”one kernel, one stall mechanism, compiler code-generation difference still unidentified” — real progress on a real question, reported at the precision the evidence actually supports rather than either overclaiming a full explanation or leaving the SASS finding’s dead end as the last word.

Paper IV full-block, real Delta CPU data, 5-repeat average (job 21294300) N=2048->4096 jump (12.5x) CONFIRMED real: persists under averaging, isolated to this transition

102

1 thread best thread count at each N

wall time (s)

101 100 10 1 10 2 102

103

Figure 19: Full-block real CPU scaling, 5-repeat average, 1 thread versus each N ’s best thread count. The N =2048 → 4096 jump is confirmed real, not measurement noise (see text). The N =2048 → 4096 jump is confirmed real, not measurement noise. Section 6.1 originally flagged this as unresolved between two candidates: genuine cache-capacity effects, or single-shot measurement artifact (this benchmark had no repeated-averaging at the time). full block bench.c was revised to match moa decode bench.c’s already-correct pattern (one untimed warmup, then a 5-repeat timed average), and re-run (job 21294300). If the jump were measurement noise, averaging over 5 repeats should have smoothed it back toward the O(n2 ) target; instead it persisted, and slightly grew (11.7× → 12.5×). Every other transition in the same sweep shows smooth 2.5–4.5× growth, consistent with mixed O(n)+O(n2 ) scaling as attention in27

creasingly dominates — making N =4096 the sole, isolated outlier, not part of a general trend. This is consistent with the cache-capacity hypothesis (at N =4096 the attention score matrix alone is ∼268MB per batch element, plausibly exceeding total L3 capacity in a way no smaller N does) and inconsistent with noise, though the cache mechanism itself has not been directly confirmed by a cache-miss profiling run — what this closes is which of the two original candidates is responsible, not the full causal chain. Full-block’s thread-count behavior is far milder than Paper III’s decode finding, on the identical hardware. The worst 128-thread penalty here is 6.98× (at N =64), collapsing to near-parity (1.0–1.3×) by N =256 and beyond — nowhere close to decode’s 535× (Section 5.2) on the same Delta topology. The likely reason: full-block’s per-thread work at any tested N (QKV projection, attention, two norms, and the FFN) is substantially larger than decode’s O(n) pertoken cost at small N , giving each thread enough to do that cross-NUMA overhead is amortized rather than dominant. This is consistent with, not contrary to, the machine-as-arrays account in Section 1.1: the same fixed NUMA cost matters more or less depending on how much real work per thread the specific ONF being measured provides.

6.2

GPU: real data, clean O(n2 ) confirmation, and Fortran’s advantage

Job 21273959 (Delta gpuA100x4) compiled cleanly — all four kernel regions (fused once in C, run once in Fortran, and full block once) generated real NVIDIA GPU code with only two harmless ”variable set but never used” compiler notes, no errors. This is the first real GPU data for Paper IV, on both tested languages. Full-block’s growth confirms O(n2 ) cleanly: 4.40×, 4.10×, 3.91× across three doublings (N =64 to 512, the range FULL BLOCK N END’s default permits on GPU), converging toward the theoretical 4× from just above it — the same qualitative pattern as Paper I’s GPU forward-pass exponent converging to its target (Section 3), here observed for the first time on the complete block rather than attention alone. Fortran is faster than C for the fused norm+MLP kernel, at every tested size: 1.45× at N =256 up to 2.81× at N =64, no clear convergence across the tested range (Figure 17). Combined with Section 6.1’s opposite CPU finding, this is the clearest evidence in this document that language/compiler choice is a real, measurable, architecture-dependent cost — orthogonal to the DNF/ONF distinction Section 1.1 centers on, since both languages implement the identical ONF, verified to machine precision against the same PyTorch reference. Whether this generalizes beyond this specific kernel pair, or is specific to nvc/nvfortran’s particular code generation choices, was investigated further in Section 6.1 (SASS instruction-count comparison, then warp stall-reason profiling) — narrowed to a specific kernel and stall mechanism, though the underlying code-generation cause is still not fully identified.

6.3

Mblock : real DRAM traffic validation

The complete-block kernel was profiled with ncu (delta profile paper4.sbatch, job 21289516, B=2, N=256, D=64, DFF=256) for a direct measured-vs-predicted comparison against Mblock = 24nd + 4ndff — the same methodology Section 3’s Mfwd validation used. ncu’s raw CSV export reports byte counts pre-scaled to Mbyte/Kbyte (not raw bytes); this was caught and corrected before drawing any conclusion, the same discipline applied throughout this document’s numeric claims. Measured DRAM traffic is 1.50× the predicted Mblock (15.74MB measured vs. 10.49MB predicted, using n = N T = B × N , the convention consistently used for every other cost function in

Paper IV: Mblock real DRAM traffic validation B=2, N=256, D=64, DFF=256 -- measured is 1.50x predicted 15.74 MB

DRAM traffic (MB)

12 10

10.49 MB

8 6 4 2 0

Predicted (M_block)

Measured (ncu, real A100)

Figure 20: Predicted vs. measured total DRAM traffic (read+write, summed across all 11 kernel launches comprising one complete-block call) at the tested size. n = N T = B ×N = 512, consistent with every other cost function in this document.

this document). This is the same class of gap Paper I found for Mfwd (2.01×, Section 3) — smaller in magnitude here, but the same qualitative finding: the memory-optimal algorithm is proven, but the current code generation moves more bytes than the DNF/ONF derivation’s idealized accounting predicts. Applying Section 3’s coalescing diagnosis here: a real but partial explanation. The same smsp sass average data bytes per sector metrics used to diagnose Mfwd ’s gap are available in this profiling run too, per kernel launch. Traffic-weighted across all 11 kernels (each kernel’s efficiency weighted by its share of the 15.74MB total, not a plain average across kernels of very different sizes): reads achieve 38.9% of theoretical coalescing efficiency, writes achieve 25.8% — both real, measured, and in the same family as Mfwd ’s 29%/57% split (here reads outperform writes; there it was the reverse, a real difference between the two kernels’ access patterns worth noting rather than glossing over). Individual kernels vary widely: one (line 254, a small elementwise pass) achieves a full 100% on both, while most of the norm/projection kernels cluster around 30–40% reads and a strikingly consistent 25% writes across seven of the eleven launches. This is real evidence that coalescing inefficiency contributes to the gap, in the same direction as Paper I’s finding — but checked arithmetically, it does not cleanly reconstruct the specific 1.50× factor: the traffic that would result from perfect coalescing at these measured efficiencies is 5.82MB, which is below the 10.49MB Mblock prediction, not equal to the 15.74MB actually measured. If coalescing inefficiency alone explained the gap, ideal-coalescing traffic should reconstruct something close to the prediction; it does not. The honest conclusion is that coalescing inefficiency is a real, measured, contributing factor, not the complete explanation — some combination of Mblock ’s element-counting methodology not mapping exactly to DRAM sector-level units even under ideal coalescing, and effects specific to chaining 11 separate kernel launches (each potentially re-reading intermediate values the DNF treats as computed once), likely also contributes. Fully closing this gap is not yet done.

Scope and Limitations

This paper’s scope is deliberately narrow, and stating that scope explicitly matters for interpreting every result above correctly. This is a study of MoA-derived designs against their own predictions, not a competitive benchmark. Every kernel measured in this paper — naive and fused attention, decode, the complete transformer block, in both C and Fortran — is a design derived by Papers I–IV’s DNF/ONF/γ pipeline. This paper asks whether each design’s own formally-derived cost function correctly predicts that design’s own measured performance; it does not ask, and does not answer, how any of these designs compares in absolute terms to externally-optimized production kernels such as FlashAttention, vendor libraries, or other hand-tuned implementations (Section 1.2 situates this work relative to that literature qualitatively, but draws no performance comparison against it). A reader looking for evidence that MoA-derived kernels are competitive with the state of the art will not find it here; that is a different, legitimate question this paper does not address. Hardware coverage is two clusters, four GPU models, one CPU vendor. All measurements are on Purdue Anvil and NCSA Delta (Section 2), AMD EPYC CPUs throughout, NVIDIA A100/H100/H200 GPUs compiled via nvc/OpenACC. No Intel CPU, no AMD GPU (Delta’s gpuMI100x8 was excluded specifically because this project’s toolchain does not target it), and no cloud-vendor hardware are represented. The cross-cluster NUMA finding (Section 5.2) shows real hardware topology materially changes measured cost, which argues for testing more

topologies before generalizing broadly, not fewer. Statistical rigor varies by benchmark, and is stated at each result rather than uniform throughout. Some measurements use a 5-repeat timed average following an untimed warmup (e.g. decode, Section 5; full-block, Section 6.1 post-revision); others, flagged explicitly where it matters (the Paper II CPU N =128 outlier, Section 4.2; RMSNorm’s noisy C-versusFortran ratio, Section 6.1), are single-shot measurements on a shared cluster node, subject to scheduling jitter this paper does not quantify with formal confidence intervals. Where a finding rests on a single-shot measurement, this is stated in the text at that point; multi-point monotonic trends (e.g. the N =4096 thread-count sweep, Section 4.2) are treated as more reliable than single isolated values for exactly this reason. Code and data. The benchmark source, SLURM submission scripts, and analysis code underlying every result in this paper are being prepared for archival release; Table 4’s SLURM job identifiers are retained for the authors’ own reproducibility in the interim.

Discussion

Five things are worth stating plainly, in roughly the order the evidence for each accumulated. First, the earliest completed real-hardware result (Paper I, CPU) already produced a finding with no prior theoretical claim behind it: the thread-count-vs-n crossover in Table 1. This is exactly the kind of result that cannot come from the correctness-only verification in Papers I–IV themselves — it is a statement about scheduling overhead on a specific machine, not about the DNF — and is arguably as valuable to a systems-oriented reader as the asymptotic cost function itself. Second, two GPU timing-methodology bugs (cold-start dominance, Section 2.3; unaveraged single-shot measurement, diagnosed from the v1 run’s physically implausible falling scaling exponent) were caught only because real hardware was used. Host-fallback OpenACC compilation, used for all correctness verification in Papers I–IV, has no comparable per-process context-initialization cost or measurement-jitter profile and would never have surfaced either. Both fixes were independently confirmed against the theory they were meant to recover: the corrected kernel’s local exponent, computed from real A100 and H200 hardware (Table 2), rises to 1.97 and 2.04 at n = 8192, matching the O(n2 ) prediction almost exactly. Third, and the sharpest test of Section 1.1’s claim in this whole document: Paper II’s GPU regression (Section 4.3) and its resolution (Sections 4.4–4.5) is a complete diagnosepredict-verify cycle conducted entirely in the ρ/ψ/ι vocabulary the ONF is written in. The DNF for fused forward+backward is identical regardless of target machine; its γ-derived ONF for GPU introduced two atomic-accumulation sites (GV and GK) that the CPU’s ONF does not need in the same form. That structural difference — not measured yet, only reasoned about from the ONF itself — predicted that fused should show a specific, countable cost on GPU that it does not show on CPU. Profiling confirmed the prediction almost exactly (2.0000× backward’s atomic instructions, matching the ONF’s GV-and-GK-versus-GK-only accounting to four decimal places). Restructuring the ONF to remove one atomic site (Section 4.5) then reversed the real-hardware result completely, on both tested GPU shapes. This is the document’s clearest demonstration that γ’s output is a genuine, falsifiable claim about a real machine’s memory system — not merely that the theory and hardware agree, but that a specific, nameable feature of the ONF (which atomic sites exist, on which array) predicted a specific, measurable hardware cost before that cost was measured. Fourth, the same DNF, run through the same γ-translation, produces different real costs on different real machines, and the difference is itself explainable in machine-topology terms rather

than shrugged off as noise. Decode’s ONF run at full thread count costs 535× on Delta (128 real cores spanning 8 NUMA domains) but never more than 2.9× on Anvil (32 real cores, 2× oversubscribed, no additional real parallelism) — two different machine-array structures producing two different measured costs from the identical γ-derived schedule (Section 5.2). This is not a counterexample to Section 1.1’s claim; it is a confirmation of it in a different form. The ONF does not claim a single universal cost — it claims a cost as a function of the target machine’s own array structure (core count, memory-domain topology). Measuring genuinely different costs on genuinely different topologies, each consistent with that topology’s own structure, is exactly what the claim predicts. Fifth, not every follow-up test produces a clean win, and this document reports the one that did not as plainly as the ones that did. Removing GK’s remaining atomic (Section 4.6), the natural next step after GV ’s complete success, helped backward uniformly but made fused measurably slower at N ≥4096 — a real, size-dependent trade-off, not a second confirmation of the same mechanism. The lesson is not that the persisted-GS technique is wrong; it is that a fix confirmed for one kernel does not transfer to a structurally different kernel without separately checking it, even when the underlying technique is identical. Separately, an anomaly flagged earlier in this document as unresolved (Section 6.1’s N =2048 → 4096 jump) was run down to a specific answer using exactly the discipline this project tries to apply throughout: instead of guessing, the benchmark was fixed to match the repeated-averaging pattern already proven correct elsewhere, and re-run. The jump persisted under averaging rather than smoothing toward the theoretical value — ruling out measurement noise directly, by the same logic that would have ruled out the cache-effect hypothesis had the jump vanished instead. Together, these five points argue for treating real-hardware validation as a required step before any cost-function claim from this project is cited externally, not an optional afterthought to the formal derivation — and they argue that when validation surfaces a gap, the productive move is to look for the gap’s cause inside the ONF’s own ρ/ψ/ι structure before looking anywhere else. Most of the time that search found a specific, fixable, correctly-diagnosed cause (the cold-start fix, the measurement-averaging fix, the GV -atomic fix). Once it did not: GK’s removal is real, measured, and genuinely mixed rather than uniformly good, and reporting that honestly is as much a part of applying this discipline as reporting the wins.

Implications: active rewrites as a scaling strategy for AI systems

Everything in this section is a claim about what this document’s results argue for, distinguished explicitly from what they prove. The GV-atomic fix, the NUMA finding, the C-versus-Fortran reversal, and the recurring toolchain churn encountered while gathering this data are proven, measured facts about these specific kernels on these two specific clusters. That the same pattern generalizes usefully to AI systems broadly is a research argument this evidence supports, not a claim this document establishes on its own.

9.1

The problem this project’s own experience illustrates

Real hardware changes constantly, in ways both large and mundane. This project hit four instances of it directly, at four different scales. Largest: the same DNF produces genuinely different optimal deployment choices on Delta versus Anvil (Section 5.2) — a 535× penalty on one machine’s topology and under 3× on another’s, from identical source code. Also large, and sharper in one respect: Paper IV’s identical denotational computation (fused norm+MLP) runs faster in C than Fortran

on CPU, and faster in Fortran than C on GPU (Sections 6.1–6.2) — not just a different magnitude of advantage depending on target hardware, but a different direction entirely, from source code that implements the same ONF in two languages. Medium: the GPU regression in Paper II (Sections 4.3–4.5) was invisible in every correctness check against PyTorch and only appeared as a real cost once actual GPU hardware ran the kernel; fixing it required rewriting the ONF, not the DNF. Smallest, but recurring throughout this project’s timeline: Delta’s nvhpc module was upgraded from 25.3 to 26.5 mid-project, silently breaking every script that hardcoded the old version, and a separate DCGM profiling conflict appeared that required a cluster-specific workaround (Section 4.4) undocumented anywhere but Delta’s own user guide. None of these four were exotic; they are the ordinary texture of running real code on real, evolving infrastructure. An AI system deployed once and never revisited accumulates exactly this kind of drift silently — the cost doesn’t announce itself, it just sits there as wasted compute until someone happens to look.

9.2

What MoA’s DNF/ONF separation buys, concretely, in this project’s own evidence

The GV-atomic fix is the clearest available case study, precisely because its cost and benefit are both measured, not estimated. The DNF for fused forward+backward did not change and did not need re-verification against PyTorch — the correctness proof this project inherited from Papers I–IV remained valid throughout. Only the ONF’s realization on GPU changed: one (b, ir)-parallelized kernel became four kernels, one (b, ic)-parallelized pass eliminating an atomic site. The diagnosis that motivated this specific rewrite came directly from reading the ONF’s own ρ/ψ/ι structure (Section 4.4), not from trial-and-error search over possible restructurings. The result was a real, measured 2–2.5× reduction in wall-clock cost on both tested GPU shapes (Section 4.5) — meaning, at fixed throughput, less than half the GPU-time and energy this kernel previously required. For a kernel as heavily reused as attention’s backward pass, that per-call saving compounds directly with call volume; this document measures the per-call number, not the aggregate, but the aggregate is a multiple of it by construction. The NUMA finding makes a related but distinct point: it is evidence against treating ”optimal” as a property of the DNF alone. Table 1 and Section 5.2 both show the correct deployment choice (thread count, in these cases) is a function of the target machine’s own array structure, not a fixed constant the DNF could specify once. A single hardcoded deployment default, chosen for one machine and never revisited, is actively harmful on a different one — not marginally suboptimal, but by a factor reaching into the hundreds in Section 5.2’s Delta measurement. Treating this as a γ-translation question (what ONF does this specific machine’s array structure call for) rather than a one-time deployment decision is the difference between that cost being found and fixed versus silently paid forever.

9.3

The broader argument, stated as a research direction

Restated in the terms Section 1.1 opened with: the DNF is a claim about what a computation produces, portable across any machine; γ’s translation to an ONF is a predictive claim about what that computation costs on one particular machine’s array structure — predictive because it is stated before measurement, in ρ/ψ/ι terms precise enough to be wrong, and this document’s entire method has been checking whether it is. Across all four papers it mostly was, and where it initially was not (Paper I’s coalescing gap, Paper II’s atomic contention), the gap itself was diagnosable in the same vocabulary the prediction was stated in, not just patched by trial and error until the numbers looked better.

That is the concrete sense in which this project argues MoA is a candidate replacement for how AI kernels are typically hand- optimized today, not merely a fast implementation of one kernel family. Conventional practice re-tunes each kernel for each new architecture by profiling, guessing, and iterating without a formal predictive model to check the guess against — expensive in engineerhours, and every un-retuned kernel in the meantime silently burns extra GPU-hours and energy doing more memory traffic than it needs to. This project’s own fixes did not work that way: the GV-atomic fix was found by reading the ONF’s structure, not by sweeping restructurings until one happened to be faster, and it was confirmed, not just hoped to generalize, against real measurement on two GPU shapes. Papers I–IV’s cost functions, verified once against PyTorch and now checked against four real-machine outcomes across two clusters, are the reusable part; each machine’s ONF, actively rewritten as Section 6.1’s C-versus-Fortran reversal and Section 5.2’s NUMA finding both show is genuinely necessary, is the part that changes — and changes with a specific, checkable target each time, rather than an open-ended search. Whether this generalizes usefully beyond the kernels tested here to the wider space of AI workloads, and beyond two clusters to the full diversity of production AI hardware, remains a research direction, not a conclusion this document reaches on its own. What this document does establish is that the pattern held, without exception, on every kernel it was tried on: four papers, two clusters, CPU and GPU, real hardware throughout. That is the evidence this broader argument stands on, and the same standard — stated precisely, then checked against real measurement — is what extending it further would require.

Conclusion

We set out to answer a narrow empirical question — do the memory-optimal cost functions Papers I–IV derive formally actually predict real hardware behavior? — and answered it across four kernel families, two independent HPC clusters, and both CPU and GPU targets. The answer is yes in the specific, falsifiable sense this paper defines: where predictions held, they held to within a diagnosable, mechanistically understood margin (a factor of roughly 1.5–2×, traced in every case we investigated to a specific, nameable feature of the machine’s memory system); where they did not initially hold, as with Paper II’s GPU regression, the gap itself pointed to its own fix in the same formal vocabulary the prediction was stated in, and that fix was confirmed, not assumed. We take this as evidence for a broader claim about how AI systems might be built and maintained as hardware continues to change: that separating a hardware-independent specification from its machine-specific realization is not merely a theoretical convenience, but a practical scaling strategy, letting the expensive, error-prone part (correctness) remain fixed while the part that must legitimately change with each new machine does so with a specific, checkable target rather than an open-ended search. Not every question we asked resolved cleanly — the C-versus-Fortran reversal in Paper IV remains only partially explained — and we report that alongside the successes on the view that an honest accounting of what real-hardware validation does and does not settle is itself part of this paper’s contribution.

Acknowledgments This work used Purdue Anvil at Purdue University and NCSA Delta at the National Center for Supercomputing Applications (local Delta account: bibg-delta), both through allocation CIS261396, from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS)

program [1], which is supported by U.S. National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296.

References [1] T. J. Boerner, S. Deems, T. R. Furlani, S. L. Knuth, and J. Towns, ACCESS: Advancing Innovation: NSF’s Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support, in Practice and Experience in Advanced Research Computing (PEARC ’23), July 23–27, 2023, Portland, OR, USA. ACM, New York, NY, USA, 4 pages. [2] L. M. Mullin and G. Hains, Attention at the Theoretical Minimum I: Forward Pass, arXiv:2606.07713, 2026. [3] L. M. Mullin and G. Hains, Attention at the Theoretical Minimum II: Backward Pass and Fused Forward+Backward, HAL:05659212, 2026. [4] L. M. Mullin and G. Hains, MoA-Structured Decode Attention: DNF Derivation, KV-Cache Accumulation, GQA/MQA, and OpenACC Kernel, arXiv:2607.19456, 2026. [5] L. M. Mullin and G. Hains, Attention at the Theoretical Minimum: A Unified Treatment of the Transformer Block, submitted to IEEE Transactions on Neural Networks and Learning Systems, manuscript TNNLS-2026-P-50044, 2026; arXiv/HAL preprint in preparation. [6] L. M. Mullin, A Mathematics of Arrays, Ph.D. dissertation, Syracuse University, 1988. [7] K. E. Iverson, A Programming Language, John Wiley & Sons, 1962. [8] P. S. Abrams, An APL Machine, Technical Report SLAC-R-114, Stanford Linear Accelerator Center, 1970. [9] S. Williams, A. Waterman, and D. Patterson, Roofline: An Insightful Visual Performance Model for Multicore Architectures, Communications of the ACM, 52(4):65–76, 2009. [10] T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré, FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, in Advances in Neural Information Processing Systems 35 (NeurIPS 2022). [11] C. Lameter, NUMA (Non-Uniform Memory Access): An Overview, ACM Queue, 11(7):40, 2013.

Real-Hardware Job Identifiers

For reproducibility, Table 4 lists the status and SLURM job identifiers for every real-hardware run this paper’s claims are drawn from, on both clusters described in Section 2.

Paper

CPU status

GPU status

I (forward)

Complete (job 21181232) Complete (job 21226440; fusion 28% faster at N=4096, 128 threads)

Complete (jobs 21202354/21202355, both shapes; Mfwd profiled, job 21202356) Complete + FIXED (root cause: GV atomic, confirmed 2.00x via job 21247666; fix applied, jobs 21250395/96 confirm complete reversal – fused now 0.4-0.9x naive, i.e. FASTER, both shapes; GK atomic also removed, job 21294299: backward uniformly faster, fused mixed (faster N<4096, slower N≥4096) but still beats naive everywhere; mechanism CONFIRMED via job 21299223: extra GS re-read moves 1.97x the DRAM traffic) Complete (job 21264888; cold-start fix confirmed on real A100, exponent settles to 0.771.16 vs the O(n) target of 1.0)

II (backward/ fused)

III (decode)

IV (block)

Complete (job 21249667, Delta; job 20001576, Anvil – cross-cluster comparison confirms two different penalty mechanisms) Complete (job 21273806; MLP: C faster than Fortran 1.21.85x; fullblock NUMA penalty far milder than Paper III’s; N=4096 jump CONFIRMED real via 5-repeat re-run job 21294300, not noise)

Complete (job 21273959; O(nˆ2) confirmed; Fortran faster than C 1.45-2.81x, opposite of CPU; M block DRAM profiled, job 21289516: measured 1.50x predicted; SASS comparison job 21299788 inconclusive on instruction count; stall-reason follow-up job 21299927: dominant kernel’s C-vs-Fortran gap (3.17x) matches its memory-latency stall gap (3.35x) closely – narrowed to one kernel, code-gen cause still open)

Table 4: Real-hardware validation status and SLURM job identifiers, by paper and target.

Record · ID 1108722 · SHA-256 552d6ddda3aa1b13
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.