Plumbline: When the Loss Lies, Validating LLM Training Correctly

P LUMBLINE : W HEN THE L OSS L IES , VALIDATING LLM T RAINING C ORRECTLY

arXiv:2611.13393v1 [cs.DC] 10 Nov 2026

Ruochen Wen Hongyang Wang Siwei Li Wenbo Chen Xian Li Jingguo Liu Leo Fan Yiming Yan Qian Qu Zheshen Wang Benoit Dumoulin

A BSTRACT Correctness is a prerequisite for training large language models (LLMs) at scale, yet it is hard to ensure: a fault can sit in any part of a run, and how it shows differs across frameworks and the libraries they depend on. Some failures are loud and some are silent, and both take time to detect and trace. A single fault can waste a long run: the run crashes late, or the model scores lower only at the final evaluation. We present Plumbline, a correctness validation system for LLM training, used in production at Amazon. Plumbline checks each part of the training lifecycle, from the data a run trains on to the model it serves, down to the new layers each model brings and up to whole runs. Its diagnosis reuses these checks to trace failures to their root causes. Both rest on a systematic correctness methodology designed to carry across models and frameworks. Plumbline supports two agentic use cases: Day-0 onboarding, which validates the training of a newly released model before teams rely on it, and root-cause tracing of training failures. Across 3 frameworks and 5 models, it found 61 problems, 32 of them silent. Of these, 42 are fixed or reported upstream, 8 of them first raised by us. We further group the 61 into seven failure patterns that show where training breaks, and report how problems recur across frameworks and models. 1

I NTRODUCTION

Teams that train large language models (LLMs) commonly build on released model checkpoints and open-source training frameworks. Errors in such training can arise in the user’s code and configuration, in the framework, and in the libraries and hardware beneath it (Hu et al., 2024; Jiang et al., 2025b). These errors are costly to resolve: on one production platform, diagnosing an escalated training failure took 34.7 hours on average and more than a day in 41.9% of cases (Jiang et al., 2025b). A failure seldom points to its root cause. A fault can surface far from where it lies, or stay silent: the loss still falls while the run trains or serves the wrong model. Whether a fault shows depends on how a run combines its model, settings, framework and libraries (§2). Tracing a fault can take reruns at the same scale. These costs argue for validating training correctness before a long run, instead of discovering errors after a run fails or a trained model underperforms. A fault can sit in any part of a run and need not show in the loss, so we check each part on its own against what correct training produces. A failed check then also shows where to look, and tracing can reuse the checks at small scale instead of rerunning the job. We Amazon. Correspondence to: Ruochen Wen <ruochw@amazon.com, ruochenwen@gmail.com>. Preprint.

built Plumbline on this idea (Fig. 1). Our contributions are: • Plumbline, a correctness validation system for LLM training. Its checks cover a run at three levels: each stage of its lifecycle, from its data to the model it serves; one training step under each supported setting; and whole runs. Its diagnosis reuses the checks at smaller scale, with tools that record what a run did, to trace a failure to its root cause. • A systematic correctness methodology designed to carry across models and frameworks. It follows two principles. Under correctness first, each check tests the training a user would run against a reference not produced by the code under test, under rules fixed per system version, with bounds above measured noise. Under modularity, each check tests one part on its own, so the checks run alone or together as a full scan, and a new model or framework changes how a check connects to it, not what it tests. The same checks and rules ran on all 13 framework–model combinations. • Plumbline is used in production at Amazon in two agentic use cases: Day-0 onboarding of newly released models and root-cause tracing of training failures. Across 3 frameworks and 5 models, Day-0 onboarding found 61 problems, 32 of them silent, and tracing located the cause of each. Of these, 42 are fixed or reported upstream, 8 of them first raised by us. We group the problems into seven failure patterns that show where

Plumbline: When the Loss Lies, Validating LLM Training Correctly

Figure 1. Plumbline’s checks along one training run, grouped by where a fault sits, and its diagnosis tools. Each check is labeled with what it compares, the training it needs and where it runs. The bottom row explains the markers.

training breaks, and report how problems recur across frameworks and models.

2

M OTIVATION

Open-source training frameworks such as Megatron-LM, DeepSpeed and TorchTitan (Shoeybi et al., 2019; Rasley et al., 2020; Liang et al., 2025) run LLM training as a distributed job: they split a model and its data across many GPUs with several forms of parallelism and train it in mixed precision. The frameworks and the libraries beneath them are developed by different teams and released at different times (Huang et al., 2023). New model architectures keep adding layer types for each framework to support (DeepSeek-AI et al., 2024; MiniMax et al., 2025). A training job therefore combines code that different teams change on their own schedules. In one study of dependency bugs in deep-learning stacks, half were exposed in a different dependency from the one that introduced them (Huang et al., 2023). On a Microsoft platform, 35% of 400 sampled program failures came from differences between a developer’s environment and the platform’s (Zhang et al., 2020). Whether a fault appears therefore depends on how a run combines its parts, so a part that passes its own tests can still fail in a particular run. In a study of 1,000 fixed bugs in four deep-learning frameworks, 24.3% produced incorrect results without a crash (Chen et al., 2023). A fault of this kind shows up late, when the trained model is evaluated or served. On the Microsoft platform, failures during model evaluation were 24% of 400 sampled program failures but used 81% of their GPU time (Zhang et al., 2020).

In a study of 428 LLM training failures on another production platform, 89.9% needed manual log analysis (Jiang et al., 2025b). Many crashes appear as generic communication errors (Dong et al., 2025a; Kokolis et al., 2025), and when a failure does not reproduce locally, developers resubmit modified jobs by trial and error (Zhang et al., 2020). Tracing a failure to its cause is thus slow and largely manual. Model libraries and training frameworks implement each new model, and their tests check that implementation (NVIDIA, 2026b; ModelScope Team, 2026; Hugging Face, 2026; PyTorch, 2026). Users then train the model with their own settings, data and checkpoints, so whether a particular run is correct depends on these choices as well.

3

S YSTEM D ESIGN

3.1

Overview

Plumbline validates a run at the configuration its user trains with. A fault in any part of a run can make it train or serve the wrong model. Some faults also need a particular setting or many steps to appear. Plumbline therefore checks each part of a run against a trusted reference, under the settings and over the run lengths in which faults appear. Two principles shape the design. The first is correctness first: a system that validates training needs to be trustworthy itself. Each check therefore tests the training a user would run against a reference we can trust, under rules fixed within each system version, a frozen release of Plumbline, so that two results are judged the same way. Each bound sits above measured noise, so noise alone does not fail a check. The second is modularity: each check tests one part on its own. The checks can therefore run one at a time or together as a

Plumbline: When the Loss Lies, Validating LLM Training Correctly Table 1. What correct training means for each part of a run, the reference each check compares with and its rule. References are of five kinds: the official release’s files, the framework’s declarations and published recipe, closed-form results, an independent implementation or recomputation, and the run’s saved state. None is produced by the code under test. Check

Correct training means

Reference

Rule

the job runs the framework commit under test with the dependency versions it declares

framework declarations

exact

Collective

each collective computes its defined result on every rank

closed form

exact on every rank

Data

training sees each conversation as inference does, and the loss covers the text the model generates

official template, tokenizer

exact on every row

Build and load

the model built and loaded is the released model

official configuration, original checkpoint

exact, and unsupported settings are refused

Save and resume

a resumed run continues from its saved state

state at the save

bitwise on every rank

Export and serving

the export is the trained model in the release’s format and serves as the original checkpoint does

original checkpoint on the same engine

bitwise at learning rate 0; equal server behavior

Optimizer step

the loss is the mean cross-entropy over the supervised tokens, and each update is the one the recipe’s optimizer settings define

fp32/fp64 recomputation

exact or within rounding bounds; injected faults caught

model’s own implementation (e.g., Hugging Face code)

relative error ≤ 4× the larger noise floor, per layer group

published recipe

the run’s records show both

Lifecycle (§3.2.1) Image

Architecture (§3.2.2) Step under each supported setting, one step’s gradients and loss equal the reference’s within noise Convergence (§3.2.3) on each representative workload, a run follows its Convergence recipe and trains stably to the end

full scan of a new model, and a failed check already narrows where to look. By design, adding a new framework or model changes how the checks connect to it, not what they test or the rules they apply. The checks sit at three levels (Fig. 1), and Table 1 lists the reference and the rule of each. Lifecycle checks cover the parts that surround the forward and backward pass and determine what the model is trained on and what is served (§3.2.1). Architecture checks cover the forward and backward pass under the supported training settings, because new layer types can take different code paths from one setting to another (§3.2.2). Convergence checks cover whole runs, for faults that need the full model or many steps (§3.2.3). When a check or a run fails, a diagnosis searches for its root cause with the same checks (§3.2.4). Each check passes or fails by the rules of §3.2.5. Because the checks are independent and their rules fixed, a coding agent can compose them for each use, while a human reviewer approves its plans and confirms each problem (§3.3). 3.2

Modular Checks and Diagnosis Tools

Each check compares as exactly as correct training allows. Most lifecycle stages have one correct output and are matched exactly, one training step passes within a bound

above measured noise, and a whole run, with no reference run yet, is checked for following its recipe and training stably to the end. 3.2.1

Lifecycle Correctness

A falling loss curve shows that the model is fitting the supervised tokens it receives. The curve does not reveal whether the data pipeline produces the right tokens and labels, whether checkpointing preserves the weights and training state through loading, resuming and export, or whether each optimizer step normalizes the loss and updates the parameters correctly. These stages run on an environment: the container image and the cluster’s communication stack. A fault in the environment or in these stages can silently corrupt what the model learns or serves, or hang the job.

Environment: image versions and collective communication. A container image fixes the framework and many separately versioned libraries. A library older than the framework code expects can change results without an error. A faulty network plugin can hang a multi-node run. The image check and the collective check run in the job’s own environment: the first in its container image, the second on its nodes and network settings.

Plumbline: When the Loss Lies, Validating LLM Training Correctly

Figure 2. The step check. The same rows train one step in the framework’s training command with one setting changed and in the reference implementation (here the model’s Hugging Face code) on CPU. A layer group fails when s > 4 max(F1 , F2 ), so each failure points to one layer group, configuration and length range. Dot positions are illustrative.

Data: chat template, tokenization and loss mask. Training learns only from the tokens that the loss mask selects. New chat templates add reasoning blocks and tool turns, which the framework can format, tokenize or mask differently from the official template without an error. The data check runs every row of each representative workload (§3.2.3) through the framework’s data pipeline on CPU. A row passes when its tokens and labels equal those of a reference built from the official chat template and tokenizer.

Checkpoint conversion and loading: build, load, save and resume, export and serving. Weights are converted several times between the original checkpoint and the serving engine. A fault in a conversion can load wrong weights, resume from another state or serve a different model, and usually raises no error. The build and load checks compare the framework’s model with the official configuration and its weights with the original checkpoint. Every parameter is filled with NaN before the load, so a skipped one shows. The save-and-resume check compares each rank’s state after the resumed run’s load with its state right after the save, bitwise. The export check first trains one step at learning rate 0, so the exported weights must equal the original checkpoint’s. The serving check compares a server on the export with the same engine on the original checkpoint.

Training status: loss normalization and optimizer update. Within a step, the loss is normalized by the number of supervised tokens across data-parallel ranks and gradientaccumulation micro-batches, and each parameter group is updated with its own learning-rate and weight-decay settings. An error in either changes what is learned without stopping training. The optimizer-step check recomputes both from the framework’s logits and gradients, the loss in fp32 and the update in fp64.

3.2.2

Architecture Correctness

New layer types, such as linear or sparse attention and new expert routing, can take different kernel paths by training setting and sequence length, so a layer that is right under the recipe may be wrong under another setting a user chooses. The step check (Fig. 2) therefore sweeps configurations. Each is the recipe with one setting changed to another that the framework documents as supported for the model. A few configurations combine settings to test how they interact. Staying within documented support keeps the sweep to configurations a user could run. A setting documented as unsupported is expected to be refused. Each configuration trains one step on fixed rows from the representative workloads, grouped into length ranges, one for each code path that sequence length selects and one for short rows. To keep each configuration and its reference small, the check uses the first layers of the released model, chosen to include every layer type with its real weights. The reference is the model’s own implementation from its release, at a version with its known fixes. For the models we study, this is their Hugging Face code. The reference runs in fp32 on CPU with each document alone, so it shares no kernel, parallel layout or packing with the framework. When the framework runs the same reference code, a bug in that code is outside what this check can detect. A layer group holds the parameters of one model component, such as attention or the experts. For each group, the check computes ∥g − gref ∥2 s= ∥gref ∥2 over the same gradient elements of the configuration (g) and the fp32 reference (gref ). The group fails when s > 4 max(F1 , F2 ), 4× the larger of two noise floors: the run-to-run floor F1 , which is s between two runs of the unchanged recipe, and the precision floor F2 , which is s

Plumbline: When the Loss Lies, Validating LLM Training Correctly

between the reference’s bf16 and fp32 runs. The worst document’s per-token loss is judged the same way. 3.2.3

Convergence Correctness

Some faults need the full model or many steps to appear: effects of depth, memory pressure, the number and timing of collectives, and errors that build up between ranks. The convergence checks therefore run the full recipe. The published recipe first runs as it is on a small dataset such as GSM8K (Cobbe et al., 2021), so a problem in the recipe itself is seen before any change. A 100-step run then trains on each representative workload: general chat (Tülu 3, Lambert et al., 2025), thinking-mode reasoning (Dolci-Think-SFT, Team Olmo et al., 2025) and long-document question answering (LongReward-10k, Zhang et al., 2025). The workloads differ in rendering and sequence length, so faults that depend on the form or length of the data have a workload that exercises them. A run passes when its records show it followed its recipe and trained stably. These checks catch faults that stop, stall or destabilize a run. The comparison with an accepted earlier run of the recipe is recorded but not yet judged. 3.2.4

Diagnosis Tools

A failed run or check shows where training went wrong but often not why: a fault on one rank appears as a wait on the others. Five tools record what a run did: our run reader, PyTorch’s flight recorder, memory snapshot and profiler, and a Python stack dump. Except for the run reader, their debug settings add overhead, so they are turned on only in a separate rerun of the failed job. A diagnosis combines these records with the checks themselves, rerun at smaller scale or with one setting changed to bisect the fault, and with the code at the commit under test (§4.3). Each diagnosis ends with the cheapest check that catches the problem again, written per component type so it carries to the next model. 3.2.5

Pass/Fail Rules

A value that floating-point rounding does not change matches its reference exactly. Other values pass within a bound set from measured noise or above that rounding. Where a bound could hide a fault, the check also injects that fault into a copy of its input and passes only when it catches it. A check whose comparison cannot be completed reports an inconclusive or not-evaluated result, never a pass. 3.3

Operation with Agents

A full scan of a new model involves many checks, configurations and reruns. The design lets a coding agent operate the scan (Fig. 3). Because each check answers one question with a rule fixed per system version, the agent chooses what

Figure 3. How the agent and the human reviewer work within one system version. Tinted steps are the agent’s, steps with the person icon the human reviewer’s. A failed check goes to diagnosis, and the human reviewer decides each critical change before a rerun. Accepted checks enter the next version.

to run but not how a result is judged. A diagnosis reruns checks, so the agent can also search for a failure’s cause by the same means. Each system version fixes the procedures, the checks, the per-framework templates that launch and record each check, and the pass/fail rules. Within a version, the agent chooses the configurations and job order, runs the checks, and proposes hypotheses and changes from the records the jobs write. The human reviewer sets the task, approves the agent’s plan and decides each critical change, such as a rerun outside the plan or a fix that changes what a check validates. A second agent rebuilds each finding from the raw records, and the human reviewer confirms it. Diagnoses feed back into the design: a check that a diagnosis names and the human reviewer accepts enters the next system version, which re-evaluates the stored records. The checks thus grow with each problem found, while every result stays tied to the version that judged it.

4

U SE C ASES

4.1

Overview

At Amazon, Plumbline runs in production in two agentic use cases, one for each way modularity lets its checks run (§3.1). The two use cases cover LLM training at scale across 3 frameworks, AutoModel (NVIDIA, 2026d), MegatronSWIFT (Zhao et al., 2025) and Miles (RadixArk et al., 2026), and 5 models (Table 2). For a newly released model, a fault can sit in any part of a run, so Day-0 onboarding (§4.2) applies the full scan before teams train it. A failed or implausible result narrows the search, so root-cause tracing (§4.3) uses a few checks and diagnosis tools to find its cause. From September to October 2026, Day-0 onboarding found 61 problems at the versions we tested, counting one root cause on one framework as one problem. Root-cause tracing located the cause of each and confirmed a fix or workaround.

Plumbline: When the Loss Lies, Validating LLM Training Correctly Table 2. The models validated, each on AutoModel, MegatronSWIFT and Miles except gpt-oss-20b, on Miles only. All five are mixture-of-experts (MoE) models. Model-specific architecture lists the parts beyond a standard MoE transformer block, with multi-token prediction (MTP), and the release’s low-precision weight formats. Nodes is the validated training scale: the number of nodes, each with eight B200 GPUs, in the training configurations that onboarding recommends for the model; a range spans the model’s workloads and the frameworks’ recipes. Model

Model-specific architecture

Nodes

gpt-oss-20b (OpenAI, 2025)

MXFP4 experts

1

Qwen3.8-FlashNext (Qiu et al., 2026)

linear attention, sparse attention and its indexer, hyper-connections, n-gram table, MTP

4

DeepSeek-V4Flash (DeepSeek-AI, 2026a)

sparse attention, hyper-connections, MTP, FP8/FP4 weights

8

GLM-5.3-Flash (GLM5-Team et al., 2026)

sparse and linear attention, hyper-connections, MTP, FP8 weights

8–12

DeepSeek-V4.1Flash (DeepSeek-AI, 2026b)

sparse attention, Engram tables, hyper-connections, DSpark draft layers, FP8/FP4 weights

8–16

Teams that use these models can apply the fixes in advance, rather than after a run fails or a trained model underperforms. Of the 61, 42 are fixed or reported upstream, 8 of them first raised by us. 4.2

Use Case 1: Day-0 Onboarding

Teams want to train a newly released model as soon as a framework supports it, with their own data and settings. A new model brings new layer types, weight formats and a chat template that few training runs have yet exercised. Problems can then lie in any part of a run, appear only under some settings or workloads, or be silent: the run goes on with no error and a falling loss. Day-0 onboarding finds such problems, each with a fix or workaround, before a team commits GPUs to its full-scale training. 4.2.1

Workflow

Each task names a framework–model combination, its commit and a container image. From what the official release and the framework’s documentation state about training, the agent builds the scan: the lifecycle checks, the architecture checks under each supported setting and length range, and the convergence checks on the representative workloads (Table 1). A CPU dry run of each job, the checks that need no training and the recipe’s first run start the cycle (Fig. 4).

Figure 4. One onboarding cycle, from reading what the official release and the framework declare to the report. The numbered steps go from top to bottom, and the jobs within a step run at once. The person icon marks the human reviewer’s approval of the plan, which the step after it waits for. Step 5 hands the results that do not pass to root-cause tracing (§4.3). Checks are grouped, colored and marked by the training they need as in Fig. 1; the collective and load checks run when their trigger holds, such as a first multi-node run or a quantized release.

Once these jobs pass and the reviewer approves the plan (§3.3), the other checks run at once. 4.2.2

Results: What the Full Scan Found

The 61 problems of §4.1 fall into seven failure patterns, each named by what goes wrong in training (Table 3). Each part of the scan found some problems first, and in six of the patterns one part found most of them. The lifecycle and architecture checks, which compare each part of training with its reference, found 29 of the 32 silent problems first. At least 16 of the 32 were in runs that passed their convergence checks, so these runs did not show them. The convergence checks found 21 problems first, only one of them silent, because they catch faults that stop, stall or destabilize a run. Tokens or loss masks differ from the release. These problems arise where the data pipeline renders, masks or tokenizes a new model’s conversations differently from the official chat template and tokenizer. The difference can lie in any detail, such as a turn marker or a reasoning block. Each changes the tokens or the supervised positions. In the silent ones, the rest of each conversation is still trained, so the loss falls as usual. For example, the loss mask of GLM-5.3-Flash left out the token that ends a reply, so the model was not trained to stop. The data check found most of these problems first, before any training run. Layers break under parallelism, packing or other kernels. A layer that matches its reference implementation can still fail when its computation is partitioned across ranks, when documents are packed into one sequence, or when

Plumbline: When the Loss Lies, Validating LLM Training Correctly Table 3. The 61 problems of §4.1 by failure pattern and by the part of the full scan that found each first. A cell counts the problems of that pattern that the part found first; silent problems are in parentheses. Code reading: problems the agent saw in the framework’s code while building the scan, including one first found by the CPU dry run of a job. Failure pattern

Lifecycle

Architecture

Code reading

Convergence

Total

Tokens or loss masks differ from the release Layers break under parallelism, packing or other kernels Failures depend on sequence length Optimizer updates are lost or misapplied Resumed state differs from the checkpoint Conversion alters the checkpoint Multi-node jobs hang or fail to launch

11 (11) – – 3 (3) 5 (5) 7 (4) –

– 9 (5) – 1 (1) – – –

– 1 (1) – – – 2 (1) 1

1 4 9 (1) 3 – 1 3

12 (11) 14 (6) 9 (1) 7 (4) 5 (5) 10 (5) 4 (0)

Total

26 (23)

10 (6)

4 (2)

21 (1)

61 (32)

another kernel runs. If the layer combines partial results across ranks wrongly or lets a document read across its boundary, only some parameters or documents are affected, so the loss still falls. If the setting selects a kernel that lacks support for the layer, the run stops. For example, the querynormalization weights of DeepSeek-V4-Flash received 8× their gradient at tensor parallelism 8, because a gradient already reduced across tensor-parallel ranks was all-reduced again. The step check found most of these problems first.

Failures depend on sequence length. Parts of training behave differently at particular sequence lengths: long sequences, very short documents, or lengths that differ between batches or ranks. This behavior covers memory use, kernels and their compilation, and rank synchronization. The resulting failures stop the run, slow it, make the loss NaN, or leave parameters without gradients. For example, the sparse-attention indexer of Qwen3.8-Flash-Next scored every token against every block of the sequence in fp32, so its activations grew with the square of the sequence length. On a workload whose longest sequence has 67,877 tokens, the run ran out of memory at step 2. The convergence checks found all of them first.

Optimizer updates are lost or misapplied. These problems came from two sources: memory-saving options and a new model’s auxiliary modules. Keeping weights, gradients or optimizer state in lower precision or on the CPU saves memory but can lose small updates or fail on mixedprecision parameters. A frozen module can still be updated or contribute to the loss. Lost or wrong updates leave the loss falling, while a failure on mixed parameters or a frozen module stops the run at its start. For example, with AdamW updating bf16 weights in place at a learning rate of 5×10−6 , 92% of sampled weights that a step should change kept their value, while the loss looked normal. The build, optimizerstep and step checks found the silent ones first.

Resumed state differs from the checkpoint. A resume loads each component of the training state separately, so one component can differ from how it was saved. Here, these were progress counters, such as the learning-rate schedule, the data position and the optimizer step, and a sharded embedding table. None of these problems raises an error, and the resumed run keeps training. For example, with its state offloaded to the CPU, the optimizer of GLM-5.3-Flash did not restore its step count after a resume. The resumed run’s loss then differed from an uninterrupted run’s by up to 3.7%, while repeated uninterrupted runs differed by at most 0.02%. The save-and-resume check found all of them first. Conversion alters the checkpoint. The model is converted between checkpoint formats twice: when the original checkpoint is loaded for training, and when the trained model is exported for serving. Whatever a conversion does not map, such as a new weight format, tensor, config field or file, is dropped, misread or misnamed. The silent ones do not show during training: an export is served only after training, and a model loaded with wrong weights still trains with a falling loss. For example, loading the MXFP4 checkpoint of gpt-oss-20b transposed the down projections of all 32 experts. The matrices are square, so no shape check caught the transposition. A run from this load started at a loss of 17.08 instead of the 1.48 of a correct load. The run’s loss still fell, and the run passed the convergence checks, which judge whether a run trains stably, not its loss value. The export check found most of these problems first. Reading the conversion code found the transposed experts, and the step check confirmed them. Multi-node jobs hang or fail to launch. The remaining problems lie outside the model’s layers, in cross-node communication, compilation on every rank and the multi-node launch. Such failures show as a hang that logs no error until a timeout, often with the GPUs fully utilized, or as a launch that stops before training. For example, the first multi-node run of Qwen3.8-Flash-Next never finished step 0: every

Plumbline: When the Loss Lies, Validating LLM Training Correctly

Figure 5. How a diagnosis narrows a failure, level by level: whole run, rank and phase, component, operation or tensor, code line. Beside each step down is the tool or test that takes it: the run reader, a check at small scale, the code under test and a CPU run, and one change to confirm. A failed run enters at the top, a failed check at the component it covers. Dashed: the check named for next time feeds back into the three levels.

GPU was busy, no line was logged, and one all-to-all of the backward pass stayed pending on a single rank. The convergence checks found the hangs first, and the CPU dry run found a failed launch before any GPU job. 4.3

Use Case 2: Root-Cause Tracing

Root-cause tracing starts whenever a check does not pass, a training run fails, or a result is implausible. Each such failure narrows where to look but seldom shows the cause. Testing each hypothesis by rerunning the job at its scale is slow and costly (§2). Tracing therefore starts from what the failed job recorded and tests its hypotheses with the diagnosis tools and a few checks at smaller scale. 4.3.1

Workflow

From the records that the failed job wrote on every rank, the agent forms a hypothesis and narrows it level by level, from the whole run to a code line (Fig. 5). At each level the agent chooses the cheapest test that can reject the hypothesis, such as a check on fewer nodes or a CPU run of one operation. The failed job, rerun with one change and no debug tools, then confirms the cause or the fix. By design, each diagnosis ends with a check that catches the problem next time (§3.2.4). When no hypothesis is left that one job can test, the human reviewer decides the next step. 4.3.2

Results: From Symptom to Root Cause

Of the 61 diagnoses, 50 reached the code at the commit under test, the lowest level in Fig. 5. The descents used the failed job’s records, the diagnosis tools and a few tests at smaller scale or on CPU. The cause often lay far from the symptom: a hang from a forward-only kernel selected for short documents, and an illegal memory access from bf16

Figure 6. The 16 logged out-of-memory requests of the eight-node run’s last stage (circles) and of the step check’s length range for long rows, on one node (squares). Each is placed at the integer row length L at which the gather’s backward, (b · L) × (b · nkv ) × 512 bf16 rounded up to 2 MiB, equals it exactly. On one node, each such L is the token count of a row in the batch. On eight nodes, the largest is the workload’s longest row, 4,914 tokens. Lines: 4 rows a micro-batch (solid) and 1 row (dotted).

and fp32 gradients mixed in one fused kernel.

Case study: an out-of-memory error. In one diagnosis, a single-node check and a CPU run explained an eight-node out-of-memory error. The recipe’s first run of DeepSeekV4-Flash trained on rows of a few hundred tokens, but on a representative workload, with rows of up to 4,914 tokens, the run stopped in its first backward pass. The run reader showed every rank of the last pipeline stage had requested one allocation of hundreds of GiB, more than twice its free memory, with no log line naming the operation. An allocation that large, growing with the rows, pointed to an operation whose memory is superlinear in row length. The step check’s length range for long rows reproduced the error on a single node in about 3 minutes, also under other parallel and recomputation settings. The failure therefore depends on row length, not on pipeline parallelism or eightnode communication. Reading the code at the commit under test located the operation. The sparse attention gathers each query’s keys and values from an expanded view. For microbatches of b rows of L tokens, each row with nkv key-value entries of 512 values, the gather’s backward allocates the whole view’s gradient of (b·L)×(b·nkv )×512 bf16 values. A small CPU run confirmed this dense gradient, and with the allocator’s rounding to 2 MiB the formula matched all 16 logged requests exactly (Fig. 6). Locating the cause took one check of about half a node-hour.

Plumbline: When the Loss Lies, Validating LLM Training Correctly

4.4

Across Use Cases

Both use cases rely on the checks’ verdicts, and together they show where problems sit. Calibration of the checks. The lifecycle checks compare exactly or bitwise. The optimizer-step check compares the loss and each update with a high-precision recomputation, within rounding tolerances that an injected fault tests on every run (§3.2.5). The step check instead passes within a multiple of measured noise. We examined the step checks of 3 models on 3 frameworks. Where a run had no known defect, the unchanged recipe’s error stayed at or below 3.44× the larger noise floor max(F1 , F2 ), under the bound of 4. Where a step carried a known defect and failed, the error was at least 4.35× that floor. Problems across models and frameworks. Of the 61 problems, 28 appeared in a model-specific part that Table 2 lists, such as its attention variant or low-precision weights. Counted by where the cause lies, 27 lay in the framework’s dependencies rather than its own code, most in MegatronLM or in the bridges that implement new models and checkpoint conversion for it. These dependencies are released apart from the framework, so its pinned versions can lack support for a new model or an upstream fix. Where a cause lies decides which combinations a problem reaches. A cause in the model can recur on each framework that trains it, as when two frameworks’ exports left out the n-gram table of Qwen3.8-Flash-Next, and a cause in a framework’s shared code can recur on each of its models, as when one export wrote no generation configuration for three of them.

5

L IMITATIONS

Each check has a defined scope. A check gives a verdict where it has a trusted reference (§3.1). DeepSeek-V4.1Flash, for example, has no Hugging Face implementation at the versions we tested, so the step check leaves its layers unjudged until one is released. We consider two fallbacks, each with a narrower scope: its inference code, which covers only the forward pass, and a comparison of each setting with the framework’s unchanged recipe, which shares the code under test. Plumbline records the latter for reference, without a verdict. The step check runs one step of a model’s first layers, so faults that need the full model or many steps are left to the lifecycle and convergence checks. A human reviewer confirms each problem. Today the agent operates Plumbline, and a human reviewer approves its plans and confirms each problem before it reaches a team (§3.3). We are moving toward full automation. Each system version adds the checks and procedures that the reviewer accepted, so fewer decisions are left to the reviewer, and

onboarding more models, frameworks and use cases makes the checks and their rules more general. Our next step is for the agent to propose and test new checks itself.

6

R ELATED W ORK

Training-correctness checking. A falling loss does not show whether a training run computes what the released model defines. The training debuggers SageMaker Debugger (Rauschmayr et al., 2021), AUTOTRAINER (Zhang et al., 2021) and DeepDiagnosis (Wardat et al., 2022) watch a run’s tensors and metrics for symptoms such as vanishing or exploding gradients. The closest systems compare training with a counterpart. TrainCheck (Jiang et al., 2025a) checks a running job against invariants inferred from high-quality example pipelines. TrainVerify (Lu et al., 2025) proves a parallel execution plan equivalent to the single-device logical plan. Entangle (Wang et al., 2026) verifies that a distributed model refines its sequential model. nnTrace (Jiang et al., 2026) aligns the tensors of one distributed iteration with a trusted reference run. OpGuard (Zhou et al., 2026) localizes a production incident to the first operator that differs bit for bit from a comparable run. These counterparts come from training code: invariants of example programs, or another plan, setting or run of a framework. When a framework first supports a new model, its settings and runs share the new model code, so comparing them can miss a fault in that code. Our references for that code come from the official release.

Framework tests, checkpointing and differential testing. Megatron-LM (NVIDIA, 2026c) tests short and resumed runs of its own model configurations against reference runs, and Megatron-Bridge’s verification card (NVIDIA, 2026a) requires a one-prompt logit comparison with the Hugging Face model and a resumed run within 1% of the uninterrupted run. ByteCheckpoint (Wan et al., 2025a) and Universal Checkpointing (Lian et al., 2025) reshard checkpoints across parallel layouts and compare the loss of a resumed run with that of the original run. CRADLE (Pham et al., 2019), Muffin (Gu et al., 2022), FreeFuzz (Wei et al., 2022) and TitanFuzz (Deng et al., 2023) compare models or API calls across library backends or devices, NNSmith (Liu et al., 2023) generates valid models to test compilers, and D3 (Wang et al., 2025) requires a generated model trained under different distributed settings to match its non-distributed run. CPU-GPU comparison missed 30 of the 31 silent bugs that TransFuzz found (Zhang et al., 2026). Plumbline instead runs each framework’s training code on the released model: the step check compares one step’s gradients with the reference implementation’s under each supported setting, and the save-and-resume check compares each rank’s saved and restored states bitwise.

Plumbline: When the Loss Lies, Validating LLM Training Correctly

Training failures at scale and their diagnosis. MegaScale (Jiang et al., 2024) and ByteRobust (Wan et al., 2025b) detect, localize and recover from failures in large jobs. Mycroft (Deng et al., 2025) traces collective communication to explain stalls and slowdowns. These systems isolate the component or code update at fault during a long job. When a run fails, our diagnoses follow similar per-rank records into the code at the commit under test. SuperBench (Xiong et al., 2024) validates nodes before jobs use them, and Aegis (Dong et al., 2025b) and GUARD (Liu et al., 2026) add such checks to their runtime monitoring. Our checks test each part of the training software before such a job.

Validation and reproducibility. MLPerf Training (Mattson et al., 2020) requires closed-division submissions to be mathematically equivalent to a reference implementation. Even with fixed random seeds, identical training runs differed in accuracy by up to 2.9 points (Pham et al., 2020), and FLEX (Dutta et al., 2021) derives the bounds of ML tests from the tail of repeated runs. Our step check also sets its tolerance from measured noise. The ML Test Score (Breck et al., 2017), schema-based data validation (Polyzotis et al., 2019), model assertions (Kang et al., 2020) and ease.ml/ci (Renggli et al., 2019) test a team’s own ML pipeline against stated conditions. Plumbline states fixed conditions for the frameworks that teams build on. In concurrent work, Cross-Stack Validation (Tran & Dang, 2026) uses an independent training stack as the oracle for LoRA training of Qwen3-0.6B, and its fault with the largest effect on the trained model lay in how the training data was formatted, which our data check compares with the official chat template and tokenizer.

LLM agents for systems debugging and testing. SWEbench (Jimenez et al., 2024), KernelBench (Ouyang et al., 2025) and AIOpsLab (Chen et al., 2025) judge the patches, kernels and failure diagnoses that LLMs and agents produce against tests, reference code and injected faults fixed by the benchmark. Ekka’s agent (Gu et al., 2026) maps a serving engine’s modules to a reference implementation and aligns their activations to diagnose silent errors. TritorX’s agent (Hammond et al., 2026) writes Triton kernels for PyTorch ATen operators on Meta’s MTIA, each counted as covered when its OpInfo tests pass. In concurrent work, ForgeTrain’s coding agent (He et al., 2026) builds a training engine that must match a trusted framework bit for bit on anchor tensors before optimizing it, with no human in the loop. OpGuard Harness (OrderLab, 2026), a code release, lets a coding agent investigate two diverging runs and reports no pass/fail verdict. Plumbline’s agent chooses which checks to run under a human reviewer’s approval, and the pass/fail rules come from the system version, not the agent.

7

C ONCLUSION

A falling loss does not show that an LLM training run is correct. Of the 61 problems that Plumbline found, 32 were silent, and checks that compare one part of training with a trusted reference found 29 of these 32 first. Many problems lay in what is new in each model or in the dependencies a framework builds on, and a problem found on one combination points to others worth checking. Plumbline puts this into practice: its checks cover a run at three levels, its diagnosis reuses them to trace a failure to its root cause, and coding agents operate both in production at Amazon. We will release Plumbline as open source.

Plumbline: When the Loss Lies, Validating LLM Training Correctly

R EFERENCES Anthropic. Claude Code. Claude Code documentation, 2026. URL https://code.claude.com/docs /en/overview. Breck, E., Cai, S., Nielsen, E., Salib, M., and Sculley, D. The ML test score: A rubric for ML production readiness and technical debt reduction. In Proceedings of the IEEE International Conference on Big Data (Big Data), pp. 1123–1132. IEEE, 2017. URL https://doi.org/ 10.1109/BigData.2017.8258038. Chen, J., Liang, Y., Shen, Q., Jiang, J., and Li, S. Toward understanding deep learning framework bugs. ACM Transactions on Software Engineering and Methodology, 32(6):135:1–135:31, 2023. doi: 10.1145/3587155. URL https://doi.org/10.1145/3587155. Chen, Y., Shetty, M., Somashekar, G., Ma, M., Simmhan, Y., Mace, J., Bansal, C., Wang, R., and Rajmohan, S. AIOpsLab: A holistic framework to evaluate AI agents for enabling autonomous clouds. In Proceedings of Machine Learning and Systems (MLSys), volume 7, 2025. URL https://proceedings.mlsys.org/paper_ files/paper/2025/hash/d1f9e4a9f109b6 e8b75ed362736f22ec-Abstract-Conferenc e.html. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. URL https://arxiv.org/abs/2110.1 4168. DeepSeek-AI. DeepSeek-V4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348, 2026a. URL https://arxiv.or g/abs/2606.19348. Model release: Hugging Face deepseek-ai/DeepSeek-V4-Flash, revision 60d8d707. DeepSeek-AI. DeepSeek-V4.1-Flash: Pushing the limits of KV cache compression. arXiv preprint arXiv:2609.19969, 2026b. URL https://arxiv.org/abs/26 09.19969. Model release: Hugging Face deepseekai/DeepSeek-V4.1-Flash, revision 52544481. DeepSeek-AI, Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Guo, D., Yang, D., Chen, D., Ji, D., Li, E., Lin, F., Dai, F., Luo, F., Hao, G., Chen, G., Li, G., Zhang, H., Bao, H., Xu, H., Wang, H., Zhang, H., Ding, H., Xin, H., Gao, H., Li, H., Qu, H., Cai, J. L., Liang, J., Guo, J., Ni, J., Li, J., Wang, J., Chen, J., Chen, J., Yuan, J., Qiu, J., Li, J., Song, J., Dong, K., Hu, K., Gao, K., Guan, K., Huang, K., Yu, K., Wang, L., Zhang, L., Xu, L., Xia, L., Zhao,

L., Wang, L., Zhang, L., Li, M., Wang, M., Zhang, M., Zhang, M., Tang, M., Li, M., Tian, N., Huang, P., Wang, P., Zhang, P., Wang, Q., Zhu, Q., Chen, Q., Du, Q., Chen, R. J., Jin, R. L., Ge, R., Zhang, R., Pan, R., Wang, R., Xu, R., Zhang, R., Chen, R., Li, S. S., Lu, S., Zhou, S., Chen, S., Wu, S., Ye, S., Ye, S., Ma, S., Wang, S., Zhou, S., Yu, S., Zhou, S., Pan, S., Wang, T., Yun, T., Pei, T., Sun, T., Xiao, W. L., Zeng, W., Zhao, W., An, W., Liu, W., Liang, W., Gao, W., Yu, W., Zhang, W., Li, X. Q., Jin, X., Wang, X., Bi, X., Liu, X., Wang, X., Shen, X., Chen, X., Zhang, X., Chen, X., Nie, X., Sun, X., Wang, X., Cheng, X., Liu, X., Xie, X., Liu, X., Yu, X., Song, X., Shan, X., Zhou, X., Yang, X., Li, X., Su, X., Lin, X., Li, Y. K., Wang, Y. Q., Wei, Y. X., Zhu, Y. X., Zhang, Y., Xu, Y., Xu, Y., Huang, Y., Li, Y., Zhao, Y., Sun, Y., Li, Y., Wang, Y., Yu, Y., Zheng, Y., Zhang, Y., Shi, Y., Xiong, Y., He, Y., Tang, Y., Piao, Y., Wang, Y., Tan, Y., Ma, Y., Liu, Y., Guo, Y., Wu, Y., Ou, Y., Zhu, Y., Wang, Y., Gong, Y., Zou, Y., He, Y., Zha, Y., Xiong, Y., Ma, Y., Yan, Y., Luo, Y., You, Y., Liu, Y., Zhou, Y., Wu, Z. F., Ren, Z. Z., Ren, Z., Sha, Z., Fu, Z., Xu, Z., Huang, Z., Zhang, Z., Xie, Z., Zhang, Z., Hao, Z., Gou, Z., Ma, Z., Yan, Z., Shao, Z., Xu, Z., Wu, Z., Zhang, Z., Li, Z., Gu, Z., Zhu, Z., Liu, Z., Li, Z., Xie, Z., Song, Z., Gao, Z., and Pan, Z. DeepSeek-V3 technical report. arXiv preprint arXiv:2412.19437, 2024. URL https://arxiv.org/abs/2412.19437. Deng, Y., Xia, C. S., Peng, H., Yang, C., and Zhang, L. Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models. In Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), pp. 423–435. ACM, 2023. URL https://doi.org/10.1145/ 3597926.3598067. Deng, Y., Zhang, L., Wang, Q., Zhi, X., Zhang, X., Jiang, Z., Xu, H., Wang, L., Song, Z., Liu, G., Bai, Y., Wang, S., Xiao, W., Ye, J., Yu, M., and Xu, H. Mycroft: Tracing dependencies in collective communication towards reliable LLM training. In Proceedings of the ACM SIGOPS Symposium on Operating Systems Principles (SOSP), pp. 254–269. ACM, 2025. URL https: //doi.org/10.1145/3731569.3764848. Dong, J., Luo, B., Zhang, J., Zhang, P., Feng, F., Zhu, Y., Liu, A., Chen, Z., Shi, Y., Jiao, H., Lu, G., Guan, Y., Zhai, E., Xiao, W., Zhao, H., Yuan, M., Yang, S., Li, X., Wang, J., Men, R., Zhang, J., Zhou, C., Cai, D., Xie, Y., and Fu, B. Enhancing large-scale AI training efficiency: The C4 solution for real-time anomaly detection and communication optimization. In Proceedings of the IEEE International Symposium on High Performance Computer Architecture (HPCA), pp. 1246–1258. IEEE, 2025a. doi: 10.1109/HPCA61900.2025.00095. Dong, J., Qian, K., Zhang, P., Zheng, Z., Chen, L., Feng,

Plumbline: When the Loss Lies, Validating LLM Training Correctly

F., Xu, Y., Zhu, Y., Lu, G., Li, X., Ren, Z., Wang, Z., Luo, B., Zhang, P., Liu, Y., Chen, Y., Guan, Y., Wang, W., Yang, C., Zhang, Y., Yuan, M., Zhao, H., Li, Y., Zhao, Z., Li, S., Zeng, X., Yao, Z., Fu, B., Zhai, E., Lin, W., Wang, C., and Cai, D. Evolution of Aegis: Fault diagnosis for AI model training service in production. In Proceedings of the USENIX Symposium on Networked Systems Design and Implementation (NSDI), pp. 865–881. USENIX Association, 2025b. URL https://www.us enix.org/conference/nsdi25/presentat ion/dong. Dutta, S., Shi, A., and Misailovic, S. FLEX: Fixing flaky tests in machine learning projects by updating assertion bounds. In Proceedings of the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), pp. 603–614. ACM, 2021. URL https://doi.org/10 .1145/3468264.3468615. GLM-5-Team, Zeng, A., Lv, X., Hou, Z., Du, Z., Zheng, Q., Chen, B., Yin, D., Ge, C., Huang, C., Xie, C., Zhu, C., Yin, C., Wang, C., Pan, G., Zeng, H., Zhang, H., Wang, H., Chen, H., Zhang, J., Jiao, J., Guo, J., Wang, J., Du, J., Wu, J., Wang, K., Li, L., Fan, L., Zhong, L., Liu, M., Zhao, M., Du, P., Dong, Q., Lu, R., Shuang-Li, Cao, S., Liu, S., Jiang, T., Chen, X., Zhang, X., Huang, X., Dong, X., Xu, Y., Wei, Y., An, Y., Niu, Y., Zhu, Y., Wen, Y., Cen, Y., Bai, Y., Qiao, Z., Wang, Z., Wang, Z., Zhu, Z., Liu, Z., Li, Z., Wang, B., Wen, B., Huang, C., Cai, C., Yu, C., Li, C., Hu, C., Zhang, C., Zhang, D., Lin, D., Yang, D., Wang, D., Ai, D., Zhu, E., Yi, F., Chen, F., Wen, G., Sun, H., Zhao, H., Hu, H., Zhang, H., Liu, H., Zhang, H., Peng, H., Tai, H., Zhang, H., Liu, H., Wang, H., Yan, H., Ge, H., Liu, H., Chu, H., Zhao, J., Wang, J., Zhao, J., Ren, J., Wang, J., Zhang, J., Gui, J., Zhao, J., Li, J., An, J., Li, J., Yuan, J., Du, J., Liu, J., Zhi, J., Duan, J., Zhou, K., Wei, K., Wang, K., Luo, K., Zhang, L., Sha, L., Xu, L., Wu, L., Ding, L., Chen, L., Li, M., Lin, N., Ta, P., Zou, Q., Song, R., Yang, R., Tu, S., Yang, S., Wu, S., Zhang, S., Li, S., Li, S., Fan, S., Qin, W., Tian, W., Zhang, W., Yu, W., Liang, W., Kuang, X., Cheng, X., Li, X., Yan, X., Hu, X., Ling, X., Fan, X., Xia, X., Zhang, X., Zhang, X., Pan, X., Zou, X., Zhang, X., Liu, Y., Wu, Y., Li, Y., Wang, Y., Zhu, Y., Tan, Y., Zhou, Y., Pan, Y., Zhang, Y., Su, Y., Geng, Y., Yan, Y., Tan, Y., Bi, Y., Shen, Y., Yang, Y., Li, Y., Liu, Y., Wang, Y., Li, Y., Wu, Y., Zhang, Y., Duan, Y., Zhang, Y., Liu, Z., Jiang, Z., Yan, Z., Zhang, Z., Wei, Z., Chen, Z., Feng, Z., Yao, Z., Chai, Z., Wang, Z., Zhang, Z., Xu, B., Huang, M., Wang, H., Li, J., Dong, Y., and Tang, J. GLM-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026. URL https://arxiv.or g/abs/2602.15763. Model release: Hugging Face zai-org/GLM-5.3-Flash, revision eb9eb208.

Gu, J., Luo, X., Zhou, Y., and Wang, X. Muffin: Testing deep learning libraries via neural architecture fuzzing. In Proceedings of the IEEE/ACM International Conference on Software Engineering (ICSE), pp. 1418–1430. ACM, 2022. URL https://doi.org/10.1145/3510 003.3510092. Gu, Y., Zhang, Z., Zhu, S., Fu, X., Wu, J., Wang, Y., and Kasikci, B. Ekka: Automated diagnosis of silent errors in LLM inference. In Proceedings of the International Conference on Machine Learning (ICML), volume 306 of Proceedings of Machine Learning Research, pp. 37360– 37375. PMLR, 2026. URL https://proceedings. mlr.press/v306/gu26q.html. Hammond, A., Markosyan, A., Dontula, A., Mahns, S., Fisches, Z., Pedchenko, D., Muzumdar, K., Supper, N., Cao, S., Zhu, H., Saroufim, M., Isaacson, J., Wang, L., Hunt, W., Gondkar, K., Levenstein, R., Synnaeve, G., Li, R., Kahn, J., and Mathews, A. Agentic operator generation for ML ASICs. In Proceedings of Machine Learning and Systems (MLSys), volume 8, pp. 1583–1594, 2026. URL https://proceedings.mlsys.or g/paper_files/paper/2026/hash/8c54e9 bfed4119c873f575d1d1e2f0a0-Abstract-C onference.html. Industry Track. He, Q., Zhu, Z., Li, S., Chen, Y., Sun, H., Chen, X., Li, L., Shen, Y., Zhao, C., Fan, M., Guan, W., Zheng, Y., Zuo, Y., Li, Z., Luo, Z., Li, Y., Han, X., and Liu, Z. ForgeTrain: Forging production-grade training frameworks via harness-driven AI development. arXiv preprint arXiv:2609.13645, 2026. URL https://arxiv.or g/abs/2609.13645. Hu, Q., Ye, Z., Wang, Z., Wang, G., Zhang, M., Chen, Q., Sun, P., Lin, D., Wang, X., Luo, Y., Wen, Y., and Zhang, T. Characterization of large language model development in the datacenter. In Proceedings of the USENIX Symposium on Networked Systems Design and Implementation (NSDI), pp. 709–729. USENIX Association, 2024. URL https://www.usenix.org/conference/ns di24/presentation/hu. Huang, K., Chen, B., Wu, S., Cao, J., Ma, L., and Peng, X. Demystifying dependency bugs in deep learning stack. In Proceedings of the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), pp. 450–462. ACM, 2023. doi: 10.1145/3611643.3616325. URL ht tps://doi.org/10.1145/3611643.3616325. Hugging Face. Transformers documentation: Writing model tests. GitHub repository huggingface/transformers, docs/source/en/testing.md, commit 8fcb5239, 2026. URL https://github.com/huggingface/trans

Plumbline: When the Loss Lies, Validating LLM Training Correctly

formers/blob/8fcb52399c2d2f914e0e943 8b5341a0b99173f53/docs/source/en/tes ting.md. Jiang, H., Zhu, S., Zhang, Z., Song, Z., Fu, X., Jia, Z., Wang, Y., and Li, J. nnTrace: Detecting and localizing silent bugs in distributed training. In Advances in Neural Information Processing Systems (NeurIPS), 2026. URL https://neurips.cc/virtual /2026/poster/154832. Earlier version: TTrace, arXiv:2506.09280. Jiang, Y., Zhou, Z., Xu, B., Liu, B., Xu, R., and Huang, P. Training with confidence: Catching silent errors in deep learning training with automated proactive checks. In Proceedings of the USENIX Symposium on Operating Systems Design and Implementation (OSDI), pp. 313– 329. USENIX Association, 2025a. URL https://ww w.usenix.org/conference/osdi25/prese ntation/jiang. Jiang, Z., Lin, H., Zhong, Y., Huang, Q., Chen, Y., Zhang, Z., Peng, Y., Li, X., Xie, C., Nong, S., Jia, Y., He, S., Chen, H., Bai, Z., Hou, Q., Yan, S., Zhou, D., Sheng, Y., Jiang, Z., Xu, H., Wei, H., Zhang, Z., Nie, P., Zou, L., Zhao, S., Xiang, L., Liu, Z., Li, Z., Jia, X., Ye, J., Jin, X., and Liu, X. MegaScale: Scaling large language model training to more than 10,000 GPUs. In Proceedings of the USENIX Symposium on Networked Systems Design and Implementation (NSDI), pp. 745–760. USENIX Association, 2024. URL https://www.usenix.org/conference/ nsdi24/presentation/jiang-ziheng. Jiang, Z., Huang, J., Yu, G., Chen, Z., Li, Y., Zhong, R., Feng, C., Yang, Y., Yang, Z., and Lyu, M. L4: Diagnosing large-scale LLM training failures via automated log analysis. In Proceedings of the ACM International Conference on the Foundations of Software Engineering (FSE Companion), pp. 51–63, 2025b. doi: 10.1145/3696630.3728531. Industry Track.

Kokolis, A., Kuchnik, M., Hoffman, J., Kumar, A., Malani, P., Ma, F., DeVito, Z., Sengupta, S., Saladi, K., and Wu, C.-J. Revisiting reliability in large-scale machine learning research clusters. In Proceedings of the IEEE International Symposium on High Performance Computer Architecture (HPCA), pp. 1259–1274. IEEE, 2025. doi: 10.1109/HPCA61900.2025.00096. URL https://do i.org/10.1109/HPCA61900.2025.00096. Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H., Brahman, F., Miranda, L. J. V., Liu, A., Dziri, N., Lyu, X., Gu, Y., Malik, S., Graf, V., Hwang, J. D., Yang, J., Le Bras, R., Tafjord, O., Wilhelm, C., Soldaini, L., Smith, N. A., Wang, Y., Dasigi, P., and Hajishirzi, H. Tülu 3: Pushing frontiers in open language model post-training. In Conference on Language Modeling (COLM), 2025. URL https://openreview.net/forum?id= i1uGbfHHpH. Lian, X., Jacobs, S. A., Kurilenko, L., Tanaka, M., Bekman, S., Ruwase, O., and Zhang, M. Universal Checkpointing: A flexible and efficient distributed checkpointing system for large-scale DNN training with reconfigurable parallelism. In Proceedings of the USENIX Annual Technical Conference (USENIX ATC), pp. 1519–1534, Boston, MA, 2025. USENIX Association. ISBN 978-1-939133-48-9. URL https://www.usenix.org/conference/ atc25/presentation/lian. Liang, W., Liu, T., Wright, L., Constable, W., Gu, A., Huang, C.-C., Zhang, I., Feng, W., Huang, H., Wang, J., Purandare, S., Nadathur, G., and Idreos, S. TorchTitan: One-stop PyTorch native solution for production ready LLM pretraining. In International Conference on Learning Representations (ICLR), 2025. URL https: //openreview.net/forum?id=SFN6Wm7YBI. arXiv:2410.06511.

Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations (ICLR), pp. 54107–54157, 2024. URL https://proceedi ngs.iclr.cc/paper_files/paper/2024/h ash/edac78c3e300629acfe6cbe9ca88fb84 -Abstract-Conference.html.

Liu, G., Patni, A., Lin, C., Zeng, Z., Wittmayer, J., Wu, J., Nihalani, A., Huang, B., Liu, Y., Na, R., Ko, A., Zhipa, A., Cheng, C., Sun, M., Rajakumar, V., Joseph, R. G., and Govindarajen, P. GUARD: Scalable straggler detection and node health management for large-scale training. In Proceedings of Machine Learning and Systems (MLSys), volume 8, pp. 1595–1607, 2026. URL https://pr oceedings.mlsys.org/paper_files/pape r/2026/hash/339caf45a6fa281cae8adc64 65343464-Abstract-Conference.html.

Kang, D., Raghavan, D., Bailis, P., and Zaharia, M. Model assertions for monitoring and improving ML models. In Proceedings of Machine Learning and Systems (MLSys), volume 2, pp. 481–496, 2020. URL https://proc eedings.mlsys.org/paper_files/paper/ 2020/hash/e851ca7b43815718fbbac8afb2 246bf8-Abstract.html.

Liu, J., Lin, J., Ruffy, F., Tan, C., Li, J., Panda, A., and Zhang, L. NNSmith: Generating diverse and valid test cases for deep learning compilers. In Proceedings of the ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS), pp. 530–543. ACM, 2023. URL http s://doi.org/10.1145/3575693.3575707.

Plumbline: When the Loss Lies, Validating LLM Training Correctly

Lu, Y., Miao, Y., Tan, C., Huang, P., Zhu, Y., Zhang, X., and Yang, F. TrainVerify: Equivalence-based verification for distributed LLM training. In Proceedings of the ACM SIGOPS Symposium on Operating Systems Principles (SOSP), pp. 237–253. ACM, 2025. URL https://do i.org/10.1145/3731569.3764850.

Bridge, skills/create-model-verification-card/SKILL.md, commit e301c817, 2026a. URL https://github .com/NVIDIA-NeMo/Megatron-Bridge/blob /e301c817218e4e626c8c264aee993551455 9a04b/skills/create-model-verificatio n-card/SKILL.md.

Mattson, P., Cheng, C., Coleman, C., Diamos, G., Micikevicius, P., Patterson, D., Tang, H., Wei, G.-Y., Bailis, P., Bittorf, V., Brooks, D., Chen, D., Dutta, D., Gupta, U., Hazelwood, K., Hock, A., Huang, X., Ike, A., Jia, B., Kang, D., Kanter, D., Kumar, N., Liao, J., Ma, G., Narayanan, D., Oguntebi, T., Pekhimenko, G., Pentecost, L., Janapa Reddi, V., Robie, T., St. John, T., Tabaru, T., Wu, C.-J., Xu, L., Yamazaki, M., Young, C., and Zaharia, M. MLPerf training benchmark. In Proceedings of Machine Learning and Systems (MLSys), volume 2, pp. 336–349, 2020. URL https: //proceedings.mlsys.org/paper_files/ paper/2020/hash/411e39b117e885341f25 efb8912945f7-Abstract.html.

NVIDIA. Contribute a new model to Megatron Bridge. NVIDIA NeMo Megatron-Bridge documentation, read 2026-10-29, 2026b. URL https://docs.nvidia. com/nemo/megatron-bridge/latest/addin g-new-models.html.

MiniMax, Chen, A., Li, A., Gong, B., Jiang, B., Fei, B., Yang, B., Shan, B., Yu, C., Wang, C., Zhu, C., Xiao, C., Du, C., Zhang, C., Qiao, C., Zhang, C., Du, C., Guo, C., Chen, D., Ding, D., Sun, D., Li, D., Jiao, E., Zhou, H., Zhang, H., Ding, H., Sun, H., Feng, H., Cai, H., Zhu, H., Sun, J., Zhuang, J., Cai, J., Song, J., Zhu, J., Li, J., Tian, J., Liu, J., Xu, J., Yan, J., Liu, J., He, J., Feng, K., Yang, K., Xiao, K., Han, L., Wang, L., Yu, L., Feng, L., Li, L., Zheng, L., Du, L., Yang, L., Zeng, L., Yu, M., Tao, M., Chi, M., Zhang, M., Lin, M., Hu, N., Di, N., Gao, P., Li, P., Zhao, P., Ren, Q., Xu, Q., Li, Q., Wang, Q., Tian, R., Leng, R., Chen, S., Chen, S., Shi, S., Weng, S., Guan, S., Yu, S., Li, S., Zhu, S., Li, T., Cai, T., Liang, T., Cheng, W., Kong, W., Li, W., Chen, X., Song, X., Luo, X., Su, X., Li, X., Han, X., Hou, X., Lu, X., Zou, X., Shen, X., Gong, Y., Ma, Y., Wang, Y., Shi, Y., Zhong, Y., Duan, Y., Fu, Y., Hu, Y., Gao, Y., Fan, Y., Yang, Y., Li, Y., Hu, Y., Huang, Y., Li, Y., Xu, Y., Mao, Y., Shi, Y., Wenren, Y., Li, Z., Li, Z., Tian, Z., Zhu, Z., Fan, Z., Wu, Z., Xu, Z., Yu, Z., Lyu, Z., Jiang, Z., Gao, Z., Wu, Z., Song, Z., and Sun, Z. MiniMaxM1: Scaling test-time compute efficiently with Lightning Attention. arXiv preprint arXiv:2506.13585, 2025. URL https://arxiv.org/abs/2506.13585. ModelScope Team. Custom Megatron model (MegatronSWIFT). https://github.com/modelscope/ ms-swift/blob/680df5c670d7e8add906ac 49381d603869410612/docs/source_en/Me gatron-SWIFT/Custom-Model.md, 2026. Accessed 2026-10-30. NVIDIA. Megatron Bridge: model verification card skill. GitHub repository NVIDIA-NeMo/Megatron-

NVIDIA. Megatron-LM functional tests: golden values, checkpoint resume and optimizer-gradient match. GitHub repository NVIDIA/Megatron-LM, tests/functional_tests, commit b9d4e2b2, 2026c. URL https://github.c om/NVIDIA/Megatron-LM/tree/b9d4e2b2c 899ef7a77b482ff81d020b3d328d8ab/test s/functional_tests. NVIDIA. NeMo AutoModel: DTensor-native SPMD library for scalable and efficient training. GitHub repository NVIDIA-NeMo/Automodel, commit 7127cb7b, 2026d. URL https://github.com/NVIDIA-NeMo/Au tomodel/tree/7127cb7bef67dfd0037aa82 4e25ab59e3c9b65c0. OpenAI. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925, 2025. URL https://ar xiv.org/abs/2508.10925. Model release: Hugging Face openai/gpt-oss-20b, revision 6cee5e81. OrderLab. OpGuard Harness: Diagnostic skills and supporting agent tools for OpGuard. Code release, version 0.1.0, GitHub repository OrderLab/OpGuard, commit 146d6bb5, 2026. URL https://github.com/Ord erLab/OpGuard/tree/146d6bb523cf8364b bf2788cb319aef1a46f966b. Ouyang, A., Guo, S., Arora, S., Zhang, A. L., Hu, W., Ré, C., and Mirhoseini, A. KernelBench: Can LLMs write efficient GPU kernels? In Proceedings of the International Conference on Machine Learning (ICML), volume 267 of Proceedings of Machine Learning Research, pp. 47356–47415. PMLR, 2025. URL https://procee dings.mlr.press/v267/ouyang25a.html. Pham, H. V., Lutellier, T., Qi, W., and Tan, L. CRADLE: Cross-backend validation to detect and localize bugs in deep learning libraries. In Proceedings of the IEEE/ACM International Conference on Software Engineering (ICSE), pp. 1027–1038. IEEE, 2019. URL ht tps://doi.org/10.1109/ICSE.2019.00107. Pham, H. V., Qian, S., Wang, J., Lutellier, T., Rosenthal, J., Tan, L., Yu, Y., and Nagappan, N. Problems and opportunities in training deep learning software systems: An

Plumbline: When the Loss Lies, Validating LLM Training Correctly

analysis of variance. In Proceedings of the IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 771–783. ACM, 2020. URL https: //doi.org/10.1145/3324884.3416545. Polyzotis, N., Zinkevich, M., Roy, S., Breck, E., and Whang, S. Data validation for machine learning. In Proceedings of Machine Learning and Systems (MLSys), volume 1, pp. 334–347, 2019. URL https://proceedings.ml sys.org/paper_files/paper/2019/hash/ 928f1160e52192e3e0017fb63ab65391-Abs tract.html. PyTorch. torchtitan documentation: Debugging. GitHub repository pytorch/torchtitan, docs/debugging.md, commit 1e30d69a, 2026. URL https://github.com /pytorch/torchtitan/blob/1e30d69a3df 2c027c7a5ea65d2fd5439bf068f69/docs/d ebugging.md. Qiu, Z., Wang, Z., Li, X., Li, Y., Xu, Y., Wang, Y., Zhang, H., Men, R., Mao, B., Zhang, C., Zhou, F., Luo, H., Huang, H., Lian, H., Huang, H., Chen, H., Zhang, J., Xu, J., Wang, J., Chen, L., Wang, L., Jiang, L., Yuan, M., Sun, M., Jin, P., Zhang, S., Wang, S., Ren, X., Wang, Y., Zhang, Y., Dong, Y., Cao, Y., Ma, Y., Mao, Y., Zheng, B., and Liu, D. On the design of Qwen3.8Next architecture: Evaluation, efficiency, and training stability. arXiv preprint arXiv:2608.30320, 2026. URL https://arxiv.org/abs/2608.30320. Model release: Hugging Face Qwen/Qwen3.8-Flash-Next, revision de4b8e4d. RadixArk, Chen, T., Cheng, M., Dong, S., Du, K., Jiang, Y., Li, J., Li, Y., Lin, T., Su, Y., Ye, A., Yuan, Y., and Zeng, Z. Miles v0.1: Production-level post-training. arXiv preprint arXiv:2609.08368, 2026. URL https://arxiv.or g/abs/2609.08368. Rasley, J., Rajbhandari, S., Ruwase, O., and He, Y. DeepSpeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD), pp. 3505– 3506, 2020. doi: 10.1145/3394486.3406703. Rauschmayr, N., Kumar, V., Huilgol, R., Olgiati, A., Bhattacharjee, S., Harish, N., Kannan, V., Lele, A., Acharya, A., Nielsen, J., Ramakrishnan, L., Chandy, I., Bhatt, I., Li, Z., Chia, K., Dodda, N., Gu, J., Choi, M., Nagarajan, B., Geevarghese, J., Davydenko, D., Li, S., Huang, L., Kim, E., Hill, T., and Kenthapadi, K. Amazon SageMaker Debugger: A system for real-time insights into machine learning model training. In Proceedings of Machine Learning and Systems (MLSys), volume 3, pp. 770– 782, 2021. URL https://proceedings.mlsys.

org/paper_files/paper/2021/hash/5c7c b28f9db7a96951df3f06fd0ccfaf-Abstract. html. Renggli, C., Karlaš, B., Ding, B., Liu, F., Schawinski, K., Wu, W., and Zhang, C. Continuous integration of machine learning models with ease.ml/ci: Towards a rigorous yet practical treatment. In Proceedings of Machine Learning and Systems (MLSys), volume 1, pp. 322–333, 2019. URL https://proceedings.mlsys.org/paper_ files/paper/2019/hash/4284d31e68c0a4 a39dcdad167ac4bd72-Abstract.html. Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-LM: Training multibillion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019. Team Olmo, Ettinger, A., Bertsch, A., Kuehl, B., Graham, D., Heineman, D., Groeneveld, D., Brahman, F., Timbers, F., Ivison, H., Morrison, J., Poznanski, J., Lo, K., Soldaini, L., Jordan, M., Chen, M., Noukhovitch, M., Lambert, N., Walsh, P., Dasigi, P., Berry, R., Malik, S., Shah, S., Geng, S., Arora, S., Gupta, S., Anderson, T., Xiao, T., Murray, T., Romero, T., Graf, V., Asai, A., Bhagia, A., Wettig, A., Liu, A., Rangapur, A., Anastasiades, C., Huang, C., Schwenk, D., Trivedi, H., Magnusson, I., Lochner, J., Liu, J., Miranda, L. J. V., Sap, M., Morgan, M., Schmitz, M., Guerquin, M., Wilson, M., Huff, R., Bras, R. L., Xin, R., Shao, R., Skjonsberg, S., Shen, S. Z., Li, S. S., Wilde, T., Pyatkin, V., Merrill, W., Chang, Y., Gu, Y., Zeng, Z., Sabharwal, A., Zettlemoyer, L., Koh, P. W., Farhadi, A., Smith, N. A., and Hajishirzi, H. Olmo 3. arXiv preprint arXiv:2512.13961, 2025. URL https://arxiv.org/abs/2512.13961. Tran, T. and Dang, L. Cross-stack validation of languagemodel training: A clinical fine-tuning case study. arXiv preprint arXiv:2608.24267, 2026. URL https://ar xiv.org/abs/2608.24267. Wan, B., Han, M., Sheng, Y., Peng, Y., Lin, H., Zhang, M., Lai, Z., Yu, M., Zhang, J., Song, Z., Liu, X., and Wu, C. ByteCheckpoint: A unified checkpointing system for large foundation model development. In Proceedings of the USENIX Symposium on Networked Systems Design and Implementation (NSDI), pp. 559–578, Philadelphia, PA, 2025a. USENIX Association. ISBN 978-1-93913346-5. URL https://www.usenix.org/confe rence/nsdi25/presentation/wan-borui. Wan, B., Liu, G., Song, Z., Wang, J., Zhang, Y., Sheng, G., Wang, S., Wei, H., Wang, C., Lou, W., Yang, X., Zhang, M., Jiang, K., Ren, C., Zhi, X., Yu, M., Nan, Z., Zheng, Z., Zhong, B., Wang, Q., Yu, H., Chi, J., Zhang, W., Li, Y., Du, Z., Zhao, S., Zhang, Y., Tang, J., Liu, Z., Wu,

Plumbline: When the Loss Lies, Validating LLM Training Correctly

C., Peng, Y., Lin, H., Xiao, W., Liu, X., and Xiang, L. Robust LLM training infrastructure at ByteDance. In Proceedings of the ACM SIGOPS Symposium on Operating Systems Principles (SOSP), pp. 186–203. ACM, 2025b. URL https://doi.org/10.1145/3731569. 3764838.

Zhang, K., Xiao, D., Wu, D., Wang, S., Zhao, J., Lin, Y., Xu, T., and Wang, S. LLM-powered silent bug fuzzing in deep learning libraries via versatile and controlled bug transfer. Proceedings of the ACM on Programming Languages, 10 (OOPSLA1):1599–1626, 2026. URL https://doi. org/10.1145/3798258.

Wang, J., Pham, H. V., Li, Q., Tan, L., Guo, Y., Aziz, A., and Meijer, E. D3 : Differential testing of distributed deep learning with model generation. IEEE Transactions on Software Engineering, 51(1):38–52, 2025. URL https: //doi.org/10.1109/TSE.2024.3461657.

Zhang, R., Xiao, W., Zhang, H., Liu, Y., Lin, H., and Yang, M. An empirical study on program failures of deep learning jobs. In Proceedings of the IEEE/ACM International Conference on Software Engineering (ICSE), pp. 1159– 1170. ACM, 2020. doi: 10.1145/3377811.3380362.

Wang, Z., Ding, D., Zhu, H., Lin, H., and Panda, A. It takes two to Entangle. In Proceedings of the ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS), pp. 1022–1039. ACM, 2026. URL https://doi.or g/10.1145/3779212.3790178. Wardat, M., Cruz, B. D., Le, W., and Rajan, H. DeepDiagnosis: Automatically diagnosing faults and recommending actionable fixes in deep learning programs. In Proceedings of the IEEE/ACM International Conference on Software Engineering (ICSE), pp. 561–572. ACM, 2022. URL https://doi.org/10.1145/3510 003.3510071. Wei, A., Deng, Y., Yang, C., and Zhang, L. Free lunch for testing: Fuzzing deep-learning libraries from open source. In Proceedings of the IEEE/ACM International Conference on Software Engineering (ICSE), pp. 995– 1007. ACM, 2022. URL https://doi.org/10.1 145/3510003.3510041. Xiong, Y., Jiang, Y., Yang, Z., Qu, L., Zhao, G., Liu, S., Zhong, D., Pinzur, B., Zhang, J., Wang, Y., Jose, J., Pourreza, H., Baxter, J., Datta, K., Ram, P., Melton, L., Chau, J., Cheng, P., Xiong, Y., and Zhou, L. SuperBench: Improving cloud AI infrastructure reliability with proactive validation. In Proceedings of the USENIX Annual Technical Conference (USENIX ATC), pp. 835–850. USENIX Association, 2024. URL https: //www.usenix.org/conference/atc24/pr esentation/xiong. Zhang, J., Hou, Z., Lv, X., Cao, S., Hou, Z., Niu, Y., Hou, L., Dong, Y., Feng, L., and Li, J. LongReward: Improving long-context large language models with AI feedback. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, Volume 1: Long Papers (ACL), pp. 3718–3739. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.acl-long.187. URL https://aclanthology.org/2025.ac l-long.187/.

Zhang, X., Zhai, J., Ma, S., and Shen, C. AUTOTRAINER: An automatic DNN training problem detection and repair system. In Proceedings of the IEEE/ACM International Conference on Software Engineering (ICSE), pp. 359– 371. IEEE, 2021. URL https://doi.org/10.110 9/ICSE43902.2021.00043. Zhao, Y., Huang, J., Hu, J., Wang, X., Mao, Y., Zhang, D., Jiang, Z., Wu, Z., Ai, B., Wang, A., Zhou, W., and Chen, Y. SWIFT: A scalable lightweight infrastructure for finetuning. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 39, pp. 29733–29735, 2025. doi: 10.1609/aaai.v39i28.35383. Demonstration Track. Zhou, Z., Zhao, Y., Zhu, H., Wang, W., Bai, Z., Zhang, Y., Wang, S., Lin, H., and Huang, P. OpGuard: Bitwise alignment for precise and general debugging of production LLM training. In Proceedings of the USENIX Symposium on Operating Systems Design and Implementation (OSDI), pp. 1385–1406. USENIX Association, 2026. URL https://www.usenix.org/conference/ osdi26/presentation/zhou-ziming.

AI U SAGE D ISCLOSURE Coding agents are part of Plumbline itself (§3.3, §4). They adapt the checks’ launch and record templates to new frameworks, run jobs, choose the settings to test and trace failed runs. A human reviewer approves each plan and each critical change and confirms every problem we report. Beyond the system, we used Claude Code (Anthropic, 2026) to assist with ideation, literature search, experiment design, parts of the code (validation code, analysis scripts and figure scripts) and drafts of text, tables and figures. The authors set the research direction and the design, and decided which experiments to run. They wrote parts of the code, reviewed and tested all of it, checked every number against the run records and every reference against its source, and revised the text. The authors take full responsibility for the content. The training workloads are public datasets.

Related documents

• Fault-tolerant foundation models#1234678
• Fault-tolerant foundation models#1227356
• How Do Hardware Faults Manifest Differently in LLM Agents?#1296763
• Free to Look, Free to Reason: Simple LLM Harnesses Outdiagnose Hand-Crafted Root Cause Analysis Agents#1459290
• Free to Look, Free to Reason: Simple LLM Harnesses Outdiagnose Hand-Crafted Root Cause Analysis Agents#1450732