Auditing the Risk Claims of Distributional Reinforcement Learning Hari Prasad [email protected]
arXiv:2607.11607v1 [cs.AI] 13 Jul 2026
Abstract Distributional reinforcement learning agents learn full return distributions that are increasingly read at face value: for interpretability, risk-sensitive control, and safety monitoring. We ask a question theory anticipates but that has not been measured directly: are the risk claims of a trained distributional agent true? Our audit combines a decision-relevant screening metric (the excess Wasserstein gap between the top two actions, which equals the mass by which first-order stochastic dominance is violated), ground truth from snapshot-restart Monte Carlo, and a statistical harness (permutation nulls, bootstrap refutation, FDR control) without which the audit itself manufactures false conclusions. Across QR-DQN, C51, and IQN on MinAtar (33 runs), 40–95% of the strongest claimed risk trade-offs are refuted at 95% confidence, the placement of the strongest claims is statistically indistinguishable from truth-blind, and essentially no claim is confirmable: for these agents, the learned “risk” reflects a training artifact rather than environment stochasticity. The artifact is structural (fully formed early in training, uncorrelated with final score, idiosyncratic to each seed) and appears unchanged at full-Atari scale, with every top Breakout claim of a pretrained near-state-of-the-art QR-DQN refuted. Positive controls of known magnitude confirm 96–100% of real claims (correlation 0.89–0.92): the reading measures the agents, not the audit. Acting on the heads’ CVaR advice at their mostflagged states ranges from beneficial to significantly worse than chance. Neither training for risk nor ensembling removes the artifact, and recalibration passes the audit only by nullifying the claims: the head is uninformative, not merely miscalibrated. We release the toolkit and document two silent pitfalls that produced convincing but wrong audits of our own.
Introduction Distributional reinforcement learning (RL) replaces the scalar value function with a full distribution over returns (Bellemare, Dabney, and Munos 2017; Dabney et al. 2018b,a). The idea has been unusually successful: distributional heads drove state-of-the-art results on the Arcade Learning Environment (Bellemare, Dabney, and Munos 2017) and anchor a growing textbook theory (Bellemare, Dabney, and Rowland 2023). Along the way, the learned distributions stopped being treated as an internal implementation detail: they are visualized to interpret agent behavior (Greydanus et al. 2018), thresholded to produce risk-
sensitive policies via conditional value-at-risk (CVaR) and related functionals (Keramati et al. 2020; Lim and Malik 2022), and monitored as safety signals from autonomous driving (Hoel, Wolff, and Laine 2021) to safe-RL pipelines broadly (Garcı́a and Fernández 2015). All of these uses share one assumption: that where the learned distribution of one action differs in shape from another (promising, say, a safe small return versus a gamble of equal mean), the environment actually contains that risk structure. Theory gives reasons for doubt. Quantile temporaldifference (QTD) learning converges to fixed points that need not coincide with the true return distribution (Rowland et al. 2023a), and the benefits of distributional methods appear even when only the mean is used (Rowland et al. 2023b; Lyle, Bellemare, and Castro 2019), suggesting the distributions can help without being correct. The uncertaintyestimation literature has long suspected the same flaw architecturally: the spread of a quantile head conflates aleatoric and epistemic uncertainty (Clements et al. 2019), motivating a series of ensemble and evidential repairs (Hoel, Wolff, and Laine 2021; Eriksson et al. 2022; Stutts et al. 2024). What is missing from both threads is a measurement: how wrong are the distributions of a trained deep agent, at which states, and does the error land on the claims a user would act on? A distribution can be wrong in ways nobody acts on, or wrong exactly where a risk-sensitive user consumes it; aggregate divergence metrics do not separate the two, a decision-level audit does. This paper provides that measurement: a diagnosis that must precede any cure, and a benchmark against which future calibrated methods can be held. Our contributions: 1. A decision-relevant audit metric. We screen states by the excess Wasserstein gap ∆x (s) between the agent’s top two actions: the W1 distance between their return distributions minus their mean gap. We note (Prop. 1) that ∆x (s) > 0 exactly when neither action first-order stochastically dominates the other, i.e. precisely when the agent claims a genuine risk trade-off where a risk-sensitive policy would deviate from a mean-greedy one. 2. Ground truth with a statistical harness. We restart the environment from state snapshots, roll out thousands of futures per action, and test each claim against a permutation null and a bootstrap refutation test, with FDR control (Benjamini and Hochberg 1995) and a per-state aliasing index
bounding partial observability. 3. The finding. Across QR-DQN on three MinAtar games and five seeds: 66–84% of the strongest claimed trade-offs are provably false; claim placement is statistically indistinguishable from truth-blind (learned–true correlation ≈0); essentially no trade-off among 735 states is confirmable; and the most extreme failures occur at doomed states whose every future yields an identical return, over which the head reports wide, crossing distributions. 4. A decision-level evaluation. Acting on the head’s CVaR ranking ranges from genuinely informative (Breakout) to significantly worse than chance (Seaquest), with no observable signal separating the regimes; C51’s advice loses to a mean-greedy baseline on all three games. 5. Replication, structure, repairs, and two pitfalls. The audit replicates on C51 and IQN (all three projection families) and on a pretrained 10M-frame SB3-zoo QR-DQN whose top Breakout claims are all refuted, so the finding is not an artifact of homegrown or weak agents: it is fully formed by 500k steps, persists while score doubles, and is seed-idiosyncratic. Nor is it repaired by the obvious fixes: a CVaR-greedy agent overclaims just as much, ensembling attenuates but does not calibrate, and recalibration passes the audit only by nullifying the claims, showing the head is uninformative rather than miscalibrated. Positive controls with trade-offs of known magnitude confirm 96–100% of real claims (correlation 0.89–0.92), so effect size does not explain the MinAtar zero. Finally we document two silent failure modes of restart-based evaluation (RNG cloning and the empirical-Wasserstein noise floor), each of which produced a convincing but wrong audit before the final design.
Related Work
Distributional RL. C51 (Bellemare, Dabney, and Munos 2017), QR-DQN (Dabney et al. 2018b), and IQN (Dabney et al. 2018a) established the deep family (Bellemare, Dabney, and Rowland 2023); our audit targets QR-DQN and replicates on C51 and IQN. Several results show these methods help without requiring distributional accuracy (representation-learning effects, Lyle, Bellemare, and Castro 2019; quantile efficiency for mean estimation, Rowland et al. 2023b), and QTD fixed points are non-unique and need not equal the true return distribution (Rowland et al. 2018, 2023a). We provide the empirical, decision-level counterpart of these warnings for deep agents. Uncertainty (mis)estimation. Clements et al. (2019) showed quantile spread mixes epistemic and aleatoric components; successors propose ensembles (Hoel, Wolff, and Laine 2021; Eriksson et al. 2022; Charpentier et al. 2022), projection-ensemble diversity (Zanger, Böhmer, and Spaan 2024), evidential calibration (Stutts et al. 2024), and conformal wrappers (Gan et al. 2025). These presume the raw distributions are unreliable and build estimators around them; none measures the per-state truth of a trained agent’s claims, which our audit supplies. Evaluation methodology. Distributional OPE with error guarantees (Wu, Uehara, and Sun 2023) addresses estimation under known conditions; we are not aware of prior state-level, statistically controlled audits of deep agents. Our
harness follows the push for rigor in deep-RL evaluation (Machado et al. 2018; Agarwal et al. 2021) and parallels Atrey, Clary, and Jensen (2020), who found saliency explanations of Atari agents do not survive counterfactual testing; the empirical-Wasserstein noise floor behind our second pitfall is classical (Fournier and Guillin 2015).
Preliminaries and the Audit Metric
P t Distributional RL. Let Z π (s, a) = t γ Rt be the random discounted return from action a in state s under π, with Qπ (s, a) = E Z π (s, a). QR-DQN (Dabney et al. 2018b) represents Z(s, a) by N quantile atoms at fixed levels; C51 (Bellemare, Dabney, and Munos 2017) fixes N atom locations and learns their probability mass after a categorical projection. Both train from the same bootstrapped targets and differ only in how the distribution is parameterized, a difference our C51 replication exploits. A risk-sensitive user consumes Z through a functional such as CVaRα (Z) = E[Z | Z ≤ FZ−1 (α)] (Artzner et al. 1999; Rockafellar and Uryasev 2000), which deviates from mean-greedy behavior only where the actions’ distributions differ in shape, not location. The excess Wasserstein gap. For the two actions a1 , a2 ranked top-2 by mean at s, the 1-Wasserstein distance P between their quantile representations is W1 = 1 i |θi (s, a1 ) − θi (s, a2 )|, computable in closed form. N Writing µj (s) = EZ(s, aj ), define ∆x (s) = W1 Z(s, a1 ), Z(s, a2 ) − µ1 (s) − µ2 (s) ≥ 0.
Proposition R1. For real-valued R distributions with CDFs F, G: ∆x = |F − G| dx − (F − G) dx = 0 iff F − G does not change sign, i.e. iff one distribution first-order stochastically dominates the other; otherwise ∆x equals twice the mass of the smaller-signed region of F − G, the magnitude of the dominance violation. R The proof (a one-line consequence of W1 (F, G) = |F − R G| and EG − EF = (F − G); Villani 2009) is in the supplement. ∆x (s) > 0 is exactly the agent’s claim that the two actions trade off risk: neither is safer at every quantile, so risksensitive and mean-greedy policies can disagree at s; when ∆x (s) = 0, every monotone risk functional ranks the actions identically. As a screening statistic, ∆x isolates the claims that carry decision content while ignoring errors both actions share: the audit is of relative risk claims, the kind acted upon. A note on what the audit targets (expanded in the supplement): the head’s distribution as consumed by downstream users, not as an intermediate Bellman quantity. Whether or not Ẑ is meant to recover Z π exactly, the interpretability, CVaR, and monitoring uses of §1 read it as if it does, and that reading is what we test.
Audit Methodology We summarize the method here; the supplement gives full detail. Screening and stratified sampling. We roll out the trained agent (ϵ-greedy, ϵ = 0.05) to collect 2,000 unique
QR-DQN
Breakout
Seaquest
Asterix
Refuted, top 2% of ranking 84% [74,91] 84% [74,91] 66% [54,76] Refuted, rest of ranking 20% [15,27] 15% [10,21] 5% [2,9] Refuted, top 2%, low-alias 80% [66,90] 100%† 91% [62,98] Confirmed (FDR 0.1), pooled 0/245 0/245 0/245 corr(learned, true excess) 0.02 −0.01 −0.04
Figure 1: Pitfall P2, measured. Empirical excess between two n-samples from the same return distribution (a highvariance audited Breakout state, σ ≈ 2.3), 300 replications per n. The floor decays as n−1/2 (Fournier and Guillin 2015); at the pilot n = 200 its 95th percentile (≈ 0.22) dwarfs a fixed 0.05 threshold, so naive thresholding “confirms” pure noise. All tests in the audit are therefore calibrated against per-state nulls at n = 2,000. states, score each by ∆x from the learned head, and rank them. Because overclaiming might concentrate anywhere, we audit seven states from each of seven rank strata (top 0.5%, 0.5–2%, 2–5%, 5–10%, 10–25%, 25–50%, 50– 100%), 49 per run. “Top strata” denotes the top 2% of the ranking: the states a practitioner consulting the head would flag first. Ground truth by snapshot restart. We store deep copies of the environment at up to 16 visits of each state, and for each audited state and top-2 action run 2,000 rollouts: restart from a stored snapshot, force the action, then follow the greedy policy to termination or horizon 103 . Each restarted copy is independently reseeded (pitfall P1). Environment stochasticity (sticky actions, random spawns, difficulty ramping) is the only noise source, matching the aleatoric semantics of Z π under the evaluation policy; the MC empirical MC excess ∆ˆx (s) uses the same quantile formula. MC Statistical harness. A claim is confirmed if ∆ˆx (s) exceeds a permutation null (excess between split halves of one action’s rollouts, pure estimation noise by construction), and refuted if the learned ∆x (s) exceeds the 97.5th bootstrap percentile of the MC excess; the estimator’s finitesample bias (P2) inflates that percentile, so refutation is conservative. Both use Benjamini–Hochberg (Benjamini and Hochberg 1995) at FDR 0.10, pooled per game. The median null is 0.07, below the typical top-stratum claim, so real trade-offs above ≈0.07 are detectable. A per-state aliasing index (dispersion of per-snapshot mean returns over total spread) flags states where the observation may hide environment state; low-alias results (< 0.25) are reported separately.
Two Pitfalls That Produce Convincing, Wrong Audits P1: RNG cloning. Restart-based evaluation typically snapshots environments with deepcopy, which clones the environment’s internal random generator. Every “random” roll-
Table 1: Main results over 5 seeds per game (245 audited states per game; Wilson 95% intervals). † single low-alias state in stratum. corr rows are per-seed means. out then replays one identical future. Our first audit, built this way, concluded all claimed risk was artifact; the tell was impossible determinism in a game with random spawns. The failure is silent, directionally plausible, and survives code review easily; it may affect other restart-based evaluations. P2: the empirical Wasserstein noise floor. For two nsamples from the same distribution with spread σ, the empirical ∆x concentrates around c σn−1/2 > 0 rather than zero (Fournier and Guillin 2015). Figure 1 measures this floor on real audit data: at our pilot n = 200 the mean floor is ≈0.07 and its 95th percentile ≈0.22, exceeding most true effect sizes; a fixed small threshold “confirmed” two-thirds of claims in our second audit, all noise. Validation of distributional claims must be calibrated against a per-state null at the deployed sample size; hence the permutation and bootstrap constructions above.
Experiments Setup. We train QR-DQN (51 quantiles, 3M steps, five seeds), C51, and IQN (three seeds) on MinAtar (Young and Tian 2019) Breakout, Seaquest, and Asterix (shared recipe in the supplement; MinAtar defaults incl. sticky actions p=0.1); Figure 2 shows the curves. We additionally audit intermediate Breakout checkpoints (0.5M/1M/2M steps) and the pretrained SB3-zoo QR-DQN agents (10M ALE frames, 200 quantiles) on full-scale ALE Breakout and Seaquest (n=1,000 rollouts, sticky evaluation p=0.25). With the positive controls below, this totals 2,450 audited states and 9.6M rollouts. C51 claims are quantized to its atom grid (bin width 0.8), restricting cross-algorithm comparison to claims above it. The full sweep cost under $9 of commodity CPU time; all code, checkpoints, and raw returns are released.
The Strongest Claims Are Mostly False In the top 2% of the agent’s own ∆x ranking, 66–84% of claimed risk trade-offs are refuted at 95% confidence (Table 1): the learned excess exceeds anything the ground-truth distributions could plausibly produce, even with the estimator’s own bias favoring the agent. The rate is not explained by aliasing: among low-alias states, where the observation approximately pins down the outcome distribution, the refutation rate is equal or higher. Refutation falls monotonically down the ranking (Fig. 3), from ≈94% in the top halfpercent of Breakout and Seaquest to near zero below the top quartile. This gradient must be read with care: small claims are unfalsifiable by construction (refutation requires the claim to
Figure 2: Training curves for all 33 MinAtar agents (rolling mean over 200 episodes). Final scores (mean of last 100 episodes, averaged over seeds): QR-DQN 14.6 / 25.2 / 22.1, C51 15.8 / 9.2 / 10.0, IQN 13.4 / 17.7 / 22.3 on Breakout / Seaquest / Asterix, consistent with published MinAtar baselines (Young and Tian 2019). C51 underperforms QR-DQN on two games; the audit is within-agent (each head is tested against its own claims), so audit conclusions do not depend on cross-algorithm score parity. pooling. Under greedy play, MinAtar’s aleatoric noise does not produce action-conditional dominance violations of the size the heads routinely claim: they are not exaggerating real trade-offs but reporting ones that do not exist.
The Audit Passes Its Positive Control
Figure 4 plots learned against true excess for all audited states: pooled correlation is 0.02 / 0.05 / −0.03 across the three games (per-seed means 0.02 / −0.01 / −0.04, no seed significantly positive). The ranking a practitioner would use to find “the states where risk matters” is precisely a map of where the representation is least trustworthy.
A zero-confirmation result invites the objection that the audit, not the agents, is broken. We therefore built an environment with a known answer key. RISKYGRID is a tenstep corridor: each episode draws per-column gamble scales g(x) ∈ {0, 0.5, . . . , 2.0}, rendered into the observation; at each step the agent chooses SAFE (reward 0.5, deterministic) or GAMBLE (reward 0.5 ± g(x), fair coin). Means are identical, so every g(x) > 0 column is a pure risk trade-off of known magnitude, g(x) = 0 columns have none, and the observation determines the full g-map (zero aliasing by construction). We train QR-DQN (three seeds, 500k) and run the identical audit. On this control the audit’s verdicts invert (Fig. 5): 143/147 = 97% [93, 99] pooled FDR-corrected confirmations (the misses are g=0 states); zero false refutations in the top strata (0/42, with 4/105 below, consistent with the FDR budget); learned–true correlation 0.92 against |r| ≤ 0.11 everywhere on MinAtar; and the head’s CVaR choice picks the truly safer action in 99–100% of 143 decidable states with realized regret ≈0. A natural objection remains: RISKYGRID’s trade-offs are large, MinAtar’s (if any) would be faint. A faint variant scales the gambles down to g ∈ {0, 0.05, 0.1, 0.2, 0.4}, straddling both the audit’s detection floor (median null95 ≈0.07) and the scale of MinAtar’s artifact spread. The audit confirms 96–100% of g > 0 states at every level including g = 0.05 (Fig. 5, center), leaves g = 0 states at the FDR budget (1/10), and the ranking still tracks truth at pooled r = 0.89. The same architecture and audit that report spurious risk on MinAtar report near-perfect risk here at every signal size the audit can detect: the MinAtar reading measures the agents, not the audit’s power or the effect size.
Essentially No Claim Is Confirmable
Specimens: What Overclaiming Looks Like
With power to detect true trade-offs of magnitude ≳0.07 (the median per-state null), pooled FDR-corrected confirmations number zero in all three games. Uncorrected per-state detections at p < 0.05 hover at the false-positive rate (≈ 5%): 9 raw per-seed detections among 735 QR-DQN states, 7 among 441 C51 states, none surviving Benjamini–Hochberg
The clearest specimens (Fig. 6) are Breakout states in which the ball is irrecoverably lost: every one of 2,000 futures returns exactly the same value (bootstrap 97.5th percentile of MC excess 0.00), yet the head reports wide, crossing distributions with learned excess up to 0.44; pooled across seeds, 14/70 = 20% of top-stratum Breakout states are this
Figure 3: Refutation rate by rank stratum (QR-DQN, pooled over 5 seeds, Wilson 95% intervals, FDR 0.1). The agent’s most confident risk claims (left) are the least true; below the top decile of the ranking, refutations largely vanish; the claims there are too small to be provably wrong. A truthblind head with the same claim magnitudes reproduces this profile (see text). exceed the MC bootstrap percentile), so a declining curve is expected from test asymmetry alone. A truth-blind null head makes the comparison exact: permuting each run’s learned claims across its audited states (magnitudes and rank strata preserved, placement randomized with respect to ground truth) and re-running the identical pipeline yields top-strata refutation of 81% ([75, 86] over 1,000 permutations), against 79% observed (165/210, p = 0.22). The heads place their strongest claims no better than truth-blind chance; the positive evidence of miscalibration is the confirmation and correlation analyses below.
Confidence Anti-Correlates with Truth
Figure 4: Learned excess vs. ground-truth (MC) excess for all 735 QR-DQN audited states (color = rank stratum, 0 = top of ranking; dashed = identity; r = pooled Pearson correlation). If the heads were even noisily calibrated, the top-stratum (dark) points would hug the diagonal; instead they lie flat on the floor.
Figure 5: Positive controls. Left: on RISKYGRID, where trade-offs exist by construction, the learned ranking tracks ground truth (pooled r = 0.92; per-seed 0.96/0.92/0.89; the head compresses magnitudes but orders them correctly); compare the flat clouds of Fig. 4. Center: the faint variant, trade-offs of scale g ∈ {0, 0.05, . . . , 0.4} straddling the audit’s detection floor: 96–100% of g > 0 states confirmed, g=0 states at the FDR budget. Right: FDR-corrected confirmation rate, controls vs. the three MinAtar games (QR-DQN, Wilson 95% intervals; the MinAtar bars are all at zero). safer pick (%) CVaR regret head
mean
head mean
QR-DQN Breakout (n=27) Seaquest (n=35) (α=0.1) Asterix (n=27)
81 34 52
44 63 48
0.20 0.65 0.20
1.36 0.30 0.26
Breakout (n=17) Seaquest (n=18) Asterix (n=14)
65 28 36
71 56 57
0.45 0.67 0.48
0.10 0.20 0.24
C51 (α=0.1)
Table 2: Decision-level evaluation at the heads’ own mostflagged (top strata) decidable states: % of states where each chooser picks the truly safer action, and mean realized CVaR regret vs. the ground-truth-optimal choice. Bold marks a clear winner. The mean-greedy column ignores risk entirely. IQN rows and α=0.25 (quoted in the text) show the same pattern; per-state data released. doomed type. The Seaquest and Asterix specimens show the modal failure: the two actions’ true distributions are indistinguishable while the head claims a decision-relevant asymmetry that exists in the network, not the environment.
Does Acting on the Heads Matter? Using the same rollouts, we compare three choosers of the safer action at level α on decidable states (true CVaR gap significant at 95% by bootstrap): the head’s CVaR ranking, the mean-greedy baseline, and ground truth (Table 2).
The outcome is strongly environment-dependent. In Breakout the QR-DQN head is genuinely informative (top strata, α=0.1: safer action chosen 81% vs. 44% for mean-greedy; regret 0.20 vs. 1.36). In Seaquest it is anti-predictive: at the states it flags as most risk-critical, its CVaR choice is truly safer only 24–34% of the time, below chance (onesided binomial p = 8 × 10−4 at α=0.25; p = 0.045 at α=0.1), roughly tripling regret relative to ignoring risk entirely. In Asterix it is a coin flip. A practitioner cannot tell which regime obtains; advice that is sometimes helpful and sometimes inverted is harder to use than uniformly uninformative. C51 fares worse, beaten by mean-greedy in all three games (28–68% vs. 48–71%; 2–4.5× the regret); IQN reproduces the QR-DQN regimes (helpful in Breakout, 79% vs. 42%; chance in Seaquest; inverted in Asterix, 47% vs. 73%). Across all three algorithms, no game-independent rule tells a practitioner when the head’s risk advice can be trusted.
C51, IQN, and a Pretrained Atari Agent Replicate the Pattern Repeating the full audit for C51 and IQN (Table 3, Fig. 7) yields the same signature under two more projections: 40– 95% of top-strata claims refuted against a 5–16% background, zero pooled confirmations among 882 states, and no consistent positive correlation, in all 18 runs. IQN overclaims somewhat less but its top-strata refutation still exceeds its background several-fold. Since C51’s categorical projection shares none of QTD’s fixed-point pathologies
Figure 7: Refutation rate by rank stratum for C51 (left) and IQN (right), pooled over 3 seeds (Wilson 95% intervals, FDR 0.1): the same monotone decline from the top of the ranking as Fig. 3.
Figure 6: Audited specimens: environment snapshot (left) and both actions’ return CDFs, learned (solid) vs. 2,000rollout MC ground truth (dashed). Top: a doomed Breakout state; every future yields the same return (dashed step), yet the head reports crossing distributions with excess 0.44. Middle: top-stratum Seaquest and Asterix states whose true distributions coincide while the learned ones cross. Bottom: the same signature in C51. Breakout
Seaquest
Asterix
Refuted, top 2% 81% [67,90] 95% [84,99] 57% [42,71] Refuted, rest 10% [5,17] 16% [10,24] 5% [2,11] C51 Confirmed, pooled 0/147 0/147 0/147 corr(learned, true) 0.02 0.05 −0.11 Refuted, top 2% 64% [49,77] 79% [64,88] 40% [27,56] Refuted, rest 7% [3,13] 7% [3,13] 8% [4,14] IQN Confirmed, pooled 0/147 0/147 0/147 corr(learned, true) 0.08 −0.11 −0.08 Refuted, top 2% 100% [78,100] 14% [4,40] ALE Refuted, rest 0% [0,10] 3% [1,15] (zoo) Confirmed, pooled 0/49 0/49 corr(learned, true) −0.22 −0.25
— — — —
Table 3: C51 and IQN audits, 3 seeds per game (147 audited states per game per algorithm; Wilson 95% intervals; FDR 0.1; corr rows are per-seed means), and full-Atari audits of the single pretrained SB3-zoo QR-DQN per game (n=1,000 rollouts, sticky-action evaluation). The QR-DQN signature of Table 1 replicates under a categorical projection, an implicit-quantile projection, and a 10M-frame professionally trained agent on the full ALE.
(Rowland et al. 2018, 2023a) and IQN replaces the fixed tau grid entirely, the overclaiming implicates the common ingredient: bootstrapped distributional targets from a changing policy under function approximation, not the representation.
Figure 8: Full audits of intermediate checkpoints (Breakout QR-DQN, 3 seeds pooled, 42 top-strata states per point). Top-strata refutation is 81/88/86/83% at 0.5/1/2/3M steps while the training score climbs from 8.6 to 14.4; confirmations are zero at every stage. The same signature appears at full-Atari scale, with a caveat: the zoo agents were trained without sticky actions, so their claimed spread cannot reflect environment stochasticity by construction; this audit quantifies the artifact’s size and placement, not an independent replication (Table 3, bottom). The pretrained SB3-zoo QR-DQN for Breakout (raw score ≈400, matching its published performance) has all 14 topstrata claims refuted, zero elsewhere (the cleanest winner’scurse profile in the study), zero confirmations among 49 states, and negative correlation: mean top-strata claim 0.184 against mean ground-truth excess 0.000. The zoo Seaquest head sits at the null floor (mean 0.021 vs. 0.03), so only 2/14 are refutable, but nothing is confirmable and correlation is again negative. Across MinAtar, 32 of 33 runs refute at least 6/14 top-strata claims against a 5–20% background (per-run counts released).
Structural, Not a Symptom of Weak Agents Three probes tie the artifact to the learning process, not agent quality. Training time: audits of intermediate checkpoints (Fig. 8) show the overclaiming fully formed by 500k steps and unchanged to 3M while score nearly doubles; the top2% claimed magnitude grows over training in 15/15 runs. Performance: across all 33 runs, score and top-strata refutation are uncorrelated (Spearman −0.07): better agents do not claim better. Seeds: under the game’s four sibling seeds, siblings reproduce a median 3.8% of a top-stratum claim’s magnitude, and the cross-seed median stays uncorrelated with truth (|r| ≤ 0.07): each seed produces its own spurious claims. Interpretation. The learned spread must originate in the training process (bootstrapped targets from a changing policy, the projection step, optimization noise) rather than en-
vironment stochasticity; the seed probe makes this concrete, and the doomed states show it directly. This is the behavioral, per-state realization of the conflation diagnosed architecturally by Clements et al. (2019) and licensed theoretically by Rowland et al. (2023a). It concentrates at the top of the ranking by selection: ranking by ∆x maximizes over noise, a winner’s curse in which face-value reads are least reliable exactly where they are most likely to be consulted.
Repairs Do Not Fix It Three repairs the literature would expect to help (risksensitive training, ensembling, and recalibration) fail to restore usable risk claims (3 seeds per game). A QR-DQN trained to be CVaR0.25 -greedy (selecting and bootstrapping from the CVaR of its own quantiles, audited under the matching policy) shows the unchanged signature: top-strata refutation 88–98%, zero pooled confirmations in all three games, pooled correlation 0.06/0.04/−0.02, despite healthy scores (Breakout 16.7, Asterix 18.4). Optimizing the tail does not make the tail-shape claims true. A five-member ensemble on a shared buffer, audited at its quantile-function average (W1 barycenter), attenuates the artifact: top-strata refutation falls from the single-member 95/100/95% to 81/98/57% and recovers 8 Breakout confirmations, but Seaquest and Asterix keep zero confirmations, pooled correlation stays near zero (0.14/0.12/−0.11), and the paired single members show the standard profile. Averaging helps but leaves the top claims uncalibrated. Miscalibrated or uninformative? We fit the best monotone recalibration (isotonic map from learned to true excess) on a calibration split and apply it to a held-out split. On the positive controls it rescues the head (refutation zero, 98–99% of genuine trade-offs asserted, correlation ≈0.9); but on MinAtar the only recalibration that lowers refutation collapses the top claims to 12–22% of their magnitude and asserts 0% of any real trade-off: it passes by saying nothing. The head is not miscalibrated but uninformative: missing information, not a fixable scale error (a supplementary distillation control confirms the architecture can fit the true distributions when supervised). Nor are the unsupported claims flaggable from the agent alone: a leak-free probe on 13 head-shape features does not beat the trivial claimmagnitude baseline (AUROC 0.92 either way), so no head signal separates them beyond claim size. Implications. (i) Interpretability claims built on visualized return distributions need ground-truth validation, as saliency maps did (Atrey, Clary, and Jensen 2020). (ii) Risksensitive policies derived from raw quantile heads (Keramati et al. 2020; Lim and Malik 2022) act on spurious risk at the critical states. (iii) Repairs that build uncertainty estimates around the raw head (Eriksson et al. 2022; Zanger, Böhmer, and Spaan 2024; Stutts et al. 2024; Gan et al. 2025) now have a measurement to be evaluated against, which the repairs we test do not clear: an uninformative head gives posthoc corrections no signal to work with. Limitations. Our claims cover QR-DQN, C51, and IQN under greedy and CVaR-greedy evaluation, ensembles, and one pretrained QR-DQN per ALE game; other calibrated variants (Stutts et al. 2024; Gan et al. 2025) may differ,
and an ALE agent trained under sticky actions remains the missing test at scale. Regimes where faint risk is entangled with dynamics have no answer key. The audit requires snapshot-restart access, though the pitfalls (P1, P2) apply to any restart-based evaluation.
Conclusion We set out to answer a question the distributional RL literature has left implicit: when a trained agent reports that two actions differ in risk, is that difference real? Answering it required a decision-level audit: a screening metric that isolates exactly the claims a risk-sensitive user would act on, ground truth from snapshot-restart Monte Carlo, and a statistical harness (permutation nulls, bootstrap refutation, FDR control) without which the audit manufactures its own false conclusions, as two silent pitfalls of restart-based evaluation taught us at our own expense. The result is consistent across the algorithms and games we test. On MinAtar, 40–95% of the strongest claimed risk trade-offs are provably false, essentially none is confirmable, and the placement of a head’s most confident claims is statistically indistinguishable from truth-blind (the null-head control of §). The effect is structural rather than a symptom of weak agents: it is fully formed early in training, grows with the very claim strength a practitioner would rank on, is uncorrelated with final score, and reproduces on a pretrained near-state-of-the-art Atari agent. Positive controls with trade-offs of known magnitude confirm the reading measures the agents, not the audit. It is also decisionrelevant: acting on the heads’ own CVaR advice at the states they flag as most risk-critical ranges from beneficial to significantly worse than chance, with no observable signal telling a practitioner which regime they are in. We also mapped the boundary of what can be done about it. Training for risk, ensembling, and recalibration all fail to produce a head whose top claims confirm; recalibration passes only by collapsing the claims to noise, showing the deficit is missing information rather than a fixable scale error, and no head feature flags the unsupported claims beyond their magnitude. Applied to an environment that contains real risk, the identical audit recovers it, isolating the deficit to the learned distributions: for the agents we test, they do not carry the risk structure they report. The implication for practice is direct. Face-value reads of distributional heads are least reliable at the states where they are most likely to be consulted, and should be validated against ground truth before they are trusted. We release the audit toolkit and a concrete standard for future methods: an audit-passing head confirms its top claims.
References Agarwal, R.; Schwarzer, M.; Castro, P. S.; Courville, A. C.; and Bellemare, M. G. 2021. Deep Reinforcement Learning at the Edge of the Statistical Precipice. In Advances in Neural Information Processing Systems 34. Artzner, P.; Delbaen, F.; Eber, J.-M.; and Heath, D. 1999. Coherent Measures of Risk. Mathematical Finance, 9(3): 203–228.
Atrey, A.; Clary, K.; and Jensen, D. 2020. Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning. In Proceedings of the 8th International Conference on Learning Representations. Bellemare, M. G.; Dabney, W.; and Munos, R. 2017. A Distributional Perspective on Reinforcement Learning. In Proceedings of the 34th International Conference on Machine Learning. Bellemare, M. G.; Dabney, W.; and Rowland, M. 2023. Distributional Reinforcement Learning. MIT Press. Benjamini, Y.; and Hochberg, Y. 1995. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society: Series B, 57(1): 289–300. Charpentier, B.; Senanayake, R.; Kochenderfer, M.; and Günnemann, S. 2022. Disentangling Epistemic and Aleatoric Uncertainty in Reinforcement Learning. arXiv preprint arXiv:2206.01558. Clements, W. R.; Van Delft, B.; Robaglia, B.-M.; Slaoui, R. B.; and Toth, S. 2019. Estimating Risk and Uncertainty in Deep Reinforcement Learning. arXiv preprint arXiv:1905.09638. Dabney, W.; Ostrovski, G.; Silver, D.; and Munos, R. 2018a. Implicit Quantile Networks for Distributional Reinforcement Learning. In Proceedings of the 35th International Conference on Machine Learning. Dabney, W.; Rowland, M.; Bellemare, M. G.; and Munos, R. 2018b. Distributional Reinforcement Learning with Quantile Regression. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence. Eriksson, H.; Basu, D.; Alibeigi, M.; and Dimitrakakis, C. 2022. SENTINEL: Taming Uncertainty with Ensemble Based Distributional Reinforcement Learning. In Proceedings of the 38th Conference on Uncertainty in Artificial Intelligence. Fournier, N.; and Guillin, A. 2015. On the Rate of Convergence in Wasserstein Distance of the Empirical Measure. Probability Theory and Related Fields, 162: 707–738. Gan, F.; Lu, Y.; Zhang, Y.; and Liu, Y. 2025. Conformal Prediction Beyond the Horizon: Distribution-Free Inference for Policy Evaluation. In Advances in Neural Information Processing Systems 38. Garcı́a, J.; and Fernández, F. 2015. A Comprehensive Survey on Safe Reinforcement Learning. Journal of Machine Learning Research, 16: 1437–1480. Greydanus, S.; Koul, A.; Dodge, J.; and Fern, A. 2018. Visualizing and Understanding Atari Agents. In Proceedings of the 35th International Conference on Machine Learning. Hoel, C.-J.; Wolff, K.; and Laine, L. 2021. Ensemble Quantile Networks: Uncertainty-Aware Reinforcement Learning with Applications in Autonomous Driving. arXiv preprint arXiv:2105.10266. Keramati, R.; Dann, C.; Tamkin, A.; and Brunskill, E. 2020. Being Optimistic to Be Conservative: Quickly Learning a CVaR Policy. In Proceedings of the 34th AAAI Conference on Artificial Intelligence. Lim, S. H.; and Malik, I. 2022. Distributional Reinforcement
Learning for Risk-Sensitive Policies. In Advances in Neural Information Processing Systems 35. Lyle, C.; Bellemare, M. G.; and Castro, P. S. 2019. A Comparative Analysis of Expected and Distributional Reinforcement Learning. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence. Machado, M. C.; Bellemare, M. G.; Talvitie, E.; Veness, J.; Hausknecht, M.; and Bowling, M. 2018. Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents. Journal of Artificial Intelligence Research, 61: 523–562. Rockafellar, R. T.; and Uryasev, S. 2000. Optimization of Conditional Value-at-Risk. Journal of Risk, 2(3): 21–41. Rowland, M.; Bellemare, M. G.; Dabney, W.; Munos, R.; and Teh, Y. W. 2018. An Analysis of Categorical Distributional Reinforcement Learning. In Proceedings of the 21st International Conference on Artificial Intelligence and Statistics. Rowland, M.; Munos, R.; Azar, M. G.; Tang, Y.; Ostrovski, G.; Harutyunyan, A.; Tuyls, K.; Bellemare, M. G.; and Dabney, W. 2023a. An Analysis of Quantile TemporalDifference Learning. arXiv preprint arXiv:2301.04462. Rowland, M.; Tang, Y.; Lyle, C.; Munos, R.; Bellemare, M. G.; and Dabney, W. 2023b. The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimation. In Proceedings of the 40th International Conference on Machine Learning. Stutts, A. C.; Erricolo, D.; Tulabandhula, T.; and Trivedi, A. R. 2024. Echoes of Socratic Doubt: Embracing Uncertainty in Calibrated Evidential Reinforcement Learning. arXiv preprint arXiv:2402.07107. Villani, C. 2009. Optimal Transport: Old and New. Springer. Wu, R.; Uehara, M.; and Sun, W. 2023. Distributional Offline Policy Evaluation with Predictive Error Guarantees. In Proceedings of the 40th International Conference on Machine Learning. Young, K.; and Tian, T. 2019. MinAtar: An Atari-Inspired Testbed for Thorough and Reproducible Reinforcement Learning Experiments. arXiv preprint arXiv:1903.03176. Zanger, M. A.; Böhmer, W.; and Spaan, M. T. J. 2024. Diverse Projection Ensembles for Distributional Reinforcement Learning. In Proceedings of the 12th International Conference on Learning Representations.
Supplementary Material Auditing the Risk Claims of Distributional Reinforcement Learning
This supplement collects the full methodology, hyperparameters, and complete per-game / per-seed results for every experiment in the paper, together with all tables and figures that did not fit the main text. Section numbers here are self-contained; references of the form “main text §X” point to the submitted paper. All numbers are regenerated from the released analysis/*.json artifacts and per-run stratified.npz files; the code paths that produce each are named inline so every result is traceable to a command.
Contents 1 Scope, Assumptions, and Anticipated Objections
2
2 Notation and the Audit Object 2.1 Proposition (dominance characterization) and proof . . . . . . . . . . . . . . . . . . . . . .
3 3
3 Audit Methodology in Full 3.1 Screening and stratified sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Ground truth by snapshot restart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 The statistical harness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 The two pitfalls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3 3 4 4 4
4 Environments
5
5 Training Details and Hyperparameters
5
6 Main QR-DQN Audit: Complete Per-Seed Results 6.1 Truth-blind null head . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6 6
7 C51 and IQN Replication
7
8 Pretrained SB3-Zoo QR-DQN on Full ALE (audit only)
8
9 Sticky-Action ALE Training: Attempt and Status
9
10 Positive Controls 10 10.1 RISKYGRID (large trade-offs) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 10.2 RISKYGRIDFAINT (detection-floor trade-offs) . . . . . . . . . . . . . . . . . . . . . . . . . 10 11 Decision-Level Evaluation (CVaR)
11
1
12 Structural Probes 12.1 Training time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2 Performance decoupling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.3 Cross-seed ensemble flag . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11 11 12 12
13 Repairs: Risk-Sensitive Training and Ensembling
12
14 Recalibration: What a Repair Can and Cannot Recover
12
15 Distillation Against Ground Truth (Capacity Control)
13
16 A Ground-Truth-Free Detector Fails
13
17 Specimen Gallery
14
18 Reproduction and Compute Accounting
14
1
Scope, Assumptions, and Anticipated Objections
Scope of the empirical claims. Our evidence covers three canonical value-based distributional algorithms (QR-DQN, C51, and IQN) on three MinAtar games (five seeds for QR-DQN, three for C51/IQN), plus one pretrained SB3-zoo QR-DQN per ALE game, under greedy and CVaR-greedy evaluation, five-member ensembles, and a monotone recalibration. We do not claim these results transfer automatically to architectures and settings we did not test, among them FQF and its variants, DSAC and continuous-control actor-critics, offline distributional RL, and model-based agents (MuZero, Dreamer). We phrase conclusions accordingly (“for the agents we test,” “across the three algorithms”), and treat broader coverage as the natural next use of the released toolkit rather than an established fact. What the evidence does establish is that the phenomenon is not idiosyncratic to one algorithm, representation, seed, domain scale, or training budget within the family we audit, and that it lands on the decision-relevant claims. Objection 1: are you auditing the learned approximation or the Bellman target? These are distinct objects, and we audit neither the target nor an idealized fixed point: we audit the trained network’s output distribution as it is consumed. The distinction matters because a defender might argue the head was never intended to recover the true return distribution Z π , that its spread is an optimization by-product with no semantic commitment. We grant the premise and observe that it does not rescue the downstream uses. Interpretability tools visualize the head’s distribution as the agent’s belief about returns; CVaR and other risk functionals are computed directly from the head’s quantiles to choose actions; safety monitors threshold that same spread. Each of these reads Ẑ as if it recovers Z π . We therefore test exactly the object these uses consume, and report where that reading is and is not warranted. The theory that the learned fixed point need not equal Z π (Rowland et al. 2023) is, on this view, not a defense but a prediction our measurement confirms at the decision level. A separate audit of the Bellman target (e.g. the projected bootstrapped distribution) is a complementary question our harness could address but that we do not claim to answer here. Objection 2: perhaps MinAtar simply contains little action-conditional aleatoric risk, so “zero confirmations” is unsurprising. This is the most important alternative explanation, and we separate the two claims it conflates. The confirmation result (essentially no true trade-off is detectable) is indeed partly a property of the environments: under near-greedy play, action-conditional dominance violations may be genuinely rare. But the paper’s core claim is the refutation result (that the heads assert large risk tradeoffs where the ground truth supports none, and that claim does not depend on how much real risk exists: it is about the gap between what the head reports and what the environment delivers). Three controls 2
make this concrete. (i) The positive controls (§10) show the identical audit confirms 96–100% of trade-offs when they are present, down to the detection floor, so a null confirmation rate is diagnostic of the agents, not a blind spot of the method. (ii) The recalibration analysis (§14) shows that on MinAtar no monotone correction recovers usable signal: the head is uninformative, which is a statement about the head, not about the scarcity of risk. (iii) The decision-level evaluation (§11) shows that acting on the heads’ claims changes outcomes, sometimes for the worse, so the asserted risk is consequential regardless of the underlying base rate. The honest residual is that our absolute confirmation numbers would rise in environments richer in action-conditional risk; the refutation and anti-correlation findings would not be explained away by such richness.
2
Notation and the Audit Object
P t Let Z π (s, a) = t γ Rt be the random discounted return from taking action a in state s and following π thereafter, with Qπ (s, a) = E Z π (s, a). A distributional agent learns an approximation Ẑ(s, a) to the law of Z π . We audit relative risk claims: for the two actions a1 , a2 ranked top-2 by mean at s, the excess Wasserstein gap ∆x (s) = W1 Ẑ(s, a1 ), Ẑ(s, a2 ) − µ̂1 (s) − µ̂2 (s) , µ̂j = EẐ(s, aj ), P with W1 = N1 i |θi (s, a1 ) − θi (s, a2 )| computed in closed form from the N quantile atoms. The audit MC compares ∆x (s) (a claim) against ∆ˆx (s), the same statistic computed from snapshot-restart Monte Carlo rollouts (ground truth).
2.1
Proposition (dominance characterization) and proof
Proposition 1. For real-valued distributions with CDFs F, G, Z Z ∆x = |F − G| dx − (F − G) dx = 0 iff F − G does not change sign, i.e. iff one distribution first-order stochastically dominates the other. When positive, ∆x equals twice the mass of the smaller-signed region of F − G: the magnitude of the dominance violation. R R Proof. InR one dimension W (F, G) = |F − G| and EG − EF = (F − G). Writing hR = F −RG, the 1 R R + R − identity |h| − | h| = 2 min( h , h ) gives both claims: the difference is zero iff h+ or h− is zero, i.e. iff h is single-signed (dominance), and otherwise equals twice the smaller tail mass. Consequently ∆x (s) > 0 is exactly the agent’s claim that neither action is safer at every quantile, so a risk-sensitive (e.g. CVaR) policy and a mean-greedy policy can disagree at s; when ∆x (s) = 0 every monotone risk functional ranks the two actions identically. Screening by ∆x therefore isolates the claims that carry decision content while ignoring errors both actions share.
3
Audit Methodology in Full
3.1
Screening and stratified sampling
We roll out the trained agent (ϵ-greedy, ϵ = 0.05) to collect 2,000 unique states, score each by ∆x from the learned head, and rank them. Because overclaiming could concentrate anywhere in the ranking, we audit a 3
stratified sample: 7 states from each of 7 rank strata (top 0.5%, 0.5–2%, 2–5%, 5–10%, 10–25%, 25–50%, 50–100%) for 49 audited states per run. “Top strata” denotes the top two strata (top 2% of the ranking). Code: distrl ot/stratify.py, strata in STRATA.
3.2
Ground truth by snapshot restart
At collection time we store deep copies of the environment at up to 16 distinct visits of each state. For each audited state and each top-2 action we run 2,000 rollouts (MinAtar; 1,000 for ALE): restart from a stored snapshot (cycling through snapshots), force the action, then follow the greedy policy to termination or horizon 103 , accumulating γ-discounted return. Rollouts are vectorized and each restarted copy is independently reseeded (pitfall P1, §3.4). Environment stochasticity (sticky actions with repeat probability 0.1 on MinAtar / 0.25 on ALE, random spawns, difficulty ramping) is the only noise source, matching the aleatoric semantics of Z π under the evaluation policy. Code: distrl ot/mc validate.py (collect snapshot pool, mc returns); ALE variant distrl ot/atari audit.py (snapshot/restore instead of deepcopy).
3.3
The statistical harness
MC Confirmation test. A claimed trade-off at s is confirmed if ∆ˆx (s) exceeds a permutation null formed by splitting the same action’s rollouts in half and computing the excess between the halves (pure estimation noise by construction; conservative because half-samples are noisier). We use 200 permutation resamples per state. Refutation test. A claim is refuted if the learned ∆x (s) exceeds the 97.5th percentile of the bootstrap distribution (500 resamples) of the MC excess. The finite-sample bias of the MC estimator (P2) inflates that percentile, making refutation conservative. FDR control. Both tests’ p-values are corrected by Benjamini–Hochberg at FDR 0.10, pooled per game across seeds. A state can be neither confirmed nor refuted, but never both. Code: distrl ot/report.py (bh discoveries, analyze run). Aliasing control. Per state we record an aliasing index: the standard deviation of per-snapshot mean returns divided by the total return standard deviation, averaged over the two actions. States with index below 0.25 are ones where the observation approximately pins down the outcome distribution; because the index is built from per-snapshot means, it is a coarse control that shape-only aliasing would evade. Headline results are reported separately for low-alias states.
3.4
The two pitfalls
P1: RNG cloning. Restart-based evaluation typically snapshots environments with deepcopy, which clones the environment’s internal random generator, so every “random” rollout replays one identical future. Our first audit, built this way, concluded all claimed risk was artifact; the tell was impossible determinism in a game with random spawns. The fix is to reseed each restarted copy independently (mc returns reseeds every clone). P2: the empirical Wasserstein noise floor. For two n-samples from the same distribution with spread σ, the empirical ∆x concentrates around c σn−1/2 > 0 rather than zero. Figure 1 measures this on a real high-variance audited Breakout state: at the pilot n = 200 the mean floor is ≈0.07 and its 95th percentile ≈0.22, exceeding most true effect sizes; a fixed small threshold “confirmed” two-thirds of claims in an early audit, all noise. All tests are therefore calibrated against per-state nulls at the deployed sample size n = 2,000.
4
Figure 1: Pitfall P2, measured. Empirical excess between two n-samples from the same return distribution (σ ≈ 2.3), 300 replications per n. The floor decays as n−1/2 ; at n = 200 its 95th percentile (≈ 0.22) dwarfs a fixed 0.05 threshold.
4
Environments
MinAtar (distrl ot/envs.py): Breakout, Seaquest, Asterix, 10 × 10 × C binary observations, sticky actions p = 0.1, random spawns and difficulty ramping. ALE (distrl ot/atari.py): full Atari with SB3-zoo preprocessing (grayscale 84 × 84, frameskip 4 with max over the last two raw frames, 4-frame stack, minimal action set, FIRE on reset, episodic-life, sign-clipped rewards); evaluation stochasticity is sticky actions p = 0.25 implemented in the wrapper. RISKYGRID (distrl ot/riskygrid.py): a ten-step corridor; each episode draws per-column gamble scales g(x) ∈ {0, 0.5, . . . , 2.0} rendered into the observation; each step the agent chooses SAFE (reward 0.5, deterministic) or GAMBLE (reward 0.5 ± g(x), fair coin). Means are identical, so every g(x) > 0 column is a pure risk trade-off of known magnitude and g(x) = 0 columns have none; the observation determines the full g-map (zero aliasing by construction). The faint variant scales gambles to g ∈ {0, 0.05, 0.1, 0.2, 0.4}.
5
Training Details and Hyperparameters
All MinAtar agents share the recipe in Table 1. QR-DQN uses 51 quantile atoms; C51 uses 51 categorical atoms on [0, 40]; IQN uses a 64-dim cosine tau embedding evaluated on the 51-tau midpoint grid. Training code: distrl ot/train.py (--algo qrdqn|c51|iqn), ensembles distrl ot/train ensemble.py, ALE distrl ot/train atari.py. Ensembles (§13) are K = 5 QR-DQN members on a shared replay buffer, bootstrapped-DQN style: members share one experience stream, differ by initialization, and the behavior member is resampled uniformly each episode; every member trains on the same batch against its own target. The CVaR0.25 -greedy agent (§13) uses the standard QR-DQN recipe but selects actions and bootstraps targets by the CVaR of its own quantiles rather than the mean. Final scores (mean of last 100 episodes, averaged over seeds), Breakout / Seaquest / Asterix: QR-DQN 14.6/25.2/22.1, C51 15.8/9.2/10.0, IQN 13.4/17.7/22.3.
5
Hyperparameter
MinAtar
ALE (sticky)
Frames / steps Optimizer Learning rate Batch size Replay buffer Warmup Target update Train frequency γ ϵ schedule Quantiles / atoms Sticky-action p Network
3M steps Adam 10−4 32 105 5k 1k steps every step 0.99 1 → 0.1 over 100k 51 0.1 MinAtar conv (16 ch)
10M frames (frameskip 4) Adam 5 × 10−5 32 2.5 × 105 20k frames 10k grad steps every 4 frames 0.99 1 → 0.01 over 10% 200 0.25 Nature CNN
Table 1: Training hyperparameters. The ALE recipe follows Dabney et al. (2018).
Figure 2: Training curves for all 33 MinAtar agents (rolling mean over 200 episodes). The audit is withinagent (each head is tested against its own claims), so conclusions do not depend on cross-algorithm score parity.
6
Main QR-DQN Audit: Complete Per-Seed Results
Table 2 gives every seed. Pooled top-strata refutation is 84/84/66% (Breakout/Seaquest/Asterix); zero confirmations among 245 states per game; pooled learned–true correlation 0.02/0.05/ − 0.03. Source: report/summary.md. Refutation falls monotonically down the ranking (Fig. 3), and the learned–true scatter (Fig. 4) shows the top-stratum points lying flat on the floor rather than hugging the diagonal.
6.1
Truth-blind null head
The declining refutation curve is partly a test-power artifact: small claims cannot be refuted regardless of truth. To make the comparison exact we permute each run’s learned claims across its audited states (magnitudes and rank strata preserved, placement randomized w.r.t. ground truth) and re-run the identical pipeline 1,000 times. Truth-blind top-strata refutation is 81% [75, 86], against 79% observed (165/210, p = 0.22): the heads place their strongest claims no better than truth-blind chance (Fig. 5). Code: scripts/null head.py, data analysis/null head.json.
6
Game
Seed
Refuted top strata
Confirmed anywhere
Breakout
0 1 2 3 4 pooled
11/14 = 79% 10/14 = 71% 13/14 = 93% 12/14 = 86% 13/14 = 93% 59/70 = 84% [74,91]
0/49 0/49 0/49 3/49 0/49 0/245
0 1 2 3 4 pooled
14/14 = 100% 12/14 = 86% 9/14 = 64% 13/14 = 93% 12/14 = 86% 59/70 = 84% [74,91]
3/49 0/49 0/49 0/49 3/49 0/245
0 1 2 3 4 pooled
8/14 = 57% 9/14 = 64% 11/14 = 79% 11/14 = 79% 8/14 = 57% 46/70 = 66% [54,76]
0/49 0/49 0/49 0/49 0/49 0/245
Seaquest
Asterix
corr 0.02
corr −0.01
corr −0.04
Table 2: QR-DQN, all 15 runs (Wilson 95% intervals on pooled rows). corr is the per-seed-mean learned– true correlation.
7
C51 and IQN Replication
Table 3 gives pooled results; the signature replicates under a categorical projection (C51) and an implicitquantile projection (IQN). Since C51’s categorical projection shares none of QTD’s fixed-point pathologies and IQN replaces the fixed tau grid, the overclaiming implicates the common ingredient (bootstrapped distributional targets from a changing policy under function approximation), not the representation. Sources: report c51/summary.md, report iqn/summary.md. Algorithm
Game
Refuted top 2%
Refuted rest
corr(learned,true)
C51
Breakout Seaquest Asterix
81% [67,90] 95% [84,99] 57% [42,71]
10% 16% 5%
IQN
Breakout Seaquest Asterix
64% [49,77] 79% [64,88] 40% [27,56]
7% 7% 8%
0.02 0.05 −0.11 0.08 −0.11 −0.08
Table 3: C51 and IQN, 3 seeds per game (147 states each), FDR 0.1. Zero pooled confirmations in all six cells.
7
Figure 3: Left: refutation rate by rank stratum (QR-DQN, pooled over 5 seeds, Wilson 95% intervals, FDR 0.1). Right: pooled confirmation rate by stratum (essentially zero everywhere).
Figure 4: Learned excess vs. ground-truth (MC) excess for all 735 QR-DQN audited states (color = rank stratum, 0 = top; dashed = identity).
8
Pretrained SB3-Zoo QR-DQN on Full ALE (audit only)
The pretrained zoo agents were trained without sticky actions, so their claimed spread cannot reflect environment stochasticity by construction; this audit quantifies the artifact’s size and placement (a consistency check, not an independent replication). Full numbers in Table 4. Source: analysis/ale audit.json. Game
n
Refuted top
Refuted rest
Confirmed
corr
mean top claim / MC
Breakout Seaquest
49 49
14/14 = 100% 2/14 = 14%
0/35 = 0% 1/35 = 3%
0 0
−0.22 −0.25
0.184/0.000 0.021/0.008
Table 4: Pretrained SB3-zoo QR-DQN (10M ALE frames, 200 quantiles), audited at n = 1000 rollouts, sticky-action evaluation p = 0.25. Breakout raw score ≈400 in our harness. The Seaquest head sits at the null floor (median null95 ≈ 0.03), so few claims are refutable but none is confirmable.
8
Figure 5: Observed top-strata refutation (red) sits inside the truth-blind null band (blue), p = 0.22.
Figure 6: Refutation by stratum for C51 (left) and IQN (right), pooled over 3 seeds. Same monotone decline as QR-DQN.
9
Sticky-Action ALE Training: Attempt and Status
The cleanest version of the full-Atari test would train a QR-DQN with sticky actions (p = 0.25), so that any claimed spread could legitimately reflect environment risk, and then audit it (in contrast to the pretrained zoo agents of §8, which were trained under deterministic dynamics). We implemented the trainer (distrl ot/train atari.py, Nature-CNN QR-DQN with the standard Atari recipe) and the matching native-checkpoint audit path (distrl ot/atari audit.py --native), and ran Breakout and Seaquest at three seeds. Within our compute budget (a single consumer GPU, 10M frames), the fromscratch agents did not reach competitive scores (sticky-action training from scratch is markedly slower than the no-sticky regime the published zoo checkpoints used), so we do not report an audit of undertrained agents, which would be uninformative. We report this openly: the sticky-action-trained ALE audit remains the natural extension of this work, and the released trainer makes it a matter of compute rather than new method. The ALE evidence we do report is the pretrained zoo audit (§8), with its training-dynamics caveat stated explicitly.
9
10
Positive Controls
10.1
RISKYGRID (large trade-offs)
On RISKYGRID, where trade-offs exist by construction, the audit’s verdicts invert: 143/147 = 97% [93, 99] pooled FDR-corrected confirmations (the misses are g=0 states); zero false refutations in the top strata (0/42, 4/105 below, consistent with the FDR budget); learned–true correlation 0.92 (per-seed 0.96/0.92/0.89); and the head’s CVaR choice picks the truly safer action in 99–100% of 143 decidable states with realized regret ≈0. Source: report riskygrid/.
Figure 7: RISKYGRID positive control: confirmation rate by stratum (left) and CVaR-choice agreement with ground truth (right).
10.2
RISKYGRIDFAINT (detection-floor trade-offs)
The faint variant straddles the audit’s detection floor. Table 5 gives the per-scale breakdown: 96–100% of g > 0 states confirmed at every level including g = 0.05, g = 0 states at the FDR budget, ranking–truth correlation pooled r = 0.89 and corr(learned, g) = 0.95. Source: analysis/faint crossover.json. g
n
Confirmed
Refuted
mean MC excess
mean learned excess
0.00 0.05 0.10 0.20 0.40
10 10 9 25 93
10% [2,40] 100% [72,100] 100% [70,100] 96% [80,99] 100% [96,100]
6 0 0 0 4
0.003 0.045 0.082 0.148 0.350
0.008 0.014 0.018 0.060 0.165
Table 5: RISKYGRIDFAINT: confirmation tracks the true gamble scale g down to the detection floor. The head compresses magnitudes (learned < MC) but orders them correctly.
10
Figure 8: Positive-control summary (main-text figure, reproduced). Left: RISKYGRID learned-vs-true scatter tracks the diagonal; center: faint-variant confirmation vs. scale g; right: FDR-corrected confirmation, controls vs. the three MinAtar games (MinAtar bars all at zero).
11
Decision-Level Evaluation (CVaR)
On decidable states (true CVaR gap significant at 95% by bootstrap) we compare three choosers of the safer action at level α: the head’s CVaR ranking, the mean-greedy baseline, and ground truth. Table 6 gives α ∈ {0.1, 0.25} for all algorithms. Source: report*/cvar analysis.md, code distrl ot/cvar.py. safer pick (%)
CVaR regret
Algo (α=0.1)
Game
head
mean
head
mean
QR-DQN
Breakout Seaquest Asterix
81 34 52
44 63 48
0.20 0.65 0.20
1.36 0.30 0.26
C51
Breakout Seaquest Asterix
65 28 36
71 56 57
0.45 0.67 0.48
0.10 0.20 0.24
IQN
Breakout Seaquest Asterix
79 51 47
42 49 73
— — —
— — —
Table 6: Decision-level evaluation at the heads’ own most-flagged states. Bold marks a clear winner. The Seaquest QR-DQN head is below chance (one-sided binomial p = 8 × 10−4 at α=0.25; p = 0.045 at α=0.1). Across all three algorithms no game-independent rule tells a practitioner when the head’s advice can be trusted.
12
Structural Probes
12.1
Training time
Full MC audits of intermediate Breakout QR-DQN checkpoints (Table 7) show the overclaiming fully formed by 500k steps and unchanged to 3M while score climbs from 8.6 to 14.4; confirmations are zero at every stage, and screening every dense checkpoint shows the top-2% claimed magnitude grows over training in 15/15 runs. Source: analysis/tier2 ckpt audit.json. 11
Figure 9: CVaR-choice agreement with ground truth, QR-DQN / C51 / IQN. Steps
Refuted top
Refuted rest
Confirmed
corr
mean top claim / score
500k 1M 2M 3M
34/42 = 81% 37/42 = 88% 36/42 = 86% 35/42 = 83%
20% 22% 20% 23%
0 0 0 0
0.10 0.06 0.13 −0.11
0.135 / 8.6 0.138 / 11.1 0.177 / 14.3 0.181 / 14.4
Table 7: Training-time audit (Breakout QR-DQN, 3 seeds pooled, 42 top-strata states/point).
12.2
Performance decoupling
Across all 33 MinAtar runs, final score and top-strata refutation are uncorrelated (Spearman −0.07 pooled; per-game +0.12/−0.23/−0.19). Per-run (score, refuted) pairs are in analysis/tier0 score vs refutation.jso
12.3
Cross-seed ensemble flag
Re-evaluating each audited observation under the game’s four sibling QR-DQN seeds: siblings reproduce a median 3.8% of a top-stratum claim’s magnitude (vs. 11% lower in the ranking), the cross-seed median stays uncorrelated with truth (|r| ≤ 0.07), and using sibling disagreement to flag refuted-within-top claims gives AUROC 0.48 (top strata). Ensembling by sibling median therefore flags irreproducibility cheaply but does not restore calibration. Source: analysis/tier0 ensemble.json.
13
Repairs: Risk-Sensitive Training and Ensembling
Table 8 gives the full revision experiment: a CVaR0.25 -greedy agent, a 5-member ensemble audited at its W1 barycenter, and the paired single member (member 0) as control, 3 seeds per game. Neither intervention restores calibration; ensembling attenuates on two games but Seaquest and Asterix keep zero confirmations and pooled correlation stays near zero. Source: analysis/revision audit.json, code distrl ot/train ensemble.py, models.py (EnsembleView, RiskPolicyView), train.py --risk-alpha.
14
Recalibration: What a Repair Can and Cannot Recover
The interventions above operate on the training procedure. We now ask the sharper question directly: is the head miscalibrated (a fixable scale error) or uninformative (no recoverable signal)? We fit the best monotone recalibration (isotonic regression, pool-adjacent-violators, of the true MC excess on the learned excess) on a calibration split of audited states, apply it to a held-out split, and measure whether the recalibrated claims g(learned) (i) stop being refuted and (ii) retain any claimed magnitude or assert any real trade-off. 200 random calibration/test splits; code scripts/recalibrate.py, data analysis/recalibrate.json. 12
Figure 10: Top-strata refutation is flat across training while score doubles. Condition
Game
Refuted top
Refuted rest
Confirmed
corr (pooled)
CVaR0.25 -greedy
Breakout Seaquest Asterix
41/42 = 98% [88,100] 40/42 = 95% [84,99] 37/42 = 88% [75,95]
46% 56% 22%
0 0 0
Ensemble (W1 bary.)
Breakout Seaquest Asterix
34/42 = 81% [67,90] 41/42 = 98% [88,100] 24/42 = 57% [42,71]
25% 46% 11%
8 0 0
0.06 0.04 −0.02
Ensemble member 0
Breakout Seaquest Asterix
40/42 = 95% [84,99] 42/42 = 100% [92,100] 40/42 = 95% [84,99]
40% 57% 27%
0 0 0
0.14 0.12 −0.11 −0.05 −0.11 −0.12
Table 8: Repair experiments, 3 seeds/game (147 states each), FDR 0.1. Final scores: CVaR agent Breakout 16.7, Asterix 18.4; ensemble members reach comparable MinAtar scores. Ensembling recovers 8 Breakout confirmations and lowers top-strata refutation on 2 games, but does not calibrate the top claims.
15
Distillation Against Ground Truth (Capacity Control)
Recalibration cannot recover signal that is not there; but is the MinAtar overclaiming a limit of the architecture, or of the bootstrapped training target? We fine-tune the trained QR-DQN head against snapshot-restart MC return distributions (quantile regression on MC samples) at a distill split of audited states, and evaluate on a held-out split the head’s projection error (W1 between its top-2 quantiles and MC) and its refutation rate, before vs. after. Code scripts/distill.py, data analysis/distill.json. The takeaway is deliberately narrow: the architecture is not the bottleneck (it fits true marginals when supervised), but the overclaiming is not a simple fitting error that naive distillation removes, consistent with the paper’s account that it originates in the bootstrapped target and the argmax selection over noise, both of which a small ground-truth fine-tune does not undo.
16
A Ground-Truth-Free Detector Fails
Can a fabricated claim be flagged from the agent alone, without MC? We extract 13 head-only features per audited state (the excess itself, W1 , mean gap, per-action spread and range, between-action crossing count, 13
Figure 11: Learned vs. MC excess for the CVaR-greedy agent (top) and the ensemble barycenter (bottom), pooled over 3 seeds per game (color = stratum). The clouds remain flat: the interventions do not move the top claims onto the diagonal. tail location of maximum divergence, quantile skew, per-action means) and fit a leak-free logistic probe to predict the per-state refutation label, under leave-one-game-out and leave-one-seed-out cross-validation (scripts/detector.py, analysis/detector.json).
17
Specimen Gallery
18
Reproduction and Compute Accounting
Code layout. Environments envs.py, atari.py, riskygrid.py; agents qrdqn.py, c51.py, iqn.py, models.py; training train.py, train ensemble.py, train atari.py; audit critical.py (screening), mc validate.py (rollouts), stratify.py (stratified audit + sharding/merge), atari audit.py (ALE), report.py (FDR/Wilson aggregation), cvar.py (decision-level); extra analyses under scripts/. Representative commands. python -m distrl ot.train --game breakout --steps 3000000 --seed 0 python -m distrl ot.stratify --run runs/breakout qrdqn s0 --mc-rollouts 2000 python -m distrl ot.report --runs-dir runs instance --out report python -m distrl ot.train atari --game breakout --frames 10000000 --device cuda python -m distrl ot.atari audit --game breakout --policy CKPT --native --out OUT python scripts/null head.py --runs-dir runs instance
14
Domain
Refuted top before
Refuted top after recal
Magnitude retained
corr(g,true) after
Real trade-offs still asserted
MinAtar Breakout MinAtar Seaquest MinAtar Asterix
82% 82% 65%
23% 37% 11%
12% 18% 22%
0% 0% 0%
RISKYGRID
0% 0%
0% 6%
134% 211%
−0.11 0.01 −0.05
RISKYGRIDFAINT
0.94 0.88
99% 98%
Table 9: Best monotone recalibration, held-out. On the positive controls (real risk) recalibration passes the audit and preserves or amplifies the compressed claims, retaining 98–99% of genuine trade-offs at correlation ≈0.9. On MinAtar the only recalibration that reduces refutation collapses claims to 12–22% of their magnitude and asserts 0% of any real trade-off: it passes by saying nothing. The MinAtar head is not miscalibrated but uninformative: the deficit is missing information, not a scale error.
Figure 12: Audit-passing recalibration: magnitude retained (solid) and real trade-offs still asserted (hatched). On the positive controls both are near or above 100%; on MinAtar magnitude collapses and no real trade-off survives. Compute. The full study audited 2,450 states over 9.6M ground-truth rollouts. MinAtar training and auditing plus positive controls ran on commodity multi-core CPU instances for under $9 total; the stickyaction ALE campaign (§9) ran on a single consumer GPU. Sharded audits are deterministic given --seed (the env RNG is seeded), so shards agree on the state pool and split the rollout work without overlap; --merge recombines them. Released artifacts. All code, configs, per-run stratified.npz (raw MC returns per audited state), analysis/*.json, and figure scripts are released, so every table and figure here regenerates without re-running rollouts.
15
Projection error (W1 to MC)
Refuted fraction
Distill split, before Distill split, after
2.11 0.45
0.36 0.30
Held-out split, before Held-out split, after
2.13 1.12
0.25 0.33
Table 10: Distillation against MC ground truth, pooled over 9 QR-DQN runs. Mixed outcome, reported in full. Projection error drops 5× in-sample and ≈2× on held-out states: the architecture can represent MinAtar’s true return distributions, so capacity is not the bottleneck. However, a light per-state distillation does not repair the relative-excess overclaiming out-of-sample (held-out refutation is unchanged to slightly higher): the excess is a fragile difference between two actions and does not generalize from ≈24 supervised states. The clean capacity argument remains the positive control (§10), where the identical architecture achieves correlation ≈0.92 with truth on a domain that contains real risk.
CV scheme Leave-one-game-out Leave-one-seed-out
Head features (13)
Learned excess only (1)
Oracle (+MC feats)
0.91 0.92
0.92 0.92
0.95 0.96
Table 11: Detector AUROC (735 QR-DQN states, 219 refuted). The 13 head features do not beat the trivial one-feature baseline of claim magnitude, and that baseline is itself near-mechanical (the refutation test fires on large claims). No head signal distinguishes fabricated claims beyond their size; an actionable detector would need to separate fabricated from genuine claims at matched magnitude, but on MinAtar essentially all top claims are fabricated, so there is no genuine class to contrast against. This makes the paper’s “undetectable from the agent alone” statement rigorous.
16
Figure 13: Audited specimens: environment snapshot and both actions’ return CDFs, learned (solid) vs. 2,000-rollout MC ground truth (dashed). Top: a doomed Breakout state (every future yields the same return, dashed step) where the head reports crossing distributions with excess 0.44. Middle: top-stratum Seaquest and Asterix states whose true distributions coincide while the learned ones cross. Bottom: the same signature in C51. Pooled across seeds, 14/70 = 20% of top-stratum Breakout states are of the doomed type.
17