ActMap: Single-Pass Uncertainty Quantification from Generation-Time Activation Maps Jacopo Dardini
Roberta Calegari
University of Bologna, Italy
verification effort goes when human oversight cannot scale to every generation (Bowman et al., 2022).
arXiv:2609.11498v1 [cs.AI] 10 Sep 2026
Abstract Practical uncertainty quantification (UQ) for large language models must decide, from a single generation, whether a specific answer should be trusted. Existing methods either sample multiple generations, read only output-token probabilities, or reduce the model’s internal computation to a single hidden state. We introduce ActMap, a white-box representation that compresses the generation-time hidden-state trajectory (every layer, every generated token) into a fixed 12 × 32 × 128 tensor of temporal-statistic channels that preserves structure across transformer depth and pooled hidden coordinates. The map is captured during the generation pass with no measurable overhead, has a fixed shape across model depths and hidden sizes, and occupies 96 KiB: a compact artifact that can be retained for audit-relevant generations and probed directly, with occlusion analysis localizing the classifier’s signal to mid-depth regions of the map. A lightweight classifier, instantiated as a compact Vision Transformer, reads an estimated correctness probability from each map in a fraction of a millisecond; capacity-matched MLPs perform comparably, indicating the representation itself carries the result. Trained and evaluated in-domain on short-answer QA, direct-answer math, and summarization factuality with three instruction-tuned 7–8B models, ActMap consistently outperforms sampling, token-probability, attention, and embedding baselines, and matches ACT-ViT, a detector trained on dense activation tensors 67× larger, at essentially the same mean AUROC with lower calibration error on ten of twelve pairs. The resulting score supports abstention, routing, and selective verification from a single generation, making it a practical primitive for scalable oversight of deployed models.
Existing UQ methods occupy three regimes, each with a structural gap. Sampling-based black-box methods such as semantic entropy measure disagreement across multiple generations (Kuhn et al., 2023; Farquhar et al., 2024); they capture meaning-level ambiguity but require N sampled generations per query, which raises total decoding compute and throughput requirements even when parallel sampling hides latency. Grey-box methods read output-token probabilities (perplexity, mean token entropy, or learned functions of the output distribution (Bar-Shalom et al., 2026)) from a single pass, but see only the final projection of the model’s computation. White-box methods look inside the model, yet most reduce the internal state to a single vector: a probe on the last token’s hidden state (Azaria and Mitchell, 2023; Marks and Tegmark, 2024), a pooled sentence representation, or aggregated logits at one self-evaluation position (Xiao et al., 2026). Between “one vector” and “ten regenerations” lies almost everything the model computed while producing the answer: how representations differ across depth and across the hidden space, and how they evolve over generated tokens; many existing methods collapse or omit much of this structure. We propose ActMap, a representation designed to keep it. During a single generation, ActMap records the hidden state of every transformer layer at every generated token, then compresses this variable-size L × T × D trajectory into a fixed 12 × 32 × 128 tensor: twelve channels of temporal statistics over the token axis, with adaptive pooling mapping the layer and hidden-dimension axes to fixed sizes. The result is compact (96 KiB per generation), fixed in shape across model depths and hidden sizes, and structured: rows index transformer depth, columns index pooled hidden coordinates, and channels are aligned temporal statistics. A learned classifier, in our main experiments a compact Vision Transformer, maps each tensor to an estimated correctness probability. The classifier never sees the generated text or output token probabilities: correctness is predicted from the internal state alone, with no extra model calls.
Introduction Large language models answer questions, solve problems, and summarize documents with fluent confidence whether or not they are right. Deployments that act on these outputs need a per-answer reliability signal: a score that says, for this specific generation, how likely it is to be correct. This answer-level uncertainty quantification (UQ) problem, detecting hallucinated or otherwise unreliable answers, is increasingly the gatekeeper for abstention, retrieval fallback, escalation, and human review (Kadavath et al., 2022; Farquhar et al., 2024), and the signal that decides where scarce
Our primary setting is in-domain deployment: an operator serves a fixed model on a fixed task, labels a set of generations once, and then assigns every subsequent answer a single correctness score, on which downstream decisions such as 1
abstention, escalation, routing, or selective verification can be thresholded. Our main contributions are as follows:
and Mitchell, 2023; Marks and Tegmark, 2024), and layeraggregated logits at a self-evaluation token improve calibration (Xiao et al., 2026). These approaches collapse the token axis entirely (one position) or the layer axis (one pooled vector), assuming the reliability signal is localized.
• Representation. A fixed-size, multi-channel activation-map summary of the full generation trajectory (layers, generated tokens, hidden dimensions, and activation dynamics), computable during one generation pass for any decoder-only transformer; ablations show robustness to the classifier choice.
Structured white-box methods. EigenScore measures the differential entropy of sampled-response embeddings in internal space (Chen et al., 2024); RAUQ aggregates attention through uncertainty-relevant heads, unsupervised, in a single pass (Vazhentsev et al., 2026); TAD learns attentionbased features of conditional dependency between generation steps (Vazhentsev et al., 2025). ACT-ViT is the closest representation-level comparison: it trains a vision transformer directly over padded layer-by-token activation tensors (Bar-Shalom et al., 2025). Unlike methods that select attention statistics or embedding geometry, ACT-ViT and ActMap both preserve joint depth and token structure. They differ in what is retained: ACT-ViT keeps a dense tensor whose width tracks the generator’s hidden size, reads at most the first 100 generated tokens, and couples the detector to the generator through a model-specific adapter; ActMap compresses the full trajectory into a fixed-shape map that is identical in geometry for every generator.
• Method. A single-pass supervised UQ method with negligible added inference cost: capture is not measurably slower than plain decoding, and scoring is one forward pass of a 2.4M-parameter classifier. • Evaluation. A unified comparison against eight baselines organized in an explicit black-/grey-/white-box taxonomy, on four tasks and three open-weight 7–8B models under a shared balanced protocol. The comparison includes ACT-ViT, evaluated with its complete published per-pair architecture sweep; ActMap reaches essentially the same mean AUROC from a 67× smaller representation with a single fixed classifier configuration. • Analysis. Ablations and controls that identify crosslayer, pooled-coordinate structure as the primary source of predictive signal; an occlusion analysis of where the classifier draws that signal; transfer experiments across datasets, tasks, generators, and model scale (with clearly reported near-chance results); and calibration, selectiveprediction, and cost analyses.
Summarization factuality. Summary factuality is judged against the source document and is known to be graded and multi-dimensional rather than binary (Maynez et al., 2020). We use MiniCheck sentence-level verification for CNN/DailyMail (Tang et al., 2024) and assign one factual or non-factual label to each summary. This supports the same correctness-prediction interface for long-form generation, although summaries near the label boundary remain difficult.
Related Work Sampling-based black-box UQ. Semantic entropy samples several answers to the same query, clusters them by bidirectional entailment, and computes entropy over the resulting meaning classes (Kuhn et al., 2023; Farquhar et al., 2024). It needs no access to model internals, but each scored query requires N sampled generations (here N =10) plus entailment inference.
Positioning. ActMap keeps what each family discards: it scores the single produced answer from one pass (vs. sampling), reads the pre-projection trajectory (vs. grey-box), preserves the depth axis and summarizes evolution over generated tokens (vs. single-vector probes), and exposes a broad structured summary for a learned classifier to mine rather than committing to one signal (Elhage et al., 2022). Viewed as a monitor, ActMap extends work that reads safety-relevant signals directly from internal activations (Burns et al., 2023). Table 2 places every baseline in this taxonomy.
Token-probability (grey-box) methods. Sequence perplexity and mean token entropy summarize the output distribution of a single generation; P(True)-style self-evaluation elicits the probability the model assigns to its own answer being correct (Kadavath et al., 2022). LOS-Net learns a detector over the full sequence of next-token distributions (BarShalom et al., 2026), evidence that learned functions of output distributions can outperform hand-crafted statistics. All of these observe only the model’s final projection onto the vocabulary.
Method Generation-Time Trajectory Let a decoder-only transformer with L layers and hidden size D generate an answer of T tokens. During decoding we record, via forward hooks on every layer, the hidden state of each newly generated token, giving a trajectory
Single-vector white-box probes. Linear probes on one hidden state can recover truthfulness information (Azaria
H ∈ RL×T ×D , 2
(1)
Table 1: Datasets, pre-balancing split sizes, and label types.
where Hℓ,t is the layer-ℓ representation of generated token t. To bound memory, each hidden vector is reduced online from D to D′ =128 coordinates by contiguous adaptive average pooling (each pooled coordinate is the mean of a fixed contiguous block of hidden dimensions), so the stored trajectory is L × T × 128. Generation itself is unchanged: one decoding pass in vLLM (Kwon et al., 2023), no extra samples.
Dataset
Rows
Test Label
TriviaQA (no ctx.) 50,000 5,051 answer correctness NQ-Open 50,000 5,064 answer correctness GSM8K (direct) 8,792 879 numeric correctness CNN/DailyMail 50,000 5,000 summary factuality
Training minimizes binary cross-entropy on maps with binary correctness labels, using AdamW (learning rate 10−3 , weight decay 0.05), cosine decay with 5 warm-up epochs, at most 80 epochs with early stopping on validation AUROC (patience 20), batch size 64, Gaussian input noise (σ=0.08) and mixup (α=0.2), and three seeds {42, 123, 456}. Reported probabilities are raw sigmoid outputs; deployment at a base rate different from the training distribution may require prior correction (see Calibration). Scoring a stored map is a single forward pass of the small network; no additional LLM call is made.
Activation-Map Construction The trajectory varies in T (answers have different lengths) and, across models, in L and D. ActMap converts it into a ′ ′ fixed tensor M ∈ RC×L ×D with C=12, L′ =32, D′ =128 in three steps (Figure 1). (1) Temporal statistics. The token axis is summarized by twelve channels, each a function RT → R applied independently at every (layer, pooled-coordinate) location: segment means (4: means over the four consecutive quarters of the answer, locating activation mass in answer time), final states (2: last-token state and mean over the final eight tokens, the information a last-token probe would see), dispersion (2: standard deviation and maximum over tokens), drift and slope (2: last-minus-first difference and least-squares slope against token index), and magnitude and dynamics (2: perlayer RMS norm, broadcast over coordinates, and mean absolute token-to-token difference). Every channel is a statistic over the whole token axis, so the output is independent of T by construction; one-token answers set the dispersion, slope, and dynamics channels to zero. (2) Layer pooling. The layer axis is adaptively averagepooled to L′ =32 rows, aligning models with different depths (L ∈ {32, 36, 64} here) onto a common axis while preserving depth ordering. (3) Normalization. Each channel is standardized to zero mean and unit variance over its 32 × 128 entries, so the classifier sees spatial patterns within a channel rather than raw magnitudes, and channels are on a common scale. Maps are stored in float16 (96 KiB per generation). We deliberately avoid interpreting individual cells; the representation’s role is to preserve where in depth, where in the pooled hidden space, and when in answer time activity differs between correct and incorrect generations.
Experimental Setup Tasks, Models, and Generation We evaluate on four datasets covering three task families and two output-length regimes (Table 1): TriviaQA (no context) and NQ-Open for short-answer factual QA (Joshi et al., 2017; Kwiatkowski et al., 2019), GSM8K for mathematical problem solving in direct-answer mode (Cobbe et al., 2021), and CNN/DailyMail for long-form summarization factuality (See et al., 2017). Generators are Qwen3-8B (Qwen Team, 2025), Llama-3.1-8B-Instruct (Llama Team, AI @ Meta, 2024), and Mistral-7B-Instruct-v0.3 (Jiang et al., 2023); Qwen3-32B (64 layers) is used for an in-domain and model-scale transfer study on the three short-form tasks. Primary answer generation is greedy (temperature 0) in vLLM with a 4,096-token context, 32 new tokens for short-answer tasks and GSM8K (direct answer, no explicit reasoning), and 384 for summaries. In total, the study produces more than 476,000 generations with captured trajectories.
Labels and Splits For TriviaQA and NQ-Open, a response is correct if its normalized answer exactly matches a normalized gold alias or attains token F1 of at least 0.8 against any alias; GSM8K uses exact numeric match. CNN/DailyMail summaries are split into sentences and each sentence is verified against the source article with MiniCheck (Flan-T5-Large); a summary is labeled factual iff every sentence is supported at threshold 0.5. Splits are disjoint by source key. Because per-model accuracy varies, all supervised training and all reported metrics use per-(dataset, model, split) balanced indices with equal numbers of correct and incorrect generations. All supervised detectors (ActMap, TAD, and ACT-ViT) train on the same balanced training rows, use the same balanced validation
Correctness Classifier The detector fθ maps a tensor M to an estimated correctness probability p(correct) = σ(fθ (M )); the uncertainty score of a generation is u = 1 − p(correct). We instantiate fθ as a compact Vision Transformer (Dosovitskiy et al., 2021) over 4 × 16 patches of the map (2.4M parameters; six pre-norm blocks, embedding width 192, six heads, a class token) with factorized row/column positional embeddings, so that depth and coordinate identity are preserved; remaining details appear in the appendix. The representation is not tied to this choice (see Ablations); all main-table results use the Vision Transformer. 3
LLM prompt → answer
capture every layer & token
12 temporal stats, pool, z-score
single forward pass
hidden trajectory L×T ×D
Vision Transformer classifier
σ
p(correct)
ActMap 12×32×128
during generation (single pass)
after generation (no extra LLM calls)
Figure 1: The ActMap pipeline. The LLM generates its answer normally while hooks capture the per-layer, per-token hiddenstate trajectory; temporal statistics, pooling, and per-channel standardization compress it into a fixed 12 × 32 × 128 map, from which a compact Vision Transformer classifier reads p(correct). Channels shown: token standard deviation, temporal slope, first segment mean (real Qwen3-8B GSM8K generation). rows for their respective model-selection protocols, and are evaluated on the identical balanced test rows.
dataset) pairs against the sampling, token-probability, attention, and embedding baselines. Its mean AUROC over those twelve pairs is .825, against .790 for TAD and .741 for the best training-free baseline (MTE).
Baseline Taxonomy Table 2 summarizes all methods. The black-box baseline is Semantic Entropy (10 sampled generations at temperature 1.0, top-p 0.9, clustered by bidirectional DeBERTaMNLI entailment). Grey-box baselines are sequence perplexity, mean token entropy (MTE), and P(True) self-evaluation. White-box baselines are TAD and ACT-ViT (supervised, like ActMap), RAUQ, and EigenScore (over 10 sampledresponse embeddings). We follow each baseline’s published protocol; for ACT-ViT this includes the authors’ full 24configuration architecture sweep on the shared balanced splits and seeds (details in the appendix).
Baseline categories behave consistently. Grey-box statistics are strong on short-answer QA, where a wrong answer usually coincides with a diffuse output distribution, but weaker on direct-answer mathematical correctness, where an incorrect numeric answer may still be produced with a concentrated token distribution: the best grey-box AUROC on GSM8K trails ActMap by .031–.087. Sampling-based Semantic Entropy pays for its ten extra generations without matching single-pass grey-box statistics on these balanced splits, consistent with its score reflecting question ambiguity rather than answer-specific reliability. Among the nontensor white-box baselines, supervised TAD is consistently the strongest; unsupervised RAUQ and EigenScore sit between grey-box statistics and the supervised methods.
Metrics and Evaluation Axes We report AUROC (↑; ranking quality across all thresholds), AUPRC (↑; precision–recall performance, with a 0.5 baseline on the balanced splits), and 10-bin expected calibration error (ECE, ↓; the gap between predicted confidence and empirical correctness) (Guo et al., 2017). Supervised detectors report the mean over seeds {42, 123, 456}; the maximum seed standard deviation of ActMap AUROC is 0.026 (GSM8K, smallest split) and below 0.010 on all splits above 1,700 test rows. The evaluation separates (i) in-domain performance, training and testing on the same (dataset, model); (ii) cross-dataset transfer within a model; (iii) cross-task transfer between short-form correctness and long-form factuality; (iv) cross-generator transfer; and (v) model-scale transfer (Qwen3-8B↔32B).
Long-form factuality is the hardest regime. On CNN/DailyMail, absolute scores drop for all methods. Summary factuality is graded: a mostly supported summary may contain one unsupported clause, making factual/non-factual labels difficult near the boundary. Training-free methods are barely better than chance here, and Semantic Entropy is at chance, since whole-response equivalence clustering is poorly matched to long-form output; only the supervised methods extract usable signal, with ActMap ahead of TAD on all three models, within .012 AUROC of ACT-ViT on every model, and keeping ECE below .05 throughout.
Compression Preserves the Dense-Tensor Signal
Main Results
We evaluate ACT-ViT with its complete published 24configuration sweep per pair rather than as a fixed-score baseline. Ranking quality is nearly identical: ActMap leads on seven of twelve pairs (one decided beyond three decimals) and ACT-ViT on five. Mean AUROC is .825 versus .823; the largest gaps are comparable (.048 and .044).
Table 3 reports the in-domain matrix over all (model, dataset, method) triplets. ActMap leads every non-tensor baseline. ActMap attains the highest AUROC and AUPRC on all twelve (model, 4
Table 2: Method taxonomy. “Extra gen.” counts generated responses beyond the scored answer. Per-answer latencies, which are specific to our implementation and hardware stack, are reported in the appendix. Method
Access
Extra gen.
Semantic Entropy (Kuhn et al., 2023)
black-box
10
Perplexity Mean token entropy P(True) (Kadavath et al., 2022)
grey-box grey-box grey-box
TAD (Vazhentsev et al., 2025) RAUQ (Vazhentsev et al., 2026) EigenScore (Chen et al., 2024) ACT-ViT (Bar-Shalom et al., 2025) ActMap (ours)
white-box white-box white-box white-box white-box
Information used
Supervision
Stored artifact
sampled answer texts
none
samples
0 0 1 short pass
output token probabilities output token distributions self-evaluation token probability
none none none
– – –
0 0 10 0 0
attention + token probabilities attention + token probabilities sampled-response embeddings padded activation tensor full hidden-state trajectory
trained none none trained trained
attn. features attn. stats embeddings activations 96 KiB map
Table 3: In-domain results on balanced test splits (AUROC ↑ / AUPRC ↑ / ECE ↓; mean over three seeds for trained methods). Bold marks the best value for that pair. ECE for methods without native probabilities uses a fixed monotone score-to-probability mapping. CNN/DailyMail EigenScore covers the label subset with sampled-response embeddings. Model
Method
TriviaQA
NQ-Open
GSM8K
CNN/DM
AUROC AUPRC ECE AUROC AUPRC ECE AUROC AUPRC ECE AUROC AUPRC ECE Qwen3-8B
ActMap (ours) ACT-ViT Semantic Entropy Perplexity Mean token entropy P(True) TAD RAUQ EigenScore Llama-3.1-8B ActMap (ours) ACT-ViT Semantic Entropy Perplexity Mean token entropy P(True) TAD RAUQ EigenScore Mistral-7B ActMap (ours) ACT-ViT Semantic Entropy Perplexity Mean token entropy P(True) TAD RAUQ EigenScore
.887 .861 .772 .837 .847 .817 .837 .813 .742 .880 .832 .802 .830 .844 .792 .849 .827 .713 .859 .885 .711 .777 .787 .774 .788 .761 .681
.883 .856 .678 .836 .841 .773 .814 .771 .694 .878 .833 .744 .834 .843 .780 .839 .799 .692 .851 .881 .614 .763 .767 .730 .748 .724 .676
.028 .059 .397 .116 .186 .367 .126 .329 .124 .039 .072 .386 .169 .171 .316 .099 .350 .125 .048 .066 .445 .070 .144 .390 .141 .329 .418
.838 .855 .752 .761 .760 .729 .823 .792 .730 .861 .848 .758 .795 .796 .707 .824 .817 .782 .905 .911 .777 .853 .850 .731 .891 .879 .855
.802 .827 .682 .753 .751 .691 .782 .740 .674 .840 .823 .719 .793 .792 .707 .798 .782 .754 .877 .878 .699 .857 .854 .663 .856 .858 .828
.057 .088 .382 .079 .116 .396 .305 .335 .140 .066 .098 .368 .150 .155 .274 .214 .355 .181 .041 .124 .410 .050 .172 .426 .337 .341 .298
.791 .835 .719 .744 .755 .685 .761 .750 .714 .931 .912 .575 .812 .827 .844 .845 .770 .535 .782 .769 .728 .751 .750 .575 .771 .757 .597
.781 .844 .651 .753 .758 .704 .734 .692 .718 .939 .906 .582 .772 .793 .867 .819 .684 .527 .813 .763 .673 .745 .745 .622 .768 .759 .585
.126 .100 .359 .134 .142 .364 .118 .335 .157 .086 .061 .228 .177 .233 .079 .076 .383 .298 .146 .159 .288 .142 .131 .504 .153 .343 .433
.705 .717 .510 .579 .582 .534 .684 .583 .573 .769 .766 .483 .576 .579 .557 .732 .580 .606 .689 .689 .495 .516 .517 .577 .676 .521 .556
.692 .703 .516 .569 .570 .506 .668 .579 .555 .754 .753 .488 .560 .559 .542 .716 .565 .587 .671 .664 .510 .514 .513 .556 .661 .518 .541
.033 .083 .367 .028 .061 .497 .058 .357 .469 .044 .103 .418 .054 .111 .405 .079 .366 .384 .039 .077 .413 .041 .089 .500 .082 .363 .208
Within-task transfer is strong: TriviaQA → NQ-Open stays close to matched in-domain training and ahead of the frozen TAD baseline, though TAD transfers with better calibration. Beyond the task boundary the picture changes: transfer from pooled QA to GSM8K is weak and model-dependent, and both directions between short-answer correctness and summary factuality are near chance. Cross-generator transfer trains on the pooled maps of two generators with the dataset held fixed and evaluates on the held-out third (a macro mean over four datasets, three targets, and three seeds); it is near chance. Model scale behaves the same way: transfer between Qwen3-8B and Qwen3-32B is near chance in both directions despite the shared model family and tokenizer, while a detector trained in domain on Qwen3-32B is at least as strong as at 8B (macro .868 AUROC, with the largest gain on GSM8K, .791 → .870). These results concern transfer of the learned decision boundary. Shared map geometry alone does not align boundaries across tasks, generators, or scales.
ActMap reaches this parity from 49,152 values per generation against the 3.3M values of ACT-ViT’s dense 8×100×4096 tensor, with one fixed classifier for all pairs while the sweep selects different architectures for different pairs, and with lower ECE on ten of twelve pairs (mean .063 vs. .091). These results indicate that temporal-statistic compression preserves the uncertainty signal in the dense activation tensor while producing a fixed, generator-agnostic map.
Transfer and Generalization A detector is deployed either in-domain, trained for a fixed generator and task (the previous section), or in transfer, where target labels are unavailable and a detector trained elsewhere must generalize. We quantify the latter along four axes with the classifier frozen after source training (Table 4). 5
Table 4: Frozen-detector transfer (AUROC ↑ / AUPRC ↑ / ECE ↓ on balanced target test splits; no target-domain training). Rows are macro means over their constituent target pairs and three seeds; model-scale and in-domain-scale rows average over the three short-form datasets. Axis
Source → target
little beyond the last-token channels in both output-length regimes (CNN/DailyMail × Qwen3-8B, summaries up to 384 tokens: .704 vs. .705), and no single channel group is critical. The temporal channels are thus a compact mechanism for reducing variable-length trajectories to a fixed shape; the predictive signal lies in the cross-layer, pooled-coordinate structure.
AUROC AUPRC ECE
TriviaQA → NQ-Open (3 models) frozen TAD, same transfer Cross-task TriviaQA+NQ → GSM8K TriviaQA+NQ → CNN/DM CNN/DM → TriviaQA+NQ Cross-generator leave-one-generator-out (12 targets) Model scale Qwen3-8B → 32B Qwen3-32B → 8B Scale (in-dom.) Qwen3-32B, short-form
Cross-dataset
.837 .804 .617 .512 .537 .510 .520 .517 .868
.821 .766 .645 .513 .532 .516 .518 .522 .860
.138 .053 .304 .463 .256 .337 .366 .403 .065
Does the pooling scheme matter? Yes. Permuting hidden coordinates before pooling preserves the marginal activation values but destroys the coordinate grouping; it is the most damaging representation variant, costing more than collapsing the map to a single final-layer vector. Random Gaussian projections of matched size also lose ground. Performance depends on the consistent coordinate grouping induced by contiguous pooling; dimension reduction alone does not preserve the signal.
Table 5: Ablations and controls on TriviaQA × Qwen3-8B (test AUROC, mean over three seeds; seed std ≤ .010 for every variant). Variants requiring re-captured trajectories use the retained class-balanced split intersection, on which the full configuration scores .886 (vs. .887 in Table 3). Configuration Full ActMap (12 ch., 32×128, Vision Transformer) Representation scope last-token channels only single mean-pooled final-layer state Pooling structure (controls) hidden-coordinate permutation before pooling Gaussian projection instead of pooling Construction choices any one channel group removed (worst–best of 5) 1 / 8 temporal segments (default 4) resolution 12 × 16 × 64 resolution 12 × 64 × 256 global normalization (not per-channel) Classifier on identical maps logistic regression (flattened) MLP (matched parameters) MLP (4× parameters) Training-set size 10% / 25% / 50% of train Sanity controls permuted labels (expect ≈.5) answer-length-only predictor
Does performance depend on the classifier architecture? No: logistic regression, a capacity-matched MLP, and a 4× MLP all perform comparably on identical maps, and the capacity-matched MLP in fact slightly outperforms the Vision Transformer; on this ablation setting, classifier choice has little effect relative to the representation. The remaining construction choices (segment count, map resolution, normalization) shift AUROC by at most .010.
AUROC .886 .879 .754 .713 .863 .881–.890 .889 / .889 .877 .886 .889
How much supervision is needed? On TriviaQA × Qwen3-8B, 10% of the training data (about 3,800 balanced examples; AUROC .815) beats Semantic Entropy and EigenScore; 25% (.855) beats every evaluated training-free baseline. As expected, a detector fit to permuted labels falls to chance, and an answer-length-only predictor stays well below the full map.
.877 .892 .893 .815 / .855 / .875 .510 .597
Depth-Wise Localization of Predictive Signal We localize the predictive signal the classifier uses within the map: on TriviaQA × Qwen3-8B, we occlude one depth-band × coordinate-band region at a time and re-evaluate the frozen detector (baseline .887 AUROC; occluding the entire map collapses it to .500). The classifier is most sensitive to midnetwork depth bands (occlusion drops of .003–.013), while the earliest and latest bands are individually more redundant; integrated-gradients and attention-rollout attributions agree on the same mid-depth concentration (Figure 2). This agreement localizes the detector’s signal without implying an explicit correctness representation in the generator, and is consistent with probing literature placing semantic and truthfulness information in intermediate layers (Azaria and Mitchell, 2023; Marks and Tegmark, 2024).
A new deployment therefore requires target-domain labels; otherwise the monitor degrades silently under shift, which is itself an oversight risk.
Ablations All ablations run on TriviaQA × Qwen3-8B with the maintable protocol and three seeds (Table 5); they answer four questions. Where does the gain come from? From the structure preserved across depth and pooled hidden coordinates; no single statistic explains it. Collapsing the map to one mean-pooled final-layer vector, the representation prior white-box probes use, costs .13 AUROC. A map built from the last-token channels alone recovers nearly all of the full map’s performance, as expected on short answers where the final state can summarize the preceding computation. Temporal summaries add
Calibration and Selective Prediction Abstention thresholds require calibrated probabilities, not only good ranking (Guo et al., 2017). ActMap’s raw sigmoid 6
Figure 2: Occlusion atlas for TriviaQA × Qwen3-8B (mean over three seeds): AUROC drop from zeroing one depth × coordinate band, with integrated-gradients (IG) and attention-rollout views of the same grid. All views concentrate in mid-depth bands. ACT-ViT are comparable (.432 and .427) against .476 for MTE.
Computational Cost Measurements use vLLM on one NVIDIA L40S with Qwen3-8B × TriviaQA; matched baseline latencies are in the appendix. Capture: forward hooks pool every layer online; hooked and unhooked decoding both averaged about 7 ms/prompt (five batches of 64), so overhead is within run-to-run variation. Storage: one 12 × 32 × 128 float16 map is 96 KiB (85–96× smaller than a 32-token float16 trajectory and 67× smaller than ACT-ViT’s dense tensor); 38k TriviaQA generations occupy 3.5 GiB as maps versus 235 GiB as dense tensors. Scoring: classifier inference takes .024 ms/map (batch 256), and training converges in under 20 GPU-minutes per seed. ACT-ViT takes .066 ms/answer with 2.1× the parameters and 25× the peak memory; its CNN/DailyMail configuration trains for about 3 GPU-hours per pair before the 24-configuration sweep. Semantic Entropy and EigenScore require 10 additional generations per answer.
Figure 3: Risk–coverage for Qwen3-8B: selective risk (error among retained answers) vs. coverage on shared balanced test rows. Markers: 80% and 90% coverage.
outputs are the best- or near-best-calibrated on short-answer QA and CNN/DailyMail. The exception is GSM8K, whose training split is an order of magnitude smaller: ActMap’s ECE rises to .086–.146 and ACT-ViT or MTE is better calibrated there (Table 3). Raw-score ECE comparisons favor trained probabilistic detectors by construction. As a diagnostic, we fit one scalar temperature per method on half of the Qwen3-8B × TriviaQA test predictions and report ECE on the other half: ActMap’s optimal temperature is ≈1.0 (ECE .027 → .024), while MTE needs strong sharpening (T =0.50) and still leaves ECE at .158.
Limitations and Ethical Considerations
Balanced splits also differ from deployment prevalence, so we simulate prevalence shift by class-conditionally resampling each balanced test set to the model’s natural test accuracy (100 replicates; temperature fit on 256 held-out target examples each). Where natural accuracy is near the balanced regime (TriviaQA, .53–.66; CNN/DailyMail, .54–.63), raw ECE stays at .02–.09 and selective prediction remains useful (18–38% coverage at 5% risk on TriviaQA). Where accuracy collapses (NQ-Open, .17–.23; Mistral-7B GSM8K, .07), ECE rises to .15–.24 and temperature scaling does not repair it: the error is a prior shift requiring prior correction, not a sharpness error.
Limitations. ActMap requires white-box access to generation-time hidden states: self-hosted or providerinstrumented models, not closed APIs. It is supervised: each deployment regime needs its own labeled generations. Label sources are imperfect (alias matching misses paraphrases; MiniCheck inherits its judge’s errors), and the score is correlational: high confidence means the trajectory resembles previously correct generations. It does not establish truth. Balanced-split evaluation differs from deployment prevalence (see Calibration). Our experiments use greedy decoding on English tasks; robustness across decoding strategies and temperatures is left to future work. In-domain results cover 7–8B generators on four tasks and Qwen3-32B on three short-form tasks; ablations cover two pairs.
The deployment-relevant summary is risk–coverage behavior (Geifman and El-Yaniv, 2017): how much generation volume can be cleared automatically at a target error rate, concentrating human review on the remainder (Figure 3). On Qwen3-8B TriviaQA, ActMap retains 18.3% coverage at 5% risk versus 14.3% for ACT-ViT and 6.2% for MTE. CNN/DailyMail is substantially harder: no method provides useful coverage at 5% risk; at 80% coverage ActMap and
Ethical considerations. A correctness score can reduce overreliance on fluent wrong answers, but a miscalibrated score can become a safety veneer. Under prevalence shift on NQ-Open and Mistral-7B GSM8K, ECE reaches .15–.24 and temperature scaling does not repair it. The score should 7
guide verification rather than replace it. Stored maps require the source text’s access controls and retention limits and provide a re-scorable audit trail for consequential answers.
Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. arXiv:2110.14168. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of the 9th International Conference on Learning Representations (ICLR), 2021.
Conclusion ActMap converts generation-time hidden activations into a fixed-size representation over depth and pooled hidden coordinates. Trained in-domain, its lightweight classifier outperforms every evaluated non-tensor baseline on all twelve 7– 8B pairs and matches dense activation-tensor learning from a 67× smaller, generator-agnostic artifact with better calibration. Capture adds no measurable overhead; results survive classifier swaps and hold in-domain at 32B scale. The main limitation is transfer: decision boundaries do not yet align across tasks, generators, or scales. Code, replication materials, and the 476,372-map dataset will be released on Hugging Face upon publication.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, et al. Toy models of superposition, 2022. arXiv:2209.10652.
References
Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. In Advances in Neural Information Processing Systems (NeurIPS), volume 30, pages 4878–4887, 2017.
Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625–630, 2024.
Amos Azaria and Tom Mitchell. The internal state of an LLM knows when it’s lying. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 967– 976, 2023.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning (ICML), pages 1321–1330, 2017.
Guy Bar-Shalom, Fabrizio Frasca, Yaniv Galron, Yftah Ziser, and Haggai Maron. Beyond token probes: Hallucination detection via activation tensors with ACT-ViT. In Advances in Neural Information Processing Systems (NeurIPS), 2025.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7B, 2023. arXiv:2310.06825.
Guy Bar-Shalom, Fabrizio Frasca, Derek Lim, Yoav Gelberg, Yftah Ziser, Ran El-Yaniv, Gal Chechik, and Haggai Maron. Beyond next token probabilities: Learnable, fast detection of hallucinations and data contamination on LLM output distributions. In Proceedings of the AAAI Conference on Artificial Intelligence, 2026.
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1601–1611, 2017.
Samuel R. Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamilė Lukošiūtė, Amanda Askell, et al. Measuring progress on scalable oversight for large language models, 2022. arXiv:2211.03540.
Saurav Kadavath, Tom Conerly, Amanda Askell, et al. Language models (mostly) know what they know, 2022. arXiv:2207.05221. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In Proceedings of the 11th International Conference on Learning Representations (ICLR), 2023.
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. In Proceedings of the 11th International Conference on Learning Representations (ICLR), 2023.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, MingWei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466, 2019.
Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. INSIDE: LLMs’ internal states retain the power of hallucination detection. In Proceedings of the 12th International Conference on Learning Representations (ICLR), 2024. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Łukasz Kaiser, Matthias Plappert, 8
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP), pages 611–626, 2023. Llama Team, AI @ Meta. The Llama 3 herd of models, 2024. arXiv:2407.21783. Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. In Proceedings of the Conference on Language Modeling (COLM), 2024. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1906–1919, 2020. Qwen Team. Qwen3 arXiv:2505.09388.
technical
report,
2025.
Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1073–1083, 2017. Liyan Tang, Philippe Laban, and Greg Durrett. MiniCheck: Efficient fact-checking of LLMs on grounding documents. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8818–8847, 2024. Artem Vazhentsev, Ekaterina Fadeeva, Rui Xing, Gleb Kuzmin, Ivan Lazichny, Alexander Panchenko, Preslav Nakov, Timothy Baldwin, Maxim Panov, and Artem Shelmanov. Unconditional truthfulness: Learning unconditional uncertainty of large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 35673– 35694, 2025. Artem Vazhentsev, Lyudmila Rvanova, Gleb Kuzmin, Ekaterina Fadeeva, Ivan Lazichny, Alexander Panchenko, Maxim Panov, Mrinmaya Sachan, Preslav Nakov, Timothy Baldwin, and Artem Shelmanov. Efficient hallucination detection for LLMs using uncertainty-aware attention heads. In Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026. Zeguan Xiao, Diyang Dou, Boya Xiong, Yun Chen, and Guanhua Chen. Enhancing uncertainty estimation in LLMs with expectation of aggregated internal belief. In Proceedings of the AAAI Conference on Artificial Intelligence, 2026.
9
Table 6: Why each hand-designed channel family is included. These working hypotheses are evaluated by the grouped ablations in the main paper; they do not prescribe what the classifier must use.
Technical Appendix This appendix gives the implementation details for the representation and classifier, specifies the ACT-ViT reproduction protocol, and reports measured costs. It also records the software environment and retained artifacts. The main paper contains all claims and results needed to assess the paper.
A
Channel family
Intended observable
Segment means
Where activation mass lies during the answer and how its depth profile changes over time. Final state and The endpoint representation and its rewindow cent context, comparable to last-token and pooled-state probes. Dispersion Variation within an answer and rare activation excursions that may separate stable from conflicted or unstable generations. Drift and slope Whether activation patterns accumulate, resolve, or change direction during the answer. Layer RMS Layer-wise activation energy independent of coordinate, retaining a coarse depth profile. Temporal differ- Local token-to-token changes that a global ences mean, endpoint, or linear slope can miss.
Exact Activation-Map Construction
For one generated answer, let H ∈ RL×T ×D contain the hidden state from every transformer block and generated token. Hidden coordinates are first reduced to D′ = 128 by one-dimensional adaptive average pooling over contiguous coordinate intervals. Denote the resulting tensor by X ∈ ′ RL×T ×D . The twelve channels below are computed independently for every layer ℓ andP pooled coordinate d. For clarity, xt = Xℓ,t,d and x̄ = T −1 t xt . 1. Four segment means. The token indices are divided by integer boundaries obtained from linspace(0, T, 5). Each channel is the mean within one consecutive segment; an empty segment for a very short answer is replaced by its nearest valid one-token interval.
The stored map M has shape 12 × 32 × 128 and float16 size 96 KiB. All statistics are computed in float32 before storage.
2. Final state. xT −1 . 1 3. Final-window mean. min(8,T )
A.1
PT −1
t=max(0,T −8) xt .
4. Temporal standard deviation. The sample standard deviation over tokens. It is zero for T = 1.
We chose a small, hand-designed channel set and fixed it before the final benchmark evaluation. We do not claim that this set is unique or optimal. The channels summarize four aspects of a generated answer: when a pattern occurs, endpoint state, global dispersion, and temporal change. Segment means and endpoint channels retain coarse phase information. Standard deviation and maximum describe the distribution. Drift and slope capture long-range direction, while temporal differences capture local movement. All statistics use the same captured hidden states, require no extra model call, and produce a fixed-size output for answers of different lengths.
5. Temporal maximum. maxt xt . 6. Endpoint drift. xT −1 − x0 . 7. Least-squares slope. P (t − t̄)(xt − x̄) tP , max{ t (t − t̄)2 , 10−8 }
(2)
set to zero for T = 1. 8. Layer RMS. Unlike the other channels, this value is computed once per layer √ over all tokens and pooled coordinates: ∥Xℓ,:,: ∥2 / T D′ . It is then broadcast across the coordinate axis.
ActMap tests whether a temporal statistic can carry different information at different depths and coordinates. The representation therefore keeps the layer and pooled-coordinate axes. In the grouped ablations, removing one channel family changes AUROC only slightly, whereas collapsing the depthresolved map causes a much larger loss. This result supports the full structured map but does not show that any single channel family is essential.
9. Mean absolute temporal difference. PT −1 (T − 1)−1 t=1 |xt − xt−1 |, set to zero for T = 1. Stacking the channels gives S ∈ R12×L×128 . Contiguous adaptive average pooling maps the ordered layer axis to 32 rows. Finally, every channel is standardized independently within each example: Mc =
Sc − µ(Sc ) . max{std(Sc ), 10−6 }
Design Rationale for the Twelve Channels
The same per-example standardization is applied to every retained channel. It removes absolute scale differences between channels and examples but keeps the relative pattern across layers and pooled coordinates. The globalnormalization control in the main paper tests whether this pattern adds information beyond the overall activation scale.
(3)
10
A.2
The parameter accounting is: 147,648 for patch projection, 3,456 for class and position parameters, 2,225,664 across the six encoder blocks, 384 for the final layer normalization, and 24,833 for the readout head. The sigmoid of the scalar logit is the reported estimated correctness probability.
Formal Representation and Computational Properties
For a trajectory H ∈ RL×T ×D , let PD : RD → R128 denote contiguous adaptive average pooling over hidden coordinates. Applying it independently to every (ℓ, t) gives Xℓ,t,: = PD (Hℓ,t,: ),
X ∈ RL×T ×128 .
(4)
C
For channel c, let ϕc be one of the twelve scalar trajectory statistics listed above. The unpooled channel maps are Sc [ℓ, d] = ϕc (Xℓ,0:T −1,d ) ,
S ∈ R12×L×128 .
Each (generator, dataset) detector is trained independently on its balanced training split with the hyperparameters in Table 8. Validation and test splits are also balanced and disjoint by source key. We use binary cross-entropy with logits. The positive-class weight is N− /N+ , which equals 1.0 for every balanced training split in the main experiments. Gaussian noise is sampled independently for every training map before batching. For each ActMap training batch, a single λ ∼ Beta(0.2, 0.2) mixes maps with a random permutation; the loss is the corresponding convex combination of the two binary losses. Validation and test maps receive no augmentation. The checkpoint is replaced whenever validation AUROC strictly improves; ties retain the earlier epoch. Training stops after 20 epochs without improvement. The selected checkpoint is then evaluated exactly once on the test split. Main-table values are arithmetic means over the three seeds; ECE uses ten equal-width probability bins. The learning-rate multiplier at zero-indexed epoch e is
(5)
Let AL be ordered adaptive average pooling from L rows to 32 rows, and let N denote independent per-channel spatial standardization. The stored ActMap is therefore the deterministic operator M = Φ(H) = N (AL (S)) ∈ R12×32×128 .
(6)
The learned detector is a separate function p̂ = fθ (Φ(H)) ,
p̂ ∈ [0, 1],
(7)
trained to estimate the correctness label for a fixed generator and task. Φ defines the reusable representation. We do not assume that fθ or its decision boundary transfers without target-domain supervision. The output shape is independent of L, T , and D for a decoder-only transformer whose hidden states can be captured. The map keeps layer and coordinate order, so the classifier can use their relative structure. During generation, the implementation reduces each hidden state to 128 pooled coordinates. It applies the twelve statistics after the pooled sequence is complete. The stored map has 12 · 32 · 128 values regardless of output length, instead of O(LT D) full hiddenstate values. Because segment boundaries depend on the final token count, the current implementation temporarily keeps the pooled O(LT · 128) sequence until generation ends. The ablations test whether this compression retains enough information for correctness prediction.
B
Optimization and Model Selection
η(e) =
C.1
( (e + 1)/5,
e < 5, h
0.01 + 0.495 1 + cos
e−5 π 80−5
i
,
e ≥ 5.
(8)
ACT-ViT Reproduction Protocol
We evaluate ACT-ViT using the authors’ released architecture. Each generated trajectory is converted to the published Leff = 8 by Neff = 100 dense activation tensor. As in the released preprocessing, the output-token axis is sliced or zero-padded to the fixed Nmax = 100 positions, and the zero-padded layer axis is max-pooled to eight groups. We retain the released model-specific linear-adapter branch and zero-padding rule. We search the full 24-configuration grid used by the authors: hidden dimensions {128, 1024}, transformer depths {1, 3}, weight decays {1, 10−3 }, and patch sizes {(1, 1), (8, 1), (4, 2)}. All configurations use four attention heads, dropout 0.3, learning rate 10−3 , 15 epochs, and the published batch size 128. Optimization uses AdamW and binary cross-entropy, with a cosine schedule and a 10% step warm-up. The released patience of 30 exceeds the 15epoch sweep horizon, so every configuration runs for all 15 epochs. We select the configuration by validation AUROC for seed 42, retrain that configuration with seeds 123 and 456, and report the three-seed mean. For each seed, the checkpoint is replaced whenever validation AUROC strictly improves; ties retain the earlier epoch. We evaluate the selected checkpoint once on the test split. This matches the authors’ test metric at the best-validation epoch; the test split is not consulted
Primary Classifier
The main experiments use the compact transformer in Table 7. A convolution whose stride equals its kernel size forms non-overlapping learned patches. It turns the 32 × 128 map into an 8 × 8 grid of 64 tokens. A learned classification token is prepended. Learned row and column embeddings preserve both spatial axes without a full 64-position table. Attention dropout is 0.1. MLP and readout dropout are 0.3, while positional dropout is 0.15. Stochastic-depth probability increases linearly from 0 in the first block to 0.05 in the sixth. The patch-projection weights use Xavier uniform initialization. The class token and factorized positional embeddings use a truncated normal distribution with standard deviation 0.02; remaining linear and normalization layers use PyTorch defaults. 11
Table 7: Exact classifier architecture. The total number of trainable parameters is 2,401,985. Component
Configuration
Output
Input Patch projection Position Encoder Attention Block MLP Readout
12 channels Conv2d, 4 × 16, stride 4 × 16 class + factorized 8 row/8 column 6 pre-norm blocks 6 heads, head width 32 192 → 576 → 192, GELU LN, 192 → 128 → 1, GELU
12 × 32 × 128 64 × 192 65 × 192 65 × 192 65 × 192 65 × 192 scalar logit
Table 8: Training hyperparameters used by every main-table ActMap detector.
Table 9: In-domain ActMap results for Qwen3-32B on balanced test splits (mean over three seeds).
Hyperparameter
Value
Dataset
AUROC
AUPRC
ECE
Optimizer Initial learning rate Weight decay Batch size Maximum epochs Warm-up Post-warm-up schedule Early-stopping patience Selection metric Gradient-norm clipping Gaussian input noise Mixup Seeds
AdamW 10−3 0.05 64 80 5 epochs, linear cosine, final multiplier 0.01 20 epochs validation AUROC 1.0 σ = 0.08 α = 0.2 42, 123, 456
TriviaQA NQ-Open GSM8K
.893 .841 .870
.892 .816 .873
.056 .057 .082
C.2
Transfer Scope and Trainable Adapters
Transfer freezes the detectors and uses no target labels or adapter updates. This is stricter than in-domain training. A shared input shape does not guarantee the same correctness boundary across tasks or generators. The in-domain ACTViT comparison trains its published model-specific adapter; transfer freezes that adapter. Its near-chance results measure zero-shot reuse. Lightweight target adaptation may improve them, but remains outside this study.
during training or model selection. ACT-ViT receives the same complete balanced training, validation, and test rows as the other supervised methods; we do not apply the upstream 10,000-example preprocessing cap. Generators, saved responses, correctness labels, supervision, splits, and random seeds are fixed across methods. Only the detector and its published model-selection protocol differ.
D
Reproduction Environment and Retained Artifacts
The experiments used Python 3.12 with package versions pinned in a lockfile. Each run manifest records the operating system, CUDA and PyTorch versions, visible accelerator, configuration, and input-artifact identifiers. We ran all experiments on NVIDIA L40S GPUs with 48 GiB of device memory. Experiments with 7–8B models used one L40S; Qwen3-32B experiments used two. The cloud VMs used x86-64 Ubuntu Noble images. CPU model and host memory varied across generation and training jobs.
Measured cost. Under the scoring protocol of the main paper (batch 256, 300 timed repeats, median per-item latency, single NVIDIA L40S), the validation-selected TriviaQA × Qwen3-8B configuration (hidden dimension 128, depth 1, weight decay 10−3 , patch size 4 × 2; 5.16M parameters) scores at .066 ms per answer with a peak of 4,949 MiB of CUDA memory for one batch; the ActMap classifier (2.40M parameters) measured identically scores at .024 ms with a 197 MiB peak. One packed ACT-ViT input tensor (8 × 100 × 4096, float16) is 6.25 MiB against ActMap’s 96 KiB map. Training the selected configuration for all three seeds took 1.4–73.7 GPU-minutes per short-form pair and 157.8–195.9 GPU-minutes per CNN/DailyMail pair on one L40S, excluding the 24-configuration sweep that selects it. The corresponding three-seed ActMap training takes about 54–57 GPU-minutes per TriviaQA pair and about 63 GPUminutes per CNN/DailyMail pair; unlike ACT-ViT, it uses one fixed configuration and no architecture sweep.
D.1
Determinism
The reproduction runner fixes the relevant random seeds and uses deterministic PyTorch settings. These controls support repeated runs on the recorded software and hardware stack; exact bitwise agreement across stacks is not claimed. Before test aggregation, each seed retains the full run configuration, the classifier constructor arguments, the validation-selected model state, the per-epoch loss and validation history, and the aggregate test metrics with one score and probability per balanced test row. The configuration in12
Table 10: Measured per-answer latency beyond producing the answer, on our implementation and hardware stack. Method
Added latency
Perplexity / MTE P(True) Semantic Entropy TAD RAUQ EigenScore ACT-ViT ActMap
negligible post-processing 17.2 ms 85 ms + 50–291 ms NLI 125.6 ms 146.2 ms 85 ms + 329 ms cov. .066 ms score no measurable capture; .024 ms score
cludes split counts, optimizer, augmentation, determinism settings, and source-artifact metadata. The three checkpoints for each (generator, dataset) pair are the frozen detectors used in the transfer experiments.
E
Figure 4: Added per-answer latency (log scale; our stack, Table 10) against mean in-domain AUROC over the twelve 7–8B model–dataset pairs of the main paper. Latencies are implementation- and hardware-specific.
Classifier Controls 125.6 ms overall. Across 512 scores, feature extraction averages 110.6 ms and readout 14.9 ms. Total latency has a median of 120.8 ms and a 95th percentile of 158.9 ms. The experiment archive retains the raw timings and software versions.
The classifier ablation keeps the maps and data splits fixed. Logistic regression maps all 49,152 entries to one logit (49,153 parameters). The matched MLP flattens the same map, applies one GELU hidden layer and dropout, and emits one logit. Its hidden width matches the primary classifier’s parameter budget. The 4× control uses four times that budget. Every architecture receives the same input information, so the comparison tests classifier capacity and architecture without changing the representation.
F
Measured Per-Answer Latencies
Table 10 reports latency after the scored answer has been produced on our stack (vLLM, one NVIDIA L40S, Qwen38B × TriviaQA, 32 new tokens). The values can change with the serving stack, kernels, batch size, and output length. For Semantic Entropy and EigenScore, sampling cost is the measured 8.5 ms marginal decode time times 10 samples. NLI cost increases with output length; parallel sampling can reduce wall-clock time by using more memory. Figure 4 relates these measurements to mean ranking quality.
G
TAD Latency Protocol
The TAD benchmark restores the fitted official two-stage checkpoint for Qwen3-8B on TriviaQA. We time the evaluation path used in the paper: teacher-forced BF16 inference with eager attention, extraction of top-10 all-layer/all-head attention and token probabilities, and the fitted Ridge readout. Loading and training are excluded. After eight warm-up rows, we score 128 balanced test rows twice on each of two L40S GPUs. Each GPU runs an isolated single-GPU workload. The two means are 126.25 and 124.86 ms, or 13