Which Signals of Code Correctness Survive Distribution Shift? Hidden-State and Causal Predictors of Program and Verdict Correctness

Which Signals of Code Correctness Survive Distribution Shift? Hidden-State and Causal Predictors of Program and Verdict Correctness Viola Campos, Thomas Klassert, and Adrian Ulges

arXiv:2611.07876v1 [cs.SE] 5 Nov 2026

RheinMain University of Applied Sciences Wiesbaden, Germany {viola.campos, adrian.ulges}@hs-rm.de thomas.klassert@student.hs-rm.de

Abstract—Large language models increasingly judge code, yet without external feedback such judgments are unreliable, and verifying them by executing tests or by extended reasoning is costly. Instead, execution-free correctness predictors use signals from the model’s output probabilities, sampled solutions, or hidden states to identify judgments that warrant further verification. However, these predictors may degrade under distribution shifts when generalizing across projects and languages. We compare six execution-free predictors from four families on Qwen3-Coder-30B and gpt-oss-20b under benchmark shift between LiveCodeBench and BigCodeBench and language shift across Python, C++ and Java. We evaluate two targets: whether the model’s own verdict on a program can be trusted, and whether the program itself is correct. Besides established approaches, we adapt two novel causal localization methods to code, which have been shown to be more robust out-of-distribution on natural language tasks. For verdict correctness, causal localization improves robustness to language shift. For Qwen3-Coder a probe on the causally localized subspace stays 0.09 AUROC ahead of output confidence under language shift (0.11 in distribution), significantly outperforming the best non-causal predictor by 0.04 AUROC on average. This advantage, however, does not generalize consistently to gpt-oss or to benchmark shift. For program correctness, predictors transfer more successfully between benchmarks, largely by identifying problems on which the model tends to fail. However, their ability to rank candidates within the same problem is limited. Index Terms—code correctness prediction, uncertainty estimation, LLM-as-a-judge, distribution shift, causal interpretability

I. I NTRODUCTION LLMs are increasingly used to judge code, including their own. They determine whether a candidate solves its task, whether a change is safe to accept, and which of several candidates to keep. Without external feedback, such judgments are unreliable. Models rarely improve by verifying their own outputs [18], [26], favor their own generations [37], and remain inconsistent as code judges [23]. Reliable verification is costly: executing tests requires an environment and a test suite [41], and reasoning about a candidate involves generating many additional tokens. A cheap signal indicating when a judgment can be trusted could gate these costlier checks rather than replace them [24], [42]. Execution-free correctness predictors

provide such a signal from the model itself, such as the probability of its answer [4], [48], the agreement among sampled solutions [2], [5], [47], or its hidden states [3], [45], [51]. Probes on hidden states are particularly cheap, since they reuse activations the model computes anyway, and are already deployed as first-stage monitors [27], [33]. Such a predictor serves two purposes: it can estimate whether a program is correct, and whether the model’s own verdict on it can be trusted. Although predictors trained on hidden states are accurate in distribution, their acknowledged weakness is distribution shift [27], [36]. For code this is critical, as a predictor fitted on one benchmark or programming language will usually be applied to other projects and languages. For natural-language question answering, Huang et al. [19] proposed causal localization as a remedy: they use counterfactual interventions, which copy part of the hidden state from one input into another and observe how the model’s answer changes, to find the subspace that robustly determines the answer under shift. Whether this promise holds for code correctness assessment remains open. Instead, prior work on code has considered only a subset of predictor families and shift settings, and has largely focused on predicting program correctness [12], [15], [20], [44]. To our knowledge, none includes causal methods or asks whether the model’s own verdict can be trusted. We therefore ask which execution-free signals of code correctness survive a change of benchmark or programming language, and what explains those that do. We adapt two causal methods of Huang et al., counterfactual simulation and value probing, to code and compare them with output confidence, a probe on the hidden state, LAT [45] and EigenScore [5]. To do so, we use Qwen3-Coder-30B and gpt-oss-20b for candidate generation and as judges; we apply shifts between LiveCodeBench and BigCodeBench and between Python, C++ and Java. We then answer four research questions: RQ1 asks how well the methods tell whether the model’s verdict can be trusted, the target the causal methods were designed for; RQ2 asks which part of the causal methods carries their signal; RQ3 asks how well the methods predict whether the program is correct, the target of most prior work; and

RQ4 asks whether agreement across samples, which requires no supervised training, tells better than a single pass which problems the model will solve. Our results show that causal localization pays off where it was designed to: under language shift, a probe on the localized subspace is the most robust predictor of Qwen3-Coder’s verdict, significantly ahead of every non-causal method, and the learned subspace itself, not the intervention, carries this gain. On gpt-oss, no method stands out. Beyond the causal methods, generalization depends on the target and the type of shift: no verdict predictor transfers reliably between benchmarks, program correctness transfers largely because the predictors recognize difficult problems, and agreement across samples gives no consistent advantage over a single pass. Our contributions are: To our knowledge, the first comparison of executionfree correctness predictors for code across four families, including causal methods, under both benchmark and language shift and for two targets. • An adaptation of causal localization [19] to code correctness prediction. • An analysis showing that the learned subspace carries the advantage of the causal methods, and a distinction between candidate-level prediction and problem-level reliability: pooled AUROC must be read alongside a predictor’s ability to rank the candidates of one problem.

•

II. R ELATED W ORK a) Code correctness prediction: Token probabilities form the most direct signal for execution-free code correctness prediction. They have been studied for calibration [4], [48], but have been found to barely separate failing from passing programs [43]. Consistency methods compare multiple generations for the same problem, some by executing them [14], [46], [50], others without execution [2], [22], [47]. Probes and representation reading recover correctness from hidden states [3], [20], [29], [45], [51]. Comparisons across the above approaches only exist between output and consistency methods so far [2]. To our knowledge, no causal methods have been investigated. b) Verdict correctness prediction: LLMs are increasingly used as judges in software engineering [13], but have been found to misjudge code correctness [7] and react to superficial edits such as renamed variables [35]. Hence, verdict correctness prediction aims at predicting the correctness not of the code itself, but of the LLM’s asssessment. Approaches have explored judges’ hidden states [40], reasoning structure [39], or have escalated uncertain cases [24]. Furthermore, probes trained for correctness can track the model’s own judgment instead [28], which supports keeping the verdict and program correctness as separate targets. To our knowledge, however, no prior work studies both code correctness and verdict correctness, nor has verdict correctness been assessed under distribution shift.

c) Generalization under shift: ML models in software engineering are well known to erode for cross-dataset and cross-project prediction [17], [54], under language shift [15], [20], and under other realistic splits [8]. This has also been found for recent LLM-based predictors [27], [36], [49], particularly for execution-free code correctness prediction: Probes based on hidden states have been found to lack robustness [44], and to fall behind under dataset shift [12]. Each of these works studies one or two families of predictors on program correctness; we compare four families of predictors, evaluate not only code correctness but also verdict correctness, and include both benchmark and language shifts. d) Causal localization: Huang et al. [19] use causal localization with distributed alignment search [10] to learn a subspace through which interchange interventions [9] control the output, and use it for correctness prediction, reporting strong robustness under shift on natural-language tasks. In general, whether a causal role makes a direction generalize better is open: some work finds that it does [31], [32], while others find that causal efficacy and generality diverge [52]. To this end, our work is the first to adapt and evaluate causal probing for code correctness and verdict estimation. That an intervention changes the output does not by itself show that the model represents the variable there: patched subspaces can act through pathways the model does not normally use, and they need not be unique [11], [30], [34]. We therefore compare the learned subspace with random subspaces of the same dimension (Section VI-B). III. P ROBLEM S ETTING We study execution-free correctness prediction for code that an LLM generates and judges itself, from signals of the model’s own generation process: no tests are executed and no additional reasoning tokens are generated. Given a programming problem xi , a model M samples candidate solutions cij ∼ M(xi ), j = 1, . . . , ni . Following common practice in code generation benchmarks, a candidate is considered correct if it passes all tests. We denote the program correctness label prog as yij ∈ {0, 1}. Tests are executed only to obtain these labels for training and evaluation, never to make a prediction. Where a statement holds for any problem and candidate, we write (x, c) and drop the indices. A. The Model’s Verdict To obtain the model’s own judgment, we prompt M with the problem description xi and the candidate solution cij and ask whether the implementation is correct, explicitely demanding a single token answer, Yes or No. We define the model’s verdict v(x, c) as the answer t ∈ {Yes, No} with the higher probability pM (t|x, c). We refer to this probability as the verdict confidence and use it as a baseline in our experiments. The same forward pass that defines the verdict – the judging pass – also generates the signals for the other prediction methods (Section IV). We extract those at the verdict token’s position: probabilities of the two answer tokens

Yes/No, the hidden states, or a subspace of the hidden states identified through interventions. B. Prediction Targets We evaluate predictors for two different targets: a) Program correctness (y prog ): The first target is to predict whether the candidate implementation actually satisfies the problem specification, as operationalized by the test suite. This target corresponds to prior work on code correctness prediction [3], [14], [45], [46], [50], and prediction scores are used to decide which candidates need testing. Note that predictors may differ from the model’s verdict v as defined in Section III-A, since they can be based on other internal signals. b) Verdict correctness (y verd ): The second target is whether the model’s verdict (as defined in Section III-A) is correct, that is, whether the model accepts a correct candidate or rejects an incorrect one:   y verd = 1 v(x, c) = Yes ⇔ y prog = 1 . A predictor for this target estimates when the model’s judgment can be trusted and when to invoke more expensive verification, such as test execution or a stronger model. It is related to uncertainty estimation for generated answers, which predicts whether a model’s own answer is correct [19], [25]. Note that these two targets serve different purposes: program correctness concerns the candidate itself, whereas verdict correctness concerns the reliability of the model’s judgment. When a model correctly rejects an incorrect candidate, for example, its verdict is correct while the program is not. We evaluate both targets with the same judging pass to investigate which internal signals support each prediction target. C. Distribution Shift We study two generalization scenarios, in which the evaluation data differ from the training data in one controlled way. In benchmark shift, we train a predictor on one benchmark, LiveCodeBench [21] or BigCodeBench [53], and evaluate it on the other. This simulates a predictor fitted on one kind of task and deployed on another: LiveCodeBench consists of algorithmic programming challenges, BigCodeBench of practical tasks that call libraries and APIs. In language shift, we train a predictor on candidates in one programming language and evaluate it on candidates in another, for the same problems in Python, C++ and Java. The problems are split once for all three languages, so no problem is seen in training in one language and in evaluation in another. IV. M ETHODS We compare six predictors from four families, grouped by the signal they read from the model: (1) its output token probabilities, (2) the consistency across sampled generations, (3) internal features of the hidden states, and (4) internal causal features, a subspace of the hidden states identified through counterfactual interventions. The first three families are established approaches to correctness prediction. The fourth was

proposed for natural-language question answering by Huang et al. [19], who report that it is more robust under distribution shift. We adapt this approach to code generation and describe it in more detail. A. Output Token Probabilities (Confidence) The simplest predictor uses the probability the model assigns to an answer in the judging pass. It requires neither hidden states nor labeled training examples, and we refer to it as confidence. For program correctness, we use the probability of Yes, pM (Yes | x, c), following Kadavath et al. [25]. For verdict correctness, we use the probability of the emitted verdict, pM (v(x, c) | x, c). A model that is uncertain of its verdict should be wrong more often [16]. Both come from a softmax over the full vocabulary, whose temperature is the only free parameter (Section V-B). B. Consistency (EigenScore) Consistency based methods do not judge a single candidate but asks how consistent the model is across several generations for the same problem: a model that knows how to solve a problem should produce similar solutions, while diverging samples indicate uncertainty. We use EigenScore [5], which measures consistency in the hidden states. For a problem with m sampled candidates, it takes the hidden state of the last layer at the last token of each candidate’s generation and computes the log-determinant of their regularized covariance matrix, normalized by m: the more the states spread out, the higher this Eigenscore. We use the negated EigenScore as an indicatore that instances from the candidate set solve the problem. EigenScore is an unsupervised method, requiring no correctness labels. Unlike all other methods, it scores a problem rather than a candidate and does not rank the candidates of one problem. We compare it with the other methods at problem level (Section VI-D). Its cost is m generations per problem, which come for free wherever multiple candidates are sampled anyway. C. Internal Features The third family reads correctness from the hidden states. We extract the hidden state hℓ (x, c) ∈ Rd after a selected layer ℓ at the last token of the judging prompt, the position from which the verdict is generated. hℓ (x, c) serves as input for two probing methods: a) Correctness probe: We use linear probes [1], i.e. logistic regressors trained on hℓ with correctness labels [3], [36], [51]: y prog for program correctness and y verd for verdict correctness. A probe’s predicted probability of the positive class serves as the score. Since the classifier is linear, it captures only linearly accessible information [38], like the other hidden states methods. b) Linear Artificial Tomography (LAT): LAT [45], [55] identifies a correctness direction: For each training problem with both a passing and a failing candidate, it computes the difference of their hidden states and applies principal component analysis (PCA) to these difference vectors. The first principal

component defines the direction, whose sign is chosen so that passing candidates tend to have a higher projection than failing ones. A candidate’s score is its projection onto this direction. In the original method, hidden states are extracted using a dedicated stimulus prompt that presents the problem and candidate [45]. In a preliminary comparison, applying the same procedure to the judging pass did not improve performance, so we use the original prompt for fairness. LAT was designed to predict program correctness, and we evaluate it for this target only. D. Internal Causal Features These methods follow the tradition of causal interpretability, which identifies a causal representation by intervening on it [9]. Here, these approaches try to localize specific features within hidden states that causally determine a model’s verdict rather than merely correlating with correctness. If a model judges code reliably when prompted, it presumably does so through a systematic internal computation. We ask whether this computation can be localized and whether it helps predicting correctness on new inputs. a) Localizing the verdict: We assume that the model computes an internal assessment of whether a given candidate is correct, a variable V ∈ {Yes, No} that determines its verdict and is encoded in the hidden state at the last prompt token. To localize it, we use Distributed Alignment Search (DAS) [10], which can be understood by analogy to debugging: Run a program on two inputs, pause both at a breakpoint, copy a variable’s value from the second run into the first, and continue: if the first run now produces the expected output of the second, the variable controls the output. The run that is modified is called the base b = (xb , cb ), the run whose value is copied the source s = (xs , cs ). Note that both are usually sampled from different problems (Figure 1). In an LLM there are no named variables whose values could be switched as in the debugging analogy, and a concept such as “this program is correct” is typically spread across many dimensions of hℓ . DAS therefore learns where to look: a kdimensional subspace, given by R ∈ Rd×k with orthonormal columns. The intervention replaces the component of the base state’s representation within this subspace with the corresponding component from the source and leaves the rest of the base’s state untouched:
hℓb←s = hℓ (b) + RR⊤ hℓ (s) − hℓ (b) . (1) The forward pass of b then continues through all later layers from hℓb←s instead of hℓ (b). Such a replacement is called an interchange intervention [9]. b) Training: DAS is trained only on candidates the model judges correctly (y verd =1). On these, the verdict coincides with program correctness y prog : Yes for the passing candidates, No for the failing ones. A training pair combines two such candidates as base and source, from the same or different problems and with the same or different verdicts. With the LLM frozen, R is trained so that the intervened base outputs the source’s verdict: Where the two verdicts

(a) Judging pass problem x candidate c Correct? Yes/No

layer `

p(Yes), p(No) verdict v

probe: h` , value probe: R> h`

confidence

(b) Interchange intervention at layer ` source s (xs , cs )

verdict v(s) copy subspace R> h` (s)

answer = v(s)?

base b (xb , cb ) h`b←s

Fig. 1. (a) The judging pass: the model reads the problem, the candidate and the question and answers Yes or No at the last prompt token. The confidence baseline reads answer probabilities, the probe the hidden state hℓ of one layer, the value probe its projection onto the learned subspace. (b) An interchange intervention: the subspace coordinates of a source example’s hidden state replace those of a base example, and the base’s forward pass continues. DAS learns a subspace so that the base then answers like the source. Counterfactual simulation scores a candidate by making it the source for B = 16 fixed bases from the training set. A source state that firmly encodes its verdict should enforce it on the bases, one on which the model is unsure should not. The more consistently the bases adopt the candidate’s verdict, the more likely it is correct.

differ, the intervention must flip the base’s answer; where they agree, it must leave it unchanged. The share of heldout interventions that produce the intended verdict is the interchange intervention accuracy (IIA) [10]. It shows whether the subspace controls the verdict, not how well the verdict predicts correctness. We learn one subspace per layer and use it for both targets. c) Counterfactual Simulation: Counterfactual simulation [19] asks whether a (source) candidate’s state in the subspace is strong enough to impose its assessment on other (base) examples. The intuition here is that a source candidate whose state clearly encodes a verdict – Yes or No – should do so, while one on which the model is unsure should not. We fix a set B of B=16 correctly judged training candidates as bases. The candidate to be scored is the source s. Let pb←s (t) be the probability of answer t in the intervened forward pass of base b. Averaged over the bases, we calculate two scores: 1 X csprog (s) = pb←s (Yes), (2) B b∈B
1 X csverd (s) = pb←s v(s) . (3) B b∈B

The first measures how strongly the candidate’s state pushes the bases’ verdicts toward Yes and predicts program correctness. The second measures how consistently the bases adopt the candidate’s own verdict and predicts verdict correctness. Note that counterfactual simulation requires B additional forward passes per candidate. d) Value Probe: The value probe [19] utilizes the subspace R for probing instead of intervening on it, so it needs

no forward pass beyond the judging pass. Like the correctness probe (Section IV-C), it uses logistic regression, but on the kdimensional projection R⊤ hℓ (x, c) of the candidate’s hidden state onto subspace R. This probe is trained differently for the two targets: For the verdict target, it learns which verdict a subspace state encodes. It is trained on the same candidates as DAS, those the model judges correctly, to predict the model’s verdict, which on these candidates equals y prog : Yes for passing and No for failing programs. Following Huang et al., the score is the probe’s confidence in its own prediction, |p(Yes) − 12 |. As in counterfactual simulation, a state that clearly reads as Yes or No indicates certainty and a probably correct verdict. For the program target, we train the probe on all training candidates to predict y prog , and its probability of Yes is the score. V. S TUDY D ESIGN We evaluate the methods of Section IV on both targets of Section III, verdict and code correctness, in distribution and under benchmark and language shift. RQ1–RQ3 judge single candidates, RQ4 whole problems. A. Models and Data a) Models: We study two open-weight models, Qwen3-Coder-30B-A3B-Instruct and OpenAI gpt-oss-20b. Each model generates candidate programs and judges its own candidates, so generation, verdicts and hidden states come from the same model. Qwen3-Coder answers the judging prompt directly. Since gpt-oss is a reasoning model, we prefill its final channel so that it also answers directly. Preliminary experiments showed that reasoning would cost about 1,100 generated tokens per candidate (median at medium effort) instead of a single forward pass, and its varying length would shift the position at which the methods read the hidden state. Judging without reasoning costs gpt-oss some accuracy (Section VIII). b) Benchmarks: We construct our datasets from LiveCodeBench [21] (release_v6) and BigCodeBench [53], both written in Python. To evaluate language shift we use 454 LiveCodeBench problems released between October 2024 and May 2025. We use the original Python versions, together with corresponding C++ versions from the LiveCodeBenchCPP dataset (v6_2408_2505)1 and Java versions sourced from the original AtCoder and LeetCode platforms. We refer to these datasets as LCB-Python, LCB-C++, and LCB-Java, respectively. LCB-Python is the subset of our LCB data on these problems. c) Candidates and labels: For each problem, each model samples n = 10 candidates at temperature t = 1. We discard generations from which no code candidate can be extracted, candidates whose judging prompts exceed 2048 tokens, and duplicate generations. EigenScore compares m non-empty candidates per problem: m = 5 on LCB and the language sets, where some generations are empty, and all ten on BCB, where 1 https://huggingface.co/datasets/nvidia/LiveCodeBench-CPP

TABLE I E VALUATION SETS OF THE FIRST PARTITION . Problems: NUMBER OF DISTINCT PROBLEMS ; Cand.: NUMBER OF SCORED CANDIDATES ; Pass: PERCENTAGE OF CANDIDATES THAT PASS THE TESTS ; Yes: PERCENTAGE OF CANDIDATES ACCEPTED BY THE MODEL ; Judge acc.: BALANCED ACCURACY OF THE MODEL’ S VERDICT AGAINST PROGRAM CORRECTNESS ( MEAN OF THE ACCEPTANCE RATE ON PASSING AND THE REJECTION RATE ON FAILING CANDIDATES ). Benchmark

Problems

Cand.

Pass

Yes

Judge acc.

Qwen3-Coder-30B LiveCodeBench BigCodeBench LCB-Python LCB-C++ LCB-Java

208 456 178 175 179

1646 3240 1423 1448 1486

55.3 40.2 41.7 24.0 37.5

58.3 71.7 44.1 32.5 31.2

74.4 58.3 73.8 63.8 64.4

gpt-oss-20b LiveCodeBench BigCodeBench LCB-Python LCB-C++ LCB-Java

189 456 150 147 141

1588 4533 1140 1116 1069

80.4 45.8 76.8 68.4 59.8

44.6 42.6 37.6 35.1 38.6

62.8 61.7 62.2 57.2 65.4

almost none are. Problems with fewer non-empty candidates are left out of the problem-level comparison for every method (93–100% of the problems are kept for Qwen3-Coder, 76–94% for gpt-oss). We obtain program-correctness labels by executing each candidate against the respective benchmark’s test suite. To obtain verdict-correctness labels, we prompt the generating model to judge each candidate and compare its verdict with the execution-based correctness label (Section III). d) Splits: We split each benchmark by problem into training, validation and test sets (60/20/20), so all candidates of a problem fall into the same split. For the language sets, with fewer than 90 test problems each, we evaluate on validation and test together. We repeat every experiment on three partitions of the problems with disjoint test sets and report the mean over the partitions. Table I summarizes the evaluation sets of the first partition and reports the fraction of candidates that pass the tests, the fraction accepted by the model, and the balanced accuracy of the model’s verdict against program correctness. The models’ balanced accuracy ranges from 57.2% to 74.4%, so the models misjudge many of their own candidates, and knowing which verdicts to trust matters. B. Training and Selection a) Fitting: The probe is a logistic regression on the hidden state of one layer, trained on 512 candidates per class. LAT uses the stimulus prompt of Ribeiro et al. [45] and is fitted on one pair of a passing and a failing candidate per training problem that has both. DAS learns a k=6-dimensional subspace per layer, a size chosen in a preliminary sweep and in line with evidence that correctness is encoded in few dimensions [6]. It is trained on pairs of 1024 correctly judged training candidates, in at most one pass over 20,480 pairs with early stopping on the training IIA. The value probe is a logistic regression on the projections of the same training candidates

onto the subspace. Counterfactual simulation uses a fixed set of B=16 correctly judged training candidates as bases. The generation probes of RQ3 apply the probe to the hidden states of the model’s own generation, replayed token by token, at the last token of the code or of the prompt; they are trained on the same candidates and program labels as the probe. Detailed settings can be found in the replication package2 . b) Selection: Each method comes with one free parameter: the layer, or – for confidence – the temperature. The probe and LAT are fitted on every layer, while DAS is trained at a subset (every second layer of L33–L43 for Qwen3-Coder and of L11–L23 for gpt-oss), since each subspace takes several GPU hours. Layers are chosen in distribution, via cross-fitting over problems. The selection is made on four fifths of the evaluation problems and read on the remaining fifth, over 20 random splits. Under shift, the choice made on all indistribution rows is read on the shifted set, so no shifted row enters selection.

TABLE II AUROC FOR VERDICT CORRECTNESS ( MEAN OVER 3 PARTITIONS ). LCB, BCB: TRAINED AND EVALUATED ON THAT BENCHMARK ; →: EVALUATED ON THE OTHER ONE . L ANGUAGE SHIFT: ID IS THE MEAN OVER THE THREE TRAINING LANGUAGES , Shift THE MEAN OVER THE SIX OOD TRAIN→ EVALUATION PAIRS (TABLE III), ∆ THEIR DIFFERENCE . B EST PER COLUMN AND LLM IN BOLD . Benchmark shift Method

Language shift

LCB →BCB BCB →LCB

ID Shift

∆

Qwen3-Coder-30B Confidence .672 Probe .699 Counterf. sim. .692 Value probe .740

.552 .480 .453 .485

.555 .639 .560 .579

.669 .516 .680 .695

.644 .761 .759 .758

.643 .668 .728 .728

−.001 −.093 −.031 −.031

gpt-oss-20b Confidence Probe Counterf. sim. Value probe

.598 .478 .572 .599

.597 .507 .601 .600

.544 .476 .556 .605

.542 .774 .808 .713

.536 .668 .691 .646

−.005 −.106 −.116 −.067

.569 .829 .779 .695

VI. E XPERIMENTS C. Evaluation Protocol a) Metrics: At candidate level (RQ1–RQ3) we report the area under the receiver operating characteristic curve (AUROC) over all candidates in an evaluation set, as is common for correctness prediction [2], [14], [15], [47], using either verdict correctness or program correctness as the positive label. AUROC measures how well a score ranks positive examples above negative ones, with 0.5 corresponding to chance and 1.0 to perfect ranking. Because candidates from different problems are pooled, this metric reflects both the ability to distinguish candidates within a problem and the ability to rank problems by how often the model produces positive outcomes. Where this distinction matters, we additionally report within-problem AUROC, computed over pairs of candidates from the same problem with different labels. It equals the pairwise accuracy of [44] and is on the same scale as the pooled AUROC. The gap between the two is the part of the pooled AUROC that comes from ranking problems. At problem level (RQ4), a problem is considered as solved if at least half of its candidates pass, and the single-pass methods are averaged over the candidates that EigenScore uses. b) Significance: We compare methods using a paired, problem-clustered bootstrap with 2,000 resamples. Each resample draws whole problems and evaluates all methods on the same examples, accounting for within-problem dependence while preserving paired comparisons. For language-shift results, we average differences over the six language pairs and three partitions, using shared resamples across language pairs because they share problems. We combine the bootstrap standard error (SE) with between-partition variability to account for variation due to training and layer selection. We consider a difference significant if |∆| > 1.96 SE and report ∆±1.96 SE.

A. RQ1: How well can execution-free methods tell whether the model’s own verdict can be trusted? Both LLMs are mediocre judges, with balanced accuracies of their verdicts of 57–74% (Table I) against the tests. RQ1 asks how well our methods identify the candidates on which the model’s verdict is correct. Table II summarizes the results, showing the mean per method over partitions. Table III breaks down the language shift by training and evaluation language. a) In distribution, hidden states outperform confidence: On LCB and the language sets, the methods that read hidden states reach 0.69–0.83, while confidence is weak for Qwen3Coder (0.64–0.67) and close to chance for gpt-oss (0.54– 0.57). On the language sets, the probe and the two causal methods are level for Qwen3-Coder (0.758–0.761); for gptoss, counterfactual simulation is best (0.808). BCB is harder, no trained method exceeds 0.639 for Qwen3-Coder or 0.601 for gpt-oss. b) Under language shift, the causal subspace is the most robust predictor for Qwen3-Coder: For Qwen3-Coder, the probe drops from 0.76 in distribution to 0.67 when the language changes, while both causal methods fall only from 0.76 to 0.73, against 0.64 for confidence. Averaged over the six language pairs, the value probe is ahead of the best noncausal method of each pair by 0.042 ± 0.021 (95% interval of the paired test, Section V-C) and ahead in each of the three partitions (+0.035, +0.053, +0.038). Counterfactual simulation reaches the same mean but varies more between partitions (±0.043). The lead does not depend on the probe’s layer choice: in 13 of the 18 pair–partition combinations, most value-probe checkpoints lie above the probe at its best layer on the shifted rows, as illustrated for Qwen3-Coder, trained on C++ in Figure 2. For gpt-oss, all trained methods lose 0.07–0.12 AUROC under language shift, and neither causal 2 https://anonymous.4open.science/r/FORGE-PCCuDS

TABLE III AUROC FOR VERDICT CORRECTNESS , IN DISTRIBUTION (ID) AND UNDER LANGUAGE SHIFT PER TRAINING LANGUAGE (LCB PROBLEMS ; ID CROSS - FITTED , SHIFTED COLUMNS AT THE LAYER CHOSEN IN DISTRIBUTION ). M EAN ± SD OVER 3 PARTITIONS OF THE PROBLEMS . B EST PER COLUMN IN BOLD ; UNDERLINED : WITHIN ONE SD OF THE BEST. N ON - CAUSAL METHODS ABOVE THE RULE , CAUSAL BELOW. Trained on Py Model

Method

Qwen3-Coder-30B

gpt-oss-20b

Probe

ID

→C++

→Java

ID

→Py

→Java

ID

→Py

→C++

.608 ±.028 .670 ±.011

.637 ±.052 .628 ±.103

.687 ±.011 .636 ±.059

.637 ±.054 .874 ±.010

.611 ±.026 .636 ±.062

.688 ±.013 .701 ±.009

.689 ±.010 .740 ±.044

.605 ±.027 .670 ±.026

.630 ±.053 .739 ±.020

Counterf. sim. Value probe

.671 ±.071 .717 ±.031

.679 ±.129 .692 ±.030

.721 ±.041 .737 ±.004

.854 ±.021 .814 ±.010

.714 ±.024 .731 ±.003

.746 ±.013 .761 ±.010

.751 ±.019 .744 ±.011

.729 ±.017 .733 ±.019

.778 ±.013 .712 ±.026

Confidence Probe

.542 ±.023 .812 ±.021

.529 ±.016 .692 ±.016

.549 ±.050 .677 ±.018

.528 ±.016 .797 ±.011

.541 ±.023 .651 ±.045

.549 ±.050 .596 ±.021

.555 ±.045 .713 ±.033

.525 ±.009 .739 ±.055

.525 ±.016 .651 ±.018

Counterf. sim. Value probe

.828 ±.099 .743 ±.010

.652 ±.144 .636 ±.028

.759 ±.034 .645 ±.028

.799 ±.088 .700 ±.007

.670 ±.168 .633 ±.023

.624 ±.057 .591 ±.007

.796 ±.027 .697 ±.005

.784 ±.092 .721 ±.018

.660 ±.091 .653 ±.015

Counterf. sim.

Value probe

Confidence

selected

pooled within problem

C++ → Java

C++ → Python

0.8

Qwen3-Coder-30B

language shift in distribution

gpt-oss-20b

Confidence

0.7

verdict target

AUROC

Trained on Java

Confidence Probe

C++ (ID)

0.9

Trained on C++

Probe

0.6

Counterf. sim.

0.5 30

40

layer

30

40

layer

30

40

Value probe

layer Confidence program target

Fig. 2. AUROC per layer on the verdict target, Qwen3-Coder trained on C++ (fold 0), in distribution (left) and under language shift. Rings mark the layer chosen in distribution; the dashed line is confidence. Under shift to Python, every causal checkpoint lies above the probe’s best layer; on Java, most do.

Probe LAT Counterf. sim.

method is significantly ahead of the best non-causal one. The robustness reported for natural language tasks [19] thus carries over to code for one of the two models; Section VI-B examines what carries the gain. c) The advantage lies in identifying unreliable problems, not candidates: Pooled AUROC combines the ability to identify problems on which the judge is unreliable with the ability to distinguish correct from incorrect verdicts within a problem (Section V-C). The second is limited. Under language shift, within-problem AUROC is 0.616–0.618 for all trained methods of Qwen3-Coder. For gpt-oss the probe reaches 0.652, counterfactual simulation 0.635 and the value probe 0.576 (Figure 3). The causal methods’ pooled advantage thus comes mainly from identifying problems on which the judge is unreliable, rather than individual candidates it misjudges. d) A change of benchmark affects every method more than a change of language: Transfer between LCB and BCB is asymmetric (Table II). From LCB to BCB, all trained methods of Qwen3-Coder fall to 0.453–0.485, below confidence (0.552). In the reverse direction, the probe stays near chance (0.516), while the causal methods reach 0.680 and 0.695, close to confidence (0.669). For gpt-oss, every method stays at 0.48– 0.61 in both directions.

Value probe 0.6

0.8 AUROC

0.6

0.8 AUROC

Fig. 3. Within-problem (hollow) against pooled AUROC (filled), in distribution (faint) and under language shift: means over training languages and partitions. Within-problem AUROC uses the problems with mixed outcomes only (Section V-C); the length of a line is how much of the pooled AUROC comes from ranking problems.

Answer to RQ1. In distribution, hidden states predict whether the model’s verdict is right far better than its confidence. Under language shift, a probe on the causally localized subspace is the most robust predictor for Qwen3-Coder (+0.04 AUROC over the best non-causal method, in all three partitions), mainly by identifying problems on which the judge is unreliable; for gptoss, no method is significantly ahead. Verdict prediction does not transfer reliably between benchmarks.

B. RQ2: Which part of the causal methods carries their signal? On Qwen3-Coder’s verdict target, the causal methods are the most robust predictors under language shift (RQ1). We now investigate which aspects of the method contribute to this performance: whether prediction requires the intervention itself, whether the learned directions matter beyond their

Qwen3-Coder AUROC (shift) gpt-oss AUROC (shift)

trained on Python

trained on C++

trained on Java

33

33

33

0.6

0.4 37

41

37

41

37

TABLE IV AUROC FOR PROGRAM CORRECTNESS ( MEAN OVER 3 PARTITIONS ). LCB, BCB: TRAINED AND EVALUATED ON THAT BENCHMARK ; →: EVALUATED ON THE OTHER ONE . L ANGUAGE SHIFT: ID IS THE MEAN OVER THE THREE TRAINING LANGUAGES , Shift THE MEAN OVER THE SIX TRAIN→ EVALUATION PAIRS , ∆ THEIR DIFFERENCE . B EST PER COLUMN AND MODEL IN BOLD .

41

Benchmark shift Method

0.6 0.5 11

15 19 layer

learned subspace (shift)

11

15 19 layer

11

random (mean, range)

15 19 layer IIA ≥ 0.9

Fig. 4. Value probe on the learned DAS subspace against value probes on three random 6-dimensional subspaces at the same layer, verdict target, mean over shifted languages (the band is the range over the random subspaces). Shaded: layers where DAS has converged (IIA ≥ 0.9). On Qwen3-Coder the learned subspace is above random at every layer; on gpt-oss it is above random before convergence and near chance once DAS has converged.

dimensionality, and whether interchange intervention accuracy (see Section IV-D) indicates when the subspace is useful. a) Reading the subspace is sufficient: Counterfactual simulation intervenes through the subspace and reads the answer token, at the cost of B additional forward passes per candidate; the value probe reads the subspace directly in the judging pass. On Qwen3-Coder the two are equally robust under language shift (mean difference over the six pairs −0.000 ± 0.033 on the verdict target, +0.004 ± 0.007 on the program target), and both are ahead of confidence, which reads the same answer token without the subspace (+0.085 ± 0.033 and +0.05 ± 0.03). The intervention thus brings no significant improvement over reading the subspace, and the value probe costs no more at prediction time than a probe. b) The learned direction matters, not the low dimension: A six-dimensional readout might transfer better simply because it is small. We therefore train the value probe on three random subspaces of the same dimension at the same layer. At the layer chosen in distribution, the learned subspace reaches 0.70–0.75 under shift for Qwen3-Coder, random subspaces 0.48–0.61, no better than the probe on the full hidden state (0.668, Table II). The same holds at every trained layer (Figure 4). c) Interchange accuracy does not guarantee predictive utility: DAS learns the subspace so that swapping it between two inputs swaps the model’s verdict. IIA measures how often this succeeds. A high IIA thus shows that the subspace controls the decision, not that it tells whether the decision is right. On gpt-oss the two diverge (Figure 4): at L11 and L13, where DAS succeeds only partly (IIA 0.54–0.64), the learned subspace transfers better than random ones (0.56–0.68 against 0.52–0.55 under shift), and L13 is the layer chosen in distribution. However, at L17–L21, where DAS almost always

Language shift

LCB →BCB BCB →LCB

ID Shift

∆

Qwen3-Coder-30B Confidence .815 Probe .844 LAT .844 Counterf. sim. .839 Value probe .838

.650 .656 .580 .644 .642

.648 .688 .590 .635 .664

.815 .697 .731 .813 .815

.759 .878 .807 .848 .861

.755 .790 .799 .806 .810

−.003 −.088 −.009 −.042 −.051

gpt-oss-20b Confidence Probe LAT Counterf. sim. Value probe

.648 .575 .618 .647 .638

.646 .649 .608 .652 .652

.721 .690 .676 .724 .712

.696 .859 .760 .820 .846

.691 .693 .666 .710 .666

−.004 −.166 −.094 −.111 −.180

.723 .793 .767 .780 .803

succeeds (IIA 0.91–0.95), it falls to 0.52–0.58, no better than random. On Qwen3-Coder the converged layers (L39–L43, IIA 0.95) remain informative. IIA thus cannot tell the two cases apart, and layers are chosen by held-out prediction instead (Section V-B). Answer to RQ2. Where the causal methods help, the learned subspace carries the gain: the value probe, which reads it in the judging pass, is as robust as counterfactual simulation, and random subspaces of the same size transfer only as well as the full hidden state. IIA does not indicate when the subspace is useful: on gpt-oss, the converged subspaces are no better than random ones.

C. RQ3: How well do the methods predict whether the program is correct? The program target asks whether a candidate passes its tests, regardless of the model’s verdict. The two targets agree only where the judge is right. We therefore ask whether the results on the verdict target carry over. Table IV summarizes the results. In distribution, all methods predict program correctness well above chance, AUROC ranges from 0.70–0.88 on LCB and the language sets, with the hidden-state probe performing best on the language sets for both models. BCB is more challenging for every method. a) Under language shift, the causal methods lose their advantage: The full-state probe loses 0.088 AUROC for Qwen3-Coder and 0.166 for gpt-oss, while confidence changes little (−0.003 and −0.004). For Qwen3-Coder, LAT and both causal methods nevertheless remain significantly ahead of confidence by 0.044–0.054 AUROC on average; the probe’s lead of 0.035 is no longer significant. For gpt-oss, no trained method significantly outperforms confidence. Importantly, neither causal method outperforms the best non-causal method: for Qwen3-Coder, the value probe and counterfactual simulation trail it by 0.010 and 0.014 AUROC, respectively, neither

significantly. For gpt-oss, the value probe is significantly behind (−0.068), while counterfactual simulation is statistically level (−0.025). Thus, the robustness of causal localization observed for verdict correctness (RQ1) does not carry over to program correctness. b) Program correctness transfers between benchmarks: In contrast to verdict correctness, predictors for program correctness transfer between LCB and BCB in both directions. When trained on LCB and evaluated on BCB, methods reach 0.580–0.656 for Qwen3-Coder, and 0.575–0.648 for gpt-oss, comparable to the respective in-distribution results on BCB (0.590–0.688 and 0.608–0.652). Transfer in the reverse direction is similarly strong, reaching 0.70–0.82 for Qwen3-Coder and 0.68–0.72 for gpt-oss. c) Pooled AUROC overstates candidate-level discrimination: As for verdict correctness, pooled AUROC reflects both the ability to identify problems the model tends to solve and the ability to distinguish candidates within a problem. Choosing a candidate needs only the second, and here every method is weak (Figure 3, bottom): under language shift, within-problem AUROC lies at 0.58–0.64 for all methods and both models, against a pooled AUROC of 0.67–0.81. For Qwen3-Coder confidence performs best within problem (0.629). The trained methods reach higher within-problem AUROCs of 0.69–0.78 only for gpt-oss in distribution, but lose this advantage under shift. Consequently, the predictors are more useful for identifying problems that warrant verification than for choosing among candidates of the same problem. Section VI-D predicts at problem level directly. d) Reading the generation is as good as asking: All methods so far read the model in a separate judging pass. The same information might already be present when the model generates the candidate. We therefore train the same probe on the hidden states of the model’s own generation (without the judgment prompt), replayed token by token, and compare it with the judging-pass probe on identical candidates, Table V reports the results. We compare different token positions here: at the last token of the candidate’s code, it is at least as good as the judging pass in distribution (0.895 against 0.886 for Qwen3-Coder, 0.864 against 0.851 for gptoss); under language shift, it is close for Qwen3-Coder (0.795 against 0.779) and behind for gpt-oss (0.673 against 0.704). A correctness score can thus be read while the code is written. Read at the last prompt token, before any code exists, the probe has seen only the problem, yet for Qwen3-Coder it matches the judging pass in distribution (0.891). Much of what the methods predict is thus the problem, not the candidate. Answer to RQ3. All methods predict program correctness well above chance, largely by recognizing the problems the model fails on, and this transfers better between benchmarks than the verdict. Under language shift, the trained methods lose most of their lead over confidence: about 0.05 remains for Qwen3-Coder, nothing for gpt-oss, and the causal methods’ robustness on the verdict target does not carry over. A probe on the model’s own generation is as good as one on a separate judging pass in distribution.

TABLE V T HE PROBE ON THE JUDGING PASS AGAINST THE SAME PROBE ON THE STATES OF THE MODEL’ S OWN GENERATION , PROGRAM TARGET ON THE LANGUAGE SETS . ID: MEAN AUROC OVER THE THREE LANGUAGES , Shift: MEAN OVER THE SIX LANGUAGE PAIRS , Within: AUROC OVER PAIRS OF CANDIDATES OF THE SAME PROBLEM . T HE LAST PROMPT TOKEN HAS SEEN ONLY THE PROBLEM . ID Readout

Lang. Shift

Pooled Within Pooled Within

Qwen3-Coder-30B Judging pass Generation, last code token Generation, last prompt token

.886 .895 .891

.553 .620 .514

.779 .795 .755

.540 .627 .531

gpt-oss-20b Judging pass Generation, last code token Generation, last prompt token

.851 .864 .720

.772 .781 .524

.704 .673 .669

.609 .623 .478

D. RQ4: Does agreement across samples improve problemlevel prediction? This question moves from the candidate to the problem level. Much of the candidate-level signal reflects which problems the model solves (Section VI-C). When several candidates are sampled for a problem, their agreement provides a further signal. EigenScore [5] reads this agreement from the hidden states of the generations (Section IV-B). Like confidence, it needs no training, so it cannot lose under shift. We compare at equal sampling cost: each single-pass method is averaged over the candidates EigenScore uses, and every method predicts whether at least half of a problem’s candidates pass (Table VI). The cheapest prediction needs no sample at all: for Qwen3Coder, a probe that has seen only the problem already ranks candidates as well as the judging pass (Section VI-C). Sampling several candidates pays off for prediction only if their agreement adds to what a single pass provides. a) Agreement provides no consistent advantage: For Qwen3-Coder, EigenScore is at chance on LCB (0.504) and on the language sets (0.528), where every single-candidate method, confidence included, is significantly ahead (0.77– 0.91). On BCB it is comparable to them (0.658 against 0.671– 0.682). For gpt-oss the benchmarks swap roles: EigenScore is at chance on BCB (0.518 against 0.622–0.712) and closer on LCB (0.660 against 0.698–0.808). Under shift, the singlepass methods lose ground. From LCB to BCB, Qwen3-Coder’s trained methods fall slightly below EigenScore (0.618–0.652 against 0.658) while confidence stays above it (0.670); under language shift, the gap remains large for Qwen3-Coder (0.775– 0.838 against 0.528) and narrows for gpt-oss (0.615–0.676 against 0.586). Answer to RQ4. Agreement across samples gives no consistent advantage over a single pass. EigenScore is level with the singlepass methods in some cells and near chance in others, and confidence, which needs no labels either, is never significantly below it. That EigenScore cannot lose under shift does not help where it starts near chance.

TABLE VI AUROC FOR WHETHER A PROBLEM IS SOLVED ( AT LEAST HALF OF ITS CANDIDATES PASS ). S INGLE - CANDIDATE METHODS ARE AVERAGED OVER THE CANDIDATES OF A PROBLEM , AT THE LAYER OF TABLE IV; E IGEN S CORE IS COMPUTED FROM THE SAMPLED CANDIDATES AND NEEDS NO TRAINING . C OLUMNS AS IN TABLE IV. ↑: SIGNIFICANTLY ABOVE E IGEN S CORE ( PAIRED PROBLEM BOOTSTRAP, 95%; FOR ID AND S HIFT ON THE MEAN OVER LANGUAGES ). A LL METHODS ON THE PROBLEMS THAT HAVE AN E IGEN S CORE . ∗ Benchmark shift Method

LCB →BCB

Language shift

BCB →LCB

Qwen3-Coder-30B EigenScore .504 Confidence .877↑ Probe .906↑ LAT .902↑ Counterf. sim. .912↑ Value probe .904↑

.658 .670 .638 .618 .646 .652

.658 .671 .671 .682 .676 .680

gpt-oss-20b EigenScore .660 Confidence .698 Probe .791 LAT .808↑ Counterf. sim. .772 Value probe .790

.518 .677↑ .615 .662↑ .655↑ .663↑

.518 .678↑ .712↑ .622↑ .685↑ .692↑

ID

Shift

.504 .872↑ .884↑ .871↑ .881↑ .889↑

.528 .774↑ .911↑ .844↑ .875↑ .890↑

.528 .775↑ .803↑ .821↑ .831↑ .838↑

.660 .692 .759 .688 .692 .676

.586 .623 .845↑ .772↑ .800↑ .834↑

.586 .615 .676↑ .645 .662 .630

∗

EigenScore needs m non-empty samples: five per problem on LCB and the language sets, ten on BCB. Problems with fewer non-empty generations are left out for every method, and the single-candidate methods are averaged over the candidates of the same samples. This keeps 93–100% of the evaluation problems for Qwen3-Coder-30B and 76–94% for gpt-oss-20b.

VII. D ISCUSSION a) Verdict and program are different prediction targets: Our results show that the usefulness of execution-free correctness prediction depends on both the prediction target and the type of distribution shift. In particular, predicting whether a model’s verdict is correct is different from predicting whether the program itself is correct, even though both targets use signals from the same judging pass. The two targets respond differently to shift. Causal localization helps only on the verdict, and between benchmarks only program correctness transfers. An execution-free correctness predictor for an LLMbased code-generation system should therefore be chosen and evaluated for the decision it supports. To decide which candidates to test in a new programming language, output confidence is a strong baseline: it needs no labels and loses almost nothing under shift, while trained predictors keep only part of their lead. To decide whether to trust the model’s verdict, confidence is weak, and no verdict predictor we tested carries over to a new benchmark. b) Triage rather than selection: Much of what the methods predict is which problems the model solves: every method is weak at ranking candidates within a problem, and a probe that has seen only the problem matches one that has seen the candidate (Table V). Pooled metrics, such as AUROC [2], [47] or accuracy and F1 [3], [51], thus mix problem difficulty with the ability to tell a correct candidate from an incorrect one. For triage across problems this is useful, while for choosing among the candidates of one problem it is not. Selection

metrics [45] and pairwise accuracy within a task [44] measure the second ability directly; we recommend reporting such a within-problem measure alongside the pooled one. c) Causal localization for code: Huang et al. [19] report that causally localized predictors are more robust under shift on natural-language tasks. For code, this holds on one of our two models and for the target the method was designed for, the model’s own verdict; on the program correctness target it gives no advantage. Where it helps, the learned subspace carries the gain: once the subspace is learned, a probe on it is as robust as simulating interventions and costs no more than any probe. IIA does not reliably identify subspaces that are useful for correctness prediction. On gpt-oss, well-converged subspaces were no better than random ones, so the subspace has to be validated on held-out data. VIII. T HREATS TO VALIDITY a) Construct validity: Correctness is defined by the benchmarks’ tests, which can miss errors. The judging prompt asks for a single token without reasoning; gpt-oss, a reasoning model, judges moderately better with reasoning: on 498 program-target candidates of the language sets, medium-effort reasoning (median 1,101 tokens) raises its balanced accuracy from 62.4% to 67.8% and the AUROC of its Yes probability from 0.68 to 0.73, still below the trained internal methods (Table IV), at the cost of the reasoning tokens. b) Internal validity: The trained methods choose among different numbers of layers: the probe and LAT among all 25– 49, the causal methods among the 4–7 layers at which DAS was trained, since each subspace takes several GPU hours. A wider choice can find a better layer, but under shift the layer best in distribution need not transfer, so the effect can go either way. Under shift, however, the causal checkpoints often lie above the probe even at the probe’s best layer (Figure 2). Counterfactual simulation uses one fixed set of 16 bases; drawing them from different problems changed the AUROC of one checkpoint by at most 0.01 AUROC. Reruns of the same configuration on different machines differ by up to 0.02 AUROC on identical rows, so we do not interpret smaller differences between single cells. EigenScore needs five nonempty generations per problem (ten on BCB), so the problemlevel comparison leaves out the problems with fewer: 40 for Qwen3-Coder, none of them solved, and 143 for gpt-oss, 47 of them solved. c) External validity: We study two open-weight models on competitive-programming problems from LiveCodeBench, on a subset of the same problems in Python, C++ and Java, which isolates the language, and on more practical Python tasks from BigCodeBench. Other models, sampling settings or repository-level tasks may behave differently. IX. C ONCLUSION We compared execution-free predictors of code correctness from four families under benchmark and language shift, for two targets: whether a program is correct and whether the model’s verdict on it can be trusted. Causal localization,

adapted from natural-language question answering, gives the most robust predictor of the model’s verdict under language shift on one of two models, and its gain lies in the learned subspace, so a simple probe on it suffices. Beyond that, how well a predictor generalizes depends on the target and the type of shift: no verdict predictor transfers reliably between benchmarks, while program correctness transfers better. Much of what the predictors recognize is the problem rather than the candidate, which makes them better suited to deciding which problems to verify than to choosing among candidates. Future work should test whether these signals can gate test execution and stronger judges in agentic settings, and whether a few labels from the target setting suffice to adapt them. R EFERENCES [1] Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings. OpenReview.net, 2017. [2] Dylan Bouchard, Mohit Singh Chauhan, Zeya Ahmad, and Ho-Kyeong Ra. Functional entropy: Predicting functional correctness in LLMgenerated code with uncertainty quantification. CoRR, abs/2605.28500, 2026. [3] Tuan-Dung Bui, Thanh Trong Vu, Thu-Trang Nguyen, Son Nguyen, and Hieu Dinh Vo. Correctness assessment of code generated by large language models using internal representations. J. Syst. Softw., 230(C), December 2025. [4] Viola Campos, Robin Kuschnereit, and Adrian Ulges. Multicalibration for llm-based code generation. In Proceedings of the 1st International Workshop on AI for Software Quality Evaluation - Judgment, Metrics, Benchmarks, and Beyond, AI-SQE ’26, page 61–71, New York, NY, USA, 2026. Association for Computing Machinery. [5] Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. INSIDE: LLMs’ internal states retain the power of hallucination detection. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. [6] Seonglae Cho, Zekun Wu, Kleyton Da Costa, and Adriano Koshiyama. The confidence manifold: Geometric structure of correctness representations in language models, 2026. [7] Giuseppe Crupi, Rosalia Tufano, Alejandro Velasco, Antonio Mastropaolo, Denys Poshyvanyk, and Gabriele Bavota. On the effectiveness of llm-as-a-judge for code generation and summarization. IEEE Trans. Software Eng., 51(8):2329–2345, 2025. [8] Yangruibo Ding, Yanjun Fu, Omniyyah Ibrahim, Chawin Sitawarin, Xinyun Chen, Basel Alomair, David Wagner, Baishakhi Ray, and Yizheng Chen. Vulnerability detection with code language models: How far are we? In 47th IEEE/ACM International Conference on Software Engineering (ICSE), 2025. [9] Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks. In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 9574–9586, 2021. [10] Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah D. Goodman. Finding alignments between interpretable causal variables and distributed neural representations. In Francesco Locatello and Vanessa Didelez, editors, Causal Learning and Reasoning, 1-3 April 2024, Los Angeles, California, USA, volume 236 of Proceedings of Machine Learning Research, pages 160–187. PMLR, 2024. [11] Satchel Grant, Simon Jerome Han, Alexa R. Tartaglini, and Christopher Potts. Addressing divergent representations from causal interventions on neural networks, 2025. [12] David Gros and Prem Devanbu. Localized calibrated uncertainty in code language models, 2025. [13] Junda He, Jieke Shi, Terry Yue Zhuo, Christoph Treude, Jiamou Sun, Zhenchang Xing, Xiaoning Du, and David Lo. LLM-as-a-judge for software engineering: Literature review, vision, and the road ahead, 2025.

[14] Weilin He, Arindam Sharma, and Cristina David. Using semantic distance to estimate uncertainty in llm-based code generation. CoRR, abs/2605.09023, 2026. [15] Yicheng He, Zheng Zhao, Kaiyu Zhou, Bryan Dai, Jie Fu, and Yonghui Yang. CodeCircuit: Toward inferring LLM-generated code correctness via attribution graphs, 2026. [16] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. [17] Seyedrebvar Hosseini, Burak Turhan, and Dimuthu Gunarathna. A systematic literature review and meta-analysis on cross project defect prediction. IEEE Transactions on Software Engineering, 45(2):111–147, 2019. [18] Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. [19] Jing Huang, Junyi Tao, Thomas Icard, Diyi Yang, and Christopher Potts. Internal causal mechanisms robustly predict language model out-of-distribution behaviors. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 1319, 2025, volume 267 of Proceedings of Machine Learning Research. PMLR / OpenReview.net, 2025. [20] Yuheng Huang, Lei Ma, Keizaburo Nishikino, and Takumi Akazaki. Risk assessment framework for code llms via leveraging internal states. In Leonardo Montecchi, Jingyue Li, Denys Poshyvanyk, and Dongmei Zhang, editors, Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering, FSE Companion 2025, Clarion Hotel Trondheim, Trondheim, Norway, June 23-28, 2025, pages 432–443. ACM, 2025. [21] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. [22] Siddhartha Jain, Xiaofei Ma, Anoop Deoras, and Bing Xiang. Lightweight reranking for language model generations. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 1116, 2024, pages 6960–6984. Association for Computational Linguistics, 2024. [23] Hongchao Jiang, Yiming Chen, Yushi Cao, Hung-Yi Lee, and Robby T. Tan. Codejudgebench: Benchmarking llm-as-a-judge for coding tasks. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens, editors, Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2026, San Diego, California, United States, July 2-7, 2026, pages 19416–19448. Association for Computational Linguistics, 2026. [24] Jaehun Jung, Faeze Brahman, and Yejin Choi. Trust or escalate: LLM judges with provable guarantees for human agreement. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. [25] Saurav Kadavath, Tom Conerly, Amanda Askell, Thomas Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zachary Dodds, Nova Dassarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, John Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom B. Brown, Jack Clark, Nicholas Joseph, Benjamin Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. Language models (mostly) know what they know. ArXiv, abs/2207.05221, 2022. [26] Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. When can LLMs Actually correct their own mistakes? A critical survey of self-correction of LLMs. Trans. Assoc. Comput. Linguistics, 12:1417– 1440, 2024.

[27] János Kramár, Joshua Engels, Zheng Wang, Bilal Chughtai, Rohin Shah, Neel Nanda, and Arthur Conmy. Building production-ready probes for Gemini, 2026. [28] Yi-Long Lu. Diagnosing correctness probes under self-judgement confounding, 2026. [29] William Lugoloobi, Thomas Foster, William Bankes, and Chris Russell. LLMs encode their failures: Predicting success from pre-generation activations, 2026. [30] Aleksandar Makelov, Georg Lange, Atticus Geiger, and Neel Nanda. Is this the subspace you are looking for? an interpretability illusion for subspace activation patching. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. [31] Alex Mallen, Madeline Brumley, Julia Kharchenko, and Nora Belrose. Eliciting latent knowledge from “quirky” language models, 2024. [32] Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. In Conference on Language Modeling (COLM), 2024. [33] Alex McKenzie, Urja Pawar, Phil Blandfort, William Bankes, David Krueger, Ekdeep Singh Lubana, and Dmitrii Krasheninnikov. Detecting high-stakes interactions with activation probes. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2-7, 2025 / Mexico City, Mexico, November 30 December 5, 2025, 2025. [34] Maxime Méloux, Silviu Maniu, François Portet, and Maxime Peyrard. Everything, everywhere, all at once: Is mechanistic interpretability identifiable? In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. [35] Jiwon Moon, Yerin Hwang, Dongryeol Lee, Taegwan Kang, Yongil Kim, and Kyomin Jung. Don’t judge code by its cover: Exploring biases in LLM judges for code evaluation. In Vera Demberg, Kentaro Inui, and Lluı́s Marquez, editors, Findings of the Association for Computational Linguistics: EACL 2026, Rabat, Morocco, March 24-29, 2026, Findings of ACL, pages 1364–1389. Association for Computational Linguistics, 2026. [36] Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Reichart, Idan Szpektor, Hadas Kotek, and Yonatan Belinkov. Llms know more than they show: On the intrinsic representation of LLM hallucinations. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. [37] Arjun Panickssery, Samuel R. Bowman, and Shi Feng. LLM evaluators recognize and favor their own generations. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024. [38] Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024. [39] Jasmine Qi, Danylo Dantsev, and Muyang Sun. VERDI: Single-call confidence estimation for verification-based LLM judges via decomposed inference, 2026. [40] Bhaktipriya Radharapu, Eshika Saxena, Kenneth Li, Chenxi Whitehouse, Adina Williams, and Nicola Cancedda. Calibrating LLM judges: Linear probes for fast and reliable uncertainty estimation. In Yunyao Li, Georg Rehm, and Mei Tu, editors, Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track), ACL 2026, San Diego, California, USA, July 2-7, 2026, pages 193–216. Association for Computational Linguistics, 2026. [41] Mohit Raghavendra, Anisha Gunjal, Bing Liu, and Yunzhong He. Agentic rubrics as contextual verifiers for SWE agents. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens, editors, Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2026, San Diego, California, United States, July 2-7, 2026, pages 15265–15290. Association for Computational Linguistics, 2026. [42] Pranav Rakasi, Maanas Lalwani, Arnav Srivastava, Arya Palanivel, Tinuade Adeleke, Ruizhe Li, and Sean Wu. When uncertainty isn’t enough: An empirical study of self-correction in code generation. CoRR, abs/2608.14659, 2026.

[43] Ravishka Rathnasuriya and Wei Yang. Characterizing overconfident failure in LLM-based code generation, 2026. [44] Francisco Ribeiro, Claudio Spiess, Premkomar Thomas Devanbu, and Sarah Nadi. On llms’ internal representation of code correctness. Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering, 2025. [45] Francisco Ribeiro, Claudio Spiess, Premkumar Devanbu, and Sarah Nadi. On llms’ internal representation of code correctness. In Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering, ICSE ’26, page 1160–1172, New York, NY, USA, 2026. Association for Computing Machinery. [46] Arindam Sharma and Cristina David. Assessing correctness in llm-based code generation via uncertainty estimation. CoRR, abs/2502.11620, 2025. [47] Yuling Shi, Caiqi Zhang, Yuexian Li, Haopeng Wang, Yeheng Chen, Nigel Collier, and Xiaodong Gu. Code is more than text: Uncertainty estimation for code generation. CoRR, abs/2606.09577, 2026. [48] Claudio Spiess, David Gros, Kunal Suresh Pai, Michael Pradel, Md. Rafiqul Islam Rabin, Amin Alipour, Susmit Jha, Prem Devanbu, and Toufique Ahmed. Calibration and correctness of language models for code. In 47th IEEE/ACM International Conference on Software Engineering, ICSE 2025, Ottawa, ON, Canada, April 26 - May 6, 2025, pages 540–552. IEEE, 2025. [49] Ponhvoan Srey, Xiaobao Wu, Cong-Duy Nguyen, Quang Minh Nguyen, Duc Anh Vu, and Anh Tuan Luu. From signals to transfer: A factorised study of probe-based uncertainty estimation in large language models, 2026. [50] Thomas Jean-Michel Valentin, Ardi Madadi, Gaetano Sapia, and Marcel Böhme. Incoherence as oracle-less measure of error in llm-based code generation. In Sven Koenig, Chad Jenkins, and Matthew E. Taylor, editors, Fortieth AAAI Conference on Artificial Intelligence, ThirtyEighth Conference on Innovative Applications of Artificial Intelligence, Sixteenth Symposium on Educational Advances in Artificial Intelligence, AAAI 2026, Singapore, January 20-27, 2026, pages 33305–33313. AAAI Press, 2026. [51] Thanh Trong Vu, Tuan-Dung Bui, Thu-Trang Nguyen, Son Nguyen, and Hieu Dinh Vo. Model-agnostic quality assessment for llm-generated code via dynamic internal representation selection. J. Syst. Softw., 237:112852, 2026. [52] Zhuofan Josh Ying, Shauli Ravfogel, Nikolaus Kriegeskorte, and Peter Hase. The truthfulness spectrum hypothesis, 2026. [53] Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, James Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, and et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. [54] Thomas Zimmermann, Nachiappan Nagappan, Harald Gall, Emanuel Giger, and Brendan Murphy. Cross-project defect prediction: A large scale experiment on data vs. domain vs. process. In Proceedings of the 7th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE), pages 91–100, 2009. [55] Andy Zou, Long Phan, Sarah Li Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, AnnKathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down approach to AI transparency. CoRR, abs/2310.01405, 2023.

Related documents

• UNBIND: UNlearning By INference-time Directional Steering for Code LLMs#1130755
• To Explore The Strange New World Beyond Data Distribution: System Behavior, Causality Tax, and Non-causal Base Model#1187153
• Precedent or Intent? What Coding Agents Follow When Evolving Existing Code#1489036
• Precedent or Intent? What Coding Agents Follow When Evolving Existing Code#1483090
• Faster Block-Diffusion Serving with Distribution-Free Risk Guarantees#1118960