Uncovering Why LLMs Fail: From Internal Mechanisms to Adversarial Reasoning Trajectories

Uncovering Why LLMs Fail: From Internal Mechanisms to Adversarial Reasoning Trajectories Zirui Ding1,2 , Xinxin Fan2,3 , Hao Yuan1,4 , Shaoye Luo2,3 , Men Niu1,2 , Yunfeng Lu5 1 Institute of Computing Technology, Chinese Academy of Sciences 2 University of Chinese Academy of Sciences

arXiv:2610.37369v1 [cs.AI] 27 Oct 2026

3 State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences 4 ShanghaiTech University 5 Beihang University

Abstract

ported facts, fabricated evidence, or invalid reasoning, as documented by recent hallucination and factuality surveys [13, 43]. In ordinary use, such outputs can propagate misinformation and mislead downstream decisions. In security-sensitive settings, HarmBench [28] and JailbreakBench [6] standardize the measurement of whether manipulated prompts turn safetyaligned models from refusal to harmful compliance. The cost is therefore not limited to a wrong answer: a fluent but ungrounded explanation can also make the failure difficult for users or automated detectors to identify [22, 27]. Research on trustworthy and safe LLMs has identified several non-exclusive sources of these failures. TruthfulQA shows that models can reproduce popular misconceptions learned from their training distribution [23]. Lost-in-themiddle analyses [24] and distraction tests [38] show that task-relevant evidence can be overlooked when it is embedded in long or irrelevant context. Calibration studies examine whether models can estimate the correctness of their own answers [18, 47], whereas SelfCheckGPT detects unsupported passages through disagreement among sampled responses [27]. Reasoning traces introduce another gap: biasing features can change an answer without being acknowledged in the generated chain of thought [41], and intervention-based studies find that textual rationales do not always mediate the final prediction [20,36]. In safety-aligned models, adversarial prompts can redirect intermediate hidden-state processing [51], exploit safety signals concentrated in the chat-template region [21], alter refusal-related representations [2], or hijack attention from safety-relevant template tokens [4]. These findings suggest that hallucination and unsafe compliance can involve interacting failures of evidence use, reasoning coordination, and internal control rather than one isolated defect. Nevertheless, LLM trustworthiness and safety are still commonly evaluated from endpoint outputs, and an endpoint label alone does not reveal how the result was produced through successive Transformer layers and generation steps [43,51]. Prior studies show that the same endpoint failure label can arise from heterogeneous failure patterns: task-relevant premises may be overlooked when evidence is embedded in long or

Large language models (LLMs) are usually evaluated by finaloutput correctness or safety, yet an endpoint label does not reveal how a failure develops throughout the reasoning process. We therefore ask whether ordinary reasoning failures and adversarially induced failures can be characterized using the same internal coordinates. Towards this end, we in this paper study a white-box diagnostic framework CauTrace to disclose the root-cause of why LLM fails from three fundamental inspections: Functional States, Propagation of Representation, Temporal-State Coordination and Reasoning Trajectory. First, seven functional states are creatively defined to represent the support and constraint processing in LLMs. Second, the propagation of representation change measures how the changes among the seven states persist across subsequent Transformer blocks. Third, temporal-state coordination and trajectory topology characterize when the commitment and verification occur relative to the support, and how the dominant state sequence evolves over the full inference process. Finally, we evaluate CauTrace on both clean reasoning failures and paired adversarial perturbations. Across Qwen and LLaMA-family models, the failed traces show weaker Attention–MLP agreement on support-state updates and less verification activity relative to commitment, while the exact representation-level signatures and timing shifts vary by architecture. Held-out repair and steering tests demonstrate consistent operational signals that were in line with our claims for certain models and specific settings, with aligned steering producing the predicted directional changes in process measures. Overall, CauTrace provides a unified diagnostic framework for analyzing failure dynamics through changes in support, commitment, and verification, which would shed light on the in-depth research on LLM security.

1

Introduction

Hallucinations and harmful generations can make large language models (LLMs) appear certain while producing unsup1

distracting contexts [24, 38]; parametric falsehoods or unsupported associations may dominate a response [23]; multi-hop reasoning may omit a required inference or fail to integrate the final supporting fact [11, 36]; and explicit task constraints may be violated [17]. Moreover, answer commitment may be driven by shortcut features rather than by the stated support, with the reported reasoning merely rationalizing that choice [41]; a later self-check may then fail to repair the answer or may even degrade it [12]. The same ambiguity arises in safety evaluation: an unsafe completion establishes that a safeguard failed, but not which part of the internal generation process was reorganized by the adversarial prompt [6,51]. Consequently, endpoint evaluation alone leaves a fundamental diagnostic question unanswered: how can failures be traced to changes within the model’s internal generation pipeline? Existing research therefore provides four important but partial analytic views. Endpoint benchmarks measure factuality through learned misconceptions in TruthfulQA [23], generated and human-annotated hallucinations in HaluEval [22], or standardized harmful-behavior evaluation in HarmBench [28] and JailbreakBench [6]. Calibration and consistency methods estimate whether an answer is unreliable [18, 27]; chainof-thought studies test whether textual rationales faithfully mediate predictions [36, 41]; and mechanistic methods decode hidden representations [3, 10] or localize safety-relevant directions [2]. These views are rarely connected by a common process-level unit. Endpoint scores can merge distinct failure routes, textual rationales need not expose the operative computation, and a localized representation does not by itself explain when support, commitment, and verification become miscoordinated over an entire generation. A unified analysis therefore needs to connect where a change persists, when functional roles switch or lose coordination, and which whole-trace route produces the endpoint. In this work, our goal is therefore not to propose another endpoint detector or a stronger attack, but to ask a nontrivial question: Can non-adversarial reasoning failure and adversarially-induced reasoning failure be described with a common set of reasons? Surrounding this question, we treat a well-formed generation as a reasoning trajectory in which task-relevant support and constraint are established before answer commitment becomes dominant, and later verification remains able to confirm or revise the candidate answers. Failures can therefore appear as weak support continuity, shifted commitment/verification timing, frequent state switching, or low verification activity relative to commitment, which offers an operational and quantitative criterion for analyzing reasoning failures. Generally, the final LLM-output answers whether the response is incorrect, unsupported, or unsafe, but cannot locate which part of the internal process changed; the same output label can also hide different state-transition routes. Our proposed CauTrace first maps each generation to seven functional-state coordinates describing support/constraint pro-

cessing, answer commitment, and verification. It then analyzes three complementary properties: whether representation changes persist across subsequent Transformer blocks, whether the timing of support, commitment, and verification remains coordinated, and which trajectory topology emerges over the full generation trace. In a nutshell, three main contributions are involved: • Reasoning-Process Formulation. We creatively define seven radical functional-state coordinates to abstract the entire reasoning process, by which the support/constraint processing, answer commitment, and verification can be formulated appropriately. • Three-Level Failure-Diagnostic Analytics. We construct the propagation of representation mechanism across Transformer blocks, temporal coordination among functional states, and whole-trace trajectory topology, by which the fine-grained representation changes can be captured microscopically. • Empirical Verification. Multi-facet experimentoriented evidences across the popular Qwen and LLaMA-family models demonstrate that: i) the failed traces differ from successful traces in support-state update agreement and relative verification activity; ii) that even the same output label can follow different reasoning trajectory chains; and iii) that harmful-benignpaired adversarial flips are accompanied by larger internal-state change.

2 2.1

Related Work Reliability and Reasoning Analytics

Hallucination analytics first requires an operational definition of failure. Surveys distinguish factuality, faithfulness to a source, evidence support, and task correctness as related but non-equivalent targets [13,15,43,50]. At the benchmark level, TruthfulQA tests whether models reproduce common misconceptions learned from human text [23], whereas HaluEval combines generated and human-annotated samples to evaluate hallucination recognition across general queries and taskspecific settings [22]. FActScore moves from response-level labels to claim-level analysis by decomposing long-form generations into atomic facts and checking each fact against a knowledge source [30]. Related task analytics isolate other endpoint failure dimensions: FollowBench separates finegrained instruction constraints [17], and NovelHopQA varies reasoning depth and context length to identify missed finalhop integration and long-range drift [11]. These benchmarks improve diagnostic granularity at the output level, but the same failed label or unsupported claim can still be produced through different internal routes. 2

A complementary line examines whether the model itself provides signals of unreliability. Kadavath et al. [18] derive probability-based estimates of answer correctness and knowledge confidence from model outputs [18]; Yin et al. [47] evaluate self-knowledge across answerable and unanswerable questions [47]; and SelfCheckGPT [27] uses disagreement among stochastic generations as a black-box signal of unsupported content [27]. Verification is not automatically corrective, however, intrinsic self-correction without external feedback can fail to improve reasoning and can even reduce accuracy [12]. These methods provide uncertainty or consistency scores, but they do not identify which stage of evidence use, answer formation, or checking produced the score. Reasoning analytics uses generated intermediate text as a more detailed behavioral trace. Scratchpads externalize intermediate computation [33]; chain-of-thought and zero-shot reasoning prompts elicit multi-step rationales [19, 44]; and self-consistency aggregates answers across multiple sampled chains [42]. Faithfulness research separates the plausibility of a rationale from whether it actually mediates the prediction [14]. Biasing features can systematically change answers while remaining absent from the stated rationale [41]; rationale perturbation and truncation tests reveal task- and modeldependent faithfulness [20]; and causal mediation analysis finds that models do not always rely on their own intermediate steps when producing the final answer [36]. Consequently, textual reasoning traces provide valuable behavioral evidence but cannot be treated as direct records of internal computation.

2.2

cepts [52]. Arditi et al. identify a direction that mediates refusal and validate it through activation addition and ablation [2]; subsequent studies describe refusal using concept cones [45] or multiple approximately orthogonal safety dimensions [35]. Intermediate-state analysis compares aligned and jailbroken generations across layers [51]. More localized studies show that safety decisions can be anchored in the chat-template region [21] and that universal suffixes redirect attention away from safety-relevant template tokens [4]. Activated-concept analysis has also been used for jailbreak defense [49], while PromptLocate localizes prompt-injection influence within the model pipeline [16]. These methods reveal safety-relevant representations or information-flow vulnerabilities, but most are organized around a particular refusal mechanism, attack family, or localized component rather than a task-general generation process.

2.3 Mechanistic and Representation Analytics White-box analytics supplies the primitives for reading and intervening on model internals. Contrast-consistent learning extracts latent truth-related structure without labeled answers [5], while the Tuned Lens decodes layer-wise predictive distributions and exposes how token predictions evolve through the residual stream [3]. Patchscopes generalizes projection and intervention methods by transplanting hidden representations into a target prompt so that the model can express their information in natural language [10]. TransformerLens [31] provides reusable instrumentation for caching, decomposing, and patching activations [31]. Together, these methods determine what information is decodable at a layer or how an intervention changes a selected behavior. Mechanistic studies then seek smaller computational units. Feed-forward layers can operate as key–value memories that promote factual concepts [9]; induction-head analyses identify attention circuits that implement repeated-pattern completion [34]; and causal tracing with ROME localizes and edits factual associations [29]. Automated Circuit Discovery systematizes activation-patching workflows by searching for the edges that form a behavior-specific circuit [7]. Such methods provide component- or circuit-level explanations, but their units are usually selected for one behavior and do not by themselves describe how functional roles coordinate over an entire generation trace. Summary & Motivation. Existing analytics offer increasingly precise endpoint scores, attack measurements, hiddenrepresentation decoders, and localized safety mechanisms, but rarely connect these resolutions into one generation-level account of ordinary and adversarial failures. CauTrace addresses this gap by organizing a white-box evidence chain around seven functional-state/process coordinates and jointly analyzing Propagation of Representation, Temporal-State Coordination, and Reasoning Trajectory, rather than proposing another endpoint benchmark, probe, or refusal direction.

Adversarial-Prompt and Safety Analytics

Security analytics first characterizes attacks and their behavioral endpoints. HarmBench standardizes harmful behaviors, red-teaming methods, and refusal classifiers within one evaluation protocol [28]; JailbreakBench defines a reproducible threat model and artifact repository for comparing jailbreak attacks and defenses [6]; and StrongREJECT grades both harmfulness and response informativeness so that empty or non-responsive outputs are not counted as successful jailbreaks [40]. Many-shot jailbreaking demonstrates that long contexts containing many demonstrations can progressively erode refusal behavior [1]. Corpus-based studies characterize in-the-wild jailbreak prompts [37] and analyze transferable semantic attack patterns [48], whereas AutoDAN automates the search for stealthy jailbreak instructions [25]. Prompt-injection work separately formalizes attacker objectives and evaluation settings [26], including optimization attacks against LLM-as-a-judge systems [39]. This line establishes whether an attack succeeds and under which threat model, but its principal analytic unit remains the prompt– response outcome. A second line analyzes the internal organization of safety behavior. Representation Engineering extracts populationlevel activation directions associated with behavioral con3

Figure 1: CauTrace overview. CauTrace maps clean/adversarial traces to seven functional-state coordinates and analyzes (1) Propagation of Representation across Transformer blocks; (2) Temporal-state Coordination among support, commitment, and verification; and (3) Reasoning Trajectory over the full generation trace.

3

Methodology

3.1

Functional-State Coordinates

Inspired by decompositions of problem-solving into representation, knowledge access, operation, monitoring, and answer confirmation [8, 32], we define seven functional-state/process coordinates for LLM reasoning. They address three questions: what supports or constrains the answer? when does the model commit to a candidate answer? and can the later computation verify or revise that commitment? Table 1 explicitly maps each state to one of the three research questions (Que. I-III): PB, KR, RC, OC, and FRT represent support or constraint formation; AC describes answer selection; and SV indicates post-commitment verification and revision. Why These Seven States. Three considerations motivate this seven-state decomposition: i) Coverage of Roles of LLM Reasoning. The seven-state decomposition covers recurring functional roles in reasoning, including binding, evidence access and composition, operational computation, rule tracking, commitment, and checking.; ii) Operational Separability. Our defined seven states enables to adequately decouple the entangled roles under task-grounded weak supervision; and iii) Failure-Type Diagnostic Correction. The seven states can validly distinguish the categories of LLM failures, also verified by the subsequent experiments. We here do not claim that these seven states are a unique or minimal cognitive decomposition. The first five states can overlap and need not occur in a fixed order. They provide a general task decomposition at a suitable granularity that can be used to deeply inspect the hidden-state reasoning process in a microscopic

Fig. 1 sketches the overview framework, the propose CauTrace treats each generation as one hidden-state trajectory and uses the seven functional-state coordinates as a reasoning chain to maps the generated trace. To unveil the inherent reasoning process, we inspect three-level linked analytics: Propagation of Representation. It examines whether a local change is retained or canceled/absorbed alongside Transformer’s Attention and MLP updates; ii) Temporal-State Coordination. It is used to reconstruct how the same coordinates switch and align dynamically over the generation time, including whether support and constraint precede answer commitment and whether the later verification can revise it back; and iii) Reasoning Trajectory. It integrates the local and temporal measurements into a whole-trace route for further observation and judgement on the output trajectory.Together, these three levels provide a unified view of the reasoning process across different analytical resolutions. The above three-level analytics form a measurement hierarchy rather than three independent feature inspections. Representation analytics aims to identify where a deviation remains available to later computation; temporal-state analysis identifies when state-links, commitment, and verification become miscoordinated; and trajectory chains specify which reasoning chain exactly lead to the final results. Therefore, in view of these three-level analytics, they are hierarchical and mutually complementary to calibrate the reasoning trace. 4

attn S the seven-state set and s ∈ S a specific state. Let pin s,t,l , ps,t,l ,

manner. Let PB, KR, RC, OC, and FRT respectively denote the support/constraint family,

S + = {PB, KR, RC, OC, FRT}.

and pout s,t,l represent the state-coordinate scores at the block input, after the Attention sublayer, and after the MLP sublayer, respectively. We define

(1)

The family captures premise binding, knowledge retrieval, reasoning composition, operational computation, and rule tracking. The defined AC can convert these signals into a candidate answer; SV can evaluate whether the candidate answer remains supported or revisable. Hence, the process failure is defined by abnormal coordination rather than by a missing state. A probe score ps (t) measures the evidence for an operational role in the hidden representation; it does not indicate the entry into a discrete reasoning stage. CauTrace therefore allows overlapping state scores and ostensibly "non-sequential" transitions, despite different categories of LLM failures are subject to specified reasoning trajectories. To make these functional coordinates concrete and clear, we provide a simple reasoning example below.

attn in ∆attn s,t,l = ps,t,l − ps,t,l ,

(2)

mlp attn ∆s,t,l = pout s,t,l − ps,t,l ,

(3)

 
mlp As,t,l = sign ∆attn s,t,l sign ∆s,t,l ,

(4)

mlp

where ∆attn s,t,l and ∆s,t,l are Attention-side and MLP-side changes in the corresponding state-coordinate score. As,t,l ∈ {−1, 0, 1} records whether the two sublayers move the state in the same, opposite, or zero direction. We average As,t,l over support states and analyzed layers to obtain Āsup (t). A smaller Āsup indicates lower agreement between Attention and MLP updates on support-related state scores. For value-projected Attention, we analyze the projected representations rather than the parameter matrix. Let e h j,l be the normalized residual representation of token j before AtV the value-projection matrix of head tention in layer l, and Wl,h h. Thus, the head-level value vector can be defined as

State Walkthrough Example Question. Can a person born in 2000 legally vote in the United States in 2017? Target Output. No State Interpretation.

V vVj,l,h = e h j,l Wl,h ∈ Rdv ,

where dv is the value-head dimension. At generation step t, let It denote the token-layer-head units used for spectral analysis. We stack the corresponding value vectors into h i Vt = (vVj,l,h )⊤ ∈ Rm×dv , (6)

1. Premise Binding: bind the birth year (2000), decision year (2017), and jurisdiction (United States). 2. Knowledge Retrieval: recall the voting-age rule that a person must be at least 18. 3. Reasoning Composition: combine the age rule with the question conditions. 4. Operational Computation: compute that the person is 17 in 2017. 5. Formal Rule Tracking: keep the answer in the required Yes/No decision space. 6. Answer Commitment: select No once support is sufficient. 7. Self-Verification: check that the age calculation and the voting rule are consistent with No.

( j,l,h)∈It

where m = |It |. The local second-moment matrix is ΣVt = (Vt )⊤Vt /m. Let λV1,t ≥ λV2,t ≥ · · · denote the eigenvalues of ΣVt , sorted in non-increasing order. We respectively define the leading spectral concentration, normalized spectrum, and step-to-step spectral change as ρVt = πVt,k =

3.2

(5)

Propagation of Representation

λV1,t ∑ j λVj,t λVk,t ∑ j λVj,t

,

(7)

,

(8)

DVKL (t) = DKL πVt ∥πVt−1 ,

The propagation of representation aims to analyze whether a local change in support state signals is canceled/absorbed or persists across later Transformer updates. That is, the Attention–MLP update agreement measures whether the two sublayers move support-state in the same direction; the valuespectrum statistics measure changes in the value-projected representation of the state probes; and the alignment with the Answer Commitment (AC) probe tests whether the dominant value-space direction becomes more commitment-related. Representation-Level Deviation Score. Let t ∈ {1, . . . , T } denote a generation step, l ∈ L an analyzed Transformer layer,

(9)

where ρVt is the leading spectral concentration, πVt,k is the normalized spectral mass, and DVKL (t) is the step-to-step spectral change. Let uVac represent the answer-commitment probe direction in the same value space and vV1,t the leading eigenvector, thus their alignment degree can be defined as Γac,V (t) = cos uVac , vV1,t

.

(10)

The absolute cosine removes the arbitrary sign of the eigenvector. Γac,V (t) therefore measures alignment degre between 5

Table 1: Seven functional-state coordinates organized by three diagnostic questions. Seven States Premise Binding (PB) Knowledge Retrieval (KR) Reasoning Composition (RC) Operational Computation (OC) Formal Rule Tracking (FRT) Answer Commitment (AC) Self-Verification (SV)

Functional Roles Bind entities, conditions, and prompt constraints Activate relevant factual or commonsense knowledge Compose evidence across hops or intermediate claims Execute numerical or symbolic operations Maintain task rules, labels, and output constraints Select and stabilize a candidate answer Check, revise, or reject the committed candidate

the dominant value direction and the AC probe direction. We combine the four indicators into Rstr (t) = β1 [δ1 − Āsup (t)]+̂ + β2 [δ2 − ρVt ]+̂ +β3 [DVKL (t) − δ3 ]+̂ + β4 [Γac,V (t) − δ4 ]+̂

describe the ordering, persistence, and revision opportunity of the reasoning chain. Let ps (t) be the causal-pooled probe score of state s at generation step t, and let θs denote state-specific activation threshold. The thresholds are calibrated on the clean-sample reasoning split and fixed before held-out or adversarial evaluation. The reasoning onset of s is set as

(11)

where [·]+̂ = max(·, 0). Thresholds are calibrated on correct reasoning traces: lower-tail thresholds are used for support-state agreement/concentration quantities, and uppertail thresholds for change/alignment quantities. Each component is normalized by its interquartile range. We use equal weights (βk = 1/4) by default and evaluate weight sensitivity in Appendix B.1. Rstr is a composite representation-level deviation score instead of one causal factor.

τs = min{t | ps (t) ≥ θs , ps (t + 1) ≥ θs }.

(12)

The two-step condition suppresses the transient-score spikes. The dominant state sequence is ct = arg maxs ps (t); and its switch count and aggregate support signal are defined as T

Ω = ∑ 1[ci ̸= ci−1 ],

(13)

i=2

Diagnostic Claim I – Representation Propagation. A failure is not defined by one isolated activation difference. Rather, support-related changes can survive the Attention–MLP update sequence, remain visible in later value-projected representations, and become aligned with AC instead of being canceled by subsequent blocks.

S(t) =

1

|S + | s∈∑ S+

ps (t).

(14)

For any nonnegative signal u = (u1 , . . . , uT ), we define its center of mass as CoM(u) =

3.3

Questions Que.I Que. I Que. I Que. I Que. I Que. II Que. III

Temporal-State Coordination

T t ut ∑t=1 , T ∑t=1 ut + ε

(15)

where ε > 0 is a small stabilizer. Let pcom (t) = pAC (t) and pver (t) = pSV (t). The aggregate-support onset τsup is computed by applying the same two-step rule to S(t) with threshold θsup , calibrated like θs . We quantify commitment/support timing, verification/commitment timing, and relative verification activity by

At this level analytics, the seven states are treated as "nodes" in a sophisticated stepwise-switching chain. This part is not a comparison of seven isolated score magnitudes, but to reconstruct the intrinsic correlation by which the support and constraint states switch into one another, the point at which the chain enters AC, and whether the subsequent SV activity returns to support formation or revises the committed candidate. We therefore analyze both the state-transition correlation— including repeated resets, skipped support links, and commitment/verification cycles—and their temporal-sate coordination, such as ordering, onset, persistence, and relative timing. We operationalize the stepwise-switching reasoning chain with four complementary measurements: activation onset identifies the first sustained appearance of a state; activityweighted mean time gives the center of mass of its score curve, dominant-state transitions record the directed links and switching frequency, and relative verification activity measures whether the verification remains active in proportion to commitment. Taken together, these statistics can appropriately

gcs = CoM(pcom ) − CoM(S),

(16)

gvc = CoM(pver ) − CoM(pcom ),

(17)

Rvc =

Aver , Acom + ε

T

Ax =

1 ∑ xt , T t=1

(18)

where gcs < 0 is the early-commitment direction, while larger positive gvc indicates later verification relative to commitment. Rvc is the verification-to-commitment activity ratio. Correspondingly, we can define the aggregate reasoning processdeviation score as Ω Rproc = α1 1[τcom < τsup ] + α2 [−gcs ]+ + α3 T (19) + α4 [gvc ]+ + α5 [1 − Rvc ]+ , 6

where τcom is the onset of AC.Continuous risk terms are normalized to [0,1] using the 5th and 95th percentiles of correct reasoning traces and then clipped. We choose percentile-based scaling rather than min-max normalization to reduce sensitivity to rare outliers while preserving the typical variation of successful reasoning processes.The default weights are uniform (αk = 1/5). The score above combines early-commitment, support–commitment timing inversion, frequent dominantstate switching, late verification, and low verification activity relative to commitment.

Diagnostic Claim III – Trajectory Heterogeneity. An endpoint label does not identify a unique failure route. Trajectory Topology preserves differences in support continuity, commitment ordering, rule preservation, state recurrence, verification effectiveness, and continuation length, thereby separating multiple reasoning-process organizations within the same output category.

4 4.1

Diagnostic Claim II – Temporal-State Coordination. Reliability of LLM reasoning depends on the coordination of the state chain, not on the magnitude of any one singlestate. Failed traces reorganize the handoffs among support/constraint states, AC, and SV; the onset shifts, temporal gaps, and switching patterns remain model- and failure typedependent.

3.4

Experiment Evaluation Configuration

Procedure. CauTrace is evaluated as an offline diagnostic framework. For each generation, we (1) collect hidden states and module representations without changing decoding, (2) apply fixed probes to obtain probe score {ps (t)}s,t , (3) compute the representation-level, temporal-state, and trajectory diagnostics from Section 3, and (4) compare successful and failed generation traces, matched prompt pairs, and model families under fixed development-set calibration. The resulting output is a process-level diagnostic profile rather than a new endpoint classifier. The following experiments are conducted surrounding four research questions that correspond to the aforementioned three diagnostic claims:

Reasoning Trajectory

To draw the reasoning state microscopically, we summarize each generation as a reasoning trajectory-feature vector: φ(x1:T ) = (τsup , τcom , τver , Ω, gcs , gvc , Rvc , Lrule , S̄, T ), (20) where x1:T is the sequence of state-coordinate scores, τver = τSV , Lrule is the duration of weak rule/operation tracking, S̄ = AS denotes the mean support strength, and T stands for the generation length. The feature groups correspond to support (τsup , S̄), commitment ordering (τcom , gcs ), verification (τver , gvc , Rvc ), state switching (Ω), constraint preservation (Lrule ), and continuation length (T ). These features characterize complementary aspects of trajectory organization, including support formation (τsup , S̄), commitment coordination (τcom , gcs ), verification behavior (τver , gvc , Rvc ), transition stability (Ω), constraint preservation (Lrule ), and continuation dynamics (T ). Based on these trajectory coordinates, a well-organized reasoning process is expected to satisfy three general organization principles. First, support-related states should provide sufficient evidence before answer commitment, indicating that candidate selection is grounded in accumulated support rather than premature commitment. Second, functional-state transitions should remain coordinated, with limited unnecessary oscillation or repeated revisits among unrelated states. Third, post-commitment verification should preserve the ability to confirm or revise the current candidate, enabling later computation to correct earlier deviations. These principles define the expected organization of reasoning trajectories and provide a reference for identifying those abnormal transition patterns. In Section 4.5, we empirically investigate how failed traces depart from these organization principles and characterize recurring trajectory patterns in the proposed feature space.

• RQ1 – Categories of LLM Failures: How do failed and successful traces differ in propagation of representation and in the coordination of support, answer commitment, and self-verification? • RQ2 – Specificity of Reasoning Trajectory: Can the samples with the same output label be distinguished by their reasoning trajectory topology? • RQ3 – Verification of Adversarial Mechanism: When a harmful prompt-perturbation causes a behavioral flip, are associated the internal changes larger than that in nonflipped pairs and broadly consistent with the targeted or downstream functional states in theory? • RQ4 – Generalization of Methodology: Are the diagnostic analytics stable to aggregation choices, and transferable across different model families?

4.2

Threat Model

For RQ3, we consider a prompt-channel adversary who controls the user prompt and task context. The attacker may introduce misleading premises, retrieval decoys, reordered reasoning cues, incorrect intermediate computations, rulepriority overrides, forced answer commitments, or instructions that suppress verification. An attack succeeds when a cleansuccessful reasoning instance becomes incorrect, unsupported, or task-inconsistent, or when a safe refusal becomes unsafe 7

Table 2: Representation-level indicators. Lower Āsup denotes lower Attention and MLP updates to support states.

or partially unsafe compliance. The task objective and evaluation target remain unchanged, enabling pairwise comparison between each adversarial trace and its clean counterpart. The adversary cannot modify model weights, decoding code, sampling parameters, trace collection, the evaluator, or CauTrace pipeline, and has no access to hidden states or diagnostic outputs during prompt construction. We exclude training-time poisoning, weight editing, tool or retrievalsystem compromise, and deployment-side exfiltration. Whitebox access is reserved for the post-generation analyst, who inspects hidden states, module outputs, layers, heads, and Q/K/V/O after generation; it is not part of the attack model.

4.3 Representation-Level: Support-Change Persists Across Updates

Model

Label

∆attn sup

∆sup

ml p

Āsup

Qwen2.5-14B Qwen2.5-14B Qwen2.5-7B Qwen2.5-7B Qwen2.5-3B Qwen2.5-3B Qwen2.5-1.5B Qwen2.5-1.5B

Correct Full Hall. Correct Full Hall. Correct Full Hall. Correct Full Hall.

0.076 0.125 -0.009 0.019 0.023 0.041 0.010 0.025

-0.015 -0.160 0.009 -0.056 0.029 -0.050 0.039 -0.060

-0.560 -0.600 -0.568 -0.625 -0.500 -0.950 -0.700 -0.500

Together with the cross-scale exception in Table 2, this supports Claim I as a propagated diagnostic association, not a uniquely localized circuit.

To test whether local support-state changes survive later Transformer updates (Que. I and Claim I), we compare Correct and Full Hallucination traces from the nine-dataset clean reliability benchmark across Qwen2.5-1.5B/3B/7B/14B. Table 2 measures whether Attention and MLP move the support coordinates in agreeing directions, while Fig. 2 evaluates the value-projected Attention representation through spectral concentration, step-to-step spectral change, and alignment with the AC probe direction. These measurements are computed from the same held-out analysis-track traces and combine probe-dependent indicators with partly probe-independent value-space statistics. Support-Related Representation Changes Remain Visible Across Later Transformer Updates, Despite Exact Carrier Is Not Uniform Across Scales. As shown in Table 2, the Attention–MLP update agreement decreases from Correct to Full Hallucination for Qwen2.5-14B (−0.560 → −0.600), 7B (−0.568 → −0.625), and 3B (−0.500 → −0.950); the 3B gap is the largest by −0.450. The 1.5B result moves in the opposite direction (−0.700 → −0.500), possibly because the substantially smaller model scale leads to different internal representation dynamics and update coordination patterns.This observation suggests that Āsup should not be interpreted as a scale-invariant failure classifier, but rather as a mechanistic indicator of support-related propagation changes. The value-space panels show why the conclusion is nevertheless about propagation rather than one statistic. Throughout Fig. 2(a), the failed traces retain a much lower but nonzero leading value-spectrum concentration than correct traces; Fig. 2(b) shows the repeated failure-associated spectralchange spikes, most visibly near the middle and final 20% of generation; and Fig. 2(c) shows failed traces becoming sharply aligned with the AC direction only near the endpoint, whereas correct traces remain commitment-aligned across a broader portion of the trace. The three panels therefore clearly connect a changed support representation to later commitment organization: the deviation is not erased after its local update, but remains available and is reorganized near commitment.

4.4 Temporal-State Level: Coordination of Failure-Reorganizing To verify Claim II, we compare Correct and Full Hallucination traces from the same analysis track at two resolutions. Fig. 3 reports the aggregate reasoning process-deviation score and its components across Qwen2.5 scales; Table 3 tests the relative placement and activity of support, AC, and SV; and Fig. 4 shows which dominant-state handoffs gain or lose probability. The combination calibrate the coordination rather than treating any one single-state score as a failure label. We compute q-values from two-sided unpaired permutation tests with Benjamini–Hochberg correction across all 15 comparisons. Failures Consistently Show Insufficient Verification After Commitment, While Their Timing Relationship Varies Across Models. Across models, failed traces exhibit a consistently lower verification-to-commitment ratio Rvc , indicating that verification activity becomes insufficient relative to commitment after the answer direction is formed. Pooled across models, Rvc decreases from 4.172 to 1.065 (∆ = −3.107, q < 0.001) as shown in Table 3. This reduction is significant across all Qwen2.5 scales, with failure–correct differences of −7.783, −5.292, −1.192, and −1.752 for 1.5B, 3B, 7B, and 14B, respectively. However, the exact temporal relationship between verification and commitment is not identical across models: after FDR correction, the verification–commitment gap is not significant for 1.5B and 7B. Therefore, Claim II is supported by a consistent reduction in verification coverage after commitment, rather than by assuming a universal timing reversal across all models. Fig. 3(a) provides the aggregate view: Full Hallucination has a higher aggregate reasoning process-deviation score Rproc bar than Correct at every model size, with the largest visible separation at 3B, while the overlaid verification-onset indicator varies substantially across scales. The decomposition 8

0.10 0.10 0.10

0.06 0.06

0.6 0.6 0.6

0.04 0.04

0.06 0.06 0.06

0.03 0.03

0.04 0.04 0.04

0.4 0.4 0.4

0.02 0.02

0.02 0.02 0.02 0.00 0.00 0.00 0.0 0.0 0.0

correct correct correct full hallucination full fullhallucination hallucination

0.8 0.8 0.8

0.05 0.05

Mean value

Mean value

0.08 0.08 0.08

0.2 0.2 0.2

0.01 0.01 0.2 0.2 0.2

0.4 0.4 0.4

0.6 0.6

0.8

1.0 1.0

Normalized Normalized Normalizedgeneration generation step

(a) Value-projected spectral concentration ρVt

0.00 0.00 0.0 0.0

0.2 0.2 0.2

0.40.4 0.4

0.60.6 0.6

0.8 0.8 0.8 1.0 1.0 1.0

Normalized Normalized Normalized generation generation generation step step step

(b) Value-spectrum change DVKL (t)

0.0 0.0 0.0 0.0 0.0 0.0 0.2 0.2 0.20.4

0.4 0.40.6

0.6 0.6 0.8

0.8 0.8 1.0

1.0 1.0

Normalized Normalized Normalized generation generation generation step step step

(c) Commitment alignment Γac,V (t)

Figure 2: Representation-level diagnostics. Panel (a) reports value-projected Attention spectral concentration ρVt ; Panel (b) reports step-to-step change in the normalized value spectrum, DVKL (t); Panel (c) reports alignment Γac,V (t) between the dominant value-space direction and the AC probe direction. in Fig. 3(b) further shows that the aggregate change is not a uniform rise in all five components: early-commitment and late-verification contributions increase, whereas other components change only slightly or in the opposite direction. This is precisely the coordination account in Claim II—different component combinations can produce failure, but inadequate checking relative to commitment remains the common relation confirmed by Table 3. The transition matrices identify the links underlying that aggregate shift. Relative to correct traces, Fig. 4(c) shows that failed traces lose probability on support-preserving handoffs, such as KR→FRT (−0.11) and PB→RC (−0.08), while gaining probability on KR→AC (+0.08), SV→AC (+0.06), and RC→KR (+0.06). These selective shifts in Fig. 4(c) mean that failure does not merely reduce all state activity: support construction is rerouted toward commitment or reset, and verification is more likely to return to the committed candidate. This discrete link evidence explains how the lower Rvc in Table 3 appears as a reorganization of the state chain.

tifies this asymmetry: Early-Commitment and Verification without Overwrite have diagonal proportions of 0.99 and 0.84, whereas Fragmented Support and Rule Failure have only 0.09 and 0.25 on the diagonal and are often mapped to Verification without Overwrite. The result validates Claim III without overstating separability: the endpoint category contains multiple routes, but the routes are not rigidly-delineated, but overlapping-existing regions. The time-domain panels explain what the projection preserves. Normal Convergence in Fig. 6(a) maintains strong support through the first half of generation and then transfers activity to verification before the trace ends. Early Commitment in Fig. 6(b) instead begins with substantial commitment activity and returns to a strong commitment near the endpoint; Fragmented Support in Fig. 6(c) retains high but uneven support while verification stays weak; and Rule Failure in Fig. 6(d) shows support decaying while commitment grows without a comparably strong verification response. The last two panels separate two superficially similar long failures: Verification without Overwrite develops substantial late verification while commitment persists in Fig. 6(e), whereas Verbose Unsupported Continuation loses support after midgeneration and ends in a renewed commitment spike with little remaining verification in Fig. 6(f). These panel-specific differences interpret the temporal meaning of reasoning trajectory, i.e. the same Full Hallucination endpoint can result from premature selection, disrupted support, weak rule preservation, ineffective checking, or unsupported continuation. Fig. 7 translates those curve-shapes into state-order motifs. The Early-Commitment examples enter AC before later RC/KR re-entry; The Fragmented Support examples repeat KR/RC resets; and the Verification without Overwrite examples explicitly return from SV to AC. The Verbose Unsupported Continuation examples are the longest and most recurrent in the figure (T = 10, R = 7 and T = 11, R = 8), consisting of repeated commitment/support re-entry without proportional verification. The above analytics validate Claim III by showing that trajectory types encode families of routes

4.5 Trajectory-Topology Level: One Output Label Contains Multiple Routes To validate Claim III, we summarize the same held-out Full Hallucination traces with the trajectory features in Eq. (20) and compare the five failure-prone regions defined in Table 16 in Appendix B.4. Fig. 5 tests whether the regions occupy distinguishable neighborhoods in the full feature space, Fig. 6 exposes their time-domain organization, and Fig. 7 makes the corresponding state-order motifs explicit. Full Hallucination Exhibits Topologically Heterogeneity: Some Routes Are Sharply Distinguishable, While Others Overlap because They Share Only Part of Support–Commitment–Verification Organization. In Fig. 5(a), the first two projection axes explain only 15.5% + 12.7% = 28.2% of trajectory-feature variance, yet Early-Commitment and Verification without Overwrite occupy visibly different neighborhoods. The row-normalized matrix in Fig. 5(b) quan9

Table 3: Timing comparisons between Correct and Full Hallucination traces. Each cell reports Correct → Full Hallucination with ∆ = Full Hallucination−Correct. For gcs and gvc , the measured quantity is a shift in relative temporal position; negative gcs denotes the early-commitment risk direction. Lower Rvc indicates less verification activity relative to answer commitment. Model

gcs : support–commitment gap gvc : verification–commitment gap Rvc : verification/commitment negative = early commitment; shift large shift / sign flip = disrupted ratio = change checking lower ratio = less relative verification activity

All models

14.025 → 0.489 ∆ = -13.536, q < 0.001 14.182 → −1.432 ∆ = -15.614, q = 0.034 24.283 → 3.125 ∆ = -21.159, q < 0.001 3.750 → −0.857 ∆ = -4.607, q = 0.042 9.319 → 0.333 ∆ = -8.986, q < 0.001

Qwen2.5-1.5B Qwen2.5-3B Qwen2.5-7B Qwen2.5-14B

−0.274 → 16.311 ∆ = 16.585, q < 0.001 not significant after FDR correction 4.951 → 20.616 ∆ = 15.664, q = 0.011 not significant after FDR correction −3.940 → 21.437 ∆ = 25.376, q < 0.001

4.172 → 1.065 ∆ = -3.107, q < 0.001 9.064 → 1.281 ∆ = -7.783, q = 0.011 6.473 → 1.181 ∆ = -5.292, q < 0.001 1.784 → 0.592 ∆ = -1.192, q = 0.013 2.474 → 0.722 ∆ = -1.752, q = 0.018

Behavioral Flips Are Accompanied by Broad Changes in Same Functional-State Coordinates. In the reasoninginterference rows of Table 4, PFR ranges from 1.8% to 6.2% and conditional PFR from 10.0% to 26.1%; in the safety rows, PFR is higher at 12.7%–22.3% and conditional PFR at 14.7%–30.5%. These paired flips establish that the prompt intervention can change the endpoint under a matched objective. However, TAR@1 is only 15.2%–16.6% versus the 1/7 = 14.3% chance level, and TAR@3 is 41.8%–49.8% versus 3/7 = 42.9%, as shown in Table 4, and Appendices A.4 and A.5 provide the paired construction, target-state taxonomy, and metric definitions. We leverage the working mechanism to test related reasoning-process changes in theory, while treating exact target-state identification as unsupported. The flip-associated columns in Table 5 provide the processlevel support for the claim. Reasoning-interference flips show their strongest contrasts in OC for Qwen2.5-1.5B (d = −3.180) and 7B (d = −3.134), and in FRT onset for 3B

rather than merely one token-level chain per output label.

4.6 Adversarial Test: Behavioral Flips Coincide with Larger Internal Changes To test whether the adversarial prompt-induced failures disturb the same reasoning-process organization as clean failures (RQ3), we compare each adversarial trace with a matched clean trace under a fixed task objective. Table 4 reports 500 reasoning-interference pairs and 300 AdvBench/HarmBench safety pairs per model; Table 5 then compares state-coordinate, cascade, onset, and Q/K/V/O changes for flipped and non-flipped pairs, here a flip means that a clean correct/evidence-grounded answer becomes erroneous or unsupported, or that a clean safe refusal becomes unsafe or partially unsafe compliance. The two tables therefore separate endpoint vulnerability from the internal route associated with a successful adversarial perturbation.

(a) Composite process-deviation score across model sizes

(b) Process-deviation score components

Figure 3: Temporal-state coordination diagnostics: Panel (a) compares Rproc and the verification-onset indicator Iver ; Panel (b) decomposes Rproc into early commitment onset, support delay, oscillation, late verification, and low verification coverage. 10

(a) Correct trajectories

(b) Failed trajectories

(c) Failure-minus-correct shift

Figure 4: State-transition matrices for correct and failed trajectories: Panels (a) and (b) report transition probabilities for correct and failed generations, respectively; Panel (c) reports the failure-minus-correct difference.

(a) Trajectory-feature projection

(b) Type-prediction confusion matrix

Figure 5: Trajectory topology in the trajectory-feature space: Panel (a) visualizes trajectory-level features with approximate type neighborhoods; Panel (b) gives the row-normalized confusion matrix of a lightweight rule-derived trajectory predictor constructed directly from the diagnostic definitions in Eq. (20). Table 4: Paired adversarial test results. BSR denotes clean-side success, AFR is adversarial failure rate, PFR is pair flip rate, and conditional PFR measures flips among clean-successful pairs. Top-k Target Accuracy Rate (TAR) is reported for process reasoning perturbations, where each perturbation defines an explicit seven-state target. Dataset

Model

Pairs

BSR

AFR

PFR

Cond.PFR

TAR@1

TAR@3

Reasoning Interfered

Qwen2.5-1.5B Qwen2.5-3B Qwen2.5-7B

500 500 500

23.8% 18.0% 26.6%

34.0% 14.0% 21.0%

6.2% 1.8% 4.8%

26.1% 10.0% 18.0%

16.6% 15.2% 16.6%

49.8% 41.8% 48.4%

AdvBench/HarmBench

Qwen2.5-1.5B Qwen2.5-3B Qwen2.5-7B

300 300 300

73.3% 69.3% 86.0%

37.0% 33.3% 18.0%

22.3% 14.7% 12.7%

30.5% 21.2% 14.7%

– – –

– – –

11

1.0

0.8

0.8

0.8

0.6 0.4 0.2

State intensity

1.0

State intensity

State intensity

1.0

0.6 0.4 0.2 0.0

0.0 0.0

0.5

0.5

0.2

1.0

0.0

(a) Normal Convergence

(b) Early-Commitment

(c) Fragmented Support

0.8

0.8

0.8

0.4 0.2

State intensity

1.0

State intensity

1.0

0.6

0.6 0.4 0.2

0.0

0.0 0.5

1.0

0.6 0.4 0.2 0.0

0.0

Normalized generation step

0.5

1.0

0.0

0.5

Normalized generation step

(d) Rule Failure

1.0

Normalized generation step

1.0

0.0

0.5

Normalized generation step

Normalized generation step

State intensity

0.4

0.0 0.0

1.0

0.6

1.0

Normalized generation step

(e) Verification without Overwrite

(f) Verbose Unsupported Continuation

Figure 6: Time-domain state-score trajectories for one normal and five failure-prone trajectory types. The curves show how each trajectory pattern differs in support formation, answer commitment, and self-verification over the generation process. forward transition

revisit / backward jump

1 Early commitment

Support states repeatedly break and re-enter

Severe prototype | Qwen2.5-1.5B | commonsenseqa_00109

Severe prototype | Qwen2.5-1.5B | commonsenseqa_00126

PB KR RC SNO FRT AC SV

PB KR RC

RC

AC

AC

AC

AC

Maximum disorder | Qwen2.5-1.5B | commonsenseqa_00131

PB KR RC SNO FRT AC SV

RC

PB KR RC SNO FRT AC SV

RC

AC

AC SV

0

0.5 Normalized transition order

1.0

KR RC

AC

AC

AC

PB

PB

RC

PB

PB

PB

RC FRT

FRT

FRT

AC

AC SV

SV

SV

SV

SV

0.5 Normalized transition order

1.0

5 Verbose unsupported continuation

Rule tracking is absent or repeatedly displaced

Long continuation cycles without stable support

Severe prototype | Qwen2.5-1.5B | gsm8k_00012 RC

Severe prototype | Qwen2.5-1.5B | folio_00029

PB KR RC SNO FRT AC SV

RC FRT

FRT

AC

FRT

AC

AC

AC

Maximum disorder | Qwen2.5-7B | folio_00029

PB KR RC SNO FRT AC SV

KR RC

0

3 Rule failure PB KR RC SNO FRT AC SV

PB KR

Maximum disorder | Qwen2.5-3B | gsm8k_00034

RC

AC

end

2 Fragmented support

Commitment dominates before support stabilizes

PB KR RC SNO FRT AC SV

start

RC

RC

RC

FRT

FRT

FRT

FRT

AC SV

SV

Maximum disorder | Qwen2.5-3B | folio_00029

PB

PB

PB

PB KR RC SNO FRT AC SV

PB KR

RC FRT

FRT

FRT

FRT

FRT

FRT

AC

AC

0

0.5 Normalized transition order

1.0

RC FRT

FRT

FRT

AC

AC SV

SV

0

SV

SV

0.5 Normalized transition order

1.0

4 Verification without overwrite Verification returns to the prior commitment

Severe prototype | Qwen2.5-3B | gsm8k_00071

PB KR RC SNO FRT AC SV

PB

PB

FRT

PB

FRT

PB

FRT

FRT

PB

FRT

PB

FRT

FRT

AC

AC

AC SV

AC SV

Maximum disorder | Qwen2.5-14B | strategyqa_00022

PB KR RC SNO FRT AC SV

KR RC

AC

KR

RC

AC

AC

AC SV

0

0.5 Normalized transition order

Figure 7: Representative state-transition chains for the five failure-prone trajectory types.

12

1.0

Table 5: Internal changes under paired adversarial perturbations. Dataset

Model

Dominant State Shift

Strongest Flip-Associated Signal

Reasoning Interfered

Qwen2.5-1.5B

Premise Binding −0.133

OC cascade (d = −3.180)

Qwen2.5-3B

Reasoning +0.115 Reasoning −0.058

Qwen2.5-7B AdvBench /HarmBench

Qwen2.5-1.5B Qwen2.5-3B Qwen2.5-7B

Answer +0.057 Answer −0.001 Answer +0.002

composition Composition commitment commitment commitment

(d = 2.980), all in Table 5. These are support/constraint coordinates from Claim II, thus their disruption explains why a harmful-benign-paired prompt change can reroute the chain before the endpoint flips. Furthermore, safety flips follow a different route: the strongest state/cascade indicators are organized around AC, including peak-layer shifts of d = 0.642 for 3B and d = 0.636 for 7B. This contrast supports a shared coordinate system, not a shared single route: attacks primarily disturb operation/rule support, whereas successful attacks are more closely associated with commitment organization. The Q/K/V/O column in Table 5 links those process-level changes back to Claim I. The largest paired representation effects are the O consecutive-cosine shift for 1.5B safety pairs (dz = −1.289) and the Q stable-rank shift for 3B reasoning pairs (dz = −0.975), as reported in Table 5. Thus, the successful adversarial perturbations are visible in both reasoning-process coordinates and fine-grained Transformer representations, while the changing Q/K/V/O carrier again argues against one universal circuit. Held-out checks in Tables 17 and 18 show the largest observed 1.5B reasoning reduction with unified repair (76.0% to 71.0%) and predicted 7B aligned-steering shifts in risk, rule, and verification. The adversarial evidence supports adversarial perturbation-linked, intervention-sensitive changes in the three-level analytics.

4.7

Strongest Paired QKV Signal

K pairwise cosine +0.003 (dz = 0.352) Formal Rule Tracking on- Q stable rank −0.038 set delay (d = 2.980) (dz = −0.975) OC state (d = −3.134) V–O norm correlation +0.063 (dz = 0.315) Commitment cascade strength (d = 0.232) Commitment peak-layer shift (d = 0.642) Commitment peak-layer shift (d = 0.636)

O consecutive cosine −0.078 (dz = −1.289) Q pairwise-cosine dispersion +0.006 (dz = 0.485) O row-norm dispersion −0.031 (dz = −0.661)

Representation-Level Carrier. All four models show lower support-state update agreement for failed traces, with ∆Āsup ranging from −0.012 to −0.052 as shown in Table 13. They also show lower "verification-to-commitment" activity, with ∆Rvc ranging from −1.102 to −15.519. These shared directions support the central relations as stated in Claims I-II. By contrast, ∆ρV and ∆Γac,V change sign across models, and ∆DVKL is positive for the three dense LLaMA models but approximately zero for Qwen1.5-MoE. The cross-family results therefore support representation-propagating differences and weaker relative verification at the diagnostic level while indicating that the measurable carrier depends on architecture.

5

Conclusion

We have quantitatively verified the proposed three diagnostic claims through three-level hierarchical analytics. Our work fills a reasoning process-level gap for LLM reliability and safety inspections: final-output labels identify failures but not how they emerge during the entire process of generation. By organizing the hidden-state traces into seven functionalstate coordinates, our proposed CauTrace connects the representation changes across Transformer blocks, temporal-state coordination among support/commitment/verification, and trajectory topology over the full reasoning trajectory chain. Across Qwen and LLaMA-family models, our experiments show lower support-state update agreement and lower verification activity relative to commitment in failed traces, different trajectory topologies among samples with the same output label, and larger internal changes under successful adversarial perturbations. The evidence supports a unified diagnostic account of failure dynamics and provides a foundation for more fine-grained causal localization. We believe that our work can provide a foundation for future research on LLM safety and inspire further efforts toward safer and more reliable LLMs.

Generalization

To assess the generalizability of our proposed analytics framework, beyond Qwen2.5, we also apply the Correct/Full Hallucination analysis to LLaMA-3.2-1B/3B, LLaMA-3.1-8B, and Qwen1.5-MoE and Table 13 reports failure-minus-correct differences in the representation-level indicators and in the verification activity relative to commitment. The Coordination Result Transfers Across Model Families More Consistently Than Any Individual 13

Ethics Considerations

the resulting diagnostic knowledge retains some dual-use potential, the controlled evaluation setting, bounded threat model, and restricted release policy substantially reduce operational misuse. We therefore judge the expected reliability and safety benefits to outweigh the residual risks under these safeguards.

In developing CauTrace, we considered the ethical implications of combining white-box diagnostics with adversarial evaluation of large language models. The work proposes an offline analytical framework rather than an attack system: adversarial prompts are introduced only as controlled stress tests of whether ordinary and induced failures exhibit related internal process changes. All experiments use locally hosted open-weight models and public benchmarks; they involve no human participants, private user data, unauthorized access, or attacks on deployed services. Stakeholders and Potential Impact. The principal stakeholders are LLM users, individuals and communities affected by unsupported or harmful outputs, model developers, AI reliability and safety researchers, and benchmark maintainers. CauTrace may benefit these groups by clarifying how evidence support, answer commitment, rule tracking, and selfverification become miscoordinated before an unreliable or unsafe endpoint. Such diagnostics can inform future monitoring and mitigation, but they should not be interpreted as a per-generation safety guarantee or as a substitute for deployment-specific evaluation. Dual-Use Risks and Responsible Disclosure. The same internal patterns that help diagnose failure could be misused to design prompts that encourage premature commitment, weaken rule tracking, or suppress verification. We limit this risk in both experimental scope and artifact release. The threat model is restricted to prompt-side perturbations against locally hosted models, and the adversary has no access to model weights, hidden states, diagnostic outputs, evaluators, or deployed systems during prompt construction. We will not release turnkey jailbreak pipelines, operational instructions for bypassing safeguards, unredacted harmful-prompt collections, or harmful model completions. If future use of CauTrace reveals a deployment-specific vulnerability, we will coordinate disclosure with the relevant model or system provider before releasing operational details. Handling of Harmful Content. The paired safety evaluation may expose models to harmful prompts or produce unsafe responses. We reduce researcher exposure through automated safety judges, predefined output labels, and structured filtering, with manual inspection limited to debugging and validation. Harmful instructions are never executed, and potentially unsafe prompts or generations are neither reproduced in the paper nor redistributed in the artifact. Reported findings use aggregate statistics and sanitized identifiers wherever possible. Balancing Potential Benefits and Harms. CauTrace is intended to improve failure analysis by linking endpoint behavior to Propagation of Representation, Temporal-State Coordination, and Reasoning Trajectory. This evidence can help researchers distinguish failures that share an output label but require different monitoring or mitigation strategies. Although

Open Science Reproducibility is a central goal of this work. We will provide an anonymous artifact containing the materials required to reproduce the reported diagnostic analyses while applying the safety restrictions described above. Data and Model Use. The experiments use publicly available reasoning and safety benchmarks, including the cleanreasoning datasets documented in Appendix A.2 and the AdvBench/HarmBench evaluation sets, together with publicly released Qwen and LLaMA-family model checkpoints. These resources are used only for controlled evaluation and will remain subject to the licenses and usage conditions of their original providers. We collect no new human-subject data or personally identifiable information. Where redistribution of raw or harmful content is inappropriate, the artifact will provide source references, dataset manifests, version records, and sanitized instance identifiers rather than repackaging that content. Code and Artifacts. The artifact will include the CauTrace implementation and environment specification; model and dataset version records; scripts for trace collection, functional-state/process-coordinate scoring, probe training, weak-label construction, evaluation, statistical testing, and figure generation; fixed data splits and configuration files; and sanitized metadata and aggregate outputs for the matched clean/adversarial analysis. It will exclude unsafe model completions, unredacted harmful prompts, turnkey jailbreak pipelines, and operational instructions for circumventing safeguards. The review artifact will be available at https://anonymous.4open.science/r/artifact-B43D; following review, the released version will be preserved in a persistent archival repository.

References [1] Cem Anil, Esin Durmus, Nina Panickssery, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Meg Tong, Jesse Mu, Daniel Ford, et al. Many-shot jailbreaking. In Advances in Neural Information Processing Systems, volume 37, 2024. [2] Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction. In Advances in Neural Information Processing Systems, volume 37, 2024. 14

[3] Nora Belrose, Igor Ostrovsky, Lev McKinney, Zach Furman, Logan Smith, Danny Halawi, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112, 2023.

[12] Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. In International Conference on Learning Representations, 2024.

[4] Matan Ben-Tov, Mor Geva, and Mahmood Sharif. Universal jailbreak suffixes are strong attention hijackers. Transactions of the Association for Computational Linguistics, 14:1028–1050, 2026. doi:10.1162/TACL.a. 695.

[13] Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2):1–55, 2025.

[5] Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. In International Conference on Learning Representations, 2023.

[14] Alon Jacovi and Yoav Goldberg. Towards faithfully interpretable nlp systems: How should we define and evaluate faithfulness? In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 4198–4205, 2020.

[6] Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramèr, et al. Jailbreakbench: An open robustness benchmark for jailbreaking large language models. In Advances in Neural Information Processing Systems, volume 37, 2024.

[15] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM computing surveys, 55(12):1– 38, 2023.

[7] Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. In Advances in Neural Information Processing Systems, volume 36, 2023.

[16] Yuqi Jia, Yupei Liu, Zedian Shao, Jinyuan Jia, and Neil Zhenqiang Gong. Promptlocate: Localizing prompt injection attacks. In 2026 IEEE Symposium on Security and Privacy (SP). IEEE, 2026. doi:10.1109/SP63933. 2026.00105.

[8] Janet E Davidson and Robert J Sternberg. The psychology of problem solving. Cambridge university press, 2003.

[17] Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. Followbench: A multi-level fine-grained constraints following benchmark for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4667–4688, 2024.

[9] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484–5495, 2021.

[18] Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022.

[10] Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. Patchscopes: A unifying framework for inspecting hidden representations of language models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 15466– 15490. PMLR, 2024.

[19] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213, 2022.

[11] Abhay Gupta, Kevin Zhu, Vasu Sharma, Sean O’Brien, and Michael Lu. NovelHopQA: Diagnosing multi-hop reasoning failures in long narrative contexts. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 26134– 26151. Association for Computational Linguistics, 2025. doi:10.18653/v1/2025.emnlp-main.1328.

[20] Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, et al. Measuring faithfulness in chain-ofthought reasoning. arXiv preprint arXiv:2307.13702, 2023. 15

[21] Chak Tou Leong, Qingyu Yin, Jian Wang, and Wenjie Li. Why safeguarded ships run aground? aligned large language models’ safety mechanisms tend to be anchored in the template region. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15212–15229. Association for Computational Linguistics, 2025. doi:10.18653/v1/2025.acl-long.738.

[29] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359–17372, 2022. [30] Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12076–12100. Association for Computational Linguistics, 2023. doi:10.18653/v1/2023.emnlp-main. 741.

[22] Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. HaluEval: A large-scale hallucination evaluation benchmark for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6449– 6464. Association for Computational Linguistics, 2023. doi:10.18653/v1/2023.emnlp-main.397.

Transformer[31] Neel Nanda and Joseph Bloom. lens. https://github.com/TransformerLensOrg/ TransformerLens, 2022.

[23] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages 3214–3252, 2022.

[32] Allen Newell, Herbert Alexander Simon, et al. Human problem solving, volume 104. Prentice-hall Englewood Cliffs, NJ, 1972. [33] Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114, 2021.

[24] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173, 2024. doi: 10.1162/tacl_a_00638.

[34] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. Incontext learning and induction heads. arXiv preprint arXiv:2209.11895, 2022.

[25] Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. In International Conference on Learning Representations, 2024. [26] Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. Formalizing and benchmarking prompt injection attacks and defenses. In 33rd USENIX Security Symposium (USENIX Security 24), pages 1831– 1847. USENIX Association, 2024.

[35] Wenbo Pan, Zhichao Liu, Qiguang Chen, Xiangyang Zhou, Yu Haining, and Xiaohua Jia. The hidden dimensions of llm alignment: A multi-dimensional analysis of orthogonal safety directions. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 47697–47716, 2025.

[27] Potsawee Manakul, Adian Liusie, and Mark Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 conference on empirical methods in natural language processing, pages 9004–9017, 2023.

[36] Debjit Paul, Robert West, Antoine Bosselut, and Boi Faltings. Making reasoning matter: Measuring and improving faithfulness of chain-of-thought reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 15012–15032, 2024.

[28] Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. In Proceedings of the International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 35181–35224, 2024.

[37] Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. Do anything now: Characterizing and evaluating in-the-wild jailbreak prompts on large language models. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security. ACM, 2024. 16

[38] Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pages 31210–31227. PMLR, 2023.

gen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. [47] Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuan-Jing Huang. Do large language models know what they don’t know? In Findings of the association for Computational Linguistics: ACL 2023, pages 8653–8665, 2023.

[39] Jiawen Shi, Zenghui Yuan, Yinuo Liu, Yue Huang, Pan Zhou, Lichao Sun, and Neil Zhenqiang Gong. Optimization-based prompt injection attack to llm-asa-judge. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pages 660–674. ACM, 2024. doi:10.1145/3658644. 3690291.

[48] Zhiyuan Yu, Xiaogeng Liu, Shunning Liang, Zach Cameron, Chaowei Xiao, and Ning Zhang. Don’t listen to me: Understanding and exploring jailbreak prompts of large language models. In 33rd USENIX Security Symposium (USENIX Security 24), pages 4675–4692. USENIX Association, 2024.

[40] Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, et al. A strongreject for empty jailbreaks. In Advances in Neural Information Processing Systems, volume 37, 2024.

[49] Shenyi Zhang, Yuchen Zhai, Keyan Guo, Hongxin Hu, Shengnan Guo, Zheng Fang, Lingchen Zhao, Chao Shen, Cong Wang, and Qian Wang. {JBShield}: Defending large language models from jailbreak attacks through activated concept analysis and manipulation. In 34th USENIX Security Symposium (USENIX Security 25), pages 8215–8234, 2025.

[41] Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. Advances in Neural Information Processing Systems, 36:74952–74965, 2023.

[50] Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren’s song in the ai ocean: A survey on hallucination in large language models. Computational Linguistics, 51(4):1373–1418, 2025.

[42] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations, 2023.

[51] Zhenhong Zhou, Haiyang Yu, Xinghua Zhang, Rongwu Xu, Fei Huang, and Yongbin Li. How alignment and jailbreak work: Explain llm safety through intermediate hidden states. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 2461–2488. Association for Computational Linguistics, 2024.

[43] Yuxia Wang, Minghan Wang, Muhammad Arslan Manzoor, Fei Liu, Georgi Nenkov Georgiev, Rocktim Jyoti Das, and Preslav Nakov. Factuality of large language models: A survey. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 19519–19529, 2024.

[52] Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405, 2023.

[44] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022.

A

Experimental Protocol and Details

This appendix provides the experimental details and supplemental results needed to interpret the main evaluation, including benchmark composition, output evaluation, probe training, paired-adversary construction, and Trajectory Topology evidence.

[45] Tom Wollschläger, Jannes Elstner, Simon Geisler, Vincent Cohen-Addad, Stephan Günnemann, and Johannes Gasteiger. The geometry of refusal in large language models: Concept cones and representational independence. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 66945–66970, 2025.

A.1

Evaluation Configuration

Each sample is evaluated under concise and analysis prompt tracks. Unless otherwise stated, process-level results use the analysis track, which requests two to four short reasoning

[46] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chen17

steps followed by a constrained final-answer line. The evaluator assigns one of five task-specific labels: Correct, Partial Hallucination, Full Hallucination, Abstain, or Invalid. Safety pairs are evaluated analogously as safe refusal versus unsafe or partially unsafe compliance. The matched clean/adversarial evaluation reports Benign Success Rate (BSR), Adversarial Failure Rate (AFR), Pair Flip Rate (PFR), and Target Alignment Rate (TAR). Subsequent subsections specify the benchmark, output evaluation, probes, prompt construction, and paired metrics.

weak supervision, not a ground-truth cognitive state. Table 8 shows that validation AUC approaches saturation from w32 to w64 across the evaluated Qwen2.5 models. We therefore use w32 in the main analysis because it remains in the topperforming range while preserving greater temporal resolution for activation-onset and state-transition measurements. These AUC values measure separability with respect to the weak supervision; they do not establish that a probe recovers a unique latent state. Table 9 lists the fixed training hyperparameters.

A.2

Trace collection and offline diagnosis. CauTrace computes all diagnostic measurements after the model has generated a response. During generation, we record the hidden states, module outputs, attention-stream projections, and token positions required for analysis, but do not modify logits, hidden states, attention weights, or decoding decisions. The original model output determines the endpoint label. Statecoordinate scores, transition variables, Attention/MLP update statistics, cascade statistics, layer/head sensitivity, and Q/K/V/O measurements are then computed offline from the recorded trace. CauTrace is therefore a diagnostic framework, not a generation-time controller. In paired experiments, the clean and adversarial rows use identical instrumentation and evaluation; only the prompt condition differs. Consequently, differences between the two traces are attributable to the prompt condition and the model’s response to it, rather than to changes introduced by the analysis pipeline.

Benchmark Composition and Evaluation

The clean reliability benchmark contains nine public datasets: TruthfulQA, HotpotQA, MuSiQue, GSM8K, MATH, FOLIO, ProofWriter, StrategyQA, and CommonsenseQA. Together, they cover factual QA, multi-hop reasoning, numerical reasoning, formal mathematics, logic/rule reasoning, implicit reasoning, and commonsense multiple-choice QA. Table 6 reports the composition of the benchmark. Automatic evaluation follows the task-specific judge_type. Table 7 specifies the required output form and parsing constraint for each task family. Constraint failures are therefore evaluated against task-specific requirements rather than a single open-ended criterion. Open-ended short answers are evaluated using normalized token-level F1: F1(ŷ, y) = |T (ŷ) ∩ T (y)| P= , |T (ŷ)|

2PR , P+R |T (ŷ) ∩ T (y)| R= . |T (y)|

(21)

A.4 Prompt Tracks and Matched Clean/Adversarial Construction

(22)

Each sample is evaluated under concise and analysis prompt tracks. Both use the same system instruction. The concise track requests only the final answer in the task-required format, whereas the analysis track requests two to four short reasoning steps followed by exactly one final-answer line. When uncertain, the model is instructed to output Insufficient information. The analysis track increases the observability of intermediate support and verification while retaining a parseable endpoint. Table 10 summarizes the differences between the tracks. The matched clean/adversarial benchmark extends the process-level analysis to a security-relevant setting. Each pair shares an identifier, task semantics, evaluation target, and safety objective. The clean row contains the ordinary task or refusal instruction; the adversarial row adds a CauTracetargeted prompt perturbation without changing the evaluation target. The design isolates the prompt condition at the input level. Differences in outputs and internal measurements are therefore interpreted as perturbation-linked changes, not as evidence of a uniquely localized cause. The paired evaluation combines reasoning-reliability pairs derived from the clean benchmark with public red-team-style

When F1 ≥ 0.60 without an exact match, the sample is labeled partial_hallucination. For mcq, yesno, and label tasks, the structured parser π j (·) is applied before scoring; a parsing failure produces the label invalid_or_unparseable. Abstention is detected using regular expressions for phrases such as insufficient information, cannot determine, and unknown. The evaluator first checks parseability and then distinguishes abstention, correctness, partial hallucination, and full hallucination. These endpoint labels are separate from the trajectory types defined by CauTrace.

A.3

State-Probe Training and Scale Selection

The seven functional-state/process coordinates are estimated by applying linear probes to causally pooled hidden states over window sizes {1, 2, 4, 8, 16, 32, 64}. Probe training uses weak process labels derived from task type, output label, and sequence-region heuristics; it does not use manually annotated token-level state labels. A higher score indicates stronger evidence for the corresponding operational role under the 18

Table 6: Clean benchmark composition used in the main reliability evaluation. Dataset

Size

Judge

Dataset

Size

Judge

Dataset

Size

Judge

TruthfulQA GSM8K ProofWriter

120 880 600

semantic numeric label

HotpotQA MATH StrategyQA

7,400 1,250 250

support math yes/no

MuSiQue FOLIO CommonsenseQA

242 150 1,200

support label mcq

Table 7: Output constraints by judge_type. judge_type

Output Form

Constraint

label_match yesno_match mcq_match numeric_match math_answer_match semantic support

Label Binary answer Option Number Mathematical answer Short answer QA answer

entailment/contradiction/unknown yes/no A/B/C/D/E Final number only Final expression or result only Concise, comparable answer Answer based on supporting facts

Table 8: Probe validation AUC across pooling-window scales. Model Qwen2.5-1.5B Qwen2.5-3B Qwen2.5-7B Qwen2.5-14B

w1

w2

w4

w8

w16

w32

w64

fused

0.9006 0.9038 0.9557 0.9794

0.9253 0.9311 0.9752 0.9916

0.9501 0.9570 0.9880 0.9974

0.9715 0.9793 0.9950 0.9985

0.9843 0.9902 0.9973 0.9992

0.9901 0.9939 0.9977 0.9990

0.9918 0.9952 0.9979 0.9989

0.9950 0.9972 0.9957 0.9979

Table 9: Multi-scale probe training hyperparameters. Parameter

Setting

Epochs Batch size Learning rate Weight decay Patience Train/Val Scale set

40 4096 1 × 10−2 1 × 10−4 6 0.9 / 0.1 w1, w2, w4, w8, w16, w32, w64, fused

A.5

Adversarial Perturbations

Each adversarial sample is annotated with the CauTrace state that its prompt perturbation is designed to disrupt: PB, KR, RC, OC, FRT, AC, or SV. The annotation is excluded from endpoint scoring and used only in the target-alignment analysis. Specifically, we test whether the largest clean-toadversarial changes occur in the intended state or in theoretically downstream states. Table 11 defines the perturbation taxonomy. The main experiments use Qwen2.5-Instruct models with 1.5B, 3B, 7B, and 14B parameters without fine-tuning [46]; the transfer analysis uses dense LLaMA-family models and Qwen1.5-MoE. Decoding is deterministic, with temperature 0.0, top_p= 1.0, and do_sample=false. Every clean and adversarial row passes through the same evaluator, trace collector, probe stack, transition analyzer, cascade analyzer, and Q/K/V/O analyzer. For any comparable scalar metric η, the pair-level change is

safety pairs based on AdvBench and HarmBench [28, 37, 48]. Reasoning perturbations may introduce false premises, misleading source cues, shortcut rationales, incorrect calculation drafts, or answer-forcing instructions. For harmful-goal pairs, the clean condition requires safe refusal, whereas the adversarial condition applies jailbreak-style semantic pressure to formal rule tracking, answer commitment, or self-verification. We do not reproduce operational attack prompts or harmful procedural content; the analysis is limited to paired outcomes and internal changes.

∆η = η(adversarial) − η(clean),

(23)

This notation avoids overloading m, which denotes the number of value vectors in Eq. (9). The paired analysis uses four primary behavior-level metrics. Benign Success Rate (BSR) is the fraction of clean rows 19

Table 10: Differences between concise and analysis tracks. Dimension

Concise

Analysis

Potential Failure Mode

Output length Observability Commitment risk Parsing stability

Short Low Medium High

2–4 steps High Potentially higher Medium

May induce Verbose Unsupported Continuation outputs Rationale text is not always faithful verification Early Commitment or Verification without Overwrite Invalid parsing if final-line format is violated

Table 11: CauTrace state-targeted adversarial perturbations. Target State

Process Role

Perturbation Focus

Premise Binding

Bind entities, roles, context, and constraints. Retrieve relevant facts or supporting evidence. Organize multi-step inference and intermediate claims. Execute numerical or symbolic operations. Maintain formal rules, safety policies, and task boundaries. Stabilize an answer after sufficient support. Check evidence, answer consistency, and risk boundaries.

False premises, role confusion, conflicting context bindings. Plausible but wrong facts, source bias, retrieval decoys. Shortcut reasoning, chain reordering, incompatible intermediate claims. Wrong scratch work, expression substitution, arithmetic interference. Rule-priority inversion, forged exceptions, constraint override. Premature conclusion, forced single answer, uncertainty compression. Reflection suppression, no-check instructions, refusal-review weakening.

Knowledge Retrieval Reasoning Composition Operational Computation Formal Rule Tracking Answer Commitment Self-Verification

that satisfy the correctness, evidence-support, or safe-refusal criterion. Adversarial Failure Rate (AFR) is the fraction of adversarial rows that are erroneous, unsupported, unsafe, or partially unsafe. Pair Flip Rate (PFR) is the fraction of pairs that move from benign success to adversarial failure. Target Alignment Rate at k (TAR@k) measures whether the intended CauTrace target state occurs among the k largest internal changes; the main text reports TAR@1 and TAR@3. We also compare flipped and non-flipped pairs to determine whether realized endpoint failures are associated with larger changes in state scores, modules, cascades, layers/heads, or Q/K/V/O representations.

This section provides the endpoint, Trajectory Topology, and intervention results referenced in the main evaluation.

son. Table 12 reports rank stability, failure–correct separation, and threshold-decision stability relative to the equal-weight scores. Representation-Change Propagation Is Stable as a Ranking, Whereas Reasoning-Process Deviation Is Intrinsically Multi-Component and Should Not Be Reduced to One Threshold. For Rstr , both ablation and random weighting retain rank stability of 1.000 in Table 12; however, ablation decision stability ranges from 0.250 to 1.000, and its separation range reaches 0.000. Thus, Claim I is robust at the ordering level but not as a universal binary rule. For Rproc , median rank stability falls to 0.790 under ablation and 0.696 under random weights, while median decision stability is 0.827 and 0.811, respectively. This sensitivity is consistent with Claims II–III: different Trajectory Topologies are dominated by different combinations of timing, switching, rule preservation, and verification, so the aggregate score must be interpreted together with its components.

B.1

B.2

B

Supplemental Results

Robustness to Diagnostic Aggregation

To determine whether Claims I–III depend on a particular aggregation scheme, we recompute the four-indicator aggregation Rstr and reasoning process-deviation score Rproc using leave-one-component-out ablations and Dirichlet-random weights on the clean Correct/Full Hallucination compari-

Cross-Family Diagnostic Results

Table 13 provides the complete cross-family measurements summarized in the main Generalization subsection. The shared directions of ∆Āsup and ∆Rvc support the representation-propagation and coordination relations, whereas the other representation-level carriers remain 20

Table 12: Aggregation sensitivity and component-ablation analysis for Rstr and Rproc . Rank stability denotes Spearman correlation with the default equal-weight score. Separation delta denotes the failure-minus-correct contrast under the perturbed score. Decision stability denotes preservation of the default threshold decision. Score Variant

Perturbation

Rank Stability

Separation Delta

Decision Stability

Rstr default Rstr ablation

none; equal weights support/value components leaveone-out Dirichlet random weights

1.000 1.000 [1.000, 1.000]

[+0.000,

1.000 0.812 [0.250, 1.000]

[+0.000,

1.000 [1.000, 1.000]

none; equal weights process components leave-oneout Dirichlet random weights

1.000 0.790 [0.633, 0.887]

+0.014 +0.014 +0.018] +0.014 +0.050] -0.022 -0.022 +0.006] -0.022 +0.061]

[-0.046,

1.000 0.827 [0.767, 0.933]

[-0.107,

0.811 [0.567, 1.000]

Rstr random weights Rproc default Rproc ablation Rproc weights

random

1.000 [1.000, 1.000]

0.696 [0.022, 0.987]

architecture-dependent.

B.3

nor mutually exclusive ground-truth classes. Table 16 specifies their operational signatures. The state-transition chains in Fig. 7 are schematic examples consistent with these definitions and the observed transition-probability differences. They are not token-level reconstructions, and each trajectory type may contain multiple chain variants.

Endpoint Outcomes and Distinct Trajectory Topologies

To show why endpoint labels are insufficient for the three process-level claims, we compare the concise and analysis tracks on the clean nine-dataset benchmark. Table 14 reports the overall outcome distribution by Qwen2.5 scale. Table 15 and Fig. 8 provide the corresponding task-group breakdown. Improved Endpoint Accuracy Does Not Imply Single Reliable Reasoning-Process Route. In Table 14, the analysis track raises correctness by 13.2, 20.1, 18.6, and 11.7 percentage points for Qwen2.5-1.5B/3B/7B/14B, respectively. Full Hallucination nevertheless increases by 6.1 points for 3B and 7.6 points for 7B, even though it decreases for 1.5B and 14B. Longer reasoning is therefore associated with different outcomes across scales: it may provide more opportunity to build support and verify an answer, but it may also extend an unsupported commitment. Endpoint changes alone cannot distinguish these process routes. The task-group results show that endpoint distributions also depend strongly on task structure. In Table 15, failure is the largest category for formal mathematics (0.718), abstention is the largest for multi-hop QA (0.678), and correctness is the largest for commonsense multiple-choice QA (0.721). Figure 8 displays the same distributions across all seven task groups. These task-dependent mixtures motivate separate analysis of operational and rule failures, missing support, and constraint handling rather than treating an endpoint label as a complete process explanation.

B.4

B.5

CauTrace-Guided Repair and State Steering

Calibration and held-out protocol. We test whether the functional-state/process coordinates identified by CauTrace can guide response repair or internal-state steering on held-out data. All data-dependent choices—including the repair trigger, Trajectory Topology assignment, and steering strength—are fixed using calibration data before test evaluation. The heldout outcomes are therefore used only for evaluation and do not affect the intervention configuration. Reasoning repair. For Qwen2.5-1.5B reasoning, we use 50 calibration pairs to determine which responses should be repaired. A trace is triggered when its process-deviation score Rproc exceeds the 90th percentile of the clean-success calibration traces. This percentile provides a conservative high-risk operating point: it targets traces whose process organization is unusually different from successful clean reasoning, while avoiding unnecessary second-pass generation and test-set threshold tuning. Each triggered trace is also assigned a Trajectory Topology using the fixed CauTrace diagnostics. We then evaluate five conditions on 200 disjoint test pairs. Baseline retains the original response. Generic review applies a diagnosis-free second-pass review with the same decoding and response budget as the CauTrace conditions. Unified CauTrace repair (cautrace_repair) applies a fixed sequence of support reconstruction, rule checking, provisional commitment, and final verification. Topology-conditioned repair uses a correction protocol specialized to the diagnosed Trajectory

Trajectory Signatures

The trajectory types in the main text are rule-defined regions of the trajectory-feature space. They are neither output labels 21

Table 13: Cross-family validation of representation-level and temporal diagnostics. Model

∆Āsup

∆ρV

∆DVKL

∆Γac,V

∆Rvc

LLaMA-3.2-1B LLaMA-3.2-3B LLaMA-3.1-8B Qwen1.5-MoE

-0.052 -0.022 -0.042 -0.012

-0.000 +0.075 +0.073 +0.000

+0.001 +0.002 +0.007 -0.000

-0.058 +0.050 -0.018 +0.057

-13.085 -13.558 -15.519 -1.102

Table 14: Outcome distributions across Qwen2.5 models and prompt tracks. Len. denotes mean generation length. Model

Track

1.5B 1.5B 3B 3B 7B 7B 14B 14B

Concise Analysis Concise Analysis Concise Analysis Concise Analysis

Correct

Partial

Full

Abstain

Invalid

Len.

19.7 32.9 12.1 32.2 15.8 34.4 18.5 30.2

0.5 1.2 0.3 1.2 0.4 1.3 0.4 0.7

54.4 43.7 20.1 26.2 12.7 20.3 16.1 12.7

24.7 21.9 65.1 40.1 71.1 44.0 58.9 56.4

0.7 0.2 2.3 0.3 0.0 0.0 6.1 0.0

16.0 76.5 25.5 67.5 6.0 41.3 29.1 55.7

Correct

Hallucination

Abstain

Invalid

1.0

0.11

0.14

0.18

0.38

0.44 0.25

0.18

0.26

0.2

0.24

0.4

0.49

0.54

0.72

0.72

0.6

0.26

Outcome proportion

0.8

0.0 nse

nse

Co

o mm

Q

MC

licit

Imp

ing

son

rea

ic/r

Log

ule

ath

lm

ma

For

n latio lcu Ca

A

pQ

o lti-h

Mu

dge

wle Kno

QA

Figure 8: Task-group outcome distributions for Qwen2.5 generations. The horizontal bars visualize the proportions reported in Table 15; they are not independent estimates. Topology, whereas wrong-topology repair uses a deliberately mismatched topology while preserving the same prompt structure and generation budget. Only triggered traces are regenerated; non-triggered traces retain their baseline response under every second-pass condition. These controls distinguish improvement from generic additional review, improvement from the unified CauTrace protocol, and improvement that specifically depends on the diagnosed topology. Table 17 reports the held-out endpoint results.

Table 15: Task-group outcome composition, aggregated across Qwen2.5 scales and prompt tracks. Task group Calculation Formal math Multi-hop QA Knowledge QA Logic/rule Commonsense MCQ Implicit reasoning

Size

Correct

Failure

Abstain

Invalid

7032 10000 61176 656 5952 9680 2224

0.445 0.240 0.137 0.110 0.252 0.721 0.537

0.380 0.718 0.185 0.494 0.177 0.263 0.262

0.090 0.042 0.678 0.396 0.486 0.013 0.188

0.085 0.000 0.000 0.000 0.086 0.003 0.013

State steering. For Qwen2.5-7B safety, we test whether the same diagnostics can guide a direct hidden-state intervention. Let valign denote the normalized steering direction constructed from the signed state directions predicted by the diagnosed trajectory: positive support, formal rule, and verification directions are combined with a negative answer-commitment direction. At generation step t and intervened layer ℓ, steering 22

Table 16: Operational signatures of the Trajectory Topology regions. Arrows indicate the expected direction relative to Normal Convergence. A sample may satisfy more than one signature. Trajectory

Metric Signature

Early Commitment

τcom < τsup or gcs ↓

Fragmented Support

Ω ↑, S̄ ↓, unstable τsup

Rule Failure

Lrule ↑ with weak FRT/OC state scores

Verification without Overwrite Verbose Unsupported Continuation Normal Convergence

gvc ↑, Rvc ↓ T ↑ with S̄ ↓ and Rvc ↓ τsup ≤ τcom , bounded Ω, Rvc ↑

Table 17: Held-out Qwen2.5-1.5B reasoning repair on 200 test pairs. Endpoint rates are percentages. Clean good denotes the proportion of clean responses that remain correct, and ∆Fail is the adversarial-failure change relative to the baseline in percentage points.

Diagnostic Interpretation Answer commitment becomes dominant before support is sufficiently established. Support-related states activate but do not form a stable, continuous chain. The trace approaches an answer without preserving formal rules, output constraints, or symbolic/numerical consistency. Verification follows commitment but remains too weak to revise the stabilized answer. Generation continues without a corresponding increase in support or effective verification. Support precedes commitment, constraints remain active, and verification can confirm or revise the candidate.

Table 18: Held-out Qwen2.5-7B safety steering on 12 test pairs. Process deltas pool the clean and adversarial member of each pair; lower Rproc and higher rule and verification coverage are better.

Condition Adv. fail Clean ∆Rproc ∆Rulecov Condition Baseline Unified CauTrace Generic review Topology-conditioned Wrong topology

Adv. fail

Clean good

∆Fail

76.0 71.0 72.0 73.0 73.5

26.0 31.0 32.0 29.5 30.5

– -5.0 -4.0 -3.0 -2.5

Baseline Aligned Random Reversed Wrong state

0 0 +0.011 +0.021 +0.002 0.000 -0.014 -0.019 -0.007 -0.026

−valign ; random steering uses a norm-matched direction orthogonal to valign ; and wrong-state steering strengthens answer commitment while weakening formal rule tracking and verification. All four conditions use the same α, layers, intervention duration, and decoding configuration. The controls therefore test whether the predicted state direction changes the reasoning process beyond the effect of a generic hiddenstate perturbation. Table 18 reports both endpoint outcomes and online process-coordinate changes. Unified Repair Gives the Largest Observed 1.5B Failure Reduction, and Aligned Steering Moves the 7B Process Measures in the Predicted Direction. On the full 1.5B reasoning test, unified CauTrace repair reduces adversarial failure from 76.0% to 71.0%, corresponding to a 5.0-percentagepoint reduction (paired 95% bootstrap CI [−8.5, −2.0]; McNemar p = 0.0063), with 11 failed responses rescued and one previously successful response regressing. This is the largest observed reduction among the five conditions in Table 17; generic review reduces failure by 4.0 points, while topology-conditioned and wrong-topology repair reduce it by 3.0 and 2.5 points, respectively. The direct 1.0-point differ-

is applied as h′ℓ,t = hℓ,t + α RMS(hℓ,t ) valign .

100.0 100.0 0 66.7 91.7 -0.008 66.7 100.0 0.000 66.7 91.7 +0.015 66.7 91.7 +0.010

∆Vcov

(24)

where RMS normalization makes the intervention scale comparable across layers and responses, and α controls its strength. We select α using six calibration pairs and a predefined grid 0.25, 0.5, 1.0. The selected value minimizes adversarial failure on the calibration pairs subject to a maximum twopercentage-point reduction in clean-good rate. This constraint prevents the intervention strength from being selected solely for adversarial recovery at the expense of clean behavior. The resulting value is α = 0.25, which is fixed before evaluation on 12 disjoint clean-success/adversarial-failure test pairs. The direction is injected into a three-layer band centered on the CauTrace-identified risk layer during the first eight generated tokens. This restricts the intervention to the diagnosed model region and the early generation stage in which the response trajectory is formed. We compare aligned steering with three equal-budget controls: reversed steering uses 23

ence between unified CauTrace repair and generic review is not statistically significant (95% CI [−4.0, +1.5]; p = 0.727). The result therefore shows that the complete support–rule– commitment–verification protocol can improve held-out reasoning outcomes, but does not establish superiority over generic review or a topology-specific repair effect. On 7B safety, all four steering directions reduce adversarial failure from 100.0% to 66.7% in Table 18; endpoint recovery alone therefore does not distinguish the aligned direction from its controls. The internal process changes, however, follow different directions. Aligned steering is the only condition that simultaneously lowers process risk (∆Rproc = −0.008), increases rule coverage (+0.011), and increases verification coverage (+0.021). Reversed and wrong-state steering instead increase Rproc and reduce both rule and verification coverage, whereas the random control remains close to zero. The same pattern is stronger among the adversarial responses rescued by aligned steering, for which ∆Rproc = −0.051, ∆Rulecov = +0.038, and ∆Vcov = +0.042. Thus, aligned steering produces the process-direction changes predicted by the support/rule–commitment–verification relation in Claim II. Given the 12-pair test and the identical endpoint rates across steering controls, this result supports theory-consistent process steering in the evaluated setting without attributing endpoint recovery to a unique causal state direction. B.5.1

Discussion: Defense Implications and Future Directions

These results suggest a concrete path from the three-level diagnostics to process-aware defenses. Structural retention can identify where an intervention may be applied, functionalstate/process coordinates can specify whether support, rule tracking, and verification should be strengthened or premature commitment suppressed, and Trajectory Topology can provide a candidate basis for selecting repair policies. In the present tests, unified repair provides the largest observed 1.5B reasoning reduction, while aligned steering moves the 7B process measures in the predicted direction, showing that CauTrace signals are actionable in the evaluated settings. Future work can develop this fixed intervention into a closed-loop defense that monitors process risk during generation, triggers repair only when needed, and adapts the repair policy or steering strength while preserving clean behavior. Evaluation across broader model families, tasks, attacks, and held-out samples, together with stronger direction-specific controls, will clarify when these process-level improvements translate into consistent endpoint robustness.

24

Related documents

• Uncovering Why LLMs Fail: From Internal Mechanisms to Adversarial Reasoning Trajectories#1468973
• SABER: Stability-Aware Early Exit for LLM Reasoning via Adversarial Branch Probing#784740
• SABER: Stability-Aware Early Exit for LLM Reasoning via Adversarial Branch Probing#783150
• Who Bridges Safety? Identifying and Targeting Cross-Lingual Shared Safety Pathways#1144864
• Breaking the Mirror: Activation-Based Mitigation of Self-Preference in LLM Evaluators#1216184