R EASONING L ENS: Hierarchical Visualization and Diagnostic Auditing for Large Reasoning Models Jun Zhang1,2 , Jiasheng Zheng1,2 , Boxi Cao1 * , Yaojie Lu1 , Hongyu Lin1 * Jia Zheng1 , Xianpei Han1 , Le Sun1 1 Chinese Information Processing Laboratory Institute of Software, Chinese Academy of Sciences 2 University of Chinese Academy of Sciences {zhangjun2025,zhengjiasheng2022,caoboxi,hongyu}@iscas.ac.cn
arXiv:2606.23404v1 [cs.CL] 22 Jun 2026
Abstract
Calculating radius $r = \sqrt{0^2 + 3^2} = \sqrt{9} = 3$. Now find $\theta$. Using $\theta = \arctan(y/x)$ gives $\arctan(3/0)$. Division by zero! Need to check the quadrant for $(0,3)$. $x=0$ means it’s on the y-axis. Since $y=3$ is positive, it must be $\pi/2$. Let me verify: $\cos\theta = x/r = 0/3 = 0$ and $\sin\theta = y/r = 3/3 = 1$. Wait, check if $3\pi/2$ is possible? No, $y$ is positive. Also, $r=3$ is positive, so it matches. Let's re-verify the conversion back: $x = 3 \cos(\pi/2) = 0$, $y = 3 \sin(\pi/2) = 3$. The point $(0, 3)$ is recovered. The reasoning is consistent. No negative radius or angle wrapping needed. Final answer is $(3, \pi/2)$.
The emergence of Large Reasoning Models has introduced exceptionally long Chain-ofThought traces, creating a transparency burden where critical logic is often buried under massive procedural text. To address this, we present R EASONING L ENS, an open-source framework designed for the hierarchical visualization and diagnostic auditing of complex reasoning chains. R EASONING L ENS addresses information necropsy by: (1) structuring traces into interactive hierarchies that separate highlevel strategy from low-level execution; (2) leveraging an agentic auditor for automated error detection and tool-augmented verification; and (3) synthesizing systemic reasoning profiles to reveal model-specific blind spots. By transforming unstructured walls of text into actionable insights, R EASONING L ENS1 provides a modular foundation for interpreting, debugging, and optimizing the next generation of reasoning-centric AI.
1
Raw Long CoT
Hierarchical Visualization
Analysis by
ERROR: Logic Error (Division by zero) SUGGESTION: Use case-based reasoning, …
Agentic Diagnosis
Systemic Profiling
ReasoningLens
Figure 1: The main components in R EASONING L ENS for analyzing raw CoTs.
critical logical dependencies become buried under an unstructured “wall of text”, leading to reduced structural transparency and increased verification burden. This scaling-induced opacity significantly challenges manual inspection, error diagnosis, and safety assurance. While previous studies have attempted to visualize the structure of reasoning traces (Pang et al., 2025; Zhou et al., 2026), they remain largely heuristic and descriptive, often restricted to superficial text rendering. Crucially, such methods lack a comprehensive framework to guide the design of a structured taxonomy, leaving a gap between simple visualization and deep-level analysis. Consequently, current work remains an isolated, ad-hoc process rather than a diagnostic tool. We argue that the structured visualization should not be designed in a vacuum, but must be purpose-driven to serve two critical functions: as a structural scaffold for understanding the model’s internal logic, and as a diagnostic foundation for localizing and profiling reasoning failures. To bridge this gap, we present R EASON ING L ENS, an open-source framework built around
Introduction
Large language models have rapidly evolved from fluent text generators into systems capable of extended, multi-step reasoning (Xu et al., 2025). Recent Large Reasoning Models (LRMs) such as Deepseek-R1 (Guo et al., 2025), GPT-5 (Singh et al., 2025), and Qwen3 (Yang et al., 2025) exhibit the ability to generate detailed chains-of-thought, decompose complex problems, self-correct intermediate steps, and even simulate deliberative processes. However, the scaling of reasoning length introduces a structural trade-off between expressiveness and interpretability (Turpin et al., 2023; Chen et al., 2025; Arcuschin et al., 2025). As reasoning traces scale to tens of thousands of tokens, * Corresponding authors. 1 Our code is available at: https://github.com/ Our dataset is available icip-cas/ReasoningLens. at: https://hf.co/datasets/LasRuinasCirculares/ LensBench. The demonstration video is available at: https://youtu.be/sVZ8yYrpCYk.
1
a principled pipeline that moves from visualization through diagnosis to profiling, providing a systematic foundation for deep-level model analysis. As demonstrated in Figure 1, R EASONING L ENS introduces a multi-granularity approach to reasoning analysis: (1) Hierarchical visualization: Guided by a meticulously designed taxonomy of reasoning behaviors, R EASONING L ENS transforms long CoT into multi-layered, interactive reasoning graphs. This provides an intuitive structural abstraction for researchers to audit the model’s cognitive trajectories across varying levels of granularity. (2) Agentic Diagnosis: Guided by a comprehensive taxonomy of reasoning errors, we implement a multi-agent system comprising Memory, Verification, and Suggestion modules. This agentic architecture enables fine-grained error localization and actionable feedback within complex reasoning traces, shifting the paradigm from passive observation to active diagnostic intervention. (3) Systemic Profiling: By leveraging cross-trajectory structured representation and a hierarchical evidence compression mechanism, R EASONING L ENS achieves a holistic modeling of model-level reasoning behaviors. This allows researchers to systematically diagnose strategic biases and stability bottlenecks, facilitating interpretable model comparison and principled iterative optimization. Overall, by transforming unstructured walls of text into actionable insights, R EASONING L ENS provides a modular foundation for interpreting, debugging and optimizing the next generation of reasoning AI. To validate the effectiveness of R EASON ING L ENS in both hierarchical visualization and agentic diagnosis, we further construct L ENS B ENCH, a unified benchmark targeting long-form CoT reasoning. Specifically, L ENS B ENCH comprises 130 instances spanning 5 representative categories of reasoning failures(Sui et al., 2025; Wang et al., 2025a,b; Song et al., 2025, 2026; Mirzadeh et al., 2025), each annotated with both explorationlevel hierarchical structure and fine-grained error types. Experimental results show that when instantiated with top-performing models, R EASON ING L ENS achieves consistent and reliable performance across both hierarchical visualization and error identification tasks. Further case studies show that R EASON ING L ENS provides reliable structural transparency and actionable diagnostic insights by transforming monolithic CoT traces into an auditable logical framework.
These findings confirm that R EASONING L ENS transcends mere heuristic rendering, providing a formalized foundation for diagnosing the internal logic of LRMs and offering a scalable path toward transparent and verifiable machine deliberation. The major contribution of this paper is summarized as follows: • We formalize a multi-granularity diagnostic framework tailored for the transparency burden of large reasoning models. • We present R EASONING L ENS, an opensource diagnostic toolkit for LRMs, enabling hierarchical visualization, automated diagnosis, and systematic profiling. • We construct L ENS B ENCH, a unified benchmark assessing both structural visualization fidelity and CoT error-annotation accuracy, which sheds light on the development of future reasoning analysis tools.
2
Related Work
The externalized thinking processes of LRMs make manual inspection increasingly difficult (Korbak et al., 2025; Guan et al., 2025), driving research into reasoning transparency, particularly error analysis and visualization. Prior work has identified failure modes such as redundant traces (Sui et al., 2025), sensitive content leakage (Green et al., 2025), and logical inconsistencies (Mündler et al., 2024). While some efforts broaden error coverage to evaluate process reward models (He et al., 2025), their taxonomies are limited to reasoning-related failures in math and coding tasks. No existing work provides a holistic taxonomy to evaluate the interplay of pathological behaviors across models. To improve trace interpretability, visualization systems render CoT as graphical interfaces to reduce cognitive load (Li et al., 2025; Pang et al., 2025; Zhou et al., 2026; Felder et al., 2025), yet rely on superficial text without capturing underlying reasoning structures. Structural approaches parse traces into step-level taxonomies to characterize reasoning behavior (Lee et al., 2026; Shahariar et al., 2026; Xue et al., 2025), but stop short of actionable error localization. R EASONING L ENS addresses this gap with a unified framework for structural modeling and automated CoT diagnosis, revealing model-specific strengths and recurring blind spots to improve transparency and guide the development of more robust LRMs. 2
Hierarchical Visualization Systemic Profiling
Agentic Diagnosis
Figure 2: The R EASONING L ENS framework enables (a) Hierarchical Visualization: mapping CoT segments to reasoning nodes via hovering; (b) Agentic Diagnosis: automatically detecting reasoning flaws (e.g., overthinking) with actionable fixes; and (c) Systemic Profiling: generating comprehensive reports on behavior patterns of LRMs.
3
System Design
2. Backtracking is a failure-triggered reasoning behavior invoked when the current strategy cannot produce a feasible solution. It prunes the failed branch and resumes search from a prior decision point to explore alternatives. 3. Validation acts as a self-correction mechanism to verify intermediate or final conclusions. Topologically, it introduces verification loops into the trajectory before committing to a path or triggering a backtrack.
As illustrated in Figure 2, R EASONING L ENS comprises three core components: Hierarchical Visualization, Agentic Diagnosis and Systemic Profiling, that collectively transform raw CoT traces into structured, actionable insights, facilitating both precise instance-level inspection and holistic modellevel behavioral profiling. 3.1
Hierarchical Visualization
Exploitation-Level, the low-level procedural execution units apply known formulas to flesh out the reasoning steps. These behaviors instantiate the plan by sourcing needed premises, applying procedural transformations, and committing intermediate states.
To establish a systematic pipeline for visualizing intricate and multifaceted reasoning processes, we first detail a comprehensive taxonomy of reasoning behaviors. Building upon this, we develop a unified yet hierarchical graphical representation. The Taxonomy Of Reasoning Behaviors Reasoning traces exhibit observable cognitive operations invoked by the model to navigate from a problem statement to its solution. Based on their functional roles, we categorize these behaviors into two distinct levels, including exploration-level and exploitation-Level. Exploration-Level, the highlevel strategic moves that orchestrate the search over the solution space. Rather than merely advancing the state, these behaviors act as navigational maneuvers that actively dictate topology transitions and redirect the global reasoning trajectory:
1. Knowledge Retrieval extracts task-relevant priors from parametric memory or provided context. It supplies the explicit premises required for subsequent procedural steps. 2. Procedural Execution performs rulegoverned transformations over instantiated inputs. This produces derived intermediate values that directly advance the local inference chain. 3. State Assertion explicitly commits local assumptions or intermediate findings into the working state. This makes critical information available for reference and propagation in downstream steps.
1. Decomposition adopts a divide-and-conquer strategy that partitions complex, multi-hop problems into a sequence of atomic sub-goals. Structurally, it expands a linear reasoning path into a tree of manageable sub-trajectories.
Based on this taxonomy, we implement a comprehensive framework to achieve structured modeling of the reasoning chain. 3
Planning Unit Extraction Guided by the proposed taxonomy, we extract reasoning behaviors from unstructured CoT text by segmenting each trace into atomic planning units, which serve as the primitive elements for all subsequent graph-based modeling, each minimal and semantically coherent. In practice, transitions between reasoning strategies are frequently signaled by decision-oriented lexical cues (e.g., “but”, “wait”, “alternatively”, “try another approach”). We leverage these linguistic markers to partition a monolithic reasoning trace into this sequence of units, enabling fine-grained labeling of exploration and exploitation behaviors at the unit level.
The Taxonomy Of Error Types While growing body of work has investigated reasoning errors in LRMs (Chua and Evans, 2025; Arrieta et al., 2025; Marjanović et al., 2026), their efforts often stem from disparate perspectives and lack an organized framework for integrating. This fragmentation hinders the rigorous analysis and detection of long CoT quality. To bridge this gap, we synthesize existing observations into a unified error taxonomy comprising five primary categories, which facilitates the precise identification and localization of reasoning failures. 1. Overthinking: Redundant reasoning cycles (e.g., repeated verification, circular loops, overelaboration of simple tasks) that increase deliberation time without yielding better results (Peng et al., 2025; Sui et al., 2025). 2. Safety: Increased risk of generating harmful content (e.g., toxicity, bias) or leaking sensitive information as LRMs incorporate external information and explore potential reasoning paths (Green et al., 2025; Qiu et al., 2025). 3. Knowledge Error: Incorrect recall or misuse of established knowledge (e.g., factual hallucinations), including the use of outdated information or incorrect definitions (Su et al., 2024; Song et al., 2025). 4. Logical Error: The use of flawed reasoning strategies or incoherent inference steps (e.g., non-sequiturs or internal contradictions) that violate logical consistency and lead to invalid conclusions (Mündler et al., 2024). 5. Formal Error: Non-compliance with strict symbolic rules (e.g., syntax, LATEX, or arithmetic) in programming and mathematical contexts, resulting in invalid formal output (Gao et al., 2023; Tong and Zhang, 2024).
Exploration-Level Modeling To expose latent intent shifts and alternative branches, we abstract the explicitly ordered trace of atomic planning units S = (u1 , . . . , uN ) into a macro-level exploration graph Gmacro = (Vmacro , Emacro ). Specifically, we leverage an LLM to partition S into M disjoint contiguous spans, collapsing each into a macro-node vj ∈ Vmacro . Each vj encapsulates a coherent strategic operation, typed by a predefined functional role TVmacro (e.g., problem decomposition, validation, intermediate answers). These macro-nodes are connected via structural edges in TEmacro (e.g., forward reasoning, backtracking, detached verification) to form a coarse-grained, tree-structured hierarchy that captures the global reasoning trajectory. Exploitation-Level Modeling Complementing the high-level strategic transitions, the exploitation level characterizes the fine-grained operational execution within a selected reasoning path. For a given macro-node vj spanning (ui , . . . , ui+k ), we further refine its internal units into a local execution subgraph. This transforms the underlying CoT segment into an ordered sequence of micronodes Vmicro (vj ) = ⟨ṽ1 , . . . , ṽL ⟩, where each ṽℓ is labeled with a specific execution behavior from TVmicro . Connected by micro-edges TEmicro , this subgraph encodes the procedural dependencies required for precise error attribution. 3.2
Agentic Error Detection To operationalize our proposed reasoning error taxonomy, we design a unified multi-agent framework for effective error detection. The framework features a memory module that incrementally inspects the CoT through memory compression, ensuring granular localization of local errors while preserving trace-level consistency. Complementing this, a verification module strategically leverages external tool invocations to resolve internal ambiguity, enabling precise and verifiable diagnostic outcomes.
Agentic Diagnosis
The expansion of CoT complexity often masks implicit reasoning errors, making them harder to identify. To tackle this, we introduce a comprehensive taxonomy of reasoning errors alongside a multiagent framework. This framework leverages three core modules (Memory, Verification, and Suggestion) to enable scalable and robust error diagnosis.
Actionable Fix Suggestions Building upon the preceding structural modeling and error local4
Trace Structuring Annotation We annotate each reasoning trace with its exploration-level structure using GPT-5.4 according to our reasoning behavior taxonomy. These annotations enable evaluation of whether R EASONING L ENS can recover the global exploration topology.
ization, R EASONING L ENS closes the analytical loop by generating actionable mitigation strategies mapped to specific error types. Rather than offering generic advice, the system leverages a curated repository covering two complementary paradigms: training-free approaches and post-training techniques aimed at refining intrinsic reasoning behaviors. By dynamically aligning identified bottlenecks with this repository, users can rapidly identify and apply relevant interventions. 3.3
Reasoning Error Annotation Since naturally occurring errors in long-CoT traces are heavily imbalanced across failure types (He et al., 2025), we introduce controlled, taxonomy-guided errors into otherwise clean trajectories. Given the query, seed trace, and error taxonomy, GPT-5.4 identifies plausible insertion points, selects compatible failure types, and rewrites targeted spans to produce globally coherent error injections.
Systemic Profiling
Characterizing model reasoning requires moving beyond instance-level trace inspection towards distribution-level trajectory analysis. To this end, the Systemic Profiling module aggregates a trajectory set into a unified behavioral representation, capturing model-level reasoning dynamics and enabling diagnosis of reasoning bottlenecks. Tree-structured artifacts and failure signals are first mapped into a shared representation to extract global invariants, including search topology (e.g., backtracking distributions) and node-level annotations (e.g., error types). LLM-driven semantic deduplication then compresses similar reasoning paths while preserving distinct heuristic features, yielding a distilled evidence base. This evidence is finally synthesized into a structured modellevel profile along three axes: exploration habits (depth–breadth trade-offs), verification reliability (self-correction consistency), and stability bottlenecks (high-variance logical structures).
4
Experiment
4.1
Evaluation Dataset Construction
Human Verification. To ensure L ENS B ENCH quality, we manually review all candidate instances, discarding those with incoherent rewrites, ambiguous failures, or unreliable annotations, resulting 130 verified examples with gold annotations for both evaluation dimensions. The verification guidelines are provided in Appendix B, and a representative annotated case is shown in Appendix C. 4.2
Experimental Setup
We evaluate R EASONING L ENS on L ENS B ENCH using five backbone evaluators: DeepSeek-V4Pro (DeepSeek-AI, 2026), MiniMax-M2.7 (MiniMax et al., 2026), Qwen3.5-27B (Team, 2026), Gemma-4-26B-A4B (Google DeepMind, 2026), and Qwen3-32B (Yang et al., 2025), spanning multiple model families and parameter scales. For hierarchical trace structuring, we report Node Type Accuracy (NTA) and Graph Edit Similarity (GES). NTA measures whether predicted structural nodes are assigned the correct functional types. GES measures the normalized similarity between predicted and gold exploration-level reasoning graphs. Formal definitions are in For reasoning error diagnosis, we report pertype and micro-averaged Precision, Recall, and F1 scores. Per-type results reveal category-specific behavior, while micro-averaged scores summarize overall diagnostic performance.
To empirically evaluate the effectiveness of R EA SONING L ENS in both Hierarchical Visualization and Agentic Diagnosis, we construct L ENS B ENCH, a benchmark providing gold annotations for trace structuring and fine-grained reasoning error over long CoT traces. Seed Selection We use Mixture-of-Thoughts (Fein-Ashley et al., 2025), a publicly available longCoT benchmark, as our source corpus. We retain only traces containing at least 10 planning units to ensure sufficient reasoning complexity, remove mixed-language traces to reduce annotation noise, and apply GPT-5.4 to filter out traces with preexisting reasoning errors, yielding a clean seed set suitable for controlled error injection.
4.3
Main Results
R EASONING L ENS’s Agentic Diagnosis module demonstrates reliable error detection capability that scales with backbone model strength, with overall diagnostic F1 ranging from 66.3 on 5
Trace Structuring
Reasoning Error Diagnosis Model
DeepSeek-V4-Pro MiniMax-M2.7 Qwen3.5-27B Gemma-4-26B-A4B Qwen3-32B
Overthinking
Safety
Knowledge Error
Logical Error
Formal Error
Overall
P
R
F1
P
R
F1
P
R
F1
P
R
F1
P
R
F1
P
R
F1
85.8 88.0 73.8 88.3 78.3
88.3 86.4 88.7 81.7 84.5
87.0 87.2 80.6 84.9 81.3
100.0 97.0 100.0 100.0 86.1
97.0 97.0 93.9 84.8 93.9
98.5 97.0 96.9 91.8 89.9
63.6 47.6 52.6 77.8 40.0
66.7 47.6 47.6 33.3 38.1
65.1 47.6 50.0 46.7 39.0
80.4 71.7 77.1 71.2 52.4
48.2 44.7 31.8 43.5 25.9
60.3 55.1 45.0 54.0 34.6
86.0 93.8 74.0 73.3 51.4
86.0 70.9 82.6 51.2 41.9
86.0 80.8 78.0 60.3 46.2
85.1 85.3 75.1 83.8 69.6
79.7 74.2 74.9 66.2 63.2
82.3 79.4 75.0 74.0 66.3
NTA
GES
79.5 73.0 77.4 70.9 74.0
72.3 69.0 70.8 68.2 68.0
Table 1: Performance of R EASONING L ENS on L ENS B ENCH. The Agentic Diagnosis module achieves reliable error detection across diverse failure types, while the Hierarchical Visualization module consistently reconstructs exploration-level reasoning graphs across all evaluated models.
Qwen3-32B to 82.3 on DeepSeek-V4-Pro. Specifically, Safety detection remains consistently high across all models, demonstrating that current safety alignment has effectively reinforced safety boundaries. In contrast, Knowledge Error and Logical Error detection are strongly model-dependent, with F1 scores as low as 39.0 and 34.6 on Qwen3-32B, yet recovering notably on stronger models, suggesting that diagnosing deeper reasoning failures relies heavily on the model’s internal reasoning capacity. R EASONING L ENS’s Hierarchical Visualization module proves consistently robust across model scales, maintaining stable NTA and GES scores of 75.0 and 69.7 on average despite substantial variation in overall diagnostic F1. By segmenting raw reasoning text into semantically coherent planning units, R EASONING L ENS converts long-context understanding into a locally-scoped labeling task, reducing model burden and validating the hierarchical visualization design as a robust component independent of backbone choice.
5
Figure 3: We compared human-annotated CoT with the structured representation generated by ReasoningLens to validate our structuring accuracy. Red highlights identify reasoning errors, while blue highlights denote structural cues for hierarchy construction.
Case Analysis
We demonstrate that R EASONING L ENS achieves reliable structural transparency and actionable diagnostic insights by transforming monolithic CoT traces into an auditable, multi-dimensional logical framework. As illustrated in Figure 3, R EA SONING L ENS decomposes the reasoning trajectory of Qwen3-32B into semantically coherent functional blocks (e.g., Strategy Shift and Verification), with high alignment to human-annotated ground truth. Crucially, the Agentic Diagnosis module surfaces critical reasoning vulnerabilities that are otherwise obscured by verbosity, including unsafe manipulation tactics and redundant overthinking, and further associates them with targeted remediation strategies (e.g., Early Stopping) in the “How to fix” panel. This demonstrates that R EASON ING L ENS not only enhances model interpretability but also effectively closes the loop between latent reasoning analysis and model alignment.
6
Conclusion
We present R EASONING L ENS, a framework that reframes Long CoT analysis from passive observation to active, structured interpretation through hierarchical reasoning graphs, providing a multigranular view of reasoning trajectories. Our multiagent diagnostic system, integrating memory, verification, and suggestion modules, enables precise error localization and actionable feedback, while our systemic profiling mechanism uncovers modellevel reasoning bottlenecks. As reasoning models continue to scale, R EASONING L ENS offers a critical step toward transparent, verifiable, and reliable machine deliberation. 6
Limitations
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Pal: Program-aided language models. Preprint, arXiv:2211.10435.
Although R EASONING L ENS establishes a systematic pipeline from visualization through diagnosis to profiling for long-form reasoning, it currently focuses primarily on static chain-of-thought traces rather than dynamic, multi-step agentic interactions. In future work, we plan to extend the framework to support agentic trajectory analysis, specifically capturing the Plan-Act-Observe cycle to better model interactive reasoning. Additionally, while the current system provides a comprehensive diagnostic foundation, its deployment remains relatively monolithic. We aim to evolve R EASON ING L ENS into a modular plugin ecosystem, facilitating lightweight integration and deployment for process-based training supervision in subsequent development.
Google DeepMind. 2026. Gemma 4 model card. Accessed: 2026-06-18. Tommaso Green, Martin Gubri, Haritz Puerto, Sangdoo Yun, and Seong Joon Oh. 2025. Leaky thoughts: Large reasoning models are not private thinkers. Preprint, arXiv:2506.15674. Melody Y. Guan, Miles Wang, Micah Carroll, Zehao Dou, Annie Y. Wei, Marcus Williams, Benjamin Arnav, Joost Huizinga, Ian Kivlichan, Mia Glaese, Jakub Pachocki, and Bowen Baker. 2025. Monitoring monitorability. Preprint, arXiv:2512.18311. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, and 175 others. 2025. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature, 645(8081):633–638.
References
Yancheng He, Shilong Li, Jiaheng Liu, Weixun Wang, Xingyuan Bu, Ge Zhang, Zhongyuan Peng, Zhaoxiang Zhang, Zhicheng Zheng, Wenbo Su, and Bo Zheng. 2025. Can large language models detect errors in long chain-of-thought reasoning? Preprint, arXiv:2502.19361.
Iván Arcuschin, Jett Janiak, Robert Krzyzanowski, Senthooran Rajamanoharan, Neel Nanda, and Arthur Conmy. 2025. Chain-of-thought reasoning in the wild is not always faithful. Preprint, arXiv:2503.08679. Aitor Arrieta, Miriam Ugarte, Pablo Valle, José Antonio Parejo, and Sergio Segura. 2025. Early external safety testing of openai’s o3-mini: Insights from the pre-deployment evaluation. Preprint, arXiv:2501.17749.
Tomek Korbak, Mikita Balesni, Elizabeth Barnes, Yoshua Bengio, Joe Benton, Joseph Bloom, Mark Chen, Alan Cooney, Allan Dafoe, Anca Dragan, Scott Emmons, Owain Evans, David Farhi, Ryan Greenblatt, Dan Hendrycks, Marius Hobbhahn, Evan Hubinger, Geoffrey Irving, Erik Jenner, and 22 others. 2025. Chain of thought monitorability: A new and fragile opportunity for ai safety. Preprint, arXiv:2507.11473.
Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato, Carson Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner, Fabien Roger, Vlad Mikulik, Samuel R. Bowman, Jan Leike, Jared Kaplan, and Ethan Perez. 2025. Reasoning models don’t always say what they think. Preprint, arXiv:2505.05410.
Jinu Lee, Shivam Agarwal, Amruta Parulekar, Siddarth Madala, Dilek Hakkani-Tur, and Julia Hockenmaier. 2026. Reasoningflow: Discourse structures for understanding llm reasoning traces. Preprint, arXiv:2606.05402.
James Chua and Owain Evans. 2025. Are deepseek r1 and other reasoning models more faithful? Preprint, arXiv:2501.08156.
Zongqian Li, Ehsan Shareghi, and Nigel Collier. 2025. Reasongraph: Visualisation of reasoning paths. Preprint, arXiv:2503.03979.
DeepSeek-AI. 2026. Deepseek-v4: Towards highly efficient million-token context intelligence.
Sara Vera Marjanović, Arkil Patel, Vaibhav Adlakha, Milad Aghajohari, Parishad BehnamGhader, Mehar Bhatia, Aditi Khandelwal, Austin Kraft, Benno Krojer, Xing Han Lù, Nicholas Meade, Dongchan Shin, Amirhossein Kazemnejad, Gaurav Kamath, Marius Mosbach, Karolina Stańczak, and Siva Reddy. 2026. Deepseek-r1 thoughtology: Let’s think about llm reasoning. Preprint, arXiv:2504.07128.
Jacob Fein-Ashley, Dhruv Parikh, Rajgopal Kannan, and Viktor Prasanna. 2025. Mixture of thoughts: Learning to aggregate what experts think, not just what they say. Preprint, arXiv:2509.21164. Ludwig Felder, Jacob Miller, Markus Wallinger, Stephen Kobourov, and Chunyang Chen. 2025. Retrace: Interactive visualizations for reasoning traces of large reasoning models. Preprint, arXiv:2511.11187.
MiniMax, :, Aili Chen, Aonian Li, Baichuan Zhou, Bangwei Gong, Binyang Jiang, Boji Dan, Changqing Yu, Chao Wang, Cheng Ma, Cheng Zhong, Cheng
7
Zhu, Chengjun Xiao, Chengyi Yang, Chengyu Du, Chenyang Zhang, Chi Zhang, Chuangyi Huang, and 188 others. 2026. The minimax-m2 series: Mini activations unleashing max real-world intelligence. Preprint, arXiv:2605.26494.
Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Na Zou, Hanjie Chen, and Xia Hu. 2025. Stop overthinking: A survey on efficient reasoning for large language models. Preprint, arXiv:2503.16419.
Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. 2025. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. Preprint, arXiv:2410.05229.
Qwen Team. 2026. Qwen3.5: Accelerating productivity with native multimodal agents. Weixi Tong and Tianyi Zhang. 2024. Codejudge: Evaluating code generation with large language models. Preprint, arXiv:2410.02184.
Niels Mündler, Jingxuan He, Slobodan Jenko, and Martin Vechev. 2024. Self-contradictory hallucinations of large language models: Evaluation, detection and mitigation. Preprint, arXiv:2305.15852.
Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. 2023. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. Preprint, arXiv:2305.04388.
Rock Yuren Pang, K. J. Kevin Feng, Shangbin Feng, Chu Li, Weijia Shi, Yulia Tsvetkov, Jeffrey Heer, and Katharina Reinecke. 2025. Interactive reasoning: Visualizing and controlling chain-of-thought reasoning in large language models. Preprint, arXiv:2506.23678.
Cheng Wang, Yue Liu, Baolong Bi, Duzhen Zhang, Zhong-Zhi Li, Yingwei Ma, Yufei He, Shengju Yu, Xinfeng Li, Junfeng Fang, Jiaheng Zhang, and Bryan Hooi. 2025a. Safety in large reasoning models: A survey. Preprint, arXiv:2504.17704.
Keqin Peng, Liang Ding, Yuanxin Ouyang, Meng Fang, and Dacheng Tao. 2025. Revisiting overthinking in long chain-of-thought from the perspective of selfdoubt. Preprint, arXiv:2505.23480.
Yanbo Wang, Yongcan Yu, Jian Liang, and Ran He. 2025b. A comprehensive survey on trustworthiness in reasoning with large language models. Preprint, arXiv:2509.03871.
Jianing Qiu, Lin Li, Jiankai Sun, Hao Wei, Zhe Xu, Kyle Lam, and Wu Yuan. 2025. Emerging cyber attack risks of medical ai agents. Preprint, arXiv:2504.03759.
Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, Chenyang Shao, Yuwei Yan, Qinglong Yang, Yiwen Song, Sijian Ren, Xinyuan Hu, Yu Li, Jie Feng, Chen Gao, and Yong Li. 2025. Towards large reasoning models: A survey of reinforced reasoning with large language models. Preprint, arXiv:2501.09686.
G M Shahariar, Erfan Shayegani, Ali Nazari, and Nael Abu-Ghazaleh. 2026. Modeling hierarchical thinking in large reasoning models. Preprint, arXiv:2510.22437.
Haoran Xue, Gias Uddin, and Song Wang. 2025. An empirical study of reasoning steps in thinking code llms. Preprint, arXiv:2511.05874.
Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, Akshay Nathan, Alan Luo, Alec Helyar, Aleksander Madry, Aleksandr Efremov, Aleksandra Spyra, Alex BakerWhitcomb, Alex Beutel, Alex Karpenko, and 465 others. 2025. Openai gpt-5 system card. Preprint, arXiv:2601.03267.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
Linxin Song, Xuwei Ding, Jieyu Zhang, Taiwei Shi, Ryotaro Shimizu, Rahul Gupta, Yang Liu, Jian Kang, and Jieyu Zhao. 2025. Discovering knowledge deficiencies of language models on massive knowledge base. Preprint, arXiv:2503.23361.
Runtao Zhou, Giang Nguyen, Nikita Kharya, Anh Totti Nguyen, and Chirag Agarwal. 2026. Improving human verification of llm reasoning through interactive explanation interfaces. Preprint, arXiv:2510.22922.
Peiyang Song, Pengrui Han, and Noah Goodman. 2026. Large language model reasoning failures. Preprint, arXiv:2602.06176. Zhaochen Su, Jun Zhang, Xiaoye Qu, Tong Zhu, Yanshu Li, Jiashuo Sun, Juntao Li, Min Zhang, and Yu Cheng. 2024. Conflictbank: A benchmark for evaluating the influence of knowledge conflicts in llm. Preprint, arXiv:2408.12076.
8
A
Evaluation Metrics
Knowledge Error, Formal Error, and Overthinking. The error also had to be identifiable from the trace content alone. Ambiguous cases were discarded. Third, for outputs with multiple injected errors, the resulting label composition had to be informative and non-redundant. Instances with imbalanced or near-identical errors were removed.
Node Type Accuracy (NTA) For each example, predicted nodes are matched to gold nodes by section span. Let [sg , eg ] and [sp , ep ] denote the section intervals of gold node g and predicted node p; their overlap is IoU(g, p) =
|[sg , eg ] ∩ [sp , ep ]| , |[sg , eg ] ∪ [sp , ep ]|
Trace Structuring Annotation. Structure labels were accepted under three conditions. First, node spans had to be non-overlapping with consistent section assignments. Each trace also had to contain exactly one final-answer node and show a plausible progression. Structural violations led to exclusion. Second, every node had to be assigned a defensible functional type drawn from the taxonomy, covering problem decomposition, reasoning step, intermediate answer, check, and final answer. Third, each edge had to reflect a discourse relation grounded in the text. Reasoning, check, and backtracking edges were retained only when textually supported. Unsupported instances were filtered out.
where |·| counts integers in the interval. Gold nodes are processed in order; each is greedily matched to the highest-IoU unmatched predicted node, accepting only if IoU(g, p) ≥ 0.5. For the resulting matched set M , NTA =
|{(g, p) ∈ M : type(g) = type(p)}| |M |
(defined as 0 when |M | = 0), averaged over all examples. Graph Edit Similarity (GES) For each example, gold and predicted exploration-level reasoning graphs G and Ĝ are constructed retaining only node and edge functional types. Let d = dGED (G, Ĝ) denote the graph edit distance with unit costs for insertion/deletion and substitution cost 1 for type mismatches. Let Z = |VG | + |VĜ | + |EG | + |EĜ |; if no distance is returned within a fixed timeout, d is set to Z. The per-example similarity is ( 1, Z = 0, GES = max(0, 1 − d/Z), Z > 0,
C
Figure 4 presents an annotation example from L ENS B ENCH, illustrating the data schema, original section texts, injected error labels, and the corresponding structured reasoning graph.
averaged over all examples.
B
LensBench Case Study
Human Verification Protocol
Each candidate instance in L ENS B ENCH was manually reviewed before inclusion. The review covered both annotation layers, including reasoning-error labels and trace-structure labels. Instances failing any criterion were discarded, yielding the final 130 verified examples. Reasoning Error Annotation. Labels were accepted only when three conditions held. First, each injected span had to remain coherent with its surrounding trace while preserving local notation and referents without surface cues that exposed the injection. Conflicting spans were excluded. Second, each error had to unambiguously satisfy exactly one category among Safety, Logical Error, 9
{
"id": 30,
"query": "7. If three points are randomly chosen on a circle, the probability that the triangle formed by these three points is an acute triangle is $\\qquad$ .", "sections": [ { "section_id": 1, "text": "Okay, so I need to find the probability that three randomly chosen points on a circle form an acute triangle. Hmm, let me recall some geometry here. First, a triangle is acute if all its angles are less than 90 degrees. Now, when dealing with circles, I remember that an angle inscribed in a circle is related to the arc it subtends. Specifically, the measure of the inscribed angle is half the measure of its subtended arc. So, if all three angles of the triangle are less than 90 degrees, then each of their subtended arcs must be less than 180 degrees because 90 degrees times 2 is 180. Wait, is that right?" }, { "section_id": 24, "text": "Therefore, the length of y for each x is (x + 1/2) - 1/2 = x. So, the area is the integral from x = 0 to x = 1/2 of x dx. Which is [ (1/2)x^2 ] from 0 to 1/2, which is (1/2)^2 = 1/4.\n\nBut this is the area in the region where x <= y. Since we fixed x <= y, the total area in the entire unit square would be twice this. So, by symmetry, the total area is 2 * 1/4 = 1/2. Therefore, the probability would be 1/2." }, { "section_id": 28, "text": "But wait, the probability that three random points lie on a semicircle is 3/8. Therefore, the probability that they don't lie on a semicircle is 1 - 3/8 = 5/8. Therefore, the probability that all arcs are less than 1/2 is 5/8. Therefore, the probability of forming an acute triangle is 5/8.\n\nBut I need to verify this. Wait, let me recall that the probability that n points on a circle all lie on some semicircle is n / 2^n. For n=3, that would be 3/8, which matches this result. Therefore, the probability that they don't lie on any semicircle is 1 - 3/8 = 5/8, which would mean that all arcs are less than 1/2. Hence, the probability of forming an acute triangle is 5/8." }, { "section_id": 32, "text": "Therefore, the probability that three randomly chosen points on a circle form an acute triangle is 1/4.\n\n**Final Answer**\n\\boxed{\\dfrac{1}{4}}" } ], "labels": { "errors": [ { "type": "Formal Error", "section_start": 24, "section_end": 24, "section_numbers": [24], "description": "Introduces an arithmetic/integration mistake: the integral of x from 0 to 1/2 is incorrectly evaluated as 1/4 instead of 1/8, then used consistently in the symmetry argument." }, { "type": "Knowledge Error", "section_start": 28, "section_end": 28, "section_numbers": [28], "description": "Misstates the known formula for the probability that n random points lie on a semicircle, using an incorrect denominator and deriving the wrong numerical result." } ],
}
}
"structure": { "layer1": { "nodes": [ {"id": "node1", "type": "problem_decomposition", "section_start": 1, "section_end": 5}, {"id": "node2", "type": "reasoning_step", "section_start": 6, "section_end": 23}, {"id": "node3", "type": "intermediate_answer", "section_start": 24, "section_end": 24}, {"id": "node4", "type": "check", "section_start": 25, "section_end": 25}, {"id": "node5", "type": "reasoning_step", "section_start": 26, "section_end": 28}, {"id": "node6", "type": "check", "section_start": 29, "section_end": 30}, {"id": "node7", "type": "check", "section_start": 31, "section_end": 31}, {"id": "node8", "type": "final_answer", "section_start": 32, "section_end": 32} ], "edges": [ {"from": "node1", "to": "node2", "type": "reasoning"}, {"from": "node2", "to": "node3", "type": "reasoning"}, {"from": "node4", "to": "node3", "type": "check"}, {"from": "node1", "to": "node5", "type": "reasoning"}, {"from": "node3", "to": "node5", "type": "backtracking"}, {"from": "node6", "to": "node5", "type": "check"}, {"from": "node5", "to": "node6", "type": "reasoning"}, {"from": "node6", "to": "node7", "type": "reasoning"}, {"from": "node7", "to": "node2", "type": "check"}, {"from": "node7", "to": "node8", "type": "reasoning"} ] } }
Figure 4: Complete LensBench case annotation for ID 30, including the data schema, original section texts with ASCII-only normalization, detected errors, and structured reasoning graph.
10