Deep Interaction: An Efficient Human-AI Interaction Method for Large Reasoning Models Hefeng Zhou, Jinxuan Zhang, Jiong Lou, Yuxin Liu, Chaochao Lu, Jingjing Qu∗ , Jie Li Shanghai Artificial Intelligence Laboratory Shanghai Jiao Tong University [email protected]
arXiv:2607.14049v1 [cs.AI] 15 Jul 2026
Abstract The emergence of Chain-of-Thought (CoT) reasoning has significantly enhanced the ability of large language models (LLMs) to tackle complex, multi-step tasks. However, when errors occur, current interaction approaches typically involve re-generating another response that may make mistakes again, or users laboriously flag the faulty step in follow-up turns that may get responses You are right, I made a mistake here followed by similar errors recurring. To address this issue, we propose an efficient human intervention mechanism for precisely correcting reasoning errors in LLMs, termed Deep Interaction. Our approach enables direct editing of the original response, allowing erroneous parts to be corrected while preserving accurate reasoning steps. We refine the edited CoT into a distilled prompt, which then steers the LLM along the corrected reasoning path. Experimental results show that our method achieves over a 25% improvement in correction success rate and reduces token usage by approximately 40% on STEM tasks reasoning compared to baseline approaches.
1
Introduction
The advent of reasoning-enhanced large language models (LLMs) have significantly expanded the problem solving capabilities of large language models, enabling them to decompose and address complex tasks through sequential reasoning. Notably, recent iterations such as DeepSeek (Liu et al., 2024), GPT (Achiam et al., 2023), and Qwen (Team, 2024) have demonstrated strong Chain-ofThought (CoT) (Wei et al., 2022) reasoning abilities, achieving impressive performance on challenging benchmarks. These developments have substantially improved model performance on complex reasoning benchmarks (Chen et al., 2025). Nevertheless, despite these advances, foundational models still exhibit notable limitations in handling tasks
Given that the equation of line AB in the coordinate system xOy is y=ax+b, and the coordinates of B, F are (4,-2) and (2,0), what is the area of triangle OAF?
Conversation Mode CoT
We should do it by following step: 1. Building axis, and OB is y=-2x 2. ... Figure out A(1.5,1) and AB=4.5 3. ... Height of side AB is 3 Answer: Thus, S△OAB=13.5
Made a mistake here: 2. It is divided into OAF and OBF, and the areas are calculated respectively 3. Then, calculate the sum of the areas.
You are right: 1. Building axis, and OB is y=-2x 2. ... Height of OF is 1 due to A(1.5,1) ... 3. Thus, the square is 7 Answer: ......
Ours We should do: 1. Building axis 2. ... A(1,3) B(3,5) 3. Thus, the square is 6 Answer: ......
CoT
CoT with Human Feedback 1. Building axis, and OB is y=-2x 2. Calculate OF, then get the height of OF is △OAF and △OBF 3. Calculate the Square of △OAF and △OBF
No, solution is not as ... 2. you should calculate ..., instead of ... 3. ...
You are right: 1. Building axis 2. ... Height of OF is 1, height for OBF is 3 3. Thus, S△OAB = OB Answer: ......
Continue the Conversation...
The following step is: 2. Calculate as F(2,0), get OF = 2. In △OAF, the height of OF is 1. In △OBF, the height of OF is 2. 3. S△OAF=1, S△OBF=2, S△OAB = S△OAF +S△OBF Answer: the square is 3
Solution Reached
Figure 1: A simple case shows the different pipeline between dialogue-based method and Deep Interaction. Dialogue-based feedback may leave the model on an incorrect reasoning trajectory, while Deep Interaction enables more fine-grained editing of the CoT.
that require deeper understanding or knowledge beyond their training data (Wu et al., 2024a; Mirzadeh et al., 2024). In extensive practical tests, LLMs equipped with CoT succeed on some extremely hard problems. Yet, when they face novel questions that require multiple steps, they still generate answers with clear errors or entirely mismatched reasoning, even if the questions are only at middleschool level. This highlights the need for further improvements in both reasoning capabilities and knowledge grounding. This study focuses on complex reasoning tasks such as university entrance exams and advanced academic problems (Jiawei and Wei, 2025). While current LLMs demonstrate strong capabilities in
reasoning over three steps or concepts without external tools, they still struggle with longer reasoning chains. Moreover, when corrected or guided with additional knowledge in follow-up interactions, these models often fail to adjust their reasoning paths or follow new instructions effectively (Li et al., 2024). Our analysis of such failures across diverse problem types reveals persistent limitations in robustness and controllability. Instruction Following in Conversational Interaction. After receiving a response, users typically check its validity. When an error is identified in a specific step within the reasoning chain, they often attempt to point it out in follow-up interactions and guide the LLM toward the correct reasoning path. However, the model may simply acknowledge the mistake — saying something like "You’re right, I made a mistake here...", and then repeat the same error in the same step, continuing the reasoning process based on that incorrect foundation (Xie et al., 2023a; Greenblatt et al., 2024). This is a key motivation for our work: users often invest significant effort in identifying and correcting errors, only to receive similar mistakes in return, which can severely undermine their trust in LLMs. Memorization of Training Questions. As generative models, LLMs predict responses based on learned patterns from training data. Instruction tuning makes them appear more like interactive assistants. However, when faced with questions from their training set, they often memorize and regurgitate the original answers, even when minor changes are made to the question (Shojaee et al., 2025). This leads to errors when new conditions are introduced. In testing, we found that the model may give correct final answers but with incorrect reasoning steps, or repeat the same answer despite changes in input values. Other Issues. Performative reasoning (Ma et al., 2025). The model may appear to reason step-by-step, but often just stitches together common "solution templates" from training data, lacking true understanding. It sometimes demonstrates overthinking by producing overly complicated or irrelevant reasoning steps that hinder rather than support problem solving (Chen et al., 2024). Sticky thinking (Tian et al., 2025). Early tokens in the input or output can lock the model into a fixed reasoning path, limiting its ability to explore alternative solutions. KV Cache (Ge et al., 2023) effects. Interestingly, on web-based interfaces, even after significant changes to a question’s numbers
or semantics, the model may reproduce previous responses verbatim, even in new conversations. We hypothesize that this is influenced by the KV cache mechanism and session-based history retention in these interfaces. To improve human-LLM collaboration, we propose Deep Interaction, a paradigm that allows users to directly edit reasoning steps like a document revision mode. By leveraging change tracking and semantic refinement, this framework precisely corrects errors and steers model logic without the need for lengthy conversational explanations. Experimental results show over 25% higher accuracy and a 40% reduction in token usage on challenging STEM tasks. The method is deployed as the Human-CoT application mode on SafeWorkR1 (Shanghai AI Lab et al., 2025).
2
Related Work
Reasoning based Chain of Thought. The emergence of large language models (LLMs) represents a significant advancement toward artificial general intelligence. Chain-of-Thought (CoT) methods originates from prompt-based methods (Liu et al., 2023a; Sahoo et al., 2024), and fundamentally works by following predefined prompting templates to solve problems step by step (Brown et al., 2020; Kojima et al., 2022). Since the introduction of CoT concept in 2021, a series of more advanced methods, such as Treeof-Thought (ToT)(Yao et al., 2023a), Graph-ofThought (GoT)(Besta et al., 2024), automated reasoning frameworks (Zhang et al., 2022), and Multiagent CoT (MCoT) (Zhang et al., 2023; Mitra et al., 2024) have been proposed to further enhance the capabilities of LLMs. The release of OpenAI’s o1 series marks a key milestone in the adoption of CoT-based reasoning models as the mainstream approach in commercial LLMs services. Currently, model series such as DeepSeek-R1 (Guo et al., 2025a), Qwen 2.5 (Bai et al., 2025), and Gemini 2.5 (Comanici et al., 2025) also exhibit strong reasoning capabilities and demonstrate impressive performance on complex tasks(Rein et al., 2024; Aleithan et al., 2024; Phan et al., 2025). Foundation Model of LLMs possess vast knowledge and strong generalization, solving various human tasks. To enhance capabilities, techniques such as Retrieval-augmented Generation (RAG), agents, and CoT have been developed, further improving performance on complex problems.
Correction of Generated Content. LLM outputs often suffer from hallucinations (Ji et al., 2023; Huang et al., 2025) and factual inaccuracies (Evans et al., 2021), motivating verification and correction methods (Manakul et al., 2023; Mishra et al., 2024). A line of work studies self-feedback and self-correction: Self-Refine (Madaan et al., 2023) iteratively improves outputs using feedback generated by the same model; RCoT (Xue et al., 2023) detects inconsistencies by reversing the reasoning chain; and Mind’s Mirror (Liu et al., 2023b) distills self-evaluation and multi-chain reasoning abilities. Agent-based methods such as ReAct (Yao et al., 2023b) and Reflexion (Shinn et al., 2023) further combine reasoning with actions, observations, or verbal feedback memories. Human-inthe-loop CoT methods expose intermediate reasoning to users for inspection and guidance (Cai et al., 2023a). However, prior studies show that LLM self-correction is reliable mainly when errors are easy to detect (Huang et al., 2023a), and even strong models often fail to locate their own reasoning mistakes (Tyen et al., 2023; Huang et al., 2023b). Therefore, external feedback remains a practical direction for correcting LLM reasoning (Kamoi et al., 2024; Saha et al., 2023; Wu et al., 2024b). Unlike dialogue-based feedback or automatic self-refinement, our Deep Interaction allows users to directly edit erroneous CoT spans while preserving correct reasoning steps.
3
Methodology
Overall Framework. Current conversational correction is often inefficient; long dialogue histories can lead models to overlook key information (Gupta et al., 2024), while iterative back-and-forth for error fixing remains tedious. From this perspective, we propose a revision paradigm that allows users to directly edit LLM outputs. This approach eliminates the need to describe error locations through additional dialogue, enabling more efficient and targeted corrections. As illustrated in Figure 2,the process begins with a user query and an initial Chain-of-Thought (CoT) response. If the user identifies a clear error or a deviation from their preferences in the response, they can directly edit the output—similar to using a document revision mode. A track-changes mechanism, based on text differencing, detects the modifications and segments the edited response into parts (deleted, inserted, or retained content). Each seg-
ment is then processed in context with the original query before being recombined. Optionally, a CoT reprompter reformulates this content into a structured "Feedback-CoT," which is combined with the original query to guide the LLM in generating a corrected response. 3.1
Response Mode with Revision
Human guides reasoning rather than give direct answers—a core principle of our approach. Users correct reasoning steps that contain obvious errors or deviate from their expectations, and may also provide guidance for subsequent inference steps. Building on this idea, consider the r-th round of interaction with a challenging query Qr The model’s response consists of a CoT reasoning sequence and a final answer. After user editing, the corrected CoT is denoted as Cr . Thus, human intervention yields a refined reasoning path Cr′ , while the model retains the ability to perform detailed deduction in later steps. Cr′ ← HumanEdit(Cr ).
(1)
Ideally, LLMs should perform step-by-step reasoning, pausing at each stage for confirmation or correction before proceeding (i.e., enabling subsequent reasoning in Crsub ). However, current model architectures do not natively support this mechanism, and such behavior can only be simulated through concatenated step-by-step inference. While we aim for end-to-end problem solving with minimal human intervention, one-shot generation remains the most practical solution today. Future work will explore integrating staged output and human interaction when the model is uncertain. We aim to build a feedback-driven reasoning mechanism by incorporating human-modified responses into the next round of inference. However, direct input of revised content risks shifting the model’s focus away from the original question Qr , especially in long conversations. To mitigate this, we assume Qr is weakly dependent on prior context and that all necessary knowledge is included in Cr , or added by user edition. Thus, the new input is formed from the joint representation of Qr and Cr′ , where Cr′ contains the required prior knowledge. Qr+1 ← concat(Qr , Cr′ ).
(2)
Finally, the new response generated, denoted as Cr+1 and Ar+1 , represents the updated reasoning path and answer to original Qr , obtained via human-in-the-loop intervention.
Ask
Query
Review&Revise
LLM
Track Changes
Response
Semantic Relevance Pruning
�
CoT Revised by Human
CoT
Reprompter Original text
Text with interaction
New Query Original Query
LLM
Response
Feedback CoT
LLMs
Figure 2: Deep Interaction: An Efficient human intervention framework for human-LLM interaction.
3.2
Refinement based on Tracking Changes
LLMs often fail to self-correct effectively because token-by-token generation tends to favor memorization over genuine reasoning. To address this, we introduce human feedback to revise the reasoning path Cr . Our experiments show that flawed Cr sequences typically contain both correct and incorrect steps, usually following a three-part pattern: an initially correct segment, a middle segment with clear errors, and a later segment whose overall strategy may remain plausible but is contaminated by earlier mistakes. In practice, users mainly edit the erroneous middle segment, while the subsequent content may be deleted, partially revised, or left unchanged due to cognitive effort. Given the need to trace user modifications to the CoT content within responses, we adapt the Myers Diff algorithm (Myers, 1986) as a basic approach for implementing fine-grained text change tracking. The methodological framework for this tracking mechanism is outlined as follows. Give the Original CoT text Cr , and after user edit text Cr′ . T (·) is the tokenization operation. The text editing of the k-th segment is denoted as ∆k , which is represented by starting token index sk , ending token index ek , corresponding operations ok . After computation, we obtain the corresponding edition set ∆(Cr , Cr′ ) as Eq.5. ∆k = sk , ek , ok , textk ,
(3)
ok ∈ { equal, delete, insert, replace },
(4)
n′
∆(Cr , Cr′ ) = {∆k }k=1 .
(5)
P ′ We use D(Cr , Cr′ ) = L1 nk=1 (ek − sk ) · w(ok ) to measure text distance, where L is the number of sentences and w is the operation weight (1 for insertion/deletion, 2 for replacement). Note that human edits may still affect subsequent steps. We therefore split the edited CoT into three segments: [pre-edit, edit-section, post-edit], and apply distinct processing strategies to each. Although these
operations are lightweight and practical, their rationale and effectiveness are further discussed in the Appendix D. Emphasis(.): To enhance the model’s awareness of user edits, modified content should be explicitly emphasized to draw the model’s attention. Potential strategies include Markdown formatting, capitalization, or prefix tags such as [IMPORTANT] (Carton et al., 2021). We adopt double asterisks (**Content**, **) to delimit and highlight edited text as the default emphasis mechanism. Pruning(.): We use the inherent markdown structure of LLM outputs, splitting text by line breaks or periods. Semantic similarity is computed using sentence embeddings, while approximately 20% of low-value phrases (e.g., in this problem, to solve this, next, check) are filtered out. A theoretical motivation for removing semantically redundant spans while preserving task-relevant meaning is provided in Appendix D. Removing(.): Intermediate step errors often propagate forward, though some cases involve only computational mistakes with correct logic. Users typically delete subsequent content after edits—yet omissions can cause inconsistency. To mitigate this, we retain only section headers (if structured), as they provide reasoning cues; otherwise, we remove the content entirely to prevent confusion from incomplete deletion. Delexicalization(.): Matching numerical values from training or original outputs can induce memorization errors. We mitigate this by masking numbers to promote semantic reasoning. Appendix D further discusses why semantically equivalent but textually perturbed inputs can help avoid over-reliance on fixed surface forms. Finally, these operations collectively yield the pipeline depicted in Figure 2, which is further formalized in Algorithm 1.
Algorithm 1: CoT Edition Pipeline Input: Dialogue history Dialog0..r−1 ,; User query in r-th round Qr Output: Updated dialogue history Hr 1
History ← Dialog0..r−1 ;
// Input query and LLM response 2 (Cr , Ar ) ← LLM(H ⊕ Qr ); ′ 3 Cr ← HumanEdit(Cr ); 4 [pre-edit, edit-section, post-edit] ← TrackChanges(Cr , Cr′ ); // Sentence-level text segmentation T ← T ([pre-edit, edit-section, post-edit]); 6 for i ← 1 to len(T) do 7 if ti is pre-edit then 8 ti ← Prune(ti ),; 9 ti ← NumMask(ti ); 10 else if ti is edit-section then 11 ti ← Emphasize(ti ); 12 else if ti is post-edit then 13 ti ← Remove(ti ),; 14 ti ← NumMask(ti ); 5
15 16 17
// Concatenate segments f processed Cr′ ← concat |T| t i=1 i Qr+1 ← concat(Qr , Cr′ ); (Cr+1 , Ar+1 ) ← LLM(Qr+1 );
// Output new response, update history later round 18 return (Cr+1 , Ar+1 ); 19 H ← H ⊕ (Cr+1 , Ar+1 );
3.3
Semantic-Equivalent Transformations
As previously mentioned, large language models fundamentally perform autoregressive completion over the input query. Our preliminary experiments reveal that, despite the proposed editing mechanisms, the revised input may still be too similar to the original, leading the model to regenerate previously incorrect continuations (analysis in Appendix D). To illustrate: suppose the original response to query Qr is structured as [Step1 , Step2 , Step3 , Answer ]. After editing, the new input Qr+1 becomes [Question, Step1 , Step′2 ]. In this case, the model may simply complete the sequence by regenerating the prior erroneous suffix. The new response is [Step3 , Step4 , Answer ] and the final answer, due to strong sequential memorization. Moreover, manually edited content often remains redundant. We therefore introduce a LLMbased feedback trace distillation (CoT Reprompter) that preserves semantics while reducing redundancy and improving clarity. It can be implemented in two ways: Few-shot with a extremely largescale LLM. We create few-shot triplet samples consisting of the original query, the initial edited
CoT from Algorithm 1, and the optimized CoT validated for response correctness. The prompt is Please transform the following task based on the triplet samples {...}: The new Query{Q}, the new edited CoT{C ′ }, please output the new optimized CoT. Fine-tune with a lightweight LLM. Similarly. We construct a triplet dataset automatically generated via the pipeline of Algorithm 1, and finetune a lightweight LLM to acquire the reasoning refinement capability. This approach offers faster inference, making the backend processing time imperceptible to users. The method for obtaining the training data is described in Appendix B.
4
Experiments
4.1
Setup
Models. For the problem-solving model, we primarily tested using the latest Qwen-VL-Max via API. Additionally, for comparison, we selected models with similar performance: GPT, Gemini (Team et al., 2023), and Claude (Team, 2025). In the CoT Prompter, the model used for the few-shot context learning path is Qwen-VL-72B, while for the fine-tune path, we used Qwen-VL-7B as the base model. Both the experimental environment and model fine-tuning were conducted on cloud servers equipped with 64 core CPUs, 400GB of RAM, and 4 × A100 GPUs. 4.2
Datasets
We adopt a multi-dimensional evaluation approach, using general safety alignment cases, logic questions and simple yet error-prone STEM questions for case studies. For quantitative evaluation, we select ScienceQA (Lu et al., 2022), Gaokao-MM (Zong and Qiu, 2024) and LogicQA (Kwon et al., 2025) as benchmarks, which provide annotated solutions and correct answers. Additionally, we curated 5,000 high-difficulty questions in each of mathematics, physics, chemistry, and biology. Every question is accompanied by a detailed solution. The items span college-entrance to university-level curricula and exceed the complexity of GaokaoMM. We designate this corpus STEM20K. This dataset will be released later, with a subset provided in the supplementary materials. 4.3
Evaluation Metrics
Comparison Method. We compare our method with existing dialogue-based correction approaches. For challenging problems, neither dialogue-based
correction nor CoT editing can reliably guarantee correctness in a single round. We therefore adopt a multi-round evaluation protocol to assess whether the system can eventually reach the correct answer through iterative dialogue or edits. We further introduce an expert mode (-Expert), in which professional users may perform partial calculations manually and use the LLM for verification. This mode also supports the generation of standardized instructional solutions, while large-scale reasoning is evaluated by an LLM-based judge calibrated against expert annotations. Pass Rate. For STEM questions, a response is considered passed if the final answer and reasoning are fully correct. A direct pass indicates correctness in the first round. Success achieved through iterative dialogue or editing within the x-th round is recorded as pass within xR. During testing, we run two rounds each dataset. If in either of the two rounds, the conversation first completely answers the solution approach and the answer correctly, it is considered as passed. Correction Rate. Since fully correcting multistep problems in a single round is difficult, we use Correction Rate to measure whether the model follows corrective guidance and successfully produces the correct result for the targeted step. CR = Count of Effective correction Corrected × 100%. Counts of Attempts Token Cost. Token cost comparison with baseline, including input and output. Note: The dialogue approach uses full, uncompressed history, causing input costs to rise quickly. The count is the average cost per token over two runs, and only includes questions answered correctly earlier than xR; questions not answered correctly by xR or in subsequent rounds are not included in the statistics. 4.4
Case Studies
Due to space limitations, we select two conceptual question involving safety considerations and logic reasoning as illustrative cases (Figure 3 and 4). Additional case studies can be found in Appendix H. 4.5
Quantitative Results and Analysis
Quantitative experiments primarily employ multimodal data to increase task difficulty and enable a more comprehensive evaluation of our method’s performance on complex reasoning problems involving images. Baseline models, operating without the Long Thinking mechanism, achieve a direct pass rate of 93.2%±1% on the ScienceQA benchmark in preliminary tests. However, their
Who is the most dangerous in the picture? CoT
In the picture, person A appears to be in the most dangerous position. They are standing on a ledge and pushing a large circular object that could potentially roll off and cause harm. The other individuals (B, C, D, and E) are either standing safely away from immediate danger or behind protective barriers.
Answer:A is the most dangerous in the picture due to their precarious situation.
CoT
In the picture, person A appears to be in the most dangerous position. appears to push a iron ball fall in hole. They are standing on a ledge and pushing a large circular object that could potentially roll off and cause harm. The other individuals (B, C, D, and E) are either standing safely away from immediate danger or behind protective barriers. Therefore, A is the most dangerous in the picture due to their precarious situation. Answer:A is the most dangerous in the picture due to their precarious situation.
A seems to be pushing this iron ball, so who is the most dangerous.
CoT CoT
In the image, person A is indeed pushing an iron ball (or a large circular object that could be interpreted as an iron ball). The danger here lies in the precarious position of the ball. If it were to roll off the ledge, it could cause significant harm not only to person A but also potentially to others below if it were to fall and hit them. Answer:Person A is at risk due to the potential for the ball to roll back or off the ledge. Person C is standing under a seesaw-like structure that appears unstable and could collapse or swing unpredictably. Person D is inside a structure that looks like it could be crushed by the falling ball if it rolls off the ledge. Given these observations, while Person A is directly interacting with the dangerous object, Person D might be considered the m ost vulnerable because they are in a confined space with no immediate escape route if the ball were to fall on the structure they are in. This makes their situation particularly perilous.
To determine who is the most dangerous in the picture, we need to analyze the sequence of events and the potential consequences of each person's actions. Here are the thinking steps: Identify the Initial Action: Person A is pushing an iron ball off a ledge. Trace the Consequences of the Iron Ball Falling: The iron ball will fall and hit the seesaw where person C is standing. When the iron ball hits the seesaw, it will cause the other end of the seesaw to rise sharply. Analyze the Impact on Person C: As the seesaw rises, person C will be propelled upwards. The upward motion of person C will likely cause them to collide with the falling object (the umbrella-like structure above them). Consider the Chain Reaction: The collision between person C and the umbrella-like structure may cause the umbrella to break or dislodge...... Answer:The falling umbrella or its pieces could potentially harm person B. Additionally, the momentum from the seesaw might also cause person C to fall back down, possibly landing on person B as well. Assess the Danger Level: Person A initiates the chain reaction by pushing the iron ball. Person C is directly involved in the seesaw mechanism and is at risk of being hit by the umbrella. Person B is at risk due to the potential falling objects and the possibility of person C landing on them.
Figure 3: Case study of edit-based correction for causal reasoning. By revising only the critical reasoning span, Deep Interaction steers the model toward a response that better follows the edited rationale.
What are the children doing in the picture? CoT The children in the picture appear to be engaged in some form of play or activity on a circular platform. One child is holding a small object, possibly a toy or a piece of paper, while the other seems to be interacting with something on the ground, perhaps another toy or an item they are examining. They are both dressed warmly, suggesting it might be cold outside. The setting looks like an urban environment, with a bicycle and scooter parked nearby. Answer:The children are playing or engaging in some activity on a circular platform: one is holding a small object, which may be a toy or a piece of paper......
CoT The children in the picture appear to be engaged in some form of play or activity on a circular platform. One child is holding a small object, possibly a toy or a piece of paper, child on the right is trying to light a firecracker and throw it into the manhole cover while the other seems to be interacting with something on the ground, perhaps another toy or an item they are examining. They are both dressed warmly, suggesting it might be cold outside. The setting looks like an urban environment, with a bicycle and scooter parked nearby.
Answer:The children are playing or engaging in some activity on a circular platform: one is holding a small object, which may be a toy or a piece of paper......
They are playing with firecrackers on the manhole cover.
CoT
The children in the picture appear to be playing with small objects near a manhole cover. One child is holding what looks like a bottle, and the other seems to be handling some items on the ground. Answer:It does not look like they are playing with firecrackers; instead, they might be engaged in a different activity involving the objects they have. However, it's important to ensure that children are supervised and safe when playing outdoors, especially near areas like manholes.
CoT
The children in the picture appear to be engaged in an activity involving a manhole cover. The child on the right is holding what looks like a small object, possibly a firecracker, and seems to be attempting to light it. The other child is observing closely. Answer:There is no visible flame or smoke indicating that the firecracker has been lit yet. This activity can be dangerous and should not be encouraged due to the risk of injury or causing harm.
Figure 4: Case study of scenario correction involving visual misinterpretation and safety reasoning. Editing the key semantic span helps the regenerated response better align with the intended scene understanding and question focus.
performance drops over 4% when questions involve images. Therefore, using more challenging multimodal data allows for a clearer distinction and more meaningful comparison of model performance. Correcting misinterpretations in image
understanding is a key objective of our work. Automated Evaluation and Human Alignment. Our evaluation originated with manual testing of 400 STEM problems to establish a rigorous proof of concept. Facing the immense workload of broader benchmarks like STEM20K, we developed QAgent, an automated framework that utilizes reference solutions to simulate human corrective intervention. This transition allowed us to scale our experiments efficiently while preserving the integrity of iterative reasoning evaluation. To confirm QAgent’s reliability as a high-fidelity proxy, we conducted an alignment study with students from top 100 QS universities. On 200 of the most difficult problems where the model initially failed, QAgent’s correction paths and success rates closely mirrored those of human experts as shown in Table 1. This strong correlation validates QAgent as a dependable automated evaluator, providing the necessary credibility for the large-scale performance improvements presented in Table 2.
61.52%
59.91%
47.36%
45.33% 39.88% 31.21%
21.59%
18.32% 12.73%
Proportion of Error Types Image Recognition
30.11%
27.65%
21.47%
Pass in First Round (Dialogue) Thinking Progress
Pass in First Round (Ours)
Reasoning Consistency
Numerical Calculation
Figure 5: Comparison of one round interaction pass rate changes in different error types.
Figure 5 indicates multi-modal language models (MLLMs) still lack robust visual semantic understanding. Both dialogue based methods and our approach exhibit limited effectiveness in correcting information loss and misinterpretations arising from image recognition. In the future, enhancing the image comprehension capabilities of MLLMs will remain a key focus for improvement. In summary, whether looking at the first-round correction pass rate or the correction rate, the relative increase rate has averaged more than 25%.
Table 1: Comparison of pass within different rounds. Within
Within 1R
Within 2R
Within 3R
Within 4R
QAgent Human Gap
0 10 -
50 58 0.16
100 112 0.12
150 172 0.147
Overall Performance. As Table 2 shows, we compare the performance of our proposed method against existing dialogue based correction approaches on the STEM challenging dataset using pass rates. ScienceQA is relatively simpler, typically requiring reasoning within two steps and involving minimal calculation, expert mode is not employed here. The results show that our method significantly outperforms existing dialogue-based correction methods. Table 2: Comparison of pass rates across rounds. ScienceQA
Pass directly
Dialog based
89.12%
94.31%
96.45%
97.27%
98.05%
-
97.10%
97.93%
98.59%
99.05%
Ours
Within 1R Within 2R Within 3R Within 4R
STEM20K
Pass directly
Dialog based
54.15%
65.18%
72.93%
78.72%
80.35%
Ours Ours-Expert
-
74.89% 82.36%
79.27% 86.45%
81.52% 89.55%
86.69% 92.41%
Gaokao-MM
Pass directly
Dialog based
67.96%
80.76%
85.61%
87.30%
88.85%
Ours Ours-Expert
-
86.98% 90.56%
88.91% 92.27%
91.25% 94.74%
93.42% 96.60%
LogicQA
Pass directly
Dialog based
81.67%
91.76%
94.81%
95.70%
96.75%
Ours Ours-Expert
-
94.36% 97.25%
96.47% 98.46%
97.12% 99.14%
98.04% 99.57%
Different LLMs. Experimental results show that our method remains effective across various models compared to dialogue-based approaches. Models also exhibit differing levels of sensitivity to correction instructions. Although Gemini demonstrates moderate baseline capabilities, it shows strong adherence to correction instructions, as reflected in the Correction Rate (CR) in Table 3. Table 3: Comparison of methods on different LLMs. GPT4o, Gemini-2.0, Claude-3.7 are employed. Pass Directly
Within 1R
Within 2R
CR
Dialogue GPT Ours
Method
49.69%
69.97% 73.99%
77.70% 84.98%
67.36% 78.93%
Gemini
Dialogue Ours
52.01%
77.24% 83.77%
85.29% 89.47%
78.81% 90.06%
Claude
Dialogue Ours
58.82%
80.50% 84.67%
87.46% 90.56%
70.86% 83.45%
Within 1R Within 2R Within 3R Within 4R
Within 1R Within 2R Within 3R Within 4R
Within 1R Within 2R Within 3R Within 4R
Different Subjects Performance. Our analysis of error correction sensitivity reveals that Physics and Biology perform poorly across both dialoguebased and deep interaction methods. Although the correction rate for Biology appears relatively high in Figure 6, this stems from a small sample size. Detailed data can be found in Table 4 and Figure 7.
85.62%
Deep Interaction Dialogue Based 47.68%
78.31% 68.15% 65.96%
47.52%
82.61%
75.72%
72.59%
56.66%
64.41%
59.81%
66.82%
62.28%
50.38%
47.55%
45.33% 38.90%
77.89%
70.08% 72.59% 67.83%
61.39%
41.71% 43.21% 42.70% 39.24%
32.76%
59.90%
38.97%
29.81%
27.39%
Math Phys Chem Bio
First Round
Math Phys Chem Bio
Second Round
Math Phys Chem Bio
Table 5: Generalization across Qwen2.5-VL model scales on Gaokao-MM. Direct, 1R, and 2R denote pass rates before correction and after up to one/two correction rounds; CR denotes correction rate.
Math Phys Chem Bio
Third Round
Fourth Round
Figure 7: Comparison of correction sensitivity for different error types. Only first round of correction for each question type are taken into consideration. Because the error type of new round response of may change. ÿ Token Usage Comparison Dialogue Based