CopT: Contrastive On-Policy Thinking with Continuous Spaces for General and Agentic Reasoning

CopT: Contrastive On-Policy Thinking with Continuous Spaces for General and Agentic Reasoning

(a) CoT

Thinking

Answer

Next Task/End

(b)

Output

×

>>>

On-policy Thinking:

>>>

Question

Discrete Inputs >>>

√

Step

× CopT

>>>

arXiv:2605.20075v1 [cs.CL] 19 May 2026

Dachuan Shi1 , Hanlin Zhu2 , Xiangchi Yuan1 , Wanjia Zhao3 , Kejing Xia1 , Wen Xiao4 , Wenke Lee1 1 Georgia Tech 2 UC Berkeley 3 Stanford University 4 Microsoft

(c) #Tokens CoT CopT* CopT

Cont. Inputs

Acc

Figure 1: (a) Conceptual comparison between C OT thinking and C OP T on-policy thinking. (b) C OP T contrasts the output distributions under discrete and continuous inputs. (c) C OP T improves peak accuracy, marked by ∗ , across mathematics, coding, and agentic reasoning tasks and nearly halves token usage at matched accuracy.

Abstract Chain-of-thought (C OT) is a standard approach for eliciting reasoning capabilities from large language models (LLMs). However, the common C OT paradigm treats thinking as a prerequisite for answering, which can delay access to plausible answers and incur unnecessary token costs even when the model is able to identify an answer before extended thinking, a behavior known as performative reasoning. In this paper, we introduce C OP T, a reformulated reasoning pipeline that reverses the usual order of thinking and answering. Instead of thinking before answering, C OP T first elicits a draft answer and then invokes subsequent on-policy thinking conditioned on its own draft answer for reflection and correction. To assess whether the draft answer should be trusted, C OP T recasts continuous embeddings as inferencetime contrastive verifiers. Specifically, it contrasts the model’s support for the same generated tokens under discrete-token inputs and continuous-embedding inputs, yielding a sequence-level reverse KL estimator for answer reliability. Our analysis shows that under certain assumptions, the expected estimate equals the mutual information between the unresolved latent state and the emitted answer token, explaining why it captures answer-relevant uncertainty rather than arbitrary uncertainty in the latent state. When the answer is deemed insufficiently reliable, C OP T performs further on-policy thinking, where a second KL estimator dynamically controls draft-answer visibility, preserving useful partial information while reducing the risk of being misled by unreliable content. Across mathematics, coding, and agentic reasoning tasks, C OP T improves peak accuracy by up to 23% and reduces token usage by up to 57% at comparable or higher accuracy, without any additional training. The code is available at https://github.com/sdc17/CopT.

Preprint.

1

Introduction

Reasoning has become one of the central capabilities of large language models (LLMs), enabling them to solve increasingly complex tasks in mathematics [Google DeepMind, 2024, Jaech et al., 2024, OpenAI, 2025, Team et al., 2025], coding [Cao et al., 2026, Hui et al., 2024, Zhu et al., 2024, Roziere et al., 2023], and agentic [Anthropic, 2025b,a, Qwen Team, 2026, Patil et al., 2024] settings. A common approach for eliciting reasoning behavior is chain-of-thought (C OT), where LLMs generate intermediate natural-language steps before producing the final answer [Wei et al., 2022, Yao et al., 2023, Goyal et al., 2024, Pfau et al., 2024, Qwen Team, 2024, 2025]. By making the thinking process explicit, C OT brings substantial improvements to complex tasks that demand advanced reasoning capabilities [Yang et al., 2025, Meta AI, 2025a,b, Guo et al., 2025, Agarwal et al., 2025, Abdin et al., 2025, Shi et al., 2025b, Abouelenin et al., 2025]. A key limitation of the predominant C OT paradigm is that it treats thinking as a prerequisite for answering. It works by first producing a thorough reasoning trace and only then arriving at the answer. However, recent work has revealed that, for many queries, LLMs exhibit performative reasoning [Boppana et al., 2026, Huang et al., 2026, Lindsey, 2026, Chen et al., 2025b], in which they insist on completing the reasoning process even when they have already internally identified a plausible answer. We propose C OP T, a reversed reasoning paradigm. Rather than thinking before answering, an LLM first drafts an answer and then performs thinking for reflection and correction afterward. This reformulated paradigm provides earlier access to answers and avoids unnecessary token consumption when the model is able to identify a plausible answer before thorough thinking. Reversing the usual order of thinking and answering raises two key challenges: when a draft answer should be trusted, and how it should be used during later thinking. We show that continuous embeddings, previously used for generation in latent C OT methods [Hao et al., 2024, Xu et al., 2025], can be recast as inference-time verifiers for this reversed reasoning setting. By contrasting the model’s support for the same generated tokens under discrete-token and continuous-embedding inputs, they provide measurable criteria for draft reliability estimation and controlled utilization. Latent C OT, where LLMs generate continuous embeddings instead of committing to discrete tokens during the thinking process [Hao et al., 2024, Shen et al., 2025, Zhu et al., 2025b, Xu et al., 2025, Tan et al., 2025], is a distinct line of recent work in parallel to explicit C OT. These approaches are motivated by the observation that latent C OT offers higher representational bandwidth per step [Zhu et al., 2025c, Yu et al., 2026]. Continuous embeddings can encode richer information by preserving uncertainty, whereas discrete tokens retain only the information carried by the sampled token at each step [Li et al., 2025, Chen et al., 2025a]. Instead of using continuous embeddings for generation during the thinking process, as in existing latent reasoning methods, C OP T keeps thinking explicit while recasting continuous embeddings as contrastive verifiers at inference time. This allows C OP T to retain the readability of explicit C OT while simultaneously leveraging uncertainty information, as in latent C OT. Meanwhile, it avoids issues that may arise when continuous embeddings are directly used for generation, such as unseen representations [Zhang et al., 2025], de-diversification [Wu et al., 2025], and drifting into noise [Shi et al., 2025a]. To address the first challenge of determining when a draft answer should be trusted, C OP T introduces a contrastive mechanism with continuous spaces to estimate the reliability of the draft answer. Specifically, it contrasts the model’s support for its own generated answer under two types of input representations: explicit inputs in discrete spaces and continuous embeddings constructed from next-token distributions and cached online along with explicit token generation. This contrast yields a sequence-level reverse KL estimator that indicates the reliability of the draft answer. If the draft answer appears sufficiently reliable, it will be accepted by the model directly. Otherwise, C OP T triggers a subsequent on-policy thinking process to either correct or support the answer. The second challenge of how to use the draft answer arises once on-policy thinking is triggered. A draft answer deemed insufficiently reliable may still contain useful partial information, but exposing it throughout the entire later thinking process risks misleading the model. To control the visibility of the draft answer during on-policy thinking across thinking steps, C OP T periodically calculates a second KL estimator within each thinking chunk using a similar contrastive mechanism with continuous 2

spaces. In this way, C OP T allows the model to use the draft answer when it appears helpful, while hiding it when the current thinking process becomes unstable. Beyond the empirical results, we further provide a latent-state interpretation of the proposed contrastive estimator. Under a local mixture-prefix view, the continuous prefix preserves uncertainty over an unresolved latent reasoning state S, while the emitted answer token is denoted by A. We show that under a mixture-linear assumption (see Section 4), the expected estimate equals the mutual information I(S; A), indicating that the estimator measures answer-relevant uncertainty rather than the entropy of the latent state itself. This explains why the score grows only when uncertainty preserved by continuous embeddings changes the model’s support for its own generated tokens, supporting its use for draft reliability estimation. Our contributions are summarized as follows: • We propose C OP T, a training-free reasoning pipeline that enables LLMs to start with a draft answer and invoke on-policy thinking conditioned on it when necessary, thereby allowing earlier access to answers and selective correction afterward. • We introduce a contrastive mechanism that measures the discrepancy between the model’s support for the same generated tokens under discrete and continuous inputs, which helps identify potential errors in draft answers and modulates their exposure during the thinking process. • We extensively validate the effectiveness of C OP T on mathematics, coding, and agentic reasoning tasks across multiple benchmarks, model architectures, and scales, demonstrating consistent gains over C OT baselines in both accuracy and token efficiency.

2

Related Work

Reasoning LLMs and explicit reasoning. Reasoning with explicit natural-language traces has become a standard way to improve the performance of LLMs on complex tasks [OpenAI, 2025, Anthropic, 2025b, Comanici et al., 2025]. Early work elicits such behavior through prompting [Wei et al., 2022, Wang et al., 2022, Yao et al., 2023]. More recent LLMs typically gain reasoning capabilities through reinforcement learning [Shao et al., 2024, Yu et al., 2025, Liu et al., 2025b] or multi-stage post-training that combines supervised fine-tuning with reinforcement learning [Liu et al., 2024, Shi et al., 2024, Yang et al., 2025, Ma et al., 2025, Yuan et al., 2025a,b]. Representative open-source examples include DeepSeek-R1 [Guo et al., 2025] and Qwen3 [Yang et al., 2025], which show that large-scale reinforcement learning and long-C OT post-training can elicit strong reasoning behaviors. Following works [Zeng et al., 2025a, Liu et al., 2025a, Yuan et al., 2026, Cao et al., 2026] further demonstrate the effectiveness of explicit reasoning across diverse mathematics, coding, and agentic tasks. Despite these advances, reasoning LLMs typically retain the standard thinking-before-answering order. In contrast, C OP T reverses this order by first eliciting a draft answer and invoking on-policy thinking conditioned on it when the answer appears insufficiently reliable. Latent reasoning with continuous embeddings. A parallel line of work explores latent reasoning in continuous spaces, where LLMs operate on continuous embeddings instead of committing to discrete tokens at every reasoning step [Hao et al., 2024, Su et al., 2025, Zhu et al., 2025b]. These methods are motivated by the observation that continuous representations can encode information from the full next-token distribution, while discrete decoding retains only the sampled token. Latent reasoning is mainly achieved by adapting LLMs into continuous spaces via modified pretraining [Zeng et al., 2025b, Tack et al., 2025] or fine-tuning [Shen et al., 2025, Xu et al., 2025, Tan et al., 2025, Wei et al., 2025, Zhu et al., 2025a, Xia et al., 2026] objectives. Recent training-free methods [Wu et al., 2025, Xu et al., 2026] instead construct continuous embeddings directly during inference, such as Soft-Thinking [Zhang et al., 2025] and SwiReasoning [Shi et al., 2025a]. These prior latent reasoning methods mainly use continuous embeddings as a medium for generation. In contrast, C OP T recasts them as inference-time verifiers. This allows C OP T to use uncertainty information preserved by continuous embeddings as in latent reasoning while retaining the readability of explicit reasoning.

3

Methodology

As shown in Fig. 2, C OP T reformulates LLM reasoning into two reversed stages: a leading draftanswer stage and, when necessary, a trailing on-policy thinking stage. The key insight is to first elicit 3

Answer

On-Policy Thinking

>>>

>>>

>>>

Chunk 1

Continuous

Effort

Chunk k

Question

...

...

Figure 2: C OP T starts with a draft answer and performs on-policy thinking conditioned on it. It contrasts the model’s support for the same chosen tokens under discrete and continuous inputs to estimate draft answer reliability, and during thinking, chunk by chunk, to determine the visibility of the draft answer across time steps.

an early-stage answer at low cost, estimate its reliability with a normalized sequence-level reverse KL estimator, and selectively trigger on-policy thinking with dynamic access to the draft answer. 3.1

Reliability Estimator of the Draft Answer

Draft answer elicitation. Let pθ denote the model with parameters θ. Let E ∈ R|V|×d be the input embedding matrix of the model, where V is the vocabulary and d is the hidden size. For any token v ∈ V, E(v) ∈ Rd denotes its embedding. Given a question token sequence q = (q1 , . . . , qm ), instead of allowing the model to think thoroughly, we force it to output </think> at the beginning and go straight into its answering mode. Reliability estimation. To estimate how likely it is that a subsequent thorough thinking process will be required for correcting potential errors, we introduce a normalized sequence-level reverse KL estimator κa . Let the draft phase generate tokens a = (a1 , . . . , aTa ). During draft answer generation, for each generated token at , we cache two items calculated from the next-token distributions: X pt := pθ (at | q, a<t ), et := pθ (v | q, a<t )E(v). v∈V

Here pt is the chosen-token probability, and et is a continuous embedding obtained as the probabilityweighted average over the vocabulary, which preserves uncertainty information at each step. After the draft answer a is completed, we calculate κa to estimate its reliability. More specifically, we compare the student discrete-prefix distribution induced by the original inputs against the teacher continuous-prefix distribution in which inputs are replaced with cached continuous embeddings. For t ∈ {1, 2, . . . Ta }, the student probability is simply pt , and all the teacher probabilities are obtained in parallel using a single forward pass with the modified input embeddings. The teacher probabilities of the original answer tokens are gathered at the corresponding output positions: pet := pθ (at | q, e<t ). QTa e This defines the continuous-prefix probability of the sampled draft answer as peθ (a|q) = t=1 pt . We define the estimator T

κa (a1:Ta ) :=

a 1 X [log pθ (at | q, a<t ) − log pθ (at | q, e<t )] . Ta t=1

For any fixed draft length Ta , κa is an unbiased estimator of the normalized sequence-level reverse KL divergence between the two distributions:   1 Ea1:Ta ∼pθ (·|q) κa (a1:Ta ) = DKL (pθ (· | q) ∥ peθ (· | q)) . Ta 3.2

On-Policy Thinking Conditioned on the Draft Answer

On-policy thinking elicitation. A large κa indicates that answer context becomes substantially less supported with teacher-forced continuous embeddings, i.e., the answer may be unreliable given additional uncertainty information. Let τa denote the reliability threshold. When κa > τa , we force the model to output <think> after the draft answer and move into a subsequent thinking process. 4

Visibility controls for draft answers. For draft answers that are deemed insufficiently reliable, the goal of on-policy thinking is to use any beneficial information when necessary, while avoiding being misled by unreliable draft content. Let the on-policy thinking phase generate tokens r = (rTa +1 , rTa +2 , . . . , rTa +Tr ). We partition the thinking trajectory into chunks of length C. Let the k-th chunk start at position sk := Ta + 1 + (k − 1)C, and span positions [sk , sk + C − 1]. Let mk := 1{a is visible in chunk k} denote the visibility mask for the k-th chunk, and define the visibility-conditioned draft input as  a, mk = 1, a(mk ) := ∅, mk = 0. For each generated token rt in chunk k, we cache pt := pθ (rt | q, a(mk ) , rTa +1:t−1 ),

et :=

X

pθ (v | q, a(mk ) , rTa +1:t−1 )E(v).

v∈V (k) Similarly, we calculate a second KL estimator κr on the current chunk whenever it reaches a

predefined length C to decide whether the previous draft answer should become visible in the next chunk. 1 For t ∈ {sk , . . . , sk + C − 1}, the student chosen-token probability is simply pt , and all the teacher probabilities within the chunk are obtained in parallel using a single forward pass with the modified intra-chunk input embeddings: pet := pθ (rt | q, a(mk ) , rTa +1:sk −1 ; esk :t−1 ). We define the estimator κ(k) r (rsk :sk +C−1 ) :=

1 C

sk +C−1 X

 log pθ (rt | q, a(mk ) , rTa +1:t−1 )−

t=sk

 log pθ (rt | q, a(mk ) , rTa +1:sk −1 ; esk :t−1 ) . (k)

For a fixed chunk length C, κr is an unbiased estimator of the normalized sequence-level reverse KL between the two chunk-level continuation distributions:   Ers :s +C−1 ∼pθ (·|q,a(mk ) ,rTa +1:s −1 ) κ(k) (rsk :sk +C−1 ) = r k k k   1 DKL pθ (· | q, a(mk ) , rTa +1:sk −1 ) peθ (· | q, a(mk ) , rTa +1:sk −1 ) . C (k)

(k)

κr estimates the reliability of the current thinking chunk. A large κr suggests that the current chunk is unstable and more vulnerable to misleading information in the draft answer. Let τr denote the stability threshold. After each complete chunk, we update the visibility of the draft answer for the next chunk by ( (k) 1, κr < τr , mk+1 = 0, otherwise.

4

Theoretical Analysis

In this section, we provide a theoretical interpretation to demonstrate the effectiveness of our C OP T method under certain assumptions. We focus on the reliability of our proposed reverse-KL estimator. Our analysis highlights a key property of the reverse-KL estimator: it measures answer-relevant uncertainty, rather than uncertainty over latent reasoning states themselves. 1 κ and κ are calculated on the already generated sequence, and therefore incur only small overhead once the corresponding a r

chosen-token probabilities and continuous embeddings are cached online during generation.

5

For convenience, we analyze a single answer position. Note that all probability distributions below are conditioned on the question (or equivalently, the prompt) q and the previous output prefix, which we omit when the context is clear. Let S be a finite set of latent reasoning states. A discrete output prefix (along with the prompt) commits the model to one latent state, while a continuous prefix may represent a superposition of several possible states. Let A be a finite set of all possible answers (or equivalently, the next token). When the prefix is discrete, for each latent state s ∈ S, let Ps (a) = pθ (a | s),

∀a ∈ A

denote the next-token distribution induced by committing to s, where θ denotes the model parameters. When the prefix is continuous, we make the following assumption on the output distribution. Assumption 1 (Mixture-linear continuous prefix). Let w be a distribution over S such that the discrete draft prefix commits to a latent state S ∼ w, and then emits the answer A ∼ PS . Let ew denote the corresponding continuous prefix which is determined by the distribution w. We assume the next-token distribution conditioned on a continuous prefix ew is determined by X pθ (a | ew ) = P̄w (a) := w(s)Ps (a), ∀a ∈ A. s∈S

Note that for the emitted answer token A, the local reverse-KL contribution is κ(S, A) = log PS (A) − log P̄w (A). Theorem 1 (Reverse KL measures answer-relevant uncertainty). Under Assumption 1, X
ES∼w, A∼PS [κ(S, A)] = w(s)DKL Ps P̄w = I(S; A), s∈S

where I(S; A) is the mutual information between the latent state S and the emitted answer token A under the joint distribution Pr(S = s, A = a) = w(s)Ps (a). The proof is deferred to Appendix F. Theorem 1 shows that C OP T does not penalize latent-state uncertainty by itself. Instead, it measures whether that uncertainty changes the next answer-token distribution. For example, the continuous prefix may represent a mixture over several possible states, S ∈ {s1 , s2 , s3 }, which can have high entropy. However, if all three states induce the same next answer token or the same next-token distribution, then the emitted token A carries no information about which state was selected. In that case, I(S; A) = 0, and the expected reverse-KL contribution is zero. Thus, high uncertainty over latent states is harmless when all plausible states agree on the next answer. Applying this argument token by token, if the mixture-prefix assumption holds at each answer position t, then the normalized draft score satisfies T

E[κa ] =

a 1 X I(St ; At ), Ta t=1

which is conditioned on the preceding context at each position. Therefore, κa estimates the average amount of answer-relevant uncertainty in the draft answer.

5

Experiments

5.1

Experimental Settings

Models. We evaluate C OP T on pure Transformer-based Qwen3 models [Yang et al., 2025] and hybrid Gated-DeltaNet Qwen3.5 models [Qwen Team, 2026] at 2B, 8B, and 35B scales. This selection allows us to validate the effectiveness of C OP T across model families, scales, and architectures, including pure Transformer, hybrid, dense, and sparse mixture-of-experts models. 6

Table 1: Comparison on mathematics, coding, and STEM reasoning benchmarks with the Qwen3-8B model. Green blocks indicate increasing reasoning effort of C OP T to achieve higher accuracy. Mathematics CoT CoT (Greedy) CopT (Ours)

GSM8K

Math500

AIME24

AIME25

Acc. (%)

# Tokens

Acc. (%)

# Tokens

Acc. (%)

# Tokens

Acc. (%)

# Tokens

95.75 95.83

2138 2240

96.00 96.40

4985 5311

75.83 70.00

12077 11680

67.50 60.00

12924 13292

96.36 (+0.61) 1813 (-15.2%) 97.60 (+1.60) 4851 (-2.7%) 79.17 (+3.34) 11525 (-4.6%) 70.42 (+2.92) 12801 (-1.0%) 95.98 (+0.23) 961 (-55.1%) 96.20 (+0.20) 3609 (-27.6%) HumanEval

LeetCode-Contest

MBPP

GPQA Diamond

Coding & STEM

Acc. (%)

# Tokens

Acc. (%)

# Tokens

Acc. (%)

# Tokens

Acc. (%)

# Tokens

CoT CoT (Greedy)

92.68 93.90

2368 2627

59.44 57.22

7306 6975

94.16 91.44

2033 2724

59.60 56.57

8123 7909

CopT (Ours)

96.34 (+3.66) 1842 (-22.2%) 66.11 (+6.67) 7607 (+4.1%) 94.55 (+1.39) 1997 (-1.8%) 61.62 (+2.02) 6851 (-15.7%) 94.51 (+1.83) 1023 (-56.8%) 61.11 (+1.67) 6993 (-4.3%)

Domains and Benchmarks. We evaluate C OP T on 10 benchmarks spanning four domains: math and STEM reasoning (GSM8K [Cobbe et al., 2021], Math500 [Hendrycks et al., 2021], AIME 2024 [HuggingFaceH4, 2024], AIME 2025 [Yentinglin, 2025], GPQA Diamond [Rein et al., 2024]); coding reasoning (HumanEval [Chen et al., 2021], MBPP [Austin et al., 2021], LeetCodeContest [Guo et al., 2024]); single-turn and multi-turn agentic reasoning (BFCL v4 [Patil et al., 2025], ZebraArena [Zhao et al., 2026]). More details are provided in Appendix E.2. 5.2

Experimental Results on Mathematics and Coding Reasoning

Tab. 1 reports accuracy and generation length on mathematics, coding, and STEM reasoning benchmarks with the Qwen3-8B model. Compared with standard C OT and greedy C OT, C OP T improves accuracy while effectively reducing generation length across most settings. When applicable, we report two sets of C OP T results: one targeting accuracy comparable to or higher than C OT, and another, shown in green, that further improves peak accuracy by increasing reasoning effort. On mathematics benchmarks, the token-saving setting of C OP T improves GSM8K accuracy by +0.23% while reducing generated tokens by 55.1%, and improves Math500 accuracy by +0.20% while reducing generated tokens by 27.6%. These results show substantial efficiency gains on problems that do not require extended thinking. With increasing reasoning effort, C OP T further improves GSM8K and Math500 accuracy by +0.61% and +1.60%, respectively. On more challenging AIME benchmarks, C OP T obtains larger accuracy gains: +3.34% on AIME24 and +2.92% on AIME25. The same trend holds on coding and STEM tasks. At matched accuracy levels, C OP T improves HumanEval accuracy by +1.83% while reducing tokens by 56.8%. With increasing reasoning effort, C OP T achieves larger accuracy gain of +3.66%, +6.67%, +1.39%, +2.02% on HumanEval, LeetCode-Contest, MBPP, and GPQA Diamond, respectively. These results suggest that C OP T improves peak accuracy by selectively invoking on-policy thinking when the draft answer appears insufficiently reliable. This is especially beneficial on harder benchmarks such as AIME24, AIME25, LeetCode-Contest, and GPQA Diamond, where draft answers are more likely to require correction. In addition, C OP T reduces unnecessary thinking on easier examples by allowing sufficiently reliable draft answers to be accepted earlier. This leads to considerable token savings on benchmarks such as GSM8K and HumanEval. 5.3

Comparison with Training-Free Continuous-Generation Methods

Tab. 2 compares C OP T with two training-free methods that directly use continuous embeddings for generation, Soft-Thinking [Zhang et al., 2025] and SwiReasoning [Shi et al., 2025a], with the Qwen3-8B model. C OP T differs from them by keeping generation fully explicit and recasting continuous embeddings as inference-time verifiers. Overall, C OP T achieves the best accuracy with the fewest generated tokens. Compared with SwiReasoning, C OP T improves accuracy by +0.30%, 7

Table 2: Comparison with training-free methods that use continuous embeddings for generation. Token counts are measured by generation steps, regardless of whether the steps are explicit or latent. Method

GSM8K

AIME25

HumanEval

GPQA Diamond

Acc. (%) # Tokens Acc. (%) # Tokens Acc. (%) # Tokens Acc. (%) # Tokens

Fully Explicit Readability

Soft-Thinking SwiReasoning

95.38 96.06

2073 2218

68.33 70.00

13665 13911

92.07 95.73

2408 2894

59.60 61.11

8153 8359

× ×

CopT (Ours)

96.36

1813

70.42

12801

96.34

1842

61.62

6851

✓

Figure 3: Left and center: Controllable reasoning effort by τa and τr . Right: C OP T reduces average per-sample latency at comparable or higher accuracy, measured on a single H200 GPU. +0.42%, +0.61%, and +0.51% on four mathematics, coding, and STEM benchmarks, while using 18.3%, 8.0%, 36.4%, and 18.0% fewer tokens. In addition, the reasoning processes of these latent C OT methods are not fully represented in natural language. In contrast, C OP T allows users to inspect the complete reasoning process while still benefiting from uncertainty information preserved in continuous embeddings. These results indicate that continuous embeddings do not need to be used as a generation strategy to improve reasoning. By recasting them as contrastive verifiers, C OP T better balances accuracy, efficiency, and readability. 5.4

Controllable Reasoning Effort and Latency Reduction

Fig. 3 studies how C OP T controls reasoning effort and how token savings translate into real latency reduction. The left and center panels show the accuracy-token trade-off obtained by sweeping τa and τr . These curves show that C OP T can reduce generation length when less thinking is needed, while allocating more on-policy thinking to obtain higher accuracy when the thresholds trigger stronger verification and correction. The right panel reports average latency per sample at comparable or higher accuracy. The reductions are consistent with the token-efficiency gains: C OP T reduces latency by 37% on GSM8K, 20% on Math500, and 69% on HumanEval. These results show that C OP T reduces actual generation latency while preserving reasoning accuracy. 5.5

Experimental Results on Agentic Reasoning

Tab. 3 evaluates C OP T on agentic reasoning benchmarks, including the non-live and live splits of BFCL v4 and multi-turn ZebraArena with one missing clue. We use Qwen3.5 models for agentic evaluation because they are designed for stronger agentic capabilities than reasoning-only models [Qwen Team, 2026]. On BFCL v4, C OP T consistently improves accuracy while reducing generation length across scales. In particular, C OP T maintains comparable or higher accuracy with 40.6% fewer tokens on Qwen3.5-2B and 44.7% fewer tokens on Qwen3.5-35B-A3B. The gains are stronger on the multi-turn ZebraArena benchmark. C OP T improves accuracy by +2.98%, +13.14%, and +23.03% on the small, medium, and large splits, respectively. For generation length, C OP T uses 3.8% more tokens on the small split, but reduces tokens by 24.4% on the medium split and 19.6% on the large split. These results suggest that C OP T is especially useful in longer agentic interactions, where improved accuracy-token trade-offs can accumulate across multiple rounds of interaction. For simplicity, we use the same reasoning effort of τr = 0, τa = 0.3 for all single-turn BFCL v4 splits and τr = 0, τa = 1.5 for all multi-turn ZebraArena splits. Better trade-offs could be achieved by setting separate reasoning effort for different task difficulties. 8

Table 3: Comparison on agentic reasoning benchmarks BFCL v4 (single-turn) and ZebraArena (multiturn). Green blocks indicate increasing reasoning effort of CopT to achieve higher accuracy. Qwen3.5-2B Method

CoT

Qwen3.5-35B-A3B

BFCL v4

BFCL v4

(non-live and live)

(non-live and live)

ZebraArena (multi-turn)

Acc. (%)

# Tokens

Acc. (%)

77.53

234

85.77

Small

Medium

Large

# Tokens Acc. (%) # Tokens Acc. (%) # Tokens Acc. (%) # Tokens 93.71

3357

78.37 (+0.84) 164 (-29.9%) 86.45 (+0.68) 168 (-28.5%) 96.69 CopT (Ours) 78.01 (+0.48) 139 (-40.6%) 86.17 (+0.40) 130 (-44.7%) (+2.98)

235

75.00

7217

59.21

8070

3486

88.14

5457

82.24

6486

(+3.8%)

(+13.14) (-24.4%) (+23.03) (-19.6%)

Figure 4: Left: The estimator κa identifies erroneous drafts more precisely than uniform selection. Right: The threshold τr trades off correction rate and token usage by controlling draft visibility. 5.6

Ablation Studies on Design Choices

Draft answer reliability estimation. We first examine whether the draft-answer reliability estimator κa can effectively identify unreliable draft answers on GSM8K. Fig. 4 (left) compares C OP T with a uniform selection under different thresholds τa . For each threshold, we report the number of caught errors, the precision defined as the fraction of presumed errors that are truly erroneous, and the safe acceptance rate defined as the fraction of directly accepted answers that are correct. As τa becomes stricter, C OP T selects fewer draft answers as unreliable, but the selected subset becomes increasingly concentrated on truly erroneous drafts. In contrast, uniform selection yields a much lower and nearly flat precision, indicating that uniform allocation of additional thinking fails to distinguish unreliable answers from reliable ones. These results confirm that κa provides a meaningful reliability estimate. Visibility control for draft answers. We next study the effect of the visibility estimator κr during on-policy thinking on Math500. Fig. 4 (right) varies the threshold τr and reports the number of actual draft errors and the number of errors that are successfully corrected. Smaller τr corresponds to stricter visibility control, exposing the draft answer less often. As τr becomes stricter, C OP T increases reasoning effort with less reliance on the potentially wrong draft answer, and corrects a larger fraction of erroneous draft answers. These results show that the estimator κr plays a direct role in reflection and correction. If the draft answer is always exposed, unreliable content may continue to influence the on-policy thinking process and prevent the model from correcting its initial mistake. By contrast, dynamic visibility control allows C OP T to hide the draft answer when the current thinking chunk appears unstable, while still allowing access to the draft when it provides useful partial information.

6

Conclusion

This paper introduces C OP T, a training-free LLM reasoning pipeline that reverses the usual order of thinking and answering. C OP T first elicits a draft answer and then invokes on-policy thinking conditioned on the draft when it appears unreliable. C OP T recasts continuous embeddings, previously used for generation in latent C OT methods, as inference-time verifiers by contrasting the model’s support for the same generated tokens under discrete-token and continuous-embedding inputs. Experiments on math, coding, and agentic reasoning tasks show that C OP T consistently improves both accuracy and token efficiency, suggesting that on-policy thinking with contrastive verification from continuous embeddings provides a practical path toward more cost-effective LLM reasoning at inference time. 9

References Marah Abdin, Sahaj Agarwal, Ahmed Awadallah, Vidhisha Balachandran, Harkirat Behl, Lingjiao Chen, Gustavo de Rosa, Suriya Gunasekar, Mojan Javaheripi, Neel Joshi, et al. Phi-4-reasoning technical report. arXiv preprint arXiv:2504.21318, 2025. Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras. arXiv preprint arXiv:2503.01743, 2025. Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925, 2025. Anthropic. System card: Claude opus 4 & claude sonnet 4, 2025a. URL https://www.anthropic. com/claude-4-system-card. Anthropic. Claude opus 4.5 system card. https://assets.anthropic.com/m/ 64823ba7485345a7/Claude-Opus-4-5-System-Card.pdf, November 2025b. System card. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. Siddharth Boppana, Annabel Ma, Max Loeffler, Raphael Sarfati, Eric Bigelow, Atticus Geiger, Owen Lewis, and Jack Merullo. Reasoning theater: Disentangling model beliefs from chain-of-thought. arXiv preprint arXiv:2603.05488, 2026. Ruisheng Cao, Mouxiang Chen, Jiawei Chen, Zeyu Cui, Yunlong Feng, Binyuan Hui, Yuheng Jing, Kaixin Li, Mingze Li, Junyang Lin, et al. Qwen3-coder-next technical report. arXiv preprint arXiv:2603.00729, 2026. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021. Xinghao Chen, Anhao Zhao, Heming Xia, Xuan Lu, Hanlin Wang, Yanjun Chen, Wei Zhang, Jian Wang, Wenjie Li, and Xiaoyu Shen. Reasoning beyond language: A comprehensive survey on latent chain-of-thought reasoning. arXiv preprint arXiv:2505.16782, 2025a. Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato, Carson Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner, Fabien Roger, et al. Reasoning models don’t always say what they think. arXiv preprint arXiv:2505.05410, 2025b. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025. 10

Google DeepMind. Ai achieves silver-medal standard solving international mathematical olympiad problems. https://deepmind.google/discover/blog/ ai-solves-imo-problems-at-silver-medal-level/, 2024. Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens. In The Twelfth International Conference on Learning Representations, 2024. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming - the rise of code intelligence. arXiv preprint arXiv: 2401.14196, 2024. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space. arXiv preprint arXiv:2412.06769, 2024. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021. Zixuan Huang, Xin Xia, Yuxi Ren, Jianbin Zheng, Xuanda Wang, Zhixia Zhang, Hongyan Xie, Songshi Liang, Zehao Chen, Xuefeng Xiao, et al. Does your reasoning model implicitly know when to stop thinking? arXiv preprint arXiv:2602.08354, 2026. HuggingFaceH4. Aime 2024 (american invitational mathematics examination 2024). Hugging Face dataset, 2024. URL https://huggingface.co/datasets/HuggingFaceH4/aime_2024. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. Qwen2.5-coder technical report. arXiv preprint arXiv:2409.12186, 2024. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024. Jindong Li, Yali Fu, Li Fan, Jiahong Liu, Yao Shu, Chengwei Qin, Menglin Yang, Irwin King, and Rex Ying. Implicit reasoning in large language models: A comprehensive survey. arXiv preprint arXiv:2509.02350, 2025. Jack Lindsey. Emergent introspective awareness in large language models. arXiv:2601.01828, 2026.

arXiv preprint

Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556, 2025a. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025b. Lu Ma, Hao Liang, Meiyi Qiang, Lexiang Tang, Xiaochen Ma, Zhen Hao Wong, Junbo Niu, Chengyu Shen, Runming He, Yanhao Li, et al. Learning what reinforcement learning can’t: Interleaved online fine-tuning for hardest questions. arXiv preprint arXiv:2506.07527, 2025. 11

Meta AI. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation. https://ai.meta.com/blog/llama-4-multimodal-intelligence/, 2025a. Meta AI. Llama 3.3 model card. https://github.com/meta-llama/llama-models/blob/ main/models/llama3_3/MODEL_CARD.md, 2025b. OpenAI. Openai o3 and o4-mini, introducing-o3-and-o4-mini.

2025.

URL https://openai.com/index/

Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. Advances in Neural Information Processing Systems, 37: 126544–126565, 2024. Shishir G. Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning, 2025. Jacob Pfau, William Merrill, and Samuel R Bowman. Let’s think dot by dot: Hidden computation in transformer language models. arXiv preprint arXiv:2404.15758, 2024. Qwen Team. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024. Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL https://qwenlm.github.io/blog/qwq-32b/. Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URL https://qwen. ai/blog?id=qwen3.5. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024. Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu, Yali Du, and Yulan He. Codi: Compressing chain-of-thought into continuous space via self-distillation. arXiv preprint arXiv:2502.21074, 2025. Dachuan Shi, Chaofan Tao, Anyi Rao, Zhendong Yang, Chun Yuan, and Jiaqi Wang. CrossGET: Cross-guided ensemble of tokens for accelerating vision-language transformers. In Proceedings of the 41st International Conference on Machine Learning, volume 235, pages 44960–44990. PMLR, 2024. Dachuan Shi, Abedelkadir Asi, Keying Li, Xiangchi Yuan, Leyan Pan, Wenke Lee, and Wen Xiao. Swireasoning: Switch-thinking in latent and explicit for pareto-superior reasoning llms. arXiv preprint arXiv:2510.05069, 2025a. Dachuan Shi, Yonggan Fu, Xiangchi Yuan, Zhongzhi Yu, Haoran You, Sixu Li, Xin Dong, Jan Kautz, Pavlo Molchanov, and Yingyan Celine Lin. LaCache: Ladder-shaped KV caching for efficient long-context modeling of large language models. In Proceedings of the 42nd International Conference on Machine Learning, volume 267, pages 54892–54903. PMLR, 2025b. DiJia Su, Hanlin Zhu, Yingchen Xu, Jiantao Jiao, Yuandong Tian, and Qinqing Zheng. Token assorted: Mixing latent and text tokens for improved language model reasoning. arXiv preprint arXiv:2502.03275, 2025. 12

Jihoon Tack, Jack Lanchantin, Jane Yu, Andrew Cohen, Ilia Kulikov, Janice Lan, Shibo Hao, Yuandong Tian, Jason Weston, and Xian Li. Llm pretraining with continuous concepts. arXiv preprint arXiv:2502.08524, 2025. Wenhui Tan, Jiaze Li, Jianzhong Ju, Zhenbo Luo, Jian Luan, and Ruihua Song. Think silently, think fast: Dynamic latent compression of llm reasoning chains. arXiv preprint arXiv:2505.16552, 2025. Kimi Team, Yifan Bai, Yiping Bao, Y Charles, Cheng Chen, Guanduo Chen, Haiting Chen, Huarong Chen, Jiahao Chen, Ningxin Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534, 2025. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, and et al. Chain-of-thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022. Xilin Wei, Xiaoran Liu, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Jiaqi Wang, Xipeng Qiu, and Dahua Lin. Sim-cot: Supervised implicit chain-of-thought. arXiv preprint arXiv:2509.20317, 2025. Junhong Wu, Jinliang Lu, Zixuan Ren, Gangqiang Hu, Zhi Wu, Dai Dai, and Hua Wu. Llms are single-threaded reasoners: Demystifying the working mechanism of soft thinking. arXiv preprint arXiv:2508.03440, 2025. Kejing Xia, Mingzhe Li, Lixuan Wei, Zhenbang Du, Xiangchi Yuan, Dachuan Shi, Qirui Jin, and Wenke Lee. Metastate: Persistent working memory enhances reasoning in discrete diffusion language models. arXiv preprint arXiv:2603.01331, 2026. Yige Xu, Xu Guo, Zhiwei Zeng, and Chunyan Miao. Softcot: Soft chain-of-thought for efficient reasoning with llms. arXiv preprint arXiv:2502.12134, 2025. Zhongxing Xu, Zhonghua Wang, Zhe Qian, Dachuan Shi, Feilong Tang, Ming Hu, Shiyan Su, Xiaocheng Zou, Wei Feng, Dwarikanath Mahapatra, et al. Thinking in uncertainty: Mitigating hallucinations in mlrms with latent entropy-aware decoding. arXiv preprint arXiv:2603.13366, 2026. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809–11822, 2023. Yentinglin. Aime 2025 (american invitational mathematics examination 2025). Hugging Face dataset, 2025. URL https://huggingface.co/datasets/yentinglin/aime_2025. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025. Xinlei Yu, Zhangquan Chen, Yongbo He, Tianyu Fu, Cheng Yang, Chengming Xu, Yue Ma, Xiaobin Hu, Zhe Cao, Jie Xu, et al. The latent space: Foundation, evolution, mechanism, ability, and outlook. arXiv preprint arXiv:2604.02029, 2026. Xiangchi Yuan, Xiang Chen, Tong Yu, Dachuan Shi, Can Jin, Wenke Lee, and Saayan Mitra. Mitigating forgetting between supervised and reinforcement learning yields stronger reasoners. arXiv preprint arXiv:2510.04454, 2025a. Xiangchi Yuan, Chunhui Zhang, Zheyuan Liu, Dachuan Shi, Soroush Vosoughi, and Wenke Lee. Superficial self-improved reasoners benefit from model merging. arXiv preprint arXiv:2503.02103, 2025b. 13

Xiangchi Yuan, Dachuan Shi, Chunhui Zhang, Zheyuan Liu, Shenglong Yao, Soroush Vosoughi, and Wenke Lee. Behavior knowledge merge in reinforced agentic models. arXiv preprint arXiv:2601.13572, 2026. Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, et al. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models. arXiv preprint arXiv:2508.06471, 2025a. Boyi Zeng, Shixiang Song, Siyuan Huang, Yixuan Wang, He Li, Ziwei He, Xinbing Wang, Zhiyu Li, and Zhouhan Lin. Pretraining language models to ponder in continuous space, 2025b. Zhen Zhang, Xuehai He, Weixiang Yan, Ao Shen, Chenyang Zhao, Shuohang Wang, Yelong Shen, and Xin Eric Wang. Soft thinking: Unlocking the reasoning potential of llms in continuous concept space. arXiv preprint arXiv:2505.15778, 2025. Wanjia Zhao, Ludwig Schmidt, James Zou, Vidhisha Balachandran, and Lingjiao Chen. Zebraarena: A diagnostic simulation environment for studying reasoning-action coupling in tool-augmented llms. arXiv preprint arXiv:2603.18614, 2026. Hanlin Zhu, Shibo Hao, Zhiting Hu, Jiantao Jiao, Stuart Russell, and Yuandong Tian. Emergence of superposition: Unveiling the training dynamics of chain of continuous thought. arXiv preprint arXiv:2509.23365, 2025a. Hanlin Zhu, Shibo Hao, Zhiting Hu, Jiantao Jiao, Stuart Russell, and Yuandong Tian. Reasoning by superposition: A theoretical perspective on chain of continuous thought. arXiv preprint arXiv:2505.12514, 2025b. Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406.11931, 2024. Rui-Jie Zhu, Tianhao Peng, Tianhao Cheng, Xingwei Qu, Jinfa Huang, Dawei Zhu, Hao Wang, Kaiwen Xue, Xuanliang Zhang, Yong Shan, et al. A survey on latent reasoning. arXiv preprint arXiv:2507.06203, 2025c.

14

A

Impact Statement

This paper studies an inference-time method for reasoning LLMs. By improving accuracy and token efficiency, C OP T may make reasoning LLMs more accessible and cost-effective for beneficial applications, such as scientific problem solving, programming assistance, and tool-using agents. At the same time, stronger and cheaper reasoning may also increase the usefulness of LLMs in harmful applications if deployed without proper safeguards. These risks are not specific to C OP T, but follow from improving the capability of reasoning LLMs. C OP T does not introduce a new class of societal risks beyond those associated with the underlying models.

B

Usage of Large Language Models

We used LLMs and coding agents to assist with language polishing, improving readability, and debugging code. LLMs were not misused intentionally in any part of this work. All technical ideas and experimental results are the product of human efforts.

C

Limitations

While C OP T reformulates LLM reasoning in a training-free manner, its current design has a few limitations. First, the proposed KL estimators are evaluated on the realized trajectory available at inference time, rather than averaged over multiple continuations sampled from the model distribution. This matches our per-instance reasoning control setting, but may lead to higher variance than multisample estimates. We mitigate this by averaging over tokens, while future work may study more adaptive or lower-variance estimators. Second, C OP T requires access to next-token probabilities. This is natural for open-weight models and inference systems that expose logits, but may be less convenient for closed APIs that only return text outputs. Future work may explore API-compatible variants that approximate the contrastive reliability using limited observable information.

D

Supplementary Experiments

D.1

Ablation on Answer-Content Granularity

Table 4: Ablation on the granularity of κa estimation. The default setting computes κa over the whole draft answer, while the answer-content setting computes it only on the extracted answer span. Granularity Whole draft answer (default) Answer content only

GSM8K

Math500

Acc. (%)

# Tokens

Acc. (%)

# Tokens

95.98 96.36 (+0.38)

961 885 (-7.9%)

96.20 96.40 (+0.20)

3609 3214 (-10.9%)

Tab. 4 studies the effect of the granularity used for κa estimation. In the default setting, we compute the KL-based reliability estimator over the whole draft answer. We find that a finer-grained variant can further improve performance when the final answer span is easy to identify. For example, in mathematics tasks, the final answer is usually enclosed by \boxed{}, which allows us to extract the answer tokens and compute κa only on this compact region. This setting improves GSM8K accuracy from 95.98% to 96.36% while reducing the average generation length from 961 to 885 tokens. On Math500, it improves accuracy from 96.20% to 96.40% and reduces the average generation length from 3609 to 3214 tokens. The results suggest that more precise reliability estimation can be useful when a task provides a clear and compact answer region. However, such extraction is task-dependent and is not always available for reasoning tasks beyond mathematics. To keep the method general across domains, all results in this paper use the default setting, where κa is computed over the whole draft. 15

Table 5: Ablation on the maximum draft-answer length. Maximum Draft Length 256 512 1,024 (default) 2,048

D.2

GSM8K

Math500

Acc. (%)

# Tokens

Acc. (%)

# Tokens

95.67 (-0.31) 95.37 (-0.61) 95.98 95.52 (-0.46)

1075 (+11.9%) 1021 (+6.2%) 961 965 (+0.4%)

96.60 (+0.40) 97.00 (+0.80) 96.20 94.40 (-1.80)

3405 (-5.7%) 3660 (+1.4%) 3609 3265 (-9.5%)

Ablation on the Maximum Draft-Answer Length

Tab. 5 studies the effect of the maximum generation length for draft answers. We set the maximum draft length to 1,024 by default. This cap serves as a practical safeguard for the draft-first setting. We observe that for a small number of challenging tasks, reasoning LLMs may not be well calibrated to directly produce a concise answer before extended thinking. In such cases, the draft-answer stage can occasionally continue with repetitive or uninformative text. Bounding the draft length prevents these cases from consuming excessive tokens before the reliability estimator and subsequent on-policy thinking are applied. Overall, the maximum draft-answer length does not behave as a highly sensitive hyperparameter. On GSM8K, the default length of 1,024 achieves the best accuracy while using the fewest tokens among all settings. On Math500, shorter limits such as 256 and 512 can improve accuracy, suggesting that task-specific draft caps may provide additional gains. However, increasing the cap to 2,048 does not improve accuracy on either benchmark. These results suggest that the draft-length cap mainly serves to prevent unusually long or repetitive drafts, rather than acting as a sensitive hyperparameter. We therefore use 1,024 as a simple default for all experiments, which provides stable performance without task-specific tuning. The only exception is the multi-turn ZebraArena benchmark, where we use a smaller maximum draft length of 512. Since ZebraArena requires multiple rounds of interaction, the effective draft budget can accumulate across turns. A shorter per-turn cap helps control total context growth while still allowing the model to produce a concise draft answer at each turn. D.3

Detailed Results on the LeetCode-Contest Benchmark

Table 6: Per-split accuracy and generation length comparison on the LeetCode-Contest benchmark. Method

Easy

Medium

Hard

Overall

# Tokens

CoT CoT (Greedy)

57.78 64.44

68.13 58.24

43.18 47.73

59.44 57.22

7306 6975

CopT (Ours)

64.44 (+6.66) 51.11 (-6.67)

72.53 (+4.40) 69.23 (+1.10)

54.55 (+11.37) 54.55 (+11.37)

66.11 (+6.67) 61.11 (+1.67)

7607 (+4.1%) 6993 (-4.3%)

Tab. 6 reports per-split results on the LeetCode-Contest benchmark. C OP T improves the overall accuracy from 59.44% to 66.11% with a similar token budget. The largest gain appears on the hard split, where C OP T improves accuracy from 43.18% to 54.55%, yielding an absolute gain of +11.37%. This suggests that on-policy thinking is especially useful on harder problems, where the initial draft answer is more likely to require further reflection and correction. The two C OP T rows show different accuracy-efficiency trade-offs. The set with increasing reasoning effort achieves the best overall accuracy, improving all three splits over standard C OT, with only a small increase in generation length. The other set reduces token usage by 4.3% while still improving overall accuracy by +1.67%. Although the other set reduces accuracy on the easy split, it preserves the gain on the hard split and still improves the medium split. These results show that C OP T can trade generation length for accuracy in a controlled way. 16

D.4

Detailed Results on BFCL v4 Benchmark Table 7: Per-split accuracy comparison on the BFCL v4 benchmark.

Model

Method

live_multiple live_parallel live_parallel_multiple live_simple multiple parallel

CoT Qwen3.5-2B

CopT (Ours) CoT

Qwen3.5-35B-A3B

Model

CopT (Ours)

Method

CopT (Ours) CoT

Qwen3.5-35B-A3B

62.50 81.25

54.17 54.17

72.87 69.77

90.50 91.50

81.50 82.00

(+1.52)

(+18.75)

(+0.00)

(-3.10)

(+1.00)

(+0.50)

81.20 81.58

87.50 87.50

79.17 79.17

85.27 85.66

93.50 95.00

91.50 91.50

(+0.38)

(+0.00)

(+0.00)

(+0.39)

(+1.50)

(+0.00)

parallel_multiple simple_java simple_javascript simple_python Overall # Tokens

CoT Qwen3.5-2B

73.41 74.93

CopT (Ours)

79.00 79.50

69.00 72.00

60.00 60.00

88.50 89.25

77.53 78.37

(+0.50)

(+3.00)

(+0.00)

(+0.75)

(+0.84) (-29.9%)

88.00 88.00

78.00 82.00

78.00 72.00

93.50 95.50

85.77 86.45

(+0.00)

(+4.00)

(-6.00)

(+2.00)

(+0.68) (-28.5%)

234 164 235 168

Tab. 7 reports per-split results on the non-live and live splits of BFCL v4. The results show that C OP T preserves or improves accuracy in most cases across different function-calling categories while reducing unnecessary reasoning tokens, suggesting that C OP T provides a better accuracy-efficiency trade-off for structured agentic reasoning.

E

Supplementary Details

E.1

Implementation Details

All experiments are conducted on a single NVIDIA H200 GPU. We use the default generation settings of Qwen3 [Yang et al., 2025] and Qwen3.5 [Qwen Team, 2026] models, including temperature of 0.6, top-p of 0.95, top-k of 20, and min-p of 0 for all experiments. The chunk size of C OP T is set to  
C = max 1, T4a , which balances timely estimation with low additional cost. By default, the draft answer is invisible to the first chunk of on-policy thinking, since the earliest value of κr is available only after the first chunk is completed. We set the maximum draft answer length to 512 for the multi-turn ZebraArena benchmark and 1,024 for all other benchmarks. E.2

Benchmark Details

We evaluate C OP T on 10 reasoning benchmarks, covering GSM8K [Cobbe et al., 2021], Math500 [Hendrycks et al., 2021], AIME 2024 [HuggingFaceH4, 2024], AIME 2025 [Yentinglin, 2025], GPQA Diamond [Rein et al., 2024] for mathematics and STEM reasoning; HumanEval [Chen et al., 2021], LeetCode-Contest [Guo et al., 2024], MBPP [Austin et al., 2021] for coding reasoning; and BFCL v4 [Patil et al., 2025], ZebraArena [Zhao et al., 2026] for agentic reasoning. • GSM8K: We evaluate on the test set of 1,319 grade-school math problems. The benchmark tests whether a model can solve natural-language arithmetic questions that often require several reasoning steps. : https://huggingface.co/datasets/openai/gsm8k. • Math500: We use the curated set of 500 problems from the MATH benchmark. The problems span multiple high-school competition mathematics areas, including algebra, geometry, number theory, precalculus, and intermediate algebra. : https://huggingface.co/ datasets/HuggingFaceH4/MATH-500. • AIME 2024: The benchmark contains 30 problems from the 2024 American Invitational Mathematics Examination, covering both AIME I and AIME II. Each problem requires a concise numeric answer and is designed to test competition-level mathematical reasoning. : https://huggingface.co/datasets/HuggingFaceH4/aime_2024. 17

• AIME 2025: The benchmark contains 30 problems from the 2025 American Invitational Mathematics Examination, covering both AIME I and AIME II. Each problem requires a concise numeric answer and continues the focus on competition-style math reasoning with challenging questions that test symbolic and logical skills. : https://huggingface. co/datasets/yentinglin/aime_2025. • GPQA Diamond: We use the Diamond subset of GPQA, which consists of 198 expertverified multiple-choice STEM questions. The questions mainly cover mathematics, physics, chemistry, biology, and computer science, and are intended to be difficult for non-experts. The problems are designed to evaluate expert-level factual knowledge and reasoning ability. : https://huggingface.co/datasets/hendrydong/gpqa_diamond_mc. • HumanEval: We use the 164 hand-written Python programming tasks from HumanEval. Each task provides a function signature and docstring, and correctness is measured by executing the generated function against unit tests. : https://huggingface.co/datasets/ openai/openai_humaneval. • LeetCode-Contest: We evaluate on 180 programming contest problems collected from LeetCode contests. The benchmark contains problems of different difficulty levels, and model outputs are judged by whether the generated solutions pass all the associated tests. : https://huggingface.co/datasets/TechxGenus/LeetCode-Contest. • MBPP: We use the sanitized test split, which contains 257 Python programming problems. Each example includes a natural-language task description, a reference solution, and unit tests used for execution-based scoring. : https://huggingface.co/datasets/ google-research-datasets/mbpp. • BFCL v4: The Berkeley Function Calling Leaderboard v4 evaluates the ability of LLMs to invoke functions and tools accurately in realistic agentic settings. We use all subtasks from the non-live and live splits, which contain 2,501 problems in total. : https: //github.com/shishirpatil/gorilla. • ZebraArena: A diagnostic simulation environment for evaluating multi-turn agentic reasoning in tool-augmented LLMs. It is built on Zebra logic puzzles under a missing-clues setting, where models must query the environment for hidden facts or relations and then solve a uniquely verifiable constraint satisfaction problem. : https://github.com/ wanjiaZhao1203/ZebraArena. Following the evaluation setup of Qwen3 and Qwen3.5, we allocate a large maximum output budget to allow sufficient reasoning. Specifically, we set the maximum generation length to 32,768 tokens for GSM8K, Math500, GPQA Diamond, HumanEval, LeetCode-Contest, MBPP, and BFCL v4, and 38,912 tokens for AIME 2024 and AIME 2025. For ZebraArena, we set the maximum generation length to 32,768 tokens for the small split, 65,536 tokens for the medium split, and 98,304 tokens for the large split. We repeat evaluations eight times and report the average accuracy for both C OP T and baselines on the AIME 2024 and AIME 2025 benchmarks. E.3

Sequence Distributions for the KL Estimators

We define the sequence distributions used by the KL estimators for fixed generated lengths. For κa in the draft answer stage. For a fixed draft length Ta , the discrete-prefix continuation distribution over the draft answer is pθ (a1:Ta | q) :=

Ta Y

pθ (at | q, a<t ).

t=1

The corresponding continuous-prefix continuation distribution is peθ (a1:Ta | q) :=

Ta Y

pθ (at | q, e<t ),

t=1

where the discrete prefix preceding each draft token is replaced by the cached continuous embeddings. 18

For κr in the on-policy thinking stage. We partition the on-policy thinking trajectory into chunks of length C. Let the k-th chunk start at position sk := Ta + 1 + (k − 1)C, and span positions [sk , sk + C − 1]. Let mk denote the visibility state of the draft answer for the k-th chunk. Conditioned on the visibility-controlled draft-answer input a(mk ) and the prefix rTa +1:sk −1 preceding the current chunk, the student chunk-level continuation distribution is pθ (rsk :sk +C−1 | q, a(mk ) , rTa +1:sk −1 ) :=

sk +C−1 Y

pθ (rt | q, a(mk ) , rTa +1:t−1 ).

t=sk

The corresponding continuous-prefix intra-chunk continuation distribution is peθ (rsk :sk +C−1 | q, a(mk ) , rTa +1:sk −1 ) :=

sk +C−1 Y

pθ (rt | q, a(mk ) , rTa +1:sk −1 ; esk :t−1 ),

t=sk

where the already generated content inside the current chunk is replaced by the cached continuous embeddings esk :t−1 , while the prefix before the current chunk remains discrete.

F

Additional Derivations for Answer-Relevant Uncertainty

In this section, we provide the full derivation and several consequences of Theorem 1. The main message is that the reverse-KL estimator measures uncertainty that affects the next answer token, rather than uncertainty over latent reasoning states themselves. Proof of Theorem 1. Recall the mixture-linear prefix model under Assumption 1. The local reverseKL contribution is κ(S, A) = log PS (A) − log P̄w (A). Taking expectation gives E[κ(S, A)] =

X

w(s)

=

  Ps (a) log Ps (a) − log P̄w (a)

a

s∈S

X

X

w(s)DKL Ps P̄w .

s∈S

Moreover, under the joint distribution Pr(S = s, A = a) = w(s)Ps (a), the marginal distribution of A is Pr(A = a) =

X

w(s)Ps (a) = P̄w (a).

s

Therefore, I(S; A) =

X

w(s)Ps (a) log

s,a

Pr(S = s, A = a) Pr(S = s) Pr(A = a)

Ps (a) P̄w (a) s,a X
= w(s)DKL Ps P̄w . =

X

w(s)Ps (a) log

s

Hence, E[κ(S, A)] = I(S; A), which completes the proof.

19

Harmless latent-state uncertainty. If all latent states in the support of w induce the same nexttoken distribution, Ps = P⋆ for all s ∈ supp(w), then P̄w =

X

w(s)Ps = P⋆ .

s

Therefore, κ(S, A) = log P⋆ (A) − log P⋆ (A) = 0 almost surely. Thus, the reverse-KL score is zero even if the latent-state entropy H(S) is large. This formalizes the intuition that a continuous prefix may encode a superposition of many states and still be reliable. If all plausible states agree on the next-token distribution, then the uncertainty is irrelevant to the answer. Stability under approximately equivalent states. More generally, suppose all state-conditioned distributions are close to a common distribution P⋆ . Then X E[κ(S, A)] ≤ w(s)DKL (Ps ∥ P⋆ ) . s

To see this, note that X X X Ps (a) w(s)DKL (Ps ∥P⋆ ) = w(s) Ps (a) log P⋆ (a) s s a   X X Ps (a) P̄w (a) w(s) Ps (a) log + log = P⋆ (a) P̄w (a) s a X X X Ps (a) X P̄w (a) = + w(s) Ps (a) log w(s) Ps (a) log P⋆ (a) P̄ (a) w s a s a " # X X X P̄w (a) = w(s)DKL (Ps ∥P̄w ) + w(s)Ps (a) log P⋆ (a) s a s =

X

=

X

w(s)DKL (Ps ∥P̄w ) +

s

X

P̄w (a) log

a

P̄w (a) P⋆ (a)

w(s)DKL (Ps ∥P̄w ) + DKL (P̄w ∥P⋆ ).

s

Since KL divergence is nonnegative, X
X w(s)DKL Ps P̄w ≤ w(s)DKL (Ps ∥ P⋆ ) . s

s

By Theorem 1, E[κ(S, A)] =

X

w(s)DKL Ps P̄w ,

s

which proves the claim. Deterministic-token case. Suppose each latent state s deterministically implies one answer token g(s) ∈ A, so that

Ps (a) = 1{a = g(s)}.

Define the marginal probability of the answer token induced by the soft prefix state as X ρ(a) = Pr(g(S) = a) = w(s). s:g(s)=a

20

Then P̄w (a) =

X

w(s)1{a = g(s)} = ρ(a).

s

Since A = g(S) deterministically, κ(S, A) = log 1 − log ρ(g(S)) = − log ρ(g(S)). Taking expectation gives E[κ(S, A)] =

X

ρ(a)(− log ρ(a)) = H(g(S)).

a

Therefore, under the deterministic-answer setting, the reverse-KL score is exactly the entropy of the induced answer token g(S) instead of the entropy of the latent state S.

21

Related documents

• Argumentative reasoning patterns#776177
• Reasoning or Rambling? Exploring the Effect of Thinking on Agent Persuasion#799887
• Conventional Norms of Reasoning#754871
• VACS: Value-Aligned Compositional Shielding for Multi-Agent Reasoning#1062037
• VLM-in-Sandbox: Visual Workspaces for Agentic Visual Reasoning#1050541