arXiv:2604.18464v1 [cs.LG] 20 Apr 2026
S EMANTIC S TEP P REDICTION : M ULTI -S TEP L ATENT F ORECASTING IN LLM R EASONING T RAJECTORIES VIA S TEP S AMPLING
Yuan Yidi Home Team Science and Technology Agency [email protected]
April 21, 2026
A BSTRACT [This arXiv paper is entirely derived from work conducted as part of the NUS CS5260 course.] Semantic Tube Prediction (STP) leverages representation geometric to regularize LLM hiddenstate trajectories toward locally linear geodesics during fine-tuning, thereby greatly improving data efficiency. The original STP recipe samples random token sub-spans, which is compatible with the base large language model (LLM) training architecture. Inspired by STP, we are interested to investigate whether the sampling position can further enhance the semantic structure of multi-step reasoning, and hence affect its geometric impact. We applied STP at consecutive semantic reasoning step boundaries and achieved 168× more accurate multi-step latent prediction than frozen baselines on ProcessBench (3,400 samples), compared to only 4× for the random-token STP. Probing the latent manifold with a learned non-linear predictor reveals that STP-shaped trajectories are smooth curves, not straight lines: a 3-layer MLP reduces prediction error by a further 3–12× over linear extrapolation on step-boundary models. Removing the language modeling loss yields trajectories that are 2× more MLP-predictable than the combined loss, revealing a tradeoff between generation quality and geometric purity. Our results identify sampling position as the critical variable in geometric regularization and establish multi-step latent prediction MSE as a new evaluation metric for this class of methods.
1
Introduction
Large Language models (LLMs) reason by generating step-by-step solutions, producing many tokens per reasoning step. Each token requires a full autoregressive forward pass, making multi-step reasoning computationally expensive. A growing body of work explores latent reasoning where computation is performed in the model’s continuous hidden-state space rather than decoding discrete tokens at every step [Hao et al., 2024, PLaT, 2026]. For latent reasoning to be feasible, the model’s hidden-state trajectory must be predictable, which is given the current reasoning state, future states must be forecastable with reasonable accuracy. Semantic Tube Prediction (STP) [Huang et al., 2026] provides a theoretical and practical framework for this. Building on the Geodesic Hypothesis, which insists that error-free token sequences trace locally linear geodesics on a smooth semantic manifold, STP adds a supplementary cosine-displacement loss during fine-tuning that forces consecutive hidden-state displacement vectors to be parallel. The resulting “Semantic Tube” confines trajectories to a tubular neighborhood of the geodesic, improving signal-to-noise ratio and enabling 16× data efficiency gains. STP samples its loss at random token sub-spans, where three indices s < r < t are drawn uniformly from the token sequence. While this method is elegant, we would like to explore if the semantic structure of the reasoning process, which is the natural boundaries where the model transitions from one reasoning step to the next, can contribute to the geometric smoothness of the method. We hypothesize that aligning the geometric regularization with the semantic structure of reasoning at step boundaries rather than at arbitrary token positions will produce more predictable trajectories. We test whether step-boundary STP creates the latent-space conditions that future JEPA [LeCun, 2022]
A PREPRINT - A PRIL 21, 2026
or COCONUT-style systems would need, as measured via post-hoc multi-step prediction accuracy of both linear and learned predictors. Our key contributions. 1. Sampling position is the critical variable. Applying STP at consecutive step boundaries achieves 168× prediction improvement vs. 4× for random-token STP—a 40× gap from where the loss is sampled, not what the loss computes (§5.1). 2. Multi-step latent prediction MSE as evaluation metric. We introduce and validate this metric with decoding fidelity tests: predicted embeddings decode to the correct next token 93.4% of the time (§5.4). 3. Trajectories are smooth curves, not straight lines. A learned MLP predictor reduces error by 3–12× over linear on STP-shaped models but finds no systematic structure on baselines, revealing three distinct manifold regimes (§6.1). 4. LNTP creates a tradeoff. Removing the language modeling loss yields 2× better MLP prediction at the cost of 3.7 pp GSM8K accuracy—generation quality vs. geometric purity (§5.1, §6.1). 5. Negative finding. Geometric smoothness does not encode step correctness (AUC ≈ 0.5; §A.2): the Semantic Tube captures organized thinking, not correct thinking.
2
Background
2.1
Semantic Tube Prediction
Huang et al. [2026] propose the Geodesic Hypothesis, which describes that token sequences generated by LLMs trace geodesics on a smooth semantic manifold, and deviations from these geodesics represent noise. They formalize this as the STP loss: LSTP = Es<r<t [1 − cos(hr − hs , ht − hr )] (1) where hs , hr , ht are hidden states at randomly sampled token positions. Minimizing LSTP forces consecutive displacement vectors to be parallel, confining trajectories to a tube around the geodesic. Combined with the standard next-token prediction loss, the training objective is L = LNTP + λ · LSTP . With the predictability of a geometric constraint, STP is able to dramatically reduce the amount of data required for training. Finetuning with 16× less training data across multiple model families [Huang et al., 2026] resulted in comparable results with full-data training. 2.2
Joint Embedding Predictive Architectures
We draw motivation from the JEPA paradigm [LeCun, 2022], which advocates predicting in latent space rather than reconstructing inputs. I-JEPA [Assran et al., 2023] and V-JEPA [Bardes et al., 2024] demonstrate this for images and video; LLM-JEPA [Huang et al., 2025] extends it to language. Interestinly, STP was interestingly introduced to solve the two-view problem of LLM-JEPA, which requires the text data to be paired with the code form of the same information. We do not implement a JEPA architecture as our training lacks target encoder, learned predictor module, and continuous embedding feedback loop. Instead, we test the preconditions that the latent space supports accurate multi-step prediction. 2.3
Geometric Analysis of LLM Hidden States
Several works analyze the geometry of LLM hidden states diagnostically. Wang et al. [2025] measure curvature across layers; Zhou et al. [2026] apply Menger curvature to reasoning trajectories; Herrmann et al. [2025] introduce PHi loss and show that hidden-state unpredictability correlates with correct solutions on difficult problems. Jiang et al. [2026] decompose reasoning traces into Progress (displacement magnitude) and Stability (curvature) at the token level. Sun et al. [2026] extract activations at step markers and find that correct and incorrect solutions diverge geometrically at late reasoning steps. Carson & Reisizadeh [2025] model reasoning trajectories as drift-diffusion systems with regime switching. Damirchi et al. [2026] find that smooth layer-wise trajectories correlate with valid reasoning. 2.4
Latent Reasoning
Latent reasoning to reduce or avoid token-wise decoding has been explored in several recent works. COCONUT [Hao et al., 2024] feeds the last hidden state back as the next input embedding, enabling latent iteration without token 2
A PREPRINT - A PRIL 21, 2026
Figure 1: Semantic Step Prediction concept. The token-level trajectory (pink) oscillates around the step-level geodesic (green). Step boundary positions (numbered 1, 2, 3) define the trajectory z = (z0 , z1 , . . . , zK ) on which the STP loss enforces consecutive displacement parallelism, confining the step-level path to a smooth tube. Table 1: Six-model experimental grid. Each model differs in loss composition and/or STP sampling strategy. model
LNTP
LSTP
STP sampling
B1 B2 C A2 A A1
— ✓ ✓ ✓ ✓ —
— — ✓ ✓ ✓ ✓
— (frozen pretrained) — (vanilla LM fine-tuning) Random tokens (original STP) Random step boundaries Consecutive step boundaries Consecutive step boundaries
decoding. PLaT [PLaT, 2026] decouples reasoning from verbalization entirely. CoLaR [Tao et al., 2025] trains a latent head with a next-compressed-embedding objective at random token positions. LightThinker [Zhang et al., 2025] compresses intermediate steps into gist tokens at step boundaries. Teoh et al. [2025] train a next-latent prediction head with Smooth L1 loss at every token, supporting multi-step rollouts for world modeling. At inference time, The Geometric Reasoner [Zhuang et al., 2026] scores candidates via bumpiness penalties at chunk boundaries, and STEP [Sun et al., 2026b] trains a classifier on step-boundary hidden states to detect reasoning errors.
3
Method
3.1
STP at Semantic Step Boundaries
We insert a special <|step|> delimiter token between reasoning steps in the training data (Figure 1), yielding sequences of the form: [question] <|step|> [step1 ] <|step|> [step2 ] <|step|> · · · <|step|> [stepK ] <|step|>
(2)
The hidden states at <|step|> positions form a trajectory z = (z0 , z1 , . . . , zK ). We compute the STP loss on consecutive triples: K−1 X 1 (zk − zk−1 ) · (zk+1 − zk ) Lstep = 1 − (3) STP K −1 ∥zk − zk−1 ∥ · ∥zk+1 − zk ∥ + ϵ k=1
This is mathematically identical to STP’s loss but differs in where the indices are sampled. We sample at semantically meaningful step boundaries rather than at random token positions. The combined objective is L = LNTP + β · Lstep STP with β = 1. Fine-tuning uses LoRA [Hu et al., 2022] with rank 16 on q, k, v, o projections (∼4.4M trainable parameters on a 1.5B base model). 3
A PREPRINT - A PRIL 21, 2026
3.2
Experimental Grid
This grid (Table 1) isolates three factors: (i) token-level vs. step-level sampling (C vs. A2), (ii) random vs. consecutive at the step level (A2 vs. A), and (iii) necessity of LNTP (A vs. A1). 3.3
Multi-Step Prediction MSE
For each trajectory (z0 , z1 , . . . , zK ) and each valid position k ≥ 1, we predict m steps ahead via linear extrapolation: ẑk+m = zk + m · (zk − zk−1 )
(4)
and compute the normalized prediction error: MSEm =
1 N
X (k,sample)
∥ẑk+m − zk+m ∥2 ∥zk+m ∥2
(5)
MSEm ≈ 0 means the trajectory is perfectly linear; MSEm ≈ 1 means the prediction error equals the signal magnitude. 3.4
Trajectory Smoothness Scores
We report the cosine score (matching the training objective): cos_scorek = 1 −
(zk − zk−1 ) · (zk+1 − zk ) ∈ [0, 2] ∥zk − zk−1 ∥ · ∥zk+1 − zk ∥
(6)
and the perpendicular score (geometric interpretation): perp_scorek =
∥dk − (dk · ŝk )ŝk ∥ = sin(θk ) ∈ [0, 1] ∥dk ∥
(7)
where dk = zk − zk−1 and sk = zk+1 − zk−1 . The two scores are monotonically related and produce identical model orderings (Spearman ρ = 1.0).
4
Experiments
4.1
Setup
models. Qwen2.5-Math-1.5B (math-specialized), Qwen2.5-1.5B (general-purpose), Llama-3.2-1B (different architecture, 2048-dim hidden states vs. 1536). All fine-tuned via LoRA with rank 16. Training data. 6,132 MATH competition problems and solutions split at paragraph boundaries with <|step|> delimiters. 3 epochs (∼1,150 optimizer steps per model). Evaluation data. ProcessBench [Zheng et al., 2024]: 3,400 step-by-step mathematical solutions across four difficulty levels with human-annotated error positions.
5
Results
5.1
Prediction MSE
Sampling at consecutive step boundaries (model A) achieves 168× more accurate prediction than the frozen baseline at m=1, while the original random-token recipe (model C) achieves only 4× (Table 2, Figure 2). Model A’s prediction at m=3 (MSE 0.093) is more accurate than B1’s prediction at m=1 (MSE 0.955). The role of LNTP sees to be a tradeoff and not a redundancy. Model A1 (Lstep STP only) tests whether LNTP can be dropped. Under linear extrapolation, A1 matches A within 12%. Under a learned MLP predictor, A1 outperforms A by 47–57% across all skip distances (§6.1). The choice is a tradeoff between generation quality (LNTP : +3.7% GSM8K) and geometric predictability where LSTP alone: 2× better MLP prediction. 4
A PREPRINT - A PRIL 21, 2026
Table 2: Six-model ablation on ProcessBench with Qwen2.5-Math-1.5B. Mean cosine score (↓, matches training objective), mean perpendicular score (↓), and multi-step prediction MSE (↓). Improvement factor computed at m=1 relative to B1. model B1 (frozen) B2 (LM only) C (rand-tok STP) A2 (rand-step STP) A (consec-step STP) A1 (STP only)
Mean cos ↓
Mean perp ↓
MSE1 ↓
MSE3 ↓
× vs B1
1.269 1.288 0.691 0.018 0.014 0.005
0.772 0.822 0.567 0.075 0.081 0.047
0.955 0.600 0.226 0.010 0.006 0.006
5.688 2.797 0.836 0.155 0.093 0.116
1× 1.6× 4× 96× 168× 152×
Figure 2: Multi-step prediction MSE (solid, left axis, log scale) and prediction accuracy (dashed, right axis) vs. skip distance m. Step-boundary models (A, A1, A2) achieve orders-of-magnitude better prediction than frozen (B1) and random-token STP (C). Model A remains usable (MSE < 1) up to m=8. 5.2
Cross-model and Cross-Dataset Validation
To investigate if geometric representation finetuning is transferable to non-math models, we similarly finetuned two models: Qwen2.5-1.5B, which is a direct comparison to Qwen2.5-Math-1.5B, as well as Llama-3.2-1B, which represents a different model architecture with different dimensions. From the results, we observed that the effect transfers across architectures, hidden dimensions, and pretraining regimes (Table 3, Figure 3a). Llama-3.2-1B achieves the strongest improvement (335×) despite having no math-specific pretraining. The improvement also holds across difficulty levels (Figure 3b) on four ProcessBench splits, GSM8K (165×), MATH (145×), OlympiadBench (183×), and Omni-MATH (180×). The geometric shaping is consistent and slightly stronger on harder problems, thereby ruling out the concern that STP overfits to easy problem structure. 5.3
Task Accuracy: No Degradation
Overall, all models perform somewhat equally on accuracy, confirming that step-boundary STP does not degrade the model’s reasoning ability. Model A achieves accuracy identical to vanilla fine-tuning (B2) while simultaneously achieving 168× better latent prediction MSE (Table 4). Random-token STP is the only variant that showed a decrease in accuracy relative to the frozen baseline. This suggests that the semantic structure of reasoning prevents interference 5
A PREPRINT - A PRIL 21, 2026
Table 3: B1 vs. A across three model families on ProcessBench. Accuracy = 1 − model Qwen2.5-Math-1.5B Llama-3.2-1B Qwen2.5-1.5B (general)
√
MSE.
B1 MSE1
A MSE1
Improvement
A acc. m=1
0.955 0.503 0.678
0.006 0.002 0.018
168× 335× 39×
92.5% 96.1% 86.8%
Figure 3: Robustness of the geometric improvement. (a) Cross-model: B1 vs. A prediction MSE at m=1 across three model families (log scale). The improvement ranges from 39× (Qwen2.5-1.5B) to 335× (Llama-3.2-1B). (b) Crossdataset: B1 vs. A on four ProcessBench splits of increasing difficulty. The improvement is consistent (145×–183×) and slightly stronger on harder datasets.
with the language model objective. The accuracy and MSE tests in fact reveal that two losses contribute orthogonally. LNTP seem to impact accuracy improvement while LSTP brings in geometric alignment to shape how models think through latent space.
5.4
Decoding Validation
The decoding validation (Table 5) experiment shows that two hidden-state vectors can be close in Euclidean distance yet differ in the specific subspace that the LM head uses for next-token prediction. The information relevant to decoding occupies a lower-dimensional manifold within the full 1536-dimensional hidden-state space. To test whether our predicted embeddings are functionally equivalent and not merely geometrically close, we pass both the predicted ẑk+m and the actual zk+m through the model’s final RMSNorm and LM head, producing two probability distributions over the vocabulary. At m=1, the distributions agree on the top-1 token 93.4% of the time (the frozen baseline B1 had 19.1%), and with a KL divergence of just 0.019 nats (vs. 4.581 for B1). The step retrieval metric provides a complementary view: the predicted embedding’s nearest neighbor among all step-boundary embeddings (excluding the source points zk and zk−1 ) is the correct target step 92.2% of the time, confirming that ẑk+m lands at the right reasoning state. An instructive contrast emerges between model A and B1 at m=1: B1 achieves relatively high step retrieval (70.2%) but low token agreement (19.1%), meaning the frozen model’s predicted embedding is geometrically near the correct step but encodes different next-token information—the geometric position and functional content are misaligned. STP training aligns both: model A’s 92.2% retrieval and 93.4% token agreement show that geometric proximity and functional equivalence coincide after geometric shaping. At longer horizons, model A’s token agreement degrades gracefully—71.1% at m=2 and 34.1% at m=3—while B1 collapses to near-chance (10.1% and 8.3%). Notably, model A at m=2 (71.1%) still exceeds B1 at m=1 (19.1%), mirroring the MSE result and confirming that the prediction advantage extends well beyond one-step forecasting. 6
A PREPRINT - A PRIL 21, 2026
Table 4: GSM8K accuracy (greedy decoding). model
Accuracy
n
MSE1
Interpretation
66.9% 68.8% 69.3% 73.0% 73.0%
882/1319 907/1319 914/1319 963/1319 963/1319
0.226 0.955 0.006 0.600 0.006
Hurts accuracy Pretrained baseline No LNTP : preserves acc. Vanilla fine-tuning Matches B2, 168× better MSE
C (rand-tok STP) B1 (frozen) A1 (STP only) B2 (LM only) A (consec-step)
Table 5: Decoding from predicted embeddings (200 ProcessBench samples).
6
Analysis
6.1
Manifold Structure
Metric
m
model A
B1 (frozen)
Step retrieval acc. ↑
1 2 3
92.2% 55.7% 47.5%
70.2% 14.9% 10.1%
Top-1 token agreement ↑
1 2 3
93.4% 71.1% 34.1%
19.1% 10.1% 8.3%
Top-5 Jaccard ↑ KL divergence ↓
1 1
0.878 0.019
0.207 4.581
We train a residual multilayer perceptron (MLP) post-hoc predictor probe, ẑk+m = zk + m(zk − zk−1 ) + gϕ (zk , zk−1 ), where gϕ is a 3-layer MLP with zero-initialized last layer (architecture details in Appendix A.3). The zero initialization ensures that the MLP starts as an exact copy of the linear predictor and can only improve upon it. Any reduction in MSE represents a genuine non-linear structure that linear extrapolation missed. The MLP probe is trained on 80% of ProcessBench step-boundary pairs and evaluated on the held-out 20%. The probe serves purely as a measurement instrument for the latent geometry of the finetuned model without changing the models. To characterize model’s manifold structure, we compared the MLP against a trivial linear probe on the same task of predicting the hidden state at step k + m when given hidden state of step k and step k − 1. The ratio of MLP MSE to and liner MSE reveals how much learnable strcture exists beyond what a striaght-line approximation captures. Crucially, a ratio that decreases with m indicates smooth, compounding carvature, which is the signature of a coherent curving manifold rather than random noise. (Table 6, Figure 4) Models B1, B2 and C show flat or weakly decreasing ratios, while models with step-boundary finetuning, such as A, A1, and A2, show strongly decreasing ratios. Model A goes from 0.285 to 0.085 (a 70% drop over m=1 to m=3), and A1′ goes from 0.137 to 0.029 (a 79% drop). Decreasing ratio with m is the signature of a smoothly curving path. curvature compounds quadratically with horizon, so longer extrapolations expose proportionally more non-linear structure for the MLP to learn. A1 (pure LSTP, no language modeling loss) has the steepest decrease and lowest ratios of any cell, producing the cleanest geometric structure. It’s MLP at m=3 achieves MSE 0.003407 — a 1,098× improvement over B1’s linear prediction at m=1 (MSE 0.954). With a learned predictor and pure geometric training, we can forecast three reasoning steps ahead more accurately than a frozen pretrained model can forecast one. 6.2
Why Sampling Position Matters
The 40× gap between model A and model C arises from optimization efficiency: step-level STP concentrates all gradient signal on the K step-boundary positions that define the trajectory we measure, while token-level STP distributes q triples across T token positions with O(1/n) dilution per cross-step triple. The effective gradient ratio is ∼ n(K−1)/q ≈ 100× for typical values. Both models use identical LoRA parameters, training data, epochs, and compute. Hence, likely the 40× MSE gap is entirely attributable to sampling position. The MLP/linear ratio decreasing with m (Table 6) follows from curvature bounds. The linear prediction error scales as O(m2 ϵ) while a curved predictor achieves O(m2 ϵ2 ), yielding ratio O(ϵ). Model A1 (ϵ = 0.005) has lower ratios than model A (ϵ = 0.014) at every m, consistent with the O(ϵ) prediction. The 2× MLP gap between A1 and A follows 7
A PREPRINT - A PRIL 21, 2026
Table 6: MLP vs. linear extrapolation on ProcessBench (all six models). Ratio = MLP MSE / Linear MSE; decreasing ratio with m indicates smooth curvature. model
m
Linear MSE
MLP MSE
Ratio
Trend
B1 (frozen)
1 2 3
0.954 2.770 5.732
0.385 0.975 2.004
0.403 0.352 0.350
weak ↓
B2 (LM only)
1 2 3
0.594 1.482 2.785
0.270 0.635 1.214
0.454 0.428 0.436
flat
C (rand-tok STP)
1 2 3
0.225 0.488 0.825
0.105 0.173 0.266
0.466 0.354 0.322
moderate ↓
A2 (rand-step)
1 2 3
0.010 0.056 0.155
0.00235 0.00554 0.00952
0.233 0.099 0.061
strong ↓
A (consec-step)
1 2 3
0.00568 0.03223 0.09240
0.00162 0.00403 0.00785
0.285 0.125 0.085
strong ↓
A1 (STP only)
1 2 3
0.00635 0.03952 0.11722
0.000868 0.002141 0.003407
0.137 0.054 0.029
strongest ↓
Figure 4: MLP/linear MSE ratio vs. skip distance m for all six models. Flat ratio (B1, B2, C) indicates noise without systematic curvature. Decreasing ratio (A2, A, A1) is the signature of a smoothly curving manifold where curvature compounds with m. A1 (STP only) has the steepest decrease. from gradient interference: LNTP adds perturbations orthogonal to the geometric direction that linear tolerates but the MLP cannot fit through. Our results establish a clear hierarchy of latent-space support for multi-step prediction. Frozen pretrained representations (model B1) that are based on token-based sampling are not suitable for prediction. The linear MSE ≈ 1.0 at m=1, means the prediction error equals the signal magnitude, faired worse compared to a simple learned MLP ( reduced to MSE ≈ 0.39). This confirmed that the frozen trajectory has no coherent geometric structure for a neural predictor to exploit. Vanilla LM fine-tuning (model B2) reduces linear MSE to 0.60, but the MLP/linear ratio remained flat at ∼0.45 across all skip distances. Random-token STP (model C) provided a further reduction to linear MSE at ≈ 0.23, however, its MLP/linear ratio (0.466) is the highest of any cell, suggesting the residual structure is less learnable than the frozen baseline. This may mean that Random-token STP denoises but it does not create a manifold. Only step-boundary STP showed capability in predictable trajectories. Model A achieved linear MSE of ≈ 0.006 with a strongly decreasing 8
A PREPRINT - A PRIL 21, 2026
MLP/linear ratio (0.285 → 0.085). The gap between Cell C (MSE 0.226) and Cell A (MSE 0.006) is 40×, arising entirely from where the loss is sampled, not from what the loss computes. Model A1, without the LNTP pushed further to MLP MSE ≈ 0.0009, about a 1,098× improvement over B1’s linear baseline.
7
Limitations and Future Work
Limitations. (1) We characterize prediction potential but do not build a latent reasoning system; utilizing the geometric structure for generation would require COCONUT-style architectural changes [Hao et al., 2024]. (2) Step boundaries are manually marked via <|step|> at paragraph breaks; some MATH solutions have noisy boundaries (e.g., individual lines of a LATEX align* block). Despite this, the 168× improvement is robust. (3) Evaluation is on math reasoning only. (4) MSE at m=1 is closely related to the STP training objective; MSE at m ≥ 2 and the MLP predictor provide independent validation. Future work. (1) Combining step-boundary STP with COCONUT curriculum training—STP straightens the trajectory that COCONUT iterates along. (2) Data efficiency evaluation. (3) Automatic step boundary detection. (4) Cross-domain evaluation.
8
Conclusion
In this paper, we investigates the effect of STP’s geometric loss and step sampling on shaping language model’s reasoning trajectories. We showed that changing STP from random token sub-spans to semantic step boundaries transforms hidden-state trajectories from unpredictable walks (linear MSE ≈ 1.0) into smoothly curving manifolds (linear MSE ≈ 0.006; MLP MSE ≈ 0.0009). And also achieved multi-step latent prediction MSE 168× to 1,098× improvement validated across three model families and four datasets.
Code Availability The code used to reproduce all experiments and figures in this work is publicly available at hhttps://github.com/ YYDreamzure/SSP/. The repository includes training scripts, evaluation pipelines, and pre-trained model checkpoints. Any additional data or materials required to reproduce the results reported in this paper are available from the corresponding author upon reasonable request.
References M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. Rabbat, Y. LeCun, and N. Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In CVPR, 2023. A. Bardes, Q. Garrido, J. Ponce, X. Chen, M. Rabbat, Y. LeCun, M. Assran, and N. Ballas. V-JEPA: Latent video prediction for visual representation learning. arXiv:2404.16930, 2024. H. Damirchi et al. Geometric analysis of transformer layer trajectories in reasoning. 2026. S. Hao, B. Sukhbaatar, D. Su, X. Li, Z. Hu, J. Weston, and Y. Tian. Training large language models to reason in a continuous latent space. arXiv:2412.06769, 2024. V. Herrmann et al. PHi: Hidden-state prediction loss for evaluating reasoning quality. In ICML, 2025. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022. Y. Huang et al. LLM-JEPA: Joint embedding prediction for language models. arXiv:2509.14252, 2025. Y. Huang et al. Semantic tube prediction. arXiv:2602.22617, 2026. Y. LeCun. A path towards autonomous machine intelligence. OpenReview, 2022. X. Wang et al. Latent cosine of expertise: Geometric analysis across transformer layers. arXiv:2410.13640, 2025. K. Zhang et al. LightThinker: Thinking step-by-step compression for LLM reasoning. 2025. C. Zheng et al. ProcessBench: Identifying process errors in mathematical reasoning. arXiv:2412.06559, 2024. Y. Zhou et al. Geometry of reasoning in large language models. arXiv:2510.09782, 2026. CoLaR. Compressed latent replay for efficient chain-of-thought. 2025. J. Wang et al. Latent Chain-of-Thought as Planning: Decoupling Reasoning from Verbalization 2026. X. Sun, Y. Dong, et al. LLM reasoning as trajectories: Representation, verification, and steering. arXiv:2604.05655, 2026.
9
A PREPRINT - A PRIL 21, 2026
Table 7: ProcessBench error detection. Perpendicular score does not encode correctness. model B1 (frozen) A (step-STP)
Binary AUC
Localization acc.
Random baseline
0.509 0.564
6.6% 4.8%
∼15–20% ∼15–20%
T. Teoh et al. NextLat: Next-latent prediction transformers for multi-step world modeling and reasoning. arXiv:2511.05963, 2025. Y. Jiang et al. TRACED: Beyond scalars—progress and stability in reasoning trajectories. arXiv:2603.10384, 2026. J. Carson and A. Reisizadeh. Statistical physics of language model reasoning. In ICML, 2025. arXiv:2506.04374. S. Zhuang et al. The Geometric Reasoner: Training-free geometric reasoning via smoothness and diversity penalties. arXiv:2601.18832, 2026. S. Sun et al. STEP: Hidden states as early signals for reasoning quality. arXiv:2601.09093, 2026. L. Tao et al. CoLaR: Compressed latent reasoning via next-embedding prediction. In NeurIPS, 2025. arXiv:2505.16552.
A
Supplementary Material
A.1
Perpendicular Score Geometry
Figure 5: Perpendicular score computation. Left: interior positions use the two-sided secant refrk = zk+1 − zk−1 as the reference direction. Right: the last position uses the one-sided secant refrK = zK−1 − zK−2 (the arriving direction). The perpendicular score is sin θk ∈ [0, 1] where θk is the angle between the displacement dk and its projection onto the reference direction. Binary detection AUC is indistinguishable from random (Table 7). The Semantic Tube captures the process of organized reasoning—a structured, predictable flow through latent space—not the outcome of correct reasoning. This is consistent with the PHi loss literature [Herrmann et al., 2025]: hidden-state unpredictability correlates with correct solutions on difficult problems. This finding resolves an apparent conflict with Damirchi et al. [2026], who report that smooth trajectories correlate with valid reasoning. The two analyses operate on different axes: smoothness across layers (vertical) may reflect well-conditioned computation, while smoothness across steps (horizontal) reflects predictable reasoning content—which need not correlate with correctness. A.2
Smoothness ̸= Correctness
A.3
Skip-Predictor Architectures
We use two post-hoc predictors to probe the geometric structure of step-boundary trajectories. Both take as input the hidden states at positions k and k−1 and predict the hidden state at position k+m, without modifying the underlying model. Linear predictor (zero parameters).
Extrapolates along the current displacement direction: ẑk+m = zk + m · (zk − zk−1 )
(8)
This assumes the trajectory is locally linear. If consecutive displacements are perfectly parallel (Lstep STP = 0), linear prediction is exact. Any prediction error reflects curvature or noise in the trajectory. 10
A PREPRINT - A PRIL 21, 2026
MLP predictor (learned, ∼6M parameters).
Adds a learned non-linear correction on top of the linear baseline:
ẑk+m = zk + m · (zk − zk−1 ) + gϕ (zk , zk−1 ) | {z } | {z } linear baseline
(9)
learned correction
where gϕ is a 3-layer MLP (input: 2D → 2048 → 2048 → D, GELU activations). The last layer is zero-initialized, so at epoch 0 the MLP prediction equals the linear prediction exactly—it can only improve, never hurt. Training minimizes MSE on 80% of ProcessBench step-boundary pairs; evaluation uses the held-out 20%. linear The ratio MSEMLP characterizes the manifold structure: ratio ≈ 1 means the trajectory is maximally linear m /MSEm (MLP adds nothing); ratio ≪ 1 means systematic non-linear structure exists that the MLP exploits. A ratio that decreases with m indicates smooth curvature that compounds over longer horizons—the signature of a curving tube rather than a straight line or random walk.
11