arXiv CSopen access

Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers

Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers

Sajad Movahedi∗ 1

Vera Milovanović∗ 1,2

Shlomo Libo Feigin∗ 1,2

Thomas Hofmann2

Valentina Boeva† 2, 3, 4

T. Konstantin Rusch† 1,5

Antonio Orvieto† 1

ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems, Tübingen AI Center 2 ETH Zurich 3 Swiss Institute of Bioinformatics 4 Université Paris Cité 5 Liquid AI {sajad.movahedi, vera.milovanovic}@tue.ellis.eu

Abstract Looped architectures provide an inductive bias toward learning step-by-step procedures for tasks that require compositional reasoning. The depth of effective layers reached by looping determines the quality of the solution these models find. Similar to deep architectures, looped architectures are prone to a signal propagation problem induced by depth as the halting decision is postponed. In this paper, we address the signal propagation issue by using pre-norm layers and residual scaling. Building on these architectural modifications, we propose FPRM: a Transformer-based Fixed-Point Reasoning Model that uses fixed-point convergence as an end-to-end halting mechanism in a looped architecture. We show that fixed-point halting allows FPRM to adapt its compute to the difficulty of the task. FPRM proves effective on common reasoning benchmarks, namely Sudoku, Maze, state-tracking and ARC-AGI. The implementation can be found here.

1

Introduction

Reasoning in neural networks has increasingly been framed as a problem of scaling test-time compute: a model should be able to spend more computation on inputs it finds harder (OpenAI, 2024; Snell et al., 2024). However, doing so requires two ingredients. (1) flexibility: the possibility of spending a variable amount of compute on the problem. Once the model is flexible, the next step is (2) adaptivity: a way to scale the compute spent on the problem; i.e., when to halt the computation. The standard way to achieve both is through a Chainof-Thought (CoT) mechanism (Wei et al., 2022). With CoT, the model scales compute through verbalization, and makes halting decisions based on predicting a specialized halting token. However, this emerging behavior requires a special training regime and hand-crafted reasoning traces (Guo et al., 2025). This makes the method complex and undermines the desirable property of endto-end training. ∗ Equal contribution. † Equal advising.

Preprint.

Task difficulty Easy

Medium

Hard

100

80

Test accuracy (%)

arXiv:2606.18206v1 [cs.AI] 16 Jun 2026

1

Alexander Theus∗ 1, 2

60 +10 accuracy points ∼ 1k (27%) layers less compute

40

20 Halting 0

101

102

FPRM (ours)

TRM

103

104

Inference compute (effective layers)

Figure 1: Signal propagation and adaptivity, FPRM vs. TRM: Sudoku-Extreme performance as a function of compute across difficulty. Despite being nonhierarchical, FPRM scales better, while correctly detecting the accuracy plateaus by using fixed-points for halting.

post-norm with RoPE + NoPE + conv + pre-norm + residual scaling diverged

256 128

Activation norm kzk2

Max seq. length (test acc. > 90%)

105

512

10

post-norm pre-norm pre-norm + residual scaling

+

Layer

4

SwiGLU

103

64 32

Norm +

102

16

+

ShortConv

101

8 8

16

32

64

128

256

512

Number of recurrent iterations

(a) Trainability vs. context length

Attention

2

8

32

128

No

512

Number of recurrent iterations

Stop?

Yes

Norm

(b) Activation norm at init.

Figure 2: The blessing and the curse of depth in Looped Transformers. Increasing the number of effective layers can unlock expressivity, but also creates a stability challenge: prenorm models without residual scaling can diverge in activation norm, while post-norm models may struggle to utilize the signal.

Figure 3: FPRM architecture. Our fixed-point Looped Transformer uses pre-norm and residual scaling for improved signal propagation.

An alternative approach to end-to-end training for reasoning models is emerging in the form of looped architectures (Dehghani et al., 2019; Bansal et al., 2022; Saunshi et al., 2025; Hao et al., 2024). Whereas in CoT, the compute scales along the sequence dimension, in looped architectures it scales along the depth dimension (i): zi+1 = fθ (zi ; x), (1) where fθ (z; x) is a neural network, x is the input, and zi is the ith latent representation. Thus, compute can be increased at test-time by looping in depth, naturally introducing flexibility from the architecture. Looped models have been shown to have an inductive bias towards learning algorithms (Yang et al., 2024; Fan et al., 2025) and have achieved remarkable success on reasoning benchmarks. For example, Hierarchical Reasoning Model (HRM) (Wang et al., 2025) and Tiny Reasoning Model (TRM) (Jolicoeur-Martineau, 2025) incorporate a hierarchical structure into the looping process, proving effective in solving puzzle tasks such as sudoku, maze, and ARC-AGI (Chollet et al., 2024). The halting decision (i.e., deciding when to stop iterating), however, is no longer trivial in looped models. Most approaches either fix or randomly sample the number of loops (Jeddi et al., 2026; Zhu et al., 2025; Geiping et al., 2026; Saunshi et al., 2025; Botev et al., 2024; Bansal et al., 2022), which eliminates adaptivity, or use external modules trained to make halting decisions (Wang et al., 2025; Jolicoeur-Martineau, 2025; Dehghani et al., 2019). The latter is associated with separate Adaptive Computation Time (ACT) networks (Graves, 2016), which introduce optimization challenges, as they require a continuous relaxation of a discrete objective. As a result, ACT can fail to provide adaptivity, which we demonstrate in the case of HRM and TRM (Figures 4, 5). We mitigate this issue by introducing a different halting mechanism: let the model loop until its hidden state converges to a fixed-point, and use the convergence itself as the halting signal. Unlike ACT, the fixed-point halting mechanism requires no external module and lets the model spend as much compute as a given input demands. A further challenge arises as the number of loops in the model increases. This is desirable for harder tasks, but unrolling the recursions yields a deep effective layer. As a result, looped architectures suffer from the same signal propagation issue as deep non-looped networks. To this end, we adapt architectural techniques originally developed to optimize traditional transformers at deep effective layers, carefully modifying them for the Looped Transformer setting. Our key intuition is that Looped Transformers are, in part, very deep transformers. Viewed this way, one design choice is surprising: looped models commonly use post-norm (Dehghani et al., 2019; Geiping et al., 2026; Wang et al., 2025; Jolicoeur-Martineau, 2025). Deep non-looped architectures, by contrast, prefer pre-norm, since post-norm causes unstable training (Xiong et al., 2020; Noci et al., 2022). Yet in looped models, post-norm serves a specific purpose. It keeps the magnitude of activations bounded as the model iterates, preventing the hidden states from diverging (Labovich, 2026) (see Figure 2). This raises the question: can we switch the post-norm to pre-norm, while ensuring the activations stay bounded in a different way? If so, this could make the training of looped models stable at large depths. In this paper, we use residual scaling parameters to give an affirmative answer to this question. In Figure 1, we observe that fixed-point halting, together 2

with better signal propagation, allows FPRM, a non-hierarchical reasoning model, to outperform TRM at a lower cost on the Sudoku-Extreme benchmark. We provide a description of how the figure was made in Section D. We summarize our contributions as follows: 1. Successfully training a looped pre-norm Transformer. We modify a Transformer layer to be trainable over deeper effective layers by switching post-norm to pre-norm and adding residual scaling parameters. 2. Reaching a fixed-point as a halting mechanism. To enable stable training, we propose a theoretically motivated modification specific to fixed-point models to limit the oscillation around the fixed-point. 3. Proposing the framework FPRM: Fixed-Point Reasoning Model. We show that it outperforms the baselines on Sudoku-Extreme, Maze-Hard, ARC-AGI-1, and state-tracking benchmarks A5 and S5 among 7M parameter models. Notably, we achieve our results without the hierarchical structure of HRM and TRM. To the best of our knowledge, FPRM is the first Transformer-based reasoning model that exhibits adaptivity of compute to task difficulty (Figures 1,4).

2

Background and Related Works

Looped models. Looped architectures address the flexibility requirement of reasoning models by decoupling effective layer from parameter count, providing variable computation per input without scaling the number of parameters. This makes looping a natural inductive bias for tasks that can be solved by repeatedly applying local or compositional subroutines. Early examples include Neural GPUs (Kaiser and Sutskever, 2016) and Universal Transformers (Dehghani et al., 2019); more recent works show that Looped Transformers can improve length generalization, learn algorithmic structure, and approximate much deeper untied models on reasoning tasks (Yang et al., 2024; Fan et al., 2025; Saunshi et al., 2025; Kapl et al., 2026; Geiping et al., 2026; Giannou et al., 2023; Kohli et al., 2026). Recent recursive reasoning models such as HRM (Wang et al., 2025) and TRM (Jolicoeur-Martineau, 2025) instantiate this principle in compact architectures. Using small networks, these models have been able to outperform prominent LLM-based reasoning models. At the core of these methods is the idea of the need for a hierarchical looping mechanism, wherein the compute is distributed between a fast-looping component and a slow-looping component. URM (Gao et al., 2025) follows up on these works and shows that by using short-conv, performance can be further improved. However, as we describe in this paper, previous works leave two central design axes open to improvement: First, how should a looped model decide how many iterations to run on each input at test-time? FPRM addresses this through fixed-point halting. Second, how to make the model utilize its deep effective layer? FPRM does so by switching post-norm to pre-norm and adding residual scaling. Importantly, FPRM achieves better performance than TRM and HRM without using a hierarchical structure for the loops. Adaptive computation. Classical ACT methods answer the adaptivity question by learning an explicit halting rule, such as halting probabilities, per-token stopping decisions, or learned distributions over computation depth (Dehghani et al., 2019; Banino et al., 2021). More recent works extend this idea by picking an adaptive depth for each token (Bae et al., 2025; Song et al., 2026). Such mechanisms can, in principle, allocate more computation to harder instances, but they all add a separate learned decision process on top of the recurrent computation itself, and this process is hard to optimize because the halting decision is discrete rather than continuous. Moreover, analyses of HRM find that ACT does not always scale the inference compute with the actual difficulty of the input (Ren and Liu, 2026; Ge et al., 2025), which can more generally also be the case with CoT (Palod et al., 2025), a limitation we also observe and address by instead halting when the iteration reaches a fixed-point. Deep equilibrium and fixed-point models. An alternative is to use the convergence of the latent dynamics as the halting criterion. In this view, computation stops when consecutive iterates become sufficiently close ∥zi+1 − zi ∥ ≤ ϵ, which corresponds to convergence toward a fixed-point z⋆ = fθ (z⋆ ; x). This perspective is also supported by recent mechanistic analyses of looped language 3

models, which observe that recurrent trajectories often converge to fixed-points, suggesting that weight-tied looped models can naturally implement an implicit fixed-point computation (Blayney et al., 2026). This view is most extensively developed in Deep Equilibrium Models (Bai et al., 2019), which replace a finite stack of layers by the equilibrium point of a weight-tied transformation. However, DEQ models usually find the fixed-point via a quasi-Newton approach such as Broyden’s method (Broyden, 1965; Bai et al., 2019) or Anderson acceleration (Anderson, 1965; Geng and Kolter, 2023) rather than fixed-point iterations, which makes them different from looped models. Moreover, DEQ models are difficult to optimize (Bai et al., 2021; Anil et al., 2022; Jolicoeur-Martineau, 2025), which we address by our proposed architectural changes. Concurrent to our work, Attractor Models (Fein-Ashley and Rashidinejad, 2026) frame the iterations of TRM as a root-finding problem similar to DEQ, which they solve with Anderson acceleration. Additionally, they propose to optionally include a larger separate network to “guess” the initial latent. Another concurrent work is Equilibrium Reasoners (EqR) (Huang et al., 2026), which also adopts a fixed-point perspective and shows that it is possible to scale the compute not only depth-wise, but also by making multiple initial guesses at training and inference time, favoring attractors with a wide basin. Our contributions are largely orthogonal: we target the signal-propagation issues that limit trainable depth, replacing post-norm with pre-norm and residual scaling, and damping the iteration to suppress oscillation around the fixed-point. Moreover, in contrast to these works we do not use the hierarchical looping of TRM. Score-based methods. Diffusion Models (Ho et al., 2020; Song et al., 2021) and Energy-Based Models (EBMs) (LeCun et al., 2006) provide another way of iterative computation. An EBM learns a scalar energy function, the core idea being that the energy represents the negative log of the unnormalized density, and solutions correspond to low energy states. EBMs generate a prediction either by descending the energy landscape (Belanger and McCallum, 2016; Belanger et al., 2017) or by sampling with MCMC methods (Du and Mordatch, 2019). In comparison, diffusion models learn the score of the underlying distribution, which can be viewed as the gradient of a time-dependent energy function, thus the two are closely connected (Salimans and Ho, 2021; Du et al., 2023). While diffusion models typically integrate an SDE over a fixed time interval and in this sense are not adaptive, energy-based reasoning models (Du et al., 2022, 2024; Gladstone et al., 2025) have an adaptive halting mechanism. In these models, the halting criterion is convergence to a local minimum of the energy landscape. Compared to these methods, we model the generative process via fixed-point iterations of a learned operator, as opposed to descending through a learned energy function. The setting of learning the operator directly is important for the theoretical analysis of our architectural modifications in Section 3. Signal propagation in looped models. As unrolled looped architectures can be viewed as deep networks, they are exposed to the same signal-propagation difficulties that arise in very deep Transformers. In deep sequence models, increasing depth can make optimization harder and can prevent later layers from being effectively used, a phenomenon often discussed as the curse of depth (Dong et al., 2021; Noci et al., 2022; Sun et al., 2025). Pre-norm is the standard remedy for these issues, but in a looped setting it removes a property the architecture relies on: bounded activations (Figure 2). Bounded activations, important for stable training, are typically enforced through post-norm (Geiping et al., 2026; Wang et al., 2025; Jolicoeur-Martineau, 2025). Indeed, in our own experiments a pre-norm looped model without further modification diverges in activation norm as the iteration count grows, and fails to train at the large loop counts where this growth is most severe (Figure 2). Therefore, our FPRM combines pre-norm with residual scaling to recover bounded and stable dynamics while still allowing gradients and representations to propagate through many iterations.

3

Method

Looped architectures rely on bounded activations for stability: as the model loops, an unbounded layer can cause the hidden states to grow without limit (Figure 2). Current methods mostly employ post-norm to satisfy the boundedness condition (Jolicoeur-Martineau, 2025; Wang et al., 2025; Geiping et al., 2026). However, in fixed-depth models post-norm introduces a signal propagation issue, often associated with unstable training and restricted effective layer, as reported by Dong et al. (2021); Noci et al. (2022); Sun et al. (2025). In Section B, we provide evidence for trainability issues in a toy looped model with post-norm, despite stability induced by bounded activations. 4

Consequently, in this section we propose to (a) switch to pre-norm to recover trainability at higher depth, while preserving boundedness by (b) scaling the residual stream and the sub-layer outputs (the attention and feed-forward maps). Together, these modifications yield a layer that is both stable under looping and trainable at large depth. Additionally, we introduce (c) a fixed-point halting mechanism that decides the effective layer adaptively per input. Interestingly, we find that with these changes we can (d) remove the hierarchy common in recent Looped Transformers (Wang et al., 2025; Jolicoeur-Martineau, 2025), resulting in a much simpler model. We provide an overview of our proposed model in Figure 3. 3.1

Improving signal propagation with pre-norm

We start by introducing pre-norm and post-norm in a Transformer layer. A Transformer layer consists of two sub-layers (with fθℓℓ (.) denoting the ℓth ∈ {1, . . . , 2L} sub-layer): multi-head attention and a feed-forward network. We denote the Looped Transformer model as defined in Equation (1), consisting of multiple Transformer layers, by fθ (.). In the Transformer layer, the two sub-layers are interleaved with layer normalization (Norm). Two canonical placements of the normalization define two variants of the layer. The original post-norm formulation (Normpost ) (Vaswani et al., 2017) applies normalization after the residual addition with the residual stream (zℓ−1 ), while the pre-norm variant (Normpre ) (Xiong et al., 2020) applies normalization to the input of each sub-layer without modifying the residual stream:


zℓ = Normpost zℓ−1 + fθℓℓ Normpre zℓ−1 , ℓ = 1, . . . , 2L, where L is the number of layers (Transformer blocks). In fixed-depth models, both normalization placements have been linked to training issues at large depth: post-norm bounds activation magnitudes but induces a signal propagation problem (Kim et al., 2025; Noci et al., 2022), while pre-norm improves signal propagation but causes exponential growth in residual magnitude (Kim et al., 2025; Xiong et al., 2020). Therefore, while using pre-norm in a deep neural network is desirable from the signal propagation standpoint, it can introduce instability due to unbounded activations. Motivated by these observations, we investigate both normalization placements in Looped Transformers (Saunshi et al., 2025; Dehghani et al., 2019). In Figure 2a, we test the positive correlation between the effective layer and the expressivity (maximum sequence length with > 90% test accuracy) of a Looped Transformer for the state-tracking task A5 (Merrill et al., 2024). We observe that increasing the effective layer of a Looped Transformer with post-norm does not translate into improved expressivity, while a pre-norm variant diverges at larger depth. This divergence can be attributed to the exponential growth of the activations, apparent in Figure 2b. Therefore, to use pre-norm and improve signal propagation in deep looped models, we must first stabilize it, which we do in the following. 3.2

Recovering boundedness via residual scaling

While pre-norm mitigates trainability problems in looped architectures, it removes the necessary boundedness condition that motivated the use of post-norm in them. This effect can be observed in Figure 2b, where the activations of the pre-norm model grow with deeper effective layer, causing trainability issues observable in Figure 2a. Consequently, we propose to restore boundedness by introducing scaling parameters applied at two different scales: one over each sub-layer of the network, and one across the iterations. Layer-wise residual scaling. Within a single application of fθ (z; x), the residual stream and sub-layer output fθℓℓ (zℓ−1 ) are weighted by tied scalars (α1 , β1 ) shared across all L layers: zℓ = α1 zℓ−1 + β1 fθℓℓ Normpre zℓ−1

,

ℓ = 1, . . . , 2L.

(2)

Iteration-wise input mixing. Between consecutive applications of fθ (z; x), we re-inject the input x with tied scalars (α2 , β2 ) shared across all iterations (Bai et al., 2019): z0i+1 = α2 z2L i + β2 x.

5

(3)

The two scaling schemes are not independent. With an appropriate coupling between them, the resulting recurrence is bounded for any input, resulting in a stable looping (Orvieto et al., 2023). In the following statement, we formalize this claim: Theorem 1 (Boundedness of FPRM iterates). Consider the model defined by Equations 2 and 3, and assume each layer map satisfies ∥fθℓℓ (u)∥ ≤ cf for all ℓ and input u. Let 0 ≤ α1 , α2 < 1, and set β2 = 1 − α2 α12L ,

β1 =

β2 (1 − α1 ) . 1 − α12L

Then the fixed-point iterates {z0i }i≥0 from Equation (3) are bounded, and if z0i → z0∞ , then ∥z0∞ ∥ ≤ ∥x∥ + α2 cf . The proof is in Section A.1. Note that the boundedness condition of the sequence model in Theorem 1 is satisfied when using pre-norm, as shown by Kim et al. (2021). However, boundedness still does not guarantee that the looping to converge to a fixed-point, which we propose to utilize for adaptivity. In the following, we show that there exists a choice of α2 that satisfies this requirement. Consequently, as empirically shown by Bansal et al. (2022), the model may become locally contractive during training. Theorem 2 (Small α2 implies convergence). Let λf be the Lipschitz constant of the L-layer model defined by Equation (2), i.e. the map z0 7→ z2L . Then the looped step fθ ( · ; x) of Equations 2–3 is Lipschitz in its first argument with constant α2 λf . In particular, if α2 λf < 1, then fθ ( · ; x) is a contraction and the iteration zi+1 = fθ (zi ; x) converges to a unique fixed-point z⋆ = fθ (z∗ ; x) at a linear rate: i ∥fθ (zi ; x) − zi ∥ ≤ (α2 λf ) ∥fθ (z0 ; x) − z0 ∥ . The proof is in Section A.2. While a contractive map is needed for conver- Algorithm 1 Fixed-point optimizer FPO PT: one gence, an excessively contractive one severely damped step with patience-based decay limits expressivity (Bai et al., 2019; Anil et al., Require: initial damping η0 ; decay γ ∈ (0, 1); 2022). We avoid this by making α1 and α2 patience P learnable. In practice, we find that initializing 1: Internal state: η ← η0 , p ← P , r ⋆ ← ∞ the network to be more contractive by setting α2 to be small yields better performance (Table 2). 2: procedure S TEP (z, z̃) In Figure 9, we observe that after training, the 3: r ← ∥z − z̃∥∞ /(∥z̃∥∞ + ϵ) ▷ residual distribution of α values widens, with the median 4: z ← η z̃ + (1 − η) z ▷ damped update very close to the initial point. Intriguingly, these 5: if r < r⋆ then observations are also in line with the common so6: r⋆ ← r, p ← P ▷ progress: reset lutions to signal propagation and rank-collapse 7: else problems in deep neural networks (Noci et al., 8: p←p−1 2022; Sun et al., 2025), suggesting a connec9: if p ≤ 0 and r > τ then tion between rank-collapse and the signal prop10: η ← γ η, p ← P ▷ decay η agation issue in looped architectures. Together, 11: end if these modifications yield a pre-norm Looped 12: end if Transformer that maintains performance over 13: return z, r longer looping horizons before saturating, com14: end procedure pared to the post-norm variant (see Figure 2). 3.3

Oscillation around the fixed-point

So far, we have been able to establish that, given a small enough α2 , the model introduced in Equation (3) becomes locally contractive and converges to a fixed-point. However, in practice the contraction factor of Theorem 2 is not itself guaranteed. For some inputs, we observe that the model often descends into an oscillatory behavior, causing the iteration to stay in a small region of latent space without converging. This non-convergent behavior is not in tension with Theorem 2, as the theorem gives a sufficient condition for convergence, not a complete characterization of the iteration’s 6

behavior. In fact, oscillation around the fixed-point can happen when the Jacobian satisfies certain conditions. Linearizing the iteration near a fixed-point z⋆ gives zi+1 − z⋆ ≈ J (zi − z⋆ ), where J = ∂fθ /∂z |z⋆ . Oscillation around z⋆ arises when J has an eigenvalue with ℜ(λi ) < 1 but |λi | ≥ 1, in which case the iteration spirals around z⋆ rather than contracting toward it. The half-plane condition ℜ(λi ) < 1 is exactly what licenses a runtime fix that does not require modifying fθ : Theorem 3 (Damping stabilizes oscillatory fixed-point dynamics). Suppose fθ ( · ; x) is continuously differentiable in a neighborhood of a fixed-point z⋆ , and that every eigenvalue λi of the Jacobian J at z⋆ satisfies ℜ(λi ) < 1. Define the damped iteration map gη,θ (z; x) := η fθ (z; x) + (1 − η) z. Then there exists η0 ∈ (0, 1) such that, for every η ∈ (0, η0 ), the iteration zi+1 = gη,θ (zi ; x) converges locally to z⋆ . Moreover, gη,θ ( · ; x) and fθ ( · ; x) have the same fixed-points. The proof is in Section A.3. Theorem 3 shows that a suitable damping factor η eliminates the oscillations while preserving the fixed-points of fθ (z; x). We measure convergence to a fixed-point at iteration i as ∥zi − fθ (zi ; x)∥∞ , ∥fθ (zi ; x)∥∞ + ϵ

ri =

which serves as the halting signal: the iteration stops once ri falls below a tolerance τ . To choose η adaptively at inference time, we use a patience mechanism that decreases η whenever this residual stops improving. We track the smallest residual observed so far, r⋆ = minj≤i rj . A geometric decay η ← γ η with γ ∈ (0, 1) is applied to the step-size after P consecutive iterations with no improvement in the residuals. The full procedure is given in Algorithm 1, which is based on the implementation provided by Movahedi et al. (2025). 3.4

Optimization of fixed-point models

One advantage of contractive fixed-point models is that they can be trained using truncated back∂fθ ⋆ ⋆ θ propagation through time (BPTT) (Geng et al., 2021). Let J = ∂f ∂z (z ; x) and P = ∂θ (z ; x) denote the Jacobians of fθ at the fixed-point with respect to the state and the parameters, respectively. Following the implicit function theorem we can write the gradient w.r.t. the parameters of the model as (Bai et al., 2019): dz⋆ = (I − J)−1 P. dθ

(4)

P The Neumann series (I − J)−1 = j≥0 J j converges, assuming fθ (z; x) is contractive. Therefore, truncating the series at depth k yields the estimate: Pk−1 j dz⋆ ≈ (5) j=0 J P, dθ which is closely related to Jacobian-free backpropagation, where the full implicit linear solve is replaced by cheaper approximate gradients (Fung et al., 2022). The truncation depth k trades off computation against accuracy. The following proposition bounds the resulting error under contractivity. Proposition 1 (Exponential decay of truncated-BPTT error). Let z⋆ = fθ (z⋆ ; x) and J = ∂fθ ⋆ D×D . If J is contractive in spectral norm, ∥J∥2 = σ < 1, then for every k ≥ 0, ∂z (z ; x) ∈ R (I − J)−1 −

Pk−1

j j=0 J

F

≤

√

D

σk . 1−σ

The proof is in Section A.4. An essentially equivalent result appears in the proof of Theorem 2 of Geng et al. (2021). 7

Proposition 1 allows for a fixed memory footprint during training, essentially decoupling the number of loops from the memory complexity of the model. In the same spirit, HRM (Wang et al., 2025) and TRM (Jolicoeur-Martineau, 2025) approximate the gradient with a small number of backward passes at the fixed-point, although TRM argues that the fixed-point condition is unnecessary in practice. 3.5

The fixed-point reasoning model

Algorithm 2 FPRM training loop with truncated BPTT and deep supervision

So far, we introduced modifications to improve signal propagation in Looped Transform- Require: Model fθ ; prediction head hϕ ; fixedpoint optimizer FPO PT; model optimizer ers (Section 3.1) without sacrificing stability M ODEL O PT; input x; target y; BPTT depth (Section 3.2). These modifications yield fixedK; initial state z0 point iterations that can be made non-oscillatory (Section 3.3) and trainable through truncated- 1: z ← z0 2: while FPO PT. CONT() do ▷ outer loop BPTT (Section 3.4), making adaptivity through for k = 1, . . . , K do ▷ BPTT window fixed-points reliable. We assemble these compo- 3: 4: z̃k ← fθ (z; x) nents into FPRM, summarized in Algorithm 2 5: z ← FPO PT.STEP(z, z̃k ) and illustrated in Figure 3. The result is a end for Looped Transformer that iterates until its hid- 6: ŷ ← hϕ (z) ▷ deep supervision den state converges, spending compute propor- 7: 8: L ← C ROSS E NTROPY(ŷ, y) tional to each input’s difficulty—the adaptivity M ODEL O PT.BACKWARD(L) our halting mechanism was designed to provide. 9: z ← detach(z) Since we observed no improvements in our ex- 10: periments by using the hierarchical structure 11: end while of Wang et al. (2025), we opt for a classic looped architecture (Dehghani et al., 2019) instead. An overview of the framework is available in Algorithm 2. In Section C we provide further details about FPRM. During training, the model performs looping in windows of k iterations, with k being a hyperparameter, which determines the truncated-BPTT value. During inference, we set k = 1. At each forward pass, the fixed-point optimizer introduced in Algorithm 1 is called to dampen the fixed-point iteration steps. Then, we get a prediction from the current state z of the model and perform a deep supervision step, following Wang et al. (2025); Jolicoeur-Martineau (2025). To truncate the computation graph between deep supervision steps during training, we detach the state z from the graph and stop when the optimizer detects fixed-points. This happens when the residual falls below the tolerance, or the step-size becomes too small.

4

Experiments

We evaluate FPRM against looped reasoning models on puzzle, adaptivity, and signal-propagation benchmarks. Our implementation builds on the public TRM codebase (Jolicoeur-Martineau, 2025) and adopts its deep-supervision training procedure. Additional experimental details are provided in Section G. We first describe the datasets, then evaluate puzzle-solving performance, adaptivity to task difficulty, and depth-induced signal-propagation effects. 4.1

Dataset description

In the following, we provide a brief description for each dataset used in this paper. We refer the reader to the cited literature for more information. Sudoku-Extreme. The task consists of exceptionally challenging, partially filled 9 × 9 Sudoku puzzles with unique solutions, introduced by Wang et al. (2025). Each sample is flattened into a sequence of length 81. The train data consists of 1000 unique samples, each augmented 1000 times, giving a total of ~1M train samples. The test data contains 422,786 samples. The evaluation metric is exact (sequence) accuracy. Maze-Hard. The task consists of difficult 30 × 30 shortest-path maze puzzles with unique solutions, introduced by Wang et al. (2025). The train data and the test data each consist of 1000 unique samples. Following Wang et al. (2025), we do not use augmentation for this problem. The evaluation metric is exact (sequence) accuracy. 8

ARC-1 and ARC-2. The Abstraction and Reasoning Corpus-1 (ARC-1) and -2 (ARC-2), introduced by Chollet et al. (2024), aim to assess the ability of the model to solve novel problems from minimal examples. ARC-1 consists of 2-3 2D grid-based input-output demonstration pairs with variable size (up to 30×30), through which the model is supposed to learn an underlying transformation rule and apply it to a held-out sample. The train data and the test data each contain ~400 samples. ARC-2 has similar characteristics to ARC-1, but with much more challenging problems involving several complex transformations, which are less susceptible to brute-force solutions. The benchmark contains ~1000 training samples and ~360 test samples. The evaluation metric is exact (sequence) pass@2 (top-2 predictions) accuracy. Importantly, pretrained reasoning LLMs with CoT often struggle with these tasks. For example, DeepSeek-R1 (671B model) achieves 15.8% on ARC-1 and 1.3% on ARC-2, Claude 3.7 Sonnet 16K achieves 28.6% on ARC-1 and 0.7% on ARC-2. State tracking. The A5 and S5 state-tracking tasks, introduced by Merrill et al. (2024), are algorithmic benchmarks based on permutation composition. Each sample consists of an initial state and a sequence of k update permutations; the model must apply the updates in order and predict the resulting final state. A5 uses the alternating group on five elements, i.e., the subgroup of even permutations, while S5 uses the full symmetric group on five elements. These tasks are useful proxies for stateful reasoning problems such as entity tracking, code execution, and game-state tracking, since solving them requires learning a composable update rule rather than memorizing computations at fixed lengths. We train on sequences containing up to 32 updates and evaluate out-of-distribution length generalization on sequences containing up to 128 updates. The evaluation metric is exact final-state accuracy. Table 1: Test accuracy on Sudoku-Extreme, Maze-Hard, ARC-AGI-1, and ARC-AGI-2. For each task, the best overall result is bold-face, and the best result with 7M parameters is underlined. The results denoted by † are reproduced using public checkpoints. The ARC results for URM, denoted by ‡ , are only reported for Pass@1. Model

# params.

Single Loop (No Hier.)

Sudoku-Ext. Pass@1

Maze-Hard Pass@1

ARC-1 Pass@2

ARC-2 Pass@2

27M 7M

✗ ✗

71.4% 72.6%

– 79.0%

– 40.0%†

– 6.2%†

27M 27M 14M 7M 7M 7M 7M

✗ ✗ ✗ ✗ ✗ ✗ ✓

55.0% 91.4% 77.6% 74.7% 93.0% 54.3% 94.2%

74.5% 93.1% – 85.3% – 46.7% 87.0%

40.3% – ≥53.8%‡ 44.6% – – 47.5%

5.0% – ≥16.0%‡ 7.8% – – 6.2%

Our reproduction attempt Attractor Model TRM Reported HRM Attractor Model URM TRM EqR Attractor Model FPRM

4.2

Puzzle tasks

We first evaluate FPRM on puzzle problems, namely the Sudoku-Extreme, Maze-Hard (Wang et al., 2025), ARC-1, and ARC-2 (Chollet et al., 2024) benchmarks. These benchmarks were designed to test whether latent recurrent reasoning models can solve symbolic search problems from limited supervision. In the following, we discuss the experimental results. For more information about the baselines, we refer the reader to Section 2. Table 1 demonstrates the effectiveness of FPRM as the best performing model with 7M parameters on Sudoku-Extreme, Maze-Hard, and ARC-1, with performance on par with TRM on ARC-2. On Sudoku-Extreme, FPRM improves upon even larger models. It is worth noting that these results are achieved without the breadth-search fixed-point method proposed by Huang et al. (2026), which is orthogonal to FPRM’s modifications. On Maze-Hard, FPRM underperforms compared to the larger 9

Attractor model with ~×4 more parameters. However, we note that we were not able to reproduce the results reported in the paper (Fein-Ashley and Rashidinejad, 2026). On ARC-2, FPRM performs similarly to the publicly available TRM checkpoints, but underperforms compared to the TRM results reported in (Jolicoeur-Martineau, 2025) and the URM baseline, which has roughly twice as many parameters. However, considering the recent trend, we emphasize that the ARC benchmarks appear to be much more sensitive to parameter count compared to the other puzzle tasks considered in this section (Shu et al., 2026; Hu et al., 2025). Therefore, we note that the comparison may not be fair in this case. Given that FPRM performs on par with or better than the hierarchical baselines that use post-norm on the tasks from Table 1, we hypothesize that the hierarchy might be alleviating signal propagation issues in these models. However, if this hypothesis holds, hierarchy’s benefit appears limited: in Figure 6, both models improve as effective layer grows, but the performance gap widens in favor of FPRM, and TRM’s performance stays below FPRM at matched effective layers. Moreover, in Table 3, we attempt to investigate the effect of introducing our architectural modifications to the Transformer layers used in TRM on its performance on Sudoku-Extreme. We find the proposed modifications to have a detrimental impact on the performance of the models, which we attribute to, among other things, hyperparameter optimization and a careful redesigning of the looping. 4.3

Adaptivity

In this section, we demonstrate the adaptivity of FPRM, and compare it to TRM. We use three benchmarks: Sudoku-Extreme from Section 4.2 and state-tracking benchmarks A5 and S5 introduced in Merrill et al. (2024). Each benchmark provides an intuitive measure of difficulty. For SudokuExtreme, this is the number of empty cells, an established proxy for difficulty (Prates and Lamb, 2018). For state-tracking, it is the sequence length.

320

80 60 40 20

2

32

64

96

Sequence length

(a) Accuracy, A5

128

Test accuracy (%)

100

TRM+conv w/o ACT

240 160 80 0

2

32

64

96

Sequence length

128

(b) Adaptivity, A5

TRM+conv w/ ACT

100 80 60 40 20

FPRM (ours)

320

Effective layers

TRM w/ ACT Effective layers

Test accuracy (%)

TRM w/o ACT

2

32

64

96

Sequence length

(c) Accuracy, S5

128

240 160 80 0

2

32

64

96

Sequence length

128

(d) Adaptivity, S5

Figure 4: Length generalization and adaptive compute as a function of sequence length. Shaded bands show 95% confidence intervals over seeds. The vertical dotted line marks the training length 32. The matched compute budget is 320 effective layers. State tracking. As shown in Figure 4, plain TRM fails to extrapolate: at length 128, TRM and TRM with ACT enabled both obtain 45.8% ± 3.9% on A5 and 39.4% ± 1.9% on S5 . Adding a causal 1D convolution layer substantially improves length generalization, reaching 91.4% ± 2.3% on A5 and 97.2% ± 2.5% on S5 . This modification is not part of the original TRM, but matches the task structure: group composition is a local left-to-right scan, si = si−1 · gi , and causal convolution provides a shared translation-equivariant primitive for this operation. However, 1D convolution does not make ACT reliable. On A5 , TRM+conv+ACT scales compute with sequence length but drops to 65.3% ± 2.2%, well below TRM+conv without ACT. On S5 , it reaches 96.2% ± 3.2%, but uses substantially more compute than FPRM; moreover, only a few seeds learn to adapt, while the others exhaust the full compute budget. In contrast, FPRM scales compute smoothly with sequence length while maintaining high accuracy, reaching 98.1% ± 2.2% on A5 and 98.8% ± 0.9% on S5 at length 128. Sudoku-Extreme. We compare the performance and efficiency of the halting mechanisms in FPRM and TRM on Sudoku puzzles of varying difficulty, measured by the number of empty cells (Figure 5). The sample counts per difficulty level are shown in Figure 11. We observe that halting mechanisms in 10

TRM w/ ACT

TRM w/o ACT

FPRM (ours) 100

80 70 60

Test accuracy (%)

90 Effective layers

Test accuracy (%)

100 103

102

50 40

45

48

51

54

57

Number of empty cells

(a) Accuracy vs. difficulty

60

45

48

51

54

57

60

Number of empty cells

90

ri

size

80 70 60

103

Inference compute (effective layers)

(b) Inference compute vs. difficulty

Figure 5: FPRM achieves (a) better accuracy, while (b) adapting Figure 6: Test-time scaling lowmore efficiently to the task difficulty. Difficulty is measured by ers residuals while improving the number of empty cells in the Sudoku grid. The max. compute accuracy. Both models run unbudget is matched across models (4788 effective layers). From (b): til the matched compute budget effective layers are reported as medians with 25th–75th percentiles is exhausted. FPRM achieves bands. The default behavior of TRM is without ACT at inference higher accuracy across the entime (in black), which exhausts the max. budget for all sample tire range. Marker size denotes difficulties. residual.

both TRM and FPRM adapt the number of loops, i.e., inference compute, to the sample difficulty. In contrast, the default behavior of TRM during inference (deactivated ACT) does not adapt the compute to the task difficulty. Even when we enable ACT with TRM, FPRM proves to be more efficient and accurate. Compute–accuracy trade-off in fixed-point halting. The previous experiments show that FPRM adapts its compute to task difficulty. We now examine how this adaptation can be controlled. In Figure 8, we demonstrate the effect of the step-size decay rate (γ) and maximum patience (P ) from Algorithm 1 on the halting time and performance at test-time. Max. inference-time budget was set to be the same (70k effective layers) for all experiments. We observe that larger decay rates improve the performance. Moreover, we observe that maximum patience has minimal impact on the performance, with its impact completely disappearing for larger decay rates. On the other hand, the increased performance comes at the cost of reaching much deeper effective layers before halting. The largest decay rate that achieves the best performance also exhausts the max. inference budget. These trends follow from how γ and P control the step size η from Algorithm 1. A decay rate closer to 1 reduces η only slightly once the residual stops improving. The state therefore keeps evolving, halting is deferred, and the model reaches deeper effective layers. The accuracy gain follows directly from these additional iterations. Patience P sets only when a decay occurs, not its magnitude, so its effect is minor; as γ approaches 1 each decay becomes negligible and the two patience curves coincide. The decay rate thus controls the compute-accuracy trade-off directly: a larger γ spends more compute and yields higher accuracy, while P offers only secondary control that vanishes as compute saturates. 4.4

Depth-induced signal propagation issues

Here, we aim to investigate the role of pre-norm with residual scaling in mitigating the depth-induced signal propagation issues in looped models. For a comprehensive investigation, we approach the problem from three angles: (1) trainability, where we show that pre-norm with residual scaling keeps activations bounded and enables stable training at large depth (Section 4.4.1); (2) depth utilization, where we show that boundedness alone is not sufficient, and that pre-norm with residual scaling additionally improves depth utilization in FPRM (Sun et al., 2025) (Section 4.4.2); and (3) residual scaling, where we analyze the training dynamics of the scaling parameters (Section 4.4.3). 11

20 0

400

800

1.0 0.8 0.6 0.4 0.2

1200

1600

2000

Inference compute (effective layers)

(a) Test accuracy

0.0

threshold = 0.1

0

400

800

1200

1600

2000

Inference compute (effective layers)

95 90 85

60k 30k 0

(b) Fixed-point residuals

Figure 7: Loop-utilization of FPRM on Sudoku. (a) test accuracy of FPRM with pre-norm with residual scales vs. post-norm. (b) median residual. The pre-norm model is better at loop utilization, while both have similar residuals. This indicates similar latent-space convergence, with more meaningful updates in the pre-norm variant, resulting in improved performance.

4.4.1

Inference compute (effective layers)

Median residual

40

0

saturates @ 1040

saturates @ 640

Test accuracy (%)

80 60

P = 10

P=5

FPRM + post-norm

Test accuracy (%)

FPRM + pre-norm + residual scaling

0.95

0.99

0.995

0.996

Decay Rate ( )

0.997

Figure 8: Decay rate and patience. Test accuracy and effective layer of FPRM with fixedpoint halting as a function of decay rate γ, for maximumpatience P ∈ {5, 10}.

Boundedness of activation norms and trainability

As noted in Section 3.1, the normalization scheme governs a trade-off between activation stability and signal propagation, which sharpens with depth. Post-norm keeps activations bounded but suffers from signal propagation issues. Pre-norm enables better signal propagation but lets activations grow exponentially. To isolate this trade-off, we adopt the Looped Transformer framework (Saunshi et al., 2025; Dehghani et al., 2019; Kaiser and Sutskever, 2016) with a fixed number of effective layers. This controls for dynamic halting, which can reduce the effective layers, as introduced in Section 3. In Figure 2b we show the norm of the final activations of the residual branch of Looped Transformer at initialization. From the figure, we conclude that the magnitude of the activations grows exponentially in the case of pre-norm Looped Transformer. In contrast, when using post-norm or pre-norm with residual scaling, the architecture benefits from bounded activations. In Figure 2a we give evidence that boundedness is a prerequisite for reaching deeper effective layers. Pre-norm architecture with its unbounded activations diverges and can’t reach deep effective layers, while bounded architectures ensure trainability. Therefore, a Looped Transformer with pre-norm and without residual scaling cannot achieve deep effective layers, highlighting one aspect of the importance of residual scaling in pre-norm. 4.4.2

Pre-norm with residual scaling enables higher depth utilization

An important measure for evaluating signal propagation issues in deep neural networks is depth utilization (Sun et al., 2025). Depth utilization measures whether all layers contribute meaningfully to the model. It is commonly probed by removing layers from a trained model and measuring the effect on performance at test-time. In models with signal propagation problems, removing deeper layers (closer to the output) usually does not impact the performance significantly, highlighting that these models fail to utilize the compute due to signal propagation issues. However, in a looped model there is no fixed stack of layers to remove, since the same block is applied to the previous latent representations. Consequently, we cannot perform the same experiment here. We therefore probe the same property from the opposite direction: instead of removing computation and measuring degradation, we add computation and measure improvement. We do so in two complementary regimes. In the first, using the state-tracking task, we ask whether the model can be trained to use the depth that a harder task requires. In the second, using the Sudoku-Extreme task, we ask whether a trained model can convert depth beyond its training regime into further gains at test time. State-tracking. In Section 4.4.1, we discussed the necessity of using normalization layers that ensure the boundedness of the activations, which is required for the stable training. However, bounded activations are not a sufficient condition to ensure effective utilization of large depth, once it 12

is reached. We demonstrate this in Figure 2a using a state-tracking task, where increasing difficulty, corresponding to longer sequences, requires training at greater depths (Movahedi et al., 2025). We train a Looped Transformer model with its number of loops (effective layer) tied to the train sequence length. We choose the sequence length from the set {8, 16, 32, 64, 128, 256, 512}. We plot the maximum sequence length solved with > 90% accuracy against effective layer. Because we also at test-time match effective layer to the training sequence length, a model with no signal-propagation bottleneck should solve exactly the length its depth permits, tracing the identity line y = x (Figure 3 in Movahedi et al. (2025)). We observe that this behavior is only present in the model equipped with pre-norm and residual scaling. We interpret this observation as strong evidence for improved trainability in Looped Transformers with pre-norm and residual scaling. Sudoku-Extreme. Compared to the previous experiment on the state-tracking task, here we run each model far beyond its trained depth, trying to detect the point where more compute no longer translates into improvements at test-time. We expect the performance of a model with fewer signal propagation issues to saturate later and with more effective layers, indicating that the model is capable of reaching deeper effective layer. On the other hand, the performance of a model bottlenecked by signal propagation problems is expected to saturate early, indicating that it cannot convert the extra compute into better predictions. For this experiment, we focus on two variants of the FPRM model, one with pre-norm and residual scaling, and the other with post-norm, both trained on the Sudoku-Extreme task. In Figure 7a, we demonstrate that scaling test-time compute improves performance for both types of normalization. Furthermore, considering Figure 7b, we also observe that the majority of the improvement comes before at least half of the samples halt. However, there are clear differences between the effective layer of the two normalization methods in Figure 7a, as the pre-norm model’s performance saturates at almost twice as much compute, indicating improved signal propagation through depth.

Table 2: Sensitivity of FPRM to residual scaling initialization on Sudoku-Extreme dataset. Each cell reports best test sequence accuracy (%) for a given pair of initial values. α2 init α1 init

0.25

0.50

0.75

0.25 0.50 0.75

83.44 84.49 94.23

78.10 89.05 91.41

83.24 86.29 85.70

2 init.

1 init.

2

1

0.00

0.25

0.50

0.75

Trained value

1.00

The advantage of pre-norm with residual scaling also extends to Figure 9: The distribution of the cross-model comparison in Figure 6, between FPRM (pre-norm the residual scales in FPRM with residual scaling) and TRM (post-norm). In this inference- after training on the Sudokutime scaling experiment, all samples are run to a varied maximum Extreme dataset. looping compute budget. For TRM, compute is scaled through deep supervision steps, which we find optimal relative to scaling L- and H-steps (see Section F). FPRM outperforms TRM across a range of effective-layer depths reached, with the gap widening at higher compute budgets, consistent with FPRM making better use of its depth. 4.4.3

The training dynamics of residual scales

While our original motivation for residual scaling was to prevent unbounded activations in pre-norm, we note a parallel relationship between our solution and common solutions to signal propagation problems (Sun et al., 2025; Noci et al., 2022). Specifically, it has been known that scaling down the output of the sub-layers when introducing them to the residual stream is beneficial to increasing effective layer, which is equivalent to increasing α1 in Equation (2). Moreover, in Theorem 2, we show that the looping will become more stable for smaller α2 in Equation (3). In order to investigate the impact of these two parameters, we perform a coarse-grained ablation on the initial value of α1 , α2 on the Sudoku-Extreme task. Table 2 demonstrates that the best initialization places α1 at a high value and α2 at a low value. Interestingly, the α1 preference matches the common solutions to signal propagation problems: keeping the residual stream dominant. Moreover, the ablation also highlights the importance of having a more contractive mapping at initialization, as our convergence analysis (Theorem 2) requires a sufficiently small α2 for the loop to reach a fixed-point. On the other hand, comparing the row with the smallest α2 choice (α2 = 0.25) with the column with the largest α1 (α1 = 0.75), we observe that increasing α1 has a slightly higher positive impact on the accuracy than a decreasing α2 . We hypothesize that this is because it is easier for the model to recover from a bad choice of α2 than α1 , 13

as the gradients for α1 come from two different sources (the MHA and MLP sub-layers), and thus can be noisier. In Figure 9, we provide the distribution statistics of the residual scales after training. Interestingly, we observe that while the median of the α1 , α2 values over channels does not deviate significantly from the initial point, the spread of the distributions widens significantly. In the case of α1 , the widening happens at a much larger scale, covering both very small and very large values. On the other hand, the α2 distribution remains more concentrated, with the majority of the values actually becoming smaller than the initial point. This can be interpreted as the model learning to become more contractive during training, which is in line with the observations in (Bansal et al., 2022). Furthermore, this observation also supports our hypothesis that it might be easier for the model to learn the optimal α2 values than the α1 values.

5

Discussion

Our experiments support three broader observations about FPRM and looped reasoning models in general, which we discuss in turn. Looped fixed-point models are adaptive. FPRM adapts to the difficulty of the problem more effectively compared to TRM (Figures 4, 5, and Section 4.3), using fewer effective layers (compute), while achieving better performance. This is a consequence of FPRM halting closer to the saturation point of accuracy (Figure 12). In contrast, TRM with its ACT halting mechanism either halts too early, resulting in lower performance, or too late, using excessive compute. Enabling ACT at inference time. The original proposed TRM does not use its trained ACT head at inference time, leading to the non-adaptive behavior. However, we find this to be an engineering challenge rather than a fundamental limitation. Therefore, in Figures 1, 4, 5, we record the number of effective layers reached at the moment when the probability of halting exceeds 0.5. Similarly, in the case of FPRM, we do so when the residual drops below the set threshold (0.1 in this case). However, note that the halted samples remain in the batch until the last sample in the batch halts. We leave the efficient implementation for future work. The role of hierarchy in HRM and TRM. While originally hierarchical reasoning was biologically motivated (Wang et al., 2025), later explanations involved likening the lower level of the hierarchy to a scratch pad, the latent representation of which is used by the higher level for prediction (Jolicoeur-Martineau, 2025). However, the role of hierarchy as the driving force behind the success of hierarchical reasoning models has been brought into question recently, with similar architectures without the hierarchy performing as well as hierarchical models (Ge et al., 2025; ARC Prize Foundation, 2025). In Section 4.4, we observe that a Transformer model with post-norm, which is the building block of TRM and HRM, suffers from a signal propagation issue. On the other hand, in Section 4.2 we were able to show that by improving signal propagation, FPRM improves upon these models without requiring the hierarchy. In Figure 13, we observe that reallocating the compute from the H- and L-steps to the additional deep supervision steps improves TRM’s performance. Since more H- and L-steps increase the effective layer of TRM within each supervision step, the signal propagation issue induced by post-norm is amplified. The improvement is therefore consistent with TRM being limited by the same signal-propagation issue we identify in Section 4.4. In light of these results, we hypothesize that there might be a simpler explanation for the success of hierarchical models: the hierarchy improves signal propagation. We identify the theoretical explanation of the role of hierarchy through the lens of optimization and signal propagation as an interesting direction for future work. Scaling behavior of FPRM. The results of Section 4 combine into a coherent picture of how FPRM scales its computation. First, with better signal propagation FPRM is able to utilize compute more efficiently (Figure 6). Second, as more difficult problems require more compute (Merrill et al., 2024; Movahedi et al., 2025), better test-time scaling of FPRM is mostly visible in harder tasks (Figures 4, 5a). Finally, because in FPRM halting is governed by the fixed-point optimizer rather than a learned module, a natural controlling mechanism for the compute-performance trade-off appears in the form of the decay rate γ and maximum patience P , allowing practitioners to select a desired point on the Pareto front. However, the optimality of the algorithms learned by looped models is 14

not guaranteed, with great variation not only possible but likely. For example, for solving A5 , CoT would require a super-logarithmic number of iterations (Merrill and Sabharwal, 2024), while an optimal algorithm could solve it in logarithmic time. This suboptimal scaling has also been observed in recurrent-in-depth state-space models (Movahedi et al., 2025), a behavior that we also observe in Figure 4. Therefore, we propose it as an open challenge to find a latent reasoning architecture that achieves a solution with logarithmic complexity to state-tracking while remaining Turing-complete (Dehghani et al., 2019). Limitations. In a similar spirit to previous literature on end-to-end reasoning (Kaiser and Sutskever, 2016; Fan et al., 2025; Wang et al., 2025; Jolicoeur-Martineau, 2025; Du et al., 2022, 2024), we test our model only on algorithmic tasks and not on natural language. It is an open challenge to demonstrate that the compositional reasoning behavior that latent models exhibit on algorithmic tasks translates to other domains. In addition, even though the base architecture of FPRM could adopt any model (e.g. CNN, MLP, state-space models), we limit our experiments to Transformers.

6

Conclusion

We present architectural modifications for looped fixed-point Transformers that enable the use of pre-norm, improving the model’s ability to exploit deeper effective layer provided by looping. These modifications allow FPRM to outperform hierarchical baselines of similar size, such as HRM and TRM, on common symbolic reasoning benchmarks. We show that on state tracking and SudokuExtreme, FPRM is able to adapt its compute to the difficulty of the task. This capability stems from dynamically scaling depth through fixed-point iterations and improving signal propagation. We hope these architectural modifications and the accompanying insights will support further progress on latent reasoning models.

Acknowledgments and Disclosure of Funding We thank Felix Sarnthein, Albert Catalan-Tatjer, Jonas Geiping, Philipp Nazari, Carl Richardson, and Nouha Dziri for the helpful discussions and comments. Alexander Theus, Vera Milovanović, and Shlomo Libo Feigin are supported by the Max Planck ETH Center for Learning Systems. Vera Milovanović and Antonio Orvieto are supported by the AI2050 program at Schmidt Sciences. Antonio Orvieto, T. Konstantin Rusch, and Sajad Movahedi acknowledge the financial support of the Hector Foundation.

15

References Donald G. M. Anderson. Iterative procedures for nonlinear integral equations. J. ACM, 12(4):547–560, 1965. doi: 10.1145/321296.321305. URL https://doi.org/10.1145/321296.321305. Cem Anil, Ashwini Pokle, Kaiqu Liang, Johannes Treutlein, Yuhuai Wu, Shaojie Bai, J. Zico Kolter, and Roger B. Grosse. Path independent equilibrium models can better exploit test-time computation. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL http://papers.nips.cc/paper_files/paper/2022/hash/ 331c41353b053683e17f7c88a797701d-Abstract-Conference.html. ARC Prize Foundation. The hidden drivers of HRM’s performance on ARC-AGI. https:// arcprize.org/blog/hrm-analysis, 8 2025. Accessed: 2025-11-24. Sangmin Bae, Yujin Kim, Reza Bayat, Sungnyun Kim, Jiyoun Ha, Tal Schuster, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Aaron C. Courville, and Se-Young Yun. Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation. CoRR, abs/2507.10524, 2025. doi: 10.48550/ARXIV.2507.10524. URL https://doi.org/10.48550/arXiv.2507.10524. Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. Deep equilibrium models. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 688–699, 2019. URL https://proceedings.neurips.cc/paper/2019/ hash/01386bd6d8e091c2ab4c7c7de644d37b-Abstract.html. Shaojie Bai, Vladlen Koltun, and J. Zico Kolter. Stabilizing equilibrium models by jacobian regularization. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, Proceedings of Machine Learning Research, pages 554–565. PMLR, 2021. URL http://proceedings.mlr. press/v139/bai21b.html. Andrea Banino, Jan Balaguer, and Charles Blundell. Pondernet: Learning to ponder. CoRR, abs/2107.05407, 2021. URL https://arxiv.org/abs/2107.05407. Arpit Bansal, Avi Schwarzschild, Eitan Borgnia, Zeyad Emam, Furong Huang, Micah Goldblum, and Tom Goldstein. End-to-end algorithm synthesis with recurrent networks: Extrapolation without overthinking. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL http://papers.nips.cc/paper_files/paper/2022/hash/ 7f70331dbe58ad59d83941dfa7d975aa-Abstract-Conference.html. David Belanger and Andrew McCallum. Structured prediction energy networks. In Maria-Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, JMLR Workshop and Conference Proceedings, pages 983–992. JMLR.org, 2016. URL http://proceedings.mlr. press/v48/belanger16.html. David Belanger, Bishan Yang, and Andrew McCallum. End-to-end learning for structured prediction energy networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, Proceedings of Machine Learning Research, pages 429–439. PMLR, 2017. URL http://proceedings.mlr.press/v70/belanger17a.html. Hugh Blayney, Alvaro Arroyo, Johan S. Obando-Ceron, Pablo Samuel Castro, Aaron C. Courville, Michael M. Bronstein, and Xiaowen Dong. A mechanistic analysis of looped reasoning language models. CoRR, abs/2604.11791, 2026. doi: 10.48550/ARXIV.2604.11791. URL https://doi. org/10.48550/arXiv.2604.11791. 16

Aleksandar Botev, Soham De, Samuel L. Smith, Anushan Fernando, George-Cristian Muraru, Ruba Haroun, Leonard Berrada, Razvan Pascanu, Pier Giuseppe Sessa, Robert Dadashi, Léonard Hussenot, Johan Ferret, Sertan Girgin, Olivier Bachem, Alek Andreev, Kathleen Kenealy, Thomas Mesnard, Cassidy Hardin, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Armand Joulin, Noah Fiedel, Evan Senter, Yutian Chen, Srivatsan Srinivasan, Guillaume Desjardins, David Budden, Arnaud Doucet, Sharad Vikram, Adam Paszke, Trevor Gale, Sebastian Borgeaud, Charlie Chen, Andy Brock, Antonia Paterson, Jenny Brennan, Meg Risdal, Raj Gundluru, Nesh Devanathan, Paul Mooney, Nilay Chauhan, Phil Culliton, Luiz Gustavo Martins, Elisa Bandy, David Huntsperger, Glenn Cameron, Arthur Zucker, Tris Warkentin, Ludovic Peran, Minh Giang, Zoubin Ghahramani, Clément Farabet, Koray Kavukcuoglu, Demis Hassabis, Raia Hadsell, Yee Whye Teh, and Nando de Frietas. Recurrentgemma: Moving past transformers for efficient open language models. CoRR, abs/2404.07839, 2024. doi: 10.48550/ARXIV.2404.07839. URL https://doi.org/10.48550/arXiv.2404.07839. C. G. Broyden. A class of methods for solving nonlinear simultaneous equations. Mathematics of Computation, 19:577–593, 1965. URL https://api.semanticscholar.org/CorpusID: 2802972. François Chollet, Mike Knoop, Gregory Kamradt, and Bryan Landers. ARC prize 2024: Technical report, 2024. URL https://doi.org/10.48550/arXiv.2412.04604. Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https://openreview.net/ forum?id=HyzdRiR9Y7. Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: pure attention loses rank doubly exponentially with depth. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, Proceedings of Machine Learning Research, pages 2793–2803. PMLR, 2021. URL http://proceedings.mlr.press/v139/dong21a.html. Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. Advances in neural information processing systems, 32, 2019. Yilun Du, Shuang Li, Joshua B. Tenenbaum, and Igor Mordatch. Learning iterative reasoning through energy minimization. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, Proceedings of Machine Learning Research, pages 5570–5582. PMLR, 2022. URL https://proceedings.mlr.press/v162/du22d.html. Yilun Du, Conor Durkan, Robin Strudel, Joshua B Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dickstein, Arnaud Doucet, and Will Sussman Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and mcmc. In International conference on machine learning, pages 8489–8510. PMLR, 2023. Yilun Du, Jiayuan Mao, and Joshua B. Tenenbaum. Learning iterative reasoning through energy diffusion, 2024. URL https://proceedings.mlr.press/v235/du24f.html. Katie E. Everett, Lechao Xiao, Mitchell Wortsman, Alexander A. Alemi, Roman Novak, Peter J. Liu, Izzeddin Gur, Jascha Sohl-Dickstein, Leslie Pack Kaelbling, Jaehoon Lee, and Jeffrey Pennington. Scaling exponents across parameterizations and optimizers. In Ruslan Salakhutdinov, Zico Kolter, Katherine A. Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, Proceedings of Machine Learning Research, pages 12666–12700. PMLR / OpenReview.net, 2024. URL https://proceedings.mlr.press/v235/everett24a.html. Ying Fan, Yilun Du, Kannan Ramchandran, and Kangwook Lee. Looped transformers for length generalization. In International Conference on Learning Representations, volume 2025, pages 14502–14520, 2025. 17

Jacob Fein-Ashley and Paria Rashidinejad. Solve the loop: Attractor models for language and reasoning, 2026. URL https://doi.org/10.48550/arXiv.2605.12466. Samy Wu Fung, Howard Heaton, Qiuwei Li, Daniel McKenzie, Stanley J. Osher, and Wotao Yin. JFB: jacobian-free backpropagation for implicit networks. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 - March 1, 2022, pages 6648–6656. AAAI Press, 2022. doi: 10.1609/AAAI.V36I6.20619. URL https://doi.org/10.1609/aaai.v36i6. 20619. Zitian Gao, Lynx Chen, Yihao Xiao, He Xing, Ran Tao, Haoming Luo, Joey Zhou, and Bryan Dai. Universal reasoning model. CoRR, abs/2512.14693, 2025. doi: 10.48550/ARXIV.2512.14693. URL https://doi.org/10.48550/arXiv.2512.14693. Renee Ge, Qianli Liao, and Tomaso A. Poggio. Hierarchical reasoning models: Perspectives and misconceptions. CoRR, abs/2510.00355, 2025. doi: 10.48550/ARXIV.2510.00355. URL https://doi.org/10.48550/arXiv.2510.00355. Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. Advances in Neural Information Processing Systems, 38: 41340–41391, 2026. Zhengyang Geng and J. Zico Kolter. Torchdeq: A library for deep equilibrium models, 2023. URL https://doi.org/10.48550/arXiv.2310.18605. Zhengyang Geng, Xin-Yu Zhang, Shaojie Bai, Yisen Wang, and Zhouchen Lin. On training implicit models. In NeurIPS, pages 24247–24260, 2021. URL https://proceedings.neurips.cc/ paper/2021/hash/cb8da6767461f2812ae4290eac7cbc42-Abstract.html. Angeliki Giannou, Shashank Rajput, Jy-yong Sohn, Kangwook Lee, Jason D. Lee, and Dimitris Papailiopoulos. Looped transformers as programmable computers. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, Proceedings of Machine Learning Research, pages 11398–11442. PMLR, 2023. URL https://proceedings.mlr.press/v202/giannou23a.html. Alexi Gladstone, Ganesh Nanduru, Md Mofijul Islam, Peixuan Han, Hyeonjeong Ha, Aman Chadha, Yilun Du, Heng Ji, Jundong Li, and Tariq Iqbal. Energy-based transformers are scalable learners and thinkers. CoRR, abs/2507.02092, 2025. doi: 10.48550/ARXIV.2507.02092. URL https: //doi.org/10.48550/arXiv.2507.02092. Alex Graves. Adaptive computation time for recurrent neural networks. CoRR, abs/1603.08983, 2016. URL http://arxiv.org/abs/1603.08983. Daya Guo, Dejian Yang, Haowei Zhang, et al. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nat., 645(8081):633–638, 2025. doi: 10.1038/S41586-025-09422-Z. URL https://doi.org/10.1038/s41586-025-09422-z. Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space, 2024. URL https: //doi.org/10.48550/arXiv.2412.06769. Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and HsuanTien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 612, 2020, virtual, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ 4c5bcfec8584af0d967f1ab10179ca4b-Abstract.html. Keya Hu, Ali Cy, Linlu Qiu, Xiaoman Delores Ding, Runqian Wang, Yeyin Eva Zhu, Jacob Andreas, and Kaiming He. ARC is a vision problem! CoRR, abs/2511.14761, 2025. doi: 10.48550/ARXIV. 2511.14761. URL https://doi.org/10.48550/arXiv.2511.14761. 18

Benhao Huang, Zhengyang Geng, and Zico Kolter. Equilibrium reasoners: Learning attractors enables scalable reasoning, 2026. URL https://arxiv.org/abs/2605.21488. Ahmadreza Jeddi, Marco Ciccone, and Babak Taati. Loopformer: Elastic-depth looped transformers for latent reasoning via shortcut modulation. CoRR, abs/2602.11451, 2026. doi: 10.48550/ARXIV. 2602.11451. URL https://doi.org/10.48550/arXiv.2602.11451. Alexia Jolicoeur-Martineau. Less is more: Recursive reasoning with tiny networks, 2025. URL https://doi.org/10.48550/arXiv.2510.04871. Lukasz Kaiser and Ilya Sutskever. Neural gpus learn algorithms. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http://arxiv.org/abs/ 1511.08228. Ferdinand Kapl, Emmanouil Angelis, Kaitlin Maile, Johannes von Oswald, and Stefan Bauer. From growing to looping: A unified view of iterative computation in llms, 2026. URL https://doi. org/10.48550/arXiv.2602.16490. Hyunjik Kim, George Papamakarios, and Andriy Mnih. The lipschitz constant of self-attention. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, Proceedings of Machine Learning Research, pages 5562–5571. PMLR, 2021. URL http://proceedings.mlr.press/v139/ kim21i.html. Jeonghoon Kim, Byeongchan Lee, Cheonbok Park, Yeontaek Oh, Beomjun Kim, Taehwan Yoo, Seongjin Shin, Dongyoon Han, Jinwoo Shin, and Kang Min Yoo. Peri-ln: Revisiting normalization layer in the transformer architecture. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon LacosteJulien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 1319, 2025, Proceedings of Machine Learning Research. PMLR / OpenReview.net, 2025. URL https://proceedings.mlr.press/v267/kim25u.html. Harsh Kohli, Srinivasan Parthasarathy, Huan Sun, and Yuekun Yao. Loop, think, & generalize: Implicit reasoning in recurrent-depth transformers. CoRR, abs/2604.07822, 2026. doi: 10.48550/ ARXIV.2604.07822. URL https://doi.org/10.48550/arXiv.2604.07822. Asher Labovich. Stability and generalization in looped transformers, 2026. URL https://doi. org/10.48550/arXiv.2604.15259. Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, Fujie Huang, et al. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7. William Merrill and Ashish Sabharwal. The expressive power of transformers with chain of thought, 2024. URL https://arxiv.org/abs/2310.07923. William Merrill, Jackson Petty, and Ashish Sabharwal. The illusion of state in state-space models. In Ruslan Salakhutdinov, Zico Kolter, Katherine A. Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, Proceedings of Machine Learning Research, pages 35492–35506. PMLR / OpenReview.net, 2024. URL https://proceedings.mlr.press/ v235/merrill24a.html. Sajad Movahedi, Felix Sarnthein, Nicola Muca Cirone, and Antonio Orvieto. Fixed-point rnns: From diagonal to dense in a few iterations, 2025. URL https://doi.org/10.48550/arXiv.2503. 10799. 19

Lorenzo Noci, Sotiris Anagnostidis, Luca Biggio, Antonio Orvieto, Sidak Pal Singh, and Aurélien Lucchi. Signal propagation in transformers: Theoretical perspectives and the role of rank collapse. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL http://papers.nips.cc/paper_files/paper/2022/hash/ ae0cba715b60c4052359b3d52a2cff7f-Abstract-Conference.html. OpenAI. Learning to reason with LLMs. learning-to-reason-with-llms/, September 2024. ing the o1 release.

https://openai.com/index/ OpenAI blog post, accompany-

Antonio Orvieto, Samuel L. Smith, Albert Gu, Anushan Fernando, Çaglar Gülçehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, Proceedings of Machine Learning Research, pages 26670–26698. PMLR, 2023. URL https://proceedings.mlr.press/v202/orvieto23a.html. Vardhan Palod, Karthik Valmeekam, Kaya Stechly, and Subbarao Kambhampati. Performative thinking? the brittle correlation between cot length and problem complexity. CoRR, abs/2509.07339, 2025. doi: 10.48550/ARXIV.2509.07339. URL https://doi.org/10.48550/arXiv.2509. 07339. Marcelo O. R. Prates and Luís C. Lamb. Problem solving at the edge of chaos: Entropy, puzzles and the sudoku freezing transition. In Lefteri H. Tsoukalas, Éric Grégoire, and Miltiadis Alamaniotis, editors, IEEE 30th International Conference on Tools with Artificial Intelligence, ICTAI 2018, 5-7 November 2018, Volos, Greece, pages 686–693. IEEE, 2018. doi: 10.1109/ICTAI.2018.00109. URL https://doi.org/10.1109/ICTAI.2018.00109. Zirui Ren and Ziming Liu. Are your reasoning models reasoning or guessing? A mechanistic analysis of hierarchical reasoning models, 2026. URL https://doi.org/10.48550/arXiv. 2601.10679. Tim Salimans and Jonathan Ho. Should ebms model the energy or the score?, 2021. Energy-Based Models Workshop, ICLR 2021. Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, and Sashank J. Reddi. Reasoning with latent thoughts: On the power of looped transformers. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https://openreview.net/forum?id=din0lGfZFd. Wen-Jie Shu, Xuerui Qiu, Rui-Jie Zhu, Harold Haodong Chen, Yexin Liu, and Harry Yang. Loopvit: Scaling visual ARC with looped transformers. CoRR, abs/2602.02156, 2026. doi: 10.48550/ ARXIV.2602.02156. URL https://doi.org/10.48550/arXiv.2602.02156. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling model parameters. CoRR, abs/2408.03314, 2024. doi: 10.48550/ ARXIV.2408.03314. URL https://doi.org/10.48550/arXiv.2408.03314. Shixiang Song, He Li, Zitong Wang, Boyi Zeng, Feichen Song, Yixuan Wang, Zhiqin John Xu, Ziwei He, and Zhouhan Lin. Adaponderlm: Gated pondering language models with tokenwise adaptive depth. CoRR, abs/2603.01914, 2026. doi: 10.48550/ARXIV.2603.01914. URL https://doi.org/10.48550/arXiv.2603.01914. Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://openreview.net/forum?id=PxTIG12RRHS. Wenfang Sun, Xinyuan Song, Pengxiang Li, Lu Yin, Yefeng Zheng, and Shiwei Liu. The curse of depth in large language models. CoRR, abs/2502.05795, 2025. doi: 10.48550/ARXIV.2502.05795. URL https://doi.org/10.48550/arXiv.2502.05795. 20

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998–6008, 2017. URL https://proceedings.neurips.cc/paper/2017/hash/ 3f5ee243547dee91fbd053c1c4a845aa-Abstract.html. Guan Wang, Jin Li, Yuhao Sun, Xing Chen, Changling Liu, Yue Wu, Meng Lu, Sen Song, and Yasin Abbasi-Yadkori. Hierarchical reasoning model. CoRR, abs/2506.21734, 2025. doi: 10.48550/ ARXIV.2506.21734. URL https://doi.org/10.48550/arXiv.2506.21734. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL http://papers.nips.cc/paper_files/paper/2022/hash/ 9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html. Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie-Yan Liu. On layer normalization in the transformer architecture. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, Proceedings of Machine Learning Research, pages 10524–10533. PMLR, 2020. URL http://proceedings.mlr.press/v119/xiong20b.html. Liu Yang, Kangwook Lee, Robert D. Nowak, and Dimitris Papailiopoulos. Looped transformers are better at learning learning algorithms. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id=HHbRxoDTxE. Rui-Jie Zhu, Zixuan Wang, Kai Hua, Tianyu Zhang, Ziniu Li, Haoran Que, Boyi Wei, Zixin Wen, Fan Yin, He Xing, Lu Li, Jiajun Shi, Kaijing Ma, Shanda Li, Taylor Kergan, Andrew Smith, Xingwei Qu, Mude Hui, Bohong Wu, Qiyang Min, Hongzhi Huang, Xun Zhou, Wei Ye, Jiaheng Liu, Jian Yang, Yunfeng Shi, Chenghua Lin, Enduo Zhao, Tianle Cai, Ge Zhang, Wenhao Huang, Yoshua Bengio, and Jason Eshraghian. Scaling latent reasoning via looped language models. CoRR, abs/2510.25741, 2025. doi: 10.48550/ARXIV.2510.25741. URL https://doi.org/10.48550/ arXiv.2510.25741.

21

A

Proofs

A.1

Fixed-point iterations bound

Proof. We simplify our notation by omitting the pre-norm layer in Equation (2). We start by unrolling the computation across the L layers within a single fixed-point iteration. By recursively applying the layer update, we obtain
2L−1 2L−1 z2L + β1 · fθ2L 2L zi i = α1 · zi

2L−2 2L−1 = α12 · z2L−2 + α1 β1 · fθ2L−1 + β1 · fθ2L 2L zi 2L−1 zi i


2L−3 2L−2 2L−1 = α13 · z2L−3 + α12 β1 · fθ2L−2 + α1 β1 · fθ2L−1 + β1 · fθ2L 2L zi 2L−2 zi 2L−1 zi i .. .   2L−1 X j 2L−j  2L−j−1  . = α12L · z0i + β1 ·  α1 · fθ2L−j zi j=0

Now, substituting this expression into the fixed-point update gives z0i+1 = α2 · z2L i + β2 · x  = α2 · α12L · z0i + β1 ·

2L−1 X j=0

 







 + β2 · x α1j · fθ2L−j z2L−j−1 2L−j i



2L−1 X

= α2 α12L · z0i + α2 β1 · 

For compactness, define ρ = α2 α12L and si =

j=0

  + β2 · x. z2L−j−1 α1j · fθ2L−j 2L−j i

P2L−1 j=0

  2L−j−1 α1j · fθ2L−j z . Then the fixed-point 2L−j i

iteration can be written as z0i+1 = ρ · z0i + α2 β1 · si + β2 · x. Unrolling this recursion over fixed-point iterations gives ! ! i i X X 0 i+1 0 k k zi+1 = ρ · z0 + β2 ρ · x + α2 β1 ρ · si−k . k=0

k=0

Since 0 ≤ α1 , α2 < 1, we have 0 ≤ ρ < 1. Therefore, the geometric series is convergent. Taking norms and using the boundedness of each layer map, we get ! ! L−1  i i X X X j z0i+1 ≤ ρi+1 z00 + β2 ρk ∥x∥ + α2 β1 ρk  α1  cf . k=0

k=0

j=0

Letting i → ∞, the first term vanishes and the two geometric sums converge, which gives   β2 α2 β1 1 − α12L 0 lim sup zi ≤ ∥x∥ + cf . 1−ρ 1 − ρ 1 − α1 i→∞

Substituting back ρ = α2 α12L , we obtain lim sup i→∞

z0i

α2 β1 1 − α12L β2
≤ ∥x∥ + cf . 1 − α2 α12L 1 − α2 α12L (1 − α1 )

We now set β2 = 1 − α2 α12L . This makes the coefficient of ∥x∥ equal to 1. Furthermore, setting (1−α1 ) β1 = β21−α makes the coefficient of cf equal to α2 . Therefore, 2L ( 1 ) lim sup z0i ≤ ∥x∥ + α2 · cf . i→∞

In particular, if the fixed-point iteration converges to z0∞ , then This completes the proof.

z0∞ ≤ ∥x∥ + α2 · cf .

22

A.2

Contractive mapping

Proof. We first show that fθ (.; x) is a contraction with respect to z. For any z, z′ , using the Lipschitzness of the L-layer model, we have ∥fθ (z; x) − fθ (z′ ; x)∥ ≤ λf ∥(α2 · z + β2 · x) − (α2 · z′ + β2 · x)∥ = α2 λf ∥z − z′ ∥ .

Therefore, if 0 ≤ α2 λf < 1, the map fθ (.; x) is strictly contractive. By the Banach fixed-point theorem, it has a unique fixed-point z⋆ , and the iteration zi+1 = fθ (zi ; x) converges to z⋆ . We now prove the residual bound. Since zi+1 = fθ (zi ; x), the residual at iteration i can be written as ∥fθ (zi ; x) − zi ∥ = ∥zi+1 − zi ∥ . Using the contraction property of fθ (.; x), we get ∥zi+1 − zi ∥ = ∥fθ (zi ; x) − fθ (zi−1 ; x)∥ ≤ α2 λf ∥zi − zi−1 ∥ . Applying this inequality recursively gives i

∥zi+1 − zi ∥ ≤ (α2 λf ) ∥z1 − z0 ∥ . Since z1 = fθ (z0 ; x), we obtain i

∥fθ (zi ; x) − zi ∥ ≤ (α2 λf ) ∥fθ (z0 ; x) − z0 ∥ . This completes the proof. A.3

Mitigating oscillation through damping

Proof. We first show that the fixed-points of gη,θ ( · ; x) and fθ ( · ; x) coincide. Since
gη,θ (z; x) − z = η fθ (z; x) − z , and η > 0, we have gη,θ (z; x) = z if and only if fθ (z; x) = z. We now study the local stability of the damped iteration around z⋆ . The Jacobian of gη,θ ( · ; x) at z⋆ is ∂gη,θ ⋆ (z ; x) = (1 − η)I + ηJ, ∂z so for every eigenvalue λi of J, the corresponding eigenvalue of the damped Jacobian is µi (η) = 1 − η + ηλi = 1 + η(λi − 1). Local asymptotic stability is implied by |µi (η)| < 1 for all i. Writing λi = ai + i bi with ai = ℜ(λi ), |µi (η)|2 = 1 + 2η(ai − 1) + η 2 |λi − 1|2 . Hence |µi (η)| < 1 if and only if η|λi − 1|2 < 2(1 − ai ), i.e., 0 < η <

2(1 − ℜ(λi )) . |λi − 1|2

By assumption ℜ(λi ) < 1 for every i, so each upper bound is strictly positive. Setting   2(1 − ℜ(λi )) η0 = min 1, min > 0, i |λi − 1|2 we obtain |µi (η)| < 1 for every i and every η ∈ (0, η0 ). Therefore z⋆ is locally asymptotically stable under the damped iteration zt+1 = gη,θ (zt ; x), and the iterates converge to z⋆ from any sufficiently close initialization. 23

A.4

Error of truncated-BPTT

Proof. Since ∥J∥2 = σ < 1, the Neumann series is convergent, and we have (I − J) Therefore, the error of the k-term truncated approximation is (I − J)

−1

−

k−1 X

J

j

=

j=0

Using the relation ∥A∥F ≤ spectral norm, we get

∞ X

=

P∞

j j=0 J .

Jj

j=k

F

≤ √

∞ X

−1

F

Jj F .

j=k

D∥A∥2 for A ∈ RD×D , together with submultiplicativity of the Jj F ≤ ≤

=

√ √

√

D Jj 2 j

D ∥J∥2 D · σj .

Substituting this into the previous inequality gives (I − J)

−1

−

k−1 X

J

j

j=0

≤

√

D

σj

j=k

F

=

∞ X

√

D·

Thus, the approximation error decays as O(σ k ).

σk . 1−σ

∂L ⋆ θ To make the corresponding gradient statement explicit, let P = ∂f ∂θ (z ; x) and δ = ∂z⋆ . The −⊤ ⊤ exact implicit gradient is ∇θ L = P (I − J) δ, whereas the k-step truncated BPTT gradient is Pk−1 ⊤
j (k) ⊤ b ∇ L=P J δ. Therefore, j=0

θ

b (k) L ∇θ L − ∇ θ

2

≤ ∥P∥2 (I − J)

−⊤

−

k−1 X

j

j=0

  ∞ X j ≤ ∥P∥  ∥J∥  ∥δ∥ 2

J⊤

2

∥δ∥2 2

2

j=k

= ∥P∥2

σk ∥δ∥2 . 1−σ

Hence, the truncated BPTT gradient error also decays exponentially with the number of backward passes k. This completes the proof.

B

A Toy Failure Mode for Recurrent Post-norm

In this section, we provide a toy example to showcase a failure mode of post-norm in a small setting. We take random x, y ∈ Rn×d to be the input-output pair of sequences, with sequence length n = 100 and hidden-size d = 2, and set z00 = x. Let fθ (z; x) be a neural network with a single sub-layer fθ11 (z) defined as: fθ11 (z) = zW (w), where we define the rank-one map:

with θ1 = w.

W (w) = w1⊤ ∈ R2×2 ,

24

w = (w1 , w2 )⊤ ,

The post-norm recurrence is defined as z0i+1 = Normpost z0i + fθ1 z0i

,

,

while the pre-norm recurrence is defined as z0i+1 = (1 − β) · z0i + β · fθ1 Normpost z0i

β=

1 . 2

Figure 10 gives a minimal version of the normalization trade-off discussed in the main text based on this toy model. For both models we sweep a 200 × 200 grid over w ∈ G = [−5, 5]2 and plot ∆L(w) = L(w) − min L(v), v∈G

where we define the loss as L(w) =

1 2 z0 (w) − y F , nd 20

i.e., at the 20th effective layer. The figure illustrates that boundedness does not imply trainability. Post-norm keeps the recurrent state bounded by construction: after every step, each row is projected back onto the unit sphere. However, the same projection also removes radial information at every iteration. In this two-parameter slice, the resulting loss is organized into thin angular sectors with sharp ridges and narrow low-loss regions. Thus a random initialization of w is likely to start in a bounded but poorly conditioned part of the landscape, where the gradient does not point into a useful basin. This is the toy analogue of the optimization difficulty of recurrent post-norm layers. The right panel is not bare pre-norm; it is pre-norm with residual scaling. This matters because naive pre-norm removes the projection that controls the recurrent state and can lead to activation growth (Figure 2b). With the scaled update, each row satisfies ∥z0i+1 ∥2 ≤ (1 − β)∥z0i ∥2 + β. So the toy dynamics remain bounded while preserving a live residual stream. The broader low-loss region in Figure 10 is therefore consistent with the architecture we use in Theorem 1: pre-normalization improves signal propagation, while residual scaling replaces the boundedness mechanism that postnormalization provided.

C

Further Details About the Architecture

Fixed-point solver. Let zi ∈ RB×T ×d denote the ith latent representation (with B denoting the batch index, T the sequence index, and d the hidden size), and zi+1 the next latent representation. We index the bth batch dimension as zi [b]. Convergence is measured per sample zi [b] by the relative L∞ norm of the residual, ∥zi+1 [b] − zi [b]∥∞ ri [b] = ∈ R. ∥zi+1 [b]∥∞ + ϵ A sample is declared converged when ri [b] < τ . In practice, we set τ to 0.1. But we observe that for a reasonably small choice of τ , the model is not sensitive to this value. Two safeguards bound the loop: (1) a hard cap on the number of iterations, and (2) early termination if the adaptive step size collapses below a minimum.

Deep supervision. We adopt a similar deep supervision mechanism as HRM (Wang et al., 2025) and TRM (Jolicoeur-Martineau, 2025). Let Tsup denote the deep supervision interval. After every Tsup iterations, the intermediate activations of the model are decoded through the output head, the loss is computed, and truncated-BPTT is performed through the k latest iterations. Then, the computation graph is detached from the previous step. For each sequence, this process is continued until the fixed-point of the input is reached. The number of backward passes per forward pass is therefore ⌈k/Tsup ⌉. In our model, we set Tsup = k, while in TRM and HRM, Tsup is usually set to a larger number. However, we observe that in practice, FPRM performs a smaller number of forward and backward passes during training compared to TRM, lowering the training cost. 25

Figure 10: Landscape visualization for the setup proposed in Section B.

Depth-wise convolutions. Depth-wise convolutions have proven effective in improving the performance of looped models, at a small time and parameter complexity (Shu et al., 2026; Gao et al., 2025). Therefore, in FPRM we apply depth-wise convolutions on the latent representations at the beginning of each loop, which we find to be most effective. An overview of FPRM is available in Figure 3. We consider both 1D and 2D convolutions, and we find the 2D variant to be more effective at 2-dimensional tasks such as Sudoku and ARC, while the 1D variant is essential in state-tracking. However, as observed in Table 3, depth-wise convolutions seem to have a detrimental impact on the performance of TRM.

D

Description of Figure 1

In this figure, we categorize the puzzles into three groups: easy, medium, and hard. The grouping is based on difficulty, which is measured by the number of empty cells in the puzzle. The sample sizes for each difficulty level are balanced and set at around 1000 samples. For FPRM, we mark the halting decision for the entire group based on the residual of the group: if the mean residual is smaller than a pre-determined threshold (set to 0.1), then the model makes the halting decision. For TRM, the halting decision is marked when the ACT module signals halting for more than half of the samples. For the sake of exposition, we exclude the hardest puzzles from the groups, since they fail to halt at the current max. set budget of 10000 effective layers.

E

Fixed-point Residuals and Halting

We provide the test accuracy and the fixed-point residuals achieved by FPRM as a function of effective layer in Figure 12. The residuals for more difficult problems decay at a much slower rate, indicating that they demand more compute. Furthermore, accuracy stops improving at roughly the same effective layer where the residual plateaus, supporting the use of fixed-points as a halting criterion. 26

10−1

increasing difficulty

10−2

100

Test accuracy (%)

Number of samples

105

Residue norm

100

Train (N=1,001,000) Test (N=422,786)

104

103

102 45

48

51 54 57 Number of empty cells

80 60 40 20 0

60

101

102

103

104

Inference compute (effective layers)

Figure 11: Sudoku-Extreme dataset is imbalanced. The number of samples per difficulty level (number of empty cells).

Figure 12: FPRM allocates more compute to harder problems. Harder inputs need more iterations before halting and peak beyond the training compute limit (dashed line); color shows residual norm.

Table 3: Effect of adding FPRM’s architectural modifications to TRM: pre-norm and residual scaling (α2 only, or both α1 and α2 ), individually and in combination, evaluated with and without the conv2d layer in the TRM core. Each column reports the change in test sequence accuracy on Sudoku-Extreme (%) relative to its own post-norm, no-scale baseline measured in this sweep, with the absolute accuracy shown alongside. w/ conv w/o conv Configuration

∆ (%)

Acc. (%)

∆ (%)

Acc. (%)

Original TRM (post-norm, no residual scaling) + residual scaling (α2 only) + residual scaling (α1 , α2 ) − post-norm + pre-norm + residual scaling (α2 only) − post-norm + pre-norm + residual scaling (α1 , α2 )

— −6.71 −4.47 −49.00 −24.52

63.98 57.27 59.51 14.98 39.46

— −4.44 −12.24 −58.87 −52.83

72.60 68.16 60.36 13.73 19.77

F

How to effectively spend loops in TRM?

The default proposed TRM uses 16 deep-supervision steps (outer loops) and variable L- and Hsteps (inner loops). The L-steps outnumber the H-steps, typically by about 2×. However, other configurations for the number of loops spent for deep supervision vs. inner loops are possible. We test the performance of other configurations with experiments shown in Figure 13. We fix the L-to-H ratio at 2 and vary deep-supervision steps (segments) against per-segment recurrence depth (inner loops, shown with numbers next to the black markers in Figure 13). We measure test accuracy as a function of the number of deep-supervision steps, with fixed inference budget at approximately 1040 steps. This budget also matches the max. number of effective layers reached by the baseline FPRM on this task. This isolates how a fixed inference budget is best allocated: toward more outer refinement steps or deeper inner recurrence. Figure 13 shows that the budget is best spent on outer, deep-supervision steps. This matches the finding that, in TRM’s post-norm Transformer, the gains from added effective layer depth get smaller compared to FPRM (Figure 6). Fewer effective layers per segment is therefore the better strategy at a fixed compute budget. We adopt it for all experiments where we scale TRM compute (effective layers). 27

85

Accuracy (%)

84

10 78

83 136

82

21

81 36

8

14

29

FPRM (ours) TRM

50

104

Deep supervision steps Figure 13: The optimal way to spend the fixed looping compute is to maximize deep supervision steps. The numbers next to markers are inner recurrence depths per each deep supervision step. The total depth of effective layers is approximately the same across all configurations of TRM and FPRM on the Sudoku-Extreme task.

G

Additional Experimental Details

Weight initialization. It seems that initializing the weights using a truncated normal distribution (LeCun initialization) is common practice in looped architectures. In our experiments, it accelerates the convergence but there is very little material difference in sequence accuracy after convergence. Grokking. There is some evidence for grokking in looped architectures, but on the maze task we observe convergence on the training data. And training the models for a longer period (up to 7 days) did not yield better performance. Hyperparameters, device specification We provide the values for some of the most important hyperparameters in the paper, per each model and dataset.

28

Table 4: Hyperparameters for Sudoku-Extreme experiments (Table 1 of the paper). Shared across all models: 1×A100-40GB, batch 768, 60 000 epochs, constant LR after 2 000-step warm-up, EMA enabled (rate 0.999), puzzle-embedding length 16. All models are trained with AdamW (Loshchilov and Hutter, 2019). TRM

FPRM

3 6 0 2 = L-cycles + 1

– – 0 2 6

Halting mechanism halt_max_steps max_iter (train) max_iter (eval) stepsize-decay / patience (eval) fp_thresh

ACT 16 – – – –

fixed-point – 12 35 000 0.997 / 10 0.1

Block / signal-prop modifications norm type residual scaling α1 , α2 init conv branch

post-norm ✗ – –

pre-norm ✓ 0.75, 0.25 2D-conv (3 × 3 kernel)

10−4 1.0 10−4 1.0

10−3 10−3 10−3 10−3

Looping structure H-cycles L-cycles H-layers L-layers nback

Optimizer learning rate weight decay puzzle-emb LR puzzle-emb WD

29

Table 5: Hyperparameters for Maze-Hard experiments (Table 1 of the paper). Shared: trained on maze-30x30-hard-1k without augmentation, 4×A100-80GB, constant LR after a 2 000-step warm-up, EMA enabled (rate 0.999), puzzle-embedding length 16. FPRM trains for 60 000 epochs (TRM 50 000). FPRM is trained using Adam-Atan2 (Everett et al., 2024); TRM is trained using AdamW. TRM

FPRM

3 4 0 2 = L-cycles + 1

– – 0 2 6

Halting mechanism halt_max_steps max_iter (train) max_iter (eval) stepsize-decay / patience (eval) fp_thresh

ACT 16 – – – –

fixed-point – 24 35 000 0.996 / 10 0.1

Block / signal-prop modifications norm type residual scaling α1 , α2 init conv branch

post-norm ✗ – –

pre-norm ✓ 0.75, 0.25 1D-conv (1 × 4 kernel)

10−4 1.0 10−4 1.0

10−4 1.0 10−2 1.0

Looping structure H-cycles L-cycles H-layers L-layers nback

Optimizer learning rate weight decay puzzle-emb LR puzzle-emb WD

30

Table 6: Hyperparameters for state-tracking experiments on A5 and S5 (Figure 4 of the paper) and for the Looped Transformer signal-propagation analysis (Figure 2). Shared: 1×A100-80GB, global batch 1024, Adam-Atan2, no LR warm-up, EMA disabled, no puzzle embedding (puzzle_emb_len=0). Trained at ktrain =32, evaluated for k ∈ [2, 128]. TRM and FPRM train for 50 epochs; the Looped Transformer analysis (Fig. 2) for 30. Looping structure H-cycles L-cycles H-layers L-layers nback

TRM

FPRM

Looped Transformer (Fig. 2)

2 4 0 4 = L-cycles + 1

– – 0 2 4

– – 0 2 4

fixed-point – 128 deterministic

fixed iters – = ktrain deterministic

pre-norm none ✓ 0.5, 0.5 1D-conv (1 × 4 kernel)

sweep† none sweep† sweep† –

10−3 10−2

10−4 10−2

Halting mechanism fixed iters or ACT (inference) halt_max_steps 16 max_iter – iter. distribution – Block / signal-prop modifications norm type post-norm norm placement – residual scaling ✗ α1 , α2 init – conv branch – Optimizer learning rate weight decay

10−4 10−2

† The Looped Transformer row sweeps the {post-norm, pre-norm, pre-norm + residual-scaling} variants from Figure 2a; the

residual-scaling variant uses α1 =0.75, α2 =0.5 and ktrain ∈ {8, 16, 32, 64}.

31

Table 7: Hyperparameters for the FPRM ARC-AGI experiments. The two runs share an identical configuration and differ only in the training corpus (ARC1-Concept vs. ARC2-Concept, both with 1000 augmentations per sample). Shared across both: 4×A100-80GB, batch 768, 100 000 epochs, constant LR with no warm-up, EMA enabled (rate 0.999), puzzle-embedding length 16, hidden size 512, 8 heads, MLP expansion 4, RoPE position encodings. Both models are trained with Adam-Atan2 (β1 =0.9, β2 =0.95). FPRM Looping structure H-cycles L-cycles H-layers L-layers nback

– – 0 2 6

Halting mechanism halt_max_steps max_iter (train) max_iter (eval) stepsize / decay / patience (eval) fp_thresh

fixed-point – 8 1000 1.0 / 0.9 / 5 0.1

Block / signal-prop modifications norm type pre-norm residual scaling ✓ α1 , α2 init 0.75, 0.25 conv branch 1D-conv (kernel 4) Optimizer learning rate weight decay puzzle-emb LR puzzle-emb WD

10−3 10−2 10−2 1.0

32