Training Crossroads for Recurrent Vision Transformers: Recurrence, Neural ODEs, and Deep Supervision

TRAINING CROSSROADS FOR RECURRENT VISION TRANSFORMERS: RECURRENCE, NEURAL ODES, AND DEEP SUPERVISION Grzegorz Gruszczynski1∗

Pawel Olszowiec1

Michal Byra1,2

Grzegorz Stefanski1

Alberto Presta1

1

2

Samsung AI Center, Warsaw, Poland Institute of Fundamental Technological Research, Polish Academy of Sciences, Warsaw, Poland

arXiv:2608.04879v1 [cs.LG] 5 Aug 2026

ABSTRACT Vision Transformers (ViTs) achieve strong image-recognition performance, but their parameter count grows linearly with depth when each block is independently parameterized. Singleblock recurrent ViTs (bViT) remove this growth by repeatedly applying one shared block. Rather than proposing a new architecture, we fix a bViT and provide a controlled empirical characterization of three training and inference regimes under a common CIFAR-100 protocol, asking: (i) when does recurrence beat independently parameterized depth—at matched FLOPs or at matched parameter memory? (ii) when a residual recurrent block is trained through an ODE solver, does solver order act as numerical refinement or as an architectural bias? and (iii) what does robustness beyond the training horizon cost in nominal accuracy? We find that standard ViTs remain preferable when FLOPs are the primary constraint, whereas recurrent ViTs offer a better accuracy–parameter trade-off under memory constraints. Consistent with the standard view of residual networks as Euler discretizations of ODEs, the continuous-time analogue of a residual recurrent block is the state-subtracted vector field ż = Fθ (z) − z; although known in principle, this distinction is easy to violate when the block is wrapped as a black-box vector field, and we qualify the cost at few accuracy points. Because the vector field is learned jointly with the solver, higher-order solvers act as a solver-induced architectural bias rather than a numerical-accuracy improvement, and their gains are not uniform. Finally, stage-wise deep supervision traces an accuracy–robustness frontier: it does not improve nominal accuracy, but degrades gracefully far beyond the training horizon, where naive recurrence collapses to near-random performance. Index Terms— vision transformer, recurrence, parameter sharing, neural ODE, deep supervision, image classification 1. INTRODUCTION AND RELATED WORK Vision Transformers (ViTs) have established state-of-the-art performance in image recognition but incur high memory costs due to their deep, independently parameterized block *Corresponding author. e-mail: g.gruszczyns@samsung.com

structures [1, 2, 3, 4]. While empirical evidence suggests that transformer layers often perform highly related computations and can be compressed or merged [5, 6, 7], standard architectures still scale their parameter footprint linearly with depth. Recurrence and parameter sharing offer a mechanism to decouple computational depth from parameter count, predominantly in language modeling, synthetic reasoning, or through partial sharing, channel slicing and distillation in vision [8, 9, 10, 11, 12, 13]. In contrast to these designs, the single-block Vision Transformer (bViT) investigates the extreme limit of full parameter sharing, replacing the entire sequence of unique layers with a single, recurrently applied operator trained from scratch [14].

2. ARCHITECTURE AND TRAINING PROCEDURE We fix the recurrent ViT architecture and dataset, and compare alternative training/inference paradigms. The purpose of this study is not to establish a new SOTA architecture for image recognition task, but to identify how different computational training regimes affect accuracy, compute, parameter efficiency, solver behavior, and extrapolation stability. We follow the standard Vision Transformer image-classification pipeline: an input image is split into non-overlapping patches, projected into token embeddings, pre-pended with a learnable CLS token, and augmented with positional embeddings. The resulting sequence is processed by pre-norm transformer blocks with multi-head self-attention and feed-forward sublayers, and the final normalized CLS token is passed to a linear classification head, see Fig. 1. We adopt the single-block recurrent topology of Byra et al. [14]: the stack of independently parameterized blocks F1 , . . . , FL of a standard ViT is replaced by a single shared transformer block F applied recurrently: zt+1 = Fθ (zt ) where t = 0, . . . , T − 1. Figure 1 shows that the bViT can be interpreted as an unrolled ViT with full parameter sharing across depth. By reusing the same multi-head self-attention and feedforward components at every step, bViT substantially reduces the model’s overall parameter count and memory footprint while preserving the standard computational budget.

Standard ViT Img Embed

Block 1

Block 2

···

Block N

Classifier

Shared Block

Classifier

Recurrent bViT Img Embed

Shared Block

Shared Block

···

Fig. 1: Standard ViT (upper) vs. bViT (lower) architecture.

2.1. Evaluations We evaluate the models on an image classification task. We use CIFAR-100 [15] dataset, which contains 100 distinct classes of RGB images, each corresponding to 600 images, 500 and 100 for training and testing, respectively. Models are trained for 200 epochs over three independent runs. We report the mean best test accuracy across runs, with per-configuration standard deviations ranging from ±0.001 to ±0.004. Training is performed using the AdamW optimizer [16] with a learning rate of 1 × 10−3 and a weight decay coefficient of 0.05 to minimize the standard cross-entropy loss. The default network and training hyperparameters are configured as follows [14]: an embedding dimension of 256, a patch size of 4, and a dropout rate of 0.1. An Exponential Moving Average (EMA) is maintained for model weights with a decay rate of 0.999. All models are trained using a standard batch size of 128. The model utilizes a SwiGLU activation function and the final layer normalization is applied. The bViT architecture consists of single parameter-sharing block evaluated recurrently 12 times. On the other hand, the classical ViT has 12 independently parameterized blocks. 3. EXPERIMENTS Unless stated otherwise, we keep the architecture fixed and evaluate different training recipes. In the following subsections we investigate the tradeoff between computations and model parameters, continuous time Neural ODE integration, and deep-supervised recurrent training. 3.1. Architecture comparison The comparison in Fig. 2 indicates that recurrent weight sharing in bViT induces a clear resource-dependent trade-off rather than a uniformly superior architecture. Under matched computational budgets, the classical ViT remains preferable, as its independently parameterized blocks provide higher accuracy for a given FLOPs budget. In contrast, when the dominant constraint is parameter storage or deployment memory, bViT becomes advantageous: by reusing a single transformer block across recurrent steps, it substantially reduces the parameter footprint while retaining competitive accuracy. Thus, standard

ViTs are better suited to compute-limited regimes, whereas bViT offers a more favorable accuracy–parameter trade-off for memory-constrained deployment. Next, we performed a detailed grid search experiment (see Fig. 3) to show how the transformer depth, D, (governed by the number of unique blocks) and steps, S, (recurrent repetition of the set of block) interacts with the effective representational power. Below a critical depth or steps threshold, the shared representational space lacks the capacity to encode the required functional diversity. Recurrence can unroll representation space just as effectively as structural depth assuming that the number of network parameters is sufficient. However, when the overall virtual length (steps multiplied by depth) becomes to long the model looses accuracy due to numerical discrepancies in gradient propagation.

3.2. bViT as a Neural ODE (bViT-node) The bViT update zn+1 = Fθ (zn ) is a discrete-time dynamical system and can be reinterpreted in continuous time [17]. Since a transformer block is residual, we may write Fθ (z) = z + Rθ (z), where Rθ denotes the non-identity transformation induced by attention, normalization, and feed-forward sublayers. Consistent with the standard view of residual networks as Euler discretizations of ODEs [18, 19, 20], the continuoustime analogue of the residual recurrence is the state-subtracted vector field ż = Fθ (z) − z = Rθ (z): explicit Euler integration with ∆t = 1 gives zn+1 = zn + (Fθ (zn ) − zn ) = Fθ (zn ), exactly recovering the discrete bViT recurrence. Although this correspondence is well known in principle, it is easy to violate in practice when the block is wrapped as a black-box vector field, as in standard Neural ODE toolkits: setting ż = Fθ (z) yields, under Euler with ∆t = 1, zn+1 = zn + Fθ (zn ) = 2zn + Rθ (zn ), which accumulates the identity component at every step. We therefore include ż = Fθ (z) as a control condition and quantify the cost of this mis-specification for recurrent ViTs. We evaluate explicit Euler, Heun2, and RK4 solvers applied to both vector fields. Table 1 shows that the statesubtracted formulation consistently outperforms the control by roughly 3–6 accuracy points—a nontrivial penalty for an error that is invisible at the API level of common ODE libraries. At matched or comparable numbers of function evaluations (NFEs), higher-order solvers do not uniformly improve performance despite their higher formal order. Because the vector field is learned jointly with the solver, higher-order methods alter the optimization landscape by compounding multi-stage gradient dependencies within each step; the resulting gains should therefore be read as a solver-induced architectural bias rather than a numerical-accuracy improvement. Overall, explicit Euler remains a highly competitive and robust baseline, and the optimal solver choice depends on both the block depth and the target discretization horizon.

(a) Accuracy grows as the FLOPs budget is (b) When the number of total network param- (c) As the bViT shares the parameters between increased. The ViT outperforms the bViT. Fi- eters is reduced, then accuracy of the bViT is blocks, the computational cost strictly correbetter compared to ViT. nally, both models reach common plateau. lates with embedding dimension.

Fig. 2: The trade-off between the classical ViT and the recurrent bViT, evaluated across compute and scale constraints. 3.3. Deep Supervision and Extrapolation (bViT-ds)

Fig. 3: Heat map of max accuracy. Sharp decline is visible when the network lacks capacity on either depth, D, or steps, S, axis. Notice that the setup with D = 1 and S = 12 represent the bViT, while D = 12 and S = 1 is the classical ViT. Other combinations of D, S corresponds to multi-block recursive variants. For example model with D = 12 and S = 12 would have 12 independent blocks in a stack, and the stack would be evaluated recursively 12 times. Such a long sequence (144 evaluations) suffers from degraded accuracy due to vanishing gradient problems.

Standard bViT is trained for a fixed recurrent horizon (T), and it learns a finite-horizon computation rather than necessarily learning a contractive map with a stable fixed point [14]. When evaluated for more recurrent steps than seen during training, the latent trajectory may drift away from the region in which the classifier was trained, causing extrapolation failure. Deep Equilibrium (DEQ) models [21, 22] attempt to resolve this by framing the forward pass as an implicit rootfinding problem, seeking z ⋆ = Fθ (z ⋆ ). However, in our experiments both the DEQ root-finding solvers (e.g., Anderson acceleration) and TRM-style supervision [23] applied directly to bViT turned out to be divergent. Related supervision strategies are studied in [23, 24, 25]. In this work, we introduce bViT-ds to bridge the gap between finite-horizon reasoning and asymptotic stability. This bViT variant partitions the unrolled computational graph into K discrete supervision stages. Within each supervision stage k ∈ {1, 2, 4, 8} of bViT-ds, the parameter sharing transformer block is evaluated for S steps using a simple fixed point iteration, zt = Fθ (zt−1 ). At the end of each stage, k, the intermediate logits ŷk are and a Pcomputed K 1 cross-entropy penalty is applied, Ltotal = K L (ŷ k=1 CE k , y). The O(1) memory scaling with respect to unrolled depth is achieved by applying state detachment after each group of S steps. Minimizing the loss from detached intermediate states explicitly penalizes trajectory divergence and constrain the phase space required for stability. Table 2 shows that simply increasing the recurrent horizon does not monotonically improve performance. Accuracy initially improves with additional recurrent steps, but eventually saturates and slowly degrades. When the same model is extrapolated beyond its training horizon, accuracy can collapse to near-random level. Stage-wise deep supervision substantially

Table 1: Comparison of integration schemes. The best and second best results for each of the integrators (Euler, Heun2, RK4) are highlighted with green and light green colors, respectively. Here, D and S denotes the number of unique transformer blocks and the number of solver steps respectively. NFE is the total number of transformer-block evaluations. “NaN” denotes a diverged training run.

D

S

Euler NFE Acc↑

Heun2 NFE Acc↑

RK4 NFE Acc↑

NODE: dz dt = Fθ (z) − z (state-subtracted) 1 1 1 1 2 2 2 2 4 4 4 4

3 6 12 24 3 6 12 24 3 6 12 24

3 6 12 24 6 12 24 48 12 24 48 96

0.630 0.656 0.658 0.667 0.658 0.664 0.668 0.682 0.659 0.669 0.677 0.669

6 12 24 48 12 24 48 96 24 48 96 192

0.654 0.657 0.663 0.670 0.663 0.670 0.675 0.689 0.673 0.676 0.701 0.633

12 24 48 96 24 48 96 192 48 96 192 384

0.655 0.660 0.665 0.671 0.665 0.668 0.674 0.692 0.673 0.675 0.677 NaN

12 24 48 96 24 48 96 192 48 96 192 384

0.637 0.635 0.631 0.631 0.651 0.652 0.654 0.653 0.664 0.668 0.666 NaN

NODE: dz dt = Fθ (z) (control) 1 1 1 1 2 2 2 2 4 4 4 4

3 6 12 24 3 6 12 24 3 6 12 24

3 6 12 24 6 12 24 48 12 24 48 96

0.591 0.611 0.608 0.606 0.640 0.642 0.642 0.641 0.658 0.657 0.661 0.659

6 12 24 48 12 24 48 96 24 48 96 192

0.618 0.621 0.619 0.622 0.646 0.650 0.648 0.648 0.658 0.660 0.663 0.664

improves extrapolation robustness—but it does not improve nominal accuracy: whenever the horizon is known at training time, naive extension is strictly better assuming sufficient GPU memory is available. 4. CONCLUSION This work characterizes recurrent ViT training through the lens of deployment trade-offs, continuous-time modeling, and extrapolation stability, answering the three research questions. (i) Recurrence is not a universally superior replacement for architectural depth: standard ViTs remain stronger at matched FLOPs, while bViTs provide a better accuracy–parameter trade-off when the model-memory footprint is the binding con-

Table 2: Nominal accuracy vs. extrapolation robustness. Increasing the number of steps within a single stage improves accuracy, which then plateaus and slowly degrades, whereas naive extrapolation beyond the training horizon collapses. Deep supervision (bViT-ds) helps constrain trajectory divergence and improves extrapolation robustness at the price of nominal accuracy. In bViT-ds we set each stage to consist of 12 steps. During training, the computational graph is detached after each stage to maintain O(1) memory depth. During inference, we add additional extrapolation stages beyond the training stages. The number of function evaluations (overall) is denoted as NFE. The best and second best results are highlighted with green and light green colors, respectively.

Acc↑

Steps per stage

Supervision stages

Extrapolation stages

NFE

bViT baseline and naive extrapolation. 0.658 0.029 0.017

12 12 12

1 1 1

0 4 8

12 60 108

Naive increase of the bViT loop horizon. 0.667 0.681 0.682 0.680 0.679 0.676

24 48 60 72 84 96

1 1 1 1 1 1

0 0 0 0 0 0

24 48 60 72 84 96

0 0 4 4 0 8 8 4 8

24 48 72 96 96 120 144 144 192

bViT-ds variants. 0.650 0.639 0.599 0.620 0.635 0.467 0.588 0.630 0.613

12 12 12 12 12 12 12 12 12

2 4 2 4 8 2 4 8 8

straint. (ii) The continuous-time analogue of a residual recurrent block is the state-subtracted vector field ż = Fθ (z)−z—a known correspondence whose violation we show costs 3–6 accuracy points. Higher-order solver gains are NFE-dependent and best interpreted as solver-induced architectural bias rather than numerical-accuracy improvement. (iii) Stage-wise deep supervision buys long-horizon robustness at the price of nominal accuracy and additional computation.

Funding acknowledgements This work was supported by Samsung AI Center, Warsaw.

Compliance with ethical standards The authors have no relevant conflicts of interest to disclose. 5. REFERENCES [1] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin, “Emerging properties in self-supervised vision transformers,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 9650–9660. [2] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” Proceedings of the International Conference on Learning Representations (ICLR), 2020. [3] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick, “Masked autoencoders are scalable vision learners,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 16000– 16009. [4] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou, “Training data-efficient image transformers & distillation through attention,” in International conference on machine learning. PMLR, 2021, pp. 10347–10357. [5] Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha, “Evolutionary optimization of model merging recipes,” Nature Machine Intelligence, vol. 7, no. 2, pp. 195–204, 2025. [6] Isha Garg, Christian Koguchi, Eshan Verma, and Daniel Ulbricht, “Revealing the utilized rank of subspaces of learning in neural networks,” in Proceedings of the AAAI Symposium Series, 2025, vol. 5, pp. 151–158. [7] Qi Sun, Marc Pickett, Aakash Kumar Nain, and Llion Jones, “Transformer layers as painters,” in Proceedings of the AAAI Conference on Artificial Intelligence, 2025, vol. 39, pp. 25219– 25227. [8] Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser, “Universal transformers,” Proceedings of the International Conference on Learning Representations (ICLR), 2019. [9] Mozes Jacobs, Thomas Fel, Richard Hakim, Alessandra Brondetta, Demba Ba, and T Andy Keller, “Block-recurrent dynamics in vision transformers,” arXiv preprint arXiv:2512.19941, 2025. [10] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut, “Albert: A lite bert for self-supervised learning of language representations,” Proceedings of the International Conference on Learning Representations (ICLR), 2020.

[11] Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, and Sashank J Reddi, “Reasoning with latent thoughts: On the power of looped transformers,” Proceedings of the International Conference on Learning Representations (ICLR), 2025. [12] Jinnian Zhang, Houwen Peng, Kan Wu, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan, “Minivit: Compressing vision transformers with weight multiplexing,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 12145–12154. [13] Zhiqiang Shen, Zechun Liu, and Eric Xing, “Sliced recursive transformer,” in European Conference on Computer Vision. Springer, 2022, pp. 727–744. [14] Michal Byra, Pawel Olszowiec, Grzegorz Stefanski, Grzegorz Gruszczynski, and Alberto Presta, “bvit: Investigating singleblock recurrence in vision transformers for image recognition,” 2026. [15] Alex Krizhevsky, Geoffrey Hinton, et al., “Learning multiple layers of features from tiny images,” 2009. [16] Ilya Loshchilov and Frank Hutter, “Decoupled weight decay regularization,” Proceedings of the International Conference on Learning Representations (ICLR), 2019. [17] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud, “Neural ordinary differential equations,” Advances in neural information processing systems, vol. 31, 2018. [18] Weinan E, “A proposal on machine learning via dynamical systems,” Communications in Mathematics and Statistics, vol. 5, no. 1, pp. 1–11, 2017. [19] Eldad Haber and Lars Ruthotto, “Stable architectures for deep neural networks,” Inverse Problems, vol. 34, no. 1, pp. 014004, 2017. [20] Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong, “Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations,” in Proceedings of the International Conference on Machine Learning (ICML), 2018, pp. 3276–3285. [21] Shaojie Bai, J Zico Kolter, and Vladlen Koltun, “Deep equilibrium models,” Advances in Neural Information Processing Systems (NeurIPS), vol. 32, 2019. [22] Shaojie Bai, Vladlen Koltun, and J Zico Kolter, “Multiscale deep equilibrium models,” Advances in neural information processing systems, vol. 33, pp. 5238–5250, 2020. [23] Alexia Jolicoeur-Martineau, “Less is more: Recursive reasoning with tiny networks,” arXiv preprint arXiv:2510.04871, 2025. [24] Guan Wang, Jin Li, Yuhao Sun, Xing Chen, Changling Liu, Yue Wu, Meng Lu, Sen Song, and Yasin Abbasi Yadkori, “Hierarchical reasoning model,” arXiv preprint arXiv:2506.21734, 2025. [25] Qianli Liao and Tomaso Poggio, “Simple recursive model: Simplified, single-state reasoning with skip connections,” Preprint, 2026.

Related documents

• Scaffold Then Internalize: Representation Injection for Diffusion Transformers#1129885
• Learning to Rotate: Temporal and Semantic Rotary Encoding for Sequential Modeling#138955
• What Visual Generators Need from Teachers: Rethinking Representation Alignment#1128990
• Data for Sequence learning in the visual and motor domain#965551
• MotionSpaceFlow: Representation-Aware Flow Matching in Direct Motion Space#1119263