arXiv CSopen access

Elastic Attention Cores for Scalable Vision Transformers

Elastic Attention Cores for Scalable Vision Transformers Alan Z. Song†1

arXiv:2605.12491v1 [cs.CV] 12 May 2026

Rui Zhang2

Yinjie Chen†2

Jiahang Cao2

Weijian Mai2

Carnegie Mellon University

†

Co-first authors, equal contribution

Muquan Yu2

Michael J. Tarr1

Deva Ramanan1 1

Mu Nan⋄2

2

Hossein Adeli3

Andrew F. Luo∗2

University of Hong Kong

3

Columbia University

⋄

∗

Corresponding author

Additional core contributor

A.Z.S.: [email protected] Y.C.: [email protected] [email protected] [email protected] [email protected]

Abstract Vision Transformers (ViTs) achieve strong data-driven scaling by leveraging all-toall self-attention. However, this flexibility incurs a computational cost that scales quadratically with image resolution, limiting ViTs in high-resolution domains. Underlying this approach is the assumption that pairwise token interactions are necessary for learning rich visual-semantic representations. In this work, we challenge this assumption, demonstrating that effective visual representations can be learned without any direct patch-to-patch interaction. We propose VECA (Visual Elastic Core Attention), a vision transformer architecture that uses efficient linear-time core-periphery structured attention enabled by a small set of learned cores. In VECA, these cores act as a communication interface: patch tokens exchange information exclusively through the core tokens, which are initialized from scratch and propagated across layers. Because the N image patches only directly interact with a resolution invariant set of C learned “core” embeddings, this yields linear complexity O(N ) for predetermined C, which bypasses quadratic scaling. Compared to prior cross-attention architectures, VECA maintains and iteratively updates the full set of N input tokens, avoiding a small C-way bottleneck. Combined with nested training along the core axis, our model can elastically trade off compute and accuracy during inference. Across classification and dense tasks, VECA achieves performance competitive with the latest vision foundation models while reducing computational cost. Our results establish elastic core-periphery attention as a scalable alternative building block for Vision Transformers. Project repository here.

1

Introduction

Recent advances in computer vision have been driven by increases in both computational resources and dataset scale. These developments have reshaped architectural design, driving a shift away from strong inductive biases and towards more flexible, data-driven models [1, 2, 3, 4, 5]. Whereas convolutional neural networks directly encode locality and translation equivariance into their structure [6, 7], these design choices constrain the representations that can be learned at scale. As datasets have grown, the value of these inductive biases has diminished. Vision Transformers (ViTs [8]) replace convolutions with self-attention and impose only weak spatial constraints through patchification, enabling consistent scaling and global receptive fields at every layer [9, 10]. Yet this flexibility comes at a substantial computational cost. ViTs rely on dense token-to-token interactions, where every

Core-Periphery

All-to-All

70 Ours-Base DINOv3-Base

60

Patch token (periphery)

50

RGB Image

Res=256

Res=384

FLOPs(G)



40 30

20 10

Core token

0

no direct patch-patch attention

256 384 512

Res=512



Res=768

Res=1024

768

Resolution (px)

89

1024

88 87

Ours (C=64)

DINOv3

DINOv2-Reg

CLIP

mloU

RGB Image

86 85

Ours-Base DINOv3-Base

84 83

RGB Image

Input

Output

8

16 24 32 40 48 56 64

Number of Core Tokens

Figure 1: Outline of our core-periphery attention structure. (a) Self-attention utilizes a fully connected attention matrix with N 2 comparisons given N input patches. VECA constructs a coreperiphery matrix with C core tokens that form a clique, requiring only 2N C + C 2 comparisons. (b) We visualize our dense features with UMAP. Our method produce stable embeddings across resolutions. (c) Top: We visualize our features and [CLS]-dense cosine similarity of our model, comparing such visuals to its teacher (DINOv3) as well as other baseline embeddings. Bottom: We visualize core attention weights with UMAP. Core attention maps start off isotropic (spherical), and become increasingly semantic. (d) Top: FLOPs of our attention block versus DINOv3 across resolutions. Bottom: We can elastically vary the number of tokens, trading off accuracy and speed.

patch attends to every other patch, leading to quadratic complexity that becomes prohibitive at high resolutions [11, 12]. Prior work has sought to accelerate self-attention by approximating the softmax operator. While these methods reduce computational cost, they still largely seek to preserve the underlying self-attention mechanism. We illustrate our framework in Figure 1. In this work, we revisit the necessity for dense patch-to-patch self-attention by introducing Visual Elastic Core Attention (VECA). VECA replaces standard self-attention with a block-sparse attention matrix mediated by a small, resolution invariant set of C learned core tokens. When viewed as an attention connectivity graph, this forms a core-periphery network [13, 14]. The core tokens constitute a fully connected clique [15] that interacts with all peripheral image patches, eliminating the need for patches to directly attend to one another. By routing information through this set of core tokens, the computational cost of the attention operation becomes (2N C + C 2 ), where N is the number of image patches. For a predetermined C, this yields linear complexity relative to image resolution. We leverage this architecture to enable adaptive computation by applying nested dropout [16] to the core tokens during training. This encourages the network to learn ordered representations, implicitly ranking the core tokens. At test time, this enables elastic adjustments to the inference cost. Building upon recent work that successfully distill representations from strong vision foundation models [17, 18, 19, 20], we supervise our model using a DINOv3 [21] teacher. We observe emergent behavior in core-to-patch spatial attention maps across layers, which evolve from isotropic (blob like) to semantically-aligned groups. Notably, this isotropic-to-semantic transform emerges without any explicit constraints encouraging such behavior. Elastically decreasing the number of cores during inference impacts early-network attention granularity, yielding noticeably coarser maps. Concretely, our contributions are as follows: (1) We propose VECA, a visual backbone that replaces quadratic self-attention with linear-time core-periphery attention in every layer, forcing token interactions through a resolution-invariant bottleneck. (2) We evaluate VECA on both classification and dense spatial tasks, where our model achieves competitive performance with the DINOv3 teacher. At ViT-B size, our model achieves less than 2% top-1 Acc. difference (81.93 versus 83.56) on Imagenet-1k classification, and less than 0.28 mIoU difference (57.46 versus 57.74 – despite 87.1% fewer attention interactions per layer) on the challenging PASCAL context segmentation benchmark. (3) We explore the properties of the core tokens. We identify emergent behavior and empirically show they exhibit object centric representations. Our work suggests that elastic core-periphery attention is a promising and scalable building block of Vision Transformers. 2

Visual Elastic Core Attention

P

Patch token

SwiGLU FFN

Unused

Core token

Transformer Block (Linear Time)

Layer Norm

Core coordinate

Linear Projection Layer 3

Layer 5

VECA 2-D RoPE Layer Norm

Sampled budget

Possible budgets

 RGB Image

R Position Linear

Layer 12

Layer 3

C=8

Layer 5

C=24

Layer 12

Layer 3

Layer 5

Layer 12

C=64

Figure 2: Architecture of VECA. (a) Our attention matrix is defined using a core-periphery structure with a graph diameter of 2. For Np patch tokens, and C active core tokens, the total connections are 2N C + C 2 . For each layer: we transform the patches, cores, and predict the spatial coordinate for each core. (b) By varying the number of core tokens, the core-patch attention granularity changes. With fewer cores, each core attends to a larger region. (c) Attention granularity varies with core count. We perform joint UMAP of core-patch attentions across multiple layers. Cores attend to larger regions when fewer are used and evolve to become more semantic at deeper layers.

2

Related work

Elastic Models and Representations. Scalable inference has explored elastic representations and adjustable cost networks [22]. Matryoshka representation learning [23] produces truncatable embeddings, while elastic neural networks enable adjustable model capacity. Parameter hierarchies can be learned via nested dropout [16, 24, 25, 26], yielding ordered components [27]. This has been further extended to width-adaptive architectures [28, 29, 30]. Such structural elasticity supports sub-network extraction [31, 32, 33, 34] with transformer specific designs [35, 36]. Input-dependent computation can be achieved via early-exiting [37, 38, 39] or iterative processing [40, 41, 42, 43]. At the data-level, elasticity can utilize token merging [44], scratchpads [45], or iterative expansion [46]. Sparse expert selection offers an additional axis of adjustment [47, 48]. In vision, recent methods leverage spatial redundancies [49, 50]. Matformer introduces elasticity along the channel dimension, which is orthogonal to our approach. Closer to our work, MQT proposes elastic queries [51], but functions as a single layer, on a frozen CLIP model which itself is quadratic time. In contrast, our model is an elastic visual backbone that is competitive with state-of-the-art vision foundation models across both classification and dense tasks. Efficient Attention. To mitigate the quadratic time cost of self-attention, some models replace softmax with approximations [52, 53, 54, 55, 56, 57] or low-rank factorization [58, 59], while others use grouping, or sliding windows [60, 61, 62, 63, 64] or completely abandon attention in favor of unparameterized transforms [4, 65]. Fixed latent bottlenecks in Set Transformers and Perceiver [66, 67, 68, 69], fixed size nested softmax [70], and multimodal fusion [71] have also been used. Recent work has shifted towards linear RNNs for sequences [72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84]. In vision, linear attention can be applied with some performance tradeoffs [85, 86, 87, 88, 89]. SwinViT and PVT perform self-attention in hierarchical local windows [90, 91], axial transformers [92] decompose attention along each axis, while Agent Attention leverages nested linear attention [93]. Our work is complementary to linear attention methods, as we achieve linear time while using softmax. Unlike Agent Attention, we do not require nested softmax. Unlike Perceiver which only refines latent queries, our model iteratively transforms both input and queries, enabling dense outputs. VECA is also natively elastic, which allows for test-time tradeoff of compute and accuracy. Differentiable Clustering and Hierarchies. Learnable clustering jointly optimizes representations and their underlying groupings. Early approaches relied on discrete assignment [94, 95, 96], while recent approaches use continuous relaxations [97, 98, 99, 100]. Unknown cluster counts can be 3

RGB Image

VECA (C=64)

DINOv3

DINOv2-Reg

CLIP

256 512 1024

Figure 3: Learned dense representations. We compare the UMAP visualizations of dense features and [CLS]-dense cosine similarity maps under increasing input resolutions with those of representative ViT backbones. The learned representations of VECA demonstrate high-quality clean dense features and remain robust at higher resolutions. inferred via bayesian methods, density estimation, or pruning [101, 102, 103, 104, 105, 106]. To capture nested semantic structure, prior work has explored learnable hierarchical embeddings [107, 108, 109, 110]. In our model, our “core” tokens can be interpreted as cluster seeds, which we observe to become decreasingly isotropic (spherical) deeper in the network. Our work is connected to slot attention [111], but is feed-forward rather than recurrent. Unlike slot attention, VECA refines both the input and cluster representations, and serves as a full vision backbone rather than only an object discovery layer. Notably, we observe emergent behavior in which core tokens correspond to objectlike regions in the image. Related work includes superpixels (SLIC [112, 113]), clustering-based transformers [114, 115, 116, 117, 118], and token aggregation methods [44, 119, 120, 121, 122, 123]. In contrast to these works, we exploit core-periphery block-sparse attention to enable elastic (adjustable) linear-cost inference. We demonstrate, for the first time, that models built entirely from these primitives can achieve performance competitive with state-of-the-art vision backbones.

3

Methods

In this section, we introduce our VECA (Visual Elastic-Core Attention) architecture, which we illustrate in Figure 2. Unlike prior Vision Transformers that construct a full self-attention matrix, we design our attention around a block-sparse core-periphery structure [13, 14]. In our notation, the image patches form the “periphery”, while the “cores” are learned from scratch and form a fully connected clique. Because patch tokens are maintained and updated throughout the network, VECA preserves spatially aligned dense representations; as shown in Figure 3, these representations remain coherent and robust as the input resolution increases. In Section 3.1, we provide background on Vision Transformers. We then describe our VECA module in Section 3.2, discuss its main properties, and finally introduce our budget-adaptive mechanism in Section 3.3. 3.1

Background on Vision Transformers

Here, we go over the fundamental components of Vision Transformers and outline recent advancements. Given an RGB image I ∈ RH×W ×3 , the image is divided into a grid of non-overlapping N = (H/P × W/P ) spatial patches, each with width P . Each patch is flattened and linearly projected to a latent dimension D. Positional embeddings are added to these projections to retain spatial coordinates, resulting in a 1D sequence X ∈ RN ×D . The sequence X is then passed through transformer blocks driven by self-attention. The input is linearly projected into Query (Q), Key (K), and Value (V) matrices. The self-attention mechanism computes contextualized representations by taking a similarity-weighted sum of the values:   QK⊤ Self-Attention(Q, K, V) = softmax √ V (1) dk where dk is the dimension of the keys. Typically an additional [CLS] and NR register tokens [124, 125, 126] are concatenated to the image tokens, creating a total attention matrix of size (N +NR +1)2 , 4

quadratic relative to input resolution (patch count). In practice, models employ Multi-Head SelfAttention, executing this operation in parallel and concatenating the outputs. While the original ViT utilized learnable positional embeddings, which were adapted to higher resolutions via interpolation, recent work has adopted 2D axial RoPE [127, 128, 129, 21]. Following DINOv3, we adopt 2D axial RoPE in our model, where core tokens have content- and layer- dependent learnable positions. 3.2

Core-Mediated Visual Attention

The basic building block of VECA is a block-sparse attention operation between active core tokens and patch tokens. Given patch tokens Z = {z1 , . . . , zN }, we additionally maintain an ordered bank of learnable core tokens RM = {r1 , . . . , rM }, where M denotes the maximum core capacity. For each image, we can select budget C ≤ M . The input sequence is formed by concatenating the active core prefix RC = RM [: C] with the patch sequence, X = [RC ; Z]. Unlike standard self-attention, patch tokens do not attend directly to all other patch tokens. Instead, core tokens attend to the full sequence, allowing them to aggregate global information, while patch tokens attend only to the active cores, allowing them to receive global context through the core interface. In standard (query, key, value) notation, the block computes R′ = Attn(RC , X, X), ′

′

Z ′ = Attn(Z, RC , RC ),

(2)

′

which are concatenated to form X = [R ; Z ], where we use 2D axial RoPE. This gives the attention matrix a core-periphery structure: fully connected core tokens serve as a compact global communication interface, while patch tokens retain spatially aligned dense representations. The resulting attention cost is (2N C + C 2 ), compared with N 2 (ignoring [CLS] and registers) for dense patch self-attention. Therefore, when C is predetermined, VECA attention cost scales linearly with image resolution while still allowing information exchange across the full image. Note that this core-periphery structure has graph diameter 2 (self-attention has graph diameter 1), and requires two blocks to achieve information mixing rather than one. As we demonstrate in section 4, our method still achieves competitive accuracy despite this limitation. In practice, each core token is associated with a continuous image- and layer-dependent planar coordinate in [−1, 1] × [−1, 1], while patch tokens use fixed image-grid coordinates. Both core and patch tokens are modulated with two-dimensional rotary positional encoding. Core coordinates are initialized to cover the image plane and are updated across layers. For layer ℓ and core i, where riℓ is the feature of core token i, ρℓi ∈ R2 is the unconstrained coordinate state, uℓi ∈ [−1, 1]2 is the coordinate used for RoPE, fℓ is a lightweight coordinate head, and αℓ is a learned layer-wise scalar: ρℓ+1 = ρℓi + αℓ fℓ (riℓ ), i

uℓ+1 = tanh(ρℓ+1 ). i i

(3)

This allows each core to maintain both a semantic representation and an evolving spatial position. 3.3

Budget-Adaptive Representation Learning

VECA is trained to support multiple active-core budgets within a single shared model. The ordered core bank allows us to select prefixes of variable size [16, 51]. This training procedure implicitly ranks the core tokens by importance. We define a distribution pC (·) over active core budgets. During training, we sample C ∼ pC (·) and retrieve the active prefix RC = RM [: C] = (r1 , . . . , rC ). Since the sampled cores form a prefix, early cores are active under more budgets and are encouraged to encode the most broadly useful information, while later cores provide additional capacity for higher-fidelity representations. This nested design induces a spatially and semantically coarse-to-fine representation. With a small core budget, the model is forced to compress the image into a compact global summary. As the budget increases, additional cores can specialize to more spatially local or semantically specific information. At inference time, we can trade accuracy and compute within the same model by selecting the number of active cores without retraining. In practice, we use the first core r1 as the [CLS] token.

4

Experiments

In this section we demonstrate that VECA can learn effective visual representations despite the lack of full patch-to-patch connectivity. We first describe the training details in section 4.1. In section 4.2, 5

RGB Image

Patch-to-Patch Similarity

Figure 4: Visualization of semantic consistency. We visualize the cosine similarity between a selected patch token, marked by a red cross, and all other tokens to examine the semantic consistency of the learned dense visual representations. Brighter colors indicate patches with higher cosine similarity to the selected token. We observe that the representations focus on spatially coherent and semantically consistent regions. Table 1: Results on dense prediction tasks. We compare backbone dense linear probing results against strong vision foundation model baselines. For semantic segmentation, we report mIoU, where higher is better. For depth estimation, we report RMSE, where lower is better. We find that our model is highly competitive despite the block-sparse attention structure. Segmentation (mIoU ↑)

Method

Depth (RMSE ↓)

VOC

Context

ADE

Stuff

Object

City

NYUv2

KITTI

Weakly Supervised CLIP-B/16 [130] OpenCLIP-B/16 [131] DFNCLIP-B/16 [132] SigLIP 2-B/16 [133]

72.87 70.74 71.58 75.41

45.99 44.06 44.56 47.51

34.55 35.98 35.64 42.13

36.18 34.47 35.23 38.70

48.76 46.27 46.11 54.58

55.34 53.38 51.83 59.53

0.6176 0.6185 0.5978 0.5296

4.3499 4.5058 4.3438 3.4349

Agglomerative AM-RADIOv2.5-B/16 [134]

85.73

56.72

50.37

46.96

62.94

69.03

0.3792

2.6051

Self-supervised DINOv2-B/14 [135] DINOv2-reg-B/14 [136] DINOv3-B/16 [21]

84.46 84.23 87.50

55.11 55.02 57.74

47.95 48.92 51.35

44.85 45.42 48.41

61.01 61.77 64.65

70.13 70.23 71.06

0.3949 0.3953 0.3684

2.7002 2.8307 2.6915

Ours-B/16 (C = 8) ▷ Gap to Best (C = 8) Ours-B/16 (C = 64) ▷ Gap to Best (C = 64)

83.84 -3.66 87.07 -0.43

53.30 -4.44 57.46 -0.28

44.51 -6.84 50.69 -0.66

43.85 -4.56 47.92 -0.49

58.27 -6.38 63.58 -1.07

61.97 -9.09 69.54 -1.52

0.4330 +0.0646 0.3705 +0.0021

3.1877 +0.5826 2.7252 +0.1201

we evaluate the performance of VECA on a diverse set of downstream tasks, including both dense prediction and image classification. Throughout the evaluation, we train a linear head with the visual model frozen. Furthermore, we analyze the impact of the core token budgets in section 4.3. Finally, we investigate the emergent behaviors of core tokens during inference in section 4.4. Additional details for the experimental setup and further evaluation results are provided in the appendix. 4.1

Training Objective and Setup

Inspired by prior work [134], we train VECA with feature distillation from a frozen DINOv3 teacher using unlabeled Objects365 images [137]. For an input image I and active core budget C, VECA (C) produces a global feature xcls from the final-layer representation of the first core token, and dense (C) patch features xdense from the final-layer patch tokens. The teacher provides the corresponding target features xcls and xdense . We optimize the high-level objective     (C) (C) L(I, C) = Lcls xcls , xcls + λLdense xdense , xdense , (4) where Lcls aligns global image representations and Ldense aligns spatial patch representations. 6

Table 2: Results on image classification. We present image classification results using a linear classifier trained on a frozen backbone. We report Top-1 accuracy (Top-1 Acc), where higher values indicate better performance. To highlight VECA’s competitive performance, we present the gap to the best result in bold. Our model is extremely competitive. Classification (Top-1 Acc ↑)

Method IN1K

INv2

IN-Real

Places

Food

SUN

Oxford

CUB

Weakly Supervised CLIP-B/16 [130] OpenCLIP-B/16 [131] DFNCLIP-B/16 [132] SigLIP 2-B/16 [133]

79.33 79.18 81.06 81.91

69.31 69.05 71.35 73.04

84.93 84.72 86.56 87.69

55.27 55.35 55.79 56.18

92.42 91.33 93.32 94.70

78.05 78.83 78.68 78.92

92.72 91.91 93.59 92.89

80.51 83.15 85.90 78.41

Agglomerative AM-RADIOv2.5-B/16 [134]

80.53

70.60

85.78

54.13

91.64

76.91

93.54

81.14

Self-supervised DINOv2-B/14 [135] DINOv2-reg-B/14 [136] DINOv3-B/16 [21]

82.41 83.44 83.56

73.22 74.75 74.92

86.81 88.18 88.65

53.37 54.81 55.39

92.02 92.87 94.47

75.67 76.85 77.59

95.61 95.97 96.54

89.06 89.52 89.82

Ours-B/16 (C = 8) ▷ Gap to Best (C = 8) Ours-B/16 (C = 64) ▷ Gap to Best (C = 64)

79.99 -3.81 81.93 -1.87

69.18 -6.03 72.25 -2.96

86.45 -2.29 87.87 -0.87

55.30 -0.88 55.82 -0.36

90.04 -4.66 92.44 -2.26

75.35 -3.57 76.60 -2.32

95.12 -1.42 95.67 -0.87

87.00 -2.82 88.02 -1.80

Training proceeds in two stages. In Stage 1, we distill VECA at a fixed 256 × 256 resolution while sampling active core budgets. In Stage 2, we continue training from the Stage-1 model with the same distillation objective, but adapt it to higher-resolution targets from {384, 512, 768}. All downstream experiments freeze the visual encoder and train only a linear prediction head. We provide additional details on the objective, optimizer, learning rate schedule, and architecture in the appendix. 4.2

Linear Probe Based Downstream Evaluation

We evaluate the quality of VECA representations on a diverse set of benchmarks using linear probes. For all dense benchmarks, we use 512 (16 px patch) or 518 (14 px patch) for feature extraction. Dense Prediction. For semantic segmentation, we utilize PASCAL VOC 2012 (VOC) [138], PASCAL Context-60 (Context) [139], ADE20K-150 (ADE) [140], COCO-Object (Object) [141], COCO-Stuff (Stuff) [142], and Cityscapes (City) [143]. Table 1 demonstrates that VECA achieves competitive performance across diverse segmentation datasets, with particularly promising results on VOC (87.07 mIoU) and Context (57.46 mIoU). Although slightly weaker than the strongest baseline, DINOv3, the performance gap is on average less than 1% mIoU delta on average across segmentation datasets. On the other hand, monocular depth estimation evaluates a model’s ability to capture the geometric structure of a scene. We evaluate on the NYUv2 [144] and KITTI [145] benchmarks. As shown in Table 1, VECA also demonstrates strong performance on both datasets, with a negligible gap (0.0021 RMSE) to DINOv3 on NYUv2. In Figure 4, we further demonstrate the qualitative visualization to show the semantic consistency of the dense features from VECA. When reducing the number of core tokens to a minimum of 8 (a mere 1.6% of the connections of the full attention matrix at 512 res), VECA still maintains considerable performance across all datasets compared to the full model with 64 core tokens. Overall, these results indicate that VECA effectively captures high-quality dense information through a compact set of learnable core tokens with linear-time inference. Image Classification. Image classification examines a model’s capability to understand global information. Table 2 includes the results on ImageNet-1K (IN1K) [146], ImageNetV2 (INv2) [147], ImageNet-ReaL (IN-Real) [148], Places365 (Places) [149], Food101 (Food) [150], SUN397 (SUN) [151], Oxford-Pets (Oxford) [152], CUB-200 (CUB) [153]. Note that for fairness, INv2 and IN-Real are direct probe transfers from IN1k without further training. Here, VECA demonstrates strong performance on image classification across all datasets. Notably, even with only 8 core tokens (only 6.3% of the connections of the full self-attention matrix at 256 res), VECA maintains at least 7

Context

59

49

57

48

56

47

NYUv2

0.44

46

54

Ours-Base DINOv3-Base

53 52

16 24 32 40 48 56 64 Number of Core Tokens

RGB Image

C=16

C=32

C=64

0.40

45

Ours-Base DINOv3-Base

44

0.38 0.36

43 8

Ours-Base DINOv3-Base

0.42

RMSE

55

0.46

mloU

58

mloU

Stuff

50

8

16 24 32 40 48 56 64 Number of Core Tokens

8

16 24 32 40 48 56 64 Number of Core Tokens

Figure 5: Results of varying the core-token budget. (a) Dense prediction performance improves consistently as more core tokens are activated. We report linear-probe results on Context, Stuff, and NYUv2 at 768 res while sweeping the active budget from C = 8 to C = 64. (b) Qualitative patch-to-patch similarity visualizations across budgets. Larger core-token budgets produce more semantically coherent and spatially localized dense features, leading to finer-grained localization.

96% of the full model’s performance. This result highlights the potential of the core-mediated design to balance strong accuracy with reduced attention cost. 4.3

The Impact of the Core Token Budgets

Benefiting from the learned ranking induced by elastic training, VECA exhibits flexibility during inference: the number of active core tokens can be varied without retraining. We therefore analyze how different core token budgets affect downstream performance. We present the results for C = 8 and C = 64 for image classification in Table 2. We further analyze the effect of varying active core tokens from 8 to 64, with a step size of 8, on dense segmentation (Context, Stuff) and depth estimation tasks (NYUv2). We observe different trends across task types. Performance on dense prediction tasks consistently improves as the token budget increases, as illustrated in Figure 5 (a), whereas image classification remains comparatively stable. In particular, using only 8 core tokens already achieves performance close to that of the full 64-token model for classification. This behavior aligns with the intended role of nested core learning. Since early cores are active across all budgets during training, they are encouraged to encode the most essential visual information, while later cores specialize in capturing finer spatial details. As a result, global recognition can be supported by a small number of highly informative core tokens, whereas dense prediction requires a larger token budget to preserve stronger semantic consistency and fine-grained localization, as shown in Figure 5 (b). 4.4

Investigating Emergent Behaviors of Core Tokens

Although core tokens serve as a communication interface for exchanging information across layers, they are not explicitly supervised during training. We therefore further investigate the emergent behaviors they acquire. From the results in Figure 6 and Figure 7, we identify two notable emergent behaviors of core tokens. First, core tokens are strongly object centric, and attend to semantically diverse objects [111, 136]. Core tokens generally attend to distinct objects or parts, for example, eggs in a bowl, as shown in Figure 6. This emergent behavior suggests that core tokens learn to focus on coherent semantics. Second, core tokens exhibit a consistent progression from isotropic representations to semantically clustered structures across layers under different active core budgets during feedforward processing, despite the lack of recurrent updates as in Slot Attention. As illustrated in Figure 7, core tokens initially form coarse partitions that depend on the active core budget in early layers. As depth increases, their representations evolve from spherical to progressively more structured and semantically coherent clusters, forming object- and region-level groups. This behavior suggests that repeated core-topatch interactions organize image information into structured semantic regions without explicit segmentation supervision, highlighting their potential to learn dense visual representations without relying on patch-to-patch interactions. When varying the number of cores from C = 16 to C = 64, we see a distinct change in the size of the attended regions. 8

RGB Image

Selected Core Tokens

UMAP

Figure 6: Complementary visual roles of core tokens. We visualize the patterns of different core tokens using the cosine similarity between selected core tokens and spatial patch features. Warmer colors indicate higher similarity. We find that different core tokens respond to distinct spatial regions, suggesting that the learned set of core tokens captures complementary visual information rather than duplicating the same global representations.

Figure 7: From isotropic to semantic clustering. We visualize the development of core token attention weights with UMAP during inference under different active token budgets. The results show that core tokens progressively cluster into semantically coherent regions, evolving from spherical and diffuse to structured representations. Please zoom in for details.

5

Discussion

Limitations and Future Work. VECA is trained with a fixed maximum core capacity and a predefined set of active-core budgets. While this enables elastic inference, the active budget is still manually selected rather than adapted to image content, resolution, or task. Future work could explore content-aware core allocation and study how core capacity should scale with model size, data scale, and task complexity. Although our visualizations suggest core specialization and feedforward clustering, more systematic analysis is needed to quantify core redundancy, stability, and semantic consistency. Finally, broader evaluation on detection, instance segmentation, video, and task-specific fine-tuning would further test the generality of VECA. 9

Conclusion. We introduce VECA, a vision architecture that replaces dense patch-to-patch selfattention with elastic core-mediated attention. By routing information through a compact set of learnable core tokens, VECA reduces attention cost while preserving spatially aligned dense features. Across frozen-backbone evaluations, VECA remains competitive on classification and is especially strong on dense prediction, closely approaching DINOv3 on segmentation and depth estimation. Our analyses show that core tokens develop semantically organized roles across depth. These results suggest that direct patch-to-patch self-attention is not strictly necessary for high-quality dense visual representation learning, making VECA a promising scalable building block for future Vision Transformers.

10

References [1] Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. Advances in neural information processing systems, 34: 24261–24272, 2021. [2] Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, et al. Resmlp: Feedforward networks for image classification with data-efficient training. IEEE transactions on pattern analysis and machine intelligence, 45(4):5314–5321, 2022. [3] Hanxiao Liu, Zihang Dai, David So, and Quoc V Le. Pay attention to mlps. Advances in neural information processing systems, 34:9204–9215, 2021. [4] James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon. Fnet: Mixing tokens with fourier transforms. In Proceedings of the 2022 Conference of the north American chapter of the Association for Computational Linguistics: human language technologies, pages 4296–4313, 2022. [5] Yongming Rao, Wenliang Zhao, Zheng Zhu, Jie Zhou, and Jiwen Lu. Gfnet: Global filter networks for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):10960–10973, 2023. [6] Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. Advances in neural information processing systems, 29, 2016. [7] Md Amirul Islam, Sen Jia, and Neil DB Bruce. How much position information do convolutional neural networks encode? arXiv preprint arXiv:2001.08248, 2020. [8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. [9] Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in neural information processing systems, 34:12116–12128, 2021. [10] Namuk Park and Songkuk Kim. arXiv:2202.06709, 2022.

How do vision transformers work?

arXiv preprint

[11] Yanyu Li, Ju Hu, Yang Wen, Georgios Evangelidis, Kamyar Salahi, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Rethinking vision transformers for mobilenet size and speed. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16889– 16900, 2023. [12] Tobias Christian Nauen, Sebastian Palacio, Federico Raue, and Andreas Dengel. Which transformer to favor: a comparative analysis of efficiency in vision transformers. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 6955–6966. IEEE, 2025. [13] M Puck Rombach, Mason A Porter, James H Fowler, and Peter J Mucha. Core-periphery structure in networks. SIAM Journal on Applied mathematics, 74(1):167–190, 2014. [14] Xiao Zhang, Travis Martin, and Mark EJ Newman. Identification of core-periphery structure in networks. Physical Review E, 91(3):032803, 2015. [15] R Duncan Luce and Albert D Perry. A method of matrix analysis of group structure. Psychometrika, 14(2):95–116, 1949. [16] Oren Rippel, Michael Gelbart, and Ryan Adams. Learning ordered representations with nested dropout. In International Conference on Machine Learning, pages 1746–1754. PMLR, 2014. 11

[17] Yixuan Wei, Han Hu, Zhenda Xie, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, and Baining Guo. Contrastive learning rivals masked image modeling in fine-tuning via feature distillation. arXiv preprint arXiv:2205.14141, 2022. [18] Mike Ranzinger, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. Am-radio: Agglomerative vision foundation model reduce all domains into one. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12490–12500, 2024. [19] Jinghuan Shang, Karl Schmeckpeper, Brandon B May, Maria Vittoria Minniti, Tarik Kelestemur, David Watkins, and Laura Herlant. Theia: Distilling diverse vision foundation models for robot learning. arXiv preprint arXiv:2407.20179, 2024. [20] Mert Bülent Sarıyıldız, Philippe Weinzaepfel, Thomas Lucas, Diane Larlus, and Yannis Kalantidis. Unic: Universal classification models via multi-teacher distillation. In European Conference on Computer Vision, pages 353–371. Springer, 2024. [21] Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Dinov3. arXiv preprint arXiv:2508.10104, 2025. [22] Shlomo Zilberstein. Using anytime algorithms in intelligent systems. AI magazine, 17(3):73–73, 1996. [23] Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, et al. Matryoshka representation learning. Advances in Neural Information Processing Systems, 35:30233–30249, 2022. [24] Edward W Staley and Jared Markowitz. Triangular dropout: variable network width without retraining. arXiv preprint arXiv:2205.01235, 2022. [25] Xinwei Shen and Nicolai Meinshausen. Distributional principal autoencoders. arXiv preprint arXiv:2404.13649, 2024. [26] Matthew Ho, Xiaosheng Zhao, and Benjamin D Wandelt. Ordered embeddings and intrinsic dimensionalities with information-ordered bottlenecks. Machine Learning: Science and Technology, 6(3):035010, 2025. [27] Saïd Ladjal, Alasdair Newson, and Chi-Hieu Pham. A pca-like autoencoder. arXiv preprint arXiv:1904.01277, 2019. [28] Hokchhay Tann, Soheil Hashemi, R Iris Bahar, and Sherief Reda. Runtime configurable deep neural networks for energy-accuracy trade-off. In Proceedings of the eleventh IEEE/acm/ifip International Conference on Hardware/Software Codesign and System Synthesis, pages 1–10, 2016. [29] Eunwoo Kim, Chanho Ahn, and Songhwai Oh. Nestednet: Learning nested sparse structures in deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8669–8678, 2018. [30] Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. arXiv preprint arXiv:1812.08928, 2018. [31] Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. arXiv preprint arXiv:1908.09791, 2019. [32] Mojtaba Valipour, Mehdi Rezagholizadeh, Hossein Rajabzadeh, Parsa Kavehzadeh, Marzieh Tahaei, Boxing Chen, and Ali Ghodsi. Sortednet: A scalable and generalized framework for training modular deep neural networks. arXiv preprint arXiv:2309.00255, 2023. [33] Kai Li and Yi Luo. Subnetwork-to-go: Elastic neural network with dynamic training and customizable inference. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6775–6779. IEEE, 2024. 12

[34] Yitian Zhang, Huseyin Coskun, Xu Ma, Huan Wang, Ke Ma, Xi Chen, Derek H Hu, and Yun Fu. Slicing vision transformer for flexible inference. Advances in Neural Information Processing Systems, 37:42649–42671, 2024. [35] Sneha Kudugunta, Aditya Kusupati, Tim Dettmers, Kaifeng Chen, Inderjit Dhillon, Yulia Tsvetkov, Hannaneh Hajishirzi, Sham Kakade, Ali Farhadi, and Prateek Jain. Matformer: Nested transformer for elastic inference. Advances in Neural Information Processing Systems, 37: 140535–140564, 2024. [36] Ruisi Cai, Saurav Muralidharan, Greg Heinrich, Hongxu Yin, Zhangyang Wang, Jan Kautz, and Pavlo Molchanov. Flextron: Many-in-one flexible large language model. arXiv preprint arXiv:2406.10260, 2024. [37] Surat Teerapittayanon, Bradley McDanel, and Hsiang-Tsung Kung. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd International Conference on Pattern Recognition (ICPR), pages 2464–2469. IEEE, 2016. [38] Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S Davis, Kristen Grauman, and Rogerio Feris. Blockdrop: Dynamic inference paths in residual networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8817– 8826, 2018. [39] Weijie Liu, Peng Zhou, Zhiruo Wang, Zhe Zhao, Haotang Deng, and Qi Ju. Fastbert: a selfdistilling bert with adaptive inference time. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 6035–6044, 2020. [40] Alex Graves. Adaptive computation time for recurrent neural networks. arXiv preprint arXiv:1603.08983, 2016. [41] Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. Universal transformers. arXiv preprint arXiv:1807.03819, 2018. [42] Zhonglin Xie, Yiman Fong, Haoran Yuan, and Zaiwen Wen. Accelerating optimization via differentiable stopping time. arXiv preprint arXiv:2505.22509, 2025. [43] Ahmadreza Jeddi, Marco Ciccone, and Babak Taati. Loopformer: Elastic-depth looped transformers for latent reasoning via shortcut modulation. arXiv preprint arXiv:2602.11451, 2026. [44] Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461, 2022. [45] Fuzhao Xue, Valerii Likhosherstov, Anurag Arnab, Neil Houlsby, Mostafa Dehghani, and Yang You. Adaptive computation with elastic input sequence. In International Conference on Machine Learning, pages 38971–38988. PMLR, 2023. [46] Ali Hojjat, Janek Haberer, Soren Pirk, and Olaf Landsiedel. Thinkingvit: Matryoshka thinking vision transformer for elastic inference. arXiv preprint arXiv:2507.10800, 2025. [47] Naibin Gu, Zhenyu Zhang, Yuchen Feng, Yilong Chen, Peng Fu, Zheng Lin, Shuohuan Wang, Yu Sun, Hua Wu, Weiping Wang, et al. Elastic moe: Unlocking the inference-time scalability of mixture-of-experts. arXiv preprint arXiv:2509.21892, 2025. [48] Yaoxiang Wang, Qingguo Hu, Yucheng Ding, Ruizhe Wang, Yeyun Gong, Jian Jiao, Yelong Shen, Peng Cheng, and Jinsong Su. Training matryoshka mixture-of-experts for elastic inferencetime expert utilization. arXiv preprint arXiv:2509.26520, 2025. [49] Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. Avit: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10809–10818, 2022. [50] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34:13937–13949, 2021. 13

[51] Wenbo Hu, Zi-Yi Dou, Liunian H Li, Amita Kamath, Nanyun Peng, and Kai-Wei Chang. Matryoshka query transformer for large vision-language models. Advances in Neural Information Processing Systems, 37:50168–50188, 2024. [52] Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. arXiv preprint arXiv:1812.01243, 2018. [53] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, pages 5156–5165. PMLR, 2020. [54] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009.14794, 2020. [55] Michael Zhang, Kush Bhatia, Hermann Kumbong, and Christopher Ré. The hedgehog & the porcupine: Expressive linear attentions with softmax mimicry. arXiv preprint arXiv:2402.04347, 2024. [56] Sahil Joshi, Agniva Chowdhury, Amar Kanakamedala, Ekam Singh, Evan Tu, and Anshumali Shrivastava. Replacing softmax similarity with a sharpened angular similarity: Theory and practice of scaling to billion-context attention. arXiv preprint arXiv:2510.04008, 2025. [57] Jiecheng Lu, Xu Han, Yan Sun, Viresh Pati, Yubin Kim, Siddhartha Somani, and Shihao Yang. Zeros: Zero-sum linear attention for efficient transformers. arXiv preprint arXiv:2602.05230, 2026. [58] Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020. [59] Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: A nyström-based algorithm for approximating self-attention. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 14138–14148, 2021. [60] Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451, 2020. [61] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283–17297, 2020. [62] Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020. [63] Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics, 9:53–68, 2021. [64] Shantanu Acharya, Fei Jia, and Boris Ginsburg. Star attention: Efficient llm inference over long sequences. arXiv preprint arXiv:2411.17116, 2024. [65] Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer: Rethinking self-attention for transformer models. In International Conference on Machine Learning, pages 10183–10192. PMLR, 2021. [66] Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation-invariant neural networks. In International Conference on Machine Learning, pages 3744–3753. PMLR, 2019. [67] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International Conference on Machine Learning, pages 4651–4664. PMLR, 2021. 14

[68] Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. ICLR, 2022. [69] Rares Dolga, Lucas Maystre, Marius Cobzarenco, and David Barber. Latte: Latent attention for linear time transformers. 2024. [70] Xuezhe Ma, Xiang Kong, Sinong Wang, Chunting Zhou, Jonathan May, Hao Ma, and Luke Zettlemoyer. Luna: Linear unified nested attention. Advances in Neural Information Processing Systems, 34:2441–2453, 2021. [71] Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun. Attention bottlenecks for multimodal fusion. Advances in neural information processing systems, 34:14200–14213, 2021. [72] Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A Smith, and Lingpeng Kong. Random feature attention. arXiv preprint arXiv:2103.02143, 2021. [73] Kazuki Irie, Imanol Schlag, Róbert Csordás, and Jürgen Schmidhuber. Going beyond linear transformers with recurrent fast weight programmers. Advances in neural information processing systems, 34:7703–7717, 2021. [74] Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models. Advances in neural information processing systems, 35:35971–35983, 2022. [75] Hao Peng, Jungo Kasai, Nikolaos Pappas, Dani Yogatama, Zhaofeng Wu, Lingpeng Kong, Roy Schwartz, and Noah A Smith. Abc: Attention with bounded-memory control. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7469–7483, 2022. [76] Huanru Henry Mao. Fine-tuning pre-trained transformers into decaying fast weights. In Proceedings of the 2022 conference on empirical methods in natural language processing, pages 10236–10242, 2022. [77] Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. arXiv preprint arXiv:2312.06635, 2023. [78] Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621, 2023. [79] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. [80] Zhen Qin, Songlin Yang, Weixuan Sun, Xuyang Shen, Dong Li, Weigao Sun, and Yiran Zhong. Hgrn2: Gated linear rnns with state expansion. arXiv preprint arXiv:2404.07904, 2024. [81] Yu Zhang, Songlin Yang, Ruijie Zhu, Yue Zhang, Leyang Cui, Yiqiao Wang, Bolun Wang, Freda Shi, Bailin Wang, Wei Bi, et al. Gated slot attention for efficient linear-time sequence modeling. Advances in Neural Information Processing Systems, 37:116870–116898, 2024. [82] Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xLSTM: Extended long short-term memory. Advances in Neural Information Processing Systems, 37: 107547–107603, 2024. [83] Bo Peng, Ruichong Zhang, Daniel Goldstein, Eric Alcaide, Xingjian Du, Haowen Hou, Jiaju Lin, Jiaxing Liu, Janna Lu, William Merrill, et al. RWKV-7 “Goose” with expressive dynamic state evolution. arXiv preprint arXiv:2503.14456, 2025. [84] Jiaxi Hu, Yongqi Pan, Jusen Du, Disen Lan, Xiaqiang Tang, Qingsong Wen, Yuxuan Liang, and Weigao Sun. Improving bilinear RNN with closed-loop control. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. 15

[85] Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, and Judy Hoffman. Hydra attention: Efficient attention with many heads. In European conference on computer vision, pages 35–49. Springer, 2022. [86] Han Cai, Chuang Gan, and Song Han. EfficientViT: Enhanced linear attention for highresolution low-computation visual recognition. arXiv preprint arXiv:2205.14756, 3(1), 2022. [87] Haoran You, Yunyang Xiong, Xiaoliang Dai, Bichen Wu, Peizhao Zhang, Haoqi Fan, Peter Vajda, and Yingyan Celine Lin. Castling-ViT: Compressing self-attention via switching towards linear-angular attention at vision transformer inference. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14431–14442, 2023. [88] Dongchen Han, Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang. Flatten transformer: Vision transformer using focused linear attention. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5961–5971, 2023. [89] Weikang Meng, Yadan Luo, Xin Li, Dongmei Jiang, and Zheng Zhang. Polaformer: Polarityaware linear attention for vision transformers. arXiv preprint arXiv:2501.15061, 2025. [90] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012–10022, 2021. [91] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 568–578, 2021. [92] Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidimensional transformers. arXiv preprint arXiv:1912.12180, 2019. [93] Dongchen Han, Tianzhu Ye, Yizeng Han, Zhuofan Xia, Siyuan Pan, Pengfei Wan, Shiji Song, and Gao Huang. Agent attention: On the integration of softmax and linear attention. In European conference on computer vision, pages 124–140. Springer, 2024. [94] Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In International Conference on Machine Learning, pages 478–487. PMLR, 2016. [95] Bo Yang, Xiao Fu, Nicholas D Sidiropoulos, and Mingyi Hong. Towards k-means-friendly spaces: Simultaneous deep learning and clustering. In International Conference on Machine Learning, pages 3861–3870. PMLR, 2017. [96] Liang Duan, Charu Aggarwal, Shuai Ma, and Saket Sathe. Improving spectral clustering with deep embedding and cluster estimation. In 2019 IEEE International Conference on Data Mining (ICDM), pages 170–179. IEEE, 2019. [97] Sohil Atul Shah and Vladlen Koltun. arXiv:1803.01449, 2018.

Deep continuous clustering.

arXiv preprint

[98] Filippo Maria Bianchi, Daniele Grattarola, and Cesare Alippi. Spectral clustering with graph neural networks for graph pooling. In International Conference on Machine Learning, pages 874–883. PMLR, 2020. [99] Bishwajit Saha, Dmitry Krotov, Mohammed J Zaki, and Parikshit Ram. End-to-end differentiable clustering with associative memories. In International Conference on Machine Learning, pages 29649–29670. PMLR, 2023. [100] Lawrence Stewart, Francis Bach, Felipe Llinares-López, and Quentin Berthet. Differentiable clustering with perturbed spanning forests. Advances in Neural Information Processing Systems, 36:31158–31176, 2023. [101] Meitar Ronen, Shahaf E Finder, and Oren Freifeld. Deepdpm: Deep clustering with an unknown number of clusters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9861–9870, 2022. 16

[102] Yazhou Ren, Ni Wang, Mingxia Li, and Zenglin Xu. Deep density-based image clustering. Knowledge-Based Systems, 197:105841, 2020. [103] Dorin Comaniciu and Peter Meer. Mean shift: A robust approach toward feature space analysis. IEEE Transactions on pattern analysis and machine intelligence, 24(5):603–619, 2002. [104] Shu Kong and Charless C Fowlkes. Recurrent pixel embedding for instance grouping. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9018– 9028, 2018. [105] Collin Leiber, Lena GM Bauer, Benjamin Schelling, Christian Böhm, and Claudia Plant. Dipbased deep embedded clustering with k-estimation. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 903–913, 2021. [106] Collin Leiber, Niklas Strauß, Matthias Schubert, and Thomas Seidl. Dying clusters is all you need-deep clustering with an unknown number of clusters. In 2024 IEEE International Conference on Data Mining Workshops (ICDMW), pages 726–733. IEEE, 2024. [107] Maximillian Nickel and Douwe Kiela. Poincaré embeddings for learning hierarchical representations. Advances in neural information processing systems, 30, 2017. [108] Xiang Li, Luke Vilnis, Dongxu Zhang, Michael Boratko, and Andrew McCallum. Smoothing the geometry of probabilistic box embeddings. In International Conference on Learning Representations, 2018. [109] Ines Chami, Albert Gu, Vaggos Chatziafratis, and Christopher Ré. From trees to continuous embeddings and back: Hyperbolic hierarchical clustering. Advances in neural information processing systems, 33:15065–15076, 2020. [110] Nicholas Monath, Manzil Zaheer, Daniel Silva, Andrew McCallum, and Amr Ahmed. Gradientbased hierarchical clustering using continuous representations of trees in hyperbolic space. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 714–722, 2019. [111] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances in neural information processing systems, 33:11525–11538, 2020. [112] Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. Slic superpixels compared to state-of-the-art superpixel methods. IEEE transactions on pattern analysis and machine intelligence, 34(11):2274–2282, 2012. [113] Varun Jampani, Deqing Sun, Ming-Yu Liu, Ming-Hsuan Yang, and Jan Kautz. Superpixel sampling networks. In Proceedings of the European conference on computer vision (ECCV), pages 352–368, 2018. [114] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18134–18144, 2022. [115] Tsung-Wei Ke, Sangwoo Mo, and Stella X Yu. Learning hierarchical image segmentation for recognition and by recognition. arXiv preprint arXiv:2210.00314, 2022. [116] James Liang, Yiming Cui, Qifan Wang, Tong Geng, Wenguan Wang, and Dongfang Liu. Clusterfomer: clustering as a universal visual learner. Advances in neural information processing systems, 36:64029–64042, 2023. [117] Wang Zeng, Sheng Jin, Lumin Xu, Wentao Liu, Chen Qian, Wanli Ouyang, Ping Luo, and Xiaogang Wang. Tcformer: Visual recognition via token clustering transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12):9521–9535, 2024. [118] Guillem Brasó, Aljoša Ošep, and Laura Leal-Taixé. Native segmentation vision transformers. arXiv preprint arXiv:2505.16993, 2025. 17

[119] Marius Aasan, Odd Kolbjørnsen, Anne Schistad Solberg, and Adín Ramirez Rivera. A spitting image: Modular superpixel tokenization in vision transformers. In European Conference on Computer Vision, pages 124–142. Springer, 2024. [120] Jieru Mei, Liang-Chieh Chen, Alan Yuille, and Cihang Xie. Spformer: Enhancing vision transformer with superpixel representation. arXiv preprint arXiv:2401.02931, 2024. [121] Marius Aasan, Martine Hjelkrem-Tan, Nico Catalano, Changkyu Choi, and Adín Ramírez Rivera. Differentiable hierarchical visual tokenization. arXiv preprint arXiv:2511.02652, 2025. [122] Savya Khosla, Sethuraman TV, Barnett Lee, Alexander Schwing, and Derek Hoiem. Ren: Fast and efficient region encodings from patch-based image encoders. arXiv preprint arXiv:2505.18153, 2025. [123] Yifan Pu, Jixuan Ying, Qixiu Li, Tianzhu Ye, Dongchen Han, Xiaochen Wang, Ziyi Wang, Xinyu Shao, Gao Huang, and Xiu Li. Linear differential vision transformer: Learning visual contrasts via pairwise differentials. arXiv preprint arXiv:2511.00833, 2025. [124] Mikhail S Burtsev, Yuri Kuratov, Anton Peganov, and Grigory V Sapunov. Memory transformer. arXiv preprint arXiv:2006.11527, 2020. [125] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint arXiv:2309.16588, 2023. [126] Yinjie Chen, Zipeng Yan, Chong Zhou, Bo Dai, and Andrew F Luo. Vision transformers with self-distilled registers. arXiv preprint arXiv:2505.21501, 2025. [127] Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. Image and Vision Computing, 149:105171, 2024. [128] Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26439–26455, 2024. [129] Zeyu Lu, Zidong Wang, Di Huang, Chengyue Wu, Xihui Liu, Wanli Ouyang, and Lei Bai. Fit: Flexible vision transformer for diffusion model. arXiv preprint arXiv:2402.12376, 2024. [130] Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021. [131] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. [132] Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. arXiv preprint arXiv:2309.17425, 2023. [133] Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786, 2025. [134] Greg Heinrich, Mike Ranzinger, Hongxu Yin, Yao Lu, Jan Kautz, Andrew Tao, Bryan Catanzaro, and Pavlo Molchanov. Radiov2. 5: Improved baselines for agglomerative vision foundation models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 22487–22497, 2025. [135] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. [136] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint arXiv:2309.16588, 2023. 18

[137] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8430–8439, 2019. [138] Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International Journal of Computer Vision, 111:98–136, 2015. [139] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 891–898, 2014. [140] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302–321, 2019. [141] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13, pages 740–755. Springer, 2014. [142] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1209–1218, 2018. [143] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213–3223, 2016. [144] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V 12, pages 746–760. Springer, 2012. [145] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(11):1231–1237, 2013. [146] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. [147] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pages 5389–5400. PMLR, 2019. [148] Lucas Beyer, Olivier J Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? arXiv preprint arXiv:2006.07159, 2020. [149] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017. [150] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101–mining discriminative components with random forests. In European conference on computer vision, pages 446–461. Springer, 2014. [151] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485–3492. IEEE, 2010. 19

[152] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498–3505. IEEE, 2012. [153] Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-ucsd birds 200. 2010. [154] Dahun Kim, Anelia Angelova, and Weicheng Kuo. Region-aware pretraining for openvocabulary object detection with vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11144–11154, 2023. [155] Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun. Rotary position embedding for vision transformer. In European Conference on Computer Vision, pages 289–305. Springer, 2024. [156] Sophie Ostmeier, Brian Axelrod, Maya Varma, Michael E Moseley, Akshay Chaudhari, and Curtis Langlotz. Liere: Lie rotational positional encodings. arXiv preprint arXiv:2406.10322, 2024. [157] Connor Schenck, Isaac Reid, Mithun George Jacob, Alex Bewley, Joshua Ainslie, David Rendleman, Deepali Jain, Mohit Sharma, Avinava Dubey, Ayzaan Wahid, et al. Learning the ropes: Better 2d and 3d position encodings with string. arXiv preprint arXiv:2502.02562, 2025. [158] Chase van de Geijn, Timo Lüddecke, Polina Turishcheva, and Alexander S Ecker. A circular argument: Does rope need to be equivariant for vision? arXiv preprint arXiv:2511.08368, 2025. [159] Hao Yu, Tangyu Jiang, Shuning Jia, Shannan Yan, Shunning Liu, Haolong Qian, Guanghao Li, Shuting Dong, and Chun Yuan. Comrope: Scalable and robust rotary position embedding parameterized by trainable commuting angle matrices. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 4508–4517, 2025. [160] Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan.github.io/posts/muon/. [161] Zichong Li, Liming Liu, Chen Liang, Weizhu Chen, and Tuo Zhao. Normuon: Making muon more efficient and scalable. arXiv preprint arXiv:2510.05491, 2025. [162] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. [163] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. [164] Kwangjun Ahn, Byron Xu, Natalie Abreu, and John Langford. Dion: Distributed orthonormalized updates. arXiv preprint: 2504.05295, 2025. [165] Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, et al. Muon is scalable for llm training. arXiv preprint arXiv:2502.16982, 2025. [166] Lizhang Chen, Jonathan Li, Kaizhao Liang, Baiyu Su, Cong Xie, Nuo Wang Pierse, Chen Liang, Ni Lao, and Qiang Liu. Cautious weight decay. arXiv preprint arXiv:2510.12402, 2025. [167] Noah Amsel, David Persson, Christopher Musco, and Robert M Gower. The polar express: Optimal matrix sign methods and their application to the muon algorithm. arXiv preprint arXiv:2505.16932, 2025. [168] MMSegmentation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. https://github.com/open-mmlab/mmsegmentation, 2020. [169] Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. Adabins: Depth estimation using adaptive bins. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4009–4018, 2021. 20

[170] Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. Attention is not only a weight: Analyzing transformers with vector norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7057–7075, 2020.

21

A

Technical Appendices and Supplementary Material

Sections 1. Architecture details of VECA (Section A.1) 2. Training and optimization details (Section A.2) 3. Core-Periphery attention pseudocode (Section A.3) 4. Baseline model details (Section A.4) 5. Implementation details for downstream tasks (Section A.5) 6. Multi-resolution evaluation across tasks (Section A.6) 7. Additional core-to-patch similarity visualizations (Section A.7) 8. Additional classification results (Section A.8) 9. Additional emergent behavior and core-budget visualizations (Section A.9) 10. Efficiency analysis and computational cost (Section A.10) 11. Additional patch-to-patch similarity visualizations (Section A.11) 12. Additional ablation studies (Section A.12)

22

A.1

Architecture details of VECA

This section specifies the concrete VECA model configurations used in our experiments. The main paper defines the core-mediated attention mechanism and its motivation; here we report implementation-level architectural settings. Model configurations. We instantiate VECA at four model sizes: Small, Small+, Base, and Large. All variants use a 16 × 16 patch embedding, a maximum core capacity M = 64, and nested activecore budgets {8, 16, 24, 32, 40, 48, 56, 64}. Effectively, we implement a granularity of 8 cores for nested training. The model scale is controlled by the number of layers, hidden dimension, number of attention heads, and feed-forward expansion ratio. Table S.1: VECA architecture configurations. All variants use patch size 16 × 16, maximum core capacity M = 64, and active-core budgets {8, 16, 24, 32, 40, 48, 56, 64}. Model

Layers

Hidden dim.

Heads

MLP ratio

12 12 12 24

384 384 768 1024

6 6 12 16

2.67 4.00 2.67 2.67

VECA-Small VECA-Small+ VECA-Base VECA-Large

Parameter counts. VECA closely matches the parameter count of the corresponding DINOv3 backbone at each model scale. The small increase comes from the learnable core tokens, core coordinates, and coordinate-update layers. Table S.2: Parameter counts. We compare VECA with the corresponding DINOv3 backbone at each model size. Model

VECA

DINOv3

VECA-Small VECA-Small+ VECA-Base VECA-Large

21.63 M 28.72 M 85.73 M 303.20 M

21.59 M 28.68 M 85.64 M 303.08 M

Shared architectural settings. Table S.3 summarizes the settings shared across model sizes. Attention is implemented using PyTorch scaled dot-product attention. Table S.3: Shared architectural settings across VECA variants. Component

Setting

Patch size Input channels Patch embedding Patch embedding normalization Dropout Maximum core capacity Core-token chunk size Active-core budgets Global representation Dense representation FFN type Attention implementation

16 × 16 3 Strided convolution None 0.0 M = 64 8 {8, 16, 24, 32, 40, 48, 56, 64} First core token r1 All patch tokens Z SwiGLU PyTorch scaled dot-product attention (SDPA)

23

Token and parameter layout.

The learnable core-token bank is denoted by RM = {r1 , . . . , rM },

where M is the maximum core capacity. The bank is stored as chunked parameters, with 8 core tokens per chunk and 8 chunks in total. For an active budget C, the model activates the first C/8 chunks, yielding the active prefix RC = RM [: C] = (r1 , . . . , rC ). Given patch tokens Z = {z1 , . . . , zN }, the input sequence to each block is ordered as X = [RC ; Z]. This fixed ordering is used by the attention implementation in Section A.3. Transformer block. Each block uses pre-normalization, core-mediated attention, and a SwiGLU feed-forward network: Xℓ′ = Xℓ + VECA_Attn(LN(Xℓ )) ,

Xℓ+1 = Xℓ′ + FFN(LN(Xℓ′ )) .

The exact implementation is given in Section A.3. Feed-forward network.

The feed-forward network uses a gated SwiGLU form:

[u, v] = W1 z,

FFN(z) = W2 (SiLU(u) ⊙ v) .

This design is used for all model sizes; VECA-Small+ differs from VECA-Small only by using a larger MLP expansion ratio. Core coordinates and positional encoding. Each patch token is assigned a fixed two-dimensional coordinate from the patch grid. Each core token is assigned a learnable two-dimensional coordinate initialized by farthest-point sampling in the normalized image plane. For core i at layer ℓ, we denote the unconstrained coordinate state by ρℓi ∈ R2 and the bounded coordinate used for RoPE by uℓi ∈ [−1, 1]2 : uℓi = tanh(ρℓi ). Both patch and core tokens use two-dimensional rotary positional encoding. For layers ℓ > 0, the core coordinate state is updated using a learned residual: ∆ρℓi = Wℓpos riℓ ,

ρℓi = ρℓ−1 + αℓ ∆ρℓi , i

uℓi = tanh(ρℓi ),

where riℓ is the feature of core token i at layer ℓ, Wℓpos is a lightweight coordinate-update head, and αℓ is a learned scalar initialized to a small value. Output representations. After the final transformer block, LayerNorm is applied to the full token sequence. The first final core token is used as the global image representation, y(x) = r1L , and the final patch tokens are used as the dense representation, L Z L (x) = {z1L , . . . , zN }.

Recent models have adopted randomly augmented embeddings [154, 18], 2D axial RoPE [127, 128, 129, 21], or more sophisticated (sometimes non-commutative) 2D RoPE variants [155, 156, 157, 158, 159]. For this work, we follow DINOv3 with the adoption of 2D axial RoPE, and leave the exploration of other positional encodings to future work.

24

A.2

Training and optimization details

This section provides the optimization and training settings used for VECA. We focus on implementation details that are not specified in the main paper. Training stages. VECA is trained in two stages. First, we train the model at 256 × 256 resolution. Second, we continue from the single-resolution checkpoint and perform multi-resolution finetuning for 50,000 steps over resolutions {256, 384, 512, 768}. Both stages use the same frozen-teacher objective and the same nested active-core training scheme. Table S.4: Training stages. Stage Single-resolution training Multi-resolution finetuning

Resolution

Duration

256 × 256 {256, 384, 512, 768}

135 epochs 50,000 steps

Training objective. For an image x and sampled active-core budget C, the student produces a (C) global representation y (C) (x) ∈ RD and dense patch representation Z (C) (x) = {zi (x)}N i=1 , where (C) zi (x) ∈ RD . The frozen teacher provides corresponding targets y ⋆ (x) and Z ⋆ (x) = {zi⋆ (x)}N i=1 . We optimize     L(x, C) = Lglobal y (C) (x), y ⋆ (x) + λdense Ldense Z (C) (x), Z ⋆ (x) . The global loss is cosine distance: Lglobal = 1 −

y (C) (x), y ⋆ (x) . y (C) (x) 2 ∥y ⋆ (x)∥2

The dense loss combines patch-wise cosine distance and mean-squared error: D E   (C) N N zi (x), zi⋆ (x) X 2 1 X (C)  + βmse 1 Ldense = 1− zi (x) − zi⋆ (x) . (C) N i=1 N D i=1 2 zi (x) ∥zi⋆ (x)∥2 2

We set λdense = 1.0, βmse = 1.0, and use normalization epsilon 10−6 . Training data. VECA is trained on Object365 images. For each training image, we apply both the global representation loss and the dense patch-level loss using targets from the frozen teacher. No labeled class supervision is used during backbone training. Teacher models. Each VECA model is trained with the corresponding DINOv3 teacher at the same scale. The teacher is frozen throughout training. When the teacher includes register tokens, we remove them before constructing dense patch targets. Table S.5: Teacher models used for VECA training. All teachers are DINOv3 models from the facebook/*-pretrain-lvd1689m family. Student

Teacher

Teacher micro-batch

VECA-Small VECA-Small+ VECA-Base VECA-Large

ViT-S/16 ViT-S+/16 ViT-B/16 ViT-L/16

128 128 128 64

Batch size. For VECA-Small, VECA-Small+, and VECA-Base, we use a per-rank batch size of 128 during single-resolution training. For VECA-Large, we use a per-rank micro-batch size of 64 and accumulate gradients for 2 steps. 25

Table S.6: Per-rank batch settings for single-resolution training. Model VECA-Small VECA-Small+ VECA-Base VECA-Large

Micro-batch

Grad. accum.

Effective batch

128 128 128 64

1 1 1 2

128 128 128 128

Input preprocessing and augmentation. For single-resolution training, images are resized to a short side of 288 pixels and cropped to 256 × 256. For multi-resolution finetuning, the crop resolution is sampled from {256, 384, 512, 768}, and the short side is resized to 1.125 times the crop resolution. Training uses random crop, random horizontal flip, and TrivialAugmentWide. Validation uses center crop. All images are normalized with µ = (0.485, 0.456, 0.406),

σ = (0.229, 0.224, 0.225).

Active-core budget sampling. During both stages, one active-core budget is sampled at each optimizer step from C ∈ {8, 16, 24, 32, 40, 48, 56, 64} with sampling weights (1, 1, 2, 2, 3, 3, 4, 4). The sampled budget is broadcast to all distributed ranks before the forward pass. Validation uses the full active-core budget unless otherwise specified. Optimization. We use a two-optimizer setup. Selected two-dimensional linear weights are optimized with NorMuon [160, 161], while the remaining trainable parameters are optimized with AdamW [162, 163]. We utilize the Dion library [164] for all optimizers. The AdamW group includes core tokens, core coordinates, coordinate-update layers, positional scaling parameters, biases, normalization parameters, patch embedding parameters, and selected output projection parameters. All other parameters are optimized with NorMuon. Table S.7: Optimizer and schedule settings. Setting AdamW learning rate NorMuon learning rate Minimum learning rate Weight decay AdamW epsilon Schedule Mixed precision Training hardware

Single-resolution

Multi-resolution finetuning

4 × 10−4 4 × 10−4 5 × 10−5 0.015 10−8 Warmup + cosine bfloat16 6 NVIDIA L40S GPUs

8 × 10−5 8 × 10−5 3 × 10−5 0.01 10−8 Warmup + cosine bfloat16 6 NVIDIA L40S GPUs

NorMuon settings. For NorMuon, we use momentum µ = 0.95, second-moment coefficient β2 = 0.95, Nesterov updates, RMS-normalized AdamW compatible learning-rate adjustment [165], and cautious weight decay [166]. The orthogonalization routine uses five Polar Express [167] iterations. We do not use parameter flattening. Distributed training and checkpointing. All models are trained using PyTorch DistributedDataParallel on 6 NVIDIA L40S GPUs. The active-core budget is sampled on the main process and broadcast to all ranks before each optimizer step. We use bfloat16 automatic mixed precision for both student and teacher forward passes. Validation metrics are averaged across distributed ranks. During multi-resolution finetuning, validation is performed over the resolution and budget grid, and the best checkpoint is selected according to the averaged validation loss.

26

Table S.8: NorMuon settings. Setting

Value

Momentum µ β2 Nesterov Learning-rate adjustment Cautious weight decay Newton–Schulz / Polar Express iterations Triton kernels Parameter flattening

0.95 0.95 True RMS norm True 5 False False

27

A.3

Core-mediated attention pseudocode

This section gives implementation-level code snippets for the VECA attention block. The model configurations are provided in Section A.1. We use M for the maximum core capacity, C ≤ M for the active number of core tokens, N for the number of patch tokens, and D for the hidden dimension. The ordered core-token bank is RM = (r1 , . . . , rM ). For an active budget C, the model uses the active prefix RC = RM [: C] = (r1 , . . . , rC ) ∈ RC×D . Given patch tokens Z ∈ RN ×D , the input sequence is ordered as X = [RC ; Z] ∈ R(C+N )×D . The listings below are generalized from the training implementation and omit model-size-specific constants. In the Python code, the variable active_c corresponds to the mathematical budget C. Core-mediated attention. The core-mediated attention layer uses separate query, key, and value projections. After applying two-dimensional RoPE to queries and keys, active core tokens attend to the full sequence, while patch tokens attend only to the active core-token prefix. class CoreMediatedAttention(nn.Module): def __init__(self, dim: int, num_heads: int, dropout: float = 0.0): super().__init__() assert dim % num_heads == 0 self.num_heads = num_heads self.head_dim = dim // num_heads self.q_proj = nn.Linear(dim, dim, bias=True) self.k_proj = nn.Linear(dim, dim, bias=True) self.v_proj = nn.Linear(dim, dim, bias=True) self.out_proj = nn.Linear(dim, dim, bias=True) self.drop_p = float(dropout) self.out_drop = nn.Identity() if dropout == 0.0 else nn.Dropout(dropout) def forward(self, x: torch.Tensor, rope: Rotary2D, coords: torch.Tensor, active_c: int) -> torch.Tensor: B, T, D = x.shape H, Hd = self.num_heads, self.head_dim assert 1 <= active_c < T q = self.q_proj(x) k = self.k_proj(x) v = self.v_proj(x) q = q.view(B, T, H, Hd).transpose(1, 2) k = k.view(B, T, H, Hd).transpose(1, 2) v = v.view(B, T, H, Hd).transpose(1, 2) cos, sin = rope.cos_sin(coords) q = rope.apply(q, cos.to(q.dtype), sin.to(q.dtype)) k = rope.apply(k, cos.to(k.dtype), sin.to(k.dtype)) p = self.drop_p if self.training else 0.0 # Active core tokens R_C attend to all active cores and all patches. y_core = F.scaled_dot_product_attention( q[:, :, :active_c, :], k, v, dropout_p=p, is_causal=False, ) # Patch tokens Z attend only to the active core-token prefix R_C. y_patch = F.scaled_dot_product_attention( q[:, :, active_c:, :], k[:, :, :active_c, :], v[:, :, :active_c, :], dropout_p=p, is_causal=False, ) y = torch.cat([y_core, y_patch], dim=2) y = y.transpose(1, 2).contiguous().view(B, T, D) y = self.out_proj(y) return self.out_drop(y)

28

For visualization, the corresponding attention pattern can be written with output rows ordered as (Z, R) and source columns ordered as (R, Z):   AZ←R 0 A= . AR←R AR←Z The zero block denotes the removed patch-to-patch interaction AZ←Z . This display ordering is only for illustrating the sparsity pattern; the implementation keeps the token sequence ordered as [RC ; Z]. VECA transformer block. Each block applies pre-normalized core-mediated attention followed by a pre-normalized SwiGLU feed-forward network. class VECABlock(nn.Module): def __init__(self, dim: int, num_heads: int, mlp_ratio: float, dropout: float = 0.0): super().__init__() self.norm_attn = nn.LayerNorm(dim) self.attn = CoreMediatedAttention(dim, num_heads, dropout) hidden = int(dim * mlp_ratio) self.norm_ffn = nn.LayerNorm(dim) self.fc1 = nn.Linear(dim, 2 * hidden, bias=True) self.fc2 = nn.Linear(hidden, dim, bias=True) self.drop = nn.Identity() if dropout == 0.0 else nn.Dropout(dropout) def forward(self, x: torch.Tensor, rope: Rotary2D, coords: torch.Tensor, active_c: int) -> torch.Tensor: x = x + self.attn( self.norm_attn(x), rope=rope, coords=coords, active_c=active_c, ) y = self.norm_ffn(x) u, v = self.fc1(y).chunk(2, dim=-1) y = F.silu(u) * v y = self.drop(y) y = self.fc2(y) y = self.drop(y) return x + y

Active core selection and coordinate update. VECA stores the core-token bank RM and the corresponding core-coordinate bank in chunks. For a budget C, the forward pass activates the first C core tokens and updates only their coordinates layer by layer. class VECAEncoder(nn.Module): def _build_active_cores(self, batch_size: int, active_c: int): n_chunks = active_c // self.core_chunk_size cores = torch.cat( [self.core_token_chunks[j] for j in range(n_chunks)], dim=1, ).expand(batch_size, -1, -1) core_raw = torch.cat( [self.core_coord_chunks[j] for j in range(n_chunks)], dim=1, ).expand(batch_size, -1, -1) return cores, core_raw @torch.no_grad() def _patch_grid(self, hp: int, wp: int, device: torch.device) -> torch.Tensor: y = (torch.arange(hp, device=device).float() + 0.5) / hp * 2.0 - 1.0 x = (torch.arange(wp, device=device).float() + 0.5) / wp * 2.0 - 1.0 gy, gx = torch.meshgrid(y, x, indexing="ij") return torch.stack([gx, gy], dim=-1).reshape(-1, 2)

VECA forward pass. The full encoder first computes patch tokens Z, concatenates the active core-token prefix RC with the patch sequence, and then applies VECA blocks. The first final core token is returned as the global representation, and the final patch tokens are returned as dense features. 29

class VECAEncoder(nn.Module): def forward(self, images: torch.Tensor, active_c: Optional[int] = None): B = images.shape[0] active_c = self.num_core_tokens if active_c is None else int(active_c) assert 1 <= active_c <= self.num_core_tokens assert active_c % self.core_chunk_size == 0 patches, (hp, wp) = self.patch_embed(images) cores, core_raw = self._build_active_cores(B, active_c) patch_coords = self._patch_grid(hp, wp, patches.device) patch_coords = patch_coords.unsqueeze(0).expand(B, -1, -1) patch_coords = self.rope.augment_coords_patch(patch_coords) core_coords = torch.tanh(core_raw) x = torch.cat([cores, patches], dim=1) for layer_idx, block in enumerate(self.blocks): if layer_idx > 0: active_cores = x[:, :active_c, :] delta = self.pos_linears[layer_idx - 1](active_cores) alpha = self.pos_alpha[layer_idx - 1].to(delta.dtype) core_raw = core_raw + alpha * delta core_coords = torch.tanh(core_raw) coords = torch.cat([core_coords, patch_coords], dim=1) x = block(x, self.rope, coords, active_c) x = self.final_norm(x) cores = x[:, :active_c, :] patches = x[:, active_c:, :] global_feature = cores[:, 0, :] dense_features = patches return global_feature, dense_features

30

A.4

Baseline model details

This section summarizes the baseline backbones used for comparison. We only describe the model choices used in the paper; task-specific preprocessing, feature extraction, and evaluation protocols are provided in Section A.5. Baseline backbones. We compare VECA with a set of widely used vision backbones spanning contrastive vision-language pretraining, self-supervised representation learning, and recent vision foundation models. All baseline models are evaluated using their publicly released checkpoints. Table S.9: Baseline models used in our evaluation. The asterisk indicates a resolution-specific checkpoint variant. Family

Backbone Checkpoint

CLIP OpenCLIP DFN-CLIP DINOv2 DINOv2-Reg DINOv3 SigLIP2∗ C-RADIOv3

ViT-B/16 ViT-B/16 ViT-B/16 ViT-B/14 ViT-B/14 ViT-B/16 ViT-B/16 ViT-B/16

openai/clip-vit-base-patch16 ViT-B-16/laion2b_s34b_b88k apple/DFN2B-CLIP-ViT-B-16 facebook/dinov2-base facebook/dinov2-with-registers-base facebook/dinov3-vitb16 google/siglip2-base-patch16 nvidia/C-RADIOv3-B

Model usage. For each baseline, we use the official or checkpoint-compatible loading interface associated with the released model. All backbone parameters are kept frozen during downstream evaluation. For OpenCLIP and DFN-CLIP, we use the visual backbone from the released checkpoint. For C-RADIOv3, we use the model-provided external preprocessor together with the released backbone.

31

A.5

Implementation details for downstream tasks

This section provides implementation details for downstream evaluation. We separate the classification protocol from dense prediction protocols. A.5.1

Image Classification

Linear probing protocol. For image classification benchmarks, we freeze the visual backbone and train only a linear classification head on top of the extracted global feature. For VECA, the global feature is the first final core token. For each baseline, we use the corresponding global visual representation defined by its released implementation. The backbone is kept in evaluation mode throughout linear probing. Optimization. The linear head is trained with SGD using momentum 0.9, learning rate 0.05, and weight decay 0. We train for 12,500 steps with batch size 128 and a cosine learning-rate schedule. Validation is performed every 1,250 steps, and we keep the checkpoint with the best validation top-1 accuracy. Resolution and preprocessing. For classification benchmarks, each backbone is evaluated at its native or fixed crop resolution. CLIP, OpenCLIP, DFN, DINOv2, and DINOv2-Reg use a 224 × 224 crop. DINOv3, SigLIP2∗ , C-RADIOv3, and VECA use a 256 × 256 crop. Before cropping, we resize the short side to 1.125 times the crop size, giving 252 for 224 × 224 models and 288 for 256 × 256 models. We use the checkpoint-compatible normalization for each backbone. For ViT-style HuggingFace backbones, positional-embedding interpolation is disabled. Evaluation metric. A.5.2

We report top-1 accuracy for classification benchmarks.

Dense Prediction

Linear probing protocol. For semantic segmentation and depth estimation benchmarks, we train a linear prediction head while keeping the backbone frozen. The entire evaluation pipeline is implemented based on the MMSegmentation framework [168]. For both VECA and all baselines, we extract dense patch features and append the CLS token to each patch representation to enrich the feature representations [169]. For SigLIP 2, which does not include a CLS token, we directly use patch features for depth evaluation. All backbones remain in evaluation mode throughout linear probing. Optimization. The linear head is trained using the AdamW optimizer with betas (0.9, 0.999), a learning rate of 0.01, and a weight decay of 0.0001, together with a polynomial learning rate scheduler. The number of training steps depends on the benchmark. We use 20,000 steps for VOC, 40,000 for Context and ADE, 80,000 for Stuff and Object, 12,000 for Cityscapes, and 38,400 for NYUv2 and KITTI. The batch size is set to 8. For evaluation, we retain the checkpoint with the best validation mIoU or RMSE, depending on the task. Resolution and preprocessing. For dense prediction benchmarks, each backbone is evaluated using a fixed crop resolution of 512 × 512 for models with patch size 16 and 518 × 518 for models with patch size 14. Before cropping, we resize the shorter side to 512 or 518 and apply random cropping during training. For CLIP, OpenCLIP, and DFNCLIP models with absolute positional embeddings, we interpolate the positional embeddings to match the input resolution. Evaluation metrics. We report mIoU for semantic segmentation benchmarks and RMSE for depth estimation benchmarks.

32

A.6

Multi-resolution evaluation across tasks

This section provides the multi-resolution evaluation protocol. We split the protocol into image classification, dense prediction, and qualitative comparison. A.6.1

Image Classification

Protocol. For multi-resolution classification, we evaluate VECA, DINOv3, and DINOv2 across model sizes. For VECA and DINOv3, we use resolutions {256, 384, 512, 768}. For DINOv2, which uses patch size 14, we use the closest patch-aligned resolutions {252, 392, 518, 770}. All backbones are frozen, and a separate linear head is trained for each model and resolution using the protocol in Section A.5.1. Table S.10: Multi-resolution classification results. Each cell reports top-1 accuracy in the order (IN1K, V2, ReaL). Column headers show the VECA/DINOv3 resolution, with the corresponding DINOv2 patch-aligned resolution in parentheses. Model

256 (252)

384 (392)

512 (518)

768 (770)

DINOv3-S DINOv3-S+ DINOv3-B DINOv3-L

77.31 / 66.69 / 84.58 79.95 / 70.31 / 86.54 83.56 / 74.92 / 88.65 86.40 / 78.84 / 90.12

78.43 / 67.84 / 85.42 80.77 / 71.32 / 87.05 84.14 / 75.80 / 88.90 86.67 / 79.28 / 90.11

78.81 / 68.32 / 85.70 81.23 / 71.88 / 87.22 84.15 / 75.84 / 88.95 86.67 / 79.50 / 90.08

78.55 / 68.05 / 85.48 81.19 / 71.63 / 87.22 84.08 / 75.53 / 88.80 86.74 / 79.41 / 90.18

DINOv2-S DINOv2-B DINOv2-L

78.77 / 68.40 / 84.41 79.76 / 70.14 / 85.17 79.94 / 70.28 / 85.26 79.41 / 69.63 / 84.82 82.51 / 73.66 / 86.97 83.02 / 74.30 / 87.06 83.34 / 74.88 / 87.28 82.81 / 74.68 / 87.10 84.58 / 76.40 / 88.01 84.79 / 76.88 / 88.20 84.74 / 76.67 / 88.03 84.41 / 76.54 / 87.98

VECA-S VECA-S+ VECA-B VECA-L

74.63 / 62.87 / 82.44 77.03 / 66.25 / 84.39 81.93 / 72.25 / 87.87 84.51 / 76.07 / 89.52

75.39 / 64.23 / 83.01 77.58 / 67.50 / 84.61 82.37 / 72.76 / 88.07 84.67 / 76.44 / 89.71

75.74 / 64.72 / 83.42 77.82 / 67.38 / 84.92 82.25 / 73.07 / 88.01 84.73 / 76.52 / 89.68

75.42 / 64.24 / 83.03 77.50 / 67.40 / 84.70 82.26 / 73.28 / 88.00 84.53 / 76.91 / 89.53

Summary. As shown in Table S.10, VECA remains competitive with DINOv2 and DINOv3 across the full resolution sweep while using a core-mediated attention interface. At Base scale, VECA-B preserves 97.7%−98.1% of DINOv3-B ImageNet-1K accuracy and 98.9%−99.1% of DINOv3B ImageNet-ReaL accuracy across all evaluated resolutions. At Large scale, VECA-L reaches 97.5%−97.8% of DINOv3-L ImageNet-1K accuracy and 99.3%−99.6% of DINOv3-L ImageNetReaL accuracy. Compared with DINOv2-L, VECA-L is nearly matched on ImageNet-1K across resolutions and is consistently higher on ImageNet-ReaL. These results indicate that the learned core interface transfers robustly across input resolutions and preserves strong global recognition performance. A.6.2

Dense Prediction

Protocol. For multi-resolution semantic segmentation and monocular depth estimation, we evaluate VECA and DINOv3 across model sizes on several benchmarks since DINOv3 is the strongest model on dense prediction tasks. For VECA and DINOv3, we use resolutions {256, 384, 512, 768}. All backbones are frozen, and a separate linear head is trained for each model and resolution using the protocol in Section A.5.2. Summary. As shown in Table S.11 and Table S.12, VECA achieves performance comparable to or better than DINOv3 across model scales through its core-mediated attention interface. The performance of both models improves as the input resolution increases, since higher resolutions provide richer spatial details that benefit dense prediction tasks. However, the FLOPs of DINOv3 grow quadratically with resolution, whereas VECA exhibits linear scaling. At the Base scale, VECAB achieves performance on par with DINOv3-B, with a gap of less than 3% across all evaluated resolutions. At the Large scale, VECA-L surpasses DINOv3-L on several datasets, including Context, ADE, and NYUv2, demonstrating strong scaling potential. These results indicate that 33

Table S.11: Multi-resolution semantic segmentation results. Each cell reports mIoU in the order (Context, ADE, Stuff, Object). Column headers show the VECA/DINOv3 resolution. Model

256

384

512

768

DINOv3-S DINOv3-S+ DINOv3-B DINOv3-L

51.78 / 41.28 / 47.74 / 54.60 51.81 / 41.67 / 42.13 / 55.54 55.02 / 46.61 / 45.87 / 60.67 56.50 / 49.75 / 47.36 / 62.55

54.06 / 44.55 / 43.82 / 57.83 54.87 / 46.44 / 44.95 / 59.51 57.07 / 49.53 / 47.97 / 63.67 58.33 / 52.66 / 48.93 / 65.13

55.01 / 46.01 / 44.67 / 59.49 55.95 / 48.57 / 45.86 / 61.12 57.74 / 51.35 / 48.41 / 64.65 59.13 / 54.56 / 49.71 / 66.53

55.10 / 46.62 / 45.04 / 59.83 56.38 / 49.53 / 46.27 / 61.69 58.40 / 52.37 / 49.04 / 65.78 59.70 / 55.52 / 50.12 / 67.28

VECA-S VECA-S+ VECA-B VECA-L

50.98 / 40.61 / 40.49 / 52.91 51.19 / 41.07 / 41.32 / 54.16 54.91 / 46.09 / 45.43 / 59.71 56.66 / 50.05 / 47.36 / 62.28

52.85 / 43.19 / 42.86 / 55.92 54.02 / 44.78 / 43.53 / 57.50 56.79 / 49.24 / 47.27 / 62.69 58.61 / 52.99 / 48.92 / 64.94

53.46 / 44.82 / 43.45 / 57.46 54.68 / 46.55 / 44.33 / 58.76 57.46 / 50.69 / 47.92 / 62.58 59.64 / 54.17 / 49.53 / 66.19

53.87 / 45.28 / 43.77 / 57.96 54.68 / 46.55 / 44.33 / 58.76 58.05 / 51.46 / 48.37 / 64.14 60.23 / 55.54 / 50.06 / 66.85

Table S.12: Multi-resolution depth estimation results. Each cell reports RMSE in the order (NYUv2, KITTI). Column headers show the VECA/DINOv3 resolution. Model

256

384

512

768

DINOv3-S DINOv3-S+ DINOv3-B DINOv3-L

0.4565 / 3.2402 0.4651 / 3.3499 0.3897 / 2.9636 0.3663 / 2.9207

0.4252 / 3.1130 0.4225 / 2.9987 0.3727 / 2.8962 0.3460 / 2.7836

0.4169 / 2.9083 0.4143 / 2.8127 0.3684 / 2.6915 0.3446 / 2.5346

0.4206 / 2.8943 0.4140 / 2.8125 0.3665 / 2.6840 0.3445 / 2.4821

VECA-S VECA-S+ VECA-B VECA-L

0.4528 / 3.2776 0.4597 / 3.3009 0.3920 / 2.9765 0.3546 / 2.8792

0.4360 / 3.2106 0.4316 / 3.1318 0.3768 / 2.8976 0.3409 / 2.8019

0.4347 / 3.0130 0.4252 / 2.9878 0.3705 / 2.7252 0.3379 / 2.5904

0.4298 / 3.0289 0.4231 / 3.0176 0.3704 / 2.7468 0.3382 / 2.5634

VECA achieves a better balance between computational cost and competitive performance at high resolutions, resulting in improved inference efficiency.

34

A.6.3

Additional Qualitative Comparison

Figure S.1: Additional qualitative multi-resolution comparison. We compare dense patch representations from VECA and baseline ViTs across increasing input resolutions. Each method column shows spatial feature organization and patch-to-patch similarity maps. VECA maintains coherent object boundaries and semantically smooth regions across resolutions, while several baselines show noisier or more fragmented similarity patterns, especially at higher resolutions.

35

Figure S.2: Additional qualitative multi-resolution comparison. We compare dense patch representations from VECA and baseline ViTs across increasing input resolutions. Each method column shows spatial feature organization and patch-to-patch similarity maps. VECA maintains coherent object boundaries and semantically smooth regions across resolutions, while several baselines show noisier or more fragmented similarity patterns, especially at higher resolutions.

36

A.7

Additional core-to-patch similarity visualizations Selected Core Tokens

Figure S.3: Additional core-to-patch similarity visualizations. For each RGB image, we show a UMAP projection of dense patch features followed by similarity maps for selected core tokens from our model. Warmer colors indicate image regions with higher cosine similarity to a given core token. The selected cores attend to complementary objects, parts, and spatial regions, illustrating that different core tokens specialize in distinct visual structures while preserving coherent scene-level organization.

37

A.8

Additional classification results

This section reports the full set of classification results used in our base model evaluation, including datasets that are not shown in the main paper. We use the same frozen-backbone linear probing protocol as Section A.5.1. All backbones are evaluated using their standard classification preprocessing: CLIP, OpenCLIP, DFN, DINOv2, and DINOv2-Reg use a 224 × 224 crop, while DINOv3, SigLIP2-256, C-RADIOv3, and VECA use a 256 × 256 crop. Before cropping, the short side is resized to 1.125 times the crop size, and we use the checkpoint-compatible normalization for each backbone. We evaluate on datasets spanning ImageNet-style recognition, scene recognition, fine-grained recognition, texture recognition, and multi-label object classification. Table S.13: Full classification results across datasets. We report top-1 accuracy for each dataset using frozen visual backbones and a trained linear head. VECA is evaluated with the full core budget unless otherwise specified; C = 8 denotes a reduced-core inference setting. Model

IN1K

V2

ReaL Places Food C10 C100 SUN Cars Aircraft VOC DTD Caltech Pets CUB Flowers

DINOv3-B 83.56 74.92 88.65 DINOv2-Reg-B 83.44 74.75 88.18 DINOv2-B 82.45 72.98 86.84 SigLIP2-B/16-256 81.95 73.04 87.69 DFN2B-CLIP-B/16 81.06 71.35 86.56 C-RADIOv3-B 80.53 70.60 85.78 CLIP-B/16 79.33 69.31 84.93 OpenCLIP-B/16 79.18 69.05 84.72 VECA-B (C = 64) 81.93 72.25 87.87 VECA-B (C = 8) 79.99 69.18 86.45

55.39 54.81 53.37 56.18 55.79 54.13 55.27 55.35 55.82 55.30

94.47 98.32 90.25 77.59 93.32 92.87 98.45 90.65 76.85 90.62 92.02 98.44 89.80 75.67 86.62 94.70 96.33 82.30 78.92 92.55 93.32 97.87 88.51 78.68 94.33 91.64 97.77 89.21 76.91 89.14 92.42 94.23 82.10 78.05 85.14 91.33 96.81 84.84 78.83 92.65 92.44 97.65 87.09 76.60 91.84 90.04 96.76 84.71 75.35 90.18

81.97 70.78 74.02 62.80 66.49 52.18 56.65 62.89 74.47 70.72

73.99 80.32 84.22 79.20 85.79 79.47 77.00 75.32 86.60 80.74 89.01 81.54 84.57 74.15 85.87 79.41 71.64 77.01 70.47 75.80

92.65 93.86 93.36 95.40 97.91 95.20 94.30 96.97 92.08 91.98

96.54 89.82 95.97 89.52 95.61 89.06 92.89 78.41 93.59 85.90 93.54 81.14 92.72 80.51 91.91 83.15 95.67 88.02 95.12 87.00

99.69 99.67 99.59 97.98 98.70 98.52 96.36 96.50 99.35 98.98

Summary. Table S.13 summarizes the full classification benchmark suite. VECA-B remains competitive across ImageNet-style recognition, scene recognition, fine-grained recognition, texture recognition, and multi-label classification. Compared with DINOv3-B, VECA-B is close on ImageNet-1K, ImageNet-ReaL and several transfer datasets, while slightly trailing on ImageNet-V2, and some fine-grained settings. VECA-B obtains strong results on Places365, Stanford Cars, Oxford Pets, CUB-200, and Flowers-102, indicating that the core-mediated representation preserves useful global recognition information across diverse visual domains. The reduced-budget VECA-B (C = 8) remains competitive on many datasets, showing that a small active core set can retain much of the classification-relevant information.

38

A.9

Additional emergent behavior visualizations

In this section, we provide additional visualizations of how VECA organizes patch representations through its learned core-token interface. A.9.1

Analysis Method

To analyze the behavior of core tokens, we visualize how patch tokens interact with the active core set across layers and core budgets. Rather than simply averaging softmax attention over heads, we compute an output-contribution attention map that accounts for the attention weight, the value feature magnitude, and the output projection. For an active budget C, let Aℓhij denote the attention probability from query token i to key token j in ℓ head h at layer ℓ, vhj denote the corresponding value vector, and WhO denote the output projection slice for head h. We define the contribution from token j to token i as eℓij =

H X

ℓ WhO , Aℓhij vhj

h=1

sℓij = P

∥eℓij ∥2

ℓ j ′ ∥eij ′ ∥2

.

(5)

Here sℓij measures the normalized output-space importance of key token j for query token i. This differs from raw attention averaging because it reflects not only where the model attends, but also how strongly the attended values contribute after the output projection [170]. For layer-wise visualizations, each patch is represented by its normalized contribution profile over the active cores. For each image and core budget, we collect these profiles from selected layers and jointly embed them into a three-dimensional space using UMAP. The resulting three coordinates are normalized and interpreted as RGB values, producing a spatial map over the image grid. We exclude the first layer from this analysis because early attention is often dominated by low-level initialization effects. A.9.2

Additional Visualizations

39

Figure S.4: Additional layer-wise emergent behavior visualizations. We visualize UMAP embeddings of patch contribution profiles over active core tokens across layers and core budgets. Please zoom in for details.

40

Figure S.5: Additional layer-wise emergent behavior visualizations. We visualize UMAP embeddings of patch contribution profiles over active core tokens across layers and core budgets. Please zoom in for details.

41

Figure S.6: Additional layer-wise emergent behavior visualizations. We visualize UMAP embeddings of patch contribution profiles over active core tokens across layers and core budgets. Please zoom in for details.

42

A.10

Efficiency analysis and computational cost

This section provides efficiency analysis for VECA. The main computational advantage of VECA comes from replacing dense patch-to-patch self-attention with core-mediated attention. For an active core budget C and N patch tokens, the attention interaction cost scales as O(N 2 )

(2N C + C 2 ).

−→

Thus, when C ≪ N , the benefit becomes more pronounced as the input resolution increases. Benchmark design. We compare VECA with the corresponding DINOv3 backbone at the Small, Base, and Large scales. To isolate the cost of the attention mechanism, we benchmark the same projection-attention-projection path for both models: query, key, and value projections, scaled dotproduct attention, and the output projection. For VECA, patch tokens attend only to the active core tokens, while core tokens attend to the full sequence. For DINOv3, all tokens use standard full self-attention. Unless otherwise stated, VECA uses the full active core budget C = 64. We measure FLOPs over a resolution sweep from 256 to 1024, and report GPU latency on an NVIDIA L40S 48GB GPU with bfloat16 precision.

Figure S.7: Resolution sweep for attention cost. We compare VECA and DINOv3 across input resolutions for Small, Base, and Large models. Left: FLOPs per image. Right: GPU latency at batch size 8 on an NVIDIA L40S 48GB GPU. VECA becomes increasingly efficient as resolution grows because the removed patch-to-patch attention term dominates at high token counts. Resolution scaling. Figure S.7 shows that the efficiency advantage grows with resolution. At lower resolutions, the sequence length is smaller and fixed projection/kernel overheads are more visible, so the gap between VECA and DINOv3 is modest. At the largest evaluated resolution, VECA substantially reduces both FLOPs and GPU latency across all model sizes. For VECA-Small, FLOPs decrease from 30.67G to 5.72G, a 5.36× reduction, while latency decreases from 1.57ms to 0.72ms. For VECA-Base, FLOPs decrease from 71.02G to 21.25G, and latency decreases from 4.08ms to 2.20ms. For VECA-Large, FLOPs decrease from 103.29G to 37.06G, and latency decreases from 6.07ms to 3.37ms. Batch-size scaling. We further evaluate GPU latency of the attention residual block over batch sizes from 1 to 128 using the same high-resolution setting. This tests whether the advantage persists under larger batched workloads. As shown in Figure S.8, the VECA attention block maintains lower latency across the batch-size sweep. At the largest batch size, VECA-Small reduces latency from 29.18ms to 17.33ms, VECABase reduces latency from 70.04ms to 37.79ms, and VECA-Large reduces latency from 99.92ms to 56.07ms. These correspond to 1.68×, 1.85×, and 1.78× speedups, respectively. 43

Table S.14: Attention cost at the largest evaluated resolution (1024 × 1024). FLOPs are reported per image. GPU latency is reported per batch with batch size 8 on an NVIDIA L40S 48GB GPU. Model VECA FLOPs DINOv3 FLOPs FLOP red. VECA ms DINOv3 ms Speedup Small Base Large

5.72G 21.25G 37.06G

5.36× 3.34× 2.79×

30.67G 71.02G 103.29G

0.72 2.20 3.37

1.57 4.08 6.07

2.19× 1.86× 1.80×

Figure S.8: Batch-size scaling at high resolution. GPU latency is measured on an NVIDIA L40S 48GB GPU. The VECA attention residual block remains consistently faster than the corresponding DINOv3 residual block across model sizes and batch sizes. Summary. Overall, the efficiency results support the intended scaling behavior of VECA. The gains are most visible at high resolution, where standard ViT self-attention becomes dominated by patch-to-patch interactions. For a fixed active core budget C, VECA reduces this cost while retaining bidirectional communication between the compact core set and the dense patch sequence, yielding lower FLOPs and consistently lower GPU latency in the regimes used for dense visual tasks.

44

A.11

Additional patch-to-patch similarity visualizations

Figure S.9: Additional patch-to-patch similarity visualizations. For each RGB image, we visualize patch-to-patch cosine similarity maps from our model. The red cross marks the selected reference patch, and warmer colors indicate patches with higher feature similarity to that reference location. Across diverse scenes, the similarity maps often recover semantically or structurally related regions, suggesting that the learned patch representations preserve coherent object-, part-, and layout-level relationships.

45

A.12

Additional ablation studies

This section provides additional ablation studies for VECA. We split the analysis into classification and dense prediction settings. A.12.1

Classification Ablation

Protocol. We evaluate the effect of the active core-token budget during linear probing. For VECA, we vary the number of active core tokens C ∈ {8, 16, 24, 32, 40, 48, 56, 64}, freeze the backbone, and train a linear head on the extracted global representation. We use the same linear probing protocol as Section A.5.1. All VECA models are evaluated at 256 × 256, consistent with the classification evaluation protocol. Table S.15: Classification ablation over active core-token budgets. Each cell reports top-1 accuracy in the order (IN1K, V2, ReaL). C denotes the number of active core tokens used during linear probing. C

VECA-S

VECA-S+

VECA-B

VECA-L

64 56 48 40 32 24 16 8

74.63 / 62.87 / 82.44 74.22 / 62.73 / 82.02 74.04 / 62.99 / 82.13 73.89 / 62.94 / 81.91 73.73 / 62.73 / 81.81 73.52 / 62.40 / 81.62 72.87 / 61.70 / 81.06 71.30 / 59.87 / 79.59

77.03 / 66.25 / 84.39 76.97 / 65.99 / 84.24 76.97 / 65.65 / 84.28 76.97 / 65.83 / 84.26 77.03 / 65.67 / 84.23 76.65 / 65.33 / 83.91 76.17 / 64.91 / 83.56 74.39 / 62.93 / 81.95

81.93 / 72.25 / 87.87 81.84 / 72.16 / 88.00 81.79 / 72.14 / 87.81 81.71 / 72.21 / 87.83 81.63 / 71.96 / 87.74 81.51 / 71.93 / 87.68 81.09 / 71.25 / 87.41 79.99 / 69.18 / 86.45

84.51 / 76.07 / 89.52 84.27 / 75.96 / 89.51 84.35 / 75.86 / 89.54 84.23 / 75.85 / 89.47 84.23 / 75.77 / 89.49 84.15 / 75.42 / 89.35 83.90 / 75.62 / 89.25 83.06 / 74.39 / 88.64

Summary. Table S.15 shows that VECA classification performance is stable across a wide range of active core-token budgets. For Base and Large models, reducing the budget from C = 64 to C = 32 produces only a small drop on ImageNet-1K and ImageNet-ReaL. The degradation becomes more visible at very small budgets such as C = 8, but the model still preserves substantial recognition performance. This supports the nested core-token design: earlier core tokens retain much of the global classification information, while larger budgets provide additional capacity. A.12.2

Dense Prediction Ablation

Protocol. We will report dense prediction ablations over the active core-token budget using the same frozen-backbone dense evaluation protocol as Section A.5. All VECA models are evaluated at 512 × 512, consistent with the classification evaluation protocol. Table S.16: Dense prediction ablation over active core-token budgets. Each cell reports the metrics in the order of mIoU for (VOC, ADE), and RMSE for NYUv2. C denotes the number of active core tokens used during linear probing. C

VECA-S

VECA-S+

VECA-B

VECA-L

64 56 48 40 32 24 16 8

81.56 / 44.82 / 0.4347 81.45 / 44.80 / 0.4352 81.51 / 44.53 / 0.4361 81.40 / 43.87 / 0.4382 81.00 / 43.98 / 0.4430 81.06 / 43.23 / 0.4476 80.40 / 41.20 / 0.4562 77.56 / 37.48 / 0.4975

84.12 / 46.55 / 0.4252 83.92 / 46.24 / 0.4259 83.94 / 46.08 / 0.4270 83.84 / 45.71 / 0.4299 83.84 / 45.56 / 0.4327 83.43 / 44.60 / 0.4375 82.60 / 43.03 / 0.4453 79.18 / 38.96 / 0.4898

87.07 / 50.69 / 0.3705 86.94 / 50.40 / 0.3751 86.91 / 50.63 / 0.3748 86.85 / 50.53 / 0.3754 86.72 / 49.49 / 0.3789 86.34 / 49.44 / 0.3829 85.37 / 47.52 / 0.3965 83.84 / 44.51 / 0.4330

88.62 / 54.17 / 0.3379 88.62 / 54.17 / 0.3398 88.59 / 54.09 / 0.3407 88.51 / 53.81 / 0.3412 88.37 / 53.90 / 0.3432 87.78 / 53.65 / 0.3474 87.67 / 52.20 / 0.3563 86.50 / 48.10 / 0.3956

Results. Table S.16 reports VECA performance on dense prediction tasks, including semantic segmentation on VOC and ADE, and depth estimation on NYUv2. Overall, performance steadily 46

improves as the number of active core tokens increases. Across all model sizes, reducing the budget from C = 64 to C = 32 leads to a noticeable performance drop, particularly on dense prediction tasks that require fine-grained spatial information. The degradation becomes more significant at very small budgets, such as C = 8. Combined with the image classification results, these trends support the effectiveness of the nested core-token design. Earlier core tokens preserve much of the global visual information needed for recognition, while larger token budgets introduce additional spatial details that are especially important for dense prediction tasks.

47