arXiv CSopen access

Visual Access Boundaries in Vision-Language Model Reasoning

Visual Access Boundaries in Vision-Language Model Reasoning

Hiroto Osaka The University of Tokyo [email protected] Gouki Minegishi The University of Tokyo [email protected]

arXiv:2607.12815v1 [cs.AI] 14 Jul 2026

Masahiro Suzuki The University of Tokyo [email protected]

Shohei Taniguchi The University of Tokyo [email protected] Kai Yamashita The University of Tokyo [email protected] Yutaka Matsuo The University of Tokyo [email protected]

Abstract Chain-of-Thought (CoT) prompting is widely used as a test-time scaling strategy for Vision-Language Models (VLMs), but it remains unclear what is extended when VLMs generate longer reasoning traces. We ask whether CoT requires continued access to image tokens, or whether it mainly operates over visual information already made available earlier in the forward pass. We introduce Visual Access Sweep, a causal intervention that masks attention from generated-token queries to image-token keys along layer depth and generation time, and define the Visual Access Boundary (VAB) as the minimal access region that preserves task accuracy. Across six model configurations from Qwen2.5-VL and InternVL3, both no-CoT direct answering and CoT prompting exhibit finite VABs. In Qwen2.5-VL-32B and InternVL3 at 14B and 38B scales, when CoT is evaluated against the noCoT full-access target, its VAB layer differs from the no-CoT boundary by at most two layers, despite substantially longer generations. This suggests that CoT does not primarily improve performance by prolonging direct image-token access throughout the reasoning trace, but by extending language-side computation over image-derived hidden-state information. We further show that CoT gains are constrained by perceptual readout. CoT helps when the queried visual attribute can be reliably read out by the model, but not when that readout is unreliable. A symbolic-attribute oracle shows that CoT can improve counting once ground-truth attributes are supplied as text, while a single-object probe-vs-decode check shows that hard attributes can be linearly recoverable from hidden states yet difficult for the model itself to output. Together, these analyses place the bottleneck at readout rather than counting.

1

Introduction

CoT prompting is a standard test-time scaling tool in LLMs [Kojima et al., 2022, Wei et al., 2022, Wang et al., 2023] and has also been adopted in VLMs [Zhang et al., 2024, Xu et al., 2025, Lin et al., 2025]. However, its benefits on visual tasks are not uniform: some studies report gains from explicit intermediate reasoning, whereas others find limited or degraded gains on perception-heavy visual QA settings [Lu et al., 2024, Liu et al., 2024b]. What mechanism underlies CoT’s contribution to VLM Preprint.

Figure 1: Overview of Visual Access Sweep and the Visual Access Boundary. (a) Visual Access Sweep. At each sweep point (tcutoff , ℓcutoff ), we allow direct image-token access only for generatedtoken queries satisfying t ≤ tcutoff and layers satisfying ℓ ≤ ℓcutoff . Outside this allowed rectangle, attention from generated-token queries to image-token keys is masked. The Visual Access Boundary (VAB) is the minimal access pair (ℓ∗ , τ ∗ ) that preserves accuracy within tolerance ϵ. (b) Key result. Under direct prompting, knocking out later layers barely affects accuracy, identifying a depth ℓ∗ beyond which continued direct image-token access is functionally redundant. Under CoT prompting, the VAB region is preserved despite a much larger generated-token count, indicating that extended language-side reasoning does not require extended direct image-token access. performance? Does CoT improve performance by prolonging direct access to image tokens during generation, or by enabling additional language-side computation over visual information already made available earlier? Existing evidence on this question is largely observational: studies document that VLM attention to image tokens decays as generation proceeds [Tong et al., 2024, Rahmanzadehgervi et al., 2024, Kang et al., 2025, Yuan et al., 2025, Yang et al., 2025a], but attention magnitude does not establish necessity. Low attention can mean that continued access is no longer used, or that access is failing; the two are indistinguishable from attention values alone. We therefore test necessity by intervention rather than by reading off attention. To answer this question, we introduce the Visual Access Boundary (VAB). We intervene by masking direct attention from generated-token queries to image-token keys across layer depth and generation time (Figure 1(a)). We call this two-axis intervention Visual Access Sweep. This intervention turns an otherwise observational question into a causal one. If performance is preserved after removing later generated-token access to image tokens, then those direct visual accesses are not necessary for the target behavior. Instead, when CoT helps, it appears to operate over image-derived information already carried in hidden states. To isolate visual grounding from downstream reasoning, we design controlled attribute-counting tasks in which object attributes and spatial layouts are fully parameterized, following the spirit of controlled visual reasoning benchmarks. This avoids cases where a model can answer from parametric knowledge or dataset bias rather than from the image itself, while keeping the reasoning operation fixed as counting and varying only the queried visual attribute. We compare no-CoT direct answering, which we refer to as Direct, with CoT prompting. We further test whether the same structure appears beyond synthetic images using a GQA-derived real-image yes/no task [Hudson and Manning, 2019]. Applied across six model configurations from Qwen2.5-VL and InternVL3, Visual Access Sweep yields two main findings. First, as schematized in Figure 1(b), a Visual Access Boundary exists clearly under both Direct and CoT prompting. In every main model-task setting we test, accuracy is preserved while a broad upper-layer region is blocked, then drops sharply once the intervention reaches earlier layers. Taken together, these results adjudicate between the two mechanisms above. In Direct answering, there is a depth after which generated tokens no longer need direct attention to image tokens. Under CoT, generation becomes much longer, but the necessary visual-access region does not expand proportionally along the generation-time axis. Thus, in the regimes we test, CoT does not primarily improve performance by continuing to re-read the image throughout the reasoning trace. 2

Instead, when CoT helps, it appears to operate over image-derived information already carried in hidden states. The same qualitative VAB structure also appears on the GQA-derived real-image task. Second, on Qwen2.5-VL across three scales, per-attribute CoT gains are predicted by multi-object perceptual readout accuracy at 3B and 7B and saturate at 32B. An oracle bypass that supplies the attribute symbolically restores CoT gains uniformly across attributes, and a probe-vs-decode gap on hard attributes provides separate single-object evidence that the bottleneck sits at perceptual readout rather than the counting operation itself (Section 6). Our contributions are as follows. (i) We introduce Visual Access Sweep and the Visual Access Boundary, a causal intervention and an operational measure that together map the functional necessity of direct image-token access along both layer depth and generation time. (ii) We show that finite VABs appear under both Direct and CoT prompting across the main Qwen2.5-VL and InternVL3 settings. In larger models, the CoT VAB layer remains within two layers of the no-CoT boundary when evaluated against the no-CoT full-access target, despite much longer generations. The same qualitative VAB structure also appears on a GQA-derived real-image yes/no task. (iii) On Qwen2.5VL, per-attribute CoT gain tracks multi-object perceptual readout accuracy across three scales and is restored uniformly by an oracle bypass on the same checkpoint without image input. A separate single-object probe-vs-decode diagnostic on hard attributes provides supporting evidence that the bottleneck sits at perceptual readout rather than the counting operation itself.

2

Related Work

Visual attention decay in VLMs. A growing body of work shows that VLM attention to image tokens often weakens, becomes noisy, or is misallocated during generation, and several methods improve performance by strengthening, redistributing, or re-injecting visual signals during inference [Tong et al., 2024, Rahmanzadehgervi et al., 2024, Kang et al., 2025, Yuan et al., 2025, Yang et al., 2025a]. These studies establish that modifying visual attention can be beneficial. Our question is complementary. In an unmodified prefix-based decoder-only VLM, when is continued direct imagetoken access functionally necessary during generation? Rather than enhancing visual attention, Visual Access Sweep removes generated-token access to image tokens along layer depth and generation time, thereby testing necessity rather than utility of attention amplification. CoT and test-time scaling in VLMs. CoT and test-time scaling are well established in LLMs [Wei et al., 2022, Kojima et al., 2022, Wang et al., 2023, Liu et al., 2025a] and have been extended to VLMs, with some studies reporting improved multimodal reasoning [Zhang et al., 2024, Xu et al., 2025, Lin et al., 2025, Wu et al., 2025] and others showing limited or degraded gains in perception-heavy settings [Lu et al., 2024, Liu et al., 2024b]. Rather than searching for the best prompting recipe, we ask what is being extended when CoT does help. Because generated rationales need not faithfully reveal the computation that produced the answer [Turpin et al., 2023, Lanham et al., 2023], we do not infer visual use from the CoT text itself; we instead test the functional necessity of image-token access by intervention. Active perception and visual re-access. Several recent approaches modify VLM inference by allowing the model to acquire additional visual evidence during reasoning, for example through visual memory, region refocusing, or explicit perception actions [Zou et al., 2025, Yang et al., 2025b, Yu et al., 2025]. These methods study how to improve models by adding a perception-reasoning feedback loop. Our work asks a complementary diagnostic question. We test whether ordinary CoT already makes continued direct access from generated tokens to the initial image-token prefix functionally necessary in prefix-based decoder-only VLMs. Mechanistic interpretability and probing in VLMs. Causal-intervention methods on attention or activations [Vig et al., 2020, Geiger et al., 2021, Meng et al., 2022, Schwettmann et al., 2023, Gandelsman et al., 2024] and probing studies showing VLM hidden states encode visual information the model fails to verbalize [Alain and Bengio, 2016, Belrose et al., 2023, Geva et al., 2023, Fu et al., 2025, Liu et al., 2025b] both inform our approach. Unlike tuned-lens-style observational decoding, the VAB measures via causal masking where direct image-token access stops being necessary, and we go beyond generic readout-bottleneck observations by relating the probe-vs-decode gap to CoT gain through oracle bypass. 3

3

Experimental Design

3.1

VLM formulation

We focus on decoder-only VLMs in which a frozen vision encoder followed by a projection layer produces image tokens that are concatenated as a prefix to text tokens, and outputs are generated autoregressively [Vaswani et al., 2017, Liu et al., 2023, 2024a, Bai et al., 2024, 2025]. Throughout this paper, when we refer to attention from generated-token queries to image-token keys, we mean the corresponding entries of the standard self-attention matrix in this prefix-based decoder-only setup, rather than a separate cross-attention module. Let timg = (timg,1 , . . . , timg,K ) denote the K image tokens, ttext the input text tokens, and tgen,1:t−1 the previously generated tokens; at step t the model conditions on Tt = [timg , ttext , tgen,1:t−1 ] and predicts tgen,t . The operational definition of visual access that our intervention manipulates is given in Section 4.1.

3.2

Tasks and datasets

We use two task families, a controlled synthetic attribute-counting task family that allows precise factor isolation and a real-image scene-graph QA task derived from GQA that probes external validity beyond synthetic stimuli.

Controlled attribute-counting tasks. The controlled tasks vary the queried visual attribute while holding the reasoning operation fixed (counting). Each scene is procedurally generated with parameterized objects whose attributes (color, shape, location, angle, size, with 4 classes each) span a range of perceptual difficulty (Appendix A.1). For each scene we issue five counting queries of the form Figure 2: Controlled attribute-counting task. “How many ATTRIBUTE = VALUE objects are in the The reasoning operation (COUNT) is fixed image?” (Figure 2), one per attribute. while the queried attribute varies across {color, The role of these tasks is diagnostic, not benchmark shape, location, angle, size}, isolating the efcoverage. Holding the reasoning operation fixed fect of perceptual difficulty on a single, fixed across attributes makes it harder to attribute differ- reasoning program. ences in CoT gain to differences in reasoning difficulty. What varies is the difficulty of obtaining the relevant attribute in a form usable by counting. Section 6 uses two diagnostics on the same attributes, namely multi-object readout on the counting scenes and a single-object probe-vs-decode check in Appendix D.1. Both test whether the queried attribute is available as a usable symbol. A concrete counting example is in Appendix A.2.

Real-image scene-graph QA. To test whether the VAB structure is specific to synthetic stimuli, we additionally construct a real-image extension from the balanced validation split of GQA [Hudson and Manning, 2019].1 GQA contains natural photographs paired with scene-graph-derived questions, and its balanced split is designed to reduce answer-distribution bias. We use the yes/no subset to keep the answer space controlled and to reduce answer-parsing ambiguity. Full preprocessing, prompting, answer extraction, and license details are in Appendix C.6. For both task families, we compare Direct prompting, which asks for a final answer with minimal intermediate output, against CoT prompting, which asks the model to produce intermediate reasoning before the final answer. Controlled counting is evaluated by constrained option-key accuracy, and the GQA-derived task by binary yes/no accuracy. Full prompt templates, answer extraction rules, decoding settings, and compute details are provided in Appendix A.3 and Appendix A.4.

1 https://huggingface.co/datasets/vikhyatk/gqa

4

4

Visual Access Sweep

4.1

Intervention

We operationally define visual access at (ℓ, t) as direct attention from the generated-token query at generation step t and layer ℓ to image-token keys (Figure 1(a)). Visual Access Sweep restricts this channel by two cutoffs. Image-token access is allowed only inside the rectangle ℓ ≤ ℓcutoff and t ≤ tcutoff , and is masked outside this region. Sweeping (ℓcutoff , tcutoff ) thus traces how much direct image-token access is needed to preserve task performance. Throughout this paper, visual access denotes this specific computational channel. Visual information that has already been incorporated into hidden states through earlier computation can still persist through the residual stream and is not removed by this intervention. Concretely, for the additive attention mask Mℓ at layer ℓ, we set Mℓij = −∞ for every (generated-query position i, image-token key position j) pair within the masked region, so that the corresponding post-softmax attention weight is exactly zero: !  ⊤ Qℓ Kℓ −∞ if intervention active, ℓ ℓ ℓ ℓ ℓ ℓ √ Attention(Q , K , V ) = softmax + M V , Mij = 0 otherwise. d (1) The layer cutoff ℓcutoff keeps direct image-token access in layers 1, . . . , ℓcutoff and removes it in all subsequent layers. The token cutoff tcutoff keeps access for the first tcutoff generated tokens and removes it thereafter. Because direct prompting and CoT prompting produce sequences of very different lengths, we use the effective window teff = min(tcutoff , Tgen ) and treat samples with Tgen < tcutoff as having full token-axis access. The full-access baseline corresponds to the sweep point with the largest layer and token cutoffs. To verify that the observed boundary is not an artifact of the masking operation, Appendix B.1 reports null-sink and late-layer query-text-block controls. These controls test for attention redistribution and query-mediated re-reading, respectively, and both preserve the qualitative VAB structure on Qwen2.5-VL-32B. 4.2

Visual Access Boundary

Figure 1(b) illustrates how the sweep is read. If accuracy remains within tolerance after masking a broad late-layer or late-token region, then continued direct image-token access in that region is not functionally necessary for the task. For compact notation, we write τ for the effective token window teff . Let Apfull denote the full-access accuracy under prompt condition p ∈ {D, CoT}, and let Ap (ℓ, τ ) denote the accuracy after applying the visual-access cutoff (ℓ, τ ) under the same prompt condition. For a tolerance ϵ, each boundary is defined as the minimal access region on the sweep grid whose criterion-specific accuracy drop is at most ϵ. We use ϵ = 0.05 throughout, corresponding to a 5-percentage-point accuracy budget. The three drops we compare against ϵ are D ∆D (ℓ, τ ) = AD full − A (ℓ, τ ), CoT ∆CoT (ℓ, τ ) = ACoT (ℓ, τ ), full − A

(2)

CoT ∆DA (ℓ, τ ) = AD (ℓ, τ ). full − A

The Direct boundary ℓ∗D is computed from ∆D . The CoT-own-max boundary ℓ∗CoT is computed from ∆CoT , asking when CoT under intervention preserves CoT’s own full-access ceiling. The direct-anchored CoT boundary ℓ∗DA is computed from ∆DA , asking when CoT under intervention reaches the Direct full-access target. We report two layer shifts, ∆ℓ∗own = ℓ∗CoT − ℓ∗D ,

∆ℓ∗DA = ℓ∗DA − ℓ∗D .

(3)

Per-setting numbers are summarized in Section 5.2 and reported in Appendix C.1. We verify in Appendix B.2 that the central CoT-vs-Direct comparison is stable under ϵ ∈ {0.03, 0.05, 0.07, 0.10} (|∆ℓ∗DA | ≤ 1 under the direct-anchored criterion at all ϵ tested). By construction, the VAB measures the functional necessity of continued direct image-token access. It does not erase image-derived information that has already propagated into hidden states. 5

(a) Direct: layer sweep for color and shape counting.

(b) CoT: layer-token sweep. Stars mark the empirical (ℓ∗ , τ ∗ ).

Figure 3: Visual Access Boundary on Qwen2.5-VL-32B. (a) Direct inference shows a sharp layerwise transition: accuracy is preserved while a broad upper-layer region is blocked, but drops once the intervention reaches earlier layers. (b) Under CoT, the high-accuracy region remains concentrated in a similar depth range despite much longer generations. Stars mark the empirical VAB under ϵ = 0.05. Exact boundary values are reported in Appendix C.1.

5

Visual Access Boundary in Direct and CoT Inference

Across model families, scales, and stimulus domains, Visual Access Sweep reveals a consistent structure. Performance is preserved after masking a broad late region of direct image-token access, and drops sharply only when the intervention reaches earlier layers or earlier generated tokens. This structure appears in Direct inference, persists under CoT prompting, and also appears on a GQA-derived real-image QA task. The main empirical result is therefore not a particular numerical layer shift, but the repeated emergence of a finite Visual Access Boundary. Extended generation does not imply continued direct image-token access throughout the reasoning trace. 5.1

Visual Access Boundary in direct inference

We first establish that direct inference exhibits a clear functional boundary on continued image-token access. Figure 3(a) shows the Direct layer sweep on Qwen2.5-VL-32B for color and shape counting. Accuracy remains near the full-access baseline while a broad upper-layer region is blocked, then drops sharply once the intervention reaches earlier layers. Exact ℓ∗ values are reported in the figure and in Appendix C.1. We interpret this boundary operationally. Beyond ℓ∗ , additional direct attention from generated tokens to image tokens is no longer needed for this task, although visual information already carried in hidden states may still be used. 5.2

Visual Access Boundary in CoT inference

We now ask whether the much longer generation produced by CoT requires a correspondingly larger region of direct image-token access. Figure 3(b) shows the joint sweep over layer and effective token cutoffs under CoT on Qwen2.5-VL-32B. The region where ∆CoT (ℓ, τ ) ≤ ϵ = 0.05 occupies the upper portion of the heatmap, and the star marks the empirical (ℓ∗ , τ ∗ ). We use the CoT-own-max and direct-anchored criteria defined in Section 4.2. The structural observation is that, across the 12 (family, scale, task) settings we test, |∆ℓ∗own | stays within 9 layers while generation length increases by approximately 50×. The direct-anchored CoT boundary is generally close to the Direct VAB on the larger models for which the target is attainable (Qwen2.5-VL-32B and the 14B and 38B variants of InternVL3 yield |∆ℓ∗DA | ≤ 2). Settings where the direct-anchored target is not reached are reported separately. The consistent pattern is the existence of a finite VAB under CoT, not a monotonic deepening of the boundary. We discuss possible mechanisms for this decoupling in Section 7. 5.3

Cross-family and cross-scale VAB sweeps

A natural concern is that the VAB structure could be specific to a single family, since differences in vision encoder, projection, and training data could in principle create an architecture-specific late-layer redundancy. We evaluate the same Visual Access Sweep on Qwen2.5-VL [Bai et al., 2025] at three scales (3B, 7B, 32B) and on InternVL3 [Zhu et al., 2025] at three scales (8B, 14B, 38B). A 6

(a) Direct: layer sweep on GQA yes/no.

(b) CoT: layer-token sweep. Star marks (ℓ∗ , τ ∗ ).

Figure 4: Real-image extension: VAB on GQA yes/no (Qwen2.5-VL-32B). (a) Direct inference reproduces the qualitative pattern of the controlled tasks. A broad upper-layer region can be blocked with little accuracy loss before a sharp transition. (b) Under CoT, the high-accuracy region is concentrated within a similar depth range despite much longer generations. Both prompting modes therefore exhibit a finite VAB on real images. Exact boundary values are reported in Appendix C.6. layer-wise VAB is observed in every case (per-model figures in Appendix C.4, architectural details in Appendix C.2). The shift is not consistently positive. Qwen2.5-VL scales tend to show non-negative CoT-own-max shifts, whereas InternVL3 does not show a systematic deepening. In some larger InternVL3 settings, the CoT boundary is comparable to or slightly shallower than the Direct boundary. What is constant across architectures is therefore the absence of generation-proportional expansion, not a single-direction shift. Cross-attention fusion architectures. We further apply the same intervention principle to a crossattention-based VLM (Llama-3.2-11B-Vision-Instruct [Grattafiori et al., 2024]), where visual access is implemented by masking the generated-token queries’ cross-attention to visual keys rather than self-attention to an image-token prefix (Appendix C.5). The CoT-side sweep again exhibits a finite VAB. This pilot check suggests that the intervention principle can be applied beyond prefix-fusion models, with explicit iterative re-perception modules remaining the main untested regime.

5.4

Real-image scene-graph QA extension

A second concern is that the VAB structure might be a property of the synthetic CLEVR-style stimuli [Johnson et al., 2017] rather than of decoder-only VLMs. We apply Visual Access Sweep to the GQA-derived real-image scene-graph QA task on Qwen2.5-VL-32B (Section 3.2). At full access, Direct accuracy is 0.84 and CoT accuracy is 0.80 (Appendix C.6). Figure 4 shows the joint sweep for both prompting modes. Although CoT does not improve this binary GQA setting, this aligns with recent evidence that longer reasoning in VLMs can weaken perceptual grounding rather than monotonically improve visual QA [Tian et al., 2026, Kancheti et al., 2026]. Both Direct and CoT sweeps still exhibit a finite VAB, with the redundant late-layer region comparable in size to synthetic counting on the same model. The GQA result thus extends the VAB phenomenon to real images while separating the existence of a visual-access boundary from whether CoT improves final accuracy.

6

Perceptual Readout Restricts CoT Gains

The Visual Access Sweep results say that, in the regime we study, CoT gains do not come from extending direct image-token access. They do not yet say what determines whether CoT gains exist at all on a given task. In this section, we show that within Qwen2.5-VL, the magnitude of CoT gains is constrained by whether the queried visual attribute can be read out into a form usable by the model’s own reasoning chain. 7

Figure 5: Perceptual readout predicts CoT gain across Qwen2.5-VL scales. Each point is one queried attribute (color, shape, location, size, angle). The x-axis is multi-object perceptual readout accuracy: the model is asked to fill in the queried attribute for every object in a multi-object scene, and accuracy is computed on the queried attribute values. The y-axis is the CoT gain on the corresponding counting task. CoT gains are largest when the queried attribute can be reliably read out. Per-attribute values and per-scale correlations are in Appendix D.2.

6.1

Defining perceptual readout

We distinguish two questions. The first is whether an attribute is encoded somewhere in the model’s hidden states. The second is whether the model can turn that attribute into a symbol that its own answer-generation process can use. This section focuses on the second question. We use two related readout diagnostics. The main predictor of CoT gain is a multi-object perceptual readout task, in which the model is given a scene with multiple objects and must fill in the queried attribute value for every object, while the remaining attributes are provided as textual scaffolding. We use this multi-object perceptual readout accuracy as the x-axis in Figure 5. Separately, Appendix D.1 reports a single-object probe-vs-decode diagnostic, which tests whether attribute information is linearly decodable from hidden states even when the model cannot reliably express it in its own output. Neither is a parallel main task. Both are evaluation devices for whether the queried visual attribute is available as a usable symbol. This distinction is necessary because latent encoding and decoded availability can diverge. On Qwen2.5-VL-32B, hard attributes such as angle, location, and size show large probe-vs-decode gaps. A linear probe on hidden states recovers them with high accuracy while the model’s own direct decoding is substantially lower (Appendix D.1, Table 12). The bottleneck is therefore not whether visual information exists internally, but whether the model can read it out into a usable symbolic form.

6.2

Perceptual readout and CoT gain correlation

We hold the reasoning operation fixed (counting) and vary only the queried attribute, then plot per-attribute CoT gain (AccCoT − AccDirect ) against multi-object perceptual readout accuracy on Qwen2.5-VL at three scales (3B, 7B, 32B) in Figure 5. The relationship is positive at every scale. Attributes with relatively higher readout accuracy within each scale tend to show larger CoT gains, whereas attributes with weaker readout (angle and size at every scale) show essentially no CoT gain or slightly negative gain. At 32B, three of five attributes saturate above readout accuracy 0.77, compressing residual variance. We therefore read ρ as the more stable summary at high scale and the linear coefficient as a within-readout-range slope. The angle limiting case (chance-level readout, VAB ill-defined) is in Appendix C.7. Because the reasoning operation is held fixed across attributes, the variation in CoT gain is difficult to explain by reasoning difficulty alone. CoT improves counting only after the queried visual attribute is available as a usable symbol, complementing Section 5 by identifying what limits CoT once extended visual access is not the source of improvement. 8

6.3

Symbolic-attribute oracle bypass

The correlation in Figure 5 is observational. Hard attributes may show small CoT gains either because the model cannot read out the relevant visual attribute, or because the corresponding counting question is intrinsically difficult to solve. To separate these possibilities, we run an oracle-bypass experiment. In this setting, we use the same Qwen2.5-VL-3B checkpoint without image tokens. No image is provided, the visual-token pathway is bypassed, and ground-truth object attributes are serialized directly into the prompt. The model is then asked the same multiple-choice counting questions under Direct and CoT prompting. The experiment therefore does not test visual perception. It tests whether the language-side counting operation can benefit from CoT once the relevant attributes are already available as symbols. We consider two oracle inputs. The minimal oracle provides only the queried attribute for each object (e.g., color values when the question asks about color counting). This is the cleanest test because it removes the visual readout step while preserving the counting operation. The full oracle provides all object attributes, adding irrelevant information and serving as a robustness condition rather than the primary evidence. Figure 6(a) shows that under the minimal oracle, CoT gains become consistent across attributes, including angle, size, and location, which showed weak gains under image input (Figure 5). The counting operation is therefore not the limiting factor once the relevant attributes are supplied as symbols. The limiting step is mak- Figure 6: Oracle bypass with the Qwen2.5-VL-3B ing the queried visual attribute available checkpoint without image input. Ground-truth atin a usable symbolic form. Figure 6(b) re- tributes are serialized as text. (a) Minimal oracle proports the full-oracle condition, where the vides only the queried attribute for each object. (b) Full larger prompt changes the amount of irrel- oracle provides all object attributes. evant information and the headroom for CoT gains. We treat it as a supplementary check. The probe-decode gap and the oracle bypass together separate three quantities that are easy to conflate. Linear probing asks whether the attribute is present in hidden states (Section 6.1). Direct decoding asks whether the model can read out the attribute as a usable symbol (Section 6.2). Oracle bypass asks whether the counting operation can benefit from CoT once that symbol is supplied (this section). The results point to the middle step as the bottleneck. CoT can improve reasoning over available symbols, but it does not reliably recover a visual attribute that the model cannot itself read out.

7

Discussion

Interpreting the VAB. This work provides a causal test of what is extended when VLMs generate longer CoT traces. Across the regimes tested here, Visual Access Sweep reveals a finite Visual Access Boundary under both Direct and CoT prompting. Extended generation does not require continued direct image-token access throughout the reasoning trace. Operationally, the VAB marks when additional direct attention from generated-token queries to image-token keys becomes functionally unnecessary. It does not imply that visual evidence is absent from the model. Such evidence may already have been transformed into hidden-state representations and carried forward through the residual stream. This reframes visual-attention decay and visual forgetting. The key question is not whether attention to image tokens decreases, but whether later direct access is still necessary for the task. Implications for CoT and active perception. The perceptual-readout results suggest that CoT helps when the needed visual attribute has first been made available in a form the model’s reasoning trace can use. When readout succeeds, longer reasoning can improve the downstream counting operation. When readout fails, CoT does not reliably recover the missing symbol, even if latent visual information may be present. This gives a practical rule for test-time scaling: before spending tokens on longer reasoning, test whether the relevant visual attribute can be read out reliably. Within Qwen2.5-VL, a cheap multi-object readout probe supports such routing, preserving most of the 9

always-CoT accuracy at substantially lower token cost (Appendix E). The same result clarifies the role of active perception methods such as MemVR [Zou et al., 2025], Look-Back [Yang et al., 2025b], and Visual Perception Token [Yu et al., 2025]. These methods are complementary: ordinary CoT in the tested regime does not by itself close the perception-reasoning loop, whereas active perception methods explicitly try to acquire new visual evidence conditioned on the reasoning state. Scope and limitations. The evidence in this paper covers controlled attribute-counting tasks across families and scales, a GQA-derived real-image setting, and an additional cross-attention fusion setting (Appendix C.5). Tasks deliberately designed to require iterative visual inspection remain outside the current evaluation. The sweep result also does not by itself identify the internal mechanism that produces the boundary. The same VAB pattern could arise because visual attributes are bound into hidden-state representations early, because late layers receive sufficient image-derived information through the residual stream, or because CoT prompting front-loads visual extraction into early generated tokens. Visual Access Sweep isolates the necessity of one channel (direct generated-tokento-image-token attention), so distinguishing these accounts will require complementary interventions on the residual stream and on the temporal distribution of perceptual extraction.

Acknowledgments and Disclosure of Funding We thank our colleagues for their valuable discussions and feedback throughout the development of this work.

10

References G. Alain and Y. Bengio. Understanding intermediate layers using linear classifier probes, 2016. URL https://arxiv.org/abs/1610.01644. J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou. Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond, 2024. URL https://openreview.net/forum?id=qrGjFJVl3m. S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin. Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923, 2025. doi: 10.48550/arXiv.2502.13923. URL https://arxiv.org/abs/ 2502.13923. N. Belrose, Z. Furman, L. Smith, D. Halawi, I. Ostrovsky, L. McKinney, S. Biderman, and J. Steinhardt. Eliciting latent predictions from transformers with the tuned lens, 2023. URL https://arxiv.org/abs/2303.08112. S. Fu, T. Bonnen, D. Guillory, and T. Darrell. Hidden in plain sight: VLMs overlook their visual representations, 2025. URL https://arxiv.org/abs/2506.08008. Y. Gandelsman, A. A. Efros, and J. Steinhardt. Interpreting CLIP’s image representation via text-based decomposition. In International Conference on Learning Representations (ICLR), 2024. A. Geiger, H. Lu, T. Icard, and C. Potts. Causal abstractions of neural networks. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pages 9574–9586, 2021. M. Geva, J. Bastings, K. Filippova, and A. Globerson. Dissecting recall of factual associations in auto-regressive language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. URL https://arxiv.org/abs/2407.21783. D. A. Hudson and C. D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. J. Johnson, B. Hariharan, L. van der Maaten, J. Hoffman, L. Fei-Fei, C. L. Zitnick, and R. Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. URL https://arxiv.org/abs/1612.06890. S. S. Kancheti, A. S. Kanade, V. N. Balasubramanian, and T. Ganu. Chain-of-thought degrades visual spatial reasoning capabilities of multimodal LLMs. arXiv preprint arXiv:2604.16060, 2026. URL https://arxiv.org/abs/2604.16060. S. Kang, J. Kim, J. Kim, and S. J. Hwang. See what you are told: Visual attention sink in large multimodal models. In International Conference on Learning Representations (ICLR), 2025. URL https://openreview.net/forum?id=7uDI7w5RQA. T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213, 2022. T. Lanham, A. Chen, A. Radhakrishnan, B. Steiner, C. Denison, D. Hernandez, D. Li, E. Durmus, E. Hubinger, J. Kernion, K. Lukošiūtė, K. Nguyen, N. Cheng, N. Joseph, N. Schiefer, O. Rausch, R. Larson, S. McCandlish, S. Kundu, S. Kadavath, S. Yang, T. Henighan, T. Maxwell, T. TelleenLawton, T. Hume, Z. Hatfield-Dodds, J. Kaplan, J. Brauner, S. R. Bowman, and E. Perez. Measuring faithfulness in chain-of-thought reasoning, 2023. URL https://arxiv.org/abs/2307.13702. Y. Lin, A. Wang, M. Chen, J. Liu, H. Liu, J. Su, and X. Xiao. Investigating inference-time scaling for chain of multi-modal thought: A preliminary study, 2025. URL https://arxiv.org/abs/ 2502.11514. 11

H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2023. H. Liu, C. Li, Y. Li, and Y. J. Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024a. R. Liu, S. Bai, N. Saunshi, and S. Kakade. Chain-of-thought can reduce performance on tasks where thinking makes humans worse. In International Conference on Learning Representations (ICLR), 2024b. URL https://openreview.net/forum?id=rpbzBXdo4x. Y. Liu, Z. Li, Z. Fang, N. Xu, R. He, and T. Tan. Rethinking the role of prompting strategies in LLM test-time scaling: A perspective of probability theory. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), 2025a. doi: 10.18653/v1/2025.acl-long.1356. URL https://arxiv.org/abs/2505.10981. Z. Liu, Z. Chen, H. Liu, C. Luo, X. Tang, S. Wang, J. Zeng, Z. Dai, Z. Shi, T. Wei, B. Dumoulin, and H. Tong. Seeing but not believing: Probing the disconnect between visual attention and answer correctness in VLMs, 2025b. URL https://arxiv.org/abs/2510.17771. P. Lu, H. Banber, L. Xia, J. Zhu, C. Li, H. Hajishirzi, H. Wang, Y. Choi, and F. Poursabzi-Sangdeh. MathVista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICLR), 2024. K. Meng, D. Bau, A. Andonian, and Y. Belinkov. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems, 2022. URL https://arxiv.org/abs/ 2202.05262. NeurIPS 2022. P. Rahmanzadehgervi, L. Bolton, M. R. Taesiri, and A. T. Nguyen. Vision language models are blind. In Proceedings of the Asian Conference on Computer Vision (ACCV), 2024. S. Schwettmann, N. Chowdhury, and A. Torralba. Multimodal neurons in pretrained text-only transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2859–2869, 2023. X. Tian, S. Zou, Z. Yang, M. He, F. Waschkowski, L. Wesemann, P. Tu, and J. Zhang. More thought, less accuracy? on the dual nature of reasoning in vision-language models. In International Conference on Learning Representations (ICLR), 2026. URL https://arxiv.org/abs/2509. 25848. S. Tong, Z. Liu, Y. Zhai, Y. Ma, Y. LeCun, and S. Xie. Eyes wide shut? exploring the visual shortcomings of multimodal LLMs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9568–9578, 2024. M. Turpin, J. Michael, E. Perez, and S. R. Bowman. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting, 2023. URL https://arxiv.org/ abs/2305.04388. NeurIPS 2023. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_ files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf. J. Vig, S. Gehrmann, Y. Belinkov, S. Qian, D. Nevo, Y. Singer, and S. Shieber. Investigating gender bias in language models using causal mediation analysis. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 12388–12401, 2020. X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, and D. Zhou. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations (ICLR), 2023. J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. Le, and D. Zhou. Chain-ofthought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2022. 12

Q. Wu, X. Yang, Y. Zhou, C. Fang, B. Song, X. Sun, and R. Ji. Grounded chain-of-thought for multimodal large language models, 2025. URL https://arxiv.org/abs/2503.12799. G. Xu, P. Jin, Z. Wu, H. Li, Y. Song, L. Sun, and L. Yuan. LLaVA-CoT: Let vision language models reason step-by-step. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2087–2098, 2025. J. Yang, C. Cui, Y. Zhou, Y. Chen, P. Xia, Y. Wei, T. Yu, Y. Huang, and L. Wang. IKOD: Mitigating visual attention degradation in large vision-language models, 2025a. URL https://arxiv.org/ abs/2508.03469. S. Yang, Y. Niu, Y. Liu, Y. Ye, B. Lin, and L. Yuan. Look-back: Implicit visual re-focusing in MLLM reasoning, 2025b. URL https://arxiv.org/abs/2507.03019. R. Yu, X. Ma, and X. Wang. Introducing visual perception token into multimodal large language model, 2025. URL https://arxiv.org/abs/2502.17425. Q. Yuan, Q. Zhang, Y. Liu, J. Chen, Y. Lu, H. Lin, J. Zheng, X. Han, and L. Sun. ShortV: Efficient multimodal large language models by freezing visual tokens in ineffective layers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. URL https://arxiv.org/abs/2504.00502. Z. Zhang, A. Zhang, M. Li, H. Zhao, G. Karypis, and A. Smola. Multimodal chain-of-thought reasoning in language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. J. Zhu, W. Wang, Z. Chen, Z. Liu, S. Ye, L. Gu, H. Tian, Y. Duan, W. Su, J. Shao, Z. Gao, E. Cui, X. Wang, Y. Cao, Y. Liu, X. Wei, H. Zhang, H. Wang, W. Xu, H. Li, J. Wang, N. Deng, S. Li, Y. He, T. Jiang, J. Luo, Y. Wang, C. He, B. Shi, X. Zhang, W. Shao, J. He, Y. Xiong, W. Qu, P. Sun, P. Jiao, H. Lv, L. Wu, K. Zhang, H. Deng, J. Ge, K. Chen, L. Wang, M. Dou, L. Lu, X. Zhu, T. Lu, D. Lin, Y. Qiao, J. Dai, and W. Wang. InternVL3: Exploring advanced training and test-time recipes for open-source multimodal models, 2025. URL https://arxiv.org/abs/2504.10479. X. Zou, Y. Wang, Y. Yan, Y. Lyu, K. Zheng, S. Huang, J. Chen, P. Jiang, J. Liu, C. Tang, and X. Hu. Look twice before you answer: Memory-space visual retracing for hallucination mitigation in multimodal large language models, 2025. URL https://arxiv.org/abs/2410.03577.

13

Appendix Contents A Experimental Details

15

A.1 Dataset Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

15

A.2 Task Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

15

A.3 Prompting, decoding, and answer extraction . . . . . . . . . . . . . . . . . . . . .

15

A.4 Compute Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

16

B Visual Access Sweep: Methodology Details

19

B.1 Control Experiments: Ruling Out Intervention Artifacts . . . . . . . . . . . . . . .

19

B.2 VAB Sensitivity to the Tolerance Parameter ϵ . . . . . . . . . . . . . . . . . . . .

19

C Visual Access Boundary: Detailed Results

21

C.1 Boundary Summary Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

21

C.2 Model Architectures Compared . . . . . . . . . . . . . . . . . . . . . . . . . . . .

21

C.3 Qwen2.5-VL Family Sweeps . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

23

C.4 InternVL3 Family Sweeps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

24

C.5 Cross-Attention Fusion Pilot . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

25

C.6 GQA Real-Image Extension . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

26

C.7 Limiting Case: Angle Counting

. . . . . . . . . . . . . . . . . . . . . . . . . . .

26

C.8 Qualitative Error Breakdown . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

26

D Perceptual Readout: Detailed Results

28

D.1 Probe-vs-Decode Gap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

28

D.2 Per-Attribute CoT Gain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

28

E Practical Implication: Selective CoT Routing

14

30

A

Experimental Details

A.1

Dataset Generation

We construct controlled synthetic tasks in which the queried visual attribute varies while the counting operation is held fixed. Images are 256×256 px RGB generated with matplotlib under a fixed random seed. Tables 1 and 2 summarize the controlled attributes and dataset sizes. Table 1: Controlled visual attributes. Each attribute has four classes. Attribute

Classes

Color Shape Size Location Angle

red / green / yellow / blue star / circle / triangle / square xs (18 px) / s (24 px) / l (30 px) / xl (36 px) four image quadrants 0◦ / 45◦ / 90◦ / 135◦

Table 2: Dataset sizes for the two task families used in the controlled experiments. Task Single-object readout Counting

A.2

Images

Objects

Queries

750 150

1 6–7

one attribute query per image five queries per image

Task Examples

Figure 7 shows one multi-object counting instance with its per-object attributes and the five corresponding queries. ID

Color

Shape

Size

Location

Angle

0 1 2 3 4 5 6

red yellow yellow yellow red green yellow

triangle square square circle square square square

s s s xs xs s s

lower_right upper_right lower_left upper_left lower_right upper_right upper_left

45◦ 90◦ 0◦ 45◦ 45◦ 90◦ 135◦

Attribute

Question target

Correct answer

Color Shape Angle Location Size

green objects square objects 135◦ orientation upper_right region s (small) objects

D: 1 C: 5 D: 1 C: 2 A: 5

Figure 7: Example controlled counting instance. Left: example image. Right: per-object attributes (top) and the five corresponding queries (bottom). The same image is paired with five queries, one per attribute, so the counting operation is fixed while the queried attribute changes. Option keys are randomly permuted per query. A.3

Prompting, decoding, and answer extraction

We use a unified prompt template across all models with greedy decoding on frozen public checkpoints. Table 3 summarizes the prompt instructions and answer-extraction rules across settings, and Table 4 lists the verbatim prompt strings. For Direct, CoT, and Oracle counting, option keys are randomly permuted per query and extracted from the model output (no constrained decoding). GQA dataset construction details are in Appendix C.6. 15

Table 3: Prompting and answer-extraction summary across settings. Setting

Prompt instruction

Answer extraction

Direct counting CoT counting Single-object readout Multi-object readout

answer with option key only think step by step, then answer in braces multiple-choice attribute question (Table 4) fill missing target attribute per object, JSON output (Table 4) short-answer format with curly-brace final line think step by step, finish with curly-brace final line serialized attributes as text (minimal or full)

option-key extraction (A–E) final braced option key (A–E) option-key extraction (A–E) per-object value parse from JSON

GQA Direct GQA CoT Oracle (counting)

A.4

last “yes” / “no” (YN _ LAST) last “yes” / “no” (YN _ LAST) option-key extraction (A–E)

Compute Resources

All experiments in this paper are inference-time interventions on frozen public VLM checkpoints. No VLM is fine-tuned. Only lightweight linear probes are trained for the probe-vs-decode analysis (Appendix D.1). Hardware and runtime are summarized in Table 5. Aggregate compute across the main results reported in this paper is on the order of ∼ 1,100 GPU-hours, and preliminary or discarded experiments (alternative sweep grids, additional probe configurations, control runs not reported) account for a comparable additional amount. Peak GPU memory is bounded by full-attention forward-pass memory at the largest scale (Qwen2.5VL-32B) with the image-token prefix, and fits within 3× H100 NVL 96 GB with standard tensorparallel sharding.

16

Table 4: Verbatim prompt strings used in our experiments. Mustache-style placeholders such as {attribute_value} and {question} are filled in per query. Image input is concatenated to the user message via the model’s default chat template. Setting

Prompt body

Counting / Direct

How many {attribute_value} objects are in the image? options: {A: 2, B: 4, C: 1, D: 3, E: 5} Answer with the option key only (no ’:’ or value). Respond with a single final line in the curly braces format:

Counting / CoT

How many {attribute_value} objects are in the image? options: {A: 2, B: 4, C: 1, D: 3, E: 5} Think step by step. You must write intermediate steps. Respond in curly braces format: {A}

Single-object readout

Which color is this object in the image? options: {A: blue, B: red, C: green, D: yellow} Answer with the option key only (no ’:’ or value). Respond with a single final line in the curly braces format:

{A}

{A}

Multi-object readout

In the image, there are {N} objects. Fill in the missing color for each object. Valid color values: red, green, yellow, blue Object 0: color=__, shape=triangle, size=l, location=upper_left, angle=135 Object 1: color=__, shape=circle, size=s, location=lower_right, angle=0 ... Output in following format: {"objects": [{"id": 0, "color": "red"}, {"id": 1, "color": "green"}, ...]}

GQA / Direct

{question} Answer with a short phrase only. Respond with a single final line in the curly braces format: Answer: {your short answer}

GQA / CoT

{question} Think step by step. You must write intermediate steps. Finish with a single final line in the curly braces format after your reasoning: Answer: {your short answer}

Oracle (minimal)

In the image, there are {N} objects. Object 0: color=blue Object 1: color=yellow ... How many {attribute_value} objects are there in the image? Answer with the option key only (no ’:’ or value). Respond with a single final line in the curly braces format:

Oracle (full)

{A}

In the image, there are {N} objects. Object 0: color=blue, shape=star, size=l, location=lower_right, angle=0 Object 1: color=yellow, shape=circle, size=s, location=upper_left, angle=135 ... How many {attribute_value} objects are there in the image? Answer with the option key only (no ’:’ or value). Respond with a single final line in the curly braces format: {A}

17

Table 5: Compute resources used in this paper. All sweeps were run on 3× NVIDIA H100 NVL 96 GB nodes; runtime is wall-clock × 3 GPUs. Experiment family

Hardware

Approx. runtime

Small-model sweeps

3× H100 NVL 96 GB

∼ 600 GPU-hours

3× H100 NVL 96 GB

∼ 500 GPU-hours

3× H100 NVL 96 GB 3× H100 NVL 96 GB

45–60 GPU-hours ∼ 21 GPU-hours

—

∼ 1,100 GPU-hours

(Qwen2.5-VL-3B/7B, InternVL3-8B, Llama-3.2-11B-Vision)

Large-model sweeps (Qwen2.5-VL-32B, InternVL3-14B/38B)

Qwen2.5-VL-32B 2D layer-×-token sweep Perceptual-readout analysis (probe, oracle, controls)

Total reported main results

18

B

Visual Access Sweep: Methodology Details

This section reports two intervention controls and an ϵ-sensitivity analysis for Visual Access Sweep. B.1

Control Experiments: Ruling Out Intervention Artifacts

We run two controls targeting possible artifacts of Visual Access Sweep. Late-layer query-text block. The first concern is a query-mediated re-reading loophole. Even if generated tokens cannot directly attend to image tokens in late layers, prompt or query-text tokens might still attend to the image in those layers and pass image-derived information to generated tokens. To test this, we apply a stricter intervention that additionally blocks query-text → imagetoken attention from the same cutoff layer upward. Importantly, early-layer query-text access is left unchanged. The control only removes the late-layer path that could bypass the generatedtoken intervention. Figure 8 shows that the qualitative VAB pattern is preserved under this stricter intervention.

(a) Direct layer sweep.

(b) CoT layer-token sweep.

Figure 8: Late-layer query-text block control (Color Counting, Qwen2.5-VL-32B). In addition to blocking generated-token → image-token attention, this control blocks query-text → imagetoken attention from the same cutoff layer upward. Early-layer query-text access is left intact. The qualitative VAB pattern is preserved, weakening the query-mediated late-rereading explanation. Null-sink control. The second concern is an attention-redistribution artifact. Setting image-token logits to −∞ changes the softmax normalization over the remaining real tokens, which could in principle distort the model’s behavior beyond simply removing image access. To test whether the observed boundary is caused by this redistribution rather than by removing image access, we redirect the removed image-token attention mass to a dummy non-visual sink slot. Figure 9 shows that the qualitative VAB pattern is again preserved. Takeaway. Together, these controls support the interpretation that the reported boundary is not primarily caused by either query-mediated late re-reading or attention-mass redistribution. They do not claim that all visual information flow stops after the boundary. Rather, they support the narrower claim used throughout the paper. Additional late direct image-token access is not functionally necessary in the tested setting. B.2

VAB Sensitivity to the Tolerance Parameter ϵ

The VAB definition (Section 4.2) depends on a tolerance ϵ, which we fix to ϵ = 0.05 throughout the main text. A natural concern is that the CoT-vs-Direct boundary comparison in Section 5.2 could rest on this specific choice of ϵ. To check this, we recompute ℓ∗ on Qwen2.5-VL-32B for color and shape counting under both criteria across ϵ ∈ {0.03, 0.05, 0.07, 0.10} (Table 6). Takeaway. Two points are stable across tolerance choices. First, under the direct-anchored criterion, the CoT boundary remains close to the Direct boundary across all tested ϵ values (|∆ℓ∗DA | ≤ 1), 19

(a) Direct layer sweep.

(b) CoT layer-token sweep.

Figure 9: Null-sink control (Color Counting, Qwen2.5-VL-32B). Removed image-token attention mass is redirected to a dummy non-visual sink instead of being redistributed over real non-image tokens. The qualitative VAB pattern is preserved, weakening the attention-redistribution artifact explanation. Table 6: VAB ϵ-sensitivity on Qwen2.5-VL-32B. ∆ℓ∗ across the two criteria and four tolerance values. “Direct-anchored” uses ∆ℓ∗DA = ℓ∗DA − ℓ∗D . “CoT-own-max” uses ∆ℓ∗own = ℓ∗CoT − ℓ∗D . Shaded columns (ϵ = 0.05) correspond to the tolerance used in the main text, with values shown in bold. Direct-anchored criterion

CoT-own-max criterion

ϵ=0.03

ϵ=0.05

ϵ=0.07

ϵ=0.10

ϵ=0.03

ϵ=0.05

ϵ=0.07

ϵ=0.10

color

ℓ∗D ℓ∗DA / ℓ∗CoT ∆ℓ∗DA ∆ℓ∗own

44 43 −1 —

43 43 0 —

43 42 −1 —

42 42 0 —

44 52 — +8

43 52 — +9

43 52 — +9

42 51 — +9

shape

ℓ∗D ℓ∗DA / ℓ∗CoT ∆ℓ∗DA ∆ℓ∗own

42 42 0 —

42 42 0 —

42 42 0 —

42 41 −1 —

42 44 — +2

42 43 — +1

42 43 — +1

42 42 — 0

Task

so the main CoT-Direct comparison is not an artifact of choosing ϵ = 0.05. Second, under the CoT-own-max criterion the boundary can be deeper because the target accuracy is higher. This reflects the stricter accuracy target rather than a proportional expansion of direct image-token access with generation length.

20

C

Visual Access Boundary: Detailed Results

This section reports per-family, per-scale, and per-task VAB results supporting Section 5: boundary summary tables, model architectures compared, family-level sweeps for Qwen2.5-VL and InternVL3, a cross-attention fusion pilot, the GQA real-image extension, an angle-attribute limiting case, and a qualitative error breakdown.

C.1

Boundary Summary Tables

Tables 7 and 8 report the per-setting VAB layers used in Section 5. The CoT-own-max criterion asks when CoT preserves its own full-access accuracy, whereas the direct-anchored criterion asks when CoT reaches Direct’s full-access accuracy. We denote the corresponding CoT-side boundaries by ℓ∗CoT and ℓ∗DA , respectively, and report shifts relative to the Direct boundary ℓ∗D following Eq. (3). We report both criteria because they answer different questions. The former measures the access region needed for CoT’s own ceiling, while the latter controls for Direct–CoT ceiling differences. N/A in Table 8 indicates that CoT never reaches the direct-anchored target in that setting, so the boundary is undefined under this criterion.

Table 7: CoT-own-max Visual Access Boundary across families and scales (ϵ = 0.05). Model

L

ℓ∗D

Color Counting ℓ∗CoT ∆ℓ∗own E[Tgen ]

ℓ∗D

Shape Counting ℓ∗CoT ∆ℓ∗own

Qwen2.5-VL-3B Qwen2.5-VL-7B Qwen2.5-VL-32B

36 28 64

25 20 43

34 21 52

+9 +1 +9

128 ± 32 85 ± 33 183 ± 33

22 20 42

29 19 44

+7 −1 +2

142 ± 29 76 ± 18 143 ± 30

InternVL3-8B InternVL3-14B InternVL3-38B

28 48 64

20 36 45

26 36 44

+6 0 −1

105 ± 20 75 ± 17 66 ± 11

20 35 46

23 37 47

+3 +2 +1

103 ± 27 86 ± 17 92 ± 21

E[Tgen ]

Table 8: Direct-anchored Visual Access Boundary across families and scales (ϵ = 0.05). N/A indicates that the direct-anchored target is not reached.

C.2

Model

L

Color Counting ℓ∗D ℓ∗DA ∆ℓ∗DA

Shape Counting ℓ∗D ℓ∗DA ∆ℓ∗DA

Qwen2.5-VL-3B Qwen2.5-VL-7B Qwen2.5-VL-32B

36 28 64

25 20 43

33 20 43

+8 0 0

22 20 42

N/A 5 42

N/A −15 0

InternVL3-8B InternVL3-14B InternVL3-38B

28 48 64

20 36 45

N/A 34 46

N/A −2 +1

20 35 46

N/A 37 47

N/A +2 +1

Model Architectures Compared

Table 9 summarizes the architectural differences across the VLMs analyzed in this paper. The cross-attention fusion VLM (Llama-3.2-11B-Vision-Instruct2 ) is reported in Appendix C.5.

2 https://huggingface.co/meta-llama/Llama-3.2-11B-Vision-Instruct

21

Table 9: Architectural summary of VLMs evaluated by Visual Access Sweep. “Fusion” indicates how the visual stream is exposed to the language decoder. Family Qwen2.5-VL [Bai et al., 2025] InternVL3 [Zhu et al., 2025] Llama-3.2-Vision-Instruct [Grattafiori et al., 2024]

Scale

Decoder layers

Vision encoder / connector

Fusion

3B / 7B / 32B

36 / 28 / 64

Qwen ViT + MLP

Prefix

8B / 14B / 38B

28 / 48 / 64

InternViT + MLP

Prefix

11B

40

CLIP ViT + cross-attn

Cross-attention

22

C.3

Qwen2.5-VL Family Sweeps

Figure 10 shows per-scale Direct VAB curves and CoT VAB heatmaps for the Qwen2.5-VL family.

(a) 3B Direct.

(b) 3B CoT.

(c) 7B Direct.

(d) 7B CoT.

(e) 32B Direct.

(f) 32B CoT.

Figure 10: Qwen2.5-VL family sweeps. Rows correspond to model scale; left panels show Direct layer sweeps and right panels show CoT layer-token sweeps. Exact boundary values are reported in Tables 7 and 8.

23

C.4

InternVL3 Family Sweeps

To assess whether the VAB phenomenon generalizes across model families, we additionally evaluated InternVL3 models (8B, 14B, 38B). These models use a different vision encoder and projection architecture compared to Qwen2.5-VL.

(a) 8B Direct.

(b) 8B CoT.

(c) 14B Direct.

(d) 14B CoT.

(e) 38B Direct.

(f) 38B CoT.

Figure 11: InternVL3 family sweeps. Rows correspond to model scale; left panels show Direct layer sweeps and right panels show CoT layer-token sweeps. Exact boundary values are reported in Tables 7 and 8.

24

C.5

Cross-Attention Fusion Pilot

The main experiments use prefix-fusion VLMs, where image tokens are concatenated with text tokens and direct visual access is realized through self-attention from generated-token queries to image-token keys. As a pilot check, we apply the same intervention principle to Llama-3.2-11B-Vision-Instruct, where visual features are exposed through cross-attention modules rather than through an image-token prefix. In this case, the sweep masks generated-token queries’ cross-attention to visual keys over layer depth and generation time. We report the CoT-side sweep only. Direct prompting on this model did not reliably follow the option-key answer format under our extraction protocol, so the Direct layer sweep was not a clean comparison. This result should therefore be read as an intervention-transfer check for cross-attention fusion, not as a full family-level comparison.

Figure 12: CoT-side Visual Access Sweep on a cross-attention fusion VLM (Llama-3.2-11B-VisionInstruct, color counting, ϵ = 0.05). The intervention masks generated-token cross-attention to visual keys. Reported as a pilot check that the sweep can be applied beyond prefix-fusion models.

25

C.6

GQA Real-Image Extension

Table 10 summarizes the construction and evaluation protocol for the GQA yes/no extension. This experiment is used as a real-image check of the VAB structure, not as a claim that CoT improves GQA accuracy. We extract the final occurrence of \b(yes|no)\b (case-insensitive) as the predicted label. Table 10: GQA yes/no extension setup and boundary summary. Item

Value

Source split Filter Sample size Label balance Model Image preprocessing Extraction rule Direct full accuracy / ℓ∗ CoT full accuracy / (ℓ∗ , τ ∗ )

val_balanced answer ∈ {yes, no} (case-insensitive) 100, shuffled with seed 0 53 no / 47 yes (chance level 0.5) Qwen2.5-VL-32B RGB conversion, resize to model native input last yes/no token (YN _ LAST) 0.84 / 40 (N ∗ = 24) 0.80 / (28, ≈ 32)

A finite VAB appears in both Direct and CoT sweeps, even though CoT does not improve over Direct on this binary-answer subset. This supports the main-text claim that the VAB phenomenon extends beyond synthetic stimuli (Section 5.4), independent of whether CoT itself improves task accuracy. C.7

Limiting Case: Angle Counting

Figure 13 shows angle counting on Qwen2.5-VL-32B. We include this as a limiting case rather than as main VAB evidence. When full-access accuracy is near the task floor, the sweep cannot cleanly localize a boundary, because preserving low accuracy is not informative about successful visual grounding. This case supports the interpretation used in Section 6: extended reasoning cannot compensate when the queried visual attribute is not reliably read out into a usable symbol.

(a) Direct layer sweep.

(b) CoT layer-token sweep.

Figure 13: Angle counting as a limiting case (Qwen2.5-VL-32B). Full-access performance is near the task floor, so the sweep does not yield an interpretable boundary in the same sense as color or shape counting. C.8

Qualitative Error Breakdown

Table 11 provides a small qualitative breakdown of representative errors. We use it only to characterize failure modes, not as a main quantitative result. Errors are grouped into four categories: alt-attributeconsistent, where the predicted count matches the count of a non-target attribute value in the same scene; tally-inconsistent, where the target attribute is identified but the written count disagrees with the visible tally; format errors, where outputs fail extraction (no valid option key or yes/no token); and other errors that match none of the above. 26

Takeaway. When direct visual access is blocked before the VAB, errors are mostly alt-attributeconsistent, suggesting that the intervention corrupts which visual attribute or value is available for counting. Under full-access CoT, the remaining errors shift toward tally inconsistency, suggesting that once the visual attribute is available, residual failures more often occur in the symbolic counting step. Table 11: Qualitative error breakdown over incorrect examples only. Categories follow the definitions above. Task

Condition

Acc

Alt-attr. cons.

Tally incon.

Other

Format

color color color color

Full access (Direct) Beyond VAB (ℓ=36) Strong block (ℓ=16) CoT full access

0.80 0.35 0.35 0.95

20 65 65 0

0 0 0 4

0 0 0 1

0 0 0 0

shape shape shape shape

Full access (Direct) Beyond VAB (ℓ=34) Strong block (ℓ=16) CoT full access

0.84 0.49 0.20 0.89

16 51 80 0

0 0 0 9

0 0 0 2

0 0 0 0

angle angle

Full access (Direct) CoT full access

0.22 0.37

77 0

0 61

0 0

1 2

27

D

Perceptual Readout: Detailed Results

This section reports two pieces of supporting evidence for Section 6: a probe-vs-decode comparison that separates latent visual grounding from decoded readout, and a per-attribute breakdown of the readout–CoT-gain relationship underlying Figure 5.

D.1

Probe-vs-Decode Gap

This appendix uses a single-object setting only for the probe-vs-decode diagnostic. This is separate from the multi-object perceptual readout accuracy used as the Figure 5 predictor in Section 6. The goal here is to test whether attribute information is present in hidden states even when the model’s own decoded answer is unreliable. Table 12 reports this comparison on Qwen2.5-VL-32B. An attribute can be linearly recoverable from hidden states (Probing Acc) while still being unavailable to the model as a decoded output symbol (Decoding Acc).

Table 12: Probe-vs-decode comparison on Qwen2.5-VL-32B. Probing Acc measures linear recoverability from hidden states; Decoding Acc measures whether the model can output the attribute value itself. Gap (highlighted) is Probe minus Decode. Attribute

Probing Acc

Decoding Acc

Gap (Probe − Decode)

Color Shape Angle Location Size

1.000 1.000 0.954 0.971 0.892

1.000 1.000 0.383 0.312 0.283

0.000 0.000 +0.571 +0.658 +0.608

Table 13: Linear-probe protocol used in Table 12.

D.2

Item

Value

Model Layer Token position Data Probe Evaluation

Qwen2.5-VL-32B Final decoder layer (after the last Transformer block) Final image-token position 500 single-object images, 400 / 100 train-validation split ℓ2 -regularized multinomial logistic regression (λ = 1.0, scikit-learn) Held-out accuracy on the 100-image validation split

Per-Attribute CoT Gain

Table 14 reports the per-attribute values underlying Figure 5. Each block corresponds to one Qwen2.5VL scale. Within each block, attributes are ordered by multi-object perceptual readout accuracy. Pearson r and Spearman ρ are descriptive summaries over five attributes per scale, not independent evidence of a universal scaling law.

Takeaway. Across Qwen2.5-VL scales, attributes with stronger multi-object readout generally show larger CoT gains, while attributes with weak readout show small or negative gains. Together with the oracle result in Figure 6, this supports the interpretation in Section 6: CoT gain is constrained by whether the queried visual attribute is available as a usable symbol. 28

Table 14: Per-attribute readout and CoT-gain values underlying Figure 5. ∆CoT is CoT minus Direct; shading indicates the sign of ∆CoT (green positive, red negative). Correlations are computed over the five attributes within each scale. Readout

Direct

CoT

∆CoT

Qwen2.5-VL-3B color 0.713 shape 0.605 location 0.601 size 0.347 angle 0.245

0.580 0.610 0.360 0.140 0.290

0.830 0.770 0.470 0.190 0.320

+0.250 +0.160 +0.110 +0.050 +0.030

Attribute

Pearson r = +0.92, Spearman ρ = +1.00

Qwen2.5-VL-7B color 0.498 shape 0.354 location 0.222 size 0.131 angle 0.005

0.790 0.770 0.370 0.310 0.440

0.890 0.850 0.420 0.340 0.320

+0.100 +0.080 +0.050 +0.030 −0.120

Pearson r = +0.88, Spearman ρ = +1.00

Qwen2.5-VL-32B color 0.920 shape 0.865 location 0.771 size 0.479 angle 0.254

0.840 0.850 0.410 0.200 0.370

0.950 0.890 0.570 0.270 0.400

Pearson r = +0.51, Spearman ρ = +0.50

29

+0.110 +0.040 +0.160 +0.070 +0.030

E

Practical Implication: Selective CoT Routing

The perceptual-readout analysis suggests a simple routing heuristic, namely using CoT only when the queried attribute can be reliably read out, and otherwise falling back to Direct inference. For each queried attribute, we route to CoT if the corresponding multi-object readout accuracy on the readout validation split exceeds 0.5. This threshold is not tuned for optimal deployment. It is used as a simple illustrative rule. Table 15 shows that this rule preserves most of the accuracy gain of always using CoT while substantially reducing output-token cost. Table 15: Selective CoT routing as an efficiency-oriented heuristic. The adaptive policy uses CoT only when the queried attribute’s multi-object readout accuracy exceeds 0.5. Gain recovered is the fraction of the Always-Direct→Always-CoT improvement preserved by the adaptive policy. Token cost vs CoT is average output tokens normalized by the Always-CoT row. The adaptive row is highlighted. Model

Policy

Acc.

Gain vs Direct Gain recovered Avg. tokens

Tokens vs CoT

Qwen2.5-VL-3B Qwen2.5-VL-3B Qwen2.5-VL-3B

Always Direct 0.396 Always CoT 0.516 Adaptive (thr=0.5) 0.500

— +0.120 +0.104

— 100% 86.7%

2.6 166.7 82.7

1.6% 100% 49.6%

Qwen2.5-VL-32B Always Direct 0.534 Qwen2.5-VL-32B Always CoT 0.616 Qwen2.5-VL-32B Adaptive (thr=0.5) 0.596

— +0.082 +0.062

— 100% 75.6%

3.0 224.3 102.9

1.3% 100% 45.9%

Takeaway. A cheap attribute-level readout check can recover most of the benefit of always using CoT while avoiding roughly half of the output-token cost in these two representative Qwen2.5-VL settings.

30