The Ebb and Flow of Multimodal Focus: Scheduling Visual Relay Windows for Grounded VLM Reasoning Wencheng Ye1 , Yi Bin1 , Yujuan Ding2 , Hongye Fang1 , Zheng Wang1 , Xing Xu1 , Jingkuan Song1 , Yun Zhang3 , Sirui Da1 , Heng Tao Shen1 1 School of Computer Science and Technology, Tongji University 2 School of Fashion and Textiles, The Hong Kong Polytechnic University 3 School of Artificial Intelligence, Shanghai Jiao Tong University Correspondence: [email protected]
Abstract
2024a; Bai et al., 2025; Wang et al., 2025b; Yue et al., 2024; Lu et al., 2024; Thawakar et al., 2025). Yet this progress also sharpens a central requirement: visual evidence must remain available across multiple inference steps and be reorganized into answer-relevant structure (Li et al., 2025; Liu et al., 2024a). This is difficult because once visual information is absorbed into the language stack, its influence can attenuate with depth, become entangled with language organization, or drift as generation unfolds (Zhong et al., 2024; Xu et al., 2026). Recent methods improve visual evidence support in VLMs through diverse control handles anchored to different actionable signals: contrastive decoding exploits the sensitivity of generation to visual perturbation (Leng et al., 2024; Zhao et al., 2024a), attention reallocation targets insufficient attention paid to visual tokens (Liu et al., 2024c; Zhang et al., 2026a), and activation steering targets activation patterns associated with ungrounded generation (Wu et al., 2025; Su et al., 2025; Liu et al., 2025; Yin et al., 2026). Together, these advances show that failures of visual evidence support can often be mitigated at inference time once the right signal is acted on at the right place. What remains less clear is how these signals, each anchored to a different point of computation, relate to one another over the course of inference: whether they should be treated as separate cues calling for separate fixes, or as partial snapshots of a single evolving process governing how visual evidence is organized and used. As a result, it remains difficult to tell how intervention should track the changing role of visual evidence as reasoning unfolds. Addressing this requires a more explicit account of multimodal computation during inference. Prior mechanistic studies show that attention-based analysis can reveal functionally meaningful structure inside VLMs, identifying how visual information is re-
arXiv:2607.11436v1 [cs.AI] 13 Jul 2026
Vision-language models increasingly succeed on multimodal reasoning benchmarks, yet their visual evidence often becomes unstable once it enters the language stack, weakening evidence-grounded reasoning. To understand this fragility, we examine the internal dynamics of VLMs through a mechanistic lens and uncover a stable three-stage redistribution of multimodal attention focus across depth: an early question-conditioned organization, a critical middle visual-dominant relay, and a late return to answer formation. We operationalize the middle phase as the Visual Relay Window (VRW), and show that its geometry varies with task demand, is causally tied to grounded generation, and distinguishes unsupported answers from stronger reasoning trajectories. Guided by this internal rhythm, we propose TRACE, a task-adaptive inference-time control framework with lightweight trained modules. It reshapes relay allocation during prefill and preserves assembled visual support after handoff during decoding. Across four open-weight VLM backbones and seven benchmarks, TRACE delivers large gains on groundingsensitive settings, improving them by 4.33 points on average and by up to 6.6 points, while also improving reasoning-heavy tasks. These results show that explicitly controlling multimodal focus across depth offers a unified and effective mechanism for strengthening evidence-grounded multimodal reasoning. The code is available at: https://github. com/gooogleshanghai/visual-relay-window.
Keywords: Vision-language models, multimodal reasoning, hallucination mitigation.
1
Introduction
Vision-language models (VLMs) have advanced rapidly in both scale and capability, with recent systems showing competitive reasoning ability and strong performance on broad multimodal benchmarks (Zhang et al., 1
Normalized Attention
Question-conditioned Organization
How many people are holding umbrellas?
Visual Relay Window
Linguistic Answer Formation
Focus shifts to visual stream. Evidence is gathered and structured. Intergrate Gather
Reasoning Based on evidence, the answer is: 5.
Retrieve
Figure 1: A unified three-stage relay pattern across vision-language models. Across ten open VLMs, the jointly normalized Answer→Query and Image→Image trajectories show a consistent depth-wise redistribution of multimodal focus: early layers emphasize question-conditioned response organization, middle layers allocate computation to visual consolidation, and later layers shift back toward answer formation. The horizontal bracket marks the relay interval, while paired markers match individual models to their trajectories.
distributed across layers, token groups, and components (Basu et al., 2024; Zhang et al., 2025b; Li et al., 2026; Jiang et al., 2025; Shanbhag et al.). Through such analyses, they find that visually relevant transfer is concentrated in a subset of middle layers and finegrained details are retrieved more locally from image tokens (Zhang et al., 2025b; Kaduri et al., 2025; Jiang et al., 2025). We therefore ask a more structural question: how does multimodal focus evolve over the course of inference as a VLM reasons toward an answer?
assembled support is handed off to generation. Figure 1 sketches this progression and shows that the same three-stage pattern appears consistently across models, marking a limited interval in which visual consolidation dominates before computation returns to language side. Crucially, this relay geometry is not fixed. Our analysis suggests that it shifts with the evidence demands of the question: tasks that require broader visual aggregation such as counting tend to exhibit longer and later relay phases, whereas tasks that place greater weight on downstream reasoning tend to exhibit earlier handoff from visual consolidation to answer formation. Success therefore depends not only on whether relevant visual content is present, but also on whether the timing of relay match the computation the task requires. Mismatch can arise in either direction. If relay ends too early, answerrelevant evidence may remain only partially assembled before generation begins; if it persists too long, computation that would otherwise support later reasoning will be occupied by unneeded visual processing. This interpretation is further supported by causal analyses: perturbing states within the relay interval changes whether later responses remain grounded far more strongly than comparable perturbations outside it. Stronger Thinking variants likewise exhibit more stable relay matching and stronger post-handoff anchoring, suggesting that part of their advantage may lie in allocating this intermediate stage more appropriately.
To answer it, we track two attention probes across layers: an answer-side probe measuring how strongly answer tokens look back to the question, and a visualside probe measuring how strongly visual tokens attend to one another. Aggregated attention mass under these two probes gives a compact proxy for how multimodal interaction is redistributed. What emerges is a clear three-stage pattern, where the two probes behave like opposing currents over depth. In the early layers, attention remains dominated by question-conditioned organization: the model is still aligning the visual input with the query and establishing an answer-relevant frame, so the question-directed probe rises while intra-visual attention falls. Midway through the network, the balance shifts toward intra-visual consolidation and attention to the question recedes, where visual tokens interact more strongly with one another and answer-relevant evidence is actively assembled. In the later layers, computation returns to language-side answer formation, where the
These findings make relay geometry a natural target 2
for intervention. We therefore propose TRACE (Taskadaptive Relay Anchoring and Controlled Evidence Scheduling, an inference-time control framework with lightweight trained modules that regulates how visual support is assembled before handoff and retained afterward. TRACE first uses lightweight prefill-time attention statistics to predict which layers are likely to participate in relay, then adaptively expands or contracts relay allocation through task-aware scheduling, and finally anchors the selected visual support after handoff so that relevant evidence remains available during decoding.
2
Related Work
2.1
Visual Evidence Degradation in Multimodal Reasoning
Vision-language models (VLMs) have increasingly shifted from perception-oriented tasks toward multistep, evidence-dependent reasoning(Zhang et al., 2024a; Thawakar et al., 2025; Zhou et al., 2026). This transition is especially visible on benchmarks such as MMMU(Yue et al., 2024) and MathVista(Lu et al., 2024), where success depends on carrying visual evidence across multiple inference steps, and is widely tied to a persistent modality gap between visual and language representations(Bai et al., 2024; Liu et al., 2024a). Since visual tokens are projected into the language stack through a lightweight connector, they often behave as weakly aligned soft prompts whose influence degrades with depth and generation length(Li et al., 2025; Chen et al., 2026). Modern VLMs therefore frequently fall back to language priors over visual evidence, leading to visual neglect across perception, spatial understanding, and long-form reasoning settings(Zhao et al., 2024b; Chandhok et al., 2025; Alam et al., 2026; Xi et al., 2026; Xu et al., 2026; Zhong et al., 2024). These findings shift attention from visual access alone to how visual evidence is maintained through reasoning.
We evaluate TRACE on four open-weight VLM backbones from two families across seven benchmarks spanning document understanding, visual reasoning, and hallucination-sensitive grounding. It delivers large gains on grounding-sensitive settings, with an average improvement of 4.33 points and gains of up to 6.6 points, while also improving reasoning-heavy settings consistently. The intervention also produces the intended internal shifts: grounding-sensitive benchmarks benefit from broader, later relay phases, whereas reasoningheavy cases improve when relay becomes shorter and hands off earlier. This highlights the unified feature of our approach: rather than designing separate fixes for hallucination mitigation or logical reasoning, we cast the diverse demands of downstream tasks into a single underlying mechanism.
2.2
Our contributions are threefold:
Mechanistic Analysis of Multimodal Computation
To understand why generation becomes less visually grounded during reasoning, recent work has increasingly used attention patterns and token-group interactions as informative probes of multimodal computation inside VLMs (Shanbhag et al.; Zhang et al., 2025b; Saporita et al., 2026). Through these analyses, several studies show that intermediate layers play an especially important role in consolidating visual evidence and aligning it with language representations (Yang et al., 2026; Kaduri et al., 2025; Jiang et al., 2025). Causal tracing and circuit-level analyses further suggest that object representations and image-to-text transfer pathways are often concentrated in a subset of layers and components (Li et al., 2026; Zhang et al., 2025a; Basu et al., 2024). Several works also report failure sources such as representational interference, attention sinks, and visual information dilution (Kang et al., 2025; Chen et al.; Savietto et al., 2026; Zhang et al., 2024b). These findings together suggest that visual information is redistributed non-uniformly across depth, but the stage-wise structure that governs evidence buildup remains less explicitly characterized. Using attention as a lens on how interaction is redistributed across token groups and depth, our
• We identify a stable stage-wise redistribution of multimodal focus across depth and operationalize its middle relay phase as the Visual Relay Window (VRW), establishing a structural measure for internal evidence assembly.
• We demonstrate that this geometry dynamically adapts to specific task demands, and establish a causal link between optimal relay timing and grounded generation, offering a mechanistic explanation for typical hallucinations and capability differences across models.
• We introduce TRACE, an intervention framework that reshapes relay allocation during prefill and preserves assembled support during decoding. By viewing the diverse demands of downstream tasks through a single computational lens, it addresses both hallucination mitigation and complex reasoning under a unified control principle. 3
analysis reveals a relay pattern in which visual consolidation and language-side organization exhibit a clear stage-wise trade-off, and operationalizes this structure as the Visual Relay Window. 2.3
Consider a vision-language model that takes an imagequestion pair (I, q) as input and autoregressively generates an answer sequence y = (y1 , . . . , yT ). Let the model contain L Transformer layers, and let A(l) ∈ RN ×N denote the self-attention matrix at layer l. We partition the sequence into three groups,
Inference-Time Control for Vision-Language Models
V ∪ Q ∪ Y = {1, . . . , N },
A large body of previous work attempts to mitigate failures of visual evidence support through inferencetime intervention. One prominent direction is contrastive decoding, including VCD(Leng et al., 2024), MARINE(Zhao et al., 2024a), and several perturbationbased variants(Park et al., 2025; Jiang et al., 2026), which contrast original and perturbed visual representations to suppress unsupported generations. These approaches mainly operate at the final decoding stage, therefore cannot directly repair upstream evidence formation errors. Another line of work intervenes in latent representations through activation steering or hiddenstate editing(Wu et al., 2025; Wang et al., 2025a; Su et al., 2025; Liu et al., 2025), which may alter entangled reasoning-relevant representations(Sivakumar et al., 2025; Cheng et al., 2026; Yin et al., 2026). More recently, attention-level and prefill-stage interventions have emerged as a promising alternative. Methods such as PTI(Zhang et al., 2026a), PAI(Liu et al., 2024c), and AVAM(Zeng et al., 2025) show that earlier intervention points can improve visual evidence support by reallocating attention or preserving visual context before decoding begins(Zhang et al., 2026b; Fu et al., 2026). Our work builds on these advances by explicitly modeling relay geometry and using it as a unified control target. This intervention improves grounded reasoning directly, while hallucination reduction emerges from the same mechanism.
3
where V, Q, and Y respectively denote visual, question, and answer tokens. We first define an answer-side probe that measures how strongly the model refers back to the question while forming a response: (l)
MA→Q =
1 X X (l) Aij , |Y|
(2)
i∈Y j∈Q
We next define a visual-side probe for intra-visual consolidation: 1 X X (l) (l) MI→I = Aij . (3) |V| i∈V j∈V
For autoregressive decoding, the answer-side probe aggregates the corresponding attention mass over answer-generation steps, yielding a layerwise estimate of question-conditioned response formation. The visual probe is computed within each layer as a summary of intra-visual consolidation. Together, these two probes provide a compact view of how multimodal computation is distributed between language-side organization and visual evidence assembly across depth. 3.2
A Shared Relay Structure Across VLMs
Cognitive accounts of scene understanding describe a stage-wise progression, which moves from perceptual encoding to evidence integration and response selection (Van Maanen et al., 2021; Epstein and Baker, 2019). Consistent with this view, after aligning layers by relative depth and jointly normalizing the two relay channels, the balance between the two probes likewise reveals a three-stage trajectory: an early questionconditioned regime, a middle visual-dominant regime, and a late return to response formation. Figure 1 shows that this three-stage pattern survives substantial architectural variation, covering ten checkpoints from QwenVL(Qwen et al., 2025; Bai et al., 2025), InternVL(Wang et al., 2025b), and LLaVA(Liu et al., 2024b) on a balanced probing set of 2,000 image-question pairs spanning four task families. Despite large differences in model size, tokenizer layout, and absolute attention scale, the same phase ordering persists. What changes across models is the geometry of the middle visualdominant segment, especially its extent and termination
Structure-Aware Control in VLMs
We first identify a relay structure that organizes multimodal evidence flow and establish it as a measurable target for grounded generation. We then show how TRACE operationalizes this structure as an inference-time control mechanism. 3.1
(1)
Attention Probes for Depth-Wise Multimodal Focus
Our goal is to examine how multimodal focus evolves across depth as a VLM reasons. To do so, we track a small set of token-group attention flows that capture complementary aspects of this process. One reflects question-conditioned response organization, while the other reflects consolidation within the visual stream. 4
(A)
Table 1: Top probe pairs ranked by reciprocal correlation on the probing split. Lower correlation indicates a stronger reciprocal trend. Source relation Corr. ↓
Probe pair
Answer→Query/ Image→Image Disjoint-source Answer→Query/ Answer→Image Same-source Query→Query/ Image→Image Disjoint-source Answer→Query/ Query→Image Disjoint-source
-0.93 -0.87 -0.84 -0.73
(B)
depth. We therefore isolate this segment and operationalize it as the Visual Relay Window (VRW). 3.3
Operationalizing the Visual Relay Window
We formalize the VRW as an operationally estimated middle-layer segment in which visual evidence dominates before answer commitment. Among non-identical probe pairs, Answer→Query and Image→Image exhibit the clearest reciprocal depth-wise structure on the probing split (Table 1), which makes them a suitable basis for this operational summary. To compare samples and model families, we jointly normalize the two relay channels within each sample, and define the relay-dominance score (l) (l) R(l) = M I→I − M A→Q . (4)
Figure 2: Task-dependent geometry of the Visual Relay Window on Qwen3-VL-4B. (A) Distributions of VRW-End and VRW-Width across task families. (B) Task-wise mean relay curves for four task families.
From the final VRW boundaries, we derive two statistics used throughout the paper:
Higher values indicate stronger visual consolidation, while lower values indicate stronger questionconditioned response organization. Intuitively, R(l) is a signed summary of which side currently dominates the computation at layer l. In practice, before comput(l) (l) ing R(l) , we jointly normalize MI→I and MA→Q within each sample by min-max scaling them over layers to [0, 1]. To focus on the middle relay phase, we define the relative depth ρl = l/L and use the heuristic setting ρl ∈ [0.2, 0.8] together with a tolerance margin δ = 0.1. This restricts the estimate to the network interior and suppresses small local ripples. We then define the relay peak as l⋆ = arg max R(l) . (5)
VRW-Width =
3.4
∀l ∈ [s0 , e0 ].
(6)
This candidate interval permits local fluctuations while requiring the retained segment to remain close to the middle-layer relay peak. The final VRW is the longest contiguous sub-interval [s, e] ⊆ [s0 , e0 ] that still contains l⋆ and preserves the global handoff trend: (e)
(s)
M I→I − M I→I > 0,
(e)
e . (8) L
Task-Dependent Relay Geometry
The cross-model regularity above motivates an immediate question: is the relay window fixed or highly adaptive? To answer this, we group evaluation samples into four families—detail-heavy perception, recognition, relation/spatial reasoning, and high-level reasoning—and test whether relay geometry shifts systematically with evidence demand. As Figure 2 shows, relay geometry is strongly taskdependent (one-way ANOVA, p < 0.005). Detail-heavy tasks tend to produce broader windows and later relay termination, indicating that visual evidence continues to be assembled deeper into the stack. High-level reasoning tasks, by contrast, often exhibit shorter windows and earlier handoff, while recognition and relation-oriented tasks occupy intermediate regimes. Taken together, these patterns show that the redistribution of multimodal focus is itself task-sensitive, which reflects a control policy that the model may have acquired during massive
The candidate interval [s0 , e0 ] is the maximal contiguous interval containing l⋆ such that ⋆
VRW-End =
VRW-Width measures the relative extent of the relay phase, while VRW-End measures the depth at which it terminates.
l: ρl ∈[0.2,0.8]
R(l) ≥ R(l ) − δ,
e−s+1 , L
(s)
M A→Q − M A→Q < 0. (7) 5
A1: It looks more like a stool. The seat appears small, backless and made of wood.
Unsupported
Table 2: Relay patching controls. Pre- and post-relay denote the full ranges before and after the estimated VRW, while shifted controls preserve the VRW width. Recovery denotes the fraction of unsupported branches that become supported after patching.
Answer—>Query Image—>Image
A2: It appears to be a chair. The person is making music and seems to be seated on a supported seat.
Figure 3: Case study of relay bifurcation for the same imagequestion pair. The grounded trajectory sustains a broader relay phase and remains visually supported, whereas the unsupported trajectory exits earlier and drifts toward language-side priors. Lower overlays visualize answer-to-visual attention after relay termination.
Patched range
Recovery (%)
Pre-relay range Post-relay range Mean over same-width shifts Estimated VRW
8.7 11.2 27.6 ± 3.3 41.4