Don’t Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programs
(a) Standard
PerceptionProgram
Multi-View Reasoning (Accuracy)
P2 Generator
Tool Output
Expert Tool
Is the camera moving (A) left or (B) right when shooting the video? Select between A and B.
Input Pair (e.g., multi-view reasoning)
Query
arXiv:2604.12896v1 [cs.CV] 14 Apr 2026
Muhammad Kamran Janjua1∗ Hugo Silva1∗ Di Niu2 Bahador Rashidi1 1 Huawei Technologies, Canada 2 University of Alberta, Canada github.com/AISmartPerception/perception-programs
(c) PerceptionProgram (Ours)
(b) Tool
P2 Generator
MLLM Q Input Pair
Q
Query
MLLM
MLLM Input Pair
Query
P2
Q Tool Output
Input Pair
Query
Tool Output
Figure 1. Teaser. Turning dense tool outputs into a Perception Program makes a general MLLM behave as if it can read the modality. Given same query and input pair, (a) standard MLLM underuses the visual signal [6], (b) a tool-only route exposes the modality but stays pixel-level, while (c) our P2 summarizes it into a language-native structure that MLLM can reliably reason over, yielding large gains.
Abstract Multimodal language models (MLLMs) are increasingly paired with vision tools (e.g., depth, flow, correspondence) to enhance visual reasoning. However, despite access to these tool-generated visual cues, MLLMs often fail to benefit fully from them. Existing approaches typically feed raw tool outputs into the model, but these dense, pixel-level representations are misaligned with the language-native reasoning strengths of LLMs, leading to weak perception due to reliance on language priors. We argue that, in problems where vision tools can provide the necessary visual cues, the bottleneck is not more tool calls or larger MLLMs, it is how tool outputs are represented. We introduce Perception Programs (P2 ), a training-free, model-agnostic method that rewrites tool outputs into compact, structured, languagenative summaries that MLLMs can directly parse and reason over. Across six perception-centric tasks in BLINK, P2 consistently yields large improvements over base models and * Equal Contribution correspondence: [email protected]
raw tool-augmented baselines. With GPT-5 Mini as the base model, P2 raises its accuracy from 41.35% to 86.47% on multi-view reasoning, from 52.42% to 81.45% on relative depth, and achieves a 19.66% overall average gain across tasks, setting new state-of-the-art results. Even on smaller MLLMs, e.g., InternVL3.5-4B and Qwen3VL-4B, we observe 21–25% absolute gains from P2 across tasks when comparing to image-based tool variants, surpassing prior agentic, supervised, and RL-based tool-use methods, without any training or model modifications.
1. Introduction Modern multimodal language models (MLLMs) are increasingly expected to perform perception-driven reasoning over visual inputs such as images and video [8, 16]. Recent advances have made it feasible to pair MLLMs with vision tools, e.g., monocular depth estimation, optical flow, visual correspondence, and object detectors, to surface perceptual signals that are not directly accessible from pixel inputs alone [1, 29, 31]. Despite this, MLLMs often under-utilize
Original Image
G.T. Depth
G.T. Quantization
GPT-5 Mini Prediction
Figure 2. Under-utilization of Visual Information. Given several ICL examples along with depth map, GPT-5 Mini fails to recover near-to-far ordering from it (see Sec. 5.1), indicating limited utilization of the modality.
the rich information provided by these tools. When raw tool outputs are serialized and supplied to the model, they appear as dense, low-level visual tokens that misalign with the language-native reasoning substrate of LLMs [6]. As a result, MLLMs exhibit limited perceptual grounding, inherit language priors from the base LLM, and frequently fail on tasks that require interpreting visual modalities, even when provided with accurate tool surrogates for those modalities, see Fig. 2 for an illustration. Prior work has attempted to mitigate this limitation through (i) program-synthesis pipelines that generate executable code to invoke tools [9, 19], (ii) chain-of-thoughtbased tool-use methods that interleave tool calls with reasoning [1, 29], (iii) fine-tuning (SFT or reinforcement-learning) to encourage tool-augmented reasoning [14, 17, 30, 33, 35], or (iv) architectures incorporating perception-oriented modules [2, 21]. These approaches, however, increase computational cost, require specialized training, or continue to operate at the same pixel-level granularity as the raw tool outputs, thus inheriting the fundamental representation mismatch between visual tool outputs and the linguistic reasoning capabilities of MLLMs. Collectively, these trends raise a central question: Is the true bottleneck in tool-augmented MLLMs the number of tool calls or model size—or the representation of the visual information itself? The main motivation of this work stems from contrasting how visual information is fundamentally captured by humans with the pixel-level processing often imposed on MLLMs. Human cue-taking varies widely depending on the type of data: surface proximity for depth-related problems, left–right movement for spatial orientation, or local similarity when looking for correspondences between images. Converting the key information in each of these problems into text reduces the models’ burden of processing pixellevel details and better equips them to interpret and reason about such information, since text is the representation that best aligns with their native reasoning capabilities. With that in mind, we introduce Perception Programs (P2 ), a training-free, model-agnostic representation that
rewrites raw tool outputs into compact, structured, languagenative summaries of visual modalities. P2 standardizes what is conveyed from a tool (the perceptual signal), where it is grounded spatially in the image, and (optionally) how local regions relate. This reformulation enables MLLMs to read the visual modality rather than infer it from dense numeric tokens. P2 requires no parameter updates to the MLLM, no architectural modifications, and no additional tool calls during inference. It serves as a plug-and-play module: the same tool output provided to a standard tool-use pipeline is instead converted into a P2 and consumed directly by any off-the-shelf MLLM, see Fig. 1 for illustration. Our contributions are summarized as follows: • Unified Representation. We introduce Perception Programs (P2 ), a symbolic framework that reformulates visual tool outputs as visually-grounded language-native summary. It generalizes across depth estimation, optical flow, visual and semantic correspondence, jigsaw-style reasoning, and object localization. In addition to requiring no model fine-tuning or multiple tool calls, it works across model scales and architectures. • Comprehensive Evaluation. We show that this representation enables robust reasoning on six BLINK [8] tasks. P2 yields an average gain of 19.48% on larger base models (GPT-5 Mini, Gemini 2.5 Pro) relative to raw tool use, while improving smaller open-source models such as InternVL3.5-2B, InternVL3.5-4B, and Qwen3VL-4B by 22.18%. Across all benchmarks, P2 achieves an average 19.66% improvement over the prior best results. • Analysis of Bottlenecks. We examine GPT-5 Mini’s ability to interpret raw visual tool outputs by prompting it to verbalize its understanding in language. For depth maps, the model fails to preserve relative patch ordering—Kendall’s τ rapidly approaches zero as grid resolution increases (see Fig. 2). For visual correspondence, it often reproduces input tokens rather than reasoning over image content. Moreover, chain-of-thought prompts that encourage explicit verbalization of visual cues produce noisy descriptions and degrade performance. We also investigate the effect of augmenting existing agentic tool-use methods with P2 , yielding 18.28% improvement on depth and localization tasks, demonstrating that P2 enhances both interpretability and effectiveness in tool-use MLLMs. These results show that when vision tools already provide the necessary visual cues, the primary bottleneck is not additional tool calls or larger models, but the representation of visual tool outputs. P2 directly addresses this bottleneck by presenting the evidence in a language-native form that MLLMs can reliably parse and reason over.
2. Related Work Fu et al. [6] show that MLLMs often under-use their encoders and lean heavily on language prior, while another
work documents limitations in their visual understanding [24]. Recent efforts attempt to mitigate these limitations either through in-context learning ability of LLMs [9], zeroshot prompting [19], or fine-tuning [33] to emit modular programs (in a programming language), designing specialized agent-and-tool collaboration mechanisms [34], or enabling MLLMs to think with images [5, 14, 29]. Tool-Use through Program Synthesis. Here, the MLLM is given an API specification plus a query (and/or image) and is asked to synthesize an executable program whose output is used to answer the question. VisProg [9] uses incontext examples with GPT-3 to generate Python programs that call vision tools and subroutines. ViperGPT [19] similarly prompts a code-generation model (GPT-3 Codex) with the query and an API specification of available modules (classical image operations, neural models, other LLMs) and executes the resulting Python on the input image or video. Thyme [33] fine-tunes Qwen2.5-VL (7B) in two stages (SFT + RL) so that, for each query, the model can emit code, reasoning, or both before answering. MMFactory [4] maintains a model/tool repository and, given a multimodal query and constraints, proposes programmatic pipelines that compose vision tools and (M)LLMs. While powerful, these systems can be computationally heavy, often require multiple LLM calls and a sand-boxed executor, and may struggle with multi-image settings [4]. Tool-Use through Chain-of-Thought. An alternative is to integrate tools directly into the reasoning chain instead of emitting full programs. Aurora [1] introduces perception tokens, discrete latent codes that approximate a visual modality, and let the MLLM ‘call’ vision tools within its chain-of-thought. Mirage [29] encourages the model to ‘visually imagine’ by generating latent image representations as intermediate states, and is trained in two stages (latent reinforcement followed by text-only prediction). Visual Sketchpad [10] leverages sketch-like intermediate tool outputs, motivated by human use of sketches for visual reasoning. LATTE [14] builds a multimodal reasoning dataset that includes expert visual tool outputs and fine-tunes a family of MLLMs on it. However, these methods still operate at pixel-level tool outputs and thus inherit under-utilization of vision encoders [6] and limited understanding of dense visual modalities. For further review, see Sec. 8 (appendix).
3. Perception Programs We formalize a Perception Program (P2 ) as a compact, symbolic summary of sensory inputs, or visual modalities, that standardizes what is present, where it is located, and optionally how different parts relate. P2 exposes a unified item schema shared across modalities and grounded in language, enabling off-the-shelf LLMs to reason over expert tool outputs through reading the visual modalities.
3.1. General Construction Let the pixel domain be Ω = {0, ..., W −1}×{0, ..., H −1}. We define a finite set of primitives P, wherein each primitive p ∈ P is associated with a spatial support Sp ⊆ Ω (e.g., a patch, a point, an image, etc.) and a normalized location cp ∈ {0, ..., 1000}2 . For any pixel coordinate (x, y), we 1000y normalize it to a canonical location as (⌊ 1000x W ⌋, ⌊ H ⌋), and we use this map to define location field where each primitive p is assigned to a location cp . Note that, when Sp is a region, we take its center and normalize it this way. For each p ∈ P, P2 emits a structured item Ip = (p, cp , rp , bp ),
(1)
where p is the primitive identifier, cp is the normalized spatial coordinate, rp is the reading from the modality data on Sp , and bp is an optional label. P2 can include a sparse set of symbolic triples T denoting relations between the primitives, defined as (pa , π, pb ) wherein pa , pb ∈ P, and π is the predicate name (e.g., darker than, adjacent to, etc.). These relations are generated by comparing item statistics for candidate pairs. We serialize P2 as a YAML-like text block summarizing the visual modality in a language-first format. The schema is invariant across modalities, the only changes are: the construction of rp , if bp is included, and whether relations are emitted. A general algorithm is given in Algorithm 1. Unless T is explicitly mentioned, it should be assumed empty (i.e. T = ∅).
3.2. Modality Instantiations We instantiate the above framework for each visual modality dependent on the task. 3.2.1. Depth An expert depth estimation tool produces a scalar field D : Ω → [0, 1] where larger values indicate proximity (nearer). We then partition this depth field into a regular P × P grid. This yields K disjoint grid cells {S1 , S2 , ..., SK } and Sp ⊂ Ω indexed in row-major order. Let cp ∈ [0, 1000]2 denote the normalized center of the cell Sp , then for each grid cell Sp , the P2 read-out, rp , is computed as rp =
min D(x, y), max D(x, y) .
(x,y)∈Sp
(x,y)∈Sp
(2)
In other words, for every patch we store the minimum and maximum value observed in that spatial support. For each (a, b) ∈ N where N is a 4-neighborhood adjacency defined on the same partition {S1 , ..., SK }, then depth instantiated P2 emits relations as ( (a, in-front of, b), if µa > µb + τ t(a,b) = , (3) (b, in-front of, a), if µb > µa + τ
Algorithm 1 Perception Program (P2 ) Construction
3.2.4. Jigsaw
Require: Data D, (W, H) Ensure: P2 = (P, {Ip }, T ) 1: P ← ExtractPrimitives(D) 2: for each p ∈ P do 3: cp ← ExtractCoordinates(Sp ) 4: rp ← ExtractReadOut(Sp , D) 5: bp ← ExtractLabel(p, D) 6: Ip ← (p, cp , rp , bp ) 7: end for 8: R ← {rp | p ∈ P} // collect all primitive read-outs 9: T ← ExtractRelations(P, R) 2 10: return P = (P, {Ip }, T )
The jigsaw problem involves an image with a missing piece and a set of candidate images that may complete it. In our setting, the missing region corresponds to the lowerright corner of the source image, and we are given two candidate pieces, denoted i ∈ {A, B}. Each primitive in this configuration represents a combination between a candidate piece and one of its relevant edges, where edges are defined with respect to the candidate image itself (hence the names left and top). Thus, the set of primitives is
P where µp = |S1p | (x,y)∈Sp D(x, y) which is the average depth of grid cell Sp and τ > 0 is a small margin that suppresses relations caused by small differences. 3.2.2. Optical Flow An expert optical flow estimation tool produces a vector field F : Ω → R2 on the same image domain Ω = {0, ..., W − 1}×{0, ..., H −1} with F (x, y) = (u(x, y), v(x, y)) where u and v denote the horizontal and vertical components, respectively. Without loss of generality, we use horizontal flow. We then partition the field into a regular P × P grid. This yields K disjoint grid cells {S1 , S2 , ..., SK } and Sp ⊂ Ω indexed in row-major order. Let cp ∈ [0, 1000]2 denote the normalized center of the cell Sp , and ūp denote P the mean horizontal component computed as ūp = 1 2 (x,y)∈Sp u(x, y), then for each grid cell Sp , the P |Sp | read-out, rp , is computed as ( ‘left’, rp = ‘right’,
if ūp < 0, if ūp ≥ 0.
P = {(left, A), (top, A), (left, B), (top, B)}.
(5)
For each primitive p = (b, i), the coordinates cp = (x0 , y0 , x1 , y1 ) denote the location of the edge in the coordinate space of candidate i, normalized to the range [0, 1000]4 . The read-out rp is defined as the average of structural, edge, and color similarity scores between the border strip close to the missing region in the reference image and the corresponding border strip of the candidate, and rp ∈ [0, 1]. 3.2.5. Object Detection An object detection tool provides a set of detections for an input image with each detection consisting of category label, confidence score ∈ [0, 1], and a bounding box localizing the object. We take the output from the detector as-is and treat each detection as one primitive p ∈ P. Given an image size (W, H), we normalize the coordinates to the common [0, 1000]2 range, following similar procedure as described earlier, and, with slight abuse of notation, set this at the location field in P2 for that primitive, i.e., cp = (x0 , y0 , x1 , y1 ). The read-out is the confidence score, while bp is the label of the object (category). 3.2.6. Semantic Correspondence
(4)
3.2.3. Visual Correspondence Visual correspondence problems have two views of the same scene with some varying condition (e.g. light or camera position) and the task is to find matching points between them. A correspondence tool produces multiple point-to-point matches between a reference image and a target image. Let the reference image have a spatial domain Ω1 = {0, ..., W1 − 1} × {0, ..., H1 − 1}, and the target image have domain Ω2 = {0, ..., W2 − 1} × {0, ..., H2 −1}. The tool outputs a set of N correspondences (1) (1) (2) (2) (1) (1) {((xi , yi ), (xi , yi ))}N i=1 , where (xi , yi ) ∈ Ω1 is (2) (2) a keypoint in the reference image and (xi , yi ) ∈ Ω2 is a matched keypoint in the target image. We do not form a grid in this modality. For each match we normalize both the reference and target locations to [0, 1000]2 , then set the former as ci and the latter as ri .
The semantic correspondence problem seeks to identify matching points between two related images based on their visual or geometric similarity. An expert tool suited for this task is a feature extractor, which takes a pair of points, one from each image, and computes a similarity score between their corresponding features. In our case, we consider one point in the source image and four candidate points in the target image, denoted A, B, C, and D. Accordingly, we define the primitive set as, P = {A, B, C, D}.
(6)
Given the discrete nature of this modality, we do not form a grid. The coordinate cp of each primitive represents the normalized pixel location of the corresponding candidate point in the target image, and the read-out rp is the similarity score produced by the expert tool. We do not include an optional label bp since the candidate identifier itself serves that role.
Multi-View Reasoning
Visual Correspondence
Object Localization
Semantic Correspondence
Relative Depth
Jigsaw
Figure 3. Perception Program Instantiations. Top: Tool outputs. Bottom: P2 instantiations of those respective tools.
4. Evaluation Setup
4.2. MLLMs Evaluated
We posit that Perception Program (P2 ) lets MLLMs read visual modalities. To test this, we consider BLINK benchmark [8], a suite of 14 perception-focused tasks. We concentrate evaluation on six sub-tasks from the benchmark where additional modalities are especially informative. We benchmark multiple reasoning MLLMs with and without P2 , and for each task, specify the modality-aware tool used to facilitate P2 construction.
We evaluate a representative mix of frontier and opensource/open-weight MLLMs: GPT-5 Mini (2025-0807) [15], Gemini 2.5 Pro [3], InternVL3.5-2B and 4B [25], and Qwen3VL-4B [22], and report accuracy on the validation split of BLINK following the original work [8]. For each of the MLLM, we use its reasoning variant and evaluate three settings: (i) Standard, wherein the model is queried as is with only the images and question, (ii) Tool, where the model additionally receives the raw tool output as an auxiliary input (simulating a tool-calling pipeline), and (iii) P2 , wherein the raw tool output is first converted into a Perception Program and then is supplied, alongside the images and question, to the model. Notably, tools are as outlined in Sec. 4.1, the main difference between settings (ii) and (iii) being that the former provides the information directly as pixels, whereas the latter processes and digests it into the P2 template. For jigsaw, we follow prior work [10, 29] and choose the raw tool to be the two images corresponding to trial-and-error of the two candidates (i.e. tentative images with bottom right corners overlaid with the candidate completions), causing the MLLM’s task to be simply the recognition of global image consistency, see Fig. 3 for tools and their P2 instantiations.
4.1. Tasks & Tools We evaluate six BLINK sub-tasks, namely multi-view reasoning, relative depth, visual correspondence, jigsaw, semantic correspondence, and object localization. We exclude datasets centered on IQ testing or commonsense compositional reasoning, as they do not directly assess visual perception. Instead of BLINK’s standard relative depth task, which considers pairwise point comparisons, we adopt HardBLINK, the harder variant introduced in Bigverdi et al. [1]. The study introduces three increasingly challenging settings with three, four, and five comparative points, respectively. Each task is paired with an off-the-shelf modality/tool that instantiates the P2 . Concretely, HardBLINK (relative depth) uses estimated monocular depth from DepthAnything [28], multi-view reasoning uses optical flow estimated with RAFT [23], visual correspondence uses dense feature matching with LoFTR [18], jigsaw uses a mix of Structural Similarity Index Measure (SSIM) [26], HSV-hist χ2 and gradient-based normalized cross-correlation (NCC) for structural alignment, color compatibility and edge continuity respectively, semantic correspondence uses feature similarity computed with DIFT [20], and object localization uses openvocabulary object detection computed with LLMDet [7]. BLINK tasks have multiple alternative options (points) overlaid on the images. We provide their coordinates to the models as an additional P2 .
4.3. Baselines We also benchmark against five strands of prior work aimed at strengthening perceptual understanding in MLLMs: (i) chain-of-thought methods that reason over tool outputs, e.g., LATTE [14], Thyme [33], Aurora [1], and Mirage [29]; (ii) architectures with dedicated perception-oriented modules, e.g., TULIP [21] and PerceptionLM [2]; (iii) data-centric pipelines that improve supervision and training signals, e.g., Zebra-CoT [11]; (iv) agentic frameworks that leverage tools during inference, e.g., Visual Sketchpad [10], MMFactory [4]; and (v) reinforcement-learning based reasoning pipelines, e.g., ReVPT [35], ViGoRL [17], and OVR [27].
Human [8] Thyme [33] LATTE [14] Visual Sketchpad (GPT-4o) [10] Visual Sketchpad (GPT-5 Mini)⋆ [10] MMFactory (GPT-4o) [4] TULIP (Llama-3.2-11B) [21] PerceptionLM-8B [2] Zebra-CoT (Anole 7B) [11] OVR-7B [27] Prior State-of-the-Art
Multi-View Reasoning
HardBLINK Depth
Visual Correspondence
Jigsaw
Object Localization
Semantic Correspondence
92.48 51.88 48.87 45.60 47.37 60.20 44.96 55.64 21.02 46.62 60.20 ( ) [4]
N/A 35.21 N/A N/A 63.71 N/A N/A 51.08 N/A 25.27 61.56 ( ) [35]
99.42 63.95 29.65 80.80 91.86 85.50 48.97 43.60 26.16 32.56 85.50 ( ) [4]
99.00 58.00 75.33 70.70 79.33 75.30 57.26 40.00 39.33 53.33 88.00 ( ) [29]
98.00 48.36 47.54 65.40 67.21 59.00 60.01 63.11 45.90 51.64 65.40 ( ) [10]
96.07 39.57 33.09 58.30 60.43 58.30 29.61 38.13 17.99 24.50 58.30 ( ) [4, 10]
GPT-5 Mini [15] (Zero-Shot)
Standard Raw Tool P2 (Ours)
41.35 45.11 86.47
52.42 65.05 81.45
76.74 75.58 94.19
76.00 66.00 91.33
58.20 59.02 93.44
53.24 53.24 64.03
Gemini 2.5 Pro [3] (Zero-Shot)
Standard Raw Tool P2 (Ours)
48.12 33.83 63.91
57.79 61.83 77.68
91.86 91.86 96.51
78.00 88.00 96.67
70.49 57.38 77.05
64.03 66.19 74.10
Instruct Standard Raw Tool P2 (Ours)
39.84 45.90 67.66 93.98
41.66 47.04 42.40 61.02
80.23 86.62 76.16 88.37
79.33 64.67 60.66 86.67
54.10 54.92 45.90 85.25
59.71 61.87 62.58 64.03
InternVL3.5-2B [25] (One-Shot)
Standard Raw Tool P2 (Ours)
49.62 56.39 69.17
29.57 25.00 32.25
34.88 41.86 39.53
61.33 62.00 83.33
47.54 50.00 55.74
33.81 34.53 46.04
InternVL3.5-4B [25] (One-Shot)
Standard Raw Tool P2 (Ours)
45.86 75.18 94.73
40.59 37.90 69.89
65.69 64.53 86.04
80.00 66.67 90.67
56.56 54.92 90.16
45.32 44.60 62.59
Qwen3VL-4B [22] (One-Shot)
Table 1. Results on BLINK [8]. Accuracy (%) on six perception-centric sub-tasks. We report three settings per model, Standard (image+question only), Raw Tool (tool output as auxiliary input), and P2 (our training-free Perception Program instantiated from same tool output). Further, we also list representative prior tool-use or perception-oriented methods, plus previous state-of-the-art for each sub-task. Bold indicates best score within each model block. Even with the much smaller InternVL3.5-4B and Qwen3VL-4B MLLMs than prior work, P2 yields large gains setting new state-of-the-art results. ⋆ indicates we ran with GPT-5 Mini as LLM and official codebase. training/fine-tuning, and indicates no parameter updates necessary.
5. Results Tables 1 and 2 (summary in Fig. 4) summarize performance across six BLINK [8] sub-tasks. Across all models and tasks, converting raw tool outputs into language-native Perception Program (P2 ) consistently and often substantially outperforms both Standard (image+question only) and Raw Tool (raw tool as auxiliary input) settings. The gap between Raw Tool and P2 highlights that merely appending tool outputs can be neutral or even harmful (e.g., Gemini 2.5 Pro on multi-view reasoning reduces by 14.29%). On stronger models, P2 sets new state-of-the-art results on every task we consider. With GPT-5 Mini, P2 surpasses prior best results by a large margin, including performing better than Visual Sketchpad [10] with GPT-5 Mini backbone. With the same GPT-5 Mini backbone, P2 also uses significantly less average tokens per sample than Visual Sketchpad [10], see Fig. 7 (in appendix). Further, Gemini 2.5 Pro shows similar trends, often making even better use of P2 , e.g., on semantic correspondence, despite the
denotes
standard Gemini 2.5 Pro already performing similarly to P2 -equipped GPT-5 Mini, its P2 performance improves by 10% rather than saturating. Smaller models benefit as well. InternVL3.5-2B [25] sees sturdy improvements from Standard to P2 . While InternVL3.5-2B is limited in general, its 4B variant matches performance of base GPT-5 Mini and Gemini 2.5 Pro when coupled with P2 . Qwen3VL-4B (reasoning variant) observes gains markedly under P2 . We also compare P2 to several methods that reason over tool outputs. For multi-view reasoning, visual and semantic correspondence tasks, the strongest prior tool method is MMFactory [4] (60.20, 85.50, 58.30, respectively). Even though it uses GPT-4o as the LLM, both InternVL3.54B and Qwen3VL-4B outperform it when coupled with P2 . Crucially, several baselines that reason over tools outputs (Thyme [33], LATTE [14]) still trail the much smaller InternVL3.5-4B and Qwen3VL-4B coupled with P2 , see Fig. 4. Overall, our results validate our premise that P2 turns the same tool signal into a representation that
Methods
Visual Correspondence
Jigsaw
Semantic Correspondence
Mirage (Qwen2.5VL-7B) [29] Mirage (Qwen2.5VL-3B) [29] Zhang et al. [32] (Qwen2.5VL-7B) ViGoRL [17] GPT-4 Turbo Visual Sketchpad [10] (GPT-4 Turbo) LLaVA-13B [12] MMFactory (LLaVA-13B) [4]
N/A N/A 69.77 N/A 48.80 52.30 29.10 34.30
88.00 85.00 74.67 56.00 64.70 68.50 58.0 64.00
N/A N/A N/A N/A 30.90 42.40 32.40 34.50
InternVL3.5-2B InternVL3.5-2B+P2
34.88 39.53
61.33 83.33
33.81 46.04
InternVL3.5-4B InternVL3.5-4B+P2
65.69 86.04
80.00 90.67
45.32 62.59
Qwen3VL-4B Qwen3VL-4B+P2
86.62 88.37
64.67 86.67
61.87 64.03
Table 2. Additional BLINK Results. Comparison of task-specific methods, i.e., methods that do not report on entire BLINK benchmark, on three sub-tasks. denotes training/fine-tuning, and indicates no parameter updates necessary.
-24.6
LATTE
-23.3
TULIP
-19.1
Thyme -10.4
GPT-5 Mini
-7.3
VS (GPT-4o)
-3.8 -2.2
MMFactory (GPT-4o) VS (GPT-5 Mini)
-1.0
Gemini 2.5 Pro
Gemini 2.5 Pro (P²)
+10.2
QwenVL3-4B (P²)
+12.2
InternVL3.5-4B (P²)
+13.4
GPT-5 Mini (P²)
MLLMs can actually read, delivering consistent, large, and architecture-agnostic gains without any training or modifications to the underlying MLLM.
5.1. Analysis We first study the quality of visual interpretation of current MLLMs and then investigate the performance gain of plugging P2 into existing frameworks.
20
15
10
5
+14.4
0
5
10
Figure 4. Mean ∆ vs. prior SOTA across BLINK. Bars show average accuracy improvement (percentage points) of each method over task-wise (except HardBLINK) prior state-of-the-art (at point zero; see Tab. 1). Positive values indicate gains over prior SOTA; negative values indicate regressions. Numeric ∆ are written inside/beyond the bars along with their method names. VS denotes Visual Sketchpad.
5.1.1. Quality of Visual Interpretation The main goal of this study is twofold: to obtain a more holistic picture of how much information is lost when tasking the MLLM with fine-grained visual interpretation, and to see whether latent reasoning abilities emerge when the MLLM is directed to spell out its own interpretation of the image. For the former goal, we note that, while allowing for efficient testing of the model’s ability, the coarse granularity of multiple-choice question answering leads to less informative insights. For the latter, we take inspiration from chain-of-thought prompting and study whether that extends to finer-grained vision tasks. Relative Depth. We provide GPT-5 Mini with a set of instructions followed by five image-P2 pairs to exemplify the conversion, then provide an incomplete P2 with the depth range redacted and ask the model to write it. Figure 2 illustrates one such reconstruction. The left panel of Fig. 5 illustrates how the relative ordering (i.e. the model’s notion of what is the closest, second closest, etc.) diminishes with higher grid size. For each granularity, we plot the distribution of per-sample Kendall τ between the ground-truth depth map and its reconstruction on HardBLINK. This quantity illustrates the ranking agreement between the two variants: values close to 1 mean perfect agreement, whereas values close to zero imply the grid orderings are uncorrelated. We see that the reconstruction has reasonable values for the 3×3 grid, but its information gets quickly corrupted on finer grids. The right panel shows evaluation of these reconstructed
P2 on HardBLINK-5 [1], compared to using original P2 . The reconstructed P2 fails to be useful: GPT-based performance remains roughly flat around 50% across all grid sizes, whereas P2 accuracy improves with finer grids, creating a large gap at 16 × 16. Visual Correspondence. Given two images with corresponding points, we study how well the model can follow a line connecting them. We provide the original image pair, and a variant with lines connecting the points (see Fig. 3), along with a few examples and partially redacted P2 with ‘r’ fields (i.e. final image coordinates) masked out. The LLM task is to fill them. Problem difficulty is increased with higher displacement between matching points, otherwise a model that neglects visual interpretation and simply copies ‘c’ field (i.e. initial image coordinates) can attain reasonable reconstructions. In Figure 6, the left panel plots reconstruction error versus true displacement. Each hexagon aggregates matches, with color indicating its count. Bins forming a narrow band near the x-axis would be indicative of good reconstruction performance. Instead, the errors are large and form a pronounced diagonal structure. Points along it indicate that the MLLM simply copied the input it was given* . In the right panel, we compare the performance of GT and reconstructed P2 on *We note that the BLINK visual correspondence has a prevalence of images with low camera movement, see supplementary material for more details
60
100
0.0 3x3
10x10
16x16
0
3x3
10x10
16x16
Figure 5. GPT-5 Depth Modality Analysis. Left: Kendall’s Tau (y-axis) between ground-truth and GPT-5 Mini reconstructed P2 decreases as the grid is refined (x-axis). Right: HardBLINK-5 accuracy (y-axis) using GPT-5 Mini’s reconstructions (GPT Recon.) across grids (x-axis).
the BLINK task. We additionally contrast them with two oracles: a naive algorithm that simply considers Euclidean distance between the reference and alternatives and an oracle procedure that follows the correspondences perfectly (see appendix). In accordance to our hypothesis, the reconstructed performance is worse than the naive baseline, whereas the correct P2 makes performance competitive with the oracle. 5.1.2. Plug-and-Play Perception Program In Tab. 3, we study the performance gain Visual Sketchpad observes if we replace some tools with their P2 variants. In particular, we post-process outputs of DepthAnything [28] and GroundingDINO [13], in addition to providing a tool that gives the coordinates of points A, B, C, D, E upon request. The table shows steady improvement on both tasks, with GPT-5 Mini as the base LLM. Interestingly, the way each model reasons with P2 output is qualitatively diverse: Visual Sketchpad writes and executes code to process P2 , whereas our variants from Tab. 1 interpret it directly.
5.2. Design Discussions & Limitations While P2 enables MLLMs to read visual modalities, here, we also discuss the limitations thereof. Scope. Notably, we evaluate six BLINK sub-tasks that have clear tool surrogates. Broader settings (e.g., 3D reasoning beyond depth, general VQA, etc.) may require richer or hierarchical programs, which we do not study here. We deliberately scope this work to perception tasks where tools directly benefit because this isolates the contribution of P2 as a representation, enabling fair understanding of whether formatting the same evidence lets MLLMs use it effectively. Tool Pairings. We adopt tool pairings suggested by the original work BLINK [8], and we note many of our tool choices overlap with tools chosen in LATTE [14] and Visual Sketchpad [10]. However, unlike LATTE’s and Visual Sketchpad’s agentic approach to tool selection, our setting does not explore tool sequencing, or composition at inference time. This is merely a design decision, and we reiterate
20 10
10 0
0
10 20 Displacement
30
1
40 20
Oracle
60
PerceptionProgram
100
30
80
GPT Recon.
1000
40 Error
GPT Recon.
20
PerceptionProgram
0.2
GPT Recon.
40
0.4
GPT Recon.
0.6
PerceptionProgram
60
PerceptionProgram
50
0.8
Naive
80
Accuracy (%)
1.0
0
Figure 6. GPT-5 Visual Correspondence Modality Analysis. Left: Plot of LoFTR [18] correspondences, with true displacement between views on the x-axis and GPT 5’s reconstruction error on the y-axis (both as % of image diagonal). Color indicates bin count. The strong diagonal indicates a failure mode where GPT often copies the left-image coordinate into the right-image field. Right: Accuracy (%) across 100 matches. Naive refers to Euclidean distance baseline, while Oracle uses the tool directly.
HardBLINK 3
4
5
Object Localization
71.77 81.45
62.90 83.06
56.45 79.84
60.43 80.33
Methods VS [10] VS + P2
Table 3. Plug-and-Play. Visual Sketchpad (VS) results with P2 .
that P2 is trivially pluggable into any agent-and-tool pipeline (e.g., feeding each tool’s output through its instantiated P2 ). We leave dynamic tool selection and composition to future. Tool Reliability. As with prior methods, P2 conveys whatever evidence the upstream tool produces, i.e., errors propagate into the program. Our method does not attempt to calibrate or reconcile conflicting tools. Nevertheless, we observe that frontier LLMs, such as GPT-5 Mini and Gemini 2.5 Pro, often utilize P2 to narrow down choices (in multiple choice questions) and cross-check evidence from P2 as part of their reasoning before generating the final answer.
6. Conclusion In this work, we proposed Perception Programs (P2 ) that rewrite dense tool outputs into compact, symbolic, languagenative summaries that specify what cue is present and where it is grounded. We show that this training-free, modelagnostic interface consistently improves performance on several perception-centric tasks, across both frontier and open-source MLLMs, outperforming prior agentic, and tooluse methods. We view P2 as a step toward a more principled interface between perception and language, suggesting that better representations of existing tool signals can be as important as, if not more than, additional tools or larger models.
References [1] Mahtab Bigverdi, Zelun Luo, Cheng-Yu Hsieh, Ethan Shen, Dongping Chen, Linda G Shapiro, and Ranjay Krishna. Perception tokens enhance visual reasoning in multimodal language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 3836–3845, 2025. 1, 2, 3, 5, 7, 4 [2] Jang Hyun Cho, Andrea Madotto, Effrosyni Mavroudi, Triantafyllos Afouras, Tushar Nagarajan, Muhammad Maaz, Yale Song, Tengyu Ma, Shuming Hu, Suyog Jain, et al. Perceptionlm: Open-access data and models for detailed visual understanding. arXiv preprint arXiv:2504.13180, 2025. 2, 5, 6 [3] Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025. 5, 6 [4] Wan-Cyuan Fan, Tanzila Rahman, and Leonid Sigal. Mmfactory: A universal solution search engine for vision-language tasks. arXiv preprint arXiv:2412.18072, 2024. 3, 5, 6, 7, 2 [5] Yue Fan, Xuehai He, Diji Yang, Kaizhi Zheng, Ching-Chen Kuo, Yuting Zheng, Sravana Jyothi Narayanaraju, Xinze Guan, and Xin Eric Wang. Grit: Teaching mllms to think with images. arXiv preprint arXiv:2505.15879, 2025. 3 [6] Stephanie Fu, Tyler Bonnen, Devin Guillory, and Trevor Darrell. Hidden in plain sight: Vlms overlook their visual representations, 2025. 1, 2, 3 [7] Shenghao Fu, Qize Yang, Qijie Mo, Junkai Yan, Xihan Wei, Jingke Meng, Xiaohua Xie, and Wei-Shi Zheng. Llmdet: Learning strong open-vocabulary object detectors under the supervision of large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 14987–14997, 2025. 5 [8] Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. In European Conference on Computer Vision, pages 148–166. Springer, 2024. 1, 2, 5, 6, 8 [9] Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14953–14962, 2023. 2, 3 [10] Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, and Ranjay Krishna. Visual sketchpad: Sketching as a visual chain of thought for multimodal language models. Advances in Neural Information Processing Systems, 37:139348–139379, 2024. 3, 5, 6, 7, 8, 1, 2, 4 [11] Ang Li, Charles Wang, Deqing Fu, Kaiyu Yue, Zikui Cai, Wang Bill Zhu, Ollie Liu, Peng Guo, Willie Neiswanger, Furong Huang, et al. Zebra-cot: A dataset for interleaved vision language reasoning. arXiv preprint arXiv:2507.16746, 2025. 5, 6, 2 [12] Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun
Zhu, et al. Llava-plus: Learning to use tools for creating multimodal agents. In European conference on computer vision, pages 126–142. Springer, 2024. 7 [13] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision, pages 38–55. Springer, 2024. 8 [14] Zixian Ma, Jianguo Zhang, Zhiwei Liu, Jieyu Zhang, Juntao Tan, Manli Shu, Juan Carlos Niebles, Shelby Heinecke, Huan Wang, Caiming Xiong, Ranjay Krishna, and Silvio Savarese. Latte: Learning to think with vision specialists. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2025. 2, 3, 5, 6, 8 [15] OpenAI. Gpt-5 system card, 2025. 5, 6, 1 [16] Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. Perception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processing Systems, 36:42748–42761, 2023. 1 [17] Gabriel Sarch, Snigdha Saha, Naitik Khandelwal, Ayush Jain, Michael J Tarr, Aviral Kumar, and Katerina Fragkiadaki. Grounded reinforcement learning for visual reasoning. arXiv preprint arXiv:2505.23678, 2025. 2, 5, 7 [18] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. Loftr: Detector-free local feature matching with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8922–8931, 2021. 5, 8, 4 [19] Dı́dac Surı́s, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11888–11898, 2023. 2, 3 [20] Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. Advances in Neural Information Processing Systems, 36:1363–1389, 2023. 5 [21] Zineng Tang, Long Lian, Seun Eisape, Xudong Wang, Roei Herzig, Adam Yala, Alane Suhr, Trevor Darrell, and David M Chan. Tulip: Contrastive image-text learning with richer vision understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4267–4277, 2025. 2, 5, 6 [22] Qwen Team. Qwen3 technical report, 2025. 5, 6 [23] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In European conference on computer vision, pages 402–419. Springer, 2020. 5 [24] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568–9578, 2024. 3 [25] Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3. 5: Advancing open-source
multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025. 5, 6 [26] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600–612, 2004. 5 [27] Yana Wei, Liang Zhao, Jianjian Sun, Kangheng Lin, Jisheng Yin, Jingcheng Hu, Yinmin Zhang, En Yu, Haoran Lv, Zejia Weng, et al. Open vision reasoner: Transferring linguistic cognitive behavior for visual reasoning. arXiv preprint arXiv:2507.05255, 2025. 5, 6, 2 [28] Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10371–10381, 2024. 5, 8 [29] Zeyuan Yang, Xueyang Yu, Delin Chen, Maohao Shen, and Chuang Gan. Machine mental imagery: Empower multimodal reasoning with latent visual tokens. arXiv preprint arXiv:2506.17218, 2025. 1, 2, 3, 5, 6, 7 [30] Runpeng Yu, Xinyin Ma, and Xinchao Wang. Introducing visual perception token into multimodal large language model. arXiv preprint arXiv:2502.17425, 2025. 2 [31] Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598, 2022. 1 [32] Bob Zhang, Haoran Li, Tao Zhang, Cilin Yan, Jiayin Cai, and Yanbin Hao. Improving the reasoning of multi-image grounding in mllms via reinforcement learning. arXiv preprint arXiv:2507.00748, 2025. 7 [33] Yi-Fan Zhang, Xingyu Lu, Shukang Yin, Chaoyou Fu, Wei Chen, Xiao Hu, Bin Wen, Kaiyu Jiang, Changyi Liu, Tianke Zhang, et al. Thyme: Think beyond images. arXiv preprint arXiv:2508.11630, 2025. 2, 3, 5, 6 [34] Zhehao Zhang, Ryan Rossi, Tong Yu, Franck Dernoncourt, Ruiyi Zhang, Jiuxiang Gu, Sungchul Kim, Xiang Chen, Zichao Wang, and Nedim Lipka. Vipact: Visual-perception enhancement via specialized vlm agent collaboration and tool-use. arXiv preprint arXiv:2410.16400, 2024. 3 [35] Zetong Zhou, Dongping Chen, Zixian Ma, Zhihan Hu, Mingyang Fu, Sinan Wang, Yao Wan, Zhou Zhao, and Ranjay Krishna. Reinforced visual perception with tools. arXiv preprint arXiv:2509.01656, 2025. 2, 5, 6, 1, 4
Don’t Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programs Supplementary Material HardBLINK
Methods 3
4
5
Avg.
Aurora [1] ReVPT-7B [35] Visual Sketchpad [10] (GPT-5 Mini)
66.90 68.55
60.50 55.65
54.80 60.48
60.73 61.56
71.77
62.90
56.45
63.71
GPT-5 Mini GPT-5 Mini + P2
62.10 82.26
53.23 87.90
41.49 74.19
52.42 81.45
Gemini 2.5 Pro Gemini 2.5 Pro + P2
66.13 77.42
55.65 83.87
45.16 71.77
55.65 77.68
InternVL3.5-2B InternVL3.5-2B+P2
39.52 35.48
28.23 29.03
20.97 32.26
29.57 32.25
InternVL3.5-4B InternVL3.5-4B+P2
55.65 72.58
36.29 70.97
29.84 66.13
40.59 69.89
Qwen3VL-4B Qwen3VL-4B+P2
55.65 73.39
46.77 61.29
38.71 48.39
47.04 61.02
40,300
Table 4. HardBLINK Breakdown. We report accuracy (%) on different sub-tasks in HardBLINK [1]: 3 point, 4 point and 5 point. Each setting is increasingly difficult from the prior, wherein more candidate points are presented to the MLLM.
957
968
912
6,408
13,749
13,793 4,626
13,486 4,495
4,404
2,951
13,512
21,238
23,455
27,550
Visual Sketchpad Perception Program
Figure 7. Average Tokens/Sample. Comparison of Visual Sketchpad (with GPT-5 Mini as LLM) and GPT-5 Mini with P2 on average token per sample across all six sub-tasks. P2 incurs significantly lower token cost.
7. Perception Program Details In this section, we discuss additional details about Perception Programs. Mainly, we provide samples of prompts for both frontier and open-source MLLMs. We also detail in-context (ICL) example that we use to query the open-source MLLMs. Recall that frontier models, GPT-5 Mini and Gemini 2.5 Pro, work as is and do not require any ICL examples. However, for both Qwen3VL and InternVL3.5, we provide a single in-context example as part of the system prompt, see Fig. 8
for an illustrative example for one question in multi-view reasoning task. Note that the P2 rationale as part of in-context sample is taken from GPT-5 Thinking [15]. We prompt GPT-5 Thinking with the question and its P2 and ask it to output a short rationale on how it uses P2 to compute the answer. We include this obtained rationale as an in-context example for open-source MLLMs. This procedure is similar for all the tasks from the BLINK benchmark we consider in this work, along with HardBLINK. Note that we do the same for raw tool setting and provide exhaustive descriptions of the tool in terms of how to use it to get to the answer, see Fig. 9. An important distinction to note is for InternVL3.5 (both 2B and 4B variants), we also include additional instructions. In all problems, we direct it to to not copy the in-context example as is, along with some problem-specific orientations and clarification. For multi-view reasoning, we mention that clockwise and left are used interchangeably. For relative depth, we explain that the comparison of which point is closest is based on depth range and not coordinates. For semantic correspondence, we reiterate to use similarity scores and not the coordinates for comparison. In visual correspondence we emphasize not to directly use euclidean distance between coordinates in different images to conclude which point corresponds to REF. However, these clarifications are not necessary for Qwen3VL. For closed-source frontier LLMs, such as GPT-5 Mini and Gemini 2.5 Pro, we just query with the question along with the tool or P2 and do not provide any example. The frontier LLMs are able to understand how to use P2 on their own.
8. Additional Related Work In this section, we give a non-comprehensive summary of methods from the related work, expanding on some that were briefly mentioned while also introducing additional ones. We additionally note that several prior state-of-the-art BLINK results were obtained by methods that do not rely on tools, which we also include here.
8.1. Tool-Use through Program Synthesis Methods in this category leverage program generation, typically Python, to structure the model’s reasoning process. Instead of reasoning purely in natural language, these approaches produce executable code that coordinates external vision modules to enable compositional and interpretable visual reasoning.
Image Pair
System Prompt + Question <system>...</system> <icl>You're given two frames from a static scene and a <perceptionprogram> block that lists per-patch horizontal motion ("left" or "right") on a 10x10 grid. One example of how to use perception program to solve the problem is given below.
Images: (beginning frame, end frame) Perception: many 'right' patches in upper rows; some 'left' in lower rows Justification: The upper rows overwhelmingly show 'right' motion and overall the grid has a right majority (roughly 60+ vs 40), indicating global motion to the right. Answer: (B)</icl> Now use this information and solve the next question. <question>The images are frames from a video. The first image is from the beginning of the video and the second image is from the end. Is the camera moving left or right when shooting the video? Select between A and B.</question>
<think>Got it, let's try to figure out the camera motion. The problem says the video is shooting a scene, and the camera is moving clockwise (left) or counter-clockwise (right) around the object. The first image is the beginning, second is the end. So we need to see the direction of motion. The perception program gives a grid of patches with their horizontal motion: "left" or "right". The key is to find the global motion. The example in the problem said that if most patches are "right", then global motion is right. Looking at the perception program data: Let's count how many "left" and "right" there are. Let's list all the r values: Looking at the items: The first 33 items (p 1 to 33) are mostly "left". Then p 34: right, p35: right, p36: left, etc. Wait, let's count all 100 patches. Let's check each patch's r. Wait, the grid is 10x10, so 100 patches. Let's count how many are "left" and "right". Let's go through each: p1-10: all left (10) p11-20: all left (10) p2130: all left (10) p31-33: left (3) p34: right (1) p35: right (1) p36: left (1) p37-40: 4 lefts p41-43: 3 lefts p44: right (1) p45: right (1) p46: right (1) p47-50: 4 lefts p51-53: 3 lefts p54: right (1) p55: right (1) p56-60: 5 lefts p61-70: 10 lefts p71-80: 10 lefts p81-90: 10 lefts p91-100: 10 lefts. Wait, let's count the "right" ones: p34, p35, p44, p45, p46, p54, p55. Let's check:p34: right (1) p35: right (2) p44: right (3) p45: right (4) p46: right (5) p54: right (6) p55: right (7) Are there any others? Let's check:p36 is left, p37-40 are left. p47-50: 4 lefts. p56: left. p57-60: 5 lefts. So the "right" patches are 7. The rest are lefts. The perception program's r is the patch's motion. The global motion is the majority direction. Since 93 out of 100 are 'left', the global motion is left.</think><answer>(A)</answer>
Figure 8. Open-Source Prompt with P2 ICL. We present a sample prompt for open-source MLLMs (e.g., Qwen3VL and InternVL3.5). We include a single in-context example describing the use of P2 . Both Qwen3VL and InternVL3.5 reason with the given P2 to compute the correct answer (A) to the question.
• VisProg [9]: A neuro-symbolic approach in which the model uses in-context learning to generate modular Python programs that call vision models and image-processing tools. Demonstrates strong performance in compositional VQA, reasoning over image pairs, object tagging, and language-guided image editing. • ViperGPT [19]: Reduces the burden on large MLLMs by equipping GPT-based models with an API of callable vision-related subroutines. The model generates Python programs executed on images or video, improving visual grounding and compositional question answering, including cases requiring external knowledge. • Thyme [33]: Enhances logical reasoning by enabling models to perform image-level manipulations such as cropping, rotation, and contrast adjustment. Trained via a two-stage pipeline combining supervised fine-tuning and GRPO with adaptive temperature sampling. • MMFactory [4]: Addresses deployment challenges such as performance constraints and computational limits. Proposes a model that composes programmatic solutions from a tool repository while also suggesting metrics and benchmarks, taking user illiteracy into account to improve realworld usability.
8.2. Tool-Use through Chain-of-Thought These methods integrate tool usage directly into the reasoning trajectory of the model, often allowing the system to call vision specialists or perform visual operations as part of its intermediate reasoning steps rather than through offline program synthesis. • LATTE [14]: Introduces 8B vision-language models trained to incorporate outputs from multiple vision specialists as part of a think–act–observe reasoning loop. Supports tasks including object recognition, depth estimation, text extraction, and mathematical operations. • VisualSketchpad [10]: A hybrid method that integrates
code generation and tool usage into chain-of-thought reasoning. Inspired by sketch-based human problem solving and implemented on a GPT backbone. • VigoRL [17]: Highlights the gap between the success of RL in math/coding and its limited impact on visually grounded tasks. Proposes grounding reasoning steps in image regions and enabling zoom operations to focus on visually relevant details. • ReVPT [35]: Uses a GRPO-based RL framework to train multimodal LLMs to reason with visual tools such as detection, zooming, edge analysis, and depth estimation. Shows notable improvements on perception-heavy benchmarks (e.g., KV-Bench, BLINK, MMVP, MMStar), with particularly strong gains for 2B-scale models. • Visual Perception Token [30]: Trains models to emit toolcalling tokens that enable selective invocation of external visual modules. Provides tools for region selection-thenzooming and for supplying enriched vision tokens from an auxiliary vision tower.
8.3. Other Prominent Methods This group includes methods that achieve strong results in multimodal reasoning without relying primarily on explicit tool calls. Many provide alternative training paradigms or new datasets that improve perception or reasoning capabilities. • TULIP [21]: Based on Llama-3.2-11B, this method addresses limitations of CLIP/SigLIP in detailed visual interpretation. Introduces contrastive and reconstruction objectives and provides a drop-in replacement vision tower, yielding improved BLINK performance. • PerceptionLM [2]: Promotes transparency by avoiding reliance on closed-source vision model annotations. Constructs a fully open perception-language model (8B LLM + vision tower) that achieves strong BLINK performance without using tools. • Zebra-CoT [11]: Tackles the scarcity of high-quality sketch/diagram reasoning data by releasing a new interleaved image–text dataset and training the Anole-7B model on it. Instead of using external tools, the model directly generates auxiliary images. • OVR [27]: Argues that prior RL approaches under-scale cognitive behavior training. Proposes a large two-stage RL paradigm achieving strong gains in mathematical reasoning and BLINK tasks. All of the methods discussed above combine to form a wide-range of baselines we compare our proposed P2 to in Tables 1 and 2; also refer to Sec. 4.
9. Additional Experimental Details In Sec. 5.1 we discussed the quality of visual interpretation of current MLLMs. We expand the discussion on on vi-
Image Pair
Tool Output
System Prompt + Question <system>...</system> <icl>You're given two frames from a static scene and an optical flow map between them. Decide the global camera motion: (A) left (B) right Interpret horizontal motion from the flow: rightward (positive u) vs leftward (negative u). You may infer direction from the color-wheel (e.g., warm hues often indicate rightward, cool hues leftward. One example of how to use the optical flow to solve the problem is given below. Images: (beginning frame, end frame, flow) Flow: dominated by warm hues (red/orange/yellow) across the scene, indicating positive horizontal flow (rightward). Justification: The optical-flow map shows global rightward motion (positive u), so the camera moved to the right. Answer: (B)</icl> Now use this information and solve the next question.<question>...</question>
<think>So, let's try to figure this out. The problem is about determining the global camera motion direction based on the optical flow between the two images. The key is to look at the color-wheel in the optical flow map. Warm hues (like red, orange, yellow) usually indicate rightward motion (positive u), and cool hues (like blue, purple) indicate leftward motion (negative u). First, let's recall the example given: if the optical flow is dominated by warm hues, it's rightward (so answer B). The optical flow image has a mix, but let's focus on the direction. Wait, the optical flow between the two frames (image 1 and image 2) shows the movement. The two images are the beginning and end. So we need to see the direction of motion. Let's imagine the first image (image 1) and second image (image 2). The optical flow map's color indicates the direction. Wait, in the optical flow, positive u is rightward. So if the flow is mostly warm (red/orange), it's right. Let's check the provided optical flow image. The optical flow is a color map where each color represents the direction. The optical flow image here: there's a sort of blueish hue? Wait, no, the description says optical flow is given as an image. Let's think about standard color wheels: red is right, blue is left...Let's assume the optical flow color wheel: red is right, blue is left. If the optical flow is mostly red, then right. Wait, in the provided optical flow image (the third image), the main part (the area where the flow is) might have a color. Let's look: the optical flow has a light color (maybe orange or red) in the center, and blue around. Wait, but maybe the direction is right. Wait, the problem says the camera is moving clockwise (left) or counter-clockwise (right). Wait, no: the problem says the camera is either moving clockwise (left) or counter-clockwise (right) around the object. Wait, no: the options are (A) left, (B) right...Wait, the optical flow image given here (the third image in the problem) is a color map. Let's look at it: the background is blue , and there's a central region that is lighter (maybe orange or yellow). Wait, the central part is a light color, which might be warm. So if the flow is warm, that's right (B). Alternatively, maybe the direction can be inferred by the overall mean horizontal flow. If the majority of the flow vectors are pointing right, then positive u, rightward. Another approach: look at the position of the objects. In image 1, the cup is on the box. In image 2, the cup's position: but maybe the camera is moving. Wait, the optical flow shows the movement between the frames. Let's see: the optical flow image. If the flow vectors are pointing right (positive u), then the camera moved right (B). Wait, in the example, warm hues mean right. So if the optical flow in the problem is dominated by warm hues (red/orange), then it's right (B). Let's assume that's the case here. Wait, but maybe the optical flow image in the problem is not the standard. Let's look at the provided optical flow image. The image shows a blue gradient with a lighter (orange) area in the center. Therefore, the flow is positive (right), so the camera moved right (B).</think><answer>(B)</answer>
Figure 9. Open-Source Prompt with Tool ICL. We present a sample prompt for open-source MLLMs (e.g., Qwen3VL and InternVL3.5). We include a single in-context example describing the use of optical flow as tool output. Note how the example clearly illustrates that blue hues indicate left while warm hues indicate right motion, the MLLM (Qwen3VL in this example) concludes the same that flow is dominated by blue hues, yet gives the wrong answer (B). Note that MLLM also uses a lot more tokens than its P2 counterpart (exhausting almost the entire 8192 token budget), we use . . . for brevity purposes in this illustrative figure.
Distribution of Matches
sual correspondence task and describe the two baselines we included, namely Naive and Oracle, also see Fig. 6.
Y coordinate
When evaluating the reconstructed P2 results, we considered the naive Euclidean and oracle baselines. We describe their setup as follows. Naive. It receives ground-truth coordinates corresponding to the BLINK alternatives: REF in the reference image and A, B, C, D, E in the target image. It simply gives the answer as the point whose coordinate is closest to REF in the normalized coordinate space. This method completely disregards the visual content of the image and is therefore unsuitable to solve the task of visual correspondence. Its performance of 85% indicates that most pairs of images indeed have low camera movement, which we later confirm in Sec. 9.2. Oracle. An implementation of correct usage of the P2 to navigate through correspondences is what we term as an oracle. Concretely, we first store the reference (REF) point coordinates. We then scan the correspondence P2 to find all the candidate points and read their ‘c’ coordinates. For
40
Samples (%)
9.1. Additional Details on Visual Correspondence
Geometric Interpretation
30 20 10
20% 5%
0
0
10
20
30
40
50
X coordinate
Displacement (% of diagonal)
Figure 10. Correspondence Distribution. Illustration of distribution of correspondence markers in the visual correspondence task from BLINK validation set.
each candidate, we compute its Euclidean distance to the reference point and select the neighbor with the smallest distance. We then take this neighbor’s ‘r’ coordinate as the mapped position in the second image. Finally, we compare this mapped location with the coordinates of alternatives A, B, C, D, E, and choose the point whose coordinates are closest in Euclidean distance as the correspondence of the original reference point.
9.2. Distribution of Displacements We here plot the distribution of visual correspondences in the BLINK validation set. The left pane illustrates the histogram and density of LoFTR [18] displacements across the whole dataset as a percentage of the diagonal. To give the reader a rough visual reference of the displacement ranges, we illustrate regions closer than 5% (blue) and 20% (green) of the diagonal, considering the normalized coordinate space. Colors between panes correspond to matching regions. We can see that the majority of displacements are closer than 5% of image diagonal, further evidencing the fact that this dataset is biased towards low displacement between images.
9.3. Breakdown on HardBLINK In Tab. 4, we present results on each sub-task, 3-, 4-, and 5-point, in HardBLINK benchmark introduced in Bigverdi et al. [1] to complement the results presented in Tab. 1 (the HardBLINK performance reported there is the average of these sub-tasks). Across all three HardBLINK settings, we observe a consistent trend: performance drops as the number of candidate points increases, but P2 substantially narrows this gap. Specialized baselines such as Aurora [1], ReVPT [35], and Visual Sketchpad (GPT-5 Mini) [10] achieve average accuracies between 60.73% and 63.71%, with modest degradation from 3-point to 5-point tasks. In contrast, raw MLLMs struggle more severely as difficulty increases, i.e., GPT-5 Mini falls from 62.10% on 3-point to 41.49% on 5-point sub-task. Both GPT-5 Mini and Gemini 2.5 Pro immensely benefit from P2 , and even the smaller InternVL3.5-4B and Qwen3VL-4B observe +29.30% and +13.98% increase. Overall, P2 not only lifts all base models, but is particularly effective in the more challenging 4- and 5-point regimes, where raw MLLMs otherwise collapse.
10. LLM Usage Statement In this manuscript, we used several MLLMs as part of our experimental setup and we have described the necessary details in Secs. 4 and 7. Other than that, we also used LLMs (ChatGPT) to help with refining the manuscript in terms of fixing grammatical errors in writing and with plotting codes for various figures. The authors did not use any LLM in any part of ideation, experimental design, analysis of results, and implementation of core methodology.