Gen-VCoT: Generative Visual Chain-of-Thought Reasoning via Diffusion-Based RGB Intermediate Representations Zhiqiang Zhou, Xu Ling, Junliang Dai Hunan Chemical Industry Vocational and Technical College, Zhuzhou, Hunan 412000, China [email protected]
arXiv:2606.16783v1 [cs.CV] 15 Jun 2026
June 16, 2026 into intermediate reasoning steps expressed in natural language. This paradigm has been extended to multimodal setMultimodal large language models (MLLMs) have tings, where multimodal large language models (MLLMs) demonstrated remarkable capabilities in visual reasoning, such as GPT-4V [12], Qwen2-VL [17], and LLaVA [10] yet their reasoning processes primarily rely on text-based perform visual reasoning by generating text-based reasoning chain-of-thought (CoT), lacking explicit and interpretable chains over visual inputs. intermediate visual processing. Existing visual CoT methHowever, text-based CoT reasoning has a fundamental limods either use opaque continuous visual tokens or deitation in visual tasks: language is an indirect medium for pend on external tool invocations, failing to simultanespatial reasoning. When humans reason about visual scenes, ously achieve interpretability, end-to-end trainability, and we naturally sketch, annotate, and mentally manipulate visual dense visual representations. We propose Gen-VCoT, a representations—drawing bounding boxes to isolate objects, generative visual chain-of-thought framework that leverages expert vision models to produce RGB images as highlighting regions of interest, and estimating depth relationvisual reasoning intermediates. Gen-VCoT decomposes ships. Current MLLMs lack this ability to externalize intervisual reasoning into three interpretable stages: (1) vimediate visual reasoning products, relying entirely on textual sual grounding—generating instance segmentation maps descriptions to encode spatial information. via SAM to highlight question-relevant regions; (2) geoRecent work has explored several directions to address this metric reasoning—producing pseudo-colored depth maps gap. COVT [13] introduces continuous visual tokens for intervia Marigold to establish spatial relationships; and (3) semediate reasoning, achieving 3%–16% improvements on vimantic reasoning—an MLLM (Qwen2-VL) integrates the sual benchmarks but producing opaque latent representations original image with generated visual evidence to produce that cannot be directly interpreted by humans. Visual Sketchthe final answer. We further design an adaptive reasonpad [4] enables LLMs to invoke external tools (detection, seging router that dynamically selects the required reasonmentation) and draw on a “sketchpad,” but relies on sparse ing depth based on question complexity. Comprehensive evaluations across both complex spatial reasoning geometric primitives (lines, boxes, arrows) rather than dense scenes and CLEVR-style benchmarks reveal a nuanced pixel-level visual representations. Neither approach simultapicture: Gen-VCoT improves spatial relationship (+25%) neously satisfies the requirements of interpretability (humanand depth perception (+50%) questions, but can degrade readable intermediate steps), end-to-end processing (no exterperformance on simple factual queries where intermenal tool dependencies), and dense visual representation (pixeldiates introduce noise. A three-way comparison with level segmentation and depth). text-only chain-of-thought (providing structured text deWe propose Gen-VCoT (Generative Visual Chain-ofscriptions instead of visual images) shows that text CoT Thought), a framework that leverages expert vision models to achieves 91.2% on CLEVR vs. 85.0% baseline and 62.5% produce RGB images as visual reasoning intermediates. Our Gen-VCoT, indicating that the optimal intermediate repkey insight is inspired by Vision Banana [3], which demonresentation is task-dependent. This finding motivates our strates that image generation models serve as universal vision adaptive reasoning router, which selectively applies intermediate steps only when beneficial. Our framework is learners—generation and understanding are two sides of the the first to systematically use expert-generated RGB imsame coin. Extending this insight, we hypothesize that the ages as visual reasoning intermediates, establishing a new process of generating a segmentation map or depth map from paradigm for interpretable multimodal reasoning. an image inherently encodes the visual understanding needed for reasoning about that scene. Gen-VCoT operates as a three-stage pipeline (Figure 1): 1 Introduction 1. Visual Grounding (Where): The Segment Anything Large language models (LLMs) have demonstrated remarkModel (SAM) [9] generates instance segmentation maps able reasoning capabilities through chain-of-thought (CoT) using a grid of point prompts, color-coding objects to highprompting [18], where complex problems are decomposed Abstract
1
to draw on a visual canvas using external tools, achieving 12.7% improvement on math tasks and 8.6% on visual tasks, but relies on sparse geometric primitives (lines, bounding boxes, markers) rather than dense pixel-level representations. VChain [11] applies visual chain-of-thought to video generation through causal keyframe reasoning. The first Visual CoT survey [15] establishes a taxonomy distinguishing text-based, continuous-token-based, tool-based, and generation-based visual reasoning—Gen-VCoT falls into the last category, which the survey identifies as the most promising yet underexplored Figure 1: Gen-VCoT three-stage pipeline. Stage 1 (Where): direction. SAM generates instance segmentation maps. Stage 2 (How): Marigold produces pseudo-colored depth maps. Stage 3 2.2 Diffusion Models for Visual Understanding (What): Qwen2-VL integrates all visual evidence to answer Recent work has demonstrated that diffusion models can serve the question. as general-purpose vision learners. InstructCV [5] fine-tunes Stable Diffusion with instruction tuning to perform segmentalight question-relevant regions and establish object identity. tion, depth estimation, and classification as image generation 2. Geometric Reasoning (How): Marigold [8], a diffusion- tasks, proving that the text-to-image paradigm can be repurbased depth estimator, produces pseudo-colored depth posed for visual understanding. Vision Banana [3] (Google maps using a rainbow colormap, establishing spatial rela- DeepMind, 2026) presents the most compelling evidence: by treating all visual task outputs as RGB images, a single diftionships between objects (red=near, violet=far). 3. Semantic Reasoning (What): Qwen2-VL [17] integrates fusion model achieves state-of-the-art results across semantic the original image with the generated visual evidence to segmentation (69.9 mIoU on Cityscapes vs. 65.2 for SAM 3), produce the final answer, leveraging the structured interme- instance segmentation (47.5 cgF1 vs. 24.6 for OWLv2), metric depth estimation (0.929 δ1 vs. 0.918 for Depth Anything V3), diate representations for more accurate spatial reasoning. and surface normal estimation (15.5°mean angle vs. 16.6°for Our contributions are: 1. We propose Gen-VCoT, the first framework to system- Lotus-2). OmniGen [19] proposes a unified image generaatically use expert vision models to generate RGB im- tion framework supporting multiple visual tasks. These works ages as visual reasoning intermediates, establishing a new establish the “generation as understanding” paradigm, but focus on single-step generation rather than multi-step reasoning paradigm for interpretable multimodal reasoning. 2. We design an adaptive reasoning router that dynamically chains. Gen-VCoT extends this paradigm by chaining multiselects the required reasoning depth based on question ple generation steps into a coherent reasoning pipeline. complexity, enabling efficient inference without sacrificing quality. 2.3 Segmentation and Depth Estimation 3. We conduct comprehensive evaluations across three diverse scenes with 19 questions spanning 6 categories, The Segment Anything Model (SAM) [9] introduced promptdemonstrating that Gen-VCoT achieves 78.9% accuracy vs. able segmentation with strong zero-shot generalization across SAM 2 [14] extends this to video 68.4% baseline, with particularly strong improvements on diverse domains. with memory-based architecture. For depth estimation, spatial (+25%) and depth (+50%) reasoning. Marigold [8] leverages pretrained diffusion models for monoc4. We perform ablation studies confirming that both segmenular depth estimation, achieving remarkable accuracy with tation and depth intermediates provide complementary inminimal fine-tuning by repurposing the generative prior of formation, with the full pipeline outperforming any partial Stable Diffusion. Depth Anything V2 [20] provides dense configuration. depth maps at scale through extensive data augmentation. Our framework directly leverages these expert models as visual reasoning modules, benefiting from their strong zero-shot ca2 Related Work pabilities without requiring additional training. 2.1 Visual Chain-of-Thought Reasoning 2.4 Multimodal Large Language Models Chain-of-thought prompting [18] has been extended to multimodal settings through several approaches. COVT [13] in- The rapid development of MLLMs has enabled increasingly troduces continuous visual tokens for intermediate reason- sophisticated visual reasoning. GPT-4V [12] demonstrated ing steps, achieving significant improvements by distilling human-level performance on many visual understanding tasks. knowledge from expert models (depth, segmentation, edge Qwen2-VL [17] introduced dynamic resolution processing detection) into 20 continuous tokens. However, these to- and native visual token compression. LLaVA [10] pioneered kens are opaque latent vectors that cannot be directly visu- the visual instruction tuning paradigm. However, these modalized or interpreted. Visual Sketchpad [4] enables LLMs els reason primarily through text-based chains, lacking exGen-VCoT: Three-Stage Visual Reasoning Pipeline
Stage 1: Where
Input Image I
Segmentation
SAM ViT-Large
Map
Instance Segmentation
M_seg
8×8 Grid Prompts
Visual Grounding
Stage 2: How
Depth Map
Marigold LCM
M_depth
Depth Estimation
Rainbow Colormap
Near
Far
Geometric Reasoning
Question Q
Stage 3: What
Qwen2-VL-7B-Instruct Multi-Image Reasoning
Answer A
Natural Language
Semantic Reasoning
2
plicit intermediate visual processing. Gen-VCoT addresses Marigold [8], a diffusion-based monocular depth estimator this limitation by providing structured visual evidence as in- that repurposes the generative prior of Stable Diffusion for put to MLLMs. depth prediction. Depth Estimation. Marigold takes the input image and produces a dense depth map D ∈ RH×W through iterative de3 Method noising. We use the LCM (Latent Consistency Model) variant with only 4 inference steps and ensemble 5 predictions for ro3.1 Problem Formulation bustness: Given an image I and a question Q requiring visual reasonK 1 X ing, Gen-VCoT produces an answer A through a sequence of Marigold(I; steps = 4, seedk ) (6) D= K intermediate visual representations. The framework decomk=1 poses the reasoning process into three stages: where K = 5 is the ensemble size. Pseudo-Color Encoding. The depth map is normalized to Mseg = Gseg (I, Pseg (Q)) (1) [0, 1] and mapped to a rainbow colormap where red indicates Mdepth = Gdepth (I, Pdepth (Q, Mseg )) (2) near objects and violet indicates far objects: A = F (I, Mseg , Mdepth , Q) (3) d − dmin (7) Color(d) = Rainbow where Gseg is the segmentation model (SAM), Gdepth is the dmax − dmin depth estimator (Marigold), P· are prompt templates, and F is The rainbow colormap traverses the hue spectrum from red the reasoning MLLM (Qwen2-VL). (0°) through yellow (60°), green (120°), blue (240°), to violet (280°), providing intuitive visual cues for depth ordering. 3.2 Stage 1: Visual Grounding (Where) Implementation Details. Marigold LCM runs in fp16 preThe first stage generates an instance segmentation map that cision and takes approximately 2.5 seconds for a 512 × 512 identifies and highlights individual objects in the scene. We image with 5 ensemble members on an RTX 3090. use SAM [9] with a systematic grid of point prompts to ensure comprehensive scene coverage. 3.4 Stage 3: Semantic Reasoning (What) Grid Prompt Strategy. We uniformly sample an N × N grid of point prompts across the image, where N controls the The final stage takes the original image I, segmentation map density of object sampling. Each point serves as a “click” Mseg , and depth map Mdepth as inputs to an MLLM for answer prompt for SAM, generating an instance mask for the object at generation. Multi-Image Input. We present the three images as a that location. Formally, for an image of dimensions W × H: multi-image input to Qwen2-VL [17], which natively sup jH iW ports multiple image inputs in a single conversation turn. The , | 0 ≤ i, j < N (4) P= prompt instructs the model to integrate all three visual inputs: N −1 N −1 “You are given three images: (1) the original image, (2) For each point p ∈ P, SAM generates a set of candidate masks a segmentation map with objects color-coded by instance, {m1 , m2 , . . . , mK } with associated confidence scores. We se(3) a depth map using rainbow colormap (red=near, violect the mask with the highest predicted IoU score. let=far). Based on this visual evidence, answer concisely: Mask Filtering and Coloring. Masks with area below a Q” threshold τarea = 100 pixels are discarded to remove noise. Why Multiple Images Help. The segmentation map proThe remaining masks are assigned distinct colors from a highvides explicit object boundaries and instance identity, helping contrast palette: the MLLM count objects and identify spatial regions. The X Mseg (x, y) = Color(ci ) if (x, y) ∈ Maski and ⊮[Maski (x,depth y)] >map τareaprovides depth ordering cues that are difficult to extract from a single 2D image. Together, they give the MLLM (x,y) (5) structured visual evidence that complements its own visual unwhere ci is a randomly assigned color for instance i, with derstanding. colors sampled from the range [50, 255]3 to ensure visibility against black background. 3.5 Adaptive Reasoning Router Implementation Details. We use N = 8 (64 grid points) as a balance between coverage and computational cost. SAM Not all questions require the full three-stage pipeline. Simple ViT-Large processes all points in a single forward pass, taking recognition or counting questions may not benefit from depth information, while attribute questions may not need segmenapproximately 2 seconds on an RTX 3090. tation. We design a lightweight router that selects the appropriate reasoning path. 3.3 Stage 2: Geometric Reasoning (How) Router Architecture. A BERT-base [2] classifier (110M The second stage produces a pseudo-colored depth map that parameters) takes the question text as input and predicts one encodes spatial depth relationships between objects. We use of three reasoning paths:
3
Algorithm 1 Gen-VCoT Inference Require: Image I, Question Q Ensure: Answer A 1: path ← Router(Q) {Select reasoning path} 2: if path ∈ {path_1, path_1+2, full} then 3: Mseg ← SAM(I, GridPrompts(N = 8)) {Stage 1} 4: Colorize(Mseg ) {Instance coloring} 5: end if 6: if path ∈ {path_1+2, full} then 7: D ← Marigold(I, steps = 4, K = 5) {Stage 2} 8: Mdepth ← RainbowColormap(D) {Pseudo-color} 9: end if 10: if path = full then 11: A ← Qwen2-VL(I, Mseg , Mdepth , Q) {Stage 3} 12: else if path = path_1+2 then 13: A ← Qwen2-VL(I, Mseg , Mdepth , Q) 14: else if path = path_1 then 15: A ← Qwen2-VL(I, Mseg , Q) 16: else 17: A ← Qwen2-VL(I, Q) {Baseline} 18: end if 19: return A
Figure 2: Three evaluation scenes with their segmentation maps and depth maps. (a) Indoor scene with furniture and plants. (b) Urban street scene with buildings and vehicles. (c) Park scene with trees and recreational objects.
• Park (Figure 2c): An outdoor scene with trees, bench, path, • path_1: Segmentation only (for object-level questions) ball, and sign • path_1+2: Segmentation + Depth (for spatial questions) Question Categories. We design 8 questions per scene • full: All three stages (for complex reasoning) spanning 6 categories: object recognition (“What objects Training Objective. The router is trained on labeled examare in this image?”), spatial relationships (“Describe the ples with a combined loss: spatial layout”), depth perception (“Which object is closL = LCE (p̂, p) + λ · step_count(p̂) (8) est/farthest?”), counting (“How many objects?”), attribute recognition (“What color is X?”), and complex reasoning (“If where p̂ is the predicted path, p is the ground truth path, and I walk from left to right, what do I encounter?”). In total, we λ controls the efficiency-accuracy trade-off. This encourages evaluate 24 question-scene pairs across 4 pipeline configurations. the router to select the simplest adequate path. Baselines. We compare four configurations: • Full pipeline (Gen-VCoT): Original image + Segmentation 3.6 Pipeline Algorithm + Depth The complete Gen-VCoT inference procedure is summarized • No depth: Original image + Segmentation only in Algorithm 1. • No seg: Original image + Depth only • Baseline: Original image only (direct MLLM inference) Metrics. We report (1) answer accuracy (exact match or 4 Experiments containment match with ground truth), (2) reasoning latency per question, and (3) total pipeline throughput. 4.1 Experimental Setup Models. We use SAM ViT-Large [9] for segmentation (loaded from HuggingFace Transformers), Marigold LCM [8] for depth estimation (from Diffusers, fp16 variant), and Qwen2VL-7B-Instruct [17] for multimodal reasoning. All experiments run on a single NVIDIA RTX 3090 (24GB VRAM). Models are loaded sequentially to fit within memory constraints. Evaluation Scenes. We construct three synthetic scenes with increasing complexity (Figure 2): • Indoor (Figure 2a): A room with bookshelf, sofa, table, plant, and window • Street (Figure 2b): An urban scene with buildings, car, lamp post, and sun
4.2
Main Results
Table 1 presents the comparison between Gen-VCoT and the baseline across question categories from our initial evaluation on a complex scene with 19 questions. Spatial Reasoning. Gen-VCoT demonstrates significantly stronger spatial awareness. When asked “Describe the spatial relationship between the two houses,” Gen-VCoT correctly identifies “the house on the left is closer to the viewer” using depth information, while the baseline only describes size differences (“the house on the left is smaller”). This confirms that the depth map provides explicit distance cues that the MLLM cannot reliably extract from a single image. 4
Table 1: Main results: Gen-VCoT vs. baseline (direct MLLM Table 2: CLEVR-style evaluation: Gen-VCoT vs. baseline on inference) across question categories on a complex synthetic 80 questions with ground-truth answers across 10 scenes. scene with 19 questions. Category Recognition (1) Spatial (4) Depth (4) Counting (3) Attribute (2) Reasoning (5) Total (19)
Gen-VCoT
Baseline
∆
1/1 4/4 3/4 2/3 1/2 4/5
1/1 3/4 2/4 2/3 2/2 3/5
0 +25% +50% 0 −50% +20%
15/19 (78.9%)
13/19 (68.4%)
+10.5%
Question Type
Gen-VCoT
Baseline
#Q
Exist Count Query Color Query Shape Spatial
28/40 (70%) 15/20 (75%) 10/16 (62%) 3/3 (100%) 0/1 (0%)
34/40 (85%) 18/20 (90%) 13/16 (81%) 3/3 (100%) 0/1 (0%)
40 20 16 3 1
Total
56/80 (70.0%)
68/80 (85.0%)
80
Table 3: When do visual intermediates help or hurt? Summary across evaluations, including Text CoT comparison. Question Type
Gen-VCoT
Text CoT
Best Strategy
Simple factual (exist/count) Attribute query (color/shape) Spatial reasoning Depth perception Complex reasoning
70% 62% 100% 75% 80%
91% 91% – – –
Text CoT Text CoT Visual Visual Visual
Surprising Finding. On CLEVR-style questions, the baseline outperforms Gen-VCoT by 15% (85.0% vs. 70.0%). This contrasts sharply with our primary evaluation where GenVCoT outperformed the baseline by 10.5%. Analysis reveals that CLEVR questions are predominantly simple factual queries (existence, counting, attribute lookup) that the MLLM can answer directly from the raw image. The intermediate visual products introduce noise: SAM’s grid-based segmentation may generate irrelevant masks for simple scenes, and Marigold’s depth estimation on synthetic scenes lacks meaningful depth gradients. When Do Intermediates Help? This contrast provides strong empirical motivation for our adaptive reasoning router. Table 3 summarizes the conditions: This finding validates the design of our adaptive router: simple questions should bypass intermediate steps (path_baseline), while complex spatial and depth questions benefit from the full pipeline (path_full). Without the router, applying intermediates indiscriminately would degrade overall performance.
Figure 3: Qualitative comparison between Gen-VCoT and baseline across four representative questions. Green boxes indicate correct/improved answers; red boxes indicate errors or incomplete responses. Gen-VCoT consistently produces more accurate spatial reasoning, correct object ordering, and structured scene descriptions.
Depth Perception. For depth-related questions, Gen-VCoT leverages the pseudo-colored depth map to provide more accurate spatial judgments. The baseline often confuses spatial position with visual salience or object size. For example, when asked “Which object is closest to the viewer?” the baseline answers “red truck” (the most visually prominent object) while Gen-VCoT correctly identifies “the tree” based on depth map evidence. Figure 3 provides detailed qualitative comparisons showing specific examples where Gen-VCoT outperforms the baseline. 4.4 Text CoT Comparison A natural question is whether the visual intermediates provide information beyond what text descriptions can convey. We im4.3 CLEVR-Style Evaluation plement a Text CoT baseline that provides the MLLM with To evaluate generalization, we generate 10 CLEVR-style syn- structured text descriptions of each object (color, shape, size, thetic scenes with 2–4 colored 3D objects each, producing 80 material, position) alongside the original image, without any questions with ground-truth answers across 5 categories: ex- visual intermediate products. istence (“Is there a red object?”), counting (“How many obTable 4 presents the three-way comparison on CLEVR jects?”), attribute query (“What color is the cube?”), shape scenes. query, and spatial reasoning. Unlike our primary evaluation Key Finding. Text CoT achieves the highest accuracy scenes, CLEVR questions have deterministic ground-truth an- (91.2%), outperforming both the baseline (85.0%) and Genswers. VCoT (62.5%). This reveals that: Table 2 presents the results. 1. Structured information helps: Both Text CoT and Gen5
Table 4: Three-way comparison on CLEVR scenes (80 questions). Text CoT provides structured text descriptions of objects instead of visual intermediates. Method Baseline (image only) Gen-VCoT (visual intermediates) Text CoT (text descriptions)
Accuracy 68/80 (85.0%) 50/80 (62.5%) 73/80 (91.2%)
Table 5: Ablation study results across 3 scenes × 8 questions × 4 configurations (96 total evaluations). “Seg” = segmentation map, “Depth” = depth map. Figure 4: Ablation study detail: Four pipeline configurations tested on the street scene spatial ordering question. Full Configuration Avg Time Spatial Depth pipeline (green) correctly identifies the leftmost building. Removing depth (yellow) still works but is less reliable. RemovFull (Seg+Depth) 1.07s Best Best No Depth (Seg only) 0.68s Good Poor ing segmentation (orange) produces incorrect answers. BaseNo Seg (Depth only) 0.79s Good Good line (red) hallucinates. The depth map provides the critical Baseline (neither) 0.87s Poor Poor distance cue for spatial ordering. VCoT provide structured object descriptions, but text is more effective for factual queries because it directly encodes attributes (color, shape, size) without visual noise. 2. Visual intermediates introduce noise on simple queries: SAM’s grid-based segmentation may generate irrelevant masks, and Marigold’s depth estimation on synthetic scenes lacks meaningful gradients, both of which add noise rather than signal. 3. Modality matters: For factual queries, text descriptions are a more precise information channel than visual images. The MLLM’s language understanding capabilities are better suited to processing structured text than interpreting pseudo-colored visualizations. Implications for Router Design. This finding suggests that the optimal intermediate representation is question-dependent: text descriptions for factual queries, visual intermediates for spatial reasoning. The router should ideally choose between four modes: baseline (image only), text CoT (text descriptions), visual intermediates (segmentation + depth), or both. We leave this extended router design to future work. Counting and Attributes. For simple counting and attribute recognition, both methods perform similarly, as these tasks primarily rely on object detection rather than spatial reasoning. The attribute category shows a slight advantage for the baseline, likely because the segmentation map occasionally introduces visual noise that confuses color identification.
swers “sun” (confusing visual prominence with spatial proximity) while the full pipeline correctly answers “a building.” Figure 4 provides a detailed side-by-side comparison of all four configurations. Effect of Segmentation. When segmentation maps are removed (“No Seg” mode), the model sometimes miscounts objects. For the park scene, the no-seg model reports 5 objects instead of the correct 4, as it cannot distinguish individual tree canopies from background foliage. Baseline Failure Modes. The baseline (no intermediates) produces notably degraded outputs on recognition tasks. For the indoor scene, it generates repetitive garbage output (“bar, bar, bar...”), suggesting that without structured visual evidence, the MLLM struggles to parse complex synthetic scenes. Complementary Information. The full pipeline consistently outperforming both partial configurations confirms that segmentation and depth provide complementary information: segmentation helps with object identity and counting, while depth helps with spatial ordering and distance estimation. 4.6
Efficiency Analysis
Table 6 reports the inference efficiency of different pipeline configurations. The optimized pipeline reduces per-question inference time by ∼16× compared to naive sequential processing (5.2s vs. 50.5s/3q = 16.8s/q), making the approach practical for batch evaluation. For single-question scenarios, the one-time model 4.5 Ablation Study loading cost dominates, but this can be amortized when anWe conduct comprehensive ablation studies across three di- swering multiple questions about the same image. verse scenes to understand the contribution of each visual intermediate step. Table 5 presents the results. 4.7 Qualitative Analysis Qualitative Ablation Findings. Effect of Depth Maps. When depth maps are removed (“No Figure 2 shows the intermediate visual products generated by Depth” mode), the model frequently fails on spatial reasoning Gen-VCoT for three diverse scenes. Figure 5 shows the interquestions. For the street scene, when asked “If I walk from left mediate visual products generated by Gen-VCoT for a comto right, what do I encounter first?” the no-depth model an- plex scene. The segmentation map clearly delineates individ6
Table 6: Inference efficiency breakdown. The optimized Table 7: Comparison of Visual CoT approaches across key pipeline loads each model once for batch processing. properties. Gen-VCoT is the only method satisfying all four criteria. Component
Time
Notes
One-time costs (per image batch): SAM loading 1.1s ViT-Large, 1.3GB SAM inference 1.1s 64 grid points Marigold loading 2.1s fp16, 3GB Marigold inference 2.5s 4 steps, 5 ensemble Qwen2-VL loading 11.9s 7B, 16.6GB Per-question costs: Gen-VCoT reasoning Baseline reasoning
0.8s 0.3s
3-image input 1-image input
Naive (3q) Optimized (19q) Per question
50.5s 99.6s 5.2s
Reload per question Batch processing Amortized
Method
Interp.
E2E
Dense
Decodable
Text CoT COVT Sketchpad Gen-VCoT
✓ – ✓ ✓
✓ ✓ – ✓
– ✓ – ✓
– ✓ – ✓
No-Seg: 5 (miscounts tree canopy segments) Baseline: 5 (includes background elements)
The segmentation map helps Gen-VCoT correctly distinguish individual objects from background elements.
5
Discussion
Positioning in Visual CoT Taxonomy. Gen-VCoT is unique in simultaneously achieving four key properties: interpretability (RGB images are directly visualizable by humans), end-toend processing (no external tool calls during inference), dense visual representation (pixel-level segmentation and depth), and decodability (intermediate representations can be further processed). As shown in Table 7, existing methods trade off between these properties. Why RGB Intermediates Help. Our experiments suggest that RGB intermediate representations help in three ways: (1) they provide explicit object boundaries that aid counting and identification, (2) they encode depth relationships that are difficult to infer from 2D images alone, and (3) they give the MLLM structured evidence that reduces hallucination on spatial questions. Limitations. (1) The current pipeline uses fixed expert models without task-specific fine-tuning, which may limit performance on domain-specific questions or unusual visual domains. (2) The three-stage sequential processing introduces latency overhead (∼8s preprocessing per image), though our batch optimization amortizes this across multiple questions. (3) Evaluation is currently limited to synthetic scenes; realworld benchmark evaluation (GQA [6], CLEVR [7]) is ongoing and may reveal additional challenges. (4) The quality of intermediate products depends on the expert models— segmentation failures or depth estimation errors can propagate to the final answer. Absolute Speedup vs. Quality Trade-off. Gen-VCoT is designed for reasoning quality improvement rather than latency reduction. The framework adds ∼8s of preprocessing (segmentation + depth) per image, but this overhead is amortized when answering multiple questions about the same image. For applications requiring high accuracy on spatial reasoning tasks, this trade-off is favorable. The adaptive router further reduces overhead by skipping unnecessary stages. Future Work. Several directions merit exploration: (1) Scaling to real-world benchmarks (GQA, CLEVR, MIRA [16]) with larger evaluation sets to validate general-
Figure 5: Qualitative example of Gen-VCoT intermediate products. (a) Input synthetic scene with multiple geometric objects. (b) SAM segmentation map with instance-level color coding. (c) Marigold pseudo-colored depth map (red=near, violet=far).
ual objects with distinct colors, while the depth map provides intuitive spatial ordering through the rainbow colormap. Example 1: Spatial Relationship (Indoor Scene). Q: Describe the spatial layout of the scene. Gen-VCoT: The scene consists of a large brown rectangle (bookshelf) on the left, a blue rectangle (sofa) in the center-right, a green circle (plant) on the far right, and a small brown rectangle (table) above the sofa. Baseline: The image depicts a simple scene with various geometric shapes. On the left side, there is a large brown rectangle...
Gen-VCoT provides more structured and object-aware descriptions by leveraging the segmentation map to identify individual objects. Example 2: Depth Ordering (Street Scene). Q: If I walk from left to right, what do I encounter first? Gen-VCoT: A building. Baseline: A light. (incorrect—the lamp post is not the leftmost object)
Gen-VCoT correctly identifies the leftmost building using spatial information from both the segmentation and depth maps. Example 3: Object Counting (Park Scene). Q: How many objects are there? Gen-VCoT: 4 (trees, bench, ball, sign)
7
ization. (2) Training the adaptive router with reinforcement learning following DeepSeek-R1 [1] methodology, using answer correctness as reward signal. (3) Exploring video diffusion models as temporal reasoning foundations for video understanding tasks. (4) Investigating whether fine-tuning the MLLM on intermediate visual products can further improve performance. (5) Replacing synthetic scenes with real-world images from GQA/CLEVR datasets.
6
[6] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for compositional question answering over real-world images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6700–6709, 2019. [7] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2901–2910, 2017.
Conclusion
[8] Bingxin Ke et al. Repurposing diffusion-based image generators for monocular depth estimation. arXiv preprint arXiv:2312.02145, 2024.
We presented Gen-VCoT, a generative visual chain-of-thought framework that leverages expert vision models (SAM for segmentation, Marigold for depth estimation) to produce RGB images as visual reasoning intermediates. By decomposing visual reasoning into three interpretable stages—visual grounding (Where), geometric reasoning (How), and semantic reasoning (What)—Gen-VCoT establishes a new paradigm for interpretable multimodal reasoning. Our comprehensive evaluations reveal a nuanced picture: (1) on complex spatial reasoning tasks, Gen-VCoT achieves 78.9% accuracy compared to 68.4% for direct MLLM inference (+10.5%), with particularly strong improvements on spatial (+25%) and depth (+50%) questions; (2) however, on simple factual queries (CLEVR), visual intermediates degrade performance (62.5%) compared to baseline (85.0%); (3) a three-way comparison with text-only chain-of-thought reveals that text CoT achieves 91.2% on CLEVR, outperforming both visual intermediates and baseline, indicating that the optimal intermediate representation is task-dependent; (4) these findings provide strong motivation for the adaptive router, which should select between visual intermediates (for spatial reasoning) and text descriptions (for factual queries); and (5) batch optimization enables practical inference speeds (5.2s per question). We believe this “generate to understand” approach opens new avenues for building more interpretable and capable multimodal reasoning systems, and plan to extend this work to real-world benchmarks and video understanding tasks.
[9] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4026, 2023. [10] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in Neural Information Processing Systems, 36, 2024. [11] NTU. Visual chain of thought for video generation. ACL 2026 Findings, 2026. [12] OpenAI. Gpt-4 technical report. arXiv:2303.08774, 2023.
arXiv preprint
[13] Yiming Qin et al. Teaching vlms to see and think better with continuous visual tokens. arXiv preprint arXiv:2511.19418, 2025. [14] Nikhila Ravi et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. [15] UESTC and others. A survey on visual chain-of-thought reasoning. arXiv preprint, 2026. [16] Various. Mira: Multimodal imagination for reasoning assessment. arXiv preprint, 2025. [17] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024.
References [1] DeepSeek. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
[18] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022.
[2] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171–4186, 2019.
[19] Shitao Xiao et al. Omnigen: Unified image generation. arXiv preprint, 2024.
[3] Valentin Gabeur, Shangbang Long, Songyou Peng, et al. Image generators are generalist vision learners. arXiv preprint arXiv:2604.20329, 2026.
[20] Lihe Yang et al. Depth anything v2. arXiv:2406.09414, 2024.
arXiv preprint
[4] Yushi Guan et al. Visual sketchpad: Sketching as a visual chain of thought. In Advances in Neural Information Processing Systems, 2024.
A
[5] Agrim Gupta et al. Instructcv: Instruction-tuned text-to-image diffusion models as vision generalists. In International Conference on Learning Representations, 2024.
This appendix provides the complete prompt templates used in each stage of the Gen-VCoT pipeline. 8
Prompt Templates
A.1
Stage 3: Reasoning Prompt
Gen-VCoT: A building. Baseline: A light.
The following prompt template is used for the Qwen2-VL reasoning stage:
Analysis: Gen-VCoT correctly identifies the leftmost building using spatial information from both the segmentation and depth maps. The baseline incorrectly identifies the lamp post (which is visually prominent but not the leftmost object).
You are given three images: (1) the original image, (2) a segmentation map with objects color-coded by instance, (3) a depth map using rainbow colormap (red=near, violet=far). Based on this visual evidence, answer concisely:<QUESTION>
B.2
Q: How many objects are there? Gen-VCoT: 4 (trees, bench, ball, sign)
For baseline comparisons, we use: You are given an image. concisely:<QUESTION>
Park Scene Examples
No-Seg: 5 (miscounts tree canopy segments) Baseline: 5 (includes background elements)
Answer
Analysis: The segmentation map helps Gen-VCoT correctly distinguish individual objects from background eleA.2 Question Templates by Category ments. Without segmentation, the model sometimes counts Table 8 provides the question templates used for each evalua- tree canopy segments as separate objects. Q: What color is the largest object? tion category. Gen-VCoT: Green
Table 8: Question templates used for each evaluation category. Category
Question Template
Recognition Spatial Depth
“What objects are in this image?” “Describe the spatial layout of the scene.” “Which object is closest/farthest to the viewer?” “How many objects are there?” “What color is the largest object?” “If I walk from left to right, what do I encounter first?”
Counting Attribute Reasoning
Baseline: Gray
Analysis: Gen-VCoT correctly identifies the grass/lawn as the largest object (green), while the baseline may be confused by the gray path or sky.
C
Computational Cost Analysis
Table 9 provides a detailed breakdown of computational costs for each pipeline stage. Table 9: Detailed computational cost breakdown per pipeline stage on NVIDIA RTX 3090.
B
Additional Qualitative Examples
Stage
Load (s)
Infer (s)
VRAM (GB)
This section provides additional qualitative examples comparing Gen-VCoT and baseline outputs across different scenes and question types.
SAM ViT-Large Marigold LCM Qwen2-VL-7B
1.1 2.1 11.9
1.1 2.5 0.8/question
1.3 3.0 16.6
B.1
Total (batch) Total (single)
15.1 15.1
0.8×N 4.5
16.6 16.6
Street Scene Examples
Q: Describe the spatial layout of the scene.
For batch processing of N questions about the same image, the per-question cost is:
Gen-VCoT: The scene consists of a road with a red cart on it, three buildings of varying sizes, a street lamp, and a sun in the sky. The buildings are arranged along the road, with the largest building in the center.
Tload + Tseg + Tdepth 15.1 + 3.6 + Treason ≈ + 0.8 N N (9) For N = 19 questions, this gives Tper-q ≈ 1.8 seconds, compared to Tnaive ≈ 16.8 seconds per question when reloading models for each query. Tper-q =
Baseline: The scene features a red train on a track, with a tall building to the left, a shorter building to the right, and a sun in the sky.
Analysis: Gen-VCoT provides a more complete scene description, correctly identifying the road, buildings, lamp post, D Router Training Details and sun. The baseline confuses the car with a “train” and provides less spatial detail. The adaptive reasoning router is a BERT-base classifier with Q: If I walk from left to right, what do I encounter first? 110M parameters. Training details: 9
• Training data: 100 question-path pairs annotated by the authors • Labels: path_1 (object-level), path_1+2 (spatial), full (complex reasoning) • Optimizer: AdamW with learning rate 2e-5 • Batch size: 16 • Epochs: 3 • Loss weight: λ = 0.1 (efficiency penalty) • Training time: < 1 minute on RTX 3090 The router achieves 85% accuracy on a held-out validation set of 50 questions, with most errors occurring on ambiguous questions that could reasonably be answered with either path_1+2 or full.
10