VisionFoundry: Teaching VLMs Visual Perception with Synthetic Images

VisionFoundry: Teaching VLMs Visual Perception with Synthetic Images

arXiv:2604.09531v1 [cs.CV] 10 Apr 2026

Guanyu Zhou1

Yida Yin1

Wenhao Chai1

Shengbang Tong2

Xingyu Fu1

Zhuang Liu1

1 Princeton University 2 New York University

Project Page

Code

Dataset

Abstract Vision-language models (VLMs) still struggle with visual perception tasks such as spatial understanding and viewpoint recognition. One plausible contributing factor is that natural image datasets provide limited supervision for low-level visual skills. This motivates a practical question: can targeted synthetic supervision, generated from only a task keyword such as Depth Order, address these weaknesses? To investigate this question, we introduce VisionFoundry, a task-aware synthetic data generation pipeline that takes only the task name as input and uses large language models (LLMs) to generate questions, answers, and text-to-image (T2I) prompts, then synthesizes images with T2I models and verifies consistency with a proprietary VLM, requiring no reference images or human annotation. Using VisionFoundry, we construct VisionFoundry-10K, a synthetic visual question answering (VQA) dataset containing 10k image–question–answer triples spanning 10 tasks. Models trained on VisionFoundry-10K achieve substantial improvements on visual perception benchmarks: +7% on MMVP and +10% on CV-Bench-3D, while preserving broader capabilities and showing favorable scaling behavior as data size increases. Our results suggest that limited tasktargeted supervision is an important contributor to this bottleneck and that synthetic supervision is a promising path toward more systematic training for VLMs.

1 Introduction Vision-language models (VLMs) have rapidly evolved into general-purpose systems capable of processing and reasoning over interleaved text and images (Dai et al., 2023; Zhu et al., 2024; Team, 2023; Liu et al., 2023). They now support a broad spectrum of applications, from visual question answering (Liu et al., 2023, 2024a) and optical character recognition (Liu et al., 2024b) to multimodal reasoning (Chen et al., 2024e; Yue et al., 2024), graphical user interface (GUI) grounding (Li et al., 2025), and mathematical problem-solving (Lu et al., 2024). Despite these advances, VLMs still exhibit persistent weaknesses in visual perception, as highlighted by recent diagnostic benchmarks (e.g., MMVP, CV-Bench, and RealWorldQA). Tong et al. (2024b,a) construct benchmarks that decouple visual perception from pure language priors, while RealWorldQA (xAI, 2024) stress-tests geometric and spatial reasoning. One plausible contributing factor behind this perception bottleneck is limited supervision in natural image datasets for these low-level visual skills. Natural image–text corpora, while vast, may not systematically cover the combinatorial range of variations required for robust visual perception. This motivates a practical question: can we synthesize targeted supervision to address these weaknesses without relying on reference images or expensive human annotation?

1

Web Natural Images Web Images

VisionFoundry: Synthetic Images

Traditional VQA Dataset Q: What color is this orange?

VisionFoundry Pipeline

VisionFoundry-10k Dataset

T2I Model

Q: Which direction is the giraffe facing?

LLM

A: Orange. Natural Images

A: Left. VLM Verification

Synthetic Q&A

VQA Dataset

Synthetic Images

Synthetic Q&A

Figure 1 Training VLMs with synthetic images. VisionFoundry uses a task-keyword-only pipeline that requires no reference text or images: it generates task-aware text-to-image (T2I) prompts and question and answer (QA) pairs, synthesizes images, and uses the resulting supervision to improve capabilities such as visual perception in visionlanguage models (VLMs). Compared with traditional web-image collection pipelines, this approach provides more controllable and task-targeted supervision with less retrieval noise.

Synthetic data is a core ingredient in training for LLMs. It can be generated on demand, targeted to specific failures, and iterated rapidly (Gunasekar et al., 2023; Abdin et al., 2024; Qin et al., 2025). A similar trend is emerging in vision, where text-to-image (T2I) models enable synthetic-caption generation and image–text corpus construction (Tian et al., 2023; Fan et al., 2024; Sharifzadeh et al., 2024). They also support multimodal tuning focused on specific visual skills (Liu et al., 2025; Jiao et al., 2025; Li and Li, 2025; Wu et al., 2025). Synthetic data is on-demand, near-unbounded, and highly controllable, allowing systematic coverage of entity, attribute, relation, viewpoint, and style combinations that are rare in natural corpora and otherwise expensive to curate. For VLMs, the more useful question is what makes synthetic images good supervision: they should provide task-specific, reliable, and scalable training signals without reference images or expensive human annotation, while improving visual perception without noticeably degrading broader capabilities (Singh et al., 2024; Ben-Melech Stan et al., 2026; Hu et al., 2025; Wu et al., 2026). We build VisionFoundry, an end-to-end synthetic data-generation pipeline that takes only a task name (or capability configuration) and produces verifier-filtered image–question–answer triplets (see Figure 1). VisionFoundry composes three components into a closed-loop system (Figure 2): (1) a large language model generates question–answer pairs and detailed T2I prompts conditioned on the target task; (2) a modern T2I model synthesizes images conditioned on those prompts; and (3) a strong multimodal judge verifies alignment between the generated image and the answer-determining visual statement, filtering out misaligned samples. This pipeline eliminates the need for reference images and manual labeling, reducing cost while using the native cross-modal capabilities of T2I models. Using VisionFoundry, we construct VisionFoundry-10K, a synthetic visual question answering (VQA) dataset of 10k image–question–answer triples spanning 10 carefully selected low-level visual perception tasks (1k samples per task), including spatial understanding, relative depth, and viewpoint variation (Section 3). We use this controlled setting as an initial validation to isolate the direct contribution of synthetic images to visual capability learning, rather than gains primarily driven by text-heavy reasoning supervision. These tasks are motivated by benchmark analyses that identify spatial and perceptual failures as persistent weaknesses of contemporary VLMs (Tong et al., 2024b,a; xAI, 2024). Across three representative open-source models, Qwen2.5-VL-3B-Instruct (Bai et al., 2025), Llama-3.2-11BVision-Instruct (Meta AI, 2024), and MiMo-VL-7B-SFT (Yue et al., 2025), finetuning on VisionFoundry-10K consistently improves visual perception benchmarks in our experiments, with gains such as +7% on MMVP and +10% on CV-Bench-3D, while preserving broader capabilities and showing favorable scaling behavior as data size increases (Section 4). We also report full benchmark-wise results on broader multimodal reasoning, OCR, and GUI grounding evaluations, where changes are benchmark-dependent: substantial gains on some benchmarks and slight fluctuations on others. We further observe a clear data-size trend: for a representative task, performance improves predictably as synthetic data size increases. We also show that an equal-sized synthetic–natural mixture outperforms pure natural subsets, suggesting that verifier-filtered synthetic supervision provides complementary signals that are hard to obtain from limited real data alone (Tian et al., 2023; Fan et al., 2024; Singh et al., 2024). 2

Task Selection

Adaptive Concept Pool

Object: [Apple]

 Color

 State  Depth

 Viewpoint

Prompt Generation

Generation & Verification

“A red apple on a wooden table in oil painting style."

VQA Dataset

·· Question Question · Question ·· Answer Answer · Answer

Attribute: [Red] LLM

 Orientation

 ……

LLM Scene: [Table] Style: [Oil Painting]

Q: What's the color of the apple? A: Red.

T2I Model

Frontier VLM

Fallback

Figure 2 VisionFoundry overview. Overview of our synthetic VQA data generation pipeline: using only task keywords, an LLM builds an adaptive concept pool; compositional sampling forms entities used to generate T2I prompts; and a T2I model generates images that are verified by a frontier VLM to produce a high-quality VQA dataset. The left, middle, and right parts correspond to Sections 2.1, 2.2, and 2.3, respectively.

Our results suggest that limited task-targeted supervision is an important contributor to the perception bottleneck in VLMs and that targeted synthetic data curation can meaningfully alleviate part of this weakness. Beyond immediate benchmark improvements, VisionFoundry is an initial step toward more systematic synthetic-data workflows for multimodal training. If synthetic data is now a first-class primitive for LLM post-training, verifier-filtered synthetic images may become an analogous primitive for VLMs. Such synthetic images can support instruction tuning and future post-training pipelines that patch capability gaps as new evaluations emerge. They may also support pretraining regimes with broader compositional coverage than web-scale natural images can provide. We hope this work can catalyze a broader shift toward fully automated synthetic multimodal data as a promising path to stronger visual understanding.

2 VisionFoundry VisionFoundry is a task-aware synthetic data generation pipeline that produces high-quality VQA supervision for VLMs. It constructs data using only task specifications, without reference images, human-written QA annotations, or real image–caption pairs. VisionFoundry is guided by three methodological principles: controllability, visual determinism, and verification. Controllability uses explicit task configurations and entity pools to systematically cover targeted capabilities. Visual determinism encodes the answer-determining facts in the prompt so that the question is answerable only from the image. Verification uses a strong proprietary multimodal judge to filter subtle prompt and image misalignments that otherwise poison supervision. We structure the section around these principles rather than the implementation details. Operationally, given a high-level task specification (e.g., spatial understanding, counting, depth reasoning), VisionFoundry has three stages: (1) task-aware generation of questions, answers, and T2I prompts; (2) image synthesis conditioned on the generated prompts; and (3) alignment verification and filtering using a strong multimodal judge. Together, these stages produce supervision that is visually grounded, unambiguous, and robust. Figure 2 provides an overview of the pipeline, with its left, middle, and right parts corresponding to Sections 2.1, 2.2, and 2.3, respectively. We provide the exact pipeline prompts in the appendix for transparency and reproducibility in downstream audits.

2.1 VQA Triplet Generation The left part of Figure 2 corresponds to the first stage of VisionFoundry, which constructs a triplet consisting of a question, its answer, and a corresponding T2I prompt. This stage is driven by GPT-5.2 (OpenAI, 2025).

3

Real-World Spatial Understanding

Viewpoint and Perspective

Q: Is the window's bottom sill higher than the mailbox? A: Yes, the window sill is higher than the top of the mailbox.

Q: From what perspective was the photo taken relative to the screen? A: From a very low, worm's-eye viewpoint near the front row.

Structural and Physical Characteristics

State and Condition

Q: What is the silhouette profile of the barometer in the image? A: Circular with a small hanging loop at the top.

Q: Is the toy ship's cabin door closed or ajar? A: Ajar.

Color and Appearance

Depth Order

Q: What two colors are the stripes on the sofa? A: Red and white.

Q: Which object is closer to the camera, the submarine or the knife? A: The submarine is closer to the camera.

Positional and Relational Context

Orientation and Direction

Q: Which object is directly above the suitcase? A: The backpack.

Q: Is the motorcycle facing toward the camera or away from the camera? A: Away from the camera.

Relative Distance

Spatial Relationship

Q: Which object is closest to the pallet stack with the barcode label? A: Cup.

Q: In the blueprint image, is the tripod located west of the sink? A: Yes, the tripod is west (left) of the sink.

Figure 3 VisionFoundry-10K examples. Randomly selected qualitative examples from VisionFoundry-10K, covering all 10 tasks. The task names are annotated at the top and serve as the only valid input to the pipeline. Each panel shows a generated image, its corresponding question, and ground-truth answer.

Task-aware generation. VisionFoundry conditions generation on an explicit task configuration. Each task specifies the target capability (e.g., spatial relations, color discrimination), the number of objects involved, and optional constraints such as required attributes or relations. To support systematic coverage, we construct an entity pool composed of objects, attributes, scenes, styles, and task-specific custom dimensions. Entities are sampled from the Cartesian product of these dimensions, yielding structured visual configurations. Given a sampled entity, the LLM is prompted to generate: (i) a question whose answer is fully determined by visible content, (ii) a concise and deterministic answer, and (iii) a highly detailed T2I prompt that explicitly encodes the answer-determining visual facts. To avoid ambiguity, the LLM is instructed to rely exclusively on visually verifiable properties and to avoid any hidden or commonsense assumptions. Design rationale. This design enforces tight coupling between language supervision and visual content at generation time. By embedding the correct answer directly into the T2I prompt, we reduce the risk of producing images that are irrelevant or underspecified with respect to the question. This is a common failure mode in synthetic multimodal data.

2.2 Image Synthesis The middle part of Figure 2 corresponds to the second stage, where VisionFoundry synthesizes images conditioned on the T2I prompts. We use a modern T2I model, instantiated as Gemini-2.5-Flash-Image (Google Cloud, 2025a; Comanici et al., 2025), which provides strong photorealism and prompt adherence. 4

Prompt-conditioned generation. Each image is generated directly from the T2I prompt produced in the previous stage. T2I prompts typically specify the main objects, their attributes, spatial arrangements, scene context, and visual style, ensuring that the resulting image contains all information required to answer the associated question. Importantly, VisionFoundry treats the T2I model as a black box. Our method does not rely on any model-specific internals, making the pipeline compatible with upcoming generators. Selective iterative refinement. To improve yield, VisionFoundry allows limited iterative refinement of images. If an initially generated image fails downstream verification, the system can request a localized edit that minimally modifies the image to better satisfy the intended visual statement. This refinement process is conceptually similar to prompt-based image editing, but is only used when necessary and does not alter the original question and answer pair.

2.3 Alignment Verification and Filtering The right part of Figure 2 corresponds to the final stage, where VisionFoundry verifies whether each generated image is consistent with its corresponding question and answer. This stage is important for maintaining dataset quality under larger generation budgets, while keeping the pipeline fully automatic. Statement-based verification. Given a question and its candidate answer, we convert it into a short declarative visual statement that captures the answer-determining fact (e.g., “The red cube is to the left of the blue sphere”). A strong proprietary multimodal model, Gemini-3-Pro (Google Cloud, 2025b), then acts as a judge over images and text. It reads the generated image together with the statement and returns an accept/reject decision based on whether the core visual relation is actually present. We ignore minor stylistic discrepancies and focus on correctness of the answer-determining visual facts. Importantly, this verification stage is also fully automated and involves no manual intervention. Binary filtering criterion. Only samples for which the judge confirms alignment are retained. If an image fails verification after limited refinement attempts, the entire sample is discarded and a new entity is sampled. Because there is no human correction loop, any false accept decision by the verifier may still pass into the final dataset. This design choice is intentional: our goal is to study the feasibility and data-size effects of a fully automated synthetic-data pipeline, rather than to present a fully perfected curation system. Why verification matters. Without explicit verification, synthetic datasets are prone to subtle misalignments, such as missing objects, incorrect spatial relations, or visually ambiguous scenes. By incorporating a multimodal verifier, VisionFoundry adds a practical automatic filtering stage to reduce obvious misalignments before training. Although the verifier is not perfect, we observe that current frontier models such as Gemini are already strong enough to make this automation effective in practice. Detailed verification-accuracy records are provided in the appendix. The output of the pipeline is a fully synthetic collection of image–question–answer triples, formatted for standard VLM instruction tuning. Each retained sample is verifier-approved and intended to be visually grounded and answerable solely from the image. Overall, VisionFoundry demonstrates how modern LLMs and T2I models can be composed into a principled system for synthetic multimodal data generation and provides a practical testbed for studying improvements in current VLMs’ visual understanding.

3 VisionFoundry-10K Dataset Using VisionFoundry, we construct VisionFoundry-10K, a synthetic VQA dataset targeting visual perception. VisionFoundry-10K contains 10k image–question–answer triples across 10 carefully selected tasks, with 1k cases per task. All images are generated by T2I models and filtered through automated multimodal alignment verification as described in Section 2. Figure 3 shows VisionFoundry-10K examples. Task selection. The tasks are chosen to emphasize fundamental visual perception skills that are known to be challenging for current VLMs and are weakly correlated with pure language ability. We treat VisionFoundry10K as a first-step validation of synthetic images as training supervision. We therefore prioritize visual

5

Benchmark

Qwen2.5-VL-3B Instruct

MiMo-VL-7B SFT

Llama-3.2-11B Vision-Instruct

Baseline

Synth

Baseline

Synth

Baseline

Synth

35.3 64.3 67.3 66.0 65.0

42.0 68.3 72.4 76.5 66.9

43.3 66.7 74.3 72.3 65.9

57.3 77.7 79.0 83.7 67.5

42.7 70.3 70.4 74.4 63.0

46.7 71.7 71.7 75.3 64.6

48.1 48.0 75.8 55.7 62.3

47.9 47.1 76.1 54.5 62.9

58.9 49.4 50.5 53.7 39.1

58.7 52.3 81.6 62.7 52.4

34.3 40.1 66.6 49.0 47.0

35.5 40.7 73.8 50.9 48.3

19.4 82.7 9.2 35.2 15.8

20.2 82.4 9.9 36.8 14.4

3.4 83.3 8.4 32.0 24.0

5.8 82.9 11.5 40.0 28.8

0.0 75.4 3.9 27.5 5.3

6.4 75.2 8.9 39.2 6.2

Visual Perception Benchmarks MMVPpair (Tong et al., 2024b) MMVPsingle (Tong et al., 2024b) CV-Bench-2D (Tong et al., 2024a) CV-Bench-3D (Tong et al., 2024a) RealWorldQA (xAI, 2024)

General Understanding and Reasoning Benchmarks BLINK (Fu et al., 2024) MMMUVal (Yue et al., 2024) MMBenchEN v1.1 (Liu et al., 2024a) MMStar (Chen et al., 2024e) MathVistaMini (Lu et al., 2024) Application-Specific Benchmarks ScreenSpot-Proavg (Li et al., 2025) OCRBench (Liu et al., 2024b) MMSI-Bench (Yang et al., 2026) 3DSRBench (Ma et al., 2025) LEGO (Tang et al., 2025)

Table 1 Main Benchmark Results. Main results on 13 benchmarks with 15 reported metrics across three VLMs. Models trained on VisionFoundry-10K improve visual perception performance while showing benchmark-dependent generalpurpose changes, including clear gains on several benchmarks and slight fluctuations on others. We report the mean score over four independent runs. Visual perception benchmarks are highlighted at the top.

perception tasks over text-heavy or long-chain reasoning tasks to isolate the visual learning effect from improvements that come mainly from textual reasoning signals. In particular, we focus on perceptionoriented tasks that depend on spatial layout, depth, and relative attributes, rather than high-level semantics or commonsense knowledge. The 10 tasks included in VisionFoundry-10K are: • Orientation and Direction • Viewpoint and Perspective • Positional and Relational Context • Spatial Relationship • State and Condition • Structural and Physical Characteristics • Color and Appearance • Depth Order • Relative Distance • Real-World Spatial Understanding

Together, these tasks cover low-level visual skills. Examples include left/right and front/behind reasoning, near/far judgments, depth ordering, viewpoint-dependent appearance, and relative attribute comparisons. Dataset construction and statistics. For each task, we define a task-specific configuration that constrains the number of objects, admissible attributes, and relational predicates. Entity configurations are systematically sampled from an underlying entity pool, ensuring diversity in object categories, attributes, scenes, and visual styles. For every sampled configuration, VisionFoundry generates a single question with a deterministic answer, plus a corresponding T2I prompt encoding the answer-determining visual facts. After image synthesis and alignment verification, exactly 1k verifier-accepted samples are retained per task, resulting in the balanced VisionFoundry-10K dataset of 10k examples. All questions are short and unambiguous, and all answers are concise (e.g., binary, categorical, or short phrases), enabling reliable finetuning and evaluation. Positioning vs. existing datasets. Unlike large-scale natural-image VQA datasets, VisionFoundry-10K is not designed to maximize semantic coverage or linguistic diversity. Instead, it prioritizes precise visual grounding and controlled difficulty, making it suitable for diagnosing and improving core perceptual weaknesses of VLMs. Compared to manually curated visual perception benchmarks, VisionFoundry-10K is 6