SABRE: Scalable and Automated Benchmarking of VLMs under Stress

SABRE: Scalable and Automated Benchmarking of VLMs under Stress Zixuan Lan∗,† University of Chicago

Luzhe Sun∗ Toyota Technological Institute at Chicago

zixuanlan@uchicago.edu

luzhesun@ttic.edu

Matthew R. Walter Toyota Technological Institute at Chicago

Jiawei Zhou Stony Brook University

mwalter@ttic.edu

jiawei.zhou.1@stonybrook.edu

Abstract

arXiv:2608.07435v1 [cs.CV] 7 Aug 2026

Vision-language models (VLMs) are improving rapidly, but benchmark development lags behind, making weaknesses hard to identify. Building stress tests is costly: samples must satisfy controlled conditions, remain answerable, and challenge current models. We present SABRE, a scalable, automated pipeline that converts a Test Primer(a Markdown Task Design with Data Schema) into structured specifications, generated or edited images, and question–answer pairs. Automated filtering removes candidates solved by a Filtering VLM, while human review verifies candidate validity and supports annotation correction and localized image repair. We instantiate SABRE-Prior to test whether VLMs follow visual evidence instead of relying on world priors—learned expectations about familiar objects and scenes. Its 600 images and 1,000 questions span Context (unexpected entities in familiar scenes), Texture (counterfactual materials), Attribute (noncanonical component counts), and Language Elicitation (answers suggested by language but unsupported by the image). Across six VLMs, macro-average accuracy ranges from 17.8% to 31.3% (22.6% mean). A real-image Attribute control is comparably difficult for the Filtering VLM. SABRE-Counting and SABRESpatial pilots show that the workflow supports other stress-test settings. These results establish SABRE as a reusable framework for constructing and refreshing VLM stress tests rather than a single fixed benchmark. Project website and resources will be available at https: //zesearch.github.io/vlm-SABRE/.

1

Introduction

Vision-language models (VLMs) are advancing rapidly, while reliable benchmarks remain slow to design, construct, and validate(Fu et al., 2024; Yang et al., 2025). As fixed benchmarks saturate, * †

Equal contribution. Work done as a research intern at Stony Brook University.

they reveal less about the weaknesses that remain and provide less guidance for further model development. This paper asks a direct question: Can we construct targeted and reliable VLM stress tests fast enough to keep pace with model development? Large-scale visual benchmarks have traditionally been built by collecting real images and annotating them with human labor. ImageNet (Deng et al., 2009) and MS COCO (Lin et al., 2015) are landmark examples of this approach. This approach is especially costly for stress tests, which evaluates a target capability under deliberately demanding, distracting, or counterintuitive conditions while keeping the task valid and answerable. Building stress tests from real-image collections is slow and labor-intensive, benchmark designers must devise enough rare and counterintuitive examples to probe the target failure mode systematically which makes it extremely hard to scale up. To reduce annotation costs, several approaches automatically derive questions or labels from existing image collections. For example, POPE constructs objectexistence questions from images and object annotations, while AutoConverter automatically turns existing visual questions into challenging multiplechoice questions (Li et al., 2023; Zhang et al., 2025). These approaches show that much of the annotation process can be automated. However, they still inherit the images from source datasets where the visual evidence in these images may not be sufficient for complicated stress test setting. Image generation alone does not produce a valid benchmark: generators may omit requested objects, alter unrelated regions, or produce incorrect counts(Hu et al., 2023; Huang et al., 2025; Ghosh et al., 2023), causing questions and reference answers derived from the benchmark specification to disagree with the actual image. Benchmark construction therefore requires both specification-faithful generation and explicit verification. To address these challenges, we introduce a mod-

SABRE

Test Primer Describe the task you want to test in natural language.

User

Scalable and Automated Benchmarking of VLMs under Stress

Task Design

(a)

(b)

(c)

LLM

⋯

Q1

Q2

Q3

Q4

A1

A2

A3

A4

Image Generation and Edition

Edit

Attribute

Texture

Spatial

Q1

Q2

Q3

Q4

A1

A2

A3

A4

Filtering VLM Convenient Human Evaluation Platform

Base Image Description Area of Interest Image Edition Plan

×N

Sample Specifications

Scalable Customized VLMs Stress Test

Various Category Test Image

Question & Answer Builder Data Schema

Automated Filtering (Reject Sampling)

Nano Banana

Flux

Language

Context

Counting

Discard

Figure 1: Overview of the SABRE benchmark-construction pipeline.

ular pipeline in which a benchmark designer provides a Test Primer which consists of a naturallanguage Markdown Task Design, a Data Schema that defines task-specific fields and validation rules, and question format, such as open generation or multiple choice. The pipeline converts this specification into structured sample specifications. It then generates the required images, applies taskoriented edits when needed, and builds the corresponding question–answer pairs. Each candidate is evaluated by a filtering VLM through a model-inthe-loop Automated filtering. Candidates solved by the filtering model are discarded, while those exposing a potential failure are retained. Because automatic filtering establishes difficulty but not validity, every retained candidate must undergo human verification.For tasks that require a specified image edit, reviewers compare the base image I b with the edited image I e to verify that the requested change is present and unrelated content is preserved. Reviewers can revise questions, reference answers, and region annotations, or repair localized image defects without regenerating the entire image. The same interface also accepts uploaded real images, which pass through the same screening and curation stages. We first instantiate the pipeline as SABRE-Prior, which tests whether VLMs follow visual evidence when it conflicts with world priors, namely expectations learned from training data about how familiar objects, scenes, and situations normally appear(Liu et al., 2025; Lee et al., 2025; Luo et al., 2025; Vo et al., 2026; Guan et al., 2024). Such conflicts are underrepresented in conventional image collections because they are intentionally unusual and often require precise image edits. Nevertheless, these atypical scenes, edited content, and noncanonical object configurations can test a basic requirement of visual grounding: a model should answer from the image supplied by

the user rather than from what it expects to see. Using the pipeline, we construct SABRE-Prior across four complementary subsets: unexpected objects in familiar scenes (Context), answers suggested by question wording but unsupported by the image (Language Elicitation), counterfactual object materials (Texture), and noncanonical component counts (Attribute). To examine the pipeline in different stress-test settings, we also apply it to Counting and Spatial reasoning using separate Test Primers. These two small-scale pilots are not intended as exhaustive evaluations on model capability; instead, they examine whether the pipeline can construct new types of stress tests without taskspecific redesign. We evaluate SABRE-Prior on six recent frontier VLMs. Across its four world-prior subsets, macro accuracy ranges from 17.8% to 31.3%, with an average of 22.6% across models. The distinct performance profiles across subsets reveal complementary failures in following visual evidence and rejecting unsupported prior expectations. Together, the SABRE-Prior results and the two pilot instantiations support the central contribution of this work: SABRE is not a single fixed benchmark, but a reusable framework for constructing, verifying, and refreshing targeted VLM stress tests as models evolve.

2

Related Work

VLM Evaluation and Stress Testing. VLM evaluation has expanded from task-specific VQA to broad suites covering perception, reasoning, instruction following, and hallucination, such as MME, MMBench, and MM-Vet, alongside targeted stress tests such as POPE and MMVP (Fu et al., 2025; Liu et al., 2024; Yu et al., 2024; Li et al., 2023; Tong et al., 2024). These benchmarks show that strong aggregate performance can coexist with systematic failures on carefully selected

visual cases. However, most are fixed evaluation sets focused on what to measure, whereas our work studies how targeted stress tests can be constructed, screened, and maintained as models and evaluation needs evolve. Automated and Generative Benchmark Construction. Prior work reduces benchmark construction costs by automatically generating questions, answers, and distractors from existing image collections. For example, POPE derives objectexistence questions from image annotations, while AutoConverter transforms existing VQA questions into challenging multiple-choice evaluations (Li et al., 2023; Zhang et al., 2025). Although these methods automate substantial annotation work, they cannot control the visual evidence contained in the source images. Image generation, editing, and procedural rendering address this limitation by directly constructing evaluation images. ImageNet-D and JourneyBench use generated images to introduce controlled or unusual visual conditions, while Vision-Language Bootstrapping, Auto-Comp, and InfiniBench use dynamic image editing, synthetic image pairs, and customizable 3D scenes (Zhang et al., 2024; Wang et al., 2024; Yang et al., 2025; Sbrolli et al., 2026; Wang et al., 2026). However, generative construction makes sample validity dependent on whether each image realizes its intended specification. Our pipeline therefore integrates structured sample specifications, image generation and targeted editing, pressure screening, human verification, annotation revision, and localized repair. It also supports generated images and uploaded real images within the same curation workflow. World Priors and Visual Evidence Conflicts. A growing body of work studies whether VLMs follow visual evidence when it conflicts with language or world knowledge. The PhD benchmark includes a counter-common-sense subset, PhD-ccs, that uses AI-generated images violating everyday expectations to evaluate visual hallucination (Liu et al., 2025). VLind-Bench measures language priors using DALL-E 3-generated counterfactual images together with auxiliary tests of commonsense knowledge, visual perception, and commonsense bias, thereby reducing alternative explanations for a model’s failure (Lee et al., 2025). ViLP constructs out-of-distribution image–question–answer triplets in which one condition can be answered from textual priors while the others require evidence from

generated images (Luo et al., 2025). VLMBias makes controlled counterfactual modifications to familiar subjects, such as well-known logos, animals, flags, and game pieces, to test whether memorized canonical knowledge overrides visual counting and identification (Vo et al., 2026). HallusionBench is broader than a world-prior benchmark: it uses human-collected and human-edited control pairs to diagnose both language hallucination and visual illusion, although some edited cases explicitly place visual content in conflict with common knowledge (Guan et al., 2024). These benchmarks directly motivate our world-prior case study, and we include them as comparison benchmarks in our experiments. Our primary goal, however, is not to introduce another isolated test of prior reliance, but to use this demanding setting to instantiate and evaluate a general pipeline for scalable stress-test construction.

3

Method

3.1

Overview and Problem Formulation

Our goal is to automatically create a VLM stresstest benchmark based on the pipeline. The pipeline takes a Test Primer, denoted by S, which consists of a natural-language Markdown Task Design, a Data Schema that defines task-specific fields and validation rules, and question format, such as open generation or multiple choice. Given S, the pipeline automatically constructs a set of Candidate Samples before human verification: C(S) = {xi }N i=1 . Each candidate xi = (Ii , qi , yi ) consists of an image Ii , a question qi , and its reference answer yi . As illustrated in Figure 1, an LLM first converts the test primer S into Sample Specifications. Each sample specification is a JSON file containing the information required by the subsequent steps, such as image generation and editing prompts. Let zi denote the i-th sample specification. An image generation model, denoted by G, generates the required images. When a design requires an image modification, an image editing model, denoted by E, edits the generated image. After candidate set construction, the candidate samples enter Automated Filtering. A VLM, denoted by M and called the Filtering VLM, serves as the Automated Filtering and evaluates each candidate sample. Candidate samples answered correctly

Attribute

Texture

Wood Mallet → Fabric Mallet

USB Drive→ Lighter

Is the dentist using a teaspoon?

Zoom In

Edit

Plastic → Fabric

Does the mallet appear to have a fabric surface?

Zoom In

Is there a black lighter visible in this digital forensics hardware-imaging lab?

Zoom In

Edit

Folding Feeler → Nail Clipper

Zoom In

Edit

Does the traffic cone appear to have a fabric surface?

Zoom In

Metal wrench → Form wrench

How many tentacles are visible on the octopus?

How many circular lamps are visible on the traffic light?

How many tines does this fork have?

World Prior: 8 Test Edition: 12

World Prior: 3 Test Edition: 5

World Prior: 4 Test Edition: 6

Is there a nail clipper visible in this engine repair workshop?

Edit

Is there a foam wrench on the workbench in the auto repair shop?

Zoom In

Edit

Language Elicitation

Context

Dentist Mirror → Teaspoon

Edit

Background: It’s early in the morning at a quiet office desk, and someone has just come back from the break area after setting down their bag and opening a laptop for the day. A mug has been placed beside the notebook and pen, ready for the first few minutes of work. Question: What drink is most likely inside the mug? Choose: A. Coffee B. Tea C. Water D. Unknown

Figure 2: Samples of SABRE-Prior in 4 different subsets.

by M are filtered out, while those answered incorrectly are retained and sent to our Convenient Human Evaluation Platform, for human verification and image repair. Candidate Samples that pass human verification become Samples and are included in the final benchmark. 3.2

Modular Task Recipes

A test primer S defines one stress-test topic. It contains a natural-language Markdown task design, an data schema, and a selected question format. The Markdown file states what to test and how to construct the images. It can specify the scene, the visual content to add, remove, or modify, and the content that must remain unchanged. The question format specifies how the VLM should answer, such as open generation or multiple choice. The data schema controls the structure of the JSON output. It defines the data types (e.g., text or lists) and other constraints. The pipeline uses the default data schema when the task recipe does not provide a custom one. A design LLM reads S and produces sample specifications in JSON. Each sample specification contains the information needed to construct one candidate sample, including an image generation or editing prompt, a question, and its reference answer. A new stress-test topic only requires a new test primer. The pipeline reuses the same image generation, pressure screening, human verification, image repair, and export workflow for all topics. This separation provides extensibility without requiring a new benchmark-construction pipeline for each topic. 3.3

Automatic Candidate Sample Construction

Each sample specification zi contains an image prompt, a question qi , and a reference answer yi . For a single-image task, the image generation

model G generates an image Ii from the prompt pi : Ii = G(pi ). Some stress-test topics test whether a VLM responds to a specific visual change, such as removing an object or changing an object attribute. These topics require image editing. The edit changes the target visual evidence while keeping the rest of the scene unchanged. This isolates the target change and tests whether the VLM updates its answer based on that change. For an editing task, zi contains a base-generation prompt pbi and an editing prompt pei . The pipeline uses G to generate the base image Iib and the image editing model E to produce the edited image Iie : Iib = G(pbi ),

Iie = E(Iib , pei ).

The editing prompt states what to change and what to preserve. A real image can replace Iib , which skips the image generation step. The pipeline pairs each applicable image with one question and its reference answer. Each sample specification produces exactly one candidate sample: x = (I, q, y). After human verification, each accepted sample forms or joins a case. A case contains one sample for one-to-one tasks and multiple samples when several image–question pairs derive from the same underlying image construction or visual intervention. The pipeline stores the prompts, model identifiers, and generation records as metadata. 3.4

Model-in-the-Loop Pressure Screening

Automated filtering removes candidate samples that the Filtering VLM M answers correctly and retains those that it answers incorrectly. For each candidate sample xi = (Ii , qi , yi ), M produces a response ri = M (Ii , qi ).

ei ∈ {0, 1},

where ei = 1 indicates a correct response and ei = 0 indicates an incorrect response. The automated filtering retains only the candidate samples that M answers incorrectly: Cpressure (S; M ) = {xi ∈ C(S) | ei = 0}. The filtering VLM measures candidate sample’s difficulty, not candidate sample’s validity. An incorrect response from M may reveal a genuine model failure, but it may also result from an unsuccessful image generation or edit, an ambiguous question, or an incorrect reference answer. Every retained candidate sample must therefore undergo human verification before it can become a sample. 3.5

Human Verification and Localized Image Repair

Candidate samples retained by the automated filtering enter human verification platform. Reviewers inspect the image Ii , question qi , reference answer yi , sample specification zi . A candidate sample passes verification only when the required visual evidence is present, the requested edit is correctly applied when needed, the question is unambiguous, and the reference answer matches the image. Reviewers can accept or reject a candidate sample, revise its question or reference answer, or repair a local image defect. For image repair, the reviewer selects a bounding box bi and provides a repair instruction ui . The platform expands the selected region to include its surrounding visual context and crops the resulting patch Pi from the image. It marks the target region inside the patch and sends the marked patch, rather than the full image, to Gemini 3.1 Flash Image Preview with ui . Restricting the model input to a local patch focuses the edit on the selected defect and prevents unintended changes to the rest of the image. The repaired patch Pei cannot be directly pasted into the image because a hard boundary may introduce visible color and texture discontinuities. We therefore construct a soft mask Ai by expanding the target region and applying Gaussian blur to its boundary. The platform blends the repaired and

SABRE-Counting

ei = Eval(ri , yi ),

SABRE-Spatial

The evaluator compares ri with the reference answer yi using the scoring rule for the selected question format:

# of yellow parking cones: 24

# of salmon nigiri: 21

# of whole mushrooms: 55

# of red cap test tube: 76

# of yellow bike helmet: 51

# of eggs: 56

Coordinate system: the object slot in the frontleft-bottom grid cell is coordinate (1,1,1), not a geometric corner point. The x axis increases from left to right, the y axis increases from front to back, and the z axis increases from bottom to top. Coordinates are positive integers. What color and shape is the object at coordinate (3,1,2)? Answer only as two words: color shape. z

GT Answer: Blue Cube

y x

Figure 3: SABRE-counting and SABRE-spatial

original patches as Pi⋆ = Ai ⊙ Pei + (1 − Ai ) ⊙ Pi . It then places Pi⋆ back into the original image. The center of the mask preserves the repaired content, while the smooth boundary gradually transitions to the original pixels. This soft blending reduces rectangular seams and integrates the repaired region more naturally into the surrounding scene. The reviewer compares the repaired image with the original and may accept it, repair it again, or reject the candidate sample. Our questionnaire study shows that the localized repair tool produces more successful and visually natural repairs than direct whole-image Gemini editing and the other evaluated methods. Participants consistently preferred the results produced by our tool. The full comparison and questionnaire results are provided in the appendix. Let x⋆i denote the candidate sample after human verification and any reviewer revisions or image repairs. The reviewer assigns a validity decision vi ∈ {0, 1}, where vi = 1 indicates acceptance. The final benchmark is D(S; M ) = {x⋆i | xi ∈ Cpressure (S; M ), vi = 1} . Each item in D(S; M ) originates from a pressureselected candidate sample and passes human verification. We call each retained item a sample. The curation platform also supports real-image authoring. Users can upload a real image, provide or revise its question and reference answer, and apply localized image repair when needed.

RANK

MODEL

CONTEXT

TEXTURE

ATTRIBUTE

LANGUAGE

MACRO AVG.

1

Claude 4.6

10

40

17

58

31.3

2

Kimi-k2.6

7

52

17

17

23.3

3

Qwen 3.5

3

46

14

29

23.0

4

Gemini 3.5

0

52

26

11

22.3

5

GPT-5.4

1

28

20

23

18.0

6

Grok-4.3

4

28

16

23

17.8

Figure 4: Accuracy (%) of six frontier VLMs on SABRE-Prior. Bars show point estimates, and whiskers indicate the corresponding confidence intervals. Context and Texture report strict All4 accuracy: a case is correct only when all four Base–Edited yes/no probes are answered correctly. Attribute uses normalized exact-match counting accuracy, while Language Elicitation uses four-option multiple-choice accuracy. Macro Average is computed across the four subset accuracies.

4

Benchmark Instantiation

We first use our pipeline to construct SABRE-Prior, a benchmark that tests whether VLMs follow visual evidence or learned expectations about what is usually true. We refer to these learned expectations as world priors. SABRE-Prior Stress-Test Subsets. The benchmark contains four complementary subsets. Context places unexpected objects in familiar scenes, such as replacing a microscope in a laboratory with a toaster. Language Elicitation uses question wording to suggest an answer that is unsupported by the image, such as asking what is inside a closed box. Texture assigns familiar objects counterfactual materials, such as giving a ceramic mug a furry surface. Attribute modifies canonical component counts while preserving object identity, such as showing a chair with five legs. Context and Texture use paired base–edit images with four complementary yes/no probes. Language Elicitation uses single-image, four-option questions whose correct answer is unknown, while Attribute uses open-ended counting questions over edited images. SABRE-Prior Construction, Scale, and Evaluation. All four subsets use the shared workflow in Section 3. For each subset, GPT-5.4 reads its task primer S and produces sample specifications. FLUX.2 [flex], FLUX.2 [pro] (Black Forest Labs, 2025), and Gemini 3.1 Flash Image (Nano Banana 2) (Raisinghani, 2026) generate or edit the images specified by these specifications. Gemini 3.5 Flash serves as the filtering VLM M . Candidate sam-

ples retained by the automated filtering undergo human verification and localized repair when needed. Each subset contains 100 cases. The final benchmark contains 400 cases, 600 images, and 1,000 samples, with one question per sample. Context and Texture are evaluated using paired Base and Edited images. Context asks whether the source and target objects are visible in each image, with expected answers yes/no for Base and no/yes for Edited. Texture uses the same fourprobe structure for the expected and counterfactual materials, again producing yes/no for Base and no/yes for Edited. A case is counted as correct only when all four yes/no samples are correct. Attribute presents a single edited image with an open-ended component-counting question; the extracted numeric response must exactly match the annotated visible count after normalizing digits and number words. Language Elicitation uses a single-image, four-option question whose wording suggests an answer that cannot be verified visually; the correct option is unknown, with its position balanced across the dataset. We report each subset’s primary accuracy and their macro-average as the overall score. 4.1

Additional Pipeline Instantiations

To evaluate the extensibility of our pipeline, we use the same workflow to construct two additional 20-example pilots: Counting and Spatial. Counting presents dense scenes containing dozens of target objects and visually similar distractors, and asks the model to report the number of visible target objects. Responses are scored by normalized exact count. Spatial presents dense multi-layer 3D lat-

Source Image

Edited Image

Source Image

Edited Image

Source Image

Edited Image

One Handle

Two Handles

Four Strings

Six Strings

Four Legs

Ten Legs

Awl

Crayfish

Wrench

Gecko

Wood board

Fabric board

Fabric Cap

Leather Cap

Handle

Rubber boot

Crab

Cardboard boot

Figure 5: Examples of generated images from SABRE

Figure 6: Additional examples of generated images from SABRE. The first row shows Spatial QA instances, where the question-answer format follows the same structure as illustrated in Figure 3. The second row presents Counting instances, including examples that require counting pink slippers, pens, oranges, and mugs from left to right. The third row provides additional Attribute instances.

tices and includes multiple question formats, such as identifying the color and shape at a specified coordinate or counting objects that satisfy spatial and visual constraints, for example red cubes on the top layer. Responses are scored by normalized exact attribute matching or exact count, depending on the question type.

5

Experiments

5.1

Experimental Setting

We evaluate six frontier VLMs on SABRE-Prior: Gemini 3.5 Flash (Google DeepMind, 2026), GPT-

5.4 (OpenAI, 2026), Claude 4.6 Sonnet (Anthropic, 2026), Kimi-k2.6 (Moonshot AI, 2026), Qwen 3.5 27B (Qwen Team, 2026), and Grok-4.3 (xAI, 2026). These models represent the current state of the art, and our stress tests are designed to identify their blindspots for future improvement. All models are evaluated with zero-shot prompts using greedy decoding (i.e., temperature = 0) when supported. We estimate 95% confidence intervals using a percentile bootstrap with 20,000 resamples, resampling cases within each subset. For each resample, we recompute every SABRE-Prior subset accuracy and their macro-average, defined

Benchmark

Images

Construction Method

Question Types

Gemini 3.5 Flash Acc. (%)

PhD-CCS VLind-Bench ViLP HallusionBench VLMBias

750 842 900 181 1,392

DALL-E 3 DALL-E 3 DALL-E 3 / FLUX Manual editing Semi-automated generation

Yes/No Yes/No Open identification Yes/No Counting and Yes/No

82.3 90.0 70.3 81.6 60.3

SABRE-Prior

600

Automated generation

All above and MCQ

22.3

Table 1: Same-model comparison using each benchmark’s primary evaluation protocol

as the unweighted mean of the subset accuracies. Detailed model identifiers, access dates, prompts, and response-parsing rules are provided in the Appendix. Baselines We additionally evaluate VCD (Leng et al., 2024) and SoM (Yang et al., 2023) on the open-source Qwen 3.5 27B model to examine whether visual-enhancement methods improve performance. We also construct a real-image variant of Attribute subset to test whether model failures persist when the edited source images are real rather than generated. We also show more examples shown in 5 and 6. 5.2

Main Results

Figure 4 reports the primary subset scores of the six evaluated VLMs. Macro-average accuracy ranges from 17.8% to 31.3%, with a mean of 22.6% across models. Because SABRE-Prior uses pressure screening and evaluates different behaviors with different question formats, the macro-average summarizes performance on this benchmark rather than providing a general ranking of model capability. Table 1 provides complementary evidence. Gemini 3.5 Flash achieves 60.3%–90.0% accuracy on five existing benchmarks but only 22.3% on SABRE-Prior. Its low score on SABRE-Prior partly reflects its role as the Filtering VLM. However, the other five models, which were not used during pressure screening, also score below 32%. These results show that the constructed cases expose failures across frontier VLMs rather than only the Filtering VLM. Context obtains the lowest scores under the strict criterion: no model exceeds 10%, and the mean is 4.2%. This reflects the difficulty of consistently recognizing both the original and counterfactual objects across the Base–Edit intervention. Texture scores are higher, ranging from 28% to 52%, suggesting that visible material evidence is more accessible than context-incongruent object identity,

although canonical material associations still exert substantial pressure. Attribute ranges from 14% to 26%, combining fine-grained structural perception and exact counting with the need to override canonical component counts. Language Elicitation shows the widest variation, from 11% to 58%. This subset should not be interpreted as a pure measure of visual recognition: the queried information is intentionally unavailable in the image. Instead, it tests whether a model recognizes insufficient visual evidence and rejects a plausible answer induced by the question wording. The large variation therefore reveals differences in visual grounding, prior reliance, and willingness to abstain. Models perform very differently across subsets. Claude 4.6 achieves the highest macro-average, largely because of its strong Language Elicitation score. Gemini 3.5 Flash and Kimi-k2.6 perform best on Texture, while Gemini 3.5 Flash leads on Attribute but answers no Context cases correctly. The model ranking therefore changes across subsets, and no model performs best on all of them. These results show that SABRE-Prior exposes distinct model weaknesses that a single aggregate score would hide. 5.3

Comparison with Existing Benchmarks

We further compare SABRE-Prior with five related benchmarks using Gemini 3.5 Flash. As shown in Table 1, the model achieves between 60.33% and 90.02% on PhD-CCS, VLind-Bench, ViLP, HallusionBench, and VLMBias, but only 22.30% on SABRE-Prior. In particular, its accuracy exceeds 80% on three of the existing benchmarks, suggesting that many previously constructed worldprior and hallucination cases are handled reliably by recent frontier models. In contrast, the cases produced by our pipeline continue to expose substantial failures. The benchmarks use different question formats and scoring rules, so their accuracy values are not directly equivalent. However, the results show that

Method Qwen 3.5 27B + VCD + SoM

Ctx

Tex

Att

Lang

Avg.

Image Source

3 1 6

46 34 24

14 16 12

29 27 25

23.0 19.5 16.8

Generated (Attribute Modify) Real-Image Control

Table 2: Effect of visual-enhancement methods on Qwen 3.5 27B. Results use the same four subset metrics as Figure 4; Avg. denotes their macro-average.

5.4

Effect of Visual-Enhancement Methods

A useful stress-test benchmark should not only expose model failures but also measure whether proposed methods fix them. We use SABREPrior to evaluate two visual-enhancement methods on Qwen 3.5 27B. Visual Contrastive Decoding (VCD) contrasts predictions from original and perturbed visual inputs, while Set-of-Mark (SoM) adds region markers to improve visual grounding. We evaluate both methods using the same questions and subset metrics as the standard Qwen 3.5 27B setting. As shown in Table 2, neither method consistently improves performance. VCD and SoM improve selected subsets but reduce accuracy on others, resulting in lower macro-averages than the standard model. These results indicate that the failures exposed by SABRE-Prior cannot be fixed simply by highlighting visual regions or changing the decoding signal. This experiment also demonstrates the diagnostic value of SABRE-Prior. It shows where a mitigation method helps, where it introduces regressions, and which model weaknesses remain unresolved. The benchmark can therefore support the development and evaluation of methods designed to ad-

Accuracy

26 / 100 6 / 20

26% 30%

Table 3: Real-image control. Exact-match counting accuracy of Gemini 3.5 Flash on generated and realimage Attribute samples. MODEL

several existing benchmarks provide limited headroom for a current frontier VLM. The low Gemini 3.5 Flash score on SABRE-Prior partly reflects its role as the Filtering VLM, but it also demonstrates that our pipeline can construct new cases that target weaknesses not exposed by increasingly saturated benchmarks. Unlike a fixed test set, our pipeline can generate new stress-test candidates on demand using updated task recipes and Filtering VLMs. This allows the evaluation to adapt as models improve. The pipeline supports flexible question formats, including yes/no, multiplechoice, and open-ended counting questions. It can therefore produce scalable stress tests across different task formats rather than remaining tied to one predefined benchmark.

Correct / Total

COUNTING

SPATIAL

Claude 4.6

1/20

0/20

Kimi-k2.6

0/20

0/20

Qwen 3.5

1/20

0/20

Gemini 3.5

0/20

0/20

GPT-5.4

1/20

0/20

Grok-4.3

1/20

0/20

Figure 7: Results on two additional stress-test instantiations. Each cell represents one sample, and colored cells indicate correct model responses.

dress specific VLM failures. 5.5

Real-Image Control

We test whether the low model accuracy on SABRE-Prior is caused by generated-image artifacts. We construct 20 control cases from real images using the same editing and evaluation procedure as the Attribute. As shown in Table 3, Gemini 3.5 Flash achieves 30% accuracy on the real-image cases, compared with 26% on the generated-image cases. The similarly low accuracies suggest that the benchmark difficulty is not primarily caused by generated-image artifacts. They also support the reliability of the generated stress tests, which expose model failures at a level comparable to tests constructed from real photographs. The pipeline supports both real and generated source images, while generated images provide greater scalability and more precise control over the visual content being tested. 5.6

Additional Stress-Test Instantiations

We use two different test primers, to construct additional stress tests for counting and spatial reasoning. Each stress test contains 20 samples and uses the same generation, pressure screening, human verification, and repair workflow. Figure 7 shows that both stress tests are challenging for all six VLMs. Each model answers at most one of the 20 Counting samples correctly, and none answers any Spatial sample correctly. These results contrast with the strong performance of frontier VLMs on many established benchmarks, which can become saturated

as models improve. They also demonstrate the extensibility of our pipeline: different task specifications can be used to construct challenging stress tests for new visual capabilities without changing the shared workflow.

6

Conclusion and Limitations

We introduced SABRE, a scalable and modular pipeline for constructing VLM stress tests from natural-language task specifications. The pipeline converts each task specification into structured case specifications, constructs the required images and question–answer pairs, retains challenging candidates through failure-only pressure screening, and supports human verification and localized repair. Using this pipeline, we constructed SABRE-Prior, covering four forms of conflict between visual evidence and world priors. Six recent VLMs achieve only 17.8%–31.3% macro accuracy and exhibit distinct failure profiles. Representative Inferencetime mitigation methods provide no consistent improvement, suggesting that the benchmark exposes persistent weaknesses in using visual evidence to override prior expectations rather than errors that can be removed by generic inference-time interventions. We also apply the same pipeline to Counting and Spatial reasoning through separate task specifications, demonstrating its use across different stress-test settings. A limitation shared by any fixed benchmark suite is that its coverage reflects a finite set of task specifications at the time of release. The modular design of SABRE mitigates this limitation by allowing new task specifications and construction modules to be added as VLM capabilities evolve.

References Anthropic. 2026. Introducing Claude Sonnet 4.6. https://www.anthropic.com/news/ claude-sonnet-4-6. Accessed: 2026-07-20. Black Forest Labs. 2025. FLUX.2: Frontier visual intelligence. https://bfl.ai/blog/flux-2. Accessed: 2026-07-20. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, Rongrong Ji,

Caifeng Shan, and Ran He. 2025. Mme: A comprehensive evaluation benchmark for multimodal large language models. Preprint, arXiv:2306.13394. Chaoyou Fu, Yi-Fan Zhang, Shukang Yin, Bo Li, Xinyu Fang, Sirui Zhao, Haodong Duan, Xing Sun, Ziwei Liu, Liang Wang, Caifeng Shan, and Ran He. 2024. Mme-survey: A comprehensive survey on evaluation of multimodal llms. Preprint, arXiv:2411.15296. Dhruba Ghosh, Hanna Hajishirzi, and Ludwig Schmidt. 2023. Geneval: An object-focused framework for evaluating text-to-image alignment. Preprint, arXiv:2310.11513. Google DeepMind. 2026. Gemini 3.5 flash: Model card. https://deepmind.google/models/ model-cards/gemini-3-5-flash/. Accessed: 2026-07-20. Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. 2024. Hallusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. Preprint, arXiv:2310.14566. Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A. Smith. 2023. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 20349–20360. Kaiyi Huang, Chengqi Duan, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. 2025. T2icompbench++: An enhanced and comprehensive benchmark for compositional text-to-image generation. Preprint, arXiv:2307.06350. Kang-il Lee, Minbeom Kim, Seunghyun Yoon, Minsung Kim, Dongryeol Lee, Hyukhun Koh, and Kyomin Jung. 2025. VLind-bench: Measuring language priors in large vision-language models. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 4129–4144, Albuquerque, New Mexico. Association for Computational Linguistics. Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2024. Mitigating object hallucinations in large visionlanguage models through visual contrastive decoding. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13872– 13882. Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. 2023. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 292–305, Singapore. Association for Computational Linguistics. Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona,

Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. 2015. Microsoft coco: Common objects in context. Preprint, arXiv:1405.0312. Jiazhen Liu, Yuhan Fu, Ruobing Xie, Runquan Xie, Xingwu Sun, Fengzong Lian, Zhanhui Kang, and Xirong Li. 2025. Phd: A chatgpt-prompted visual hallucination evaluation dataset. Preprint, arXiv:2403.11116. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2024. Mmbench: Is your multi-modal model an all-around player? Preprint, arXiv:2307.06281. Tiange Luo, Ang Cao, Gunhee Lee, Justin Johnson, and Honglak Lee. 2025. Probing visual language priors in VLMs. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 41120–41156. PMLR. Moonshot AI. 2026. Kimi K2.6: Advancing opensource coding. https://www.kimi.com/blog/ kimi-k2-6. Accessed: 2026-07-20. OpenAI. 2026. Introducing GPT-5.4. https:// openai.com/index/introducing-gpt-5-4/. Accessed: 2026-07-20. Qwen Team. 2026. Qwen3.5: Towards native multimodal agents. https://qwen.ai/blog?id=qwen3. 5. Accessed: 2026-07-20. Naina Raisinghani. 2026. Nano banana 2: Combining pro capabilities with lightning-fast speed. https://blog.google/innovation-and-ai/ technology/ai/nano-banana-2/. Accessed: 2026-07-20. Cristian Sbrolli, Matteo Matteucci, and Toshihiko Yamasaki. 2026. Auto-comp: An automated pipeline for scalable compositional probing of contrastive vision-language models. Preprint, arXiv:2602.02043. Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. 2024. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9568–9578. An Vo, Khai-Nguyen Nguyen, Mohammad Reza Taesiri, Vy Tuong Dang, Anh Totti Nguyen, and Daeyoung Kim. 2026. Vision language models are biased. Preprint, arXiv:2505.23941. Haoming Wang, Qiyao Xue, and Wei Gao. 2026. Infinibench: Infinite benchmarking for visual spatial reasoning with customizable scene complexity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21594–21605.

Zhecan Wang, Junzhang Liu, Chia-Wei Tang, Hani Alomari, Anushka Sivakumar, Rui Sun, Wenhao Li, Md. Atabuzzaman, Hammad Ayyubi, Haoxuan You, Alvi Ishmam, Kai-Wei Chang, Shih-Fu Chang, and Chris Thomas. 2024. Journeybench: A challenging onestop vision-language understanding benchmark of generated images. In Advances in Neural Information Processing Systems, volume 37, pages 63110– 63123. Curran Associates, Inc. xAI. 2026. Grok 4.3. https://docs.x.ai/ developers/models/grok-4.3. Accessed: 202607-20. Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. 2023. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. Preprint, arXiv:2310.11441. Yue Yang, Shuibo Zhang, Kaipeng Zhang, Yi Bin, Yu Wang, Ping Luo, and Wenqi Shao. 2025. Dynamic multimodal evaluation with flexible complexity by vision-language bootstrapping. In International Conference on Learning Representations, volume 2025, pages 21399–21420. Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2024. MM-vet: Evaluating large multimodal models for integrated capabilities. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 57730–57754. PMLR. Chenshuang Zhang, Fei Pan, Junmo Kim, In So Kweon, and Chengzhi Mao. 2024. Imagenet-d: Benchmarking neural network robustness on diffusion synthetic object. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21752–21762. Yuhui Zhang, Yuchang Su, Yiming Liu, Xiaohan Wang, James Burgess, Elaine Sui, Chenyu Wang, Josiah Aklilu, Alejandro Lozano, Anjiang Wei, Ludwig Schmidt, and Serena Yeung-Levy. 2025. Automated generation of challenging multiple-choice questions for vision language model evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 29580– 29590.

A

Detailed Experimental Setup

Evaluated Models. Table 4 lists the exact model identifiers and inference interfaces used in our experiments. All models were accessed on July 20, 2026. API models were evaluated using the versions exposed under these identifiers on that date. Each model received one image–question sample at a time without in-context examples. Inference Prompts. We use zero-shot, format-specific prompts and do not request explanations or chain-of-thought reasoning. Context and Texture use the following prompt: Question: {question} Answer with only one word: yes or no.

Attribute and Counting use: Question: {question} Answer with only the count or shortest possible count phrase. Do not explain.

Language Elicitation presents the question followed by four options labeled A–D and uses: Question: {question and options} Answer with only the option letter. Do not explain.

Spatial uses the same short-answer instruction as Counting. Depending on the question, the expected response is either a count or a short combination of attributes such as a color and shape. For the local Qwen evaluation, these instructions are wrapped in the model’s standard chat template with a format-only system message requesting no reasoning or additional text. Decoding and Image Input. The maximum output length is restricted to 8–16 tokens for API models, depending on the response format. Qwen uses at most 32 newly generated tokens. Images are supplied at their original stored resolution to API models. GPT-5.4 uses the high image-detail setting. For local Qwen inference, images are converted to RGB and processed using the checkpoint’s AutoProcessor. Qwen is evaluated with do_sample=false, bfloat16 precision, batch size 1, and automatic device placement. The four subsets are distributed across four CUDA devices, with the subsets evaluated independently. Response Parsing. Yes/no responses are lowercased and mapped to yes or no when the response begins with the corresponding word. If necessary, we accept the answer when exactly one of the two words occurs in the response; ambiguous or unparseable outputs are marked incorrect. Multiple-choice responses are parsed as the first standalone letter from A to D. For open-count questions, responses are lowercased, punctuation is removed, and number words from zero to twenty are mapped to digits. We then extract the first valid numeric count. Equivalent forms such as “seven” and “7” are therefore treated identically, while explanations or other text do not receive partial credit. Spatial attribute responses are normalized for capitalization, spacing, and punctuation before exact matching. An output that cannot be normalized to the required response format is counted as incorrect. Scoring. Attribute, Language Elicitation, Counting, and Spatial are scored independently at the sample level. Context and Texture contain four samples for each Base–Edited case: two questions are asked about the Base image and two about the Edited image. A case is correct only if all four responses are correct, which we refer to as strict All4 accuracy. Language Elicitation is evaluated using four-option multiple-choice accuracy, with the position of the unknown answer balanced across the dataset. Attribute and Counting use normalized exact-count accuracy. Spatial uses normalized exact count or exact attribute matching according to the question type. The overall SABRE-Prior score is the unweighted macro-average of the Context, Texture, Attribute, and Language Elicitation subset accuracies. Confidence Intervals. We compute 95% confidence intervals using a percentile bootstrap with 20,000 resamples and random seed 42. Resampling is performed over cases within each subset. Thus, the four samples belonging to a Context or Texture case remain grouped and are resampled together. For every bootstrap replicate, we recompute the four subset accuracies and their unweighted macro-average. The reported interval endpoints are the 2.5th and 97.5th percentiles of the resulting bootstrap distributions.

Reported Name

Model Identifier

Interface

Inference Configuration

Gemini 3.5 Flash GPT-5.4

gemini-3.5-flash gpt-5.4

Claude 4.6 Kimi-k2.6

claude-sonnet-4-6 kimi-k2.6

Qwen 3.5 27B

Qwen3.5-27B

Grok-4.3

grok-4.3

Google Gen AI API Temperature 0; thinking budget 0 OpenAI Responses API Temperature 0; reasoning effort none; image detail high OpenRouter Temperature 0 Official Moonshot API Thinking disabled; provider-supported deterministic configuration Local Hugging Face Greedy decoding; thinking disabled; bfloat16 Transformers OpenRouter Temperature 0

Table 4: Models and inference configurations used in our experiments. Greedy decoding is used whenever supported by the corresponding inference interface. Field

Type

Purpose

concept_id

String

pressure_test_type objective generation_mode base_prompt edit_prompt image_prompt task_attributes

String String Enum String String String List

questions

List

review_targets

List

generation_notes

List

Unique identifier used to reject duplicate specifications and connect subsequent construction records. Name of the stress-test topic, such as context_prior. Capability or failure mode that the specification is intended to test. Either paired_edit or single_image. Prompt used to construct the Base image for a paired intervention. Local transformation applied to the Base image to produce the Edited image. Prompt used for tasks that require only one generated image. Task-specific structured values, such as source entity, target entity, material, count, or spatial relation. Question specifications containing the probe identifier, image role, question type, prompt, reference answer, evaluation rule, and optional multiple-choice options. Instructions identifying the visual evidence and image region that a human reviewer should verify. Additional constraints used during image generation, editing, and review.

Table 5: Principal fields in the default Sample Specification schema.

B

Test Primers and Sample Specifications

A benchmark designer begins by providing a Test Primer, which consists of a natural-language Task Design written in Markdown and a Data Schema that defines the required output fields and validation rules. The Task Design describes the capability or failure mode to test, the visual construction requirements, and the desired question format. The schema converts these requirements into a machine-verifiable contract. A designer may use the default schema provided by SABRE or replace it with a task-specific Pydantic schema when additional fields or validation rules are required. In our implementation, GPT-5.4 receives the Task Design together with the schema and generates one structured Sample Specification at a time. The output is accepted only if it satisfies the schema; invalid outputs and duplicate concept identifiers are returned to the design model with validation feedback for regeneration. Valid specifications are stored as JSONL records and passed to the image-construction and question-building modules. Default Data Schema. Table 5 summarizes the principal fields in the default Sample Specification schema. The schema supports paired image editing and single-image generation, as well as yes/no, multiple-choice, and open-generation questions. The schema also enforces cross-field constraints. A paired_edit specification must contain both a Base prompt and an edit prompt, and its questions may refer only to the Base or Edited image. A single_image specification must instead contain an image prompt, and all questions must refer to that image. Yes/no questions require yes_no_exact evaluation and a reference answer of either yes or no; multiple-choice answers must identify one of the provided options; and open-generation questions must specify an exact-match, counting, or containment-based evaluation rule.

Sample Specification

Context / paired_edit

{ "concept_id": "ctx01_repack_tape_bagel", "pressure_test_type": "context_prior", "generation_mode": "paired_edit", "objective": "Test whether a VLM follows visible evidence when an expected object is replaced by an unexpected object.", "base_prompt": "Generate a documentary photograph of a busy warehouse repacking floor. Include exactly one tan roll of masking tape on the lower shelf. No bagel or additional tape roll is visible. ...", "edit_prompt": "Replace only the masking tape with one plain bagel at the same location. Preserve all other objects, people, lighting, camera properties, and scene content. ...", "task_attributes": [ {"name": "source_entity", "value": "roll of masking tape"}, {"name": "target_entity", "value": "plain bagel"}, {"name": "review_location", "value": "lower shelf of the central repacking bench"} ], "questions": [ {"probe_id": "base_source", "image_role": "base", "question_type": "yes_no", "prompt": "Is a roll of masking tape visible?", "answer": "yes", "eval_type": "yes_no_exact"}, {"probe_id": "base_target", "image_role": "base", "question_type": "yes_no", "prompt": "Is a plain bagel visible?", "answer": "no", "eval_type": "yes_no_exact"}, {"probe_id": "edited_source", "image_role": "edited", "question_type": "yes_no", "prompt": "Is a roll of masking tape visible?", "answer": "no", "eval_type": "yes_no_exact"}, {"probe_id": "edited_target", "image_role": "edited", "question_type": "yes_no", "prompt": "Is a plain bagel visible?", "answer": "yes", "eval_type": "yes_no_exact"} ], "review_targets": [ {"label": "source object", "image_role": "base", "description": "Verify one tape roll at the specified location."}, {"label": "target object", "image_role": "edited", "description": "Verify that one bagel replaces the tape roll."} ] }

Figure 8: Abbreviated Context Sample Specification. The record connects visual construction, question definition, and human verification within a schema-validated specification. Long prompts are shortened for presentation.

Context Task Design. We use the Context subset as a running example. Its Task Design instructs the design model to construct a natural Base scene containing one contextually expected source entity and no target entity. The Edited image must replace only the source entity with a visually compatible but contextually unexpected target while preserving the rest of the scene. The Task Design also defines four complementary yes/no probes: 1. Source entity visible in Base: yes. 2. Target entity visible in Base: no. 3. Source entity visible in Edited: no. 4. Target entity visible in Edited: yes.

The Markdown further requires both entities to remain human-recognizable while occupying a minor region of a visually dense scene. It prohibits duplicate source or target instances, readable labels, large isolated objects, obvious editing artifacts, and changes outside the specified region. Example Context Sample Specification. The following abbreviated record is an actual schemacompatible specification generated for the Context task. Long image prompts are shortened only for presentation 8. Question Instantiation. The questions are specified before image construction; the Question Builder does not use another model to generate them. Once the required images are available, it associates each question with the appropriate image through its image_role, assigns a unique identifier, and exports the canonical records used for filtering, human review, and final evaluation. For Context, the four probes share the same case identifier and Base–Edited assets and are evaluated jointly using All4. Other subsets use the same question-instantiation interface with their task-specific

Instantiation

Image Mode

Context

Paired Edited

Task-Specific tion

Construc- Question mat

For- Primary Validity Requirement

Base– Replace one contextually ex- Four yes/no The source appears only in Base, the pected source entity with a probes target appears only in Edited, and all visually compatible but connon-target content is preserved. textually unexpected target. Texture Paired Base– Change the visible surface Four yes/no Canonical and counterfactual surface cues Edited of a familiar entity from its probes are visually distinguishable, and the intervention does not change the entity or canonical material to a counterfactual material while presurrounding scene. serving its identity and geometry. Attribute Single final im- Construct a familiar entity Open-ended Every counted component is visible and age with a noncanonical number count individually distinguishable, and the annotated count matches the image of visible components while preserving its category idenexactly. tity. Language Elici- Single image Construct a scene in which Four-option The requested information cannot be tation the question suggests a plau- MCQ determined from the image, making sible answer although the unknown the uniquely supported answer. queried evidence is hidden, occluded, unreadable, or outside the frame. Counting Independently Construct dense scenes con- Open-ended Every target instance has a visible scored images taining many instances of a count boundary, distractors are distinguishable, precisely defined target catand the reference count is exact. egory together with visually similar distractors. Each question has one visually verifiable Spatial Independently Construct dense multi-layer Open-ended scored images 3D lattices with controlled attribute or count answer under an explicitly defined object colors, shapes, cocoordinate system or spatial constraint. ordinates, and spatial relations.

Table 6: Task-specific instantiations of the shared SABRE pipeline. The main SABRE-Prior benchmark contains Context, Texture, Attribute, and Language Elicitation. Counting and Spatial are reported separately as additional pipeline instantiations.

question formats and scoring rules. Complete Test Primers and schemas will be released with the benchmark-construction code.

C

Task-Specific Instantiations

We instantiate the pipeline on the four SABRE-Prior subsets and two additional pilots. Each instantiation uses a different Test Primer, visual construction requirement, question format, and validity criterion, while retaining the same construction, filtering, verification, repair, and export workflow. Table 6 summarizes these settings. Context and Texture use paired Base–Edited cases, whereas Attribute and Language Elicitation use a single final image. Counting and Spatial are evaluated at the individual-image level and do not use the All4 case-level metric. Context. A Context specification defines a dense, coherent Base scene containing exactly one source entity that is natural for the scene and no instance of the target entity. The source is placed in a precisely described review location and remains identifiable without becoming the focal object. The target is selected to have a similar apparent size, orientation, color, or silhouette so that it can occupy the same location after editing while remaining contextually unexpected. The edit instruction replaces only the source with the target and preserves the camera, people, lighting, composition, surrounding objects, and local occlusion pattern. The specification contains four question records: base_source, base_target, edited_source, and edited_target, with reference answers yes, no, no, and yes, respectively. A candidate is visually valid only when both entities are human-recognizable in their intended images, neither entity appears in the wrong image, and no unrelated scene content changes in a way that affects the questions.

Texture. A Texture specification begins with a familiar entity displaying its expected surface, such as a plastic watering can or a wooden rolling pin. The counterfactual material is chosen to conflict with the entity’s canonical appearance while providing recognizable visual evidence, such as visible fabric fibers, foam structure, glass transparency, or a paper-like surface. The intervention changes the visible material of the target entity but preserves its category, geometry, scale, pose, and location. The four probes ask whether the entity has the canonical or counterfactual surface in the Base and Edited images. Their answer pattern is again yes/no for Base and no/yes for Edited. Human validation checks that the surface change covers the intended visible region, that the material cues are sufficiently clear for direct visual judgment, and that the edit does not deform the entity or alter neighboring objects. Attribute. Attribute specifications target familiar objects whose components normally follow a canonical count, including animal legs, chair legs, shirt buttons, instrument holes, and repeated symbols on playing cards. The image is constructed or edited to contain a noncanonical number of visible components while retaining the object’s recognizable identity. The intervention must produce complete components rather than duplicated fragments, merged structures, shadows, or ambiguous partial instances. Each exported image is associated with one open-ended question asking for the visible component count. The reference answer is derived from the verified image rather than assumed from the construction prompt. During validation, the reviewer confirms the object identity, checks each visible component, and ensures that occlusion or malformed geometry does not make the count ambiguous. Language Elicitation. Language Elicitation specifications construct scenes in which contextual wording makes one answer appear plausible although the necessary evidence is not available in the image. Examples include asking about the contents of a closed opaque container, text on a face-down object, information hidden by glare or blur, or content outside the image boundary. The image prompt must ensure that the queried information is genuinely unavailable and that no incidental visual cue reveals the intended distractor. Each question contains three plausible content answers and one unknown option. The plausible alternatives are selected to reflect answers that may be suggested by the scene or question wording, rather than visually supported answers. The position of unknown is shuffled and balanced across the subset. A candidate is valid only when none of the three content options can be confirmed from the image and unknown is therefore the uniquely supported response. Counting. Counting specifications define a target category using visible properties such as object type, color, pattern, or location. The scene contains dozens of target instances distributed across natural clusters, together with distractors sharing selected properties with the target. For example, a scene may contain yellow lemons among other yellow or round objects, or four-hole blue buttons among buttons and beads with similar colors and shapes. Targets are not arranged in regular rows and may touch or overlap slightly, but each instance must retain a visible boundary. The construction process may produce Base and Edited images with different target counts. The edit adds or removes a specified number of targets while preserving the scene and distractors. Unlike Context and Texture, each image is exported as an independent open-count sample. Human verification determines the final visible count and rejects images containing merged, duplicated, heavily occluded, or otherwise uncountable target instances. Spatial. Spatial specifications construct dense multi-layer 3D lattices containing repeated colored shapes at controlled cells. A fixed coordinate convention defines the front-left-bottom cell as the origin cell, with the horizontal axis increasing from left to right, depth increasing from front to back, and height increasing from bottom to top. The same convention is stated explicitly in every coordinate question to avoid dependence on an unstated viewpoint. The task supports coordinate lookup questions, such as identifying the color and shape at a specified cell, as well as constrained counting questions, such as counting red cubes on the top layer. Specifications may also move a target object between cells while preserving the lattice, camera, and all other objects. As with Counting, Base and Edited images are evaluated independently. A candidate is valid only when

Context Repair Review Current 1/1 · Reviewed 1/1

gecko_wrench_workbench__v01 · standalone · gecko_wrench_workbench__v01 generated_without_screening

Saved: unsure

Exactly one small silver adjustable wrench. → Exactly one small realistic gecko. Review location

Front-right area of the midground workbench cluster, diagonally positioned beside and partly overlapped by a grease-stained rag. Local cluster

A crowded mixed-tool cluster containing pliers, hex keys, tire lever, socket, chain, spokes, washers, bolts, rag, and unbranded bicycle components; exactly one adjustable wrench is present in Base and no additional adjustable wrench appears anywhere. Edited visual cues

—

Human verifiability

Verify that exactly one recognizable small silver adjustable wrench is visible in the front-right midground tool cluster and that no other adjustable wrench appears anywhere in the scene.; Verify that exactly one recognizable small gecko occupies the wrench location, with toe-padded feet, head, mottled body, and tapering tail visible, and that no adjustable wrench remains anywhere.; Verify that the rag, surrounding tools, people, bicycles, camera angle, lighting, focus, composition, and all non-designated scene content are unchanged.

Base: mark the source entity

Edited: mark the inserted entity or repair region

Drag a box or click for a default box

Drag a box or click for a default box

Showing original edited image.

Decision Keep

Needs recheck

Edited replacement failed

Base invalid

Scene changed

Unsure

Notes Optional: ambiguity, inaccurate box, or rejection reason

Questions and answers

Editable benchmark content. The model prediction and result are the historical filter-gate record. Probe Question probe_01 Is there an adjustable wrench in the Base image?

Figure 9: Overview of the SABRE annonation platform. Answer yes

Filter prediction

Filter result Incorrect

probe_02

Is there a gecko in the Base image?

no

Incorrect

probe_03

Is there an adjustable wrench in the Edited image?

no

Incorrect

the lattice orientation is visually recoverable, the queried cells or layers remain visible, the target objects are distinguishable from distractors, and the reference answer can be verified without relying on the generation prompt. probe_04

Is there a gecko in the Edited image?

D

Human Verification and Curation Platform

D.1

Annotation Platform

yes

Incorrect

Figure 9 shows the platform used to verify candidates retained by automated pressure filtering. For paired-image cases, the Base and Edited images are displayed side by side together with the intended source-to- target transformation and a natural-language description of the review location. This location cue helps reviewers find the relevant evidence in visually dense scenes. Reviewers mark the source entity in the Base image and the inserted entity or edited region in the Edited image using bounding boxes. The normalized box coordinates are stored as curation metadata, allowing the regions to be highlighted during later inspection and reused for localized repair. These annotations are not provided to the evaluated VLMs. Reviewers inspect whether the Base image is valid, whether the intended transformation is correctly realized, and whether unrelated scene content is preserved. The interface also presents the four pressure-screening probes, including their reference answers, filtering-model predictions, and correctness. A valid case is marked as Keep; otherwise, it can be marked as requiring reinspection, having a failed edit, having an invalid Base image, containing unintended scene changes, or being uncertain. Reviewers may also record notes explaining ambiguous cases or rejection decisions. The platform additionally provides localized image repair and real-image authoring tools. Reviewers can select a defective region in an Edited image, generate a repaired candidate, compare it with the current image, and either accept the repair or restore the original version. The real-image authoring interface supports importing photographs, specifying source and target entities, constructing edited candidates, defining question–answer pairs, and exporting the resulting annotations in the same structured format. D.2

Localized Image Repair

Fine-grained full-image editing is frequently unreliable, especially when the requested change affects only a small object in a visually complex scene. A generation model may ignore the requested change, leave visible remnants of the original entity, or modify unrelated objects and scene content. Repeatedly

Subset

Metric

Context Prior

All4 (P) ↑ Edit2 ↑ Q1 Base+ ↑ Q2 Base- ↑ Q3 Edit- ↑ Q4 Edit+ ↑

0 0 97 82 0 86

1 2 99 67 3 82

10 13 72 94 33 52

7 9 85 84 17 58

3 7 93 85 10 68

4 6 97 62 7 88

Texture

All4 (P) ↑ Edit2 ↑ Q1 Base+ ↑ Q2 Base- ↑ Q3 Edit- ↑ Q4 Edit+ ↑

52 52 99 100 81 58

28 28 100 99 30 82

40 41 98 100 54 65

52 54 98 99 69 67

46 47 99 100 63 65

28 29 99 100 38 65

Attribute

Count Acc. (P) ↑

26

20

17

17

14

16

Language Elicitation MCQ Acc. (P) ↑

11

23

58

17

29

23

22.3

18.0

31.3

23.3

23.0

17.8

Overall

Macro Avg. ↑

Gemini 3.5 Flash GPT-5.4 Claude 4.6 Kimi-k2.6 Qwen 3.5 27B Grok-4.3

Table 7: Detailed results on SABRE-Prior. Rows marked with (P) are the primary subset metrics used to compute the macro-average. For Context Prior and Texture, All4 requires all four Base–Edited probes to be correct, while Edit2 requires both Edited-image probes to be correct. Q1 and Q2 respectively test the expected and counterfactual evidence in the Base image; Q3 and Q4 test the same evidence after editing, with expected answers yes/no/no/yes. Attribute reports normalized exact-match counting accuracy, and Language Elicitation reports four-option multiplechoice accuracy. All values are percentages. Bold values indicate the highest score in each row.

regenerating the complete image can therefore destroy otherwise valid visual evidence and weaken the controlled relationship between the Base and Edited images. To address this problem, our curation platform provides localized patch repair, as illustrated in Figure 10. When a reviewer identifies a defective region, they draw a bounding box around the failed edit or local artifact and select a repair operation. The system expands the selected region slightly to retain its surrounding visual context, crops the resulting patch, and sends only this marked patch to the image generation model together with a task-specific repair instruction. The repaired patch is resized when necessary and blended back into the original image using a soft mask. This procedure preserves the image outside the local neighborhood and reduces unintended changes to unrelated scene content. The platform supports four repair operations: removing an unwanted entity, swapping one entity for another, cleaning local visual artifacts, and restoring the background. Figure 10 presents the two entity-level operations. For Swap Entity, the reviewer specifies both the source and target entities; for Remove Entity, the selected object is removed and the exposed region is completed using its local context. Reviewers may further refine the repair instruction to specify expected appearance, scale, lighting, or content preservation constraints. A generated repair is treated as a candidate rather than being accepted automatically. The platform displays the original Edited image and repair preview side by side, marks the case as requiring reinspection, and allows the reviewer to accept the repair or return to the previous image. When a repair is accepted, the previous Edited image is retained as a backup and the repair operation is recorded. The repaired case must then pass human verification before it is included in the benchmark. D.3

Blind User Study of Localized Repair Quality

We conduct a randomized, method-anonymous user study to evaluate the perceptual quality of images produced by the localized repair tool. The survey interface are shown in Figures 11. The study contains 20 image-repair cases. Each question first presents an original image with a red bounding box indicating the entity to be removed, followed by four candidate repair results. Participants select the result that most naturally and seamlessly realizes the requested edit. The instructions emphasize that the marked entity should be completely removed while unrelated image content should remain unchanged. We compare four methods: Gemini prompt-only full-image editing, OpenCV inpainting, Gemini patch

Repair Method Gemini full-image prompt OpenCV inpainting Gemini patch + hard paste Ours: patch + soft blend

Votes Preference (%) 16 0 10 374

4.0 0.0 2.5 93.5

Table 8: Method preferences from 20 participants over 20 image-repair cases, yielding 400 selections in total.

generation with hard pasting, and our Gemini patch generation with soft-mask blending. The hard-paste baseline and our method use the same generated patch, isolating the effect of the compositing strategy. Candidate results are displayed only as Method 1–Method 4, without revealing their underlying methods. Their display positions are independently shuffled for every question, and the system records both the randomized order and the selected method. Each question requires one selection and cannot be skipped. Forty survey sessions were initiated. We analyze the 20 sessions that completed all 20 required comparisons, yielding 400 valid selections. We use entity removal throughout the study to maintain a consistent and unambiguous evaluation objective. The study therefore directly evaluates the removal setting and the shared patch-generation and compositing components of the localized repair mechanism. Entity swapping is demonstrated qualitatively in Figure 10 but is not included in this user study. As shown in Table 8, our method is selected in 374 of the 400 comparisons, corresponding to an overall preference rate of 93.5%. Because every participant completes the same 20 questions, this is also the mean participant-level preference rate. The standard deviation across participants is 8.1 percentage points, with individual preference rates ranging from 70% to 100%. Participants therefore show a clear preference for localized patch generation with soft-mask blending over full-image editing, OpenCV inpainting, and hard patch compositing on the tested entity-removal cases. D.4

Illustrative Pipeline Cost

Exact historical costs cannot be recovered because early task exploration and benchmark construction shared the same API accounts. We therefore report an illustrative marginal-cost estimate for the stabilized Context recipe using the Gemini-based image pipeline. For every 100 initial candidates, we estimate that approximately 50 pass pressure filtering and enter human review, of which 30 are accepted directly, 10 are accepted after localized repair, and 10 are rejected. This yields approximately 40 retained cases.

E

Additional Results

Table 7 localizes the failures within the paired-image subsets. Base-image accuracy is generally high, whereas performance drops after editing. Indicating that models may recognize the inserted target while still reporting the replaced source. These results motivate the strict All4 metric over individual-probe accuracy.

Stage

Model

100 Initial Candidates

Estimated Cost

Cost per 100 Retained

Sample specification Base-image generation Targeted image editing Pressure filtering Localized repair

GPT-5.4

100 structured outputs

$2.0–$5.0

$5.0–$12.5

Gemini 3.1 Flash Image Gemini 3.1 Flash Image Gemini 3.5 Flash Gemini 3.1 Flash Image

100 image generations

$6.7

$16.8

100 image edits

$6.7

$16.8

400 image–question evaluations approximately 10 repair calls

$1.1–$2.2 $0.7–$1.4

$2.8–$5.5 $1.8–$3.5

–

$17.2–$22.0

$43.0–$55.0

Human verifica- – tion Repair and rein- – spection

50 reviewed cases

25–50 min

1.0–2.1 h

approximately 10 cases

10–20 min

0.4–0.8 h

Total active hu- – man time

–

35–70 min

1.5–2.9 h

Total API cost

–

Table 9: Illustrative marginal cost of the stabilized Context recipe. Image costs assume 1K outputs from Gemini 3.1 Flash Image. The estimate includes sample-specification generation, Base-image generation, targeted editing, pressure filtering, localized repair, and active human review. It excludes one-time task exploration, prompt development, framework engineering, and final benchmark evaluation. Costs and retention rates vary with the task specification and required visual intervention.

Context Repair Review Current 1/1 · Reviewed 1/1

gecko_wrench_workbench__v01 · standalone · gecko_wrench_workbench__v01 generated_without_screening

Saved: needs_recheck

Exactly one small silver adjustable wrench. → Exactly one small realistic gecko. Review location

Front-right area of the midground workbench cluster, diagonally positioned beside and partly overlapped by a grease-stained rag. Local cluster

A crowded mixed-tool cluster containing pliers, hex keys, tire lever, socket, chain, spokes, washers, bolts, rag, and unbranded bicycle components; exactly one adjustable wrench is present in Base and no additional adjustable wrench appears anywhere. Edited visual cues

—

Human verifiability

Verify that exactly one recognizable small silver adjustable wrench is visible in the front-right midground tool cluster and that no other adjustable wrench appears anywhere in the scene.; Verify that exactly one recognizable small gecko occupies the wrench location, with toe-padded feet, head, mottled body, and tapering tail visible, and that no adjustable wrench remains anywhere.; Verify that the rag, surrounding tools, people, bicycles, camera angle, lighting, focus, composition, and all non-designated scene content are unchanged. Upload image for free repair Choose File No file chosen

Use current benchmark image

Current edited image

Repair preview

Repair input: current edited image.

Preview: repairs/gecko_wrench_workbench__v01__20260805T181554__gemini_patch__repaired.jpg

Draw the repair box here

Appears after running patch repair

Repair edited image

Repair status: needs_recheck Current repaired image: repairs/gecko_wrench_workbench__v01__20260805T181554__gemini_patch__repaired.jpg

Draw a box on the Edited image around the residual entity, replacement target, or local artifact. Repair only that marked region, then re-review the repaired image before keeping it. Repair type Swap entity Engine Current 1/1 · Reviewed 1/1 Gemini local patch repair Gemini model gecko_wrench_workbench__v01 · standalone · gecko_wrench_workbench__v01 generated_without_screening Saved: needs_recheck gemini-3.1-flash-image-preview Source entity realistic gecko Review location Target entityarea of the midground workbench cluster, diagonally positioned beside and partly overlapped by a grease-stained rag. Front-right small Local cluster ARepair crowded mixed-tool cluster containing pliers, hex keys, tire lever, socket, chain, spokes, washers, bolts, rag, and unbranded bicycle components; exactly one adjustable wrench is present in Base and no additional adjustable wrench appears anywhere. instruction Edited visual cues Replace the realistic gecko inside the marked region with small. The new small must be clearly visible and naturally integrated with the scene. Match the surrounding lighting, perspective, scale, shadows, and occlusion. Remove all visible traces of the realistic gecko. Keep everything outside the marked region unchanged. Swap the realistic gecko into a small ball dont make others changed

Context Repair Review

Exactly one small silver adjustable wrench. → Exactly one small realistic gecko.

—

Human verifiability

Verify that exactly one recognizable small silver adjustable wrench is visible in the front-right midground tool cluster and that no other adjustable wrench appears anywhere in the scene.; Verify that exactly one recognizable small gecko occupies the wrench location, with toe-padded feet, head, mottled body, and tapering tail visible, and that no adjustable wrench remains anywhere.; Verify that the rag, surrounding tools, people, bicycles, camera angle, lighting, focus, composition, and all non-designated scene content are unchanged.

Questions and answers

Editableimage benchmark content. The model prediction and result are the historical filter-gate record. Upload for free repair Probe Question Use current benchmark image Choose File No file chosen probe_01

Current edited image Draw the repair box here probe_02

probe_03

Is there an adjustable wrench in the Base image? Is there a gecko in the Base image? Is there an adjustable wrench in the Edited image?

Repair input: current edited image.

Answer

Repair preview

Filter prediction

yes

Appears after running patch repair

Filter result Incorrect

no

Incorrect

no

Incorrect

Preview: repairs/gecko_wrench_workbench__v01__20260805T181506__gemini_patch__repaired.jpg

Repair edited image

Repair status: needs_recheck Current repaired image: repairs/gecko_wrench_workbench__v01__20260805T181506__gemini_patch__repaired.jpg

Draw a box on the Edited image around the residual entity, replacement target, or local artifact. Repair only that marked region, then re-review the repaired image before keeping it.

Figure 10: Localized image repair in the curation platform. Top: Swap Entity replaces the bagel inside the selected region with a ball. Bottom: Remove Entity removes the bagel and reconstructs the surrounding tabletop. The current Edited image is shown on the left and the repair preview on the right. Reviewers can accept the repaired image or revert to the original Edited image. Repair type Remove entity Engine Gemini local patch repair Gemini model gemini-3.1-flash-image-preview Entity to remove realistic gecko Repair instruction Remove the realistic gecko inside the marked region. Do not leave any visible part, silhouette, shadow, label, or texture of the realistic gecko. Remove the realistic gecko only, dont make others changed

Questions and answers

Editable benchmark content. The model prediction and result are the historical filter-gate record. Probe Question probe_01 Is there an adjustable wrench in the Base image?

Answer yes

Filter result Incorrect

probe_02

Is there a gecko in the Base image?

no

Incorrect

probe_03

Is there an adjustable wrench in the Edited image?

no

Incorrect

Filter prediction

Image Editing Quality Evaluation We are comparing four methods for removing objects from images. Please help us evaluate which result looks best.

Best viewed on a desktop or laptop computer

HOW EACH QUESTION WORKS 1

Original image has a red box marking the entity to remove

Original

M1

M2

M3

M4

→

2

4 anonymous methods each attempt the removal

3

Click the result that looks most natural and seamless

✦ Only the entity inside the red box should be removed — nothing else in the

image should change ✦ Select the result that looks most natural and seamless ✦ 20 questions total — take as long as you need ✦ If a result fails to remove the entity, or accidentally removes/changes

something else — that is the method's failure, not your mistake. Please judge honestly based on what you see.

No right or wrong answers — your honest impression is what matters.

Start Evaluation

Figure 11: Instructions for the method-anonymous repair-quality study. Participants are asked to select the most natural result that removes only the marked entity while preserving all unrelated image content.

Related documents

• OmniGameArena: A Unified UE5 Benchmark for VLM Game Agents with Improvement Dynamics#267678
• Orchra: Stateful-aware Cross-slice Workload Migrations in the 6G Control Plane#662085
• Orchra: Stateful-aware Cross-slice Workload Migrations in the 6G Control Plane#661035
• Orchra: Stateful-aware Cross-slice Workload Migrations in the 6G Control Plane#480018
• ExploreAI: Agentic Exploration Knowledge Bases for Reproducible Observable-Regression Testing of Black-Box VR and 3D Applications#482226