Are Prompt Optimizers Blind? Cross-Modal Visual Feedback for Automatic Prompt Optimization Haoyue Liu1 , Xiaoyu Ma1 , Ye Chen2 , Yuexian Zou3,4 , Xiaoying Tang1∗ 1
The Chinese University of Hong Kong, Shenzhen, China 2 Xi’an Jiaotong University, China 3 Peking University, China 4 Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology, China
arXiv:2607.24354v1 [cs.AI] 27 Jul 2026
Abstract Automatic prompt optimization (APO) has been widely adopted to adapt vision-language models (VLMs) to downstream tasks without weight updates, yielding promising results. However, on multimodal tasks the effectiveness of APO is fundamentally bottlenecked by a blind feedback channel: the optimizer reads the question, the prediction, and the gold answer, but never the input image on which the model failed, and therefore cannot diagnose visually grounded errors. As a remedy, we introduce Cross-Modal Visual Feedback (CMVF). CMVF incorporates (1) a failure-conditioned visual diagnosis stage, in which a stronger optimizer VLM inspects each failed image without access to predictions or labels, and (2) an error-aware aggregation stage that compresses these observations into reusable, task-level visual blind-spot patterns that drive the prompt rewrite. Crucially, the image is consumed only during optimization; the deployed artifact is an ordinary text prompt that runs at the same inference cost as any text-only baseline. Extensive results across 12 VQA datasets and 4 target VLMs demonstrate that CMVF consistently ranks first, improving over the strongest baseline on every target by +2.4 points on average (up to +6.5 on individual benchmarks), while the optimizer self-organizes into expert-style visual checklists that transfer across models without re-optimization. Our code and optimized prompts will be released.
1
Introduction
Vision-language models (VLMs) are the default substrate for visual question answering, medical imaging, chart understanding, and document interpretation (Masry et al. 2022; Mathew, Karatzas, and Jawahar 2021), yet their accuracy is notoriously sensitive to the wording of the textual prompt— on our benchmarks, mean accuracy on Qwen3.5-4B varies from 70.3% to 75.0% across competitive prompts alone. Automatic prompt optimization (APO)—spanning evolutionary search (Fernando et al. 2024; Guo et al. 2024), textualgradient methods (Pryzant et al. 2023; Yuksekgonul et al. 2025), response-trajectory tracking (Zhang et al. 2025), and declarative program compilation (Khattab et al. 2024)—has become an effective, weight-free way to adapt these models, yielding promising results on text-centric tasks. However, when applied to multimodal tasks these optimizers all share the same structural gap: their feedback channel ∗
Corresponding author.
Figure 1: Motivation for CMVF. Existing text-only APO sees scores and error triples but misses the visual causes of failures. CMVF uses failed images only during optimization to aggregate visual blind spots into a reusable text prompt, with no extra visual-feedback calls at test time.
is blind. The optimizer sees the question, the prediction, and the gold answer, but never the image on which the target failed (Figure 1)—like a teacher grading an exam without ever seeing the figures. It can register that an error occurred but not why: a confused anatomical slice level, an overlooked axis label, or a global scene prior that overrides local evidence. We argue that the bottleneck of multimodal APO is not the optimization machinery but the evidence available to it. As a remedy, we introduce Cross-Modal Visual Feedback (CMVF), built on a simple principle: optimization feedback must have access to the modality in which failures originate. CMVF lets a stronger optimizer VLM inspect the failed images during optimization—first describing question-relevant visual evidence without seeing the prediction or ground-truth answer, then aggregating these descriptions with the error triples into reusable visual blind spots that drive the prompt rewrite. Unlike multimodal prompt optimization (MPO) (Choi et al. 2026), which jointly optimizes a (text, image) prompt pair and keeps the image in the loop at deployment, CMVF pays for the visual signal only once, at optimization time, and compiles it into a purely textual prompt; deployment therefore adds no VLM calls and reuses the same serving interface as any text-only prompt. Our contributions are three-fold: 1. Problem. We identify and characterize the visualfeedback blindness shared by all text-only APO methods on multimodal tasks, and show the bottleneck is evidence,
not optimization machinery. 2. Method. We propose CMVF, which incorporates (1) failure-conditioned, label-free visual diagnosis and (2) error-aware aggregation into reusable visual blindspot patterns, realizing a train-time-multimodal, deploytime-text-only design at zero added inference cost. 3. Results & analysis. Across 12 datasets and 4 target VLMs, CMVF ranks first everywhere, beating the strongest baseline by +2.4 pp on average (up to +6.5 pp); token-budget-matched ablations attribute the gain to visual perception, the optimizer self-organizes into expertstyle visual checklists, and these strategies transfer across models without re-optimization.
2
Cross-Modal Visual Feedback
The preceding discussion raises two questions that define the design of CMVF. First, how can a prompt optimizer obtain visual diagnostic information without leaking the groundtruth answer into the visual analysis? Second, how can this visual information improve the deployed prompt without adding any inference-time VLM calls? CMVF answers these two questions with a two-stage visual-gradient channel (Section 2.3) and a fusion-update loop that preserves a standard textual-prompt inference interface (Section 2.4).
2.1
Problem Setup
Given a target VLM MT and training data D = {(xi , qi , yi )}N i=1 (xi image, qi question, yi ground-truth answer), we seek a text prompt p∗ ∈ P that maximizes task accuracy: N
1 X 1[MT (p, xi , qi ) = yi ]. p∈P N i=1
p∗ = arg max
(1)
Following TextGrad (Yuksekgonul et al. 2025), we optimize two prompts jointly: pyn for yes/no questions and poe for open-ended questions, both with a {fs} placeholder for few-shot examples. At step t ∈ {1, . . . , T }, let ŷi := MT (pt , xi , qi ) be the prediction under the current (t) (t) prompt pt = (pyn , poe ); we form the correct set Ct and the wrong set Wt = {(xi , qi , ŷi , yi ) : ŷi ̸= yi }. A validation set V selects the best prompt across steps, and if a rewrite drops the {fs} placeholder we restore the validation-best prompt.
2.2
The Visual-Feedback Bottleneck
Standard APO constructs feedback from accuracy and tex tual error triples, gttext = acc(pt ), Etsub with Etsub ⊆ {(qi , ŷi , yi )}. This exposes which questions failed but omits the visual evidence that caused them: a medical VQA model may confuse adjacent anatomical slice levels, a chart model may miss a small axis label, a real-world model may rely on a global scene prior. The textual triple sees the symptom; the cause stays hidden. CMVF closes this loop with the two-stage visual channel below.
2.3
Visual Diagnostic Channel
CMVF adds a visual-gradient channel to the feedback loop. During optimization only, a stronger optimizer VLM MO
inspects failed images and summarizes recurring visual blind spots. Figure 2 gives the overview. Stage 1: question-aware visual description. For every wrong example, MO receives only the image and question and produces a question-aware visual description: vi = MO (xi , qi ; πvis ),
∀(xi , qi , ŷi , yi ) ∈ Wt ,
(2)
where πvis is a fixed Stage-1 meta-prompt asking for questionrelevant visual evidence (supplementary material). Crucially, the prediction ŷi and ground truth yi are withheld at this stage. This prevents answer-conditioned rationalization—the optimizer cannot manufacture a post-hoc explanation for a label it has not yet seen—and forces it to record perceptionlevel evidence rather than rationalize a known label. Stage 2: error-aware aggregation. Individual descriptions are noisy and idiosyncratic. We therefore pair each vi with its error triple and ask MO to summarize recurring blind spots: gtvis = Agg {(qi , ŷi , yi , vi ) : (xi , qi , ŷi , yi ) ∈ Wt } . (3) The aggregator receives at most 30 wrong cases per step with field-level truncation. The two-stage decomposition separates visual perception from supervised error attribution: Stage 1 records what is visible; Stage 2 identifies which visual patterns repeatedly correlate with mistakes, compressing them into a compact set of reusable diagnostic instructions (e.g., “check small text,” “verify anatomical slice level,” “avoid global scene priors”). This compression is what makes broad inspection at Stage 1 affordable: many descriptions in, one prompt-sized signal out.
2.4
Fusion, Prompt Update, and Inference
Fused feedback. We fuse accuracy statistics and visual blind spots into a unified feedback string: gtCM = Concat acc(pt ), |Ct |, |Wt |, gtvis , s , (4) where s instructs the optimizer to rewrite the prompt so the target model explicitly checks the identified visual features before answering. Because gtvis is constructed from both visual descriptions and textual error triples, CMVF augments rather than replaces the existing textual error signal. Update and inference. The prompt is updated by pt+1 = Update(pt , gtCM ). The default Update is a TextGrad-style rewrite (Yuksekgonul et al. 2025); CMVF∗ swaps in a REVOLVE-style alternative (Zhang et al. 2025) on the same visual-gradient pipeline, isolating whether gains come from the visual channel rather than a particular rewriter. At deployment, the target VLM receives only the image, question, and optimized text prompt (pyn , poe )—no additional visual analysis at inference (Algorithm 1).
2.5
Modeling Intuition
The role of visual feedback can be understood through a simple error-cause decomposition. Let αV ∈ [0, 1] denote the fraction of currently wrong examples whose failure has a visual component, and let IV ∈ [0, 1] denote how
Figure 2: CMVF optimization pipeline. The target VLM is evaluated on training examples; the optimizer VLM inspects failed images, aggregates visual blind spots, and rewrites the text prompt. Inference uses the optimized text prompt with no extra visual-feedback calls. Algorithm 1 CMVF Prompt Optimization Require: Target VLM MT , optimizer VLM MO , training set D, validation set V, steps T 1: Initialize prompts (pyn , poe ) and validation-best prompts 2: for t = 1 to T do 3: Evaluate MT on D to obtain Ct and Wt 4: Evaluate on V and update the validation-best prompts if improved 5: for each (xi , qi , ŷi , yi ) ∈ Wt do 6: vi ← MO (xi , qi ; πvis ) {image + question only} 7: end for 8: gtvis ← Agg({(qi , ŷi , yi , vi ) : (xi , qi , ŷi , yi ) ∈ Wt }) 9: gtCM ← Concat(acc(pt ), |Ct |, |Wt |, gtvis , s) 10: (pyn , poe ) ← Update((pyn , poe ), gtCM ) 11: Restore validation-best prompts if {fs} is lost 12: end for 13: return validation-best prompts
informative the optimizer’s visual analysis is about these causes. Feedback without visual inspection carries only indirect visual hints; denote its residual visual signal by δT . With error rate L(p) := 1−acc(p) and per-step reduction ∆L := L(pt )−L(pt+1 ), under an additive view of textual and visual error sources the expected advantage of crossmodal feedback scales as E ∆LCM − ∆Ltext ≳ αV IV − δT . (5) This intuition (not a convergence guarantee) predicts CMVF should help most when αV is large (medical VQA, TextVQA OCR, RealWorldQA) and least when αV → 0; both match our empirical pattern in Section 3.2. This also clarifies the role of Stage-2 aggregation. A single visual description may be noisy or instance-specific, so its information about the underlying visual cause is limited. By aggregating many failed
cases, CMVF suppresses instance-specific noise, surfaces recurring visual patterns, and converts per-image observations into a compact prompt-level signal. In this sense, aggregation increases the effective IV , which is consistent with the ablation results in Section 3.5.
3
Experiments
In this section, we run experiments to address five questions: • Q1: Does CMVF beat APO baselines that do not incorporate visual feedback, across multiple target VLMs? (Section 3.2) • Q2: Does the optimizer learn anything interpretable about the failure modes? (Section 3.3) • Q3: Do the optimized strategies transfer across target models without re-optimization? (Section 3.4) • Q4: Where do the gains come from—image-aware perception, or extra feedback tokens? (Section 3.5) • Q5: Does CMVF generalize and pay an acceptable training cost with unchanged inference? (Section 3.6) Stage-2 variants (CM-Mem, Diff), dataset details, prompt templates, and case studies are in the Appendix.
3.1
Setup
Models and benchmarks. We use Qwen3.5-9B as the optimizer VLM MO and evaluate four target VLMs: Qwen3.5-4B, Qwen2.5-VL-7B (Wang et al. 2024), LLaVA1.6-Mistral-7B (Li et al. 2023), and Phi-3.5-vision (Abdin et al. 2024). The suite covers 12 datasets: medical VQA datasets SLAKE CT/MRI/X-Ray, VQA-RAD, and PathVQA (Liu et al. 2021; Lau et al. 2018; He et al. 2020); reasoning and chart/OCR datasets ScienceQA, ChartQA, TextVQA, and MMMU (Lu et al. 2022; Masry et al. 2022; Singh et al. 2019; Yue et al. 2024); and RealWorldQA, AI2D, and DocVQA (xAI 2024; Kembhavi et al. 2016; Mathew, Karatzas, and Jawahar 2021). We focus the main compari-
son on eight benchmarks and report the remaining four (ScienceQA, AI2D, DocVQA, MMMU) separately in Table 2. Baselines and protocol. We compare against ProTeGi (Pryzant et al. 2023), OPRO (Yang et al. 2024), DSPy (Khattab et al. 2024), TextGrad (TG) (Yuksekgonul et al. 2025), and REVOLVE (RE) (Zhang et al. 2025). Each benchmark uses 150 training samples, 150 validation samples, and the full test set. All methods share data splits, initialization prompts, the same optimizer VLM, and 20 optimization steps. Results are averaged over 3 runs. Evaluation follows each benchmark’s official scoring protocol (exact match after lowercasing and punctuation removal, plus a small set of domain synonyms, e.g., “x-ray”/“xray”).
3.2
A1: CMVF Ranks First on All Four Target VLMs
Headline result. Table 1 restores the full comparison across all five baselines without visual feedback, CMVF, and CMVF∗ for all four target VLMs. CMVF ranks first on mean accuracy for every target, with the largest absolute gain on LLaVA-1.6-Mistral-7B (+3.1 pp) and a comparable +3.0 pp on Qwen3.5-4B. Where the wins concentrate. The per-dataset pattern is consistent with the visual-feedback hypothesis: gains cluster on visually demanding benchmarks. On Qwen3.5-4B, CMVF improves RealWorldQA by +6.5 pp, TextVQA by +2.4 pp, and PathVQA by +1.5 pp over the strongest baseline without visual feedback. On LLaVA, ChartQA jumps by +6.0 pp and PathVQA by +3.3 pp. On the four additional benchmarks, CMVF again ranks first, by +1.12 to +3.88 pp over the strongest baseline (Table 2). This matches the modeling intuition in Equation (5): the gain scales with αV , the fraction of visually-grounded errors, which is largest precisely on perception-heavy tasks. The gains are statistically significant. Exact McNemar tests against the strongest baseline, on paired per-example predictions, confirm the improvements: p = 7.9 × 10−14 on LLaVA/TextVQA, p = 0.021 on LLaVA/ChartQA, and p = 0.015 on Qwen2.5-VL/TextVQA—significant precisely on OCR, chart-grounding, and fine-grained visual tasks, where visual evidence is essential. It is not the rewriter. CMVF∗ uses a REVOLVE-style rewriter on top of the same visual-gradient pipeline. Relative to its counterpart without visual feedback, adding the visualgradient channel improves a REVOLVE-style update on most targets (e.g., 70.3→71.9 on Qwen3.5-4B, 54.7→58.5 on LLaVA, and 53.7→55.8 on Phi). This shows that the visual channel is not tied to a TextGrad-style rewriter, while the remaining gap to CMVF also makes clear that the rewriter choice still matters.
3.3
A2: The Optimizer Self-Discovers Expert-Style Visual Inspection Checklists
Beyond the headline accuracy gains, we observe a striking qualitative effect: the optimizer VLM is not given datasetspecific visual failure rules, yet it converges on instruc-
tions that look like a domain expert’s checklist. This emergent self-diagnostic behavior is, in our view, the most interesting observation of this paper. Prompt contrast and source audit. Table 3 lists representative deployed prompts on Qwen3.5-4B. Baselines without visual feedback converge on generic, vision-agnostic hedges (“read step by step,” “be precise”), while CMVF produces task-specific visual checklists. Crucially, this is not an artifact of hand-written rules: the Stage-1 prompt is deliberately broad (“key diagnostic finding” for medical, “key visual details” for general) and never mentions chart axes, small-text OCR, slice-level checks, or local-object priors—yet the final prompts recover exactly these task-level routines from failed-image evidence. What CMVF rewrites the prompt to say. Feedback without visual inspection typically yields generic advice (“verify your reasoning against the visual evidence,” “be careful with details”). CMVF instead discovers task-specific routines across the 8 benchmarks: (1) SLAKE-CT/MRI prompts ask the model to verify anatomical slice level and separate similar-density regions; (2) VQA-RAD/PathVQA prompts locate the focal abnormality before classification; (3) ChartQA prompts read axes, units, and legends before comparing values; (4) TextVQA prompts scan for small or non-salient text; and (5) RealWorldQA prompts check occlusion and avoid global scene priors for local-object questions. These instructions are not pre-specified in the seed prompt or Stage-1 description prompt; they arise from Stage-2 aggregation over Stage-1 perception, showing that a frozen optimizer VLM can self-organize reusable visual checklists from failed-image evidence alone. Why this matters. The phenomenon suggests that the bottleneck of multimodal APO is not optimization machinery but evidence: once the optimizer can see failed images, recurring visual failure modes become extractable in language and can be carried by a static inference-time text prompt.
3.4
A3: Optimized Strategies Transfer Across Target Models
If CMVF merely memorized target-specific quirks, its prompts would not survive being moved to a different model. We test this directly: the CMVF prompt optimized on each of Qwen2.5-VL-7B, LLaVA-1.6-7B, and Phi-3.5-vision is deployed unchanged on Qwen3.5-4B, with no re-optimization (Table 4). Every transferred prompt outperforms a TextGrad prompt optimized directly on Qwen3.5-4B (70.3), by +1.2 to +2.8 pp on the 8-benchmark mean, and the advantage holds per-dataset rather than on a few benchmarks—on ChartQA a transferred prompt even exceeds the in-domain one. This indicates that Stage-2 aggregation surfaces reusable, task-level visual strategies rather than spurious correlations bound to a single model, directly addressing the concern that access to ground-truth labels could induce dataset- or target-specific overfitting. The in-domain prompt optimized on 4B remains best overall (75.0), as expected.
Model
Medical VQA ↑
Method
General VQA ↑
Mean ↑
CT
MRI
X-Ray
RAD
Path
Chart
Text
RWQA
Qwen3.5-4B
ProTeGi OPRO DSPy TG RE CMVF∗ CMVF
75.3±2.8 73.2±3.4 77.6±1.1 75.9±0.3 75.5±0.6 76.8±0.4 77.9±1.3
78.1±0.9 67.7±6.1 77.7±1.4 75.3±1.3 79.0±0.1 77.9±2.2 80.7±3.1
81.7±1.4 81.8±1.0 80.7±1.3 80.7±1.6 83.8±0.5 84.1±0.2 84.5±1.2
59.4±3.1 55.2±4.7 62.1±0.8 57.9±3.1 59.2±2.0 61.3±1.4 63.4±1.0
45.4±0.5 44.6±1.9 47.5±2.8 45.4±2.0 44.3±1.1 46.9±0.8 49.0±2.8
78.0±3.4 25.1±28.4 81.6±1.9 78.6±2.7 75.1±2.7 78.8±2.0 81.4±2.0
65.2±2.3 69.1±0.6 66.3±2.2 63.8±1.4 64.3±1.6 65.7±0.8 71.5±2.7
80.4±1.9 81.7±3.3 82.6±0.9 84.8±0.7 80.9±2.2 83.3±1.8 91.3±3.8
70.4 62.3 72.0 70.3 70.3 71.9 75.0 ↑3.0
Qwen2.5-VL-7B
ProTeGi OPRO DSPy TG RE CMVF∗ CMVF
64.0±1.3 60.1±6.4 60.8±1.1 64.5±2.0 63.5±0.2 63.8±0.9 65.0±1.1
59.4±4.9 61.4±8.3 61.0±2.8 69.2±1.3 69.4±0.6 68.7±3.2 69.4±2.3
73.5±1.1 69.4±7.5 72.1±1.3 73.5±1.6 72.1±0.2 73.2±2.2 75.0±3.0
47.5±2.3 42.6±0.1 47.0±0.6 48.8±1.6 48.3±1.0 49.9±1.4 51.0±1.8
37.9±0.9 29.5±4.4 32.7±1.3 36.8±1.9 38.6±0.9 35.4±2.7 40.2±2.2
56.3±0.6 54.2±3.8 59.4±2.1 54.4±1.9 57.5±0.3 55.7±1.2 56.1±2.9
59.8±0.6 60.0±0.2 55.5±2.6 59.4±1.8 57.1±3.7 59.3±1.5 62.3±1.7
58.3±11.0 50.4±8.0 60.0±8.6 61.9±2.7 58.7±2.1 61.9±0.7 62.2±3.2
57.1 53.4 56.1 58.6 58.2 58.5 60.1 ↑1.6
LLaVA-1.6-7B
ProTeGi OPRO DSPy TG RE CMVF∗ CMVF
58.4±0.9 58.3±1.6 58.5±0.6 58.9±2.4 56.4±1.2 60.3±0.3 59.6±0.6
72.7±0.1 65.6±0.3 66.9±1.1 70.1±2.6 69.6±2.8 70.9±1.0 71.1±0.4
76.8±0.4 71.9±1.2 74.5±0.9 75.7±0.2 76.1±0.5 75.8±0.9 77.7±0.6
45.6±0.6 41.1±0.8 44.3±1.7 43.8±1.1 45.1±0.3 48.0±2.5 48.2±0.1
38.6±0.2 36.3±0.2 37.8±0.7 34.9±2.0 37.6±1.4 40.6±0.2 41.9±3.3
25.5±0.9 32.3±3.7 31.4±5.2 30.9±2.1 29.1±1.1 37.7±3.7 38.3±4.4
45.3±1.8 49.5±2.4 49.4±3.0 45.9±3.4 43.6±1.8 53.5±1.5 51.3±1.9
81.6±1.2 80.1±0.5 80.3±1.3 76.1±6.8 80.4±0.6 81.0±0.2 81.3±0.4
55.6 54.4 55.4 54.5 54.7 58.5 58.7 ↑3.1
Phi-3.5-vision
ProTeGi OPRO DSPy TG RE CMVF∗ CMVF
58.2±2.1 59.2±1.0 57.4±1.7 57.7±2.1 56.4±0.8 60.1±1.1 58.9±1.9
72.0±4.9 63.8±2.5 64.0±1.4 68.5±2.6 68.6±2.1 71.9±0.9 72.4±1.5
76.0±0.6 69.5±2.2 74.1±2.1 75.0±0.7 75.0±1.2 77.3±1.6 77.1±2.3
45.5±2.4 40.4±2.9 42.6±1.3 43.0±2.3 43.3±0.6 46.4±3.0 46.5±0.5
32.5±1.6 31.9±0.8 34.4±1.0 34.1±1.1 37.1±0.8 38.5±1.0 38.0±1.5
50.9±1.8 45.0±6.4 51.1±1.8 51.2±1.6 48.6±1.1 54.4±2.6 53.4±1.1
47.4±1.7 49.3±2.0 51.2±1.5 49.5±2.1 51.8±0.7 52.3±1.4 51.4±1.9
45.8±2.0 48.1±3.9 55.7±6.8 45.7±5.0 48.6±3.7 45.2±1.2 49.8±1.4
53.5 50.9 53.8 53.1 53.7 55.8 55.9 ↑2.1
Table 1: Accuracy on 8 benchmarks. Rows group target VLMs; per model the CMVF (our method) row is shaded green and the runner-up by mean gray. Bold/underline mark best/second best; gray subscripts show std over 3 runs. Method
AI2D
DocVQA
MMMU
SciQA
Mean
ProTeGi OPRO DSPy TG RE CMVF
78.46 76.83 74.21 72.68 80.91 84.62
68.31 64.28 66.74 62.53 70.86 74.74
47.62 45.89 43.47 41.16 47.62 48.74
81.42 85.30 78.95 76.63 83.07 87.60
68.95 68.08 65.84 63.25 70.62 73.93
Table 2: Accuracy (%) on the four additional benchmarks (Qwen3.5-4B). The CMVF row is highlighted; the runnerup by mean (RE) is shaded gray. Bold: best; underline: best baseline (ProTeGi and RE tie on MMMU). CMVF ranks first on all four and on their mean.
3.5
A4: Image-Aware Perception Drives the Gains, Not Extra Tokens
Tokens are cheap; pixels are decisive. A2 attributed the gains to image-aware evidence; Table 5 now rules out longer feedback as the cause under a token-budget-matched protocol on Qwen3.5-4B. CM-Blind adds the same number of feedback tokens as CMVF but without image access; it reaches only 71.9 (+1.6 pp over TG). CM-Caption feeds the optimizer generic, question-agnostic captions and reaches 72.2. CM-Vis keeps the visual channel but drops the fused error attribution—concatenating raw per-image descriptions
into the rewrite prompt, closely matching the IPO-style (Du, Sun, and Snoek 2024) recipe adapted from CLIP classtemplate prompts to VQA instruction prompts—reaching 72.8, while the full CMVF adds Stage-2 aggregation over all wrong samples and reaches 75.0. The implication is direct: longer prompts alone account for at most +1.6 pp over TG, while question-aware visual perception with aggregation accounts for the remaining +3.1 pp. CMVF’s deployed prompt averages ∼205 BPE tokens (excluding {fs}), comparable to TG/RE (185–192) and 14–26% shorter than DSPy/OPRO/ProTeGi (239–276); the gain cannot be attributed to a longer inference-time prompt. The ordering generalizes across targets. The same monotone channel progression—CM-Blind < CM-Caption < CM-Vis < CMVF—holds on every (target, dataset) pair for two additional target VLMs (Table 5), so the effect is not specific to Qwen3.5-4B. The margin widens on the cross-family target LLaVA-1.6-7B (CMVF +7.8 over CM-Vis on RealWorldQA and +7.5 on TextVQA), consistent with a weaker base model leaving more visually-groundable headroom for the visual channel to recover. Two alternative Stage-2 aggregation modes further isolate why unified aggregation is preferred: CM-Mem (72.1, −2.9 vs. CMVF), which carries a cross-step memory pool and is misled by stale signals from earlier weaker prompts, and Diff (72.2, −2.8), which fragments the visual signal by partitioning errors before aggregation. Both underperform
Method
Deployed prompt (visual-specific clauses highlighted)
Acc
■ TextVQA (scene text, OCR) OPRO
Imagine your mind is a vast, still pond where ripples of distraction fade away until only the single drop of truth answering the question can be seen; gently surface to inspect this one isolated ripple. . .
69.1
TG
Analyze the image to extract only visual information, explicitly ignoring any text, labels, or metadata that cannot be verified by visual inspection alone.
63.8
CMVF
Meticulously scan the entire image to verify specific counts, read small text, and confirm fine details rather than relying on generic associations 71.5 ↑2.4 or prominent but unrelated features.
■ ChartQA (chart reasoning, axes/legends) TG
Examine the chart and answer the question. Be precise and double-check your interpretation before giving the final answer.
78.6
RE
Analyze the chart step by step, revise any inconsistent interpretation, and return the final value or category.
75.1
CMVF
Read axis labels, units, and the legend mapping before reasoning about magnitudes; identify the requested data point and verify its exact value 81.4 ↑2.8 before answering.
■ RealWorldQA (real-world VQA, scene reasoning) DSPy TG
You are a visual question answering assistant. Inspect the image and answer the question accurately. Look at the image carefully and answer the question. Make sure your reasoning is grounded in what is actually visible.
CMVF
Scan for small, low-contrast, occluded, or background objects; verify orientation, signs, and object states; avoid global scene priors when 91.3 ↑6.5 the question is about a local object.
82.6 84.8
Table 3: Representative deployed prompts on Qwen3.5-4B. For each benchmark we show two baselines (without visual feedback, gray italics) and CMVF (green). Bold marks the decisive clause: baselines stay generic or, on TextVQA, even instruct the model to ignore text, whereas CMVF names the exact visual check the task demands. Acc is test accuracy. Prompt source → Qwen3.5-4B
CT
MRI
X-Ray
RAD
Path
Chart
Text
RWQA
Mean
TextGrad (direct on 4B) CMVF ← Qwen2.5-VL-7B CMVF ← LLaVA-1.6-7B CMVF ← Phi-3.5-vision CMVF (direct on 4B)
75.9 76.0 76.4 76.0 77.9
75.3 76.8 78.7 79.0 80.7
80.7 82.5 83.4 83.0 84.5
57.9 57.4 59.4 58.3 63.4
45.4 44.8 48.2 47.6 49.0
78.6 83.2 80.5 84.3 81.4
63.8 66.1 65.5 69.7 71.5
84.8 84.8 84.3 87.0 91.3
70.3 71.5 72.1 73.1 75.0
Table 4: Cross-target transfer: a CMVF prompt optimized on one model and deployed on Qwen3.5-4B without re-optimization still beats TextGrad optimized directly on 4B. Every transferred prompt exceeds the direct TextGrad baseline; on ChartQA transfer even beats the in-domain prompt. Bold: best per column. Qwen3.5-4B Qwen2.5-VL-7B
LLaVA-1.6-7B
Method
Mean
RWQA
Text
RWQA
Text
CM-Blind CM-Caption CM-Vis CMVF
71.9 72.2 72.8 75.0
56.1 58.3 60.5 62.2
56.1 58.2 60.4 62.3
69.1 71.3 73.5 81.3
39.5 41.5 43.8 51.3
Table 5: Feedback-channel ablation across three target VLMs (methods defined in text; token-budget matched). Qwen3.5-4B is the 8-benchmark mean; the other two targets are per-dataset (official protocol). Every column rises monotonically—CMVF (green) best, CM-Vis (gray) runnerup.
unified CMVF (full design-space sweep in the appendix), confirming that the gain comes from question-aware perception of all wrong samples with a single unified aggregation step, not from any individual component.
3.6
A5: CMVF Generalizes Better at ~25% Training-Time Overhead and No Additional Inference Cost
Late-stage improvement. Figure 3 shows that CMVF continues improving after baselines without visual feedback flatten on these settings. Once feedback without visual inspection has corrected the linguistically obvious failures, the remaining errors are more likely to be visually grounded, so an image-grounded gradient can keep moving. Generalization gap. Figure 4(a) shows that CMVF has the smallest average validation-to-test gap among all six methods, and Figure 4(b) confirms this advantage holds perdataset rather than coming from a few easy benchmarks. We attribute this to aggregation: instead of overfitting prompts to isolated validation errors, CMVF rewrites them around recurring visual patterns—a form of implicit regularization. Cost in practice. The visual-gradient pipeline adds a oneshot training-time overhead but no additional inference-time cost. End-to-end optimization on Qwen3.5-4B takes 2.6 h for CMVF versus 2.1 h for TextGrad and 2.4 h for REVOLVE—a ∼ 20–25% wall-clock increase that buys +3.0 pp mean accu-
validation acc (%)
(a) Phi · MRI
70.0 67.5 65.0 62.5 60.0 57.5 55.0 52.5
(b) Phi · X-Ray
84 82
5.0
7.5
10.0
12.5
optimization step
15.0
17.5
80
85
78
80
76
TextGrad REVOLVE CMVF 2.5
70
72
20.0
2.5
5.0
(e) Phi · CT
7.5
10.0
12.5
optimization step
15.0
17.5
20.0
(f) Phi · PathVQA
68 66 64 62 2.5
5.0
7.5
10.0
12.5
optimization step
15.0
17.5
2.5
5.0
7.5
10.0
12.5
optimization step
15.0
17.5
20.0
2.5
5.0
7.5
10.0
12.5
optimization step
15.0
17.5
20.0
2.5
5.0
7.5
10.0
12.5
optimization step
15.0
17.5
20.0
17.5
20.0
(h) Phi · TextVQA 90
97
89
96
88
95
87
94
86
93
85 84
92
20.0
82.5 80.0 77.5 75.0 72.5 70.0 67.5 65.0
(g) Phi · ChartQA
98
52 50 48 46 44 42 40 38
70
(d) Phi · RealWorldQA
75
74
72
validation acc (%)
(c) Phi · VQA-RAD 90
2.5
5.0
7.5
10.0
12.5
optimization step
15.0
17.5
20.0
2.5
5.0
7.5
10.0
12.5
optimization step
15.0
Figure 3: Validation accuracy over 20 optimization steps on eight Phi-3.5-vision benchmarks (top: MRI, X-Ray, VQA-RAD, RealWorldQA; bottom: CT, PathVQA, ChartQA, TextVQA). CMVF (red) stays above both text-only baselines (TextGrad, REVOLVE) and keeps improving after they plateau; the shaded band marks CMVF’s margin over the stronger baseline. (a) Average Overfitting Gap
(b) Per-Dataset Overfitting
10.7
30
DSPy
11.9
25
TextGrad
13.4
REVOLVE
13.8
Per-dataset gap (%)
CMVF
16.0
ProTeGi
19.1
OPRO
20
TextGrad
REVOLVE
CMVF
high
5.5
15
9.7
10 medium
8.6
5 low
0 0
5
10
Avg validation-to-test gap (%)
15
20
lower is better
CT
MRI
y A A A A X-Ra VQA-RAD PathVQ ChartQ TextVQ lWorldQ Rea
Figure 4: Generalization gap (validation-best accuracy minus test accuracy; lower is better) on 8 benchmarks. CMVF has the smallest average gap and reduces high-gap cases such as ChartQA and VQA-RAD. racy. Compared to ProTeGi (5.9 h), CMVF is more than 2× faster while delivering substantially higher accuracy. Crucially, the final deployed artifact is a standard text prompt, so inference uses the same serving interface and call count as a vanilla deployment.
4
Related Work
Automatic prompt optimization. Early APO methods search over prompt candidates or use LLMs as metaoptimizers (Yang et al. 2024; Zhou et al. 2023), including reinforcement-learning, black-box, and evolutionary variants (Deng et al. 2022; Chen et al. 2024; Fernando et al. 2024; Guo et al. 2024). DSPy compiles declarative LLM programs into optimized prompt pipelines (Khattab et al. 2024), gradient-inspired methods such as ProTeGi and TextGrad use natural-language critiques as textual gradients (Pryzant et al. 2023; Yuksekgonul et al. 2025), and REVOLVE tracks response evolution across iterations (Zhang et al. 2025). As prompts are highly sensitive to surface form (Sclar et al. 2024), recent work also explores symbolic and structurefactorized prompt spaces (Ou et al. 2025; Liu et al. 2026). These methods keep the feedback channel textual even when the downstream task is multimodal.
Prompt learning for vision-language models. A parallel line adapts VLMs by learning continuous prompts: CoOp and CoCoOp tune soft context vectors for CLIP-style classifiers (Zhou et al. 2022b,a), VPT prepends learnable visual tokens (Jia et al. 2022), and MaPLe couples prompts across the vision and language branches (Khattak et al. 2023). These require gradient access to the backbone and deploy a nontextual prompt, whereas CMVF optimizes a discrete, humanreadable instruction and treats the target VLM as a black box at both optimization and deployment time. Multimodal prompting, visual feedback, and selfrefinement. The closest prior work is IPO (Du, Sun, and Snoek 2024), which lets an LMM describe training images and rewrites a CLIP class-template prompt for zero-shot classification. CMVF shares its recipe of answer-agnostic visual descriptions + LMM optimizer + textual-prompt deployment, but inspects only the wrong set Wt , adds a Stage-2 error-aware aggregation into reusable blind-spot patterns (removing it costs 2.2 pp; Table 5), and targets generative VQA prompts on four VLMs (4B–7B) rather than a fixed CLIP backbone. Concurrently, multimodal prompt optimization (MPO) (Choi et al. 2026) keeps the deployed prompt multimodal at a per-query visual cost, whereas CMVF deploys a standard textual prompt. Unlike self-refinement methods that critique outputs per instance at inference time (Madaan et al. 2023; Shinn et al. 2023), CMVF aggregates cross-example visual evidence once, at optimization time.
5
Conclusion
We introduce CMVF, a two-stage method that lets the optimizer VLM inspect the images the target model failed on and compiles that evidence into an ordinary text prompt. CMVF ranks first on all four target VLMs, beating the strongest baseline by +2.4 pp on average (up to +6.5 pp) at no added inference cost, and the optimizer self-organizes into transferable expert-style visual checklists. The bottleneck of multimodal APO is visual evidence, not optimization machinery.
References Abdin, M.; Aneja, J.; Awadalla, H.; Awadallah, A.; Awan, A. A.; Bach, N.; Bahree, A.; et al. 2024. Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone. arXiv preprint arXiv:2404.14219. Chen, L.; Chen, J.; Goldstein, T.; Huang, H.; and Zhou, T. 2024. InstructZero: Efficient Instruction Optimization for Black-Box Large Language Models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, 6503– 6518. PMLR. Choi, Y.; Kim, D.; Baek, J.; and Hwang, S. J. 2026. Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMs. In International Conference on Learning Representations. Deng, M.; Wang, J.; Hsieh, C.-P.; Wang, Y.; Guo, H.; Shu, T.; Song, M.; Xing, E. P.; and Hu, Z. 2022. RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), 3369–3391. Du, Y.; Sun, W.; and Snoek, C. G. 2024. IPO: Interpretable Prompt Optimization for Vision-Language Models. Advances in Neural Information Processing Systems, 37: 126725–126766. Fernando, C.; Banarse, D. S.; Michalewski, H.; Osindero, S.; and Rocktäschel, T. 2024. Promptbreeder: Self-Referential Self-Improvement via Prompt Evolution. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, 13481–13544. PMLR. Guo, Q.; Wang, R.; Guo, J.; Li, B.; Song, K.; Tan, X.; Liu, G.; Bian, J.; and Yang, Y. 2024. Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers. In International Conference on Learning Representations. He, X.; Zhang, Y.; Mou, L.; Xing, E.; and Xie, P. 2020. PathVQA: 30000+ Questions for Medical Visual Question Answering. arXiv preprint arXiv:2003.10286. Jia, M.; Tang, L.; Chen, B.-C.; Cardie, C.; Belongie, S.; Hariharan, B.; and Lim, S.-N. 2022. Visual prompt tuning. In European Conference on Computer Vision, 709–727. Springer. Kembhavi, A.; Salvato, M.; Kolve, E.; Seo, M.; Hajishirzi, H.; and Farhadi, A. 2016. A diagram is worth a dozen images. In European Conference on Computer Vision, 235– 251. Springer. Khattab, O.; Singhvi, A.; Maheshwari, P.; Zhang, Z.; Santhanam, K.; Vardhamanan, S.; Haq, S.; Sharma, A.; Joshi, T. T.; Moazam, H.; Miller, H.; Zaharia, M.; and Potts, C. 2024. DSPy: Compiling Declarative Language Model Calls into State-of-the-Art Pipelines. In International Conference on Learning Representations. Khattak, M. U.; Rasheed, H.; Maaz, M.; Khan, S.; and Khan, F. S. 2023. MaPLe: Multi-Modal Prompt Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 19113–19122.
Lau, J. J.; Gayen, S.; Ben Abacha, A.; and Demner-Fushman, D. 2018. A dataset of clinically generated visual questions and answers about radiology images. Scientific Data, 5(1): 180251. Li, C.; Wong, C.; Zhang, S.; Usuyama, N.; Liu, H.; Yang, J.; Naumann, T.; Poon, H.; and Gao, J. 2023. LLaVAMed: Training a Large Language-and-Vision Assistant for Biomedicine in One Day. Advances in Neural Information Processing Systems, 36: 28541–28564. Liu, B.; Zhan, L.-M.; Xu, L.; Ma, L.; Yang, Y.; and Wu, X.-M. 2021. SLAKE: A Semantically-Labeled KnowledgeEnhanced Dataset for Medical Visual Question Answering. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), 1650–1654. IEEE. Liu, H.; Wang, Z.; Guo, Y.; Shou, H.; and Tang, X. 2026. Adaptive Prompt Structure Factorization: A Framework for Self-Discovering and Optimizing Compositional Prompt Programs. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 11690–11714. Association for Computational Linguistics. Lu, P.; Mishra, S.; Xia, T.; Qiu, L.; Chang, K.-W.; Zhu, S.-C.; Tafjord, O.; Clark, P.; and Kalyan, A. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35: 2507–2521. Madaan, A.; Tandon, N.; Gupta, P.; Hallinan, S.; Gao, L.; Wiegreffe, S.; Alon, U.; Dziri, N.; Prabhumoye, S.; Yang, Y.; Gupta, S.; Majumder, B. P.; Hermann, K.; Welleck, S.; Yazdanbakhsh, A.; and Clark, P. 2023. Self-Refine: Iterative Refinement with Self-Feedback. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 46534–46594. Masry, A.; Long, D. X.; Tan, J. Q.; Joty, S.; and Hoque, E. 2022. ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, 2263–2279. Mathew, M.; Karatzas, D.; and Jawahar, C. 2021. DocVQA: A Dataset for VQA on Document Images. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2200–2209. Ou, Y.; Zhou, W.; Ding, S.; Li, L.; Wu, J.; Wang, T.; Chen, J.; Wang, S.; Xu, X.; Zhang, N.; Chen, H.; and Jiang, Y. E. 2025. Symbolic learning enables self-evolving agents. AI Open, 6: 314–322. Pryzant, R.; Iter, D.; Li, J.; Lee, Y.; Zhu, C.; and Zeng, M. 2023. Automatic Prompt Optimization with “Gradient Descent” and Beam Search. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 7957–7968. Sclar, M.; Choi, Y.; Tsvetkov, Y.; and Suhr, A. 2024. Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting. In International Conference on Learning Representations.
Shinn, N.; Cassano, F.; Gopinath, A.; Narasimhan, K.; and Yao, S. 2023. Reflexion: Language Agents with Verbal Reinforcement Learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 8634–8652. Singh, A.; Natarajan, V.; Shah, M.; Jiang, Y.; Chen, X.; Batra, D.; Parikh, D.; and Rohrbach, M. 2019. Towards VQA Models That Can Read. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8317–8326. Wang, P.; Bai, S.; Tan, S.; Wang, S.; Fan, Z.; Bai, J.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; et al. 2024. Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution. arXiv preprint arXiv:2409.12191. xAI. 2024. RealWorldQA. https://huggingface.co/datasets/ xai-org/RealworldQA. Dataset on Hugging Face; accessed 2026-07-24. Yang, C.; Wang, X.; Lu, Y.; Liu, H.; Le, Q. V.; Zhou, D.; and Chen, X. 2024. Large language models as optimizers. In International Conference on Learning Representations. Yue, X.; Ni, Y.; Zhang, K.; Zheng, T.; Liu, R.; Zhang, G.; Stevens, S.; Jiang, D.; Ren, W.; Sun, Y.; Wei, C.; Yu, B.; Yuan, R.; Sun, R.; Yin, M.; Zheng, B.; Yang, Z.; Liu, Y.; Huang, W.; Sun, H.; Su, Y.; and Chen, W. 2024. MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9556–9567. Yuksekgonul, M.; Bianchi, F.; Boen, J.; Liu, S.; Lu, P.; Huang, Z.; Guestrin, C.; and Zou, J. 2025. Optimizing Generative AI by Backpropagating Language Model Feedback. Nature, 639(8055): 609–616. Zhang, P.; Jin, H.; Hu, L.; Li, X.; Kang, L.; Luo, M.; Song, Y.; and Wang, H. 2025. REVOLVE: Optimizing AI Systems by Tracking Response Evolution in Textual Optimization. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, 75216–75233. PMLR. Zhou, K.; Yang, J.; Loy, C. C.; and Liu, Z. 2022a. Conditional Prompt Learning for Vision-Language Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 16816–16825. Zhou, K.; Yang, J.; Loy, C. C.; and Liu, Z. 2022b. Learning to Prompt for Vision-Language Models. International Journal of Computer Vision (IJCV), 130(9): 2337–2348. Zhou, Y.; Muresanu, A. I.; Han, Z.; Paster, K.; Pitis, S.; Chan, H.; and Ba, J. 2023. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations.