MM-IssueLoc: A Controlled Benchmark for Evaluating Visual Evidence in Multimodal Repository-Level Issue Localization

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

MM-IssueLoc: A Controlled Benchmark for Evaluating Visual Evidence in Multimodal Repository-Level Issue Localization

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

artificialintelligence knowledgerepresentation reasoning

artificial intelligence, reasoning, knowledge representation

MM-IssueLoc: A Controlled Benchmark for Evaluating Visual Evidence in Multimodal Repository-Level Issue Localization Shaoxiong Zhan1,2 * , Shi Hu2 * , Boyu Feng2 , Hai Lin1 , Andrew Gong2 , Zhengda Zhou2 , Jiaying Zhou2 , Yunyun Hou2 , Hao Su2 , Hai-Tao Zheng1 † 1 Tsinghua University 2 JD.com [email protected], [email protected], [email protected]

Abstract

and identifies the code locations relevant to resolving the issue (Chen et al., 2025). For automated software engineering (SE) systems, localization is a prerequisite for issue resolving: errors at this step propagate into every downstream stage of patch generation. Real-world repository issues are natively multimodal; developers routinely attach screenshots, UI states, error dialogs, rendered outputs, and log excerpts alongside textual descriptions. Recent work has begun to extend SE benchmarks beyond text, e.g., SWE-bench Multimodal (Yang et al., 2024b), which studies whether SE agents generalize to visually grounded tasks. However, no existing benchmark isolates the role of visual evidence at the localization step. Textonly benchmarks such as SWE-bench and its multilingual extensions strip images entirely (Jimenez et al., 2023; Zan et al., 2025). SWE-bench Multimodal (Yang et al., 2024b) retains images but evaluates end-to-end patch generation, so localization is folded into repair, and whether an image helped identify the right file or function cannot be read off the final pass/fail signal. Neither setting provides fine-grained localization gold, per-image annotations, or paired with-image versus without-image runs. These are the ingredients needed to treat visual evidence as a first-class input to repositorylevel localization. We introduce MM-IssueLoc, a controlled benchmark and evaluation protocol for multimodal repository level issue localization. Each instance is grounded in a real GitHub issue, its linked pull request, and the repository snapshot before the fix. MM-IssueLoc contains 652 issue PR instances across 23 programming languages, with 1,050 issue images, file level gold labels for all instances, and function level gold labels for 343 instances. Each image is annotated with one of seven evidence categories, such as UI screenshot, error message, rendering result, code screenshot, log output, and data visualization, as well as a relevance score

arXiv:2607.15205v1 [cs.SE] 16 Jul 2026

Real repository issues routinely include visual evidence such as screenshots, error dialogs, rendered UI states, and logs, yet repositorylevel issue localization is evaluated mostly as a text-only task. Existing multimodal SE benchmarks evaluate end-to-end repair, entangling localization with patch synthesis and obscuring whether visual input helped, hurt, or was ignored. We introduce MM-IssueLoc, a controlled benchmark and evaluation protocol for repository-level localization with visual evidence. MM-IssueLoc contains 652 issuePR instances across 23 languages, with annotations for 7 image categories and 4 relevance levels. It provides file-level and functionlevel gold labels, paired text-only and withimage evaluation, and VCE-based diagnostics that convert images into structured textual evidence. We evaluate LLM-based and retrievalbased systems, including MM-IssueLoc-VLEmb as a controlled multimodal retriever. Results show that existing systems remain far from reliable multimodal repository localization: the strongest agent reaches 38.96 file Acc@5 and 22.45 function Acc@10, while the strongest retriever reaches 33.86 function Acc@10. Cross-benchmark comparisons show that high localization scores on text-dominant SWE benchmarks do not transfer cleanly to multimodal issue localization. MM-IssueLoc turns visual evidence into an explicit evaluation variable, enabling future work to test whether systems improve by using visual evidence for localization, rather than by relying on text-only cues or downstream patchgeneration effects. https://github.com/ Jasaxion/MM-IssueLoc-Bench

Introduction

Repository-level issue localization takes an issue description and a repository snapshot as input, * Equal contribution. †

Corresponding author.

MM-IssueLoc EvalData Dashboard Top Languages

Image Category

652 File-level 343 Function-level 1050 Attached Images 608 Open Repositories 202 Median Repo Stars

Relevance Score

Annotation Provenance

Difficulty Bucket

Rule-based Construction & Human Review

Changed Files Easy: 1 Medium: 2~3 Hard: > 4

Figure 1: Dashboard of MM-IssueLoc. MM-IssueLoc contains 652 file-level instances and 343 function-level instances, drawn from 608 repositories across 23 programming languages, with 1,050 attached issue images. Each panel reports the distribution of one annotation dimension: programming language, image category, relevance score, annotation provenance, and difficulty bucket.

indicating whether the image is harmful, neutral, helpful, or vital for localization. The benchmark also includes 55 human reviewed harmful image instances, which serve as a robustness stress test for visually plausible but localization incorrect evidence.

the best function level agent result reaches 22.45 Acc@10. Controlled ablations further show that visual evidence can provide useful localization signal: removing images reduces MM-IssueLocVL-Embedding by 4.44 to 4.91 File@5 points, while VCE improves several agent framework pairs by converting visual content into structured text. These results motivate MM-IssueLoc as a controlled benchmark for evaluating when and how visual evidence contributes to repository level localization. Our contributions are three-fold: 1) we introduce MM-IssueLoc, a controlled multimodal issue localization benchmark with 652 real issue PR instances, 1,050 images, seven evidence categories, four relevance levels and file/function level gold labels; 2) we provide a controlled evaluation protocol with text only, with image, VCE, and VCE+image modes for isolating visual evidence under strict Acc@K metrics; and 3) we evaluate representative agents and retrievers, introduce MM-IssueLoc-VLEmbedding as a controlled ablation model, and show that visual evidence offers useful but model dependent localization signal.

MM-IssueLoc is designed as an evaluation instrument rather than as a leaderboard for a single model family. It supports four paired input modes: text-only, with-image, Visual Content Evidence (VCE), and VCE+image. These modes allow us to separate the effect of visual information from the effect of raw image conditioning. We evaluate both llm-based and retrieval-based localization systems, including AgentLess, LocAgent, OpenHands, Mini-SWE-Agent, BM25, SweRank, Qwen3-VLEmbedding, and our controlled multimodal retriever, MM-IssueLoc-VL-Embedding (Xia et al., 2024; Chen et al., 2025; Wang et al., 2025; Reddy et al., 2025; Li et al., 2026). We further compare representative systems across SWE-bench Lite, SWE-bench Verified, SWE-bench Multimodal, and MM-IssueLoc (Appendix B.4). These crossbenchmark results show that strong localization performance on text-dominant SWE benchmarks does not transfer cleanly to multimodal issue localization, which highlights the need for a benchmark that controls visual evidence explicitly.

Related Work

Repository-level SE benchmarks. SWE-bench establishe repository issue resolving as a scalable benchmark for automated SE systems (Jimenez et al., 2023). Its later variants and related multilingual efforts broaden the issue-resolving ecosystem, including multilingual coverage in Multi-

Our findings show that MM-IssueLoc exposes a capability gap that is difficult to observe in text dominant benchmarks. On MM-IssueLoc, the best file level result reaches 38.96 Acc@5, and 2

Benchmark

SWE-bench (Zan et al., 2025) and larger dataconstruction efforts such as SWE-smith (Yang et al., 2025). SWE-bench Multimodal is the closest prior artifact to MM-IssueLoc: it asks whether SE agents generalize to visual, user-facing JavaScript tasks (Yang et al., 2024b). However, these benchmarks evaluate issue resolving as a whole. MMIssueLoc instead isolates localization, adds perimage relevance labels and harmful-image controls, and reports both file-level and function-level gold across 23 languages.

SWE-bench / Verified SWE-bench Multimodal Multi-SWE-bench LocAgent / Loc-Bench MM-IssueLoc

Multimodal

Repo-level

Image relevance

Function gold

No Yes No No Yes

Yes Yes Yes Yes Yes

– No – – Yes

Partial No Patch-derived Yes Yes

Table 1: Positioning of MM-IssueLoc. The benchmark is designed to isolate visual evidence in repository-level localization, not to replace end-to-end issue-resolving benchmarks.

on this line but serves a different purpose: it is a controlled retriever for testing image-presence, stage-curriculum, and hard-negative design under a fixed benchmark protocol. Its contrastive objective follows the general InfoNCE/CLIP-style retrieval paradigm (Oord et al., 2018; Radford et al., 2021).

Repository Search and Localization agents. Recent SE agents provide different ways for models to inspect and navigate repositories. SWEagent studies agent-computer interfaces for software engineering (Yang et al., 2024a), Agentless argues that much of end-to-end repair can be decomposed into simpler localization and patching components (Xia et al., 2024), and OpenHands exposes a production-oriented SDK for software agents (Wang et al., 2025). Beyond these general software-agent frameworks, recent work has increasingly focused on specialized repository search. LocAgent and CoSIL formulate localization as an agentic, iterative search process over repository structure, where the model repeatedly inspects files, functions, and contextual evidence before selecting suspicious locations (Chen et al., 2025; Jiang et al., 2025). RepoSearcher and CodeScout instead emphasize learned repository search policies, including reinforcement-learning-based training, to improve how models navigate large codebases and identify relevant code regions (Ma et al., 2025; Sutawika et al., 2026). MM-IssueLoc uses these systems as baselines but evaluates only localization outputs. This design separates localization ability from patch-generation success and from framework-specific submission failures.

Multimodal code understanding. Recent work has introduced visual evidence into softwareengineering evaluation. MMCode studies visually rich programming problems (Li et al., 2024), while FailureMem and long-horizon multimodal search examine visual evidence in broader repair or search pipelines (Ma et al., 2026; Du et al., 2026). These studies motivate multimodal software reasoning, but they do not provide a controlled benchmark for repository-level issue localization. MMIssueLoc complements this line by isolating localization from repair and by evaluating how visual evidence affects file- and function-level localization.

Benchmark Construction

MM-IssueLoc evaluates whether visual evidence in real issue reports helps localize code responsible for repository-level failures. To make this measurable, each instance is grounded in a GitHub issue, linked pull request, and fixed repository snapshot; supports paired evaluation with and without images; includes per-image evidence type and localization relevance annotations; and provides both file-level and function-level gold labels.

Code retrieval and embedding models. Retrieval methods offer lower-latency alternatives to full agents. SWERank and SWERank+ formulate issue localization as code ranking, including function-level and multilingual settings (Reddy et al., 2025; Gangi Reddy et al., 2025). CodeXEmbed and large-scale code representation learning study general-purpose code embeddings (Liu et al., 2024; Zhang et al., 2024). Multimodal embedding models such as Qwen3-VL-Embedding and jina-embeddings-v4 extend dense retrieval to visual and textual inputs (Li et al., 2026; Günther et al., 2025). MM-IssueLoc-VL-Embedding builds

3.1

Task Definition

Given an issue report and a repository snapshot, the task is to rank code locations by their likelihood of being responsible for the reported failure. Each MM-IssueLoc instance is defined as xi = (ti , bi , Ii , Ri @ci ).

(1)

where ti is the issue title, bi is the issue body, Ii = {ii,1 , . . . , ii,n } is a set of attached images, 3

and Ri @ci denotes the repository snapshot at the base commit of the linked pull request. A system returns either an ordered list of candidate files F̂i = [fi,1 , . . . , fi,K ],

Hands Claude scores 36.00 on the human subset and 37.13 on the AI subset, while AgentLess Claude scores 27.39 and 28.00. The ranking is stable, suggesting that the AI-assisted extension broadens coverage without dominating the benchmark behavior. further benchmark construction details are provided in Appendix C.

(2)

for file-level localization, or an ordered list of qualified functions Ĝi = [gi,1 , . . . , gi,K ],

3.3

(3)

Function-level labels are extracted by intersecting unified-diff line ranges with tree-sitter AST function spans. Functions newly introduced by the PR are excluded from both gold and candidate pools because the task is localization in the base repository, not prediction of new code. Languages with weak grammar coverage can still be evaluated at file level. Naturally harmful images are rare. In the human pool, images that actively mislead localization occur at less than 1%. MM-IssueLoc therefore includes controlled harmful-image synthesis for robustness testing. The synthesis uses three strategies: same-repository image-text mismatch, TFIDF-similar issues with disjoint edit files, and promotion of misleading comment-thread screenshots into the issue-body image slot. Each synthetic instance is scored along seven dimensions, including text sufficiency, surface relevance, image direction, direction deviation, misleading score, developer impact, and suggested relevance label, followed by human review.

for function-level localization. The gold file set Fi⋆ consists of files edited by the human-vetted fixing pull request. For instances with function-level annotations, the gold function set G⋆i consists of edited functions that can be matched to functions in the pre-fix repository snapshot Ri @ci . Functions newly introduced by the pull request are excluded, since they are absent from the input repository and therefore cannot be localized. Function-level gold labels are provided only when tree-sitter extraction succeeds and at least one edited pre-existing function can be identified. 3.2

Function-Level Gold and Harmful-Image Controls

Data Collection and Annotation

MM-IssueLoc is built from public GitHub repositories with at least 50 stars, closed and merged pull requests, non-bot authorship, deduplication, GitHub API enrichment, issue-PR linking, pre/post file retrieval, and complete repository snapshots at the base commit. The released benchmark contains 652 instances, 1,050 localized issue images, and 650 repository snapshots. Its file-level view covers all 652 instances; its function-level view covers 343 instances. Human annotation covers 450 instances by six annotators over approximately 20 hours. Each instance receives one image category among ui_screenshot, behavior_demo, error_message, rendering_result, code_screenshot, data_visualization, and log_output; a relevance score in {−1, 0, +1, +2}; and a difficulty bucket derived from the number of changed files. The AI-assisted extension adds 202 instances through a four-stage VLM gate: relevance scoring, category classification with confidence threshold, patch-alignment verification, and a second-model consistency check. All AI rows retain annotation_by=ai, which enables separate sanity checks. This split is audited in the evaluation rather than treated as bookkeeping. On file Acc@5, Open-

3.4

Training Data and Controlled Retriever

The companion MM-IssueLoc training data is disjoint from the benchmark at the instance_id level and follows the right branch of the pipeline in Figure 2. It contains 13,196 file-level rows over 7,664 instances and 13,769 function-level rows over 3,999 instances. Each anchor is multimodal and uses at most two images. Multipositive expansion emits one row per gold file or function, aligning training with the multi-gold evaluation protocol. For each anchor, hard negatives (Robinson et al., 2021) are mined from samerepository non-gold candidates ranked by Qwen3VL-Embedding-8B, so the model contrasts the true target with plausible repository-local distractors. MM-IssueLoc-VL-Embedding is initialized from Qwen3-VL-Embedding-2B or 8B. The query contains issue text and images; candidates are file or function strings prefixed with File: or Function:. 4

Data Collection Pipeline (Github Crawl)

MM-IssueLoc Embedding Training Data Pipeline

Github Source PR/Issue Pairs

(star ≥ 50, pr closed, Non-bot)

1. Coarse Filter

2. Deduplication

3. API Fetch

Diff, Patch, Images

Annotation & Synthesis Process

4. Merge PR/Issue

A:Relevance

• • • •

MM-IssueLoc EvalSet HumanVerified

C:Patch Align

B:Category

Rule-based harmful image entries

Evaluation

At base commit

Four-stage VLM gate

450 cleaned instances

1. Title Mismatch 2. Disjoint “diff_files” 3. Image Swap

6. Repo Snapshot

Pre/Post File Contents

AI-driven Annotation Track (202 instances)

Human-Annotation Track

(Relevance = -1)

5. Content Download

(652 instances)

D: 2nd VLM Judge

7-class taxonomy 4-level relevance score File & Function Level Gold Difficulty Buckets

MM-Issue VCE Side-Channel Structured-text Image Output

Hard Negative Mining Anchor=Issue+Image Positive=Gold file/function Negative=In-repo negatives via cosine ranking (strong VLEmbedding)

Two-stage Curriculum Stage 1：File-level contrastive training

Code Localization Paradigm

Image-use toggle: Text-only / Image

Leakage Prevention

Separation from evalset

Retrieval-based Retrievers

File-level Gold Evaluation Tools: Strict Acc@k

LLM-based Agents

Function-level Gold

(InfoNCE loss)

Stage 2：Function-level continuation, initialised from Stage 1 (InfoNCE loss)

Figure 2: End-to-end construction and evaluation flow for MM-IssueLoc. The benchmark separates human and AI annotation, explicit harmful-image controls, file/function-level gold extraction, multimodal training data construction, and controlled image-conditioning evaluation modes.

Let qi , pi , and nij denote the normalized query, positive, and hard-negative embeddings. We optimize Li = − log

Retrieval-based Methods. This paradigm directly ranks repository files or functions based on relevance score, including BM25 (Robertson and Zaragoza, 2009), bge-m3 (Chen et al., 2024), SWERank-small, SWERank-large (Reddy et al., 2025), Qwen3-VL-Embedding-2B/8B (Li et al., 2026), and MM-IssueLoc-VL-Embedding-2B/8B. Llm-based methods explore the repository with tool use before returning localized code locations, including AgentLess (Xia et al., 2024), LocAgent (Chen et al., 2025), OpenHands (Wang et al., 2025), and Mini-SWE-Agent (Yang et al., 2024a) with frontier and open-weight multimodal models. All methods are scored by the same evaluation harness. Agents are constrained to output strict JSON for file/function localization and are not allowed to generate patches or modify repositories. For multimodal benchmarks, we evaluate four input modes. The original mode passes issue text and raw images. The text-only mode removes images. The +VCE mode replaces images with a structured visual text block. The +VCE+image mode passes both the structured text and raw images.

exp(qi⊤ pi /τ ) P , exp(qi⊤ pi /τ ) + j exp(qi⊤ nij /τ )

with τ = 0.05. Training follows a file-to-function curriculum: Stage 1 fine-tunes on file-level rows, and Stage 2 continues on function-level rows. Overall, MM-IssueLoc contains 652 issue instances, 1,050 images, and covers 24 programming languages. All instances support file-level evaluation, and 343 also support function-level evaluation. Each instance records annotation provenance, image evidence category, image relevance score, and difficulty, enabling controlled analyses by language, visual evidence type, relevance, annotation source, and search-space size. Figure 1 summarizes these benchmark statistics, and the released JSONL files contain the same fields used in evaluation, making the statistics directly reproducible.

Experimental Setup

4.1

Baselines 4.2

We evaluate two representative paradigms for issue localization: retrieval-based localization and llm-based localization. These baselines cover both direct relevance ranking and interactive repository exploration, allowing us to evaluate whether visual evidence improves localization under different levels of search and reasoning complexity.

Visual Context Extraction (VCE)

VCE is a non-trainable image-to-text adapter. For each image, a VLM extracts structured fields such as OCR text, error signals, UI elements, user actions, code hints, saliency, confidence, and notes. The fields are rendered as a compact textual block and appended to the issue body. The text-only to 5

+VCE comparison measures the value of structured visual content, while original to +VCE+image measures whether raw pixels add further signal. See Appendix D for the VCE workflow and case studies. 4.3

independently encoded, so their outputs are naturally aligned with the evaluation space. In contrast, LLM-based systems must generate function identifiers as text, where incomplete names, missing qualifiers, or invalid output formats can reduce exact-match scores. Figure 3 further explains the aggregate results. First, performance collapses with difficulty. OpenHands GPT-5.2 drops from 83.10 Acc@10 on easy instances to 2.84 on hard instances, while MMIssueLoc-VL-Emb-8B drops from 74.18 to 3.98, showing that multi-edit issues remain the main bottleneck. Second, language slices reveal nonuniform localization behavior. Both systems perform better on JavaScript and related frontend languages, where screenshots often provide direct UI or rendering evidence, but performance is weaker on languages such as C++ and Go, where the visual signal is less directly tied to localized code. Third, agent results are also shaped by harness reliability. With the same Claude-Sonnet-4.6 backend, OpenHands achieves a 98.6% valid submission rate and 37.42 File@5, whereas Mini-SWE-Agent achieves 47.1% and 18.71. Thus, RQ1 shows that current systems are limited not only by model capability, but also by difficulty, language properties, and finaloutput reliability.

Evaluation Granularities and Metrics

The harness evaluates both file-level (n = 652) and function-level (n = 343) localization. Following prior localization work (Chen et al., 2025), we use strict Acc@K as the primary metric: N

Acc@K =

1 X 1 [G⋆i ⊆ TopK(ri )] . N

(4)

i=1

A prediction is successful only if all gold files or qualified functions appear in the top-K ranked list. This all-gold criterion is important for multilocation issues, where finding only one correct location may still fail to recover the full edit scope. Function-level results are reported at K ∈ {5, 10}, since strict Acc@1 and Acc@3 are necessarily zero whenever an instance contains more than one gold function. We also report MRR, Recall@K, and Hit@K as supporting metrics.

Results and Analysis

5.1

RQ1: How Well Do Current Systems Localize Multimodal Repository Issues?

5.2

RQ2: Is Visual Evidence Useful, and Do Systems Use It Reliably?

RQ2 separates visual evidence from visual use. MM-IssueLoc is designed for this distinction: each image is annotated with evidence category and relevance, and the harmful subset introduces visually plausible but localization-incorrect cues. Therefore, the no-image ablation evaluates whether systems exploit image-conditioned localization evidence, rather than whether images are merely attached to issue reports. Cross-benchmark results further motivate this question. As reported in Appendix B.4, current systems already achieve strong localization performance on text-dominant SWE benchmarks. For example, OpenHands reaches 94.53 File@5 on SWE-bench-Lite and 90.20 File@5 on SWE-benchVerified. However, performance drops substantially on SWE-bench-MM, where the best File@5 is 43.14. This gap suggests that repository-level localization is relatively mature in text-only settings, but remains much less developed when issue evidence involves images. MM-IssueLoc fills this diagnostic gap by providing controlled image relevance,

RQ1 establishes the capability boundary of current systems. As shown in Table 2, multimodal repository-level issue localization remains far from solved. The strongest file-level result is obtained by OpenHands with GPT-5.2, which reaches 38.96 File@5, while OpenHands with Claude-Sonnet-4.6 achieves the best File@1 and File@3. However, even the best system still misses all correct files in the top five for more than 60% of instances. This suggests that current systems can provide useful candidates, but are not yet reliable enough for downstream repair-oriented workflows. The results further reveal a clear paradigm difference. LLM-based agents are stronger at filelevel localization, likely because they can inspect the repository and follow issue-level clues during exploration. Retrieval-based methods are more competitive at function-level localization: MMIssueLoc-VL-Emb-8B achieves the best Func@10 score of 33.86. This advantage is partly structural. Retrievers rank pre-enumerated functions that are 6

Paradigm

Model

File@1

File@3

File@5

Func@5

Func@10

GPT-5.2 Claude-Sonnet-4.6 Kimi-K2.5 Kimi-K2.5 Gemini-3-Pro Qwen3.5-122B Claude-Sonnet-4.6

23.93 25.61 24.23 18.15 17.87 21.36 15.18

36.35 36.66 35.28 28.83 27.70 25.39 18.71

38.96 37.42 35.74 32.74 30.11 25.70 18.71

21.87 20.41 20.12 12.14 12.92 12.98 10.79

22.45 20.41 20.12 12.14 12.92 12.98 11.08

Text Text Text Text Multimodal Multimodal Multimodal Multimodal

6.75 7.36 3.83 10.12 11.66 12.11 13.19 13.96

16.56 14.88 7.98 21.32 23.16 26.69 25.61 27.61

21.01 20.40 10.74 26.69 29.29 32.06 32.06 32.82

12.54 18.66 19.83 20.95 24.20 25.36 25.36 22.74

19.83 26.24 26.24 27.82 30.90 32.65 32.36 33.86

LLM-based localization OpenHands OpenHands OpenHands AgentLess AgentLess LocAgent Mini-SWE-Agent Retrieval-based localization BM25 bge-m3-0.5B SWERank-small-0.1B SWERank-large-7B Qwen3-VL-Emb-2B Qwen3-VL-Emb-8B MM-IssueLoc-VL-Emb-2B MM-IssueLoc-VL-Emb-8B

Table 2: Overall comparison on MM-IssueLoc. File metrics are Acc@1/3/5, and function metrics are Acc@5/10. Values are percentages. Bold numbers indicate the best result in each column, and underlined numbers indicate the best retrieval-based file-level results. 90

Difficulty collapse 83.10

80 74.18

OpenHands harness (Claude-Sonnet-4.6)

File Acc@5 (%)

Acc@10 (%)

Agent harness reliability 40

70 50 40

31.94 25.86

30 20

30 20 10

10 0

Language sensitivity

OpenHands GPT-5.2 MM-IssueLoc-VL-Emb-8B

OpenHands GPT-5.2 MM-IssueLoc-VL-Embedding-8B

File Acc@5 (%)

3.98 2.84

Easy

Medium

Hard

30 25 20 15

a n + pt pt Scri Pytho vaScri C+ Jav Ja Type

(98.6%, 37.42)

Mini-SWE-Agent harness (Claude-Sonnet-4.6)

(47.1%, 18.71)

Valid final submission rate (%)

100

Figure 3: Current capability frontier on MM-IssueLoc. Agents are stronger at broad file discovery, retrievers are competitive at function ranking, hard instances collapse across paradigms, and language slices expose additional sensitivity that aggregate scores obscure.

category, and harmful-image annotations. The main results also indicate that MM-IssueLoc contains usable visual signal. In Table 2, text-only retrievers reach 19.83–27.82 Func@10, whereas MM-IssueLoc-VL-Embedding-8B achieves 33.86 Func@10. This shows that visual evidence can be converted into fine-grained localization signal when a retriever is trained for multimodal issue localization. Detailed retriever ablations are reported in Appendix B.2. Table 3 reports the controlled no-image ablation. We define

hurts. The results show uneven image use across systems. AgentLess GPT-5.2 benefits from images (∆img = −2.66), while OpenHands changes by less than one point. MM-IssueLoc-VL-Embedding is more image-sensitive: removing images reduces File@5 by 4.91 points for the 2B model and 4.44 points for the 8B model. Thus, visual evidence is useful, but current agent systems do not exploit it consistently. Fine-grained slices explain why aggregate image gains are unstable. Relevance labels do not induce monotonic accuracy changes, since harmful images are not always recognized as misleading and some instances remain solvable from text alone. Category slices further show that images are most helpful when they are directly tied to observable

∆img (m) = Acc@5file (m, text-only) − Acc@5file (m, text+image), so negative values indicate that removing images 7

Method

Model

AgentLess OpenHands OpenHands Retriever Retriever

GPT-5.2 GPT-5.2 Claude-Sonnet-4.6 MM-IssueLoc-VL-Embed-2B MM-IssueLoc-VL-Embed-8B

No-image File@5 With-image File@5 ∆ File@5 26.93 38.65 38.34 27.15 28.38

29.59 38.96 37.42 32.06 32.82

-2.66 -0.31 +0.92 -4.91 -4.44

Table 3: With-image versus no-image ablation on MM-IssueLoc. ∆ is no-image minus with-image file Acc@5. Negative values mean that removing images hurt the cell.

software behavior, such as UI screenshots, behavior demos, rendering results, code screenshots, logs, or data visualizations. These signals often help identify relevant files, but they do not always map cleanly to edited functions. Additional relevance, category, difficulty, and annotation slices are reported in Appendix B.1.

exhibits much larger variation: Claude-Sonnet-4.6 drops by 5.62 points, GPT-5.2 gains 4.74 points, and Qwen3.5-122B-A10B drops by 11.69 points. This indicates that VCE exposes useful image content, while the effect of raw pixels is mediated by the framework’s visual-input integration.

6 5.3

RQ3: Does Localization Depend on Image Content or Raw Image Format?

Discussion, Limitations, and Future Work

Discussion. MM-IssueLoc follows prior issueresolving benchmarks by using merged pull request diffs as localization gold labels. Although such labels are not minimal causal explanations, they provide a scalable and auditable proxy for code locations modified by human developers. We therefore use strict all-gold Acc@K to measure whether a method recovers the full edit scope, and provide function-level labels when tree-sitter can reliably map edits to pre-existing functions. We also release per-instance predictions to support reanalysis under alternative scoring rules.

RQ3 diagnoses whether the benefit of visual evidence comes from image content or from the raw image format. We use Visual Content Evidence (VCE)(fig. 6) to convert each image into structured textual evidence. The +VCE setting removes raw images and provides only structured visual text, while the +VCE+image setting provides both VCE and the original image. Thus, the gain from the original setting to +VCE indicates whether image content is useful after being textualized, and the difference between +VCE and +VCE+image measures the residual value of raw pixels. Table 4 reports the VCE diagnostic results using the same three model backends under AgentLess and LocAgent. VCE improves file-level localization in several settings. For AgentLess, ClaudeSonnet-4.6 improves from 25.15 original File@5 to 28.53 with +VCE, and GPT-5.2 improves from 28.15 to 29.07. For LocAgent, Qwen3.5-122BA10B improves substantially from 25.70 to 35.04. These results show that images contain useful localization evidence and that such evidence can be consumed in structured textual form. However, raw images do not consistently add value once VCE is provided. We define the rawpixel residual as

Limitations. MM-IssueLoc is designed for controlled diagnosis, and its results should be interpreted within this scope. The harmful-image subset is primarily rule-constructed and human-reviewed, so it serves as a robustness stress test rather than an estimate of real-world frequency. VCE is a diagnostic protocol, not an upper bound on visual reasoning, since it uses fixed extraction fields and a fixed downstream localization setting. Function-level labels may be noisy for language features that are difficult for tree-sitter to resolve, such as macros, templates, or generated code. Our retriever also uses at most two images per issue, leaving manyimage issues as a future stress case. Finally, MMIssueLoc evaluates localization only, rather than end-to-end software repair or downstream patch generation.

ρpix = Acc@5f (VCE+img) − Acc@5f (VCE). For AgentLess, the residuals are small: +0.22 for Claude-Sonnet-4.6, +0.03 for GPT-5.2, and −2.15 for Qwen3.5-122B-A10B. In contrast, LocAgent

Future work. Future work can extend MMIssueLoc along three directions: collecting more 8

Original

+VCE

+VCE+Image

ρpix

Claude-Sonnet-4.6 GPT-5.2 Qwen3.5-122B-A10B

25.15 28.15 28.07

28.53 29.07 26.53

28.75 29.10 24.38

0.22 0.03 -2.15

Claude-Sonnet-4.6 GPT-5.2 Qwen3.5-122B-A10B

27.73 19.13 25.70

30.93 16.12 35.04

25.31 20.86 23.35

-5.62 4.74 -11.69

Method

Model

AgentLess AgentLess AgentLess LocAgent LocAgent LocAgent

Table 4: VCE diagnostic results on MM-IssueLoc, measured by file-level Acc@5. VCE converts image content into structured text, while ρpix measures the residual value of raw pixels after VCE has already been supplied.

naturally occurring misleading images and manyimage issues, improving function-level annotation with language-specific parsers or hybrid static analysis, and studying whether better multimodal localization improves broader software-engineering workflows such as repository understanding, debugging, triage, and downstream repair.

2026. Towards long-horizon agentic multimodal search. arXiv preprint arXiv:2604.12890. Revanth Gangi Reddy, Ye Liu, Wenting Zhao, JaeHyeok Doo, Tarun Suresh, Daniel Lee, Caiming Xiong, Yingbo Zhou, Semih Yavuz, and Shafiq Joty. 2025. Swerank+: Multilingual, multi-turn code ranking for software issue localization. arXiv e-prints, pages arXiv–2512.

Conclusion

Michael Günther, Saba Sturua, Mohammad Kalim Akram, Isabelle Mohr, Andrei Ungureanu, Bo Wang, Sedigheh Eslami, Scott Martens, Maximilian Werk, Nan Wang, and 1 others. 2025. jina-embeddings-v4: Universal embeddings for multimodal multilingual retrieval. In Proceedings of the 5th Workshop on Multilingual Representation Learning (MRL 2025), pages 531–550.

MM-IssueLoc shows that multimodal repositorylevel issue localization is a distinct and underevaluated capability. Cross-benchmark results indicate that strong localization performance on textdominant SWE benchmarks is not sufficient evidence of multimodal issue localization ability. By pairing real issue-PR instances with file-level and function-level gold labels, image-category and relevance annotations, and controlled text-only, with-image, VCE, and VCE+image modes, MMIssueLoc makes visual evidence an explicit evaluation variable. Our results show that visual evidence can provide useful localization signal, but current agents and retrievers use it unevenly. Overall, MMIssueLoc provides a controlled basis for evaluating how visual evidence affects repository-level localization.

Zhonghao Jiang, Xiaoxue Ren, Meng Yan, Wei Jiang, Yong Li, and Zhongxin Liu. 2025. Issue localization via llm-driven iterative code graph searching. In 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 3034– 3045. IEEE. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? In The twelfth international conference on learning representations. Kaixin Li, Yuchen Tian, Qisheng Hu, Ziyang Luo, Zhiyong Huang, and Jing Ma. 2024. Mmcode: Benchmarking multimodal large language models for code generation with visually rich programming problems. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 736–783.

References Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216, 4(5).

Mingxin Li, Yanzhao Zhang, Dingkun Long, Keqin Chen, Sibo Song, Shuai Bai, Zhibo Yang, Pengjun Xie, An Yang, Dayiheng Liu, and 1 others. 2026. Qwen3-vl-embedding and qwen3-vlreranker: A unified framework for state-of-the-art multimodal retrieval and ranking. arXiv preprint arXiv:2601.04720.

Zhaoling Chen, Robert Tang, Gangda Deng, Fang Wu, Jialong Wu, Zhiwei Jiang, Viktor Prasanna, Arman Cohan, and Xingyao Wang. 2025. Locagent: Graphguided llm agents for code localization. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8697–8727.

Ye Liu, Rui Meng, Shafiq Joty, Silvio Savarese, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. 2024. Codexembed: A generalist embedding model family for multiligual and multi-task code retrieval. arXiv preprint arXiv:2411.12644.

Yifan Du, Zikang Liu, Jinbiao Peng, Jie Wu, Junyi Li, Jinyang Li, Wayne Xin Zhao, and Ji-Rong Wen.

Ruize Ma, Yilei Jiang, Shilin Zhang, Zheng Ma, Yi Feng, Vincent Ng, Zhi Wang, Xiangyu Yue, Chuanyi Li, and Lewei Lu. 2026. Failuremem: A failure-aware multimodal framework for autonomous software repair. arXiv preprint arXiv:2603.17826.

John Yang, Carlos E Jimenez, Alex L Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R Narasimhan, and 1 others. 2024b. Swe-bench multimodal: Do ai systems generalize to visual software domains? In The Thirteenth International Conference on Learning Representations.

Zexiong Ma, Chao Peng, Qunhong Zeng, Pengfei Gao, Yanzhen Zou, and Bing Xie. 2025. Tool-integrated reinforcement learning for repo deep search. arXiv preprint arXiv:2508.03012.

John Yang, Kilian Lieret, Carlos E Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. 2025. Swe-smith: Scaling data for software engineering agents. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PmLR.

Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Shulin Xin, Linhao Zhang, Qi Liu, Aoyan Li, Lu Chen, Xiaojian Zhong, and 1 others. 2025. Multiswe-bench: A multilingual benchmark for issue resolving. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

Revanth Gangi Reddy, Tarun Suresh, JaeHyeok Doo, Ye Liu, Xuan Phi Nguyen, Yingbo Zhou, Semih Yavuz, Caiming Xiong, Heng Ji, and Shafiq Joty. 2025. Swerank: Software issue localization with code ranking. arXiv preprint arXiv:2505.07849.

Dejiao Zhang, Wasi Ahmad, Ming Tan, Hantian Ding, Ramesh Nallapati, Dan Roth, Xiaofei Ma, and Bing Xiang. 2024. Code representation learning at scale. arXiv preprint arXiv:2402.01935.

Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond, volume 4. Now Publishers Inc.

Training and Evaluation Setup

A.1

Training Hyperparameters

We report the training details of MM-IssueLoc-VLEmbedding in the multimodal setting, where each input contains the issue text and attached images. Table 5 summarizes the two-stage protocol and the function-only ablation. Stage 1 uses file-level supervision for coarse localization, while Stage 2 continues from the Stage-1 checkpoint with functionlevel supervision. The function-only ablation directly trains on function-level data to assess the effect of file-level initialization. Table 6 lists the shared optimization settings. We use full fine-tuning with the vision encoder frozen to reduce training cost while preserving pretrained visual representations. Table 7 reports the hardware and batch configurations. All experiments use four NVIDIA H200 GPUs. The effective batch size equals the perGPU batch size multiplied by gradient accumulation steps and the number of GPUs.

Joshua David Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka. 2021. Contrastive learning with hard negative samples. In International Conference on Learning Representations. Lintang Sutawika, Aditya Bharat Soni, Apurva Gandhi, Taha Yassine, Sanidhya Vijayvargiya, Yuchen Li, Xuhui Zhou, Yilin Zhang, Leander Melroy Maben, Graham Neubig, and 1 others. 2026. Codescout: An effective recipe for reinforcement learning of code search agents. arXiv preprint arXiv:2603.17829. Xingyao Wang, Simon Rosenberg, Juan Michelini, Calvin Smith, Hoang Tran, Engel Nyst, Rohit Malhotra, Xuhui Zhou, Valerie Chen, Robert Brennan, and 1 others. 2025. The openhands software agent sdk: A composable and extensible foundation for production agents. arXiv preprint arXiv:2511.03690. Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2024. Agentless: Demystifying llm-based software engineering agents. arXiv preprint arXiv:2407.01489.

A.2

John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024a. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528– 50652.

Agent Evaluation Parameters

We configure each agent framework following the setup reported in its original paper, and restrict every framework to localization-only output on a shared evaluation harness. 10

Table 5: Training configurations for the proposed two-stage training and function-only ablation. Setting Stage-1 File Stage-2 Function Function-only Ablation

Data level

Initialization

Epochs

Seed

File-level Function-level Function-level

Qwen3-VL-Embedding Stage-1 checkpoint Qwen3-VL-Embedding

2 3 3

1 × 10−5 5 × 10−6 1 × 10−5

42 43 42

Table 6: Shared optimization hyperparameters. Hyperparameter

Value

Max sequence length Contrastive temperature Warmup ratio Logging steps Checkpoint interval Vision encoder Training strategy Image input in MM setting Image input in text-only setting

8192 0.05 0.05 10 200 steps Frozen Full fine-tuning Enabled Disabled

GPUs

Batch/GPU

Accum. Steps

Global Batch

2B 8B

4×H200 4×H200

4 1

8 16

128 64

Extended Results

B.1

Stratified Evaluation Slices

Helpful

Neutral

Harmful

36.14 35.15 30.20

40.16 29.66 32.28

21.43 21.43 28.57

34.55 32.73 34.55

Category

Behavior

Error

Rendering

Code

Log

Data-viz

Instances

177

least 0.6. Stage C judges whether the proposed patch plausibly addresses the issue. Stage D repeats relevance and category with a second VLM and accepts only if relevance differs by at most one level and category matches. The accepted rows are marked with annotation_by=ai.

Table 16: Image category distribution. Categories are mutually exclusive at the instance level. Relevance

Vital (+2)

Helpful (+1)

Neutral (0)

Harmful (-1)

Instances

202

381

C.4 Table 17: Per-image relevance distribution used for stratified evaluation.

C.2

The harmful subset uses same-repository imagetext mismatches, TF-IDF-similar issues with disjoint edit files, and comment-thread image promotion. Each candidate receives a seven-axis quality assessment: text sufficiency, surface relevance, image direction, direction deviation, misleading score, developer impact, and suggested label. Human review filters implausible or insufficiently misleading candidates.

Benchmark Examples

Figure 5 presents representative examples from MM-IssueLoc, organized by the image categories defined in Table 16. Each example illustrates the issue context, the associated visual evidence, and the corresponding gold annotations at both file and function levels. C.3

Harmful-Image Synthesis

Visual Content Evidence Extraction

D.1

VCE Mechanism

AI Annotation Gate

The AI-assisted extension uses four stages. Stage A estimates relevance and admits only +1 or +2 rows for the main augmented subset. Stage B assigns one of seven image categories with confidence at

Visual Content Evidence (VCE) converts issue images into structured textual evidence before localization. Instead of directly using the raw image 13

as model input, VCE extracts task-relevant visual content, such as error messages, UI states, code snippets, logs, rendering artifacts, and chart patterns, into a compact textual representation. This design provides a diagnostic side channel for separating the contribution of visual content from the effect of direct image conditioning. In our evaluation, VCE is used to test whether localization improvements come from information contained in the image or from the model’s ability to process the image modality itself. D.2

VCE Extraction Examples

Figure 7 shows representative image-to-VCE extraction examples across different image categories. Each example contains the original visual evidence, the extracted VCE text, and its corresponding interpretation for file-level and function-level localization. D.3

VCE Prompt

The full prompt template is provided in Figure 8.

Prompt Templates

E.1

AI-Driven Harmful Impact Prompt

The full prompt template is provided in Figure 9. E.2

AgentLess Prompt

The full prompt template is provided in Figure 10. E.3

LocAgent Prompt

The full prompt template is provided in Figure 11. E.4

OpenHands Prompt

The full prompt template is provided in Figure 12. E.5

Mini-SWE-Agent Prompt

The full prompt template is provided in Figure 13.

Category: Rendering Result

Category: Rendering Result Image

Image

Issue title:Landing page as per new designs

Issue title: Orthogonal view feature

Body: I have worked on the landing page , and created a pull request for the same. <image>

Body: Hi, does the polyscope support an orthogonal view of the camera? I need an orthogonal view but I could not find it. I found I can adjust the FOV may lower the quality XXX

GT file: "client/public/index.html", "client/src/Themes/Theme.js", "client/src/components/App.js", "client/src/providers/DarkThemeProvider.jsx"

GT function: "include/polyscope/view.h:polyscope"

Category: UI Screenshot Image

Category: UI Screenshot Issue title:WooCommerce Related products section CSS issue

Image

Issue title: Shapes on certain layer not selectable

Body: Reported: https://secure.helpscout.net/conversation/724266 382/166432?folderId=607541 <image> This is something I XXX

Body: Version: KLayout 0.29.7, commit 2d4f5d822c2e8c2649387ad5c5e242bbb85c56ec I can‘t select shapes on `mcon.pin 78748`, XXX GT function: "src/edt/edt/edtServiceImpl.cc:edt"

GT file: "assets/front/css/_parts/1_0_plugin_compat.css"

Category: Log Output

Category: Log Output Image

Body: I noticed that after some time without receiving any commands, the bot has problems determining whether or not it is connected XXX

Image

Issue title: Feature request: support for /32 DHCP allocations Body: Hi, Thanks for iPXE, I‘m slowly getting familiar with it. On Linux it’s possible to assign a /32 v4 IP XXX

GT file: "config/config.py", "vulkan/music/Player.py", "vulkan/music/utils.py"

GT function: "src/net/ipv4.c:ipv4_add_miniroute", "src/net/ipv4.c:ipv4_is_fragment", "src/net/ipv4.c:ipv4_route ( unsigned int scope_id,\n\t\t\t\t struct in_addr *dest )"

Issue title: Bot stop responding after some time

Category: Data Visualization

Category: Data Visualization Issue title: Feature request: Dashboard Pie Charts Display percentage instead of hours

Image

Body: Improve plot_splits for time series splits. Currently the plot present some limitation. Here is an exemple with code: ``` import pandas as pd, XXX

Body: Ability to change hovertext display on piechart segments from hours to a percentage, or at least include the XXX

GT function: "mlxtend/evaluate/time_series.py:plot_split_indices"

GT file: "ui/src/dashboard/Entry/DashboardPieChart.tsx"

Category: Behavior Demo

Category: Behavior Demo Image

Image

Issue title: When the ContentType is set to `multipart/form-data` and query XXX

GT file: "src/main/java/com/ly/doc/model/torna/Apis.java", "src/main/java/com/ly/doc/model/torna/ DubboInfo.java", "src/main/java/com/ly/doc/utils/TornaUtil.java"

Image

GT function: "crates/bevy_editor_pls_core/src/editor.rs:Editor.v iewport_interaction_active"

Category: Code Screenshot

Issue title: Feature Request: Support another case of using Promises.all()

Image

Body: Related to https://github.com/playwrightcommunity/eslint-plugin-playwright/issues/107 It looks like I am XXX GT file: "src/rules/missing-playwright-await.ts", "test/spec/missing-playwright-await.spec.ts"

Issue title: DataTable became blank after window resizing Body: Hello, I found a little bug when I resize my window on a dashboard, one of my enhanced table became blank. Here is the Javascript error that appear after resizing: ``` TypeError: "col.agg XXX GT function: "public/enhanced-table-viscontroller.js:EnhancedTableVisController"

Category: Error Message Image

Issue title: viewport_interaction_active function error Body: When the editor window is created but not entered by the mouse or loses focus directly from the viewport area of the editor window: `assert!(self.pointer_used, false);` XXX

Body: When the ContentType is set to multipart/form-data and query parameters (annotated with `@RequestParam`) are XXX

Category: Code Screenshot

Issue title: Enhance plot

Category: Error Message Issue title: Core functionality crashes while working with FSE

Image

Body: Steps to Reproduce 1. Install a Block theme 2. Try to use the Insert Special Characters feature via FSE 3. Observe it crashes with XXX

Issue title: `Drawer` will cause mismatch in SSR mode Body: Reproduction link [https://stackblitz.com/edit/nextjszxvknz?file=pages/index.js](https://stackblitz.com/edit /nextjs-zxvknz?file=pages/index.js) ### Steps to reproduce 1. XXX

GT file: "src/index.js"

GT function: "src/Drawer.tsx:Drawer"

Figure 5: Representative MM-IssueLoc examples grouped by image category.

VCE (Visual Content Extraction) overview Screenshot Input

Screenshot-to-Text Extraction

Textualized Visual Evidence

Augmented Issue Text

Code Localization Granularity

Convert screenshots into structured textual clues

Screenshot(s)

OCR text

• Button label

UI elements

• Error message

Error signal

• Function/file clue

Issue input

File-level

Function-Level

• User action

User action

Extracted Textual Evidence

Used as input For localization

Code hint

Figure 6: Overview of the VCE extraction mechanism. Issue images are converted into structured textual evidence and then used as an auxiliary diagnostic input for issue localization.

Example of VCE Category: Rendering Result

VCE + Extracted Textual Evidence OCR text:

":: File View Project\n(clear)\n(fill (text 0 40 40 \"Hello\") \"black\")\n(stroke (text 0 80 40 \"Hello\") \"black\") \nHello\nHello”

UI elements：

"code editor panel","canvas rendering area","menu bar (File View Project)”

Error signal:

{ "kind": "rendering_bug", "type": "text_rendering_failure", "message": "Text rendered as outlined/hollow glyphs instead of proper filled/stroked text", "stack_hint": ["text", "fill", "stroke"] }

User action:

"User typed fill and stroke text commands in Ronin editor and observed incorrect rendering"

Screenshot

Example of VCE Category: Error Message

Code hint：

{ "function_names": ["clear","fill","text","stroke"], "file_patterns": null, "frameworks": ["Ronin"] }

VCE + Extracted Textual Evidence OCR text:

“> date -u && date Wed 15 Jul 2020 06:47:51 PM UTC Thu 16 Jul 2020 03:47:51 AM JST~> http ‘https://codeforces.com/enter‘ HTTP/1.1 200 Cache-Control: private,no-cache,no-store,max-age=0,must-revalidate Connection: keep-alive Content Encoding: gzip Content-Type: text/html;charset=UTF-8 Date: Wed, 15 Jul 2020 18:48:02 GMT Expires: -1 Pragma: no-cache Server: kittenx Set-Cookie: JSESSIONID=...Set-Cookie: ... StrictTransport-Security: max-age=86400 Transfer-Encoding: chunked Vary: Accept-Encoding X-Content-Type-Options: nosniff X-Frame-Options: sameorigin X-XSS-Protection: 1; mode=block"

UI elements:

"terminal output", "HTTP response headers", "HTML source code"

Error signal:

{ "kind": "http", "type": "HTTP 200 but login not working", "message": "Codeforces login endpoint returns HTTP/1.1 200 but login functionality is broken - requires CSRF token and session handling changes", "stack_hint": null }

User action:

"User attempted HTTP request to codeforces.com/enter to debug login issue"

Code hint:

Screenshot

{ "function_names": ["adjustViewport"], "file_patterns": ["codeforces.com/enter"], "frameworks": ["oj-api", "online-judge-tools"] }

Figure 7: Representative VCE extraction examples across image categories, including error messages, UI screenshots, behavior demos, rendering results, code screenshots, data visualizations, and log outputs.

VCE prompt VCE+ Structured Extraction Task You are a screenshot-to-structured-text extractor for GitHub issue images. Your task is to extract structured visual evidence from each image and return a concise JSON object. System Instruction: Output only a single JSON object that conforms exactly to the schema given below. Do not include prose outside the JSON object. Use null for fields that are not applicable. Keep each string concise, preferably fewer than 25 words. Do not invent content that is not visible in the image. Input: • Issue Category: {category} • Issue Title: {issue title} • Issue Body: {issue body} • Issue Image: {image} Extraction Schema: {

}

Field Definitions: • ocr text: verbatim readable text in the image. • error signal: structured error information, such as {kind, type, message, stack hint[]}. • ui elements: salient UI components mentioned or visible in the image. • user action: one-line natural-language description of the user action. • code hint: code-related hints, such as {function names[], file patterns[], frameworks[]}. • visual saliency: the part of the image that is visually highlighted or abnormal. • confidence: extractor’s self-assessed confidence score from 0 to 1. • notes: a short summary of at most 15 words. Category-Specific Extraction Focus: • ui screenshot: focus on UI elements and user action. Usually no error signal is needed. Include visible button, label, or menu text in ocr text. code hint may include framework or component names. • behavior demo: focus on the user action and visual saliency. Describe what the user does step by step and list the widgets involved. • error message: focus on error signal, including error kind, type, message, and stack hints. Include the full verbatim message in ocr text if it is short. Extract likely function or file names from the stack trace. • rendering result: focus on what is visually wrong and which widget renders the output. Include axis labels, numbers, or visible text if legible. Add component or framework names when possible. • code screenshot: focus on transcribing the code. Populate code hint.function names. Include visible line numbers in notes. • data visualization: focus on chart titles, axis labels, and abnormal series or bars. Add visualization frameworks such as matplotlib, plotly, or similar names when visible or strongly indicated. • log output: focus on the log text. If the log is short, transcribe it verbatim; otherwise provide a condensed summary. Fill error signal if an error line is present. Use code hint.file patterns for mentioned paths. User Prompt Template: Issue category: {category} Issue title: {issue title} Issue body: {issue body} Extraction focus for this category: {category hint} Return a JSON object following exactly the schema above. Use null for fields you cannot fill in, and never invent content not visible in the image.

Output Requirement: Return only the JSON object. Do not add explanations, markdown fences, or extra text.

Figure 8: VCE prompt

AI-driven harmful impact prompt You are a software engineering expert. Please analyze the relationship between the images and the problem described in the following GitHub issue. Issue Information: • Title: {issue title}

• Repository: {repo full name}

• Language: {repo language} • Body: {issue body} Images in the Issue: {issue images}

Image Source Description: This is a synthetic sample generated by {synthesis strategy}. The images are taken from another issue in the same {image source issue title}.

repository:

Ground-truth Fixed Files: {diff files} Important Note: The ground-truth fixed files are provided only for evaluation. Do not treat them as information available in the original issue. Instructions: Please answer strictly according to the following seven dimensions: 1. Q1 TEXT SUFFICIENT: [YES/NO] Based only on the textual issue description, can one infer the approximate location that needs to be modified? Q1 REASONING: Give a one-sentence explanation. 2. Q2 SURFACE RELEVANCE: [1--5] Rate the surface-level relevance between the images and the issue description. 1 means completely irrelevant, and 5 means highly relevant. Q2 REASONING: Explain in one sentence why the images appear relevant or irrelevant. 3. Q3 IMAGE DIRECTION: If the images are considered, which direction, component, or module would they guide the developer toward? Q3 REASONING: Give a one-sentence explanation. 4. Q4 DIRECTION DEVIATION: [1--5] Rate the deviation between the image-guided direction and the actual fixed location. 1 means the directions are consistent, and 5 means they are completely divergent. Q4 REASONING: Explain the difference between the image-guided direction and the actual fixed files. 5. Q5 MISLEADING SCORE: [0--5] Rate the overall harmful or misleading impact of the images on locating the correct fix. 0 means no misleading impact, and 5 means severely misleading. Q5 REASONING: Give a one-sentence explanation. 6. Q6 DEVELOPER IMPACT: [LOW/MEDIUM/HIGH] Estimate how much additional debugging time a developer may waste if they rely on the images. Q6 REASONING: Give a one-sentence explanation. 7. Q7 LABEL: [-1/0/1/2] Give the final usefulness label for the images in this issue: • -1: harmful, because the images mislead the repair direction; • 0: not helpful; • 1: helpful but not necessary; • 2: extremely helpful and necessary. Q7 REASONING: Give a one-sentence explanation.

Figure 9: AI-Driven Harmful Impact Prompt

Agentless prompt You are a code localization expert. Given a repository structure, candidate file contents, and a bug report or feature request, identify the most likely files and functions that need to be modified. Task Overview: The localization process follows a two-stage hierarchical procedure: 1. File-level localization: identify candidate files from the repository structure and issue description. 2. Function-level localization: identify candidate functions or methods from the selected files and issue description. Stage 1: File-level Localization System Instruction: You are a code localization expert. Given a repository structure and a bug report or feature request, identify the most likely files that need to be modified. Return a JSON array of file paths, ordered by relevance from most relevant to least relevant. Return at most 10 files. Input: • Repository Structure: {repo tree}

• Issue Description: {problem statement}

• Issue Images: {issue images}

optional, only used in multimodal setting

User Prompt: List the files that most likely need to be changed. Return a JSON array of file paths. Expected Output Format: ["path/to/file1.py", "path/to/file2.py", "..."] Stage 2: Function-level Localization System Instruction: You are a code localization expert. Given file contents and a bug report, identify the specific functions or methods that need to be modified. Return a JSON array of function identifiers, ordered by relevance from most relevant to least relevant. Return at most 10 functions. Input: • Candidate Files: {candidate files}

• Candidate File Contents: {file contents} • Issue Description: {problem statement}

• Issue Images: {issue images}

optional, only used in multimodal setting

User Prompt: Identify the functions or methods that need modification. Return a JSON array of identifiers in "file:function" format. Expected Output Format: ["file path:ClassName.method name", "file path:function name", "..."] Important Notes: • The output must be a valid JSON array. • Do not include explanations outside the JSON array. • Rank files and functions by relevance. • Return at most 10 files in the file-level stage. • Return at most 10 functions in the function-level stage. • If images are provided, use them as auxiliary evidence together with the issue description.

Figure 10: AgentLess Prompt

LocAgent prompt You are a code localization agent. Your task is to find the files and functions in a repository that need to be modified to address a given issue. Input: • Issue Description: {problem statement} • Issue Images: {issue images} ting

optional, only used in the multimodal set-

• Repository: accessible through tool calls Task Instruction: Find the files and functions that need to be modified. Available Tools: You have the following tools available. At each turn, call exactly one tool by outputting a JSON object. 1. find files: list files matching a glob pattern. Example: {"tool": "find files", "pattern": "**/*.py"} 2. grep: search for a regex pattern in the repository. Example: {"tool": "grep", "pattern": "some function", "include": 3. read file: read a file, optionally with a line range. Example: {"tool": "read file", "path": "src/foo.py", "start line": "end line": 100}

"*.py"}

4. submit: submit the final localization result. You must call this tool when done. Example: {"tool": "submit", "files": ["path/to/file.py"], "functions": ["path/to/file.py:ClassName.method"]} Guidelines: • Start by understanding the issue. • Search for relevant code using repository-level tools. • Explore broadly first, then narrow down to specific files and functions. • When confident, call submit to provide the final answer. • Return files and functions ordered by relevance, from most relevant to least relevant. Final Output Format: The final answer must be submitted through the submit tool: {"tool": "submit", "files": ["..."], "functions": ["..."]} Fallback Instruction: If the maximum number of tool calls is reached, submit the best current guess immediately using the following format: {"tool": "submit", "files": [...], "functions": [...]}

Figure 11: LocAgent prompt

Openhands prompt Repository-Level Issue Localization Task You are a localization agent. Your only goal is to identify which source files and functions in repo/ are most likely affected by the issue below. You must not edit, patch, commit, install dependencies, or run tests. You only inspect the repository in a read-only manner and answer in JSON. Input: • Issue Title: {title} • Issue Body: {body}

• Vision Block: {vision block} vided

optional, only used when images are pro-

• Repository: repo/ Procedure: Follow the steps exactly: 1. Run ls repo or a similar command to skim the top-level repository layout. 2. Use grep -rn or a similar command to find symbols, error messages, or strings from the issue inside repo/. 3. Read at most 2–3 files that look most relevant. 4. Stop after at most 8 total actions. Even if uncertain, you must answer. 5. Call the finish tool with the required JSON object as the value of message. 6. Place the same JSON object as the final assistant message text. Do not include prose, markdown fences, or explanations. Output JSON Format: { "file level": [ "relative/path/to/most likely file.ext", "relative/path/to/next file.ext" ], "function level": [ "relative/path/to/file.ext:bare function name", "relative/path/to/file.ext:ClassName.method name" ] }

Format Rules: • file level: return 1–10 file paths, ordered by descending likelihood. • function level: return 1–10 function identifiers, ordered by descending likelihood. • function level may be empty only if the issue genuinely has no function-level target. • File paths must be relative to the repository root. • Use POSIX / separators. • Do not use a leading ./, trailing /, or repo/ prefix. • Each function identifier must use exactly one of the following formats: – relative/path/to/file.ext:bare function name – relative/path/to/file.ext:ClassName.method name • Do not include line numbers, argument lists, type hints, parentheses, decorators, or explanations. • Every function path should also appear in file level. • Do not list functions added by the fix. If a brand-new function is needed, point to the nearest existing anchor. • If unsure whether the scope is a class or module, prefer the bare function form. • Emit valid JSON: use double-quoted keys and strings, with no trailing commas and no comments. Reminders: • Prefer source files over tests, docs, generated files, or config files unless the issue clearly points to them. • Always emit the JSON. Empty answers score 0; imperfect answers can still receive partial credit. • Do not use terminal to run interactive commands, npm install, pytest, pip install, or similar commands. • Do not call task to spawn sub-agents. • Do not use browser use. • Use only terminal and file editor. Final Instruction: When confident, emit the final JSON object as the last message and stop.

Figure 12: Openhands prompt

Mini-SWE-Agent prompt Repository-Level Issue Localization Task You are given a repository snapshot and an issue report. Your task is not to fix the issue. Your task is only to localize the source files and functions that most likely need to be modified. Allowed Behavior: • You may inspect the repository. • You may use shell commands to search and read code. • You may reason about the issue, stack traces, UI screenshots, logs, rendered outputs, and code structure. • You must not edit files. • You must not create patches. • You must not commit changes. • You must not run long package installation commands. • You must prefer source files over tests, docs, generated files, or config files unless the issue clearly points to them. Repository: The repository root is: {repo dir} You are already working inside the repository root. Issue: • Title: {title} • Body: {body}

• Comments Block: {comments block} • Visual Block: {visual block}

Final Submission: When you have enough evidence, write the final JSON answer to the file path stored in $MSWEA PRED FILE. This is a per-instance path pre-allocated by the runner. Do not use /tmp/something.json, because parallel agents share the host’s /tmp directory. Then issue exactly this bash action, and nothing else, to complete the task: echo COMPLETE TASK AND SUBMIT FINAL OUTPUT && cat "$MSWEA PRED FILE"

Output JSON Schema: { "file level": [ "path/to/file1.ext", "path/to/file2.ext" ], "function level": [ "path/to/file1.ext:function name", "path/to/file2.ext:Class.method" ] }

Format Rules: • file level: return 1–10 repository-relative file paths, ordered from most likely to least likely. • function level: return 0–20 entries of the form file:function or file:Class.method. • Every file referenced in function level must also appear in file level. • Use POSIX / separators. • Do not prepend ./ or repo/. • Do not include line numbers, parentheses, argument lists, decorators, or comments in the function identifiers. • Prefer source files over tests, docs, generated files, or config files unless the issue clearly points to them.

Figure 13: Mini-SWE-Agent Prompt

Record · ID 373422 · SHA-256 b639f109ccf17776
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.