SIExBench: A Benchmark for Automated Repair of Sensitive Information Exposure

arXiv:2611.10776v1 [cs.SE] 7 Nov 2026

SIExBench: A Benchmark for Automated Repair of Sensitive Information Exposure MINGMING ZHANG, Beihang University, China XU WANG, Beihang University, China JIAN ZHANG∗ , Beihang University, China DIHAO FAN, Beihang University, China XIANGXIN MENG, Beihang University, China CHUNMING HU, Beihang University, China Modern software systems routinely process credentials, personal records, and other sensitive information. A sensitive information exposure (SIEx) defect makes such information accessible to an unauthorized actor and can cause serious and often irreversible disclosure. However, prior research has focused largely on detecting such exposures rather than repairing the underlying defects, and existing repository-level issue-resolution benchmarks do not specifically target this defect class. We present SIExBench, to our knowledge the first benchmark dedicated to automated repair of SIEx defects. SIExBench contains 161 execution-verified instances from 64 widely used Python and Java repositories, each with an issue report, buggy repository snapshot, developer patch, executable exposure and regression tests, and defect characterization. We evaluate six repair systems with three LLMs, yielding 18 configurations, and systematically analyze their overall effectiveness, performance across defect characteristics, and failure mechanisms. The strongest configuration resolves only 39.1% of the instances, while 40.4% remain unresolved by all 18 configurations. More importantly, 62.1% of unsuccessful attempts produce a valid patch that still leaves sensitive information exposed, making incomplete exposure removal a central challenge. Systems often generate plausible patches and modify relevant code without fully eliminating the exposure, indicating that effective repair must ensure that all relevant paths to sensitive-information exposure are closed. Overall, SIExBench provides a foundation for systematically evaluating and advancing automated repair of sensitive information exposure defects. CCS Concepts: • Software and its engineering → Software testing and debugging; Maintaining software; • Security and privacy → Software security engineering. Additional Key Words and Phrases: sensitive information exposure, automated program repair, repository-level issue resolution, benchmark, large language models ACM Reference Format: Mingming Zhang, Xu Wang, Jian Zhang, Dihao Fan, Xiangxin Meng, and Chunming Hu. 2018. SIExBench: A Benchmark for Automated Repair of Sensitive Information Exposure. 1, 1 (November 2018), 23 pages. https://doi.org/XXXXXXX.XXXXXXX ∗ Corresponding author.

Authors’ Contact Information: Mingming Zhang, mmzhang@buaa.edu.cn, Beihang University, Beijing, China; Xu Wang, xuwang@buaa.edu.cn, Beihang University, Beijing, China; Jian Zhang, zhangj_cs@buaa.edu.cn, Beihang University, Beijing, China; Dihao Fan, fandihao@buaa.edu.cn, Beihang University, Beijing, China; Xiangxin Meng, mengxx@buaa.edu.cn, Beihang University, Beijing, China; Chunming Hu, hucm@buaa.edu.cn, Beihang University, Beijing, China. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. © 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM XXXX-XXXX/2018/11-ART https://doi.org/XXXXXXX.XXXXXXX , Vol. 1, No. 1, Article . Publication date: November 2018.

2

1

Zhang et al.

Introduction

Modern software systems routinely process credentials, personal records, and other sensitive information. When a software defect makes such information accessible to an actor who is not explicitly authorized to access it, it creates a sensitive information exposure defect (CWE-200) [36], which we refer to as a SIEx defect. Such exposure is particularly consequential because disclosure is generally irreversible: once sensitive information is logged, returned to an unintended client, or stored in an accessible location, a later patch cannot retract what has already been observed. Exposed credentials and personal records may also enable subsequent attacks, while software vulnerabilities impose substantial economic costs [3, 32]. Figure 2 shows two real-world examples from our benchmark. In Redisson, a connection password is exposed through exception messages and logs, while in Home Assistant, a device location is exposed to another user’s UI. Together, these examples show that the same security consequence can arise from different root causes. Redisson directly emits sensitive information, whereas Home Assistant exposes it through incorrect program logic. Repairing SIEx defects therefore requires not only identifying sensitive information, but also reasoning about the underlying cause and how the information becomes observable. Prior work on information exposure has largely focused on detection and diagnosis rather than repair. Static and dynamic taint analyses track sensitive information through programs [4, 14], while other techniques target specific exposure channels, including network traffic, application logs, and server-side disclosure [8, 24, 45]. More recently, LLM-based approaches such as Argus [55] and SIExVulTS [23] directly detect CWE-200 sensitive-information leakage. In contrast, automated program repair has mainly targeted general software bugs [35, 66] and broader security vulnerabilities [29], leaving SIEx defects largely unexplored. Existing issue resolution benchmarks do not fill this gap. SWE-bench and its successors [13, 19, 21, 26, 75, 81] neither specifically curate SIEx defects nor provide dedicated repair evaluation. Moreover, generic issue-resolution benchmarks do not identify the sensitive information, exposure channel, and unauthorized actor that characterize SIEx defects, limiting systematic analysis of repair behavior and failure causes.The closest existing benchmark, SecLogging [73], covers security issues in logging code, including sensitive-information exposure, but is limited to logging-related defects. To address this gap, we present SIExBench, to our knowledge the first benchmark dedicated to automated repair of information exposure defects. SIExBench contains 161 execution-verified instances from 64 widely used Python and Java repositories. Each instance includes an issue report, buggy repository snapshot, developer patch, executable exposure tests that verify that sensitive information is no longer observable, and regression tests that preserve existing behavior. We further characterize each defect along six dimensions, enabling fine-grained analysis of repair effectiveness. We evaluate six repair systems with three LLMs, yielding 18 configurations, and analyze effectiveness, performance across defect characteristics, and failure mechanisms. Current systems leave substantial room for improvement: the strongest configuration resolves only 39.1% of SIExBench, while 40.4% of instances are unresolved by all 18 configurations. More importantly, failure is dominated not by the absence of a patch, but by incomplete exposure removal: 62.1% of unsuccessful attempts produce a valid patch that leaves sensitive information exposed. This reveals a challenge of SIEx repair. Systems often generate plausible patches and modify relevant code without fully eliminating the exposure. Effective repair therefore requires reasoning about whether all relevant paths through which sensitive information can become observable have been closed. In summary, we make the following contributions. • Problem Formulation. We formulate automated repair of sensitive information exposure as a distinct repair task, requiring reasoning about how sensitive information becomes observable to unauthorized actors across diverse root causes and exposure mechanisms. , Vol. 1, No. 1, Article . Publication date: November 2018.

SIExBench

3 CWE-201: Sent-data exposure

CWE-598: Query-string exposure

CWE-213: Conflicting policies

CWE-204: Response differences

CWE-206: Internal exposure

CWE-205: Behavior differences

CWE-200 Information exposure

CWE-207: Product-specific behavior

CWE-203: Observable differences

CWE-208: Timing differences

CWE-1254: Comparison granularity

CWE-215: Debug exposure

CWE-1300: Physical side channels

CWE-1255: Power side-channel

CWE-359: Personal-data exposure

CWE-1303: Shared CPU resources

CWE-1273: Credential sharing

CWE-210: Self-error exposure

CWE-535: Shell exposure

CWE-209: Sensitive error messages

CWE-211: External exposure

CWE-536: Servlet exposure

CWE-1295: Debug-message exposure

CWE-550: Server exposure

CWE-537: Java exposure

CWE-497: System-info exposure CWE-1431: Crypto-value exposure CWE-538: Accessible-file exposure

CWE-214: Visible-args exposure CWE-548: Directory listings CWE-532: Log exposure

CWE-531: Test exposure

CWE-540: Source exposure

CWE-541: Include exposure

CWE-651: WSDL exposure

CWE-615: Code-comment exposure

Fig. 1. CWE-200 and its descendants; names are abbreviated.

• Benchmark. We introduce SIExBench, to our knowledge the first benchmark dedicated to automated SIEx repair. It comprises 161 execution-verified instances from 64 Python and Java repositories and provides exposure-oriented test oracles, regression tests, and fine-grained defect characterization. • Extensive Evaluation. We evaluate six repair systems with three LLMs, yielding 18 configurations, and systematically analyze overall repair effectiveness, performance across defect characteristics, and failure mechanisms. Our results show that incomplete exposure removal dominates unsuccessful repairs and reveal that effective SIEx repair requires reasoning beyond patch generation to fully eliminate exposure paths. 2 2.1

Background and Related Work Sensitive Information Exposure Defects

A sensitive information exposure defect (SIEx defect) is a software defect that exposes sensitive information to an actor not explicitly authorized to access it, corresponding to CWE200 [36]. Figure 1 shows CWE-200 and its descendants. Our scope also includes defects whose root cause lies outside this hierarchy but whose consequence is a sensitive information exposure. Specifically, we define SIEx defects as defects whose resultant weakness falls within the CWE-200 hierarchy, without imposing any restriction on their primary weakness.1 Such defects are particularly consequential because disclosure is generally irreversible: a later patch Figure 2. Two SIExBench instances: issue excerpt, decan prevent further exposure but cannot retract veloper patch, and exposure path, with the fix in green. information already observed. Exposed sensitive data may also enable subsequent attacks [36]. Given the substantial economic impact of software vulnerabilities and cybercrime [3, 32], repairing these defects is an important software security problem [29]. (a) Redisson #7031

(b) Home Assistant #24674

Issue report

Issue report

What is the Actual behavior? The Redis password (and username if set) is included in plaintext in exception messages and log output [...] What is the Expected behavior? [...] the password should be masked/redacted [...]

Description of problem: When any of the 3 devices (unique device_trackers) changes location, the other 2 devices get set to be in the same location, when they aren’t actually [...] - 3 different user accounts - 3 iPhones

Developer patch

Developer patch

public String toString() { if (username != null && password != null) { return getScheme() + "://" + username + ":" + password + "@" + trimIpv6Brackets(host) + ":" + port; return getScheme() + "://***:***@" + trimIpv6Brackets(host) + ":" + port; } else if (password != null) { return getScheme() + "://" + password + "@" + trimIpv6Brackets(host) + ":" + port; + return getScheme() + "://***@" + + trimIpv6Brackets(host) + ":" + port; } ... } + +

Exposure path and fix password

toString()

+ +

def _async_receive_data(self, device, location, location_name): if device != self._name: return self._location_name = location_name self._location = location self.async_write_ha_state()

Exposure path and fix

mask

logs

log reader

location

receive()

verify

state (UI)

other user

1We follow the CWE Glossary definitions of Primary Weakness and Resultant Weakness.

, Vol. 1, No. 1, Article . Publication date: November 2018.

4

Zhang et al.

Figure 2 shows two representative instances. In Redisson (issue #7031), a connection password reaches exception messages and logs through RedisURI.toString(), and the patch masks it before emission. In Home Assistant (issue #24674), a device location is incorrectly written to another device’s state and exposed through the UI; the patch rejects mismatched device updates. The former is a direct exposure, where sensitive data is emitted directly, whereas the latter is a resultant exposure, where exposure follows from an underlying logic error. 2.2

Sensitive Information Exposure Detection

Research on SIEx defects has focused primarily on detection rather than repair. Static taint analyses such as FlowDroid [4], Amandroid [60], IccTA [28], and EdgeMiner [7] track sensitive data across program and component boundaries, although their accuracy depends on configuration and target [77]. Dynamic approaches instead observe concrete executions, including TaintDroid [14], TaintART [48], and MutaFlow [33]. Other work targets specific exposure settings, including privacy leaks in applications [18, 57], network traffic [10, 25, 45, 47], and access-control or policy enforcement [5, 15, 54]. More recent studies address application logs [8], server-side oversharing [24], and microservice backends [30], while LLM-based detectors such as Argus [55] and SIExVulTS [23] directly target sensitive-information leakage and CWE-200 exposures. Overall, prior work identifies or diagnoses information exposure rather than repairing such defects. APR research has instead focused on general software bugs [35, 66] and security vulnerabilities [29], leaving SIEx defects largely unexplored as a repair target. No dedicated benchmark exists for evaluating their automated repair; SIExBench fills this gap. 2.3 LLM-Based Repository-Level Issue Resolution Large language models (LLMs) have substantially advanced automated program repair (APR) [2, 6, 27, 40, 42, 59, 64, 67, 71, 76, 78]. SWE-bench [21] further catalyzed research on resolving real-world repository issues from natural-language descriptions [9, 16, 17, 31, 46, 49, 58, 62, 63, 68, 79]. Agent-based systems let LLMs interact with repositories through tools. SWE-agent [68], OpenHands [58], and mini-swe-agent [1] support repository inspection, editing, and test execution with varying levels of autonomy and complexity. In contrast, workflow-based systems use predefined stages; Agentless [62], for example, separates fault localization, patch generation, and patch selection. Developer-facing coding agents such as OpenCode [53] and Codex [37] bring similar capabilities to practical development. We evaluate representative systems from both paradigms on SIExBench to assess their effectiveness in repairing SIEx defects. 2.4

Benchmarks for Repository-Level Issue Resolution

Repository-level issue resolution is supported by a growing family of benchmarks. SWE-bench [21] established the setting with 2,294 issues from 12 Python repositories, pairing each issue with its fixing pull request and regression tests. Later benchmarks broaden this setting to longer-horizon tasks [13, 20], multiple languages [70, 75], and multimodal issues [19, 69]. Other benchmarks target specific domains or defect classes, including patch backporting [81], application domains [80], language- or API-specific defects [22, 34, 56], and security vulnerabilities [26, 61, 82]. The closest related benchmark, SecLogging [73], evaluates LLM-based detection and repair of 101 manually reviewed security issues in logging code, including cases of sensitive-information exposure. However, it is restricted to logging-related issues, whereas SIEx defects may arise from diverse root causes and exposure channels. Existing benchmarks therefore do not specifically curate SIEx defects or provide a dedicated evaluation setting for their repair. SIExBench fills this gap as the first benchmark dedicated to automated repair of SIEx defects. , Vol. 1, No. 1, Article . Publication date: November 2018.

SIExBench

5

1 Repository Selection

2 Issue–PR Pair Collection and Filtering Collecting

Top 500

Filtering

3 Screening and Review Screening

Review

GitHub API 500 Py repos 500 Java repos

157,730 issue–PR pairs

4 Task Instance Construction and Verification Construcing

210 confirmed pairs

28,574 issue–PR pairs

210 confirmed pairs

5 Defect Characterization

Verifying

210 task instances

940 candidate pairs

Characterizing

161 verified instances

161 verified instances + defect characterizations

Fig. 3. Overview of SIExBench construction.

3

SIExBench Construction

SIExBench is constructed in six stages that progressively refine issue–PR pairs into verified benchmark instances. We ❶ select repositories, ❷ collect and filter issue–PR pairs, ❸ pre-screen and manually review candidates, ❹ construct task instances and validate them through execution, and ❺ characterize the confirmed defects along six dimensions. Figure 3 summarizes the pipeline, and Section 4.1.1 reports the number of candidates retained at each stage. 3.1

Problem Definition

A SIEx repair task requires generating a patch that eliminates an information exposure in a repository. The input consists of the repository at the commit immediately preceding the fix, referred to as the buggy commit, and the issue report describing the defect; the expected output is exactly one patch for that snapshot. This input–output formulation follows the repository-level issue-resolution setting established by SWE-bench [21]. A task is resolved if the patch makes the exposure-revealing tests pass while preserving all previously passing tests. 3.2

Repository Selection

We sample instances from widely used open-source projects, using GitHub stars as a proxy for popularity. We focus on Python and Java, two widely used languages well represented in repositorylevel issue-resolution benchmarks [21, 70, 75]. For each language, we consider the 500 most-starred repositories and exclude private, archived, disabled, unlicensed, forked, or template repositories, as well as those that do not use GitHub Issues or pull requests. The resulting pool contains popular and legally reusable projects with established issue–PR workflows. 3.3

Issue–PR Pair Collection and Filtering

Following prior issue-resolution benchmarks [41, 70], we require a one-to-one issue–PR mapping: each retained issue is resolved by exactly one merged PR, and each retained PR resolves exactly one issue. This establishes a clear correspondence between the issue report as the task description and the PR as its reference fix, avoiding ambiguous task boundaries when a PR addresses multiple issues or an issue is resolved across multiple PRs. We further verify each issue–PR link using recorded cross-reference events to avoid incidental mentions. Moreover, following prior APR work [62, 64, 72], we define the scope of our study as source-code changes and exclude fixes requiring changes to configuration, documentation, or infrastructure. Following SWE-bench [21], we further require the fixing PR to include test changes, which provide candidate tests for constructing an executable evaluation oracle. Accordingly, we retain pairs whose changes are limited to production and test , Vol. 1, No. 1, Article . Publication date: November 2018.

6

Zhang et al.

source code and modify at least one test file and one non-test source file. These criteria define our benchmark as executable, source-code-level repair and exclude issues without accompanying tests, fixes spanning multiple PRs, and fixes requiring non-code changes. 3.4

Screening and Review

Rule-based filtering cannot determine whether a candidate describes a SIEx defect. We therefore use two-stage screening: recall-oriented LLM pre-screening followed by manual review. Two authors with over three years of research experience in software engineering or security independently review the retained candidates, and every instance in SIExBench is manually confirmed. LLM-Based Pre-screening. We assess each issue–PR pair with DeepSeek-V4-Pro [65], providing the complete CWE-200 taxonomy and asking whether the issue describes a SIEx defect. Each pair is assessed three times and retained if any assessment identifies an exposure. This recall-oriented policy is intended to reduce false negatives, while false positives can be removed during manual review. The LLM therefore serves only as a candidate filter and makes no final inclusion decisions. Human Review. Two authors independently review every retained candidate and determine whether it constitutes a SIEx defect based on three required elements: the sensitive information exposed, the exposure channel, and the unauthorized actor. A candidate is confirmed only if all three can be identified, with disagreements resolved through discussion. 3.5

Task Instance Construction and Verification

A confirmed defect becomes a benchmark instance only after being packaged into an executable environment and validated through execution. Task Instance Construction. For each confirmed defect, we separate the merged pull request into the developer fix and test changes, and combine them with the issue report. Each instance is anchored at the buggy commit, with the issue report serving as the problem statement. We package the repository and its dependencies into a Docker image and disable network access during evaluation to prevent retrieval of external information. Figure 5 shows a complete instance. Execution-Based Verification. We execute the tests on both the buggy and developer-fixed versions. Following SWE-bench [21], tests that fail before and pass after the fix are classified as FAIL_TO_PASS, while those that pass in both versions are PASS_TO_PASS. During verification, we find that some developer-written FAIL_TO_PASS tests encode implementation details of the reference fix rather than the intended security outcome. For example, Figure 5 requires credentials to be replaced by exactly three asterisks, which would reject an alternative but secure masking strategy. We therefore manually inspect all FAIL_TO_PASS tests and rewrite such implementation-specific assertions to test the intended exposure outcome instead, e.g., that the username and password are no longer exposed through the tested behavior. For tests requiring normalization, two authors independently review each rewritten test against the issue report, developer patch, and original test to verify that it removes only implementation-specific constraints while preserving the intended exposure property. When disagreements arise, the rewritten test is revised as necessary and rereviewed until both authors reach consensus. These normalized tests form the final FAIL_TO_PASS set. We retain only instances with at least one FAIL_TO_PASS test. We exclude instances whose tests (i) cannot run on the buggy commit because they depend on symbols introduced by the fix, (ii) behave identically before and after the fix, or (iii) require unavailable external services. Patch Evaluation Protocol. For each candidate patch, we reset the repository to the buggy commit, apply the test changes and candidate patch, and execute the test suite. An instance is resolved only if all FAIL_TO_PASS and PASS_TO_PASS tests pass; no partial credit is awarded. As construction sanity checks, the buggy commit must fail the final FAIL_TO_PASS tests, whereas the developer fix must pass all FAIL_TO_PASS and PASS_TO_PASS tests. , Vol. 1, No. 1, Article . Publication date: November 2018.

SIExBench

7

Table 1. The six characterization dimensions and their label vocabularies. Dimension

Labels

Primary CWE

The CWE identifier of the initial weakness that constitutes the root cause of the defect and is not restricted to the CWE-200 hierarchy.

Resultant CWE

The CWE identifier of the downstream exposure weakness, which always lies within the CWE-200 hierarchy and equals the primary CWE when the exposure itself is the root cause.

Sensitive information

credential or secret HTTP response

Exposure channel

Unauthorized actor Repair pattern

private record

user interface

business record

log

SQL result

error message

UI viewer

other authenticated user

chat message

HTTP request

file content

deployment config

shell output

log reader

adjacent client

redirect target

SQL user

mask or redact

add authorization check

remove field

defer resolution

Confirmed Final

73 47

137

531 409 940

102

104 106 Pairs / instances (log)

reject hardlink

strip on redirect

PASS_TO_PASS

remove path echo

(c) Patch details 3 / 5

Changed hunks

1 / 3

FAIL_TO_PASS

17 / 36

2,875 / 4,430

Test patch Python Java Total

161

infrastructure reader

telemetry collector

557 / 1,599

15,655 12,919 28,574

210 114

HTTP request header

(b) Task, patch, and test sizes

94,851 62,879 157,730

Rule-filtered

system information

network message

cross-tenant actor

workspace-restricted user

(a) Construction funnel

directory listing

tool result

telemetry export

chat recipient

Raw pairs

LLM candidates

file path

2,013 / 3,780

Patch Task input

IQR Median Mean Min–Max

100

Changed lines 1 / 2

1,485 / 2,237

102 104 Size (log scale)

Changed files

100

IQR Median Mean Min–Max

101 102 Count (log scale)

Fig. 4. SIExBench overview: (a) construction funnel, (b) task, patch, and test statistics, and (c) patch details.

3.6

Defect Characterization

To support fine-grained analysis, we characterize each instance along six dimensions (Table 1). Two follow CWE weakness chains. The primary CWE denotes the root-cause weakness, while the resultant CWE denotes a downstream weakness caused by it. They may differ when a SIEx defect results from another weakness; for example, a missing authorization check may be primary while the resulting exposure is resultant. When the exposure itself is the root cause, we assign the same CWE to both. Following the CWE-200 definition of information exposure, we further characterize the sensitive information disclosed, the exposure channel through which it becomes accessible, and the unauthorized actor that gains access. The final dimension, repair pattern, captures how the developer patch removes the exposure. Two authors derive the label vocabularies through open coding. They first identify candidate labels from a sample of confirmed defects, guided by the CWE taxonomy for the two CWE dimensions and by observed exposure and repair mechanisms for the others. The labels are then refined over the remaining defects, with ambiguous categories merged and disagreements resolved through discussion. Each defect receives exactly one, most-specific applicable label per dimension. Table 1 lists the resulting categories. 4 4.1

SIExBench Statistics Benchmark Overview

4.1.1 Construction Funnel and Composition. SIExBench comprises 161 task instances collected from 64 widely used open-source repositories, spanning both Python and Java projects. Specifically, the , Vol. 1, No. 1, Article . Publication date: November 2018.

8

Zhang et al.

benchmark includes 114 Python instances and 47 Java instances, enabling evaluation of automated SIEx repair across both language ecosystems. Figure 4(a) summarizes the construction funnel. From 157,730 issue–PR pairs, rule-based filtering retained 28,574, LLM prescreening 940, and human review confirmed 210 SIEx defects. Rule-based filtering keeps an issue–PR pair only when every changed file is source code, at least one changed file is a test, and at least one changed file is a non-test source file. Of the 129,156 discarded pairs, 87,777 (68.0%) modify at least one non-source file, 36,566 (28.3%) modify source code but no test, and 4,813 (3.7%) modify only test files. Agreement among the three LLM assessments correlated with human confirmation: 177/313 (57%) candidates flagged three times were confirmed, compared with 22/186 (12%) flagged twice and 11/441 (2%) flagged once. To assess false negatives from LLM pre-screening, we randomly sampled 600 pairs rejected by all three assessments. This sample size provides at least 95% probability of detecting one or more missed defects if such defects constitute 0.5% or more of the rejected pairs. Manual review using the same inclusion criteria identified one missed SIEx defect, corresponding to 0.17% of the audited rejected candidates (95% Wilson CI: 0.03%–0.94%). Execution verification Figure 5. An example SIExBench retained 161 instances, roughly one per thousand initial pairs; instance. the remaining 49 were excluded because tests depended on symbols introduced by the fix (24), behaved identically before and after the fix (23), or required unavailable external services (2). All subsequent analyses use these 161 instances. Figure 5 illustrates one retained instance, where a startup log exposes credentials from DATABASE_URL; the developer patch masks them, and the exposure test checks that the username and password are absent. The figure also illustrates test normalization: in 57 instances, developer-written tests encode the exact reference-fix output, as in Figure 5, where the test requires the literal redaction ***:*** and would reject alternative secure repairs, e.g., masking the same credentials with four asterisks instead of three. We therefore rewrite such assertions to check the exposure outcome rather than the specific implementation. vectorize-io/hindsight #1262

Python

Metadata

Repository

vectorize-io/hindsight (Python)

Issue / fix PR

#1262 / #1316

Base commit

300a8c1e

FAIL_TO_PASS / PASS_TO_PASS

1 / 1,098

Defect profile

Primary / resultant CWE

CWE-532 / CWE-532

Sensitive info

credential / secret

Exposure channel

log output

Unauthorized actor

log reader

Repair pattern

mask / redact

Developer patch (config.py)

- logger.info(f"Database: {self.database_url}")

+ logger.info(f"Database: {mask_network_location(self.database_url)}")

Test patch, as in the PR

+ assert "hindsight_user" not in log_output + assert "plain-password" not in log_output + assert "migration_user" not in log_output

+ assert "migration-password" not in log_output

+ assert "postgresql://***:***@db:5432/hindsight_db" in log_output

+ assert "postgresql://***:***@db-admin:5432/hindsight_db" in log_output

Test patch, exposure test (manually fixed)

+ assert "hindsight_user" not in log_output + assert "plain-password" not in log_output + assert "migration_user" not in log_output

+ assert "migration-password" not in log_output + assert "db:5432/hindsight_db" in log_output

+ assert "db-admin:5432/hindsight_db" in log_output

4.1.2 Task, Patch, and Test Statistics. Figure 4(b) plain text summarizes task, patch, and test-suite sizes. Task demodel scriptions, developer patches, and test patches have dirkey prefect median lengths of 1,485, 2,013, and 2,875 characters, list respectively. Each instance contains a median of one read file FAIL_TO_PASS test but 557 PASS_TO_PASS tests, refile read endpoint secret flecting the focused role of exposure tests and the access broader role of regression tests. The distributions auth header maskedinformation work are right-skewed; for example, PASS_TO_PASS tests average 1,599 per instance. Figure 4(c) shows that Figure 6. Issue-title keyphrases. a typical developer patch changes one file, 17 lines, and three hunks, with two thirds of instances being single-file fixes, while larger multi-file changes provide variation in repair complexity. Finally, Figure 6 highlights frequent issue-title keyphrases, including password, privacy, logs, API, user, and access, reflecting recurring themes in SIExBench. bedrocklitellm content filter

加密

database

litellm

path traversal

log user

password

, Vol. 1, No. 1, Article . Publication date: November 2018.

proxy

show extra

redirect

data

client

guardrail handle silent

correctly response

constrain

hide

directory

virtual keys

environment variables

api key

code

mcp

handle first matching

security visible

filter guardrail

content filter guardrail

creating

litellm content

payload transformation

output

report task

logs apiusers connection

url

SIExBench

9

Table 2. Repositories contributing to SIExBench, sorted by instance count. The top 20 are shown individually and the remaining 44 are aggregated. Repository

Lang

License

keycloak/keycloak BerriAI/litellm home-assistant/core odysseus-dev/odysseus PrefectHQ/prefect getsentry/sentry HKUDS/nanobot agno-agi/agno apache/airflow PrefectHQ/fastmcp StarRocks/starrocks halo-dev/halo kestra-io/kestra openai/openai-agents-python paperless-ngx/paperless-ngx zulip/zulip PostHog/posthog apache/shardingsphere apache/superset deepset-ai/haystack

Java Py Py Py Py Py Py Py Py Py Java Java Java Py Py Py Py Java Py Py

Apache-2.0 NOASSERTION Apache-2.0 AGPL-3.0 Apache-2.0 NOASSERTION MIT Apache-2.0 Apache-2.0 Apache-2.0 Apache-2.0 GPL-3.0 Apache-2.0 MIT GPL-3.0 Apache-2.0 NOASSERTION Apache-2.0 Apache-2.0 Apache-2.0

Remaining 44 repositories

–

–

Total

Stars

#Files

Size (MB)

#LOC

36k 55k 89k 62k 23k 44k 46k 41k 46k 27k 12k 39k 28k 28k 44k 26k 37k 21k 74k 26k

9,364 3,998 9,518 913 2,139 6,254 487 4,838 7,172 628 7,101 1,025 2,775 1,260 650 2,695 6,351 9,839 4,848 436

589 1243 838 42 224 896 51 294 661 55 732 86 121 33 190 622 5700 684 1046 211

1,320,860 1,041,068 1,617,741 331,091 487,610 1,041,806 164,127 846,146 1,214,695 144,123 1,537,497 119,300 264,901 302,045 84,029 504,163 1,203,896 715,560 925,660 99,044

–

132,562

9,725

214,853

24,043

#Inst Domain

Description

16 15 15 9 8 8 5 4 4 3 3 3 3 3 3 3 2 2 2 2

Security & Identity Identity and access management server for modern applications and services AI & Machine Learning Gateway and SDK for calling many LLM APIs Home Automation Open source home automation platform prioritizing local control AI & Machine Learning Self-hosted AI workspace for tools and workflows Data Infrastructure Workflow orchestration framework for building resilient data pipelines DevOps & Cloud Developer-first error tracking and performance monitoring platform AI & Machine Learning Lightweight open source AI agent for tools and chats AI & Machine Learning Framework for building and managing AI agent platforms Data Infrastructure Platform to author schedule and monitor data workflows AI & Machine Learning Python framework for building MCP servers and clients Data Infrastructure Distributed OLAP query engine for real-time analytics Web & API Open source CMS and website building platform Data Infrastructure Event driven orchestration and scheduling platform for workflows AI & Machine Learning Lightweight framework for building multi-agent workflows Document Management Document management system to scan index and archive documents Web & API Open source team chat server and web application Data Infrastructure Product analytics platform with session replay and feature flags Data Infrastructure Distributed SQL database middleware for sharding and data encryption Data Infrastructure Open-source business intelligence and data visualization web platform AI & Machine Learning Framework for building LLM and retrieval-augmented generation applications

21,057,583

48

Omitted for brevity

35,022,945

161

64 repositories

Instances

4.1.3 Temporal and Fix-Time Statistics. Figure 7 shows the distribution of the 161 instances by issue75 Total Python Java 63 creation year, with 2026∗ denoting a partial year 50 covering issues before August 2026. SIExBench contains recent instances, with the annual count rising 25 19 25 15 13 after 2022 and reaching 63 in partial-year 2026. Fix 6 6 4 4 3 3 0 latency, measured from issue creation to the merge 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 2026 of the corresponding fix, has a median of 4 days and Issue creation year an interquartile range of 1–18 days. The distribution is strongly right-skewed, with a mean of 54 days and Figure 7. Instances by issue-creation year. a maximum exceeding three years. Thus, SIExBench covers both rapidly resolved defects and substantially longer repair cycles. 4.2

*

Source Distribution

4.2.1 Languages, Repositories, and Popularity. The 161 instances come from 64 repositories, including 41 Python repositories with 114 instances and 23 Java repositories with 47 instances (Table 2, Figure 8). Repository popularity ranges from 7,200 to 89,500 GitHub stars, with a median of 27,600, while repository size ranges from 89 to 16,986 source files, with a median of 1,946. Instance counts are uneven: 40 repositories contribute one instance, whereas the six largest contributors, keycloak, litellm, home-assistant, odysseus, prefect, and sentry, account for 71 of the 161 instances. 4.2.2 Application Domains. We assign each repository a primary application domain based on its GitHub topics and description, manually review all assignments, and consolidate them into seven categories (Figure 8(d)). AI and machine learning is the largest domain, with 23 repositories and 58 instances, followed by data infrastructure with 16 repositories and 33 instances, and web and API frameworks with 10 repositories and 15 instances. Security and identity, DevOps and cloud, and developer tools each contribute four repositories, while three single-repository domains are grouped as Others. These domains cover systems that routinely handle sensitive information such as credentials, prompts, user records, and access tokens. Notably, AI and machine-learning repositories first contribute instances in 2023 and account for 42 of the 63 instances created in 2026, reflecting their growing presence among recent SIEx defects in SIExBench. These repositories also exhibit domain-specific exposure patterns. For example, one LLM gateway masks one provider’s key format but not another’s when logging deployments, exposing an AWS access key and secret in , Vol. 1, No. 1, Article . Publication date: November 2018.

10

Zhang et al.

(a) Star distribution 30

(c) Language split

15 10

101

58 (23 repos) 33 (16 repos)

Data infra 114

Count

20

(d) Application domain

Repositories AI & ML Instances

150

Repositories

25

Repositories

(b) Instances per repo

Python Java

100

Security

19 (4 repos)

Others

19 (3 repos) 15 (10 repos)

Web & API

50

47

41

5

12 (4 repos)

DevOps

23

5 (4 repos)

Dev tools

100

0 0

0

20 40 60 80

1 3 5 8

GitHub stars (k)

16

0

Python Java

Language

Instances per repository

20

40

60

Instances

Fig. 8. Repository statistics, including popularity, instance distribution, language coverage, and application domains. Others groups three single-repository domains. (a) Sensitive information credential or secret 45 (28%)

private record 22 (14%)

business record file content

13 (8%)

file path

7 (4%)

directory listing

4 (2%)

CWE-201 CWE-862 CWE-532 CWE-538 CWE-22 CWE-59 CWE-550 CWE-598 CWE-613 CWE-210 Others

(b) Unauthorized actor UI viewer other authenticated user log reader adjacent client cross-tenant actor infrastructure reader chat recipient workspace-restricted user redirect target SQL user Others

(c) Exposure channel

33 (20%)

70 (43%)

29 (18%) 29 (18%) 16 (10%) 12 (7%) 11 (7%) 9 (6%) 7 (4%) 7 (4%) 7 (4%) 1 (1%)

HTTP response user interface log tool result chat message deployment config HTTP request header SQL result error message HTTP request Others

41 (25%) 29 (18%) 26 (16%) 14 (9%) 12 (7%) 12 (7%) 8 (5%) 7 (4%) 6 (4%) 3 (2%) 3 (2%)

Instances

Instances

Instances

(d) Primary CWE

(e) Resultant CWE

(f) Repair pattern

51 (32%) 36 (22%) 26 (16%) 12 (7%) 9 (6%) 4 (2%) 4 (2%) 3 (2%) 2 (1%) 2 (1%) 12 (7%)

CWE-201 CWE-532 CWE-538 CWE-359 CWE-550 CWE-204 CWE-548 CWE-598 CWE-210 CWE-497 Others

91 (57%) 26 (16%) 19 (12%) 8 (5%)

47 (29%) 36 (22%)

remove field

4 (2%) 3 (2%) 2 (1%) 2 (1%) 2 (1%) 1 (1%) 3 (2%)

Instances

58 (36%)

mask or redact

add authorization check

Instances

defer resolution

7 (4%)

reject hardlink

6 (4%)

strip on redirect

4 (2%)

remove path echo

3 (2%)

Instances

Fig. 9. Distribution of the six characterization dimensions over 161 instances. The ten most frequent values are shown and the remainder grouped as Others.

plaintext. Such cases illustrate how modern AI software introduces heterogeneous credentials and providers that can create new opportunities for information exposure, underscoring the growing relevance of SIEx repair in this domain. 4.3

Defect Distribution

Each instance is characterized along six dimensions: primary and resultant CWE, sensitive information, exposure channel, unauthorized actor, and repair pattern. Figure 9 shows their distributions. The exposure dimensions are dominated by several recurring categories. Credentials and secrets are the most common sensitive information (70 instances, 43.5%), followed by private records (45, , Vol. 1, No. 1, Article . Publication date: November 2018.

SIExBench

11

28.0%), together accounting for 71.4% of SIExBench. The leading exposure channels are HTTP responses (41, 25.5%), user interfaces (29, 18.0%), and logs (26, 16.1%), totaling 59.6%. Unauthorized actors are similarly concentrated, with UI viewers, other authenticated users, and log readers each representing 18.0–20.5% of the benchmark. The CWE dimensions further show that the weaknesses causing an exposure often differ from the weaknesses through which the exposure ultimately manifests. By definition, all resultant CWEs in SIExBench fall within the CWE-200 hierarchy, and they are concentrated in a small number of categories, particularly CWE-201 (sensitive information in sent data; 56.5%), CWE-532 (sensitive information in log files; 16.1%), and CWE-538 (sensitive information in externally accessible files or directories; 11.8%). In contrast, primary CWEs are more diverse and are not restricted to this hierarchy. CWE-201 serves as the primary CWE in 31.7% of instances, while CWE-862 (missing authorization; 22.4%) and CWE-22 (path traversal; 5.6%) are notable primary root causes that lie outside the CWE-200 hierarchy. Overall, the primary and resultant CWEs are identical in 63.4% of instances, whereas 35.4% of instances originate from primary weaknesses outside the CWE-200 hierarchy. Consequently, restricting benchmark construction to defects whose root causes fall within CWE-200 would exclude more than one third of the observed SIEx defects. 5 5.1

Experimental Setup Research Questions

We investigate the following research questions: RQ1 How do current repair methods perform on SIExBench? (Overall Performance) RQ2 How does repair performance vary across defect characteristics? (Stratified Performance) RQ3 Which failure modes limit automated repair on SIExBench? (Failure Analysis) 5.2

Studied Systems

We evaluate six systems in two groups. The first includes issue-resolution methods commonly used in prior research, spanning procedural and agentic designs. MSWE-agent [52], Magentless [50], and MopenHands [51] are the Multi-SWE-bench [75] multilingual forks of SWE-agent [68], Agentless [62], and OpenHands [58], enabling the same methods to cover both Python and Java. We also include mini-swe-agent [1], a minimal bash-only agent widely used as a reference baseline [13, 20, 63]. The second group includes developer-facing coding agents: the open-source OpenCode [53] and commercial Codex [37]. All agentic systems receive a budget of 100 interaction steps, while Magentless uses a 10 × 10 patch space comprising 10 localization groups and 10 sampled patches per group. To prevent unintended access to external information or future repository history, we configure the Docker containers to block all network access except LLM API requests and remove all commits after the buggy commit. 5.3

Model Selection

We conduct evaluation experiments with two groups of base LLMs. Table 3. Base LLMs in the first (leakage-controlled) model group. Model

Open

Parameters

Release

Knowledge cutoff

DeepSeek-V3.2 Qwen3.5-Plus GPT-5.6 Luna

Yes Yes No

671B (37B active) 397B (17B active) Undisclosed

2025-12-01 2026-02-16 2026-07-09

Not published Not published 2026-02-162

, Vol. 1, No. 1, Article . Publication date: November 2018.

12

Zhang et al.

Table 4. Overall repair performance of the 18 configurations. Column-best values are bolded and darkestshaded, with lower cost preferred; steps are reported as min–max (mean).

Coding agents

Issue-resolution systems

Type

Method

Model

MSWE-agent [52]

DeepSeek-V3.2 Qwen3.5-Plus GPT-5.6 Luna

Magentless [50]

Steps

#R (%R) on leakage subsets

#R (%R) by language

Pass details per test suite

≤02-16 (105)

>02-16 (56)

Python (114)

Java (47)

#F2P (%F2P)

#F2P′ (%F2P′ )

#P2P (%P2P)

#R (%R)

Avg. $

$/Solved

15–100 (72) 7–100 (57) 8–100 (32)

27 (25.7%) 28 (26.7%) 30 (28.6%)

15 (26.8%) 18 (32.1%) 21 (37.5%)

22 (19.3%) 29 (25.4%) 30 (26.3%)

20 (42.6%) 17 (36.2%) 21 (44.7%)

50 (31.1%) 56 (34.8%) 60 (37.3%)

39 (24.2%) 46 (28.6%) 48 (29.8%)

87 (54.0%) 98 (60.9%) 107 (66.5%)

42 (26.1%) 46 (28.6%) 51 (31.7%)

$0.119 $0.401 $0.065

$0.456 $1.405 $0.204

DeepSeek-V3.2 Qwen3.5-Plus GPT-5.6 Luna

– – –

15 (14.3%) 10 (9.5%) 18 (17.1%)

10 (17.9%) 6 (10.7%) 11 (19.6%)

18 (15.8%) 8 (7.0%) 19 (16.7%)

7 (14.9%) 8 (17.0%) 10 (21.3%)

29 (18.0%) 20 (12.4%) 35 (21.7%)

24 (14.9%) 16 (9.9%) 30 (18.6%)

90 (55.9%) 52 (32.3%) 84 (52.2%)

25 (15.5%) 16 (9.9%) 29 (18.0%)

$0.104 $0.204 $0.059

$0.671 $2.052 $0.328

MopenHands [51]

DeepSeek-V3.2 Qwen3.5-Plus GPT-5.6 Luna

1–100 (77) 14–100 (94) 4–100 (28)

33 (31.4%) 40 (38.1%) 29 (27.6%)

21 (37.5%) 23 (41.1%) 20 (35.7%)

34 (29.8%) 44 (38.6%) 31 (27.2%)

20 (42.6%) 19 (40.4%) 18 (38.3%)

68 (42.2%) 71 (44.1%) 60 (37.3%)

55 (34.2%) 57 (35.4%) 52 (32.3%)

118 (73.3%) 115 (71.4%) 120 (74.5%)

54 (33.5%) 63 (39.1%) 49 (30.4%)

$0.141 $0.232 $0.023

$0.421 $0.594 $0.074

mini-swe-agent [1]

DeepSeek-V3.2 Qwen3.5-Plus GPT-5.6 Luna

28–100 (73) 27–100 (74) 4–26 (10)

39 (37.1%) 36 (34.3%) 36 (34.3%)

17 (30.4%) 18 (32.1%) 20 (35.7%)

38 (33.3%) 35 (30.7%) 34 (29.8%)

18 (38.3%) 19 (40.4%) 22 (46.8%)

65 (40.4%) 64 (39.8%) 65 (40.4%)

58 (36.0%) 53 (32.9%) 52 (32.3%)

108 (67.1%) 99 (61.5%) 131 (81.4%)

56 (34.8%) 54 (33.5%) 56 (34.8%)

$0.099 $0.114 $0.007

$0.284 $0.341 $0.019

OpenCode [53]

DeepSeek-V3.2 Qwen3.5-Plus GPT-5.6 Luna

28–100 (58) 11–100 (61) 10–100 (26)

42 (40.0%) 38 (36.2%) 28 (26.7%)

20 (35.7%) 17 (30.4%) 22 (39.3%)

39 (34.2%) 36 (31.6%) 34 (29.8%)

23 (48.9%) 19 (40.4%) 16 (34.0%)

77 (47.8%) 63 (39.1%) 58 (36.0%)

60 (37.3%) 49 (30.4%) 48 (29.8%)

121 (75.2%) 117 (72.7%) 100 (62.1%)

62 (38.5%) 55 (34.2%) 50 (31.1%)

$0.165 $0.160 $0.022

$0.429 $0.468 $0.071

Codex [37]

DeepSeek-V3.2 Qwen3.5-Plus GPT-5.6 Luna

8–100 (67) 6–100 (56) 1–70 (14)

35 (33.3%) 37 (35.2%) 37 (35.2%)

18 (32.1%) 21 (37.5%) 21 (37.5%)

36 (31.6%) 39 (34.2%) 38 (33.3%)

17 (36.2%) 19 (40.4%) 20 (42.6%)

67 (41.6%) 69 (42.9%) 66 (41.0%)

53 (32.9%) 53 (32.9%) 51 (31.7%)

106 (65.8%) 115 (71.4%) 117 (72.7%)

53 (32.9%) 58 (36.0%) 58 (36.0%)

$0.120 $0.122 $0.042

$0.366 $0.340 $0.118

As shown in Table 3, the first group includes DeepSeek-V3.2 [11], Qwen3.5-Plus [43], and GPT-5.6 Luna [38], crossed with all six studied systems, yielding 3 × 6 configuration pairs. We select these models to balance capability, cost, and release date, covering competitive open-weight models and a cost-effective proprietary model. To support leakage-aware analysis, we split the 161 SIExBench instances at 2026-02-16 (the latest knowledge cutoff among the three models): 105 have issue dates on or before 2026-02-16 and 56 after it. DeepSeek-V3.2 and Qwen3.5-Plus were released before the boundary, while GPT-5.6 Luna has a documented knowledge cutoff of 2026-02-16. Thus, none of the three models could have had access to the 56 post-boundary instances during training. The second group comprises more recent models released after the boundary: DeepSeek-V4.1Flash [12], GLM-5.3-Flash [74], Qwen3.8-Flash [44] and GPT-6 Luna [39]. Following prior benchmark studies [13, 20], we evaluate these models only with mini-swe-agent to isolate model effects under a fixed agent. As these models postdate the boundary, their results are reported separately from those of the first group. 5.4

Evaluation Metrics

Following prior work [19, 21, 58, 62, 68], we report the following metrics: • #Resolved (%Resolved): the number (and the corresponding percentage) of benchmark instances that are successfully resolved; the tables abbreviate the metric as #R (%R). • Avg. $: the average cost per instance in USD. 6

Evaluation

6.1 RQ1: Overall Performance We first evaluate six repair methods with three LLMs, yielding 18 configurations over all 161 instances. Table 4 reports repair success, test outcomes, API cost, and interaction steps for each configuration. Here, F2P denotes the normalized FAIL_TO_PASS tests for exposure removal, 𝐹 2𝑃 ′ the original developer-written tests before normalization, and P2P the PASS_TO_PASS regression tests; an instance is resolved only if both F2P and P2P pass. Overall repair effectiveness. Resolve rates range from 9.9% for Magentless with Qwen3.5Plus to 39.1% for MopenHands with the same LLM (63/161), with a mean of 30.3% across all 18 configurations. Thus, substantial room remains even for the strongest evaluated configuration. 2 https://developers.openai.com/api/docs/models/gpt-5.6-luna

, Vol. 1, No. 1, Article . Publication date: November 2018.

SIExBench

13

Table 5. Repair performance under mini-swe-agent. Models marked † are newly evaluated and postdate the leakage boundary; steps are min–max (mean). Model

DeepSeek-V4.1-Flash† GLM-5.3-Flash† Qwen3.8-Flash†

GPT-6 Luna† DeepSeek-V3.2

GPT-5.6 Luna

Qwen3.5-Plus

58 (36.0%) 31.6% / 46.8%

56 (34.8%) 29.8% / 46.8%

54 (33.5%) 30.7% / 40.4%

#Resolved (%Resolved) Python / Java

65 (40.4%) 37.7% / 46.8%

65 (40.4%) 36.0% / 51.1%

58 (36.0%) 34.2% / 40.4%

56 (34.8%) 33.3% / 38.3%

Avg. $ / $/Solved Steps (min–max, mean)

$0.030 / $0.073 6–100 (49)

$0.023 / $0.056 5–100 (29)

$0.030 / $0.082 $0.004 / $0.012 $0.099 / $0.284 $0.007 / $0.019 $0.114 / $0.341 6–100 (55) 6–85 (17) 28–100 (73) 4–26 (10) 27–100 (74)

P2P pass rates are consistently higher than exposure-test pass rates, ranging from 32.3% to 81.4% (65.0% on average), compared with 12.4%–47.8% (36.0%) for F2P and 9.9%–37.3% (29.1%) for F2P′ . This indicates that eliminating the exposure is more challenging than preserving previously passing behavior under our protocol. Test normalization also matters: at least one configuration passes the normalized exposure tests for 103 instances, compared with 89 under the original developer-written tests, reducing rejections caused by implementation-specific assertions. Method and model effects. Averaged over methods, the three LLMs achieve similar resolve rates of 30.2%–30.3%, whereas method-level averages range from 14.5% for Magentless to 35.0% for Codex. Thus, among the similarly capable models evaluated here, performance varies more across repair methods than across LLMs. Resource use also varies widely, with average cost ranging from $0.007 to $0.401 per instance and $0.019 to $2.052 per resolved instance. Java instances outperform Python instances in 17 of 18 configurations, averaging 37.0% and 27.5%, respectively. Repository composition, however, precludes attributing this gap to language alone. Resolve rates are similar before and after the leakage boundary (29.5% vs. 31.6%), providing no aggregate evidence of a pre-boundary advantage while not ruling out contamination of individual instances. Instance coverage. Figure 10 shows that 65 of 161 instances (40.4%) remain unresolved by all 18 configurations. Among the #configs that repair the instance 96 resolved by at least one configuration, 51 are resolved by ten or more. The union of all configurations therefore reaches 59.6% Figure 10. Distribution of incoverage, compared with 39.1% for the best individual configurastances by the number of configtion. This bimodal pattern suggests that many instances are either urations that resolve them. broadly solvable or consistently difficult. Complementarity across configurations. No single configOpenCode uration subsumes the repairs of the others. As shown in Figure 11, (DS) the four highest-performing configurations collectively resolve 85 0 instances, 22 more than the best configuration, while the remaining MopenHands Codex 0 (Qwen) (Qwen) 0 0 14 contribute 11 additional repairs, bringing total coverage to 96 in1 0 0 4 0 1 1 stances (59.6%). Thus, configurations exhibit complementary repair 0 0 0 0 capabilities, although 40.4% of SIExBench remains unresolved. 34 0 6 4 8 0 Model sensitivity under a fixed agent. To isolate LLM choice 1 5 0 from the repair workflow, we fix the method to mini-swe-agent 2 2 6 2 6 and compare additional models. Among the three primary LLMs, 11 2 Others Codex DeepSeek-V3.2 and GPT-5.6 Luna each resolve 56 instances (34.8%), (GPT) while Qwen3.5-Plus resolves 54 (33.5%). Their resolve rates are similar, but costs vary substantially, with GPT-5.6 Luna being the Figure 11. Repair overlap across cheapest ($0.007 per instance and $0.019 per resolved instance). configurations. We further evaluate four newer LLMs: DeepSeek-V4.1-Flash, 100

Python Java 65 instances: no configuration repairs them

60

instances

80

50

60

40 30

40

20

20

10 0

cumulative share (%)

70

0

0

3

6

9

12

15

18

, Vol. 1, No. 1, Article . Publication date: November 2018.

14

Zhang et al.

Table 6. Repair performance by defect characteristic across 18 configurations. Py/Java and ≤02-16/>02-16 denote language and issue-date splits; Others groups values with fewer than five instances. Characteristic

#Inst

MSWE-agent

Py Java ≤02-16 >02-16

Magentless

MopenHands

mini-swe-agent

OpenCode

Codex

min

DS

Qwen

GPT

DS

Qwen

GPT

DS

Qwen

GPT

DS

Qwen

GPT

DS

Qwen

GPT

DS

Qwen

GPT

max

≥1 Avg

Sensitive information type credential_secret private_record business_record file_content file_path Others

70 30.6% 37.8% 45 22.6% 38.5% 22 33.3% 35.7% 13 23.1% 0.0% 7 6.9% 38.9% 4 37.5% –

28.5% 29.5% 35.4% – 21.3% 31.5%

41.5% 23.6% 30.6% 21.4% 16.7% 55.6%

30.0% 26.7% 27.3% 7.7% 14.3% 25.0%

27.1% 31.1% 31.8% 15.4% 28.6% 50.0%

30.0% 28.9% 40.9% 23.1% 42.9% 50.0%

20.0% 13.3% 13.6% 7.7% 0.0% 25.0%

14.3% 11.1% 4.5% 0.0% 0.0% 0.0%

15.7% 15.6% 22.7% 23.1% 28.6% 25.0%

38.6% 33.3% 27.3% 15.4% 42.9% 25.0%

44.3% 28.9% 54.5% 30.8% 14.3% 50.0%

28.6% 33.3% 36.4% 23.1% 14.3% 50.0%

41.4% 31.1% 40.9% 23.1% 0.0% 25.0%

35.7% 31.1% 36.4% 30.8% 14.3% 50.0%

35.7% 33.3% 40.9% 23.1% 28.6% 50.0%

41.4% 33.3% 54.5% 23.1% 28.6% 25.0%

38.6% 31.1% 40.9% 23.1% 14.3% 25.0%

30.0% 31.1% 27.3% 38.5% 28.6% 50.0%

35.7% 31.1% 36.4% 7.7% 42.9% 50.0%

41.4% 28.9% 40.9% 30.8% 14.3% 50.0%

41.4% 14.3% 44.3% 64.3% 28.9% 11.1% 33.3% 55.6% 36.4% 4.5% 54.5% 68.2% 38.5% 0.0% 38.5% 38.5% 14.3% 0.0% 42.9% 57.1% 50.0% 0.0% 50.0% 50.0%

32.8% 27.9% 34.1% 21.4% 20.6% 37.5%

Unauthorized actor ui_viewer other_authenticated_user log_reader adjacent_client cross_tenant_actor infra_artifact_reader chat_recipient workspace_restricted_user redirect_target sql_user Others

33 30.2% 40.7% 29 19.4% 27.8% 29 36.0% 63.9% 16 14.8% 31.0% 12 14.6% 51.4% 11 28.4% 0.0% 9 35.8% – 7 37.3% – 7 23.6% 37.0% 7 27.8% 24.1% 1 0.0% –

34.0% 26.8% 36.9% 27.2% 13.3% 11.1% 35.7% – 35.6% 24.6% 0.0%

24.1% 16.7% 58.6% 15.1% 36.5% 44.4% 36.1% 37.3% 13.9% – –

27.3% 17.2% 44.8% 18.8% 25.0% 9.1% 22.2% 14.3% 28.6% 42.9% 0.0%

30.3% 13.8% 37.9% 25.0% 33.3% 27.3% 44.4% 42.9% 28.6% 14.3% 0.0%

30.3% 17.2% 34.5% 31.2% 33.3% 18.2% 66.7% 57.1% 28.6% 42.9% 0.0%

21.2% 10.3% 31.0% 6.2% 8.3% 18.2% 0.0% 14.3% 14.3% 0.0% 0.0%

12.1% 6.9% 17.2% 6.2% 8.3% 9.1% 0.0% 0.0% 28.6% 0.0% 0.0%

21.2% 13.8% 24.1% 12.5% 8.3% 9.1% 33.3% 28.6% 28.6% 0.0% 0.0%

36.4% 17.2% 48.3% 31.2% 50.0% 27.3% 33.3% 14.3% 42.9% 28.6% 0.0%

45.5% 31.0% 55.2% 18.8% 33.3% 27.3% 44.4% 57.1% 28.6% 42.9% 0.0%

27.3% 27.6% 44.8% 18.8% 41.7% 27.3% 22.2% 42.9% 28.6% 14.3% 0.0%

42.4% 24.1% 51.7% 25.0% 16.7% 27.3% 33.3% 42.9% 42.9% 28.6% 0.0%

39.4% 24.1% 48.3% 18.8% 33.3% 27.3% 22.2% 57.1% 28.6% 28.6% 0.0%

33.3% 31.0% 48.3% 25.0% 33.3% 27.3% 44.4% 42.9% 14.3% 42.9% 0.0%

48.5% 37.9% 51.7% 37.5% 25.0% 27.3% 33.3% 28.6% 28.6% 14.3% 0.0%

36.4% 27.6% 44.8% 18.8% 25.0% 36.4% 44.4% 28.6% 42.9% 42.9% 0.0%

30.3% 27.6% 44.8% 25.0% 25.0% 18.2% 55.6% 57.1% 0.0% 14.3% 0.0%

39.4% 24.1% 44.8% 31.2% 25.0% 18.2% 55.6% 28.6% 28.6% 14.3% 0.0%

33.3% 27.6% 55.2% 25.0% 33.3% 36.4% 33.3% 57.1% 28.6% 28.6% 0.0%

24.2% 12.1% 48.5% 66.7% 27.6% 6.9% 37.9% 55.2% 58.6% 17.2% 58.6% 75.9% 18.8% 6.2% 37.5% 37.5% 25.0% 8.3% 50.0% 50.0% 27.3% 9.1% 36.4% 36.4% 55.6% 0.0% 66.7% 77.8% 57.1% 0.0% 57.1% 57.1% 57.1% 0.0% 57.1% 57.1% 42.9% 0.0% 42.9% 71.4% 0.0% 0.0% 0.0% 0.0%

32.2% 22.6% 43.7% 21.9% 26.9% 23.2% 35.8% 37.3% 29.4% 24.6% 0.0%

Exposure channel http_response ui log tool_result chat_message deployment_config http_request_header sql_result error_message Others

41 19.9% 32.5% 29 28.9% 36.7% 26 34.0% 63.9% 14 24.2% – 12 18.5% – 12 28.3% 0.0% 8 23.6% 38.9% 7 27.8% 24.1% 6 48.1% 42.6% 6 46.3% –

24.5% 32.7% 37.0% 5.6% 29.4% 13.2% 35.7% 24.6% 66.7% 22.2%

27.5% 9.3% 56.9% 29.3% 3.3% 44.4% 0.0% – 24.1% 58.3%

22.0% 27.6% 42.3% 7.1% 16.7% 8.3% 25.0% 42.9% 50.0% 33.3%

22.0% 27.6% 38.5% 21.4% 16.7% 33.3% 37.5% 14.3% 50.0% 50.0%

26.8% 27.6% 34.6% 35.7% 33.3% 25.0% 25.0% 42.9% 33.3% 66.7%

9.8% 20.7% 30.8% 7.1% 0.0% 16.7% 12.5% 0.0% 16.7% 33.3%

7.3% 13.8% 15.4% 0.0% 0.0% 8.3% 25.0% 0.0% 16.7% 16.7%

14.6% 17.2% 26.9% 21.4% 25.0% 8.3% 25.0% 0.0% 0.0% 33.3%

24.4% 27.6% 50.0% 42.9% 16.7% 25.0% 50.0% 28.6% 50.0% 50.0%

31.7% 41.4% 53.8% 35.7% 25.0% 33.3% 25.0% 42.9% 66.7% 50.0%

36.6% 20.7% 42.3% 21.4% 16.7% 25.0% 25.0% 14.3% 50.0% 50.0%

22.0% 37.9% 53.8% 35.7% 25.0% 25.0% 50.0% 28.6% 33.3% 50.0%

31.7% 41.4% 46.2% 21.4% 16.7% 25.0% 25.0% 28.6% 66.7% 16.7%

31.7% 34.5% 46.2% 21.4% 16.7% 33.3% 25.0% 42.9% 66.7% 50.0%

39.0% 44.8% 53.8% 28.6% 25.0% 25.0% 37.5% 14.3% 50.0% 33.3%

31.7% 34.5% 42.3% 21.4% 8.3% 41.7% 50.0% 42.9% 50.0% 33.3%

24.4% 31.0% 46.2% 35.7% 25.0% 16.7% 12.5% 14.3% 50.0% 66.7%

31.7% 31.0% 42.3% 14.3% 33.3% 16.7% 37.5% 14.3% 66.7% 66.7%

31.7% 31.0% 53.8% 28.6% 16.7% 33.3% 25.0% 28.6% 66.7% 66.7%

24.4% 34.5% 57.7% 35.7% 16.7% 25.0% 50.0% 42.9% 33.3% 66.7%

53.7% 65.5% 76.9% 57.1% 41.7% 41.7% 50.0% 71.4% 66.7% 66.7%

25.7% 30.3% 43.2% 24.2% 18.5% 23.6% 31.2% 24.6% 45.4% 46.3%

Repair pattern mask_redact add_authorization_check remove_field defer_resolution reject_hardlink Others

58 30.7% 50.6% 47 25.1% 30.6% 36 27.8% 38.9% 7 40.7% 0.0% 6 26.9% – 7 6.9% 16.7%

37.1% 24.7% 25.6% 24.4% 66.7% 6.7%

38.9% 28.5% 42.2% 0.0% 18.9% 22.2%

34.5% 21.3% 30.6% 14.3% 0.0% 0.0%

31.0% 25.5% 33.3% 28.6% 33.3% 0.0%

36.2% 25.5% 33.3% 28.6% 33.3% 28.6%

19.0% 12.8% 19.4% 14.3% 0.0% 0.0%

13.8% 4.3% 11.1% 14.3% 0.0% 14.3%

20.7% 14.9% 16.7% 0.0% 33.3% 28.6%

43.1% 23.4% 33.3% 14.3% 50.0% 28.6%

51.7% 38.3% 30.6% 28.6% 33.3% 0.0%

32.8% 34.0% 27.8% 14.3% 33.3% 14.3%

46.6% 25.5% 36.1% 14.3% 33.3% 14.3%

41.4% 34.0% 30.6% 14.3% 33.3% 0.0%

43.1% 31.9% 36.1% 28.6% 16.7% 0.0%

48.3% 38.3% 36.1% 14.3% 33.3% 0.0%

43.1% 29.8% 27.8% 28.6% 33.3% 28.6%

37.9% 25.5% 36.1% 14.3% 33.3% 0.0%

41.4% 29.8% 36.1% 14.3% 0.0% 14.3%

44.8% 31.9% 38.9% 14.3% 33.3% 0.0%

46.6% 13.8% 51.7% 72.4% 29.8% 4.3% 38.3% 53.2% 30.6% 11.1% 38.9% 55.6% 14.3% 0.0% 28.6% 28.6% 50.0% 0.0% 50.0% 50.0% 28.6% 0.0% 28.6% 57.1%

37.5% 26.5% 30.2% 17.5% 26.9% 11.1%

Primary CWE CWE-201 CWE-862 CWE-532 CWE-538 CWE-22 Others

51 35.4% 27.4% 36 18.6% 29.9% 26 34.0% 63.9% 12 16.2% 0.0% 9 23.5% – 27 24.7% 41.4%

33.8% 22.2% 37.0% 8.3% – 35.3%

31.9% 23.6% 56.9% 47.2% 23.5% 25.0%

29.4% 19.4% 42.3% 8.3% 11.1% 25.9%

29.4% 25.0% 38.5% 16.7% 11.1% 33.3%

39.2% 25.0% 34.6% 16.7% 22.2% 33.3%

17.6% 5.6% 30.8% 8.3% 11.1% 14.8%

11.8% 2.8% 15.4% 8.3% 0.0% 14.8%

21.6% 11.1% 26.9% 0.0% 22.2% 18.5%

39.2% 22.2% 50.0% 16.7% 11.1% 37.0%

41.2% 30.6% 53.8% 25.0% 33.3% 40.7%

27.5% 33.3% 42.3% 8.3% 33.3% 29.6%

43.1% 16.7% 53.8% 16.7% 22.2% 37.0%

31.4% 25.0% 46.2% 16.7% 33.3% 44.4%

37.3% 30.6% 46.2% 25.0% 33.3% 29.6%

45.1% 33.3% 53.8% 16.7% 22.2% 33.3%

33.3% 27.8% 42.3% 41.7% 22.2% 37.0%

33.3% 25.0% 46.2% 8.3% 44.4% 25.9%

47.1% 25.0% 42.3% 8.3% 11.1% 25.9%

35.3% 25.0% 53.8% 16.7% 33.3% 44.4%

35.3% 11.8% 47.1% 58.8% 25.0% 2.8% 33.3% 58.3% 57.7% 15.4% 57.7% 76.9% 8.3% 0.0% 41.7% 50.0% 44.4% 0.0% 44.4% 44.4% 40.7% 14.8% 44.4% 55.6%

33.2% 22.7% 43.2% 14.8% 23.5% 31.5%

Resultant CWE CWE-201 CWE-532 CWE-538 CWE-359 Others

91 28.0% 28.5% 26 34.0% 63.9% 19 21.2% 13.0% 8 25.9% 41.1% 17 22.8% 42.9%

30.2% 37.0% 15.4% 19.4% 33.8%

24.3% 56.9% 29.6% 51.4% 25.9%

23.1% 42.3% 10.5% 37.5% 29.4%

25.3% 38.5% 26.3% 37.5% 29.4%

33.0% 34.6% 21.1% 37.5% 29.4%

13.2% 30.8% 10.5% 12.5% 11.8%

8.8% 15.4% 5.3% 12.5% 11.8%

19.8% 26.9% 5.3% 12.5% 11.8%

28.6% 50.0% 26.3% 37.5% 41.2%

35.2% 53.8% 36.8% 50.0% 35.3%

27.5% 42.3% 15.8% 50.0% 35.3%

33.0% 53.8% 21.1% 37.5% 29.4%

28.6% 46.2% 31.6% 37.5% 41.2%

33.0% 46.2% 21.1% 50.0% 35.3%

39.6% 53.8% 21.1% 37.5% 29.4%

30.8% 42.3% 36.8% 37.5% 35.3%

29.7% 46.2% 10.5% 37.5% 35.3%

35.2% 42.3% 10.5% 25.0% 35.3%

29.7% 53.8% 31.6% 50.0% 41.2%

33.0% 8.8% 39.6% 59.3% 57.7% 15.4% 57.7% 76.9% 15.8% 5.3% 36.8% 47.4% 37.5% 12.5% 50.0% 50.0% 41.2% 11.8% 41.2% 52.9%

28.1% 43.2% 19.9% 35.4% 31.0%

Primary CWE scope In CWE-200 Outside CWE-200

104 30.3% 39.8% 57 22.2% 32.0%

31.2% 25.4%

37.9% 30.8% 24.9% 17.5%

29.8% 26.3%

33.7% 28.1%

19.2% 8.8%

12.5% 5.3%

18.3% 17.5%

39.4% 22.8%

41.3% 35.1%

28.8% 33.3%

41.3% 22.8%

34.6% 31.6%

37.5% 29.8%

42.3% 31.6%

37.5% 28.1%

32.7% 28.1%

39.4% 21.1%

38.5% 31.6%

37.5% 12.5% 42.3% 60.6% 33.3% 5.3% 35.1% 57.9%

33.1% 25.1%

Application domain AI & Machine Learning Data Infrastructure Security & Identity Others Web & API DevOps & Cloud Developer Tools

58 27.4% 58.3% 33 26.9% 28.5% 19 0.0% 29.2% 19 28.9% – 15 50.6% 53.7% 12 12.5% 37.5% 5 11.1% 63.9%

26.2% 26.7% 20.7% 27.5% 50.4% 22.7% 44.4%

29.7% 31.5% 29.9% 41.7% 72.2% 0.0% 88.9%

24.1% 27.3% 26.3% 31.6% 53.3% 8.3% 60.0%

29.3% 33.3% 31.6% 26.3% 46.7% 16.7% 60.0%

17.2% 12.1% 5.3% 15.8% 26.7% 16.7% 20.0%

6.9% 6.1% 15.8% 5.3% 20.0% 8.3% 40.0%

22.4% 9.1% 10.5% 10.5% 33.3% 8.3% 60.0%

34.5% 36.4% 31.6% 21.1% 40.0% 16.7% 80.0%

39.7% 36.4% 31.6% 42.1% 60.0% 33.3% 20.0%

27.6% 24.2% 26.3% 21.1% 73.3% 25.0% 40.0%

31.0% 30.3% 21.1% 42.1% 60.0% 33.3% 60.0%

27.6% 33.3% 26.3% 36.8% 66.7% 25.0% 40.0%

29.3% 36.4% 31.6% 26.3% 60.0% 33.3% 60.0%

34.5% 32.8% 39.4% 33.3% 26.3% 21.1% 36.8% 26.3% 73.3% 46.7% 25.0% 33.3% 60.0% 100.0%

34.5% 24.2% 26.3% 31.6% 46.7% 8.3% 60.0%

32.8% 21.2% 26.3% 36.8% 66.7% 25.0% 40.0%

34.5% 33.3% 31.6% 36.8% 60.0% 25.0% 40.0%

36.2% 6.9% 39.7% 55.2% 39.4% 6.1% 39.4% 60.6% 26.3% 5.3% 31.6% 47.4% 36.8% 5.3% 42.1% 63.2% 40.0% 20.0% 73.3% 80.0% 16.7% 8.3% 33.3% 50.0% 80.0% 20.0% 100.0% 100.0%

28.4% 27.6% 24.6% 28.9% 51.9% 20.8% 53.3%

17.2% 21.2% 26.3% 36.8% 60.0% 16.7% 40.0%

7.3% 13.8% 15.4% 0.0% 0.0% 8.3% 12.5% 0.0% 0.0% 16.7%

39.0% 44.8% 57.7% 42.9% 33.3% 41.7% 50.0% 42.9% 66.7% 66.7%

GLM-5.3-Flash, Qwen3.8-Flash, and GPT-6 Luna. Because they postdate the leakage boundary, we report them separately from the primary 18 configurations (Table 5). All four improve under the same agent, resolving 58–65 instances (36.0%–40.4%). DeepSeek-V4.1-Flash and GLM-5.3-Flash each resolve 65 (40.4%), exceeding the best primary configuration at 63 (39.1%), although these results are not leakage-controlled. The newer models are also cheaper ($0.012–$0.082 per resolved instance) and generally require fewer steps than the two step-heavy primary models. RQ1 Summary: Current systems remain limited on SIExBench: the best of the 18 configurations resolves 39.1% of instances, while 40.4% remain unresolved by all. Exposure removal is harder than preserving existing behavior, and test normalization prevents implementationspecific assertions from rejecting secure repairs. Complementary configurations increase overall coverage to 59.6%. Under a fixed agent, newer post-boundary models achieve 36.0%–40.4% resolve rates, though these results are not leakage-controlled.

, Vol. 1, No. 1, Article . Publication date: November 2018.

SIExBench

15

Table 7. Failure-cause distributions across the 18 configurations. Values are percentages of analyzed failures within each column. Scope indicates the generality of each cause: generic to issue resolution ( any fix ), generic to security repair ( sec. fix ), or specific to SIEx repair ( SIEx ). Cause

Description

C1. no_patch the method produced no source change C2. patch_invalid the patch does not apply or build C3. exposure_not_removed the sensitive value is still exposed after the patch C3.1. localization_error the patch edits code outside the leak’s flow C3.2. incomplete_sanitization redaction misses nested fields or is reversible C3.3. another_sink_uncovered the same value is still exposed through another output C3.4. secret_in_logs_or_errors only log level or one error branch changed C3.5. wrong_value_or_fallback a stale, default or wrong value reaches the consumer C3.6. unsafe_channel_routing guarded content goes to the wrong recipient C3.7. ineffective_authorization permission rule missing or wrong identity C3.8. stale_shared_state cached or shared state keeps revoked access C3.9. path_confinement path checks miss links or part of the operations C3.10. req_validation redirect, host or parameter checks are bypassable C4. regression_over_fix exposure fixed, other behaviour broken C4.1. over_broad_change redaction too broad, breaks legitimate callers C4.2. wrong_value_regression changed value or fallback breaks other behaviour C4.3. other_regression other behaviour stops working after the fix C5. non_termination the patched code hangs, evaluation cannot finish Analysed failures

6.2

Scope any fix any fix any fix SIEx SIEx SIEx SIEx SIEx SIEx sec. fix sec. fix sec. fix SIEx SIEx any fix any fix

Py Java ≤02-16 >02-16 26.7% 15.2% 7.5% 12.3% 61.7% 63.0% 14.2% 11.1% 11.4% 13.4% 5.2% 3.6% 6.1% 0.5% 3.4% 4.8% 1.5% 0.0% 6.4% 16.8% 4.2% 3.9% 6.3% 0.7% 3.0% 8.2% 3.6% 9.5% 2.5% 9.3% 0.1% 0.2% 1.0% 0.0% 0.4% 0.0%

24.1% 10.1% 59.6% 15.1% 12.7% 3.9% 5.3% 5.2% 0.0% 9.7% 5.2% 0.3% 2.3% 5.8% 4.6% 0.1% 1.1% 0.4%

MSWE-agent

Magentless

MopenHands

mini-swe-agent

OpenCode

DS Qwen

DS Qwen

DS Qwen

DS Qwen

DS Qwen

GPT

GPT

GPT

GPT

Codex GPT

DS Qwen

All GPT

21.9% 31.9% 19.5% 21.2% 22.5% 59.2% 26.4% 11.1% 15.1% 8.3% 34.7% 40.5% 3.9% 4.2% 22.0% 25.0% 20.8% 12.3% 18.7% 6.3% 22.0% 21.8% 13.8% 9.9% 12.5% 12.3% 4.2% 4.1% 7.1% 0.0% 1.3% 2.6% 5.6% 2.4% 7.5% 10.4% 13.7% 1.3% 67.3% 42.9% 55.2% 61.2% 64.0% 27.5% 58.5% 72.2% 72.6% 77.4% 61.3% 48.1% 89.6% 81.9% 68.3% 58.8% 59.7% 68.5% 76.0% 9.6% 7.7% 19.5% 12.5% 18.9% 12.5% 17.9% 27.8% 17.8% 16.7% 6.7% 5.1% 11.7% 18.1% 17.1% 6.2% 10.4% 2.7% 8.0% 10.4% 6.6% 8.0% 11.2% 12.6% 3.3% 13.2% 4.2% 13.7% 15.5% 13.3% 12.7% 20.8% 19.4% 15.9% 8.8% 11.7% 15.1% 14.7% 6.5% 0.0% 3.4% 7.5% 3.6% 1.7% 9.4% 5.6% 15.1% 4.8% 1.3% 0.0% 5.2% 4.2% 6.1% 7.5% 1.3% 4.1% 6.7% 2.9% 5.5% 0.0% 3.8% 4.5% 0.8% 0.9% 5.6% 5.5% 7.1% 6.7% 6.3% 5.2% 2.8% 1.2% 10.0% 6.5% 5.5% 6.7% 0.8% 4.4% 3.4% 6.2% 1.8% 0.0% 1.9% 2.8% 1.4% 4.8% 2.7% 3.8% 9.1% 4.2% 6.1% 5.0% 2.6% 4.1% 6.7% 3.3% 0.0% 0.0% 0.0% 1.8% 0.0% 0.9% 1.4% 0.0% 2.4% 2.7% 2.5% 1.3% 1.4% 1.2% 1.2% 1.3% 1.4% 0.0% 9.0% 5.5% 9.2% 8.8% 7.2% 5.0% 6.6% 13.9% 4.1% 10.7% 10.7% 7.6% 16.9% 12.5% 9.8% 7.5% 10.4% 13.7% 16.0% 1.8% 4.4% 1.1% 3.8% 5.4% 1.7% 3.8% 5.6% 5.5% 3.6% 6.7% 0.0% 5.2% 5.6% 4.9% 7.5% 2.6% 4.1% 4.0% 13.9% 5.5% 4.6% 3.8% 5.4% 0.0% 2.8% 2.8% 4.1% 8.3% 4.0% 6.3% 6.5% 5.6% 2.4% 2.5% 10.4% 6.8% 5.3% 9.0% 3.3% 5.7% 3.8% 2.7% 2.5% 0.9% 2.8% 5.5% 3.6% 6.7% 3.8% 7.8% 8.3% 3.7% 2.5% 2.6% 11.0% 8.0% 4.5% 3.3% 3.4% 3.8% 2.7% 0.8% 1.9% 11.1% 8.2% 7.1% 4.0% 10.1% 3.9% 8.3% 7.3% 8.8% 9.1% 4.1% 4.0% 4.3% 3.3% 3.4% 2.5% 1.8% 0.8% 0.9% 9.7% 8.2% 6.0% 2.7% 10.1% 3.9% 6.9% 6.1% 6.2% 7.8% 2.7% 2.7% 0.2% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 1.2% 0.0% 0.0% 0.0% 0.0% 0.0% 1.2% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 1.2% 0.9% 0.0% 0.9% 1.4% 0.0% 0.0% 1.3% 0.0% 0.0% 1.4% 1.2% 1.2% 1.3% 1.4% 1.3% 0.0% 0.0% 0.0% 0.0% 0.9% 0.0% 0.9% 1.4% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 1.4% 0.0%

23.4% 8.9% 62.1% 13.3% 12.0% 4.8% 4.5% 3.8% 1.1% 9.4% 4.1% 4.7% 4.5% 5.4% 4.5% 0.1% 0.7% 0.3%

100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0%

100.0%

RQ2: Stratified Performance

Whereas RQ1 examines aggregate repair performance, RQ2 investigates how effectiveness varies across defect characteristics. We stratify the 161 instances along eight characteristics, comprising the six characterization dimensions, primary-CWE scope, and application domain. The experimental results are shown in Table 6. Repairability varies across defect characteristics. Resolve rates differ markedly across both characteristic values and configurations. For example, among credential and secret exposures, individual configurations resolve 14.3%–44.3% of instances. Moreover, no configuration achieves the highest resolve rate in more than 16 of the 51 fine-grained categories. Thus, aggregate performance alone does not capture how repair effectiveness varies across different forms of SIEx defects. Direct exposure mitigation is more tractable than structural repair. A consistent pattern emerges across repair patterns and root causes. Instances repaired through masking or redaction achieve a mean resolve rate of 37.5%, compared with 26.5% for authorization checks and 17.5% for deferred resolution. Similarly, CWE-532 defects involving sensitive information in logs average 43.2%, whereas CWE-862 defects involving missing authorization average 22.7%. These results suggest that current systems more readily handle defects where the exposed value can be directly suppressed than those requiring changes to authorization or other underlying program behavior. Root causes outside CWE-200 are harder on average. Instances whose primary CWE lies outside the CWE-200 hierarchy achieve a lower mean resolve rate than those within it (25.1% vs. 33.1%). Nevertheless, 57.9% of these instances are resolved by at least one configuration, compared with 35.1% for the best individual configuration. Thus, these defects are challenging for individual systems rather than uniformly unsolvable. This finding also supports including resultant exposures whose root causes fall outside CWE-200, as restricting the benchmark to the hierarchy would omit a substantial and relatively difficult subset. RQ2 Summary: Repairability varies markedly across defect characteristics, with no configuration consistently leading across categories. Defects addressed through direct sanitization are repaired more successfully than those requiring authorization or other structural changes. Moreover, exposures originating outside the CWE-200 hierarchy are harder on average, underscoring the importance of including such resultant exposures in repair evaluation.

, Vol. 1, No. 1, Article . Publication date: November 2018.

16

6.3

Zhang et al.

RQ3: Failure Analysis

To investigate the limitations identified in RQ1 and RQ2, we analyze the unsuccessful configuration– instance attempts across all 18 configurations. We classify each failure according to its primary cause (Table 7), examine how these causes vary with defect characteristics (Figure 12), and illustrate recurring failure mechanisms through four representative cases (Figure 13). Our taxonomy distinguishes five main outcomes: no source change (C1), an invalid patch (C2), a valid patch that leaves the exposure unresolved (C3), a patch that eliminates the exposure but introduces functional regressions (C4), and non-termination (C5). We further decompose C3 and C4 into specific causes. All percentages are computed over failed attempts, with each attempt assigned to exactly one main cause and, where applicable, one corresponding subcategory. Incomplete exposure removal dominates failures. As shown in Table 7, most failed attempts produce a valid patch that still leaves the exposure unresolved (C3, 62.1%). By comparison, 23.4% produce no source change (C1), 8.9% produce an invalid patch (C2), 5.4% remove the exposure but introduce regressions (C4), and 0.3% do not terminate (C5). Thus, the main challenge is not merely generating a valid patch or preserving functionality, but actually eliminating the exposure. A closer examination of C3 shows that sanitization and information-handling failures (C3.2–C3.6) account for 26.0% of all failures, including 12.0% from incomplete or reversible sanitization alone. Access-control and state-management failures (C3.7–C3.8) contribute another 13.5%, followed by localization errors (C3.1, 13.3%) and containment or request-validation failures (C3.9–C3.10, 9.2%). Although localization is the largest individual subcategory, it accounts for only about one fifth of C3 failures. Nearly four fifths arise after systems reach the relevant code, indicating that localization alone is insufficient. Successful repair must also account for the conditions, representations, and execution paths through which sensitive information remains accessible. This pattern is even stronger for the 65 instances unresolved by every configuration. C3 accounts for 71.5% of their failed attempts, compared with 53.0% for instances resolved by at least one configuration. Thus, consistently unresolved instances are characterized less by a failure to generate patches than by repeated failure to eliminate the underlying exposure. Failure profiles differ across configurations. Although C3 dominates overall, failure profiles vary substantially across configurations. No-patch failures (C1) account for 3.9%–59.2% of unsuccessful attempts, invalid patches (C2) for 0%–22.0%, and surviving exposures (C3) for 27.5%–89.6%. For example, Magentless has more no-patch failures than the agentic methods (36.8% vs. 19.5%) but fewer surviving exposures (49.3% vs. 65.8%). Thus, Magentless more often fails to produce a patch, whereas agentic methods more often generate executable patches that leave the exposure unresolved. Overall resolve rates therefore obscure where different repair processes fail. Failure mechanisms are associated with defect characteristics. Figure 12 further relates the failure taxonomy to the characteristics examined in RQ2. Distinct exposure mechanisms exhibit markedly different failure profiles. Sanitization failures dominate log exposures (48.9%), whereas containment and validation failures are more common for tool-result exposures (34.2%) and especially file-content exposures (58.3%). These results indicate that unsuccessful repairs fail in different ways depending on how sensitive information becomes accessible. The CWE dimensions provide consistent evidence. Sanitization accounts for 48.9% of failures involving CWE-532 (Insertion of Sensitive Information into Log File), while access-control and state failures account for 35.1% of those rooted in CWE-862 (Missing Authorization). For CWE-22 (Path Traversal), containment and validation dominate at 60.0%. Similar associations appear in repair patterns: 40.5% of failed masking or redaction repairs involve sanitization problems, whereas 38.4% of failed authorization-check repairs involve access-control and state failures. Defects requiring hard-link rejection exhibit an even stronger concentration in containment-related failures (75.6%).

, Vol. 1, No. 1, Article . Publication date: November 2018.

SIExBench

17

( a ) Unauthorized actor

( c ) Exposure channel

90

( f ) Sensitive information type

UI viewer

26.8%

9.6%

18.4%

37.3%

2.6%

0.9%

4.1%

0.3%

HTTP response

23.9%

10.5%

8.0%

13.4%

27.7%

10.5%

6.0%

0.0%

credential secret

23.1%

7.8%

17.1%

33.0%

3.1%

5.9%

9.6%

0.5%

other auth user

23.4%

10.3%

8.8%

7.3%

36.6%

8.1%

5.5%

0.0%

UI

25.7%

10.6%

16.2%

32.0%

6.6%

1.0%

7.6%

0.3%

private record

23.6%

10.4%

9.3%

27.6%

24.7%

2.4%

1.8%

0.2%

6.7%

16.9%

log reader

21.9%

5.6%

18.7%

47.4%

0.4%

0.4%

4.4%

1.2%

log

19.8%

5.5%

19.0%

48.9%

0.4%

0.4%

4.6%

1.3%

business record

33.8%

10.8%

29.7%

0.5%

1.5%

0.0%

adjacent client

16.5%

9.4%

10.6%

30.0%

19.4%

10.6%

3.5%

0.0%

tool result

20.1%

14.1%

4.7%

10.7%

12.8%

34.2%

3.4%

0.0%

file content

17.3%

8.6%

2.2%

7.9%

4.3%

58.3%

1.4%

0.0%

cross-tenant actor

26.7%

11.1%

6.7%

0.7%

28.9%

20.7%

5.2%

0.0%

chat message

26.5%

0.0%

21.7%

33.7%

14.5%

0.0%

3.6%

0.0%

file path

11.7%

11.7%

19.1%

37.2%

2.1%

10.6%

7.4%

0.0%

infra artifact reader

21.8%

0.0%

9.1%

32.7%

1.8%

30.9%

3.6%

0.0%

deploy config

25.0%

0.0%

10.3%

32.4%

2.9%

25.0%

4.4%

0.0%

Others

23.8%

19.0%

4.8%

0.0%

42.9%

0.0%

9.5%

0.0%

chat recipient

33.3%

3.2%

22.2%

27.0%

9.5%

0.0%

4.8%

0.0%

request header

7.9%

3.2%

14.3%

17.5%

19.0%

28.6%

9.5%

0.0%

( g ) Repair pattern

workspace user

22.9%

12.5%

2.1%

10.4%

0.0%

50.0%

2.1%

0.0%

SQL result

29.7%

17.6%

20.9%

28.6%

0.0%

2.2%

1.1%

0.0%

mask/redact

23.0%

8.7%

21.0%

40.5%

0.2%

0.8%

5.2%

0.6%

redirect target

9.0%

7.5%

3.0%

11.9%

1.5%

35.8%

31.3%

0.0%

error message

20.0%

15.0%

25.0%

35.0%

0.0%

0.0%

5.0%

0.0%

add auth check

26.0%

11.1%

7.7%

5.9%

38.4%

7.2%

3.5%

0.2%

sql user

29.7%

17.6%

20.9%

28.6%

0.0%

2.2%

1.1%

0.0%

Others

32.7%

1.9%

13.5%

21.2%

28.8%

0.0%

1.9%

0.0%

Others

11.1%

0.0%

0.0%

5.6%

83.3%

0.0%

0.0%

0.0%

( d ) Resultant CWE

( b ) Application domain

remove field

31.6%

10.9%

10.0%

30.3%

7.5%

3.4%

6.2%

0.0%

defer resolution

11.8%

2.9%

14.7%

32.4%

22.5%

14.7%

1.0%

0.0%

9.8%

13.0%

23.3%

15.5%

8.8%

4.7%

0.1%

reject hardlink

11.5%

1.3%

3.8%

5.1%

0.0%

75.6%

2.6%

0.0%

6.2%

11.4%

25.8%

9.5%

17.0%

3.7%

0.0%

CWE-532

19.8%

5.5%

19.0%

48.9%

0.4%

0.4%

4.6%

1.3%

Others

8.7%

6.5%

13.0%

28.3%

2.2%

22.8%

18.5%

0.0%

28.2%

11.3%

22.9%

24.9%

7.3%

1.7%

3.7%

0.0%

CWE-538

25.3%

1.9%

16.5%

25.3%

1.3%

20.9%

8.9%

0.0%

( h ) Primary CWE

CWE-359

21.5%

13.8%

1.5%

0.0%

60.0%

0.0%

3.1%

0.0%

CWE-201

23.6%

9.4%

15.0%

33.0%

8.8%

5.1%

4.9%

0.2%

Others

17.6%

15.3%

7.6%

17.6%

15.3%

18.3%

8.4%

0.0%

CWE-862

25.9%

10.4%

11.9%

10.0%

35.1%

4.0%

2.7%

0.0%

19.8%

5.5%

19.0%

48.9%

0.4%

0.4%

4.6%

1.3%

23.0%

8.2%

15.2%

36.7%

6.2%

4.2%

6.0%

0.4%

CWE-538

32.0%

1.9%

13.6%

38.8%

1.9%

0.0%

11.7%

0.0%

Dev tools

17.1%

14.6%

9.8%

31.7%

2.4%

17.1%

7.3%

0.0%

Outside CWE-200

24.0%

10.0%

10.4%

9.2%

25.2%

17.0%

4.3%

0.0%

CWE-22

20.0%

11.2%

1.2%

6.2%

1.2%

60.0%

0.0%

0.0%

Others

19.1%

11.3%

10.3%

15.7%

8.3%

24.0%

11.3%

0.0%

60

50

40

30

20

10

0

no nte

ce

co nt a

no

ac

pa tc

rm no nte

co nt a

ce ac

lo ca

no

sa ni tiz

pa tc

rm no nte

co nt a

ss ce ac

lo ca

sa ni tiz

pa tc no

70

rm

CWE-532

In CWE-200

in

( e ) Primary CWE scope

0.0%

re gr es s

0.0%

7.4%

ss

0.0%

2.7%

lo ca

0.0%

17.6%

sa ni tiz

3.1%

27.7%

h

35.7%

11.5%

in va lid

15.3%

12.8%

in

15.3%

20.3%

re gr es s

30.6%

DevOps

e

Web & API

ss

Others

2.0%

liz .

0.0%

2.0%

h

18.1%

1.5%

in va lid

15.3%

21.7%

in

32.2%

32.5%

re gr es s

13.0%

9.9%

e

8.5%

liz .

1.1% 12.3%

h

11.9% 18.2%

in va lid

Security

e

24.9%

26.4%

liz .

CWE-201

AI & ML Data infra

80

Fig. 12. Failure-family distributions across the eight defect characteristics in Table 6, row-normalized by characteristic value. The eight families are no patch (C1), invalid patch (C2), localization (C3.1), sanitization (C3.2–C3.6), access control and state (C3.7–C3.8), containment and validation (C3.9–C3.10), regression (C4.1– C4.3), and non-termination (C5). (1) localization (C3.1)

(2) sanitization (C3.2)

(3) access control (C3.7)

(4) containment (C3.9)

airflow#17604

shardingsphere#25616

keycloak#36838

agno#8632

Py · 16/18 repeat

Java · 16/18 repeat

Java · 18/18 repeat

Py · 15/18 repeat

Issue Task logs leak secrets held in the exception context

Issue from-x = to-y returns the raw value

Issue one user's grant read as a grant to all users

Issue symlinks in agent_directory upload outside files

secrets_masker.py

KeepFromXToY…Algorithm.java

UserPermissionsV2.java

antigravity.py

+ def _redact_exception_ + _with_context(self, e): + e.args = redact(e.args)

- if (… || toY <= fromX) return result; + checkState(fromX <= toY);

+ // an "all-users" grant? + if (findByName(…) == null) + return false;

- cfg = path/"agent.yaml" + cfg = safe_join(path, + "agent.yaml")

Exposure path and fix

Exposure path and fix

Exposure path and fix

Exposure path and fix

env var

masker

task log

SQL result

mask algo

client

admin request

perms V2

all users

directory

safe_join

upload

Failed patch · MSWE-agent · DS

Failed patch · MSWE-agent · DS

Failed patch · MSWE-agent · DS

Failed patch · MSWE-agent · DS

+ mask_env_vars = False + # KubernetesPodOperator only + # masker unchanged

+

+ Resource r = perm.get(); + if (r == null || + r.getId() != res.getId())

+ if f.is_symlink(): + continue # workspace/ + # agent.yaml not covered

Why it fails the patch never touches the leaking path

Why it fails the equal-bound case still returns the raw value

Why it fails the check never runs on the exercised path

Why it fails the sibling paths are left unchecked

|| toY <= fromX) || toY < fromX)

Fig. 13. Four representative cases illustrating distinct failure families on instances unresolved by all 18 configurations: localization (C3.1), sanitization (represented by C3.2), access control and state (represented by C3.7), and containment and validation (represented by C3.9).

Taken together, these results show that unsuccessful repairs are not uniformly distributed across failure mechanisms. Rather, their composition is closely associated with the exposure mechanism, underlying weakness, and required mitigation. Thus, different defect types differ not only in repair success, as shown in RQ2, but also in the technical failure modes encountered during repair. Case studies of recurring incomplete repairs. To illustrate how incomplete repairs arise in practice, Figure 13 presents four instances unresolved by all 18 configurations. Each represents a distinct C3 failure family and contrasts the developer patch and exposure path with an unsuccessful patch from MSWE-agent with DeepSeek-V3.2. Airflow #17604 (C3.1) illustrates localization failure: the developer patch sanitizes exception arguments exposing retained secrets through task logs, whereas the failed patch modifies environmentvariable masking but misses the exposure path. ShardingSphere #25616 (C3.2) illustrates incomplete sanitization: the failed patch changes a boundary condition but allows the equality case to return , Vol. 1, No. 1, Article . Publication date: November 2018.

18

Zhang et al.

an unsanitized value. In both cases, generated changes modify related code but do not block the exposure path (Airflow #17604) or block it only partially (ShardingSphere #25616). The other two cases concern authorization and path containment. In Keycloak #36838 (C3.7), the generated patch adds a resource-identity check that is not reached on the path causing unauthorized access, while the developer fix corrects the underlying permission evaluation. In Agno #8632 (C3.9), the generated patch rejects some symbolic links but leaves sibling upload paths insufficiently constrained, whereas the developer fix applies safe path resolution to prevent uploads outside the intended directory. Again, the generated changes modify security-related code without blocking the exposure: the added identity check never runs on the exercised path (Keycloak #36838), and the symlink rejection leaves the sibling paths unchecked (Agno #8632). Across these four instances, 15–18 of the 18 configurations exhibit the corresponding failure. Their recurrence across methods and LLMs points to a common limitation: generated patches often address a visible manifestation of the defect without closing all exposure paths. This motivates future work that explicitly discovers and validates all relevant exposure paths, ensuring that a repair closes all of them rather than only the one it edits. RQ3 Summary: Incomplete exposure removal dominates failures, accounting for 62.1% of unsuccessful attempts. Nearly four fifths of these failures occur after the relevant code is reached, driven mainly by sanitization, access-control, and containment failures rather than localization alone. Representative cases further show that systems often modify related code without fully closing the exposure path. Effective SIEx repair therefore requires discovering and eliminating all relevant exposure paths.

7

Threats to Validity

Training-data contamination. Because LLMs may have seen the public GitHub data underlying SIExBench during training, we split the benchmark at February 16, 2026, the latest knowledge cutoff among the three evaluated LLMs, into 105 pre-boundary and 56 post-boundary instances and report both separately. Contamination may still affect the pre-boundary subset. Unintended external information access. Agentic systems may access unavailable information through Web search or future Git history. We mitigate this threat by blocking all network access except LLM API requests and removing future commits. However, LLM API providers may offer server-side Web search that bypasses this isolation. It was initially enabled for Codex with Qwen3.5Plus, yielding 64 resolved defects versus 58 after disabling it. We corrected the setting for subsequent runs and inspected all trajectories, finding no other such access. Test-oracle validity. Developer-written FAIL_TO_PASS tests may encode reference-fix details rather than intended security outcomes. We therefore inspect all tests, fix implementation-specific assertions in 57 of 161 instances, and verify each final exposure test on both buggy and fixed versions. Nevertheless, finite tests may miss exposure paths or regressions, motivating future SIEx-specific methods for assessing whether all relevant exposure paths are blocked. Benchmark representativeness. SIExBench focuses on Top-ranked Python and Java repositories and defects with merged source-code fixes and executable regression tests, and may not represent other languages, projects, or repair settings. Future work can extend it to broader scenarios. Benchmark construction and annotation. LLM pre-screening may miss relevant defects, while manual review and characterization involve subjective judgment. We mitigate the former by assessing each candidate three times, retaining it if any assessment identifies an exposure, and auditing 600 pairs rejected by all three, finding one missed SIEx defect. For manual review, two authors , Vol. 1, No. 1, Article . Publication date: November 2018.

SIExBench

19

independently assess retained candidates using evidence of sensitive information, exposure channel, and unauthorized actor, resolving disagreements through discussion. The six characterization dimensions are likewise developed and annotated by two authors. 8

Conclusion

We present SIExBench, the first benchmark for automated repair of information exposure defects, with 161 execution-verified instances from 64 Python and Java repositories. Across six repair systems and three LLMs, the best configuration resolves 39.1% of the instances, while 40.4% remain unresolved by all 18 configurations. Most failures involve incomplete exposure removal, highlighting the need to close every path through which the sensitive information can still be disclosed. We hope SIExBench provides a foundation for systematically evaluating and advancing future SIEx repair techniques. Data Availability We will publicly release SIExBench, along with the evaluation scripts and results, to support reproducibility and future research on SIEx repair. References [1] SWE agent Team. [n. d.]. GitHub - SWE-agent/mini-swe-agent: The 100 line AI agent that solves GitHub issues or helps you in your command line. Radically simple, no huge configs, no giant monorepo—but scores >74% on SWE-bench verified! — github.com. https://github.com/SWE-agent/mini-swe-agent. [Accessed 19-09-2026]. [2] Nikta Akbarpour, Mahdieh Sadat Benis, Fatemeh Hendijani Fard, Ali Ouni, and Mohamed Aymen Saied. 2025. Collaborative Agents for Automated Program Repair in Ruby. arXiv preprint arXiv:2511.03925 (2025). [3] Afsah Anwar, Aminollah Khormali, Hisham Alasmary, Sung J Choi, Saeed Salem, David Mohaisen, et al. 2020. Measuring the Cost of Software Vulnerabilities. EAI Endorsed Transactions on Security & Safety 7, 23 (2020). [4] Steven Arzt, Siegfried Rasthofer, Christian Fritz, Eric Bodden, Alexandre Bartel, Jacques Klein, Yves Le Traon, Damien Octeau, and Patrick McDaniel. 2014. Flowdroid: Precise context, flow, field, object-sensitive and lifecycle-aware taint analysis for android apps. ACM sigplan notices 49, 6 (2014), 259–269. [5] Alastair R. Beresford, Andrew Rice, Nicholas Skehin, and Ripduman Sohan. 2011. MockDroid: trading privacy for application functionality on smartphones. In Proceedings of the 12th Workshop on Mobile Computing Systems and Applications. ACM, Phoenix Arizona, 49–54. doi:10.1145/2184489.2184500 [6] Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. 2024. Repairagent: An autonomous, llm-based agent for program repair. arXiv preprint arXiv:2403.17134 (2024). [7] Yinzhi Cao, Yanick Fratantonio, Antonio Bianchi, Manuel Egele, Christopher Kruegel, Giovanni Vigna, and Yan Chen. 2015. EdgeMiner: Automatically Detecting Implicit Control Flow Transitions through the Android Framework.. In NDSS. https://www.yinzhicao.org/EdgeMiner/2015_ndss_edgeminer.pdf [8] Zhiyuan Chen. 2024. A comprehensive study of privacy leakage vulnerability in android app logs. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 2510–2513. [9] Zimin Chen, Yue Pan, Siyu Lu, Jiayi Xu, Claire Le Goues, Martin Monperrus, and He Ye. 2025. Prometheus: Unified Knowledge Graphs for Issue Resolution in Multilingual Codebases. arXiv preprint arXiv:2507.19942 (2025). [10] Andrea Continella, Yanick Fratantonio, Martina Lindorfer, Alessandro Puccetti, Ali Zand, Christopher Kruegel, and Giovanni Vigna. 2017. Obfuscation-Resilient Privacy Leak Detection for Mobile Apps Through Differential Analysis.. In NDSS, Vol. 17. 10–14722. https://sites.cs.ucsb.edu/~vigna/publications/2017_NDSS_Agrigento.pdf [11] DeepSeek-AI. 2025. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv:2512.02556 [cs.CL] https://arxiv.org/abs/2512.02556 [12] DeepSeek-AI. 2026. DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression. arXiv:2609.19969 [cs.CL] https://arxiv.org/abs/2609.19969 [13] Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Vijay Bharadwaj, Jeff Holm, Raja Aluri, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu, and Brad Kenstler. 2025. SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? arXiv:2509.16941 [cs.SE] https://arxiv.org/abs/2509.16941 [14] William Enck, Peter Gilbert, Seungyeop Han, Vasant Tendulkar, Byung-Gon Chun, Landon P. Cox, Jaeyeon Jung, Patrick McDaniel, and Anmol N. Sheth. 2014. TaintDroid: An Information-Flow Tracking System for Realtime Privacy Monitoring on Smartphones. ACM Transactions on Computer Systems 32, 2 (June 2014), 1–29. doi:10.1145/2619091 , Vol. 1, No. 1, Article . Publication date: November 2018.

20

Zhang et al.

[15] Pietro Ferrara, Omer Tripp, and Marco Pistoia. 2015. MorphDroid: Fine-grained Privacy Verification. In Proceedings of the 31st Annual Computer Security Applications Conference. ACM, Los Angeles CA USA, 371–380. doi:10.1145/2818000. 2818037 [16] Pengfei Gao, Zhao Tian, Xiangxin Meng, Xinchen Wang, Ruida Hu, Yuanan Xiao, Yizhou Liu, Zhao Zhang, Junjie Chen, Cuiyun Gao, et al. 2025. Trae agent: An llm-based agent for software engineering with test-time scaling. arXiv preprint arXiv:2507.23370 (2025). [17] Paul Gauthier and Aider-AI Contributors. 2024. Aider is ai pair programming in your terminal. https://aider.chat [18] Clint Gibler, Jonathan Crussell, Jeremy Erickson, and Hao Chen. 2012. Androidleaks: Automatically detecting potential privacy leaks in android applications on a large scale. In International conference on trust and trustworthy computing. Springer, 291–307. [19] Lianghong Guo, Wei Tao, Runhan Jiang, Yanlin Wang, Jiachi Chen, Xilin Liu, Yuchi Ma, Mingzhi Mao, Hongyu Zhang, and Zibin Zheng. 2025. Omnigirl: A multilingual and multimodal benchmark for github issue resolution. Proceedings of the ACM on Software Engineering 2, ISSTA (2025), 24–46. [20] Wenqi Huang, Charley Lee, Leonard Tng, and Serena Ge. 2026. DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks. arXiv:2607.07946 [cs.SE] https://arxiv.org/abs/2607.07946 [21] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770 (2023). [22] René Just, Darioush Jalali, and Michael D Ernst. 2014. Defects4J: A database of existing faults to enable controlled testing studies for Java programs. In Proceedings of the 2014 international symposium on software testing and analysis. 437–440. [23] Kyler Katz, Sara Moshtari, Ibrahim Mujhid, Mehdi Mirakhorli, and Derek Garcia. 2025. SIExVulTS: Sensitive Information Exposure Vulnerability Detection System using Transformer Models and Static Analysis. doi:10.48550/arXiv.2508.19472 arXiv:2508.19472 [cs] version: 1. [24] William Koch, Abdelberi Chaabane, Manuel Egele, William Robertson, and Engin Kirda. 2017. Semi-automated discovery of server-based information oversharing vulnerabilities in Android applications. In Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, Santa Barbara CA USA, 147–157. doi:10.1145/3092703.3092708 [25] Anh Le, Janus Varmarken, Simon Langhoff, Anastasia Shuba, Minas Gjoka, and Athina Markopoulou. 2015. AntMonitor: A System for Monitoring from Mobile Devices. In Proceedings of the 2015 ACM SIGCOMM Workshop on Crowdsourcing and Crowdsharing of Big (Internet) Data. ACM, London United Kingdom, 15–20. doi:10.1145/2787394.2787396 [26] Hwiwon Lee, Ziqi Zhang, Hanxiao Lu, and Lingming Zhang. 2025. SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks. arXiv preprint arXiv:2506.11791 (2025). [27] Fengjie Li, Jiajun Jiang, Jiajun Sun, and Hongyu Zhang. 2025. Hybrid automated program repair by combining large language models and program analysis. ACM Transactions on Software Engineering and Methodology 34, 7 (2025), 1–28. [28] Li Li, Alexandre Bartel, Tegawendé F Bissyandé, Jacques Klein, Yves Le Traon, Steven Arzt, Siegfried Rasthofer, Eric Bodden, Damien Octeau, and Patrick McDaniel. 2015. Iccta: Detecting inter-component privacy leaks in android apps. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Vol. 1. IEEE, 280–291. [29] Ying Li, Faysal Hossain Shezan, Bomin Wei, Gang Wang, and Yuan Tian. 2025. Sok: Towards effective automated vulnerability repair. arXiv preprint arXiv:2501.18820 (2025). [30] Fengyu Liu, Yuan Zhang, Tian Chen, Youkun Shi, Guangliang Yang, Zihan Lin, Min Yang, Junyao He, and Qi Li. 2025. Detecting Taint-Style Vulnerabilities in Microservice-Structured Web Applications. In 2025 IEEE Symposium on Security and Privacy (SP). 972–990. doi:10.1109/SP61157.2025.00137 [31] Yizhou Liu, Pengfei Gao, Xinchen Wang, Jie Liu, Yexuan Shi, Zhao Zhang, and Chao Peng. 2024. Marscode agent: Ai-native automated bug fixing. arXiv preprint arXiv:2409.00899 (2024). [32] Cybercrime Magazine. 2020. Cybercrime To Cost The World $10.5 Trillion Annually By 2025. https:// cybersecurityventures.com/hackerpocalypse-cybercrime-report-2016. [Accessed 05-11-2025]. [33] Björn Mathis, Vitalii Avdiienko, Ezekiel O. Soremekun, Marcel Böhme, and Andreas Zeller. 2017. Detecting information flow by mutating input data. In 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 263–273. https://ieeexplore.ieee.org/abstract/document/8115639/ [34] Rahil Mehta, Pushpak Katkhede, and Manish Motwani. 2026. Defects4REST: A Benchmark of Real-World Defects to Enable Controlled Testing and Debugging Studies for REST APIs. In Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering. 1597–1608. [35] Xiangxin Meng, Zexiong Ma, Pengfei Gao, and Chao Peng. 2026. LLM-based Agents for Automated Bug Fixing: How Far Are We?. In Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering. 855–867. [36] MITRE. [n. d.]. CWE-200: Exposure of Sensitive Information to an Unauthorized Actor. https://cwe.mitre.org/data/ definitions/200.html. [Accessed 30-09-2026].

, Vol. 1, No. 1, Article . Publication date: November 2018.

SIExBench

21

[37] OpenAI. [n. d.]. GitHub - openai/codex: Lightweight coding agent that runs in your terminal — github.com. https: //github.com/openai/codex. [Accessed 19-09-2026]. [38] OpenAI. 2026. GPT-5.6 Luna. https://developers.openai.com/api/docs/models/gpt-5.6-luna. [Accessed 19-09-2026]. [39] OpenAI. 2026. GPT-6 Luna. https://openai.com/index/introducing-gpt-6-sol-and-luna/. [Accessed 24-09-2026]. [40] Shuyin Ouyang, Jie M Zhang, Zeyu Sun, and Albert Merono Penuela. 2025. Knowledge-enhanced program repair for data science code. arXiv preprint arXiv:2502.09771 (2025). [41] Adamenko Pavel, Ivanov Mikhail, Aidar Valeev, Rodion Levichev, Pavel Zadorozhny, Ivan Lopatin, Dmitrii Babaev, Alena Fenogenova, and Valentin Malykh. 2025. SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Ivan Habernal, Peter Schulam, and Jörg Tiedemann (Eds.). Association for Computational Linguistics, Suzhou, China, 440–452. doi:10.18653/v1/2025.emnlp-demos.30 [42] Yun Peng, Shuzheng Gao, Cuiyun Gao, Yintong Huo, and Michael Lyu. 2024. Domain Knowledge Matters: Improving Prompts with Fix Templates for Repairing Python Type Errors. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (Lisbon, Portugal) (ICSE ’24). Association for Computing Machinery, New York, NY, USA, Article 4, 13 pages. doi:10.1145/3597503.3608132 [43] Qwen Team. 2026. Qwen3.5: Towards Native Multimodal Agents. https://qwen.ai/blog?id=qwen3.5 [44] Qwen Team. 2026. Qwen3.8-Flash. https://www.qianwenai.com/models/qwen3.8-flash. [Accessed 19-09-2026]. [45] Jingjing Ren, Ashwin Rao, Martina Lindorfer, Arnaud Legout, and David Choffnes. 2016. Recon: Revealing and controlling pii leaks in mobile network traffic. In Proceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services. 361–374. [46] Haifeng Ruan, Yuntong Zhang, and Abhik Roychoudhury. 2025. SpecRover: Code Intent Extraction via LLMs. In Proceedings of the IEEE/ACM 47th International Conference on Software Engineering (Ottawa, Ontario, Canada) (ICSE ’25). IEEE Press, 963–974. doi:10.1109/ICSE55347.2025.00080 [47] Yihang Song and Urs Hengartner. 2015. PrivacyGuard: A VPN-based Platform to Detect Information Leakage on Android Devices. In Proceedings of the 5th Annual ACM CCS Workshop on Security and Privacy in Smartphones and Mobile Devices. ACM, Denver Colorado USA, 15–26. doi:10.1145/2808117.2808120 [48] Mingshen Sun, Tao Wei, and John C.S. Lui. 2016. TaintART: A Practical Multi-level Information-Flow Tracking System for Android RunTime. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. ACM, Vienna Austria, 331–342. doi:10.1145/2976749.2978343 [49] Wei Tao, Yucheng Zhou, Yanlin Wang, Wenqiang Zhang, Hongyu Zhang, and Yu Cheng. 2024. Magis: Llm-based multi-agent framework for github issue resolution. Advances in Neural Information Processing Systems 37 (2024), 51963–51993. [50] ByteDance Seed Team. [n. d.]. GitHub - multi-swe-bench/MagentLess: Agentless for multi-swe-bench. — github.com. https://github.com/multi-swe-bench/MagentLess. [Accessed 19-09-2026]. [51] ByteDance Seed Team. [n. d.]. GitHub - multi-swe-bench/MopenHands: Multi-SWE-Bench infer with OpenHands. — github.com. https://github.com/multi-swe-bench/MopenHands. [Accessed 19-09-2026]. [52] ByteDance Seed Team. [n. d.]. GitHub - multi-swe-bench/MSWE-agent: Swe-agent for multi-swe-bench. — github.com. https://github.com/multi-swe-bench/MSWE-agent. [Accessed 19-09-2026]. [53] OpenCode Team. [n. d.]. GitHub - anomalyco/opencode: The open source coding agent. — github.com. https: //github.com/anomalyco/opencode. [Accessed 19-09-2026]. [54] Omer Tripp and Julia Rubin. 2014. A bayesian approach to privacy enforcement in smartphones. In 23rd USENIX Security Symposium (USENIX Security 14). 175–190. [55] Bin Wang, Hui Li, Liyang Zhang, Qijia Zhuang, Ao Yang, Dong Zhang, Xijun Luo, and Bing Lin. 2025. Argus: A Multi-Agent Sensitive Information Leakage Detection Framework Based on Hierarchical Reference Relationships. doi:10.1145/3744916.3773208 arXiv:2512.08326 [cs]. [56] Jian Wang, Xiaofei Xie, Qiang Hu, Shangqing Liu, Jiongchi Yu, Jiaolong Kong, and Yi Li. 2025. Defects4C: Benchmarking Large Language Model Repair Capability with C/C++ Bugs. doi:10.48550/arXiv.2510.11059 arXiv:2510.11059 [cs]. [57] Xiaolei Wang, Andrea Continella, Yuexiang Yang, Yongzhong He, and Sencun Zhu. 2019. LeakDoctor: Toward Automatically Diagnosing Privacy Leaks in Mobile Applications. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 1 (March 2019), 1–25. doi:10.1145/3314415 [58] Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. 2024. Openhands: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407.16741 (2024). [59] Xu Wang, Mingming Zhang, Xiangxin Meng, Jian Zhang, Yang Liu, and Chunming Hu. 2025. Element-Based Automated DNN Repair with Fine-Tuned Masked Language Model. Proceedings of the ACM on Software Engineering 2, FSE (2025), 106–129.

, Vol. 1, No. 1, Article . Publication date: November 2018.

22

Zhang et al.

[60] Fengguo Wei, Sankardas Roy, Xinming Ou, and Robby. 2018. Amandroid: A Precise and General Inter-component Data Flow Analysis Framework for Security Vetting of Android Apps. ACM Transactions on Privacy and Security 21, 3 (Aug. 2018), 1–32. doi:10.1145/3183575 [61] Zichao Wei, Jun Zeng, Ming Wen, Zeliang Yu, Kai Cheng, Yiding Zhu, Jingyi Guo, Shiqi Zhou, Le Yin, Xiaodong Su, and Zhechao Ma. 2025. PATCHEVAL: A New Benchmark for Evaluating LLMs on Patching Real-World Vulnerabilities. doi:10.48550/arXiv.2511.11019 arXiv:2511.11019 [cs]. [62] Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2025. Demystifying llm-based software engineering agents. Proceedings of the ACM on Software Engineering 2, FSE (2025), 801–824. [63] Chunqiu Steven Xia, Zhe Wang, Yan Yang, Yuxiang Wei, and Lingming Zhang. 2025. Live-swe-agent: Can software engineering agents self-evolve on the fly? arXiv preprint arXiv:2511.13646 (2025). [64] Chunqiu Steven Xia and Lingming Zhang. 2024. Automated program repair via conversation: Fixing 162 out of 337 bugs for $0.42 each using ChatGPT. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 819–831. [65] Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chenchen Ling, et al. 2026. Deepseek-v4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348 (2026). [66] Boyang Yang, Zijian Cai, Fengling Liu, Bach Le, Lingming Zhang, Tegawendé F Bissyandé, Yang Liu, and Haoye Tian. 2025. A survey of LLM-based automated program repair: Taxonomies, design paradigms, and applications. arXiv preprint arXiv:2506.23749 (2025). [67] Boyang Yang, Haoye Tian, Jiadong Ren, Hongyu Zhang, Jacques Klein, Tegawendé F Bissyandé, Claire Le Goues, and Shunfu Jin. 2026. Morepair: Teaching llms to repair code via multi-objective fine-tuning. ACM Transactions on Software Engineering and Methodology 35, 2 (2026), 1–38. [68] John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems 37 (2024), 50528–50652. [69] John Yang, Carlos E Jimenez, Alex L Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R Narasimhan, et al. 2024. Swe-bench multimodal: Do ai systems generalize to visual software domains? arXiv preprint arXiv:2410.03859 (2024). [70] John Yang, Kilian Lieret, Carlos E. Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. 2025. SWE-smith: Scaling Data for Software Engineering Agents. arXiv:2504.21798 [cs.SE] https://arxiv.org/abs/2504.21798 [71] He Ye, Aidan ZH Yang, Chang Hu, Yanlin Wang, Tao Zhang, and Claire Le Goues. 2025. Adverintent-agent: Adversarial reasoning for repair based on inferred program intent. Proceedings of the ACM on Software Engineering 2, ISSTA (2025), 1398–1420. [72] Xin Yin, Chao Ni, Shaohua Wang, Zhenhao Li, Limin Zeng, and Xiaohu Yang. 2024. Thinkrepair: Self-directed automated program repair. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1274–1286. [73] He Yang Yuan, Xin Wang, Kundi Yao, An Ran Chen, Zishuo Ding, and Zhenhao Li. 2026. Towards Secure Logging: Characterizing and Benchmarking Logging Code Security Issues with LLMs. Proceedings of the ACM on Software Engineering 3, FSE (2026), 745–765. [74] Z.ai. 2026. GLM-5.3-Flash. https://z.ai/blog/glm-5.3-flash. [Accessed 19-09-2026]. [75] Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Shulin Xin, Linhao Zhang, Qi Liu, Li Aoyan, Lu Chen, Xiaojian Zhong, et al. 2026. Multi-swe-bench: A multilingual benchmark for issue resolving. Advances in Neural Information Processing Systems 38 (2026). [76] Jiayi Zhang, Kai Huang, Jian Zhang, Yang Liu, and Chunyang Chen. 2025. Repair Ingredients Are All You Need: Improving Large Language Model-Based Program Repair via Repair Ingredients Search. arXiv preprint arXiv:2506.23100 (2025). [77] Junbin Zhang, Yingying Wang, Lina Qiu, and Julia Rubin. 2022. Analyzing Android Taint Analysis Tools: FlowDroid, Amandroid, and DroidSafe. IEEE Transactions on Software Engineering 48, 10 (Oct. 2022), 4014–4040. doi:10.1109/TSE. 2021.3109563 [78] Quanjun Zhang, Chunrong Fang, Tongke Zhang, Bowen Yu, Weisong Sun, and Zhenyu Chen. 2023. Gamma: Revisiting template-based automated program repair via mask prediction. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 535–547. [79] Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. Autocoderover: Autonomous program improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1592–1604.

, Vol. 1, No. 1, Article . Publication date: November 2018.

SIExBench

23

[80] Dewu Zheng, Yanlin Wang, Ensheng Shi, Xilin Liu, Yuchi Ma, Hongyu Zhang, and Zibin Zheng. 2025. Top General Performance = Top Domain Performance? DomainCodeBench: A Multi-domain Code Generation Benchmark. doi:10. 48550/arXiv.2412.18573 arXiv:2412.18573 [cs]. [81] Zhiqing Zhong, Jiaming Huang, and Pinjia He. 2025. BackportBench: A Multilingual Benchmark for Automated Backporting of Patches. arXiv. doi:10.48550/arXiv.2512.01396 arXiv:2512.01396 [cs]. [82] Yuxuan Zhu, Antony Kellermann, Dylan Bowman, Philip Li, Akul Gupta, Adarsh Danda, Richard Fang, Conner Jensen, Eric Ihli, Jason Benn, Jet Geronimo, Avi Dhir, Sudhit Rao, Kaicheng Yu, Twm Stone, and Daniel Kang. [n. d.]. CVE-Bench: A Benchmark for AI Agents’ Ability to Exploit Real-World Web Application Vulnerabilities. In ICML.

Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009

, Vol. 1, No. 1, Article . Publication date: November 2018.

Related documents

• CVE-2026-67080 — MEDIUM 6.5#1493807
• Outcome Monitors: Recovery Affordances for Silent Tool Failures#471320
• CVE-2026-67104 — MEDIUM 5.3#1167480
• CVE-2026-67077 — MEDIUM 4.2#1493805
• Information recovery problems#712130