SWE-NFI: Studying and Benchmarking Coding Agents for Non-Functional Improvements Pengyu Xue
He Yang Yuan
Xin Wang
[email protected] York University Toronto, Ontario, Canada
[email protected] York University Toronto, Ontario, Canada
[email protected] The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, Guangdong, China
Junkai Chen
Haonan Zhang
Boyuan Chen
[email protected] Singapore Management University Singapore, Singapore
[email protected] University of Waterloo Waterloo, Ontario, Canada
[email protected] Independent Researcher Canada
Zishuo Ding
Zhenhao Li
Weiyi Shang
[email protected] The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, Guangdong, China
[email protected] York University Toronto, Ontario, Canada
[email protected] University of Waterloo Waterloo, Ontario, Canada
Abstract Although coding agents have achieved impressive performance on correctness-oriented benchmarks, their ability to make behaviorpreserving non-functional improvements (NFIs) remains underexplored. In real-world software development, developers continuously improve software quality without changing observable behavior, yet existing benchmarks primarily evaluate functional correctness and provide limited support for assessing these non-functional improvements. In this paper, we present SWE-NFI, a benchmark for evaluating coding agents on NFIs beyond functional correctness. Our benchmark contains 188 tasks constructed from real merged pull requests in open-source Python projects. We operationalize developeroriented NFIs into 92 executable rules and develop a comprehensive evaluation suite that combines functional correctness testing with rule-based NFI evaluation. We evaluate state-of-the-art commercial and open-source coding agents. Although the best-performing agent achieves a 70.0% functional correctness rate, all evaluated agents generally fall short of human developers in overall NFI capability. The gap is particularly evident for structural code improvements, where agents’ NFI scores range from 0.0 to 1.3, compared with 1.5 for the human reference. Our benchmark and findings provide a reproducible foundation for evaluating and advancing coding agents beyond functional correctness.
1
Introduction
Coding agents powered by large language models have become increasingly integrated into modern software engineering workflows. These agents are capable of assisting developers in a wide range of functionality-oriented software engineering tasks, such as resolving GitHub issues, fixing bugs, implementing feature requests [7, 9, 11, 31, 34, 72, 74, 85], and supporting emerging vibe coding workflows [10, 55].
In practice, real-world software development and maintenance involve far more than implementing new functionality or fixing bugs. Developers continuously make non-functional improvements (NFIs) to existing code, such as refining documentation [29], improving error handling [47, 61] and security [45], modernizing library usage [66], enhancing coding style [12, 82], constructing system observability [19, 35–38, 40, 67, 87, 88], and simplifying program structure [52]. Although these changes typically preserve the original functionality, they play a critical role in improving software maintainability, readability, and long-term evolution [6, 46]. Consequently, NFIs constitute an indispensable part of software maintenance. Despite the importance of NFIs, existing benchmarks for coding agents predominantly evaluate functionality-oriented tasks. Representative benchmarks typically evaluate the agents by executing unit tests or comparing the generated code against expected behavior [5, 13, 14, 31, 42]. Such evaluations have substantially advanced the development of coding agents [28, 41], but provide limited insights into their ability to perform NFIs. More importantly, functional correctness and NFIs capture complementary aspects of software quality. A functionally correct solution may still exhibit poor documentation, inadequate error handling, outdated library usage, or unnecessarily complex program structure. Since these issues affect maintainability and long-term evolution, evaluating coding agents solely by functional correctness gives an incomplete picture of practical usefulness. To bridge this gap, we present SWE-NFI, a benchmark for systematically evaluating coding agents on NFIs beyond functional correctness. Following the five high-level aspects of non-functional improvements discussed in SWE-IF [86], we derive 92 executable rules from prior studies, real-world issues, and coding guidelines. Based on these rules, we construct 188 benchmark tasks from real merged GitHub pull requests, including 145 single-file tasks and 43 multi-file tasks. Each task is accompanied by a comprehensive
ArXiv, 2026, Online
Pengyu Xue, He Yang Yuan, Xin Wang, Junkai Chen, Haonan Zhang, Boyuan Chen, Zishuo Ding, Zhenhao Li, and Weiyi Shang
evaluation suite consisting of manually implemented functional test cases and rule-based NFI evaluation. Since the benchmark is constructed entirely from real merged pull requests, it captures the non-functional improvements that developers routinely perform during software development and maintenance. We systematically evaluate state-of-the-art commercial and opensource coding agents using SWE-NFI. Our results reveal a substantial gap between current coding agents and human developers. Although the best-performing agent achieves a 70.0% functional correctness rate, all evaluated agents mostly fall short of the corresponding human reference across multiple NFI aspects, especially for logic patterns, where the average improvement score ranges from 0.0 to 1.3 across all evaluated agents, compared with 1.5 for the human reference. We further find that multi-file tasks remain considerably more challenging than single-file tasks, while repeated executions exhibit relatively stable NFI performance. In addition, we investigate the consistency of coding agents across repeated executions of the same task, as well as the practical costs of making non-functional improvements in terms of execution time and token consumption. These findings suggest that, while coding agents have made substantial progress in generating functionally correct code, enabling them to perform high-quality non-functional improvements remains an important open challenge. Our main contributions are summarized as follows: • We operationalize NFIs into an executable evaluation framework. Specifically, following the five high-level NFI aspects proposed in prior work, we systematically derive 92 executable rules from multiple complementary sources, including prior studies, real-world open-source development practices, and coding guidelines. • We construct SWE-NFI, a benchmark from real merged GitHub pull requests. The benchmark contains 188 NFI tasks, including 145 single-file and 43 multi-file. Each task is accompanied by a comprehensive evaluation suite consisting of manually implemented functional test cases and rule-based NFI evaluation. • We conduct a comprehensive empirical study of state-of-theart commercial and open-source coding agents on SWE-NFI. Our study provides a systematic evaluation of coding agents on NFIs beyond functional correctness and discusses insights into their effectiveness, robustness, and practical deployment costs.
2 Related Work 2.1 Functional Code Generation Benchmarks In the early days of large language models, code generation benchmarks, such as HumanEval [14] and MBPP [5], generally consist of method-level, self-contained programming problems with several test cases for functional evaluation. Motivated by these works, a series of follow-up benchmarks has emerged, improving upon them in various respects, such as task granularity (e.g., ClassEval [21], RepoEval [83]), language coverage (e.g., MultiPL-E [8], MBXP [4]), domain diversity (e.g., BigCodeBench [89], DS-1000 [32]), and test enhancement (e.g., EvalPlus [42]). In particular, SWE-bench [31] advances the evaluation of code generation to a new stage: given an issue description, the model
is required to edit multiple files to resolve it, with the resulting patch verified against comprehensive test suites. This work opens a new direction that more closely mirrors real-world development workflows, and has become the prevailing standard for evaluating the coding capabilities of LLMs and agents. It also inspires a number of works that extend and refine the original SWE-bench (e.g., feature addition [34], multilingual [81], and multimodal [75] extensions, among many others [17, 57, 79, 84]). Very recently, we have also observed that the generation of entire repositories or complete software systems [18, 27, 76] is nascent but developing rapidly. For example, ProgramBench [76] has attracted considerable attention from the community, which asks LLM agents to reproduce program executables and evaluate their function with agentic black-box fuzzing. Compared with these works focusing on functional evaluation, SWE-NFI proposes five non-functional aspects for code generation evaluation, with rules mined from the open-source community, constituting a complementary and distinctive contribution.
2.2
Code Generation Evaluation Beyond Correctness
In addition to functional correctness, a growing number of studies examine other aspects of LLM-generated code. Among these, code efficiency has received considerable attention [20, 26, 30, 39, 43, 44, 54, 56, 70]. For example, EffiBench [30] collects efficiencycritical problems from LeetCode and compares the running time of LLM-generated solutions against human-written canonical ones. COFFE [54] uses computationally intensive test inputs and the number of CPU instructions for a more discriminative and precise evaluation of code efficiency. The security of code produced by LLMs and agents is another dimension beyond functionality [10, 51, 53, 60]. Pearce et al. [51] pioneer this direction by auditing GitHub Copilot completions and find that approximately 40% of them may be vulnerable. SecureVibeBench [10] reconstructs scenarios in which developers introduce security issues and uses them to evaluate agents. Beyond efficiency and security, more specialized aspects of code have been studied, such as portability [23, 66, 71], compliance [73], and fairness [22]. In contrast to the aforementioned research, which typically focuses on evaluating classical code properties such as efficiency and security, SWE-NFI is grounded in real-world coding behaviors and assesses developer-oriented dimensions mined from large-scale open-source development. Moreover, to achieve verifiable and deterministic evaluation, we codify these abstract, descriptive criteria into reproducible, deterministic rules, rather than relying on less reliable measurements such as manual assessment or LLM-as-a-judge, which significantly distinguishes SWE-NFI from similar work [86].
2.3
LLM-Based Agents in Software Engineering
LLM-based agents have shown great capabilities in various software engineering tasks, including program repair, vulnerability discovery, and software refactoring [28, 41]. Bouzenia et al. [7] present the first autonomous LLM agent for program repair, namely RepairAgent. Built around tool use, this work designs a well-crafted tool set and employs dynamic templates together with a state machine to guide tool invocation for the LLM. AutoCodeRover [85] utilizes
SWE-NFI : Studying and Benchmarking Coding Agents for Non-Functional Improvements
Stage 1: NFI Rules Academic Literature
Coding Documents
GitHub Issues
ArXiv, 2026, Online
Stage 2: Task Construction
Stage 3: Evaluation Suite
Pull Request Collection
Functional Correctness Evaluation
PR Cleaning
Keyword Matching
Manual Rule Analysis
NFI Rule Evaluation NFI Rule-base Filtering
92 NFI Rules
Data Cleaning
Human Validation
Result Analysis Deterministic
Threshold
Construct Task Description
Cross Validation
NFI Qualification
NFI Scoring
RQ1: Performance Comparison RQ2: Scope Impact
NFI Task Dataset (188 tasks)
RQ3: Consistency Analysis
RQ4: Practical Cost
Figure 1: Overview of SWE-NFI. the reasoning ability of LLMs to retrieve code, localize relevant elements, and generate patches in an iterative manner. SWE-agent [74] proposes the idea of an Agent-Computer Interface, which transforms common computer operations into agent-friendly instructions to enable effective iteration between action and feedback. Building on SWE-agent, Abramovich et al. [2] design interactive agent tools and a context summarization strategy tailored for vulnerability discovery tasks. RefAgent [50] introduces a multi-agent framework for automatic code refactoring, where several specialized agents are responsible for roles such as planning and testing during the refactoring process. Beyond academia, efforts from industry and the open-source community, exemplified by Claude Code [15], Codex [48], and OpenHands [68], have also joined in releasing and iterating on software engineering agents. As these agents become more capable, the need for a multi-dimensional evaluation beyond functionality alone becomes increasingly pressing, which is precisely what SWENFI targets.
3 SWE-NFI Benchmark Framework 3.1 Benchmark Overview SWE-NFI is designed to evaluate whether coding agents can make developer-oriented non-functional improvements beyond functional correctness. As shown in Figure 1, SWE-NFI consists of three main stages: (1): NFI Rules: We operationalize five aspects of NFIs into 92 executable rules; (2) Task Construction: We construct 188 benchmark tasks from real-world merged GitHub pull requests; (3) Evaluation Suite: We implement an evaluation suite that combines functional correctness tests with NFI rule-based evaluation. We illustrate each component in detail below.
3.2
NFI Rules
Non-Functional Improvement Aspects. Non-functional improvements cover a broad range of software quality concerns, such as maintainability, performance, security, reliability, and usability. In this work, we focus on developer-oriented, behavior-preserving
improvements that are routinely performed during software maintenance. Specifically, following prior study [86], we study five representative aspects: Documentation, Error Handling, Library Constraints, Coding Style, and Logic Patterns. Although these aspects provide a high-level characterization of non-functional improvements, they do not define executable evaluation criteria. Therefore, we further operationalize them into concrete rules that can be automatically evaluated. In this study, we focus on Python because it provides mature static-analysis tools and well-established coding standards, enabling reproducible benchmark construction and deterministic evaluation. Rule Derivation. We derive rules from three complementary sources following a similar process to prior studies [69, 77, 80]: prior studies, open-source software artifacts, and Python coding guidelines. First, we collect studies related to software maintainability and non-functional improvements from major software engineering venues (e.g., ICSE, FSE, ASE) and AI venues (e.g., AAAI, NeurIPS, ACL), published between 2020 and 2025. We retrieve candidate papers using aspect-specific keywords associated with the five NFI aspects, (eg., "static analysis", "linter", "code quality", "refactoring", "code smells", etc.) resulting Coding Style with 95 papers, Logic Patterns with 101 papers, Documentation with 78 papers, Error Handling with 97 papers, and Library Constraints with 39 papers. Following an in-depth manual screening of the literature, we narrowed the selection to 35 highly relevant papers that provided the theoretical foundation for our rule derivation. Second, we collect real-world developer discussions from issues in widely used open-source Python projects. We start from the Top-100 starred Python projects in GitHub-Ranking and remove repositories that correspond to tutorials, demos, educational materials, instructions, or curated lists. We then use the GitHub Issue Search API to run aspect-specific keyword queries over the remaining repositories, restricting the search to issues created between 2020 and 2025 and removing duplicates by URL. To focus on discussions with clear developer engagement, we keep only issues with at least 10 reactions and comments in total. This procedure
ArXiv, 2026, Online
Pengyu Xue, He Yang Yuan, Xin Wang, Junkai Chen, Haonan Zhang, Boyuan Chen, Zishuo Ding, Zhenhao Li, and Weiyi Shang
Table 1: Summary of the 92 executable rules for NonFunctional Improvements evaluation. Deterministic
Threshold-based
Aspect
Total Required
Prohibited
Quantitative
Documentation (Doc) Error Handling (EH) Library Constraints (LC) Coding Style (CS) Logic Patterns (LP)
12 4 1 7 0
4 6 19 10 8
3 2 2 8 6
19 12 22 25 14
Total
24
47
21
92
yields 326 issues across five aspects: Coding Style (54), Logic Patterns (21), Documentation (104), Error Handling (31), and Library Constraints (116). These issues, together with the literature and coding guidelines, inform the candidate NFI rules. Third, we incorporate coding guidelines that define recommended Python development practices. We systematically categorize and synthesize these guidelines into rules corresponding to our five evaluation aspects. Our rule set integrates standards from PEP 8 [65], PEP 3134 [78], PEP 585 [33], PEP 257 [24], and PEP 484 [58]. We also incorporate best practices from reputable technical sources, including the Tufts University coding standards [64], the pyupgrade documentation [62], and industry-standard metrics [59]. Each deterministic and threshold-based rule is derived from these official documentations and specialized technical resources to ensure that our benchmark accurately reflects established professional coding standards. Manual Analysis and Refinement. We then manually analyze the collected artifacts to identify concrete code improvement rules that can be mapped to the five NFI aspects. During the analysis, we focus on rules that satisfy two requirements. First, they should represent practices that frequently appear in prior studies or realworld software improvements. Second, they should be measurable through static analysis without requiring evaluation-time human annotation or LLM-based judgment. The first author initially extracted candidate rules from the collected data. The extracted rules were then reviewed and refined by a second author. Disagreements were resolved through discussion until a consensus was reached. Rules Summary. Through this process, we derive 92 executable rules across the five NFI aspects. These rules transform high-level NFI aspects into concrete and measurable evaluation targets. To provide a clear overview of our evaluation mechanism, Table 1 reports the number of rules in each aspect, and Table 2 summarizes representative rules for each NFI aspect, categorized by their type of measurement and requirement. Due to space constraints, we list a portion of example rules in the table; the full list of all 92 rules, including specific threshold parameters and static analysis configurations, is provided in our replication package [1]. NFI Rules Overview. Through this process, we derive 92 executable rules across the five NFI aspects. We further organize these rules into two categories according to their measurement characteristics: Deterministic Rules and Threshold-based Rules:
• Deterministic rules correspond to rules that can be directly verified using static analysis without additional calibration. These rules are further divided into Required Rules and Prohibited Rules. Required rules describe coding practices that should be present in the generated code (e.g., function names should follow snake_case), whereas prohibited rules describe undesirable patterns that should be avoided (e.g., wildcard imports should not be used). Each deterministic rule is converted into a binary indicator (i.e., 1 if the code complies with the rule and 0 otherwise). • Threshold-based Rules quantify code quality using numerical thresholds. Unlike deterministic rules, these rules require comparing a measured value against a predefined threshold. Examples include maximum function length, maximum nesting depth, cyclomatic complexity, and the number of parameters per function. These thresholds are configurable and can be adjusted for different projects or coding standards. In our benchmark, we adopt threshold values based on widely accepted Python coding guidelines and static analysis recommendations to reflect common software engineering practices. Non-Functional Improvements Quantification. For each NFI aspect 𝑝, let 𝑃𝑝 (𝑐) denote the number of rules satisfied by code version 𝑐 under aspect 𝑝, and let 𝑇𝑝 denote the total number of rules for aspect 𝑝. We first define the normalized rule score of code version 𝑐 as: 𝑆𝑝 (𝑐) =
𝑃𝑝 (𝑐) . 𝑇𝑝
(1)
Therefore, 𝑆𝑝 (𝑐) ∈ [0, 1]. Given two code versions 𝑐 before and 𝑐 after , we define the normalized NFI improvement under aspect 𝑝 as: 𝐼𝑝 (𝑐 before, 𝑐 after ) = 𝑆𝑝 (𝑐 after ) − 𝑆𝑝 (𝑐 before ) =
𝑃𝑝 (𝑐 after ) − 𝑃𝑝 (𝑐 before ) . 𝑇𝑝
(2)
A positive value indicates an improvement under NFI aspect 𝑝, zero indicates no measurable change, and a negative value indicates a regression. For benchmark instance 𝑖, let 𝑐 𝑖before denote the original code and let 𝑐 𝑖after (𝑥) denote the completed code produced by source 𝑥. For dataset instances, 𝑥 = ℎ denotes the human reference implementation; during agent evaluation, 𝑥 = 𝑎 denotes the output generated by agent 𝑎. We compute the NFI improvement of source 𝑥 under aspect 𝑝 as: 𝐼𝑖,𝑝 (𝑥) = 𝐼𝑝 𝑐 𝑖before, 𝑐 𝑖after (𝑥) . (3) We report all NFI improvement values after scaling 𝐼𝑖,𝑝 (𝑥) to a percentage scale. Here, 𝐼𝑖,𝑝 (ℎ) denotes the human reference improvement, and 𝐼𝑖,𝑝 (𝑎) denotes the improvement achieved by agent 𝑎.
3.3
Benchmark Dataset Construction
We construct the dataset of SWE-NFI through a multi-stage pipeline that progressively transforms raw pull request data into benchmark tasks. Starting from real-world merged pull requests, we first collect candidate records from production-grade Python repositories and identify developer-driven non-functional improvements using the
SWE-NFI : Studying and Benchmarking Coding Agents for Non-Functional Improvements
ArXiv, 2026, Online
Table 2: Rules catalogue for the five NFI aspects. Aspect
Measurement
Requirement
Example / Threshold
Explanation
Documentation
Deterministic Deterministic Threshold-based
Required Prohibited Quantitative
Modules, classes, and methods No empty docstrings, no TODOs Docstring/Type hint coverage ≥ 80%
Docstring presence enforcement Content quality verification Documentation coverage targets
Error Handling
Deterministic Deterministic Threshold-based
Required Prohibited Quantitative
Mandatory ‘raise X from e’ No bare ‘except:’, no silent drops Try-block LOC ratio ≤ 0.5, handlers ≤ 3
Exception chaining enforcement Safety and exception robustness Exception scope management
Library Constraints
Deterministic Deterministic Threshold-based
Required Prohibited Quantitative
Use pathlib, collections.abc No tf.Session, np.bool, .ix[] Deprecated API count ≤ 2 per file
Modern API migration Legacy API prevention API usage frequency monitoring
Coding Style
Deterministic Deterministic Threshold-based
Required Prohibited Quantitative
snake_case, PascalCase, Line break after operators, UPPER_CASE No trailing whitespace, no mixed tabs Line length ≤ 100, magic numbers ≤ 5
Naming conventions enforcement Formatting standards maintenance Complexity & style density control
Logic Patterns
Deterministic Deterministic Threshold-based
Required Prohibited Quantitative
Assertions for critical logic No mutable default args, no ‘while True’ Cyclomatic complexity ≤ 10, depth ≤ 3
State validation requirement Anti-pattern avoidance Complexity and logic density
aspect-specific rule checkers introduced in Section 3.2. We then apply automatic filtering and human validation to remove noisy, ambiguous, and unsuitable samples. Next, we manually construct task descriptions that preserve the original development intent while preventing benchmark information leakage. Overall, SWENFI contains 188 benchmark tasks, including 145 single-file tasks and 43 multi-file tasks, constructed from real-world merged pull requests. Step 1: Real-world Pull Request Collection. We construct SWENFI from real-world merged pull requests in production-grade Python repositories. We first collect repositories using the GitHub Search API and filter them by programming language, repository status, and popularity. We exclude forks, archived repositories, tutorials, demos, educational projects, and curated lists, yielding 100 production-grade Python repositories, each with at least 45,000 stars (median 66,254). We then collect merged pull requests from 2020 to 2025. To focus on developer-driven non-functional improvements, we remove bot-generated pull requests, dependency updates, release pull requests, version bumps, and revert commits. We further apply aspect-specific keyword matching to increase the likelihood of collecting modifications related to the five NFI aspects. The basic unit of data collection is a PR-file pair, where each record corresponds to one merged pull request and one modified Python file. After deduplication, we obtain 226,955 raw records. Step 2: NFI Rule-Based Filtering. We next identify records with measurable non-functional improvements by computing the NFI improvement score using Eq. (3). We retain only records satisfying 𝐼𝑖,𝑝 (ℎ) > 0. This process yields 6,775 positive-improvement records, 211,914 records without measurable changes, 7,566 regressions, and 700 scorer errors. Only the positive-improvement records are retained for subsequent benchmark construction. Step 3: Data Cleaning and Human Validation. We further clean and validate the remaining records to construct high-quality benchmark candidates. We first apply automatic filters, including repository blocklists, diff size and file length constraints, perrepository caps, and SHA-256 deduplication, resulting in 1,215 candidate records for manual review.
Two authors independently review each candidate to verify that it is focused and represents a genuine non-functional improvement. We further exclude instances requiring substantial external context, large-scale migrations, public API changes, or semantic changes. Disagreements are resolved through discussion until consensus is reached (𝜅 = 0.71). This process yields 484 validated instances for subsequent benchmark construction. Step 4: Task Construction. For each validated instance, we manually construct a natural-language task description that converts the original pull request into a developer-facing request. To preserve benchmark integrity, the description omits the final aspect, executable rules, code patches, and the reference implementation while preserving the original development intent. One author drafts each task description based on the pull request context and code changes, and a second author verifies and refines it for clarity, self-containment, and fidelity. Disagreements are resolved through discussion until consensus is reached. The benchmark contains both single-file and multi-file tasks. Single-file tasks modify one file, whereas multi-file tasks require coordinated modifications across related files sharing a common improvement objective. After a final selection that discards non-self-contained or untestable instances, we obtain the final 188 tasks.
3.4
Evaluation Suite
The evaluation suite of SWE-NFI consists of two complementary components: functional correctness and NFI rule-score evaluation. We first verify whether the generated code preserves the intended functionality. NFI evaluation is performed only on valid, functionally correct outputs. The entire process is fully automated and does not rely on LLM-as-a-judge or human annotation. Functional Correctness Evaluation. For each benchmark task, we manually implement unit test cases to verify functional correctness. One author drafts the tests, and a second author verifies and refines them to ensure comprehensive coverage of the intended functionality. Disagreements are resolved through discussion until consensus is reached. We validate the evaluation suite by executing all test cases on the human reference implementation, where every benchmark instance passes successfully.
ArXiv, 2026, Online
Pengyu Xue, He Yang Yuan, Xin Wang, Junkai Chen, Haonan Zhang, Boyuan Chen, Zishuo Ding, Zhenhao Li, and Weiyi Shang
NFI Rule-Score Evaluation. We implement rule-based checkers using Ruff, Radon, AST-based analyses, and custom scripts to evaluate the 92 executable rules introduced in Section 3.2. Ruff detects coding-style violations and library usage issues, Radon measures structural properties (e.g., cyclomatic complexity and function length), AST analyses identify Python-specific patterns, and custom scripts implement rules unsupported by existing tools. Benchmark Summary: SWE-NFI operationalizes five NFI aspects into 92 executable rules and contains 188 real-world tasks, including 145 single-file and 43 multi-file tasks. Each task is accompanied by a comprehensive evaluation suite combining functional correctness tests and rule-based NFI evaluation.
4 Experimental Setup 4.1 Studied Agents We evaluate eight coding agent configurations, including two commercial coding agents and two open-source coding agents paired with three different foundation models. Commercial Coding Agents. We study OpenAI Codex [48] and Claude Code Agent [15], two commercial coding agents designed for software engineering tasks. In our experiments, Codex is configured with gpt-5.3-codex, while Claude Code Agent uses claudesonnet-4-6. Open-source Coding Agents. We follow prior work [10] and study two open-source coding agents, Aider [3] and mini-SWEagent [63], the officially recommended successor to SWE-agent [74]. To examine the influence of different foundation models, we equip both agents with the same three backbone models: gpt-4.1 [49], gemini-2.5-flash [25], and deepseek-chat [16].
4.2
Experimental Protocol
To reduce the impact of stochasticity, we set the temperature to zero whenever the corresponding agent supports temperature control. For each agent configuration and each benchmark task, we repeat the experiment five times and report the average results across all runs. All agents are given the same task input, consisting of the natural-language task description and the corresponding code_before. Agents are expected to generate the complete revised code_after. The human reference implementation, i.e., the humanwritten code_after extracted from real pull requests, is never exposed to the agents and is only used to compute the human reference improvement 𝐼𝑖,𝑝 (ℎ).
4.3
invalid Python output are excluded from subsequent functional and NFI rule-score analysis. Functional Correctness. We then execute the manually implemented unit tests introduced in Section 3.4. A valid agent run is considered functionally correct if the generated code passes all corresponding test cases. Since SWE-NFI focuses on non-functional improvements beyond functional correctness, NFI rule-score analysis is performed only on functionally correct agent runs. NFI Improvement. For each valid and functionally correct agent run, we compute the NFI improvement score under the target aspect 𝑝 using Eq. (3). In this evaluation setting, the source 𝑥 in Eq. (3) is instantiated as the evaluated agent 𝑎, and 𝑐 𝑖after (𝑎) denotes the evaluated output generated by that agent. The human reference improvement 𝐼𝑖,𝑝 (ℎ) is pre-computed from the dataset by instantiating 𝑥 = ℎ, where 𝑐 𝑖after (ℎ) denotes the human reference implementation. Both quantities are computed relative to the same code_before, and the human reference implementation is never exposed to agents. Aggregate Analysis. We aggregate NFI results over valid and functionally correct agent runs. Unless otherwise specified, reported NFI values are average improvements computed from 𝐼𝑖,𝑝 (𝑎) in Eq. (3) and scaled to a percentage. For aspect-level analysis, we compute averages separately for each NFI aspect. For subset-level analysis, such as single-file versus multi-file tasks, we compute averages over the corresponding evaluated outputs. If an agent has no valid and functionally correct outputs for a given subset, its NFI improvement for that subset is removed from calculation rather than treated as zero.
5
Results
In this section, we systematically evaluate coding agents on SWENFI and analyze their ability to perform tasks on non-functional improvements. Specifically, we investigate the following research questions: • RQ1: How well do coding agents perform on non-functional improvements compared with human reference improvements? • RQ2: How does task scope affect the ability of coding agents to make non-functional improvements? • RQ3: How consistent are coding agents in producing nonfunctional improvements across repeated runs? • RQ4: What is the practical cost of making non-functional improvements in terms of execution time and token consumption?
Results Analysis Protocol
Output Validity. Before evaluating the code generated by agents, we first verify whether the generated output is valid. An output is considered valid if it is non-empty and syntactically parseable Python. We report output validity first because an agent that fails to produce valid code cannot be meaningfully evaluated by our evaluation suites. We report valid-output rate, which measures the fraction of agent runs where the generated output is both non-empty and syntactically valid Python. Agent runs with empty output or syntactically
RQ1: How well do coding agents perform on non-functional improvements compared with human reference improvements? Motivation. Modern coding agents have demonstrated strong capabilities in achieving functional correctness. However, it remains unclear whether these agents can also produce measurable nonfunctional improvements. In professional software development, these aspects are critical for long-term maintainability and usability. RQ1 evaluates whether agent-generated revisions produce valid,
SWE-NFI : Studying and Benchmarking Coding Agents for Non-Functional Improvements
ArXiv, 2026, Online
Table 3: Performance comparison of coding agents on non-functional improvements and functional correctness (RQ1). Agent
Valid (%)
Func. Pass (%)
Doc
EH
LC
CS
LP
Codex Claude Code Agent Aider + GPT-4.1 Aider + DeepSeek Aider + Gemini 2.5 Mini-SWE + GPT-4.1 Mini-SWE + DeepSeek Mini-SWE + Gemini 2.5
99.3 93.7 96.4 73.9 95.8 98.4 83.8 91.1
64.8 70.0 66.3 67.0 59.1 4.8 11.5 5.3
6.8− 7.4− 8.1+ 6.0− 7.0− 0.0 2.7− 0.0
2.1+ 2.1+ 2.3+ 2.5+ 2.1+ 0.0 1.6− 0.0
1.6+ 1.4+ −1.1− 0.5− −0.5− 0.0 2.1+ 0.5−
2.0+ 1.0− 0.1− 1.2− 0.6− 0.0 1.0− 0.9−
1.3− 0.7− 0.9− 0.7− 0.7− 0.0 0.6− 0.0
Human Reference
100.0
100.0
8.0
2.0
1.1
1.5
1.5
Table 4: Average NFI improvement by task scope (RQ2). Single-file (S) vs. multi-file (M) tasks. Improvement (Single-file %)
S − M (Difference %)
Improvement (Multi-file %)
Agent Codex Claude Code Agent Aider + GPT-4.1 Aider + DeepSeek Aider + Gemini 2.5 Mini-SWE + GPT-4.1 Mini-SWE + DeepSeek Mini-SWE + Gemini 2.5
Doc
EH
LC
CS
LP
Avg.
Doc
EH
LC
CS
LP
Avg.
Doc
EH
LC
CS
LP
Avg.
6.8 6.9 7.9 6.1 6.4 0.0 2.4 0.0
2.3 2.4 2.6 2.7 2.3 0.0 2.0 0.0
1.2 1.1 −1.1 0.3 −0.5 0.0 1.9 0.5
2.0 1.1 0.4 1.4 0.6 0.0 1.1 0.9
1.5 0.7 0.9 0.7 0.9 0.0 0.9 0.0
2.8 2.4 2.1 2.2 1.9 0.0 1.7 0.3
6.8 9.3 9.2 5.6 9.9 – 4.0 –
1.1 0.7 1.1 1.5 1.2 – 0.8 –
3.3 2.5 −1.3 1.7 0.1 – 2.7 –
1.9 0.7 −1.0 0.7 0.4 – 0.9 –
0.4 0.5 0.6 0.4 −0.1 – 0.0 –
2.7 2.8 1.7 2.0 2.3 – 1.7 –
0.0 −2.4 −1.3 +0.4 −3.5 – −1.6 –
+1.2 +1.7 +1.4 +1.3 +1.1 – +1.2 –
−2.1 −1.4 +0.2 −1.4 −0.5 – −0.8 –
+0.1 +0.4 +1.4 +0.7 +0.2 – +0.2 –
+1.1 +0.1 +0.3 +0.3 +1.0 – +0.9 –
+0.1 −0.3 +0.4 +0.3 −0.3 – 0.0 –
functionally correct, and measurable NFI. We compare these aspectlevel improvements with human-reference improvements from real pull requests. Approach. For RQ1, we evaluate each agent on the 188 tasks in SWE-NFI. We first report valid output rate and functional pass rate to separate executable outputs from functionally correct revisions. For NFI performance, each aspect column in Table 3 reports the average NFI improvement defined in Eq. (3). For an agent 𝑎, the reported value is the average of 𝐼𝑖,𝑝 (𝑎) over evaluated outputs under aspect 𝑝. For the Human row, the reported value is the average of 𝐼𝑖,𝑝 (ℎ), where code_after denotes the human reference implementation. Results. Table 3 shows the results of this RQ. For each aspect, a “+” or “-” following the number indicates whether an agent’s average improvement is above or below the corresponding human reference improvement for that aspect. Overall, current coding agents remain substantially behind human developers in making non-functional improvements. Although the best-performing agent achieves a functional correctness rate of 70.0%, most agent configurations fail to match the corresponding human improvements across the five NFI aspects. Among the evaluated aspects, logic patterns exhibit the largest gap: all evaluated agents remain below the human reference (0.0 - 1.3 vs. 1.5). These results suggest that, while current coding agents can successfully preserve functionality, they are still limited in performing the structural code improvements routinely made by human developers. Despite this overall gap, several agents demonstrate competitive performance on specific aspects. Codex achieves the highest validoutput rate (99.3%) and exceeds the human reference on coding style, error handling, and library constraints. Claude Code Agent achieves the highest functional correctness rate (70.0%) and exceeds the human reference on error handling and library constraints.
Aider + GPT-4.1 achieves the highest documentation improvement (8.1 vs. 8.0) and also exceeds the human reference on error handling. Performance also varies considerably across NFI aspects. Documentation and error handling are the strongest aspects for most configurations, whereas logic-pattern improvements consistently remain below the human reference. Library-constraint improvements show larger variation across agents: Codex, Claude Code Agent, and Mini-SWE + DeepSeek outperform the human reference, whereas Aider + GPT-4.1 and Aider + Gemini 2.5 produce negative average improvements. The Mini-SWE configurations reveal a functional-correctness bottleneck. Although they maintain relatively high valid-output rates (83.8% - 98.4%), their functional correctness rates remain between 4.8% and 11.5%, which leaves few outputs eligible for subsequent NFI evaluation. RQ1 Summary: Current coding agents remain substantially behind human developers in making non-functional improvements. Especially, all evaluated agents consistently underperform on logic-pattern improvements (0.0 - 1.3 vs. 1.5), which suggests that structural code improvements remain a major challenge.
RQ2: How does task scope affect the ability of coding agents to make non-functional improvements? Motivation. Real-world non-functional improvement tasks vary in scope. Some require localized modifications within a single file, whereas others involve coordinated changes across multiple files. Since modern coding agents are increasingly expected to handle repository-level development tasks, it is important to understand whether task scope influences their ability to make non-functional
ArXiv, 2026, Online
Pengyu Xue, He Yang Yuan, Xin Wang, Junkai Chen, Haonan Zhang, Boyuan Chen, Zishuo Ding, Zhenhao Li, and Weiyi Shang
improvements. RQ2 compares agent performance on single-file and multi-file tasks to investigate how task scope affects NFI improvements across different quality aspects. Approach. We split SWE-NFI into its single-file and multi-file subsets. For each agent and each subset, we compute the average NFI improvement using Eq. (3) with 𝑥 = 𝑎. The reported values are computed over valid and functionally correct evaluated outputs only. For multi-file tasks, rule scores are aggregated across the modified files before computing task-level NFI improvement. Table 4 reports aspect-level improvements and their average across aspects. The S−M columns subtract the multi-file average from the singlefile average. Negative values indicate higher average improvement on multi-file tasks, while positive values indicate higher average improvement on single-file tasks. Results. Table 4 shows that multi-file tasks remain more challenging for current coding agents. Overall, the effect of task scope varies across NFI aspects. Documentation occasionally benefits from multifile tasks (e.g., Claude Code Agent: 6.9→9.3 and Aider + Gemini 2.5: 6.4→9.9), suggesting that adding documentation across related files is relatively straightforward. In contrast, error-handling and logicpattern improvements consistently decrease on multi-file tasks across all evaluated agents. For example, Codex’s logic-pattern improvement drops from 1.5 to 0.4, while Claude Code Agent’s error-handling improvement decreases from 2.4 to 0.7. Notably, the two Mini-SWE configurations paired with GPT-4.1 and Gemini-2.5 fail to produce functionally correct outputs on multi-file tasks and are therefore excluded from the comparison. These results suggest that current coding agents can often extend localized improvements such as documentation across multiple files, but still struggle to coordinate more complex non-functional improvements involving multiple related files. RQ2 Summary: Overall, multi-file tasks remain more challenging than single-file tasks for current coding agents. While documentation occasionally benefits from additional cross-file context, multi-file tasks consistently reduce improvements in coding style (up to 1.4), error handling (up to 1.7), and logic patterns (up to 1.1).
RQ3: How consistent are coding agents in making non-functional improvements across repeated executions of the same task? Motivation. Coding agents driven by large language models may exhibit stochastic behavior across repeated executions, even under the same benchmark task and agent configuration. For software maintenance tasks, stable behavior matters at multiple stages: producing valid outputs, preserving functional correctness, and achieving measurable non-functional improvement. RQ3 evaluates whether these stages remain stable across repeated runs. Approach. For RQ3, we use five repeated executions of each agent configuration to measure run-to-run consistency. We analyze three metrics: valid-output rate, functional pass rate, and aggregate agent NFI improvement. The aggregate improvement, denoted as 𝐼 NFI (𝑎), is computed using Eq. (3) with 𝑥 = 𝑎, over valid and functionally
Table 5: Results over five repeated runs (RQ3). Agent
Metric
Run 1 Run 2 Run 3 Run 4 Run 5 Avg. Std.
Codex
Valid (%) Func. Pass (%) 𝐼 NFI (𝑎)
98.4 62.7 12.4
100.0 68.6 12.5
99.5 63.6 13.4
98.9 64.5 12.6
99.5 64.7 12.6
99.3 0.5 64.8 2.0 12.7 0.4
Valid (%) Claude Code Agent Func. Pass (%) 𝐼 NFI (𝑎)
97.3 67.1 13.1
92.0 71.0 13.1
94.0 69.8 12.8
92.0 70.1 12.8
93.1 71.9 13.4
93.7 2.0 70.0 1.6 13.0 0.2
Aider+GPT
Valid (%) Func. Pass (%) 𝐼 NFI (𝑎)
96.3 66.1 13.7
96.8 67.4 13.7
95.7 67.1 13.3
96.8 65.8 13.3
96.3 65.0 14.0
96.4 0.4 66.3 0.9 13.6 0.3
Aider+DS
Valid (%) Func. Pass (%) 𝐼 NFI (𝑎)
76.0 66.2 11.6
74.0 69.4 11.7
72.9 67.9 12.0
75.6 65.0 11.5
71.3 66.3 11.8
73.9 1.8 67.0 1.5 11.7 0.1
Aider+Gemini
Valid (%) Func. Pass (%) 𝐼 NFI (𝑎)
96.8 59.2 11.7
96.3 59.4 12.4
95.3 60.2 11.9
94.7 58.5 12.0
95.8 58.3 12.9
95.8 0.7 59.1 0.7 12.2 0.4
Mini-SWE+GPT
Valid (%) Func. Pass (%) 𝐼 NFI (𝑎)
98.4 4.8 0.0
98.4 4.8 0.0
98.4 4.8 0.0
98.4 4.8 0.0
98.4 4.8 0.0
98.4 0.0 4.8 0.0 0.0 0.0
Mini-SWE+DS
Valid (%) Func. Pass (%) 𝐼 NFI (𝑎)
79.8 10.7 8.6
87.6 12.0 6.6
85.7 11.1 5.5
85.7 13.8 7.8
80.3 9.8 5.0
83.8 3.1 11.5 1.4 6.7 1.4
Valid (%) Mini-SWE+Gemini Func. Pass (%) 𝐼 NFI (𝑎)
91.5 5.1 0.9
90.4 5.1 0.9
91.5 5.6 0.8
92.5 4.9 0.9
89.4 5.7 0.8
91.1 1.1 5.3 0.3 0.9 0.0
correct evaluated outputs. For each agent and metric, we report the five run values, their average, and standard deviation. Lower standard deviation indicates more stable behavior across repeated runs on the same tasks. Results. Table 5 shows that NFI improvements are generally stable across repeated runs. The best-performing configurations, including Codex, Claude Code Agent, and the three Aider-based agents, all exhibit relatively low standard deviations in 𝐼 NFI (𝑎) below 0.4, indicating that their measured NFI improvements remain consistent once outputs are valid and functionally correct. Greater variation appears before NFI evaluation. Valid-output and functional-pass rates fluctuate more noticeably for Aider + DeepSeek and Mini-SWE + DeepSeek. The results suggest that repeated executions primarily affect whether outputs become functionally correct and eligible for NFI evaluation rather than the measured NFI improvements themselves. Finally, consistency should be interpreted together with effectiveness. For example, Mini-SWE + GPT and Mini-SWE + Gemini show almost no variation across runs, but their functional pass rates remain below 6% and result in little or no measurable NFI improvement. RQ3 Summary: Aggregate NFI improvements remain consistent across repeated runs once outputs pass validity and functional-correctness checks. Most run-to-run variation arises before NFI evaluation, primarily from valid-output and functional pass rates. High consistency alone does not necessarily imply effective non-functional improvements.
SWE-NFI : Studying and Benchmarking Coding Agents for Non-Functional Improvements
ArXiv, 2026, Online Evaluation of Execution Time and Token Usage Across Different Agents
Approach. For each agent configuration, we measure two computationalcost metrics: execution time and token consumption. Execution time is measured from task initiation to completion, including model inference and agent execution overhead. Token consumption is computed as the total number of input and output tokens used during the interaction. We report the average values across all valid runs and compare them with the NFI performance reported in previous RQs. Results. Figure 2 shows that per-run execution time and token consumption vary by orders of magnitude across agent configurations. Claude Code Agent has the highest cost, averaging 14,330 seconds and 50.7 million tokens per run, followed by Mini-SWE + DeepSeek with 11,780 seconds and 32.1 million tokens. In contrast, Codex and the Aider configurations require fewer than one million tokens per run on average, while Mini-SWE + GPT-4.1 has the lowest measured cost, with 816 seconds and 327 thousand tokens. However, higher computational cost does not necessarily translate into proportionally better NFI performance. Claude Code Agent consumes approximately 100× more tokens than Codex and over 50× more than Aider + GPT-4.1, yet its functional pass rate as shown in Table 3 is only 5.2 and 3.7 percentage points higher, respectively. Moreover, Aider + GPT-4.1 achieves higher documentation (8.1 vs. 7.4) and error-handling improvements (2.3 vs. 2.1), while Codex outperforms Claude Code Agent in coding style (2.0 vs. 1.0) and library constraints (1.6 vs. 1.4). These results suggest diminishing returns in computational cost, where substantially higher resource consumption does not necessarily yield proportional gains in nonfunctional improvements. The results also show that computational cost is not explained by a simple commercial versus open-source distinction. Claude Code Agent is substantially more expensive than Codex despite both being commercial agents, while the open-source configurations exhibit a wide spectrum of resource consumption, ranging from the lightweight Aider framework to the considerably more expensive Mini-SWE framework. Overall, resource efficiency should be evaluated together with functional correctness and NFI performance. RQ4 Summary: Higher computational cost does not necessarily lead to proportionally better non-functional improvements. The cost pattern is also not explained by a simple commercial vs. open-source distinction. Resource cost should therefore be interpreted together with validity, functional correctness, and aggregate NFI improvement.
14330 10000
504,160 2450
32,139,360
50,741,364
11780 991,751 438,519 870,624 3654 3680 327,391 2854
7,518,460 5801
100,000,000
10,000,000 1,000,000 100,000
816
1000
Time(s)
Motivation. Beyond NFI performance, the practical utility of coding agents also depends on their computational cost. An agent that achieves strong NFI improvements may require substantially more execution time or token consumption than alternative configurations. RQ4 evaluates the computational cost of different coding agents and examines the relationship between resource consumption and NFI performance.
100000
10,000 100
Tokens
RQ4: What is the practical cost of making non-functional improvements in terms of execution time and token consumption?
1,000 100
10
10 1
Codex (gpt-5.3)
Claude Aider + GPTAider + Agent 4.1 DeepSeek (sonnet-4-6)
Aider + Gemini 2.5
Mini-SWE + Mini-SWE + Mini-SWE + GPT-4.1 DeepSeek Gemini 2.5
1
Agents
Avg Time (s)
Avg Token
Figure 2: Execution time and token usage across agents Task description Add type annotations and docstrings to g4f/debug.py to improve clarity for maintainers and static analysis tools, without changing runtime behavior. Code before … log_handler: callable = print logs: list = [] def log(text, file = None): if logging: log_handler(text, file=file) …
Sample 1
Sample 2
Claude r1 | NFI_Documentation +21.05%
Codex r1 | NFI_Documentation +10.53%
… log_handler: Callable[..., None] = print """Callable used to emit log messages.""" logs: list[str] = [] """Accumulated log messages.""" def log(text: str, file: Optional[TextIO] = None) -> None: """Write *text* via ``log_handler`` if enabled.
… log_handler: Callable[..., None] = print logs: list = [] def log(text: str, file: Optional[TextIO] = None) -> None: """Log text through the configured handler when logging is enabled.""" if logging: log_handler(text, file=file) …
The optional ``file`` stream is forwarded to ``log_handler`` (e.g. ``sys.stderr``). """ if logging: log_handler(text, file=file) …
Figure 3: Comparison of different Agent outputs
6 Discussion 6.1 Case Study Figure 3 shows a task from SWE-NFI that is related to improving the aspect of Documentation. The task requires adding type annotations and docstrings to improve maintainability and static-analysis support without changing runtime behavior. For readability, the figure shows abbreviated versions of the task description and source code. Both agents successfully improve the original implementation, but to different extents. Claude provides more comprehensive documentation, introduces more precise type annotations (e.g., list[str]), and documents the logging behavior in greater detail, resulting in a 21.05 Documentation rule-score improvement. Codex also adds type annotations and docstrings, but its revisions are less comprehensive, achieving a 10.53 Documentation improvement. This example illustrates that SWE-NFI can quantitatively distinguish the NFI produced by different coding agents.
6.2
Implications
Implication 1: Current coding agents still lag behind human developers in non-functional improvements. The benchmark instances in SWE-NFI are constructed from real merged pull requests, reflecting non-functional improvements that developers
ArXiv, 2026, Online
Pengyu Xue, He Yang Yuan, Xin Wang, Junkai Chen, Haonan Zhang, Boyuan Chen, Zishuo Ding, Zhenhao Li, and Weiyi Shang
actively make during software maintenance. These changes demonstrate that improving documentation, error handling, library usage, and code structure is an integral part of real-world software development rather than an auxiliary activity. However, our results show that current coding agents consistently fall short of the corresponding human reference improvements across multiple NFI aspects. This gap suggests that, although coding agents have achieved strong functional capabilities, their ability to perform high-quality nonfunctional improvements remains limited. Future research should place greater emphasis on developing coding agents that can better support software maintenance beyond functional correctness. Implication 2: Deterministic evaluation enables reproducible benchmarking. Unlike LLM-as-a-judge evaluation, SWE-NFI operationalizes non-functional improvements using 92 executable rules. This enables deterministic, reproducible, and scalable evaluation without requiring evaluation-time human annotation. Although rule-based evaluation cannot capture every aspect of software quality, it provides an objective foundation for comparing coding agents on developer-oriented non-functional improvements and is also flexible to extend. Implication 3: NFI capability of current coding agents is aspect-dependent and often favors localized additions over structural revisions. The aspect-level results show that agent performance is not uniform across NFI aspects. Documentation and error handling are the strongest sources of improvement for several configurations, while logic-pattern improvements remain below the human reference improvement for all evaluated agents. Library-constraint improvements also vary across configurations, including cases of negative average improvement despite functional correctness. These findings suggest that future coding agents should move beyond localized insertions toward reasoning about broader program structure and maintainability. Implication 4: SWE-NFI provides actionable guidance for non-functional improvements. The five-aspect framework and 92 executable rules provide a structured and practical foundation for improving non-functional aspects of software beyond functional correctness. By operationalizing common maintenance practices into measurable rules, SWE-NFI provides actionable guidance for both researchers and practitioners to evaluate and improve documentation, coding style, error handling, library usage, and code structure. Beyond serving as an evaluation benchmark, the proposed framework can facilitate the development of coding agents and software engineering tools that better support real-world software maintenance. Implication 5: NFI outcomes depend on the agent system and resource cost must be interpreted with output quality. The results show that NFI performance is shaped by the full agent configuration, not only by the backbone model or resource budget. Agent scaffolds affect whether outputs pass validity and functionalcorrectness checks, and thus how many outputs become eligible for NFI scoring. Meanwhile, higher execution time or token consumption does not by itself imply stronger NFI improvement, and lowcost configurations may still produce weak evaluated outputs. Practical use of coding agents for non-functional maintenance should
therefore consider resource cost together with validity, functional correctness, and measured NFI improvement.
7
Threats to Validity
Construct Validity. SWE-NFI operationalizes non-functional improvements using 92 executable NFI rules derived from various sources, including prior studies, open-source artifacts, and coding guidelines. Although these rules cannot fully capture every aspect of software quality, we intentionally focus on improvements that can be measured automatically and reproducibly without requiring evaluation-time human annotation or LLM-based judgment. Our benchmark is also flexible to extend more rules. Internal Validity. The stochastic nature of coding agents may introduce variability into the experimental results. To mitigate this threat, we set the temperature to zero whenever supported and repeat each experiment five times. We report the average results across all runs to reduce the influence of randomness. Our benchmark construction involves manual procedures, such as manually validating the data to be included in our benchmark, which may introduce subjectivity. To mitigate this threat, two authors independently performed these procedures, and disagreements were resolved through discussion until consensus was reached. Functional tests may also affect internal validity, because they may not cover all possible cases for each task. To mitigate this threat, we only apply NFI scoring to outputs that are valid and pass the task-specific tests. We report the functional pass rate separately, so outputs that fail the tests are not counted as negative NFI improvements. External Validity. Our benchmark focuses exclusively on Python projects. Although Python is one of the most widely used programming languages and has a rich open-source ecosystem, the findings may not directly generalize to other programming languages or software ecosystems. Future studies may extend our benchmark to study additional programming languages. Due to computational and API costs, we evaluate a limited set of coding agents. Although we include both commercial and open-source agents, our results may not fully represent the rapidly evolving landscape of coding agents. Future studies could expand the set of evaluated agents and foundation models to further validate our findings.
8
Conclusion
In this paper, we present SWE-NFI, a benchmark for evaluating coding agents on non-functional improvements beyond functional correctness. SWE-NFI contains 188 single-file and multi-file tasks constructed from real merged Python pull requests, and operationalizes five NFI aspects into 92 executable rules with a comprehensive evaluation suite combining functional correctness testing and deterministic rule-based NFI evaluation. Our results show that, although current coding agents can produce measurable NFIs after generating functionally correct code, they still consistently fall short of human developers, particularly on structural improvements such as logic patterns. SWE-NFI provides a reproducible foundation for evaluating and advancing coding agents toward better support for real-world software development and maintenance.
SWE-NFI : Studying and Benchmarking Coding Agents for Non-Functional Improvements
References [1] 2026. Replication Package. https://figshare.com/s/f14692e227368f6c392b. Last accessed June 2026. [2] Talor Abramovich, Meet Udeshi, Minghao Shao, Kilian Lieret, Haoran Xi, Kimberly Milner, Sofija Jancheska, John Yang, Carlos E Jimenez, Farshad Khorrami, Prashanth Krishnamurthy, Brendan Dolan-Gavitt, Muhammad Shafique, Karthik R Narasimhan, Ramesh Karri, and Ofir Press. 2025. EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities. In Forty-second International Conference on Machine Learning. https: //openreview.net/forum?id=Of3wZhVv1R [3] Aider. 2026. Aider: An ai pair programming tool. https://aider.chat/. [4] Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Sujan Kumar Gonugondla, Hantian Ding, Varun Kumar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Giaquinto, Haifeng Qian, Murali Krishna Ramanathan, Ramesh Nallapati, Baishakhi Ray, Parminder Bhatia, Sudipta Sengupta, Dan Roth, and Bing Xiang. 2023. Multi-lingual Evaluation of Code Generation Models. In The Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=Bo7eeXm6An8 [5] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021. Program Synthesis with Large Language Models. arXiv preprint arXiv:2108.07732 (2021). [6] Amiangshu Bosu, Michaela Greiler, and Christian Bird. 2015. Characteristics of Useful Code Reviews: An Empirical Study at Microsoft. In IEEE/ACM Working Conference on Mining Software Repositories (MSR). [7] Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. 2025. Repairagent: An autonomous, llm-based agent for program repair. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 2188–2200. [8] Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna PhippsCostin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. 2023. Multipl-e: A scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software Engineering 49, 7 (2023), 3675–3691. [9] Junkai Chen, Xing Hu, Zhenhao Li, Cuiyun Gao, Xin Xia, and David Lo. 2024. Code search is all you need? improving code suggestions with code search. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE). 1–13. doi:10.1145/3597503.3639085 [10] Junkai Chen, Huihui Huang, Yunbo Lyu, Junwen An, Jieke Shi, Chengran Yang, Ting Zhang, Haoye Tian, Yikun Li, Zhenhao Li, Xin Zhou, Xing Hu, and David Lo. 2026. SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarios. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Maria Liakata, Viviane P. Moreira, and David Jurgens (Eds.). Association for Computational Linguistics, San Diego, California, United States, 24144–24168. https://aclanthology.org/2026.acl-long.1107/ [11] Junkai Chen, Zhenhao Li, Xing Hu, and Xin Xia. 2026. Nlperturbator: Studying the robustness of code llms to natural language variations. ACM Transactions on Software Engineering and Methodology 35, 4 (2026), 1–20. [12] Junkai Chen, Zhenhao Li, Qiheng Mao, Xing Hu, Kui Liu, and Xin Xia. 2025. Understanding practitioners’ expectations on clear code review comments. Proceedings of the ACM on Software Engineering 2, ISSTA (2025), 1257–1279. doi:10.1145/3728931 [13] Junkai Chen, Zhiyuan Pan, Xing Hu, Zhenhao Li, Ge Li, and Xin Xia. 2025. Reasoning Runtime Behavior of a Program with LLM: How Far Are We?. In Proceedings of the IEEE/ACM 47th International Conference on Software Engineering (ICSE). [14] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, et al. 2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374 (2021). [15] Claude Code. 2026. Claude Code Docs. https://docs.anthropic.com/en/docs/ claude-code. Last Access: 2026. [16] Deepseek. 2026. DeepSeek. https://www.deepseek.com/en/. Last Access: 2026. [17] Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, et al. 2025. Swe-bench pro: Can ai agents solve long-horizon software engineering tasks? arXiv preprint arXiv:2509.16941 (2025). [18] Jingzhe Ding, Shengda Long, Changxin Pu, Huan Zhou, Hongwan Gao, Xiang Gao, Chao He, Yue Hou, Fei Hu, Zhaojian Li, et al. 2026. NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents. arXiv preprint arXiv:2512.12730 (2026). [19] Zishuo Ding, Yiming Tang, Yang Li, Heng Li, and Weiyi Shang. 2023. On the temporal relations between logging and code. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 843–854. [20] Mingzhe Du, Luu A Tuan, Bin Ji, Qian Liu, and See-Kiong Ng. 2024. Mercury: A code efficiency benchmark for code large language models. Advances in Neural Information Processing Systems 37 (2024), 16601–16622.
ArXiv, 2026, Online
[21] Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2024. Evaluating large language models in class-level code generation. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–13. [22] Yongkang Du, Jen-tse Huang, Jieyu Zhao, and Lu Lin. 2025. Faircoder: Evaluating social bias of llms in code generation. arXiv preprint arXiv:2501.05396 (2025). [23] Ryo Fujii, Makoto Morishita, Kazuki Yano, and Jun Suzuki. 2026. TimeMachinebench: A Benchmark for Evaluating Model Capabilities in Repository-Level Migration Tasks. arXiv preprint arXiv:2601.22597 (2026). [24] David Goodger and Rossum Guido. 2001. PEP 257 - Docstring Conventions. https://peps.python.org/pep-0257/. [25] Google. 2026. Gemini API. https://ai.google.dev/gemini-api/docs/models. Last Access: 2026. [26] Xinyi He, Qian Liu, Mingzhe Du, Lin Yan, Zhijie Fan, Yiming Huang, Zejian Yuan, and Zejun Ma. 2025. Swe-perf: Can language models optimize code performance on real-world repositories? arXiv preprint arXiv:2507.12415 (2025). [27] Ruida Hu, Xinchen Wang, Chao Peng, Cuiyun Gao, and David Lo. 2026. Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios. arXiv preprint arXiv:2604.06742 (2026). [28] Xing Hu, Feifei Niu, Junkai Chen, Xin Zhou, Junwei Zhang, Junda He, Xin Xia, and David Lo. 2025. Assessing and advancing benchmarks for evaluating large language models in software engineering tasks. ACM Transactions on Software Engineering and Methodology (2025). [29] Xing Hu, Xin Xia, et al. 2022. Practitioners’ Expectations on Automated Code Comment Generation. In IEEE/ACM International Conference on Software Engineering (ICSE). [30] Dong Huang, Yuhao Qing, Weiyi Shang, Heming Cui, and Jie M. Zhang. 2025. Effibench: Benchmarking the efficiency of automatically generated code. Advances in Neural Information Processing Systems 37 (2025), 11506–11544. [31] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world GitHub Issues? arXiv preprint arXiv:2310.06770 (2024). [32] Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2023. DS-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning. PMLR, 18319–18345. [33] Lukasz Langa. 2019. PEP 585 – Type Hinting Generics In Standard Collections. https://peps.python.org/pep-0585/. [34] Wei Li, Xin Zhang, Zhongxin Guo, Shaoguang Mao, Wen Luo, Guangyue Peng, Yangyu Huang, Houfeng Wang, and Scarlett Li. 2025. Fea-bench: A benchmark for evaluating repository-level code generation for feature implementation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 17160–17176. [35] Zhenhao Li, An Ran Chen, Xing Hu, Xin Xia, Tse-Hsun Chen, and Weiyi Shang. 2023. Are they all good? studying practitioners’ expectations on the readability of log messages. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 129–140. doi:10.1109/ASE56229.2023.00136 [36] Zhenhao Li, Tse-Hsun Chen, and Weiyi Shang. 2020. Where Shall We Log? Studying and Suggesting Logging Locations in Code Blocks. In 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020. 361–372. doi:10.1145/3324884.3416636 [37] Zhenhao Li, Tse-Hsun Chen, Jinqiu Yang, and Weiyi Shang. 2019. DLFinder: Characterizing and Detecting Duplicate Logging Code Smells. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). doi:10.1109/ICSE. 2019.00032 [38] Zhenhao Li, Heng Li, Tse-Hsun Chen, and Weiyi Shang. 2021. Deeplv: Suggesting log levels using ordinal based neural networks. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 1461–1472. doi:10.1109/ICSE43902.2021.00131 [39] Zhengquan Li, Zhenhao Li, and Zishuo Ding. 2025. From Feedback to Failure: Automated Android Performance Issue Reproduction. arXiv preprint arXiv:2508.11147 (2025). [40] Zhenhao Li, Chuan Luo, Tse-Hsun Chen, Weiyi Shang, Shilin He, Qingwei Lin, and Dongmei Zhang. 2023. Did we miss something important? Studying and exploring variable-aware log abstraction. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). 830–842. [41] Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Zhenpeng Chen, Lingming Zhang, and Yiling Lou. 2025. Large language model-based agents for software engineering: A survey. ACM Transactions on Software Engineering and Methodology (2025). [42] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. In Advances in Neural Information Processing Systems (NeurIPS). [43] Jiawei Liu, Songrun Xie, Junhao Wang, Yuxiang Wei, Yifeng Ding, and Lingming Zhang. 2024. Evaluating language models for efficient code generation. arXiv preprint arXiv:2408.06450 (2024).
ArXiv, 2026, Online
Pengyu Xue, He Yang Yuan, Xin Wang, Junkai Chen, Haonan Zhang, Boyuan Chen, Zishuo Ding, Zhenhao Li, and Weiyi Shang
[44] Jeffrey Jian Ma, Milad Hashemi, Amir Yazdanbakhsh, Kevin Swersky, Ofir Press, Enhui Li, Vijay Janapa Reddi, and Parthasarathy Ranganathan. 2025. SWEfficiency: Can Language Models Optimize Real-World Repositories on Real Workloads? arXiv preprint arXiv:2511.06090 (2025). [45] Qiheng Mao, Zhenhao Li, Xing Hu, Kui Liu, Xin Xia, and Jianling Sun. 2025. Towards explainable vulnerability detection with large language models. IEEE Transactions on Software Engineering (2025). [46] Shane McIntosh, Yasutaka Kamei, Bram Adams, and Ahmed E. Hassan. 2016. An Empirical Study of the Impact of Modern Code Review Practices on Software Quality. In Empirical Software Engineering, Vol. 21. 2146–2189. [47] Tam Nguyen, Phong Vu, and Tung Nguyen. 2020. Code Recommendation for Exception Handling. In ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). [48] OpenAI. 2025. Codex CLI: A Lightweight Coding Agent That Runs in Your Terminal. https://github.com/openai/codex. Accessed: 2026-06-28. [49] OpenAI. 2026. OpenAI Developer. https://developers.openai.com/api/docs/ models. Last Access: 2026. [50] Khouloud Oueslati, Maxime Lamothe, and Foutse Khomh. 2026. RefAgent: A Multi-agent LLM-based Framework for Automatic Software Refactoring. In IEEE/ACM International Conference on Software Engineering (ICSE). [51] Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2025. Asleep at the keyboard? assessing the security of github copilot’s code contributions. Commun. ACM 68, 2 (2025), 96–105. [52] Norman Peitek, Sven Apel, Chris Parnin, André Brechmann, and Janet Siegmund. 2021. Program Comprehension and Code Complexity Metrics: An fMRI Study. In IEEE/ACM International Conference on Software Engineering (ICSE). [53] Jinjun Peng, Leyi Cui, Kele Huang, Junfeng Yang, and Baishakhi Ray. 2025. Cweval: Outcome-driven evaluation on functionality and security of llm code generation. In 2025 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code). IEEE, 33–40. [54] Yun Peng, Jun Wan, Yichen Li, and Xiaoxue Ren. 2025. Coffe: A code efficiency benchmark for code generation. Proceedings of the ACM on Software Engineering 2, FSE (2025), 242–265. [55] Veronica Pimenova, Sarah Fakhoury, Christian Bird, Margaret-Anne Storey, and Madeline Endres. 2025. Good Vibrations? A Qualitative Study of Co-Creation, Communication, Flow, and Trust in Vibe Coding. arXiv:2509.12491 [cs.SE] [56] Yuhao Qing, Boyu Zhu, Mingzhe Du, Zhijiang Guo, Terry Yue Zhuo, Qianru Zhang, Jie Zhang, Heming Cui, Siu Ming Yiu, Dong Huang, et al. 2026. Effibenchx: A multi-language benchmark for measuring efficiency of llm-generated code. Advances in Neural Information Processing Systems 38 (2026). [57] Muhammad Shihab Rashid, Christian Bock, Yuan Zhuang, Alexander Buchholz, Tim Esler, Simon Valentin, Luca Franceschi, Martin Wistuba, Prabhu Teja Sivaprasad, Woo Jung Kim, et al. 2026. Swe-polybench: A multi-language benchmark for repository level evaluation of coding agents. arXiv preprint arXiv:2504.08703 (2026). [58] Guido Rossum, Jukka Lehtosalo, and Lukasz Langa. 2001. PEP 484 - Type Hints. https://peps.python.org/pep-0484/. Understanding McCabe Cyclomatic Complexity. [59] SciTools. 2026. https://support.scitools.com/support/solutions/articles/70000582297understanding-mccabe-cyclomatic-complexity. [60] Chihao Shen, Connor Dilgren, Purva Chiniya, Luke Griffith, Yu Ding, and Yizheng Chen. 2026. SecRepoBench: Benchmarking Code Agents for Secure Code Completion in Real-World Repositories. arXiv preprint arXiv:2504.21205 (2026). [61] Xinyu Shi, Zhenhao Li, and An Ran Chen. 2025. Enhancing LLM-based Fault Localization with a Functionality-Aware Retrieval-Augmented Generation Framework. arXiv preprint arXiv:2509.20552 (2025). [62] Anthony Sottile. 2021. pyupgrade Documentation. https://github.com/asottile/ pyupgrade. Last Access: 2026. [63] SWE-agent Team. 2026. mini-SWE-Agent. https://mini-swe-agent.com/latest/. [64] Tufts University. 2026. CS 40 Coding Standards. https://www.cs.tufts.edu/comp/ 40/docs/coding-standards.html. [65] Guido Van Rossum, Barry Warsaw, and Nick Coghlan. 2001. PEP 8 - Style Guide for Python Code. https://peps.python.org/pep-0008/. [66] Chong Wang, Kaifeng Huang, Jian Zhang, Yebo Feng, Lyuye Zhang, Yang Liu, and Xin Peng. 2025. LLMs Meet Library Evolution: Evaluating Deprecated API Usage in LLM-based Code Completion. In IEEE/ACM International Conference on Software Engineering (ICSE). arXiv:2406.09834. [67] Xin Wang, Yang Feng, Jiaoxiao Qian, Yang Zhang, Zhenhao Li, and Zishuo Ding. 2026. Logging Like Humans for LLMs: Rethinking Logging via Execution and Runtime Feedback. arXiv preprint arXiv:2603.29122 (2026). [68] Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. 2025. Openhands: An open platform for ai software developers as generalist agents. In International Conference on Learning Representations, Vol. 2025. 65882–65919. [69] Xin Wang, Zhenhao Li, and Zishuo Ding. 2025. Defects4Log: Benchmarking LLMs for Logging Code Defect Detection and Reasoning. In 40th IEEE/ACM International Conference on Automated Software Engineering, ASE 2025, Seoul, Korea, Republic
of, November 16-20, 2025. IEEE, 1931–1942. doi:10.1109/ASE63991.2025.00161 [70] Xin Wang, Zhenhao Li, and Zishuo Ding. 2026. LLM4Perf: Large Language Models Are Effective Samplers for Multi-Objective Performance Modeling. In Proceedings of the IEEE/ACM 48th International Conference on Software Engineering (ICSE). [71] Tongtong Wu, Weigang Wu, Xingyu Wang, Kang Xu, Suyu Ma, Bo Jiang, Ping Yang, Zhenchang Xing, Yuan-Fang Li, and Gholamreza Haffari. 2024. Versicode: Towards version-controllable code generation. arXiv preprint arXiv:2406.07411 (2024). [72] Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2024. Agentless: Demystifying LLM-based Software Engineering Agents. arXiv preprint arXiv:2407.01489 (2024). [73] Weiwei Xu, Kai Gao, Hao He, and Minghui Zhou. 2025. Licoeval: Evaluating llms on license compliance in code generation. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 1665–1677. [74] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. In Advances in Neural Information Processing Systems (NeurIPS). [75] John Yang, Carlos E Jimenez, Alex L Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R Narasimhan, et al. 2025. SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?. In The Thirteenth International Conference on Learning Representations. [76] John Yang, Kilian Lieret, Jeffrey Ma, Parth Thakkar, Dmitrii Pedchenko, Sten Sootla, Emily McMilin, Pengcheng Yin, Rui Hou, Gabriel Synnaeve, et al. 2026. ProgramBench: Can Language Models Rebuild Programs From Scratch? arXiv preprint arXiv:2605.03546 (2026). [77] Lanxin Yang, Jinwei Xu, Yifan Zhang, He Zhang, and Alberto Bacchelli. 2023. Evacrc: Evaluating code review comments. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 275–287. [78] Ka-Ping Yee. 2005. PEP 3134 – Exception Chaining and Embedded Tracebacks. https://peps.python.org/pep-3134/. [79] Boxi Yu, Yuxuan Zhu, Pinjia He, and Daniel Kang. 2025. Utboost: Rigorous evaluation of coding agents on swe-bench. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 3762–3774. [80] He Yang Yuan, Xin Wang, Kundi Yao, An Ran Chen, Zishuo Ding, and Zhenhao Li. 2026. Towards Secure Logging: Characterizing and Benchmarking Logging Code Security Issues with LLMs. Proceedings of the ACM on Software Engineering 3, FSE (2026), 745–765. [81] Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Shulin Xin, Linhao Zhang, Qi Liu, Li Aoyan, Lu Chen, Xiaojian Zhong, et al. 2026. Multi-swe-bench: A multilingual benchmark for issue resolving. Advances in Neural Information Processing Systems 38 (2026). [82] Beiqi Zhang, Peng Liang, Qiong Feng, Yujia Fu, and Zengyang Li. 2024. Copilotin-the-Loop: Fixing Code Smells in Copilot-Generated Python Code using Copilot. In IEEE/ACM International Conference on Automated Software Engineering (ASE), NIER Track. arXiv:2401.14176. [83] Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. Repocoder: Repository-level code completion through iterative retrieval and generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2471–2484. [84] Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, et al. 2026. Swe-bench goes live! Advances in Neural Information Processing Systems 38 (2026). [85] Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. Autocoderover: Autonomous program improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1592–1604. [86] Ming Zhong, Xiang Zhou, Ting-Yun Chang, Qingze Wang, Nan Xu, Xiance Si, Dan Garrette, Shyam Upadhyay, Jeremiah Liu, Jiawei Han, Benoit Schillings, and Jiao Sun. 2026. SWE-IF: Aligning Code Evaluation with Human Preference. In Proceedings of the 43rd International Conference on Machine Learning (ICML) (PMLR, Vol. 306). [87] Renyi Zhong, Yichen Li, Yulun Wu, Jinxi Kuang, Yintong Huo, and Michael R Lyu. 2026. Single-Language Evidence Is Insufficient for Automated Logging: A Multilingual Benchmark and Empirical Study with LLMs. arXiv preprint arXiv:2604.17529 (2026). [88] Renyi Zhong, Yichen Li, Guangba Yu, Wenwei Gu, Jinxi Kuang, Yintong Huo, and Michael R Lyu. 2026. Larger is not always better: Exploring small open-source language models in logging statement generation. ACM Transactions on Software Engineering and Methodology 35, 7 (2026), 1–41. [89] Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. 2025. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. In International Conference on Learning Representations, Vol. 2025. 66602–66656.