What Survives the Next Model? Benchmarking LLM-Based Techniques Against Single-Prompts

arXiv:2609.00468v1 [cs.SE] 31 Aug 2026

What Survives the Next Model? Benchmarking LLM-Based Techniques Against Single-Prompts Nahian Salsabil

Joy Saha

Simantika Bhattacharjee Dristi

Nicholas Phair

University of Virginia abj6hv@virginia.edu

University of Virginia daa7mv@virginia.edu

University of Virginia nwc8gr@virginia.edu

University of Virginia np4ay@virginia.edu

Nusrat Jahan Mozumder

Matthew B. Dwyer

Sebastian Elbaum

University of Virginia nm8tm@virginia.edu

University of Virginia matthewbdwyer@virginia.edu

University of Virginia selbaum@virginia.edu

Abstract—The software engineering research community has enthusiastically embraced the integration of Large Language Models (LLMs) into complex techniques to solve a wide variety of tasks. However, the extent to which this investment is strategic remains unclear, as the native capabilities of successive frontier model generations can rapidly render existing techniques obsolete. To assess this research investment, we analyze 35 LLMbased technique papers from ICSE 2026. We evaluate whether their complex tools can be outperformed by the simplest possible alternative: a single, automatically generated prompt executed on a newer generation model, without any iterative refinement. We find that for between 37% and 63% papers, a newer model with a single prompt natively outperforms the heavily engineered tooling proposed just a year prior. We identify that constructive techniques like code generation or repair are more amenable to substitution by a single-prompt. We also identify a surviving set of papers relying on strategies that provide additional insights to the model where newer LLMs will amplify the proposed technique. Our findings raise questions about the cost-benefit proposition of techniques designed as workarounds to temporary model deficits and the need to focus on enduring challenges that scale synergistically with future model generations. Our source codes and results are made publicly available at https://anonym ous.4open.science/r/ICSE2027-What Survives The Next Model.

I. I NTRODUCTION We are seeing amazing growth in the ability of LLMs to solve complex problems. The performance of frontier LLMs on programming challenges has been doubling year over year [14], [15]. Beyond these raw performance gains, their applicability has also broadened to encompass the entire software development lifecycle [12]. Industry has jumped in to adopt these techniques, with some reports indicating that 90% of developers are using AI-based tools [5] and investment in “tokens” to use them has been exploding [8]. The software engineering community has mirrored this enthusiasm. As shown in Figure 1, the fraction of ICSE Research Track papers mentioning “LLM” in their abstract grew from 2% (5 of 207) in 2023 to 45% (144 of 321) in 2026. This enormous growth and sudden shift in research focus has similarities with a historical pattern in artificial intelligence research described by Richard Sutton in his essay, The Bitter Lesson [24]. As Sutton observed, “. . . [AI] researchers always

Fig. 1. ICSE Research Track papers with “LLM” in the abstract, 2023–2026. Each bar is the number of Research Track papers that year; the blue-shaded segment shows those whose abstract mentions “LLM”.

tried to make systems that worked the way the researchers thought their own minds worked -they tried to put that knowledge in their systems- but it proved ultimately counterproductive, and a colossal waste of researcher’s time, when, through Moore’s law, massive computation became available and a means was found to put it to good use.” This insight warns that specialized human engineering designed to patch the immediate deficits of AI systems risks being outpaced and rendered obsolete by scaling. Today, the software engineering research community is at risk of experiencing a manifestation of this lesson. While the scaling mechanism driving the current software engineering shift is no longer just raw hardware compute but the compounding capabilities of frontier LLMs, the underlying dynamic remains remarkably similar. As a stark illustration, consider the publication of AutoVerus [27] in October 2025, which established a new state-of-the-art in LLM-based verification of Rust code by incorporating a highly sophisticated collection of bespoke agents that codified human expertise, verification strategies, and iterative feedback loops. Yet, just two months later in December 2025, a new challenging benchmark for Rust verification was released [28]. While AutoVerus

successfully resolved 20% of the problems, a newer frontier model (Sonnet 4.5), using a single, straightforward prompt, natively solved 81% of the benchmark. A single generation of model improvement effectively deprecated the engineering complexity of the specialized technique, while simultaneously yielding a four-fold performance improvement. A single instance does not constitute proof across all software engineering domains, but it serves as a cautionary tale of a potential broader trend. We hypothesize that a substantial volume of current software engineering research may face a similar fate by allocating significant effort to two activities with a very short half-life: (1) identifying the quirky, localized limitations of a specific generation of LLMs and (2) engineering complex, bespoke workarounds to bypass them. Such short-term patches are highly likely to be rendered obsolete by the native capabilities of future model generations, making them temporary fixes rather than enduring scientific contributions. To test this hypothesis and quantify the extent of this pattern across the broader software engineering literature, we analyze the techniques of 35 LLM-based software engineering papers published in the most recent ICSE proceedings from 2026. We reproduce their key results and evaluate their core techniques against a bare baseline: a single, few-shot prompt submitted to a newer generation model than the one utilized in the original study. By design, our baseline represents the absolute minimum possible effort: the prompt is automatically generated directly from the paper’s text, involves no iterative refinement, requires no execution loops, and lacks access to external tools or infrastructure. Our findings revealed a split in the longevity and value proposition of current LLM-based SE research. First, we observe a high ephemerality among complex bespoke techniques, finding that between 37% and 63% of the heavily engineered software engineering techniques using LLMs published less than six months ago can be effectively replaced by our singleprompt baseline. Constructive tasks, such as automated code generation or program repair, are the most susceptible to being natively absorbed by next-generation frontier models. For most of these techniques, the half-life is less than a single calendar year. We also discover that a single-prompt is insufficient with tasks requiring semantic understanding, complex feedback validation loops, or project-wide knowledge grounding. However, even for some of these domains, we project that simple additions, such as wrapping our singleprompts in a basic execution loop or granting them tool access, could bridge much of the remaining performance gap. On a positive note, our study highlights a resilient set of papers where model advancements do not replace the technique but rather amplify it. These papers provide a path forward for the community; they are characterized not by temporary workarounds, but by domain-specific processing, symbolic integration, and inputs structured specifically to provide richer context. Ultimately, this study provides the arguments to guardrail our research efforts from bridging temporary, fast-evaporating capability gaps in current LLMs.

Instead, we should focus on enduring challenges that naturally scale with model capabilities rather than being erased by them. II. BACKGROUND Large language models (LLMs) are increasingly being used across many software engineering domains. This trend has been documented for several years. As early as 2023, Fan et al. published a survey [7] identifying potential LLM applications in coding, testing, requirements engineering, repair, refactoring, and documentation. More recent surveys show the realization of that potential. Hou et al. [12] analyzed 395 SE papers from venues such as ICSE, FSE, and ASE, similarly documenting broad LLM adoption, for example, for code generation [6], judging [11], and agents [25] that plan, critique, select, vote, or coordinate subtasks, and assessing optimization strategies, datasets, and evaluation mechanisms. Following these patterns, our study of 35 ICSE papers also cuts across tasks and strategies. LLM usage across such a diverse set of tasks has driven the evolution of how we interact with LLMs in at least two ways. First, through the elevation of prompting to the level of a pseudo programming language for LLMs [3], [19]. Multiple prompting surveys [4], [20], [22] catalog evolving prompting strategies (i.e., zero to few-shot, chain of thought to trees of thought) and offer templates that capture best practices [26]. Beyond prompting, the ability to leverage multiple LLMs with greater levels of autonomy has led to the emergence of agents. A recent survey [10] shows that LLM-based SE techniques increasingly use agentic systems with planning, reasoning, memory, and tool augmentation. Against this backdrop, our study deliberately examines the simplest replacement strategy for software engineering techniques: a single-prompt to a single newer LLM, without access to any external tools and no multi-step interaction for iterative refinement. Prior works [9], [15], [21] have also evaluated LLM capabilities for software engineering tasks, measuring how well a model performs on a particular benchmark or task in isolation. A recent systematic review [13], for instance, characterizes how LLM capability is measured for standardized software engineering tasks by surveying 291 benchmarks. Their study explores these benchmarks, the SE tasks they cover, their construction methods, evaluation metrics, and remaining limitations in benchmark design. In this paper, we also perform a form of benchmarking. However, instead of purposefully defining different tasks or artifacts, we keep those fixed and introduce an alternative technique as treatment, substituting the original technique in a paper with a single-prompt to a newer LLM. As alluded to in previous work [17] and as we shall see in our study, performing this benchmarking in a systematic form across many papers is non-trivial. In this work we pose a different question: will current LLMbased software engineering techniques become obsolete as newer models emerge? This is not a question that has been asked, but it is critical, as the answer defines the operational half-life of the techniques we are developing.

Fig. 2. Methodology of single-prompt LLM approach

Fig. 3. Paper selection process.

III. R ESEARCH Q UESTIONS

hypothesis is focused on papers with an LLM component, from the remaining 236 papers, we remove 103 papers that do not use an LLM. Out of the 133 remaining papers, only 105 had publicly available artifacts to enable us to run our prompt-based approach. When inspecting their artifacts, we had to discard another 39 papers that did not provide the full dataset to perform the comparison, included an evaluation that required manual assessment, focused on the internals of the model instead of the software engineering task, or requiring specialized software (e.g., proprietary system, advanced simulator) to produce the input used as the data for evaluation. Next, we removed 19 papers that required extremely large inputs, as the cost of running such analysis is beyond our budget and LLMs are known to struggle in handling such large context [16]. For example, [23] required full C sourcecode repositories as input to translate them into Rust. This requirement makes single-call substitution difficult because the dataset contains 25 repositories and roughly 71K C/C header files in total, with 18 repositories containing at least 100 such files. The processing cost for only the input would be on the order of roughly $500, while including generated Rust output could raise the cost to several thousand dollars for a single pass over the full repository set. After applying these exclusion criteria, we were left with a final pool of 47 papers. Of this final pool, we analyzed a random set of 35 papers in this study (cited with the prefix EP, and listed under ‘Evaluated Papers’ in references).

We investigate the following research questions: • RQ1: Given a software engineering task, how effectively does single-prompt inference on state-of-the-art LLMs compete with a recent software engineering tool? • RQ2: What factors make a technique more amenable to single-call LLM substitution? • RQ3: How much performance gain is achieved by embedding the original paper’s methodology as reasoning guidance into the Pw compared to Pb ? IV. M ETHODOLOGY This methodology, as shown in Figure 2 aims to provide evidence for a solid comparison between techniques with LLM components implemented in a subset of the ICSE 2026 papers and a single-prompt invoking a newer LLM when applied to the same software engineering tasks. To carry out this comparison, we required the following: • A pool of candidate papers that implemented a technique using an LLM to solve a software engineering task and provided the necessary artifacts for evaluation of the prompt-based alternative. • A consistent mechanism to generate the single-prompts to solve each paper’s task. • An execution framework to run the generated prompts on the selected LLM with the same inputs and conditions established in the candidate papers. • An evaluation protocol to assess the prompt-driven LLMgenerated outputs against the paper’s tool over the same artifacts using the same process. A. Pool of Candidate Papers Figure 3 illustrates how we arrived at the pool of candidate papers, starting from the initial pool of 321 papers from the Research Track of the International Conference in Software Engineering 2026 [1], the latest instance of the flagship conference in our field. We first filtered 85 papers that did not introduce an automated technique to solve a task, this includes papers conducting empirical studies and surveys, developing benchmarks, or conducting human studies. Since the scope of our

B. Black-box and White-box Prompt Design and Generation To implement our single-prompt approach, for each paper, we needed prompts that instruct the LLM about the task it needs to perform, along with the corresponding inputs and outputs. We designed two types of prompts: Black-box (Pb ) and White-box (Pw ). These prompts share a common set of fields: Role, Task, Input, Output, Example input-output pairs, and Instructions. The Pb does not make any reference to the original paper’s methodology, asking the LLM to infer the functionality to solve the task given the inputs and outputs provided. The Pw extends the common structure with one additional field, Steps, that embeds the paper’s methodology as internal reasoning guidance, asking the LLM to realize the

functionality following the paper’s approach. This distinction allows us to examine whether providing the paper’s methodology as guidance offers any additional benefit to the LLM’s reasoning, a comparison that we later present in Section V-C. The full structure of both prompts is shown in Table I. Given the two prompt designs, we set to automate their generation by designing a meta-prompt that, given a paper and input/output sample pairs, would invoke an LLM to generate paper-specific black-box and white-box prompts that instantiate the fields in Table I. The design of this metaprompt was iterative, refined using a set of 5 calibration papers until: the two prompts generated contained complete and identical content in every field except the Steps section, the Pb did not include any methodological details from the paper, and the required input and output format was correct. To address this, and throughout this process, we found that it was crucial to extract at least a single sample input/output pair from each paper’s dataset, since the manuscripts themselves rarely detail these concrete execution artifacts. In general, we provided as few examples as possible but the number varied across papers. For instance, [EP10] uses six datasets, and the expected output structure varies across them; therefore, we provided one example from each dataset. Similarly, in [EP31], the output schema contains 15 fields, but no single dataset instance assigns meaningful values to all 15 fields. We therefore provided three examples whose combined outputs illustrate the full schema. During this meta-prompt refinement we only evaluated the structure of the generated prompts; we did not run experiments to assess the resulting prompts’ performance on the target task and iterate on the meta-prompt. Once the meta-prompt satisfied the required criteria across the calibration papers, we adopted it as the single template used across all experiments. The template of the meta-prompt is shown in Figure 4. C. Prompt Executor This component invokes the LLM with the generated prompts and the dataset from the candidate paper. Before executing the prompt, we first manually checked that the papers’ datasets satisfied the following criteria. First, they had to provide inputs and expected outputs against which we could check our results. If they did not, we removed the portions of the datasets that lack such ground truth. For example, [EP14] reports results on three datasets, but only one includes ground-truth vulnerability labels, thus we removed the other two datasets from our assessment. Second, the inputs used by the paper’s technique had to be readily available. If the inputs required, for example, any preprocessing or remote retrieval, we conducted those steps so that the generated prompts received the same inputs as the paper’s technique. For example, [EP1] provides a set of Python programs but then it constructs its own evaluation dataset by removing certain function arguments from each file, stores the removed arguments as ground truth, and then asks their technique to recover them. We therefore replicated this step ourselves using Python’s ast module to remove the corresponding arguments

Meta-Prompt for Single-prompt LLM Approach Role: Expert AI Engineer specializing in Prompt Engineering and Technical Synthesis. Task: Analyze a research paper and generate two self-contained replacement prompts, Prompt Bb and Prompt Wb, that replace the paper’s full multi-step technique with a single LLM call. Rules for Both Prompts: - Be fully self-contained... - The prompts must replace the paper’s entire multi-step technique... - ... - Embed the reference examples as a few-shot demonstration inside both prompts... Steps vs. Instructions Boundary: - Steps: The methodology... - Instructions: Behavioral constraints... - Do not duplicate content across Steps and Instructions. Output Format: Provide both prompts as separate fenced Markdown code blocks. Prompt Bb: {See Table I for structure} Prompt Wb: {See Table I for structure} Reference Examples: The following is a set of concrete input-output pairs representing exactly what the technique receives at the very start and produces at the very end... "examples": [ { "inputs": { "<input_field_1>": "<value>", "<input_field_2>": "<value>" }, "outputs": { "<output_field_1>": "<value>" } } ]

Fig. 4. Meta-prompt template, together with the candidate paper, is sent to the LLM to generate Pw and Pb .

and produce the same incomplete-source inputs used by the paper’s technique. Once the dataset met our criteria, our scripts made a single LLM call with the prompt and input. To control the execution costs of the study, we provided an LLM budget of $15 per paper, which meant that not every dataset in every paper was fully analyzed. To accommodate this constraint fairly, we randomly sampled the inputs to analyze from each paper’s dataset. Figure 5 summarizes the percentage of each paper’s evaluation dataset covered under this fixed budget. In the end, for 15 papers, that budget was enough to complete the analysis on the full dataset. For 16 papers, it covered 10% to 70% of the dataset. For 4 papers [EP25], [EP23], [EP32], [EP1], we covered only 5%–7% and since the results were mostly negative, we did not pursue them further. We stored the model’s raw response (for traceability) and the output in the requested structured json format. The formatted outputs (i.e., code, labels, test cases, constraints, prediction), were then evaluated against the original paper’s expected outputs following its evaluation protocol. LLMs’ capacities advance so rapidly that yesterday’s frontier-leading model quickly becomes today’s trailing model. For example, since the inception of this study, Anthropic has released several frontier models: Claude Opus 4.7 and Opus 4.8, Mythos, and Fable [2]. For this study, we selected the trailing-edge Claude Sonnet 4.6 (released in February 2026). This decision was driven by three factors. First, Sonnet

TABLE I W HITE -B OX AND B LACK - BOX P ROMPT S TRUCTURE Field

Description

Field Content

Role

Task-specific LLM role.

You are an Expert software engineer specializing in API usage analysis and code ...

Task

Brief task objective.

You receive a code snippet with one or more missing API method arguments, indicated by a ‘/* missing */‘ placeholder or similar comment...

Input

Input fields and definitions.

preceding_code: A string containing the full code snippet up... call_line: A string identifying the specific line or expression...

Output

Output schema and field definitions.

A JSON object with a single key arguments, whose value is an array of objects, each containing: position: integer value: string

Example

Demonstrative input-output pairs.

inputs: preceding_code: "import os.path as path PROJECT_ROOT = path.realpath(path.dirname(__file__)) ...", call_line: "MEDIA_ROOT = path.join(/* missing */)" outputs: arguments: position: 0, value: "PROJECT_ROOT" position: 1, value: "media"

Steps

White-box-only methodology steps.

1. Identify the incomplete method call... 2. Extract variable definitions and their types... 3. Build an internal data-flow map... 4. Reason about each missing argument position... 5. Verify internal consistency... 6. Compile and format the output...

Instructions

Fixed and task-specific constraints.

- Solve the task using only the provided input. - Do not use any external tools, APIs, web search... - Provide a concrete value for every missing argument position... - Output only valid JSON matching the schema above. Do not include explanation... ...

D. Evaluator

100 90

Dataset coverage (%)

80 70 60 50 40 30 20 10 20 25 38 42 54 77 97 98 103 108 186 189 219 237 256 73 18 10 82 254 232 19 102 104 52 86 88 14 66 303 154 6 70 11 32

0 Paper ID

Fig. 5. Percentage of each paper’s evaluation dataset covered under the fixed LLM budget. Papers are sorted by coverage percentage.

4.6 represented the state of the frontier when we initiated this research; re-executing our entire approach on subsequent iterations would have required substantial renewed efforts and disrupted our established baseline. Second, Sonnet provides a non-trivial cost differential compared to leading-edge alternatives, as Anthropic’s Opus models are approximately 1.67x more expensive than Sonnet for both input and output tokens, making cost a pragmatic consideration given the scale of our experiments across 35 papers. Third, this choice establishes a capability floor: if a trailing-edge model can successfully perform the target task, we can presume that future frontieredge models will achieve at least the same performance.

This component aims to reproduce each paper’s evaluation methodology to support a direct comparison between the paper-reported results and the results produced by our single-prompt approach. This phase consumed the most effort because each paper defines its own evaluation protocol that includes target metrics, ground truth, and execution environment. To build each evaluator we followed three steps. First, we identified the main research question (RQ). Papers typically pose multiple RQs. Since our goal is to compare against the tool’s overall performance, we selected the RQ that most directly measures the end-to-end performance of the paper’s technique. Then, for that RQ, we identified the metrics used by the paper and adopted them in our evaluation to make our results directly comparable to the ones in the paper. Second, we prepared the ground truth needed to compute those metrics. For 18 papers, the dataset provided explicit ground truth, such as labels, reference outputs, or affected program elements. In these cases, evaluation consisted of comparing the LLM-produced output directly against the provided ground truth using the paper’s reported metrics. For the other 17 papers, the ground truth requires additional execution or checking the generated artifact. These cases required running tests, coverage tools, compilers, semantic equivalence checkers, or benchmark-specific evaluation harnesses before computing the final metric. Third, we replicated the paper’s evaluation setup so that the generated outputs were evaluated under the same conditions

and metrics as the original technique. To give an idea of the complexity of this effort, we share a couple of examples. [EP26] generates tests to distinguish a plausible patch from an oracle patch, revealing behavioral discrepancies even when SWE-bench deems the patch correct. Checking whether a generated test differentiates the patches requires running it against both, which implies we had to construct and launch a Docker container with an SWE-bench dataset, run the SWEharness on both patches, and compare them. As another example, for [EP29], which evaluates PostgreSQL-to-MySQL SQL dialect translation, we had to reproduce its executionbased setup by launching PostgreSQL and MySQL Docker containers, generating and loading identical database contents, running the source query on PostgreSQL and the translated query on MySQL, and comparing their result sets. Overall, this evaluation process required a paper-specific evaluator for each of the 35 techniques, consuming several hours to understand, prepare, revise, and package. Each evaluator encoded the corresponding paper’s metrics, ground truth format, and, when necessary, execution environment or external checking procedure. We make these evaluator scripts, along with the generated outputs and corresponding results, available in our replication repository. E. Artifact Quality Control and Paper Categorization We perform two additional cross-cutting processes to ensure artifact quality and enrich the analysis. First, we validate the correctness of the artifacts used in our experiments. Second, we label each paper on two technical factors that may affect whether a single-prompt can replace an existing technique: the type of software engineering task being performed and the key strategy used by the original technique to accomplish that task. To ensure the soundness of the produced artifacts, we performed an independent cross-reviewing process for each paper. We first defined a common checklist so that reviewers evaluated each artifact using the same criteria. Using this checklist, reviewers verified that the selected paper was appropriate for the study, the dataset was correctly identified, the evaluated research question matched the paper’s main technique, and the metrics corresponded to that research question. They also verified that each prompt included at least one correct inputoutput example from the chosen dataset, the input-output formulation matched the paper’s task, and the evaluator was correctly implemented to match each paper’s evaluation setup, including the correct ground truth or the required execution environment. Each reviewer inspected multiple artifacts using this checklist. When a reviewer identified an issue, it was brought up for discussion and, as needed, corrected and reran before including it in the final analysis. After completing the artifact checks, we categorized the papers along the two factors introduced above: task type and strategy. To establish categories for these factors, rather than forcing papers into a predefined taxonomy, we utilized an incremental coding process. First, several reviewers independently characterized each paper’s target domain and core technical aspects. A primary encoder then synthesized these

characterizations alongside the papers to explicitly code the stated software engineering domain and technical contribution. We then iteratively grouped, refined, and consolidated these codes to maximize distinctness and minimize overlap across our corpus. This process concluded once a consensus was reached among two reviewers, ensuring the resulting categories were sufficiently granular to capture technical key differences yet distinct enough to remain mutually exclusive. This process resulted in the following task labels according to software engineering tasks: code generation, bug finding, repair, impact analysis, requirement formalization, log analysis, test generation, and verification. Because some papers address more than one task or combine multiple technical strategies, we allow a paper to receive multiple task or strategy labels when appropriate. For strategy, the resulting labels require further explanation. Knowledge Grounding refers to techniques that retrieve, construct, or inject additional project context, API relationships, dependency information, or static-analysis facts inside their approach. Feedback and Validation refers to methodologies that use execution results, tests, compiler messages, traces, or other validation signals as feedback for other stages of the approach. Structured Reasoning refers to techniques that decompose the task into explicit reasoning stages, agents, or hierarchical subtasks. Search and Selection refers to approaches that generate multiple candidate solutions and rank, filter, or select among them. Input/Output Control refers to frameworks whose main contribution is constraining, reducing, filtering, normalizing, or decoding the model’s input or output schema. Domain-Specific Processing refers to techniques that encode specialized rules, parsers, annotations, or procedures for a particular software engineering domain. Model Adaptation refers to techniques that train, tune, or adapt a model or policy to improve performance on the target task. V. R ESULTS We now share the results for each research question. A. RQ1: Single-Prompt LLM vs. Engineered SE Approaches Table II summarizes the comparison between the original techniques and the single-prompt replacements. The ID column gives the paper identifier. The Paper’s LLM column reports the LLM used by the original technique. When a paper’s technique is instantiated with multiple models, we pick the strongest model used for comparison. Dataset and Metrics identify the benchmark and paper-specific criteria used in our replication. Original Result reports the paper’s result on that dataset and metric. Pb and Pw correspond to the prompt results. Finally, Diff(nl) Performance compares the best single-prompt result against the original technique: positive (+) means the single-prompt fully outperformed the original technique, mixed (+−) when it outperformed the original only for a portion of the datasets or some but not all metrics, and negative (−) when it consistently underperformed the original. Bolded numbers indicate best performance in that row for a particular dataset portion and metric.

TABLE II C OMPARISON OF PAPER - REPORTED RESULTS AGAINST S INGLE -P ROMPT LLM ID

Paper’s LLM

Dataset

Metrics

[EP14]

Claude-3.5-Sonnet

EchoFuzz D2

#Vuln detected/TP/FP/FN

103/103/0/8

[EP20]

Claude-3.5-Sonnet

StatType-SO

F1

90.8

Pass@5

90.42, 87.34, 80.81, 90.46, 85.99, 86.40, 92.89, 99.23, 90.66, 85.89, 86.09

Pw

Diff(nl) Performance

287/111/176/0

297/111/186/0

+−

42.9

43.1

−

93.15, 93.15, 81.25, 92.86, 63.38, 100.00, 96.77, 100.00, 92.86, 88.24, 100.00

86.3, 90.41, 81.25, 96.55, 66.20, 100.00, 90.32, 89.92, 89.47, 100.00, 90.00

+

Pb

Original Result

[EP17]

Claude-3.5-Sonnet

xCodeEval (C, C#, C++, Go, Java, JS, Kotlin, PHP, Python, Ruby, Rust)

[EP16]

CodeLlama-7B

Custom security code review dataset

F1/SecureBLEU

71.98/29.31

36.62/18.59

32.97/16.59

−

[EP21]

DeepSeek-Coder-V2-Lite

HumanEval, MBPP

Accuracy/RSR

94.5/76.3, 80/39

100/100 , 91/72.56

98.17/92.11 , 90/69.51

+

[EP30]

DeepSeek-R1

FoundRoot A-D

MRR

56.9, 61, 85.8, 93.1

61.33, 58, 84.44, 92.52

64.85, 61.79, 83, 90.6

+−

[EP13]

DeepSeek-V3

CodaMosa

(Line + Branch) coverage

62

42.55

46.17

−

[EP27]

DeepSeek-V3

DB2 (Ethereum)

F1

88.46

40

28

−

[EP10]

DeepSeek-V3

FSD, RAC, PURE, BP, US, LMC

N-F1/R-F1/, Pass rate

71.32/60.48/ 97.78

48.53/49.99/ 96.34

49.65/50.71/ 98.38

+−

[EP22]

DeepSeek-V3

FSD

F1

50.94

38.64

35.71

−

[EP33]

DeepSeek-V3

LFTBench

Pass@1

45.9

68

72

+

[EP8]

DeepSeek-V3

SynthNL, ConfoNL, LangNL, SpecNL

Accuracy

82

61.01

58.18

−

[EP19]

Gemini-2.5-Flash

FSM-AP, FSM-S, REG, RobotExplain, Ventilator, Deepstl-test

Pass@10

100, 100, 100, 71.7, 61.2, 100

100, 66.7, 100, 0, 20.7, 25

100, 100, 100, 20, 27.6, 100

+−

[EP9]

Gemini-2.5-Flash

HumanEval, HumanEval+, ClassEval, BigCodeBench

Pass@1

99.39, 98.17, 82 , 85

100, 93.33, 71, 87.72

100 , 93.33 , 74, 86.84

+−

[EP24]

Gemini-1.5-Flash

APPS, CodeContests, xCodeEval

Pass@1

65.33, 36.36, 38.67

70.54, 40.85, 55.04

64.34, 43.66, 48.06

+

[EP23]

GLM4-2.0-Flash

BGL, Spirit, Thunderbird, HDFS, Hadoop2, Hadoop3, Spark2, Spark3

F1

97.5

18.5

14.9

−

[EP31]

GPT-4.1

LV-Parser evaluation dataset

Mean accuracy

93.60

82.99

86.77

−

[EP4]

GPT-4.1-mini

MultiPL-E Java

Pass@1

61.90

78.85

77.88

+

[EP32]

GPT-4o

Alexandria

Hit@k/F1

69/25

40.0/20.9

40.0/17.3

−

[EP11]

GPT-4o

AGORA+

Precision/Recall

85.1/83

53.7/41.6

61.8/40.6

−

[EP34]

GPT-4o

Defects4J V1.2, Defects4J V2.0

Pass test

52.94, 43.38

43.33, 38.67

46, 47.33

+−

[EP18]

GPT-4o

HumanEval, LiveCodeBench

Pass@1

90.2, 50.2

99.1, 52.51

98.2, 52.8

+

[EP1]

GPT-4o

Py150, Netbeans

Precision/Recall

73.1/68.1, 72.11/69.46

66.12/65.47, 72.7/72.57

70/69.06, 71.5/71.38

+−

[EP28]

GPT-4o

PrimeVul

PC

18.62

13.37

22.17

+

[EP35]

GPT-4o

PrOntoQA-OOD, ProofWriter, FOLIO

Average macro F1

94.26, 91.24, 84.42

76.02, 65.79, 56.82

77.74, 84.28, 66.66

−

[EP29]

GPT-4o

TPC-DS, SQLProcBench

Translation accuracy

97.98

60.61

61.62

−

[EP7]

GPT-4o-mini

CoderEval

Pass@1

36.52

54.62

54.55

+

[EP3]

GPT-o3-mini

GuideSyn .mls benchmark

%solved

78

81.25

83.75

+

[EP26]

GPT-4o mini

SWE-bench

Differentiating rate

29.3, 27.2

15.56 ,36.73

11.11, 24.49

+−

[EP6]

Llama 3.3 70B

Defects4J

Line/Branch coverage, Pass rate

78.62/69.25, 62.91

89.3/80.3, 98.9

91.7/82.9, 98.4

+

[EP25]

Qwen2.5-14B

HPC (Loghub-2k)

PA/PTA/RTA/GA

99.3/71.7/82.6/93.4

90.5/72.9/76.1/90

90.5/78.3/78.3/90.5

+−

[EP12]

Qwen2.5-32b-instruct

Custom GitHub derailment dataset

F1@threshold 0.3

90.10

85.82

90.91

+

[EP2]

Qwen2.5-72B

CoCoClaNeL

MCC

25.9

65.72

65.41

+

[EP5]

Qwen2.5-Coder-7B-Instruct

HumanEval, MBPP, LiveCodeBench

Pass@1

90.9, 88.6, 36.7

100, 100, 48.68

100, 99.29, 50.94

+

[EP15]

StarCoder2-7B (v1), CodeLlama-Python-7B (v2)

Tfv1 Synthetic

Exact match@1

83.53 (v1), 60.86 (v2)

71.4 (v1), 36.5 (v2)

70.9 (v1), 36.3 (v2)

−

TABLE III D ISTRIBUTION OF OUTCOMES BY TASK AND STRATEGY CATEGORY. Task Category + - ± Bug Finding 1 7 3 Code Generation 6 2 1 Impact Analysis 0 2 0 Log Analysis 1 1 1 Repair 3 1 2 Req. Formalization 0 1 2 Test Generation 1 2 1 Verification 1 1 0 Total 13 17 10

Strategy Category Domain-Specific Processing Feedback and Validation Input/Output Control Knowledge Grounding Model Adaptation Search and Selection Structured Reasoning

+ 1 3 2 1 1 3 5

4 3 2 4 1 0 1

± 0 3 0 3 1 0 3

Total

16 15 10

Overall, the results show that a single-prompt to a newer LLM is a surprisingly competitive substitute for many recent SE techniques engineered over older LLMs. Among the 35 studied papers, the single-prompt inference consistently outperformed the original technique in 13 cases and had mixed results in 9 additional cases. RQ1. Between 37% and 63% of recent SE techniques can be effectively replaced by an automatically generated single-prompt to a general-purpose LLM.

B. RQ2: Amenability to single-prompt LLM substitution Building on the paper categorizations introduced earlier, we explore how the target task type and the underlying technical strategies used by a given technique affect a single-prompt’s viability as a replacement. We find that the effectiveness of the single-prompt varies substantially across task categories and strategies, but there are a few emerging patterns that we highlight through Table III and the heatmap in Figure 6. In Table III, each row denotes either a task or strategy category with columns denoting positive, negative and mixed results count per category. In Figure 6, each row represents a task category and each column a strategy. Green cells indicate the single-prompt outperforms the paper’s technique, and red indicates the opposite. Cell’s scores range from −1.0 (singleprompt underperforms in every paper using that strategy-task combination) to +1.0 (outperforms in every paper), where 0.0 reflects a balance; n denotes the number of papers per cell, and empty cells lack papers for that combination. Papers performing constructive tasks such as code generation and program repair are more likely to be replaced by single-prompts. As we can see in Table III, single-prompts for code generation outperform the techniques in six [EP7], [EP18], [EP5], [EP4], [EP3], [EP24] out of nine papers, obtain mixed results for one [EP1], and underperform in the other two [EP15], [EP29]. For example, in [EP4], the single-prompt achieved 78.85% on pass@1, whereas the paper’s original approach got 61.9%. Code generation techniques that integrate strategies that constrain or guide the model were the easiest to outperform. For every instance of code generation using a ‘Search and Selection’ strategy, a single-prompt yielded consistently better results [EP18], [EP4], [EP3] (cell of code

generation and ‘Search and Selection’ in Figure 6). Overall, we find that newer LLMs excel at synthesizing functional code when they are anchored by simple constraints. Similarly, code repair techniques are also somewhat susceptible to quick and simple prompt-based replacements. As seen in Table III, for three of the six repair papers a singleprompt outperforms the original technique [EP17], [EP21], [EP33], and two results were mixed [EP9], [EP34]. Among them, single-prompts were specifically well-suited to replace techniques that performed ‘Structured Reasoning’ (two positive result [EP17], [EP21] and one mixed result [EP9] as seen in Figure 6, indicating that newer models may be able to capture those natively. For example, [EP17] uses structured reasoning through a three-agent repair workflow, where the agents translate the buggy program, generate repairs, and validate or revise the fix, yet the single-prompt replacement outperformed it. Surprisingly, single-prompts did not produce any negative result with one positive [EP17] and one mixed [EP9] on repair tasks even when the underlying techniques used ‘Feedback and Validation’ as a strategy. In contrast, single-prompts could not compete with tasks that require a deeper or specific semantic understanding. For example, as seen in the first row of Figure 6 for Bug-finding tasks, single-prompts struggle specially where the underlying techniques leverage ’Domain-Specific Processing’ [EP16], [EP31], [EP22], ‘Knowledge Grounding’ [EP23], [EP27], or ‘Feedback and Validation’ [EP35]. This limitation extends to the Impact Analysis [EP32], [EP20] tasks as well, as seen in the Table III having no positive or mixed results. The remaining tasks had fewer papers and did not show enough consistent patterns to report. In Figure 6, looking at the strategies independent of the task, techniques relying on ‘Search and Selection’ were the most replaceable, having all positive results, followed by ‘Structured Reasoning’, As seen in Table III, all three papers [EP18], [EP4], [EP3] using ‘Search and Selection’ outperformed the original technique. Of nine papers using structured reasoning, five outperformed [EP5], [EP2], [EP17], [EP21], [EP28], three were mixed [EP9], [EP30], [EP10], and one underperformed [EP11]. These results suggest that newer LLMs can often internalize such selection-based techniques and structured reasoning with a single-prompt. As seen in Figure 6, the hardest techniques to replace are those that relied on ‘Domain-Specific Processing’ and ‘Knowledge Grounding’ strategies. According to Table III, four of the five ‘Domain-Specific Processing’ papers could not be outperformed by single-prompts. These techniques encode domain knowledge in task-specific procedures. For example, [EP8] is trained on dataset-specific translation guidelines by comparing generated formulas with ground-truth LTL, whereas our single-prompt approach receives no such dataset-level adaptation. This suggests that a single-prompt struggles when the original approach’s advantage comes from specialized procedures learned or constructed for a particular domain. ‘Knowledge Grounding’ showed a similar pattern, with only one positive result [EP7], three mixed results [EP1], [EP34],

(n=3)

-0.33 (n=3)

-1.00

+0.50

+1.00

(n=3)

-0.67

Code Generation

(n=3)

-1.00

Impact Analysis

-0.67 (n=2)

+0.00

(n=1) (n=1)

(n=3)

+1.00 (n=3)

+1.00 (n=1)

-1.00

(n=1)

(n=2)

0.5

Log Analysis

+1.00

Repair

-1.00

+0.50

Req Formalization

-1.00

+0.00

+0.00

Test Generation

+0.00

-1.00

Verification

-1.00

+1.00

(n=1) (n=1) (n=1)

-1.00

+0.00

+0.00

-1.00

(n=1)

+1.00

(n=2)

(n=1)

(n=1)

(n=1) (n=1)

(n=1)

cific

back Feed

lid.

& Va

(n=3)

(n=1)

(n=1)

-Spe

+0.67 (n=1)

(n=3)

ain Dom

1.0

(n=1)

g g ol ction ation onin ndin ontr Sele dapt Reas Grou A & . . l t w h e c c Kno Mod Stru Sear

I/O C

0.0

Performance Score (-1: Under, 0: Balanced, 1: Out)

-1.00

Bug Finding

0.5

1.0

Fig. 6. Heatmap of tasks and strategies. Green cells show areas where the single-prompt outperforms the original technique, red indicates the opposite. +1.0 means that every single paper that used this strategy for this task showed that the single-prompt outperformed the technique in the paper. 0.0 means there was an equal balance of positive and negative. −1 means every paper using this strategy for this task underperformed compared to the paper’s technique.

[EP14], and four negative results [EP23], [EP32], [EP20], [EP27], as the single-prompt approach could not derive such knowledge internally from the raw task input. These techniques generate intermediate knowledge or provide additional context, such as method dependencies [EP32], static-analysis results [EP20], [EP27], and API relationships [EP1], and pass it to subsequent stages of the methodology. The remaining categories showed no consistent pattern. We conjecture that two additional factors may contribute to these findings. First, the capability of the models used by the techniques in the papers. We expected techniques employing weaker models (see “Paper’s LLM” column in Table II) to be more susceptible to replacement, but the diversity of models, tasks, and strategies obscures any definitive pattern. Second, the evaluation datasets. Just a handful of popular benchmarks appear on multiple examined papers, so we state such trends with caution. We notice, for example, that the single-prompt outperformed every technique for the HumanEval [18] benchmark. This hints at that newer models may have utilized these benchmarks during training, however, the benchmarks’ age (e.g., HumanEval was first released in 2021) means older models likely consumed them as well. In other words, benchmark leakage could have happened in both cases, but we cannot assert how much it plays into the results. Future evaluations must keep this limitation in mind and adhere to the use of multiple contemporary benchmarks.

RQ2. Single-prompt replacement is most effective for Code Generation and Repair, particularly with Search and Selection and Structured Reasoning strategies, but remains challenging for Bug Finding, Program Analysis, and Requirement Formalization, especially when techniques rely on Knowledge Grounding or Domain-Specific Processing.

TABLE IV B EST- PERFORMING PROMPT TYPE ACROSS PAPERS . Best Prompt Type Pb

Pw

Mixed

Paper IDs [EP7], [EP23], [EP26], [EP2], [EP16], [EP32], [EP8], [EP15], [EP17], [EP21], [EP4], [EP27], [EP14], [EP22] [EP18], [EP20], [EP29], [EP28], [EP33], [EP13], [EP3], [EP31], [EP12], [EP10], [EP35], [EP34] [EP25], [EP5], [EP6], [EP1], [EP9], [EP30], [EP11], [EP19], [EP24]

Count 14

12

9

C. RQ3: The Value of White-Box Prompting The white-box prompt extends the black-box prompt with guidance derived from the original paper’s technique. We expected this additional methodological guidance to make it more effective. However, as seen in Table IV the results show a more balanced pattern. Pb outperformed Pw in 14 of the 35 papers, while Pw outperformed Pb in 12 papers. The remaining 9 papers produced mixed results, where one prompt was better on some datasets or metrics but not on all of them. Thus, the white-box guidance did not provide a consistent advantage over the simpler black-box prompt. A more detailed examination of the papers hints at why. Many of the techniques for which Pb outperform Pw depend on components or workflows that cannot be mimicked by the LLM. For example, [EP27] relies on static programdependence and slicing analysis to trace and extract the parts of the smart contract code that influence signature checks before detecting replay vulnerabilities. Similarly, [EP14] uses static analysis to identify state dependencies, branch conditions, and function-call sequences; it then runs a fuzzer over those sequences, with runtime oracles built into the fuzzing process to detect when a vulnerability is actually triggered. These examples illustrate how having access to a description of the original procedure is insufficient for Pw when the LLM cannot mimic or invoke the tools to generate the necessary

data. We note again that by design we aimed to have the simplest alternative LLM-based implementation possible. As such, we do not allow the model to invoke tools or receive feedback, however, a natural next step is to explore agentic implementations, where the model can autonomously select and invoke external tools, the key capability that distinguishes agents from plain LLM prompting. These additional instructions to guide Pw , however, do increase prompt complexity. Across all papers, only for the prompt, Pw requires an average of 80% more tokens than Pb . We conjecture that such additional information, when the model cannot leverage it, acts as a distraction and is counterproductive, explaining the weaker Pw results. We did not find any pattern in the task types to explain the performance differences between Pw and Pb . However, at the strategy level, Pb outperformed Pw for ‘Knowledge Grounding’ (5 out of 8, with 2 mixed), and slightly outperformed Pw for ‘Domain-Specific Processing’ (3 out of 5, with none mixed), and ‘Structured Reasoning’ (3 out of 9, with 4 mixed). ‘Feedback and Validation’ was the only strategy category where Pw slightly outperformed Pb (4 out of 9, with 2 mixed). These methodology descriptions often specify both the validation criteria and the feedback signal, such as test results, compiler errors, traces, or counterexamples. Although an LLM cannot execute these steps, we conjecture that these validation details may help the model anticipate how its output will be judged, and the described feedback loop may force it to revise its answer internally before producing the final output. We also compared the estimated dollar cost for running Pb and Pw that include the prompt, input, and output tokens. Pb cost $6.5 per paper on average, while Pw cost $7.5, with standard deviations of slightly over $4 for both. Thus, Pw was approximately 15% more expensive than Pb . RQ3. Pb matches or outperforms Pw in 40% to 65% of the studied SE tasks. In most cases LLMs cannot take advantage of the technical insights from the paper when provided through a single-prompt and without access to external tools. D. Threats to Validity a) External Validity: Several factors may limit the generalizability of our findings. First, while our study examines a representative sample of papers, the selection from the ICSE proceedings does not capture the breath of software engineering, and even less of the practices in the broader community. Second, accessibility constraints to models driven by version decommissioning, high costs, or evolving geopolitical and regulatory restrictions may similarly affect generalizability. For instance, over the course of this study, model providers like Anthropic updated their model families, deprecate older models, and altered global access boundaries due to compliance and security protocols. We have mitigated these concerns by documenting our paper selection criteria and choosing contemporary, widely accessible model for our evaluation.

b) Internal Validity: Threats to internal validity relate to the inherent difficulties in performing a partial replication of highly diverse empirical studies across 35 distinct research groups. Given the scale of this task, although we aimed for standardization across prompt generation, execution setups, and evaluation pipelines, our replication can only guarantee a best-effort approximation. To minimize errors, we implemented a cross-checking protocol involving multiple researchers that led to the refinement of many evaluations. Some of results depend on our categorization of techniques by task type and strategy; this classification scheme was adequate for our set of papers, but it may not be sufficient more broadly. Budget constraints bounded the evaluation depth for a subset of the analyzed papers, which we mitigated by sampling. Finally, there is an unquantifiable threat regarding the evaluation benchmarks themselves; if the models were exposed to these datasets during training, their performance may be inflated. We have explicitly documented the datasets to provide the necessary context for this threat. c) Construct Validity: Threats to construct validity concern whether our broad characterization (outperforming, mixed, underperforming) captures the performance differences of the techniques studied. To enable a unified comparison over such a highly diverse set of techniques, datasets, and benchmarks, the constructs we chose are necessarily coarse. This characterization allows us to observe trends across heterogeneous studies, but they may lack nuance to uncover smaller patterns, or subtle qualitative dimensions unique to few techniques. To mitigate this limitation we commented on the possibility of such patterns when we found them, and we provide the core data in the paper and the rest in our repository to allow researchers to scrutinize these findings. d) Conclusion Validity: The inherent heterogeneity of the papers and their evaluation process, and the small sample size of papers, prevent us from conducting a statistical significance tests on our observations. We have tempered our conclusions accordingly, presenting our findings as indicative trends rather than definitive causal relationships. VI. C ONCLUSION This paper is the first to systematically investigate the longevity and strategic value of software engineering techniques built around LLMs. Our study of 35 SE papers from ICSE 2026 makes it recent and thus relevant; our automated prompt-generation, execution, and evaluation framework ensures that our methodology is both systematic and reproducible. That said, we acknowledge that this work has limitations, particularly regarding fixed execution budgets, dataset sampling constraints, and categorization schemas, and that it represents just the first step in this line of inquiry. Our findings reveal that between 37% and 63% of SE techniques published less than six months ago can be effectively replaced by the simplest possible instance of LLM invocation: a single, automatically generated prompt executed on a newer generation model, without any iterative refinement or access to other tools. These findings provide a warning to the

software engineering research community about the ephemeral nature and short half-life of many of the heavily engineered workarounds our community is currently producing. Our findings also reveal that techniques aimed at constructive tasks like generating code or repairing code are the easiest to be replaced as frontier models get more powerful. For tasks requiring deeper semantic understanding or relying on feedback loops, existing techniques still fare better, but some of the performance gains could be matched if the single prompts are simply placed within a basic loop, delegated to agents, or granted access to other tools, something that future work should explicitly investigate. On the other hand, papers involving domain-specific processing or project-wide knowledge grounding are the least likely to be replaced. Overall, these insights point to the critical need for our community to be far more strategic in our research investments. We are currently relearning AI’s ‘Bitter Lesson’ within the software engineering domain: heavily engineered human heuristics designed to patch temporary model deficits are rapidly cleared away by computational scaling. Enduring software engineering research must shift from building transient scaffolds for LLMs towards establishing where symbolic approaches can add complementary value. Our community needs to stop chasing the moving target of model capabilities and instead build techniques that scale alongside them.

R EFERENCES [1] ACM, “ICSE 2026 research track,” https://conf.researchr.org/track/icse2026/icse-2026-research-track, 2026. [2] Anthropic, “Claude fable 5 and claude mythos 5,” https://www.anthropi c.com/news/claude-fable-5-mythos-5, 2026. [3] L. Beurer-Kellner, M. Fischer, and M. Vechev, “Prompting is programming: A query language for large language models,” vol. 7, no. PLDI, Jun. 2023. [Online]. Available: https://doi.org/10.1145/3591300 [4] B. Chen, Z. Zhang, N. Langrené, and S. Zhu, “Unleashing the potential of prompt engineering for large language models,” Patterns, vol. 6, no. 6, p. 101260, 2025. [Online]. Available: https://www.sciencedirect. com/science/article/pii/S2666389925001084 [5] D. DeBellis, K. Storer, N. Harvey, M. Beane, R. Edwards, E. Fraser, B. Good, E. Kalliamvakou, G. Kim, E. Maxwell, S. D’Angelo, S. Inman, A. Murillo, and D. Villalba, “2025 DORA report: State of AI-assisted software development,” Google Cloud, Tech. Rep., 2025. [Online]. Available: https://dora.dev [6] Y. Dong, X. Jiang, J. Qian, T. Wang, K. Zhang, Z. Jin, and G. Li, “A survey on code generation with llm-based agents,” ArXiv, vol. abs/2508.00083, 2025. [Online]. Available: https://api.semanticscholar.or g/CorpusID:280416987 [7] A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, “Large language models for software engineering: Survey and open problems,” in 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE), 2023, pp. 31–53. [8] Gartner, Inc., “Gartner predicts AI coding costs will surpass average developer’s salary by 2028 as token consumption surges,” Gartner Press Release, June 2026. [Online]. Available: https://www.gartner.com/en/new sroom/press-releases/2026-06-24-gartner-predicts-ai-coding-costs-will-s urpass-average-developer-salary-by-2028-as-token-consumption-surges [9] G. F. Gunawan and M. Amien, “Comprehensive evaluation of large language models on software engineering tasks: A multi-task benchmark,” 2026. [Online]. Available: https://arxiv.org/abs/2602.07079 [10] J. Guo, S. Huang, M. Li, D. Huang, X. Chen, R. Zhang, Z. Guo, H. Yu, S.-M. Yiu, P. Lio, and K.-Y. Lam, “A comprehensive survey on benchmarks and solutions in software engineering of llm-empowered agentic system,” 2025. [Online]. Available: https: //arxiv.org/abs/2510.09721 [11] J. He, J. Shi, T. Y. Zhuo, C. Treude, J. Sun, Z. Xing, X. Du, and D. Lo, “Llm-as-a-judge for software engineering: Literature review, vision, and the road ahead,” ACM Trans. Softw. Eng. Methodol., Feb. 2026, just Accepted. [Online]. Available: https://doi.org/10.1145/3797276 [12] X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engineering: A systematic literature review,” ACM Transactions on Software Engineering and Methodology, vol. 33, pp. 1–79, December 2024. [Online]. Available: https://doi.org/10.1145/3695988 [13] X. Hu, F. Niu, J. Chen, X. Zhou, J. Zhang, J. He, X. Xia, and D. Lo, “Assessing and advancing benchmarks for evaluating large language models in software engineering tasks,” ACM Trans. Softw. Eng. Methodol., Dec. 2026, just Accepted. [Online]. Available: https://doi.org/10.1145/3786771 [14] N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica, “Livecodebench: Holistic and contamination free evaluation of large language models for code,” in The Thirteenth International Conference on Learning Representations, 2025. [Online]. Available: https://openreview.net/forum?id=chfJJYC3iL [15] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “SWE-bench: Can language models resolve real-world github issues?” in The Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/forum ?id=VTF8yNQM66 [16] N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language models use long contexts,” Transactions of the Association for Computational Linguistics, vol. 12, pp. 157–173, 02 2024. [Online]. Available: https://doi.org/10.1162/tacl a 00638 [17] A. Muttakin, S. Mondal, and C. K. Roy, “The state of open science in software engineering research: A case study of icse artifacts,” arXiv preprint arXiv:2601.02066, 2026. [18] OpenAI, “Humaneval,” https://huggingface.co/datasets/openai/openai h umaneval, 2021.

[19] L. Reynolds and K. McDonell, “Prompt programming for large language models: Beyond the few-shot paradigm,” in Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, ser. CHI EA ’21. New York, NY, USA: Association for Computing Machinery, 2021. [Online]. Available: https://doi.org/10.1145/3411763.3451760 [20] P. Sahoo, A. K. Singh, S. Saha, V. Jain, S. Mondal, and A. Chadha, “A systematic survey of prompt engineering in large language models: Techniques and applications,” 2025. [Online]. Available: https://arxiv.org/abs/2402.07927 [21] E. Sarıkayak, W. Gu, H. Ghonim, and C. Chen, “Evaluating llms on real-world software performance optimization,” 2026. [Online]. Available: https://arxiv.org/abs/2606.25530 [22] S. Schulhoff, M. Ilie, N. Balepur, K. Kahadze, A. Liu, C. Si, Y. Li, A. Gupta, H. Han, S. Schulhoff, P. S. Dulepet, S. Vidyadhara, D. Ki, S. Agrawal, C. M. Pham, G. C. Kroiz, F. Li, H. Tao, A. Srivastava, H. D. Costa, S. Gupta, M. L. Rogers, I. Goncearenco, G. Sarli, I. Galynker, D. Peskoff, M. Carpuat, J. White, S. Anadkat, A. M. Hoyle, and P. Resnik, “The prompt report: A systematic survey of prompting techniques,” ArXiv, vol. abs/2406.06608, 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID:270380093 [23] M. Shiraishi, Y. Cao, and T. Shinagawa, “Smartc2rust: Iterative, feedback-driven c-to-rust translation via large language models for safety and equivalence,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026. [24] R. Sutton. (2019) The bitter lesson. [Online]. Available: http: //www.incompleteideas.net/IncIdeas/BitterLesson.html [25] Y. Wang, W. Zhong, Y. Huang, E. Shi, M. Yang, J. Chen, H. Li, Y. Ma, Q. Wang, and Z. Zheng, “Agents in software engineering: survey, landscape, and vision,” Automated Software Engg., vol. 32, no. 2, Aug. 2025. [Online]. Available: https://doi.org/10.1007/s10515-025-00544-2 [26] J. White, Q. Fu, S. Hays, M. Sandborn, C. Olea, H. Gilbert, A. Elnashar, J. Spencer-Smith, and D. C. Schmidt, “A prompt pattern catalog to enhance prompt engineering with chatgpt,” in Proceedings of the 30th Conference on Pattern Languages of Programs, ser. PLoP ’23. USA: The Hillside Group, 2023. [Online]. Available: https://dl.acm.org/doi/10.5555/3721041.3721046 [27] C. Yang, X. Li, M. R. H. Misu, J. Yao, W. Cui, Y. Gong, C. Hawblitzel, S. Lahiri, J. R. Lorch, S. Lu, F. Yang, Z. Zhou, and S. Lu, “Autoverus: Automated proof generation for rust code,” Proc. ACM Program. Lang., vol. 9, no. OOPSLA2, Oct. 2025. [Online]. Available: https://doi.org/10.1145/3763174 [28] C. Yang, N. Neamtu, C. Hawblitzel, J. R. Lorch, and S. Lu, “Verusage: A study of agent-based verification for rust systems,” 2026. [Online]. Available: https://arxiv.org/abs/2512.18436

E VALUATED PAPERS [EP1] W. Akram, Y. Jiang, H. A. Khan, F. Jalil, and H. Liu, “Llm-based api argument completion with knowledge-augmented prompts,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026. [EP2] D. S. Bouras, Y. Dai, T. Wang, Y. Xiong, and S. Mechtaev, “Hoareprompt: Structural reasoning about program correctness in natural language,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026. [EP3] H. Cho, J. Lee, and W. Lee, “Distance-guided search in program synthesis with imperfect llm solutions,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026. [EP4] G. Crupi, R. Tufano, and G. Bavota, “Improving code generation via small language model-as-a-judge,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026. [EP5] S. Gao, C. Wang, C. Gao, and M. R. Lyu, “Seer: Enhancing chainof-thought code generation through self-exploring deep reasoning,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026. [EP6] S. Gu, N. Nashid, and A. Mesbah, “Llm test generation via iterative hybrid program analysis,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026.

[EP7] W. Gu, J. Chen, Y. Wang, T. Jiang, X. Li, M. Liu, X. Liu, Y. Ma, and Z. Zheng, “What to retrieve for effective retrieval-augmented code generation? an empirical study and beyond,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026. [EP8] J. Hu, J. Sun, C. Wang, Y. Huang, J. Feng, Y. Xu, Y. Li, K. Wang, W. Miao, J. S. Dong, and G. Pu, “Adarule: Llm-driven natural language to ltl conversion via pattern-adaptive rule induction,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026. [EP9] J. Huang, W. Ye, W. Sun, J. Zhang, M. Zhang, and Y. Liu, “Tracecoder: A trace-driven multi-agent framework for automated debugging of llm-generated code,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026. [EP10] Y. Huang, M. Hu, X. Chen, Z. Jin, and S. Xiao, “Modeling like peeling an onion: Layerwise analysis-driven automatic behavioral model generation,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026. [EP11] H. Huynh, T. Le, T. Nguyen, V. Nguyen, V. Nguyen, and T. N. Nguyen, “Rbctest: Leveraging llms to mine and verify oracles of api response bodies for restful api testing,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026. [EP12] M. M. Imran, R. Zita, R. R. Rahman, P. Chatterjee, and K. Damevski, “Toxicity ahead: Forecasting conversational derailment on github,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026. [EP13] C. C. Le, C. D. Van, T. D. Vu, M. V. T. Pham, N. H. Phan, H. Phan, and T. N. Nguyen, “Testweaver: Execution-aware, feedbackdriven regression testing generation with large language models,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026. [EP14] J. Li, P. Qian, G. Li, R. Wang, P. Wang, Z. Tang, F. Ma, Y. Chen, and L. Zhang, “Echofuzz: Empowering smart contract fuzzing with large language models,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026. [EP15] Y. Li, J. Li, G. Li, and Z. Jin, “Adaptrack: Constrained decoding without distorting llm’s output intent,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026. [EP16] F. Liu, S. Liu, Y. Zhu, X. Lian, and L. Zhang, “Securereviewer: Enhancing large language models for secure code review through secure-aware fine-tuning,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026. [EP17] W. Luo, J. W. Keung, B. Yang, J. Klein, T. F. Bissyande, H. Tian, and B. Le, “Unlocking llm repair capabilities through cross-language translation and multi-agent refinement,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026. [EP18] T. Mahmud, B. Duan, C. Păsăreanu, and G. Yang, “Enhancing llm code generation with ensembles: A similarity-based selection approach,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026. [EP19] D. Mendoza, A. Mavridou, A. Katis, and C. Trippel, “Automating requirements formalization: Using llms and low-complexity distinguishing traces for semantic validation,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026. [EP20] X. Rong, A. Yadavally, and T. N. Nguyen, “Large language modelaided partial program dependence analysis,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026. [EP21] Y. Shi, S. Wang, C. Wan, M. Wang, and X. Gu, “From code to correctness: Closing the last mile of code generation with hierarchical debugging,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026. [EP22] X. Song, Y. Wu, B. Chen, Z. Lu, S. Liu, and X. Peng, “Characterizing regression bug-inducing changes and improving llm-based regression bug detection,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026.

[EP23] Y. Tao, H. Zhang, V.-H. Le, and Y. Xiao, “Knowledge-augmented log anomaly detection with large language models,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026. [EP24] Z. Tian and J. Chen, “Aligning requirement for large language model’s code generation (specine),” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026. [EP25] Y. Wang, P. Chen, H. Huang, Z. He, G. Tan, C. Zhang, J. He, and Z. Zheng, “Inferlog: Accelerating llm inference for online log parsing via icl-oriented prefix caching,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026. [EP26] Y. Wang, M. Pradel, and Z. Liu, “Are ”solved issues” in swe-bench really solved correctly? an empirical study,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026. [EP27] Z. Wang, J. Chen, Z. Lin, W. Chen, K. Ning, J. Yu, Y. Feng, Y. Zhang, W. Zhang, and Z. Zheng, “One signature, multiple payments: Demystifying and detecting signature replay vulnerabilities in smart contracts,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026. [EP28] R. Widyasari, M. Weyssow, I. C. Irsan, H. W. Ang, F. Liauw, E. L. Ouh, L. K. Shar, H. J. Kang, and D. Lo, “Let the trial begin: A mockcourt approach to vulnerability detection using llm-based agents,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026. [EP29] X. Xie, Y. Zhang, W. Dou, Y. Gao, Z. Cui, J. Song, R. Yang, and J. Wei, “Rise: Rule-driven sql dialect translation via query reduction,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026. [EP30] Z. Xie, Z. Li, X. He, S. Zhang, L. Xu, Y. Yang, T. Zhang, J. Chen, R. Shi, and D. Pei, “Foundroot: Towards foundation model for root cause analysis via structured deep thinking,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026. [EP31] W. Xu, H. Ye, K. Gao, and M. Zhou, “Small changes, big trouble: Demystifying and parsing license variants for incompatibility detection in the pypi ecosystem,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026. [EP32] A. Yadavally and T. N. Nguyen, “From seed to scope: Reasoning to identify change impact sets,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026. [EP33] B. Yang, L. Ren, X. Yin, J. Ren, H. Tian, and S. Jin, “Input reduction enhanced llm-based program repair,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026. [EP34] J. Zhang, K. Huang, J. Zhang, Y. Liu, and C. Chen, “Repair ingredients are all you need: Improving large language model-based program repair via repair ingredients search,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026. [EP35] X. Zheng, N. Li, X. Luan, K. Wang, L. Shi, M. Sun, and H. Wang, “Beyond correctness: Exposing llm-generated logical flaws in reasoning via multi-step automated theorem proving,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ser. ICSE 2026, 2026.

Related documents

• Where Do LLMs Decide to Break the Rules? Mechanistic Localization of Prompt Injection Compliance#1142049
• Where Do LLMs Decide to Break the Rules? Mechanistic Localization of Prompt Injection Compliance#1122191
• RAISE: Diagnosing Acquisition Collapse in Costly LLM Signals#1131639
• How Much Prompt Is Enough? A Blackbox Minimization of Few-Shots in LLMs#1131131
• How Much Prompt Is Enough? A Blackbox Minimization of Few-Shots in LLMs#1122378