Foundation Models as Oracles for Refactoring Correctness Detection
arXiv:2605.02096v1 [cs.SE] 3 May 2026
Rohit Gheyi · Rian Melo · Jonhnanthan Oliveira · Márcio Ribeiro · Baldoino Fonseca
Abstract Refactoring tools in popular Integrated Development Environments (IDEs) can introduce unintended behavioral changes or compilation errors, a persistent challenge that undermines developer trust in automated transformations. Traditional detection approaches rely on handcrafted preconditions, and static and dynamic analyses, yet remain limited in adaptability and can miss subtle correctness issues. This study examines the potential of foundation models to serve as oracles for detecting refactoring bugs in Java programs. We evaluate zero-shot prompting, without task-specific training, across 226 real refactoring bugs collected over more than a decade from widely used Java IDEs (IntelliJIDEA, Eclipse, and NetBeans), spanning 47 refactoring types. Our results indicate that foundation models can be effective for this task, although performance varies across models. In the first-run setting, GPT-OSS-20B achieved 80.5% accuracy, while GPT-5.4 reached 93.8%. We also evaluated other open and proprietary models: Gemma-4-31B achieved the strongest result among open models, and Gemini-3.1-Pro-Preview achieved the best overall result among all evaluated models. Metamorphic testing further shows that model predictions are largely consistent under intended semantics-preserving code variations, suggesting that superficial pattern matching may not fully account for the observed behavior. Beyond detection accuracy, foundation models can provide short explanations that may help support developer inspection, operate across refactoring types without explicitly encoded refactoring-specific rules, and may serve as lightweight triage aids in development workflows. Our findings suggest that foundation models can complement traditional refactoring checks by flagging suspicious transformations for developer inspection. R. Gheyi and R. Melo and J. Oliveira Federal University of Campina Grande E-mail: [email protected], [email protected], [email protected] M. Ribeiro and B. Fonseca Federal University of Alagoas E-mail: [email protected], [email protected]
2
Rohit Gheyi et al.
Keywords Refactoring, Foundation Models, Oracle, Behavior Change, Compilation Error.
1 Introduction Refactoring [1, 2], the systematic restructuring of code to improve its internal quality while preserving observable behavior, has become a cornerstone of modern software development. Since Roberts [3] introduced the first automated refactoring tool, popular Integrated Development Environments (IDEs) such as IntelliJ-IDEA, Eclipse, and NetBeans have integrated refactoring implementations that make complex transformations accessible to large developer communities. This automation is not merely a convenience but an important form of support: recent empirical studies reveal that over 40% of developers perform refactorings daily [4], making automated support important for maintaining code quality and supporting the long-term sustainability of increasingly complex software systems. Despite decades of research and engineering effort, automated refactoring tools remain imperfect: they can still introduce subtle behavioral changes that violate the core promise of behavior preservation. This challenge represents more than a technical inconvenience: it undermines developer trust and can lead to production failures that escape both code review and existing test suites. The difficulty stems from the inherent complexity of ensuring behavior preservation across program transformations, where seemingly straightforward changes can have far-reaching and unexpected consequences. Traditional approaches to refactoring correctness rely on handcrafted preconditions and extensive static and dynamic analyses [5, 6,7]. Tools like SafeRefactor [8] generate and execute test cases to expose unintended changes, while others implement complex, transformation-specific analyses [9]. However, these approaches face important limitations: defining sound and a set of complete preconditions is challenging [10], implementing the corresponding analyses requires substantial engineering effort, and proposing new analyses for additional refactoring scenarios is far from straightforward. Moreover, such approaches are difficult to adapt to new refactoring types, must continuously evolve alongside programming languages, or both. The complexity is compounded by the rich semantics of modern programming languages. While individual language constructs may appear straightforward in isolation, their interactions create a combinatorial explosion of edge cases. Java exemplifies this challenge with its numerous traps, pitfalls, and corner cases [11] involving inheritance, method resolution, exception handling, and type inference. This complexity may explain why many developers continue to prefer manual refactorings despite the availability of automated tools [12]. They simply do not trust automated refactoring implementations to preserve program behavior reliably. The emergence of foundation models has catalyzed a paradigm shift in software engineering, demonstrating strong performance in code understanding,
Foundation Models as Oracles for Refactoring Correctness Detection
3
generation, and analysis tasks [13, 14]. Unlike traditional rule-based approaches that require explicit specification of every possible edge case, foundation models can exhibit capabilities that may support reasoning-like behavior and allow them to perform across diverse scenarios without task-specific training. This generalization potential is particularly compelling for refactoring validation, where the space of possible transformations and their interactions is vast and evolving. In this article, we use the term foundation models to refer to general-purpose models trained on broad data and adaptable to many downstream tasks. Large language models (LLMs) are a subclass of foundation models specialized in processing and generating natural language. They can analyze code at multiple levels of abstraction simultaneously, identify semantic relationships that may not be explicitly encoded in traditional analyses, and provide natural language rationales for their decisions, helping developers inspect why a transformation might be problematic rather than simply flagging it as incorrect. In addition, some open models can be executed locally or on low-cost cloud hardware, making them attractive candidates for integration into development workflows where cost, latency, and privacy matter. However, the application of foundation models to refactoring correctness remains relatively unexplored. While these models have shown promise in various software engineering tasks [13], their effectiveness in detecting the subtle behavioral changes that characterize refactoring bugs, particularly their ability to distinguish between surface-level code changes and potential behavior-changing differences, represents an open research question [15]. This article investigates the extent to which foundation models may serve as complementary oracle-like components for detecting behavioral changes (BCs) and compilation errors (CEs) introduced by Java program refactorings. We address this question through an empirical study that evaluates foundation models on real-world refactoring bugs rather than synthetic examples, providing evidence about their possible use in development settings. Our evaluation encompasses 226 refactoring bugs collected over more than a decade from three widely used Java IDEs (IntelliJ-IDEA, Eclipse, NetBeans), spanning 47 distinct refactoring types. This dataset represents a broad sample of refactoring challenges encountered in practice, including both compilation errors and subtle behavioral changes that traditional tools often miss. We employ zeroshot prompting in the main benchmark, and use MetaPrompting to derive a complementary diff-oriented prompt for large-project feasibility analysis. This allows us to assess model behavior without domain-specific training while also exploring how the approach behaves when full project context is unavailable. Our key findings indicate that foundation models can achieve encouraging accuracy in refactoring-bug detection. In the first-run setting, the open model GPT-OSS-20B attains 80.5% accuracy, while the proprietary GPT-5.4 model reaches 93.8%. Additional proprietary and open models show that performance varies across model families. Importantly, metamorphic testing provides evidence that these results are not explained solely by exact-match memorization of the original inputs. A possible cost-conscious deployment strategy is suggested by the results: use inexpensive local or open models for initial screening
4
Rohit Gheyi et al.
and selectively employ stronger proprietary models for uncertain or high-risk cases, balancing accuracy, latency, and resource utilization. The implications of our findings may extend beyond refactoring to software engineering tasks that require reasoning about behavioral preservation. Compiler-optimization validation, mutation testing, and programtransformation verification face the same core challenge of distinguishing behavior-preserving from behavior-changing transformations [16, 17,18,19]. Our results suggest that foundation models may serve as complementary oraclelike components for semantic equivalence assessment across these domains. The integration of foundation models into modern development environments, exemplified by AI-enabled IDEs such as Antigravity [20], Windsurf [21] and Cursor [22], offers a plausible path for further integration. Rather than replacing existing verification techniques, foundation models can augment them by providing explanation-oriented triage that may reduce the computational burden on heavyweight analyses while improving developers’ understanding of transformation risks. Open challenges remain, including context-window limits, cost and latency, model non-determinism, and the choice of behavioralequivalence criteria. Nevertheless, the results indicate that foundation models are a promising complement to existing refactoring-validation techniques. This article is organized as follows. Section 2 presents a motivating example that illustrates the challenges of refactoring correctness. Section 3 details our experimental methodology, including dataset construction, model selection, and evaluation metrics. Section 4 presents our empirical findings across different models and refactoring types. Section 5 discusses our results. Section 6 situates our work within the broader landscape of refactoring research and foundation model applications. Finally, Section 7 concludes with a synthesis of our contributions and directions for future research.
2 Motivating Example Ensuring that a refactoring transformation preserves program correctness requires satisfying a set of preconditions. These preconditions are intended to ensure that the resulting program compiles, and preserves the original behavior. For instance, in the Extract Class refactoring [1], the newly introduced class must not conflict with an existing class name. In practice, developers may perform refactorings manually, an error-prone and time-consuming process, or rely on automated tools available in IDEs such as VSCode [23], IntelliJIDEA [24], Eclipse [25], and NetBeans [26]. Behavior preservation is typically verified by compiling the refactored program and executing its test suite [1]. While this strategy is often effective at identifying compilation errors, it may fail to capture more subtle behavioral deviations. Rachatasumrit and Kim [27] find that, in practice, many developers’ test suites are insufficient, as they do not adequately exercise many refactored methods and fields. Moreover, many refactoring implementations
Foundation Models as Oracles for Refactoring Correctness Detection
5
check only a subset of the necessary preconditions rather than encoding them comprehensively, which increases the likelihood of correctness violations. To illustrate, consider the program in Figure 1(a), where method m in class B invokes super.k(). Applying the Push Down Method refactoring in Eclipse JDT moves B.m to subclass C, producing the program in Figure 1(b). Although compilation succeeds, the behavior changes: executing C.main prints 10 before refactoring and 20 afterwards. This Eclipse bug1 occurs because the refactoring fails to update the call to super.k(), causing the call to resolve to a different method implementation.
(a) Original program
(b) Refactored program
public class A { public int k() { return 10; } } public class B extends A { public int k() { return 20; } public int m() { return super.k(); } } public class C extends B { public static void main(String[] ,→ args) { C c = new C(); System.out.println(c.m()); } }
public class A { public int k() { return 10; } } public class B extends A { public int k() { return 20; } } public class C extends B { public int m() { return super.k(); } public static void main(String[] ,→ args) { C c = new C(); System.out.println(c.m()); } }
Fig. 1: Applying Push Down Method B.m to C using Eclipse JDT introduces a behavioral change.
As another example, consider applying Extract Method in IntelliJ-IDEA to the conditional inside m. In the original program (Figure 2(a)), the execution prints ok because the array access x[0] is protected by the guard condition c. After refactoring (Figure 2(b)), however, IntelliJ-IDEA with Fold parameters enabled extracts the expression x[0] as an argument in the call g(c, x[0]). This transformation alters the original control flow: since Java eagerly evaluates method arguments, x[0] is now evaluated unconditionally before entering the method, leading to an ArrayIndexOutOfBoundsException that did not occur in the original code. This behavioral change constitutes a correctness bug in the refactoring tool, which was subsequently reported to IntelliJ-IDEA.2 While this issue may be obvious in such a small program, detecting similar problems in larger, real-world codebases is more challenging. Assessing the correctness of a transformation requires careful handling of subtle language semantics and numerous corner cases, which is a nontrivial task. Even experienced IntelliJIDEA developers were unable to understand the behavioral change described in the initial bug report. 1
https://bugs.eclipse.org/bugs/show_bug.cgi?id=356698 https://youtrack.jetbrains.com/issue/IDEA-92815/Extract-method-refactoringwith-Fold-parameters-is-too-clever-breaks-functioning-code 2
6
Rohit Gheyi et al.
(a) Original program
(b) Refactored program
class A { public static void main(String[] a) { new A().m(); } void m(){ Object[] x = {}; boolean c = false; if (c) System.out.println(x[0]); else System.out.println("ok"); } }
class A { public static void main(String[] a) { new A().m(); } void m(){ Object[] x = {}; boolean c = false; g(c, x[0]); } void g(boolean c, Object y){ if (c) System.out.println(y); else System.out.println("ok"); } }
Fig. 2: Applying Extract Method using IntelliJ-IDEA introduces a behavioral change.
Although simple, this example mirrors issues reported in real-world systems. Gligoric et al. [28] studied five large open-source Java projects and uncovered 77 refactoring-related bugs in Eclipse, many resembling the problem in Figure 1. Such bugs are not isolated: IDEs including IntelliJ-IDEA, VSCode, and NetBeans have exhibited similar issues due to the inherent difficulty of defining complete correctness preconditions. Formalizing refactoring semantics is difficult [10], leading many tools to implement only partial checks. As a result, developers may distrust automated tools and prefer manual refactorings [12]. Prior research has proposed several techniques to test and improve the correctness of refactoring implementations [5,6,7,29,30]. For example, SafeRefactor [8] automatically generates test cases to detect unintended behavioral changes introduced by transformations, while other approaches rely on specialized type-specific analyses and carefully designed refactoring conditions [9]. Although effective in some settings, these techniques are often labor-intensive and costly to adapt to new refactoring types. Moreover, their effectiveness depends on the adequacy [31] of the underlying validation mechanisms: existing test suites are often insufficient to reveal all behavioral changes introduced by refactorings [27], and static or dynamic analyses may fail to capture subtle interactions among language constructs. Refactoring implementations must account for a broad range of programming-language constructs, which is far from trivial. Even when the semantics of individual constructs seem simple in isolation, their interactions frequently give rise to subtle and unexpected behaviors. Java, for instance, is well known for its many traps, pitfalls, and corner cases, which make assessing program transformations particularly challenging [11]. This complexity makes it difficult to define and enforce complete preconditions for all refactorings. As a result, existing tools do not reliably detect all behavior-changing transformations, and refactoring correctness remains a persistent open problem.
Foundation Models as Oracles for Refactoring Correctness Detection
7
Fig. 3: Overview of the research method. Stage 1 summarizes dataset construction, including filtering, reconstruction, and validation of 226 refactoring bugs spanning multiple refactoring types. Stage 2 summarizes the evaluation pipeline, including stability and sensitivity assessment with metamorphic operators, prompt-based model inference with GPT-OSS-20B and GPT-5.4, correctness assessment, and computation of accuracy and stability metrics.
More recently, foundation models have emerged as promising complementary techniques for software analysis. Rather than replacing existing validation mechanisms used by developers and refactoring tools, such as compiler checks, static analyses, and regression testing, we position foundation models as an additional oracle-like check over the transformation. In contrast to traditional static and dynamic techniques, which are often computationally expensive and tightly coupled to language-specific semantics, foundation models may provide relatively lightweight analyses, in some cases on consumer-grade hardware [13, 14]. Such analyses can provide additional evidence about whether a transformation may introduce a compilation error or a behavioral change, especially when available tests or handcrafted analyses are incomplete. However, their effectiveness in detecting refactoring-related bugs, especially those involving subtle behavioral changes, remains, to the best of our knowledge, an open research question [15].
3 Research Method This section details the research method employed in our study. Figure 3 presents an overview of the research method, which is detailed next.
8
Rohit Gheyi et al.
3.1 Research Goal and Questions The objective of this study is defined as follows: analyze the accuracy of foundation models for the purpose of detecting refactoring bugs, including compilation errors and behavioral changes, with respect to previously reported refactoringrelated bugs from the point of view of software engineering researchers. We define the following research questions: RQ1 To what extent can an open foundation model (GPT-OSS-20B) detect refactoring bugs, including compilation errors and behavioral changes? RQ2 To what extent can a proprietary foundation model (GPT-5.4) detect refactoring bugs, including compilation errors and behavioral changes? RQ3 To what extent is the performance of these models affected by potential data leakage?
3.2 Dataset We evaluate 226 refactoring bugs reported across major Java IDEs, namely IntelliJ-IDEA (Mar 2016–Mar 2024), Eclipse (Mar 2005–Jun 2023), and NetBeans (Apr 2012–Apr 2024). Our study builds on the dataset organized by Wang et al. [32], from which we retain cases where the transformation introduced compilation errors (CEs) or behavioral changes (BCs). We extend the dataset of Wang et al. [32], a minor contribution of this article, by extracting, for each report, both the original Java program before the refactoring and the resulting Java program produced after the faulty transformation. For several reports, this process required manual reconstruction because the original submissions were not directly executable artifacts, but rather bug reports mixing code fragments, natural-language explanations, and follow-up discussion [33]. In some cases3 , the bug report already contained a complete executable example, which we reused in our dataset after minor normalization steps, such as removing comments. In other cases4 , the report provided only partial code snippets embedded in the description or discussion thread; for those instances, we reconstructed the missing class declarations and surrounding program context from the reported fragments and the clarifications provided in follow-up comments. Our final dataset contains 226 bugs in total, including 185 compilation errors and 41 behavioral changes. Regarding tool distribution, Eclipse accounts for 73 bugs, NetBeans for 89, and IntelliJ-IDEA for 64. The original program size of the examples ranges from 3 to 28 LOC, with an average of 10.8 LOC per instance. To minimize bias in model evaluation, we removed all code comments from the examples. We also performed a sanity check to avoid redundancy by collapsing similar bug reports that affect multiple tools into a single representative case. 3 4
ID 97: https://github.com/eclipse-jdt/eclipse.jdt.ui/issues/1530 ID 257: https://bugs.eclipse.org/bugs/show_bug.cgi?id=87483
Foundation Models as Oracles for Refactoring Correctness Detection
9
For every instance in the dataset, we store both Java versions involved in the refactoring scenario: the original program and the resulting program. To validate the reconstruction, we compiled all original and resulting programs using javac from OpenJDK-Temurin-21.0.7+6 and stored the corresponding compiler logs in the dataset. Every original program compiles successfully. For instances labeled as compilation errors, the resulting program fails to compile, consistently with the reported bug; the corresponding compiler diagnostics provide executable evidence for the compilation-error label. For instances labeled as behavior changes, the resulting program compiles successfully, again consistently with the report, but its behavior differs from that of the original program. In this case, we additionally provide one JUnit test class containing a single test case that exposes the behavioral difference. Each such test case was compiled successfully against both the original and the resulting programs using the same Java environment, and we store the corresponding test-compilation logs. We then executed the same test unchanged on both versions and stored the execution results. In all 41 cases, the test passes on the original program and fails on the resulting program, thereby confirming the reported behavior change after refactoring through executable evidence. None of these test cases relies on Java reflection. The Java programs in our dataset exercise a broad range of Java features that commonly interact with refactoring behavior. They include fundamental object-oriented constructs such as classes, packages, imports, access modifiers, fields, methods, constructors, inheritance, interfaces, abstract classes, method overloading and overriding, field hiding, and qualified member access through this, super, and enclosing instances. The programs also cover several forms of nested and local declarations, including member classes, static nested classes, local classes, anonymous classes, nested interfaces, and enums. The dataset further includes common control-flow and expression-level constructs, such as conditionals, loops, enhanced for loops, switch statements and switch expressions, ternary expressions, array accesses, varargs, casts, instanceof checks, assignments with side effects, and try-catch and try-with-resources statements. It also exercises type-system features such as generics, bounded type parameters, raw types, wildcards, generic methods, arrays, primitive and boxed types, and final fields and variables. In addition, several programs rely on annotations, static imports, static members, initializer blocks, synchronization, exceptions, and standard-library APIs. The programs also include more recent Java constructs, including lambda expressions, method references, functional interfaces, default and static interface methods, var-based local type inference, and switch expressions with yield. The dataset also spans a broad and diverse spectrum of refactorings reported by practitioners. Table 1 summarizes the ten most frequent refactoring types in the dataset, with all remaining cases grouped into Others. The most frequent categories are Move Method, Inline Method, Pull Up Method, Extract Local Variable, Rename Method, Change Method Signature, Extract Method, Move Class, Inline Variable, and Push Down Method. Beyond these common categories, the dataset also includes Add Parameter, Encapsulate Field, Pull
10
Rohit Gheyi et al.
Table 1: Top 10 refactoring types in the dataset. Refactoring Move Method Inline Method Pull Up Method Extract Local Variable Rename Method Change Method Signature Extract Method Move Class Inline Variable Push Down Method Others
Instances 32 20 18 15 14 13 12 11 10 6 75
Up Field, Push Down Field, Rename Class, Rename Field, Rename Variable, Introduce Variable, Pull Up Interface, Introduce Constant, Extract Superclass, Generalize Type, Join Variable Declaration, Extract Variable, Introduce Factory, Extract Class, Inline Field, Convert Anonymous to Nested Class, Refactoring to Conditional, Introduce Field, Extract Variable as Enum, Replace Inheritance with Delegation, Replace Anonymous with Lambda, Convert to AtomicBoolean, Delete Unused Variable, Extract Parameter, Extract Interface, Replace Constructor with Factory, Convert Boolean, Delete Unused Method, Convert to Instance Method, Type Migration, Inline Interface, Change Class Signature, Move Field, Move Inner to Outer Level, and Replace Constructor with Builder. This diversity is important because it exposes the models to both frequent and less common transformations, ranging from method- and field-level changes to broader type- and design-level restructurings.
3.3 Behavior Preservation Notion In this study, we adopt a client-observable notion of behavioral equivalence tailored to the analysis of refactoring bugs. This notion is aligned with the equivalence notion used in SafeRefactor [8]. Given an initial program and a resulting program produced after a refactoring, behavior is considered to be preserved only if the resulting program remains compilable and both versions exhibit the same observable behavior for the exercised public APIs and externally visible effects of the methods they have in common. Thus, if the initial program compiles but the resulting program does not, we treat the transformation as behavior-changing, since client code can no longer execute the transformed program. For pairs in which both versions compile, two programs are considered behaviorally equivalent if the same client-level code, executed unchanged against both versions, produces the same return values, printed output, thrown exceptions, and externally visible state changes when exercising their public interface. This definition deliberately focuses on effects visible to clients of the program, rather than on internal implementation details, since
Foundation Models as Oracles for Refactoring Correctness Detection
11
refactorings are expected to preserve externally observable behavior even when the internal structure changes. This notion is reflected in the kinds of behavioral deviations present in our dataset. In some cases, the difference is directly observable when executing the main method, for instance through changes in printed output. In many other cases, the deviation becomes evident only when invoking specific methods and checking their return values or the values they print. A smaller subset of bugs manifests through more subtle externally visible changes, such as altered method-call resolution, modified field access, changes in the accessible API, or the introduction of new exceptions. Our notion of behavior preservation is designed to capture not only output mismatches, but also failures that affect which method is invoked, whether a call throws an exception, or whether the visible program state observed through public operations remains the same. The motivating examples in Section 2 illustrate how this notion of equivalence is applied in practice. In the Push Down Method example from Figure 1, both versions compile and can be executed through the same public entry point, namely C.main. However, the observable output changes from 10 in the original program to 20 in the refactored program. Under our notion, this is a behavioral change because a client executing the same public operation observes a different printed value. Similarly, in the Extract Method example from Figure 2, the original program executes A.main and prints ok, whereas the refactored program evaluates x[0] before entering the extracted method and therefore throws an ArrayIndexOutOfBoundsException. This also violates behavior preservation: the same client-level execution changes from normal termination with an observable output to abnormal termination with an exception. To make this notion operational, for each of the 41 behavioral-change bugs in the dataset, we provide one JUnit test class containing a single test case that exposes the behavioral difference between the original and resulting programs. Importantly, the exact same test is compiled and executed unchanged against both program versions. A bug is considered a behavioral-change instance only when this shared test succeeds in revealing different outcomes across the two versions. These tests do not rely on Java reflection; instead, they interact with the programs only through ordinary Java constructs available to client code. This choice keeps the notion of observable behavior aligned with client-level usage scenarios. The tests also provide a concrete view of the kinds of behavioral properties considered in our study. Many tests assert return values, such as checking that a method returns 1, 0, 10, or another expected value. Others capture and compare standard output, which is necessary for programs whose externally visible behavior is expressed through printing. Several tests assert that no exception should be thrown, thereby capturing cases where a faulty refactoring introduces failures such as ClassCastException, or other unintended runtime exceptions. In a few cases, the tests check externally visible state after invoking a public method, such as verifying the value of a field that should have been updated according to the original behavior. Finally, the tests range from 8 to 23 LOC, with an average size of 15 LOC. This compact size suggests that,
12
Rohit Gheyi et al.
Table 2: Accuracy and stability metrics used in our study.
Accuracy
Metric Mean Accuracy
pass@k
Accuracy Spread Stability
tar@k
cons@k
Definition Computes the rate of correct answers across k responses for a question. The overall mean accuracy is the average over all questions. Binary metric that returns 1 if there is at least one correct answer across k responses for a question, and 0 otherwise. The overall pass@k is the average over all questions. Higher values indicate that the model is more likely to produce a correct response eventually. Computes the difference between the maximum and minimum mean accuracy across k attempts over all questions. Lower values indicate that the model is more stable in accuracy across attempts. Binary metric that returns 1 if all k answers for a question are identical, regardless of correctness, and 0 otherwise. The overall tar@k is the average over all questions. Higher values indicate that the model produces more self-consistent responses across attempts. Binary metric that returns 1 if the unique most frequent answer, or consensus, across k responses for a question is correct, and 0 otherwise. The overall cons@k is the average over all questions. Higher values indicate that the model is more likely to produce a correct response consistently.
although the behavioral changes caused by faulty refactorings may be subtle, they can still be exposed by small and direct client-level tests in our dataset.
3.4 Metrics To assess the accuracy and consistency of foundation models in detecting compilation errors and behavioral changes, we adopt the accuracy and stability metrics presented in Table 2. These metrics [34,35] allow us to capture not only whether a model produces correct results, but also whether it does so consistently across multiple attempts, an important property for tools intended to support developers. Accuracy is measured through mean accuracy, which reflects the overall proportion of correct answers across responses, and pass@k, which evaluates whether at least one correct answer emerges within k attempts. While mean accuracy indicates the model’s average effectiveness, pass@k highlights its potential to eventually provide a correct response under repeated queries, a scenario that may be useful in automated pipelines. Because our benchmark contains only positive instances, there are no true negatives; therefore, in our setting, the reported accuracy numerically coincides with recall.
Foundation Models as Oracles for Refactoring Correctness Detection
13
Stability metrics complement these measures by examining how dependable model outputs are across different executions. Accuracy spread quantifies the gap between the best and worst results over multiple attempts, thereby reflecting the predictability of performance. tar@k (Total Agreement Rate) measures the extent to which all responses for a question are identical, regardless of correctness, providing insights into the model’s self-consistency. Finally, cons@k evaluates whether the most frequent response across attempts aligns with the correct answer, thus combining correctness with consistency. Together, these metrics provide a multi-dimensional evaluation framework. Accuracy indicates whether models can identify the target transformations, while stability measures whether their predictions are consistent across runs. Considering both dimensions is important, since even accurate models may be difficult to use in developer-facing workflows if their behavior is inconsistent.
3.5 Models The models evaluated include GPT-OSS-20B and GPT-5.4, both developed by OpenAI. GPT-OSS-20B was chosen as a representative open-weight model that can be executed locally, offering potentially improved reproducibility, deployment flexibility, and lower-cost experimentation. In contrast, GPT-5.4 was selected as a strong proprietary-model baseline, ranked highly in LLM Arena [36]. This contrast allows us to investigate an important practical tradeoff between accessibility and performance, namely how far locally deployable open models can go for refactoring-bug detection and how they compare with stronger closed models in terms of effectiveness and stability. We accessed GPT-5.4 (gpt-5.4-2026-03-05) via its API, using the default settings, including a reasoning effort of none, while GPT-OSS-20B was executed locally using the Ollama framework in Python on a Mac Mini M4 Pro with 64GB of RAM. For GPT-OSS-20B, we specified the model name and base URL, and used the default configurations of the LangChain Ollama API [37], with the exception of the temperature parameter, which was set to 0.5 (see Section 5.2). For GPT-5.4, we used the Python API with default settings, requiring only the API key [38]. We ran GPT-OSS-20B and GPT-5.4 five times each to assess prediction stability under repeated executions, since foundation models may vary even for the same input. We limited the evaluation to five runs per model to keep computational costs manageable. All analyses were conducted in April 2026.
3.6 Correctness Assessment We assess each model response by comparing it against the ground truth established in Section 3.2. Since all 226 instances in our dataset correspond to faulty refactorings, each instance is expected to be classified either as a behavioral change or as a compilation error. Therefore, a response that claims
14
Rohit Gheyi et al.
behavior preservation, i.e., YES, is always counted as incorrect in this benchmark. To make this process systematic and reproducible, we implemented a dedicated analysis tool that automatically evaluates each model answer according to the expected bug type and the corresponding validation criteria. For compilation-error instances, a response is considered correct only when the model classifies the transformation as NO - COMPILATION ERROR. This classification is checked against the dataset evidence showing that the original program compiles successfully whereas the resulting program does not. For behavioral-change instances, a response is considered correct only when the model classifies the transformation as NO - BEHAVIOR CHANGE and the JUnit test provided in the model response constitutes valid executable evidence of the behavioral difference. Specifically, our tool compiles the model-provided test against both the original and resulting programs using OpenJDK-Temurin21.0.7+6 and records the corresponding compiler logs. It then executes the same test on both versions and records the execution results. The response is counted as correct only if the same model-provided test compiles successfully in both versions and exposes different outcomes across them, that is, it succeeds on one version and fails on the other.
3.7 Prompt We adopt a zero-shot prompting strategy [39,40,41], in which the model is asked to solve the task without task-specific fine-tuning or example-based demonstrations. In addition, we employed MetaPrompting [42] to help refine the prompt design. More specifically, we iteratively asked GPT-5.4 to improve an initial prompt designed to assess whether a refactoring introduces compilation errors or behavioral changes. Through this process, we obtained a revised prompt with a more structured output format and clearer instructions regarding compilation correctness and behavioral preservation. The final prompt takes as input the original program before refactoring (code1) and the resulting program after refactoring (code2), and asks the model to check two conditions: (i) whether the resulting program compiles successfully, and (ii) whether the transformation preserves behavior according to the notion of behavior preservation defined in Section 3.3. To support automatic processing, the model must return valid JSON with three fields: verdict, explanation, and junit_test. The verdict must be exactly one of YES, NO COMPILATION ERROR, or NO - BEHAVIOR CHANGE; the explanation must provide a brief evidence-based justification; and junit_test must be null unless the verdict is NO - BEHAVIOR CHANGE. In that case, the model must generate one deterministic JUnit test, as a complete Java source file, that compiles and runs unchanged against both versions and exposes the behavioral difference. This JSON-based design makes the output easier to process automatically and reduces ambiguities associated with free-form answers. Next, we present the prompt used in our experiments.
Foundation Models as Oracles for Refactoring Correctness Detection
15
Consider the following initial program: <CODE1 {code1} CODE1> After applying a refactoring, it yields the following resulting program: <CODE2 {code2} CODE2> Check the following: 1. Compilation correctness: The resulting program compiles successfully, with no syntax, type, name-resolution, import, or linkage errors. 2. Behavioral equivalence: Ignore methods not present in both programs; compare only methods with the same signature that are public in both versions. For those common public methods, the initial and resulting programs preserve the same observable behavior, including return values, printed output, thrown exceptions, and externally visible state changes. Return ONLY valid JSON. Do not use markdown. Do not add any text before or after the JSON. Use exactly this schema: { "verdict": "YES | NO - COMPILATION ERROR | NO - BEHAVIOR CHANGE", "explanation": "1-4 sentences", "junit_test": null } Rules: - "verdict" must be exactly one of: "YES" "NO - COMPILATION ERROR" "NO - BEHAVIOR CHANGE" - "explanation" must briefly justify the answer with concrete evidence. - "junit_test" must be null unless the verdict is "NO - BEHAVIOR CHANGE". - If "junit_test" is not null, it must contain exactly one deterministic JUnit test as a complete Java source file. - The content of "junit_test" must be directly copyable into a Java IDE such as VS Code. - The exact same generated test code must compile and run against both code1 and code2 without any modification. - The test must expose a behavioral difference by producing one outcome on code1 and a different outcome on code2. - The test must include all necessary imports, exactly one public test class, and exactly one test method. - The test must be minimal, concrete, deterministic, and focused on the changed public behavior. - The test must not rely on randomness, time, concurrency, network, filesystem, or environment-specific behavior. - Do not invent compilation errors.
16
Rohit Gheyi et al.
- Do not claim behavior change unless you can point to a specific differing return value, output, exception, or externally visible state. - Do not include markdown fences such as java inside "junit_test". - Write "junit_test" as plain Java code text. Return JSON only.
In this prompt, code1 and code2 denote the program versions before and after the refactoring, respectively. For example, these placeholders correspond to the programs shown in Figures 1(a) and 1(b), respectively. 3.8 Metamorphic Testing A key threat to validity when evaluating foundation models is data contamination [43], that is, the possibility that a model has been exposed during training to identical or highly similar examples, which may artificially inflate performance. To reduce this threat, we applied metamorphic testing (MT) [44, 45] to all 226 transformations in our benchmark. The main idea is to generate behaviorally equivalent variants of the original programs while altering their lexical and structural form. This allows us to assess whether the models produce similar predictions when the same underlying refactoring bug is presented through a syntactically different original program. For each bug instance, we applied a metamorphic transformation to the original program before evaluating the corresponding refactoring scenario. The resulting program was kept unchanged. This setup tests whether the model can still identify the same refactoring bug when the original program is presented through a behavior-preserving variant with a different lexical or structural form. The metamorphic transformation operators were intended to preserve behavior under the equivalence notion adopted in this study, as defined in Section 3.3. The operators used in this analysis do not change existing control flow, methodcall resolution, return expressions, printed output, exception behavior, or public operations exercised by the tests. Instead, they introduce behaviorally inert modifications, such as unused declarations, comments, imports, and auxiliary classes. Therefore, these transformations are expected not to change whether an instance is classified as a behavioral change or as a compilation error. We implemented six metamorphic transformation operators on top of Spoon [46]: AddFieldOperator (AF), CommentsOperator (CO), InnerClassOperator (IC), JavaImportOperator (JI), LocalVariableDeclarationOperator (LVD), and TopLevelClassOperator (TLC). These operators introduce intended behavior-preserving changes such as adding unused fields, injecting comments, creating auxiliary inner or top-level classes, inserting unused imports, and declaring dead local variables. Table 3 shows representative examples of these transformations applied to the original program. Although Table 3 shows simplified examples, the actual transformations are randomized. Both the choice of operator and the internal choices within each operator vary across instances. Depending on the operator, this includes random
Foundation Models as Oracles for Refactoring Correctness Detection
17
Table 3: Examples of metamorphic transformation operators (AddFieldOperator (AF), CommentsOperator (CO), InnerClassOperator (IC), JavaImportOperator (JI), LocalVariableDeclarationOperator (LVD), and TopLevelClassOperator (TLC)) applied to the original program to generate behavior-preserving variants. Operator Before AF
After
class A { int m() { return 1; } }
class A { int field = 42; int m() { return 1; } }
class A { int m() { return 1; } }
class A { // comment int m() { return 1; } }
class A { int m() { return 1; } }
import java.util.List;
class A { int m() { return 1; } }
class A { int m() { int local = 7; return 1; } }
class A { int m() { return 1; } }
class A { class Aux { int value = 0; } int m() { return 1; } }
class A { int m() { return 1; } }
class A { int m() { return 1; } }
CO
JI
class A { int m() { return 1; } }
LVD
IC
TLC
class Aux { int value = 0; }
variation in identifiers, literals, comments, imports, declared types, field names, class names, and generated class structures. For example, AF randomly chooses the type and initial value of the injected field; CO randomly selects among several comments; JI randomly inserts one among several imports; and LVD randomly varies both the declared type and assigned literal. Similarly, IC and TLC generate auxiliary classes with randomized names, field names, field types,
18
Rohit Gheyi et al.
Table 4: Metamorphic operators applied to the 226 instances. Operator AddFieldOperator CommentsOperator InnerClassOperator JavaImportOperator LocalVariableDeclarationOperator TopLevelClassOperator Total
Abbrev. AF CO IC JI LVD TLC
Instances 32 40 37 37 45 35 226
and default values. As a result, even when two instances are transformed by the same operator, the transformed original programs may differ at both lexical and structural levels. To further strengthen the validity of the metamorphic analysis, we checked the generated variants through compilation and test execution. For the models evaluated in this analysis, we stored the compiler logs for the transformed original programs and the corresponding resulting programs, the compiler logs for the JUnit tests provided in the model responses, and the JUnit execution results obtained by running those tests on both program versions. Table 4 summarizes the number of transformed instances per operator. The distribution is reasonably balanced, with all six operators being applied dozens of times. An additional minor contribution of this study is the implementation infrastructure itself. We developed the metamorphic transformation operators as a reusable Java-based framework, including implementations for AF, CO, IC, JI, LVD, and TLC. This framework can be used or extended by future studies interested in generating intended behavior-preserving program variants for robustness assessment in refactoring analysis and foundation-model evaluation.
4 Results Next we answer our research questions.
4.1 RQ1 . To what extent can an open foundation model (GPT-OSS-20B) detect refactoring bugs, including compilation errors and behavioral changes? Across individual runs, GPT-OSS-20B exhibits relatively stable behavior in this setting, although non-negligible variation remains even under a fixed configuration. Overall accuracy ranges from 0.757 to 0.805. The mean overall accuracy across the five runs is 0.784. Figure 4 summarizes five complementary metrics over the first k attempts: cumulative overall success rate (Acc@), agreement in binary correctness across attempts (tar@), strict-majority correctness (cons@), cumulative success on behavioral-change cases (BC@), and cumulative success on compilation-error cases (CE@). The model is consistently stronger on behavioral-change instances than on compilation-error instances in our
Foundation Models as Oracles for Refactoring Correctness Detection
19
experiments. BC accuracy ranges from 0.854 to 0.927, whereas CE accuracy ranges from 0.724 to 0.778. GPT-OSS-20B performs better on BC than on CE in every attempt. The strongest overall result (Attempt 1) combines the highest BC accuracy (0.927) with one of the two highest CE accuracies (0.778). Considering cumulative success across attempts reveals that repeated sampling increases cumulative coverage. With only the first attempt, GPT-OSS20B correctly handles 182 out of 226 instances. Acc@ increases to 0.858 after two attempts, 0.912 after three attempts, 0.925 after four attempts, and 0.929 after all five attempts. The cumulative curves also highlight a clear contrast between BC and CE. For BC, cumulative success increases from 0.927 with one attempt to 0.951 with two attempts and reaches 1.000 with three attempts. For CE, cumulative success increases more gradually, from 0.778 with one attempt to 0.838, 0.892, 0.908, and 0.914 after two, three, four, and five attempts, respectively. This indicates that repeated sampling is effective in this setting for recovering BC cases that fail in individual runs, but it does not eliminate the hardest CE failures. The agreement and majority-based metrics provide a complementary view of stability. The tar@ curve decreases from 1.000 for a single attempt to 0.845, 0.739, 0.677, and 0.642 when considering the first two, three, four, and five attempts, respectively. This decline shows that a notable fraction of instances changes correctness status across runs, even though many of these fluctuations are beneficial for cumulative coverage. The cons@ curve is not monotonic because it depends on the strict-majority outcome among the first k attempts. It starts at 0.805, decreases to 0.704 for two attempts due to unresolved ties, and then reaches 0.796, 0.761, and 0.819 for three, four, and five attempts. The BC subset is largely attempt-sensitive. No BC instance is misclassified in all five runs. However, a few cases recur across attempts. IDs 72 (Change Method Signature) and 281 (Rename Field ) are the most difficult BC cases, each failing in three out of five attempts. ID 72 fails either because the generated test does not compile or because it does not expose the behavioral change, suggesting instability in test generation. ID 281 repeatedly fails because the model states that behavior was preserved, indicating a recurring incorrect semantic verdict. BC errors are not concentrated in a single structural category, but instead arise from transformations that require sensitivity to subtle changes in data flow, conditions, name binding, or interface contracts. The CE subset, in contrast, contains a more persistent core of difficult instances. Sixteen CE IDs are misclassified in all five attempts. These cases represent recurring failure modes rather than occasional fluctuations. In these persistent CE failures, the model repeatedly assigns a semantic verdict, stating either that behavior changed or that behavior was preserved, when the correct label is compilation error. The persistent CE failures are concentrated in transformations that alter declarations, member placement, signatures, or class hierarchies. The CE cases misclassified in all five attempts include Push Down Method (IDs 15 and 345), Move Method (IDs 94 and 131), Inline Variable (IDs 106 and 326), Rename Method (IDs 111 and 397), Introduce Constant (IDs 122 and 123), Pull Up Method (IDs 154, 192, and 398), Move Class
20
Rohit Gheyi et al.
(ID 203), Pull Up Field (ID 225), and Inline Method (ID 349). This indicates that the observed difficulty appears to be driven less by raw frequency and more by the semantic and structural challenges posed by transformations that affect name resolution, inheritance, member visibility, and compilation validity.
1 0.95
Success rate
0.9 Acc@ tar@ cons@ BC@ CE@
0.85 0.8 0.75 0.7 0.65 0.6
1
2
3 k
4
5
Fig. 4: GPT-OSS-20B results across five repeated attempts. Acc@ = cumulative overall success rate considering whether an instance is solved in at least one of the first k attempts, tar@ = agreement in binary correctness across the first k attempts, cons@ = correctness of the strict-majority outcome across the first k attempts, BC@ = cumulative success rate on behavioral-change cases, and CE@ = cumulative success rate on compilation-error cases. CE = Compilation Error, and BC = Behavioral Change.
RQ1 Answer GPT-OSS-20B shows moderate stability in this setting, with stronger and more recoverable performance on behavioral-change cases than on compilation-error cases. Repeated sampling improves cumulative coverage to 92.9%, indicating that multiple attempts can recover some missed bugs. However, strict-majority aggregation is less effective because several instances alternate between correct and incorrect outcomes. Overall, the main limitation of GPT-OSS-20B lies in persistent CE cases involving name resolution, inheritance, member visibility, and compilation validity.
Foundation Models as Oracles for Refactoring Correctness Detection
21
4.2 RQ2 . To what extent can a proprietary foundation model (GPT-5.4) detect refactoring bugs, including compilation errors and behavioral changes? GPT-5.4 achieves high single-attempt performance in this benchmark and remains stable across repeated executions. In individual runs, overall accuracy ranges from 0.938 to 0.947, with a spread of only 0.009. This indicates that, in our setting, GPT-5.4 already provides high performance without requiring multiple attempts. Figure 5 reports five complementary metrics for k: cumulative overall success rate (Acc@), agreement in correctness and error status across attempts (tar@), strict-majority correctness (cons@), cumulative success on behavioral-change cases (BC@), and cumulative success on compilation-error cases (CE@). The cumulative results show that additional attempts provide modest but consistent gains. Acc@ increases from 0.938 at k=1 to 0.956 at k=2, 0.965 at k=3, 0.969 at k=4, and 0.973 at k=5. Across all five runs, GPT-5.4 correctly handles 220 out of 226 instances at least once. These results show that repeated sampling still improves coverage, but most of the model’s effectiveness in this benchmark is already achieved in the first attempt. By type, GPT-5.4 performs slightly better on compilation-error instances than on behavioral-change instances. Across individual runs, BC accuracy ranges from 0.878 to 0.927, whereas CE accuracy ranges from 0.941 to 0.957. The cumulative curves follow the same pattern. BC@ increases from 0.927 at k=1 to 0.951 at k=2, reaches 0.976 at k=3, and remains at 0.976 through k=5. CE@ increases from 0.941 at k=1 to 0.957, 0.962, 0.968, and 0.973 for k=2 through k=5. The agreement and majority-based metrics further support the model’s stability in this setting. The tar@ curve decreases from 1.000 for a single attempt to 0.973, 0.947, 0.929, and 0.925 when considering the first two, three, four, and five attempts, respectively. This decline is expected because exact agreement becomes harder as more attempts are considered, but the values remain high. The cons@ curve also remains high: it starts at 0.938, decreases slightly to 0.929 for two attempts due to unresolved ties, and reaches 0.947 for k=3, k=4, and k=5. This indicates that the majority answer is usually correct even when individual attempts differ. For BC, there are 20 total errors across the five attempts. The most common failure mode is an incorrect semantic verdict that behavior is preserved, accounting for 10 cases. In 4 cases, the model incorrectly indicates a compilation error. Test-generation failures account for the remaining cases: 5 errors occur because the generated test does not compile, and 1 occurs because the generated test does not expose the behavioral change. For CE, there are 46 total errors across the five attempts. The dominant failure mode is assigning a semantic verdict to an uncompilable transformation: in 30 cases, the model indicates a behavior change when the correct label is compilation error, while in 16 cases it states that behavior is preserved. This suggests that the hardest CE failures are not caused primarily by output instability, but by treating syntactically invalid transformations as if they were semantically analyzable.
22
Rohit Gheyi et al.
In the BC subset, only ID 112 (Rename Method ) is misclassified in all five attempts, and it repeatedly fails because the model states that behavior was preserved. ID 281 (Rename Field ) fails in four attempts, alternating between indicating a compilation error and stating that behavior was preserved. ID 361 (Rename Field ) fails in three attempts because the generated test does not compile. In the CE subset, five IDs are misclassified in all five attempts: 104 (Inline Variable), 108 (Inline Variable), 215 (Move Method ), 282 (Inline Variable), and 360 (Push Down Field ). Additional CE IDs recur in four out of five attempts. The recurring CE errors again reflect semantic misclassification: the model either indicates a behavior change or states that behavior was preserved when the correct answer is compilation error. Figure 6 summarizes this analysis by comparing GPT-5.4 and GPT-OSS-20B across the 10 most frequent refactoring types in the dataset.
1
Success rate
0.98 0.95
Acc@ tar@ cons@ BC@ CE@
0.93 0.9 0.88 0.85
1
2
3 k
4
5
Fig. 5: GPT-5.4 results across five repeated attempts. Acc@ = cumulative overall success rate considering whether an instance is solved in at least one of the first k attempts, tar@ = agreement in correctness and error status across the first k attempts, cons@ = correctness of the strict-majority outcome across the first k attempts, BC@ = cumulative success rate on behavioral-change cases, and CE@ = cumulative success rate on compilation-error cases. CE = Compilation Error, and BC = Behavioral Change.
Foundation Models as Oracles for Refactoring Correctness Detection GPT-OSS-20B
23 GPT-5.4
CE
BC
CE
BC
Move Method (n=32)
0.79
0.67
0.93
1.00
Inline Method (n=20)
0.73
1.00
1.00
1.00
Pull Up Method (n=18)
0.69
1.00
0.88
1.00
Extr. Local Variable (n=15)
1.00
1.00
1.00
1.00
Rename Method (n=14)
0.50
1.00
0.90
0.75
Change Meth. Sig. (n=13)
0.91
0.50
1.00
1.00
Extract Method (n=12)
0.91
1.00
1.00
1.00
Move Class (n=11)
0.82
–
1.00
–
Inline Variable (n=10)
0.80
–
0.70
–
Push Down Method (n=6)
0.00
1.00
1.00
1.00
Others (n=75)
0.80
0.93
0.95
0.86
Fig. 6: Heatmap of model accuracy across the 10 most frequent refactoring types in the dataset, separated into compilation-error (CE) and behavioralchange (BC) instances. All remaining refactoring categories are grouped into Others.
RQ2 Answer GPT-5.4 is effective and stable at detecting reported refactoring bugs in this benchmark across repeated attempts. Repeated sampling improves cumulative coverage from 93.8% to 97.3%, but most gains are achieved in the first attempt, suggesting that repeated attempts provide limited additional benefit. The remaining failures are concentrated in a small number of persistent cases: BC failures mainly involve incorrect semantic verdicts or test-generation failures, whereas CE failures are dominated by transformations that affect bindings, member resolution, and structural relationships.
4.3 RQ3 . To what extent is the performance of these models affected by potential data leakage? To investigate whether the reported performance could be influenced by data leakage, we conducted a metamorphic testing analysis. We applied intended semantics-preserving source-code transformations to the input programs and re-evaluated both models on the transformed instances. In this analysis, the original setting corresponds to the first execution of each model, whereas the
24
Rohit Gheyi et al.
Table 5: Performance before and after metamorphic testing. CE = Compilation Error; BC = Behavioral Change. Model GPT-OSS-20B GPT-OSS-20B GPT-5.4 GPT-5.4
Setting Original Metamorphic testing Original Metamorphic testing
Overall Acc. 0.805 0.801 0.938 0.942
BC Acc. 0.927 0.951 0.927 0.902
CE Acc. 0.778 0.768 0.941 0.951
metamorphic-testing setting corresponds to the execution on the transformed programs. The rationale is that, if predictions were driven mainly by memorization of superficial training artifacts, then small structural perturbations that preserve the intended semantics would be expected to lead to larger and less stable degradation. In contrast, if the models are sensitive to program-relevant information beyond surface form, performance should remain relatively stable under these transformations. Table 5 summarizes the results before and after metamorphic testing. Overall, the metamorphic transformations had only limited impact on both models. For GPT-OSS-20B, overall accuracy changes only slightly, from 0.805 in the original setting to 0.801 under metamorphic testing. For GPT-5.4, the overall effect is similarly small: accuracy increases from 0.938 to 0.942. BC accuracy decreases from 0.927 to 0.902, whereas CE accuracy increases from 0.941 to 0.951. These results indicate that the predictions of both models are stable under the applied metamorphic transformations. A closer inspection of the error IDs shows that metamorphic testing does not completely change which instances are difficult. Instead, it preserves a meaningful portion of the hard cases observed in the original execution, especially in the CE subset. For GPT-OSS-20B, the BC overlap between the original and metamorphic settings is empty. In the CE subset, 22 IDs are misclassified in both settings. For GPT-5.4, the overlap is even more concentrated. In the BC subset, IDs 112 and 281 are misclassified in both the original and metamorphic settings. In the CE subset, 8 of the metamorphic CE errors were also present in the original run: IDs 104, 108, 215, 278, 282, 360, 398, and 401. This overlap suggests that metamorphic testing preserves many of the hardest compilation-error cases, especially for GPT-5.4. We also analyzed whether certain metamorphic operators (see Table 3) were associated with more errors than others. Table 6 summarizes the distribution of metamorphic-testing errors by operator. For GPT-OSS-20B, the MT errors are spread across all six operators, suggesting that no single transformation type fully explains the failures. In absolute terms, InnerClassOperator accounts for the largest number of CE errors, followed by JavaImportOperator, CommentsOperator, and LocalVariableDeclarationOperator. In the BC subset, GPT-OSS-20B produces only two MT errors, associated with JavaImportOperator and AddFieldOperator. For GPT-5.4, the error distribution is more concentrated: LocalVariableDeclarationOperator is the main source of MT errors, followed by AddFieldOperator and TopLevelClassOperator. Notably,
Foundation Models as Oracles for Refactoring Correctness Detection
25
Table 6: Metamorphic-testing errors by operator (AddFieldOperator (AF), CommentsOperator (CO), InnerClassOperator (IC), JavaImportOperator (JI), LocalVariableDeclarationOperator (LVD), and TopLevelClassOperator (TLC)). Counts are descriptive and not normalized by operator frequency. CE = Compilation Error; BC = Behavioral Change. Model GPT-OSS-20B GPT-OSS-20B GPT-5.4 GPT-5.4
Subset BC CE BC CE
AF 1 3 1 2
CO 0 8 0 0
IC 0 11 0 1
JI 1 9 1 1
LVD 0 8 2 3
TLC 0 4 0 2
CommentsOperator does not produce any MT error for GPT-5.4 in this run. This suggests that GPT-5.4 is less affected by comment-only changes in this run, whereas structural edits involving local declarations, added fields, or top-level rearrangements remain more challenging. Taken together, these findings suggest that the results are not easily explained by contamination alone. If the models mainly relied on memorized surface patterns, the semantics-preserving perturbations would likely cause broader degradation and a larger reshuffling of the error set. Instead, performance remains high, GPT-5.4 is only slightly affected, and many hard CE cases remain hard after transformation. This suggests that the models capture signals related to code structure and behavior, rather than relying only on exact memorization. Nevertheless, metamorphic testing does not rule out data leakage. A model may still benefit from prior exposure to related code, refactoring patterns, or abstract problem structures. Thus, our results do not prove the absence of contamination; they only indicate that the observed performance is unlikely to be explained solely by exact-match memorization or leakage of the original instances. RQ3 Answer The performance of both models is only marginally affected by semanticspreserving metamorphic transformations, which is not consistent with a simple data-leakage explanation for the observed results. GPT-5.4 remains almost unchanged under metamorphic testing, while GPTOSS-20B also shows stable overall performance, despite a small shift between BC and CE results. Moreover, many of the hardest cases from the original execution remain difficult after transformation, especially for compilation-error instances, indicating that model failures are not primarily tied to memorized surface patterns. Although these results do not rule out contamination entirely, they strengthen the interpretation that both models rely, to some extent, on signals related to code behavior and compilation validity.
26
Rohit Gheyi et al.
5 Discussion Next we discuss the results of our work, with emphasis on the scope and limitations of the observed evidence.
5.1 Accuracy When comparing GPT-OSS-20B (Figure 4) and GPT-5.4 (Figure 5), GPT5.4 shows higher accuracy and agreement overall. At k=1, GPT-5.4 reaches Acc@ 0.938, compared with 0.805 for GPT-OSS-20B; at k=5, the values increase to 0.973 and 0.929, respectively. The main difference is in compilationerror cases, where GPT-5.4 consistently obtains higher CE@ values. For behavioral-change cases, the gap is smaller: GPT-5.4 starts higher, but GPTOSS-20B reaches full BC coverage by k=3. Agreement metrics show the same trend. GPT-5.4 maintains higher tar@ and cons@ across repeated attempts, indicating more stable predictions, whereas GPT-OSS-20B benefits more from repeated sampling but varies more across runs. Overall, GPT-5.4 is more accurate and stable in this benchmark, especially for compilation-error detection, while GPT-OSS-20B remains competitive on behavioral-change cases when multiple attempts are considered. Both GPT-OSS-20B and GPT-5.4 correctly identify the behavioral change shown in Figure 1 (ID 14) in all five runs when applying the Push Down Method refactoring using Eclipse JDT. Next, we show an example output produced by GPT-5.4. Besides returning the correct label, the model explains why the behavior changes and generates a JUnit test that compiles against both versions and exposes the expected difference: it passes before the refactoring and fails after it. This additional evidence may help developers inspect why the transformation is behavior-changing. verdict: NO - BEHAVIOR CHANGE explanation: The refactoring moved method m() from class B to class C. In the original program, m() calls super.k(), which invokes A.k() and returns 10. In the refactored program, super refers to B, so m() calls B.k() and returns 20, changing the observable output. junit_test: import static org.junit.Assert.assertEquals; import org.junit.Test; public class RefactoringBehaviorTest { @Test public void testMBehavior() { assertEquals(10, new C().m()); } }
Foundation Models as Oracles for Refactoring Correctness Detection
27
The Inline Variable refactoring performed with NetBeans, shown in Figure 7 (ID 104), illustrates a compilation-error case. The refactored program does not compile because the conditional expression flag ? 1 : 2 has primitive type int; therefore, the method invocation .byteValue() is invalid, since primitive values do not have members. GPT-OSS-20B correctly reports this compilation error in all five runs and provides an explanation consistent with the relevant Java typing rule. verdict: NO - COMPILATION ERROR explanation: The expression (flag ? 1 : 2) has type int. In Java, primitives do not have methods, so calling .byteValue() on an int is a compile-time error. The original code relies on autoboxing to Integer, which is not applied in the refactored form. In contrast, GPT-5.4 fails to identify this refactoring bug in all five runs. It consistently classifies the transformation as behavior-preserving and incorrectly assumes that the conditional expression has type Integer due to boxing. Its explanation is reproduced below. verdict: YES explanation: The refactored code compiles: the conditional expression (flag ? 1 : 2) has type Integer due to boxing, so calling .byteValue() is valid. Behavior is unchanged because the local variable iii was not used for anything other than immediately invoking byteValue(), and both versions evaluate the same conditional and method call with the same result and side effects.
(a) Original program
(b) Refactored program
public class A { private void compIndex(boolean flag) { Integer iii = flag ? 1 : 2; iii.byteValue(); } }
public class A { private void compIndex(boolean flag) { (flag ? 1 : 2).byteValue(); } }
Fig. 7: Applying Inline Variable to iii using NetBeans introduces a compilation error.
5.2 Temperature Temperature is a key hyperparameter in foundation models that controls the randomness of generated outputs [47]. Lower values make the model
28
Rohit Gheyi et al.
more deterministic, favoring high-probability outputs and reducing variation, whereas higher values increase randomness and may produce more diverse but potentially less stable responses. Evaluating different temperature settings is therefore important to understand how the model balances sensitivity and variability in a task where consistency is important. To assess the sensitivity of GPT-OSS-20B to decoding variability, we evaluated temperatures ranging from 0.0 to 1.0 in increments of 0.1. Figure 8 summarizes the accuracy results for the full benchmark as well as for the two subsets considered in our study: behavioral-change (BC) and compilation-error (CE) instances. Overall accuracy remains relatively stable across temperatures, ranging from 0.757 at t=0.2 to 0.805 at t=0.5. This limited variation suggests that GPT-OSS-20B is relatively insensitive to temperature changes on the full benchmark. The best overall result is obtained at t=0.5, where the model correctly identifies 182 out of 226 instances. A more fine-grained analysis, however, reveals different sensitivity patterns for BC and CE cases. BC accuracy varies from 0.780 at t=0.8 to 0.927 at t=0.1 and t=0.5, corresponding to a spread of 14.6%. CE accuracy is more stable, ranging from 0.724 at t=0.2 to 0.784 at t=0.8. Thus, although the overall results appear relatively stable, BC cases are more sensitive to decoding configuration than CE cases. The best BC performance is achieved at t=0.1 and t=0.5, where the model correctly identifies 38 out of 41 instances. By contrast, the best CE performance is obtained at t=0.8, where GPT-OSS-20B correctly identifies 145 out of 185 instances. However, t=0.8 also yields the weakest BC performance, with only 32 out of 41 BC cases correctly identified. This suggests a trade-off between the two subsets: the temperature that maximizes CE accuracy is not the best choice for BC detection. In contrast, t=0.5 provides the highest overall accuracy while maintaining the best BC accuracy and near-best CE accuracy. The error patterns help explain this difference. For BC instances, the dominant failure mode is not only an incorrect semantic classification, but also the inability to construct an effective supporting test. Across temperatures, BC errors frequently arise because the generated test does not compile or because it compiles but does not expose the behavioral difference. Across all temperatures, BC errors include cases in which the model states that behavior was preserved, cases in which the generated test does not compile, and cases in which the generated test does not expose a behavioral change. This suggests that, for BC cases, the bottleneck involves both behavioral assessment and the construction of a valid and discriminative test oracle. For CE instances, the dominant failure mode is an incorrect verdict, with the model repeatedly stating that the program changed behavior or preserved behavior when the correct label is compilation error. Thus, CE mistakes stem less from inadequate test generation and more from repeatedly treating uncompilable transformations as if they were semantically analyzable. This pattern is consistent across temperatures and helps explain the limited variation in CE accuracy: changing the decoding randomness does not eliminate the same underlying classification errors. The distribution of error IDs suggests that only a limited subset of instances is persistently difficult for GPT-OSS-20B across decoding configurations. In
Foundation Models as Oracles for Refactoring Correctness Detection
29
the BC subset, no instance is misclassified at all temperatures, indicating that every behavioral-change case can be correctly identified under at least one temperature setting. Still, some BC instances are more challenging than others, most notably ID 281, which is misclassified under most temperature settings. In contrast, the CE subset exhibits a small but important core of persistent failures. Some CE instances are misclassified at every temperature, such as IDs 94 and 298. Several additional CE instances recur in 10 out of 11 temperatures, such as IDs 15 and 105. This indicates that some uncompilable transformations repeatedly lead the model to an incorrect classification path, regardless of decoding randomness. At the same time, considering the union of all temperatures, GPT-OSS20B correctly handles 219 out of 226 instances at least once. This includes all 41 BC cases and 178 out of 185 CE cases. Therefore, most failures are temperature-sensitive rather than universally hard. The remaining unsolved cases are concentrated entirely in the CE subset, reinforcing that the most persistent limitation is the model’s difficulty in recognizing certain compilation errors. Taken together, the results reveal a trade-off across subsets. Lower temperatures around t=0.1 are favorable for BC detection, whereas t=0.8 provides the highest CE accuracy but harms BC performance. When selecting a single decoding configuration for GPT-OSS-20B, t=0.5 offers the best observed balance between accuracy and stability: it achieves the highest overall accuracy, matches the best BC accuracy, and remains close to the best CE accuracy. 94 92 90 Accuracy (%)
88 86 Overall BC CE
84 82 80 78 76 74 72
0
0.1
0.2
0.3
0.4 0.5 0.6 Temperature
0.7
0.8
0.9
1
Fig. 8: Accuracy across decoding temperatures for GPT-OSS-20B on the full benchmark, behavioral-change cases (BC), and compilation-error cases (CE). We also performed an additional GPT-5.4 run with T = 0, keeping all other parameters unchanged. In this deterministic decoding setting, GPT-5.4 obtained worse results than under the default-temperature configuration: 88.5%
30
Rohit Gheyi et al.
overall accuracy, with 58.5% accuracy on behavioral-change cases and 95.1% on compilation-error cases. Thus, compilation-error performance remained similar, whereas behavioral-change detection dropped substantially. In the behavioralchange subset, GPT-5.4 incorrectly classified 15 cases as behavior-preserving and 2 cases as introducing compilation errors.
5.3 Evaluation on Correct Refactorings (True Positives) Beyond detecting behavioral changes, it is important to assess whether models correctly recognize behavior-preserving transformations. For this purpose, we used a subset of the dataset from prior work [48] covering 10 refactoring types. Each transformation applies a single refactoring to a small program generated by JDolly [5], an automatic program generator. We evaluated 50 transformations spanning Add Parameter, Encapsulate Field, Move Method, Pull Up Field, Pull Up Method, Push Down Field, Push Down Method, Rename Field, Rename Method, and Rename Class. All transformations were independently validated as behavior-preserving by SafeRefactor [8, 5,49] and by successfully compiling both the original and refactored programs. On the 50 true-positive refactoring instances, GPT-OSS-20B correctly classified 43 cases, yielding an accuracy of 86%. The remaining 7 errors were concentrated in Add Parameter, Move Method, Pull Up Field, and Push Down Field case, and mostly reflected situations in which the model inferred a behavioral difference from changes in dynamic dispatch or overload resolution, or incorrectly concluded that the refactored program did not compile. In contrast to its earlier behavior, GPT-OSS-20B handled the other refactoring types consistently well. Using the setup described in Section 3, GPT-5.4 with reasoning effort set to medium correctly classified 48 out of 50 transformations, yielding an accuracy of 96%. The two errors occurred for Push Down Field and Rename Field. In both cases, the model incorrectly claimed that the transformation changed behavior and generated a JUnit test that relied on reflection to access a field, thereby exposing a difference that falls outside the intended notion of equivalence for this benchmark. In terms of efficiency, the 50 analyses consumed a total of 77,370 tokens, including 30,239 reasoning tokens, at a total cost of USD 0.72. On average, each instance used 1,547.4 total tokens and 604.8 reasoning tokens. The total runtime was approximately 1,258.7 s, corresponding to a mean latency of 25.2 s per instance. These results indicate that GPT-5.4 and GPT-OSS-20B can recognize behavior-preserving refactorings with high accuracy in this additional analysis, but still benefit from a clearly specified equivalence criterion. In particular, the false positives suggest that the model may overapproximate externally visible behavior by considering reflective access to members that are not part of the intended public interface. This reinforces the importance of precise prompting when evaluating behavioral equivalence under refactoring.
Foundation Models as Oracles for Refactoring Correctness Detection
31
5.4 Other Open Models We also evaluated other open models using the same dataset, prompt, and methodology adopted for GPT-OSS-20B in Section 3. More specifically, we ran Llama-3.2-3B, Phi-4-14B, Gemma-4-31B, and Qwen-3.6-35B via Ollama, using each model’s default parameters with a fixed temperature of 0.5. Figure 9 summarizes the results in three settings: overall accuracy, accuracy on behavioral changes (BC), and accuracy on compilation errors (CE). The results show variation among the evaluated open models. Llama-3.23B did not solve any benchmark instance, obtaining 0.0% overall accuracy. Its failures included incorrect semantic verdicts, invalid top-level labels, malformed outputs marked as PARSE_ERROR, and non-compiling generated tests. Thus, under our prompt and evaluation protocol, Llama-3.2-3B did not reliably follow the required output format or distinguish behavioral changes from compilation errors. Phi-4-14B also achieved limited performance, with 27.9% overall accuracy. It was particularly weak on behavior-change instances, correctly classifying only 14.6% of them, and performed slightly better on compilation-error instances, with 30.8% correctly classified. Some outputs were also marked as PARSE_ERROR, indicating malformed JSON or outputs that could not be reliably parsed. Qwen-3.6-35B performed better, correctly classifying 75.7% of the cases. Its performance was stronger on behavior-change instances (90.2%) than on compilation-error instances (72.4%). GPT-OSS-20B achieved a similar overall result (80.5%), but with a more balanced distribution across subsets. Gemma-4-31B achieved the strongest results among the evaluated open models, with 96.5% overall accuracy. It performed consistently well on both subsets, correctly classifying 97.6% of BC instances and 96.2% of CE instances. These results indicate that Gemma-4-31B was effective at distinguishing behavior-change cases from compilation-error cases while also following the required output format. Under the single-run configuration used for this comparison, Gemma-4-31B also outperformed the first-run results of GPT-5.4 presented in Section 4.
5.5 Other Proprietary Models We evaluated two additional proprietary large language models using the same dataset, prompt, and methodology described in Section 3: Gemini-3.1-ProPreview and Claude-4.6-Sonnet. Gemini-3.1-Pro-Preview was executed via API with default parameters, whereas Claude-4.6-Sonnet was evaluated manually through the web interface, also with default parameters. All analyses were conducted in April 2026. Table 7 summarizes the first-run results for all evaluated models, including GPT-OSS-20B and GPT-5.4 for comparison. Overall, Gemini-3.1-ProPreview achieved the highest observed performance, with 225 correct classifications out of 226 cases, corresponding to an overall accuracy of 99.6%. It was followed by Claude-4.6-Sonnet with 94.7%, GPT-5.4 with 93.8%, and
32
Rohit Gheyi et al. 100
Accuracy (%)
80
60
Overall BC CE
40
20
0
m
3 a-
a Ll
3B
.2-
4B
0B
-1
P
4 hi-
1B
-2
S OS
T-
GP
3 4-
Ge
m
ma
B 35
3.6 n-
e Qw
Fig. 9: Overall, behavioral-change (BC), and compilation-error (CE) accuracy of the evaluated open models at temperature 0.5.
GPT-OSS-20B with 80.5%. These results indicate a clear performance gap between the strongest proprietary model in this evaluation and the remaining models, particularly the smaller open-weight model considered in our study. The per-category results provide a more detailed view of these differences. For behavioral-change (BC) cases, Gemini-3.1-Pro-Preview achieved the highest accuracy, correctly handling 40 out of 41 instances. The other three models obtained the same BC accuracy. However, their failures differed. Gemini3.1-Pro-Preview missed only ID 50, where the generated test did not compile. GPT-5.4 also failed on ID 50, but by predicting a compilation error instead of a behavioral change, and additionally failed on IDs 112 and 281. Claude-4.6Sonnet failed on IDs 20, 75, and 302 by stating that behavior was preserved. GPT-OSS-20B failed on IDs 72, 97, and 263 because the generated tests did not expose a behavioral change. For compilation-error (CE) instances, the differences between models were more pronounced. Gemini-3.1-Pro-Preview correctly classified all 185 CE cases, achieving perfect CE accuracy. Claude4.6-Sonnet made 9 CE errors, while GPT-5.4 made 11 CE errors. In contrast, GPT-OSS-20B made 41 CE errors. Although GPT-OSS-20B matched GPT5.4 and Claude-4.6-Sonnet on BC accuracy, most of its overall performance gap came from its substantially weaker ability to recognize compilation errors. These results suggest that BC cases remain challenging because correctness depends not only on identifying that behavior changed, but also on generating a valid test that compiles and exposes the difference between the original and resulting programs. CE cases pose a different challenge: the model must recognize that the resulting program is uncompilable rather than interpreting the transformation as behavior-preserving or behavior-changing. Overall, the strongest proprietary models achieved high accuracy on both categories, with
Foundation Models as Oracles for Refactoring Correctness Detection
33
Table 7: Results for all evaluated models in the first run. Model GPT-OSS-20B GPT-5.4 Gemini-3.1-Pro-Preview Claude-4.6-Sonnet
Overall Accuracy 182/226 (0.805) 212/226 (0.938) 225/226 (0.996) 214/226 (0.947)
BC Accuracy 38/41 (0.927) 38/41 (0.927) 40/41 (0.976) 38/41 (0.927)
CE Accuracy 144/185 (0.778) 174/185 (0.941) 185/185 (1.000) 176/185 (0.951)
Gemini-3.1-Pro-Preview obtaining the best observed result in this evaluation.
5.6 Mixture of Experts We also evaluated a simple mixture-of-experts (MoE) strategy over the four models considered in this analysis: GPT-OSS-20B, GPT-5.4, Gemini-3.1Pro-Preview, and Claude-4.6-Sonnet. In this setting, an instance is considered correctly solved if at least one model succeeds; that is, we combine model outputs using a logical OR. Since the results reported here rely on the first attempt of each model, this MoE analysis should be interpreted as a complementary analysis of cross-model agreement and residual-error diversity rather than as a repeated-sampling strategy over multiple runs. Figure 10 summarizes the overlap among the bugs correctly detected by each model. The four models jointly solved 163 instances, forming a large common core of cases that were consistently handled across model families. In addition, 39 instances were solved by the three strongest proprietary models, namely Gemini-3.1-Pro-Preview, GPT-5.4, and Claude-4.6-Sonnet, but missed by GPT-OSS-20B. This region captures most of the gap between GPTOSS-20B and the proprietary models. Smaller regions reveal complementary behavior among the remaining systems: 10 instances were solved by Gemini3.1-Pro-Preview, Claude-4.6-Sonnet, and GPT-OSS-20B but missed by GPT-5.4, while 6 instances were solved by Gemini-3.1-Pro-Preview, GPT5.4, and GPT-OSS-20B but missed by Claude-4.6-Sonnet. The remaining non-empty regions were smaller: 4 instances were solved only by Gemini-3.1Pro-Preview and GPT-5.4, 2 only by Gemini-3.1-Pro-Preview and GPTOSS-20B, 1 only by Gemini-3.1-Pro-Preview and Claude-4.6-Sonnet, and 1 only by Claude-4.6-Sonnet and GPT-OSS-20B. A simple OR-based MoE is therefore effective in this benchmark, but its interpretation should be limited to an upper-bound complementarity analysis: it assumes that an external mechanism can identify or select a correct answer whenever at least one model produces one. Nevertheless, the result is useful because it shows that combining model outputs can recover residual errors left by individual systems. This finding motivates future work on cost-aware MoE strategies, such as cascaded evaluation pipelines in which a cheaper or faster model is queried first and stronger models are invoked only for uncertain or unresolved cases.
34
Rohit Gheyi et al.
Gemini-3.1-Pro-Preview
Claude-4.6-Sonnet
1
39
10 1
163
4 GPT-5.4 6
2
GPT-OSS-20B Claude-4.6-Sonnet Gemini-3.1-Pro-Preview
GPT-5.4 GPT-OSS-20B
Fig. 10: Diagram of bugs correctly detected by each model. Numbers indicate the number of bugs in each overlap region.
5.7 Costs Cost is an important consideration when deploying these models at scale. As with the execution-time analysis, our goal is not to provide a definitive or exact cost benchmark. Costs depend on several factors, including hardware availability, cloud provider pricing, API pricing, batching strategy, implementation overhead, and changes in model-serving infrastructure. Therefore, the values reported here should be interpreted as indicative estimates under the conditions of our evaluation and current pricing. We also expect these costs to decrease over time as inference infrastructure improves, hardware becomes more efficient, and providers adjust pricing. Although we run GPT-OSS-20B locally on a Mac Mini M4 Pro, estimating an equivalent cloud rental cost still provides useful perspective on resource efficiency. Based on the listed rental price of a Mac Mini cloud instance, EUR 0.22 per hour5 , and on the recorded end-to-end runtime of the first GPT-OSS-20B run over the 226 instances, the estimated infrastructure cost is approximately EUR 0.31 per complete run. In this run, the mean per-call latency was 22.27 s, the median was 13.61 s, the minimum was 5.52 s, the maximum was 512.67 s, and the total runtime was 1.40 h. These values account only for infrastructure rental time and do not include engineering time, storage, network transfer, or operational overhead. This indicates that GPT-OSS-20B can be evaluated at low monetary cost when suitable local or low-cost cloud 5
https://www.scaleway.com/en/mac-mini-m4/
Foundation Models as Oracles for Refactoring Correctness Detection
35
hardware is available, although its per-instance latency is higher than that of GPT-5.4. For GPT-5.4, the average cost was approximately $0.79 per complete benchmark run. Across these runs, GPT-5.4 consumed approximately 180K tokens per run over the 226 benchmark, comprising about 153K input tokens and 27K output tokens per attempt. The input-token cost is largely fixed by the benchmark prompts, while the small variation in total cost across runs is mainly attributable to differences in generated output length. These numbers highlight a deployment-relevant trade-off between the two main configurations considered in this cost discussion. Compared with the local GPT-OSS-20B setup, GPT-5.4 offers higher observed effectiveness and lower per-call latency, but at a higher monetary cost per complete run. At the same time, the API-based deployment reduces operational overhead because it does not require provisioning or maintaining local inference hardware. This trade-off is especially relevant in scenarios requiring repeated executions, such as pass@k analyses, temperature-sensitivity studies, metamorphic-testing sensitivity analyses, or large-scale benchmarks.
5.8 Execution Time To provide a general sense of runtime, we report the observed latency of each model in our experiments, without aiming at a controlled benchmarking study. These measurements should therefore be interpreted as indicative rather than definitive, since they are affected by factors such as hardware configuration, API latency, implementation overhead, network conditions, and background system load. Table 8 summarizes the latency observed for each model. Among the evaluated models, Llama-3.2-3B was the fastest, with a mean latency of 1.76 s per instance and a median of 1.65 s. This corresponds to approximately 6.6 minutes to process the full benchmark of 226 transformations. However, as discussed in Section 5.4, this low latency came with very poor predictive performance, since Llama-3.2-3B did not correctly solve any instance under our evaluation protocol. GPT-5.4 was also very fast, with a mean response time of 3.25 s per instance and a median of 2.71 s. It processed the full benchmark in approximately 12.2 minutes, although one outlier reached 43.75 s. This makes GPT-5.4 one of the lowest-latency models in our evaluation, combining low latency with high observed predictive performance. Phi-4-14B was slower than Llama-3.2-3B and GPT-5.4, but still relatively efficient, with a mean latency of 9.41 s and a median of 7.88 s, corresponding to approximately 35.4 minutes for the full benchmark. Gemini-3.1-Pro-Preview also showed moderate latency. Its mean response time was 11.41 s per instance and its median was 7.46 s, corresponding to approximately 43 minutes for the full benchmark. Most responses were relatively fast, but one outlier reached 446.78 s. Given that Gemini-3.1-ProPreview achieved the highest observed predictive performance among all
36
Rohit Gheyi et al.
Table 8: Observed execution time per instance. Total time is estimated from the sum of per-instance latencies over the 226 transformations. Model Llama-3.2-3B GPT-5.4 Phi-4-14B Gemini-3.1-Pro-Preview GPT-OSS-20B Gemma-4-31B Qwen-3.6-35B
Mean 1.76s 3.25s 9.41s 11.41s 22.27s 110.73s 611.02s
Median 1.65s 2.71s 7.88s 7.46s 13.61s 69.28s 82.64s
Min. 1.28s 1.75s 5.53s 3.57s 5.52s 26.90s 24.99s
Max. 3.52s 43.75s 22.45s 446.78s 512.67s 870.88s 23,361.43s
Total 6.6m 12.2m 35.4m 43.0m 1.4h 7.0h 38.4h
evaluated models, this result indicates a favorable observed accuracy–latency trade-off in our setting. GPT-OSS-20B showed higher latency than GPT-5.4, Phi-4-14B, and Gemini-3.1-Pro-Preview, with a mean response time of 22.27 s and a median of 13.61 s per instance. This corresponds to approximately 1.4 hours to analyze all 226 transformations. Its runtime was also more variable, with responses ranging from 5.52 s to 512.67 s. The larger open models were slower. Gemma-4-31B achieved the highest observed predictive performance among the evaluated open models, and even outperformed the first-run results of GPT-5.4 and GPT-OSS-20B under the configuration used in our experiments. However, Gemma-4-31B required a mean latency of 110.73 s per instance and a median of 69.28 s, corresponding to approximately 7.0 hours for the full benchmark. Qwen-3.6-35B also exhibited high latency, with a median response time of 82.64 s per instance. Its runtime was highly variable, with several extreme outliers above 23,000 s, which increased its mean latency to 611.02 s and its estimated total runtime to approximately 38.4 hours. Gemini-3.1-Pro-Preview achieved very high accuracy with moderate latency, while GPT-5.4 provided high accuracy with low latency. Beyond runtime alone, these models also provide explanatory feedback about behavioral changes and compilation errors, a capability that may complement traditional refactoring implementations and may be useful in AI-assisted IDEs such as Antigravity [20], Cursor [22], and Windsurf [21].
5.9 Statistical Analysis of Model Accuracy We evaluated four models (GPT-OSS-20B, GPT-5.4, Gemini-3.1-ProPreview, and Claude-4.6-Sonnet) on a dataset of N =226 bugs. For each bug, we recorded a binary outcome indicating whether the model produced the correct classification. Table 9 reports the overall accuracy of each model together with 95% Wilson confidence intervals. Gemini-3.1-Pro-Preview achieved very high observed accuracy, making only one error in the entire benchmark. Because this leaves almost no variability for inferential comparison, we use Gemini-3.1-Pro-Preview mainly as a descriptive reference in this section and focus the formal paired significance analysis on GPT-OSS-20B,
Foundation Models as Oracles for Refactoring Correctness Detection
37
Table 9: Overall accuracy with 95% Wilson confidence intervals. Model GPT-OSS-20B GPT-5.4 Claude-4.6-Sonnet Gemini-3.1-Pro-Preview
Correct 182/226 212/226 214/226 225/226
Accuracy 0.805 0.938 0.947 0.996
95% CI [0.749, 0.852] [0.899, 0.963] [0.909, 0.969] [0.975, 0.999]
Table 10: Pairwise two-sided exact McNemar tests with Holm correction. Pair GPT-OSS-20B vs. GPT-5.4 GPT-OSS-20B vs. Claude-4.6-Sonnet GPT-5.4 vs. Claude-4.6-Sonnet
n11 169 174 202
n10 13 8 10
n01 43 40 12
n00 1 4 2
∆ −0.133 −0.142 −0.009
pexact 7.33×10−5 3.31×10−6 8.32×10−1
pHolm 1.47×10−4 9.92×10−6 8.32×10−1
GPT-5.4, and Claude-4.6-Sonnet. This choice is also methodologically useful because these three models still exhibit a non-trivial number of errors, allowing more informative paired comparisons. Moreover, they represent three practically relevant profiles: a smaller open-weight model that can be executed locally (GPT-OSS-20B), a strong proprietary model (GPT-5.4), and a strong alternative proprietary model from another provider (Claude-4.6-Sonnet). To compare GPT-OSS-20B, GPT-5.4, and Claude-4.6-Sonnet statistically, we used tests for paired dichotomous outcomes. We first applied Cochran’s Q test [50,51] as an omnibus test across the three models. The result indicates that at least one model differs significantly from the others (Q=30.60, p<0.001). We then performed pairwise McNemar exact tests [52] (two-sided), with Holm–Bonferroni correction [53] to control the family-wise error rate over the three pairwise contrasts. To make the analysis fully explicit, Table 10 reports the exact paired contingency counts for each comparison: n11 (both models correct), n10 (A correct and B wrong), n01 (A wrong and B correct), and n00 (both models wrong). We also report the accuracy difference ∆(A−B) = (n10 − n01 )/226. The results show that both GPT-5.4 and Claude-4.6-Sonnet outperform GPT-OSS-20B, with absolute accuracy gains of 13.3% and 14.2%, respectively. In contrast, the difference between GPT-5.4 and Claude-4.6-Sonnet is very small, less than 1%, and is not statistically significant after correction. Thus, the inferential analysis supports an observed separation between GPTOSS-20B and the two stronger proprietary models, while GPT-5.4 and Claude-4.6-Sonnet do not show a statistically significant difference on this benchmark. For completeness, Gemini-3.1-Pro-Preview obtained the highest descriptive accuracy, correctly classifying 225 out of 226 bugs. However, because its performance is very high, we avoid over-interpreting pairwise significance tests involving Gemini-3.1-Pro-Preview.
38
Rohit Gheyi et al.
5.10 Large Projects Many codebases exceed the context window supported by current models; consequently, applying the full-source evaluation prompt from Section 3.7 is impractical for such projects. As a feasibility study, we therefore evaluate large codebases using unified diffs from real projects, following a similar idea [48]. Because this experiment involves real projects and diff-only reasoning, we treat it as a complementary feasibility study rather than as a definitive benchmark. Unlike the controlled benchmark used in the main evaluation, correctness judgments in this setting cannot always be established mechanically from the diff alone. The goal is therefore not to provide a mechanically verified benchmark over large projects, but to examine how foundation-model-based assessment behaves on realistic refactoring diffs and to identify practical difficulties that arise in this setting. We adopt MetaPrompting [42] to guide the model in generating or refining task prompts. Since the model does not have access to the entire source code in this setting, we do not ask it to generate a test case. I used the following prompt to evaluate refactoring correctness in small programs. prompt However, due to the context window, I cannot use it to evaluate larger programs. Create a similar prompt that evaluates the refactoring correctness based on the diff between the source and refactored programs. Do not need to generate a test case when identifying a behavior change.
The variable prompt refers to the prompt described in Section 3.7. In response, the model produced a diff-oriented prompt that targets both compilation errors and behavioral changes while retaining the same verdict schema used in the controlled benchmark. The variable diff includes added, removed, and minimal unchanged context lines. Because diff-only reasoning may not provide enough information to determine correctness with confidence, the model conservatively added a fourth possible verdict: UNKNOWN. Consider the following diff between an initial program and the resulting program after applying a refactoring: <DIFF {diff} DIFF> The diff may include one or more files. Lines removed from the initial program are marked with "-". Lines added in the resulting program are marked with "+". Context lines are unchanged. Evaluate whether the resulting program is a correct refactoring of the initial program, based only on the information available in the diff. Check the following:
Foundation Models as Oracles for Refactoring Correctness Detection
39
1. Compilation correctness: Determine whether the changes shown in the diff introduce an evident compilation problem in the resulting program, such as syntax errors, type errors, unresolved names, invalid imports, invalid method calls, invalid overriding, incompatible access modifiers, or linkage-related problems. 2. Behavioral equivalence: Determine whether the changes shown in the diff preserve the observable behavior of the common public API. Ignore methods not present in both programs; compare only methods with the same signature that are public in both versions. For those common public methods, the initial and resulting programs preserve the same observable behavior, including return values, printed output, thrown exceptions, and externally visible state changes. Return ONLY valid JSON. Do not use markdown. Do not add any text before or after the JSON. Use exactly this schema: { "verdict": "YES | NO - COMPILATION ERROR | NO - BEHAVIOR CHANGE | UNKNOWN", "explanation": "1-4 sentences" } Rules: - "verdict" must be exactly one of: "YES" "NO - COMPILATION ERROR" "NO - BEHAVIOR CHANGE" "UNKNOWN" - Use "YES" only when the diff provides sufficient evidence that the resulting program compiles and preserves the behavior of the common public API. - Use "NO - COMPILATION ERROR" only when the diff provides concrete evidence of a compilation problem introduced by the resulting program. - Use "NO - BEHAVIOR CHANGE" only when the diff provides concrete evidence of a behavioral difference in a common public method, such as a changed return value, changed printed output, changed thrown exception, changed mutation of externally visible state, or changed public-visible interaction. - Use "UNKNOWN" when the diff does not contain enough surrounding context to determine compilation correctness or behavioral equivalence with confidence. - Do not invent compilation errors. - Do not claim behavior change unless the diff shows a specific differing return value, output, exception, state change, or public-visible interaction. - Do not assume behavior preservation merely because the change is labeled as a refactoring. - Do not assume missing declarations, imports, fields, methods, or classes unless they are visible in the diff. - If a symbol is added, removed, renamed, or has its type, visibility, signature, inheritance relationship, or initialization changed, consider whether the diff provides enough context to determine whether all affected uses remain valid. - If a public method is added or removed, ignore that method for behavioral equivalence, but consider whether the change affects existing common public methods. - Private, protected, or package-private changes are relevant only if they affect compilation or the observable behavior of common public methods. - Formatting-only changes, comment-only changes, import reordering, or equivalent renamings should be considered behavior-preserving unless they create a concrete compilation or behavior issue.
40
Rohit Gheyi et al.
- The "explanation" must briefly justify the answer with concrete evidence from the diff. - Return JSON only.
We evaluated 44 refactorings applied to real-world open-source Java projects using IntelliJ-IDEA 2024.1.4 (Table 11). The dataset covers twelve widely used refactoring types: Extract Class, Extract Interface, Extract Superclass, Inline Method, Move Method, Pull Up Field, Pull Up Method, Push Down Field, Push Down Method, Rename Field, Rename Method, and Rename Class. Each transformation corresponds to a single refactoring instance manually applied in IntelliJ-IDEA. In April 2026, we used the GPT-5.4 API to adapt the full-source prompt to a diff-only setting, setting reasoning effort to medium and leaving all other configuration parameters at their default values. GPT-5.4 returned YES for 24 transformations, NO - BEHAVIOR CHANGE for 2 transformations, and UNKNOWN for 18 transformations. After author adjudication, the 24 YES verdicts were judged as supported by the visible diff evidence. In these cases, the edited declarations, implementations, and visible call sites were updated coherently, and no concrete compilation error or behavioral difference in the common public API was evident from the diff. The two NO BEHAVIOR CHANGE verdicts were not confirmed by the authors: although the model identified a potential behavioral concern, the explanation did not provide sufficient evidence from the diff to establish a concrete behavioral difference under our adopted equivalence notion. The 18 UNKNOWN cases were mainly due to insufficient contextual information in the diffs. In several instances, public or protected members were renamed, moved, or removed, such as methods, fields, interfaces, and nested types, but the diffs did not show all project-wide references, subclasses, constructor declarations, imports, or external callers that could still depend on the old declarations. In other cases, the visible changes appeared locally consistent and no clear behavioral change was evident in common public methods, but correctness depended on unseen code, such as whether moved methods were implemented elsewhere, whether fields remained accessible across packages, whether callbacks could be null or behaviorally different, or whether all concrete subclasses still satisfied required interfaces. Even for a human reviewer, deciding behavior preservation from such diff-only evidence can be difficult in large programs, since the relevant semantic dependencies may be distributed across files, subclasses, package boundaries, generated code, framework callbacks, or external clients. A seemingly local refactoring can have non-local effects, for example by changing overload resolution, enabling or disabling method overriding, affecting dynamic dispatch, or altering which declarations remain visible to clients. Because the diff-only setting does not always provide enough context to mechanically establish whether a transformation preserves behavior, we manually assessed the model judgments in this analysis. Each GPT-5.4 judgment was independently reviewed by two authors. The reviewers applied the same decision criterion: a judgment was considered correct only when the model’s
Foundation Models as Oracles for Refactoring Correctness Detection
41
Table 11: Java projects used in the large-project evaluation. Project Lettuce
Domain KLOC Stars Contr. Transf. A scalable thread-safe Redis 234 5.6 135 30 client for synchronous, asynchronous, and reactive usage. 454 2.6 115 1 Apache Gobblin A distributed data integration framework. Google Maps Services A Java client for Google Maps 38 1.7 96 3 Services. A framework to create Spring674 77.1 1,156 4 Spring Boot based applications. RefMiner A refactoring detection tool. 127 0.4 18 6
verdict was supported by evidence visible in the diff, under the behavioralequivalence notion adopted in this study. The two initial reviewers agreed on 42 out of 44 cases. The two disagreements involved borderline cases in which the diff suggested possible API-level or visibility-related changes, but the available context was insufficient to determine all behavioral consequences mechanically. After discussion, the reviewers decided to keep the baseline classification that the transformations did not change behavior, because the diff alone did not provide sufficient evidence of an observable behavioral change. Therefore, we interpret this large-project diff-only analysis as a feasibility study rather than as a definitive benchmark. While the diff-only strategy is a useful exploratory step, important challenges remain in assessing behavior preservation for real-world programs using foundation models. The approach is particularly difficult when the decisive evidence lies outside the edited regions or depends on runtime semantics that cannot always be inferred from static analysis over the diff alone. This limitation is not specific to foundation models: diff-only review can also be challenging for human reviewers when behavior preservation depends on global semantic relationships that are not visible in the transformation. As future work, we plan to refine the prompt to make the adopted notion of behavioral equivalence more explicit and to investigate retrieval-augmented generation techniques that provide richer project context for patch analysis, building on recent work on large-scale refactoring automation [54].
5.11 Threats to Validity Several threats to validity apply to this study [43]. 5.11.1 Internal Validity Internal-validity threats stem from prompt and context sensitivity, model nondeterminism, provider-specific execution settings, ground-truth reconstruction, human adjudication, and possible data leakage. Although we used standardized prompt templates across models, small wording changes or different context
42
Rohit Gheyi et al.
representations may affect predictions. This is particularly relevant because the task requires both classification and, for behavioral-change cases, generation of an executable test oracle. We fixed prompts and evaluation scripts, ran repeated trials where feasible, and reported stability measures such as accuracy spread, cumulative coverage, tar@, and cons@. Still, we did not systematically ablate prompt wording, context order, context amount, or all decoding configurations. Ground-truth reliability is another internal-validity threat. Not all bug reports provide self-contained examples, and reconstructing minimal reproducing programs is manual and potentially error-prone. Reproduction may depend on JDK versions, IDE versions, project configuration, dependencies, and historical tool behavior. To reduce this risk, we relied on available attachments and version information, recorded environment details, compiled all programs, validated behavior-preserving cases with SafeRefactor, and manually rechecked ambiguous cases. We revised the dataset construction and validation procedure to make the ground truth more explicit, executable, and reproducible, and removed structurally redundant near-duplicate cases found during revalidation. Nevertheless, reconstruction remains a possible source of error. Data leakage is a further concern. We applied metamorphic testing to all benchmark instances to assess whether intended semantics-preserving changes would substantially alter predictions. The results showed little overall degradation for GPT-OSS-20B and GPT-5.4, which is not consistent with a simple exact-match leakage explanation. However, metamorphic testing does not rule out leakage. A model may still benefit from prior exposure to related code, refactoring patterns, or abstract bug structures, and our operators cover only a subset of possible surface-level variations. Model and infrastructure effects may also influence the results. Some models were executed through vendor APIs, others through local wrappers such as Ollama, and Claude-4.6-Sonnet was evaluated through the web interface. These settings may differ in hidden system prompts, decoding defaults, truncation behavior, rate limits, wrappers, or model revisions. Proprietary models may also change over time. Therefore, the results should be understood as a snapshot of the models and configurations available during our evaluation. 5.11.2 Construct Validity Construct-validity threats arise from how we operationalize refactoring correctness, model outputs, and evaluation metrics. Our benchmark uses binary labels for compilation errors and behavioral changes, but this abstraction may hide partial correctness, uncertainty, or explanations that are technically plausible but insufficiently justified. Some model outputs contain correct-looking verdicts with weak explanations, while others identify plausible risks without enough evidence to establish a compilation error or behavioral change under our adopted equivalence notion. Disagreements may also arise from different interpretations of behavioral preservation, such as how to treat exception locations, public API changes, removed public members, or visibility changes. To mitigate this, we compiled programs, compiled and executed the model-generated tests with
Foundation Models as Oracles for Refactoring Correctness Detection
43
JUnit to check whether they actually expose the reported behavioral differences, used SafeRefactor on behavior-preserving transformations, manually reviewed explanations, and distinguished confirmed bugs from unsupported concerns and insufficient-evidence cases. Future work should score explanation quality and explore uncertainty-aware metrics. The main bug-detection benchmark contains only positive bug instances. Consequently, accuracy on this benchmark is numerically equivalent to recall and does not measure false-positive behavior on truly behavior-preserving refactorings. We partially address this limitation through the complementary true-positive study in Section 5.3, which evaluates 50 behavior-preserving refactorings. However, this study is smaller and does not fully characterize falsepositive behavior across the broader space of valid refactorings, projects, and refactoring types. Thus, our results provide stronger evidence about detecting known refactoring bugs than about deployment-level false-positive rates. 5.11.3 External Validity Our findings may not generalize beyond the studied setting. The benchmark focuses on Java refactoring bugs from popular IDEs and does not exhaustively cover all Java features, libraries, build systems, reserved keywords, refactoring tools, JDK versions, or ecosystems. Some features remain untested or underrepresented, such as assert, volatile, transient, native, and strictfp. The dataset is also imbalanced, since compilation errors outnumber behavioral changes. We therefore report per-category BC and CE metrics, but broader validation on larger, stratified corpora across languages, projects, refactoring kinds, IDEs, and JDK versions is needed. The large-project diff-only study introduces additional external-validity threats. Since the model receives unified diffs rather than full project context, it may lack information about unseen call sites, imports, subclasses, dependencies, build files, framework callbacks, or external clients. For this reason, the prompt includes an UNKNOWN verdict, and we treat this experiment as a feasibility study rather than as a definitive benchmark. Human review reduces the risk of blindly trusting model outputs, but does not eliminate author bias in borderline cases involving API compatibility or behavioral equivalence. 5.11.4 Conclusion Validity The statistical and comparative analyses should be interpreted cautiously. Statistical tests and confidence intervals are conditional on this dataset and model snapshot, and limited repeated runs for some proprietary models constrain the ability to estimate variance. Differences between models may partly reflect training data, inference infrastructure, hidden system prompts, model revisions, or reasoning style rather than inherent capability gaps. We mitigated this threat by evaluating all models on the same benchmark and by using paired statistical tests for the main model comparisons, but the conclusions remain conditional on the evaluated setting.
44
Rohit Gheyi et al.
The mixture-of-experts, execution-time, and cost analyses also have limitations. The MoE analysis uses an optimistic OR-based criterion, measuring potential cross-model complementarity rather than a deployable decision procedure. Runtime and cost depend on hardware, provider, API latency, network conditions, batching, implementation overhead, and changing prices. These values are useful for understanding approximate trade-offs under our experimental conditions, but they should not be read as stable or universal benchmarks. Future work should increase sample sizes, expand repetitions, evaluate additional models, and investigate cost-aware and confidence-aware routing strategies. Taken together, these threats delimit the strength of the empirical evidence. The study supports claims about observed bug-detection performance on known Java refactoring bugs, robustness under the tested metamorphic transformations, executable oracle generation for the evaluated benchmark, and complementary evidence from a limited set of behavior-preserving refactorings. It does not establish complete semantic reliability, general-purpose behavioral-equivalence checking, or deployment-ready correctness guarantees.
6 Related Work Opdyke and Johnson [55, 2] introduced the foundational concept of refactoring, while Roberts [3] pioneered the automation of basic refactoring operations. Subsequent work by Tokuda and Batory [56] demonstrated that Opdyke’s preconditions alone were insufficient to fully ensure behavior preservation, highlighting the complexity of ensuring correctness in refactoring transformations. AlOmar et al. [57] performed a comprehensive systematic mapping study on behavior preservation during software refactoring, providing valuable insights into current practices, challenges, and research gaps in the field. Schäfer et al. [10] further emphasized the challenges of ensuring correctness across all language constructs. Unlike these foundational works, which focus on formalizing, implementing, or proving refactorings, our work investigates whether foundation models can serve as oracle-like aids for detecting errors in existing refactoring transformations. We therefore position foundation models as a complementary correctness layer rather than as a replacement for traditional refactoring implementations or formal analyses.
6.1 Testing Refactoring Implementations Daniel et al. [6] proposed automated testing methodologies for Java refactoring implementations by generating programs and employing programmatic oracles to uncover faults such as overly weak preconditions. Their approach primarily detected issues manifesting as compilation errors through systematic program generation. Soares et al. [5] later developed a systematic approach using JDolly to generate Java programs and SafeRefactor [8] to detect behavioral changes through automatically generated tests. Their methodology
Foundation Models as Oracles for Refactoring Correctness Detection
45
successfully identified 106 bugs, including compilation errors and behavioral changes, across 39 refactoring implementations. Drienyovszky et al. [58] similarly validated Erlang refactoring tools using automated property-based random testing based on formal refactoring specifications. Our work differs from these approaches in both objective and oracle design. Rather than testing refactoring implementations by generating new input programs or relying on hand-crafted properties, we evaluate whether foundation models can judge the correctness of already observed refactoring transformations. This allows us to analyze real bug reports from mature IDEs using zero-shot prompting, without relying directly on specialized generators, formal refactoring specifications, or executable testgeneration infrastructure. These directions are complementary: dynamic and property-based oracles provide strong evidence when executable tests or formal properties are available, whereas model-based assessment may be useful for historical bugs, heterogeneous transformations, and scenarios where executable oracles are difficult to configure. Dong et al. [30] presented a ChatGPT-driven framework for testing refactoring implementations. Their approach synthesizes test programs using LLMs by mining feature libraries from bug reports and existing test cases, encoding preconditions, and guiding generation through prompt templates. The generated programs are then used for differential testing across multiple refactoring implementations, leading to the discovery of 115 Java refactoring bugs. In contrast, our goal is not to synthesize programs for testing refactoring implementations, but to evaluate whether a given transformation is correct. Our oracle directly classifies refactoring outcomes as behavior-preserving, behavior-changing, or compilation-invalid. These approaches are complementary: LLM-generated test inputs can expose new refactoring-implementation defects, while our modelbased oracle can help assess whether the resulting transformations appear to preserve behavior. Wang et al. [32] conducted an extensive manual study analyzing 518 refactoring bugs across three widely used IDEs (Eclipse, NetBeans, IntelliJIDEA), systematically identifying root causes, bug symptoms, and inputprogram characteristics. Their transferability analysis uncovered 130 previously unreported bugs, demonstrating the persistent nature of refactoring-related defects. Building on their dataset, we focus on two critical bug classes: compilation errors and behavioral changes. We refine the original artifacts by reconstructing executable Java input–output program pairs, compiling them to identify refactoring-induced compilation errors, and providing a JUnit test for each behavioral-change case to expose the behavioral difference. Our study then asks whether foundation models can detect these failures directly, providing an orthogonal perspective on refactoring-implementation reliability.
6.2 Tools Schäfer et al. [9] improved Java refactoring support in Eclipse by formalizing transformations and implementing them in the JRRT tool, advancing soundness
46
Rohit Gheyi et al.
guarantees but requiring substantial language-specific engineering effort. Kim et al. [59] showed that developers often apply refactorings manually, with the notable exception of Rename refactoring, emphasizing the need for more trustworthy automated support. Murphy-Hill et al. [60] and Eilertsen and Murphy [61] further showed that refactoring-tool usability depends not only on correctness, but also on informative feedback, control, and developer trust. Our work complements these tool-oriented studies by focusing on an explanationoriented validation layer for refactoring correctness. Rather than implementing refactorings or replacing existing IDEs, foundation models may help identify suspicious transformations and explain potential correctness risks. This aligns with prior calls for more informative refactoring feedback and suggests a possible role for foundation models in AI-assisted IDEs. Horikawa et al. [62] presented a large-scale empirical study on refactoring activities performed by AI coding agents in real-world open-source Java projects. They find that agentic commits frequently include refactorings, mostly lowlevel and consistency-oriented edits. Our work complements their study by examining how foundation models can evaluate the correctness of refactorings, including those produced by AI agents. In this sense, our oracle-like analysis may serve as one validation component for agentic refactoring workflows.
6.3 LLM-Assisted Refactoring Xu et al. [63] evaluated LLMs for code-related tasks, while Hou et al. [13] surveyed the broader use of LLMs in software engineering. Fan et al. [15] highlighted open research problems at the intersection of LLMs and refactoring. More directly related to our topic, Martinez et al. [64] conducted a systematic literature review of LLM-based refactoring research. Their review shows that the area is growing rapidly, but remains fragmented in terms of datasets, prompting strategies, tooling, and, especially, how correctness and accuracy are defined and measured. They also identify recurring challenges such as hallucinations, erroneous code generation, context loss, difficulties with complex refactorings, and scalability limitations. Our study addresses one of these central gaps: systematic correctness assessment for LLM-assisted and tool-produced refactorings. Rather than asking models to generate refactorings, recommend opportunities, or improve code quality, we evaluate whether foundation models can serve as oracle-like components for already-applied transformations, detecting both compilation errors and behavioral changes. White et al. [65] proposed prompt patterns for applying refactorings with ChatGPT, and AlOmar et al. [66] analyzed developer–AI conversations involving refactoring requests. Liu et al. [67] studied GPT-4 and Gemini for identifying refactoring opportunities and recommending edits, introducing RefactoringMirror to reapply inferred transformations through a vetted refactoring implementation. Depalma et al. [68] evaluated ChatGPT on Java refactoring tasks and reported both useful improvements and variability across identical prompts. Pomian et al. [69] proposed EM-Assist, an IntelliJ-IDEA
Foundation Models as Oracles for Refactoring Correctness Detection
47
plugin that uses LLMs to suggest and rank Extract Method refactorings. Other work has explored smell removal [70], Python simplification and transformation [71, 72], refactoring recommendation [73,74], and expert-guided or agentic refactoring [75, 76]. These studies primarily focus on generating, recommending, applying, or improving refactorings. Our work is complementary: we focus on validating the correctness of transformations after they have been produced, including transformations that originate from an IDE, a developer, or an AI coding agent. This makes our oracle-style approach a potential downstream validation layer for LLM-assisted refactoring pipelines.
6.4 Comparison with Traditional Baselines To position foundation models with respect to executable approaches, we compared our method against two traditional baselines on the same benchmark: Java compilation checking for compilation-error detection and SafeRefactor [8] for behavior-preservation assessment. For compilation errors, OpenJDK-Temurin21.0.7+6 correctly identifies all CE cases in our dataset, as expected, since compilation validity is ultimately determined by the Java language implementation and compiler version. SafeRefactor performs both compilation checking and dynamic behavior-preservation assessment through affected-entity analysis and automatically generated tests. In our setting, SafeRefactor correctly handles 221 out of 226 benchmark cases, corresponding to 97.8% accuracy. The five incorrect cases are IDs 122, 137, 140, 189, and 190. In our benchmark, these are compilation-error cases: the source program compiles, but the resulting program does not. However, SafeRefactor reported that both the source and resulting programs failed to compile. As a result, it misclassified the failure as an invalid source-program setup rather than as a compilation error introduced by the refactoring. Thus, these failures indicate that SafeRefactor could not properly analyze the transformation under its required execution environment. SafeRefactor depends on Randoop [77] for test generation and requires a Java 8 execution environment. However, some benchmark programs use newer Java features, such as var, switch expressions with yield, or static nested classes in contexts supported only by later Java versions. Consequently, these failures are caused by version incompatibilities in the execution environment rather than by the refactoring transformations themselves. In comparison, Gemini-3.1-Pro-Preview achieved a correct classification rate of 99.6%, slightly outperforming SafeRefactor on this benchmark, while GPT-5.4 and GPT-OSS-20B detected 93.8% and 80.5% of the bugs, respectively, in the first-run setting. Thus, although SafeRefactor remains highly competitive and provides executable evidence when its Java 8-based toolchain is applicable, the strongest foundation model was able to match or exceed its observed accuracy while also handling newer Java features present in our dataset. This comparison highlights a key trade-off. Compilers and tools such as SafeRefactor provide strong executable evidence and are generally deterministic, fast, and inexpensive when applicable. Foundation models, in
48
Rohit Gheyi et al.
contrast, are less deterministic and do not provide formal guarantees, but they may be more adaptable with respect to language evolution, can produce naturallanguage explanations, and can be applied in settings where dynamic test generation or project configuration is difficult. We therefore view foundation models as complementary oracles rather than replacements for traditional analysis-based approaches.
6.5 Robustness Sallou et al. [43] examined systemic risks associated with deploying LLMs in software engineering contexts, including data leakage, reproducibility challenges, and dependence on closed-source models. They propose mitigation guidelines emphasizing rigorous empirical validation. Zhang et al. [78] surveyed LLM hallucinations and classify them into input-, context-, and fact-conflicting categories. Our study explicitly considers these risks. We evaluate repeated attempts, temperature sensitivity, and metamorphic transformations to assess stability and sensitivity to input-preserving variations. We also analyze malformed outputs, incorrect semantic verdicts, and cases in which generated tests fail to compile or fail to expose behavioral differences. These analyses do not eliminate threats such as data leakage, hallucination, or provider-specific behavior, but they provide evidence, within our setting, about when model predictions are stable and when they remain fragile.
7 Conclusions In this article, we investigated the potential of foundation models to serve as oracle-like components for detecting refactoring bugs, including both behavioral changes and compilation errors introduced by program refactorings. Our empirical results show that these models can provide informative signals about refactoring correctness across real-world bugs reported over more than a decade in widely used Java IDEs (IntelliJ-IDEA, Eclipse, and NetBeans). Using zero-shot prompting without task-specific training, the models were evaluated across multiple refactoring types and Java language features, while also producing natural-language rationales that may help developers inspect their predictions. The evaluation reveals substantial performance differences between models. In the first-run setting, GPT-OSS-20B achieved 80.5% overall accuracy, while GPT-5.4, Claude-4.6-Sonnet, and Gemini-3.1-Pro-Preview reached 93.8%, 94.7%, and 99.6%, respectively. Among the evaluated open models, Gemma-4-31B achieved the strongest result, correctly classifying 96.5% of the cases. Repeated sampling further improved cumulative coverage for GPT-OSS-20B and GPT-5.4, although majority voting was less effective than using multiple attempts to recover at least one correct answer. The execution-time and cost analyses highlight deployment-relevant tradeoffs. Smaller open models were faster but less accurate, while stronger open
Foundation Models as Oracles for Refactoring Correctness Detection
49
models such as Gemma-4-31B achieved high accuracy at higher runtime cost. Gemini-3.1-Pro-Preview and GPT-5.4 offered the best latency–accuracy balance among the strongest models in our setting. Local cloud execution of GPT-OSS-20B may be useful for exploratory analyses or first-pass triage, whereas stronger API-based models may be preferable when higher accuracy, stability, or low latency is required. These estimates are indicative rather than definitive, since prices and inference infrastructure are likely to evolve. Our findings also indicate that LLMs may help analyze refactoring transformations before they are applied to an entire codebase. This preventive use case may be useful in large projects, where full builds or test suites may be costly, slow, or dependent on complex environments. The large-project feasibility study reinforces this potential, but also shows that diff-only reasoning often requires an UNKNOWN outcome because decisive evidence may lie outside the visible patch. An additional insight concerns the impact of language evolution on refactoring-validation techniques. Traditional tools that rely on automatic test generation, such as SafeRefactor, require continual engineering effort to support new language constructs and are constrained by the test generators on which they depend. In contrast, foundation models were able, in our dataset, to analyze refactorings involving newer Java features and provide explanations tied to their semantics. This suggests that LLM-based oracles may offer a flexible complement to traditional validation tools as programming languages evolve, although this flexibility does not provide formal guarantees and must be validated per setting. Overall, we do not view foundation models as replacements for existing validation tools, but rather as complementary mechanisms. Deterministic techniques should be preferred whenever a property can be checked directly, especially for compilation errors. At the same time, foundation models provide a unified analysis interface and useful signals in settings where traditional tools are incomplete, costly, or difficult to configure. Hybrid workflows combining deterministic tooling with LLM-based analysis, and explanation appear to be a promising direction for practical adoption. Nevertheless, several limitations remain. The models are sensitive to prompt design, context representation, decoding configuration, and the adopted definition of behavioral equivalence. Context-window limitations restrict direct analysis of large projects, and model non-determinism introduces stability concerns for production settings. Human oversight remains crucial, especially when model explanations are plausible but insufficiently grounded. For future work, we plan to expand the evaluation to additional programming languages, larger and more diverse codebases, and a broader set of refactoring scenarios. We also intend to investigate tighter integration with compilers, static analyzers, test-generation tools, and retrieval-augmented project context, particularly to improve systematically hard compilation-error cases and ambiguous diff-only scenarios. In addition, we plan to explore structured prompting, temperature and decoding-parameter sensitivity for larger closed models such as GPT-5.4, ensemble, mixture-of-experts, and agentic strategies to improve stability and reduce attempt-sensitive errors. Finally, we see a promising opportunity in developing lightweight refactoring-aware checkers
50
Rohit Gheyi et al.
and integrating foundation models into AI-augmented IDEs for interactive refactoring assistance, validation, and decision support.
Acknowledgments We want to thank the anonymous reviewers for their insightful suggestions. This work was partially supported by CNPq, FAPESQ-PB and FAPEAL grants.
Funding This work was partially supported by CNPq, FAPESQ-PB, and FAPEAL grants.
Ethical Approval Not applicable.
Informed Consent Not applicable.
Author Contributions Rohit Gheyi: Methodology, Investigation, Conceptualization, Software, Data Curation, Writing – Original Draft. Jonhnanthan Oliveira: Software, Data Curation, Validation, Writing – Review and Editing. Rian Melo: Software, Data Curation, Validation, Writing – Review and Editing. Márcio Ribeiro: Methodology, Investigation, Conceptualization, Writing – Original Draft. Baldoino Fonseca: Methodology, Investigation, Conceptualization, Writing – Original Draft.
Data Availability All data is available as supplementary material.
Conflict of Interest The authors declare no conflicts of interest relevant to this article.
Foundation Models as Oracles for Refactoring Correctness Detection
51
References 1. M. Fowler, Refactoring: improving the design of existing code, Addison-Wesley, 1999. 2. W. Opdyke, Refactoring Object-oriented Frameworks, Ph.D. thesis, UIUC (1992). 3. D. Roberts, Practical Analysis for Refactoring, Ph.D. thesis, University of Illinois at Urbana-Champaign (1999). 4. Y. Golubev, Z. Kurbatova, E. A. AlOmar, T. Bryksin, M. W. Mkaouer, One thousand and one stories: a large-scale survey of software refactoring, in: Proceedings of the Foundations of Software Engineering, ACM, 2021, p. 1303–1313. 5. G. Soares, R. Gheyi, T. Massoni, Automated Behavioral Testing of Refactoring Engines, IEEE Transactions on Software Engineering 39 (2) (2013) 147–162. 6. B. Daniel, D. Dig, K. Garcia, D. Marinov, Automated testing of refactoring engines, in: Proceedings of the Foundations of Software Engineering, ACM, 2007, pp. 185–194. 7. F. Steimann, A. Thies, From public to private to absent: Refactoring Java programs under constrained accessibility, in: Proceedings of European Conference on Object-Oriented Programming, Springer, 2009, pp. 419–443. 8. G. Soares, R. Gheyi, D. Serey, T. Massoni, Making program refactoring safer, IEEE Software 27 (4) (2010) 52–57. 9. M. Schäfer, O. de Moor, Specifying and implementing refactorings, in: Proceedings of the Object-Oriented Programming, Systems, Languages, and Applications, ACM, 2010, pp. 286–301. 10. M. Schäfer, T. Ekman, O. de Moor, Challenge Proposal: Verification of Refactorings, in: Proceedings of the International Conference on Programming Languages Meets Program Verification, ACM, 2008, pp. 67–72. 11. J. Bloch, N. Gafter, Java Puzzlers: Traps, Pitfalls, and Corner Cases, Addison-Wesley, 2005. 12. E. Tempero, T. Gorschek, L. Angelis, Barriers to Refactoring, Communications of the ACM 60 (10) (2017) 54–61. 13. X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, H. Wang, Large language models for software engineering: A systematic literature review, Transactions on Software Engineering and Methodology 33 (8) (2024) 220:1–220:79. 14. J. Wang, Y. Huang, C. Chen, Z. Liu, S. Wang, Q. Wang, Software testing with large language models: Survey, landscape, and vision, IEEE Transactions on Software Engineering 50 (2024) 911–936. 15. A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, J. M. Zhang, Large language models for software engineering: Survey and open problems, in: Proceedings of the International Conference on Software Engineering: Future of Software Engineering, IEEE, 2023, pp. 31–53. 16. F. Steimann, A. Thies, From behaviour preservation to behaviour modification: constraintbased mutant generation, in: Proceedings of the International Conference on Software Engineering, ACM, 2010, pp. 425–434. 17. R. A. DeMillo, R. J. Lipton, F. G. Sayward, Hints on test data selection: Help for the practicing programmer, Computer 11 (4) (1978) 34–41. 18. Y. Jia, M. Harman, An analysis and survey of the development of mutation testing, IEEE Transactions on Software Engineering 37 (5) (2011) 649–678. 19. A. V. Aho, R. Sethi, J. D. Ullman, Compilers: Principles, Techniques, and Tools, AddisonWesley, Reading, Massachusetts, U.S.A., 1986. 20. Google, Antigravity, https://antigravity.google (2026). 21. Windsurf, Windsurf AI IDE, https://windsurf.com/editor (2026). 22. Cursor, The AI Code Editor, https://www.cursor.com (2026). 23. VSCode, Visual studio code, https://code.visualstudio.com (2026). 24. JetBrains, IntelliJ IDEA, https://www.jetbrains.com/idea/ (2026). 25. Eclipse.org., Eclipse Project, https://www.eclipse.org/topics/ide/ (2026). 26. Apache, Netbeans IDE, http://www.netbeans.org (2026). 27. N. Rachatasumrit, M. Kim, An empirical investigation into the impact of refactoring on regression testing, in: Proceedings of the International Conference on Software Maintenance, IEEE, 2012, pp. 357–366.
52
Rohit Gheyi et al.
28. M. Gligoric, F. Behrang, Y. Li, J. Overbey, M. Hafiz, D. Marinov, Systematic testing of refactoring engines on real software projects, in: Proceedings of the European Conference on Object-Oriented Programming, Springer Berlin, 2013, pp. 629–653. 29. M. Mongiovi, R. Gheyi, G. Soares, L. Teixeira, P. Borba, Making refactoring safer through impact analysis, Science of Computer Programming 93 (2014) 39–64. 30. C. Dong, Y. Jiang, Y. Zhang, Y. Zhang, H. Liu, ChatGPT-based test generation for refactoring engines enhanced by feature analysis on examples, in: Proceedings of the International Conference on Software Engineering, ACM, 2025, pp. 746–746. 31. J. B. Goodenough, S. L. Gerhart, Toward a theory of test data selection, Transactions on Software Engineering 1 (2) (1975) 156–173. 32. H. Wang, Z. Xu, H. Zhang, N. Tsantalis, S. H. Tan, Towards understanding refactoring engine bugs, Transactions on Software Engineering and Methodology 35 (5) (2026) 1–55. 33. T. Zimmermann, R. Premraj, N. Bettenburg, S. Just, A. Schröter, C. Weiss, What makes a good bug report?, IEEE Transactions on Software Engineering 36 (5) (2010) 618–643. 34. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, W. Zaremba, Evaluating large language models trained on code (2021). arXiv:2107.03374. 35. B. Atil, S. Aykent, A. Chittams, L. Fu, R. J. Passonneau, E. Radcliffe, G. R. Rajagopal, A. Sloan, T. Tudrej, F. Ture, Z. Wu, L. Xu, B. Baldwin, Non-determinism of “deterministic” LLM settings (2025). arXiv:2408.04667. 36. W.-L. Chiang, L. Zheng, Y. Sheng, A. N. Angelopoulos, T. Li, D. Li, B. Zhu, H. Zhang, M. I. Jordan, J. E. Gonzalez, I. Stoica, Chatbot arena: an open platform for evaluating LLMs by human preference, in: Proceedings of the International Conference on Machine Learning, PMLR, 2024, pp. 8359–8388. 37. LangChain API, OllamaLLM, https://api.python.langchain.com/en/latest/ollama/ llms/langchain_ollama.llms.OllamaLLM.html (2026). 38. OpenAI, Python API library, https://github.com/openai/openai-python (2026). 39. P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, G. Neubig, Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing, ACM Computing Surveys 55 (9) (2023) 1–35. 40. DAIR.AI, Prompt Engineering Guide, https://www.promptingguide.ai/techniques (2026). 41. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, Language models are unsupervised multitask learners, https://cdn.openai.com/better-language-models/ language_models_are_unsupervised_multitask_learners.pdf (2019). 42. Y. Hou, H. Dong, X. Wang, B. Li, W. Che, MetaPrompting: Learning to learn better prompts (2022). arXiv:2209.11486. 43. J. Sallou, T. Durieux, A. Panichella, Breaking the silence: the threats of using LLMs in software engineering, in: Proceedings of the International Conference on Software Engineering: New Ideas and Emerging Results, ACM, 2024, pp. 102–106. 44. L. Applis, A. Panichella, R. Marang, Searching for quality: Genetic algorithms and metamorphic testing for software engineering ML, in: Proceedings of the Genetic and Evolutionary Computation Conference, ACM, 2023, pp. 1490–1498. 45. T. Y. Chen, F. Kuo, H. Liu, P. Poon, D. Towey, T. H. Tse, Z. Q. Zhou, Metamorphic testing: A review of challenges and opportunities, Computing Surveys 51 (1) (2018) 4:1–4:27. 46. R. Pawlak, M. Monperrus, N. Petitprez, C. Noguera, L. Seinturier, Spoon: A Library for Implementing Analyses and Transformations of Java Source Code, Software: Practice and Experience 46 (9) (2015) 1155–1179. 47. A. Holtzman, J. Buys, L. Du, M. Forbes, Y. Choi, The curious case of neural text degeneration, in: International Conference on Learning Representations, OpenReview.net, 2020.
Foundation Models as Oracles for Refactoring Correctness Detection
53
48. P. Simões, R. Gheyi, R. Melo, J. Oliveira, M. Ribeiro, W. Assunção, Refmodel: Detecting refactorings using foundation models, in: Proceedings of the Brazilian Symposium on Software Engineering, SBC, 2025, pp. 811–817. 49. M. Mongiovi, G. Mendes, R. Gheyi, G. Soares, M. Ribeiro, Scaling testing of refactoring engines, in: Proceedings of the International Conference on Software Maintenance and Evolution, IEEE, 2014, pp. 371–380. 50. W. G. Cochran, The comparison of percentages in matched samples, Biometrika 37 (3/4) (1950) 256–266. 51. W. J. Conover, Practical Nonparametric Statistics, Wiley, 1999. 52. Q. McNemar, Note on the sampling error of the difference between correlated proportions or percentages, Psychometrika 12 (2) (1947) 153–157. 53. S. Holm, A simple sequentially rejective multiple test procedure, Scandinavian Journal of Statistics 6 (2) (1979) 65–70. 54. F. Batole, A. Bellur, M. Dilhara, M. R. Ullah, Y. Zharov, T. Bryksin, K. Ishikawa, H. Chen, M. Morimoto, S. Motoura, T. Hosomi, T. N. Nguyen, H. Rajan, N. Tsantalis, D. Dig, Leveraging LLMs, IDEs, and semantic embeddings for automated move method refactoring (2025). arXiv:2503.20934. 55. W. Opdyke, R. Johnson, Refactoring: An Aid in Designing Application Frameworks and Evolving Object-Oriented Systems, in: Proceedings of the Symposium Object-Oriented Programming Emphasizing Practical Applications, Marist College, 1990, pp. 274–282. 56. L. Tokuda, D. Batory, Evolving Object-Oriented Designs with Refactorings, Automated Software Engineering 8 (1) (2001) 89–120. 57. E. A. AlOmar, M. W. Mkaouer, C. D. Newman, A. Ouni, On preserving the behavior in software refactoring: A systematic mapping study, Information and Software Technology 140 (2021) 106675. 58. D. Drienyovszky, D. Horpácsi, S. J. Thompson, Quickchecking refactoring tools, in: Proceedings of Workshop on Erlang, ACM, 2010, pp. 75–80. 59. M. Kim, T. Zimmermann, N. Nagappan, An empirical study of refactoring challenges and benefits at microsoft, IEEE Transactions on Software Engineering 40 (7) (2014) 633–649. 60. E. Murphy-Hill, A. P. Black, Breaking the barriers to successful refactoring: Observations and tools for extract method, in: Proceedings of the International Conference on Software Engineering, ACM, 2008, p. 421–430. 61. A. M. Eilertsen, G. C. Murphy, The usability (or not) of refactoring tools, in: Proceedings of the International Conference on Software Analysis, Evolution and Reengineering, IEEE, 2021, pp. 237–248. 62. K. Horikawa, H. Li, Y. Kashiwa, B. Adams, H. Iida, A. E. Hassan, Agentic refactoring: An empirical study of AI coding agents (2025). arXiv:2511.04824. 63. F. F. Xu, U. Alon, G. Neubig, V. J. Hellendoorn, A systematic evaluation of large language models of code, in: Proceedings of the International Symposium on Machine Programming, ACM, 2022, pp. 1–10. 64. S. Martinez, L. Xu, M. Elnaggar, E. A. AlOmar, Software refactoring research with large language models: A systematic literature review, Journal of Systems and Software 235 (2026) 112762. 65. J. White, S. Hays, Q. Fu, J. Spencer-Smith, D. C. Schmidt, ChatGPT prompt patterns for improving code quality, refactoring, requirements elicitation, and software design (2023). arXiv:2303.07839. 66. E. A. AlOmar, A. Venkatakrishnan, M. W. Mkaouer, C. D. Newman, A. Ouni, How to refactor this code? An exploratory study on developer-ChatGPT refactoring conversations, in: Proceedings of International Conference on Mining Software Repositories, ACM, 2024, pp. 202–206. 67. B. Liu, Y. Jiang, Y. Zhang, N. Niu, G. Li, H. Liu, An empirical study on the potential of LLMs in automated software refactoring (2024). arXiv:2411.04444. 68. K. Depalma, I. Miminoshvili, C. Henselder, K. Moss, E. A. AlOmar, Exploring ChatGPT’s code refactoring capabilities: An empirical study, Expert Systems with Applications 249 (2024) 123602. 69. D. Pomian, A. Bellur, M. Dilhara, Z. Kurbatova, E. Bogomolov, A. Sokolov, T. Bryksin, D. Dig, EM-assist: Safe automated extract method refactoring with LLMs, in: Companion Proceedings of Foundations of Software Engineering, ACM, 2024, pp. 582–586.
54
Rohit Gheyi et al.
70. J. Cordeiro, S. Noei, Y. Zou, An empirical study on the code refactoring capability of large language models (2024). arXiv:2411.02320. 71. A. Shirafuji, Y. Oda, J. Suzuki, M. Morishita, Y. Watanobe, Refactoring programs using large language models with few-shot examples, in: Proceedings of Asia-Pacific Software Engineering Conference, IEEE, 2023, pp. 151–160. 72. M. Dilhara, A. Bellur, T. Bryksin, D. Dig, Unprecedented code change automation: The fusion of LLMs and transformation by example, Proceedings of the ACM on Software Engineering 1 (FSE) (2024) 631–653. 73. Y. Zhang, Y. Li, G. Meredith, K. Zheng, X. Li, Move method refactoring recommendation based on deep learning and LLM-generated information, Information Sciences 697 (2025) 121753. 74. H. Liu, Y. Wang, Z. Wei, Y. Xu, J. Wang, H. Li, R. Ji, RefBERT: A two-stage pretrained framework for automatic rename refactoring, in: Proceedings of the International Symposium on Software Testing and Analysis, ACM, 2023, pp. 740–752. 75. Y. C. K. Piao, J. C. Paul, L. D. Silva, A. M. Dakhel, M. Hamdaqa, F. Khomh, Refactoring with LLMs: Bridging human expertise and machine understanding (2025). arXiv: 2510.03914. 76. K. Oueslati, M. Lamothe, F. Khomh, Refagent: A multi-agent LLM-based framework for automatic software refactoring (2026). arXiv:2511.03153. 77. C. Pacheco, S. K. Lahiri, M. D. Ernst, T. Ball, Feedback-directed random test generation, in: Proceedings of the International Conference on Software Engineering, IEEE, 2007, pp. 75–84. 78. Y. Zhang, Y. Li, L. Cui, D. Cai, L. Liu, T. Fu, X. Huang, E. Zhao, Y. Zhang, Y. Chen, L. Wang, A. T. Luu, W. Bi, F. Shi, S. Shi, Siren’s song in the AI ocean: A survey on hallucination in large language models (2023). arXiv:2309.01219.