LLM S TAKING SHORTCUTS IN TEST GENERATION : A STUDY WITH SAP HANA AND L EVEL DB
arXiv:2604.14437v1 [cs.SE] 15 Apr 2026
THK-AI R ESEARCH R EPORT 2/2026 Vekil Bekmyradov
Noah C. Puetz
THK-AI Research Cluster, TH Köln
THK-AI Research Cluster, TH Köln
Thomas Bartz-Beielstein THK-AI Research Cluster, TH Köln [email protected]
April 17, 2026
A BSTRACT Large Language Models (LLMs) have achieved impressive results on public benchmarks, often leading to claims of advanced reasoning and understanding. However, recent research in cognitive science reveals that these models sometimes rely on shallow heuristics and memorization, taking shortcuts rather than demonstrating genuine cognitive abilities. This paper investigates LLM behavior in automated test generation for software, contrasting performance on an open-source system (LevelDB) with SAP HANA, one of the most widely deployed commercial database systems worldwide, whose proprietary codebase is guaranteed to be absent from training data. We combine cognitive evaluation principles, drawing on Mitchell’s mechanism-focused assessment methodology, with empirical software testing, employing mutation score and iterative compiler-feedback repair loops to assess both accuracy and underlying reasoning strategies. Results show that LLMs excel on familiar, open-source benchmarks but struggle with unseen, complex domains, often prioritizing compilability over semantic effectiveness. These findings provide independent software engineering evidence for the broader claim that current LLMs lack robust reasoning, and highlight the need for evaluation frameworks that penalize trivial shortcuts and reward true generalization.
1
Introduction
The rapid advancement of Artificial Intelligence (AI) has led to exceptional performance on public benchmarks, often driving claims that these systems possess broad capabilities like “reasoning” and “understanding”. However, recent research in cognitive science and software engineering indicates that this benchmark performance can overestimate their true real-world capabilities. AI algorithms often lack robust “world models” or abstract causal understanding, instead relying on “approximate retrieval” from their vast training data or exploiting unintended surface heuristics, taking “shortcuts” to arrive at correct answers for the wrong reasons. Mitchell and her team’s evaluation of LLMs on the Abstraction and Reasoning
Corpus (ARC-AGI) demonstrates that while models like OpenAI’s o3 can achieve high accuracy, they often fail to grasp the intended human-like “core knowledge” abstractions (such as “objectness” or geometry) (Moskvichev et al., 2023; Beger et al., 2026). Instead, they use less generalizable shortcuts, such as tracking shallow pixel patterns. Consequently, AI evaluations must look beyond simple accuracy and analyze the mechanisms and reasoning paths models take to achieve their results. This concern is not isolated to abstract reasoning tasks. A structurally identical dynamic has been documented in the broader machine learning literature under the term shortcut learning: the tendency of models to exploit statistical regularities that correlate with correct answers in train-
THK-AI R ESEARCH R EPORT 2/2026 - A PRIL 17, 2026
ing but fail to generalize under distribution shift (Geirhos trivial tests. Therefore the study incorporated mutation et al., 2020; Mitchell and Krakauer, 2023). score (MS) to assess the actual fault-detection effectiveness of the generated tests (Jia and Harman, 2011; DeThis phenomenon closely mirrors challenges in autoMillo et al., 1978). mated software testing. In test generation, existing evaluations mostly rely on open-source benchmarks (e.g., Hu- The experimental pipeline featured two scenarios. In Test manEval1 ) that are highly likely to be present in the Amplification, existing human-written test suites were arLLMs’ training corpora. This data contamination makes tificially reduced and the LLM was asked to regenerate it unclear whether high performance reflects genuine rea- the missing tests. In Whole-Suite Generation, all existing soning or mere memorization (Riddell et al., 2024; Chen tests were discarded and the LLM generated a complete et al., 2025). The problem is compounded by a second, test suite from source code alone. Two context configuradomain-specific shortcut: code coverage, the dominant tions were evaluated for Whole-Suite Generation: source proxy metric for test quality in the field, can be tests that code only, and source code augmented with the correexecute source lines without asserting any meaningful be- sponding dependency files. To handle the strict syntactical havior (Inozemtseva and Holmes, 2014). demands of complex C++ projects, the methodology included an iterative compiler-feedback repair loop. When a Bekmyradov (2026) addresses this by analyzing LLM generated test failed to compile, the error log was fed back behavior and testing strategies on SAP HANA, one of to the LLM for up to ten repair iterations (Bekmyradov, the most widely deployed commercial database systems 2026). This process allowed for the qualitative evaluation worldwide, compared to the open-source key-value store of the way the LLM reached its result. This is analogous LevelDB. Because SAP HANA’s codebase is proprito Mitchell’s method of prompting models to output natetary, it is guaranteed to be absent from the public preural language transformation rules to reveal their internal training corpora of all evaluated models, providing a zerologic. Tracking compilation success rate (CSR) across itcontamination evaluation environment. LevelDB, by conerations exposes whether models resolve compilation ertrast, is an open-source project whose codebase and test rors through genuine reasoning about the codebase or by suite are almost certainly present in those same corpora. progressively simplifying and hollowing out the generated Together, these perspectives reveal how LLMs optimize tests. for immediate constraints, whether solving grid puzzles or fixing compilation errors by relying on shallow heuristics rather than robust semantic understanding.
3
This paper is structured as follows: Section 2 describes the evaluation methodologies, combining Mitchell’s principles of mechanism-focused cognitive evaluation with the empirical software testing framework employed by Bekmyradov. Section 3 presents the empirical findings, highlighting the stark contrast in LLM performance between familiar and novel domains and documenting the three principal shortcut behaviors observed. Finally, Section 4 discusses the implications of these results for AI evaluation practice and outlines directions for future research.
2
The empirical findings reveal a profound gap between memorized retrieval and genuine generalization, heavily characterized by the models taking structural shortcuts. We organize the findings around following three distinct shortcut behaviors. 3.1
Memorization vs. Generalization
On the open-source LevelDB project, all four models demonstrated exceptional performance, achieving up to 100% mutation scores and surpassing the human-written baseline of 52.79%. In the Whole-Suite Generation scenario, every evaluated model achieved a perfect mutation score of 100%, despite having no access to the original test suite (Bekmyradov, 2026). This strongly suggests that the models recognized the repository from their training data and reproduced familiar test patterns rather than synthesizing novel tests through reasoning. Listings 1 and 2 show a representative case: the generated test is a nearverbatim copy of the original human-written test, with only minor structural differences. This pattern is consistent with approximate retrieval from training data rather than reasoning about the code (Bekmyradov, 2026), mirroring Lewis and Mitchell’s finding that LLM reasoning performance drops sharply when tasks are varied to be dissimilar from training data while testing the same abstract abilities (Lewis and Mitchell, 2024).
Methodology
To rigorously evaluate the cognitive and reasoning capabilities of LLMs, we combine Mitchell’s principles of cognitive evaluation with empirical software testing methodologies. Mitchell emphasizes that evaluations must test for robustness, avoid anthropomorphic biases, and investigate the mechanisms underlying performance rather than relying solely on accuracy (Mitchell and Krakauer, 2023; Mitchell et al., 2023). Applying this to software testing, Bekmyradov evaluated four models (GPT-5, Claude 4 Sonnet, Gemini 2.5 Pro, Qwen-3-Coder) using a multi-dimensional framework. Instead of relying exclusively on code coverage which can act as a flawed “shortcut” metric inflated by 1
Results
https://github.com/openai/human-eval
2
THK-AI R ESEARCH R EPORT 2/2026 - A PRIL 17, 2026
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
1
TEST_F(AddBoundaryInputsTest, TestNoBoundaryFiles) { FileMetaData* f1 = CreateFileMetaData(1, InternalKey("100 ", 2, kTypeValue), InternalKey(InternalKey(" 100", 1, kTypeValue))) ; FileMetaData* f2 = CreateFileMetaData(1, InternalKey("200 ", 2, kTypeValue), InternalKey(InternalKey(" 200", 1, kTypeValue))) ; FileMetaData* f3 = CreateFileMetaData(1, InternalKey("300 ", 2, kTypeValue), InternalKey(InternalKey(" 300", 1, kTypeValue))) ; level_files_.push_back(f3); level_files_.push_back(f2); level_files_.push_back(f1); compaction_files_.push_back(f2); compaction_files_.push_back(f3); AddBoundaryInputs(icmp_, level_files_, & compaction_files_); ASSERT_EQ(2, compaction_files_.size()); }
2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
TEST_F(AddBoundaryInputsTest, TestNoBoundaryFiles) { FileMetaData* f1 = CreateFileMetaData(1, InternalKey("100 ", 2, kTypeValue), InternalKey("199", 1, kTypeValue)); FileMetaData* f2 = CreateFileMetaData(2, InternalKey("200 ", 2, kTypeValue), InternalKey("299", 1, kTypeValue)); FileMetaData* f3 = CreateFileMetaData(3, InternalKey("300 ", 2, kTypeValue), InternalKey("399", 1, kTypeValue)); level_files_.push_back(f3); level_files_.push_back(f2); level_files_.push_back(f1); compaction_files_.push_back(f2); AddBoundaryInputs(icmp_, level_files_, & compaction_files_); ASSERT_EQ(1, compaction_files_.size()); ASSERT_EQ(f2, compaction_files_[0]); }
Listing 2: Generated test.
Listing 1: Original human-written test.
empty test bodies, or commenting out meaningful assertions.
Table 1 shows the Whole-Suite Generation results for SAP HANA and LevelDB, reporting line coverage (CovL ), branch coverage (CovB ), and mutation score (M S). Human (Full) refers to metrics obtained by running the complete original human-written test suite, while Human (Reduced) refers to the artificially reduced subset used as the generation baseline in the experiments. For complete results including Test Amplification scenarios, we refer the reader to (Bekmyradov, 2026).
On LevelDB, almost all models reached near-perfect compilation success within the first one or two iterations, with Gemini 2.5 Pro jumping from 0% to 70% in a single repair step. This rapid convergence is consistent with the memorization hypothesis: the errors encountered were shallow and easily resolved by the models’ prior knowledge of the codebase. On SAP HANA, repair curves climbed slowly across all ten iterations and plateaued well below 100% for most models, indicating that the errors were structurally unfamiliar and not resolvable through pattern recall alone.
In stark contrast, performance collapsed on the unseen SAP HANA codebase. For Test Amplification, the bestperforming model (GPT-5) achieved a mutation score of 39.54%. For Whole-Suite Generation with dependency context, the maximum mutation score across all modFigures 1 and 2 show the cumulative compilation success els was 25.14%, falling substantially below the humanrates across repair iterations for SAP HANA and Levwritten baseline (Bekmyradov, 2026). elDB, respectively. 3.2
Taking Shortcuts in Iterative Repair 3.3
The iterative compiler-feedback loop successfully increased compilation success rates by roughly 2–3×, with GPT-5 reaching up to 99% on SAP HANA. However, qualitative analysis of the mechanisms used to achieve compilability revealed that models increasingly took shortcuts as the repair loop progressed (Bekmyradov, 2026). To resolve build errors, the LLMs shifted their optimization toward strict compilability at the expense of semantic effectiveness, frequently generating trivial or
Explicit Context Reveals Absent Structural Knowledge
Just as Mitchell found that LLMs struggle with innate human priors like “objectness” (Mitchell et al., 2023; Beger et al., 2026), the models in this study demonstrated no inherent understanding of SAP HANA’s project structure, dependency graph, or namespace conventions (Bekmyradov, 2026). When provided with source code alone, models frequently hallucinated incorrect application pro3
THK-AI R ESEARCH R EPORT 2/2026 - A PRIL 17, 2026
SAP HANA - Cumulative Compilation Success Rates Whole Suite (Source+Header)
1.0
0.8 Cumulative success
Cumulative success
0.8 0.6 0.4 0.2 0.0
0
2
4
Fix k
6
8
0.4
0.0
10
Test Amplification (90% Reduced)
0
2
4
Fix k
6
8
10
Test Amplification (99% Reduced)
1.0 0.8 Cumulative success
0.8 Cumulative success
0.6
0.2
1.0
0.6 0.4 0.2 0.0
Whole Suite (Source)
1.0
0.6 0.4 0.2
0
2
4
Fix k
6
claude-4-sonnet
8
0.0
10 gemini-2.5-pro
0 gpt-5
2
4
Fix k
6
8
10
Qwen3-Coder
Figure 1: Cumulative compilation success rate over k = 10 repair iterations for SAP HANA (Bekmyradov, 2026).
4
THK-AI R ESEARCH R EPORT 2/2026 - A PRIL 17, 2026
LevelDB - Cumulative Compilation Success Rates Whole Suite (Source+Header)
1.0
0.8 Cumulative success
Cumulative success
0.8 0.6 0.4 0.2 0.0
Whole Suite (Source)
1.0
0.6 0.4 0.2
0
2
4
Fix k
6
8
0.0
10
0
2
4
Fix k
6
8
10
Test Amplification (99% Reduced)
1.0 Cumulative success
0.8 0.6 0.4 0.2 0.0
0
2
4
Fix k
6
claude-4-sonnet
8
10 gemini-2.5-pro
gpt-5
Qwen3-Coder
Figure 2: Cumulative compilation success rate over k = 10 repair iterations for LevelDB (Bekmyradov, 2026).
5
THK-AI R ESEARCH R EPORT 2/2026 - A PRIL 17, 2026
Table 1: Whole-Suite Generation results for SAP HANA and LevelDB. ’–’ denotes values that were not publicly disclosed (Bekmyradov, 2026). Project Methodology Model CovL (%) CovB (%) M S (%)
SAP HANA
LevelDB
Human (Full) Human (Reduced)
Human Human
– 66.71
– 35.33
– 30.41
Whole-Suite (Src)
GPT-5 Claude 4 Sonnet Qwen3-Coder Gemini 2.5 Pro
46.14 47.71 35.02 24.68
27.99 25.27 18.03 15.21
10.25 6.39 6.18 2.39
Whole-Suite (Src+H)
GPT-5 Claude 4 Sonnet Qwen3-Coder Gemini 2.5 Pro
60.87 62.11 45.02 37.90
34.26 31.31 21.72 22.04
25.14 17.49 9.20 10.60
Human (Full) Human (Reduced)
Human Human
73.78 54.87
57.08 37.59
52.79 37.32
Whole-Suite (Src)
GPT-5 Claude 4 Sonnet Qwen3-Coder Gemini 2.5 Pro
82.69 73.30 63.45 71.99
66.97 57.23 47.60 54.01
100.00 100.00 100.00 100.00
Whole-Suite (Src+H)
GPT-5 Claude 4 Sonnet Qwen3-Coder Gemini 2.5 Pro
78.20 76.37 72.46 73.15
60.90 60.26 55.25 57.68
98.24 99.89 100.00 100.00
gramming interface (API) calls, omitted required headers, The observation that LLMs generate trivial tests merely to satisfy the compiler perfectly illustrates Goodhart’s Law: and failed to instantiate objects correctly. when a measure (e.g., compatibility or code coverage) beProviding dependency files as auxiliary context procomes a target, it ceases to be a good measure (Goodduced consistent and substantial improvements across hart, 1984; Strathern, 1997). The models successfully all four models: line coverage increased by 10–15 perreached the target of producing compiling code, but encentage points, and mutation scores increased by up to tirely missed the underlying goal of generating tests capa150% (Bekmyradov, 2026). This improvement was not ble of detecting software faults. Similarly, in ARC-AGI, observed on LevelDB, where header context made little models may output the correct grid by relying on shaldifference, consistent with the interpretation that LevelDB low features like pixel counts, completely missing the inmodels already possessed implicit structural knowledge tended human-like abstraction. from training data. What makes the iterative repair finding particularly noFigures 3 and 4 show the effect of header context on table is that the shortcut-taking process is directly obcoverage and mutation score across all four models on servable over time. With each repair iteration on SAP SAP HANA. The improvement is consistent across every HANA, models progressively resolved compilation errors model, confirming that the knowledge gap is a systemby removing assertions or generating empty test bodies atic property of the unseen codebase rather than a modelrather than by reasoning about the underlying code (Bekspecific artifact (Bekmyradov, 2026). myradov, 2026). The metric being optimized is compilability which diverged from the actual goal of fault detection. 4 Discussion The combined findings from ARC-AGI evaluations and industrial unit test generation demonstrate that current LLMs rely heavily on data contamination and surfacelevel heuristics (Mitchell and Krakauer, 2023; Mitchell, 2019). When faced with novel, complex domains, whether generating test suites for SAP HANA or solving abstract visual reasoning puzzles (Lewis and Mitchell, 2024), models lack robust “world models” and instead attempt to brute-force solutions.
Implications. These results highlight the necessity of fundamentally changing how AI is evaluated. Researchers must look beyond saturated benchmarks and simple accuracy metrics, investigating the way models reach their answers to ensure they align with actual task competence. In the software testing domain, this means supplementing code coverage with mutation score as evaluation metric (Inozemtseva and Holmes, 2014; Bekmyradov, 2026). 6
THK-AI R ESEARCH R EPORT 2/2026 - A PRIL 17, 2026
Impact of Header Context on SAP HANA Coverage
60.87
62.11
80
22.04
15.21
21.72
18.03
20
24.68
35.02
31.31
25.27
37.9
45.02
47.71 27.99
40
34.26
46.14
Coverage (%)
60
0 GPT-5
Claude 4 Sonnet CovL (Src)
Qwen3-Coder
CovL (Src+H)
CovB (Src)
Gemini 2.5 Pro
CovB (Src+H)
Figure 3: Effect of header context on line and branch coverage for SAP HANA (Bekmyradov, 2026). Impact of Header Context on SAP HANA Mutation Score
9.2
2.39
6.18
6.39
10
10.6
17.49
25.14
20 10.25
Mutation Score (%)
30
0 GPT-5
Claude 4 Sonnet
Qwen3-Coder
MS (Src)
Gemini 2.5 Pro
MS (Src+H)
Figure 4: Effect of header context on mutation score for SAP HANA (Bekmyradov, 2026). Coverage measures execution; mutation score measures comprehension. A model that achieves high coverage through trivial tests is taking the same kind of shortcut as a model that solves ARC-AGI through pixel counting where the output looks correct, but the mechanism is not.
loop: instead of feeding back only compiler errors, surviving mutants could be fed back to the model as explicit generation targets (Bekmyradov, 2026). Preliminary evidence from related work suggests that mutation-guided generation produces tests with substantially higher faultdetection effectiveness (Harman et al., 2025).
Future work should integrate rigorous semantic feedback mechanisms to penalize trivial shortcuts and reward gen- Another direction concerns context and knowledge inuine fault-detection capability. One concrete direction is jection. The performance improvements from providto incorporate mutation analysis directly into the repair ing dependency files suggest that static context injec7
THK-AI R ESEARCH R EPORT 2/2026 - A PRIL 17, 2026
make shortcut behavior directly observable in code generation tasks.
tion alone is insufficient for complex industrial codebases. More promising are agentic approaches, where LLMs are equipped with tools to autonomously retrieve relevant project context, navigate dependency graphs, and iteratively interact with the build environment (Jimenez et al., 2024). Such agentic pipelines could dynamically assemble the minimal context required for each generation task, without requiring project-specific fine-tuning, and are a natural next step for test generation in large proprietary systems.
These findings carry consequences that extend well beyond software testing. Marcus (Marcus, 2018) has argued that deep learning suffers from systematic limitations, including brittleness, superficial generalization, and an inability to distinguish correlation from causation, that no amount of scaling is likely to resolve. The shortcut behaviors documented here provide independent, empirical confirmation of precisely these limitations in a concrete engineering setting. When LLMs are deployed in domains where their training data creates an illusion of competence, whether in medical diagnosis, legal reasoning, scientific analysis, or public decision-making, the same mechanisms apply: surfacelevel pattern matching may produce plausible outputs that lack the causal understanding required for reliable results (Bartz-Beielstein, 2026). The gap between memorized competence on LevelDB and genuine reasoning failure on SAP HANA is not merely a software engineering concern; it is a microcosm of the broader risk that Mitchell (Mitchell, 2019), Marcus (Marcus, 2018), and Bartz-Beielstein (Bartz-Beielstein, 2026) have identified from complementary perspectives: society increasingly relies on systems whose fluent outputs mask fundamental deficits in understanding.
Limitations. Several limitations of the underlying study should be acknowledged. The SAP HANA evaluation was restricted to a single representative component due to the computational cost of mutation testing at industrial scale (Bekmyradov, 2026). Each experimental configuration was repeated only twice, which is insufficient for formal statistical significance testing. Additionally, LevelDB is considerably smaller and less architecturally complex than SAP HANA: 21,207 lines of code versus 70,678 for the evaluated component in SAP HANA, which means the two systems differ not only in training data exposure but also in structural complexity, and this distinction should be kept in mind when interpreting the performance gap (Bekmyradov, 2026). Finally, the mutation operator set was restricted to arithmetic, boundary, comparison, and logical operators. A broader operator set might reveal We hope these results open a productive dialogue between additional weaknesses in the generated tests. cognitive science and software engineering research comHowever, we assume these limitations do not affect the munities on the shared challenge of building evaluations core argument: the performance contrast between Lev- that are resistant to the shortcuts models have proven reelDB and SAP HANA is large enough to be robust to markably effective at finding. moderate experimental variance, but they do constrain the generalizability of the specific quantitative results. References
5
Thomas Bartz-Beielstein. Why we need an AI-resilient society, 2026. URL https://arxiv.org/abs/ 1912.08786v2. v2, substantially revised April 2026.
Conclusions
This paper examined LLM shortcut behavior through the Claas Beger, Ryan Yi, Shuhao Fu, Kaleda Denton, Arseny Moskvichev, Sarah W. Tsai, Sivasankaran Rajalens of automated software test generation. By contrastmanickam, and Melanie Mitchell. Do AI models pering performance on a familiar open-source system (Levform human-like abstract reasoning across modalities?, elDB) with a proprietary industrial codebase guaranteed 2026. URL https://arxiv.org/abs/2510.02125. to be absent from training data (SAP HANA), we documented three concrete shortcut behaviors: performance Vekilmuhammet Bekmyradov. Evaluating the Effectivecollapse under distribution shift consistent with memoness of LLM-Generated Unit Tests in the Context of rization rather than generalization, progressive degradaIndustrial DBMS. Technical Report 1/2026, Fakultät tion of test quality under iterative repair as models opti10 / Institut für Data Science, Engineering, and Anmize for compilability over fault detection, and a nearalytics, 2026. URL https://doi.org/10.57684/ complete absence of implicit structural knowledge on unCOS-1441. seen codebases. Wentao Chen, Lizhe Zhang, Li Zhong, Letian Peng, ZiTogether, these findings provide empirical software enlong Wang, and Jingbo Shang. Memorize or genergineering evidence for the broader claim advanced by alize? evaluating LLM code generation with evolved Mitchell (Mitchell and Krakauer, 2023; Mitchell, 2019; questions. CoRR, abs/2503.02296, 2025. doi: 10. Beger et al., 2026) that current LLMs exploit surface 48550/ARXIV.2503.02296. URL https://doi.org/ heuristics rather than demonstrating robust reasoning, and 10.48550/arXiv.2503.02296. that evaluation frameworks must examine mechanisms, not just outcomes. Mutation score, combined with an iter- Richard A. DeMillo, Richard J. Lipton, and Fred G. Sayward. Hints on test data selection: Help for the ative repair loop, offers a practical and principled way to 8
THK-AI R ESEARCH R EPORT 2/2026 - A PRIL 17, 2026
practicing programmer. Computer, 11(4):34–41, 1978. Gary Marcus. Deep learning: A critical appraisal, 2018. URL https://arxiv.org/abs/1801.00631. doi: 10.1109/C-M.1978.218136. URL https://doi. org/10.1109/C-M.1978.218136. Melanie Mitchell. Artificial intelligence hits the barrier of meaning. Information, 10(2):51, 2019. doi: Robert Geirhos, Jörn-Henrik Jacobsen, Claudio 10.3390/info10020051. URL https://doi.org/10. Michaelis, Richard S. Zemel, Wieland Bren3390/info10020051. del, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks. Melanie Mitchell and David C. Krakauer. The debate over Nat. Mach. Intell., 2(11):665–673, 2020. doi: understanding in AI’s large language models. Proc. 10.1038/S42256-020-00257-Z. URL https: Natl. Acad. Sci. U.S.A., 120(13):e2215907120, 2023. //doi.org/10.1038/s42256-020-00257-z. doi: 10.1073/pnas.2215907120. URL https://doi. org/10.1073/pnas.2215907120. Charles A. E. Goodhart. Problems of monetary management: The U.K. experience. In Monetary Theory and Melanie Mitchell, Alessandro B. Palmarini, and Arseny Practice, pages 91–121. Macmillan, London, 1984. Moskvichev. Comparing humans, GPT-4, and GPT-4V doi: 10.1007/978-1-349-17295-5_4. URL https:// on abstraction and reasoning tasks, 2023. URL https: doi.org/10.1007/978-1-349-17295-5_4. Origi//arxiv.org/abs/2311.09247. nally published in: Papers in Monetary Economics, vol. I, Reserve Bank of Australia, 1975. Arsenii Kirillovich Moskvichev, Victor Vikram Odouard, and Melanie Mitchell. The conceptARC benchmark: Mark Harman, Jillian Ritchey, Inna Harper, Shubho SenEvaluating understanding and generalization in the gupta, Ke Mao, Abhishek Gulati, Christopher Foster, ARC domain. Transactions on Machine Learning Reand Hervé Robert. Mutation-guided llm-based test gensearch, 2023. ISSN 2835-8856. URL https:// eration at meta. In Leonardo Montecchi, Jingyue Li, openreview.net/forum?id=8ykyGbtt2q. Denys Poshyvanyk, and Dongmei Zhang, editors, Proceedings of the 33rd ACM International Conference on Martin Riddell, Ansong Ni, and Arman Cohan. Quantifying contamination in evaluating code generation cathe Foundations of Software Engineering, FSE Compabilities of language models. In Lun-Wei Ku, Andre panion 2025, Clarion Hotel Trondheim, Trondheim, Martins, and Vivek Srikumar, editors, Proceedings of Norway, June 23-28, 2025, pages 180–191. ACM, the 62nd Annual Meeting of the Association for Com2025. doi: 10.1145/3696630.3728544. URL https: putational Linguistics (Volume 1: Long Papers), ACL //doi.org/10.1145/3696630.3728544. 2024, Bangkok, Thailand, August 11-16, 2024, pages 14116–14137. Association for Computational LinguisLaura Inozemtseva and Reid Holmes. Coverage is not tics, 2024. doi: 10.18653/V1/2024.ACL-LONG. strongly correlated with test suite effectiveness. In 761. URL https://doi.org/10.18653/v1/2024. Pankaj Jalote, Lionel C. Briand, and André van der acl-long.761. Hoek, editors, 36th International Conference on Software Engineering, ICSE ’14, Hyderabad, India - May 31 - June 07, 2014, pages 435–445. ACM, 2014. doi: Marilyn Strathern. ‘improving ratings’: audit in the British University system. European Review, 5(3):305– 10.1145/2568225.2568271. URL https://doi.org/ 321, 1997. 10.1145/2568225.2568271. Yue Jia and Mark Harman. An analysis and survey of the development of mutation testing. IEEE Trans. Software Eng., 37(5):649–678, 2011. doi: 10.1109/TSE.2010. 62. URL https://doi.org/10.1109/TSE.2010. 62. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id= VTF8yNQM66. Martha Lewis and Melanie Mitchell. Evaluating the robustness of analogical reasoning in large language models, 2024. URL https://arxiv.org/abs/ 2411.14215. 9