Towards Evaluation of Implicit Software World Models in Coding LLMs Egor Bogomolov 1,∗ , Yaroslav Zharov 1,∗
arXiv:2606.27406v1 [cs.SE] 25 Jun 2026
1
JetBrains Research • [email protected], [email protected]
CRUXEval (Gu et al., 2024) predicts function inputs given outputs (and vice versa) on 800 short Python functions. REval (J. Chen et al., 2025) decomposes function execution into predicting code coverage, program state along the path, the next executed statement, and the final output. ThrowBench (Prenner and Robbes, 2025) predicts the runtime exception type. BigO(Bench) (Chambon et al., 2025) classifies asymptotic complexity. CodeMind (Liu, Y. Chen, and Jabbarvand, 2025) bundles three reasoning tasks on 1,450 short programs. RE2-Bench (Liu, Ghazanfari, et al., 2025) evaluates reasoning code on repository-scale. The reasoning benchmarks mostly operate on isolated, often synthetically created, Python functions, classes, or short programs, and mostly focus on return value or a derivative.
Software engineering, whether performed by humans or by AI agents, requires reasoning about how software behaves. We call the internal model that supports such reasoning the software world model, and view current code-execution benchmarks as covering one well-studied slice of it—control flow. In this paper, we take a step toward a broader evaluation by shifting the observable axis to execution resources: alongside test outcome and exception class, we predict peak memory, wall-clock time, and ranked profiler outputs at method and line granularity. We use SWE-bench Verified as the source of data to hold the test close to realworld software engineering tasks. All tested models, frontier ones included, show modest performance and brittle behaviour, suggesting a notable lack of understanding of how software is executed, as opposed to how its source code is written.
Reasoning about code execution in terms of the control flow is, however, only one slice of the broader skill software engineers exercise daily — predicting how a build will resolve dependencies, how a test suite will behave under a given change, how a service will respond at runtime, how concurrent code will interleave, how a patch will interact with the surrounding repository. We use the term software world model for this broader internal model of software-system behaviour, of which the control flow is the most studied facet. Following the established usage, we distinguish implicit world models— the capability spontaneously acquired by coding LLMs trained on general code corpora—from explicit world models, models trained for this purpose specifically, such as the recent CWM (FAIR et al., 2025).
Data: https://huggingface.co/collections/JetBrainsResearch/dl4c26-evaluation-of-software-worldmodels Code: https://github.com/JetBrainsResearch/cwm-execution-tracer
1 Introduction The most straightforward way of assessing coding LLMs is by how well they write code. Functionlevel benchmarks such as HumanEval (M. Chen et al., 2021) and MBPP (Austin et al., 2021) score whether a generated solution passes a heldout test suite; repository-level benchmarks such as SWE-Bench (Jimenez et al., 2024) and AiderPolyglot (Aider-AI, 2026) score whether a generated patch fixes a real bug or implements a feature; and repository-scale benchmarks such as Commit0 (Zhao et al., 2024) ask a model to reproduce an entire Python repository from its specification and unit-test suite. These evaluations measure the capabilities of model in code generation.
A complete evaluation of an implicit software world model would span build and dependency resolution, test and CI behaviour, runtime errors, deployment and runtime environments, concurrency, and the agentic, repo-level workflows that connect them — work well beyond a single paper. As a first step in this direction, we stay within the moststudied facet, code execution, and push it toward more realistic software contexts. We address two gaps in current code-execution evaluation. First, the function-scoped snippets used in most reasoning benchmarks do not match the complexity of practical software, where behaviour depends on a surrounding library implementation. Second, the return value is only one of many facets of how a piece of software actually executes. To bridge these
Another line of benchmarks evaluates coding LLMs on their understanding of program execution. Equal contribution; author order was determined by a coin toss. *
1
gaps, we (1) design a set of metrics that captures a wider slice of execution behaviour, (2) collect a dataset of library-level cases derived from SWEbench Verified, and (3) present results for a broad set of recent code-fluent LLMs, both proprietary and open-weight. We position this contribution as a template for further extensions of software world model evaluation, beyond code execution alone. The data package is available on Hugging Face1 , and the code on GitHub2 .
profiling metrics (time and memory). We record wall-clock time in milliseconds and memory in kilobytes. For the method profiler, we record fully qualified function names, and for the line profiler, we record lines in the format <file>:<line>. In all cases, we record the top 20 rows, ordered by the respective metric, and limit the scope of profiling to the repository.
2 Data
We target three qualitatively different prediction tasks and use a distinct metric family for each. Outcome prediction is evaluated by classification accuracy; resource prediction by linear calibration on a log scale, since wall time and memory span several orders of magnitude; and profiling as a ranking task. All metrics are computed per instance and aggregated over the dataset.
3 Metrics
We build the dataset from SWE-bench Verified (Jimenez et al., 2024), a curated collection of 500 real GitHub issues and verified gold patches across 12 Python repositories. We build samples from the pairs of tests that were failing and then got fixed with the gold patch. To make the task solvable for most models, we only retain the examples that require less than 500K characters of context. As SWE-bench Verified is dominated by samples from Django, we then downsample the data to 435 examples, while preserving diversity at the repository level. To collect the ground truth observables, for each instance we inject a custom sys.settrace/sys.monitoring-based tracer into the SWE-bench Docker container, and run the designated tests. We run the procedure before and after the patch, to catch different behaviours of the same test. The tracer records a number of observables detailed further in this section.
Test failure. We treat test failure prediction as a binary classification and report precision, recall, and F1. Linear calibration for wall time and peak memory. Both quantities span several orders of magnitude, so we evaluate on the log10 scale. We fit a linear model ŷ = a y ∗ + b where ŷ = log10 (predicted) and y ∗ = log10 (actual), and report slope a (calibration), intercept b (systematic bias), and Mean Absolute Error. We additionally replace zero predictions of models with 0.01 ms for time and 10KB for peak memory. These values are smaller than all the groundtruth numbers in our dataset, yet they allow us to compute log-scale metrics.
Test outcome. The tracer records whether each test passed or failed. We count a test as failed if it raises an AssertionError or another exception. For tests that failed, we additionally record the exception class name.
Profiler ranking. For each of the four ranked lists (functions × {time, memory} and lines × {time, memory}) we report two ranking metrics: recall and NDCG. Recall@k yields 1 for a sample if the actual top-method/-line is within top k predicted methods/lines and 0 otherwise. DCG@k is defined Pk as i=1 ri / log2 (i + 1), where ri is the measured time/memory attributed to the function or line predicted at rank i. We report NDCG@5—DCG@5 normalized by the optimal DCG@5—and Recall@5. To gauge the model’s ability to estimate the execution scope we additionally measure execution rate, which defines a portion of predicted methods or lines that were actually executed during the run.
Wall-clock time. We record the wall-clock time as the elapsed time between the start and finish of the function execution. To alleviate the noise from other measurements, we do this on a clean run without any additional tracing. Peak memory. Memory required for the test to run. We record it as a maximum of memory consumption recorded during the test execution, measured at the line level. Profiler. For each run, we record four profiling types, formed by the Cartesian product of two profiling scopes (method-level and line-level) and two
4 Experiment Setup
1 https://huggingface.co/
Models. We evaluate three Anthropic models via the Anthropic API: claude-haiku-4-5 (Anthropic, 2025), claude-sonnet-4-6 (Anthropic,
collections/JetBrains-Research/ dl4c26-evaluation-of-software-world-model 2 https://github.com/JetBrains-Research/ cwm-execution-tracer
2
Table 1. Test failure prediction, sorted by F1 ↓. Model
Prec. ↑ Rec. ↑
gpt-5.5 gpt-oss-120b claude-sonnet-4-6 gpt-5-mini claude-opus-4-7 Qwen3.5-397B gpt-5.2 gpt-5.4 CWM claude-haiku-4-5 Qwen3-235B Qwen3-30B
0.987 0.773 0.907 0.669 0.821 0.762 0.720 0.959 0.636 0.597 0.800 0.833
Table 2. Peak memory consumption prediction calibration (log10 scale). Ideal: slope = 1, bias = 0. Sorted by MAE ↓.
F1 ↑
0.735 0.842 0.495 0.604 0.390 0.545 0.395 0.497 0.345 0.486 0.320 0.451 0.270 0.393 0.235 0.378 0.210 0.316 0.185 0.282 0.080 0.145 0.025 0.049
Model
Slope ↑
Bias ↓
MAE ↓
gpt-5.4 gpt-5.2 Qwen3-30B gpt-5.5 claude-sonnet-4-6 gpt-oss-120b Qwen3.5-397B claude-opus-4-7 Qwen3-235B claude-haiku-4-5 CWM gpt-5-mini
0.738 0.661 0.312 0.802 0.760 0.733 0.660 0.713 0.665 0.640 0.186 0.405
+1.176 +1.883 +2.844 +1.616 +1.711 +1.765 +1.920 +2.097 +2.235 +2.469 +3.761 +4.063
0.567 0.631 0.723 0.729 0.756 0.789 0.828 0.833 0.869 0.911 1.004 1.188
Table 3. Wall-time prediction calibration (log10 scale). Ideal: slope = 1, bias = 0. Sorted by MAE ↓.
2026b), and claude-opus-4-7 (Anthropic, 2026a). Via the OpenAI API we evaluate gpt-5-mini (OpenAI, 2025a), gpt-5.2 (OpenAI, 2025c), gpt-5.4 (OpenAI, 2026a), and gpt-5.5 (OpenAI, 2026b). We run five openweight models locally: gpt-oss-120b (OpenAI, 2025b) from OpenAI; Qwen3.5-397B-A17B (Qwen Team, 2026), Qwen3-235B-A22B-Instruct, and Qwen3-30B-A3B-Instruct (Qwen Team, 2025) from Alibaba; and CWM (FAIR et al., 2025) by FAIR. Context. The user message consists of two blocks. The first block is a slice of the library containing all executed code. For each file, where at least one line was executed, we include module preambles (all content before the first class or function definition), followed by every function (or method body enclosed in class scaffolding) in which at least one line was executed. For classes with executed lines we also keep everything outside methods such as class definition and its fields. The remaining budget (if any) up to 500 K characters is then filled with non-executed functions and methods. The second block is the test file, windowed to at most 60 K characters centered on the target test function. The prompt template is shared in Appendix A.
Model
Slope ↑
Bias ↓
MAE ↓
gpt-5.4 gpt-5.5 gpt-5.2 gpt-oss-120b Qwen3-235B Qwen3.5-397B gpt-5-mini claude-opus-4-7 Qwen3-30B claude-sonnet-4-6 claude-haiku-4-5 CWM
0.800 0.895 0.763 0.630 0.405 0.626 0.442 0.779 0.305 0.848 0.663 0.244
+0.500 +0.652 +0.880 +0.917 +1.057 +1.154 +1.148 +1.221 +1.328 +1.382 +1.649 −0.640
0.578 0.686 0.893 0.944 1.021 1.078 1.095 1.182 1.295 1.350 1.625 1.816
5 Results Discussion Table 1 reports test-outcome classification; Tables 2 and 3 report resource-prediction calibration; Tables 4 and 5 report profiler-ranking quality. We note that all the models, including the frontier models available by API aren’t scoring high on the proposed tasks. For test outcome prediction, most models have F1 scores consistently low, largely due to very low recall. For both peak memory and walltime, model predictions are systematically biased, with every model having slope and bias significantly different from optimal (1, 0) values. For all types of profiling, the best recall@5 never reaches 0.2, indicating that models rarely identify the most consuming entity.
Task. The model is instructed to return a single JSON object with ten fields: reasoning (a 2–4 sentence explanation), outcome (passed, failed, or error), failure_line and exception_type (null when outcome is passed), peak_bytes and wall_ms (integer and float respectively), and four ranked lists of up to 20 entries each: hot_methods_time, hot_methods_alloc, hot_lines_time, and hot_lines_alloc. These fields correspond directly to the observables described in Section 2 and are scored by the metrics in Section 3.
For the test outcome prediction, we note that most models have low recall, indicating a strong bias towards tests passing. We attribute this to LLMs’ tendency to follow the natural-language semantics of code rather than its structure, as shown by Lam et al. (2025). Optimizing tests to elicit unbiased predictions from LLMs may be a promising research 3
Table 4. Memory-profiler ranking quality. exec: fraction of predicted names present in the execution trace. NDCG@5 and recall@5 assess ranking quality against ground-truth allocation profiles. Sorted by Method NDCG@5 ↓. Method
Line
Model
exec ↑
NDCG@5 ↑
rec@5 ↑
exec ↑
NDCG@5 ↑
rec@5 ↑
gpt-5.5 gpt-5-mini gpt-5.4 gpt-5.2 Qwen3.5-397B claude-opus-4-7 gpt-oss-120b claude-sonnet-4-6 Qwen3-30B Qwen3-235B claude-haiku-4-5 CWM
0.222 0.254 0.259 0.268 0.200 0.219 0.199 0.188 0.168 0.188 0.108 0.113
0.127 0.123 0.120 0.119 0.111 0.108 0.103 0.092 0.091 0.091 0.067 0.054
0.136 0.145 0.141 0.136 0.118 0.108 0.111 0.101 0.097 0.088 0.069 0.048
0.349 0.221 0.169 0.165 0.163 0.252 0.138 0.196 0.103 0.135 0.145 0.043
0.095 0.057 0.017 0.018 0.023 0.022 0.020 0.017 0.010 0.009 0.010 0.004
0.094 0.046 0.011 0.016 0.014 0.014 0.023 0.016 0.009 0.005 0.007 0.005
Table 5. Time-profiler ranking quality. exec: fraction of predicted names present in the execution trace. NDCG@5 and recall@5 assess ranking quality against ground-truth time profiles. Sorted by Method NDCG@5 ↓. Method
Line
Model
exec ↑
NDCG@5 ↑
rec@5 ↑
exec ↑
NDCG@5 ↑
rec@5 ↑
gpt-5.4 gpt-5.5 gpt-5.2 claude-opus-4-7 gpt-5-mini Qwen3.5-397B gpt-oss-120b claude-sonnet-4-6 Qwen3-30B Qwen3-235B claude-haiku-4-5 CWM
0.309 0.268 0.328 0.294 0.305 0.255 0.245 0.253 0.219 0.241 0.152 0.138
0.197 0.188 0.186 0.184 0.183 0.177 0.161 0.159 0.153 0.152 0.130 0.077
0.154 0.168 0.150 0.159 0.154 0.145 0.129 0.136 0.122 0.138 0.127 0.067
0.178 0.361 0.167 0.260 0.209 0.165 0.143 0.200 0.106 0.139 0.147 0.048
0.021 0.111 0.017 0.039 0.080 0.024 0.026 0.027 0.012 0.014 0.018 0.004
0.007 0.106 0.007 0.023 0.085 0.018 0.023 0.016 0.005 0.007 0.007 0.000
direction for Software Engineering.
are dominated by the line numbers and methods that weren’t actually executed. We additionally measured NDCG@5 of a set obtained by correct ordering of model predictions and noticed a significant boost (On average ×1.5 for method profiling and ×2 for line profiling), which indicates that hallucinating execution scope is not the sole problem — correct ranking is challenging for models.
For the peak memory consumption and time, in addition to the presence and strength of systematic errors, we note that the bias itself is universal. We observe unanimous slope compression (models giving predictions closer to the average) and bias towards overestimation (for most of the models the difference is at least an order of magnitude). The unique case of CWM can be explained by 247 cases where it predicts 0 ms execution time, motivating it in reasoning by inability to calculate it precisely. This conservative estimations make models less useful in giving predictions when it comes to load brought by particular tasks3 .
In general, we note that smaller and especially openweight models have a tendency to predict "round" numbers, with particularly peculiar numbers like "12345 ms" dominating the output of Qwen3-30B. This is especially well notable on the scatter plots of predicted and measured values shown in Appendix B. Surprisingly, CWM’s performance on code generation and reasoning, is not transferring to reasoning about software—while it produces coherent reasoning traces and outputs, it falls behind the Qwen3-30B (a model of the similar size) in most experiments. These findings reinforce our point about the need for a wider understanding of
Most often, the predictions for the profiling tasks 3 On an anecdotal observation, while we were running experiments for this paper, the runs were supervised by Sonnet 4.6, which systematically mispredicted the time needed to complete a run by always predicting a value around 20 minutes, whether in reality it took 2 hours or 7 minutes.
4
Software World Models beyond Code World Models.
Austin, Jacob et al. (2021). Program Synthesis with Large Language Models. arXiv: 2108.07732 [cs.PL]. url: https://arxiv.org/abs/2108. 07732. Chambon, Pierre et al. (2025). BigO(Bench) – Can LLMs Generate Code with Controlled Time and Space Complexity? arXiv: 2503.15242 [cs.CL]. url: https://arxiv.org/abs/2503.15242. Chen, Junkai et al. (2025). “Reasoning Runtime Behavior of a Program with LLM: How Far Are We?” In: Proceedings of the IEEE/ACM 47th International Conference on Software Engineering. IEEE Press, pp. 1869–1881. isbn: 9798331505691. url: https://doi.org/10.1109/ICSE55347. 2025.00012. Chen, Mark et al. (2021). Evaluating Large Language Models Trained on Code. arXiv: 2107 . 03374 [cs.LG]. url: https://arxiv.org/abs/ 2107.03374. FAIR et al. (2025). CWM: An Open-Weights LLM for Research on Code Generation with World Models. arXiv: 2510.02387 [cs.SE]. url: https://arxiv.org/abs/2510.02387. Gu, Alex et al. (21–27 Jul 2024). “CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution”. In: Proceedings of the 41st International Conference on Machine Learning. Ed. by Ruslan Salakhutdinov et al. Vol. 235. Proceedings of Machine Learning Research. PMLR, pp. 16568–16621. url: https://proceedings. mlr.press/v235/gu24c.html. Jimenez, Carlos E et al. (2024). “SWE-bench: Can Language Models Resolve Real-world Github Issues?” In: The Twelfth International Conference on Learning Representations. url: https : / / openreview.net/forum?id=VTF8yNQM66. Lam, Man Ho et al. (Oct. 2025). CodeCrash: Exposing LLM Fragility to Misleading Natural Language in Code Reasoning. arXiv:2504.14119. doi: 10 . 48550 / arXiv . 2504 . 14119. url: http : / / arxiv . org / abs / 2504 . 14119 (visited on 05/15/2026). Liu, Changshu, Yang Chen, and Reyhaneh Jabbarvand (2025). CodeMind: Evaluating Large Language Models for Code Reasoning. arXiv: 2402. 09664 [cs.SE]. url: https://arxiv.org/abs/ 2402.09664. Liu, Changshu, Alireza Ghazanfari, et al. (2025). “Evaluating Code Reasoning Abilities of Large Language Models Under Real-World Settings”. In: arXiv preprint arXiv:2512.14917. OpenAI (2025a). GPT-5 System Card. https:// openai.com/index/gpt-5-system-card/. Accessed 2026-04. – (2025b). gpt-oss-120b & gpt-oss-20b Model Card. https://openai.com/index/gpt-oss-modelcard/. Accessed 2026-05. – (2025c). Update to GPT-5 System Card: GPT5.2. https : / / openai . com / index / gpt - 5 -
6 Limitations We see three points where further work can make the results more diverse: data, context, and answer elicitation techniques. The dataset is derived from SWE-bench Verified and further limits its scope by filtering the tests and libraries where the context doesn’t fit in 500k characters. We leave generalisation to other languages, less-curated codebases, or broader test populations to future work. This paper only evaluates a single oracle-based context-collection strategy, thus establishing the upper boundary of the context collection performance. Future work should explore more realistic scaffolding. Finally, this paper does not explore broader ways to elicit better answers from LLMs, leaving such strategies as advanced prompting, multi-shot voting, or probing the latent space open for further exploration.
7 Conclusion Code execution is one facet of the broader software world modeling; this paper is intended as a first probe into the rest of that space. We extend execution evaluation to library-level cases from SWEbench Verified and to four tasks beyond the return value: test outcome, peak memory consumption, wall time, and ranked profiler outputs at method and line granularity. Across twelve models, including the trace-trained CWM and frontier models, performance is modest. We envision further extensions to the area of software world modeling in such tasks as build resolution, CI, deployment, concurrency, and agentic workflows; we release the data, prompts, and tracing harness to contribute to help advance the research in this area.
References Aider-AI (2026). aider: AI pair programming in your terminal. GitHub repository. https : / / github.com/Aider-AI/aider. Anthropic (2025). Claude Haiku 4.5 System Card. https://www.anthropic.com/claude-haiku4-5-system-card. Accessed 2026-05. – (2026a). Claude Opus 4.7 System Card. https: / / www . anthropic . com / claude - opus - 4 - 7 system-card. Accessed 2026-05. – (2026b). Claude Sonnet 4.6 System Card. https: //www.anthropic.com/claude- sonnet- 4- 6system-card. Accessed 2026-05.
5
system - card - update - gpt - 5 - 2/. Accessed 2026-05. OpenAI (2026a). GPT-5.4 Thinking System Card. https : / / openai . com / index / gpt - 5 - 4 thinking-system-card/. Accessed 2026-05. – (2026b). GPT-5.5 System Card. https : / / openai.com/index/gpt- 5- 5- system- card/. Accessed 2026-05. Prenner, Julian Aron and Romain Robbes (2025). ThrowBench: Benchmarking LLMs by Predicting Runtime Exceptions. arXiv: 2503 . 04241 [cs.SE]. url: https://arxiv.org/abs/2503. 04241. Qwen Team (2025). Qwen3: Think Deeper, Act Smarter. https://qwenlm.github.io/blog/ qwen3/. Accessed 2026-05. – (2026). Qwen3.5: Towards Native Multimodal Agents. https : / / www . alibabacloud . com / blog/602894. Accessed 2026-05. Zhao, Wenting et al. (2024). Commit0: Library Generation from Scratch. arXiv: 2412 . 01769 [cs.SE]. url: https://arxiv.org/abs/2412. 01769.
6
A Prompt Template Each sample is sent to the model as a two-message conversation. The system message is identical for all samples; the user message is constructed per sample as described in Section 4. Variable parts are shown in 〈angle brackets〉.
System message 1
You are an expert Python developer analyzing a software project. You will be shown a slice of the project’s source code and the full test file. You will predict a specific runtime property of running that test against THIS source code.
2 3
You do NOT execute the code. Reason about it by reading the source. Respond with ONLY a JSON object — no prose, no markdown fences.
User message 1
## Source files (slice of the project)
2 3 4 5
6
### ‘<rel/path/to/file.py>‘ ‘‘‘python <executed functions and module preamble, then non-executed functions up to 400 000 characters total across all files> ‘‘‘
7 8 9
### ‘<rel/path/to/another_file.py>‘ ...
10 11
## Test file: ‘<rel/path/to/test_file.py>‘
12 13
<test file content, up to 60 000 characters, windowed around the target test function>
14 15
## Task
16 17
Predict the runtime behavior of running the test <test_id> against the source code shown above. Return a single JSON object with all of the following keys:
18 19 20
21
22
23
24
25
26
reasoning — 2-4 sentences explaining your overall analysis outcome — "passed", "failed" (AssertionError), or "error" (non-assertion exception) failure_line — 1-based line in test file <test_file> where the failure occurs; null if outcome == passed exception_type — exception class name ("AssertionError", "TypeError", ...); null if outcome == passed peak_bytes — peak memory the test needs above its baseline, in bytes (int). i.e. how much additional RAM the system must have free for the test to run correctly. Concretely: the high-water mark of memory usage during test execution MINUS the memory already in use when the test started; taken as the larger of two complementary measurements: * Python-heap (tracemalloc) peak delta — catches lists / dicts / strings. * Process-RSS peak delta — catches large numpy / C-extension buffers that bypass pymalloc. Allocate-then-free patterns count their peak, NOT the cumulative bytes: a loop of 100 iterations each allocating + freeing 80 MB has peak_bytes ≈ 80 MB. Process-baseline (imports + state from prior tests in the session) is NOT counted.
7
27
28
29
30
31
wall_ms — total wall-clock time to run the test, in milliseconds (float). What a stopwatch would show from when the test framework (pytest / unittest) invokes the test method to when it returns — the test method body plus any setUp / fixtures / tearDown. Includes time spent in stdlib, numpy, database drivers, network I/O, etc. Does NOT include test-runner collection or reporting time outside the test invocation. hot_methods_time — up to 20 fully-qualified function names from this project, ranked by total time spent executing them during the test. Uses EXCLUSIVE wall time: time in each function’s own body, with time in nested in-project calls credited to the callee. Time in stdlib / numpy / third-party calls invoked from a method IS credited to it (we don’t trace into those frames). Hottest first. Synthetic frames (<lambda>, <listcomp>, <dictcomp>, <genexpr>) are eligible. hot_methods_alloc — up to 20 fully-qualified function names from this project, ranked by total bytes ALLOCATED (directly or indirectly via library calls) while executing during the test. Counts EVERY allocation event, including transient allocations that are freed before the function returns — a method that builds a 100 MB array, uses it, and discards it inside one call gets full credit. Combines Python-heap (tracemalloc) and process-RSS deltas to capture pymalloc AND C-extension buffers. Exclusive: allocations in stdlib / numpy / third-party calls invoked from this method ARE credited to it (not traced), but allocations inside in-project child methods are credited to those children. Hottest allocator first. hot_lines_time — up to 20 <rel_file_path>:<line_number> strings, ranked by total wall time spent executing that line during the test (summed across every execution of the line). Use the paths from the source slice; only lines in those files are eligible. Hottest first. hot_lines_alloc — up to 20 <rel_file_path>:<line_number> strings, ranked by total bytes allocated when that line executes (summed across every execution of the line). Captures the line’s own allocation activity plus any library / stdlib allocations made by code called from that line. Largest allocator first.
32 33
Use "reasoning" to think before committing to a value. Return fewer than 20 entries in any list if you expect fewer than many to be relevant. Wrong names or paths in the lists score 0 for that slot.
34 35 36 37 38 39 40 41 42 43 44 45 46 47
Return JSON: { "reasoning": "<2-4 sentences>", "outcome": "passed" | "failed" | "error", "failure_line": <int> | null, "exception_type": "<ClassName>" | null, "peak_bytes": <int>, "wall_ms": <float>, "hot_methods_time": ["fn1", "fn2", ...], "hot_methods_alloc": ["fn1", "fn2", ...], "hot_lines_time": ["path/to/file.py:42", ...], "hot_lines_alloc": ["path/to/file.py:42", ...] }
B Calibration Scatter Plots Figures 1 and 2 show per-model scatter plots of predicted versus ground-truth values for peak heap allocation and wall-clock time, respectively. Each panel reports the log-log linear fit (ŷ = s · x + b, slope s and bias b) together with the mean absolute log10 error (MAE). The dashed diagonal marks perfect calibration (y=x); the solid blue line is the fitted regression. Points are coloured by ground-truth test outcome (pass / fail). All models consistently overestimate both quantities (positive bias), with slope below 1 indicating compression of the dynamic range.
8
Peak Memory: Ground Truth vs. Predicted (log-log) GPT-5.5
Predicted log10 (bytes)
10 9
10 8
10 8
10 7
10 7
10 7
10 7
10 6
10 6
10 6
10 6
10 5
10 5
10 5
10 5
10 4
10 4
10 4
10 4
10 3
10 4
10 5
10 6
10 7
10 8
10 9
Claude Opus 4.7
10 3
10 3
10 5
10 6
10 7
10 8
10 3
10 9
Claude Sonnet 4.6
10 9
MAE=0.83 s=0.71 b=2.10
10 4
10 3
10 5
10 6
10 7
10 8
10 9
Claude Haiku 4.5
10 9
MAE=0.76 s=0.76 b=1.71
10 4
10 3
10 8
10 8
10 8
10 7
10 7
10 7
10 7
10 6
10 6
10 6
10 6
10 5
10 5
10 5
10 5
10 4
10 4
10 4
10 4
10 3
10 3
10 3
10 4
10 5
10 6
10 7
10 8
10 9
Qwen3.5-397B
10 9
10 3
10 5
10 6
10 7
10 8
10 9
Qwen3-235B-A22B
10 9
MAE=0.83 s=0.66 b=1.92
10 4
10 3
10 5
10 6
10 7
10 8
10 9
Qwen3-30B-A3B
10 9
MAE=0.87 s=0.66 b=2.23
10 4
10 3
10 8
10 8
10 8
10 7
10 7
10 7
10 7
10 6
10 6
10 6
10 6
10 5
10 5
10 5
10 5
10 4
10 4
10 4
10 4
10 3
10 4
10 5
10 6
10 7
10 8
Ground truth log10 (bytes)
10 9
10 3
10 3
10 4
10 5
10 6
10 7
10 8
Ground truth log10 (bytes) y = x (perfect)
10 3
10 9
Log-log fit
10 3
10 4
10 5
10 6
10 7
10 3
10 8
Ground truth log10 (bytes) Pass Fail
10 9
10 3
10 4
10 5
10 6
10 7
10 8
10 9
10 8
10 9
10 8
10 9
GPT-oss-120B MAE=0.79 s=0.73 b=1.76
10 3
10 4
10 5
10 6
10 7
CWM
10 9
MAE=0.72 s=0.31 b=2.84
10 8
10 3
MAE=1.19 s=0.41 b=4.06
10 9
MAE=0.91 s=0.64 b=2.47
10 8
10 3
GPT-5-mini
10 9
MAE=0.63 s=0.66 b=1.88
10 8
10 9
Predicted log10 (bytes)
GPT-5.2
10 9
MAE=0.57 s=0.74 b=1.18
10 8
10 3
Predicted log10 (bytes)
GPT-5.4
10 9
MAE=0.73 s=0.80 b=1.62
MAE=1.00 s=0.19 b=3.76
10 3
10 4
10 5
10 6
10 7
Ground truth log10 (bytes)
Figure 1. Peak heap allocation: predicted vs. ground-truth log10 (bytes) for all twelve models. Each panel’s legend reports MAE, fitted slope s, and bias b; dashed diagonal is y=x (perfect calibration). Points are coloured by test outcome (green = pass, red = fail).
9
Wall Time: Ground Truth vs. Predicted (log-log) 10 5
GPT-5.5 MAE=0.69 s=0.89 b=0.65
Predicted log10 (ms)
10 4
10 6 10 5
10 3
10 1
10 6 10 5
Claude Opus 4.7 MAE=1.18 s=0.78 b=1.22
10 6 10 5
10 4
10 4
10 3
10 3
10 −2
10 −1
10 0
10 1
10 2
10 3
10 4
Claude Sonnet 4.6
10 −3
10 5
MAE=1.35 s=0.85 b=1.38
10 1
10 1
10 0
10 0 10 −2 −3
10
10 5
10 −3 10 −2 10 −1 10 0 10 1 10 2 10 3 10 4 10 5 10 6
Qwen3.5-397B MAE=1.08 s=0.63 b=1.15
10 4
10
10 4
Claude Haiku 4.5
10 −1
10 0
10 1
10 2
10 3
10 4
GPT-oss-120B MAE=0.94 s=0.63 b=0.92
10 3 10 2 10 1 10 0
10 0
10 −1
10 −1 10 −3 10 −2 10 −1 10 0 10 1 10 2 10 3 10 4 10 5 10 6
Qwen3-235B-A22B
10 −2
10 4
MAE=1.02 s=0.41 b=1.06
10 3
10 3
10 2
10 2
10 1
10 1
10 0
10 0
10 −2 10 −2 10 −1 10 0
10 1
10 2
10 3
10 4
10 5
Qwen3-30B-A3B
10 −3
10 5
MAE=1.29 s=0.30 b=1.33
10 −3 10 −2 10 −1 10 0 10 1 10 2 10 3 10 4 10 5
CWM MAE=1.82 s=0.24 b=-0.64
10 4 10 3
10 2 10 1 10 0
10 2 10 1 10 0
10 −1
10 −1
10 −2 10
10 −2
10 4
10 3
−3
10 −2
10 5
MAE=1.62 s=0.66 b=1.65
10 3
10 1
−3
10 −3 10 −2 10 −1 10 0 10 1 10 2 10 3 10 4 10 5 10 6
10 4
10 2
10 −1
10 −1
10 −2
10 2
10 −2
10 0
10 −1
10 2
10 −1
10 1
10 0
10 −1 10 −3 10 −2 10 −1 10 0 10 1 10 2 10 3 10 4 10 5
10 2
10 1
10 −1
GPT-5-mini MAE=1.10 s=0.44 b=1.15
10 3
10 2
10 0
10 −2
10 4
10 3
10 1
10 0
GPT-5.2 MAE=0.89 s=0.76 b=0.88
10 4
10 2
10 2
10 −3
Predicted log10 (ms)
GPT-5.4 MAE=0.58 s=0.80 b=0.50
10 3
10 −2
Predicted log10 (ms)
10 4
10 −3 10 −2 10 −1 10 0 10 1 10 2 10 3 10 4 10 5
Ground truth log10 (ms)
10 −1
10 −1
−2
10 −2
10
10 −2
10 −1
10 0
10 1
10 2
10 3
Ground truth log10 (ms) y = x (perfect)
10 4
Log-log fit
10 −2 10 −2
10 −1
10 0
10 1
10 2
10 3
Ground truth log10 (ms) Pass Fail
10 4
10 −3
10 −3 10 −2 10 −1 10 0 10 1 10 2 10 3 10 4 10 5
Ground truth log10 (ms)
Figure 2. Wall-clock time: predicted vs. ground-truth log10 (ms) for all twelve models. Layout and colour coding identical to Figure 1. CWM is the only model whose bias is negative (b=−0.64), reflecting systematic under-prediction of execution time.
10