CppPerf: An Automated Pipeline and Dataset for Performance-Improving C++ Commits Tommy Ho
Khashayar Etemadi
Zhendong Su
[email protected] ETH Zurich Switzerland
[email protected] ETH Zurich Switzerland
[email protected] ETH Zurich Switzerland
arXiv:2605.10890v1 [cs.SE] 11 May 2026
Abstract Recent progress in automated repair of performance bugs demands realistic, executable benchmarks. However, existing C++ performance benchmarks are largely built from competitive programming submissions, and recent real-world benchmarks predominantly target Python and .NET. To fill this gap, we present CppPerf-Mine, a configurable pipeline that mines execution-time-improving patches from open-source C++ repositories on GitHub by combining structural commit filtering, an LLM-based commit classifier, and a containerized build & test stage that produces fully reproducible Docker images for each patch. Using CppPerf-Mine, we build CppPerf-DB, a benchmark comprising 347 manually verified patches from 42 mature C++ repositories, 39% of which are multi-file, enabling the evaluation of repository-level repair tools. In our preliminary study, OpenHands correctly fixes only 13.5% of the patches in CppPerfDB, confirming that real-world C++ performance repair remains an open challenge. CppPerf-Mine and CppPerf-DB are open-source and publicly available at: https://doi.org/10.5281/zenodo.20097425. In addition, a demonstration video is available at: https://www.youtube.com/watch?v=nixlupIgSdM.
builds and passes tests, and outputs the executable execution-timeimproving commits as Docker images that provide full reproducibility. With this approach, CppPerf-Mine can build extensible benchmarks by collecting additional and newer commits. Using CppPerf-Mine, we build CppPerf-DB, a benchmark with 347 real-world executable C++ patches that fix execution-time bugs and are collected from 42 different repositories. The patches in CppPerf-DB are collected from repositories that have between 306 and 28,718 stars, and 39% (136/347) of them modify multiple files, making CppPerf-DB diverse and suitable for evaluating repositorylevel patch generation. Our preliminary study shows that OpenHands, a state-of-the-art agentic tool, fixes only 13.5% of the bugs in CppPerf-DB, indicating the challenging nature of automated repair of real-world performance bugs. In summary, we make the following contributions: • CppPerf-Mine, a configurable tool for building benchmarks of executable C++ performance bug-fixing patches. • CppPerf-DB, a benchmark with 347 real-world C++ execution time bug-fixing patches collected from 42 repositories that span 6 years of development history. • An evaluation of the diversity of patches in CppPerf-DB and the precision of CppPerf-Mine, and a preliminary study using CppPerf-DB to assess the effectiveness of state-of-theart agentic tools in fixing C++ performance bugs.
Keywords Execution Time Optimization, Benchmark, C++, Program Repair
2 1
Introduction
After years of advancements in the field of automated program repair (APR), which has been mainly focused on fixing functional bugs [10, 11], recent research has started to build new techniques for automated repair of performance bugs [3, 5, 15]. One of the challenges in this area is the scarcity of benchmarks of real-world performance bug-fixing patches. The benchmarks of performance patches that are most commonly used for evaluating APR tools contain online competition programs [2, 9, 14], introductory-level programs [12], and generated programs [8]. Recently, several benchmarks have been proposed that are collected from real-world projects in Python [6, 13, 15, 16], C# [4], Java [19], and C++ [14]. However, they are typically collected from a small number of projects and provide no configurable tool for extending them according to user needs [13, 19]. In this paper, we introduce CppPerf-Mine, a novel and configurable tool for building benchmarks of executable performance bugfixing patches in C++ repositories. CppPerf-Mine searches the commit history of GitHub C++ repositories, filters them based on userprovided configurations, employs an LLM-based technique to identify commits that improve execution time, checks that the project
CppPerf-Mine: The Pipeline
CppPerf-Mine takes user-defined settings as input and uses them to find commits that fix execution time bugs in C++ repositories. For each such commit, it produces a containerized environment that builds and runs tests on the corresponding patch. Figure 1 shows an overview of the CppPerf-Mine workflow, which has four main components: (1) collecting repositories, (2) collecting commits, (3) LLM-based commit classification, and (4) containerized building and testing. All components consider userdefined settings to select and build containerized execution-timeimproving patches that meet user expectations.
2.1
Collecting Repositories and Commits
CppPerf-Mine starts by crawling GitHub to select repositories that meet five requirements. First, the repository should have at least the minimum number of stars specified by the user to ensure that it is a solid and popular project. Second, C++ should be the primary language of the repository. Third, the project should have a CMakeLists.txt file at its root and use CMake to build, allowing us to build and test selected projects with a unified technique. Fourth, the repository must define executable tests within its CMake configuration, which are used to check that the patch does not break
Conference’17, July 2017, Washington, DC, USA
Tommy Ho, Khashayar Etemadi, and Zhendong Su
Containerized Execution Time Improving Patches
User-defined Settings
1: Collecting Repositories
Selected Repos
2: Collecting Commits
Selected Commits
3: LLM-based Commit Classification
Execution Time Improving Commits
4: Containerized Building & Testing
Figure 1: An overview of the CppPerf-Mine workflow.
the functionality of the program. Fifth, we run the tests on the latest version of the project to confirm that they build and pass successfully. After selecting the repositories that meet the requirements, CppPerfMine examines their commit histories and filters commits according to three structural criteria. First, the commit should be from the userdefined period of time. Second, the commit should not modify more than the maximum number of files specified by the user. Finally, the commit should only change C++ source code files; CppPerf-Mine excludes commits that change test files to ensure that a fixed set of tests can serve as a specification of program functionality in both the original and patched versions. Commits that meet these criteria are selected and passed on to the next step.
2.2
LLM-based Commit Classification
CppPerf-Mine uses a two-phase LLM-based technique to assess whether a selected commit focuses on improving execution time. In the first phase, two LLMs are presented with the commit message and, if available, the description of the linked issue, and asked to determine if the commit is focused on execution time improvement by answering “Yes”, “No”, or “Maybe”. If both LLMs agree on “Yes” or “No”, we take that label as the final classification. This ensures that we classify a commit based on its commit message and linked issue description only when two LLMs are highly confident. If the LLMs do not agree, we proceed to the second phase, in which a single LLM is presented with the commit message, the issue description, and the code diff, and asked to respond with either “Yes” or “No”. The code diff provides detailed information about the modifications beyond the high-level commit message and issue description, but it can be large and costly to process. Therefore, we include it only for the limited number of commits that reach the second phase.
2.3
code, and execution logs. This image is the output of CppPerf-Mine as a “containerized execution time improving patch”.
2.4
Implementation
CppPerf-Mine is programmed in Python. By default, it collects commits from 2020 to 2025 in repositories with at least 300 stars and changing at most 20 files. The LLM-based commit classifier uses qwen2.5:7b and qwen3:8b, with the latter also used in the second phase. These open-source models are chosen to enable local deployment. In the containerized building and testing step, the tests are executed 31 times by default; the first execution is treated as a warm-up and discarded. Following the previous work [6], a test is considered to demonstrate a statistically significant execution time improvement if it shows at least a 5% improvement with a p-value below 0.05 in Mann-Whitney test. CppPerf-Mine and CppPerf-DB are publicly available on both Zenodo and GitHub [7].
2.5
Usage
Both CppPerf-Mine and CppPerf-DB are used through a commandline interface provided by a main.py script. To build a benchmark, users supply their desired filters and configurations as arguments, and CppPerf-Mine produces a set of Docker images together with a JSON file per image describing the patch source and execution data. To evaluate a patch generation tool against CppPerf-DB, users pull the Docker image of a selected patch and invoke main.py with the patch id and their generated patch; the script runs the tests and saves the results to a JSON file. Collected patches and evaluation results can be inspected via the same CLI.
Containerized Building & Testing
In the final step, CppPerf-Mine builds and runs the tests for each commit classified as execution-time-improving in a dockerized environment. It first fetches a base Docker image with an appropriate version of GCC and clones the original and patched versions of the repository into separate directories. Then it applies various heuristics to install the libraries required to build the project, falling back to an LLM-based iterative technique to detect and install missing libraries if the build fails. Finally, CppPerf-Mine builds and runs the tests on both versions a user-specified number of times to ensure that the build & test process is consistently successful. If both versions build and pass all runs, the result is saved as a Docker image containing the required compiler, libraries, both versions of the
3
Experiments
To evaluate CppPerf-Mine, we begin by running it with its default configuration to build CppPerf-DB, an extensible benchmark of real-world execution time bug-fixing patches in C++ projects. Using CppPerf-Mine and the resulting CppPerf-DB, we then answer three research questions: RQ1 (CppPerf-DB characteristics): What are the characteristics of the patches in CppPerf-DB in terms of scale, diversity, and scope? RQ2 (CppPerf-DB in practice): To what extent can off-the-shelf advanced coding agents fix execution time bugs in CppPerf-DB? RQ3 (CppPerf-Mine precision): How precise is CppPerf-Mine for building benchmarks of C++ execution time bug-fixing patches?
CppPerf: An Automated Pipeline and Dataset for Performance-Improving C++ Commits
Metric Min Q1 Median Q3 Max Commits 126 2,032 3,468 5,829 14,428 Stars 306 1,307 1,849 5,493 28,718 Table 1: The number of commits and stars across the repositories from which the 347 patches in CppPerf-DB are collected.
Conference’17, July 2017, Washington, DC, USA
Table 3: Confusion matrix of CppPerf-Mine’s LLM-based classifier on a manually labeled sample of 405 commits. It indicates a precision of 86.67%.
Table 2: OpenHands results on CppPerf-DB.
Manual Label
Predicted Positive
Predicted Negative
Total
Positive Negative
13 (TP) 2 (FP)
18 (FN) 372 (TN)
31 374
15
390
405
Total Result Correct Patch Correct Location Incorrect
3.1
All (n=347)
Single-file (n=211)
Multi-file (n=136)
47 (13.5%) 101 (29.1%) 199 (57.3%)
37 (17.5%) 57 (27.0%) 117 (55.5%)
10 (7.4%) 44 (32.4%) 82 (60.3%)
RQ1 Experiment (CppPerf-DB characteristics)
In total, CppPerf-Mine scans 65,942 commits, of which 25,715 meet the structural criteria and are passed to the LLM-based classifier. The classifier identifies 1,120 commits as focused on execution time bugs, and CppPerf-Mine successfully builds and runs tests for 493 of them in a fully reproducible containerized environment. We then manually verify these executable patches: a patch is retained if (a) the commit message or linked issue explicitly mentions speed improvement, or (b) the code diff contains a recognizable optimization pattern (e.g., algorithmic improvement, caching, reduced allocations). This step excludes 146 patches that are not clearly focused on execution time improvement and might add noise to the evaluation of patch generation tools. Consequently, CppPerf-DB consists of 347 manually verified, real-world execution time bug-fixing patches that build and pass all tests in a reproducible environment. For 35 of these patches, an existing test case demonstrates a significant improvement in execution time and can be used for automated and objective assessment of generated patches. The 347 patches in CppPerf-DB are collected from 42 repositories. As shown in Table 1, these repositories have between 126 and 14,428 commits (median 3,468) and between 306 and 28,718 stars (median 1,849), indicating a diverse mix of well-established and mature projects. This places CppPerf-DB well beyond the toy programs considered in widely-used benchmarks [14]. Furthermore, 39% (136/347) of the patches are multi-file, making CppPerf-DB suitable for evaluating repository-level patch generation tools [17, 18, 20], and patches modify between 1 and 6,069 lines of code (median 28), spanning a wide range of scopes. Answer to RQ1: CppPerf-DB contains 347 containerized execution time improvement patches from 42 repositories with a median of 1,849 stars and 3,468 commits, indicating their maturity and popularity. 39% of the patches modify multiple files, enabling the evaluation of repository-level patch generation tools.
3.2
RQ2 Experiment (CppPerf-DB in practice)
We perform a preliminary study to assess the effectiveness of OpenHands, a state-of-the-art patch generation tool, on CppPerf-DB.
For each patch, we give OpenHands the original code, the commit message, the issue description (if available), and the list of modified files of the ground-truth patch, and ask it to generate a patch that improves execution time. We provide the list of modified files to constrain the search space and reflect realistic development scenarios, and configure OpenHands to use gpt-5-mini. We then manually compare each generated patch with its ground-truth: a patch is correct patch if it performs the same semantic modification, correct location if it modifies the same code but with different semantics, and incorrect otherwise. Table 2 shows the results. OpenHands generates a semantically equivalent patch for 13.5% (47/347) of the bugs in CppPerf-DB, including 17.5% of single-file and 7.4% of multi-file bugs. This indicates that CppPerf-DB contains challenging bugs, especially among its multi-file ones, that are suitable for evaluating tools aimed at advancing the state of the art. The table also shows that in 57.3% of the cases, the generated patch does not even correctly locate the bug, reaffirming the importance of accurate fault localization in fixing performance bugs. Answer to RQ2: OpenHands generates a patch semantically matching the ground-truth for 13.5% (47/347) of performance bugs in CppPerfDB, indicating that CppPerf-DB contains many challenging bugs suitable for evaluating advanced patch generation tools.
3.3
RQ3 Experiment (CppPerf-Mine precision)
To assess whether the LLM-based classifier of CppPerf-Mine correctly labels commits, we randomly select and manually analyze 405 commits passed to the classifier. Table 3 shows the resulting confusion matrix. CppPerf-Mine predicts 15 commits as positive, 13 of which are manually verified to focus on improving execution time, yielding a precision of 86.67%. The classifier’s recall is 41.94%, showing that a notable number of execution time improvement patches are discarded. While higher recall is desirable, we consider precision more important for a patch collection tool: a patch included in the benchmark should be highly likely to be useful for evaluation. We therefore intentionally design a conservative classifier that favors precision, even at the cost of excluding some execution-time-improving patches. Answer to RQ3: Manual analysis of 405 randomly selected commits shows that the LLM-based classifier of CppPerf-Mine has a precision of 86.67%, making it a reliable tool for finding execution-timeimprovement patches.
Conference’17, July 2017, Washington, DC, USA
4
Threats to Validity
External Validity: CppPerf-Mine considers C++ projects that use CMake and relies on heuristics and LLM-based techniques to build them, excluding projects that use other build tools or that cannot be built with our approach. Extending CppPerf-Mine to handle additional build systems and projects dismissed by the current version is an opportunity for future work. Construct Validity: CppPerf-Mine relies on test execution times across multiple runs to decide if a test demonstrates an improvement. As reported in Section 3.1, only 35 of the 347 patches in CppPerf-DB have an existing test that detects execution time improvement, reflecting a known limitation: human-labeled performance patches may target code paths not exercised by existing tests, or improvements may be too small to surface above measurement noise. To mitigate this, CppPerf-Mine employs the configurable statistical testing described in Section 2.3, which increases confidence in the tests that demonstrate improvement.
5
Related Work
Performance Benchmarks. Traditional APR benchmarks such as Defects4J [11] and SWE-Bench [10] target functional defects rather than performance issues. Early performance benchmarks are built from competitive programming submissions, including PIE [14], EffiBench [9], and Mercury [2], but consist of small, self-contained programs that differ substantially from real-world software [1]. More recent benchmarks mine performance patches from real repositories: SWE-Perf [6], SWE-fficiency [13], GSO [16], PeacExec [15], and PerfBench [4]. Despite C++ being a dominant language for performance-critical software, the only widely-used C++ performance benchmark remains PIE [14], built from online competition submissions. CppPerf-DB addresses this gap with 347 manually verified, containerized patches mined from 42 real-world C++ repositories, and CppPerf-Mine enables extending the benchmark with newly mined commits. Automated Performance Repair. Function-level methods include RAPGen [5], SBLLM [3], and EFFI-EARNER [8], while PEACE [15] targets project-level optimization. General-purpose agentic frameworks, e.g., OpenHands [17], SWE-Agent [18], AutoCodeRover [20], have also been evaluated on performance tasks, with reported success rates remaining low (e.g., 4.9% on GSO and 3% on PerfBench) [4, 16]. Our preliminary study of OpenHands on CppPerfDB reports 13.5%, consistent with these results and reaffirming that real-world C++ performance repair remains an open challenge.
6
Conclusion
We presented CppPerf-Mine, a configurable pipeline that mines reproducible execution time improving patches from open-source C++ repositories, and CppPerf-DB, a benchmark of 347 such patches spanning 42 mature projects, 39% of which modify multiple files. A preliminary evaluation shows that OpenHands semantically matches the ground-truth for only 13.5% of bugs in CppPerf-DB, confirming that real-world C++ performance repair remains an open challenge. We release CppPerf-Mine so that the community
Tommy Ho, Khashayar Etemadi, and Zhendong Su
can extend CppPerf-DB with newer commits and tailor benchmarks to their evaluation needs.
References [1] Tristan Coignion, Clément Quinton, and Romain Rouvoy. 2024. A performance study of llm-generated code on leetcode. In Proceedings of the 28th international conference on evaluation and assessment in software engineering. 79–89. [2] Mingzhe Du, Luu A Tuan, Bin Ji, Qian Liu, and See-Kiong Ng. 2024. Mercury: A code efficiency benchmark for code large language models. Advances in Neural Information Processing Systems 37 (2024), 16601–16622. [3] Shuzheng Gao, Cuiyun Gao, Wenchao Gu, and Michael R. Lyu. 2025. SearchBased LLMs for Code Optimization. IEEE Press, 578–590. https://doi.org/10.1109/ ICSE55347.2025.00021 [4] Spandan Garg, Roshanak Zilouchian Moghaddam, and Neel Sundaresan. 2025. PerfBench: Can Agents Resolve Real-World Performance Bugs? arXiv preprint arXiv:2509.24091 (2025). [5] Spandan Garg, Roshanak Zilouchian Moghaddam, and Neel Sundaresan. 2025. Rapgen: An approach for fixing code inefficiencies in zero-shot. In 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 124–135. [6] Xinyi He, Qian Liu, Mingzhe Du, Lin Yan, Zhijie Fan, Yiming Huang, Zejian Yuan, and Zejun Ma. 2025. Swe-perf: Can language models optimize code performance on real-world repositories? arXiv preprint arXiv:2507.12415 (2025). [7] Tommy Ho, Khashayar Etemadi, and Zhendong Su. 2026. CppPerf: An Automated Pipeline and Dataset for Performance-Improving C++ Commits. https://github. com/vizual1/CppPerf [8] Dong HUANG, Jianbo Dai, Han Weng, Puzhen Wu, Yuhao QING, Heming Cui, Zhijiang Guo, and Jie Zhang. 2024. EffiLearner: Enhancing Efficiency of Generated Code via Self-Optimization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=KhwOuB0fs9 [9] Dong Huang, Yuhao Qing, Weiyi Shang, Heming Cui, and Jie M Zhang. 2024. Effibench: Benchmarking the efficiency of automatically generated code. Advances in Neural Information Processing Systems 37 (2024), 11506–11544. [10] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. SWE-bench: Can Language Models Resolve RealWorld GitHub Issues? ArXiv abs/2310.06770 (2023). https://api.semanticscholar. org/CorpusID:263829697 [11] René Just, Darioush Jalali, and Michael D Ernst. 2014. Defects4J: A database of existing faults to enable controlled testing studies for Java programs. In Proceedings of the 2014 international symposium on software testing and analysis. 437–440. [12] Jiawei Liu, Songrun Xie, Junhao Wang, Yuxiang Wei, Yifeng Ding, and Lingming Zhang. 2024. Evaluating language models for efficient code generation. arXiv preprint arXiv:2408.06450 (2024). [13] Jeffrey Jian Ma, Milad Hashemi, Amir Yazdanbakhsh, Kevin Swersky, Ofir Press, Enhui Li, Vijay Janapa Reddi, and Parthasarathy Ranganathan. 2025. SWEfficiency: Can Language Models Optimize Real-World Repositories on Real Workloads? arXiv preprint arXiv:2511.06090 (2025). [14] Aman Madaan, Alexander Shypula, Uri Alon, Milad Hashemi, Parthasarathy Ranganathan, Yiming Yang, Graham Neubig, and Amir Yazdanbakhsh. 2023. Learning Performance-Improving Code Edits. ArXiv abs/2302.07867 (2023). https: //api.semanticscholar.org/CorpusID:256868633 [15] Xiaoxue Ren, Jun Wan, Yun Peng, Zhongxin Liu, Ming Liang, Dajun Chen, Wei Jiang, and Yong Li. 2025. PEACE: Towards Efficient Project-Level Efficiency Optimization via Hybrid Code Editing. arXiv preprint arXiv:2510.17142 (2025). [16] Manish Shetty, Naman Jain, Jinjian Liu, Vijay Kethanaboyina, Koushik Sen, and Ion Stoica. 2025. Gso: Challenging software optimization tasks for evaluating swe-agents. arXiv preprint arXiv:2505.23671 (2025). [17] Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. 2025. OpenHands: An Open Platform for AI Software Developers as Generalist Agents. In The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=OJd3ayDDoF [18] John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems 37 (2024), 50528–50652. [19] Lirong Yi, Gregory Gay, and Philipp Leitner. 2025. An Experimental Study of RealLife LLM-Proposed Performance Improvements. arXiv preprint arXiv:2510.15494 (2025). [20] Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. Autocoderover: Autonomous program improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1592–1604.