Test-Oriented Programming: rethinking coding for the GenAI era Jorge Melegati [email protected] INESC TEC, Faculty of Engineering University of Porto Porto, Portugal
arXiv:2604.08102v1 [cs.SE] 9 Apr 2026
Abstract Large language models (LLMs) have shown astonishing capability of generating software code, leading to its use to support developers in programming. Proposed tools have relied either on assistants for improved auto-complete or multi-agents, in which different model instances are orchestrated to perform parts of a problem to reach a complete solution. We argue that LLMs can enable a higher-level of abstraction, a new paradigm we called Test-Oriented Programming (TOP). Within this paradigm, developers only have to check test code generated based on natural language specifications, rather than focusing on production code, which could be delegated to the LLMs. To evaluate the feasibility of this proposal, we developed a proof-of-concept tool and used it to generate a small command-line program employing two different LLMs. We obtained promising results and identified challenges for the use of this paradigm for real projects.
CCS Concepts • Software and its engineering → Software creation and management.
Keywords AI4SE, generative artificial intelligence, test-oriented programming ACM Reference Format: Jorge Melegati. 2026. Test-Oriented Programming: rethinking coding for the GenAI era. In 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE-Companion ’26), April 12–18, 2026, Rio de Janeiro, Brazil. ACM, New York, NY, USA, 2 pages. https://doi.org/10.1145/3774748.3795666
1
Introduction
The emergence of more powerful large language models (LLMs) employed for Generative AI (GenAI) promises to increase developers’ productivity and to improve the quality of produced software [2]. To achieve this aim, current approaches include auto-complete assistants, such as Github Copilot, vibe coding, or the use of multiple agents. However, these approaches still require the inspection of production code by developers which, despite the increased speed of code generation, do not represent an increased level of abstraction. We argue that GenAI allows the increase of the abstraction level of software development, i.e., instructions could be given to the machine in a language closer to natural languages. This shift is
This work is licensed under a Creative Commons Attribution 4.0 International License. ICSE-Companion ’26, Rio de Janeiro, Brazil © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2296-7/2026/04 https://doi.org/10.1145/3774748.3795666
similar to the emergence of high-level languages, e.g., C, that increased the abstraction level of assembly instructions. To this aim, we propose Test-Oriented Programming (TOP), a novel paradigm consisting of the complete delegation of code generation based on natural language specifications, in which developers are only responsible for verifying test code to solve the ambiguity issue of natural languages. To assess the feasibility of our proposal, we developed Onion, a proof-of-concept tool that implements systems in Python based on configuration files. Using Onion, a developer only needs to checks and, if needed, modifies configuration files and test code. We evaluated the capacity of the tool for generating a proposed system when relying on different LLM models. In this process, we also identified challenges to be overcome to take this paradigm to real projects.
2
Test-Oriented Programming
Current research and practice have explored the use of GenAI for SE. Initially, coding assistants, e.g., GitHub Copilot, were integrated in IDEs to generate snippets of code, complete functions or classes based on natural-language descriptions or function signatures. More recently, multiple agents [3], i.e., different instances of LLMs, perform specific tasks towards a common goal. Finally, the term “vibe coding” represents a development workflow based on a conversational process between the developer and a chatbot. However, these proposals do not change the level of abstraction of programming since the output is still in conventional programming languages to be checked by developers. To reach higher levels of abstraction, a reasonable candidate would be natural languages, which, however, suffer from ambiguities. In conventional programming languages, these ambiguities are solved by formal specifications. Another issue is the lack of determinism of LLMs, for which a mitigation strategy being explored is the Assured LLM-based Software Engineering [1]. This approach consists of the application of LLMs to SE in which all LLM responses come “with some verifiable claim to its utility” [1]. For example, test-driven development (TDD) can be employed to guarantee that there exists a way to verify the generated code [5]. We argue that the combination of LLMs and TDD represents a higher-level programming paradigm, which we call Test-Oriented Programming (TOP). TOP is based on creating code for automatic tests (test code), which, alongside natural language specifications, is used to automatically generate production code. It represents an increased abstraction level of programming by completely automatizing the creation of production code, which was previously performed by developers. TOP is related to several existent concepts in SE, therefore it is essential to discuss its novelty. First, even though TOP is also based on the idea of having tests created before production code as TDD, it proposes another level of programming
ICSE-Companion ’26, April 12–18, 2026, Rio de Janeiro, Brazil
abstraction which is not the case for TDD in which tests and production code stay at the same level. This shift is only possible thanks to the automatic generation of code enabled by GenAI. Second, TOP aims to be a generic paradigm suitable to any problem, requiring trained developers capable of inspecting test code and, as such, is not equivalent of low-code development tools, that are adequate for citizen developers but tailored for specific contexts.
3
Preliminary results
To evaluate the feasibility and identify potential challenges to implement TOP, we developed a proof-of-concept tool called Onion. It is an iterative, command-line tool that, based on a configuration file following a pre-defined template in YAML, generates code by interacting with a LLM with pre-defined prompts and with the developer, responsible for verifying the test code. The configuration file contains, as lists of natural language statements, i) a description of the project describing several details of system, including its goal, library dependencies, and outputs and ii) the description of acceptance tests. Based on this information, the tool generates a structure file, also in YAML, containing the description of all the packages to be created and their classes and respective methods. This file could be changed by the developer. Then, the tool creates a file containing the acceptance tests and, for each class, a file containing tests. The developer verifies these files and, if needed, modify them. The execution of the tests are then used to generate the code of the classes. If, after a defined number of tries, the generation fails, the tool aborts and the developer should modify the tests, the structure or even the main configuration file. Once all the classes’ tests pass, the acceptance tests are run and used to generate the main file of the system. The prompts used in the process were defined through an iterative process to allow the completion of the task when using OpenAI’s GPT-4o-mini. The tool is available here1 . To evaluate the tool, we employed it to generate a command-line tool to handle BibTEXentries, persisting them into a pre-defined file, with options to add entries, list all the entries and search for specific text in the entries. We employed two different LLMs: OpenAI’s GPT4o mini and Google’s Gemini 2.5-Flash, to compare a reasoning and a non-reasoning model. First, we used the tool to generate a structure configuration file containing all the classes and the respective methods, which we accepted without changes. Then, keeping the structure unchanged, we generated the production code five times for each LLM from scratch, interacting with the tool as needed. For each run, we collected the interactions needed to complete the tasks, the generated code and all the logs, including the prompts sent to the models. All this data is available as a supplemental package2 . All the tries were successful but with different levels of developer intervention and code quality. In none of the tries, we had to directly modify the production code. In most of the tries, the generation of the production code failed because of inconsistencies in the test code. Once we fixed these issues, the production code was correctly implemented. However, in two tries, one for each model, we had to add comments in the test code to guide the generation of the production code since the models were always creating code which 1 https://github.com/TOProgramming/onion 2 https://doi.org/10.5281/zenodo.17227298
Jorge Melegati
failed the same tests. For GPT-4o-mini, just in one try, we had to modify the test code before checking the implementation. This issue happened more often with the Gemini 2.5-Flash model, however, these issues were more related to supporting code, such as class imports or test execution. A possible explanation for these issues is the fact that prompts had been optimized for another model. The speed and amount of test code, especially for the Gemini 2.5-Flash model, represent a challenge for the developer responsible for checking it. In some tries, the fact that the model was not able to generate code that fulfilled the tests prompted the developer to check the test code, identifying issues on it and fixing it. This issue indicates the need for automatic tools to support the developer in the verification of the test code as summarized in the following challenge: the amount of the generated test code represents a challenge for developers verifying it. We also observed the issue of the lack of determinism of LLMs in the generation of production and test code: most of the tries led to the generation of different outputs. In summary, for the same model, there is some variability of the generated code. Finally, we observed a large difference of the generated code: the ones created by GPT-4o-mini were shorter and had less comments, while the ones created by Gemini 2.5-Flash were longer and contained several comments. This issue can be related to the increased verbosity observed in reasoning models [4]. This observation leads us to the following challenge: there is a huge difference to the generated code for different models.
4
Discussion and Conclusion
Onion is a proof-of-concept and, at the current stage of development, not adequate for the development of a real, complex system. We expect that more complex systems would need to be broken into modules, which in their turn could be generated by the tool. For example, in a microservices architecture, each microservice could be a Onion project. However, our evaluation already brings pieces of evidence to support the feasibility of TOP: i) the automatic generation of production code based on test code without human intervention, and ii) of test code based on specifications with minimal human intervention. We also identified three challenges that need to be handled to employ TOP in real projects.
References [1] Nadia Alshahwan, Mark Harman, Inna Harper, Alexandru Marginean, Shubho Sengupta, and Eddy Wang. 2024. Assured Offline LLM-Based Software Engineering. In Proceedings of the ACM/IEEE 2nd International Workshop on Interpretability, Robustness, and Benchmarking in Neural Software Engineering, Vol. 1. ACM, New York, NY, USA, 7–12. [2] Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M. Zhang. 2023. Large Language Models for Software Engineering: Survey and Open Problems. In 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE). IEEE, 31–53. [3] Junda He, Christoph Treude, and David Lo. 2025. LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision, and the Road Ahead. ACM Transactions on Software Engineering and Methodology 34, 5 (jun 2025), 1–30. [4] Joonwon Jang, Jaehee Kim, Wonbin Kweon, Seonghyeon Lee, and Hwanjo Yu. 2025. Verbosity-Aware Rationale Reduction: Sentence-Level Rationale Reduction for Efficient and Effective Reasoning. In Findings of the Association for Computational Linguistics: ACL 2025. Association for Computational Linguistics, Stroudsburg, PA, USA, 20769–20784. [5] Moritz Mock, Jorge Melegati, and Barbara Russo. 2025. Generative AI for Test Driven Development: Preliminary Results. In Agile Processes in Software Engineering and Extreme Programming – Workshops. Springer Nature Switzerland, Cham, 24–32.