SaaSBench: Exploring the Boundaries of Coding Agents in Long-Horizon Enterprise SaaS Engineering
arXiv:2605.17526v1 [cs.SE] 17 May 2026
Qingnan Ren1 , Shun Zou1,2 , Shiting Huang1 , Ziao Zhang1 , Kou Shi1 , Zhen Fang1 , Yiming Zhao1 , Yu Zeng1 , Qisheng Su1 , Lin Chen1 , Yong Wang2,† , Zehui Chen1 , Xiangxiang Chu2 , Feng Zhao1,† 1
University of Science and Technology of China † Corresponding authors
2
AMAP, Alibaba Group
Abstract As autonomous coding agents become capable of handling increasingly longhorizon tasks, they have gradually demonstrated the potential to complete end-toend software development. Although existing benchmarks have recently evolved from localized code editing to from-scratch project generation, they remain confined to structurally simplified, single-stack applications. Consequently, they fail to capture the heterogeneous environments, full-stack orchestration, and system-level complexity of real enterprise Software as a Service (SaaS) systems, leaving a critical gap in assessing agents under realistic engineering constraints. To fill this gap, we introduce SaaSBench, the first benchmark designed to explore the boundaries of AI agents in enterprise SaaS engineering. Spanning 30 complex tasks across 6 SaaS domains with 5,370 validation nodes, it incorporates 8 programming languages, 6 databases, and 13 frameworks to meticulously mirror real-world software heterogeneity. Furthermore, we design a dependency-aware hybrid evaluation paradigm tailored for complex systems with long horizons and multi-component coupling, enabling fine-grained, reproducible assessment. Crucially, our extensive experiments reveal a striking insight: the primary bottleneck for state-of-the-art agents is not generating isolated code logic, but successfully configuring and integrating a multi-component system. Over 95% of task failures occur before agents even reach deep business logic, with models often falling victim to overconfidence and prematurely halting during foundational system setup, or getting trapped in ineffective debugging loops. We hope SaaSBench serves as a practical and challenging testbed to drive the evolution of reliable, system-level coding agents. The code is available at https://github.com/ShadeCloak/SaaSbench.
1
Introduction
With the rapid development of large language models (LLMs) [2, 4, 20, 27, 41], coding agents have evolved from early tools primarily designed for function completion and localized editing into systems with composite capabilities, including requirement understanding, system design, code generation, environment interaction, and iterative debugging [1, 3, 11, 27, 40]. They are also entering real software development workflows in diverse forms [7, 13, 19, 26, 45, 48]. At the same time, coding agents continue to lower the technical barriers to software development, enabling users without development experience to drive the construction of complete software systems from scratch through natural language requirements [14, 42, 43]. Meanwhile, corresponding benchmarks continue to evolve. As shown in Table 1, this trajectory aligns with the expanding capabilities of coding agents. Existing code benchmarks can be broadly divided into two categories. The first mainly focuses on localized and isolated software engineering Preprint.
Pass@1
OpenHands
Claude Code
Figure 1: Up-to-Date Leaderboard: Coding agent performance on SaaSBench evaluation tasks. tasks, such as function-level code generation, patch fixing, and localized modifications within repositories [5, 6, 9, 17, 21, 25, 28, 30]. These benchmarks are better suited for measuring shorthorizon and localized engineering behaviors, but they struggle to reflect the holistic capabilities required for end-to-end software development. The second category begins to examine the ability of agents to build complete code repositories or projects from scratch based on natural language requirements, thereby placing higher demands on long-horizon planning, cross-file coordination, and system-level consistency [10, 12, 23, 29, 31, 32, 39]. Although recent project-level and repositorylevel benchmarks have made progress, they still face three key limitations: 1. Lack of real-market grounding. Existing benchmarks typically define task instances first and then abstract categories from them. As a result, tasks often lack clear market origins, stable product categories, and well-defined business boundaries. This makes it difficult to assess whether an agent truly possesses the ability to build real commercial Software as a Service (SaaS) products. 2. Limited system complexity. Most existing benchmarks operate in software development settings centered on a single language, a single component, or weakly coupled architectures. In contrast, real SaaS system development typically requires the joint design and implementation of the frontend, backend, database, authentication, deployment, and cross-component workflows. 3. Insufficient evaluation mechanisms. Existing evaluations for end-to-end development tasks usually rely on flat end-to-end signals, such as execution outcomes and unit test pass rates. These evaluation methods lack clear definitions and sufficient constraints. They are suitable only for relatively simple software development tasks and fail to characterize prerequisite dependencies, state dependencies, and other constraints in complex real-world business workflows. To address these limitations, we introduce SaaSBench, the first coding agent benchmark systematically designed for real enterprise-level SaaS development scenarios. SaaSBench starts from real software development markets and their open-source product implementations, and constructs the benchmark through a rigorous multi-stage process with strict quality validation. It contains 30 task instances across 6 high-level SaaS domains, covering mainstream SaaS software development scenarios. Each task consists of a long-context product requirements document (PRD), an ambiguity-resolution knowledge base (KB), a standardized runtime environment, and an accompanying DAG-based test suite. This design evaluates whether coding agents can complete the full engineering loop from scratch, including requirement understanding, system implementation, debugging, deployment, and execution. Overall, the PRDs in SaaSBench contain approximately 4,363 lines on average. The benchmark includes 5,370 executable validation nodes and covers 8 programming languages, 6 database types, and 13 frontend and backend development frameworks, reflecting the complexity and diversity of real-world software development. In addition, we design a dependency-aware hybrid evaluation paradigm for long-horizon and highly interactive end-to-end system development tasks. The paradigm centers on a directed acyclic graph (DAG), where each validation node is compiled into a linear checking chain composed of executable primitives. Through prerequisite dependency gating, failure propagation control, and three scoring mechanisms, namely binary, weighted, and llm-as-judge, it enables reproducible and objective evaluation. The validation nodes cover six capability dimensions: deployment availability, data modeling, API contract consistency, business logic correctness, access control, and engineering 2
Table 1: Comparison of SaaSBench with representative coding agent benchmarks. ✓ fully incorporated; ✓ partially incorporated; ✗ not incorporated; – not applicable. Task Realism Benchmark
System Complexity
From Market- Deployable MultiScratch Grounded Runtime Language
Language
Evaluation Fidelity
PRD CrossWorkflow Semantic Workflow Component Depth Judge Dependency Lines Ç
Snippet-Level Coding Benchmarks
HumanEval [6] MBPP [5] APPS [17] DS-1000 [22] CodeContests [25] EvoCodeBench [24] SWE-Bench [21] GitTaskBench [36]
✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗
✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓
✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗
✗ ✗ ✗ ✗ ✓ ✗ ✗ ✗
Python Python Python Python Py / Ja / C++ Python Python Python
✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗
✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓
– – – – – – – –
✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗
✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗
Repository- & Project-Level Coding Benchmarks
ProjectEval [29] NL2Repo-Bench [10] RepoGenesis [39] PRDBench [12] ProjDevBench [31]
✓ ✓ ✓ ✓ ✓
✗ ✗ ✗ ✗ ✗
✗ ✗ ✓ ✗ ✗
✗ ✗ ✓ ✗ ✗
Python Python Py / Ja Python C++
✗ ✗ ✓ ✗ ✗
✗ ✗ ✗ ✗ ✗
34.45 2452.64 210.17 105.22 283.85
✗ ✗ ✗ ✓ ✓
✗ ✗ ✗ ✗ ✗
SaaSBench (Ours)
✓
✓
✓
✓
Py / Ja / Go . . . (8 langs.)
✓
✓
4362.7 Ç
✓
✓
quality. These dimensions systematically cover the key engineering aspects that must be verified across the lifecycle of real software development. As shown in Figure 1, our experiments reveal that even state-of-the-art coding agents exhibit substantial capability gaps on SaaSBench, highlighting their limitations in long-horizon task planning and cross-component coordination. These findings provide a foundation for further improving the capabilities of future coding agents. Our main contributions are summarized as follows: • We introduce SaaSBench, the first benchmark platform designed to evaluate the ability of coding agents to generate and deploy enterprise-level SaaS systems from scratch. It covers mainstream software development markets. • We design a dependency-aware hybrid evaluation paradigm for end-to-end complex system development tasks. It provides a reproducible and reliable evaluation mechanism and comprehensively covers the key engineering dimensions in the lifecycle of real software development. • We systematically evaluate a broad range of agents and models on SaaSBench. The results show that even the strongest current agents still face severe challenges in enterprise-level SaaS development.
2
Related Work
Autonomous Coding Agents. As the coding capabilities of LLMs continue to improve [4, 20, 41], coding agents have become essential tools in everyday software development. Modern coding agents can be broadly divided into two categories. The first consists of IDE-integrated assistants, such as Cursor, Claude Code, and Codex, which evolve from context-aware code completion toward cross-file modification and repository-level iterative assistance [1, 7, 15, 37]. The second consists of autonomyoriented frameworks, such as OpenHands, Qwen-Agent, and SWE-agent, which incorporate the terminal, file system, and runtime environment into a unified agent loop to support longer-horizon planning, implementation, and debugging [18, 45, 48]. Despite differences in interaction interfaces and product forms, the two categories exhibit a common trend: they integrate terminal access, script execution, dependency installation, and test feedback into the standard workflow, enabling agents to handle end-to-end software engineering tasks with complex dependencies and long feedback loops. Code-Centric Agent Benchmarks. Benchmarks for coding agents have continuously expanded in coverage. Early works such as HumanEval, MBPP, APPS, and CodeContests mainly evaluate function-level code generation in isolated settings [5, 6, 17, 25, 46, 47, 50]. Later, RepoBench and SWE-Bench extend evaluation to real code repositories, requiring agents to perform completion, editing, and issue fixing across multiple files [9, 16, 21, 28, 30, 34, 36]. However, these settings remain largely incremental and primarily measure localized, short-horizon engineering capabilities. 3
(a) Task Construction
Real Business Software Market Real SaaS Business Product
SaaS Domain
Seed Repositories
Star count
Goal and Business Context
Activity level
Technical Requirements
SaaS completeness
Entity Structure and Relations
Business boundary
Workflows and Business Rules
When deleting a customer, how should the associated projects and time records be handled? Use cascading soft deletion. Set the customer’s visible flag to false, automatically set associated projects to visible=false, and keep existing time records unchanged.
…
Product Requirement Document (PRD)
GitHub Repositories
DAG Test Suite
Expert with Agent
Distinct Core Technical Challenges
API Contracts
Human Verified
Docker Environments
Deep Analysis
Knowledge Base (KB)
(c) Evaluation Protocol
(b) System Generation
P1
DAG Test Suite
P2 P3
Coding Agent in Sandbox
+ PRD KB
Rule-Based
Prerequisite Dependencies If (
Running SaaS system
Primitive Chain
or
) Score == 0
then
LLM-as-Judge Rubric Rules
Score: 0
Score: W1*P1+W2*0+ W3*P3
Binary Score
Weighted Score
LLM Judge Score
Figure 2: Overview of SaaSBench. The benchmark is grounded in real software development markets and constructs tasks through a multi-stage human-agent collaborative process. Evaluation is conducted with a reproducible dependency-aware hybrid evaluation paradigm. A recent line of work further requires agents to build complete code repositories or projects from scratch. NL2Repo-Bench [10] generates complete Python projects from specification documents. PRDBench [12] uses product requirements documents (PRDs) as the core input. RepoGenesis [39] targets repository-level web microservice generation. ProjDevBench [31] further incorporates Online Judge diagnostic signals and LLM-based code review. Although these works make progress in repository-level and project-level evaluation, a substantial gap remains between their settings and real enterprise-level SaaS system development. They also lack stable automated evaluation protocols for highly interactive and multi-dependency systems, which is the gap that SaaSBench aims to fill.
3
SaaSBench
As shown in Figure 2, the construction of SaaSBench is carried out through collaboration between experienced doctoral researchers and Cursor [7]. Building a single task requires a multi-stage systematic workflow, including candidate repository auditing, PRD writing, KB organization, standardized container environment preparation, DAG test-suite implementation, and strict quality validation. The detailed construction workflow is presented in the following subsections. 3.1
Benchmark Construction
SaaS Domain Definition and Seed Repository Selection. SaaSBench defines candidate domains from real software development markets. Specifically, we refer to industry taxonomies, publicly available commercial product landscapes, and consultations with domain experts. We retain only domains that satisfy two conditions. First, the domain corresponds to stable commercial SaaS use cases and identifiable product forms. Second, the core technical challenges introduced by the domain 4
are not substantially redundant with those of other selected domains. The resulting task space is therefore clearly grounded in real markets while preserving diversity in engineering patterns. For each selected domain, we further select corresponding seed repositories. Candidate repositories must satisfy the following requirements. They need to show signals of continuous maintenance and community activity, provide a complete SaaS system form, and maintain a clear primary business boundary, meaning that each repository mainly serves one interpretable business domain. Annotators then conduct cold-start validation on the candidate repositories, requiring each repository to be independently built, successfully launched, and verified through basic smoke tests. Detailed descriptions of the domains and repositories are provided in Appendix A.1 and A.3. PRD Construction. After determining the seed repositories, we construct PRDs through a rigorous workflow. First, annotators and agents analyze each repository in depth, systematically examining its code structure, configuration files, route definitions, data models, existing tests, and key business logic. Based on this analysis, we generate comprehensive long-context PRDs. Unlike most benchmarks that retain only short problem descriptions or feature lists, the PRDs in SaaSBench preserve as much key information as possible for system-level development, including technical requirements, complete data models, core business workflows, API contracts, permission policies, boundary rules, deployment constraints, and build steps. This makes them closer to the long-document requirement inputs used in real enterprise development. We ensure that each PRD provides complete coverage of all major aspects of the corresponding repository. KB Construction and Environment Building. In real-world development, clients often provide further revisions and detailed feedback based on an initial product prototype. Similarly, a PRD alone is insufficient to express all evaluation-sensitive details. We therefore further construct an ambiguity-resolution KB. Each KB record corresponds to a behavioral detail that affects correctness but is difficult to express stably in natural language requirements, such as default pagination rules, deletion semantics, or fallback logic. This reduces ambiguity in requirement descriptions and helps ensure the stability and auditability of evaluation. In addition, we build a standardized runtime environment for each task. The environment artifacts are containerized and preinstall the required system packages, system dependencies, database services, port mappings, and environment variables for the corresponding task. 3.2
DAG Evaluation Protocol
Motivation. For end-to-end enterprise-level SaaS development, a conventional list of unit tests is insufficient for reliable evaluation. First, failures in foundational capabilities often introduce secondary noise into many downstream tests, obscuring the true bottlenecks. Second, if evaluation relies only on shallow signals, such as file existence or basic CRUD functionality, an agent may receive a high score even when it fails to correctly implement key business semantics. The fundamental reason is that multi-user interactions, multi-model data operations, and cross-module business workflows in real SaaS systems are not independent. Instead, they form long-horizon interaction processes built on shared application states and explicit prerequisite dependencies. Definition of the DAG-based Hybrid Evaluation Paradigm. Based on these observations, we organize the evaluation paradigm as a DAG G = (V, E). Each node v ∈ V corresponds to an independently scored validation unit, and each edge e ∈ E explicitly represents a prerequisite dependency between nodes. Each node contains a primitive chain composed of basic validation primitives executed in sequence. The executable checks include HTTP requests, authentication login, and rubric-based LLM judgment, among others, as detailed in Table 12. Node scoring falls into three categories. binary is used for scenarios that must be fully correct, such as permission gating and security constraints. weighted is used for scenarios that allow partial completion, such as multi-step CRUD workflows. llm-as-judge is used only when deterministic assertions cannot adequately characterize the target, such as the reasonableness of page layout. In addition, we assign each evaluation node to one of six engineering capability dimensions: Deploy, Data, API, Logic, AuthZ, and Quality, enabling comprehensive evaluation of software engineering dimensions. DAG Test Suite Construction. The DAG is not constructed by manually listing test items in an arbitrary manner. Instead, it follows a comprehensive and complete definition and is systematically compiled from the task artifacts. First, annotators collaborate with agents to scan the PRD and map each verifiable requirement to a candidate node. Next, any assertion involving potential ambiguity 5
SaaS Bench
(a) Programming Languages
(b) Database Types
(c) Frontend Frameworks
(d) Backend Frameworks
Figure 3: Statistical overview of SaaSBench. Left: SaaSBench includes six key SaaS domains and 30 fine-grained categories, covering mainstream software development markets. Right: Distribution of tasks across programming languages, database types, and frontend and backend frameworks. must be aligned with the KB. Finally, each node must be compiled into an executable linear chain of primitives and assigned prerequisite dependencies that reflect real business workflows. Detailed definitions are provided in Appendix B.5. Evaluation Pipeline. As shown in Figure 2, during evaluation, the agent receives two inputs, the PRD and the KB, together with a carefully designed prompt, as detailed in Appendix C, and runs in an isolated, pre-built Docker environment. Within the specified workspace of this environment, the agent is granted full autonomy to build and deploy a runnable and accessible SaaS system from scratch, without any human intervention. The evaluation system then topologically sorts the DAG test suite corresponding to the task and executes the evaluation nodes on the running system one by one in dependency order. If any prerequisite dependency of a node is not satisfied, the node is not simply marked as a direct failure. Instead, it is marked as Skipped dependency, which prevents foundational errors from being repeatedly penalized across all downstream nodes. 3.3
Task Quality Validation
PRD Alignment Verification. To ensure the completeness and accuracy of each PRD, we introduce an independent review and revision loop. After the initial PRD is completed, two additional annotators inspect the seed repository and verify the PRD with a structured checklist. For each missing, inconsistent, or underspecified requirement, the reviewers record a revision item and return it to the PRD author for refinement. The revised PRD is checked again until the reviewers confirm that it covers the key capabilities of the repository. This process reduces requirement omissions and hallucinated requirements, and improves the alignment between each task and the corresponding executable SaaS system. Test-Suite Quality Assurance. To avoid subtle errors in the test suite, such as incorrect assertions or fragile chains of atomic capability calls, we conduct strict quality validation for each task. Specifically, we deploy the upstream source code of the seed repository in the same standardized runtime environment used for evaluation, and require the reference implementation to pass the full test suite. For llm-as-judge nodes, we allow bounded variance. Only tasks that pass this validation are included in the benchmark. Tasks that fail this gate are revised until they converge. 3.4
Benchmark Statistics
SaaSBench exhibits clear characteristics of real SaaS development in terms of task coverage, technology-stack diversity, and system complexity. As shown in Figure 3, SaaSBench contains 30 tasks, covering 6 high-level domains and 30 fine-grained SaaS categories. Each task typically includes a frontend interface, backend APIs, persistent data models, a role-based permission system, 6
Table 2: Main results on SaaSBench across different agent–model configurations. We report the overall Pass@1 and Node Coverage, together with scores over the six SaaS domains. Within each coding agent block, bold numbers indicate the best performance, and underlined numbers indicate the second-best performance. Overall
Coding Agent
SaaS Domain
Pass@1
Node Cov.
CG
PC
CF
DCI
SI
DW
4.09 8.36 6.78 7.44 8.10 10.97 10.23 18.12
5.85 8.94 6.43 8.83 8.91 10.55 11.21 18.24
4.19 5.78 3.59 7.63 1.94 4.29 14.03 36.16
3.42 6.97 5.87 2.23 3.37 9.97 7.02 8.62
2.10 2.60 1.98 0.98 2.30 4.35 3.97 4.62
2.43 5.79 13.22 6.34 8.46 10.37 10.65 7.69
3.70 17.45 3.28 14.25 25.25 21.93 16.22 24.90
9.38 14.88 13.98 15.96 14.20 20.60 8.12 20.55
5.95 9.70 9.26 10.62 10.12 13.19 13.60 20.68
6.10 9.70 10.62 9.81 10.12 12.13 15.28 18.50
6.85 4.49 11.07 11.90 11.29 15.96 11.08 21.51
7.12 9.90 5.63 4.17 4.28 8.98 10.57 9.30
2.92 0.48 4.92 2.40 1.47 4.48 6.75 15.65
6.48 10.51 8.46 14.50 9.97 12.78 13.91 37.14
0.53 22.07 17.57 20.62 30.15 22.90 25.93 34.35
10.39 14.38 8.56 11.41 5.60 14.16 16.73 7.06
OpenHands Qwen 3.6 Plus Kimi K2.6 MiniMax M2.7 GPT-5.4 Gemini 3.1 Pro DeepSeek V4 Pro GLM 5.1 Claude Opus 4.7 Claude Code Qwen 3.6 Plus Kimi K2.6 MiniMax M2.7 GPT-5.4 Gemini 3.1 Pro DeepSeek V4 Pro GLM 5.1 Claude Opus 4.7
and deployment configurations, forming a system structure that is clearly distinct from function-level, patch-level, and toy project-level or repository-level benchmarks. Overall, the PRDs in SaaSBench contain 4,363 lines on average. The benchmark includes 5,370 executable validation nodes connected by 6,167 prerequisite dependency edges, and covers 8 programming languages, 6 types of database systems, 5 types of frontend frameworks, and 8 types of backend frameworks. More fine-grained per-task statistics and technology-stack distributions are provided in Appendix A.3.
4
Experiments
4.1
Experimental Settings
Evaluated Agents and LLM Backends. We evaluate eight state-of-the-art open-source and closed-source large models, including GPT-5.4 [38], Gemini 3.1 Pro [44], Claude Opus 4.7 [4], Kimi K2.6 [35], Qwen 3.6 Plus [40], DeepSeek V4 Pro [8], GLM 5.1 [49], and MiniMax M2.7 [33]. We integrate these models into two representative coding agent frameworks: OpenHands [45] and Claude Code [1]. For llm-as-judge nodes, we use Claude Sonnet 4.5 [3] as the rubric judge and set temperature = 0 to maximize reproducibility. More detailed experimental settings, agent framework descriptions, and model information are provided in Appendix B.3. Evaluation Metrics. We mainly report the average pass@1 across all tasks, as well as performance across the six SaaS domains: Customer & Growth (CG), Productivity & Collaboration (PC), Commerce & Finance (CF), Data & Content Infrastructure (DCI), Security, Identity & Infrastructure (SI), and Domain & Workflow Platforms (DW). We also report the node-coverage rate, defined as the proportion of validation nodes that reach the Passed state across all tasks. Detailed metric definitions are provided in Appendix B.4. 7
4.2
Main Results
SaaSBench is a Highly Challenging Benchmark. As shown in Table 2 and Figure 1, SaaSBench poses a substantial challenge to current coding agents. The best result is only 20.68%, achieved by Claude Opus 4.7 under Claude Code. On average, Claude Code achieves a performance of 11.64%, while OpenHands achieves 9.26%. These results show that current LLM-based coding agents still struggle to reliably generate complete enterpriselevel SaaS systems from scratch based on natural language requirements, and that end-to-end SaaS construction remains far from solved. Unlike prior toy-level project or repository generation from scratch, SaaSBench requires agents to jointly handle long-context requirement understanding, multistep task planning, cross-component implementation, persistent data modeling, permission logic, and deployment-level execution. Performance Varies across Task Domain. We further break down the evaluation results by the six high-level SaaS domains. Overall, models perform relatively better on SI and DCI, while their performance is consistently weaker on CF and PC. This difference suggests that current coding agents can more easily handle infrastructure-oriented tasks with clear structures and well-defined interface boundaries. However, they still face substantial challenges in scenarios involving complex business semantics, long-horizon state dependencies, and multi-user collaborative behavior. In particular, billing, order, and financial-state consistency in CF tasks, as well as calendar interactions and shared-state management in PC tasks, often require agents to maintain cross-component consistency among the frontend, backend, database, and permission logic. These results further characterize the capability boundaries of current models across different SaaS product forms.
5
Fine-grained Analysis
5.1
Performance by Engineering Capability Dimension
We further report the evaluation results Table 3: Engineering capability dimensions results on of agents across six engineering capa- SaaSBench. We report per-category scores for Deploy, bility dimensions: Deploy, Data, API, Data, API, Logic, AuthZ, and Quality. Logic, AuthZ, and Quality. Detailed defCapability dimensions score initions are provided in Appendix B.6. Coding Agent As shown in Table 3, failures are not Deploy Data API Logic AuthZ Quality uniformly distributed. Deploy is usually OpenHands the highest-scoring dimension, suggest- Qwen 3.6 Plus 14.12 8.48 1.06 1.02 3.03 1.46 ing that current coding agents already Kimi K2.6 18.68 11.29 5.39 5.85 7.66 2.28 16.75 7.85 2.85 2.64 4.08 0.93 possess some ability in service startup MiniMax M2.7 19.31 7.34 4.33 5.97 6.81 1.76 and basic runtime environment config- GPT-5.4 3.1 Pro 12.81 8.37 6.53 6.03 8.24 3.59 uration. Data remains at an intermedi- Gemini DeepSeek V4 Pro 18.18 9.67 6.84 6.13 8.11 3.02 ate level, while API, Logic, and AuthZ GLM 5.1 18.54 8.04 3.36 4.11 6.34 2.18 obtain lower scores. This indicates that Claude Opus 4.7 19.46 12.40 6.72 6.87 9.43 2.99 agents still face clear difficulties in in- Claude Code terface contract consistency, persistent Qwen 3.6 Plus 13.07 5.97 0.27 0.69 2.46 0.54 11.57 7.08 3.96 4.64 5.35 1.18 state modeling, business state transitions, Kimi K2.6 18.03 11.92 3.74 3.80 6.18 1.79 and role-based access control. The most MiniMax M2.7 GPT-5.4 19.75 9.27 3.82 5.29 7.48 1.80 prominent bottleneck lies in the Quality Gemini 3.1 Pro 17.69 10.84 6.12 6.77 7.98 2.32 dimension, where scores are substantially DeepSeek V4 Pro 21.31 10.36 4.25 5.23 6.77 2.00 19.25 12.88 6.37 7.81 7.79 3.02 lower than those of the other engineer- GLM 5.1 ing capability dimensions. This indicates Claude Opus 4.7 22.76 14.57 7.92 8.79 11.55 2.66 that even when agents can generate services that start successfully and implement some local functionality, they still lack sufficient code organization, frontend rendering quality, edge-case handling, and overall engineering robustness. This trend is qualitatively associated with higher structural and interaction complexity, and provides guidance for improving future coding agents. 8
(a) Agent Framework Performance
(b) Failure mode breakdown
Figure 4: Left: Performance analysis of agent frameworks. Right: We classify capability units into five execution trajectories. T4 and T5 account for 95.6% of all units, showing that most failures occur before agents reach deep business logic. See Appendix B.7 for definitions. 5.2
Agent Frameworks
To examine the impact of agent frameworks beyond the underlying model, we evaluate GPT-5.4 and Claude Opus 4.7 under three agent frameworks: OpenHands, Claude Code, and Codex CLI. As shown in Figure 4 (a), the same underlying model can exhibit substantial performance differences across different agent frameworks. Commercial IDE-based frameworks often outperform open-source agent frameworks. This gap mainly stems from how each framework manages tool calls, context, and execution feedback. Commercial IDE-oriented frameworks are usually more tightly integrated with file editing, terminal execution, diagnostic information, and intermediate project states, thereby reducing the burden on the model to track workspace changes and recover from failed commands. In contrast, more open frameworks often require the model to bear greater coordination costs. In long-horizon SaaS tasks, these differences gradually accumulate and lead to clear gaps in deployment stability, dependency repair, schema consistency, and error recovery. This highlights that SaaSBench does not evaluate an isolated LLM, but rather a coupled system composed of the model, tool interfaces, execution loop, environment feedback mechanism, memory mechanism, and error recovery strategy. 5.3
Interaction Turns and Performance
Due to the high complexity of SaaSBench tasks, agents of- Table 4: Interaction behavior and executen perform long autonomous multi-turn interactions. As tion cost of OpenHands on SaaSBench. shown in Table 4 (a), we report steps, total token consump- Model Pass@1 Steps Tokens Time tion, and average time to analyze interaction behavior and Qwen 3.6 Plus 4.09 258 12.4M 1h 12m execution cost in end-to-end SaaS construction. We find Kimi K2.6 8.36 159 9.0M 2h 17m that more steps do not guarantee higher system comple- MiniMax M2.7 6.78 279 24.4M 56m 12s 7.44 36 1.4M 7m 16s tion. Successful construction depends on the ability of the GPT-5.4 3.1 Pro 8.10 84 4.1M 12m 52s agent to use environmental feedback, identify root causes Gemini DeepSeek V4 Pro 10.97 223 9.8M 1h 24m of build errors, runtime exceptions, interface responses, GLM 5.1 10.23 218 8.2M 1h 52m and data-state issues, and apply targeted fixes. Claude Opus 4.7 18.12 102 12.1M 29m 55s Specifically, we observe two typical phenomena. The first is insufficient interaction or premature convergence: GPT-5.4 executes only 36 steps yet achieves a score of 7.44%, suggesting high singlestep generation quality but limited long-horizon debugging and system validation. The second is ineffective long-horizon interaction: MiniMax M2.7 executes 279 steps yet achieves only 6.78%, indicating that many attempts may still result in repeated debugging, local patching, or inefficient exploration. These results show that long-horizon SaaS development depends more on high-quality reason-act-observe loops than on a larger interaction budget. 5.4
Error Analysis and Failure Modes
To examine where the development process breaks down, we analyze 480 capability units from two agents and define five execution-trajectory categories, as detailed in Appendix B.7. As shown in Figure 4 (b), the dominant failure mode in SaaSBench is not weakness in a single capability dimension. Instead, most failures occur before agents reach deep business logic. Overall, in 63.5% of the capability units, the generated stack never runs stably. Another 32.1% are only superficially 9
accessible but structurally incomplete. Only 3.8% progress to a stage where incomplete business logic becomes the main bottleneck. This indicates that real enterprise-level SaaS development first tests process discipline, deployment stability, dependency management, schema correctness, and reproducible execution, rather than isolated algorithmic or API skills. Therefore, the key limitation of current state-of-the-art coding agents is not failure to implement specific advanced business logic. Rather, they lack the ability to stably complete end-to-end engineering setup. This is exactly the challenge that future coding agents must overcome.
6
Conclusion
In this paper, we introduce SaaSBench, a comprehensive benchmark for evaluating the ability of coding agents to develop and deploy enterprise-level SaaS systems from scratch. Through extensive experiments, we find that even Claude Opus 4.7, the strongest model in our evaluation, performs poorly in delivering a complete system from scratch. We hope that SaaSBench can provide an important foundation for the future development of coding agents and help move the field toward practical “Vibe Coding”.
References [1] Anthropic. Claude code: Ai-powered coding assistant, 2024. URL https://www.claude. com/product/claude-code. Accessed: 2026-05-03. [2] Anthropic. System card: Claude opus 4 & claude sonnet 4, 2025. URL https://www-cdn. anthropic.com/4263b940cabb546aa0e3283f35b686f4f3b2ff47.pdf. Accessed: 202605-03. [3] Anthropic. System card: Claude sonnet 4.5, 2025. URL https://assets.anthropic.com/ m/12f214efcc2f457a/original/Claude-Sonnet-4-5-System-Card.pdf. Accessed: 2026-05-03. [4] Anthropic. Introducing Claude Opus 4.7, 2026. URL https://www.anthropic.com/news/ claude-opus-4-7. Accessed: 2026-05-03. [5] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models. CoRR, abs/2108.07732, 2021. URL https://arxiv.org/abs/ 2108.07732. [6] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021. URL https://arxiv.org/abs/2107.03374. [7] Cursor AI. Cursor: The ai code editor, 2024. URL https://www.cursor.com. Accessed: 2026-05-03. [8] DeepSeek-AI. Deepseek-v4: Towards highly efficient million-token context intelligence. Technical report, DeepSeek-AI, 2026. URL https://huggingface.co/deepseek-ai/ DeepSeek-V4-Pro/blob/main/DeepSeek_V4.pdf. Accessed: 2026-05-03. [9] Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Vijay Bharadwaj, Jeff Holm, Raja Aluri, Chen Bo Calvin Zhang, 10
Noah Jacobson, Bing Liu, and Brad Kenstler. Swe-bench pro: Can AI agents solve long-horizon software engineering tasks? CoRR, abs/2509.16941, 2025. URL https://arxiv.org/abs/ 2509.16941. [10] Jingzhe Ding, Shengda Long, Changxin Pu, Huan Zhou, Hongwan Gao, Xiang Gao, Chao He, Yue Hou, Fei Hu, Zhaojian Li, Weiran Shi, Zaiyuan Wang, Daoguang Zan, Chenchen Zhang, Xiaoxu Zhang, Qizhi Chen, Xianfu Cheng, Bo Deng, Qingshui Gu, Kai Hua, Juntao Lin, Pai Liu, Mingchen Li, Xuanguang Pan, Zifan Peng, Yujia Qin, Yong Shan, Zhewen Tan, Weihao Xie, Zihan Wang, Yishuo Yuan, Jiayu Zhang, Enduo Zhao, Yunfei Zhao, He Zhu, Liya Zhu, Chenyang Zou, Ming Ding, Jianpeng Jiao, Jiaheng Liu, Minghao Liu, Qian Liu, Chongyang Tao, Jian Yang, Tong Yang, Zhaoxiang Zhang, Xinjie Chen, Wenhao Huang, and Ge Zhang. Nl2repo-bench: Towards long-horizon repository generation evaluation of coding agents. CoRR, abs/2512.12730, 2026. URL https://arxiv.org/abs/2512.12730. [11] Yihong Dong, Xue Jiang, Jiaru Qian, Tian Wang, Kechi Zhang, Zhi Jin, and Ge Li. A survey on code generation with llm-based agents. CoRR, abs/2508.00083, 2025. URL https: //arxiv.org/abs/2508.00083. [12] Lingyue Fu, Bolun Zhang, Hao Guan, Yaoming Zhu, Lin Qiu, Weiwen Liu, Xuezhi Cao, Xunliang Cai, Weinan Zhang, and Yong Yu. Automatically benchmarking LLM code agents through agent-driven annotation and evaluation. CoRR, abs/2510.24358, 2025. URL https: //arxiv.org/abs/2510.24358. [13] Pengfei Gao, Zhao Tian, Xiangxin Meng, Xinchen Wang, Ruida Hu, Yuanan Xiao, Yizhou Liu, Zhao Zhang, Junjie Chen, Cuiyun Gao, Yun Lin, Yingfei Xiong, Chao Peng, and Xia Liu. Trae agent: An llm-based agent for software engineering with test-time scaling. CoRR, abs/2507.23370, 2025. URL https://arxiv.org/abs/2507.23370. [14] Yuyao Ge, Lingrui Mei, Zenghao Duan, Tianhao Li, Yujia Zheng, Yiwei Wang, Lexin Wang, Jiayu Yao, Tianyu Liu, Yujun Cai, Baolong Bi, Fangda Guo, Jiafeng Guo, Shenghua Liu, and Xueqi Cheng. A survey of vibe coding with large language models. CoRR, abs/2510.12399, 2025. URL https://arxiv.org/abs/2510.12399. [15] GitHub. Github copilot: Your ai pair programmer, 2021. URL https://copilot.github. com/. Accessed: 2026-05-03. [16] Xinyi He, Qian Liu, Mingzhe Du, Lin Yan, Zhijie Fan, Yiming Huang, Zejian Yuan, and Zejun Ma. Swe-perf: Can language models optimize code performance on real-world repositories? arXiv preprint arXiv:2507.12415, 2025. URL https://arxiv.org/abs/2507.12415. [17] Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with APPS. CoRR, abs/2105.09938, 2021. URL https://arxiv.org/ abs/2105.09938. [18] Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. Metagpt: Meta programming for A multi-agent collaborative framework. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=VtmBAGCN7o. [19] Siming Huang, Tianhao Cheng, Jason Klein Liu, Weidi Xu, Jiaran Hao, Liuyihan Song, Yang Xu, Jian Yang, Jiaheng Liu, Chenchen Zhang, Linzheng Chai, Ruifeng Yuan, Xianzhen Luo, Qiufeng Wang, YuanTao Fan, Qingfu Zhu, Zhaoxiang Zhang, Yang Gao, Jie Fu, Qian Liu, Houyi Li, Ge Zhang, Yuan Qi, Yinghui Xu, Wei Chu, and Zili Wang. Opencoder: The open cookbook for top-tier code large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 33167–33193, 2025. URL https://aclanthology.org/2025.acl-long.1591/. [20] Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation. ACM Transactions on Software Engineering and Methodology, 35 (2):1–72, 2026. URL https://dl.acm.org/doi/10.1145/3747588. 11
[21] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? CoRR, abs/2310.06770, 2024. URL https://arxiv.org/abs/2310.06770. [22] Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, pages 18319–18345. PMLR, 2023. URL https://proceedings.mlr.press/v202/lai23b.html. [23] Bowen Li, Wenhan Wu, Ziwei Tang, Lin Shi, John Yang, Jinyang Li, Shunyu Yao, Chen Qian, Binyuan Hui, Qicheng Zhang, Zhiyin Yu, He Du, Ping Yang, Dahua Lin, Chao Peng, and Kai Chen. Prompting large language models to tackle the full software development lifecycle: A case study. In Proceedings of the 31st International Conference on Computational Linguistics, pages 7511–7531, 2025. URL https://aclanthology.org/2025.coling-main.502/. [24] Jia Li, Ge Li, Xuanming Zhang, Yihong Dong, and Zhi Jin. Evocodebench: An evolving code generation benchmark aligned with real-world code repositories. CoRR, abs/2404.00599, 2024. URL https://arxiv.org/abs/2404.00599. [25] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode. Science, 378(6624):1092–1097, 2022. doi: 10.1126/science. abq1158. URL https://www.science.org/doi/abs/10.1126/science.abq1158. [26] Jiaye Lin, Yifu Guo, Yuzhen Han, Sen Hu, Ziyi Ni, Licheng Wang, Mingguang Chen, Hongzhang Liu, Ronghao Chen, Yangfan He, Daxin Jiang, Binxing Jiao, Chen Hu, and Huacan Wang. Se-agent: Self-evolution trajectory optimization in multi-step reasoning with llm-based agents. CoRR, abs/2508.02085, 2025. URL https://arxiv.org/abs/2508.02085. [27] Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al. Deepseek-v3.2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556, 2025. URL https://arxiv.org/ abs/2512.02556. [28] Jiaheng Liu, Ken Deng, Congnan Liu, Jian Yang, Shukai Liu, He Zhu, Peng Zhao, Linzheng Chai, Yanan Wu, Ke Jin, Ge Zhang, Zekun Moore Wang, Guoan Zhang, Yingshui Tan, Bangyu Xiang, Zhaoxiang Zhang, Wenbo Su, and Bo Zheng. M2RC-EVAL: massively multilingual repository-level code completion evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15661–15684, 2025. URL https://aclanthology.org/2025.acl-long.763/. [29] Kaiyuan Liu, Youcheng Pan, Yang Xiang, Daojing He, Jing Li, Yexing Du, and Tianrun Gao. Projecteval: A benchmark for programming agents automated evaluation on project-level code generation. In Findings of the Association for Computational Linguistics, pages 20205–20221, 2025. URL https://aclanthology.org/2025.findings-acl.1036/. [30] Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repositorylevel code auto-completion systems. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=pPjZIOuQuF. [31] Pengrui Lu, Shiqi Zhang, Yunzhong Hou, Lyumanshan Ye, Chaoyi Huang, Zixi Chen, Ji Zeng, Hantao Jiang, Pengfei Liu, Yiwei Wang, and Ming-Hsuan Yang. Projdevbench: Benchmarking AI coding agents on end-to-end project development. CoRR, abs/2602.01655, 2026. URL https://arxiv.org/abs/2602.01655. [32] Zimu Lu, Yunqiao Yang, Houxing Ren, Haotian Hou, Han Xiao, Ke Wang, Weikang Shi, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Webgen-bench: Evaluating llms on generating interactive and functional websites from scratch. CoRR, abs/2505.03733, 2025. URL https: //arxiv.org/abs/2505.03733. 12
[33] MiniMax. MiniMax M2.7: Early echoes of self-evolution, 2026. URL https://www.minimax. io/news/minimax-m27-en. Accessed: 2026-05-03. [34] Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke. SWE-lancer: Can frontier LLMs earn $1 million from real-world freelance software engineering? In Proceedings of the 42nd International Conference on Machine Learning, volume 267, pages 44412– 44450. PMLR, 2025. URL https://proceedings.mlr.press/v267/miserendino25a. html. [35] Moonshot AI. Kimi K2.6: Advancing open-source coding, 2026. URL https://www.kimi. com/blog/kimi-k2-6. Accessed: 2026-05-03. [36] Ziyi Ni, Huacan Wang, Shuo Zhang, Shuo Lu, Ziyang He, Wang You, Zhenheng Tang, Sen Hu, Bo Li, Chen Hu, Binxing Jiao, Daxin Jiang, Yuntao Du, and Pin Lyu. Gittaskbench: A benchmark for code agents solving real-world tasks through code repository leveraging. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 32564–32572, 2026. URL https://doi.org/10.1609/aaai.v40i38.40533. [37] OpenAI. Codex cli, 2025. URL https://github.com/openai/codex. Accessed: 2026-0503. [38] OpenAI. GPT-5.4 Thinking System Card, 2026. URL https://deploymentsafety.openai. com/gpt-5-4-thinking/gpt-5-4-thinking.pdf. Accessed: 2026-05-03. [39] Zhiyuan Peng, Xin Yin, Pu Zhao, Fangkai Yang, Lu Wang, Ran Jia, Xu Chen, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. Repogenesis: Benchmarking end-to-end microservice generation from readme to repository. CoRR, abs/2601.13943, 2026. URL https://arxiv. org/abs/2601.13943. [40] Qwen Team. Qwen3.6-Plus: Towards real world agents, 2026. URL https://qwen.ai/ blog?id=qwen3.6. Accessed: 2026-05-03. [41] Qwen Team. Qwen3.6-27B: Flagship-level coding in a 27B dense model, 2026. URL https: //qwen.ai/blog?id=qwen3.6-27b. Accessed: 2026-05-03. [42] Ranjan Sapkota, Konstantinos I. Roumeliotis, and Manoj Karkee. Vibe coding vs. agentic coding: Fundamentals and practical implications of agentic AI. CoRR, abs/2505.19443, 2025. URL https://arxiv.org/abs/2505.19443. [43] Advait Sarkar and Ian Drosos. Vibe coding: programming through conversation with artificial intelligence. arXiv preprint arXiv:2506.23253, 2025. URL https://arxiv.org/abs/2506. 23253. [44] The Gemini Team. Gemini 3.1 Pro: a smarter model for your most complex tasks, 2026. URL https://blog.google/innovation-and-ai/models-and-research/ gemini-models/gemini-3-1-pro/. Accessed: 2026-05-03. [45] Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. Openhands: An open platform for AI software developers as generalist agents. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=OJd3ayDDoF. [46] Zihan Wang, Siyao Liu, Yang Sun, Hongyan Li, and Kai Shen. Codecontests+: High-quality test case generation for competitive programming. CoRR, abs/2506.05817, 2025. URL https: //arxiv.org/abs/2506.05817. [47] Jingxuan Xu, Ken Deng, Weihao Li, Songwei Yu, Huaixi Tang, Haoyang Huang, Zhiyi Lai, Zizheng Zhan, Yanan Wu, Chenchen Zhang, Kepeng Lei, Yifan Yao, Xinping Lei, Wenqiang Zhu, Zong-Xian Feng, Han Li, Junqi Xiong, Dailin Li, Zuchen Gao, Kun Wu, Wen Xiang, Ziqi Zhan, Yuanxing Zhang, Wuxuan Gong, Ziyuan Gao, Guanxiang Wang, Yirong Xue, Mengtong Li, Mengfei Xie, Xiaojiang Zhang, Jinghui Wang, Wenhao Zhuang, Zheng Lin, 13
Huiming Wang, Zhaoxiang Zhang, Yuqun Zhang, Haotian Zhang, Bin Chen, and Jiaheng Liu. Swe-compass: Towards unified evaluation of agentic coding abilities for large language models. CoRR, abs/2511.05459, 2025. URL https://arxiv.org/abs/2511.05459. [48] John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528– 50652, 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/ hash/5a7c947568c1b1328ccc5230172e1e7c-Abstract-Conference.html. [49] Z.AI. Glm-5.1: Towards long-horizon tasks, 2026. URL https://z.ai/blog/glm-5.1. Accessed: 2026-05-03. [50] Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang, Binyuan Hui, Niklas Muennighoff, David Lo, Daniel Fried, Xiaoning Du, Harm de Vries, and Leandro Von Werra. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=YrycTjllL0.
A
Benchmark Details and Statistics
A.1
Domain Selection Principles
The domain space of SaaSBench is not assembled bottom-up from an arbitrary collection of repositories. Instead, it is derived top-down from the commercial SaaS market. We begin with broad market segments and progressively refine them into categories that are distinguishable in terms of engineering patterns, until each retained category satisfies three conditions simultaneously: (i) it corresponds to a stable commercial use case, (ii) it has a recognizable product form, and (iii) there exists at least one production-grade open-source implementation that annotators can successfully build and launch. At the macro level, the final 30 categories are organized into 6 families along two axes: position in the value chain (front office / middle office / back office) and target service recipient (external customers vs. internal users or systems). This yields the six families Customer & Growth (CG), Productivity & Collaboration (PC), Commerce & Finance (CF), Data & Content Infrastructure (DCI), Security, Identity & Infra (SI), and Domain & Workflow (DW). These macro families are used only for conceptual organization and visualization. They do not alter the actual scoring dimensions of the benchmark, nor do they replace fine-grained orthogonality analysis. Beyond market realism, we also explicitly ensure category independence through a four-dimensional orthogonality audit. For any candidate category pair (A, B), we compare four dimensions in sequence: (D1) the core business objects, (D2) the core user roles, (D3) the dominant data read/write patterns, and (D4) the core architectural challenges. The decision rule is straightforward: 4/4 distinct indicates full orthogonality; 3/4 distinct indicates that the categories are still independent, with only one dimension exhibiting explainable local overlap; 2/4 distinct requires additional stress testing; and 0–1/4 distinct is treated as category overlap and must be merged or removed. This audit framework deliberately ignores generic capabilities that are shared by almost all SaaS systems, such as authentication, CRUD operations, and notifications. What truly matters is not whether two systems share boilerplate components, but whether their most distinctive engineering bottlenecks require fundamentally different technical solutions. We apply this audit to all 30 2 = 435 category pairs. The final result is as follows: 421 pairs are fully orthogonal, and the remaining 14 pairs exhibit only bounded local overlap in one dimension. No category pair falls into the risk zone of 2/4 distinct or below. These 14 pairs are retained not because they are problematic but temporarily tolerated. Rather, their overlap is primarily semantic rather than engineering-essential: along the other three dimensions, especially the dimension of dominant engineering bottlenecks, they remain stably distinguishable. For example, Workflow Automation and Low-Code / No-Code both provide visual construction interfaces, but the former is centered on DAG execution and connector orchestration, whereas the latter is centered on UI rendering 14
Table 5: SaaSBench category definitions and engineering scope (Part I). Rows 1–15 summarise the functional scope and distinctive engineering challenge of each category. #
Dom.
Category
1
CG
Email & Newsletter
2
CG
3
CG
4
CG
5
CG
6
CG
7
CG
8
PC
Project Management & Issue Tracking
9
PC
Knowledge Base & Wiki
10
PC
Time Tracking
11
PC
Gamified Productivity
12
PC
Scheduling & Booking
13
PC
Learning Management System
14
CF
E-commerce Platform
15
CF
Billing & Subscription
Definition
Core Engineering Challenge
SaaS that composes, segments, and delivers bulk transactional and marketing email, and tracks open/click/bounce funnels. Customer Relationship Manage- System of record for accounts, contacts, opportument nities and deal pipelines, with activity logging and forecasting. Help Desk & Ticketing Omni-channel inbox (email, chat, voice, social) that unifies customer requests as tickets with SLAs, macros, and agent workflows. Form Builder & Survey No-code builder for web forms, surveys, and quizzes with conditional logic and response analytics. Community Forum Threaded public/private discussion platform with trust levels, moderation tooling, and topic-level SEO. Real-time Communication Team messaging with channels, threads, direct messages, presence, and file sharing, over persistent WebSocket. Video Conferencing Real-time audio/video meetings with screen share, recording, and large-room broadcasting. Tool for planning, tracking, and reporting work items across sprints, epics, and cross-functional teams. Hierarchical collaborative document system with rich-text editing, versioning, search, and permissioned sharing. Tool that records time spent per task/project, generates timesheets, and exports billable hours. Habit/to-do app that models tasks as RPG quests with rewards, streaks, and social challenges. Self-service scheduling links that expose a user’s availability and let external parties book within configurable rules. Platform that authors courses, delivers content, runs assessments, and reports learner outcomes. Multi-channel storefront with product catalogue, cart, checkout, payment and order management. Engine that meters usage, prices subscription plans, and issues recurring invoices with dunning and renewals.
High-throughput SMTP pipeline with DKIM/SPF/DMARC signing, bounce/ complaint handling, and pertenant IP-warming and reputation. Multi-object relational graph (account/contact/opportunity/activity) with custom fields, role-based views, and pipeline state machines. Channel unification, SLA/priority timers, skill-based routing, and agent collision detection across concurrent conversations. Conditional-logic branching engine, schema versioning, and response ingestion with partial-submit and anti-fraud controls. Trust-level reputation model, spam/abuse moderation queue, and topic ranking with SEO-grade public rendering. Persistent WebSocket fan-out, channel/thread model, presence and typing indicators, and federation across workspaces. WebRTC SFU media routing with simulcast, bandwidth adaptation, and server-side recording / transcoding. Multi-view (board / list / timeline / Gantt) synchronisation, sprint engine, and dependency graph with cycle detection. Hierarchical page tree, full-text search over Markdown/AST, CRDT- or OT-style concurrent editing with versioning. Live timer engine with project switching, timesheet aggregation, and offline / cross-device reconciliation. Deterministic RPG state machine (HP/XP/streaks), economy balance, and asynchronous social quest consensus. Cross-calendar availability algorithm, timezone-aware conflict detection, and buffer / round-robin booking policies. Course/module engine, quiz grading with item analysis, and SCORM / xAPI content interoperability. Cart and order state machine, payment-gateway orchestration, and inventory consistency under concurrent checkout. Usage metering at scale, proration and mid-cycle plan changes, and idempotent invoicing with revenuerecognition safety.
and dynamic schema binding. Similarly, Web & Product Analytics and Business Intelligence both present dashboards, but the former mainly revolves around event collection and behavior telemetry under privacy constraints, whereas the latter mainly revolves around SQL / OLAP queries and chart rendering. Likewise, Identity & Access Management and Password & Secrets Management both belong to the security software stack, but the former focuses on protocol federation and session control, whereas the latter focuses on zero-knowledge secret storage and cross-device synchronization. At the higher-level grouping layer, these six macro families satisfy the principle of mutually exclusive and collectively exhaustive (MECE): each of the 30 categories belongs to one and only one family, while the six families together fully cover the entire task space of the benchmark without overlap. The result is a task space that is both market-grounded and low in redundancy, maximizing the diversity of core engineering primitives under a fixed benchmark budget. A.2
SaaS Domain Statistics
SaaSBench contains 30 tasks in total, spanning six macro domains: Customer & Growth (7 tasks), Productivity & Collaboration (6), Commerce & Finance (4), Data & Content Infrastructure (5), Security, Identity & Infra (4), and Domain & Workflow (4). As shown in Tables 5, 6, 7, 8, for each category we provide a brief definition, the core engineering challenge that motivates its inclusion in the benchmark, the corresponding commercial market and analyst taxonomy, and representative commercial products. Taken together, these tables make it clear that SaaSBench is not an arbitrary collection of tasks, but a benchmark grounded in a coherent commercial SaaS categorization framework. 15
Table 6: SaaSBench category definitions and engineering scope (Part II). Continuation of Table 5. #
Dom.
Category
16
CF
Accounting & Invoicing
17
CF
18
DCI
19
DCI
20
DCI
21
DCI
22
DCI
23
SI
24
SI
25
SI
26
SI
27
DW
28
DW
29
DW
30
DW
A.3
Definition
Core Engineering Challenge
SMB bookkeeping with chart of accounts, invoices, Double-entry ledger invariants, multi-currency conexpenses, tax and financial reports. version and tax computation, and audit-grade report generation. Inventory & Warehouse System that tracks stock levels, locations, move- Stock-movement ledger, multi-location transfer and ments, and BOMs across warehouses. BOM hierarchy, and barcode / label pipeline with scanner integration. Headless CMS API-first content platform where editors model con- Schema-to-API generation, media processing pipeline, tent types and consume them via REST/GraphQL draft/publish workflow, and webhook-driven content from any front end. delivery. Business Intelligence Tool for exploring data, authoring dashboards, and SQL authoring / semantic layer, SQL-→-chart rendersharing interactive visualisations across an organi- ing, and governed multi-source connector fleet. sation. Data Catalog & Lineage Central registry of datasets, schemas, owners, usage Metadata ingestion from heterogeneous sources, and column- level lineage across the data stack. lineage-graph construction, and impact-analysis queries. Feature Flag & Experimentation SDK + console for toggling features by segment Low-latency flag evaluation at the edge, targeting-rule and running A/B and feature-rollout experiments. engine, and statistical rigour of A/B analysis. Web & Product Analytics Lightweight tracker that ingests page/event data and High-cardinality event ingest, real-time aggregation, reports traffic, conversion and engagement metrics. and privacy- preserving (cookieless) attribution. Monitoring & Security Ops Dashboards, alerts and time-series exploration over Probe scheduling, time-series storage and dashboardmetrics, logs, traces and synthetic probes, often ing, alert-rule evaluation with notification routing, and doubling as the SOC/SecOps visualisation layer. log/event correlation for security operations. Identity & Access Management Centralised identity server that federates OID- Protocol coverage (OIDC/SAML/LDAP/SCIM), C/SAML/LDAP, enforces MFA, and manages users, MFA and session model, and tenant/realm isolation roles and realms. with delegated administration. Password & Secrets Manage- Encrypted vault for passwords and secrets with Zero-knowledge end-to-end encryption, client-side ment cross-device sync and organisation-level sharing. vault model, and secure cross-device sync / recovery. File Storage & Sync Self-hosted cloud drive with chunked upload, multi- Chunked / resumable upload, multi-device delta sync, device sync, sharing links, and collaboration apps. and external sharing with link-scoped access control. E-Signature & Contract Manage- Platform to prepare, route, sign and archive docu- Signing-order orchestration, tamper-evident hash ment ments with legal audit trail. chain, and audit- log evidence compliant with eIDAS / ESIGN. Low-Code / No-Code Visual builder that assembles internal business apps Visual page/component renderer, dynamic schema on top of existing databases and APIs. binding, and multi- source (DB/REST/GraphQL) data orchestration. Electronic Health Records Clinical record system for patient charts, encounters, FHIR / HL7 interoperability, ICD / CPT / SNOMED prescribing, and billing. coding, and e-prescribing with clinical-decision support. Workflow Automation Visual DAG engine that chains SaaS APIs into au- Visual DAG orchestration, connector ecosystem, and tomations with triggers, conditions and error han- durable execution with retries / idempotency. dling.
Task Statistics
In Table 9, we list the 30 final seed repositories that constitute SaaSBench. During the data curation stage, we first collect multiple candidate repositories for each category. However, under a fixed construction and annotation budget, only one seed repository is ultimately retained for each category and enters the subsequent pipeline of PRD writing, environment preparation, and evaluation construction. The retained repository must satisfy several requirements simultaneously: it should represent the typical system form of the category, exhibit signals of active maintenance, be practically buildable and deployable, and have business boundaries that are clearly aligned with the target category. Before being formally included in the benchmark, annotators also perform cold-start build validation and basic smoke testing on it. Stars denotes the approximate number of GitHub stars at the time of benchmark construction, rounded to the nearest thousand. In addition, as shown in Table 10, we further release the candidate repository pool for each category. Given sufficient budget, the number of tasks in SaaSBench can be further expanded.
B
Additional Experimental Details
B.1
Inference Configuration
For all agent-model configurations, we report pass@1, corresponding to a single rollout per task for each configuration. We do not perform oracle-style retries after task failure. Within each rollout, the agent receives an interaction budget of up to 500 reasoning–action steps (OpenHands max_iterations; Codex CLI and Claude Code follow the analogous internal limit of each tool) and a wall-clock budget of 10,800s for OpenHands, Codex CLI and Claude Code. These limits cap the total time that the 16
Table 7: Market grounding of the SaaSBench categories (Part I). Rows 1–15 list the commercial segment, representative analyst rankings, and flagship proprietary products of each category. #
Dom.
Category
1
CG
2
CG
3
CG
4
CG
5
CG
6
CG
7
CG
8
PC
9
PC
10
PC
11
PC
12
PC
13
PC
14
CF
15
CF
Email Marketing & Marketing Automation. Gartner MQ for B2B Marketing Automation; Forrester Wave for Email Marketing; G2 Grid for Email Marketing. Customer Relationship Manage- CRM / Sales Force Automation. Gartner MQ for Sales ment Force Automation; Forrester Wave for CRM Suites; IDC MS for CRM. Help Desk & Ticketing Customer Service & Support. Gartner MQ for the CRM Customer Engagement Center; Forrester Wave for Customer Service Solutions. Form Builder & Survey Online Survey & Form Building. G2 Grid for Survey / Online Form Builder; Forrester mentions in ExperienceManagement landscape. Community Forum Online Community Management. G2 Grid for Online Community Management; Forrester Wave for Community Platforms. Real-time Communication Team Collaboration & Messaging. Gartner MQ for Unified Communications as a Service (UCaaS); G2 Grid for Business Instant Messaging. Video Conferencing Meeting Solutions / UCaaS. Gartner MQ for Meeting Solutions; IDC MS for Worldwide UCaaS. Project Management & Issue Project & Portfolio Mgmt. / Agile Work Mgmt. Gartner Tracking MQ for Adaptive Project Mgmt. & Reporting; Forrester Wave for Enterprise Agile Planning Tools. Knowledge Base & Wiki Knowledge Mgmt. / Collaborative Docs. Gartner MQ for Insight Engines (adjacent); G2 Grid for Knowledge Mgmt. and for Note-Taking Software. Time Tracking Time Tracking & Professional Services Automation. G2 Grid for Time Tracking; Gartner MQ for PSA (adjacent). Gamified Productivity Habit Tracking / Personal Productivity. G2 Grid for Task Mgmt. (adjacent); niche leader in the Quantified-Self / habitapp landscape. Scheduling & Booking Online Appointment Scheduling. G2 Grid for Online Appointment Scheduling; Gartner mention in Digital Commerce Experience. Learning Management System Learning Management / Corporate LMS. Gartner MQ for Higher-Ed & Corporate LMS; Forrester Wave for Learning Platforms. E-commerce Platform Digital Commerce. Gartner MQ for Digital Commerce; Forrester Wave for B2B / B2C Commerce Solutions. Billing & Subscription Recurring Billing & Subscription Management. Gartner MQ for Recurring Billing; IDC MS for SaaS/Subscription Billing.
Market Segment & Rankings
Email & Newsletter
Representative Products Mailchimp (Intuit); HubSpot Marketing Hub.
Salesforce Sales Cloud; Microsoft Dynamics 365 Sales. Zendesk; Freshdesk (Freshworks).
Typeform; SurveyMonkey (Momentive).
Discourse (commercial cloud); Higher Logic Vanilla. Slack (Salesforce); Microsoft Teams.
Zoom Meetings; Cisco Webex. Atlassian Jira; monday.com Work OS.
Notion; Confluence (Atlassian).
Toggl Track; Harvest. Habitica (SaaS tier); Todoist (“karma” system as closest commercial analog). Calendly; Microsoft Bookings.
Canvas (Instructure); Blackboard Learn (Anthology). Shopify Plus; Adobe Commerce (Magento). Stripe Billing; Zuora Billing.
agent may spend on a single task. Single tool calls, such as npm install or prisma migrate, are allowed to run for up to 900s each before interruption, which empirically suffices for the most expensive installation steps in our benchmark. Network-level failures, including HTTP 429/529 errors, gateway overloads, and transient connection resets, are retried up to five times with exponential backoff between 10s and 60s. Once an agent rollout terminates, either through self-declared completion or budget exhaustion, no additional retries are attempted on the same task. B.2
Score Aggregation
For a single rollout, we sum the achieved scores across validation nodes and normalize the result by the task-specific total_maxScore, yielding a per-task score st ∈ [0, 1] for each task t. The reported P30 1 benchmark score for an agent-LLM configuration is the unweighted mean S = 30 t=1 st , scaled to the [0, 100] range. We also report a node-coverage rate, defined as the fraction of validation nodes that reach the PASSED state across all tasks. B.3
Evaluated Agents
We evaluate three coding-agent frameworks. They differ in how they structure planning, sandbox execution, and tool calls, but all are driven by a single underlying LLM. OpenHands. We use OpenHands [45] as our reference open-source agent framework and run its built-in CodeActAgent. OpenHands is launched in local runtime mode without nested Docker, so shell tool calls of the agent execute directly inside the host shell of our orchestrator and access the 17
Table 8: Market grounding of the SaaSBench categories (Part II). Continuation of Table 7. #
Dom.
Category
Market Segment & Rankings
16
CF
Accounting & Invoicing
17
CF
18
DCI
19
DCI
20
DCI
21
DCI
22
DCI
23
SI
24
SI
25
SI
26
SI
27
DW
28
DW
29
DW
30
DW
Small-Business Accounting. G2 Grid for Small-Business Accounting; Gartner Critical-Capabilities for Cloud Core Financials (adjacent). Inventory & Warehouse Inventory & Warehouse Management. Gartner MQ for Warehouse Management Systems; G2 Grid for Inventory Control. Headless CMS Content Management Systems (Headless). Gartner MQ for Digital Experience Platforms; Forrester Wave for Content Management Systems (Hybrid & Headless). Business Intelligence Analytics & Business Intelligence Platforms. Gartner MQ for ABI Platforms; Forrester Wave for Augmented BI Platforms. Data Catalog & Lineage Active Metadata Management / Data Catalogs. Gartner MQ for Metadata Management Solutions / Active Metadata; Forrester Wave for Machine-Learning Data Catalogs. Feature Flag & Experimentation Feature Management & Experimentation. Gartner Cool Vendor recognition in Software Engineering; G2 Grid for Feature Management. Web & Product Analytics Digital & Product Analytics. Gartner MQ for Digital Analytics; Forrester Wave for Digital Intelligence Platforms. Monitoring & Security Ops Observability / APM / Security Operations. Gartner MQ for Observability Platforms; Gartner MQ for SIEM (adjacent via SecOps dashboards). Identity & Access Management Access Management / Workforce IAM. Gartner MQ for Access Management; Forrester Wave for Customer Identity & Access Management. Password & Secrets Manage- Password Management / Enterprise Secrets. Gartner MQ ment for Privileged Access Management (adjacent); G2 Grid for Password Manager. File Storage & Sync Enterprise File Sync & Share (EFSS) / Content Collaboration. Gartner MQ for Content Collaboration Platforms; Forrester Wave for Content Platforms. E-Signature & Contract Manage- Electronic Signature / Contract Lifecycle Mgmt. Gartner ment MQ for Electronic Signature; Forrester Wave for CLM. Low-Code / No-Code Enterprise Low-Code Application Platforms (LCAP). Gartner MQ for Enterprise LCAP; Forrester Wave for LowCode Development Platforms. Electronic Health Records Ambulatory / Acute EHR. KLAS Research EHR rankings; Gartner Hype Cycle for U.S. Healthcare Payers & Providers. Workflow Automation iPaaS / Integration & Workflow Automation. Gartner MQ for Integration Platform as a Service (iPaaS); Forrester Wave for iPaaS.
Representative Products QuickBooks Online (Intuit); Xero.
NetSuite Inventory Mgmt. (Oracle); Fishbowl Inventory. Contentful; Sanity.
Tableau (Salesforce); Microsoft Power BI.
Alation Data Catalog; Collibra Data Intelligence Platform. LaunchDarkly; Optimizely Feature Experimentation. Google Analytics 4; Adobe Analytics. Datadog; Splunk Observability Cloud.
Okta Workforce Identity; Microsoft Entra ID (Azure AD). 1Password Business; Bitwarden (commercial tier). Dropbox Business; Box.
DocuSign; Adobe Acrobat Sign. Microsoft Power Apps; OutSystems.
Epic EpicCare; Oracle Cerner Millennium. Zapier; Workato.
task container via docker exec. For each task-model rollout, we instantiate an OpenHands TOML configuration with the following fields: max_iterations = 500, default_agent = "CodeActAgent", an [llm] block with temperature = 0, num_retries = 5, retry_min_wait = 10, and retry_max_wait = 60, and a [sandbox] block with timeout = 900. The MCP integration of the agent is disabled (enable_mcp = false); the agent uses only the native OpenHands tool set, including file read/write, shell, and IPython. The full reasoning trajectory of the agent is persisted to trajectory.json for post-hoc analysis. We use OpenHands version v1.6.0. Codex CLI.
We use OpenAI Codex CLI [37] through its single-shot execution mode:
codex exec –model <id> –json –sandbox danger-full-access –skip-git-repo-check – <prompt>
The danger-full-access sandbox is required because the task workspace is itself a Docker container. From the perspective of the harness, the container defines the sandbox boundary, and we therefore set the internal sandbox of Codex to behave as a no-operation layer. The –skip-git-repo-check flag avoids spurious failures on workspaces that are not Git-initialized. The internal step budget of Codex is governed by the same wall-clock budget of 10,800 s. A thin wrapper retries at most once on transient gateway errors, including high demand, Reconnecting, and stream disconnected, with a 60 s backoff. Standard output, the structured JSONL event stream, and standard error are written to disk for post-hoc analysis. We use Codex CLI version codex-cli 0.128.0. Claude Code.
We use Anthropic Claude Code CLI [1] in headless mode:
18
Table 9: SaaSBench task catalog (30 tasks). Each task corresponds to one SaaS category and is instantiated from one selected open-source seed repository. For each category, we curated multiple candidate repositories but retained a single representative seed repository for benchmark construction under a fixed annotation budget. We report the selected seed repository, its approximate GitHub Stars, primary language, backend framework, frontend stack, and main database engine. Repository URLs are shown inline for readability; stars are rounded to the nearest thousand at the time of benchmark construction. #
SaaS Category
Seed Repository
Stars
Language
Backend
Frontend
Database
1 2 3 4 5 6 7
Email & Newsletter Customer Relationship Management Help Desk & Ticketing Form Builder & Survey Community Forum Real-time Communication Video Conferencing
listmonk twenty chatwoot formbricks discourse mattermost jitsi-meet
19k 40k 28k 12k 41k 35k 29k
Go TypeScript Ruby TypeScript Ruby Go Java / JS
Go stdlib NestJS + GraphQL Rails Next.js API Rails Go stdlib Jitsi Videobridge
Vue React Vue React (Next) Ember.js React React
PostgreSQL PostgreSQL PostgreSQL PostgreSQL PostgreSQL PostgreSQL N/A (Stateless SFU)
8 9 10 11 12 13
Project Management & Issue Tracking Knowledge Base & Wiki Time Tracking Gamified Productivity Scheduling & Booking Learning Management System
plane outline kimai habitica cal.com canvas-lms
46k 32k 5k 12k 41k 5k
TypeScript TypeScript PHP JavaScript TypeScript Ruby
Django (DRF) Express (Node.js) Symfony Express (Node.js) Next.js API + tRPC Rails
React (Next) React SSR (Twig) Vue React (Next) SSR
PostgreSQL PostgreSQL MySQL / MariaDB MongoDB PostgreSQL MySQL / MariaDB
14 15 16 17
E-commerce Platform Billing & Subscription Accounting & Invoicing Inventory & Warehouse
medusa lago firefly-iii InvenTree
33k 10k 23k 7k
TypeScript Ruby PHP Python
Node.js / TS Rails Laravel Django (DRF)
React / Next.js React SSR / Vue React
PostgreSQL PostgreSQL MySQL / MariaDB PostgreSQL
18 19 20 21 22
Headless CMS Business Intelligence Data Catalog & Lineage Feature Flag & Experimentation Web & Product Analytics
payload superset datahub flagsmith plausible
42k 60k 12k 6k 24k
TypeScript Python Java Python Elixir
Next.js / Node.js Flask / Django-style Spring Django (DRF) Phoenix
React React React React React
PostgreSQL PostgreSQL MySQL / MariaDB PostgreSQL ClickHouse (+ PG)
23 24 25 26
Monitoring & Security Ops Identity & Access Management Password & Secrets Management File Storage & Sync
grafana keycloak vaultwarden nextcloud
66k 21k 34k 34k
Go Java Rust PHP
Go stdlib Quarkus Actix-web Native PHP (self)
React React External (BW clients) Vue
PostgreSQL PostgreSQL SQLite MySQL / MariaDB
27 28 29 30
E-Signature & Contract Management Low-Code / No-Code Electronic Health Records Workflow Automation
docuseal appsmith openemr n8n
11k 39k 5k 180k
Ruby TypeScript PHP TypeScript
Rails Spring (Java) + TS Native PHP (Laminas) Express (Node.js)
SSR React SSR Vue
PostgreSQL PostgreSQL (+Mongo) MySQL / MariaDB PostgreSQL
claude -p –output-format stream-json –verbose –model <id> –permission-mode bypassPermissions –allowedTools <tool-list> –add-dir <workspace> – <prompt>
-p/–print runs Claude Code non-interactively. –permission-mode bypassPermissions is the in-container analogue of danger-full-access in Codex: it removes interactive permission prompts that would otherwise block automation. We restrict Claude Code to a fixed set of 12 core tools — Bash, Edit, Write, Read, Glob, Grep, LS, WebFetch, NotebookEdit, NotebookRead, TodoRead, and TodoWrite — via –allowedTools to keep the request body small and reduce protocol-layer instability. As with Codex, transient gateway errors trigger at most one retry with a 60 s backoff, and the structured event stream is persisted to claude_events.jsonl. We use Claude Code CLI version @anthropic-ai/claude-code v2.1.126. LLM Backend Versions. Table 11 specifies the exact model snapshots used as agent backends. To support reproducibility, we record the full snapshot identifier of each vendor when available. we set the temperature to 0 where supported. For reasoning models that require non-zero sampling under their thinking modes (Claude Opus, Kimi, Qwen, DeepSeek, GLM, MiniMax, and GPT-5.4 with reasoning_effort=xhigh), we follow each vendor’s recommended temperature=1.0 and enable the corresponding thinking/reasoning channel. For Gemini, we additionally enable thought signatures 19
Table 10: Candidate repository pool for the 30 SaaSBench categories. Each row lists the curated open-source candidate repositories for one category; the repository ultimately selected as the benchmark seed is shown in bold. Rows follow the benchmark task order used throughout the paper. #
SaaS Category
Candidate Repositories
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
Email & Newsletter Customer Relationship Management Help Desk & Ticketing Form Builder & Survey Community Forum Real-time Communication Video Conferencing Project Management & Issue Tracking Knowledge Base & Wiki Time Tracking Gamified Productivity Scheduling & Booking Learning Management System E-commerce Platform Billing & Subscription Accounting & Invoicing Inventory & Warehouse Headless CMS Business Intelligence Data Catalog & Lineage Feature Flag & Experimentation Web & Product Analytics Monitoring & Security Ops Identity & Access Management Password & Secrets Management File Storage & Sync E-Signature & Contract Management Low-Code / No-Code Electronic Health Records Workflow Automation
listmonk; postal; mautic twenty; krayin; SuiteCRM; espocrm chatwoot; UVdesk; zammad; freescout formbricks; typebot; heyform discourse; forem; NodeBB mattermost; Rocket.Chat; zulip jitsi-meet; bigbluebutton; livekit plane; openproject; leantime outline; docmost; BookStack kimai; solidtime habitica cal.com; rallly; easyappointments canvas-lms; openedx; moodle medusa; saleor;bagisto; vendure; sylius lago ;killbill; firefly-iii;akaunting; invoiceninja InvenTree; grocy payload;strapi; directus superset; metabase; redash datahub; OpenMetadata; amundsen flagsmith; growthbook; unleash plausible; umami; matomo grafana; uptime-kuma; wazuh; graylog; gatus; oneuptime keycloak; authentik; zitadel; casdoor vaultwarden; bitwarden; passbolt nextcloud; seafile; owncloud docuseal; documenso; OpenSign appsmith; nocodb; ToolJet; budibase; nocobase openemr; openmrs n8n; activepieces; automatisch
Table 11: LLM backends evaluated as drivers of every agent framework. Vendor
Model (paper name)
API snapshot id
OpenAI Google Anthropic Moonshot AI Alibaba DeepSeek Zhipu AI MiniMax
GPT-5.4 Gemini 3.1 Pro Claude Opus 4.7 Kimi K2.6 Qwen 3.6 Plus DeepSeek V4 Pro GLM 5.1 MiniMax M2.7
gpt-5.4-2026-03-05-xhigh gemini-3.1-p claude-opus-4-7 kimi-k2-6 qwen3.6-plus deepseek-v4-pro glm-5-1-260408 Minimax-M2.7
(thinking.include_thoughts = true, thinking.budget_tokens = 8192) and increase max_output_tokens to 65,536, which empirically prevents premature truncation on long tool-call chains. Comparison with the Original IDE-Integrated Experience. We include a pragmatic note on faithfulness. All three frameworks, particularly Codex CLI and Claude Code, are routinely used inside an interactive IDE, where a human user steers the trajectory, supplies missing context, and rejects bad tool calls. Our evaluation deliberately disables this human-in-the-loop channel: the agent receives the PRD and KB once, runs autonomously inside the container until it self-declares completion or exhausts its budget, and is scored on the final container state. We expect this setting to provide a strict lower bound on the score that the same framework would achieve in real interactive use. This design isolates the autonomous engineering capability of the agent from the steering capability of the human user, which is the quantity that SaaSBench aims to measure. 20
B.4
Metric Detail
Per-node Scoring Rules. Let a validation node v have a primitive chain ⟨p1 , p2 , . . . , pk ⟩ executed in order, where each primitive pi returns a Boolean success flag ⊮[pi passes]. Let Mv ∈ R>0 denote the pre-declared maxScore of the node, and let method(v) ∈ {binary, weighted, llm-as-judge} denote its scoring method. The achieved score sv is computed as follows. (i) Binary nodes. These nodes are used for security-critical assertions where any failure invalidates the entire claim, such as deployment health, RBAC denials, and authentication checks: ( Qk Mv if i=1 ⊮[pi passes] = 1, binary sv = 0 otherwise. For efficiency, primitive execution is short-circuited at the first failure within a binary chain. (ii) Weighted nodes. These nodes are used for multi-step CRUD workflows and coverage checks where partial completion should receive partial credit: k j X k sweighted = k1 ⊮[pi passes] · Mv , v 0.1
i=1
where ⌊·⌋0.1 denotes rounding to one decimal place. Unlike binary nodes, weighted chains run to completion, and the achieved score scales with the fraction of primitives that pass. (iii) LLM-as-judge nodes. These nodes are used only when deterministic primitives cannot adequately characterize the target, such as page-layout reasonableness or architectural-organization quality. Each such node materializes a single P17 primitive that bundles a rubric prompt, a per-node max_score, and an evidence source, such as the workspace codebase, the last HTTP response body, or a rendered page screenshot/HTML. The judge model is asked to return a JSON object containing a numerical score and free-form reasoning; we then clip the parsed score into the legal range: sllm-judge = clip( Judge(rubricv , evidencev ), 0, Mv ) . v The judge is invoked with temperature = 0 to maximize reproducibility. Concrete rubric and judge prompts are listed verbatim in Appendix C. Status Taxonomy and Dependency Gating. Each node terminates in one of six statuses: PASSED (score > 0 on a chain that ran cleanly), FAILED (the chain ran but produced score 0), ERROR (the chain raised an unhandled exception and is treated as FAILED for scoring), SKIPPED_DEPENDENCY (some prerequisite of v is not in PASSED, so v is not executed), SKIPPED_LLM (the judge API failed or LLM judging was disabled for this run), and DRY_RUN (an administrative skip used during evaluator authoring). Dependency-induced skips are assigned sv = 0, but their Mv remains in the denominator of the task score. This design makes foundational failures cascade fairly: if an agent never starts the application, every dependent node is correctly counted as 0/Mv rather than silently dropped. At the same time, the failure is attributed to one root cause, namely the unmet prerequisite, rather than being re-charged at every downstream node, because the harness only executes the prerequisite chain once. By contrast, SKIPPED_LLM nodes are removed from both the numerator and the denominator, so that a transient judge-API outage cannot artificially deflate the score of an agent. The harness logs the dropped maxScore so that reviewers can inspect how much of the rubric pool was excluded. Aggregation: Node → Task → Benchmark. Let Vt denote the node set of task t, and let Vt† = Vt \ {v : status(v) = SKIPPED_LLM} denote the LLM-pruned set. The per-task score, scaled to [0, 100], is P P v∈Vt† sv v∈Vt† ,method(v)̸=llm-judge sv non-llm St = 100 · P , St = 100 · P , v∈V † Mv v∈V † ,method(v)̸=llm-judge Mv t
t
where the second formula gives the deterministic-only subscore that we additionally report for transparency. The benchmark-level score for an (agent, model) configuration is the unweighted mean across all 30 tasks, 30 X 1 S = 30 St . t=1
21
We prefer the mean over a max-pooled sum because the tasks deliberately span a wide range of total_maxScore values, from 174 to 1,489 in the present benchmark. An unweighted mean of normalized per-task scores prevents oversized DAGs from dominating the headline number. Per-category and Per-trajectory Diagnostics. In addition to the headline score S, the harness emits per-category aggregates, such as Authentication, RBAC, Frontend, and Deployment, computed with the same numerator/denominator formula restricted to the relevant nodes. It also emits per-trajectory subscores for tasks that declare named user trajectories, such as happy_path and advanced_workflows. B.5
Primitive Taxonomy of DAG Validation Nodes
The primitive chain inside each node is composed from a fixed library of primitives. Table 12 lists the primitives that appear in at least one node across the 30 tasks, together with the number of observed node-level invocations out of 18,196 primitive calls in total. Primitive identifiers are intentionally short codes (P01–P29, together with the browser-related RENDER_DOM and SCREENSHOT primitives and the analytics-specific P_INGEST primitive) so that DAG JSON files remain compact and human-auditable. Table 12: DAG validation primitives, grouped by purpose. Counts are the total number of times the primitive is invoked across the all task DAGs. One node may invoke several primitives in sequence.
B.6
Group
Code
Purpose
Calls
File / artefact
P01 P02 P03
file existence file content match (regex / substring) file count under a path
90 143 48
HTTP / API
P04 P05 P06 P07
HTTP request end-to-end CRUD round-trip JSON schema match JSON value assert (path-based)
3,941 47 123 2,595
Database
P08 P09 P10 P11
raw SQL / NoSQL query table existence column type / nullability check index existence
1,143 533 506 33
Container / runtime
P12 P21 P23
exec inside container log content check file upload / download
Auth / RBAC
P13 P14 P15 P16
authenticated login (C RITICAL) permission / role gate check HTTP status-code assert response-time check
3,453 277 3,329 4
Browser / UI
P18 RENDER_DOM SCREENSHOT
browser interaction (form, click) full DOM rendering screenshot capture
68 54 26
LLM judge
P17
rubric-based LLM scoring
402
Domain-specific
P_INGEST P19 / P22 / . . . P25 P27
event ingestion (analytics tasks) reserved / per-task misc. asserts webhook delivery probe
72 — 17 14
438 15 9
Six Evaluation Backbones and Category Mapping
Each validation node in SaaSBench is annotated by the task author with a fine-grained category string. Across the all tasks, this annotation yields 289 distinct categories, ranging from broadly applicable concepts such as DataModel (644 nodes) and RBAC (482 nodes) to highly task-specific concepts such as BusinessLogic_DoubleEntry and XMPPSignaling. Reporting per-category scores at this level of granularity would be difficult to read and unsuitable for cross-task comparison. We therefore define a deterministic and exhaustive mapping from these 289 fine-grained categories into six high-level evaluation backbones. 22
Backbone Definitions. The six backbones are designed so that each captures an orthogonal axis of an enterprise SaaS system that an autonomous coding agent must implement correctly. • Deploy — the runnable artifact starts cleanly inside its container, including dependency installation, schema migration, environment configuration, fixture seeding, process supervision, and health checks. Source categories include Deployment, Setup, Build, Configuration, Maintenance, Teardown, TestFixture, BackgroundJobs, and CLI. • Data — persistent state is correctly modeled and reachable, including tables, columns, indices, foreign keys, datasource wiring, file/object storage, import/export, and event-stream ingestion. Source categories include DataModel*, Datasource, DatabaseDiscovery, Metadata, Cache*, FileSystem, MediaManagement, Upload, ImportExport, IngestionCLI, EventProcessing, RLS, and Lineage. • API — the system implements correct protocols, including RESTful CRUD, GraphQL/tRPC, WebSocket/realtime communication, outbound webhooks, search, notification fan-out, and OpenAPI conformance. Source categories include API* and all API_* variants, such as API_GraphQL, API_FHIR, and API_v2_EE; CRUD/*CRUD; GraphQL*; tRPC; WebSocket; Webhook*; Realtime/NotificationRealtime; Search*; Action*; ChatSystem; TransactionalEmail; and Validation. • Logic — domain-specific business behavior built on top of the data and API layers, including orders, billing, subscription lifecycle, gamification, moderation, conferencing flows, dashboards, plugin systems, sharing/collaboration, and content templates. Source categories include all BusinessLogic*; Workflow; Cron*; Async*; the e-commerce and billing families (Order, Cart, Subscription, Payment, Tax, Inventory, InvoiceLifecycle, ChargeModels, . . . ); the gamification family (BadgeSystem, QuestSystem, PetMount, . . . ); the conferencing family (Recording, BreakoutRooms, Whiteboard, ConferenceFlow, . . . ); and broad-coverage building blocks (ModerationReview, TrustLevel, TemplateManagement, Plugin, HookSystem, Sharing, . . . ). • AuthZ — authentication, authorization, and security auditing, covering user identity, permitted actions, and how these decisions are recorded. Source categories include Authentication, Authorization*, RBAC, AccessControl, Permission, AuditLog, AuditCompliance, PasswordPolicy, BruteForceDetection, APIKey, Security*, OAuth*, OIDCProtocol, SAMLProtocol, TokenExchange, IdentityProvider, KeyManagement, ProtocolMappers, UserManagement, BusinessLogic_2FA, and BusinessLogic_Identity. • Quality — non-functional and presentation aspects that distinguish a working prototype from a production-ready system, including code architecture, frontend rendering, edge-case robustness, error handling, internationalization, and administrative user experience. Source categories include ArchitectureQuality, Architecture, Frontend*, UI*, EdgeCases, ErrorHandling, AdminPanel, AdminUI, Internationalization, Localization, and ConfigAndAdmin. Mapping Algorithm. The mapping from a fine-grained category c to a backbone B(c) is fully deterministic and consists of a hand-curated exact-match dictionary plus four prefix rules applied in order. We release the mapping as a single category_to_backbone.json file alongside the benchmark, so any third party can re-derive the per-backbone scores from raw node-level reports. The procedure is: 1. If c is in the exact-match dictionary, return the assigned backbone. 2. Otherwise, scan the prefix rules in order and return the first match: API*/Api* → API, BusinessLogic* → Logic, DataModel* → Data, Architecture*/Frontend*/UI* → Quality, Auth* → AuthZ. 3. No category falls through; the algorithm is verified to cover all 5,370 nodes (Table 13). Resulting Distribution. Table 13 shows how the 5,370 validation nodes and the total maxScore of 17,299.1 are distributed across the six backbones under this mapping. The distribution is intentionally non-uniform: Logic dominates by maximum score (27.2%) because business behavior is the primary factor that distinguishes a real SaaS product from a generic web application, whereas Deploy is the lightest backbone (3.7% of maxScore) because each task requires only a small number of nodes to certify that the service is running. We do not reweight backbones in the headline Pass@1 score. The 23
Table 13: Distribution of the 5,370 validation nodes and total maxScore across the six evaluation backbones, after applying the deterministic mapping in Appendix B.6. Backbone
#Nodes
% Nodes
Total maxScore
% maxScore
Deploy Data API Logic AuthZ Quality
289 854 1216 1179 1023 809
5.4% 15.9% 22.6% 22.0% 19.1% 15.1%
640.8 1901.0 3253.3 4698.0 3239.5 3566.6
3.7% 11.0% 18.8% 27.2% 18.7% 20.6%
Total
5370
100.0%
17299.1
100.0%
per-backbone scores in Table 3 are computed independently within each backbone, so an agent that is strong on Logic but weak on Deploy is visible as such. B.7
Failure-Mode Taxonomy
This appendix provides detailed definitions for the five execution-trajectory types used in Section 5.4. The goal of this taxonomy is not to rank capability units by aggregate score, but to identify where the agent’s development process breaks down. We classify each capability unit using its execution trace and node-level failure profile over six capability backbones: deployment, data, API, business logic, authorization, and quality. The five types are ordered from the strongest engineering trajectory (T1) to the earliest and most severe breakdown (T5). T1: Disciplined end-to-end execution. T1 denotes an idealized trajectory in which the agent proceeds through the full SaaS development pipeline in a disciplined order: deployment, data schema, authentication, business routes, authorization policy, and quality checks. At each stage, the agent verifies the current layer before building downstream components. A T1 unit therefore represents a stable, reproducible, and well-validated implementation. No capability unit in our 480-unit sample falls into this category. T2: Single-backbone bottleneck. T2 denotes an otherwise functional system with one dominant capability bottleneck. The stack is runnable, core data and authentication layers are largely in place, and most business workflows are implemented, but one capability backbone exhibits a localized failure. Examples include missing a specific security requirement, omitting a quality constraint, or failing a single policy-related check. This type corresponds to the common “weakest link” explanation, but it accounts for only 0.6% of our units. T3: Runnable but shallow business logic. T3 denotes a system whose infrastructure and main execution path are working, but whose business semantics remain incomplete. The agent can usually start the application, create the main schema, expose endpoints, and satisfy simple happy paths. However, it fails to operationalize detailed SaaS requirements such as edge cases, quotas, trust levels, error handling, workflow constraints, and policy rules. In this type, the bottleneck has moved beyond setup into incomplete product behavior. T4: Superficially reachable but structurally incomplete. T4 denotes a system that appears reachable from the outside but lacks reliable foundations. For example, the HTTP entry point may return a response, while migrations, schema constraints, authentication bootstrap, session handling, or RBAC prerequisites remain incomplete. Downstream API and business-logic failures then arise because they are built on an unstable base. This type captures the case where the agent has made the project look runnable, but has not completed the underlying engineering setup. T5: Non-runnable or unstable stack. T5 denotes the earliest and most severe failure mode. The generated system never becomes reliably runnable, or it becomes unstable under basic probes. Typical causes include dependency conflicts, incorrect Docker or service configuration, missing health checks, broken migrations, wrong working directories, port or volume errors, and premature claims of success before the stack is actually running. This type accounts for 63.5% of all capability units, making it the dominant failure mode on SaaSBench. 24
C
Full Prompts
We provide the full prompt templates used in SaaSBench. These templates cover two core procedures: first, prompting the agent to complete each task; second, eliciting rubric-based scores from the judge model at llm-as-judge validation nodes. All templates follow the exact concatenation logic used in our public release. C.1
Agent Task Prompt
For each (task, agent, model) configuration, the harness constructs a single textual prompt by concatenating four blocks. Square-bracketed placeholders are populated on a per-task basis, while all other text remains unchanged across the 30 tasks. Block 1 — anti-cheat banner Mandatory anti-cheat policy. You MUST implement the platform from scratch within this Docker environment. Cloning, copying, or otherwise importing any pre-existing open-source codebase (via git clone, wget, curl, container image extraction, package downloads of unrelated projects, etc.) is strictly forbidden and will be detected by the harness. Trajectories that fetch external source repositories receive a score of 0 regardless of the resulting test outcomes. Block 2 — role and environment specification You are a senior full-stack engineer. Your task is to build a complete [task-specific product description] from scratch inside an already-running Docker environment. Environment. The Docker environment is already up and running — you do not need to pull images or start containers. You work inside the container [container-name], with /app as the working directory (currently empty). Run commands inside the container like this: docker exec [container-name] bash -c "your command". Tools pre-installed in the container (Ubuntu 22.04): [per-task list of language runtimes, package managers, native libraries, database clients]. Database ([engine name and version], already running): host db, port [port], database [db], username [user], password [pwd], extensions [ext-list]. The application MUST listen on port [port]. What you need to do: (1) create a complete [stack name] project inside /app; (2) install dependencies; (3) run database migrations; (4) create the following 3 evaluation users (admin / moderator / regular user) with the credentials specified in the PRD; (5) start the application server, listening on 0.0.0.0:[port]. Key technical requirements. [per-task framework version, ORM choice, cache, job queue, auth header conventions, etc.] Block 3 — requirements document and KB pointer ### Requirements Document The complete product requirements document has been placed in the working directory at /app/task.md. Read it carefully and implement everything it describes. In addition, the working directory contains a supplementary knowledge base at /app/knowledge_base.json that clarifies ambiguous points in the PRD. Each entry has the shape {id, question, answer, source_reference, confidence}. Consult it as a reference when task.md is unclear on details such as field semantics, state transitions, serialization, filtering, or permissions. You can read it on demand with jq, cat, or grep (e.g.
25
jq ’.clarifications[] id, question, answer’ /app/knowledge_base.json|); you do not have to read the whole file at once. Block 4 — success condition ### Success Condition The application server must be reachable on the designated port by the time you consider the task done. Anything that needs to happen before that — dependency install, schema migration, fixture creation, process supervision — is part of what is being evaluated; the harness will not enumerate the steps for you. The PRD and KB are not embedded directly in the prompt body. Instead, they are placed in the workspace as files, namely /app/task.md and /app/knowledge_base.json. This design allows the agent to re-read them on demand without unnecessarily expanding the rolling context window. Since PRDs in SaaSBench contain 4,363 lines on average, embedding them in every model turn would otherwise consume the entire context budget. C.2
LLM-as-Judge Rubric Prompt
For each llm-as-judge node, the P17 primitive collects a node-specific rubric together with a piece of evidence, such as a workspace codebase listing, the last HTTP response body, or a rendered page screenshot/HTML. It then issues a single chat-completion call to the judge model, Claude Sonnet 4.5, with temperature = 0. The two-message template is defined as follows. System message You are an expert evaluator. Score the evidence against the rubric. Respond ONLY with a JSON object: {"score": <0-{max_score}>, "reasoning": "<brief explanation>"} User message ## Rubric {rubric_prompt} ## Evidence {evidence} The reply from the judge is parsed as a JSON object. The integer score is clipped into [0, Mv ] to guard against rare cases in which the judge ignores the requested upper bound. Markdown code fences in the reply are stripped before json.loads is applied. If parsing fails or the upstream API is unavailable, the node is marked as SKIPPED_LLM and excluded from both the numerator and the denominator of the per-task score, as detailed in Appendix B.4. Concrete rubric example. For illustration, we include the actual rubric used for the FE_HOMEPAGE_LAYOUT node of the Community Forum task. The task identifier is task_aoiwqoiq, the maximum score is 6, and the evidence source is the rendered homepage screenshot together with HTML. Concrete rubric example Evaluate the homepage layout quality: (1) Does a topic list display with titles, authors, timestamps? (2) Is there a navigation bar with logo, search, login/signup? (3) Is there a category sidebar or navigation? (4) Are visual hierarchy and spacing professional? Score 0–6. The remaining 401 P17 invocations across the 30 tasks follow the same rubric structure: a short numbered checklist of 3–6 observable criteria paired with an explicit score range. Together, these 26
rubrics form a consistent and audit-friendly contract between the benchmark authors and the judge model.
D
Simplified Task Example
This section uses a concrete task, Discourse, to illustrate two core artifacts of a SaaSBench instance. The following two simplified snippets show how a single SaaSBench task is instantiated in practice. Simplified PRD ### Task: Build a Full-Featured Community Forum Platform Product Background. The task asks the agent to build a production-ready community discussion platform for large-scale, structured conversations. The system supports threaded topics, posts, categories, tags, badges, private messages, notifications, full-text search, moderation workflows, and an extensible plugin architecture. It also includes an administrative dashboard for site settings, users, groups, email, backups, themes, API keys, webhooks, reports, and operational monitoring. Technical Stack. The required backend stack is Ruby 3.4 with Ruby on Rails 8.0, PostgreSQL 13+, ActiveRecord, Redis, Sidekiq, mini_scheduler, and MessageBus-based real-time updates. The frontend is implemented with Ember.js and Glimmer components, using pnpm and Node.js 20+. Search is implemented with PostgreSQL full-text search, and file storage must support local storage or S3-compatible object storage. API Conventions. The platform does not use a global /api/ prefix. JSON responses are served through the same Rails controllers as HTML, selected by either a .json suffix or the Accept: application/json header. Authentication supports browser sessions, global API keys through Api-Key and Api-Username headers, and per-user API keys through User-Api-Key. Cookie-authenticated mutating requests require CSRF protection, and clients obtain a token from GET /session/csrf. Errors follow a structured JSON shape with errors and error_type. Data Model. The PRD specifies a large PostgreSQL schema with over one hundred domain entities, including users, posts, topics, categories, groups, tags, badges, notifications, uploads, bookmarks, invites, reviewables, API keys, drafts, topic timers, post actions, topic users, group memberships, user emails, post revisions, themes, color schemes, webhooks, polls, search data, user security keys, and many administrative or audit-related tables. The schema uses ActiveRecord conventions, foreign keys, indexed lookup columns, soft-deletion fields, polymorphic associations, and domain-specific enums. Core Business Workflows. The forum implements a progressive trust-level system from TL0 to TL4. Trust levels control posting, private messages, uploads, likes, edits, moderation-like privileges, and topic management. Promotions are automatic for TL1–TL3 based on reading activity, visits, posts, replies, likes, and flag history, while TL4 is manually granted. The authorization layer is modeled as a per-request Guardian permission system that checks category visibility, topic creation, post editing, post deletion, staff actions, category moderation, and user management permissions. Post and Topic Lifecycle. Post creation is handled by a service pipeline that validates spam rules, slow mode, rate limits, PM recipient limits, and category permissions before creating posts and topics. Raw Markdown is converted into sanitized HTML through a markdown-it and server-side sanitization pipeline. New topics and posts update user statistics, topic counters, tracking states, search data, and MessageBus channels. Topic lifecycle behavior includes closing, archiving, pinning, unlisting, private messages, topic timers, auto-close behavior, post moving, revisions, and whisper posts for staff-only discussions. Moderation, Search, and Notifications. The system includes flag queues, reviewable
27
objects, review scores, threshold actions, silence and suspension workflows, watched words, spam handling, screened emails/IPs/URLs, user warnings, and staff action logs. Notifications cover replies, mentions, quotes, likes, private messages, high-priority events, muting rules, watched topics, and first-post watching. Search supports PostgreSQL full-text search, ranked results, category and tag filters, user filters, status filters, date filters, and advanced query operators. Frontend Requirements. The Ember.js frontend must expose topic list views, category-scoped and tag-scoped listings, topic detail pages, user profiles, user preferences, admin dashboard pages, search pages, review queues, group pages, tag pages, private-message views, badges, bookmark manager, static content pages, safe-mode diagnostics, setup wizard, authentication pages, and error or permission-boundary pages. Interactive components include the topic composer, Markdown preview, uploads, emoji picker, mentions, hashtag autocomplete, notification menus, admin editors, filters, pagination, and responsive layouts. User and Permission System. Authentication includes browser session cookies, global API keys, user API keys, OAuth/OmniAuth, SSO, passkeys, two-factor authentication, and passwordless email login. Roles include admin, moderator, staff, and trust-level users. Permissions combine role checks, trust levels, category permissions, ownership, staff status, group membership, rate limits, and visibility rules. Restricted categories and private messages require explicit user or group authorization. Deployment and Runtime Contract. The generated system must run in the provided Docker environment with Rails, PostgreSQL, Redis, Sidekiq, Node.js, pnpm, and the Ember build pipeline. The application must expose the configured HTTP port, initialize the database, run migrations and seed data, start background workers, compile frontend assets, and provide a health-checkable web service. Simplified local DAG fragment example ### Core Forum Workflow Fragment Node 1: DEPLOY_HEALTH. Objective: verify that the generated forum application is reachable on localhost:8020. Actions: send GET /. Validations: the returned status code belongs to the accepted healthy-status set. Node 2: AUTH_SESSION_LOGIN. Prerequisites: AUTH_CSRF_TOKEN, AUTH_CREATE_ADMIN. Objective: verify browser-session login. Actions: fetch a CSRF token from /session/csrf.json, then submit administrator credentials to /session. Validations: the response status is 200 and the returned user is eval_admin. Node 3: CRUD_TOPIC_CREATE. Prerequisite: AUTH_CREATE_ADMIN. Objective: verify topic creation. Actions: submit POST /posts.json with a title, body, and category. Validations: the response contains a topic_id, the first post has post_number = 1, and the topic row exists in PostgreSQL. Node 4: CRUD_POST_CREATE. Prerequisite: CRUD_TOPIC_CREATE. Objective: verify reply creation. Actions: create a reply under the newly created topic. Validations: the returned post has post_number = 2, and the topic’s posts_count is updated to 2 in the database. Node 5: CRUD_CATEGORY_CREATE. Prerequisite: AUTH_CREATE_ADMIN. Objective: verify category creation. Actions: submit POST /categories.json with category name and colors. Validations: the response returns the created category name, and the category row exists in PostgreSQL. Node 6: SEARCH_BASIC. Prerequisite: CRUD_TOPIC_CREATE. Objective: verify fulltext search. Actions: create a topic containing the unique keyword SearchXyz42, wait for
28
indexing, and query /search.json?q=SearchXyz42. Validations: search results exist and the result blurb contains the keyword. Dependency structure. AUTH_CREATE_ADMIN → CRUD_TOPIC_CREATE → {CRUD_POST_CREATE, SEARCH_BASIC} AUTH_CSRF_TOKEN, AUTH_CREATE_ADMIN → AUTH_SESSION_LOGIN AUTH_CREATE_ADMIN → CRUD_CATEGORY_CREATE
E
Broader Impact
SaaSBench aims to support more realistic and diagnostic evaluation of long-horizon coding agents. By focusing on enterprise-level SaaS development scenarios, the benchmark can help researchers better understand whether current agents can move beyond localized code generation toward endto-end system construction, deployment, and validation. We hope that SaaSBench can promote the development of coding agents that are more reliable, transparent, and better aligned with practical software engineering needs. At the same time, more capable coding agents may lower the barrier to software creation in both beneficial and harmful ways. They can improve developer productivity, broaden access to software development, and help non-expert users rapidly prototype useful systems. However, they may also generate insecure code, propagate hidden defects, or be misused to automate harmful software behavior.
F
Limitations and Future Work
Limitations. Real-world software development is highly diverse and continues to evolve with changes in organizational contexts, deployment practices, engineering conventions, and product requirements. Therefore, SaaSBench may not cover all possible variants of enterprise software construction, nor can it exhaust all qualitative factors involved in engineering decisions. These factors do not diminish the value of the benchmark. Rather, they indicate that SaaSBench still has room for further refinement as coding agents and software engineering workflows continue to develop. Future Work. Future work can further extend SaaSBench in two directions. First, it can increase the number of tasks, expand the set of SaaS categories, and cover a broader range of technology stacks to improve the representativeness of the benchmark. Second, future versions can support more dynamic software development scenarios, such as iterative requirement updates, system maintenance tasks, and multi-stage product evolution.
29