AgentScript-Eval: Benchmarking LLMs and Agents for Code Generation in an Enterprise DSL

AgentScript-Eval: Benchmarking LLMs and Agents for Code Generation in an Enterprise DSL

arXiv:2611.13183v1 [cs.SE] 9 Nov 2026

Dipin Khati1,2∗ Shubham Mehrotra2 Bin Bi2 Zhou Yu2 Denys Poshyvanyk1 James Zhu2 Sitaram Asur2 Phil L. Mui2 1 William & Mary 2 Salesforce dkhati@wm.edu

Abstract Domain-specific languages (DSLs) are widely used in enterprise software, giving a compact interface to complex tasks, from data access to infrastructure automation. Platforms increasingly extend them to agents, where a developer declares the topics, permitted actions, and guardrails of an agent rather than writing generalpurpose code. Large language models (LLMs) write general-purpose code well, but that ability has been measured almost entirely on languages abundant in public pretraining data, and enterprise DSLs are neither abundant nor public. We introduce AgentScript-Eval, an execution-graded benchmark of 100 real tasks for AgentScript, the declarative DSL that Salesforce uses in Agentforce to build conversational agents. Tasks span authoring an agent from a specification, debugging a broken definition, and optimizing a working one. Evaluating five models zero-shot, we find that the regime is hard: the best model reaches only 25.0% pass@3, and every model scores 0% on authoring from scratch. We further find that model scale and price are weak predictors of success—a 31B open-weight model attains the highest compilation rate of any model tested (48.0%) and matches the strongest frontier model on partial-credit accuracy at the lowest token cost. Adding an agentic compile-and-edit loop is the single largest lever, roughly doubling partial-credit accuracy. Beyond that gain, the wording of domain guidance matters more than its packaging. The same skill pack scores 28% as plain documentation but only 14% as a live plugin. Rewriting one skill inside that plugin then raises it to 37%, our best result, with catalog and tooling untouched. A per-category breakdown shows the difficulty is concentrated in authoring from scratch, where no zero-shot model succeeds, and even the best agentic configuration stays under 17%. Larger models did not reliably outperform smaller ones; what mattered was the harness surrounding the model and the guidance supplied to it.

1

Introduction

Code generation is among the most extensively benchmarked capabilities of LLMs, and by prevailing measures it appears close to saturation: frontier systems exceed 90% on HumanEval-style function synthesis [Chen et al., 2021, EvalPlus Team, 2025] and, embedded in agentic harnesses, resolve a substantial share of real GitHub issues [Jimenez et al., 2023, Yang et al., 2024, SWE-bench Team, 2025]. These results rest on an assumption that is rarely made explicit: that the target language is well represented in pretraining data. Python and Java dominate existing benchmarks in part because they are also among the languages most abundantly represented in the public code corpora on which these models are trained. Thus, benchmark performance may overstate the generality of LLM code-generation capabilities across programming languages. ∗ Work done during an internship at Salesforce.

40th Conference on Neural Information Processing Systems (NeurIPS 2026). Workshop: 2nd Workshop on Agentic AI Benchmarks and Applications for Enterprise Tasks (AABA4ET), NeurIPS 2026.

This limitation is particularly consequential in enterprise software, where substantial portions of production systems are built using domain-specific and declarative languages rather than generalpurpose programming languages. Conversational agents, through which customers and employees interact with organizations, are authored on major enterprise platforms through declarative surfaces rather than general-purpose code [Microsoft, 2025, Google Cloud, 2025, Amazon Web Services, 2025]. These surfaces let developers specify an agent’s behavior, including what it may discuss, which actions it may invoke, and which guardrails constrain it, while the platform handles execution. Such DSLs deliberately constrain the programming interface to expose abstractions tailored to a particular domain [Mernik et al., 2005]. Precisely because these languages are specialized and closely tied to particular enterprise platforms, they are also among the settings in which least is known about LLM code generation. They are scarce or absent from public pretraining corpora, evolve alongside their underlying platforms rather than through broadly adopted standards, and lack established public benchmarks. Existing DSL research offers limited coverage of this setting, targeting languages that are themselves public and well represented, such as SQL [Yu et al., 2018] and IT-automation-YAML [Pujar et al., 2023], while studies of resource scarcity vary the language only across general-purpose targets [Cassano et al., 2022]. We are aware of no publicly executed benchmark for an enterprise-specific DSL used in active production. Whether the competence documented above transfers to such languages, and what can restore it when it does not, therefore remains an open question. We address this question in the context of one such language. AgentScript is a declarative DSL used in Salesforce Agentforce to author conversational agents [Salesforce, 2025]. We introduce AgentScript-Eval, a benchmark of 100 tasks drawn from real agent-development work and spanning the developer lifecycle: authoring an agent from a specification, repairing a broken agent, and optimizing a working one. Each task is graded by compiling the generated program and executing it against multi-turn acceptance checks, distinguishing programs that parse but misbehave from those that fail to compile. We evaluate five models across seven generation configurations, ranging from single-shot generation to agentic loops with and without domain-skill-packs. Our contributions are threefold. First, we introduce AgentScript-Eval, to our knowledge the first verifiable benchmark for enterprise-specific, in-production agent-authoring DSL (Section 2). Second, we provide a controlled comparison across models and generation configurations that separates compilation from behavioral correctness and reports token cost per task (Section 3). Third, we identify three findings that complicate the extrapolation of general-purpose coding. The regime is substantially harder than the existing results would suggest: the strongest model achieves only 25.0% pass@3, and no model solves an authoring task zero-shot. Scale and price are weak predictors: a 31B open-weight model compiles more programs than any hosted model we evaluate, at the lowest cost per task. Finally, once a model can compile and edit its own output, the wording of domain guidance matters more than its packaging: rewriting a single skill within an otherwise unchanged plugin yields a +23 point improvement and our strongest configuration at 37%.

2

Method

AgentScript. AgentScript is a declarative DSL for authoring conversational agents in Salesforce Agentforce [Salesforce, 2025]: a program specifies the agent’s topics, the actions it may invoke with their inputs and outputs, and guardrails on behavior (Appendix A). Actively developed and absent from public corpora, it demands both valid syntax and behavior matching a specification. Tasks. AgentScript-Eval contains 100 real agent-development tasks: authoring (30) produces an agent from a natural-language specification, debugging (40) repairs an uncompiling AgentScript, and optimization (30) improves a working one against a stated objective. Authoring is most demanding, since debugging and optimization supply an existing program as scaffold. Each task ships with an input context and executable checks; the defect taxonomy and construction details are in Appendix B. Grading. Text-similarity metrics correlate poorly with functional correctness for code [Evtikhiev et al., 2022, Liu et al., 2023], so we grade by compilation and execution. If the generated agent compiles, it is executed as an agent against the task’s acceptance checks, which encode expected actions, state, and responses over multi-turn dialogues with stubbed backends. 2

execution feedback (agentic)

Task

Generate

Compile

Execute

Score

spec + program

direct or agentic

compile rate

stubbed dialogues

pass rate, pass@k

compile feedback (agentic)

Figure 1: The evaluation harness. All configurations share the same tasks, environment, and checks, differing only in how the program is produced Table 1: Pass rate is mean per-task partial credit; tokens are mean output per task Configuration

Model

pass@k (%)

Compile (%)

Pass rate (%)

Tokens/task

25.0 20.0 15.0 14.0 14.0

38.0 38.0 48.0 32.0 21.0

31.3 29.1 28.1 20.9 17.4

9.9k 9.5k 7.0k 9.2k 8.2k

Agentic, Opus 4.8 (k=1); Sonnet 5 in Appendix E Baseline (no skills) Opus 4.8 33.0 Skills v1 (inert docs) Opus 4.8 28.0 Skills v2 (live plugin) Opus 4.8 14.0 Skills v3 (rewritten) Opus 4.8 37.0 afcode (minimal CLI) Opus 4.8 21.0

64.0 89.0 86.0 98.0 89.0

46.5 51.5 43.6 61.1 45.3

835k 480k 389k 440k 293k

Direct generation, zero-shot (k=3) Claude Opus 4.8 GPT-5.5 Gemma-4 31B IT GPT-5 mini Claude Sonnet 5

Metrics. We report compile rate, the fraction of tasks whose program compiles; pass rate, the mean per-task fraction of checks that succeed (partial credit); pass@k, the fraction of tasks passing every check in at least one of k runtime executions of a fixed compiled program; and mean output tokens per task. Direct generation compiles once and executes k=3 times, since a single generation is never re-run. Agentic configurations already execute the program repeatedly within their trajectory, so we score their final draft once, at k=1 (Appendix D). Models. We evaluate five models: three frontier hosted models (GPT-5.5, Claude Opus 4.8, Claude Sonnet 5), one smaller hosted model (GPT-5 mini), and one open-weight model (Gemma-4 31B Instruction-Tuned). The direct-generation leaderboard includes all five. Agentic configurations are run on Opus 4.8 and Sonnet 5, the hosted models supported by the coding-agent CLIs we study. Configurations. Holding task, environment, and checks fixed, we vary the prompting context and whether the model may iterate against a compiler. Direct generation is one shot with no tools: zero-shot supplies only the specification and any starting program, and two-shot adds two held-out examples; both use pass@3 to capture runtime variability. The remaining configurations are agentic and use pass@1, since they can already execute and revise code repeatedly. The baseline places the zero-shot prompt in a sandbox with a general-purpose coding agent that may read, edit, compile, and run the agent against the task’s checks over multiple turns, but has no knowledge of AgentScript. Three domain-skill packs add generation, observation, and testing skills, differing only in how that guidance is authored and delivered: v1 as inert documentation, v2 as a live executable plugin, and v3 as that same plugin with only the authoring skill’s guidance rewritten (Appendix F) . Finally, afcode is an alternative platform CLI with a built-in agent and a minimal tool set.

3

Results

Finding 1: models struggle, and the difficulty is concentrated in authoring. The best model reaches only 25.0% pass@3 in a narrow 14.0–25.0% field. Even the strongest compile rate is under half (48.0%), so most failures never parse into valid AgentScript. Among programs that do compile many still misbehave, Gemma-4 compiling 48.0% of samples but solving only 15.0%. Few-shots widen this gap rather than close it, lifting Opus’s compile rate to 60.0% while lowering its pass@3 to 21.0% (Table 3). By category (Table 5), every model scores 0.0% on authoring and no configuration 3

exceeds 16.7%, whereas optimization is already 53.3% zero-shot and debugging rises from 22.5% to 45.0% under the strongest skill pack. That authoring collapses while scaffolded editing does not points to generative coverage of the DSL, not local reasoning, as the bottleneck; and no configuration wins all three categories. Finding 2: scale and price are weak predictors. Were capability tracking scale and price, the largest hosted models would dominate every column; they do not. Gemma-4 31B, the smallest and only open-weight model, has the highest compile rate of any model (48.0% vs. 38.0%), the lowest token cost and a pass rate (28.1%) within three points of the best (31.3%). A gap remains on strict pass@3 (15.0% vs. 25.0%), so smaller models are not equal, but the ordering is not the price ordering: among hosted models, Sonnet 5 trails the smaller GPT-5 mini on both compile rate and pass rate. Defaulting to the most expensive model is not clearly justified in this regime. Finding 3: an agentic loop is the largest lever, and how domain guidance is written matters more than how it is packaged. Giving Opus a compiler to iterate against raises pass@k from 25.0% to 33.0% and increases mean pass rate (31.3% → 46.5%): the largest gain from any harness change, driven by the ability to compile, run, and revise its own program rather than framework choice. Adding skills on top makes the outcome sensitive to how that pack is authored. Moving from inert docs (v1) to a live plugin (v2) changes delivery breadth and catalog together and regresses below the skill-free baseline (Opus 28.0 → 14.0%, Sonnet 27.0 → 16.0%), inviting the conclusion that executable delivery is the culprit. It is not. Version v3 holds delivery fixed and rewrites only the authoring skill’s guidance, halving its entry point (830 → 461 lines) and splitting its references for progressive disclosure at unchanged word count. That single edit takes Opus to 37.0% pass@1, 98.0% compile, and 61.1% pass rate, the best result in our study: packaging was never the binding constraint, guidance density was. Under v2 the model spent more turns on failed than passed tasks (15.4 vs. 13.1); under v3 mean turns fall to 12.0.

4

Discussion and Future Work

Current LLMs remain unreliable for DSLs authored by proprietary agents and larger models provide limited gains. The largest single lever is allowing the model to compile and run its own program; beyond that, the gains are uneven and must be measured rather than assumed. Our sharpest result concerns domain-skill packs, whose value depends less on packaging than on how their guidance is written: rewriting one skill for progressive disclosure improved strict accuracy by +23 points, exceeding gains from changes in delivery or catalog, suggesting that teams shipping skills to agents should treat guidance density as a design variable. Agentic loops and domain scaffolding largely overcome the DSL’s surface-form difficulty, with compile rates reaching 98%, but behavioral correctness does not follow: even with compiler errors and graded results in hand, our best configuration leaves 40 of 100 tasks at partial credit. The bottleneck is therefore not the absence of correctness signals, but the ability to act on them, pointing to targeted repair from failing cases rather than broad regeneration [Chen et al., 2023b, Madaan et al., 2023, Shinn et al., 2023, Chen et al., 2023a, Ni et al., 2023] as a promising next step.

5

Related Work

Code-generation benchmarks target general-purpose languages: function synthesis [Chen et al., 2021, Austin et al., 2021], competition programming [Li et al., 2022], and repository-level issue resolution [Jimenez et al., 2023, Yang et al., 2024]. DSL and low-resource generation covers public, well-represented DSLs such as text-to-SQL [Yu et al., 2018] and IT-automation YAML [Pujar et al., 2023], and resource level across general-purpose languages [Cassano et al., 2022]; we target instead a proprietary, in-production agent-authoring DSL absent from public data. In-context learning [Brown et al., 2020, Dong et al., 2023, Scholak et al., 2021, Geng et al., 2023] and execution-guided selfcorrection [Chen et al., 2023b, Madaan et al., 2023, Shinn et al., 2023, Chen et al., 2023a, Ni et al., 2023] are candidate mechanisms; our results point to the latter. Agents [Yao et al., 2023] are here the artifact to be authored, not the harness. 4

References Amazon Web Services. What is Amazon Lex V2? — Amazon Lex V2 developer guide. https: //docs.aws.amazon.com/lexv2/latest/dg/what-is.html, 2025. Accessed: 2026-08-27. Jacob Austin, Augustus Odena, Maxwell Nye, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020. Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, et al. Multiple: A scalable and extensible approach to benchmarking neural code generation. arXiv preprint arXiv:2208.08227, 2022. Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests. In International Conference on Learning Representations (ICLR), 2023a. Mark Chen, Jerry Tworek, Heewoo Jun, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128, 2023b. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, et al. A survey on in-context learning. arXiv preprint arXiv:2301.00234, 2023. EvalPlus Team. EvalPlus leaderboard. https://evalplus.github.io/leaderboard.html, 2025. Accessed: 2026-08-13. Mikhail Evtikhiev, Egor Bogomolov, Yaroslav Sokolov, and Timofey Bryksin. Out of the bleu: how should we assess quality of the code generation models? arXiv preprint arXiv:2208.03133, 2022. Saibo Geng, Martin Josifoski, Maxime Peyrard, and Robert West. Grammar-constrained decoding for structured nlp tasks without finetuning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. Google Cloud. Flows — Dialogflow CX documentation. https://cloud.google.com/ dialogflow/cx/docs/concept/flow, 2025. Accessed: 2026-08-27. Carlos E Jimenez, John Yang, Alexander Wettig, et al. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, et al. Competition-level code generation with alphacode. arXiv preprint arXiv:2203.07814, 2022. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. In Advances in Neural Information Processing Systems (NeurIPS), 2023. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, et al. Self-refine: Iterative refinement with self-feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2023. Marjan Mernik, Jan Heering, and Anthony M Sloane. When and how to develop domain-specific languages. ACM Computing Surveys, 37(4):316–344, 2005. Microsoft. Create and edit topics — Microsoft Copilot Studio. https://learn.microsoft.com/ en-us/microsoft-copilot-studio/authoring-create-edit-topics, 2025. Accessed: 2026-08-27. Ansong Ni, Srinivasan Iyer, Dragomir Radev, Veselin Stoyanov, Wen-tau Yih, Sida I Wang, and Xi Victoria Lin. Lever: Learning to verify language-to-code generation with execution. In International Conference on Machine Learning (ICML), 2023. 5

Saurabh Pujar, Luca Buratti, Xiaojie Guo, Nicolas Dupuis, Burn Lewis, et al. Automated code generation for information technology tasks in yaml through large language models. arXiv preprint arXiv:2305.02783, 2023. Salesforce. Get started with Agent Script. https://developer.salesforce.com/docs/ai/ agentforce/guide/agent-script.html, 2025. Agentforce Developer Guide. Accessed October 2026. Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. Picard: Parsing incrementally for constrained auto-regressive decoding from language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023. SWE-bench Team. SWE-bench leaderboard. https://www.swebench.com/, 2025. Accessed: 2026-08-13. John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems (NeurIPS), 2024. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, et al. Spider: A largescale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018.

A

The AgentScript language

AgentScript is an indentation-structured declarative DSL. A program declares a system block (persona and canned messages), a config block (deployment identifiers), typed mutable variables that persist across a conversation, and one or more topics, each of which bundles the actions the agent may invoke with natural-language reasoning instructions that govern when to invoke them. Actions declare typed inputs and outputs and bind those outputs back into conversation variables. Listing 1 shows an abridged program in the shape the benchmark uses; identifiers are anonymized. Two properties make the language a hard generation target. Semantics are carried by fields whose names read as ordinary English but whose values change runtime behavior (require_user_confirmation on a read-only lookup inserts a confirmation turn), and the binding between action outputs and conversation variables is positional and easy to permute without breaking compilation. Both yield programs that compile and misbehave, which is the failure mode our results are dominated by.

B

Benchmark construction

Provenance. Tasks are seeded from real agent-development work. Each authoring task pairs a natural-language specification with a human-authored gold agent that satisfies it. Debugging and optimization tasks are built by injecting defects of known type into a gold agent, so every task carries a recorded ground-truth fault, its location, and its intended repair. Composition. 100 tasks: 30 authoring, 40 debugging, 30 optimization; 7 are labelled basic, 22 intermediate, and 71 advanced. The 40 debugging tasks carry a median of 3 injected defects each (max 5) drawn from eight recurring classes observed in real development, listed in Table 2. Optimization tasks inject a single behavioral defect into a working agent—the program compiles and runs, but violates a stated objective (for example, a read-only lookup that pauses for user confirmation). 6

system: instructions: "You are an order tracking assistant for a retailer." messages: welcome: "Hi! Share an order number or email to get started." error: "I can’t reach the order system right now." variables: customer_email: mutable string = "" description: "Customer’s email address" customer_verified: mutable boolean = False description: "Whether customer identity has been verified" start_agent order_locator: description: "Locates and validates customer order information" actions: Get_Customer_Info: description: "Retrieves customer information using email address" require_user_confirmation: False inputs: email: string is_required: True is_user_input: True outputs: customer_found: boolean customer_name: string target: "flow://GetCustomerInfo" before_reasoning: if @variables.customer_email == "": set @variables.customer_verified = False Listing 1: An abridged AgentScript program.

Table 2: Defect classes injected into debugging tasks, with the number of tasks containing at least one defect of that class. Counts exceed 40 because tasks carry multiple defects. Defect class

Tasks

Reasoning lifecycle (hook ordering, side effects) Variable declarations (scope, shadowing, missing decls) DSL syntax (malformed blocks, indentation) Type serialization (input/output type mismatch) Config validation Locale / internationalization Voice channel Markdown rendering

35 30 25 23 11 6 5 5

Acceptance checks. Tasks ship 630 executable test cases in total, a median of 5 per task (range 1–13). A test case is a scripted multi-turn dialogue (median 3 turns, max 15) together with the expected agent behavior at its end: the actions the agent must have invoked, the conversation-variable state it must have reached, and constraints on its response. Backend actions are stubbed with fixed return values, so a dialogue is reproducible across runs and models. Every task is scored all-or-nothing at the task level: a program passes only if all of its test cases pass, and a program that fails to compile scores zero.

C

Prompts

Zero-shot. The default generation prompt in our codebase bundles a syntax example into every call. Because that example is itself a demonstration, our zero-shot condition removes it, so the model receives the specification and constraints and no syntax help at all: 7

You are an expert AgentScript developer. Your task is to generate AgentScript code that meets this specification: {specification} Constraints: {constraints} Return ONLY valid AgentScript code that satisfies the specification and compiles cleanly. Do not include prose commentary, explanations, or a diff.

Debugging. Debugging tasks supply the broken program and ask for the full corrected file, explicitly forbidding diffs and unrelated refactors so that grading sees a complete program: The following AgentScript file has a bug. Analyze the problem, identify the root cause, and return the FULL corrected file. {program} Constraints the fix must satisfy: {constraints}

Optimization. Optimization prompts supply the working program and the observed symptom only. We deliberately omit the constraints block here: for these tasks the constraints spell out the intended repair, and including them would leak the answer. Two-shot. The two-shot condition prepends two (specification, program) demonstrations drawn from a held-out pool of gold agents that no benchmark task uses. Agentic. Agentic configurations receive the same zero-shot prompt seeded into a sandbox containing the starting program (where the task supplies one) and helpers for compiling and running the agent against the task’s checks, so they may iterate on both compiler errors and graded execution results. Direct generation receives neither.

D

Harness and protocol

Each generated program is compiled in a real project environment on the host platform; programs that compile are deployed and driven through each test case’s dialogue by a scripted user simulator, with stubbed backend actions. Agentic runs execute in a per-task sandbox capped at 50 assistant turns and 900 s of wall clock, whichever comes first; the final draft.agent in the sandbox is what gets graded, whether or not the agent announced completion. Why k differs by configuration. Runtime execution is stochastic even for a fixed compiled program: the same program can produce different outcomes across runs. We therefore use different values of k for direct LLM generation and agentic configurations to reflect the different opportunities each configuration has to respond to this runtime variability. For direct generation, the model produces a program in a single shot and does not observe or respond to its execution outcome. We therefore compile the generated program once and execute it three times, using pass@k with k=3 to account for runtime variability. The task is counted as passed if any of the three executions clears every check. Thus, these three trials correspond to repeated executions of the same generated program, not to three independently generated programs. Agentic configurations have a different evaluation protocol. The agent can compile and execute its program during its trajectory and use compiler errors, execution outcomes, and check results to revise the program over multiple turns. We therefore use pass@k with k=1 and report the best result achieved during the agent’s trajectory. Specifically, at each point where the agent produces or revises a program, we evaluate it against the task’s checks, and the agent receives the corresponding feedback to guide subsequent revisions. The reported score is the outcome of the best program encountered during this process. This gives agentic configurations multiple opportunities to respond to runtime variability as part of their normal execution loop, whereas direct-generation configurations receive three independent executions of their single generated program. 8

E

Additional results

Table 3: Two-shot direct generation (100 tasks; pass@3). Two-shot lift compile rate substantially for some models (Opus 38.0 → 60.0) but do not convert into correctness: only Sonnet 5 improves on pass@3, and by just two points (14.0→16.0); the strongest model drops four. Demonstrations teach syntax, not intent. Model

pass@3 (%)

Compile (%)

Pass rate (%)

Tokens/task

21.0 20.0 16.0 15.0 14.0

60.0 36.0 31.0 24.0 47.0

30.9 25.9 18.4 19.1 28.4

14.4k 11.0k 9.8k 10.7k 7.0k

Claude Opus 4.8 GPT-5.5 Claude Sonnet 5 GPT-5 mini Gemma-4 31B IT

Table 4: Agentic configurations for both models (pass@1). The body table reports the Opus rows only; Sonnet is included here in full. v3 lifts Sonnet above v2 on every metric yet still trails v1’s inert docs on strict pass@1. Configuration

Model

pass@1 (%)

Compile (%)

Pass rate (%)

Tokens/task

Baseline (no skills) Baseline (no skills) Skills v1 (inert docs) Skills v1 (inert docs) Skills v2 (live plugin) Skills v2 (live plugin) Skills v3 (rewritten) Skills v3 (rewritten) afcode (minimal CLI) afcode (minimal CLI)

Opus 4.8 Sonnet 5 Opus 4.8 Sonnet 5 Opus 4.8 Sonnet 5 Opus 4.8 Sonnet 5 Opus 4.8 Sonnet 5

33.0 20.0 28.0 27.0 14.0 16.0 37.0 23.0 21.0 28.0

64.0 65.0 89.0 79.0 86.0 62.0 98.0 91.0 89.0 79.0

46.5 40.4 51.5 43.7 43.6 29.4 61.1 49.5 45.3 52.7

835k 1,563k 480k 1,125k 389k 1,123k 440k 914k 293k 400k

Table 5: Per-category pass@k for the strongest model (Opus 4.8) across all configurations. Direct rows are pass@3; agentic rows are pass@1. Authoring resists every configuration (best 16.7%); optimization is already tractable zero-shot; no single configuration wins all three categories. Configuration Zero-shot direct Two-shot direct Agentic baseline Skills v1 (inert docs) Skills v2 (live plugin) Skills v3 (rewritten) afcode (minimal CLI)

F

Authoring (%)

Debugging (%)

Optimization (%)

0.0 10.0 16.7 6.7 13.3 10.0 6.7

22.5 15.0 25.0 32.5 15.0 45.0 20.0

53.3 40.0 60.0 43.3 13.3 53.3 36.7

Skill-pack versions

All three versions cover the same three capabilities—generation (author an agent from a specification), observation (inspect an agent and localize faults), and testing (execute an agent against checks). They differ as follows. v1 (inert docs) is a curated three-skill subset copied into the sandbox as static markdown. The agent may read the guidance; the accompanying hooks and scripts do not execute. v2 (live plugin) ships the same core skills inside the platform’s real agent plugin: 40 skills, working hooks, helper scripts, a bundled language server, and a platform CLI the agent may shell out to. Its authoring skill’s entry point is 830 lines with 41 reference files. 9

v3 (rewritten guidance) keeps v2’s plugin shell, its 40-skill catalog, its tool policy, and its bundled language server unchanged. It rewrites only the authoring skill: the entry point shrinks to 461 lines (−44%) and its references are split from 41 into 49 finer, on-demand files, holding total reference word count essentially flat (88.3k → 89.0k words). The change is a progressive-disclosure restructure—a lean entry point that pulls in detail only when a task needs it—not additional content. Comparing v1 with v2 therefore varies delivery and catalog breadth together, and cannot attribute the regression to either alone. Comparing v2 with v3 isolates guidance: one file, restructured, everything else held fixed. Neither v2 nor v3 ever invoked the bundled language server on any of the 100 tasks— both validated through the harness compile.sh—so v3’s compile-rate gain cannot be attributed to a new validation surface.

10

Related documents

• LLM-based Hardware Development with Hierarchical IRs and End-to-End Multi-Agent Workflow#818715
• LLM-based Hardware Development with Hierarchical IRs and End-to-End Multi-Agent Workflow#808723
• ACEM: A Cost Estimation Model for Agentic Software Engineering#424031
• Developing LLM-based Multi-Agent Systems in Software Engineering: A Mixed-Method Experience Report#448000
• BuildBench: Benchmarking LLM Agents on Compiling Real-World Open-Source Software#1013446