arXiv:2605.09610v1 [cs.MA] 10 May 2026
SmartEval: A Benchmark for Evaluating LLM-Generated Smart Contracts from Natural Language Specifications Abhinav Goel
Chaitya Shah
Columbia University New York, NY, USA [email protected]
Columbia University New York, NY, USA [email protected]
Agostino Capponi
Alfio Gliozzo
Columbia University New York, NY, USA [email protected]
IBM T.J. Watson Research Center Yorktown Heights, NY, USA [email protected]
Abstract
Keywords
We introduce SmartEval, a benchmark for systematically evaluating the quality of Solidity smart contracts generated by large language models (LLMs) from natural language specifications. SmartEval provides a corpus of 9,000 generated contracts paired with expert-written ground-truth implementations drawn from the FSMSCG dataset, a five-dimensional evaluation rubric covering functional completeness, variable fidelity, state-machine correctness, business-logic fidelity, and code quality, and a reproducible generationand-evaluation pipeline. To validate the benchmark’s reliability, we conduct three independent empirical studies: a five-condition ablation study (N=300 per condition) isolating the contribution of each pipeline component, a human expert evaluation by three Columbia University PhD researchers confirming automated scores align with expert judgment to within 0.34 points, and external security analysis via the Slither static analyzer confirming 79.4% agreement between the LLM auditor and a non-LLM rule-based tool. Systematic analysis of 9,000 generated contracts reveals characteristic failure modes (logic omissions at 35.3%, state transition errors at 23.4%, and complexitydriven degradation) and quantifies a +8.29 composite-score advantage of generated contracts over ground-truth implementations, attributable to LLMs’ literal specification-following behavior. SmartEval establishes a reproducible, validated foundation for empirical research on LLM smart contract synthesis quality, with all data, evaluation code, and generated contracts publicly released.
Code-Generating Language Models; Program Synthesis; Smart Contracts; Agentic AI; LLM Evaluation
CCS Concepts • Software and its engineering → Software verification and validation; • Computing methodologies → Machine learning; • Information systems → Data mining; • Security and privacy → Software and application security. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. Conference’17, Washington, DC, USA © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-XXXX-X/2018/06 https://doi.org/XXXXXXX.XXXXXXX
ACM Reference Format: Abhinav Goel, Chaitya Shah, Agostino Capponi, and Alfio Gliozzo. 2026. SmartEval: A Benchmark for Evaluating LLM-Generated Smart Contracts from Natural Language Specifications. In . ACM, New York, NY, USA, 15 pages. https://doi.org/XXXXXXX.XXXXXXX
1
Introduction
Smart contracts, self-executing programs deployed on blockchain networks that automatically enforce agreement terms without intermediaries, have become critical infrastructure for decentralized finance (DeFi), non-fungible tokens (NFTs), and decentralized autonomous organizations (DAOs) [4, 5, 22]. However, developing secure and correct smart contracts remains a significant challenge due to the complexity of languages like Solidity, the immutability of deployed contracts, and the high-stakes nature of financial applications [1, 6]. The gap between domain experts who understand contract requirements and developers who can implement them in Solidity creates bottlenecks in the development process. Traditional approaches require extensive back-and-forth between legal or business stakeholders and blockchain developers, often leading to specification misinterpretation and implementation errors. These errors have resulted in significant financial losses, with notable examples including the DAO hack ($60M) and the Parity wallet freeze ($300M) [13, 15]. Recent work by Anthropic’s red team [23] demonstrated that frontier AI agents can now autonomously exploit realworld smart contract vulnerabilities, collectively extracting $4.6M in simulated stolen funds from contracts exploited after model knowledge cutoffs, with exploit revenue doubling every 1.3 months. Recent advances in Large Language Models (LLMs) have demonstrated remarkable capabilities in code generation [2, 7, 19]. Models such as GPT-4 [18] and Code Llama [19] can generate syntactically correct code from natural language descriptions. However, generating production-ready smart contracts requires more than syntactic correctness; it demands semantic fidelity to specifications, proper state machine implementation, security best practices, and correct economic logic. Emerging techniques such as SmartInv [20] and SmartSys [21] leverage multimodal and foundation models to uncover “machine un-auditable” smart contract bugs that evade traditional static analysis. These results highlight both the promise and
Conference’17, July 2017, Washington, DC, USA
the urgency of developing systematic, validated quality evaluation frameworks for AI-generated smart contracts. In this paper, we introduce SmartEval, a benchmark and evaluation framework for studying how well LLMs translate natural language contract specifications into correct, secure, and deployable Solidity code. SmartEval applies to a broad class of contract types including NDA agreements, employment contracts, rental agreements, service contracts, token standards, and governance systems, any domain where natural language must become verifiable on-chain logic. The benchmark provides three core assets: (i) a corpus of 9,000 LLM-generated contracts paired with 9,000 expert ground-truth implementations, (ii) a validated five-dimensional evaluation rubric with deterministic score computation, and (iii) a reproducible generation-and-evaluation pipeline with full provenance metadata. A complete worked example for a token staking contract is provided in Appendix K and Table 13. Our contributions include:
(1) SmartEval Benchmark and Dataset: A corpus of 9,000 LLM-generated Solidity contracts paired with expert groundtruth implementations, spanning six contract categories (token standards, governance, staking, escrow, NDA/legal, and general service contracts), with per-contract quality scores, security reports, and compilation results, all publicly released for reproducible evaluation. (2) Five-Dimensional Quality Evaluation Rubric: The first rubric to treat FSM state machine correctness as a weighted first-class metric in LLM code generation evaluation, measuring Functional Completeness (25%), Variable Fidelity (15%), State Machine Correctness (15%), Business Logic Fidelity (35%), and Code Quality (10%), with deterministic composite recomputation eliminating drift from model-generated aggregates. (3) Systematic Failure Mode Analysis: Characterization of LLM error patterns across 2,398 lower-performing contracts, revealing that logic omissions (35.3%) and state transition errors (23.4%) are the dominant failure modes, with performance degrading sharply on high-complexity specifications (8+ functions, 5+ states: avg. score 71.8 vs. 87.2 for low-complexity). (4) Three-Way Benchmark Validation: Independent confirmation of rubric reliability via (a) human expert scoring by three Columbia University PhD researchers (within 0.34 points of automated evaluator), (b) external Slither static analysis (79.4% LLM–tool vulnerability category agreement), and (c) a five-condition ablation study with N=300 per condition and cross-seed Cohen’s 𝑑 < 0.2. (5) Generation Pipeline with Severity-Gated Refinement: A reproducible multi-agent pipeline (parser, generator, auditor, refiner, evaluator) that produces all benchmark artifacts, with a control-flow security gate whose removal increases output standard deviation by 111% and reduces compilation by 5.2 percentage points. (6) Behavioral Analysis of LLM vs. Expert Developers: Quantitative characterization of the +8.29 composite gap
Abhinav Goel, Chaitya Shah, Agostino Capponi, and Alfio Gliozzo
between LLM-generated and expert-written contracts, attributing it to LLMs’ literal specification-following behavior versus expert developers’ architectural judgment, gasefficiency trade-offs, and pattern reuse.
2
Background
Smart contracts are programs stored on blockchain networks that execute automatically when predetermined conditions are met [4, 17]. Ethereum introduced Solidity as its primary programming language [9, 22]. Key features include state variables, functions with visibility and mutability modifiers, access-control modifiers, event logging, and contract inheritance. The immutability of deployed contracts makes correctness paramount: bugs cannot be patched post-deployment without complex migration procedures [24]. Common vulnerability classes include reentrancy attacks, integer overflow, access control flaws, and timestamp manipulation. Static analysis tools like Slither [10] and Mythril [16] can identify many vulnerability patterns but produce raw findings requiring expert interpretation. LLM-based approaches have demonstrated capabilities in translating natural language to functional code [3, 7], but smart contract generation demands blockchain-specific semantics: a “token” must conform to ERC20, “staking” requires careful asset lock handling, and economic invariants must be rigorously preserved. Multi-agent frameworks like CrewAI [8] and IBM Agentics [11] enable specialized agents to focus on distinct subtasks, preventing optimistic bias and enabling iterative refinement. We leverage Finite State Machine (FSM) representations [14] as formal behavioral specifications, modeling contract states, valid transitions, guard conditions, and triggered actions, providing rigorous validation beyond static analysis.
3
Benchmark Construction
SmartEval's benchmark artifacts are produced by a seven-phase agentic pipeline orchestrated by the IBMAgenticContractTranslator class, illustrated in Figure 1. Each phase is designed to be independently reproducible; all intermediate outputs are stored as structured JSON artifacts.
3.1
Phase 1: Requirement Specification Agent
The Requirement Specification Agent extracts structured information from natural language contract specifications into a UniversalContractSchema. The agent is instructed to extract exact terminology; if the contract says initializeLease the schema records initializeLease, not initialize. Every described function or operation is mapped to an obligation entry with the authorized party identified. The obligations array is never left empty when functions are described. The schema captures parties (name, role, blockchain address), financial terms (amount, currency, purpose, frequency, due date), dates, assets, obligations with deadlines and breach penalties, FSM conditions (exact function names, variable names, state names, state transitions, events, and logic conditions), and termination criteria. The schema is domain-agnostic: it handles NDA agreements, employment contracts, rental agreements, service contracts, ERC20 token standards, governance systems, and escrow mechanisms through the same typed structure.
SmartEval: A Benchmark for Evaluating LLM-Generated Smart Contracts from Natural Language Specifications
Conference’17, July 2017, Washington, DC, USA
semantic analysis, 12 mandatory rules (enforcing on-chain logic for every specification guarantee, full ERC20/ERC721 compliance where applicable, explicit state machine enforcement via enumbased variables and transition guards, and complete conservative economic logic), and a set of forbidden patterns. Target contract length is 150–400 lines.
3.3
Phase 3: Security Auditing
The Auditor Agent performs systematic security analysis across eight vulnerability categories: (1) reentrancy attacks (external calls followed by state changes, CEI pattern violations); (2) access control (missing modifiers on critical functions, constructor initialization); (3) arithmetic safety (unchecked operations, division by zero); (4) ether handling (payable access control, locked ether, withdrawal validation); (5) denial-of-service (unbounded loops, reverting recipients); (6) input validation (require statements, zero-address checks, amount validation); (7) timestamp dependence; and (8) external call safety (return value checking, low-level call error handling). The audit returns only valid JSON with fields: severity_level (none/low/medium/high/critical), approved boolean, issues array with function-specific exploit paths, recommendations with linelevel fixes, vulnerability_count, and security_score (A–F). No markdown or prose is returned.
3.4
Phase 4: Severity-Gated Reinforcement Loop
A key architectural contribution is the severity-gated reinforcement loop. Unlike standard actor-critic pipelines that pass a gradient signal, our should_refine() function implements a hard blocking gate: code with medium-or-higher audit severity cannot proceed to downstream artifact release and re-enters the refinement loop. Listing 1: Severity-Gated Refinement Decision Logic
Figure 1: SmartEval benchmark pipeline. Natural language contract specifications flow through seven agents: the Requirement Specification Agent extracts a structured schema, the Generator Agent produces initial Solidity code, the Auditor/Refiner loop applies severity-gated security refinement, and the Quality Evaluator scores the result across five dimensions and compares against expert ground-truth implementations. Optional deployment generates ABI and MCP server wrappers for testnet interaction.
3.2
Phase 2: Solidity Generation
The Generator Agent (role: Senior Solidity Smart Contract Engineer) transforms the parsed schema into production-ready Solidity code. The agent is designed to reason about each function in terms of three questions: what real-world operation does this represent, what invariant must hold before and after, and what can go wrong. Generation follows a structured eight-phase approach covering
def should_refine ( audit_report , refinement_count , max_iterations =2) : if refinement_count >= max_iterations : return False severity = audit_report . get ( ' severity_level ') approved = audit_report . get ( ' approved ' , False ) if ( not approved and severity in [ ' medium ' , ' high ' , ' critical ' ]) : return True return False
The Refiner Agent applies established secure coding patterns (Checks-Effects-Interactions, reentrancy guards, strict access control, comprehensive input validation) and each iteration is re-audited. Separating Generator and Refiner agents (each with distinct roles and backstories) prevents optimistic self-repair bias. Our ablation study (§5.7) demonstrates the gate’s measured effect: disabling it raises output standard deviation by 111% (std 10.31 vs. 4.89) and reduces compilation by 5.2 percentage points.
3.5
Phase 5 & 6: Optional Deployment
The ABI Agent generates Application Binary Interface specifications from refined Solidity code. The MCP Agent uses these to generate a complete Model Context Protocol server that exposes
Conference’17, July 2017, Washington, DC, USA
Abhinav Goel, Chaitya Shah, Agostino Capponi, and Alfio Gliozzo
contract endpoints as standardized interfaces, enabling AI assistants and automation tools to interact with the deployed contract on a Ganache blockchain environment.
4 Benchmark Design 4.1 Dataset We utilize the FSM-SCG dataset [12], a collection of 21,976 smart contract specifications each containing: (i) natural language requirements (avg. 120 words), (ii) formal FSM specifications with states, transitions, guards, and actions, and (iii) expert-written groundtruth Solidity implementations (avg. 85 lines). The dataset spans Solidity versions 0.4.x–0.8.x and covers token standards (ERC20, ERC721, ∼42%), crowdsales, governance systems, staking, escrow, and access control (∼58%). Table 1 summarizes key characteristics. Table 1: The FSM-SCG dataset [12] Characteristic Total entries Avg. requirement length Avg. contract length Solidity version range Standard tokens (ERC20/721) Custom business logic Components per entry
4.2
Value 21,976 120 words 85 lines 0.4.x–0.8.x ∼42% ∼58% Requirement + FSM + Code
Quality Evaluation Framework
The five-dimensional rubric assesses generated contracts as follows. Functional Completeness (M1, 25%) verifies functions are implemented with correct naming and complete logic. Variable Fidelity (M2, 15%) ensures naming consistency and appropriate data types. State Machine Correctness (M3, 15%) evaluates whether the FSM is properly defined with valid transitions and enforced guards. Business Logic Fidelity (M4, 35%) measures accuracy of obligations, financial logic, temporal constraints, and conditional flows. Code Quality (M5, 10%) checks for placeholders, error handling, and structure. The composite score is: Score = 0.25𝑀1 + 0.15𝑀2 + 0.15𝑀3 + 0.35𝑀4 + 0.10𝑀5
(1)
Composite scores are deterministically recomputed from raw metric scores in post-processing rather than accepted from LLM-generated aggregates, eliminating a systematic source of score drift. Error modes in lower-performing contracts are identified through the Quality Evaluator’s structured critical_gaps JSON output field, which categorizes deficiencies per metric into logic omissions, state transition errors, compilation failures, incomplete financial logic, and access control gaps, cross-referenced with solcx compiler logs.
4.3
Main Experiment Configuration
We evaluated the pipeline on 9,000 unique contracts processed in six parallel batches of 1,500. All contracts used GPT-4o-mini for all agent tasks, up to two refinement cycles, Solidity 0.8.x compilation validation, and the five-dimensional rubric. All six batches produced consistent results (compilation rate range: 86.48%–87.18%).
4.4
Ablation Study Design
To isolate the contribution of each pipeline component, we ran five conditions on the same randomly sampled set of 300 contracts (seed=42), with cross-seed significance testing on additional seeds to confirm reproducibility. • E – Single raw LLM call: GPT-4o-mini with a plain prompt, no schema extraction, no agents, no quality evaluation. Compilationonly baseline. • A – Schema extraction only, no reinforcement: full Requirement Specification Agent plus Generator Agent, but no security auditing or refinement loop. • B – Full pipeline, reinforcement disabled: all pipeline phases active, but should_refine() always returns False. • C – Full pipeline, max_iter=1: the configuration used for the 9,000-contract main evaluation. • D – Full pipeline, max_iter=2: two refinement iterations permitted. A key distinction of Conditions B–D (full pipeline runs) is that the Quality Evaluator operates with enhanced semantic matching during evaluation. Rather than scoring only exact specification matches, the evaluator applies camelCase-to-snake_case equivalence mapping, architectural synonym recognition (e.g., isActive credited for contractActive), gas-optimization pattern credit (e.g., boolean flags accepted as valid alternatives to explicit enum-based state tracking), and partial-implementation scoring for functions that satisfy the specification’s intent through different but semantically equivalent code patterns. These matching enhancements are a deliberate feature of the ablation evaluation design, enabling a more nuanced assessment of specification fidelity than keywordonly matching. They directly affect M3 (State Machine Correctness) scores in particular: GT contracts that implement state logic through boolean flags and timestamp comparisons receive full Path B credit under this scoring, raising GT’s M3 scores relative to the main 9,000contract evaluation and contributing to the slight GT advantage on that metric observed in Condition C.
4.5
Human Expert Validation Protocol
Three Columbia University PhD students (Enbei Zhang, Xiaoyang Liu, and Victoria Ruojie Lie) with research backgrounds in blockchain systems and formal methods independently scored 30 randomly sampled pipeline-generated contracts using the same five-dimensional rubric. Each evaluator received only the naturallanguage specification and the generated Solidity code, with no access to automated scores or each other’s assessments. This directly tests whether the LLM-based evaluator is calibrated to expert human judgment.
4.6
Slither External Validation Protocol
We ran all generated contracts through Slither (Trail of Bits), a non-LLM rule-based static analyzer, both before and after the reinforcement loop. This provides evaluation fully independent of any language model, testing whether the LLM auditor’s findings correspond to patterns a deterministic tool also detects.
SmartEval: A Benchmark for Evaluating LLM-Generated Smart Contracts from Natural Language Specifications
5
5.3
Benchmark Analysis
Table 2 presents aggregate performance statistics across the full SmartEval corpus of 9,000 evaluated contracts.
5.1
Value
Average Composite Score Standard Deviation Compilation Success Rate Average Processing Time (s) Total Evaluation Time (h)
81.54 12.87 86.54% 109.96 274.9
5.4
Grade Distribution (N=9,000)
7.3
B (80–89)
66.4
C (70–79)
23.1
D/F (<70)
Statistic
Value
Total Contracts Checked Successful Compilations Failed Compilations Not Checked Success Rate
8,824 7,637 1,187 176 86.54%
The pipeline achieved an 86.54% compilation success rate across 8,824 checked contracts, consistent across all six parallel batches (range: 86.48%–87.18%), confirming reproducibility.
Grade Distribution
A (≥90)
Compilation Validation Table 4: Compilation Statistics (N=9,000)
Table 2: Overall Performance Statistics (N=9,000) Metric
AI Generated vs. Expert Implementations (N=9,000)
Table 5 presents the full paired comparison between pipelinegenerated contracts and expert ground-truth implementations across all 9,000 evaluated contracts. Table 5: Generated vs. Ground Truth: Aggregate and PerMetric Comparison (N=9,000)
3.2
Generated
Ground Truth
Δ
Δ%
Aggregate Avg Composite Score Std Deviation
81.54 12.87
73.25 14.52
+8.29 —
+11.3% —
Per-metric breakdown M1: Functional Completeness M2: Variable Fidelity M3: State Machine Correctness M4: Business Logic Fidelity M5: Code Quality
84.45 84.62 83.12 76.73 83.85
77.82 79.13 79.28 66.41 75.19
+6.63 +5.49 +3.84 +10.32 +8.66
+8.5% +6.9% +4.8% +15.5% +11.5%
Metric
0
20
40
60
Percentage of contracts (%)
Figure 2: Grade distribution across all 9,000 generated contracts. B-grade contracts (80–89) account for 66.4% of output. The 3.2% D/F rate reflects failures concentrated in highcomplexity specifications (8+ functions, 5+ states). The pipeline produces predominantly B-grade contracts (66.4%), with 7.3% achieving A-grade and only 2.2% failing completely. The strong central tendency around B-grade reflects consistent pipeline performance across standard specifications, while the D/F tail corresponds to high-complexity contracts as discussed in §5.5.
5.2
Conference’17, July 2017, Washington, DC, USA
Individual Metric Performance
Table 3: Five-Dimensional Metric Averages (N=9,000) Metric
Weight
Avg Score
Contribution
Functional Completeness Variable Fidelity State Machine Correctness Business Logic Fidelity Code Quality
25% 15% 15% 35% 10%
84.45 84.62 83.12 76.73 83.85
21.11 12.69 12.47 26.86 8.39
Composite Score
100%
—
81.52
Business Logic Fidelity (76.73) is the lowest-scoring metric despite its highest weight (35%), capturing the hardest semantic aspects: economic logic, obligations, timing constraints, and conditional flows. Its weighted contribution (26.86 points) nonetheless dominates the composite score, making M4 the primary lever for pipeline improvement.
Generated contracts outperform ground-truth implementations by 8.29 composite points on average. Figure 3 overlays the full score distributions, showing not only the mean shift but the difference in spread: generated contracts (blue, std=12.87) are more tightly concentrated around the mean than ground-truth implementations (orange, std=14.52), whose higher variance reflects the diversity of implementation styles among expert developers. The gap is largest in M4 (Business Logic Fidelity, +10.32 pts) and smallest in M3 (State Machine Correctness, +3.84 pts), as shown in Table 5. Figure 4 presents a radar chart comparing per-metric profiles, making the behavioral differences immediately visible: generated contracts achieve higher scores on the five specification-fidelity metrics (M1, M2, M3, M4, M5).
5.5
Error Modes and Complexity Analysis
Logic omissions (35.3%), missing implementation of specified obligations, conditions, or side effects, represent the dominant failure mode. State transition errors (23.4%) occur in contracts with complex state machines where the system omits edge-case transitions or enforces guards inconsistently. Both failure modes are concentrated in high-complexity specifications, as shown in Table 6. Performance degrades with complexity: high-complexity contracts (8+ functions, 5+ states) score 15.4 points below low-complexity contracts with a 20.9 pp lower compilation rate, suggesting that
Conference’17, July 2017, Washington, DC, USA
Abhinav Goel, Chaitya Shah, Agostino Capponi, and Alfio Gliozzo
Score Distribution: Generated vs. Ground Truth (N=9,000)
maintaining semantic coherence across lengthy multi-obligation specifications is a fundamental challenge for single-pass LLM generation. Figure 5 provides a visual breakdown of error mode prevalence and the compound effect of specification complexity on both score and compilation rate.
𝜇 =81.54 Generated (𝜎 =12.87) Ground Truth (𝜎 =14.52)
𝜇 =73.25
Density
3
2 Complexity vs. Performance
Error Mode Breakdown
100
1 Logic Omission
35.3
90 30
40
50
60
70
80
90
100
23.4
State Trans.
110
Score / Compile%
0
Composite Score Compilation
Figure 3: Approximate score distributions for LLM-generated (blue) and expert ground-truth (orange) contracts, modeled as Gaussians from empirical means and standard deviations. Generated contracts are more tightly concentrated (lower variance) and centered 8.29 points higher. The distribution shapes reveal that generated contracts are more predictable; ground truth spans a wider range reflecting diverse expert implementation styles.
17.6
Financial Logic
13.3
80
70 10.3
Access Control
60 0
10
20
30
40 Low (1–3f)
Count (%)
Med (4–7f)
Avg Score
High (8+f)
Compile%
Figure 5: Left: Breakdown of error modes in the 2,398 lowerperforming contracts (C/D/F grade). Logic omissions are the dominant failure, followed by state transition errors. Right: Both average score and compilation rate degrade sharply as specification complexity increases from low (1–3 functions) to high (8+ functions), with a 15.4-point score gap and 20.9 pp compilation gap between extremes.
Functional
M1
84
77 Quality
Variable
M5
M2 71
5.6
Security Refinement Loop Effectiveness
Table 7: Security Refinement Loop Effectiveness (N=9,000) Metric
M3
Contracts w/ Med+ Severity Avg Security Issues/Contract Critical Vulnerabilities Compilation Success Rate
M4
State
Business
Before
After
4,127 (45.9%) 2.8 287 81.2%
1,203 (13.4%) 0.7 34 86.5%
Generated (N=9,000) Ground Truth (N=9,000)
Figure 4: Radar chart of per-metric quality profiles for LLM-generated contracts versus expert ground-truth implementations (N=9,000 pairs). Generated contracts (blue) cover more area on M1, M2, M3, M4, and M5, reflecting literal specification-following. M4 (Business Logic) shows the widest gap (+10.32 pts). Table 6: Performance by Specification Complexity Complexity Low (1–3 funcs, 1–2 states) Medium (4–7 funcs, 3–4 states) High (8+ funcs, 5+ states)
N
Avg Score
Compile%
3,245 4,517 1,229
87.2 81.4 71.8
94.1 86.7 73.2
The severity-gated reinforcement loop reduced medium-or-higher severity contracts from 45.9% to 13.4% (70.9% reduction) and cut critical vulnerabilities from 287 to 34 (88.2% reduction). Slither independently confirms a 43.8% reduction in total static analysis findings post-refinement, validating that the improvements are not artifacts of the LLM auditor evaluating its own outputs.
5.7
Ablation Study
Table 8 presents results from the five-condition ablation (N=300 per condition, same contracts across all conditions). Figure 6 visualizes the compilation rate progression across conditions, illustrating the incremental contribution of each component. Several findings emerge from the ablation. First, each component makes a measurable, monotone contribution: schema extraction adds +14.0 pp compilation (E→A), structured generation adds +16.0 pp (A→B), and the reinforcement gate adds +5.2 pp (B→C). The full
SmartEval: A Benchmark for Evaluating LLM-Generated Smart Contracts from Natural Language Specifications
Table 8: Five-Condition Ablation Study (N=300 each, seed=42). Condition C corresponds to the paper’s main 9,000-contract configuration.
Table 9: Generated vs. Ground Truth: Aggregate and PerMetric Comparison (Condition C, N=300 paired) Generated
Ground Truth
Δ
Δ%
Aggregate Avg Composite Score Std Deviation GT Compilation Rate
83.44 4.89 —
76.83 15.14 40.7%
+6.61 — —
+8.6% — —
Per-metric breakdown M1: Functional Completeness M2: Variable Fidelity M3: State Machine Correctness M4: Business Logic Fidelity M5: Code Quality
87.36 82.97 80.41 81.19 86.78
78.82 78.30 82.86 71.12 80.53
+8.54 +4.66 -2.45 +10.06 +6.25
+10.8% +6.0% -3.0% +14.1% +7.8%
Metric
Cond.
Description
Score
±Std
Compile%
E A B C D
Single raw LLM call Schema only, no reinforce Full pipeline, reinforce OFF Full pipeline, max_iter=1 Full pipeline, max_iter=2
— 78.98 82.64 83.44 83.70
— 6.55 10.31 4.89 5.57
49.3 63.3 79.3 84.5 85.2
Paper (N=9K)
Full pipeline
81.54
12.87
86.54
Conference’17, July 2017, Washington, DC, USA
Ablation: Compilation Rate by Condition
Compilation Rate (%)
100 84.5
85.2
C
D
79.3
80 63.3
60 49.3
40 20 0 E
A
B
Figure 6: Compilation rate increases monotonically as pipeline components are added. The full pipeline (C) improves compilation by +35.2 pp over the raw LLM baseline (E). The paper’s reported 86.54% is reproduced within 2 pp at N=300. pipeline achieves 84.5% compilation against a raw-LLM baseline of 49.3%, a +35.2 percentage point improvement. Second, the reinforcement loop’s most important contribution is variance reduction: Condition B has std=10.31 while Condition C has std=4.89, a 53% reduction. For a production code generation system, consistent quality across contracts is as important as mean quality. Third, Condition C closely reproduces the paper’s 86.54% compilation rate (84.5% at N=300), validating that the main experiment results generalize across different random samples. Cross-seed pairwise significance testing across three seeds confirms all intercondition effect sizes are negligible (Cohen’s 𝑑 < 0.2 on all metrics), establishing that results are stable and not sampling-dependent. Finally, the marginal benefit of a second refinement iteration (D vs. C: +0.27 score, +0.7 pp compile) occurs at 2.2× the compute cost (206s vs. 93s per contract), a clear diminishing-returns result. We recommend max_iter=1 for production deployments prioritizing throughput.
5.8
AI Generated vs. Expert Implementations
The per-metric pattern is consistent with the full 9,000-contract results (Table 5): M4 (Business Logic Fidelity) shows the largest generated advantage (+10.06 pts) and the generator leads on M1, M2, and M5. M3 (State Machine Correctness) shows a slight GT
advantage in this ablation subsample (−2.45 pts). This is partly explained by the ablation evaluation applying more granular semantic matching to state machine implementations: the Quality Evaluator’s M3 rubric includes Path B scoring, which awards full credit for stateless designs using boolean flags, timestamp comparisons, and access-control guards as semantically equivalent alternatives to explicit enum-based FSMs. In the ablation, GT contracts that relied on these implicit state-tracking patterns were more consistently recognized as correct under Path B, raising GT’s M3 score from 79.28 in the full experiment to 82.86 in the ablation subsample and producing the slight GT advantage on that single metric. Figure 7 presents a unified view of per-metric scores across all three reference points: the full 9,000-contract evaluation, the ablation-generated contracts (Condition C, N=300), and the ablation ground-truth implementations. Comparing the teal bars (9,000-contract main evaluation) to the blue bars (ablation Condition C) reveals a modest but meaningful difference. The ablation contracts score notably higher on M4 (Business Logic Fidelity: 81.19 vs. 76.73, +4.46 pts). The most likely explanation is natural sampling variance: the ablation draws 300 contracts from the full 21,976-entry dataset, and M4’s dominant 35% weight amplifies any distributional skew in the sampled subset. Contracts that happen to have more explicit obligations and financial logic in their specifications will score higher on M4 regardless of pipeline configuration, and a 300-contract sample is not guaranteed to match the full dataset’s distribution on this dimension. Despite these per-metric differences, composite scores are consistent (81.54 main vs. 83.44 ablation, a 1.90-point difference), and the central finding that Business Logic Fidelity is the weakest and most consequential dimension holds across both experimental settings.
5.9
Human Expert Validation
To validate whether the LLM-based evaluator produces scores aligned with expert human judgment, three Columbia University PhD students (Enbei Zhang, Xiaoyang Liu, and Victoria Ruojie Lie) independently scored 30 randomly sampled pipeline-generated contracts using the five-dimensional rubric from §4.2. The aggregate human expert composite score (81.88) aligns with the automated evaluator (81.54) to within 0.34 points, well within the natural inter-evaluator variability of 5.1 points mean absolute deviation. The grade distribution from the human study closely mirrors the automated distribution from 9,000 contracts. If the LLM
Conference’17, July 2017, Washington, DC, USA
Abhinav Goel, Chaitya Shah, Agostino Capponi, and Alfio Gliozzo
Per-Metric Scores: 9,000-Contract Evaluation vs. Ablation Generated vs. Ablation Ground Truth
M1: Functional Completeness
M2: Variable Fidelity
M3: State Machine
M4: Business Logic Ablation Ground Truth (N=300) Ablation Generated (N=300)
M5: Code Quality Main Evaluation Generated (N=9,000)
65
70
75
80 Score (0–100)
85
90
95
Figure 7: Unified per-metric comparison across three reference points: the full 9,000-contract evaluation (teal), ablationgenerated contracts (blue, Condition C, N=300), and ablation ground-truth implementations (orange). M4 (Business Logic Fidelity) shows the largest generated advantage across all three series. Differences between the teal and blue bars reflect sampling variance across the two contract subsets. Table 10: Human Expert Validation Results (N=30 contracts, 3 evaluators)
Table 11: Slither External Validation Results Metric
Metric
Human Aggregate
Mean Composite Score Delta Inter-evaluator MAD Grade distribution comparison A (≥90) B (80–89) C (70–79) D/F (<70)
LLM Evaluator
81.88 +0.34 (human higher) 5.1 pts 10.0% 60.0% 23.3% 6.7%
81.54 —
7.3% (paper) 66.4% (paper) 23.1% (paper) 3.2% (paper)
evaluator were systematically inflating scores or introducing selfreferential bias, independent domain experts applying the same rubric would have arrived at substantially lower values. They did not.
Compilation rate Avg Slither issues/contract Total issue reduction LLM Auditor – Slither Agreement Category agreement rate
External Security Validation via Slither
To address the concern that LLM-based auditing of LLM-generated code may produce correlated rather than independent assessments, we ran all generated contracts through Slither [10], a non-LLM rule-based static analyzer, providing security evaluation fully independent of any language model. The most significant finding is the 79.4% category agreement rate: when the LLM auditor flags a vulnerability category (reentrancy, access control, input validation, timestamp dependency), Slither independently detects an issue in the same category 79.4% of the time. This directly confirms that the LLM auditor is detecting
Post-Refine
78.1% 6.4 43.8%
82.3% 3.6
79.4%
real, independently verifiable patterns rather than hallucinating security concerns. The refinement loop produces a 43.8% reduction in total Slither findings (6.4 to 3.6 per contract), with improvements concentrated in timestamp dependency issues, missing input validation, and local variable shadowing; genuine security-relevant findings, not stylistic changes.
5.11 5.10
Pre-Refine
Summary
Together, these results establish that the pipeline produces highquality, deployable smart contracts from natural language specifications with well-characterized performance. The five-condition ablation demonstrates that each pipeline component contributes measurably, with the full pipeline achieving a +35.2 pp compilation improvement over a raw LLM baseline. Human expert validation confirms that the automated evaluation metric is calibrated to expert judgment within 0.34 points. The per-metric breakdown further shows that the rubric captures genuine quality differences rather than uniformly favoring LLM output. External Slither validation confirms a 79.4% LLM–tool agreement rate, ruling out circular self-evaluation bias.
SmartEval: A Benchmark for Evaluating LLM-Generated Smart Contracts from Natural Language Specifications
6
Conclusion
We presented an end-to-end agentic pipeline for generating, auditing, and evaluating Solidity smart contracts from natural language specifications, and validated it with three independent empirical methods: a five-condition ablation study (N=300 per condition), a human expert evaluation (three Columbia University PhD researchers, 30 contracts), and external static analysis via Slither. Together, these experiments produce a coherent picture of what the pipeline contributes and where its limits lie. The most concrete result is the compilation improvement. A raw single-LLM call achieves 49.3% compilation; the full pipeline with schema extraction and the severity-gated reinforcement loop reaches 84.5%, a +35.2 percentage point gain. Each component contributes measurably, and the reinforcement loop’s most important effect is not the mean score gain (+0.80 pts over no-reinforcement) but the variance reduction: output standard deviation drops from 10.31 to 4.89 when the gate is enabled, producing consistently reliable code across contracts rather than occasionally excellent code with frequent failures. Human expert evaluation resolves the evaluator reliability question directly. Three independent PhD domain experts scored the same 30 contracts using the same rubric and arrived at a mean of 81.88/100, within 0.34 points of the automated evaluator’s 81.54. The grade distributions also closely agree. The Slither external validation corroborates the security auditor’s findings: 79.4% category agreement between the LLM auditor and a non-LLM rule-based tool confirms that the auditor detects real patterns, not self-referential artifacts. Together these two forms of external validation address the most fundamental concern about LLM-based evaluation pipelines: that they may be measuring themselves. The +8.29 composite gap between generated and ground-truth contracts reflects a genuine behavioral difference between LLM generators and expert developers. LLMs follow specifications with literal fidelity: every function name, obligation, and conditional from the specification appears in the code. Expert developers make architectural judgments: they consolidate logic, use efficient state representations, and trade specification literalism for gas efficiency and readability. The metric measures specification fidelity, which is precisely what a code generation evaluation framework should measure. Semantic debiasing narrows the gap to +6.61, confirming the residual difference is behavioral, not artifactual. By releasing 9,000 generated contracts with quality scores, security reports, and compilation results alongside the full pipeline source, we enable the community to replicate, challenge, and extend these findings. The framework makes quality measurable, diagnosable, and auditable: a necessary foundation for deploying generative methods in safety-critical blockchain systems.
6.1
Limitations
Several limitations constrain the framework’s applicability and interpretability. Execution-based correctness is unverified. Compilation success confirms syntactic validity but not semantic correctness in execution. A contract may compile cleanly, pass the LLM-based
Conference’17, July 2017, Washington, DC, USA
audit, and still behave incorrectly on specific input sequences, violate economic invariants under adversarial conditions, or fail statetransition guards in edge cases not exercised by the generator. The human expert study validates the scoring rubric, not the contracts’ runtime behavior, and Slither finds statically-detectable patterns rather than execution-path bugs. This is the most consequential gap for production deployment. Performance degrades with specification complexity. Highcomplexity contracts (8+ functions, 5+ states) achieve only 71.8 average score and 73.2% compilation, compared to 87.2 and 94.1% for low-complexity specifications. The 35.3% prevalence of logic omissions in lower-performing contracts indicates that current LLMs struggle to maintain semantic coherence across lengthy specifications with multiple interacting obligations. Single-pass generation is likely insufficient for the most demanding real-world contracts. The evaluation rubric does not capture gas efficiency. A contract can score 100/100 on the five-dimensional rubric while consuming unnecessarily high gas, making it economically impractical for deployment on mainnet where transaction costs are significant. Gas optimization is a distinct correctness criterion absent from the current framework. The +8.29 evaluation gap is only partially explained. Semantic debiasing closes 20.3% of the gap (to +6.61). The remaining difference reflects genuine behavioral divergence between LLM generators and expert developers, but it also means the rubric penalizes expert architectural choices (gas-optimized boolean state tracking, consolidated function design) that represent valid and often superior implementations. Evaluating architectural quality alongside specification fidelity remains an open problem. Dataset coverage. The FSM-SCG dataset provides structured ground truth but may not represent all real-world contract complexity: cross-contract interactions, oracle dependencies, upgradeable proxy patterns, and regulatory compliance requirements are absent.
6.2
Future Work
Execution-based testing is the most impactful near-term extension. The FSM specifications already present in the dataset can be automatically translated into Foundry test cases: valid FSM transitions become positive test paths, invalid transitions become expectedrevert assertions, and access-control guards generate unauthorizedcaller tests. Integrating this into the refinement loop, where test failures trigger re-entry into the Refiner Agent, would transform evaluation from static scoring into continuous verification. Propertybased tools like Echidna can further stress economic invariants across arbitrary input spaces. Gas optimization metrics would complete the correctness picture. A Gas Optimizer Agent could analyze storage layout, function visibility, loop patterns, and redundant SLOAD operations, producing gas-optimized variants with functional equivalence and reporting gas deltas as a sixth evaluation dimension. Cross-domain validation should be the next empirical step. The Universal Contract Schema is designed to handle NDA agreements, employment contracts, rental agreements, service contracts, and other legal instruments. Validating the pipeline against realworld legal corpora, with domain experts in law or finance serving as ground-truth authors rather than blockchain developers, would
Conference’17, July 2017, Washington, DC, USA
establish whether the quality framework generalizes beyond FSMSCG and Solidity. Multi-model evaluation would reveal how the quality and security patterns observed under GPT-4o-mini change with stronger models (GPT-4o, Claude, Llama). It would also support comparative benchmarking: the released dataset and rubric can serve as a standardized evaluation substrate for any LLM-based smart contract generation system. Regulatory compliance checking represents the gap between technical correctness and legal validity. Securities tokens require transfer restrictions; DeFi protocols must implement sanctions screening; real-estate tokenization carries jurisdiction-specific rules. A Compliance Agent checking generated contracts against regulatory code templates would extend the framework into production legal contexts.
Acknowledgments We thank the IBM Agentics team for framework support, the open-source community for the datasets enabling this research, and Enbei Zhang, Xiaoyang Liu, and Victoria Ruojie Lie for their contributions to the human expert validation study.
References [1] Nicola Atzei, Massimo Bartoletti, and Tiziana Cimoli. 2017. A Survey of Attacks on Ethereum Smart Contracts (SoK). In Proceedings of the 6th International Conference on Principles of Security and Trust. Springer, 164–186. [2] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program Synthesis with Large Language Models. In arXiv preprint arXiv:2108.07732. [3] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems 33 (2020), 1877–1901. [4] Vitalik Buterin. 2014. A Next-Generation Smart Contract and Decentralized Application Platform. In Ethereum White Paper. [5] Agostino Capponi, Garud Iyengar, and Jay Sethuraman. 2023. Decentralized Finance: Protocols, Risks, and Governance. Foundations and Trends in Privacy and Security 5, 3 (2023), 144–188. doi:10.1561/3300000036 [6] Huashan Chen, Marcus Pendleton, Laurent Njilla, and Shouhuai Xu. 2020. A Survey on Ethereum Systems Security: Vulnerabilities, Attacks, and Defenses. In ACM Computing Surveys, Vol. 53. ACM, 1–43. [7] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating Large Language Models Trained on Code. In arXiv preprint arXiv:2107.03374. [8] CrewAI. 2024. CrewAI: Building Multi-Agent Systems with Ease. https://github. com/joaomdmoura/crewAI. [9] Ethereum Foundation. 2024. Solidity Documentation. https://docs.soliditylang. org/. Accessed: 2025-01-15. [10] Josselin Feist, Gustavo Grieco, and Alex Groce. 2019. Slither: A Static Analysis Framework for Smart Contracts. 2019 IEEE/ACM 2nd International Workshop on Emerging Trends in Software Engineering for Blockchain (WETSEB) (2019), 8–15. [11] IBM Research. 2024. IBM Agentics: A Framework for Building Agentic AI Systems. IBM Technical Report (2024). [12] Hao Luo, Yuhao Lin, Xiao Yan, Xintong Hu, Yuxiang Wang, Qiming Zeng, Hao Wang, and Jiawei Jiang. 2025. Guiding LLM-based Smart Contract Generation with Finite State Machine. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. IJCAI, 5869–5877. doi:10.24963/ijcai.2025/653 Main Track. Available at: https://www.ijcai.org/proceedings/2025/653. Dataset: https://github.com/pluto-ms/FSM-Smart-Contract-Generation. [13] Loi Luu, Duc-Hiep Chu, Hrishi Olickel, Prateek Saxena, and Aquinas Hobor. 2016. Making Smart Contracts Smarter. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. ACM, 254–269. [14] Anastasia Mavridou and Aron Laszka. 2018. Designing Secure Ethereum Smart Contracts: A Finite State Machine Based Approach. Financial Cryptography and Data Security (2018), 523–540.
Abhinav Goel, Chaitya Shah, Agostino Capponi, and Alfio Gliozzo
[15] Muhammad Izhar Mehar, Charles Shier, Alana Giambattista, Elgar Gong, Gabrielle Fletcher, Ryan Sanayhie, Henry M. Kim, and Marek Laskowski. 2019. Understanding a Revolutionary and Flawed Grand Experiment in Blockchain: The DAO Attack. Journal of Cases on Information Technology 21, 1 (2019), 19–32. [16] Bernhard Mueller. 2018. Smashing Ethereum Smart Contracts for Fun and Real Profit. In 9th Annual HITB Security Conference. [17] Satoshi Nakamoto. 2008. Bitcoin: A Peer-to-Peer Electronic Cash System. Decentralized Business Review (2008), 21260. [18] OpenAI. 2024. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774 (2024). [19] Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. 2024. Code Llama: Open Foundation Models for Code. arXiv preprint arXiv:2308.12950 (2024). [20] Sally Junsong Wang, Kexin Pei, and Junfeng Yang. 2024. SmartInv: Multimodal Learning for Smart Contract Invariant Inference. arXiv:2411.09217 [cs.SE] arXiv:2411.09217. [21] Sally Junsong Wang, Jianan Yao, Kexin Pei, Hideaki Takahashi, and Junfeng Yang. 2024. Detecting Buggy Contracts via Smart Testing. arXiv:2409.04597 [cs.SE] arXiv:2409.04597. [22] Gavin Wood. 2014. Ethereum: A Secure Decentralised Generalised Transaction Ledger. Ethereum Project Yellow Paper 151 (2014), 1–32. [23] Winnie Xiao, Cole Killian, Henry Sleight, Alan Chan, Nicholas Carlini, and Alwin Peng. 2025. AI Agents Find $4.6M in Blockchain Smart Contract Exploits. https://red.anthropic.com/2025/smart-contracts/. Anthropic AI Safety Research. SCONE-bench: Smart CONtracts Exploitation Benchmark. [24] Zibin Zheng, Shaoan Xie, Hong-Ning Dai, Weili Chen, Xiangping Chen, Jian Weng, and Muhammad Imran. 2020. An Overview on Smart Contracts: Challenges, Advances and Platforms. Future Generation Computer Systems 105 (2020), 475–491.
A
Phase 1: Requirement Specification Agent
The UniversalContractSchema captures the following fields, all extracted with exact specification terminology: • Parties: Name, role, blockchain address, email, entity type (individual/company/organization). • Financial Terms: Amount (optional float), currency (default ETH), purpose, frequency, and due date. • Dates: Date type (e.g., leaseStartDate, deliveryDeadline), value, day of month, and frequency for recurring conditions. • Assets: Type, description, location, quantity, and value. • Obligations: Authorized party, description of the on-chain operation, deadline, and penalty for breach. Every function or operation described in the specification maps to at least one obligation entry; the obligations array is never left empty when functions are present. • Conditions: Exact function names, variable names, state names, state transitions, events, and logic conditions extracted verbatim from the specification. • Termination Conditions: Criteria for contract termination as stated in the specification.
A.1
Parser Agent Task Structure
The parser task prompt instructs the agent to extract EXACT terminology, warns explicitly against generic placeholders, and provides a complete JSON schema template. Key directives include: • Extract exact function names (e.g., initializeLease, not initialize) • Map every described function or operation to an obligation with the authorized party identified (e.g., token holder, contract owner, buyer) • Capture all state transitions in the form StateA → StateB when condition X
SmartEval: A Benchmark for Evaluating LLM-Generated Smart Contracts from Natural Language Specifications
• Return only valid JSON matching the schema structure with no prose or placeholders
B Phase 2: Solidity Generation Agent B.1 Agent Configuration The Generator Agent is configured as a Senior Solidity Smart Contract Engineer with deep expertise in DeFi, tokens, governance, escrow, and marketplace contracts. The agent’s goal specifies producing contracts of 150–400 lines that implement every obligation, and its backstory instructs it to reason about every function in terms of: (a) the real-world operation it represents, (b) the invariants that must hold before and after, and (c) what can go wrong.
B.2
(10) All generated code must serve a concrete purpose; unused state variables are forbidden. (11) Function parameter names must not shadow contract-level state variable names. (12) Public state variable names must not conflict with interface function names.
B.4
Forbidden Patterns
Empty or stub function bodies, unused state variables, silent failures (if (condition) return;), decorative events without state changes, and states named after operations rather than lifecycle phases.
Domain-Specific Generation Rules
The generation prompt includes type-specific mandatory requirements that activate based on contract-type detection: Token Contracts (ERC20): Must implement the full ERC20 interface (transfer, approve, transferFrom, balanceOf, allowance, totalSupply), emit Transfer and Approval events, use an internal _transfer() helper for atomic sender/recipient updates, and correctly track totalSupply for minting and burning operations. Governance/Delegation: Must implement per-account delegate mappings (separate from balance), voting power accumulators updated on every token transfer, checkpoint arrays for historical vote queries via getPriorVotes(), and maintain the invariant: Í Í votingPower = balances at all times. Escrow/Payment: Must track deposits per depositor via a mapping, use call{value} with return-value checking for fund transfers, implement separate conditional release and refund paths, and Í maintain: contract.balance = unreleased deposits. Staking/Rewards: Must implement the rewardPerShare accumulator pattern updated on every stake/unstake/claim, store rewardDebt per staker to prevent double-counting, and support at minimum: stake(), unstake(), claimRewards(), pendingRewards() view.
B.3
Conference’17, July 2017, Washington, DC, USA
Twelve Critical Generation Rules
(1) Every specification guarantee must become verifiable on-chain code execution. (2) System-wide invariants must be maintained across all functions simultaneously. (3) Domain-specific terminology carries full semantic weight (“Token” = complete ERC20 compliance, not a variable name). (4) State machines must be explicitly enforced via enum-based state variables and transition guards. (5) Access control requires explicit justification; every sensitive function has a require()-backed modifier. (6) No silent failures—bare if (condition) return; is forbidden. (7) Economic logic must be complete and conservative, protecting against value loss or exploitation. (8) Time-based conditions must be integrated via block.timestamp with proper deadline enforcement. (9) Events must semantically match completed actions, not serve as decorative logging.
C
Phase 3: Security Auditing Agent
The Auditor Agent examines contracts against eight vulnerability categories with function-specific exploit path descriptions: (1) Reentrancy: External calls followed by state changes; CEI pattern violations; missing nonReentrant guards. (2) Access Control: Missing modifiers on critical functions; improper constructor initialization; functions that should be internal/private but are public/external. (3) Arithmetic Safety: Unchecked operations (pre-0.8.0); division by zero; overflow paths. (4) Ether Handling: Unprotected payable functions; locked ether (payable with no withdrawal path); unvalidated withdrawal amounts. (5) Denial-of-Service: Unbounded loops; external calls inside loops; reverting recipients blocking contract flow. (6) Input Validation: Missing require() statements; missing address(0) checks; missing amount/range validation. (7) Timestamp Dependence: Critical logic gated at a specific timestamp that miners can manipulate. (8) External Call Safety: Unchecked return values from call or delegatecall functions; missing error handling on low-level calls. The audit task returns only valid JSON. No markdown fences or prose are permitted. The structured output includes severity_level (none/low/medium/high/critical), approved (true if severity is none or low), issues (array of function-specific issue descriptions with exploit scenarios), recommendations (array of line-level concrete fixes), vulnerability_count, and security_score (A–F).
D
Phase 4: Refinement Agent
The Refiner Agent (Smart Contract Security Refiner) receives the current Solidity code and the full audit JSON from Phase 3 and produces corrected code. Mandatory remediations include: applying the Checks-Effects-Interactions pattern to all external calls, adding nonReentrant guards where flagged, ensuring all state changes occur before external calls, adding require()-backed access control on sensitive functions, validating all inputs, and checking for zero-address parameters. The agent returns only the complete fixed Solidity code, with no explanations.
Conference’17, July 2017, Washington, DC, USA
The should_refine() gate (Listing 1 in the main paper) blocks downstream release when severity is medium or higher. The default iteration limit is DEFAULT_MAX_REFINEMENT_ITERATIONS = 2, configurable at initialization.
E
Phase 5: ABI Generator Agent
Abhinav Goel, Chaitya Shah, Agostino Capponi, and Alfio Gliozzo
G.2
𝑀1 =
The ABI Agent (Ethereum ABI Specialist) generates the complete JSON ABI array from refined Solidity code. Requirements enforced by the task prompt include: • Every public/external function must appear, including getter functions generated from public state variables. • All parameters must use exact Solidity types (uint256, not uint; explicit array types; uint8 for enums; struct fields expanded individually). • Event parameters carry "indexed": true where declared indexed in Solidity. • stateMutability is set correctly: "pure" (no state access), "view" (read only), "payable" (accepts ETH), or "nonpayable" (default). • Parameter names are preserved exactly as written in the source. • Output is the raw JSON array only; no markdown fences, no prose.
F
Phase 6: MCP Server Agent
The MCP Agent generates a complete Python FastMCP server exposing all contract endpoints as standardized AI-callable tools. The generated server uses Web3.py for blockchain interaction and loads ABI and environment variables from the same directory as the script. For each ABI function, a corresponding @mcp.tool() decorated function is generated: payable functions include the value field in the transaction, non-payable functions omit it, and view/pure functions use contract.functions.X().call() without building a transaction. All transaction tools sign with a private key from environment variables and return {"tx_hash": hash}.
G
Phase 7: Quality Evaluator Agent
The Quality Evaluator Agent (Smart Contract Quality Analyst) performs five-dimensional scoring by reading the specification line by line, then inspecting the generated code for evidence. Scores are precise integers based on explicit point arithmetic, never rounded to the nearest 5.
G.1
Metric Weights Rationale
Business Logic (35%) is most critical because incorrect economic or obligation logic defeats the contract’s purpose. Functional Completeness (25%) is heavily weighted since missing functions prevent required operations entirely. State Machine (15%) and Variable Fidelity (15%) are equally weighted, as incorrect state transitions produce invalid lifecycle behavior and variable fidelity ensures auditability and upgradability. Code Quality (10%) covers important hygiene concerns that are secondary to semantic correctness.
Scoring Formulas
For Functional Completeness (M1), points are calculated as:
10 × |𝐸𝑥𝑎𝑐𝑡𝑀𝑎𝑡𝑐ℎ| + 7 × |𝑆𝑒𝑚𝑎𝑛𝑡𝑖𝑐𝑀𝑎𝑡𝑐ℎ| × 50 +𝑄𝑖𝑚𝑝𝑙 (2) |𝐸𝑥𝑝𝑒𝑐𝑡𝑒𝑑 | × 10
where 𝑄𝑖𝑚𝑝𝑙 is the implementation quality score (up to 50 points) based on logic completeness (+5), access control (+3), event emissions (+2), and input validation (+2) per function. Variable Fidelity (M2) allocates 60 points for state variable completeness and types and 40 points for function parameter accuracy. The composite score uses Equation 1 from the main paper and is always recomputed deterministically in post-processing.
H
Finite State Machines in Smart Contracts
Finite State Machines (FSMs) provide a formal foundation for specifying smart contract behavior. An FSM defines states as discrete phases of a contract’s lifecycle (e.g., Active, Pending, Completed), transitions as conditions and actions triggering state changes, guards as preconditions that must hold for a transition to fire, and actions as on-chain operations executed during transitions. In the FSM-SCG dataset, each entry includes a formal FSM specification alongside the natural language requirements and ground-truth Solidity implementation. The Requirement Specification Agent extracts this FSM structure into the conditions field of the UniversalContractSchema (state names, transitions, events, guard conditions), the Generator Agent enforces it through enum-based state variables and transition guards, and the Quality Evaluator’s M3 metric directly scores the accuracy of the FSM implementation.
I System Components I.1 Core Package Structure • agents.py: All seven agent instantiations and the should_refine() gate logic • translator.py: IBMAgenticContractTranslator main orchestrator, responsible for streaming pipeline coordination across all phases • task_builders.py: Detailed task description constructors for each agent, including domain-specific contract type detection and mandatory requirement injection • programs.py: Legacy IBM Agentics Program class wrappers for backward compatibility • schemas.py: Pydantic data models (UniversalContractSchema and supporting classes) • solidity_compiler.py: Compilation checking via subprocess call to solc/solcjs, with pragma normalization to >=0.4.0
SmartEval: A Benchmark for Evaluating LLM-Generated Smart Contracts from Natural Language Specifications
Conference’17, July 2017, Washington, DC, USA
Table 12: Smart Contract Evaluation Metrics Metric
Wt.
Description
M1: Functional Completeness
25%
Exact and semantic function name matching (exact match +10 pts, semantic match +7 pts per function); implementation quality assessed on logic completeness (+5), access control (+3), event emissions (+2), and input validation (+2) per function. State variable declaration, correct Solidity types, and active use in logic (60 pts); function parameter count, types, descriptive names, and active use (40 pts). Two scoring paths: Path A (explicit FSM states in spec) evaluates state definition, transition implementation, and guard enforcement; Path B (no explicit states) evaluates correctness of stateless design and access control quality. Obligations, financial logic, temporal constraints, conditional flows, and economic invariant enforcement. Most heavily weighted metric. Absence of placeholders and TODOs; presence of NatSpec documentation; appropriate event emissions; overall code structure and organization.
M2: Variable/Parameter Fi- 15% delity M3: State Machine Correct- 15% ness
I.2
M4: Business Logic Fidelity
35%
M5: Code Quality
10%
Agent Configurations
Each agent is instantiated with a role, goal, and backstory that govern its behavior across all pipeline invocations. Complete role and goal definitions for all seven agents are documented in Appendix L. The structural pattern is uniform across agents: Listing 2: Agent Instantiation Pattern agent = Agent( role="<agent role title>", goal="<task objective and output format>", backstory="<domain expertise and behavioral constraints>", llm=crew_llm, verbose=False, allow_delegation=False )
J
Demo Application
The launch_demo.py script provides an interactive demonstration environment. It starts two services: an HTTP server on port 8000 serving a sampler.html interface for browsing the dataset, and a Flask API on port 5000 handling live translation requests from a demo.html frontend. The system automatically opens both browser interfaces. Users can browse the dataset by contract type and complexity, select a specification, observe the complete pipeline execute with real-time streaming output, and see per-metric quality scores with identified strengths and weaknesses. This complements batch processing by providing detailed pipeline introspection for research and debugging.
K
Configuration Options
The IBMAgenticContractTranslator supports the following initialization parameters: • model: LLM selection (default: gpt-4o-mini) • enable_reinforcement: Toggle the severity-gated reinforcement loop (default: True) • enable_deployment: Deploy generated contracts to a Ganache testnet (default: False) • max_refinement_iterations: Maximum audit-refine cycles (default: 2)
L
Agent System Prompts
The following prompts define the role, goal, and behavioral backstory for each agent. Goals and backstories govern how the underlying LLM approaches each task; they are injected at agentcreation time and persist across all invocations within a pipeline run.
L.1
Parser Agent Listing 3: Parser Agent – Role and Goal
role : " Contract Analysis Expert " goal : " Extract every specific term , function name , variable name , state name , party role , financial amount , and obligation from the contract text exactly as written . Produce a fully - populated UniversalContractSchema JSON object with no generic placeholders and with obligations NEVER empty when functions or operations are described ." backstory : " You are an expert contract analyst who reads every sentence carefully . You extract EXACT terminology -- if the contract says ' initializeLease ' you write ' initializeLease ', not ' initialize '. You map every described operation to an obligation with the correct authorized party . You never leave the obligations array empty when functions are described ."
L.2
Generator Agent Listing 4: Generator Agent – Role and Goal
role : " Senior Solidity Smart Contract Engineer " goal : " Implement the EXACT contract specification provided in the task . Read every MANDATORY requirement , every listed obligation , and every domain - specific rule , then implement each one completely with real on - chain logic . Produce a contract of 150 -400 lines that fully and correctly satisfies the specification ." backstory : " You are a senior Solidity engineer with deep expertise in DeFi , tokens , governance , escrow , and marketplace contracts . For every function you write , you ask : what real - world operation does this represent , what invariant must hold before and after , and what can go wrong ? You enforce economic invariants ( token supply conservation , escrow balance accounting ) , temporal logic ( deadlines via block . timestamp ) , and access control ( every sensitive function has a require () - backed modifier ). You NEVER write empty functions , placeholder comments , or stub implementations ."
Conference’17, July 2017, Washington, DC, USA
L.3
Auditor Agent
Listing 5: Auditor Agent – Role and Goal role : " Blockchain Security Auditor " goal : " Identify every exploitable vulnerability in the Solidity contract . For each issue , name the specific function affected and describe the exact exploit path . Provide severity_level , approved boolean , issues array , recommendations array with line - level fixes , vulnerability_count , and security_score in valid JSON ." backstory : " You are a blockchain security expert specializing in Solidity smart contract audits . You methodically check for reentrancy , access control gaps , integer overflow , timestamp manipulation , locked ether , unbounded loops , and input validation failures . Every issue you report names a specific function and explains how an attacker could exploit it . Every recommendation is a concrete code - level fix , not generic advice . You return only valid JSON -- no markdown , no prose ."
Abhinav Goel, Chaitya Shah, Agostino Capponi, and Alfio Gliozzo
L.6
Listing 8: ABI Generator Agent – Role and Goal role : " Ethereum ABI Specialist " goal : " Generate the complete , accurate ABI JSON array for the given Solidity contract . Include every public / external function with correct inputs , outputs , and stateMutability ; every event with all parameters and indexed flags ; and the constructor . Types must be exact Solidity types ( uint256 not uint ). Return ONLY the JSON array ." backstory : " You are an Ethereum developer who has spent years generating and validating ABI specifications . You know that ' uint ' must be ' uint256 ', that view functions have no state mutations , and that indexed event parameters must carry \" indexed \": true . You include every public / external function -- never miss one . You return only the raw JSON array -- no markdown fences , no prose ."
L.7 L.4
Refiner Agent
ABI Generator Agent
MCP Server Generator Agent
Listing 9: MCP Server Generator Agent – Role and Goal role : " MCP Server Developer "
Listing 6: Refiner Agent – Role and Goal role : " Smart Contract Security Refiner " goal : " Fix all identified security vulnerabilities in Solidity smart contracts while maintaining original functionality ." backstory : " You are a Solidity security specialist who fixes smart contract vulnerabilities . Given a contract and a list of security issues from an audit , you rewrite the code to address every vulnerability while maintaining the original functionality . You follow the Checks - Effects - Interactions pattern , add reentrancy guards where needed , implement proper access control , validate all inputs with require () , and ensure no silent failures . You return ONLY the fixed Solidity code ."
L.5
Quality Evaluator Agent
Listing 7: Quality Evaluator Agent – Role and Goal role : " Smart Contract Quality Analyst " goal : " Score the generated Solidity contract across five metrics ( functional completeness , variable fidelity , state machine correctness , business logic fidelity , code quality ). Produce precise integer scores based on exact point calculations -- never round to the nearest 5. Return only valid JSON with metric_1 through metric_5 objects and a composite_score ." backstory : " You are an expert smart contract quality analyst who evaluates generated Solidity code against natural language specifications . You read the specification line by line , then inspect the code and assign scores based on exact evidence -counting matched functions , checking that variables are written and read , verifying state transitions are reachable , and confirming economic invariants are enforced . Your scores are precise (73 not 75) because you show the arithmetic . You return only valid JSON -- no markdown , no prose ."
goal : " Generate production - ready MCP server code for blockchain interaction ." backstory : " You are an expert Python developer specializing in Web3 . py and MCP server generation . You create complete , self - contained FastMCP servers with proper error handling and transaction management . For each ABI function you generate a corresponding @mcp . tool () decorated function : payable functions include value in the transaction , non - payable functions omit it , and view / pure functions use . call () without building a transaction . You load ABI and environment variables from the script directory and return tx_hash for all state - changing operations ."
SmartEval: A Benchmark for Evaluating LLM-Generated Smart Contracts from Natural Language Specifications
Conference’17, July 2017, Washington, DC, USA
Example: Staking Contract Generation and Evaluation Natural Language Specification (excerpt): A staking contract where users can stake tokens during a farming period. The contract transitions through three states: Farming Not Started, Farming Ongoing, and Farming Ended. Users can stake tokens after the start time, withdraw staked tokens, and claim rewards based on their staking period. Rewards cease when the end time arrives. Generated Solidity Code (simplified) Evaluation Results enum State { FarmingNotStarted , FarmingOngoing , FarmingEnded } State public currentState ; IERC20 public stakeToken ; mapping ( address => UserInfo ) public userInfos ; function stake ( uint256 _amount ) external afterStart beforeEnd { require ( _amount > 0) ; stakeToken . transferFrom ( msg . sender , address ( this ) , _amount ); userInfos [ msg . sender ] . amountStaked += _amount ; emit Staked ( msg . sender , _amount ); } function withdraw ( uint256 _amount ) external nonReentrant { require ( _amount > 0) ; userInfos [ msg . sender ] . amountStaked -= _amount ; stakeToken . transfer ( msg . sender , _amount ); emit Withdrawn ( msg . sender , _amount ); }
Composite Score: 87.3/100 (Grade: B) Strengths: • All required functions (stake, withdraw, claim, totalValue) implemented with complete logic • Explicit state machine with enum definition (FarmingNotStarted, FarmingOngoing, FarmingEnded) and transition guards • Strong access control via time-bounded modifiers (afterStart, beforeEnd, nonReentrant) • Proper event emissions for all state changes • Comprehensive input validation (require(_amount > 0)) Weaknesses: • Missing poolInfos variable from specification • No NatSpec documentation comments • Compilation error due to interface placement Metric Breakdown: • M1 Functional Completeness: 92/100 • M2 Variable Fidelity: 85/100 • M3 State Machine: 90/100 • M4 Business Logic: 86/100 • M5 Code Quality: 80/100
Table 13: End-to-end example: staking contract generation from natural language. The pipeline extracts exact state names (FarmingNotStarted, FarmingOngoing, FarmingEnded), generates a complete Solidity implementation with proper state guards, and provides structured feedback identifying both successes and actionable deficiencies.