arXiv:2605.17535v1 [cs.SE] 17 May 2026
AgentModernize: Preserving Business Logic in Legacy Modernization with Multi-Agent LLMs and Behavioral Specification Graphs Sheikh Nazib Ahmed
Marnim Galib
University of Texas at Arlington Arlington, TX, USA [email protected]
University of Texas at Arlington Arlington, TX, USA [email protected]
Abstract—Legacy modernization breaks business logic. Not sometimes — routinely. Most tools and LLM-based approaches treat modernization as syntax translation: convert COBOL to Java, swap PL/SQL for Python, ship it. The implicit rules, edgecase handling, and cross-module constraints that keep production systems running are lost in the process. Nobody notices until something fails in production. We present AgentModernize, a multi-agent framework that treats modernization as a behavioral preservation problem instead. Four specialized agents handle extraction, specification, code generation, and validation. The key intermediate artifact — a Behavioral Specification Graph (BSG) — forces extracted business logic to be explicit and inspectable before any code is generated, creating a trust boundary between what the LLM understood and what it produces. We evaluated on LegacyModernize-8, eight scenarios spanning telecom and banking, using three models (GPT-4o-mini, GPT-4o, GPT-5.3-codex) under a fair protocol: same gold-standard tests, 3 trials, temperature 0.0. The result was consistent across all three: Full AgentModernize with feedback was the only configuration with non-zero mean BER under every backbone. SP-LLM and CoT-LLM scored 0.0% on every scenario, on every backbone. AgentModernize without feedback scored 0.0% mean BER with GPT-4o-mini and GPT-5.3-codex; under GPT-4o it achieved nonzero BER only on S1 (44.4%; 5.6% mean over scenarios) — still far below full AgentModernize with feedback. Mean BER for full AgentModernize was 9.4% (mini), 8.1% (GPT-4o), and 19.4% (codex), with individual scenarios reaching 75.0%. The feedback loop remains decisive: without it, mean BER is 0% on mini and codex, and stays well below the full pipeline even under GPT-4o. The BSG captures 91.2% of gold-standard rules, confirming that the bottleneck is code generation, not extraction. For regulated industries, the pipeline’s traceable artifacts — business rule inventory, BSG, equivalence reports — provide an audit trail that no single-prompt approach can match. Index Terms—legacy modernization, multi-agent systems, large language models, business logic preservation, behavioral equivalence, software engineering
I. I NTRODUCTION Telecommunications carriers, financial institutions, and healthcare providers run their core operations on systems built twenty or thirty years ago. COBOL batch jobs process billing. PL/SQL procedures enforce contract logic. Shell scripts orchestrate provisioning workflows. These systems work — and the business rules encoded in them have been refined through
years of production experience, bug fixes, and regulatory adaptation [1]. But they are expensive to maintain, difficult to integrate with modern platforms, and increasingly fragile as the engineers who wrote them retire [2]. The obvious response is modernization. The less obvious problem is that most modernization efforts destroy the very thing they should preserve: behavioral semantics. Business rules in legacy systems are rarely documented. They live in control flow patterns, conditional branches, exception handlers, and configuration files [3]. A line-by-line COBOL-to-Java translation can compile and run yet silently break edge-case handling, alter validation logic, or drop constraints that were never written down [4]. We have seen this firsthand in telecom provisioning workflows where a syntactically correct translation passed compilation but silently dropped a suspendedaccount exemption that had been in production for fifteen years. The result is a system that looks modern but behaves differently. Large Language Models (LLMs) offer a natural tool for this problem — they can parse legacy code, reason about intent, and generate modern equivalents [5], [6]. But simply prompting an LLM to “convert this legacy code to a modern API” is brittle for at least three reasons. First, context windows cannot hold an entire legacy codebase at once. Second, a single generation pass provides no mechanism to verify that behavior was preserved. Third, a monolithic prompt conflates understanding, specification, transformation, and validation — tasks that benefit from separation [7]. Our approach, AgentModernize, decomposes legacy modernization into four agent-handled phases: 1) A Legacy Analyzer that extracts implicit business rules, control flows, and operational constraints from legacy artifacts. 2) A Specification Generator that encodes the extracted logic into Behavioral Specification Graphs (BSGs) — a structured intermediate representation with preconditions, postconditions, and invariants. 3) A Modernization Transformer that generates modern service implementations from the BSG, preserving behavioral contracts.
4) An Equivalence Validator that checks the generated code against the BSG through automated test generation and differential trace analysis, feeding failures back for iterative correction. The key insight — and we return to this repeatedly in Sections V and VI — is that the BSG acts as a “glass box” between legacy understanding and modern generation. It forces extracted business logic to be explicit and inspectable before any code is written. When the Validator detects a behavioral divergence, it triggers a targeted correction loop, not a full re-generation. The feedback loop turns out to be the single most important mechanism in the entire pipeline (Section V, Table VI). We evaluate on LegacyModernize-8, a benchmark of eight scenarios covering telecom and banking modernization. We measure behavioral equivalence rate, business rule preservation (distinguishing explicit from implicit rules), and manual effort reduction against single-prompt and chain-of-thought LLM baselines. We should note upfront that absolute BER numbers are modest — mean 9.4% with GPT-4o-mini, 8.1% with GPT-4o, 19.4% with codex — but the relative result is what matters: SP-LLM and CoT-LLM score 0.0% under the same protocol. A. Contributions Our contributions are: • A multi-agent framework that decomposes legacy modernization into extraction, specification, transformation, and validation — with a feedback loop for iterative correction. • Behavioral Specification Graphs, an intermediate representation that captures business rules, pre/post-conditions, data constraints, and control flow dependencies in an inspectable, verifiable form. • An automated equivalence checking approach based on test oracle generation from BSG specifications, combined with differential trace analysis. • An empirical evaluation on an eight-scenario benchmark (including a cross-domain COBOL banking scenario), with ablation analysis quantifying each agent’s contribution, a model comparison (GPT-4o-mini vs. GPT-4o) demonstrating complementary strengths, and a frontier model study (GPT-5.3-codex) confirming that AgentModernize with feedback outperforms all other configurations — including the no-feedback variant — under our protocol.
models requires manual effort that often rivals the cost of the modernization itself — which defeats the purpose. Industry tools — IBM’s Rational Asset Analyzer, Micro Focus Enterprise Analyzer, COBOL-to-Java transpilers — handle syntactic translation competently [9]. But “compile-and-run” is not behavioral equivalence. None of these tools verify that the translated system behaves the same as the original, and none of them publish behavioral equivalence metrics we could compare against. Our work targets precisely this gap — and as we show in Section V, even LLM-based approaches fail to preserve behavior without an explicit feedback mechanism. Comparison to Established IRs. The Knowledge Discovery Metamodel (KDM) [24] and Architecture-Driven Modernization (ADM) standards from OMG provide formal representations of legacy system semantics for modernization tooling. Behavioral Interface Specification Languages such as JML [22] and Eiffel [23] encode pre/post/invariants per method. BSG draws from both traditions but differs in three key respects: (i) it is designed for LLM-mediated extraction with per-rule confidence scoring and source traceability; (ii) it explicitly distinguishes explicit from implicit business rules — a categorization absent from KDM/JML; and (iii) it serves as an inspectable trust boundary between LLM extraction and LLM generation, a role unnecessary in pre-LLM modernization workflows. We view BSG as adapting the well-established graph-IR-with-contracts paradigm to the specific demands of LLM-driven modernization, rather than introducing a fundamentally new representation. B. LLMs for Software Engineering LLMs have shown strong results in code generation [5], bug detection [10], code review [11], and program repair [12]. The work most relevant to ours is Pan et al. [13], who found that GPT-4 produces syntactically correct cross-language translations but introduces semantic bugs in complex control flows. This is exactly the failure mode we observed in our own experiments (Section V): every single-prompt baseline compiled and ran but scored 0.0% on behavioral tests. The common thread in all of these studies is that they treat modernization as a single-pass task. One model, one prompt, one output. No intermediate representation, no verification, no feedback. For a self-contained utility function, that might work. For a 280-line COBOL provisioning workflow with thirteen interacting business rules and implicit constraints buried in exception handlers, it does not. Our results in Table III confirm this empirically.
II. R ELATED W ORK A. Legacy System Modernization
C. Multi-Agent LLM Systems
Comella-Dorda et al. [1] catalogued modernization strategies ranging from wrapping to full re-engineering; the horseshoe model [3] formalized the process as iterative abstraction, transformation, and refinement. These frameworks are useful for thinking about the problem. They do not solve it. Modeldriven approaches like MoDisco [8] formalize legacy knowledge using KDM and ADM standards, but constructing the
MetaGPT [14] partitions software development across architect, engineer, and tester agents. ChatDev [15] models an entire software company through multi-agent dialogue. AgentCoder [16] pairs a code generator with an adversarial test generator. Self-Refine [25] and Reflexion [26] show that LLM agents can improve their own outputs through self-generated feedback — a principle our feedback loop shares.
But all of these target greenfield development. Building new code from scratch is a fundamentally different problem than modernizing existing code with undocumented business rules. None of them need to extract implicit logic from decadesold COBOL, represent it in a verifiable form, or confirm that generated code preserves behavioral semantics that were never written down. Compared to these systems, AgentModernize has one advantage and one disadvantage: it solves a harder problem (behavioral preservation, not just code generation), but it relies on a domain-specific intermediate representation (the BSG) that limits its generality to the modernization use case.
Legacy Artifact Bundle
Agent 1: Legacy Analyzer
Business Rule Inventory
Agent 2: Spec Generator
Behavioral Spec Graph
trust boundary
D. Behavioral Equivalence Formal program equivalence — bisimulation, trace equivalence, observational equivalence — provides the theoretical grounding for our verification approach [17]. Full formal verification is intractable for most real-world systems and we do not attempt it. Practitioners rely on lighter-weight alternatives: differential testing [18] compares outputs across system versions; metamorphic testing [19] checks whether known input transformations produce expected output transformations. Our Equivalence Validator combines both ideas, with one important twist: it generates test oracles from the BSG specification, not from the legacy code directly. This avoids the circularity of testing a translation against itself — a subtle but critical point that most automated testing approaches miss. We do not claim formal guarantees. What we claim, and demonstrate in Section V, is that the approach catches a meaningful class of behavioral regressions that single-pass methods miss entirely. III. T HE AGENT M ODERNIZE F RAMEWORK A. Overview The framework takes a bundle of legacy artifacts — source code, configuration files, database schemas, and whatever documentation exists — and produces a modernized service implementation along with a report quantifying how faithfully the original behavior was preserved. Four agents handle this in sequence, each reading from and writing to a shared pipeline state. Fig. 1 shows the architecture. The critical design choice is that the pipeline is not purely feed-forward. When the Equivalence Validator (Agent 4) detects behavioral divergences in the generated code, it feeds specific failure information back to the Modernization Transformer (Agent 3) for targeted correction. This loop runs up to three iterations, allowing the pipeline to self-correct without human intervention. B. Legacy Artifact Bundle The input to the framework is a Legacy Artifact Bundle L = {S, C, D, N } where: • S = Source code files (e.g., COBOL programs, PL/SQL procedures, shell scripts)
Agent 3: Transformer
Modernized Service
feedback
Agent 4: Validator
Equivalence Report Fig. 1. AgentModernize Pipeline Architecture. Solid arrows show the forward pipeline; the dashed red arrow shows the feedback loop from the Validator to the Transformer. The BSG (dotted box) acts as a trust boundary between extraction and generation.
C = Configuration files (e.g., JCL, parameter files, cron schedules) • D = Database schemas (e.g., DDL statements, copybook layouts, data dictionaries) • N = Natural language documentation (optional; e.g., runbooks, comments, requirement documents) •
C. Agent 1: Legacy Analyzer The Legacy Analyzer Agent performs deep analysis of the artifact bundle to extract implicit business logic. This agent operates in three sub-phases: Phase 1a — Structural Parsing: The agent identifies program entry points, module boundaries, data flows, and external dependencies. For procedural legacy code, this involves identifying PERFORM/CALL hierarchies, file I/O operations, and database interactions. Phase 1b — Business Rule Extraction: The agent identifies conditional logic, validation rules, computation formulas, state transitions, and exception handling patterns. Each extracted rule is annotated with: a natural language description, the source location (file, line range), input variables and output effects, and a confidence score (high/medium/low) based on extraction clarity. Phase 1c — Constraint Discovery: The agent identifies implicit constraints including data type restrictions, value ranges, referential integrity rules, temporal ordering requirements, and business-specific invariants (e.g., “order total must equal sum of line items”).
Output: A structured Business Rule Inventory (BRI) — a JSON document containing all extracted rules, constraints, and their metadata. D. Agent 2: Specification Generator The Specification Generator Agent transforms the Business Rule Inventory into a Behavioral Specification Graph (BSG) — the core intermediate representation of AgentModernize. Definition 1 (Behavioral Specification Graph). A BSG is a directed acyclic graph G = (V, E, P re, P ost, Inv) where: • V is a set of operation nodes, each representing a discrete business operation (e.g., “validate order”, “calculate tax”). • E ⊆ V × V × L is a set of labeled edges representing control flow and data dependencies, where L = {sequence, conditional, parallel, error} is the label set. • P re : V → P maps each node to a set of preconditions — logical predicates that must hold before the operation executes. • P ost : V → P maps each node to a set of postconditions — logical predicates that must hold after the operation completes. • Inv : G → P is a set of global invariants — predicates that must hold throughout the entire workflow. Each operation node v ∈ V is further annotated with: inputs(v) (required input data fields with types), outputs(v) (produced output data fields with types), rules(v) (set of business rule IDs from the BRI), and error behavior(v) (expected behavior under error conditions). Concrete Example (S1, Order Validation). The following BSG node, extracted from scenario S1’s COBOL order validation program, illustrates the representation in practice: { "operation": "ValidateDisconnectOrder", "source_rule": "BR-004", "source_location": "ORDER_VALIDATION.cob:118-142", "preconditions": [ "account_status in [’ACTIVE’, ’SUSPENDED’]", "order_type == ’DISCONNECT’" ], "postconditions": [ "suspended accounts proceed for disconnect orders", "non-disconnect orders rejected for suspended" ], "invariants": ["order_total == sum(line_items)"], "confidence": "high" } Listing 1. BSG Node — S1 Disconnect Order Validation
This node captures an implicit rule — suspended accounts are normally blocked, but disconnect orders are exempted — that would be lost in a direct syntax translation. The BSG makes it explicit and inspectable before code generation begins. E. Agent 3: Modernization Transformer The Modernization Transformer Agent generates a modern service-oriented implementation from the BSG. The agent operates under the following transformation principles:
Principle 1 — Operation-to-Endpoint Mapping: Each BSG operation node maps to a discrete function or API endpoint in the modernized system. Principle 2 — Contract Preservation: Pre/postconditions from the BSG are translated into input validation, output assertions, and error handling in the generated code. Principle 3 — Data Model Derivation: Input/output annotations from BSG nodes are aggregated to derive modern data models (e.g., Pydantic models, dataclasses) with type constraints. Principle 4 — Flow Preservation: BSG edge labels (sequence, conditional, parallel, error) determine the orchestration pattern in the modernized implementation. Output: A Modernized Service Package comprising: API endpoint implementations (Python/FastAPI), data models with validation rules, orchestration logic preserving the BSG control flow, error handling matching the BSG error behaviors, and generated documentation mapping each endpoint to its source BSG node. F. Agent 4: Equivalence Validator The Equivalence Validator Agent verifies that the modernized implementation maintains behavioral equivalence with the legacy system as specified by the BSG. The agent performs three types of validation: Type 1 — Contract Verification: For each BSG node, the agent generates test cases that exercise the corresponding modernized endpoint, verifying that preconditions are enforced, postconditions hold, and invariants are maintained across multi-step workflows. Type 2 — Boundary Testing: The agent generates boundary test cases based on BSG constraints (e.g., minimum/maximum values, empty inputs, null handling, type boundaries). Type 3 — Differential Trace Analysis: The agent constructs end-to-end workflow traces from the BSG and executes them against the modernized implementation, comparing actual outputs against expected BSG-specified outputs. Output: A Behavioral Equivalence Report (BER) containing: per-node equivalence status (PASS/FAIL/PARTIAL), overall behavioral equivalence rate, failed test cases with root cause analysis, and recommendations for remediation. Feedback Loop: When the BER identifies behavioral divergences, the pipeline re-invokes the Modernization Transformer with targeted correction instructions, including the failing test cases and expected behaviors. This feedback loop runs for a configurable maximum number of iterations (default: 3). G. Orchestration and State Management The four agents are orchestrated through a stateful pipeline implemented using a directed graph execution engine. The pipeline maintains a shared state object that accumulates artifacts across phases: State = { legacy_bundle: LegacyArtifactBundle, business_rules: BusinessRuleInventory, bsg: BehavioralSpecificationGraph, modern_code: ModernizedServicePackage,
equiv_report: BehavioralEquivalenceReport, iteration: int, status: RUNNING | COMPLETED | FAILED } Listing 2. Pipeline State Object
Each agent reads from and writes to this shared state, enabling traceability from legacy artifacts through intermediate representations to the final modernized output. IV. E VALUATION D ESIGN A. Research Questions We evaluate AgentModernize with respect to the following research questions: • RQ1 (Behavioral Equivalence): To what extent does AgentModernize preserve the behavioral semantics of legacy systems in modernized implementations? • RQ2 (Business Rule Preservation): How effectively does the framework extract and preserve individual business rules through the modernization pipeline? • RQ3 (Effort Reduction): How does the manual effort required for AgentModernize-assisted modernization compare to single-prompt LLM baselines? • RQ4 (Agent Contribution): What is the individual contribution of each agent to the overall modernization quality? • RQ5 (BSG Quality): How accurately does the Behavioral Specification Graph capture the gold-standard business rules? • RQ6 (Model Sensitivity): How does modernization quality vary across LLM model sizes? B. Benchmark: LegacyModernize-8 We construct a benchmark of eight legacy modernization scenarios, seven drawn from the telecommunications domain and one cross-domain banking scenario written in COBOL to test generalization beyond the telecom domain. Each scenario consists of a legacy artifact bundle (COBOL/PL-SQL procedures with embedded business logic) and a gold-standard behavioral specification curated by a domain expert (AIassisted drafting, human-reviewed). Table I summarizes the benchmark scenarios. TABLE I L EGACY M ODERNIZE -8 B ENCHMARK S CENARIOS ID
Scenario
Domain
Complexity
Rules
LOC
S1 S2 S3 S4 S5 S6 S7 S8
Order Validation Billing Disputes Service Activation Circuit Inventory Fault Escalation Contract Renewal Account Migration Bank Transactions
Orders Billing Provisioning Inventory Fault Mgmt Contracts Accounts Banking
Medium High High High High High Very High Very High
12 12 14 13 13 12 15 15
248 195 245 280 230 210 280 310
Scenario Design Principles: Each scenario contains explicitly documented business rules and at least three implicit
rules that must be inferred from code patterns. Complexity levels reflect the number of conditional branches, external dependencies, and state transitions. Scenarios S1–S7 cover telecom-domain challenges: multi-step validation workflows (S1), tier-based conditional logic (S2), capacity management with exemptions (S3), lifecycle management with parent-child dependencies (S4), SLA tracking with tiered escalation (S5), stacking discount calculations with loyalty tiers (S6), and multi-phase migration with rollback semantics (S7). S8 is a synthetic COBOL banking transaction processor — authored to test cross-domain generalization — with multi-type transactions, tier-based withdrawal limits, overdraft protection, transfer fees, fraud detection, and dormant account handling. C. Metrics Both metrics are computed against a gold-standard test suite that is withheld from the pipeline — the system never sees these tests during execution. They are used only for evaluation, ensuring an independent measure of behavioral preservation. The test scripts are synthesized by GPT-4omini from the gold-standard specification and a reference implementation each trial; assertions are executed deterministically by pytest, making BER protocol-dependent on the test-generation prompt. M1 — Behavioral Equivalence Rate (BER): The percentage of gold-standard test cases that pass when run against the generated code. BER =
|passing tests| × 100% |total tests|
(1)
M2 — Business Rule Preservation Score (BRPS): The percentage of gold-standard business rules correctly represented in the modernized output. A rule is considered preserved if: (1) it appears in the BSG, AND (2) the corresponding modernized code enforces it, as verified by targeted test cases. |preserved rules| BRP S = × 100% (2) |total rules| D. Baselines B1 — Single-Prompt LLM (SP-LLM): A single GPT4o-mini prompt that receives the entire legacy artifact bundle and instructions to produce a modernized implementation. No intermediate representation, no verification. B2 — Chain-of-Thought LLM (CoT-LLM): A single GPT-4o-mini prompt with chain-of-thought instructions: “First analyze the business logic, then design the modern API, then implement it.” Still a single model call, but with structured reasoning. B3 — AgentModernize (No Feedback): The full AgentModernize pipeline with the feedback loop disabled, measuring the contribution of iterative refinement. E. Implementation Details All agents use GPT-4o-mini (OpenAI) as the default LLM backend, with GPT-4o used for the model comparison study and GPT-5.3-codex for the frontier model study (Section V-G).
The pipeline is orchestrated using LangGraph (Python) for stateful execution. Test cases are executed using pytest. Legacy artifacts are written in COBOL/PL-SQL; modernized output targets Python (FastAPI). Temperature is set to 0.2 for extraction/specification agents and 0.0 for code generation. The feedback loop runs for a maximum of 3 iterations. The model override is implemented at the pipeline level, allowing any OpenAI-compatible model to be substituted without architectural changes.
B. RQ2: Business Rule Preservation Table IV breaks down preservation scores by rule type.
TABLE II R EPRODUCIBILITY D ETAILS Item
Details
Repository Python Core libraries Models Temperature Trials Random seed Run command
https://github.com/nazib123/agent-modernize 3.11 LangGraph, pytest, OpenAI SDK GPT-4o-mini, GPT-4o, GPT-5.3-codex 0.2 (extraction), 0.0 (generation/eval) 3 per scenario per method N/A (temperature 0.0 for generation) python run_fair_eval_existing.py --model all --trials 3 fair_eval_summary.json, per-scenario folders < $15 for full evaluation suite
Output Total API cost
The absolute numbers are low — we are the first to acknowledge that. But the relative result matters more: the feedback loop is the decisive mechanism, as the ablation in Table VI confirms. Without it, the multi-agent pipeline performs no better than a single-prompt baseline. With it, AgentModernize is the only approach that correctly preserves any business rules under independent gold-standard verification. We discuss why the gap between extraction quality and end-to-end BER is so large in Section VI-A.
V. R ESULTS A. RQ1: Behavioral Equivalence Table III compares behavioral equivalence rates across all scenarios and methods. TABLE III B EHAVIORAL E QUIVALENCE R ATE (%) — FAIR E VALUATION (3 TRIALS ; σ=0.0 EXCEPT AM ON S5: 14.4 AND S8: 6.7) Scenario
SP-LLM
CoT-LLM
AM (No FB)
AM
S1 S2 S3 S4 S5 S6 S7 S8
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
25.0 0.0 0.0 0.0 16.7 0.0 0.0 33.3
Avg
0.0
0.0
0.0
9.4
Under the fair evaluation protocol, AgentModernize with GPT-4o-mini is the only method to pass any gold-standard tests. All three baselines — SP-LLM, CoT-LLM, and AM without feedback — achieve 0.0% BER across all scenarios with this model. AgentModernize achieves non-zero BER on 3 of 8 scenarios: S1 (25.0%), S8 (33.3%), and S5 (16.7%), for a mean of 9.4%. Results are deterministic across 3 trials (σ = 0.0) except for S5 (σ = 14.4) and S8 (σ = 6.7), where the feedback loop’s iterative patching introduces trial-level variation. Notably, S8 (the COBOL banking scenario) achieves the highest single-scenario BER with GPT-4o-mini (33.3%), demonstrating cross-domain generalization to banking.
TABLE IV B USINESS RULE P RESERVATION S CORE (%) — FAIR E VALUATION ID
Tests
Explicit
Implicit
SP-LLM
AM
S1 S2 S3 S4 S5 S6 S7 S8
12 11 10 12 12 10 13 15
8 7 6 8 7 6 8 8
4 4 4 4 5 4 5 7
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
25.0 0.0 0.0 0.0 16.7 0.0 0.0 33.3
Under fair evaluation with GPT-4o-mini, SP-LLM preserves zero business rules across all scenarios. AgentModernize preserves rules in S1 (3/12 tests passing), S5 (2/12), and S8 (5/15). In S1, the preserved rules include the core order validation logic and the implicit suspended-account exemption (BR-004). In S8, the COBOL banking scenario, AgentModernize correctly preserved transaction validation and balance computation logic — demonstrating cross-domain generalization. In S5 (fault escalation), the framework preserved SLAbased escalation thresholds. The remaining scenarios expose a fundamental challenge we return to in Section VI-B: when the generated code’s API structure diverges from the gold-standard expectations, even correctly implemented business logic fails verification. C. RQ3: Residual Behavioral Failures A framework that preserves more behavior automatically should leave fewer behavioral failures for engineers to address. Table V reports failing test counts as a proxy for residual work. TABLE V R ESIDUAL B EHAVIORAL FAILURES — FAIR E VALUATION ID
Tests
SP-LLM
CoT-LLM
AM (No FB)
AM
S1 S2 S3 S4 S5 S6 S7 S8
12 11 10 12 12 10 13 15
12/12 11/11 10/10 12/12 12/12 10/10 13/13 15/15
12/12 11/11 10/10 12/12 12/12 10/10 13/13 15/15
12/12 11/11 10/10 12/12 12/12 10/10 13/13 15/15
9/12 11/11 10/10 12/12 10/12 10/10 13/13 10/15
All baselines fail every gold-standard test across all scenarios. AgentModernize reduces test failures in S1 (−3),
S5 (−2), and S8 (−5) — the same three scenarios where non-zero BER appears in Table III. S8’s reduction (−5) is the largest; the banking scenario’s transaction logic seems particularly amenable to iterative correction, possibly because each transaction type is relatively self-contained. The remaining five scenarios show universal failure. The generated code’s structural divergence from gold-standard API expectations is too large — even correctly implemented business logic cannot pass the standardized tests when the endpoint shape is wrong. This is the structural mismatch problem we discuss further in Section VI-B. D. RQ4: Ablation Study We compare four configurations to isolate the contribution of each component. Table VI reports summary results. TABLE VI A BLATION S TUDY — BER (%) BY C ONFIGURATION (FAIR E VALUATION ) Config
SP-LLM
CoT
AM (No FB)
Full AM
0.0 0/8
0.0 0/8
0.0 0/8
9.4 3/8
Avg BER Non-zero scenarios
The ablation under fair evaluation reveals a clear finding: The feedback loop is essential, not optional. Without it, AgentModernize scores 0.0% — identical to all baselines. The multi-agent decomposition alone (extraction → BSG → code generation) produces code that compiles and runs but does not pass any gold-standard behavioral tests. Only when the Equivalence Validator feeds failing test information back to the Transformer does the generated code begin to match expected behaviors. Feedback is necessary but not sufficient. Even with the feedback loop, AgentModernize achieves non-zero BER in only 3 of 8 scenarios. In S2–S4, S6, and S7 (with GPT-4omini), the structural divergence between generated code and gold-standard expectations was too large for iterative patching to bridge. This suggests the feedback loop is effective at fixing point errors (wrong constant, missing condition) but cannot overcome structural mismatches (different API shape, wrong endpoint semantics). No baseline passes any gold-standard test. SP-LLM, CoT-LLM, and AM without feedback all score 0.0% across all 8 scenarios. Under fair, independent evaluation, singlepass LLM approaches produce code that looks plausible but fails every behavioral check. A reviewer might reasonably ask whether our gold-standard tests are too strict. We believe they are not — they test for specific business rules (e.g., “suspended accounts may proceed with disconnect orders”) that any correct implementation must satisfy. The baselines fail because they produce structurally different code, not because the tests are unreasonable. E. RQ5: BSG Quality Table VII evaluates the Behavioral Specification Graph as an intermediate representation by measuring how many gold-
standard business rules are captured (recall) and what fraction of extracted rules correspond to real rules (precision). TABLE VII BSG RULE E XTRACTION Q UALITY — P RECISION AND R ECALL (%) ID
Gold Rules
BSG Rules
Precision
Recall
Missed
S1 S2 S3 S4 S5 S6 S7
12 12 14 13 13 12 15
25 10 13 12 10 15 14
48.0 100.0 100.0 100.0 100.0 80.0 100.0
100.0 83.3 92.9 92.3 76.9 100.0 93.3
0 2 1 1 3 0 1
Avg
13.0
14.1
89.7
91.2
1.1
Table VII reports S1–S7; S8 is excluded because its goldstandard rules were not structured for BSG-level precision/recall analysis at evaluation time. The BSG achieves a mean recall of 91.2% and precision of 89.7% across the seven evaluated scenarios. This is a key finding: the extraction pipeline captures the vast majority of gold-standard business rules, even though downstream code generation fails to preserve many of them in executable form. The gap between BSG recall (91.2%) and end-to-end BER (9.4%) indicates that the bottleneck is not rule extraction but rather code generation and structural alignment. S1 and S6 achieve 100% recall but lower precision (48% and 80%), meaning the Legacy Analyzer extracted additional rules beyond the gold standard — plausible business logic that the human annotator did not include. S5 has the lowest recall (76.9%), missing 3 rules related to SLA tracking thresholds, which were deeply embedded in nested conditional logic. The BRI-to-BSG transfer is lossless: every rule extracted by Agent 1 is faithfully represented in Agent 2’s BSG output. This confirms that the BSG acts as a reliable intermediate representation with no information loss between pipeline stages. The implication for practitioners is concrete: if you inspect the BSG after Agent 2 and the rules look right, the problem is downstream in code generation, not upstream in extraction. F. Qualitative Analysis Successfully Preserved Implicit Rule (S1, BR-004). The COBOL code contains an implicit exemption where suspended accounts are allowed to proceed with disconnect orders. The Legacy Analyzer correctly identified this, the Specification Generator encoded it as a conditional edge in the BSG, and the Transformer generated appropriate Python logic. The Validator’s test confirmed the exemption was preserved. Cross-Domain Success (S8, Banking). The COBOL banking transaction processor — the most complex scenario at 310 LOC with 15 business rules — achieves 33.3% BER with GPT-4o-mini, the highest single-scenario score for this model. The feedback loop successfully corrected transaction validation, balance computation, and overdraft protection logic across three iterations. This result demonstrates that AgentModernize generalizes beyond its telecom-domain training scenarios.
G. RQ6: Model Comparison To evaluate how model capability affects modernization quality, we ran the full AgentModernize pipeline with GPT4o and GPT-5.3-codex (a code-specialized frontier model) on all eight scenarios under the same fair evaluation protocol. Table VIII shows the results. TABLE VIII M ODEL C OMPARISON — AGENT M ODERNIZE BER (%) FAIR E VALUATION Scenario
GPT-4o-mini
GPT-4o
GPT-5.3-codex
S1 S2 S3 S4 S5 S6 S7 S8
25.0 0.0 0.0 0.0 16.7 0.0 0.0 33.3
58.3 0.0 6.7 0.0 0.0 0.0 0.0 0.0
44.4 0.0 0.0 0.0 75.0 0.0 0.0 35.6
Avg Non-zero
9.4 3/8
8.1 2/8
19.4 3/8
Stronger models yield higher BER. GPT-5.3-codex achieves the highest mean BER (19.4%), more than double GPT-4o-mini’s 9.4%. S1 (order validation) is the only scenario where all three models score above zero, with GPT-4o reaching 58.3% — the highest per-scenario BER for that model. GPT5.3-codex achieves 75.0% on S5 (fault escalation), the highest single-cell BER in the study.
Complementary strengths persist across three models. Each model succeeds on different scenarios: GPT-4o-mini on S1/S5/S8, GPT-4o on S1/S3, and GPT-5.3-codex on S1/S5/S8. No single model dominates all scenarios. An ensemble selecting the best model per scenario could achieve non-zero BER on 4 of 8 scenarios (S1, S3, S5, S8). The framework is model-agnostic. The pipeline architecture, BSG representation, and feedback loop work identically across all three models. The only change is the LLM backend. GPT-4o-mini
GPT-4o
80 BER (%)
Failure Case (S3, All Rules). The Service Activation scenario involves multi-step validation where each step depends on the previous step’s output. The LLM-generated code implemented individual validation functions but failed to wire them together correctly, leading to all tests failing. The feedback loop could not recover because the structural issue was in orchestration, not individual rules. Feedback Loop Convergence (S4). Under internal evaluation (pipeline-generated tests), iteration 1 produced code with 3/11 internal tests passing; the Validator fed back specific pytest failures, and the Transformer’s incremental patch reached 7/11 by iteration 3. However, under fair evaluation against gold-standard tests, S4 scores 0.0% BER — the generated code’s API structure diverges too far from goldstandard expectations for any behavioral test to pass, despite internal improvements. Feedback-Induced Regression (S6). Under internal evaluation (pipeline-generated tests), the initial code without feedback achieved 91.7% on its own test suite. The feedback loop attempted to fix the single failing internal test but introduced cascading errors across the pricing logic, degrading internal performance to 0.0%. Under fair evaluation, both configurations score 0.0% BER — but the internal regression illustrates a real risk: feedback loops can make things worse when fixing one test breaks others. This motivates future work on regression-aware patching that reverts to the best-performing iteration.
60
GPT-5.3-codex 75
58.3 44.4
40
33.3 35.6 25 16.7
20 6.7 0 0 0 0
0 S1
S2
0 0 0 0
S3
S4
0
0 0 0 0 0 0
S5
S6
S7
0
S8
Fig. 2. Per-scenario BER comparison across three models. All three models score non-zero on S1; GPT-5.3-codex reaches 75% on S5. An ensemble could achieve non-zero BER on 4/8 scenarios (S1, S3, S5, S8).
H. RQ6b: Frontier Model Study We designed AgentModernize for cost-effective models like GPT-4o-mini. The obvious question: if you throw a frontier model at the problem, does the pipeline still matter? We ran all four methods with GPT-5.3-codex under the same fair evaluation protocol. Table IX shows the results. TABLE IX F RONTIER M ODEL S TUDY — BER (%) WITH GPT-5.3- CODEX Scenario
SP-LLM
CoT-LLM
AM (No FB)
AM
S1 S2 S3 S4 S5 S6 S7 S8
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
44.4 0.0 0.0 0.0 75.0 0.0 0.0 35.6
Avg Non-zero
0.0 0/8
0.0 0/8
0.0 0/8
19.4 3/8
No crossover. Even with GPT-5.3-codex, SP-LLM and CoTLLM score 0.0% across all eight scenarios. The pattern is the same as with GPT-4o-mini and GPT-4o: feedback is the decisive mechanism, regardless of model capability. The pipeline scales with model capability. Mean BER increases with model strength: 9.4% (mini) to 19.4% (codex). The pipeline amplifies model improvements rather than canceling them out. Stronger models give the feedback loop more correctable errors to work with, which is probably why codex’s 75.0% on S5 is the highest single-cell BER in the study.
Auditability beyond BER. Even if a future model could match AgentModernize’s BER in a single prompt, it would produce no Business Rule Inventory, no BSG, no equivalence report. In telecom and banking, “the model said so” is not an acceptable audit trail. The intermediate artifacts — inspectable, traceable, version-controllable — matter in regulated environments regardless of what the BER numbers say. Cost. Total API cost for the final evaluation — all 8 scenarios, 4 methods, 3 trials each, across three models — was under $15. The entire study is reproducible for the cost of lunch. VI. D ISCUSSION A. BSGs as a Trust Boundary We spent more time debating the BSG than any other design decision, and it turned out to be the most consequential one. The BSG creates a trust boundary: everything upstream is extraction (potentially noisy, confidence-scored), and everything downstream is generation under contract (verifiable against the BSG). Without this boundary, the pipeline is a chain of LLM calls with no checkpoint — you get code at the end and no way to tell where it went wrong. With the BSG, a human reviewer can inspect after Agent 2 and catch extraction errors before any code is generated. The RQ5 results (Table VII) provide empirical support for this design. The BSG captures 91.2% of gold-standard rules with 89.7% precision — yet end-to-end BER is only 9.4% with GPT-4o-mini (19.4% with codex). This 72–82-point gap between extraction quality and code quality pinpoints the bottleneck: the Modernization Transformer (Agent 3), not the extraction pipeline, is the weak link. B. Why the Feedback Loop Is Decisive Under fair evaluation, full AgentModernize with feedback is the only configuration with non-zero mean BER under GPT-4o-mini, GPT-4o, and GPT-5.3-codex (9.4%, 8.1%, and 19.4% respectively), while SP-LLM and CoT-LLM score 0.0% on every backbone. Without feedback, mean BER is 0.0% on GPT-4o-mini (Table VI) and GPT-5.3-codex (Table IX), matching SP-LLM and CoT-LLM there. On GPT-4o, the nofeedback variant is an exception: 44.4% BER on S1 (5.6% averaged across all scenarios) — non-zero, but far below the full pipeline’s 58.3% on S1 and 8.1% mean. The feedback mechanism’s effectiveness depends on whether the initial code has correctable point errors versus structural mismatches. In S1 (order validation), the generated code implemented most business rules correctly but miscalculated a tier threshold — a localized error the Validator identified and the Transformer patched. In S3 (service activation), by contrast, the generated code failed to wire multi-step validation stages together — a structural error that incremental patching cannot fix. The model comparison (Table VIII) confirms this pattern: switching models does not recover S2, S4, S6, or S7; it merely shifts which scenarios have correctable errors.
For practitioners, the implication is clear: the feedback loop should not be optional in any LLM-based modernization workflow. But it is not a silver bullet — when the initial code architecture diverges fundamentally from the gold standard, iterative correction cannot bridge the gap. C. The Implicit Rule Problem Explicit rules are the easy case. A condition like IF ACCOUNT_STATUS = ‘INACTIVE’ THEN REJECT is visible in the source and straightforward to extract. The harder — and more interesting — case is implicit rules: a default value that silently initializes a field, an error handler that doubles as a business constraint, a cross-module data dependency that enforces referential integrity without ever stating it. Our evaluation distinguishes these two categories deliberately. We expect the gap between explicit and implicit preservation rates to reveal where LLM-based extraction struggles, and to motivate future work on specialized extraction strategies for implicit logic. D. Model Capability and Architectural Sophistication We expected frontier models to make the pipeline redundant. They did not. In Tables VIII and IX, SP-LLM and CoT-LLM remain at 0.0% BER; AgentModernize without feedback is likewise 0.0% under GPT-5.3-codex (Table IX). Full AgentModernize with feedback is the only configuration with non-zero mean BER under every backbone we tested; the GPT-4o no-feedback outcome (Section VI-B) does not match that cross-backbone pattern. The pipeline does not become optional as model capability increases. What actually happens is more interesting: the pipeline amplifies model capability. Mean BER scales from 9.4% (mini) to 19.4% (codex). Stronger models produce better initial code, which gives the feedback loop more correctable point errors and fewer structural mismatches to deal with. We think of it this way: a good model gets you 80% of the way on the first pass, and the feedback loop closes some of the remaining gap. A weak model gets you 40% of the way, and the feedback loop has too much ground to cover. The architecture and the model work together, and neither is sufficient alone. We should be transparent about statistical power. With N = 8 scenarios, we cannot make strong claims. Most scenariomethod cells yield σ = 0.0 across 3 trials (temperature 0.0, identical prompts), though S5 and S8 show non-zero σ (up to 14.4) due to feedback-loop variation in which patches get applied in which order. Extraction uses temperature 0.2 but produced identical outputs across all trials in practice. The pattern is consistent — SP-LLM and CoT-LLM achieve 0.0% BER under every backbone — but a larger benchmark would strengthen the claim. The practical takeaway: the BSG pipeline pays for itself at every model tier. With budget models, it is the only path to non-zero BER. With frontier models, it achieves the highest BER in this study (19.4%). In regulated domains — telecom, banking, healthcare — the pipeline’s traceability is an additional advantage that no single-prompt method can match.
E. Comparison Scope We compare against LLM baselines (SP-LLM, CoT-LLM) rather than commercial modernization tools (IBM Rational, Micro Focus, TSRI) or recent agentic coding systems (SWEAgent [20], OpenHands [21]). This is deliberate. Commercial tools target syntactic translation (e.g., COBOL-to-Java) and do not publish behavioral equivalence metrics, making direct comparison infeasible without access to proprietary systems. SWE-Agent and OpenHands solve a fundamentally different problem — resolving GitHub issues in existing codebases — rather than modernizing legacy systems with undocumented business rules. We view these as complementary rather than competing approaches: an agentic coding system could potentially serve as the code generation backend within AgentModernize’s Transformer agent. One baseline we did not include is an “LLM + feedback loop without BSG” configuration — i.e., direct prompting with validator feedback but no intermediate specification. This would isolate whether improvements come from the BSG or from the feedback loop alone. Our ablation (Table VI) shows feedback is necessary, but does not fully prove BSG is sufficient. We flag this as a priority for follow-up work. F. What This Approach Cannot Do (Yet) We evaluated on scenarios of 100–310 lines of code. Real legacy systems are orders of magnitude larger. Scaling AgentModernize would require chunking strategies for the Legacy Analyzer, hierarchical BSGs that compose sub-workflows, and incremental validation that avoids re-running the entire test suite after each correction. These are engineering challenges, not fundamental limitations, but they are non-trivial. Our model comparison (Section V-G) confirms that the framework is model-agnostic across three models: GPT-4omini (9.4%), GPT-4o (8.1%), and GPT-5.3-codex (19.4%), each succeeding on different scenarios. A natural follow-up would evaluate across model families (Claude, Llama, Gemini) and explore ensemble strategies that select the best model per scenario. VII. T HREATS TO VALIDITY Internal validity. All eight benchmark scenarios are synthetic, though S8 targets a different domain (banking) to test cross-domain generalization. While we designed them to reflect realistic complexity — multi-step workflows, implicit constraints, cross-module dependencies — they inevitably lack the full messiness of production legacy codebases that have accumulated decades of patches and undocumented workarounds. The contribution should therefore be interpreted as an early empirical framework and benchmark study, not a production-ready modernization solution. The gold-standard business rules were authored by a single domain expert; a second annotator would strengthen confidence in the ground truth. External validity. We evaluated primarily on telecom scenarios with one banking scenario (S8), using COBOL and PL/SQL legacy languages. Healthcare and government legacy
systems share structural similarities but also have domainspecific patterns (e.g., regulatory audit trails, HIPAA constraints) that we have not tested. The S8 results provide preliminary evidence of cross-domain generalization, but further study is needed. Construct validity. Test-based verification cannot guarantee complete behavioral equivalence — it can only demonstrate the absence of divergence for tested paths. Untested edge cases may harbor latent behavioral regressions. We accept this tradeoff as inherent to any test-based approach and note that formal verification, while stronger in principle, is impractical at the scale we target. Reliability. LLM outputs are non-deterministic. We mitigate this by using low temperature settings (0.0 for code generation, 0.2 for extraction). Our fair evaluation uses 3 trials per scenario; most cells yield σ = 0.0, though S5 (σ = 14.4) and S8 (σ = 6.7) show trial-level variation introduced by the feedback loop’s iterative patching. We evaluated with three models (GPT-4o-mini, GPT-4o, GPT-5.3-codex), confirming that results are not artifacts of a single model’s behavior. Model availability. GPT-5.3-codex is a frontier model with restricted availability and significantly higher cost than GPT4o-mini. The scaling pattern observed (Section V-H) may not generalize to other frontier models or future model releases, and the cost differential limits practical applicability for largescale modernization. Selection bias. The first author designed the benchmark scenarios and authored the gold-standard business rules. This dual role creates a risk that scenarios are inadvertently tuned to the framework’s strengths. We mitigate this partially by including S8 (a banking-domain scenario structurally different from S1–S7), but a fully independent benchmark authored by separate domain experts would strengthen validity. LLM-as-evaluator. The Equivalence Validator (Agent 4) is itself an LLM, raising the question of whether it introduces systematic bias into the BER metric. Our fair evaluation protocol mitigates this: gold-standard test scripts are synthesized by GPT-4o-mini from the scenario specification and a reference implementation (not by the pipeline under evaluation) and executed deterministically by pytest. BER is therefore protocoldependent on the test-generation prompt. The Validator’s LLM role during the pipeline run is separate; the fair-eval BER is computed entirely from deterministic test execution against independently generated tests. VIII. A RTIFACT AVAILABILITY The AgentModernize framework implementation, LegacyModernize-8 benchmark scenarios (COBOL/PL/SQL source, gold-standard business rules, and behavioral test suites), BSG schema definitions, evaluation scripts, and sample prompts are publicly available at: https://github.com/nazib123/agent-modernize. IX. AI U SE D ISCLOSURE AI tools (OpenAI GPT-4o) were used for grammar editing and formatting assistance during manuscript preparation. All
technical content, experimental design, implementation, evaluation, and analysis were performed by the authors. X. C ONCLUSION Legacy modernization is not a translation problem. It is a behavioral preservation problem. We present a framework that treats it as such, and the results — while modest in absolute terms — confirm one thing clearly: the feedback loop changes everything. AgentModernize decomposes modernization into extraction, specification, transformation, and validation, connected through Behavioral Specification Graphs. SP-LLM and CoTLLM scored 0.0% BER across all eight scenarios on every backbone. The pipeline without feedback scored 0.0% mean BER on GPT-4o-mini and GPT-5.3-codex; on GPT-4o it achieved non-zero BER only on S1 (44.4%; 5.6% mean over scenarios), still well below full AgentModernize with feedback. The only configuration with non-zero mean BER under all three backbones was full AgentModernize with feedback (9.4% GPT-4o-mini, 8.1% GPT-4o, 19.4% GPT-5.3codex). The BSG captures 91.2% of gold-standard rules; the bottleneck is code generation, not extraction. We expected frontier models to make the pipeline redundant. They did not. The pipeline amplifies model capability rather than competing with it. And the artifacts it produces along the way — business rule inventory, BSG, equivalence reports — are the kind of audit trail that regulated industries need and that no single-prompt method can provide. Three directions seem most promising: hierarchical BSGs for codebases larger than 300 lines, ensemble strategies that select the best model per scenario (our three-model results suggest 4/8 coverage is within reach), and a human-in-the-loop study to test whether domain experts actually trust modernized code more when they can inspect the intermediate BSG. That last one, we think, is the most important question this work opens up. R EFERENCES [1] S. Comella-Dorda, K. Wallnau, R. Seacord, and J. Robert, “A survey of legacy system modernization approaches,” Software Engineering Institute, Carnegie Mellon University, Tech. Rep. CMU/SEI-2000-TN003, 2000. [2] H. P. Breivold, I. Crnkovic, and M. Larsson, “A systematic review of software architecture evolution research,” Information and Software Technology, vol. 54, no. 1, pp. 16–40, 2012. doi:10.1016/j.infsof.2011.08.002 [3] R. Kazman, S. G. Woods, and S. J. Carrière, “Requirements for integrating software architecture and reengineering models: CORUM II,” in Proc. Working Conference on Reverse Engineering (WCRE), 1998. doi:10.1109/WCRE.1998.723185 [4] S. Rugaber and K. Stirewalt, “Model-driven reverse engineering,” IEEE Software, vol. 21, no. 4, pp. 45–53, 2004. doi:10.1109/MS.2004.23 [5] M. Chen et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021. [6] R. Li et al., “StarCoder: may the source be with you!” arXiv preprint arXiv:2305.06161, 2023. [7] C. Hou et al., “Large language models for software engineering: A systematic literature review,” ACM Transactions on Software Engineering and Methodology, 2024. doi:10.1145/3695988
[8] H. Brunelière, J. Cabot, G. Dupé, and F. Madiot, “MoDisco: A model driven reverse engineering framework,” Information and Software Technology, vol. 56, no. 8, pp. 1012–1032, 2014. doi:10.1016/j.infsof.2014.04.007 [9] A. De Lucia, A. R. Fasolino, and M. Napoli, “An approach for reverse engineering of COBOL-based applications,” in Proc. European Conference on Software Maintenance and Reengineering, 2001. doi:10.1109/CSMR.2001.914982 [10] Z. Li et al., “Large language models for code analysis: Do LLMs really do their job?” arXiv preprint arXiv:2310.12357, 2023. [11] Z. Li et al., “Automating code review activities by large-scale pretraining,” in Proc. 30th ACM Joint European Software Engineering Conference (ESEC/FSE), 2022. doi:10.1145/3540250.3549081 [12] C. S. Xia and L. Zhang, “Less training, more repairing please: revisiting automated program repair via zero-shot learning,” in Proc. 30th ACM Joint European Software Engineering Conference (ESEC/FSE), 2022. doi:10.1145/3540250.3549101 [13] R. Pan et al., “Lost in translation: A study of bugs introduced by large language models while translating code,” in Proc. IEEE/ACM International Conference on Software Engineering (ICSE), 2024. doi:10.1145/3597503.3639226 [14] S. Hong et al., “MetaGPT: Meta programming for a multi-agent collaborative framework,” in ICLR, 2024. [15] C. Qian et al., “ChatDev: Communicative agents for software development,” in ACL, 2024. doi:10.18653/v1/2024.acl-long.810 [16] K. Huang et al., “AgentCoder: Multi-agent-based code generation with iterative testing and optimisation,” arXiv preprint arXiv:2312.13010, 2023. [17] R. Milner, Communication and Concurrency. Prentice Hall, 1989. ISBN: 978-0-13-115007-2. [18] W. McKeeman, “Differential testing for software,” Digital Technical Journal, vol. 10, no. 1, pp. 100–107, 1998. [19] T. Y. Chen, S. C. Cheung, and S. M. Yiu, “Metamorphic testing: A new approach for generating next test cases,” Tech. Rep. HKUST-CS98-01, 1998. [20] J. Yang et al., “SWE-agent: Agent-computer interfaces enable automated software engineering,” in NeurIPS, 2024. [21] X. Wang et al., “OpenHands: An open platform for AI software developers as generalist agents,” arXiv preprint arXiv:2407.16741, 2024. [22] G. T. Leavens, A. L. Baker, and C. Ruby, “JML: A notation for detailed design,” in Behavioral Specifications of Businesses and Systems, H. Kilov, B. Rumpe, and I. Simmonds, Eds. Springer, pp. 175–188, 1999. doi:10.1007/978-1-4615-5229-1_12 [23] B. Meyer, “Applying design by contract,” IEEE Computer, vol. 25, no. 10, pp. 40–51, 1992. doi:10.1109/2.161279 [24] Object Management Group, “Architecture-Driven Modernization: Knowledge Discovery Meta-Model (KDM), v1.4,” OMG Standard formal/2016-02-01, 2016. Available: https://www.omg.org/spec/KDM/1. 4 [25] A. Madaan et al., “Self-Refine: Iterative refinement with self-feedback,” in Advances in Neural Information Processing Systems (NeurIPS), 2023. [26] N. Shinn et al., “Reflexion: Language agents with verbal reinforcement learning,” in Advances in Neural Information Processing Systems (NeurIPS), 2023.