Article
Iterative Audit Convergence in LLM-Managed Multi-Agent Systems: A Case Study in Prompt Engineering Quality Assurance Elias Calboreanu 1,2 1 2
arXiv:2605.12280v1 [cs.SE] 12 May 2026
*
Swift (North) AI Lab, The Swift Group, LLC, Maryland, USA Capitol Technology University, Laurel, MD 20708, USA Correspondence: [email protected]
Abstract Prompt specifications for multi-agent large language model (LLM) systems carry data contracts and integration logic across many interdependent files but are rarely subjected to structured-inspection rigor. This paper reports a single-system empirical case study of iterative, agent-driven auditing applied to AEGIS (Autonomous Engineering Governance and Intelligence System), a production seven-lane orchestration pipeline whose prompt-specification surface comprises approximately 7150 lines: 6907 across seven lane PROMPT.md files and a 245-line shared Ticket Contract. Nine sequential audit rounds, executed by Claude sub-agents using a checklist-driven walkthrough adapted from Weinberg and Freedman, surfaced 51 prompt-specification consistency defects, distinct from the 51 STRIDE-categorized adversarial code findings reported in the companion preprint. Per-round counts were 15, 8, 12, 2, 8, 1, 4, 1, and 0. We report a seven-category post-hoc defect taxonomy with explicit coding rules, observed non-monotonic convergence consistent with cascading edits and audit-scope expansion, and an audit protocol distilled from the study, with the final locked checklist released as a reproducibility appendix. Single-file review missed defect classes that were surfaced only by later expanded-scope rounds in this system. The same LLM family authored and audited the specifications; replication with dissimilar models and human reviewers is required before generalization. Keywords: prompt engineering; multi-agent systems; iterative auditing; quality assurance; large language models; specification consistency; software inspection; cross-document validation; agentic automation; configuration management
1. Introduction 1.1. Motivation Prompt engineering for multi-agent systems presents a quality assurance challenge distinct from traditional software testing. When an LLM-orchestrated pipeline spans multiple autonomous lanes, each with its own behavioral specification, data contracts, and integration points, the prompt layer becomes the system’s source of truth. Errors in prompt specifications can propagate as runtime failures: wrong field names may break data handoffs, stale references can trigger invalid API calls, and missing labels can corrupt downstream classification. The empirical evidence presented in this paper is consistent with such failure modes but does not establish their relative frequency. Yet prompt specifications are rarely subjected to the rigor applied to source code. They are authored in natural language, reviewed informally, and updated incrementally. When a design change touches one document, ripple effects across other documents are easy to miss. Software inspection research has long established that structured review catches defects that testing alone may miss [1–3]. This paper extends that principle to prompt specifications and reports the iteration required in one production specification surface.
2 of 13
1.2. Research Questions The research questions are stated descriptively rather than causally: • •
RQ1. What defect classes remained after the initial audit pass and surfaced only through later iterative expanded-scope rounds in this case study? RQ2. What convergence behavior was observed, and how many rounds were required for this 7150-line, eight-document specification surface?
1.3. Contributions This paper offers three contributions, each scoped to the case study: •
•
•
C1 (taxonomy). A seven-category post-hoc taxonomy of the 51 surfaced defects, with category definitions, coding rules, and one representative example per category (Section 3 and Appendix G). Percentages are descriptive; no inferential claims are made. C2 (observed convergence). Per-round defect counts and a category-by-round cross-tabulation showing observed non-monotonic intermediate behavior consistent with cascading edits and audit-scope expansion. We do not claim that the curve is predictable in advance; we report what was observed and offer a phase-based interpretation. C3 (audit protocol distilled from the study). An audit protocol distilled from the case study, with the final locked checklist, prompt template, finding schema, convergence criterion, and context-loading chronology released as a reproducibility appendix (Appendix F and Appendix H). We distinguish protocol used during the study, which co-evolved with discovery, from protocol recommended after the study.
We do not claim single-pass full-scope review is universally insufficient — only that single-file review cannot, by construction, detect cross-file defects, and that one-pass termination would have left later-discovered defects unresolved under this study’s protocol. A single-pass full-scope control was not run. 1.4. Background and Related Work 1.4.1. Software Inspection and Structured Review Fagan-style code inspection [1] established that structured review detects defects that testing alone may miss. Boehm and Basili’s “Software Defect Reduction Top 10 List” [2] reaffirmed this in 2001 with broader empirical evidence, identifying peer reviews and inspections as cost-effective defect-reduction mechanisms across multiple project contexts. Weinberg and Freedman [3] articulated lighter-weight walkthroughs and reviews, observing that effectiveness depends on checklist quality and reviewer expertise without requiring the full Fagan moderator/reader/recorder/inspector role separation. Our audit protocol adapts the walkthrough tradition rather than full Fagan inspection. We did not have multiple human inspectors with separate role assignments; we had one LLM family acting as inspector against a checklist authored by the same family. The locked checklist in Appendix F serves as the inspection guide, the sub-agent serves as the inspector, and the convergence criterion replaces the single-pass exit gate. The key limitation, that the inspector belongs to the same LLM family that authored the specifications, is documented in Section 4.5. 1.4.2. Specification Drift, Aging, and Traceability Parnas [4] identified software aging as the phenomenon by which documentation accuracy degrades as the system it describes evolves. Gotel and Finkelstein [5] analyzed the requirements traceability problem, finding that cross-artifact consistency is the most labor-intensive and error-prone aspect of requirements management. Our findings are consistent with both observations in the promptspecification domain: 12 of the 51 defects (23.5%) were stale Jira references and five (9.8%) were cross-lane schema mismatches.
3 of 13
1.4.3. LLM Self-Evaluation and Self-Refinement Self-consistency [6] samples multiple reasoning paths and selects the most consistent answer within a single context window. Self-Refine [7] iterates a critique-and-revise loop within a single LLM. Our work applies an iterative principle to specification artifacts spanning many persistent files across multiple sessions, not to single completions within one context. The persistence of file-based state changes the convergence dynamics: drift can accumulate across rounds in ways that intra-context refinement is not designed to address. 1.4.4. Recent LLM-Based SE and Agentic Quality Assurance He, Treude, and Lo [8] survey LLM-based multi-agent systems across the software development lifecycle and identify robustness and synergy gaps. Cemri et al. [9] propose a 14-failure-mode taxonomy for multi-agent LLM systems with human-annotated inter-rater agreement (κ = 0.88); their work directly motivates Appendix G’s coding rules. Lubos et al. [10] evaluate LLMs for requirements quality assurance against ISO 29148-style requirement-quality characteristics. Chen et al. [11] argue that prompts are software artifacts requiring lifecycle methods (the promptware position). Patil [12] and the agentic-AI assessment-framework literature [13] are consistent with the position that final-pass success on its own is an inadequate quality signal. Repository-level agentic code reasoning benchmarking [14] provides a parallel for cross-file consistency challenges. Yang et al.’s SWE-agent [15] establishes a leading framework for LLM-driven single-agent repository interaction; AgentBench [16] provides standardized evaluation of LLMs-as-agents in synthetic environments. The present case study complements both lines of work by reporting production-deployment evidence rather than benchmark performance. MetaGPT [17] operationalizes Standard Operating Procedures for multi-agent collaboration; AEGIS’s seven-lane structure is a domain-specialized instance of that pattern. Naqvi, Baqar, and Mohammad [18] contrast iterative closed-loop multi-agent testing against static single-shot test generation. 1.4.5. Companion Work The companion preprint [19] addresses the runtime behavior of the same AEGIS pipeline, including STRIDE-based adversarial testing and FMEA-based safety analysis. The 51 specification defects reported here are distinct from the 51 STRIDE-categorized adversarial code findings reported in the companion: those are P0–P3 priority code vulnerabilities (lock handling, race conditions, subprocess management) found by a two-person human review panel across three injection rounds; ours are spec-consistency defects (version drift, stale Jira refs, schema mismatches) found by Claude sub-agents across nine audit rounds. The numerical coincidence is genuine but the populations are distinct in source, method, and content. This paper exercises the prompt-specification layer that configures AEGIS; it does not validate the upstream Context Engineering [20], MANDATE [21], LATTICE [22], or TRACE [23] frameworks themselves. Those frameworks provide system context only.
2. Materials and Methods 2.1. System Under Test: AEGIS AEGIS (Autonomous Engineering Governance and Intelligence System) is a seven-lane agentic automation pipeline operating against a Jira backlog that had grown to 2100+ tickets as of April 2026, up from the companion paper’s approximately 1602-row consolidated backlog snapshot reported in [19]. Each lane is a Claude agent governed by a PROMPT.md file that defines: • • • • •
Behavioral rules and constraints; Data schemas for inter-lane communication; Model Context Protocol (MCP) [24] tool permissions and rate limits; Jira transition contracts (status IDs, label conventions); Worker pool configuration (parallelism, isolation);
4 of 13
•
Quality gates.
The seven lanes share a common TICKET_CONTRACT.md that codifies status-transition IDs, creation authority by lane, mandatory labels, and shared schemas. The lanes communicate via JSON intermediary files (for example, Lane 2 writes findings to lane_02_untracked_findings.json; Lane 3 ingests them and creates Jira tickets). A design change in any one lane can invalidate assumptions in three to four other documents. AEGIS is also governed by separately documented Context Engineering [20], MANDATE [21], LATTICE [22], and TRACE [23] specifications. These frameworks provide system context, but the present study evaluates the consistency of the prompt-specification layer rather than validating the frameworks themselves. Specification-surface metrics are summarized in Table 1. Table 1. Specification surface metrics for the AEGIS pipeline at the time of the audit. Line counts as reported in the companion paper, Section 6.1.
Metric
Value
Configuration files audited Lane PROMPT.md files Total lane PROMPT lines Ticket Contract lines Total specification lines
8 (seven lane PROMPT.md files plus one shared TICKET_CONTRACT.md) 7 (one per lane: Lanes 1–7) 6907 245 ~7152 37 identified (catalogued by grep + manual confirmation across the eight files) 4 (fix_queue, untracked_findings, connectivity, fallback) 4 (To Do, In Progress, On Hold, Done) 4
Cross-document references Shared data schemas Shared transition IDs Ticket-creating lanes
2.2. Case Study Design We follow Runeson and Höst [25] case-study reporting conventions. Case selection rationale. AEGIS was selected because (a) it is a production-deployed multi-agent system whose specification surface had grown organically across lane consolidations and architectural changes, (b) the author had artifact-level access required to run the audit, and (c) the system had reached sufficient operational maturity for cross-document audit to surface meaningful defects. Unit of analysis. The unit of analysis is the prompt-specification surface (eight files) and the audit process (nine rounds). Convergence is measured at the audit-round level; defect classification is measured at the individual-finding level. Propositions. RQ1 implies the proposition that defect classes exist that single-file review cannot detect by construction. RQ2 implies the proposition that iterative auditing exhibits non-monotonic convergence in this case. Chain of evidence. Audit logs (per-round finding lists with file/line citations) are the primary data; the AEGIS specification files are the substrate; the author’s classification is the interpretation layer. The reproducibility bundle (Appendix H) preserves the audit logs and classification labels. 2.3. Audit Protocol (As Used in the Study) Each audit round consisted of spawning one or more Claude sub-agents (via the Agent tool) with a checklist covering seven dimensions: 1. 2. 3. 4. 5. 6.
Version consistency within and across files; Cross-lane data contract alignment (field names, schemas); Jira permission boundaries (which lanes may create, comment, transition); Label conventions per the Ticket Contract; Lane-count propagation (all references must cite seven lanes); Cadence and scheduler alignment;
5 of 13
7.
Internal contradictions between sections of the same document.
Agents read each file top-to-bottom and returned structured findings with exact line numbers. Fixes were applied between rounds using targeted edits, and verification greps confirmed each fix. Following Weinberg and Freedman [3], we adopted walkthrough discipline rather than full Fagan inspection: there was one inspector (the LLM family) rather than the moderator/reader/recorder/inspector role separation that Fagan inspection requires. Section 4.5 documents the limitations this entails. The audit protocol co-evolved with the fixes. Early rounds used per-file agents; later rounds explicitly loaded producer and consumer specifications together for cross-lane checks. Section 2.6 documents this chronology. Consequently, the convergence curve in Section 3 reflects two interleaved processes—defect remediation and audit refinement—and should be read as describing the joint process rather than as evidence about a fixed audit function. The final locked protocol used after the study is reproduced in Appendix F; that locked version is the one we recommend for replication. 2.4. Defect Coding The seven-category taxonomy reported in Section 3.2 was derived post-hoc by a single coder (the author) and is therefore subject to single-coder bias. This is foregrounded as a central methodological limitation, not a peripheral one. Cemri et al. [9] achieved κ = 0.88 inter-rater agreement on a comparable taxonomy by using multiple coders; the present study did not. Future replications should run a quick inter-rater check on a defect subsample. Category definitions, decision rules for boundary cases, a worked boundary case, and one representative example per category are provided in Appendix G. Categories were treated as mutually exclusive at coding time; any defect that could plausibly fit two categories was assigned to the one matching the trigger of detection, not the apparent fix. 2.5. Convergence Criterion (Operational, Not Mathematical) The audit loop terminated when a full-scope audit (all eight files, all seven checklist dimensions) returned zero findings on round 9. Operationally, the loop resembles audit-to-quiescence: repeat audit-and-fix until a full-scope pass returns no new findings. This is analogous to fixed-point iteration but the analogy is methodological rather than mathematical: the audit function changed across rounds (Section 2.3 caveat), the LLM auditor is stochastic, and a single clean pass is not a mathematical fixed point of a stationary operator. A stricter protocol would require two consecutive clean passes to guard against oscillation. We did not adopt that stricter criterion in this study; future replications should consider doing so. Appendix F.3 specifies the recommended stricter criterion. 2.6. Context-Loading Chronology The audit protocol used during the study evolved as documented in Table 2. The nine rounds were conducted across approximately five weeks (early April through early May 2026), after Lane 7 was introduced on 31 March 2026 per the companion paper [19]. The specification surface was held stable across audit rounds modulo the targeted fixes applied between rounds; no architectural changes were introduced during the audit window.
3. Results 3.1. Convergence Data Table 3 presents the per-round convergence data. The audit surfaced 51 total prompt-specification defects, distinct from the 51 STRIDE-categorized adversarial code findings reported in [19] (Table 4 of the companion). Per-round counts were 15, 8, 12, 2, 8, 1, 4, 1, and 0. Figure 1 shows the convergence curve with per-round discovery rates and cumulative totals.
6 of 13
Table 2. Context-loading chronology for the nine audit rounds. Round
Files loaded per call
Cross-lane comparison?
1 2 3
1 1 1–2
No No Partial
4
2–3
Yes
5 6
2–3 8
Yes Yes
7
8
Yes
8
8
Yes
9
8
Yes
Notes Per-file structural pass Per-file structural pass First cross-lane checks for label conventions First explicit producer-consumer schema comparison Stabilized cross-lane comparison strategy First full-scope pass Full-scope, including changelog and example data Full-scope; surfaced priority_score vs fix_priority mismatch Clean pass; no findings
Table 3. Audit convergence across nine rounds. Round
Issues found
1 2
15 8
3
12
4 5 6 7 8 9
2 8 1 4 1 0
Primary category
Cumulative
Structural: stale versions, removed features, missing lanes Operational: fallback references, footer versions, lock commands Semantic: misleading env-var docs, Jira templates, cadence references Schema: timing assumptions, fix_queue field alignment Contract: fallback_enabled flag, labels, Status Contract coverage Changelog: stale fallback queue reference in version history Cross-lane: formula mismatch, planned scripts, transition scope Data contract: priority_score vs fix_priority field-name break Clean pass: zero issues across all eight files
15 23 35 37 45 46 50 51 51
3.2. Defect Taxonomy Table 4 presents the post-hoc seven-category taxonomy; Figure 2 shows the same distribution as a horizontal bar chart with the highest-severity category highlighted. Category definitions, coding rules, and examples are in Appendix G. Stale Jira references (23.5%) were the most prevalent category, reflecting ripple effects of an earlier architectural change that removed Jira ticket creation from Lane 2. Cross-lane schema mismatches (9.8%), though fewer in count, were author-coded as high severity because such mismatches can cause silent runtime failures (Section 3.3 and Appendix F). Table 4. Defect classification (n = 51). Percentages sum to 99.9 due to rounding. Defect type
n
%
Version drift Stale Jira references Cross-lane schema mismatch Missing recent-extension coverage (Lane 7) Label / contract gaps Semantic misleading text Formula / timing drift
9 12 5 7 6 8 4
17.6 23.5 9.8 13.7 11.8 15.7 7.8
Severity distribution (author-coded per Appendix F.4 rubric): high severity, 5 (9.8 %); medium, 32 (62.7 %); low, 14 (27.5 %). The high-severity category and the cross-lane schema mismatch category are coextensive in this study: every cross-lane schema mismatch was coded high severity per Appendix F.4’s silent-runtime-failure criterion, and no other defect class reached the high-severity threshold. Defect density. 51 defects across approximately 7150 specification lines yields approximately 7.1 defects per thousand lines of specification. This provides an order-of-magnitude reference point, but direct comparison to code-inspection rates from the Fagan-tradition empirical literature [1,2] is not
7 of 13
Audit Convergence Across 9 Rounds (n = 51 defects) 15
50
Cumulative findings
Issues found per round
15.0
60
51
Issues found per round Cumulative findings
17.5
12
12.5 10.0
40
8
30
8
7.5
20
5.0
4 2
2.5 0.0
1
2
3 scope expansion
4
1
1
5 6 scope expansion Audit round
7
8
10
0
0
9
Figure 1. Audit convergence across nine rounds. Bars show per-round defect counts (left axis); the line shows cumulative totals (right axis). Shaded regions in rounds 3 and 5 indicate observed scope expansion.
warranted without comparable inspection-effort normalization: the artifact class (natural-language prompt specifications versus source code), the inspector (LLM versus human), the defect definition, and the inspection depth all differ.
Defect Taxonomy Distribution (n = 51, post-hoc 7-category coding) Stale Jira References
12 (23.5%)
Version Drift
9 (17.6%)
Semantic Misleading Text
8 (15.7%)
Missing Lane 7 Coverage
7 (13.7%)
Label / Contract Gaps
6 (11.8%)
Cross-Lane Schema Mismatch
5 (9.8%)
Formula / Timing Drift
Highest severity (silent runtime failure risk)
4 (7.8%) 0
2
4
6
8
Number of defects (n = 51)
10
12
14
Figure 2. Defect taxonomy distribution (n = 51). Cross-lane schema mismatches (highlighted) are the highestseverity category by author coding.
3.3. Key Observations Observed non-monotonic convergence. Rounds 3 and 5 surfaced more issues than round 2. We interpret this as the audit scope expanding rather than a regression: structural fixes in earlier rounds revealed previously masked inconsistencies. Auditor-variance vs. mechanism interpretation is discussed in Section 4.5.4. Cross-lane defects are the hardest to catch with single-file review. The most operationally consequential defect (round 8) was a pre-remediation field-name mismatch: Lane 3 emitted priority_score while Lane 4 was specified to consume fix_priority. This could plausibly have caused a silent runtime failure, such as empty sort results, with no error signal. The defect was remediated before the
8 of 13
companion paper’s final architecture review, and both lanes now use priority_score (companion paper [19], Section 3.2). By construction, single-file review cannot detect this class of defect. A full-scope single-pass review might detect it; we did not run such a control. Specification aging outlasts structural fixes. After all structural issues were resolved, rounds 6 through 8 continued to surface semantic inconsistencies. These were not detected by the grepbased verification used in this study. Parnas [4] identified this phenomenon as software aging; the prompt-specification analog observed here is specification aging. 3.4. Defect Category Progression Figure 3 cross-tabulates defect categories against audit rounds; Table 5 provides full numeric detail. Three patterns are visible. First, version drift and stale Jira references concentrate in rounds 1–3 (19 of 21 combined, or 90.5%), consistent with structural issues surfacing earliest. Second, semantic misleading text peaks in round 3 (5 of 8 total). Third, cross-lane schema mismatches appear exclusively in rounds 4–8 (all 5 occurrences), consistent with the observation that this class requires multi-file visibility, which (Table 2) only became standard from round 4. Table 5. Defect cross-tabulation: category by round. Defect type
R1
R2
R3
R4
R5
R6
R7
R8
R9
Total
Version drift Stale Jira refs Cross-lane schema Missing Lane 7 Label / contract Semantic text Formula / timing
5 5 0 4 1 0 0
3 3 0 0 1 0 1
0 3 0 2 2 5 0
0 0 1 0 0 0 1
1 1 1 1 2 2 0
0 0 0 0 0 0 1
0 0 2 0 0 1 1
0 0 1 0 0 0 0
0 0 0 0 0 0 0
9 12 5 7 6 8 4
Column total
15
8
12
2
8
1
4
1
0
51
Version Drift
5
3
·
·
1
·
·
·
·
Stale Jira Refs
5
3
3
·
1
·
·
·
·
Cross-Lane Schema
·
·
·
1
1
·
2
1
·
Missing Lane 7
4
·
2
·
1
·
·
·
·
Label / Contract
1
1
2
·
2
·
·
·
·
Semantic Text
·
·
5
·
2
·
1
·
·
Formula / Timing
·
1
·
1
·
1
1
·
·
R1
R2
R3
R4
R5
R6
R7
R8
R9
=15
=8
=12
=2
Audit=8round
=1
=4
=1
5 4 Defects per cell
Defect category
Defect Round (cross-tabulation, n = 51) StructuralCategory by Audit Semantic Cross-lane / contract
3 2 1
=0
Figure 3. Defect category by audit round (cross-tabulation, n = 51). Phase brackets above the heatmap mark structural, semantic, and cross-lane/contract phases.
4. Discussion 4.1. Three Distinct Notions of Review “Single-pass review” is ambiguous. We use three distinct terms: •
Single-file review. The auditor reads one specification file at a time without explicit cross-file comparison. By construction, single-file review cannot detect cross-file schema mismatches.
9 of 13
• •
Initial full-scope pass. The auditor reads all files in one context window, once, with cross-file comparison enabled. We did not run this as a control. Iterative full-scope auditing. Repeated full-scope passes with audit-and-fix between rounds, until quiescence. This is the protocol distilled from the case study.
The defensible empirical claim of this paper is that, in this system, single-file review missed cross-document defect classes by construction, and one-and-done audit (any single-pass design) left later-discovered defects unresolved under the case-study protocol. We do not claim that no full-scope single-pass design could in principle have surfaced them. 4.2. Cascading Edits Fixing issue A in round N can create a new inconsistency B that was masked by A. For example, removing fallback_enabled: true from Lane 2’s config (round 5) made the changelog entry referencing jira_fallback_queue.md newly incorrect (caught in round 6). This is analogous to regression introduction in traditional code review. 4.3. Cross-Document Visibility No single-file audit can detect that Lane 4 expects fix_priority while Lane 3 emits priority_score. This requires reading both files in the same pass and comparing schemas field-by-field. The audit protocol itself was iteratively refined to support this, as documented in Table 2; convergence in this case study therefore depends on both fixing defects and improving the audit prompt. 4.4. Implications for Prompt Engineering Practice (Bounded) These implications are observations from this case, not generalizations: • •
• •
Treat prompt specifications as code. Consistent with the promptware position [11]. Prompt files define system behavior, carry data contracts, and contain integration logic. Budget for iterative auditing. This case took nine rounds at this scale (approximately 7150 lines and eight files); the protocol surfaced cross-document defects in rounds 7–8 that earlier rounds had not. The metric we used was a full-scope clean pass, not a fixed round count. Cross-document audits matter for contract validation. Single-file review missed every cross-lane data-contract issue surfaced in this study. Verification greps are useful but insufficient in this case. Automated string matching caught structural issues but missed semantic drift.
4.5. Threats to Validity We identify six categories following Runeson and Höst [25] and Wohlin et al. [26]. 4.5.1. Internal Validity The same LLM family (Claude) both authored and audited the specifications, creating a possible shared blind spot. A stronger replication would use a dissimilar LLM and at least one human reviewer. 4.5.2. External Validity This study examines a single system (n = 1) without a control group. The seven-category taxonomy is post-hoc and may not be exhaustive. 4.5.3. Construct Validity Convergence to zero findings does not establish correctness, only internal consistency. Runtime validation remains a necessary complementary quality gate. 4.5.4. Statistical and Conclusion Validity Sample sizes are modest (n = 9 rounds, n = 51 defects). With a stochastic LLM auditor and nine rounds, the bumps at rounds 3 and 5 cannot be statistically distinguished from auditor variance
10 of 13
without a control comparison or repeated trials per round; the phase-based interpretation is one explanation among several. Following Wohlin et al. [26], conclusion validity concerns the relationship between treatment and outcome: with one stochastic auditor, a co-evolving protocol, and no repeat trials, we cannot rule out alternative explanations for the observed convergence pattern. Future replications should fix the protocol before remediation begins and run repeat-trial passes per round. 4.5.5. Prompt-Overfitting and Sequence Effects Because the audit protocol co-evolved with discovered defects, later prompts may have become tuned to the known defect space, weakening the convergence interpretation. Related, later rounds operated on artifacts already changed by prior fixes, so the order in which fixes were applied may have shaped the observed category progression: a different fix-order might have surfaced different categories at different rounds. The locked Appendix F protocol was finalized after the study and should be applied as-is in future replications to address both confounds. 4.5.6. Coding Reliability The taxonomy was derived by a single coder (the author). No inter-rater check was conducted. This is the central methodological limitation of the taxonomy claim. The synthetic mini-specification released in the reproducibility bundle (Appendix H) includes ground-truth coding labels for five seeded defects across four categories, providing a non-proprietary artifact against which inter-rater agreement on Appendix G’s rules can be measured.
5. Conclusions This paper presented a single-system empirical case study of iterative, agent-driven auditing of prompt specifications in a production seven-lane orchestration pipeline. Across nine rounds, 51 prompt-specification defects were surfaced and remediated. The audit terminated when round 9 returned zero findings under the case-study protocol’s one-clean-pass stopping rule. We make three case-bounded claims. First, in this case study, single-file review missed crossdocument defect classes by construction. Second, the observed convergence path was non-monotonic, consistent with cascading edits and audit-scope expansion, though sampling-noise alternatives cannot be ruled out without a control. Third, the audit protocol itself co-evolved with the fixes; the locked, recommended version is reproduced in Appendix F. Iterative full-scope auditing is a candidate practice for LLM-orchestrated systems where prompt specifications serve as the behavioral contract between autonomous agents. Replication across diverse architectures, with dissimilar LLMs and human reviewers, with a single-pass full-scope control, and with multiple independent coders, is required to validate generalizability. Author Contributions: Conceptualization, methodology, software, validation, formal analysis, investigation, resources, data curation, writing—original draft preparation, writing—review and editing, visualization, supervision, project administration: E.C. The author has read and agreed to the published version of the manuscript. Funding: This research received no external funding. Institutional Review Board Statement: Not applicable. The study did not involve human or animal subjects. Informed Consent Statement: Not applicable. Data Availability Statement: A reproducibility bundle accompanies this submission as Supplementary Material containing per-round CSVs, the anonymized 51-row defect catalog, the locked checklist and prompt template, the severity rubric, the context-loading chronology, and a synthetic mini-specification for protocol validation. Source AEGIS specifications and Jira data are proprietary to The Swift Group, LLC, and are not released; the bundle is sufficient for replication of the audit method against an independent specification surface. Conflicts of Interest: The author is employed by The Swift Group, LLC, which holds commercial licensing rights to products built on the AEGIS reference implementation. The audit method and data presented here are released
11 of 13
independently of any commercial product. The employer had no role in the study design, the audit protocol design, the analysis, the interpretation, or the decision to submit.
Abbreviations The following abbreviations are used in this manuscript: AEGIS FMEA LLM MCP MLT QA RQ STRIDE
Autonomous Engineering Governance and Intelligence System Failure Mode and Effects Analysis Large Language Model Model Context Protocol MANDATE / LATTICE / TRACE governance stack Quality Assurance Research Question Spoofing, Tampering, Repudiation, Information disclosure, Denial of service, Elevation of privilege
Appendix F Locked Audit Checklist, Prompt Template, and Method Specifics Appendix F.1 Locked Checklist (Seven Dimensions) The full locked checklist is reproduced in the supplementary file audit_checklist_locked.md. The seven dimensions are: version consistency; cross-lane data contract alignment; Jira permission boundaries; label conventions; lane-count propagation; cadence and scheduler alignment; internal contradiction. Appendix F.2 Locked Prompt Template The full locked prompt template is reproduced in the supplementary file audit_prompt_template.md. It specifies the structured-output schema (id, file, line, dimension, category, severity, description, suggested_fix, uncertainty), the trigger-of-detection rule for category assignment, and the severity rubric. Appendix F.3 Recommended Convergence Criterion Terminate when two consecutive full-scope passes return zero findings. (This is stricter than the criterion used in the case study, which required only one clean pass.) Appendix F.4 Severity Rubric The full severity rubric is reproduced in the supplementary file severity_rubric.md. High severity covers silent runtime failure risk and producer-consumer schema breaks; medium covers documented-behavior errors and operator-misleading text; low covers cosmetic and internal-doc drift. Appendix F.5 Method Specifics for LLM-Based Auditing For replication, report the following: LLM family (Claude, this study), specific model versions (mix of Claude Opus and Sonnet across nine rounds), decoding parameters (default temperature; no custom sampling), context-window policy (single-file rounds 1–2, expanding to multi-file rounds 3–5, full-scope rounds 6–9), sub-agents per round (1–3), tooling (Claude Agent tool with file-read access; no cached memories; fresh session per round). A more rigorous replication would lock model version and decoding parameters before remediation begins.
Appendix G Defect Coding Rules and Examples Coder. Single coder (the author). Inter-rater validation was not performed; this is acknowledged as a central limitation in Section 4.5. Mutual exclusivity. Categories are mutually exclusive. Boundary cases were assigned to the category matching the trigger of detection, not the apparent fix. Worked boundary case. A defect surfaced in round 5 read: “Lane 5’s diagnostic checklist instructs the operator to confirm that the fallback_enabled flag is set to true if Lane 2 is in degraded mode, but
12 of 13
Lane 2’s config file has fallback_enabled: false after the prior architectural change.” This defect could plausibly be coded as either semantic_text (the diagnostic instruction has become misleading) or label_contract (the flag-value contract has drifted). The trigger of detection was a label-contract dimension check (dimension 4), so it was coded label_contract. The seven category definitions and one example per category are reproduced in the supplementary file audit_checklist_locked.md and the manuscript’s full markdown source.
Appendix H Reproducibility Artifact The reproducibility bundle (Supplementary Material) contains: round_counts.csv, defect_catalog.csv (anonymized 51-row catalog), audit_checklist_locked.md, audit_prompt_template.md, severity_rubric.md, context_loading_chronology.csv, synthetic_mini_specification.md (a four-file mini-specification with five seeded defects across four taxonomy categories for protocol validation against ground truth), and a README.md with replication instructions. The source AEGIS specifications and Jira data are proprietary to The Swift Group, LLC, and are not released. The bundle is sufficient to replicate the audit method against an independent specification surface.
1. 2. 3. 4. 5.
6.
7.
8. 9. 10.
11.
12.
13. 14.
15.
Fagan, M.E. Design and Code Inspections to Reduce Errors in Program Development. IBM Systems Journal 1976, 15, 182–211. Boehm, B.; Basili, V.R. Software Defect Reduction Top 10 List. Computer 2001, 34, 135–137. Weinberg, G.M.; Freedman, D.P. Reviews, Walkthroughs, and Inspections. IEEE Transactions on Software Engineering 1984, SE-10, 68–72. Parnas, D.L. Software Aging. In Proceedings of the Proceedings of the 16th International Conference on Software Engineering (ICSE), Sorrento, Italy, 1994; pp. 279–287. Gotel, O.C.Z.; Finkelstein, A.C.W. An Analysis of the Requirements Traceability Problem. In Proceedings of the Proceedings of the 1st International Conference on Requirements Engineering (ICRE), Colorado Springs, CO, USA, 1994; pp. 94–101. Wang, X.; Wei, J.; Schuurmans, D.; Le, Q.; Chi, E.; Zhou, D. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In Proceedings of the Proceedings of the International Conference on Learning Representations (ICLR), Kigali, Rwanda, 2023. Madaan, A.; Tandon, N.; Gupta, P.; Hallinan, S.; Gao, L.; Wiegreffe, S.; Alon, U.; Dziri, N.; Prabhumoye, S.; Yang, Y.; et al. Self-Refine: Iterative Refinement with Self-Feedback. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), New Orleans, LA, USA, 2023. He, J.; Treude, C.; Lo, D. LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead, 2024, [arXiv:cs.SE/2404.04834]. Cemri, M.; Pan, M.Z.; Yang, S.; et al. Why Do Multi-Agent LLM Systems Fail?, 2025, [arXiv:cs.AI/2503.13657]. Lubos, S.; Felfernig, A.; Tran, T.N.T.; Garber, D.; El Mansi, M.; Polat Erdeniz, S.; Le, V.M. Leveraging LLMs for the Quality Assurance of Software Requirements, 2024, [arXiv:cs.SE/2408.10886]. RE 2024 — Requirements Engineering Next! track. Chen, Z.; Wang, C.; Sun, W.; Liu, X.; Zhang, J.M.; Liu, Y. Promptware Engineering: Software Engineering for Prompt-Enabled Systems, 2025, [arXiv:cs.SE/2503.02400]. Accepted by ACM Transactions on Software Engineering and Methodology (TOSEM). Patil, A. A Blueprint for AI-Driven Software Quality: Integrating LLMs with Established Standards, 2025, [arXiv:cs.SE/2505.13766]. Earlier versions of this preprint circulated as "Advancing Software Quality: A Standards-Focused Review of LLM-Based Assurance Techniques.". Akshathala, S.; Adnan, B.; Ramesh, M.; Vaidhyanathan, K.; Muhammed, B.; Parthasarathy, K. Beyond Task Completion: An Assessment Framework for Evaluating Agentic AI Systems, 2025, [arXiv:cs.AI/2512.12791]. Li, J.; Su, Y.; Lyu, M.R. From Laboratory to Real-World Applications: Benchmarking Agentic Code Reasoning at the Repository Level, 2026, [arXiv:cs.SE/2601.03731]. Introduces the RepoReason benchmark. Accepted by ACL 2026 (main track). Yang, J.; Jimenez, C.E.; Wettig, A.; Lieret, K.; Yao, S.; Narasimhan, K.; Press, O. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, 2024, [arXiv:cs.SE/2405.15793].
13 of 13
16.
17. 18. 19.
20. 21.
22.
23.
24. 25. 26.
Liu, X.; Yu, H.; Zhang, H.; Xu, Y.; Lei, X.; Lai, H.; Gu, Y.; Ding, H.; Men, K.; Yang, K.; et al. AgentBench: Evaluating LLMs as Agents. In Proceedings of the Proceedings of the International Conference on Learning Representations (ICLR), Vienna, Austria, 2024. Hong, S.; Zhuge, M.; Chen, J.; Zheng, X.; Cheng, Y.; Wang, J.; Zhang, C.; Wang, Z.; Yau, S.K.S.; Lin, Z.; et al. MetaGPT: Meta Programming for a Multi-Agent Collaborative Framework, 2023, [arXiv:cs.AI/2308.00352]. Naqvi, S.; Baqar, M.; Mohammad, N.A. The Rise of Agentic Testing: Multi-Agent Systems for Robust Software Quality Assurance, 2026, [arXiv:cs.SE/2601.02454]. Calboreanu, E. Closed-Loop Autonomous Software Development via Jira-Integrated Backlog Orchestration: A Case Study in Deterministic Control and Safety-Constrained Automation, 2026, [arXiv:cs.SE/2604.05000]. Preprint, Swift North AI Lab, The Swift Group, LLC. Calboreanu, E. Context Engineering: A Methodology for Structured Human-AI Collaboration, 2026. Working Paper v2.5, Capitol Technology University. ORCID: https://orcid.org/0009-0008-9194-0589. Calboreanu, E. MANDATE: A Tolerance-Based Framework for Autonomous Agent Task Specification, 2026. SSRN paper 6170328. Title cited per the SSRN canonical entry. The acronym MANDATE is the project name; an alternative expansion appears in earlier companion artifacts., https://doi.org/10.2139/ssrn.6170328. Calboreanu, E. LATTICE: A Governance-First Architecture for Authorized Autonomous AI Operations, 2026. SSRN paper 6151128. Title cited per the SSRN canonical entry. The acronym LATTICE is the project name; an alternative expansion appears in earlier companion artifacts., https://doi.org/10.2139/ssrn.6151128. Calboreanu, E. TRACE: A Governance-First Execution Framework Providing Architectural Assurance for Autonomous AI Operations, 2026. SSRN paper 6212818. Title cited per the SSRN canonical entry. The acronym TRACE is the project name; an alternative expansion appears in earlier companion artifacts., https://doi.org/10.2139/ssrn.6212818. Anthropic. Model Context Protocol Specification, 2024. Online documentation; accessed 12 May 2026. Runeson, P.; Höst, M. Guidelines for Conducting and Reporting Case Study Research in Software Engineering. Empirical Software Engineering 2009, 14, 131–164. Wohlin, C.; Runeson, P.; Höst, M.; Ohlsson, M.C.; Regnell, B.; Wesslén, A. Experimentation in Software Engineering; Springer: Berlin, Germany, 2012.