1
N-Version Programming with Coding Agents Javier Ron* , Benoit Baudry† , and Martin Monperrus* KTH Royal Institute of Technology † Université de Montréal [email protected] [email protected] [email protected]
arXiv:2606.20158v1 [cs.SE] 18 Jun 2026
*
Abstract—This paper revisits the classical concept on N-version programming in the setting of contemporary AI coding agents. Revisiting the seminal Knight–Leveson experiment, we study whether diversity across agent systems, models, and implementation languages creates diverse failure modes. Using the Knight–Leveson’s, Launch Interceptor Program Specification, we evaluate 48 agent-generated implementations on a shared oracle and a campaign of 1,000,000 randomized test inputs. The results show substantial common-mode failure, along the findings of Knight–Leveson. Further analysis that many of those co-occuring failures can be traced to where is specification is particularly hard or ambiguous. We also demonstrate that diversity from coding agents provides practical benefit: across majority voting three-version units, the mean failure count drops from 387.44 for single versions to 130.99 for triples, and 11,844 N-version units exhibit zero observed failures. Our original results is the strongest evidence to date that N-Version Programming with coding agents is a useful engineering strategy.
I. I NTRODUCTION N-version programming (NVP) promises reliability through diversity: multiple independently produced implementations of the same specification are executed in parallel, and a voting rule masks individual faults. Its classical reliability argument, however, depends on a strong condition: the versions must fail independently, or at least be diverse enough that coincident failures remain rare. That assumption was challenged in the human-programmer era: the seminal Knight–Leveson experiment showed that independently developed human implementations of the same specification still exhibited substantial common-mode failure [15]. AI coding agents make this question relevant again [11, 25]. Compared with recruiting independent human teams, it is now straightforward to generate many implementations of the same task while varying three key components: the coding agent, the underlying foundation model, and the target programming language. With agentic coding, the central question of NVersion Programming remains unchanged: do the generated variants behave like independent versions, or do they converge to the same latent defects? This paper revisits the classical N-Version Programming literature in the modern agentic setting. We design and perform a reproduction of the original Knight–Leveson experiment with AI coding agents. We also revisit the broader questions about N-version software: whether the considered diversity mechanisms reduce fault correlation, which shared fault families dominate, and whether redundancy helps even if fault independence fails. As specification, we take Knight–Leveson’s one for the Launch Interceptor Program (LIP), ane example defense soft-
N-version
Version Generation
P1
engineering
python
Program Specification
Testing and
P2
N-version units
fault analysis
rust
P0 P3
Claude
pascal
Code Codex
P19
P42
Version selection
...
Coding Agents
Pn
Fig. 1: N-version units are generated by AI coding agents, they improve reliability over single programs on average.
ware system. We strictly follow the structure of the original experiment, only replacing human programmers with contemporary AI coding agents. Across five agent systems, 23 models, and three target languages, our design space comprises 69 [harness, model, language] triples. We evaluate the resulting implementations with an oracle-based acceptance screen and a shared 1,000,000-case campaign. We analyze the observed failures through population-level coincidentfailure statistics, pairwise overlap across language and agent boundaries, function-level fault localization, and exhaustive majority-vote three-version units. Our results are clearcut. First, the idealized fault independence hypothesis fails: among the 48 admitted implementations in the campaign archive, the experiment produces 429 coincident-failure cases where the random independence model predicts only 115.36 (z = 29.20), and strong pairwise clusters can be observed across both language and agent boundaries. The failures are not diffuse; they concentrate in a small number of recurring bug families that reappear across ostensibly different implementations. Second, despite fault correlation, redundancy does provide measurable benefit: when all 48 = 17,296 3-Version units are evaluated, the 3 mean failure count drops from 387.44 for single versions to 130.99 for triples, and 11,844 triples exhibit zero observed failures. All code and generated versions are available at https://github.com/ASSERT-KTH/Knight-Leveson-Redux To sum up, the paper makes four contributions. First, it provides a systematic experimental framework for studying fault independence and reliability in agentgenerated software, faithful to Knight–Leveson. • Second, it shows that modern coding agents can feasibly generate enough versions to cheaply perform N-Version programming at scale. • Third, it measures and demonstrates failure overlap across •
2
[harness, model, language] diversity axes. Most of the errors can be traced back to weaknesses in the specification. • Fourth, it shows that majority voting N-version units do provide practical reliability gains, despite fault correlation.
II. BACKGROUND
b) Subsequent N-Version Work: Littlewood and Miller [18] argued that diversity mechanisms such as development methodology variation may improve reliability, while Bishop [4] identified ambiguities and omissions in the specification as important sources of common-mode faults. Hatton [12] later showed that even without independence, multi-version systems may still deliver useful reliability gains in practice.
A. N-Version Programming C. The Launch Interceptor Specification N-version programming was proposed by Chen and Avizienis [6] as a software analog of hardware N-modular redundancy. The design calls for N independent teams to implement the same specification from a common requirements document, then execute all versions in parallel on each input and determine the output by majority vote or other consensus mechanism. Avizienis [3] formalized the reliability model: provided that the failure events of distinct versions on any given input are mutually independent and that individual failure probabilities are small, the probability of majority failure decreases exponentially with N .
B. N-Version and Fault Independence The idealized reliability benefit of NVP is entirely contingent on fault independence. Eckhardt and Lee [9] provided a theoretical treatment showing that independence cannot be assumed as a matter of principle. Because all versions are developed from the same specification, and because specifications are finite and sometimes ambiguous, there exists a nonzero set of inputs for which the specification is underspecified or misinterpreted in a common way. Programmers who share a training background, a programming language, or exposure to the same reference materials will tend to make the same misinterpretation, creating systematic coincident failure modes. Littlewood and Miller [18] extended this analysis, arguing that the very process of translating a specification into code creates correlations among any set of implementations derived from it. a) The Knight–Leveson Experiment (1986): The Knight –Leveson experiment [15] was an empirical test of fault independence in NVP. Twenty-seven programmers from two universities independently implemented the ”Launch Interceptor Program” specification, working without mutual communication. Their implementations were evaluated against a reference implementation on one million randomly drawn test cases. Of the 24 versions with nonzero failure rates, failures were strongly correlated: the observed count of simultaneous failures exceeded the expectation under independence by a statistically significant margin. Subsequent work by the same team [5] examined the coincidental faults, finding that a small number of fault categories accounted for the majority of coincident events. These faults were most prominently related to implementations of specific geometric computations required by the specification. Hatton [12] conducted a partial replication with a different benchmark and similarly found that independence was not achieved.
The specification for a ‘Launch Interceptor Program’ (LIP) was defined by NASA [8] and used in the original Knight –Leveson study. The implementation task is to implement a DECIDE function that computes a missile launch authorization decision, given n planar radar data points with Cartesian coordinates and a set of parameters representing incoming threats. The implementation is not trivial because several functions involve non-trivial geometric computations that are known fault attractors [5]. The computation is specified with four stages: (1) the Conditions Met Vector (CMV), a vector of 15 Boolean Launch Interceptor Conditions (LICs), each encoding a geometric predicate on subsets of the input points; (2) the Preliminary Unlocking Matrix (PUM), a 15×15 Boolean matrix derived from the CMV and a programmer-supplied Logical Connector Matrix (LCM) whose entries are ANDD, ORR, or NOTUSED; (3) the Final Unlocking Vector (FUV), a 15-element Boolean vector derived from PUM column-wise conjunctions; and (4) the scalar LAUNCH decision, which is true iff all FUV entries are true. The Launch Interceptor Program is well suited to randomized testing: its input domain is (1) large, (2) the input types are simple (floating-point and enumeration parameters), and (3) random sampling covers the full condition space. D. Coding Agents as N-Version Generators Contemporary AI coding agents represent a qualitatively new kind of software developer. Chen et al. [7] demonstrated that LLMs trained on code can solve a substantial fraction of programming challenges; and modern coding agents can use tools, and perform iterative self-correction, and long-horizon planning [13]. Unlike earlier program synthesis systems, these agents are able to operate from general-purpose, natural language specifications. Recent work such as Galapagos [22] has begun to exploit LLMs to construct functionally equivalent variants for N-version deployments. Our present paper asks whether AI coding agents can generate diverse program versions whose failures behave like the independent faults assumed by classical N-version programming studies, or whether they reproduce the same kinds of correlated failure modes those studies repeatedly found in practice. III. E XPERIMENTAL M ETHODOLOGY We design and perform an original experimental plan, that both replicates and extends the Knight–Leveson experiment.
3
Version Generation
? Statistical Acceptance Screening
Launch P1
Interceptor
Test Campaign
Analysis
Pi
All version faults
Program Specification
? P2
200
1000000
Acceptance
Shared random
tests
test cases
Pj
z-statistic
test
<>
Fault correlation Pairwise 𝛟
?
?
Pij
P3
Pk
Pik
Pjk
Analysis
/ independence evidence
Pairwise faults
Coding Agent
? Pn
rejected versions
Fig. 2: Experimental workflow for revisiting Knight–Leveson with agentic coding: (1) version generation by each agent against the LIP specification, (2) oracle-based acceptance screening on 200 independently drawn random cases (all must pass), (3) shared one-million test campaign across admitted versions, and (4) statistical analysis (Knight–Leveson z-statistics, pairwise correlation analysis). Versions that fail acceptance are excluded from the campaign and from all downstream analyses.
A. Research Questions The core idea is to ask distinct coding agents to implement the same specification: LIP. From there, we answer five research questions:
Beyond quantitatively measuring correlation, we qualitatively investigate the origin of failures. We study whether faults in agent-generated programs can be attributed to a small number of challenging or ambiguous parts of the specification.
RQ1. (Generation Capabilities) To what extent are AI coding agents able to implement the LIP specification?
RQ5. (N-version Reliability) To what extent do N-version units generated by AI agents provide practical reliability benefits?
We first ask whether modern coding agents can serve as generators of complete program versions from the seminal LIP specification. This inquiry is a necessary condition for collecting a sufficiently large and varied set of working implementations for comparative analysis in the subsequent questions.
Failure correlation falsifies the ideal case for N-version programming, but does not by itself imply that N-version is useless. Following the perspective of Hatton’s study [12], we measure whether combining multiple AI-generated versions yields worthwhile error reduction.
RQ2. (Fault Independance) Are failures of AI-generated implementations mutually independent, per idealized random fault model?
B. Methodology Overview
This is the central question of the original Knight–Leveson study, translated to the agentic world. We test whether simultaneous failures across generated implementations do not happen (consistent with independence), or whether joint failures occur substantially more often than that model predicts (consistent with independence). Our experimental design follows the Knight–Leveson methodology as closely as the agentic setting permits. Table I summarizes the relationship between the original study and the present one. RQ3. (Diversity Dimensions) To what extent does diversity across programming languages, coding agents, and AI models impact fault independence? Littlewood and Miller [18] argued that diverse software generation mechanisms, such as development methodology, may improve the dependability of multi-version systems. We therefore ask whether varying the language, agent, or underlying model decorrelates failures, or whether shared failure modes remain dominant across these dimensions. RQ4. (Fault Explanation) What are the principal sources of faults in AI-generated implementations?
At a high level, we answer the research questions by reproducing the Knight–Leveson experimental structure faithfully. Table I summarizes which elements are carried over directly and the extent of which the experimental design has been adapted for AI coding agents. As summarized in Table I, we preserve the original acceptance filter, campaign size, failure definition, and primary Knight–Leveson hypothesis test. RQ1. First, we adapt the source of diversity from human programmers to [harness, model, language] configurations. We generate candidate versions from a controlled set of [harness, model, language] configurations and apply the original Knight–Leveson-style acceptance screen; the number of admitted versions is the feasibility evidence of agentic NVersion programming. RQ2. Second, we run every admitted version on the same million-input campaign and record a binary failure vector for each implementation; these vectors are the core data for the independence test. RQ3. Third, we stratify the pairwise co-failure data by language pair and by same-agent versus cross-agent comparisons to test whether these diversity dimensions reduce overlap in failure behavior. RQ4. Fourth, for failed executions we retain the triggering inputs and output mismatches, then aggregate them at LIC
4
Dimension
Knight–Leveson (1986)
This Study
Subjects
27 human programmers (graduate and undergraduate students at 2 universities) Pascal
69 coding agents over 5 harnesses (Cursor, Claude Code, OpenAI Codex, Gemini, OpenCode) Pascal, Python, Rust
Specification
Original LIP specification document + 15 input/output examples + realcompare function
Admitted version count
27
Faithful; Original LIP specification document + 15 input/output examples + realcompare function + a prompt of the task to perform 48
Test-campaign size
1,000,000 random cases
Faithful: 1,000,000 random cases
Acceptance filter
200-case pre-screen; all must pass
Faithful: 200-case pre-screen; all must pass
Oracle
Pascal reference implementation
Failure definition
Any of 241 output bits differs from oracle
Python reference implementation validated by 82 unit tests covering the full spec. Faithful: any of 241 output bits differs from oracle
Primary statistic
Knight–Leveson z-statistic (Eq. 4)
Independence enforcement
Different universities, no communication between students Shared specification document, shared university curriculum
Implementation Language
Primary confounders
Faithful; Knight–Leveson z-statistic, supplemented by pairwise correlation analysis, cross-language and cross-agent stratification, root cause analysis, and majority-vote unit analysis Distinct agent harnesses and vendors; distinct underlying model lineages; distinct target languages Faithful; shared specification document, shared LLM training corpora, overlapping model lineages
TABLE I: Relationship to Knight–Leveson (1986): faithful methodological elements, adaptations and novelty (in italics).
level to determine the major failure sources and trace back the problem to the specification. RQ5. Finally, we reuse the same failure vectors to simulate majority-vote N-version units, and quantitatively measure the reliability improvements brought by agentic N-version. C. Version Generation with LLMs Five AI coding agent systems serve as the “programmers” in our study: Cursor [2], Claude Code [1], OpenAI Codex [7], Gemini [10], and OpenCode [21]. Each agent is configured with a list of underlying models spanning, where applicable, multiple vendors and generations as described in Table II. We configure Cursor with the Composer models; Claude Code with Anthropic’s Haiku, Sonnet, and Opus variants; Codex with multiple GPT-5.x revisions; Gemini with its 2.5 and 3.x preview variants; and OpenCode with Qwen and Gemma models. The functional specification given to every agent is the original Knight–Leveson specification document, preserved verbatim as the authoritative source of truth for all conditions, LICs, and realcompare semantics. Agents are additionally given: a file with 15 input/output examples, and a reference realcompare implementation. The agents also receive a short directive that describes the provided information, input and output formats, and the expected deliverable (Python, Rust, or Pascal). No algorithm is suggested in the prompt, no code skeletons are provided. We pair each [harness, model] tuple with each target programming language, producing three corresponding [harness, model, language] triples. Each triple produces one candidate DECIDE program per invocation. The resulting pool of versions therefore consists of 69 unique [harness, model, language] runs. The exact agent system, underlying model identifier, and target language are recorded in
the per-version metadata for reproducibility. This metadata is later used to separate failures by agent and language when answering RQ3. D. Oracle and Acceptance Testing This part of the methodology defines the reference implementation and the admission filter that determines which generated versions are eligible for the main campaign. We develop a reference implementation of DECIDE in Python, validated by an automated test suite of 82 unit tests covering all known boundary conditions for the 15 LICs. This reference implementation serves as the oracle for differential testing: on every campaign input, each admitted version and the oracle implementation are both evaluated, and the version is recorded as failing on that case if any of its 241 output bits differs from the oracle’s. This matches the failure definition used by Knight–Leveson. Before entering the main test campaign (see III-E), each generated version undergoes an acceptance screening, using the terminology of the original Knight–Leveson protocol. Acceptance testing works as follows. First, we check Pascal and Rust implementations for compilation errors. Then, we use two hundred test cases and evaluate each version against the oracle. Compilation and the small set of tests are used to filter out clearly non-functional implementations. As in Knight –Leveson, a version is admitted only if it passes all 200 tests. The admitted-version count is the first empirical outcome of the study: it measures whether AI coding agents can produce complete-enough implementations to support the replicated Nversion experiment in RQ2. E. Main Test Campaign At this stage, versions that crash immediately or fail any acceptance test have been filtered out. Therefore, the main
5
Harness
Models
Cursor
composer-2.5, composer-2 anthropic/claude-opus-4.6, anthropic/claude-opus-4.5, anthropic/claude-sonnet-4.6, anthropic/claude-sonnet-4.5, anthropic/claude-haiku-4.5 gpt-5.4, gpt-5.4-mini, gpt-5.3-codex, gpt-5.2-codex, gpt-5.2 gemini-3.1-pro-preview, gemini-3-flash-preview, gemini-2.5-pro, gemini-2.5-flash, gemini-2.5-flash-lite qwen/qwen3.6-plus, qwen/qwen3.5-flash-02-23, qwen/qwen3.5-plus-02-15, qwen/qwen3.5-397b-a17b, google/gemma-4-26b-a4b-it, google/gemma-4-31b-it
Claude Code OpenAI Codex Gemini OpenCode
TABLE II: Coding harnesses and underlying models used for version generation. A coding agent is a combination of a harness and a model.
campaign will only analyze implementation-level disagreements among the admitted versions. We draw uniformly T = 1,000,000 test cases at random from the input domain using a fixed seed to ensure reproducibility. All admitted versions are evaluated on exactly the same set T against the oracle. For all test runs, we record metadata: coding agent, model, language, and a binary pass/fail outcome. The resulting failure data is the common measurement for the rest of the paper: (1) aggregate coincident failures are used for the statistical analysis in RQ2; (2) stratified pairwise overlaps are used for the cross-language and cross-agent analysis in RQ3; and (3) majority-vote simulations are used for RQ5. For failed test runs, we also record the input that caused the failure, as well as the specific CMV, PUM, FUV, or LAUNCH oracle mismatches. These fault records are later aggregated at LIC level and linked back to representative implementations and triggering inputs for RQ4. F. Failure Correlation Statistical Analysis We implement an exact replicate of the Knight–Leveson statistical framework [15] to answer RQ2. Let N be the number of admitted versions, T the number of campaign test cases, fi the failure count for version i, and pi = fi /T the empirical failure rate. Let K denote the number of test cases on which two or more versions fail simultaneously. K is the aggregate measure of failure overlap: it counts how often the test campaign encounters an input on which independent implementations break at the same time. We use the z-statistic to test the observed distribution of failures against an approximate normal distribution of failures. Specifically, under the null hypothesis H0 that failures are mutually independent Bernoulli events: P0 = P1 =
N Y
(1 − pi )
i=1 N h X i=1
pi
i Y (1 − pj )
(1)
(2)
j̸=i
Pm = 1 − P 0 − P 1
(3)
where P0 , P1 , Pm are the probabilities that exactly zero, one, or two or more versions fail on a given test case. In particular, Pm is the quantity of interest under H0 : it is
the independence-based prediction for the chance that a test case triggers a coincident failure event. Under H0 , K is approximately normally distributed with mean µ = T Pm and p standard deviation σ = T Pm (1 − Pm ), giving the Knight –Leveson z-statistic: K −µ . (4) σ Because K is taken to be approximately normal under H0 , tail probabilities and two-sided p-values for the observed z follow from the standard normal distribution. We reject H0 at the 99% confidence level if |z| > 2.576, matching the Knight –Leveson threshold. The alternative hypothesis H1 is that failures are positively correlated, i.e. coincident failures exceed the independence prediction. A large positive z therefore means that the observed overlap in failures is many standard deviations above what this independence model would predict. In addition to the pooled and per-language Knight–Leveson counts, we compute a global all-pairs similarity view over the admitted population. For every pair of admitted versions, we compute the Pearson ϕ correlation between their binary failure vectors over the million-case campaign. Here, ϕ measures cofailure similarity: ϕ = 1 means two versions fail on exactly the same campaign inputs, ϕ ≈ 0 means little overlap in their failure patterns, and negative values indicate anticorrelation. We use ϕ as the main descriptive overlap measure between pairs. This complements the aggregate Knight–Leveson statistic by showing whether coincident failures are concentrated in clusters of versions or dispersed across the population. z=
G. Cross-Language and Cross-Agent Failure Analysis RQ3 asks whether implementation language and coding agent behave as meaningful axes of diversity, rather than as wrappers around the same failure mode. We answer that question by reusing the full pairwise failure data from RQ2 and stratifying it along the diversity dimensions directly available in the experiment. To study language diversity, we partition the full pairwise set into cross-language pairs and inspect their ϕ distribution. This asks whether changing implementation language tends to decorrelate failure behavior in the observed population. Third, to study agent diversity, we partition the same pairwise set into same-agent and cross-agent subsets and
6
compare their ϕ distributions. This asks whether crossing an agent boundary decorrelates failures, or whether similar failure profiles remain common even across different tools.
Total 69
70
20.0 17.5 15.0 12.5 10.0 7.5 5.0 2.5 0.0
60 48
Versions
50 40 30 20
H. Root Cause Analysis
10 0
We identify the sources of correlated faults by manually analyzing failures patterns. Because the 15 LIC predicates form the main functional decomposition inside DECIDE, they provide a natural first unit for fault localization. First, we count how many distinct [harness, model, language] triples fail on each LIC, and we stratify those LIC-level counts by target language and by coding agent. Then, we trace those failures back to implementation choices in the generated source code, and compare against the oracle and the specification.
All
15 11
By Language
25
18 15 13
15
20
23
23
18
17
23
15
10
8
6
13
10 5
ex de Cod ude Co Cla
sor nCode emini G Ope
Cur
Passed
0
on
Pyth
t Rus
al Pasc
Failed
Fig. 3: RQ1. Acceptance testing summary. Left: the total configured population of 69 triples. Center: counts by agent system. Right: counts by target language. In all panels, green denotes admitted versions and red denotes excluded versions; the stacked height is the total number of configured triples in that category.
27 25
We construct synthetic N-version units from the admitted versions and evaluate them on the same T = 1,000,000 campaign inputs. A unit of size n fails on an input when at least ⌈n/2⌉ of its members fail on that input. For the core RQ5 analysis, we enumerate all possible threeversion units that can be formed from the admitted pool and compute their majority-vote failure statistics over the campaign. This gives an exhaustive picture of how often N-version voting helps improve reliability in the observed population.
20
Triple count
I. N-Version Unit Analysis
IV. E XPERIMENTAL R ESULTS
By Agent
Agent/Model/Language Triples by Failed-Test Bucket
18
15 10 5 0
0
0 <10
2 <100
<1000
Failed test cases
0 <10000
1 <100000
Fig. 4: Failure-count distribution across the 48 admitted [harness, model, language] according to the reference implementation, grouped into buckets over the million test case campaign. The distribution is strongly right-tailed, with many near-perfect versions and a much smaller set of high-failure outliers.
A. RQ1: Acceptance Success Fig. 3 summarizes the outcome of acceptance testing. Across the five agent systems and three target languages, 69 configured [harness, model, language] triples were generated and 48/69 passed all 200 acceptance cases, yielding an admission rate of 70%. The remaining 21 configured triples were excluded before the campaign. The admission rate varies markedly by agent (center panel): the Cursor programs are admitted for all 6 of 6 attempts (100%) across all three languages, compared with 13 of 15 (87%) for Claude Code, 11 of 15 (73%) for Codex, 8 of 15 (53%) for Gemini, and 10 of 18 (56%) for OpenCode. Similar variation was observed per language (right panel): Python is the most successful target language, with 18/23 admitted attempts (78%), while Rust has 17 of 23 (74%) and Pascal 13 of 23 (57%). Agent Failure Modes. Among the 21 excluded triples, 12 failed to produce the required artifact. Among the remaining excluded versions, 5 crashed immediately when executed, and 4 returned wrong output on at least one acceptance case. Consistent with the original Knight–Leveson acceptance protocol, all such versions are excluded from the campaign and from all subsequent analyses.
Answer to RQ1. AI coding agents can correctly implement the specification of the Knight–Leveson study. We obtained 48 functional implementations which pass the 200-case acceptance screen. The pool of admitted implementations is large enough to compute the Knight –Leveson independence test and to support additional analyses by language, harness, model, and fault source.
B. RQ2: Coincident Failure Statistics The admitted versions span a wide range of failure counts in the main campaign: 27 are failure-free, while the worst version fails 10,469 of the 106 inputs. Fig. 4 groups the 48 admitted [harness, model, language] triples into coarse failurecount buckets. Most versions cluster at exactly zero failures; 2 more fall between 10 and 99 failures, 18 fall between 102 and 103 cases, none fall between 103 and 104 , and 1 exceeds 104 failures. The right tail shows that the agent population contains both a large near-perfect core and a smaller set of high-failure outliers.
7
Table III presents the Knight–Leveson statistics for the whole 48-version campaign. The Knight–Leveson test asks whether the observed number of campaign inputs on which at least two versions fail simultaneously is compatible with the independence model derived from the individual version failure rates. If the observed count K is much larger than the independence expectation µ, the resulting large positive z indicates common-mode failure rather than accidental overlap. Pooled across all languages, the independence model predicts µ = 115.36 coincident-failure cases over the T = 106 campaign; the observed count is K = 429, an excess of K/µ ≈ 3.7× over the independence prediction, yielding z = 29.20 with p ≈ 1.765 × 10−187 . This infinitesimal pvalue decisively rejects the independence hypothesis. Next, we do a per-language analysis. Every per-language analysis rejects H0 with over 99.99% confidence, with K/µ ranging from 17.5× (Python) to 155.1× (Pascal) and z from 80.7 to 253.3. The systematic rejection rules out the explanation that the pooled rejection is being driven by only one implementation language. The fact that Z is lower at the whole population level shows that the pooled population is more heterogeneous than the within-language slices, reducing the relative concentration of coincident failures in the aggregate. We next turn from coincident-failure counts to pairwise failure similarity. This global all-pairs view shows how the 48 admitted versions relate to one another as a population and whether the same dependence appears beyond the pooled Knight–Leveson statistic. For every pair of admitted versions, we compute the Pearson ϕ correlation between their binary failure vectors over the main test campaign. In this setting, ϕ measures co-failure similarity: ϕ = 1 means two versions fail on exactly the same inputs, ϕ ≈ 0 means little overlap in their failure patterns, and negative values would mean their failures are anticorrelated. Fig. 5 shows a filtered heatmap of the pairwise ϕ matrix, restricted to versions with at least one observed campaign failure. Large dark regions indicate families of versions that fail on the same inputs, and those regions cross both agent and language boundaries rather than aligning neatly with a single tool or target language. Again, the immediate implication is that nominal diversity in agent, model, or language does not automatically buy behavioral diversity. Pale rows and columns in the heatmap mark failing implementations whose failures do not overlap with the main co-failure clusters.
Slice
N
µ
K
z
p-value
All languages Python Rust Pascal
48 18 17 13
115.36 23.97 4.91 2.70
429 419 419 419
29.20 80.69 186.93 253.30
1.765 × 10−187 1.701 × 10−1416 1.693 × 10−7590 3.327 × 10−13935
TABLE III: RQ2. Knight–Leveson correlated failure statistics. N is the number of versions; µ = T · Pm is the expected simultaneous-failure count according to the Bernoulli model; K is the observed count in the respective population; z is the z-statistic (see Eq. 4); and p is the corresponding two-tailed p-value. All slices decisively reject H0 : the theoretical random failure model and the actual failure mode do not match. splitting the pairwise co-failure analysis along the diversity dimensions directly available in the experiment: implementation language and coding agent. We first refine the global picture from RQ2 by looking specifically at cross-language pairs. Here, every admitted version written in one language is paired with every admitted version written in a different language, regardless of agent or model, and we compute the same co-failure statistics as in the global analysis. This yields 761 cross-language pairs in total; for 615 of them, ϕ is undefined because at least one of the two versions is failure-free, leaving 146 pairs in the ϕ distribution analysis. Figure 6 summarizes the resulting pairwise ϕ distributions under both diversity splits. For cross-language pairs, 146 pairs have defined ϕ; 81 land exactly in the ϕ = 1 bucket, indicating perfect co-failure agreement. The remaining defined crosslanguage pairs are fewer: 40 are non-positive, 13 fall in (0.1, 0.2], and 12 fall in (0.6, 0.7]. The language-pair stack shows that perfect co-failures are distributed across all three language pairings: Pascal–Python contributes 17 pairs, Pascal– Rust contributes 32, and Python–Rust contributes 32. The same figure also shows the corresponding same-agent versus cross-agent split. The observed pattern is similar in both groups: among the 221 same-agent pairs, 52 have defined ϕ, with 34 at exact ϕ = 1 and 6 non-positive; among the 907 cross-agent pairs, 158 have defined ϕ, with 87 at exact ϕ = 1 and 50 non-positive. Crossing an agent boundary therefore does not eliminate highly correlated failure profiles: exact cofailure clusters remain common even between different tools, although the population also contains many genuinely distinct cross-agent pairs.
Answer to RQ2. The generated AI-agent implementations do not fail independently, according to a uniform random model. At the pooled level, the experiment produces 429 coincident-failure cases where the independence model predicts only 115.36; the null hypothesis is rejected with p ≈ 1.765 × 10−187 .
Answer to RQ3. Cross-language and cross-agent program generation do not provide enough diversity to make failure correlation disappear. In Littlewood and Miller’s terms [18], varying the [harness, model, language] tuple does not equate to achieving fundamentally different methodologies that generate diverse program distributions.
C. RQ3: Cross-Language and Cross-Agent Failure Analysis
D. RQ4: Root Cause Analysis Next, we take a deeper look at the failure modes on the 1M random test cases.
RQ3 studies whether language and agent variance produce diversity in the observed failure behavior. We answer it by
8
Pairwise failure analysis claude-code-claude-haiku-4.5-l-rust
Phi correlation
1.0
1.000 0.657 1.000 1.000 1.000 1.000 1.000 -0.000 1.000 1.000 -0.000 -0.000 1.000 1.000 1.000 0.199 1.000 1.000 1.000 1.000
claude-code-claude-opus-4.6-l-rust 1.000
0.657 1.000 1.000 1.000 1.000 1.000 -0.000 1.000 1.000 -0.000 -0.000 1.000 1.000 1.000 0.199 1.000 1.000 1.000 1.000
claude-code-claude-sonnet-4.5-l-pascal 0.657 0.657
0.657 0.657 0.657 0.657 0.657 -0.001 0.657 0.657 -0.000 -0.000 0.657 0.657 0.657 0.129 0.657 0.657 0.657 0.657
claude-code-claude-sonnet-4.5-l-rust 1.000 1.000 0.657
1.000 1.000 1.000 1.000 -0.000 1.000 1.000 -0.000 -0.000 1.000 1.000 1.000 0.199 1.000 1.000 1.000 1.000
claude-code-claude-sonnet-4.6-l-pascal 1.000 1.000 0.657 1.000
1.000 1.000 1.000 -0.000 1.000 1.000 -0.000 -0.000 1.000 1.000 1.000 0.199 1.000 1.000 1.000 1.000
claude-code-claude-sonnet-4.6-l-python 1.000 1.000 0.657 1.000 1.000
0.8
1.000 1.000 -0.000 1.000 1.000 -0.000 -0.000 1.000 1.000 1.000 0.199 1.000 1.000 1.000 1.000
cursor-composer-2.5-l-pascal 1.000 1.000 0.657 1.000 1.000 1.000
1.000 -0.000 1.000 1.000 -0.000 -0.000 1.000 1.000 1.000 0.199 1.000 1.000 1.000 1.000
cursor-composer-2.5-l-rust 1.000 1.000 0.657 1.000 1.000 1.000 1.000
-0.000 1.000 1.000 -0.000 -0.000 1.000 1.000 1.000 0.199 1.000 1.000 1.000 1.000
cursor-composer-2-l-python -0.000 -0.000 -0.001 -0.000 -0.000 -0.000 -0.000 -0.000
-0.000 -0.000 -0.000 -0.000 -0.000 -0.000 -0.000 -0.002 -0.000 -0.000 -0.000 -0.000
gemini-gemini-2.5-pro-l-python 1.000 1.000 0.657 1.000 1.000 1.000 1.000 1.000 -0.000
0.6
1.000 -0.000 -0.000 1.000 1.000 1.000 0.199 1.000 1.000 1.000 1.000
gemini-gemini-2.5-pro-l-rust 1.000 1.000 0.657 1.000 1.000 1.000 1.000 1.000 -0.000 1.000
-0.000 -0.000 1.000 1.000 1.000 0.199 1.000 1.000 1.000 1.000
gemini-gemini-3.1-pro-preview-l-pascal -0.000 -0.000 -0.000 -0.000 -0.000 -0.000 -0.000 -0.000 -0.000 -0.000 -0.000
1.000 -0.000 -0.000 -0.000 -0.000 -0.000 -0.000 -0.000 -0.000
gemini-gemini-3.1-pro-preview-l-python -0.000 -0.000 -0.000 -0.000 -0.000 -0.000 -0.000 -0.000 -0.000 -0.000 -0.000 1.000
-0.000 -0.000 -0.000 -0.000 -0.000 -0.000 -0.000 -0.000
opencode-qwen-qwen3.5-397b-a17b-l-pascal 1.000 1.000 0.657 1.000 1.000 1.000 1.000 1.000 -0.000 1.000 1.000 -0.000 -0.000
0.4
1.000 1.000 0.199 1.000 1.000 1.000 1.000
opencode-qwen-qwen3.5-397b-a17b-l-python 1.000 1.000 0.657 1.000 1.000 1.000 1.000 1.000 -0.000 1.000 1.000 -0.000 -0.000 1.000
1.000 0.199 1.000 1.000 1.000 1.000
opencode-qwen-qwen3.5-397b-a17b-l-rust 1.000 1.000 0.657 1.000 1.000 1.000 1.000 1.000 -0.000 1.000 1.000 -0.000 -0.000 1.000 1.000
0.199 1.000 1.000 1.000 1.000
opencode-qwen-qwen3.5-flash-02-23-l-python 0.199 0.199 0.129 0.199 0.199 0.199 0.199 0.199 -0.002 0.199 0.199 -0.000 -0.000 0.199 0.199 0.199
0.199 0.199 0.199 0.199
opencode-qwen-qwen3.5-plus-02-15-l-python 1.000 1.000 0.657 1.000 1.000 1.000 1.000 1.000 -0.000 1.000 1.000 -0.000 -0.000 1.000 1.000 1.000 0.199
0.2
1.000 1.000 1.000
opencode-qwen-qwen3.5-plus-02-15-l-rust 1.000 1.000 0.657 1.000 1.000 1.000 1.000 1.000 -0.000 1.000 1.000 -0.000 -0.000 1.000 1.000 1.000 0.199 1.000
1.000 1.000
opencode-qwen-qwen3.6-plus-l-pascal 1.000 1.000 0.657 1.000 1.000 1.000 1.000 1.000 -0.000 1.000 1.000 -0.000 -0.000 1.000 1.000 1.000 0.199 1.000 1.000
1.000
opencode-qwen-qwen3.6-plus-l-rust 1.000 1.000 0.657 1.000 1.000 1.000 1.000 1.000 -0.000 1.000 1.000 -0.000 -0.000 1.000 1.000 1.000 0.199 1.000 1.000 1.000
cla
cla
ud e
-co de -c u d cla e-c laud ud o e-c de- e-hai ku od cla cla e-cl ude -4.5 ud aud -op -l-r cla e-co e-so us-4 ust ud denn .6-l e c e cla -code laud t-4.5 rust e ud e-c -clau -son l-pas od de- net ca l e-c so lau nne 4.5-l r de t u 4 s cu -son .6-l t rso r-c net-4 pasc om .6- al cu pos l-py rso ert 2 ho r cu -com .5-l-p n rs p a ge or-co oser scal -2. mi m ni5 ge pose -l-ru m rs ge mi gem ini-2 2-l-p t niini .5-p ytho g em -ge ge ro n op mini ini-3 mini -l-py en t -ge . h 2 on .5 co m 1-p op de-q ini-3 ro-pr -proen l w e . r co en- 1-pr view ust de -l- qw o-p op qwen en3. revie pasc 5-3 en a w q op cod wen 97b -l-p l y en e co -qw 3.5- -a17 thon d 3 e op e-qw n-qw 97b b-l-p -a1 as en e e n n c co de -qwe 3.5- 7b-l- al -qw n3 39 py 7 t op en en-q .5-fla b-a1 hon co de wen3 sh-0 7b-l-q 2ru . op wen 5-plu 23-l- st -qw s-0 py en co de en3 2-15 thon op -qwe .5-plu -l-py en n co -qw s-02 thon de -qw en3.6 -15-l en -pl -rus -qw ust en l-pa 3.6 sc -pl al us -l-r us t
0.0
Fig. 5: Filtered pairwise co-failure heatmap for the 21 versions with at least one observed campaign failure. Each cell shows the Pearson ϕ correlation between the binary failure vectors of two versions; rows and columns are ordered by agent system and then by language. Dark blocks correspond to clusters of versions that fail on exactly the same campaign inputs. The dark zones cross agent and language boundaries.
Fig. 7 provides the LIC-level view. For each LIC condition from the specification, it plots the number of distinct [harness, model, language] triples that fail on at least one campaign case, showing the same counts under two stackings: by implementation language and by coding agent. Fig. 7 shows that the failures are not equally distributed over the whole specification. Two LICs tend to be incorrectly implemented: 9 and 14. The failures happen over all languages and across multiple agents. Rust programs only fail on LICs #9 and #14, while Pascal and Python programs also fail on some other LICs. For the admitted versions, all agents generated programs with failures, except Codex.
LICs #9 and #14 are both spaced-point variants of the minimum-enclosing-circle predicate: LIC #9 asks whether a selected triple cannot fit inside a circle of radius RADIUS1, while LIC #14 strengthens that pattern into a two-part condition requiring one spaced triple outside RADIUS1 and one spaced triple inside or on RADIUS2. Inspection of the faulty versions reveals the recurring mistake behind LICs #9 and #14: many implementations compute the circumcircle of the selected triple instead of the minimum enclosing circle. That shortcut is incorrect because the minimum enclosing circle of three points is not always the circumcircle. Fig. 8 shows an excerpt from the admitted claude_code/claude-sonnet-4.5 Pascal version that
9
140
Language pair Pascal-Python Pascal-Rust Python-Rust
Version-pair count
120 100
function CircleContainsThreePoints(x1, y1, x2, y2, x3, y3, radius: real): boolean; var d12, d23, d13: real; cx, cy, r: real; a, b, c, d, e, f, g: real; begin d12 := Distance(x1, y1, x2, y2); d23 := Distance(x2, y2, x3, y3); d13 := Distance(x1, y1, x3, y3);
Agent relation Same Agent Cross Agent
80 60 40 20 0
) ] ] ] ] ] ] ] ] ] 0 <= .0,0.1 .1,0.2 .2,0.3 .3,0.4 .4,0.5 .5,0.6 .6,0.7 .7,0.8 .8,0.9 (0.9,1 (0 (0 (0 (0 (0 (0 (0 (0 (0
=1
Phi correlation range
Fig. 6: RQ3: Bucketed distributions of pairwise ϕ correlations under two diversity axes. For each bucket, the left stacked bar counts cross-language pairs by language pair, while the right stacked bar counts all version pairs by same-agent versus cross-agent relation. The exact-match bucket ϕ = 1 is separated from the near-perfect bucket (0.9, 1), showing that the high-correlation mass is dominated by correlated failures.
Distinct failing triples
20
Fig. 8: Representative LIC 9/14 implementation mistake: the version checks whether all pairwise distances fit within the diameter, then computes a circumcircle center and radius rather than the minimum enclosing circle.
Target language Coding agent Pascal Python Rust
Claude Code Cursor Gemini OpenCode
15 10 5 0
1
2
3
4
5
6
7
if (REALCOMPARE(d12, 2 * radius) <> GT) and (REALCOMPARE(d23, 2 * radius) <> GT) and (REALCOMPARE(d13, 2 * radius) <> GT) then begin ... cx := (d * e - b * f) / g; cy := (a * f - c * e) / g; r := Distance(cx, cy, x1, y1); CircleContainsThreePoints := REALCOMPARE(r, radius) <> GT; end else CircleContainsThreePoints := False;
8
LIC
9
10 11 12 13 14 15
Fig. 7: RQ4: failure counts per specification items (aka LIC in the LIP specification). For each LIC, the bars show the number of distinct [harness, model, language] triples that fail on at least one campaign test case. Within each LIC, the left stacked bar groups failures by target language and the right stacked bar groups the same failures by coding agent. Failures overwhelmingly concentrate in LICs 9 and 14, indicating that the failures are driven by a small number of difficult or ambiguous parts of the specification.
is representative of the mistake. LICs #3 and #10 expose a more subtle issue. The original specification is ambiguous about whether the angle predicate should be implemented directly as an interior angle in [0, π] or indirectly through a complementary angle in [0, 2π). In exact arithmetic those formulations are equivalent, but they are not equivalent under the benchmark’s relative-tolerance REALCOMPARE. For the failing gemini-3.1-pro-preview versions, the oracle compares a interior angle against the threshold π −ϵ, while the candidate compares the complementary angle against the threshold π+ϵ. In many test cases with razor-sharp differences, comparisons against the thresholds yield different results. Across the remaining LICs, the dominant mechanisms are
concrete implementation errors rather than specification mistakes. Besides the minimum-enclosing-circle family in LICs #9 and #14, we observed a wrong circumradius formula in LIC #2, a segment-distance substitution for the infinite-line distance required by LIC #7, and dropped applicability guards in LICs #8, #12, and #13. Answer to RQ4. The dominant failure modes are concentrated in LICs #9 and #14, where many agents compute the circumcircle instead of the minimum enclosing circle. The remaining recurrent faults are smaller but still structured: LICs #3 and #10 expose a specification ambiguity caused by REALCOMPARE, while LICs #2, #7, #8, #12, and #13 fail for specific geometric or applicability-check mistakes. These findings are consistent with Brilliant et al.’s observation that coincident failures concentrate in a small number of shared fault categories [5] and with Bishop’s emphasis on specification weaknesses as drivers of correlated faults [4].
E. RQ5: N-Version Unit Reliability The preceding analyses show that the failures are not Bernoulli independent, but they do not by themselves answer whether N-Version units are useful. We therefore construct every possible three-version unit from the admitted implementations and evaluate each unit on the same million campaign inputs. This yields 48 3 = 17,296 triple combinations. Fig. 9 compares bucketed failure-count distributions for majority-vote triple failure counts against the corresponding distribution of single-version failure counts. The 3-version distribution is better on average: the mean triple failure count
10
Bucketed failure-count distribution: single-version vs 3-version unit
80
68.5%
Percentage of versions
70 60
Single version (n = 48) 3-version unit, all triples (n = 17,296)
56.2%
50 40
37.5% 31.3%
30 20 10 0
4.2%
0
0.3%
(0,10]
0.0%
2.1%
0.0%
0] (10,10
000] (100,1
Oracle failures K (of 1,000,000 tests)
0.0%
>1000
Fig. 9: RQ5: Bucketedfailure-count distributions for single versions and for all 48 3 = 17,296 majority-vote triples. The x-axis groups versions into the buckets 0, (0, 10], (10, 100], (100, 1000] , and > 1000 failure counts, and the y-axis shows the percentage of the corresponding population in each bucket, with percentages annotated above the bars. The triple distribution shifts toward the lower-failure buckets, indicating that triple redundancy does reduce the number of failures even though the full population is not independent. This is clear evidence in favor of N-Version programming with coding agents.
is 130.99, compared with a mean single-version failure count of 387.44. At the low end, 11,844 triples (68.48%) have zero majority-vote failures, compared with 27 failure-free individual versions (56.25%). Both distributions have minimum 0 and median 0, so the meaningful differences appear in the upper tail: at P95 the single-version count is 429 while the triple count is 419, at P99 it is 6,004 versus 419, and at the maximum it is 10,469 versus 419. This shows that majority voting substantially compresses the rare high-failure tail. Answer to RQ5. N-version units improve reliability even when the full population exhibit fault correlation. Majority-voting triples substantially improve over single versions: the mean failure count drops from 387.44 to 130.99, and 11,844 3-Version units exhibit zero observed failures (as opposed to 27 individual versions). This is a constructive result that confirms Hatton’s claim that NVersion programming is useful, even in the presence of correlated faults [12].
running stateful services. Replication over other specifications is a direction for future work. Sampling variability. Each admitted version corresponds to a single [harness, model, language] tuple; we therefore measure diversity across distinct configurations rather than withinconfiguration variability due to LLM sampling. Rerunning the generation stage at either a different temperature or at a different date would likely change the exact failure counts. VI. R ELATED W ORK Recent work on LLM-based code generation has shown that diversity can improve correctness. Chen et al. [7] and Li et al. [17] established that sampling multiple candidates from a single model can improve pass@k solve rates. Kodati et al. [16] further show that sampling candidates from different LLMs increases solve rates compared to single model sampling. In a related line of work, Wang et al. [24] show that self-consistency improves accuracy in code generation. Mahmud et al. [20] use ensembles of LLM outputs together with syntactic and behavioral similarity signals to improve HumanEval pass rates, and Valentin et al. [23] argue that cross-candidate incoherence can serve as an oracle-less proxy for error. Differential testing of generated programs has also been suggested by Kessel et al. [14] as a way to enhance generation quality. These papers all exploit disagreement or agreement across candidates as useful engineering signals. The key difference with our work is that they do not study the reliability of a population of generated implementations under the lens of N-version programming models. A separate thread of work emphasizes that modern software systems are often assembled and revised under changing requirements instead of built once from a complete specification. Liu et al. [19] frame this as a move toward just-intime systems, where specifications, interfaces, and generated code evolve together. This perspective is relevant to N-version generation because repeated failures across independently generated programs may reveal not only implementation mistakes, but also parts of the specification that are difficult for agents to interpret consistently. The closest work to our setting is Galapagos [22], which consists in generating diverse but functionally equivalent variants with guarantees. In contrast to our work, (1) Galapagos uses a reference implementation as input to LLMs while we use a natural language specification; (2) Galapagos uses so2023 single-shot prompting, as opposed to our usage of agentic coding.
V. T HREATS TO VALIDITY Scope of Interpretation The present study is about implementation-level faults relative to a fixed specification and a fixed oracle provided by a reference implementation. It does not test whether multi-version execution mitigate runtime or transient failures such as network outages, external tool crashes, or language-specific stack components. Single benchmark. The experiment is based on a single specification: the LIP problem. Conclusions drawn from it may not generalize to specifications in other domains, such as long-
VII. C ONCLUSION This paper asked whether modern AI coding agents can serve as effective generators of diverse software versions for N-version programming. Our results show that the answer is yes in an operational sense: agent systems can readily generate enough working implementations to recreate fully working NVersion units. On the question of failure independence, the resulting versions do not fail independently, confirming Knight–Leveson’s
11
result from TSE 1986. Across the 48 admitted implementations in the campaign archive, the experiment produces 429 coincident-failure cases where the independence model predicts 115.36. Strong pairwise similarity persists across both language and agent boundaries. That result rules out the idealized assumption or random faults behind Knight– Leveson’s paper. Yet, this result does not make agent-generated redundancy irrelevant in practice. First, the structure of those coincidental failures provides relevant information about the specification itself. The implementation faults are not arbitrary: they can be repeatedly traced back to a small set of specification weaknesses. Second, we creating N-version units from those correlated versions, the majority voting mean failure count drops significantly. To sum, N-Version programming with coding agents is both doable and useful. An interesting avenue of future research is to study how to use the correlated faults to detect and refine ambiguous specifications in an automated manner.
R EFERENCES [1] Anthropic. 2024. Claude Code: Agentic Coding in the Terminal. https://docs.anthropic.com/en/docs/ claude-code. Accessed: 2026. [2] Anysphere Inc. 2024. Cursor: The AI Code Editor. https: //www.cursor.com. Accessed: 2026. [3] Algirdas Avizienis. 1985. The N-Version Approach to Fault-Tolerant Software. IEEE Transactions on Software Engineering 11, 12 (1985), 1491–1501. [4] Peter G. Bishop. 1995. Review of Software Design Diversity. In Software Fault Tolerance, Michael R. Lyu (Ed.). John Wiley & Sons, New York, NY, 211–229. [5] Susan S. Brilliant, John C. Knight, and Nancy G. Leveson. 1990. Analysis of Faults in an N-Version Software Experiment. IEEE Transactions on Software Engineering 16, 2 (1990), 238–247. [6] Liming Chen and Algirdas Avizienis. 1978. N-Version Programming: A Fault-Tolerance Approach to Reliability of Software Operation. In Digest of Papers, Eighth Annual International Symposium on Fault-Tolerant Computing (FTCS-8). Toulouse, France, 3–9. [7] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374 (2021). [8] Janet R. Dunham and John L. Pierce. 1986. An Experiment in Software Reliability. NASA Contractor Report NASA-CR-172553. NASA Langley Research Center, Hampton, VA. https://ntrs.nasa.gov/api/citations/ 19860020075/downloads/19860020075.pdf Prepared by Research Triangle Institute under Contract NAS1-16489; originally dated March 1985, revised May 1986. [9] David E. Eckhardt and Larry D. Lee. 1985. A Theoretical Basis for the Analysis of Multiversion Software Subject to Coincident Errors. IEEE Transactions on Software Engineering 11, 12 (1985), 1511–1517.
[10] Google DeepMind. 2024. Gemini CLI: An Open-Source AI Agent. https://github.com/google-gemini/gemini-cli. Accessed: 2026. [11] Ahmed E. Hassan, Hao Li, Dayi Lin, Bram Adams, Tse-Hsun Chen, Yutaro Kashiwa, and Dong Qiu. 2025. Agentic Software Engineering: Foundational Pillars and a Research Roadmap. arXiv:2509.06216 [cs.SE] https: //arxiv.org/abs/2509.06216 [12] Les Hatton. 1997. N-Version Design Versus One Good Version. IEEE Software 14, 6 (1997), 71–76. [13] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. In Proceedings of the 12th International Conference on Learning Representations (ICLR). OpenReview.net, Vienna, Austria. [14] Marcus Kessel and Colin Atkinson. 2024. N-version assessment and enhancement of generative AI: differential GAI. IEEE Software 42, 2 (2024), 76–83. [15] John C. Knight and Nancy G. Leveson. 1986. An Experimental Evaluation of the Assumption of Independence in Multiversion Programming. IEEE Transactions on Software Engineering 12, 1 (1986), 96–109. [16] Abhishek Kodati, Foutse Khomh, and Ashkan Sami. [n. d.]. MAC: Multi-Agent LLM Coder is All You Need. Available at SSRN 5887028 ([n. d.]). [17] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022. Competition-Level Code Generation with AlphaCode. Science 378, 6624 (2022), 1092–1097. [18] Bev Littlewood and Douglas R. Miller. 1989. Conceptual Modeling of Coincident Failures in Multiversion Software. IEEE Transactions on Software Engineering 15, 12 (1989), 1596–1614. [19] Shu Liu, Alexander Krentsel, Shubham Agarwal, Mert Cemri, Ziming Mao, Soujanya Ponnapalli, Alexandros G. Dimakis, Sylvia Ratnasamy, Matei Zaharia, Aditya Parameswaran, and Ion Stoica. 2026. The Time is Here for Just-in-Time Systems: Challenges and Opportunities. arXiv:2605.24096 [cs.DB] https://arxiv.org/abs/ 2605.24096 [20] Tarek Mahmud, Bin Duan, Corina S. Păsăreanu, and Guowei Yang. 2025. Enhancing LLM Code Generation with Ensembles: A Similarity-Based Selection Approach. arXiv preprint arXiv:2503.15838 (2025). [21] OpenCode Contributors. 2024. OpenCode: The AI Coding Agent Built for the Terminal. https://opencode.ai. Accessed: 2026. [22] Javier Ron, Diogo Gaspar, Javier Cabrera-Arteaga, Benoit Baudry, and Martin Monperrus. 2025. Galapagos: Automated N-Version Programming with LLMs. ACM Transactions on Software Engineering and Methodology (2025). doi:10.1145/3785363 [23] Thomas Valentin, Ardi Madadi, Gaetano Sapia, and Marcel Böhme. 2025. Incoherence as Oracle-less Measure of Error in LLM-Based Code Generation. arXiv:2507.00057 [cs.PL] https://arxiv.org/abs/2507.
12
00057 [24] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed H. Chi, and Denny Zhou. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In Proceedings of the 11th International Conference on Learning Representations (ICLR). OpenReview.net. [25] Junjun Zheng, Hiroyuki Okamura, and Tadashi Dohi. 2025. Can Generative AI Enhance the Effectiveness of NVersion Programming?. In International Symposium on Software Fault Prevention, Verification, and Validation. Springer, 1–16.