Beyond Surface Metrics: Multi-Condition Requirements in Enterprise Maintenance and AI Coding Benchmarks

Beyond Surface Metrics: Multi-Condition Requirements in Enterprise Maintenance and AI Coding Benchmarks Heeyong Eun KPMG Korea, Seoul, South Korea heeyonge@gmail.com

arXiv:2611.03162v1 [cs.SE] 2 Nov 2026

Abstract Repository-level AI coding benchmarks measure whether agents solve sampled software tasks, but task size and benchmark success do not directly describe the conditions that constitute maintenance completion. We examine multi-condition fulfillment: situations in which a maintenance activity must satisfy several operational conditions, sometimes linked by directional dependencies. Using a frozen 18-criteria taxonomy, we analyze 2,925 Apache Fineract issues and 944 AI benchmark tasks (SWE-bench Pro, SWE-PolyBench Java, and SWE-EVO) through three complementary lenses: attribute breadth (|B|), corpus-level co-occurrence (ϕ), and source-grounded directional requirement relationships. First, 1,177 benchmark task pairs matched on issue classification, modified-file count, and code churn have mean fulfillment-set Jaccard distance 0.8138; 57.4% have disjoint condition sets. Second, attribute count does not explain the structural contrast: SWE-bench Pro averages |B| = 2.47 versus Fineract 1.78. Temporal (B4 ) and numerical (B2 ) conditions have ϕ = +0.430 in Fineract versus −0.013 in the pooled AI corpus; among B4 -tagged tasks, B2 is also specified in 48.85% (254/520) versus 5.81% (5/86), respectively. These associations are descriptive, not causal. Third, 16 source-grounded Fineract requirement relationships illustrate directional obligations involving time, calculations, and state preservation. Four selected benchmark cases then illustrate how parts of such fulfillment contracts appear in inspected evaluation assertions; they do not estimate benchmark-wide oracle omissions. Our results show that matched surface task profiles can conceal disjoint fulfillment-condition sets, while condition counts alone do not capture observed co-specification patterns. We therefore advocate reporting condition breadth, co-occurrence, and documented directional relationships as distinct properties of maintenance benchmarks. The corpus contrast is descriptive and may reflect domain differences; no production failure rate or causal selection mechanism is inferred.

Keywords: AI Coding Agents; Software Maintenance; Benchmark Evaluation; Multi-Condition Fulfillment; Requirement Dependencies; Enterprise Software

1

Introduction

1.1

Motivation: What Counts as Maintenance Completion?

The rapid evolution of Large Language Models (LLMs) and autonomous scaffolding frameworks has transformed automated program repair into repository-scale software engineering. Benchmarks such as SWE-bench [1], SWE-bench Pro [2], SWE-PolyBench [3], and SWE-EVO [4] evaluate coding agents on tasks derived from software issues, repository changes, or evolution histories. Although coding-agent benchmarks provide quantitative performance measurements, their leaderboard scores do not directly establish readiness for enterprise software maintenance. A fundamental question precedes test adequacy: what combination of conditions defines completion of a maintenance task? In enterprise services, a single change can entail temporally ordered recalculation, preservation of prior data, and consistent downstream state. These are not necessarily independent checklist items: one operation can trigger another, and the task is complete only when the applicable conditions hold together. We call this multi-condition fulfillment, distinguishing the number of conditions from their relationships. Apache Fineract backdating requirements provide concrete instances of such linked obligations; whether comparable structures are visible in benchmark task descriptions is an empirical question. 1

The distinction matters for benchmark interpretation. A task may mention several conditions without stating how they interact, while a compact issue may specify a sequence of dependent operations. Evaluation tests are one downstream way to check those conditions, not the definition of the conditions themselves. We therefore first compare requirement structures across corpora, and only then inspect selected benchmark tests as illustrative consequences for evaluation design.

1.2

Limitations of Surface Complexity Metrics

Existing benchmark frameworks predominantly evaluate task difficulty through surface syntactic and navigational metrics: (1) Syntactic Churn (LOC added/deleted, file counts); (2) Repository Topography (search space breadth, AST exploration radius, directory depth [5]); and (3) Generic Labels ("Bug Fix", "Feature"). We examine whether surface task descriptors obscure differences in fulfillment-condition structures. Some maintenance activities (A) impose dependent requirements (B), as illustrated by backdated financial operations: triggers ∆S A −−→ Btemporal −−−−→ Bnumeric |= Binvariant (1) The analytical question is whether tasks with similar surface profiles describe comparable sets and relationships of fulfillment conditions; evaluation coverage is a subsequent implication, not the primary outcome measured here.

1.3

Research Questions

This investigation is guided by two fundamental research questions: • RQ1 (Semantic Heterogeneity & Attribute Diversity): To what extent do conventional benchmark task categories and surface churn metrics conceal semantic heterogeneity, and how does the diversity of required maintenance attributes (|B|) compare between enterprise maintenance and AI coding benchmarks? • RQ2 (Multi-Condition Fulfillment Structure): (a) Corpus-Level Co-occurrence (RQ2a): How do observed patterns of jointly specified conditions (ϕ) differ between the studied enterprise and benchmark corpora? (b) Directional Dependencies (RQ2b): What source-grounded action– condition and condition–condition relationships characterize selected enterprise maintenance requirements, and what do selected benchmark cases illustrate about representing these obligations in evaluation?

1.4

Core Contributions

We present a mixed-methods empirical study grounded in verified raw data across three distinct analytical levels: 1. Unmasking Semantic Heterogeneity (Level 1 / E1): In 1,177 controlled task pairs matched on repository native types and code churn (714 SWE-bench Pro pairs, 463 PolyBench Java pairs, 581 unique tasks), matching on surface task descriptors does not guarantee similar fulfillmentcondition sets (mean dJ = 0.8138, median 1.0000, σ = 0.2745), with 57.4% of pairs (676 pairs) exhibiting completely disjoint criteria (dJ = 1.0). 2. Attribute Diversity Paradox (Level 2 / E2 Macro): Across 944 AI tasks (SWE-bench Pro N = 731, PolyBench N = 165, SWE-EVO N = 48), SWE-bench Pro exhibits higher attribute diversity (mean |B| = 2.47) than enterprise core banking (Fineract 1.78, Welch’s t = 15.78, p < 10−50 ). However, linear co-occurrence between temporal backdating (B4 ) and numerical recalculation (B2 ) in AI benchmarks shows near-zero correlation (ϕ = −0.013, p = 0.825, 5 tasks) vs. enterprise banking (ϕ = +0.430, p < 10−90 , 254 issues), showing that attribute breadth alone does not characterize joint-condition structure.

2

3. Directional Fulfillment Contracts (Level 3 / E2 Micro): We trace 16 source-grounded Fineract requirement relationships (19 quotes) to illustrate how operations such as backdating, recalculation, and state preservation form joint completion obligations. Four selected benchmark cases illustrate possible alignment boundaries between explicit conditions and inspected assertions; they do not establish a population-level failure rate. 4. Reporting Implications: We distinguish attribute breadth, statistical co-occurrence, and source-grounded directionality as non-interchangeable properties of maintenance tasks, with secondary implications for specification-aligned benchmark checks.

2

Related Work & Research Positioning

2.1

Evolution of AI Coding Benchmarks

Automated code generation benchmarks have evolved rapidly from isolated synthetic function completion (e.g., HumanEval [6], MBPP) to repository-level software maintenance. SWE-bench [1] pioneered the extraction of historical pull requests from Python repositories, packaging them with reproducible execution containers. Subsequent benchmarks extended this paradigm: (1) SWE-PolyBench [3] supports repository-level evaluation across Java, JavaScript, TypeScript, and Python; (2) SWEbench Pro [2] targets long-horizon, multi-file software engineering tasks across public, held-out, and commercial splits; and (3) SWE-EVO [4] introduces longitudinal software evolution across sequential commits to evaluate regression avoidance. Across all these benchmarks, candidate patches are evaluated by executing automated test harnesses to verify whether target unit tests transition from failing to passing (fail_to_pass) without inducing regressions in pre-existing tests (pass_to_pass).

2.2

Characterizing Benchmark Complexity & Audit Studies

Several empirical frameworks have investigated the demands of repository-level coding tasks. Syntactic & Navigational Complexity (SNC) [5] profiles Spread (file distribution entropy), Novelty (the proportion of added versus deleted code), and Centrality (dependency graph importance), examining task demands through code-change and repository structure. In contrast, UTBoost [7] identifies insufficient issue-specific benchmark tests that can allow incorrect candidate patches to pass, and augments tests to improve patch discrimination. These studies examine repository-level demands and patch evaluation, respectively, rather than the specified structure of multi-condition maintenance completion. The Distinct Dimension: Multi-Condition Fulfillment Structure. Test-adequacy studies such as UTBoost address whether existing tests discriminate correct and incorrect patches. We instead examine what a maintenance task requires: which completion conditions are specified, which appear together, and which are directionally dependent. We do not claim that test coverage has been unexplored, nor that our descriptive corpus comparisons prove a missing condition in any entire benchmark harness.

2.3

Research Positioning: From Task Size to Joint Fulfillment Conditions

As summarized in Table 1, repository-complexity studies analyze properties of code changes and navigation demands, while test-audit work examines patch evaluation. Our unit of analysis is the maintenance fulfillment contract: developer activities (A), specified conditions (B), and, where supported by source text, relationships among those conditions. We distinguish three complementary observations: (1) condition breadth within an issue (|B|); (2) corpus-level co-occurrence (ϕ), which does not establish direction; and (3) explicit or source-grounded directional obligations linking actions to downstream states. Selected oracle inspections are a secondary application of this characterization. The principal research question is therefore not simply whether a test catches a buggy patch. It is: Which fulfillment conditions co-occur across the analyzed corpora, and what source-grounded dependencies characterize completion in selected enterprise maintenance requirements?

3

Table 1: Research positioning by unit of analysis and primary contribution. Analytical mension

Di-

SNC Audit [5]

UTBoost [7]

This Work

Primary Problem

Repository topography ambiguity and task demand categorization

Incomplete tests and false pass judgments

Multi-condition structure of maintenance completion

Unit of Analysis

Repository graph metrics (AST distance, spread, novelty)

Tests and candidate patches in SWE-bench tasks

Requirement fulfillment invariants (A → ∆S → B)

Requirement Perspective

Surface issue labels (Bug, Feature, Enhancement)

Task-specific tests and patch evaluation

Ontological taxonomy (ISO 14764 / 25010) with verified quotes

Oracle Evaluation Scope

Accepts existing benchmark test harnesses as ground truth

Augments benchmark tests to flag incorrect passes

Secondary case-level assertion inspection

Empirical Scope

SWE-bench Lite/Verified (Python repository graphs)

SWE-bench Python repos (LLM-augmented test suites)

Analyzed AI tasks (N = 944) vs. Fineract issues (N = 2, 925)

3

Empirical Foundations & Methodology

3.1

Reference Corpora

Our empirical foundation comprises two extensive software maintenance corpora: • Enterprise Reference Corpus (Apache Fineract): Apache Fineract is an open-source core banking platform maintained by the Apache Software Foundation [8], providing financial service infrastructure (loan scheduling, multi-currency ledgers, portfolio management). We analyze a collected corpus of 2,925 maintenance issues from its issue tracker, comprising 3,811 granular requirement units, including 2,393 issues identified as resolved in the source metadata. Fineract serves as a domain-specific enterprise reference corpus in which maintenance issues can impose financial calculations, audit requirements, and state-consistency constraints; its banking domain limits direct generalization to other systems. • AI Coding Benchmark Corpus (944 Analyzed Tasks): We analyze 944 tasks from three benchmark sources: (1) SWE-bench Pro public V1: 731 tasks (77.44%) from its original public split; (2) SWE-PolyBench Java: 165 tasks (17.48%) from its Java split; and (3) SWE-EVO: 48 tasks (5.08%) included in the analyzed corpus. These counts define the scope of our study rather than the complete populations of all benchmark releases. Across these 944 tasks, 3,446 itemized criteria assignments were systematically annotated with exact text quotes and code diff references.

3.2

The Immutable 18-Criteria Maintenance Taxonomy: Provenance and Grounding

To support a transparent and consistently applied cross-domain comparison, we establish an immutable 18-criteria maintenance taxonomy in three operational stages: (1) Standard-Based Concept Derivation: Development Activities (A1 –A6 ) were adapted from the maintenance categories of ISO/IEC/IEEE 14764 [9] (Corrective, Adaptive, Perfective, Preventive), while Fulfillment Conditions (B1 –B12 ) were adapted from the quality characteristics of ISO/IEC 25010 (SQuaRE) [10] formalized as state invariants; (2) Empirical Concretization & Codebook Freezing: Through inductive thematic analysis across 2,925 Apache Fineract maintenance issues (3,811 requirement units), standard categories were operationalized into concrete criteria and formally frozen into an immutable codebook (framework-snapshot.json); and (3) Uniform Cross-Corpus Application: This frozen taxonomy was applied without alteration across the 944 analyzed AI benchmark tasks, maintaining consistent operational definitions across the selected corpora. The proposed 18-criteria taxonomy is 4

an empirically operationalized classification informed by ISO/IEC/IEEE 14764 and ISO/IEC 25010, rather than a direct reproduction of either standard. The six activity categories and twelve fulfillment conditions were developed through inductive analysis of Apache Fineract maintenance requirements, with their operational definitions and classification boundaries documented in the accompanying codebook. 3.2.1

Ontological Separation: Activities (A) vs. Conditions (B)

Conventional benchmarks collapse maintenance tasks into 1-dimensional categorical labels ("Bug Fix", "Feature"). However, in enterprise service maintenance, the developer’s maintenance activity (Activity, A) is ontologically distinct from the multi-step system invariants (Conditions, B) required to declare the ∆S

triggers

task complete: A −−→ Btemporal −−−−→ Bnumeric |= Binvariant . The taxonomy establishes 6 Development Activities (A1 –A6 : Product Operation, Structure/Refactoring, Testing/QA, Execution/Operations, Deployment/Migration, Process/Docs) and 12 Fulfillment Conditions (B1 –B12 : Functional Result, Numerical Precision, Data Consistency, Temporal Semantics, State Transition, Legacy Preservation, Concurrency Boundaries, Authorization, Performance, Service Continuity, Interface Compatibility, User Interaction), quantified across corpora in Table 2. 3.2.2

Annotation Procedure, Semantic Validity, and Boundary Resolution

To ensure criteria assignments reflect authentic operational semantics rather than keyword matching, we enforced strict annotation protocols: (1) Operational Rules: A labels required concrete code diff intent (A1 functional logic, A2 structural refactoring), while B conditions required explicit specification mandates for state invariants or quality constraints; (2) Boundary Adjudication: (a) B1 vs. B2 : input validation and boolean returns were coded as B1 , whereas financial calculations, rounding, or multi-threaded counters were coded as B2 ; (b) A1 vs. A2 : functional behavior modifications were classified as A1 , whereas internal refactoring preserving behavior was classified as A2 ; (c) B3 vs. B6 : multi-entity referential integrity was coded as B3 , whereas legacy data retention and non-regression guarantees were coded as B6 ; (3) Batch Protocol & Verification: Semantic coding across the 944 tasks was conducted by the primary investigator via an iterative batch protocol binding all 3,446 criteria assignments to verbatim textual quotes and rationales. An automated pipeline (validate_*.py) programmatically verified contiguous task sequences, file hashes, exact string offsets in raw JSON, and codebook schema compliance, ensuring quotation fidelity rather than dual-coder consensus (documented in Section 6); and (4) Procedural Consistency: Applying this immutable codebook across both corpora ensures procedural consistency while preserving domain characteristics.

3.3

Three-Level Analytical Protocol: Distinct Dimensions of Construct Validity

To separate the breadth of completion conditions from their relationships, we use three complementary lenses (Figure 1): (1) Condition Breadth (|B|) counts distinct tagged fulfillment criteria in a task, without asserting that they interact; (2) Corpus Co-occurrence (ϕ) measures pairwise association across issue specifications, without identifying direction or causality; and (3) Source-Grounded Directional Dependencies traces action–condition or condition–condition obligations when supported by requirement text. The figure combines findings from FINERACT-753 and FINERACT-2338 as an illustrative synthesis, not a verbatim single-issue specification. Subsequent inspection of selected benchmark assertions is an optional evaluation-facing application of these three lenses, not a fourth measurement of task complexity.

4

E1: Unmasking Semantic Heterogeneity Beyond Surface Metrics

4.1

Unmasking Semantic Heterogeneity Beyond Surface Churn

Conventional benchmark leaderboards and dataset summaries commonly categorize maintenance tasks under broad native issue classifications (e.g., "Bug Fix") alongside summary metrics of code churn and

5

A maintenance task may require multiple conditions to hold together. 1. Condition breadth

2. Pairwise association

3. Directed obligations

Count the specified conditions: B4 temporal ordering B2 numerical recalculation B3 data consistency B6 state preservation |B| = 4 is not coupling.

Measure co-occurrence across issues:

Trace relations in source text:

Fineract: ϕ(B4 , B2 ) = +0.430 AI corpus: ϕ(B4 , B2 ) = −0.013

A1 → B4 → B2 B2 → B6 (illustrative) Source evidence is required.

Association is not direction.

Figure 1: Three complementary lenses for multi-condition fulfillment: breadth, co-occurrence, and source-grounded directionality. The illustrated dependency chain synthesizes FINERACT-753 and FINERACT-2338 rather than quoting a single task; oracle inspection is a downstream application. file counts. To empirically evaluate whether tasks sharing comparable surface characteristics exhibit comparable requirement structures, we constructed a controlled matched-pair analysis across the 944 benchmark tasks. We identified pairs of tasks (ti , tj ) that satisfied three equivalence constraints: 1. Identical native repository issue classification. 2. Identical modified file count (|F iles(ti )| = |F iles(tj )|). 3. Code churn: |LOCi − LOCj | ≤ max{3, 0.30 max(LOCi , LOCj )}. For each matched pair, we computed the Jaccard distance over their active fulfillment condition sets (B(ti ), B(tj )): |B(ti ) ∩ B(tj )| dJ (ti , tj ) = 1 − |B(ti ) ∪ B(tj )| We emphasize that this matched-pair construction controls for surface task similarity (repository issue classification, file modification count, and line churn magnitude), but does not imply or guarantee identical cognitive problem-solving difficulty. Rather, it isolates whether tasks with matched issue classifications and file counts and churn within tolerance share common fulfillment criteria. Across N = 1, 177 surface-matched pairs (714 SWE-bench Pro pairs and 463 PolyBench Java pairs, spanning 581 unique tasks), the mean Jaccard distance is d¯J = 0.8138 (median 1.0000, sample standard deviation σ = 0.2745). Crucially, 57.4% of all controlled pairs (676 pairs) exhibit dJ = 1.0000, meaning their underlying fulfillment conditions are completely disjoint despite matching issue classifications and modified-file counts, with code churn within the specified tolerance. A representative example is observed in internetarchive/openlibrary: OpenLibrary #0143 (29 LOC, 1 file, tagged Bug Fix) modifies author-edition entity relationships requiring relational state consistency (A1 → B3 , B5 ), whereas OpenLibrary #0233 (27 LOC, 1 file, tagged Bug Fix) corrects string sanitization and character encoding compatibility (A1 → B1 , B6 , B11 ). Although both tasks have comparable surface characteristics (1 modified file each, 27 and 29 LOC, native ’Bug Fix’ tag), their underlying semantic fulfillment conditions are completely disjoint (dJ = 1.0000), demonstrating that surface metadata fails to reflect whether a task demands relational database integrity (#0143) or character encoding transformations (#0233).

4.2

Stratified Benchmark Family Profile (N=944)

Rather than aggregating AI benchmarks into a monolithic entity, we analyze empirical distributions stratified across the three constituent benchmark families in Table 2 (gate3_stratified_family_ metrics.csv). 6

Table 2: Stratified 18-Criteria Distribution Across Benchmark Families (N = 944). Code Codebook Criterion Name

AI All (N = 944) Pro (N = 731) Poly (N = 165) EVO (N = 48)

Product Operation & Business Rules Structure & Implementation Method Testing & Quality Assurance Execution & Operations Deployment & Environment Migration Development Process & Docs

426 (45.13%) 572 (60.59%) 94 (9.96%) 122 (12.92%) 60 (6.36%)

300 (41.04%) 505 (69.08%) 70 (9.58%) 93 (12.72%) 39 (5.34%)

84 (50.91%) 50 (30.30%) 10 (6.06%) 20 (12.12%) 9 (5.45%)

42 (87.50%) 17 (35.42%) 14 (29.17%) 9 (18.75%) 12 (25.00%)

53 (5.61%)

25 (3.42%)

17 (10.30%)

11 (22.92%)

315 (33.37%) 65 (6.89%) 214 (22.67%) 86 (9.11%) 131 (13.88%)

276 (37.76%) 49 (6.70%) 184 (25.17%) 77 (10.53%) 109 (14.91%)

25 (15.15%) 11 (6.67%) 25 (15.15%) 7 (4.24%) 20 (12.12%)

14 (29.17%) 5 (10.42%) 5 (10.42%) 2 (4.17%) 2 (4.17%)

368 (38.98%)

336 (45.96%)

17 (10.30%)

15 (31.25%)

42 (4.45%)

35 (4.79%)

4 (2.42%)

3 (6.25%)

154 (16.31%) 83 (8.79%) 87 (9.22%)

130 (17.78%) 60 (8.21%) 72 (9.85%)

16 (9.70%) 13 (7.88%) 8 (4.85%)

8 (16.67%) 10 (20.83%) 7 (14.58%)

391 (41.42%)

311 (42.54%)

51 (30.91%)

29 (60.42%)

B12

Functional Result & Input Condition Numerical & Calculation Precision Data Relationship & Consistency Temporal Semantics & Ordering State Transition & Business Workflow Preservation of Existing Data & Behavior Concurrency & Transaction Boundaries Authorization & Data Protection Performance & Resource Utilization Service Continuity & Failure Handling Interface & Environmental Compatibility User Interaction Semantics

183 (19.39%)

169 (23.12%)

4 (2.42%)

10 (20.83%)

4.3

Multi-Attribute Co-presence Complexity (|B|)

A1 A2 A3 A4 A5 A6 B1 B2 B3 B4 B5 B6 B7 B8 B9 B10 B11

Table 3 reports the distribution of |B| (gate3_coupling_complexity.csv). Table 3: Fulfillment-Condition Breadth: Distribution of |B| (not a measure of dependency strength). Metric / Bin

Fineract (N = 2, 925) AI All (N = 944) Pro (N = 731) Poly (N = 165) EVO (N = 48)

Mean |B| Median |B|

1.78 (1.7798) 2.0

2.24 (2.2447) 2.0

2.47 (2.4733) 2.0

1.22 (1.2182) 1.0

2.29 (2.2917) 2.0

|B| = 0 |B| = 1 |B| = 2 |B| = 3 |B| = 4 |B| = 5 |B| = 6 |B| = 7 |B| = 8

331 (11.32%) 872 (29.81%) 1,097 (37.50%) 451 (15.42%) 112 (3.83%) 37 (1.26%) 21 (0.72%) 4 (0.14%) 0 (0.00%)

3 (0.32%) 248 (26.27%) 357 (37.82%) 231 (24.47%) 79 (8.37%) 15 (1.59%) 6 (0.64%) 3 (0.32%) 2 (0.21%)

0 (0.00%) 109 (14.91%) 304 (41.59%) 219 (29.96%) 77 (10.53%) 12 (1.64%) 6 (0.82%) 2 (0.27%) 2 (0.27%)

3 (1.82%) 126 (76.36%) 33 (20.00%) 3 (1.82%) 0 (0.00%) 0 (0.00%) 0 (0.00%) 0 (0.00%) 0 (0.00%)

0 (0.00%) 13 (27.08%) 20 (41.67%) 9 (18.75%) 2 (4.17%) 3 (6.25%) 0 (0.00%) 1 (2.08%) 0 (0.00%)

Subtotal (|B| ≥ 2)

1,722 (58.87%)

693 (73.41%)

622 (85.09%)

36 (21.82%)

35 (72.92%)

4.4

Attribute Breadth Is Not Joint Fulfillment Structure

In the analyzed corpora, SWE-bench Pro has a higher mean number of tagged fulfillment conditions (|B| = 2.47) than Apache Fineract (1.78, Welch’s t = 15.78, p < 10−50 ). Thus the contrast cannot be reduced to a claim that benchmark tasks contain fewer conditions. But |B| measures breadth only: two tasks with the same count may differ in which conditions must hold jointly and whether one condition triggers another. This result does not by itself show that enterprise issues are universally deeper or more difficult. This motivates the next question: how do specified conditions combine? Section 5 first examines corpus-level pairwise association and then traces source-grounded directional relationships in enterprise issues. Selected benchmark cases subsequently illustrate why the distinction can matter for evaluation, without treating test-assertion coverage as our principal empirical outcome.

7

5

E2: Joint Fulfillment Patterns and Directional Dependencies

5.1

Macro Statistical Co-occurrence Divergence (ϕ)

To measure how often pairs of fulfillment conditions are specified together, we compute Pearson’s ϕ across the studied corpora (Table 4; gate3_phi_correlation_matrix.csv). This is a measure of association, not evidence that one condition triggers another. The near-zero ϕ for B4 and B2 in the pooled AI tasks (−0.013, p = 0.825) indicates little observed pairwise association, not proof of independence or the absence of dependent tasks. Conditional and higher-order co-specification. To make the joint-condition contrast interpretable, we calculate the proportion of B4 -tagged issues that also specify B2 . In Fineract, this is 254/520 (48.85%); in the pooled AI benchmark corpus, it is 5/86 (5.81%). The descriptive conditional risk ratio is 8.40 (95% CI: 3.57–19.76; two-sided Fisher’s exact test, p = 3.14 × 10−16 ). An exploratory higher-order check finds B4 and B2 jointly specified with at least one of B3 (data consistency) or B6 (preservation) in 108/2,925 Fineract issues versus 1/944 benchmark tasks. These counts are computed from the frozen criterion annotations (gate5_multi_condition_summary.csv); they measure co-specification, not directed dependency rates or causal depth. Because the corpora represent different software domains, the contrast cannot be attributed to benchmark design alone. To examine directionality, Section 5.2 separately inspects source-grounded requirements. Table 4: Pairwise Co-occurrence (ϕ); associations do not establish causal direction. Pair

Condition Pair (non-directional)

B4 ↔ B2 A1 ↔ B4 A1 ↔ B2 A1 ↔ B3 B4 ↔ B3 B4 ↔ B6 B2 ↔ B3 B2 ↔ B6 B3 ↔ B6 B7 ↔ B6 B7 ↔ B3

Temporal / Numerical Business Rule / Temporal Business Rule / Numerical Business Rule / Consistency Temporal / Consistency Temporal / Preservation Numerical / Integrity Numerical / Preservation Integrity / Preservation Isolation / Preservation Isolation / Integrity

5.2

Fineract (N = 2, 925) AI Bench (N = 944) Co-occur / ϕ Co-occur / ϕ (p) 254 / +0.430 471 / +0.309 440 / +0.343 538 / +0.329 138 / +0.069 84 / +0.068 189 / +0.226 64 / +0.036 89 / +0.053 8 / +0.005 20 / +0.040

5 / −0.013 (0.825) 42 / +0.024 (0.496) 34 / +0.039 (0.246) 115 / +0.094 (0.005) 22 / +0.022 (0.501) 28 / −0.042 (0.246) 14 / −0.007 (1.000) 19 / −0.054 (0.113) 63 / −0.106 (0.001) 14 / −0.025 (0.519) 7 / −0.031 (0.451)

Source-Grounded Directional Fulfillment in Enterprise Maintenance

To characterize what a joint completion obligation can look like, we examine 16 source-grounded business requirement relationships in Apache Fineract (Table 5; gate2_dependencies.csv). The FINERACT-2338 examples connect backdating to repayment recalculation and preservation constraints; FINERACT-2836 specifies transaction boundaries for savings-charge processing. Supporting quotations were checked against source text (gate2_raw_source_snippets.json). These purposively documented relationships illustrate directionality; they are not a frequency estimate or evidence that all Fineract issues exhibit this structure.

5.3

Illustrative Evaluation Implications in Four Benchmark Tasks

The preceding results concern specified conditions and their relationships, not agent success or general test quality. As a secondary illustration, we inspected task text, source diffs, and selected test assertions for four benchmark tasks (Table 6; data_and_tables/ai_case_sources/). These cases show why a multi-condition specification can call for more than a localized check, and also show instances of appropriate alignment. They are not representative samples and do not establish full-harness behavior. We establish a unified 3-Question Comparative Protocol applied systematically across all four axes: (1) Specification Mandate (Rspec ): operational state transformations and invariants required by the raw task description and DoD; (2) Oracle Assertion Scope (Reval ): specific conditions directly asserted by test code in the benchmark’s evaluation patch; and (3) Construct 8

Table 5: Representative Source-Grounded Requirement Dependencies in Apache Fineract Maintenance (Full set of 16 documented relationships in the replication package). ID

Issue

Classification

Structural Chain

Verbatim Jira Excerpt

Causal Cascade Focus: Multi-Tier Loan Modification (FINERACT-2338) “Backdate can be done on DEP-01 FINERACT-2338 Explicit Dependency A1 → B4 already paid Installments as well” “Repayment schedule will be DEP-02 FINERACT-2338 Explicit Dependency B4 → B2 recalculated with New EMI and Interest from the Interest applied schedule date” “No of Installments will remain DEP-03 FINERACT-2338 Explicit Dependency B2 → B6 the same, only EMI and Interest would get affected” DEP-04 FINERACT-2338 Explicit Dependency B4 → B3 “standard accounting rules and not charge-off accounting rules” Cross-Entity Invariant Preservation & Transaction Isolation DEP-05 FINERACT-753 Explicit Dependency A1 → B4 , B2 DEP-07

FINERACT-2836

Explicit Dependency

B7 → B6

“now reverse the deposit of 500 .” “Each due savings-charge installment is processed in an independent transaction.”

Alignment or Structural Gap: structural correspondence or divergence between mandate and assertions. To distinguish task demands from implementation details, requirement conditions are categorized into three provenance tiers: (a) Explicit Requirement (directly stated in prompt or interface); (b) Developer-Implemented Obligation (concrete implementation in reference patch); and (c) Inferred Requirement (deduced from architectural context but unstated in prompt). 5.3.1

Comparative Case Analysis across the Three Questions and Provenance Tiers

Applying the protocol illustrates different alignments between explicit task obligations and the inspected assertions; it does not quantify the prevalence of evaluation gaps: Axis 1 (NodeBB #0030: Construct Underrepresentation). Mandate (Rspec ): The issue explicitly mandates an upgrade path in src/upgrades/4.3.0/chat_allow_list.js to migrate legacy restrictChat=1 records (Explicit Requirement), implemented via a 44-line batch migration script (batch.processSortedSet, 500 records/chunk) in the developer patch (+383/-50 lines across 12 files). Oracle (Reval ): Static inspection of the analyzed test patch (test/messaging.js:L103-121) confirms it creates mock users in memory with pre-populated chatAllowList arrays, asserting Messaging.canMessageUser() with zero assertions exercising src/upgrades/. Structural Gap: Static comparison confirms developer migration omission in an ablated patch modifying 11 files, coupled with complete migration coverage absence in analyzed test assertions. Within this verified static inspection scope, assertions evaluate localized leaf permissions while leaving the database migration unasserted. Axis 2 (Element-Web #0139: Mocked Logic vs. Inferred Cache Persistence). Mandate (Rspec ): The issue mandates recomputing unread badges when receipts arrive across thread timelines, implemented in src/Unread.ts (Explicit Requirement and Developer Obligation). Persistent IndexedDB cache synchronization across network streams is an Inferred Requirement. Oracle (Reval ): Test code in test/Unread-test.ts asserts in-memory helper returns under synthetic mocked events (mocked(haveRendererForEvent)). Structural Gap: While in-memory state transitions for the Explicit Requirement are validated, persistent IndexedDB synchronization and asynchronous event reordering remain unasserted, reflecting the boundary between unit mock verification and inferred persistent client state. Axis 3 (RocketMQ #0819: JVM Contention vs. Transaction Boundaries). Mandate 9

Table 6: Comparative Case Analysis: Test Oracle Construct Coverage Across Four Qualitative Benchmark Axes (Unabridged matrix in replication package). Axis Task

/

Specification (Rspec )

Mandate

Evaluation (Reval )

Test

Oracle

Observed Scope

Construct Alignment or Gap Analysis

Axis 1 (Direct) NodeBB #0030

Explicit (A5 → B6 ): DB migration script (chat_allow_list.js, 44 LOC) migrating legacy restrictChat records.

Leaf Unit Assertion: Asserts canMessageUser() on mock objects in memory (test/messaging.js, L103– 121).

Inmemory leaf

Construct Underrepresentation: Direct test patch asserts leaf permissions; DB migration script unasserted in analyzed test patch.

Axis 2 (Scope) ElementWeb #0139

Explicit badge update; inferred cache persistence: Recalculate unread badges; IndexedDB cache synchronization is inferred rather than explicitly mandated.

Mocked State: Asserts helper return values under synthetic event mocks (test/Unread-test.ts).

Inmemory mock

Mocked Logic vs. Persistent Cache: In-memory transitions verified; asynchronous IndexedDB cache persistence outside evaluation scope.

Axis 3 (Hierarchy) RocketMQ #0819

Explicit (B7 , B2 ): Thread-safe counter insertion (StatsItemSet, 20kL) under JVM thread contention.

Thread Stress Test: 100 worker threads executing 10,000 tasks (StatsItemSetTest.java).

JVM thread contention

Single JVM Concurrency Met: Checks concurrent counter accumulation within a single JVM; distributed transaction boundaries are not mandated by the task.

Axis 4 (Aligned) Navidrome #0495

Explicit (A4 , B6 , B10 ): Online SQLite DB backup, destructive DDL recovery, and retention pruning.

Integrated Engine Test: Executes backup, runs destructive DDL, restores DB, asserts pruning (db/backup_test.go).

Persistent DB state

Aligned Invariant Testing: Positive case in which the inspected tests directly address the explicitly required persistentstate operations.

(Rspec ): The task mandates thread-safe counter accumulation under concurrent thread contention in StatsItemSet (Explicit Requirement), resolved using putIfAbsent (Developer Obligation); distributed transactions were neither requested nor implied. Oracle (Reval ): Test code in StatsItemSetTest.java spawns 100 worker threads executing 10,000 tasks, asserting atomic accumulation of 20,000L. Structural Alignment: The inspected tests exercise concurrent accumulation within a single JVM, in line with the Explicit Requirement. Axis 4 (Navidrome #0495: Aligned Invariant Testing Counterexample). Mandate (Rspec ): The task explicitly mandates online SQLite database backup, destructive DDL recovery, and backup retention pruning (Explicit Requirement), implemented across cmd/backup.go and db/backup.go (Developer Obligation). Oracle (Reval ): Test code in db/backup_test.go creates real SQLite backups, executes destructive DDL (DELETE FROM sqlite_master), verifies schema restoration, and asserts retention pruning policies. Structural Alignment: The inspected tests provide direct evidence of alignment with the explicitly specified SQLite backup, schema restoration, and retention requirements.

6

Threats to Validity

6.1

Internal & Construct Validity

A primary threat to internal and construct validity is potential subjective bias in applying the 18-criteria taxonomy. Semantic coding across 944 AI benchmark tasks was conducted by a single investigator against frozen codebook definitions rather than a multi-researcher adjudication committee. While all 3,446 criteria assignments were validated through automated scripts (validate_*.py), programmatic offset checking guarantees textual quotation fidelity and source hash integrity rather than multi-coder consensus; we make no claim of dual-coder agreement statistics. We mitigated potential subjective drift through four procedural controls: (1) freezing the 18-criteria operational definitions and boundary rules prior to corpus-wide annotation (framework-snapshot.json), anchoring criteria in domain-neutral standards (ISO/IEC/IEEE 14764, ISO/IEC 25010); (2) requiring an explicit verbatim textual quote from the task specification for every criterion assignment; (3) providing all evidence quotes, code diff references, and validation scripts in our replication package for independent audit; and (4) verifying all 16 Fineract dependency chains via automated byte-level offset matching against raw Jira JSON archives (gate2_raw_source_snippets.json). Additionally, to address potential matched-pair clustering in E1 10

across identical repositories, we confirmed that semantic heterogeneity persists across cross-repository pairs regardless of repository origin.

6.2

External Validity & Scope Boundaries

Evaluating 944 benchmark tasks across three established benchmarks spanning multiple languages (Python, Java, TypeScript, Go), we explicitly delimit our empirical scope: (1) Empirical Scope vs. Universal Generalization: Findings reflect quantitative replicability and static verification across the analyzed corpora, rather than a universal claim that all AI benchmarks lack causal dependencies; well-designed harnesses can evaluate persistent state operations (Navidrome #0495); (2) Static Oracle Analysis vs. Dynamic Solve Rates: We examine test oracle construct representation via diffs and assertions, without claiming dynamic execution pass rates for unexecuted models; (3) Domain Confounding: Fineract is a banking system with temporally dependent ledger requirements, while the benchmark corpus spans different projects. Using a shared ISO-informed taxonomy makes labels comparable but does not control for domain composition or demonstrate benchmark selection bias. This also applies to the conditional and exploratory higher-order co-specification rates, whose sparse benchmark cells limit precision; and (4) Dependency and Case Selection: The 16 source-grounded relationships and four oracle case studies illustrate possible structures, not their prevalence in the populations. We do not establish that enterprise maintenance universally requires more dependencies, measure production failure rates, or assess a complete execution harness from an inspected test patch.

7

Discussion & Benchmark Design Implications

7.1

Re-evaluating the Benchmark Complexity Paradigm

Our findings demonstrate that software maintenance complexity cannot be adequately characterized solely by surface metadata such as lines of code churn or file navigation graphs. In E1, 1,177 pairs matched on native issue classification and modified-file count, with code churn within the prespecified tolerance, exhibited mean fulfillment-set Jaccard distance 0.8138; 57.4% (676 pairs) had completely disjoint condition sets (dJ = 1.0000). This indicates that matching on native issue classification, exact modified-file counts, and code churn within the specified tolerance does not ensure semantic similarity in fulfillment conditions. Furthermore, SWE-bench Pro has higher mean attribute breadth (|B| = 2.47) than Apache Fineract (1.78) in the studied corpora. Thus the number of tagged fulfillment conditions alone cannot characterize maintenance requirement structure; their co-specification patterns and source-grounded dependencies must also be considered. The Fineract examples show how a backdated operation can impose linked recalculation and preservation obligations. The co-occurrence contrast and the additional conditional rates (P (B2 | B4 ): 48.85% in Fineract versus 5.81% in the pooled AI corpus) are consistent with different patterns of jointly specified conditions. They do not measure causal dependency strength, and domain composition prevents attributing this contrast to benchmark design alone. Selected oracle inspections illustrate implications for checking explicit obligations rather than forming the principal empirical claim.

7.2

Implications for Describing and Evaluating Maintenance Tasks

The empirical results motivate three reporting and design considerations, rather than a claim that any one test architecture is universally superior: 1. Report fulfillment-condition profiles alongside surface size: Task descriptions and benchmark reports should distinguish native issue labels, file counts, and churn from the types and breadth (|B|) of required completion conditions. In E1, superficially matched tasks frequently have different condition sets. 2. Represent linked conditions when the specification warrants them: Beyond listing attributes, descriptions should preserve source-grounded dependencies, such as a backdated change that triggers

11

financial recalculation while retaining historical consistency. The observed ϕ differences motivate closer analysis but do not themselves prove directional dependencies or broader causal mechanisms. 3. Align checks with explicitly required multi-condition completion: If a task specifies a migration, transaction boundary, or state-preservation obligation, evaluation design should consider checking that obligation jointly with the local functional change. NodeBB and Navidrome illustrate contrasting scopes in inspected patches. This is a downstream implication of requirement analysis, not evidence that benchmark oracles generally fail. When possible, benchmark reports should make these verification boundaries transparent.

8

Conclusion

Using 944 analyzed coding-benchmark tasks and 2,925 Apache Fineract issues, we examined how maintenance completion conditions differ in breadth, co-occurrence, and source-grounded directionality. RQ1 (Semantic Heterogeneity and Breadth): Among 1,177 pairs matched on issue type, modifiedfile count, and churn tolerance, mean fulfillment-set Jaccard distance was 0.8138, with 57.4% disjoint sets. SWE-bench Pro also had greater mean attribute breadth (|B| = 2.47) than Fineract (1.78): an explanation based on fewer benchmark attributes is therefore insufficient for these corpora. RQ2a (Co-occurrence): Temporal and numerical conditions exhibited positive association in Fineract (B4 /B2 , ϕ = +0.430, 254 issues) but near-zero association in the pooled benchmark corpus (ϕ = −0.013, p = 0.825, 5 tasks). Among items specifying B4 , B2 also appears in 254/520 Fineract issues (48.85%) versus 5/86 benchmark tasks (5.81%). These domain-confounded co-specification contrasts do not establish directional dependency rates or prove missing dependencies across AI benchmarks. RQ2b (Directional Fulfillment): Sixteen source-grounded Fineract relationships illustrate how maintenance completion may depend on multiple linked obligations, including backdating, recalculation, and preservation constraints. Four selected benchmark cases show that the inspected evaluation assertions address explicit obligations with varying scope; these cases do not establish a general oracle-failure rate. Together, our results establish that matching surface task characteristics does not ensure similar fulfillment conditions and that condition counts alone do not characterize observed joint-specification patterns. Maintenance benchmarks should therefore distinguish condition breadth, statistical association, and source-grounded directional obligations. Test-assertion alignment remains a downstream application rather than the principal empirical claim.

Data Availability Analysis datasets, selected source-grounded evidence, and targeted verification scripts are publicly available in the replication package: public GitHub repository. Some upstream source materials and complete benchmark execution environments are not redistributed.

References [1] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations (ICLR), 2024. [2] Xiang Deng, Jeff Da, Edwin Pan, et al. SWE-Bench Pro: Can ai agents solve long-horizon software engineering tasks? arXiv preprint arXiv:2509.16941, 2025. URL https://arxiv.org/abs/2509. 16941. [3] Muhammad Shihab Rashid, Christian Bock, Yuan Zhuang, Alexander Buchholz, Tim Esler, Simon Valentin, Luca Franceschi, Martin Wistuba, Prabhu Teja Sivaprasad, Woo Jung Kim, Anoop Deoras, Giovanni Zappella, and Laurent Callot. SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents. arXiv preprint arXiv:2504.08703, 2025. URL https://arxiv.org/abs/2504.08703. 12

[4] Minh V. T. Thai, Tue Le, Dung Nguyen Manh, Huy Phan Nhat, and Nghi D. Q. Bui. SWEEVO: Benchmarking coding agents in long-horizon software evolution scenarios. arXiv preprint arXiv:2512.18470, 2025. [5] Radin Shayanfar, Keheliya Gallaba, and Ahmed E. Hassan. What does an agentic software engineering benchmark measure? profiling task demands and agent behaviour beyond what category labels reveal. arXiv preprint arXiv:2609.01271, 2026. URL https://arxiv.org/abs/ 2609.01271. [6] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. [7] Boxi Yu, Yuxuan Zhu, Pinjia He, and Daniel Kang. UTBoost: Rigorous evaluation of coding agents on SWE-bench. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3762–3774, 2025. doi: 10.18653/v1/2025.acl-long.189. [8] The Apache Software Foundation. Apache Fineract: An open source system for core banking solutions. https://fineract.apache.org, 2024. [9] ISO/IEC/IEEE. ISO/IEC/IEEE 14764:2022: Software engineering — software life cycle processes — maintenance. International Organization for Standardization, 2022. [10] ISO/IEC. ISO/IEC 25010:2023: Systems and software engineering — systems and software quality requirements and evaluation (SQuaRE) — product quality model. International Organization for Standardization, 2023.

13

Related documents

• Machine Learning-Based Predictive Maintenance System for CNC Machines: A Deep Learning Approach to Fault Detection.#489507
• Hybrid-augmented intelligence in predictive maintenance with digital intelligent assistants#421596
• AI-accelerated End-to-End Framework for Rapid Professional Upskilling#370361
• Rebuild Dossier: Mechanically-Enforced Specs for Agentic App Rebuilds, and What Model-Tier Failures Reveal#1048793
• Hive-AI: a defended multi-service honeypot framework for generative AI APIs#680872