Mining Architectural Quality Under Agentic AI Adoption: A Causal Study of Java Repositories Oliver Aleksander Larsen1⋆ and Mahyar T. Moghaddam1
arXiv:2606.13298v1 [cs.SE] 11 Jun 2026
SDU Software Engineering, University of Southern Denmark, Odense, Denmark {olar,mtmo}@mmmi.sdu.dk
Abstract. AI coding tools are now used by a majority of developers, and agentic use of these tools has popularized the practice colloquially called “vibe coding”. Yet causal evidence on their effect on software architecture is scarce. Prior causal work has measured code-level outcomes (complexity, static analysis warnings); whether such degradation propagates to architecture-level outcomes remains unknown. We mine 151 open-source Java repositories, 74 with detectable agentic AI adoption (identified via configuration files and Co-Authored-By commit trailers) and 77 propensity-matched controls, across a 13-month per-repository window yielding 1,811 monthly Arcan snapshots. We estimate the causal effect of adoption on architectural smell density (ASD) with a staggered difference-in-differences design and the Borusyak imputation estimator, applying a causal design recently used for code-level metrics to the architecture level. Total smell counts are essentially unchanged (+1.1%, p = 0.82) while lines of code grow +12.8% (p = 0.003); the resulting 6.7% ASD decline (p = 0.004) is therefore a denominator effect rather than an architectural improvement. Per-type estimates and robustness checks (wild cluster bootstrap, Lee bounds, stale-observation sensitivity) corroborate the pattern; pre-trends are flat (Wald p = 0.90), consistent with parallel trends. Density-normalized outcomes can mislead when treatment affects system size: raw counts and explicit decomposition are required for causal mining studies of AI tool adoption. The complete replication package, including the curated 151-repository monthly panel, is publicly available.
Keywords: architectural smells · architectural erosion · AI-assisted development · vibe coding · mining software repositories · difference-in-differences
1
Introduction
AI coding tools have moved from inline autocomplete to fully agentic operation: given a natural-language intent, tools such as Cursor, GitHub Copilot, Claude Code, and Aider generate, edit, and refactor code across multiple files [16,5,4]. ⋆
Corresponding author.
This manuscript has been accepted for presentation at the 52nd Euromicro Conference on Software Engineering and Advanced Applications (SEAA) 2026, held in Krakow, Poland, 02 – 04 September 2026, and for publication in Springer Lecture Notes in Computer Science (LNCS) proceedings. This is the author’s accepted manuscript version. The final authenticated publication will be available via Springer.
2
O. A. Larsen and M. T. Moghaddam
The resulting paradigm, popularized as “vibe coding” by Karpathy [25], leaves observable artifacts in repositories: tool-specific configuration files, which encode project-specific directives consistent with agentic engagement [22], and Co-Authored-By commit trailers; neither signal arises from inline autocomplete. These artifacts make agentic adoption visible to repository mining at scale, even as practitioner surveys report concerns about long-term maintainability [39,14]. These concerns have empirical support at the code level. He et al. [20] studied 806 Cursor-adopting repositories using a staggered difference-in-differences (DiD) design and found that AI adoption increases static analysis warnings by 30% and code complexity by 41%. Yet code-level complexity and architectural quality are distinct: a method with high cyclomatic complexity does not necessarily create a cyclic dependency; a duplicated code block does not produce a hublike component. Esposito et al. [13] showed that 33.8% of static analysis warnings never co-occur with any architectural smell, confirming that code-level and architectural degradation are only partially overlapping concerns. Whether code-level degradation propagates to architecture-level degradation remains unanswered. The gap matters. Architectural smells are recurring structural anti-patterns in a system’s dependency graph [15] and are indicators of architectural erosion [19,21] that affects maintainability and evolvability [23,33]. Qualitative work suggests the risk is real [3], yet no study has measured architectural impact with a causal mining design. The code-level evidence above predicts that agentic AI adoption should increase architectural smell density (ASD), particularly structural anti-patterns plausibly arising from file-scoped generation. RQ1–RQ3 test that prediction at the architectural level; RQ4 decomposes any observed ASD effect into numerator and denominator channels. RQ1: Does the adoption of agentic AI coding tools affect architectural smell density (ASD) in open-source Java repositories? RQ2: Which categories of architectural smells show the largest treatment effect? RQ3: How does the effect on ASD evolve over time after agentic AI tool adoption? RQ4: Is the observed ASD effect driven by changes in architectural smell counts (numerator), code volume (denominator), or both? We mine 151 open-source Java repositories, 74 with observable agentic AI adoption (detected via configuration files and Co-Authored-By trailers) and 77 propensity-matched controls [31,36], yielding 1,811 monthly Arcan snapshots [6,32] over 13 months per repository. The causal effect on ASD is estimated with a staggered difference-in-differences design and the Borusyak imputation estimator [9], applied here at the architectural level (one abstraction higher than He et al. [20]). ASD declines by 6.7% (p = 0.004), the opposite-direction finding to that prediction. Decomposition reveals the mechanism: total smell counts are essentially unchanged (+1.1%, p = 0.82) while code volume grows substantially (+12.8%, p = 0.003); the decline is therefore a composition effect driven by the denominator. Pre-trends are flat (Wald p = 0.90), consistent with parallel trends.
Agentic AI and Architectural Quality
3
This paper contributes: (1) a causal mining pipeline (151 repositories, 1,811 monthly snapshots, Borusyak DiD on architectural metrics) reusable for other AI-tool quality outcomes; (2) a methodological warning that density-normalized outcomes mislead when treatment affects system size, requiring explicit numerator/denominator decomposition; (3) the first causal evidence on architecturelevel effects of agentic AI adoption, consistent with divergence between codeand architecture-level effects [20,13]; and (4) a complete open-source replication package with the curated 151-repo monthly panel.
2
Background and Related Work
2.1
AI Coding Tools and the Detection Problem in Repository Mining
AI tools are now used by 63.2% of professional developers [35]. The major tools (GitHub Copilot, Cursor, Claude Code) have each evolved beyond inline autocomplete to offer agentic capabilities: given a natural-language intent, the tool autonomously generates, edits, and refactors code across multiple files [16,5,4]. Practitioner accounts describe developers accepting AI-generated code with minimal structural review [25,14,39]. The key distinction for repository mining is between usage modes, not between tools. In autocomplete mode, the developer accepts inline suggestions; no project-level configuration is needed and no attribution metadata is produced. In agentic mode, developers create tool-specific configuration files (which encode project-specific directives [22]) and produce Co-Authored-By commit trailers; neither artifact arises from autocomplete-only usage. Any detection strategy based on repository artifacts (including ours and He et al.’s [20]) therefore observes only agentic usage (Sec. 3.2). This is also the mode posing the greatest architectural risk: agentic tools operate at file and function scope, while architecture is an emergent property of inter-module relationships [8]. 2.2
Architectural Erosion and Architectural Smells
Perry and Wolf [29] formalized software architecture as the set of design decisions governing a system’s structural organization. Architectural erosion is the gradual divergence between intended and implemented architecture [19,21]. Architectural smells operationalize erosion as detectable structural anti-patterns in dependency graphs [15]. We measure four types, each violating a known design principle [28,27]: – Cyclic Dependency (CD): circular dependency chains among packages, violating the Acyclic Dependencies Principle. – Unstable Dependency (UD): a package depending on a less-stable package, violating the Stable Dependencies Principle. – Hub-Like Dependency (HL): a component with both high fan-in and high fan-out, undermining modularity.
4
O. A. Larsen and M. T. Moghaddam
– God Component (GC): an oversized package concentrating too many responsibilities, violating modularity. We detect these smells with Arcan [6], with reported precision in the 70–100% range across Java projects [6,32]. Architectural smell density (ASD = total smells / KLOC) is used as a proxy for the rate of erosion; erosion proper requires comparison against an intended architecture [19], which is unavailable for open-source projects at scale. ASD requires decomposition when treatment affects code volume (Sec. 4.5). 2.3
Empirical Studies of AI-Assisted Development
He et al. [20] conducted the closest predecessor study: a staggered DiD analysis of 806 Cursor-adopting repositories, reporting +30% static-analysis warnings and +41% complexity alongside higher development velocity. We follow their mining design one abstraction level higher: from SonarQube’s intra-method metrics to Arcan’s inter-package metrics. Esposito et al. [13] confirmed the gap between these two levels: 33.8% of static analysis warnings are architecturally benign, unassociated with any smell, indicating that code-level findings cannot be assumed to translate to architecture-level effects. Beyond causal studies, Cotroneo et al. [11] found AI-generated code more prone to vulnerabilities, while Yetiştiren et al. [41] report mixed quality results across major AI tools; both are cross-sectional and measure code-level properties exclusively. Amasanti and Jahić [3] survey practitioners about architecturalquality concerns under AI assistance, finding mixed signals; Waseem et al. [39] document related qualitative concerns. Jiang and Nam [22] characterized Cursor configuration files across 401 repositories, finding project-specific directives consistent with agentic engagement. A companion study by Agarwal et al. [1] extends He et al.’s analysis by distinguishing IDE-based AI assistants from autonomous coding agents, finding larger quality effects for agentic usage. These studies establish a code-level signal but do not address the architectural channel; the present work fills that gap. 2.4
Architectural Smell Evolution
Longitudinal studies of smell evolution provide the baseline against which AIinduced changes must be assessed. Sas et al. [33] found that smells persist, compound, and resist removal: erosion is the natural trajectory of organic development. Gnoyke et al. [17] confirmed these patterns across 485 releases of 14 opensource systems. Our DiD design controls for this natural trajectory by measuring the differential change in treated repositories relative to controls. Related work links smells to reduced maintainability [23], validates Arcan against LLM-based detection [38], and identifies conformance checking as underexplored [34]. Summary. No prior work applies a causal mining design to measure the architectural impact of AI-assisted development. The closest causal precedent (He et al.) operates on code-level metrics; the closest architectural-level work (Amasanti and Jahić) is qualitative.
Agentic AI and Architectural Quality Mining 204 treatment 580 control Phase 1
window filter
Matching
monthly snapshots
Extraction
Arcan (80.8%)
5
Analysis
90 + 90 repos
175 repos 2,241 snapshots
1,811 obs 151 repos
Phase 2
Phase 3
Phase 4
Fig. 1. Study pipeline with sample sizes at each stage.
3
Mining Design and Statistical Model
3.1
Study Design
We employ a staggered difference-in-differences (DiD) design to estimate the causal effect of observable agentic AI adoption on architectural quality. Figure 1 summarizes the four phases (mining, matching, extraction, analysis) whose sample sizes we trace below. Treatment is the first detectable adoption of an agentic AI coding tool (Sec. 3.2); controls are matched non-adopters observed over the same calendar periods. Each repository contributes a 13month window (six pre-adoption, adoption month, six post-adoption). The unit of analysis is the repository-month; the primary outcome is log(ASD + 1) with ASD = total smells/KLOC (thousand lines of code); effects are reported as 100 × (exp(β̂) − 1)% (approximate under log(x + 1); see Sec. 4.5 for the strict-log specification on positive observations). Identification relies on repository and time fixed effects under parallel trends. Repository sampling follows the methodological caveats of Kalliamvakou et al. [24] (forks and inactive repositories filtered). 3.2
Treatment Detection and Sample Construction
Treatment identification. AI tool adoption is identified through two categories of repository artifacts. First, tool-specific configuration files per tool: Cursor (.cursorrules, .cursor/rules/), Copilot (.github/copilot-instructions.md), Claude Code (CLAUDE.md or claude.md), Aider (.aider.conf.yml, .aider.model.settings.yml), and Codex (AGENTS.md). Second, Co-Authored-By commit trailers matching known AI tool patterns. Detection patterns. Each configuration marker is matched as a literal pathspec via git log --all --diff-filter=A; all markers are file pathspecs except .cursor/rules/, which is a directory pathspec (any addition under it counts). Commit-message signals are matched via git log --all -E --grep against four POSIX-ERE patterns: three case-insensitive Co-Authored-By: trailers naming Claude, aider, or Copilot, plus the literal subject-line tag (aider) that the Aider CLI appends to commits it authors. The adoption date is the earliest event across both signal types. Ambiguous markers (CLAUDE.md/claude.md, AGENTS.md) are content-validated, while high-reliability markers (.cursorrules, copilot-instructions.md, .aider.*) are accepted directly. Full patterns and code are in the replication package.
6
O. A. Larsen and M. T. Moghaddam
Both categories indicate configured, agentic usage rather than inline autocomplete, which produces no detectable artifacts (Sec. 2.1). The construct measured is observable agentic AI adoption, distinct from the broader behavioral pattern of accepting AI output without review. Undetected autocomplete in controls induces attenuation bias, making estimates conservative. Sensitivity to autocomplete contamination. If a fraction c of controls use undetected AI tools, attenuation bias yields β̂true ≈ β̂obs /(1 − c) [2]; the corrected effect at the empirical c = 8.9% is −7.4%, amplifying rather than altering the neutrality conclusion. Tool composition (matched / post-Arcan retained): Claude Code 52/44 (one of which also uses Aider), Copilot 21/14, Cursor 16/15, Codex 1/1. Single-event sensitivity is in Sec. 4.7; tool-stratified analysis in Sec. 4.6. We validate the proxy empirically via temporal alignment: LOC growth emerges at adoption dates (Sec. 4.5). Inclusion criteria. Treatment candidates satisfy: (1) Java as primary language; (2) GitHub repository size ≥2,000 KB at mining time as a code-volume proxy (LOC and package counts are post-hoc Arcan outputs); (3) ≥2 commits per month on average; (4) ≥6 months pre-adoption history; and (5) ≥3 months post-adoption history. These criteria yield 90 treatment repositories from 204 candidates (Fig. 1). Control matching. The control pool comprises 580 Java repositories without detectable AI markers. We perform 1:1 nearest-neighbor matching [31,36] on Euclidean distance over six standardized covariates (four log-transformed to address skew: stars, forks, commits, size), with a propensity-score caliper of 0.2 × SD(logit(p̂)) [30,7]. Matching yields 90 pairs (the 90+90 in Fig. 1). Postmatching, four of six covariates satisfy |SMD| < 0.1 (standardized mean difference; below 0.1 is well balanced); repository size (SMD = −0.18) and commit count (SMD = −0.12) exceed it, addressed via covariate adjustment in Sec. 4.7. 3.3
Architectural Smell Extraction
Monthly snapshots and Arcan analysis. For each repository-month we extract the last commit before the first of the month (git rev-list --first-parent --before) and materialize it as a temporary working copy. Of the 180 matched repositories (90 treatment, 90 control), 5 controls are dropped before analysis for having no resolvable commit in their study window (insufficient git history), leaving 175 repositories (90 treatment, 85 control) that yield 2,241 monthly snapshots. Each snapshot is analyzed with Arcan 2 CLI [6] (trial edition, pinned Docker image) detecting the four smell types from Sec. 2.2; analysis succeeded for 1,811 of 2,241 snapshots (80.8%, failures from JVM memory limits or incompatible structures). A further 24 repositories with zero successful snapshots (16 treatment, 8 control) are excluded, yielding the final panel of 1,811 observations across 151 repositories (74 treatment, 77 control). Total attrition is thus 29 of 180 matched repositories (16.1%; Sec. 6).
Agentic AI and Architectural Quality
7
Metrics. Primary outcome: ASD = total smells/KLOC, aggregated across class and package granularity (a container-only sensitivity is in Sec. 4.7). Cyclic dependencies dominate (mean CD = 333 vs. <10 for other types), so we complement aggregate ASD with per-type analysis (RQ2), balanced-ASD, and total-count decomposition. Secondary metrics: efferent coupling (Ce), instability (I = Ce/(Ca + Ce)), and distance from the main sequence. All outcomes logtransformed as log(x + 1). The primary specification uses fixed effects only; the secondary specification, two-way fixed effects with the Sun and Abraham [37] interaction-weighted correction (TWFE-SA), additionally adjusts for monthly contributors. 3.4
Statistical Model
Because repositories adopt at different calendar months (staggered adoption), standard two-way fixed effects estimation can produce biased estimates under heterogeneous treatment effects [18,12]. We therefore employ estimators designed for staggered settings, and adopt Borusyak imputation as primary because, unlike standard TWFE, it never uses already-treated repositories as controls and so stays unbiased under heterogeneous treatment effects. Primary estimator: Borusyak imputation. Following Borusyak et al. [9]: log(ASDit + 1) = αi + γt + β · Dit + εit
(1)
where i indexes repositories, t months, αi and γt are repository and time fixed effects, and Dit = 1 once repository i has adopted. The imputation procedure fits on untreated observations (controls plus pre-adoption periods of treated), predicts counterfactual Ŷit (0) for treated post-adoption observations, and estid = Yit − Ŷit (0). mates the average treatment effect on the treated (ATT) as ATT Standard errors are clustered at the repository level. Architectural metrics exhibit high inertia (intraclass correlation ICC = 0.979); fixed effects absorb between-repository variance, isolating within-repo treatment effects. GC nulls should be interpreted as potentially underpowered (77.5% zeroinflation). Secondary estimator and event study. The Sun and Abraham [37] interaction-weighted estimator provides an independent concordance check. Event-study coefficients are estimated for h ∈ {−6, . . . , +6} with h = −1 as reference; a Wald test on h = −6 through h = −2 jointly zero is the parallel-trends falsification check. Post-adoption coefficients trace the trajectory (RQ3). Denominator decomposition (RQ4). Since ASD is a ratio (smells/KLOC), a treatment effect can reflect changes in the numerator, the denominator, or both. We run separate DiD models on log(total smells + 1) and log(KLOC + 1) to identify the mechanism. Robustness checks. Pre-registered: a placebo test at a fake adoption date and an alternative matching specification (subsumed by Mahalanobis with regression adjustment). Added post-results: wild cluster bootstrap [10,40] (few-cluster
8
O. A. Larsen and M. T. Moghaddam
Table 1. Main diff-in-diff estimates: effect of AI tool adoption on architectural smell density (log ASD). Borusyak
TWFE-SA
ATT (β̂) -0.0698*** (0.0239) Effect (%) -6.7%
-0.0681*** (0.0238) -6.6%
∗
p < 0.1; ∗∗ p < 0.05; ∗∗∗ p < 0.01 Both: 1,811 obs., 151 repos, repo and time FE, SEs clustered by repo.
robustness); Lee [26] bounds (worst-case attrition); smells per package; stale observations; Bacon decomposition (staggered-DiD weighting); size stratification; container-only. Post-hoc: tool-type stratification; dose-response.
4
Results
4.1
Sample and Descriptive Statistics
The final panel comprises 1,811 observations across 151 repositories (74 treatment, 77 control), after excluding 29 of 180 matched repositories (16.1%; Sec. 6). Groups are comparable in size (mean KLOC: 44.9 vs. 45.2), maturity (73.8 vs. 80.1 months), and stars (918 vs. 920). ASD is right-skewed (median = 3.0, max = 58.8), motivating the log transformation. Treatment repos exhibit higher mean ASD (6.1 vs. 5.1); with ICC = 0.979, repository fixed effects absorb this variation. 4.2
RQ1: Effect of AI Adoption on Architectural Smell Density
Table 1 reports the main DiD estimates. The Borusyak imputation estimator yields β̂ = −0.070 (SE = 0.024, 95% CI [−0.117, −0.023], p = 0.004), corresponding to 6.7% lower ASD in repositories with observable agentic AI adoption. Sun and Abraham confirms concordance (−6.6%, p = 0.005). Against the theoretical expectation of architectural degradation (Sec. 1), the negative point estimate is the opposite-direction finding; the mechanism behind the headline number is the focus of Sec. 4.5. 4.3
RQ2: Disaggregated Effects by Smell Type
Figure 2 presents per-type treatment effects (Borusyak imputation). Because we test four smell types, we report Holm-adjusted p-values, which raise the bar so multiple tests do not yield a false positive. Hub-like dependency density decreases by 5.0% (p = 0.001; Holm-adjusted p = 0.003), the only type surviving multiplicity correction. Cyclic dependency shows a comparable estimate (−5.0%, p = 0.023) but is non-significant after Holm adjustment (p = 0.070). UD and GC show no significant density change (p = 0.205, p = 0.877). The GC null likely reflects low power: 77.5% of repo-months record zero god components.
Agentic AI and Architectural Quality
9
Treatment Effect by Architectural Smell Type
Unstable Dependency
God Component
Cyclic Dependency
Hub−Like Dependency
−7.5
−5.0
−2.5
0.0
Treatment Effect on Smell Density (%) Not significant
Significant (Holm)
Fig. 2. Per-type treatment effects (Borusyak imputation) with 95% CIs. Filled markers: significant after Holm correction.
Raw UD counts show a suggestive increase of 5.8% (raw p = 0.032; Holmadjusted across four raw-count tests, p = 0.128). The density null masks this marginal signal, discussed in Sec. 5.1. Density and count tests answer distinct mechanistic questions, with family-wise error controlled within each family. 4.4
RQ3: Dynamic Treatment Effects
Figure 3 presents the event study. Pre-period coefficients (h = −6 to h = −2) cluster near zero, and the Wald test cannot reject joint nullity (χ2 = 1.64, df = 5, p = 0.896), consistent with the parallel trends assumption (here a high p-value is reassuring: no divergence before adoption). Post-adoption, the effect appears immediately at h = 1 (−5.5%) and grows to −9.5% by h = 6, consistent with cumulative code growth diluting smell density (Sec. 4.5). The effect is still expanding at h = 6, so longer post-adoption windows are needed to see whether smell counts eventually grow in step with LOC and the density gap closes. 4.5
RQ4: Decomposing the ASD Effect
The 6.7% ASD reduction could reflect fewer smells (numerator) or faster code growth (denominator). Separate DiD models (Fig. 4) reveal the mechanism. Raw smell counts are unchanged (+1.1%, p = 0.82) while code volume grows significantly faster in adopters than controls (+12.8%, p = 0.003). The ASD decrease is a composition effect: the marginal code carries lower smell density than the pre-existing codebase. A strict-log specification on positive observations (which preserves the ratio identity that log(x + 1) approximates) yields β̂ = −0.139 (−13.0%, p < 0.001), implying an even larger denominator share than under log(x + 1).
10
O. A. Larsen and M. T. Moghaddam Event Study: Architectural Smell Density Borusyak (solid blue) vs. Sun & Abraham (dashed grey) Adoption
Effect on ASD (%)
5
0
−5
−10
−15 −6
−5
−4
−3
−2
−1
0
1
2
3
4
5
6
Months Relative to AI Tool Adoption
Fig. 3. Event study: ASD effect by relative month. Pre-period coefficients near zero (Wald p = 0.896); post-adoption effect grows to ∼−9.5% at h = 6. Sun & Abraham overlay (grey dashed) confirms concordance.
Per-type raw counts confirm the pattern: CD, HL, and GC show no significant absolute changes (all p > 0.25). UD shows a suggestive increase (+5.8%, raw p = 0.032; Holm-adjusted p = 0.128), the strongest per-type signal though non-significant after multiplicity correction. Coupling metrics (Ce, I, D) are similarly stable (all p > 0.05). Package count also grows (+9.7%, p = 0.032), yet smell counts remain unchanged despite expanding modular structure. The methodological implication for empirical mining is discussed in Sec. 5.2. 4.6
Exploratory: Treatment Heterogeneity by Tool
Tool-stratified Borusyak estimates are consistently negative (Codex, nT = 1, omitted): Claude Code (nT = 44, −5.1%, p = 0.039), Copilot (nT = 14, −4.0%, p = 0.005), and Cursor (nT = 15, −13.5%, p < 0.001). Magnitudes differ (Cursor strongest), but with a non-significant dose-response interaction (p = 0.716) and post-hoc stratification, the larger Cursor effect is a hypothesis for replication, not a finding. 4.7
Robustness Checks
Table 2 reports supplementary analyses. Covariate adjustment yields −6.6% (p = 0.004). The Bacon decomposition reports 89.1% of the TWFE weight from clean comparisons (80.2% treated-vs-never-treated, 8.9% treated-vs-notyet-treated) and 10.8% from the potentially-biased later-vs-already-treated comparison, which the Borusyak primary estimator avoids by construction; a placebo at a fake adoption date (t − 3) is null (p = 0.427). The wild cluster bootstrap [10,40] (small-sample-safe significance) yields −6.0% (p = 0.010); Lee [26] bounds (worst-case attrition) span [−9.9%, −2.3%] (both negative, though the upper-bound CI includes zero, p = 0.239). Size-stratified estimates are consistent
Agentic AI and Architectural Quality
11
(a) Raw Smell Counts
Effect (%)
10
0
−10 −6
−5
−4
−3
−2
−1
0
1
2
3
4
5
6
−2
−1
0
1
2
3
4
5
6
(b) Lines of Code (KLOC) 30
Effect (%)
20
10
0
−10 −6
−5
−4
−3
Months Relative to AI Tool Adoption
Fig. 4. Decomposition: separate DiD on raw smell counts (top) and LOC (bottom). Smells unchanged (+1.1%, p = 0.82); LOC grows significantly (+12.8%, p = 0.003).
(−6.3%/−7.2%); balanced ASD (−4.6%, p < 0.001) and a multi-event restriction excluding repositories with a single detection event (≥ 2 events, −6.8%, p = 0.020) retain significance, indicating the result is not driven by one-off configuration commits. Container-level (package) smells alone (an outcome restriction; −3.0%, p = 0.074) are directionally consistent but attenuated. Only aggressive stale filtering (N = 94, p = 0.223) attenuates to non-significance; the effect is directionally consistent (−3% to −8%, or −13% under strict-log) across all other specifications.
5
Discussion
5.1
Code Growth Without Proportional Architectural Degradation
Architectural smells are topological: cyclic dependencies require mutual package imports; hub-like dependencies require excessive fan-in and fan-out. Neither increases despite growth in code volume (+12.8%) and package count (+9.7%, p = 0.032), suggesting new modules integrate without creating structural violations. UD’s suggestive count increase (+5.8%, raw p = 0.032; Holm-adjusted p = 0.128) would, if confirmed by replication, be consistent with AI tools preferentially adding imports to less-stable packages [28]. It does not survive multiplicity correction. The limited warning-smell overlap (33.8% non-co-occurring [13]) offers a candidate, untested reconciliation with He et al. [20]: AI tools may write more complex code within modules while respecting between-module boundaries.
12
O. A. Larsen and M. T. Moghaddam
Table 2. Supplementary analyses: denominator decomposition, inference robustness, attrition bounds, and sample sensitivity. Analysis
ATT (β̂)
Denominator decomposition Raw smell count LOC growth (log KLOC) Smells per package
0.0108 0.0486 [-0.0845, 0.1061] 0.8246 0.1201*** 0.0401 [0.0415, 0.1986] 0.0027 -0.0318* 0.0171 [-0.0653, 0.0018] 0.0636
1.1% 12.8% -3.1%
Inference robustness Wild cluster bootstrap
-0.0615*** 0.0249 [-0.1098, -0.0129] 0.0097
-6.0%
Attrition bounds (Lee 2009) Upper bound (pessimistic) Lower bound (optimistic)
-0.0229 0.0195 [-0.0611, 0.0152] 0.2387 -0.1046*** 0.0192 [-0.1422, -0.0671] 0.0000
-2.3% -9.9%
Sample sensitivity Excl. stale (≥3 zero-mo, N=94) -0.0413 0.0339 [-0.1078, 0.0252] 0.2234 Excl. stale (≥6 zero-mo, N=128) -0.0556** 0.0251 [-0.1048, -0.0063] 0.0269 Packages ≥ 30 (N=109) -0.0590*** 0.0216 [-0.1013, -0.0167] 0.0063 Balanced ASD (equal type weight) -0.0469*** 0.0139 [-0.0742, -0.0196] 0.0007 Excl. outliers (KLOC ≥ 1) -0.0791*** 0.0188 [-0.1159, -0.0423] 0.0000
-4.0% -5.4% -5.7% -4.6% -7.6%
∗
SE
95% CI
p-value Effect (%)
p < 0.1; ∗∗ p < 0.05; ∗∗∗ p < 0.01. Primary ASD estimate: β̂=-0.070, p=0.004.
5.2
A Methodological Lesson for Density-Normalized Mining Outcomes
Without decomposition, the −6.7% result would invite the misleading headline “AI tools improve architecture.” The correct interpretation is that adopters grow code volume faster than matched controls without a proportional increase in architectural violations. This generalizes: any density-normalized metric (defects/KLOC, vulnerabilities/KLOC, churn/KLOC) where treatment plausibly affects KLOC requires explicit decomposition. He et al.’s [20] raw-count SonarQube metrics show warning growth (+30%) exceeding LOC growth (+28.6%), indicating per-line degradation at the code level. Smells are topological yet ASD normalizes by KLOC (volumetric): the smells-per-package alternative normalization (a denominator change: −3.1%, p = 0.064) shows a smaller, marginal effect consistent with the denominator explanation. 5.3
Implications
For practitioners. These findings do not support escalating architectural safeguards in response to agentic AI adoption in the short term, but they warrant three monitoring practices: (1) track raw smell counts alongside ASD, since density alone will look improving even when counts are stagnant; (2) treat 6-month windows as preliminary and re-examine at 12+ months given the trajectory at h = 6 shows no inflection; and (3) monitor coupling metrics (Ce, I) alongside smell counts as a complementary signal. For researchers. Density-normalized outcomes require numerator/denominator decomposition under any treatment plausibly affecting system size. The highest-
Agentic AI and Architectural Quality
13
value extensions are greenfield projects (where AI influences architecture from inception), longer observation windows, and dependency-graph topology diffs to identify AI-introduced edges.
6
Threats to Validity
Internal Validity. Differential attrition is the most serious concern. From the matched 90+90, 16/90 treatment (17.8%) and 13/90 control (14.4%) were lost (5 controls dropped pre-Arcan for insufficient git history; 24 repositories, 16 treatment and 8 control, lost during Arcan analysis to JVM memory limits, concentrated among the largest projects). Lee bounds bracket the effect at [−9.9%, −2.3%]: both bounds are negative, but the upper bound’s 95% CI includes zero (p = 0.239). Under worst-case attrition, the effect cannot be distinguished from null. Parallel trends are supported by the event study Wald test (p = 0.896), though this cannot exclude smooth pre-treatment effects absorbed by repository fixed effects. Estimator sensitivity: Borusyak and TWFESA agree closely (−6.7%/−6.6%). The Bacon decomposition (Sec. 4.7) confirms 89.1% of weight comes from clean comparisons, the bias Borusyak avoids by construction. Stale observations: 19.3% of consecutive month-pairs share identical commits; aggressive filtering (N = 94) attenuates the effect to non-significance, though moderate filtering (N = 128, p = 0.027) retains it. Interference: we assume no spillover between units (SUTVA): shared libraries or fork relationships could violate this assumption, though any spillover would attenuate estimates toward zero. Co-adoption and selection: the treatment may capture a modernization bundle. Repository fixed effects and matching mitigate time-invariant confounders, but unobserved time-varying changes could confound the estimate. External Validity. We focus on mature Java OSS systems to isolate architectural effects where structure is already established; findings apply to shortterm effects under agentic AI usage. The effect may differ in greenfield projects where AI influences architecture from inception, or under strict governance constraining AI-generated changes. Generalization is further limited by language (Java, an Arcan constraint). Survivorship is a further threat: repositories abandoned before sampling are invisible, potentially biasing toward neutrality. Construct Validity. AI adoption proxy: our treatment captures observable agentic AI usage through configuration artifacts and commit metadata. Treated repositories are a purposive subset, restricting external validity. While this excludes autocomplete-only usage (false negatives) and may include one-off experiments (false positives), prior work shows such artifacts encode project-specific directives [22]. We validate this proxy empirically (Sec. 3.2): temporal alignment and cross-signal consistency support its reliability. Excluding single-event repos retains significance (−6.8%, p = 0.020). Control contamination: 8/90 matched controls (8.9%) showed AI markers during their study windows. The corrected effect is −7.4% (Sec. 3.2), and the qualitative pattern (significant ASD decline, unchanged raw counts, large LOC growth) is unchanged. Arcan and architecture operationalization: Arcan precision is reported in the 70–100% range [6,32]. The
14
O. A. Larsen and M. T. Moghaddam
trial edition was used. Arcan detects smells at class and package level: our primary ASD aggregates both, with 57% of cyclic dependencies at class level. A container-only sensitivity (−3.0%, p = 0.074) is directionally consistent but nonsignificant. We operationalize architecture through dependency-graph structure, which does not capture design intent, conformance, or semantic violations. ASD normalization: the ratio metric conflates numerator and denominator changes. This is addressed via explicit decomposition (Sec. 4.5) and alternative normalization (smells per package, p = 0.064). Reproducibility: mitigated by detectionpattern disclosure (Sec. 3.2), pinned tool versions, fixed random seeds, and a complete replication package.
7
Conclusion
This study presented the first causal mining study of agentic AI adoption’s effect on software architecture, analyzing 1,811 monthly snapshots across 151 Java repositories with a staggered difference-in-differences design. The apparent 6.7% ASD reduction is a composition effect: repositories with observable agentic AI adoption exhibit increased code volume without proportionally more architectural smells. We find no evidence that the observed agentic AI tools degrade software architecture in established projects over a six-month window, though under worst-case attrition the confidence interval includes zero. Future work should extend the observation window and study greenfield projects. Dependency-graph topology diffs are needed to confirm the mechanism. The denominator-decomposition lesson generalizes to any density-normalized mining outcome under size-affecting treatment, and we encourage empirical AItool studies to adopt this practice by default.
Data Availability The replication package (pipeline scripts, processed monthly panel, configurations, fixed seeds, Docker setup) is archived on Zenodo (https:// doi.org/10.5281/zenodo.20510047) and mirrored at https://github.com/ Oliver1703dk/seaa2026-replication-package; all tables and figures reproduce via make analyze.
References 1. Agarwal, S., He, H., Vasilescu, B.: AI IDEs or autonomous agents? Measuring the impact of coding agents on software development (2026), accepted at MSR 2026 2. Aigner, D.J.: Regression with a binary independent variable subject to errors of observation. Journal of Econometrics 1(1), 49–59 (1973). https://doi.org/10. 1016/0304-4076(73)90005-5 3. Amasanti, G., Jahić, J.: The impact of AI-generated solutions on software architecture and productivity: Results from a survey study. In: Software Architecture: ECSA 2025 Tracks and Workshops. Lecture Notes in Computer Science, vol. 15982, pp. 89–104. Springer (2025). https://doi.org/10.1007/978-3-032-04403-7_10
Agentic AI and Architectural Quality
15
4. Anthropic: Claude Code: An agentic coding tool. Software tool (2025) 5. Anysphere: Cursor: The AI code editor. Software tool (2024) 6. Arcelli Fontana, F., Pigazzini, I., Roveda, R., Tamburri, D.A., Zanoni, M., Di Nitto, E.: Arcan: A Tool for Architectural Smells Detection. In: Proceedings of the IEEE International Conference on Software Architecture Workshops (ICSAW). pp. 282– 285 (2017). https://doi.org/10.1109/ICSAW.2017.16 7. Austin, P.C.: Optimal caliper widths for propensity-score matching when estimating differences in means and differences in proportions in observational studies. Pharmaceutical Statistics 10(2), 150–161 (2011). https://doi.org/10.1002/pst. 433 8. Bass, L., Clements, P., Kazman, R.: Software Architecture in Practice. AddisonWesley, 4th edn. (2021) 9. Borusyak, K., Jaravel, X., Spiess, J.: Revisiting event-study designs: Robust and efficient estimation. Review of Economic Studies 91(6), 3253–3285 (2024). https: //doi.org/10.1093/restud/rdae007 10. Cameron, A.C., Gelbach, J.B., Miller, D.L.: Bootstrap-based improvements for inference with clustered errors. The Review of Economics and Statistics 90(3), 414–427 (2008). https://doi.org/10.1162/rest.90.3.414 11. Cotroneo, D., Improta, C., Liguori, P.: Human-written vs. AI-generated code: A large-scale study of defects, vulnerabilities, and complexity. In: Proceedings of ISSRE. pp. 252–263 (2025). https://doi.org/10.1109/ISSRE66568.2025.00035 12. de Chaisemartin, C., d’Haultfœuille, X.: Two-way fixed effects estimators with heterogeneous treatment effects. American Economic Review 110(9), 2964–2996 (2020). https://doi.org/10.1257/aer.20181169 13. Esposito, M., Robredo, M., Arcelli Fontana, F., Lenarduzzi, V.: On the correlation between architectural smells and static analysis warnings. Software Quality Journal 33(4), 33 (2025). https://doi.org/10.1007/s11219-025-09730-7 14. Fawzy, A., Tahir, A., Blincoe, K.: Vibe coding in practice: Motivations, challenges, and a future outlook—a grey literature review. In: Proceedings of ICSE-SEIP (2026), arXiv:2510.00328, to appear 15. Garcia, J., Popescu, D., Edwards, G., Medvidovic, N.: Toward a catalogue of architectural bad smells. In: Proceedings of the International Conference on the Quality of Software Architectures (QoSA). pp. 146–162 (2009). https://doi.org/10. 1007/978-3-642-02351-4_10 16. GitHub: GitHub Copilot: Your AI pair programmer. Software tool (2025) 17. Gnoyke, P., Schulze, S., Krüger, J.: Evolution patterns of software-architecture smells: An empirical study of intra- and inter-version smells. Journal of Systems and Software 217, 112170 (2024). https://doi.org/10.1016/j.jss.2024.112170 18. Goodman-Bacon, A.: Difference-in-differences with variation in treatment timing. Journal of Econometrics 225(2), 254–277 (2021). https://doi.org/10.1016/j. jeconom.2021.03.014 19. van Gurp, J., Bosch, J.: Design erosion: Problems and causes. Journal of Systems and Software 61(2), 105–119 (2002) 20. He, H., Miller, C., Agarwal, S., Kästner, C., Vasilescu, B.: Speed at the cost of quality: How Cursor AI increases short-term velocity and long-term complexity in open-source projects (2025), accepted at MSR 2026 21. Hochstein, L., Lindvall, M.: Combating architectural degeneration: A survey. Information and Software Technology 47(10), 643–656 (2005) 22. Jiang, S., Nam, D.: Beyond the prompt: An empirical study of Cursor Rules. In: Proceedings of the 23rd International Conference on Mining Software Repositories (MSR) (2026), arXiv:2512.18925, to appear
16
O. A. Larsen and M. T. Moghaddam
23. Jolak, R., Karlsson, S., Dobslaw, F.: An empirical investigation of the impact of architectural smells on software maintainability. Journal of Systems and Software 225, 112382 (2025). https://doi.org/10.1016/j.jss.2025.112382 24. Kalliamvakou, E., Gousios, G., Blincoe, K., Singer, L., German, D.M., Damian, D.: An in-depth study of the promises and perils of mining GitHub. Empirical Software Engineering 21(5), 2035–2071 (2016). https://doi.org/10.1007/ s10664-015-9393-5 25. Karpathy, A.: Vibe coding. X (formerly Twitter) (February 2025), https://x. com/karpathy/status/1886192184808149383 26. Lee, D.S.: Training, wages, and sample selection: Estimating sharp bounds on treatment effects. Review of Economic Studies 76(3), 1071–1102 (2009). https: //doi.org/10.1111/j.1467-937X.2009.00536.x 27. Lippert, M., Roock, S.: Refactoring in Large Software Projects: Performing Complex Restructurings Successfully. John Wiley & Sons (2006) 28. Martin, R.C.: Agile Software Development: Principles, Patterns, and Practices. Prentice Hall (2003) 29. Perry, D.E., Wolf, A.L.: Foundations for the study of software architecture. ACM SIGSOFT Software Engineering Notes 17(4), 40–52 (1992) 30. Rosenbaum, P.R., Rubin, D.B.: Constructing a control group using multivariate matched sampling methods that incorporate the propensity score. The American Statistician 39(1), 33–38 (1985). https://doi.org/10.1080/00031305.1985. 10479383 31. Rubin, D.B.: Bias reduction using Mahalanobis-metric matching. Biometrics 36(2), 293–298 (1980). https://doi.org/10.2307/2529981 32. Sas, D., Avgeriou, P.: An architectural technical debt index based on machine learning and architectural smells. IEEE Transactions on Software Engineering 49(8), 4169–4195 (2023). https://doi.org/10.1109/TSE.2023.3286179 33. Sas, D., Avgeriou, P., Uyumaz, U.: On the evolution and impact of architectural smells—an industrial case study. Empirical Software Engineering 27, 86 (2022) 34. Schmid, L., Hey, T., Armbruster, M., Corallo, S., Fuchß, D., Keim, J., Liu, H., Koziolek, A.: Software architecture meets LLMs: A systematic literature review (2025) 35. Stack Overflow: Stack overflow developer survey 2024. Online survey (2024) 36. Stuart, E.A.: Matching methods for causal inference: A review and a look forward. Statistical Science 25(1), 1–21 (2010). https://doi.org/10.1214/09-STS313 37. Sun, L., Abraham, S.: Estimating dynamic treatment effects in event studies with heterogeneous treatment effects. Journal of Econometrics 225(2), 175–199 (2021). https://doi.org/10.1016/j.jeconom.2020.09.006 38. Tessa, C., Bochicchio, M., Arcelli Fontana, F.: Exploring architectural smells detection through LLMs. In: Proceedings of the 19th European Conference on Software Architecture (ECSA). Lecture Notes in Computer Science, vol. 15929, pp. 90–98. Springer (2025). https://doi.org/10.1007/978-3-032-02138-0_6 39. Waseem, M., Ahmad, A., Kemell, K.K., Rasku, J., Lahti, S., Mäkelä, K., Abrahamsson, P.: Vibe coding in practice: Flow, technical debt, and guidelines for sustainable use (2025) 40. Webb, M.D.: Reworking wild bootstrap-based inference for clustered errors. Canadian Journal of Economics 56(3), 839–858 (2023). https://doi.org/10.1111/ caje.12661 41. Yetiştiren, B., Özsoy, I., Ayerdem, M., Tüzün, E.: Evaluating the code quality of AI-assisted code generation tools: An empirical study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT (2023)