Beyond Simpson’s Paradox: A Cascade of Confounders in AI Agent Pull-Request Co-Authorship Haoran Yu
Independent Researcher Seattle, WA, USA [email protected]
arXiv:2606.22711v1 [cs.SE] 21 Jun 2026
Su Wang
Carnegie Mellon University Pittsburgh, PA, USA [email protected]
Xiaochong Jiang
Independent Researcher Seattle, WA, USA [email protected]
Pin Qian
Carnegie Mellon University Pittsburgh, PA, USA [email protected]
Abstract Pooled across five AI coding agents, pull requests (PRs) with a human Co-Authored-By trailer merge less often than purelyautonomous ones (53.8% vs. 79.8%)—yet this aggregate finding is a textbook Simpson’s Paradox. Stratifying 33,596 PRs from the AIDev dataset by agent identity reverses the conclusion: Copilot and Devin show large positive within-agent gaps (+41.2 and +33.5 pp, both 𝑝<0.001), while Cursor, Claude Code, and Codex show small effects whose cross-sectional 95% CIs span zero. The paradox is driven entirely by agent composition: Codex, which dominates 64.9% of the dataset, achieves high merge rates while rarely using co-authorship. But Simpson’s Paradox is only the first layer of a cascade of confounders: within-repo controls eliminate Devin’s gap (+33.5 → +1.6 pp, 𝑝 = 0.73); a commit-count control further halves Copilot’s within-repo gap (+36.2 → +24.4 pp); restricted to multi-commit PRs, the Copilot within-repo effect dissolves to +4.8 pp (𝑝 = 0.59). No agent retains a clear co-authorship effect once both repository selection and PR structure are controlled. Our findings caution against reporting agent-pooled statistics without stratification and demonstrate that cross-sectional co-authorship associations are largely selection and PR-structure artefacts rather than evidence of a causal benefit.
CCS Concepts • Software and its engineering → Software creation and management; • Computing methodologies → Machine learning.
Keywords AI coding agents, pull requests, Simpson’s paradox, confounding, co-authorship, human-AI collaboration, empirical software engineering
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. AgenticSE’26, Jeju, Korea © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM.
Lifei Liu
Independent Researcher Seattle, WA, USA [email protected]
Yihang Chen
Georgia Institute of Technology Atlanta, GA, USA [email protected]
ACM Reference Format: Haoran Yu, Xiaochong Jiang, Lifei Liu, Su Wang, Pin Qian, and Yihang Chen. 2026. Beyond Simpson’s Paradox: A Cascade of Confounders in AI Agent Pull-Request Co-Authorship. In Proceedings of KDD 2026 Workshop on Agentic Software Engineering (AgenticSE’26). ACM, New York, NY, USA, 5 pages.
1
Introduction
AI coding agents (GitHub Copilot, Devin, Cursor, Codex, and Claude Code), built on large language models trained on code [1], can now draft, commit, and open pull requests (PRs) with minimal human direction [3, 11]. They are part of a broader proliferation of largelanguage-model-based systems across software and data-intensive applications [2]. A central question for practitioners is: how often do these PRs actually get merged, and does human involvement matter? A first-pass answer is deceptively simple to compute. Using the Co-Authored-By trailer that GitHub and many agent workflows insert into commit messages [5], one can classify each PR as humanagent collaborative or purely autonomous. Pooling all 33,596 PRs from five agents, collaborative PRs merge at only 53.8% while autonomous PRs merge at 79.8%, a gap of −26.0 pp that appears to condemn human participation as counterproductive. We show this conclusion is entirely spurious. The aggregate signal is driven by agent composition, not by co-authorship itself: Codex, which accounts for 64.9% of all PRs and achieves high merge rates, rarely uses Co-Authored-By (1.2% of its PRs), while Copilot and Devin, whose autonomous PRs succeed at 6.2% and 21.6% respectively, rely on co-authorship for over 89% and 95% of their submissions. Stratifying by agent, four of five agents show a positive within-agent co-authorship effect, a classic reversal that constitutes Simpson’s Paradox [6, 7]. However, the within-agent reversal is itself the start of a longer story. A staged robustness analysis exposes a cascade of confounders: within-repo controls collapse Devin’s apparent gap to +1.6 pp, and a commit-count control further halves Copilot’s within-repo effect, which dissolves entirely (+4.8 pp, 𝑝=0.59) when restricted to multicommit PRs. No agent retains a clear within-stratum co-authorship effect once both repository selection and PR structure are controlled. Our data are purely descriptive: causality cannot be established from observational PR records. This paper addresses three research questions:
AgenticSE’26, August 2026, Jeju, Korea
RQ1. Does the presence of a Co-Authored-By trailer in agentsubmitted PRs correlate with merge rate, and does this relationship hold within individual agents? RQ2. How do author/committer collaboration modes (fully autonomous, agent-draft, human-both) relate to PR merge outcomes? RQ3. Does adopting a second agent in a repository change existing merge rates?
2
Background and Related Work
Pull-request dynamics. Gousios et al. [4] established that reviewer workload, PR size, and contributor reputation are strong predictors of merge outcomes. Yu et al. [12] showed that reviewer assignment strongly influences acceptance. We extend these lines of inquiry to the emerging setting of AI-generated PRs. Bots and automation in SE.. Wessel et al. [8, 10] found that adopting code review bots on open-source projects changes PR workflows and contributor behaviour; a follow-up study [9] confirmed downstream effects on PR acceptance rates. Our work complements this literature by studying generative AI agents, not merely notification/review bots. AI agents and code generation. Dakhel et al. [3] evaluated Copilot’s code quality; Xia et al. [11] introduced Agentless, a minimal agent baseline. Neither study focuses on PR-level merge outcomes or human collaboration signals, which is our contribution. Simpson’s Paradox. Simpson’s Paradox [7] arises when a confounding variable (here, agent identity) drives opposite associations at the aggregate and stratum levels. Pearl [6] formalises conditions under which the within-stratum analysis is causally preferred; we invoke this to argue that the per-agent results are the correct lens for our data.
3
Method
Dataset. We use the AIDev dataset [5], which contains 33,596 pull requests submitted by five AI coding agents: Codex (𝑛 = 21, 799, 64.9%), Copilot (𝑛 = 4, 970, 14.8%), Devin (𝑛 = 4, 827, 14.4%), Cursor (𝑛 = 1, 541, 4.6%), and Claude Code (𝑛 = 459, 1.4%). PRs span 2024-12 to 2025-07 (218 days), but agent observation windows differ: Devin appears earliest (2024-12), Codex latest (from 2025-05); within-agent and within-repo analyses absorb time-invariant confounding, but readers should bear this asymmetry in mind when interpreting pooled counts. Each PR is linked to its hosting repository and records the final PR state (merged, closed-unmerged, or open at collection time). Co-Authored-By mining. GitHub’s commit-message convention supports a Co-Authored-By: Name <email> trailer [5]. Agent workflows (e.g., Copilot Workspace, Devin) routinely inject this trailer to attribute human editors. We parse all commit messages in each PR and mark a PR as collaborative (coauth=1) if at least one commit contains a Co-Authored-By trailer; otherwise the PR is marked purely autonomous (coauth=0). In total, 10,764 PRs (32.0%) are collaborative and 22,832 (68.0%) are purely autonomous. Because the trailer is agent-inserted (most often by Copilot Workspace or
Yu et al.
Devin to attribute human editors), in practice these flagged coauthors are humans rather than bots; we discuss this assumption as a measurement limitation in Section 5. Author/Committer classification. Beyond the trailer, GitHub exposes a distinction between the author (who wrote the commit) and the committer (who applied it to the branch). We derive three collaboration modes: (i) human_both: human author and human committer; (ii) agent_draft: bot author, human committer (human applied the agent’s commit); and (iii) fully_autonomous: bot author and bot committer. A fourth logical combination (human author, bot committer) occurs in only 16 PRs and is excluded from the analysis due to insufficient sample size. Outcome. The binary outcome is merge rate: the proportion of PRs with state=merged at the time of dataset collection. We treat PRs that are still open at collection time (2,312 PRs, 6.9%) as “not merged” for conservatism, acknowledging this may undercount eventual merges for recent PRs. As a robustness check, restricting to state=closed PRs (excluding the open ones) preserves the qualitative findings: the pooled paradox direction holds (−21.5 pp) and Copilot’s pure-autonomous merge rate remains low (7.1% vs. the 6.2% headline figure). Identifying Simpson’s Paradox. A Simpson’s Paradox occurs when an aggregate association has the opposite sign to the withinstratum association for most or all strata. We operationalise this as: (1) pooled co-authorship effect < 0, and (2) majority of agents showing co-authorship effect > 0. We additionally verify that the effect reversal is explained by the distributional imbalance in agentlevel coauth rates (i.e. that high-merge agents are systematically low-coauth). Statistical tests. We use Pearson 𝜒 2 tests (with continuity correction) for proportion comparisons and report two-sided 𝑝-values. Effect sizes are reported as percentage-point differences (Δ pp). For the multi-agent analysis (RQ3) we use a difference-in-differences regression on weekly merge rates, controlling for repository fixed effects and a linear time trend. No causal model is assumed; all relationships are described as associations.
4 Results 4.1 RQ1: The Simpson’s Paradox in Co-Authorship Pooled analysis (misleading). Table 1 shows the aggregate result. Co-authored PRs merge at 53.8% while purely-autonomous PRs merge at 79.8%, a difference of −26.0 pp (𝜒 2 = 2424, 𝑝≈0). A practitioner stopping here would conclude that human co-authorship reduces merge probability. Per-agent analysis (true picture). Table 2 stratifies by agent identity. The within-agent picture almost completely reverses the pooled finding. Anatomy of the paradox. The confound is Codex: it contributes 64.9% of all PRs, achieves an 82.6% merge rate, and uses Co-Authored-By in only 1.2% of its submissions, inflating the pooled autonomous rate far above any individual agent’s. Conversely, Copilot and Devin (both high-coauth) have much
Beyond Simpson’s Paradox: A Cascade of Confounders in AI Agent Pull-Request Co-Authorship
Table 1: Pooled merge rates by co-authorship status (all agents combined). The −26.0 pp gap is entirely confounded by agent composition. Co-Authored-By
𝑛 PRs
Merge Rate
Yes (collaborative) No (pure agent)
10,764 22,832
53.8% 79.8% −26.0 pp
Delta
Agent Codex Copilot Cursor Devin Claude Code
Table 3: Merge rates by author/committer collaboration mode across all agents (𝑛 = 33,564 PRs; excludes 16 humanauthor/bot-committer PRs and 16 PRs lacking commit metadata).
𝜒 2 = 2424, 𝑝≈0
Table 2: Per-agent merge rates by co-authorship status. Copilot and Devin show large, highly significant positive gaps; Codex’s −4.4 pp and the small positive gaps for Cursor and Claude Code have 95% CIs that include zero. The pooled negative is an artefact of agent composition (Simpson’s Paradox). % coauth
MR (coauth)
MR (pure)
Δ
1.2% 89.4% 69.8% 95.9% 76.7%
78.3% 47.4% 66.3% 55.1% 60.2%
82.6% 6.2% 62.7% 21.6% 55.1%
−4.4 pp +41.2 pp +3.7 pp +33.5 pp +5.1 pp
lower overall merge rates, depressing the pooled collaborative rate. The within-agent gaps are striking: +41.2 pp for Copilot and +33.5 pp for Devin, with pure-autonomous merge rates of just 6.2% and 21.6%. Of Copilot’s 528 purely-autonomous PRs, 54.5% are explicit drafts ([WIP] in title); excluding these raises pure-autonomous MR from 6.2% to 13.3%. Devin’s 199 pure-autonomous PRs contain no WIP markers. By contrast, Codex’s tiny −4.4 pp gap is consistent with its low co-authorship rate reflecting design choice rather than neglect. Within-repo robustness check. To absorb repository-level confounders, we estimate merged𝑖 = 𝛼𝑟,𝑎 + 𝛽 · coauth𝑖 + 𝜀𝑖 with repo×agent pair fixed effects on the 391 repo–agent pairs (12,301 PRs) with both co-authored and pure PRs (cluster-robust SEs at pair level). The overall effect is +15.2 pp (SE = 2.5, 𝑝<0.001, 95% CI [10.4, 20.0] pp). Crucially, this effect is driven almost entirely by Copilot (+36.2 pp, 𝑝<0.001); a leave-one-out check excluding the top-3 Copilot repositories (microsoft/vscode, dotnet/aspire, mlflow/mlflow) yields +35.4 to +36.4 pp, so the result is not a single-repo artefact. However, Copilot’s pure-autonomous PRs are mostly singlecommit (median 1 vs. 4 for co-authored), and adding log(commits) as a control reduces the within-repo Copilot effect to +24.4 pp (𝑝<10−10 ). Restricting to multi-commit PRs (which removes the single-commit-draft confound), the within-repo Copilot effect is +4.8 pp (𝑝 = 0.59): not statistically distinguishable from zero. The headline Copilot association therefore reflects a structural workflow correlate (single-commit drafts vs. multi-commit submissions) as much as human co-authorship per se. The multi-commit Copilot null is also underpowered: only 𝑛=38 multi-commit pureautonomous Copilot PRs exist, giving a minimum detectable effect of roughly ±22 pp at 80% power, so we cannot rule out moderate effects. Devin’s cross-sectional gap (+33.5 pp) collapses to +1.6 pp
AgenticSE’26, August 2026, Jeju, Korea
Mode
𝑛 PRs
Merge Rate
human_both agent_draft fully_autonomous
23,200 336 10,028
82.0% 64.3% 47.6%
(𝑝 = 0.73) within repos. Codex (+2.5 pp), Cursor (+2.1 pp), and Claude Code (+7.4 pp) also show near-zero within-repo effects, though Codex’s within-repo sample is concentrated (top-3 repos contribute 69% of its 6,186 PRs). The practical implication is more cautious than the cross-sectional numbers suggest: once repository selection and PR structure are controlled, no agent shows a clear within-stratum co-authorship effect.
4.2
RQ2: Author/Committer Collaboration Modes
Table 3 presents merge rates across the three collaboration modes derived from author/committer attribution. The pattern is strictly monotone: every additional step of human involvement is associated with a higher merge rate. Moving from fully-autonomous to agent-drafted (human committer) raises the merge rate by +16.7 pp; moving further to human-both raises it by a further +17.7 pp. Crucially, this gradient survives the commit-count control that dissolved the RQ1 co-authorship effect: within every commit-count bin (1, 2, 3–5, >5 commits) and within every agent with sufficient data, human_both exceeds fully_autonomous by 25–49 pp. The author/committer signal, unlike the Co-Authored-By marker, is not absorbed by PR-structure controls. The small agent_draft category (336 PRs) at 64.3% sits in an informative intermediate position, consistent with human sign-off compensating for agent-authored content.
4.3
RQ3: Multi-Agent Adoption Patterns
We briefly examined whether adopting a second agent changes existing merge rates (233 multi-agent repos). A difference-in-differences regression on weekly merge rates, controlling for repository fixed effects and a linear time trend, estimates a treatment effect of −12.1 pp (SE = 3.6, 𝑝<0.001, 95% CI [−19.2, −5.0] pp). This suggests that introducing a second agent is associated with a meaningful decline in merge rates, though the effect may reflect compositional changes (the second agent contributing lower-quality PRs) rather than disruption to the first agent’s workflow.
5
Discussion and Limitations
Direction of causality. Our data are purely observational. The positive within-agent association between co-authorship and merge rate does not imply that adding a human co-author to a PR causes it to be merged. An equally plausible interpretation is selection: developers may invest effort in co-authoring only those agent PRs they already believe are worthwhile, so Co-Authored-By marks
AgenticSE’26, August 2026, Jeju, Korea
Yu et al.
Table 4: A collaboration-level framework for human involvement in agent-generated PRs, ordered by increasing human participation. L0/L1/L3 merge rates from author/committer classification (Table 3); L2 from co-authorship analysis (pooled). Level
Description
MR
L0: Fully autonomous L1: Agent draft L2: Co-authored L3: Human-both
bot author + bot committer bot author, human committer Co-Authored-By present human author + human committer
47.6% 64.3%
Table 5: Agent-specific deployment recommendations. The “Min. Level” column indicates the minimum collaboration level at which the within-repo evidence supports deployment.
∗
82.0%
Agent composition dominance. The Simpson’s Paradox is manufactured almost entirely by Codex, which contributes 64.9% of the dataset. Interpretations of pooled statistics in multi-agent datasets must control for agent identity, just as cross-study meta-analyses must control for study characteristics. We recommend against reporting agent-pooled merge rates without stratification in future empirical work on AI agents. Measurement limitations. The Co-Authored-By trailer is agentinserted and not standardised; some agents may omit it even when humans edited commits. Our headline definition counts any trailer as “collaborative,” which is conservative for Copilot/Cursor/Devin (where ≥ 99% of trailers are human emails) but inflates apparent collaboration for Claude Code and Codex, which often self-attribute via [email protected]. Restricting to non-bot co-author emails leaves the within-repo Copilot finding essentially unchanged (+36.1 vs +36.2 pp) but reveals a previously masked positive Claude Code effect (cross-sectional +33.8 pp, 𝑛=47). We measure merge rate at collection time; open PRs are coded as “not merged.” Scope. This study covers five agents in AIDev and may not generalise beyond them. We do not control for PR size, task type, or reviewer experience [4, 12], and the dataset exhibits survivorship bias (only repos that adopted agents are included). A collaboration-level framework. Our results motivate a fourlevel taxonomy of human involvement in agent-generated PRs, ordered by increasing human participation: Combining this with the within-repo analysis yields agent-specific deployment recommendations: As Table 5 shows, Copilot is the only agent with a significant within-repo co-authorship association at the cross-sectional level, but this effect is largely explained by PR structure: restricted to multi-commit PRs, Copilot’s within-repo effect is +4.8 pp (𝑝=0.59). For Codex, Cursor, and Devin, autonomous operation shows no within-repo penalty. Practically, teams deploying Copilot should be aware that the apparent benefit of mandating co-authorship is weaker than headline numbers suggest once PR structure is
Min. Level
Codex Copilot Cursor Devin Claude Code
L0 (autonomous OK) L2+ tentative† L0 (autonomous OK) L0 (autonomous OK) L0–L1 (tentative)
Within-repo Δ +2.5 pp (𝑝=0.40) +36.2 pp (𝑝<0.001) +2.1 pp (𝑝=0.62) +1.6 pp (𝑝=0.73) +7.4 pp (𝑝=0.35)
† Within-repo Copilot effect drops to +4.8 pp (𝑝=0.59)
∗ Pooled 53.8%; within-repo +15.2 pp.
quality rather than creating it. The cascade of confounders documented in §4.1 is consistent with this: pure-autonomous Copilot PRs are mostly single-commit drafts (54.5% [WIP]), so the crosssectional gap captures PR maturation, repository selection, and curation as much as co-authorship per se. Full causal identification would require a randomised experiment.
Agent
when restricted to multi-commit PRs.
controlled; the strongest practical signal remains Copilot’s overall low pure-autonomous merge rate (∼13% non-WIP), which itself reflects that pure-autonomous Copilot PRs are predominantly single-commit drafts.
6
Conclusion
We documented a textbook Simpson’s Paradox in AI agent pullrequest data: pooled across five agents, co-authorship appears to hurt merge rates (−26.0 pp), yet within-agent analysis reveals positive gaps for four of five agents, with Copilot and Devin showing +41.2 and +33.5 pp cross-sectionally. This reversal, however, is only the first layer of a cascade of confounders: within-repo controls collapse Devin’s gap to +1.6 pp, and a commit-count control reduces Copilot’s within-repo gap from +36.2 to +24.4 pp; restricted to multi-commit PRs the Copilot effect is +4.8 pp (𝑝=0.59). A complementary author/committer analysis confirms a graded pattern (47.6% → 64.3% → 82.0%). These findings argue for agent-stratified analyses in SE empirical studies and demonstrate that cross-sectional co-authorship associations can be misleading without within-repo and withinPR-structure controls: what appears as a universal or even agentspecific human-involvement benefit is, once both confounders are controlled, more cautiously characterised as a structural correlate (single-commit drafts vs. multi-commit submissions) than evidence of a causal benefit of co-authorship per se. Analysis scripts are available for replication at https://anonymous.4open.science/r/prcoauth-simpsons-paradox-7DC2.
Acknowledgments We thank the maintainers of the AIDev dataset for making this research possible, and the anonymous reviewers for their constructive feedback.
References [1] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374 (2021). [2] Zhiyuan Cheng, Longying Lai, Yue Liu, and Yu Sun. 2026. Toward Sustainable On-Device Intelligence: A Survey on Energy-Efficient RAG Systems with Small Language Models. Available at SSRN 6698538 (2026). [3] Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C. Desmarais, and Zhen Ming Jack Jiang. 2023. GitHub Copilot AI pair programmer: Asset or liability? Journal of Systems and Software 203 (2023), 111734. doi:10.1016/j.jss.2023.111734
Beyond Simpson’s Paradox: A Cascade of Confounders in AI Agent Pull-Request Co-Authorship
[4] Georgios Gousios, Margaret-Anne Storey, and Alberto Bacchelli. 2016. Work Practices and Challenges in Pull-Based Development: The Contributor’s Perspective. In Proceedings of the 38th International Conference on Software Engineering (ICSE). ACM, 285–296. doi:10.1145/2884781.2884826 [5] Hao Li, Haoxiang Zhang, and Ahmed E. Hassan. 2025. The Rise of AI Teammates in Software Engineering (SE) 3.0: How Autonomous Coding Agents Are Reshaping Software Engineering. arXiv:2507.15003 [cs.SE] [6] Judea Pearl. 2014. Comment: Understanding Simpson’s Paradox. The American Statistician 68, 1 (2014), 8–13. doi:10.1080/00031305.2014.876829 [7] Edward H. Simpson. 1951. The Interpretation of Interaction in Contingency Tables. Journal of the Royal Statistical Society, Series B 13, 2 (1951), 238–241. doi:10.1111/j.2517-6161.1951.tb00088.x [8] Mairieli Wessel, Alexander Serebrenik, Igor Wiese, Igor Steinmacher, and Marco A. Gerosa. 2020. Effects of Adopting Code Review Bots on Pull Requests to OSS Projects. In Proceedings of the 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 1–11. doi:10.1109/icsme46990.2020.
AgenticSE’26, August 2026, Jeju, Korea
00011 [9] Mairieli Wessel, Alexander Serebrenik, Igor Wiese, Igor Steinmacher, and Marco A. Gerosa. 2022. Quality Gatekeepers: Investigating the Effects of Code Review Bots on Pull Request Activities. Empirical Software Engineering 27, 5 (2022). doi:10.1007/s10664-022-10130-9 [10] Mairieli Wessel, Igor Wiese, Igor Steinmacher, and Marco Aurelio Gerosa. 2021. Don’t Disturb Me: Challenges of Interacting with Software Bots on Open Source Software Projects. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (2021), 1–21. doi:10.1145/3476042 [11] Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2024. Agentless: Demystifying LLM-Based Software Engineering Agents. arXiv preprint arXiv:2407.01489 (2024). [12] Yue Yu, Huaimin Wang, Gang Yin, and Tao Wang. 2016. Reviewer Recommendation for Pull-Requests in GitHub: What Can We Learn from Code Review and Bug Assignment? Information and Software Technology 74 (2016), 204–218. doi:10.1016/j.infsof.2016.01.004