arXiv CSopen access

A meta-analysis of the effect of generative AI on productivity and learning in programming

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

software-architecture software-engineering testing

software engineering, software architecture, testing

arXiv:2605.04779v1 [cs.SE] 6 May 2026

A meta-analysis of the effect of generative AI on productivity and learning in programming Sebastian Maier1,2,† , Moritz Gunzenhäuser1,† , Jonas Schweisthal1,2 , Manuel Schneider1 , and Stefan Feuerriegel* 1,2 1 2

LMU Munich, Munich, Germany

Munich Center for Machine Learning (MCML), Munich, Germany †

These authors contributed equally to this work.

* Corresponding author: [email protected]

Abstract Generative artificial intelligence (GenAI) is increasingly used for programming, yet it remains unclear when and where GenAI tools lead to productivity gains. Evidence on the effects of GenAI on the long-term development of programming skills is similarly mixed. Here, we present a meta-analysis of n = 23 studies reporting k = 27 effect sizes to quantify the effect of GenAI-powered coding assistants on productivity and learning. We systematically searched (i) ACM, (ii) arXiv, (iii) Scopus, and (iv) Web of Science for studies published between 2019 and 2025. Studies were required to compare GenAI-assisted with unassisted programming using quantitative measures of (1) productivity (i.e., task completion time, commits, and lines of code) and (2) learning (i.e., exam performance). We assessed the risk of bias using RoB2 and ROBINS-I and compared standardized effect sizes using Hedges’ g. We find a statistically significant, but moderate positive effect of GenAI assistance on developer productivity (g = 0.33, 95% CI: [0.09, 0.58]), yet with substantial heterogeneity across settings. Notably, productivity gains tend to be larger in controlled experimental settings, while effects are smaller in open-source and enterprise contexts. In contrast, we find no statistically significant effect of GenAI assistance on learning outcomes (g = 0.14, 95% CI: [−0.18, 0.47]). Overall, these results highlight that GenAI coding assistants can increase developer productivity, although these gains depend strongly on context. In educational settings, however, the use of GenAI does not consistently translate into improved learning or skill development, which highlights the need for careful integration of GenAI into computer science education.

Introduction Generative artificial intelligence (GenAI) tools such as GitHub Copilot [1] and Cursor [2] have rapidly become a crucial part of the software engineering landscape [3–5]. Adoption rates of GenAI tools for coding grow rapidly [6], with 84% of developers already using or planning to use GenAI tools in their workflow [7]. This leads to a fundamental paradigm shift in how code is written and maintained [8–10]. Many GenAI tools based on state-of-the-art large language models, such as GPT, Claude, Gemini, and DeepSeek, now achieve near-perfect scores on established coding benchmarks and reach or even surpass the vast majority of human developers [11–15]. At the same time, GenAI has enabled “vibe coding”, a style of development in which programmers iteratively interact with GenAI through prompts and accept generated code based on perceived plausibility rather than a detailed understanding of its logic or behavior [16–18]. Here, we analyze the effect of GenAI tools on productivity and learning in programming. Empirical evidence on the productivity effects of GenAI tools remains mixed. On the one hand, developers report productivity gains from automating repetitive coding tasks and from lower cognitive load required for recalling syntax [19–22]. GenAI tools can also lower entry barriers and help developers, particularly novice developers, to understand unfamiliar code bases [23–25]. On the other hand, using GenAI for programming changes work from active problem solving to passive evaluation and review of AI-generated code [26]. This shift may also increase cognitive load and thus decrease performance, especially for experienced developers [3, 27–29]. In addition, AIgenerated code can introduce errors such as security vulnerabilities [30, 31] and “technical debt” due to reduced code maintainability and deviations from established design practices [32]. Vibe coding further encourages software development based on plausibility rather than true understanding of the underlying code [16–18]. While this may speed up initial development, it can increase maintenance costs over time, particularly for ensuring code quality [27]. In sum, the impact of GenAI on productivity can vary widely across developers, tasks, and settings. 3

GenAI use is also reshaping how programming skills are acquired. Although proficiency with GenAI tools is increasingly expected in the labor market, such use by novice learners without welldeveloped foundational skills may alter how students engage with core learning processes and thus raises important challenges for computer science education [33, 34]. On the one hand, students benefit from personalized AI-based tutoring, for example, by asking GenAI systems to explain code fragments or solution strategies, which provides on-demand answers and explanations that are perceived as helpful and accessible [24, 35–37]. On the other hand, reliance on GenAI can allow students to bypass core learning processes, such as trial-and-error learning, debugging, and iterative refinement, when they delegate core tasks to GenAI tools [38–40]. This phenomenon is often referred to as cognitive offloading [41, 42] and may reduce active engagement with problem solving, which, in turn, can lead to shallow understanding and ultimately impair the development of fundamental programming skills [43, 44]. As a result, the capabilities students develop during AI-assisted coding may prove tool-dependent, thus failing to persist without AI support [45] or even yielding lower skill acquisition compared to unassisted learners [46]. Despite a growing body of work on GenAI-assisted programming, evidence on the effects on productivity and learning remains fragmented and often inconsistent [27, 28, 47, 48]. A key challenge is the substantial heterogeneity across studies, including differences in developer experience, and study settings (e.g., controlled experiments with standardized tasks versus real-world contexts). This heterogeneity makes it difficult to draw generalizable conclusions and leaves the question of when and where GenAI tools are effective. To address this, we focus on two core outcomes of GenAI-assisted programming: (1) productivity, operationalized through task completion time and code output, and (2) learning, measured by exam performance. We thus study the following two research questions (RQs): RQ1: How does GenAI assistance affect programmer productivity? RQ2: How does GenAI assistance affect learning outcomes for computer science students? Here, we conduct a pre-registered meta-analysis to quantify the impact of GenAI on produc4

tivity and learning (see Figure 1). After screening 10,115 records during our literature search, we identified n = 23 relevant studies. We extracted quantitative measures of productivity and learning from each study and then computed standardized effect sizes based on Hedges’ g under a random-effects model. To explain variability in outcomes across studies, we conducted moderator analyses to examine differences across study context, participant characteristics, and technological implementation. This approach allows us to identify when and where GenAI assistance is effective.

Figure 1: Research overview. a, Overview of the systematic process for literature review and data synthesis. b, The meta-analysis estimates the effect of GenAI on productivity (measured by task completion time and code output) and learning (measured by exam performance) across the identified studies. c, Moderator analyses examine how study context, task type, and assessment conditions contribute to variation across reported effect sizes. This allows to identify when and where GenAI assistance is effective.

Results Productivity effect of GenAI (RQ1)

Productivity Proxies

Study Setting

Task completion time

Laboratory

Code commits

Open−Source

Code output

Enterprise

6 Studies (n)

Participant Level

3 4 5 Studies (n)

Programming Language

Experienced

Python

Students

Multi−language

Mixed

JavaScript C++

3 4 5 Studies (n)

3 4 Studies (n)

Figure 2: Characteristics of productivity studies.

Study overview To analyze the effect of GenAI on productivity (RQ1), we conducted a systematic synthesis of the existing literature (see Method section). Overall, our search identified 16 effect size estimates derived from n = 14 unique studies that examine the impact of GenAI assistance on developer productivity (see Fig. 1 for a descriptive overview and Table 1 for the full list of studies). The productivity studies comprise studies on a participants level (m = 3535 participants in total) as well as studies conducted on a repository level (r = 6355 repositories in total). The studies operationalized productivity using common proxies for measuring development activities in software engineering research (see Fig. 2), including task completion time (n = 9), the number of commits (n = 4), and the lines of output code (n = 1). The studies span open-source (n = 4), enterprise (n = 3), and controlled laboratory settings (n = 7). The studies were published between 2022 and 2025, and include both peer-reviewed publications (n = 6) and preprints (n = 8), which reflect 7

the rapid pace of GenAI research. The research methods of the studies vary and include randomized controlled trials (n = 10), natural experiments (n = 2), and quasi-experiments (n = 2). Evidence is generated based on within-subject comparisons (n = 6), between-repository comparisons (n = 3), and betweensubject comparisons (n = 5). The primary GenAI assistant was GitHub Copilot (n = 6 studies), while the other studies used tools such as Cursor (n = 2) and other GenAI-based systems (n = 6). Python was the most frequently used language (n = 6), with the remaining studies conducted in multi-language settings (n = 5), JavaScript (n = 2), or C++ (n = 1). Taken together, the existing studies focusing on productivity effects reveal substantial heterogeneity in the study design.

Main meta-analysis To examine how GenAI assistance affects programmer productivity (RQ1), we synthesized evidence from existing studies using a random-effects meta-analysis. Specifically, we consolidated the 16 independent effect size estimates that compared GenAI-assisted development with a baseline condition involving manual programming into a single pooled estimate, while accounting for variation across study contexts. Overall, we find a statistically significant, positive but moderate increase in productivity associated with GenAI assistance (Hedges’ g = 0.33, 95% confidence interval [CI]: [0.09, 0.58], SE = 0.13, p = 0.008). The forest plot is shown in Fig. 3. At the same time, productivity effects vary substantially, which is reflected by a substantial heterogeneity across studies (I 2 = 99%), with an estimated between-study variance of τ 2 = 0.22 (Q(15) = 206.06, p < 0.001). This suggests that GenAI assistance does not yield uniform productivity gains, but that effects depend strongly on where and how these tools are used.

Forest Plot: Productivity P002_R001 P004_R001 P006_R001 P014_R001 P007_R001 P003_R001 P013_R001 P001_R001 P001_R002 P011_R001 P010_R001 P009_R001 P005_R001 P012_R001 P010_R002 P008_R001

4.8% 5.3% 4.4% 5.5% 5.4% 5.9% 6.9% 7.0% 7.0% 6.8% 7.0% 7.0% 6.2% 7.1% 6.7% 6.8% Hedges' g = 0.3315

−0.5

0.0

0.5 1.0 1.5 2.0 Effect size (Hedges' g)

2.5

Figure 3: Forest plot of the pooled effect of GenAI assistance on developer productivity. To answer RQ1, the plot summarizes individual effect size estimates (Hedges’ g) and the corresponding 95% CIs, with study weights shown on the right. The vertical line at g = 0 denotes a null effect, while estimates to the right indicate higher productivity gain from GenAI assistance. The orange dashed line shows the pooled estimate, and the orange shaded area represents the 95% CI of the pooled estimate. Overall, the meta-analysis indicates a moderate positive effect (Hedges’ g = 0.33, 95% CI: [0.09, 0.58], SE = 0.13, p = 0.008), alongside substantial heterogeneity across studies (I 2 = 99%).

Heterogeneity in productivity effects To understand when and where GenAI is effective, we conducted moderator analyses to identify key determinants of productivity gains. Specifically, we analyzed six moderators: (a) study setting (i.e., laboratory, enterprise, open-source), (b) the GenAI interface (e.g., GitHub Copilot), (c) programming language (e.g., Python), (d) the participant level (students, experienced, mixed), (e) whether the study was randomized (randomized, non-randomized), and (f) the experimental design (between-repositories, between-subject, within-subject). The moderators were selected to reflect commonly reported study characteristics, while ensuring sufficient representation and thus 9

statistical power across subgroups. All moderator analyses were performed using mixed-effects meta-regression models; the results are summarized in Fig. 4. We examined whether the context in which GenAI tools are evaluated influences observed productivity gains by comparing controlled laboratory experiments, enterprise settings, and opensource environments. The study setting may shape outcomes because tasks in controlled laboratory experiments are typically shorter, more structured, and require less familiarization with existing code bases, whereas real-world development involves coordination overhead, constraints imposed by existing and legacy code bases, and quality assurance. Differences may further arise between enterprise and open-source settings due to different collaboration practices, governance structures, and programmer backgrounds. Interestingly, the moderation analysis reveals that the study setting significantly moderates the productivity effect of GenAI (QM (2) = 10.51, p = 0.005), accounting for approximately 36% of the heterogeneity across studies. Laboratory experiments yield a large and significant effect (g = 0.73, p < 0.001), whereas effects in enterprise (g = 0.19, p = 0.448) and open-source settings (g = 0.01, p = 0.975) are substantially smaller and do not differ significantly from zero. This result suggests that the productivity gains with GenAI tools observed in experiments diminish substantially in more realistic contexts. None of the remaining moderators reached statistical significance at common significance thresholds, which is the case for the GenAI interface (QM (1) = 0.21, p = 0.644), programming language (QM (1) = 0.43, p = 0.514), and the participant level (QM (2) = 3.97, p = 0.138). Also, in terms of study design, neither randomization (QM (1) = 1.63, p = 0.202), nor experimental design was a significant moderator (QM (2) = 1.35, p = 0.510), suggesting that the substantial residual heterogeneity observed across studies could not be accounted for by whether studies were randomized or not and employed a between-repositories, within-subject, or between-subject design. However, these results should be interpreted cautiously, given that the small number of studies and limited variability across moderator levels may have been insufficient to reliably detect meaningful differential effects. 10

Subgroup

Hedges' g with CIs

All effect sizes (**) Pooled

Study Setting (**) Laboratory

Enterprise

Open−Source

GenAI Interface GitHub Copilot

Other

Programming Language Python

Others

Participant Level Students

Experienced

Mixed

Randomization Randomized

Non−randomized

Experimental Design Between−repositories 4 Between−subject

Within−subject

6 −1

−0.5

0.5

1.5

Figure 4: Predicted productivity effect sizes by moderator subgroup. Predicted effect sizes (Hedges’ g) derived from univariate random-effects meta-regressions (REML estimator). Thick bars represent 90% confidence intervals; thin bars represent 95% confidence intervals. k denotes the number of effect sizes per subgroup. Asterisks in parentheses after moderator category labels indicate statistically significant omnibus tests (∗ p < 0.05; ∗∗ p < 0.01; ∗∗∗ p < 0.001). The vertical dashed line represents a null effect (g = 0); positive values indicate a beneficial effect of GenAI assistance on productivity.

Time trend of productivity effects To evaluate how the reported productivity effects of GenAI have changed over time, we conducted a cumulative meta-analysis ordered by publication year (see Fig. 5). In the cumulative analysis,

studies are chronologically ordered and then added one by one to the meta-analysis to update the pooled estimate with the overall effect size from all studies up to a specific year. This approach thus tracks how the estimated productivity effect evolves as new studies are published. The cumulative meta-analysis shows a consistent, modest productivity gain that remains stable as studies are added. Despite the rapid advancement of underlying foundation models over this period, the cumulative evidence does not reflect a corresponding increase in reported productivity gains. Rather, the cumulative estimate stabilized at a small, consistent productivity gain (g = 0.331) as evidence accumulated. This suggests that raw model capability or AI literacy may not be the primary drivers of productivity gains in GenAI-assisted coding tasks. Productivity ( All Studies ): Evidence Evolution 2022 2023

2024

2025

Final: g = 0.331

Cumulative Effect Size (g)

Identifier of included study

Figure 5: Longitudinal analysis of productivity effects by publication year. The plot shows a cumulative meta-analysis over time. Each point shows the pooled effect size after sequentially adding studies in chronological order from 2022 to 2025, while background colors indicate the publication year. Shaded bands represent 95% CIs around the cumulative estimates. Light background shading distinguishes different time periods. The horizontal dashed line (in red) denotes a null effect, and the dotted line represents the final pooled estimate of the overall effect. Study identifiers on the x-axis (e.g., P012, P005) correspond to the order in which studies and the respective effect sizes enter the cumulative analysis.

Robustness checks We conducted several further analyses as robustness checks. We examined whether effect sizes differed between studies rated as lower versus higher risk of bias using RoB2 and ROBINS-I (see Online Supplement C). Risk of bias was not a significant moderator (QM (1) = 2.90, p = 0.089). However, lower-risk studies showed descriptively smaller effect sizes than higher-risk studies (g = 0.08, 95% CI: [−0.29, 0.45] vs. g = 0.50, 95% CI: [0.19, 0.80]). The leave-one-out sensitivity analysis showed that the pooled effect size was robust to the exclusion of individual studies (range: g = 0.24–0.38, all p < 0.05), but where the omission of P002 [49] and P004 [50] produced the largest reductions. Influence diagnostics further identified these two studies as exerting disproportionate influence on the pooled estimate, evidenced by Cook’s d values (0.55 and 0.56) and studentized residuals exceeding the conventional |z| > 3 threshold (3.01 and 2.83), suggesting their effect sizes deviated substantially from the remaining studies. However, as the effect remained positive and statistically significant across all leave-oneout iterations, these studies did not alter the overall interpretation. Notably, heterogeneity remained consistently high across all iterations (I 2 = 98.8–99.6%), thereby confirming that no single study accounted for the substantial between-study variance. We further examined the potential presence of publication bias and small-study effects using Egger’s mixed-effects regression test [51], which indicated significant funnel plot asymmetry (z = 4.52, p < 0.001). To account for this asymmetry, we applied the trim-and-fill procedure [52], which imputed no missing studies, suggesting that the detected asymmetry does not reflect a systematic suppression of null results. This could imply that the funnel plot asymmetry is due to true heterogeneity rather than publication bias per se.

Learning effect from GenAI (RQ2) Study overview To analyze the effect of GenAI on learning (RQ2), we again conducted a systematic literature search to then perform a meta-analysis (see Method section). Overall, we identified n = 10 unique studies examining the impact of GenAI on programming-related learning outcomes, comprising m = 1, 069 participants in total, from which we extracted 11 effect size estimates (see Fig. 6 for a descriptive overview and Table 2 for the full list of studies). All effect sizes are based on quantitative measures of learning outcomes, specifically comparisons of exam performance following instruction phases in which students either had access to GenAI assistants or worked without them. The studies were published between 2023 and 2025, all in peer-reviewed outlets, and span a geographically diverse set of countries, including Slovenia, South Korea, China, Taiwan, Oman, and the United States. Methodologically, n = 4 studies employed experimental designs, while n = 6 studies used quasi-experimental designs to compare exam performance between subjects. Most studies are based on samples of university students (n = 6), while the remaining studies involve high school (n = 2), middle school (n = 1), and mixed K-12 samples (n = 1). Variation in GenAI tools was narrow, with OpenAI models accounting for the majority of deployments across all but two studies. Python was the most commonly used programming language (n = 5), followed by C++ (n = 3), Java (n = 1), and C# (n = 1). The study duration also varied substantially: n = 7 studies were from GenAI interventions lasting fewer than ten weeks, while n = 3 studies were based on studies where the duration of the GenAI intervention exceeded ten weeks. In most cases, GenAI assistance was not allowed during final assessments (n = 7), whereas n = 3 studies did allow the usage of GenAI in the treatment group for the assessment.

Study Duration

Exam Environment

< 10 weeks

No GenAI

>= 10 weeks

GenAI allowed

4 5 Studies (n)

Participant Level

4 5 Studies (n)

Programming Language

University

Python

Pre−University

C++ Java C#

3 4 Studies (n)

Figure 6: Characteristics of learning effect studies.

Main meta-analysis To examine how GenAI assistance affects learning outcomes in programming education (RQ2), we synthesized evidence from 11 independent effect sizes from n = 10 studies that compared GenAIsupported learning with traditional educational approaches. For this, we again use a random-effects model while accounting for variation across education contexts. Notably, the included studies differed in whether GenAI assistance was available during the assessment itself or only during the learning phase, a distinction that, as we show in the section below, proves critical for interpreting the pooled results. Overall, we find a small but statistically non-significant pooled effect of GenAI assistance on learning outcomes (Hedges’ g = 0.14, 95% CI: [−0.18, 0.47], SE = 0.17, p = 0.389). The corresponding CI includes zero, indicating no reliable overall improvement in exam performance associated with GenAI use across the analyzed studies. The forest plot is shown in Fig. 7. At the same time, substantial heterogeneity was observed, with an I 2 statistic of 86% and an estimated between-study variance of τ 2 = 0.25 (Q(10) = 54.96, p < 0.001). This level of heterogeneity suggests that the effects of GenAI assistance on learning outcomes depend strongly on the contextual setting, as for example on whether students retained access to GenAI during the test. 15

Forest Plot: Learning L004_R001 L009_R001 L010_R001 L002_R001 L003_R001 L008_R001 L001_R001 L006_R001 L007_R002 L007_R001 L005_R001

6.2% 10.0% 9.6% 10.3% 9.3% 9.0% 10.1% 10.1% 8.8% 9.3% 7.3% Hedges' g = 0.1449

−1.5 −1.0 −0.5 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Effect size (Hedges' g)

Figure 7: Forest plot of the pooled effect of GenAI assistance on learning outcomes (RQ2). Rows represent the individual effect size estimates from different studies, together with the 95% CIs as shaded areas and study weights shown on the right. The vertical line at g = 0 denotes a null effect, while estimates to the right indicate a higher learning effect from GenAI assistance. The orange dashed line shows the pooled estimate (Hedges’ g) from random-effects meta-analysis, and the orange shaded area represents the 95% CI of the pooled estimate. Overall, the pooled effect size is small and statistically non-significant (Hedges’ g = 0.14, SE = 0.17, p = 0.389, 95% CI [−0.18, 0.47]), alongside substantial heterogeneity across studies (I 2 = 86%).

Heterogeneity in learning effects To understand when and where GenAI affects learning outcomes in programming education, we conducted univariate moderator analyses to identify key sources of variation across studies. Again, the moderator analyses were performed using mixed-effects meta-regression models; the results are summarized in Fig. 8. The most influential moderator was the exam environment, meaning whether GenAI assistance was permitted during assessment. The overall test of moderation was significant (QM (1) = 7.23, p = 0.007), indicating that the testing environment explained between-study heterogeneity. Indeed, the moderator accounted for 45.49% of the total heterogeneity, representing a substantial reduction from the unconditional model. When participants were not allowed to use AI for the assessment, the effect was non-significant and even pointed slightly in a negative direction (g = 16

−0.06, 95% CI: [−0.36, 0.24], p = 0.674). In contrast, when AI tools were permitted during the assessment, the estimated effect was large and statistically significant (g = 0.76, 95% CI: [0.24, 1.28], p = 0.004). This pattern suggests that students perform better on assessments when AI tools are available, but these performance gains may reflect tool-dependent augmentation rather than durable learning, raising concerns about the transferability of GenAI-assisted instruction to unaided contexts. None of the remaining moderators significantly explained variation in learning effect sizes. The type of GenAI interface did not moderate the effect (QM (1) = 0.70, p = 0.404), nor did the programming language taught (QM (1) = 0.06, p = 0.807). Similarly, study duration (QM (1) = 0.48, p = 0.487), and the participant educational level (university vs. pre-university; QM (1) = 0.06, p = 0.812) were all non-significant moderators. Further, as for the productivity outcomes, the randomization was not a significant moderator ( QM (1) = 1.53, p = 0.216). Notably, none of these five moderators accounted for any of the observed heterogeneity (R2 = 0.00% in each case), and residual heterogeneity remained substantial throughout (I 2 > 85%). This suggests that, among the study-level characteristics we coded, only the exam environment accounts for a meaningful share of the observed heterogeneity.

Subgroup

Hedges' g with CIs

All effect sizes Pooled

Exam Environment (**) No GenAI

GenAI allowed

GenAI Interface ChatGPT

Other

Programming Language Python

Others

Duration <10 weeks

>10 weeks

Participant Educational Level University

Pre−University

Randomization Randomized

Non−randomized

4 −1

−0.5

0.5

1.5

Figure 8: Predicted learning effect sizes by moderator subgroup. Predicted effect sizes (Hedges’ g) derived from univariate random-effects meta-regressions (REML estimator). Thick bars represent 90% confidence intervals; thin bars represent 95% confidence intervals. k denotes the number of effect sizes per subgroup. Asterisks in parentheses after moderator category labels indicate statistically significant omnibus tests (∗ p < 0.05; ∗∗ p < 0.01; ∗∗∗ p < 0.001). The vertical dashed line represents a null effect (g = 0); positive values indicate a beneficial effect of GenAI assistance on learning outcomes.

Robustness checks To assess robustness, we first performed a leave-one-out sensitivity analysis, which indicates that the pooled effect size was generally robust to the exclusion of individual studies, with estimates ranging from g = 0.06 to g = 0.21. Importantly, no single study altered the overall conclusion. Additionally, and consistent with the productivity analysis, risk of bias was not a significant mod18

erator of learning effects (QM (1) = 1.15, p = 0.284; see Online Supplement B). Further, to assess potential publication bias, we conducted Egger’s regression test for funnel plot asymmetry [51] and a trim-and-fill analysis [52]. Egger’s test revealed no significant asymmetry (z = 0.55, p = 0.585), and the trim-and-fill procedure estimated zero missing studies, leaving the pooled estimate unchanged. Together, these results provide no indication of publication bias in the learning outcome. Similarly, influence diagnostics suggest that the learning dataset is relatively robust to outliers as no study exceeded conventional thresholds.

Discussion Summary of findings Our meta-analysis demonstrates that GenAI-based coding assistants can improve developer productivity, yet the evidence does not support a corresponding benefit for learning outcomes in programming education. This finding challenges overly optimistic claims from artificial benchmark results suggesting universal productivity gains [11, 15, 53], while the broader meta-analytic evidence also counters the notion that current GenAI tools are ineffective in practice [28]. Instead, the strong variation across studies indicates that the impact of GenAI assistance depends heavily on contextual characteristics. For productivity (RQ1), GenAI assistance is associated with a moderate positive effect, though gains are substantially larger in controlled laboratory environments than in open-source or real-world settings, suggesting that the simplified tasks in experimental settings may amplify measured effects relative to everyday development workflows. For learning (RQ2), significant effects emerged only when students retained access to GenAI during the assessment itself, suggesting that observed gains reflect tool-augmented performance rather than durable skill acquisition. This finding aligns with recent evidence that the educational benefits of GenAI only materialize when it is integrated deliberately into the learning process, as unrestricted access may hinder genuine skill acquisition [44, 46]. Implications Our analysis provides robust evidence that GenAI-based coding assistants can improve developer productivity, but the benefits depend strongly on context. Productivity gains are larger in controlled laboratory settings than in open-source or real-world development environments. This suggests that results from controlled experiments may overestimate the productivity improvements that organizations can expect in everyday software development, where code bases are more complex and

coordination costs are higher [27, 47, 48, 54]. For practice, this means that GenAI tools should not be treated as a one-size-fits-all productivity solution. Instead, the specific design choices of how GenAI tools are integrated into existing workflows, review processes, and quality controls [3,4,55] may matter just as much as the technical capabilities of the models alone. As GenAI tools take over more routine code generation [56], developers may spend less time writing code and more time reviewing, validating, and integrating AI-generated outputs [27, 28, 57]. Companies should therefore be careful when generalizing results from experimental studies to production environments [48]. In this context, emerging agentic workflows that better align GenAI assistance with developer roles and responsibilities may be particularly beneficial. Overall, our results suggest that the long-term value of GenAI in software engineering will depend not only on continued advances in model capability but also considerably on how effectively organizations embed these tools within real-world development processes. This points to a needed shift in focus away from simple coding benchmarks and toward a more human-centered understanding of how these tools can complement developer expertise and intent [58, 59]. While GenAI tools are now widely used among computer science students, our analysis does not support a robust improvement in learning outcomes. Although students perform better on assessments when allowed to use AI, these benefits do not transfer to settings without AI support. This suggests that GenAI may help students complete tasks without internalizing the underlying reasoning. When students rely on GenAI to generate or repair code, they may bypass critical learning processes such as reasoning through errors and internalizing core concepts. This pattern mirrors previous research in other educational domains [38, 44, 60] and is consistent with concerns about cognitive offloading [41,42,61]; as a result, such reliance on GenAI tools may backfire when students are required to explain, adapt, or debug AI-generated code. These risks are particularly salient given ongoing shifts in software engineering practice, where developers increasingly take on roles that emphasize code review, integration, and oversight rather than routine code production [40, 62]. If students graduate without strong foundations in code 21

comprehension and reasoning, they may struggle in these higher-level roles. At the same time, banning GenAI tools is neither realistic nor desirable given the growing role of GenAI in professional programming practice [63, 64]. Paradoxically, the very skills that GenAI makes more important in professional practice, critical evaluation, code reasoning, and independent problem solving, are precisely those that may atrophy when students rely on GenAI during learning [46]. This tension is amplified by the fact that novice developers, although they might experience immediate productivity gains from GenAI-assisted prototyping and debugging [23, 47], often lack the understanding needed to critically evaluate generated code [26, 65]. Instead, experienced developers might leverage these tools differently by using them to offload routine implementation while redirecting their effort toward complex problem solving [6, 66]. Our findings thus suggest the need to rethink curricula and the integration of GenAI by moving beyond code production alone and placing greater emphasis on explanation, reasoning, and critical evaluation, so that students develop the skills required to work safely and effectively with GenAI assistance [33, 43, 67, 68]. Current state of the literature Our meta-analysis provides a thorough analysis on the effect of GenAI assistance on productivity and learning outcomes by deliberately focusing on comparable outcomes across various settings. However, as with other studies, ours is subject to limitations that reflect the early and rapidly evolving nature of research on GenAI-assisted coding. First, GenAI capabilities evolved rapidly during the observation period, yet our cumulative meta-analysis shows that reported productivity effects remained stable over time rather than increasing with new model improvements. Second, with 16 effect sizes for productivity and 11 for learning, the evidence base constrains the granularity of moderator analyses. This reflects both the emerging state of the field and our deliberate decision to exclude studies relying on subjective productivity measures, which capture a fundamentally different construct [69]. We therefore reduced the number of covariates and relied on univariate meta-regressions, where the risk of overfitting would outweigh interpretive gains. 22

Third, as preregistered, we included both randomized and non-randomized studies, which relies on the assumption that both are comparable [70]. Further, the actual GenAI use was often not controlled for, which implies that the study design captures treatment assignment, not adherence to GenAI use. Additionally, factors such as repository size, coordination overhead, legacy code, quality assurance processes, and the specific mode of interaction with GenAI might be relevant characteristics for productivity and learning outcomes, yet remain largely unobserved in the available evidence. Recommendations for future research Our literature synthesis also highlights impactful opportunities for future research. For productivity (RQ1), more emphasis should be placed on field studies conducted under realistic development conditions. In contrast to controlled laboratory settings, professional software development involves large code bases with legacy code, coordination overhead, and strict quality and maintainability requirements. Studying GenAI-assisted coding in such environments is important to better assess the external validity of productivity effects and to explain the gap between laboratory and real-world findings observed in our meta-analysis. Such analyses should explicitly account for key confounding factors, including team structure, project complexity, and organizational processes. Second, future research should investigate the interaction formats and how to implement GenAI tools in systematic comparisons across different GenAI technologies, including reasoning-capable GenAI, agentic systems, and emerging practices such as “vibe coding”, to better understand how GenAI capabilities and usage patterns influence productivity. Third, future research would benefit from a deeper understanding of developer heterogeneity. Separating expertise (e.g., novices, junior and senior roles) from task type (e.g., code review, quality assurance, and maintenance) would help resolve this, while also clarifying whether GenAI primarily accelerates routine work or enables developers to reallocate attention to higher-order activities such as architectural design and code review [71]. 23

Similarly, for learning outcomes (RQ2), future research should focus on how GenAI tools can be leveraged as effective tutoring systems for programming skill acquisition. In particular, studies should investigate the mechanisms and boundary conditions through which GenAI assistance influences learning. Experimental designs that systematically vary how GenAI outputs are presented, such as providing full solutions versus guardrails that present structured hints or constructive feedback [44], could help distinguish when GenAI encourages superficial problem solving and when it supports conceptual understanding. Moreover, research should examine how GenAI assistance can be adaptively designed to meet the needs of users across different expertise levels, from novice programmers to experienced developers, accounting for individual differences in learning styles and user needs. Finally, longitudinal studies are needed to assess the long-term effects of GenAI assistance on learning, knowledge retention, and skill development, including potential risks from deskilling after the adoption of GenAI by experienced developers. GenAI is driving a major transformation in software development by shifting work from manually writing code toward reviewing and integrating AI-generated output. As these tools become increasingly embedded in software development workflows, a key tension emerges: productivity gains do not necessarily translate into stronger human skill development and may, in some cases, undermine it. Addressing this tension is important for computer science education, given that effective human oversight will become a key skill in GenAI-enabled programming.

Methods Search strategy Our data collection process follows the PRISMA 2020 guidelines [72] for systematic literature reviews. The meta-analysis was preregistered on December 4, 2025 using the PRISMA-P protocol (see https://osf.io/5h2s3). We operationalized productivity and learning as follows to ensure consistency across the metaanalysis: (1) Productivity was defined as the efficiency of code production [23, 47, 73]. In the included studies, this was measured via task completion time in a controlled laboratory experiment, the number of commits, or the number of merged pull requests. (2) Learning was defined as the measurable acquisition of programming skills. In the context of software engineering education, this is primarily measured via exam scores following a learning phase with or without GenAI support [50, 74]. We performed a systematic search across four major electronic databases commonly used to disseminate research findings in computer science and software engineering: (i) ACM Digital Library, (ii) arXiv, (iii) Scopus, and (iv) Web of Science. To account for the rapid evolution of generative AI research, our search included both peer-reviewed publications and preprints to capture emerging research that may not yet be formally published in peer-reviewed conferences or journals. In addition, we performed backward and forward citation searches for all included studies to identify additional relevant publications not retrieved by the initial search query. The search was restricted to publications in English language released between January 1, 2019, and December 5, 2025, which includes the public release of GenAI-based coding assistants such as GitHub Copilot but also research based on earlier large language models such as GPT-2. We formulated two search queries to identify studies examining the effects of GenAI assistance on (1) developer productivity and (2) learning outcomes. Both queries were structured around three

building blocks that filtered studies along a specific dimension: (i) the technology (i.e., GenAI, large language models, . . . ), (ii) the application context (i.e., programming, software engineering, . . . ), and (iii) the outcome of interest, namely, either productivity or learning. Within each building block, we used a set of synonyms, including common abbreviations, to ensure that the search is sufficiently broad to cover relevant studies. The formulation was inspired by prior research aimed at analyzing the effects of GenAI on human behavior in other contexts [75, 76] but was tailored to programming. The boolean search strings are shown below, while the exact versions adjusted to the specific databases are provided in Online Supplement A. Search query (Productivity) TITLE("GitHub Copilot" OR "Copilot" OR "CodeWhisperer" OR "Gemini" OR "Claude" OR "ChatGPT" OR "Large Language Model" OR "LLM" OR "AI" OR "Generative AI" OR "GenAI" OR "AI programming assistant" OR "programming tool" OR "AI code assistant") AND TITLE-ABS-KEY("developer" OR "software engineer" OR "programmer" OR "software development") AND TITLE-ABS-KEY("productivity" OR "efficiency" OR "experience" OR "behavior" OR "human-AI collaboration")

Search query (Learning) TITLE-ABS-KEY("GitHub Copilot" OR "Copilot" OR "CodeWhisperer" OR "Gemini" OR "Claude" OR "GPT" OR "ChatGPT" OR "Large Language Model" OR "LLM" OR "LLMs" OR "programming assistant" OR "code assistant" OR "code completion" OR "AI-assisted" OR "Chatbot") AND (("student*" OR "novice*" OR "learner*" OR "junior*") AND ("developer*" OR "programmer*"))

Eligible studies had to meet the following inclusion criteria: (i) the study compared human performance with GenAI assistance against a control condition without GenAI assistance or a valid alternative baseline; (ii) the study reported empirical, quantitative results; (iii) the study measured at least one relevant outcome variable related to developer productivity (e.g., task completion time, output volume) or learning (e.g., test or exam scores); and (iv) the study provided sufficient statistical information to compute standardized effect sizes (e.g., means, standard deviations, sample sizes), or such information could be reliably gathered from supplementary materials. Screening 26

and eligibility assessment were conducted by the first author. The inclusion flowchart is shown in Fig. 9. Our database search yielded a total of n = 10,115 records from ACM (n = 342), arXiv (n = 3,149), Scopus (n = 5,596), and Web of Science (n = 1,028). We identified duplicate records based on (i) the DOI and (ii) a combination of the first author’s surname and the title. For the latter, we used a normalized format where all special characters and whitespace were removed prior to comparison. After removing n = 1,690 duplicate records, n = 8,425 records remained for title and abstract screening. Of these, n = 8,204 records were excluded due to a lack of relevance to the topic of our meta-analysis. The full texts of the remaining n = 221 reports were sought for retrieval, of which n = 17 could not be accessed, leaving n = 204 reports for eligibility assessment. During this stage, n = 135 reports were excluded due to insufficient study design, n = 25 due to insufficient statistical reporting, and n = 27 because the study addressed a different domain, totaling to n = 187 ineligible reports. When studies were otherwise eligible but lacked sufficient statistical information, we contacted the corresponding authors via email to request the necessary data. None provided the information needed for inclusion. Consequently, n = 17 studies retrieved from the four databases met our inclusion criteria. To minimize the risk of missing relevant works, we additionally conducted citation-based searches (forward and backward) for the studies retrieved from the databases, which identified n = 71 further records. All retrieved studies were assessed for eligibility, resulting in the exclusion of n = 65 studies, either due to inappropriate study design (n = 43), insufficient statistics (n = 4), or an incorrect domain (n = 18). In total, n = 23 studies met all inclusion criteria and were retained for the meta-analysis. Of these, n = 13 studies examined only productivity outcomes, n = 9 focused solely on learning outcomes, and n = 1 study reported on both types of outcomes.

Records identified from: ACM (n = 342) arXiv (n = 3149) Scopus (n = 5596) Web of Science (n = 1028)

Records removed before screening: Duplicate records removed (n = 1690)

Identification of studies via other methods

Records identified from: Citation searching (n = 71)

Total (n = 10115)

Records screened (n = 8425)

Records excluded (n = 8204)

Reports sought for retrieval (n = 221)

Reports not retrieved (n = 17)

Reports sought for retrieval (n = 71)

Reports not retrieved (n = 0)

Reports assessed for eligibility (n = 204)

Reports excluded: Inappropiate study design (n = 135) Insufficient statistics (n = 25) Wrong domain (n = 27)

Reports assessed for eligibility (n = 71)

Reports excluded: Inappropiate study design (n = 43) Insufficient statistics (n = 4) Wrong domain (n = 18)

Total (n = 187)

Included

Screening

Identification

Identification of studies via databases and registers

Total (n = 65)

Studies included in review (n = 23) Productivity (n = 14) Learning (n = 10) (1 Study in both concepts)

Figure 9: Study inclusion flowchart. The diagram shows the study selection process following the PRISMA 2020 guidelines [72].

Data collection All studies that met the predefined inclusion criteria (n = 23) were manually transferred into a spreadsheet. Data extraction was performed by the first and the second author. The lists of included studies for productivity and learning are provided in Table 1 and Table 2, respectively. The complete dataset with the extracted statistics is available in our GitHub repository as supplementary material (see the data and code availability statement). For each study, we extracted metadata (e.g., author, title, publication year, publication stage), details about the experimental design (e.g., sample size, study design, choice of GenAI assistant, programming language), and outcome statistics (e.g., mean, standard deviation, t-value, F -value, standardized β). The variables were extracted to explore heterogeneity across different contexts (see Online Supplement E). Statistical analysis To compare the statistical results across the included studies, we computed effect sizes using Hedges’ g. We aimed to extract Cohen’s d directly from the studies where possible, yet most studies only reported other statistical measures (means, standard deviations, sample sizes, t-values, F -values), based on which we computed Cohen’s d using established conversion formulas (see Online Supplement D). For studies employing a pre-post-control group design, we computed the effect size dppc2 as recommended by Morris [77], defined as the difference in mean pre-post change between the treatment and control groups, divided by the pooled pretest standard deviation. To integrate effect sizes from studies using within-subject designs with those using between-subject designs, we followed the framework of Morris and DeShon [78]. Where the pre-post correlation was not reported, we assumed r = 0.5, a conservative default consistent with recommendations in the literature [78]. For the detailed conversion formulae, see Online Supplement D. The biascorrection factor J was applied to all estimates to mitigate small-sample bias [79]. In the main paper, all point estimates are reported with their corresponding 95% confidence interval (CI) and

standard error (SE). When studies reported multiple objective outcome measures within the same outcome category, we selected a single representative effect size per study to preserve statistical independence. For productivity outcomes, we prioritized commit-based measures, as these reflect discrete, externallyverifiable metrics that measure the contribution to software development. Measures based on lines of code were omitted whenever alternatives were available, as such measures may be inflated by redundant code generated by GenAI (e.g., comments or other form of in-code documentation) and tend to be less directly tied to meaningful development output. For learning outcomes, most studies reported test-based performance measures. When both intermediate assessments (e.g., midterms) and final exams were available, we selected the final exam score, as these represent a more comprehensive assessment of learning over the full instructional period. In line with Cochrane’s recommendations [80], we employed a random-effects model, as the included studies vary considerably in their design, context, and methodology, making the assumption of a single common effect implausible. We estimated the pooled effect sizes using the Restricted Maximum Likelihood (REML) estimator [81] and 95% CIs, which represent the expected range of true effects. Separate random-effects models were fitted for productivity-related outcomes and learning-related outcomes. Between-study heterogeneity was assessed using Cochran’s Q test, the I 2 statistic to estimate the proportion of total variance attributable to heterogeneity, and the estimated between-study variance (τ 2 ) [70]. We conducted subgroup analyses and meta-regressions to explore systematic sources of heterogeneity from moderators such as the study setting, the programming language, etc. The moderators differed between productivity and learning, reflecting the specific context and available data in each domain. All data processing and statistical modeling were performed in R (version 4.5.1) using the metafor package (version 4.8.0) [82]. The data and code to replicate our analyses are available via our GitHub repository (see the data and code availability statement). 30

Robustness checks We performed several robustness checks following best practice. First, we performed leave-oneout sensitivity analyses, in which the meta-analysis was re-estimated repeatedly while sequentially excluding one study at a time. This procedure evaluates whether the overall results are driven by any single study and helps identify influential studies that disproportionately affect the pooled effect size or heterogeneity. In addition, we computed influence diagnostics following established procedures [83]. Potential publication bias was evaluated using funnel plot asymmetry and Egger’s regression test [51], conditional on a minimum of ten effect sizes per analysis, following Cochrane’s guidance [80]. We applied the trim-and-fill procedure [52] to estimate the number and impact of potentially missing studies due to selective reporting. To address the statistical non-independence introduced by multiple effect sizes drawn from the same study [84], we additionally estimated a three-level meta-analytic model with effect sizes nested within studies (see Table 5 in Online Supplement B). Study quality We evaluated study quality by assessing the risk of bias of each effect size individually. Given the diverse range of study designs included in this meta-analysis, a single risk-of-bias tool would not have adequately captured the methodological variability across studies. We therefore selected the most appropriate tool for each study design: ROBINS-I for non-randomized studies (n = 9), and the appropriate RoB-2 variant for randomized studies, choosing between the individually randomized parallel group trials (n = 11), crossover (n = 5), and cluster (n = 1) versions depending on the specific design (see Tables 1 and 2). This ensured that the risk-of-bias evaluation was appropriately adapted to the methodological characteristics of each study design. In our case, two raters independently assessed the risk of bias of each study by answering the corresponding questions and building an algorithmic overall evaluation. Differences were then

discussed and a consensus reached. We used the risk of bias ratings to evaluate whether preprints, that are not yet peer-reviewed, account for a higher portion in high risk studies. Second, we categorized studies as either lower risk (low or some concerns) or higher risk (high or critical) and conducted a moderator analysis to evaluate whether risk-of-bias level was associated with differences in effect size estimates. Across both productivity and learning outcomes, the overall study quality was mixed (see Online Supplement C). Of all effect sizes, 59.3% were rated as having a higher risk of bias, whereas 40.7% had a lower risk of bias. Common issues were deviations from intended interventions for randomized studies, within-subject design studies that did not account for carry-over effects, and the widespread absence of preregistration. We conducted a univariate moderation analysis to check whether the overall risk of bias moderates the effect (see Online Supplement B). Risk of bias was not a statistically significant moderator for either productivity (QM = 2.90, p = 0.089) or learning (QM = 1.15, p = 0.284), indicating that the findings are robust regardless of study quality. Descriptively, however, higher risk studies tended to report larger effect sizes than lower risk studies for both outcomes.

ID P001

RID R001

P001

R002

P002 P003

R001 R001

P004

R001

P005 P006 P007

R001 R001 R001

P008 P009 P010 P010 P011 P012 P013

R001 R001 R001 R002 R001 R001 R001

P014

R001

Study Yeverechyahu et al. [85] Yeverechyahu et al. [85] Weber et al. [49] Paradis et al. [48] Kazemitabaar et al. [50] Nam et al. [25] Shihab et al. [54] Piscitelli et al. [60] Becker et al. [28] Xu et al. [27] Cui et al. [47] Cui et al. [47] Xu et al. [86] He et al. [87] Gambacorta et al. [71] Peng et al. [23]

Lang. Python

GenAI Asst. Copilot

Base LLM GPT-3

Setting Open-source

Outcome Commits

Unit Count/Qtr

Peer-Reviewed No

Bias Tool ROBINS-I

Overall Bias Moderate

Rust

Copilot

GPT-3

Open-source

Commits

Count/Qtr

ROBINS-I

Moderate

Python C++

Copilot –

GPT-3 –

Laboratory Enterprise

Time Time

Seconds Minutes

Yes No

RoB 2 (co) RoB 2 (IRPG)

High Some concerns

Python

Codex

Davinci

Laboratory

Time

Seconds

Yes

RoB 2 (IRPG)

High

Python Mixed JS

GILT Copilot ClueBot

3.5-turbo – GPT-4o

Laboratory Laboratory Laboratory

Time Time Time

Sec/Task Seconds Seconds

Yes Yes Yes

RoB 2 (co) RoB 2 (co) RoB 2 (co)

High High High

Python Mixed Mixed Mixed Python Mixed Mixed

Cursor Copilot Copilot Copilot NL2Code Cursor CodeFuse

Sonnet GPT-3 GPT-3 GPT-3 – – –

Open-source Open-source Enterprise Enterprise Laboratory Open-source Enterprise

Time Commits Commits Commits Time Commits Code Output

% Change Log Count Count/Wk Count/Wk Seconds Log Count Lines of Code

No No No No Yes No No

RoB 2 (IRPG) ROBINS-I RoB 2 (IRPG) RoB 2 (IRPG) RoB 2 (co) ROBINS-I ROBINS-I

Some concerns Moderate High High High Serious Moderate

JavaScript

Copilot

Codex

Laboratory

Time

Minutes

RoB 2 (IRPG)

High

Table 1: Overview of included studies related to productivity. Note: ID = Unique identifier for each study; RID = Unique identifier for each effect size estimate derived from the study. Bias Tool: RoB 2 = Revised Cochrane Risk-of-Bias tool (IRPG = individually randomized parallel-group; co = crossover trial); ROBINS-I = Risk Of Bias In Non-randomised Studies – Interventions. Overall Bias = Overall risk-of-bias judgment.

ID L001 L002 L003 L004 L005 L006 L007 L007 L008 L009 L010

RID R001 R001 R001 R001 R001 R001 R001 R002 R001 R001 R001

Study Kosar et al. [88] Choi & Kim [74] Sun et al. [89] Abdulla et al. [90] Johnson et al. [91] Yang et al. [92] Suh et al. [93] Suh et al. [93] Kazemitabaar et al. [50] Fan et al. [94] Tang et al. [95]

Lang. C++ Python Python C# C++ C++ Python Python Python Java Python

Setting Course Course Course Course Laboratory Course Course Course Laboratory Course Course

Dur. >10w <10w <10w <10w <10w <10w >10w >10w <10w >10w <10w

Exam No GenAI No GenAI GenAI allowed GenAI allowed No GenAI No GenAI No GenAI No GenAI No GenAI GenAI allowed No GenAI

Outcome Exam performance Exam performance Exam performance Exam performance Exam performance Exam performance Exam performance Exam performance Exam performance Exam performance Exam performance

Unit 0-100 0-100 0-100 0-100 0-100 0-100 0-10 0-10 0-100 0-100 n.s.

Bias Tool RoB 2 (IRPG) ROBINS-I RoB 2 (IRPG) ROBINS-I RoB 2 (IRPG) ROBINS-I RoB 2 (IRPG) RoB 2 (IRPG) RoB 2 (IRPG) RoB 2 (cluster) ROBINS-I

Overall Bias High Moderate High Critical High Moderate Some concerns Some concerns High High Moderate

Table 2: Overview of included studies related to learning. Note: ID = Unique identifier for each study; RID = Unique identifier for each effect size estimate derived from the study. Setting: Course = Course-integrated. Bias Tool: RoB 2 = Revised Cochrane Risk-of-Bias tool (IRPG = individually randomized parallel-group; cluster = clusterrandomized); ROBINS-I = Risk Of Bias In Non-randomised Studies – Interventions. Overall Bias = Overall risk-of-bias judgment.

Author contributions S.M. and M.G. contributed equally to this work. S.M. and S.F. had the idea. M.G. conducted the initial literature search. S.M. and M.G. performed study screening, risk-of-bias assessment, and statistical analyses. All authors contributed to conceptualization, manuscript writing, and approved the final manuscript.

Funding S.F. acknowledges funding via the Swiss National Science Foundation (SNSF), Grants 197485 and 186932.

Competing interests The authors declare no competing interests.

Data and code availability The data and the code to replicate our analyses are available via our Git repository at https://github. com/SM2982/MetaanalysisGenAICoding.git. The repository also includes the PRISMA checklist.

References [1] GitHub Copilot - Your AI pair programmer (2025). URL https://github.com/features/copilot. [2] Cursor: The best way to code with AI (2025). URL https://cursor.com. [3] Banh, L., Holldack, F. & Strobel, G. Copiloting the future: How generative AI transforms software engineering. Information and Software Technology 183, 107751 (2025). [4] Russo, D. et al. Generative AI in software engineering must be human-centered: The Copenhagen manifesto. Journal of Systems and Software 216, 112115 (2024). [5] Feuerriegel, S., Hartmann, J., Janiesch, C. & Zschech, P. Generative AI. Business & information systems engineering 66, 111–126 (2024). [6] Daniotti, S., Wachs, J., Feng, X. & Neffke, F. Who is using AI to code? Global diffusion and impact of generative AI. Science 391, 831–835 (2026). [7] Stack Overflow.

Stack Overflow developer survey 2025 (2025).

URL https://survey.

stackoverflow.co/2025/. [8] Qiu, K., Puccinelli, N., Ciniselli, M. & Di Grazia, L. From today’s code to tomorrow’s symphony: The AI transformation of developer’s routine by 2030 (2024). URL https://arxiv. org/abs/2405.12731. [9] Ulfsnes, R., Moe, N. B., Stray, V. & Skarpen, M. Transforming software development with generative AI: Empirical insights on collaboration and workflow (2024). URL https://arxiv. org/abs/2405.01543. [10] Otten, D., Stalnaker, T., Wintersgill, N., Chaparro, O. & Poshyvanyk, D.

Prompting

in practice: Investigating software developers’ use of generative AI tools (2025). URL https://arxiv.org/abs/2510.06000. 36

[11] Jain, N. et al. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974 (2024). [12] DeepSeek-AI et al. DeepSeek-V3.2: pushing the frontier of open large language models (2025). URL http://arxiv.org/abs/2512.02556. [13] Gemini 3: Introducing the latest Gemini AI model from Google (2025). URL https://blog. google/products/gemini/gemini-3/#gemini-3. [14] Quan, S. et al. Codeelo: Benchmarking competition-level code generation of llms with human-comparable elo ratings. arXiv preprint arXiv:2501.01257 (2025). [15] Chen, M. et al.

Evaluating large language models trained on code.

arXiv preprint

arXiv:2107.03374 (2021). [16] Ge, Y. et al. A survey of vibe Coding with large language models (2025). URL https: //arxiv.org/abs/2510.12399. [17] Pimenova, V., Fakhoury, S., Bird, C., Storey, M.-A. & Endres, M. Good vibrations? A qualitative study of co-creation, communication, flow, and trust in Vibe Coding (2025). URL https://arxiv.org/abs/2509.12491. [18] Fawzy, A., Tahir, A. & Blincoe, K. Vibe Coding in practice: motivations, challenges, and a future outlook - A grey literature review (2025). URL https://arxiv.org/abs/2510.00328. [19] Russo, D. Navigating the complexity of generative AI adoption in software engineering. ACM Transactions on Software Engineering and Methodology 33, 1–50 (2024). [20] Weisz, J. D. et al. Examining the use and impact of an AI code assistant on developer productivity and experience in the enterprise. In Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, 1–13 (ACM, Yokohama Japan, 2025). 37

[21] Lyu, Y. et al. "my productivity is boosted, but ...": Demystifying users’ perception on AI coding assistants (2025). URL https://arxiv.org/abs/2508.12285. [22] Wivestad, V. T., Barbala, A. & Stray, V. Copilot’s island of joy: Balancing individual satisfaction with team interaction in agile development. In Marchesi, L. et al. (eds.) Agile Processes in Software Engineering and Extreme Programming – Workshops, 524, 123–129 (Springer Nature Switzerland, Cham, 2025). [23] Peng, S., Kalliamvakou, E., Cihon, P. & Demirer, M. The impact of AI on developer productivity: Evidence from GitHub Copilot (2023). URL https://arxiv.org/abs/2302.06590. [24] Prather, J. et al. The widening gap: The benefits and harms of generative AI for novice programmers. In Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1, 469–486 (ACM, Melbourne VIC Australia, 2024). [25] Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B. & Myers, B. Using an LLM to help with code understanding. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, 1–13 (ACM, Lisbon Portugal, 2024). [26] Barke, S., James, M. B. & Polikarpova, N. Grounded Copilot: How Programmers Interact with Code-Generating Models. Proceedings of the ACM on Programming Languages 7, 85– 111 (2023). [27] Xu, F., Medappa, P. K., Tunc, M. M., Vroegindeweij, M. & Fransoo, J. C. AI-assisted programming may decrease the productivity of experienced developers by increasing maintenance burden (2025). URL https://arxiv.org/abs/2510.10165. [28] Becker, J., Rush, N., Barnes, E. & Rein, D. Measuring the impact of early-2025 AI on experienced open-source developer productivity (2025). URL http://arxiv.org/abs/2507.09089v2.

[29] Liang, J. T., Yang, C. & Myers, B. A. A large-scale survey on the usability of AI programming assistants: Successes and challenges. In Proceedings of the IEEE/ACM 46th international conference on software engineering, Icse ’24 (Association for Computing Machinery, Lisbon, Portugal, 2024). [30] Perry, N., Srivastava, M., Kumar, D. & Boneh, D. Do users write more insecure code with ai assistants? In Proceedings of the 2023 ACM SIGSAC conference on computer and communications security, 2785–2799 (2023). [31] Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B. & Karri, R. Asleep at the keyboard? assessing the security of github copilot’s code contributions. Communications of the ACM 68, 96–105 (2025). [32] Yetistiren, B., Ozsoy, I. & Tuzun, E. Assessing the quality of GitHub Copilot’s code generation. In Proceedings of the 18th International Conference on Predictive Models and Data Analytics in Software Engineering, 62–71 (ACM, Singapore Singapore, 2022). [33] Kirova, V. D., Ku, C. S., Laracy, J. R. & Marlowe, T. J. Software engineering education must adapt and evolve for an LLM environment. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1, 666–672 (ACM, Portland OR USA, 2024). [34] Lau, S. & Guo, P. From "ban it till we understand it" to "resistance is futile": How university programming instructors plan to adapt as more students use AI code generation and explanation tools such as ChatGPT and GitHub Copilot. In Proceedings of the 2023 ACM Conference on International Computing Education Research V.1, 106–121 (ACM, Chicago IL USA, 2023). [35] Aviv, I., Leiba, M., Rika, H. & Shani, Y. The impact of ChatGPT on students’ learning programming languages. In Zaphiris, P. & Ioannou, A. (eds.) Learning and Collaboration Technologies, 14724, 207–219 (Springer Nature Switzerland, Cham, 2024). 39

[36] Leinonen, J. et al. Comparing code explanations created by students and large language models. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1, 124–130 (ACM, Turku Finland, 2023). [37] Laato, S., Morschheuser, B., Hamari, J. & Björne, J. AI-assisted learning with ChatGPT and large language models: Implications for higher education. In 2023 IEEE International Conference on Advanced Learning Technologies (ICALT), 226–230 (IEEE, Orem, UT, USA, 2023). [38] Becker, B. A. et al. Programming is hard - or at least it used to be: Educational opportunities and challenges of AI code generation. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1, 500–506 (ACM, Toronto ON Canada, 2023). [39] Güner, H. & Er, E. AI in the classroom: Exploring students’ interaction with ChatGPT in programming learning. Education and Information Technologies 30, 12681–12707 (2025). [40] Lee, H.-P. H. et al. The impact of generative AI on critical thinking: Self-reported reductions in cognitive effort and confidence effects from a survey of knowledge workers. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, 1–22 (ACM, Yokohama Japan, 2025). [41] Vivian, R. Coding with ChatGPT: empirical Evidence of cognitive offloading in computer science education. Clareus Scientific Science and Engineering 2, 10–19 (2025). [42] Jose, B. et al. The cognitive paradox of AI in education: Between enhancement and erosion. Frontiers in Psychology 16, 1550621 (2025). [43] Rahe, C. & Maalej, W. How do programming students use generative AI? Proceedings of the ACM on Software Engineering 2, 978–1000 (2025).

[44] Bastani, H. et al. Generative AI without guardrails can harm learning: Evidence from high school mathematics. Proceedings of the National Academy of Sciences 122, e2422633122 (2025). [45] Wiles, E. et al. GenAI as an Exoskeleton: Experimental evidence on knowledge workers using GenAI on new skills. Available at SSRN 4944588 (2024). [46] Shen, J. H. & Tamkin, A. How AI Impacts Skill Formation (2026). URL http://arxiv.org/abs/ 2601.20245. [47] Cui, Z. et al. The effects of generative AI on high skilled work: Evidence from three field experiments with software developers (2024). URL https://www.ssrn.com/abstract=4945566. [48] Paradis, E. et al. How much does AI impact development speed? An enterprise-based randomized controlled trial (2024). URL https://arxiv.org/abs/2410.12944. [49] Weber, T., Brandmaier, M., Schmidt, A. & Mayer, S. Significant productivity gains through programming with large language models. Proceedings of the ACM on Human-Computer Interaction 8, 1–29 (2024). [50] Kazemitabaar, M. et al. Studying the effect of AI code generators on supporting novice learners in introductory programming. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 1–23 (ACM, Hamburg Germany, 2023). [51] Egger, M., Smith, G. D., Schneider, M. & Minder, C. Bias in meta-analysis detected by a simple, graphical test. BMJ (Clinical research ed.) 315, 629–634 (1997). [52] Duval, S. & Tweedie, R. Trim and Fill: A simple funnel-plot–based method of testing and adjusting for publication bias in meta-analysis. Biometrics. Journal of the International Biometric Society 56, 455–463 (2000).

[53] Jimenez, C. E. et al. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770 (2023). [54] Shihab, M. I. H. et al. The effects of GitHub copilot on computing students’ programming effectiveness, efficiency, and processes in brownfield coding tasks. In Proceedings of the 2025 ACM Conference on International Computing Education Research V.1, ICER ’25, 407–420 (Association for Computing Machinery, New York, NY, USA, 2025). [55] Simkute, A. et al. Ironies of generative AI: Understanding and mitigating productivity loss in human-AI interactions (2024). URL https://arxiv.org/abs/2402.11364. [56] Jin, K., Wang, C.-Y., Pham, H. V. & Hemmati, H. Can ChatGPT support developers? An empirical evaluation of large language models for code generation (2024). [57] Bird, C. et al. Taking flight with Copilot. Communications of the ACM 66, 56–62 (2023). [58] Collins, K. M. et al. Building machines that learn and think with people. Nature Human Behaviour 8, 1851–1863 (2024). [59] Wang, Z. Z. et al. Position: Humans are missing from AI coding agent research . [60] Piscitelli, A., Costagliola, G., De Rosa, M. & Fuccella, V. Influence of large language models on programming assignments – A user study. In Proceedings of the 2024 16th International Conference on Education Technology and Computers, 33–38 (ACM, Porto Vlaams-Brabant Portugal, 2024). [61] Prather, J. et al. The robots are here: Navigating the generative AI revolution in computing education. In Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education, 108–159 (ACM, Turku Finland, 2023). [62] Acharya, V. Generative AI and the transformation of software development practices (2025). URL https://arxiv.org/abs/2510.10819. 42

[63] Bardach, A. & Murrah, H. Bridging the skills gap: A course model for modern generative AI education (2025). URL https://arxiv.org/abs/2511.11757. [64] Beale, R. Computer science education in the age of generative AI (2025). URL https://arxiv. org/abs/2507.02183. [65] Gu, K., Shang, R., Althoff, T., Wang, C. & Drucker, S. M. How Do Analysts Understand and Verify AI-Assisted Data Analyses? In Proceedings of the CHI Conference on Human Factors in Computing Systems, 1–22 (ACM, Honolulu HI USA, 2024). [66] Hoffmann, M., Boysel, S., Nagle, F., Peng, S. & Xu, K. Generative AI and the nature of work. Harvard Business School Strategy Unit Working Paper 25–021 (2025). [67] Ma, B. et al. Scaffolding metacognition in programming education: Understanding studentAI interactions and design implications (2025). URL https://arxiv.org/abs/2511.04144. [68] Denny, P. et al. Computing education in the era of generative AI. Communications of the ACM 67, 56–67 (2024). [69] Meyer, A. N., Fritz, T., Murphy, G. C. & Zimmermann, T. Software developers’ perceptions of productivity. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, 19–29 (ACM, Hong Kong China, 2014). [70] Borenstein, M., Hedges, L. V., Higgins, J. P. T. & Rothstein, H. R. Introduction to metaanalysis (Wiley, 2009), 1 edn. [71] Gambacorta, L., Qiu, H., Shan, S. & Rees, D. M. Generative AI and labour productivity: a field experiment on coding, 1208 (Bank for International Settlements, Monetary and Economic Department, 2024). [72] Page, M. J. et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ (Clinical research ed.) n71 (2021). 43

[73] Ziegler, A. et al. Productivity assessment of neural code completion. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, 21–29 (ACM, San Diego CA USA, 2022). [74] Choi, S. & Kim, H. The impact of a large language model-based programming learning environment on students’ motivation and programming ability. Education and Information Technologies 30, 8109–8138 (2025). [75] Hölbling, L., Maier, S. & Feuerriegel, S. A meta-analysis of the persuasive power of large language models. Scientific Reports 15, 43818 (2025). [76] Holzner, N., Maier, S. & Feuerriegel, S. Generative AI and creativity: a systematic literature review and meta-analysis (2025). URL https://arxiv.org/abs/2505.17241. [77] Morris, S. B. Estimating Effect Sizes From Pretest-Posttest-Control Group Designs. Organizational Research Methods 11, 364–386 (2008). [78] Morris, S. B. & DeShon, R. P. Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. Psychological Methods 7, 105–125 (2002). [79] Hedges, L. V. Distribution theory for Glass’s estimator of effect size and related estimators. Journal of Educational Statistics 6, 107–128 (1981). [80] Higgins, J. P. et al. Cochrane handbook for systematic reviews of interventions (Cochrane, 2024), version 6.5 (updated august 2024) edn. [81] Viechtbauer, W. Bias and efficiency of meta-analytic variance estimators in the randomeffects model. Journal of Educational and Behavioral Statistics 30, 261–293 (2005). [82] Viechtbauer, W. Conducting meta-analyses in R with the metafor package. Journal of Statistical Software 36 (2010).

[83] Viechtbauer, W. & Cheung, M. W.-L. Outlier and influence diagnostics for meta-analysis. Research Synthesis Methods 1, 112–125 (2010). [84] Cheung, M. W.-L. Modeling dependent effect sizes with three-level meta-analyses: a structural equation modeling approach. Psychological methods 19, 211 (2014). [85] Yeverechyahu, D., Mayya, R. & Oestreicher-Singer, G. The impact of large language models on open-source innovation: Evidence from GitHub Copilot (2024). URL https://arxiv.org/ abs/2409.08379. [86] Xu, F. F., Vasilescu, B. & Neubig, G. In-IDE code generation from natural language: Promise and challenges. ACM Transactions on Software Engineering and Methodology 31, 1–47 (2022). [87] He, H., Miller, C., Agarwal, S., Kästner, C. & Vasilescu, B. Does AI-assisted coding deliver? A difference-in-differences study of Cursor’s impact on software projects (2025). URL https: //arxiv.org/abs/2511.04427. [88] Kosar, T., Ostojić, D., Liu, Y. D. & Mernik, M. Computer science education in ChatGPT era: Experiences from an experiment in a programming course for novice programmers. Mathematics 12, 629 (2024). [89] Sun, D., Boudouaia, A., Zhu, C. & Li, Y. Would ChatGPT-facilitated programming mode impact college students’ programming behaviors, performances, and perceptions? An empirical study. International Journal of Educational Technology in Higher Education 21, 14 (2024). [90] Abdulla, S., Ismail, S., Fawzy, Y. & Elhag, A. Using ChatGPT in teaching computer programming and studying its impact on students performance. Electronic Journal of e-Learning 22, 66–81 (2024). 45

[91] Johnson, D. M., Doss, W. & Estepp, C. M. Using ChatGPT with novice Arduino programmers: Effects on performance, interest, self-efficacy, and programming ability. Journal of Research in Technical Careers 8, 1 (2024). [92] Yang, T.-C., Hsu, Y.-C. & Wu, J.-Y. The effectiveness of ChatGPT in assisting high school students in programming learning: Evidence from a quasi-experimental research. Interactive Learning Environments 33, 3726–3743 (2025). [93] Suh, J., Lee, K. & Lee, J. Programming education with ChatGPT: outcomes for beginners and intermediate students. Education and Information Technologies 30, 19511–19536 (2025). [94] Fan, G., Liu, D., Zhang, R. & Pan, L. The impact of AI-assisted pair programming on student motivation, programming anxiety, collaborative learning, and programming performance: A comparative study with traditional pair programming and individual approaches. International Journal of STEM Education 12, 16 (2025). [95] Tang, B., Liang, J., Hu, W. & Luo, H. Enhancing Programming Performance, Learning Interest, and Self-Efficacy: The Role of Large Language Models in Middle School Education. Systems 13, 555 (2025).

Search Queries

Table 3: Search strings. Below are the search strings adapted for the different databases. Database

Topic

Query

Scopus

Productivity

TITLE("GitHub Copilot" OR "Copilot" OR "CodeWhisperer" OR "Gemini" OR "Claude" OR "ChatGPT" OR "Large Language Model" OR "LLM" OR "AI" OR "Generative AI" OR "GenAI" OR "AI programming assistant" OR "programming tool" OR "AI code assistant") AND TITLE-ABS-KEY("developer" OR "software engineer" OR "programmer" OR "software development") AND TITLE-ABS-KEY("productivity" OR "efficiency" OR "experience" OR "behavior" OR "human-AI collaboration")

Learning

TITLE-ABS-KEY("GitHub Copilot" OR "Copilot" OR "CodeWhisperer" OR "Gemini" OR "Claude" OR "GPT" OR "ChatGPT" OR "Large Language Model" OR "LLM" OR "LLMs" OR "programming assistant" OR "code assistant" OR "code completion" OR "AI-assisted" OR "Chatbot") AND (("student*" OR "novice*" OR "learner*" OR "junior*") AND ("developer*" OR "programmer*"))

Web of Productivity Science

ACM

arXiv

TI=("GitHub Copilot" OR "Copilot" OR "CodeWhisperer" OR "Gemini" OR "Claude" OR "ChatGPT" OR "Large Language Model" OR "LLM" OR "AI" OR "Generative AI" OR "GenAI" OR "AI programming assistant" OR "programming tool" OR "AI code assistant") AND TS=("developer" OR "software engineer" OR "programmer" OR "software development") AND TS=("productivity" OR "efficiency" OR "experience" OR "behavior" OR "human-AI collaboration")

Learning

TS=(("GitHub Copilot" OR "Copilot" OR "CodeWhisperer" OR "Gemini" OR "Claude" OR "GPT" OR "ChatGPT" OR "Large Language Model" OR "LLM" OR "LLMs" OR "programming assistant" OR "code assistant" OR "code completion" OR "AI-assisted" OR "Chatbot") AND (("student*" OR "novice*" OR "learner*" OR "junior*") AND ("developer*" OR "programmer*")))

Productivity

"query": { Title:("GitHub Copilot" OR "Copilot" OR "CodeWhisperer" OR "Gemini" OR "Claude" OR "ChatGPT" OR "Large Language Model" OR "LLM" OR "AI" OR "Generative AI" OR "GenAI" OR "AI programming assistant" OR "programming tool" OR "AI code assistant" OR "AI-powered programming") AND (Title:("developer" OR "software engineer" OR "programmer" OR "software development") OR Abstract:("developer" OR "software engineer" OR "programmer" OR "software development")) AND (Title:("productivity" OR "efficiency" OR "experience" OR "behavior" OR "human-AI collaboration") OR Abstract:("productivity" OR "efficiency" OR "experience" OR "behavior" OR "human-AI collaboration")) } "filter": { }

Learning

"query": { Title:("GitHub Copilot" OR "Copilot" OR "CodeWhisperer" OR "Gemini" OR "Claude" OR "ChatGPT" OR "Large Language Model" OR "LLM" OR "AI" OR "Generative AI" OR "GenAI" OR "AI programming assistant" OR "programming tool" OR "AI code assistant" OR "AI-powered programming") AND (Abstract:("developer" OR "software engineer" OR "programmer" OR "software development") OR Title:("developer" OR "software engineer" OR "programmer" OR "software development")) AND (Abstract:("productivity" OR "efficiency" OR "experience" OR "behavior" OR "human-AI collaboration") OR Title:("productivity" OR "efficiency" OR "experience" OR "behavior" OR "human-AI collaboration")) } "filter": { }

Productivity

Title:($Github Copilot$) OR Title: (Copilot OR Gemini OR Claude OR GPT OR Cursor OR LLM OR AI) AND Abstract: (developer OR programmer OR software development) AND Abstract: (productivity OR efficiency)

Learning

Title:($Github Copilot$) OR Title: (Copilot OR Gemini OR Claude OR GPT OR ChatGPT OR AI OR LLM OR Chatbot) AND Abstract: (student OR novice OR learner OR junior) AND Abstract: (developer OR programmer)

Robustness Checks

Table 4: Moderator analysis results for risk of bias across productivity and learning outcomes. Predicted effect sizes (Hedges’ g) with 95% confidence intervals derived from univariate mixed-effects meta-regressions (REML estimator). k = number of effect sizes per subgroup. QM and p reflect the omnibus moderator test. Productivity Subgroup

Risk of Bias Lower risk Higher risk

6 0.081 10 0.498

Learning QM

95% CI

[−0.287, 0.449] 2.90 0.089 [0.190, 0.805]

95% CI

5 6

−0.043 0.319

[−0.521, 0.434] [−0.140, 0.778]

1.15

0.284

Table 5: Multilevel meta-analysis accounting for effect size dependencies. Results from threelevel meta-analytic models with effect sizes (level 1) nested within studies (level 2), estimated via REML. σ12 = between-study variance; σ22 = within-study variance. Within-study variance is estimated at zero for both outcomes, confirming that the pooled estimates and conclusions from the main random-effects analysis remain robust when accounting for dependencies among effect sizes from the same study.

Outcome

Productivity 16 0.38 0.15 Learning 11 0.19 0.18 ∗∗∗

95% CI

σ12

σ22

[0.09, 0.67] 0.010 0.266 0.000 [−0.154, 0.54] 0.275 0.261 0.000

p < .001; k = number of effect sizes.

Q 206.06*** 54.96***

Risk of Bias

L001 / R001 L003 / R001 L005 / R001 L007 / R001

Learning

L007 / R002 L008 / R001 L009 / R001 P002 / R001 P003 / R001 P004 / R001 P005 / R001 P006 / R001 P007 / R001

Productivity

P008 / R001 P010 / R001 P010 / R002 P011 / R001 P014 / R001 ss

R 1:

s mi

p on ati

ta m ro da s f ns me on ntio i o t c t ia ve ou ev ter : D in ing D2 ded iss n M e : int D3

f to en me em tco r su ou ea he :M t

Judgment

Low

n cts tio ffe ly) ca ly) r e r on tifi r on e n v o ve ide ste rry so of (clu ca ros ing d & (c im o i T er b: :P D1 DS

the of ult on res i t lec ted Se or 5: rep

Some concerns

all er Ov

High

Figure 10: Risk-of-bias assessment of included studies using the revised Cochrane Risk of Bias tool (RoB 2). Each row represents a study–result pair, grouped by outcome domain (Learning and Productivity). Columns correspond to bias domains: D1: Randomization process; D2: Deviations from intended interventions; D3: Missing outcome data; D4: Measurement of the outcome; D5: Selection of the reported result; D1b: Timing of identification or recruitment of participants (cluster-randomized trials only); DS: Period and carryover effects (crossover trials only). Judgments are color-coded as low risk of bias (green), some concerns (yellow), or high risk of bias (red).

L002 / R001

L004 / R001 Learning

L006 / R001

L010 / R001

P001 / R001

P001 / R002

P009 / R001

Productivity

P012 / R001

P013 / R001 of on ts cti an ele ticip S 2: par

ing

: D1

n Co

fou

of on s ati ion ific vent s las ter : C in

m fro s ns ion tio vent a i ev ter : D in D4 ded n e t in

Judgment

Low

Moderate

ing ss Mi

f to en es em m ur utco s ea o

ta da

all er Ov

Serious

Critical

Figure 11: Risk-of-bias assessment of included non-randomized studies using the Risk Of Bias In Non-randomized Studies of Interventions (ROBINS-I) tool. Each row represents a study– result pair, grouped by outcome domain (Learning and Productivity). Columns correspond to bias domains: D1: Confounding; D2: Selection of participants; D3: Classification of interventions; D4: Deviations from intended interventions; D5: Missing data; D6: Measurement of outcomes. Judgments are color-coded as low risk of bias (green), moderate risk of bias (yellow), serious risk of bias (orange), or critical risk of bias (red).

Effect Size Conversion Formulas

All effect sizes were computed as Cohen’s d and subsequently converted to Hedges’ g to correct for small-sample bias [79]. For each effect size, the corresponding sampling variance v was computed and used as the basis for inverse-variance weighting for the meta-analytic model [70]. Five conversion paths were used, depending on the statistics reported in each primary study. Throughout, nT and nC denote the treatment and control group sample sizes, and zα/2 = 1.96 is the critical value for 95% confidence intervals. When only a standard error (SE) or confidence interval is reported, √ the standard deviation is recovered via SD = SE · n, with SE = (CIupper − CIlower )/(2 × 1.96) where applicable. D.1

Within-Subject: Standardised Mean Change (Raw-Score SD)

For single-group pretest–posttest designs, the effect size is the standardised mean change using the raw-score (pretest) standard deviation [78]

dSMCR =

X̄post − X̄pre SD pre

(1)

with sampling variance 2(1 − r) n − 1 v= n n−3

n d2SMCR d2 1+ − SMCR 2 , 2(1 − r) c(n − 1)

(2)

where r is the pre–post correlation (set to r = 0.5 when unreported; [78]) and c(df ) = 1 − 3 / (4 df − 1) is the Hedges bias function.

D.2

Between-Subject: Standardised Mean Difference

For independent-groups designs reporting group means and variation (SD, SE, or CI), the standardised mean difference [70] is computed: s

X̄T − X̄C d= , SD pool

SD pool =

(nT − 1) SD 2T + (nC − 1) SD 2C nT + n C − 2

(3)

with sampling variance v=

D.3

nT + n C d2 + nT nC 2(nT + nC )

(4)

Pre-Post-Control Group Design: dppc2 (Morris, 2008)

For studies with a pre-post-control group (PPC) design, dppc2 is used [77]. This estimator standardizes the difference-in-differences by the pooled pre-test standard deviation, which is unaffected by the treatment and therefore more consistent across studies. Where pre- and post-test instruments differed considerably in content or scoring (e.g., [95]), a post-test-only Hedges’ g was computed instead. The pooled pre-test SD is: s SD pool =

D.3.0.1

(nT − 1) SD 2pre,T + (nC − 1) SD 2pre,C nT + nC − 2

(5)

From raw pre/post means. When pretest and posttest means are available for both

groups: dppc2 =

D.3.0.2

(X̄post,T − X̄pre,T ) − (X̄post,C − X̄pre,C ) SD pool

(6)

From a percentage treatment effect. When a study reports the treatment effect as

a percentage of the control-group baseline (%eff ) together with the control-group baseline mean

(X̄base,C ), the raw difference is first recovered: ∆raw SD pool

(7)

1 d2ppc2 1 + + v = 2(1 − r) nT nC 2(nT + nC )

(8)

∆raw = %eff × X̄base,C ,

D.3.0.3

dppc2 =

Sampling variance (both variants).

where r is the pre–post correlation, set to r = 0.5 when unreported. D.4

Regression Coefficient to Standardized Effect

For observational panel studies whose identification relies on controls absorbed within the model (e.g., unit and time fixed effects in TWFE/DiD designs), we computed effect sizes from β̂ and the pooled standard deviation (SD pooled ) where available.

β̂ SD pooled

SE β̂ SD pooled

2 (9)

Note: When separate pretest SDs for treatment and control groups were unavailable, we used SD overall (pooled across all units and time periods). This yields a conservative effect size because SD overall includes between-unit variance that the fixed effects have already absorbed from the numerator [77].

Definitions of moderators

Moderator

Description

Extracted values

Study setting

The context in which the study was con-

Open-Source, Enterprise, Labora-

ducted.

tory

Levels: Laboratory, Open-Source, Enterprise GenAI interface

The specific IDE or interface provided

GitHub Copilot, Cursor Pro, Codex,

to participants.

GILT, ClueBot, not stated

Levels: GitHub Copilot, Other Programming language

Primary programming language of the

Python, C++, Rust, JavaScript,

task.

Mixed

Levels: Python, Other Participant level

Professional experience of participants.

Experienced, Mixed, Students

Developers contributing to open-source projects or working in an organizational context were classified as experienced, while the remaining studies involved student samples or mixed groups. Levels: Experienced, Students, Mixed Randomization

Whether treatment assignment was ran-

Randomized, Non-randomized

domized. Levels: Randomized (e.g. RCT), Nonrandomized (e.g. Quasi-Experiment)

Table 6: Moderator variables for productivity outcomes included in the meta-regression analyses, with the specific values extracted from the included studies. Two pre-registered moderators were excluded: LLM base model was omitted because model specifications varied substantially and several studies did not report the exact model; coding task type was excluded as no meaningful categorization across studies was feasible.

Moderator

Description

Extracted values

Exam environment

Whether GenAI tool use was permitted

GenAI allowed, No GenAI allowed

during the post-test assessment in the treatment condition. Levels: GenAI allowed, No GenAI GenAI interface

The specific IDE or interface provided

ChatGPT, Codex, Custom interface

to participants. Levels: ChatGPT, Other Programming language

Primary programming language of the

Python, C++, C#, Java

task. Levels: Python, Other Study duration

Length of the AI-assisted learning inter-

Short-term

weeks),

vention.

Semester-long (> 10 weeks)

Levels: <10 weeks, ≥10 weeks Participant educational

Studies involved learners at varying ed-

Undergraduate,

level

ucational stages, ranging from K-12

university (K-12, High School)

Graduate,

Pre-

and high school students to university undergraduates and graduates, which we collapsed into two levels. Levels: University, Pre-University Randomization

Whether treatment assignment was ran-

Randomized, Non-randomized

domized. Levels: Randomized, Non-randomized

Table 7: Moderator variables for learning outcomes included in the meta-regression analyses, with the specific values extracted from the included studies.

Record · ID 168399 · SHA-256 e255556dc74916bf
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.