arXiv CSopen access

Prompt Quality and Pull Request Outcomes: A Stage-Based Empirical Study of LLM-Assisted Development

Prompt Quality and Pull Request Outcomes: A Stage-Based Empirical Study of LLM-Assisted Development

arXiv:2606.19644v1 [cs.SE] 17 Jun 2026

Richard Sserunjogi, Daniel Ogenrwot, John Businge Department of Computer Science, University of Nevada Las Vegas, 4505 S. Maryland Pkwy., Las Vegas, 89154, Nevada, United States of America.

*Corresponding author(s). E-mail(s): [email protected]; Contributing authors: [email protected]; [email protected]; Abstract Large language model (LLM)-powered tools such as ChatGPT are increasingly used in collaborative software engineering workflows, yet little is known about how prompt structure influences downstream pull request (PR) outcomes. Prior studies primarily examine conversational helpfulness, productivity, or coarsegrained adoption metrics, leaving the role of prompt structure in collaborative integration behavior insufficiently understood. We analyze 265 manually validated developer–ChatGPT interactions derived from self-admitted ChatGPT usage in open-source pull requests. Building on prior research on developer-facing artifacts and prompt engineering, we operationalize prompt structure using three dimensions: Context, Specificity, and Verification. We first evaluate whether LLM-assisted annotation can reliably reproduce human judgments of prompt structure, finding substantial variation across prompt dimensions and workflow contexts. Specificity exhibits the most stable agreement with human judgment, whereas Context is systematically under-scored by the LLM and Verification remains difficult to assess consistently, motivating a hybrid human–LLM annotation strategy. Using this validated framework, we then examine how prompt structure influences actionable code generation, code adoption, and integration depth across AI-assisted PR workflows. Specificity and Context are most strongly associated with actionable code generation, Verification emerges as the primary predictor of code adoption, and integration depth is most strongly associated with Context, suggesting that deeper reuse depends on alignment with the surrounding implementation and project context. Overall, our findings show that prompt characteristics exert distinct stage-dependent effects across collaborative

1

AI-assisted software engineering workflows. The results suggest that prompt construction functions as an important upstream factor in AI-assisted pull request workflows, influencing not only code generation but also downstream adoption and integration outcomes through contextual grounding, task specificity, and evaluability cues. Keywords: Large language models Prompt quality Pull requests AI-assisted software engineering Human–AI collaboration Code generation

1 Introduction Large language model (LLM)-powered tools such as ChatGPT and GitHub Copilot are increasingly shaping collaborative software engineering workflows by supporting code generation, debugging, documentation, testing, and pull request (PR) review activities [1–4]. Developers increasingly incorporate conversational AI systems into day-to-day development tasks, often sharing these interactions directly within GitHub issues and PR discussions [5–8]. Prior work has examined the use of LLMs for code generation, debugging, productivity improvement, and issue resolution [9–14]. Recent studies further show that developers use conversational AI for a broad range of software-engineering activities extending beyond direct code generation [5, 8]. Consider a common scenario in AI-assisted pull-request workflows: a developer asks an LLM to generate or revise code for a specific issue, but the usefulness of the response depends heavily on how the request is framed. Prompts with clear contextual grounding and constraints may produce actionable code that can be integrated with minimal revision, whereas vague or underspecified requests may yield incomplete, irrelevant, or unverifiable suggestions. During review and refinement, developers often revise prompts, clarify requirements, and provide additional context before deciding whether generated code should be adopted, adapted, or discarded. Prior work on ChatGPTassisted pull requests shows that generated solutions are frequently refined, selectively reused, or substantially adapted rather than incorporated verbatim, underscoring the central role of developer evaluation and adaptation in AI-assisted workflows [15]. Prior work further shows that persistent interaction failures and declining confidence in LLM-generated outputs can lead developers to abandon AI assistance altogether [16]. Consequently, the influence of LLMs on pull-request workflows is shaped not only by model capability, but also by the structure and quality of developer prompts. Recent work characterizes prompts as first-class software artifacts within promptenabled systems and argues that prompt development requires systematic softwareengineering methodologies beyond ad hoc trial-and-error practices [8, 17, 18]. Complementary empirical work on prompt programming further shows that, when prompts are embedded in software systems and executed over variable inputs, developers must iteratively refine them, reason about model behavior, and address testing, debugging, and maintenance challenges that partly parallel traditional software development [19]. Despite these advances, empirical understanding of how prompt characteristics influence collaborative AI-assisted development workflows remains limited. Existing studies

2

primarily examine issue-resolution helpfulness, broad adoption or usage patterns, and generated-code quality [14, 20–22], leaving prompt-driven integration behavior within real-world pull request workflows insufficiently understood. However, important challenges remain in AI-assisted software development. LLMgenerated outputs can contain hallucinations, incomplete assumptions, and contextually inappropriate implementations [23]. Multi-file and semantically coordinated changes remain especially difficult due to limitations in project-level reasoning and contextual alignment [24]. These challenges highlight the need to better understand how prompt characteristics influence downstream code-generation, adoption, and integration outcomes in collaborative software-development workflows. Prior work further suggests that AI-generated pull requests exhibit distinct communicative and integration characteristics compared to traditional human-authored contributions [25, 26]. Recent evidence also shows that AI-assisted pull requests involve complex review, adaptation, and integration processes that extend beyond direct code generation [15, 27]. Understanding how prompt structure influences code generation, adoption, and downstream integration is therefore essential to improving AI-assisted collaboration. Moreover, recent evidence suggests that advances in foundation models do not eliminate the need for prompt engineering entirely. While some sophisticated prompting strategies provide diminished benefits for advanced LLMs, execution-feedback-based and task-specific prompting techniques continue to improve performance on complex software-engineering tasks [28]. To address this gap, we develop and validate a framework for operationalizing prompt structure in collaborative AI-assisted software engineering. Building on prior research on bug-report quality, pull request evaluation, testing practices, and prompt engineering [29–35], we characterize prompt quality using three dimensions: Context, Specificity, and Verification (CSV), which capture grounding, task precision, and evaluability properties associated with effective developer-facing artifacts. In this view, prompt structure represents an additional layer of developer-facing communication that may influence collaborative review and integration decisions alongside traditional pull request factors. Reliable operationalization of these constructs remains challenging, however, because large-scale manual annotation of developer–LLM interactions is costly and difficult to scale. Recent software-engineering guidelines and broader annotation research increasingly examine LLMs as annotators or evaluators, while cautioning that LLMgenerated labels can be sensitive to model configuration, prompt design, task context, bias, and insufficient human validation [36–39]. We therefore first examine whether LLM-based annotation can reliably reproduce human judgments of prompt structure across the CSV dimensions and identify the conditions under which selective annotation automation is appropriate. Building on this validated measurement framework, we model AI-assisted pull request workflows as a multi-stage decision process. Rather than treating AI assistance as a binary success-or-failure outcome, we examine how prompt structure influences actionable code generation, code adoption, and integration depth across successive stages of real-world pull request workflows. Importantly, we do not interpret adoption outcomes as direct indicators of intrinsic code quality or model capability; rather, they

3

reflect developer and maintainer decision-making within collaborative pull request workflows. To study these dynamics systematically, we analyze a manually validated dataset of 265 pull requests that contain developer-shared links to ChatGPT conversations. These cases are drawn from self-admitted ChatGPT usage (SACU) in open-source repositories [15]. Unlike prior studies that primarily examine conversational helpfulness, developer adoption, or coarse-grained pull request outcomes, our study focuses on how distinct prompt-quality dimensions influence code generation, code adoption, and integration depth within collaborative AI-assisted software development workflows. We further evaluate whether LLM-assisted annotation can reliably reproduce human judgments of prompt structure, motivating a hybrid annotation strategy that combines selective automation with human oversight across different prompt dimensions and workflow contexts. Our analysis shows that different prompt-quality dimensions become important at different stages of AI-assisted software development. More concretely, Specificity and Context are most strongly associated with actionable code generation, Verification is most strongly associated with code adoption, and Context again becomes the dominant factor for deeper integration. These findings suggest that effective collaboration depends not only on model capability or generated-code quality, but also on how developers construct prompts. Prompt construction thus emerges as a collaborative activity in which contextual grounding, task specification, and evaluability cues shape downstream development outcomes. The paper makes the following contributions:

• We develop a framework for operationalizing prompt structure in AI-assisted software engineering using three dimensions: Context, Specificity, and Verification (CSV), enabling systematic analysis of developer–LLM interactions in collaborative workflows. • We evaluate the reliability of LLM-assisted prompt annotation, showing that annotation performance varies across prompt dimensions and workflow contexts, and derive a dimension- and context-aware hybrid human–LLM annotation strategy for scalable prompt-quality assessment. • We introduce a stage-based workflow framework that relates prompt structure to actionable code generation, code adoption, and integration depth in AI-assisted pull request workflows. • We conduct a mixed-methods empirical study of 265 developer–ChatGPT interactions linked to open-source pull requests, showing that different prompt-quality dimensions become important at different stages of AI-assisted development workflows. • We release a public replication package to support reproducibility and future research on AI-assisted collaborative software engineering [40].

4

2 Operational Definitions and Scoring Rubric This section defines the key constructs used in our study and explains how they are operationalized for analysis. We first introduce the terminology used to describe ChatGPT-assisted interactions in pull requests, then present the prompt-quality dimensions and their associated scoring scheme, and finally describe the PR workflow stages. Together, these constructs establish the foundation for the analyses presented in subsequent sections.

2.1 Terminology We adopt and restate key terminology from prior work on ChatGPT-assisted pull request workflows [15]. All definitions are provided here to ensure that this paper remains self-contained. • Code Snippet: A small, standalone block of code intended for reuse, demonstrating a specific task or programming concept. In this study, “code snippet” refers exclusively to discrete pieces of code suggested by ChatGPT, excluding code examples embedded within descriptive explanations. • Self-Admitted ChatGPT Usage (SACU): Instances where developers explicitly acknowledge, within a pull request discussion, commit message, or issue thread, that they used ChatGPT to generate, modify, or assist in writing code. • Outcome Classes: Following the work of Ogenrwot and Businge [15], we classify ChatGPT-assisted pull requests into four categories: – Patch Applied (PA): ChatGPT generates a code snippet, and the suggested code is incorporated into the pull request, either directly or after some modification. This outcome reflects cases where ChatGPT output contributes directly to implementation, though often through adaptation rather than verbatim reuse. – Patch Not Applied (PN): ChatGPT generates a code snippet, but the suggested code is not incorporated into the pull request, or is modified beyond recognizable correspondence. Despite the lack of direct integration, these cases often reflect meaningful developer engagement with the generated solution, including adaptation to project constraints, exploration of alternative implementations, or validation of design decisions. – No Existing Patch (NE): The ChatGPT interaction provides textual guidance or explanation, but does not produce a code snippet that can be evaluated as a patch. These interactions nevertheless contribute to development by supporting tasks such as debugging, conceptual clarification, documentation refinement, and exploration of implementation strategies. – Closed (CL): The pull request is closed without being merged, regardless of whether ChatGPT-generated code was present. These cases may still involve code generation or developer interaction with ChatGPT, but the pull request does not result in integration into the codebase. Closure can arise from a range of factors, including quality concerns, workflow constraints, duplication, or misalignment with project requirements. In this study, CL is treated as a pull request–level outcome and is analyzed separately from code-focused analyses.

5

These definitions establish the basis for characterizing prompt structure, which we operationalize through three dimensions described next. The outcome classes are later mapped to stages in the pull request workflow, enabling analysis of how prompt structure influences code generation, adoption, and integration.

2.2 Prompt Quality Dimensions We ground our prompt-quality dimensions in prior software engineering research on bug report quality, pull request evaluation, and testing/CI practices [29–34, 41]. Prior work highlights the importance of contextual information, clear communication of developer intent, and evidence supporting the correctness of proposed changes when developers create, evaluate, and integrate software artifacts. Drawing on these recurring themes, we operationalize prompt quality using three dimensions: Context, Specificity, and Verification. We score each dimension on a common three-point ordinal scale (0–2), where 0 indicates absence of the relevant cue, 1 indicates partial or implicit presence, and 2 indicates explicit and well-developed presence. This scale reflects the recurring distinction in prior software-engineering work between missing, incomplete, and actionable information in developer-facing artifacts, including bug reports, pull requests, tests, CI signals, and external references [29, 30, 32–34]. Using the same scale across Context, Specificity, and Verification provides a consistent representation of prompt quality while limiting annotation ambiguity.

Context (C) Prior research on bug reports and pull request evaluation highlights the importance of concrete technical information and supporting evidence when diagnosing and assessing software changes [29, 30, 32]. We define Context as the extent to which a prompt is grounded in concrete technical artifacts. High-context prompts identify the locus of change and provide supporting evidence such as code snippets, logs, or error messages. We assign C = 2 when both are present, C = 1 when only one is provided, and C = 0 when neither is present. Specificity (S) Studies of pull-based development and prompt engineering emphasize the importance of clearly communicating intended changes, constraints, and expectations [31, 35]. We define Specificity as the clarity and precision with which a prompt articulates the intended task. High-specificity prompts define the goal, scope, and relevant constraints, such as API requirements, output formats, or example behaviors. We assign S = 2 when all three are specified, S = 1 when scope or constraints are incomplete, and S = 0 when the prompt is vague or underspecified. Verification (V) Evidence such as tests, CI outcomes, examples, and external references is commonly used to assess whether software changes satisfy expected behavior [33–35]. We define Verification as the presence of signals that allow developers to assess whether generated outputs are valid. Such signals may include input–output examples, rules, tests, 6

or expected behaviors. We assign V = 2 when the prompt includes a concrete, checkable evaluation criterion, V = 1 when expected behavior is implied but not explicitly defined, and V = 0 when no evaluation signal is present. We also considered Prompt Efficiency as a potential dimension, capturing the number of interaction turns required to obtain a satisfactory LLM-generated solution. However, we excluded this dimension due to construct validity concerns. ChatGPT conversations shared in pull request discussions represent only the subset of interactions developers choose to disclose and may omit exploratory prompting performed offline. Developers may therefore share only the final successful interaction, making observed turn counts an unreliable proxy for actual effort. Preliminary analysis further revealed no stable relationship with adoption outcomes. We therefore focus on structural prompt properties (Context, Specificity, and Verification) that can be measured consistently from shared artifacts.

2.3 PR Workflow Stages To analyze how prompt structure influences pull request outcomes, we conceptualize the development process as a sequence of stages reflecting how ChatGPT-generated content is used in practice. These stages are operationalized using the PR outcome classes defined earlier.

• Code Generation: Whether a prompt leads ChatGPT to produce a code snippet that can be considered for reuse. This stage distinguishes between interactions where no code is generated (NE ) and those where code is produced (PA & PN ). • Adoption: Whether the generated code is incorporated into the pull request, either directly or after modification. This stage distinguishes between cases where generated code is adopted (PA) and those where it is not (PN ). • Integration depth: The extent to which generated code contributes to the final implementation. This stage focuses on PA cases and captures variation in the degree of incorporation rather than binary adoption. In addition to these code-level stages, we consider pull request lifecycle outcomes, which capture whether a pull request is ultimately merged or closed (CL), as well as the time required to reach resolution.

2.4 Concrete Example To illustrate the constructs used in this study, we present a real-world example of a ChatGPT-assisted pull request interaction. In this case, a developer sought to eliminate duplication in a TypeScript discriminated union by deriving the union from a mapping type. The prompt includes the original type definitions and asks whether a more concise formulation is possible. The full interaction is available online1 . ChatGPT responded by introducing a mapped type and indexed-access pattern that removed the need for manual enumeration. Examination of the corresponding pull request2 shows 1 2

https://chat.openai.com/share/f09f38e5-f541-4f98-9483-e183f5650398 https://github.com/nbd-wtf/nostr-tools/pull/241

7

Step 1 — Dataset construction PatchTrack validated dataset

Filtering and prompt extraction

Final PR–Prompt dataset

Step 2 — Prompt annotation and reliability analysis (RQ1) Annotation rubric C – Context S – Specificity V– Verification

Agreement analysis

Human annotation

Outcome-aware reliability LLM-assisted annotation

Annotated Prompt dataset

Class-aware evaluation

Step 3 — Gate-based workflow modeling (RQ2) Gate 2 Integration depth (PA)

Gate 1 Adoption (PA vs PN)

Gate 0 Generation (NE vs PA&PN)

Axis B — PR lifecycle modeling Merge / close hazards

Step 4 — Interpretive and stability analysis Qualitative interpretation Representative examples

Robustness validation Sensitivity checks

Stage-dependent workflow interpretation

Fig. 1: Overview of the study design.

that the solution was largely incorporated into the repository with only minor adaptations. Using PatchTrack, an analytical instrument for classifying patch outcomes and usage patterns under AI assistance [15], we estimate that approximately 60% of the ChatGPT-generated code was retained in the final version. From the perspective of our prompt-quality dimensions, this interaction exhibits strong contextual grounding because the prompt includes concrete code and clearly specifies the transformation goal; partial specificity because the desired outcome is stated without fully constraining the solution; and implicit verification because correctness is inferred from type equivalence rather than explicit tests or formal specifications. This example illustrates the type of prompt–response interaction analyzed in this study and motivates our investigation of how prompt structure influences code generation, adoption, and integration outcomes.

3 Study Design This section describes the overall research methodology, including dataset construction, prompt-quality measurement, and the quantitative and qualitative analyses used to investigate the relationship between prompt structure and pull request outcomes. The study proceeds in four stages. First, we construct the PR–prompt dataset and prepare it for analysis. Second, we annotate prompts using the Context, Specificity, and Verification (CSV) framework and evaluate the reliability of LLM-assisted annotation. Third, we model AI-assisted pull request workflows using a gate-based framework that examines code generation, adoption, integration depth, and pull request lifecycle 8

outcomes. Finally, we conduct qualitative and robustness analyses to interpret and validate the observed stage-dependent relationships. Figure 1 provides an overview of the study design.

3.1 Analytical Scope and Inclusion Criteria Our analysis distinguishes between prompt-level interactions and pull request level outcomes to isolate the effects of prompt structure from broader project and workflow influences. We focus on stages corresponding to code generation, adoption, and integration, while treating overall pull request lifecycle outcomes separately. The outcome classes PA, PN, and NE map directly to these code-level stages and therefore form the basis of our gate-level analysis. Specifically, NE captures cases where no code is generated, PN captures cases where generated code is not adopted, and PA represents cases where generated code is incorporated into the pull request. In contrast, Closed (CL) pull requests may still involve high-quality prompts and ChatGPT-generated code, but closure often reflects factors external to prompt structure, such as project policies, workflow constraints, duplicate effort, or reviewer decisions. Including CL cases in gate-level models would therefore conflate promptlevel effects with project-level influences. Accordingly, we restrict gate-level analyses to PA, PN, and NE cases, while analyzing CL cases separately as part of pull request lifecycle outcomes (Axis B).

3.2 Research Questions This study investigates how the structural properties of developer prompts influence progression through AI-assisted pull request workflows. We first evaluate the reliability of prompt-structure measurement, and then analyze how validated prompt dimensions relate to code generation, adoption, integration depth, and pull request lifecycle outcomes.

• RQ1: How reliably can large language models reproduce human judgments of prompt structure? Reliable operationalization of prompt-quality constructs is essential for large-scale empirical studies of developer–LLM interactions. This question evaluates whether LLM-based annotation can reproduce human judgments of Context, Specificity, and Verification, and whether annotation reliability varies across workflow contexts in ways that affect the design of scalable human–LLM annotation strategies. – RQ1a (Overall Agreement): To what extent do LLM-generated annotations align with human consensus across prompt dimensions, and do systematic biases emerge? – RQ1b (Outcome-Conditioned Agreement): Does annotation agreement vary across pull request outcome classes (PA, PN, NE, CL)? Variations in agreement may indicate that certain workflow contexts are inherently more difficult to assess.

9

– RQ1c (Hybrid Annotation Policy): Which prompt dimensions and workflow contexts are suitable for LLM-assisted annotation without compromising construct validity? This question informs the design of a class-aware hybrid annotation strategy balancing scalability with human oversight.

• RQ2: How does prompt structural quality influence progression through the pull request integration pipeline? Using validated annotations, we examine how prompt structure relates to code generation, adoption, integration depth, and broader pull request lifecycle dynamics. – RQ2a (Code Generation): Does prompt structure influence whether ChatGPT produces actionable code (NE vs. non-NE)? – RQ2b (Adoption): Does prompt structure influence whether generated code is adopted in the pull request (PA vs. PN)? This stage reflects whether generated code is considered sufficiently useful and trustworthy for inclusion. – RQ2c (Extent of Integration Depth: Among adopted cases, does prompt structure relate to the fraction of ChatGPT-generated code integrated? This stage distinguishes superficial reuse from deeper integration. – RQ2d (Lifecycle Outcomes): Does prompt structure relate to pull request outcomes (merged vs. closed) and time-to-resolution?

3.3 Dataset Construction and Prompt Extraction To address our research questions, we construct a dataset of developer–ChatGPT interactions grounded in real-world pull request (PR) workflows. Our approach builds on the growing use of self-admitted GenAI usage artifacts as an empirical data source for studying developer interactions with conversational AI systems in open-source software development [8]. We build on the manually validated PatchTrack corpus [15], which contains pull requests with developer-shared links to ChatGPT conversations classified according to how generated code was used during development. By refining this validated corpus for prompt-level analysis, we obtain a traceable dataset linking prompts, generated code, and pull request outcomes, enabling systematic analysis of how prompt structure relates to code generation, adoption, integration depth, and broader workflow outcomes.

Step 1: Source Dataset and Outcome Classification The original PatchTrack validated dataset contains 338 pull requests with developershared ChatGPT conversations. In the PatchTrack study, each PR was classified according to how ChatGPT-generated code was used within the development workflow using token-based similarity analysis between generated snippets and pull request diffs [42]. Based on this analysis, each PR was assigned one of four outcome classes: Patch Applied (PA), Patch Not Applied (PN), No Code Patch / Guidance Only (NE), and Closed Unmerged (CL). To ensure reliability, all classifications were manually validated through inspection of ChatGPT conversations, pull request discussions, and

10

code diffs, resulting in a high-confidence subset of 283 validated cases (PA-89, PN-63, NE-84, and CL-47).

Step 2: Selection of Validated Subset To ensure consistency and avoid propagating classification errors, we restrict our analysis to this manually validated subset. We treat the PatchTrack labels as ground truth and do not re-derive or modify them in this study. The validated dataset provides a balanced representation of adoption, non-adoption, guidance-only interactions, and closed pull requests, enabling robust analysis across outcome types. Step 3: Prompt Availability Filtering Because our analysis focuses on prompt quality, we further filter the validated dataset based on the availability of developer prompts. In some cases, the original ChatGPT shared links were no longer accessible (e.g., deleted or inactive), preventing reliable extraction of prompt content. These cases were removed to ensure that all included instances contain complete and analyzable prompt data. After filtering, the final dataset consists of 265 PR-linked cases. The reduction is primarily due to missing prompt data in a subset of Patch Not Applied (PN) cases, with additional exclusions in the No Existing Patch (NE) and Closed (CL) categories (PA-89, PN-53, NE-80, and CL-43). Step 4: Prompt Extraction and Data Representation For each remaining case, we extract the developer prompt text from the ChatGPT conversation, including any refinements present within the interaction. Each case is represented as a tuple consisting of the prompt text, the corresponding PR outcome class, and traceability links to both the ChatGPT conversation and the GitHub pull request. Each instance is assigned a unique Case ID, which serves as the primary key throughout all stages of analysis, including annotation, validation, and modeling. Step 5: Data Integrity and Reproducibility All preprocessing steps, including dataset filtering, prompt extraction, and case mapping, are documented in the replication package to ensure full transparency and reproducibility. Our procedures follow established guidelines for empirical software engineering studies involving LLMs, emphasizing artifact preservation, traceability, and reproducible data preparation [36].

3.4 Prompt Evaluation and Validation Methodology To address RQ1, we design a structured methodology to (i) operationalize prompt quality, (ii) evaluate the reliability of LLM-based annotation, and (iii) derive a validated annotation strategy for large-scale analysis. Our approach integrates rubricbased scoring, LLM-assisted annotation, and human validation to ensure both scalability and measurement validity.

11

3.4.1 Prompt Quality Annotation We operationalize prompt quality using a structured rubric capturing three dimensions: Context, Specificity, and Verification (CSV ). Each dimension is scored on a three-point ordinal scale (0–2), reflecting the extent to which a prompt is grounded in technical artifacts, clearly specifies the intended task, and provides verifiable correctness criteria. Following recommendations for LLM-assisted empirical studies [36], we adopt a hybrid human–LLM annotation workflow that combines human validation with automated annotation to balance scalability and construct validity. A stratified 30% subset of prompts from each outcome class (CL: n = 13, PN: n = 16, PA: n = 27, and NE: n = 24) was independently annotated by two PhDlevel researchers using the CSV rubric. To minimize potential bias, annotators were blinded to all LLM-generated labels during both annotation and adjudication. Because prompt-quality labels are subsequently used in all downstream analyses, establishing annotation reliability is a necessary prerequisite for construct validity. Prior to any discussion or reconciliation, inter-rater agreement was assessed using weighted Cohen’s Kappa (κw ), which is appropriate for ordinal annotation tasks. Following reliability assessment, disagreements were resolved through discussion, and unresolved cases were adjudicated by a senior researcher (third co-author). The resulting consensus labels constituted the gold-standard reference used in all subsequent Human–LLM agreement analyses. To support transparency and reproducibility, the replication package contains the annotation rubric, annotation instructions, independent annotator labels, disagreement records, adjudication decisions, consensus labels, annotation prompts, evaluation logs, and configuration details [40]. These materials are provided in accordance with recommended reporting practices for LLM-assisted empirical studies [36].

3.4.2 LLM Annotation Procedure LLM-based annotation was performed using a structured prompt (LLM-v1) that operationalizes the Context–Specificity–Verification (CSV) rubric. For each interaction, LLM-v1 receives the developer-authored prompt extracted from the shared ChatGPT conversation together with the rubric definitions and scoring criteria. The model assigns an ordinal score (0–2) for each dimension and provides a brief justification. Human annotators were blinded to all LLM-generated labels during annotation and adjudication to prevent automated scores from influencing construction of the gold-standard dataset. Annotations were generated using ChatGPT under its default configuration, including default decoding parameters. Annotation experiments were conducted between 14 November 2025 and 21 January 2026. During this period, ChatGPT 5.1 generated annotations for 222 interactions and ChatGPT 5.2 generated annotations for 43 interactions. To assess the impact of the model transition, we compared annotations from both versions against the human-adjudicated gold standard. No systematic differences in scoring behavior or rubric interpretation were observed, suggesting that the version change did not materially affect the reported results.

12

3.4.3 Human–LLM Agreement Evaluation After establishing the human gold standard, we evaluate whether LLM-generated annotations can reliably reproduce human judgments of prompt quality. LLMgenerated scores are compared against the consensus human labels on the goldstandard subset. Because the CSV dimensions are measured on an ordinal three-point scale (0–2), we use complementary metrics that capture both agreement and error characteristics. Specifically, we compute quadratic weighted Cohen’s κw to measure chance-adjusted ordinal agreement while assigning partial credit to adjacent-category disagreements [43, 44]. We additionally report mean absolute error (MAE) to quantify the average magnitude of scoring deviations [45] and directional bias to identify systematic overor under-scoring relative to the human reference labels. To assess robustness across pull request contexts, we additionally compute classspecific agreement using quadratic weighted κw for each outcome class (PA, PN, NE, CL). Following common practice, commonly cited Kappa interpretation categories [46] are used only as descriptive reference points rather than strict decision thresholds. Annotation reliability is instead assessed using the combined evidence provided by agreement statistics, error magnitude, directional bias, and qualitative error analysis. Beyond aggregate agreement metrics, we examine representative Human–LLM disagreement cases to better understand the sources of annotation variation. Specifically, we review prompts exhibiting substantial disagreement to assess whether misalignment reflects boundary ambiguity between adjacent categories, differences in rubric interpretation, or systematic scoring tendencies. This diagnostic analysis contextualizes the quantitative results and is motivated by recent work showing that aggregate agreement metrics alone can obscure important differences between annotation ambiguity, systematic model bias, and context-dependent labeling failures in LLM-assisted annotation workflows [37–39].

3.4.4 Annotation Policy Derivation The agreement analysis described in Section 3.4.3 was used to determine which prompt dimensions and outcome classes could be reliably annotated using LLM assistance and which required continued human evaluation. Rather than assuming uniform reliability across workflow contexts, we adopt a selective verification strategy in which LLMgenerated labels are accepted only when agreement with the human gold standard is sufficient, while dimensions exhibiting persistent disagreement remain human-scored. This design follows recent recommendations for hybrid human–LLM annotation workflows that combine automated labeling with targeted human validation for unreliable cases [37, 38]. The policy was derived using the combined evidence from agreement strength, error magnitude, directional bias, and qualitative disagreement analysis rather than any single metric in isolation. Consequently, dimensions and workflow contexts exhibiting reduced reliability were retained as human-scored, yielding a conservative, class-aware, and dimension-specific annotation strategy that balances scalability, reliability, and construct validity. The resulting policy was subsequently applied to construct the

13

final prompt-quality dataset used in RQ2, with annotations assigned through either LLM-assisted or human evaluation according to the reliability criteria established above.

3.5 Modeling Prompt Effectiveness Across Development Stages To address RQ2, we adopt a structured modeling framework to evaluate how prompt structure influences AI-assisted pull request outcomes across multiple stages of the development workflow. Prompt quality is operationalized using a Prompt Quality Score (PQS), defined over three dimensions: Context (C), Specificity (S), and Verification (V). Each dimension is scored on a three-point ordinal scale (0–2) and aggregated as: PQS = C + S + V ∈ [0, 6]

(1)

While PQS provides a summary measure, all primary models are specified using the individual dimensions (C, S, V) to preserve interpretability and isolate their respective effects. We model prompt effectiveness as a sequence of stages reflecting how LLM outputs are generated and incorporated into pull requests: Gate 0 captures whether a prompt produces code, Gate 1 whether generated code is adopted, and Gate 2 the extent to which generated code is integrated. This decomposition separates code generation, adoption, and reuse rather than treating AI assistance as a single aggregated outcome. We additionally consider Axis B (Pull Request Lifecycle Outcomes), which captures pull request resolution status and time-to-resolution separately from the gate-level workflow analysis. Regression-based models are aligned with the structure of each stage. Logistic regression is used for binary outcomes (Gates 0 and 1), while models appropriate for bounded proportional data are used for Gate 2. All models include the prompt dimensions (C, S, V). To account for pull request complexity, we include Log(PR Size) as a control variable. PR Size is measured as total code churn, computed as the sum of lines added and deleted in a pull request, and is log-transformed to reduce skewness [47–49]

3.5.1 Model Diagnostics To evaluate the suitability of the statistical models, we conducted a series of diagnostic checks. Multicollinearity among predictors was assessed using variance inflation factors (VIF), following established regression diagnostics practices [50, 51]. Logistic regression models were examined for complete or quasi-separation, which can lead to unstable coefficient estimates [52]. For time-to-event analyses associated with Axis B, proportional hazards assumptions were evaluated using Schoenfeld residuals [53, 54]. We additionally conducted robustness and sensitivity analyses using alternative model specifications and reduced predictor sets to assess the stability of the findings.

14

3.5.2 Robustness and Sensitivity Analysis To assess the stability of the gate-level findings, we conduct a series of robustness and sensitivity analyses targeting repository heterogeneity, skewed pull request distributions, dominant technical domains, and alternative outcome operationalizations. We evaluate alternative model specifications, including repository-clustered standard errors and mixed-effects models with repository-level random effects, to account for potential within-repository dependence. We further repeat the analyses after excluding dominant repositories, extreme pull request sizes, and the most frequent programming languages to assess potential outlier and language-specific effects. Additional analyses examine alternative operationalizations of integration depth in Gate 2, compare aggregate Prompt Quality Score (PQS) models against individual CSV dimensions, and incorporate prompt-length controls and hold-out stability checks where applicable. Across these analyses, our focus is on the qualitative stability of stage-dependent relationships rather than exact coefficient replication.

3.5.3 Directed Qualitative Content Analysis To complement the quantitative modeling, we conduct a directed qualitative content analysis [55] to interpret how prompt structure relates to different stages of the pull request workflow. The analysis uses the Context, Specificity, and Verification rubric to guide examination of prompt rationale descriptions and associated developer– LLM interactions. Rather than seeking an exhaustive thematic characterization of developer–LLM interactions, we adopt a targeted interpretive approach to explain and contextualize the statistical relationships observed in the quantitative analysis. For each workflow stage, the reviewers examined a small set of representative and contrasting cases (approximately ten per stage) spanning high-, medium-, and low-scoring prompt configurations, multiple outcome classes, technical domains, and workflow outcomes. Sampling intentionally included cases that both aligned with and diverged from the dominant quantitative trends to reduce reliance on confirmatory examples and surface alternative explanations where applicable. Through cross-case comparison, the reviewers identified recurring patterns in how prompts formulate implementation tasks, express constraints, provide contextual grounding, and support evaluation-related reasoning. Initial interpretations were refined by revisiting the original prompts and conversations, and disagreements were resolved through iterative discussion and consensus refinement. The qualitative analysis was conducted across all workflow stages, and representative examples were selected only after recurring patterns had been identified through cross-case review. Accordingly, the examples are intended to illustrate and contextualize the quantitative findings rather than serve as standalone evidence or support statistical generalization. Additional examples exhibiting similar characteristics are included in the replication package to support transparency and cross-case comparison.

15

4 Results This section presents the key findings of our study. We first evaluate the reliability of LLM-based prompt annotation (RQ1) and then examine how validated prompt dimensions relate to progression through the pull request workflow (RQ2).

4.1 RQ1: Reliability of LLM-Based Prompt Annotation We assess the extent to which LLM-based annotation reproduces human judgments of prompt structural quality and whether such annotations can support downstream modeling.

Gold-standard annotation reliability. As described in Section 3.4, the gold-standard dataset was established through independent human annotation followed by adjudication. Prior to reconciliation, inter-rater agreement was assessed using weighted Cohen’s Kappa (κw ) to evaluate the reliability of the CSV rubric across the validation subset. Agreement varied substantially across dimensions and outcome classes. Context agreement ranged from κw = −0.059 (CL) to κw = 0.563 (NE), Specificity from κw = −0.573 (CL) to κw = 0.440 (NE), and Verification from κw = −0.043 (NE) to κw = 0.467 (PA). Raw agreement ranged from 23.1% to 91.7% across the evaluated combinations. The weakest agreement was observed primarily in CL and, to a lesser extent, NE, where small validation subsets and highly imbalanced score distributions contributed to unstable κw estimates. These findings indicate that annotation reliability varies substantially across prompt dimensions and workflow contexts, underscoring the challenges of promptquality assessment and motivating the class-aware evaluation strategy developed in RQ1c. Following reliability assessment, all disagreements were resolved through discussion and adjudication, and the resulting consensus labels served as the gold-standard reference for subsequent Human–LLM agreement analyses.

4.1.1 RQ1a: Overall Human–LLM Agreement LLM-based annotation exhibits moderate agreement with human judgment for both Context and Specificity. However, Context exhibits a consistent under-scoring tendency relative to human annotations, whereas Verification achieves only fair agreement, indicating greater difficulty in operationalizing verification-related signals. We evaluate agreement between annotations produced by LLM-v1 and blinded human consensus annotations across the human-validation subset (N = 80). Context achieves moderate agreement (κw = 0.526), but with a relatively high mean absolute error (MAE = 0.475) and substantial negative bias (−0.450), indicating that LLM-v1 systematically assigns lower Context scores than the human reference labels. Specificity also exhibits moderate agreement (κw = 0.431), with lower error (MAE = 0.325) and a smaller negative bias (−0.200). Verification shows only fair agreement (κw = 0.316), with comparable error (MAE = 0.338) and minimal directional bias (−0.037). Overall, these results suggest that LLM-based annotation more reliably captures observable prompt characteristics than implicit correctness and verification cues. 16

Explaining Agreement Patterns. The stronger agreement observed for Context and Specificity is consistent with their reliance on explicit structural features such as code artifacts, system details, and task constraints that are directly observable in the prompt text. In contrast, Verification often depends on implicit correctness expectations and evaluative reasoning that may be underspecified or distributed across the broader conversational context, making consistent interpretation more difficult. The negative bias observed for Context further suggests that LLM-v1 requires more explicit contextual evidence than human annotators before assigning higher Context scores.

4.1.2 RQ1b: Outcome-Conditioned Agreement Patterns Agreement between LLM and human annotations varies across outcome classes, particularly for Verification. To assess robustness across workflow contexts, we compute quadratic weighted κw separately for each outcome class and dimension (Table 1). Class

N

Context κw

Specificity κw

Verification κw

PA PN CL NE

27 16 13 24

0.427 0.308 0.544 0.621

0.053 0.282 0.581 0.438

0.073 0.667 -0.020 -0.075

Table 1: Per-class quadratic weighted κw values.

Agreement varies markedly across outcome classes. Verification shows very low agreement in PA (κw = 0.073) and negative agreement in CL and NE, indicating substantial human–LLM disagreement on verification-related cues. Specificity also shows very low agreement in PA (κw = 0.053), but reaches moderate levels in CL and NE. Context is more stable, with moderate-to-substantial agreement in CL (κw = 0.544) and NE (κw = 0.621), though weaker in PN. Overall, annotation reliability is outcomedependent, with some dimensions harder to score consistently in particular workflow contexts.

Explaining Outcome-Conditioned Agreement. The observed variation across outcome classes likely reflects differences in how information and correctness expectations are expressed within prompts. During annotation and adjudication, NE and CL prompts often contained localized structural cues, making Context and Specificity easier to identify consistently. In contrast, PA prompts frequently involved iterative refinement, partial implementations, or loosely specified requirements, causing relevant signals to be more distributed and difficult to interpret. Verification agreement appears particularly sensitive to the clarity of correctness cues. In PA cases, verification signals are often implicit and must be inferred from the interaction, contributing to very low agreement. By contrast, PN cases frequently contain explicit evaluation or rejection reasoning, consistent with the substantially higher Verification agreement (κw = 0.667). 17

4.1.3 RQ1c: Deriving an Empirically Grounded Evaluation Policy The RQ1a and RQ1b results indicate that annotation reliability varies substantially across prompt dimensions and outcome classes. As a result, no single automation strategy achieves consistent agreement across all workflow contexts, motivating a classaware evaluation policy. The recommended annotation policy was derived primarily from agreement strength (κw ) and evidence of systematic over- or under-scoring identified through the bias analysis, with MAE used as a supplementary indicator of scoring behavior. Following widely used interpretations of Cohen’s κw , values of κw ≥ 0.40 were used as a reference benchmark for identifying dimensions that achieved at least moderate agreement [46, 56, 57]. We therefore derived the dimension- and class-aware evaluation policy shown in Table 2 using the combined evidence from agreement strength, error magnitude, directional bias, and qualitative examination of disagreement patterns. The class-specific agreement estimates are based on stratified subsets of the dataset and should therefore be interpreted as indicative of relative annotation reliability across workflow contexts rather than as precise population parameters. Accordingly, the policy should be viewed as a conservative recommendation informed by multiple sources of evidence rather than a strict threshold-based decision rule. Metric Context Specificity Verification

PA

PN

CL

NE

Human Human Human

Human Human LLM

Human LLM Human

Human LLM Human

Table 2: Recommended human–LLM annotation policy derived from agreement and bias analyses.

Context is retained as human-scored across all outcome classes because LLM-v1 consistently assigns lower Context scores than human annotators despite moderate agreement levels. In contrast, LLM-assisted scoring is used selectively for Specificity (CL and NE) and Verification (PN) in workflow contexts where agreement is sufficiently reliable, while all remaining dimension–outcome combinations remain human-scored.

Interpreting the Evaluation Policy. The resulting policy highlights that annotation reliability depends on both prompt dimension and workflow context, motivating selective human oversight rather than uniform automation.

18

Summary of RQ1 Results: LLM-based annotation achieves moderate agreement with human judgments for some prompt dimensions, but reliability varies substantially across prompt dimensions and workflow contexts. Context and Specificity achieve moderate agreement overall, although Context consistently receives lower scores from the LLM and Specificity exhibits substantial variation across outcome classes. Verification remains difficult to assess when correctness expectations are implicit. Agreement also varies across outcome classes, indicating that annotation reliability depends not only on the construct being measured but also on the surrounding workflow context. These findings suggest that uniform annotation automation is inappropriate. Instead, reliable operationalization of prompt structure requires a class-aware hybrid human–LLM strategy in which LLM-assisted scoring is selectively applied only in dimension–outcome contexts exhibiting acceptable agreement and limited systematic bias, while human oversight is retained for contexts that remain difficult to annotate consistently.

4.2 RQ2: Prompt Structure and Progression Through the Pull Request Workflow Building on the validated prompt-quality dataset produced through the class-aware hybrid annotation strategy developed in RQ1, we analyze how prompt structure influences progression through the pull request workflow. We conceptualize this process as a sequence of stages corresponding to code generation, adoption, and integration, followed by broader pull request lifecycle outcomes. This stage-based framework allows us to examine how the role of prompt structure evolves across the development workflow. We combine quantitative modeling with targeted qualitative analysis to interpret the observed effects. Quantitative results identify statistical relationships between prompt dimensions and workflow outcomes, while illustrative examples are used to explain the mechanisms underlying these patterns. Together, these analyses provide insight into how prompt structure shapes LLM-assisted development in practice.

4.2.1 Descriptive Statistics We summarize the distributions of Context (C), Specificity (S), and Verification (V) across all prompts.

Mean Std. Dev.

C

S

V

1.48 0.62

1.08 0.58

0.58 0.57

Class

Mean

Median

N

PA PN NE CL

3.96 3.00 2.11 3.53

4 3 2 4

89 53 80 43

(a) Prompt structure (C, S, V) (b) PQS by outcome class Table 3: Prompt structure dimensions and PQS by outcome class.

19

Fig. 2: Distribution of Prompt Quality Score (PQS) across outcome classes.

Prompts exhibit higher levels of Context (mean = 1.48) and Specificity (mean = 1.08) than Verification (mean = 0.58), indicating that prompts more frequently provide contextual grounding and task definition than explicit correctness signals. Prompt quality also varies substantially across outcome classes. PA cases exhibit the highest mean PQS (3.96), whereas NE cases exhibit the lowest (2.11), suggesting a strong association between prompt quality and successful code generation. PN and CL occupy intermediate ranges with substantial overlap, indicating that mid-range prompt quality alone does not uniquely determine outcomes, as also reflected in the overlapping distributions shown in Figure 2. Notably, CL cases exhibit relatively high PQS despite not being merged, suggesting that prompt structure is more closely associated with early-stage outcomes such as code generation and adoption than with downstream pull-request lifecycle outcomes. Pull request size and contributor experience exhibit highly right-skewed distributions, with large discrepancies between mean and median values and the presence of extreme outliers. These characteristics motivate the use of log transformations and robust modeling techniques in subsequent analyses. Together, these descriptive patterns suggest that the influence of prompt structure evolves across the pull request workflow, motivating the stage-based modeling framework introduced next, which decomposes the workflow into code generation, adoption, integration, and lifecycle phases.

4.3 Stage-Based Modeling of Prompt Structure Effects To examine how prompt structure influences progression through the pull request workflow, we model the process as a sequence of stages corresponding to code generation, adoption, integration, and lifecycle outcomes. This decomposition isolates how different prompt dimensions influence each transition rather than treating pull request outcomes as a single aggregated process. We report results for each stage independently, beginning with code generation (Gate 0), followed by adoption (Gate 1), integration depth (Gate 2), and lifecycle

20

outcomes. For each stage, we estimate regression models that quantify the effects of Context, Specificity, and Verification while controlling for pull request characteristics.

Qualitative interpretation and illustrative examples. To complement the quantitative modeling, we use the directed qualitative analysis described in Section 3.5.3 to identify recurring patterns in prompt formulation, implementation framing, evaluability cues, and downstream integration behavior. For each workflow stage, we summarize these patterns and present purposefully sampled representative and contrasting examples spanning different prompt-quality configurations, workflow outcomes, and technical contexts. These examples are intended to illustrate recurring patterns identified during the qualitative review rather than serve as standalone evidence or estimates of prevalence. Additional examples are included in the replication package, and each case is labeled as PA-X, PN-X, or NE-X to support cross-referencing.

4.3.1 Gate 0: Code Generation Gate 0 examines whether prompt structure influences whether ChatGPT produces actionable code. This stage distinguishes cases where code was generated (PA and PN) from interactions that produced only textual or conceptual guidance (NE). After excluding closed pull requests (CL) and observations with missing values, the effective sample size is n = 218. Table 4 reports odds ratios (OR) and 95% confidence intervals from logistic regression models predicting code generation. Model 1 includes the three prompt dimensions (Context, Specificity, and Verification), while Model 2 adds logtransformed pull request size as a control variable. Table 4: Gate 0: Logistic Regression Results for Code Generation

Context (C) Specificity (S) Verification (V) Log(PR Size) Observations

(1) Baseline

(2) + PR Size

2.16* [1.14, 4.10] 62.96*** [8.52, 465.19] 0.90 [0.43, 1.88] — —

2.14* [1.13, 4.07] 65.84*** [8.90, 487.14] 0.90 [0.43, 1.88] 1.12 [0.91, 1.36]

218

218

Notes: Odds ratios reported with 95% confidence intervals in brackets. *** p < 0.001, ** p < 0.01, * p < 0.05.

Both Context and Specificity are strong predictors of actionable code generation. A one-unit increase in Context is associated with approximately a twofold increase in the odds of generating actionable code (OR ≈ 2.1), and this effect remains statistically significant across model specifications. Specificity exhibits the strongest association with code generation (OR ≈ 66), indicating that prompts with clearer goals, constraints, and implementation requirements are substantially more likely to produce actionable

21

code. Although the estimated effect size is large, the corresponding confidence interval is also wide, and its precise magnitude should therefore be interpreted cautiously. In contrast, Verification does not exhibit a statistically significant effect, suggesting that correctness-related cues do not determine whether code is generated. Adding pull request size does not materially change the estimates, and PR size itself is not statistically significant. Overall, these findings suggest that code generation depends primarily on prompt grounding and task specification, whereas evaluative and correctness-oriented cues appear more relevant to downstream workflow stages.

Recurring structural patterns. Across the reviewed Gate 0 cases, recurring structural differences emerged between prompts that produced actionable code and those that did not. Prompts associated with actionable generation frequently framed bounded implementation objectives, specified concrete transformation or construction tasks, or constrained the expected output format. In contrast, prompts associated with non-code outcomes were often exploratory, conceptual, or discussion-oriented despite being grounded in valid technical contexts. These patterns suggest that actionable code generation is most commonly associated with prompts that frame bounded implementation tasks rather than exploratory or discussion-oriented requests. Illustrative examples. To further interpret these effects, we examine representative PN and NE cases reflecting the recurring structural patterns identified during the qualitative review. The examples contrast implementation-oriented and conceptual prompt formulations to illustrate how prompt structure influences code generation outcomes. ✓Implementation-oriented prompts enabling code generation. Case PN19 (C=1, S=1, V=1) illustrates how implementation-oriented prompts can generate actionable code even when the generated solution is not ultimately adopted. The developer asks: “Hello can you give me a regex to match ULID format?” and later evaluates a candidate expression: “This one is correct? [0-7][0-9A-HJKMNP-TV-Z]{25}”. Although the prompt provides only moderate contextual grounding, it defines a clear implementation target and bounded coding task, enabling ChatGPT to generate a usable regex solution. This example illustrates why Context and Specificity strongly influence code generation at Gate 0. ✓Conceptual prompts limiting code generation. By contrast, Case NE-3 (C=1, S=0, V=0) asks: “how are express.js middleware functions commonly named? [...] What might you call a middleware that adds a content security policy [...]?”. Although the prompt is grounded in a concrete technical domain, it does not define a bounded implementation task or request an executable artifact. As a result, ChatGPT produces descriptive guidance rather than actionable code. This example illustrates that contextual grounding alone is insufficient to support code generation when prompts remain conceptual rather than implementation-oriented.

22

Summary of Gate 0 Results: Code generation is driven primarily by Specificity and Context. Prompts that clearly define bounded implementation objectives and provide relevant technical grounding are substantially more likely to produce actionable code, whereas Verification does not significantly influence whether code is generated. The qualitative analysis reinforces this pattern by showing that implementation-oriented prompts tend to yield executable solutions, while conceptual or exploratory prompts more often produce explanatory guidance.

4.3.2 Gate 1: Code Adoption Gate 1 examines whether prompt structure influences whether generated code is adopted within the pull request. This analysis is restricted to cases where code was generated (PA and PN outcomes). After removing observations with missing values, the effective sample size is n = 141. Table 5 reports odds ratios (OR) and 95% confidence intervals from logistic regression models predicting code adoption. Model 1 includes Context, Specificity, and Verification, while Model 2 adds log-transformed pull request size as a control variable. Table 5: Gate 1: Logistic Regression Results for Code Adoption

Context (C) Specificity (S) Verification (V) Log(PR Size) Observations

(1) Baseline

(2) + PR Size

2.28* [1.02, 5.10] 0.61 [0.23, 1.58] 7.78*** [3.40, 17.82] — —

2.22 [0.97, 5.09] 0.79 [0.29, 2.15] 8.45*** [3.50, 20.43] 1.36** [1.08, 1.71]

141

141

Notes: Odds ratios reported with 95% confidence intervals in brackets. *** p < 0.001, ** p < 0.01, * p < 0.05.

Verification is the strongest and most consistent predictor of code adoption. A oneunit increase in Verification is associated with approximately eightfold higher odds of patch adoption (OR ≈ 7.8–8.5), and this effect remains highly significant across model specifications. In contrast, Specificity does not exhibit a statistically significant association with adoption, indicating that clearly specifying task requirements alone is insufficient to distinguish adopted from non-adopted patches once actionable code has been generated. Context shows a positive association with adoption in the baseline model, but the effect becomes marginal after controlling for pull request size, suggesting a more limited role at this stage of the workflow. Including pull request size reveals a significant positive effect, with larger pull requests more likely to incorporate LLM-generated code. Importantly, adding this

23

control does not materially change the estimated effect of Verification, indicating that the observed relationship is robust to differences in pull request complexity. Overall, these results suggest that adoption depends primarily on whether generated outputs can be evaluated against explicit correctness expectations. Whereas Context and Specificity help developers communicate and constrain implementation tasks during code generation, Verification provides validation criteria and evaluative signals that appear to increase confidence in generated solutions and their suitability for integration. This contrast highlights a transition from task formulation during generation to correctness assessment during adoption.

Recurring structural patterns. The qualitative analysis revealed recurring prompt patterns associated with both successful and unsuccessful adoption outcomes. Prompts linked to adopted code frequently included explicit correctness criteria, observable output expectations, or other evaluative signals that enabled developers to assess generated solutions with confidence. In contrast, prompts associated with non-adopted code often produced technically plausible outputs and sometimes provided substantial contextual grounding and task specification, yet lacked sufficiently explicit validation criteria or acceptance conditions to support confident evaluation within the target implementation context. These recurring patterns qualitatively reinforce the quantitative finding that Verification plays a central role during adoption, whereas Context and Specificity alone are often insufficient to explain adoption decisions. Illustrative examples. To further interpret these effects, we examine representative PA and PN cases reflecting the recurring structural patterns identified during the qualitative review. The examples illustrate how adoption depends less on contextual grounding alone and more on whether generated outputs are sufficiently constrained and evaluable for developers to assess correctness with confidence. Some examples intentionally reappear across gates to illustrate how the same prompt may exhibit different strengths and limitations across successive workflow stages. ✓Explicit constraints and observable correctness enabling adoption. Case PA-22 (C=1, S=2, V=2) illustrates how high Specificity and Verification support direct adoption of generated code. The developer asks: “Please generate code using the Kotlin sksamuel/scrimage library to draw text on an image and add an outline to the text. The text should be white and the outline black.”. Although the prompt provides only moderate contextual grounding, it defines a narrowly scoped implementation objective together with explicit output constraints. The prompt also includes observable correctness conditions, namely that the generated image must contain white text with a black outline, making the generated solution directly evaluable and reducing ambiguity regarding expected behavior. This example illustrates why Specificity and Verification emerge as significant predictors of adoption at Gate 1. ✓Generated but weakly evaluable code limiting adoption. By contrast, Case PN-3 (C=2, S=2, V=0) illustrates how code may fail to be adopted despite strong contextual grounding and task specification. The developer asks whether contextual information normally managed through SLF4J’s MDC can instead be attached directly through a fluent log.atLevel() logging approach in a highly concurrent environment and requests an example Log4j2 configuration. Although the prompt provides substantial technical context and a clearly defined objective, it does not specify expected 24

behavior, acceptance criteria, or other correctness conditions that would allow the generated solution to be evaluated within the target system. This example illustrates that adoption depends not only on obtaining plausible code or configuration guidance, but also on whether prompts provide explicit evaluative signals that support trustworthy assessment.

Summary of Gate 1 Results: Gate 1 results show that code adoption is driven primarily by Verification. Prompts containing explicit correctness expectations or validation criteria are substantially more likely to produce code that developers adopt, whereas Specificity does not distinguish adopted from non-adopted code once generation has occurred. Context plays a weaker role and becomes marginal after accounting for pull request size. The qualitative analysis supports this finding by showing that adoption depends less on task formulation and more on whether generated outputs can be evaluated against clear correctness expectations. Together, these results suggest that Gate 1 represents a transition from code generation to solution evaluation.

4.3.3 Gate 2: Integration Depth Gate 2 examines whether prompt structure influences the extent to which LLMgenerated code is integrated into the final implementation. This analysis is restricted to adopted cases (PA outcomes). After removing observations with missing values, the effective sample size is n = 89, spanning 71 repositories. The dependent variable, fraction adopted ∈ (0, 1], represents the proportion of generated code incorporated into the final implementation. Table 6: Gate 2: Fractional Logit Results for Fraction of Code Adopted

Variable

AME

Context (C)

0.125* [0.016, 0.233] -0.013 [-0.131, 0.105] -0.008 [-0.106, 0.089] -0.036 [-0.074, 0.002]

Specificity (S) Verification (V) Log(PR Size) Observations

89

Notes: Average marginal effects reported with 95% confidence intervals in brackets. Standard errors are clustered at the repository level. *** p < 0.001, ** p < 0.01, * p < 0.05.

Table 6 reports average marginal effects (AME) from the fractional-logit model with repository-clustered robust standard errors. Context is the only statistically significant predictor of integration depth. A one-unit increase in Context is associated 25

with an increase of approximately 12.5 percentage points in the expected fraction of adopted code. In contrast, Specificity and Verification do not exhibit statistically significant effects, indicating that while these dimensions influence earlier workflow stages, they do not determine how extensively code is reused once accepted. Log(PR Size) shows a negative but statistically insignificant association, and including this control does not materially affect the estimated impact of Context. Overall, these results suggest that deeper integration depends primarily on contextual fit. Once generated code has been accepted, the extent of reuse appears to depend less on task specification or correctness cues and more on how well the generated solution aligns with the surrounding implementation environment.

Recurring structural patterns. The qualitative analysis revealed recurring prompt patterns associated with different levels of downstream reuse after adoption. Prompts linked to deeper integration frequently embedded generated tasks within rich implementation context, including surrounding code structures, existing APIs, configuration details, or repository-specific logic. This contextual grounding enabled generated outputs to align more closely with the surrounding implementation environment and therefore required less downstream adaptation before reuse. In contrast, prompts with high Specificity but limited contextual grounding often produced technically correct yet relatively generic outputs that required substantial refinement or restructuring prior to integration. These recurring patterns qualitatively reinforce the finding that deeper reuse depends primarily on contextual fit rather than task specification or correctness cues alone. Illustrative examples. To further interpret these effects, we examine representative PA cases reflecting recurring structural patterns observed during the qualitative review. The examples illustrate how deeper integration depends primarily on contextual grounding and the extent to which generated solutions align with existing implementation structures and surrounding code artifacts. Some examples intentionally reappear across gates to illustrate how the same prompt may exhibit different strengths and limitations across successive workflow stages. ✓Strong contextual grounding enabling deep integration. Case PA-78 (C=2, S=2, V=1; Fraction=84.84%) illustrates how strong contextual grounding enables extensive reuse after adoption. The developer provides a complete Python program and asks: “Refactor the following code to improve readability and structure: [...]”. The prompt includes multiple function definitions, control-flow structures, and surrounding implementation details, clearly situating the task within an existing codebase. This rich Context enables the generated solution to align closely with the surrounding implementation, allowing most of the output to be incorporated with minimal modification. This example illustrates why Context emerges as the dominant predictor of integration depth at Gate 2. ✓High specificity with limited contextual grounding. By contrast, Case PA24 (C=1, S=2, V=1; Fraction=7.69%) illustrates how highly specific implementation requests may still result in limited reuse when contextual grounding is weak. The developer asks: “Create a TypeScript enum for the following values and initialize the corresponding variables [...]”. Although the prompt clearly defines the intended

26

structure and functionality, it provides little information about the surrounding implementation environment or existing code structures. As a result, the generated output remains relatively generic and requires substantial downstream adaptation before incorporation into the pull request. This example illustrates why contextual alignment, rather than Specificity alone, drives deeper reuse at Gate 2.

Summary of Gate 2 Results: Gate 2 results show that integration depth is driven primarily by Context. Prompts containing rich implementation context are associated with deeper reuse of generated code, whereas Specificity and Verification do not significantly influence how extensively accepted code is incorporated into the final implementation. The qualitative analysis reinforces this finding by showing that generated solutions are reused more extensively when they align closely with existing code structures, APIs, and surrounding implementation details. Together, these results suggest that Gate 2 is primarily a problem of implementation alignment rather than task specification or correctness evaluation.

4.4 Axis B: PR Lifecycle Results Axis B models pull request lifecycle as a time-to-event process, capturing the rate at which pull requests are merged or closed without merge. After removing observations with missing values, the effective sample size is n = 261. Pull requests resolve via either merge or closure, with unresolved cases treated as right-censored observations. Table 7 reports hazard ratios (HR) and 95% confidence intervals from cause-specific Cox proportional hazards models for merge and close events. None of the prompt dimensions (Context, Specificity, or Verification) exhibit statistically significant effects in the merge model, indicating that prompt structure does not substantially influence the rate at which pull requests are merged. In contrast, Verification is a significant predictor in the close-hazard model (HR ≈ 1.77), indicating that higher levels of verification are associated with faster closure without merge. Context and Specificity do not exhibit significant effects in this model. Log(PR size) is significant in both models, with hazard ratios below one for merge (HR ≈ 0.93) and closure (HR ≈ 0.83), indicating that larger pull requests resolve more slowly overall. The stronger effect for closure suggests that larger pull requests are less likely to be quickly rejected or abandoned, but also require more time to reach resolution. Overall, these results suggest that lifecycle dynamics are driven by factors distinct from those influencing earlier workflow stages. While prompt structure strongly affects code generation, adoption, and integration depth (Gates 0–2), it does not appear to substantially influence merge decisions once code has entered the pull request workflow. Instead, Verification is associated primarily with faster negative resolution outcomes. One possible explanation is that prompts emphasizing correctness constraints make implementation mismatches or unmet requirements easier to identify during review, allowing maintainers to reach closure decisions more quickly

27

Table 7: Axis B: Cause-Specific Cox Results for PR Lifecycle

Context (C) Specificity (S) Verification (V) Log(PR Size) Observations

Close Hazard

Merge Hazard

1.72 [0.95, 3.13] 0.63 [0.34, 1.16] 1.77* [1.03, 3.05] 0.83** [0.72, 0.95]

1.02 [0.80, 1.30] 1.07 [0.81, 1.41] 0.84 [0.65, 1.09] 0.93* [0.87, 1.00]

261

261

Notes: Hazard ratios reported with 95% confidence intervals in brackets. *** p < 0.001, ** p < 0.01, * p < 0.05.

when generated solutions do not align with project expectations. More broadly, lifecycle outcomes appear to reflect repository-level coordination and review processes that extend beyond prompt design.

Summary of Axis B Results: Axis B results show that prompt structure does not significantly influence the rate at which pull requests are merged, although verification signals are associated with faster closure without merge. In contrast to earlier workflow stages, life cycle timing is driven primarily by pull request size and repository-level review processes. These findings suggest that prompt design has limited influence on pull request life cycle dynamics, reflecting a transition from LLM-driven effects to broader coordination and review factors.

4.5 Robustness and Sensitivity Analysis We conducted a series of robustness and sensitivity analyses to assess whether the stage-dependent findings were sensitive to repository heterogeneity, dominant repositories or programming languages, skewed pull request distributions, alternative Gate 2 operationalizations, prompt-length effects, and model-specification choices. Repository-level robustness checks using repository-clustered robust standard errors and repository random-intercept sensitivity specifications preserved the primary findings across all gates. Gate 0 continued to show positive effects for Context and Specificity, Gate 1 consistently identified Verification as the primary predictor of code adoption, and Gate 2 continued to identify Context as the primary predictor of integration depth. Additional sensitivity analyses excluding dominant repositories, frequent programming languages, and extreme pull request outliers did not materially alter the qualitative interpretation of the Gate 0 and Gate 1 models. Gate 2 Context effects remained directionally positive across restricted samples, although they became less stable under smaller PA-only subsets, indicating greater sensitivity at the integrationdepth stage. Alternative Gate 2 operationalizations based on ordinal reuse levels 28

similarly identified Context as the strongest positive predictor, while Specificity and Verification remained comparatively weak. Models using the aggregate Prompt Quality Score (PQS) positively predicted Gate 0 and Gate 1 outcomes, but did not consistently predict Gate 2 integration depth, reinforcing the importance of analyzing prompt dimensions separately for deeper reuse behaviors. Finally, introducing prompt-length controls did not materially alter the primary interpretations, suggesting that the observed Context effects were not simply artifacts of prompt verbosity. Stratified 80/20 hold-out stability checks also preserved the principal coefficient directions, although several effects weakened under smaller hold-out subsets, particularly for Gate 2 and other restricted PA-only analyses. Overall, the robustness analyses support the interpretation that the observed stage-dependent relationships reflect stable characteristics of developer–LLM interaction behavior rather than artifacts of a particular repository subset, language distribution, or modeling specification.

4.6 Model Diagnostics To evaluate the validity of the statistical models, we conducted a series of diagnostic checks across all specifications. Variance inflation factors (VIF) remained below 2 for all predictors, indicating no evidence of problematic multicollinearity and suggesting that Context, Specificity, Verification, and PR size capture distinct aspects of prompt structure and workflow behavior. All logistic regression models associated with Gates 0 and 1 converged successfully within a small number of iterations. Similarly, the fractional logit models used for Gate 2 converged without numerical instability, and fitted values aligned closely with the observed integration-depth distributions, supporting the suitability of the modeling framework for bounded reuse outcomes. For Axis B, proportional hazards diagnostics based on Schoenfeld residuals indicated mild deviations from the proportional hazards assumption for certain predictors, particularly PR size. Accordingly, the reported hazard ratios should be interpreted as approximate relative effects rather than strictly time-invariant estimates. Overall, the diagnostic analyses did not identify severe violations of model assumptions, supporting the validity and stability of the reported results.

4.7 Overall Findings and Discussion (RQ1–RQ2) Our study examined how prompt structure relates to different stages of LLM-assisted pull request workflows, ranging from annotation reliability and code generation to adoption and integration depth. Across the analyzed pull requests, prompts were associated with a wide range of development activities, including implementation requests, debugging, refactoring, architectural reasoning, configuration changes, and conceptual discussion. Rather than treating LLM usage as a binary notion of success or failure, our analysis models AI-assisted development as a staged collaborative workflow in which different prompt characteristics become important at different stages of developer interaction, evaluation, and integration.

29

These findings complement prior studies of self-admitted generative-AI usage in open-source projects, which show that developers increasingly share ChatGPT interactions in issues and pull requests for tasks such as code generation, debugging, explanation, and design discussion [5]. Whereas such work primarily characterizes the breadth of developer GenAI usage, our study examines what happens after these interactions enter collaborative pull request workflows: whether prompts generate actionable code, whether developers adopt the generated code, and how deeply the generated code is integrated into the surrounding implementation context. Synthesizing findings across RQ1 and RQ2, we identify several cross-cutting patterns that clarify how developers structure prompts and evaluate LLM-generated outputs during collaborative software development.

1. Reliable prompt annotation requires selective human oversight. RQ1 shows that annotation agreement and automation suitability are not identical. Although Context achieves the strongest overall human–LLM agreement, it also exhibits systematic under-scoring bias, making fully automated scoring unreliable under our evaluation criteria. In contrast, Specificity exhibits lower aggregate agreement but more stable scoring behavior across several workflow contexts, allowing selective automation in structurally explicit contexts. Verification remains unstable when correctness signals are implicit or distributed across interactions. Agreement further varies across workflow outcomes. Conceptual and descriptive interactions often contain more localized structural signals, whereas iterative code-centric exchanges may involve more distributed context and implicit evaluative reasoning, reducing annotation stability. These findings suggest that reliable operationalization of prompt structure requires class-aware hybrid human–LLM workflows rather than uniform automation. More broadly, they demonstrate that the reliability of LLM-assisted annotation depends both on the construct being measured and the workflow context in which it is observed. These findings align with a growing body of research showing that LLM-assisted annotation workflows require selective human oversight rather than fully automated labeling pipelines [37–39]. Prior work highlights how annotation reliability may vary depending on task complexity, verification difficulty, contextual ambiguity, and interaction design. Our results extend these observations to developer–LLM interactions by showing that annotation reliability depends not only on the target construct, but also on conversational context and the distribution of evaluative reasoning across interactions. Consequently, for empirical software-engineering studies that rely on LLM-based annotation, validation should be performed at the level of individual constructs and workflow contexts rather than assuming uniform annotation reliability across a dataset. 2. Prompt effectiveness is workflow-stage-dependent. The results reveal a clear stage-dependent relationship between prompt structure and workflow outcomes. Prompts that provide technical grounding and bounded implementation objectives are more likely to generate actionable code, whereas downstream adoption depends primarily on verification-oriented cues, including correctness expectations, evaluability constraints, and observable output criteria that support developer assessment and

30

trust. Deeper integration, in turn, depends primarily on contextual alignment with the surrounding implementation environment rather than task definition alone. These findings extend recent work on ChatGPT-assisted issue resolution, which shows that conversational helpfulness depends on task, project, and interaction characteristics [14]. They also complement prior work on AI-assisted pull request integration. PatchTrack showed that ChatGPT-generated patches frequently undergo adaptation, selective reuse, or complete rejection during collaborative review, indicating that successful AI assistance cannot be understood solely through generated-code quality or merge outcomes [15]. Whereas PatchTrack focused on the downstream fate of generated patches, our analysis investigates an earlier stage of the workflow by examining how prompt structure influences actionable code generation, code adoption, and integration depth. Together, the two studies suggest that variation in downstream integration outcomes may partly originate from upstream differences in prompt construction and contextual grounding. The findings further refine recent prompt-engineering results showing that prompt effectiveness remains task-dependent even for advanced LLMs [28]. Our results suggest that prompt effectiveness is also workflow-stage-dependent: the prompt characteristics supporting code generation are not identical to those associated with downstream adoption and integration. Specifically, Specificity and Context are most strongly associated with actionable code generation, Verification emerges as the primary predictor of code adoption, and Context again becomes the dominant factor for deeper integration. These findings suggest that different dimensions of prompt quality become important at different stages of AI-assisted software development. 3. AI-assisted development reflects iterative evaluation, adaptation, and alignment. Across workflow stages, developers evaluate generated outputs using traditional software engineering criteria such as implementation fit, maintainability, architectural alignment, and project conventions. LLM-generated suggestions are therefore incorporated into existing review and collaboration processes rather than bypassing them. Prior work further shows that repeated failures, poor contextual fit, and unmet developer expectations can erode trust in LLM-generated suggestions and ultimately lead developers to abandon AI assistance, highlighting the importance of evaluation and alignment within AI-assisted development workflows [16]. The findings further suggest that verification-oriented reasoning becomes increasingly important during downstream evaluation and adoption decisions. Prompts containing explicit correctness expectations, evaluability constraints, or observable output criteria are more likely to support developer trust and confident integration. At the same time, qualitative analyses show that such evaluative signals are often implicit or distributed across interactions, helping explain why verification-oriented constructs remain more difficult to operationalize reliably through automated annotation. More broadly, these findings support recent arguments that prompts should be treated as first-class software artifacts within prompt-enabled systems [17, 18] and align with qualitative evidence that developers engage in iterative “prompt programming” practices [19]. Our results extend this emerging literature by showing how

31

prompt structure relates not only to prompt construction or conversational helpfulness, but also to downstream collaborative software-engineering outcomes such as code adoption and integration depth. Viewed collectively, the findings suggest that AI-assisted software development is best understood as a staged collaborative process rather than a single generation event. In this view, prompt construction, code generation, adoption, and integration form a connected workflow in which upstream prompting decisions influence downstream collaborative outcomes. This perspective complements prior work on patch integration [15] by extending the analysis further upstream to the prompts that initiate AI-assisted development activities. The results further demonstrate that studying these workflows requires careful operationalization of prompt-quality constructs and selective use of LLM-assisted annotation, linking methodological choices about measurement directly to substantive conclusions about AI-assisted software-development practices.

Temporal scope and model evolution. The dataset spans a period during which ChatGPT and related LLMs underwent substantial capability improvements. Because historical conversations rarely disclose the exact model version used, we cannot directly assess the impact of model evolution on individual outcomes. Consequently, our findings should not be interpreted as measuring the performance of a specific ChatGPT release. Prior work suggests that although advances in LLM capabilities may reduce the effectiveness of some prompting strategies, prompt structure and task-specific guidance remain important for software-engineering tasks [28]. Accordingly, our results should be interpreted as characterizing AI-assisted development workflows during the period represented by the dataset rather than the behavior of a particular model version.

4.8 Implications Implications for Practitioners The findings suggest that effective AI-assisted software development depends on structuring prompts differently across workflow stages. During early-stage code generation, prompts benefit from clear technical grounding and bounded implementation objectives, including relevant frameworks, APIs, code snippets, runtime behavior, and expected functionality. Once code has been generated, successful adoption depends primarily on verification-oriented cues such as expected behavior, validation examples, output constraints, acceptance criteria, and other signals that help developers evaluate correctness and trustworthiness. The results further show that deep integration of LLM-generated code depends heavily on contextual alignment with the surrounding implementation environment. Even highly specific prompts may produce correct and adoptable code that nevertheless requires substantial downstream adaptation if insufficient implementation context is provided. Developers seeking reusable LLM-generated solutions should therefore prompt from within the target architectural and repository context rather than treating code generation as an isolated task.

32

These findings further suggest that many downstream integration challenges may originate upstream during prompt construction. Developers often focus on improving generated code after it is produced, yet our results indicate that contextual grounding and task framing influence whether generated solutions are actionable, adoptable, and ultimately reusable. Organizations seeking to improve the effectiveness of AI-assisted development should therefore treat prompt design as part of the software-development process rather than as a separate interaction with an AI tool. Beyond workflow outcomes, the findings demonstrate that organizations adopting LLM-assisted annotation or evaluation pipelines should employ selective human verification rather than uniform automation. Annotation reliability varies substantially across prompt dimensions and interaction contexts, indicating that human effort should be concentrated where construct ambiguity, contextual reasoning, or evaluative judgment remain difficult for LLMs to reproduce reliably. Although Specificity can be selectively automated in several workflow contexts, Context requires continued human oversight due to systematic under-scoring bias, and Verification remains less stable when evaluative reasoning is implicit or distributed across interactions. Collectively, these findings suggest that different dimensions of prompt quality become important at different stages of AI-assisted development: task definition and contextual grounding support code generation, verification-oriented cues support adoption, and contextual alignment becomes increasingly important for deeper integration. More broadly, the findings reinforce that LLM-generated outputs should be treated as provisional artifacts requiring critical evaluation, refinement, and adaptation within existing software engineering workflows rather than as authoritative solutions. This interpretation is consistent with recent evidence that developers frequently reassess, revise, or abandon LLM assistance when generated outputs fail to satisfy contextual requirements or maintain developer trust [16].

Implications for Researchers This study highlights important methodological considerations for future research on AI-assisted software engineering. First, prompt-quality constructs such as Context, Specificity, and Verification cannot be assumed to be uniformly observable or reliably annotatable across all workflow settings. Second, LLM-assisted annotation reliability depends both on the construct being measured and the workflow context in which it is applied, motivating construct-aware and context-aware validation strategies. More broadly, the findings suggest that understanding AI-assisted software development requires linking upstream prompting behavior to downstream integration outcomes. Prior studies have examined either developer–LLM interactions or the fate of generated artifacts in collaborative workflows. Our results indicate that these perspectives should be studied jointly, as prompt construction may represent an important source of variation in subsequent adoption, adaptation, and integration behavior. The findings further demonstrate that scalable prompt-structure operationalization requires adaptive hybrid annotation strategies rather than uniform LLM automation. Future research on LLM-assisted empirical software engineering should therefore move beyond aggregate agreement metrics and develop evaluation protocols

33

that explicitly account for construct-specific variation, contextual ambiguity, systematic annotation bias, and the differing reliability of prompt dimensions across workflow contexts. From a methodological perspective, the results suggest that future empirical studies using LLMs as annotators should report construct-level reliability, systematic bias analyses, and workflow-specific validation procedures rather than relying solely on aggregate agreement measures. Such reporting would improve transparency, facilitate comparison across studies, and help researchers identify contexts in which automated annotation remains appropriate or requires additional human oversight. Finally, the results suggest that future benchmark design, prompt-engineering research, and evaluations of LLM effectiveness should incorporate richer implementation context and repository-level information while recognizing that different prompt dimensions become important at different stages of the development workflow. Evaluations based solely on code generation or correctness may overlook factors that influence downstream adoption and integration.

Implications for Educators The findings suggest that AI-assisted software development should be taught as a staged collaborative workflow rather than as a simple code generation activity. Educational use of tools such as ChatGPT should therefore emphasize not only prompt construction, but also critical evaluation, adaptation, and integration of AI-generated outputs within real development settings. Students should also learn that prompt construction influences not only the quality of generated outputs, but also the effort required for downstream evaluation, adaptation, and integration. Prompting should therefore be taught as a software-engineering activity that affects the entire lifecycle of AI-assisted development rather than only the initial generation step. The results further highlight the importance of incorporating human oversight and evaluative reasoning into AI-assisted software engineering education. Cases involving plausible but weakly evaluable or poorly integrated solutions provide opportunities to teach reviewer trust, implementation fit, maintainability, and collaborative decisionmaking. More broadly, the findings support instructional approaches that position generative AI as a collaborative development aid requiring structured human judgment rather than as an autonomous code-generation system. These insights are directly applicable to project-based software engineering courses where students increasingly incorporate generative AI into collaborative development activities. For example, they are currently being integrated into senior design instruction at UNLV through industry-based case studies illustrating how developers evaluate, adapt, and integrate AI-generated solutions within collaborative pull request workflows.

34

5 Related Work 5.1 LLMs in Software Engineering and Developer–LLM Interactions Large language model (LLM)-powered tools such as ChatGPT and GitHub Copilot are increasingly integrated into software engineering workflows, supporting tasks including code generation, debugging, testing, documentation, and issue resolution [7, 58]. Recent studies show that developers frequently use conversational LLMs within collaborative development environments and openly share these interactions in GitHub issues and pull requests [6, 59]. Ehsani et al. [14] analyzed developer–ChatGPT conversations in GitHub issue threads and found that successful issue-resolution interactions depend on conversational, project, and issue-level characteristics. Complementary work identified prompt-level knowledge gaps, including missing context, missing specifications, and unclear instructions, as recurring factors associated with ineffective LLM-guided issue resolution [60]. Recent work has also begun studying self-admitted GenAI usage (SAGU) as a lens for understanding AI adoption in open-source software development. Xiao et al. [8] analyzed more than 1,200 public disclosures of GenAI usage across GitHub repositories and developed taxonomies of developer tasks, generated content, and usage purposes. Their findings establish self-admitted GenAI usage as a valuable empirical artifact for studying developer–LLM interactions, but focus primarily on adoption and disclosure behavior rather than downstream integration outcomes. A complementary line of research has examined failure modes in developer–LLM interactions. Tie et al. [16] identified multiple categories of LLM failures, including incorrect responses, context loss, and cognitive overload, and found that persistent failures can erode developer trust and ultimately lead to abandonment of AI assistance. Whereas their work focuses on why developers disengage from LLM assistance, our study investigates how prompt structure influences successful progression through subsequent workflow stages, including code generation, adoption, and integration. Beyond individual developer–LLM interactions, recent research has examined how generative AI influences collaborative software engineering workflows. Tufano et al. [5] showed that developers use ChatGPT and GitHub Copilot for activities extending beyond code generation, including debugging, testing, documentation, refactoring, and code review. Similarly, PatchTrack showed that ChatGPT-generated patches frequently undergo adaptation, selective reuse, or complete rejection during pull request integration, highlighting the socio-technical factors that shape AI-assisted development outcomes [15]. Together, these studies highlight the importance of understanding how developer–LLM interactions translate into downstream collaborative outcomes. Whereas prior work primarily focuses on conversational helpfulness, adoption behavior, abandonment, or the downstream fate of generated artifacts, the present study investigates an earlier stage of the workflow by examining how prompt structure influences actionable code generation, code adoption, and integration depth across AI-assisted pull request workflows.

35

5.2 Prompting Practices and Prompt Quality Recent research increasingly highlights the importance of prompt construction in AI-assisted software engineering. Ehsani et al. [60] identified recurring deficiencies in developer–ChatGPT interactions, including missing context, underspecified requirements, competing objectives, and unclear instructions, and proposed promptimprovement heuristics emphasizing specificity, contextual richness, and clarity. Similarly, Otten et al. [61] found that developers frequently rely on iterative multi-turn prompting and adapt their prompts according to the software engineering task being performed. Midolo et al. [35] further derived practical prompting guidelines for code generation, emphasizing explicit input/output specifications, examples, preconditions, postconditions, and ambiguity clarification. Recent studies have also examined how prompt strategies influence softwareengineering outcomes. Wang et al. [28] found that several prompt-engineering techniques developed for earlier LLMs provide diminished benefits for advanced models, whereas execution feedback and task-specific guidance remain important for complex software-engineering tasks. Porta et al. [22] similarly reported that common prompt patterns do not uniformly improve maintainability, security, or reliability outcomes, suggesting that prompt effectiveness remains strongly task- and context-dependent. A complementary line of work argues that prompts should be treated as softwareengineering artifacts rather than merely natural-language queries. Chen et al. [17] characterize prompts as first-class artifacts in prompt-enabled systems and propose “promptware engineering” as a systematic methodology for prompt requirements, design, testing, evolution, deployment, and monitoring. Villamizar et al. [18] similarly emphasize prompt management, reuse, traceability, and maintenance, while Liang et al. [19] provide qualitative evidence that developers engage in iterative “prompt programming,” treating prompts as programmable artifacts that require refinement, testing, and evaluation. Collectively, these studies establish that prompt formulation is an important developer-facing artifact that shapes LLM behavior and software-engineering outcomes. However, prior work primarily focuses on conversational effectiveness, prompting practices, code quality, or prompt management. In contrast, our study examines how distinct prompt-structure dimensions influence successive stages of AI-assisted pull request workflows, including actionable code generation, code adoption, and integration depth.

5.3 Pull Request Evaluation and Developer-Facing Artifacts Prior work shows that successful software collaboration depends on how effectively technical information is communicated to developers and reviewers. Studies of bug-report quality emphasize the importance of contextual information such as reproduction steps, stack traces, test cases, and environment details that help developers localize and understand a problem [29, 30]. Similarly, research on pull request evaluation shows that contribution acceptance depends on both technical and social signals, including change quality, contributor experience, review discussion, and repository

36

context [31, 32]. These findings motivate our Context and Specificity dimensions, which capture technical grounding and the communication of a bounded, actionable task. Prior work also highlights the importance of evidence that enables developers to assess whether a change is correct or acceptable. External references help document and justify pull request changes [34], while continuous integration and testing provide automated signals for evaluating incoming contributions [33]. These studies motivate our Verification dimension, which captures whether a prompt provides information that supports correctness assessment, such as tests, expected behavior, examples, constraints, or references to specifications. Building on this literature, we treat prompts as developer-facing artifacts that communicate context, intent, and evaluability cues in much the same way as bug reports and pull request descriptions, thereby influencing downstream development and review decisions.

5.4 LLM-Assisted Annotation and Empirical SE Methodology LLMs are increasingly used not only as software-engineering tools, but also as instruments for empirical research, including coding, labeling, synthesis, and artifact assessment. Recent methodological guidelines for empirical SE studies involving LLMs emphasize that researchers should explicitly report the role of the LLM, model configuration, prompts, interaction logs, validation procedures, and limitations, particularly when LLMs are used as annotators or judges [36]. These recommendations are especially relevant because LLM outputs are affected by model non-determinism, evolving model versions, prompt sensitivity, and context-dependent interpretation. Broader work on LLM-based data annotation similarly shows that LLMs can reduce annotation cost and scale empirical analysis, but that their labels require careful assessment before being used as ground truth [62]. Recent human–LLM annotation studies further caution against fully replacing human judgment with automated labels. Wang et al. [37] propose a collaborative annotation framework in which LLMs first generate labels and explanations, a verifier identifies potentially unreliable labels, and humans re-annotate low-confidence cases. Similarly, Takehi et al. [38] show that LLM judgments can assist large-scale relevance assessments, but that their effectiveness varies across assessment contexts. They therefore advocate selective human involvement to calibrate and verify LLM predictions where reliability is uncertain. Xu et al. [39] further argue that aggregate agreement metrics can obscure important differences between task-inherent ambiguity and model-driven errors, especially for ordinal and interpretive annotation tasks. These studies motivate our treatment of prompt-quality annotation as a methodological research question rather than a preprocessing step. Accordingly, we evaluate human–LLM agreement across prompt dimensions and workflow outcome classes, and derive a class-aware hybrid annotation policy that uses LLM labels only where they are sufficiently aligned with human judgment while retaining human annotation where reliability concerns remain or where additional human verification is warranted. Taken together, prior work establishes that prompt construction influences LLM behavior and that prompts increasingly function as developer-facing artifacts within collaborative AI-assisted workflows. Existing studies, however, primarily focus on 37

conversational effectiveness, prompt-development practices, code quality, or artifact integration outcomes. In contrast, our study examines how distinct prompt-structure dimensions influence successive stages of AI-assisted pull request workflows and evaluates whether these dimensions can be measured reliably through LLM-assisted annotation.

6 Threats to Validity Construct Validity. Our measurement of prompt structure relies on the operationalization of Context, Specificity, and Verification. Although these dimensions are grounded in prior softwareengineering and prompt-engineering research, they may not capture all characteristics of effective prompting or developer intent. Prompt quality in practice may additionally depend on factors such as conversational strategy, domain knowledge, reasoning style, or broader interaction history that are not explicitly modeled by our rubric. To mitigate subjectivity, we employed a structured codebook, conducted blinded human annotation on a validation subset, and evaluated agreement with LLM-assisted annotation using ordinal agreement metrics, error analysis, and bias assessment. Because annotation reliability varied across dimensions and workflow outcome classes, we derived a class-aware hybrid annotation policy in which dimensions exhibiting instability or systematic bias remained human-scored. The final analytical dataset used in RQ2 was constructed using this empirically validated policy rather than a uniform automated annotation strategy, reducing the risk that downstream findings are driven by annotation artifacts. Agreement estimates for some outcome-class subsets (particularly CL, and to a lesser extent NE for Verification) should nevertheless be interpreted cautiously because small validation sample sizes, skewed score distributions, and prevalence effects can produce unstable agreement statistics, including negative weighted kappa estimates despite relatively high observed agreement. We additionally provide the annotation prompts, codebook, and replication artifacts to support transparency and reproducibility. Internal Validity. The study design is observational and does not establish causal relationships between prompt structure and pull request outcomes. Although we control for factors such as pull request size, contributor experience, repository context, and prompt length, unobserved confounding variables may still influence the observed relationships. In addition, the stage-based workflow framework represents an analytical abstraction of collaborative development behavior rather than a strictly sequential process, and some workflow stages may partially overlap in practice. A further threat concerns conversational completeness. The shared ChatGPT conversations analyzed in this study may not fully capture all developer–LLM interactions associated with a pull request. Developers may omit intermediate prompts, continue conversations outside publicly shared artifacts, or substantially revise generated outputs before integration. Consequently, the observed prompts may represent partial views of broader iterative workflows. To mitigate these risks, we conducted robustness

38

checks, sensitivity analyses, hold-out stability tests, and alternative model specifications. These analyses preserved the principal stage-dependent relationships, although some effects became less stable in smaller restricted samples, particularly for Gate 2 integration-depth models estimated on PA-only subsets.

External Validity. Our dataset is derived from self-admitted ChatGPT usage (SACU) cases in public GitHub pull request workflows. Consequently, the findings may not generalize to undisclosed AI usage, private development settings, or interactions involving other LLMs and coding agents. As observed in prior work on self-admitted GenAI usage in opensource software development [8], developers who publicly share AI conversations may differ behaviorally from those who do not disclose AI assistance, introducing potential survivorship and disclosure bias into the observed workflow patterns. Furthermore, the dataset reflects interactions collected during earlier generations of ChatGPT, and newer models may respond differently to the same prompts. However, our focus is on how prompt structure shapes collaborative workflow behavior, including code generation, adoption, and integration, which is less sensitive to specific model versions than raw model capability. Future work should examine whether these workflow patterns persist across newer LLM ecosystems and alternative collaborative development settings. Conclusion Validity. Our conclusions are based on statistical modeling and agreement analysis, both of which may be affected by sample size, class imbalance, model assumptions, and measurement error. In addition, the workflow outcome classes used throughout the analysis are inherited from the manually validated PatchTrack dataset, and residual classification ambiguity may still affect some downstream analyses, particularly for partially integrated or substantially adapted patches. We mitigate these risks through model diagnostics, repository-clustered robustness specifications, sensitivity analyses, holdout stability checks, and alternative operationalizations of workflow outcomes. Across these analyses, the principal stage-dependent relationships remained qualitatively stable, with Specificity and Context supporting code generation, Verification remaining most strongly associated with code adoption, and Context continuing to exhibit the strongest relationship with integration depth. Additionally, the qualitative analysis was intended to contextualize quantitative findings rather than support standalone thematic claims. Consequently, the examples should be interpreted as illustrative of recurring patterns rather than representative estimates of their prevalence, reducing the risk of overgeneralization.

6.1 Limitations and Future Work Although the proposed stage-based framework provides a useful abstraction for modeling AI-assisted pull request workflows, real-world collaborative development often involves overlapping, iterative, and conversationally dependent behaviors that may not be fully captured by discrete analytical stages. Similarly, our operationalization of

39

prompt structure through Context, Specificity, and Verification intentionally emphasizes interpretable developer-facing dimensions, but may not capture finer-grained conversational or semantic characteristics that influence LLM-assisted development outcomes. Future work should investigate how prompt-aware tooling, review support systems, and developer education practices can leverage workflow-aware prompt characteristics to improve collaborative AI-assisted software engineering. Additional research could also examine richer conversation-level representations, longitudinal developer–LLM interaction patterns, and the role of emerging agent-based coding systems in shaping collaborative development workflows.

7 Ethical Implications This study analyzes publicly accessible GitHub pull requests and developer-shared ChatGPT conversations involving self-admitted AI usage. Although these artifacts are publicly available, we recognize the importance of responsible handling of developergenerated content. To minimize privacy risks, we restrict our analysis to publicly shared artifacts, avoid collecting private or personally sensitive information, and present illustrative examples only when necessary to support empirical findings. We additionally provide a public replication package containing datasets, analysis scripts, annotation artifacts, and supplementary documentation to support transparency, reproducibility, and responsible reuse of the research artifacts [40].

8 Conclusion This paper investigated how prompt structure relates to progression through AIassisted pull request workflows. Using a manually validated dataset of 265 pull request-linked developer–ChatGPT interactions, we examined how three prompt dimensions, Context, Specificity, and Verification, relate to actionable code generation, code adoption, integration depth, and broader pull request workflow outcomes. Our findings show that prompt characteristics exert distinct stage-dependent associations across collaborative software engineering workflows. Specifically, Specificity and Context are most strongly associated with actionable code generation, Verification emerges as the primary predictor of code adoption, and Context again becomes the dominant factor for deeper integration. These results highlight the importance of contextual grounding, evaluability, and implementation alignment throughout AI-assisted pull request workflows. The study further demonstrates that reliable operationalization of prompt structure requires adaptive hybrid human–LLM annotation strategies rather than uniform automation, as annotation reliability varies substantially across prompt dimensions and workflow contexts. More broadly, the results show that the reliability of LLMassisted annotation depends both on the construct being measured and the context in which it is applied, highlighting the need for construct-aware and context-aware validation in future empirical studies. Overall, the results suggest that effective AI-assisted software engineering depends not only on generating plausible code, but also on how developers construct prompts 40

that support subsequent evaluation, adoption, adaptation, and integration activities. Viewed collectively, the findings position prompt construction as an upstream factor in AI-assisted software-development workflows, influencing not only code generation but also downstream collaborative outcomes within pull request processes. We hope this work motivates future research on workflow-aware prompt engineering, context-aware AI-assisted development tools, scalable methodologies for studying developer–LLM interactions, and human-centered evaluation frameworks for collaborative software engineering with LLMs.

Declarations Funding This research was supported by NSF Grant Award No. #2519136.

Ethics approval Not applicable

Informed consent Not applicable

Author contribution John Businge: Conceptualization, supervision, and review and editing of the manuscript. Richard Sserunjogi: Data collection, analysis, writing of the original draft. Daniel Ogenrwot: Data analysis and writing of the original draft.

Data availability The datasets, analysis scripts, and supplementary documentation are publicly available at https://zenodo.org/records/20451325.

Conflict of interest The authors declare that there are no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Clinical trial number in the manuscript Not applicable

References [1] Russo, D.: Navigating the complexity of generative ai adoption in software engineering. ACM Trans. Softw. Eng. Methodol. (2024) https://doi.org/10.1145/ 3652154 41

[2] Huang, Y., Chen, Y., Chen, X., Chen, J., Peng, R., Tang, Z., Huang, J., Xu, F., Zheng, Z.: Generative software engineering. arXiv preprint arXiv:2403.02583 (2024) https://doi.org/10.48550/arXiv.2403.02583 . Submitted on 5 Mar 2024, last revised 3 Apr 2024 (this version, v2) [3] Ebert, C., Louridas, P.: Generative ai for software practitioners. IEEE Software 40(4), 30–38 (2023) https://doi.org/10.1109/MS.2023.3265877 [4] Sauvola, J., Tarkoma, S., Klemettinen, M., Riekki, J., Doermann, D.: Future of software development with generative ai. Automated Software Engineering 31(26) (2024) https://doi.org/10.1007/s10515-024-00330-1 [5] Tufano, R., Pepe, F., Zampetti, F., Mastropaolo, A., Dabić, O., Di Penta, M., Bavota, G.: Developers and generative ai: A study of self-admitted usage in open source projects. Empirical Software Engineering 31(108) (2026) https://doi.org/ 10.1007/s10664-026-10848-w [6] Xiao, T., Hata, H., Treude, C., Matsumoto, K.: Generative ai for pull request descriptions: Adoption, impact, and developer interventions. Proc. ACM Softw. Eng. 1(FSE) (2024) https://doi.org/10.1145/3643773 [7] Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., Wang, H.: Large language models for software engineering: A systematic literature review. ACM Trans. Softw. Eng. Methodol. 33(8) (2024) https://doi.org/10.1145/3695988 [8] Xiao, T., Fan, Y., Calefato, F., Treude, C., Kula, R.G., Hata, H., Baltes, S.: Selfadmitted genai usage in open-source software. IEEE Transactions on Software Engineering, 1–18 (2026) https://doi.org/10.1109/TSE.2026.3681886 [9] Peng, S., Kalliamvakou, E., Cihon, P., Demirer, M.: The impact of ai on developer productivity: Evidence from github copilot. arXiv preprint arXiv:2302.06590 (2023) https://doi.org/10.48550/arXiv.2302.06590 . Submitted on 13 Feb 2023 [10] Siddiq, M.L., Da Silva Santos, J.C., Tanvir, R.H., Ulfat, N., Al Rifat, F., Carvalho Lopes, V.: Using large language models to generate junit tests: An empirical study. In: Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering, pp. 313–322 (2024) [11] Jin, K., Wang, C.-Y., Pham, H.V., Hemmati, H.: Can chatgpt support developers? an empirical evaluation of large language models for code generation. In: Proceedings of the 21st International Conference on Mining Software Repositories. MSR ’24, pp. 167–171. Association for Computing Machinery, New York, NY, USA (2024). https://doi.org/10.1145/3643991.3645074 . https://doi.org/10.1145/3643991.3645074

42

[12] Guo, Q., Cao, J., Xie, X., Liu, S., Li, X., Chen, B., Peng, X.: Exploring the potential of chatgpt in automated code refinement: An empirical study. In: Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. ICSE ’24, pp. 1–13. Association for Computing Machinery, New York, NY, USA (2024). https://doi.org/10.1145/3597503.3623306 . https://doi.org/10.1145/3597503.3623306 [13] Deng, Y., Xia, C.S., Yang, C., Zhang, S.D., Yang, S., Zhang, L.: Large language models are edge-case generators: Crafting unusual programs for fuzzing deep learning libraries. In: Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. ICSE ’24, pp. 1–13. Association for Computing Machinery, New York, NY, USA (2024). https://doi.org/10.1145/3597503. 3623343 . https://doi.org/10.1145/3597503.3623343 [14] Ehsani, R., Pathak, S., Rodriguez, E.P., Haiduc, S., Chatterjee, P.: What characteristics make chatgpt effective for software issue resolution? an empirical study of task, project, and conversational signals in github issues. Empirical Software Engineering 31(1), 22 (2026) https://doi.org/10.1007/s10664-025-10745-8 . Published online: 18 November 2025 [15] Ogenrwot, D., Businge, J.: Patchtrack: A comprehensive analysis of chatgpt’s influence on pull request outcomes. Empirical Software Engineering 31(5), 136 (2026) https://doi.org/10.1007/s10664-026-10869-5 [16] Tie, J., Yao, B., Li, T., Fang, H., Ahmed, S.I., Wang, D., Zhou, S.: ”should i give up now?” investigating llm pitfalls in software engineering. ACM Trans. Softw. Eng. Methodol. (2026) https://doi.org/10.1145/3801972 . Just Accepted [17] Chen, Z., Wang, C., Sun, W., Liu, X., Zhang, J.M., Liu, Y.: Promptware engineering: Software engineering for prompt-enabled systems. ACM Trans. Softw. Eng. Methodol. (2026) https://doi.org/10.1145/3796535 [18] Villamizar, H., Fischbach, J., Korn, A., Vogelsang, A., Mendez, D.: Prompts as software engineering artifacts: A research agenda and preliminary findings. In: Product-Focused Software Process Improvement: 26th International Conference, PROFES 2025, Salerno, Italy, December 1–3, 2025, Proceedings, pp. 470–478. Springer, Berlin, Heidelberg (2025). https://doi.org/10.1007/978-3-032-12089-2 32 . https://doi.org/10.1007/978-3-032-12089-2 32 [19] Liang, J.T., Lin, M., Rao, N., Myers, B.A.: Prompts are programs too! understanding how developers build software containing prompts. Proc. ACM Softw. Eng. 2(FSE) (2025) https://doi.org/10.1145/3729342 [20] Grewal, B., Lu, W., Nadi, S., Bezemer, C.-P.: Analyzing developer use of chatgpt generated code in open source github projects. In: Proceedings of the 21st International Conference on Mining Software Repositories. MSR ’24, pp. 157– 161. Association for Computing Machinery, New York, NY, USA (2024). https: 43

//doi.org/10.1145/3643991.3645072 . https://doi.org/10.1145/3643991.3645072 [21] Siddiq, M.L., Roney, L., Zhang, J., Santos, J.C.D.S.: Quality assessment of chatgpt generated code and their use by developers. In: Proceedings of the 21st International Conference on Mining Software Repositories. MSR ’24, pp. 152– 156. Association for Computing Machinery, New York, NY, USA (2024). https: //doi.org/10.1145/3643991.3645071 . https://doi.org/10.1145/3643991.3645071 [22] Della Porta, A., Lambiase, S., Palomba, F.: Do prompt patterns affect code quality? a first empirical assessment of chatgpt-generated code. In: Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering. EASE ’25, pp. 181–192. Association for Computing Machinery, New York, NY, USA (2025). https://doi.org/10.1145/3756681.3756938 . https://doi.org/10.1145/3756681.3756938 [23] Tanzil, M.H., Khan, J.Y., Uddin, G.: Chatgpt incorrectness detection in software reviews. In: Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. ICSE ’24. Association for Computing Machinery, New York, NY, USA (2024). https://doi.org/10.1145/3597503.3639194 . https://doi.org/10.1145/3597503.3639194 [24] Nashid, N., Ding, D., Gallaba, K., Hassan, A.E., Mesbah, A.: Characterizing multi-hunk patches: Divergence, proximity, and llm repair challenges. arXiv preprint arXiv:2506.04418 (2025) [25] Ogenrwot, D., Businge, J.: How ai coding agents modify code: A large-scale study of github pull requests. In: Proceedings of the 23rd International Conference on Mining Software Repositories (MSR 2026). IEEE/ACM, ??? (2026). Accepted for publication. [26] Ogenrwot, D., Businge, J.: Agenticflict: A large-scale dataset of merge conflicts in ai coding agent pull requests on github. In: Proceedings of the 3rd ACM International Conference on AI-powered Software (AIWare 2026). ACM, ??? (2026). Accepted for publication. [27] Ogenrwot, D., Businge, J.: Patchtrack: Analyzing chatgpt’s impact on software patch decision-making in pull requests. In: Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. ASE ’24, pp. 2480– 2481. Association for Computing Machinery, New York, NY, USA (2024). https: //doi.org/10.1145/3691620.3695338 . https://doi.org/10.1145/3691620.3695338 [28] Wang, G., Sun, Z., Ye, S., Gong, Z., Chen, Y., Zhao, Y., Liang, Q., Hao, D.: Do advanced language models eliminate the need for prompt engineering in software engineering? ACM Trans. Softw. Eng. Methodol. (2025) https://doi.org/10.1145/ 3771933 [29] Bettenburg, N., Just, S., Schröter, A., Weiss, C., Premraj, R., Zimmermann,

44

T.: What makes a good bug report? In: Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of Software Engineering. SIGSOFT ’08/FSE-16, pp. 308–318. Association for Computing Machinery, New York, NY, USA (2008). https://doi.org/10.1145/1453101.1453146 . https://doi.org/10.1145/1453101.1453146 [30] Zimmermann, T., Premraj, R., Bettenburg, N., Just, S., Schroter, A., Weiss, C.: What makes a good bug report? IEEE Transactions on Software Engineering 36(5), 618–643 (2010) https://doi.org/10.1109/TSE.2010.63 [31] Gousios, G., Storey, M.-A., Bacchelli, A.: Work practices and challenges in pullbased development: the contributor’s perspective. In: Proceedings of the 38th International Conference on Software Engineering. ICSE ’16, pp. 285–296. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/ 10.1145/2884781.2884826 . https://doi.org/10.1145/2884781.2884826 [32] Tsay, J., Dabbish, L., Herbsleb, J.: Influence of social and technical factors for evaluating contribution in github. In: Proceedings of the 36th International Conference on Software Engineering. ICSE 2014, pp. 356–366. Association for Computing Machinery, New York, NY, USA (2014). https://doi.org/10.1145/ 2568225.2568315 . https://doi.org/10.1145/2568225.2568315 [33] Vasilescu, B., Yu, Y., Wang, H., Devanbu, P., Filkov, V.: Quality and productivity outcomes relating to continuous integration in github. In: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering. ESEC/FSE 2015, pp. 805–816. Association for Computing Machinery, New York, NY, USA (2015). https://doi.org/10.1145/2786805.2786850 . https://doi.org/10.1145/2786805.2786850 [34] Zampetti, F., Ponzanelli, L., Bavota, G., Mocci, A., Di Penta, M., Lanza, M.: How developers document pull requests with external references. In: 2017 IEEE/ACM 25th International Conference on Program Comprehension (ICPC), pp. 23–33 (2017). https://doi.org/10.1109/ICPC.2017.30 [35] Midolo, A., Giagnorio, A., Zampetti, F., Tufano, R., Bavota, G., Penta, M.D.: Guidelines to prompt large language models for code generation: An empirical characterization. In: Proceedings of the International Conference on Program Comprehension (ICPC) (2026). To appear [36] Baltes, S., Angermeir, F., Arora, C., Barón, M.M., Chen, C., Böhme, L., Calefato, F., Ernst, N., Falessi, D., Fitzgerald, B., Fucci, D., Kalinowski, M., Lambiase, S., Russo, D., Lungu, M., Prechelt, L., Ralph, P., Tonder, R., Treude, C., Wagner, S.: Guidelines for empirical studies in software engineering involving large language models (2025) arXiv:2508.15503 [cs.SE]. arXiv preprint arXiv:2508.15503v3 [cs.SE] [37] Wang, X., Kim, H., Rahman, S., Mitra, K., Miao, Z.: Human-llm collaborative 45

annotation through effective verification of llm labels. In: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. CHI ’24. Association for Computing Machinery, New York, NY, USA (2024). https://doi.org/10.1145/ 3613904.3641960 . https://doi.org/10.1145/3613904.3641960 [38] Takehi, R., Voorhees, E.M., Sakai, T., Soboroff, I.: Llm-assisted relevance assessments: When should we ask llms for help? In: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR ’25, pp. 95–105. Association for Computing Machinery, New York, NY, USA (2025). https://doi.org/10.1145/3726302.3729916 . https://doi.org/10.1145/3726302.3729916 [39] Xu, Z., Khatri, V., Dai, Y., Liu, X., Li, S., Zhang, X., Yu, R.: Enhancing llm-based data annotation with error decomposition. In: Proceedings of the LAK26: 16th International Learning Analytics and Knowledge Conference. LAK ’26, pp. 325– 335. Association for Computing Machinery, New York, NY, USA (2026). https: //doi.org/10.1145/3785022.3785070 . https://doi.org/10.1145/3785022.3785070 [40] Sserunjogi, R., Ogenrwot, D., Businge, J.: Replication Package: A Stage-Based Empirical Study of LLM-Assisted Development. https://doi.org/10.5281/zenodo. 20451325 . https://doi.org/10.5281/zenodo.20451325 [41] Kalliamvakou, E., Gousios, G., Blincoe, K., Singer, L., German, D.M., Damian, D.: The promises and perils of mining github. In: Proceedings of the 11th Working Conference on Mining Software Repositories. MSR 2014, pp. 92–101. Association for Computing Machinery, New York, NY, USA (2014). https://doi.org/10.1145/ 2597073.2597074 . https://doi.org/10.1145/2597073.2597074 [42] Ramkisoen, P.K., Businge, J., Bladel, B., Decan, A., Demeyer, S., De Roover, C., Khomh, F.: Pareco: patched clones and missed patches among the divergent variants of a software family. In: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ESEC/FSE 2022, pp. 646–658. Association for Computing Machinery, New York, NY, USA (2022). https://doi.org/10.1145/3540250. 3549112 [43] Cohen, J.: Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin 70(4), 213–220 (1968) https: //doi.org/10.1037/h0026256 [44] Fleiss, J.L.: Statistical Methods for Rates and Proportions. John Wiley & Sons, New York (1981) [45] Willmott, C.J., Matsuura, K.: Advantages of the mean absolute error (mae) over the root mean square error (rmse) in assessing average model performance. Climate Research 30(1), 79–82 (2005). Accessed 2026-04-28

46

[46] Landis, J.R., Koch, G.G.: The measurement of observer agreement for categorical data. Biometrics 33(1), 159–174 (1977) https://doi.org/10.2307/2529310 [47] Businge, J., Openja, M., Nadi, S., Bainomugisha, E., Berger, T.: Clone-based variability management in the Android ecosystem. In: International Conference on Software Maintenance and Evolution, pp. 625–634. IEEE, ??? (2018) [48] Businge, J., Decan, A., Zerouali, A., Mens, T., Demeyer, S., De Roover, C.: Variant forks – motivations and impediments. In: Proceedings of the 29th Edition of the IEEE International Conference on Software Analysis, Evolution and Reengineering, pp. 867–877. IEEE Computer Society, ??? (2022). https://doi.org/10. 1109/SANER53432.2022.00105 [49] Businge, J., Openja, M., Nadi, S., Berger, T.: Reuse and maintenance practices among divergent forks in three software ecosystems. Empirical Softw. Engg. 27(2) (2022) [50] Kutner, M.H., Nachtsheim, C.J., Neter, J., Li, W.: Applied Linear Statistical Models, 5th edn. McGraw-Hill Irwin, Boston (2005) [51] James, G., Witten, D., Hastie, T., Tibshirani, R.: An Introduction to Statistical Learning: with Applications in R. Springer Texts in Statistics. Springer, New York (2013). https://doi.org/10.1007/978-1-4614-7138-7 [52] Albert, A., Anderson, J.A.: On the existence of maximum likelihood estimates in logistic regression models. Biometrika 71(1), 1–10 (1984) https://doi.org/10. 1093/biomet/71.1.1 [53] Schoenfeld, D.: Partial residuals for the proportional hazards regression model. Biometrika 69(1), 239–241 (1982) https://doi.org/10.1093/biomet/69.1.239 [54] Grambsch, P.M., Therneau, T.M.: Proportional hazards tests and diagnostics based on weighted residuals. Biometrika 81(3), 515–526 (1994) https://doi.org/ 10.1093/biomet/81.3.515 [55] Hsieh, H.-F., Shannon, S.E.: Three approaches to qualitative content analysis. Qualitative Health Research 15(9), 1277–1288 (2005) https://doi.org/10.1177/ 1049732305276687 [56] González-Prieto, Á., Perez, J., Diaz, J., López-Fernández, D.: Reliability in software engineering qualitative research through inter-coder agreement. Journal of Systems and Software 202, 111707 (2023) [57] Dı́az, J., Pérez, J., Gallardo, C., González-Prieto, Á.: Applying inter-rater reliability and agreement in collaborative grounded theory studies in software engineering. Journal of Systems and Software 195, 111520 (2023)

47

[58] Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., Zhang, J.M.: Large language models for software engineering: Survey and open problems. In: 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE), pp. 31–53 (2023). https://doi.org/10.1109/ ICSE-FoSE59343.2023.00008 [59] Hao, H., Hasan, K.A., Qin, H., Macedo, M., Tian, Y., Ding, S.H.H., Hassan, A.E.: An empirical study on developers’ shared conversations with chatgpt in github pull requests and issues. Empirical Softw. Engg. 29(6) (2024) https://doi.org/10. 1007/s10664-024-10540-x [60] Ehsani, R., Pathak, S., Chatterjee, P.: Towards detecting prompt knowledge gaps for improved llm-guided issue resolution. In: 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR), pp. 699–711 (2025). https: //doi.org/10.1109/MSR66628.2025.00107 [61] Otten, D., Stalnaker, T., Wintersgill, N., Chaparro, O., Poshyvanyk, D.: Prompting in practice: Investigating software practitioners’ use of generative ai tools. arXiv preprint arXiv:2510.06000 (2025) https://doi.org/10.48550/arXiv.2510. 06000 [62] Tan, Z., Li, D., Wang, S., Beigi, A., Jiang, B., Bhattacharjee, A., Karami, M., Li, J., Cheng, L., Liu, H.: Large language models for data annotation and synthesis: A survey. In: Al-Onaizan, Y., Bansal, M., Chen, Y.-N. (eds.) Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 930–957. Association for Computational Linguistics, Miami, Florida, USA (2024). https:// doi.org/10.18653/v1/2024.emnlp-main.54 . https://aclanthology.org/2024.emnlpmain.54/

48

Related documents

• Experimental Settings in LLM-Based Program Repair: A Study of Inputs, Tool Access, Feedback, and Validation#965480
• A11yLTLNav: Automatic Detection of Accessibility Navigation Failures#965481
• Detecting Argument-Swap Bugs Using Context-Enhanced Code Representations#965482
• A Large-Scale Empirical Study of Quality Assurance Practices and Gaps in AI Agents#965483
• CARE: A Responsibility-Oriented Architecture for Domain-Scoped Resolution in Modular and Upgradeable Smart Contracts#965484