Doc2CI: A Multi-Service Study of CI Configuration Generation Using Large Language Models

1

Doc2CI: A Multi-Service Study of CI Configuration Generation Using Large Language Models

arXiv:2608.01451v1 [cs.SE] 2 Aug 2026

Taher A. Ghaleb

Abstract—Adopting Continuous Integration (CI) often requires writing YAML configurations that are error-prone and challenging to maintain. Despite increasing LLM use in software engineering, their ability to generate CI configurations from natural language across services and model families remains unclear. This paper presents a large empirical study on using LLMs to generate CI configurations. We introduce D OC 2CI, a benchmark of 3,363 description-to-YAML pairs collected from the official documentation of four CI services, and evaluate 14 open-weight models from 7B–34B parameters together with GPT-4o and GPT-4.1, producing over 53,000 configurations. We assess both reference alignment and schema validity to determine whether the generated configurations are structurally valid. We further develop a failure taxonomy from a manual analysis of 385 configurations and examine why LLMs disagree. Across models and services, exact reference reproduction never exceeds 3.1%, and while 97% of outputs parse as YAML, only 71% satisfy service schemas. Larger models improve structural validity, but code specialization provides no consistent advantage over comparable general models. Model differences are driven largely by output completeness: for the same request, some models generate the expected fragment while others produce a full workflow. Finally, a training-free schema-guided repair method improves schema validity to 94%, while fine-tuning improves similarity to documentation but reduces standalone validity. This suggests that similarity and validity are distinct objectives for CI generation and motivate schema-aware evaluation and tooling for LLM-based configuration generation. Index Terms—Continuous Integration (CI), Large Language Models (LLMs), CI services, CI configuration, Benchmark

I. I NTRODUCTION ONTINUOUS Integration (CI) is central to modern software development, automating building and testing of source code on every change [1]. Configuring CI requires developers to write YAML files that are challenging [2] both syntactically, due to strict indentation and structure, and semantically, due to service-specific keywords, job matrices, reusable components, and third-party integrations [3]. This is particularly apparent for developers who are new to CI concepts or a certain service, and poor configurations can be associated with long build durations and failures [4, 5]. Large Language Models (LLMs) now automate many software engineering tasks, such as code generation, documentation, and bug fixing. However, generating CI configurations remains challenging because CI files are non-executable, declarative YAML with domain-specific semantics rather than imperative code [6], so success in code generation does not necessarily generalize to this setting. Prior work studied CI configuration challenges [7–10] and explored automated generation in limited contexts: Mastropaolo et al. [11] complete

C

Taher A. Ghaleb is with the Department of Computer Science, Trent University, Peterborough, Ontario, Canada (e-mail: taherghaleb@trentu.ca).

partial GitHub Actions workflows, while Zhang et al. [12] evaluate LLMs on manually crafted, syntax-focused scenarios. A prior single-service evaluation [13] showed that LLMs generate GitHub Actions configurations with at most 3% exact reference reproduction and that models pretrained on code do not outperform comparable general models. However, it remains unknown whether these findings generalize across CI services, model scales, and model families, or why models fail. This paper addresses these gaps through a large empirical study of LLM-powered CI configuration generation. We construct D OC 2CI, a benchmark of 3,363 description-toYAML pairs curated from the official documentation of GitHub Actions, CircleCI, GitLab CI, and Travis CI. We evaluate 14 open-weight models from 7B–34B parameters, pairing general families with code-specialized counterparts, together with GPT-4o and GPT-4.1, under zero-shot prompting [14], producing 53,808 configurations. Unlike prior evaluations focused primarily on similarity, we assess both reference alignment through eight lexical and structural metrics and validity through YAML parsing and conformance to service schemas. We further manually analyze a stratified sample of 385 configurations using open coding [15] to derive a failure taxonomy and examine the causes of model disagreement. Our results show that exact reproduction rarely exceeds 3%: although 97% of configurations parse, only 71% satisfy service schemas. Larger models improve structural accuracy, but similarity gains are limited, and code specialization offers no overall advantage over comparable general models. Performance varies substantially by the target CI service, with CircleCI being significantly harder than the others. Failures are driven mainly by misinterpretation and omission, while model disagreement primarily reflects differences in output completeness, from minimal fragments to complete workflows. Finally, schema-guided repair raises validity to 94%, whereas fine-tuning improves similarity at the cost of standalone validity, showing that these objectives must be optimized separately. Contributions. This paper makes the following contributions. • We introduce D OC 2CI, a benchmark of 3,363 descriptionto-YAML pairs across four CI services for evaluating natural language-based CI configuration generation. • We conduct a large-scale evaluation of 16 LLMs and 53,808 configurations, showing that high reference similarity does not necessarily imply valid CI configurations. • We study model scale and code specialization, showing that scale improves structural validity while code specialization provides no aggregate advantage and yields benefits only at larger scales within a model family.

2

We develop an LLM failure taxonomy through open coding of 385 configurations, revealing that failures are dominated by misinterpretation and omission, while LLM disagreement arises from output completeness. • We propose two complementary approaches for CI generation through fine-tuning (which improves reference similarity) and schema-guided repair (which improves validity), with their combination achieving the best trade-off. Paper organization. Section II gives background. Section III describes our study design. Section IV reports results across six research questions. Section V discusses implications and validity threats. Section VI presents related work. Finally, Section VII concludes the paper and outlines future work. •

II. BACKGROUND Continuous Integration and its configurations. CI merges developer changes frequently into a shared repository, triggering automated builds and tests for fast feedback [1]. These workflows are encoded in YAML, but each service defines its own vocabulary and structure: GitHub Actions uses on triggers and jobs with steps that call reusable actions; CircleCI uses jobs, workflows, orbs, and executors; GitLab CI uses top-level job names with stages, rules, and include; and Travis CI uses language, script, and deploy providers. The same intent can demand very different configurations across services. For example, a developer migrating a mature project from Travis CI to GitHub Actions may use an LLM to generate a workflow from a brief build description. The file may look fluent and on-topic yet still leak Travis-specific idioms into GitHub Actions, omit a required keyword, or produce only a fragment rather than a runnable configuration. These failures are not captured by similarity measures alone, as they appear only through parsing, schema validation, or expert review, which novice developers typically lack. This gap between fluent generation and reliable configuration motivates our study. Large Language Models for configuration. Transformerbased LLMs such as the GPT, Llama, Gemma, and Qwen families are trained on large text and code corpora and have been applied to generating configuration and infrastructure files from descriptions [16, 17]. Two factors structure the model landscape we study: scale, the parameter count, and specialization, whether a model is general or pretrained on source code. A third, often conflated with specialization, is instruction tuning, whether a model is trained to follow instructions rather than merely continue text. Distinguishing these factors is central to our analysis. Evaluating generated YAML. Given that CI YAML is nonexecutable and sensitive to structure, no single metric captures quality. Lexical metrics such as BLEU [18], CrystalBLEU [19], ROUGE-L [20], and chrF [21] capture token and character overlap. Embedding-based cosine similarity [22] captures semantic closeness, while structural metrics such as tree edit distance [23] and canonicalized Levenshtein distance capture layout. Validity is orthogonal: a configuration may resemble a reference yet fail to parse, or parse yet violate the service schema.

III. S TUDY D ESIGN Fig. 1 outlines our study: we curate description-to-YAML pairs from four CI services, prompt 16 models to generate a configuration for each description, and score each output for similarity and validity. Three analyses address our six research questions: quantitative scores address RQ1 to RQ3, open coding of a stratified sample addresses RQ4 and RQ5, and fine-tuning with schema-guided repair addresses RQ6. All data, generated outputs, scripts, and detailed results are in our replication package [24]. Each generation, fine-tuning, repair, and similarity analysis runs on a dedicated compute node in a shared cluster, each with one NVIDIA H100 GPU (80 GB HBM3), an AMD EPYC 9654 CPU (96 cores), 755 GB RAM, Rocky Linux 9.8, PyTorch 2.9.0, and CUDA 12.6.

A. The D OC 2CI Benchmark We developed a Selenium-based crawler to traverse the official documentation of GitHub Actions, CircleCI, GitLab CI, and Travis CI, following internal links and extracting instructional content while discarding navigation, sidebars, and footers. These docs explain keywords step by step and include example configurations and best practices, making them a natural source of description-to-YAML pairs. Each pair consists of a task-oriented natural language description (the section title plus its explanatory text) and a reference YAML that implements the described behavior. We manually validated the pairs, fixed indentation and whitespace, and removed entries with empty descriptions. Yet, the same information can appear multiple times in documentation, but for different reasons that require different treatments. First, providers reorganize and heavily cross-link their docs, so an identical section, with the same description and reference, is therefore reachable via several URLs, and the crawler collects each copy. We remove these exact-duplicate pairs, which account for 974 of the initial 4,337 crawled pairs (22.5%). Second, a section can document a single task using multiple example configurations, and the same description can legitimately map to different reference snippets, as illustrated in Fig. 2, where one description is accompanied by a configuration that inherits all secrets, and another that forwards only one. We keep these as distinct pairs, since each generated output is scored against its own reference, and merge them into a single target only when fine-tuning (Section IV-F). After removing exact duplicates, D OC 2CI contains 3,363 unique pairs (3,317 unique descriptions): 725 for GitHub Actions, 786 for CircleCI, 841 for GitLab CI, and 1,011 for Travis CI. Descriptions and reference YAML configurations vary in complexity, from a single keyword to multi-job workflows with matrices and conditional execution. One characteristic of documentation-derived references shapes our interpretation throughout: many references are minimal fragments that illustrate one keyword (e.g., a singleline cache: pip or a two-line permissions block), and some are non-workflow artifacts (an action.yml metadata file, a Helm values.yaml, or a Kubernetes manifest) or, in migration guides, snippets in another tool’s language.

3

Score and adapt Research questions

Generate

Curate D OC 2CI

Similarity: Cosine, Euclidean, BLEU, CrystalBLEU, ROUGE-L, chrF, Tree-Edit, Levenshtein CI docs: GitHub Actions CircleCI GitLab CI Travis CI

Curate pairs

D OC 2CI 3,363 pairs

Prompt 16 LLMs: 2 proprietary, 14 open-weight (7 general, 7 code; small/medium/large)

53,808 configurations

RQ1 Generation quality RQ2 Scale and specialization RQ3 Target CI service

Validity: Parse, Schema

Open coding: Stratified samples

RQ4 Failure taxonomy RQ5 Models disagreement

Fine-tune & Schema repair (D OC 2CI train/test)

RQ6 Fine-tuning and Schema-guided repair

Fig. 1. Overview of our study. Cylinders denote datasets, boxes denote processing steps, and stacked shapes group research questions. We curate descriptionto-YAML pairs from four CI services into D OC 2CI, generate configurations with 16 models, and analyze outputs through quantitative evaluation (RQ1–RQ3), open coding of stratified samples (RQ4–RQ5), and fine-tuning with schema-guided repair (RQ6).

Fig. 2. Example of a D OC 2CI entry from the GitHub Actions documentation. The section title and explanatory prose form the description, while each YAML block is a reference configuration. Here, one description maps to two valid references: one inherits all secrets, and one forwards a single secret.

B. Selected Models We retain the six models evaluated in the prior singleservice study [13]: two proprietary models (GPT-4o [25] and GPT-4.1 [26]) and two general–code model pairs (Llama3.18B [27] with CodeLlama-7B [28], and Gemma3-12B [29] with CodeGemma-7B [30]). This allows direct comparison with the earlier results while expanding the evaluation to parameter scale and code specialization. To vary each factor cleanly, we add the Qwen2.5 family, the one open family with size-matched general and code-specialized checkpoints at 7B, 14B, and 32B, yielding three directly comparable general-versus-code pairs with family and scale fixed and specialization varying. Around this basis, we include all other released checkpoints from the selected model families within a locally runnable 7B–34B range: CodeLlama at 13B and 34B, Gemma3 at 27B, and Gemma2 at 9B. This yields 14 openweight models grouped into three size tiers: small (7–8B), medium (9–15B), and large (> 15B). The tier boundaries follow the available model sizes, with no checkpoint between 14B and 27B and a natural separation between the 7–8B and

9–14B clusters. This setting is not a complete family-by-tier grid, as released checkpoints do not cover all combinations. In the 7–34B general range, only Llama3.1-8B is available (the next checkpoint is 70B), and no single Gemma generation spans all tiers, requiring Gemma2-9B and Gemma3-12B/27B. CodeGemma is available only at 7B. Consequently, only the three Qwen2.5 pairs match on both scale and generation; Llama and Gemma comparisons also vary in size or generation. We therefore use Qwen2.5 for the controlled specialization test and treat the remaining families as broader coverage of scale and specialization trends. The proprietary GPT-4o and GPT-4.1 serve as frontier references and, like the open models, were evaluated on all four services. Because their parameter counts and training data are undisclosed, we exclude them from the controlled scale and specialization analyses in RQ2 while retaining them for overall quality and cross-service comparisons (RQ1 and RQ3). GitHub Actions, the service used in prior work and the validation setting for our study (Section IV-A), is also the most common CI service in public code, providing a favorable setting for closed models and an upper-bound reference when interpreting per-service results. Open models were run locally through Ollama with default 4-bit quantization, allowing even 32B models to run on a single 80 GB GPU, and with their maximum supported context length. GPT models were accessed through the OpenAI API, and the exact Ollama model tags are listed in the replication package. All models used temperature=0; with fixed quantization and greedy decoding, generation is deterministic and reproducible across runs. C. Prompting and Generation We use a single standardized zero-shot prompt (Listing 1) to assess each model’s ability to turn a description into a structurally valid configuration without in-context examples, reflecting realistic first-attempt developer usage. The serviceparameterized prompt requests a minimal, single, valid configuration with no comments or markdown. We stripped markdown fences from generated text before storage. Prompting all 16 models across four services yielded 53,808 configurations.

4

System: You are a highly skilled software developer and DevOps engineer. User: Given the description below, generate a minimal and single valid {Service} YAML configuration. Do not provide any reasoning. Only output the single YAML configuration, with no comments, markdown, or extra formatting. The YAML must match the described task exactly. Do not add any unneeded directives unless told. {Description}

Listing 1. The standardized zero-shot prompt. {Service} is instantiated per CI service and {Description} holds the natural-language task.

D. Evaluation Metrics We score each generated configuration against its reference along two dimensions. Similarity. We use eight metrics capturing lexical, semantic, and structural agreement: Cosine similarity of sentence embeddings from all-MiniLM-L6-v2 [22] (semantic metric); Euclidean similarity (inverse embedding distance); BLEU [18]; CrystalBLEU [19], which reduces the influence of shared n-grams; ROUGE-L [20]; chrF [21]; tree edit similarity on parsed YAML trees [23]; and canonicalized Levenshtein similarity. All are scaled to [0, 1] (higher is better), except for cosine similarity, whose range is [−1, 1]. A configuration is an exact match when its canonicalized Levenshtein similarity to the reference is 1. We use all-MiniLM-L6-v2 for the primary cosine metric because it is a compact encoder with strong semantic textual similarity performance relative to its size [22, 31], is widely used in recent software engineering studies [32, 33], and efficiently scores all 53,808 outputs on CPU. Validity. Similarity alone does not guarantee that a configuration is valid. Hence, we add two validity checks that range from lenient to strict. Parse validity holds when the output loads as a YAML mapping. Schema validity holds when a configuration also conforms to its service’s JSON schema: the GitHub Actions [34] and Travis CI [35] schemas from SchemaStore, the CircleCI schema from the official CircleCI YAML language server [36], and the GitLab CI schema from the GitLab repository, used by its web editor [37]. We validate each output with the JSON Schema validator for the draft declared by its schema. We parse with a YAML 1.2 loader so that GitHub Actions on: is treated as a string rather than a boolean, and we use the same procedure for all four services to keep checks uniform and reproducible offline. Each service also has a dedicated validator, namely actionlint [38] for GitHub Actions, circleci config validate [39] for CircleCI, the CI Lint tool [40] for GitLab CI, and travis-yml [41] for Travis CI. We do not use these as quality metrics as each assumes a complete, standalone configuration file, but our prompt requests a minimal configuration, and much of the documentation and generated outputs are intentionally partial fragments (Section III-A). Linting these fragments would confuse incompleteness with incorrectness, and each validator applies service-specific, non-comparable rules. Schema conformance, applied uniformly across services, already captures the structural validity these validators check, which is why we use it as our strict check.

E. Sampling and Open Coding To understand why models fall short (RQ4) and diverge on identical inputs (RQ5), we complement automatic metrics with manual qualitative analysis. Automatic scores measure distance between an output and its reference but do not reveal failure modes or causes of divergence, which coding uncovers under the same protocol. We draw stratified random samples of generated configurations with a target confidence level and margin of error, and analyze them using open coding [15]. For each sampled configuration, the primary coder (software engineering researcher with CI expertise) inspected three artifacts side by side: the task specification (service name and description), the documentation reference, and the model output. Coding followed constant comparison, starting from a small seed codebook and iteratively adding or merging codes as new discrepancy types emerged. Earlier items were revisited when the codebook changed, after which the finalized codebook was applied to the full sample and open codes were consolidated into higher-level categories. Reliability. For independent coding, we assessed reliability through a second coder, a researcher with the same software engineering and CI expertise, on a service-stratified subsample using the finalized codebook and target service labels. We report Cohen’s κ [42, 43] between coders on both the three-point verdict (usable, partial, unusable) and the binary usable-versus-not decision. All disagreements were resolved through discussion to reach consensus (Section IV-D), and the subsample size and resulting coefficients are reported with the analyses they validate. F. Statistical Analysis We use the Friedman test [44] to compare models on perdescription paired scores within each service, and to compare CI services on the 14 common models’ per-service means, following standard non-parametric practice [45, 46]. We use the Wilcoxon signed-rank test [47] for paired contrasts, the Mann-Whitney U test [48] for the unpaired general-versuscode comparison, and apply Holm correction [49] for multiple comparisons. We report effect sizes alongside p-values: Spearman’s ρ [50] for model size–quality association, Cliff’s δ for the general-versus-code contrast, and bootstrap 95% confidence intervals for per-service means. IV. E VALUATION R ESULTS A. RQ1: How well do LLMs generate CI configurations? 1) Motivation: To understand how well LLMs generate usable CI configurations, we establish a baseline at scale. Developers care about two distinct properties that a single score conflates: whether a configuration matches their intent and whether it is usable, meaning that it parses and conforms to the service schema. Prior work evaluated models on a single service using similarity alone, leaving unclear whether similar outputs are runnable. We evaluate all 16 models across four services while separating similarity from validity, distinguishing lexically close but unusable outputs from usable but differently phrased ones.

5

2) Methodology: For each generated configuration, we compute the eight similarity metrics, using mean cosine similarity as the primary metric and releasing the others in the replication package. Cosine similarity is our main metric because it captures semantic agreement despite variation in CI YAML (reordered keys, quoting, whitespace), which lexical and edit-based metrics penalize even for equivalent configurations. We show below that all eight metrics rank models consistently, so emphasizing cosine does not affect our conclusions. We also compute exact-match rate (canonicalized Levenshtein = 1) and two validity checks: YAMLmapping parse validity and conformance to the service JSON schema. We report per-model means across services and perservice validity. We compare our GitHub Actions results with the earlier GPT-4o evaluation [13] to verify consistency and apply the statistical tests described in Section III-F for model comparisons. 3) Results: Our GitHub Actions results are consistent with the earlier GPT-4o evaluation: GPT-4o achieves a mean cosine of 0.675 and an exact-match rate of 2.6%, compared with the previously reported 0.69 and 3%. Across the four services, the exact-match rate does not exceed 3.1% (GPT-4o) and remains below 2% for most models (Table I), confirming at scale that LLMs rarely reproduce reference configurations. GPT-4.1 (0.668) and GPT-4o (0.652) lead on cosine similarity, and the best open model (Qwen2.5-Coder-14B) achieves a relatively comparable score of 0.639, while the weakest (CodeLlama13B) achieves 0.532. Models differ significantly within every service (Friedman, all p < 10−100 ). GPT-4.1 achieves a small but significant edge over GPT-4o (Wilcoxon p < 10−18 , rankbiserial r = 0.19), and the proprietary lead over the strongest open model is likewise significant but small (GPT-4.1 vs. Qwen2.5-Coder-14B, p < 10−39 , r = 0.26; the two GPT models vs. the 14 open models, Cliff’s δ = 0.18 per-configuration). The other seven metrics agree with cosine on model ranking: across the 16 models, the per-model Spearman correlation between cosine and each other metric ranges from 0.73 for CrystalBLEU to 0.92 for chrF and Euclidean similarity, with all lexical, structural, and edit-based metrics in between. This indicates that the observed differences are driven by model outputs rather than by the choice of similarity metric, with CrystalBLEU as the only outlier because it downweights common n-grams shared by CI files. Configuration validity reveals different trends for both YAML parsing and schema conformance (Table II). Almost all configurations parse as YAML mappings (97.1% overall), ranging from 94.0% for Gemma2-9B to 99.0% for Qwen2.5Coder-14B, with slight variation across tiers (96.9% small, 96.6% medium, 97.9% large). In contrast, only 70.8% conform to the service schema, with schema validity ranging from 56.3% for CircleCI to 83.0% for Travis CI. Schema validity is non-monotonic with scale (67.7% small, 63.2% medium, 79.3% large) due to a drop in Gemma models. Still, it differentiates models clearly: the best open models, Qwen2.5-32B (82.2%) and Qwen2.5-Coder-32B (83.0%), approach GPT-4.1 (87.8%), while Gemma2-9B falls to 43.6%. One illustrative case shows a mismatch between similarity

TABLE I P ER - MODEL QUALITY ACROSS FOUR SERVICES , GROUPED BY SIZE TIER AND RANKED BY MEAN COSINE . B OLD MARKS THE BEST VALUE WITHIN EACH TIER / METRIC . Ú DENOTES GENERAL AND Ð DENOTES CODE - SPECIALIZED . P ROPRIETARY GPT MODELS ( SHADED ) ARE EXCLUDED FROM RQ2. Model

Type

Cosine

Exact%

Parse%

Proprietary (all services) GPT-4.1 GPT-4o

Ú Ú

0.668 0.652

2.7 3.1

96.7 98.3

Small (7 to 8B) Qwen2.5-7B Llama3.1-8B Qwen2.5-Coder-7B CodeGemma-7B CodeLlama-7B

Ú Ú Ð Ð Ð

0.609 0.602 0.600 0.571 0.570

1.1 1.1 0.8 0.9 1.2

97.2 96.6 97.0 96.4 97.6

Medium (9 to 15B) Qwen2.5-Coder-14B Qwen2.5-14B Gemma2-9B Gemma3-12B CodeLlama-13B

Ð Ú Ú Ú Ð

0.639 0.612 0.586 0.583 0.532

1.5 0.7 1.5 2.0 0.7

99.0 96.5 94.0 97.2 96.3

Large (> 15B) Qwen2.5-Coder-32B Qwen2.5-32B Gemma3-27B CodeLlama-34B

Ð Ú Ú Ð

0.638 0.636 0.606 0.588

1.5 1.9 0.7 1.3

97.7 97.5 97.7 98.6

TABLE II B Y SERVICE : MEAN COSINE , PARSE VALIDITY, AND SCHEMA VALIDITY ( OVER ALL 16 MODELS ). Service

Cosine

Parse%

Schema%

GitHub Actions CircleCI GitLab CI Travis CI

0.641 0.550 0.586 0.640

97.1 94.9 96.9 99.1

66.6 56.3 73.0 83.0

Overall

0.606

97.1

70.8

and validity within a single service. On GitHub Actions, GPT-4o has the highest similarity of any model yet is only 17.2% schema-valid, because it follows the minimal instruction literally and generates fragments in 80% of cases (e.g., a bare permissions block) that match many fragment references but are not valid standalone files. GPT-4.1 instead generates complete GitHub Actions workflows (2% fragments) and achieves 94.8% schema validity, indicating a sharp withinservice split (McNemar p < 10−100 ). This completeness effect is specific to GitHub Actions: on CircleCI, GitLab CI, and Travis CI, GPT-4o produces complete files (near 0% fragments) with 84%–92% schema validity, and its low GitHub Actions validity therefore reflects output completeness rather than an inability to produce valid YAML. On CircleCI, the pattern reverses: GPT-4o produces complete, schema-valid configurations that diverge from short references, ranking ninth of sixteen models on similarity and well below the best open model (rank-biserial r = −0.28). GPT-4.1, by contrast, ranks first on CircleCI, GitLab CI, and Travis CI and second on GitHub Actions, without GPT-4o’s completeness-driven variability. Similarity thus substantially overstates usefulness: a lexically close configuration can still be unparseable, schemainvalid, or only a valid fragment rather than a runnable file.

6

100

Schema validity (%)

GPT-4.1

80

GPT-4o

60

40

20

general code proprietary

0 0.525 0.550 0.575 0.600 0.625 0.650 0.675 0.700

Mean cosine similarity

Fig. 3. Similarity is not validity: each model’s mean cosine (x) versus schema validity (y), across all four services and nearly uncorrelated across models.

Fig. 3 shows both measures across all models. RQ1 Summary. Similarity is an unreliable proxy for usable CI configurations: only 70.8% of outputs satisfy service schema despite 97.1% parsing successfully. Exact reproduction is rare (≤ 3.1%). GPT-4o highlights this gap by achieving the highest GitHub Actions similarity but only 17.2% schema-valid outputs. Schema validation is therefore essential for evaluating LLM-generated CI. B. RQ2: How do model scale and code specialization correlate with quality? 1) Motivation: Model choice for CI generation involves a trade-off between cost and quality. Larger models require substantially more resources, while code-specialized models are often expected to perform better on structured targets such as YAML. Whether either factor improves CI configuration quality remains unclear: the constrained nature of CI files may limit scale benefits, and prior single-service evidence suggests code specialization may not help. We therefore isolate model scale and code specialization across services to determine which, if either, provides a measurable advantage. 2) Methodology: Across the 14 open models, we measure the association between parameter count and quality using Spearman’s ρ on per-model means, with both semantic (mean cosine) and structural metrics (tree edit, ROUGE-L, chrF) to capture effects beyond embedding similarity. We compare size tiers on paired per-configuration scores using Mann-Whitney tests and Cliff’s δ. To evaluate code specialization, we first compare general and code-specialized models in aggregate, then perform the cleaner paired analysis: each general model versus its code-specialized counterpart on identical descriptions using the Wilcoxon signed-rank test. The three sizematched Qwen2.5 pairs (7B, 14B, 32B) isolate specialization by holding scale and model family fixed. 3) Results: Scale improves structural validity, but its effect on embedding similarity is modest within this lineup of midto-large models. Parameter count correlates significantly with

the structural metrics (tree edit ρ = 0.65, ROUGE-L ρ = 0.61, chrF ρ = 0.65, all p < 0.05), yet its correlation with mean cosine is positive and not significant at this sample size (ρ = 0.47, p = 0.09, n = 14). Mean cosine is essentially flat across the small and medium tiers (both ≈ 0.59) and only modestly higher in the large tier (0.62). Across the 47,082 paired per-configuration scores from the 14 open models, the small-to-medium contrast is not even significant (Cliff’s δ = −0.010, Mann-Whitney p = 0.12), and although the medium-to-large and small-to-large contrasts are significant (p < 10−3 ) their effect is negligible in magnitude (Cliff’s δ = −0.068 and −0.081, both below the 0.147 negligible threshold [51, 52]). Scale therefore has at most a practically negligible effect on embedding similarity once every model is already at least 7B parameters. The usable rate from the open coding (Section IV-D) provides a clearer view of the scale effect, increasing monotonically from 34.7% for small models to 40.0% for medium models and 46.9% for large models, with the proprietary models highest at 50.0%. Code specialization gives no clear aggregate benefit on similarity, and its effect is non-monotone in scale. General and code models have almost identical mean cosine (0.605 versus 0.591; Mann-Whitney p = 0.46; Cliff’s δ = 0.27, small), though code models are somewhat more schema-valid in aggregate (73.8% versus 65.0%). The within-family contrasts, each measured on the identical descriptions evaluated across all four services, explain the flat aggregate effect: the code-specialized variant wins only at the two larger Qwen sizes (Qwen2.5-Coder-14B 0.639 versus Qwen2.5-14B 0.612, p < 10−27 ; Qwen2.5-Coder-32B 0.638 versus Qwen2.532B 0.636, p < 10−2 ), whereas the general model wins at the smallest Qwen size (Qwen2.5-7B 0.609 versus Qwen2.5Coder-7B 0.600, p < 10−2 ) and in the Llama and Gemma families (Llama3.1-8B 0.602 versus CodeLlama-7B 0.570, p < 10−22 ; Gemma2-9B 0.586 versus CodeGemma-7B 0.571, p < 10−6 ). The Llama and Gemma pairs are not sizematched: each code-specialized model is smaller than its general counterpart (7B versus 8–9B), and for Gemma a generation older. Part of the general model’s advantage in these two families may therefore reflect scale rather than a cost of specialization, and only the Qwen pairs cleanly identify specialization effects. Even where significant, these withinfamily effects are small (|r| ≤ 0.22) and point in opposite directions across families. Thus, code pretraining helps only at larger scale within a strong instruction-tuned family and is neutral or harmful otherwise. Fig. 4 shows both patterns together: the general line declines at the medium tier before recovering, and the code and general lines converge only at the large tier on either factor. RQ2 Summary. Scale predicts CI quality better than code specialization. The overall general-vs-code comparison shows no significant advantage (Cliff’s δ = 0.27, small, p = 0.46), and Qwen2.5 pairs indicate specialization helps only at 14B and 32B while slightly hurting at 7B. Larger general models are thus strong baselines, with code specialization offering limited gains only at higher scales.

7

80

0.615

Schema validity (%)

Mean cosine similarity

0.620

0.610 0.605 0.600 0.595 0.590 0.585

general

75

70

65

60

code

0.580 Small (7-8B)

Medium (9-15B)

Large (>15B)

Small (7-8B)

Medium (9-15B)

Large (>15B)

Fig. 4. Quality by size tier for code-specialized and general open models (tier means): similarity (left) and schema validity (right). The two types converge only at the large tier.

C. RQ3: How does the target CI service affect quality? 1) Motivation: The four services in D OC 2CI express the same automation concepts through different surface syntax, from GitHub Actions’ jobs and steps to Travis CI’s flat lifecycle phases, and they appear unevenly in public code and therefore in pretraining data. A model that writes fluent GitHub Actions may thus falter on CircleCI’s orbs or GitLab’s stages, and a study that measures only the most common service would produce an optimistic estimate of performance that may not transfer across services. Quantifying the effect of target service reveals where LLM assistance is reliable and where human oversight remains necessary, while directly testing whether single-service conclusions from prior work generalize across the CI ecosystem. 2) Methodology: Because all 16 models were run on every service, the cross-service comparison uses the full set. For each model, we compute its mean cosine per service, then test whether the target service changes quality with a Friedman test on the paired per-model means, which controls for model identity by treating each model as its own block. A significant omnibus result is followed by Holm-corrected Wilcoxon signed-rank contrasts between service pairs. We report bootstrap 95% confidence intervals on each service mean and quantify every pairwise gap with Cliff’s δ on the per-configuration scores, so that significance at this sample size is not mistaken for a large effect. 3) Results: The target service significantly affects quality (Friedman χ2 = 43.3, p < 10−8 ). Mean cosine is highest for GitHub Actions (0.641, 95% confidence interval [0.630, 0.651]) and Travis CI (0.640, [0.618, 0.664]), which are statistically tied, then GitLab CI (0.586, [0.570, 0.601]), and lowest for CircleCI (0.550, [0.526, 0.571]). Five of the six pairwise contrasts are significant after Holm correction; only GitHub Actions and Travis CI are statistically indistinguishable (Holm = 0.94, Cliff’s δ = 0.004), while CircleCI is significantly the hardest service. The per-configuration effect sizes confirm the ordering and put it in perspective: CircleCI’s disadvantage relative to the easy services is small (Cliff’s δ up to 0.23). The target service, though the largest single

factor we measure, therefore moves similarity only modestly, though by more than model scale or specialization do. Two forces explain the CircleCI gap, both visible in the open coding (Section IV-D): its documentation contributes many server-installation references (Helm values.yaml, Kubernetes manifests) that are not workflow configurations, and models frequently leak GitHub Actions or GitLab idioms into CircleCI output. GitHub Actions and Travis CI are the easiest, with GitHub Actions benefiting from its prevalence in training data and Travis CI from its short references that are simple to reproduce. The manual coding agrees on the ordering and quantifies it: usable rates are highest for Travis CI (56.0%) and GitHub Actions (53.0%) and far lower for CircleCI (26.7%) and GitLab CI (27.1%), consistent with CircleCI also being the least schema-valid service. RQ3 Summary. CI service dominates generation outcomes: CircleCI is the most challenging, while GitHub Actions and Travis CI are easier due to service-specific formats and idioms. Similarity differences are modest, but usability varies substantially (56% usable on Travis CI vs. 27% on CircleCI). Single-service evaluations therefore overstate general performance. D. RQ4: Why do LLMs fall short in generation? 1) Motivation: Similarity and validity scores locate a model on a scale but do not reveal what a wrong configuration got wrong: a low cosine can indicate a missing step, an invented key, a wholesale misreading of the task, or merely a shorter but correct output than the reference. Those causes call for entirely different remedies, from better prompting to schema tooling to more training data, and no automatic metric distinguishes among them. A grounded, human-coded account of failure is therefore what turns the quantitative difference into an actionable diagnosis, and it is the only way to separate genuine capability limits from artifacts of using documentation as the ground truth source. We build such an account by opencoding a stratified sample of outputs into a failure taxonomy that later research and tooling can target directly.

8

2) Methodology: Following the protocol of Section III-E, we drew a stratified random sample of 385 generated configurations (95% confidence level, ±5% margin of error) and open-coded them. We stratified the sample across the four services (83 GitHub Actions, 90 CircleCI, 96 GitLab CI, 116 Travis CI), proportional to each service’s share of the deduplicated benchmark and, within each service, approximately evenly across all 16 models under a fixed random seed, ensuring every model was coded on every service. Because the sample is balanced across models rather than weighted by output volume, the taxonomy characterizes the range of failure modes and how they vary with model and service, not their frequency in the generated outputs. Coding instrument. For each configuration, the coder recorded a three-part structured judgment for the item’s target service. The first part is an overall verdict on a three-point scale, defined so the boundaries are clearly decidable rather than a matter of degree: • usable: runs on the target service, or would after only trivial edits (filling an obvious placeholder, renaming an identifier, reformatting), and expresses the documented behavior; a different-but-valid realization counts as usable; • partial: implements the mechanism described in the task but needs a non-trivial fix, such as a missing required key or a wrong value on the key the task is about; • unusable: does not express the documented behavior, is schema-invalid for the target service, or is degenerate. A degenerate flag is recorded on top of the verdict for output that is empty, prose instead of YAML, a stub, or repetitive, so that degeneration is counted without requiring the coder to separate “wrong” from “broken” on the main scale. When the reference is not a comparable standalone configuration for the target service (a bare fragment, a non-workflow artifact such as a Helm values.yaml, or a snippet in another tool’s language), the coder does not grade by matching it but on whether the output expresses the documented concept for the service, and records a reference-quality code. The second part is one or more issue codes that name each discrepancy (e.g., missing_key, wrong_service_idiom, hallucinated_step) and the YAML key it touches. The third is an attribution of each discrepancy to one of three sources: the model (a genuine error given a clear description), the description (vague, underspecified, misleading, or based on a fragment or non-workflow reference, making the output defensible), or acceptable variation (a legitimate alternative, not a defect). This attribution separates genuine model failures from artifacts of documentation-derived references. Reliability. A service-stratified subsample of 77 configurations (95% confidence level, ±10% margin of error) was independently re-coded by a second researcher using the same codebook, with the target service provided for each item. We report Cohen’s κ on the three-point verdict and on the binary usable-versus-not decision. Agreement on the threepoint verdict was substantial, κ = 0.78, and rises further on the binary usable-versus-not decision on which our findings are mainly based, κ = 0.81. The 11 disagreements out of 77 are all one level apart, either usable versus partial or partial

versus unusable, and each turns on a single sub-judgment the codebook leaves open, namely whether a required edit is trivial, or whether an output that uses the right mechanism with an invalid key is repairable enough to keep. None is a gross disagreement about whether the configuration works, which the earlier five-point scale had produced by splitting the same judgment across four boundaries. We resolved each disagreement through a consensus discussion meeting, in which the two coders re-examined the specification, reference, and output together, discussed the competing verdicts, and settled on the single verdict used in the analysis. Considering that all disagreements were adjacent rather than gross, discussion converged in every case, and each resolution clarified how the borderline sub-judgment above should apply so that the same reading was used consistently across the full sample. We open-coded all 385 configurations, yielding 560 discrete coded issues, and consolidated the codes into six categories. We report the overall verdict distribution, category frequencies, and how these vary by service, model size, and model type. 3) Results: Overall, 41.3% of sampled configurations are usable, 31.7% are partial, and 27.0% are unusable; 2.3% fall under the degenerate flag (empty, non-YAML, or repetitive). Table IV shows the six failure categories. Misinterpretation is the most common (36.1% of issues), covering wrong values, misread tasks, and wrong triggers, followed by Omission (24.1%). The remaining issues divide almost evenly: Reference/data quality (13.4%), a category absent from prior single-service coding, captures cases where the documentation reference is itself a fragment, a more elaborate configuration than the description warrants, a non-workflow artifact, or a snippet in another tool’s language, so that a correct output is penalized; Superfluous additions (13.0%); and Invalidity (12.9%, schema violations, non-YAML output, and crossservice idiom leakage, reserved for output that does not parse or violates the service schema, not valid YAML that is merely off-target). Attribution reveals the source of the observed failures: of the 560 coded issues, 432 (77.1%) are genuine model failures, 84 (15.0%) trace to the description or reference, and 44 (7.9%) are acceptable variation. Table III grounds the categories in concrete cases drawn from the coding. The taxonomy varies across dimensions. By service, CircleCI has the highest share of reference-quality issues (20.1% of its issues, tied with GitHub Actions), reflecting the values.yaml and Kubernetes-manifest references of RQ3, alongside high invalidity (17.5% of its issues) from cross-service idiom leakage. GitLab CI failures are instead dominated by content errors split between misinterpretation (36.5%) and omission (33.3%), where models generate a plausible generic workflow that ignores the specific keyword the task requires (e.g., a generic cache block that drops the cache:key:files directive, or a plain needs list where the task requires needs:project). By scale, the nature of failure shifts: invalidity peaks in the medium tier (19.6% of issues, against 12.8% small) and then drops to 7.7% in the large tier, while misinterpretation and superfluous additions rise (superfluous additions reach 14.6% in the large tier). Larger models thus fail more by writing valid YAML

9

TABLE III R EPRESENTATIVE EXAMPLES OF OPEN - CODED FAILURE CATEGORIES ( MODEL OUTPUTS ARE EXCERPTED ). Failure (model, service)

Task

Reference

Model output (excerpt)

Misinterpretation (CodeLlama-13B, GitHub Actions)

Run a job in a container with a Redis service container.

services: redis: image: redis

container: redis steps: - run: npm test redis set as the job container, not a service

Invalidity: idiom leakage (CodeGemma-7B, CircleCI)

Specify the CircleCI configuration version.

version: 2.1

jobs: build_and_test: runs-on: ubuntu-latest steps: - uses: actions/checkout GitHub Actions syntax in a CircleCI file

Omission (Llama3.1-8B, GitLab CI)

Add the Cobertura coverage report to artifacts.

artifacts: reports: coverage_report: coverage_format: cobertura

artifacts: paths: [$ARTIFACTS] omits reports:coverage_report

Reference/data quality (GPT-4o, GitHub Actions)

Give the --config-file-path file that targets converted reusable workflows.

reusable_workflows: - name: ... target_url: ...

jobs: audit: ... migrate: ... a workflow; the reference is a configuration file

TABLE IV RQ4 FAILURE TAXONOMY OVER 560 ISSUES FROM 385 CONFIGURATIONS . Category Misinterpretation Omission Reference/data quality Superfluous additions Invalidity Surface variation

%

Representative codes

36.1 wrong value, misread task, wrong trigger 24.1 missing key/step, incomplete step 13.4 fragment, over-elaborate, non-workflow or wrong-language reference 13.0 hallucinated step, unnecessary configuration, over-specification 12.9 schema violation, non-YAML, idiom leakage 0.5 renamed identifier, reformatting

that means the wrong thing or adds unrequested content than by generating broken output. Outright degeneration is concentrated rather than spread across scale: it accounts for only 2.3% of verdicts overall and is essentially confined to CodeLlama, whose prose lead-ins drive the medium tier’s 6.7% degenerate rate and raise the code models’ degenerate rate (5.2%) well above the general models’ (0.0%). RQ4 Summary. Only 41.3% of sampled outputs are usable as-is. Most failures are semantic rather than syntactic, driven by keyword misinterpretation (36%) and omission (24%), while reference artifacts (13%) reveal a limitation of documentation-derived evaluation. As models scale, the remaining challenge shifts from producing valid YAML to producing the intended configuration.

E. RQ5: Why do LLMs disagree in their generations? 1) Motivation: Results of RQ1 to RQ3 reveal that one model may score near-perfectly while others score near zero on the same input. Such disagreements could mean several different things, each with a different practical response. If they reflect genuinely different but defensible task interpretations, ensembling or sampling several models is the right lever; if they reflect a capability threshold that only some models clear, model selection is; and if they are an artifact of how the reference is written or scored, the implication concerns

the benchmark design and evaluation procedure rather than the models. RQ4 characterizes failures across all cases, but the key disagreements are where the metric is least stable and therefore where its interpretation most needs investigation. This RQ addresses what drives the largest inter-model gaps. 2) Methodology: To study why models diverge on identical inputs, we identify 852 high-dispersion descriptions where at least 14 models generated outputs, the best cosine score was ≥ 0.72, and the worst was ≤ 0.35. From these, we draw a service-proportional random sample of 86 cases (95% confidence level, ±10% margin of error, with median cosine range of = 0.79). For each case, through collaborative review, we inspect the highest-scoring output and the failures of the two lowest-scoring outputs, then apply the constant-comparison open-coding procedure from Section III-E to assign the dominant driver of the best-versus-worst split. The resulting five categories are: completeness mismatch (fragment references matched by one model but expanded into full workflows or under-produced by others), degenerate or invalid (placeholder, prose, non-YAML, or hallucinated outputs), reference quality (the reference is not a CI workflow configuration, e.g., a Helm values.yaml), valid alternative (both outputs are valid but differ in form), and spec underdetermined (the description admits multiple valid configurations). Table V summarizes the categories with representative examples. 3) Results: Most inter-model disagreements do not reflect differences in generation ability (Table V). Completeness mismatch accounts for 51 of the 86 cases (59%): the reference is a minimal fragment (e.g., permissions:, dist:, or git: depth: false), the winner reproduces that snippet, and lower-scoring models generate a complete, often valid, workflow around it. Another 15 cases (17%) are referencequality artifacts, and 2 (2%) are valid alternatives or underdetermined specifications. Together, these account for 68 of 86 disagreements (79%), indicating benchmark or outputcompleteness effects rather than capability gaps. Only 18 cases (21%) are genuine failures, where losing outputs are placeholder stubs, prose, or non-YAML. The conclusion does not depend on ambiguous labels. Completeness mismatch

10

TABLE V T HE 86 INTER - MODEL DISAGREEMENTS , GROUPED BY DOMINANT DRIVER , WITH ONE REPRESENTATIVE CASE PER CATEGORY. Category

n (%)

What drives the split

Representative case (service)

Completeness mismatch

51 (59.3)

The reference is a minimal documented fragment; the winner generates exactly that snippet while the losers produce a complete, often valid, workflow (or underproduce a bare fragment against a full reference).

GitHub Actions’s permissions: {}: the winner reproduces the fragment and scores near 1, while the loser wraps it in a complete build workflow, valid YAML, and scores near 0.

Degenerate or invalid

18 (20.9)

A losing model’s output is genuinely broken: a placeholder stub, a prose essay, non-YAML, a hallucinated dump, or the wrong language.

Travis’s git: depth: false: the winner reproduces it, whereas the loser returns Here is a minimal... with no YAML body.

Reference quality

15 (17.4)

The gold reference is not a CI configuration, and matching it measures agreement with the reference rather than the ability to generate valid CI workflows.

CircleCI’s reference is a Helm values.yaml: the model generates a YAML configuration instead.

Valid alternative

1 (1.2)

Best and worst are both valid configurations differing only in surface form or key spelling.

Both outputs are valid variables: blocks that differ only in content.

Spec underdetermined

1 (1.2)

The description admits several correct configurations, so different models pick different valid options.

“Use Ubuntu 22.04” maps to dist: jammy for one model and os: ubuntu-22.04 for another model, and both are correct.

alone accounts for 51 of 86 cases (59%), a majority by itself, and is based on a structural distinction: a short reference, a winner matching that granularity, and losers expanding it into a full pipeline. The only ambiguous cases are weak CircleCI outputs labeled degenerate versus reference-quality, all within CircleCI’s 30 cases. Even if all were counted as genuine failures, the failure rate would rise only to at most 37%, so failures would still be a minority. The examples illustrate why similarity-based ranking can be misleading. On one GitHub Actions task whose reference is the single line permissions: {}, GPT-4o reproduces exactly that block and scores near 1, whereas CodeLlama-7B and CodeLlama-34B wrap the same permission in a full build workflow and score near 0 despite producing valid YAML. On a Travis CI task whose reference is git: depth: false, GPT-4o generates the bare key while other models generate complete language configurations. The “best” model on such fragments is the one whose default output completeness matches the reference. The proprietary models win most often (GPT-4.1 in 22 cases and GPT-4o in 16) because they tend to generate exactly the requested snippet, followed by mid-sized Qwen2.5-Coder-14B (9); the low scorer is typically CodeLlama, whose placeholder lead-ins (Here is a minimal...) make CodeLlama-13B the worst output in 11 of the 18 genuine-failure cases and CodeLlama-7B in 10. The mechanism is also service-dependent. On CircleCI, disagreement splits between reference quality (14 of its 30 sampled cases, where the reference is a CircleCI-server installation manifest such as a Helm values.yaml or Kubernetes ConfigMap rather than a .circleci/config.yml workflow) and genuine degeneration (16 of 30, where a low-scoring model generates a placeholder stub or a prose explanation instead of YAML). In contrast, GitHub Actions, GitLab CI, and Travis CI disagreement is almost entirely completeness mismatch. Because high-dispersion references are often minimal fragments, the metric rewards models that generate the small requested snippet and penalizes models that produce complete (still valid) workflows. The result is a measure of output completeness rather than correctness,

motivating the analysis of RQ6, where fine-tuning on these fragments improves similarity while reducing validity. F. RQ6: To what extent can fine-tuning and schema-guided repair improve generation? 1) Motivation: Previous RQs evaluate models in their default form under zero-shot prompting and show that errors concentrate on misinterpreting or omitting task-specific keywords, with a smaller and largely mechanical invalidity tail rather than deep misunderstanding (RQ4). This suggests that targeted adaptation may reduce the gap without requiring larger models. We therefore evaluate two interventions: finetuning a modest open model on D OC 2CI to learn servicespecific conventions from description-to-configuration pairs, and a training-free schema-guided repair pass to measure how much invalidity can be corrected from validator feedback. Together, these interventions determine whether D OC 2CI serves only as a diagnostic benchmark or also as a resource for improving configuration generation. 2) Methodology: We test two lightweight interventions against the validity and quality gap identified in RQ1. Both operate on the deduplicated D OC 2CI pairs and reuse the exact zero-shot prompt from Listing 1, so that only the intervention, and not the prompt or decoding settings, changes. The first intervention is supervised fine-tuning, evaluated with k-fold cross-validation (k = 5) stratified by service. We take one target configuration per unique description (n = 3,317 unique descriptions; 46 of the 3,363 pairs share a description with a different reference snippet and collapse to a single target), ensuring that a description never appears in both training and test. Because D OC 2CI is deduplicated (Section III-A), no reference leaks across folds. For each fold, we fine-tune on the remaining folds and generate on the held-out fold, then combine the held-out outputs across all folds so that every pair is evaluated outof-fold and the fine-tuned and zero-shot models are compared on identical inputs. We fine-tune the small and large members of the strongest open family from RQ1, Qwen2.5-Coder-7BInstruct and Qwen2.5-Coder-32B-Instruct, allowing the effect

11

of adaptation to be measured at two scales rather than one. We use low-rank adaptation(LoRA) [53] over 4-bit quantized weights [54] (rank 16, three epochs, cosine schedule), which keeps training within an academic GPU budget, and decode greedily to match the zero-shot runs. We refer to the resulting fine-tuned setting as D OC 2CI–FT (at both scales), giving later work building on D OC 2CI a named similarity baseline for comparison, and share the generated outputs upon request. For each model, the baseline is the same base model generating zero-shot outputs for all pairs, regenerated with identical greedy settings to isolate the effect of adaptation. 3) Results: The two interventions shift different metrics in opposite directions, and that contrast is the result: finetuning greatly improves reference alignment but reduces standalone validity, while the repair pass boosts validity without affecting anything else.Fig. 5 places both moves in the similarity-validity plane: from the same zero-shot starting point, D OC 2CI–FT pulls down and to the right, while D OC 2CI–SR pushes straight up. 95

Schema validity (%)

90 85 80

7B

32B

75 70 65

zero-shot

60

fine-tuned (Doc2CI-FT)

55

combined (Doc2CI-FT+SR)

repaired (Doc2CI-SR)

0.60

0.65

0.70

0.75

0.80

Mean cosine similarity Fig. 5. RQ6 interventions in the similarity-validity plane. Finetuning (D OC 2CI–FT) increases similarity while reducing validity, repair (D OC 2CI–SR) increases validity at fixed similarity, and their combination (D OC 2CI–FT+SR) improves over zero-shot on both axes.

Both bases are among the most schema-valid models in RQ1, with zero-shot schema validity already high (about 80%, well above the 71% cross-model average), which makes them a demanding rather than a flattering starting point. On the combined out-of-fold test set, fine-tuning pulls each model’s output strongly toward the documentation’s own level of completeness (Table VI). For Qwen2.5-Coder-7B-Instruct, mean cosine rises from 0.575 to 0.723 and the exact-match rate rises from 0.5% to 10.2%, about a twentyfold increase; for Qwen2.5Coder-32B-Instruct the same shift appears, from 0.627 to 0.779 cosine and from 1.9% to 17.3% exact. Yet schema validity falls, from 80.3% to 54.6% for Qwen2.5-Coder-7BInstruct (a change of −25.7 points, paired bootstrap 95% confidence interval [−27.6, −23.7]) and from 81.0% to 56.7% for Qwen2.5-Coder-32B-Instruct (−24.3, [−26.1, −22.5]), while parse validity edges down from about 96% to about 89%. Both movements are decisive and opposed. The similarity gain is a large paired effect (matched-pairs rank-biserial r = +0.66 for

TABLE VI F INE - TUNING AND THE COMBINED D OC 2CI–FT+SR PASS (D OC 2CI–SR REPAIR OVER D OC 2CI–FT) VERSUS GREEDY ZERO - SHOT ON THE COMBINED OUT- OF - FOLD D OC 2CI TEST SET (n = 3,317). B OLD MARKS THE BEST UNDER EACH MEASURE WITHIN A BASE MODEL . Cosine

Exact%

Parse%

Schema%

Qwen2.5-Coder-7B-Instruct Zero-shot D OC 2CI–FT D OC 2CI–FT+SR

0.575 0.723 0.686

0.5 10.2 4.4

95.9 89.0 97.6

80.3 54.6 84.7

Qwen2.5-Coder-32B-Instruct Zero-shot D OC 2CI–FT D OC 2CI–FT+SR

0.627 0.779 0.727

1.9 17.3 8.1

95.5 89.2 98.6

81.0 56.7 87.9

Qwen2.5-Coder-7B-Instruct and +0.69 for Qwen2.5-Coder32B-Instruct, Wilcoxon p < 10−230 ), far larger than any model-scale or specialization effect in RQ2, whereas the validity loss is equally significant in the other direction: on the paired schema outcomes McNemar’s test [55] counts 1,072 of Qwen2.5-Coder-7B-Instruct’s configurations turning from valid to invalid against only 221 corrected (p < 10−130 ), and 995 versus 189 for Qwen2.5-Coder-32B-Instruct. The drop is not noise but a direct consequence of what the model learns to imitate. As RQ5 showed, most D OC 2CI references are minimal documentation fragments that illustrate a single keyword rather than a runnable file, and fine-tuning on them reproduces the same short, fragmentary output: the output matches the reference more closely (higher cosine and exact-match) but omits the wrapper keys a standalone file needs to satisfy the schema. The effect tracks how fragmentary each service’s references are. It is most severe on GitHub Actions, where schema validity collapses from 70.2% to 22.0% for Qwen2.5-Coder-32B-Instruct because its references so often omit the surrounding on: and jobs: scaffold, and on CircleCI (76.4% to 28.1%), but is mild on GitLab CI and essentially absent on Travis CI (91.4% to 86.8%), whose short references are themselves usually schema-valid. Fine-tuning on D OC 2CI therefore improves stylistic agreement with the documentation but should not be optimized on similarity alone, since on a fragment-heavy reference set that objective can come at the expense of the validity RQ1 found lacking. The training-free repair pass moves the opposite way. Feeding the validator’s own error back for a single correction raises schema validity from 81.0% to 93.7% for Qwen2.5-Coder32B-Instruct, fixing 423 of the 631 invalid configurations (67.0%) in one round, and from 80.3% to 89.7% for Qwen2.5Coder-7B-Instruct (312 of 655, 47.6%); both gains are significant by McNemar’s test (p < 10−90 ). The pass is also monotone by construction: because it only revisits outputs the validator already rejected, it never turns a valid configuration invalid (0 regressions for either model), in sharp contrast to fine-tuning’s wholesale validity loss. The recovered cases are concentrated exactly where fine-tuning did the most damage, namely GitHub Actions (70.2% to 93.9% for Qwen2.5-Coder32B-Instruct) and CircleCI, which confirms that much of the invalidity is mechanical: a missing or misplaced key that the schema message names precisely and the model can reinstate without any training and without disturbing similarity. The

12

larger model repairs more of its own errors than the smaller one (67% versus 48% of invalid outputs), and repair capability, like generation quality, therefore scales with model size. Combining fine-tuning with repair. Because fine-tuning and repair move validity in opposite directions, we also apply them in sequence, running the D OC 2CI–SR repair pass over the combined D OC 2CI–FT outputs, a combined setting we call D OC 2CI–FT+SR (Table VI). This assesses whether one model can retain the fine-tuned similarity gains while recovering the standalone validity that fine-tuning gives up, since repair reinstates exactly the wrapper scaffold (on:, jobs:, version:) that fine-tuning learned to omit. For Qwen2.5-Coder-32B-Instruct, repair raises the fine-tuned model’s schema validity from 56.7% to 87.9% and its parse validity from 89.2% to 98.6%, while cosine settles back from 0.779 to 0.727 and exact-match from 17.3% to 8.1%; Qwen2.5-Coder-7B-Instruct follows the same pattern, with schema validity increasing from 54.6% to 84.7% while cosine decreases from 0.723 to 0.686. The similarity gain largely survives, since D OC 2CI–FT+SR retains 66% of fine-tuning’s cosine improvement over zero-shot at 32B (0.627 to 0.727) and 75% at 7B while ending well above the zero-shot schema rate (87.9% against 81.0%). Consequently, D OC 2CI–FT+SR is the only setting that outperforms the zero-shot baseline on both similarity and validity simultaneously. It does not match the schema validity of repairing the zero-shot output directly (93.7%), because the fragmentary text produced by fine-tuning is harder to complete than the complete zero-shot output. The cosine drop from D OC 2CI–FT to D OC 2CI–FT+SR is a large, highly consistent paired effect, since repair lowers cosine on nearly every configuration it edits (matchedpairs rank-biserial r = −0.70 for Qwen2.5-Coder-32BInstruct and −0.58 for Qwen2.5-Coder-7B-Instruct, Wilcoxon p < 10−80 ), yet its absolute magnitude is modest: a mean cosine loss of 0.052 (paired bootstrap 95% confidence interval [0.047, 0.057]) for Qwen2.5-Coder-32B-Instruct and 0.037 ([0.033, 0.041]) for Qwen2.5-Coder-7B-Instruct. The reduction is not an artifact of the mean, since the median and geometric-mean cosine follow the same trend (median 0.813 to 0.755 at 32B and 0.748 to 0.708 at 7B). On the paired schema outcomes, McNemar counts 1,037 configurations turned valid against 0 turned invalid for Qwen2.5-Coder32B-Instruct (1,000 versus 0 for Qwen2.5-Coder-7B-Instruct, both p < 10−300 ). The per-service pattern reflects the expected trade-off: on Travis CI, whose references are already valid, repair adds validity (86.8% to 95.6%) at almost no similarity cost (0.818 to 0.804 cosine), whereas on the fragment-heavy GitHub Actions case it incurs a larger similarity cost (0.836 to 0.722) to restore schema validity (22.0% to 80.3%). RQ6 Summary. Fine-tuning on D OC 2CI substantially improves reference similarity (cosine 0.627 → 0.779 at 32B) but reduces standalone schema validity (81.0% → 56.7%) because the data contains many minimal fragments. Repair restores validity (93.7% schema validity), and combining the two achieves the best balance, improving both similarity and validity over zero-shot generation.

V. D ISCUSSION A. Implications 1) Similarity is necessary but not sufficient: The gap between similarity and validity is the primary practical implication of our findings. A mean cosine near 0.7 for the best models coexists with an exact-match rate of at most 3.1% and, although parse failures are rare (∼3%), a schema-conformance rate of only 71% across services. The GPT-4o case on GitHub Actions, where it achieves the highest similarity yet is among the least schema-valid because it generates fragments, makes the point vivid. Evaluations that report only similarity, as most prior work does, overstate how close LLMs are to usable CI automation. Validity checks against service schemas are cheap and should be standard in this setting. 2) Output completeness over code specialization: On a lineup of instruction-tuned models, the code-versus-general distinction is weak and non-monotone (RQ2): code specialization helps only at larger Qwen scale and is neutral or harmful elsewhere. What actually separates outputs is their completeness, namely whether a model generates the configuration specified by the task or wraps it in a larger workflow, which is the dominant driver of cross-model disagreement (RQ5). The only residual output-discipline failures are CodeLlama’s occasional prose lead-ins. For CI generation, teams should therefore prioritize scale and output discipline over the codeversus-general label. 3) Documentation is an imperfect oracle: Our references come from official documentation, which is authoritative but was not written to be a generation target. Many references are minimal fragments or non-workflow artifacts, which inflates apparent error under similarity and explains part of the low exact-match rate and the Travis CI divergence between similarity and manual usability. This is a caution for benchmark builders as much as for model developers, and it motivates references that are complete, structurally valid, and executable configurations. 4) Repair, not fine-tuning, is the validity lever: The two adaptations of RQ6 make the similarity-validity split concrete as an engineering choice. Fine-tuning on D OC 2CI, released as D OC 2CI–FT, does exactly what supervised learning should, pulling outputs toward the training targets, which raises cosine and exact-match sharply at both model scales, and D OC 2CI–FT at 32B achieves the highest similarity of any model in the study (0.779 cosine), which makes it the reference point later similarity-based work on this benchmark will compare against. Because those targets are largely minimal documentation fragments, however, the same move lowers standalone schema validity by about 25 points, most of all on GitHub Actions, where references routinely omit the on: and jobs: scaffold. Optimizing a CI generator on reference similarity is therefore self-defeating for validity on a fragment-heavy reference set, a caution that generalizes to any benchmark developed from documentation snippets. The training-free D OC 2CI–SR repair pass moves the other way, raising schema validity to 93.7% and fixing two thirds of invalid configurations in a single round by feeding back the validator’s own message, precisely because much of the

13

invalidity is a mechanical missing or misplaced key rather than a conceptual error (RQ4). For teams, these results suggest prioritizing validation before fine-tuning: apply a validatorin-the-loop repair step before turning to fine-tuning when the goal is runnable output. 5) Practical recommendations: For CI engineers. Treat LLM output as a draft, not a deliverable. With only 41.3% of sampled configurations usable as-is and 27% unusable, a generated file should be parsed and schema-checked before it is committed, and read against the specific keyword the task requires, since omission and misinterpretation of that keyword are the two most common failures. Practical value varies by service: expect the most help on Travis CI (56.0% usable) and GitHub Actions (53.0%) and the least on CircleCI (26.7% usable, and only 56% schema-valid across all models), and prefer a larger model, weighting output discipline over the code-versus-general label. Where resources permit, prefer larger models despite their higher latency and memory requirements, prioritizing scale and output discipline over the code-versus-general label. For CI platforms. Many LLM failures are detectable before execution. Editor-integrated schema validation, already available for human authors, would catch the small fraction of configurations that do not parse and the nearly one-third that violate the schema before they reach a workflow. Returning those validation errors to the model recovers two thirds of the invalid configurations in a single repair round (RQ6). Servicespecific scaffolds could further reduce the cross-service idiom leakage that contributes to CircleCI’s low scores. For LLM developers. On a capable lineup, outright degeneration is rare (2.3%), with the remaining output-discipline failures largely limited to CodeLlama’s occasional prose leadins. Constraining generation to valid YAML and the target service schema, whether through schema-aware decoding or the validator-in-the-loop repair of RQ6, removes most invalid outputs at low cost. Fine-tuning on documentation improves alignment with reference configurations but not standalone validity, making it complementary rather than a replacement. Because misinterpretation and omission are the dominant failure modes, retrieving service documentation during generation [56] to supply the required service-specific keywords is likely the higher-leverage intervention. For researchers. Evaluate across services and report validity alongside similarity. A single-service study would have missed that CircleCI is substantially harder than GitHub Actions, while a similarity-only evaluation would have missed that GPT-4o, the top model on GitHub Actions by similarity, is among the least schema-valid on that service. Documentationderived benchmarks also require reference-quality controls: without them, the 13% of coded issues (and 15% by attribution) caused by fragment, non-workflow, or wrong-artifact references would be misinterpreted as model errors. B. Threats to Validity 1) Construct validity.: We use each service’s JSON schema, the same schemas used by IDE tooling, applied uniformly

across all four services. Two schemas are maintained by the community-run SchemaStore project (GitHub Actions and Travis CI) and two by service-specific editor tooling (the CircleCI YAML language server and GitLab web-editor schema). Because schemas can lag behind newly introduced features, schema conformance approximates acceptance by the live service rather than certifying it; any lag makes our validity estimates conservative. We verified that all schemas resolve and validate known-good references but did not independently assess their feature coverage. Although each service provides a dedicated validator (actionlint, circleci config validate, GitLab CI Lint, and travis-yml), these tools assume complete standalone files and enforce service-specific rules. Applying them to our fragment-heavy benchmark would conflate incompleteness with incorrectness and prevent comparable cross-service measurement. We therefore report uniform schema conformance. Consequently, valid fragments matching fragment references may still be marked schema-invalid when a service schema requires a complete file; this distinction motivates the GPT-4o fragment analysis in RQ4. We treat canonicalized Levenshtein similarity of 1 as an exact match, allowing trivial formatting differences. However, fragment and non-workflow references can penalize correct outputs under similarity, which we quantify through the reference-quality category in RQ4. Finally, we use a single zero-shot prompt (Listing 1) for all 16 models and four services. Although this may not elicit each model’s best output, it avoids permodel prompt engineering and attributes quality differences to models and services rather than prompting. We leave more advanced prompting strategies to future work. 2) Internal validity.: Open coding involves human judgment, making coding reliability the main validity concern. Each discrepancy was attributed to model, description, or acceptable variation. For independent coding, a second researcher re-coded a stratified subsample using the three-point codebook, yielding substantial agreement on both the verdict (κ = 0.78) and the usable-versus-not decision (κ = 0.81). Disagreements were one level apart and resolved by consensus. RQ5 categories were instead assigned collaboratively: the two researchers reviewed each case together and agreed on its dominant driver, so no independent agreement coefficient is reported. This limitation has limited impact because the main failure category, degenerate or invalid, is based on concrete failures (placeholder stubs, prose, non-YAML, or hallucinated outputs). Moreover, the key conclusion that most disagreements are not capability gaps follows from the completenessmismatch majority, independent of the ambiguous CircleCI boundary (Section IV-E). Residual boundary decisions (e.g., accepting a renamed job) could still shift category frequencies. The sample size of 385, though grounded in a 95%/±5% design, is smaller than the full population. The disagreement analysis (RQ5) draws a random 95%/±10% sample from a high-dispersion frame and therefore characterizes sharp-split cases rather than the benchmark as a whole. The fine-tuning of RQ6 uses a single fixed lowrank configuration (rank 16, three epochs) rather than a swept hyperparameter search, and the magnitude of the validity drop could therefore shift with the training budget, though its

14

direction is consistent across both model scales and follows from the fragment nature of the training targets. 3) External validity.: We study four CI services, 16 models, and English documentation. Other services, newer or larger proprietary models, few-shot or agentic prompting, and nonEnglish descriptions may behave differently. The proprietary GPT models were evaluated on all four services alongside the open models. Possible exposure of the documentation during pretraining could inflate their scores, but the low exact-match rate (at most 3.1%) suggests limited leakage. 4) Conclusion validity.: We use non-parametric tests appropriate for skewed similarity distributions [46], report effect sizes and confidence intervals, and apply Holm correction for multiple comparisons. Because 53,808 configurations make even small differences statistically significant, we base substantive claims on effect sizes rather than p-values alone. VI. R ELATED W ORK A. CI configuration and its challenges Hilton et al. [7] surveyed developers and found CI adoption limited by inexperience, while Widder et al. [8] and Zampetti et al. [9] documented recurring configuration difficulties, and Vassallo et al. [10] detected configuration smells. Poor CI configurations were linked to long builds and failures [4, 5, 57], Ghaleb et al. [58] characterized CI/CD configuration practices in open-source apps, and Rostami Mazrae et al. [59] analyzed the usage and migration of CI/CD tools. This body of work motivates automation but does not generate configurations. B. LLMs for CI and configuration generation Closest to our work, Mastropaolo et al. [11] study the completion of partial GitHub Actions workflows, and Zhang et al. [12] evaluate LLMs on manually crafted GitHub Actions scenarios, emphasizing syntactic validity and execution. Both target a single service and a small model set. Broader configuration-generation work translates descriptions into infrastructure or deployment files [16, 17]. A prior single-service evaluation [13] examined six models on GitHub Actions only. Our study differs by spanning four CI services rather than one, covering 16 models across scales and specializations rather than a handful, and combining similarity, validity, and a grounded failure taxonomy rather than reporting a single quality score. It further uses open coding to characterize failure modes and sources of disagreement across models. VII. C ONCLUSION We presented a large empirical study on generating CI configurations from natural language, across four CI services and 16 LLMs, on a new benchmark called D OC 2CI containing 3,363 description-to-YAML pairs. LLMs remain far from reliable: exact reproduction rarely exceeds 3%, and although nearly all configurations parse, only 71% conform to the official service schema. Larger models are more structurally faithful, though the gain in embedding similarity is not significant within our mid-to-large lineup, and code specialization confers no aggregate advantage, helping only at larger

scale within a model family. The target service matters, with CircleCI hardest, and our failure taxonomy is led by misinterpretation and omission rather than invalidity, while crossmodel disagreement reflects output completeness (the minimal documented fragment versus a complete workflow) rather than divergent understanding. These results argue for CI-aware, instruction-tuned models, for validity-based evaluation, and for editor-integrated schema support. Future work. Our results suggest training CI-native models on large datasets of executable workflows and their run outcomes, allowing them to learn structural validity and service semantics directly rather than adapting from general code or text. Another direction is execution-grounded training that rewards configurations which parse, satisfy schema constraints, and execute successfully, making validation part of the optimization objective instead of a post-processing step. We also plan to explore few-shot, tool-augmented prompting, and agentic solutions in this context. Moreover, we aim to extend D OC 2CI with executable references, more CI services, and newer proprietary models to enable consistent benchmarking. ACKNOWLEDGMENT This work is supported by the Natural Sciences and Engineering Research Council of Canada (NSERC): RGPIN-202505897. The study was enabled in part by the Digital Research Alliance of Canada. R EFERENCES [1] M. Fowler, “Continuous Integration,” https://martinfowler.com/articles/ originalContinuousIntegration.html, accessed May 2025. [2] S. G. Saroar and M. Nayebi, “Developers’ perception of GitHub Actions: A survey analysis,” in Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering. ACM, 2023, pp. 121–130. [3] “Workflow syntax for GitHub Actions,” https://docs.github.com/en/ actions/writing-workflows, Accessed May 2025. [4] T. A. Ghaleb, D. A. Da Costa, and Y. Zou, “An empirical study of the long duration of continuous integration builds,” Empirical Software Engineering, vol. 24, no. 4, pp. 2102–2139, 2019. [5] T. A. Ghaleb, S. Hassan, and Y. Zou, “Studying the interplay between the durations and breakages of continuous integration builds,” IEEE Transactions on Software Engineering, vol. 49, no. 4, pp. 2476–2497, 2022. [6] S. Pujar, L. Buratti, X. Guo, N. Dupuis, B. Lewis, S. Suneja, A. Sood, G. Nalawade, M. Jones, A. Morari et al., “Automated code generation for information technology tasks in YAML through large language models,” in Proceedings of the 60th ACM/IEEE Design Automation Conference. IEEE, 2023, pp. 1–4. [7] M. Hilton, T. Tunnell, K. Huang, D. Marinov, and D. Dig, “Usage, costs, and benefits of continuous integration in open-source projects,” in Proceedings of the 31st IEEE/ACM international conference on automated software engineering, 2016, pp. 426–437. [8] D. G. Widder, M. Hilton, C. Kästner, and B. Vasilescu, “A conceptual replication of continuous integration pain points in the context of Travis CI,” in Proceedings of the 2019 27th acm joint meeting on european software engineering conference and symposium on the foundations of software engineering, 2019, pp. 647–658. [9] F. Zampetti, C. Vassallo, S. Panichella, G. Canfora, H. Gall, and M. Di Penta, “An empirical characterization of bad practices in continuous integration,” Empirical Software Engineering, vol. 25, no. 2, pp. 1095–1135, 2020. [10] C. Vassallo, S. Proksch, A. Jancso, H. C. Gall, and M. Di Penta, “Configuration smells in continuous delivery pipelines: a linter and a sixmonth study on GitLab,” in Proceedings of the 28th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering, 2020, pp. 327–337.

15

[11] A. Mastropaolo, F. Zampetti, G. Bavota, and M. Di Penta, “Toward automatically completing GitHub workflows,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, 2024, pp. 1–12. [12] X. Zhang, S. Muralee, S. Cherupattamoolayil, and A. Machiry, “On the effectiveness of large language models for GitHub workflows,” in Proceedings of the 19th International Conference on Availability, Reliability and Security, 2024, pp. 1–14. [13] T. A. Ghaleb and D. Rathnayake, “Can LLMs write CI? a study on automatic generation of GitHub Actions configurations,” in Proceedings of the IEEE International Conference on Software Maintenance and Evolution, 2025. [14] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, “Large language models are zero-shot reasoners,” Advances in neural information processing systems, vol. 35, pp. 22 199–22 213, 2022. [15] A. Strauss and J. Corbin, Basics of Qualitative Research: Grounded Theory Procedures and Techniques. Sage Publications, 1990. [16] G. Rosa, A. Mastropaolo, S. Scalabrino, G. Bavota, and R. Oliveto, “Automatically generating dockerfiles via deep learning: Challenges and promises,” in Proceedings of the IEEE/ACM International Conference on Software and System Processes. IEEE, 2023, pp. 1–12. [17] D. Mehta, K. Rawool, S. Gujar, and B. Xu, “Automated DevOps pipeline generation for code repositories using large language models,” arXiv preprint arXiv:2312.13225, 2023. [18] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “BLEU: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002, pp. 311–318. [19] A. Eghbali and M. Pradel, “CrystalBLEU: precisely and efficiently measuring the similarity of code,” in Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, 2022, pp. 1–12. [20] C.-Y. Lin, “ROUGE: A package for automatic evaluation of summaries,” in Text summarization branches out, 2004, pp. 74–81. [21] M. Popović, “chrF: character n-gram F-score for automatic MT evaluation,” in Proceedings of the tenth workshop on statistical machine translation, 2015, pp. 392–395. [22] N. Reimers and I. Gurevych, “Sentence-BERT: Sentence embeddings using siamese bert-networks,” in Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing, 2019, pp. 3982–3992. [23] K. Zhang and D. Shasha, “Simple fast algorithms for the editing distance between trees and related problems,” SIAM Journal on Computing, vol. 18, no. 6, pp. 1245–1262, 1989. [24] T. A. Ghaleb, “Doc2CI: A multi-service study of CI configuration generation using large language models (replication package),” https: //figshare.com/s/c52fa9b2e6471e61eb40, 2026. [25] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., “GPT-4 technical report,” arXiv preprint arXiv:2303.08774, 2023. [26] OpenAI, “Introducing GPT-4.1 in the API,” https://openai.com/index/gpt-4-1, 2025, accessed June 2025. [27] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “LLaMA: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023. [28] B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, R. Sauvestre, T. Remez et al., “Code llama: Open foundation models for code,” arXiv preprint arXiv:2308.12950, 2023. [29] G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ramé, M. Rivière et al., “Gemma 3 technical report,” arXiv preprint arXiv:2503.19786, 2025. [30] H. Zhao, J. Hui, J. Howland, N. Nguyen, S. Zuo, A. Hu, C. A. Choquette-Choo, J. Shen, J. Kelley, K. Bansal et al., “Codegemma: Open code models based on gemma,” arXiv preprint arXiv:2406.11409, 2024. [31] N. Muennighoff, N. Tazi, L. Magne, and N. Reimers, “MTEB: Massive text embedding benchmark,” in Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, 2023, pp. 2014–2037. [32] Y. Zhang, W. Dong, J. Liu, S. Wang, D. Wang, T. Ma, Y. Li, and K. Yang, “A little help goes a long way: Tutoring LLMs in solving competitive programming through hints,” IEEE Transactions on Software Engineering, 2025.

[33] J. Cordeiro, S. Noei, and Y. Zou, “An empirical study on the code refactoring capability of large language models,” ACM Transactions on Software Engineering and Methodology, 2024. [34] SchemaStore, “GitHub Actions JSON Schema,” https://json. schemastore.org/github-workflow.json, accessed June 2026. [35] ——, “Travis CI JSON Schema,” https://json.schemastore.org/travis. json, accessed June 2026. [36] “circleci-yaml-language-server: JSON schema for CircleCI configuration,” https://github.com/CircleCI-Public/circleci-yaml-language-server, accessed June 2026. [37] “GitLab CI/CD configuration JSON schema,” https://gitlab.com/ gitlab-org/gitlab/-/blob/master/app/assets/javascripts/editor/schema/ci. json, accessed June 2026. [38] “actionlint: Static checker for GitHub Actions workflow files,” https: //github.com/rhysd/actionlint, 2024. [39] Fernando, “How to validate your CircleCI configuration,” https://support.circleci.com/hc/en-us/articles/ 360006735753-How-to-validate-your-CircleCI-configuration, accessed June 2026. [40] “CI lint: Validate GitLab CI/CD configuration,” https://docs.gitlab.com/ ci/yaml/lint, accessed June 2026. [41] “travis-yml: Travis CI configuration processing and validation,” https: //github.com/travis-ci/travis-yml, accessed June 2026. [42] J. Cohen, “A coefficient of agreement for nominal scales,” Educational and psychological measurement, vol. 20, no. 1, pp. 37–46, 1960. [43] J. R. Landis and G. G. Koch, “The measurement of observer agreement for categorical data,” Biometrics, pp. 159–174, 1977. [44] M. Friedman, “The use of ranks to avoid the assumption of normality implicit in the analysis of variance,” Journal of the american statistical association, vol. 32, no. 200, pp. 675–701, 1937. [45] J. Demšar, “Statistical comparisons of classifiers over multiple data sets,” Journal of Machine Learning Research, vol. 7, no. Jan, pp. 1–30, 2006. [46] A. Arcuri and L. Briand, “A practical guide for using statistical tests to assess randomized algorithms in software engineering,” in Proceedings of the 33rd international conference on software engineering, 2011, pp. 1–10. [47] F. Wilcoxon, “Individual comparisons by ranking methods,” Biometrics bulletin, vol. 1, no. 6, pp. 80–83, 1945. [48] H. B. Mann and D. R. Whitney, “On a test of whether one of two random variables is stochastically larger than the other,” The Annals of Mathematical Statistics, pp. 50–60, 1947. [49] S. Holm, “A simple sequentially rejective multiple test procedure,” Scandinavian journal of statistics, pp. 65–70, 1979. [50] C. Spearman, “The proof and measurement of association between two things,” The American journal of psychology, vol. 100, no. 3/4, pp. 441– 471, 1987. [51] N. Cliff, “Dominance statistics: Ordinal analyses to answer ordinal questions.” Psychological bulletin, vol. 114, no. 3, p. 494, 1993. [52] J. Romano, J. D. Kromrey, J. Coraggio, and J. Skowronek, “Appropriate statistics for ordinal level data: Should we really be using t-test and Cohen’s d for evaluating group differences on the nsse and other surveys,” in Annual meeting of the Florida Association of Institutional Research, vol. 177, no. 34, 2006. [53] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen et al., “LoRA: Low-rank adaptation of large language models,” in Proceedings of the Tenth International Conference on Learning Representations, 2022. [54] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “QLoRA: Efficient finetuning of quantized LLMs,” Advances in neural information processing systems, vol. 36, pp. 10 088–10 115, 2023. [55] Q. McNemar, “Note on the sampling error of the difference between correlated proportions or percentages,” Psychometrika, vol. 12, no. 2, pp. 153–157, 1947. [56] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel et al., “Retrievalaugmented generation for knowledge-intensive NLP tasks,” Advances in neural information processing systems, vol. 33, pp. 9459–9474, 2020. [57] T. A. Ghaleb, D. A. Da Costa, Y. Zou, and A. E. Hassan, “Studying the impact of noises in build breakage data,” IEEE Transactions on Software Engineering, vol. 47, no. 9, pp. 1998–2011, 2019. [58] T. Ghaleb, O. Abduljalil, and S. Hassan, “CI/CD configuration practices in open-source Android apps: An empirical study,” ACM Transactions on Software Engineering and Methodology, 2025. [59] P. Rostami Mazrae, T. Mens, M. Golzadeh, and A. Decan, “On the usage, co-usage and migration of CI/CD tools: A qualitative analysis,” Empirical Software Engineering, vol. 28, no. 2, p. 52, 2023.

Related documents

• CivBench: A Long-Horizon Benchmark for Tool-Mediated Agents in Civilization VI#924341
• CivBench: A Long-Horizon Benchmark for Tool-Mediated Agents in Civilization VI#920110
• PDEU-Bench: Benchmarking the Personalized Planning Lifecycle of Tool-Calling LLM Agents#1129540
• MTAC-IFBench: Benchmarking Instruction-Following in Multi-Turn Agentic Coding#1084892
• Beyond Problem Solving: UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive Programming#271916