arXiv:2605.02091v1 [cs.SE] 3 May 2026
How Compliant Are GitHub Actions Workflows? A Checklist-Based Study with LLM-Assisted Auditing Edward Abrokwah
Taher A. Ghaleb
Department of Computer Science Trent University Peterborough, Ontario, Canada [email protected]
Department of Computer Science Trent University Peterborough, Ontario, Canada [email protected]
Abstract
1
GitHub Actions (GHA) CI workflows are critical infrastructure, but current tooling offers only syntactic or heuristic checks and does not enforce documented best practices for security, maintainability, or performance. Consequently, issues like over-privileged permissions, weak secrets management, and missing failure notifications remain undetected in real-world pipelines. This paper proposes a novel, documentation-grounded GHA compliance checklist with 30 criteria spanning four workflow sections and eight themes, and assesses Large Language Models (LLMs) for scalable compliance auditing. On 95 real-world Java workflows (2,850 assessments) using four open-weight LLMs, we find only fair agreement (Fleiss’ 𝜅 = 0.28), with systematic disagreement on structural reasoning and security-sensitive judgments. To address this, we introduce a multi-tier adjudication framework in which GPT 5 resolves model conflicts before targeted manual review, reducing verification effort by 81% while retaining 87% agreement with expert judgment. At scale, it reveals major compliance gaps: overall compliance is 28%, dropping to 4% for permission controls; Security (26%) lags far behind Clarity (68%). Our results show that LLMs enable scalable compliance measurement but cannot replace experts, highlighting the need for hybrid human–AI auditing and providing empirical benchmarks and guidance for defensible GHA workflow audits.
Continuous Integration (CI) pipelines depend on correct, compliant workflow configurations to ensure reliable, secure, and reproducible software development. GitHub Actions (GHA), the dominant CI service on GitHub, offers powerful YAML-based automation, but this flexibility increases the risk of misconfiguration, compounded by evolving documentation and widespread third-party actions [17, 21]. Recent supply-chain attacks, such as the 2021 Codecov breach [8], exposed secrets from thousands of build servers, and studies show many GHA workflows are vulnerable to privilege escalation, code injection, and unauthorized token access [9, 25]. Existing validation tools check only syntactic correctness and structural well-formedness, missing workflow compliance with documented best practices [23]. Linters like actionlint and yamllint detect malformed YAML or invalid action references, but cannot assess adherence to GitHub’s security, maintainability, or performance recommendations [30]. Prior work on workflow smells and configuration issues [24] focuses on heuristic indicators rather than normative compliance. For instance, permissions: write-all is syntactically valid and may not trigger smell detectors, yet it violates least privilege, leaving issues such as improper permission scoping, weak secrets management, missing failure notifications, and absent caching strategies undetected in production pipelines. Despite GHA’s widespread adoption, non-compliance in real workflows is common, violation patterns remain unclear, and the ability of modern language models in detecting workflow compliance failures at scale is largely unknown. Addressing these gaps requires an empirical study of GHA compliance quality. To address this gap, this paper proposes a novel, documentationgrounded checklist comprising 30 criteria across four workflow sections and eight themes, enabling automated evaluation of workflow compliance. We empirically evaluate four open-weight Large Language Models (LLMs) on 95 real-world workflows (2,850 assessments), revealing only fair inter-model agreement and systematic weaknesses in structural reasoning, security judgments, and context-dependent interpretation. Finally, we introduce a multi-tier adjudication framework combining GPT 5 dispute resolution with targeted manual review, reducing verification effort by 81% while maintaining 87% agreement with expert judgment. Together, these contributions provide a practical, scalable approach to measuring and mitigating compliance violations in GHA workflows. Contributions. This paper makes the following contributions: (1) A novel, documentation-grounded checklist for GHA compliance across four workflow sections and eight themes.
CCS Concepts • Software and its engineering → Software development process management; Software configuration management and version control systems; Software maintenance tools.
Keywords GitHub Actions (GHA), Compliance, Large Language Models (LLMs), CI workflows, GHA documentation, Auditing, Checklist ACM Reference Format: Edward Abrokwah and Taher A. Ghaleb. 2026. How Compliant Are GitHub Actions Workflows? A Checklist-Based Study with LLM-Assisted Auditing. In The 30th International Conference on Evaluation and Assessment in Software Engineering (EASE ’26), 9–12 June, 2026, Glasgow, Scotland, United Kingdom. ACM, New York, NY, USA, 11 pages. https://doi.org/To-Be-Inserted-ByACM
This work is licensed under a Creative Commons Attribution 4.0 International License. EASE 2026, Glasgow, Scotland, United Kingdom © 2026 Copyright held by the owner/author(s). ACM ISBN To-Be-Inserted-By-ACM https://doi.org/To-Be-Inserted-By-ACM
Introduction
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
(2) Empirical evaluation of four LLMs and a hybrid GPT 5+human adjudication framework, revealing fair agreement, systematic weaknesses, and insights into disagreement patterns. (3) Analysis of real-world open-source workflows, reporting compliance rates and common patterns of violation. Paper Organization. The rest of this paper is organized as follows. Section 2 presents background and related work. Section 3 presents our empirical study across four research questions (RQs). Section 4 discusses the implications of our findings. Section 5 details threats to validity. Section 6 concludes and suggests future work.
Edward Abrokwah and Taher A. Ghaleb
Positioning. Despite these advances, prior work has not studied CI workflow compliance auditing using structured, documentationgrounded checklists, nor analyzed inter-model agreement on compliance judgments. In the context of CI workflow compliance checking, the use of a strong adjudicating LLM to resolve model disagreements, together with targeted manual verification to establish reliable ground truth, also remains unexplored. These gaps motivate our investigation into LLM-based compliance detection and the factors driving inter-model disagreement.
3 2
Background and Related Work
Continuous Integration (CI). CI automates building, testing, and deployment, enabling early detection of integration issues [10, 40]. Workflows are defined via YAML configuration files specifying jobs and steps, but ensuring correctness and compliance with platform best practices remains challenging. GitHub Actions (GHA). Since its 2019 release, GHA has seen wide adoption for its GitHub integration and extensive documentation [18] on how to set up workflow jobs with each with a clear, unique name, suitable runners, and well-defined steps using pinned third-party actions. Shell scripts (run) must follow safe practices; environment variables (env) should not hardcode secrets; inputs (with) should be validated; and GITHUB_TOKEN permissions should be minimal. Properly structuring jobs, steps, dependencies, inputs, and permissions is essential for secure CI pipelines. Large Language Models (LLMs). LLMs are increasingly applied to automate compliance evaluation in GHA workflows. While they scale better than manual audits, LLMs exhibit biases, anchoring effects, and inconsistencies across samples [26, 36, 43, 47], potentially overlooking novel workflow patterns. Mitigation strategies improve reliability and consistency, but understanding these limitations is crucial for safe application in CI compliance auditing [26, 36]. Related Studies on CI (Mis)Configurations. CI configurations can directly impact build time and status [14–16] and are linked to mobile app success [48], underscoring the need to study how well CI workflows follow guidelines and best practices. Prior work has examined CI configuration complexity [13], misconfigurations [11], and smells [24, 45], as well as automated techniques to detect and repair these issues [41, 46]. Such approaches, however, largelybut these typically rely on heuristics or narrow issue taxonomies rather than systematic, documentation-grounded compliance reasoning. Recent LLM-based methods support software quality tasks such as test smell detection [34], code smell assessment [37], and repositorylevel audits [19], yet their suitability for structured compliance reasoning over CI configuration guidelines remains unexplored. Related Studies on LLMs for CI. Recent work explores using LLMs for CI configuration, automation, and analysis. Studies show LLMs can generate GitHub Actions workflows from natural language [17], automate DevOps pipeline creation [28], migrate CI across services [22], analyze CI practices in open-source projects [6], and assist in failure diagnosis and remediation [44]. While promising, these approaches face challenges in correctness, semantic alignment, and structural reasoning, motivating our focus on normative, documentation-grounded compliance evaluation.
Empirical Analysis and Results
This section describes our study setup, including the construction, curation, and refinement of the proposed compliance checklist, as well as the collection and preparation of the GHA workflow dataset. We also outline the LLM-based auditing pipeline used in the study (Figure 1), which defines how multiple models independently evaluated workflows, how disagreements were resolved, and how final compliance labels were obtained. The following subsections detail the checklist’s structure, its grounding in official GHA documentation, and the data sources, selection criteria, and preprocessing steps used to ensure consistent, reproducible, and fair evaluation across all Large Language Models.
3.1
RQ1. What are the core compliance criteria for GHA workflow jobs?
3.1.1 Motivation. Workflow jobs are central to GHA CI pipelines, coordinating build, test, and deployment. Yet no prior work systematically defines actionable, documentation-grounded compliance criteria for jobs and steps. Existing studies highlight CI antipatterns but lack a structured checklist for reliable human or automated audits. Misconfigurations therefore persist, causing security, performance, and reliability issues, motivating our systematic approach to defining and structuring workflow job compliance criteria. 3.1.2 Approach. We derived a structured compliance checklist using a three-phase methodology: documentation review,iterative refinement and structural organization. • Documentation Review. We systematically analyzed GitHub Actions’ official documentation, including workflow syntax references, guidelines, and specifications [18]. From these, we extracted required configuration elements (e.g., runner selection, permission scopes) and recommended practices (e.g., action pinning, caching strategies) for workflow jobs. Each candidate criterion was mapped to authoritative documentation to ensure platform-defined grounding. Criteria were derived only from statements with directive terms (e.g., “must”, “should”, “avoid”) to capture enforceable or recommended practices. Each criterion was recorded with its source and labeled as required or recommended to support traceability and consistency. • Iterative Refinement. Two co-authors with CI expertise collaboratively refined the checklist over three iterations. In each round, we applied it to randomly sampled pilot workflows spanning single- and multi-job pipelines and varied runner configurations. Pilot workflows were selected to maximize diversity, helping uncover edge cases and ambiguities in the criteria. Items were assessed for interpretability, measurability, and
How Compliant Are GitHub Actions Workflows? A Checklist-Based Study with LLM-Assisted Auditing
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
Checklist Derivation Compliance Auditing Pipeline GHA Documentation
Expert Review (2 Co-Authors)
Yes
30-item Checklist Independent Model Analysis Input YAML Workflows
8,924 Java Projects
GHA Filter 11,031 workflows 22,990 YAML files
Quality Filter ≥10 stars, ≥50 runs 1,576 projects 5,749 YAMLs
Stratified Sample 2,850 checklist items 95% CI, 10% MoE
Phi4 14B Gemma-3 12B mistral-0.3 7B LLaMA-3.1 8B
No
4/4 or 3/4?
GPT 5 Tie-Breaker
Resolved
Final Compliance Labels
Unresolved
Human Review
Dataset Construction
Figure 1: Full study pipeline. Left: checklist derivation from GHA documentation (top) and dataset construction from 8,924 Java projects, filtered and sampled to 2,850 checklist evaluations (bottom). Right: LLM-based compliance auditing (unanimous or near-unanimous model verdicts are accepted; disagreements escalate to GPT 5 and, if still unresolved, to manual review). cross-project applicability. Problematic items were revised or removed if they could not be clearly measured or observed, or if they duplicated higher-level criteria without adding meaningful value. Disagreements were resolved through discussion, using GitHub documentation as the final reference. • Structural Organization. The final checklist was organized along two dimensions. Each criterion was mapped to its corresponding workflow Section (workflow, jobs, permissions, or steps) following the GHA schema and assigned a primary compliance Theme (e.g., Security, Error/Failure Handling, Modularity, etc.). This dual structure enables both targeted section-level auditing and thematic compliance analysis. Checklist derivation followed established guidelines for operationalizing qualitative concepts in empirical software engineering [35]. One co-author extracted candidate criteria from the documentation, which were then reviewed and verified with the second co-author through collaborative discussion, resolving any disagreements by consulting official GHA documentation. Each criterion was formalized as explicit binary decision rules to enable consistent yes/no assessment. For example, “proper secrets management” was formalized as requiring secrets to be referenced via ${{ secrets.NAME }}, never hardcoded, and scoped to individual steps rather than job-level environments. Similarly, “failure handling” was formalized as requiring explicit failure conditions (e.g., if: failure()) or notification steps triggered on job failure. 3.1.3 Findings. Table 1 presents the proposed compliance checklist, consisting of 30 criteria organized by workflow sections and thematic concerns. The checklist was derived from official GitHub Actions documentation and defines the compliance dimensions evaluated in this study. Step-level configurations drive most compliance violations in GHA workflows. Our analysis produced a 30-item compliance checklist across four workflow sections and eight themes. Step-level configurations account for half of the checklist (15 items), capturing fine-grained execution logic where security vulnerabilities and operational failures commonly occur. Job-level criteria (11 items) focus on build coordination, environment configuration, and performance considerations, while workflow-level (3 items) and permissions (1 item) criteria address global properties affecting the entire pipeline. This distribution aligns with prior empirical evidence showing that CI violations disproportionately stem from low-level configuration
decisions [11, 45]. Notably, step-level criteria concentrate most Input Validation, Maintainability, and Security checks, highlighting that compliance failures often stem from fine-grained execution logic rather than high-level workflow structure. Security dominates compliance concerns, while performance, maintainability, and modularity are equally important. From a thematic perspective, Security dominates the checklist with six criteria spanning all workflow sections, covering secrets management, action pinning, runner authorization, and vulnerability exposure. Several themes are represented by four criteria each, including Performance, Error/Failure Handling, Input Validation, Maintainability, and Modularity, highlighting that robust CI pipelines require balanced attention across multiple quality dimensions rather than optimization along a single axis. Environment and Clarity form foundational themes with two criteria each, capturing runner selection and configuration decisions that affect workflow portability and correctness. Unlike prior work that infers workflow smells from historical change patterns [24], our checklist is explicitly grounded in GitHub documentation, supporting normative compliance auditing. Several compliance themes span multiple workflow sections, necessitating cross-layer analysis. The checklist reveals crosscutting patterns: security criteria appear at workflow, job, and step levels, while Modularity and Environment span multiple configuration layers. This suggests that effective compliance tooling must pair section-specific checks with cross-section reasoning. Practically, the checklist helps DevOps teams prioritize CI improvements: security criteria form an automatable baseline, while performance and maintainability practices can be added incrementally. Over half of the criteria target step-level logic, and input validation remains a frequent, poorly automated source of errors. RQ1 Summary: A documentation-based checklist of 30 GHA compliance criteria reveals that step-level configurations are the primary source of violations, while security matters span all workflow sections. This framework supports targeted auditing and underscores the need to balance workflow complexity, maintainability, and compliance.
3.2
RQ2. To what extent can LLMs detect GHA compliance in open source projects?
3.2.1 Motivation. Prior research investigated the use of LLMs for CI-related tasks [17], yet their ability to assess GHA workflow
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
Edward Abrokwah and Taher A. Ghaleb
Table 1: Compliance checklist for GHA workflows: 30 criteria organized by compliance sections and themes Section
Theme
ID
Criterion
Workflow (W)
Error/Failure Handling Environment Security
W1 W2 W3
Workflow should handle failures properly and provide notifications. Workflow should use documented and supported runner environments. Workflow should follow security and maintainability best practices.
Clarity Clarity Error/Failure Handling Environment Modularity Modularity Performance Performance Performance Performance Security
J1 J2 J3 J4 J5 J6 J7 J8 J9 J10 J11
Job names should be clear and unique. All jobs must be defined properly in the main jobs block. Jobs should enable runner debug logging to allow better diagnosis of job execution. Runners must be appropriate for each job. Jobs should remain modular and separated (setup, test, deploy). Jobs should be isolated to avoid unintended side effects. Dependencies and tools should be cached effectively across jobs. Caching strategy should be portable across environments. Parallelism settings should be optimized and validated for better resource usage. Caching must be used to reduce build time. Unauthorized runners must not be used.
Modularity Modularity Input Validation Input Validation Input Validation Input Validation Error/Failure Handling Error/Failure Handling Maintainability Maintainability Maintainability Maintainability Security Security Security
S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 S14 S15
Any complex run commands should be split into smaller steps for clarity. Build/deploy commands should be split into steps with error handling and caching. Inputs should be validated or sanitized to prevent unexpected behavior. User inputs for platform parameters should be validated. Boot JDK platform inputs should be validated. Validation checks must not be disabled without justification. Steps should enable debug logging to make errors clear and easily traced. Command-line tools should detect and report failures properly. Repository-specific conditions should be avoided or made configurable. Weak file-change detection (e.g., git status) should be avoided. Conditional jobs should be done using native GitHub strategies like matrix filters. Conditional expressions should be documented and maintainable. Third-party actions must be pinned to specific commits SHA. Reusable or third-party actions should be kept up to date with stable versions. Steps should include dedicated static/dynamic security analysis.
Security
P1
Secrets must be stored securely (no hardcoding).
Jobs (J)
Steps (S)
Permissions (P)
compliance remains unexplored. Building on the structured checklist from RQ1, we investigate whether LLMs can reliably evaluate real-world workflows against these criteria, highlighting both their potential and limitations for automated compliance auditing. 3.2.2 Approach. Our study evaluates the ability of LLMs to detect GHA workflow compliance in open-source projects using the checklist developed in RQ1. We follow the following process. Dataset Preparation. We used a dataset of 8,924 Java projects spanning multiple CI services from a recent study [7]. Projects were filtered to include only those using GHA, yielding 11,031 workflows across 22,990 YAML files. Following prior established criteria [4], we retained projects with at least 10 stars and 50 workflow runs to focus on non-trivial, actively used repositories. This resulted in 1,576 projects with 5,749 distinct YAML workflows. Sampling Workflows. To balance coverage and manual feasibility, we randomly sampled 95 workflows (95% confidence level, ±10% error margin) from the filtered dataset. This sample represents 95 × 30 = 2, 850 possible checklist checks per LLM. LLM Selection and Configuration. We evaluated four openweight LLMs representing diverse architectures and reasoning capabilities: LLaMA-3.1 8B [39], Gemma-3 12B [38], mistral-0.3 7B [29], and Phi-4 14B [1]. GPT 5 (a proprietary reasoning model) [31] was additionally used as an adjudicator for disagreements. Openweight models were run via Ollama [27] with temperature = 0 to
ensure deterministic and reproducible outputs and reduce sampling variance, aligning with best practices for stable LLM benchmarking [5]. In contrast, GPT 5 operates at a fixed temperature = 1 (not configurable via the API), which supports controlled sampling for more robust reasoning and helps avoid degenerate outputs [32, 33]. Table 2 gives more details about the LLMs used in our study. Checklist Question Formulation and Prompting. Each checklist item was converted into a structured question answerable with “YES,” “NO,” or “NOT APPLICABLE”. We then designed a zero-shot prompt with two roles, to ensure deterministic, section-aligned responses suitable for automated analysis (Listing 1), as follows: • System role: Instructed the LLM to act as a senior DevOps expert auditing GHA workflows against the checklist. • User role: Provided the workflow YAML and requested structured JSON output aligned with checklist sections. LLM Evaluation and Agreement Analysis. Each sampled workflow was evaluated by all four LLMs, producing 11,400 total outputs. To understand reliability, outputs were categorized into three agreement bands (agreement distribution is summarized in Table 3): • Unanimous (4/4): All models agree. • Near-unanimous (3/4): Single model disagrees; treated as strong agreement. • Split (2/2 or 2/1/1): Significant disagreement, indicating ambiguous or context-dependent criteria.
How Compliant Are GitHub Actions Workflows? A Checklist-Based Study with LLM-Assisted Auditing
Table 2: LLMs used in our study Model LLaMA-3.1 8B [39]
Gemma-3 12B [38]
mistral-0.3 7B [29]
Phi-4 14B [1]
GPT 5 [31]
Description Decoder-only model from Meta with Grouped Query Attention for long-context reasoning, fine-tuned with supervised and RL methods for helpfulness, coherence, and safety. Multimodal (text+image) decoder-only transformer from Google DeepMind with 400M vision encoder, supporting 140+ languages; optimized for reasoning, summarization, QA, and vision-language tasks. Optimized transformer with grouped query and sliding window attention, efficient for long sequences, strong on reasoning, math, and code tasks. Transformer-based model trained on synthetic data, cleaned web content, academic papers, and QA datasets. OpenAI’s advanced multimodal model with autonomous task execution, routing queries between a fast general engine and deep reasoning core.
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
3.2.3 Findings. Table 4 presents the performance of the evaluated LLMs in terms of overall agreement rates, pairwise Cohen’s 𝜅 values, and McNemar test results, highlighting both the extent of concordance between models and statistically significant differences in their compliance judgments. LLM performance varies widely. Gemma-3 12B achieved the highest agreement rate at 90%, followed by Phi-4 14B (71%),mistral0.3 7B (70%), and LLaMA-3.1 8B (61%) (Table 4). This suggests that parameter count alone does not predict performance; instead, architecture and training data appear to be stronger determinants of compliance detection capability. The 29 percentage point gap between the best and worst model further indicates that results are highly model-dependent, making single-model evaluations unreliable. Table 4: Model Agreement Rates, Pairwise Cohen’s 𝜅, and McNemar Test Results Model
Agreement Rate
Pairwise Model
Cohen’s 𝜅
McNemar p-value
Gemma-3 12B
90%
LLaMA-3.1 8B mistral-0.3 7B Phi-4 14B
0.57 0.57 0.70
0.02 0.02 0.04
LLaMA-3.1 8B
61%
mistral-0.3 7B Phi-4 14B
0.47 0.73
1.00 0.73
mistral-0.3 7B
70%
Phi-4 14B
0.40
0.81
Phi-4 14B
71%
–
–
–
Table 3: Agreement Categories Across Checklist Questions Category
Count
Percentage
Unanimous (4/4) Near-unanimous (3/4) Split (2/2 or 2/1/1)
758 1,104 988
27% 39% 35%
Total
2,850
100%
System : You are a senior DevOps expert specialized in secure , efficient , and standardized CI pipeline practices . Your task is to rigorously audit GHA YAML workflow files for compliance with industry - recognized best practices . You understand both functional correctness and structural quality of CI workflows . Use a sectioned JSON format matching the checklist structure . User : Audit the following GHA YAML workflow using the compliance checklist provided . Return your results as a structured JSON matching the section / question layout from the checklist . Answer each question with " YES ", " NO ", or " NOT APPLICABLE ". Return only valid JSON , with no commentary or markdown . { YAML }
Listing 1: Prompt Template used for LLMs Hybrid Adjudication for Split Cases. When LLMs disagreed on a workflow, we used GPT 5 to parse the raw YAML files and generate a consistent input, enabling systematic resolution of split decisions. Cases that remained unresolved after GPT 5 adjudication were escalated to manual review. From these, a stratified subset of 79 cases was manually adjudicated, ensuring proportional representation across checklist items and achieving a 95% confidence level with a ±10% margin of error. The remaining 345 cases were not considered for manual review and thus excluded from our subsequent analyses. This process yielded 295 individual checklist item evaluations used as ground truth for model performance assessment.
Fleiss’ 𝜅 = 0.28
Consensus across checklist questions is limited. Across the checklist, unanimous agreement occurred on 27% of questions and near-unanimous on 39%, leaving 35% of cases split across models (Table 3). This shows that a significant fraction of checklist items are ambiguous or context-dependent, suggesting that automated auditing cannot treat model outputs as definitive and requires mechanisms to resolve unresolved items. Pairwise agreement reveals systematic divergences. Pairwise 𝜅 and McNemar tests show uneven alignment: the highest agreement is between LLaMA-3.1 8B and Phi-4 14B (73%), and the lowest between mistral-0.3 7B and Phi-4 14B (40%). Gemma-3 12B differs significantly from all models (𝑝 < 0.05), indicating distinct heuristics for interpreting checklist items and motivating multiLLM setups with tie-breaking or human review is necessary to ensure reliable compliance assessments. In contrast, LLaMA-3.1 8B, mistral-0.3 7B, and Phi-4 14B show no statistically significant pairwise differences (p > 0.05), suggesting their disagreements may stem from random variation rather than systematically distinct reasoning strategies. Models performance in the manually validated set. The results in Table 5 are computed over 295 manually validated instances, which form the ground-truth evaluation set. Phi-4 shows the best overall performance, with the highest accuracy (0.49), macro precision (0.44), macro recall (0.45), and macro F1 (0.42), indicating the most balanced performance across classes. GPT5 ranks second with a macro F1 of 0.39, limited by weak performance on the YES class (Precision = 0.17, F1 = 0.20), despite strong performance on
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
Edward Abrokwah and Taher A. Ghaleb
Table 5: Model Performance on Manual Validation (n=295). Bold represents the results of the highest performing model.
Model
Acc
Pr.𝑚𝑎𝑐𝑟𝑜
Re𝑚𝑎𝑐𝑟𝑜
F1𝑚𝑎𝑐𝑟𝑜
Pr.𝑌 𝑒𝑠
Re𝑌 𝑒𝑠
F1𝑌 𝑒𝑠
Pr.𝑁 𝑜
Re𝑁 𝑜
F1𝑁 𝑜
Pr.𝑁 𝐴
Re𝑁 𝐴
F1𝑁 𝐴
Manual
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
GPT5 mistral-0.3 7B Phi-4 14B LLaMA-3.1 8B Gemma-3 12B
0.48 0.36 0.49 0.22 0.25
0.41 0.30 0.44 0.30 0.36
0.38 0.29 0.45 0.31 0.32
0.39 0.28 0.42 0.22 0.27
0.17 0.14 0.24 0.16 0.32
0.26 0.30 0.34 0.45 0.36
0.20 0.19 0.28 0.24 0.34
0.41 0.18 0.27 0.19 0.12
0.30 0.13 0.49 0.38 0.42
0.35 0.15 0.35 0.25 0.19
0.64 0.59 0.81 0.57 0.65
0.58 0.43 0.52 0.11 0.18
0.61 0.50 0.64 0.19 0.28
the N/A class (F1 = 0.61). Across all models, the YES class has the lowest F1 scores, peaking at 0.34 (Gemma-3), with precision consistently low (≤ 0.32). For the NO class, Phi-4 reaches the highest recall (0.49) with an F1 of 0.35, while other models perform worse and less consistently. The N/A class is the most reliably predicted, with all models achieving their highest precision on this class, especially Phi-4 (Precision = 0.81, F1 = 0.64), though recall varies widely (e.g., 0.11 for LLaMA-3.1). Overall, models perform best on N/A and consistently struggle to detect YES instances in the manually annotated evaluation set. Section- and theme-level agreement rates reported below are computed from the same 2,850 assessments and are available in full in the replication package [2]. Section-level reliability is uneven. Agreement is highest for Workflow items (74%), followed by Steps (68%), Jobs (62%), and lowest for Permissions (33%). LLMs handle global workflow rules more consistently than fine-grained logic, highlighting that auditing pipelines should incorporate human or advanced-model review for critical or security-sensitive sections, particularly permissions. Theme-level trends highlight strengths and weaknesses. Models achieved the strongest agreement on Modularity (91%) and Maintainability (73%), moderate agreement on Performance (68%) and Error/Failure Handling (64%), and weakest agreement on Environment (41%) and Security (55%). This pattern suggests that structural, deterministic rules are easier for LLMs to assess, while context-sensitive or configuration-dependent criteria require careful human-in-the-loop verification in practice. Overall, LLMs alone are insufficient. Although some models perform well individually, their collective outputs show only fair agreement (Fleiss’ 𝜅 = 0.28) with many split cases. LLMs can detect straightforward compliance issues but cannot reliably judge ambiguous or context-dependent checklist items. This highlights the need for hybrid pipelines that combine multiple LLMs, tie-breaking via GPT 5, and targeted human review to ensure trustworthy, reproducible compliance auditing in real-world DevOps settings. RQ2 Summary: LLM performance on GHA compliance auditing is inconsistent, with Gemma-3 12B reaching 90% agreement but 35% of criteria still requiring adjudication, showing that human oversight remains essential for reliable auditing.
3.3
RQ3. Why do LLMs struggle to agree in GHA compliance auditing?
3.3.1 Motivation. RQ2 revealed that LLMs evaluate GHA compliance inconsistently, with disagreements that are systematic rather than random—some models over-flag certain rule categories while
under-detecting others. This highlights a key challenge: resolving conflicting LLM judgments in a principled way. Existing AI-assisted auditing approaches rarely address multi-model disagreement, often assuming a single authoritative model or using majority voting, which is inadequate when models differ due to heuristics, reasoning styles, or interpretations of compliance criteria. 3.3.2 Approach. RQ3 builds on the multi-tier adjudication strategy from RQ2 to examine why LLMs struggle with GHA compliance. Our approach combines automated adjudication with stratified manual validation to resolve disagreements among weaker models. Automated adjudication with GPT 5. We employ GPT 5 as a dispute resolution tool for three key reasons: (1) Its larger training corpus and parameter count provide superior reasoning compared to open-weight models [20]. (2) It reduces manual review burden from 424 to 79 items (81% reduction), making large-scale validation feasible. (3) Stratified manual verification of 79 items shows 87% agreement with expert judgment, confirming its effectiveness. Importantly, GPT 5 was not treated as ground truth; instead, it amplified consensus among weaker models, with manual review providing independent validation. This mirrors judicial review, where higher courts resolve lower-court disagreements, subject to final expert adjudication [3]. Stratified manual validation. To assess GPT 5 adjudication reliability, we manually review a statistically representative subset of 79 items (19% of flagged cases), following empirical LLM evaluation guidelines [42]. The sample guarantees coverage of: • All 30 checklist criteria: At least two disagreements per criterion. • All disagreement patterns: Single-model disagreements, systematic multi-model errors, and borderline context-dependent cases. • All compliance categories: Security, Clarity, Performance, Modularity, Environment, Input Validation, Maintainability, and Error Handling, proportionally represented. Each sampled item is manually reviewed by a blinded author using the original checklist rules. Disagreements with GPT 5 are resolved through discussion with a second author and reference to official GHA documentation. This ensures that systematic errors in GPT 5 adjudication would be detectable. Validation outcomes. The 79-item stratified sample confirms that GPT 5 adjudication aligns closely with expert judgment (87% agreement) across all criteria types. The absence of systematic misclassification patterns supports GPT 5’s reliability as a first-tier dispute
How Compliant Are GitHub Actions Workflows? A Checklist-Based Study with LLM-Assisted Auditing
resolver, while final adjudication of any remaining ambiguous cases ensures high-quality, trustworthy compliance judgments. 3.3.3 Findings. Table 6 outlines the key reasons and underlying causes of disagreement among LLMs in GHA compliance auditing, along with corresponding findings from manual validation and the relative frequency of each issue. Automated adjudication resolves most, but not all, model disagreement. Of the 988 checklist items where the four openweight LLMs disagreed, GPT 5 aligned with the majority judgment in 564 cases (57%), resolving them without human intervention. This increased the proportion of items with stable, non-contradictory labels from 65% to 85%. The remaining 424 cases (43%) still required manual verification, indicating that adjudication reduces but does not eliminate ambiguity in LLM-based compliance auditing. Disagreement concentrates on a small number of recurring compliance themes. Manual analysis of a stratified sample of 79 unresolved cases shows that disagreement is not uniformly distributed across the checklist. The most frequent source of conflict concerns Job Structure & Runners (23.1%), where models inconsistently interpreted missing jobs, reusable workflows, and runner specifications. A second major source is Inputs & Conditional Logic (19.2%), where LLMs often assumed validation or conditional checks were required even when workflows exposed no inputs or conditions, leading to systematic over-flagging. Security-related criteria trigger systematic but inconsistent risk judgments. Security practices account for a substantial portion of disagreement, spanning Secrets and Scanning (13.5%) and Action Pinning & Reusable Workflows (7.7%). LLMs frequently overreported violations by assuming that all production workflows must include security scanning or SHA-pinned actions, even when such requirements were context-dependent or inapplicable (e.g., local reusable workflows). These conflicts reflect differences in implicit risk tolerance rather than random error. Operational conventions expose limits of context-free reasoning. Criteria related to Error/Failure Handling (11.5%) and Caching & Performance Optimization (11.5%) reveal a recurring issue: LLMs often expected explicit error-handling logic or caching steps, overlooking implicit guarantees provided by tools like Gradle or existing cache setups. Similarly, Change Detection (5.8%) disagreements occurred when models inferred brittleness without sufficient contextual evidence. These cases require understanding tool semantics and workflow intent, which is hard to infer from YAML alone. Structural clarity rarely causes disagreement. Only a small fraction of conflicts involved Modularity & Maintainability (3.8%), where LLMs occasionally flagged simple or compact step definitions as insufficiently modular. Manual review confirmed that, in most such cases, simplicity did not reduce clarity or maintainability, explaining the low frequency of disagreement for these criteria. Overall, disagreement is widespread but highly structured. Across the checklist, 27 of 30 compliance criteria (90%) had at least one case where none of the five LLMs reached consensus. These conflicts are dominated by structural interpretation, conditional applicability, and security judgment, rather than uniformly affecting all checklist items. This indicates that LLM disagreement in GHA
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
compliance auditing is partially systematic and predictable, highlighting specific areas where automated auditing needs adjudication or human oversight. RQ3 Summary: LLM disagreement in GHA compliance auditing is partially systematic: Gemma-3 12B shows distinct, statistically significant divergence from all other models, while disagreements among the remaining three models appear less consistent, driven by structural reasoning, context dependence, and security judgment.
3.4
RQ4. To what extent do open-source GitHub workflows adhere to GHA compliance checklist?
3.4.1 Motivation. Our checklist captures GitHub’s recommendations for secure, maintainable, and reliable CI workflows, but it does not show whether open-source projects actually follow them. GHA’s flexibility lets maintainers favor convenience, historical configurations, or project-specific needs over formal compliance, leaving real-world adherence uncertain. Measuring actual compliance establishes a baseline for CI quality, distinguishes widely adopted from routinely ignored practices, and reveals systematic gaps that may need better tooling, documentation, or community education. This RQ therefore investigates the extent to which open-source GHA workflows align with our validated checklist. 3.4.2 Approach. To systematically evaluate GHA workflow compliance, we employed a multi-model consensus framework that balances automated assessment with targeted manual validation. For each checklist question derived from RQ2, every model produced one of three judgments: YES, NO, or NOT APPLICABLE (N/A). The interpretation of these judgments depends on the checklist item’s intent: for security-positive checks (e.g., “Is the action pinned to a commit SHA?”), YES indicates compliance; for security-negative checks (e.g., “Are secrets exposed in plaintext?”), NO indicates compliance. Items flagged as N/A were excluded from compliance frequency calculations, as they do not apply to the workflow in question. Overall, 43% of checklist evaluations were marked N/A, with the highest rates on criteria S2, J8, and S9, reflecting workflows where modularity, performance optimisation, and maintainability practices were absent and those criteria could not be evaluated. The high N/A rate indicates that many checklist criteria cover features that are not present in all workflows. This reflects the varying scope of the studied Java projects rather than non-compliance. Given that individual model outputs can contain errors or inconsistencies, we implemented a multi-level agreement framework to enhance reliability. A workflow checklist item is classified as compliant when at least one of the following conditions is met: (1) Three or all four models produce the same judgment. (2) At least two models agree with GPT 5, used as a reference model for its superior reasoning capabilities. (3) Two models agree, and their shared judgment is confirmed by manual expert review. Items failing to meet any of these criteria are classified as noncompliant. This design prioritizes automated consensus while
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
Edward Abrokwah and Taher A. Ghaleb
Table 6: Summary of reasons for LLM disagreement in GHA compliance auditing Theme
Main Issue
Why LLMs Disagreed
Job Structure & Run- Missing jobs, unclear naming, LLMs often misinterpreted reusable ners runner configuration workflows, treated missing runners as implied non-compliance, or inconsistently judged job completeness.
Manual Check Findings
Share of Disagreements (%)
Manual review revealed some workflows lacked full job configuration, others used valid runners (e.g., ubuntu-latest), and a small fraction used unauthorized runners.
23.1
Inputs & Conditional Logic
Input validation and condi- LLMs treated the absence of inputs or Many workflows legitimately defined tional expressions conditions as violations, assuming vali- no inputs or conditions, making validadation logic was required. tion not applicable.
19.2
Security Practices
Secrets handling and security scanning
LLMs over-flagged missing secrets or scanning, assuming all workflows must include security checks regardless of context.
No secrets were hardcoded; however, many workflows omitted security scanning entirely.
13.5
Han- Detecting failures or explicit er- LLMs assumed explicit failure-handling ror handling blocks were required, overlooking tools that fail on non-zero exit codes.
Gradle correctly failed on errors, but several workflows lacked explicit logging or failure notifications.
11.5
Caching & Perfor- Use or absence of caching mech- LLMs disagreed on when caching mance Optimization anisms was required or misidentified existing caching behavior.
Some workflows used effective caching, while others had incomplete or missing caching.
11.5
Action Pinning & Reusable Workflows
Pinning third-party or local ac- LLMs incorrectly required SHA pinning Local reusable workflows cannot be tions for local reusable workflows and evalu- SHA-pinned by design; third-party acated tags and SHAs inconsistently. tions varied in correctness.
7.7
Change Detection
Brittle vs. dynamic file change detection
Manual review confirmed mixed practices, including both robust and brittle approaches.
5.8
Modularity & Main- Step structure and decomposi- LLMs occasionally flagged simple com- Simplicity did not reduce maintainabiltainability tion mands as insufficiently modular. ity in these cases.
3.8
Error/Failure dling
LLMs sometimes flagged detection logic as brittle or assumed alternatives were required.
Table 7: Workflow adherence across validation stages Validation Stage
Compliant
Rate
Initial 3/4 LLM consensus After GPT 5 adjudication After manual review
789 / 2,850 1,062 / 2,850 1,079 / 2,850
28% 37% 38%
retaining human verification for ambiguous or borderline cases, ensuring both accuracy and efficiency. Using the validated compliance outcomes, we then computed compliance rates across checklist sections (Section-wise), themes (Theme-wise), and individual criteria (Criterion-wise). 3.4.3 Findings. Table 7 shows how adherence progressed as judgments were consolidated via multi-model consensus, GPT 5 adjudication, and targeted manual review. Open-source project workflows generally show low adherence to the GHA compliance checklist. Across 95 workflows and 30 checklist items, only 28% of items initially achieved consensus among four LLMs. GPT 5 adjudication raised this to 37%, with manual review contributing a marginal one percentage point (38%). This indicates that most recommended practices are not followed, revealing widespread CI quality gaps in open-source projects. Adherence varies across workflow components. Normalized compliance rates were highest for WORKFLOW and JOBS (40%) and lowest for PERMISSIONS (4%), while STEPS achieved 38%. Although
STEPS contained half of all criteria, its normalized adherence closely matched the overall average, suggesting that no single component consistently outperforms others in practice. Thematic compliance is uneven. Among themes, Clarity had the highest adherence (68%), indicating generally acceptable naming, structure, and readability. Adherence was moderate for Input Validation (47%) and Environment (43%), while Error/Failure Handling (39%), Modularity (38%), and Performance (38%) were inconsistently implemented. Maintainability (29%) and Security (26%) showed the lowest adherence, revealing ongoing weaknesses in robustness, maintainability, and secure configuration. Even widely recommended practices are rarely applied. At the criterion level, the most frequently satisfied items were handling workflow failures with notifications (9%), clear job names (8%), decomposing complex commands (8%), validating boot JDK inputs (8%), and avoiding weak file-change detection (7%) (Table 8). This indicates that no checklist item achieves near-universal adoption, even for fundamental practices. Automated auditing reveals systematic CI gaps. GPT 5 adjudication improves LLM-based multi-model evaluation, enabling scalable workflow quality assessment with minimal human effort. Applying this approach to open-source workflows shows that, while pipelines are generally readable and structured, they frequently lack robust error handling, modular decomposition, performance optimization, and security controls, highlighting persistent gaps in CI quality and the need for better tooling and community guidance.
How Compliant Are GitHub Actions Workflows? A Checklist-Based Study with LLM-Assisted Auditing
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
Table 8: Most frequently satisfied workflow checklist items. Rate = compliant frequency/total assessed per criterion Category
Subcategory / Criterion
Compliant Frequency
Rate
Section-wise
Jobs Workflow Steps Permissions
422 114 539 4
40% 40% 38% 4%
Theme-wise
Clarity Input Validation Environment Error / Failure Handling Modularity Performance Maintainability Security
129 178 82 147 143 143 110 147
68% 47% 43% 39% 38% 38% 29% 26%
Workflow handles failures / provides notifications Job names clear and unique Complex commands split into smaller steps Boot JDK platform inputs validated Avoid weak file-change detection (e.g., git status)
95 88 86 85 74
9% 8% 8% 8% 7%
Top 5 Criterion-wise
RQ4 Summary: Open-source GitHub workflows show low adherence to recommended practices, with overall compliance at 28–38%. Clarity is relatively high (68%), but security (26%) and maintainability (29%) remain weak. Under 10% follow even the most common practices, indicating persistent gaps in CI robustness, modularity, and security.
4
Discussion
Compliance is driven by fine-grained, cross-layer decisions. RQ1 shows that most compliance criteria target step-level configurations, yet many of these criteria cut across workflow, job, and step boundaries. Security, environment setup, and modularity cannot be assessed in isolation because their correctness depends on how decisions propagate across layers. This explains why syntax-level or section-local checks are insufficient: many violations emerge only when execution logic, runner context, and global settings are considered together. Effective compliance analysis therefore requires cross-section reasoning that captures execution semantics rather than surface structure alone. LLMs exhibit systematic, not random, inconsistency. RQ2 and RQ3 jointly show that disagreement among LLMs is widespread but patterned. Models agree on deterministic, structural criteria such as modularity and naming, but diverge on security, environment, and conditional logic. These disagreements stem from differing implicit assumptions about applicability, risk tolerance, and defaults, not from noise. As a result, LLM judgments are neither interchangeable nor safely composable without explicit resolution mechanisms. Treating LLM outputs as authoritative labels is therefore unjustified for CI compliance tasks. Adjudication mitigates ambiguity but exposes hard limits. RQ3 demonstrates that adjudication with a stronger model resolves a majority of conflicts and substantially increases label stability. However, nearly half of disputed cases still require human judgment, especially where compliance depends on workflow intent, tool semantics, or context-specific security expectations. These cases
reveal a fundamental limit of context-free reasoning over YAML: some compliance questions are underspecified without projectlevel knowledge. Adjudication improves efficiency, but it does not eliminate the need for expert review. Low compliance reflects Gaps in practice, Not just detection. RQ4 shows that adherence to recommended GHA practices is uniformly low across open-source workflows, including for basic security, permissions, and error-handling criteria. Even after adjudication and manual review, no checklist item approaches widespread adoption. This suggests that non-compliance is not primarily an artifact of model disagreement but reflects genuine gaps in CI practice. Workflows tend to be readable and functional, yet systematically under-instrumented for robustness, performance, and security.
4.1
Implications
For CI Workflow Developers. The results show that most compliance violations stem from step-level logic and conditionals, not high-level structure. Developers should prioritize validating inputs, constraining permissions, and making execution intent explicit over simply making workflows runnable. Security and error handling must be encoded in each workflow step, not assumed or inherited. Checklists based on platform documentation provide a practical baseline for self-auditing before using automated tools. For AI Tool Developers. Our results show that LLMs struggle most with context-dependent and non-applicable compliance criteria, causing systematic over-flagging. Auditing tools should expose uncertainty and distinguish mandatory from conditional rules instead of forcing binary decisions, potentially leveraging AI-assisted mechanisms such as CI/CD agents [12] to flag uncertain cases and defer ambiguous decisions. Multi-model adjudication is more reliable than single-model, but must surface unresolved cases for human review rather than hiding disagreement. For Researchers. This study derives 30 compliance criteria from workflow and job documentation, yet many violations occur at the step level and in execution logic. This suggests that CI smell research should therefore look beyond workflow or job structure, even
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
when based on documentation. Future work can refine documentationdriven approaches with finer-grained criteria for command composition, conditional execution, input handling, and security-sensitive steps. These extensions would complement existing smell taxonomies and better reflect where compliance failures actually occur. For CI Service Providers (e.g., GitHub Actions). The low adherence rates indicate that best practices are either hard to discover, hard to apply, or poorly enforced. CI providers can improve compliance by offering stronger defaults, clearer normative guidance, and first-class validation for security and permissions. Native tooling that explains why a configuration is risky or non-compliant, rather than merely flagging it, would reduce ambiguity for both humans and automated auditors.
5
Threats to Validity
Construct Validity. The compliance checklist was derived from official GitHub Actions documentation and refined through iterative application to real workflows. This grounding reduces the risk of inventing ad hoc or undocumented criteria. However, the checklist focuses primarily on workflow- and job-related guidance, even though many violations manifest at the step level. While this was an intentional design choice to anchor criteria in documented platform guarantees, it may underrepresent undocumented or communitydriven practices. Some criteria are conditionally applicable (e.g., caching, security scanning, input validation), and applicability depends on workflow intent that is not always explicit in YAML. Although the N/A option was used to mitigate this issue, misclassification of applicability remains possible and may contribute to disagreement. Finally, some YAML files represent partial CI configurations (e.g., reusable workflows or auxiliary pipelines), which may appear non-compliant when assessed in isolation despite being correct in the project context. Internal Validity. LLM outputs may be influenced by prompt formulation, model defaults, or undocumented heuristics. We mitigated this by using a fixed zero-shot prompt, deterministic decoding for open-weight models, and structured JSON outputs. Still, different prompting strategies or few-shot examples might yield different results, and prompt sensitivity remains an avenue for future investigation. Manual adjudication introduces potential reviewer bias. To reduce this threat, we used stratified sampling with full criterion coverage, blinded review, and documentation-backed resolution of disagreements. GPT 5 was used only as a dispute resolver and not treated as ground truth. Nevertheless, some borderline cases remain inherently subjective, especially for security and performance practices, which could affect adjudication outcomes. External Validity. The study focuses on Java-based open-source projects using GitHub Actions. While this controls for ecosystem variability, results may differ for other languages, build tools, or private repositories with stricter security policies. The selected LLMs represent diverse open-weight architectures plus a stronger proprietary model, but findings may not generalize to other models or future versions. The workflows analyzed are from public repositories with minimum activity thresholds, likely biasing the sample toward better-maintained projects. This means the low compliance rates observed are probably conservative estimates rather than overstatements.
Edward Abrokwah and Taher A. Ghaleb
Conclusion Validity. Agreement metrics and compliance rates depend on how N/A cases are handled and how split decisions are resolved. While we report intermediate agreement distributions and use multiple resolution paths (consensus, adjudication, manual review), alternative aggregation strategies could yield slightly different absolute rates. However, the main conclusions—systematic LLM disagreement, concentration of violations at fine-grained levels, and low overall adherence—are robust across resolution stages.
6
Conclusion
This paper introduced a documentation-grounded checklist of 30 compliance criteria for auditing GitHub Actions (GHA) workflows and evaluated the consistency of large language models (LLMs) in applying it. The criteria were derived from official GHA documentation and focused on workflow compliance, providing a reproducible basis for automated auditing. Our evaluation shows substantial variation across models. Gemma-3 12B achieved the highest agreement rate (90%), while LLaMA-3.1 8B performed lowest (61%). Overall agreement was limited, with Fleiss’ 𝜅 = 0.28, indicating only fair consistency. Split cases (35%) revealed recurring challenges in workflow reasoning, including job dependencies, secrets usage, and caching behavior. Using GPT 5 as an adjudicator resolved 57% of split cases and reduced manual effort, but human review remained necessary for ambiguous or context-dependent decisions. Overall, the results suggest that LLMs can support CI compliance auditing, but reliable use requires a hybrid setup combining automated reasoning with expert adjudication. For CI developers, the checklist can support the design and review of GitHub Actions workflows. For AI tool builders, the results highlight the need for stronger documentation grounding and better reasoning over cross-job dependencies. For researchers, the findings suggest defining compliance criteria directly from documentation and extending analysis beyond isolated jobs to workflow-level properties. For CI providers such as GitHub Actions, the criteria could support native linting and auditing features to improve clarity and trust in automated checks. Future work. Future work should extend the checklist to incorporate additional criteria about workflow structure, triggers, built-in features, and reusable components. We plan to evaluate our work on a larger and more diverse set of workflows across programming languages, repository types, and CI frameworks. We also aim to improve LLM grounding through fine-tuning and retrieval over official CI documentation to reduce disagreement and hallucination. Finally, we will conduct cross-project analyses to study variations in compliance patterns and LLM performance across domains and development practices.
Artifact Availability A replication package (scripts, data, and raw results) used to produce the findings of this study is available online on GitHub [2].
Acknowledgments This work is funded by the Natural Sciences and Engineering Research Council of Canada (NSERC): RGPIN-2025-05897.
How Compliant Are GitHub Actions Workflows? A Checklist-Based Study with LLM-Assisted Auditing
References [1] Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. 2024. Phi-4 technical report. arXiv preprint arXiv:2412.08905 (2024). [2] Edward Abrokwah and Taher A. Ghaleb. 2026. Auditing GitHub Actions Workflows: A Compliance Checklist and Evaluation Using LLMs (Replication Package). https://github.com/Taher-Ghaleb/GHACompliance-EASE2026. [3] Alireza Amiri-Margavi, Iman Jebellat, Ehsan Jebellat, and Seyed Pouyan Mousavi Davoudi. 2025. Enhancing answer reliability through inter-model consensus of large language models. In IFIP International Conference on Artificial Intelligence Applications and Innovations. Springer, 299–316. [4] Moritz Beller, Georgios Gousios, and Andy Zaidman. 2017. Travistorrent: Synthesizing Travis CI and GitHub for full-stack research on continuous integration. In 2017 IEEE/ACM 14th International Conference on Mining Software Repositories. IEEE, 447–450. [5] Robert E Blackwell, Jon Barry, and Anthony G Cohn. 2024. Towards reproducible llm evaluation: Quantifying uncertainty in llm benchmark scores. arXiv preprint arXiv:2410.03492 (2024). [6] Łukasz Chomątek, Jakub Papuga, Przemyslaw Nowak, and Aneta PoniszewskaMarańda. 2025. Decoding CI/CD Practices in Open-Source Projects with LLM Insights. In Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering. 1638–1644. [7] Nitika Chopra and Taher A Ghaleb. 2025. From First Use to Final Commit: Studying the Evolution of Multi-CI Service Adoption. In International Conference on Software Maintenance and Evolution. IEEE, 773–778. [8] Codecov. 2021. Codecov Bash Uploader Security Incident. https://about.codecov. io/security-update. Accessed: 2026-01-20. [9] Cycode Security Research. 2024. How we found vulnerabilities in GitHub Actions CI/CD pipelines. https://cycode.com/blog/github-actions-vulnerabilities. Accessed: 2026-01-20. [10] Martin Fowler. [n. d.]. Continuous Integration. https://martinfowler.com/articles/ originalContinuousIntegration.html. [11] Keheliya Gallaba and Shane McIntosh. 2018. Use and misuse of continuous integration features: An empirical study of projects that (mis) use Travis CI. IEEE Transactions on Software Engineering 46, 1 (2018), 33–50. [12] Taher A Ghaleb. 2026. When AI Agents Touch CI/CD Configurations: Frequency and Success. In Proceedings of the 23rd International Conference on Mining Software Repositories. ACM, 1–5. [13] Taher A Ghaleb, Osamah Abduljalil, and Safwat Hassan. 2026. CI/CD Configuration Practices in Open Source Android Apps: An Empirical Study. ACM Transactions on Software Engineering and Methodology 35, 2 (2026), 1–40. [14] Taher Ahmed Ghaleb, Daniel Alencar da Costa, and Ying Zou. 2019. An empirical study of the long duration of continuous integration builds. Empirical Software Engineering 24, 4 (2019), 2102–2139. [15] Taher Ahmed Ghaleb, Daniel Alencar da Costa, Ying Zou, and Ahmed E Hassan. 2019. Studying the Impact of Noises in Build Breakage Data. IEEE Transactions on Software Engineering (2019), 1–14. doi:10.1109/TSE.2019.2941880 [16] Taher A Ghaleb, Safwat Hassan, and Ying Zou. 2022. Studying the interplay between the durations and breakages of continuous integration builds. IEEE Transactions on Software Engineering 49, 4 (2022), 2476–2497. [17] Taher A Ghaleb and Dulina Rathnayake. 2025. Can LLMs Write CI? A Study on Automatic Generation of GitHub Actions Configurations. In 2025 IEEE International Conference on Software Maintenance and Evolution. IEEE, 767–772. [18] GitHub. [n. d.]. GitHub Actions Documentation. https://docs.github.com/en/ actions. Accessed: 2025-11-17. [19] Jinyao Guo, Chengpeng Wang, Xiangzhe Xu, Zian Su, and Xiangyu Zhang. 2025. Repoaudit: An autonomous llm-agent for repository-level code auditing. arXiv preprint arXiv:2501.18160 (2025). [20] Aayush Gupta. 2026. ReliabilityBench: Evaluating LLM Agent Reliability Under Production-Like Stress Conditions. arXiv preprint arXiv:2601.06112 (2026). [21] Michael Hilton, Nicholas Nelson, Timothy Tunnell, Darko Marinov, and Danny Dig. 2017. Trade-offs in continuous integration: assurance, security, and flexibility. In Proceedings of the 11th Joint Meeting on Foundations of Software Engineering. 197–207. [22] Md Nazmul Hossain and Taher A Ghaleb. 2025. CIgrate: Automating CI Service Migration with Large Language Models. arXiv preprint arXiv:2507.20402 (2025). [23] IN-COM DATA SYSTEMS. 2025. What Is the Difference Between Static Code Analysis and Linting? https://www.in-com.com/blog/what-is-the-differencebetween-static-code-analysis-and-linting. Accessed: 2026-01-20. [24] Ali Khatami, Cćdric Willekens, and Andy Zaidman. 2024. Catching smells in the act: A GitHub Actions workflow investigation. In 2024 IEEE International Conference on Source Code Analysis and Manipulation (SCAM). IEEE, 47–58. [25] Legit Security Research Team. 2024. Vulnerable GitHub Actions Workflows Part 1: Privilege Escalation Inside Your CI/CD Pipeline. https://www.legitsecurity. com/blog/github-privilege-escalation-vulnerability. Accessed: 2026-01-20. [26] Yiqi Liu, Nafise Sadat Moosavi, and Chenghua Lin. 2024. LLMs as narcissistic evaluators: When ego inflates evaluation scores. In Findings of the Association for Computational Linguistics: ACL 2024. 12688–12701.
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
[27] Francisco S Marcondes, Adelino Gala, Renata Magalhães, Fernando Perez de Britto, Dalila Durães, and Paulo Novais. 2025. Using Ollama. In Natural Language Analytics with Generative Large-Language Models: A Practical Approach with Ollama and Open-Source LLMs. Springer, 23–35. [28] Deep Mehta, Kartik Rawool, Subodh Gujar, and Bowen Xu. 2023. Automated devops pipeline generation for code repositories using large language models. arXiv preprint arXiv:2312.13225 (2023). [29] Mistral AI. 2024. Mistral 7B. https://mistral.ai/news/mistral-7b/ Accessed: 2025-10-18. [30] Oligo Security. 2025. Static Code Analysis: Top 7 Methods, Pros/Cons and Best Practices. https://www.oligo.security/academy/static-code-analysis. Accessed on Jan 20, 2026. [31] OpenAI. 2025. Introducing GPT-5. https://openai.com/gpt-5 Online; accessed 27-August-2025. [32] OpenAI. 2026. Reasoning Models Guide. https://developers.openai.com/api/docs/ guides/reasoning. Accessed:2026. [33] Charilaos Pipis, Shivam Garg, Vasilis Kontonis, Vaishnavi Shrivastava, Akshay Krishnamurthy, and Dimitris Papailiopoulos. 2025. Wait, Wait, Wait... Why Do Reasoning Models Loop? arXiv preprint arXiv:2512.12895 (2025). [34] Enio G Santana Jr, Jander Pereira Santos Junior, Erlon P Almeida, Iftekhar Ahmed, Paulo Anselmo da Mota Silveira Neto, and Eduardo Santana de Almeida. 2025. Evaluating LLMs Effectiveness in Detecting and Correcting Test Smells: An Empirical Study. arXiv preprint arXiv:2506.07594 (2025). [35] Klaas-Jan Stol, Paul Ralph, and Brian Fitzgerald. 2016. Grounded theory in software engineering research: a critical review and guidelines. In Proceedings of the 38th International conference on software engineering. 120–131. [36] Rickard Stureborg, Dimitris Alikaniotis, and Yoshi Suhara. 2024. Large language models are inconsistent and biased evaluators. arXiv preprint arXiv:2405.01724 (2024). [37] Davide Taibi, Andrea Janes, and Valentina Lenarduzzi. 2017. How developers perceive smells in source code: A replicated study. Information and Software Technology 92 (2017), 223–235. [38] Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786 (2025). [39] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023). [40] Pablo Valenzuela-Toledo, Alexandre Bergel, Timo Kehrer, and Oscar Nierstrasz. 2024. The Hidden Costs of Automation: An Empirical Study on GitHub Actions Workflow Maintenance. In 2024 IEEE International Conference on Source Code Analysis and Manipulation (SCAM). IEEE, 213–223. [41] Carmine Vassallo, Sebastian Proksch, Anna Jancso, Harald C Gall, and Massimiliano Di Penta. 2020. Configuration smells in continuous delivery pipelines: a linter and a six-month study on GitLab. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 327–337. [42] Stefan Wagner, Marvin Muñoz Barón, Davide Falessi, and Sebastian Baltes. 2025. Towards evaluation guidelines for empirical studies involving LLMs. In 2025 IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering (WSESE). IEEE, 24–27. [43] Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, et al. 2024. Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. 9440–9450. [44] Weiyuan Xu, Juntao Luo, Tao Huang, Kaixin Sui, Jie Geng, Qijun Ma, Isami Akasaka, Xiaoxue Shi, Jing Tang, and Peng Cai. 2025. LogSage: An LLM-Based Framework for CI/CD Failure Detection and Remediation with Industrial Validation. In 40th IEEE/ACM International Conference on Automated Software Engineering. [45] Fiorella Zampetti, Carmine Vassallo, Sebastiano Panichella, Gerardo Canfora, Harald Gall, and Massimiliano Di Penta. 2020. An empirical characterization of bad practices in continuous integration. Empirical Software Engineering 25, 2 (2020), 1095–1135. [46] Chen Zhang, Bihuan Chen, Junhao Hu, Xin Peng, and Wenyun Zhao. 2022. BuildSonic: Detecting and repairing performance-related configuration smells for continuous integration builds. In Proceedings of the 37th IEEE/ACM international conference on automated software engineering. 1–13. [47] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging LLM-as-a-Judge with MT-Bench and chatbot arena. Advances in neural information processing systems 36 (2023), 46595–46623. [48] Xiaoxin Zhou, Taher A Ghaleb, and Safwat Hassan. 2026. Role of CI Adoption in Mobile App Success: An Empirical Study of Open-Source Android Projects. In Proceedings of the 23rd International Conference on Mining Software Repositories. ACM, 1–12.