GHA-SmellFix: Taking Action with Agentic LLMs for Detecting and Refactoring GitHub Actions Workflow Smells

arXiv:2611.14963v1 [cs.SE] 11 Nov 2026

GHA-SmellFix: Taking Action with Agentic LLMs for Detecting and Refactoring GitHub Actions Workflow Smells ANDERSON UCHÔA✉ , Federal University of Ceará, Brazil and North Carolina State University, USA JHEFFERSON SOUZA, Federal University of Ceará, Brazil EMANUEL ÁVILA, Federal University of Ceará, Brazil GABRIEL SILVA, Federal University of Ceará, Brazil KEVIN LIRA, North Carolina State University, USA HUI SUN, North Carolina State University, USA NELSON ARAÚJO, Federal University of Ceará, Brazil ARTHUR CORDEIRO, Federal University of Ceará, Brazil ROHIT GHEYI, Federal University of Campina Grande, Brazil WESLEY ASSUNÇÃO, North Carolina State University, USA Continuous integration and continuous delivery workflows have become key artifacts in modern software projects. In GitHub Actions, they encode build, test, release, deployment, and security automation through YAML configuration files. Unfortunately, these workflows often contain recurring configuration smells that can affect security posture, reliability, maintainability, or operational stability. Existing methods to address GitHub Actions workflow smells have two limitations: (i) they are mostly rule-based; however, rule creation and evolution are difficult tasks, and (ii) they stop at reporting findings; developers must still interpret these findings and craft fixes. Agentic LLM pipelines are a natural fit for GitHub Actions smell remediation because the task combines contextual detection, transformation selection, and patch generation. Assigning these responsibilities to specialized agents enables the pipeline to condition refactoring on positive detection verdicts and use task-specific reasoning at each stage, potentially avoiding unnecessary modifications. Yet, it remains unclear whether this decomposition improves effectiveness or mainly changes operational overhead. We present GHA-SmellFix, an agentic LLM pipeline for detecting and refactoring GitHub Actions workflow smells. GHA-SmellFix is instantiated with two orchestration designs: (i) Joint pipeline, a single agent detects and refactors, and (ii) Decomposed pipeline, separate agents detect and then refactor. To empirically compare these two designs, we operationalize 19 smells from an existing catalog with explicit detection criteria, refactoring expectations, exclusion rules, and safety checks, and build a manually curated ground truth of 1,710 (workflow, smell) pairs. We instantiate the two designs with four LLMs (GPT-5.4-nano, Gemini 3.5 Flash, Claude Haiku 4.5, and DeepSeek V4 Flash) and evaluate them on 570 pairs sampled from this ground truth. Our results show that the key value of agentic decomposition is not universally better patch quality, but better control over whether the agent attempts patch generation. Decomposition particularly helps models prone to missing smells and can reduce operational overhead by gating patch generation. Once a smell is correctly detected, clean refactoring is generally effective, making target-smell detection the primary end-to-end bottleneck. However, false-positive detections can trigger unnecessary edits to execution-relevant workflow constructs. Thus, practical CI/CD agents need not only effective patch generation, but also reliable abstention and construct-aware validation to avoid unnecessary or execution-relevant workflow changes. CCS Concepts: • Software and its engineering → Software maintenance tools; Software evolution; • Computing methodologies → Artificial intelligence. Additional Key Words and Phrases: GitHub Actions, CI/CD, configuration smells, large language models, agentic pipelines, refactoring, empirical software engineering

1

Introduction

Continuous Integration and Continuous Delivery (CI/CD) pipelines have become key maintenance artifacts in modern software engineering [15, 18, 39, 47]. GitHub Actions is widely adopted to implement such pipelines through repository-level YAML files, called workflows, that automate

Uchôa et al.

builds, tests, releases, deployments, and security-related tasks [8, 9]. Although workflows have a declarative syntax [17], they are not simple configuration artifacts. They may invoke third-party actions, access repository secrets, publish artifacts, define permissions, execute scripts, and interact with deployment environments [25]. As a result, the way a workflow is written can affect the security posture [24, 25], reliability, maintainability, and operational stability of software projects [15, 40, 47]. Prior studies show that CI/CD workflows frequently contain configuration smells or anti-patterns, i.e., recurring patterns that may increase the risk of vulnerabilities, unstable builds, excessive resource usage, or performance degradation [15, 24, 25, 40, 47]. Existing methods for analyzing quality and security problems in GitHub Actions workflows are largely rule-based [4, 14, 24]. These methods encode known patterns and provide inexpensive, reproducible screening. However, they have two main limitations. First, their detection logic depends on engineered rules whose design and maintenance can require substantial effort as workflow constructs, platform features, and smell definitions evolve [2, 9, 28]. For instance, checking effective permissions may require combining workflow-level and job-level permissions, while fork-guard smells depend on event triggers, conditional expressions, and repository context rather than on a single YAML key. Second, only reporting a smell does not complete the maintenance task: currently, developers must still judge whether the finding is relevant, decide how to remediate it, and assess whether the resulting patch is correct or introduces unrelated behavior changes [3, 22, 46]. This is challenging because workflow refactoring may involve constructs such as permissions, triggers, conditions, runners, dependencies, deployment steps, caches, matrices, and reusable actions or workflows [2, 28]. Thus, the open problem is not only detecting smells, but generating and evaluating candidate refactorings that are structurally valid, scope-bounded, and abstention-aware. Large Language Models (LLMs) offer a promising mechanism for this broader maintenance task because they can interpret configuration context, reason about intended changes, and generate candidate patches [20]. However, LLM-generated edits may be risky: a patch may remove the target smell while changing unrelated triggers, permissions, runners, dependencies, or deployment behavior. Agentic pipelines can organize LLM capabilities into explicit roles, for instance, by separating smell detection from patch generation [41]. Yet, we still lack a clear understanding of how decomposing workflow maintenance tasks affects detection and refactoring performance, as well as its trade-offs in execution latency, token efficiency, and API cost. This paper presents GHA-SmellFix, an agentic LLM pipeline for identifying and refactoring GitHub Actions configuration smells. Given smell definitions, a workflow file, and repository-level context, GHA-SmellFix detects whether a smell is present and, when appropriate, generates a scope-bounded candidate refactoring. We compare two GHA-SmellFix orchestration designs: (D1) a joint design that performs detection and, when applicable, refactoring in one pass; and (D2) a decomposed design in which a detection agent first decides whether the target smell is present, and a refactoring agent is invoked only for detected smells. These designs differ in task decomposition, prompt context, number of LLM calls, and control flow; therefore, we use them as practical designs for studying effectiveness, patch quality, abstention behavior, and operational overhead, not as an isolated causal test of adding one agent. To evaluate these designs, we operationalize 19 GitHub Actions smells from an existing catalog [24], defining explicit detection criteria, refactoring expectations, exclusion rules, construct-level scope, and risk checks. We then instantiate the two designs using four LLMs (GPT-5.4-nano, Gemini 3.5 Flash, Claude Haiku 4.5, and DeepSeek V4 Flash) and compare them in paired cross-model and cross-design evaluations on 570 annotated (workflow, smell) pairs. These pairs were sampled from a 1,710-pair ground truth manually curated for this study, covering positive and negative cases for 19 smells. To assess generated refactorings, we use a construct-aware evaluation protocol that distinguishes clean smell removals, removals with unexpected edits, removals with potential

GHA-SmellFix: Taking Action on Workflow Smells

behavior changes, failed/no-patch cases, and unnecessary edits on ground-truth-negative pairs. We analyze detection effectiveness, end-to-end and conditional refactoring outcomes, abstention behavior, and overhead in terms of latency, token usage, and estimated API cost. Our results show that agentic decomposition provides selective rather than universal benefits for GitHub Actions maintenance. Separating detection from refactoring modestly improves aggregate detection, increasing average F1 from 86.34% to 87.55% and recall from 79.12% to 82.37%, with the largest gain for GPT-5.4-nano, whose F1 rises from 81.38% to 87.15%; models already strong in the joint setting change little. More importantly, the main end-to-end bottleneck is detecting when the target smell is present rather than generating the patch itself : clean removal reaches 74% overall, but rises to 92.6% once the target smell is correctly detected. At the same time, successful smell removal does not guarantee patch quality, and abstention remains challenging: the pipelines produced 93 unnecessary edits on negative cases, 54.84% of which affected behavior-relevant workflow constructs. Decomposition also need not increase operational cost, as the detection gate reduces latency for three of the four evaluated models by avoiding unnecessary refactoring calls. Overall, these findings position agentic LLM pipelines as promising review-support mechanisms for CI/CD maintenance, whose practical value depends on model- and smell-aware orchestration, reliable abstention, and construct-aware validation rather than on decomposition alone. Our contributions include: (i) GHA-SmellFix, an agentic LLM pipeline for target-smell detection and candidate refactoring; (ii) a manually curated ground truth of 1,710 annotated (workflow, smell) pairs, including positive and negative cases, designed to support paired evaluation of detection and refactoring behavior; (iii) a construct-aware evaluation protocol for LLM-generated workflow patches, covering smell removal, workflow validity, expected construct changes, unexpected edits, potential behavior-impacting changes, failed/no-patch cases, and unnecessary edits; and (iv) an empirical evaluation of two GHA-SmellFix orchestration designs across four LLMs, covering effectiveness, abstention, construct-level safety, and operational overhead. 2

Related Work

CI/CD anti-patterns and workflow quality. Prior work has shown that CI/CD pipelines accumulate recurring configuration problems that affect maintainability, feedback, security, and delivery behavior. Hilton et al. [18] discuss trade-offs in CI adoption, including assurance, security, and flexibility. Gallaba and McIntosh [15] show that CI features are often used or misused in ways that affect build behavior and feedback. Zampetti et al. [47] characterize bad practices in CI pipelines, while Vassallo et al. [39, 40] study CI/CD anti-patterns, decay, configuration smells, and linter-based reporting. These studies establish CI/CD configuration quality as a recurring maintenance concern, but they focus mainly on characterizing, detecting, or reporting problematic practices rather than generating and evaluating candidate refactoring. GitHub Actions workflows, smells, and scanners. GitHub Actions brings CI/CD quality concerns to repository-level YAML workflows. Prior work has studied the adoption and evolution of GitHub Actions workflows [8, 9, 28, 37, 38], their security risks [25], and opportunities for resourceusage optimization [6]. Recent work also shows that workflow reliability and maintenance effort are related to specific constructs, such as triggers, jobs, steps, permissions, matrices, conditions, and reusable components [2]. This construct-level perspective is important for refactoring because an edit may be syntactically valid while still changing behavior-relevant parts of the automation. Several tools and catalogs target GitHub Actions quality and security. Khatami et al. [24] introduced a catalog of smells and the GHA analyzer. GHAST [4] and GASH [14] inspect workflows for recurring security or configuration problems, while Fares et al. [12] show that workflow security scanners differ substantially in scope, detected weaknesses, and usability. These approaches are

Uchôa et al.

useful as scalable detectors, but they stop at reporting findings. They do not assess whether generated patches remove a target smell, stay within the expected construct scope, or abstain when the smell is absent. Our work evaluates generated candidate patches, not only detects smell instances. LLM-based agents for software maintenance. LLM-based agents have recently been applied to software maintenance tasks that involve repository interaction, patch generation, and validation. SWE-agent [45] and OpenHands [42] frame maintenance as an interactive repository-level task, while RepairAgent [5] uses agentic repair loops to plan and validate program fixes. Recent work on test smells shows that agentic language models can support both smell detection and refactoring in source-code-centric artifacts [31]. GitHub Actions workflows differ from these settings: small YAML edits can alter permissions, event exposure, secret access, runner selection, artifact publication, deployment behavior, or reusable-workflow invocation. Thus, patch quality cannot be assessed only by syntactic validity or smell removal. Moreover, recent evidence cautions against assuming that more complex agentic orchestration necessarily improves software engineering outcomes; simpler localization–repair–validation pipelines can compete with more autonomous agents [44]. We therefore treat decomposition as an empirical design choice rather than an assumed improvement. 3

Study Settings

To design our study, we adopted the Goal-Question-Metric (GQM) template [7]. The goal is to analyze GHA-SmellFix orchestration designs; for the purpose of detecting and refactoring configuration smells in GitHub Actions workflows; with respect to (i) target-conditioned detection effectiveness, (ii) refactoring correctness, construct-level consistency, and abstention behavior, and (iii) the effectiveness–overhead trade-offs of task decomposition; from the viewpoint of researchers studying LLM-based CI/CD maintenance; in the context of a manually curated ground truth of (workflow, smell) pairs covering 19 configuration smells. The research questions (RQ) are: RQ1 : How effectively can each GHA-SmellFix orchestration design detect target GitHub Actions configuration smells? – This RQ evaluates target-conditioned detection over explicit (workflow, smell) pairs. Each prediction is a binary decision about whether the target smell is present or absent, compared with the annotated ground truth using precision, recall, accuracy, and F1-score. RQ2 : To what extent can each GHA-SmellFix orchestration design generate structurally valid and scope-bounded refactorings for GitHub Actions workflow smells? – This RQ evaluates end-to-end and conditional refactoring outcomes. We examine whether generated patches remove the target smell, preserve YAML and GitHub Actions structural validity, stay within the expected construct scope, and avoid potential behavior-impacting changes. We also analyze unnecessary edits on ground-truth negative pairs, where abstention is expected. RQ3 : What is the effectiveness–overhead trade-off of task decomposition in GHA-SmellFix? – This RQ compares (D1) a joint detect-and-refactor design with (D2) a decomposed detect-then-refactor design. These designs differ in task decomposition, prompt context, number of LLM calls, and control flow. We compare detection effectiveness, end-to-end and conditional refactoring outcomes, abstention behavior, latency, token usage, and estimated cost. 3.1

Study Steps and Procedures

Step 1: Repository selection and workflow extraction. We relied on the GitHub Actions workflow dataset by Cardoen et al. [8] to identify open-source repositories that use GitHub Actions, deriving 49,510 candidate repositories. We filtered repositories based on criteria adapted from prior studies [23, 32, 34]. We kept repositories that: (i) had at least 100 GitHub stars; (ii) had at least 100 commits; (iii) had activity within the last 12 months; (iv) were at least six months old; (v) used main or master as the default branch; and (vi) contained at least two workflow files. We excluded repositories that were: (i) forks; (ii) archived; (iii) named as learning, tutorial, clone, template,

GHA-SmellFix: Taking Action on Workflow Smells

actions, workflows, or reusable examples; or (iv) created primarily for saving workflows or actions. At the workflow level, we kept files that: (i) were triggered by pull_request or push events; and (ii) had at least 100 execution runs. We excluded workflow files that: (i) had no modification history; (ii) had names or filenames containing documentation-related terms, such as docs.yml; (iii) were empty or contained placeholder content; or (iv) had invalid YAML syntax. With these criteria, we selected a dataset comprising 27,704 repositories and 49,836 unique GitHub Actions workflow files. We then used the GitHub API to retrieve repository metadata and collect the current configuration files under the .github/workflows/ directory, adopting a snapshot cut-off on April 9, 2026. Step 2: Selection and operationalization of configuration smells. We rely on Khatami et al.’s catalog [24] because it provides fine-grained GitHub Actions smells and tool support through the GHA analyzer, which we use for sampling. The catalog is grounded in evidence of workflow evolution: candidate smells were derived from recurring patterns of workflow change and examined through a contribution study with maintainers. The catalog contains 22 smells. We selected 19 and excluded AddCommentsToWorkflows, CorrectIndentation, and NameRunSteps, because they mainly capture readability or formatting conventions rather than security, performance, or CI/CD execution concerns. We refined the selected smells into explicit detection conditions, exclusion criteria, uncertainty-handling rules, and counting units. These refinements support consistent manual annotation and agent-based evaluation, especially for ambiguous GitHub Actions constructs such as reusable workflows, local actions, matrices, fork guards, and deployment-related steps. We also cross-checked the refined definitions against the official GitHub Actions documentation [17]. Based on the refined definitions, two authors classified each smell as localized or context-dependent through discussion and consensus. Localized smells rely on explicit evidence within a bounded set of workflow constructs, such as a job, step, action reference, permission block, timeout, or runner declaration. context-dependent smells require combining evidence across workflow constructs, scopes, relationships, or repository-level context, such as triggers, fork guards, deployment behavior, concurrency semantics, matrices, reusable workflows, or workflow intent. Table 1 overviews the smells and the core operational criterion for each smell. Table 1. GitHub Actions configuration smells, refined from Khatami et al. [24], and ground-truth coverage. P/N denotes positive/negative pairs; full definitions are in the replication package. P/N

Localized

Core operational criterion

MissingPermExternalActions A job uses an external step-level GitHub action and has no effective workflow- or job-level permissions block. UnpinnedActionVersion A step-level GitHub action reference uses owner/repo@ref, but ref is not a full 40-character commit SHA. MissingTokenPermissions A job uses GITHUB_TOKEN or authenticated GitHub operations without an effective permissions block. MissingJobTimeout A job defined in the current workflow file does not declare job-level timeout-minutes. UnsafeArtifactUploadFork A PR-triggered workflow uploads retained GitHub Actions artifacts without an effective fork guard. MissingUploadCondition A retained artifact-upload step lacks an explicit step-level or job-level if condition. MutableRunnerVersion A job uses a mutable GitHub-hosted runner label ending in -latest. SuboptimalCacheUsage A workflow uses actions/cache for dependencies already supported by setup-action built-in caching. MultiCommandStep A run step combines separable commands from different workflow phases. UnpinnedDependencies A run step installs external packages or tools without fixed versions or equivalent immutable references.

50/50 36/36 50/50 48/48 33/33 38/38 50/50 49/49 25/25 38/38

Contextdependent

Type Configuration smell

MissingForkGuardPR MissingConcurrencyPR MissingConcurrencyBranch MissingPathFilters MissingForkGuardSchedule UnsafeDeployOnFork MissingConcurrencyGlobal SingleOSExecution SingleLangVersionCI

An issue/PR-triggered workflow performs issue or pull-request actions without an effective fork guard. A PR-triggered workflow lacks applicable concurrency.cancel-in-progress: true to cancel older runs. A branch-push workflow lacks applicable concurrency.cancel-in-progress: true to cancel older runs. A CI workflow triggered by push or pull_request lacks paths or paths-ignore filters. A scheduled workflow can run outside the intended repository without a repository, owner, or fork-status guard. A workflow can run in a fork-related PR context and perform external delivery without an effective fork guard. A repeated or side-effectful task can overlap because no effective concurrency group is defined. A test job effectively runs on only one operating system and has no multi-OS matrix. A CI job validates only one explicit language/runtime version and has no multi-version matrix.

50/50 50/50 50/50 39/39 50/50 50/50 50/50 50/50 49/49

Total

855/855

Step 3: Ground-truth construction and manual annotation. Figure 1 summarizes how we built the manually annotated ground truth for the smells in Table 1.

Uchôa et al.

2 Manual annotation

3 Balancing & subset

GHA-flagged 1,416 inspected

902 confirmed positives

Balanced ground truth 855 pos. + 855 neg.

GHA-unflagged 1,310 inspected

944 confirmed negatives

LLM-mutated 49 candidates

45 validated positives

1 Candidate retrieval

GHA-guided stratified sampling ≤ 1 instance per smell LOC ≤ 209

Manually labeled negative workflows of UnsafeDeployOnFork

514 discarded (absent)

incl. 45 semi-synthetic pos. undersampled: − 92 pos., − 89 neg.

366 discarded (present)

4 discarded (invalid)

Shared evaluation set 285 pos. + 285 neg. 15 + 15 pairs per smell

Fig. 1. Overview of ground-truth construction, manual annotation, and shared-set sampling.

Stratified sampling. We adopted pair-based stratified purposive sampling guided, but not labeled, by the GHA analyzer [24]. For each smell, we sampled positive candidates from workflows in which GHA analyzer reported exactly one target-smell instance and negative candidates from workflows where the target smell was not flagged. To reduce annotation ambiguity, we kept only workflows with at most one GHA-reported instance of any smell and removed LOC outliers using Tukey’s IQR filter [11] (threshold: 209 LOC). We ranked the remaining candidates by ascending total_unique_smells and descending workflow_run_count, prioritizing workflows with fewer cooccurring smells and more execution evidence. For each smell, we sampled rounds of 100 positive and 100 negative candidates, repeating for up to six rounds until we reached 50 validated pairs per class, and then balanced each smell by undersampling the larger validated class. The only exception was UnsafeDeployOnFork, for which we found only five natural positives. We therefore used Claude Opus 4.8 to mutate real workflows manually labeled negative for this smell, restricting edits to the target-smell construct. After manual validation, we retained 45 of the 49 mutants as positives, which entered balancing together with the natural pairs. We did not synthesize any negative pairs. Manual annotation. Six annotators worked in three mixed-experience pairs (two master’s– undergraduate pairs and one PhD–undergraduate pair). They had one to three years of software engineering experience and prior exposure to GitHub Actions and CI/CD workflows. Before annotation, we provided annotators with the refined smell catalog, annotation guidelines, and positive/negative examples for each smell. Within each pair, annotators independently inspected the same (workflow, smell) pairs, labeled the target smell as present or absent, and recorded the number of instances. We replaced pairs in which annotators identified multiple target-smell instances using the same iterative procedure, and each team resolved disagreements by consensus. We measured pre-consensus agreement over the 1,710 evaluated (workflow, smell) pairs from 902 unique workflow files using Cohen’s 𝜅 [29], obtaining almost perfect agreement [26] (𝜅 = 0.848); we report team-level values in the replication package. We discarded candidates whose manual label contradicted their sampling stratum rather than relabeling them into the natural ground truth (Figure 1). After adding the 45 validated semi-synthetic positives and balancing per smell, the final ground truth comprised 855 positive and 855 negative pairs, with workflows ranging from 7 to 207 LOC. From this ground truth, we sampled a shared cross-model evaluation set of 15 positive and 15 negative pairs per smell (𝑛 = 570). Table 1 also summarizes their distribution; per-smell workflow-size statistics and pair-level agreement results are available in our replication package. Step 4: GHA-SmellFix orchestration designs. We instantiate GHA-SmellFix with two agentic LLM orchestration designs that differ in task decomposition and control flow, while keeping the input format, smell catalog, output schemas, and underlying LLM fixed within each comparison. Figure 2 illustrates the two designs. D1 is a joint detect-and-refactor design: a single agent classifies the target smell as “present” or “absent” and, when applicable, returns a candidate patch. D2 is a decomposed detect-then-refactor design: a detection agent first classifies the target smell; if the smell is predicted as “absent”, the original workflow is returned unchanged; otherwise, a refactoring

GHA-SmellFix: Taking Action on Workflow Smells

D1 – Joint Design

D2 – Decomposed Design absent: original workflow returned

workflow, target smell, repository context

verdict and optional candidate patch

Detection and Refactoring

In

Out

workflow, target smell, repository context

In

present and detection result

candidate patch Refactoring

Detection

Out

Fig. 2. GHA-SmellFix orchestration designs for target-conditioned smell detection and refactoring.

agent generates a candidate patch. The agents are not given repository-level CLI access, external validators, or runtime services, allowing us to assess basic agentic capability. We implemented both designs using the Agno framework [10], with XML-tagged prompts and Pydantic schemas for structured outputs. Prompts combined role-based prompting with reasoningoriented instructions and prompt-level constraints [35]. These guardrails constrained agents to the target smell, required the smallest scope-bounded edit, discouraged unrelated workflow changes, enforced raw YAML outputs, and required confidence justifications grounded in workflow evidence. We logged execution metadata for each LLM call, including token usage and latency, to support overhead analysis. Complete prompt templates and schemas are provided in the replication package. Our study focuses on target-conditioned detection. GHA-SmellFix receives the target smell definition from our refined catalog, the workflow, and repository-level context (owner/repo, owner, default branch, and file path) to interpret repository-specific locations. We executed D1 and D2 on the shared 570-pair set for paired comparison; the full 1,710-pair ground truth serves as complementary evidence for DeepSeek V4 Flash. Step 5: LLM selection and execution. We selected four LLMs, presented in Table 2, varying in provider, family, scale, and access mode. We prioritized code-related capabilities, instruction following, and support for structured outputs because LLM effectiveness on SE tasks varies across models and deployment settings [20, 21, 33], and agentic pipelines depend on role-specific instructions and structured responses [5, 27, 36, 45]. We kept the input artifacts, role-specific prompt templates, and structured output schemas fixed across agents, changing only the underlying LLM. Between D1 and D2, the workflow input and target smell also remained fixed, while the orchestration changed by decomposing the task, changing prompt context, and adding a detection gate before refactoring. We used temperature 0 whenever supported to reduce sampling variability. GPT-5.4-nano and Claude Haiku 4.5 required temperature 1 in their thinking modes and were executed 𝑘 = 3 times to assess detection stability. To control execution cost, GPT-5.4-nano, Claude Haiku 4.5, and Gemini 3.5 Flash were evaluated on a balanced subset with 15 positive and 15 negative pairs per smell. Gemini 3.5 Flash was executed once. DeepSeek V4 Flash was evaluated once on the 1,710-pair ground truth. Overall, we ran (2 × 2 × 570 × 3) + (2 × 1 × 570) + (2 × 1 × 1,710) = 11,400 pipeline runs. Table 2. Overview of the selected LLMs. Costs are reported in USD per 1M tokens (June 2026). Model

Access

Type

GPT-5.4-nano Claude Haiku 4.5 Gemini 3.5 Flash DeepSeek V4 Flash

OpenAI API Anthropic API Gemini API OpenRouter API

Proprietary Proprietary Proprietary Open-weight

Params. – – – 284B (13B active)

Temp.

Thinking

In (1M/USD)

Out (1M/USD)

1 1 0 0

Medium 2,048 tokens Medium 2,048 tokens

0.20 1.00 1.50 0.064

1.25 5.00 9.00 0.251

GPT-5.4-nano and Gemini 3.5 Flash expose reasoning effort as a qualitative level rather than an explicit token budget; their token allocation is provider-managed.

Step 6: Data Analysis. We based our analysis on the shared set of 570 ground-truth (workflow, smell) pairs, covering 19 smells with 15 positive and 15 negative instances each. The full groundtruth results for DeepSeek V4 Flash are provided in the replication package.

Uchôa et al.

1

Input

2

Automated processing

3

4

Blinded assessment

Reported views

Dimensions: smell removal · validity · construct-level consistency invalid ⇒ Failed or No Patch

4,560

Workflow validity

outputs

YAML parsing + GHA structure

4 models

× 2 designs × 570 pairs

Positive pairs (smell removal + scope) Failed or No Patch

all GT positives

Removal w/ Unexpected Changes

Potential Behavior-Impacting Change

Conditional

Diff vs. original 2,280 GT-positive

per output

GT positives verdict = present

Negative pairs (modified outputs only)

2,280 GT-negative

End-to-end

Clean Removal

Construct extraction 117/197 relevant constructs [2]

Cosmetic Edit

Localized Unnecessary Edit

Abstention all GT negatives

Behavior-Relevant Unnecessary Edit unmodified ⇒ Correct Abstention

Two blinded annotators label each output independently (smell definition, expected construct scope, rationale, diff, and extracted constructs).

A third author adjudicates disagreements; both uncertain ⇒ not assessable. Runtime-dependent facts are not validated. Pre-consensus 𝜅 = 0.758.

Fig. 3. Overview of the refactoring assessment protocol. Dashed arrows indicate automatic assignments for invalid positive outputs and unmodified negative outputs; all other outputs are assessed by blinded annotators. Conditional rates use GT-positive outputs whose structured detection verdict is present.

Detection Analysis (RQ1 ). We assess detection effectiveness using Precision, Recall, F1-score, and Accuracy [13]. Each prediction is a binary decision indicating whether a target smell is present in a workflow; Accuracy therefore captures the proportion of correct present/absent classifications over positive and negative (workflow, smell) pairs. We also assess detection stability for GPT-5.4nano and Claude Haiku 4.5, which were executed three times under the same setup. Following prior work [16], we report accuracy spread, the difference between maximum and minimum accuracy across the 𝑘 = 3 executions for each (model, design) pair. This stability analysis is restricted to detection and to models executed three times. Refactoring Assessment (RQ2 ). Figure 3 summarizes our assessment protocol. We evaluate patches along three dimensions: smell removal, workflow validity, and construct-level consistency. Because refactoring depends on detection, we report end-to-end outcomes over all ground-truthpositive outputs and conditional outcomes restricted to correctly detected positives. For workflow validity, we check YAML parsability and required GitHub Actions structure; invalid outputs are unsuccessful refactorings. For the other dimensions, we diff original and refactored workflows, extract modified YAML paths, and normalize them into GitHub Actions constructs using Bardsiri et al. [2]. Of their 197 constructs, we use the 117 relevant to our 19 smells and map each smell to its expected modification scope. For instance, MissingJobTimeout targets jobs.<job_id>.timeout-minutes, whereas SingleOSExecution may involve jobs.<job_id>.strategy.matrix.<matrix_var>. These mappings define expected fix scope rather than fixed templates and are provided in the replication package. We manually inspected 4,560 outputs: 2,280 positive-pair outputs for refactoring assessment and 2,280 negative-pair outputs for abstention. The ground-truth annotator pairs assessed outputs blindly to model and design using the smell definition, expected construct scope, agent rationale, diff, and modified constructs. Because multiple valid fixes may exist, no single reference patch was used. Positive outputs were classified as Clean Removal when the smell was removed, validity preserved, and only expected constructs changed; Removal with Unexpected Changes when unrelated constructs were modified; Removal with Potential Behavior-Impacting Change when changes could affect workflow execution, security posture, or deployment behavior; or Failed or No Patch when the smell was not removed, validity checks failed, or no applicable patch was produced. For negative pairs, where abstention was expected, modified outputs were classified as Cosmetic Edit limited to formatting, comments, or naming; Localized Unnecessary Edit, for plausible but

GHA-SmellFix: Taking Action on Workflow Smells

unnecessary smell-related changes; or Behavior-Relevant Unnecessary Edit, for changes to executionrelevant constructs. If multiple edit types occurred, we assigned the most behavior-relevant category. Annotators could mark cases as uncertain when static evidence was insufficient. Disagreements were adjudicated by a third author, except when both annotators independently marked a case uncertain; these were treated as not assessable. Pre-consensus agreement was substantial (𝜅 = 0.758) [26, 29]. We do not validate runtime-dependent facts such as commit hashes, package versions, credentials, external actions, or services. Orchestration effect (RQ3 ). We compare D1 and D2 as practical GHA-SmellFix designs to assess whether task decomposition improves effectiveness or mainly changes overhead. This comparison does not isolate the effect of adding one agent. We reuse RQ1 metrics for detection, RQ2 outcomes for refactoring, and compare latency, token usage, and estimated API cost for overhead. For each LLM, we compare D1 and D2 outputs over the same (workflow, smell) pairs using McNemar’s exact test [30] for detection correctness and end-to-end clean removal, and paired Wilcoxon signed-rank tests [43] for latency and estimated cost, with Holm correction [19] within each metric. Execution protocol and stability. For GPT-5.4-nano and Claude Haiku 4.5, which were nondeterministic in our setup, we ran each model–design pair three times under D1 and D2. We use the first run in the main analyses to avoid post-hoc run selection, and use the repetitions only for stability analysis. The maximum accuracy spread is 2.11%, indicating low run-to-run variation. Gemini 3.5 Flash and DeepSeek V4 Flash were configured with temperature zero and executed once; therefore, stability conclusions are limited to GPT-5.4-nano and Claude Haiku 4.5. All repeated-run outputs are provided in the replication package. 4

Results and Discussion

4.1 Detection Effectiveness (RQ1 ) To answer RQ1 , we analyze detection from two perspectives: (i) aggregate effectiveness, comparing models and designs to assess whether task decomposition changes detection; and (ii) smell-level effectiveness, examining how these effects vary across the 19 smells and across models. Task decomposition modestly imTable 3. Overall detection effectiveness across designs. The proves aggregate detection effectiveness. best and second-best model values per configuration are Table 3 overviews the aggregate detection shown in bold and underlined, respectively. results. We observe that D2 achieves the highest aggregate detection effectiveness, Design Model 𝑛 F1↑ Pr↑ Re↑ Acc↑ reaching an average F1-score of 87.55% and Claude Haiku 4.5 570 88.81% 93.08% 84.91% 89.30% DeepSeek V4 Flash 570 88.27% 94.05% 83.16% 88.95% accuracy of 88.29%. This gain is mainly D1 Gemini 3.5 Flash 570 86.89% 98.23% 77.89% 88.25% recall-driven: recall increases from 79.12% GPT-5.4-nano 570 81.38% 96.17% 70.53% 83.86% Avg. – 86.34% 95.38% 79.12% 87.59% in D1 to 82.37% in D2, while precision reClaude Haiku 4.5 570 88.52% 93.73% 83.86% 89.12% mains high but decreases from 95.38% to DeepSeek V4 Flash 570 87.99% 92.97% 83.51% 88.60% D2 GPT-5.4-nano 570 87.15% 92.86% 82.11% 87.89% 93.44%. Thus, task decomposition shifts the Gemini 3.5 Flash 570 86.53% 94.21% 80.00% 87.54% Avg. – 87.55% 93.44% 82.37% 88.29% precision–recall balance toward recovering more true smell instances, at the cost of a small increase in false positives. However, this effect is model-dependent. GPT-5.4-nano benefits the most from D2, improving from 81.38% to 87.15% F1, driven by a recall increase from 70.53% to 82.11%. Claude Haiku 4.5 and DeepSeek V4 Flash are already strong under D1 and remain stable across designs, while Gemini 3.5 Flash shifts from high precision in D1 to higher recall in D2. The results also suggest that orchestration effects are not explained by model availability alone. DeepSeek V4 Flash, the open-weight model in our study, achieves the second-best F1-score in both designs, closely matching Claude Haiku 4.5, but changes little under decomposition. In contrast,

Uchôa et al.

GPT-5.4-nano benefits substantially from D2. Overall, D2 improves aggregate detection, but not uniformly across models; its main benefit is increasing recall, especially for models that under-detect in the joint detect-and-refactor design. Finding 1: Task decomposition modestly improves aggregate detection, raising average F1 from 86.34% to 87.55% and recall from 79.12% to 82.37%. Task decomposition helps selectively, depending on smell semantics and model behavior. Table 4 shows that D2 gains vary by smell and model. Overall, we observe high precision across configurations, while recall varies more substantially. Smells with explicit workflow evidence are easier to detect: MissingPermExternalActions and MissingTokenPermissions reach 100% F1 in nearly all (model, design) pairs, and MissingPathFilters, UnsafeArtifactUploadFork, and SuboptimalCacheUsage remain close to 100% in most cases. For these construct-explicit smells, decomposition provides little additional value because the single-agent configuration already performs well. Table 4. Smell-level detection effectiveness by model and GHA-SmellFix orchestration design. Best F1-score per smell/model is in bold; second-best distinct F1-score is underlined. The Avg. row reports macro-averages across smells for each model/design pair. Gray rows highlight selected difficult smells discussed in the text.

Localized

GPT-5.4-nano Gemini 3.5 Flash Claude Haiku 4.5 DeepSeek V4 Flash D1 D2 D1 D2 D1 D2 D1 D2 F1 Pr Re Acc F1 Pr Re Acc F1 Pr Re Acc F1 Pr Re Acc F1 Pr Re Acc F1 Pr Re Acc F1 Pr Re Acc F1 Pr Re Acc

MissingPermExternalActions UnpinnedActionVersion MissingTokenPermissions MissingJobTimeout UnsafeArtifactUploadFork MissingUploadCondition MutableRunnerVersion SuboptimalCacheUsage MultiCommandStep UnpinnedDependencies

97 100 93 97 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 80 100 67 83 80 100 67 83 80 100 67 83 80 100 67 83 80 100 67 83 80 100 67 83 80 100 67 83 80 100 67 83 97 100 93 97 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 82 74 93 80 75 60 100 67 100 100 100 100 79 65 100 73 75 60 100 67 75 60 100 67 77 62 100 70 75 60 100 67 85 100 73 87 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 70 100 53 77 93 100 87 93 89 100 80 90 93 100 87 93 93 100 87 93 93 100 87 93 89 100 80 90 93 100 87 93 89 100 80 90 93 100 87 93 93 100 87 93 93 100 87 93 89 100 80 90 89 100 80 90 93 100 87 93 89 100 80 90 70 100 53 77 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 70 100 53 77 80 100 67 83 57 100 40 70 64 100 47 73 70 100 53 77 64 100 47 73 70 100 53 77 64 100 47 73 100 100 100 100 100 100 100 100 80 100 67 83 89 100 80 90 97 100 93 97 97 100 93 97 97 100 93 97 97 100 93 97

Contextdependent

Type Smell

MissingForkGuardPR MissingConcurrencyPR MissingConcurrencyBranch MissingPathFilters MissingForkGuardSchedule UnsafeDeployOnFork MissingConcurrencyGlobal SingleOSExecution SingleLangVersionCI

42 100 27 63 42 100 27 63 42 100 27 63 24 100 13 57 64 100 47 73 50 100 33 67 42 100 27 63 42 100 27 63 100 100 100 100 97 100 93 97 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 87 87 87 87 80 80 80 80 94 88 100 93 91 83 100 90 94 88 100 93 91 83 100 90 91 83 100 90 91 83 100 90 97 100 93 97 100 100 100 100 97 100 93 97 100 100 100 100 97 100 93 97 93 100 87 93 97 100 93 97 97 100 93 97 42 100 27 63 80 100 67 83 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 75 100 60 80 89 100 80 90 89 100 80 90 89 100 80 90 89 100 80 90 89 100 80 90 85 100 73 87 85 100 73 87 86 92 80 87 76 79 73 77 79 85 73 80 58 78 47 67 88 79 100 87 90 88 93 90 90 88 93 90 88 79 100 87 64 100 47 73 67 89 53 73 24 100 13 57 61 88 47 70 48 83 33 63 67 89 53 73 67 89 53 73 61 88 47 70 75 100 60 80 86 92 80 87 70 100 53 77 80 100 67 83 86 92 80 87 85 100 73 87 75 100 60 80 85 100 73 87

Avg.

79 98 71 84 86 95 82 88 84 99 78 88 84 95 80 88 88 95 85 89 87 96 84 89 87 96 83 89 87 95 84 89

In contrast, several smells remain more challenging. Context-dependent smells such as MissingForkGuardPR, SingleOSExecution, and MissingConcurrencyGlobal show lower or more variable F1 scores, reflecting the difficulty of recovering true positives when evidence depends on fork context, concurrency semantics, or OS coverage. The localized but intent-sensitive MultiCommandStep is also difficult, ranging from 57% to 80% F1. A different pattern appears for UnpinnedActionVersion: all models and configurations are conservative, with 100% precision but only 67% recall. These results suggest that detection becomes harder when models must infer workflow intent, distinguish separable command phases, or consistently apply strict pinning criteria to action references. We find that task decomposition helps mostly when the model under-detects in D1. GPT-5.4-nano benefits the most from D2, improving from 79% to 86% in smell-level macro-F1, mainly through recall gains from 71% to 82%. These gains are visible for SuboptimalCacheUsage, MissingUploadCondition, MissingForkGuardSchedule, UnsafeArtifactUploadFork, and SingleLangVersionCI. However, this pattern does not generalize to all models: Gemini 3.5 Flash remains stable on average, while Claude Haiku 4.5 and DeepSeek V4 Flash are already strong under D1 and change little under D2. Although the open-weight DeepSeek V4 Flash remains competitive with closed models, it benefits little from decomposition, whereas GPT-5.4-nano benefits substantially. Thus, decomposition gains

GHA-SmellFix: Taking Action on Workflow Smells

are not explained by model availability alone, but by how each model handles task decomposition and smell-specific reasoning demands. Finding 2: Detection gains are selective: D2 mainly helps GPT-5.4-nano and a subset of target smells, while models with stronger D1 performance change little under decomposition. 4.2

Refactoring Outcomes (RQ2 )

To answer RQ2 , we evaluate whether the studied GHA-SmellFix orchestration designs generate structurally valid and scope-bounded candidate refactorings for workflow smells. End-to-end construct-aware outcomes by target smell. Following the refactoring assessment procedure described in Step 6, we classify positive-pair outputs as Clean Removal (CR), Removal with Unexpected Changes (UX), Removal with Potential Behavior-Impacting Change (BR), or Failed/No Patch (FN). For negative pairs, we distinguish Correct Abstention (CA) from Unnecessary Edit (UE). Table 5 reports these end-to-end outcomes by smell, together with the Smell Removal Rate (SRR = CR + UX + BR). Because FN conflates missed detections and failed patches after detection, we separate them in the conditional analysis below. Arrows indicate the desired direction. Table 5. Construct-aware refactoring outcomes by target smell. Values aggregate models and designs; percentages include counts in parentheses. Gray rows highlight the lowest-CR smells. Ground-truth-positive outputs

Ground-truth-negative outputs

SRR ↑

CR ↑

UX ↓

BR ↓

FN ↓

Assessable/Expected Pos. Outputs

Localized

Smell

MissingPermExternalActions UnpinnedActionVersion MissingTokenPermissions MissingJobTimeout UnsafeArtifactUploadFork MissingUploadCondition MutableRunnerVersion SuboptimalCacheUsage MultiCommandStep UnpinnedDependencies

99% (117) 66% (78) 99% (119) 93% (105) 95% (110) 77% (75) 82% (96) 91% (105) 47% (54) 89% (102)

98% (116) 40% (47) 99% (119) 91% (103) 95% (110) 77% (75) 81% (95) 86% (99) 46% (53) 70% (81)

1% (1) 0% (0) 0% (0) 0% (0) 0% (0) 0% (0) 0% (0) 1% (1) 0% (0) 1% (1)

0% (0) 26% (31) 0% (0) 2% (2) 0% (0) 0% (0) 1% (1) 4% (5) 1% (1) 17% (20)

1% (1) 34% (40) 1% (1) 7% (8) 5% (6) 23% (23) 18% (21) 9% (10) 53% (62) 11% (13)

Contextdependent

Type

MissingForkGuardPR MissingConcurrencyPR MissingConcurrencyBranch MissingPathFilters MissingForkGuardSchedule UnsafeDeployOnFork MissingConcurrencyGlobal SingleOSExecution SingleLangVersionCI

26% (30) 99% (118) 96% (115) 91% (107) 84% (91) 73% (86) 75% (85) 37% (41) 65% (73)

25% (29) 97% (115) 96% (115) 81% (95) 81% (88) 70% (83) 74% (84) 30% (33) 65% (73)

1% (1) 3% (3) 0% (0) 9% (10) 3% (3) 1% (1) 1% (1) 0% (0) 0% (0)

0% (0) 0% (0) 0% (0) 2% (2) 0% (0) 2% (2) 0% (0) 7% (8) 0% (0)

78% (1,707)

74% (1,613)

1% (22)

3% (72)

Overall

CA ↑

UE ↓

Assessable/Expected Negative Outputs

118/120 118/120 120/120 113/120 116/120 98/120 117/120 115/120 116/120 115/120

100% (120) 100% (120) 100% (120) 56% (65) 100% (120) 100% (120) 100% (120) 100% (120) 100% (120) 100% (120)

0% (0) 0% (0) 0% (0) 44% (52) 0% (0) 0% (0) 0% (0) 0% (0) 0% (0) 0% (0)

120/120 120/120 120/120 117/120 120/120 120/120 120/120 120/120 120/120 120/120

74% (87) 1% (1) 4% (5) 9% (10) 16% (17) 27% (32) 25% (28) 63% (69) 35% (40)

117/120 119/120 120/120 117/120 108/120 118/120 113/120 110/120 113/120

100% (120) 100% (120) 88% (100) 100% (120) 100% (120) 100% (120) 83% (100) 95% (114) 99% (119)

0% (0) 0% (0) 12% (14) 0% (0) 0% (0) 0% (0) 17% (20) 5% (6) 1% (1)

120/120 120/120 114/120 120/120 120/120 120/120 120/120 120/120 120/120

22% (474)

2,181/2,280

96% (2,178)

4% (93)

2,271/2,280

Table 5 shows that clean removal is often high for smells whose fixes target explicit workflow constructs, such as MissingPermExternalActions, MissingConcurrencyPR, and UnsafeArtifactUploadFork. However, two limitations remain. First, high SRR does not always imply clean refactoring: UnpinnedActionVersion and UnpinnedDependencies show SRR–CR gaps mainly due to potential behavior-impacting changes, while MissingPathFilters shows a smaller gap due mostly to unexpected changes. Second, difficult cases are not limited to one group: context-dependent smells such as MissingForkGuardPR and SingleOSExecution, as well as the localized but intent-sensitive MultiCommandStep, remain difficult, with low CR and high FN. Figure 4 illustrates these construct-aware distinctions, showing that patches may be clean, include collateral edits, depend on external facts, or modify workflows where the target smell is absent. Figure 4(a) shows a clean-removal case for MissingConcurrencyBranch. In the inspected outputs for this workflow, Gemini 3.5 Flash produces the same construct-level modification under D1 and D2, and DeepSeek V4 Flash shows the same behavior. Although placement or formatting may vary 1 across outputs, the modified construct is the same: a single concurrency block added at marker ○ in  pr-merged.yaml. The added group scopes concurrent runs by workflow and reference, while

Uchôa et al. (a) Clean removal 1 2 3 4 5 6 7 8 9 10 11 12 13

--- original +++ patch @@ -6,6 +6,9 @@ branches: - master - release-* 1 ¥ +concurrency: ○ + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true permissions: contents: read jobs:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

(b) Unexpected changes 1 --- original 2 +++ patch 3 @@ -10,14 +10,17 @@ 4 permissions: 5 contents: read 6 2 7 - id-token: "write" # AWS GitHub OIDC required: write ○ + id-token: "write" 8 packages: read 9 1 ¥ +concurrency: ○ + group: ${{ github.workflow }}-${{ github.event. 10 ↩→ pull_request.number || github.ref }} 11 + cancel-in-progress: true 12 jobs: flowzone: name: Flowzone uses: product-os/flowzone/.github/workflows/flowzone. ↩→ yml@master # prevent duplicate workflow executions for pull_request 3 ↩→ and pull_request_target ○ if: |

(c) Potential behavior-impacting change 1 --- original 2 +++ patch 3 @@ -21,7 +21,7 @@ 4 timeout-minutes: 5 5 steps: - name: Run microbenchmark 6 7 1 q uses: elastic/oblt-actions/buildkite/run@v1 ○ + uses: elastic/oblt-actions/buildkite/run@a1b2c3...a1b 8 9 2 ¥ ↩→ ○ 10 with: 11 pipeline: "apm-agent-microbenchmark" 12 token: ${{ secrets.BUILDKITE_TOKEN }} 13 14 15 16 17 18 19 20 21

(d) Unnecessary (neg.) --- original +++ patch @@ -1,6 +1,3 @@ -# This workflow will install Python dependencies 1 ↩→ ... ○ @@ -41,6 +38,7 @@ jobs: unittest_ci_gpu: 2 + timeout-minutes: 60 ○ strategy: @@ -91,7 +89,7 @@ python: version: "3.11" 3 + version: "3.10" ○ @@ -125,12 +123,15 @@ uses: pytorch/test-infra/.github/workflows/ 4 ↩→ linux_job_v2.yml@main ○ ... 5 + steps: ○ + - name: Checkout + uses: actions/checkout@v4 ...

Fig. 4. Representative refactoring outcomes: clean removal (CR), removal with unexpected changes (UX), removal with potential behavior-impacting change (BR), and unnecessary edit on a negative pair (UE). Markers n denote the target smell construct (q), the intended fix (¥), and other edits (○).

cancel-in-progress: true cancels older in-flight runs when a newer run in the same group is triggered. Because the patch is purely additive and touches no existing job or step, it is classified as a clean removal. This helps explain the high CR and 0% UX for MissingConcurrencyBranch in Table 5; the next examples show why smell removal alone is insufficient to assess patch quality. Figure 4(b) shows a removal-with-unexpected-changes case for MissingConcurrencyPR in the  flowzone.yml. All eight outputs for this workflow, from the four models under D1 and D2, converge on the same construct-level fix: a top-level concurrency block with cancel-in-progress: 1 ; only the group key varies. In D1, however, DeepSeek V4 Flash also true, added at marker ○ 2 , deletes two comments outside the fix: the note on why id-token: write is required, at marker ○ 3 . These deletions do not change and the note on the purpose of the job-level if guard, at marker ○ how the workflow runs, but they remove documentation of the existing configuration. Thus, the intended fix is present, but the patch is not clean. The other seven outputs apply the same fix without collateral edits, so the collateral edit comes from one output, not from the fix itself. This helps explain why MissingConcurrencyPR still has 3% UX despite its 97% CR in Table 5. Figure 4(c) shows a potential behavior-impacting change. In D1, Claude Haiku 4.5 refactors UnpinnedActionVersion by replacing the mutable @v1 tag in the action-reference construct at 1 with a 40-character commit-like reference at marker ○ 2 in  microbenchmark.yml. marker ○ Structurally, this is the expected fix: the action reference is no longer pinned to a mutable version tag. However, correctness depends on a fact outside the workflow file, namely whether the generated reference corresponds to a real commit in the action repository. In this example, the a1b2c3... pattern suggests a placeholder-like value, so the patch is well-formed but requires repository-level validation before use. Thus, the smell is removed syntactically, but the patch introduces a potential behavior-impacting change because it may break the step or pin an unintended revision. This explains why UnpinnedActionVersion has low CR and non-trivial BR in Table 5 (40% CR, 26% BR). Figure 4(d) shows an unnecessary-edit case on a negative pair, where MissingJobTimeout is absent and abstention is expected. In D1, GPT-5.4-nano nevertheless edits in the  unittest_ci.yml: it 2 , but also changes unrelated constructs, removing a header adds timeout-minutes at marker ○ 1 , changing the Python matrix from 3.11 to 3.10 at marker ○ 3 , and replacing a comment at marker ○ 4 with inline steps at marker ○ 5 . This is not a failed refactoring, reusable-workflow call at marker ○ but an unnecessary rewrite of a workflow that should have remained unchanged. Apart from comment deletion, edits are behavior-relevant rather than cosmetic, since they may alter job duration, runtime environment, and workflow reuse. This helps explain why MissingJobTimeout is the smell most prone to unnecessary edits, with a 44% UE rate on negative pairs.

GHA-SmellFix: Taking Action on Workflow Smells

Finding 3: Smell removal overestimates refactoring quality. Some smell-removing patches still introduce unexpected edits or potential behavior-impacting changes, so CI/CD refactoring evaluation should distinguish smell removal from construct-aware patch quality. End-to-end failures are mostly missed detections. Table 6 reports end-to-end (E2E) and conditional (Cond.) outcomes; Det. miss and Patch fail denote detection misses and post-detection patch failures. End-to-end, the pipelines remove the target smell in 78.3% of assessable outputs and produce clean removals in 74%. Among the 474 Failed/No Patch outcomes, 439 (92.6%) are detection misses and only 35 are post-detection patch failures. Conditioning on correct positive detecTable 6. End-to-end and conditional refactoring outcomes tion raises smell removal to 98.0% and clean on positive outputs. removal to 92.6%. This refines the interpretation of RQ2 : many end-to-end failures reflect Det. Patch SRR ↑ CR ↑ missed detections rather than unsuccessful Perspective Design Out. miss fail patch generation. It also clarifies the role D1 1,092 238 21 76.3% (833) 71.8% (784) D2 1,089 201 14 80.3% (874) 76.1% (829) of task decomposition. D2 improves end-to- E2E All 2,181 439 35 78.3% (1,707) 74.0% (1,613) end results in part because it misses fewer positives than D1 (201 vs. 238), producing D1 854 – 21 97.5% (833) 91.8% (784) D2 888 – 14 98.4% (874) 93.4% (829) gains of 4.0 percentage points in SRR and Cond. All 1,742 – 35 98% (1,707) 92.6% (1,613) 4.3 percentage points in CR. However, once the target smell is detected, patch quality is similar across configurations: D2 exceeds D1 by only 0.9 percentage points in conditional SRR and 1.6 percentage points in conditional CR. Thus, D2’s refactoring benefit is primarily detection-driven, not evidence of systematically better patch generation. Finding 4: End-to-end refactoring is detection-limited. Most Failed/No Patch outcomes are missed detections rather than patch-generation failures; when the target smell is correctly detected, clean removal rises from 74% to 92.6%. Model-level outcomes show where D2’s gains come from. Table 7 shows that D2’s aggregate refactoring gain is model-dependent. GPT-5.4-nano explains most of the reduction in detection misses, avoiding 33 of the 37 misses reduced by D2 (84 vs. 51); its end-to-end CR increases by 9.8 percentage points, while conditional CR remains similar (93.1% vs. 92.0%). Thus, for this model, decomposition mainly helps by detecting more positives. DeepSeek V4 Flash shows a different pattern: detection misses barely change (48 vs. 47), but conditional CR increases from 88.9% to 94.2%, indicating a gain after detection. Gemini 3.5 Flash and Claude Haiku 4.5 show smaller changes. Overall, D2’s refactoring benefit is model-dependent: it is mainly detection-driven for GPT-5.4-nano, while the largest post-detection gain occurs for DeepSeek V4 Flash. Section 4.3 tests whether these differences are statistically significant. Table 7. Detection misses and Clean Removal (CR) under D1 and D2, by model. Δ = D2 − D1 in percentage points, computed from unrounded values. Detection misses

End-to-end CR

Model

D1

D2

D1

D2

Δ

Conditional CR D1

GPT-5.4-nano Gemini 3.5 Flash Claude Haiku 4.5 DeepSeek V4 Flash

84 63 43 48

51 57 46 47

64.3% 70.1% 79.3% 73.3%

74.1% 71.8% 80.7% 77.9%

+9.8 +1.7 +1.3 +4.7

93.1% 91.3% 94.0% 88.9%

All models

238

201 71.8% 76.1% +4.3 91.8% 93.4% +1.6

D2

Δ

92.0% -1.1 90.1% -1.2 96.9% +2.9 94.2% +5.3

Uchôa et al.

Types of unnecessary edits. For negative pairs, abstention is expected; any workflow modification is therefore unnecessary. Following Step 6 of Section 3.1, we classify modified negatives by their most behavior-relevant edit type. Since no modified negative output was classified as cosmetic-only, Table 8 focuses on localized unnecessary edits and behavior-relevant edits. This distinction helps developers prioritize mitigation: improve abstention for plausible but unnecessary fixes, and add stronger safeguards for edits that may affect workflow execution. Table 8. Types of unnecessary edits in modified ground-truth-negative pairs by design and model. Edit type

Model

D1

D2

Overall

GPT-5.4-nano 57.14% (4/7) 40.00% (6/15) 45.45% (10/22) Gemini 3.5 Flash 0.00% (0/3) 12.50% (1/8) 9.09% (1/11) Localized but unnecessary edits Claude Haiku 4.5 50.00% (7/14) 60.00% (9/15) 55.17% (16/29) DeepSeek V4 Flash 50.00% (7/14) 47.06% (8/17) 48.39% (15/31)

Behavior-relevant edits

GPT-5.4-nano 42.86% (3/7) 60.00% (9/15) 54.55% (12/22) Gemini 3.5 Flash 100.00% (3/3) 87.50% (7/8) 90.91% (10/11) Claude Haiku 4.5 50.00% (7/14) 40.00% (6/15) 44.83% (13/29) DeepSeek V4 Flash 50.00% (7/14) 52.94% (9/17) 51.61% (16/31)

Unnecessary edits

All models

38

55

93

Across D1 and D2, 54.84% of modified negative outputs are behavior-relevant (51/93), while 45.16% are localized but unnecessary edits (42/93). This pattern is similar across configurations: behavior-relevant edits account for 52.63% of modified negative outputs in D1 (20/38) and 56.36% in D2 (31/55). Thus, when pipelines over-refactor, they often change executable or behavior-relevant workflow constructs rather than formatting, comments, or other cosmetic elements. Negativepair outcomes also reveal smell-specific over-refactoring. As shown in Table 5, unnecessary edits concentrate in MissingJobTimeout, MissingConcurrencyGlobal, and MissingConcurrencyBranch, with UE rates of 44.44%, 16.67%, and 12.28%, respectively. These cases, illustrated in Figure 4(d), suggest that agents sometimes apply plausible fix templates even when the target smell is absent. Finding 5: Negative pairs reveal model-specific abstention failures. All models produced unnecessary edits when the target smell was absent, although at different rates; overall, 54.84% of these edits touched behavior-relevant workflow constructs. 4.3

Effectiveness–Overhead Trade-off of Task Decomposition (RQ3 )

To answer RQ3 , we compare D1 and D2 reusing RQ1 detection outcomes and RQ2 end-to-end clean removal, adding overhead measures: latency, token usage, and estimated API cost. Absolute overhead varies by model. For each run, we aggregate provider-reported billable input and output tokens across all LLM calls and estimate API cost by multiplying them by the corresponding model-specific prices per one million tokens in Table 2. Token counts include reasoning or thinking tokens when reported as billable. Table 9 reports median latency, token usage, and per-run estimated cost multiplied by 1,000 for readability. The paired tests report median within-pair D2–D1 differences, so the summaries may differ slightly. We can observe in the table that D2 does not uniformly increase overhead. For GPT-5.4-nano and Gemini 3.5 Flash, D2 reduces median latency, token usage, and estimated cost. For Claude Haiku 4.5, D2 reduces token usage and estimated cost while keeping the same median latency. For DeepSeek V4 Flash, D2 reduces input tokens and keeps estimated cost nearly unchanged, but increases median latency and slightly increases output tokens. This pattern follows from D2’s detection gate, which invokes refactoring only for predicted positives. Effectiveness–overhead trade-offs are model-dependent. Detection correctness, latency, and estimated cost use all 570 pairs; end-to-end clean removal uses only ground-truth-positive

GHA-SmellFix: Taking Action on Workflow Smells

Table 9. Median operational overhead per (workflow, smell) run by design and model. Cost reports median per-run estimated cost multiplied by 1,000. Δ = D2–D1. Latency (s)

Input tokens

Median est. cost × 1K (USD)

Output tokens

Model

Δ

D1

14.5 11.1 -3.4 3,888 2,655 -1,233 17.9 9.7 -8.2 12,743 2,941 -9,802 15.1 15.1 +0.0 4,741 3,144 -1,597 16.5 21.9 +5.4 3,964 2,782 -1,182

2,210 2,660 1,887 1,579

D1 GPT-5.4-nano Gemini 3.5 Flash Claude Haiku 4.5 DeepSeek V4 Flash

D2

Δ

D1

D2

D2

Δ

1,546 -664 3.63 2.47 1,186 -1,474 42.04 14.71 1,465 -422 14.44 10.35 1,657 +78 0.65 0.63

-1.16 -27.33 -4.09 -0.02

D2

Δ

D1

pairs whose D1 and D2 outputs are both assessable (255–268 pairs per model). All effects are reported as D2–D1: for binary outcomes, 𝐸 𝑓 𝑓 = rate𝐷2 − rate𝐷1 ; for overhead outcomes, 𝐸 𝑓 𝑓 = median𝑖 (𝑋𝑖,𝐷2 − 𝑋𝑖,𝐷1 ). Thus, positive binary effects favor D2, while negative overhead effects indicate that D2 reduces latency or estimated cost. Table 10 reports the resulting comparisons. Table 10. Paired D1–D2 tests for effectiveness and overhead. Binary effects are D2–D1 rate differences; overhead effects are median paired D2–D1 differences. 𝑝-values are Holm-corrected within each metric. Detection correctness Comparison

Model

D1 vs. D2

GPT-5.4-nano Gemini 3.5 Flash Claude Haiku 4.5 DeepSeek V4 Flash

End-to-end clean removal

Latency

Est. cost

𝑝 Holm

Eff.

Sig.

𝑝 Holm

Eff.

Sig.

𝑝 Holm

Eff.

Sig.

𝑝 Holm

Eff.

Sig.

0.015 1.000 1.000 1.000

+0.04 -0.01 0.00 0.00

Yes No No No

0.026 1.000 1.000 0.242

+0.08 +0.01 +0.00 +0.05

Yes No No No

<0.001 <0.001 <0.001 <0.001

-3.5 -3.5 -0.8 +3.7

Yes Yes Yes Yes

<0.001 <0.001

-1.00 -12.70 -3.58 -0.04

Yes Yes No No

0.933 0.933

Table 10 shows that D2 significantly improves effectiveness only for GPT-5.4-nano, increasing detection correctness by 4 percentage points and end-to-end clean removal by 8 percentage points. However, the clean-removal gain is fragile: it is no longer significant when UnsafeDeployOnFork is excluded (Section 4.4). The detection component of this gain is also smell-dependent: GPT-5.4-nano benefits from D2 mainly for SuboptimalCacheUsage, MissingUploadCondition, MissingForkGuardSchedule, UnsafeArtifactUploadFork, and SingleLangVersionCI, whereas MissingJobTimeout and concurrency-related smells show smaller gains or favor D1. For Gemini 3.5 Flash, Claude Haiku 4.5, and DeepSeek V4 Flash, D1 and D2 do not differ significantly in effectiveness. In contrast, overhead changes are more widespread, with latency differing significantly for all models. D2 is faster for GPT-5.4-nano, Gemini 3.5 Flash, and Claude Haiku 4.5, but slower for DeepSeek V4 Flash. The estimated cost decreases significantly only for GPT-5.4-nano and Gemini 3.5 Flash. Overall, task decomposition is an effectiveness–overhead trade-off rather than a default improvement. Finding 6: Task decomposition is not uniformly better: D2 improves effectiveness significantly only for GPT-5.4-nano, while overhead changes vary by model. 4.4 Robustness and Diagnostic Analyses Four diagnostic analyses were conducted to check if our conclusions depend on key design choices. GHA-based candidate retrieval. The GHA analyzer should not be treated as a labeling oracle. Of the 1,416 GHA-flagged candidates inspected manually, annotators confirmed 902 and rejected 514; only 9/19 smells reached ≥ 90% confirmation, while 5/19 were below 70%. GHA also missed positives: 366 of 1,310 GHA-unflagged candidates were manually labeled positive (27.9%). The lowest confirmation rate, 2.8% for UnsafeDeployOnFork, reflects a rule that flags any job without a repository guard, whether or not the job deploys. Some misses stem from narrower rules; for example, GHA does not flag branch references such as @main for UnpinnedActionVersion. Thus, GHA is useful for reproducible retrieval and stratified sampling, but labels and reported effectiveness must

Uchôa et al.

rely on the manual ground truth. Moreover, the final natural ground truth keeps only candidates whose manual label agrees with the sampling stratum; GHA matches every natural ground-truth label by construction, so we use it neither as a detection baseline for RQ1 nor, since it does not generate patches, as a refactoring baseline for RQ2 . Balanced evaluation. Precision in our balanced 570-pair evaluation should not be interpreted as field precision under natural smell prevalence. We therefore recomputed precision using candidatepool prevalence estimated from the GHA-guided sampling pool and manual confirmation/miss rates among the 14,385 workflows eligible for Step 3. Across the 19 smells, the estimated positive-pair prevalence was 29.92% on average, ranging from 1.97% to 93.06%. Keeping predictions fixed, we recomputed precision as 𝜋𝑅/(𝜋𝑅 + (1 − 𝜋)FPR), where 𝜋 is the estimated prevalence, 𝑅 is recall, and FPR is the false-positive rate. Averaged over the four models, prevalence-adjusted precision decreases from 95.38% to 89.92% for D1 and from 93.44% to 85.89% for D2, while recall remains 79.12% and 82.37%, respectively. The main interpretation of RQ1 remains: D1 is more precise, while D2 recovers more smells. However, it nearly removes D2’s aggregate F1 advantage, reducing the D2–D1 gap from 1.21 to 0.14 percentage points because D2’s additional false positives in the balanced set weigh more heavily under lower positive-pair prevalence. This diagnostic reframes the precision under a less balanced prevalence assumption; it does not estimate the field precision because the candidate pool was GHA-guided and ranked rather than randomly sampled. Semi-synthetic UnsafeDeployOnFork. Because natural positives for UnsafeDeployOnFork were scarce, we generated 49 semi-synthetic candidates from real workflows labeled negative for this smell; 45 were accepted as positives after manual validation. As a sanity check, GHA flagged 48/49 mutated workflows, including all accepted positives, while every base workflow was GHA-unflagged by selection. However, GHA also flagged 3/4 rejected candidates, reinforcing that final labels must come from manual validation. We repeated the main analyses after excluding this smell, i.e., its 30 pairs in the 570-pair set: 15 positives (10 semi-synthetic and 5 natural) and 15 natural negatives. Reusing the same outputs and metadata, the exclusion changes detection F1 by ≤ 0.33 percentage points and Clean Removal by ≤ 0.73 percentage points per model and design. We do not report changes in Unnecessary Edit, latency, or estimated cost because these outcomes do not depend on the semi-synthetic positive labels. The exclusion changes one RQ3 significance conclusion: GPT5.4-nano’s end-to-end clean-removal gain remains similar in size (8.2 vs. 7.5 percentage points), but is no longer significant (𝑝 Holm : 0.026 → 0.070), because three of its 38 pairs with clean removal only under D2 are UnsafeDeployOnFork pairs. Detection correctness, latency, and estimated-cost conclusions do not change. Thus, the main findings are not driven by this semi-synthetic smell, but the significance of GPT-5.4-nano’s clean-removal gain rests on a narrow margin. Workflow-level feasibility diagnostic. Because GHA-SmellFix is target-conditioned, we ran a small diagnostic on eight real workflows to assess workflow-level smell-set recovery through exhaustive target-conditioned screening. Two annotators labeled all 19 smells per workflow, agreeing on 138/152 pairs (𝜅 = 0.759); the workflows contain 40 present pairs across 16 smells. Using Claude Haiku 4.5, we ran D1 and D2 once per target smell, yielding 304 runs, and aggregated predictedpresent smells into workflow-level sets. Both designs recovered all 40 annotated pairs (recall 100%) and produced nine predicted-present pairs not annotated as present (precision 81.63%); exact workflow-level matches occurred for 3/8 workflows under D1 and 2/8 under D2. We also inspected patches independently per target smell, without merging them into coordinated workflow-level refactorings. Both designs generated patches for all 40 present pairs, with 32/40 D1 and 33/40 D2 patches being valid smell-specific refactorings or requiring only external validation. On absent pairs, each design generated nine unnecessary patches, all affecting behavior-relevant constructs. Overall, this diagnostic supports the feasibility of workflow-level smell-set recovery and smell-specific patch generation, while reinforcing the need for reliable abstention and construct-aware validation.

GHA-SmellFix: Taking Action on Workflow Smells

5

Study Implications

For researchers: evaluate CI/CD agents as maintenance systems, not only smell detectors. Existing rule-based work on GitHub Actions quality and security problems has mostly focused on detecting known configuration patterns [4, 14, 24]. Because we build on a developer-informed smell catalog, our results shift the focus from smell relevance to whether LLM-based pipelines can detect, refactor, and abstain safely when targeting these smells. A patch may remove the target smell while still introducing unexpected edits, potential behavior-impacting changes, or unnecessary edits, so CI/CD agents should be evaluated beyond detection and smell-removal rates. For tool builders: place LLM-generated workflow patches behind validation gates. GHASmellFix can draft useful candidate patches, especially when targeting explicit constructs, but should not be treated as an autonomous refactoring tool. GitHub Actions workflows depend on behavior-relevant constructs such as permissions, triggers, runners, secrets, artifacts, deployments, reusable workflows, and repository-specific policies [17, 25]; small YAML edits can have executionlevel consequences. Tool builders should combine LLM-generated patches with YAML and GitHub Actions structural checks, policy checks, construct-aware diff analysis, abstention checks, and repository-level validation for external facts like action SHAs or dependency versions. A practical deployment path is a hybrid static-analysis–agentic-LLM pipeline where rule-based scanners, like the GHA analyzer, provide first-level screening, while agentic LLM components generate candidate patches passing through construct-aware validation gates. Orchestration should be model- and smell-aware, since decomposition changed effectiveness and overhead in model-dependent ways. For developers and maintainers: treat generated patches as review candidates, not readyto-merge fixes. The studied designs produce useful candidate patches, but human review remains necessary. Maintainers should inspect whether a patch changes only expected workflow constructs, preserves intended triggers, permissions, runners, dependencies, and reusable-workflow calls, and uses contextually valid values. In practice, LLM agents are better positioned as review-support components for CI/CD configuration maintenance than as fully automatic refactoring systems.

6

Threats to Validity

Construct validity. We derived our smell definitions from an existing GitHub Actions smell catalog [24] grounded in workflow evolution evidence and examined through a contribution study with maintainers. We operationalized the selected smells with explicit detection conditions, exclusion criteria, uncertainty rules, and counting units, and cross-checked them against official GitHub Actions [17]. Still, some smells may depend on unobservable repository intent from workflow files, especially concerning fork guards, deployment behavior, concurrency semantics, reusable workflows, or workflow intent. Our refactoring assessment is construct-aware rather than runtimesemantic: using automatic checks, diffs, and manual inspection, we evaluate smell removal, YAML parsability, GitHub Actions structural validity, expected construct scope, and evident behaviorrelevant changes. We do not execute workflows or verify runtime-dependent facts like commit hashes, package versions, credentials, external services, branch rules, or repository-specific policies. Thus, our results characterize structurally plausible candidate patches, not production-ready repairs. Internal validity. Our ground truth was manually annotated through independent inspection, consensus discussion, and agreement analysis with Cohen’s Kappa [26, 29]. Some annotators are co-authors, which may introduce confirmation bias. We mitigated this risk through explicit guidelines, pairwise annotation, consensus, fixed ground-truth labels before any agent run, and blinded inspection of refactoring outputs with respect to model and design identities. In addition, none of the annotators implemented the agentic pipelines or executed the experiments.

Uchôa et al.

We used GHA analyzer [24] only for stratified sampling; final labels came from manual annotation. This reduces dependence on the initial heuristic tool, but may underrepresent positives missed by GHA analyzer or workflows with multiple co-occurring smells. The balanced ground truth supports paired comparisons between smells, models, and designs, but it does not estimate the prevalence of smells in the real-world. Thus, precision, recall, abstention, and unnecessary-edit rates should be interpreted under the sampled distribution, not as prevalence-adjusted field estimates. For UnsafeDeployOnFork, natural positives were scarce, so we used LLM-assisted semi-synthetic positives derived from real workflows by mutating only the target-smell construct. These cases were manually validated, and no synthetic negative pair was generated. Since semi-synthetic cases may differ from natural ones, we avoid conclusions that rely on this smell alone. Section 4.4 shows that excluding this smell changes detection F1 and clean removal by less than one percentage point, but makes GPT-5.4-nano’s clean-removal gain no longer statistically significant. Conclusion validity. We mitigate threats by using paired comparisons over the same (workflow, smell) inputs, McNemar’s exact test for binary outcomes [30], Wilcoxon signed-rank tests for paired overhead measures [43], and Holm correction for multiple comparisons [19]. Still, the balanced shared set has limited per-smell sample size, and Gemini 3.5 Flash and DeepSeek V4 Flash were executed once, which may limit estimate precision. LLM executions may also vary due to providerside nondeterminism, infrastructure changes, or model updates; we repeated GPT-5.4-nano and Claude Haiku 4.5 three times and limit stability conclusions to those models. Non-significant results should not be interpreted as evidence of equivalence [1]. We report effect directions, significance results, generated outputs, and execution metadata in the replication package. External validity. We focus on GitHub Actions workflows sampled from a workflow-history dataset [8] and 19 configuration smells [24]. The findings may not generalize to other CI/CD platforms, smell catalogs, proprietary workflows, larger workflows, repositories with many cooccurring smells, or teams with different CI/CD conventions. We evaluate target-conditioned detection and refactoring over explicit (workflow, smell) pairs, not open-ended discovery, target-free refactoring, or coordinated multi-smell repair. Although Section 4.4 reports a small workflow-level diagnostic, our main evaluation remains target-conditioned; larger-scale workflow-level screening and coordinated multi-smell refactoring require broader annotations and strategies for resolving interacting patches. Although GHA-SmellFix is conceptually extensible to other CI/CD platforms, its smell definitions, prompts, construct mappings, and validation rules would need to be adapted to each platform’s configuration language and execution semantics. 7

Conclusion and Future Work

This paper presented GHA-SmellFix, a target-conditioned agentic pipeline for detecting and refactoring GitHub Actions configuration smells. We evaluated a joint detect-and-refactor design and a decomposed detect-then-refactor design with four LLMs on 570 shared (workflow, smell) pairs sampled from a 1,710-pair manually curated ground truth covering 19 smells. Our results show that task decomposition helps selectively, but is not a default improvement. D2 modestly improves aggregate detection, increasing average F1 from 86.34% to 87.55% and recall from 79.12% to 82.37%, with the strongest gains for models that under-detect in D1. However, once the target smell is correctly detected, conditional patch quality is similar across D1 and D2. Thus, D2’s main benefit is detection-driven rather than evidence of systematically better patch generation. Refactoring results are promising but require safeguards. Across assessable positive outputs, 78.3% remove the target smell and 74% are clean removals; given correct detection, smell removal increases to 98% and clean removal to 92.6%. At the same time, smell removal alone is insufficient to characterize patch quality: some patches introduce unexpected edits or potential behavior-impacting changes, and negative pairs reveal 93 unnecessary edits, 54.84% of which affect behavior-relevant

GHA-SmellFix: Taking Action on Workflow Smells

workflow constructs. Overall, GHA-SmellFix supports review-oriented CI/CD maintenance, but requires validation and human review. Future work should incorporate richer repository context, external validators, sandboxed CI execution, broader CI/CD platforms and smell catalogs, and independent evaluation of generated patches with maintainers or external CI/CD experts. Data Availability All datasets, prompts, outputs, and analysis scripts is available at: https://github.com/reset-ufc/GHASmellFix-preprint. References [1] Douglas G Altman and J Martin Bland. 1995. Absence of evidence is not evidence of absence. British Medical Journal 311, 7003 (1995), 485–486. [2] Aref Talebzadeh Bardsiri, Alexandre Decan, and Tom Mens. 2026. On the GitHub Actions Language: Usage, Evolution, and Workflow Reliability. In 42nd 2026 IEEE International Conference on Software Maintenance and Evolution (ICSME). https://arxiv.org/abs/2605.26825 [3] Moritz Beller, Radjino Bholanath, Shane McIntosh, and Andy Zaidman. 2016. Analyzing the state of static analysis: A large-scale evaluation in open source software. In 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER), Vol. 1. IEEE, 470–481. [4] Giacomo Benedetti, Luca Verderame, and Alessio Merlo. 2022. Automatic security assessment of github actions workflows. In Proceedings of the 2022 ACM Workshop on Software Supply Chain Offensive Research and Ecosystem Defenses. 37–45. [5] Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. 2025. Repairagent: An autonomous, llm-based agent for program repair. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 2188–2200. [6] Islem Bouzenia and Michael Pradel. 2024. Resource usage and optimization opportunities in workflows of github actions. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering. 1–12. [7] Victor R Basili-Gianluigi Caldiera and H Dieter Rombach. 1994. Goal question metric paradigm. Encyclopedia of software engineering 1, 528-532 (1994), 6. [8] Guillaume Cardoen, Tom Mens, and Alexandre Decan. 2024. A dataset of GitHub Actions workflow histories. In Proceedings of the 21st International Conference on Mining Software Repositories (Lisbon, Portugal) (MSR ’24). Association for Computing Machinery, New York, NY, USA, 677–681. doi:10.1145/3643991.3644867 [9] Alexandre Decan, Tom Mens, Pooya Rostami Mazrae, and Mehdi Golzadeh. 2022. On the use of github actions in software development repositories. In 2022 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 235–245. [10] Hana Derouiche, Zaki Brahmi, and Haithem Mazeni. 2025. Agentic ai frameworks: Architectures, protocols, and design challenges. arXiv preprint arXiv:2508.10146 (2025). [11] Wade C Driscoll. 1996. Robustness of the ANOVA and Tukey-Kramer statistical tests. Computers & Industrial Engineering 31, 1-2 (1996), 265–268. [12] Madjda Fares, Yogya Gamage, and Benoit Baudry. 2026. Unpacking Security Scanners for GitHub Actions Workflows. arXiv preprint arXiv:2601.14455 (2026). [13] Tom Fawcett. 2006. An introduction to ROC analysis. Pattern Recogn. Lett. 27, 8 (2006), 861–874. [14] Matheus B Freitas and Lincoln S Rocha. 2024. GASH–The GitHub Actions Smell Hunter. In Workshop de Visualização, Evolução e Manutenção de Software (VEM). SBC, 103–112. [15] Keheliya Gallaba and Shane McIntosh. 2018. Use and misuse of continuous integration features: An empirical study of projects that (mis) use Travis CI. IEEE Transactions on Software Engineering 46, 1 (2018), 33–50. [16] Rohit Gheyi, Rian Melo, Jonhnanthan Oliveira, Márcio Ribeiro, and Baldoino Fonseca. 2027. Foundation models as oracles for refactoring correctness detection. Empirical Software Engineering 32, 1 (2027), 32. [17] GitHub. 2026. Workflow syntax for GitHub Actions. https://docs.github.com/actions/using-workflows/workflowsyntax-for-github-actions. Accessed: 2026-04-17. [18] Michael Hilton, Nicholas Nelson, Timothy Tunnell, Darko Marinov, and Danny Dig. 2017. Trade-offs in continuous integration: assurance, security, and flexibility. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. 197–207. [19] Sture Holm. 1979. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics (1979), 65–70. [20] Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large language models for software engineering: A systematic literature review. ACM Transactions on

Uchôa et al.

Software Engineering and Methodology 33, 8 (2024), 1–79. [21] Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2026. A survey on large language models for code generation. ACM Transactions on Software Engineering and Methodology 35, 2 (2026), 1–72. [22] Brittany Johnson, Yoonki Song, Emerson Murphy-Hill, and Robert Bowdidge. 2013. Why don’t software developers use static analysis tools to find bugs?. In 2013 35th International Conference on Software Engineering (ICSE). IEEE, 672–681. [23] Eirini Kalliamvakou, Georgios Gousios, Kelly Blincoe, Leif Singer, Daniel M German, and Daniela Damian. 2014. The promises and perils of mining github. In Proceedings of the 11th working conference on mining software repositories. 92–101. [24] Ali Khatami, Cćdric Willekens, and Andy Zaidman. 2024. Catching smells in the act: A github actions workflow investigation. In 2024 IEEE International Conference on Source Code Analysis and Manipulation (SCAM). IEEE, 47–58. [25] Igibek Koishybayev, Aleksandr Nahapetyan, Raima Zachariah, Siddharth Muralee, Bradley Reaves, Alexandros Kapravelos, and Aravind Machiry. 2022. Characterizing the security of github {CI} workflows. In 31st USENIX Security Symposium (USENIX Security 22). 2747–2763. [26] J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics (1977), 159–174. [27] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. 2024. Agentbench: Evaluating llms as agents. In International Conference on Learning Representations, Vol. 2024. 52989–53046. [28] Pooya Rostami Mazrae, Alexandre Decan, Tom Mens, and Mairieli Wessel. 2026. An empirical study of the evolution of GitHub Actions workflows. Journal of Systems and Software (2026), 112824. [29] Mary L McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica 22, 3 (2012), 276–282. [30] Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12, 2 (1947), 153–157. [31] Rian Melo, Pedro Simões, Rohit Gheyi, Marcelo d’Amorim, Márcio Ribeiro, Gustavo Soares, Eduardo Almeida, and Elvys Soares. 2026. Agentic LMs: Hunting Down Test Smells. IEEE Software 43, 1 (2026), 32–40. [32] Nuthan Munaiah, Steven Kroh, Craig Cabrey, and Meiyappan Nagappan. 2017. Curating github for engineered software projects. Empirical Software Engineering 22, 6 (2017), 3219–3253. [33] Rares Patcas and Simona Motogna. 2026. An evaluation study of large language models for addressing code quality issues. Empirical Software Engineering 31, 5 (2026), 118. [34] Peter Pickerill, Heiko Joshua Jungen, Mirosław Ochodek, Michał Maćkowiak, and Miroslaw Staron. 2020. PHANTOM: Curating GitHub for engineered software projects using time-series clustering. Empirical Software Engineering 25, 4 (2020), 2897–2929. [35] Sander Schulhoff, Michael Ilie, Nishant Balepur, Konstantine Kahadze, Amanda Liu, Chenglei Si, Yinheng Li, Aayush Gupta, HyoJung Han, Sevien Schulhoff, et al. 2024. The prompt report: A systematic survey of prompt engineering techniques. arXiv preprint arXiv:2406.06608 (2024). [36] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. Advances in neural information processing systems 36 (2023), 8634–8652. [37] Hui Sun, Yinan Wu, Wesley KG Assunção, and Kathryn T Stolee. 2026. Test Code Review in the Era of GitHub Actions: A Replication Study. arXiv preprint arXiv:2603.15935 (2026). [38] Pablo Valenzuela-Toledo and Alexandre Bergel. 2022. Evolution of github action workflows. In 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 123–127. [39] Carmine Vassallo, Sebastian Proksch, Harald C Gall, and Massimiliano Di Penta. 2019. Automated reporting of antipatterns and decay in continuous integration. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 105–115. [40] Carmine Vassallo, Sebastian Proksch, Anna Jancso, Harald C Gall, and Massimiliano Di Penta. 2020. Configuration smells in continuous delivery pipelines: a linter and a six-month study on GitLab. In Proceedings of the 28th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering. 327–337. [41] Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2024. A survey on large language model based autonomous agents. Frontiers of Computer Science 18, 6 (2024), 186345. [42] Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. 2025. Openhands: An open platform for ai software developers as generalist agents. In International Conference on Learning Representations, Vol. 2025. 65882–65919. [43] Frank Wilcoxon. 1945. Individual comparisons by ranking methods. Biometrics bulletin 1, 6 (1945), 80–83. [44] Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2025. Demystifying llm-based software engineering agents. Proceedings of the ACM on Software Engineering 2, FSE (2025), 801–824.

GHA-SmellFix: Taking Action on Workflow Smells

[45] John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems 37 (2024), 50528–50652. [46] He Ye, Matias Martinez, and Martin Monperrus. 2021. Automated patch assessment for program repair at scale. Empirical Software Engineering 26, 2 (2021), 20. [47] Fiorella Zampetti, Carmine Vassallo, Sebastiano Panichella, Gerardo Canfora, Harald Gall, and Massimiliano Di Penta. 2020. An empirical characterization of bad practices in continuous integration. Empirical Software Engineering 25, 2 (2020), 1095–1135.

Related documents

• GHA-SmellFix: Taking Action with Agentic LLMs for Detecting and Refactoring GitHub Actions Workflow Smells#1648993
• Continuous Assurance of Agentic Security Auditors for Software Delivery Decision Gates#1129860
• Continuous Assurance of Agentic Security Auditors for Software Delivery Decision Gates#1108613
• Unpacking Security Scanners for GitHub Actions Workflows#1402170
• Comment and Control: Hijacking Agentic Workflows via Context-Grounded Evolution#178826