GH-AWTrace: A Large-Scale Dataset of GitHub Agentic Workflow Execution Logs

arXiv:2611.06866v1 [cs.SE] 4 Nov 2026

GH-AWTrace: A Large-Scale Dataset of GitHub Agentic Workflow Execution Logs Jasem Khelifi

Hao Li

Ali Ouni

ÉTS Montreal, University of Quebec Montreal, Canada jasem.khelifi.1@ens.etsmtl.ca

Queen’s University Kingston, Canada hao.li@queensu.ca

ÉTS Montreal, University of Quebec Montreal, Canada Ali.Ouni@etsmtl.ca

Mohammed Sayagh

Mohamed Aymen Saied

ÉTS Montreal, University of Quebec Montreal, Canada Mohammed.Sayagh@etsmtl.ca

Laval university Quebec city, Canada mohamed-aymen.saied@ift.ulaval.ca

Abstract—Agentic workflows enable AI agents to perform recurring repository tasks on schedules or in response to repository events. Existing studies examine workflow definitions and outputs, but provide limited evidence of what happens during execution. We present GH-AWTrace, a dataset linking 25,014 verified GitHub Agentic Workflows across 5,051 repositories to 4,112,526 run records and 562,877 nonempty execution log archives. The dataset preserves execution evidence subject to GitHub’s log-retention limits and links it to repository metadata and generated workflow configurations. Illustrative cases show how the logs expose tool denials, provider failures, changes in an agent’s approach, and delegation among agents. These observations enable research on execution failures, tool use, and the relationship between reported run outcomes and visible task progress. Coverage varies across workflows, and the logs provide partial observations rather than complete agent trajectories. Video: https://youtu.be/KIlfmZhHxtA.

I. I NTRODUCTION Agentic AI can inspect repositories, edit code, run tests, and resolve software issues through development tools [1]. In common coding-agent use, a developer starts each task by assigning an issue or writing a prompt, then reviews the proposed change [2]. Agentic workflows let developers specify recurring tasks once and run them on schedules or repository events, without a person starting each execution [3]. GitHub Agentic Workflows (gh-aw) implements this model in Markdown: a file combines the agent’s task with its trigger, permissions, and allowed outputs [3]. gh-aw compiles the file into a GitHub Actions YAML workflow that starts a coding agent and can produce reviewable issues, comments, or pull requests [3]. In Azure/azure-dev, a weekly workflow reads feedback on merged pull requests and turns recurring reviewer requests into proposed updates to the repository’s Copilot instructions [4]. In August 2026, it examined 49 pull requests and proposed rules to reject ignored command-line flags and redact credentials from URLs before printing them in logs [5]. Repository settings prevented the workflow from opening a pull request, so it saved a branch and an issue with the patch; a maintainer refined and merged the changes through pull request #9731 [5], [6]. Khelifi et al. [7] studied how developers specify and maintain gh-aw workflows. Valenzuela-Toledo et al. [8] released histories that link Markdown definitions to generated YAML.

Sun et al. [9] measured whether comments from AI codereview Actions led to code changes. Wang et al. [10] used static analysis to find prompt-injection paths in agentic GitHub Actions workflows. Isbarov et al. [11] tested prompt-injection attacks in live AI-powered CI/CD workflows. These studies provide complementary perspectives on workflow specifications, outputs, and security. However, workflow definitions describe intended behavior, while run-level outcomes summarize execution without explaining the intervening actions. Execution logs provide additional evidence of tool interactions, access restrictions, infrastructure failures, and recovery attempts. Preserving and linking this evidence is particularly important because GitHub’s retention limits make it unavailable over time. To support research on agentic workflow execution, we release GH-AWTrace, linking 25,014 verified gh-aw workflows from 5,051 repositories to 4,112,526 run records and 562,877 nonempty execution log archives. Its contributions are threefold: (1) an archive of execution logs that may otherwise become unavailable under GitHub’s retention policies; (2) linked workflow, repository, run, and log metadata that support traceable analyses; and (3) illustrative use cases showing how the data can support studies of failures, tool interactions, and within-run agent coordination. II. DATASET C OLLECTION AND C URATION We construct GH-AWTrace in three stages: 1) identifying generated gh-aw workflows, 2) collecting their run metadata, and 3) archiving available execution logs. Figure 1 summarizes coverage across these stages. A. GitHub Agentic Workflow Identification First, we search for gh-aw generated workflows through the GitHub REST API code search [12] using the following search query on both .yml and .yaml files separately: "This file was automatically generated by gh-aw" path:.github/workflows extension:yml fork:true size:0..393216

GitHub returns at most 1,000 code-search results per query [12]. To reach files beyond this limit, we search nonoverlapping file-size ranges in bytes. When a range reported

the maximum allowed 1,000 matches, we split it in half and searched both smaller ranges. We retrieve every page from the final ranges and remove duplicate files using the repository ID and exact file path. Upon further investigation, we notice that the target phrase could appear as a quoted text rather than a generator comment. We fetched each file at the blob revision returned by search and accepted it only when the generator phrase appeared in an opening YAML comment before any configuration key. Of 25,227 candidate files in 5,059 repositories, 63 lacked the required header, and 150 indexed blobs could not be retrieved. This process yielded 25,014 verified gh-aw workflow files across 5,051 repositories.

25,014 Verified workflows

Phase A: Workflows

11,396 With runs

13,618 No recorded runs

Phase B: Runs 4,112,526 Run records

2,453,263 Within 90 days

1,659,263 Outside 90 days

Phase C: Logs

B. GitHub Agentic Workflow Runs Collection We use the GitHub REST API to collect runs for each ghaw workflow, querying by workflow ID and saving all data fields returned by the run listing API [13]. We link runs to the YAML revision indexed during discovery by repository ID and exact path. This revision may differ from the one used by a run if the workflow changed. The run metadata retains its head_sha. This phase yields 4,112,526 run records dated August 14, 2025, through September 21, 2026, since GitHub retention policy deletes runs after 400 days [14]. C. Log Collection Finally, GitHub automatically deletes public workflow execution logs after at most 90 days [14]. We therefore select 2,453,263 run records created from June 23 through September 21, 2026, the 90 days before our run cutoff. For these runs, we download and save 1,633,215 execution log archives (56.1 GB), one ZIP per downloaded attempt [13]. A run has one GitHub run ID. Each rerun keeps that ID and receives a new attempt number, so we identify an archive by run ID and attempt number. We check each ZIP’s integrity and record its SHA-256 hash and file list. Of the selected runs, 821,672 have no downloaded archive, which may reflect retention, deletion, access restrictions, or an unfinished run. Among the downloaded ZIPs, 1,070,338 contain no files and 562,877 contain log files. The 562,877 archives with files belong to 561,367 runs, and 18,337 archives come from attempts after the first. In an early audit of 2,900 empty ZIPs with saved job metadata, 2,699 show only skipped jobs in the available pages and 195 list no jobs. We cannot determine the cause of every empty ZIP. Figure 1 summarizes workflow, run-record, and log-archive coverage. III. DATASET C HARACTERISTICS A. Schema and Stored Files Figure 2 shows how repositories, gh-aw workflows, runs, and log archives are linked. A repository can contain one or more gh-aw workflows, linked through repo_id. When a workflow executes, it has one or more runs, linked through repo_id and its exact YAML path. A run may have no downloaded log archive or an archive linked through run_id. If a run is rerun, each downloaded attempt archive is distinguished by its attempt number.

1,631,591 With log archive 1,633,215 ZIPs

562,877 With log files

821,672 Without log archive

1,070,338 Empty archive

Fig. 1. Workflow, run, and log coverage across the three collection phases. The final split counts attempt ZIPs, including reruns.

B. Available Content The JSON fields retain additional GitHub metadata, including each run’s event, actors, branch, commit SHA, and timestamps. CSV indexes support direct joins, while the raw YAML, API responses, and ZIPs preserve the underlying content. A download row records whether an archive was saved or unavailable; a saved ZIP can still contain zero files. Its receipt lists each member and checksum. Runs skipped because of age have no downloaded archive. C. Execution Distributions Figure 3 shows the distribution for the 6,040 workflows with logs. The median is 16 logs per workflow (interquartile range: 6–60). Each downloaded execution attempt contributes one log archive. Table I groups nonempty log archives by their run’s recorded event. The workflow column counts verified workflows with at least one log under that event. A workflow can appear under multiple events, so workflow counts are not additive. Percentages use the 6,040 workflows with logs and 562,877 nonempty archives in the fixed 90-day window. IV. A NALYSIS & P OTENTIAL U SE C ASES The excerpts below reproduce text from each cited run’s archived attempt-1 agent job log. We omit repeated leading UTC timestamps. Colors are editorial, and ellipses mark omitted text or intervening lines. A. Agentic Workflow Failure Analysis Of 562,877 nonempty log archives, 178,151 (31.7%) are linked to run records with a failure conclusion. This does

Repository

Attempt log

Run

Workflow 1:N

integer

id full_name stars fetched_at metadata

text integer

1:N

repo_id path

time

workflow_id name

JSON

sha verified state metadata discovered_at

repository.json

integer text integer

1:N

id repo_id path

text

created_at

text

status

integer integer text time text

run_id attempt

integer

kind state path

text

boolean text

conclusion attempts

text integer

bytes

JSON

verification metadata

text

files error

time

source .yml / .yaml

JSON

run-page JSON / run.json

integer text text integer text integer

sha256

text

logs.zip / logs.receipt.json

Fig. 2. Dataset schema. Colored dots mark key and join fields. Connectors show one-to-many relationships, and shaded footers name preserved files.

Example 2: FAILURE SCANNER Logs per workflow

10,000

Run 35294030895 [16]

New, stable QUIC failure. I’ll pair ... Authentication failed with provider at

1,000

http://172.30.0.30:10002 (HTTP 403). [copilot-harness] attempt 1: provider authentication

100

failed after partial execution - will retry once ... [copilot-harness] attempt 2: authentication failed ---

10

not retrying ##[error]Process completed with exit code 1.

1 Workflows with logs

Fig. 3. Distribution of logs per workflow across 6,040 workflows. The logscaled axis shows the median (dot) and interquartile range (line).

The scanner’s two attempts are retries inside one GitHub run. Such paired messages allow studies of task, access-policy, and infrastructure failures, including whether recovery was attempted.

TABLE I W ORKFLOWS AND NONEMPTY LOGS BY OBSERVED RUN TRIGGER . Trigger schedule pull request target pull request issue comment workflow dispatch issues workflow run check suite pull request review comment push check run discussion deployment status pull request review release discussion comment repository dispatch create Total with logs

Workflows 4,766 (78.91%) 59 (0.98%) 365 (6.04%) 171 (2.83%) 971 (16.08%) 304 (5.03%) 156 (2.58%) 3 (0.05%) 23 (0.38%) 101 (1.67%) 5 (0.08%) 11 (0.18%) 13 (0.22%) 5 (0.08%) 13 (0.22%) 8 (0.13%) 3 (0.05%) 3 (0.05%) 6,040 (100.00%)

Logs 252,691 (44.89%) 73,594 (13.07%) 61,224 (10.88%) 58,312 (10.36%) 46,165 (8.20%) 26,719 (4.75%) 18,944 (3.37%) 12,061 (2.14%) 6,351 (1.13%) 3,375 (0.60%) 2,063 (0.37%) 617 (0.11%) 425 (0.08%) 201 (0.04%) 56 (0.01%) 46 (< 0.01%) 21 (< 0.01%) 12 (< 0.01%) 562,877 (100.00%)

not attribute every run failure to its agent. Examples 1 and 2 show a blocked task and a provider failure, respectively.

B. Visible Agent Decision Traces Agent-job logs can record a prompt-file path and size, visible agent messages, tool calls, and responses. In the cases below, the full prompt text is absent, so these traces should not be treated as complete prompts or private reasoning. In Examples 3 and 4, one agent changes its publishing command after a tool denial, while another inspects a dependency update before posting its verdict. In these Copilot runs, the generated YAML labels the execution step Execute GitHub Copilot CLI with id: agentic_execution. Its job log prints [INFO] Executing agent command... before the harness starts. The ZIP member *_agent.txt is a whole job log, not a role label for each message. Example 3: ISSUE TRIAGE

Run 35351834895 [17]

• Read issue-context.md Example 1: ISSUE TRIAGE

Run 30125677266 [15]

✗ Publish triage summary for issue #50643 (shell)

✗ issue_read (MCP: github) · method: "get", ...

⌞ Permission to run this tool was denied due to the

⌞ MCP server ’github’: McpError: MCP error 0: [Filtered]

following rules: ‘shell(cat)‘

filtered by integrity policy (Resource

Let me use a different approach:

... I cannot read its content to perform triage.

Perfect! The triage summary has been successfully

• report_incomplete (MCP: safeoutputs) · reason: ...

published.

Example 4: DEPENDENCY CHECK

Run 33057082627 [18]

Now I need to check the diff between 2.4.1 and 2.5.0 for async_blocking and security, and check the PR link. • Get diff between 2.4.1 and 2.5.0 (shell) Now I have all the info needed. Let me compile the verdicts: • add_comment (MCP: safeoutputs) · body: ... ⌞ {"result":"success","temporary_id":"aw_k0yXps72", "comment":"#aw_k0yXps72"}

These sequences support studies of tool selection, adaptation after denial, and how evidence is turned into an action. The logged task statements are agent output, not a substitute for the complete prompt. C. Multi-Agent Coordination Example 5 shows a review run that dispatched three background reviewers and used agreement between the two respondents to select findings. It shows within-run delegation and synthesis, while providing no evidence of memory shared across runs. Example 5: EXPERT CODE REVIEW

Run 31484611735 [19]

[7] ♦ This is a documentation-only PR (4 files, all docs/README). Now dispatching 3 parallel expert reviewers. [8] ✓ task(name: reviewer-1, description: Reviewer 1: deep reasoning review, ...) [10] ✓ task(name: reviewer-3, description: Reviewer 3: alternative perspective revi..., ...) [97] ✓ read_agent(agent_id: reviewer-2, timeout: 180, wait: true) [101] ♦ Reviewer 3 produced no response → 2-reviewer mode. Both completed reviewers agree (2/2) ...

This trace supports analysis of division of labor, missing responses, and how agreement among agents changes a final review. V. C ONCLUSION We presented GH-AWTrace, a dataset linking 25,014 verified GitHub Agentic Workflows across 5,051 repositories to 4,112,526 run records and 562,877 nonempty execution log archives. By preserving execution evidence alongside workflow and run metadata, the dataset supports analyses that cannot be conducted from workflow definitions or run conclusions alone. Illustrative cases reveal access restrictions, provider failures, changes in agent actions, and within-run delegation. The dataset provides partial observations of execution. Log availability is constrained by retention and retrieval coverage, and nonempty archives do not necessarily contain complete agent traces. Analyses should therefore distinguish workflow runs from execution attempts, reported outcomes from observed task completion, and agent behavior from infrastructure failures. Concrete research directions include developing failure taxonomies, examining recovery after tool denials, and investigating agreement between run conclusions and visible task progress. GH-AWTrace provides preserved evidence for

pursuing these questions, while broader claims require systematic sampling and validation. DATA AVAILABILITY GH-AWTrace, including its linked data tables, nonempty log archives, schema, and access instructions, is publicly available at https://huggingface.co/datasets/jaykay9999/GHAWTrace. A Python example joins the tables, reproduces the 31.7% result, and extracts a cited agent log. R EFERENCES [1] J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “SWE-agent: Agent-computer interfaces enable automated software engineering,” in Advances in Neural Information Processing Systems, 2024. [2] GitHub, “Kick off a task with GitHub Copilot coding agent,” https: //docs.github.com/en/copilot/how-tos/copilot-on-github/use-copilot-age nts/kick-off-a-task. [3] ——, “About GitHub Agentic Workflows,” https://docs.github.com/en/c opilot/concepts/agent-capabilities/agent-powered-workflows-and-opera tions/about-github-agentic-workflows, accessed: 2026-10-22. [4] Azure/azure-dev, “Update instructions from PR reviews, agentic workflow definition,” https://github.com/Azure/azure-dev/blob/6a8fa2ff7b6 c457767cf95c8c7a7d8d044595957/.github/workflows/update-instructi ons-from-pr-reviews.md, 2026. [5] ——, “Update Copilot instructions from PR review patterns, issue 9507,” https://github.com/Azure/azure-dev/issues/9507, 2026. [6] ——, “Update Copilot instructions from PR review patterns, pull request 9731,” https://github.com/Azure/azure-dev/pull/9731, 2026. [7] J. Khelifi, I. Oukhay, A. Ouni, M. Sayagh, and M. A. Saied, “Specifying and maintaining agentic workflows: An empirical study of GitHub Agentic Workflows,” arXiv:2609.27263, 2026. [8] P. Valenzuela-Toledo, T. Kehrer, and S. Panichella, “GHAW-H: A dataset of GitHub Agentic Workflow histories,” Zenodo, version 0.1.2. https: //doi.org/10.5281/zenodo.22084012, 2026. [9] K. Sun, H. Kuang, S. Baltes, X. Zhou, H. Zhang, X. Ma, G. Rong, D. Shao, and C. Treude, “Does AI code review lead to code changes? a case study of GitHub Actions,” arXiv:2508.18771, 2025. [10] S. Wang, X. Hou, Z. Liu, Y. Zhao, X. Cheng, Q. Zou, X. Zhang, and H. Wang, “Demystifying and detecting agentic workflow injection vulnerabilities in GitHub Actions,” arXiv:2605.07135, 2026. [11] J. Isbarov, U. Suleymanov, I. Shumailov, and M. Kantarcioglu, “GitInject: Real-world prompt injection attacks in AI-powered CI/CD pipelines,” arXiv:2606.09935, 2026. [12] GitHub, “REST API endpoints for search,” https://docs.github.com/en/r est/search/search, accessed: 2026-10-21. [13] ——, “REST API endpoints for workflow runs,” https://docs.github.co m/en/rest/actions/workflow-runs. [14] ——, “Managing GitHub Actions settings for a repository,” https://do cs.github.com/en/repositories/managing-your-repositorys-settings-and-f eatures. [15] open-feature/dotnet-sdk-contrib, “Issue triage, GitHub Actions run 30125677266, attempt 1,” https://huggingface.co/datasets/jaykay99 99/GHAWTrace/blob/c6ec1a7109e3e2c463f29611db1f001bcf903070/e xamples/30125677266/attempt-1/3 agent.txt, 2026. [16] dotnet/runtime, “CI outer-loop failure scanner, GitHub Actions run 35294030895, attempt 1,” https://huggingface.co/datasets/jaykay99 99/GHAWTrace/blob/c6ec1a7109e3e2c463f29611db1f001bcf903070/e xamples/35294030895/attempt-1/3 agent.txt, 2026. [17] microsoft/PowerToys, “AI issue triage, GitHub Actions run 35351834895, attempt 1,” https://huggingface.co/datasets/jaykay 9999/GHAWTrace/blob/c6ec1a7109e3e2c463f29611db1f001bcf903070 /examples/35351834895/attempt-1/4 agent.txt, 2026. [18] home-assistant/core, “Check requirements (AW), GitHub Actions run 33057082627, attempt 1,” https://huggingface.co/datasets/jaykay9999/G HAWTrace/blob/c6ec1a7109e3e2c463f29611db1f001bcf903070/examp les/33057082627/attempt-1/3 agent.txt, 2026. [19] dotnet/maui-labs, “Expert code review (auto), GitHub Actions run 31484611735, attempt 1,” https://huggingface.co/datasets/jaykay99 99/GHAWTrace/blob/c6ec1a7109e3e2c463f29611db1f001bcf903070/e xamples/31484611735/attempt-1/3 agent.txt, 2026.

Related documents

• Specifying and Maintaining Agentic Workflows: An Empirical Study of GitHub Agentic Workflows#1063474
• A Methodology for Investigating AI Patterns Prevalence in Software Repositories#329161
• Silent Failures in Agent-Tool Interaction: An Audit of ToolUniverse#1450633
• Silent Failures in Agent-Tool Interaction: An Audit of ToolUniverse#1063047
• Unsupervised Tool Evolution for Coding Agents in Data-Lake Analytics#1402715