Descriptive Execution of HPC Applications and Workflows Vanessa Sochat∗
Daniel Milroy
[email protected] Lawrence Livermore National Laboratory Livermore, California, USA
[email protected] Lawrence Livermore National Laboratory Livermore, California, USA
arXiv:2607.10081v1 [cs.DC] 11 Jul 2026
Abstract The means to execute and orchestrate software components has changed from human-written code to descriptive prose. In high performance computing, this transition is represented in application orchestration, workload management, and system monitoring and debugging, to name a few. The underlying means to enable descriptive definition of tasks is the use of Large Language Model with associated tool functions and resources. A combination of a model with access to such resources, modeled in software, encompasses an autonomous framework. As fully automated and agentic frameworks are developed for science, it is important to assess reliability and strategies scoped to specific tasks. In this work, we assess the extent to which an agentic framework can optimize and run an HPC scaling study with a low latency network in Amazon Web Services, accurately transform HPC job specifications between workload managers, and design and run an entire biosciences workflow. We find that the framework completes all three tasks while surfacing task-specific failure modes. In the scaling study, agents deploy and optimize applications but monitor running jobs inefficiently, preferring conservative fixed waits over event subscriptions. In job translation, they convert specifications between Slurm and Flux with high accuracy, with processor-affinity flags the most common error. In the bioscience workflow, the agent reproduces an expertwritten variant-calling pipeline almost exactly—agreeing with the reference call set in 18 of 19 completed runs—and reaches this result through many distinct yet functionally equivalent workflow implementations. This information is invaluable moving forward to developing multi-cluster setups with scheduling and transformation handled by agents.
1
Introduction
The need for representation of tasks into inputs that can processed by a machine can be traced back as far as the 1890 U.S. Census, when punch cards were used to automate voting [6]. The method expanded in the 1900s to handle more tasks, from serving as library cards to proving mathematical theorems, and were foundational to modern computing today. The study of how people communicate with computers is Human-Computer Interaction (HCI) [9] and is based on the idea of mapping human intent into tasks that can be executed by a computer. Software is an attempt to materialize human understanding into operations that a computer can perform. Data captures states of phenomena that, for scientific contexts, we typically want to use to model the real world. A data format is an attempt to standardize the phenomena into structures that can best be processed by precise instructions. A modern workflow is capturing and executing a set of tasks with states, and dependencies. Even in early computing, insert of a punch card or execution of ∗ Corresponding Author
code was a rudimentary conversation between man and machine. A successful interaction leads to a meaningful computational result that is aligned with the initial intent. The different strategies that computational scientists have adopted to use software and data to accomplish scoped tasks can be described as progressive stages. Andrej Kaparthy defines Software 1.0, 2.0, and 3.0 [13] as a transition through stages of traditional coding, neural network methods, and using Large Language Models (LLM), respectively. We might consider units of execution changing from single programs to workflows and now conversational artificial intelligence (AI). Tasks are delegated to agents instead of programs. The 1960s through 1990s were defined by direct management of hardware paired with command line or software interfaces [3, 5], and the 2000s by a shift from running single programs to graphs of dependencies [4]. Two decades later, the 2020s are becoming defined by Artificial Intelligence and Machine Learning (AI/ML) orchestration empowered by LLMs. It might be pointed out that the way in which we interact with machines is becoming closer to how we interact with one another. It is flexible, and conversational. It is not clear if this is advantageous due to expressibility, or deleterious due to loss of control. While a detailed analysis of the cultural and social implications are outside of the scope of this paper, we might summarize some of the broad challenges. The first challenge is representation of work, tasks, and outcomes. Traditional computing uses precise machine instructions [14] in programs paired with return codes. The user goal is directly programmed, affording a tight level of control over program executions. The gap between the user goal and actions required to execute it grows with workflow tools that add a layer of abstraction to describe interactions and dependencies between tasks [4]. Adding LLMa moves the initial task to represent intent one step earlier. Instead of modeling our intent directly into software, we provide LLM agents with a goal that they must process to derive the entire execution, including inputs, tasks, and interaction with the environment. In this context, an ability to understand the environment and discover resources [24] becomes paramount, along with setting and enforcing strong constraints on agent behavior. While the interactions are conversational, communication with an LLM agent is different than that with a human [8]. It cannot be assumed that agents will always make the correct or same decisions. The next challenge is heterogeneity of resources, and complexity of scientific workflows. Whether it is a human or an agent executing tasks toward a goal, the higher degree of complexity of the resources leads to a greater number of possible configurations, which makes the task harder, and sometimes computationally infeasible [15]. The inherent non-deterministic nature of agents makes their outputs often not reproducible [12]. A lack of understanding that is created by way of allocating responsibility of ownership to
Conference, Nov 15-20, 2026, Chicago, IL
LLMs is likely a barrier to establishing trust. Finally, there is a conflict between precision and flexibility. high performance computing (HPC) environments are unforgiving in the formats and commands they expect. Agents driven by LLMs are asked to achieve goals that are non-specific, such as optimizing a HPC application. As an example, a prompt to schedule a workload might ask for optimizing for memory efficiency, or ensuring there is CPU affinity. An agent not only needs to understand the meaning of the requests, but to map the meaning to a specific workload manager that can provide it, and done exactly with the flags and arguments that the manager expects. Understanding the degree to which LLM agents can perform this task, and strategies for going about it, are the goals of this paper. In this work, we explore using descriptive language as more rigorous input for HPC optimization and agent-led job conversion. In our Method (Section 2) we demonstrate the ability of an agentic framework to design and run a scaling study across 5 well known HPC applications and benchmarks, perform translation of job specifications between workload managers for Slurm and Flux Framework [1], and design and execute bioscience workflows. We review results (Section 3), demonstrating that an agentic framework can optimize and run HPC scaling studies, translate the majority of job specifications between Slurm and Flux, and design and execute bioscience workflows that reproduce expert-written results with high fidelity, while revealing concrete, task-specific limitations in job monitoring, affinity translation, and adherence to specification. We make the following contributions: • Execution and analysis of 4 HPC applications in Kubernetes • Best practices for HPC translation prompts • Agentic design and run of Snakemake workflows • Evaluation of outcomes against human expertise • Software prototype for agentic orchestration We start with an introduction to agentic roles and definitions (Section 2) and describe our methods to orchestrate agentic execution and respond to failure. We describe experiments and results (Section 3). We finish with a discussion of lessons learned and best practices for collaboration between agents and humans.
2 Methods 2.1 Overview Our study includes four experiments that focus on real-world use cases. The first (Section 2.2) is intended to test the ability of an agent to translate a human request and intent to run and scale an application into an orchestrated scaling study. The second experiment looked at job translation, testing an agent’s ability to translate 1,000 job specifications from open source repositories into the Flux Framework workload manager (Section 2.3). The third experiment uses applications from our optimization study and our transformer agent. We vary performance flags across 1-5 nodes, and ask the translation agent to do an equivalent transformation to Slurm (2.4). We then test the scripts and assess performance and the agent’s ability to predict negative implications of the translation. Finally, we demonstrate design and execution of 3 Snakemake workflows, where the agent is provided with input data and a goal, and required to dynamically write and execute the workflow (Section 2.5).
Sochat and Milroy.
Agentic Framework An agentic framework is a design philosophy about how to combine interactions with an LLM with an initial user prompt, and tools. Specifically, the tools allow for exploration of the environment and discovery of resources, along with enabling the agent to execute interactions needed toward a task. We designed software, fractale [16] that provides an agent expert abstraction [17], where the agent has expertise toward a scoped task. The expertise is defined by the prompt, and the agent is designed to work in a discovery, action, evaluation, and analysis loop, discovering tools available in the context of the task requested, and making turns of calls and analysis of the result until a desired outcome or a maximum number of attempts is reached. The software has a modular design for agents, allowing us to design experts for an optimization study, a job transformer, and a workflow orchestrator. The software supports agent backends for OpenAI family and Google Gemini models, and we chose to use Google Gemini for this work.
2.2
Scaling Study
We aimed to deploy 4 HPC applications and benchmarks in the leading cloud orchestrator, Kubernetes, using previously built containers from prior work [23]. This task is motivated by our own experiences running scaling studies, for which the manual work is often arduous and time consuming, even with declarative management and automation [22, 25]. We chose Flux Framework [1] as our workload manager because it is easy to deploy in Kubernetes with a single manifest with the Flux Operator [21], and our team has expertise in using it. Our applications include a set that are intended to strong scale (LAMMPS, Kripke), weak scale (AMG2023) or be used as benchmarks (OSU Benchmarks). We used the Amazon Web Services hpg7g.16xlarge instance type with the Elastic Fabric Adapter (EFA) to enable low-latency networking [2]. For each application, we choose problem sizes that will run at approximately 5 minutes for the smallest number of nodes (N=1). We run 10 iterations for each size from 1 to 5 nodes, and have described each application in detail in previous work [25]. Optimization Agent The Flux Operator optimization agent is an expert at creation and deployment of the MiniCluster custom resource definition (CRD). To work successfully, it must authenticate with the cluster, discover resources available, generate and apply the CRD, and monitor the status of the job. An optimization agent goal should include a explicit task, constraints, criteria for completion, and other points of expertise from the human executioner. The agent must generate a command for Flux, along with ensuring the deployment matches the node. The optimization agent can use any set of tools discovered via the Model Context Protocol (MCP). For our experiments, we deploy the expert alongside our mcp-server framework [24] that provisions 16 tools to interact with Kubernetes [17] along with a Kubernetes event subscription. Agent Instructions The optimization agent is given a 5 node cluster as resources. For each scaling study, we instruct the agent to discover cluster resources, and deploy a MiniCluster CRD starting at size 1 that uses the maximum cores available. We instruct the agent to maximize or minimize a figure of merit (FOM) relevant to each application at one node. As an example, for LAMMPS, we instruct the agent to start at problem size x,y,z of 20 20 20, and then go up
Descriptive Execution of HPC Applications and Workflows
in increments of 1, and adjust individual sizes when OOMKilled. When the run is successful and deemed optimized by the agent, we instruct the agent to pin problem size parameters and scale up to the maximum size of the cluster, 5 nodes, and report a final structured result in JSON. In practice, this means the agent must write and create the custom resource MiniCluster, check status and debug errors that arise, clean up on error, wait for success, retrieve and parse logs to extract the FOM. The exception is with Kripke, for which it is not possible to deploy on 3 or 5 nodes, and we provide the agent with a 6 node cluster. We ask the agents to return FOMs, reasons for decisions, and literature references to justify them. For each of 5 applications and 5 sizes, we performed 5 scaling studies for a total of 25 studies and 125 size, application, and iteration combinations.
2.3
Job Translation
A second challenge for HPC practitioners is translation of jobs to run between workload managers. Moving work between managers has become an increasingly common task given resource contention and the need to use multi-cluster environments. For this experiment, we start with an open source database of 33,744 job specifications parsed from GitHub [18] and remove outliers outside of the 95th percentile (𝑁 = 32, 001). We randomly select 1,000 job specifications with 1,000 or fewer tokens to be more conservative with respect to sending tokens to the Gemini API. To ensure the sample is comparable in complexity to the larger set, we calculate the cyclomatic complexity using shellmetrics [10]. The cyclomatic complexity is a quantitative measure of the number of linearly independent paths through a program’s source code. Importantly, our 1K subset (𝑁 = 1, 000, 𝜇 = 1.16, 𝜎 = 0.70) is comparable to the larger sample (𝑁 = 32, 001, 𝜇 = 1.64, 𝜎 = 1.31), with a Z-score of −0.369 (within 0.37 standard deviations of the larger sample mean). Job Translation Instructions For each transformation, we provide the agent with full help output for the Flux submit command, and the instruction for how to represent a flag in a batch script. We additionally tested a more detailed case, asking the agent to ignore accounts, partitions, queues, mail, banks, notifications, and commented out directives. We also provide clear instruction for how Flux models affinity after observing mixing up with Slurm affinity options. Job Translation Agent The job translation agent is an expert at transforming job specifications between workload managers, honoring a user request for a source and destination manager, along with user-specific goals. The agent is instructed to preserve as many options as possible from the original, and leave out directives have no means to translate, with comment about performance implications. For our translation task, we request to leave out directives related to user or cluster identity such as queue or partition and email notifications. We instruct the agent that if a validation tool is available, it should be used to validate the job specification. Finally, the agent is instructed to derive a list of issues with labels UNKNOWN_TO_ME (a parameter not familiar with that cannot be figured out), NO_ANALOGOUS, and MISSING for when there is no possible conversion.
Conference, Nov 15-20, 2026, Chicago, IL
2.4
Translation Performance
To combine our first two experiments, we use the final optimized container images from our scaling and optimization study (Section 2.2) and ask the transformer agent to transform each of Kripke, LAMMPS, AMG, and OSU Benchmarks osu_latency and osu_allreduce from Flux into Slurm. For each, we generated and requested job specifications between 1 and 5 nodes, with and without affinity. We ran 10 iterations for each application at each size, and for with and without affinity. Our next goal was to test the actual performance of the translated job specifications compared to the original. For our original job specifications, we again used the Flux Operator [21], this time manually orchestrating the work from 1 to 5 nodes. For Slurm, although we first intended to use AWS Parallel Cluster or AWS Parallel Compute Service with Singularity [7] containers, we were unable to deploy the setups customized for our needs. As an alternative, and a solution closer to the Flux Operator in using a deployment of the workload manager directly in Kubernetes with the same containers, we used the Slurm Operator for Kubernetes [19]. A successful translation result will be indicated if the performance between runs at equivalent scales and affinity are not significantly different.
2.5
Workflow Orchestration
A final challenge for descriptive execution is design and orchestration of workflows. The Snakemake workflow manager [11] is ideal for automation by an agent because it defines Snakemake wrappers – packaged single unit steps that are common across bioinformatics workflows. By way of designing a catalog of Snakemake tools for the agent via our server and providers software [20], the agent is able to design the and execute the entire workflow. The mcp-server is deployed in the context of a job, with a specific scope to execute one workflow. A read only data input directory is provided alongside a read and write working directory. When the server starts, it stages input data and workflow steps to the working directory for the agent to work from. It dynamically downloads and parses the snakemake wrappers repository. Traditionally, a human might write the workflow file that defines steps with inputs and outputs, called a Snakefile. In this case, the agent will be tasked to write it, and provided with functions to view the current Snakefile, roll back a step, or get metadata about a specific wrapper. Upon startup, the agent receives a prompt with a specific goal (e.g., variant calling or sequence alignment) and is able to see a suite of tools to interact with snakemake, including listing and getting details for wrappers, viewing or editing the current Snakefile, inspecting data and inputs, and executing wrappers or rules. The staging design is ideal as it protects the agent from destructing actions against input data, and allows for multiple agents to work on the data at once. There is no redundancy because the data is linked, and the links can be deleted without harming the initial data. The software rolls back failed steps automatically, which includes removing the Snakefile rule and the step output directory. To assess the ability of an agent to orchestrate a workflow, we run a Snakemake variant calling workflow 20 times and compare against a ground truth, manually generated Snakefile. Snakemake Agent The Snakemake workflow expert is given input data, and tasked with a scientific objective. It must execute a
Conference, Nov 15-20, 2026, Chicago, IL
Sochat and Milroy.
complete workflow until the goal is reached. The agent is given a structured input directory, permissions and path details, and instructed to inspect the computational environment to inspect wrappers and data. For execution, the agent adds and executes one step at a time and ensures success or makes another attempt. A structured JSON object must be returned upon completion. During the prototyping stage, We tested us rolling back and agent rolling back, and allowing agent to explicitly delete step. The latter worked better and we roll back failed steps. Rollbacks are immutable.
optional steps such as filtering, normalization, merging, and readgroup assignment) by keyword. For each run we record the number of steps, the sequence of canonical operations, whether rules were expressed via Snakemake wrappers or shell commands, and the presence of any additional operations. Across runs we count the distinct step-name layouts, the distinct canonical operation sequences, and, per operation, the number of distinct step-directory names used to implement it. The analysis is implemented as command-line tools depending only on bcftools and a faidx-indexed reference.
Snakemake Analysis Our analysis goal is to compare agentgenerated workflow results against a manually written reference pipeline that can be considered a ground truth. The reference pipeline maps short reads to a reference genome, sorts and indexes the alignments, and jointly calls genomic variants across three samples. We are careful to install the same samtools and bcftools as in the wrappers to ensure comparability of output – any deviance in results can be attributable to the workflow orchestration. For each output iteration, the agent will generate a full output directory and Snakefile. Because the agent decides the specific steps to use and the naming of the output step directories, there can be variation in the output structure, and so the analysis does not assume a fixed directory layout or file paths. For each run we locate the final variant-call file based on content rather than path. We classify each candidate file using the provenance bcftools writes into its output headers and the presence of a genotype (GT) field. A file produced by bcftools call is taken as a call set, whereas a file produced only by bcftools mpileup, lacking genotypes, is treated as an intermediate. When multiple call sets are present, the one consumed as input by another based on input paths embedded in the header is treated as an upstream. Each call set is left aligned and decomposed into biallelic records against the reference (bcftools norm -f ref -m -any), sorted, and restricted to variant sites (records with an alternate allele), so that the reference output (bcftools call -mv, variant sites only) and the agentic output (bcftools call -m, all covered sites) are directly comparable. We intersect the reference and agentic call sets with bcftools isec, partitioning sites into shared (true positives, TP), reference-only (false negatives, FN), and agentic-only (false positives, FP), with the reference taken as truth. From these we compute precision, recall, 𝐹 1 , and the Jaccard index. A run is an exact reproduction when FP = FN = 0. Comparison is at the site level (chrom/pos/ref/alt) and is therefore independent of sample labels. Joint calling is performed across the sample set defined by the reference. The agent’s sample set is recorded for each run, and where a run carries a different set, its call set is restricted to the samples common with the reference (bcftools view -s ⟨samples⟩ –trim-alt-alleles) before comparison. For runs that apply a step after calling (e.g. a hard-quality filter) absent from the reference, we report the concordance of the terminal deliverable and, separately, of the upstream call-stage output. We may find that we arrive at identical results haven taken different paths, or using a different structure of steps. We would want to understand the overall categorization of steps. To characterize how each workflow is constructed, we parse the per-run step directories and rule definitions and mapped each step to a canonical scientific operation (alignment, sorting, indexing, pileup, calling, and
3 Results 3.1 Scaling Study We performed a scaling study across five nodes and three applications, asking the agent to optimize at the smallest size and scale up appropriately to maximize or minimize a FOM of interest. Best achieved FOM for each optimization size are reported in Table 1 and results across independent studies shown in Figure 2. For LAMMPS, variance was exceptionally small, reflecting consistency in the runs. Kripke was the most erroneous, with the majority of the runs being unsuccessful to deploy the application to an odd number of nodes. For all applications, we were predominantly interested in the agent’s ability to perform the study and adjust parameters when required. The variance for AMG is large due to independent scaling studies arriving at different maximum problem sizes. An interface to explore full results is available 1 . Table 1: Best Achieved FOM Performance
Application Kripke AMG LAMMPS
3.2
Metric
Best Value
Nodes
Time (s) FOM Overall M-atom steps/s
5.6584 × 10−10
5 5 5
6.9176 × 1009 3.5010
Job Translation
Of the sample, 18 jobs were not converted due to not being recognizable batch scripts associated with a workload manager, leaving 982 jobs. Of the 982 jobs, all translations were validated by production Flux, meaning successful submission of a dry run without report of error. Importantly, a validation does not indicate that the translation was able to reliably map the intention of the job specification. Across translations, the agent reported no analogous translation 1033 times, s missing parameter 237 times, and an unknown directive 62 times. We generated 8 categories of issues based on groups of flags reported problematic by the agent, including I/O, memory, topology, GPU specification, MPI, job arrays, environment, and scheduling Figure 3. The translation agent has the most difficulty with I/O (N=802 issues), primarily due to being unable to handle Slurm templates for input and output files translated to Flux. While important for preserving data in real world runs, these flags with no direct analogous mapping do not influence application performance. Memory allocation (N=436) was an expected result, as Flux deploys predominantly to clusters that require exclusive node access (all memory) and has no flags to specify memory for a job. Flags related 1 https://converged-computing.org/fractale-experiments/optimization-
study/data/single-node-1/
Descriptive Execution of HPC Applications and Workflows
Conference, Nov 15-20, 2026, Chicago, IL
Figure 1: Translation performance for OSU Benchmarks . OSU Latency results between Slurm and Flux to Slurm were almost identical, and overlap on the plot. OSU All Reduce is reported across message sizes for 5 nodes.
Figure 2: Maximum FOM achieved for HPC applications for 1-5 nodes.
to slurm gres were noted as problematic issues by agents (N=110), along with a lack of job arrays (N=54). An interface with complete results for exploration is available 2 .
Figure 3: Categorized issues for Flux conversions. Flux does not have a memory flag, and the agent
3.3
Translation Performance
We used application containers from our scaling study with each of Slurm and Flux operators in Kubernetes to run job commands 2 https://converged-computing.org/fractale-experiments/jobspec-agent-
conversion/results/
generated by human experts as compared to an agentic translation. Specifically, we did a conversion from Flux to Slurm. Results for each of AMG, LAMMPS, and Kripke are reported in Figure 4 and for OSU Benchmarks in 1. The performance of the agent translation from Flux to Slurm at 5 nodes for AllReduce (𝜇 = 11645.25, 𝜎 = 4237.68) showed a 28.79% increase in mean latency compared to human-derived Slurm (𝜇 = 9042.21, 𝜎 = 353.3). Due to high performance variability, the difference was not statistically significant (𝑝 = 3.99𝑒 − 01). There were no significant differences for any other application between Flux, Slurm, and the Flux to Slurm translation, with the exception at LAMMPS at size 5 for the Flux to Slurm translation (𝜇 = 2.920, 𝜎 = 0.03) as compared to Slurm with and without affinity (𝜇 = 3.06, 𝜎 = 0.03). For the set of translations at this size, jobs that attempted to set affinity were not successful, meaning that the result only includes jobs without affinity. All other sizes have at least one affinity run. Lack of affinity would likely be more apparent at larger sizes. Closer inspection of all applications and benchmarks revealed performance differences were due to not asking Slurm correctly to set affinity. For a total of 76 Slurm commands across 5 applications between 64 and 320 tasks, 63 ran successfully, and 13 produced erroneous cpu-bind flags, asking for one of task, tasks, or the Flux flag per-task. Looking at agent translation logs, in all erroneous cases the agent identified the Flux flag as not having an analogous translation, but added the incorrect flag anyway. This addition
Conference, Nov 15-20, 2026, Chicago, IL
violated an instruction given to our agent. We believe this results from the agent getting caught up on the discrepancy between cpuaffinity being an option flag, and Slurm not having option flags. In assessing this discrepancy, the agent forgets the initial prompt to not hallucinate flags or options. Finally, the most interesting result was the discrepancy in performance between Slurm (𝑚𝑢 = 7.41, 𝜎 = 0.03) and Flux (𝑚𝑢 = 11.32, 𝜎 = 0.22) for osu-latency shown as the gap between blue and green in Figure 1.
3.4
Workflow Orchestration
We ran a full workflow for variant calling with Snakemake using a Snakemake agent and compared to 21 iterations of an equivalent ground truth run. The agent followed a similar pattern of behavior each time. We observe the agent looking at the environment, specifically the data input directory, wrappers available, and metadata about wrappers of interest. We then see the agent calling bcftools for variant calling for each of three samples. The agent is able to see the Snakefile on request, delete or execute rules or wrappers, and list contents of the input or working directory. Of 21 agentic iterations, 19 produced a usable variant call set. The remaining two did not (one resulted in a partial workflow with no final call set likely due to connectivity error, and the other was not successful to produce output). These runs were excluded from concordance analysis (Figure 5). Of the 19 runs we could evaluate, 18 (95%) reproduced the reference call set exactly (zero false positives and zero false negatives), giving a mean 𝐹 1 of 0.989 and mean Jaccard of 0.982 against the reference. Per-run concordance was at the ceiling for precision, recall, 𝐹 1 , and the Jaccard index, with a single exception (Figure 6). The lone divergent run (iteration 20) applied a hard-quality filter after calling that the reference workflow did not. This reduced its recall (terminal 𝐹 1 = 0.79) by removing 230 true variant sites, while its precision remained 1.0. Its upstream call-stage output was perfectly concordant with the reference (call-stage 𝐹 1 = 1.00), localizing the discrepancy to the added filter rather than to the variant caller. Because all runs that we were able to evaluate processed the reference sample set, the as-produced and sample-matched comparisons were identical, and no run was penalized for sample-set differences. Although the scientific result was almost invariant, the workflow implementations were not. Every run performed the core operation graph: alignment was present in 100% of the 20 runs that composed a workflow, and sorting, indexing, pileup, and calling each in 90–95% (Table 2). The small shortfalls reflect the partial run that produced no call set together with occasional folding of an operation into an adjacent step (e.g. sorting within another rule, or fusion of pileup and calling). Only a single run introduced an additional operation (a post-calling filter, and one run a merge step; 1/20 each). At the same time, the surface form varied substantially runs used between 3 and 11 steps (median 5), and a given operation was expressed under up to 6 distinct step names across runs (Table 2). In aggregate the 20 runs exhibited 10 distinct step-name layouts but only 7 distinct canonical operation sequences (Table 3), indicating that the agent’s variability is concentrated in naming and organization rather than in the scientific procedure.
Sochat and Milroy.
Taken together, the agent reproduces the reference scientific result with high reliability (exact reproduction in 95% of runs we could evaluate), and its few deviations are workflow-design choices—an added filtering step in one run, and failure to produce a usable call set in two—rather than errors in the underlying analysis.
4
Discussion
In this work, we use expert agents to semi-autonomously deploy and optimize HPC applications, convert job specifications between managers, and orchestrate workflows. We use the same underlying orchestration software with different expert agents across tasks. There are several topics for discussion.
4.1
Scaling Study
The variability of a FOM (AMG) or inability to deploy an application at a particular node size (Kripke) reflects an inadequacy of tools to expose the environment and application deployment difficulty. A tool that might provide hints or validation about parameter choices would avoid deployment, wait, and failure of erroneous commands. The serial design of our scaling study reflects a need to monitor execution, and is not representative of an ideal execution environment to run jobs in parallel with a workload manager queue. For our experiments, an agent running in an asynchronous loop to execute, wait for, and retrieve logs for an application must know when a job is complete. The task of waiting, in retrospect, led to many errors by agents concluding an incomplete log indicated an erroneous job when it required more execution time. While we provided and provisioned event notifications and gave the agent the ability to subscribe to job events, our expert agents more often chose to call an explicit sleep function between job executions, with increasing time as an experiment progressed. This setup does not afford optimal utilization of resources as agents often overestimate waiting time. A more intelligent design would use models to predict expected running times, and enforce the agents to subscribe to events. Subscribing to events would make it easer for one agent to handle more than application execution at once given a workload manager that delivers notifications across jobs. We also ran the study from a tabula rasa each time without giving agents additional hints about errors or preserving learning across iterations. An agent that is able to use a tool that exposes a model or database of previously successful runs could more quickly get an application working, and Table 2: Canonical scientific operations across agentic runs that produced a workflow. Runs counts those performing the operation, and Names is the number of distinct step-directory names used to implement it.
Operation
Runs (of 20)
Distinct names
Core operations Alignment 20 (100%) Sorting 18 (90%) Indexing 19 (95%) Pileup 19 (95%) Calling 19 (95%)
6 5 6 3 3
Additional operations Filtering 1 (5%) Merging 1 (5%)
1 1
Descriptive Execution of HPC Applications and Workflows
Conference, Nov 15-20, 2026, Chicago, IL
Figure 4: Translation performance for a scaling study across up to 5 sizes for two affinity variants for each of Flux and Slurm. There are no observable differences with respect to adding affinity. Asking the agent to
then better optimize. Further, a more realistic scenario is one of agent managing multiple jobs, and needing to explicitly respond to events. Providing tools and models that make it possible to better estimate job running times is not just important for checking status, but also for the initial scheduling. While we are actively working on parallel execution with events, we chose a serial design for this early study to make it easy and possible for the developer human user to observe the experiments. The serial task of performing optimization via submitting and waiting for jobs often does not reflect the human experience of
Table 3: Structural variation across the 20 agentic workflows.
Quantity Runs producing a workflow Steps per workflow (min / median / max) Distinct step-name layouts Distinct operation sequences Distinct operation sets Runs using wrapper / shell rules Runs adding a filter step
Value 20 3 / 5 / 11 10 7 5 20 / 4 1
testing applications interactively. To compare to what a human would do, experiments might test giving an agent access to the same MiniCluster from the lead broker, akin to running the application on bare metal across an interactive allocation. Without needing to wait for wrapping orchestration of Kubernetes, for example, the optimization and subsequent experiments could be run much faster.
4.2
Figure 5: Outcomes across 21 agentic runs. Exact reproduction of the reference call set, post-filter divergence, and runs producing no output. Percentages are of all runs.
4.3 Figure 6: Per-run concordance with the reference. We show precision, recall, 𝐹 1 , and the Jaccard index. Each point is one run and boxes summarise the distribution. Almost all runs sit at 1.0. The single outlier (iteration 20) added a post-calling filter and is otherwise concordant at the calling stage.
Performance Translation
The agent did well to translate jobs between workload managers Flux and Slurm, with the exception of affinity flags. Even with help provided for each of Slurm and Flux in the prompt, providing an incorrect flag option was common. The translation agent would often make more errors when distracted with an irrelevant detail. For affinity, Flux uses broker flags that are requested with an -o option. In cases where the agent made a mistake, it commented verbosely on the inability to map the broker option to Slurm, and then subsequently hallucinated an incorrect affinity flag. We think that this issue might be resolved given a Slurm validator tool or an adversarial agent approach where one agent derives a command and a second is required to scrutinize it. A surprising and pleasant result in our work was observing relative equality of performance between orchestration of applications using Flux and Slurm. In a highly contentious landscape of workload managers, we can attest from our work that both are good choices in this set.
Workflow Design and Reproduction
The biosciences task asked the agent to design and execute a complete variant-calling workflow—read alignment, sorting, indexing, and joint variant calling-rather than to fill in a fixed template. Assessed against a manually-written reference, the agent was reliable. and a single divergence attributable to an extra hard-filtering step
Conference, Nov 15-20, 2026, Chicago, IL
the agent added of its own accord rather than to an error in calling. What the agent did not do reliably was converge on a single implementation, exhibiting ten distinct step layouts but only seven distinct operation sequences. The framework thus reproduces the scientific result more consistently than the code that produces it. This separation between functional and surface variability has a practical consequence for how agentic workflows should be evaluated. Comparing agent output to a reference by file paths, filenames, or even rule text is misleading, because two correct workflows rarely share them. We found it necessary to compare the scientific artifacts directly—normalizing and intersecting the called variant sites—and to recover each workflow’s structure from tool provenance rather than from naming. While we expect this content- and provenancebased evaluation to generalize to other scientific workflows, we recognize that this task would be challenging given result files that cannot be compared. Finally, the deviations we observed—an unrequested filtering step and two empty outputs—are similar to failure we saw in other experiments in this larger study. Akin to the hallucinated affinity flags in translation, an agent that adds or omits a step would benefit from a validation tool or an adversarial reviewer that checks the produced workflow against the specification before it is run.
4.4
Limitations
Our findings are conditioned on a single model and version, a fixed prompt design, and a particular tool set. Other LLM application programming interface (API) endpoints, newer model versions, or different prompt and tool configurations could shift both accuracy and failure modes, and we did not test across these choices. Agent behavior is also sensitive to prompt phrasing in ways we observed but did not quantify. For example, the translation agent made more errors when distracted by irrelevant detail. Absolute error rates should be read as characteristic rather than definitive. Each task was evaluated at experimental scale, including a serial scaling study, a bounded set of translation cases, and 21 independent bioscience workflow designs. These are sufficient to explore behavior and failure states, but are too few to report tight confidence intervals or to claim that rare failures have been observed. The serial, human-observable execution design further means our execution is not optimized. An optimized parallel, event-driven deployment would be ideal. As agentic approaches mature, we expect execution to be better optimized. Our agentic outputs are assessed against references and a small set of well-known applications, benchmarks, and a tutorial-derived biosciences pipeline. A single expert reference is just one way to perform a task. An agent output that diverges may be wrong, or merely different and still correct. Our concordance metrics treat the reference as ground truth when in fact there may be multiple derivatives of correct. Our experiment tasks themselves are representative but narrow, including five HPC applications, two workload managers, and one variant-calling workflow. Generalization to larger, multi-step, or less-documented scientific workflows remains to be shown. The biosciences comparison is also a site-level assessment of the final call set, and the workflow runs very quickly. We did not evaluate genotype-level concordance, runtime, or cost, which may matter for other workflows.
Sochat and Milroy.
We ran each experiment from a blank slate, without preserving learning across iterations or supplying error hints, and with a human able to observe but largely not intervene. This strategy does not measure the more realistic collaborative setting in which an agent accumulates context, manages multiple concurrent jobs, and escalates to a human on uncertainty. We do not yet know when an agent should defer to a human, nor did we measure how much the failures we report (e.g. hallucinated affinity flags or an unrequested filtering step) would be caught by a validation or adversarial-review mechanism. Finally, allowing semi-autonomous agents to deploy and execute code at HPC scale raises safety, cost, and reproducibility concerns.
5
Conclusion
Across three tasks, our experiments show that agentic orchestration is already reliable enough to be useful and revealing enough to be improved. Agents optimized and ran scaling studies, translated the majority of job specifications between Slurm and Flux, and reproduced an expert-designed biosciences workflow while exploring a wide space of equivalent implementations. The errors that remain, including inefficient job monitoring, flag hallucination, and occasional departures from specification, are concrete and addressable through validation tools, event-driven execution, and persistent memory of past runs. Agentic frameworks represent a transformative shift for the HPC community and the broader software landscape. The realization of autonomous, converged HPC infrastructure enables a collaborative paradigm where machines and humans jointly orchestrate the full lifecycle of scientific workloads. By integrating autonomous task execution and AI-optimized scheduling, these advancements will fundamentally redefine the workflow of scientists and engineers. We are entering a new era of HPC – one defined by unprecedented speed, precision, and accessibility.
Acknowledgments We are grateful for new cities and mountains to explore, colorful climbing walls, bows in hair, and the color green. This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DEAC52-07NA27344 and was supported by the LLNL-LDRD Program under Projects No. 24-SI-005 (LLNL-CONF-2021324).
References [1] Dong H. Ahn, Jim Garlick, Mark Grondona, Don Lipari, Becky Springmeyer, and Martin Schulz. 2014. Flux: A Next-Generation Resource Management Framework for Large HPC Centers. In 2014 43rd International Conference on Parallel Processing Workshops. 9–17. doi:10.1109/ICPPW.2014.15 [2] Amazon Web Services. 2022. The Elastic Fabric Adapter (EFA). https://docs.aws. amazon.com/AWSEC2/latest/UserGuide/efa.html Accessed: 2024-12-12. [3] Lyndon Clarke, Ian Glendinning, and Rolf Hempel. 1994. The MPI Message Passing Interface Standard. In Programming Environments for Massively Parallel Distributed Systems. Birkhäuser Basel, Basel, 213–218. [4] Ewa Deelman, Karan Vahi, Mats Rynge, Rajiv Mayani, Rafael Ferreira da Silva, George Papadimitriou, and Miron Livny. 2019. The evolution of the Pegasus workflow management software. Comput. Sci. Eng. 21, 4 (July 2019), 22–36. [5] Rolf Hempel and David W Walker. 1999. The emergence of the MPI message passing standard for parallel computing. Computer Standards & Interfaces 21, 1 (1999), 51–62. doi:10.1016/S0920-5489(99)00004-5 [6] Smithsonian Institute. 2026. Punch Cards for Data Processing. https://www.si. edu/spotlight/punch-cards/punch-cards-data-processing. Accessed: 2026-4-18.
Descriptive Execution of HPC Applications and Workflows
[7] Gregory M Kurtzer, Vanessa Sochat, and Michael W Bauer. 2017. Singularity: Scientific containers for mobility of compute. PloS one 12, 5 (2017), e0177459. [8] Zhicheng Lin. 2024. How to write effective prompts for large language models. Nat. Hum. Behav. 8, 4 (April 2024), 611–615. [9] Shaun Marsh. 1990. Human computer interaction: an operational definition. SIGCHI Bull. 22, 1 (June 1990), 16–22. [10] T.J. McCabe. 1976. A Complexity Measure. IEEE Transactions on Software Engineering SE-2, 4 (1976), 308–320. doi:10.1109/TSE.1976.233837 [11] Felix Mölder, Kim Philipp Jablonski, Brice Letcher, Michael B Hall, Christopher H Tomkins-Tinch, Vanessa Sochat, Jan Forster, Soohyun Lee, Sven O Twardziok, Alexander Kanitz, Andreas Wilm, Manuel Holtgrewe, Sven Rahmann, Sven Nahnsen, and Johannes Köster. 2021. Sustainable data analysis with Snakemake. F1000Res. 10 (Jan. 2021), 33. doi:10.12688/f1000research.29032.3 [12] Nature Machine Intelligence. 2026. Recognizing reproducibility and reusability in times of fast science. Nat. Mach. Intell. 8, 3 (March 2026), 293–294. [13] Paul Pajo. 2025. Software 3.0: A detailed examination of Andrej karpathy’s vision for AI-driven software development paradigms, natural language programming interfaces, and the emergent infrastructure role of large language models in the era of artificial intelligence. [14] Santa Cruz Operation, Inc. 1997. System V Application Binary Interface (edition 4.1 ed.). https://refspecs.linuxfoundation.org/elf/gabi41.pdf Generic System V ABI. [15] Aasish Kumar Sharma and Julian Kunkel. 2025. A review of tools and techniques for optimization of workload mapping and scheduling in heterogeneous HPC system. arXiv [cs.DC] (May 2025). [16] Vanessa Sochat. 2026. converged-computing/fractale 0.1.0. doi:10.5281/zenodo. 19654652 [17] Vanessa Sochat. 2026. converged-computing/fractale-agents: Fractale Agents Release v0.0.12. doi:10.5281/zenodo.19654729 [18] Vanessa Sochat. 2026. converged-computing/jobspec-database: Jobspec Database 0.0.1. doi:10.5281/zenodo.20112396 [19] Vanessa Sochat. 2026. converged-computing/slurm-operator: Slurm Operator Release v0.0.1. doi:10.5281/zenodo.20128848 [20] Vanessa Sochat. 2026. Resource Secretary (converged-computing/resource- secretary). doi:10.5281/zenodo.19476501 [21] Vanessa Sochat, Aldo Culquicondor, Antonio Ojea, and Daniel Milroy. 2024. The Flux Operator. F1000Res. 13, 203 (March 2024), 203. [22] Vanessa Sochat and Daniel Milroy. 2025. If we build them, they will run: Automated HPC apps deployment and profiling with eBPF in cloud. F1000Res. 14, 1235 (Nov. 2025), 1235. [23] Vanessa Sochat and Daniel Milroy. 2026. Agentic Orchestration of HPC Applications in Cloud. arXiv [cs.DC] (July 2026). [24] Vanessa Sochat and Daniel Milroy. 2026. Server Architecture for Agentic Science. SC26 Submission. Under Review. [25] Vanessa Sochat, Daniel Milroy, Abhik Sarkar, Aniruddha Marathe, and Tapasya Patki. 2025. Usability Evaluation of Cloud for HPC Applications. In Proceedings of the SC ’25 Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC Workshops ’25). Association for Computing Machinery, New York, NY, USA, 135–150. doi:10.1145/3731599. 3767353
Conference, Nov 15-20, 2026, Chicago, IL