Generated, Parallel, Scalable? A Study of Agentic AI-Generated Julia Code on Supercomputers Linus Bantel 1 , Anna-Lena Roth 2 , Jonas Posner 2 , and Dirk Pflüger 1
arXiv:2606.16534v1 [cs.DC] 15 Jun 2026
1
Institute for Parallel and Distributed Systems, University of Stuttgart, Germany {linus.bantel, dirk.pflueger}@ipvs.uni-stuttgart.de 2 Fulda University of Applied Sciences, Germany {anna-lena.roth, jonas.posner}@cs.hs-fulda.de
Abstract Julia is increasingly used in High-Performance Computing (HPC) as a single-language alternative to combining high-level scripting with low-level systems languages, but achieving scalable performance still requires expertise in parallel programming. Large Language Models (LLMs) are increasingly used for code generation and are advancing rapidly with each new version. Yet, existing studies focus on single-shot prompting rather than agentic settings, in which an LLM autonomously plans, generates, and refines code through tool use. Using an OpenCode-based agent extended with a Julia-documentation Model Context Protocol (MCP) server, we study agentic generation of parallel Julia code, focusing on task-based execution with Dagger.jl. We evaluate three LLMs, OpenAI GPT-5.5, Anthropic Claude Opus 4.7, and the open-weight Qwen3-Coder-Next, on three problems with distinct parallel structures: π approximation, tiled general matrix multiplication, and tiled Cholesky decomposition. The generated Dagger.jl implementations are compared against agent-generated Base.Threads and MPI.jl baselines, with shared-memory experiments scaling to 192 cores and distributed-memory experiments on two nodes. The agents reliably produce executable code for small inputs but fail at larger scales due to deadlocks, oversubscription, or out-of-memory errors, with the open-weight model affected most severely. The two commercial models scale comparably on Base.Threads and MPI.jl, while their Dagger.jl implementations expose recurring weaknesses in task dependencies, granularity, and scheduling. Agentic AI is promising for producing parallel Julia code, but generating robust, performance-aware implementations for large-scale HPC systems remains an open challenge. Keywords: LLM · Agentic AI · Julia · HPC
1
Introduction
Scientific software development in HPC often relies on multiple programming languages with different levels of abstraction. While high-level languages such as Python, MATLAB, and R support intuitive problem formulation, performancecritical components are commonly implemented in lower-level languages such as
2
Bantel et al.
C, C++, or Fortran, frequently combined with parallel programming models such as MPI, OpenMP, or CUDA to efficiently exploit modern HPC architectures. This separation creates the two-language problem, in which productive high-level code must be reimplemented for performance, increasing expertise requirements and maintenance effort while reducing code reusability [1,2,3]. Julia addresses this problem by combining a dynamic, interactive syntax suited to numerical and scientific computing [4,5] with LLVM-based JIT compilation to efficient native code, achieving performance close to traditional HPC languages for numerical kernels [6,7]. Recent studies show that Julia is competitive with established parallel programming models such as OpenMP, Kokkos, OpenCL, CUDA, HIP, and SYCL on selected HPC benchmarks [6], and that MPI.jl matches C MPI for most collective operations [5,8]. While these results demonstrate Julia’s potential for HPC, scalable performance on modern systems still requires applications to expose and coordinate parallelism across threads, processes, and distributed compute nodes. Using established models such as MPI or CUDA forces developers to manage communication, synchronization, data movement, and load balancing explicitly, reintroducing complexity despite Julia’s high-level model. Asynchronous Many-Task (AMT) programming models provide a higher-level abstraction by decomposing computations into fine-grained tasks connected via dependencies, with scheduling delegated to a runtime system. This is particularly attractive for irregular or dynamic workloads, where parallelism is difficult to express through static loopbased or message-passing approaches. In the Julia ecosystem, Dagger.jl [9] implements this idea by representing computations as directed acyclic graphs that can be scheduled across local and distributed workers. In parallel, LLMs have become powerful tools for natural language understanding, reasoning, and code generation, drawing growing attention in the HPC community. Available models include OpenAI’s GPT family[10], Anthropic’s Claude[11], and open-weight alternatives such as Qwen [12], with new versions appearing rapidly. Recent work has evaluated LLMs on HPC-specific challenges [13,14,15,16], but mostly in single-shot prompting rather than agentic settings, where an LLM plans, writes, executes, and refines code via tools. One effort to systematize such evaluations across rapidly evolving models is ParEval [14], which benchmarks LLMs on parallel code generation. Its results show strong performance on boilerplate and common patterns, but weaknesses in concurrency control, synchronization, and performance optimization, indicating that LLMs currently complement rather than replace expert knowledge. In this work, we study LLM-generated parallel Julia code in an agentic setting, focusing on task-based execution with Dagger.jl. Building on ParEval’s prompt design, we adapt its evaluation to this setting. We compare two commercial models, OpenAI GPT-5.5[10] and Anthropic Claude Opus 4.7[11], with the open-weight Qwen3-Coder-Next (35B active parameters) [12], covering tradeoffs in capability, deployment flexibility, and kernel generation time. All models are driven by an OpenCode-based agent extended with a Julia-documentation MCP server. We consider three representative algorithmic problems with distinct
A Study of Agentic AI-Generated Julia Code on Supercomputers
3
parallel structures, namely π approximation via numerical integration (embarrassingly parallel), tiled general matrix multiplication (regular structured), and tiled Cholesky decomposition (irregular dependencies), and compare agent-generated implementations across Base.Threads, MPI.jl, and Dagger.jl. All experiments run on the Otus supercomputer, using up to 192 cores on one node for shared memory and up to 384 cores across two nodes for distributed memory. Our results show that the two commercial models reliably produce executable, reasonably scaling code for the established Base.Threads and MPI.jl baselines, while the open-weight model frequently fails at the problem sizes required for meaningful scaling. Across all models, Dagger.jl implementations reveal recurring weaknesses in task dependencies, granularity, and scheduling, with several runs failing at higher core counts due to deadlocks or out-of-memory errors. Section 2 presents related work, Section 3 the experimental setup, and Section 4 the results. Section 5 discusses the findings, and Section 6 concludes.
2
Related Work
Generating parallel code for HPC using LLMs is challenging, as correctness alone is insufficient; performance and efficiency are equally critical [17]. Domainspecific approaches such as HPC-GPT [18], MonoCoder [19], LASSI [20], and CodeRosetta [21] investigate integrating LLMs into HPC workflows to improve productivity, support domain-specific programming, and enable code translation. However, their effectiveness remains inconsistent for complex parallel workloads. Recent work examining LLM capabilities across parallel programming frameworks such as MPI, OpenMP, CUDA, and Kokkos identifies a pronounced parallelism gap: while models perform well on serial tasks, they frequently struggle with data dependencies, synchronization, and thread safety [14]. Broader multilanguage evaluations further show that LLMs can generate functional code and accompanying unit tests for relatively simple scientific applications, but their performance degrades for parallel and distributed workloads [16]. Similarly, studies on performance optimization find that current models can identify obvious antipatterns yet often fail to apply architecture-specific optimizations that require deeper hardware awareness [22]. At the same time, specialized models such as CUDA-LLM [23] demonstrate that high efficiency can be achieved when models are tailored to specific architectures, particularly for GPU kernel generation. Within this broader landscape, task-based parallelization in Julia remains underexplored. Evaluations indicate that LLMs can generate usable code for mature Julia programming models such as Base.Threads and CUDA.jl, but confirm that parallel code generation is substantially more challenging than serial scientific programming [15,24]. Julia-native task runtimes are therefore natural targets for LLM-assisted parallelization, as their APIs expose dependency information explicitly, e.g., through Dagger’s Datadeps annotations, while remaining close to the mathematical code users intend to write [9,24]. To the best of our knowledge, no prior work has systematically studied LLM-based agents for task-based parallelization in Julia, particularly for performance-oriented refinement.
4
Bantel et al.
3
Experimental Setup
We describe the algorithmic problems (Section 3.1), the evaluated LLMs and codegeneration environment (Section 3.2), and our two-stage workflow (Section 3.3). 3.1
Algorithmic Problems
Our initial objective was to reproduce the full ParEval benchmark [14] and adapt its tasks and prompts for parallel, task-based Julia, replacing manual prompting with LLM-based agents that also test and optimize the generated code themselves. However, a preliminary evaluation revealed two limitations. First, ParEval’s prompts are primarily designed for completion-based systems, such as GitHub Copilot-style autocompletion, rather than for agentic systems that iteratively plan, generate, and revise code. Second, applying the full 60experiment benchmark to agentic systems would require substantial time and token budgets, and extensive manual analysis to assess correctness, scalability, and code quality. We therefore restrict our evaluation to the following three representative algorithmic problems. Approximation of π via numerical integration is an embarrassingly parallel problem. The integration domain can be divided into independent subintervals, where each task computes a partial result that is aggregated to obtain the final approximation. Its computational complexity is O(1/d), where d is the step width of the numerical integration. Since no data movement is required beyond the final aggregation, the problem is strictly compute-bound. The problem description provided to the agent is: Approximate pi using the numerical integration f (x) = 4/(1 + x2 ) from 0 to R1 1: 0 4/(1 + x2 )dx = 4 ∗ arctan(1) = 4 ∗ π/4 = π. The function returns the approximated value. Matrix-Matrix Multiplication (GEMM) is a regular, structured workload with predictable communication patterns and substantial arithmetic intensity per task. Its tiled formulation maps naturally to task-based execution, and the use of optimized BLAS kernels is explicitly permitted for the agents. Multiply the matrix A by the matrix B. Store the results in the matrix C. A is an MxK matrix, B is a KxN matrix, and C is an MxN matrix. The matrices are stored in column-major order. Cholesky decomposition is a standard benchmark for task-based runtime systems. In contrast to embarrassingly parallel workloads, it exhibits structured task dependencies and therefore serves as a fitting workload for task-based scheduling frameworks. Typical parallel implementations follow a block-based decomposition, in which the factorization is divided into smaller matrix blocks that can be processed concurrently while respecting data dependencies. We therefore expect task-based frameworks such as Dagger.jl to be well suited to
A Study of Agentic AI-Generated Julia Code on Supercomputers
5
this workload, as their scheduling mechanisms can exploit task parallelism while managing inter-task dependencies. Factorize the symmetric positive definite matrix A into A = L * LT where L is a lower triangular matrix. The lower triangular part (including the diagonal) contains L. The upper triangular part can be left unchanged or ignored. A is an NxN matrix stored in column-major order. Use POTRF, TRSM, SYRK, and GEMM block-based. 3.2
LLMs and Code Generation Environment
We evaluate three LLMs, each assessed independently under identical conditions for a controlled comparison. ChatGPT-5.5 (GPT) [10] is a proprietary general-purpose LLM provided by OpenAI. Since the model is commercial, local deployment and direct inspection of weights or architectural details are not possible. We access it through OpenAI’s ChatGPT Plus subscription plan. Claude Opus 4.7 (Opus) [11] is a proprietary LLM developed by Anthropic. The model is not open source, so its weights and architectural details cannot be inspected or deployed locally. We access it through Anthropic’s Claude Pro subscription plan. Qwen3-Coder-Next (Qwen) [12] is an open-weight code-specialized LLM developed by Alibaba’s Qwen team. It is designed for agentic coding workflows, including program synthesis, debugging, code modification, and tool-based software development. Unlike the proprietary models, it can be deployed locally without subscription or API fees, improving reproducibility and cost efficiency. We use the 35B active-parameter version, which we run locally on three NVIDIA A100 40 GB GPUs. OpenCode is an open-source terminal-based coding agent that enables LLM-based software development directly within a local project environment. It allows LLMs to inspect files, modify source code, execute commands, and interact with external tools. We use OpenCode 1.14.20 and extend it with a Juliadocumentation MCP server [25] that provides structured access to the official Julia documentation, enabling the agent to retrieve relevant API references and usage patterns during code generation. In addition, we use a dedicated system prompt for Julia-based HPC tasks. The prompt defines the agent as an expert in Julia HPC and emphasizes parallel programming, performance optimization, profiling, memory management, and efficient program design. The agent is explicitly instructed to use juliadoc for documentation retrieval and to validate generated implementations. Our Sandbox is an isolated Ubuntu 24.04 environment in which all code generation and execution are performed. It is preconfigured with Julia, MPI, and Slurm to emulate realistic HPC conditions, allowing the agent to autonomously generate, execute, and validate parallel and distributed workloads in a consistent software environment. The isolated setup improves reproducibility and prevents side effects between experiments.
6
3.3
Bantel et al.
Code Generation
To evaluate LLM-based agents on parallel task-based Julia code, we compare Dagger.jl with the established approaches Base.Threads and MPI.jl. Dagger.jl is particularly relevant in this context because it supports task-graphbased execution and can be used in shared-memory, distributed-memory, or hybrid configurations. For comparability, we distinguish between shared- and distributed-memory settings, as shared-memory Dagger.jl implementations may rely on different data structures and execution mechanisms than distributed ones. The shared-memory Dagger.jl variant is compared against Base.Threads, while the distributed variant uses Distributed.jl and is compared against MPI.jl. Hybrid implementations are excluded from the evaluation. We analyze how effectively agents implement these algorithms across the programming models, comparing correctness, scalability, and performance. A key design consideration is the trade-off between constraint enforcement and model flexibility: strict constraints improve reproducibility and reduce invalid implementations, but may limit solution strategies, while less restrictive prompts allow more independent decisions at the cost of correctness, efficiency, and comparability. To balance both, our workflow uses two sequential stages in the same OpenCode session: test generation defines the interface and validation, and kernel generation implements the algorithm based on the generated test program. In the first stage, a test program is generated which provides the execution harness for the actual kernel. It defines how the kernel is invoked, generates and prepares the input data sequentially, collects the kernel output, and validates it using correctness checks. This setup separates test orchestration from the measured kernel execution while still leaving the concrete data-passing strategy between the test program and the kernel implementation. Shared-memory implementations can usually operate on standard Julia arrays, whereas distributed-memory variants may require communication, data copies, or distributed abstractions such as Dagger.jl’s DArray. Accordingly, the kernel interface is not fixed during test generation; instead, the LLM may define an interface suitable for the respective programming model and transform or distribute the input data before invoking the kernel. For comparability, only the outer test-function interface is fixed: each test receives the number of experiments, the problem size, and the number of warmup runs, with an additional block-size parameter for task-based implementations to control decomposition granularity. In the second stage, the agent implements the kernel in the same OpenCode session, using the generated test program as the interface, execution harness, and correctness checks. Unlike the test-generation prompt, the kernel-generation prompt provides guidance for efficient, comparable implementations: external dependencies that directly solve the target problem are disallowed, while BLAS calls are permitted. The agent is then instructed to test the implementation with the generated test program and kernel, run scaling experiments with appropriate problem sizes, and refine the code based on the results. Correctness is assessed using the generated test program, while scaling experiments encourage performance-oriented improvements.
A Study of Agentic AI-Generated Julia Code on Supercomputers
4
7
Experimental Results
To evaluate the ability of different LLMs to generate parallel Julia code and to compare the established parallelization approaches with task-based Dagger.jl, we analyze both the generation process and the performance of the generated programs. First, we examine the generation logs to identify differences between models, algorithmic problems, and Julia frameworks, including the time required for kernel generation. Second, we execute the generated programs on the Otus supercomputer and evaluate their strong-scaling performance. The dataset containing raw results, generation logs, and prompts is available at [26]. Each model–problem–framework combination is generated only once, and although generation is non-deterministic, the agentic workflow’s iterative testing and refinement mitigates this variability. The Otus supercomputer comprises 636 compute nodes, each with two AMD EPYC 9655 “Turin” CPUs, each with 96 cores at 2.6–4.5 GHz, for a total of 192 cores and 768 GiB of main memory per node. Shared-memory experiments run on a single node with up to 192 cores, whereas distributed-memory experiments run on two nodes with up to 384 cores. All experiments use Julia 1.12.6; Dagger.jl implementations use Dagger.jl 0.19.4, and MPI implementations use MPI.jl 0.20.26 with Open MPI 5.0.7. All runs are submitted as Slurm batch jobs and Slurm handles node allocation, process placement, and CPU binding. In distributed runs, processes are spread evenly across nodes. OpenBLAS is restricted to a single thread throughout all experiments to prevent nested parallelism from influencing the measured scaling behavior. Each experiment is repeated five times, and the reported results are averaged across these runs. 4.1
Agentic AI for Julia
Across the logs of LLM-generated Julia implementations, agents typically start with API exploration, then perform correctness testing, performance experiments, and iterative optimization. Correctness is usually tested before evaluating performance across thread or process counts, problem sizes, and, where applicable, block sizes. The logs mainly differ in structure: GPT and Opus usually plan and proceed step by step, whereas Qwen more often uses direct trial-and-error and includes larger portions of generated or modified code. The models also differ in their use of Dagger.jl abstractions. GPT often uses future-oriented constructs with deferred execution and explicit result resolution, while Opus more often expresses computations through explicit task dependencies or DAG-like execution patterns. Qwen spans a wider range of abstraction levels, from high-level Dagger.jl interfaces down to lower-level or internal API interactions. At the same time, even GPT and Opus often prefer established implementation strategies over less familiar distributed data abstractions. For example, when the prompt does not explicitly require Dagger.jl’s distributed data structures, GPT may recognize DArray as suitable but still choose manual data transfers to worker processes. Since such transfers can increase memory pressure in distributed-memory settings, we adapted the distributed Dagger.jl prompts
8
Bantel et al.
to explicitly require its distributed data structures. Some GPT logs also attempt explicit task placement, using Dagger.Scope annotations or custom round-robin schemes. The resulting error patterns reflect these abstraction choices: GPT and Opus logs mainly show API adjustments, type corrections, data-distribution changes, and task-dependency fixes before reaching executable implementations. In contrast, Qwen logs show more API errors, type errors, runtime crashes, and timeouts. In several Qwen cases, implementations are adjusted by reducing the problem size or modifying the test configuration. Performance-related decisions are mostly empirical across all LLMs. The logs contain limited evidence of an explicit model relating problem size, chunk size, worker count, scheduler overhead, and expected runtime. Instead, the agents run benchmark experiments and adapt the implementation based on observed results. Opus tends to test more extensively and often repeats correctness checks after performance experiments. GPT more frequently accepts limited local scaling results or notes that local results may not reflect larger-cluster behavior. Qwen often uses comparatively small problem sizes despite prompt instructions to choose sizes with sufficiently long serial runtime, causing several measurements to be dominated by overhead. Figure 1 shows that kernel generation time varies substantially across algorithmic problems. The π kernel has the shortest generation times in most configurations, while the Cholesky kernel generally requires the longest, consistent with its more complex numerical dependencies and blocked execution structure. Figure 1a, which averages over problems, shows lower generation times for Base.Threads than for the other frameworks, consistent with its comparatively direct fork-join structure. Distributed-memory frameworks require additional decisions about data distribution, communication, task placement, and process coordination. Figure 1b, averaging over frameworks, further shows that variation between algorithmic problems is larger than variation between frameworks. Differences between LLMs are also visible in the timing results. Opus shows increased generation time for Cholesky, consistent with the more extensive testing and repeated validation observed in its logs, and Qwen shows a distinct timing profile compared to GPT and Opus. However, these timings must be interpreted alongside correctness results, since shorter generation times do not necessarily imply successful or correct implementations. Because generation runs against hosted APIs and a local deployment, the measured times are affected by uncontrolled factors such as rate-limiting, token-throttling, and queueing. Thus, we treat kernel generation time as an indicative proxy rather than a precise measurement. 4.2
Performance Evaluation
The generated test and kernel files enable agents to detect and correct common issues in LLM-generated code, including missing imports, interface mismatches, and simple runtime errors. For small problem sizes and up to four threads or processes, all agents produced executable implementations for all algorithmic
A Study of Agentic AI-Generated Julia Code on Supercomputers 4000
Dagger
Threads
Dist Dagger
GEMM
Cholesky
4000
Generation Time [s]
Generation Time [s]
3000
3000
2000
2000
1000 0
PI
MPI
9
1000
GPT
Opus
Qwen
(a) Averaged over different Problems
0
GPT
Opus
Qwen
(b) Averaged over Frameworks
Figure 1: Kernel generation time (code generation, testing, and optimization)
problems. However, correctness differs between models: GPT and Opus consistently generated implementations that passed the correctness checks for all problems, whereas Qwen implementations were executable in several cases but produced incorrect results in others. In later experiments, some implementations failed beyond certain problem sizes or thread/process counts, with failure modes including deadlocks, oversubscription, out-of-memory errors, crashes, and timeouts. This affected Qwen in particular: for problem sizes suitable for scaling experiments and successfully used with GPT and Opus, its implementations failed for all problems, often already at small thread or process counts. Therefore, the following performance evaluation plots include only GPT and Opus. Missing data points in the performance figures indicate configurations where the corresponding implementation failed at the specified number of threads or processes. Numerical π Integration. Figure 2a shows the shared-memory results. Base.Threads is faster than Dagger.jl due to lower scheduling overhead. Runtime decreases with more threads, but scaling efficiency drops at higher parallelism. Both LLMs achieve comparable performance and near-ideal scaling. A noticeable discontinuity occurs in the Dagger.jl runtime between one and two threads. This is caused by Dagger.jl reserving one interactive thread for task scheduling, leaving only num_threads − 1 threads for worker execution. As a result, the effective worker count is shifted relative to the Base.Threads implementation. This behavior applies to all shared-memory Dagger.jl implementations discussed in the following. As explained in Section 3.3, the generated Dagger.jl implementations differ between shared- and distributed-memory settings, so their runtimes may differ as well. Figure 2b presents the distributed-memory results. For this embarrassingly parallel workload, MPI achieves near-ideal scaling, since only a final reduction is required. Distributed Dagger.jl also scales well initially, but task-management overhead reduces efficiency at higher process counts. Both LLMs generate reduction strategies that avoid naive centralized communication and therefore achieve good distributed scaling behavior. Qwen generated a distributed Dagger.jl implementation based on a DArray decomposition, which introduces substantial communication and allocation overhead. Therefore, its runtime is orders of mag-
10
Bantel et al. GPT Dagger GPT Threads
GPT Dist Dagger GPT MPI
Opus Dagger Opus Threads
102
Opus Dist Dagger Opus MPI
Runtime [s]
Runtime [s]
103 102
101
101 100
20
21
22
23 24 25 # threads/workers
26
27
(a) Runtimes in seconds on one node
28
20
21
22
23 24 25 26 # processes/workers
27
28
29
(b) Runtimes in seconds on two nodes
Figure 2: Strong scaling of the π approximation with 241 intervals for shared(left) and 243 for distributed-memory (right).
nitude higher than that of GPT and Opus. The runtime difference between the GPT and Opus MPI implementations is caused by differences in the generated kernel code. Opus relies on a single SIMD-annotated loop, whereas GPT uses manual loop unrolling, which performs worse in this case. General Matrix Multiplication. A tiled GEMM kernel is, similar to the π benchmark, a highly structured workload with regular communication patterns. At the same time, it is in principle well suited to task-based execution, since the multiplication can be decomposed into tile-level tasks with explicit data dependencies and substantial per-task computation. With sufficiently large tiles, this structure provides enough arithmetic intensity to amortize scheduling overhead. As shown in Figure 3a, all implementations initially exhibit comparable performance. However, as thread count increases, the Opus Dagger.jl implementation degrades more rapidly and fails to complete beyond 32 threads, indicating an unresolved scheduling dependency. This suggests that the generated synchronization and dependency structure does not scale reliably to larger task graphs, despite the algorithmic structure’s suitability for task-based parallelism. Inspection of the generated implementations indicates that this behavior is largely related to scheduling overhead and dependency management. The Opus Dagger.jl implementation relies on spawn_datadeps(), whereas GPT uses a simpler future-based approach in which asynchronously spawned tile computations are stored as task handles and synchronized explicitly after task creation. Since tiled GEMM requires coordination both before computation, to distribute matrix tiles, and after computation, to collect partial results, the additional dependency tracking introduced by spawn_datadeps() increases overhead and appears to limit scalability in the generated implementation. The distributed-memory results in Figure 3b show an even clearer separation between programming models. The MPI-based implementations scale reasonably well with increasing process counts, whereas the distributed Dagger.jl
A Study of Agentic AI-Generated Julia Code on Supercomputers GPT Dagger GPT Threads
102
Opus Dagger Opus Threads
GPT Dist Dagger GPT MPI
11
Opus Dist Dagger Opus MPI
102 Runtime [s]
Runtime [s]
101
100 20
21
22
23 24 25 # threads/workers
26
27
(a) Runtimes in seconds on one node
28
101 100
10 1
20
21
22
23 24 25 26 # processes/workers
27
28
29
(b) Runtimes in seconds on two nodes
Figure 3: Strong scaling of the tiled GEMM for 214 × 214 matrices with tile size 2048 in shared- (left) and distributed-memory (right) settings.
implementations exhibit little to no scalability. Both GPT and Opus again make extensive use of spawn_datadeps(). For this regular workload, where communication and synchronization patterns can be expressed relatively directly, the generated dependency-based Dagger.jl implementations introduce more coordination overhead than the MPI implementations, even though the latter use a naive column-wise output decomposition rather than established, optimized GEMM algorithms. Cholesky Decomposition. The tiled Cholesky decomposition has an irregular dependency structure and is therefore a common benchmark for task-based runtime systems. For the shared-memory implementations, Figure 4a shows behavior similar to the Opus GEMM implementation: the GPT and Opus Dagger.jl variants encounter deadlocks at 24 and 32 tasks, respectively. In contrast, the forkjoin-based Base.Threads implementations run up to 192 threads, with scaling saturating around 64 threads. Inspection of the shared-memory Dagger.jl implementations shows that both models express the algorithm through futures but organize dependency tracking differently. Opus wraps the decomposition in a single spawn_datadeps() region and encodes dependencies explicitly using In and InOut annotations, relying on a globally managed dependency scope. GPT instead stores the futures returned by @spawn directly in the chunk array and uses them to construct dependencies implicitly, resulting in a more local scheduling structure without an additional global dependency region. The same distinction appears in the distributed-memory variants: Opus again uses spawn_datadeps() with annotated accesses, whereas GPT relies on futurebased composition. The distributed MPI and Dagger.jl implementations show comparable performance at low process or worker counts, with only limited overall scaling. At higher parallelism levels, the Dagger.jl implementations exhibit smoother scaling, whereas MPI shows more irregular behavior.
12
Bantel et al. GPT Dagger GPT Threads
GPT Dist Dagger GPT MPI
Opus Dagger Opus Threads
Opus Dist Dagger Opus MPI
103
102 Runtime [s]
Runtime [s]
102
101
101 100
100 20
21
22
23 24 25 # threads/workers
26
27
(a) Runtimes in seconds on one node
28
20
21
22
23 24 25 26 # processes/workers
27
28
29
(b) Runtimes in seconds on two nodes
Figure 4: Strong scaling of the tiled Cholesky decomposition for a 214 × 214 matrix with tile size 2048 in shared- (left) and distributed-memory (right) settings.
5
Discussion
Our experimental results in Section 4 show that agentic code generation can detect and correct many common issues in LLM-generated code, such as missing imports, interface mismatches, and simple runtime errors, since programs are executed and tested by the agent during generation. As a result, agents generally produce executable parallel Julia code for small problem sizes and low degrees of parallelism. However, in an HPC context, correctness is only a necessary condition; robustness, scalability, and performance on supercomputers matter equally. Several implementations that pass small correctness tests fail when problem size or parallelism is increased, for example, due to deadlocks, oversubscription, timeouts, or out-of-memory errors. This shows that the generated parallel code must be evaluated under scaling conditions, not only through functional tests. The model comparison further shows that generation logs are essential for interpreting the results. GPT and Opus follow a more structured development process involving planning, correctness testing, and performance experiments, with Opus testing particularly extensively, whereas Qwen shows less structured behavior and more often uses problem sizes that are too small for meaningful scaling experiments. Overall, the implementations generated by GPT and Opus perform similarly, but GPT achieves better performance and scalability in most Dagger.jl cases. A possible explanation is that Opus spends more effort optimizing and testing code on the local execution system, which may improve performance in the observed test environment but can lead to implementations that are overly tuned to it and therefore transfer less effectively to supercomputers. This points to a broader limitation: during generation, agents lack access to distributed-memory execution, realistic process placement, node-level memory constraints, and large core counts, making it difficult to predict performance on HPC systems, especially for distributed and task-based implementations in which data distribution, communication, task granularity, synchronization, and scheduling overhead are decisive. Ideally, agents would optimize directly on the
A Study of Agentic AI-Generated Julia Code on Supercomputers
13
target system, where these effects become observable. In practice, however, this is difficult because supercomputers are accessed through batch schedulers, offer limited interactive feedback, and repeated agent-driven benchmark runs would consume substantial shared resources. If this barrier were lowered, for example through agent-accessible batch submission, agents could optimize directly on the target system, likely narrowing the observed gap between locally tuned and supercomputer behavior, especially for distributed and task-based implementations. The trade-off then becomes economic, weighing the cost of repeated on-system runs against the resulting gains. The results also reveal a tension between task-based programming models and the agents’ implementation strategies. Although frameworks such as Dagger.jl delegate scheduling and data movement to the runtime, generated implementations often reintroduce manual control through explicit task placement, custom scheduling, or manual data transfer. Agents also favor familiar data structures and execution patterns over framework-specific abstractions. Thus, the results do not show that task-based execution is unsuitable for these workloads, but that agents do not always exploit the runtime effectively. The prompt adaptations used in this study demonstrate that explicit constraints are necessary for fair comparisons across frameworks. If prompts are too permissive, agents may use a framework syntactically correctly without adhering to its intended programming model, for example by manually moving data in distributed Dagger.jl implementations instead of using distributed data abstractions. If prompts are too restrictive, they may suppress framework-specific solution strategies. Prompt design therefore requires a trade-off between comparability, reproducibility, and implementation flexibility. The models differ in cost, though our data supports only a qualitative assessment: the proprietary GPT and Opus incur subscription fees, whereas the open-weight Qwen runs locally without per-token cost but requires dedicated GPU hardware. Kernel generation time (Figure 1) is the only cost dimension we measure directly. A full cost–time–value analysis would additionally require token counts and human guidance time, which we leave to future work. Finally, two limitations stand out. Our comparison lacks a non-Julia ground truth: Base.Threads and MPI.jl act as within-Julia baselines, but a hand-written or vendor-optimized reference would be needed to separate framework-inherent inefficiencies from those of the generated code. We also do not trace implementation choices to training sources. The full logs and prompts are available [26].
6
Conclusion
This work evaluated LLM-based agents for generating parallel Julia code with Base.Threads, MPI.jl, and Dagger.jl. The results show that Agentic AI can produce executable implementations and automatically correct many typical coding errors through iterative testing. However, small-scale correctness does not guarantee robustness or scalability in an HPC setting. Several implementations fail at larger problem sizes or higher thread and process counts, exposing deadlocks,
14
Bantel et al.
oversubscription, or out-of-memory errors. This highlights the need to evaluate and optimize generated HPC implementations under realistic conditions. Overall, the agents handle established models such as Base.Threads and MPI.jl more reliably than task-based Dagger.jl implementations, which require more complex decisions about task granularity, data distribution, dependencies, and scheduling. Thus, LLM-based agents are promising for producing initial parallel code, but generating robust, performance-aware implementations for largescale HPC systems remains an open challenge. Future work should investigate how agentic optimization can be coupled more tightly to the target system, for example through batch-aware feedback loops and resource-conscious benchmarking. Acknowledgments. This research was supported by Advantest as part of the Graduate School “Intelligent Methods for Test and Reliability” (GS-IMTR) at University of Stuttgart. This research was partially funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under project number 558599020. The authors gratefully acknowledge the computing time made available to them on the high-performance computer Otus at the NHR Center Paderborn Center for Parallel Computing (PC2). This center is jointly supported by the Federal Ministry of Research, Technology and Space and the state governments participating in the National High-Performance Computing (NHR) joint funding program. Disclosure of Interests. The authors have no competing interests to declare that are relevant to the content of this article.
References 1. Eschle, J., Gál, T., Giordano, M., et al.: Potential of the Julia Programming Language for High Energy Physics Computing. Computing and Software for Big Science (2023), 10.1007/s41781-023-00104-x 2. Bezanson, J., Edelman, A., Karpinski, S., et al.: Julia: A Fresh Approach to Numerical Computing. SIAM Review (2017), 10.1137/141000671 3. Bezanson, J., Chen, J., Chung, B., et al.: Julia: Dynamism and Performance Reconciled by Design. ACM on Programming Languages (2018), 10.1145/3276490 4. Stewart, G.A., Moreno Briceño, A., Gras, P., et al.: Julia in HEP. In: EPJ Web of Conferences. EDP Sciences (2025), 10.1051/epjconf/202533701266 5. Byrne, S., Wilcox, L.C. and Churavy, V.: MPI.jl: Julia Bindings for the Message Passing Interface. JuliaCon Conferences (2021), 10.21105/jcon.00068 6. Lin, W.C. and McIntosh-Smith, S.: Comparing Julia to Performance Portable Parallel Programming Models for HPC. In: International Workshop on Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS). IEEE (2021), 10.1109/PMBS54543.2021.00016 7. Godoy, W.F., Valero-Lara, P., Dettling, T.E., et al.: Evaluating Performance and Portability of High-Level Programming Models: Julia, Python/Numba, and Kokkos on Exascale Nodes. In: International Parallel and Distributed Processing Symposium Workshops (IPDPSW). IEEE (2023), 10.1109/IPDPSW59300.2023.00068 8. Hunold, S. and Steiner, S.: Benchmarking Julia’s Communication Performance: Is Julia HPC Ready or Full HPC? In: International Workshop on Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS). IEEE (2020), 10.1109/PMBS51919.2020.00008
A Study of Agentic AI-Generated Julia Code on Supercomputers
15
9. Alomairy, R., Tome, F., Samaroo, J., et al.: Dynamic Task Scheduling with Data Dependency Awareness Using Julia. In: High Performance Extreme Computing Conference (HPEC). IEEE (2024), 10.1109/HPEC62836.2024.10938467 10. OpenAI: GPT-5.5 (2026), https://openai.com/ 11. Anthropic: Claude (2026), https://www.anthropic.com/claude 12. Alibaba Cloud and Qwen Team: Qwen (2026), https://qwenlm.github.io/ 13. Bantel, L., Strack, M., Strack, A., et al.: From Prompts to Performance: Evaluating LLMs for Task-based Parallel Code Generation. In: Workshop on Asynchronous Many-Task Systems and Applications (WAMTA). Springer (2026), to appear; preprint available on arXiv, 10.48550/arXiv.2602.22240 14. Nichols, D., Davis, J.H., Xie, Z., et al.: Can Large Language Models Write Parallel Code? In: International Symposium on High-Performance Parallel and Distributed Computing (HPDC). ACM (2024), 10.1145/3625549.3658689 15. Diehl, P., Nader, N., Brandt, S., et al.: Evaluating AI-Generated Code for C++, Fortran, Go, Java, Julia, Matlab, Python, R, and Rust. In: International European Conference on Parallel and Distributed Computing Workshops (Euro-Par). Springer (2025), 10.1007/978-3-031-90200-0_20 16. Diehl, P., Nader, N., Moraru, M., et al.: LLM Benchmarking with LLaMA2: Evaluating Code Development Performance Across Multiple Programming Languages. Journal of Machine Learning for Modeling and Computing (2025), 10.1615/JMachLearnModelComput.2025058957 17. Ljaljevic, S., Jorba, J. and Iserte, S.: Exploring the Role of Large Language Models in High-Performance Computing Programming: A Survey. Future Generation Computer Systems (2026), 10.1016/j.future.2026.108618 18. Ding, X., Chen, L., Emani, M., et al.: HPC-GPT: Integrating Large Language Model for High-Performance Computing. In: Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis (SC-W). ACM (2023), 10.1145/3624062.3624172 19. Kadosh, T., Hasabnis, N., Vo, V.A., et al.: MonoCoder: Domain-Specific Code Language Model for HPC Codes and Tasks. In: High Performance Extreme Computing Conference (HPEC). IEEE (2024), 10.1109/HPEC62836.2024.10938441 20. Dearing, M.T., Tao, Y., Wu, X., et al.: LASSI: An LLM-Based Automated SelfCorrecting Pipeline for Translating Parallel Scientific Codes. In: International Conference on Cluster Computing Workshops (CLUSTER Workshops). IEEE (2024), 10.1109/CLUSTERWorkshops61563.2024.00029 21. TehraniJamsaz, A., Bhattacharjee, A., Chen, L., et al.: CodeRosetta: Pushing the Boundaries of Unsupervised Code Translation for Parallel Programming. In: Advances in Neural Information Processing Systems (NeurIPS) (2024), 10.48550/ arXiv.2410.20527 22. Cui, B., Ramesh, T., Hernandez, O., et al.: Do Large Language Models Understand Performance Optimization? arXiv (2025), 10.48550/arXiv.2503.13772 23. Chen, W., Zhu, J., Fan, Q., et al.: CUDA-LLM: LLMs Can Write Efficient CUDA Kernels. arXiv (2025), 10.48550/arXiv.2506.09092 24. Godoy, W.F., Valero-Lara, P., Teranishi, K., et al.: Large Language Model Evaluation for High-Performance Computing Software Development. Concurrency and Computation: Practice and Experience (2024), 10.1002/cpe.8269 25. Plavin, A.: julia-mcp: MCP server for persistent Julia sessions (2026), https: //github.com/aplavin/julia-mcp 26. Bantel, L., Roth, A.L., Posner, J., et al.: Agentic AI-Generated Julia Code on Supercomputers (2026), https://github.com/BaLinuss/ Agentic-AI-Generated-Julia-Code-on-Supercomputers