Real-time Optimization of Simulation and Data Processing Pipelines for Experiments on Exascale Computing Platforms

arXiv:2610.02498v1 [cs.DC] 1 Oct 2026

Real-Time Optimization of Simulation and Data Processing Pipelines for Experiments on Exascale Computing Platforms Thomas Wester

Harikrishna Tummalapalli

Christine M. Simpson

Enrico Fermi Institute University of Chicago Chicago, IL, USA 0000-0001-6668-7595

Argonne Leadership Computing Facility Argonne National Laboratory Lemont, IL, USA 0000-0002-1055-3130

Argonne Leadership Computing Facility Argonne National Laboratory Lemont, IL, USA 0000-0001-9985-1814

Abstract—Current and next-generation large-scale experiments, especially in high-energy physics and cosmology, involve increasing volumes of data that require exascale computing power to analyze and model. Exascale machines present challenges for workflow scaling and efficient resource usage. We propose a multi-stage task scheduling approach for optimizing resource usage for these workflows. We demonstrate task scheduling for an example Monte Carlo simulation pipeline from the ShortBaseline Near Detector (SBND), and present theoretical and numerical tools for modeling task scheduling in these workflows that can be used to identify optimal scheduling strategies and choose resources for a given problem size. Index Terms—workflow, scheduling

I. I NTRODUCTION Current and next-generation scientific experiments will collect petabytes of raw instrument data [1]. Data-intensive analysis patterns have been identified as a key pillar of the workloads that national computational facilities will need to support to realize the scientific impact of many upcoming experiments [2], and exascale high-performance computing (HPC) systems provide the necessary computational resources to process this volume of data on the timescales necessary for typical data analysis cycles. However, extreme-scale systems are often many times bigger than resources experimentalists are accustomed to using near their experiments and therefore pose a challenge in porting and optimizing pipelines. Experimental data are typically organized into discrete units, e.g., events or objects, which must be processed through staged computational pipelines in order to extract high-level quantities of interest [3]. Optimal resource utilization of these computational pipelines is often linked to finding the optimal schedule. Pipeline scheduling has been extensively studied [4]. However, these studies often consider deterministic workloads, where the task durations are predictable. In practice, advance knowledge is not always available: real experimental data and Monte Carlo simulations often contain statistical fluctuations that affect the run times of each processing stage [5]. For example, a particle detector may observe a random number of particles in a given event, each of which must be processed through a pattern recognition algorithm. Static scheduling

SC26 Workshops, November 15-20, 2026, Chicago, Illinois, USA 979-8-3195-1221-5/26/$31.00 ©2026 IEEE

algorithms often perform poorly on these stochastic workloads [6]. The problem is exacerbated by the heterogeneity of modern HPC systems. In this work, we show how a real-time task scheduling strategy can improve resource utilization and reduce the makespan, or wall time, for stochastic pipeline workflows on production HPC systems. Additionally, we provide theoretical and numerical tools to estimate makespan, resource utilization, and other performance metrics for general workflow directed acyclic graphs (DAGs) with random task durations. Our scheduling strategy divides computational pipelines into two stages: a short prediction stage which is used to estimate the run time of a second, longer stage. This strategy is broadly applicable to science domains which process variable amounts of data in multiple steps, and we show its direct application to a highenergy physics (HEP) Monte Carlo simulation workflow. The paper is structured as follows: we first describe relevant theoretical expressions used for optimally scheduling tasks with unknown run times on HPC systems. We then present a workflow simulator which can be used to accurately estimate makespan for different workflow configurations. Finally, we compare the theoretical and simulated results to a real HEP application for simulating neutrino interactions running on an HPC system. The consistency of the three approaches validates that a two-stage pipeline with a dynamic scheduling policy reduces the makespan and improves worker occupancy on HPC systems and provides a tool for exploring scaling and optimization of workloads as described in the last section. II. S CHEDULING T HEORY Optimal pipeline scheduling is well studied for particular cases in the literature [4], [7], and it is well known that minimizing the makespan is NP-hard for an arbitrary configuration of task run times [7]. Consequently, scheduling strategies are employed to assign computational tasks from pipelines to HPC resources in a particular order, e.g., based on task characteristics or at random. We consider submitting a workflow consisting of M tasks to an HPC job or “block” for processing. An HPC block

consists of N workers which will be in use until an unknown makespan, or wall time, T , at which point the final task finishes. The tasks are assigned to workers from a shared, perfect queue, i.e., when one worker finishes, it begins processing the next available task immediately unless there are no more tasks in the queue, at which point it is idle. The performance of the job can be measured by the makespan, T , and the idle time, I. Shorter T and smaller I represent better usage of the block. Workflows can be optimized trivially to minimize T by setting N = M . However, when the variance of task run times is non-zero, this configuration introduces a maximum inefficiency as measured by I, since many workers are idle while longer tasks finish. On the other extreme, setting N = 1 minimizes I but is impractical as the makespan T is maximized instead. HPC users must choose an acceptable T and I given the number of tasks and the run-time variance of their particular real-world application. To derive the relationship of T and I for different scheduling strategies, we assume tasks have random run times from a distribution q(t) with mean µ and variance σ 2 . We consider two scheduling scenarios: either the task run times are not known in advance of the block and are randomly assigned to workers, or the task run times are known in advance and can be ordered with a scheduling strategy. Since task run-time distributions may have long tails, it is useful to discuss these quantities as a function of the fraction of completed tasks, p, i.e., the T or I at which a fraction p of the work is done, which we denote as Tp and Ip . A. Unknown Run Times (Unordered)

tq(t) , µ

Substituting Eq. 1, we find Z 1 − Q(τ ) 1 ∞ q(t)dt = r(τ ) = , µ τ µ

(1)

1 This assumption should be considered optimistic and holds only when M ≫ N (in the limiting case where M = N , tdrain starts immediately so the run times are not uniformly distributed, and r(τ ) would be exactly the run time distribution, q(t), rather than a residual time distribution).

(3)

where Q(t) is the cumulative distribution function of q(t). The number of tasks completed within a time τ after tdrain is then given by integrating r, Z τ r(δ)dδ ≡ N R(τ ), (4) mdrain = N 0

where R(τ ) is the cumulative distribution of remaining run times. The makespan Tp can be expressed as a function of the completed fraction of tasks as ( Tp =

pM µ N mbulk µ + R−1 N

mdrain N

mbulk /M ≥ p, , mbulk /M < p,

(5)

where R−1 (x) is the inverse of R, mbulk ≡ M − N , and mdrain = pM − mbulk is the number of tasks that must complete during the draining phase to reach p. The idle time, Ip , can then be expressed as Ip = N Tp − pM µ.

This scheduling algorithm assumes no prior knowledge about the tasks, such that the tasks are taken from the queue at random. The block can be divided into two phases: a bulk phase when there are tasks in the queue, and a draining phase which starts at a time tdrain when the final task is removed from the queue, which is approximately (M − N )µ/N , i.e., the time at which M − N tasks have completed. To find Tp , we consider the number of tasks completed during each phase. In the bulk phase, before the queue is empty, all workers are busy. In the draining phase, which begins when the final task is assigned to a worker at tdrain , we denote the remaining run time of each task by τ . The distribution of remaining run times, r(τ ), can be obtained by assuming the remaining run times are uniformly distributed with respect to tdrain 1 . This assumption also has the consequence that the probability of a task being present on a worker is proportional to its run time, t, i.e., longer-duration tasks are more likely to be running at tdrain . Under this assumption, the remaining tasks have run times from a distribution given by qdrain (t) =

and the conditional probability of τ for a given task with run time t is 1/t. r(τ ) then considers all tasks with at least τ seconds remaining via the integral over all t, Z ∞ qdrain (t) dt. (2) r(τ ) = t τ

(6)

B. Known Run Times (Ordered) When run times are known, a standard heuristic-based scheduling policy is to prioritize either longest or shortest tasks. Since the residual duration of the block is proportional to the task durations at the end of the queue, scheduling the longest tasks first is the better strategy. In this case, we also have a bulk phase and draining phase. To find T and I, we need only consider the final task to be submitted to a worker. By construction, the final task is assigned to the first available worker after all other tasks are assigned, so it runs on the worker with the minimum sum of task run times. In the worstcase scenario, this last task starts at the moment all other tasks finish and extends T by its run time, giving an upper bound on the completion of the final task. When M ≫ N , this sets the makespan2 M −1

TLF ≤

1 X ti + t M . N i=1

(7)

Here, the ti are the run times of the other tasks, and tM is the final (shortest run time) task. From this, we also obtain the idle time, I, as 2 If the longest-running task is still running at the time the last task is assigned to a worker, its run time could extend the makespan instead. However, this case is atypical when M ≫ N .

(8)

The expressions derived in this section rely on idealized assumptions and are therefore optimistic estimates for the different scheduling algorithms. As we have seen, information about the task run times allows for a more favorable reduction in both the makespan and idle time, resulting in better utilization of HPC systems. III. S CHEDULING S IMULATOR Since the theoretical estimates of the makespan T and worker idle time I require simplifying assumptions, we developed a scheduling simulator to numerically estimate these quantities. The simulator extends the pipeline scheduling framework described in [4] to general DAGs and decomposes scheduling into a series of ordering and mapping steps. An architectural overview is shown in Fig. 1. The simulator consists of four main components: WorkflowModel, TaskOrderer, Mapper, and SystemModel. The WorkflowModel holds a set of Workflow objects, each representing a workflow as a DAG. Each task in the workflow (WorkflowTask) is characterized by a compute_cost and a data_size parameter, representing the computational cost and the output data size of the task, respectively. Once all dependencies of a task are resolved, it is marked as ready for execution. Ready tasks are inserted into a priority queue (TaskHeap) using a key defined by the chosen ordering policy. We provide several built-in policies, including first-in-first-out (FIFO), shortest-first, and longest-first. The ordered tasks are then mapped onto the compute system. The SystemModel consists of compute nodes (ComputeNode), each containing

ta

ta

0

We then find Ip = N Tp − Wbulk − Wdrain .

Ready Tasks

Da

i.e., the total run time of the longest M − N tasks, assuming the draining phase starts with N tasks remaining. Similarly, Z k/M Wdrain = M Q−1 (t)dt. (10)

Task 2

Task 3

TaskHeap

Ordered Tasks Slot 1

Comm

Slot 1 Slot 2

Mapped Tasks

Mapper

mm

Slot 2

Co

mm

N/M

Orderer

Task 1

Co

which only scales with the number of workers allocated during the block. Importantly, for known, randomly distributed run times scheduled according to a longest-first policy, tM decreases as M increases, since the final task is the shortest by definition. For a random distribution of task run times, this means I decreases for large M , which is the opposite trend of the unknown run-time case. We obtain Tp and Ip for this case by defining the number of tasks k required to reach the desired completion fraction p once the queue is empty. Since the tasks are run in order, T is extended by the run time of the k th task, which is, in general, found by evaluating Q−1 (k/M ). The idle time Ip is found by integrating the run times over the bulk and draining phases, Z 1 Q−1 (t)dt, (9) Wbulk = M

Workflow Model

Da

ILF ≤ N tM ,

Slot 1 Slot 2

System Model Scheduling Simulator

Fig. 1. Illustration of the workflow scheduling simulator.

a collection of compute slots (ComputeSlot). Each slot is characterized by a compute power parameter. The inter-node network topology is represented as a networkx [8] graph, where each edge carries a bandwidth parameter. The simulator drives these components through a discreteevent simulation (DES) loop, presented in Alg. 1. The loop maintains the current simulation time step, t, an event priority queue, Q, and a waiting priority queue, W. At initialization, all ready tasks are obtained from the workflow model, M, and inserted into W with priorities assigned by the orderer, O. These tasks are then scheduled onto the system, S, via the mapper, F. Within the S CHEDULE procedure, the completion time tc of each allocated task is computed and inserted into the event queue along with the task, τ , and its allocated slots, s. At each iteration, the earliest completion event is extracted from Q and the simulation clock is advanced to tc . The corresponding resources are released in S, and the task is marked as completed in M. Any newly ready successors are inserted into W and scheduled. The loop terminates when Q is empty. In the following sections we validate the simulator by comparing to a production workflow on an HPC system, and use it to investigate the performance of our scheduling strategy at different node counts and task duration variance. IV. H IGH -E NERGY P HYSICS W ORKFLOW HEP experiments typically consist of a detector which produces a data stream of triggered events. Events are a record of the state of the detector during readout windows, where the activity in the detector largely depends on the particle multiplicities and energies present during the window. Events undergo several pre-processing steps before being ultimately processed through pattern recognition algorithms to identify high-level features of interest such as particle tracks, mo-

Algorithm 1 Discrete-event workflow scheduling simulation Require: Workflow model M, system model S, mapper F, orderer O Ensure: Makespan T 1: t ← 0; Q ← ∅; W ← ∅ 2: for each τ ∈ M.R EADY TASKS() do 3: Insert τ into W with priority O(τ ) 4: end for 5: S CHEDULE(W, S, F, Q, t) 6: while Q ̸= ∅ do 7: (t, τ, s) ← E XTRACT M IN(Q) 8: S.R ELEASE(s) 9: M.M ARK C OMPLETED(τ, t) 10: for each v ∈ M.R EADY TASKS(τ ) do 11: Insert v into W with priority O(v) 12: end for 13: S CHEDULE(W, S, F, Q, t) 14: end while 15: return t

Fig. 3. Task time distribution and correlation between the two stages of the HEP workflow. The banding pattern is a feature of the discrete number of events generated in the first stage.

16: procedure S CHEDULE(W, S, F, Q, t) 17: if W = ∅ or S.FREE = 0 then return 18: end if 19: for each (τ, s) ∈ F(W, S) do 20: tc ← t + c(τ ) / p(s) 21: I NSERT(Q, (tc , τ, s)) 22: end for 23: end procedure

computational resources, each processing a small portion of the total work while respecting ordered dependencies.

Pipeline Task 1

Task 2

Generator

Geant4

Det. Sim.

Reco.

Interaction models Flux prediction

Particle propagation Light simulation

Electronics response

Pattern recognition

Fig. 2. The HEP Monte Carlo processing pipeline discussed in the text. The pipeline has four stages that each depend on the previous stage. The scheduling strategies considered either execute all the stages from a pipeline in sequence on a worker, or separately schedule stages grouped into tasks 1 and 2, potentially on different workers.

mentum estimates, etc. Most HEP experiments also compare data to Monte Carlo simulations in which particle interactions are generated according to a particular physics model, and events are simulated according to a realistic description of the detector geometry. A typical HEP Monte Carlo processing pipeline involves a number of stages: generation of initial particle multiplicities and kinematics, propagation of the generated particles through a detector medium, simulated detector readout of the propagated particles, and “reconstruction,” i.e., running signal processing and pattern recognition algorithms on the detector readout. Simulation pipelines can be described as a DAG; the stages of each pipeline are run in sequence, and the results of each stage provide the inputs for the next. Typical simulation sizes are in the millions of events for modern HEP experiments [1]. To produce a large number of events quickly, the stages of each DAG are distributed across parallelized

We compare the efficacy of two scheduling algorithms on a simulation workflow from the Short-Baseline Near Detector (SBND) experiment at Fermilab [9], [10]. SBND is a 112 t active mass liquid argon time projection chamber (LArTPC) placed in Fermilab’s Booster Neutrino Beam (BNB). SBND has been collecting data since 2024 and has recorded millions of neutrino interactions. SBND’s simulation software is based on LA R S OFT [11], a configurable, module-based library collection with specific utilities for LArTPCs. A typical LA R S OFT workflow consists of many copies of a pipeline with several stages. The first stage runs the G ENIE event generator [12] to simulate neutrino interactions according to specified physics models, followed by G EANT 4 [13] to simulate the propagation of the particles through liquid argon. The next stage simulates the detector readout using a realistic simulation of the electric field configuration in the detector and an electronics response model implemented with W IRE C ELL [14]. Finally, the last stage produces high-level information for analysis by reconstructing the simulated events with the PANDORA [15] pattern recognition algorithm. The steps of the workflow are illustrated in Fig. 2. Because each pipeline begins with a generation step that produces a random number of particles, downstream steps are expected to vary in run time. However, there is a strong correlation between the run times of the early stages of the pipeline and later stages. The correlation is illustrated in Fig. 3 for the LA R S OFT workflow described above, shown for 100,000 independent pipelines. Using this correlation in a multi-stage scheduling strategy allows us to modify the ordering of later stages as earlier stages complete to prioritize stages with longer estimated times first in a dynamic queue. Scheduling tests of the case study application were conducted on Aurora at the Argonne Leadership Computing

Fig. 4. Worker occupancy versus block start time (top panels) and run time versus start time for tasks (bottom panels) for two scheduling strategies on Aurora. The worker count is 26,112 and the pipeline count is 104,448. The left column shows the unordered single-stage case, while the right column shows the dynamic multi-stage case. Colors correspond to task types, either an entire pipeline (unordered single-stage) or generator and downstream stages (dynamic multi-stage). Color shade indicates the iteration number, i.e., how many tasks of each type have already run on each worker. In the lower panels, the white lines show the median stage run times. The discontinuity around 10 minutes in the unordered single-stage case is due to task start time synchronization at the start of the job, and the banding pattern results from the discrete, randomized number of events simulated in each pipeline.

Facility (ALCF). Aurora is a 10,624-node HPE Cray EX system. Each node has two Intel Xeon CPU Max 9470C processors, of which 102 of 104 cores are available for science, with 1 TB of DDR5 memory and 128 GB of HBM, along with six Intel Data Center GPU Max Series GPUs. We developed the package SBN PARSL, a wrapper around the PARSL library [16], to bundle and execute large numbers of DAGs representing LArTPC simulation and data processing pipelines on HPC systems. Internally, SBN PARSL leverages PARSL’s HighThroughputExecutor to distribute worker processes across nodes in multi-node block allocations. We ran 102 workers per node, one per available core. GPUs were not used in this test for simplicity, but we note that the scheduling strategies presented here could be generalized for other queue-based workflows that include GPUs (e.g., [17]). To support different task scheduling priorities within PARSL, SBN PARSL modifies PARSL’s DataFlowKernel, which prioritizes tasks. The following task scheduling strategies are tested: 1) Unordered single-stage — Schedules each pipeline DAG on a single worker, which executes the stages sequentially, effectively reducing the DAG to a single task. 2) Dynamic multi-stage — DAGs are split into two execution units as shown in Fig. 2. The generator stages are prioritized ahead of the downstream stages. As the generator stages complete, their corresponding simula-

tion/reconstruction stages are given an updated priority according to the run length of the generator stage. Workflow timelines are shown in Fig. 4. Top panels show the worker occupancy, as measured by task start and end times recorded by SBN PARSL. Bottom panels show the stage run times as a function of their start time. In the unordered singlestage case, only one computational task processes the entire pipeline. In the dynamic multi-stage case, the generator stages run first and set the priorities of the downstream simulation stages; we see the median stage run time decrease at later start times, as is characteristic of longest-first scheduling. V. R ESULTS We tested the scheduling approaches for the HEP workflow at 256-node scale (N = 26,112 workers), with a generous wall time request to fully capture the long run-time tail produced by a small number of pipelines. The total number of pipelines was M = 104,448, so that M/N = 4. We measured the makespan to complete different fractions of the pipelines, Tp , and the idle fraction at that time, ϕp = Ip /(N × Tp ). We refer to its complement, 1 − ϕp , as the worker occupancy. We compared the HEP workflow results to the theoretical results and to the simulator in Fig. 5 and tabulated comparison metrics in Tab. I. The theoretical results were calculated from Eqs. 5–10, using an empirical task run-time distribution, q(t), from the timings obtained from the empirical case, and using

Worker Occupancy (%) Remaining Work (1-p)

Unordered, single-stage

100

Dynamic, multi-stage

10−1 10−2 Longest-first (theory) Simulator Empirical

Unordered (theory) Simulator Empirical

10−3 100 80 60 40 20 0

0

50

100

150

Wall Time (mins)

200

0

50

100

150

Wall Time (mins)

200

Fig. 5. Comparison of scheduling results from this paper. Upper panels show the remaining fraction of work to complete (1 − p) and lower panels show the worker occupancy. The left and right columns show the unordered single-stage and dynamic multi-stage scheduling strategies, respectively.

TABLE I Expected makespans, Tp , and idle fractions, ϕp , for the theoretical, simulator, and empirical HEP workflow cases. The theory values are idealized estimates, calculated using the task run time distribution from the empirical case and with assumptions discussed in the text; they are expected to underestimate Tp . Theory (Limit) p (%)

Tp (min)

99 99.9 99.99

157.2 172.8 199.1

ϕp (%)

Simulator Tp (min)

ϕp (%)

Empirical Tp (min)

ϕp (%)

166.7 176.6 212.1

11.0 15.9 30.0

161.8 164.6 214.7

4.1 6.7 27.7

Unordered single-stage 7.1 14.7 25.9

166.2 179.6 234.7

11.3 17.9 37.1

Dynamic multi-stage 99 99.9 99.99

153.3 153.6 153.6

4.5 4.0 4.0

157.4 157.6 228.6

2.2 2.3 32.6

numerical methods to obtain Q(t) and its inverse from this distribution. We set M = 104,448 and N = 26,112 for consistency with the empirical case. For the simulator, we used the task run times from the production workflow to build the corresponding workflow model. For the unordered single-stage strategy, we used the FIFO orderer, and for the dynamic multistage strategy, we implemented a custom orderer that uses the tuple of stage number and negative of task duration to set the task priority. In Fig. 5, we see that the dynamic multi-stage strategy has noticeably better performance in terms of smaller Tp and smaller idle fraction ϕp than the unordered case, confirming an improvement in performance by using the predicted run time distribution for ordering the second pipeline stages. The results are confirmed numerically in Tab. I, with the

disclaimer that ϕ99.99 and T99.99 are measured with approximately M × (1 − 0.9999) ≈ 10 tasks only, and are therefore subject to large statistical variance. We find the real-world workflow follows the expected trend from the theoretical results. In Fig. 5, the empirical and simulator worker occupancies have similar slopes to the theoretical predictions. The lag in makespan, as indicated by larger remaining work fractions at equivalent makespans for the empirical and simulator cases in the upper panels, is a direct consequence of the simplifying assumptions made in the equations. For example, the assumption that the tasks are uniformly distributed at the moment the queue empties is optimistic, and holds only for M/N ≫ 1, whereas our measurements are at M/N = 4. We see a similar pattern in the dynamic multi-stage case compared to the longestfirst theoretical limit, though both the simulator and the measurement reproduce the sharp drop in remaining work and worker occupancy characteristic of this scheduling strategy. VI. P IPELINE S IMULATIONS : P ROPERTIES AND S CALING Using the scheduling simulator which we have demonstrated provides high-fidelity reconstructions of scheduling of the SBND workflow, we test several variations of the workflow, varying node count and run time variance. We show how the theory and simulation tool described here can help inform optimized runs at larger scales on Aurora and other systems. For this part of the study, we include a small adaptation to the simulator. Rather than adopt the empirical run times from the case study runs, we use the mean and variance of the task times presented in Fig. 3 to generate collections of estimated task times with high fidelity to the science workflow. For the unordered single-stage case this just requires drawing

Unordered, single-stage Dynamic, multi-stage

150 100 50

Runtime (minutes)

Scheduling Simulations vs Node Count

256 node, 104,448 pipelines simulated

300 Dynamic, multi-stage μ1 = μ2 = 10, σ1 = σ2 varies 250 μ1 = μ2 = 10, σ1 = 1, σ2 varies μ1 = μ2 = 10, σ2 = 1, σ1 varies 200 Unordered, single-stage μ = 20, σ varies 150 100

100

100

75

80

50 25 0 28

29

210

211

Node count

212

213

Fig. 6. Strong scaling tests run with the scheduling simulator for task run times generated from the run-time distributions measured in the SBND case study. These simulations assume 102 workers per node and simulate a fixed number of pipelines (104,448).

task times from a single normal distribution; for the dynamic multi-stage case, we draw task times for the two stages from independent normal distributions and then couple them in pipelines by sorting them and matching tasks in sorted order. This gives results that match the empirical telemetry in Fig. 5. We can then explore some scenarios of optimizing for makespan and worker occupancy. First, we show simulations of strong scaling. We fix the number of pipelines simulated to be the same as the number processed in the empirical HEP workflow runs. We show the worker occupancy in Fig. 6 for the unordered single-stage and dynamic multi-stage scheduling strategies. As the node count increases, the worker occupancy drops while the makespan flattens. This indicates an optimal node count for this pipeline and workload size where the minimum makespan is achieved with minimal loss of worker occupancy. Second, we conduct tests where we fix the node count to 256 as used in the empirical HEP workflow runs, and increase the standard deviation, σ, of task run times (see Fig. 7). For the dynamic multi-stage pipeline, we consider three variants: both stage variances increase together; stage 1 is fixed and stage 2 increases; and stage 2 is fixed and stage 1 increases. For the unordered single-stage pipeline case, we fix the mean of the tasks to be twice the mean run time of the single stages in the dynamic multi-stage case and steadily increase σ. The performance of the dynamic multi-stage case is far less sensitive to task run-time variance than the unordered single-stage case. Increasing the variance of both task run times together gives the worst results, and at sufficiently high variance it converges to the unordered single-stage case. In the case where only one task stage run time increases in variance, the dynamic multi-stage case remains more performant than the unordered single-stage case. However, the case where the early stage is fixed in variance and the downstream stage varies tends to do worse. VII. C ONCLUSIONS Optimal scheduling is necessary at exascale: our results show a high-variance distribution of task run times can lead

Node utilization (%)

Runtime (minutes) Node utilization (%)

200

60 40 20 0 0

5

10

15

20

25

30

Task-duration σ (minutes)

35

40

Fig. 7. Tests exploring differing task distributions with the scheduling simulator. Three cases of a two-stage pipeline are shown where the mean task run time for both stages is 10 minutes and the standard deviation in the task run times of different stages is allowed to vary. Also shown is a version of the unordered single-stage case where the mean task run time is 20 minutes and the standard deviation of the task run times differs.

to large idle worker times and increased makespans. As an example, in cases where approximately 15% of node compute time is left idle, as we observed at our test scale, a substantial number of node-hours can be wasted. If the workload were increased relative to node count in such a way as to fix the makespan, at 2,048 nodes this would be approximately 1,000 node-hours wasted for the HEP pipeline we tested. This number can be reduced to hundreds of node-hours instead using the dynamic multi-stage scheduling strategy. By partitioning pipelines into multiple stages, and leveraging the correlation between the stages to inform task priority, we have shown it is possible to optimize task submission. We validated this policy against a robust theoretical and simulatorbased framework for assessing optimal scheduling of pipelinebased workflows on HPC systems, and against a production Monte Carlo workflow from the SBND experiment. We designed an extensible workflow simulator that can be used to develop and investigate scheduling algorithms for general workflow DAGs, with both deterministic and stochastic task durations. We validated the simulator against real runs on Aurora at the ALCF, and used it to explore properties of the workflow. If properties of the pipeline vary (e.g., task run times or number of stages), the scheduling problem becomes more complex. However, the scheduling simulator we employ can help experimentalists make decisions on how to deploy pipelines if the task run-time distribution can be measured. For this study we did not explore the efficiency of our task launcher, PARSL. Fig. 4 shows there is a small but noticeable launching inefficiency in the 256-node tests we present here. Comparisons of PARSL with other workflow tools have shown that task launching latency becomes an issue as node counts increase and will be an aspect of future study [18]. We also only validated and ran tests on a single resource type (CPUs). Future work will consider mixed resource types (CPUs and GPUs). While per-task performance can improve on GPUs, the total makespan of workflows may not necessarily, in certain circumstances, be improved by the inclusion of GPUs, which

are less numerous on compute nodes. The importance of data-intensive compute patterns for experimental science has been noted as a priority for some time [2]. This study attempts to wrestle with the challenges inherent to the scale needed to deploy these patterns for a representative case from an in-production HEP experiment. More studies of this type will help the experimental community make progress on delivering science from these experiments.

[9]

[10]

ACKNOWLEDGMENT An award of computer time was provided by the ASCR Leadership Computing Challenge (ALCC) program. This research used resources of the Argonne Leadership Computing Facility, which is a U.S. Department of Energy Office of Science User Facility operated under contract DE-AC0206CH11357.

[11]

[12]

R EFERENCES [1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

DUNE Collaboration, DUNE offline computing conceptual design report, 2022. arXiv: 2210 . 15665 [physics.data-an]. [Online]. Available: https:// arxiv.org/abs/2210.15665 W. L. Miller et al., “Integrated research infrastructure architecture blueprint activity (final report 2023),” US Department of Energy (USDOE), Washington, DC (United States). Office of Science; Lawrence Berkeley National Laboratory, Berkeley, CA (United States), Tech. Rep., Jul. 2023. DOI: 10.2172/1984466 [Online]. Available: https://www.osti.gov/biblio/1984466 B. Abolfathi et al., “The LSST DESC DC2 simulated sky survey,” The Astrophysical Journal Supplement Series, vol. 253, no. 1, p. 31, Mar. 2021, ISSN: 1538-4365. DOI: 10.3847/1538-4365/abd62c A. Benoit, Ü. V. Çatalyürek, Y. Robert, and E. Saule, “A survey of pipelined workflow scheduling: Models and algorithms,” ACM Computing Surveys (CSUR), vol. 45, no. 4, pp. 1–36, 2013. A. S. Villarreal, Y. Babuji, T. Uram, D. S. Katz, K. Chard, and K. Heitmann, “Extreme scale survey simulation with Python workflows,” in 2021 IEEE 17th International Conference on eScience (eScience), IEEE, Sep. 2021, pp. 206–214. DOI: 10.1109/escience51609. 2021.00031 W. Zheng and R. Sakellariou, “A Monte-Carlo approach for full-ahead stochastic DAG scheduling,” in 2012 IEEE 26th International Parallel and Distributed Processing Symposium Workshops & PhD Forum, IEEE, 2012, pp. 99–112. A. Benoit, “Scheduling pipelined applications: Models, algorithms and complexity,” Ph.D. dissertation, École Normale Supérieure de Lyon, 2009. A. Hagberg, P. J. Swart, and D. A. Schult, “Exploring network structure, dynamics, and function using NetworkX,” Los Alamos National Laboratory (LANL), Tech. Rep., 2007.

[13]

[14]

[15]

[16]

[17]

[18]

R. Acciarri et al., “A Proposal for a Three Detector Short-Baseline Neutrino Oscillation Program in the Fermilab Booster Neutrino Beam,” Mar. 2015. arXiv: 1503.01520. P. A. Machado, O. Palamara, and D. W. Schmitz, “The Short-Baseline Neutrino Program at Fermilab,” Annual Review of Nuclear and Particle Science, vol. 69, no. 1, pp. 363–387, Oct. 2019, ISSN: 0163-8998, 1545-4134. DOI : 10 . 1146 / annurev - nucl - 101917 - 020949 arXiv: 1903.04608 [hep-ex]. E. D. Church, “LArSoft: A Software Package for Liquid Argon Time Projection Drift Chambers,” 2013. DOI : 10 . 48550 / arxiv . 1311 . 6774 arXiv: 1311 . 6774 [physics.ins-det]. C. Andreopoulos et al., “The GENIE neutrino Monte Carlo generator,” Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, vol. 614, no. 1, pp. 87–104, 2010, ISSN: 01689002. DOI: 10.1016/ j.nima.2009.12.009 arXiv: 0905.2517 [hep-ph]. S. Agostinelli et al., “Geant4—a simulation toolkit,” Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, vol. 506, no. 3, pp. 250–303, 2003, ISSN: 01689002. DOI: 10.1016/S0168-9002(03) 01368-8 MicroBooNE Collaboration, “Wire-cell 3D pattern recognition techniques for neutrino event reconstruction in large LArTPCs: Algorithm description and quantitative evaluation with MicroBooNE simulation,” Journal of Instrumentation, vol. 17, no. 01, P01037, Jan. 2022. DOI : 10.1088/1748-0221/17/01/P01037 J. S. Marshall and M. A. Thomson, “The Pandora software development kit for pattern recognition,” The European Physical Journal C, vol. 75, no. 9, p. 439, Sep. 2015, ISSN: 1434-6044, 1434-6052. DOI: 10 . 1140 / epjc / s10052 - 015 - 3659 - 3 arXiv: 1506 . 05348 [physics.data-an]. Y. Babuji et al., “Parsl: Pervasive Parallel Programming in Python,” in Proceedings of the 28th International Symposium on High-Performance Parallel and Distributed Computing, Phoenix AZ USA: ACM, Jun. 2019, pp. 25–36, ISBN: 978-1-4503-6670-0. DOI: 10 . 1145/3307681.3325400 K. Meagher and J. van Santen, “Parallelizing air shower simulation for background characterization in IceCube,” EPJ Web of Conf., vol. 295, p. 11 016, 2024. DOI: 10. 1051/epjconf/202429511016 [Online]. Available: https: //doi.org/10.1051/epjconf/202429511016 H. Tummalapalli et al., Overcoming orchestration bottlenecks at exascale: A decentralized, policy-driven approach for Sim-AI ensembles, 2026. arXiv: 2607.12211 [cs.DC].

Related documents

• Real-time Optimization of Simulation and Data Processing Pipelines for Experiments on Exascale Computing Platforms#1186812
• nvidia/ToolScale#786801
• Descriptive Dispatch of Computational Work#985207
• The UPC++ PGAS library for Exascale Computing: Extended Abstract#614615
• Modelo de Inferencia Holográfica del Sustrato Computacional: Extensión Matemática del Principio de Clausura#89311