arXiv:2607.26877v1 [cs.DC] 29 Jul 2026
Hybrid Workflow Composition for Extreme-Scale Data Processing: A Case Study on the HL-LHC (Extended Version) Alan Malta Rodrigues
Douglas Thain
Department of Computer Science and Engineering University of Notre Dame Notre Dame, IN [email protected]
Department of Computer Science and Engineering University of Notre Dame Notre Dame, IN [email protected]
Abstract—High-Throughput Computing (HTC) environments tailored for high-concurrency resource efficiency require sophisticated orchestration to manage petabyte-scale data across heterogeneous resources. A critical but often overlooked challenge is workflow composition: the strategic grouping of tasksets within a Directed Acyclic Graph (DAG) to mitigate execution overhead while maximizing resource utilization. This paper presents a novel simulation framework for characterizing the interplay between taskset granularity and system-level constraints (e.g., job latency, failure rate, throughput, and I/O bandwidth). By exploring a high-dimensional parameter space, we quantify the performance sensitivity of diverse workflow topologies. Our results demonstrate that hybrid composition strategies, which dynamically balance taskset independence with execution grouping, can yield up to 3.8× throughput increase and a 14.9× reduction in network overhead. We further propose a multi-metric objective function that enables policy-driven optimization, allowing system architects to navigate the Pareto frontier between throughput, I/O cost, and CPU efficiency. These findings provide a rigorous foundation for automated workflow synthesis in distributed systems, offering a scalable model for next-generation scientific pipelines. All artifacts are publicly available. Index Terms—Directed Acyclic Graphs (DAGs), DiscreteEvent Simulation, High-Throughput Computing, Performance Modeling, Resource Optimization, Scientific Workflows, Workflow Management Systems.
I. I NTRODUCTION Workflow construction and execution strategies have a significant impact on resource utilization, event throughput, and operational efficiency in large-scale scientific computing. As data-intensive domains transition toward exascale processing, the interplay between workflow topology (the logical grouping of execution stages) and heterogeneous resource constraints becomes a primary performance bottleneck. In High-Energy Physics (HEP), processing pipelines operate on events: atomic physics collision records generated by particle detectors. Workflows are modeled as Directed Acyclic Graphs (DAGs) where nodes represent tasksets (distinct A. Malta Rodrigues is a researcher with the Department of Physics, University of Notre Dame. He serves as the Lead Developer for the CMS Workload Management System; the findings in this study are intended to guide the architectural evolution of the next-generation CMS orchestration software.
computational steps, such as detector simulation or track reconstruction). During execution, these tasksets expand, meaning they are instantiated and split across event batches into hundreds of thousands of schedulable grid jobs that execute across distributed sites. While batch schedulers determine when and where jobs run across grid sites, initial workflow composition determines what is packaged inside each job. Grouping dependent tasksets into a single job enables direct data exchange via local node storage (scratch disk), whereas executing them as separate jobs forces intermediate data to be written to and read from remote grid storage. The Compact Muon Solenoid (CMS) [1] experiment at the Large Hadron Collider (LHC) [2] typifies this challenge, generating hundreds of petabytes of data annually. To process this volume, HEP currently relies on the Worldwide LHC Computing Grid (WLCG) [3], a global federation of 1.4 million cores. However, as processing demands increase for the HighLuminosity LHC (HL-LHC) era [4], workflow orchestration must adapt to leverage increasingly heterogeneous opportunistic, cloud and HPC resources alongside traditional grid sites. The CMS workflow management system has evolved over the past 15 years of LHC operations, primarily providing two static execution models: TaskChain, which executes each taskset as an independent, standalone grid job, and StepChain, which merges all sequential tasksets into a single monolithic job running on a worker node. While effective, these represent two extremes of a vast, unexplored compositional spectrum. There exists a significant research gap in exploring adaptive, group-based compositions that could better leverage the increasingly heterogeneous set of opportunistic, cloud, and High-Performance Computing (HPC) resources. To systematically evaluate these compositional trade-offs, we investigate the following Research Questions (RQs): RQ1: How can dependent tasksets be grouped into execution units that respect resource constraints while optimizing event throughput? • RQ2: What are the fundamental trade-offs between I/O activity, resource utilization, and throughput across the •
Fig. 1: Execution efficiency across 16 workflow constructions for a 5-taskset realistic sequential workflow (12h target job length, 5% failure rate, 100 MB/s network bandwidth). Green bars and red points represent mean values across N = 10 independent simulation runs with randomized failure seeds. Vertical whiskers indicate the Standard Error of the Mean (SEM ≈ 0.1%). Constructions 1 (StepChain) and 16 (TaskChain) map existing production paradigms, exposing critical trade-offs between resource utilization and processing overhead.
Fig. 2: The CMS Data Processing Pipeline, illustrating the data flow from the online detector triggers to prompt reconstruction and archival storage. Projected HL-LHC data rates (750 kHz L1 and 10 kHz HLT) represent a 10× increase over current operations, necessitating the optimized offline processing and simulation orchestration evaluated in this study.
grouping spectrum? RQ3: What is the sensitivity of this composition strategy A. The CMS Processing Model as an Extreme-Scale Use Case to target job length, failure rate, network bandwidth, and The CMS experiment at the LHC generates data at an workflow profiles? unprecedented scale, necessitating a multi-stage filtering and To address this gap, this paper introduces a framework for processing chain (Fig. 2). As the field transitions to the Highthe design, simulation, and evaluation of group-based DAG Luminosity LHC (HL-LHC) era, the system must evolve to workflow strategies. Our primary contributions include: handle a significant increase in data throughput. While current • Workflow Composition Engine: A formalization and operations manage a Level-1 trigger rate of 100 kHz and framework that parses generic workflows and enumerates a High-Level Trigger (HLT) output of 1 kHz, the HL-LHC all valid group-based compositions based on taskset is projected to increase these rates to 750 kHz and 10 kHz, dependencies and hard constraints. respectively [6], [7]. • DAGFlowSim Engine: A high-fidelity simulation engine From a distributed systems perspective, this nearly ten-fold [5] capable of executing thousands of compositions under increase in event rates - coupled with rising event complexity parameterized scenarios, including variable failure rates, - represents a massive orchestration challenge. The Offline job length and client-side network bandwidth. Processing stage must transform raw data into physics-ready • Multi-Metric Analytic Framework: A quantitative char- formats through complex reconstruction and simulation DAGs. acterization of the trade-offs between throughput and These workflows are executed across the Worldwide LHC efficiency, providing a policy-driven score function for Computing Grid (WLCG), a heterogeneous infrastructure strategy selection. where system performance is increasingly dominated by I/O • Open-Source Artifacts: We provide the simulator, analy- constraints and the structural efficiency of the workload rather sis scripts, and a dataset covering thousands of simulated than peak floating-point performance. compositions to support community reproducibility. B. Workflow Orchestration and Composition Fig. 1 illustrates the utility of this framework, showing Scientific workflows are typically modeled as DAGs, where processing efficiency metrics across 16 workflow constructions nodes represent computational tasksets and edges denote evaluated over N = 10 independent simulation runs per data dependencies. In High-Energy Physics (HEP), Workflow construction. By exposing the spectrum between existing Management Systems (WMS) such as WMCore [8], PanDA extrema (represented by constructions 1 and 16), our framework [9], and DIRAC [10] serve as middleware layers that decouple enables the identification of optimal intermediate strategies. To logical workflow definitions from physical execution units. ensure the reliability of these simulated results, we performed Recent characterizations of extreme-scale WMS [11], [12] a validation study using real CMS workflow schemas, demonidentify a critical design gap: the transition from logical DAGs strating that the simulator accurately reflects the trade-offs to schedulable execution units. While traditional tools like observed in production environments. Pegasus [13] and DAGMan [14] manage job-level dependencies, the composition strategy - the logic used to aggregate tasks II. BACKGROUND AND R ELATED W ORK into jobs - remains a static policy. Thomas and Thain [15] Efficiently executing petabyte-scale scientific workflows established that dependency-based grouping improves data requires integrating Directed Acyclic Graph (DAG) orchestra- locality; however, as workloads become increasingly datation, resilient High-Throughput Computing (HTC), and high- plane driven [16] and cross-facility in nature [17], the need for fidelity simulation. This section situates DAGFlowSim within scalable composition techniques that can navigate the tradethe broader landscape of extreme-scale computing middleware. off between taskset granularity and I/O pressure becomes •
paramount [18]. DAGFlowSim addresses this by enabling the systematic exploration of the composition spectrum. C. HTC Resilience and Heterogeneity Unlike High-Performance Computing (HPC), HTC focuses on aggregate throughput over long-running campaigns. HEP workloads are predominantly HTC-based, relying on batch schedulers (e.g., HTCondor, Slurm) to match millions of jobs to distributed, often opportunistic resources. Resilience in these environments is achieved through retry policies to mitigate hardware and network faults. Since failure rates directly inflate wall-clock time and resource waste, DAGFlowSim incorporates configurable failure models to study how different taskset composition patterns (e.g. monolithic vs granular) impact global system resilience and throughput. D. Simulation and Modeling Approaches Simulation is essential for evaluating composition policies without the prohibitive cost of full-scale grid experiments. While robust simulation frameworks like WRENCH [19] (built on SimGrid) excel at high-fidelity network topology and queuing emulation, their setup complexity and execution overhead scale significantly when conducting high-dimensional combinatorial parameter sweeps. DAGFlowSim complements these established frameworks by providing a lightweight, specialized discrete-event engine optimized specifically for the structural analysis of HEP-style workloads. By decoupling heavy network-state simulation from taskset aggregation dynamics, DAGFlowSim makes evaluating thousands of multi-variate scenarios (e.g., 7,680 simulation runs) computationally tractable. Specifically, it differentiates itself from general-purpose simulators by emphasizing: (1) event-based taskset scaling; (2) group-based scheduling and execution; (3) systematic sensitivity analysis across job lengths, failure rates, and bandwidth constraints; and (4) a specialized multi-metric analytic framework for characterizing the Pareto frontier of workflow composition and execution. III. S YSTEM M ODEL AND DAGF LOW S IM A RCHITECTURE The primary objective of the proposed framework is to transform an abstract DAG workflow description into an optimized, group-based execution plan tailored to specific infrastructure policies. The system architecture, illustrated in Fig. 3, integrates taskset grouping logic with a high-fidelity simulation engine to navigate the compositional trade-off space. A. Design Goals and Policy-Driven Selection The framework satisfies four primary requirements: (1) automated taskset grouping that enforces hard constraints; (2) discrete-event simulation of group-based execution; (3) systematic composition search space exploration; and (4) a flexible, multi-objective score function. Here, hard constraints represent parameters strictly necessary for the taskset grouping decision, such as DAG dependency ordering, target OS version, and CPU architecture. In contrast, soft constraints represent parameters that do not require strict
Fig. 3: System architecture of the proposed orchestration framework, illustrating the pipeline from generic workflow specification to optimized policy-driven compositional ranking.
compatibility, such as memory and CPU cores required for taskset execution, input/output data rates, and other variables that influence operational optimization objectives and site policies - such as targeting ideal job durations (e.g., 8–12 hours) or capping remote network bandwidth usage. B. Assumptions and Scope To maintain a focused and computationally tractable simulation environment, the following assumptions define the model’s analytical scope: • Deterministic Taskset Characterization: Tasksets are defined by CPU time and data size per event, assuming uniform CPU performance across worker nodes. Rationale: This isolates the structural impact of workflow composition from hardware noise and node heterogeneity (analytical isolation). • Operational Overhead: The model incorporates taskset bootstrap costs (60s) and client-side network bandwidth for remote I/O. Rationale: Baseline overhead values are grounded in production WLCG execution logs [8], while high-level WMS bookkeeping and site-level storage queuing latencies are excluded to isolate job-level composition dynamics. • Structural Constraints: The framework supports Directed Acyclic Graphs (DAGs), including sequential chains and fork structures. Rationale: For this baseline study, tasksets are assumed to be software-compatible, allowing any grouping configuration that respects DAG dependency order; hardware-incompatible tasksets are enforced via hard constraints in the compatibility matrix H. • Comparative Analysis: DAGFlowSim is designed for the relative ranking of compositions rather than the absolute prediction of production metrics. Rationale: This enables researchers to evaluate and identify optimal compositional strategies for a given scenario without requiring an exhaustive, high-overhead model of every transient grid anomaly. • Workflow Granularity and Scale: In HTC environments typified by CMS, extreme scale derives from event concurrency and petabyte-scale data volume rather than DAG topological depth. Standard production workflows consist of compact sequential or fork structures (N ≤ 10 processing stages, e.g., GEN-SIM-DIGI-RECO-NANOAOD). For these representative lengths, complete compositional
Job Sizing and Events: The total workload is defined by a request for N events. The simulator calculates the batch size (events per job) by dividing the target job length by the aggregate T imeP erEvent of all tasksets within a group. This ensures that every grid job instance within a group processes a C. Mathematical Formulation of Taskset Grouping uniform number of events, optimizing for typical site-specific We formalize the workflow composition as a constrained time limits (e.g., 8-hour or 24-hour slots). Event-Driven Flow: A group enters the Ready state only partitioning problem. Let W = {T1 , T2 , . . . , Tn } denote the set of workflow tasksets. A compatibility matrix H is defined when its parent dependencies in the DAG are satisfied. We such that Hij = 1 if Ti and Tj satisfy hardware homogeneity model a steady-state throughput environment where jobs are (CPU/OS) and share a dependency path in the workflow DAG. dispatched in batches. Upon completion, processed events Definition 1 (Valid Group): A subset of tasksets G ⊆ W propagate to child groups. This flow continues until all N events reach the DAG leaf nodes. is a valid execution group if it satisfies: Overhead and Resilience: Each job’s wall-clock time 1) Pairwise Compatibility: ∀Ti , Tj ∈ G, Hij = 1. includes a fixed bootstrap overhead (60s per taskset) and 2) Path Containment: For any Ti , Tj ∈ G, every taskset Tk a variable I/O latency determined by SizeP erEvent and on the directed path between Ti and Tj in the DAG must a configurable network bandwidth (default 100 MB/s). A also satisfy Tk ∈ G. probabilistic failure model forces failed jobs to consume partial The path containment requirement ensures that groups do resources before returning their events to the group buffer for not skip intermediate dependencies, preserving execution order retry, enabling robust characterization under grid instability. within a single atomic execution unit (a grid job). Definition 2 (Workflow Composition): A composition C is F. Performance Metrics a partition of W into disjoint To enable a multi-dimensional comparison of compositions, Smgroups {G1 , G2 , . . . , Gn } such that Gk ∈ G for all k, and k=1 Gk = W . the framework calculates job-level performance metrics and The workflow builder enumerates the space of all valid aggregate them into workflow-level metrics. Table I categorizes compositions C, ranging from the fully-ungrouped TaskChain these metrics. (m = n) to the fully-grouped StepChain (m = 1). enumeration is computationally trivial (16 valid sequential partitions for N = 5). For hypothetical ultra-large DAGs (N > 10), partition scaling would require heuristic search techniques left for future work.
TABLE I: Taxonomy of Workflow and Job-Level Metrics.
D. Workflow Model and Taskset Properties We modeled the workflow as an annotated DAG where each node Ti represents a taskset of idempotent jobs and edges represent data dependencies. Unlike job-level simulators, our model enriches the taskset definition with metadata required for structural orchestration and data-locality optimization. • Taskset Characteristics: Each taskset is defined by its computational intensity (CPU time per event) and I/O footprint (data size per event). Resource requirements - including memory, core count, OS version and CPU architecture constraints - are treated as invariants that dictate group compatibility. A KeepOutput flag designates whether a taskset’s data is transient (local to the group) or permanent (requiring remote storage write-back), unless data dependency is across group boundaries. • Group-Based Execution: A group G represents the unit of grid scheduling. Tasksets within a group execute sequentially on the same worker node, enabling intra-job data locality where intermediate data is passed via local scratch space rather than shared storage. This localized I/O mitigates network pressure and drives the performance gains evaluated in Section IV. E. Simulation Engine DAGFlowSim employs a discrete-event, batch-synchronous simulation model to characterize workflow performance under varying system pressures. The execution logic is governed by three primary mechanisms.
Category
Metric (Description)
Throughput
Event Throughput: Normalized events per CPUsecond.
Latency
Total Turnaround: End-to-end workflow makespan. Total Wall-time: Aggregated job wall-clock time.
Efficiency
CPU Utilization: Ratio of used vs. allocated CPU. Memory Occupancy: Ratio of used vs. allocated memory. Job Overhead: Aggregated bootstrap and networkbound I/O latencies. Success Rate: Fraction of requested events completed. Job Failure Rate: Observed fraction of failed job attempts.
I/O Stress
Network Transfer per Event: Remote I/O footprint per unit of work. Local/Remote Ratio: Balance of scratch vs. net I/O.
These metrics provide the raw data for the policy-driven score function, which allows system operators to navigate the Pareto frontier of workflow execution based on localized infrastructure priorities. IV. E XPERIMENTAL D ESIGN AND S CENARIO M ODELING To evaluate the proposed grouping strategies, we execute a multi-variate factorial experiment across diverse workflow profiles and four environmental stressors. While the framework supports complex DAGs - including fork structures - this study focuses on a rigorous characterization of three sequential profiles to achieve causal isolation. This allows for a precise
analysis of the interplay between taskset granularity and resource constraints without the confounding variables of nonlinear dependencies. The broader dataset, including results for an additional three fork-based profiles, is documented and available in the public repository.
TABLE II: Taskset Resource Requirements across Workflow Profiles. seq real Taskset T1 T2 T3 T4 T5
seq homo
seq hetero
Cores
RAM
Cores
RAM
Cores
RAM
8 4 4 2 4
3GB 7GB 8GB 4GB 4GB
8 8 8 8 8
8GB 8GB 8GB 8GB 8GB
1 8 64 4 4
2GB 16GB 64GB 10GB 8GB
A. Workflow Profiles and Composition Search Space We define three 5-taskset sequential profiles to capture the diversity of HEP workloads, with their specific resource requirements detailed in Table II. Each profile is evaluated across 16 different compositions (denotated as Const 1 through Const 16, short for Workflow Construction 1 to 16), ranging from the fully-grouped StepChain (Const 1) to the fullyindependent TaskChain (Const 16), as illustrated in Fig. 4. Fig. 5: Characterization of baseline taskset parameters across the • Realistic (seq real): A CMS-inspired sequential produc- three evaluated workflow profiles. Output data products for Taskset 1 tion chain featuring heterogeneous taskset requirements. and Taskset 2 are transient within the runtime pipeline and are not As detailed in Fig. 5, tasksets are characterized by specific persisted for final end-user analysis. T imeP erEvent and SizeP erEvent values calibrated from real-world WLCG production runs. This profile bandwidths, resulting in 480 unique scenarios for each compoincludes transient data handling for intermediate tasksets. sition. For a sequential fully compatible 5-tasksets workflow, there are 16 valid workflow constructions, leading to a total • Homogeneous (seq homo): Features uniform resource requirements across all tasksets. This profile isolates the of 7,680 unique simulation runs. Fig. 6 summarizes these pure I/O and execution overhead by eliminating resource experimental factors and their systems-level rationale. To ensure a controlled environment, we assume a fixed bootstrap overhead fragmentation and allocation waste. (60s) and a saturated resource pool with unlimited job slots to • Heterogeneous (seq hetero): Features extreme variance focus purely on the structural efficiency of the compositions. in requirements, specifically a 64-core requirement for the intermediate reconstruction stage (Taskset3). This profile evaluates the over-provisioning penalty: coarse-grained grouping (Const 1) forces the system to allocate peak requirements (64 cores) for the entire execution duration, leading to significant resource fragmentation. B. Experimental Matrix The study explores the complete combinatorial space of workflow profiles, target job lengths, failure rates and network
Fig. 6: Multi-variate experimental matrix mapping the 480 evaluation scenarios designed to expose architectural limits and systems-level scaling trade-offs.
C. Empirical Validation and Calibration Before conducting wide-scale parameter sweeps, DAGFlowSim was validated against production WLCG execution traces using the Realistic template. We calibrated baseline simulation parameters using aggregated metadata Fig. 4: Structural overview of the 16 evaluated workflow configufrom actual CMS StepChain (Const 1) and TaskChain (Const rations, illustrating the spectrum from a monolithic grouped model (Const 1: StepChain) to a fully decomposed independent model (Const 16) runs, normalized across a 5-million event envelope. These 16: TaskChain). Color-coded segments indicate shared execution operational extrema represent the only execution modes boundaries for Tasksets 1–5 (e.g., Const 7 aggregates Tasksets 2– natively supported by current CMS workload management 4 while isolating Tasksets 1 and 5). infrastructure (WMCore); validating intermediate hybrid
Fig. 7: Simulator validation against empirical WLCG baselines under zero-failure conditions (12h target length, 100 MB/s bandwidth). Hatched bars denote empirical metrics normalized across successful jobs; solid bars represent deterministic simulations (zero variance). Though absolute empirical I/O is lower due to unconsumed secondary data products, structural scaling trends and ratios between StepChain and TaskChain are accurately captured.
constructions (e.g., Const 13) on production grid infrastructure would require extensive middleware refactoring, a software development effort that this paper’s simulation framework is specifically designed to motivate and inform. To isolate structural I/O dynamics from transient infrastructure noise, empirical metrics were normalized across successful job executions, imputing sample means for transient job failures. Corresponding simulations were executed under a baseline 0% failure rate policy, yielding deterministic results with zero variance across independent runs. As shown in Fig. 7, the simulator accurately replicates fundamental structural I/O trade-offs. The total absolute data volume is lower in empirical runs because production workflows generate secondary data products that are persisted but never read by downstream tasksets in the processing chain. However, the performance scaling and proportional I/O ratio between the aggregated StepChain and decomposed TaskChain models remain tight and consistent, validating the model’s predictive accuracy for comparative structural analysis. V. P ERFORMANCE E VALUATION AND T RADE - OFF A NALYSIS We evaluate the 16 compositions across the full factorial experimental matrix to identify the Pareto optimal points within the grouping spectrum. This evaluation is structured as a threestage analysis pipeline:
Fig. 8: I/O volume breakdown across the 16 compositions under baseline CMS realistic profile parameters. Top: Per-event data volume isolated by operation type. Bottom: Cumulative stacked workflow data volume. Grouped compositions (Const 1) optimize local data reuse, whereas full independence (Const 16) maximizes remote data-staging overhead.
1) Explore: A baseline characterization of the fundamental trade-offs between I/O pressure, resource efficiency, and aggregate throughput. 2) Generalize: A sensitivity analysis under environmental stressors - including infrastructure faults, bandwidth constraints, and varying taskset granularities - to assess composition robustness. 3) Rank: A policy-driven synthesis using a multi-metric score function to identify optimal compositions for specific operational regimes. By traversing this evaluation pipeline, we demostrate how the optimal grouping strategy shifts from extrema toward hybrid compositions, as environmental constraints and infrastructure volatility increase. A. Baseline Compositional Trade-offs We establish a baseline utilizing the seq real profile (12h jobs, 100 MB/s, 0% failure) to isolate the structural impact of composition on I/O pressure, resource fragmentation, and aggregate throughput. I/O and Storage Footprint: As shown in Fig. 8, maximal grouping (Const 1) reduces remote I/O by 14.9× compared to the full independence (Const 16) by leveraging workernode scratch space. Conversely, fully ungrouped composition maximizes the network-transfer-per-event, as every taskset transition incurs a remote staging cost. Resource Fragmentation and Efficiency: While Const 1 excels in I/O, it suffers from severe resource underutilization
Fig. 9: Resource utilization profiles across all 16 compositions under the baseline scenario (seq real, 12h target length, 0% failure rate, 100 MB/s bandwidth). Monolithic grouping (Const 1) minimizes network transfer but underutilizes CPU and memory; conversely, high independence (Const 16) maximizes CPU and memory efficiency at the cost of severe network-staging overhead.
of decomposition: Const 16 incurs up to 14.9× more remote network activity than Const 1. While a 10 MB/s network bandwidth severely constrains I/O-intensive tasks (causing overheads up to ∼4,000s), performance stabilizes at 100 MB/s as network latency ceases to be the primary bottleneck. Reliability and Failures: While higher failure rates (up to 25%) increase network waste due to retries, the throughput degradation remains largely consistent across all compositions. Notably, the hybrid Const 13 matches TaskChain throughput while achieving a 2.7× network efficiency improvement over the fully decomposed model. This demonstrates that hybrid compositions provide superior resilience by limiting the I/O blast radius without sacrificing productivity. Temporal Dynamics and Makespan: Target job length creates a sharp trade-off between amortized overhead and completion speed. Short-lived jobs (15m-1h) amplify the fixed 60s bootstrap overhead per taskset, consuming up to 33% of total wall-clock time for grouped units. However, makespan (Total Turnaround Time) heavily favors Const 1, which completes in 36h compared to 71h for Const 16 (at 12h length, 5% failure). This confirms that sites prioritizing rapid completion of urgent tasks may favor StepChains, while those prioritizing global efficiency should favor hybrids. Strategic Takeaway: While event throughput is resilient to failures, resource efficiency (network and overhead) is highly sensitive to the operating environment. This volatility necessitates a hybrid approach to balance the low latency of StepChains with the high resource utilization of TaskChains.
due to over-provisioning. Because a grouped job must request the maximum CPU and memory required by any single taskset C. Policy-Driven Composition Ranking for its entire duration, it leads to significant allocation waste The final stage of our evaluation synthesizes multiduring low-intensity tasks. Fig. 9 quantifies this fragmentation dimensional metrics into an actionable utility ranking. We by showing the resource utilization rate differences among the define a weighted score function S for each composition C: configurations. Hybrid compositions (e.g., Const 13) bridge this X gap, yielding a 31.5% higher throughput and reducing CPU S(C) = (wi × m̂i ) (1) and memory overhead by 23.6% and 39.2%, respectively. i Fig. 10 visualizes the resulting network-throughput Pareto frontier. where m̂i is the normalized, feature-scaled value of metric i Strategic Takeaway: The grouping spectrum reveals a non- (throughput, CPU/event, Mem/event, and Network/event) such monotonic relationship between granularity and performance, where hybrid compositions mitigate the diminishing returns of both over-aggregation and over-decomposition. B. Sensitivity to Environmental Stressors We evaluate the robustness of composition strategies by subjecting them to four dimensions of operational stress: Temporal (Job Length), Reliability (Failures), Infrastructure (Bandwidth), and Structural (Workflow Profiles). The ”Best Hybrid” for each scenario is defined by the highest event throughput, with network transfer used as a tiebreaker. Structural and Infrastructure Constraints: Workflow topology fundamentally dictates strategy success. As shown in Fig. 11a, the Heterogeneous profile exhibits the highest sensitivity; grouping diverse tasksets into a monolithic unit (Const 1) results in a 3.8× throughput penalty due to resource over-provisioning. Conversely, Fig. 11b illustrates the I/O cost
Fig. 10: Trade-off evaluation of the 16 compositions under the baseline scenario (seq real, 12h target length, 0% failure rate, 100 MB/s bandwidth), mapping event throughput against shared-storage write volumes. The distribution identifies an optimal composition that achieves maximum throughput while reducing remote write overhead by a factor of ∼3×.
Fig. 11: Impact of workflow profiles on system performance (5% failure rate, 12h target length). (a) Event Throughput: Const 16 provides the highest performance gain in heterogeneous workloads, while homogeneous chains exhibit minimal variance. (b) Network Efficiency: Const 1 minimizes per-event I/O overhead whereas Const 16 maximizes it across all workflow profiles; hybrid configurations achieve a Pareto-optimal balance.
Fig. 13: Workflow composition score breakdown under the Default Policy (maximizing throughput). The chart illustrates the cumulative contribution of each evaluation metric to the final score. Although Const 13 and Const 16 both maximize throughput (yielding the full 0.4 score component), the network footprint acts as the decisive differentiator favoring the hybrid configuration strategy.
focus to specific system bottlenecks. TABLE III: Optimization Policy Weight Configurations Policy
Thr.
CPU
Mem
Net.
Primary Objective
Default
0.4
0.2
0.2
0.2
I/O-Prior.
0.2
0.2
0.2
0.4
Res.-Prior.
0.2
0.3
0.3
0.2
Balance throughput and efficiency. Minimize shared storage footprint. Maximize per-event utilization.
In our realistic case study, Const 13 remains the top-ranked configuration across all three policies. It successfully mitigates Fig. 12: Heatmap of normalized metric scores for all 16 compositions under the realistic sequential workflow (5% failure rate, 12h target the static resource fragmentation of Const 1 while bypassing length, 100 MB/s bandwidth). Highly grouped (e.g., Const 1) compo- the severe I/O penalties of Const 16. Fig. 13 visualizes this sitions excel at I/O efficiency while high independence (e.g., Const 16) behavior under the Default Policy, showing the breakdown of optimizes throughput and resource utilization; hybrid configurations how each metric’s weighted contribution (wi × m̂i ) accumulates successfully balance both performance extremes. toward a perfect utility score of 1.0. While both Const 13 and Const 16 maximize the throughput component (capturing the that higher values denote more desirablePperformance, and wi full 0.4 weight allocation), Const 16 suffers a heavy network represents the policy-specific weights ( wi = 1). penalty. This makes network efficiency the decisive factor The metric heatmap in Fig. 12 provides a global visualization that establishes the hybrid Const 13 as the globally optimal of this design space. The two extrema - the fully aggregated construction. StepChain (Const 1) and the fully independent TaskChain Our results confirm that the optimal composition is highly (Const 16) - occupy opposite ends of the spectrum, each sensitive to the underlying workload profile. In the Homogeexcelling in one dimension while sacrificing others. Specifically, neous profile, the StepChain becomes competitive (Score: 0.74) highly independent compositions optimize throughput and as the over-provisioning penalty is eliminated. However, in the resource utilization but exhibit poor performance in makespan Heterogeneous profile, aggressive ungrouping (Const 15 and and remote I/O efficiency. Conversely, highly aggregated 16) becomes mandatory to avoid resource waste. This confirms compositions lead in I/O efficiency but lack throughput and that a static, invariant grouping strategy is fundamentally resource-allocation efficiency. Hybrid constructions, particularly flawed for modern, heterogeneous grid computing; instead, Const 13, emerge as Pareto-efficient configurations, capturing composition must be a policy-driven, workload-aware decision. high scores across multiple dimensions and avoiding the catastrophic inefficiencies of the extreme policies. VI. D ISCUSSION AND R ECOMMENDATIONS To demonstrate the framework’s flexibility, we evaluate three optimization policies (Table III) tailored to distinct operational The simulation results validate our central thesis: optimal and infrastructure constraints. The Default Policy prioritizes workflow compositions are rarely found at the static extrema throughput while balancing resource consumption, whereas (StepChain or TaskChain) but rather in hybrid configurations the I/O-Prioritized and Resource-Prioritized policies shift the that balance I/O locality with resource throughput.
Analyzing the seq real profile under the Default Policy, VII. C ONCLUSION AND F UTURE W ORK Const 13 (a 1+2+1+1 grouping) emerges as the ParetoThis paper introduces a formal framework for taskset comoptimal construction for this specific operational scenario. position alongside a multi-metric score function to automate We emphasize that Const 13 is an exemplar output of our extreme-scale scientific workflow orchestration. By exploring policy-driven framework rather than a universal static recipe the spectrum between traditional grouped (StepChain) and for all HEP workloads. In this specific 5-taskset workflow, fully decomposed (TaskChain) models, we demonstrate that Const 13 targets the most I/O-intensive segment: Tasksets workload-aware hybrid workflow compositions can match the 2 and 3 are grouped to keep heavy intermediate data local throughput of independent tasks while significantly mitigating to the worker node scratch disk, while Tasksets 1, 4, and the I/O burden on shared infrastructure. Ultimately, this frame5 execute independently. This eliminates a major remote work provides the High-Energy Physics and broader distributed I/O bottleneck without incurring the severe resource overcomputing communities with a quantitative, policy-driven provisioning penalties of a full StepChain. methodology to dynamically identify optimal configurations A. Practical Recommendations for Workflow Engineering and balance the competing demands of data locality and To assist practitioners in navigating this trade-off space, we resource efficiency. Future Work: We plan to extend this framework for the synthesize our findings into a clear decision logic based on HL-LHC era across four primary vectors. For Topological infrastructure and workflow-specific constraints: Complexity, we will support multi-parent DAG structures (e.g., • For Network-Constrained Sites (< 100 MB/s): Highdiamond or join shapes) to accommodate complex physics progranularity compositions (e.g., TaskChain) incur processing chains. For Hardware Heterogeneity, we will integrate hibitive network overhead. Designers should favor localitysupport for heterogeneous accelerators (GPUs/FPGAs) and heavy hybrids (e.g., Const 3 or 7), which offer superior opportunistic HPC resources, where resource over-provisioning throughput compared to a StepChain while maintaining a is heavily penalized. For I/O Characterization, we plan to minimal remote I/O footprint. enrich workflow descriptions to account for unconsumed sec• For Heterogeneous Workflows: When tasksets exhibit ondary data products, tracking the precise ratio between stored high variance in resource requirements, functional decomand active downstream inputs per event. Finally, for WMS position is mandatory. Since a StepChain approach incurs Integration, we plan to integrate a lightweight framework a 3.8× throughput penalty in such scenarios, designers version into the future CMS Workload Management System to should use utilization-optimized hybrids (e.g., Const 13enable autonomic workflow composition based on site policies. 16) to enable fine-grained resource allocation. • For Time-Critical Campaigns: If makespan is prioritized R EFERENCES over global efficiency, StepChains (Const 1) remain superior for highly concurrent resource pools, as they [1] CMS Collaboration, “The CMS experiment at the CERN LHC,” Journal eliminate inter-job scheduling latencies and data-staging of Instrumentation, vol. 3, p. S08004, 2008. [2] L. Evans and P. Bryant, “LHC machine,” Journal of Instrumentation, delays. B. Scope and Limitations While DAGFlowSim is calibrated against real WLCG production runs, it is designed for comparative trade-off analysis rather than absolute metric prediction. Current limitations include a simplified failure model omitting site-correlated outages and an assumption of worker homogeneity. Nevertheless, relative performance rankings and the identification of Pareto-efficient hybrid compositions remain robust across these operational variables. Furthermore, specific compositional outcomes, such as Const 13 emerging as optimal in the seq real baseline, are strictly instance-dependent, governed by the DAG topology, taskset resource variance, and localized policy weights (wi ). As demonstrated by our seq homo and seq hetero evaluations, changing the underlying taskset requirements shifts the Pareto frontier, rendering more aggregated or decomposed configurations optimal. The primary contribution of this work is therefore not any single static construction index, but rather the optimization framework itself, which consistently identifies the superior hybrid operating point for a given workflow and site profile.
vol. 3, p. S08001, 2008. [3] I. Bird, P. Buncic, S. Campana et al., “Update of the computing models of the WLCG and the LHC experiments,” CERN, Geneva, Switzerland, Tech. Rep. CERN-LHCC-2014-014, 2014. [Online]. Available: https://cds.cern.ch/record/1695401 [4] O. Aberle et al., High-Luminosity Large Hadron Collider (HL-LHC): Technical Design Report, ser. CERN Yellow Reports: Monographs, O. Brüning, L. Rossi, A. Rossi, N. C. Moros, and K. Klein, Eds. Geneva, Switzerland: CERN, 2020, vol. 10, no. CERN-2020-010. [5] A. Malta Rodrigues, “DAGFlowSim: DAG workflow simulator,” https: //github.com/amaltaro/wflow sim, 2026, accessed: Apr. 2026. [6] CMS Collaboration, “CMS offline software and computing for HL-LHC conceptual design report,” CERN, Geneva, Switzerland, Tech. Rep. CERN-LHCC-2026-003, LHCC-G-186, 2026. [Online]. Available: https://cds.cern.ch/record/2957472 [7] E. Meschi et al., “The phase-2 upgrade of the CMS data acquisition,” EPJ Web Conf., vol. 251, p. 04023, 2021. [8] CMS Collaboration, “WMCore: Workload management core,” CERN, 2026, accessed: Apr. 2026. [Online]. Available: https://github.com/ dmwm/WMCore [9] T. Maeno, K. De, T. Wenaus et al., “Overview of ATLAS PanDA workload management,” Journal of Physics: Conference Series, vol. 331, no. 7, p. 072024, 2011. [10] A. Tsaregorodtsev et al., “DIRAC: A community grid solution,” Journal of Physics: Conference Series, vol. 119, p. 062048, 2008. [11] R. Ferreira da Silva, R. Filgueira, I. Pietri, M. Jiang, R. Sakellariou, and E. Deelman, “A characterization of workflow management systems for extreme-scale applications,” Future Gener. Comput. Syst., vol. 75, pp. 228–238, 2017.
[12] M. Turilli, V. Balasubramanian, A. Merzky, I. Paraskevakos, and S. Jha, “Middleware building blocks for workflow systems,” Comput. Sci. Eng., vol. 21, no. 4, pp. 62–75, 2019. [13] E. Deelman, D. Gannon, M. Shields, and I. Taylor, “Pegasus: A workflow management system for science automation,” Future Gener. Comput. Syst., vol. 46, pp. 17–35, 2015. [14] HTCondor Team, “HTCondor: A distributed job scheduler,” University of Wisconsin–Madison, 2026, version 24.0.6-1. [Online]. Available: https://htcondor.org/ [15] C. Thomas and D. Thain, “Liberating the data aware scheduler to achieve locality in layered scientific workflow systems,” in Proc. IEEE eScience, 2025. [16] F. Suter, R. F. Da Silva, A. Gainaru, and S. Klasky, “Driving nextgeneration workflows from the data plane,” in Proc. IEEE 19th Int. Conf. e-Science (e-Science). IEEE, 2023, pp. 1–10. [17] K. B. Antypas, D. J. Bard, J. P. Blaschke et al., “Enabling discovery data science through cross-facility workflows,” in Proc. IEEE Int. Conf. Big Data (Big Data). IEEE, 2021, pp. 3671–3680. [18] D. H. Ahn, X. Zhang, J. Mast et al., “Scalable composition and analysis techniques for massive scientific workflows,” in Proc. IEEE 18th Int. Conf. e-Science (e-Science). IEEE, 2022, pp. 32–43. [19] H. Casanova, R. Ferreira da Silva, R. Tanaka, S. Pandey et al., “Developing accurate and scalable simulators of production workflow management systems with WRENCH,” Future Gener. Comput. Syst., vol. 112, pp. 162–175, 2020.