arXiv CS open access

SEER: A Scalable Event-Driven Runtime for Parallel Execution of Multi-Agent Reinforcement Learning Environments

Jiming Su et al. · arxiv_cs

arXiv CS · Papers

cloud distributed-computing parallel-computing

Multi-agent reinforcement learning (MARL) is increasingly used in systems where many entities interact in a shared environment. In these workloads, policy inference can be batched on GPUs, but the CPU-side environment path still updates entity states, resolves local interactions, and builds observat

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

SEER: A Scalable Event-Driven Runtime for Multi-Agent Reinforcement Learning Environment Execution Jiming Su, Feng Zhu, Yuhao Xiao, Miao Zhang

Abstract Multi-agent reinforcement learning (MARL) is increasingly used in systems where many entities interact in a shared environment. In these workloads, policy inference can be batched on GPUs, but the CPU-side environment path still updates entity states, resolves local interactions, and builds observations inside each env.step. This path can dominate latency in interaction-intensive workloads after policy batching. Existing vectorized environment interfaces and general event-driven runtimes do not directly address the fine-grained cross-entity work inside one bounded MARL step while preserving the step-level semantics expected by learning systems. We present SEER (Scalable Event-Driven Environment Runtime), a runtime for MARL environment execution under an explicit event-locality contract. SEER models each environment step as a bounded stream of timestamped events. If direct state updates are entity-local and cross-entity effects are represented as ordered future events, SEER can run independent events in parallel while preserving serial-equivalent boundary outputs. To make this contract practical, SEER uses a CAS-based dual-layer event queue: local event queues preserve per-entity causal order, while a parallel executive list exposes ready work to global worker threads. SEER further uses STPA, a preflight-guided thread-pool selection mechanism that combines short runtime traces with static workload and hardware features. We evaluate SEER on two parameterized MARL workloads: compute-intensive UAV swarm control (Aircraft) and communication-intensive wireless management (PCS). On Aircraft, SEER reduces inference latency by 16.2% over the baseline. On PCS, SEER reduces environmentexecution latency by 46.9%, 19.5%, and 64.9% over ROSS-T, NBSE, and ROOT-Sim, respectively. These results suggest that fine-grained event scheduling and workload-aware thread selection can reduce MARL environment-execution overhead in the evaluated interaction-intensive workloads.

Multi-agent reinforcement learning (MARL) increasingly supports large shared environments, including robot swarms, wireless networks, transportation systems, and digital twins (Vinyals et al. 2019; Jaderberg et al. 2019; Luong et al. 2019; Orr and Dutta 2023). After GPU policy batching, CPU-side state updates, interaction processing, and observation construction can become the dominant remaining cost (Weng et al. 2022). We study how to parallelize this intra-environment execution within one shared MARL step. Copyright © 2027, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.

Agents (Running on GPU) Agents 1 Policy 1

Environment (Running on CPU)

env.step(At ) Action

Agents 2

Entity 2

Policy  2

Read action

Update local state

Entity 1

3 Process interactions

...

arXiv:2611.07996v1 [cs.DC] 5 Nov 2026

NUDT jimings@nudt.edu.cn,

Agents n Policy  n

Reward Observation

Entity m 4

Aggregate reward and next observation

Figure 1: MARL environment-execution workflow.

Vectorized environments parallelize independent instances but do not expose irregular interaction work inside one shared env.step(At ) (Terry et al. 2021; Weng et al. 2022). Such work includes entity updates, collision or contention handling, message processing, and observation construction. The key problem is therefore to expose intra-step parallelism while preserving the learning-loop outputs for environments that satisfy an explicit event-locality contract. Fig. 1 shows the execution pattern. After agents produce actions, the CPU updates entity states, processes interactions, and constructs rewards and observations. Conventional traversal preserves determinism but serializes many entitylocal operations that could execute concurrently. Generic parallel discrete-event simulation does not directly satisfy the learning-facing boundary of a MARL step: the next rewards, termination flags, and observations must preserve the original environment semantics. SEER therefore adopts an event-locality contract in which direct writes are entity-local, cross-entity effects are represented as ordered events, and outputs are emitted only after the in-step event window completes. SEER separates semantic order, work sharing, and resource selection. Entity-local queues preserve causal order, a global ready-event list exposes executable work, and preflight-guided thread selection chooses a suitable degree of parallelism. Unlike environment-level vectorization or openended PDES, SEER targets bounded execution inside one shared MARL step.

Aircraft

15.0

Per-step latency (ms)

• We identify intra-environment execution as a major postbatching bottleneck and define an event-locality contract for entity-causal parallel execution. • We design a CAS-based dual-layer queue and dynamic scheduler that separate local causal order from global ready-work sharing. • We use preflight traces and static workload features to predict a suitable thread-pool size for short MARL executions. • We evaluate throughput, latency, load balance, thread prediction, and component contributions on Aircraft and PCS against ROSS-T, NBSE, and ROOT-Sim.

17.5

Obs gather GPU policy infer CPU env advance Reward / next-obs

PCS 84.9%

83.4%

12.5 10.0

84.0%

7.5 5.0

82.9% 83.8%

206

400

800

1261

1426

Obs gather GPU policy infer CPU env advance Reward / next-obs

84.0%

40 83.1%

83.1%

20 10

2.5 0.0

Per-step latency (ms)

This paper makes the following contributions:

81.9% 81.5%

800

1600

2500

Number of entities

(a)

(b)

3600

4900

Figure 2: Single-step inference-time distribution in representative MARL environment scenarios. The policy network is MAPPO (Yu et al. 2022), and one agent corresponds to one environment entity.

Problem Model and Motivation This section defines the event model used by SEER, measures where time is spent in representative MARL inference steps, and turns these observations into design requirements.

Event-Based Model of MARL Environment Execution We model the CPU path as a bounded event-processing problem. The environment contains N entities L = {L1 , . . . , LN } with states si ∈ Σi , while M agents produce joint action At . The environment maps At to entity inputs and transforms (St , At ) into (St+1 , Rt , Dt , Ot+1 ) through events:

∀i, Li consumes events in Qi in nondecreasing order of κ. (4) LCC orders events only within the same entity. Under the event-locality contract, concurrently executed events modify disjoint entity states, while every cross-entity effect is materialized as an ordered event before it can update the target entity. Therefore, reordering independent events does not change the state observed at the step boundary. SEER claims serial-equivalent boundary outputs only for workloads that satisfy this contract and complete all in-step events before emitting outputs.

Motivation e = ⟨τ, ι, η, π, seq⟩,

τ ∈ T , ι ∈ {1, . . . , N }, η ∈ H, seq ∈ N. Observation 1. CPU-side environment execution dominates (1) the measured representative MARL inference steps (Weng Here, τ is the timestamp, ι is the target entity, η is the et al. 2022). In two MAgent2 scenarios (Zheng et al. 2018; event type, π is the payload, and seq is a deterministic Terry et al. 2021), GPU policy inference in an 800-UAV global sequence number. SEER orders events by κ(e) = cooperative-control task takes 1.32 ms, while CPU environ(τ (e), seq(e)) and stores the pending events of each entity in ment advancement takes 7.26 ms and accounts for 84.0% of Qi . The local transition is: step latency (Fig. 2(a)). In a denser wireless-communication scenario, the CPU-side fraction rises from 81.5% to 86.0% as δi : Σi × Ei → Σi × 2E . (2) the number of entities increases from 800 to 4900. Batched policy inference remains comparatively stable. These meawhere Ei contains events targeting Li . Function δi updates surements show that environment updates and cross-entity only Li and may generate new events. Cross-entity effects interactions are the main remaining bottleneck after policy must be encoded as future ordered events or explicit conflictbatching. resolution events. Generated events satisfy κ(e′ ) > κ(e); Observation 2. Event-based execution exposes useful environments with direct shared writes outside this contract intra-step parallelism (Fujimoto 1990; Weng et al. 2022). are not covered by SEER’s equivalence claim. In this motivation study, we replace the MAgent2 backend The t-th environment step is a bounded execution with ROSS-T while keeping the motivation-study scenarios window. SEER encodes each input ui,t as eact = i,t and resource configurations unchanged. Across 10 parame⟨τt , i, ACT, ui,t , seq⟩, executes ready events in [τt , τt+1 ), terized configurations, ROSS-T improves end-to-end MARL and emits outputs at τt+1 : inference throughput by 1.03–1.22× over MAgent2 (Fig. 3). [ These experiments are designed only to verify the existence N Stept (St , At ) = Outτt+1 Exec[τt ,τt+1 ) St , Qi ∪{eact . of intra-step parallelism and use workload configurations i,t }i=1 different from those in the main experimental evaluation. i (3) Accordingly, the ratios in Fig. 3 should not be directly comHere, Exec consumes in-step events in nondecreasing κ pared with the absolute environment-execution throughput order, and Out executes the boundary output events. Coror single-thread-normalized speedups reported in the main rect parallel execution follows the Local Causality Constraint evaluation. The event scheduler introduces about 2% over(LCC) (Fujimoto 1990; Jefferson 1985): head, but its net benefit generally increases with entity count

1.30

1.07×

1.075 1.050

1.05×

1.06×

1.03×

1.025 1.000 0.975

MAgent2 wins

0.950 206

400

800

Number of entities

(a)

1261

1426

1.22× 1.20×

1.20

1.17×

1.15 1.10×

1.10 1.05 1.00

Key insight. SEER parallelizes entity-local event execution rather than the entire shared state. Entity queues preserve semantic order, a global ready list exposes parallel work, and preflight traces determine the degree of parallelism.

0.98×

MAgent2 wins

0.95 10

100

200

Event time

300

400

(b)

Figure 3: End-to-end MARL inference-throughput ratio between ROSS-T and MAgent2 in the motivation-study workloads. The ratio is computed as ROSS-T throughput divided by MAgent2 throughput under the same motivation-study configuration. Event time denotes the average execution time of events within one environment step. These workloads differ from those used in the main evaluation.

and average event time. This pattern indicates that the environment path contains exploitable parallel work, while also showing that scheduling overhead must be controlled. Together, the two observations motivate a runtime that is more fine-grained than serial environment stepping, but more workload-aware than directly applying a conventional event-driven runtime. MARL steps are short, hot entities move over time, and cross-entity events appear frequently. A useful runtime must therefore expose parallelism without making queues, locks, or thread oversubscription the new bottleneck.

Challenges and Design Goals for Parallel Event Processing Balancing work while preserving entity order. Fixed entity-to-thread partitioning improves locality, but it performs poorly when computation-intensive entities shift over time (Fujimoto 1990). Work stealing reduces idle time, but it can add state-migration overhead (Blumofe and Leiserson 1999). SEER therefore needs global event scheduling under entity-local order. Avoiding queue contention during dynamic scheduling. A single global queue enables flexible work assignment, but frequent cross-entity events can cause concurrent enqueue/dequeue conflicts and atomic-write contention (Michael and Scott 1996). Such a queue can also mix same-entity events in an unsafe order. SEER must separate local causal ordering from global work sharing. Predicting the useful thread scale. Too few threads leave parallelism unused. Too many threads increase context switching, cache invalidation, memory-bandwidth pressure, and synchronization overhead (Hill and Marty 2008; Pusukuri, Gupta, and Bhuyan 2011; Popov, Jimborean, and Black-Schaffer 2019). Because the best thread count depends on entity scale, event granularity, interaction structure, and hardware hierarchy, SEER needs low-overhead thread prediction before formal execution begins (Pusukuri, Gupta, and Bhuyan 2011; Popov, Jimborean, and Black-Schaffer 2019; Alcaraz et al. 2022).

CAS-Based Dual-Layer Event Queue SEER uses a distributed CAS-based dual-layer event queue to support fine-grained event scheduling (Fig. 4). Each schedulable entity (SE) encapsulates an environment object, owns its state buffer, and consumes its local event sequence in timestamp order. Direct cross-entity state writes are not allowed under the event-locality contract. Cross-entity effects must be expressed as events, which keeps the state boundary explicit and makes parallel execution easier to reason about. Distributed Local Event Queue (LEQ). Each SE owns an LEQ for events addressed to that entity. The LEQ preserves the entity-local order required by LCC and reduces pressure on a single global structure. Multiple workers may concurrently generate events for the same target entity, while ready-event publication may overlap with insertion; therefore, LEQ operations use atomic synchronization rather than assuming single-threaded access. Parallel Executive Event List (PEL). Ready events are published from LEQs to the PEL. The PEL contains executable events rather than all pending events. Worker threads therefore compete only for work that can currently run, while local queues continue to enforce per-entity order. A worker retrieves a PEL event, executes it against the target SE’s state buffer, and inserts generated events into the corresponding target LEQs. SE state is not migrated between workers; only event execution is reassigned. This enables global load sharing while preserving entity-local causality, although it may trade some cache locality for better load balance. SE eventa

SE State buffer

eventd

eventb eventx

eventx eventy

State buffer

eventc

Event handler

...

1.100

Design

DES wins

800 entities (3 trials)

1.25

eventy

eventz

LEQ

eventz

PEL

Get eventa

Get eventb

Get eventc

Execute event new event

• • •

DES wins

• • •

Event time = 50 μs (3 trials)

1.125

Throughput ratio (ROSS-T / MAgent2)

1.150

Figure 4: SEER CAS-based dual-layer event queue for MARL environment execution.

Dynamic Parallel Scheduling Algorithm Algorithm 1 summarizes dynamic scheduling. Each SE maintains ordered events in its LEQ, and only executable

events enter the PEL for worker assignment. The runtime drains the PEL by executing events and inserting generated events into target LEQs. When the PEL is empty and all dispatched events at the current timestamp have completed, the runtime advances tcur to the minimum timestamp remaining in the LEQs and republishes the next ready events. The loop preserves LCC because an entity publishes only its next ready timestamp. At the same time, all published events are visible to the worker pool, which lets idle threads process ready work from other entities. Algorithm 1 Dynamic Parallel Event Scheduling. Description: N , number of SEs; entity i ,0 < i ≤ N , the i-th SE; eventij , the j-th environment event of SE entity i ; tij , timestamp of environment event eventij ; tcur , current environment clock; tend , environment end clock; LEQi , LEQ for SE entity i ; T P , the thread pool; threadi , the i-th thread. 1: while P EL is not empty or active workers exist do 2: if P EL is not empty and T P is not empty then 3: dequeue an available threadi from T P 4: dequeue a ready eventab from P EL 5: execute eventab on threadi and obtain generated event set G 6: for each eventxy ∈ G do 7: enqueue eventxy into LEQx 8: end for 9: release threadi and enqueue it into T P 10: else 11: wait until an active worker completes 12: end if 13: end while

Preflight-Guided Thread-Pool Selection The best thread-pool size depends on both static workload attributes and dynamic runtime behavior. Static attributes, such as SE count, event granularity, and hardware resources, describe the available parallel work and execution capacity. Dynamic signals, such as contention, critical-section access, context switching, and cache behavior, describe the current cost of using more threads. STPA is a one-shot configuration mechanism for short MARL environment steps. It collects short preflight traces, learns a temporal representation of runtime behavior, fuses this representation with static parameters, and predicts the thread-pool size used for formal execution. SEER monitors environment scale, context switching, cache hit rate, and memory consumption. Because perthread metrics change dimension with the thread-pool size, STPA aggregates runtime metrics by mean, variance, and range, X(t) = [µ, δ 2 , R] ∈ R3 . This yields a feature tensor F ∈ RN ×T ×3 for N metrics over T preflight steps. Monitoring is non-intrusive: pidstat reads thread-level OS counters, and perf collects PMU/hardware-counter signals. The temporal representation module uses TimesURL (Liu and Chen 2024), and the regression module uses XGBoost (Chen and Guestrin 2016).

The temporal encoder is trained with contrastive representation learning and masked reconstruction: L=

1 X (i,t) (i,t) (ℓ + ℓinst ) + αLrecon . |B|T i,t temp

(5)

The learned representation is passed to XGBoost to predict the normalized thread count. This division keeps the mechanism modular: the temporal encoder summarizes short runtime traces, while the regressor combines that representation with static workload and hardware features. The preflight sampling frequency controls a direct trade-off between feature fidelity and monitoring overhead.

Implementation We implemented SEER on top of ROSS-T. The implementation contains three modules: the CAS-based dual-layer event queue (LDEQ), the dynamic parallel scheduling algorithm (DPSA), and the preflight-guided thread-pool selection mechanism (STPA), as shown in Fig. 6. The modules have separate roles. LDEQ stores ordered local events and publishes ready work. DPSA assigns ready events to worker threads. STPA selects the thread-pool size before formal execution. ROSS-T base runtime. The baseline is ROSS-T (Jagtap, Abu-Ghazaleh, and Ponomarev 2012). It binds threads to CPU cores, lets each thread process a private SE task queue, and follows conservative time management. SEER keeps this event-driven execution base but replaces static work binding with the LDEQ-DPSA scheduling path. LDEQ-DPSA. Events are consumed by multiple worker threads. Instead of lock-based queues, LDEQ-DPSA uses atomic operations, including compare-and-swap, to support concurrent enqueue and dequeue operations under high load (Michael and Scott 1996). The Get interface retrieves ready events from the PEL, the Execute interface runs event handlers, and generated events are returned to the target LEQs for later publication. Idle workers block on the runtime’s worker-queue synchronization primitive and are awakened when new ready events are published, avoiding continuous busy waiting. STPA. STPA predicts the required thread-pool size from preflight event-arrival rates, system load, and resource utilization. It prevents two common failure modes: insufficient parallelism when too few workers are used, and excessive overhead when too many workers compete for queues and shared resources. STPA assumes that the short preflight phase is representative of the following formal execution. It does not continuously resize the pool within an episode; substantial workload phase changes may therefore reduce prediction quality. At runtime, SEER forms two coupled loops. The queuemanagement loop moves ready events from LEQs to the PEL and recirculates generated events. The monitoringprediction-adjustment loop observes short preflight traces and configures the worker pool. This modular design keeps semantic ordering, work dispatch, and resource selection separate, which makes the runtime easier to extend and evaluate.

pre-process Simulation Application

Time Series Representation Model

runtime information program analysis

feature fusion optimal thread pool size

Regression Model

static variable Feature Fusion

Temporal Representation Model(TimeURL)

Regression Prediction Model(XGBoost)

Encoder

Mask

Decoder

Pooling

FTAug

Pooling

cropping FreqMix

Pooling

Residual Contrastive Learning

Ldual

Time Reconstruct

Lrecon

entity_num taskque_mean machine cores

Result

Residual

Result

Optimal Thread Pool Size

Figure 5: STPA thread-pool prediction for event-driven MARL environment execution. predicting TP size continuously monitoring the runtime state

STPA

ROSS-T

Event queue management

LDEQ-DPSA

Get eventc

Execute eventb new event

Execute eventc new event

• • •

Get eventb

worker thread

• • •

Schedule worker thread

Figure 6: Architecture of the environment-execution runtime.

worker thread

scalable allocating the WT resource

SEER

event-driven

Experimental Evaluation We evaluate whether SEER improves MARL environment execution under the event-locality contract and without changing the policy network. The evaluation covers two hardware platforms, compute-intensive and communication-intensive workloads, end-to-end inference throughput, environment-execution latency, load balance, thread-selection quality, and ablations.

Experimental Design Hardware platforms. We conduct the evaluation on two Linux-based multicore platforms with different processor scales and memory-system characteristics. The two platforms allow us to evaluate SEER under different levels of available hardware parallelism and thread oversubscription. Unless otherwise stated in a figure caption, each group of comparisons is performed on the same platform using identical affinity, compiler, and measurement settings. Workloads. PCS, the Personal Communication Sys-

tem (Luong et al. 2019), is a communication-intensive benchmark in which agents represent communication nodes and generate five task types on a network topology. Aircraft is a compute-intensive UAV navigation and collision-avoidance benchmark using paths derived from real flight data (Orr and Dutta 2023). By varying entity scale, task complexity, and interaction frequency, the two parameterized workloads test SEER under different event granularities and contention patterns. Comparison systems. SEER w/o STPA enables the duallayer event queue and dynamic parallel scheduling while using a manually specified thread-pool size. SEER denotes the complete system, which additionally enables STPA to select the thread-pool size from preflight traces and static workload features. We compare them with ROSS-T, which statically binds entities to threads; NBSE, which uses one global queue; and ROOT-Sim, which uses optimistic execution and dynamic rebinding (Fujimoto 1990; Jefferson 1985; Michael and Scott 1996). All comparisons within the same figure use the same workload configuration and hardware platform. Metric definitions. Environment-execution throughput is reported as the average number of completed environment steps per second. Unless otherwise specified, speedup is computed by normalizing the throughput of a parallel configuration to that of the corresponding single-thread configuration under the same workload, platform, event handlers, and scenario parameters: Speedup(m) =

Throughput(m threads) . Throughput(1 thread)

(6)

Therefore, absolute throughput values such as 16.92 and 41.22 steps/s should not be interpreted as speedup ratios. The motivation study in Fig. 3 uses a different workload configuration and directly compares end-to-end inference throughput between ROSS-T and MAgent2; its 1.03–1.22× ratios are therefore not directly comparable with the absolute environment-execution throughput reported in the main experimental evaluation.

Aircraft(non-communication), n=400

Aircraft(non-communication), n=800

ROSS-T(32) ROSS-T(64) SEER w/o STPA(32) SEER w/o STPA(64) SEER

Throughput (steps s− 1 )

100

ROSS-T(32) ROSS-T(64) SEER w/o STPA(32) SEER w/o STPA(64) SEER

200

300

400

500

600

700

100

800

200

300

400

500

600

700

800

Event time (μs)

(b)

(a)

Figure 7: Absolute environment-execution throughput of ROSS-T, SEER w/o STPA, and SEER under different event granularities in the Aircraft workload on Platform A. Throughput is measured in environment steps per second; the reported values are not speedup ratios. Aircraft(non-communication), t=200μs

Aircraft(non-communication), t=500μs

ROSS-T(64) SEER w/o STPA(64) SEER

Single-step latency (ms)

160

120

240

ROSS-T(112) SEER w/o STPA(112) SEER

160

0 206

400

800

1261

Entity count

(a)

1426

206

400

800

1261

1426

Load Balance and Thread Prediction Sources of load balancing. In PCS with 3600 entities and 10 µs events, CPU-utilization variance drops from 23.79 with ROSS-T to 2.59 with SEER w/o STPA. SEER also improves single-thread-normalized speedup by 44.4% over ROSS-T and 33.8% over SEER w/o STPA. The per-thread distribution in Fig. 11 supports the intended effect of local queues and global ready-work sharing. Thread selection. The useful thread count depends on event granularity, interaction intensity, and platform resources. In PCS with 3600 entities and 10 µs events, STPA’s 20-thread choice yields 5.3× the NBSE throughput of the physical-core configuration. Thread scans show a measured rise-then-fall trend, motivating preflight-guided selection. Table 1 compares STPA with TimesURL, XGBoost, LSTM, and TR+GP. STPA has the lowest test MAE, 0.3394, although the gap from its training MAE of 0.0108 indicates reduced generalization to unseen configurations. Table 1: Thread-count prediction errors of the evaluated thread-pool selection methods.

Entity count

(b)

Figure 8: Environment-execution latency of ROSS-T, SEER w/o STPA, and SEER under different entity scales in the Aircraft workload on Platform A.

Measurement protocol. Within each comparison, all systems use the same event handlers, event-generation logic, policy network, action traces, compiler settings, CPU-affinity policy, and measurement windows. We report the aggregate throughput and latency measurements obtained from the evaluated configurations. Thread-prediction accuracy is evaluated separately using the training and test errors reported in Table 1.

Overall Performance On Aircraft, average environment-execution throughput increases from 16.92 steps/s with ROSS-T to 41.22 steps/s with SEER w/o STPA and 52.93 steps/s with SEER (Fig. 7). Dynamic event sharing therefore provides a 143.63% gain, while STPA adds 28.41%. Across 10 Aircraft configurations, SEER reduces latency by 68.32% over ROSS-T and 31.17% over SEER w/o STPA (Fig. 8), consistent with the roles of dynamic event sharing and thread selection. Figure 9 shows that physical- or logical-core-sized pools are not consistently optimal. With 800 entities and 400 µs events, SEER achieves a 2.81× single-thread speedup using four workers. Across the measured configurations, STPA improves speedup by 14.52% and reduces latency by 16.17% over SEER w/o STPA. End-to-end PCS results confirm that thread selection affects the full MARL loop (Fig. 10). At n = 3600 and t = 10 µs, SEER w/o STPA with 20 threads reaches 5.3× its 112-thread throughput; the corresponding ROSS-T ratio is 2.4×.

Model LSTM XGBoost TimesURL TR+GP STPA

Training Set MAE MSE 8.9976 155.5296 0.5525 0.6449 1.8344 13.7815 2.6535 18.3720 0.0108 0.0109

Test Set MAE MSE 9.3542 161.8647 1.2832 5.1508 2.2386 16.6097 3.6187 31.2561 0.3394 4.5944

Ablation Analysis ROSS-T uses static binding, SEER w/o STPA adds the duallayer queue and dynamic scheduling, and SEER additionally enables STPA. Their average Aircraft throughputs are 16.92, 41.22, and 52.93 steps/s, respectively, showing that dynamic event sharing provides the larger gain and STPA adds 28.41%. On PCS, SEER reduces latency by 46.9%, 19.5%, and 64.9% over ROSS-T, NBSE, and ROOT-Sim. It uses 1.71 GB on average, about 1/47 of ROOT-Sim and 0.44 GB more than ROSS-T.

Related Work Parallel MARL environments. Vectorized runtimes such as EnvPool parallelize execution across independent environment instances (Weng et al. 2022). SEER instead targets event-level parallelism among interacting entities within one shared environment step; the two approaches are complementary and rely on different execution abstractions. This work relates to two lines of research: load balancing in event-driven execution and resource configuration for parallel runtimes. Load balancing. Load balancing is a central problem in event-driven execution. Existing runtimes often bind logical processes or environment entities to processing units, and then rely on object migration (Trabes, Wainer, and Gil-Costa 2023; Cárdenas, Arroba, and Risco-Martín 2025; Marotta, Montesano, and Quaglia 2023) or dynamic regrouping (Miao et al. 2021) to rebalance work. These designs are effective

7.5

6.0

ROSS-T(112) ROSS-T(224) SEER w/o STPA(112) SEER w/o STPA(224) SEER

2.4

2.2

2.0

1.8

Aircraft, complexity=400

Aircraft, complexity=200 1600

ROSS-T(112) SEER w/o STPA(112) SEER

1600

Single-step latency (ms)

Speedup over single thread

9.0

Aircraft, n=800

Single-step latency (ms)

ROSS-T(112) ROSS-T(224) SEER w/o STPA(112) SEER w/o STPA(224) SEER

Speedup over single thread

Aircraft, n=400

1200

800

400

0 10

100

200

300

400

Event time (μs)

100

200

Event time (μs)

300

400

(b)

(a)

ROSS-T(112) SEER w/o STPA(112) SEER

1200

800

400

0 206

400

800

1261

1426

206

400

800

Entity count

(c)

(d)

1261

1426

Figure 9: MARL environment-execution performance for Aircraft: (a–b) single-thread-normalized speedup versus average event time, and (c–d) single-step environment-execution latency versus entity scale. Speedup is computed relative to the corresponding single-thread throughput under the same workload and platform configuration. PCS, n=1600 ROSS-T(20) ROSS-T(112) SEER w/o STPA(20) SEER w/o STPA(112) SEER

150

100

ROSS-T(20) ROSS-T(112) SEER w/o STPA(20) SEER w/o STPA(112) SEER

Throughput (steps s− 1 )

PCS, t=10μs

8 0 400

900

122

160

202

Entity count

(a)

0 250

0 360

100

200

300

400

Event time (μs)

(b)

SEER

Figure 10: End-to-end MARL inference throughput of PCS on Platform B under varying event granularities and entity scales. Throughput is measured in inference steps per second.

Figure 11: Per-thread CPU-time distribution of ROSS-T, ROOT-Sim, and SEER in the evaluated PCS configuration. A more uniform distribution across worker threads indicates better load balance.

when migration is cheap, but they can be costly when entity state is large or frequently updated. Other studies improve load sharing through spatial or temporal locality (Montesano, Marotta, and Quaglia 2024), adaptive synchronization protocols (Eker et al. 2021; Piccione, Andelfinger, and Pellegrini 2023), fine-grained task scheduling (Faxen 2010; Wan et al. 2025), or lock-free data structures (Marotta et al. 2024; Contini et al. 2023; Yang et al. 2023). SEER differs by treating environment events, rather than entities, as the scheduling unit. SEER uses a distributed local-queue plus global-readylist structure to preserve per-entity order and share ready

work across threads. Resource configuration. Thread-pool size strongly affects parallel environment-execution performance, but existing event-driven runtimes provide limited support for flexible configuration across MARL workloads (Pellegrini, Vitali, and Quaglia 2011; Carothers, Bauer, and Pearce 2002). Setting the thread count equal to the number of physical cores (Lorenzon and Beck Filho 2019; Shoshany 2024) is simple, but it can be suboptimal when event granularity or interaction intensity changes. Performance modeling techniques (Hafezan et al. 2021; Su et al. 2024; Zhao et al. 2015) analyze the relation between thread count and execution time, but simplified models may miss queue contention and cache effects in event-driven workloads. Online tuning methods (Freire, Frantz, and Roos-Frantz 2020; Bahadur, Ahmad, and Algarni 2025; Freire et al. 2021) can adjust thread counts, but iterative tuning is better suited to longrunning services than to short environment steps. Profilingbased studies (Chheda et al. 2024) and machine-learning approaches (Gao et al. 2024; Akash et al. 2021) predict resource parameters from application features. SEER combines short preflight traces with static workload and hardware features, making thread-pool selection practical for the short formal executions considered in this paper.

Conclusion This paper presented SEER, an event-driven runtime for reducing CPU-side environment-execution overhead in interaction-intensive MARL inference. SEER models an environment step as ordered events under an explicit eventlocality contract, preserves correctness through entity-local causality, and exposes ready events to global worker threads through a CAS-based dual-layer queue. It further uses STPA to select a workload-aware thread scale from short preflight traces and static features. Across compute-intensive Aircraft and communication-intensive PCS workloads, SEER improved load balance, avoided excessive-thread contention, and accelerated environment execution while preserving the checked step outputs under the event-locality contract. These results suggest that single-step environment execution is an important systems target for scalable MARL inference.

References Akash, L.; Fernando, D.; Jayasinghe, M.; et al. 2021. Machine Learning Based Thread Pool Tuning via Program Analysis. In 2021 IEEE 23rd International Conference on High Performance Computing & Communications; 7th International Conference on Data Science & Systems; 19th International Conference on Smart City; 7th International Conference on Dependability in Sensor, Cloud & Big Data Systems & Application (HPCC/DSS/SmartCity/DependSys), 648–653. Alcaraz, J.; TehraniJamsaz, A.; Dutta, A.; Sikora, A.; Jannesari, A.; Sorribes, J.; and Cesar, E. 2022. Predicting number of threads using balanced datasets for OpenMP regions. Computing. Bahadur, F.; Ahmad, Z.; and Algarni, A. 2025. PoolRunner: An Extensible Performance Testing Simulation Tool for Thread-Pool Middleware. IEEE Access. Blumofe, R. D.; and Leiserson, C. E. 1999. Scheduling multithreaded computations by work stealing. Journal of the ACM, 46(5): 720–748. Cárdenas, R.; Arroba, P.; and Risco-Martín, J. L. 2025. LockFree Simulation Algorithm to Enhance the Performance of Sequential and Parallel DEVS Simulators in Shared-Memory Architectures. Journal of Parallel and Distributed Computing, 105105. Carothers, C.; Bauer, D.; and Pearce, S. 2002. ROSS: A high-performance, low-memory, modular Time Warp system. Journal of Parallel and Distributed Computing, 62(11): 1648–1669. Chen, T.; and Guestrin, C. 2016. A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 785– 794. Chheda, S.; Verma, G.; Tian, S.; Chapman, B.; and Doerfert, J. 2024. Evaluating Tuning Opportunities of the LLVM/OpenMP Runtime. In SC24-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis, 919–929. IEEE. Contini, N.; Ramesh, B.; Kandadi Suresh, K.; Tran, T.; Michalowicz, B.; Abduljabbar, M.; Subramoni, H.; and Panda, D. 2023. Enabling reconfigurable HPC through MPIbased inter-FPGA communication. In Proceedings of the 37th International Conference on Supercomputing, 477–487. Eker, A.; Arafa, Y.; Badawy, A.-H. A.; Santhi, N.; Eidenbenz, S.; and Ponomarev, D. 2021. Load-aware dynamic time synchronization in parallel discrete event simulation. In Proceedings of the 2021 ACM SIGSIM Conference on Principles of Advanced Discrete Simulation, 95–105. Faxen, K.-F. 2010. Efficient work stealing for fine grained parallelism. In 2010 39th International Conference on Parallel Processing, 313–322. IEEE. Freire, D.; Frantz, R.; and Roos-Frantz, F. 2020. Towards optimal thread pool configuration for run-time systems of integration platforms. International Journal of Computer Applications in Technology, 62(2): 129–147.

Freire, D. L.; Mazzonetto, A.; Frantz, R. Z.; Roos-Frantz, F.; Sawicki, S.; and Basto-Fernandes, V. 2021. Performance evaluation of thread pool configurations in the run-time systems of integration platforms. International Journal of Business Process Integration and Management, 10(3-4): 318– 329. Fujimoto, R. M. 1990. Parallel discrete event simulation. Communications of the ACM, 33(10): 30–53. Gao, W.; Zhang, X.; Huang, S.; Guo, S.; Sun, P.; Wen, Y.; and Zhang, T. 2024. Autosched: An adaptive self-configured framework for scheduling deep learning training workloads. In Proceedings of the 38th ACM International Conference on Supercomputing, 473–484. Hafezan, M.; Azari, H.; Dabaghan, A.; and Beigi, L. 2021. An Accurate Learning-Based Performance/Power Model for System-Level Design of a Multicore Multithreaded Network Processor. In 2021 4th International Conference on Information and Computer Technologies (ICICT), 149–157. IEEE. Hill, M. D.; and Marty, M. R. 2008. Amdahl’s Law in the multicore era. Computer, 41(7): 33–38. Jaderberg, M.; Czarnecki, W. M.; Dunning, I.; Marris, L.; Lever, G.; Castañeda, A. G.; Beattie, C.; Rabinowitz, N. C.; Morcos, A. S.; Ruderman, A.; Sonnerat, N.; Green, T.; Deason, L.; Leibo, J. Z.; Silver, D.; Hassabis, D.; Kavukcuoglu, K.; and Graepel, T. 2019. Human-level performance in 3D multiplayer games with population-based reinforcement learning. Science, 364(6443): 859–865. Jagtap, D.; Abu-Ghazaleh, N.; and Ponomarev, D. 2012. Optimization of parallel discrete event simulator for multi-core systems. In 2012 IEEE 26th International Parallel and Distributed Processing Symposium, 520–531. IEEE. Jefferson, D. R. 1985. Virtual time. ACM Transactions on Programming Languages and Systems, 7(3): 404–425. Liu, J.; and Chen, S. 2024. TimesURL: Self-supervised contrastive learning for universal time series representation learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 13918–13926. Lorenzon, A. F.; and Beck Filho, A. C. S. 2019. Parallel computing hits the power wall: principles, challenges, and a survey of solutions. Springer Nature. Luong, N. C.; Hoang, D. T.; Gong, S.; Niyato, D.; Wang, P.; Liang, Y.-C.; and Kim, D. I. 2019. Applications of deep reinforcement learning in communications and networking: A survey. IEEE Communications Surveys & Tutorials, 21(4): 3133–3174. Marotta, R.; Ianni, M.; Pellegrini, A.; and Quaglia, F. 2024. A conflict-resilient lock-free linearizable calendar queue. ACM transactions on parallel computing, 11(1): 1–32. Marotta, R.; Montesano, F.; and Quaglia, F. 2023. Effective access to the committed global state in speculative parallel discrete event simulation on multi-core machines. In Proceedings of the 2023 ACM SIGSIM Conference on Principles of Advanced Discrete Simulation, 107–117. Miao, Z.; Yong, P.; Mei, Y.; et al. 2021. A discrete PSO-based static load balancing algorithm for distributed simulations in a cloud environment. Future Generation Computer Systems, 115: 497–516.

Michael, M. M.; and Scott, M. L. 1996. Simple, fast, and practical non-blocking and blocking concurrent queue algorithms. In Proceedings of the Fifteenth Annual ACM Symposium on Principles of Distributed Computing, 267–275. Montesano, F.; Marotta, R.; and Quaglia, F. 2024. Spatial/temporal locality-based load-sharing in speculative discrete event simulation on multi-core machines. ACM Transactions on Modeling and Computer Simulation, 35(1): 1–31. Orr, J.; and Dutta, A. 2023. Multi-agent deep reinforcement learning for multi-robot applications: A survey. Sensors, 23(7): 3625. Pellegrini, A.; Vitali, R.; and Quaglia, F. 2011. The rome optimistic simulator: Core internals and programming model. In SIMUTools’11: Simulation Tools and Techniques, 96–98. Piccione, A.; Andelfinger, P.; and Pellegrini, A. 2023. Hybrid speculative synchronisation for parallel discrete event simulation. In Proceedings of the 2023 ACM SIGSIM Conference on Principles of Advanced Discrete Simulation, 84–95. Popov, M.; Jimborean, A.; and Black-Schaffer, D. 2019. Efficient thread/page/parallelism autotuning for numa systems. In Proceedings of the ACM International Conference on Supercomputing, 342–353. Pusukuri, K. K.; Gupta, R.; and Bhuyan, L. N. 2011. Thread reinforcer: Dynamically determining number of threads via OS level monitoring. In 2011 IEEE International Symposium on Workload Characterization. Shoshany, B. 2024. A C++ 17 thread pool for highperformance scientific computing. SoftwareX, 26: 101687. Su, J.; Yao, Y.; Zhu, F.; Yuan, H.; Fang, Y.; Chen, K.; Sui, S.; Zhao, T.; Zhang, H.; and Zhang, Y. 2024. An Adaptive Approach For Parallel Discrete Event Simulation Thread Pool Prediction. In ECMS, 352–358. Terry, J. K.; Black, B.; Grammel, N.; Jayakumar, M.; Hari, A.; Sullivan, R.; Santos, L.; Perez, R.; Horsch, C.; Dieffendahl, C.; Williams, N. L.; Lokesh, Y.; and Ravi, P. 2021. PettingZoo: Gym for multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, volume 34. Trabes, G. G.; Wainer, G. A.; and Gil-Costa, V. 2023. A parallel algorithm to accelerate DEVS simulations in shared memory architectures. IEEE Transactions on Parallel and Distributed Systems, 34(5): 1609–1620. Vinyals, O.; Babuschkin, I.; Czarnecki, W. M.; Mathieu, M.; Dudzik, A.; Chung, J.; Choi, D. H.; Powell, R.; Ewalds, T.; Georgiev, P.; Oh, J.; Horgan, D.; Kroiss, M.; Danihelka, I.; Huang, A.; Sifre, L.; Cai, T.; Agapiou, J. P.; Jaderberg, M.; Vezhnevets, A. S.; Leblond, R.; Pohlen, T.; Dalibard, V.; Budden, D.; Sulsky, Y.; Molloy, J.; Paine, T. L.; Gulcehre, C.; Wang, Z.; Pfaff, T.; Wu, Y.; Ring, R.; Yogatama, D.; Wünsch, D.; McKinney, K.; Smith, O.; Schaul, T.; Lillicrap, T.; Kavukcuoglu, K.; Hassabis, D.; Apps, C.; and Silver, D. 2019. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782): 350–354. Wan, Z.; Dong, X.; Wang, L.; Zhu, E.; Gu, Y.; and Sun, Y. 2025. Parallel contraction hierarchies can be efficient and scalable. In Proceedings of the 39th ACM International Conference on Supercomputing, 670–688.

Weng, J.; Lin, M.; Huang, S.; Liu, B.; Makoviichuk, D.; Makoviychuk, V.; Liu, Z.; Song, Y.; Luo, T.; Jiang, Y.; Xu, Z.; and Yan, S. 2022. EnvPool: A highly parallel reinforcement learning environment execution engine. In Advances in Neural Information Processing Systems, volume 35. Yang, X.; Li, S.; Yuan, F.; Dong, D.; Huang, C.; and Wang, Z. 2023. Optimizing multi-grid computation and parallelization on multi-cores. In Proceedings of the 37th International Conference on Supercomputing, 227–239. Yu, C.; Velu, A.; Vinitsky, E.; Gao, J.; Wang, Y.; Bayen, A.; and Wu, Y. 2022. The surprising effectiveness of PPO in cooperative multi-agent games. In Advances in Neural Information Processing Systems, volume 35. Zhao, J.; Cui, H.; Xue, J.; et al. 2015. Predicting crosscore performance interference on multicore processors with regression analysis. IEEE Transactions on Parallel and Distributed Systems, 27(5): 1443–1456. Zheng, L.; Yang, J.; Cai, H.; Zhou, M.; Zhang, W.; Wang, J.; and Yu, Y. 2018. MAgent: A many-agent reinforcement learning platform for artificial collective intelligence. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1).

SEER: A Scalable Event-Driven Runtime for Parallel Execution of Multi-Agent Reinforcement Learning Environments

Analyze & explore related research in Conceptio

Related documents