OACM: Optimistic Asynchronous Communication Model for Large-Scale SNN Simulation

OACM: Optimistic Asynchronous Communication Model for Large-Scale SNN Simulation Guangnan Fenga , Yangle Zenga , Honghui Lianga , Zhiguang Chena and Yutong Lua,∗

arXiv:2610.04236v1 [cs.DC] 3 Oct 2026

a Sun Yat-sen University, Guangzhou, 510006, China

ARTICLE INFO

ABSTRACT

Keywords: Spiking Neural Network Brain Simulation Parallel Discrete-event Simulation Communication Optimization

Spiking Neural Network (SNN) simulation serves as a crucial tool for understanding brain dynamics and advancing neuromorphic computing, but faces significant scalability challenges in large-scale distributed implementations. The primary bottlenecks arise from frequent synchronization overhead in time-driven simulators and extensive secondary rollbacks in optimistic PDES approaches, compounded by inefficient communication patterns that underutilize network bandwidth. In this paper, we propose OACM, an Optimistic Asynchronous Communication Model that addresses these challenges through three key innovations. First, we design Optimistic Hybrid SNN Simulation that combines the implementation simplicity of time-driven approaches with reduced synchronization frequency through strategic rollback mechanisms within synchronization windows. Second, we implement asynchronous one-sided communication using the UNR library, eliminating handshake latency and achieving complete computation-communication overlap. Third, we develop adaptive message aggregation and routing strategies within a 2D-HyperX virtual topology to optimize bandwidth utilization for small message traffic. Experimental evaluation on a high-performance computing cluster demonstrates that OACM achieves up to 1.4× speedup over the original CORTEX simulator and over 22.6× speedup compared to NEST when simulating a multi-area marmoset brain model at scales of up to 174 compute nodes.

1. Introduction Spiking Neural Network (SNN) simulation is pivotal for bridging computational neuroscience and artificial intelligence, offering a biologically realistic model to decipher brain dynamics and inspire machine intelligence. The brain’s unparalleled energy efficiency and robust learning capabilities provide a crucial blueprint for overcoming key bottlenecks in modern AI, such as its massive data dependency and lack of generalization, which hinder progress toward Artificial General Intelligence. In contrast, the human brain operates with remarkable energy efficiency (approximately 20 watts), seamlessly integrates multimodal information, and exhibits robust generalization. Additionally, understanding and treating neurological disorders such as Alzheimer’s and Parkinson’s diseases necessitates a system-level investigation of aberrant neural activity, a pursuit for which largescale SNN simulation is an indispensable tool. As researchers strive to emulate SNNs at the scale of mammalian brains, which encompass 10 million to 100 billion neurons interconnected through trillions of synapses [1, 2, 3, 4], the computational complexity grows exponentially. This scaling challenge is further compounded by biologically plausible models requiring millisecond-level temporal resolution and intricate synaptic dynamics. Consequently, the development of high-performance SNN simulators capable of balancing biological fidelity with computational ∗ Corresponding author

fenggn7@mail.sysu.edu.cn (G. Feng); zengyle@mail2.sysu.edu.cn (Y. Zeng); lianghh33@mail2.sysu.edu.cn (H. Liang); chenzhg29@mail.sysu.edu.cn (Z. Chen); luyutong@mail.sysu.edu.cn (Y. Lu) ORCID (s): 0000-0002-1382-280X (G. Feng); 0009-0003-2512-9839 (Y. Zeng); 0009-0006-6509-3000 (H. Liang); 0000-0002-9318-5715 (Z. Chen); 0000-0001-5315-3375 (Y. Lu)

Feng et al.: Preprint submitted to Elsevier

efficiency has emerged as a critical frontier in computational neuroscience and neuromorphic engineering. To meet these demands, modern simulators increasingly adopt parallel acceleration paradigms, achieving remarkable progress in scalability and performance. Time-driven simulators like NEST [5], CORTEX [6], and MONET [1] have demonstrated exceptional scalability through their straightforward implementation and efficient parallelization. Eventdriven approaches and optimistic PDES techniques have shown promise in reducing computational overhead by updating only upon spike events and relaxing synchronization constraints. Communication optimizations, including asynchronous protocols [1, 7], dedicated communication threads [6], and virtual topology routing [8, 9], have significantly improved computation-communication overlap and bandwidth utilization. These advancements collectively enable simulations at unprecedented scales, bringing the field ever closer to whole-brain emulation. Despite these significant achievements, fundamental challenges persist in synchronization and communication efficiency. Time-driven simulators suffer from the overhead of frequent global synchronization, while optimistic PDES approaches face extensive secondary rollbacks under load imbalance. Existing communication implementations, whether synchronous or asynchronous, introduce implicit synchronization overhead through handshake protocols or pre/post-synchronization requirements. Moreover, the irregular communication patterns in SNN simulations generate numerous small messages that severely underutilize network bandwidth. These bottlenecks collectively limit the scalability and efficiency of large-scale SNN simulations, demanding more sophisticated approaches to synchronization and communication. Page 1 of 12

OACM: Optimistic Asynchronous Communication Model for Large-Scale SNN Simulation

To address these challenges, we propose OACM, an Optimistic Asynchronous Communication Model for largescale SNN simulation, and make the following contributions: • We design Optimistic Hybrid SNN Simulation, avoiding over-synchronization in time-driven simulators and the spread of secondary rollbacks in Optimistic PDES.

(a) Neuron

• We implement asynchronous one-sided communication, increasing computation-communication overlap and eliminating handshake latency. • We develop adaptive message aggregation and routing strategies within a 2D-HyperX virtual topology to optimize bandwidth utilization for small message traffic.

(b) Population

(c) Area

• We utilize up to 174 compute nodes of an HPC cluster to simulate a multi-area SNN model of the marmoset brain, achieving up to 1.4× speedup. The remainder of this paper is structured as follows. Section 2 provides the background and related work on SNN synchronization and communication. Section 3 details the rollback mechanism. Section 4 outlines the asynchronous communication utilizing UNR. Section 5 introduces our strategy of message routing, aggregation, and process mapping. Section 6 presents the experimental studies. Section 7 and Section 8 summarize our contributions and prospects for future work.

(d) Brain Figure 1: The components of the SNN model. (a) Neurons handle and generate spikes, which are transmitted by synapses. (b) Neurons with similar biological features are grouped together to form populations. (c) Areas are composed of many populations and are responsible for brain functions. (d) The brain comprises multiple interconnected areas, forming a complex network structure.

2. Background and Related Work This section provides the necessary background to contextualize our work. Section 2.1 introduces the foundational SNN model, explaining why synchronization and communication become bottlenecks in large-scale simulations. We then review existing SNN simulators, categorized into synchronization strategies (Section 2.2) and communication mechanisms (Section 2.3).

2.1. SNN Simulation The SNN model can be depicted in a hierarchical view, as illustrated in Figure 1. Neurons and synapses are two basic components of the SNN. Neurons, the fundamental building blocks of SNNs, mimic the functionality of biological neurons. They process incoming spike signals and determine whether to generate output spikes based on changes in their membrane potential. Synapses serve as the communication channels between neurons, facilitating the transmission of spike signals. Each synapse has a weight that modulates the signal’s strength. From a more macroscopic perspective, the SNN model is composed of populations and projections [10]. Populations are made up of groups of neurons that share similar properties. Projections define the synaptic connections between these populations. Brain areas, which are larger functional units composed of multiple distinct populations, correspond Feng et al.: Preprint submitted to Elsevier

to the diverse functional regions in biological brains, such as the visual cortex. These areas are also interconnected through synapses, forming a complex network structure that enables the brain to carry out advanced informationprocessing tasks. SNN simulations involve detailed modeling of the interactions between neurons, populations, and brain areas via synapses, as well as the monitoring of inputs and outputs. In large-scale SNN simulations, the key challenges arise primarily from frequent synchronization and irregular communication patterns, which can profoundly impact the parallel efficiency of simulations [11, 7]. The need for frequent synchronization stems from the fact that, within each effective refractory period, every neuron must confirm the arrival of all spike signals sent by its input synapses in the previous period before it can proceed, effectively serializing the computation across dependent processes. Meanwhile, the communication pattern is inherently irregular because only a small fraction of neurons fire spikes in a period, and the specific target neurons connected via output synapses differ for each sender, leading to unpredictable and finegrained communication. These application characteristics collectively result in severe performance bottlenecks, manifesting as frequent processor idle time due to synchronization waits and a flood of small, inefficient network packets Page 2 of 12

Init

Process1

Barrier & Step+1

Barrier & Start

N1 N2 N3 T0 T0 T0 Init

N1 N2 N3 T1.2 T1.5 T1.4

Barrier & Start

Barrier & Step+1

Barrier

Barrier & Step+1

N1 T3.8

N2 N2 T5.1 T5.2

Process2 Process1

Optimistic Asynchronous Large-Scale N2 SNNN2 Simulation N4 N5 N6 N5 Model forN1 N6 N5 N4 N2 N5OACM: N6 N1 N2Communication N3 N1 N3 T0.1 T0.2 T0.3 T2.7 T6.6 T0 T0 T0 T1.2 T1.5 T1.4 T3.8 T5.1 T5.2

Process3 Process2 Process1

N7 N5 N8 N6 N9& Start N4 N5 N6 N4 N1 T0 N2 T0 N3 T0 T0.1 T0.2 T0.3 T0 T0 T0

Process3 Process2

N7 N8 N9 N4 T0 N5 T0 N6 T0 T0 T0 T0

Process3

N7 N8 N9 T0 T0 T0

Process1

N1 N2 N3 T0 T0 T0

Process2 Process1

N4 N2 N5 N3 N6 N1 T0 T0 T0

N4 N5 N1N6 N2 N5N3 T0.1 T0.2T1.2 T0.3T1.5T2.8 T1.4

Process3 Process2 Process1

Process3 Process2

Init

N4 N5 N6 T0.1 T0.2 T0.3

N1

N2

N7 N8 N9 N1 T1.4 N2 T1.4 N3 T1.3 T1.2 T1.5 T1.4

N7 N9 N5 T2.6 T2.6 T2.7

N8 N6 N1 T3.4 T3.8 T3.8

N7 N2 N2 T5.3 T5.1 T5.2

N8 N5 T6.3 T6.6

N9 N7 T7.7 T7.8

N7 N8 N9 T1.3 T1.4 T1.4

N7 N9 N5 T2.6 T2.6 T2.7

N8 N6 T3.4 T3.8

N7 T5.3

N8 N5 T6.3 T6.6

N9 N7 T7.7 T7.8

N7 N8 N9 T1.3 T1.4 T1.4

N7 N9 T2.6 T2.6

N8 T3.4

N7 T5.3

N8 T6.3

N9 N7 T7.7 T7.8

N3

N1

N1

N2

N3

N2

N3

N1 N2 N6 N3 T3.8 T5.2 T3.8 T5.3

N5N2 T6.7 T5.1

N3 T5.2

N5 T6.6

N7 N5 N8 N6 N9 N4 N1 T0 N2 T0 N3 T0 T0 T0 T0

N7 N7 N4 N5 N7N6 N8 N5N9 N5 N8 N6 N9 N6 N8 N5 N7 N1 N2 T2.8 N3 T2.6 N1 T3.4 N1 T5.4 N2 T3.8 N3 T6.4 T6.7 N2 T1.4 T5.3 T0.1 T0.2T1.3 T0.3T1.4 T2.7 T3.9 T2.6 T1.2 T1.5 T1.4 T3.9 T3.8 T5.2 T5.3 T5.1 RollBack

N9 N3 T7.8 T5.2

N8 N5 N7 T6.3 T6.6 T7.9

N9 T7.7

N7 T7.8

N7 N8 N9 N4 T0 N5 T0 N6 T0 T0 T0 T0

N7 N8 N9 N7 N8 N9 N7 N4 N5 T1.3 N6 T1.4 N5 N5 T3.4 N6 T2.6 T5.4 T1.4 T2.6 T0.1 T0.2 T0.3 T2.8 T2.7 T3.9

N9 T7.8

N8 N7 T6.3 N5 T7.9 T6.6

N9 T7.7

N7 T7.8

T1.2 T1.5 T1.4 T3.8Synchronize T5.2 T5.3 after T5.1 T5.2 (a) Time-Driven SNNT3.9 Simulation: each time step. N5 N1 N6 T2.7 T3.9 T3.9

Event

N8 N7 N6 T6.4 N5T5.3 T3.8 T6.7 RollBack Event

N7 N8 N9 SNN N7 N8 N9 N7No synchronization N8 N7 N9rollback N8 N7 N9 N7 N8 N9 Parallel Distributed Simulation: occur. Process3 (b)N7 T0Optimistic T0 T0 T1.3 T1.4 Event T1.4 T2.6 T3.4 T2.6 T5.4 T6.4 T5.3 and T7.8 T6.3when T7.9errors T7.7 T7.8

N1 T3.9

N1 T3.8

N5 N1 N6 T2.7 T3.9 T3.9

N1 T3.8

RollBack Event N2

Process1

N1 N2 N3 T0 T0 T0

N1 N2 N3 T1.2 T1.5 T1.4

N2 T5.1 T5.2

Process2 Process1

N4 N2 N5 N3 N6 N1 T0 T0 T0

N4 N5 N1N6 N2 N5N3 T0.1 T0.2T1.2 T0.3T1.5T2.8 T1.4

Process3 Process2 Process1

N7 N5 N8 N6 N9 N4 N1 T0 N2 T0 N3 T0 T0 T0 T0

N7 N4 N5 N7N6 N8 N5N9 N5 N8 N6 N9 N1 N2 T2.8 N3 T2.6 N1 T3.4 N1 T1.4 T0.1 T0.2T1.3 T0.3T1.4 T2.7 T3.9 T2.6 T1.2Synchronization T1.5 T1.4 Window T3.9 T3.8Empty Msg

Process3 Process2

N7 N8 N9 N7 N8 N9 N8 N7 N8 N9 N7 N7 N8 N9 N4 N5 T1.3 N6 T1.4 N5 N5 T3.4 N6 T2.6 N6 N4 T0 N5 T0 N6 T0 T1.4 T2.6 T3.4 T6.3 N5 T7.7 T7.8 (c) Optimistic Hybrid Simulation (ours): Synchronize every 𝑘 stepsT5.3 and rollback within window. T0 T0 T0 T0.1 T0.2SNN T0.3 T2.8 T2.7 T3.9 T3.8 T6.6Emptyeach Empty Msg Msg Synchronization Window

& Step+3

& Step+3

N6 T3.8

N2 N2 T5.1 T5.2

N5 T6.6

N8 N6 T3.4 T3.8

N7 N8 N9 N7 N5 N2 N2 T6.3 T6.6 T5.3 T7.7 T7.8 T5.1 T5.2 Empty Msg & Step+4

& Step+4

Figure 2: Synchronization Mechanisms Time-Driven, andN8 OACM N7 SNN Simulation. N# represents N7 inN8 N9 N7 Event-Driven, N8 N9 N8 N9 N7 Neuron#, and T# N7 N8 N9 Process3 T0 T0 T0 T1.3 T1.4 T1.4 T2.6 T3.4 T2.6 T3.4 T5.3 T6.3 T7.7 T7.8 represents the spike that arrives at Virtual Time#. Empty Msg Empty Msg Synchronization Window & Step+3

that underutilize bandwidth. Therefore, existing works have conducted extensive research on synchronization and communication issues in SNN simulation.

2.2. Synchronization Mechanisms in Time-Driven and Event-Driven SNN Simulation There are two primary methods for large-scale SNN simulation: time-driven and event-driven simulation. The most evident difference between the two simulation methods is how the simulation time advances, which leads to different synchronization mechanisms. Time-driven simulators operate by advancing through fixed timesteps, which must be set smaller than the effective refractory period to ensure accurate neural dynamics, thus requiring frequent synchronization and communication as illustrated in Figure 2a. Each simulation step is typically divided into two distinct phases. In the neuron computation phase, all active neurons (i.e., not in a refractory state) integrate incoming signals and update their internal states based on the neuron model, deciding whether to emit a spike to post-synaptic neurons. In the spike propagation phase, spikes generated in the current or preceding steps are transmitted to their target synapses, taking synaptic delays into account. Synapses then update their states and deliver the resulting inputs to the post-synaptic neurons. Generally, there is an explicit or implicit global synchronization in the spike propagation phase. Currently, time-driven simulators dominate the field due to their ease of implementation [5, 12, 13, 14, 11, 15, 16, 17, 18, 1, 7, 19, 20, 21, 22, 23, 8, 3, 24, 4, 6], despite incurring substantial synchronization overhead as the simulation scale increases. Feng et al.: Preprint submitted to Elsevier

& Step+4

On the other hand, event-driven simulators offer arbitrary time resolution and advance the simulation via variable time increments. They only trigger updates upon the arrival of specific events, such as spike exchanges, eliminating the need for fixed time steps and reducing computational overhead by only updating relevant components. However, conservative event-driven SNN simulation requires an event to be processed if and only if all of its earlier events have been processed, which leads to synchronization between connected neurons to ensure correctness. Thus, most eventdriven SNN simulators are limited by their reliance on specialized neuromorphic hardware [25, 26, 27], a lack of multi-threading simulation support [28, 29, 30, 31], or no support for Parallel Distributed Event Simulation (PDES) [32, 33, 34, 35], which restricts their versatility and scalability. To improve scalability, the simulators of [36, 37] utilize optimistic PDES to relax the synchronization condition under which incorrect event computation can be rolled back. Nevertheless, the heterogeneous activity patterns across different brain regions lead to inter-process load imbalance, which may result in a large number of secondary rollbacks, as shown in Figure 2b. The simulator of [38] combines event-driven and timedriven strategies. The low-activity parts of the neural network can be simulated on CPU using event-driven methods, while the high-activity subsystems can be simulated on either CPU or GPU using time-driven methods. In contrast, OACM seamlessly integrates both strategies by leveraging sparse spiking activity and bounded synaptic delay.

Page 3 of 12

OACM: Optimistic Asynchronous Communication Model for Large-Scale SNN Simulation

2.3. Communication Implementation in SNN Simulation At present, most synchronization mechanisms in SNN simulators are implemented by MPI communication operations. Some simulators employ synchronous MPI functions, such as point-to-point communication (e.g., MPI_Send and MPI_Recv) [24, 39, 9] and collective communication (e.g., MPI_Alltoall and MPI_Allgatherv) [5, 11, 39, 6]. These functions inherently guarantee the delivery of messages and the synchronization of all processes, thereby obviating the need for additional explicit synchronization measures. The other simulators [1, 7, 3, 24] incorporate asynchronous communication protocols to achieve synchronization. These simulators harness asynchronous communication operations, such as MPI_Isend/MPI_Irecv, in conjunction with MPI_Request to track communication requests. Subsequently, synchronization is implemented by MPI_Waitall to confirm message arrival and to ensure data consistency across processes. While the former streamlines implementation by integrating synchronization within communication functions, the latter gains enhanced flexibility and control, especially in situations that demand meticulous coordination or the overlapping of computation with communication. As for communication optimization, overlapping communication with computation is a common approach: CORTEX [6] optimizes communication by using a dedicated thread for communication in each process to overlap communication and computation. This approach allows synaptic interactions to be handled independently of the computation threads, enabling efficient dataflow and spike propagation without dependencies at both the process and thread levels. MONET [1] utilizes a tile partitioning method and asynchronous point-to-point communication, where the communication interval is set to half of the minimum transmission delay, thus enabling the computation to overlap the asynchronous communication of MPI_Isend/MPI_Irecv. DTB [4] optimizes communication by employing multiple threads for parallel computation and communication, with GPUs managing state variables and spike sequences to enhance efficiency. It also implements a method to prevent communication contention by prioritizing the order of data transmission and using a sending thread to manage information flow, ensuring smooth and efficient data exchange. Other optimization techniques, such as message aggregation and routing, have also been applied in some studies. HRCM [8, 9] proposes a virtual communication topology (VCT), which organizes processes into an n-dimensional virtual topology and uses multi-stage message merge-forward to reduce the total number of messages transmitted. Du et al. [3] place neurons with strong communication demands on the same GPU and use two-level routing to optimize communication, where they cluster GPUs into groups and minimize intergroup connections. Carlos et al. [7] employ hypergraphbased partitioning to minimize inter-process communication by carefully placing neurons and introducing neighborhood

Feng et al.: Preprint submitted to Elsevier

exchange algorithms to overlap the communication handshake with the send-receive data phase. Communication, along with synchronization, has long been recognized as the most time-consuming part of distributed SNN simulations [11, 7, 40]. To tackle the synchronization and communication bottleneck, we propose a more versatile approach that does not depend on the minimum synaptic delay and incorporates communication optimizations that further minimize the synchronization frequency and enhance the communication efficiency of simulations. The subsequent sections will provide a detailed exposition of our strategies.

3. Optimistic Hybrid SNN Simulation 3.1. Design Overview Optimistic Hybrid SNN Simulation seamlessly integrates time-driven simulation and Optimistic PDES. It inherits the implementation simplicity of time-driven simulators while eliminating the need for frequent global synchronization. Simultaneously, it strategically applies Optimistic PDES techniques locally, thereby avoiding the extensive secondary rollbacks caused by load imbalance. The core design of Optimistic Hybrid SNN Simulation involves performing global synchronization every 𝑛 timesteps, which we refer to as a “synchronization window”, as shown in Figure 2c. By adopting this approach, the total number of synchronizations is reduced by a factor of 𝑛. To ensure simulation correctness, we incorporate the rollback strategy from Optimistic PDES, but confine rollbacks within each synchronization window. The pseudocode for the entire rollback procedure is presented in Algorithm 1.

3.2. Implementation To begin with, we establish a clear taxonomy of spike types, listed in Table 1. Central to this categorization is the concept of the “effective timestep” 𝑒. This term refers to the exact timestep at which the synaptic input, triggered by a spike, affects the post-synaptic neurons. The effective timestep is determined by adding the spike’s generation timestep 𝑔 to the synaptic delay 𝑑. With this concept in mind, spikes can be classified by comparing 𝑒 with the current timestep 𝑡, and by determining if they are anti-spikes: • Timely spike: Occurs when 𝑒 ≥ 𝑡. Processed immediately if timestamps match, otherwise queued for future processing. Table 1 Taxonomy of Spike Types Types Condition Action Rollback Timely spike 𝑒≥𝑡 Add No Timely anti-spike 𝑒≥𝑡 Sub No Late spike 𝑒<𝑡 Add Yes Late anti-spike 𝑒<𝑡 Sub Yes

Page 4 of 12

OACM: Optimistic Asynchronous Communication Model for Large-Scale SNN Simulation

Algorithm 1 Rollback Procedure Input: 𝑡: current timestep, 𝐵[]: received spike buffer, 𝑅[]: rollback step of neurons, 𝐷[]: synaptic delay 1: // Handle spikes 2: for each spike 𝑖 in buffer 𝐵 do 3: for each target neuron 𝑗 do 4: 𝑒 = 𝐵[𝑖].𝑔𝑒𝑛𝑇 𝑖𝑚𝑒 + 𝐷[𝐵[𝑖].𝑠𝑟𝑐][𝑗] 5: Write to neuron 𝑗’s input at timestep 𝑒 6: if 𝑒 < 𝑡 then 7: 𝑅[𝑗] = 𝑡 - 𝑒 8: end if 9: end for 10: end for 11: // Update neurons 12: for each neuron 𝑖 do 13: Set the neuron to the state of timestep (𝑡 − 𝑅[𝑖] − 1) 14: for 𝑗 from 𝑅[𝑖] to 1 do 15: Record whether neuron spikes before update. 16: Update and save neuron state of timestep (𝑡 − 𝑗) 17: Record whether neuron spikes after update. 18: if Spiked before update but not after then 19: Generate anti-spike with timestep (𝑡 − 𝑗) 20: else if Spiked after update but not before then 21: Generate spike with timestep (𝑡 − 𝑗) 22: end if 23: end for 24: Update and save neuron state of timestep 𝑡 25: Record whether neuron spikes after update. 26: if Spiked after update then 27: Generate spike with timestep 𝑡 28: end if 29: end for • Late spike: Arises when 𝑒 < 𝑡, typically due to network delays or rollback recalculations from other processes. Triggers rollback operations. • Timely anti-spike: Used to neutralize erroneous spikes that are generated before rollback. Occurs when 𝑒 ≥ 𝑡. Processed immediately or queued. • Late anti-spike: Functions similarly to Timely antispike. Occurs when 𝑒 < 𝑡 and necessitates rollback operations. • Straggler spike: Refers to both Late spike and Late anti-spike, which can cause rollbacks. Next, we manage the rollback state of each neuron in a fine-grained manner, rather than employing uniform control over the entire process. Each neuron is associated with a circular array whose length matches the size of the synchronization window. This array records the state of the neuron at every timestep within the specified synchronization window, encompassing parameters such as membrane potential, refractory period count, and synaptic input. Such fine-grained storage enables us to perform rollbacks for individual neurons rather than for all neurons within a process, avoiding Feng et al.: Preprint submitted to Elsevier

unnecessary repeated computation. Since only a subset of neurons undergoes rollback, the recalculation for timestep 𝑖 + 1 is significantly shortened. Here, for implementation, we employ the checkpoint technique rather than incremental computation. This decision is predicated on the fact that the dynamic equations used for calculating neuron states do not support incremental methods. Then, we design a mechanism to handle different spike message types. Upon receiving spikes, a process first calculates the effective timestep 𝑒 for each target neuron using the spike’s generation time 𝑔 and synaptic delay 𝑑, and then modifies that neuron’s input buffer at timestep 𝑒. For an anti-spike, the corresponding value is subtracted from that neuron’s input buffer; otherwise, the value is added. If 𝑒 < 𝑡, that neuron’s rollback step count 𝑟 is set to 𝑡 − 𝑒. During neuron state updates, a non-zero 𝑟 triggers the rollback procedure: the neuron’s state is reset to the state at 𝑡 − 𝑟 − 1, and then the state is recalculated and saved step by step from 𝑡 − 𝑟 onward. If the spike emission during the rollback does not match the history, new spikes or anti-spikes will be generated. Finally, the state at 𝑡 is calculated, and it is checked whether a new spike is generated. All generated spikes are buffered and transmitted later. Finally, the synchronization mechanism is further complicated by the introduction of rollbacks. Upon reaching the synchronization timestep 𝑠 at the end of each synchronization window, a process halts and verifies the receipt of all inter-process messages before updating neuron state at 𝑠. Spikes generated during updates at 𝑠 are retained for transmission after synchronization. The neuron update procedure calculates a local maximum rollback count 𝑟𝑚𝑎𝑥 in that process, and then computes the global maximum rollback count 𝑅𝑚𝑎𝑥 via an MPI_Allreduce operation. Next, each process reverts all the states to timestep 𝑠−𝑅𝑚𝑎𝑥 −1 and resumes simulation from timestep 𝑠−𝑅𝑚𝑎𝑥 . Synchronization serves to purge historical states stored prior to timestep 𝑠 − 𝑅𝑚𝑎𝑥 and to set an upper limit on rollback, preventing states from being reverted further than timestep 𝑠 − 𝑅𝑚𝑎𝑥 .

3.3. Design Analysis This implementation offers three key advantages: First, ease of implementation is crucial for SNN simulation, as SNN models frequently require adjustments. Using PDES for large-scale SNN simulation would require domain experts to understand complex concepts such as LP, KP, GVT, and fossil collection, whereas time-driven implementations are more comprehensible. This explains why the vast majority of known simulators employ timedriven approaches. Second, the synchronization window imposes an upper bound on the historical states that need to be stored, thereby avoiding the massive memory consumption issues in traditional Optimistic PDES under load imbalance. This is particularly important for SNN simulations, which typically require substantial memory to store connection structures. Finally, this approach eliminates the overhead of precise event time ordering, requiring only

Page 5 of 12

NIC Send NIC Send

B1 B2 B3 B4 B5 B6 B7 B8 B9 B1 B2 B3 B4 B5 B6 B7 B8 B9

UNR RDMA Wrt to Rmt UNR RDMA Wrt to Rmt

OACM: Optimistic Asynchronous Communication Model for Large-Scale SNN Simulation

bucket sorting as in time-driven simulators, rather than the exact event ordering required in Optimistic PDES.

NIC Recv B1B2B3B4B5B6B7B8B9 NIC Recv B1B2B3B4B5B6B7B8B9 CPU Comp CPU Comp

B1 B2 B3 B4 B5 B6 B7 B8 B9 B1 B2 B3 B4 B5 B6 B7 B8 B9

NIC Send NIC Send

4. Asynchronous One-Sided Communication Model 4.1. Communication Requirements and Existing Limitations Communication latency and computation-communication overlap are critical factors for SNN simulation performance. As introduced in Section 3, our Optimistic Hybrid SNN Simulation still relies on rollback operations within each synchronization window to ensure correctness, where communication delays directly impact whether spike data can arrive in a timely manner between processes. Achieving complete computation-communication overlap is therefore essential to maximize simulation efficiency. Existing approaches typically employ MPI-based asynchronous communication, such as two-sided non-blocking operations (MPI_Isend/MPI_Irecv) or one-sided operations (MPI_Put/MPI_Get). However, these implementations suffer from implicit communication overhead. Two-sided nonblocking operations require a handshake between sender and receiver (Rendezvous Protocol), or an additional memory copy (Eager Protocol). One-sided operations require presynchronization to ensure buffer or data readiness, and postsynchronization to notify the peer that data is ready. These synchronization operations are typically performed through MPI_Win_Fence, consuming at least a single trip time. Note that the synchronization referred to in this section represents the synchronization between sender and receiver, rather than the global barrier synchronization operation referred to in previous Section 3.

4.2. Implementation To overcome these limitations, we leverage the Unified Notification RDMA Library (UNR) [41]. UNR provides one-sided RDMA Read and RDMA Write operations like MPI, but it utilizes low-level networking programming interfaces on different platforms to obtain the transmission completion and data arrival signals of these operations, thus avoiding manual post-synchronization. These signals could be automatically placed into a queue by UNR, enabling efficient data processing. As for pre-synchronization, UNR also provides a paradigm that hides it within preceding communications. Building upon UNR’s capabilities, we implement an optimized asynchronous one-sided communication pattern specifically designed for SNN spike data transfer within each synchronization window. The spike data are directly written into the receiver buffer using the one-sided RDMA Write operation provided by UNR. The pre-synchronization and post-synchronization can be hidden or avoided in our communication model. As for pre-synchronization, we use a dedicated buffer for each sender process within the current synchronization window; thus, all the pre-synchronization is hidden in the global Feng et al.: Preprint submitted to Elsevier

B1B2B3B4B5B6B7B8B9 B1B2B3B4B5B6B7B8B9

(a) Serial Computation-communication. NIC Recv NIC Recv CPU Comp CPU Comp NIC Send NIC Send

Handshake in MPI(Rendezvous Protocol) Handshake in MPI(Rendezvous Protocol)

B1 B2 B1 B2

B3B4 B3B4

B1 B2 B1 B2

B5 B5

B6 B6

B3 B4 B5 B3 B4 B5

B1 B2 B1 B2

B7B8B9 B7B8B9

B6 B6

B7 B8 B9 B7 B8 B9

B3B4 B5 B6 B3B4 B5 B6

B7 B8B9 B7 B8B9

(b) Computation-communication overlap with MPI. UNR RDMA Wrt from Rmt

NIC Recv B1 B2 B3B4 CPU Comp

NIC Send

B5

B6 B7B8B9

B1 B2 B3 B4 B5

B6 B7 B8 B9

B1 B2 B3 B4 B5

B6 B7 B8 B9

UNR RDMA Wrt to Rmt

(c) Computation-communication overlap with UNR (ours). Figure 3: Computation-communication overlap mechanisms

NIC Recv B1B2B3B4B5B6B7B8B9 CPU Comp

B1 B2 B3 B4 B5 B6 B7 B8 B9

B1B2B3B4B5B6B7B8B9 NIC Send barrier operations among synchronization windows. By isolating buffer access per sender process, multiple source processes can write to the same destination process concurrently without requiring coordination or synchronization between B1 addition, B2 B3B4 B5 B6is B7B8B9 NIC Recv In senders. there a buffer size upper bound for each sender B1process is aB7global barrier operation B2 B3since B4 B5 there B6 B8 B9 CPU Comp after each synchronization window. The buffer only needs B1 B2 B3B4 B5 B6 B7 B8B9 NIC Send to cache spike data for each step within a synchronization window. A typical total receive buffer size for each process is 256MB in our experiments, which is relatively acceptable. Although this strategy seems to lack scalability, the biological characteristics of SNNs dictate that when the number of neurons simulated by a process is fixed, the number of pulse signals received will not increase as the total simulation scale grows, and the virtual topology introduced in Section 5 further reduces the buffer used for redundancy. As for post-synchronization, we directly leverage UNR to obtain the arrival signals of spike data. Based on timely message arrival notifications provided by UNR, we implement a progressive communication truncation mechanism as shown in Figure 3. Upon detecting the arrival of a message containing some spike data, the process proactively terminates the communication phase without waiting for the arrival of future messages, enabling the complete overlap of computation and communication. The effectiveness and correctness of this approach are guaranteed by our rollback mechanism introduced in the previous section. When messages are not received in time, the simulator will trigger the rollback mechanism to reprocess the corresponding spikes and state updates. So there is no need to worry about the causal correctness of the simulation results. The overall simulation procedure and dataflow are illustrated in Figure 4. The key mechanism is the ‘SynapseUpdate II’ phase, which reserves spikes whose effective Handshake in MPI(Rendezvous Protocol)

Page 6 of 12

OACM: Optimistic Asynchronous Communication Model for Large-Scale SNN Simulation

Figure 4: The procedure and dataflow of the simulation. (1) The newly received spikes initially undergo ‘SynapseUpdate I’. (2) Those spikes with an effective timestep exceeding the next timestep are reserved for ‘SynapseUpdate II’ to overlap communication. (3) Those in-time spikes are immediately processed to accumulate synapse weights into neuron inputs. (4) Upon completion of ‘SynapseUpdate II’, the system stops detecting incoming messages. (5) Those reserved spikes are computed and added to neuron inputs once they reach the corresponding timestep. (6) Spikes generated during ‘NeuronUpdate’ are transmitted via UNR RDMA write operations, concluding a timestep loop.

timestep has not yet arrived so that they can be processed during communication, thereby enabling full computationcommunication overlap.

4.3. Design Analysis Our Asynchronous One-Sided Communication Model essentially uses RDMA-based one-sided communication and eliminates pre-synchronization and post-synchronization at the cost of additional memory consumption. Since UNR provides fine-grained data arrival information, it enables computation-communication overlap at fine granularity.

5. Message Routing and Aggregation 5.1. Routing on Virtual Topology Although UNR offers high-performance point-to-point communication, it does not provide any collective communication primitives. The existing MPI collective communication is inefficient and introduces extra global synchronization overhead, making it impossible to employ our aforementioned optimizations and failing to satisfy the requirements for irregular communication in SNN simulations. By contrast, naive point-to-point communication among all processes involves a large number of small data packets and its quadratic scaling complexity in the communication phase. To optimize bandwidth utilization and enhance communication efficiency, we leverage dimension-order routing in a virtual 2D-HyperX topology consisting of all processes. This approach significantly reduces the number of communication connections and small data packets while preserving functional equivalence to all-to-all message passing. We logically organize 𝑁 processes into a 2D-HyperX √ √ of size (⌊ 𝑁⌋, ⌈ 𝑁⌉), balancing row and column dimensions. Each process is uniquely identified by its coordinate. Feng et al.: Preprint submitted to Elsevier

Communication between processes is restricted to those sharing the same X or Y dimension. When process 𝑎 sends a message to process 𝑏, it first checks for direct adjacency. If 𝑏 is not adjacent, the message is sent to process 𝑐, which shares the same 𝑦 coordinate with 𝑏 and the same 𝑥 coordinate with 𝑎. Process 𝑐 then forwards the message horizontally to 𝑏 along the X-axis. This is known as YX routing, and the reason for not using XY routing will be explained in Section 5.3. This orthogonal decomposition ensures that any inter-process communication requires at most two hops, reducing the√number of communications per process from 𝑂(𝑁) to 𝑂( 𝑁) compared to the naive all-to-all pattern. While routing in a virtual topology is common in HPC applications, it is seldom used in SNN simulations and our work surpasses prior efforts. Earlier GPU cluster-based large-scale SNN simulations [8, 9] have adopted the 2DHyperX virtual topology. Their results indicate that 2D routing optimally balances communication traffic and runtime performance. Going further, our approach differs from VCT in three key ways: (a) We use the UNR library for asynchronous one-sided communication, avoiding the synchronous MPI_Send/MPI_Recv calls. (b) They split communication into 2 stages and wait for all messages before proceeding, introducing extra synchronization overhead. In contrast, our method sends messages immediately upon spike arrival or generation, incorporating a rollback mechanism to reduce synchronization overhead. (c) Our approach considers communication heterogeneity and includes process mapping as detailed in Section 5.3.

5.2. Message Aggregation To further optimize inter-process communication efficiency and enhance bandwidth utilization, we implement a two-layer message aggregation mechanism. The structure of an example message is illustrated in Figure 5. Each message includes two types of components: a header and a spike payload. The header contains multiple metadata entries, each corresponding to a destination process and encoding essential parameters, including destination ID, the address of the next header, and the number of spikes. The spike payload is categorized into two types of spikes: forward spikes, which originate from other processes and are routed by the current process, and direct spikes, which are locally generated to influence processes on the same axis or to be forwarded through them. Rather than employing the conventional approach of separately filling in specific spikes for each destination process, all direct spikes pertaining to the same brain area are aggregated into a unified spike payload. Subsequently, destination-specific headers are appended, thereby maximizing message coalescence. Upon receiving a message, a process first parses header metadata and distributes spike payload to the corresponding buffers based on the destination identifiers. During the synapse update, if a spike does not match any post-synaptic neuron resident in the process, then it will be discarded. This filtering occurs prior to

Page 7 of 12

OACM: Optimistic Asynchronous Communication Model for Large-Scale SNN Simulation

(a) Data Structures and Aggregation Layers.

(b) Routing Example.

(c) Process C Internal Lifecycle.

Figure 5: Message routing and aggregation. Process A sends message I to C. Its content is also intended for D and E. Upon receiving, C stores the relevant spikes and discards the irrelevant ones. Then, C sends message II to D and message III to E, both of which include the spikes forwarded from A and the spikes generated within C.

synaptic weight evaluation, thereby eliminating unnecessary computation cycles for non-local interactions. Thus, a two-layer aggregation is established: The first layer combines direct and forwarded spikes into a single transmission unit, exploiting spatial contiguity of target processes along topology dimensions. The second layer consolidates spikes generated by the same source neuron to multiple destinations, distributing the filtering burden to destination processes.

5.3. Process Mapping When simulating large-scale brain models with multiple interconnected areas, model partitioning and process mapping emerge as critical challenges that directly impact simulation efficiency. While achieving basic load balancing through uniform distribution of neural units across processes, the widely adopted round-robin partitioning method fundamentally fails to account for the intrinsic communication patterns observed in biological neural networks. Neuroimaging studies reveal that inter-area connectivity follows a distance-dependent principle where intra-area communication density exceeds inter-area communication density by orders of magnitude. This biological reality renders the conventional round-robin partition suboptimal, as it leads to a significant amount of avoidable inter-process communication. To address this mismatch, we implement the area partition method proposed in [42], which divides each area into 𝑛 parts (𝑛 varies among areas and is determined by the workload of that area) and designates an exclusive process to simulate each part. Process mapping must consider the non-uniform communication costs inherent in modern HPC architectures: Feng et al.: Preprint submitted to Elsevier

the costs of inter-process communication vary significantly based on physical interconnect, with lower costs between processes that are physically closer. Although processes are logically arranged into a 2D Mesh, their physical deployment follows a linear numbering scheme within computing clusters, resulting in non-uniform communication characteristics between dimensions. Communication along the Xaxis typically occurs between adjacent processes under the same switch hierarchy, exhibiting lower latency. Conversely, communication along the Y-axis traverses multiple compute nodes across different switch layers, incurring higher communication costs. Given that communication within brain areas is dense, our mapping strategy prioritizes co-locating processes from the same area along shared Y-axis coordinates. This design choice underpins the preference for YX routing over XY routing: the latter would introduce additional relay operations for inter-area communication. Consequently, we design a greedy area-clustering algorithm that maps communication-intensive parts to nearby processes. Specifically, it prioritizes mapping the same brain area to the same Y-dimension, while different brain areas on the same Y-dimension have denser communication. Figure 6 provides examples of different routing strategies and mapping methods.

6. Experiments 6.1. Experiment Setup Experiments were conducted on an HPC cluster connected via a 3-layer fat-tree TH Express network. Each compute node is equipped with dual 32-core 2.6 GHz Intel 8358P CPUs, 512 GB memory, and two new 200 Gbps TH Page 8 of 12

OACM: Optimistic Asynchronous Communication Model for Large-Scale SNN Simulation

(a) CORTEX’s simulation result. (a) Using XY routing without (b) Using YX routing with greedy area-clustering. greedy area-clustering. Figure 6: Message routing and process mapping. Each circle represents a process, and different colors represent different areas. Process A of the red area sends spikes to the processes of the purple area. (a) If XY routing without greedy areaclustering is used, sending spikes from A costs 3 slow communications on the Y-axis and 2 fast communications on the X-axis. (b) If YX routing with greedy area-clustering is used, sending spikes from A only costs 1 slow communication on the Y-axis and 2 fast communications on the X-axis.

Express NICs. Each compute node is configured to host 4 MPI processes. Due to limited cluster resources, we can only use a maximum of 174 compute nodes. The software stack includes GCC 11.4.0, MPICH 4.1.3, and UNR [41]. OACM is implemented based on the CORTEX [6] simulator, which is specially designed for large-scale SNN simulation. We evaluated performance using two benchmarks: Potjans’ cortical microcircuit model [43] and a multi-area marmoset brain model adapted from the CORTEX suite. Specifically, Potjans’ model comprises four layers (L2/3 to L6) with 7.7 × 105 neurons and 2.9 × 108 synapses. The multi-area model spans 116 cortical areas, with 9.7 × 106 neurons and 3.7 × 1010 synapses. Simulations ran for 15,000 timesteps with a resolution of 0.1 ms per step.

6.2. Correctness Evaluation To validate that our optimization preserves simulation fidelity, we employed Potjans’ model following the verification methodology in [5, 6]. We analyzed population-specific firing rates and spike rasters to compare OACM against the baseline CORTEX. As shown in Figure 7, the firing rate distributions (box plots) and spike timings (rasters) of OACM closely match those of CORTEX. Minor discrepancies are observed; however, these are attributable to stochastic factors, such as random connectivity and background noise, and remain within acceptable statistical bounds.

6.3. Performance Evaluation The trade-off between synchronization window size and runtime performance is as follows. Results indicate that increasing the window size initially reduces runtime, despite a concurrent rise in the average number of rollbacks. However, extending the window size beyond 20 yields diminishing returns, significantly increasing rollbacks while causing runtime to plateau or slightly increase. This pattern reflects the balance between minimizing synchronization overhead and Feng et al.: Preprint submitted to Elsevier

(b) OACM’s simulation result. Figure 7: The correctness comparison using Potjans’ model.

managing rollback costs. Consequently, a synchronization window size of 20 was selected for our experiments as it provided optimal performance across both models. Subsequently, we evaluate and compare the performance of CORTEX and OACM. Here, overall runtime is used for comparison without dissecting each optimization method via ablation studies. The reason is that each optimization method is sequentially dependent and only takes effect when combined. Figure 8 illustrates the runtime and relative speedup of OACM and CORTEX. The line chart represents the speedup, while the bar chart depicts the runtime. Figure 9 further breaks down the average time consumption of functional components and their proportions across all processes for CORTEX (left) and OACM (right). Since CORTEX employs the MPI_Allgatherv method for communication, which inherently includes synchronization, it is hard to collect statistics separately. Therefore, we have classified the synchronization part of CORTEX into the communication part. In Figure 9, the black slashed segment over the ‘Communication’ portion represents the percentage of ‘SynapseUpdate II’, which is dedicated to overlapping computation and communication. For OACM, this portion almost entirely covers the asynchronous communication, thereby achieving a complete overlap of computation and communication. In small-scale simulations of the Potjans’ model, the number of nodes varies from 1 to 8. As shown in Figure 8, there is minimal runtime difference between OACM and CORTEX. OACM underperforms slightly at 1 node but achieves a 1.1× speedup at 2 nodes. As shown in Figure 9(a), communication and synchronization are not dominant bottlenecks at smaller node counts. Additionally, OACM introduces extra computational overhead, such as rollback operations and anti-spike processing. Given

Page 9 of 12

OACM: Optimistic Asynchronous Communication Model for Large-Scale SNN Simulation

(a) Simulating the Potjans’ model with 2 nodes. (a) The Potjans’ model.

(b) Simulating the multi-area model with 174 nodes.

(b) The multi-area model. Figure 8: The runtime performance comparison. OACM achieves up to a 1.40× speedup over CORTEX in the multiarea model. In the small-scale Potjans’ model, OACM also attains a 1.10× speedup at 2 nodes.

its communication-intensive nature and all-to-all interprocess communication pattern, the model aligns well with CORTEX’s MPI_Allgatherv primitive. As the number of nodes increases, the 2D Mesh-based routing mechanism fails to deliver messages in a timely manner, leading to more rollback operations and subsequent performance degradation. In contrast, the multi-area model exhibits dense intraarea communication but sparse inter-area communication, with irregular connections between areas causing irregular inter-process communication among many processes. In this regime, OACM’s advantages become pronounced. When simulating the multi-area model, the number of nodes scales from 58 to 174. CORTEX’s simulation time initially decreases with more nodes but increases at 116 nodes due to synchronous communication bottlenecks. By contrast, OACM shows better scalability and outperforms CORTEX at various node counts. The speedup ratio rises with the number of nodes but declines at 174 nodes, peaking at 1.4× at 116 nodes. As shown in Figure 9(b), communication and synchronization account for over 70% of CORTEX’s

Feng et al.: Preprint submitted to Elsevier

Figure 9: The runtime percentage of each function. The black slashed segment, representing the ‘SynapseUpdate II’ used to overlap communication with computation, almost entirely covers the ‘Communication’ portion in OACM, indicating that nearly all of OACM’s communication time is hidden by computation.

runtime, whereas OACM significantly reduces communication and synchronization time despite introducing additional computational overhead. Synchronization overhead reduction is the main driver of OACM’s performance gain over CORTEX. Furthermore, we compare the runtime performance between OACM and NEST [5] using the multi-area model. NEST is an open-source, MPI-enabled simulator specialized for large-scale SNN simulations. It is widely adopted in computational neuroscience and brain-inspired computing research, including the Human Brain Project. To present the data in a visually appealing manner, we give this comparison in tabular form, with the results shown in Table 2. When the number of nodes increased from 58 to 174, NEST exhibited poor scalability and runtime performance, with runtime increasing significantly. In contrast, OACM demonstrates superior scalability and runtime performance, achieving speedups of more than 4× across different node counts. Specifically, with 174 nodes, OACM achieves a speedup of 22.6× compared to NEST. This indicates that OACM is significantly more efficient and scalable than NEST, making it a more suitable choice for large-scale SNN simulations.

Page 10 of 12

OACM: Optimistic Asynchronous Communication Model for Large-Scale SNN Simulation Table 2 Runtime performance comparison with NEST Number of Nodes NEST runtime OACM runtime Speedup 58 72.08s 17.99s 4× 87 89.39s 15.84s 5.6× 116 124.62s 13.83s 9× 174 295.1s 13.01s 22.6×

7. Conclusion In this paper, we present OACM, an Optimistic Asynchronous Communication Model that addresses the fundamental synchronization and communication bottlenecks in large-scale SNN simulations. By leveraging the biological characteristics of SNNs – sparse spiking activity and bounded synaptic delays – we strategically integrate Optimistic PDES techniques into time-driven simulation frameworks. This hybrid approach reduces synchronization frequency while maintaining correctness through localized rollback mechanisms within synchronization windows, preserving the simplicity of implementation and the parallelization advantages of time-driven methods. The introduced rollback mechanism’s sensitivity to message transmission delays necessitated advanced communication optimizations. We employ asynchronous one-sided communication via the UNR library to minimize communication latency and achieve complete computationcommunication overlap, while managing receive buffers through application-level flow control. Furthermore, to address the bandwidth underutilization caused by numerous small messages, we implement adaptive message aggregation and routing within a 2D-HyperX virtual topology, consolidating small packets into larger transmissions and delivering messages through at most one forwarding hop. Experimental evaluation demonstrates that OACM achieves significant performance improvements, with up to 1.4× speedup over the original CORTEX simulator and more than 22.6× speedup compared to NEST when simulating the multi-area marmoset brain model at scales of up to 174 compute nodes. These results validate our approach’s effectiveness in overcoming the synchronization and communication bottlenecks that have long constrained large-scale SNN simulations.

8. Future Work While our work successfully integrates time-driven and event-driven approaches to optimize communication efficiency, the current rollback mechanism does not yet explicitly support evolving synaptic states, such as those introduced by synaptic plasticity. Extending OACM to richer synaptic dynamics introduces additional temporal dependencies, since synaptic states may evolve in response to spike history and must remain causally consistent with neuronal states. Future work will therefore investigate lightweight mechanisms for jointly checkpointing, rolling back, and restoring neuronal and synaptic states within Feng et al.: Preprint submitted to Elsevier

synchronization windows, while controlling the associated memory and recomputation overhead. This extension would enable OACM to support more biologically sophisticated large-scale SNN simulations.

CRediT authorship contribution statement Guangnan Feng: Conceptualization, Methodology, Investigation, Writing – review & editing, Supervision. Yangle Zeng: Methodology, Software, Writing – original draft. Honghui Liang: Software, Investigation. Zhiguang Chen: Project administration. Yutong Lu: Funding acquisition.

Declaration of interests The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References [1] J. Igarashi, H. Yamaura, T. Yamazaki, Large-scale simulation of a layered cortical sheet of spiking network model using a tile partitioning method, Frontiers in Neuroinformatics 13 (2019) 71. [2] H. Yamaura, J. Igarashi, T. Yamazaki, Simulation of a human-scale cerebellar network model on the k computer, Frontiers in neuroinformatics 14 (2020) 16. [3] X. Du, Y. Liu, Z. Lu, Q. Duan, J. Feng, J. Wu, B. Chen, Q. Zheng, A low-latency communication design for brain simulations, IEEE Network 36 (2) (2022) 8–15. [4] W. Lu, L. Zeng, X. Du, W. Zhang, S. Xiang, H. Wang, J. Wang, M. Ji, Y. Hou, M. Wang, et al., Digital twin brain: a simulation and assimilation platform for whole human brain, arXiv preprint arXiv:2308.01241 (2023). [5] M.-O. Gewaltig, M. Diesmann, Nest (neural simulation tool), Scholarpedia 2 (4) (2007) 1430. [6] T. Lyu, M. Sato, S. Aoki, R. Himeno, Z. Sun, Cortex: Large-scale brain simulator utilizing indegree sub-graph decomposition on fugaku supercomputer, arXiv e-prints (2024) arXiv–2406. [7] C. Fernandez-Musoles, D. Coca, P. Richmond, Communication sparsity in distributed spiking neural network simulations to improve scalability, Frontiers in neuroinformatics 13 (2019) 19. [8] Y. Liu, X. Du, Z. Lu, Q. Duan, J. Feng, M. Wang, J. Wu, Regularizing sparse and imbalanced communications for voxel-based brain simulations on supercomputers, in: Proceedings of the 51st International Conference on Parallel Processing, 2022, pp. 1–11. [9] X. Du, M. Wang, Z. Lu, Q. Duan, Y. Liu, J. Feng, H. Wang, Hrcm: A hierarchical regularizing mechanism for sparse and imbalanced communication in whole human brain simulations, IEEE Transactions on Parallel and Distributed Systems (2024). [10] A. P. Davison, D. Brüderle, J. M. Eppler, J. Kremkow, E. Muller, D. Pecevski, L. Perrinet, P. Yger, Pynn: a common interface for neuronal network simulators, Frontiers in neuroinformatics 2 (11) (2009) 1–10. [11] F. Zenke, W. Gerstner, Limits to high-speed simulations of spiking neural networks using general-purpose computers, Frontiers in neuroinformatics 8 (76) (2014) 1–15. [12] H. E. Plesser, J. M. Eppler, A. Morrison, M. Diesmann, M.-O. Gewaltig, Efficient parallel simulation of large-scale neuronal networks on clusters of multiprocessor computers, in: European conference on parallel processing, Springer, 2007, pp. 672–681. [13] D. F. Goodman, R. Brette, Brian: a simulator for spiking neural networks in python, Frontiers in neuroinformatics 2 (5) (2008) 1–10. [14] M. Migliore, C. Cannia, W. W. Lytton, H. Markram, M. L. Hines, Parallel network simulations with neuron, Journal of computational neuroscience 21 (2006) 119–129.

Page 11 of 12

OACM: Optimistic Asynchronous Communication Model for Large-Scale SNN Simulation [15] E. Yavuz, J. Turner, T. Nowotny, Genn: a code generation framework for accelerated brain simulations, Scientific reports 6 (1) (2016) 1–14. [16] N. Ahmad, J. B. Isbister, T. S. C. Smithe, S. M. Stringer, Spike: A gpu optimised spiking neural network simulator, bioRxiv (2018) 461160. [17] T.-S. Chou, H. J. Kashyap, J. Xing, S. Listopad, E. L. Rounds, M. Beyeler, N. Dutt, J. L. Krichmar, Carlsim 4: An open source library for large scale, biologically detailed spiking neural network simulation using heterogeneous clusters, in: 2018 International joint conference on neural networks (IJCNN), IEEE, 2018, pp. 1–8. [18] M. Stimberg, R. Brette, D. F. Goodman, Brian 2, an intuitive and efficient neural simulator, Elife 8 (2019) e47314. [19] B. Golosio, C. De Luca, E. Pastorelli, F. Simula, G. Tiddia, P. S. Paolucci, Toward a possible integration of neurongpu in nest, in: NEST Conference, Vol. 7, 2020. [20] P. Qu, Y. Zhang, X. Fei, W. Zheng, High performance simulation of spiking neural network on gpgpus, IEEE Transactions on Parallel and Distributed Systems 31 (11) (2020) 2510–2523. [21] B. Golosio, G. Tiddia, C. De Luca, E. Pastorelli, F. Simula, P. S. Paolucci, Fast simulations of highly-connected spiking cortical models using gpus, Frontiers in Computational Neuroscience 15 (2021) 627620. [22] D. Bautembach, I. Oikonomidis, A. Argyros, Multi-gpu snn simulation with static load balancing, in: 2021 International Joint Conference on Neural Networks (IJCNN), IEEE, 2021, pp. 1–8. [23] L. Niedermeier, K. Chen, J. Xing, A. Das, J. Kopsick, E. Scott, N. Sutton, K. Weber, N. Dutt, J. L. Krichmar, Carlsim 6: An open source library for large-scale, biologically detailed spiking neural network simulation, in: 2022 International Joint Conference on Neural Networks (IJCNN), IEEE, 2022, pp. 1–10. [24] G. Tiddia, B. Golosio, J. Albers, J. Senk, F. Simula, J. Pronold, V. Fanti, E. Pastorelli, P. S. Paolucci, S. J. van Albada, Fast simulation of a multi-area spiking network model of macaque cortex on an mpigpu cluster, Frontiers in neuroinformatics 16 (2022) 883333. [25] J. Zhang, H. Wu, J. Wei, S. Wei, H. Chen, An asynchronous reconfigurable snn accelerator with event-driven time step update, in: 2019 IEEE Asian Solid-State Circuits Conference (A-SSCC), IEEE, 2019, pp. 213–216. [26] H. Lee, C. Kim, Y. Chung, J. Kim, Neuroengine: a hardware-based event-driven simulation system for advanced brain-inspired computing, in: Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, 2021, pp. 975–989. [27] H. Lee, C. Kim, M. Kim, Y. Chung, J. Kim, Neurosync: A scalable and accurate brain simulator using safe and efficient speculation, in: 2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA), IEEE, 2022, pp. 633–647. [28] X. Peng, Z. Wang, F. Han, G. Song, S. Ding, A novel time-eventdriven algorithm for simulating spiking neural networks based on circular array, Neurocomputing 292 (2018) 121–129. [29] E. Ros, R. Carrillo, E. M. Ortigosa, B. Barbour, R. Agís, Event-driven simulation scheme for spiking neural networks using lookup tables to characterize neuronal dynamics, Neural computation 18 (12) (2006) 2959–2993. [30] L. Mo, Z. Tao, Evtsnn: Event-driven snn simulator optimized by population and pre-filtering, Frontiers in Neuroscience 16 (2022) 944262. [31] D. Pecevski, D. Kappel, Z. Jonke, Nevesim: event-driven neural simulation framework with a python interface, Frontiers in neuroinformatics 8 (70) (2014) 1–20. [32] L. Mo, X. Chen, G. Wang, Edha: Event-driven high accurate simulator for spike neural networks, Electronics 10 (18) (2021) 2281. [33] D. Bautembach, I. Oikonomidis, A. Argyros, Even faster snn simulation with lazy+ event-driven plasticity and shared atomics, in: 2021 IEEE High Performance Extreme Computing Conference (HPEC), IEEE, 2021, pp. 1–8. [34] D. Bautembach, I. Oikonomidis, N. Kyriazis, A. Argyros, Faster and simpler snn simulation with work queues, in: 2020 International Joint Conference on Neural Networks (IJCNN), IEEE, 2020, pp. 1–8.

Feng et al.: Preprint submitted to Elsevier

[35] G. Susi, P. Garces, A. Cristini, E. Paracone, M. Salerno, F. Maestu, E. Pereda, Fns: An event-driven spiking neural network simulator based on the lifl neuron model, arXiv preprint arXiv:1801.00864 (2018). [36] M. Plagge, C. D. Carothers, E. Gonsiorowski, N. Mcglohon, Nemo: A massively parallel discrete-event simulation model for neuromorphic architectures, ACM Transactions on Modeling and Computer Simulation (TOMACS) 28 (4) (2018) 1–25. [37] A. Pimpini, A. Piccione, B. Ciciani, A. Pellegrini, Speculative distributed simulation of very large spiking neural networks, in: Proceedings of the 2022 ACM SIGSIM Conference on Principles of Advanced Discrete Simulation, 2022, pp. 93–104. [38] F. Naveros, N. R. Luque, J. A. Garrido, R. R. Carrillo, M. Anguita, E. Ros, A Spiking Neural Simulator Integrating Event-Driven and Time-Driven Computation Schemes Using Parallel CPU-GPU CoProcessing: A Case Study, IEEE Transactions on Neural Networks and Learning Systems 26 (7) (2015) 1567–1574. doi:10.1109/TNNLS. 2014.2345844. [39] P. Qu, H. Lin, M. Pang, X. Liu, W. Zheng, Y. Zhang, Enlarge: An efficient snn simulation framework on gpu clusters, IEEE Transactions on Parallel and Distributed Systems (2023). [40] F. Cremonesi, F. Schürmann, Understanding computational costs of cellular-level brain tissue simulations through analytical performance models, Neuroinformatics 18 (3) (2020) 407–428. [41] G. Feng, J. Xie, D. Dong, Y. Lu, Unr: Unified notifiable rma library for hpc, in: SC24: International Conference for High Performance Computing, Networking, Storage and Analysis, IEEE, 2024, pp. 1–15. [42] Y. Zeng, G. Feng, Z. Chen, Y. Lu, N. Xiao, Atm: Area-based partition and topology-aware mapping for large-scale snn simulation, IEEE International Symposium on Parallel and Distributed Processing with Applications (ISPA) (2024). [43] T. C. Potjans, M. Diesmann, The cell-type specific cortical microcircuit: relating structure and activity in a full-scale spiking network model, Cerebral cortex 24 (3) (2014) 785–806.

Page 12 of 12

Related documents

• OACM: Optimistic Asynchronous Communication Model for Large-Scale SNN Simulation#1199510
• SpikeLite: Lightweight Spiking Neural Networks for Time-Series Forecasting#1129698
• Performance Comparison of Machine Learning Across Metal, Cuda, and Software-Based Neuromorphic Simulation#1081910
• The Dormant Spiking Neuron: A State-Driven Mechanism for Efficient Spiking Neural Networks#1357403
• Autonomous Chaotic Time Series Prediction using Physical Neuromorphic Networks#973975