FrontGI: Architecture-Aware Parallel GNN Inference on Chiplet-Based Multicore Systems

FrontGI: Architecture-Aware Parallel GNN Inference on Chiplet-Based Multicore Systems Binglin Ji1 , Chenfeng Zhao2† , Roger Chamberlain3*

arXiv:2610.34911v1 [cs.DC] 26 Oct 2026

1

Dept. of Electrical and Systems Engineering, Washington Univ. in St. Louis, One Brookings Dr., St. Louis, MO, 63130, USA. 2 GE Vernova Advanced Research, Niskayuna, NY, USA. 3 Dept of Computer Science and Engineering, Washington Univ. in St. Louis, One Brookings Dr., St. Louis, MO, 63130, USA.

*Corresponding author(s). E-mail(s): roger@wustl.edu; Contributing authors: binglin.j@wustl.edu; chenfeng.zhao@gevernova.com; † Work completed while at Washington Univ. in St. Louis. Abstract The realm of high-performance computing has expanded to include machine learning tasks of many forms. Graph Neural Networks (GNNs) are becoming increasingly popular, with their applications expanding across diverse domains. As the scale of graph data continues to grow, including larger numbers of nodes, edges, and higher embedding dimensions, standardized libraries such as DGL and PyG have been developed to facilitate GNN computation. These libraries allow the computation to be described a a high level of abstraction, providing high productivity on the part of the developer. However, with the rapid increase in the number of processor cores and the evolution of multicore architectures, these libraries often show poor scalability and fail to execute GNN inference efficiently on the latest multicore systems, particularly those with upwards of a hundred cores. To address this limitation, we introduce FrontGI, a fast and scalable set of techniques that advances the frontier of GNN inference on large-scale graphs (over 100,000 vertices) by leveraging graph-property- and architectureaware design. FrontGI employs different parallelization strategies optimized for diverse graph structures, maximizing the utilization of multi-level cache hierarchies in multicore systems. We evaluate a number of GNN models with FrontGI on a 128-core AMD EPYC system. Compared to state-of-the-art libraries, FrontGI achieves up to 3.35×, 3.20×, 2.89× and 3.10× speedups over DGL for GCN, GraphSAGE, GIN, and SGC models, respectively, and even greater improvements over PyG, across a range of large-scale, high-dimensional graph datasets with diverse structural properties. Keywords: Graph Neural Network, Graph Convolutional Network, GraphSAGE, Graph Isomorphism Network, Simple Graph Convolution

1 Introduction Recently, large-scale graph data has gained popularity across numerous domains, including social network analysis [1, 2], recommendation systems [3], bioinformatics [4], and electronic circuit design [5], all of which use high performance computing for their solution. Graph Neural Networks (GNNs) are a specialized type of deep learning model that excels at learning graph data representations. The emergence of GNNs has garnered significant interest from both academic and

1

industrial communities. As the application of GNNs becomes increasingly widespread to computational intensive topics, such as computer vision [6] and natural language processing [7, 8], the dimensionality of GNN embeddings has gradually increased to thousands in recent research. Standard libraries like PyTorch-Geometric (PyG) [9] and Deep Graph Library (DGL) [10] are widely used, providing a high-productivity development environment for GNNs. In effect, these libraries allow the application developer to focus on the specifics of the GNN model, removing requirements that the developer port across execution platforms or implement low-level functions. By optimizing sparse operations, in an automated way, these libraries currrently enable users to build and compute graph neural networks with high productivity. One of the clear productivity benefits of libraries like DGL and PyG is that a common source code expression of a GNN application can be efficiently deployed on a wide range of execution platforms. These range from traditional processor cores to a range of computational accelerators. We propose to expand the scope of execution platforms that libraries like DGL and PyG can exploit to include high core-count multicore systems. Although promising performance results for GNN training and inference are achieved by adopting accelerators such as GPUs [11–14], FPGAs [15–17], and NPUs [18], multicores remain advantageous and indispensable for GNN execution. First, multicores are ubiquitous across all computing platforms, from data centers to personal computers. For instance, since a significant number of cores may be idle during off-peak hours in data centers, leveraging them to perform GNN tasks can substantially reduce the cost [19]. Second, memory capacity poses a significant bottleneck for GNN execution. Large graphs, often reaching hundreds of gigabytes or more, exceed the memory limits of accelerators like GPUs and FPGAs which typically offer only tens of gigabytes of onboard memory. This necessitates frequent data movement between disjoint memories, substantially impacting the resulting performance [20]. In contrast, multicore systems capable of supporting terabytes of memory effectively mitigate this constraint for large-scale GNN tasks [19]. Third, multicore platforms can address the poor model scalability observed with discrete GPUs and FPGAs, which suffer from expensive communication bottlenecks [21]. Fourth, while many multicore systems have integrated GPU capabilities, as a general rule, they are substantially less capable than traditional dedicated GPU cards [22]. Finally, libraries such as DGL and PyG offer multicores as target platforms, so improved performance on those platforms can be beneficial to the user community. Due to these issues, there is sustained interest in GNN execution on multicore systems, particularly as their computing capability increases. Several strategies have been proposed to enhance GNN execution on multicores. GIN [23] introduces a graph learning method that automatically optimizes and generates computing kernels for GNN inference models based on input graph topology. Graphite [19] improves GNN inference through both software and hardware optimizations by fusing operations in each layer, exploiting sparsity from activation functions, and utilizing Direct Memory Access (DMA) engines for data prefetching. DistGNN [24] employs offline graph partitioning methods to reduce communication costs across computing nodes during GNN training. ARGO [25] uses a Bayesian approach to devise an optimal core-binding strategy, enhancing the scalability of GNN training tasks with standard libraries. Despite promising results, these approaches typically depend on offline preprocessing, co-design assumptions, or training-specific optimization, and therefore do not directly address a fundamental challenge in GNN inference on modern multicores, especially when offline preprocessing is undesirable or infeasible. This invites a critical question: how can we effectively accelerate GNN inference by leveraging modern multicore architectures while avoiding offline preprocessing for various graph topologies? In this work, which extends [26], we present FrontGI, a graph-property-aware and architectureaware set of techniques for GNN inference acceleration on modern multicore processor platforms. The goal is to develop methods that can be incorporated into the existing libraries, maintaining the ability to express the problem at a high level, while enabling efficient implementations to be invoked in an automated way. Here, we significantly broaden the scope of [26] along three key dimensions. First, motivated by our observation that BLAS parallelization models rely on cache assumptions that do not hold for multicore processors with distributed last-level caches (LLC), FrontGI parallelizes the Update phase in addition to the Aggregate phase, which consists of dense matrix operations that dominate the computation in many GNN models. We further investigate how operators can be efficiently fused under distributed LLC architectures. Second, to explore diverse sparse–dense computation patterns, we design and evaluate FrontGI across four representative GNN models: GCN, GraphSAGE, GIN, and SGC. These models exhibit distinct

2

Aggregate patterns—including additional linear functions and degree-normalized operations—as well as different operator connectivity inside of the Aggregate operator. Finally, beyond the qualitative analysis, we perform a comprehensive quantitative evaluation using twelve large-scale graph datasets spanning diverse graph sizes, average degrees, and degree distributions, allowing us to systematically characterize how FrontGI selects and adapts parallel execution strategies according to graph structural properties. These strategies are straightforward to incorporate into the standard libraries, maintaining the ability to express the model at a high level of abstraction. Our contributions are as follows: • We identify how a distributed last-level cache organization in modern chiplet-based multicores fundamentally influences the performance trade-offs of GNN inference under different graph properties. • We show that architectural characteristics impose distinct execution requirements for the Aggregate phase, and derive two architecture-aware hierarchical execution strategies— hierarchical node-level parallelism and mixed node–tensor parallelism—to mitigate memory access irregularity. • We design a simple parallelization strategy for the Update phase of GNNs that leverages BLAS GEMM kernels to enable faster GNN inference on large-scale graphs with modern multicore architectures, by alleviating the inconsistency between BLAS’s implicit shared-cache access assumptions and the distributed LLC organization of multicore systems. • Through extensive evaluation on a wide range of GNN models (GCN, GraphSage, GIN and SGC) and graph datasets with varying properties, we validate our design principles and provide practical guidelines for selecting appropriate parallel execution strategies under different graph structures. GNN inference using the FrontGI techniques achieves a speedup of 3.35× compared to the maximum performance of the state-of-the-art GNN computation library (DGL). Once incorporated into these libraries, these techniques will continue to improve the performance achievable in a high-productivity development environment.

2 Motivation and Background 2.1 Motivation It is not uncommon for the performance properties of many applications to change, often significantly, as the core count increases [27]. Frequently this is due to some property of the architecture or the algorithm that is different for large core counts versus more modest core counts. Systems that allow the expression of applications at a high level of abstraction have been required to automatically adapt to these changes for years. For example, solvers for linear systems are regularly tuned to newly released execution platforms empirically [28].

Fig. 1 Speedup of DGL 2.4.0 (relative to 8 cores).

3

Figure 1 shows the speedup (relative to the 8-core execution) of GCN inference on three graphs with 1024-dimensional embedded features: OGBN-Proteins (PT) [29], Reddit (RD) [30] and OGBG-Molhiv (MH) [29] with DGL v2.4.0, executing on a 128-core AMD EPYC system. Up to 32 cores, the parallel efficiency is reasonably good, with each additional core providing significant performance gains. However, the performance gains clearly wane at larger core counts. Specifically, for the OGBG-Molhiv and OGBN-Proteins, performance decreases once the execution exceeds about 90 cores, whereas for the Reddit graph, the decline begins at 64 cores. One of the potential causes for this rollover in performance is that 128-core systems are architected differently than 16- or 32-core systems. Recently, the designs of high-performance processors have increasingly adopted chiplet-based architectures, driven by the physical and economic limitations of monolithic silicon scaling [31]. As transistor density growth slows and die sizes approach reticle limits, fabricating large monolithic dies becomes cost-prohibitive and less yield-efficient. To address these challenges, major vendors have adopted chiplet-based designs across various architectures. In multicore processors, AMD employs Core Complex Dies (CCDs) in its EPYC series [32], while Intel uses tile-based designs in processors such as Sapphire Rapids [33]. Take AMD EPYC for instance, the new architecture partitions the multicore processor into multiple chiplets, each integrating a group of physical cores and a dedicated slice of the Last-Level Cache (LLC), interconnected through a high-speed on-package interconnect, as illustrated in Figure 2. This trend extends beyond traditional processors—many recent accelerator solutions [34] also adopt chiplet architectures to improve scalability.

CHIPLET

CHIPLET

MEMORY/IO DIES

CHIPLET

CHIPLET

CORE

L2

CORE

L2

CORE

L2

CORE

L2

SHARED LLC

CHIPLET

L2

CORE

L2

CORE

L2

CORE

L2

CORE

CHIPLET

Fig. 2 AMD’s multicore system using Core Complex Dies (CCDs).

Data communication within a chiplet is handled through the shared cache, offering low latency, while inter-chiplet communication relies on high-speed interconnects. This architecture significantly improves the scalability and resource utilization of multicore systems through distributed caching and core grouping. However, the backends of standard libraries often leverage OpenMP for data parallelism, typically using graph nodes as the granularity of parallelism. For operators which exhibit highly irregular memory access patterns, common in GNNs, this approach can result in substantial inter-chiplet communication overhead, thereby significantly limiting the scalability of parallel computing. Moreover, for a given hardware platform, the placement of LLC slices across chiplets is entirely determined by the underlying chiplet-based architecture and cannot be controlled directly by software. Consequently, since the cache organization is fixed by the hardware, an important question naturally arises: Can we derive a general parallel execution method that adapts computation to the underlying cache organization? Specifically, given different computation patterns—including varying graph sparsity and sparse–dense data flows—can computation be partitioned in a manner that naturally localizes memory accesses within the hardware-managed LLC of each chiplet while minimizing inter-chiplet communication? To address this question and leverage the potential of the new hardware organization, we explore alternative parallelization techniques for GNNs that attempt to alleviate some of the interchiplet communications implied by exclusive use of node-level parallelism. We also measure the impact of these techniques on last-level cache misses. Finally, we articulate how a high-productivity development environment can choose which techniques to deploy.

4

2.2 Background In this section, we first introduce notation shown in Table 1, which is used throughout the paper. We then describe different GNN operations and models as well as tensor-program optimizations for neural networks. Table 1 Notation. Symbol Description G(V, E)

Symbol Description

Graph with nodes V and edges E

N (v)

All v’s neighbors

V

Node list of V

dv

Degree of node v

H(k)

Hp

p-th partition of H

hv

(k)

Node v feature vector at k-th layer

E

Edge list of E Feature matrix after k-th layer

(k)

Aggregated feature matrix at k-th layer

Ap

p-th partition of A

av

(k)

Aggregated feature of node v at k-th layer

W(k)

Weight matrix of k-th layer

C

Number of cores in the system

A

D

Number of chiplets in the system

2.2.1 Graph Neural Network Operations Graph Neural Networks (GNNs) compute node feature vectors (embeddings) iteratively on a graph G (V, E ). For a node v at layer k + 1, its feature vector is derived with two steps based on the embeddings from its neighbors at layer k . Feature Matrix of k-th Layer

Graph Structure 2

1

4

0 5

3 Neighbor Nodes

0 1 2 3 4 5

Feature Matrix after Aggregation

0 1 2 3 4 5

Update

Aggregate

Target Nodes

Feature Matrix of (k+1)-th Layer

Aggregate

Update

Fig. 3 An example graph and corresponding GNN computation pipeline.

The first step is Aggregate, as shown in Equation (1):

  a(k+1) = Aggregate(k+1) {h(k) v u | u ∈ N (v ) ∪ {v}}

(1)

(k)

where N (v ) represents all neighbors of vertex v , hu represents node embeddings that need to be aggregated in the k th layer. The Aggregate operator passes messages from v ’s neighbor nodes, (k+1) returning aggregated embedding av . (k+1) In the next step, the GNN layer updates av with Equation (2):

  h(k+1) = Update(k+1) a(k+1) v v

(2)

where the Update operator is usually a neural network function, a single neural network layer containing a linear and a non-linear (activation) transformation, or a multi-level perceptron. The whole computing pipeline is illustrated in Figure 3.

2.2.2 Graph Convolutional Network The Graph Convolutional Network (GCN) [35] model is one of the most popular GNN models, it sums up all the neighbor node’s embeddings in the aggregation step, then updates the aggregated 5

node embeddings with a single neural network layer. Equation (3) shows a GCN layer with ReLU activation function applying on a graph node v :



 X

h(k+1) = ReLU W(k) v

 h(k) u

(3)

u∈N (v)

where W(k) ∈ Rdout ×din is a learnable weight matrix.

2.2.3 GraphSAGE GraphSAGE [30] is a GNN model designed for processing large-scale graphs. It aggregates neighbor embeddings and concatenates the aggregated features with the input node features before applying a learnable transformation and a non-linear activation, enabling the model to learn these two types of information separately and assign different weights to them. Equation (4) shows a GraphSAGE layer with mean value aggregation and ReLU activation function applying on a graph node v :







h(k+1) = ReLU W(k) Concat h(k) v v ,

1 |N (v )|

X

 h(k) u

u∈N (v)





(k) = ReLU Wa(k) h(k) v + Wb

1 |N (v )|

X

(4)

 h(k) u

u∈N (v)

where W(k) ∈ Rdout ×2din is a learnable weight matrix. To facilitate parallelization and reduce (k) the memory access footprint, we decompose W(k) into two separate matrices, Wa ∈ Rdout ×din (k) and Wb ∈ Rdout ×din , which are applied to the input node features and the aggregated features, respectively.

2.2.4 Graph Isomorphism Network The Graph Isomorphism Network (GIN) [36] model is designed to achieve maximum discrimination among graph structures by mimicking the Weisfeiler-Lehman graph isomorphism test [37]. GIN employs a sum aggregator, as summation is proven to be more expressive in distinguishing graph structures. After aggregation, GIN applies a learnable transformation using a multi-layer perceptron (MLP) followed by a non-linear activation. This design allows GIN to capture subtle differences in graph topology more effectively. Equation (5) shows a GIN layer applied on a graph node v :   X (k) (k) (k) (k) h(k+1) = MLP (1 + ϵ ) · h + h (5) v v u u∈N (v)

where ϵ

(k)

is a learnable or fixed scalar, and MLP

(k)

is a multi-layer perceptron.

2.2.5 Simple Graph Convolution The Simple Graph Convolution (SGC) [38] model removes nonlinearities between layers and collapses intermediate weights into a single linear layer. It performs K rounds of normalized propagation, i.e., aggregating information from the K -hop neighborhood, then applies a weight matrix. The normalization parameter for a node v is defined as the inverse square root of its degree dv after adding a self-loop. Equations (6) and (7) show the SGC model applied to a node v :

h̃(k+1) = v

1 h̃(k) u , ˜ dv d˜u

X p u∈N (v)∪{v}

hv = W h̃(K) v

k = 0, . . . , K

(6)

(7) (k) ˜ where dv = dv + 1, h̃u is the intermediate aggregated result of node u after k -th propagation (0) (0) round, h̃u = hu is the input feature, hv is the final output and W ∈ Rdout ×din is a learnable weight matrix. 6

We select these four models due to their representative dataflow characteristics: GCN and GraphSAGE layers embody the fundamental sparse–dense computation pattern, GIN embodies a sparse-to-multiple-dense computation pattern, while SGC represents multiple sparse operations followed by a dense computation, as illustrated in Figure 4. As shown in the figure, the dense computations correspond to the update phase, whereas the sparse and fused computations correspond to the aggregation phase. A. GCN Layer

Sparse

B. GraphSage Layer

C. GIN Layer

Sparse

Sparse

D. SGC Layer

Sparse Linear Ops Sparse

Dense Dense

Dense Operations

Fused Operations

Linear Operations

Feature Matrices

MLP

Dense

Sparse Operations

Linear Ops

Dense

Dense

k-Hops

Linear Ops

Dense

Matrices after Aggregation

Fig. 4 GNN Layer Computation models and dataflows.

2.2.6 Tensor Parallelism for DNNs With the increasing scale and dimensionality of datasets for deep learning tasks, tensor parallelism has been proposed to parallelize feature vector computations. It was first proposed to handle large language model pre-training problems on multiple GPUs by partitioning both the feature and weight matrices’ dimensionalities inside of model parallelism [39], then later was used in high performance AI systems [40] and subsequently integrated into standard libraries like PyTorch [41]. The tensor parallelism method can be illustrated as shown in Algorithm 1.

Algorithm 1 Tensor Parallelism Require: Input tensor H, weight matrix W, number of devices D, activation function σ Ensure: Output tensor H′ 1: Split H into [H1 , H2 , . . . , HD ] along feature dimension T 2: Split W into [W1 , W2 , . . . , WD ] along rows 3: for i = 1 to D (in parallel) do 4: Compute Htempi ← Hi · Wi 5: end for PD 6: Synchronize Partial Results: Htemp ← i=1 Htempi ′ 7: Apply activation function: H ← σ (Htemp )

When dealing with linear operations, on a system that has D devices, for a batch of data with high dimensional embeddings, we split the input feature matrix H and weight matrix W into D parts along feature (column) and row, respectively. Each device processes a partial linear problem. Finally, synchronize and accumulate linear operations, then apply non-linear operations like layer activation.

7

3 FrontGI Design To push the performance frontier of GNN inference on large-scale graphs using modern multicore processors, we design FrontGI, an efficient and scalable set of computing techniques that synergistically optimizes the computational characteristics of GNNs with the architecture of multicore systems, especially high core count systems built with chiplets. Drawing from ideas that work well in FPGA implementations [17], we first organize the inference computation in a pipelined manner as Figure 3, where graph data flows from the Aggregate phase to the Update phase. While there is a buffer between the two phases to hold intermediate results, this buffer can be relatively small (i.e., it will fit in cache), so that the intermediate results do not have to be written back to main memory. In addition, by separating the Aggregate phase from the Update phase, instead of executing a large sparse matrix-matrix multiply (SpMM), we are performing a series of vector query and dense vector-matrix multiplies at each node. To further leverage the latest multicore architectures, we designed the Aggregate phase of FrontGI with two different parallel decompositions according to the sparsity of input graphs: mixed tensor/node parallelism for dense graphs and hierarchical node parallelism for sparse graphs. The intermediate results are subsequently stored in a feature matrix buffer, which allows us to more effectively exploit existing matrix libraries during the Update phase. In both cases, the Update phase is implemented using the BLAS GEMM library [42] with our tailored parallelization strategy designed to fully leverage the underlying chiplet-based processor architecture.

3.1 Graph Processing Systems for GNNs General purpose graph processing systems have emerged that handle a wide variety of graph processing problems. Exploiting near-memory processing methods to exploit data locality [43], high efficiency I/O system design to optimize resource usage [44], and novel graph representation to utilize SIMT architectures [45, 46], these systems can handle large scale traditional graph processing problems. However, these systems are often not well suited for GNN processing, primarily because most graph processing algorithms’ performance is dominated by irregular memory access while GNN processing has both irregular memory access and a high computational load. Specifically, Aggregate involves irregular memory accesses and Update contains vector-matrix or matrixmatrix multiplication operations that are computationally intensive.

Algorithm 2 Aggregate With CSR Format Require: Input feature matrix H(k) , feature dimension dim, feature buffer bv , edge list E, edge range (estart , eend ) Ensure: Aggregated feature vector av for node v 1: Initialize btemp as an array of zeros of size dim 2: for e = estart to eend − 1 do 3: Create a read buffer brd as an array of size dim 4: ndst ← E[e].src 5: for i = 0 to dim − 1 do 6: brd [i] ← H(k) [ndst · dim + i] 7: if e = estart then 8: btemp [i] ← 0 9: end if 10: btemp [i] ← btemp [i] + brd [i] 11: if e = eend − 1 then 12: bv [i] ← btemp [i] 13: end if 14: end for 15: end for 16: av ← bv 17: return av

8

In this work, by executing Aggregate and Update separately in a pipelined manner, we are allowed to tune these two phases individually to address their divergent performance concerns. Algorithm 2 illustrates how to execute the Aggregate operation for a node v with graph data stored in a CSR format (a commonly used format for sparse graphs). The sum operation inherent in GCN’s Aggregate is shown in line 10. When optimizing various GNN models, we substitute this operation with the appropriate alternative.

3.2 Parallelization of Aggregate Phase 3.2.1 Mixed Tensor/Node Parallelism When loading data from memory, a segment of data is loaded into the cache, allowing the processor to exploit data locality. In GNN computing problems, we store node features in a matrix where each row represents a feature vector of a specific node. However, during aggregation, the irregular memory access patterns result in large amounts of data—most of which will not be accessed—being loaded into higher-level caches, causing frequent cache misses due to cache corruption. The higher the graph embedding dimension, the better the locality of the data access for feature vectors at an individual core, but correspondingly, the proportion of node features in the cache that will not be accessed also increases. Chiplet 1: CSR

CSR

Chiplet 2: CSR

Tensor Parallelism Fig. 5 Tensor parallelism in aggregation phase.

Similar to the tensor parallelism challenge in DNNs, where a single device struggles to handle all features of a batch of data efficiently, in the multicore system scenario, the data segment loading and irregular memory access patterns prevent each chiplet from efficiently handling the full-batch of high-dimensional node features. Therefore, for denser graphs (i.e., graphs with high average node degree), we introduce tensor parallelism during the aggregation phase. Specifically, we apply tensor parallelism to the input feature matrix by treating each chiplet with a shared L3 cache as a “device.” Within each chiplet, we exploit node-level parallelism. In mixed tensor/node parallelism, each chiplet holds its own copy of data, recording all nodes’ neighbor information and processes in a partial feature matrix along a subset of its dimensions. After all chiplets have processed their corresponding feature subsets, the intermediate results of the aggregation phase are concatenated. Figure 5 illustrates applying tensor parallelism to aggregation on a multicore system with two chiplets. The left-hand side shows the data decomposition, reading CSR-formatted graph data, and the right-hand side illustrates the mapping to distinct chiplets. In this way, for large core count systems with multiple chiplets, each shared LLC holds a significantly larger portion of node features compared to the pre-tensor-parallelization scenario, thereby reducing LLC load misses during the aggregation phase for denser graphs. As shown in Figure 6 (left), when many cores (Core 0 and Core 1) within a chiplet need the feature vectors of the neighbors of the nodes they are currently processing, they first check the cache (here, specifically, the shared LLC within the chiplet). If a cache miss occurs, they will load data from the main memory. By applying tensor parallelism to reduce the dimensionality of the feature vectors, many times more feature vectors than in the pre-tensor-parallelism case can be retained in the cache during the previous data loading process. For dense graphs, this reduces the likelihood of cache misses in LLC. In the figure, the feature size on the left has not been parallelized, so the features occupy more of the cache. On the right, the features are effectively a smaller size, so the cache can hold more of them.

9

Memory Access on Dense Graphs TID

Feature Matrix

In Parallel

TID

Memory Access on Sparse Graphs

Feature Matrix

TID

In Parallel

dim = N Nodes in Process

Cached Data

Feature Matrix

In Parallel

TID In Parallel

dim = N

dim = N/2 Feature in Cache

Feature Matrix

Feature not in Cache

dim = N/2 Cache Hits

Cache Misses

Fig. 6 Memory accesses of tensor parallelism on dense (left) and sparse (right) graphs.

Algorithm 3 shows our proposed mixed parallelism for a GNN layer. On a system with D chiplets, for a large input graph with high average node degree and heavy embeddings, we apply tensor parallelism as in Algorithm 1 during the Aggregate phase by splitting the input feature matrix into D parts along the feature dimension. Each chiplet processes a partial aggregation problem with Algorithm 2.

Algorithm 3 Mixed Tensor/Node Parallelism Require: Input feature matrix H(k) , number of chiplets D, edge list E, node list V, number of nodes N Ensure: Output feature matrix H(k+1) 1: // Start Aggregate (k) (k) (k) (k) 2: [H1 , H2 , . . . , HD ] ← split H 3: // Tensor Parallelism 4: for d = 1 to D in parallel do 5: // Compute core IDs for each chiplet 6: (IDstart , IDend ) ← CoreGroup(d) 7: Place all threads in (IDstart , IDend ) 8: // Node Parallelism 9: for v = 0 to (N − 1) in parallel do 10: Create a buffer bv with size dim/D 11: (estart , eend ) ← V(v ) (k) 12: av ← Aggregate(Hd , bv , E, estart , eend ) (k+1) 13: Ad [v ] ← av 14: end for 15: end for (k+1) (k+1) (k+1) (k+1) 16: A ← concatenate [A1 , A2 , . . . , AD ] 17: // Start Update (k+1) 18: H ← Update(A(k+1) , W) (k+1) 19: return H Although OpenMP provides policy-based thread affinity mechanisms, our execution strategies require explicit and finer-grained control over the mapping between worker threads and chiplet-local cores. Therefore, we implement a custom thread affinity routine using the Linux sched setaffinity() system call. For a two-level nested OpenMP parallel region, each outer worker invokes this routine during the creation of the outer parallel region. The routine constructs a processor affinity mask from the specified IDstart and IDend values, thereby restricting the worker to a predefined range of cores. The inner OpenMP parallel region is subsequently created within this restricted processor domain, allowing its worker threads to execute only on the assigned cores. Consequently, the operating system enforces the specified CPU affinity and prevents the inner worker threads from migrating outside their assigned processor domain. This mechanism enables 10

us to control thread placement across different stages of GNN execution, facilitating both mixed tensor/node parallelism and hierarchical node parallelism. A simple implementation of the custom thread affinity routine is shown in Listing 1. Listing 1 A simple implementation of the thread affinity routine. 1

# include < sched .h >

2 3 4 5

void set_cpu_affinity ( int ID_start , int ID_end ) { cpu_set_t mask ; CPU_ZERO (& mask ) ;

6

for ( int cpu = ID_start ; cpu <= ID_end ; ++ cpu ) CPU_SET ( cpu , & mask ) ;

7 8 9

sched_setaffinity (0 , sizeof ( mask ) , & mask ) ;

10 11

}

Then we parallelize the Aggregate phase at the node level within this range. Specifically, on a system with C cores, for a feature matrix with dimension dim, H ∈ RN ×dim , during tensor parallelism, it is divided into D partial matrices Hd ∈ RN ×(dim/D) , where d ∈ D. Each chiplet with IDstart = (C/D) × (d − 1), and IDend = (C/D) × d − 1 processes its own partial matrix with Aggregate using node parallelism. With all Hd processed, we have aggregated feature matrices (k+1) Ad , then we concatenate them into a matrix with full feature dimensions A(k+1) , and the Update function is applied.

3.2.2 Hierarchical Node Parallelism For graphs with low average node degree, tensor parallelism is not well-suited due to two main limitations. First, the overhead of applying tensor parallelism is high. The additional overhead of data partitioning, merging and synchronization in tensor parallelism can take up a large portion of computational cost of the Aggregate operation, leading to reduced overall performance. Second, the impact of hardware prefetching mechanisms shows significant differences for different graph topologies. For highly regular large embedded graphs, hardware prefetching mechanisms can fully exploit data locality and load node features sequentially into the cache, in which case the benefits of hardware locality are predictable and stable. For highly irregular large embedded graphs, even if the last-level cache can store more node features, due to the irregularity and dispersion of memory access, the node features that need to be aggregated cannot always be loaded into the LLC, and the cache hit/miss problem is difficult to improve significantly. This makes it difficult to guarantee improvement in data locality even when tensor parallelism is used. As illustrated in Figure 6 (right), when multiple cores process a highly irregular graph with low-degree nodes, applying tensor parallelism indeed results in more feature vectors being loaded in the LLC. However, the number of feature vectors that Core 0 and Core 1 can retrieve from the LLC remains similar to the pre-tensor-parallelism case. Moreover, since tensor parallelism reduces the feature dimensionality, the actual proportion of data that can be hit in the LLC decreases, degrading locality. Therefore, for sparser graphs, we still apply node parallelism, but optimize the overhead of inter-chiplet communication in a hierarchical way. Based on the analysis of the memory access pattern of graphs with extremely low average node degrees, we adopted a more conservative strategy, namely hierarchical node parallelism implemented with the same two-level nested parallel execution model. As illustrated in Algorithm 4, the outer parallel construct partitions the nodes into D parts according to the node order and maps each partition to a chiplet using the custom thread affinity routine. The worker threads within each chiplet then process approximately N/D nodes using Algorithm 2 with node parallelism in the inner parallel construct. For regular graphs, each node has a limited number of neighbors arranged in a predictable pattern. As a result, when we execute Aggregate with node parallelism within each chiplet, the feature vectors needed are already loaded or prefetched into the LLCs. For graphs with highly uneven distribution, although the memory access pattern is still irregular, as we fixed certain range of nodes for each chiplet, the variability of executing Aggregate is reduced compared to single level node parallelism. Threads are scheduled dynamically by OpenMP according to work-load balance regardless of graph structure in single level node parallelism, which brings more uncertainty 11

Algorithm 4 Hierarchical Node Parallelism Require: Input feature matrix H(k) , number of chiplets D, edge list E, node list V, number of nodes N Ensure: Output feature matrix H(k+1) 1: // Start Aggregate, Node Parallelism across chiplets 2: for d = 1 to D in parallel do 3: // Compute core IDs for each chiplet 4: vstart = (d − 1) ∗ (N/D) 5: vend = vstart + (N/D) 6: (IDstart , IDend ) ← CoreGroup(d) 7: Place all threads in (IDstart , IDend ) 8: // Node Parallelism across Cores within a chiplet 9: for v = vstart to (vend − 1) in parallel do 10: Create a buffer bv with size dim 11: (estart , eend ) ← V(v ) 12: av ← Aggregate(H(k) , bv , E, estart , eend ) 13: A(k+1) [v ] ← av 14: end for 15: end for 16: // Start Update (k+1) 17: H ← Update(A(k+1) , W) (k+1) 18: return H to feature vectors in each last level cache. When hierarchical parallelism is applied, feature vectors are managed in a more regular way on an inter-chiplet level, ensuring a more deterministic distribution of the feature vectors across chiplets. This more deterministic data distribution may not be optimal, but it provides a baseline for memory access, compared to the less predictable data replacement relying solely on the OpenMP scheduling strategy. After Aggregate, the feature vector will be written to the aggregated feature matrix A(k+1) directly. When all feature vectors are written, Update is applied.

3.3 Parallelization of Update Phase To optimize performance further, we parallelize the Update phase, i.e., dense neural network layers. Deep learning libraries like PyTorch heavily rely on BLAS as their backend, but BLAS is not specifically optimized for chiplet-based architectures. In the design of classical BLAS [47], the cache hierarchy is modeled as a single logical cache layer, without distinguishing execution units or cache domains. Cache residency is treated as a global modeling state that enables amortization of data reuse, and the model does not assume or specify any form of communication or coherence mechanism between multiple caches. However, on chiplet-based CPUs, reuse across these cache partitions requires explicit cross-LLC communication, which incurs substantial latency and bandwidth overhead; moreover, the access latency from each core’s private cache to the LLC is no longer uniform, contradicting the implicit uniform-LLC-access assumption underlying BLAS kernels. For example, see Figure 7, where matrix A is loaded into the LLC. On monolithic chips, A resides in a shared LLC that is directly accessible by the private caches of all cores, allowing private-cache–to–LLC access latency to be modeled uniformly across cores. In contrast, on chiplet-based architectures, A is partitioned into A′ and A′′ and placed in the LLCs of different chiplets; due to the expensive cost of cross-LLC communication, intra-chiplet LLC accesses and inter-chiplet LLC accesses exhibit fundamentally different latencies and cannot be captured by a unified access model. As a result, cache reuse patterns in matrix multiplication that are effective under the classical cache abstraction may lead to significant performance degradation on chiplet-based architectures. To minimize inter-chiplet communication while still leveraging the highly optimized BLAS kernels at lower cache levels, a simple and effective approach is to execute high-performance BLAS independently within each compute unit that owns private L1–L2 caches. This design avoids modifying BLAS itself, preserves its optimized use of low-level caches, and prevents excessive cache-to-cache traffic caused by the mismatch between the classical BLAS cache abstraction and the physically distributed last-level caches on chiplet-based CPUs. Each unit performs a single 12

Monolithic Chip Private

Private

Private

Chiplets Private

Private

Matrix A

Cache

Private

Matrix A'

Matrix

Chip

Local LLC Access

Private

Private

A"

Chiplet Remote LLC Access

Fig. 7 Cache accesses of BLAS on monolithic chip architecture (left) and chiplet-based architecture (right).

cblas sgemm call on its assigned block and writes results to the corresponding output region. This design leverages OpenMP to manage parallelism, reducing scheduling overhead. Algorithm 5 Parallel Update phase across Chiplets N ×din 1: Input: Feature matrix A ∈ R , weight matrix W ∈ Rdin ×dout , output matrix H ∈ N ×dout R , number of threads T 2: Output: Updated matrix H 3: Divide N rows into T chunks: chunk size ← ⌊N/T ⌋ 4: for t = 0 to T − 1 in parallel do 5: rstart ← t × chunk size 6: if t = T − 1 then 7: rows ← N − rstart 8: else 9: rows ← chunk size 10: end if 11: Hrstart :rstart +rows,: ← BLAS GEMM(Arstart :rstart +rows,: , W) 12: end for The proposed strategy is particularly effective when the embedding dimensionality is relatively small compared to the number of nodes, as it improves weight-matrix reuse across multiple row blocks. Moreover, by preserving data locality within each block and maintaining thread affinity, the approach naturally mitigates cross-chiplet data movement—a critical bottleneck in multi-chiplet processor architectures. Figure 8 compares the inference time of matrix multiplication in the update phase of GCN between OpenBLAS SGEMM and the optimized FrontGI implementation. All experiments use a feature dimension of 1024 and are evaluated on OGBN-Proteins (132,534 nodes), Reddit graph (232,965 nodes), Yelp graph (716,847 nodes), and OGBG-Molhiv (1,049,163 nodes). See Table 2 for information on each of the graphs.

3.4 Fusion of Linear Operators When a linear operator exists between the Aggregate and Update phases, or is embedded within the Aggregate phase itself—such as in GIN inference illustrated in Figure 4, where the aggregated features are combined with the node’s initial features through a weighted sum (i.e., an AXPY operation)—we leverage the decomposability and composability of linear operators to fuse this step within the aggregation parallelism. Specifically, during hierarchical node parallelism, each chiplet executes Aggregate-Linear for approximately N/D nodes. Similarly, under mixed parallelism, after partitioning the input feature dimensions across chiplets, each chiplet performs Aggregate-Linear on its assigned feature dimensions, followed by a synchronization step to merge partial results.

13

Fig. 8 Inference time comparison between OpenBLAS and FrontGI’s matrix multiplication in GCN’s update phase.

3.5 Memory Access Threads Placement As discussed in Section 2.1, under a chiplet-based architecture, inter-chiplet communication is relatively costly. Therefore, for memory-intensive operations such as dimension-wise partitioning and concatenation in tensor parallelism, as well as activation functions without complex operations, such as ReLU, we aim to localize these operations within as few chiplets as possible. This approach minimizes inter-chiplet data communication and maximizes data locality. At the same time, we leverage all available threads within these chiplets to exploit parallelism and achieve higher computational efficiency.

4 Experiments 4.1 Experimental Methodology 4.1.1 Datasets To evaluate our proposed methods, we adopt 12 representative graph datasets from the Open Graph Benchmark [29], Konect Graphs [48], Graph500 [49] and other public datasets [30, 50–52] with diverse properties, as shown in Table 2. These graph datasets feature distinct average node degrees ranging from 2.1 to 597.0 and follow either the uniform or power-law [53] degree distribution, indicating diverse irregular memory access patterns which pose unique challenges for graph neural network inference. To quantify the degree distribution, we adopt the power-law exponent α. A lower α value indicates a less uniform graph degree distribution, which corresponds to a more irregular memory access pattern, and vice versa. Typically, highly power-law-like distributions are characterized by α < 3. Here we note that α is not applicable to graphs with degree distributions close to standard uniform (e.g., MT and MH shown in Table 2). These graphs are converted to CSR format using GNNHLS [17]. In addition, we set feature dimensions to be 1024 and adopt 32-bit floating point as the data format, consistent with previous work.

4.1.2 Evaluation Methods To perform the evaluation, we use an AMD EPYC 9754 multicore server featuring 128 cores at 2.25 GHz. Each core has its own L1d (32 KiB) cache, L1i (32 KiB) cache, and L2 (1 MiB) cache. These cores, along with their exclusive caches, are symmetrically distributed across 16 chiplets, with each chiplet sharing a 16 MiB L3 cache. The system is equipped with 128 GiB of main memory and runs Rocky Linux 9.5. All code is compiled using GCC 11.5.0 with OpenMP 4.5, and simultaneous multithreading (SMT) is disabled. Therefore, 128 threads are available for model execution. We compare our methods with two widely-used standard GNN libraries, DGL 2.4.0 and PyG 2.6.1, with PyTorch 2.3.1. For mixed parallelism during the Aggregate phase, we split the H ∈ RN ×1024 into 16 partial matrices, each chiplet processes a corresponding submatrix Hd ∈ RN ×64 . For hierarchical node parallelism, each chiplet handles N/16 nodes. During the Update phase, we adopt the OpenBLAS library based on OpenMP to perform matrix multiplication, configured to leverage all the 128 cores.

14

Table 2 Graph Information.

ID

Name

Size

Ref.

Max.

Avg.

Power-Law α

145,459 302,190 6 1,049,163 2,259,376 10 169,343 1,166,243 13,155 716,847 13,954,819 4,886 159,756 4,887,982 28,372 335,319 15,459,349 41,357 149,701 10,897,472 80,635 621,512 46,358,909 193,228 781,266 121,132,159 345,076 1,000,991 256,804,235 468,366 232,965 114,848,857 21,657 132,534 79,122,504 7,750

2.1 2.2 6.9 19.5 30.6 46.1 72.8 74.6 155.1 256.8 492.9 597.0

N/A N/A 2.56 2.41 2.27 1.54 2.12 2.26 1.71 2.34 5.01 6.89

Nodes MT MH AX YP KA S19 PC HU RT YH RD PT

OGBG-Moltox21 OGBG-Molhiv OGBN-Arxiv Yelp Graph Wikipedia (ka) Graph500-Scale19 Petster-Carnivore Wikipedia (hu) Reuters Graph Yahoo Music Reddit Graph OGBN-Proteins

[29] [29] [29] [50] [48] [49] [48] [48] [52] [51] [30] [29]

Degree Edges

4.2 Evaluation In this section, we present results of our evaluation, including performance, scalability, and LLC hit/miss rates.

4.2.1 Performance We first evaluate performance improvement by showing the overall speedup, defined as the execution time of FrontGI relative to DGL and PyG. Tables 3, 4, 5, and 6 show the execution time of DGL and both FrontGI methods on a single GCN, GraphSAGE, GIN and SGC layer, where FrontGI-H refers to Hierarchical Node Parallelism and FrontGI-M refers to Mixed Tensor/Node Parallelism. For some of the graphs, the table includes execution time for PyG, where OoM denotes an Out-of-Memory error. Table 3 Execution Time (s) per GCN Layer.

Table 4 Execution Time (s) per GraphSAGE Layer.

Name

FrontGI-M

Name

PyG

DGL

FrontGI-H

FrontGI-M

MT MH AX YP KA

0.3320 2.2967 0.5256 2.3794 0.7776

0.2862 1.9868 0.3427 1.8101 0.3776

0.1527 0.9590 0.1989 1.0098 0.2683

0.2029 1.3165 0.2685 1.2182 0.3095

S19 PC HU RT YH RD PT

2.0299 1.9756 OoM OoM OoM OoM OoM

0.8036 0.5162 1.8782 3.9889 9.7020 2.0868 1.0003

0.6773 0.5470 1.3938 8.7681 24.4510 3.1063 1.2602

0.6016 0.2438 1.0912 2.1252 3.4640 0.6724 0.3128

PyG

DGL

FrontGI-H

MT MH AX YP

0.1808 1.2745 0.2992 1.7879

0.1550 1.0367 0.1955 0.9806

0.0952 0.4923 0.1294 0.7271

0.1492 1.0875 0.1983 0.9139

KA S19 PC HU RT YH RD PT

0.5835 1.6908 1.6025 OoM OoM OoM OoM OoM

0.2257 0.5120 0.3846 1.2663 3.3629 8.8921 1.9101 0.8764

0.2081 0.5584 0.4736 1.1742 8.5288 24.3678 2.7651 1.1554

0.1962 0.4397 0.2158 0.8178 2.0984 3.1785 0.5859 0.2615

Table 5 Execution Time (s) per GIN Layer.

Table 6 Execution Time (s) per SGC Layer.

Name

PyG

DGL

FrontGI-H

FrontGI-M

Name

PyG

DGL

FrontGI-H

FrontGI-M

MH AX KA

2.3629 0.5015 0.7733

2.0592 0.3694 0.3920

0.9508 0.2291 0.2765

1.3556 0.2849 0.2797

MH AX KA

2.2995 0.8240 1.9546

1.5261 0.3060 0.3765

0.5886 0.1997 0.3225

2.1087 0.3626 0.3501

PC YH RD

1.8834 OoM OoM

0.5357 9.8108 2.0795

0.5246 24.1607 2.8511

0.2712 3.5115 0.7204

PC YH RD

5.7420 OoM OoM

0.7198 17.5066 3.5987

0.9028 50.0570 5.6496

0.3644 6.7910 1.1618

15

Figures 9, 10, 11, and 12 respectively show the speedup for GCN, GraphSAGE, GIN, and SGC achieved by PyG standard library and our proposed FrontGI methods, including mixed tensor/node parallelism and hierarchical node parallelism, relative to DGL.

Fig. 9 Single GCN layer speedup relative to DGL standard library (Higher is better).

Fig. 10 Single GraphSAGE layer speedup relative to DGL standard library.

Fig. 11 Single GIN layer (with a 2-layer-MLP) speedup relative to DGL.

Fig. 12 SGC model (with K = 2) speedup relative to DGL.

In comparison with DGL, a common trend is observed. For graphs with lower average degrees, FrontGI’s hierarchical node parallelism outperforms DGL, achieving speedup from 1.35× to 1.82× for a GCN layer, 1.72× to 2.07× for a GraphSAGE layer, 1.42× to 2.17× for a GIN layer with a 2-layer MLP, and 1.17× to 2.59× for a SGC model with 2 rounds within Aggregate. In contrast, for graphs with high average node degree, the mixed tensor/node parallelism method delivers superior performance, reaching approximately 3.35× the inference speed of DGL for GCN, 3.20× for GraphSAGE, 2.89× for GIN and 3.10× for SGC. When compared to PyG, both FrontGI strategies outperform its implementation (on PC graph, FrontGI achieved up to 7.42×, 8.10×, 6.94× and 15.76× with different models), since PyG’s performance is consistently far below that of DGL and it frequently runs into out-of-memory errors on large graphs with high average degree. Since the irregularity of GNN models is mainly reflected in the Aggregate phase, we next focus our evaluation on this phase for both proposed methods across all the graph datasets. Table 7 shows the execution time of the Aggregate phase and the Update phase for mixed parallelism and hierarchical node parallelism.

16

Table 7 Profiling a Single GCN Layer on 128 Cores. Name

Hierarchical Node

Mixed Tensor/Node

Aggregate

Update

Aggregate

Update

Overhead

MT MH AX

0.0185 0.0966 0.0491

0.0749 0.3940 0.0800

0.0419 0.3504 0.0753

0.0628 0.4672 0.0708

0.0422 0.2578 0.0493

YP KA S19 PC HU RT YH RD PT

0.4278 0.1166 0.3915 0.3850 0.8993 8.2007 23.9598 2.6429 1.0845

0.2896 0.0879 0.1638 0.0817 0.2933 0.3197 0.4010 0.1194 0.0691

0.4201 0.0782 0.1935 0.0772 0.3895 1.4582 2.4293 0.4239 0.1628

0.3338 0.0677 0.1650 0.0934 0.2896 0.3632 0.4496 0.0956 0.0586

0.1521 0.0480 0.0771 0.0432 0.1316 0.2674 0.2864 0.0651 0.0382

In particular, the last column Overhead means the overhead of implementing tensor paral(k) (k+1) lelism within mixed parallelism, i.e., partitioning Hd and merging Ad . Figure 13 shows the normalized inference time and the comparison between different techniques. The comparison shown in the table and the figure confirms our assumption regarding the impact of processor architecture features on irregular memory access.

Fig. 13 Normalized GCN layer profiling (for each dataset, DGL = 1).

When mixed parallelism is applied, the Aggregate time for graphs with an average degree larger than 19 (as in YP) begins to decrease. When the average degree reaches approximately 46 (as in S19), the Aggregate time is reduced significantly enough to offset the additional overhead introduced by tensor parallelism. Additionally, we calculate the overhead-to-gain ratio ROTG when mixed parallelism outperforms hierarchical node parallelism, which quantifies the proportion of overhead of mixed parallelism relative to the performance improvement as follows:

toverhead toverhead = (8) tgain thierarch − (tmixed − toverhead ) where toverhead is Mixed Overhead in Table 7, tgain stands for the benefit obtained from mixed parallelism compared to the best performance of hierarchical parallelism. We calculate ROTG only when tgain is positive. thierarch is the minimum execution time of Aggregate with hierarchical node parallelism across different core number settings, tmixed is the execution time of Aggregate with mixed tensor/node parallelism. Comparing to tensor parallelism’s benefit, its overhead is acceptable (ROTG = 3.96% for RT, 1.33% for YH, 4.14% for PT and 2.93% for RD). For sparser ROTG =

17

graphs, the time taken by tensor parallelism’s overhead is comparable to the Aggregate inference time of hierarchical node parallelism in the 128-core setting, and it can take up to 2.46× as long as hierarchical node parallelism Aggregate in the optimal 32-core setting on graphs with regular memory access.

4.2.2 Impact of Graph Properties We observe that the choice of the best-performing parallelization strategy is strongly correlated with the average node degree of the graph. For graphs such as MT, MH, AX, and YP, all of which have an average degree less than 20, hierarchical node parallelism clearly performs the best. For KA and S19, whose average degrees range from about 20 up to around 50, hierarchical node parallelism and mixed tensor/node parallelism perform similarly. In contrast, for graphs like PC, HU, RT, YH, RD, and PT, which have average degrees greater than 50, mixed tensor/node parallelism significantly outperforms hierarchical node parallelism. This is counter to conventional wisdom, in which one assumes that the distribution of the graph degree would have a large performance impact. Here, MT and MH are fairly uniform in distribution, while S19 and RT are quite non-uniform (low α power-law exponent). Yet, this factor was not what distinguished the best performing parallelization technique.

4.2.3 Scalability Figure 14 shows the scalability of FrontGI (hierarchical node parallelism for MH and mixed tensor/node parallelism for PT and RD) and repeats the scalability of DGI from Figure 1. The inference speed of 8 cores (for both FrontGI and DGL on each graph) is normalized to 1. In this experiment, we scale the system by progressively adding chiplets rather than merely increasing core counts. This setup directly follows our chiplets-aware system design, ensuring that the scalability evaluation reflects realistic architectural expansion. Comparing to Figure 1, the performance of FrontGI significantly surpasses DGL with increasing core counts. When scaling up to 32 cores, the speedup relative to 8 cores for DGL begins to plateau, while FrontGI continues to exhibit rapid performance growth until the growth rate starts to decrease when scaling above 64 cores. When the number of cores exceeds 96, DGL’s performance begins to decline, while FrontGI maintains and keeps improving its performance up to the full utilization of 128 cores. FrontGI’s scalability is without question an improvement on that of the standard library.

Fig. 14 Comparison of FrontGI and DGL scalability on PT, RD and MH graphs (relative to 8 cores).

18

4.2.4 LLC Hit/Miss In modern multicore systems, LLC cache accesses have a significant impact on the communications between chiplets. We believe that for multicore architectures with multiple chiplets designs, programming paradigms like OpenMP, which implement fork-join parallelism through simple pragmas, are likely to cause a large number of LLC misses due to the lack of thread-level control at the chiplet level. Moreover, LLC misses can result in the most significant cache miss latency and affect cache miss events in other levels. In our experiments we measure the LLC hit and miss events for each of three parallel methods, all using the FrontGI code base: traditional node parallelism, mixed tensor/node parallelism, and hierarchical node parallelism. To facilitate analysis of these data for both high and low degree graphs, we will define a cache “event” change ratio, where what constitutes an event is different for distinct comparisons. For all definitions of an event, the change ratio tells us how much FrontGI is different than traditional node parallelism for that type of event. We define the cache event change ratio REvent as follows: |EventFrontGI − EventNode | (9) REvent = EventNode where Event represents one of: (1) cache hits, (2) cache misses, or (3) cache accesses (the sum of hits and misses). Table 8 shows cache hit and miss events on high degree graphs before and after applying tensor parallelism (i.e., for pipelined node-level parallelism and mixed tensor/node parallelism). Looking Table 8 LLC Hit/Miss counts For High Degree Graphs. L3 hit

S19 PC HU RT YH PT RD

L3 miss

Node

FrontGI-M

Node

FrontGI-M

622 M 918 M 2 430 M 4 746 M 3 149 M 3 003 M 2 053 M

906 M 434 M 1 013 M 3 337 M 11 460 M 4 280 M 4 333 M

3 575 M 1 543 M 6,336 M 12 494 M 26 812 M 5 287 M 9 874 M

1 057 M 503 M 2,302 M 6 267 M 12 578 M 659 M 2 417 M

first at cache misses (Event = Miss), we observe a significant decrease (RMiss = 70.43% on S19, 67.40% on PC, 63.67% on HU, 49.83% on RT, 53.08% on YH, 87.54% on PT and 75.52% on RD). This aligns with our expectations for mixed tensor/node parallelism. Table 9 shows cache miss and hit events on low degree graphs before and after applying hierarchical node parallelism (i.e., for pipelined node-level parallelism and hierarchical node parallelism). We see that both of the cache miss and hit events decrease, implying a better hit rate on the set of L2 caches (i.e., there aren’t as many accesses to the L3 cache). Though the effects are not as significant as in the mixed parallelism case, as we illustrated, the hierarchical node parallelism is a conservative parallel strategy, which aims to reduce main memory access. Here Event = Access = (Miss + Hit), and we observe a significant ratio decrease in total accesses to the last level cache, where RAccess = 82.81% on MT, 83.17% on MH, 73.66% on AX, 68.13% on YP, and 66.10% on KA. Meanwhile, the ratio of LLC misses decreases by 74.28% on MT, 76.36% on MH, 65.68% on AX, 62.46% on YP, and 66.76% on KA. The above corroborates the notion that cache behavior is one of the mechanisms that benefit FrontGI’s parallelism approaches.

5 Related Work Optimizing GNN training and inference performance across various hardware platforms has emerged as a popular research topic. Most of the previous work has been dedicated to enhancing GNN performance on GPUs and FPGAs (with some very recent work accelerating training on NPUs [18]), exploring techniques such as kernel fusion, memory access optimization, and

19

Table 9 LLC Hit/Miss counts For Low Degree Graphs. L3 hit

MT MH AX YP KA

L3 miss

Node

FrontGI-H

Node

FrontGI-H

1 017 M 5 209 M 712 M 2 376 M 463 M

97 M 427 M 93 M 432 M 166 M

914 M 6 608 M 1 186 M 5 731 M 1,360 M

235 M 1 562 M 407 M 2 151 M 452 M

customized accelerators. These efforts aim to address the challenges posed by the irregular memory access patterns and dataflow inherent to GNN workloads, making them better suited for high-performance inference and training on specialized hardware.

5.1 GNNs on GPUs Sampling based GNN inference is the most widely used method to accelerate GNN inference on GPUs. In real world graphs, different vertices in a graph may share many common neighbors, causing redundant vertex feature vector access, loading and computation. GraphTensor [11] presents an efficient GNN processing framework by pipelining graph sampling, graph processing, and neural network execution behaviors according to the system’s hyperparameters, significantly shortening GNN execution latency. To better leverage tensor core units in GNN computing on GPUs, computing systems need to remove redundant sparse representations and balance the load of tensor cores. TC-GNN [12] was proposed to translate sparse matrices to a denser representation, compressing large sparse matrices to small sparse blocks, thus finding a balance between sparse operations and high performance dense operations fixed by hardware. PckGNN [13] considers load balancing and data locality, categorizes graph matrices into different types with sparsity levels, enhancing load balance for sparser graphs and cache data reuse for denser graphs respectively, improving sparse matrices operation speed, thus improving the aggregation operator’s efficiency. RT-GNN [14] exploits both graph sparsity and GPU computing cores (tensor cores and CUDA cores). It manages the intermediate aggregation results hierarchically to avoid redundant computation, partitioning graphs to graph blocks, then assigning graph blocks to different kinds of cores according to the blocks’ sparsity, processing blocks with tensor cores and CUDA cores simultaneously. With the growth of graph size and diversity of GNN architecture, existing libraries are often not able to handle different workflow types of different GNNs. GNNAdvisor [54] was proposed to accelerate GNN inference by optimizing hierarchical GPU memory use utilizing the properties of input graphs and GNN architectures.

5.2 GNNs on FPGAs FPGAs are a compelling solution for GNN inference acceleration due to their massive parallelism, better performance per watt than general-purpose computing platforms, reconfigurability for enhanced flexibility, and the ability to leverage concurrent data flow for efficient execution. AWB-GCN [15], one of the earliest FPGA-based GCN accelerators, addresses workload imbalance caused by the distinct sparsity levels of different graph components using HDL. GenGNN [16] introduces a framework for accelerating GNNs with real-time requirements by storing the entire graph and corresponding intermediate results in the on-chip resources of the FPGA. GNNHLS [17] presents a comprehensive framework comprising both software and hardware components to accelerate various GNN models on FPGAs using High-Level Synthesis (HLS). HLPerf [55] proposes a simulation-based performance evaluation framework for GNN inference, enabling early design space exploration and significantly reducing performance evaluation cycles. The develop time for GNN acceleration on FPGAs remains non-trivial due to the excessive design space to be explored, prolonged compilation times, and the inherent differences from conventional software programming languages. In fact, the performance gap between a well-optimized and a naive implementation of the same kernel can span two to three orders of magnitude [56, 57].

20

6 Conclusions and Future Work Libraries such as DGL and PyG already offer an excellent high-productivity development environment for graph neural network applications. However, as the available execution platforms modernize, it is essential to explore how they implement the underlying computations. As processor core counts scale up, the techniques needed to achieve high performance on many applications vary. This is clearly the case for inference on GNNs executing on high core-count modern processors. Motivated by the memory hierarchy of these systems, we have proposed a pair of parallelization techniques for GNN inference and have assessed their efficacy empirically using several GNN models and twelve representative graphs. The techniques we propose are straightforward to incorporate into any of the high-productivity environments designed for GNN inference. Our experimental results show performance gains of up to 3.35× relative to the higher performing of two commonly used standard libraries. In addition, which parallelization technique is better depends in large part on the topological properties of the graph. In our case, graphs with low average degree (< 20) scaled better using the hierarchical node parallelism technique, graphs with high average degree (> 50) scaled better using the mixed tensor/node parallelism technique and for graphs whose average degree falls in the range of 20 to 50, either parallelization strategy of FrontGI proves effective. This work can be extended in a number of ways. First, we explored scaling up the inference of GCN, GraphSAGE, GIN, and SGC. However, additional models worth investigation are described in a recent review by Khemani et al. [58], e.g., parallelizing graph attention in GAT [59] with our method. Second, while this work is focused on inference, there is a possibly larger need for acceleration of GNN training, where automatically selecting efficient parallel strategies may provide increasing benefits as the number of cores in a system grows. Before one incorporates FrontGI approaches into production libraries such as DGL or PyG, its effectiveness across a broader set of circumstances (both GNN models and graph datasets) is required.

References [1] Fan, W., Ma, Y., Li, Q., Wang, J., Cai, G., Tang, J., Yin, D.: A graph neural network framework for social recommendations. IEEE Transactions on Knowledge and Data Engineering 34(5), 2033–2047 (2022) https://doi.org/10.1109/TKDE.2020.3008732 . IEEE [2] Tabassum, S., Pereira, F.S., Fernandes, S., Gama, J.: Social network analysis: An overview. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 8(5), 1256 (2018) https://doi.org/10.1002/widm.1256 . Wiley [3] Wu, S., Sun, F., Zhang, W., Xie, X., Cui, B.: Graph neural networks in recommender systems: A survey. ACM Comput. Surv. 55(5), 97 (2022) https://doi.org/10.1145/3535101 . ACM [4] Zhang, X.-M., Liang, L., Liu, L., Tang, M.-J.: Graph neural networks and their current applications in bioinformatics. Frontiers in Genetics 12, 690049 (2021) https://doi.org/10.3389/ fgene.2021.690049 . Frontiers Media SA [5] Dong, Z., Cao, W., Zhang, M., Tao, D., Chen, Y., Zhang, X.: CktGNN: Circuit graph neural network for electronic design automation. In: Int’l Conference on Learning Representations (2023) [6] Han, K., Wang, Y., Guo, J., Tang, Y., Wu, E.: Vision GNN: An image is worth graph of nodes. Advances in Neural Information Processing Systems 35, 8291–8303 (2022) [7] Wu, L., Chen, Y., Ji, H., Liu, B.: Deep learning on graphs for natural language processing. In: Int’l Conf. on Research and Development in Information Retrieval, pp. 2651–2653 (2021). https://doi.org/10.1145/3404835.3462809 . ACM [8] Wu, X., Shen, Y., Shan, C., Song, K., Wang, S., Zhang, B., Feng, J., Cheng, H., Chen, W., Xiong, Y., et al.: Can graph learning improve planning in LLM-based agents? Advances in Neural Information Processing Systems 37, 5338–5383 (2024)

21

[9] Fey, M., Lenssen, J.E.: Fast graph representation learning with PyTorch Geometric. arXiv preprint arXiv:1903.02428 (2019). https://doi.org/10.48550/arXiv.1903.02428 [10] Wang, M.Y.: Deep Graph Library: Towards efficient and scalable deep learning on graphs. In: ICLR Workshop on Representation Learning on Graphs and Manifolds (2019) [11] Jang, J., Kwon, M., Gouk, D., Bae, H., Jung, M.: GraphTensor: comprehensive GNNacceleration framework for efficient parallel processing of massive datasets. In: Int’l Parallel and Distributed Processing Symp., pp. 2–12 (2023). https://doi.org/10.1109/IPDPS54959. 2023.00011 . IEEE [12] Wang, Y., Feng, B., Wang, Z., Huang, G., Ding, Y.: TC-GNN: Bridging sparse GNN computation and dense tensor cores on GPUs. In: USENIX Annual Technical Conference, pp. 149–164 (2023) [13] Hu, Z., Sun, J., Li, Z., Sun, G.: PckGNN: Optimizing aggregation operators with packing strategies in graph neural networks. In: Int’l Parallel and Distributed Processing Symp., pp. 2–13 (2024). https://doi.org/10.1109/IPDPS57955.2024.00010 . IEEE [14] Yan, J., Jiang, W., He, D., Wen, S., Li, Y., Jin, H., Shao, Z.: RT-GNN: Accelerating sparse graph neural networks by Tensor-CUDA kernel fusion. ACM Trans. on Architecture and Code Optimization 22(1), 32 (2025) https://doi.org/10.1145/3702001 . ACM [15] Geng, T., Li, A., Shi, R., Wu, C., Wang, T., Li, Y., Haghi, P., Tumeo, A., Che, S., Reinhardt, S., Herbordt, M.C.: AWB-GCN: A graph convolutional network accelerator with runtime workload rebalancing. In: 53rd Int’l Symp. on Microarchitecture, pp. 922–936 (2020). https: //doi.org/10.1109/MICRO50266.2020.00079 . IEEE [16] Abi-Karam, S., He, Y., Sarkar, R., Sathidevi, L., Qiao, Z., Hao, C.: GenGNN: A Generic FPGA Framework for Graph Neural Network Acceleration. arXiv preprint arXiv:2201.08475 (2022). https://doi.org/10.48550/arXiv.2201.08475 [17] Zhao, C., Dong, Z., Chen, Y., Zhang, X., Chamberlain, R.D.: GNNHLS: Evaluating graph neural network inference via high-level synthesis. In: 41st Int’l Conference on Computer Design, pp. 574–577 (2023). https://doi.org/10.1109/ICCD58817.2023.00092 . IEEE [18] Ai, X., Zhang, B., Wang, Q., Zhang, Y., Yuan, H., Gong, S., Yu, G.: NeutronAscend: Optimizing GNN training with Ascend AI processors. ACM Transactions on Architecture and Code Optimization 22(4), 123 (2025) https://doi.org/10.1145/3762662 . ACM [19] Gong, Z., Ji, H., Yao, Y., Fletcher, C.W., Hughes, C.J., Torrellas, J.: Graphite: optimizing graph neural networks on CPUs through cooperative software-hardware techniques. In: 49th Int’l Symposium on Computer Architecture, pp. 916–931 (2022). https://doi.org/10.1145/ 3470496.3527403 . ACM [20] Chen, K., Su, H., Yang, X., An, Z., Dou, Y., Li, D.: Quantitative analysis and performance optimization of graph neural networks on multi-core CPUs. ACM Transactions on Architecture and Code Optimization 23(2), 53 (2026) https://doi.org/10.1145/3807957 . ACM [21] Ma, L., Yang, Z., Miao, Y., Xue, J., Wu, M., Zhou, L., Dai, Y.: NeuGraph: Parallel deep neural network computation on large graphs. In: USENIX Annual Technical Conference, pp. 443–458 (2019) [22] Özsoy, A.: A comprehensive performance comparison of dedicated and embedded GPU systems. Dicle Üniversitesi Mühendislik Fakültesi Mühendislik Dergisi 11(3), 1011–1020 (2020) https://doi.org/10.24012/dumf.742732 . Dicle University [23] Fu, Q., Huang, H.H.: Automatic generation of high-performance inference kernels for graph neural networks on multi-core systems. In: 50th Int’l Conf. on Parallel Processing (2021).

22

https://doi.org/10.1145/3472456.3473511 . ACM [24] Vasimuddin Md, Misra, S., Ma, G., Mohanty, R., Georganas, E., Heinecke, A., Kalamkar, D., Ahmed, N.K., Avancha, S.: DistGNN: Scalable distributed training for large-scale graph neural networks. In: Int’l Conference for High Performance Computing, Networking, Storage and Analysis (2021). https://doi.org/10.1145/3458817.3480856 . ACM [25] Lin, Y.-C., Chen, Y., Gobriel, S., Jain, N., Jha, G.K., Prasanna, V.: ARGO: An auto-tuning runtime system for scalable GNN training on multi-core processor. In: International Parallel and Distributed Processing Symposium, pp. 361–372 (2024). https://doi.org/10.1109/ IPDPS57955.2024.00039 . IEEE [26] Ji, B., Zhao, C., Chamberlain, R.D.: FGI: Fast GNN inference on multi-core systems. In: IEEE Int’l Parallel and Distributed Processing Symposium Workshops, pp. 748–757 (2025). https://doi.org/10.1109/IPDPSW66978.2025.00119 . IEEE [27] Gupta, A., Kumar, V.: Performance properties of large scale parallel systems. Journal of Parallel and Distributed Computing 19(3), 234–244 (1993) https://doi.org/10.1006/jpdc.1993.1107 . Elsevier [28] Whaley, R.C., Petitet, A., Dongarra, J.J.: Automated empirical optimizations of software and the ATLAS project. Parallel computing 27(1-2), 3–35 (2001) https://doi.org/10.1016/ S0167-8191(00)00087-9 . Elsevier [29] Hu, W., Fey, M., Zitnik, M., Dong, Y., Ren, H., Liu, B., Catasta, M., Leskovec, J.: Open Graph Benchmark: Datasets for machine learning on graphs. Advances in Neural Information Processing Systems 33, 22118–22133 (2020) [30] Hamilton, W., Ying, Z., Leskovec, J.: Inductive representation learning on large graphs. Advances in Neural Information Processing Systems 30 (2017) [31] Li, T., Hou, J., Yan, J., Liu, R., Yang, H., Sun, Z.: Chiplet heterogeneous integration technology—status and challenges. Electronics 9(4), 670 (2020) https://doi.org/10.3390/ electronics9040670 . MDPI [32] Bhargava, R., Troester, K.: AMD Next-Generation “Zen 4” Core and 4th Gen AMD EPYC Server CPUs. IEEE Micro 44(3), 8–17 (2024) https://doi.org/10.1109/MM.2024.3375070 . IEEE [33] Nassif, N., Munch, A.O., Molnar, C.L., Pasdast, G., Lyer, S.V., Yang, Z., Mendoza, O., Huddart, M., Venkataraman, S., Kandula, S., et al.: Sapphire Rapids: The next-generation Intel Xeon scalable processor. In: Int’l Solid-State Circuits Conference, vol. 65, pp. 44–46 (2022). https://doi.org/10.1109/ISSCC42614.2022.9731107 . IEEE [34] Smith, A., Chapman, E., Patel, C., Swaminathan, R., Wuu, J., Huang, T., Jung, W., Kaganov, A., McIntyre, H., Mangaser, R.: AMD InstinctTM MI300 series modular chiplet package–HPC and AI accelerator for exa-class systems. In: Int’l Solid-State Circuits Conference, vol. 67, pp. 490–492 (2024). https://doi.org/10.1109/ISSCC49657.2024.10454441 . IEEE [35] Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016). https://doi.org/10.48550/arXiv.1609.02907 [36] Xu, K., Hu, W., Leskovec, J., Jegelka, S.: How powerful are graph neural networks? arXiv preprint arXiv:1810.00826 (2018). https://doi.org/10.48550/arXiv.1810.00826 [37] Weisfeiler, B., Leman, A.: The reduction of a graph to canonical form and the algebra which appears therein. Nauchno-Technicheskaya Informatsia 2(9), 12–16 (1968)

23

[38] Wu, F., Souza, A., Zhang, T., Fifty, C., Yu, T., Weinberger, K.: Simplifying graph convolutional networks. In: International Conference on Machine Learning, pp. 6861–6871 (2019). PMLR [39] Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., Catanzaro, B.: Megatron-LM: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 (2019). https://doi.org/10.48550/arXiv.1909.08053 [40] Wang, B., Xu, Q., Bian, Z., You, Y.: Tesseract: Parallelize the tensor parallelism efficiently. In: 51st Int’l Conference on Parallel Processing (2022). https://doi.org/10.1145/3545008.3545087 . ACM [41] Paszke, A., et al.: PyTorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems 32 (2019) [42] Dongarra, J.J., Du Croz, J., Hammarling, S., Duff, I.S.: A set of level 3 basic linear algebra subprograms. ACM Transactions on Mathematical Software 16(1), 1–17 (1990) https://doi. org/10.1145/77626.79170 . ACM [43] Zhao, C., Chamberlain, R.D., Zhang, X.: SuperCut: Communication-aware partitioning for near-memory graph processing. In: 20th Int’l Conference on Computing Frontiers, pp. 42–51 (2023). https://doi.org/10.1145/3587135.3592209 . ACM [44] Kyrola, A., Blelloch, G., Guestrin, C.: GraphChi: Large-Scale graph computation on just a PC. In: 10th USENIX Symposium on Operating Systems Design and Implementation, pp. 31–46 (2012) [45] Gera, P., Kim, H.: Traversing large compressed graphs on GPUs. In: Int’l Parallel and Distributed Processing Symposium, pp. 25–35 (2023). https://doi.org/10.1109/IPDPS54959. 2023.00013 . IEEE [46] Khorasani, F., Vora, K., Gupta, R., Bhuyan, L.N.: CuSha: Vertex-centric graph processing on GPUs. In: 23rd Int’l Symp. on High-performance Parallel and Dist. Computing, pp. 239–252 (2014). https://doi.org/10.1145/2600212.2600227 . ACM [47] Goto, K., van de Geijn, R.A.: Anatomy of high-performance matrix multiplication. ACM Transactions on Mathematical Software (TOMS) 34(3), 1–25 (2008) https://doi.org/10.1145/ 1356052.1356053 . ACM [48] Kunegis, J.: KONECT: the Koblenz network collection. In: 22nd Int’l Conference on World Wide Web, pp. 1343–1350 (2013). https://doi.org/10.1145/2487788.2488173 . ACM [49] Murphy, R.C., Wheeler, K.B., Barrett, B.W., Ang, J.A.: Introducing the Graph 500. In: Cray Users Group (2010) [50] Zeng, H., Zhou, H., Srivastava, A., Kannan, R., Prasanna, V.: GraphSAINT: Graph sampling based inductive learning method. arXiv preprint arXiv:1907.04931 (2019). https://doi.org/10. 48550/arXiv.1907.04931 [51] Dror, G., Koenigstein, N., Koren, Y., Weimer, M.: The Yahoo! Music Dataset and KDDCup’11. In: Proceedings of KDD Cup 2011, pp. 3–18 (2012). PMLR [52] Lewis, D.D., Yang, Y., Rose, T.G., Li, F.: RCV1: A new benchmark collection for text categorization research. Journal of Machine Learning Research 5, 361–397 (2004) [53] Clauset, A., Shalizi, C.R., Newman, M.E.: Power-law distributions in empirical data. SIAM Review 51(4), 661–703 (2009) https://doi.org/10.1137/070710111 [54] Wang, Y., Feng, B., Li, G., Li, S., Deng, L., Xie, Y., Ding, Y.: GNNAdvisor: An adaptive and efficient runtime system for GNN acceleration on GPUs. In: 15th USENIX Symposium 24

on Operating Systems Design and Implementation, pp. 515–531 (2021) [55] Zhao, C., Faber, C.J., Chamberlain, R.D., Zhang, X.: HLPerf: Demystifying the performance of HLS-based graph neural networks with dataflow architectures. ACM Transactions on Reconfigurable Technology and Systems 18(1), 2 (2025) https://doi.org/10.1145/3655627 . ACM [56] Sohrabizadeh, A., Yu, C.H., Gao, M., Cong, J.: AutoDSE: Enabling software programmers to design efficient FPGA accelerators. ACM Transactions on Design Automation of Electronic Systems 27(4), 32 (2022) https://doi.org/10.1145/3494534 . ACM [57] Faber, C.J., Harris, S.D., Xiao, Z., Chamberlain, R.D., Cabrera, A.M.: Challenges designing for FPGAs using high-level synthesis. In: Proc. of High-Performance Extreme Computing Conference (2022). https://doi.org/10.1109/HPEC55821.2022.9926398 . IEEE [58] Khemani, B., Patil, S., Kotecha, K., Tanwar, S.: A review of graph neural networks: concepts, architectures, techniques, challenges, datasets, applications, and future directions. Journal of Big Data 11(1), 18–11842 (2024) https://doi.org/10.1186/s40537-023-00876-4 . Springer [59] Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., Bengio, Y.: Graph attention networks. arXiv preprint arXiv:1710.10903 (2017) https://doi.org/10.48550/arXiv.1710.10903

25

Related documents

• AcOrch: Accelerating Sampling-based GNN Training under CPU-NPU Heterogeneous Environments#246523
• FrontGI: Architecture-Aware Parallel GNN Inference on Chiplet-Based Multicore Systems#1458065
• GPU-Accelerated Hypergraph Partitioning and Placement to Map SNNs on Neuromorphic Hardware#984489
• mKernel: Fast Multi-GPU, Multi-Node Fused Kernels#1071362
• mKernel: Fast Multi-GPU, Multi-Node Fused Kernels#919318