GARDEN: Efficient GPU-Accelerated Billion-Scale GNN Training via Caching-Sampling-Batching Co-Design

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

1

GARDEN: Efficient GPU-Accelerated Billion-Scale GNN Training via Caching-Sampling-Batching Co-Design

arXiv:2611.11043v1 [cs.DC] 8 Nov 2026

Qingrui Zhu , Lei Liu , Yushu Cai , Kai Sheng , Zhiwen Chen , Member, IEEE,, Xin He

Abstract—Graph Neural Networks (GNNs) have shown impressive performance in diverse tasks such as social network analysis and recommendation systems. However, scaling their training to billion-edge graphs is hindered by the memory wall: GPU memory capacity falls short of the massive storage requirements for node features and topology. Sampling-based GNN training frameworks mitigate this by streaming subgraphs from CPU to GPU, but they incur prohibitive PCIe data transfers and non-coalesced memory accesses, resulting in bandwidth underutilization and low compute efficiency. We propose GARDEN, a GPU-accelerated sampling-based system that co-designs caching, sampling, and batching to eliminate these bottlenecks. GARDEN introduces three key innovations: (1) an Adaptive High-Fidelity Cache Allocator (AHFCA) that combines offline reuse profiling and online hotness tracking to dynamically cache features and adjacency lists within a fixed GPU memory budget, avoiding static allocation waste; (2) a Grouping-based Parallel Sampler that reorders stochastic neighborhood accesses(GPS) to maximize warp occupancy and hide latency on power-law graphs; and (3) a Super-batch Degree Classifier(SDC) that fuses mini-batches into degree-homogeneous groups, enabling cache-friendly coalesced transfers and high GPU utilization. Evaluations on real-world billion-scale graphs demonstrate that GARDEN achieves 4.54×, 1.27×, and 2.97× end-to-end training speedups over DGL-UVA, DUCATI, and FastGL, respectively, while maintaining model convergence and accuracy. Index Terms—Graph Neural Networks, GPU Acceleration, Sampling-based GNN Training, adaptive caching, neighborhood sampling

I. I NTRODUCTION

G

Neural Networks (GNNs) [1] have emerged as a powerful tool for learning representations on graphstructured data, achieving state-of-the-art performance in diverse domains such as social network analysis [2], recommendation systems [3], and molecular property prediction [4]. However, scaling GNN training to real-world graphs with billions of edges (e.g., citation networks like Papers100M (PA) [5], containing over 100 million nodes and 1.6 billion edges) poses a fundamental memory wall challenge. For instance, the Papers100M dataset requires hundreds of gigabytes for feature storage alone, far surpassing the limited capacity of GPU memory [6]–[9]. This makes full-graph training [10] RAPH

Corresponding author: Xin He. Qingrui Zhu, Lei Liu, Yushu Cai, Kai Sheng, and Xin He are with the Guangzhou Institute of Technology, Xidian University, Guangzhou 510530, China (e-mail: zhuqingrui@stu.xidian.edu.cn; liulei2303@stu.xidian.edu.cn; yushucai@stu.xidian.edu.cn; kaisheng@xidian.edu.cn; hexin@xidian.edu.cn). Zhiwen Chen is with the School of Computer Science, Xiangtan University, Xiangtan 411105, China (e-mail: zhiwenchen@xtu.edu.cn).

approaches impractical due to prohibitive memory footprints and CPU-GPU communication bottlenecks. To bridge this gap, sampling-based GNN training has become a predominant paradigm, where subgraphs are sampled from the full graph and processed in mini-batches on GPUs [11], [12]. In practice, sampling-based pipelines typically adopt a CPU-GPU heterogeneous setup: the CPU hosts the entire graph topology and node features, while the GPU handles forward and backward propagation. Each training iteration involves three stages: (1) sampling a subgraph on the host, (2) transferring the sampled topology and features to the GPU via PCIe (the memory I/O phase), and (3) executing GNN computations on the GPU. While this sampling-based approach enables training on massive graphs, it introduces two critical system bottlenecks, particularly for billion-scale graph scenarios. First, frequent large-scale data transfers between host and device incur high I/O costs. Second, the sampling process itself involves irregular memory accesses and complex logic, leading to substantial latency on graphs with power-law degree distributions [13]. These bottlenecks are exacerbated in irregular topologies, where random accesses dominate and constrain overall training throughput. Existing efforts attempt to mitigate these overheads but fail to address them holistically. Early GPU-side caching systems like GNNLab [6] and FastGL [14] rely on single-tier caching (storing only features), neglecting the structural data required for sampling. While recent dual-cache designs like DUCATI [8] co-cache features and topology, they suffer from static allocation inefficiencies and low-fidelity profiling. They allocate GPU memory based on offline heuristics that do not account for the diminishing returns of cache size or dynamic hotness shifts during training. Furthermore, these systems’ sampling kernels use generic thread scheduling that is illsuited to the skewed degree distributions of real-world graphs, causing load imbalance and underutilized memory bandwidth. To overcome these bottlenecks, we propose GARDEN, a GPU-accelerated sampling-based system that co-designs caching, sampling, and batching for efficient billion-scale GNN training. GARDEN introduces three key innovations: (1) an Adaptive High-Fidelity Cache Allocator (AHFCA) that integrates offline reuse profiling with online hotness tracking to dynamically manage GPU memory under fixed budgets, thereby mitigating inefficiencies from static allocation; (2) a Grouping-based Parallel Sampler that reorders stochastic neighborhood accesses to improve warp occupancy and hide memory latency on graphs with skewed degree distributions;

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

and (3) a Super-batch Degree Classifier that fuses mini-batches into degree-homogeneous groups to enable cache-efficient data transfers and maximize GPU utilization. We implement GARDEN as an end-to-end framework and evaluate it on billion-scale real-world datasets including Papers100M and IGB-large. Our evaluation demonstrates that GARDEN effectively overcomes the memory wall and compute irregularity of billion-scale GNNs, enabling practical, high-throughput training of billion-edge GNNs on a single multi-GPU node. In summary, this paper makes the following contributions: We propose the Adaptive High-Fidelity Cache Allocator, a dynamic memory management strategy that integrates end-to-end access statistics with runtime adaptation to maximize cache hits under strict memory budgets. • We design a Grouping-based Parallel Sampler that optimizes thread scheduling for stochastic sampling, significantly reducing warp divergence and increasing computational throughput on skewed graphs. • We develop a Super-batch Degree Classifier that reorganizes training batches to exploit data locality, enhancing both PCIe transfer efficiency and GPU core utilization. • We build GARDEN, an end-to-end framework integrating these techniques. Our extensive evaluation on billionscale real-world datasets show that GARDEN achieves speedups of 4.54× over DGL-UVA [15], 1.27× over DUCATI [8], and 2.97× over FastGL [14], respectively, while maintaining model accuracy.

2

seed nodes

layer0 nodes

layer1 nodes

Message aggregation (Src to Dst) Neighbor sampling (Dst to Src)

output nodes

Layer0： 1:Src nodes

1:Dst nodes

Layer1：

Layer2：

2:Src nodes

input nodes

2:Dst nodes

2:Src nodes

•

II. BACKGROUND A. Graph Neural Networks Graph Neural Networks (GNNs) are deep learning models that operate on graph-structured data, capable of end-to-end learning for tasks including node classification, link prediction, and graph-level regression. Formally, let G = (V, E) be a graph with node set V (|V| nodes) and edge set E ⊆ V × V (|E| edges). Each node v ∈ V is associated with an initial feature vector xv ∈ Rd0 that encodes domain information, where d0 is the dimensionality of the input node features. GNNs employ an iterative message-passing scheme across K layers. At each layer k, a node u aggregates information from its neighborhood N (u) = {v | (v, u) ∈ E} using a differentiable message function ϕ(k) (·): (k)

mu

= AGGREGATE(k)

  o n (k−1) (k−1) ϕ(k) hv , hu | v ∈ N (u) (1)

(k−1)

where hu is the hidden state of u from the previous layer, and AGGREGATE is a permutation-invariant function such as sum, mean, or max. The node then updates its representation by combining its previous state with the aggregated message:   (k) h(k) h(k−1) , m(k) (2) u = U PDATE u u where U PDATE(k) (·) is typically a learnable function such as (K) an MLP or GRU. After K layers, each node’s embedding hu captures both topological information and feature content from its K-hop neighborhood, enabling effective graph reasoning.

Fig. 1. Sampling-based GNN training framework with a per-layer fanout of [2,2]. Each layer-wise block connects source (src) nodes to destination (dst) nodes.

B. Sampling-based GNN Training with Caching Training GNNs on large-scale graphs is challenging because the number of involved neighbors grows rapidly with model depth, making full-neighbor aggregation memory-intensive and impractical for billion-edge graphs. Sampling-based training addresses this problem by selecting a fixed number of neighbors at each layer, thereby bounding the size of the computation graph and enabling memory-efficient mini-batch training. As illustrated in Figure 1, neighbor sampling begins with a set of seed nodes V0 at Layer 0 and recursively expands toward the outer layers according to the specified fanouts. At sampling layer ℓ, the node set Vℓ−1 serves as the Destination (Dst) set, whose representations are the aggregation targets of that layer. The sampled neighbors are then added to form the corresponding Source (Src) set, Vℓ = Vℓ−1 ∪ Nℓ (Vℓ−1 ). Therefore, Src and Dst denote layer-specific roles rather than disjoint node categories, and the Dst set of each layer is contained in its corresponding Src set. The two endpoint sets play different roles in sampling-based training. The seed set V0 is the output-node set, because the model produces final representations and evaluates predictions and losses for these nodes. In contrast, Vk , obtained after k sampling layers, is the input-node set, because the raw features of all nodes in this complete Src set must be fetched before GNN computation. Consequently, neighbor sampling expands from the output nodes at Layer 0 to the input nodes at Layer k, whereas message aggregation proceeds in the reverse direction, from the input nodes through the intermediate layers to the output nodes. From a system’s viewpoint, recursive sampling converts irregular propagation into sparse, localized operations, facilitating asynchronous fetching and dynamic scheduling. However, implementations vary in efficiency. PyTorch Geometric (PyG) [16] relies on CPU sampling, limiting scalability for massive graphs. DGL accelerates sampling on GPUs but incurs overhead from synchronization and host-device transfers. Many frameworks also suffer from low cache hit rates due to graph irregularity, exacerbating bottlenecks. Single-featurecache systems such as PaGraph [17], Data Tiering [18], AliGraph [19], and GNNLab [6] cache node features to reduce

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

data transfers but overlook adjacency structures, constraining performance for large subgraphs. FastGL employs singlecache optimization and reduces synchronization but introduces online overhead and fails to handle random accesses effectively. Dual-cache systems like DUCATI and DCI [20] improve on this by separately caching features and adjacency, better suiting divergent access patterns. Yet, their static allocation and coarse profiling limit adaptability to dynamic workloads, often leading to inefficiencies in billion-scale scenarios. GNNBoost [21] introduces a cost-model-driven cache allocation framework for jointly optimizing topology and feature caching, but it suffers from several limitations. Its additional RNGS cache competes for limited GPU memory and its sampling strategy fails to hide random access latency, leaving significant I/O bottlenecks. Moreover, its reliance on offline models limits adaptability to dynamic workload changes. This highlights the need for more adaptive caching and sampling mechanisms.

3

the system design, whereas a × indicates that it is not included in the stated design scope. GARDEN’s main design distinction is the integration of these capabilities in one end-to-end framework. Full-link offline profiling and Dual-Cache organization preserve stable global reuse, while the bounded Dynamic-Cache compensates for recently recurring residual hot nodes missed by the static cache. GARDEN further couples cache management with sampling and batching through GPS and SDC, connecting cache decisions with execution organization. Among the listed systems, GARDEN is the only framework covering all five design dimensions. Table I summarizes this capability coverage and clarifies the design scope rather than providing a direct performance ranking. TABLE I C APABILITY- ORIENTED COMPARISON OF CACHE ORGANIZATION AND SYSTEM CO - DESIGN .

Host Memory Full Graph Adjacdency

Work

Full Node Features

Cache miss(use UVA)

Cache miss(use UVA)

GPU Memory

Allocate Dual-Cache

Adj-Cache

Nfeat-Cache

(Graph Structure)

(Graph Structure)

Use Adj-Cache

Use Nfeat-Cache ... ... ...

Offline Cache Preprocessing seed nodes

subgraph

Adj-Sampling

subgraph

Training Computation

features

Nfeat-Selecting

Fig. 2. Architecture of the dual-cache GNN training system, coordinating feature and adjacency caches for efficient sampling.

Next, we outline the Dual-Cache System (DCS) pipeline, as shown in Figure 2. DCS comprises three stages. In offline preprocessing, it profiles access frequencies for adjacency lists and features, computes utility as frequency divided by size, and greedily populates Adj-Cache and Nfeat-Cache under GPU memory constraints. During training, Adj-Sampling queries Adj-Cache for lists, falling back to Unified Virtual Addressing (UVA) [22]for host misses in Compressed Sparse Column (CSC) format, enabling low-latency sampling despite irregularity. Nfeat-Selecting follows suit for features, using pinned host memory on misses, with dense storage for bandwidth efficiency. DCS decouples caching for Adj-Sampling (latencybound due to sparse pointer-chasing) and Nfeat-Selecting (bandwidth-bound from contiguous reads), avoiding singlecache trade-offs and improving throughput and utilization. C. Capability-Oriented Comparison Table I summarizes explicit design capabilities rather than measured performance. The comparison covers offline profiling, Dual-Cache organization, recent residual adaptation, sampling co-design, and batching co-design. A ✓ indicates that the corresponding capability is explicitly incorporated into

GNNLab [6] DUCATI [8] FastGL [14] GNNBoost [21] GARDEN

Offline profile

DualCache

Recent residual adapt.

✓ ✓ ✓ ✓ ✓

× ✓ × ✓ ✓

× × × × ✓

Sampling Batching co-design co-design × ✓ ✓ ✓ ✓

× × × × ✓

III. M OTIVATION To understand the fundamental bottlenecks in billion-scale GNN training, we conduct a series of micro-benchmarks on state-of-the-art frameworks (DGL [23], DUCATI [8], and FastGL [14]) using massive datasets such as Papers100M [5], IGB-large [24] and UK-2006-05 [25]. Our analysis reveals two primary systemic inefficiencies suffered by existing approaches. A. The Static Cache Dilemma: Mismatched Allocation and Diminishing Returns GNN training on billion-scale graphs demands efficient data management to handle massive node features and irregular graph structures. Existing caching strategies, however, fall short in adapting to diverse sampling algorithms and dynamic workloads. Single-cache systems, such as GNNLab and FastGL, typically cache only node features, neglecting graph topology like adjacency lists. This leads to high overhead in the sampling phase, where repeated CPU-GPU transfers dominate runtime.These single-cache approaches also underutilize GPU memory, often leaving several gigabytes of VRAM idle due to their static handling of GNN workloads. Dualcaching mechanisms, like DUCATI, co-cache node features and adjacency lists to improve data reuse and reduce I/O. Yet, even these systems struggle with billion-scale graphs, as static caching of node features mismatches on-demand training needs. As shown in Figure 3, the Nfeat-selecting phase emerges as a dominant bottleneck in systems like FastGL and DUCATI across benchmarks such as Papers100M, IGB-large, and UK, consuming a substantial fraction (e.g., on average 50%) of total













 

















 





1



















6DPSOH7LPH

3$

8.

1IHDW6HOHFWLQJ7LPH

TABLE II F EATURE C ACHE H IT R ATE (%) Framework

Hit Rate (%)

IGB-large

GNNLab DUCATI FastGL PaGraph AliGraph

49.42 61.34 16.96 12.49 13.98

Papers100M

GNNLab DUCATI FastGL PaGraph AliGraph

83.35 94.94 73.94 64.92 70.34

UK

GNNLab DUCATI FastGL

41.06 72.77 8.72

0.16

0.17

0.15

5

6

7

0.53 0.05

−1 −2

-2.50 1

2

3

4

Expansion Step

8

9

10

11

Fig. 4. Marginal diminishing-return ratio ri of DUCATI on the UK dataset over 11 cache expansion steps. Step i corresponds to the transition from (i+3) to (i + 4) GB; the 3 → 4 GB transition provides the reference marginal gain.

7UDLQ7LPH

Fig. 3. Execution time breakdown for GraphSAGE training on billion-scale graph dataset using NVIDIA RTX 3090 GPU.

Dataset

0.71

0.40



'*/ '8&$7, IDVW*/ '*/ '8&$7, IDVW*/ '*/ '8&$7, IDVW*/

,*%

0.46

0.46

0

−3



1.96

2

runtime. This overhead arises from a fundamental mismatch between offline-preprocessed caches and runtime feature demands: offline frequency-based profiling fails to capture the stochastic nature of multi-hop sampling, causing ’hot’ features to be evicted or missed entirely during training, resulting in cache hit rates that are predominantly below 90% across all evaluated billion-scale workloads, as detailed in Table II. Enhancing cache allocation during preprocessing is essential to improve hit rates and utilization. GPU memory allocation further exacerbates inefficiencies. While software-managed feature caches reduce I/O latency, empirical analysis shows diminishing returns with increasing cache size. To quantify this, we define the marginal decay ratio as: ∆i ri = 1 − , i = 1, 2, . . . , 11 (3) ∆i−1 Here, ∆0 denotes the reference latency reduction from 3 to 4 GB, and ∆i denotes the latency reduction from (i + 3) to (i + 4) GB for i = 1, . . . , 11. Thus, Steps 1–11 correspond exactly to the 4→5 GB through 14→15 GB expansions shown in Figure 4 and listed in Table IV. A negative ri means that the current expansion stage provides a larger marginal

benefit than the preceding stage. The 11-step DUCATI measurement exhibits non-monotonic marginal returns, including a pronounced fluctuation at Step 4. More importantly, all ratios from Steps 6–11 are positive, indicating that after Step 5 each further cache expansion provides a smaller marginal latency reduction than the preceding expansion. The large positive ratio of 1.96 at Step 11 further shows that DUCATI remains constrained by diminishing marginal returns as cache capacity increases. Implication. This insight reveals that static memory allocation to feature caches is suboptimal under fixed GPU budgets. Dynamic-Cache reallocation toward regions with the highest marginal gains could maximize training throughput. In essence, billion-scale GNN training requires optimizations that address static precaching limitations and enable adaptive memory use. B. The Sampling Bottleneck: Uncoalesced Access and Bandwidth Underutilization *OREDO/RDG6HFWRU(IILFLHQF\

GJO



JVDPSOHU

GXFDWL

  





 %DWFKVL]H



6DPSOH.HUQHO0HPRU\7KRXJKWSXW 7KRXJKWSXW*E\WHVV



3URSRUWLRQ

(QGWRHQGWUDLQLQJWLPHSURSRUWLRQ



4

Marginal Diminishing Return Ratio

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

GJO



JVDPSOHU

GXFDWL

 %DWFKVL]H



   





Fig. 5. Fine-grained performance metrics measured using NVIDIA Nsight Compute on the Papers100M dataset.

Sampling in large-scale GNN training introduces severe bottlenecks due to the irregular topology of billion-scale graphs. Irregular access patterns hinder GPU parallelism, while node ID remapping and inter-thread dependencies impose synchronization overhead, reducing throughput. Profiling sampling kernels in frameworks like DGL, GSampler, DUCATI, and FastGL using Nsight Compute [26] reveals persistent inefficiencies. A key metric, Global Load Sector (GLS) Efficiency, measures bandwidth utilization by the ratio of valid data bytes to total sector size (e.g., 128 bytes on Ampere architectures). Values near 100% indicate coalesced accesses, while low values (e.g., below 25%) signal wasted bandwidth from uncoalesced patterns.

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

5

As illustrated in Figure 5, even DUCATI, which employs adjacency list caching, shows poor GLS efficiency despite higher memory throughput, indicating that most transferred data remains unused. All frameworks achieve throughput far below hardware limits (e.g., 1008 GB/s on RTX 3090 Ti), confirming sampling as a critical system bottleneck. We identify three root causes for these inefficiencies. First, the inherent randomness of graph sampling generates uncoalesced memory accesses that bypass GPU cache prefetching mechanisms. Second, existing caching strategies optimize data transfer but fail to address fundamental locality issues. Third, thread divergence and load imbalance in highly parallel GPU architectures exacerbate these problems, particularly for power-law graphs where degree distributions vary by orders of magnitude. Implication. These bottlenecks collectively constrain GNN training efficiency, necessitating a co-designed approach that optimizes caching, sampling, and batching simultaneously. Our proposed GARDEN framework addresses these limitations through three synergistic innovations: an Adaptive HighFidelity Cache Allocator for dynamic memory management, a Grouping-based Parallel Sampler for optimized GPU utilization, and a Super-batch Degree Classifier for efficient batch processing.

CPU-GPU data transfers. A cache-hit detector, based on recoded ID ranges, enables cache-aware sampling kernels to differentiate cached data from that requiring host fetches. On the device side, each GPU incorporates a Super-batch Degree Classifier and a Grouping-based Parallel Sampler. GARDEN introduces super-batching, which coalesces multiple mini-batches into a single contiguous super-batch to amortize setup overheads. The Super-batch Classifier performs node deduplication and classification via hash-based mapping and parallel prefix sums, avoiding costly atomics. It categorizes nodes into full nodes (requiring complete neighbor sampling) and random nodes (using probabilistic sampling), generating metadata pointers for downstream processing. The Grouping Sampler then invokes two specialized kernels: for random nodes, a Grouping-based Parallel Sampler that organizes warp threads and preloads adjacency lists into shared memory, transforming irregular global accesses into coalesced on-chip operations; for full nodes, a lightweight full-sampling kernel. These kernels run concurrently using GPU streams to maximize compute and memory utilization. Overall, This design ensures end-to-end training efficiency by aligning cache management with sampling parallelism, addressing key bottlenecks in largescale GNN training.

GPU 2 GPU 1

Host Memory Adaptive ① High-Fidelity Allocator

B. Adaptive High-Fidelity Cache Allocator

GPU 0

CPU

②

Super batch Degree Classifier

...

batch 2

③Grouping

batch 1

Random Nodes

based Parallel Sampler

batch 0

Full Nodes

TABLE III F EATURE - CACHE HIT RATE OF DUCATI ON UK UNDER DIFFERENT NUMBERS OF OFFLINE PRE - SAMPLING EPOCHS .

subgraph

Adj. Cache

Nfeat. Cache

Data Traffic with Host Memory

Dynamic Cache

Existing static-cache designs rely on offline pre-sampling to estimate long-term access frequencies. This temporal scope is useful for stable global reuse, but extending the preprocessing period does not guarantee continuously increasing cache coverage. The following sensitivity result illustrates this limitation before we introduce AHFCA’s adaptive cache organization.

④

Trainer

Data Traffic on/between GPU Memory

Fig. 6. Overall architecture of GARDEN

IV. D ESIGN A. Overview We present GARDEN, an efficient sampling-based system for billion-scale GNN training. The system holistically cooptimizes caching, sampling, and batching to maximize training throughput on multi-GPU platforms. Figure 6 illustrates GARDEN’s architecture. On the host side, the Multiple Dual Cache Allocator builds a dual-cache structure in each GPU’s global memory, consisting of an adjacency cache (Adj-Cache) and a feature cache (Feat-Cache). Using offline pre-sampling statistics on node access frequencies, the allocator reassigns compact IDs to high-frequency nodes and populates the caches with their adjacency lists and feature vectors. This minimizes

Pre-epoch

2

3

4

5

6

Nfeat cache hit rate (%) 71.73 75.63 76.88 75.31 72.62

On UK, the feature-cache hit rate increases from 71.73% at two pre-epochs to 76.88% at four pre-epochs, but decreases to 72.62% at six pre-epochs. An excessively long offline preprocessing phase can accumulate transient access variations in the global profile, blur the persistent hot set, and reduce the accuracy of distinguishing long-term hot nodes from shortlived accesses. This non-monotonic trend shows that a longer static warm-up cannot by itself guarantee sustained hit-rate improvement. It motivates a bounded Dynamic-Cache that complements, rather than replaces, the static cache by targeting recently recurring residual hot nodes. Efficient caching strategies are critical for accelerating GNN training on large-scale graphs, particularly in the context of mini-batch sampling. Traditional methods often suffer from two key limitations: inaccurate offline prediction and diminishing returns from online cache maintenance. To overcome these challenges, we propose the Adaptive High-Fidelity Cache Allocator (AHFCA), a dual-phase (offline + online) caching

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

Host Adaptive High-Fidelity Cache Allocator Offline Allocate Dual-Cache Offline Cache Preprocessing with full-link access statistics

6

Normal nodes

GPU2 Memory GPU1 Memory GPU0 Memory

Adj-counts nodes

Nfeat-counts nodes

Seed nodes

Layer0：

Adj-Cache

Nfeat-Cache

(Graph Structure)

(Graph Structure)

Layer1：

Dynamic-Cache

Layer2：

1:Src nodes

1:Dst nodes

Online cache adjustment Adaptive Dynamic Feature Caching

node counts

2:Src nodes

(Graph Structure)

subgraph

2:Dst nodes

(a)

Fig. 7. The basic structure of Adaptive Dynamic Feature Caching

Normal nodes

Adj-counts nodes

Nfeat-counts nodes with Layer Weight

Seed nodes

Layer0：

strategy designed to enhance cache efficiency and mitigate data I/O bottlenecks during GNN training on massive graphs. As illustrated in Figure 7, AHFCA significantly improves the initial cache hit rate by leveraging detailed, full-link access statistics during the offline pre-sampling phase. During this phase, AHFCA allocates three distinct types of caches: adjacency cache, feature cache, and Dynamic-Cache, across all GPUs, ensuring synchronized cache availability. This strategy minimizes redundant data transfers between the host and GPU by enabling cache-aware sampling, which determines whether data is already present in the cache or needs to be fetched from main memory. Once training commences, AHFCA introduces the Adaptive Dynamic Feature Caching (ADFC) mechanism, which continuously monitors node access patterns to update the Dynamic-Cache. Using a marginal-gain model, ADFCA terminates cache updates when further adjustments yield negligible performance improvements, effectively balancing caching overhead with training efficiency. In AHFCA, the Dynamic-Cache adapts to variation in the training access sequence rather than changes in graph topology or graph structure, and it does not alter the graph sampling procedure. The static cache captures stable long-term global hotness from the offline profile, whereas the Dynamic-Cache is a bounded secondary compensation cache that supplements static-cache misses with recently recurring residual hot nodes. Because enlarging the static cache or extending its warmup cannot avoid diminishing marginal returns, the DynamicCache uses limited remaining memory to target recent accesses with higher short-term utility. Existing offline cache preprocessing techniques often fail to accurately capture real-world GNN access patterns, leading to suboptimal cache allocation and reduced training efficiency. For example, as shown in Figure 8a, traditional approaches are limited by two significant design flaws. First, they record adjacency accesses only for destination (dst) nodes, neglecting the role of source (src) nodes, which are frequently accessed during neighbor sampling and layer-wise aggregation. Second, these methods only account for input-layer nodes in feature accesses, ignoring intermediate nodes that serve as both src and dst across multiple computational blocks, thereby underestimating inter-layer reuse. To address these issues, we propose a more comprehensive approach that captures full-link access patterns, including all src, dst, and seed nodes across every layer of the GNN. By adopting a layer-aware weighting strategy, we account for the reuse intensity of nodes based on their sampling depth. The

1:Src nodes

1:Dst nodes

Layer1：

Layer2：

2:Src nodes

2:Dst nodes

(b)

Fig. 8. An example to show Node statistics during offline preprocessing: (a) conventional access frequency calculation; (b) our full-link access statistics method for offline cache preprocessing.

refined feature access model, denoted as: Wf eat (v) = 1(v ∈ Vin ) +

L X

ωl · 1(v ∈ Vsrc ) + 1(v ∈ Vdst ) (l)

(4)

l=1

allows us to accurately track the contributions of each node to the overall access frequency, ensuring more reliable cache prioritization and improved hit rates. The static cache is constructed from full-link offline access statistics collected during the offline profiling period. This profiling period is intended to estimate stable long-term global hotness across the training workload, rather than short-term fluctuations. Because the static cache optimizes global frequency, simply increasing its capacity or extending the profiling period cannot indefinitely improve the hit rate; once the most reusable global items have been selected, the marginal benefit of additional entries decreases. AHFCA therefore uses the Dynamic-Cache as a bounded secondary compensation cache, not as an independent replacement for the static cache. It targets only residual feature accesses that are missed by the static cache and recur within recent access windows. Although high-fidelity offline caching improves initial performance, the long-term profile does not fully describe every recent training access sequence. To address this limited residual locality, AHFCA incorporates the ADFC mechanism. ADFC uses an Exponentially Weighted Moving Average (EWMA) to emphasize recent access frequency. The DynamicCache Cdyn is updated periodically based on these hotness scores, as represented by: X Score(v, t) = γScore(v, t − 1) + I(i = v) (5) i∈Bcachehit,t

Here, Score(v, t − 1) is the historical hotness score of node v, γ ∈ (0, 1) is the decay coefficient, and Bcachehit,t denotes the

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

TABLE IV S TEP - WISE MARGINAL DIMINISHING - RETURN RATIOS ri ON THE UK DATASET. T HE TWO PANELS PARTITION THE 11 EXPANSION STEPS FOR A COMPACT SINGLE - COLUMN PRESENTATION . Steps 1–6

Steps 7–11

Step DUCATI GARDEN

Step DUCATI GARDEN

1 2 3 4 5 6

7 8 9 10 11

0.46 0.46 0.40 -2.50 0.16 0.17

-0.15 0.56 -0.73 -0.01 0.44 0.23

0.15 0.71 0.05 0.53 1.96

-0.30 0.69 -2.21 0.66 0.03

cache-hit nodes in the current batch. The summation counts the occurrences of node v among these cache-hit nodes. Thus, the EWMA retains a decayed history of previous hotness while assigning greater weight to recent access frequency. At each update interval Tupdate , the system selects the top Kdyn nodes with the highest hotness scores to update the Dynamic-Cache using a Top-K operation. This bound prevents the compensation cache from expanding into a second full static cache and limits the memory and maintenance overhead of online adaptation:
Cdyn,t+1 = TopK {Score(v, t)}v∈V , Kdyn (6) This bounded update policy addresses the overhead of online cache maintenance. In the measured configuration, DynamicCache updates are performed for a fixed schedule of 10 repeated epochs and then stop, while the cache-processing work overlaps with GPU training. Thus, AHFCA balances adaptation to recent residual hotness with bounded maintenance cost: it changes feature residency without altering the graph sampling procedure or the training objective. Larger Dynamic-Cache capacities may provide only limited additional hit-rate improvements while increasing the maintenance cost of EWMA score updates and Top-K selection. Accordingly, the Dynamic-Cache capacity is exposed as a tunable parameter and remains bounded by the GPU memory available after reserving space for the model and static caches. The runtime, memory, and accuracy effects of this bounded Dynamic-Cache maintenance policy are evaluated in Table VI. The following step-wise comparison reports the marginal diminishing-return ratio for DUCATI and GARDEN. A negative ri means that the current expansion provides a larger marginal improvement than the preceding expansion. Although local fluctuations remain, GARDEN exhibits negative ratios at multiple expansion steps and does so more frequently than DUCATI. This pattern shows that GARDEN’s cache strategy effectively mitigates marginal diminishing returns by recovering useful cache benefits across multiple expansion stages. C. Grouping-based Parallel Sampler In billion-scale GNN training, the sampling phase often faces critical system-level bottlenecks. These arise from the mismatch between the irregular topology of real-world

7

graphs and the architectural characteristics of GPU hardware. Specifically, the adjacency matrix representation of largescale graphs consumes substantial memory resources. During sampling, random accesses to adjacency lists exhibit high stride variability, which reduces spatial locality, particularly when high-degree nodes are involved. These irregular memory access patterns hinder the coalescing of memory transactions, resulting in suboptimal memory bandwidth utilization and decreased sampling kernel efficiency. Moreover, thread load imbalance and suboptimal hardware utilization further impede performance. In mini-batch training, heterogeneity in node degrees introduces significant variance in workload distribution across computing units. This imbalance, exacerbated by the irregular branching structure of graphs and conservative fanout hyperparameters, impairs the ability of sampling kernels to maintain high thread occupancy within GPU streaming multiprocessor (SM) warps. Consequently, warp divergence occurs, leading to idle threads and reduced parallel execution efficiency. Thus, the primary challenges are: (1) high computational overhead due to irregular memory accesses during sampling, (2) dynamic load imbalance, and (3) the mismatch between sampling strategies and GPU hardware architectures. To address these challenges, we propose a fine-grained node classification and sampling optimization framework built upon DUCATI’s offline adjacency cache preprocessing. DUCATI’s offline graph reordering ranks nodes based on historical access frequency, assigning smaller identifiers to high-frequency nodes. This reordering establishes a solid foundation for load balancing and optimization within the sampling kernels. During the preprocessing stage, the system determines the optimal adjacency list cache capacity, defines cache boundaries, and selectively caches adjacency information for nodes with identifiers below a designated threshold. This comprehensive preprocessing improves memory locality, workload distribution, and resource utilization during the sampling phase. Building on this foundation, we implement a fine-grained classification mechanism for seed nodes based on their dynamic characteristics and structural properties. Specifically, we use the sampling hyperparameter, fanout, to categorize nodes into two types: random nodes (with degrees greater than the fanout threshold) and full nodes (with degrees less than or equal to the fanout threshold). For random nodes, the sampling kernel generates random offsets to select neighbors, while for full nodes, all neighbors are directly sampled in one operation. As illustrated in Figure 9a-b, most sampling systems allocate GPU resources at the warp or block level, dedicating computational units to either random or full nodes. However, this approach often leads to low thread utilization, especially when processing low-degree nodes or using small fanout values. With many threads in a warp remaining idle, hardware resources are underutilized. Additionally, for random nodes, the irregular memory accesses hinder memory transaction coalescing, reducing data throughput and wasting bandwidth. To optimize thread load balancing and execution efficiency, we introduce grouping and scheduling optimizations for these two types of nodes. During sampling preprocessing, we rely on precomputed node degrees to facilitate efficient parallel sam-

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

8

the activate threads the idle threads

A warp

A warp

...

...

...

...

... Randomly sampled nodes' neighboring nodes

Fully sampled nodes' neighboring nodes

(a)

(b)

the grouping activate threads

Fig. 9. Sampling processes for (a)A warp random nodes and (b) full nodes in the idle threads conventional sampling systems.

pling computations. The complete sampler method is shown in ... ... Figure 10, for random nodes, the Grouping Parallel Sampler partitions 32 threads within a warp into multiple groups, where the width g of each group corresponds to the vertex group 0fanout. share memoryThis mapping group 1 share memory groupwarp 4 share memory sampling allows a...single to process ⌊32/g⌋ vertices in parallel. As threads within each group ... ... ... ... execute identical logic (selecting target nodes and generating All the nodes to be sampled within a warp random offsets), warp divergence is eliminated, resulting in (c) synchronized execution and higher thread occupancy. the grouping activate threads

A warp

...

the idle threads

...

...

A warp

...

...

...

share memory share memory ... share memory

group 0

...

group 1

...

group 4

...

group 0

...

group 4

group 1

...

random nodes

full nodes

Adj-Cache

Fig. 10. Grouping Parallel Sampler operation: with fanout=5, a single warp simultaneously processes six nodes.

In contrast, full nodes do not require random selection. Instead, they read the complete neighbor list of each node, which results in contiguous memory access, eliminating the need for shared memory allocation. The grouping method for full nodes is similar to that of random nodes, ensuring efficient memory utilization. To efficiently manage both types of sampling, we schedule the random and full sampling kernels to execute concurrently through stream processing, maximizing hardware resource utilization and improving system throughput. For memory access optimization, the system allocates contiguous, exclusive buffers in shared memory for nodes with degrees below a preset threshold Dmax , with a buffer size set to Dmax . For example, setting Dmax to 64 for the Papers100M dataset covers about 97% of nodes. This configuration ensures that the majority of nodes benefit from optimized random access while managing shared memory resources effectively. To facilitate efficient population of these buffers, the system employs a Warp-based Collaborative Coalesced Loading strategy. Upon kernel launch, the warp serves as the execution unit for neighbor retrieval, with threads collaboratively

fetching adjacency lists into shared memory via vectorized instructions and strided access patterns. This transforms fragmented random accesses into efficient, coalesced transactions, thereby maximizing memory bandwidth utilization. Once the adjacency data is staged in shared memory, subsequent random sampling operations by warp threads proceed with registerlevel latency. For nodes with degrees exceeding the Dmax threshold, the framework employs a direct-access fallback strategy. Instead of using shared memory, threads perform random indexed lookups directly in global memory to retrieve the required neighbor information. Although this incurs higher latency, the relatively low frequency of high-degree nodes preserves overall bandwidth and efficiency. By integrating the Adj-Cache and Unified Virtual Addressing (UVA) [22], the system ensures a robust memory access strategy throughout the sampling process. Initially, adjacency data is fetched from the Adj-Cache. If a cache miss occurs, the system seamlessly uses UVA to retrieve the data from host memory. This unified caching and retrieval framework ensures high data throughput and system robustness, even when working with billion-scale graphs. As shown in Figure 10, thread groups use a coalesced load mechanism to transfer adjacency lists from global memory to shared memory in one operation. The shared memory structure, with its 32way bank parallelism, allows low-latency access, ensuring that subsequent neighbor accesses are efficiently executed within shared memory. Our grouping-based parallel sampler further improves stochastic neighbor access by reorganizing threadgroup execution to increase warp occupancy and reduce branch divergence under the skewed degree distributions common in power-law graphs. Importantly, it preserves the original fixedfanout uniform sampling protocol used in standard frameworks such as GraphSAGE [11]: we do not alter sampling probabilities, apply pruning or reweighting, or otherwise change the sampling distribution. As a result, these scheduling optimizations preserve the unbiasedness of the stochastic gradient estimator and maintain training fidelity, as also demonstrated in our empirical results (Section VI-E). This memory optimization strategy operates in conjunction with the overall caching system. First, high-frequency adjacency lists are preloaded into GPU caches to reduce CPU-GPU data migration overhead. Second, shared memory buffering is activated for nodes meeting the degree criteria, improving memory locality during sampling. This two-tiered caching design, integrating global and shared memory caches, significantly reduces memory access latency and enhances bandwidth utilization during GNN training. D. Super-batch Degree Classifier In grouping-based parallel sampling, pre-sampling node classification is essential for optimizing memory access and load balance. However, existing approaches typically perform classification independently for each mini-batch, incurring substantial redundant computation and synchronization overhead. In GPU environments, accurate node categorization often relies on atomic operations to ensure correctness, which leads to severe thread contention and increased latency. More-

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

over, graph learning workloads commonly employ small minibatches, whose limited workload fails to fully occupy GPU resources, resulting in low SM utilization. Prior work (e.g., gSampler [27]) has shown that increasing batch size improves GPU throughput. However, under the conventional mini-batch paradigm, this also amplifies the cost of repeatedly executing classification and deduplication logic. The sampling nodes at this layer

Random nodes after classification Fully nodes after classification

batch 0

+ After classification

batch 1

+ +

batch 2

...

...

Fig. 11. Super-batch classification: nodes are categorized as random or full nodes with per-batch segmentation before sampling.

Super-batch Construction and Deduplication. To address these limitations, we propose a Super-batch Degree Classifier that jointly optimizes batching, deduplication, and node classification. The key idea is to aggregate multiple minibatches into a single super-batch with contiguous storage, enabling batch-wide preprocessing and classification in one pass. A batch pointer, denoted as BatchPtr, encodes sub-batch boundaries, preserving logical isolation between mini-batches while allowing unified processing. During super-batch construction, node sets at each sampling layer are incrementally merged with previously sampled nodes. As illustrated in Figure 11, this process eliminates redundancy both within and across mini-batches. We maintain a GPU-resident hash table that maps global node identifiers to unique entries, ensuring that each node appears exactly once in the super-batch view. This design avoids repeated node traversals and enables efficient deduplication via parallel hash lookups. Degree-based Classification without Atomics. On top of the deduplicated super-batch, we introduce a lightweight degree-based classification scheme. Nodes are categorized into two types: Full Nodes, whose degrees do not exceed the fanout and are sampled exhaustively; and Random Nodes, which require stochastic neighbor selection. The classification result is encoded as a logical partition boundary, BoundPtr, which separates full nodes and random nodes within each subbatch. Rather than relying on atomic counters, the classifier transforms category assignment into deterministic index computation. As shown in Algorithm 1, the HashmapsForUniqueClassify kernel simultaneously performs node deduplication and category marking (lines 2–8). Each node is assigned a binary flag stored in the DegreeFlag tensor, where 1 denotes a full node and 0 denotes a random node (line 5). Subsequently, the BatchPartition kernel applies a parallel prefix-sum over DegreeFlag using CUB’s ExclusiveSum [28] (line 10), producing the DegreeScan tensor. This scan yields, for each node, its offset within

9

Algorithm 1 The Super-batch Degree Classifier Algorithm Input: MergeTensor[0 . . . N − 1], BatchPtr[0 . . . K], indptr, fanout Output: UniqueClassify, BoundPtr, DegreeFlag 1: function H ASHMAPS F ORU NIQUE C LASSIFY(MergeTensor, fanout) 2: UniqueTensor ← U NIQUE H ASHMAPS(MergeTensor) 3: for each thread tid = 0 . . . N − 1 in parallel do 4: v ← UniqueTensor[tid]; deg ← indptr[v + 1] − indptr[v] 5: DegreeFlag[tid] ← (deg ≤ fanout)?1 : 0 6: end for 7: return DegreeFlag 8: end function 9: function BATCH PARTITION(UniqueTensor, DegreeFlag, BatchPtr) 10: DegreeScan ← EXCLUSIVE S UM(DegreeFlag) 11: for i = 0 . . . K − 1 do ▷ Boundary per batch 12: BoundPtr[i] ← DegreeScan[BatchPtr[i + 1]] − DegreeScan[BatchPtr[i]] 13: end for 14: for each thread tid = 0 . . . N − 1 in parallel do 15: Find i such that BatchPtr[i] ≤ tid < BatchPtr[i + 1] 16: batchStart ← BatchPtr[i]; batchID ← tid − batchStart 17: pos ← DegreeScan[tid] − DegreeScan[batchStart] 18: if DegreeFlag[tid] == 1 then ▷ Full node 19: outIdx ← batchStart + pos 20: else ▷ Random node 21: Rank ← batchID − pos; outIdx ← BoundPtr[i] + Rank 22: end if 23: UniqueClassify[outIdx] ← UniqueTensor[tid] 24: end for 25: return UniqueClassify, BoundPtr 26: end function

the full-node partition of its corresponding sub-batch. The number of full nodes per sub-batch is then computed using a lightweight difference operation over scan results, yielding BoundPtr (line 12). Importantly, this approach converts classification into spatial placement: each thread independently computes its write position in the output tensor (lines 15–22), completely eliminating atomic operations and synchronization stalls. Cache-friendly Layout and Sampling Readiness. Using the computed offsets, full nodes are placed contiguously at the beginning of each sub-batch, followed by random nodes. For random nodes, their position is derived as Rank = batchID pos, where pos is the number of preceding full nodes obtained from the prefix-sum. This layout guarantees contiguous storage for both node types within each sub-batch, significantly improving memory locality for downstream sampling kernels. Crucially, this design collapses preprocessing overhead from being proportional to the number of mini-batches to a single batch-wide operation. Once UniqueClassify and BoundPtr are generated, the sampler can directly locate target nodes through simple pointer arithmetic, enabling highly efficient differentiated sampling. Overall, the Super-batch Degree Classifier achieves efficiency gains through a three-level optimization: (1) Algorithmically, it fuses deduplication, classification, and layout into a single super-batch pass, eliminating redundant work; (2) At the parallel execution level, it replaces atomic operations with prefix-sum–based index computation, removing thread contention; (3) At the hardware utilization level, it increases effective batch size, improving SM occupancy and fully exploiting GPU parallelism. Together, these optimizations enable scalable, cache-friendly, and synchronization-free sampling for billion-scale GNN training.

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

3.99× 3.50×

UU

PA

1.13× 1.00× 3.40× 3.06× 1.85× 1.55× 1.00× 2.55× 2.22× OOM 1.16× 1.00×

UK

2.45×

4.75× 3.99× 3.03×

GCN

IGB

DUCATI

UK

UU

GNNLab

ours

GAT

UK

2.74×

IGB

OOM OOM 1.00×

PA

UU

3.55× 3.09× 2.24× 1.00×

2.11×

UK

1.00×

FastGL

3.03×

gSampler

DGL-UVA

GCN

OOM OOM 1.00×

IGB

4.16× 3.78× 2.51× 1.00×

PA

3.76×

UU

UK

GraphSAGE

GAT

3.40× 3.08×

Multiple GPUs 6.49× 4.76× 3.15× 1.00×

IGB

IGB

6.35× 5.77× 3.78× 1.00×

PA

PA

1.30× 1.00× 4.05× 3.32× 2.30× 1.24× 1.00× 3.53× 2.99× OOM 1.15× 1.00× 5.79× 5.17× 2.84× 1.26× 1.00×

UU

OOM OOM 1.00×

1.00×

8.33× 7.06× 4.55× 1.00×

0

4.88× 4.19× 3.04×

20 10

UK

GraphSAGE

1.00×

IGB

4.48× 3.93× 2.94×

PA

1.51× 1.00× 4.88× 3.75× 3.49× 1.35× 1.00× 4.10× 3.28× OOM 1.29× 1.00× 6.32× 5.27× 3.23× 1.13× 1.00×

5.87× 4.66× 3.61×

8.05× 6.29× 4.00× 1.23× 1.00×

Single GPU

4.75× 4.20× 2.92× 1.00×

Runtime per Epoch (s)

Runtime per Epoch (s)

40 30 20 10 0

10

UU

Fig. 12. Average runtime per epoch of three GNN models across multiple datasets under single- and multi-GPU configurations. Numbers above the bars are baseline-to-GARDEN runtime ratios, and OOM denotes a GPU-memory capacity failure under the reported configuration.

V. I MPLEMENTATION We implement GARDEN atop PyTorch [29], leveraging DGL v1.0.0 for efficient graph loading and subgraph sampling via its data loader module. For the Adaptive HighFidelity Cache Allocator, we introduce two Python classes: Constructor for building the dual-cache structures (AdjCache, Nfeat-Cache, and dynamic cache) and Allocator for runtime allocation and updates. When GPU memory is underutilized, GARDEN repurposes idle VRAM as an additional cache layer to further reduce I/O latency. The Grouping Parallel Sampler is realized through custom CUDA kernels that enforce memory-aware access patterns, including warpbased collaborative loading and shared memory buffering. Following gSampler’s matrix-style API, we encapsulate these kernels into modular interfaces, enabling easy integration with diverse GNN models. For super-batching, we extend gSampler’s CUDA implementation with the Super-batch Degree Classifier, optimizing node deduplication and partitioning before seamless integration into DGL’s sampling pipeline. Multi-GPU training on a single machine uses NCCL [30] for gradient synchronization, supporting data-parallel execution. GARDEN’s modular design allows individual components to be incorporated into other frameworks with minimal code changes, promoting reusability. VI. E VALUATION A. Experiment Setup Benchmarks. We evaluate GARDEN using three representative GNN models: GraphSAGE [11], Graph Convolutional Network (GCN) [31], and Graph Attention Network (GAT) [32]. All models employ 3-hop random neighbor sampling with fanouts of [5, 10, 15] for the first three layers, matching prior configurations. The batch size is 8000, consistent with GNNLab baselines. GraphSAGE and GCN use a hidden dimension of 128; GAT employs 8 attention heads, each with dimension 8.

Datasets. We test on four real-world graphs of varying scales, as summarized in Table V. These include Papers100M (PA, a citation network) from the Open Graph Benchmark, IGB-large (a large-scale homogeneous academic citation graph from the Illinois Graph Benchmark), and two billion-scale web graphs, UK-2006-05 (UK) and UK-Union (UU, a union of UK web snapshots). For UK and UU, which lack official node features, we use randomly generated features with length 128 for runtime and scalability evaluation; these datasets are therefore excluded from the accuracy comparison. TABLE V T HE STATISTICS OF DATASETS USED IN THIS WORK (M: M ILLION , B: B ILLION ). Dataset

Nodes

Edges

Features

Classes

IGB-large [24] Papers100M [5] UK-2006-05 [25] UK-Union [25]

100M 111M 77.7M 134M

1.2B 1.6B 3.0B 5.5B

128 128 128 128

19 172 100 100

Baselines. We compare GARDEN against state-of-theart systems: DGL-UVA [15], gSampler [27], GNNLab [6], DUCATI [8], and FastGL [14]. DGL-UVA conducts sampling using the GPU by utilizing the UVA technique. GNNLab uses efficient caching and multi-GPU parallelism for sampling and training. GSampler provides a general GPU-based sampling framework. DUCATI extends GNNLab with dual caching to reduce I/O. FastGL combines match-reorder and memoryaware techniques for large-scale sampling. PaGraph [17] and AliGraph [19] use CPU-based sampling and graph-processing paths in their original implementations. Because these paths differ from our GPU-accelerated sampling and caching pipeline, we select systems with comparable GPU-accelerated execution paths that can be evaluated under matched datasets, sampling configurations, hardware, and timing protocols as direct baselines.

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

11

As illustrated in Figure 12, we report average end-to-end runtime per epoch for GARDEN and the baselines across the evaluated datasets and GNN models. The experiments include both single-GPU and multi-GPU configurations. The bar heights show absolute runtime, while the labels above the bars show the corresponding baseline-to-GARDEN runtime ratios. An OOM entry indicates that the workload exceeds the available GPU memory under the reported configuration. The absolute runtimes and per-case ratios show the combined effect of GARDEN’s cache, sampling, and batching optimizations. Across the evaluated workloads, GARDEN achieves average end-to-end training speedups of 4.54×, 3.95×, 4.04×, 1.27×, and 2.97× over DGL-UVA, gSampler, GNNLab, DUCATI, and FastGL, respectively. The largest gap is observed against DGL-UVA, whose stochastic sampling generates scattered feature requests and reduces effective memory-bandwidth utilization. GARDEN’s warp-centric, coalesced data loader restores more efficient memory access and achieves up to a 4.5× speedup in training throughput. Compared with DUCATI, GARDEN provides an additional 1.13–1.55× improvement and reduces sampling latency by 27–32%. These gains result from coordinating the Groupingbased Parallel Sampler with the Super-batch Degree Classifier: the former transforms irregular accesses into coalesced sharedmemory operations, while the latter reduces redundant vertex expansion and amortizes synchronization across a super-batch. Together with the cache design, these optimizations balance data movement, sampling, and batch-level execution. C. Performance Breakdown To precisely attribute the performance gains and validate the effectiveness of our three proposed techniques, we conduct detailed component-level analyses comparing GARDEN against state-of-the-art baselines across multiple datasets and configurations. Effectiveness of the Adaptive High-Fidelity Cache Allocator. Built on the single-machine multi-GPU dual-cache runtime, where preprocessing is performed on one GPU and broadcast to others, AHFCA combines multi-dimensional profiling with precise pre-sampling statistics to enhance cache

î

î





î

,*%

3$

8.

,*%

(a)



3$

î

î

î

î



î

î





8.

1RUPDOL]HG6DPSOH7LPH6SHHGXS



 



RXUV







)HDWXUH&DFKH+LW5DWH







(b)

Fig. 13. Cache performance comparison: (a) feature cache hit rate, (b) node feature selection time per iteration. ADFC

AHFCA

TABLE VI DYNAMIC -C ACHE OVERHEAD ON PA.

50

PA

Metric

65.54

61.08

75

78.32

72.77

80.41

85.45

99.58

98.76

100

94.82

Naive

Feature Cache Hit Rate (%)

B. Overall Performance



IDVW*/



'8&$7,

Notably, several baselines such as FastGL require the entire graph adjacency information to be stored in GPU memory, which is infeasible for certain billion-scale datasets. During the overall training process, we follow the convention adopted by DUCATI and FastGL and exclude the offline preprocessing time from the end-to-end training time. As demonstrated in prior work [33], in practical training scenarios, models typically require on the order of thousands of epochs; consequently, the preprocessing time is negligible compared to the total training time. Environment. Experiments run on a server with two AMD EPYC 7702 CPUs (80 cores total), 512 GB DRAM, and six NVIDIA RTX 3090 GPUs (24 GB VRAM each). The software environment includes Python v3.8.10, PyTorch v1.12.1, CUDA v11.6, and DGL v1.0.0.

IGB

UK

Fig. 14. Feature-cache hit rate for Naive, ADFC, and AHFCA.

Static- DynamicOnly Cache

E2E time (s) 2.464 Repeated epochs 0 Update/epoch (s) 0 Peak memory (GB) 17.68 Final accuracy (%) 67.23

2.015 10 2.83 18.92 67.23

utilization and hit rates. We benchmark it against DUCATI’s dual-cache allocator and FastGL’s single-cache design under varying batch sizes, tracking the feature-cache hit rate and the average time spent in the nfeat-selecting process. As shown in Figure 13, AHFCA achieves the highest feature-cache hit rate among the evaluated baselines. By profiling source and destination vertices across all GNN layers, its memory-driven allocation strategy captures both immediate one-hop demands and multi-hop reuse patterns, retaining 5–76% more frequently accessed features in on-chip caches than DUCATI and FastGL. As shown in Figure 13b, AHFCA also reduces nfeatselecting time through high-fidelity offline allocation and adaptive online caching. By leveraging full-link access statistics, it preloads frequently reused features into GPU caches, achieving hit rates above 87% early in execution compared with below 62% for the baselines. The ADFC mechanism further tracks hot nodes with an Always-On Exponentially Weighted Moving Average (EWMA), reducing nfeat-selecting time to less than 26% of its initial value within 5–8 epochs. Across PA, UK, and IGB, this produces an average 5.05× speedup in nfeat-selecting time. The cache-isolation experiment in Figure 14 quantifies the contributions of the optimized static allocation and the bounded Dynamic-Cache. Naive uses the original static dualcache policy, ADFC applies full-link profiling to optimize static-cache allocation, and AHFCA further augments ADFC with the bounded Dynamic-Cache. Averaged across PA, IGB, and UK, ADFC increases the feature-cache hit rate by an absolute 4.65% over Naive. Adding the Dynamic-Cache provides a further absolute improvement of 7.61%, yielding a total absolute increase of 12.26% from Naive to AHFCA. The additional gain is smaller on PA because its static cache already provides high coverage, whereas the gain is larger on IGB and UK because the Dynamic-Cache captures recurring hot nodes that remain uncovered by the static cache.



(a)

î î î î

î î

î î î

î î

3$

î î

î

î

î

î

,*%

î

î î



î



RXUV î

IDVW*/



î

1RUPDOL]HG6LPSOH7LPH

'8&$7,

î

'*/89$



12

8.

,*%

3$

8.

1RUPDOL]HG3URFHVVLQJRI*OREDO/RDG6HFWRU(IILFLHQF\

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021





TABLE VII T HREAD EXECUTION EFFICIENCY AND LONG - SCOREBOARD STALL FOR DGL-UVA AND GARDEN.



Mean Thread execution Long-scoreboard Dataset degree Framework efficiency stall



PA

15.55

DGL-UVA GARDEN

90.86% 96.75%

97.07% 59.03%

IGB

13.24

DGL-UVA GARDEN

91.28% 98.22%

93.11% 64.31%

UK

38.14

DGL-UVA GARDEN

90.48% 98.85%

92.98% 65.52%



(b)

Fig. 15. Comparison of sampling time and Global Load Sector Efficiency across different frameworks on the GraphSAGE model.

IDVW*/

RXUV

3$

(a)

8.

î î

î

3$





î

î

,*%





î

î

î î î î

î

î

î

î

,*%

î



î î î



î

 

î

î î



1RUPDOL]HG6DPSOH.HUQHO0HPRU\7KURXJKSXW

'8&$7,

î

'*/89$ 

1RUPDOL]HG6DPSOH7LPH

As reported in Table VI, the Dynamic-Cache overhead measurement gives average E2E times of 2.464 s for StaticOnly and 2.015 s for Dynamic-Cache, peak memories of 17.68 GB and 18.92 GB, and the same final accuracy of 67.23%. The Dynamic-Cache changes feature residency and cache maintenance only; it does not change the graph sampling procedure, neighbor fanouts, sampled neighborhoods, batch construction, model architecture, or training objective. Therefore, the two configurations produce the same final accuracy. In the measured configuration, Dynamic-Cache updates are performed for a fixed schedule of 10 repeated epochs and then stop. The 2.83 s cache-processing time per repeated epoch overlaps with GPU training and should not be added serially to the E2E runtime, so the bounded maintenance work does not translate into an equivalent additional online training cost. Effectiveness of the Grouping-based Parallel Sampler. We quantify the advantages of the Grouping-based Parallel Sampler from both micro-architectural and system-level throughput perspectives. Figure 15a presents the end-to-end GPU kernel time for the pure sampling phase, excluding feature aggregation. Our implementation reduces sampling latency to 10–87% of that in DGL, FastGL, and DUCATI, achieving speedups of 1.09× to 7.07×. The largest improvement occurs on the large-scale PA dataset, validating the effectiveness of our warp-level multi-node parallelism and sharedmemory coalescing strategy in mitigating irregular memory access patterns. Using Nsight Compute, we further analyze memory efficiency. Specifically, we measure Global Load Sector Efficiency (GLSE), which is the ratio of useful bytes loaded to total bytes transferred by the hardware. As shown in Figure 15b, DGL achieves only 30% GLSE on PA, indicating that 65% of L2 sector bandwidth is wasted on redundant 128-byte transactions. In contrast, our Grouping Sampler buffers index accesses in shared memory, limiting random lookups to onchip storage and issuing a single coalesced global memory load. This approach increases GLSE to 55%, representing a 1.83× improvement over DGL. Overall, the Grouping-based Parallel Sampler reduces compute latency and transforms irregular global memory accesses into efficient, coalesced shared-memory hits. This optimization unlocks memory-level parallelism and frees up global memory bandwidth for subsequent GNN training stages, all without requiring additional hardware. As reported in Table VII, GARDEN achieves higher thread

execution efficiency and lower long-scoreboard stall than DGL-UVA on PA, IGB, and UK. The mean degree is reported to characterize and distinguish the structural properties of the evaluated graphs. The higher thread execution efficiency is attributable to GPS’s degree-aware thread grouping, which keeps more execution lanes active and reduces warp divergence. The reduced long-scoreboard stall reflects lower global-memory access waiting enabled by GPS’s shared-memory buffering and coalesced global-memory loading.

8.



(b)

Fig. 16. The sampling iteration time for each epoch and memory throughput achieved by the Super-batch Degree Classifier.

Effectiveness of the Super-batch Degree Classifier. The Super-batch Degree Classifier further enhances sampling efficiency and improves sampling throughput across multiple datasets. As shown in Figure 16a, it achieves average sampling speedups of 6.47×, 1.42×, and 2.71× over DGL-UVA, DUCATI, and FastGL, respectively. Figure 16b demonstrates substantial improvements in end-to-end throughput compared to all baseline systems, with overall speedups ranging from 8.22× to 38.79×. These gains stem from two key optimizations enabled by the Super-batch Degree Classifier: (1) it coalesces multiple mini-batches into a single super-batch, significantly improving GPU utilization; and (2) it reduces the overhead of node classification in the Grouping-based Parallel Sampler. Together, these mechanisms lead to a dramatic increase in system-level throughput. To quantify the trade-off introduced by super-batching, we analyze its impact on execution granularity, memory footprint, and computational cost. By aggregating multiple mini-batches into a single super-batch, GARDEN substantially increases the effective batch size. Since the number of sub-batches K is typically set between 6 and 10, this design improves GPU SM occupancy and throughput, while also reducing the degreeclassification overhead during sampling. The main overheads of super-batching come from two

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

13

(a)

(b)



î

(c)



î î

î î

î

î î

î

î



î



î

î

1RUPDOL]HG5XQWLPH

  )HDWXUH'LPHQVLRQ

î

î

î

î î

î î



î





î

  %DWFKVL]H





î



RXUV

î

1RUPDOL]HG5XQWLPH

î î

î

î î





î î



î

î î



î

î

î î

  1XPEHURI*38V



î

î

î

î î

î î

î



IDVW*/



î î



î



î



î

î

1RUPDOL]HG5XQWLPH



'8&$7,

1RUPDOL]HG5XQWLPH

'*/89$

>@ >@ >@ 1XPEHURI1HLJKERUV

(d)

 

 

 













 

 



  

 

5XQWLPHV

 





 

3HDN0HPRU\*%



 

Fig. 17. Scalability analysis: (a) GPU count, (b) batch size, (c) feature dimension, and (d) number of sampled neighbors and GNN layers.

 





















  6XSHUEDWFK.







3$0HPRU\

8.0HPRU\

3$5XQWLPH

 

8.5XQWLPH

Fig. 18. Impact of super-batch hyperparameter K on peak GPU memory usage and sampling time per epoch across two datasets (PA and UK). The bars show peak memory consumption, while the lines represent sampling time per epoch.

sources: the temporary memory required by the super-batch hashmap and auxiliary tensor structures, and the cost of GPUresident hash-based deduplication. In practice, node overlap across mini-batches is high, so the deduplication ratio is also high. For large tensors, we use pinned memory to reduce GPU memory usage, keeping the additional footprint manageable. For deduplication, we adopt a lightweight GPU hash table for parallel insertion and lookup. Although hash operations introduce extra computation relative to atomic operations in per-mini-batch classification, this cost is largely offset by eliminating repeated small-batch kernel launches and reducing atomic contention. As shown in Figure 18, as K increases, the sampling time of the super-batch degree classifier is 1.2-1.4× that of the naive per-mini-batch method, while peak GPU memory increases by only 6.8%-8.5% on average. These results indicate that the performance gains from improved parallelism and reduced synchronization overhead outweigh the modest increases in memory and hashing cost, making super-batching a favorable design for billion-scale GNN training. Ablation Study. On PA with GraphSAGE and DGL as the baseline, the framework-level results in Figure 19 report speedups of 2.46× for AHFCA, 1.83× for GPS, and 1.31× for SDC. The combined configurations achieve 4.31× for AHFCA+GPS, 2.63× for AHFCA+SDC, and 2.36× for GPS+SDC, while the complete GARDEN configuration reaches 5.43×. The AHFCA+SDC configuration provides

only a modest gain over AHFCA alone, whereas SDC provides a more evident improvement when combined with GPS: GPS+SDC reaches 2.36×, compared with 1.83× for GPS alone. This behavior arises because SDC’s degreeclassification results are directly consumed by GPS’s grouped sampling execution, making SDC the key link between classification results and sampling execution. The complete GARDEN configuration reflects the combined effect of cache, sampling, and batching co-design. D. Scalability We evaluate GARDEN’s scalability across multiple dimensions to demonstrate its robustness under various hardware configurations and workload characteristics. Unless specified otherwise, experiments use the setup described in Section 6.1: batch size 8000 on the PA dataset using a single GPU. GPU Scaling. Figure 17a shows training throughput as we scale from 1 to 6 GPUs. For multi-GPU setups (n ≥ 2), we include GNNLab, which allocates 1 GPU for sampling when n ≤ 4 and 2 GPUs when n > 4. GARDEN consistently outperforms all baselines, with the performance gap widening as GPU count increases. This results from GARDEN’s coordination of host-device communication and super-batch processing. Batch Size Scaling. Figure 17b shows performance across batch sizes (4,000 to 12,000) on PA. GARDEN achieves 1.2× to 8.1× speedups over DGL, DUCATI, and FastGL, with larger batches yielding greater advantages. This trend stems from two factors: (1) increased node overlap across mini-batches enables

Fig. 19. Framework-level speedup over DGL for individual and combined GARDEN components on PA with GraphSAGE.

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

TABLE VIII ACCURACY AND EPOCH TIME OF THREE - LAYER G RAPH SAGE NODE CLASSIFICATION ON PA AND IGB. Dataset System

Accuracy

Epoch Time (s)

PA

DGL DUCATI FastGL GARDEN

0.670 ± 0.003 0.672 ± 0.001 0.672 ± 0.004 0.671 ± 0.003

29.79 4.86 15.32 3.95

IGB

DGL DUCATI FastGL GARDEN

0.581 ± 0.002 0.581 ± 0.004 0.583 ± 0.004 0.582 ± 0.001

31.13 7.99 19.17 5.30

more effective deduplication via our Super-batch Classifier, and (2) larger workloads allow the Grouping Parallel Sampler to achieve higher warp occupancy and more efficient memory coalescing. Feature Dimension Scaling. As shown in Figure 17c, we vary feature dimensions (128 to 512) on PA, measuring both end-to-end training time (bars) and computation time only (lines). GARDEN maintains 1.2× to 11.3× speedups across all dimensions. The persistent gap between total and computation time confirms that our AHFCA effectively reduces data movement overhead even with larger features by strategically caching high-utility data. Graph Complexity Scaling. Figure 17d evaluates increasingly complex sampling patterns ([5, 10], [5, 10, 15], [5, 5, 10, 10]) on PA. GARDEN achieves 1.2× to 8.18× speedups over baselines. This efficiency stems from (1) adjacency caching with offline reordering and (2) warp-level grouped sampling that converts random global accesses to coalesced shared memory operations. While FastGL matches GARDEN on simpler patterns like [5, 10], its performance degrades on complex patterns like [5, 5, 10, 10] where subgraphs exceed cache capacity, making sampling latency the bottleneck. In contrast, GARDEN’s adaptive caching maintains low overhead regardless of subgraph size. E. Training Accuracy We compare GraphSAGE training on PA and IGB using DGL, DUCATI, FastGL, and GARDEN. As shown in Table VIII, GARDEN achieves accuracy close to the baselines on both datasets: 0.671 ± 0.003 on PA and 0.582 ± 0.001 on IGB. Its epoch times are 3.95 s and 5.3 s, respectively, lower than those of the compared systems on the same datasets. These results indicate that GARDEN’s cache, sampling, and batching optimizations accelerate training without changing the model training objective or sampling semantics. UK and UK-Union are excluded from this accuracy comparison because official node features are unavailable; their randomly generated features are used only for runtime and scalability evaluation.

14

sampled subgraphs across epochs to lower sampling frequency while maintaining accuracy. GPU-accelerated samplers like NextDoor [35], WholeGraph [10], and gSampler [27] optimize subgraph extraction through parallel algorithms. NeutronOrch [7] and NeutronHeter [36] employ CPU offloading and workload orchestration in heterogeneous systems to balance resources and enhance throughput. Cache-based GNN Training. To mitigate feature gathering overheads, Quiver [9] implement GPU feature caching to minimize CPU-GPU transfers. GNNLab [6] uses a unified feature cache for node loading but omits adjacency structures, limiting sampling efficiency. Embedding caches in SANCUS [37], NeutronOrch [7], and FreshGNN [38] reduce aggregation data movement. FastGL [14] combines memory-aware pipelines with sampling and loading optimizations for higher throughput. DUCATI [8] extends caching to include both features and topology on GPUs, improving allocation under diverse workloads. For graphs exceeding memory capacity, DiskGNN [39] enables efficient out-of-core training by optimizing disk I/O to eliminate read amplification while maintaining full model accuracy. Micro-batch-based GNN Training. To tackle the GPU memory wall for large-scale GNN training, Betty [40] proposes batch-level graph partitioning that decomposes minibatches into redundancy-aware micro-batches, enabling deeper models and larger graphs under limited GPU memory with gradient accumulation. Building on this idea, Buffalo [41] further proposes memory-efficient node bucketing with flexible mappings and lightweight organization, reducing partitioning overhead, fragmentation, and redundant feature loading. Cherry [42] addresses the high cost of data preparation by constructing a Global Message-passing Flow Graph, which eliminates repetitive per-epoch partitioning and improves load balance while reducing node redundancy. More recently, PDGNN [43] targets micro-batch training with degree-pruned partitioning and explicit redundancy elimination, pruning lowcontribution edges and duplicated accesses to improve efficiency in small micro-batches and multi-GPU settings. Overall, these systems alleviate memory pressure by partitioning graphs to support larger batches or deeper models, but they primarily optimize computation partitioning within and across devices rather than irregular sampling latency, dynamic dualcache allocation under strict single-GPU budgets, or PCIeefficient sampling workflows. GARDEN differs from existing approaches by cooptimizing caching, sampling, and batching in a hardwareaware framework. It achieves higher hit rates and lower latency through adaptive allocation, grouped parallelism, and superbatch classification, enabling efficient billion-scale training in settings where prior systems remain bottlenecked by I/O or compute imbalance.

VII. R ELATED W ORK

VIII. C ONCLUSION

Sample-based GNN Training. GraphSAGE [11] introduced fixed-size neighbor sampling and a general aggregation framework for inductive learning on large graphs. Subsequent efforts target sampling bottlenecks: LazyGCN [34] reuses

In this paper, we tackle the critical challenge of scaling GNN training to billion-edge graphs, where the GPU memory wall imposes severe limitations on efficiency. Existing sampling-based frameworks struggle with excessive PCIe data

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

transfers and irregular memory access patterns, leading to poor bandwidth utilization and computational inefficiency. To overcome these bottlenecks, we introduce GARDEN, a GPU-accelerated sampling-based GNN system that co-designs caching, sampling, and batching through three key innovations: the Adaptive High-Fidelity Cache Allocator for dynamic memory management under fixed budgets, the Groupingbased Parallel Sampler for optimized warp occupancy on power-law graphs, and the Super-batch Degree Classifier for cache-friendly data transfers. Extensive evaluations on billionscale graphs demonstrate that GARDEN achieves significant speedups over the state-of-the-art baselines while maintaining model convergence and accuracy. R EFERENCES [1] J. Zhou, G. Cui, S. Hu, Z. Zhang, C. Yang, Z. Liu, L. Wang, C. Li, and M. Sun, “Graph neural networks: A review of methods and applications,” AI Open, vol. 1, pp. 57–81, 2020. [Online]. Available: https://arxiv.org/abs/1812.08434 [2] L. Wu, P. Sun, Y. Fu, R. Hong, X. Wang, and M. Wang, “Socialgcn: An efficient graph convolutional network based model for social recommendation,” arXiv preprint arXiv:1811.02815, 2019. [Online]. Available: [https://arxiv.org/abs/1811.02815](https://arxiv.org/abs/1811. 02815) [3] Z. Yang and S. Dong, “HAGERec: Hierarchical attention graph convolutional network incorporating knowledge graph for explainable recommendation,” Knowledge-Based Systems, vol. 204, p. 106194, 2020. [Online]. Available: https://www.sciencedirect.com/science/article/pii/ S0950705120304196 [4] K. Yang, K. Swanson, W. Jin, C. Coley, P. Eiden, H. Gao, A. GuzmanPerez, T. Hopper, B. Kelley, M. Mathea et al., “Analyzing learned molecular representations for property prediction,” Journal of chemical information and modeling, vol. 59, no. 8, pp. 3370–3388, 2019. [5] W. Hu, M. Fey, M. Zitnik, Y. Dong, H. Ren, B. Liu, M. Catasta, and J. Leskovec, “Open graph benchmark: Datasets for machine learning on graphs,” arXiv preprint, vol. arXiv:2005.00687, 2020. [Online]. Available: https://arxiv.org/abs/2005.00687 [6] J. Yang, D. Tang, X. Song, L. Wang, Q. Yin, R. Chen, W. Yu, and J. Zhou, “GNNLab: A factored system for sample-based GNN training over GPUs,” in Proc. Seventeenth European Conference on Computer Systems, ser. EuroSys ’22, New York, NY, USA, Apr. 2022, pp. 417–434. [Online]. Available: https://doi.org/10.1145/3492321.3519557 [7] X. Ai, Q. Wang, C. Cao, Y. Zhang, C. Chen, H. Yuan, Y. Gu, and G. Yu, “NeutronOrch: Rethinking sample-based GNN training under CPU-GPU heterogeneous environments,” Proc. VLDB Endow., vol. 17, no. 8, pp. 1995–2008, Apr. 2024. [Online]. Available: https://doi.org/10.14778/3659437.3659453 [8] X. Zhang, Y. Shen, Y. Shao, and L. Chen, “DUCATI: A dual-cache training system for graph neural networks on giant graphs with the GPU,” Proc. ACM Manag. Data, vol. 1, no. 2, Jun. 2023. [Online]. Available: https://doi.org/10.1145/3589311 [9] Z. Tan, X. Yuan, C. He, M.-K. Sit, G. Li, X. Liu, B. Ai, K. Zeng, P. R. Pietzuch, and L. Mai, “Quiver: Supporting gpus for low-latency, high-throughput gnn serving with workload awareness,” ArXiv, vol. abs/2305.10863, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:258762769 [10] D. Yang, J. Liu, J. Qi, and J. Lai, “Wholegraph: A fast graph neural network training framework with multi-gpu distributed shared memory architecture,” in SC22: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 2022, pp. 1–14. [11] W. L. Hamilton, R. Ying, and J. Leskovec, “Inductive representation learning on large graphs,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, ser. NIPS’17, Red Hook, NY, USA, 2017, p. 1025–1035. [12] D. Zou, Z. Hu, Y. Wang, S. Jiang, Y. Sun, and Q. Gu, Layer-dependent importance sampling for training deep and large graph convolutional networks. Curran Associates Inc., 2019. [13] X. Liu, X. Xiong, M. Yan, R. Xue, S. Pan, S. Pei, L. Deng, X. Ye, and D. Fan, “DropNaE: Alleviating irregularity for large-scale graph representation learning,” Neural Networks, vol. 183, p. 106930, 2024. [Online]. Available: https://doi.org/10.1016/j.neunet.2024.106930

15

[14] Z. Zhu, P. Wang, Q. Hu, G. Li, X. Liang, and J. Cheng, “Fastgl: A gpu-efficient framework for accelerating sampling-based gnn training at large scale,” in Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 4, ser. ASPLOS ’24, 2025, p. 94–110. [Online]. Available: https://doi.org/10.1145/3622781.3674167 [15] S. W. Min, K. Wu, S. Huang, M. Hidayetoğlu, J. Xiong, E. Ebrahimi, D. Chen, and W.-m. Hwu, “Large graph convolutional network training with gpu-oriented data communication architecture,” Proc. VLDB Endow., vol. 14, no. 11, p. 2087–2100, Jul. 2021. [Online]. Available: [https://doi.org/10.14778/3476249.3476264](https: //doi.org/10.14778/3476249.3476264) [16] M. Fey and J. E. Lenssen, “Fast graph representation learning with pytorch geometric,” CoRR, vol. abs/1903.02428, 2019. [Online]. Available: http://arxiv.org/abs/1903.02428 [17] Z. Lin, C. Li, Y. Miao, Y. Liu, and Y. Xu, “Pagraph: Scaling gnn training on large graphs via computation-aware caching,” in Proceedings of the 11th ACM Symposium on Cloud Computing, ser. SoCC ’20. Association for Computing Machinery, 2020, p. 401–415. [Online]. Available: https://doi.org/10.1145/3419111.3421281 [18] S. W. Min, K. Wu, M. Hidayetoglu, J. Xiong, X. Song, and W.-m. Hwu, “Graph neural network training and data tiering,” in Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, ser. KDD ’22. New York, NY, USA: Association for Computing Machinery, 2022, p. 3555–3565. [Online]. Available: [https://doi.org/10.1145/3534678.3539038](https: //doi.org/10.1145/3534678.3539038) [19] R. Zhu, K. Zhao, H. Yang, W. Lin, C. Zhou, B. Ai, Y. Li, and J. Zhou, “Aligraph: A comprehensive graph neural network platform,” Proceedings of the VLDB Endowment, vol. 12, no. 12, pp. 2094–2105, 2019. [Online]. Available: https://www.vldb.org/pvldb/vol12/p2094-zhu. pdf [20] Y. Luo, Y. Wang, Q. Wang, Y. Song, H. Wu, Q. Wang, and J. Huang, “DCI: An efficient workload-aware dual-cache allocation GNN inference acceleration system,” in Euro-Par 2025: Parallel Processing, W. E. Nagel, D. Goehringer, and P. C. Diniz, Eds. Springer, 2025, pp. 18–32. [21] Y. Tan, Y. Gan, Z. Zeng, Z. Bai, L. Qiao, D. Liu, K. Zhong, and A. Ren, “Gnnboost: Accelerating sampling-based gnn training on large scale graph by optimizing data preparation,” Journal of Systems Architecture, vol. 167, p. 103456, 2025. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1383762125001286 [22] NVIDIA Corporation, CUDA C++ Programming Guide, latest version ed., 2025, design Guide, section on Unified Virtual Addressing. [Online]. Available: https://docs.nvidia.com/cuda/ cuda-c-programming-guide/ [23] M. Wang, L. Yu, D. Zheng, Q. Gan, Y. Gai, Z. Ye, M. Li, J. Zhou, Q. Huang, C. Ma, Z. Huang, Q. Guo, H. Zhang, H. Lin, J. J. Zhao, J. Li, A. Smola, and Z. Zhang, “Deep Graph Library: Towards efficient and scalable deep learning on graphs,” arXiv preprint arXiv:1909.01315, 2019. [Online]. Available: https://arxiv.org/abs/1909.01315 [24] A. Khatua, V. S. Mailthody, B. Taleka, T. Ma, X. Song, and W.-m. Hwu, “Igb: Addressing the gaps in labeling, features, heterogeneity, and size of public graph datasets for deep learning research,” in Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, ser. KDD ’23, 2023, p. 4284–4295. [Online]. Available: [https://doi.org/10.1145/3580305.3599843](https: //doi.org/10.1145/3580305.3599843) [25] P. Boldi and S. Vigna, “The webgraph framework i: compression techniques,” in Proceedings of the 13th International Conference on World Wide Web, ser. WWW ’04, 2004, p. 595–602. [Online]. Available: [https://doi.org/10.1145/988672.988752](https: //doi.org/10.1145/988672.988752) [26] NVIDIA Corporation, “Nvidia nsight compute,” 2025, interactive kernel profiler for CUDA applications. [Online]. Available: https: //developer.nvidia.com/nsight-compute [27] P. Gong, R. Liu, Z. Mao, Z. Cai, X. Yan, C. Li, M. Wang, and Z. Li, “gSampler: General and efficient GPU-based graph sampling for graph learning,” in Proceedings of the 29th Symposium on Operating Systems Principles, ser. SOSP ’23. New York, NY, USA: ACM Press, 2023, pp. 562–578. [Online]. Available: https://doi.org/10.1145/3600006.3613168 [28] NVIDIA Corporation, “CUB: Cooperative Primitives for CUDA C++,” Part of CUDA Core Compute Libraries (CCCL), 2026. [Online]. Available: https://nvidia.github.io/cccl/cub/ [29] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style,

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

high-performance deep learning library,” in Advances in Neural Information Processing Systems 32. Curran Associates, Inc., 2019, pp. 8024–8035. [Online]. Available: http://papers.neurips.cc/paper/ 9015-pytorch-an-imperative-style-high-performance-deep-learning-library. pdf [30] NVIDIA Corporation, “Nccl: Optimized primitives for collective multigpu communication,” https://github.com/NVIDIA/nccl, 2023, accessed: 2026-01-01. [31] T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” in International Conference on Learning Representations, 2017. [Online]. Available: https://openreview.net/ forum?id=SJU4ayYgl [32] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio, “Graph attention networks,” in International Conference on Learning Representations, 2018. [Online]. Available: https: //openreview.net/forum?id=rJXMpikCZ [33] Y. Luo, L. Shi, and X.-M. Wu, “Classic gnns are strong baselines: reassessing gnns for node classification,” in Proceedings of the 38th International Conference on Neural Information Processing Systems, ser. NIPS ’24. Red Hook, NY, USA: Curran Associates Inc., 2024. [34] M. Ramezani, W. Cong, M. Mahdavi, A. Sivasubramaniam, and M. Kandemir, “Gcn meets gpu: Decoupling “when to sample”from “how to sample”,” in Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 18 482– 18 492. [Online]. Available: https://proceedings.neurips.cc/paper files/ paper/2020/file/d714d2c5a796d5814c565d78dd16188d-Paper.pdf [35] A. Jangda, S. Polisetty, A. Guha, and M. Serafini, “Accelerating graph sampling for graph machine learning using GPUs,” in Proceedings of the Sixteenth European Conference on Computer Systems, ser. EuroSys ’21. New York, NY, USA: ACM Press, 2021, pp. 311–326. [Online]. Available: https://doi.org/10.1145/3447786.3456244 [36] C. Cao, X. Ai, Q. Wang, Y. Zhang, Z. Fu, H. Yuan, M. Cao, C. Chen, Y. Wen, Y. Gu, and G. Yu, “Neutronheter: Optimizing distributed graph neural network training for heterogeneous clusters,” Proc. ACM Manag. Data, vol. 3, no. 4, Sep. 2025. [Online]. Available: [https://doi.org/10.1145/3749175](https://doi.org/10.1145/3749175) [37] J. Peng, Z. Chen, Y. Shao, Y. Shen, L. Chen, and J. Cao, “Sancus: staleness-aware communication-avoiding fullgraph decentralized training in large-scale graph neural networks,” Proc. VLDB Endow., vol. 15, no. 9, p. 1937–1950, May 2022. [Online]. Available: [https://doi.org/10.14778/3538598.3538614](https: //doi.org/10.14778/3538598.3538614) [38] K. Huang, H. Jiang, M. Wang, G. Xiao, D. Wipf, X. Song, Q. Gan, Z. Huang, J. Zhai, and Z. Zhang, “Freshgnn: Reducing memory access via stable historical embeddings for graph neural network training,” Proc. VLDB Endow., vol. 17, no. 6, p. 1473–1486, Feb. 2024. [Online]. Available: [https://doi.org/10.14778/3648160.3648184](https: //doi.org/10.14778/3648160.3648184) [39] R. Liu, Y. Wang, X. Yan, H. Jiang, Z. Cai, M. Wang, B. Tang, and J. Li, “Diskgnn: Bridging i/o efficiency and model accuracy for out-of-core gnn training,” Proc. ACM Manag. Data, vol. 3, no. 1, Feb. 2025. [Online]. Available: https://doi.org/10.1145/3709738 [40] S. Yang, M. Zhang, W. Dong, and D. Li, “Betty: Enabling large-scale gnn training with batch-level graph partitioning,” in Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ser. ASPLOS 2023. New York, NY, USA: Association for Computing Machinery, 2023, p. 103–117. [Online]. Available: https: //doi.org/10.1145/3575693.3575725 [41] S. Yang, M. Zhang, and D. Li, “ Buffalo: Enabling LargeScale GNN Training via Memory-Efficient Bucketization ,” in 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA). Los Alamitos, CA, USA: IEEE Computer Society, Mar. 2025, pp. 1066–1081. [Online]. Available: https: //doi.ieeecomputersociety.org/10.1109/HPCA61900.2025.00083 [42] Y. Wang, Q. Guo, H. Kong, K. Sheng, Z. Xie, H. Chen, W. Jia, D. Tao, and X. He, “Cherry: Breaking the gpu memory wall for large-scale gnn training via micro-batching,” in Proceedings of the 39th ACM International Conference on Supercomputing, ser. ICS ’25. Association for Computing Machinery, 2025, p. 88–103. [Online]. Available: https://doi.org/10.1145/3721145.3730429 [43] L. Zhang, M. Jia, P. Gong, Z. Lai, D. Li, Y. Fu, A. Shen, and K. Lu, “Pdgnn: Efficient micro-batch gnn training via degreepruned partitioning and redundancy elimination,” ACM Trans. Archit. Code Optim., vol. 22, no. 4, Dec. 2025. [Online]. Available: https://doi.org/10.1145/3767325

16

Related documents

• EmbodiedIntentBench: Benchmarking Intent-Driven Runtime Adaptation for Mobile Robotic Missions#1605227
• Reproducible LoRA Inference Through Deterministic Split-K Reduction#1605228
• Continual Instruction Tuning for LLM-ased Management of Continuum Resources#1605229
• FirstSwap: Pre-Consensus Trade Approvals for Decentralized Market Design#1605230
• PACE: Keeping the Right Tokens in the KV Cache Is Not Enough for Long-Horizon LLM Agents#1605231