Analyzing 10 Petabit/s Network Data with Accelerated Associative (Token) Arrays Jeremy Kepner, Hayden Jananthan, LaToya Anderson, William Arcand, David Bestor, William Bergeron, Chansup Byun, Alex Bonn, Daniel Burrill, Vijay Gadepally, Michael Houle, Matthew Hubbell, Michael Jones, Piotr Luszczek, Peter Michaleas, Lauren Milechin, Julie Mullen, Andrew Prout, Albert Reuther, Antonio Rosa, Charles Yee, Alex Pentland
arXiv:2609.32978v1 [cs.NI] 26 Sep 2026
MIT Abstract—As networks expand and become an ever more critical infrastructure to modern society the need to analyze these networks with the highest regard for privacy is essential to ensure their proper function. Depending on the level of the network layer to be analyzed, sources and destinations can be any combination of physical, logical, or persona/agentic endpoints, which requires the ability to handle diverse data. Invaluable to these analyses are mathematical tools that enable sophisticated mathematical algorithms to be expressed succinctly while achieving scalable vertical (within a compute node), horizontal (across compute nodes), and temporal (over different generations of hardware) performance. Associative (token) array mathematics and corresponding libraries is one approach that can meet these requirements. Accelerating these libraries with GPUs enables the analysis of the largest networks. The MIT/IEEE/Amazon Anonymized Network Sensing Graph Challenge provides a venue for highlighting the applicability of accelerated associative arrays for these types of problems. The D4M associative library has been implemented in a number of languages. This work benchmarks a prototype Matlab D4M GPU accelerated implementation of the Anonymized Network Sensing challenge across a wide range of CPU and GPU hardware. Scalable performance is demonstrated within and across CPU cores, CPU nodes, and GPU nodes. Horizontal scaling across multiple nodes was linear. Running on hundreds of GPU nodes simultaneously achieved a sustained processing rate sufficient to potentially analyze a 10 Petabit/s network. Index Terms—network analysis, token arrays, AI, GPUs, distributed arrays, parallel processing, vertical scaling, horizontal scaling
high scalability can play a key role in advancing the field. Associative (token) array mathematics [11] and corresponding libraries [12]–[15] accelerated with GPUs is one approach for assisting with the analysis of the largest networks. The Anonymized Network Sensing Graph Challenge seeks to enable large, open, community-based approaches to protecting networks by providing a common venue for highlighting relevant innovations in network sensing and analysis [16], [17]. This challenge is an ideal venue for exploring accelerated associative array mathematics for large-scale network analysis. The presentation of the rest of the paper is as follows. A more detailed description of the mathematical analysis performed within the Anonymized Network Sensing challenge is provided followed by a discussion of the mathematics of associative arrays. Next, is a discussion of some of the key steps for adapting the specific associative array math library used in this work to take advantage of GPUs. A key step for porting an application to GPUs is to port the underling basic functions; a short description of the functions for the corresponding associative array library is given. Subsequently, the accelerated Anonymized Network Sensing challenge associative array code is presented. This is followed by a description of the benchmarking hardware and the vertical (within node), horizontal (across nodes), and temporal (across time) scaling results.
I. I NTRODUCTION
II. A NONYMIZED N ETWORK S ENSING C HALLENGE
Many large-scale networking problems can only be solved with community access to very broad data sets with the highest regard for privacy and strong community buy-in [1]– [3]. As networks expand and become an even more critical infrastructure for modern society, the need to analyze these networks is essential to ensure their proper function [4]– [10]. Mathematical tools that enable sophisticated mathematical algorithms to be expressed succinctly while achieving
The MIT/IEEE/Amazon GraphChallenge has fostered many community approaches for developing new solutions for analyzing graphs and sparse data derived from social media, sensor feeds, and scientific data to discover relationships between events as they unfold in the field [18]–[41]. The anonymized network sensing Graph Challenge [16], [17] seeks to enable large, open, community-based approaches to protecting networks [42]–[47]. Many large-scale networking problems can only be solved with community access to very broad data sets with the highest regard for privacy and strong community buy-in. Such approaches often require communitybased data sharing. In the broader networking community (commercial, federal, and academia) anonymized source-todestination traffic matrices with standard data sharing agreements have emerged as a data product that can meet many of these requirements.
Research was sponsored by the Department of the Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-19-2-1000. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Department of the Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein. Use of this work is controlled by the human-tohuman license listed in Exhibit 3 of https://doi.org/10.48550/arXiv.2306.09267
source packets (packets from a source)
unique sources
source fan-out
NV
unique links
X
At (i, j) = NV
i,j
valid source packet window
valid destination packet window
NV=217,218,…,227
NV=217,218,…,227
(time window)
(time window)
link packets
destination fan-in
unique destinations
destination packets (packets to a destination)
Fig. 1. Anonymized Network Sensing Challenge Quantities. Internet traffic streams of NV valid packets are divided into a variety of quantities for analysis by the anonymized network sensing challenge: source packets, source fan-out, unique source-destination pair packets (or links), destination fan-in, and destination packets. Figure adapted from [48]. TABLE I N ETWORK Q UANTITIES FROM T RAFFIC M ATRICES Formulas for computing network quantities from a traffic matrix At at time t in both summation and matrix notation. 1 is a column vector of all 1’s, T is the transpose operation, and | |0 is the zero-norm that sets each nonzero value of its argument to 1 [49]. These formulas are unaffected by matrix permutations and work on anonymized data. Underlined quantities are those specified in the anonymized network sensing Graph Challenge. Table adapted from [50]. Aggregate Property Valid packets NV Unique links Link packets from i to j Max link packets Unique sources Packets from source i Max source packets Source fan-out from i Max source fan-out Unique destinations Destination packets to j Max destination packets Destination fan-in to j Max destination fan-in
Summation Notation P P At (i, j) i P Pj i j |At (i, j)|0 At (i, j) max A (i, j) P P ij t i | Pj At (i, j)|0 Pj At (i, j) maxi Pj At (i, j) Pj |At (i, j)|0 maxi j |At (i, j)|0 P P j | P i At (i, j)|0 Pi At (i, j) maxj P i At (i, j) Pi |At (i, j)|0 maxj i |At (i, j)|0
Matrix Notation 1T At 1 1T |At |0 1 At max(At ) 1T |At 1|0 At 1 max(At 1) |At |0 1 max(|At |0 1) |1T At |0 1 1T At max(1T At ) 1T |At |0 max(1T |At |0 )
A core concept in the Anonymized Network Sensing challenge is that traffic data can be viewed as a traffic matrix where each row is a source and each column is a destination. A primary benefit of constructing anonymized traffic matrices is the efficient computation of a wide range of network quantities via matrix mathematics. Figure 1 illustrates essential quantities found in all streaming dynamic networks. These quantities are all computable from anonymized traffic matrices created from the source and destination addresses found in Internet packet headers [51]–[54]. To reduce statistical fluctuations, the streaming data is partitioned so that for any chosen time window all data sets have the same number of valid packets [50]. At a given time t, NV consecutive valid packets are aggregated from the network traffic into a matrix At , where At (i, j) is the number of valid packets between the source i and destination j. The sum of all the entries in At is equal to
Constant packet, variable time samples simplify the statistical analysis of the heavy-tail distributions commonly found in network traffic quantities [48], [55], [56]. All the network quantities depicted in Figure 1 can be readily computed from At using the formulas listed in Table I. III. A SSOCIATIVE (T OKEN ) A RRAYS Depending on the level of the network layer to be analyzed, sources and destinations can be any combination of physical, logical, or persona/agentic endpoints, such as, AI agents communicating via MCP (Model Context Protocol) [57]. These endpoints are often represented as strings. Associative arrays, generalize matrices and their operations to much broader data sets [11]. More specifically, an array A(i, j) = v is a mapping from a pair of indices i and j to a value v where i ∈ I, j ∈ J, v ∈ V For an N ×N real-valued matrix A : RN ×N I = J = {1, ..., N }, V = R An associative array generalizes this concept by allowing I, J, V to be any strict totally ordered set, which includes numbers and strings. In AI large language model (LLM) terminology, strict totally ordered sets are simply tokens; often words or parts of words that are used as the labels of the rows and columns of the many matrices used insides LLMs. In many respects, LLMs are simply collections of associative (token) arrays. In the Anonymized Network Sensing Graph Challenge the endpoints are set by Internet Protocol version 4 (IPv4) using 32 bit unsigned numbers. An IPv4 source/destination can be displayed as an integer (67305985), hexadecimal value (01020304), binary value (00000001000000100000001100000100), padded dotted quad string (001.002.003.004), or unpadded dotted quad string (1.2.3.4). Associative arrays encompass all of these and allow the same mathematics to be performed regardless of the underlying representation. The Dynamic Distributed Dimensional Data Model (D4M) mathematical library (see d4m.mit.edu) implements associative arrays in a number of high-level programming languages [12]–[15]. IV. ACCELERATED A RRAYS Porting D4M to a GPU enables all of these representations to benefit from accelerated computing. The focus of this work is on the Matlab D4M implementation that leverages the extensive GPU programming system within Matlab. A cornerstone of the Matlab GPU system are two functions gpuArray() and gather() that respectively copy a Matlab variable from a CPU to a CPU and from GPU to a CPU
Agpu = gpuArray(Acpu)
- Copies variable to GPU - Changes type to GPU variable Acpu = gather(Agpu)
- Copies from GPU - Changes type to CPU variable In addition, the type of the variable is also changed from a CPU variable to GPU variable and from GPU variable to a CPU variable, respectively. Thus, a Matlab GPU program can often appear identical to a Matlab CPU program as type information determines whether any given function is run on a CPU or a GPU. Thus, a Matlab accelerated GPU program can often appear identical to a normal Matlab CPU program as type information determines whether any given function is run on a CPU or a GPU. D4M employs object type and function overloading in a similar manner. The core D4M associative array data structure can be converted to a GPU object via corresponding function overloading AgpuD4M = gpuArray(AcpuD4M) AgpuD4M.row = gpuArray(uint16(AcpuD4M.row)) AgpuD4M.col = gpuArray(uint16(AcpuD4M.col)) AgpuD4M.val = gpuArray(uint16(AcpuD4M.val)) AgpuD4M.A = gpuArray(AcpuD4M.A) AcpuD4M = gather(AgpuD4M) AcpuD4M.row = char(gather(AgpuD4M.row)) AcpuD4M.col = char(gather(AgpuD4M.col)) AcpuD4M.val = char(gather(AgpuD4M.val)) AcpuD4M.A = gather(AgpuD4M.A)
Subsequent D4M operations on these fields will then be performed on the GPU. The D4M data structure consists of three character array fields (.row, .col, .val) and a sparse matrix (.A) holding an index to the corresponding entry in .val) [Note: if .val is empty, then the associative array values are numeric and are as given in .A]. An important choice in the original D4M design was to store character array fields as flat vectors using the convention that the last character in the vector was the string separator. This enabled operations on these fields to be performed as vector operations that are performant on both CPUs and GPUs. At the time of writing, the Matlab character type was not directly supported on the GPU, so it is converted to a GPU supported unsigned 16-bit integer. Porting of D4M functions to operate on GPU associative arrays is mostly an exercise in rewriting code to use performant functions that are supported on both the CPU and GPU, which allows the code to be independent of where it is run. Perhaps the most common trick used to achieve this end is rewriting array index operations, which have limited GPU support on sparse matrices, to employ sparse matrix multiply of corresponding diagonal matrices A(i, j) = Idiag A Jdiag where Idiag has 1’s along the diagonal corresponding to the rows to be selected (i.e., Idiag (i(i), i(i)) = 1), and Jdiag has 1’s along the diagonal corresponding to the columns to
TABLE II BASIC A SSOCIATIVE A RRAY O PERATIONS Construction Addition Subtraction Array Multiplication Elementwise Multiplication
A = Assoc(row,col,val) C = A + B C = A - B C = A * B C = A .* B
be selected (i.e., Jdiag (j(j), i(j)) = 1). The above approach generally results in improved performance on both CPUs and GPUs, while often allowing very complicated index remapping operations to be combined into a single step. Sparse matrix libraries such as the GraphBLAS use this method to implement matrix indexing as it a leverages the already heavily optimized matrix multiplication function. D4M also uses matrix multiplication to implement indexing, thus D4M indexing function itself required little adjustment to run on a GPU. Generally speaking, the more vectorized the code is, the more it will run well on both CPUs and GPUs without modification. It is particularly worth noting the impressive Matlab profiling environment that works identically for both CPU and GPU code. The profiler clearly highlights the most time consuming parts of the code. It is then simple to insert a breakpoint before the corresponding code and perform live in situ performance tuning to optimize the code. The profiler makes Matlab a highly effective environment for discovering optimal GPU algorithms for subsequent translation into other languages. V. BASIC F UNCTIONS Associative array libraries rely on a few basic functions for implementing algorithms (see Table II). Initial GPU porting and benchmarking starts with these functions. An important aspect of benchmarking these functions is the data used to randomly populate the appropriate sparse associative arrays. Two important parameters are the sizes of N ×N associative arrays and the number of non-empty entries M . A common way to connect these parameters is via the formula M = kN , where k ∼ 10. Finally, the associative array indices are stored as their decimal character representation padded to the length of N . For example, for N = 210 the indices would be drawn from set of strings {0001, ..., 1024}. VI. B ENCHMARK C ODE The Anonymized Network Sensing Graph Challenge consists of six steps 1) Read/stream each network packet capture (PCAP) file. 2) Extract the source IP and destination IP addresses from the packet headers and buffer NV valid packets. 3) Anonymize the source IP and destination IP. 4) Construct sequential traffic matrices from NV valid packets. 5) Save the traffic matrices to files. 6) Read in the traffic matrix files, sum the traffic matrices associated with a PCAP file into one large traffic matrix, and perform the analysis highlighted in Table I.
Furthermore, “Graph Challenge participants are free to select (with accompanying explanation) the Graph Challenge elements that are appropriate for highlighting their innovations.” [16] Along these lines, this work uses the Matlab reference code available on the GraphChallenge.org website and adapts it to use associative arrays. The benchmarking focus of the work is on the analysis part of Step (6). Thus, the code begins by reading in GraphBLAS versions of the random traffic matrices that are available from the GraphChallenge.org website. The matrices are converted to associative arrays and then the analysis are executed and timed. The GraphBLAS traffic matrix data was converted to associative arrays by extracting the row, column, and value triples. For the CPU, the rows/columns were converted to padded dotted quad representations used to construct corresponding associative arrays. For the GPU, since strings are already being converted to 16-bit unsigned integers an additional optimization was be performed by converting each byte of the row/column IPv4 address to 16-bit unsigned integer (offset by 1 to avoid the side effects of using 0 when integrating with sparse libraries). This results in the GPU indices being ∼1/3 the size of the CPU indices. The analysis code operations corresponding to Table I were implemented as follows using the same code on CPUs and GPUs AA = Adj(A) v = nonzeros(AA) Npackets = sum(v) Nlinks = length(v) Nsrc = size(A,1) Ndest = size(A,2) MaxPackets = max(v) colVec = ones(Ndest,1,localType) MaxSrcPackets = max(full(AA*colVec)) MaxFanOut = max(full(sign(AA)*colVec)) rowVec = ones(1,Nsrc,localType) MaxDestPackets = max(full(rowVec*AA)) MaxFanIn = max(full(rowVec*sign(AA)))
The above code is performant on both CPUs and GPUs. The potential GPU nature of the code is only manifest in the use of the localType variable that instructs the ones() constructor where to create data. The code simultaneously uses the associative array A, sparse adjacency matrix AA, and value list v representations of the data to provide a variety of options for selecting operations that perform well on both CPUs and the GPUs. For example, summing and maxing v as the fastest way to compute Npackets and Nlinks. Likewise, multiplying AA by a dense row/column vectors to compute MaxSrcPackets, MaxFanOut, MaxDestPackets, and MaxFanIn. VII. B ENCHMARKING Our team has developed a high-productivity scalable platform—the MIT SuperCloud—for providing scientists and engineers the tools they need to analyze large-scale dynamic
TABLE III C OMPUTER H ARDWARE S PECIFICATIONS MIT SuperCloud maintains a diverse set of hardware running an identical modern software stack providing a unique platform for comparing performance over different eras. Node Label
Processor
Memory
Era
Part
Clock
Cores
Part
Size
amd-e9
2024
Dual AMD EPYC 9254
2.9 GHz
48
DDR5
750 GB
- h100nvl
2024
Dual Nvidia H100 NVL
1.7 GHz
HBM3
188 GB
xeon-p8
2020
Dual Xeon Platinum 8260
2.4 GHz
48
DDR4
192 GB
xeon-g6
2018
Dual Xeon Gold 6248
2.5 GHz
40
DDR4
384 GB
- v100
2018
Dual Nvidia V100
1.2 GHz
xeon-e5
2014
Dual Xeon E5-2683 v3
2.0 GHz
28
HBM2
64 GB
DDR4
256 GB
TABLE IV S INGLE N ODE BASIC F UNCTIONS PARAMETERS Number of processes (NP ), number of cores/GPUs per process (NCP P ), array dimension N , and average number of non-empty entries per row/column (k) used for single node benchmarking of the associative array basic functions. These benchmark parameters were chosen to balance consistency with the memory capacity of the different hardware. NPxNCPP, M/NP, k
Node Label
Construction
Addition & Subtraction
Array Multiplication
Elementwise Multiplication
Core
Node
Core
Node
Core
Node
Core
Node
amd-e9
1x1, 220, 16
48x1, 220, 16
1x1, 221, 8
48x1, 221, 8
1x1, 219, 8
48x1, 219, 8
1x1, 221, 8
48x1, 221, 8
- h100nvl
1x1, 220, 16
2x1, 220, 16
1x1, 221, 8
2x1, 221, 8
1x1, 219, 8
2x1, 219, 8
1x1, 221, 8
2x1, 221, 8
xeon-p8
1x1, 220, 16
48x1, 219, 16
1x1, 219, 8
48x1, 219, 8
1x1, 219, 8
48x1, 219, 8
1x1, 220, 8
48x1, 219, 8
xeon-g6
1x1, 220, 16
40x1, 220, 16
1x1, 219, 8
40x1, 219, 8
1x1, 219, 8
40x1, 219, 8
1x1, 220, 8
40x1, 220, 8
- v100
1x1, 220, 16
2x1, 220, 16
1x1, 219, 8
2x1, 219, 8
1x1, 219, 8
2x1, 219, 8
1x1, 220, 8
2x1, 220, 8
xeon-e5
1x1, 220, 16
28x1, 220, 16
1x1, 219, 8
28x1, 219, 8
1x1, 219, 8
28x1, 219, 8
1x1, 220, 8
28x1, 220, 8
data [12], [58], [59]. The MIT SuperCloud provides interactive analysis capabilities accessible from high level programming environments that scale to thousands of processing nodes. MIT SuperCloud maintains a diverse set of hardware running an identical software stack that allows direct comparison of hardware from different eras. A typical benchmarking run can be launched in a few seconds using the MIT SuperCloud using the triples-mode hierarchical launching system [59] enabling rapid interactive benchmarking. The launch parameters are [Nnode Nppn Ntpn ], corresponding to Nnode nodes, Nppn Matlab processes per node, and Ntpn OpenMP threads per process. The total number
TABLE V S INGLE N ODE A NONYMIZED N ETWORK S ENSING PARAMETERS Number of processes (NP ), number of cores/GPUs per process (NCP P ) and total packets per process NV /NP used for single node benchmarking. Anonymized network sensing benchmark parameters were chosen to balance consistency with the memory capacity of the different hardware. For multiple nodes the parameters highlighted in bold were used. Node Label
NPxNCPP, NV/NP Ncore= 1
2
4
amd-e9
1x1, 227
2x1, 227
4x1, 227
- h100nvl
1x1, 227
2x1, 227
xeon-p8
1x1, 226
2x1, 226
4x1, 226
xeon-g6
1x1, 226
2x1, 226
4x1, 226
- v100
1x1, 226
2x1, 226
xeon-e5
1x1, 226
2x1, 226
4x1, 226
7
8
28
40
8x1, 227
48 8x6, 227
4x12, 226 8x1, 226
7x1, 226
8x5, 226
7x4, 226
Construction
Token Operations per Second
1.00E+09 109
GPU node
1.00E+08 108
CPU node 180x
1.00E+07 107
30x
1.00E+06 106
CPU core
1.00E+05 105
2010
2015
2020
2025
2030
Hardware Year 1.00E+10 1010
1.00E+09 109
1.00E+09 109
GPU node CPU node
1.00E+08 108
Subtraction
Token Operations per Second
Addition
Token Operations per Second
1.00E+10 1010
GPU node CPU node
1.00E+08 108
170x 40x
1.00E+07 107
170x 40x
1.00E+07 107
CPU core 1.00E+06 106
2010
2015
2020
CPU core 2025
2030
1.00E+06 106
2010
2015
Hardware Year Array Multiplication
1.00E+09 109
1.00E+10 1010
GPU node
Token Operations per Second
Token Operations per Second
1.00E+10 1010
190x 35x
1.00E+07 107
2025
2030
Elementwise Multiplication
1.00E+09 109
CPU node
1.00E+08 108
2020
Hardware Year
GPU node CPU node
1.00E+08 108
170x 40x
1.00E+07 107
CPU core
CPU core 1.00E+06 106
2010
2015
2020
Hardware Year
2025
2030
1.00E+06 106
2010
2015
2020
2025
2030
Hardware Year
Fig. 2. Associative (Token) Array Basic Functions. Performance is measured for the different hardware configurations (see Table III). All plots show consistent excellent vertical scaling within a node (approximately linear in number of CPU cores or GPUs) and temporal scaling over multiple eras of hardware.
10,000,000,000,000 1013
GPU node
1,000,000,000,000 1012
109 1.00E+09
100,000,000,000 1011
375x GPU
CPU node
108 1.00E+08
8x
10,000,000,000 1010
• • • • • •
h100nvl v100 amd-e9 xeon-g6 xeon-e5 xeon-p8
1012
10,000,000 107
2025
1014
100,000,000 108
single node
1,000,000 106 2020
1015
1013
CPU core
2015
1016
1,000,000,000 109
107 1.00E+07
106 1.00E+06 2010
1017
2030
Hardware Year
0.01
0.1
1
1011
multi node 10
Number of CPUs or GPUs
Bits per Second
Anonymized Network Sensing
Packets Analyzed per Second
Packets Analyzed per Second
1010 1.00E+10
1010 100
1000
Fig. 3. Anonymized Network Sensing Challenge Temporal, Vertical, and Horizontal Scaling. Performance is measured for the different hardware configurations (see Table III) using the random data sets available from GraphBLAS.org. All plots show consistent excellent vertical scaling within a node (approximately linear in number of CPU cores or GPUs up to memory limits) and temporal scaling over multiple eras of hardware (left panel). Likewise, the horizontal scaling is linear with the number of CPUs or GPUs (right panel). Note: for CPUs a single socket has a value of 1 on the x-axis; using a single core on a 24 core CPU would be 1/24 = 0.041 of a CPU.
of processes is given by NP = Nnode Nppn . On each node, each of the Nppn processes and their corresponding Ntpn threads were pinned to adjacent cores to minimize interprocess contention and maximize cache locality [60]. Within each Matlab process, OpenMP parallelism is used as provided by the underling math libraries. The number hardware cores/GPUs per process is given by NCP P . NCP P = Ntpn on CPUs and NCP P = 1 on GPUs. Triples mode makes it easy to explore horizontal scaling across nodes, vertical scaling by examining combinations of processes and threads on a node, and temporal scaling by running on diverse hardware from different eras. The computing hardware consists of many different types of nodes spanning over a decade (see Table III). The MIT SuperCloud maintains the same modern software stack across all nodes, which allows for direct comparison of hardware performance differences. The parameters used to benchmark the associative array Basic Functions are shown in Table IV. These benchmark parameters were chosen to balance consistency with the memory capacity of the different hardware. In general, the associative arrays sizes N were 1019 , 1020 , or 1021 and the average nonempty entries per row/column k was 8 or 16. The memory intensive nature of these operations meant that the optimal whole node performance is achieved by running a number of processes equal to the number of cores or GPUs on the node. The parameters used to benchmark the associative array implementation of the Anonymized Network Sensing challenge are shown in Table V. These benchmark parameters were chosen to balance consistency with the memory capacity of the different hardware. The Anonymized Network Sensing challenge data sets are large and smaller memory nodes could only run a few instances simultaneously. The number of processes per node ranged were 1, 2, 3, 4, 7, or 8. In general, the packets analyzed per process NV /NP were 1026 or 1027 . For multi-node benchmarks the values highlighted in bold in Table V were used.
VIII. P ERFORMANCE R ESULTS Figure 2 shows the within node (vertical scaling) and temporal scaling (across time) of the Basic Functions for all the different configurations of hardware listed in Table III. All plots show excellent vertical scaling within a node and temporal scaling over multiple eras of hardware. These Basic Function performance results set the foundation for running the Anonymized Network Sensing challenge. Figure 3 (left) shows the within node (vertical scaling) and temporal scaling (across time) of the Anonymized Network Sensing challenge for all the different configurations of hardware listed in Table III. Figure 3 (right) shows the across node (horizontal) and temporal scaling Anonymized Network Sensing challenge, which scales linearly with the number of nodes. These packet per second results are also shown in terms of bits per second using the standard approximation of 1 packet ≈ 10,000 bits. All plots show excellent vertical scaling within a node, horizontal scaling across nodes, and temporal scaling over multiple eras of hardware. Finally, it is worth mentioning that a popular way to report performance in accelerator hardware is via SpFlops (Sparse Flops), which uses dense operation counts when performing sparse operations. In some sense, this gives credit for work that was not done. However, it has also become a popular way to highlight algorithmic improvements in a way that is more broadly appreciated. The Graph Challenge benchmarks have faced this question for all their benchmarks and have chosen to report the more conservative edge based operation counts. These are the values listed in Figures 2 and 3. For comparison, the peak SpFlops numbers are provided in Table VI and highlights the well-known dramatic performance gains from using sparse matrix operations on sparse data. IX. C ONCLUSION The need to analyze networks with the highest regard for privacy is essential to ensure their proper function and is
TABLE VI G IGA S P F LOPS Sparse Flops (SpFlops) equivalent of the best node performance for each benchmark. All units are in Giga. Largest values are in the hundreds of ExaSpFlops. Highlights the well-known dramatic performance gains of sparse matrix operations on sparse data. Benchmark xeon-g6 xeon-p8 amd-e9254 v100 h100nvl Constuct 532 498 1,240 3,876 12,916 Plus 3,129 3,300 25,803 8,980 104,318 Minus 2,415 2,506 22,609 8,672 100,775 Times 6,095 5,622 27,142 19,333 112,329 MatMul 1,589,137,900 1,510,271,563 68,849,757,411 10,391,243,876 366,068,652,573 AnonNetSense 24,181,370,389 12,145,394,560 17,909,144,186 281,593,137,894 862,723,041,517
becoming important as this critical infrastructure expands. The ability to handle diverse data is key requirement as the network layers use different representations of sources and destinations that can be any combination of physical, logical, or persona/agentic endpoints. Associative (token) arrays naturally encompass diverse data while still providing scalable mathematical capability. The large scale of modern networks suggest accelerating these libraries with GPUs can be beneficial. The MIT/IEEE/Amazon Anonymized Network Sensing Graph Challenge provides a venue for highlighting the applicability of accelerated associative (token) arrays for these types of problems. This work benchmarks a prototype Matlab D4M GPU accelerated implementation of the Anonymized Network Sensing challenge across a wide range of CPU and GPU hardware. Scalable performance is demonstrated within and across CPU cores, CPU nodes, and GPU nodes. Horizontal scaling across multiple nodes was linear. Running on hundreds of GPU nodes simultaneously achieved a sustained processing rate sufficient to potentially analyze a 10 Petabit/s network. ACKNOWLEDGEMENT The authors wish to acknowledge the following individuals for their contributions and support: Guillermo Morales, S. Atkins, B. Bond, M. Cafarella, B. Cashman, K Claffy, B. Cook, C. Demchak, A. Edelman, P. Fisher, J. Gottschalk, T. Hardjono, C. Hill, C. Leiserson, S. Madden, K. Malvey, C. Milner, S. Mohindra, H. Perry, S. Pisharody, C. Prothmann, S. Rejto, S. Ruppel, D. Rus, P. Saunders, M. Sherman, S. Somin, S. Van Broekhoven, S. Weed. R EFERENCES [1] J. Kepner, J. Bernays, S. Buckley, K. Cho, C. Conrad, L. Daigle, K. Erhardt, V. Gadepally, B. Greene, M. Jones, R. Knake, B. Maggs, P. Michaleas, C. Meiners, A. Morris, A. Pentland, S. Pisharody, S. Powazek, A. Prout, P. Reiner, K. Suzuki, K. Takhashi, T. Tauber, L. Walker, and D. Stetson, “Zero botnets: An observe-pursue-counter approach.” Belfer Center Reports, 6 2021. [2] S. Pisharody, J. Bernays, V. Gadepally, M. Jones, J. Kepner, C. Meiners, P. Michaleas, A. Tse, and D. Stetson, “Realizing forward defense in the cyber domain,” in 2021 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–7, IEEE, 2021. [3] A. Pentland, “Building a new economy: data, ai, and web3,” Communications of the ACM, vol. 65, no. 12, pp. 27–29, 2022. [4] S. Atkins and C. Lawson, “An improvised patchwork: success and failure in cybersecurity policy for critical infrastructure,” Public Administration Review, vol. 81, no. 5, pp. 847–861, 2021. [5] S. Atkins and C. Lawson, “Cooperation amidst competition: cybersecurity partnership in the us financial services sector,” Journal of Cybersecurity, vol. 7, no. 1, 2021.
[6] C. Demchak, “Achieving systemic resilience in a great systems conflict era,” The Cyber Defense Review, vol. 6, no. 2, pp. 51–70, 2021. [7] S. Weed, “Beyond zero trust: Reclaiming blue cyberspace,” Master’s thesis, United States Army War College, 2022. [8] S. Atkins and C. Lawson, “Beyond zero trust: Reclaiming blue cyberspace with ai,” Cyber Defense Review, vol. 7, no. 1, 2023. [9] J. Kepner, H. Jananthan, M. Jones, W. Arcand, D. Bestor, W. Bergeron, D. Burrill, A. Buluc, C. Byun, T. Davis, V. Gadepally, D. Grant, M. Houle, M. Hubbell, P. Luszczek, L. Milechin, C. Milner, G. Morales, A. Morris, J. Mullen, R. Patel, A. Pentland, S. Pisharody, A. Prout, A. Reuther, A. Rosa, G. Wachman, C. Yee, and P. Michaleas, “What is normal? a big data observational science model of anonymized internet traffic,” in 2024 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–7, 2024. [10] S. Somin, K. Erhardt, T. Cohen, J. Kepner, and A. Pentland, “Temporal fingerprints for identity matching across fully encrypted domains,” Nature Communications, vol. 16, no. 1, pp. 1–11, 2025. [11] J. Kepner and H. Jananthan, Mathematics of Big Data. MIT Press, 2018. [12] J. Kepner, W. Arcand, W. Bergeron, N. Bliss, R. Bond, C. Byun, G. Condon, K. Gregson, M. Hubbell, J. Kurz, et al., “Dynamic distributed dimensional data model (d4m) database and computation system,” in 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5349–5352, IEEE, 2012. [13] A. Chen, A. Edelman, J. Kepner, V. Gadepally, and D. Hutchison, “Julia implementation of the dynamic distributed dimensional data model,” in High Performance Extreme Computing Conference (HPEC), IEEE, 2016. [14] M. Jones, J. Kepner, A. Prout, T. Davis, W. Arcand, D. Bestor, W. Bergeron, C. Byun, V. Gadepally, M. Houle, M. Hubbell, H. Jananthan, A. Klein, L. Milechin, G. Morales, J. Mullen, R. Patel, S. Pisharody, A. Reuther, A. Rosa, S. Samsi, C. Yee, and P. Michaleas, “Deployment of real-time network traffic analysis using graphblas hypersparse matrices and d4m associative arrays,” in 2023 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–8, 2023. [15] D. Quach, H. Jananthan, and J. Kepner, “Integrating python-graphblas into d4m.py,” in 2024 IEEE MIT Undergraduate Research Technology Conference (URTC), pp. 1–6, 2024. [16] H. Jananthan, M. Jones, W. Arcand, D. Bestor, W. Bergeron, D. Burrill, A. Buluc, C. Byun, T. Davis, V. Gadepally, D. Grant, M. Houle, M. Hubbell, P. Luszczek, P. Michaleas, L. Milechin, C. Milner, G. Morales, A. Morris, J. Mullen, R. Patel, A. Pentland, S. Pisharody, A. Prout, A. Reuther, A. Rosa, G. Wachman, C. Yee, and J. Kepner, “Anonymized network sensing graph challenge,” in 2024 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–8, 2024. [17] I. Voloshchuk, H. Jananthan, C. Byun, and J. Kepner, “Improving the graph challenge reference implementation,” in 2025 IEEE MIT Undergraduate Research Technology Conference (URTC), pp. 1–5, 2025. [18] R. Pearce, “Triangle counting for scale-free graphs at scale in distributed memory,” in High Performance Extreme Computing Conference (HPEC), IEEE, 2017. [19] M. M. Wolf, M. Deveci, J. W. Berry, S. D. Hammond, and S. Rajamanickam, “Fast linear algebra-based triangle counting with kokkoskernels,” in High Performance Extreme Computing Conference (HPEC), IEEE, 2017. [20] T. M. Low, D. G. Spampinato, A. Kutuluru, U. Sridhar, D. T. Popovici, F. Franchetti, and S. McMillan, “Linear algebraic formulation of edgecentric k-truss algorithms with adjacency matrices,” in 2018 IEEE High Performance extreme Computing Conference (HPEC), pp. 1–7, 2018. [21] R. Pearce and G. Sanders, “K-truss decomposition for scale-free graphs at scale in distributed memory,” in 2018 IEEE High Performance extreme Computing Conference (HPEC), pp. 1–6, 2018. [22] A. Yaşar, S. Rajamanickam, M. Wolf, J. Berry, and U. V. Çatalyurek, “Fast triangle counting using cilk,” in 2018 IEEE High Performance extreme Computing Conference (HPEC), pp. 1–7, 2018. [23] T. A. Davis, M. Aznaveh, and S. Kolodziej, “Write quick, run fast: Sparse deep neural network in 20 minutes of development time via suitesparse:graphblas,” in 2019 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–6, 2019. [24] S. Pandey, X. S. Li, A. Buluc, J. Xu, and H. Liu, “H-index: Hashindexing for parallel triangle counting on gpus,” in 2019 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–7, 2019. [25] R. Pearce, T. Steil, B. W. Priest, and G. Sanders, “One quadrillion triangles queried on one million processors,” in 2019 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–5, 2019.
[26] A. Yaşar, S. Rajamanickam, J. Berry, M. Wolf, J. S. Young, and U. V. Catalyurek, “Linear algebra-based triangle counting via finegrained tasking on heterogeneous environments : (update on static graph challenge),” in 2019 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–4, 2019. [27] B. W. Priest, A. Dunton, and G. Sanders, “Scaling graph clustering with distributed sketches,” in 2020 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–7, 2020. [28] M. Hidayetoğlu, C. Pearson, V. S. Mailthody, E. Ebrahimi, J. Xiong, R. Nagi, and W.-m. Hwu, “At-scale sparse deep neural network inference with efficient gpu implementation,” in 2020 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–7, 2020. [29] S. Ghosh and M. Halappanavar, “Tric: Distributed-memory triangle counting by exploiting the graph structure,” in 2020 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–6, 2020. [30] D.-L. Lin and T.-W. Huang, “A novel inference algorithm for large sparse neural network using task graph parallelism,” in 2020 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–7, 2020. [31] J. Xin, X. Ye, L. Zheng, Q. Wang, Y. Huang, P. Yao, L. Yu, X. Liao, and H. Jin, “Fast sparse deep neural network inference with flexible spmm optimization space exploration,” in 2021 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–7, 2021. [32] A. J. Uppal, J. Choi, T. B. Rolinger, and H. Howie Huang, “Faster stochastic block partition using aggressive initial merging, compressed representation, and parallelism control,” in 2021 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–7, 2021. [33] Y. Sun, L. Zheng, Q. Wang, X. Ye, Y. Huang, P. Yao, X. Liao, and H. Jin, “Accelerating sparse deep neural network inference using gpu tensor cores,” in 2022 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–7, 2022. [34] S. Xu, M. Wu, L. Zheng, Z. Shao, X. Ye, X. Liao, and H. Jin, “Towards fast gpu-based sparse dnn inference: A hybrid compute model,” in 2022 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–7, 2022. [35] M. Dun, X. Zhang, H. Cao, Y. Zhang, J. Huang, and X. Ye, “Adaptive sparse deep neural network inference on resource-constrained costefficient gpus,” in 2023 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–7, 2023. [36] Z. Wang, Z. Meng, X. Li, X. Lin, L. Zheng, C. Tian, and S. Zhong, “Smog: Accelerating subgraph matching on gpus,” in 2023 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–7, 2023. [37] F. Wanye, V. Gleyzer, E. Kao, and W.-c. Feng, “An integrated approach for accelerating stochastic block partitioning,” in 2023 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–7, 2023. [38] Z. Lin, C. Xu, K. Meng, and G. Tan, “Mercury: Efficient subgraph matching on gpus with hybrid scheduling,” in 2024 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–7, 2024. [39] M. Qin, C. Zhang, Y. Gao, Y. Ding, W. Jiang, W. Zhang, W. Han, and B. Bai, “Towards faster graph partitioning via pre-training and inductive inference,” in 2024 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–7, 2024. [40] D. Chen, Y. Huang, Y. Huang, B. Lin, Y. Zhang, L. Zheng, X. Liao, and H. Jin, “Prism: Practical in-memory acceleration for subgraph matching at scale,” in 2025 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–7, 2025. [41] M. Dun, J. Zhou, H. Cao, S. Song, Y. Sun, M. Yan, and X. Ye, “Towards efficient sparse deep neural network inference via multi-level concurrency orchestration,” in 2025 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–7, 2025. [42] Z. Han, A. Briasco-Stewart, M. Zink, and M. Leeser, “Extracting tcpip headers at high speed for the anonymized network traffic graph challenge,” in 2024 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–6, 2024. [43] S. Lockton, J. Kepner, M. Stonebraker, H. Jananthan, L. Anderson, W. Arcand, D. Bestor, W. Bergeron, A. Bonn, D. Burrill, C. Byun, T. Davis, V. Gadepally, M. Houle, M. Hubbell, M. Jones, P. Luszczek, P. Michaleas, L. Milechin, C. Milner, G. Morales, J. Mullen, M. Pelletier, A. Poliakov, A. Prout, A. Reuther, A. Rosa, C. Yee, and A. Pentland, “Dbos network sensing: A web services approach to collaborative awareness,” in 2025 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–8, 2025. [44] M. Mandulak, S. Ghosh, S. M. Ferdous, M. Halappanavar, and G. Slota, “Anonymized network sensing using c++26 std::execution on gpus,” in
2025 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–7, 2025. [45] C. Milner, M. Houle, H. Jananthan, M. Jones, J. Kepner, P. Michaleas, I. Voloshchuk, and A. Pentland, “Interactive trillion packet anonymized network analysis with the graphblas,” in 2025 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–6, 2025. [46] S. Samsi, D. Campbell, E. Scoullos, and O. Green, “Combining performance and productivity: Accelerating the network sensing graph challenge with gpus and commodity data science software,” in 2025 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–7, 2025. [47] J. Wang, W. Tang, C. Shi, Z. Zhang, D. Chen, M. Chen, and W. Xiao, “Sanst: Sensing anonymized network via sorted triplets,” in 2025 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–7, 2025. [48] J. Kepner, K. Cho, K. Claffy, V. Gadepally, P. Michaleas, and L. Milechin, “Hypersparse neural network analysis of large-scale internet traffic,” in 2019 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–11, 2019. [49] J. Karvanen and A. Cichocki, “Measuring sparseness of noisy signals,” in 4th International Symposium on Independent Component Analysis and Blind Signal Separation, pp. 125–130, 2003. [50] J. Kepner, C. Meiners, C. Byun, S. McGuire, T. Davis, W. Arcand, J. Bernays, D. Bestor, W. Bergeron, V. Gadepally, R. Harnasch, M. Hubbell, M. Houle, M. Jones, A. Kirby, A. Klein, L. Milechin, J. Mullen, A. Prout, A. Reuther, A. Rosa, S. Samsi, D. Stetson, A. Tse, C. Yee, and P. Michaleas, “Multi-temporal analysis and scaling relations of 100,000,000,000 network packets,” in 2020 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–6, 2020. [51] A. Soule, A. Nucci, R. Cruz, E. Leonardi, and N. Taft, “How to identify and estimate the largest traffic matrix elements in a dynamic environment,” in ACM SIGMETRICS Performance Evaluation Review, vol. 32, pp. 73–84, ACM, 2004. [52] Y. Zhang, M. Roughan, C. Lund, and D. L. Donoho, “Estimating pointto-point and point-to-multipoint traffic matrices: an information-theoretic approach,” IEEE/ACM Transactions on Networking (TON), vol. 13, no. 5, pp. 947–960, 2005. [53] P. J. Mucha, T. Richardson, K. Macon, M. A. Porter, and J.-P. Onnela, “Community structure in time-dependent, multiscale, and multiplex networks,” science, vol. 328, no. 5980, pp. 876–878, 2010. [54] P. Tune, M. Roughan, H. Haddadi, and O. Bonaventure, “Internet traffic matrices: A primer,” Recent Advances in Networking, vol. 1, pp. 1–56, 2013. [55] J. Nair, A. Wierman, and B. Zwart, “The fundamentals of heavy tails: Properties, emergence, and estimation,” Preprint, California Institute of Technology, 2020. [56] J. Kepner, K. Cho, K. Claffy, V. Gadepally, S. McGuire, L. Milechin, W. Arcand, D. Bestor, W. Bergeron, C. Byun, M. Hubbell, M. Houle, M. Jones, A. Prout, A. Reuther, A. Rosa, S. Samsi, C. Yee, and P. Michaleas, “New phenomena in large-scale internet traffic,” in Massive Graph Analytics (D. Bader, ed.), pp. 1–53, Chapman and Hall/CRC, 2022. [57] T. South, S. Nagabhushanaradhya, A. Dissanayaka, S. Cecchetti, G. Fletcher, V. Lu, A. Pietropaolo, D. H. Saxe, J. Lombardo, A. M. Shivalingaiah, S. Bounev, A. Keisner, A. Kesselman, Z. Proser, G. Fahs, A. Bunyea, B. Moskowitz, A. Tulshibagwale, D. Greenwood, J. Pei, and A. Pentland, “Identity management for agentic ai: The new frontier of authorization, authentication, and security for an ai agent world,” 2025. [58] V. Gadepally, J. Kepner, L. Milechin, W. Arcand, D. Bestor, B. Bergeron, C. Byun, M. Hubbell, M. Houle, M. Jones, et al., “Hyperscaling internet graph analysis with d4m on the mit supercloud,” in 2018 IEEE High Performance extreme Computing Conference (HPEC), pp. 1–6, IEEE, 2018. [59] A. Reuther, J. Kepner, C. Byun, S. Samsi, W. Arcand, D. Bestor, B. Bergeron, V. Gadepally, M. Houle, M. Hubbell, M. Jones, A. Klein, L. Milechin, J. Mullen, A. Prout, A. Rosa, C. Yee, and P. Michaleas, “Interactive supercomputing on 40,000 cores for machine learning and data analysis,” in 2018 IEEE High Performance extreme Computing Conference (HPEC), pp. 1–6, Sep. 2018. [60] C. Byun, J. Kepner, W. Arcand, D. Bestor, W. Bergeron, M. Hubbell, V. Gadepally, M. Houle, M. Jones, A. Klein, L. Milechin, P. Michaleas, J. Mullen, A. Prout, A. Rosa, S. Samsi, C. Yee, and A. Reuther, “Optimizing xeon phi for interactive data analysis,” in 2019 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–6, 2019.