arXiv CSopen access

Task-Centered Benchmark for Interactive Network Visualization & Analysis

Task-Centered Benchmark for Interactive Network Visualization & Analysis Experiments & Analysis Ameya Patil

University of Washington Seattle, USA [email protected]

arXiv:2607.03725v1 [cs.DB] 4 Jul 2026

Ishan Sinha

University of Washington Seattle, USA [email protected]

ABSTRACT Interactive network visualization and analysis (INVA) enables iterative, visual and algorithmic analysis of large network datasets. Although numerous benchmarks have been developed to evaluate different graph analysis algorithms and systems, we observe a lack of such efforts for interactive network data understanding. In this work, we address the question - How well do existing graph systems serve the purpose of Interactive Network Visualization and Analysis? To this end, we build and demonstrate the use of the first task-centered benchmarking framework to evaluate a variety of graph system backends on INVA workloads. Our benchmarking results highlight a gap between both the capabilities and performance of existing graph systems for INVA use cases, and uncover possible bugs in these systems. Based on our benchmarking results, we reveal new opportunities for research and development to better support interactive network visualization and analysis. PVLDB Reference Format: Ameya Patil, Wei Jun Tan, Ishan Sinha, and Leilani Battle. Task-Centered Benchmark for Interactive Network Visualization & Analysis. PVLDB, 14(1): XXX-XXX, 2020. doi:XX.XX/XXX.XX PVLDB Artifact Availability: The source code, data, and/or other artifacts have been made available at https://github.com/WeiJun428/graph-system-benchmark and https://osf.io/3z4hk

1

INTRODUCTION

The ever-growing importance of large-scale network data has sparked increased interest in visualizing and analyzing large networks [10]. In response, many graph algorithms [36, 51, 59, 63], graph processing frameworks and systems [32, 33, 53, 65] have been developed This work is licensed under the Creative Commons BY-NC-ND 4.0 International License. Visit https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy of this license. For any use beyond those covered by this license, obtain permission by emailing [email protected]. Copyright is held by the owner/author(s). Publication rights licensed to the VLDB Endowment. Proceedings of the VLDB Endowment, Vol. 14, No. 1 ISSN 2150-8097. doi:XX.XX/XXX.XX

Wei Jun Tan

University of Washington Seattle, USA [email protected]

Leilani Battle

University of Washington Seattle, USA [email protected] to support large graph analytics. However, these solutions are predominantly opaque to domain experts or non-programmers, and fail to account for human-in-the-loop or interactive understanding of network data. In this work, we focus on a relatively understudied aspect of network data understanding: interactive network visualization and analysis (INVA). INVA facilitates quick exploration and sensemaking of large complex network datasets including critical tasks across the network analysis pipeline such as data import, visualization and algorithmic processing, and exporting visualizations for information dissemination, commonly performed through intuitive graphical user interfaces [4]. As has been shown in prior work in benchmarking for relational data [8, 9, 17], interactive visualization and analysis workloads can trigger distinct queries and bursty frequencies, which existing graph analytics benchmarks do not simulate [18, 25]. Individual graph algorithms used within such workloads have been evaluated for efficacy, speed and resource utilization only in a standalone manner [16, 20, 37, 43, 50, 51, 63], and not in the context of INVA. Thus there is a need to evaluate graph systems comparatively in the context of human-in-the-loop or interactive network visualization and analysis scenarios. In this work, we answer the overarching question - how well do existing graph systems support interactive network visualization and analysis (INVA)? Towards this end, we (1) develop a task-centered model of how analysts perform INVA to generate representative INVA workloads, and (2) design and implement a task-centered benchmarking framework to measure how well existing graph systems perform on these INVA workloads over large-scale networks. We demonstrate the use of our benchmarking framework by evaluating the backends of a set of prominent INVA and graph database systems. Through the results of the benchmarking evaluation, we reveal a gap both in the kind of analyses supported by these systems, and their performance for INVA use cases, while also exposing correctness issues in some of our evaluated graph systems. We conclude with a discussion of new opportunities for research and development in Interactive Network Visualization and Analysis. Our benchmark redefines graph system performance in the context of human-in-the-loop scenarios, shifting away from abstract complexity measures and towards measuring how these systems

advance (or hinder) human insight. To the best of our knowledge, our work is the first to build a comprehensive benchmark for human-in-the-loop network data understanding. We view this work as an initial performance study over network data understanding systems for large networks, demonstrating the value of developing benchmarks for INVA, which can help researchers develop and evaluate new graph visualization and analysis systems. In summary, we make the following contributions: (1) We develop a task-centered model of Interactive Network Visualization and Analysis (INVA) to generate representative INVA workloads. (2) We design and implement a task-centered benchmarking framework to evaluate network data understanding systems for INVA workloads on large-scale network data. (3) We present results from benchmarking prominent INVA and graph database systems supporting INVA use cases, which reveal new avenues for future database research. For the rest of the paper, we adopt the same terminology as [4] where ‘network’ refers to real-life data which carries with it the context or semantics of the data, and ‘graph’ refers to the mathematical model or abstraction that is used to enable analysis of network data. A network has nodes and links, while a graph has vertices and edges.

2

2.1

Base Terminologies

We define the following basic terms in the context of INVA to build upon further and describe the benchmarking framework: 2.1.1 Dataset (𝐷). Network data having a set of nodes 𝑁 and a set of links 𝐿, with attribute sets 𝑁 𝑎 and 𝐿𝑎 respectively. A dataset may also be expressed using the relational model as 𝑁 (𝑛𝑎 1, 𝑛𝑎 2, ..., 𝑛𝑎𝑝 ) and 𝐿(𝑙𝑎 1, 𝑙𝑎 2, ..., 𝑙𝑎𝑞 ), where 𝑛𝑎𝑖 ∈ 𝑁 𝑎 and 𝑙𝑎𝑖 ∈ 𝐿𝑎 are the node and link attributes respectively. 2.1.2 Interaction (𝑖). An analysis or visualization operation performed on the dataset 𝐷, optionally taking in arguments, and generating a result, a set of results, or a visualization as output. 2.1.3 Workflow (𝑤). The dataset 𝐷 or its subset, and a sequence of interactions 𝐼 =< 𝑖 1, 𝑖 2, ...𝑖𝑛 > performed on the data, representing an analysis approach or goal (blue box in fig. 2). 2.1.4 Workload (𝑊 ). A sequence of workflows < 𝑤 1, 𝑤 2, ...𝑤𝑚 > where each workflow represents a potentially different analysis approach or goal (gray box in fig. 2). The entire workload thus represents different approaches of understanding the data.

2.2

Workload Generator Module

An important aspect of benchmarking is the creation of workloads on which to test the systems. Prior work in benchmarking for interactive analytics [9, 17] performed user studies to create a model of interactive analysis of relational data and generate corresponding workloads. However, such models are better informed by the analyses performed by domain experts rather than by crowdsourced users [24, 64]. Thus, instead of resorting to crowdsourced studies, we build our model of INVA workloads (section 2.2.1) based on prior research in graph task taxonomies [4, 21, 43], and graph analysis use cases by domain experts [5, 11–13, 22, 27, 29, 44, 61, 62, 66, 67]. We generate our workloads using this INVA model (section 2.2.2). We first present our INVA model which captures how humans visualize and analyze network data to understand it, and then describe our INVA workload generator (fig. 1a) which uses this model.

BENCHMARKING FRAMEWORK

Prior benchmarking work for graph systems has primarily focused on graph databases and graph processing systems for individual graph algorithms run in a standalone manner (section 5.1). Our aim is to address the lesser explored needs of human analysts of network data, who perform a sequence of graph operations to understand the network data. We thus present our task-centered benchmarking framework for interactive network visualization and analysis (INVA) of large-scale networks. To the best of our knowledge, this is the first benchmark addressing systems for human-in-the-loop understanding of network data. Our benchmarking framework is designed based on prior work in graph task taxonomies, graph benchmarking and benchmarking for interactive data analytics, as described in section 5. It consists of 3 modules: (a) the workload generator, (b) the data generator, and (c) the benchmark driver, of which the workload generator module is the main contribution. Figure 1 shows how these individual modules work together in the framework. We now describe some terminologies used in our framework followed by the design and implementation of each module.

Workload Workflow Data Scope - WHAT to analyze? 1. Data points

2. Data attributes

Select and filter by Graph Topology / Attributes

Select node or edge attributes

x1

Interaction (Analysis Scope - HOW to analyze?)

b

Network Dataset (D) Data Generator

c

Benchmark Driver

1. Analysis Focus (high-level intent)

2. Analysis Task (genre)

3. System Operation (executable solution)

Graph / Attributes / Comparison

As per analysis focus

As per analysis focus and task

xn

xm

Dataset Specification Logger Transition Matrix (M)

a

Workload Generator

Test System

Figure 2: Characterizing INVA workflow and workload. Each workflow has a data scope and n interactions. m such workflows are concatenated to form a workload.

Workload (W)

2.2.1 INVA Model. To generate interactive network visualization and analysis workloads for benchmarking purposes, we first need a formal model of interactive network visualization and analysis,

Figure 1: Architecture of the benchmarking framework 2

(1) Graph Focus: Interactions with a graph focus are intended to understand the topology of the network, and require the use of information about both the nodes and links. We have the following analysis tasks under graph focus: (a) Connections: Tasks for moving between nodes e.g., shortest paths, network diffusion/flow, spanning trees, etc. (b) Group: Tasks for detecting or working with groups of nodes e.g., clusters, connected components, k-core, pattern matching etc. (c) Metrics: Tasks for computing topology based node/link properties e.g., centrality measures, local clustering coefficient etc. (d) Summary: Tasks for computing statistics which summarize the topology e.g., triadic census, average path length, graph diameter etc. (e) Visualize: Tasks for visualizing topology e.g., graph layout (node-link) or node-ordering (adjacency matrix) (f) Edit: Tasks for editing the network using both the nodes and links information e.g., add/delete node/link, add new node/link attribute using adjacent link/node information etc. (2) Attribute Focus: Interactions with an attribute focus are intended to understand the attributes of the network, and require the use of either the node attribute data or the link attribute data, but not both together. It consists of the following analysis tasks: (a) Edit: Tasks for editing either the node or link information in the network (b) Compute: Tasks for computing attribute based node/link properties e.g., global aggregation of node/link attributes, transformation of node/link attributes, count etc. (3) Comparison Focus: Interactions with a comparison focus are intended to compare attributes for different nodes/links, or nodes/links with certain attribute values. This focus contains the following analysis tasks: (a) Encode: Tasks for encoding any node/link attribute in the graph visualization e.g., encode in color, size etc. (b) Sort: Tasks for sorting nodes/links based on attribute values (c) Correlate: Tasks for gauging correlation between attributes e.g., creating scatter plots, statistical tests etc. (d) Extrema: Tasks for gauging the distribution of node/link attributes e.g., min/max value, outliers, histograms etc.

to characterize how humans analyze network data. We begin by defining two terms in this regard. (1) Data Scope: The subset 𝐷 𝑗 of the network data, being analyzed during a workflow 𝑤 𝑗 , such that all the interactions within the workflow are focused on this subset (fig. 2 yellow box). It is characterized by (1) the data points (𝑑𝑝) i.e., a subset of the nodes/links, or rows of the node/link table, and (2) the data attributes (𝑑𝑎) i.e., a subset of the node/link attributes, or columns of the node/link table. Thus, 𝐷 𝑗 ⊆ 𝐷 for workflow 𝑤 𝑗 𝑑𝑝 (𝐷 𝑗 ) = (𝑁 𝑗 , 𝐿 𝑗 ), where 𝑁 𝑗 ⊆ 𝑁 & 𝐿 𝑗 ⊆ 𝐿 𝑑𝑎(𝐷 𝑗 ) = (𝑁 𝑎 𝑗 , 𝐿𝑎 𝑗 ) where 𝑁 𝑎 𝑗 ⊆ 𝑁𝑎 & 𝐿𝑎 𝑗 ⊆ 𝐿𝑎 (2) Analysis Scope: The characterization of an interaction performed within a workflow (fig. 2 green box). This characterization is done in a hierarchical manner moving from a high-level intent behind performing the analysis (analysis focus) to a low-level executable solution for the analysis (system operation). Analysis Focus (𝑎𝑓 ): A high-level intent of analyzing the data scope during each interaction in a workflow. Analysis Task (𝑎𝑡): A genre of analysis performed under each analysis focus, where each genre corresponds to a semantic grouping of analysis types. This semantic grouping helps us simulate a realistic and semantically meaningful INVA workload, as detailed in section 2.2.2. System Operation (𝑜𝑝): An executable solution which facilitates performing the analysis task. This represents the lowest level of abstraction in our INVA model, in terms of which we issue commands to the test system for benchmarking. Thus, an interaction is expressed as 𝑖 = {𝑎𝑓 , 𝑎𝑡, 𝑜𝑝} We refer to prior work in graph task taxonomy (section 5.2) for the different kinds of analysis foci, analysis tasks and system operations. One of the earliest graph task taxonomies by Lee et al. [31] categorizes graph tasks as (1) Topology based: adjacency, accessibility, common connection, connectivity; (2) Attribute based: filtering, distribution, range, compute derived attributes; (3) Browsing and (4) Overview. Nobre et al. [43] concisely taxonomize graph analysis tasks as analyzing either the topology, or the attributes of the network data. We use these two graph task taxonomies along with the survey of operational INVA tasks by [4] to create our hierarchical INVA model containing three different analysis foci, along with the constituent analysis tasks for each focus, and the constituent system operations within each analysis task. The hierarchical nature of our model is inspired from the work of Gathani et al. [21] following which, we map low level system operations to each high level analysis focus. Thus through our INVA model, we provide a tangible use case for graph task taxonomies and grammars, thereby proving their utility. We now describe these analysis foci and the constituent analysis tasks, along with some example system operations under each analysis task. Table 1b lists only those system operations which are currently supported in our benchmarking framework.

Apart from identifying different semantic genres of INVA, the analysis foci also separate different INVA genres based on data access patterns. Graph focus accesses both node and link data, while attribute and comparison foci access either node or link data. 2.2.2 INVA Workload Generator. We now detail the implementation of the workload generator built using our INVA model. Algorithm 1 shows the pseudocode for the workload generation. Each workflow 𝑤 𝑗 is synthesized by choosing (1) a data scope 𝐷 𝑗 , and (2) a sequence of interactions 𝐼 𝑗 pertaining to the data scope, in a top-down manner as per the hierarchy of our INVA model. (1) Data Scope Selection: (lines 3-10) The data scope 𝐷 𝑗 is chosen in two steps: 3

Table 1: Components of the INVA model (a) Data Scope, and (b) Analysis Scope. Each column represents a dimension along which we characterize the respective scope, while each row represents the possible options to characterize a scope along the concerned dimension.

|𝑁 𝑗 −1 |) (line 3). Analysts are less likely to drill down deeper into an already reduced dataset. In both the choice of the Select operation and the data attributes, we use random selection. The use of more sophisticated selection techniques like choosing a data scope as per the exploration goals of the analyst [46] is discussed in future work section 4.1. Table 1a summarizes the possible characterizations of the data scope currently supported in our benchmarking framework.

(a) Data Scope - What to analyze? 1. Data Points (𝑑𝑝 ) (rows)

2. Data Attributes (𝑑𝑎) (columns)

Select (𝜎 ) (k-hop neighborhood subgraph / Node attribute filter / Link attribute filter) None (entire network is being analyzed)

(2) Interaction Sequence Generation: (lines 13-26) We generate interaction sequences for a workflow such that each new interaction has a semantic dependency on both the previous interaction in the workflow, and the data scope of the workflow. Thus,

Project (Π ) (Node attributes / Link attributes)

𝑖𝑘 = 𝑓 (𝑖𝑘 −1, 𝐷 𝑗 ), for workflow 𝑤 𝑗

(b) Analysis Scope - How to analyze? 1. Analysis Focus (𝑎𝑓 )

Graph Focus

Attributes Focus Comparison Focus

2. Analysis Task (𝑎𝑡 )

3. System Operations (𝑜𝑝 )

Connections

Path Existence Check / Shortest Path Finding / Single Source Shortest Path

Group

Graph Clustering / Connected Components

Metrics

Centrality Measures / Clustering Coefficient

Summary

Triadic Census / Graph Diameter / Avg Path Length / Global Clustering Coefficient

Visualize

Graph Layout (node-link) / Node Ordering (adj matrix)

Edit

Update nodes/links / Add node/link / Delete node/link

Edit

Update node/link

Compute

Custom Computation / Count

Encode

Encode attribute channel / Toggle Labels

Sort

Sort node/link attribute

Correlate

Statistical Test / Scatterplot

Extrema

Minimum / Maximum / Distribution / Outliers

We model the dependency on the previous interaction using a Markov model with a transition probability matrix (𝑀) to choose the next interaction given the previous interaction. Using this matrix, each interaction in the interaction sequence is generated as per the hierarchy of our proposed INVA model. First, the analysis focus is generated, followed by the analysis task and finally the system operation (lines 16-23). Workloads are created by concatenating multiple workflows (line 29). We make some simplifying assumptions in configuring the transition matrix to avoid combinatorial explosion and make the transitions tractable. However, we keep this transition matrix configurable to generate a diverse variety of workloads. The dependency of an interaction on the data scope (line 23) is implemented by sampling the system operation also considering the data attributes in the data scope. Unlike prior graph benchmarks which have a fixed set of operations to evaluate the test systems on, our benchmark creates INVA workloads considering the supported interactions common to all the test systems which are being evaluated ([𝑡𝑠 ]) (line 23). This enables a fair comparison when creating workloads to evaluate test systems supporting different sets of interactions. We implemented the workload generator in Python using NetworKit library [56] for graph focus interactions and Dask library [15] for attribute and comparison foci interactions.

• Data points i.e., the nodes/links to be analyzed, are chosen by performing a Select (𝜎) operation (line 5) on the dataset. This includes using attribute filters for nodes or links, or fetching the k-hop neighborhood of certain nodes of choice. The result includes the induced subgraph of the filteredin nodes/links. When no Select operation is performed to choose the data points (𝜎 = none), the entire network 𝐷 is being analyzed in the workflow. • Data attributes i.e., the node/link attributes to be analyzed in a data scope (equivalent to the Project (Π) operation) are chosen from the set of both the node and link attributes (line 6). We sample at least 2 node/link attributes.

2.2.3 Representativeness of the INVA Model. We studied prior work employing INVA workflows [5, 11–13, 22, 27, 29, 44, 61, 62, 66, 67] to create our INVA model which enables the creation of workloads representative of how human analysts analyze network data. Analysts analyze the data with a certain analysis goal, and they begin by choosing a subset of the data or attributes to analyze, i.e., setting a data scope (fig. 2 yellow box). This is followed by a sequence of interactions performed on the data scope as per the analysis goal (fig. 2 green box). Each interaction is conceived in terms of what property of the network is to be analyzed - topology, attributes or comparison between the two, i.e., the analysis focus. The idea of analysis task gives more details about the kind of analysis to be performed for the interaction. Finally, the system operation provides an executable means to the end. We thus compose one workflow (fig. 2 blue box). Analysts may repeat the entire procedure multiple times with different analysis goals. Our model enables simulating this by generating multiple workflows and concatenating them to get a workload (fig. 2 gray box). We keep the

Analysts may sometimes want to drill down into a subgraph of the previous data scope by applying more filters, or analyze a different set of attributes for the previous data scope. Thus, the data scope from the previous workflow may be reused to perform the Select and Project operations for the next workflow. We simulate this scenario by assigning probabilities to the choice of the original data 𝐷 (or previous data scope 𝐷 𝑗 −1 ), proportional to the node count of the filtered out data |𝑁 | − |𝑁 𝑗 −1 | (or previous data scope 4

Algorithm 1 Workload Generation Pseudocode

2.3

Input: (1) Network Data (𝐷 (𝑁 , 𝐿)) (2) Transition Probability Matrix (𝑀) (3) Workload length (𝑚) (4) Test systems ([𝑡𝑠 ]) Output: Workload 𝑊 = ⟨𝑤 1, 𝑤 2, . . . , 𝑤𝑚 ⟩ 1: while 𝑊 .length < 𝑚 do 2: // setting the data scope 𝐷 𝑗 for workflow 𝑤 𝑗 3: 𝐷𝑠 ← WeightedSample({𝐷, 𝐷 𝑗 −1 }) 4: 𝜎 ∼ U ( [node_filter, link_filter, k-hop_filter, none]) 5: 𝑑𝑝 (𝐷 𝑗 ) ← 𝜎 (𝐷𝑠 ) 6: 𝑑𝑎(𝐷 𝑗 ) ∼ U (𝑁 𝑎 ∪ 𝐿𝑎, 2)

Prior works in network data generation either do not support node/link attribute generation [18, 25], or support limited connectivity patterns or require expensive hardware and infrastructure [1, 14, 49]. We address these limitations in our own implementation of the network data generator module. Our data generator (fig. 1b) enables creating synthetic network datasets using a specification for the node and link attributes, the distribution of these attribute values, and the connectivity properties of the network—degree distribution, link probability and node:link ratio. Workloads can be created, and INVA systems can be evaluated on these synthetic datasets as an alternative to realworld datasets. We implemented the data generator in Python using the NetworKit network analysis library [56].

7: 8: 9: 10: 11: 12: 13: 14: 15: 16:

if |𝑁 𝑗 | = 0 then continue // reject empty data scope end if

2.4

18: 20:

// generating interaction sequence 𝐼 𝑗 for workflow 𝑤 𝑗 𝐼 𝑗 ← ⟨⟩ 𝑘 ← 0 // interaction_counter while true do 𝑖𝑘 .𝑎𝑓 ← 𝑀 (𝑖𝑘 −1 .𝑎𝑓 , 𝑘) if 𝑖𝑘 .𝑎𝑓 = ’EOW’ then break // encountered ’end of workflow’ token end if

3

22: 24: 25: 26:

EVALUATION

In this section, we describe the evaluation metrics, the setup used to demonstrate our benchmarking framework, and present benchmarking results for the backends of three types of graph systems as per the categorization presented in [4] - INVA systems, graph databases (GDB) and graph scripting libraries (GSL). Our main focus is on evaluating graph systems for INVA use cases which are more human-oriented. Nevertheless, the underlying system performance is also crucial. Thus, we consider two types of metrics: Performance metrics, and Expressiveness metrics. We describe the performance metrics, which are more traditional or system-oriented in the following subsection, and expressiveness metrics which are more human user or developer oriented, along with the corresponding results in section 3.4.

21: 23:

Benchmark Driver

The benchmark driver module (fig. 1c) serves as the interface between the benchmarking framework and the test system. It serves three purposes (1) provides a blueprint for test system interface development, (2) scaffolds running the workload on the test system, and (3) logs the results for offline analysis. We refer our readers to the supplementary material for more details about this module. We implemented the benchmark driver module in Python.

17: 19:

Data Generator Module

𝑖𝑘 .𝑎𝑡 ← 𝑀 (𝑖𝑘 −1 .𝑎𝑓 , 𝑖𝑘 .𝑎𝑓 ) 𝑖𝑘 .𝑜𝑝 ← System_Operations(𝑡𝑠 , 𝑖𝑘 .𝑎𝑓 , 𝑖𝑘 .𝑎𝑡, 𝑑𝑎(𝐷 𝑗 )) 𝐼 𝑗 .add(𝑖𝑘 ) 𝑘 ←𝑘 +1 end while

27: 28:

𝑤 𝑗 ← (𝐷 𝑗 , 𝐼 𝑗 ) 𝑊 .add(𝑤 𝑗 ) // concatenating workflows to create workload 30: end while 31: return 𝑊

29:

3.1

transition probability matrix configurable to enable modeling different INVA behaviors, but also provide a default transition matrix based on our observations from prior work employing INVA workflows. Our INVA model can also accommodate new interactions in a semantic way as per their focus and task category. Our approach of generating INVA workflows (data scope + interaction sequence) is also corroborated by the kind of analysis constructs, certain tools like the Neo4j Graph Data Science [40] plugin facilitate. Section 4.1 describes future work to validate and further improve the workload generator. In its current state however, our generator can be used to generate workloads so that INVA systems can be tested for preliminary performance results, before evaluating them more thoroughly with human network analysts. These preliminary results can then be used to guide the research and development of new systems for INVA. Our model is not meant to substitute human network analysts, but rather to generate representative INVA workloads in order to test graph systems.

Performance Metrics

3.1.1 Scalability Limit. The upper limit on dataset size beyond which the test system fails to execute the workload. The failure could be due to any reason e.g., out-of-memory (OOM), no response for a long time (Timeout), failure to load the data correctly, etc. 3.1.2

Data Load Time. The time to ingest the data in the system.

3.1.3 Response Time for Interactions. The time between issuing an interaction and getting a response. This is measured for both data scope operations and interaction system operations. 3.1.4 Correctness. Although most graph operations are guaranteed to return a deterministic and exact result across different graph systems, some graph operations are implemented using variants or approximations in different graph systems e.g., betweenness and closeness centrality. To enable a fair comparison of the correctness of such graph operations across systems, we use a multi-pronged 5

approach for this metric, partly borrowing some approaches from the LDBC Graphalytics Benchmark [25]:

benchmark but cannot report benchmarking results to remain in legal compliance. We thus evaluate representatives from three categories of graph systems - INVA, GDB and GSL, which provides a reference benchmark performance for systems in these categories on INVA workloads, and helps us understand how different are they from each other. Our implementation challenges also demonstrate why a benchmark is critical for INVA use cases; otherwise, many of these tools will continue to lack basic interactive features, scalability tests, and benchmarking support.

Exact: For deterministic queries which return exact answers like data scope size, number of connected components, shortest path etc. we report their accuracy by comparing the exact values (exact match [25]) of test system results, with those of the workload generator (reference results). Ranking: For deterministic queries, the results for which differ slightly owing to algorithmic variants, e.g., centrality measures, we report accuracy by computing the Spearman’s ranking correlation coefficient between rankings of the nodes/links as per the computed attribute (for a limited sample of nodes/links) for the test system and the workload generator (reference results).

3.2.3 Test System Configuration. We now describe specific configurations for each chosen test system and their interfaces, using which the experiments were run. ➢Cytoscape [55] (Java): We used Cytoscape v3.10.4 with its inbuilt CyREST API and the python driver package py4cytoscape [47] v1.12.0 to implement the interface. The use of py4cytoscape requires the Cytoscape GUI tool to be running alongside. We run our experiments on a headless machine, but because Cytoscape requires a display to run the GUI tool, we create a virtual display using Xvfb utility. ➢Tulip [3] (C++): We used Tulip python package tulip-python [60] v6.0.0 to write the Tulip interface. Unlike Cytoscape, the tulippython driver can be used standalone and does not require the GUI tool running alongside. ➢Neo4j [38] (Java): We used Neo4j v2025.08.0 Community Edition along with the python driver neo4j [41] v5.28.0, and additional utility libraries designed for network analysis—Neo4j Graph Data Science Library [40] v2.21.0 and Neo4j APOC library [39] v2025.08.0 to implement the interface. Neo4j also provides a visualization library - Neo4j Bloom. However, we do not use it in our experiments because it is only available for the enterprise edition of Neo4j. The Neo4j server is configured to use a maximum of 127 GB of heap memory. Additional server configuration settings can be found in the neo4j.conf file in the docker container. ➢Memgraph [34] (C++): We used Memgraph v3.7.2, an in-memory graph database, along with its inbuilt network analysis library– Memgraph Advanced Graph Extensions (MAGE) [35]. Memgraph uses the same python driver package as Neo4j, with custom tweaks made for Memgraph. We were thus able to use Claude AI to easily adapt the Neo4j interface for Memgraph. The Memgraph server is configured to enable optimization for graph OLAP as opposed to graph OLTP, with a maximum query execution time of 3 hours after which it times out. Additional server configuration settings can be found in the memgraph.conf file in the docker container. ➢NetworKit [56] (C++): We used NetworKit v11.1.post1. Of the chosen test systems, Neo4j, Tulip and Memgraph have multiple driver options available. However we use the Python driver to keep our codebase uniform, for ease of integration, and to enable fair comparison with other systems which have only Python drivers. In a way, we test not only the test system backends, but also the python interfaces to these backends for executing various interactions. Thus the results presented can also help developers and researchers to decide which interface to use when developing applications on top of graph systems. For a fair comparison, we ran our experiments on the single machine execution setup of all the test systems, even for systems

We do not report correctness for graph operations which are stochastic in nature or may be configured by the user to return different results e.g., graph clustering and layout generation, due to the lack of a single correct answer for the same. 3.1.5 Workload Completion Time. The time to run the entire workload as experienced by the human analyst. This includes the time taken to load data, set data scopes and execute system operations for all workflows in the workload, in addition to the transformations required between intermediate data formats, from fetching query results to displaying the results on screen during the execution of workloads. It does not include the think time required by the human analyst to understand the results.

3.2

Experimental Setup

3.2.1 Hardware Configuration. We performed our experiments on a computer with a 48 core Intel Xeon 2.6 GHz CPU, having an L1 cache of 768 KB per core, L2 cache of 6 MB per core, and a shared L3 cache of 60 MB, 512 GB of main memory and 3.2 TB of disk space, running Rocky Linux 9.6. For reproducibility, better portability and sandboxing of the setup, we dockerized the benchmarking framework along with the test systems 1 . The docker container was configured to use Ubuntu 22.04 with 128 GB of main memory, and an additional 128 GB of swap space. 3.2.2 Test System Selection. We evaluate one INVA system– Cytoscape [55] and two GDBs – Neo4j [38] and Memgraph [34]. We also report results on Tulip [3] (INVA system) for data load time and all expressiveness metrics, but we do not run workloads on Tulip (no response time and correctness metrics) due to difficulties in implementing the Tulip interface owing to insufficient documentation for the APIs exposed for its capabilities. We consider the workload generator results generated using Python NetworKit [56], a graph scripting library (GSL) as reference results. Apart from the aforementioned chosen test systems, we also considered Gephi [6] (INVA system), Python NetworkX [23] (GSL), TigerGraph [58] and Kuzu [30] (GDB). We rejected Gephi due to insufficient resources to develop the interface to the benchmark driver. We rejected NetworkX due to known scalability issues, and Kuzu because it does not have its own graph analytics capabilities (it uses NetworkX instead, which limits its scalability). We were able to successfully support TigerGraph and evaluate it using our 1 Docker container available at - https://osf.io/inva-benchmark

6

which support a distributed setup. We also unit tested each test system interface to prevent any interface implementation issues from confounding the results of our evaluation. Our drivers run the workloads against test system backends, hence the benchmarking results do not reflect the extra cost incurred by GUI interaction processing, but they do account for the time spent in processing the python driver API calls. Finally, for system operations like graph layout, graph clustering etc. which have multiple algorithm options available, we evaluate the default algorithm set in the system/GUI. We essentially evaluate the system for users having limited experience with using the system - they would resort to the defaults.

Data Load Time (s)

1,000

10

Test System 1

cytoscape tulip neo4j memgraph networkit (reference)

0.1

0.01

3.2.4 Datasets & Workloads. We evaluated the test systems using 8 datasets. One workload was generated for each dataset. The dataset and workload characteristics are shown in section 3.2.4 and section 3.2.4 respectively. The synthetic datasets were generated using our data generator module, while the real-world datasets (highlighted in section 3.2.4) were obtained from the Network Repository [52]. All the network datasets used for testing are undirected. We do so to allow for benchmarking of as many system operations across as many test systems as possible; most algorithms for directed networks also work on undirected networks, but the reverse is not true. We run each workload on each test system thrice and present both the individual and averaged results.

3.3

100

1K

10K

100K

1M

10M

100M

Data Scale (|N|+|L|)

Figure 3: Average data load time across 3 runs with raw values vs. data scale for the test systems, along with workload generator (NetworKit) reference results.

3.3.3 Data Scope Operation Response Time & Correctness. Figure 4 (bottom row) shows that all test systems except for Cytoscape perform data scope operations correctly at all scales. Although Cytoscape returns correct results at all scales for the k-hop neighborhood operations, it returns incorrect results for node and link attribute filters around the 100K data scale mark and beyond, with inconsistent reproducibility. The Cytoscape team has acknowledged the intermittent correctness issue at scale, uncovered by our benchmark. In terms of response time, fig. 4 (top row) shows that when Cytoscape does work, it performs almost comparably (within an an order of magnitude) with Neo4j across all data scope operations and data scales. However Cytoscape does not scale as much as Neo4j as shown both in section 3.3.1 and fig. 4. Memgraph performs at least around an order of magnitude faster than Cytoscape and Neo4j for all data scope operations with minor variations w.r.t. data scale. The faster execution of Memgraph compared to Cytoscape and Neo4j could be attributed to the use of vectors in Memgraph vs. linked lists in Cytoscape and Neo4j. The in-memory nature of Memgraph further strengthens its case compared to Neo4j. Thus for data load time, Cytoscape is among the slower test systems, followed by the GDBs, while surprisingly, NetworKit is the fastest across all data scope operations and nearly all data scales, with exceptional performance for k-hop neighborhood operations.

Performance Results

In this section, we report the performance results of the chosen test systems on our benchmark, along with the workload generator (NetworKit) results which we consider as reference results. 3.3.1 Scalability Limit. As shown in section 3.3.1, Neo4j, Memgraph and NetworKit scaled up to the largest dataset D8 without issues. Cytoscape scaled up to D4 with timeout failures for D5, D7 and D8. We observe that graph DBMSs (GDB) are generally designed for scalability more than INVA systems as also observed in prior work [4]. The OpenMP based parallelized implementation of graph operations in NetworKit partially explains its scalability. 3.3.2 Data Load Time. Figure 3 shows that Cytoscape and Neo4j consistently take more time than Tulip, Memgraph and NetworKit, with the difference being around an order of magnitude at the 100K data scale mark and beyond. Both Cytoscape and Neo4j are implemented in Java and store nodes/links using linked lists. In contrast, Tulip, Memgraph and NetworKit are implemented in C++, and store nodes/links in contiguous memory arrays (vectors) which partially explains why they perform better than Cytoscape and Neo4j. NetworKit uses OpenMP for parallelized execution which also explains its faster speed especially at smaller data scales. Memgraph is an in-memory GDB as opposed to Neo4j which is a disk-based, which also explains their performance difference. Furthermore, the results for Neo4j are obtained with explicit indexing on nodes and batching during data load, without which Neo4j takes an unreasonably long time for loading data, and yet it is marginally slower than Cytoscape for more than the 1M data scale mark. In all, we see clear performance difference between systems which use linked-list vs. contiguous memory based network data storage.

3.3.4 System Operation Response Time & Correctness. To save space, we present results for only a select few system operations which reveal correctness differences across test systems due to differences in implementation, as shown in fig. 5. For a meaningful comparison, we evaluate correctness of a system operation for a given test system only when the test system is able to correctly execute the data scope operation of the corresponding workflow within the workload. As a reference for how responsive each system operation is for INVA use cases, we annotate all response time plots with a dashed red line at 1s denoting an interactivity threshold which when exceeded, can disrupt the flow of thought of the analyst [42]. Given the lack of prior work evaluating interactivity 7

Table 2: Characteristics of the datasets used for evaluation, listed in increasing order of data scale (last column). Real-world datasets are highlighted in yellow, the rest are synthetic datasets generated using our data generator. Dataset

#Nodes |𝑁 |

#Links |𝐿 |

Node Attributes

Link Attributes

|𝑁 | : |𝐿| ratio

Log Data Scale 𝑙𝑜𝑔10 ( |𝑁 | + |𝐿 | )

D1 D2 D3 D4 D5 D6 D7 D8

1,000 50,000 1,000 203,769 822,942 50,000 2,146,057 600,000

1020 49,760 100,055 234,355 1,348,374 5,003,843 5,743,132 15,003,456

Float: 2, String: 2 Float: 2, Bool: 2 Float: 2, String: 2 Int: 1, Float: 3 String: 1, Int: 1, Float: 2 Float: 2, Bool: 2 None Float: 2

Float: 1, Bool: 1 Float: 1, String: 2 Float: 1, Bool: 1 None None Float: 1, String: 2 Int: 2 Float: 2

1:1 1:1 1:100 1:1.15 1:1.6 1:100 1:2.7 1:25

3.305 4.999 5.005 5.642 6.337 6.704 6.897 7.193

Data Scope Operation

Response Time (s)

Node Attribute Filter

Link Attribute Filter

2-hop Neighborhood

3-hop Neighborhood

1e2 1e0 1e-2 1e-4

Accuracy (%)

100 75

Test System

50

cytoscape neo4j memgraph networkit (reference)

25 0 1K 10K

100K

1M

10M100M 1K 10K

100K

1M

10M100M 1K 10K

100K

1M

10M100M 1K 10K

100K

1M

10M100M

Data Scale (|N|+|L|)

Figure 4: Average response time across 3 runs with raw values (top row) and average accuracy across 3 runs (bottom row) vs. data scale for the test systems. Accuracy plots show overplotting for test systems which report 100% accuracy. specifically for graph analysis, we call for future work in this space in section 4.3. Closeness Centrality. For the limited number of instances of closeness centrality system operation, fig. 6 shows that both Memgraph and NetworKit consistently run around an order of magnitude faster than Cytoscape and Neo4j. While Neo4j runs faster than Cytoscape, NetworKit runs faster than Memgraph. All test systems fail to meet the 1s interactivity threshold once the data scale reaches the 100K mark. Although Memgraph is one of the faster graph systems evaluated in our experiments, it has severe correctness issues with accuracy dropping down to less then 50% in some cases. The centrality values for all failed instances of Memgraph agree with the reference results except for some nodes for which the closeness centrality value is reported as ‘undefined’. These are isolated nodes in the data scope to which Neo4j, Cytoscape and NetworKit assign a value of 0. Cytoscape also shows an accuracy between 80% and 90% across all instances, which possibly hints towards an approximated implementation of closeness centrality in Cytoscape. Thus

our benchmarking framework revealed possible differences in Cytoscape, and possible bugs in Memgraph for closeness centrality computation. Node Betweenness Centrality For the limited number of instances of node betweenness centrality, fig. 7 shows that all 3 test systems along with NetworKit have relatively similar runtimes at lower scale compared to at higher scale, where Cytoscape might not be as fast as others. There is insufficient data to comment on at what data scale node betweenness centrality fails to meet the 1s interactivity threshold. Memgraph shows reduced correctness with the same issue as that for closeness centrality–unreachable nodes being assigned an undefined value instead of 0. With 0% accuracy at close to the 10M data scale mark, our benchmark reveals another possible bug in the node betweenness centrality computation at scale for Cytoscape. Connected Components As shown in Figure 8, Cytoscape runs around two orders of magnitude slower than the GDBs as the data 8

Add Node

100

100

100

Closeness Centrality

86

100

48

Connected Components

100

100

49

Delete Edge

100

100

100

Accuracy (%) 100

80

60

W1 (8) W2 (9) W3 (7) W4 (9) W5 (8) W6 (9) W7 (6) W8 (7)

5 6 6 7 4 8 5 7

17 7 6 9 9 8 3 7

10 7 7 12 13 8 6 9

11 6 12 7 14 8 4 5

Delete Node

100

100

100

Global Clustering Coefficient

100

100

100

Local Clustering Coefficient

100

100

100

Node Betweenness Centr…

50

100

84

Total Degree Centrality

N/A

100

100

Count

100

100

100

Global Custom Computation

100

100

100

Local Custom Computation

84

100

100

Update Edge

100

100

100

Update Node

100

100

100

Maximum

92

100

100

Minimum

100

100

100

Dataset

Cytoscape

Neo4j

Memgraph

NetworKit

Sort Edge Attribute Value

100

100

100

Sort Node Attribute Value

100

100

100

neo4j

memgraph

D1 (3.305) D2 (4.999) D3 (5.005) D4 (5.642) D5 (6.337) D6 (6.704) D7 (6.897) D8 (7.193)

✓ ✓ ✓ ✓ Timeout ✓ Timeout Timeout

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

40

20

0

Comparison Focus System Operations

Attributes Focus System Operations

#comparison focus interactions

100

#attribute focus interactions

100

#graph focus interactions

100

#data scope operations

Graph Focus System Operations

Add Edge

Workload (#workflows)

Table 3: Characteristics of the evaluation workloads. Each workload is created for the corresponding numbered dataset.

Correctness wrt NetworKit (reference)

cytoscape

Table 4: Scalability limit results; Log data scale reported in parentheses; Timeout means no response for 3 hours.

Test System

Figure 5: Summary of correctness in terms of accuracy, of evaluated system operations for different test systems w.r.t. NetworKit (reference) results, averaged across all data scales. Graph > Metrics > Closeness Centrality

scale increases. It also fails to meet the interactivity threshold before the 10K data scale mark. Memgraph and Neo4j have similar runtimes and remain interactive roughly up to the 1M data scale mark. While both Memgraph and Neo4j are around an order of magnitude slower than NetworKit, this difference decreases as the data scale increases. However in spite of its competitive runtime, Memgraph has severe correctness issues in some cases suggesting a possible bug in its connected components implementation. Local Custom Computation & Maximum Figure 9 and fig. 10 show that the runtime for attribute and comparison focus system operations increases linearly w.r.t. data scale but with a smaller constant factor as opposed to graph focus system operations. This can be explained by the time complexity of attribute and comparison focus system operations which is either 𝑂 (|𝑁 |) or 𝑂 (|𝐿|), whereas for graph focus system operations the time complexity is generally 𝑂 (|𝑁 | + |𝐿|) (e.g., connected components), 𝑂 (|𝑁 |.|𝐿|) (e.g., centrality measures) or worse. Because of this, these system operations can meet the interactivity threshold up to slightly beyond the 1M data scale mark, which is more than that for graph focus system operations. Comparatively, Cytoscape and Neo4j have similar runtimes while Memgraph runs faster in most cases followed by NetworKit which is the fastest. In terms of correctness, Cytoscape has some issues for both local custom computation and maximum system operations, hinting towards possible bugs.

Response Time (s)

1e4

1e2

1e0

1e-2

Accuracy (%)

100 75 50

Test System

25

cytoscape neo4j

0 100

1K

10K

100K

memgraph networkit (reference) 1M

10M

Data Scale (|N|+|L|)

Figure 6: Results for reponse time and correctness (ranking based accuracy) for computing closeness centrality

9

Attributes > Compute > Local Custom Computation

Graph > Metrics > Node Betweenness Centrality 1e4

Response Time (s)

Response Time (s)

1e2

1e2 1e0 1e-2

75 50 25

1e-2

1e-4

100

Accuracy (%)

Accuracy (%)

100

1e0

Test System cytoscape neo4j

0

memgraph networkit (reference)

75

Test System

50

cytoscape neo4j memgraph networkit (reference)

25 0

100

1K

10K

100K

1M

10M

10

100

Data Scale (|N|+|L|)

1K

10K

100K

1M

10M

Data Scale (|N|+|L|)

Figure 7: Results for response time and correctness (ranking based accuracy) for computing node betweenness centrality

Figure 9: Results for response time and query correctness (exact accuracy) for transforming a node/link attribute i.e., local custom computation

Graph > Group > Connected Components Comparison > Extrema > Maximum

1e2

Response Time (s)

Response Time (s)

1e4

1e0 1e-2 1e-4

1e-4

100

Test System Accuracy (%)

Accuracy (%)

1e-2

1e-6

100 75

1e0

cytoscape neo4j memgraph networkit (reference)

50 25 0 10

100

1K

10K

100K

1M

10M

75

Test System 50

cytoscape neo4j memgraph networkit (reference)

25 0

100M

Data Scale (|N|+|L|)

10

100

1K

10K

100K

1M

10M

Data Scale (|N|+|L|)

Figure 8: Results for response time and query correctness results (exact accuracy) for counting the number of connected components

Figure 10: Results for response time and query correctness (exact accuracy) for finding the maximum value of a node/link attribute

3.3.5 Workload Completion Time. Figure 11 shows that, of the evaluated test systems, Memgraph is the most suitable for INVA in terms of scalability and response time. Although Cytoscape is a dedicated INVA system, it shows higher workload completion time along with scalability issues (W5, W7, W8). This shows how graph databases are better equipped to be used for INVA for large networks than dedicated INVA systems. While NetworKit is faster than all the test systems, its programming heavy nature can make it less favorable for domain experts or non-programmers.

3.3.6 Takeaways. Both in terms of correctness and response time, Cytoscape clearly struggled to compete with the GDBs, and NetworKit was comprehensively better than the GDBs. For all system operations across all three analysis foci, Cytoscape almost always failed to meet the interactivity threshold earlier i.e., at a smaller data scale, than other test systems. Neo4j, Memgraph and NetworKit in that specific order, remain interactive up to a higher data scale mark than the prior system. 10

Test System

cytoscape

neo4j

memgraph

networkit (reference)

Analyses Coverage (#system operations)

Workload Completion Time (s)

1M 100K 10K 1K 100 10

40

30

Test Systems

31/43 29/43 (72%) 29/43 (67%) (67%)

total cytoscape neo4j memgraph

22/28 (78%) 20/28 (71%) 17/28 (60%)

20

5/5 5/5 5/5 (100%) (100%) (100%)

10

7/10 (70%)

4/10 4/10 (40%) (40%)

1 0

all foci

0.1 0.01

graph focus

W1

W2

W3

W4

W5

W6

W7

W8

comparison focus

(a) Analyses Coverage Test Systems

Workload

Figure 11: Workload completion time for each test system averaged across 3 runs. ✖ indicates failed workload run.

cytoscape neo4j memgraph 0

300

600

Lines of Code

While Neo4j performed with 100% accuracy for all system operations across all data scales, our benchmarking evaluation showed that Memgraph has correctness issues. The correctness bugs uncovered in Cytoscape and Memgraph demonstrate the value of our benchmark in performance and correctness testing for INVA workloads. Cytoscape has acknowledged some of the bugs while Memgraph is yet to confirm the possible implementation bugs revealed by our benchmark. Going ahead, we need more experiments with different graph systems which support INVA, and at higher data scales, to understand if dedicated INVA systems like Cytoscape can still survive, and if GSLs like NetworKit can still remain competitive with GDBs.

3.4

attribute focus

Analysis Focus

900

1.2K

0

1

2

3

4

5

Ease of Implementation (5-point Likert)

(b) Lines of Code and Ease of Implementation

Figure 12: Expressiveness metrics for the test systems requiring explicit conversion from directed to undirected for certain interactions. These factors contribute to a higher Lines of Code compared to Cytoscape which exposes more of the required functionality through concise APIs as shown in fig. 12b. 3.4.3 Ease of Implementation. 5-point Likert scale rating based on the ease of availability and comprehensiveness of the documentation, developer support, and activity/responses on user forums, to help implement the test system interfaces to the benchmarking framework using the test system driver APIs. All three test systems—Cytoscape, Neo4j and Memgraph provide good developer support with active user base and user/developer forums. Owing to its recency, Memgraph does not have as large a user base as Neo4j, based on the number of followers for their respective github repositories. Neo4j and Memgraph both use the same python driver which is well documented, fetching them 5 points for Ease of Implementation. Cytoscape provides good documentation for the py4cytoscape API but not for CyREST API, and thus we gave it 4 points on Ease of Implementation as shown in fig. 12b.

Expressiveness Metrics & Results

We now describe the more human user or developer oriented soft metrics, along with the results on our chosen test systems. These metrics help us gauge how easy or difficult it can be for analysts to use these systems, and for developers to work with them. 3.4.1 Analyses Coverage. Percentage of system operations from the benchmark which are supported in the test system. Figure 12a shows that Cytoscape supports fewer graph focus system operations than Neo4j and Memgraph, e.g., path finding, graph clustering etc. and also does not facilitate a graph querying language. On the other hand, both Neo4j and Memgraph provide better support for graph focus interactions through their dedicated graph analysis libraries—Neo4j GDS and MAGE respectively. Cytoscape supports attribute focus system operations through node/link table manipulations in the GUI, while the same can be achieved using the Cypher query language facilitated in GDBs. Finally, Cytoscape exposes more capabilities to visualize the network data along with auxiliary visualizations for node/link attributes (comparison focus), than Neo4j and Memgraph. We had the limitation of working without Bloom visualization plugin for Neo4j Community edition, and Memgraph does not expose a visualization API.

4

DISCUSSION

In this section, we discuss some limitations and future work, and also highlight certain insights from our benchmarking evaluation.

4.1

Towards Efficient Evaluation

Our INVA model does not cover exhaustively all possible system operations, and thus the generated workloads may not cover exhaustively all possible network analysis scenarios. However, our model facilitates adding more such system operations, and a configurable transition probability matrix, using which a variety of semantic INVA workloads may be generated. User analysis goal based transition probability matrices may also be used for more realistic workload generation [46]. We also need formal validation of how closely the workloads generated using our model resemble the workflow of human network analysts. Since INVA emphasizes

3.4.2 Lines of Code. Number of lines of code required to implement the test system interface using their respective driver APIs. Both Neo4j and Memgraph require writing Cypher queries for data scope operations and some graph and attribute focus interactions. Furthermore, they allow the creation of only directed graphs, 11

human-in-the-loop network data understanding, we also need user studies to understand the user experience with different graph systems for INVA. Such studies can compliment our evaluation of graph system backends thus providing a holistic picture of how good a graph system is for INVA. Based on our experience developing the interface for each test system, and the growing importance of graph applications across domains, we envision a unified interface for all graph systems, which could ease the benchmarking of different graph systems, also making it easy for developers to test different graph system backends when developing graph applications.

4.2

focus of our work. Ours is the first benchmark to accommodate INVA workloads which iterate between interactive graph operations like inserts/deletes, operations on local neighborhoods etc. and non-interactive graph operations like community detection, centrality measure computation etc. While the LDBC has separate benchmarks for different data domains like social network analysis [18] and financial network analysis [48], our benchmark can generate a variety of workloads using a configurable transition probability matrix, and for any network data domain, thus making it versatile. The benchmarking work of Eichmann et al. [17] and Battle et al. [9] for interactive analysis of relational data is more relevant to our work owing to its user behavior modeling in the interactive environment. Nevertheless, we refer to the work of LDBC for choosing datasets with appropriate diversity and coverage, and performance metrics for evaluation.

INVA vs. Graph DBMS vs. Scripting Libraries

The results of our benchmarking evaluation reveal certain insights which could be extended to the graph system categories of INVA systems, graph DBMS (GDB) and graph scripting libraries (GSL). In addition to providing network data storage solutions and efficient graph analytics, GDBs today also provide plugins for graph analysis and visualization accessible through GUIs, along with AI agents e.g., Memgraph and FalkorDB [19], thus making them as convenient to use if not more, as INVA systems. This could be crucial for domain experts or non-programmers who form a significant portion of INVA systems user base. Newer GSLs are also outperforming dedicated INVA systems, even competing with GDBs up to the 1M data scale mark. This partially explains why newer GDBs are leveraging GSLs to provide graph analytics capabilities e.g., NetworkX for Memgraph and Kuzu. The only drawback of GSLs is the lack of a GUI which might restrict their user base to programming experts. With the inability of INVA systems to keep up with GDBs for scalability and interactivity requirements of INVA, and the increasing adoption of user friendly affordances in GDBs, GDBs seem to be well equipped to replace dedicated INVA systems.

4.3

5.2

Graph Task Taxonomies

We primarily base our INVA model on the graph task taxonomies of Lee et al. [31], Nobre et al. [43] and the operational graph task survey of [4]. Apart from that, Filipov et al. [20] provide a comprehensive survey of the prior work in graph task taxonomies, some of which we mention here. Pretorius et al. [45] presented a more refined version of Lee et al.’s taxonomy. Kerracher et al. [28] extended the same taxonomy for time-varying graphs, wherein they added an orthogonal dimension of time as timestamps or time intervals. The taxonomy of Ahn et al. [26] for dynamic graphs categorizes tasks along three aspects of network data - the type of data points being analyzed (nodes, links, clusters), the temporal aspect, and the data attributes. Saket et al. [54] provide an alternative perspective on graph task taxonomies by considering clusters or groups of nodes instead of just raw nodes/links.

OLAP vs. OLTP for Graphs

6

The data management community interprets interactivity differently for graph OLTP and OLAP. Graph OLTP queries, which consist of queries operating on small sub-graphs, local neighborhoods or shortest path queries are considered interactive, but graph OLAP queries which consist of long running iterative algorithmic graph analysis e.g., centrality measures, community detection etc. are not considered interactive [18, 57]. However Battle et al. [7] have shown that the threshold for interactivity in visual analytics depends on the complexity of the task being performed, and with the current state of advancements in graph analysis hardware e.g., GPU, and algorithms e.g., parallelized algorithms, progressive algorithms, some of these tasks can be performed reasonably interactively. This raises an important question - What is a reasonable interactive latency for INVA tasks? We thus need to rethink how we approach interactivity in graph OLAP.

CONCLUSION

Acknowledging the growing need for human-in-the-loop or interactive understanding of large scale network data, we developed a benchmarking framework grounded in an empirical model of network visualization and analysis workflows, to comparatively evaluate different graph systems for interactive network visualization and analysis (INVA). We demonstrated the value of our benchmarking framework using one INVA system and two graph DBMSs. In addition to uncovering correctness issues, our findings show how purpose built INVA systems face the danger of being replaced by graph DBMSs given their inability to scale up to the large scale graphs in use today. We also call for consideration of interactivity in graph analysis so as to enable the development of scalable and user friendly systems to facilitate interactive network visualization and analysis.

ACKNOWLEDGMENTS

5 RELATED WORK 5.1 Benchmarking Graph Systems

We thank the members of the IDL and Database labs at UW for their feedback, Alex Pico and team (Cytoscape), Xuanlei Lin and Songting Chen (TigerGraph), and Ante Javor (Memgraph) for helping with the development of the respective interfaces for use with our benchmarking framework, and explaining some of the results from our evaluation. This research was supported in part by Google and NSF awards IIS-2402718, IIS-2141506 and IIS-2514565.

Prior work in benchmarking graph systems, most of which comes from the Linked Data Benchmark Council (LDBC), is focused more on offline graph processing systems like Giraph, PowerGraph and GraphX [16, 25, 37] or on interactive (graph OLTP) workloads but for graph databases [2, 18] and not for INVA, which is the main 12

REFERENCES

International Conference on Management of Data. ACM, Melbourne Victoria Australia, 619–630. https://doi.org/10.1145/2723372.2742786 [19] FalkorDB [n.d.]. FalkorDB Graph Database. https://www.falkordb.com/. [20] Velitchko Filipov, Alessio Arleo, and Silvia Miksch. 2023. Are We There Yet? A Roadmap of Network Visualization from Surveys to Task Taxonomies. Computer Graphics Forum 42, 6 (Sept. 2023), e14794. https://doi.org/10.1111/cgf.14794 [21] Sneha Gathani, Shayan Monadjemi, Alvitta Ottley, and Leilani Battle. 2022. A Grammar-Based Approach for Applying Visualization Taxonomies to Interaction Logs. Computer Graphics Forum 41, 3 (June 2022), 489–500. https://doi.org/10. 1111/cgf.14557 [22] Piotr Hadaj, Dominik Strzałka, Marek Nowak, Małgorzata Łatka, and Paweł Dymora. 2022. The use of PLANS and NetworkX in modeling power grid system failures. Scientific Reports 12, Article 17445 (Oct. 2022), 17445 pages. https://doi.org/10.1038/s41598-022-22268-z [23] Aric Hagberg, Pieter J. Swart, and Daniel A. Schult. 2008. Exploring network structure, dynamics, and function using NetworkX. (1 2008). https://www.osti. gov/biblio/960616 https://networkx.org/documentation/stable/index.html. [24] Oliver Huang and Carolina Nobre. 2025. ViStruct: Simulating Expert-Like Reasoning Through Task Decomposition and Visual Attention. 2025 IEEE Visualization and Visual Analytics (VIS) (2025), 216–220. https://api.semanticscholar.org/ CorpusId:280010918 [25] Alexandru Iosup, Ahmed Musaafir, Alexandru Uta, Arnau Prat-Pérez, Gábor Szárnyas, Hassan Chafi, Ilie Gabriel Tanase, Lifeng Nai, Michael J. Anderson, Mihai Capota, Narayanan Sundaram, Peter A. Boncz, Siegfried Depner, Stijn Heldens, Thomas Manhardt, Tim Hegeman, Wing Lung Ngai, and Yinglong Xia. 2020. The LDBC Graphalytics Benchmark. CoRR abs/2011.15028 (2020). https://doi.org/10.48550/arXiv.2011.15028 arXiv:2011.15028 [26] Jae-wook Ahn, Catherine Plaisant, and Ben Shneiderman. 2014. A Task Taxonomy for Network Evolution Analysis. IEEE Transactions on Visualization and Computer Graphics 20, 3 (March 2014), 365–376. https://doi.org/10.1109/TVCG. 2013.238 [27] Ghazal Kalhor, Amin Asadi Sarijalou, Niloofar Sharifi Sadr, and Behnam Bahrak. 2022. A new insight to the analysis of co-authorship in Google Scholar. Applied Network Science 7 (2022). https://api.semanticscholar.org/CorpusID:248028142 [28] Natalie Kerracher, Jessie Kennedy, and Kevin Chalmers. 2015. A Task Taxonomy for Temporal Graph Visualisation. IEEE Transactions on Visualization and Computer Graphics 21, 10 (Oct. 2015), 1160–1172. https://doi.org/10.1109/TVCG. 2015.2424889 [29] Vesa Kuikka, Lauri Pykälä, Tuomas Takko, and Kimmo K. Kaski. 2025. Network modelling in analysing cyber-related graphs. Frontiers in Complex Systems Volume 3 - 2025 (2025). https://doi.org/10.3389/fcpxs.2025.1620260 [30] Kuzu [n.d.]. Kuzu. https://kuzudb.github.io/. [31] Bongshin Lee, Catherine Plaisant, Cynthia Sims Parr, Jean-Daniel Fekete, and Nathalie Henry. 2006. Task taxonomy for graph visualization. In Proceedings of the 2006 AVI workshop on BEyond time and errors: novel evaluation methods for information visualization. ACM, Venice Italy, 1–5. https://doi.org/b2pm7w [32] Yucheng Low, Danny Bickson, Joseph Gonzalez, Carlos Guestrin, Aapo Kyrola, and Joseph M. Hellerstein. 2012. Distributed GraphLab: a framework for machine learning and data mining in the cloud. Proceedings of the VLDB Endowment 5, 8 (April 2012), 716–727. https://doi.org/10.14778/2212351.2212354 [33] Grzegorz Malewicz, Matthew H. Austern, Aart J.C Bik, James C. Dehnert, Ilan Horn, Naty Leiser, and Grzegorz Czajkowski. 2010. Pregel: a system for largescale graph processing. In Proceedings of the 2010 ACM SIGMOD International Conference on Management of data. ACM, Indianapolis Indiana USA, 135–146. https://doi.org/10.1145/1807167.1807184 [34] MemGraph [n.d.]. MemGraph Graph Database. https://memgraph.com/. [35] Memgraph MAGE [n.d.]. MemGraph Advanced Graph Extensions. https: //memgraph.com/docs/advanced-algorithms/install-mage. [36] Lingkai Meng, Yu Shao, Long Yuan, Longbin Lai, Peng Cheng, Xue Li, Wenyuan Yu, Wenjie Zhang, Xuemin Lin, and Jingren Zhou. 2024. A Survey of Distributed Graph Algorithms on Massive Graphs. ACM Comput. Surv. 57, 2, Article 27 (Oct. 2024), 39 pages. https://doi.org/10.1145/3694966 [37] Lingkai Meng, Yu Shao, Long Yuan, Longbin Lai, Peng Cheng, Xue Li, Wenyuan Yu, Wenjie Zhang, Xuemin Lin, and Jingren Zhou. 2025. Revisiting Graph Analytics Benchmark. Proc. ACM Manag. Data 3, 3, Article 208 (June 2025), 28 pages. https://doi.org/10.1145/3725345 [38] Neo4j [n.d.]. Neo4j Graph Database. https://neo4j.com/. [39] Neo4j APOC [n.d.]. Neo4j - APOC Library. https://neo4j.com/labs/apoc/. [40] Neo4j GDS [n.d.]. Neo4j - Graph Data Science Library. https://neo4j.com/docs/ graph-data-science/current/. [41] Neo4j Python Driver [n.d.]. Neo4j - Python Driver. https://neo4j.com/docs/api/ python-driver/current/. [42] Jakob Nielsen. [n.d.]. Response Times: The 3 Important Limits. https://www. nngroup.com/articles/response-times-3-important-limits/. [43] Carolina Nobre, Marc Streit, Miriah Meyer, and Alexander Lex. 2019. The State of the Art in Visualizing Multivariate Networks. Computer Graphics Forum (EuroVis) 38 (2019), 807–832. https://doi.org/10.1111/cgf.13728

[1] Maksudul Alam and Kalyan Perumalla. 2021. Fast GPU-Based Generation of Large Graph Networks From Degree Distributions. Frontiers in Big Data Volume 4 - 2021 (2021). https://doi.org/10.3389/fdata.2021.737963 [2] Timothy G. Armstrong, Vamsi Ponnekanti, Dhruba Borthakur, and Mark Callaghan. 2013. LinkBench: a database benchmark based on the Facebook social graph. In Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data. ACM, New York New York USA, 1185–1196. https: //doi.org/10.1145/2463676.2465296 [3] David Auber. 2004. Tulip — A Huge Graph Visualization Framework. In Graph Drawing Software, Gerald Farin, Hans-Christian Hege, David Hoffman, Christopher R. Johnson, Konrad Polthier, Michael Jünger, and Petra Mutzel (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 105–126. https://doi.org/10. 1007/978-3-642-18638-7_5 Series Title: Mathematics and Visualization. [4] Anonymous authors. 2026. Scalability and Interactivity in Network Visualization & Analysis: a Survey. Available as supplementary material with this submission. [5] Hadis Azizi, Mohammad Sadra Amini, Sadegh Sulaimany, and Aso Mafakheri. 2025. Visibility graph analysis for educational data: potentials and a case study of predicting at-risk online students. Scientific reports 15, 1 (September 2025), 32036. https://doi.org/10.1038/s41598-025-17760-1 [6] Mathieu Bastian, Sebastien Heymann, and Mathieu Jacomy. 2009. Gephi: An Open Source Software for Exploring and Manipulating Networks. Proceedings of the International AAAI Conference on Web and Social Media 3, 1 (March 2009), 361–362. https://doi.org/10.1609/icwsm.v3i1.13937 [7] Leilani Battle, R. Jordan Crouser, Audace Nakeshimana, Ananda Montoly, Remco Chang, and Michael Stonebraker. 2020. The Role of Latency and Task Complexity in Predicting Visual Search Behavior. IEEE Transactions on Visualization and Computer Graphics 26, 1 (2020), 1246–1255. https://doi.org/10.1109/TVCG.2019. 2934556 [8] Leilani Battle, Philipp Eichmann, Marco Angelini, Tiziana Catarci, Giuseppe Santucci, Yukun Zheng, Carsten Binnig, Jean-Daniel Fekete, and Dominik Moritz. 2020. Database Benchmarking for Supporting Real-Time Interactive Querying of Large Data. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data (Portland, OR, USA) (SIGMOD ’20). Association for Computing Machinery, New York, NY, USA, 1571–1587. https://doi.org/10.1145/ 3318464.3389732 [9] Leilani Battle, Philipp Eichmann, Marco Angelini, Tiziana Catarci, Giuseppe Santucci, Yukun Zheng, Carsten Binnig, Jean-Daniel Fekete, and Dominik Moritz. 2020. Database Benchmarking for Supporting Real-Time Interactive Querying of Large Data. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data. ACM, Portland OR USA, 1571–1587. https://doi.org/ 10.1145/3318464.3389732 [10] Angela Bonifati, M. Tamer Ozsu, Yuanyuan Tian, Hannes Voigt, Wenyuan Yu, and Enjie Zhang. 2025. A Roadmap to Graph Analytics. SIGMOD Rec. 53, 4 (Jan. 2025), 43–51. https://doi.org/10.1145/3712311.3712323 [11] Denny Borsboom, Marie Katharina Deserno, Mijke Rhemtulla, Sacha Epskamp, Eiko I. Fried, Richard J. McNally, Donald J. Robinaugh, Marco Perugini, Jonas Dalege, Giulio Costantini, Adela-Maria Isvoranu, Anna C. Wysocki, Claudia D van Borkulo, Riet van Bork, and Lourens J. Waldorp. 2021. Network analysis of multivariate data in psychological science. Nature Reviews Methods Primers 1 (2021). https://api.semanticscholar.org/CorpusID:238689940 [12] Theodosia Charitou, Kenneth Bryan, and {David J.} Lynn. 2016. Using biological networks to integrate, visualize and analyze genomics data. Genetics Selection Evolution 48, 1 (31 March 2016). https://doi.org/10.1186/s12711-016-0205-1 Publisher Copyright: © 2016 Charitou et al. [13] Ting Chen, Zihao Li, Yuxiao Zhu, Jiachi Chen, Xiapu Luo, John Chi-Shing Lui, Xiaodong Lin, and Xiaosong Zhang. 2020. Understanding Ethereum via Graph Analysis. ACM Transactions on Internet Technology 20, 2 (May 2020), 1–32. https://doi.org/10.1145/3381036 [14] Sajad Darabi, Piotr Bigaj, Dawid Majchrowski, Artur Kasymov, Pawel Morkisz, and Alex Fit-Florea. 2025. A Framework for Large-Scale Synthetic Graph Dataset Generation. IEEE transactions on neural networks and learning systems 36, 8 (August 2025), 14258—14268. https://doi.org/10.1109/tnnls.2025.3540392 [15] Dask Development Team. 2016. Dask: Library for dynamic task scheduling. http://dask.pydata.org [16] Laxman Dhulipala, Jessica Shi, Tom Tseng, Guy E. Blelloch, and Julian Shun. 2020. The Graph Based Benchmark Suite (GBBS). In Proceedings of the 3rd Joint International Workshop on Graph Data Management Experiences & Systems (GRADES) and Network Data Analytics (NDA). ACM, Portland OR USA, 1–8. https://doi.org/10.1145/3398682.3399168 [17] Philipp Eichmann, Emanuel Zgraggen, Carsten Binnig, and Tim Kraska. 2020. IDEBench: A Benchmark for Interactive Data Exploration. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data. ACM, Portland OR USA, 1555–1569. https://doi.org/qzdr [18] Orri Erling, Alex Averbuch, Josep Larriba-Pey, Hassan Chafi, Andrey Gubichev, Arnau Prat, Minh-Duc Pham, and Peter Boncz. 2015. The LDBC Social Network Benchmark: Interactive Workload. In Proceedings of the 2015 ACM SIGMOD 13

[65] Reynold S. Xin, Joseph E. Gonzalez, Michael J. Franklin, and Ion Stoica. 2013. GraphX: a resilient distributed graph system on Spark. In First International Workshop on Graph Data Management Experiences and Systems (New York, New York) (GRADES ’13). Association for Computing Machinery, New York, NY, USA, Article 2, 6 pages. https://doi.org/10.1145/2484425.2484427 [66] Álvaro Brandón, Marc Solé, Alberto Huélamo, David Solans, María S. Pérez, and Victor Muntés-Mulero. 2020. Graph-based root cause analysis for serviceoriented and microservice architectures. Journal of Systems and Software 159 (2020), 110432. https://doi.org/10.1016/j.jss.2019.110432 [67] Örjan Bodin and Beatrice I. Crona. 2009. The role of social networks in natural resource governance: What relational patterns make a difference? Global Environmental Change 19, 3 (2009), 366–374. https://doi.org/10.1016/j. gloenvcha.2009.05.002

[44] Derek O’Callaghan, Derek Greene, Maura Conway, Joe Carthy, and Pádraig Cunningham. 2013. Uncovering the wider structure of extreme right communities spanning popular online networks. In Proceedings of the 5th Annual ACM Web Science Conference. ACM, Paris France, 276–285. https://doi.org/10.1145/ 2464464.2464495 [45] Johannes Pretorius, Helen C. Purchase, and John T. Stasko. 2014. Tasks for Multivariate Network Analysis. In Multivariate Network Visualization, Andreas Kerren, Helen C. Purchase, and Matthew O. Ward (Eds.). Vol. 8380. Springer International Publishing, Cham, 77–95. https://doi.org/10.1007/978-3-319-067933_5 Series Title: Lecture Notes in Computer Science. [46] Joanna Purich, Anthony Wise, and Leilani Battle. 2025. An Adaptive Benchmark for Modeling User Exploration of Large Datasets. Proc. ACM Manag. Data 3, 1, Article 8 (Feb. 2025), 24 pages. https://doi.org/10.1145/3709658 [47] Py4Cytoscape [n.d.]. Py4Cytoscape Driver. https://py4cytoscape.readthedocs. io/en/latest/. [48] Shipeng Qi, Bing Tong, Jiatao Hu, Heng Lin, Yue Pang, Wei Yuan, Songlin Lyu, Zhihui Guo, Ke Huang, Xujin Ba, Qiang Yin, Youren Shen, Yan Zhou, Tao Lv, Jia Li, Lei Zou, Yongwei Wu, Gábor Szárnyas, Xiaowei Zhu, Wenguang Chen, and Chuntao Hong. 2025. The LDBC Financial Benchmark: Transaction Workload. Proc. VLDB Endow. 18, 9 (September 2025), 3007–3020. https://doi.org/10.14778/ 3746405.3746424 [49] Masoomeh Sadat Razavi, Abdolreza Mirzaei, and Mehran Safayani. 2026. Hierarchical community-based graph generation model for improving structural diversity. Pattern Recognition 172 (2026), 112320. https://doi.org/10.1016/j. patcog.2025.112320 [50] Pedro Ribeiro, Pedro Paredes, Miguel E. P. Silva, David Aparicio, and Fernando Silva. 2022. A Survey on Subgraph Counting: Concepts, Algorithms, and Applications to Network Motifs and Graphlets. Comput. Surveys 54, 2 (March 2022), 1–36. https://doi.org/10.1145/3433652 [51] Mayra Z. Rodriguez, Cesar H. Comin, Dalcimar Casanova, Odemir M. Bruno, Diego R. Amancio, Luciano da F. Costa, and Francisco A. Rodrigues. 2019. Clustering algorithms: A comparative approach. PLOS ONE 14, 1 (Jan. 2019), e0210236. https://doi.org/10.1371/journal.pone.0210236 [52] Ryan A. Rossi and Nesreen K. Ahmed. 2015. The Network Data Repository with Interactive Graph Analytics and Visualization. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence (Austin, Texas) (AAAI’15). AAAI Press, 4292–4293. http://networkrepository.com [53] Amitabha Roy, Ivo Mihailovic, and Willy Zwaenepoel. 2013. X-Stream: edgecentric graph processing using streaming partitions. In Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles (Farminton, Pennsylvania) (SOSP ’13). Association for Computing Machinery, New York, NY, USA, 472–488. https://doi.org/10.1145/2517349.2522740 [54] Bahador Saket, Paolo Simonetto, and Stephen Kobourov. 2014. Group-Level Graph Visualization Taxonomy. In EuroVis - Short Papers, N. Elmqvist, M. Hlawitschka, and J. Kennedy (Eds.). The Eurographics Association. https://doi.org/ 10.2312/eurovisshort.20141162 [55] Paul Shannon, Andrew Markiel, Owen Ozier, Nitin S. Baliga, Jonathan T. Wang, Daniel Ramage, Nada Amin, Benno Schwikowski, and Trey Ideker. 2003. Cytoscape: A Software Environment for Integrated Models of Biomolecular Interaction Networks. Genome Research 13, 11 (Nov. 2003), 2498–2504. https: //doi.org/10.1101/gr.1239303 [56] Christian L. Staudt, Aleksejs Sazonovs, and Henning Meyerhenke. 2015. NetworKit: A Tool Suite for Large-scale Complex Network Analysis. arXiv:1403.3005 [cs.SI] https://arxiv.org/abs/1403.3005 [57] Yuanyuan Tian. 2023. The World of Graph Databases from An Industry Perspective. SIGMOD Rec. 51, 4 (Jan. 2023), 60–67. https://doi.org/qzds [58] TigerGraph [n.d.]. TigerGraph Graph Database. https://www.tigergraph.com/. [59] Alok Tripathy, Fred Hohman, Duen Horng Chau, and Oded Green. 2018. Scalable K-Core Decomposition for Static Graphs Using a Dynamic Graph Data Structure. In IEEE International Conference on Big Data (Big Data). IEEE. https://doi.org/ 10.1109/BigData.2018.8622056 [60] Tulip Python [n.d.]. Tulip Python. https://tulip.labri.fr/Documentation/current/ tulip-python/html/index.html. [61] Zhaohua Wang, Xuewei Li, Xin Zhu, Jing Li, Fan Wang, and Fei Wang. 2021. Big data-driven public transportation network: a simulation approach. Complex & Intelligent Systems 9 (07 2021). https://doi.org/10.1007/s40747-021-00462-2 [62] Deborah J. Wojcik, Nicole M. Ardoin, and Rachelle K. Gould. 2021. Using social network analysis to explore and expand our understanding of a robust environmental learning landscape. Environmental Education Research 27 (2021), 1263 – 1283. https://api.semanticscholar.org/CorpusID:236345328 [63] Shufei Wu and Jianfeng Hou. 2023. Graph partitioning: an updated survey. AKCE International Journal of Graphs and Combinatorics 20, 1 (Jan. 2023), 9–19. https://doi.org/10.1080/09728600.2022.2148589 [64] Yifan Wu, Ziyang Guo, Michalis Mamakos, Jason Hartline, and Jessica Hullman. 2023. The Rational Agent Benchmark for Data Visualization. IEEE Transactions on Visualization and Computer Graphics (2023), 1–10. https://doi.org/10.1109/ TVCG.2023.3326513 14

A

(2) Does the test system provide declarative constructs (e.g., query language support) to help implement the interaction? If yes, continue to 3. If no, do not implement the interaction.

BENCHMARK DRIVER

The benchmark driver module serves three purposes (1) provides a blueprint for test system interface development, (2) scaffolds running the workload on the test system, and (3) logs the results for offline analysis. In this section, we provide more details on how it serves as a blueprint for test system interface development. The benchmark driver module has a base class which is to be derived by individual test system interface implementations to interface with the benchmarking framework. This base class defines the APIs for different operations for data scope selection and graph analysis. By default the base class implementation marks various operation implementations as ‘unsupported’. Individual test system interfaces can override these operation methods in their respective derived test system interfaces, if they support that operation. Different test systems may support a particular graph or data scope operation differently. Some test system drivers expose a single API to execute the desired operation, while some test system drivers may require using multiple APIs together to achieve the functionality, which may also go to the extent of writing our own implementation from scratch. Since we intend to benchmark the test system, and not our own algorithmic implementations, we use the following rubric to determine if we should implement the API manually for a graph or data scope operation for a given test system. (1) Does the test system driver expose a single API or a set of APIs to be used together to execute the operation? If yes, use it/them. If no, continue to 2.

(3) Does the implementation around the available constructs require iterating over the data scope points manually? If yes, do not implement the interaction. If no, use the constructs and implement the interaction.

B

EASE OF IMPLEMENTATION RUBRIC

In this section, we describe how we ranked our test systems based on the expressiveness metric of Ease of Implementation. Each author individually rated the test systems they developed the interfaces for, on the following criteria along a 5 point Likert scale: (1) How easy was it to find the documentation for the test system and its driver on the system website? (2) How comprehensive is the documentation? Does it provide examples? (3) How easy was to find resources for help on forums like stack overflow or the system’s own forum? (4) How quick are the developers to respond to queries? How useful or helpful are their responses? This was followed by a round of discussion and reconciliation among the authors to arrive at the final ratings, which we discuss in the paper.

15

Related documents

• TrajectoryDB: A New Database for Agent Trajectories#673620
• Preserving contextual information in cultural heritage metadata through multidimensional knowledge graphs#673621
• X-DigCheck: Co-Evolving Application Profiles and Knowledge Graphs, Demonstrated on the RTI Documentation of Rupe Magna#673622
• Open Tabular Insight Extraction: Where Do We Stand, and Where Should We Go?#673623
• Protocol effects on feature-based hardware-Trojan detection across Trust-Hub families#673624