BiJuTy: An Interactive HPC-Aware Big Data Cluster Lifecycle Manager and Performance Assessment Utility for JupyterHub
arXiv:2606.24412v1 [cs.DC] 23 Jun 2026
Apurv Deepak Kulkarni1,2,3[0009−0003−3110−6801] , Jan Frenzel2,3[0009−0007−5755−1427] , and Siavash Ghiasvand1,2,3[0000−0001−6627−0159] 1
Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI) 2 Center for Interdisciplinary Digital Sciences (CIDS) 3 TUD Dresden University of Technology, Dresden, Germany {apurv.kulkarni,jan.frenzel,siavash.ghiasvand}@tu-dresden.de
Abstract. The increasing demand for data processing has created a pressing need for access to high-performance computing (HPC) systems. Nevertheless, leveraging these systems to execute complex big data processing workflows remains a significant challenge, especially for beginners. This work presents BiJuTy, a solution designed to bridge the accessibility gap for big data workflows on HPC systems within the Jupyter ecosystem. By providing an interactive and user-friendly interface, BiJuTy simplifies cluster lifecycle management and performance assessment, making it more accessible on HPC systems to beginners and experienced users alike. The solution is presented as an interactive interface that guides the user through the entire process, from setting up the cluster configuration to carrying out initial performance assessments. Additionally, the framework enables seamless management of multiple clusters directly within the Jupyter Notebook interface, eliminating the need to switch outside of working environment. The collection of performance metrics from various sources further simplifies the optimization workflow. Furthermore, an illustrative example is provided to demonstrate how BiJuTy can be deployed to optimize the performance of a big data processing application. This example showcases how the entire big data processing lifecycle can be iteratively executed and optimized in just a few clicks, helping to reach the goal of optimization easily and interactively. By facilitating such workflows, this work contributes in bringing the field of big data computing and high-performance computing one step closer to the goal of seamless interaction and usability. Keywords: Big Data · Jupyter · HPC · Slurm · Apache Spark · Apache Flink
1
Introduction
The advancements in the field of machine learning and artificial intelligence (AI) increasingly motivate interdisciplinary researchers to use sophisticated analysis
2
Kulkarni et al.
models. This usage necessitates the access to larger computational resources such as high performance computing (HPC) clusters. However, despite the wide-range of efforts and rapid advancements in democratizing HPC resources, there remains a significant gap which stems from the batch-processing nature of HPC clusters in contrast to the interactive approach of users who are coming from other fields. Currently, the most accessible computation resources are the commercial cloud providers which provide HPC resources with a minimal barrier via their own graphical user interface (GUI) [21]. However, despite the widespread availability and accessibility of cloud systems, HPC clusters remain a significant computation platform even for non-HPC workloads, due to reasons beyond user convenience. HPC clusters are purpose-built and have optimized architecture to provide extreme performance on specialized hardware within standardized environments while maintaining consistency across multiple runs. Additionally, increasing demands for data privacy and control motivate many institutions to maintain their own on-premise HPC clusters [1,24]. These systems mainly run on open-source software and are tightly-coupled into the research environment over decades. JupyterHub [26] is one of the successful software infrastructures that bridges the gap and provides interactive access to HPC resources, particularly for users without prior experience, and is being actively used on both cloud and HPC clusters. Its GUI requires only a modern web browser and serves as a single point of entry, while hiding all configurations from the user, which plays a significant role in the popularity of JupyterHub among the vast majority of users. Currently, many HPC centers provide JupyterHub as an alternative to the classical command line interface (CLI)-based access to their HPC clusters. Although this interface cannot fully replace the CLI-based approach, since it does not align with the batch-processing nature of HPC systems, it provides an easy access to computational resources and assists the new users to get onboard and familiarize themselves with the HPC environment with minimal efforts. Among the advanced analytical approaches, big data frameworks (BDF) [34] are not exceptions. BDFs are software environments and toolsets that are designed to efficiently ingest, store, process, and analyze massive volumes of data. These frameworks have a wide range of usability in various interdisciplinary domains and often they need HPC resources to properly handle the vast amount of data. However, setting up, managing, and monitoring BDF on HPC systems is itself a challenge for beginners due to a high level of complexity imposing a significant barrier on the path towards effective use of BDF on HPC systems. Beginners would significantly benefit from graphical representations of the configuration, guidance by predefined UI actions and simplified access to performance metrics. Thus, this work proposes BiJuTy 4 a comprehensive infrastructure as code (IaC) toolbox that integrates into Jupyter ecosystem and extends its capabilities to (1) manage individual users’ configurations for various big data frameworks through a user-friendly interface, (2) control the lifecycle of standalone 4
https://github.com/scads/bijuty
BiJuTy
3
big data clusters (BDC), (3) visualize Spark/Flink job information, (4) visualize framework and system-level metrics along with external metric systems, and (5) manage configuration and workflow on a per-user and per-framework basis. BiJuTy streamlines interaction with big data frameworks on HPC systems, making it accessible to novice users and efficient for experts. It automates the set up, monitor, and tear down of BDCs. Users will obtain performance-metric data with a single click, facilitating rapid debugging and deeper insight into underlying operations. It enables users to manage configurations on an individual and perframework basis. All of these are provided directly from within the familiar JupyterHub ecosystem.
2
Literature Review
Several efforts focus on simplifying the access to HPC resources as a vital prerequisite for executing complex analysis pipelines which often need to handle large amount of data [8,32,20,39]. Since this work is focusing on BDF, once an intuitive access to HPC resources is established e.g., via JupyterHub, the next step would be deployment of BDCs on the underlying HPC environment. For deployment of BDCs on HPC clusters, many CLI tools are available for various HPC scheduling systems and either generic support for various BDC [14,13,19] or tailored solutions for Apache Spark-based clusters [22,10,31,7]. These tools reduce repetitive manual steps and streamline cluster setup for experienced practitioners. However, this work focuses on automated mechanisms that minimize user intervention, thereby enabling researchers to concentrate on their core scientific objectives while abstracting the complexities of BDC management through systematic automations. Various configuration management tools such as Chef [25], Puppet [23], Ansible [30], and Salt [38] exist which can assist experienced users in installing and managing BDCs. However these tools are primarily designed for cloud environments with persistent infrastructure and full administrative access, rather than for HPC centers with resource scheduling and job-based execution environments. Consequently, they do not assist in provisioning BDCs within such environments and are targeted at experienced users. Other automation tools such as Pulumi [29] address the provisioning of underlying infrastructure, however most HPC clusters particularly in research institutes mainly rely on Slurm [33] or PBS [2] as job scheduler which therefore requires a hybrid IaC approach. Tools such as Open-OnDemand [6] on the other hand have their specific hardware and software requirements and need deep integration into the software stack of the HPC clusters, which is out of scope for normal HPC users. Therefore, although the above mentioned tools may simplify the deployment and provisioning of BDCs, none of them can provide an accessible approach for beginners. Besides community efforts such as BDWatchdog [11], that provides an accessible approach for monitoring and profiling BDF, many attempts have also been made by maintainers of BDF such as Spark [5] and Flink [4] to make monitoring of these frameworks more accessible and user-friendly.
4
Kulkarni et al.
Another approach, that is particularly beneficial, is maintaining JupyterHub as the central entry point while integrating the relevant features of BDCs directly into Jupyter ecosystem. With this approach, users can access the required functionality directly from within the familiar JupyterHub interface. Streaming Jupyter integration [17], as an example, enables interactive execution of Flink SQL jobs in Jupyter notebooks, SparkMonitor [18] embeds the resource monitoring of Spark resources into Jupyter notebooks, and JupyterLab Spark [40] integrates the native Spark web UI into the JupyterHub interface. Although these approaches contribute to the accessibility of BDCs monitoring, to the best of our knowledge, no tool exists that combines configuration, provisioning and monitoring of BDC within JupyterHub interface.
3
BiJuTy
BiJuTy, as described before, is an Interactive HPC-Aware BDC lifecycle manager and performance assessment utility that integrates its features seamlessly into JupyterHub providing a consistent and flexible environment. Current implementation is compatible with all HPC clusters that utilize Slurm batch job scheduler. 3.1
Workflow
The workflow, as illustrated in Fig. 1, starts by initiation of JupyterHub as a Slurm job on HPC cluster. Then, an appropriate pre-configured kernel (e.g. ’bigdata-kernel’) should be selected which in turn, opens a notebook. In the notebook, the user imports the BiJuTy library in one of the cells (ideally at the top). Upon completion of the import, the user is presented with a interface that displays all existing settings. Subsequently, the user can proceed to the next stages of the workflow. PHASE I: SETUP Job Launch
Kernel Selection
Library Import
Dashboard
JupyterHub HPC
bigdata-kernel
import bijuty
GUI Config
PHASE II: EXECUTION Validation vs. SLURM Limits
Start the Cluster
Monitor Metrics & Jobs
Stop the Cluster
Fig. 1. BiJuTy schematic workflow
The interface presents different options and buttons to manage the standalone big data cluster, as shown in Fig. 2. It consists of different sections such as “Cluster Configurator”, “Resource Allocation Overview”, “Cluster Controls”, “Performance Metric” and interface log panel. Sections are (de-)activated accordingly to guide the user through out the process. Tooltips are also offered where additional information or context is required.
BiJuTy
5
Fig. 2. BiJuTy interface
The “Cluster Configurator” section consists of basic options required to setup a standalone cluster. The default values are set such that they are sufficient for performing basic computation for testing purpose. Parameters in this section utilize generic names to ensure adaptability to various naming conventions used in major BDFs. For example, Apache Spark uses the notion of a Driver [35] for the component which steers distributed execution while such a component is named JobManager [36] in case of Apache Flink. The interface uses the term Coordinator instead to emphasize its role. User may initialize the standalone cluster using the provided default values or can use their own configuration template for a more customized usage, with an optional unique path for setting location of initialization. The section “Resource Allocation Overview” provides a high-level visual overview about resource distribution among different processes and nodes inside Slurm job. The visualization projects a dynamic overview according to the user’s configuration. A standalone cluster can be started and stopped from the Cluster Control section. Once the required parameters are set, the configuration can be loaded to the current environment and a standalone cluster can be started by clicking on “Start Cluster” button. In current implementation, only standalone cluster is supported for Apache Spark and Apache Flink frameworks. The integrated web interfaces provided by each BDF can be also accessed using the relevant buttons directly from BiJuTy interface. Following a successful start of the cluster processes, the “Performance Metric” section is enabled for interaction, providing a performance overview. All desired processes’, framework’s and external metrics can be monitored in real-time using the Performance Metrics section.
6
Kulkarni et al.
These visualizations and overviews provide users with a deeper understanding of big data processing workflow. BiJuTy allows users to execute the workflow iteratively to experiment with various configurations. Furthermore, multiple frameworks such as Apache Flink and Apache Spark can be used and managed simultaneously within a single interface. Users can manage these views by adding or removing tabs via the buttons located in the top-right corner. Fig. 3 further extends the schematic workflow illustrated in Fig. 1 by incorporating the interactions between the JupyterHub, Slurm and BDC. The JupyterHub Spawner instructs Slurm to schedule a job. On one of the allocated resources (green boxes), the JupyterHub Spawner starts a Jupyter process, in which a BiJuTy session is started. Via the BiJuTy session (orange), a user can configure and start the processes for the BDC (gray boxes). BDC Master schedules BDC tasks
Jupyterhub Spawner
BiJuTy session
BDC Worker1
BDC Worker2
Available Infrastructure
runs user notebook
runs BDC tasks
runs BDC tasks
Slurm scheduler
Compute node 1
Compute node 2
Compute node 3
Compute node 4
Compute node 5
Available Infrastructure
Available Infrastructure
part of Slurm job
part of Slurm job
Available Infrastructure
part of Slurm job
Fig. 3. Interactions between JupyterHub, Slurm, BiJuTy and the BDC.
3.2
Implementation Psutils
Browser Frontend WidgetView
Slurm
Ext. Metrics BDC
Python Kernel WidgetModel
Widget
Comms
Comms Jupyter WebServer
Fig. 4. BiJuTy integration in JupyterHub (adapted from [28]).
The BiJuTy package follows Jupyter’s widget architecture, as shown in Fig. 4. This architecture facilitates maintaining synchronized widget state between the Python kernel and the frontend, thus helping in decoupling python backend from GUI. The client side GUI is implemented using IPyWidgets [28]. The widget that is loaded in the kernel interfaces with backend components such as Slurm, psutil, external metric collectors and BDF backend. The widgets used in the BiJuTy package have representations both in Python kernel (Widget) and the frontend (WidgetModel). A bidirectional state synchronization exists between these objects through Jupyter’s communication protocol (Comms) [27]. Displaying a widget creates a WidgetView instance representing that WidgetModel. Any
BiJuTy
7
update from the WidgetView goes to the WidgetModel and synchronizes with the Widget. Similarly, Python kernel outputs, such as data processing output or plot, are sent to the Widget which then synchronizes with the WidgetModel to update the WidgetView. The Slurm interface present in this package extracts the job information using Slurm’s native commands and the environment variables. In Slurm-based HPC environments, software packages are typically installed centrally on shared filesystems and made available cluster-wide through environment modules or software stacks. Since these frameworks may be accessed concurrently by multiple jobs and users, it is crucial to ensure that the individual user configuration remain isolated and do not interfere with each other. To address this requirement, BiJuTy uses a template approach from BigDataFrameworkConfigure [16], where the Python kernel derives a configuration from framework-specific templates and values that are present in the interface, and initializes an isolated configuration along with setting environment variables required by BDFs. The Python kernel also uses the configuration to invokes bash scripts of the selected BDF, in order to start or stop the BDC. The metrics are collected asynchronously across three different levels, namely: Process, Framework and External levels. The Process level metrics are collected using psutil [12] Python package, that collects metrics such as CPU and Memory usage. Additionally, metrics such as heap memory usage, garbage collection (GC) statistics, and other relevant metrics are collected using the API endpoints provided by BDFs. The metrics on External level are collected from external services such as Pika [9]. Pika in this case provides node level metrics including Power usage and IO Read/Write among the others. All these metrics are plotted and regularly updated at an interval that displays system status near real-time.
4
Results and discussion
This section describes a workflow for deploying the BiJuTy package and utilizing it for the use-case of assessing performance of and application through an experimental evaluation. The experiment follows the optimization cycle illustrated in Fig. 5. Initially, the standalone cluster is started from the BiJuTy interface and the application is submitted with a baseline configuration. Application performance is then observed. If the observed performance is unsatisfactory, the cluster is stopped and restarted from the interface with a revised configuration; the application is subsequently resubmitted to the reconfigured cluster, and performance metrics are reassessed. For this demonstration, the cycle is limited to a single iteration. BiJuTy’s applicability is demonstrated on the Barnard system of the ZIHClusters [41], which uses Slurm for batch scheduling and provides shared access to Apache Spark installation via Module System [15]. A JupyterHub session is launched and attached to a Slurm job with 8 CPUs and 8GB of total memory on single node. Upon successful spawning of JupyterHub, the BiJuTy package can be installed by running the following command inside a notebook cell: pip
8
Kulkarni et al. Start
Finish
Start cluster Revise configuration
Submit application
Stop cluster
Observe performance
Yes
No
Interaction in BiJuTy interface
Satisfactory?
Fig. 5. Optimization use-case workflow using BiJuTy.
install git+https://github.com/scads/bijuty@main. This step needs to be done only once for any Python kernel that is being used. Following the installation, the web browser page is reloaded and the Python kernel is restarted, after which, this kernel is used for all subsequent operations. Apache Spark’s PySpark [37] is used to showcase how BiJuTy can help in running the complete big data processing workflow and assessing the impact of configuration changes on performance metrics. The workload 5 comprises of a data processing and analysis on Uber taxi pickup data [3] comprisng 4+ Million rows of location, date and time and base codes of compnies affiliated to Uber pickup. The example involves grouping and aggregation computation.
(a) Phase I
(b) Phase II Fig. 6. Cluster Parameters
The BiJuTy interface is launched by importing the bijuty package (import bijuty). In this interface, Apache Spark is chosen as a framework of choice for the experiment. The evaluation is conducted in two phases, one with lower resources and one with higher resources with the aim of checking if the higher resource really improves the performance or not. For phase I, parameters are set to minimum before deploying Apache Spark standalone cluster, as illustrated in Fig. 6a. Upon successful startup of the cluster, the section “Performance Metric” activates, allowing direct access to performance data on different levels, as discussed before. Phase II of the experiment can be executed with increased resources by stopping the cluster, restarting it with revised parameter values (as 5
https://github.com/scads/bijuty/tree/main/example
BiJuTy
9
presented in Fig. 6b), and re-running the same PySpark code. It can be observed that the workload executes with increased memory allocation and CPUs count within each compute unit (Executor in Apache Spark [35]). Fig. 7 depicts the “Performance Metric” section of Phase I and Phase II. It can be observed that Phase II completes the computation in less than half of the wall-clock time of Phase I (timings on X-axis of Execution Time plot), even though the execution times (i.e. cumulative CPU times) remain comparable across both phases. GC Time increases in Phase II, likely due to higher objectallocation rates associated with the increased core count. The memory pressure is comparable for both phases. Collectively, these finding indicate that increased parallelism is effective strategy for this use-case, as the reduction in wall-clock time outweighs the minor increase in garbage collection overhead. Phase I
Phase II
Fig. 7. Performance Metrics for Executor
This use-case of application optimization demonstrates BiJuTy’s capability to streamline the initial workflow for performance tuning by integrating cluster lifecycle management and performance metric observation directly within the Jupyter Notebook, thereby enhancing accessibility for beginner and experienced users alike. However, despite its capabilities, BiJuTy has some limitations that can impact its efficiency in certain scenarios. Currently, it is best suited for relatively small applications, as continuous plot updates can affect the availability of the Python kernel. Furthermore, IFrame and cross-origin restrictions within IPyWidgets prevent direct embedding of the BDF native Web-UI in the BiJuTy interface. Currently, BiJuTy only supports Apache Spark and Apache Flink. While these limitations do not compromise functionality, they may slightly affect user experience.
10
5
Kulkarni et al.
Conclusion
This work introduced BiJuTy, a comprehensive yet intuitive tool that streamlines the setup, management, and monitoring of BDF on HPC systems using a fully automated mechanism, right inside the familiar JupyterHub’s (Jupyter Notebook) graphical user interface. BiJuTy eliminates the entry barrier for beginners and makes the process efficient for experts, encouraging broader adoption of data processing on HPC resources. By providing performance information, users can make informed decisions about resource allocation and code optimization, an important measure which is often overlooked by cross-domain users. This capability is a concrete step towards a more accessible HPC environments, which is particularly important as data-processing demands surge with the growth of data intensive workloads. Future work will focus on addressing the limitations of the current implementation, including improving the retrieval of additional data from the BDF Web-UI to provide more comprehensive insights. Additionally, optimizing widget performance will be a key area of focus to improve overall usability and responsiveness. Furthermore, developing an implementation that is independent of the current Python kernel is also planned, allowing for greater flexibility and compatibility, which will contribute to a more robust and user-friendly experience. Acknowledgments. The authors acknowledge the financial support by the Federal Ministry of Research, Technology and Space of Germany and by Sächsische Staatsministerium für Wissenschaft, Kultur und Tourismus in the programme Center of Excellence for AI-research “Center for Scalable Data Analytics and Artificial Intelligence Dresden/Leipzig”, project identification number: ScaDS.AI The authors gratefully acknowledge the GWK support for funding this project by providing computing time through the Center for Information Services and HPC (ZIH) at TU Dresden. Disclosure of Interests. The authors have no competing interests to declare that are relevant to the content of this article.
References 1. Agrawal, N., Binns, R., Van Kleek, M., Laine, K., Shadbolt, N.: Exploring design and governance challenges in the development of privacy-preserving computation. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. CHI ’21, Association for Computing Machinery, New York, NY, USA (2021). https://doi.org/10.1145/3411764.3445677 2. Altair Engineering, Inc.: OpenPBS: Open PBS Professional, https://www. openpbs.org/, accessed: 2026-05-23 3. Andrew Flowers, Reuben Fischer-Baum, D.M.: Uber TLC FOIL Response, https: //github.com/fivethirtyeight/uber-tlc-foil-response, accessed: 2026-05-28 4. Apache Software Foundation: Flink Operations Playground - Flink WebUI, https://nightlies.apache.org/flink/flink-docs-stable/docs/try-flink/ flink-operations-playground/#flink-webui, accessed: 2026-03-15
BiJuTy
11
5. Apache Software Foundation: Web UI - Spark 4.0.0 Documentation, https:// spark.apache.org/docs/latest/web-ui.html, accessed: 2026-03-15 6. Azure CycleCloud Team: cyclecloud-open-ondemand, https://github.com/ Azure/cyclecloud-open-ondemand, accessed: 2026-03-15 7. Baer, T., Peltz, P., Yin, J., Begoli, E.: Integrating apache spark into pbs-based hpc environments. In: Proceedings of the 2015 XSEDE Conference: Scientific Advancements Enabled by Enhanced Cyberinfrastructure. XSEDE ’15, Association for Computing Machinery, New York, NY, USA (2015). https://doi.org/10. 1145/2792745.2792779 8. Chalker, A., Franz, E., Rodgers, M., Dockendorf, T., Johnson, D., Sajdak, D., White, J.P., Plessinger, B.D., Zia, M., Gallo, S.M., Settlage, R.E., Hudak, D.E.: Open ondemand: State of the platform, project, and the future. Concurrency and Computation: Practice and Experience 33(19), e6114 (2021). https://doi.org/ 10.1002/cpe.6114 9. Dietrich, R., Winkler, F., Knüpfer, A., Nagel, W.: Pika: Center-wide and job-aware cluster monitoring. In: 2020 IEEE International Conference on Cluster Computing (CLUSTER). pp. 424–432 (2020). https://doi.org/10.1109/CLUSTER49012. 2020.00061 10. DrudgeCAS: spark-in-slurm, https://github.com/DrudgeCAS/spark-in-slurm, accessed: 2026-03-15 11. Enes, J., Expósito, R.R., Touriño, J.: Bdwatchdog: Real-time monitoring and profiling of big data applications and frameworks. Future Generation Computer Systems 87, 420–437 (2018). https://doi.org/10.1016/j.future.2017.12.068 12. giampaolo: psutil: Cross-platform lib for process and system monitoring in Python, https://github.com/giampaolo/psutil, accessed: 2026-05-18 13. glennklockwood: myhadoop, https://github.com/glennklockwood/myhadoop, accessed: 2026-03-15 14. HPC UGent: hanythingondemand, https://github.com/hpcugent/ hanythingondemand, accessed: 2026-03-15 15. Modules. https://hpc-wiki.info/hpc/Modules, accessed: 2026-05-18 16. Jan Frenzel, A.D.K.: Bigdataframeworkconfigure (2025), https://gitlab.hrz. tu-chemnitz.de/scads.ai/bigdataframeworkconfigure, accessed: 2026-05-22 17. Kołakowski, G.: streaming-jupyter-integrations: Jupyterlab extensions for streaming data processing (flink sql), https://github.com/getindata/ streaming-jupyter-integrations, accessed: 2026-03-15 18. Krishnan, R., SWAN Team at CERN: Sparkmonitor, https://github.com/ swan-cern/sparkmonitor, accessed: 2026-03-15 19. Lawrence Livermore National Laboratory: Magpie. https://github.com/LLNL/ magpie (2026), accessed: 2026-03-15 20. Liberati, F., Marino, T.M.P., Bottoni, P., Canestrelli, D., Castrignanò, T.: Hpc-t-assembly: a pipeline for de novo transcriptome assembly of large multispecie datasets. BMC Bioinformatics 26 (2025). https://doi.org/10.1186/ s12859-025-06121-4 21. Microsoft: Manage resources for Apache Spark cluster on Azure HDInsight, https://learn.microsoft.com/en-us/azure/hdinsight/spark/ apache-spark-resource-manager, accessed: 2026-05-23 22. NIH-HPC: spark-slurm, https://github.com/NIH-HPC/spark-slurm, accessed: 2026-03-15 23. Perforce Software, Inc.: Puppet - infrastructure automation and configuration management. (2025), https://www.puppet.com/, accessed: 2026-05-23
12
Kulkarni et al.
24. Pramanik, M.I., Lau, R.Y.K., Hossain, M.S., Rahoman, M.M., Debnath, S.K., Rashed, M.G., Uddin, M.Z.: Privacy preserving big data analytics: A critical analysis of state-of-the-art. WIREs Data Mining and Knowledge Discovery 11(1), e1387 (2021). https://doi.org/10.1002/widm.1387 25. Progress Software Corporation: Chef- infrastructure automation platform (2025), https://www.chef.io/, accessed: 2026-05-23 26. Project Jupyter: JupyterHub, https://jupyter.org/hub, accessed: 2026-05-23 27. Project Jupyter Contributors: Messaging in jupyter — jupyter_client 8.8.1.dev0 documentation, https://jupyter-client.readthedocs.io/en/ latest/messaging.html, accessed: 2026-05-19 28. Project Jupyter Contributors: Widget — ipywidgets documentation, https://ipywidgets.readthedocs.io/en/latest/examples/Widget%20Low% 20Level.html, accessed: 2026-05-18 29. Pulumi Corporation: Pulumi (2025), https://www.pulumi.com/, accessed: 202605-23 30. Red Hat, Inc.: Ansible - it automation platform (2025), https://www.ansible. com/, accessed: 2026-05-23 31. Roškar, R.: sparkhpc, https://github.com/rokroskar/sparkhpc, accessed: 202603-15 32. Samuel, J., Brennan-Tonetta, M., Samuel, Y., Subedi, P., Smith, J.: Strategies for democratization of supercomputing: Availability, accessibility and usability of high performance computing for education and practice of big data analytics. JBDTP Professional 1(1), 51–65 (2022). https://doi.org/10.54116/jbdtp.v1i1.16 33. SchedMD LLC: SLURM: Simple Linux Utility for Resource Management (2025), https://slurm.schedmd.com/overview.html, accessed: 2026-05-23 34. Shahnawaz, M., Kumar, M.: A comprehensive survey on big data analytics: Characteristics, tools and techniques. ACM Comput. Surv. 57(8) (Mar 2025). https://doi.org/10.1145/3718364 35. The Apache Software Foundation: Cluster mode overview - spark 4.1.1 documentation. https://spark.apache.org/docs/latest/cluster-overview.html, accessed: 2026-05-18 36. The Apache Software Foundation: Jobs and scheduling. https://nightlies. apache.org/flink/flink-docs-stable/docs/internals/job_scheduling/, accessed: 2026-05-18 37. The Apache Software Foundation: PySpark overview - PySpark 4.1.1 documentation. https://spark.apache.org/docs/latest/api/python/index.html, accessed: 2026-05-18 38. VMware, Inc.: Salt- intelligent it automation software (2025), https:// saltproject.io/, accessed: 2026-05-23 39. Wu, D., Zhu, L., Xu, X., Sakr, S., Sun, D., Lu, Q.: Building pipelines for heterogeneous execution environments for big data processing. IEEE Software 33(2), 60–67 (2016). https://doi.org/10.1109/MS.2016.35 40. Zhang, M.: jupyterlab spark: A jupyterlab extension to show spark application ui in a jupyterlab panel, https://github.com/manuzhang/jupyterlab_spark, accessed: 2026-03-15 41. ZIH Team: HPC Systems, https://tu-dresden.de/zih/hochleistungsrechnen/ hpc, accessed: 2026-05-17