RSE of a Quantum Transport Code and its Effects Christoph Conrads
1∗ , Edoardo Di Napoli
2
1 [email protected] 2 [email protected]
arXiv:2605.21334v1 [cs.SE] 20 May 2026
∗ Corresponding author
Jülich Supercomputing Centre, Forschungszentrum Jülich, Germany Abstract: This paper presents our research software engineering (RSE) experiences over two years with libNEGF, a quantum transport code. We describe practical approaches to code quality assurance–including continuous integration, automated testing, and compiler warning correction–and performance engineering through continuous benchmarking. Our systematic application of these practices revealed critical defects: uninitialized memory reads, out-of-bounds writes, and notably, a misunderstood mathematical model in our boundary condition handling. We also document how continuous benchmarking exposed performance regressions caused by HPC system configuration changes. Our findings provide data points suggesting that a dangerous class of defects–equivalent to undefined behavior in C/C++ and processor-dependent behavior in Fortran–is as prevalent in Fortran scientific codes as elsewhere. While libNEGF is implemented in Fortran, most recommendations are applicable to scientific software regardless of implementation language, and they can be implemented selectively or in their entirety for both new and existing projects. Keywords: continuous integration, continuous benchmarking, high-performance computing, research software engineering, software sustainability, version control
1
Introduction
Within computational physics, density functional theory (DFT) is a model for examining the electronic structure of many-body systems. The density functional-based tight binding (DFTB) approximation of DFT is among the most popular approaches for simulation codes. libNEGF is an implementation of the non-equilibrium Green’s function (NEGF) method and enables DFTB-based simulations to compute quantities like the electron density or the Hartree potential [PPSC08]. The NEGF ansatz delivers sufficiently accurate results for large numbers of atoms for which, e. g., ab-initio methods are too expensive in the foreseeable future. libNEGF is implemented in Fortran with GPU code written in CUDA. Under the umbrella of the EoCoE-III project,1 co-development with Forschungszentrum Jülich started in 2024 with the goal of achieving exascale performance. This article describes the research software engineering (RSE) efforts of this project and the outcomes. The general aim of this document is to provide other RSE practitioners with data points to improve their decision making and priorization efforts. 1 https://www.eurohpc-ju.europa.eu/research-innovation/our-projects/eocoe-iii en
The first part of the article concerns software engineering. We start by presenting the mindset with which we approach the development tasks before discussing a variety of topics culminating in the introduction of continuous integration (CI). With CI in place, the code can be built automatically and static as well as dynamic code analysis can be run at the push of a button. In the second part, we describe our setup for job submission and continuous benchmarking. This section requires that the code can be built automatically; if CI is not available, then code is more likely to fail to build on target systems. Next, we talk about four highlights of our RSE efforts including tracking server-side performance changes of HPC system and a diagnosed bug. We close with a section about best practices that we should have implemented. We do not expect readers of this article to be able or willing to implement all of the suggested measures for a variety of good reasons. We propose instead to read this paper to get an impression of the problems that are likely to be present in a code base, how they affect the correctness and performance, and to which degree these problems can be resolved. Then recommendations can be applied selectively as the reader sees fit. In terms of minimum viability, we recommend to set up CI first with the reader’s scientific software at least being successfully compiled in a container in the case of compiled languages or with the software successfully running in a container in the case of interpreted languages. With CI set up, a single gateway for quality assurance is in place with many options to extend it. For large legacy code bases, finding out how to reliably build the code in a container together with all of its dependencies may be a time-consuming task in itself. The outcome is a reusable and up-to-date description of the build and its dependencies accessible to all project members. The article is focused on CI and CB with a focus on compiled languages. Readers need to know how to write and fix code, how to build their code and how to fix broken builds, how to write tests, and how to use the revision control system of their choice.
2
Software Engineering
In this section we will focus on software engineering (SE) aspects. For the purposes of this section, this is anything but performance engineering. We will explain which steps we took to turn libNEGF into a software that can be deployed with confidence at the push of a button on a variety of target systems. The mindset underlying our SE efforts is explained first before we dive into code and code history challenges that have to be resolved. Then, the code development is moved onto a code forge (e. g., GitLab or GitHub) where a CI pipeline ensures a consistent quality: the code formatting is checked, the code is built and tested in a variety of configurations. For completeness, we want to mention that CI is possible without a code forge as well. Finally, we discuss the beneficial effects of fixing compiler warnings.
2.1
Mindset
We assume our simulations to contain a special class of defects called untrapped errors which we introduce in first. Readers will quickly note that this concept originates from programming language design and may not apply to the language that they are using. We ask readers to
bear with us because in the following subsection we explain how mixing executable code from different sources (e. g., through the use of libraries) can cause untrapped errors even in safe languages. 2.1.1
Untrapped Errors caused by Code Defects
The mindset underlying our RSE activities is fundamental skepticism [OR10, §2.2] applied to a code written in an unsafe language [Car04, p. 2]: It is useful to distinguish between two kinds of execution errors: the ones that cause the computation to stop immediately, and the ones that go unnoticed (for a while) and later cause arbitrary behavior. The former are called trapped errors, whereas the latter are untrapped errors. A program fragment is safe if it does not cause untrapped errors to occur. Languages where all program fragments are safe are called safe languages. Therefore, safe languages rule out the most insidious form of execution errors: the ones that may go unnoticed. Examples of untrapped errors are reads of uninitialized variables, out-of-bounds memory accesses or dereferencing an invalid pointer. Examples of unsafe languages are C, C++, and Fortran independently of the specific standard used. In C and C++ terms, untrapped errors are a cause of undefined behavior; in Fortran they are a cause of processor-dependent behavior. Untrapped errors are the cause of many well-documented real-world software problems [Lat11, Reg10] including many security vulnerabilities. Possible effects relevant to scientific computing include crashes or silent corruption of results. These effects can occur but they do not necessarily happen. They may be triggered when changing the compiler, the compiler version, the compiler’s optimization level, something else in the software stack, or the machine on which the code is compiled. For the sake of clarity, we provide real-world examples of untrapped errors here. We dealt once with a C++ program that was not terminating on x86 CPUs whereas on ARMv7 CPUs, the same code would reliably terminate but with an incorrect output. The cause was an uninitialized variable.2 Another example of untrapped errors arose when two tests written in Fortran and C, respectively, were added to the libNEGF test suite. The two tests were run successfully in the CI environment in six different configurations on an x86-64 CPU with GCC (see Section 2.7 for details), they ran successfully on the supercomputer JUWELS Booster when building with the Intel compiler suite (2021 release) and GCC but the Intel compilers 2023 release correctly diagnosed a deallocation of unallocated memory. To verify the presence of a problem in the code, the code was rebuilt with GCC and the undefined behavior sanitizer enabled.3 The sanitizer did not signal problems but the tests suddenly failed for numerical reasons. These are two typical examples showing how untrapped errors are present in real-world scientific codes, they have no clear set of symptoms associated with them, and they may break the program at any time. 2 https://gitlab.inria.fr/melissa/melissa-sa/-/commit/dc1b1263d69d2dc81d54faec460fac97c441baa4 3 The appropriate choice would have been the address sanitizer.
The analysis tool named Stack found 40 % of all Debian packages to contain at least one instance of untrapped errors [WZKS13] and another study found 16 % of the Debian packages under consideration to contain undefined integer behavior [DLRA15]. Coverity’s static analysis tool arrived at an average defect density of 0.45 per 1,000 lines of code [Cov11]. These references are not meant to be comprehensive but just to show how pervasive this problem is. These numbers are emphatically lower bounds. Proving program correctness (and with it, proving the absence of untrapped errors) is in general not possible. Thus, these studies focus on a particular set of problems with simplifying assumptions. When focussing on scientific software, we do not believe these numbers to improve significantly. As a supporting data point we mention here the static analysis of CERN code resulting in 40,000 bug fixes in a code base of ca. 3.5 million lines of C++ code [Cla11]. Similar but less recent examples exist for Fortran, too [OR10, §4.5]. Given the statistics above and our fundamental skepticism, we always assume that our software contains undetected defects. This assumption quickly turned out to be justified. To date, our quality assurance efforts detected signed integer overflows, several out-of-bounds writes, three cases of reads of uninitialized memory, a memory leak, double frees (deallocation of unallocated pointers), and dereferencing of null pointers. 2.1.2
Untrapped Errors caused by Build Defects
All data in a computer are represented by zeros in ones in computer memory. Meaning is assigned through the operations applied to the data: bytes processed by the CPU’s instruction decoder are taken to be CPU instructions, bytes processed by the floating-point unit are taken to be floats, and bytes consumed by a read (load) instruction are taken to be a reference, and so on [BO11, §2, §3]. The use of different programming languages within one project is a common practice in scientific computing [BCC+ 08]; one of the most common examples is Python developers calling LAPACK which is written in a C-derivative, Fortran, and/or assembly. Programs written in different languages can always interact through the use of files but for performance reasons, it is much more common to have code from different sources run as part of the same program. If this is done, then all codes must agree on the representation and semantics of the data structures in memory, that is, they must all implement the same application binary interface (ABI) and this is not trivially fulfilled. For example, on 64-bit systems many compiled languages and interpreters use unsigned 64-bit integers for indexing (e. g., C, C++, and Rust) whereas others use signed 32-bit integers (e. g., Fortran). Sometimes ABI compatibility is broken (intentionally or accidentally) by compiler vendors4 and different versions of a library may induce ABI changes. On modern HPC systems, usually several compilers are available together with several libraries built in multiple configurations for each compiler. These library versions may not match the library version of the Linux distribution in use. If a build mixes incompatible libraries or if code is built with one instance of a library but then uses a different one at run-time, then the combined code is again broken. That is, if one has two defect-free pieces of code in different languages and they share data at the binary level, then the combined software may contain defects that are untrapped errors. Therefore, untrapped errors are a risk for all HPC users. 4 Example: https://gcc.gnu.org/wiki/Cxx11AbiCompatibility
2.1.3
The Mindset in Practice
In practice, a social challenge is likely to arise from this mindset. Most contributors in HPC have no formal training in software development and in our experience in some programming communities, the concept of untrapped errors is unheard of. Imagine that such a contributor is told that his software contains many undetected defects. The developer may counter the defect claims saying that that he fixed all of the bugs that he is aware of. Since there are no unresolved bugs in the issue tracker and since the simulation output is as expected, the software must be defect-free for all practical purposes. In this situation, it is necessary to explain the concept of untrapped errors and elaborate in detail on its possible consequences. The bugs that were fixed during development are only those untrapped errors that were triggered on the developers’ machines. In our experience, previous test efforts may provide only weak support for claims of correctness. In the case of libNEGF, there exists an extended test suite but the output of these tests is just compared against output computed by earlier libNEGF releases and a comparison with other data sources, e. g., measurements, may simply not be feasible. The original authors of a simulation software should not be blamed for this shortcoming because the very reason for the existence of simulation codes is often that conducting experiments is infeasible or undesirable [OR10, §1.1]. In our case, the simulation of material properties is a research topic just as much as the synthesis of these materials is [MF21]. The bottom line is that we suggest projects with unsafe languages to adopt a mindset that the code contains defects and to make all contributors aware of the possible consequences of untrapped errors. A large number of the bugs found in libNEGF since the project’s inception are an outcome of adopting this mindset. Users of safe language relying on code in different programming languages should be aware that their software stack may introduce untrapped errors.
2.2
Maintaining the Git History
In our experience most software projects nowadays use a revision control system (RCS), usually git. Among other things, git sets itself apart from other RCS through its ability to handle a history where a commit may have more than one immediate successor and more than one immediate predecessor in a single branch. In this section we will take a closer look at the git history. Figure 1 shows two months of a git history of a real-world HPC software project with arrows pointing in the direction of descendants of a commit (i. e., more recent commits are below their predecessors). Linear subgraphs are simplified and collapsed to a single node. The two commits at the top have common ancestors. All of the project contributors were regularly committing their code changes and occasionally pulling changes directly from their co-workers. One of the key properties and a major strength of this history is its accurate reflection of how a certain piece of code came into existence. Nevertheless, this particular history may be considered lacking because there is no assurance that commits on one side of the graph do not break changes on the other. Similarly, two sibling commits5 may contain changes to the same parts of the same file causing git to refuse to merge these changes automatically once these commits are 5 Two commits are considered siblings if they are contained in the same connected component of a graph but neither
is an ancestor nor a descendant of the other.
merged into one branch; this is an instance of a merge conflict. It is then the responsibility of the developer initiating the merge to resolve this conflict; this can be a challenge if he did not author the conflicting pieces of code. From the project management perspective such a history makes it difficult to answer the question if a certain commit contains a certain property (e. g., a known bug or a certain feature). Keep in mind that the graph in Figure 1 was simplified: the graph with its 41 nodes was generated from almost 100 commits. Finally, such a history may contain redundant commits as we will show now. Figure 2a shows what we will call an unrestricted git history in the following where each contributor can merge and add commits as he sees fit. Here the commits C and D are descendants of a commit B just like the commits X and Y. A developer then merged these two branches in merge commit M and added another commit E. A common mistake is contributors merging into the wrong branch. That is, let the branch at the bottom be the default branch and the branch at the top be the feature branch, then the graph in Figure 2a is a result of merging the feature branch into the default branch. In Figure 2c, the developer did the opposite and merged the default branch into the feature branch resulting in a merge commit N. Once the developer notices his mistake (his changes can nowhere be found in the default branch even after pushing), he will then perform the correct merge but retain the now superfluous commit N. In Figure 2b, the commits X and Y were rebased with git rebase on top of D and then the feature branch was merged into the default branch with git merge −−ff−only; ff stands for fast forward. X and Y are greyed out because they only exist in the feature branch and will be lost once the feature branch is deleted. Note that while a merge is performed, there is no merge commit. This strategy of maintaining the default branch is called a linear history. It is possible to create a merge request even when rebasing and this alternative is shown in Figure 2d. Here, the contributor again rebases but the merge is performed with git merge −−no−ff. This variant is called a semi-linear history. Note that feature branches can be squashed into a single commit before merging them. In terms of Figure 2, this means that X’ and Y’ are combined into a single commit before executing git merge. There exist other strategies to maintain git histories (e. g., Git flow6 or Git V7 ) but we present specifically these two because they demonstrate basic git workflows and can be enforced in GitLab.8 In our experience there exists a major social challenge with implementing any git strategy: many git users have no mind model of how git operates and cannot relate the git commands they execute to the resulting git history; this is true for IDE and command-liner users alike. Also some developers never look at the git history. For such users there is no benefit in any strategy but their workflow is impaired. Strategies requiring rebasing may be challenging to adhere to if there are parts of the source code that are frequently concurrently edited by developers. Then, changes to these pieces of code are likely to cause merge conflicts. Without the need to rebase, a contributor is forced to resolve merge conflicts at most once (when merging his feature branch into the default branch). With a rebase, merge conflicts have to be resolved for each commit to be rebased in the worst case when the same contested piece of code was changed in every commit. This can be frustrating and 6 https://nvie.com/posts/a-successful-git-branching-model/ 7 https://finitestate.io/blog/git-v-branching-model 8 https://docs.gitlab.com/user/project/merge requests/methods/
increase the risk of breaking previously working code. In general, the difficulty of rebasing and merging increases with the number of changes accumulated in a feature branch. Therefore, avoid large multi-purpose feature branches if possible and take precautions in the case of large-scale changes. Our recommendation is to take a look at a project’s git history and to decide which properties the history should have. Based on these properties and the make-up of the development team, a strategy should be chosen. We have a hard time giving advice more specific than this due to the large variety of projects. For example, a large number of contributors and a high commit frequency may make enforcement of a linear history infeasible. As we mentioned above, some contributors never looked at a git history. This is one of the reasons why their commit messages may turn out to be useless in hindsight. We have the following recommendations: contributors should at very least be aware of the intended format of a commit message with a caption of 50 characters and a message body with text in the present tense; the message body is separated by an empty line from the caption [Pop08]. The commit can contain keywords that will make GitHub and GitLab close all issues mentioned in the commit once it is added to the default branch.910
2.3
Transition to a Code Forge
At the start of the EoCoE-III project, the libNEGF source code had already been under version control for more than a decade and a GitHub project existed for libNEGF. Nevertheless, development efforts were not being organized using the code forge and consequently, decision making and the origin of data are sometimes not traceable. As an example, one of the key design aspects of the GPU code is its manual, synchronous memory management. The original design envisioned here a way to fully utilize GPU memory but the motivation for this decision is evident neither from the source code nor from the GitHub issues. To ensure the survival of such project-specific knowledge, the following development workflow was implemented: every new feature and bug is first discussed in an issue, then a feature branch and an associated pull request (merge request in GitLab lingo) are opened, and finally new code is merged only after assuring its quality (e. g., by reviewing it). In practice, there is a social aspect when using code forges. Contributors may habitually use, e. g., e-mail or their employer-internal chat system to discuss issues. These discussion should be moved to the code forge because they may not be accessible to all or future contributors and because these discussions may be hard to find again if needed.
2.4
Small and Fast Tests
Software testing can be performed at different granularities, e. g., at the level of individual functions, at the level of individual modules, or at the application level. Originally, libNEGF possessed only whole-device tests (device here means a quantum device); these are practically system tests [Duv07, §6] [OR10, §4.3.3.1]. Writing tests for scientific software is often challenging 9 https://docs.gitlab.com/user/project/issues/managing issues/#default-closing-pattern 10
issue
https://docs.github.com/en/issues/tracking-your-work-with-issues/using-issues/linking-a-pull-request-to-an-
2ccba327
408df56c
05f3b72e
c25d91ed
2c76f2d4
26a7556f
a396475f
fe1b6637
28f45aec
e8a676f3
c764a03a
3ca4a4ce
3ee2c594
26b76613
c8e686a2
284ea788
b13c1fb1
2a6212c3
228130c0
ecb74d33
e7b56b16
6d83a041
2f1dfb93
8df9af43
35155bc8
20fa68e3
c07549df
21fb1672
2fd12e31
e9655ec4
c904edeb
1f691286
ee401f42
3e0c3b6c
2b066cba
609e3dce
1294cfeb
ce6c1c68
061a7ff8
388f875b
9fc4c067
Figure 1: Parts of a git history of HPC software. The oldest commits are at the top and the arrows point in the direction of successors. For readability, most commits that are not merge commits were elided. The two commits at the top have a common ancestor. This common ancestor has 500 descendant commits made over a period of 13 months before all branches were merged into the development branch. Section 2.2 discusses how avoidance of such complicated histories.
A
B
X
Y
C
D
M
A
E
B
(a) Unrestricted history
A
B
X
Y
C
D
X
Y
C
D
X’
Y’
E
M
E
(b) Linear history
N M
E
(c) Unrestricted history with superfluous merge
A
B
X
Y
C
D
X’
Y’
(d) Semi-linear history
Figure 2: Various git histories. Circles indicate commits with descendants on the right. X’ and Y’ are modified variants of the commit X and Y, respectively, that were rebased.
already when the goal is just to test the solution of simple subproblems, e. g., because there may not exist analytical solutions or the domain of algorithm is unknown. Our suggestion therefore is to have test cases with a limited resource consumption suited for desktop PCs. This allows contributors to run these tests frequently on their machines and avoids the need to spend precious compute hours for this purpose. Furthermore, fast-running tests with a low memory footprint are amenable to augmentation, i. e., additional, expensive checks can be enabled, the code can be subjected to dynamic analyses, or it can be debugged on the developer’s office machine. With CI, small tests provide faster feedback to committers.
2.5
Nonzero Exit on Failure
The successful execution of a computer program is indicated by its exit status and it has to be indicated reliably because other parts of the software stack rely on this information (cf. Exit () in [The24]11 ) including shell scripts, CI pipelines, and batch schedulers. To the best of our knowledge, Fortran programs still commonly rely on the STOP intrinsic when encountering an unrecoverable error. The correct choice in such situations are the use of STOP x, where x is a nonzero integer or a string, or ERROR STOP. libNEGF was made to use ERROR STOP in the case of errors which immediately lead to the discovery of a broken test. The test in question was trying to open a nonexisting file, printed an error message to standard output, and exited with status 0 indicating success. Our advice for other developers is to • Ensure familiarity of every code contributor with exit codes and their common interpretation (zero indicates success, nonzero indicates failure). • Determine which tools are offered by a programming language to indicate the exit status (e. g., an uncaught Python exception triggers automatically a nonzero exit). • Check existing code for exits with status zero in case of error and fix these. 11 https://pubs.opengroup.org/onlinepubs/9799919799/functions/ Exit.html
Linux distribution MPI implementation Compute device
Rocky Linux 9, Debian 12 MPICH, Open MPI CPU, GPU (CUDA)
Table 1: Build configurations in use. GPU code is tested only on Debian leading to six different configurations overall.
2.6
Checking Return Values
We strongly recommend to always check all return values even in production runs. In libNEGF, the GPU code was originally discarding return values and once we started to check them, we immediately discovered several bugs and found the causes of incorrect simulation outputs. In one instance an algorithm reliably failed to converge in our tests but this problem went unnoticed because the iteration count was not checked, see Section 4.3 for details. Our recommendation is to always check return values. If a return value indicates failure, there exist a variety of handling strategies [McC04, §8.3] with one approach being to print an error message and to terminate the simulation with a nonzero status. We suggest this practice because log files may be cluttered with warnings even during normal execution and because we do not expect teams to have a process in place to systematically check the log output for problems when the simulation exits successfully. Checking the return value may be superfluous when a developer knows that the return value will always indicate success. This is likely never the case when a function has an error return. For example, MPI is widely used in HPC and it will never return errors if the the default error handler MPI ERRORS ARE FATAL is in use but this error handler can be changed by every piece of code having access to the associated MPI communicator. Similarly, malloc () is unlikely to return NULL pointers because many operating system are configured to allow overcommitment of memory. Unless the kernel settings are under your control, there is no guarantee of this behavior.
2.7
Continuous Integration
We use continuous integration (CI, [Duv07]) with great success in combination with containers [Con, McC18]. The CI setup itself is simple with a style check and a build followed by a run of the test suite. We use the ability of our CI provider to run parameterized jobs, i. e., building and testing is executed several times but with different variable values; these variables are the target device (i. e., whether the code runs on CPU or GPU), the Linux distribution, and the MPI implementation, see Table 1. Figure 3 shows an abridged version of our GitLab CI configuration and due to space constraints we cannot elaborate in detail on this file. The key here is the limited overhead needed to build and test the code in a variety of environments; no changes have to be made to the simulation code. The goal of using different container images is the imitation of different environments: Rocky Linux is a near-clone of RHEL (which powers most HPC systems) whereas Debian and its derivatives are frequently found on consumer machines. MPI is supposed to be implementation
1 2 3 4 5 6 7 8 9 10 11 12 13 14
.setup: parallel: matrix: - DISTRO: [debian, rockylinux] MPI: [mpich, openmpi] DEVICE: [cpu] - DISTRO: [debian] MPI: [mpich, openmpi] DEVICE: [gpu] tags: - public-docker image: $CI_REGISTRY/nexgf/libnegf/$DISTRO/$MPI-$DEVICE variables: BUILD_DIR: build-$DISTRO-$MPI-$DEVICE
15 16 17 18 19 20 21 22 23 24 25 26 27 28
build: extends: .setup stage: build script: - mkdir "$BUILD_DIR" - WITH_GPU="$( [ "$DEVICE" = ’gpu’ ] && echo ON || echo OFF )" - cmake -DWITH_TRANSPORT_GPU="$WITH_GPU" -B "$BUILD_DIR" -S . - num_cpus="$(getconf _NPROCESSORS_ONLN)" - cmake --build "$BUILD_DIR" --parallel "$num_cpus" artifacts: paths: - $BUILD_DIR
29 30 31 32 33 34 35 36
test: extends: .setup stage: test script: - cd "$BUILD_DIR" - ctest --output-on-failure Figure 3: The CI configuration of libNEGF. The file was edited to fit onto one page: the C, C++ compiler flags, the code formatting checks, and some CMake arguments were removed. The original file can be found at https://github.com/libnegf/libnegf/blob/master/.gitlab-ci.yml
agnostic but we test this with MPICH and OpenMPI. The way we use containers is decidedly different from the way containers are commonly used: instead of shipping the built software as a container image to avoid portability problems, we use a multitude of different containers throughout development to ensure portability. For developers who prefer to test changes before pushing, the container images can be built and run locally on a workstation, providing the same environment used by the CI pipeline. With this setup we found a bug in the libNEGF C bindings when OpenMPI was in use. Furthermore, we never faced build problems on Jülich Supercomputing Centre systems whose configuration closely matches the container setup. On other HPC systems, the closer the system configuration to our container setup, the smaller the number of build problems encountered. We recommend to set up CI as soon as possible with at least one build environment in which the code can be successfully run. We also suggest to use container images with all dependencies pre-installed to speed up the CI for otherwise, it is necessary to download all required packages whenever a pipeline is launched. To ensure the presence of all dependencies, there may exist suitable images online (e. g., on DockerHub) and if not, you can build your own images. A Dockerfile is a set of rules to for building a container and for simplicity, we show our Debian Dockerfile in Figure 4 below. Clearly, this is a simple file because all libNEGF dependencies can be met with packages available in the Debian repositories. Sometimes users have to build dependencies on their own, either manually or with the aid of a package manager. In this situation Dockerfiles can still be used through multi-stage builds.12 Due to space constraints, we cannot elaborate on this concept here. Dockerfiles should be checked into version control [Duv07, pp. 109]. Similarly, while Docker was a container pioneer, many alternatives (e. g., Podman) with compatible command line interfaces exist nowadays that can consume Dockerfiles. The Dockerfile in Figure 4 can be built and run with d o c k e r b u i l d −− b u i l d − a r g mpi= openmpi −− b u i l d − a r g d e v i c e = cpu \ −− t a g l i b n e g f − d e b i a n − f D o c k e r f i l e . docker run − i − t libnegf −debian
2.8
Fix Compiler Warnings
Coming back to untrapped errors, fixing compiler warnings is considered a best practice in C and C++, e. g., [McC04] repeatedly mentions paying attention to compiler warnings and even increasing the warning level. Similar advice can be easily found on internet forums or in the guidelines of various industry consortia, e. g., the C and C++ hardening guidelines13 of the Open Source Security Foundation. With respect to the actual outcome, the support for the value of compiler warnings is mostly anecdotal [KKNR22]. Given our positive experiences with C++ compiler warnings, the project went ahead and fixed compiler warnings. Given the comparatively small amount of CUDA code (less than 1,000 lines), a comprehensive set of warnings was enabled with the flags −Wextra −Wall −pedantic in the CI setup. For Fortran, warnings were enabled selectively because of the large number of warnings generated 12 https://docs.docker.com/build/building/multi-stage/ 13 https://best.openssf.org/Compiler-Hardening-Guides/Compiler-Options-Hardening-Guide-for-C-and-C++
1 2 3
ARG mpi ARG device FROM docker.io/library/debian:12-slim AS base
4 5 6 7 8 9 10 11 12 13 14
RUN apt-get update RUN apt-get install --no-install-recommends -y \ build-essential \ ca-certificates \ cmake \ gfortran \ git \ git-lfs \ libopenblas-dev \ python3-minimal
15 16 17 18
# Install MPI FROM base AS flavor-openmpi RUN apt-get install --no-install-recommends -y libopenmpi-dev
19 20 21
FROM base AS flavor-mpich RUN apt-get install --no-install-recommends -y libmpich-dev
22 23
FROM flavor-${mpi} as base+mpi
24 25 26
# Install CUDA on GPU FROM base+mpi AS flavor-cpu
27 28 29 30 31 32 33 34
FROM base+mpi AS flavor-gpu RUN sed --in-place \ ’/Components: main/ s/$/ contrib non-free non-free-firmware/’ \ /etc/apt/sources.list.d/debian.sources RUN apt-get update RUN apt-get install --no-install-recommends -y \ nvidia-cuda-toolkit
35 36 37 38 39 40
# Finalize FROM flavor-${device} AS final RUN adduser --gecos ’’ --disabled-password docker WORKDIR /home/docker USER docker Figure 4: The Dockerfile for Debian containers (the file was slightly edited to fit onto one page). Depending on the values of the build arguments, four different container images can be built from one file.
by the gfortran −Wall option. As a compromise, we settled on enabling all warnings and disabling a subset of warnings considered less relevant for correctness, e. g., warnings about unused functions. These warnings helped us find at least two defects in the code including an iteration counter that was never updated. Our advice is to increase the warning level and to fix all compiler warnings immediately: developers should not get used to ignore warnings in their compiler output; warnings arising from new code should be instantly recognizable. Since developers may not check the logs of the CI runs and instead rely only on the success indicator in the GUI out of convenience (this is not a bad thing), warnings should be turned into errors in the CI. Obviously, this forces contributors to merge only warning-free code. It is therefore necessary to discuss with the project members which warnings are the most relevant and should never be triggered by the code base. In large code bases, an overwhelming number of warnings may be emitted by flags like −Wall. In this situation we suggest to enable a subset of warnings that is the most likely to uncover bugs and with an amount of warnings that can be quickly fixed. For example, warnings about uninitialized variables could be prioritized over warnings of floating-point comparisons with zero. If you do not know which warnings are most relevant, then you could prioritize by the number of warnings emitted but this relies on a feature of modern compilers: they often show the warning together with the flag causing the compiler to emit this warning. With this feature, you can count the number of warnings triggered by each compiler flag and then fix the least frequent warnings first. Finally, we only encountered one project where all warnings had been disabled. For instance, the C compiler tried to warn about comparisons that will always evaluate to the same boolean value, confounding pointers and arrays thereof, pointers mistaken for integer arguments, and missing declarations (missing declarations imply no type checking).14 All of these were true positives and for this reason, we ask readers not to disable warnings.
3
Performance Engineering
libNEGF automatically runs strong scaling benchmarks on a weekly basis. The setup requires an automated benchmarking tool, a GitLab instance, and access to a supercomputer from the GitLab instance. We want to emphasize that this section requires the code to build successfully whenever the benchmark is run. Our CI setup from the previous section ensures that this is the case.
3.1
Use of a Benchmarking Tool
Supercomputers are only accessible by a small set of users and compute time on them is precious. For this reason it is of utmost importance to know what is being run: the simulation code, its dependencies, the build options, the simulation input and its outputs including error messages are all affecting the outcome of a job. For this purpose, a benchmarking tool is very helpful. This project uses JUBE [BWS+ 24] which downloads the libNEGF source code, loads the necessary modules, compiles the code, sets up the simulation, and runs it, and –most importantly– 14 https://gitlab.inria.fr/melissa/melissa-sa/-/commit/680e58b9982134bb60caa749e37f630256a83f45
it performs each run in a dedicated directory. This approach provides several major advantages: • It is more convenient to run simulations for scientists. • There is a single point of reference on how to run on a given supercomputer. • All of the inputs and outputs of every simulation run are saved. In short, the user can focus on the actual experiment and its outcome. Regarding the single point of reference, the Jülich Supercomputing Centre is home to four HPC systems to which the authors have (or had) access. For each system, one can run on CPU and GPU entailing different build and job scheduler options. The JUBE script hides the details of these eight configurations. Note that we follow a strictly layered approach in libNEGF development: it is still possible to run libNEGF without JUBE and this is occasionally done for development purposes. In our experience if a certain user needs to deviate from the setup realized by JUBE, it is often more convenient to modify the JUBE script. Saving all of the inputs and output provides significant benefit when a simulation unexpectedly fails because it provides developers with certainty about the build configuration as well as inputs and outputs of the simulation. Our recommendation for other developers is simple: use a tool to manage and archive supercomputer runs.
3.2
Continuous Benchmarking
Performance is a key aspect of high-performance computing. Continuous benchmarking (CB) makes performance assessments an integral step of the development process on the same level as building, testing, and deploying [ALP+ 24, §1]. To ensure fulfilling the EoCoE-III performance goals, the developer team set up CB in mid-2025 with the goal of regular execution on one of the HPC systems targeted by EoCoE-III (here: JUWELS Booster). Ideally, we would run CB whenever a contributor pushes code but the compute time on HPC systems is limited and such an approach would be a significant drain on the compute time budget.15 Therefore we limit our CB run to once per week or less frequently. The CB setup benefits significantly from on-going CB efforts of JSC [BFS24, HAA+ 24, BBRH26]. Before elaborating on the details of the CB setup, let us briefly review GitLab’s CI pipelines. A CI script describes one or more jobs (not to be mistaken with batch scheduler jobs) possibly with dependencies between them that are executed whenever a pipeline is triggered. Each job is executed on a dedicated machine called runner16 . Pipelines can commonly be triggered by pushing to a repository or by a scheduled event17 . libNEGF stores its benchmarking data together with its JUBE script (see Section 3.1) in its own git repository called libnegf-benchmarks in the same GitLab group. For CB, the CI script 15 One has to apply for access to HPC machines and we question, too, if the funding agency would agree to a project
proposal spending most of its resources on CB. 16 https://docs.gitlab.com/runner/ 17 https://docs.gitlab.com/ci/pipelines/schedules/
in Figure 5 was added to libnegf-benchmarks and a schedule was set up. With its choice of tags in lines 29–32, the script ensures that the GitLab server picks JUWELS Booster as runner. The shell script in lines 6 to 16 is then executed on a JUWELS Booster front-end node from which we can submit batch scheduler jobs. Here we re-use the JUBE script from Section 3.1; the benchmark to be executed is configured by means of the environment variables in lines 22– 27. The JUBE output is stored in the persistent storage of the libNEGF compute time project on JUWELS Booster (persistent storage in contrast to the scratch storage which is purged after 90 days). The CB runs are evaluated on demand with a Python script. In addition to the JUBE data, the batch scheduler is queried for job information. Coming back to the CB efforts at JSC, our setup is simple because JSC provides GitLab runners executing on JUWELS Booster18 and these runners use Jacamar CI19 . The advantages of this setup include its ease of use (as the batch scheduler details are hidden in the JUBE script), no code duplication, and easy debugging. A disadvantage is the CB run being marked as successful in the GitLab web interface as soon as JUBE exits successfully, i. e., after successful batch scheduler job submission; simulation failures are only discovered when the Python analyzer script is run afterwards.
4
Case Studies: Diagnosing Software and Performance Issues
4.1
Diagnosing Rare Performance Issues
With a benchmarking tool and the code forge in place, the libNEGF team successfully tracked down a rare performance issue. In September 2024 during a strong scaling run from eight to 192 nodes on JUWELS Booster, the run on eight nodes unexpectedly timed out even with a generous time limit 50 % above the estimated wall-clock time for the job. The problem was immediately investigated with the aid of the artifacts saved by JUBE. After a review no cause could be identified; human error was precluded as a cause. Thereafter a GitLab issue was opened. The Jülich Supercomputing Centre generates reports for every job; the tool generating these reports is called LLview [FR07]. These were attached to the issue and showed the GPUs idling on all nodes except one. The code forge enabled a constructive discussion with user’s being able to add tables and images to their comments. With JUBE in place, it was possible to check older runs for occurrences of this problem and the problem had previously occurred without causing a time-out. Regrettably the discussion did not lead to a quick resolution. Figure 6 shows a part of an LLview job report.20 Note that most nodes start idling after approximately 75 minutes. Even after accounting for load imbalance, the pattern seen in the left portion (the part before 9:30 pm) should be seen throughout the lifetime of the job. LLview job reports show memory consumption for each node over time and after encountering the problem several times, a peculiar pattern began to emerge in the memory consumption of concerned jobs: at the start of the job, the node would already exhibit high memory consumption. This 18 https://apps.fz-juelich.de/jsc/hps/juwels/jacamar.html 19 https://ecp-ci.gitlab.io/index.html 20 This figure was not taken from the job running on eight nodes mentioned above but a different job exhibiting the
same problem. This decision was taken for expository reasons.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
.setup: id_tokens: SITE_ID_TOKEN: aud: "https://gitlab.jsc.fz-juelich.de" script: - hostname - pwd - jutil env activate --project mat4energy - module load JUBE/2.7.1 - echo "PROJECT=$PROJECT" - env | fgrep LIBNEGF - time jube run --error --outpath="$PROJECT/continuous-benchmarking" --tag ${LIBNEGF_JUBE_TAGS:?LIBNEGF_JUBE_TAGS not set} -- JUBE/benchmark-libnegf.xml
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
run_jube_juwels_booster: extends: .setup variables: LIBNEGF_JUBE_TAGS: "juwels_booster gpu inelastic" LIBNEGF_INPUT_SIZE: "2x2" LIBNEGF_NUM_NODES: "2,4,8,16" LIBNEGF_NUM_GROUPS: "8" LIBNEGF_NUM_EPOINTS: "64" LIBNEGF_TIMELIMIT_MIN: "40" tags: - juwels_booster - jacamar - login - shell Figure 5: The CI configuration of libnegf-benchmarks. The command in line 11 is a superficial check for the presence of the environment variables set below. If these are not exported by the callers, then the JUBE script will use default values.
Figure 6: The GPU activity of a job on JUWELS Booster throughout its lifetime. The abscissa shows time and the ordinate the nodes. Idling is indicated by purple, high utilization by greenish colors. The plot was taken from the LLview job report.
is unexpected because the simulation’s memory usage is initially low until all of the inputs are consumed. Given our understanding of the JUWELS Booster architecture, we were then able to formulate an hypothesis about the cause of the slowdowns. Each JUWELS Booster node possesses eight NUMA [HP19, §5.1] domains but only four of those have a direct (and thereby fast) GPU connection. By default, libNEGF is run with four MPI tasks per node with the tasks mapped to CPU cores whose primary NUMA domain is one of the domains with direct GPU connection. Clearly, if some foreign data is already occupying these NUMA domains, then libNEGF would be slowed down. This theory puts the blame squarely on the HPC system. At this point we contacted JSC support (Ticket #1097823). With JUBE in place, the JUBE data stored on JUWELS Booster, and the LLview job reports, it was easy to convince the system administrators of our findings. With their support the cause was tracked down to a Linux configuration quirk. In general, the operating system caches filesystem data. Consider the situtation where one NUMA domain is fully occupied by cached filesystem data and the data of an application. When the application requests more memory, then the operating system has to take a decision: either the cached data is freed to satisfy the application’s request or the application’s request is fulfilled with data from another NUMA domain. The best decision in this scenario is application-dependent and for this reason the behavior can be configured21 . Finally, the best course of actions was found to be flushing the filesystem buffers before the start of a new job. This resolved the issue.
21 See zone reclaim mode [Lin]
Figure 7: This plot shows the outcome of the strong scaling runs of the libNEGF CB setup. After a JUWELS Booster update on September 9, 2025, performance slightly decreases on two and four nodes. Crosses indicate elapsed time (left axis), circles indicate energy consumption of the job (right axis), the color indicates recency. To highlight the impact of the maintenance, jobs running after it are shown in bright colors and jobs running before it are shown in dark colors.
4.2
Tracking HPC Center Peformance
The key advantage of CB is tracking application performance over time but the performance is dependent on the underlying machine, too, and with CB one can track the latter. With CB in place for several weeks, we were able to identify a performance degradation of the simulation code after JUWELS Booster maintenance without any changes in our code. Figure 7 shows a plot of the strong scaling runs of our CB setup with runs after the maintenance period in bright colors. Observe that for two and four nodes, the run-time (indicated by crosses) is higher after the update; on four nodes, the energy consumption (indicated by circles) is also higher. For performance engineering purposes, awareness of such events is important for the understanding of the performance evolution of a software.
4.3
Defect Chains hiding Misunderstood Mathematics
In the previous sections, we mentioned several fixed libNEGF issues relating to warnings and not checking return values. These issues turned out to be connected and occluding a bigger issue. There exists an iterative algorithm in libNEGF for dealing with boundary conditions of the physical domain. Its input are two pairs of square matrices (S1 , H1 ) and (S2 , H2 ). In a first step, complex scalars αi and linear combinations Ai = αi Si + Hi are computed, where i = 1, 2. In a second step, the algorithm output is calculated from A1 , A2 by an iterative method. This algorithm is implemented in CUDA and called directly from Fortran code. CUDA code updates an iteration count in a memory locations provided by the caller of the CUDA code. Initially, the Fortran code was passing an iteration counter by value to CUDA by mistake; “by value” implies that the Fortran caller code never had access to the actual iteration count. The CUDA compiler diagnosed this problem after enabling warnings. Once the iteration counter was passed by reference, we started to check the counter at the call site after the CUDA code had returned. We then found out that in one test of this algorithm, the maximum iteration count was exceeded. Due to the simplicity of the test, we arrived at the conclusion that the test was proper and that the algorithm should converge. After an investigation, we realized that our comprehension of the algorithm was lacking. In the test S1 , S2 were chosen randomly but for convergence, zS2 + S1 + z−1 S2∗ must be Hermitian positive definite for all complex scalars z with modulus one. Luckily, for real-world inputs the matrices Si possess this property meaning simulations were unaffected by this oversight. In summary, two innocuous defects were concealing nonconvergence of an algorithm with nonconvergence caused by our lack of understanding of certain aspects of the algorithm. The defects were found using our suggested practices and before our faulty assumptions affected production runs.
5
Failure to Follow Best Practices
5.1
Standards Compliance
libNEGF aims for portability across all major HPC systems and this includes being able to compile libNEGF on these machines with the provided compilers. Fortran is a standardized programming language but in practice, two problems often arise: the code is not standards compliant or the compiler does not implement the language standard in its entirety. We found out that the libNEGF Fortran code does not compile with the Fujitsu compiler (found on, e. g., Fugaku) nor with the Cray Fortran compiler (LUMI) because libNEGF is not standards compliant. It just happened to be written in a dialect compatible with the most frequently used compilers (GCC and Intel). We are aware of one language feature in our code not supported by one of the aforementioned compilers. Our recommendation is to determine if portability is desired and if so, compilers should be instructed to compile only standards-compliant code. This change should happen as early as possible to avoid accumulating code that needs to be fixed later. Our second recommendation is to check if the compilers on target systems are fully supporting the Fortran subset in use by the project because compiler support for a language standards may be extensive but not complete.
5.2
Lack of Defensive Programming
libNEGF has been actively developed for more than a decade with its focus on the prediction of material properties. As a side-effect, the software contains no “consistency” checks, i. e., neither assertions about the program state nor checks of the mathematical objects. This approach has drawbacks. First, errors may not be detected at all unless a contributor with sufficient training in the relevant physics takes a look at the simulation output. This is a problem when the project’s goal recently shifted to performance improvement and a large number of simulation runs become a common event. Second, once an error has been identified, the defect causing it has to be traced starting with a complete simulation run. Errors that went unnoticed for some time include broken GPU code and nonsensical simulation output for unknown reasons. We strongly urge other teams to have some kind of consistency checks to avoid the aforementioned problems, e. g., assertions [McC04, §8.2], backward error computations [Hig02, §1.5], or checks of residuals. These suggestions can be seen as defensive programming applied to scientific software [McC04, §8].
5.3
Code formatting
Proper code formatting helps contributors to understand code more quickly [McC04, §31] and it avoids clutter in commits. With the latter we mean that programmers may accidentally make semantically insignificant changes to the code, e. g., adding empty lines, adding spaces at the end of a line, or adding line breaks within existing code. These changes are picked up by the revision control system (RCS). In severe cases, formatting differences in semantically equivalent code may break the RCS’ capabilities to merge different code branches. This happened during libNEGF development where a fork of the DFTB+ software had to be created to accommodate for newly added libNEGF GPU code. After several months of development, an attempt a merging this branch with the latest DFTB+ code in its main branch failed due to significant formatting differences. We recommend using an automatic code formatting tool. They are widely available for almost all languages (even CMake) and literally work at the push of a button. In our experience, the ease of use and the sensible layout generated out of the box by these tools outweighs any concerns contributors may have with regards to the style. They also save the programmer from memorizing large documents for each language in use by a project (e. g., the Google C++ Style is approximately 62 pages long22 ). Code formatting should be the first thing to be checked in a CI pipeline because it is fast and computationally cheap in comparison to other typical operations.
6
Conclusion
Research software engineering techniques were applied when making libNEGF run on massively parallel supercomputers. In Section 2 we presented our software engineering approach aimed at ensuring buildability and defect-free code. The performance engineering approach detailed in 22 The Google C++ Style Guide was downloaded on April 15, 2026 from https://google.github.io/styleguide/
cppguide.html. The document contains 31,074 words or approximately 62 pages at 500 words per page.
Section 3 simplifies benchmarking and large-scale experiment runs for nonprogrammers. Our practices uncovered a misunderstood mathematical model and performance changes induced by the HPC cluster on which libNEGF ran (see Section 4). Finally, there are more common practices that should have been implemented by us as laid out in Section 5. By sharing our experiences we aim to provide other RSE practitioners with data points to support their decision-making. In particular, assuming code contains undetected defects proved to be a useful engineering practice. In the case of libNEGF, untrapped errors seem to be as common as for any other software written in an unsafe language. Acknowledgements: This project has received funding from the European High Performance Computing Joint Undertaking under grant agreement n°101144014. The authors gratefully acknowledge the Gauss Centre for Supercomputing e.V. (www.gausscentre.eu) for funding this project by providing computing time through the John von Neumann Institute for Computing (NIC) on the GCS Supercomputer JUWELS[Jü21] at Jülich Supercomputing Centre (JSC). Data availability: Figure 1 and Figure 7 can be reproduced from [Con26]. The LLview job report from which Figure 6 was taken can be found in [Con26], too. Declaration on the use of AI: Successive drafts of this manuscript were repeatedly proofread with the help of Blablador [Str26].
Bibliography [ALP+ 24] C. Alt, M. Lanser, J. Plewinski, A. Janki, A. Klawonn, H. Köstler, M. Selzer, U. R. and. A continuous benchmarking infrastructure for high-performance computing applications. International Journal of Parallel, Emergent and Distributed Systems 39(4):501–523, 2024. doi:10.1080/17445760.2024.2360190 [BBRH26] J. Badwaik, M. Bode, M. Rajski, A. Herten. exaCB: Reproducible Continuous Benchmark Collections at Scale Leveraging an Incremental Approach. 2026. To appear. doi:10.48550/arXiv.2603.22251 [BCC+ 08] V. R. Basili, J. C. Carver, D. Cruzes, L. M. Hochstein, J. K. Hollingsworth, F. Shull, M. V. Zelkowitz. Understanding the High-Performance-Computing Community: A Software Engineer’s Perspective. IEEE Software 25(4):29–36, 2008. doi:10.1109/MS.2008.103 [BFS24]
D. Brömmel, J. Fritz, R. Speck. Integrated Continuous Benchmarking. 2024. doi:10.34734/FZJ-2024-01995
[BO11]
R. E. Bryant, D. R. O’Hallaron. Computer Systems: A Programmer’s Perspective. Pearson Education, Inc., Boston, MA, USA, 2nd edition, 2011.
[BWS+ 24] T. Breuer, J. Wellmann, F. Souza Mendes Guimarães, C. Himmels, S. Lührs. JUBE. 2024. doi:10.5281/zenodo.7534372 [Car04]
L. Cardelli. Type Systems. In Tucker (ed.), Computer Science Handbook. Chapter 97. Chapman and Hall/CRC, Boca Raton, FL, USA, second edition, 2004.
[Cla11]
G. Clarke. CERN’s boson hunters tackle big data bug infestation. 2011. https://www.theregister.com/2011/09/22/cern coverity/
[Con]
What is a container? Accessed: 2026-04-13. https://docs.docker.com/get-started/docker-concepts/the-basics/what-is-acontainer/
[Con26]
C. Conrads. Replication Data for: RSE of a Quantum Transport Code and its Effects. 2026. doi:10.26165/JUELICH-DATA/9JHYGV
[Cov11]
Coverity Scan 2011 Open Source Integrity Report. Technical report, Coverity, Inc., 2011. https://web.archive.org/web/20120226115247/https://www.coverity.com/library/ pdf/coverity-scan-2011-open-source-integrity-report.pdf
[DLRA15] W. Dietz, P. Li, J. Regehr, V. Adve. Understanding Integer Overflow in C/C++. ACM Trans. Softw. Eng. Methodol. 25(1):1–29, 2015. doi:10.1145/2743019 [Duv07]
P. M. Duvall. Continuous Integration: Improving Software Quality and Reducing Risk. Addison-Wesley Signature Series. Addison-Wesley Professional, 2007.
[FR07]
W. Frings, M. Riedel. LLview: User-level Monitoring in Computational Grids and e-Science Infrastructures. 2007.
[HAA+ 24] A. Herten, S. Achilles, D. Alvarez, J. Badwaik, E. Behle, M. Bode, T. Breuer, D. Caviedes-Voullième, M. Cherti, A. Dabah, S. E. Sayed, W. Frings, A. GonzalezNicolas, E. B. Gregory, K. H. Mood, T. Hater, J. Jitsev, C. M. John, J. H. Meinke, C. I. Meyer, P. Mezentsev, J.-O. Mirus, S. Nassyr, C. Penke, M. Römmer, U. Sinha, B. v. S. Vieth, O. Stein, E. Suarez, D. Willsch, I. Zhukov. Application-Driven Exascale: The JUPITER Benchmark Suite. In SC24: International Conference for High Performance Computing, Networking, Storage and Analysis. Pp. 1–45. 2024. doi:10.1109/SC41406.2024.00038 [Hig02]
N. J. Higham. Accuracy and Stability of Numerical Algorithms. Society for Industrial and Applied Mathematics, 2 edition, 2002. doi:10.1137/1.9780898718027
[HP19]
J. L. Hennessy, D. A. Patterson. Computer Architecture. Morgan Kaufmann Publishers, 6 edition, 2019.
[Jü21]
Jülich Supercomputing Centre. JUWELS Cluster and Booster: Exascale Pathfinder with Modular Supercomputing Architecture at Juelich Supercomputing Centre. Journal of large-scale research facilities 7(A183), 2021. doi:10.17815/jlsrf-7-183
[KKNR22] G. Kudrjavets, A. Kumar, N. Nagappan, A. Rastogi. The unexplored terrain of compiler warnings. In Proceedings of the 44th International Conference on Software Engineering: Software Engineering in Practice. ICSE-SEIP ’22, pp. 283–284. Association for Computing Machinery, New York, NY, USA, 2022. doi:10.1145/3510457.3513057 [Lat11]
C. Lattner. What Every C Programmer Should Know About Undefined Behavior #1/3. Online, 2011. Accessed: 2026-04-09. https://blog.llvm.org/2011/05/what-every-c-programmer-should-know.html
[Lin]
Documentation for /proc/sys/vm/. Online. Accessed: 2026-04-13. https://docs.kernel.org/admin-guide/sysctl/vm.html
[McC04]
S. McConnell. Code Complete: A Practical Handbook of Software Construction. Microsoft Press, Redmond, WA, USA, second edition, 2004.
[McC18]
S. McCarty. A Practical Introduction to Container Terminology. 2018. Accessed: 2026-04-13. https://developers.redhat.com/blog/2018/02/22/container-terminology-practicalintroduction
[MF21]
L. Maggini, R. R. Ferreira. 2D material hybrid heterostructures: achievements and challenges towards high throughput fabrication. Journal of Materials Chemistry C 9:15721–15734, 2021. doi:10.1039/D1TC04253J
[OR10]
W. L. Oberkampf, C. J. Roy. Verification and Validation in Scientific Computing. Cambridge University Press, 2010.
[Pop08]
T. Pope. A Note About Git Commit Messages. 2008. https://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html
[PPSC08] A. Pecchia, G. Penazzi, L. Salvucci, A. D. Carlo. Non-equilibrium Green’s functions in density functional tight binding: method and applications. New Journal of Physics 10(6), 2008. doi:10.1088/1367-2630/10/6/065022 [Reg10]
J. Regehr. A Guide to Undefined Behavior in C and C++, Part 1. Online, 2010. Accessed: 2026-04-09. https://blog.regehr.org/archives/213
[Str26]
A. Strube. Blablador. 2026. Visited: 2026-04-27. http://helmholtz-blablador.fz-juelich.de
[The24]
The Open Group. The Open Group Base Specifications Issue 8. 2024. POSIX.12024. https://pubs.opengroup.org/onlinepubs/9799919799/
[WZKS13] X. Wang, N. Zeldovich, M. F. Kaashoek, A. Solar-Lezama. Towards optimizationsafe systems: analyzing the impact of undefined behavior. In Digney (ed.), SOSP’13: ACM SIGOPS 24th Symposium on Operating Systems Principles. Pp. 260–275. Association for Computing Machinery, New York, NY, USA, 2013. doi:10.1145/2517349.2522728