Computer Science and Information Systems 00(0):0000–0000https://doi.org/10.2298/CSIS123456789X
RAMSES: Secure high-performance computing for sensitive data Peter Heger1 , Lech Nieroda1 , Roland Pabel1 , Christoph Stollwerk1 , Stefan Borowski1 , Kamil Tokmakov2 , Michael Commer1 , Martin Peifer3 , Stefan Wesner1 , and Viktor Achter1
arXiv:2606.27919v1 [cs.DC] 26 Jun 2026
1
IT Center Cologne (ITCC) University of Cologne Weyertal 121 50931 Cologne, Germany [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected] 2 NEC Deutschland GmbH Fritz-Vomfelde-Straße 14-16 40547 Düsseldorf, Germany [email protected] 3 Department of Translational Genomics University of Cologne 50931 Cologne, Germany [email protected]
Abstract. Traditionally, the architecture of high-performance computing (HPC) systems is tailored for speed, while highly secure computer systems must sacrifice speed for security. However, a wide range of scientific domains, such as the life sciences, call for a combination of performance and security to allow processing sensitive data at scale. Here, we present RAMSES (Research Accelerator for Modeling and Simulation with Enhanced Security), an HPC system designed from the ground up to deliver high performance within a robust security framework. RAMSES integrates hardware-based memory encryption of AMD processors with state-of-the-art file encryption from IBM Storage Scale and the Thales CipherTrust manager, establishing an HPC platform that ensures continuous encryption throughout the data life cycle—at rest, in transit, and in use—in compliance with major data protection standards (European General Data Protection Regulation, ISO/ IEC 27001 certification, and Federal Information Processing Standards). In addition, we implemented advanced operating system hardening, a multilayered security architecture, and mandatory multi-factor authentication to adapt the HPC environment to increased security demands. Benchmark results from the biomedical sector demonstrate that the performance impact of the secure environment is limited and that integration of the conflicting requirements speed and security can be achieved while preserving a coherent, flexible, and user-friendly system. Keywords: Secure high-performance computing, confidential computing, end-to-end encryption, memory encryption, Hardware Security Module (HSM), AMD SEV/SME, Trusted Research Environment (TRE), Thales,
2
Heger et al. multi-factor authentication (MFA), Information Security Management System (ISMS), key management, operating system hardening, threat model, secure boot, SR-IOV, SPANK.
1. Introduction HPC systems are typically composed of numerous individual compute nodes and storage devices, coupled through high-speed interconnects, and are widely seen as drivers of scientific discovery and innovation [44,52,62,4]. The performance of the fastest supercomputers has reached the exascale level, with the execution of more than one exaFLOP/s (floating point operations per second) on the latest German supercomputer, Jupiter (https://www.fz-juelich.de/de/jupiter), the 4th fastest computer in the world (as of November 2025; https://top500.org/lis ts/top500/2025/11/). While access to supercomputers is restricted and users are generally not allowed to perform system-critical tasks, such as installing and modifying system software, conventional HPC systems rely on Linux’s standard security mechanisms which often lack advanced features for system and/or data security [6,21]. In contrast, highly secure computer systems, e. g. from the banking or military sector, are typically small in comparison to TOP500-listed systems (https: //top500.org/). Their design follows official security recommendations and often involves dedicated hardware and software layers on top of OS (operating system) level protection, for example HSMs (hardware security modules), TPMs (trusted platform modules), secure boot, tamper detection (hardware level), and specific software solutions that address different aspects of cybersecurity, such as key management systems, network security tools, or monitoring solutions [51,18,10]. Many of these measures are considered mandatory to ensure data integrity, transaction throughput, or regulatory auditability in their specific contexts [15,30,11,23]. According to NIST (National Institute of Standards and Technology) principles, HPC system architecture can be divided into several zones, such as access, compute, management, and storage. Each zone is exposed to distinct threat classes. They include denial-of-service attacks, perimeter scanning, and traffic interception in the access zone; side-channel and co-tenancy attacks in the compute zone; privilege escalation in the management plane; and data confidentiality or integrity violations in the storage layer [21]. Consequently, the design of secure HPC architectures requires an in-depth approach that reduces zone-specific threats through appropriate countermeasures, including encryption of data at rest, in transit, and in use; strong identity and access management; comprehensive audit and logging mechanisms; compute and workload isolation; network segmentation; and hardware- and software-based cryptographic protection. RAMSES addresses the above-mentioned security threats. Specifically designed for secure data processing, RAMSES integrates hardware-enforced memory encryption, cryptographically protected storage, and systematic operating-system hardening, while preserving usability for scientific workloads.
RAMSES: Next-generation HPC security
3
2. State of the Art 2.1. Secure Environments for Scientific Computing Implementations of secure computing environments cover a broad spectrum, from regulatory compliance frameworks to hardware-enforced protection of data in use. At the basic level, many HPC centres adopt formal certification standards such as ISO/IEC 27001, which define requirements for an information security management system (ISMS), including risk assessment, access control, incident response, and continuous auditing [26]. The use of validated cryptographic modules is often an additional requirement. For example, the U.S. FIPS 140-2 and FIPS 140-3 standards specify security requirements and approved algorithms for cryptographic modules, which are independently evaluated under the Cryptographic Module Validation Program (CMVP) and assigned assurance levels based on physical and logical protections [41,43,25]. These certifications warrant process maturity and cryptographic correctness, but do not require specific technical safeguards such as encryption of data, tenant isolation, or protection against privileged insider threats. While a growing number of academic HPC centres in Europe and North America operate under such certification programs (Table 1), certified HPC systems often still adhere to conventional architectures and remain unsuitable for sensitive workloads. Trusted Research Environments (TREs) represent a more mature level of security. TREs are controlled computing environments explicitly designed for sensitive or regulated data and typically align with established frameworks such as the Five Safes model for safe research access to data [12]. They combine strong identity and access management, restricted network connectivity, monitoring and audit logging, and user training to reduce risks across access, compute, management, and storage zones [58,34]. Architectural guidance and operational practices for TREs are reliable and widely adopted, particularly in the biomedical and social sciences. As of late 2024, more than 40 TREs have been identified across Europe [55], including HPC-enabled platforms at Cambridge, the University of Edinburgh, or Imperial College London (Table 1). However, TREs primarily rely on software isolation and organisational controls and typically do not protect data in use or at rest. Further improvements in protection are achieved through Trusted Execution Environments (TEEs), which introduce hardware-enforced isolation mechanisms that protect code and data even from privileged system software [20]. TEEs address threats explicitly highlighted in NIST SP 800-223, such as memory disclosure, remapping, and privileged access in the compute zone, by providing encryption and integrity protection for data in use, i. e., memory encryption [21]. While TEEs are increasingly deployed in public cloud infrastructures, their integration into shared, large-scale HPC environments remains limited due to operational complexity and performance considerations. Nevertheless, recent attempts to operate TEEs in HPC context report acceptable performance losses and thus encourage the productive use of TEEs, for example for bioinformatics workloads [2,35,13,8]. An intermediate approach bridging conventional HPC and hardware-backed confidential computing was proposed by Nolte et al. [45]. Their secure HPC workflow combines encrypted containers and datasets, centralised key management via
4
Heger et al.
HashiCorp Vault (https://www.vaultproject.io/), signed job scripts, and strict node and network isolation. Data is encrypted prior to upload and decrypted only within isolated execution environments on the cluster, supporting multi-node workflows and parallel encrypted I/O. Although this approach substantially improves the protection of data at rest and in transit, it introduces significant complexity for users and does not address threats to data in use, such as physical memory extraction or privileged RAM inspection. In summary, existing secure HPC solutions demonstrate increasing levels of protection, from organisational certification through TRE-style controlled environments to hardware-enforced confidential execution in TEEs, but no single approach yet provides comprehensive, zone-spanning security as recommended by NIST SP 800-223. Therefore, achieving integrated protection for data at rest, in transit, and in use within shared HPC systems remains an open challenge. 2.2. Encryption Techniques: Symmetric, Asymmetric, Hybrid, Homomorphic Encryption is a central component of any secure computing environment. Symmetric encryption uses a shared secret key for both encryption and decryption. Its classic implementation is the Advanced Encryption Standard (AES), specified in FIPS publication 197 [42]. Symmetric encryption methods work with fixed-size data blocks using simple deterministic transformations, which can be efficiently parallelized and are widely supported by dedicated CPU instructions (e. g., AESNI). As a result, symmetric encryption achieves high throughput and low latency, making it well-suited for bulk data protection, such as encrypting entire file systems or object-store volumes. Owing to its performance characteristics, symmetric encryption is the standard mechanism for protecting data at rest and data in transit, e. g., within the protocol for transport layer security (TLS). Asymmetric encryption (public-key cryptography) relies on a key pair consisting of a public key for encryption or verification and a private key for decryption or signing. Typical applications include key exchange (e. g., during a TLS handshake), digital signatures, or the protection of small data items such as cryptographic keys. In contrast to symmetric encryption, asymmetric methods avoid pre-shared secrets and instead rely on computationally hard mathematical problems to ensure that decryption is feasible only with the private key. As it does not benefit to the same degree from parallelism or hardware acceleration, asymmetric encryption is unsuitable for bulk data protection. Instead, modern systems employ hybrid encryption: asymmetric cryptography is used to authenticate users and securely establish or wrap a symmetric session key, which then encrypts the bulk of data. This hybrid design powers TLS, secure file-sharing systems, and cloud key management infrastructures. Finally, homomorphic encryption (HE) allows computation on encrypted data without decryption—theoretically attractive for «data in use» protection in untrusted compute environments. Multiple recent articles examine HE in HPC and cloud contexts. For instance, Meftah et al. (2022) utilise HPC systems to accelerate HE for convolutional neural networks [40]. Similarly, Jung et al. (2020) propose algorithms to reduce the computational cost of HE by avoiding expensive multiplications [31]. Prantl et al. (2023) demonstrate comparable performance analyses
RAMSES: Next-generation HPC security
5
Site
Certification
Extras
Link
Year
HLRS (Höchstleistungsrechenzentrum Stuttgart), University of Stuttgart
ISO27001
—
a
2023
Secure Research Computing Platform (SRCP), University of Cambridge
ISO27001, NHS Data Security and Protection Toolkit
encryption, monitoring
b
2016
SDU eScience Center, University of Southern Denmark
ISO27001
—
c
2020
Secure Research Infrastructure (Citadel), Princeton University
NIST SP 800171, NIST SP 800-53, HIPAA
encryption, MFA, monitoring
d
2021
Centre for Advanced Research Computing, University College London
ISO27001, NHS Data Security and Protection Toolkit
TRE, MFA, Data Security Awareness course
e
2014
GWDG, Göttingen University
ISO9001, ISO27001
Secure HPC workflow [45]
f
2023
SURF (Samenwerkende Universitaire Rekenfaciliteiten) Research Cloud
ISO27001
—
g
2020
Edinburgh Parallel Computing Centre (EPCC), University of Edinburgh
ISO27001, NHS Data Security and Protection Toolkit
TRE
h
2018
Big Data and Analytical Unit (BDAU), Imperial College London
ISO27001, NHS Data Security and Protection Toolkit
TRE
i
2017
Computational Research, Engineering and Technology Environment (CREATE), King’s College London
ISO27001, NHS Data Security and Protection Toolkit
TRE, MFA, encryption
j
2021
CINECA (Consorzio Interuniversitario per il Calcolo Automatico dell’Italia Nord Orientale)
ISO9001, ISO27001, ISDP 10003
—
k
2022
Advanced Research Computing, Cardiff University
ISO27001, ISO9001
—
l
2025
Table 1: List of certified HPC centers in North America and Europe. Measures that improve security beyond formal certification are listed in column «Extras». MFA: Multi-factor authentication; TRE: Trusted Research Environment; TEE: Trusted Execution Environment. For web links to the respective centers, use the hyperlinks provided in column «Link». Please note that this list contains a selection of highly visible, certified HPC centers, but may be incomplete.
in a practical linear-regression scenario [50]. These results highlight that HE is a promising field, but its current runtime overhead is still orders of magnitude
6
Heger et al.
higher than native plaintext processing due to lattice-based arithmetic, ciphertext noise management, and the absence of efficient hardware acceleration. Thus, for large-scale HPC workloads HE remains impractical and requires significant compute overhead or architectural redesign. That in turn undermines one of the main goals of high-performance computing: fast throughput and low latency. Hybrid approaches that encrypt part of the data or isolate sensitive segments are presently more feasible. In sum, the encryption strategy in a secure computing environment often uses symmetric encryption for bulk data, asymmetric/hybrid for key distribution and access, and reserves homomorphic encryption or secure enclaves for highly sensitive or small-volume operations where a performance penalty is acceptable. 2.3. Hardware Memory Encryption Standards Across CPU and GPU Vendors Modern secure computing environments increasingly rely on in-situ memory encryption to protect against physical attacks (e. g., cold-boot, direct memory access, DIMM bus probing) and to harden multi-tenant isolation. Across vendors, this is achieved via an inline AES engine at the memory controller with address-dependent tweaks, ephemeral keys generated in hardware, and per-tenant keys plus integrity metadata. Still, design choices differ in scope (whole-system vs. per-VM), granularity (page-level selection vs. «encrypt everything»), and whether integrity/freshness is provided in addition to confidentiality. AMD (SME, TSME, SEV, SEV-SNP). AMD’s Secure Memory Encryption (SME) integrates AES-XTS, a tweakable AES- and physical-address-based encryption mode, into the memory controller and encrypts any page marked with the C-bit using a 256-bit key derived by the AMD Secure Processor (SP) at boot. Transparent SME (TSME) extends this to system-wide DRAM encryption where a single key encrypts all memory. Secure Encrypted Virtualization (SEV) builds on SME by assigning each virtual machine its own memory encryption key, preventing the hypervisor or co-tenants from inspecting guest memory. The latest development, SEV-SNP (Secure Nested Paging), adds integrity protection via the Reverse Map Table (RMP) and related mechanisms, thereby blocking remapping and replay attacks ([1]). This feature adds the attestation mechanisms needed for confidential computing, allowing tenants to verify the platform’s hardware and security state before releasing sensitive data—an attestation workflow now standard in major public clouds [32]. Intel. Intel’s memory encryption technologies span several layers. Total Memory Encryption (TME) provides baseline protection by encrypting all external DRAM with AES-XTS using a single ephemeral key generated inside the CPU during boot; this protects against physical memory attacks and does not require OS changes, but it applies a single key to the entire system. Multi-Key TME (MKTME) generalizes this design by supporting multiple encryption keys and allowing the OS or hypervisor to select keys on a per-page basis, enabling isolation between different workloads or VMs. Trust Domain Extensions (TDX) build on MKTME to create confidential VMs (trust domains) that not only use distinct memory
RAMSES: Next-generation HPC security
7
encryption keys, but also add integrity protection mechanisms (such as per-cacheline MACs and anti-replay measures), remote attestation, and a reduced host-side trusted computing base. Together, TME, MKTME, and TDX form a progression from system-wide memory encryption toward a full confidential-computing model comparable in goals to AMD SEV-SNP, albeit through different architectural mechanisms [24]. Arm (CCA/RME). Arm’s Confidential Compute Architecture (CCA) introduces Realms: hardware-isolated execution environments that protect code and data even from privileged system software. Realms extend the traditional secure/non-secure model with a third, strongly separated domain and rely on the Memory Protection Engine (MPE) to enforce external-memory encryption and, where implemented, integrity or freshness protection. System memory is divided into Physical Address Spaces (PAS), each with their own encryption context, while the MMU/SMMU (memory management unit) assigns pages to PASs and ensures that Realm keys are discarded upon teardown. CCA delivers strong architectural isolation with compulsory encrypted-memory boundaries, while moving specific cryptographic mechanisms to dedicated hardware [3]. Despite the strong architectural isolation in Arm’s CCA, the concept is too new to be applied in current HPC systems which are not yet powered by Armv9 processors (top500.org). NVIDIA. With the Hopper H100, NVIDIA extended confidential computing to GPU accelerators. The PCIe interconnect between Confidential Virtual Machines (CVMs; i. e., VMs running under AMD SEV-SNP or Intel TDX) and the GPU is protected by hardware-enforced authenticated encryption using AES-GCM, ensuring that data traversing the CPU-GPU interface remain confidential and protected against interception or tampering. In addition, the GPU’s firmware and boot process are cryptographically measured and remotely attested, allowing a CVM to verify the integrity of the GPU before entrusting it with sensitive workloads. Vendor documentation and early measurements indicate that compute-heavy workloads reach near-native performance, whereas I/O-bound jobs pay a higher overhead due to encrypted paging across the CPU–GPU interconnect [47]. Functionally, AMD SME/SEV-SNP and Intel TME/MKTME/TDX converge on inline AES with key isolation and CVM-grade attestation, while Arm CCA/RME focuses on per-domain memory encryption and a system-level isolation model. NVIDIA extends these principles to accelerators with end-to-end encrypted data transfer between the CPU and the GPU. For our use case of secure scientific computing on a multi-tenant HPC system, the AMD platform is the preferred choice because of its native 256-bit hardware encryption engines, the mature ecosystem for running multi-tenant encrypted VMs, its long-established kernel support and simple deployment. Although Intel-based CPUs offer more sophisticated attestation mechanisms and will likely catch up in the future, they currently lack operational experience and require a more complex setup. Arm CPUs with Confidential Compute Architecture seem promising, but ARM-based HPC clusters are not yet available, and limitations in compiler ecosystems and application porting present additional barriers for now.
8
Heger et al.
2.4. Key Management: Secure Key Servers and Life Cycle Management Encryption alone is not sufficient in secure computing environments. Secure key management is equally important, as cryptographic protection is ruined with compromised keys or improper key handling [36]. Therefore, modern secure computing environments rely on centralised key management services or hardware security modules (HSMs) to organize the generation, storage, rotation, access control, auditing, and retirement of keys. Among five evaluated cryptographic key management systems (KMSs), HashiCorp Vault stood out for its suitability to small businesses in terms of features and usability [38]. Vault’s enterprise version supports HSM integration, automatic unsealing, and FIPS-compliant seal wrapping (https://developer.hashicorp.com/vault/docs/enterprise/hsm). On the enterprise side, Thales provides Luna HSMs and cloud HSM services, which can integrate with Vault or other KMSs to ensure keys never leave tamper-resistant hardware and comply with regulatory requirements [57]. In multicloud or HPC settings, centralised key life cycle management has been argued to be critical: For example, Thales’s White Paper describes how disparate cloud consoles and BYOK (bring-your-own-key) scenarios complicate governance unless a unified key management layer is used [56]. In the HPC or cloud computing context, deploying such key management services means the compute environment can encrypt data volumes at rest, wrap keys via HSMs, enforce role-based access, log key use, handle key rotation and retirement, and integrate with job schedulers or container orchestration. Without robust key management, even strong encryption is undermined by poor key hygiene.
3. Secure hardware implementation 3.1. File encryption File encryption on RAMSES is implemented through the data encryption capabilities of IBM Storage Scale (GPFS) and the Thales CipherTrust Manager (CTM) as a management tool for cryptographic keys. This layout enforces strict access control to data at rest and ensures that encrypted storage areas are accessible exclusively to authorized users. RAMSES’s architecture for the management of file encryption keys is layered and involves three core concepts: the File Encryption Key (FEK), the Master Encryption Key (MEK), and Remote Key Management (RKM). The FEK is a newly generated, unique key; each individual file within an encrypted domain is encrypted by its own FEK. The MEK encrypts the FEKs created during file operations; it is specific to a given file set and securely stored in a hardware security module. Finally, the RKM—implemented via the Thales CipherTrust Manager (CTM)—orchestrates FEK encryption/decryption and secure MEK handling, and protects these critical cryptographic operations with a dedicated, tamper-resistant HSM (Figure 1). Importantly, file encryption is fully transparent from the user perspective: read, write, copy, and move operations proceed without impacting the user’s workflow, as encryption and decryption are performed automatically at the file system level.
RAMSES: Next-generation HPC security
9
Unencrypted storage
RKM /projects/fileset_a file1 + FEK1 MEK_a
file2 + FEK2 file3 + FEK3 Encrypted GPFS
/projects/fileset_b
Encrypted TLS
file4 + FEK4 file5 + FEK5
MEK_b
L u n a K 7
file6 + FEK6
Fig. 1: GPFS-based file encryption on RAMSES. File encryption is managed through the GPFS kernel module and the Thales CipherTrust Manager for Remote Key Management (RKM). Two encrypted file sets (fileset a, fileset b) with three files each are shown. Each file is encrypted via its unique File Encryption Key (FEK1-6), which is stored in the file’s extended attributes. FEKs in turn are encrypted by a file set-specific MEK (Master Encryption Key; MEK a and MEK b) stored safely in the Luna K7 HSM of the Thales CipherTrust Manager. FEK traffic between file sets and the RKM uses an encrypted TLS connection. Access to encrypted file sets is controlled via security policies of the secure partition and corresponding certificates.
However, prior to operation, system administrators must define appropriate encryption policies for each encrypted domain by specifying the target file set, the MEK to apply, and the encryption algorithm to be used. Once a file set is designated for encryption, the following sequence of actions is executed transparently in the background when creating or accessing an encrypted file: The proprietary GPFS kernel module generates a new random FEK with the CPU’s hardware-based random number generator locally on the compute node and encrypts the file with the FEK using the CPU’s AES engine. It then establishes a TLS connection to the CTM, authenticates via its X.509 public key certificate, and transmits the plaintext FEK to the CTM over the encrypted TLS channel, requesting that it be wrapped with the MEK specified for the file set. Within the CTM, the request is passed to the Luna K7 HSM which retrieves the MEK from its internal secure storage, uses it to encrypt (wrap) the FEK, and returns the encrypted FEK to the CTM software layer. Thus, all MEK operations occur strictly within the HSM boundary. MEKs never leave the tamper-resistant HSM hardware and do not exist in the memory of the CTM or the host CPU. After the encrypted FEK is sent back to the compute node over TLS, the GPFS kernel module stores the FEK in the file’s extended attributes. Similar processes take place when accessing an already existing encrypted file. In this case, an encrypted FEK is read from the file’s metadata, sent to the CTM over TLS, unwrapped inside the CTM’s HSM, and returned in plaintext to the compute node over encrypted TLS, where it is used to decrypt the file. Thus, for file encryption to operate correctly, each compute node or virtual machine must be individually configured to authenticate against the CTM via its KMIP (Key Management Interoperability Protocol) interface in order to obtain
10
Heger et al.
the required MEK. On RAMSES, KMIP certificates for authentication are embedded in the file set-specific VM disk images stored read-only on nodes of the secure partition, i. e. the KMIP configuration is already in place upon start-up of the transient VM overlay images. The drawback of this per-node configuration is substantial administrative overhead, as the numerous virtual machines of a multitenant HPC environment all require independent RKM configuration. Moreover, any file set designated for encryption can be accessed only by nodes that have been fully configured in advance; incomplete or missing KMIP configuration prevents decryption and renders the file set inaccessible to that node. 3.2. Memory encryption The ultimate goal of a secure HPC system is that sensitive data is encrypted at all times, and therefore safe at rest, in transit, and in use. To achieve this, we combined virtualization techniques, sophisticated key management, and the Slurm workload manager [29] with dedicated hardware to obtain a coherent workflow for secure data processing (Figure 2). To provide a secure HPC environment, we implemented a distinct «secure partition» on RAMSES, consisting of AMD EPYC Genoa nodes configured with Secure Encrypted Virtualization (SEV) [1,32]. These nodes are hardware-identical to standard 192-core SMP nodes but are booted with SEV enabled in the system firmware, allowing the memory controller’s AES engine to transparently encrypt memory pages, i. e., encryption is performed entirely in hardware without requiring modifications to applications or the guest operating system. Because SEV activation is controlled at the firmware level, the size of the secure partition can be flexibly adjusted by toggling SEV support and rebooting selected nodes. Secure execution is provided through virtual machines managed with a KVM (kernel-based virtual machine; https://linux-kvm.org) and QEMU (quick emulator; https://www.qemu.org) software stack. While KVM acts as the in-kernel hypervisor, QEMU supplies the virtual hardware model and device emulation. During VM instantiation, the hypervisor verifies SEV availability using an extended function leaf for hardware-level memory encryption present on AMD processors. If enabled, the hypervisor sets the SEV bit in the VM control structure before issuing VMRUN, causing the guest OS to boot with SEV-protected memory. As a consequence, the hypervisor cannot read encrypted memory contents, protecting VMs booted under SEV from malicious administrators or compromised hypervisors. RAMSES’s secure VMs are based on customized Red Hat Enterprise Linux 9.6 images that include KMIP client certificates to retrieve file encryption keys from the Thales CipherTrust Manager. For each user or project, administrators provision dedicated virtual machines with a unique certificate and passphrase for Thales authentication and key retrieval, allowing the user decryption of their corresponding file sets (see section 3.1). To enable storage access within secure VMs, the InfiniBand interconnect must also be made available in a virtualized form. Since a physical host channel adapter to the InfiniBand network cannot be shared directly across multiple VMs, we employed Single Root I/O Virtualization (SR-IOV) [49] on RAMSES’s ConnectX-6 InfiniBand controllers to expose up to 127 virtual functions (VFs) from a single
RAMSES: Next-generation HPC security
11
InfiniBand card [46]. These VFs can be assigned to different VMs running on a single node, providing high-performance access to the parallel file system via IPover-IB. Custom automation scripts configure and allocate the VFs to VMs on demand. To take into account that one user or group may submit several secure jobs simultaneously, we provide multiple VM disk images for each encrypted file set. Finally, confidential workloads are integrated into the cluster workflow through Slurm. A custom SPANK plugin for Slurm (available upon request) verifies that the submitting user is authorized for secure computing. Specifically, our plugin confirms that the corresponding VM exists and that the user is member of a group with an assigned secure VM before placing the job onto a secure node. Slurm then triggers VM instantiation and coordinates the retrieval of the necessary Thales keys before the job is executed. In addition, the Slurm plugin defines custom parameters that allow administrators to access and modify the permanent storage of specific secure VMs within the limits of Slurm jobs. Importantly, the secure VMs of RAMSES are transient, i. e., they are destroyed upon shutdown. They operate on temporary overlay files derived from immutable backing disk images (read-only). Thus, any modifications made to regular operating system directories during runtime are discarded. Only job output and result files are stored in encrypted form in mounted GPFS directories and will therefore be accessible after job completion. In combination, these mechanisms provide end-to-end confidentiality for data at rest (through KMIP and Thales-managed encryption) and data in use (through AMD SEV).
4. Secure software implementation 4.1. User access via multi-factor authentication (MFA) RAMSES provides SSH access for unprivileged users via dedicated login nodes. They can be reached directly from within the University of Cologne (UoC) campus network or via the UoC virtual private network (VPN). VPN access requires multifactor authentication using the password for the UoC identity management system (uniKIM) [27] and an additional factor. At the University of Cologne, the Cisco Duo application [5] is the predominant solution to deliver the second factor. The most common MFA scenario involves authentication through the Duo Mobile App, available for Android, iOS, iPadOS, and watchOS. The second factor can then be a simple touch response to the app’s pop up question upon SSH login. Using a mobile device has the advantage of introducing another safety layer created by the device’s own access control (PIN, face recognition, or other). Registration of the Duo Mobile App is restricted to the campus network, while external users are subject to an additional identity verification step. As alternatives to a mobile device, we also support hardware tokens and time-based one-time passwords (TOTP) for MFA, the latter only for temporary accounts, such as course accounts. Finally, SSH access to the RAMSES login nodes again requires MFA, using SSH key-based authentication [54] and the specified second factor (Figure 2). To
12
Heger et al.
support users in generating SSH keys and to streamline key upload to RAMSES, we deployed a dedicated web service accessible from within the UoC network (Figure 3) [28]. The frontend is implemented in Go and incorporates a JavaScript component that performs key generation locally on the client device. The tool supports ED25519 elliptic-curve keys and RSA (Rivest–Shamir–Adleman) keys with a minimum length of 4096 bits and enforces passphrase protection with defined complexity requirements (e. g., length, character classes). The backend component provisions the public key via an LDAP (Lightweight Directory Access Protocol) directory service, making it available to RAMSES for authentication. This approach ensures that private keys remain exclusively on the user’s device while simplifying onboarding for users with limited experience in SSH-based access. As some users may be unfamiliar with the concept of SSH keys and MFA-related mechanisms, we made comprehensive RAMSES documentation publicly available on GitHub (https://gitlab.git.nrw/uzk-itcc-hpc/itcc-hpc-ramses/-/wi kis/home) and provide beginner-oriented tutorials that describe access and login procedures (link to video). 4.2. Administrative access and operating system hardening Access to the system follows the principles of privileged access management, where physical and logical system components are separated into different security levels, each with different security measures and restrictions. Privileged access is strictly unidirectional: from higher to lower security levels, to prevent escalation of privileges [61]. As such, nodes and services dedicated to system administration tasks are placed at the higher security level, while nodes directly accessible by unprivileged users, such as login and compute nodes, are at the lowest level (Figure 4). To enforce strict security measures, administrative nodes are placed behind hardened jump hosts and protected with multi-factor authentication and restrictive firewall policies. Internally, administrative services are further differentiated into the following hierarchy: Critical services, including administrative logins, monitoring and logging, image and package repositories occur at the highest security level. The level below contains provisioning and resource management services for login and compute nodes. Administrative nodes running critical services follow unidirectional privileged access, in which management connections must originate from the nodes with higher security levels to the nodes with lower security levels, with all reverse connections blocked. This is enforced by firewall rules and restrictive SSH configurations. 4.3. Information Security Management System (ISMS) To govern the secure handling of sensitive data on RAMSES, the IT Center Cologne (ITCC) is implementing an Information Security Management System (ISMS) in accordance with the ISO/IEC 27001 standard [26]. An ISMS is a structured, risk-based framework designed to establish, implement, maintain, and continually improve information security within an organization.
RAMSES: Next-generation HPC security
13
The ISMS scope covers all organizational and technical aspects of processing and storing sensitive data on RAMSES. It is specifically designed to support research workflows that include special-category data as defined by Article 9 of the General Data Protection Regulation (GDPR), such as human genomic data and data derived from medical imaging procedures [14]. Relevant use cases within the scope of the ISMS are rooted primarily in the life sciences and encompass commissioned data processing, research collaborations, diagnostic workflows, and data processing in the context of third-party funded projects. The core services provided within the Secure Computing and Storage Environment (SCSE) comprise secure compute resources for the analysis of sensitive data, high-security storage capacity for such data, and operational support for the use of these resources. From a technical perspective, data confidentiality within the SCSE is enforced through multiple complementary mechanisms: Data at rest are protected by encrypted file systems (see section 3.1 and Figure 2); data in use are protected by memory encryption during processing (see section 3.2 and Figure 2); and all data transport is secured by encrypted communication channels. These technical measures operate in conjunction with access-control mechanisms and a layered security layout (Figure 3, 4) that are enforced throughout the system. Of note, our organizational implementation of the ISMS together with the comprehensive encryption architecture described in this article go beyond the requirements of the ISO/IEC 27001 standard. Information Security concerning the ITCC and its services is handled by the ITCC Security Operations Team. All security incidents and related issues outside the ITCC fall under the responsibility of the UoC Chief Information Security Officer (CISO), who directly answers the UoC rectorate. More information is provided in the UoC guidelines on information security [59].
5. Validation of the approach in the biomedical domain To analyse the impact of encryption on job performance, we executed two representative workflows from the life sciences—the primary field with requirements in confidential computing—under various encryption modes. As an I/O-heavy workload, we used the software RepeatMasker [53] which identifies repeat elements in a genome sequence. In addition, we used the software BWA-MEM2 [60] for aligning short read sequences against a reference genome—a life science workflow with high memory requirements. In total, we devised seven different encryption regimes and measured workflow execution times, CPU efficiency, and memory consumption in six replicates per setup and workflow. Our analyses reveal that node-exclusive jobs on standard SMP nodes without any form of encryption (native bare-metal) complete fastest in both workflows (Figure 5, SMP-M⊖F⊖). As expected, running the same jobs in the most elaborate encryption configuration—from within a VM with memory and file encryption (VM-M⊕F⊕)—takes the longest. While the difference in execution times between the least secure (SMP-M⊖F⊖) and the most secure job variant (VMM⊕F⊕) is 4.4 % for the I/O-heavy RepeatMasker workflow, this difference increases
14
Heger et al.
to 18.0 % for the memory-intensive BWA-MEM2 workflow (Table 2). Thus, the full encryption setup is associated with a measurable performance decrease that may vary depending on the particular workflow, but retains high-throughput processing capabilities for confidential data. Software consisting of alternating parallel and serial stages is common in bioinformatics pipelines, e. g. for variant calling [39] or genome annotation [17]. We conclude from our RepeatMasker results that workflows with alternating parallel and serial stages experience only minor performance penalties under the full encryption scenario (Figure 5), possibly because performance drops during serial stages outweigh overheads introduced by encryption and virtualization, as illustrated by the reduction of CPU efficiency from 91 % to 68 % between BWA-MEM2 and RepeatMasker runs (Figure 5). However, to verify if this interpretation is valid in general, it would be necessary to analyse the runtime behaviour of similar workflows in more detail. Compared to standard runs, the full encryption scenario introduces overhead from virtualization, memory encryption (SEV), and file system encryption. To isolate the impact of virtualization on job performance, we carried out runs from within a VM on standard SMP nodes lacking memory and file encryption (VMM⊖F⊖) and compared them to bare-metal performance (SMP-M⊖F⊖). Our results indicate that substantial parts of the observed encryption costs originate from virtualization (Figure 5, Table 2). In particular, virtualization accounts for a penalty of 8.1 %, or 45.0 % of the overall overhead (18.0 %), in the BWA-MEM2 use case and for a penalty of 2.2 %, or 50.0 % of the overall overhead (4.4 %) in the RepeatMasker test environment, suggesting that the abstraction layer is responsible for roughly half of the observed performance losses (Table 2). Although an optimized environment with explicit CPU pinning and NUMA-aware memory binding (non-uniform memory access) may reduce overhead, the virtualization costs reported here align well with published data [19,22,37]. Importantly, the results described here represent common workloads from widely used bioinformatics applications that have not been designed for benchmark studies or the operation in an encrypted environment. As such, our setup might involve undesirable algorithm-specific branching patterns, cache behavior, and adaptive processing paths which are less pronounced in synthetic benchmark design. Our workflows might thus emphasize performance differences between job types (e. g., with and without encryption) that are less visible in carefully devised benchmark tests, as already put forward in our previous work [35]. Which type of encryption is more costly, file encryption, or memory encryption? In our setting, adding file encryption to a memory-encrypted context has almost no effects on performance compared to otherwise identical runs without file encryption, whether on SME nodes or within a virtual machine. Similarly, enabling memory and file encryption without the use of a dedicated VM has negligible impact on job performance, as demonstrated by runs on memory-encrypted SME nodes (Figure 5, SME-M⊕F⊕ vs. SME-M⊕F⊖ and SMP-M⊖F⊖). In contrast, combining virtualization with file and memory encryption reduces performance by up to 18 % relative to the unencrypted bare-metal reference. As mentioned above, approximately half of this penalty is attributable to virtualiza-
RAMSES: Next-generation HPC security
15
tion overhead. The remainder is predominantly caused by memory encryption, as combining file encryption with virtualization in the absence of memory encryption is largely performance-neutral in both workflows (VM-M⊖F⊖ vs. VM-M⊖F⊕). This effect is particularly pronounced in the memory-intensive BWA-MEM2 workflow, where enabling memory encryption alone accounts for nearly the full performance reduction observed in the combined encryption scenario (VM-M⊕F⊖ vs. VM-M⊕F⊕), suggesting that memory bandwidth saturation may increase memory encryption overhead in memory-bound workloads. We therefore conclude that memory encryption and virtualization overhead jointly account for the majority of the observed performance penalty, whereas file encryption imposes no significant cost regardless of the execution environment. Despite its performance impact, the use of virtual machines for the secure handling of file system keys and certificates is a critical part of our security architecture, and therefore a replacement with less resource-intensive alternatives is currently not meaningful. Nevertheless, the two representative workflows tested here demonstrate that RAMSES is capable of executing production jobs under full encryption with acceptable performance.
Application
Job Type
Mean [s]
BWA-MEM2
SMP-M⊖F⊖ SME-M⊕F⊖
RepeatMasker
Std Dev [s]
Rel. Runtime [%]
2895.66
8.60
100.0
3041.57
75.05
105.0
SME-M⊕F⊕
2938.13
5.50
101.5
VM-M⊖F⊖
3128.91
29.96
108.1
VM-M⊖F⊕
3112.62
36.10
107.5
VM-M⊕F⊖
3404.14
16.24
117.6
VM-M⊕F⊕
3416.86
9.18
118.0
SMP-M⊖F⊖
16 563.32
19.45
100.0
SME-M⊕F⊖
16 859.92
247.82
101.8
SME-M⊕F⊕
16 737.50
103.17
101.1
VM-M⊖F⊖
16 927.98
44.08
102.2
VM-M⊖F⊕
17 073.85
159.94
103.1
VM-M⊕F⊖
17 083.98
205.83
103.1
VM-M⊕F⊕
17 296.67
206.23
104.4
Table 2: Runtime summary of two workloads under various encryption regimes (job types). Relative runtime of the bare-metal SMP node (SMP-M⊖F⊖) is set to 100 %. M: memory encryption; F: file encryption; ⊕: enabled; ⊖: disabled.
16
Heger et al.
6. Conclusions The RAMSES platform demonstrates that it is possible to design HPC environments which meet the most stringent data protection requirements without sacrificing the scientific productivity that HPC users expect. By integrating three complementary layers—hardware-based memory encryption (AMD SEV-SNP) [1], end-to-end file system encryption organized through the Thales CipherTrust Manager (FIPS-validated HSM) [56,57], and a secure identity-and-access framework built on multi-factor authentication and hardened zones for privileged acces—RAMSES enforces end-to-end encryption across the full data life cycle: at rest, in transit, and in use. Comparison with cloud-based confidential computing. Confidential computing services in the public cloud (e. g., on AMD SEV-SNP or Intel TDX instances) provide strong isolation but typically require users to migrate data to external environments, incurring additional legal and logistical overheads for regulated datasets. In addition, cloud users are dependent on what the cloud provider supports in terms of audit logs and monitoring interfaces, which may be limited. In contrast, RAMSES retains data within the university’s trusted boundary, complying with GDPR, ISO/IEC 27001, and NIST SP 800-53 requirements while offering the same hardware-level guarantees as cloud providers [7]. Moreover, the on-premises deployment permits detailed auditing, custom policy enforcement, and physical access control that are often unavailable or cost-prohibitive in public clouds, especially as our services are provided to users free of charge. Sensitive data never leave the university’s trusted infrastructure, ensuring full institutional control and preventing risks associated with third-party data custody. Advances over existing secure HPC architectures. Compared with certified HPC centres that rely solely on organisational measures such as ISO 27001 certification, or with Trusted Research Environments that restrict protection to data at rest and in transit [58,34,55], RAMSES extends the security envelope to encompass data in use through hardware-enforced memory encryption—a capability absent from previously reported secure HPC prototypes, including the GWDG workflow [45]. By implementing memory encryption at the hardware level and automating key management through the Thales CipherTrust Manager, RAMSES addresses this gap without compromising usability: a single Slurm parameter activates a secure execution environment. The necessary virtual machine and cryptographic keys are provisioned automatically, and the user’s workflow requires no modifications. Performance impact and scalability. Our benchmark suite—representing both I/O-bound (RepeatMasker) and memory-bound (BWA-MEM2) bioinformatics pipelines—shows that the full security stack incurs a modest overall runtime penalty (4 % to 18 %). Detailed analysis reveals that roughly half of this overhead originates from virtualization, while memory encryption contributes the remainder in memory-intensive workloads; file-system encryption adds negligible cost. These figures align with recent studies on confidential-computing overheads [19,22,37] and confirm that RAMSES can support production-scale scientific workloads with acceptable throughput.
RAMSES: Next-generation HPC security
17
Future work. To further narrow the performance gap and broaden the security envelope, we plan to: – Automate VM handling: automate VM provisioning to reduce administrative overhead and implement secure and reliable VM logging mechanisms. – Reduce virtualization overhead: investigate lightweight container-based isolation (e. g., Kata Containers [33]) and NUMA-aware VM tuning to minimise the VM-related penalty while preserving the secure key-management model. – Extend confidential computing to accelerators: integrate confidential compute capabilities of both NVIDIA and AMD GPUs, thereby protecting sensitive GPU-resident data, while taking advantage of multi-tenant GPU sharing. Conclusion. RAMSES establishes a practical blueprint for next-generation secure HPC: a tightly integrated stack that satisfies regulatory mandates, matches the security guarantees of cloud confidential computing, and delivers performance competitive with traditional HPC systems. By making our source code available upon request, we enable interested sites to evaluate, adapt, and build upon this architecture, supporting the emergence of a future class of HPC platforms that can handle sensitive scientific data without compromise.
7. Materials and Methods 7.1. I/O-intensive workloads with encryption To analyse the influence of encryption on job performance, we executed typical workflows from the life sciences under various encryption schemes and compared the outcome to standard runs without encryption. All measurements were conducted in six replicates on exclusively reserved compute nodes to avoid interference from unrelated workloads. As an example of I/O-heavy workloads, we used RepeatMasker version 4.1.5 [53] to identify repeat elements in the genome sequence of the great tinamou, Tinamus major , with the help of a repeat library that we previously created [48]. For this purpose, we used the tinamou reference genome assembly ASM3246655v1 (https://www.ncbi.nlm.nih.gov/datasets/genome/GCA_0 32466555.1/) and a de novo library containing 13 712 distinct repeat elements, collected from 25 bird species using RepeatModeler [16] and Mitetracker [9]. RepeatMasker runs were carried out in SMP (symmetric multi-processor) mode using 12 cores and 20 GB main memory, resulting in runtimes of approximately 280 min. We performed seven identical runs for each of the following encryption configurations: 1) Standard SMP node without memory and without file encryption; 2) SME-enabled node with memory encryption, without file encryption; 3) SMEenabled node with memory encryption and file encryption; 4) Standard SMP node plus VM, without memory and without file encryption; 5) Standard SMP node plus VM, without memory, but with file encryption; 6) SEV-enabled node plus VM with memory encryption, without file encryption. 7) SEV-enabled node plus VM with memory and file encryption. We recorded the runtimes of the jobs (results of the time and seff <jobid> command) and plotted their means and standard deviations using jupyter lab 4.5.2 (https://jupyter.org/) and a standard python
18
Heger et al.
stack for scientific computing (matplotlib 3.10.8; numpy 2.4.1; pandas 3.0.0; plotly 6.5.2; seaborn 0.13.2; scipy 1.17.0). 7.2. Memory-bound workloads with encryption We used BWA-MEM2 version 2.2.1 [60] to align Illumina short read sequences against a reference genome assembly—a typical workflow from the life sciences with high memory but low I/O requirements. BWA-MEM2 runs were carried out in SMP (symmetric multi-processor) mode using 48 cores and 200 GB main memory, resulting in runtimes of approximately 50 min. In each run, we mapped Illumina HiSeq X Ten paired end sequencing reads (downloaded from the European Nucleotide Archive; accession: SRR7733443 with 378 590 104 reads) against the Homo sapiens GRCh38 primary genome assembly (https://ftp.ensembl.org/ pub/release-115/fasta/homo_sapiens/dna/Homo_sapiens.GRCh38.dna.pr imary_assembly.fa.gz) using standard parameters (https://github.com/bwa -mem2/bwa-mem2?tab=readme-ov-file). Like above, analyses were carried out under seven different encryption layouts.
8. Code availability – The source code for the custom Slurm SPANK plugin and VM provisioning is available to academic institutions upon request. – The source code for the RAMSES key upload website is available to academic institutions upon request.
9. Acknowledgements We thank the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation)—project number INST 216/512-1 FUGG, the MKW (Ministry of Culture and Science of the German state of North Rhine-Westphalia—project number 124-4-01.03.02), and the BMFTR (Federal Ministry of Research, Technology and Space) for funding; Oliver Kaiser and Lion Rexhepi for programming the JavaScript component of the RAMSES SSH key upload web service; Dr. Anja Kootz (Institute for African Studies and Egyptology, UoC) for help with RAMSES’s hieroglyphic artwork; Nikolai Wansart, Aileen Diefenthal, and Mauritius Hoevels for help with the RAMSES video tutorials.
10. Author’s contributions Viktor Achter: conceptualization, funding acquisition, writing—review and editing. Stefan Borowski: software and hardware implementation, writing—review and editing. Michael Commer: writing—review and editing. Peter Heger: formal analysis, investigation, methodology, validation, visualization, writing—original draft, writing—review and editing. Lech Nieroda: conceptualization, investigation, methodology, software and hardware implementation, writing—review and editing.
RAMSES: Next-generation HPC security
19
Roland Pabel: software and hardware implementation. Martin Peifer: Resources. Christoph Stollwerk: software and hardware implementation, writing—review and editing. Kamil Tokmakov: writing—review and editing. Stefan Wesner: funding acquisition, writing—review and editing.
References 1. Advanced Micro Devices, Inc.: AMD SEV-SNP: Strengthening VM isolation with integrity protection and more. Tech. rep., AMD (Jan 2020), https://www.amd.com/ content/dam/amd/en/documents/epyc-business-docs/white-papers/SEV-SNP-s trengthening-vm-isolation-with-integrity-protection-and-more.pdf 2. Akram, A., Giannakou, A., Akella, V., Lowe-Power, J., Peisert, S.: Performance analysis of scientific computing workloads on general purpose TEEs. In: 2021 IEEE International Parallel and Distributed Processing Symposium (IPDPS). pp. 1066– 1076 (May 2021), https://doi.org/10.1109/IPDPS49936.2021.00115 3. Arm Ltd.: Realm Management Extension (RME) — System Architecture. Tech. rep., Arm Ltd. (Jun 2021), https://documentation-service.arm.com/static/60d330 9b677cf7536a55bae0 4. Chen, C., Nguyen, D.T., Lee, S.J., Baker, N.A., Karakoti, A.S., Lauw, L., Owen, C., Mueller, K.T., Bilodeau, B.A., Murugesan, V., Troyer, M.: Accelerating computational materials discovery with machine learning and cloud high-performance computing: from large-scale screening to experimental validation. Journal of the American Chemical Society 146(29), 20009–20018 (2024), https://doi.org/10.102 1/jacs.4c03849 5. Cisco Systems, Inc.: Complete identity security and MFA solutions | Duo Security (2026), https://duo.com/, accessed: 2026-01-14 6. Cloud-Security-Alliance: Survey report—security practices in HPC & HPC cloud (2020), https://cloudsecurityalliance.org/artifacts/survey-report-securit y-practices-in-hpc-cloud, accessed: 2025-10-27 7. Confidential Computing Consortium: Confidential computing: Hardware-based trusted execution for applications and data. Whitepaper Version 1.3, Confidential Computing Consortium, Linux Foundation (Nov 2022), https://confidentialcomp uting.io/wp-content/uploads/sites/10/2023/03/CCC_outreach_whitepaper_up dated_November_2022.pdf 8. Coppolino, L., D’Antonio, S., Mazzeo, G., Romano, L.: An experimental evaluation of TEE technology: Benchmarking transparent approaches based on SGX, SEV, and TDX. Computers & Security 154, 104457 (2025), https://www.sciencedirect.com/ science/article/pii/S0167404825001464 9. Crescente, J.M., Zavallo, D., Helguera, M., Vanzetti, L.S.: MITE Tracker: an accurate approach to identify miniature inverted-repeat transposable elements in large genomes. BMC Bioinformatics 19, 348 (Oct 2018), https://doi.org/10.1186/s128 59-018-2376-y 10. Cyber Risk Institute (CRI): The CRI profile: A financial sector use case for the NIST cybersecurity framework. Tech. rep., Cyber Risk Institute (May 2023), https: //cyberriskinstitute.org/the-profile 11. Department of Defense Chief Information Officer, Zero Trust Portfolio Management Office: DoD zero trust strategy. Strategy Document Version 1.0, United States Department of Defense, Office of the Chief Information Officer (DoD CIO), Washington, D.C., USA (Oct 2022), https://dodcio.defense.gov/Portals/0/Documents/Libra ry/DoD-ZTStrategy.pdf, prepared by the Zero Trust Portfolio Management Office (ZT PfMO). Initial published version: 21 October 2022.
20
Heger et al.
12. Desai, T., Ritchie, F., Welpton, R.: Five Safes: Designing data access for research. Economics Working Paper Series 1601, University of the West of England, Bristol, Bristol, UK (2016), https://www2.uwe.ac.uk/faculties/BBS/Documents/1601.pdf 13. Dokmai, N., Zhu, K., Sahinalp, S.C., Cho, H.: Secure phasing of private genomes in a trusted execution environment with TX-Phase. Genome Research 35(12), 2626–2636 (2025), http://genome.cshlp.org/content/35/12/2626.abstract 14. European Parliament, Council of the European Union: Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation). Regulation 2016/679, Official Journal of the European Union (2016), https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32016R0679, oJ L 119, 4.5.2016, pp. 1–88. Corrigendum: OJ L 127, 23.5.2018, p. 2 15. Financial Stability Board (FSB): Summary report on financial sector cybersecurity: Regulations, guidance and supervisory practices. Tech. rep., Financial Stability Board (Oct 2017), https://www.fsb.org/wp-content/uploads/P131017-1.pdf 16. Flynn, J.M., Hubley, R., Goubert, C., Rosen, J., Clark, A.G., Feschotte, C., Smit, A.F.: RepeatModeler2 for automated genomic discovery of transposable element families. Proceedings of the National Academy of Sciences of the United States of America 117, 9451–9457 (Apr 2020), https://doi.org/10.1073/pnas.1921046117 17. Gabriel, L., Brůna, T., Hoff, K.J., Ebel, M., Lomsadze, A., Borodovsky, M., Stanke, M.: Braker3: Fully automated genome annotation using RNA-seq and protein evidence with Genemark-ETP, AUGUSTUS, and TSEBRA. Genome Research 34, 769– 777 (Jun 2024), https://doi.org/10.1101/gr.278090.123 18. Gaddam, S., Luykx, A., Sinha, R., Watson, G.: Reducing HSM reliance in payments through proxy re-encryption. In: Proceedings of the 30th USENIX Security Symposium. pp. 4061–4080. USENIX Security ’21, USENIX Association (2021), https://www.usenix.org/conference/usenixsecurity21/presentation/gaddam 19. Giallorenzo, S., Mauro, J., Poulsen, M.G., Siroky, F.: Virtualization costs: Benchmarking containers and virtual machines against bare-metal. SN Computer Science 2(5) (2021), https://doi.org/10.1007/s42979-021-00781-8 20. GlobalPlatform: TEE Internal Core API Specification, public release. Technical Specification (2021), https://globalplatform.org/wp-content/uploads/2021/03/GPD _TEE_Internal_Core_API_Specification_v1.3.1_PublicRelease_CC.pdf 21. Guo, Y., Chandramouli, R., Wofford, L., Gregg, R., Key, G., Clark, A., Hinton, C., Prout, A., Reuther, A., Adamson, R., Warren, A., Bangalore, P., Deumens, E., Farkas, C.: High-performance computing security: Architecture, threat analysis, and security posture. Tech. Rep. NIST SP 800-223, National Institute of Standards and Technology, Gaithersburg, MD (2024), https://doi.org/10.6028/NIST.SP.800-223 22. Hanussek, M., Bartusch, F., Krüger, J.: Performance and scaling behavior of bioinformatic applications in virtualization environments to create awareness for the efficient use of compute resources. PLoS Computational Biology 17, e1009244 (Jul 2021), https://doi.org/10.1371/journal.pcbi.1009244 23. Hussain, M.A., Samrouth, K., Bakir, N.: A survey on malware attacks in industrial air-gap systems. International Journal of Information Security 24(3), 146 (2025), https://doi.org/10.1007/s10207-025-01044-w 24. Intel Corporation: Intel® hardware shield – Intel® Total Memory Encryption. Tech. rep., Intel Corporation (2022), https://www.intel.com/content/dam/www/centra l-libraries/us/en/documents/white-paper-intel-tme.pdf 25. International Organization for Standardization and International Electrotechnical Commission: ISO/IEC 19790:2025 — Information security, cybersecurity and pri-
RAMSES: Next-generation HPC security
21
vacy protection — Security requirements for cryptographic modules. International Standard 19790, ISO/IEC (2025), https://www.iso.org/standard/82423.html 26. ISO/IEC: ISO/IEC 27001:2022 — information security, cybersecurity and privacy protection — information security management systems — requirements. International Standard ISO/IEC 27001:2022, International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC), Geneva, Switzerland (Oct 2022), https://www.iso.org/standard/27001, third edition, published 2022-10-25. Amendment 1 published 2024-02-23 (Climate action changes). Technical Committee: ISO/IEC JTC 1/SC 27. ICS: 35.030 27. ITCC, University of Cologne: Identity management – uniKIM (2024), https://itcc .uni-koeln.de/en/services/accounts-communication/identity-management-u nikim, accessed: 2026-01-14 28. ITCC, University of Cologne: RAMSES SSH Key Upload (2026), https://ramses-u mc.itcc.uni-koeln.de/web-sshkey/login, accessed: 2026-01-14 29. Jette, M.A., Wickberg, T.: Architecture of the Slurm workload manager. In: Klusáček, D., Corbalán, J., Rodrigo, G.P. (eds.) Job Scheduling Strategies for Parallel Processing. pp. 3–23. Springer Nature Switzerland, Cham (2023), https://link.springer. com/chapter/10.1007/978-3-031-43943-8_1 30. Joint Task Force Transformation Initiative: Security and privacy controls for information systems and organizations (NIST special publication 800-53 rev. 5). Tech. Rep. 800-53 Rev. 5, National Institute of Standards and Technology (Sep 2020), https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final 31. Jung, W., Lee, E., Kim, S., Kim, J., Kim, N., Lee, K., Min, C., Cheon, J.H., Ahn, J.H.: Accelerating fully homomorphic encryption through architecture-centric analysis and optimization. IEEE Access 9, 98772–98789 (2021), http://dx.doi.org/10.1109/A CCESS.2021.3096189 32. Kaplan, D., Powell, J., Woller, T.: AMD Memory Encryption – White Paper. Advanced Micro Devices, Inc. (2021), https://www.amd.com/content/dam/amd/en/doc uments/epyc-business-docs/white-papers/memory-encryption-white-paper.p df 33. Kata Containers Community: Kata containers — open source container runtime software. Open Infrastructure Foundation (2024), https://katacontainers.io, accessed: 2024 34. Kavianpour, S., Sutherland, J., Mansouri-Benssassi, E., Coull, N., Jefferson, E.: Nextgeneration capabilities in Trusted Research Environments: Interview study. J Med Internet Res 24(9), e33720 (Sep 2022), https://doi.org/10.2196/33720 35. Keßler, R., Nieroda, L., Volpert, S., Gräf, M., Achter, V., Hunhold, L., Wesner, S.: Evaluating trusted execution environment performance for genome sequence alignment: An AMD SEV case study. In: Proceedings of the SC ’25 Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis. p. 1951–1958. SC Workshops ’25, Association for Computing Machinery, New York, NY, USA (2025), https://doi.org/10.1145/3731599.3767558 36. Krähenbühl, C., Perrig, A.: Key management. In: Mulder, V., Mermoud, A., Lenders, V., Tellenbach, B. (eds.) Trends in Data Protection and Encryption Technologies, pp. 15–20. Springer Nature Switzerland, Cham (2023), https://doi.org/10.1007/97 8-3-031-33386-6_4 37. Kuity, A., Peddoju, S.K.: Investigating performance metrics for container-based HPC environments using x86 and OpenPOWER systems. Journal of Cloud Computing 12(1) (Dec 2023), https://doi.org/10.1186/s13677-023-00546-z 38. Kuzminykh, I., Ghita, B., Shiaeles, S.: Comparative analysis of cryptographic key management systems. In: Galinina, O., Andreev, S., Balandin, S., Koucheryavy, Y.
22
Heger et al.
(eds.) Internet of Things, Smart Spaces, and Next Generation Networks and Systems. pp. 80–94. Springer International Publishing, Cham (2020), https://link.springe r.com/chapter/10.1007/978-3-030-65729-1_8 39. McKenna, A., Hanna, M., Banks, E., Sivachenko, A., Cibulskis, K., Kernytsky, A., Garimella, K., Altshuler, D., Gabriel, S., Daly, M., DePristo, M.A.: The Genome Analysis Toolkit: a MapReduce framework for analyzing next-generation DNA sequencing data. Genome Research 20, 1297–1303 (Sep 2010), https://doi.org/10.1 101/gr.107524.110 40. Meftah, S., Tan, B.H.M., Aung, K.M.M., Yuxiao, L., Jie, L., Veeravalli, B.: Towards high performance homomorphic encryption for inference tasks on CPU: An MPI approach. Future Generation Computer Systems 134, 13–21 (2022), https://www.sc iencedirect.com/science/article/pii/S0167739X22001145 41. National Institute of Standards and Technology: FIPS 140-2: Security requirements for cryptographic modules, update 2 (including Annex A: Approved security functions). Tech. rep., National Institute of Standards and Technology (May 2001), https://csrc.nist.gov/files/pubs/fips/140-2/upd2/final/docs/fips140 2annexa.pdf 42. National Institute of Standards and Technology: FIPS 197: Advanced Encryption Standard (AES). Tech. rep., National Institute of Standards and Technology (Nov 2001), https://doi.org/10.6028/NIST.FIPS.197-upd1 43. National Institute of Standards and Technology: FIPS 140-3: Security requirements for cryptographic modules. Tech. rep., National Institute of Standards and Technology (Mar 2019), https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.140-3.pdf 44. National Science Foundation: Ten ways advanced computing catalyzes science. https: //phys.org/news/2015-11-ten-ways-advanced-catalyzes-science.html (Nov 2015), retrieved October 27, 2025 45. Nolte, H., Spicher, N., Russel, A., Ehlers, T., Krey, S., Krefting, D., Kunkel, J.: Secure HPC: A workflow providing a secure partition on an HPC system. Future Generation Computer Systems 141, 677–691 (2023), https://www.sciencedirect.com/scienc e/article/pii/S0167739X2200423X 46. NVIDIA / Mellanox: Single root I/O virtualization (SR-IOV) — NVIDIA Mellanox documentation. NVIDIA Mellanox OFED / Networking docs (online) (2023), https: //docs.nvidia.com/networking/display/MLNXOFEDv53100143/Single+Root+IO+ Virtualization+%28SR-IOV%29 47. NVIDIA Corporation: Confidential compute on NVIDIA Hopper H100. White Paper WP-11459-001, v1.0, NVIDIA Corporation (Jul 2023), https://images.nvidia.co m/aem-dam/en-zz/Solutions/data-center/HCC-Whitepaper-v1.0.pdf 48. Palitzsch, K., Wiehe, T., Heger, P.: Decay of the CTCF paralog BORIS in neognathous birds. bioRxiv (2025), https://www.biorxiv.org/content/early/2025/0 2/16/2025.02.12.637905 49. PCI-SIG: Single root I/O virtualization and sharing specification, revision 1.1. Specification (Jan 2010), https://pcisig.com/PCIExpress/Specs/IOV/SingleRootIOVi rtualizationandSharing_1.1 50. Prantl, T., Engel, S., Horn, L., Kaiser, D., Iffländer, L., Bauer, A., Krupitzer, C., Kounev, S.: Performance impact analysis of homomorphic encryption: A case study using linear regression as an example. In: Meng, W., Yan, Z., Piuri, V. (eds.) Information Security Practice and Experience. pp. 284–298. Springer Nature Singapore, Singapore (2023), https://link.springer.com/content/pdf/10.1007/978-981-9 9-7032-2.pdf 51. Rose, S., Borchert, O., Mitchell, S., Connelly, S.: Zero trust architecture (NIST special publication 800-207). Tech. Rep. 800-207, National Institute of Standards and
RAMSES: Next-generation HPC security
23
Technology (Aug 2020), https://nvlpubs.nist.gov/nistpubs/SpecialPublicati ons/NIST.SP.800-207.pdf 52. Schlick, T., Portillo-Ledesma, S.: Biomolecular modeling thrives in the age of technology. Nature Computational Science 1(5), 321–331 (2021), https://doi.org/10.1 038/s43588-021-00060-9 53. Smit, A., Hubley, R., Green, P.: RepeatMasker Open-4.0. 2013–2015. Software (2015), http://www.repeatmasker.org 54. SSH: What is an SSH key? An overview of SSH keys (2025), https://www.ssh.com/ academy/ssh-keys, accessed: 2026-01-14 55. Tenhunen, V., van der Meer, L., Azab, A., Reale, M., Ruzicka, M., Moed, M., Manzi, A., Sipos, G.: Trusted Research Environments – Landscape Report. Tech. rep., EGI Foundation (Oct 2024), https://documents.egi.eu/public/RetrieveFile?docid =4169&filename=EGI_TRE_WG_Landscape_Report_2024-10-25.pdf&version=2 56. Thales Trusted Cyber Technologies: The case for centralized multicloud encryption key management. White paper, Thales Trusted Cyber Technologies (Sep 2022), https: //www.thalestct.com/wp-content/uploads/2022/09/The-case-for-centralized -multicloud-encryption-key-lifecycle-management-tct-wp.pdf 57. Thales Trusted Cyber Technologies: HashiCorp Vault with Luna HSMs – Integration Guide. Integration guide / White paper, Thales Trusted Cyber Technologies (Jul 2022), https://cpl.thalesgroup.com/sites/default/files/content/integratio n_guides/field_document/2022-07/HashiCorpVault_LunaHSM_IntegrationGuid e_RevE.pdf 58. UK Health Data Research Alliance: Trusted Research Environments (TRE) Green Paper. Tech. rep., UK Health Data Research Alliance (2020), https://ukhealthda ta.org/wp-content/uploads/2020/04/200430-TRE-Green-Paper-v1.pdf 59. Universität zu Köln: Informationssicherheitsleitlinie der Universität zu Köln (ISLUzK). Amtliche Mitteilungen 96/2025, Universität zu Köln, Köln, Deutschland (Oct 2025), https://am.uni-koeln.de/e45267/data/records52760/AM_2025-96_ISL_ UzK_ger.pdf, herausgegeben vom Rektor der Universität zu Köln, Albertus-MagnusPlatz, 50923 Köln. Erscheinungsdatum: 17. Oktober 2025. 60. Vasimuddin, M., Misra, S., Li, H., Aluru, S.: Efficient architecture-aware acceleration of BWA-MEM for multicore systems. In: 2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS). pp. 314–324 (May 2019), https://doi.org/10.1109/IPDPS.2019.00041 61. Waltermire, K., Conroy, T., Harriston, M., Irrechukwu, C., Krishnan, N., MemoleDoodson, J., Nkrumah, B., Perper, H., Prince, S., Wynne, D.: Sp 1800-18: Privileged account management for the financial services sector. Draft Practice Guide NIST SP 1800-18, National Institute of Standards and Technology (NIST), National Cybersecurity Center of Excellence (NCCoE), Gaithersburg, MD, USA (Sep 2018), https://www.nccoe.nist.gov/sites/default/files/legacy-files/fs-pam-nis t-sp1800-18-draft.pdf 62. Xia, W., Sakurai, M., Balasubramanian, B., Liao, T., Wang, R., Zhang, C., Sun, H., Ho, K.M., Chelikowsky, J.R., Sellmyer, D.J., Wang, C.Z.: Accelerating the discovery of novel magnetic materials using machine learning–guided adaptive feedback. Proceedings of the National Academy of Sciences of the United States of America 119(47), e2204485119 (2022), https://doi.org/10.1073/pnas.2204485119
24
Heger et al.
Standard workflow
Secure workflow User
1.
MFA with Cisco Duo
UoC VPN/Uklan
2.
MFA with Cisco Duo
Frontend Nodes
3. Slurm SPANK plugin
4.
Secure Partition
Standard Partition
VM KMIP SR-IOV
FEKs
with MEKs
GPFS Encrypted GPFS
CipherTrust Manager with Luna HSM
GPFS
Fig. 2: Secure vs. traditional HPC workflow on RAMSES. 1: UoC members connect to the UoC network using multi-factor authentication (MFA) based on their IDM credentials and Cisco Duo. 2: Registered HPC users login to RAMSES using multi-factor authentication with an SSH key pair and Cisco Duo. 3: On the frontend node(s), users submit their jobscript for confidential (left side) or traditional computing (right side) to the Slurm workload manager. 4: Left: A dedicated Slurm job parameter activates a custom SPANK plugin essential for confidential computing. After successful verification, a user-specific VM is launched on an SEV-enabled node with active memory encryption. The VM disk image includes a KMIP client certificate, enabling transparent decryption of the associated file set via the Thales CipherTrust Manager and SR-IOV–based GPFS access. FEKs travel over encrypted TLS between the secure partition and the CTM while MEKs stay within the Luna HSM. Upon job completion, the VM is terminated and all runtime state is discarded. Right: Jobs submitted without the confidentialcomputing parameter follow the standard Slurm-managed execution path. Secure and traditional workloads coexist on the same system and are selected at submission time: running a secure job requires the addition of a single Slurm parameter. Abbreviations: FEK, file encryption key; IDM, identity management system; MEK, master encryption key; SEV, secure encrypted virtualization; SPANK, Slurm Plug-in Architecture for Node and job (K)control; SR-IOV, single root I/O virtualization; Uklan, University of Cologne local area network; UoC, University of Cologne; VM, virtual machine; VPN, virtual private network.
RAMSES: Next-generation HPC security
25
A RAMSES SSH Key Upload Please enter your UniKIM username and password to log in.
Username:
username
Password:
password
Submit
B
Key Generation Successful!
C
SSH Key Generator
To log into RAMSES via SSH, you need an SSH key pair, see RAMSES Documentation. This tool generates a secure key pair with modern encryption and runs entirely in your browser, i.e., your keys never leave your local machine. If you loose the private key, we can not recover it for you because we never possessed it.
Please note: • After submitting a public key here, please allow at least 15 minutes to pass before trying to log into RAMSES.
Please Note: • Choose a strong and unique password which meets the requirements listed below to
• Below, you will find your generated private key, which will allow you to log onto RAMSES using ssh. Store it safely and do not share it with anyone else. • Private keys are usually saved in a directory .ssh, which is located at ~/.ssh (on MacOS/Linux) or %USERPROFILE%\.ssh (on Windows). If this
directory does not exist yet, you will have to create it yourself. • This page does not save your private key. Upon leaving, you will not be able to access the generated private key anymore; Please save your key before submitting this form.
-----BEGIN OPENSSH PRIVATE KEY----b3BlbnNzaC1rZXktdjEAAAAACmFlczI1Ni1jdHIAAAAGYmNyeXB0AAAAGAAAABD4322o pk 6tp3LCvwhe3e5oAAAAGAAAAAEAAAAzAAAAC3NzaC1lZDI1NTE5AAAAIGBw+EddhEvNmS Bm krthsFJv86Z47bJoLdP8OnOf5fPMAAAAkPJBBbjNz5hEyJ2LK/ QoYlOonmIqdOJcEgzKwE KNWE2WMqL+5r6rKPSxvNhpf8oZMOPxQEDl8IaT7pGxN6CDjo1uNz0C1bDQVfi/
encrypt your private key. This password is required to access your private key and cannot be recovered if lost. • While the password requirements are helpful as a guideline, they do not guarantee that your password is safe. For example, it is strongly discouraged to reuse the same password for multiple services, even if it meets all password requirements. There are password generators available, for example included in most password managers, to make generating a truly secure and unique password easier. • Enter a descriptive comment for your key (e.g. “ramses-access”). This helps you identify the key later and will appear at the end of your public key file. • After choosing a password and clicking "Generate", you will be presented with your private key, which you MUST then download and save on your local machine. Finally, the tool will allow you to upload the public key to our servers, allowing you to use the private key to log into RAMSES. • Important: Your private key has to be kept strictly confidential. • This tool only allows you to add keys to your RAMSES account. If you want to remove an existing key from your account, you need to log onto RAMSES and delete the line containing that key from your Copy to clipboard ~/.ssh/authorized_keys file.
Password
uEqhQlOw /eWaeHrsC8F0fHleRVFYuzTwIkMrhd9h0X407EPfmvr2Cx1n/ c2b7wtxUEidq+DG0cwwDV
Confirm Password
KvESIvt8Ajk+4/6Q== -----END OPENSSH PRIVATE KEY-----
Password Requirements (3 character classes required)
Download SSH private key Comment
Generate I confirm that I have saved the password.
Fig. 3: The RAMSES SSH key upload website. A: UoC members with approved RAMSES account can connect to the RAMSES key upload website (https://ramses -umc.itcc.uni-koeln.de/web-sshkey/login) from within the University network, using their UoC IDM credentials and Cisco Duo. B, C: To improve user experience and security, a JavaScript-Plugin carries out SSH key pair generation and ensures its protection with a strong password. The public key is subsequently copied to the user’s ~/.ssh/authorized_keys file on RAMSES while the private key, including its passphrase, must be downloaded to the user’s computer. SSH keys have been generated successfully.
26
Heger et al.
s
lo
S3: S2: S1: S0: images, repositories, admin login, audits ad min master gi n a ser vic es ode nd c n ompute
Fig. 4: RAMSES’s security layout. The physical and logical system components of RAMSES are separated into four security levels, S0 to S3, illustrated as concentric circles. Unprivileged users can only access login and compute nodes of the lowest security level, S3 (outer ring). Access to nodes of higher security levels (S0 to S2) is restricted to privileged accounts and implemented through unidirectionality, hardened entry points, blocking of reverse connections, strict firewall policies, and restrictive SSH configurations.
RAMSES: Next-generation HPC security
27
RepeatMasker CPU [%]
68.6
67.7
68.2
91.8
91.6
91.6
n.a.
n.a.
n.a.
n.a.
n.a.
n.a.
Runtime [s]
15000
10000
5000
0
BWA-MEM2 CPU [%]
n.a.
n.a.
Runtime [s]
3000
2000
1000
⊕ VM F⊕ -M ⊖ F⊖ VM -M ⊖ F⊕ VM -M ⊕ F⊖ VM -M ⊕ F⊕
M E-
⊕ F⊖ M SM
E-
SM
SM
P-
M ⊖
F⊖
0
Job Type Fig. 5: Performance of two contrasting workloads under various encryption regimes. RepeatMasker (top; I/O-heavy) and BWA-MEM2 (bottom; memoryintensive) runs were carried out in seven different configurations with six replicates each: 1) Standard SMP node without memory and without file encryption (SMP-M⊖F⊖); 2) SME-enabled node with memory encryption, without file encryption (SME-M⊕F⊖); 3) SME-enabled node with memory encryption and file encryption (SME-M⊕F⊕); 4) VM on standard SMP node without memory encryption and without file encryption (VM-M⊖F⊖); 5) VM on standard SMP node without memory encryption, but with file encryption (VM-M⊖F⊕); 6) VM on SEV-enabled node with memory encryption, without file encryption (VM-M⊕F⊖); 7) VM on SEV-enabled node with memory and file encryption (VM-M⊕F⊕). Numbers on top denote average CPU efficiency of the respective job type. Red circles represent individual measurements. n.a.: data not available. M: memory encryption; F: file encryption; ⊕: enabled; ⊖: disabled.