Invisible Yet Dominant: Big Stalls of Kernel I/O Mechanisms in Cloud OLTP Databases Mitsumasa Kondo NTT, Inc. [email protected]
arXiv:2609.12597v1 [cs.DB] 11 Sep 2026
1
Introduction
The standard block device in the cloud is distributed block storage. Because cloud providers must manage their infrastructure efficiently while guaranteeing data durability, directattached storage (DAS) offered by most cloud services is volatile: it cannot be used by applications such as databases that require persistence and durability. Consequently, compute– storage disaggregated databases [2, 5, 6, 8, 9]—physically separating the compute layer from the storage layer—have become the dominant architecture for high-performance cloud databases. The driving reason is a well-known phenomenon: when a database that issues many random writes runs on top of distributed block storage, its I/O path stalls and high performance cannot be achieved. The core idea of disaggregation is to provision database-dedicated storage and to write only the database’s Write-Ahead Log (WAL) directly to it, bypassing the kernel I/O layer; database pages are then reconstructed from the WAL on the storage side, eliminating random writes. Despite the success of this approach, the random-write stalls themselves have remained poorly understood. They have conventionally been attributed to network PPS ceilings, bandwidth limitations, and general kernel-layer overhead. SteelDB [7] confronted this folklore head-on: why do random writes actually stall on distributed block storage? It discovered that the root cause is a design misalignment across three layers—the distributed block storage, the kernel, and the database—and proposed the SteelDB architecture, a crosslayer orchestration zero-patch architecture that achieves high performance without modifying any of these layers. Specifically, the kernel’s flusher thread (KFT), responsible for writing dirty pages back to disk, operates as a single thread per block device. On distributed block storage, whose per-I/O latency is inherently higher due to network-attached access and data replication, this single KFT cannot drain dirty pages fast enough, triggering the kernel’s memory protection mechanism—IO-less Dirty Throttle [1]—which forcibly pauses foreground write system calls. Furthermore, distributed block storage typically exposes only two I/O queues per device for multi-tenant QoS isolation, and this limited number of queues exacerbates congestion when the single KFT cannot drain dirty pages promptly. SteelDB resolves these bottlenecks, achieving 3.1× the throughput of Amazon Aurora [9] and 1.4× that of Google AlloyDB [2] on TPC-C at less than half the cloud cost. Its advantage is not
Figure 1. Linux Kernel Writeback Architecture limited to runtime metrics: an analysis of historical release records shows that Aurora takes a median of 254 days to port its proprietary patches to a new PostgreSQL major version, versus zero days for the patch-free SteelDB. Whereas SteelDB evaluated performance from the database perspective, this paper shifts focus to the kernel perspective. We use eBPF to analyze, from inside the kernel, the bottlenecks that SteelDB resolved. These stalls are largely invisible to standard counters such as /proc/diskstats, and we quantify their impact on cloud OLTP performance.
2
Analysis of Kernel I/O Bottlenecks
SteelDB’s cross-layer resolution. Figure 1 illustrates the kernel I/O path from a database process to distributed block storage, including the bottlenecks described in section 1. SteelDB resolves these by provisioning multiple disks and strategically placing database data across them, thereby multiplying both the number of KFTs and I/O queues while physically isolating their I/O paths. This enables parallel dirty-page writeback, alleviates I/O queue congestion, suppresses IO-less Dirty Throttle, and delivers high database performance. The SteelDB architecture is the product of a cross-layer exploration spanning the kernel’s internal I/O characteristics, the architecture of distributed block storage, and the I/O patterns of the database. For further details we refer the reader to the original paper. eBPF-based profiling. We developed an eBPF-based tool that visualizes kernel I/O-path stalls invisible to standard counters. It attaches to the block:block_rq_issue and blo ck:block_rq_complete tracepoints to capture every request traversing the block layer, from which it monitors the aggregate disk I/O bandwidth over time (Figure 2a). To classify
(a) Disk I/O bandwidth (total provisioned: 80K IOPS, 2.0 GB/s) via block:block_rq_issue/complete.
(b) Disk I/O bandwidth by issuing context (KFT writeback vs. foreground), via writeback:writeback_start/written.
(c) IO-less Dirty Throttle pause events captured via writeback:balance_dirty_pages.
Table 1. TPC-C Benchmark Results and Disk I/O. Method 1: Single disk 2: WAL split 3: SteelDB
Throughput Ave Write. Ave Read. Max Trans (NOPM) (MB/s) (MB/s) Lat. (ms) 444,326 511,480 544,354
679.4 754.1 827.1
50.7 62.1 60.2
1,238.2 848.1 502.5
each request by issuing context—separating KFT-originated writeback from other contexts such as foreground fsync (Figure 2b)—it tags each bio with its issuing context at submission time. It then correlates these events with the writeback:writ eback_start/writeback_written tracepoints, which mark the intervals during which a kworker is executing wb_writeb ack(). In addition, to capture IO-less Dirty Throttle pauses— intentional delays of write system calls when dirty pages exceed the threshold—we attach to the writeback:balance_di rty_pages tracepoint and aggregate invocations into a timeseries bin graph (Figure 2c). Evaluation setup. All experiments were conducted on AWS using Rocky Linux 9.8 with Linux kernel 5.14. The database was PostgreSQL 16.4, and TPC-C [4] benchmarks were driven by HammerDB [3]. The VM instance type was c6in.8xlarge (32 vCPUs, 64 GB RAM). The database size was 1K WH (approximately 100 GB) with 256 VU. We compared three disk configurations with identical total provisioned IOPS (80K) and bandwidth (2.0 GB/s): Method 1 uses a single gp3 volume (1 disk); Method 2 separates the WAL onto a dedicated disk (WAL: 10K IOPS, 0.7 GB/s; DATA: 70K IOPS, 1.3 GB/s); Method 3 is the SteelDB configuration with 4 disks— WAL (10K IOPS, 0.7 GB/s), and three tablespaces (30K IOPS, 0.6 GB/s; 25K IOPS, 0.5 GB/s; 15K IOPS, 0.2 GB/s).
3
Evaluation and Future Work
Our eBPF analysis reveals that Method 1 loses a substantial fraction of its provisioned I/O capacity to kernel-internal stalls that are entirely invisible to standard profiling tools. The impact on database performance is significant: as Table 1
shows, Method 3 (SteelDB), which eliminates these stalls, delivers the highest throughput—a 23% improvement over Method 1—and the average write bandwidth improves by 21.7%. More tellingly, Method 3 reduces the maximum New– Order transaction latency by 59.4% (1,238.2 ms → 502.5 ms). Figure 2a shows the disk I/O bandwidth over time. Despite being provisioned with the highest single-volume IOPS (80K), Method 1 fails to reach its ceiling and delivers the lowest sustained bandwidth. Method 3, by splitting I/O paths across multiple devices, momentarily reaches the provisioned bandwidth ceiling. The cause becomes clear in Figure 2b, which separates I/O by issuing context. Method 3 achieves the highest KFT writeback throughput because its three DATA disks provide three independent KFTs that drain dirty pages in parallel. In Method 1, the single KFT is saturated; dirty pages back up, and the majority of writeback is instead driven by foreground processes through checkpoint fsync and other synchronous paths—an inherently less efficient mechanism that further contends with user-facing I/O. This KFT saturation triggers the final link in the causal chain: IO-less Dirty Throttle. Figure 2c plots the pause events over time. Method 1, with its overwhelmed single KFT, suffers 2,513 pause episodes across the run; Method 2 reduces this to 2,094 through WAL separation, but the DATA-side single-KFT bottleneck persists. Method 3 records only 755 pauses—a 70.0% (2,513 → 755) reduction from Method 1—and these residual pauses are notably sparse and low in amplitude. The substantial reduction in pauses directly explains the 59.4% improvement in maximum latency: the tail stalls that dominated Method 1’s worst-case response times are suppressed. In this paper, we leverage eBPF to quantitatively identify kernel I/O bottlenecks in cloud database environments— invisible yet dominant overheads that standard profiling tools fail to capture. As future work, we will further investigate these cloud-specific kernel I/O bottlenecks and explore their relationships with existing kernel tuning parameters.
References [1] 2012. Fengguang Wu. IO-less Dirty Throttling. Accessed: March 30, 2026. https://events.static.linuxfound.org/images/stories/pdf/lcjp2012_ wu.pdf [2] 2026. AlloyDB for PostgreSQL. Accessed: March 30, 2026. https: //cloud.google.com/products/alloydb [3] 2026. HammerDB. Accessed: March 30, 2026. https://www.hammerdb. com/ [4] 2026. TPC-C Homepage. Accessed: March 30, 2026. https://www.tpc. org/tpcc/ [5] Panagiotis Antonopoulos, Alex Budovski, Cristian Diaconu, Alejandro Hernandez Saenz, Jack Hu, Hanuma Kodavalla, Donald Kossmann, Sandeep Lingam, Umar Farooq Minhas, Naveen Prakash, Vijendra Purohit, Hugh Qu, Chaitanya Sreenivas Ravella, Krystyna Reisteter, Sheetal Shrotri, Dixin Tang, and Vikram Wakade. 2019. Socrates: The New SQL Server in the Cloud. In Proceedings of the ACM International Conference on Management of Data. (SIGMOD ’19). 1743–1756.
[6] Alex Depoutovitch, Chong Chen, Jin Chen, Paul Larson, Shu Lin, Jack Ng, Wenlin Cui, Qiang Liu, Wei Huang, Yong Xiao, and Yongjun He. 2020. Taurus Database: How to be Fast, Available, and Frugal in the Cloud. In Proceedings of the ACM International Conference on Management of Data. (SIGMOD ’20). [7] Mitsumasa Kondo. 2026. SteelDB: Diagnosing Kernel-Space Bottlenecks in Cloud OLTP Databases. arXiv:2603.29052 [cs.DB] https://arxiv.org/ abs/2603.29052 [8] Xi Pang and Jianguo Wang. 2024. Understanding the Performance Implications of the Design Principles in Storage-Disaggregated Databases. In Proceedings of the ACM International Conference on Management of Data. (SIGMOD ’24), Vol. 2. 180:1–180:26. [9] Alexandre Verbitski, Anurag Gupta, Debanjan Saha, Murali Brahmadesam, Kamal Gupta, Raman Mittal, Sailesh Krishnamurthy, Sandor Maurice, Tengiz Kharatishvili, and Xiaofeng Bao. 2017. Amazon Aurora: Design Considerations for High Throughput Cloud-Native Relational Databases. In Proceedings of the ACM International Conference on Management of Data. (SIGMOD ’17). 1041–1052.