PhotonZip: Lifecycle-Aware Compression for HDF5-Based Synchrotron Workflows

PhotonZip: Lifecycle-Aware Compression for HDF5-Based Synchrotron Workflows Haoxu Li1,2 , Wenjing Huang1,2 , Chengyu Liu2,3 , Shiyuan Fu3 , Hao Hu3 , Guangming Tan1 , and Dingwen Tao1

arXiv:2611.04138v1 [cs.DC] 3 Nov 2026

1

Institute of Computing Technology, Chinese Academy of Sciences 2 University of Chinese Academy of Sciences 3 Institute of High Energy Physics, Chinese Academy of Sciences taodingwen@ict.ac.cn

Abstract. Synchrotron detectors generate data at rates that challenge storage and I/O systems. Compression can reduce data movement, but codec-only measurements do not capture the end-to-end behavior of HDF5 workflows. We present PhotonZip, a lifecycle-aware compression framework that tests candidate modes through a common HDF5 path and selects among eligible measured configurations according to either compression ratio or round-trip throughput. Its modular filters preserve standard HDF5 access when the required plugin is available. Across four beamline-derived datasets, PhotonZip reaches lossless compression ratios up to 1.629× and HDF5 write throughput up to 3.40GB/s write and 7.75GB/s read. The throughput-oriented configuration achieves 1.529× lossless data reduction while improving HDF5 write and read throughput by 21.2% and 15.0%, respectively. Lossy preview modes provide up to 16× data reduction and reduce preview latency by up to 8.35 s in the evaluated file-per-rank HDF5 configuration. Keywords: Scientific data compression · HDF5 filters · Synchrotron data · MPI

1

Introduction

Modern synchrotron light sources employ high-frame-rate detectors that continuously generate large volumes of scientific images [15]. These data are commonly represented as multi-byte integer arrays and persisted as chunked HDF5 datasets for online processing, downstream analysis, and long-term archival [5, 10, 13]. As detector resolution, acquisition rate, and the number of beamlines continue to increase, data production is growing faster than the available storage capacity and I/O bandwidth. Data compression is therefore important not only for reducing storage consumption, but also for decreasing the volume of data transferred through acquisition and storage pipelines. Applying compression in a production HDF5 workflow, however, is more difficult than selecting a codec with high standalone throughput. First, codec performance on in-memory buffers does not directly translate into end-to-end

2

H. Li et al.

HDF5 performance, which is jointly affected by compression, data transformation, chunk and metadata management, synchronization, and storage I/O [9]. Second, a higher compression ratio often requires a more complex compression pipeline, introducing additional computation and reducing throughput. Compression ratio and throughput must therefore be considered as a workloaddependent trade-off rather than optimized independently. Third, comparative studies show that the relative effectiveness of compressors and preprocessing pipelines can change across datasets and operating configurations [3, 17]. Finally, deploying compression in an actual scientific workflow requires users to understand codec-specific parameters, manually compare alternatives, integrate them with HDF5, and preserve sufficient metadata for subsequent decompression. These requirements impose a substantial configuration and integration burden on beamline users. To address these challenges, we present PhotonZip, a lifecycle-aware compression system for HDF5-based synchrotron data workflows. Rather than relying on one codec for every data product, PhotonZip maps workflow intent to complementary compression paths: lossy compression for online processing when information loss is acceptable and lossless compression for persistent archival. PhotonZip tests candidate modes through a common HDF5 procedure and uses hard eligibility constraints followed by a single-objective decision rule to select a measured configuration. Codec-specific filter properties and payload headers preserve the parameters required for decoding, allowing applications to retain standard HDF5 read and write calls when the corresponding plugin is installed. We further characterize the complete I/O path using file-per-rank MPI execution and backend threading rather than treating compression as an isolated kernel. In summary, this paper makes the following contributions:

– Transparent compression for HDF5 workflows. We integrate complementary lossless and lossy backends as HDF5 filters whose identifiers, parameters, and payload headers support standard HDF5 reads and writes when the required plugins are available. – Goal-directed compression configuration. We develop a lightweight configuration mechanism that tests candidate modes through a common HDF5 procedure, filters the measured candidates by fidelity and data compatibility, and selects the best configuration for the user’s ratio or throughput objective. – Modular and extensible implementation. We provide modular interfaces that separate compression backends, preprocessing strategies, configuration policies, and HDF5 integration, allowing additional compression algorithms to be incorporated without redesigning the end-to-end data path. – End-to-end workflow characterization. We evaluate PhotonZip using beamline-derived datasets and a file-per-rank MPI benchmark, quantifying compression effectiveness and complete HDF5 I/O performance across process–thread configurations.

PhotonZip for HDF5-Based Synchrotron Workflows

3

Large data throughput beamlines: ID07, ID09, ID21, ID19, ID30 …

Fig. 1. PhotonZip’s integration point in a synchrotron data workflow.

2

Background and Gap Analysis

2.1

Synchrotron Data Workflows

Many synchrotron imaging experiments use high-frame-rate detectors to acquire sequences of multidimensional images. The resulting multi-byte integer arrays move through data acquisition to online processing, persistent storage, and downstream analysis [5, 13]. Their distributions, noise levels, and temporal correlations vary across detectors and experiments, producing dataset-dependent compression behavior [3]. HDF5 stores multidimensional arrays and metadata and underlies scientific formats such as NeXus [6, 10]. Its chunked filter pipeline can compress each chunk during a write and decompress it on demand during a read [16]. Applications retain standard HDF5 interfaces when the required filter is available, while the file records the filter and parameters needed for decoding. Figure 1 shows the target integration point. PhotonZip is placed between data acquisition and HDF5 storage to reduce downstream data movement without changing surrounding HDF5 interfaces. When information loss is acceptable, it can generate a lossy product for online processing; the archival branch instead applies lossless compression directly to the acquired data. 2.2

Gap Analysis and Design Goals

Standalone codec performance does not represent HDF5 performance. In-memory compression ratio and throughput do not determine whether compression benefits an HDF5 workflow. Data conversion, filter execution, synchronization, metadata, and storage behavior also contribute to end-to-end performance [9]; parallel execution additionally depends on work partitioning and process configuration. A fixed configuration cannot satisfy different workflow objectives. Compression methods occupy different points in the ratio–throughput–fidelity space. PhotonZip currently integrates MANS and LC for lossless storage and PFPL and

4

H. Li et al.

spatial downsampling for lossy preview; their different data types and operating characteristics motivate lifecycle-aware selection. Online products may use either lossy mode when information loss is acceptable, while archival products require lossless compression and may prioritize either ratio or throughput [8, 2]. Codec-specific integration creates deployment burden. HDF5 filters and scientific compressors expose heterogeneous choices such as filter availability, pipeline composition, compression level, and lossy quality setting [16, 2, 11]. Users must translate workflow requirements into these settings and ensure that the corresponding filter remains available for decoding. Tightly coupling this logic to HDF5 access also makes codecs and policies harder to replace or extend. Based on these gaps, PhotonZip has four design goals: – G1: Transparent HDF5 integration. Preserve standard HDF5 reads and writes through installed filters and retain the codec-specific parameters required for decoding. – G2: Lifecycle-aware configuration. Test candidate modes through a common HDF5 procedure, filter them by workflow and fidelity requirements, and select for compression ratio or throughput. – G3: Modular extensibility. Separate backends, preprocessing, configuration, and HDF5 integration so compatible algorithms can be added without redesigning the data path. – G4: End-to-end characterization. Evaluate file-per-rank HDF5 execution with backend threading across process–thread configurations.

3

PhotonZip Design

3.1

System Overview

PhotonZip bridges workflow-level requirements and codec-specific execution through four components: a lightweight configuration procedure, modular compression backends, codec-specific HDF5 filters, and an end-to-end evaluation layer. Users specify a dataset’s purpose, fidelity requirement, and optimization objective. PhotonZip tests eligible modes through a common HDF5 procedure and applies a deterministic rule to choose a measured configuration. Compression and decompression then operate through the selected HDF5 filter without requiring application code to invoke codec kernels directly. The key observation is that synchrotron data products can have different requirements at different lifecycle stages. Archival products require lossless compression, whereas an online consumer may explicitly permit lossy compression to reduce data movement. PhotonZip treats this fidelity choice as a hard constraint before comparing measured candidates by the requested objective. If both products are required, they must be generated as separate branches from the acquired data; applying a lossless codec to lossy reconstructed data does not create a lossless archive.

PhotonZip for HDF5-Based Synchrotron Workflows

3.2

5

Transparent HDF5 Data Path

PhotonZip treats HDF5 as a data contract: applications access logical data arrays through HDF5 calls, while installed filters manage their physical compressed representation. The HDF5 chunk is the encoding boundary. On a write, the selected filter transforms a logical chunk into an encoded buffer; on a read, it applies the inverse path before returning the chunk to the application [6, 16]. HDF5 records the filter identifier and its client parameters in the dataset creation properties, while each backend payload stores additional fields needed by its decoder, such as the original chunk size. These mechanisms preserve codecspecific decoding information, but they do not remove the deployment dependency: a compatible filter plugin must be installed on the reader. The standalone PhotonZip container additionally records array shape, data type, backend parameters, and preprocessing information; this richer container metadata is separate from the current HDF5 filter paths. 3.3

Lifecycle-Aware Goal-Directed Configuration

PhotonZip accepts a compact request consisting of data purpose, a lossless or lossy fidelity choice, the ratio or throughput objective, and an optional quality preset for lossy compression. It first filters the available modes by hard constraints, including fidelity, data type, dimensionality, and backend availability. Lossless requests admit only exactly reconstructable modes, while lossy modes are eligible only when the user permits information loss. Fidelity is therefore established before performance is compared. All eligible modes are tested on the same representative HDF5 sample under a common measurement procedure. Let S denote its logical data, BHDF5 (c, S) the resulting HDF5 storage size, and Cvalid the modes that pass the hard constraints. PhotonZip measures CR(c, S) =

|S| , BHDF5 (c, S)

QRT (c, S) =

2|S| , Twrite (c, S) + Tread (c, S)

(1)

where write and read times cover the complete HDF5 path, including filtering, chunk handling, synchronization, and storage I/O. The user requirement then determines the measured winner: ( arg maxc∈Cvalid CR(c, S), ratio priority, c∗ = (2) arg maxc∈Cvalid QRT (c, S), throughput priority. This single-objective decision requires neither a weighted score nor a Pareto search. The selected mode and parameters are fixed for the complete dataset rather than changed independently for individual chunks. 3.4

Modular Compression Architecture

Figure 2 shows PhotonZip’s HDF5-oriented modular path. After configuration selection, applications retain standard HDF5 read and write calls, while codecspecific chunk filters dispatch data to the selected backend. Filter parameters and

6

H. Li et al.

Workflow purpose, fidelity, and ratio/throughput objective requirements

Apply hard constraints

Test eligible modes through HDF5

Choose measured objective winner

selected configuration

Standard HDF5 read/write interface

decode

Codec-specific HDF5 chunk filters

MANS lossless integers

LC lossless uint16

PFPL lossy float32

Filter parameters + payload headers

Downsample lossy preview

Lossless and lossy backends expose compatible interfaces while retaining their own data paths.

Fig. 2. PhotonZip’s modular organization.

payload headers preserve the state required for decoding. Backend data types, parameters, threading, and encoded formats remain encapsulated, so codecs compatible with the filter interface can be added without redesigning the upper layers. The current implementation integrates MANS for lossless multi-byte integers, LC for lossless 16-bit unsigned data, and two lossy preview modes: PFPL and spatial downsampling. PFPL operates on single-precision data, while downsampling reduces the spatial data volume before reconstruction. Both lossy modes are candidates for the G2 selection procedure and can be compared by compression ratio or end-to-end HDF5 throughput. A separate CLI prototype optionally applies whole-volume inter-frame delta to 16-bit unsigned arrays along axis 0. It computes differences in a widened type and clips them to signed 16-bit values; reconstruction is exact only when no clipping occurs. This prototype is not part of the current HDF5 filters. 3.5

Hybrid Process–Thread Execution

PhotonZip adopts a file-per-rank execution model. Each MPI rank processes a rank-local data slab, applies the same selected mode, and reads or writes a private HDF5 file. HDF5 invokes the selected filter independently for each chunk, so codec execution requires no communication between ranks. MPI coordinates the ranks and aggregates their results. Process-level concurrency is combined with backend-internal OpenMP parallelism. For a fixed CPU budget, PhotonZip assigns approximately P/R threads to each of R ranks, where P is the number of available cores. Overall throughput is computed as PR |Dr | , (3) Q = r=1 maxr Tr

PhotonZip for HDF5-Based Synchrotron Workflows

7

where |Dr | and Tr are the logical source bytes and end-to-end HDF5 time of rank r. This model balances independent HDF5 streams against compression threads per stream while keeping the selected mode unchanged across ranks.

4

Evaluation

4.1

Experimental Setup

We conduct all experiments on a dual-socket Intel Xeon Gold 6530 server. The detailed hardware configuration is summarized in Table 1. Table 1. Hardware configuration of the experimental platform.

4.2

Component

Specification

Processor CPU cores CPU threads NUMA Vector ISA Memory Storage

2 × Intel Xeon Gold 6530 32 cores per socket; 64 physical cores in total 1 thread per core; 64 logical CPUs 2 NUMA nodes AVX-512 503 GB 3.5 TB NVMe SSD

Compression and HDF5 Performance

Figure 3 compares the lossless compression operating points across four datasets. PhotonZip-Ratio consistently favors stronger compression, reaching 1.424×– 1.629× across the four datasets, whereas PhotonZip-Speed trades a small amount of compression ratio for substantially higher codec throughput. The two configurations thus cover complementary lossless operating points for storage- and throughput-oriented workloads. We include a 32 × 2 configuration to examine a decomposition aligned with the two NUMA domains. The process–thread results below show that the best decomposition depends on the selected configuration. We further evaluate these operating points in the end-to-end HDF5 path. Table 3 also includes the two integrated lossy preview modes, PhotonZip-PFPL and PhotonZip-Downsample, which are additional candidates for goal-directed selection and provide substantially higher data reduction. To examine the effect of process–thread decomposition, we further compare representative PhotonZip configurations under different parallel settings. Table 2 reports the resulting end-to-end HDF5 throughput. PhotonZip-Ratio benefits strongly from increased process-level parallelism up to 32 ranks. For PhotonZip-Speed, the 32 × 2 configuration outperforms 64 × 1, indicating that the optimal process–thread decomposition depends on the system configuration and backend execution behavior.

H. Li et al.

Encoding Throughput (MB/s)

JPEG-LS PhotonZip-ratio

APS-2017 103 102 101 1.50

1.52

1.54

1.56

1.58

1.60

Compression Ratio

1.62

1.64

Ti-621

103 102 101 1.44

1.46

1.48

1.50

Compression Ratio

1.52

1.54

Encoding Throughput (MB/s)

Encoding Throughput (MB/s)

JPEG2000 JPEG-XR

Encoding Throughput (MB/s)

8

PhotonZip-speed Pareto frontier

B3 103 102 101 1.34

1.36

1.38

1.40

1.42

1.60

1.62

Compression Ratio

tomo

103 102 101 1.52

1.54

1.56

1.58

Compression Ratio

1.64

Fig. 3. Compression-ratio–throughput trade-off of lossless methods across four datasets. Dashed lines indicate the Pareto frontier. Table 2. PhotonZip HDF5 throughput under different configurations (MiB/s). Method

Ranks × Threads Write Read

PhotonZip-Ratio 1 × 64 8×8 16 × 4 32 × 2

202 993 1374 1995

176 1025 1482 2062

PhotonZip-Speed 32 × 2 64 × 1

3399 3110

7573 4020

PhotonZip-Speed improves both HDF5 write and read throughput over raw I/O while retaining lossless compression. The lossy operating points provide much larger data reduction, with PFPL reaching 15.96× and PhotonZip-Downsample achieving a similar reduction at higher reconstruction quality. Together, these results show that PhotonZip spans a broad trade-off space from lossless highthroughput I/O to aggressive lossy data reduction. 4.3

Ablation of Goal-Directed Configuration

We compare PhotonZip with fixed MANS and LC configurations across four datasets. PhotonZip-Ratio targets compression effectiveness, while PhotonZipSpeed targets throughput.

PhotonZip for HDF5-Based Synchrotron Workflows

9

Table 3. End-to-end HDF5 performance across PhotonZip operating points. Method

Data Reduction Write (MiB/s) Read (MiB/s) PSNR

Raw HDF5 1.00× PhotonZip-Ratio 1.629× PhotonZip-Speed 1.529× PhotonZip-PFPL 15.96× PhotonZip-Downsample 16.00×

2804 2615 3399 3956 60435

6586 3664 7573 6814 29664

– Lossless Lossless 40.2 dB 45.45 dB

PhotonZip vs. Fixed Compression Configurations Across Four Datasets PhotonZip-Ratio

B3

LC

1.614×

249

1.3

185

102

1.2

Fixed MANS

PhotonZipRatio

Fixed LC

1.424×

1.4 1.3

PhotonZipSpeed

103 1.346×

1.284× 241

1.335×

178

1.2

102

1.1

34

1.1 1.0

103

1.0

Throughput (MB/s) Compression Ratio

1.4

1.514×

PhotonZipRatio

Fixed LC

1.3

1.445× 244

PhotonZipSpeed

1.433×

181

1.0

103

102

1.2 1.1

32

Fixed MANS

1.528×

1.378×

35

Fixed MANS

PhotonZipRatio

Fixed LC

4180 1.629×

1.6

1.540×

1.5

1.469×

1.4

246

1.3

1.528×

182

1.0

103

102

1.2 1.1

PhotonZipSpeed

104

1.8 1.7

1.5 1.4

tomo

104 4150

1.6

1.5 1.526×

1.456×

Throughput (MB/s) Compression Ratio

Compression Ratio

1.5

Throughput

Ti-621 1.7

1.7 1.6

PhotonZip-Speed

104 4260

1.6

4105

Throughput (MB/s)

1.8

104

Throughput (MB/s) Compression Ratio

MANS

APS-2017

33

Fixed MANS

PhotonZipRatio

Fixed LC

PhotonZipSpeed

Fig. 4. Comparison of PhotonZip with fixed compression configurations.

PhotonZip-Ratio consistently improves compression ratio over fixed MANS. On tomo, the ratio increases from 1.469× to 1.629×. In contrast, PhotonZipSpeed selects the throughput-oriented configuration and sustains up to 4.18 GB/s end-to-end HDF5 throughput with only a small reduction in compression ratio. These results show that goal-directed configuration enables PhotonZip to adapt to different compression and throughput objectives. 4.4

Synchrotron Workflow Integration

We finally evaluate PhotonZip for fast preview in the synchrotron workflow. Figure 5 compares the reconstructed data produced by PhotonZip-PFPL and PhotonZip-Downsample with the original data. The end-to-end preview time is reduced from 42.33 s for raw data to 39.44 s with PhotonZip-PFPL (2.89 s reduced) and 33.98 s with PhotonZip-Downsample (8.35 s reduced). Since visualization alone takes approximately 33 s, the remaining data-processing overhead is reduced from 9.33 s to 6.44 s with PhotonZipPFPL and to only 0.98 s with PhotonZip-Downsample. These results show that PhotonZip accelerates scientific preview by reducing the data movement and preparation cost while preserving sufficient reconstruction quality for rapid inspection.

5

Related Work

Scientific Data Compression. Scientific compressors span lossless methods, which preserve exact values, and lossy methods, which trade reconstruction fidelity for greater data reduction. Lossless approaches combine entropy coding

10

H. Li et al.

and reversible transformations to exploit structure in typed arrays; MANS targets multi-byte integers, while LC composes lightweight transformations and encoders into configurable pipelines [14, 8, 2]. Representative lossy compressors include SZ, ZFP, MGARD, and SPERR, which use prediction, block transforms, multilevel representations, or wavelet coding [4, 12, 1, 11]. Comparative studies show that compression ratio and throughput vary across datasets and configurations [3, 17]; workflows must therefore choose among complementary methods rather than assume one fixed codec is universally preferable. Compression in HDF5 and Scientific Workflows. HDF5 applies compression through chunk-level filters while preserving its dataset interface [6, 16]. Third-party filters extend the available codecs, but applications must still select a filter, configure codec-specific parameters, and retain the implementation required for decoding. Scientific I/O systems further combine data transformation with parallel storage and movement. Studies of parallel HDF5 compression and ADIOS2 show that workflow performance includes filter execution, synchronization, metadata, layout, and physical I/O, and therefore cannot be inferred from codec-only throughput [9, 7]. These systems provide execution mechanisms for compressed I/O, but do not by themselves provide a workflow-level policy for coordinating lossless and lossy codecs.

6

Conclusion

PhotonZip provides lifecycle-aware compression for HDF5-based synchrotron workflows by combining codec-specific filters, modular backends, and a fidelityfirst single-objective configuration procedure. Across four beamline-derived datasets, its lossless modes reach compression ratios up to 1.629×, while the throughputoriented mode achieves 1.529× data reduction with improved end-to-end HDF5 performance. The lossy preview path provides nearly 16× data reduction and reduces preview latency by up to 8.35 s. These results are obtained on a singlenode, file-per-rank HDF5 configuration and do not represent shared-file Parallel HDF5 or continuous beamline deployment. Acknowledgments. This work was supported by the National Natural Science Foundation of China through the Excellent Young Scientists Fund Program (Overseas) (Grant No. E401470), the National Key Research and Development Program of China (Grant Nos. 2020YFA0907000 and 2025YFB30037002), the National Natural Science Foundation of China (Grant Nos. 62032023 and T2125013), and Ant Group. The AIdriven experiments, simulations, and model training were performed on the robotic AI-Scientist platform of the Chinese Academy of Sciences.

References 1. Ainsworth, M., Tugluk, O., Whitney, B., Klasky, S.: Multilevel techniques for compression and reduction of scientific data—the multivariate

PhotonZip for HDF5-Based Synchrotron Workflows

11

Fig. 5. Reconstruction quality of PhotonZip fast-preview modes. PFPL and PhotonZip-Downsample provide different trade-offs between data reduction and visual fidelity.

case. SIAM Journal on Scientific Computing 41(2), A1278–A1303 (2019). https://doi.org/10.1137/18M1166651 2. Azami, N., Fallin, A., Burtscher, M.: Efficient lossless compression of scientific floating-point data on CPUs and GPUs. In: Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1. pp. 395–409. Association for Computing Machinery (2025). https://doi.org/10.1145/3669940.3707280 3. Bernstein, H.J., Jakoncic, J.: Investigation of fast and efficient lossless compression algorithms for macromolecular crystallography experiments. Journal of Synchrotron Radiation 31(4), 647–654 (2024). https://doi.org/10.1107/S160057752400359X 4. Di, S., Cappello, F.: Fast error-bounded lossy HPC data compression with SZ. In: 2016 IEEE International Parallel and Distributed Processing Symposium. pp. 730–739. IEEE (2016). https://doi.org/10.1109/IPDPS.2016.11 5. Donath, T., Šišak Jung, D., Burian, M., Radicci, V., Zambon, P., Fitch, A.N., Dejoie, C., Zhang, B., Ruat, M., Hanfland, M., Kewish, C.M., van Riessen, G.A., Naumenko, D., Amenitsch, H., Bourenkov, G., Bricogne, G., Chari, A., SchulzeBriese, C.: EIGER2 hybrid-photon-counting X-ray detectors for advanced synchrotron diffraction experiments. Journal of Synchrotron Radiation 30(4), 723–738 (2023). https://doi.org/10.1107/S160057752300454X 6. Folk, M., Heber, G., Koziol, Q., Pourmal, E., Robinson, D.: An overview of the HDF5 technology suite and its applications. In: Proceedings of the 2011

12

H. Li et al.

EDBT/ICDT Workshop on Array Databases. pp. 36–47. Association for Computing Machinery (2011). https://doi.org/10.1145/1966895.1966900 7. Godoy, W.F., Podhorszki, N., Wang, R., Atkins, C., Eisenhauer, G., Gu, J., Davis, P., Choi, J., Germaschewski, K., Huck, K., Huebl, A., Kim, M., Kress, J., Kurc, T., Liu, Q., Logan, J., Mehta, K., Ostrouchov, G., Parashar, M., Poeschel, F., Pugmire, D., Suchyta, E., Takahashi, K., Thompson, N., Tsutsumi, S., Wan, L., Wolf, M., Wu, K., Klasky, S.: ADIOS 2: The adaptable input output system. a framework for high-performance data management. SoftwareX 12, 100561 (2020). https://doi.org/10.1016/j.softx.2020.100561 8. Huang, W., Yang, J., Yin, S., Li, H., Gu, Y., Liu, Z., Jing, X., Wei, Z., Fu, S., Hu, H., Tan, G., Tao, D.: MANS: Efficient and portable ANS encoding for multi-byte integer data on CPUs and GPUs. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. pp. 1299–1314. Association for Computing Machinery (2025). https://doi.org/10.1145/3712285.3759825 9. Jin, S., Tao, D., Tang, H., Di, S., Byna, S., Lukić, Z., Cappello, F.: Accelerating parallel write via deeply integrating predictive lossy compression with HDF5. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. pp. 1–15. IEEE (2022). https://doi.org/10.1109/SC41404.2022.00066 10. Könnecke, M., Akeroyd, F.A., Bernstein, H.J., Brewster, A.S., Campbell, S.I., Clausen, B., Cottrell, S., Hoffmann, J.U., Jemian, P.R., Männicke, D., Osborn, R., Peterson, P.F., Richter, T., Suzuki, J., Watts, B., Wintersberger, E., Wuttke, J.: The NeXus data format. Journal of Applied Crystallography 48(1), 301–305 (2015). https://doi.org/10.1107/S1600576714027575 11. Li, S., Lindstrom, P., Clyne, J.: Lossy scientific data compression with SPERR. In: 2023 IEEE International Parallel and Distributed Processing Symposium. pp. 1007–1017. IEEE (2023). https://doi.org/10.1109/IPDPS54959.2023.00104 12. Lindstrom, P.: Fixed-rate compressed floating-point arrays. IEEE Transactions on Visualization and Computer Graphics 20(12), 2674–2683 (2014). https://doi.org/10.1109/TVCG.2014.2346458 13. Marone, F., Studer, A., Billich, H., Sala, L., Stampanoni, M.: Towards on-the-fly data post-processing for real-time tomographic imaging at TOMCAT. Advanced Structural and Chemical Imaging 3(1), 1 (2017). https://doi.org/10.1186/s40679016-0035-9 14. Masui, K., Amiri, M., Connor, L., Deng, M., Fandino, M., Höfer, C., Halpern, M., Hanna, D., Hincks, A.D., Hinshaw, G., Mena Parra, J., Newburgh, L.B., Shaw, J.R., Vanderlinde, K.: A compression scheme for radio data in high performance computing. Astronomy and Computing 12, 181–190 (2015). https://doi.org/10.1016/j.ascom.2015.07.002 15. SLAC National Accelerator Laboratory: LCLS-II: Linac coherent light source ii. https://lcls.slac.stanford.edu/lcls-ii (2025), accessed January 13, 2025 16. The HDF Group: HDF5 User’s Guide: Data Pipeline Filters. The HDF Group (2026), accessed August 12, 2026 17. Zhao, K., Di, S., Liang, X., Li, S., Tao, D., Bessac, J., Chen, Z., Cappello, F.: SDRBench: Scientific data reduction benchmark for lossy compressors. In: 2020 IEEE International Conference on Big Data. pp. 2716–2724. IEEE (2020). https://doi.org/10.1109/BigData50022.2020.9378449

Related documents

• Effectiveness and predictability of in-network storage cache for Scientific Workflows#1038674
• High-Performance Star-M SVD for Big Data Compression#194285
• WZIP and WLZ4: LZ77 Codecs with Match-Length-Dependent Sliding Windows#1324680
• FSZ: Breaking the Prediction-Throughput Trade-off in GPU Lossy Compression#381719
• FTP data compression#208839