Official Review · ICLR 2026

The paper presents LongCA-Bench, a unified benchmark framework designed to evaluate long-context attention mechanisms across both single-device and distributed settings. The author integrates existing attention kernels (dense and sparse) and distributed context-parallel strategies into a modular, extensible evaluation platform. Through extensive experiments on up to 96 GPUs, the benchmark compares efficiency, scalability, and memory usage across diverse mask patterns, sequence lengths, and distributed configurations of existing methods. The work provides a valuable empirical reference for practitioners and researchers working on long-context training but does not seem to introduce novel attention algorithms or distributed mechanisms beyond benchmarking existing approaches. 1. **Comprehensive benchmarking.** The paper systematically benchmarks a wide range of attention implementations, including dense, sparse, and distributed mechanisms, under a unified framework. The experimental coverage (up to 96 GPUs and multiple mask types) is extensive and provides a clear view of current attention efficiency trends. 2. **Sound experimental methodology.** The experiments are well-organized, use realistic settings (e.g., long context lengths, different mask patterns), and report consistent metrics (throughput, memory). The methodology is reproducible and technically competent. 3. **Useful empirical reference.** The benchmark results could serve as a useful reference for practitioners and researchers seeking guidance on the performance trade-offs of existing attention kernels and distributed strategies. 1. **Limited Analysis.** The paper reports extensive throughput and memory results, but offers limited discussion on the underlying causes of observed performance trends of the benchmarked methods. 2. **Incomplete coverage of the most critical setting — distributed sparse attention.** The integration of sparse attention (particularly dynamic block-sparse attention with TopK/TopP selection criterion) into distributed contexts remains an unexplored and practically important challenge. The paper does not explore this aspect (does not touch block sparse distributed attention, only discusses full/causal/document), which somewhat limits its impact given the stated motivation of benchmarking “long-context attention.” 1. How can the proposed framework offer insight on sparse attention in distributed settings? Typically, how should we overlap communication with computation, and address the load balancing problem of arbitrary block sparse pattern? The author does not need to provide a complete solution to this problem, and what I would like to see is how this work could contribute to future research on these challenges. 2. Could the authors provide more analysis to understand the trends observed in kernel or distributed scaling performance?