arXiv CS open access

LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexing

2026 · arxiv_cs

arXiv CS · Papers · 2026

cloud distributed-computing parallel-computing

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexing Wen Zan, Jiaqi Zhang∗, Jianchao Tan, Hong Liu, Cunguang Wang, Xiang Li, Duyue Ma, Guanyu Wu, Yifan Lu, Fengcun Li, Yerui Sun, Peng Pei, Yuchen Xie, Xunliang Cai Meituan LongCat Team

arXiv:2608.01662v1 [cs.AI] 3 Aug 2026

A BSTRACT DeepSeek Sparse Attention (DSA) enables efficient long-context modeling through its Lightning Indexer. However, practical deployment remains constrained by the indexer’s expensive O(L2 ) scoring overhead and the hardware-inefficient, discontinuous memory-access patterns induced by its outputs. To address these system-level bottlenecks, we introduce LongCat Sparse Attention (LSA), a hardware–algorithm co-designed framework comprising three complementary and orthogonal strategies: (1) Streaming-Aware Indexing, which selectively converts scattered KV entries into hardware-aligned contiguous layouts to enable coalesced HBM access; (2) Cross-Layer Indexing, which amortizes indexing overhead by reusing the results produced by a single layer across consecutive layers, supported by cross-layer distillation; and (3) Hierarchical Indexing, which adopts a coarse-to-fine scoring scheme to progressively narrow the candidate set for each query, thereby substantially reducing indexing computation. Extensive scaling experiments, ranging from 69B-A3B to 560B-A27B models, demonstrate that LSA consistently achieves performance on par with full attention across both generalpurpose and long-context benchmarks. Moreover, LSA supports native training with context lengths of up to one million tokens and underpins the development of LongCat-2.0 (1.6T-A48B). To facilitate further research, we also introduce and open-source LongCat-Flash-Lite-Sparse (69B-A3B), which integrates LSA into LongCat-Flash-Lite and incorporates an updated long-context training corpus. Directly Reusing the Indices from the Owner Layer Sharing Parameters Top-k Indices Non-Streaming Tokens

Block Indexer

Top-k Indices

Top-k Selector

Token Indexer Top-k Selector

Non-Contiguous KV (~50% budget)

Top-k Selector

Attn Streaming Tokens

Full KV Tokens

Indexed KV tokens

Query Token

Attn

Contiguous KV (~50% budget)

Full KV Tokens

Indexed KV tokens

Query Token LSA from the Owner Layer

LSA from the Reuse Layer

Figure 1: The architecture of the proposed LongCat Sparse Attention (LSA), featuring the streaming-aware hierarchical cross-layer indexer. (Sink tokens are omitted for clarity.)

Corresponding authors: zhangjiaqi39@meituan.com

LongCat Sparse Attention

Contents 1

Introduction

Sparse Attention and the DSA Baseline

2.1

From Fixed Patterns to Retrieval-Based Sparse Attention . . . . . . . . . . . . . . . . . . . . . . . .

2.2

A Brief Recapitulation of DSA Mechanism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2.3

Profiling the Efficiency Bottlenecks of DSA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

LongCat Sparse Attention (LSA)

3.1

Streaming-Aware Indexing: Improving Locality for Hardware-Aligned Coalesced Access . . . . . . .

3.2

Cross-Layer Indexing: Amortizing Indexing Overhead via Inter-layer Pattern Redundancy . . . . . .

3.3

Hierarchical Indexing: Coarse-to-Fine Sparse Selection . . . . . . . . . . . . . . . . . . . . . . . . .

Kernel Design and Efficiency Analysis

4.1

Efficient Kernel Design and Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4.1.1

Hybrid Sparse Attention Operator for SI: Partitioning and Overlapped Execution . . . . . . .

4.1.2

Index-Selection Operator for HI: Two-Stage Coarse-to-Fine Selection . . . . . . . . . . . . .

4.2

Attention Layer Training Speedup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4.3

End-to-End Inference Speedup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Experiments and Results

5.1

Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5.2

Results of LSA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5.3

Ablation Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5.3.1

Streaming-Aware Indexing Preserves Quality with Half Fixed Budget . . . . . . . . . . . . .

5.3.2

Cross-Layer Sharing Preserves Quality Up to N =2 . . . . . . . . . . . . . . . . . . . . . . .

5.3.3

Cross-Layer Distillation Is Essential for Index Reuse . . . . . . . . . . . . . . . . . . . . . .

5.3.4

CLI Extends to MTP Layers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5.3.5

Hierarchical Indexing Preserves Quality with Proper Configuration . . . . . . . . . . . . . .

5.3.6

LSA Is Robust to Conversion Timing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

LongCat-Flash-Lite-Sparse

6.1

Model Informations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6.2

Model Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Conclusion, Limitations, and Future Directions

8 Acknowledgement

A Training Efficiency of LSA versus Dense MLA

B Serving setup and KV-cache partitioning

28 2

LongCat Sparse Attention

C Case Study: LSA Behavior on Needle-in-a-Haystack

C.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

C.2 Indexer Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

C.3 Attention Weights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

LongCat Sparse Attention

Introduction

Long-context capabilities are increasingly essential for modern large language models (LLMs), underpinning applications such as repository-level agentic coding, long-horizon task execution, and extended reasoning. However, scaling LLMs to long contexts remains fundamentally constrained by the quadratic O(L2 ) complexity of standard self-attention [Vaswani et al., 2017], where L denotes the sequence length. Sparse attention mitigates this bottleneck by restricting each query to a subset of key-value (KV) tokens, but its effectiveness depends critically on selecting the most relevant tokens both accurately and efficiently. DeepSeek Sparse Attention (DSA) [DeepSeek-AI, 2025a] addresses this challenge with a dedicated Lightning Indexer, which scores prefix tokens for each query and performs fine-grained, token-level selection. DSA thereby achieves near-lossless quality relative to full attention and has been adopted by production-scale foundation models, including DeepSeek-V3.2 [DeepSeek-AI, 2025b] and GLM-5 [Zeng et al., 2026]. Despite its advantages over full attention at long sequence lengths, DSA still leaves substantial room for efficiency improvement. Each DSA layer sequentially executes two operators: the Lightning Indexer, which scores prefix tokens and selects a budget of K candidates, and the core sparse attention operator, which attends only to the selected subset. Our profiling reveals two corresponding system-level bottlenecks: Indexer Output Discontiguity and Indexer High Overhead. Indexer Output Discontiguity arises because dynamically selected indices require the core sparse-attention operator to gather KV vectors through scattered, non-coalesced memory accesses. Consequently, despite its modest O(LK) computational complexity, the operator becomes severely memory-bound and achieves only ∼4.5% of peak HBM bandwidth on our AI accelerators. Indexer High Overhead, meanwhile, stems from scoring the entire prefix for every query, which retains a quadratic O(L2 ) complexity. Although DSA reduces the indexer’s constant factors through fewer heads, smaller head dimensions, and FP8 quantization, its quadratic cost increasingly dominates layer latency as the context grows, accounting for up to 90% of the per-layer latency at a 1024K context length. We characterize both bottlenecks in detail in Section 2.3. To address Indexer Output Discontiguity, we introduce Streaming-Aware Indexing (SI). SI allocates part of the sparse attention budget to predictable and contiguous regions, specifically a fixed attention sink and a local sliding window, while using the remaining budget for dynamic token-level selection. This design is motivated by the streaming patterns observed in full attention [Xiao et al., 2024a,b]. By combining dynamically selected tokens with contiguous local regions, SI improves memory coalescing and restores high HBM throughput without compromising long-context accuracy. To reduce Indexer High Overhead, we introduce two complementary mechanisms: Cross-Layer Indexing (CLI) and Hierarchical Indexing (HI). CLI amortizes indexing computation across model depth. Motivated by the observation that salient tokens are often highly consistent across adjacent layers [Yang et al., 2024, Deshmukh et al., 2025, Gao et al., 2026], CLI partitions consecutive layers into groups of size N . The first layer in each group, termed the owner layer, executes the indexer, while the remaining reuse layers share its selected token set. Naively reusing the owner layer’s indices, however, leads to substantial quality degradation. We therefore introduce a cross-layer distillation objective that trains the owner indexer to capture tokens salient to the entire group. This reduces indexing computation to approximately 1/N of that required by standard DSA while preserving model quality. HI reduces the cost of each indexing pass through a coarse-to-fine selection scheme. It first uses block-level representations to retrieve the top-M candidate pages and then performs fine-grained token scoring only within those pages. This reduces the per-query selection complexity from O(L) to O(L/P + M P ), where P denotes the page size. Unlike SI and CLI, which are incorporated during both training and inference, HI is a training-free, plug-and-play module that can be applied directly at inference time. Together, SI, CLI, and HI form LongCat Sparse Attention (LSA), a co-designed sparse-attention framework whose components target complementary bottlenecks and can be composed seamlessly (Fig. 1). We evaluate LSA at two scales within the LongCat family: LongCat-Flash-Lite (69B-A3B) and LongCat-Flash (560B-A27B). Across both scales, LSA consistently matches full attention on general, reasoning, coding, and long-context benchmarks. Moreover, its substantially improved training efficiency over full MLA enables the training of LongCat-2.0 (1.6T-A48B)1 with context lengths of up to one million tokens under a limited compute budget. Building on LSA, we further extend the long-context training of LongCat-Flash-Lite [Liu et al., 2026], scaling its native context length to one million tokens. The resulting model, LongCat-Flash-Lite-Sparse, achieves substantially higher long-context inference efficiency than its dense predecessor while exhibiting stronger agentic capabilities. Our main contributions are summarized as follows: 1

https://huggingface.co/meituan-longcat/LongCat-2.0

LongCat Sparse Attention

• We systematically profile DSA and identify two major efficiency bottlenecks: Indexer Output Discontiguity and Indexer High Overhead (Section 2.3). • We propose LSA, which addresses these bottlenecks through three complementary mechanisms: Streaming-Aware Indexing for hardware-friendly memory access, Cross-Layer Indexing for amortizing indexing computation across layers, and Hierarchical Indexing for reducing the token-level scoring cost (Section 3). • Through extensive ablation studies, we show that the training-aware SI and CLI mechanisms improve both training and inference efficiency while maintaining performance parity with full attention. The training-free HI module provides further inference speedups with only marginal quality degradation (Sections 4 and 5). • We release LongCat-Flash-Lite-Sparse, an open-source model built with LSA, to facilitate research and practical deployment of sparse-attention architectures (Section 6).

Sparse Attention and the DSA Baseline

Before presenting our method, we review the evolution of sparse attention, formalize the DSA mechanism that serves as our baseline, and profile its efficiency bottlenecks. 2.1

From Fixed Patterns to Retrieval-Based Sparse Attention

The central challenge in scaling self-attention [Vaswani et al., 2017] to long contexts is the quadratic cost O(L2 ) in sequence length L. Sparse attention methods address this by restricting each query to attend only a small subset of key-value positions. These methods differ primarily in how they determine the positions attended by each query. Along this design dimension, approaches have evolved from content-blind fixed patterns to increasingly sophisticated learned retrieval mechanisms. Fixed patterns. Early sparse attention methods define sparsity structures a priori, such as sliding windows, strided patterns, or designated global tokens [Child et al., 2019, Beltagy et al., 2020, Zaheer et al., 2020]. These fixed patterns are hardware-friendly due to their regularity but entirely content-blind: the same positions are attended regardless of input. Query-aware but non-learnable retrieval. A natural next step is to make the sparsity pattern content-dependent. Reformer [Kitaev et al., 2020] applies locality-sensitive hashing (LSH) to route queries and keys into shared buckets, restricting attention to within-bucket pairs. RetrievalAttention [Liu et al., 2024a] builds an approximate nearestneighbor (ANN) index over stored keys and retrieves relevant KV entries at inference time. In both cases the selection is query-aware, yet the retrieval criterion cannot be directly optimized through training to improve selection quality. Coarse-grained learned retrieval. The key leap in recent work is making the retrieval mechanism itself trainable. MoBA [Lu et al., 2025] and NSA [Yuan et al., 2025] partition the KV sequence into contiguous blocks and learn which blocks are most relevant to each query. Their indexing mechanisms directly reuse the core attention’s query and key representations (after compression): MoBA computes block affinity via mean-pooled keys, while NSA repurposes its compression branch’s attention scores as block importance signals. However, these block-level methods inherently blur token-level importance distinctions within each block, leading to possibly suboptimal performance relative to full attention on some long-context and reasoning tasks. Fine-grained learned retrieval. DSA [DeepSeek-AI, 2025a] advances to token-level fine-grained learned sparse attention, scoring every individual token in the prefix for each query. Unlike MoBA and NSA which directly reuse the core attention’s representations for scoring, DSA introduces a dedicated Lightning Indexer with its own query and key projections, whose inputs are detached from the model’s computation graph during training. This fine-grained indexing combined with the indexer’s architectural independence enables DSA to achieve near-lossless performance relative to full attention. DSA has since been adopted by multiple production systems including DeepSeek-V3.2 [DeepSeek-AI, 2025b,a] and GLM-5 [Zeng et al., 2026], validating its deployability at scale. 2.2

A Brief Recapitulation of DSA Mechanism

We first review the formal details of DSA to establish notation used throughout this paper. Indexer scoring. Given a query token with hidden state ht , the Lightning Indexer computes a saliency score for every token s ≤ t: HI X I It,s = wt,j · ReLU qIt,j · kIs , (1) j=1

LongCat Sparse Attention

I where H I is the number of indexer heads, qIt,j and wt,j are derived from ht via learned projections, and kIs is derived from hs . The indexer adopts an MQA [Shazeer, 2019] pattern: a single key kIs is shared across all indexer heads. The ReLU activation is chosen for throughput efficiency.

Top-K selection and sparse attention.

For each query position t, the indexer selects the K highest-scoring tokens: St = arg topK({It,s }s≤t , K),

(2)

and attention is computed only over this sparse subset: ut = Attn(ht , {cs | s ∈ St }) ,

(3)

where cs denotes the MLA latent KV representation [Liu et al., 2024b] of token s. DSA is instantiated under MLA’s absorbed MQA [Shazeer, 2019] mode, where each latent vector is shared across all query heads as the key-value entry. With K = 2048 in a 128K-token context, this achieves approximately 98.4% sparsity while maintaining near-lossless quality. Two-stage training.

DSA training proceeds from a pre-trained full-attention model in two stages:

1. Dense warm-up: The base model is frozen, and only the indexer parameters are trained. The objective aligns the indexer output with the aggregated attention distribution via KL divergence: X Ldense = DKL (pt,: ∥ Softmax(It,: )) , (4) I t

where pt,: is the sum of attention weights across all heads, normalized to a distribution. 2. Sparse training: Both the indexer and the base model are trained jointly. The indexer is now supervised only over the selected token set St , with the attention target renormalized over this subset: X (5) Lsparse = DKL pt,St Softmax(It,St ) , I t

where pt,St denotes the head-aggregated attention weights restricted to St and renormalized to a distribution, and It,St the corresponding indexer scores. The indexer parameters are updated by this KL loss, while its input is detached from the model’s computation graph so that the indexer’s gradients do not perturb the base model. The base model parameters are updated solely via the next-token prediction loss. Why “Lightning”. The indexer is designed to be lightweight relative to the core attention. It uses fewer heads (typically half that of the core attention) and a smaller head dimension (roughly one-third, since the indexer’s nope dimension is much smaller than MLA’s absorbed latent dimension). It further supports FP8 computation. These choices collectively minimize the per-token cost of indexing, justifying the “Lightning” designation. Complexity. The overall computational cost of DSA consists of indexer scoring O(L2 ) and sparse attention O(LK). Despite its low per-token cost, the indexer scales quadratically with L. Consequently, for sufficiently long sequences, the indexer becomes the dominant bottleneck and can incur greater computational cost than the core attention operation it is designed to serve. We quantify this crossover through systematic profiling in the next section. 2.3

Profiling the Efficiency Bottlenecks of DSA

The execution cost of a DSA layer is partitioned between two primary operators whose latencies scale differentially with respect to the KV length L. For each incoming query token, the Lightning Indexer (LI) must evaluate all L prefix keys, resulting in a linear complexity (O(L)). Conversely, the Sparse Flash Attention (SFA) restricts its attention to a fixed set of K selected tokens, thereby remaining essentially decoupled from L. This divergence creates a clear regime shift: SFA dominates computational overhead in short contexts, while LI takes over at extended sequence lengths, intersecting at approximately ∼100K tokens. This breakdown during the decoding stage is quantified in Table 1 for KV lengths ranging from 4K to 1024K. These regimes reveal two intrinsic efficiency bottlenecks in DSA’s sparse selection mechanism: Indexer Output Discontiguity and Indexer High Overhead. We examine both bottlenecks in detail below. Bottleneck 1: Non-coalesced memory access arising from Indexer Output Discontiguity. Token-level sparse selection forces each memory transaction to retrieve a single, non-contiguous KV vector, severely degrading High Bandwidth Memory (HBM) utilization. On our AI accelerators, a single core can sustain approximately 50 in-flight 6

LongCat Sparse Attention

Table 1: Per-layer latency breakdown of baseline DSA during inference decode (batch_size = 4, qlen = 1, BF16 precision, K = 2048, causal masking). SFA is constant in L (fixed K), while the indexer grows linearly, so the dominant cost shifts from SFA to LI near ∼100K. KV Length

Indexer (ms)

SFA (ms)

Total (ms)

Indexer %

4K 8K 16K 32K 64K 128K 256K 512K 1024K

0.034 0.036 0.041 0.053 0.078 0.154 0.302 0.523 0.930

0.097 0.096 0.098 0.098 0.097 0.100 0.102 0.102 0.102

0.131 0.132 0.139 0.151 0.175 0.254 0.404 0.625 1.032

26% 27% 29% 35% 45% 61% 75% 84% 90%

cachelines (512 B each) under ideal memory coalescing, maintaining a memory window of ∼25.6 KB. In DSA, however, each selected token is fetched via an independent gather of a single latent KV vector (1,152 B in BF16), which spans only 3 cachelines. Consequently, two inefficiencies compound: first, this gather occupies merely 3 of the ∼50 outstanding memory slots (yielding ≈ 6% memory-level parallelism); second, even within these 3 cachelines, data packing efficiency is only ≈ 75%. The resulting net effective bandwidth thus plummets to a mere ≈ 4.5% (≈ 1/22) of peak performance. While the aforementioned analysis focuses on the forward pass during inference, this output discontiguity inflicts a more severe performance penalty during training. In the backward pass, gradient updates rely on scatter_add operations over the identical non-contiguous token indices. Due to the scattered nature and input-dependent selection of these indices, independent computing cores frequently attempt to write to overlapping KV gradient regions in HBM. When different cores concurrently target identical gradient positions, these probabilistic write conflicts force the HBM transactions to serialize, severely degrading hardware concurrency. The indexer backward pass suffers from this same cross-core serialization as it scatters gradients across all L scored positions. Bottleneck 2: High Indexer Overhead from linear-scaling scoring and Top-K selection. Consider long-context decoding, where a fixed number of query tokens attends to a KV cache whose length L grows continuously. The computational cost of the LI scales linearly with L for each query because both of its internal stages operate over the entire context. Specifically, the scoring stage performs an MQA-style QK matrix multiplication with O(L) complexity, after which Top-K selection incurs an additional O(L) cost over all candidate scores. By contrast, SFA attends to a fixed budget of K selected tokens, making its decoding cost independent of L. This asymmetry explains the latency breakdown reported in Table 1. As the context length increases from 4K to 1024K, SFA latency remains nearly constant at ∼0.10 ms, whereas LI latency increases by 27×, from 0.034 ms to 0.930 ms. The dominant per-layer bottleneck therefore shifts from SFA at short context lengths to LI at long context lengths. At a 1024K context length, the indexer accounts for up to 90% of the total per-layer latency. Moreover, although LI has linear complexity per decoding query, applying it to every query during prefill or training results in an aggregate O(L2 ) indexing cost, fundamentally limiting the scalability of standard DSA.

LongCat Sparse Attention (LSA)

Building upon the DSA baseline formalized in Section 2.2, LSA introduces three orthogonal efficiency improvements to the indexer: Streaming-Aware Indexing (SI; Section 3.1), Cross-Layer Indexing (CLI; Section 3.2), and Hierarchical Indexing (HI; Section 3.3). Each module targets a distinct bottleneck dimension identified in Section 2.3 and can compose naturally with the others. We describe each component in detail in the remainder of this section. 3.1

Streaming-Aware Indexing: Improving Locality for Hardware-Aligned Coalesced Access

Attention sinks and streaming heads. StreamingLLM [Xiao et al., 2024a] reveals that a small number of initial tokens act as attention sinks and absorb a disproportionate amount of attention weight because of softmax normalization constraints. It further demonstrates that retaining only these sink tokens together with a local sliding window is sufficient to maintain stable perplexity during streaming inference. DuoAttention [Xiao et al., 2024b] further deepens this finding by showing that attention heads exhibit clear functional specialization: Streaming Heads primarily attend to sinks and 7

LongCat Sparse Attention

Sink (16 tokens)

SWA (1024 tokens)

Middle (non-sink, non-SWA)

(a) Mass vs. query position (layer-averaged)

(b) Per-layer mass (last 2048 queries)

avg sink+SWA mass = 83.1%

1.0

Cumulative Attention Mass

1.0

0.8

0.6

0.4

0.2

0.0

0.8

0.6

0.4

0.2

0.0 1-2k

2-3k

3-4k

4-5k

5-6k

6-7k

7-8k

Query Position Segment

Layer Index

Figure 2: Attention mass distribution on a full-attention LongCat-Flash-Lite (69B-A3B) model with 14 short-cut MoE blocks, each containing two serial attention operations (28 attention layers total). Data: 20 InfBench-QA samples, seq_len = 8192. (a) Layer-averaged breakdown by query position segment: the streaming ratio converges to ∼83% beyond 5k tokens, confirming the pattern persists at long range. (b) Per-layer breakdown over the last 2048 queries: the streaming regions (Sink + SWA) capture an average of ∼83% of attention mass across all layers. recent tokens (capturing local context and numerical stability), while Retrieval Heads are responsible for long-range information retrieval. Streaming-aware budget partitioning. Departing from head-wise classification, we analyze the aggregate attention mass distribution across all heads. Our empirical analysis reveals a persistent streaming pattern: the sink and sliding window regions consistently capture a substantial share of the attention weight (see Fig. 2). Recognizing this as a stable structural feature, we formalize their allocation as a deterministic budget. This approach allows us to partition the attention operation into a hardware-aligned fixed streaming component (sink and sliding window) and a dynamic sparse component. Formally, we decompose the attended token set into three disjoint subsets: St =

Ssink ∪ Sswa ∪ Ssparse , | {z }

(6)

fixed streaming budgets

where Ksink = |Ssink |, Kswa = |Sswa |, and Ksparse = |Ssparse | denote the sizes of each subset, satisfying K = Ksink + Kswa + Ksparse . Specifically, Ssink = {1, . . . , Ksink } covers the sink region, Sswa = {t − Kswa + 1, . . . , t} is a sliding window around the query, and Ssparse contains the remaining Ksparse tokens dynamically selected by the indexer from the non-fixed positions. The indexer’s scoring range correspondingly shrinks: Ssparse = arg topK({It,s }s∈S / sink ∪Sswa , Ksparse ),

(7)

Training. Under the standard two-stage training (Section 2.2), the dense warm-up stage requires no modification: the indexer is supervised against the full-sequence attention distribution (Eq. (4)). In the sparse training stage, the distillation target becomes the attention distribution renormalized over the selected subset St . Although at inference time the indexer only scores tokens in the middle region (s ∈ / Ssink ∪ Sswa ), during training we deliberately distill the indexer over the entire selected set, including the sink and SWA portions. Since these streaming regions capture a substantial fraction of the attention mass (Fig. 2), including them in the distillation target provides richer supervisory signal and enables the indexer to more thoroughly learn the overall attention structure, improving its ability to identify the most relevant tokens in the remaining middle region. Design choices and benefits. We set sink size Ksink = 16 and sliding window size Kswa = 1024, yielding a fixedto-sparse budget ratio of approximately 1:1 (determined by ablation, see Section 5.3), which makes roughly 50% of the selected tokens reside in contiguous memory regions. This partitioning yields three benefits: (a) the sink and sliding-window portions are accessed as contiguous blocks, enabling efficient coalesced HBM reads; (b) the indexer’s effective scoring range shrinks from L to L − Ksink − Kswa , modestly reducing computation; and (c) the deterministic structure provides a natural interface for KV cache offloading and speculative decoding, where consecutive decode steps share predictable cache regions. 3.2 Cross-Layer Indexing: Amortizing Indexing Overhead via Inter-layer Pattern Redundancy Stability of salient tokens across layers. Recent studies [Yang et al., 2024, Deshmukh et al., 2025, Gao et al., 2026] have shown that the set of salient tokens, defined as those receiving the highest attention mass, remains remarkably 8

LongCat Sparse Attention

(a) Top-K Overlap

(b) Attention Mass Captured by Reuse

1.0

0.8

0.9

0.8

0.6 12

0.4

0.8

0.7

Value

Source Layer

0.6

0.4

0.2 24

0.6

0.0 0

0.0 Target Layer (j)

Coverage(i→j) = ∑

s∈

(i)

αt,(j)s

Layer Distance

Target Layer

Overlap(i, j) = | ∩  | / K (i)

Top-K index overlap Attn mass coverage

0.2

(α: head-averaged attention weight)

Figure 3: Cross-layer Top-K index analysis on full-attention LongCat-Flash-Lite (69B-A3B). Each layer independently selects its Top-K tokens from the full attention distribution. Only the upper triangle is shown in (a) and (b), corresponding to the CLI reuse direction, where an earlier source layer provides the index set for later target layers). (a) Pairwise overlap of Top-K token sets. (b) Cumulative attention mass captured when reusing one layer’s Top-K on another. (c) Mean overlap and coverage as a function of layer distance, with shaded regions indicating min–max range across all layer pairs at each distance; adjacent layers share ∼57% of their Top-K budget while retaining ∼93% of attention mass. Statistics are collected from 20 InfBench-QA samples with sequence length 8192, computed over the last 2048 query positions of each sample.. stable across consecutive Transformer layers. This observation has motivated a family of sparse attention methods that exploit cross-layer stability to avoid repeatedly selecting salient tokens in each layer. These methods typically designate a few full-attention layers as oracles to identify important tokens and let remaining sparse layers reuse their selections. A natural question arises: does DSA’s Lightning Indexer at a given layer have the capacity to identify the shared salient tokens across itself and subsequent layers? We verify this on LongCat-Flash-Lite by letting each layer independently run its own indexer and measuring cross-layer agreement (Fig. 3). Adjacent layers share 57.4% of their Top-K budget on average. Despite this partial overlap, reusing an adjacent layer’s index set still captures 93.2% of the target layer’s attention mass. This cross-layer consistency opens up the possibility of reusing indexer outputs across layers. Enabling cross-layer indexer reuse with cross-layer distillation. We partition consecutive layers into CLI groups of size N . Only the first layer in each group executes the indexer; the subsequent N − 1 layers directly reuse the index set St produced by the first layer, reducing the total number of indexing passes from Llayers to Llayers /N . Naively reusing indices without training adaptation degrades performance (Section 5.3), because each indexer is originally trained to predict saliency only for its own layer. To enable faithful cross-layer reuse, we make a minimal modification to the standard DSA distillation loss (Eq. (4)): instead of supervising each indexer against only its own layer’s attention, the indexer at the first layer of each group is trained to predict the attention patterns of all layers within the group: LCLI =

N −1 X

(l+i)

(8)

i=0 (l+i)

where l is the first layer in the group, N is the group size, and LI is the indexer distillation loss that supervises the shared indexer output against the attention distribution of layer l + i. Note that this cross-layer distillation applies to both the dense warm-up and sparse training stages. At inference time, the shared indexer runs once and broadcasts the resulting index set to all layers in the group. Integration with Multi-Token Prediction. Multi-Token Prediction (MTP) [Liu et al., 2024c, Zeng et al., 2026] appends D sequential prediction steps (each a Transformer layer with its own attention) after the main model to predict D additional future tokens, where each step combines the previous step’s representation with the next token’s embedding. We extend CLI to the MTP dimension: all D MTP steps form their own independent CLI group (separate from the main model’s groups) and share a single index set produced by the first MTP step’s indexer, trained via PD (MTPk ) cross-layer distillation: LMTP . Unlike CLI in the main model, where adjacent layers process the CLI = k=1 LI same input and naturally exhibit correlated saliency patterns, MTP steps operate sequentially. Each step attends to representations that incorporate different future token embeddings, which may make index reuse across MTP steps appear less justified. However, the cross-layer distillation loss ensures that the shared indexer produces a selection jointly optimized for all MTP steps. At inference time, the first MTP step’s indexer has already been trained to account 9

LongCat Sparse Attention

for the attention patterns of subsequent steps, making CLI reuse across MTP steps equally well-founded. Ablation experiments (Section 5.3) confirm that this approach maintains quality parity with independent per-step indexing. Design choices. Although our analysis reveals that certain layers can sustain high attention mass coverage even with large reuse depths (Fig. 3c), practical LLM architecture design favors simplicity and uniform structure for robust scaling across model sizes. We therefore adopt a uniform interleave pattern with a fixed group size N across all layers. We further constrain N to be even for two reasons: (1) the LongCat-Flash-Lite series adopts a shortcut-connected architecture [Cai et al., 2024, Team et al., 2025] where each shortcut layer contains two serial attention operations, and (2) even group sizes ensure uniform partitioning across pipeline-parallel stages during training. From Fig. 3(c), we observe that the minimum coverage across layer pairs begins to drop noticeably when the reuse distance exceeds 4. Ablation experiments (Section 5.3) show that N = 4 incurs measurable accuracy loss on long-context validation, so we set N = 2, which halves the indexing compute with no measurable quality loss. For MTP steps, since they serve as draft proposers for speculative decoding whose outputs are verified by the main model, their accuracy does not affect final generation quality. We therefore let all 3 MTP steps share a single index (i.e., N = 3) to maximize the efficiency gain. Relation to concurrent work. After our CLI strategy was validated and deployed in production, we became aware of IndexCache [Bai et al., 2026], an independent and concurrent effort that arrives at a highly similar training-aware cross-layer distillation mechanism for index reuse. Despite the shared core idea, several differences are worth noting. (1) Architecture: we validate on LongCat-Flash-Lite’s shortcut-connected structure (two serial attention operations per shortcut layer), whereas IndexCache targets a standard Transformer layout. (2) Effective reuse ratio: IndexCache reports that 1/4 indexer retention (i.e., N =4) remains within 0.4% of baseline on their evaluation benchmarks, whereas our ablations (Section 5.3) show that N =4 incurs measurable accuracy loss on long-context validation, leading us to adopt the more conservative N =2. This discrepancy might stem from the differences in model architecture and context length noted above. (3) Composability: beyond validating CLI in isolation, we demonstrate that it composes effectively with the other two LSA components (Streaming-aware and Hierarchical Indexing), achieving greater aggregate speedup than any single strategy alone. (4) Multi-Token Prediction: we further verify that CLI remains effective across MTP steps, extending index reuse to the speculative decoding path. 3.3

Hierarchical Indexing: Coarse-to-Fine Sparse Selection

The indexer scores every token in the non-fixed region for each query, yielding O(L) per-query cost that dominates latency at long contexts. Can we eliminate most irrelevant tokens cheaply before invoking fine-grained scoring? Our solution is a two-stage coarse-to-fine scoring scheme: a cheap coarse stage first recalls a small subset of candidate sub-sequences, shrinking the candidate set on which the expensive fine-grained scoring is then performed. Stage 1: Block-level coarse filtering. We partition the sequence into contiguous pages of size P , and first perform a coarse selection to obtain Top-M pages. For each page, we split it into sub-blocks of B tokens each, and precompute the per-dimension mean of each sub-block’s constituent keys: kmean = means∈sub-blockn kIs . The coarse saliency score n for every page p ≤ ⌈t/P ⌉ is computed by aggregating its sub-block scores in a manner similar to Eq. (1): I

page It,p =

H X j=1

I wt,j ·

ReLU qIt,j · kmean , n

(9)

n∈pagep

I where H I is the number of indexer heads and qIt,j , wt,j are the same per-head query and weight as in Eq. (1). We select the Top-M pages as candidates according to their saliency scores, and let Stpage denote the set of tokens they cover, reducing the scoring space from O(L) tokens to M · P : [ [ page Stpage = sub-blockn , where Pt = arg topK {It,p }p≤⌈t/P ⌉ , M . (10) p∈Pt n∈pagep

Stage 2: Token-level refinement. For the M selected candidate pages (comprising M · P tokens), we apply the standard indexer scoring of Eq. (1) to obtain per-token scores. Specifically, using the same indexer query qIt,j , the fine-grained indexer score for every token s within the recalled pages is computed as I

It,s =

H X

I wt,j · ReLU qIt,j · kIs ,

j=1

s ∈ Stpage .

(11)

LongCat Sparse Attention

We select the final Top-Ksparse tokens, yielding the same sparse set Ssparse of Eq. (7) but scored only over the recalled pages: Ssparse = arg topK {It,s }s∈Stpage ∩[1,t] , Ksparse . (12) As analyzed in Section 4.1.2, the Top-K selection dominates the indexer cost while the scoring itself is comparatively cheap, so we account only for the selection cost. Under this measure, the two Top-K operations in Eqs. (10) and (12) cost O(L/P ) over pages and O(M · P ) over the recalled tokens, respectively, reducing the original O(L) per-query selection to O(L/P + M · P ). Design choices. Hierarchical Indexing is training-free: it requires no additional parameters or fine-tuning, operating purely as an inference-time optimization over the existing indexer. The mean value of the blocks is precomputed once per sequence and stored in a cache. We fix the page size P to 128, while treating the sub-block size B and candidate count M as tunable hyperparameters. The ablation study in Section 5.3.5 identifies B=8 and M =1024 as the optimal configuration. Profiling under this setting (Table 4) reveals that the two-stage design introduces non-negligible overhead from block-mean maintenance, coarse-grained scoring, and candidate gathering. As a result, Hierarchical Indexing provides net efficiency gains only for sequence lengths of ≥ 256K, and we enable it adaptively beyond this crossover point.

Kernel Design and Efficiency Analysis

4.1

Efficient Kernel Design and Implementation

Among the three components of LSA, CLI reuses the indexer across layers and therefore requires no kernel-level modification. This section thus focuses on the kernel design of SI and HI, which reshape the DSA layer at the operator level to address the bottlenecks identified in Section 2.3. 4.1.1

Hybrid Sparse Attention Operator for SI: Partitioning and Overlapped Execution

Recall that SI partitions the attention budget into two components of approximately equal size—a sliding window Sswa and a dynamic sparse selection Ssparse (plus a small sink Ssink of only 16 tokens, which we omit in the analysis below for brevity). To translate this algorithmic partitioning into tangible hardware efficiency gains, we design an integrated operator, Hybrid Sparse Attention (HFA). In the forward pass, instead of processing the entire attention budget with a single SFA operator, HFA decomposes core attention into an SFA operator over Ssparse and an SWA operator over Sswa . The two operators are dispatched to separate non-blocking hardware streams for overlapped execution, and their partial outputs are merged through online-softmax rescaling. In the backward pass, beyond confronting non-coalesced memory access issues similar to those in the forward pass, HFA encounters a unique and more severe bottleneck during gradient accumulation. Specifically, gradient updates in the SFA backward pass are performed by allocating a gradient buffer for the entire sequence length in High Bandwidth Memory (HBM) and executing a scatter_add operation over the selected sparse index set. This process constitutes the dominant execution cost of the backward pass. Beyond the low memory bandwidth utilization caused by discontinuous reads, it introduces a severe write-conflict bottleneck: when different processing cores select overlapping indices, they must write to the same physical memory addresses in HBM, resulting in serialized writes to shared cachelines that heavily restrict throughput. By assigning roughly half of the fixed attention budget to the sliding window Sswa , HFA correspondingly reduces the active sparse index set, thereby decreasing the number of discrete gather/scatter operations and lowering the probability of write conflicts during gradient accumulation. Tables 2 and 3 summarize the speedups achieved by our optimized HFA over the SFA baseline. Under the training configuration (Table 2), HFA accelerates core attention by up to 1.91× in the forward pass and 1.73× in the backward pass. For inference (Table 3), we evaluate improvements at both the core-attention and full-layer levels. At the core-attention level, HFA achieves consistent speedups of 1.56–1.69× during prefill and 1.11–1.26× during decode. After including indexer overhead, however, the full-layer speedup becomes increasingly constrained by indexing cost, reaching a peak of 1.49× for prefill at 4K context and 1.14× for decode at 8K context, before gradually diminishing to 1.02–1.04× at longer contexts. This result motivates the complementary roles of our designs: SI targets short-context efficiency by alleviating core-attention bottlenecks, while CLI and HI address long-context scenarios where indexing overhead dominates. 11

LongCat Sparse Attention

KV Length

16K

32K

64K

128K

256K

512K

1024K

fwd

SFA HFA speedup

23.59 18.46 1.28×

23.65 19.31 1.22×

23.64 19.70 1.20×

23.71 19.15 1.24×

23.98 19.53 1.23×

24.25 19.42 1.25×

24.46 19.84 1.23×

36.80 19.27 1.91×

bwd

Table 2: Core attention latency (ms) under a representative training setting. Measurements use batch_size = 1, qlen = 8192, BF16, and K = 2048. For SI, we use Ksparse = 1024 and Kswa = 1024, omitting Ksink = 16 for simplicity. SFA uses the full budget for dynamic sparse selection, while HFA splits it into parallel sparse (SFA) and contiguous (SWA) branches, with outputs merged via online-softmax rescaling.

SFA HFA speedup

165.99 113.86 1.46×

215.42 135.16 1.59×

238.60 137.77 1.73×

267.57 155.02 1.73×

277.92 165.74 1.68×

296.01 175.04 1.69×

319.91 189.85 1.69×

354.48 220.12 1.61×

Table 3: Core attention and full-layer latency (ms) under representative inference settings. Results are measured with BF16 precision and K = 2048 for prefill (batch_size = 1, qlen = 2048) and decode (batch_size = 4, qlen = 1). We compare core attention latency (SFA vs. HFA) and full-layer latency including indexer overhead: SFA + LI (Lightning Indexer) versus HFA + SI (Streaming-Aware Indexing).

Decode

Prefill

KV Length

4.1.2

16K

32K

64K

128K

256K

512K

1024K

SFA HFA speedup

5.946 3.775 1.57×

5.961 3.797 1.57×

5.936 3.809 1.56×

5.964 3.821 1.56×

6.004 3.826 1.57×

6.017 3.863 1.56×

6.099 3.857 1.58×

6.672 4.170 1.60×

9.737 5.756 1.69×

SFA + LI HFA + SI speedup

6.644 4.471 1.49×

7.401 5.242 1.41×

8.877 6.763 1.31×

11.902 9.787 1.22×

17.955 15.825 1.13×

30.006 27.907 1.08×

54.137 51.829 1.04×

102.848 100.479 1.02×

202.180 198.407 1.02×

SFA HFA speedup

0.097 0.085 1.14×

0.096 0.083 1.16×

0.098 0.085 1.15×

0.097 0.087 1.11×

0.100 0.081 1.23×

0.102 0.083 1.24×

0.102 0.084 1.21×

0.102 0.081 1.26×

SFA + LI HFA + SI speedup

0.131 0.116 1.13×

0.132 0.116 1.14×

0.139 0.124 1.12×

0.151 0.135 1.12×

0.175 0.164 1.07×

0.254 0.224 1.13×

0.404 0.374 1.08×

0.625 0.588 1.06×

1.032 0.995 1.04×

Index-Selection Operator for HI: Two-Stage Coarse-to-Fine Selection

Within the indexing computation, the main bottleneck is not score computation (Eq. (1)) but Top-K selection (Eq. (2)). Score computation benefits from high-throughput matrix processing units, whereas Top-K requires sorting the full candidate set on much slower vector processing units. To overcome the latency bottleneck caused by global Top-K selection in long-context regimes, we implement a custom operator for the Hierarchical Indexer (HI) formulated in Section 3.3. We then profile the execution latency of this proposed two-stage indexer (Stage 1 coarse page recall followed by Stage 2 fine-grained token selection) against the flat Lightning Indexer (LI) baseline in Table 4. Our analysis reveals two distinct performance regimes. Below the recall budget (L≤128K), Stage 2 still processes nearly the entire sequence and its latency scales with L. Consequently, the additional overhead of the two-stage design outweighs its benefits, resulting in a net slowdown of 0.79–0.82×. We therefore enable HI only beyond the crossover point (around 200K tokens in this setting), while falling back to the flat LI for shorter contexts. Beyond the recall budget, Stage 2 saturates at a constant latency of 27.8ms, whereas the flat baseline continues to grow linearly with

Table 4: Indexer latency (ms) under representative prefill settings (qlen = 2048, BF16). We compare the proposed hierarchical two-stage indexer with the flat baseline. The hierarchical design uses B = 8, P = 128, and M = 1024 pages in Stage 1 (128k-token candidate budget), followed by Stage 2 refinement to select the final Ksparse tokens. Speedup is computed as Flat / Total. KV Length

32K

64K

128K

256K

512K

1024K

HI Stage 1 HI Stage 2 HI Total

0.666 6.814 7.480

1.264 13.353 14.617

2.457 27.769 30.226

4.912 27.769 32.681

9.758 27.769 37.527

19.162 27.769 46.931

Flat LI speedup

5.934 0.79×

11.950 0.82×

23.977 0.79×

48.025 1.47×

96.139 2.56×

192.698 4.11×

LongCat Sparse Attention Forward

Backward

1.92x

DSA LSA 1.92x Speedup Ratio

400 300 200

1.45x

800

1.55x 1.55x

200

1.81x

100

1.71x

100 1.42x

300

32K

1.58x

64K

128K 256K Sequence Length

1024K

1.61x

1000

1.41x

400

Latency (ms)

500

Total (Fwd + Bwd)

1.34x

500

Latency (ms)

600

1.51x

600 1.50x

400

1.53x

1.55x

200

32K

64K

128K 256K Sequence Length

1024K

32K

64K

128K 256K Sequence Length

1024K

Figure 4: Single-attention-layer training latency of LSA and DSA across context lengths. Bars report forward, backward, and total latency, including kernel execution and CP communication; arrows indicate LSA speedup over DSA. The CP degrees are 4, 8, 16, 32, and 128 for context lengths of 32K, 64K, 128K, 256K, and 1024K, respectively.

context length. As a result, the advantage of HI increases substantially with longer contexts, achieving 4.11× speedups at 1024K tokens. 4.2

Attention Layer Training Speedup

We compare the training efficiency of LSA and DSA by measuring the forward and backward latency of a single attention layer across a spectrum of context lengths, reflecting practical production training setups. Speedup analysis. As shown in Fig. 4, LSA consistently reduces total attention-layer latency, achieving a 1.53× speedup at 32K and a 1.61× speedup at 1024K. Since HI is used only for inference, the training gains come from SI and CLI: (1) CLI reduces forward-only indexer overhead. By training a shared indexer across N = 2 layers through cross-layer distillation, CLI amortizes one indexer forward over two attention layers. Since this sharing does not reduce backward computation, CLI exclusively benefits the forward pass, with larger gains emerging at long contexts where indexer cost dominates. (2) SI accelerates both passes, especially backward. As shown in Section 4.1, SI improves both forward and backward efficiency by resolving the SFA backward write conflict. Since the backward pass is intrinsically more expensive than the forward pass, eliminating this bottleneck translates into a larger absolute saving and a stronger contribution to overall training throughput. Together, they deliver 1.42–1.92× forward speedup and 1.34–1.55× backward speedup. 4.3

End-to-End Inference Speedup

We compare the end-to-end inference efficiency of LSA and DSA at the LongCat-Flash-Lite (69B-A3B) model scale under representative prefill and decoding workloads. End-to-end performance. To support long-context inference, we use KV-cache Partition (KVP) to shard the KV cache across multiple ranks, alleviating per-device memory pressure. KVP is enabled for decoding at sequence lengths of at least 256K; detailed serving configurations and implementation are provided in Appendix B. We enable SI and CLI for all sequence lengths. HI is enabled only for prefill at sequence lengths of at least 256K, following the crossover point in Table 4. We disable HI during decoding because its two-stage overhead outweighs its benefit at short context lengths, while KVP reduces the per-rank KV length in the long-context regime where HI would otherwise be beneficial. As shown in Fig. 5, LSA achieves a 1.42–3.60× prefill speedup and a 1.25–1.40× decode speedup over DSA. The prefill speedup increases with context length because the indexer accounts for a growing fraction of latency. Decode speedup peaks at 128K and decreases slightly at 256K and beyond, where the serving configuration switches to KVP. By sharding the KV cache, KVP reduces the indexer’s per-rank workload and thus narrows LSA’s relative advantage over DSA. Speedup analysis. SFA scales as O(LK) in prefill and O(K) in decoding, whereas the indexer scales as O(L2 ) and O(L), respectively. The indexer therefore accounts for an increasing fraction of latency as the context length grows. At short context lengths, SFA dominates and most of the gain comes from SI. At long context lengths, the indexer dominates. CLI reduces its cost by reusing indices across layers, while HI mitigates its super-linear prefill scaling beyond 256K. 13

LongCat Sparse Attention Prefill

Decode

3.60x

DSA LSA 3.60x Speedup Ratio 5 10

1.65x

TPOT (ms)

TTFT (ms)

1.90x

104 1.56x 1.50x

103

1.43x 1.63x

16K

15 10

1.25x

1.26x

1.29x

16K

1.36x

1.37x

1.28x

1.40x

1.42x

1.70x

1.32x 1.26x

K 32K 64K 128 Sequence Length

256

512

102

K 32K 64K 128 Sequence Length

256

512

102

Figure 5: End-to-end inference latency of LSA versus the DSA baseline: prefill time-to-first-token (TTFT, log scale) and decode time-per-output-token (TPOT) across context lengths. The arrow annotates the LSA speedup over DSA. Table 5: Mean acceptance length of a 3-step MTP module under LSA (DSA with SI and CLI) versus dense MLA, across four representative tasks. Higher is better; the theoretical maximum for 3-step MTP is 4. 3-step MTP with LSA 3-step MTP with MLA

HumanEval

GSM8K

AIME

MRCR

Avg

2.96 2.97

3.17 3.20

2.88 2.83

3.46 3.59

3.11 3.15

Compatibility with KV-cache offloading. KV-cache offloading complements KVP by storing the cache in host memory and transferring only the chunks accessed by sparse attention. Its efficiency depends on temporal locality across decoding steps. SI improves this locality by reserving half of the attention budget for contiguous sink and sliding-window regions, increasing the average inter-step chunk overlap from 65.05% to 82.04% and reducing per-layer reload latency from 53.88 µs to 30.46 µs. CLI further enables asynchronous prefetching for index-reuse layers, reducing the visible latency to 15.23 µs, or 28% of the DSA baseline. Integration with multi-step MTP. CLI also allows the three MTP steps used for speculative decoding to reuse the indices produced by a single indexer invocation (Section 3.2). Since draft tokens are verified by the main model, changes in draft quality affect acceptance length rather than final generation quality. As shown in Table 5, LSA achieves an average acceptance length of 3.11, compared with 3.15 for dense MLA. The difference indicates negligible impact on speculative-decoding efficiency.

Experiments and Results

5.1

Setup

Model Configuration. We validate the effectiveness of LongCat Sparse Attention (LSA) on two model scales from the LongCat family: LongCat-Flash-Lite (69B-A3B) and LongCat-Flash (560B-A27B). Both adopt the shortcut-connected MoE architecture [Cai et al., 2024] with Multi-head Latent Attention (MLA) [Liu et al., 2024b]. LongCat-Flash-Lite contains 14 shortcut layers (each with two serial attention operations, yielding 28 attention layers total) with 32 attention heads; LongCat-Flash consists of 28 shortcut layers (56 attention layers total) with 64 attention heads. We evaluate three attention configurations: (1) MLA: the standard Multi-head Latent Attention baseline with no sparsity; (2) DSA: the standard DeepSeek Sparse Attention with K = 2048; and (3) LSA: our proposed method with total K = 2048, of which sink Ksink = 16 and window Kswa = 1024 are fixed, combined with hierarchical indexing and cross-layer indexing (N = 2). For LongCat-Flash, we only compare LSA with MLA due to its larger scale and training resource constraints. Detailed model configurations are listed in Table 6. Training. Starting from mid-training checkpoints, we extend the context window through a two-stage long-context training process. LongCat-Flash-Lite is trained for 100B tokens at 128K followed by 100B tokens at 512K, while LongCat-Flash is trained for 100B tokens at 128K followed by 20B tokens at 256K. We convert MLA checkpoints to DSA and LSA during the final one-third of long-context training (512K for LongCat-Flash-Lite and 128K for LongCat-Flash), a transition that begins with a 1,000-step warm-up (7.5B tokens) before entering full sparse training. Ablations (Section 5.3.6) show that the conversion point has no measurable impact on final performance, and the chosen 14

LongCat Sparse Attention

Table 6: Model configurations for the two model scales used in experiments. LongCat-Flash-Lite

LongCat-Flash

# Parameters (total / active)

69B / 3B

560B / 27B

Layers (shortcut / attention) Attention heads (core / indexer) LoRA rank (query / kv) QK head dim (rope / nope) Indexer head dim (rope / nope)

14 / 28 32 / 16 1536 / 512 64 / 128 64 / 64

28 / 56 64 / 32 1536 / 512 64 / 128 64 / 64

512K

256K

Max Sequence Length

points are primarily for training efficiency. After long-context training, both models undergo supervised fine-tuning with the same settings as their corresponding technical reports [Liu et al., 2026, Team et al., 2025]. Evaluation Benchmarks.

We evaluate across two capability groups:

• Long-context: for chat and thinking models we adopt HELMET [Yen et al., 2024], a comprehensive long-context benchmark covering Recall (RULER [Hsieh et al., 2024]), RAG (HotpotQA [Yang et al., 2018] + PopQA [Mallen et al., 2023]), Re-rank (MS MARCO [Bajaj et al., 2016]), LongQA (NarrativeQA [Kočiský et al., 2018] + ∞Bench [Zhang et al., 2024]), Citation (ALCE [Gao et al., 2023], with ASQA [Stelmakh et al., 2022] and QAMPARI [Rubin et al., 2022]), and Summarization (Multi-LexSum [Shen et al., 2022] + ∞Bench [Zhang et al., 2024]). For base models, which we evaluate throughout our ablation studies (Section 5.3), we use the line retrieval task of LongEval [Li et al., 2023a], an n-shot needle-in-a-haystack task that we find effective for distinguishing base model quality. • Overall: general capability is assessed with MMLU [Hendrycks et al., 2021a], MMLU-Pro [Wang et al., 2024], and CMMLU [Li et al., 2023b], C-Eval [Huang et al., 2024]; reasoning with GPQA-Diamond [Rein et al., 2024], MATH500 [Hendrycks et al., 2021b], and AIME 2024/2025 [MAA, 2024, 2025]; coding with HumanEval+ [Liu et al., 2023], MBPP+ [Liu et al., 2023] and LiveCodeBench [Jain et al., 2024]. 5.2

Results of LSA

Long-context evaluation. Table 7 reports HELMET results. On LongCat-Flash-Lite scale, LSA achieves an average score of 59.02, compared to 58.50 (MLA) and 58.60 (DSA), indicating that LSA retains the long-context capability of full attention despite its sparse indexing. On LongCat-Flash scale, LSA achieves a larger gain over MLA (64.43 vs. 62.70), driven primarily by the Re-rank category (+9.3). We attribute this gap to the evaluation setup: LongCat-Flash is a thinking model that tends to produce long reasoning traces. Our analysis reveals that LSA generates slightly shorter outputs than MLA, resulting in a smaller fraction of responses being truncated by the maximum generation length. On the Re-rank subset specifically, MLA produces notably longer generations, leading to more frequent truncation and lower scores. On the remaining categories the two architectures perform comparably. Overall capability evaluation. Table 8 compares general knowledge, reasoning, and coding benchmarks. Across both model scales, LSA, DSA, and MLA achieve comparable scores on all evaluated tasks, with no consistent winner among the three. This further confirms that LSA preserves the capability of full attention across different model scales and diverse task types. Table 7: HELMET evaluation of LSA vs. MLA on LongCat-Flash-Lite (69B-A3B) and LongCat-Flash (560B-A27B). Each category score is the average across its constituent benchmarks. Model

Attn

Recall

RAG

Re-rank

LongQA

Cite

Summ

Avg

LongCat-Flash-Lite (69B-A3B, Chat)

MLA DSA LSA

98.83 99.13 98.63

64.61 65.10 64.38

71.34 70.73 72.64

44.03 42.98 44.46

35.83 36.91 37.53

36.38 36.78 36.48

58.50 58.60 59.02

LongCat-Flash (560B-A27B, Thinking)

MLA LSA

97.30 97.38

85.40 84.60

62.09 71.36

38.89 38.97

43.98 45.20

48.53 49.10

62.70 64.43

LongCat Sparse Attention

Table 8: Standard benchmark evaluation of LSA vs. MLA and DSA on LongCat-Flash-Lite (69B-A3B) and LongCatFlash (560B-A27B).

5.3

LongCat-Flash-Lite (69B-A3B)

LongCat-Flash (560B-A27B)

Category

Benchmark

MLA

DSA

LSA

MLA

LSA

General

MMLU MMLU-Pro CMMLU C-Eval

85.54 78.39 82.46 86.33

85.27 79.68 81.82 86.31

85.50 78.33 82.61 85.81

89.87 82.65 87.02 88.38

90.05 82.39 87.23 89.50

Reasoning

GPQA-Diamond MATH500 AIME 2024 AIME 2025

68.72 98.20 72.60 59.90

68.66 96.60 71.87 63.65

69.51 97.20 73.44 64.27

83.49 98.00 90.83 89.89

83.46 98.40 91.04 90.00

Coding

HumanEval+ MBPP+ LiveCodeBench

86.59 78.31 41.19

83.54 79.10 42.07

86.59 79.37 41.85

70.73 72.75 80.40

71.95 71.69 79.74

Ablation Studies

All ablations are performed at the LongCat-Flash-Lite (69B-A3B) scale. LSA variants are compared with the MLA baseline through training/validation loss and Needle-in-a-Haystack evaluation, followed by HELMET evaluation after supervised fine-tuning. Streaming-Aware Indexing Preserves Quality with Half Fixed Budget (a) Training Loss Gap vs. MLA

(b) Validation Loss (128K Context) LI (0% fixed) SI (25% fixed) SI (50% fixed) SI (75% fixed) SI (100% fixed)

0.15

0.10

0.05

MLA (full baseline) LI (0% fixed) SI (25% fixed) SI (50% fixed) SI (75% fixed) SI (100% fixed)

4.4 4.2 Validation Loss

Δ LM Loss (variant − MLA)

0.20

4.0

100

95 Accuracy (%)

5.3.1

3.8 3.6 3.4 3.2

0.00

85 MLA (full baseline) LI (0% fixed) SI (25% fixed) SI (50% fixed) SI (75% fixed)

3.0 0.0

0.2

0.4 0.6 Training Progress

0.8

1.0

0.2

0.4

0.6 Training Progress

0.8

1.0

16k

32k Context Length

64k

128k

Figure 6: Streaming-Aware Indexing ablation. “X% fixed” = fraction of budget K allocated to the fixed window and sink tokens. (a) Training loss gap relative to MLA. (b) Validation loss on long-context set. (c) Needle-in-a-haystack retrieval accuracy.

Table 9: HELMET evaluation for Streaming-Aware Indexing ablation (chat models). MLA LI (0% fixed) SI (50% fixed)

Recall

RAG

Re-rank

LongQA

Cite

Summ

Avg

95.68 96.28 94.20

64.23 66.28 65.68

67.33 65.61 68.08

39.63 39.98 39.33

35.13 34.66 37.35

33.30 33.78 34.91

55.88 56.10 56.59

A larger fixed window improves hardware efficiency through contiguous memory access, but reduces the tokens freely chosen by the indexer, risking degraded long-context modeling. We ablate this trade-off by varying the fraction of the total token budget K allocated to the fixed window (denoted “X% fixed”, including 16 sink tokens). Since long-context training is computationally expensive, we train MLA and DSA variants with different fixed ratios (0%, 25%, 50%, 75%, 100%) at 128K sequence length and compare their training dynamics and downstream quality. For brevity, we denote DSA with the standard Lightning Indexer as LI, and DSA with the Streaming-Aware Indexer as SI. Fig. 6 summarizes the results. Panel (a) plots each variant’s training loss gap relative to MLA: configurations at 75% fixed or below show negligible difference throughout training, whereas 100% fixed (i.e., pure window attention) exhibits 16

LongCat Sparse Attention

a significantly higher loss. Panel (b) shows validation loss on a long-context set, where 100% fixed again shows significantly higher loss and was therefore abandoned. Panel (c) reports needle-in-a-haystack evaluation (LongEval line retrieval task, which we find discriminative for base models) on final checkpoints: at 128K context, 75% fixed suffers a clear accuracy drop while 0%–50% fixed all match or slightly exceed MLA. We therefore discard the 75% setting and apply supervised fine-tuning to the remaining groups. Table 9 evaluates the resulting chat models on HELMET: SI (50% fixed) achieves comparable performance to both LI (0% fixed) and MLA. We select 50% fixed (a ∼1:1 fixed-to-dynamic ratio) as the default, as it maximizes the fixed window size without degrading long-context quality. 5.3.2

Cross-Layer Sharing Preserves Quality Up to N =2 (a) Training Loss Gap vs. MLA

(b) Validation Loss (128K Context) 3.20

LI (N=1) CLI (N=2) CLI (N=4) CLI (N=4, topk=4k)

0.04

MLA (full baseline) LI (N=1) CLI (N=2) CLI (N=4) CLI (N=4, topk=4k)

3.15

95 90

0.03

0.02

3.10

Accuracy (%)

Validation Loss

Δ LM Loss (variant − MLA)

0.05

3.05 3.00

max gap ≈ 0.002

0.01

85 80 75 70

MLA LI (N=1) CLI (N=2) CLI (N=4) CLI (N=2) w/o cross-layer distill

2.95 65

0.00

2.90 0.0

0.2

0.4 0.6 Training Progress

0.8

1.0

0.2

0.4

0.6 Training Progress

0.8

1.0

16k

32k Context Length

64k

128k

Figure 7: Cross-Layer Indexing ablation (base models). (a) Training loss gap relative to MLA. (b) Long-context validation loss. (c) Needle-in-a-haystack retrieval accuracy across context lengths.

Table 10: HELMET chat evaluation for Cross-Layer Indexing ablation. N = 2 performs on par with MLA and the LI baseline (N = 1), confirming that halving the indexer passes preserves quality. MLA LI (N =1) CLI (N =2)

Recall

RAG

Re-rank

LongQA

Cite

Summ

Avg

95.68 96.28 93.05

64.23 66.28 65.80

67.33 65.61 68.13

39.63 39.98 40.45

35.13 34.66 33.30

33.30 33.78 33.92

55.88 56.10 55.78

We study the impact of CLI group size N under uniform layer partitioning. Increasing N reduces indexer computation by allowing more layers to share one indexer, but also makes the shared indexer responsible for a broader range of layer-wise saliency patterns. Following Section 5.3.1, we train all variants at 128K sequence length and compare MLA against DSA with CLI group sizes N ∈ {1, 2, 4}, where N =1 corresponds to the standard Lightning Indexer (LI). To examine whether a larger selection budget can compensate for more aggressive sharing, we additionally evaluate N =4 with the top-k budget increased from 2K to 4K. We denote DSA with the standard indexer as LI and DSA with CLI as CLI; unless specified otherwise, all variants use the default top-k budget of 2K. Fig. 7 summarizes the training and evaluation results. Panel (a) shows the training loss gap relative to MLA. Although CLI (N =4) exhibits a slightly larger gap, the final difference remains below 0.002 in absolute loss (less than 0.5% relative to the overall loss), suggesting that training loss alone is insufficient to capture long-context quality. This is likely because the variable-length training mixture contains many short sequences where sparse indexing differences have limited impact. Panel (b) evaluates validation loss on a long-context set. While the curves show some variance, N =4 consistently underperforms at the end of training, and increasing the top-k budget to 4K fails to recover the degradation. This indicates that the challenge of sharing one indexer across four layers cannot be addressed simply by expanding the selection budget. Panel (c) evaluates the final base-model checkpoints using needle-in-a-haystack retrieval (omitting the N =4, topk=4k variant since its validation behavior is similar to N =4). Beyond 32K context, N =4 suffers a clear accuracy degradation that grows with sequence length, whereas N =1 and N =2 remain comparable to or slightly better than MLA up to 128K. Based on these results, we discard N =4 and perform supervised fine-tuning on the remaining configurations. Table 10 shows that CLI (N =2) achieves comparable HELMET performance to both LI (N =1) and MLA, demonstrating that 17

LongCat Sparse Attention

halving the number of indexer computations preserves long-context quality. We therefore adopt N =2 as the default CLI group size. 5.3.3

Cross-Layer Distillation Is Essential for Index Reuse

As shown in Fig. 3, adjacent layers exhibit substantial overlap in their salient token sets. This raises a key question: is such inherent inter-layer consistency sufficient for cross-layer index reuse, or is the cross-layer distillation loss (Section 3.2), which explicitly trains the owner indexer to serve multiple layers within a group, necessary? This ablation requires no additional training. Since the indexer input is detached from the model computation graph during training (Section 2.2), indexers at different layers are optimized independently. We therefore construct a “CLI (N=2) w/o cross-layer distill” variant by removing redundant indexers from a trained LI (N=1) model and sharing the remaining indexer across two layers at inference time. We evaluate this variant on needle-in-a-haystack retrieval. As shown in Fig. 7(c), removing cross-layer distillation causes severe degradation at long contexts: accuracy drops to 70% at 128K, below both distilled CLI (N=4, 82%) and distilled CLI (N=2, 96%). These results show that naive index reuse is insufficient. Cross-layer distillation is essential because it explicitly trains the owner indexer to capture the joint saliency patterns of all layers in the group, rather than producing indices optimized for only a single layer. 5.3.4

CLI Extends to MTP Layers MTP Layer 1

MTP Layer 2 0.0005

0.004

0.0175

mean Δ loss=+0.0005 mean Δ acc=-0.0002

MTP Layer 3

mean Δ loss=+0.0006 mean Δ acc=-0.0001

0.0150

0.0000 0.003

0.001

mean Δ loss=+0.0006 mean Δ acc=-0.0001

0.020 0.000

0.001

0.000 0.015

0.0125

−0.001

−0.0005

−0.001

−0.0015

0.001

−0.002

0.010

−0.003

0.0050

Δ Acc

−0.002 0.0075

Δ Loss

−0.0010

Δ Acc

0.002

Δ Loss

0.0100

−0.003 0.005

−0.004

0.0025

−0.004

−0.0020 Δ Loss Δ Acc

0.000

0.0000

−0.005

0.000

−0.005

−0.0025

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4 0.6 Training Progress

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Figure 8: MTP metrics delta (DSA w/ CLI − MLA) for all 3 MTP steps. Accuracy and loss differences remain tightly centered around zero throughout training, confirming that CLI across MTP steps preserves prediction quality. We validate our design in Section 3.2 by comparing MLA with 3-step MTP against DSA with CLI shared across all 3 MTP steps (N =3). Since MTP outputs are draft proposals verified by the main model, any degradation appears as reduced acceptance length. As shown in Fig. 8, after sparse training converges, the LM loss gap remains below 10−3 and prediction accuracy differences stay within 0.1% across all MTP steps. Consistently, Table 5 shows nearly identical acceptance lengths between LSA and MLA across HumanEval, GSM8K, AIME, and MRCR (3.11 vs. 3.15). These results demonstrate that CLI reuse across MTP steps preserves draft quality while eliminating redundant indexer computation. 5.3.5

Hierarchical Indexing Preserves Quality with Proper Configuration

Hierarchical Indexing is training-free (Section 3.3): it is applied purely at inference time and introduces no change to the trained model. Its only quality risk is insufficient coarse-stage recall, which the fine stage cannot recover. In this section we ablate the three knobs that govern this approximation—the pooling method, which indexer layers it is applied to, and the coarse-level recall budget—to identify the configuration that minimizes quality loss while retaining its speedup. Pooling method. The previous work AsyncTLS [Hu et al., 2026] applied max-pooling to compute the block-level score, and the concurrent work HISA [Xu et al., 2026a] also suggests max-pooling as a future direction; we compared the mean pooling to the max-pooling option. Specifically, instead of the mean-pooled dot product qIt,j · kmean in Eq. (9), n max I for each sub-block n we precompute the per-dimension extrema of its constituent keys, kn,d = maxs∈sub-blockn ks,d and min I kn,d = mins∈sub-blockn ks,d , and replace the ReLU argument with a block-level score that upper-bounds the maximum 18

LongCat Sparse Attention

token score within that sub-block: s̃t,n,j =

D X

I max I min max qt,j,d · kn,d , qt,j,d · kn,d ,

(13)

d=1

PH I I PN page where D is the indexer head dimension, so the coarse score becomes It,p = j=1 wt,j n=1 ReLU(s̃t,n,j ). We conduct ablation studies for the pooling method across different pooling sizes under the NIAH 128K task. Mean pooling with a pooling size of 8 is the optimal choice, considering the effectiveness-efficiency tradeoff, as shown in Table 11. Table 11: Ablation studies for the different pooling methods and the pooling sizes under the Needle-in-a-haystack (NIAH) 128K task, the coarse level recalls M = 128 pages (16K tokens). The selected configuration is marked in bold. NIAH 128K Method / Size

Mean MinMax

82 82

76 64

80 60

78 68

74 68

Layer selection. We conduct ablation studies to choose which indexer to turn off the HI. The ablation shows that the shallow layers are more sensitive to the possible recalling error introduced by the HI. The final choice is to turn off the first 4 indexers to help preserve the model performance, as shown in Table 12. Table 12: Ablation studies for turning off the HI of which indexers under the Needle-in-a-haystack (NIAH) 128K task. Mean Pooling with a pooling blocksize B of 8, recalling M = 256 pages (32K tokens). The selected configuration is marked in bold. Options

NIAH 128K

No turning off HI Turn off HI for the first 2 indexers Turn off HI for the first 3 indexers Turn off HI for the first 4 indexers Turn off HI for the first 2 indexers and the last 2 indexers

84 84 86 92 84

Recall budget. To determine the recall pages of the coarse-level indexer of HI, we compared the performance of the different recall numbers in Table 13. Recalling M = 1024 pages is the boundary choice to preserve the performance in the MRCR tasks. Table 13: Ablation studies for the different number of recalled pages under MRCR tasks. Mean Pooling with a pooling blocksize of 8, turning off HI for the first 4 indexers. The selected configuration is marked in bold. TopK of Pages

MRCR 256K

MRCR 512K

256 (32K) 512 (64K) 1024 (128K)

30.96 30.17 32.34

24.32 23.09 30.28

LSA baseline without HI

31.49

27.07

Optimal setting. Based on the results of the three ablation studies, to preserve the lossless performance, the final optimal setting is the Mean pooling method with a pooling blocksize of 8, and turning off HI for the first 4 indexers and recalling M = 1024 pages (128K tokens) for all long context tasks with sequence lengths ≥ 256K. At the operator level, HI under this optimal setting achieves up to a 4.11× indexer speedup over the baseline at 1024K (Table 4). We also conduct a comprehensive evaluation for the Hierarchical Indexer with this optimal setting, across various long context tasks with sequence lengths up to 1024K in Table 15. 5.3.6

LSA Is Robust to Conversion Timing

Following standard practice, LSA is converted from a well-trained MLA model during long-context extension. Since LSA outpaces MLA beyond ∼128K (Appendix A), throughput-optimal training dictates converting at 128K and 19

LongCat Sparse Attention

(a) Stage-1 Loss Gap (128K)

(b) Stage-2 Loss Gap (512K)

0.025

0.020

0.015

0.010 0.005 0.000

0.005 0.000 −0.005

−0.010 0.0

0.4 0.6 Training Progress

0.8

1.0

LSA (128K start) LSA (512K late)

0.04

0.010

−0.005

0.2

LSA (128K start) LSA (512K late)

Δ Loss (LSA − MLA)

LSA (128K start)

Δ Loss (LSA − MLA)

0.025

0.02

0.00

−0.02

−0.04 0.2

0.4 0.6 Training Progress

0.8

1.0

0.0

0.2

0.4 0.6 Training Progress

0.8

1.0

Figure 9: Context-length extension ablation. (a) 128K-stage loss gap (LSA − MLA). (b) 512K-stage loss gap (LSA − MLA). (c) SFT loss gap (LSA − MLA). Values near zero indicate negligible quality difference. Table 14: HELMET evaluation for context-extension strategies. Both early (128K start) and late (512K start) LSA introduction match the MLA baseline across all long-context categories. MLA (baseline) LSA (128K start) LSA (512K late)

Recall

RAG

Re-rank

LongQA

Cite

Summ

Avg

98.83 99.35 98.63

64.61 64.07 64.38

71.34 71.77 72.64

44.03 44.05 44.46

35.83 37.51 37.53

36.38 37.01 36.48

58.50 58.96 59.02

remaining sparse thereafter. However, prior works [DeepSeek-AI, 2025a, Zeng et al., 2026] defer conversion to the final extension stage. We hypothesize this stems from indexer distribution shifts: in sparse training, the KL loss supervision couples to the indexer’s own top-K token selection (St , Eq. (5)). Abrupt context-length transitions force the indexer through out-of-distribution shifts, potentially degrading supervision quality if converted too early. To verify whether the conversion timing affects quality, we compare two schedules on LongCat-Flash-Lite: (1) LSA (128K start), which converts from MLA to LSA at the very beginning of the 128K stage (100B tokens) and then completes the entire 512K stage (100B tokens) in sparse mode; and (2) LSA (512K late), which keeps full MLA throughout the 128K stage and converts to LSA only in the final third of the 512K stage—following the standard late-conversion recipe above. As shown in Fig. 9, while the late-conversion schedule maintains a slightly tighter pre-training loss gap relative to MLA, both schedules keep this gap strictly below 0.01 throughout (< 0.5% of the absolute loss, ∼2), and the SFT loss gap is centered at zero. Downstream evaluation on HELMET (Table 14) confirms that early conversion achieves performance parity with both late conversion and full MLA (58.96 vs. 59.02 and 58.50, respectively). Since downstream performance is closely matched, we recommend converting to LSA as early as the 128K crossover stage to maximize training efficiency.

LongCat-Flash-Lite-Sparse

Building on the techniques developed in this work, we introduce LongCat-Flash-Lite-Sparse, which integrates the full LSA recipe into LongCat-Flash-Lite [Liu et al., 2026]. Compared with the previously released model with full attention, the new model replaces dense MLA with LSA for substantially improved inference efficiency, extends the native context length from 128K to 1M, achieves stronger agentic capabilities, and preserves strong reasoning and general-knowledge performance. 6.1

Model Informations

Architecture. The released model is based on LongCat-Flash-Lite (69B-A3B) and replaces dense MLA attention with LSA using the configuration validated in Section 5.1: a total attention budget of K = 2048 tokens, including Ksink = 16 sink tokens, Kswa = 1024 sliding-window tokens, and dynamically selected sparse tokens. The model uses Cross-Layer Indexing with group size N = 2, while Hierarchical Indexing is applied as a training-free inference optimization beyond its efficiency crossover point. We further integrate a 3-step MTP module for speculative decoding, where all MTP steps share one indexer through CLI (N = 3). 20

LongCat Sparse Attention

Training pipeline. We extend LongCat-Flash-Lite’s long-context training pipeline through five stages: 32K, 64K, 128K, 256K, and 1M. Based on profiling and ablation studies, we convert dense MLA to LSA at the beginning of the 128K stage, followed by sparse training across the 128K, 256K, and 1M stages. The MTP module is introduced at the 32K stage and converted to LSA together with the main model at 128K. After long-context extension, the model undergoes supervised fine-tuning. Note that only Streaming-Aware Indexing and Cross-Layer Indexing participate in training; Hierarchical Indexing is training-free and applied only during inference. 6.2

Model Evaluation

We first evaluate long-context capabilities with and without HI (Table 15), followed by a comprehensive evaluation on agentic, general-knowledge, and reasoning tasks (Table 16). For clarity, in these comparisons, Lite-Dense denotes LongCat-Flash-Lite, while Lite-Sparse (w/o HI) and Lite-Sparse (w/ HI) denote the sparse variants evaluated without and with Hierarchical Indexing, respectively. Benchmarks.

We evaluate across four capability groups:

• Long-context: We adopt ATLAS [Huang et al., 2026], which unifies existing long-context benchmarks under a common length axis and evaluates them at 8K, 16K, 32K, 64K, 128K, 256K, 512K, and 1M tokens. The evaluation covers retrieval (MRCR 8-needle [OpenAI, 2025a]), aggregation (OOLong-Synth [Bertsch et al., 2025]), multi-step reasoning (GraphWalks [OpenAI, 2025b]), question answering (LOFT-Text Retrieval [Lee et al., 2024]), in-context learning (HELMET-ICL [Yen et al., 2024]), code understanding (LongCodeQA [Rando et al., 2025]), long-range memory (AMemBench-ACU [Jiayang et al., 2026]), and holistic evaluation (LongBench-v2 [Bai et al., 2024] and AA-LCR [Artificial Analysis, 2025]). • Agentic: We evaluate agentic coding (SWE-Bench Verified, SWE-Bench Pro, SWE-Bench Multilingual [Jimenez et al., 2023], and TerminalBench [Merrill et al., 2026]), agentic tool use (τ 2 -Bench [Barres et al., 2025], VitaBench [He et al., 2025], and MCP-Atlas [Bandi et al., 2026]), and agentic search (BrowseComp [Wei et al., 2025, Zhou et al., 2025] and RWSearch [Zhu et al., 2026]). • General Domains: MMLU [Hendrycks et al., 2021a], MMLU-Pro [Wang et al., 2024], CMMLU [Li et al., 2023b], and C-Eval [Huang et al., 2024]. • Mathematical Reasoning: GPQA-Diamond [Rein et al., 2024], MATH500 [Hendrycks et al., 2021b], AIME 2026 [Dekoninck et al., 2026], HMMT 2026 [Dekoninck et al., 2026], BeyondAIME [ByteDance Seed, 2025], and IMO AnswerBench [Luong et al., 2025]. Long-context evaluation. Table 15 compares Lite-Sparse with and without HI on nine long-context benchmarks up to 1M context lengths. Overall, HI maintains highly competitive performance across all tasks. On most benchmarks, the score variations remain within roughly 1 point (e.g., LongBench-v2: 53.64 vs. 52.50, MRCR: 44.47 vs. 44.66, and AMemBench-ACU: 33.13 vs. 33.25), with a slight drop observed on code understanding (LongCodeQA: 59.37 vs. 62.30). These results demonstrate a favorable efficiency-accuracy trade-off: training-free Hierarchical Indexing largely preserves long-context capabilities while delivering substantial inference speedups with only minor quality degradation. Table 15: Long-context evaluation of LongCat-Flash-Lite-Sparse on the ATLAS [Huang et al., 2026] benchmark suite, organized by capability dimension. Every component supports context lengths up to 1M tokens. Layer

Dimension

Benchmark

Foundational

Retrieval Aggregation Multi-step Reasoning Question Answering In-context Learning Code Understanding Long-range Memory

Application

Holistic Assessment

Lite-Sparse (w/o HI)

Lite-Sparse (w/ HI)

MRCR (8-needle) OOLong-Synth GraphWalks Extend

44.66 38.42 66.27

44.47 37.88 65.63

LOFT Retrieval Extend HELMET-ICL Extend LongCodeQA AMemBench-ACU LongBench-v2 AA-LCR

43.75 91.63 62.30 33.25 52.50 48.00

44.38 90.50 59.37 33.13 53.64 47.33

Overall capability evaluation. Table 16 summarizes the overall capability of Lite-Sparse compared with Lite-Dense, whose scores are taken from its technical report [Liu et al., 2026]. Lite-Sparse achieves strong performance on agentic benchmarks, including 68.20 on SWE-Bench Verified, 59.33 on SWE-Bench Multilingual, 95.18 on τ 2 -Telecom, and 21

LongCat Sparse Attention

Table 16: Evaluation results of LongCat-Flash-Lite-Sparse compared with LongCat-Flash-Lite. Benchmark

Lite-Dense

Lite-Sparse (w/o HI)

Lite-Sparse (w/ HI)

Agentic Coding SWE-Bench Verified (acc) SWE-Bench Pro (acc) SWE-Bench Multilingual (acc) TerminalBench 2.0 (acc)

54.40 – 38.10 33.75

68.20 40.63 59.33 33.70

65.20 39.40 56.00 32.58

95.18 21.67 45.60

96.05 20.42 45.00

48.62 61.94 68.50

48.18 61.59 66.00

85.31 79.24 84.25 85.76

85.14 78.68 84.51 85.71

Agentic Tool Use τ 2 -Telecom (avg@4) VitaBench (avg@4) MCP-Atlas

72.80 7.00 – Agentic Search

BrowseComp (pass@1) BrowseComp-zh (pass@1) RWSearch (pass@1)

– – – General Domains

MMLU (acc) MMLU-Pro (acc) CMMLU (acc) C-Eval (acc)

85.52 78.29 82.48 86.55

Mathematical Reasoning GPQA-Diamond (avg@16) MATH500 (acc) AIME 2026 (avg@32) HMMT 2026 Feb (avg@32) BeyondAIME (avg@10) IMO AnswerBench (avg@4)

69.49 95.80 65.73 40.53 44.20 49.38

66.78 96.80 – – – –

69.03 96.80 64.90 41.47 42.30 46.69

21.67 on VitaBench (without HI). Enabling HI introduces a modest quality trade-off on several agentic tasks (e.g., SWE-Bench Verified: 68.20 → 65.20, SWE-Bench Multilingual: 59.33 → 56.00, and RWSearch: 68.50 → 66.00). Overall, Lite-Sparse preserves the dense model’s reasoning and general-knowledge capabilities while improving its agentic performance and enabling substantially more efficient long-context inference.

Conclusion, Limitations, and Future Directions

We presented LongCat Sparse Attention (LSA), a co-designed sparse-attention framework that addresses the inefficient memory access and high indexing overhead of DSA through three complementary mechanisms: Streaming-Aware Indexing, Cross-Layer Indexing, and Hierarchical Indexing. Evaluations at both model scales show that LSA achieves nearly lossless performance relative to full attention while delivering substantial training and inference speedups. The resulting training-efficiency gains make native training with context lengths of up to one million tokens practical and support the development of LongCat-2.0 (1.6T-A48B). To facilitate further research, we also release LongCat-FlashLite-Sparse (69B-A3B), an open-source model that integrates LSA into LongCat-Flash-Lite and incorporates an updated long-context training corpus. A primary constraint of LSA is that while it substantially reduces attention computation, it leaves the total KVcache footprint intact, as every token must still store a KV entry. Although KV-cache partitioning and host-memory offloading (Section 4.3) mitigate per-device memory pressure, they do not lower the aggregate storage overhead. A promising future direction is to combine LSA with complementary KV-cache compression paradigms. For instance, Cross-Layer Attention (CLA) [Brandon et al., 2024] shares KV states across layers to compress the cache along the depth dimension, whereas DeepSeek-V4’s Compressed Sparse Attention (CSA) [Xu et al., 2026b] achieves sequencedimension compression via block-level sparse selection. Fusing LSA with these orthogonal techniques holds great potential for scaling long-context models that are simultaneously compute- and memory-efficient.

LongCat Sparse Attention

Acknowledgement

We extend our sincere gratitude to both the infrastructure team and evaluation team for their invaluable support and constructive feedback throughout this project. The primary contributors from these teams include: Yuxuan Hu Jiaxin Hou Rumei Li

Gang Liu Yuwei Jiang Dengchang Zhao

Li Wei Bole Zhou

Hongjun Wu Xing Hu

LongCat Sparse Attention

References Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017. DeepSeek-AI. Deepseek-v3.2-exp: Boosting long-context efficiency with deepseek sparse attention. Technical Report, 2025a. URL https://github.com/deepseek-ai/DeepSeek-V3.2-Exp. DeepSeek-AI. Deepseek-v3.2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556, 2025b. Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, et al. Glm-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. International Conference on Learning Representations, 2024a. Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. Duoattention: Efficient long-context llm inference with retrieval and streaming heads. arXiv preprint arXiv:2410.10819, 2024b. Lijie Yang, Zhihao Zhang, Zhuofu Chen, Zikun Li, and Zhihao Jia. Tidaldecode: Fast and accurate llm decoding with position persistent sparse attention. arXiv preprint arXiv:2410.05076, 2024. Dhruv Deshmukh, Saurabh Goyal, Nipun Kwatra, and Ramachandran Ramjee. Kascade: A practical sparse attention method for long-context llm inference. arXiv preprint arXiv:2512.16391, 2025. Yizhao Gao, Jianyu Wei, Qihao Zhang, Yu Cheng, Shimao Chen, Zhengju Tang, Zihan Jiang, Yifan Song, Hailin Zhang, Liang Zhao, et al. Hysparse: A hybrid sparse attention architecture with oracle token selection and kv cache sharing. arXiv preprint arXiv:2602.03560, 2026. Hong Liu, Jiaqi Zhang, Chao Wang, Xing Hu, Linkun Lyu, Jiaqi Sun, Xurui Yang, Bo Wang, Fengcun Li, Yulei Qian, et al. Scaling embeddings outperforms scaling experts in language models. arXiv preprint arXiv:2601.21204, 2026. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019. Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020. Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems, 33, 2020. Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. International Conference on Learning Representations, 2020. Di Liu, Meng Chen, Baotong Lu, Huiqiang Jiang, Zhenhua Han, Qianxi Zhang, Qi Chen, Chengruidong Zhang, Bailu Ding, Kai Zhang, et al. Retrievalattention: Accelerating long-context llm inference via vector retrieval. arXiv preprint arXiv:2409.10516, 2024a. Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, et al. Moba: Mixture of block attention for long-context llms. arXiv preprint arXiv:2502.13189, 2025. Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y. X. Wei, Lean Wang, Zhiping Xiao, et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention. arXiv preprint arXiv:2502.11089, 2025. Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150, 2019. Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024b. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024c. Weilin Cai, Juyong Jiang, Le Qin, Junwei Cui, Sunghun Kim, and Jiayi Huang. Shortcut-connected expert parallelism for accelerating mixture-of-experts. International Conference on Machine Learning, 2024. Meituan LongCat Team, Bei Li, Bingye Lei, Bo Wang, Bolin Rong, Chao Wang, Chao Zhang, Chen Gao, Chen Zhang, Cheng Sun, et al. Longcat-flash technical report. arXiv preprint arXiv:2509.01322, 2025. 24

LongCat Sparse Attention

Yushi Bai, Qian Dong, Ting Jiang, Xin Lv, Zhengxiao Du, Aohan Zeng, Jie Tang, and Juanzi Li. Indexcache: Accelerating sparse attention via cross-layer index reuse. arXiv preprint arXiv:2603.12201, 2026. Howard Yen, Tianyu Gao, and Danqi Chen. Helmet: How to evaluate long-context language models effectively and thoroughly. arXiv preprint arXiv:2410.02694, 2024. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2369–2380, 2018. doi:10.18653/v1/D18-1259. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), pages 9802–9822, 2023. doi:10.18653/v1/2023.acl-long.546. Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268, 2016. Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317–328, 2018. doi:10.1162/tacl_a_00023. Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. ∞bench: Extending long context evaluation beyond 100k tokens. arXiv preprint arXiv:2402.13718, 2024. Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. Enabling large language models to generate text with citations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6465–6488, 2023. Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. ASQA: Factoid questions meet long-form answers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8273–8288, 2022. Samuel Joseph Amouyal Ohad Rubin, Ori Yoran, Tomer Wolfson, Jonathan Herzig, and Jonathan Berant. QAMPARI: An open-domain question answering benchmark for questions with many answers from multiple paragraphs. arXiv preprint arXiv:2205.12665, 2022. Zejiang Shen, Kyle Lo, Lauren Yu, Nathan Dahlberg, Margo Schlanger, and Doug Downey. Multi-LexSum: Real-world summaries of civil rights lawsuits at multiple granularities. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph E. Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. How long can open-source llms truly promise on context length?, June 2023a. URL https: //lmsys.org/blog/2023-06-29-longchat. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. International Conference on Learning Representations, 2021a. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574, 2024. Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint arXiv:2306.09212, 2023b. Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, et al. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. Advances in Neural Information Processing Systems, 36, 2024. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2024. 25

LongCat Sparse Attention

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. Advances in Neural Information Processing Systems, 34, 2021b. MAA. Aime 2024, 2024. URL https://maa.org/math-competitions/ american-invitational-mathematics-examination-aime. MAA. Aime 2025, 2025. URL https://artofproblemsolving.com/wiki/index.php/ AIMEProblemsandSolutions. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 215–227, 2023. doi:10.1145/3597926.3598030. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024. Yuxuan Hu, Jianchao Tan, Jiaqi Zhang, Wen Zan, Pingwei Sun, Yifan Lu, Yerui Sun, Yuchen Xie, Xunliang Cai, and Jing Zhang. Asynctls: Efficient generative llm inference with asynchronous two-level sparse attention. arXiv preprint arXiv:2604.07815, 2026. Yufei Xu, Fanxu Meng, Fan Jiang, Yuxuan Wang, Ruijie Zhou, Zhaohui Wang, Jiexi Wu, Zhixin Pan, Xiaojuan Tang, Wenjie Pei, et al. Hisa: Efficient hierarchical indexing for fine-grained sparse attention. arXiv preprint arXiv:2603.28458, 2026a. Deli Huang, Cunguang Wang, Hongyin Tang, Zhe Tang, Linsen Guo, Dongyu Ru, Ruoshi Yuan, Ziyue Zhu, Xiaoyu Li, Ziwen Wang, Chen Zhang, Anchun Gui, Wen Zan, Jiaqi Zhang, Xuezhi Cao, Jingang Wang, Xunliang Cai, and Yixin Cao. Atlas: All-round testing of long-context abilities across scales. arXiv preprint arXiv:2605.28079, 2026. OpenAI. OpenAI MRCR: Long context multiple needle in a haystack benchmark. Hugging Face dataset, 2025a. URL https://huggingface.co/datasets/openai/mrcr. Amanda Bertsch, Adithya Pratapa, Teruko Mitamura, Graham Neubig, and Matthew R. Gormley. Oolong: Evaluating long context reasoning and aggregation capabilities, 2025. OpenAI. Graphwalks dataset, 2025b. URL https://huggingface.co/datasets/openai/graphwalks. Jinhyuk Lee, Anthony Chen, Zhuyun Dai, Dheeru Dua, Devendra Singh Sachan, Michael Boratko, Yi Luan, Sébastien MR Arnold, Vincent Perot, Siddharth Dalmia, et al. Can long-context language models subsume retrieval, rag, sql, and more? arXiv preprint arXiv:2406.13121, 2024. Stefano Rando, Luca Romani, Alessio Sampieri, Luca Franco, John Yang, Yuta Kyuragi, Fabio Galasso, and Tatsunori Hashimoto. Longcodebench: Evaluating coding llms at 1m context windows, 2025. URL https://arxiv.org/ abs/2505.07897. Cheng Jiayang, Dongyu Ru, Lin Qiu, Yiyang Li, Xuezhi Cao, Yangqiu Song, and Xunliang Cai. Amemgym: Interactive memory benchmarking for assistants in long-horizon conversations. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=sfrVLzsmlf. Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. arXiv preprint arXiv:2412.15204, 2024. Artificial Analysis. Announcing artificial analysis long context reasoning (aa-lcr). Artificial Analysis article, 2025. URL https://artificialanalysis.ai/articles/announcing-aa-lcr. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023. Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, et al. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces, 2026. Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. τ 2 -bench: Evaluating conversational agents in a dual-control environment. arXiv preprint arXiv:2506.07982, 2025. Wei He, Yueqing Sun, Hongyan Hao, Xueyuan Hao, Zhikang Xia, Qi Gu, Chengcheng Han, Dengchang Zhao, Hui Su, Kefeng Zhang, Man Gao, Xi Su, Xiaodong Cai, Xunliang Cai, Yu Yang, and Yunke Zhao. Vitabench: Benchmarking llm agents with versatile interactive tasks in real-world applications. arXiv preprint arXiv:2509.26490, 2025. Chaithanya Bandi, Ben Hertzberg, Geobio Boo, Tejas Polakam, Jeff Da, Sami Hassaan, Manasi Sharma, Andrew Park, Ernesto Hernandez, Dan Rambado, et al. Mcp-atlas: A large-scale benchmark for tool-use competency with real mcp servers. arXiv preprint arXiv:2602.00933, 2026. 26

LongCat Sparse Attention

Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.12516, 2025. Peilin Zhou, Bruce Leon, Xiang Ying, Can Zhang, Yifan Shao, Qichen Ye, Dading Chong, Zhiling Jin, Chenxuan Xie, Meng Cao, Yuxin Gu, Sixin Hong, Jing Ren, Jian Chen, Chao Liu, and Yining Hua. Browsecomp-zh: Benchmarking web browsing ability of large language models in chinese. arXiv preprint arXiv:2504.19314, 2025. Mingyang Zhu, Wei Liu, Ziwen Wang, Lin Qiu, and Xuezhi Cao. Rw-search: A high-order logic and real-world search evaluation benchmark for agents, 2026. URL https://github.com/AGI-Eval-Official/RW-Search. Jasper Dekoninck, Nikola Jovanović, Tim Gehrunger, Kári Rögnvaldsson, Ivo Petrov, Chenhao Sun, and Martin Vechev. Beyond benchmarks: Matharena as an evaluation platform for mathematics with llms. arXiv preprint arXiv:2605.00674, 2026. ByteDance Seed. Beyondaime: Advancing math reasoning evaluation beyond high school olympiads, 2025. URL https://huggingface.co/datasets/ByteDance-Seed/BeyondAIME. Thang Luong, Dawsen Hwang, Hoang H. Nguyen, Golnaz Ghiasi, Yuri Chervonyi, Insuk Seo, Junsu Kim, Garrett Bingham, Jonathan Lee, Swaroop Mishra, Alex Zhai, Clara Huiyi Hu, Henryk Michalewski, Jimin Kim, Jeonghyun Ahn, Junhwi Bae, Xingyou Song, Trieu H. Trinh, Quoc V. Le, and Junehyuk Jung. Towards robust mathematical reasoning. arXiv preprint arXiv:2511.01846, 2025. William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, and Jonathan Ragan-Kelley. Reducing transformer key-value cache size with cross-layer attention. Advances in Neural Information Processing Systems, 37: 86927–86957, 2024. Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chenchen Ling, et al. Deepseek-v4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348, 2026b.

LongCat Sparse Attention

Training Efficiency of LSA versus Dense MLA

Section 4.2 benchmarks LSA against the DSA sparse baseline. For completeness, Fig. 10 compares the per-layer training latency against dense MLA, highlighting a clear length-dependent crossover point. At short sequence lengths (< 64K), dense MLA runs faster because LSA incurs indexing overheads (yielding a net latency of 0.83× at 32K). However, beyond 64K, the quadratic scaling of dense MLA takes over—especially in the backward pass—allowing LSA to reverse the gap sharply, reaching up to a 7.73× speedup at 1024K. This demonstrates that replacing dense MLA with LSA delivers rapidly compounding efficiency gains precisely for long-context regimes (≥ 64K). Note that these microbenchmarks are measured with fixed sequence lengths; when accounting for variable-length sequence packing in our actual data mixture, the practical efficiency crossover occurs at 128K.

2.21x 1.92x

102

Backward

2.91x

3.77x

103 2.28x

1.47x

64K

128K 256K Sequence Length

1024K

32K

7.73x

3.39x

103 2.21x

1.37x

1.39x

0.79x

1.08x

32K

Total (Fwd + Bwd)

11.76x

Latency (ms)

Forward MLA LSA 2.91x Speedup Ratio

Latency (ms)

103

0.83x

64K

128K 256K Sequence Length

1024K

32K

64K

128K 256K Sequence Length

1024K

Figure 10: Single-attention-layer training latency of LSA and dense MLA across context lengths. Bars report forward, backward, and total latency, including kernel execution and CP communication; arrows indicate LSA speedup over MLA. The CP degrees are 4, 8, 16, 32, and 128 for context lengths of 32K, 64K, 128K, 256K, and 1024K, respectively.

Serving setup and KV-cache partitioning

For long-context inference, the KV cache may exceed the memory capacity of a single accelerator. We therefore introduce KV-cache Partition (KVP), which shards cache pages across multiple ranks. Specifically, page i is assigned to rank i mod NKVP , balancing memory usage across the KVP group. Our serving system uses length-aware routing to dispatch requests to instances with parallel configurations optimized for their context lengths. • Prefill. Prefill uses a 16K context chunk size with TP=8, EP=8, PP=2, and a CP/KVP group size of 8. The indexer K cache and attention KV cache are sharded across KVP ranks at page granularity. Before processing each chunk, the complete cache blocks are all-gathered so that every rank can attend to the full context. • Decoding. Decoding uses a 4K chunk size. Short-context requests are served with data-parallel attention and expert parallelism (DP=16, EP=16). Requests with at least 256K context tokens are routed to two data-parallel replicas, each using an eight-rank KVP group. With KVP, the indexer and SFA operate on local cache shards. Each rank selects its local Top-K candidates; the resulting KNKVP candidates are then all-gathered and re-ranked to obtain the global Top-K. In parallel, SFA computes attention over each local KV shard and combines the partial outputs using log-sum-exp normalization statistics, yielding the same result as attention over the global cache.

LongCat Sparse Attention

Case Study: LSA Behavior on Needle-in-a-Haystack

To illustrate how LSA’s sparse selection behaves, we present a detailed case study on the RULER NIAH (needle-in-ahaystack) multi-key task, which directly probes the indexer: retrieving a single needle line from many near-identical distractors aligns precisely with the indexer’s objective of isolating the few relevant tokens within a long, distractorheavy context. Each prompt comprises a task description, a sequence of key–value lines (“One of the special magic uuids for <key> is: <value>”), and a closing question that queries the value of one specific key; the single needle line holds the answer (an abridged example is shown in Fig. 11). The model adopts two of the three LSA components. Streaming-Aware Indexing partitions the budget into a fixed sink (Ksink =16), a sliding window (Kswa =1024), and a dynamic sparse set (a ∼1:1 fixed-to-sparse ratio); Cross-Layer Indexing with group size N =2 lets each pair of consecutive layers share one indexer, so we label the shared figures by the owner layer (e.g. L26 for the L26–L27 pair). We omit Hierarchical Indexing: it is a training-free, inference-time approximation that accelerates the selection rather than reshaping it during training, and thus lies outside the learned selection behavior studied here. For display clarity, all figures are computed on a single representative sample of length S ≈ 6K. A special magic uuid is hidden within the following text.

Make sure to memorize it.

I will quiz you about the uuid

afterwards. One of the special magic uuids for fb62b35d-...-03bd48468d49 is: fb2d6908-...-f1eaf31b7c54. One of the special magic uuids for 1b20a57e-...-12df0ef04ec3 is: b364c1fd-...-6f649741c0fa. . . . (72 more distractor lines) One of the special magic uuids for e8ea1605-...-0cd28545ce6d is: 9ac11a87-...-04225be0162e. . . . (3 more distractor lines) What is the special magic uuid for e8ea1605-...-0cd28545ce6d mentioned in the provided text? The special magic uuid for e8ea1605-...-0cd28545ce6d mentioned in the provided text is Target:

9ac11a87-...-04225be0162e

Figure 11: Abridged prompt for the case-study sample. The three spans analyzed in the visualizations below are colorcoded: the task description , the target needle line (which supplies the answer), and the question . The remaining 77 lines are same-format distractors; UUIDs are shortened for display.

C.1

Overview

Fig. 12 gives an overview spanning four quantities: the indexer score, the selection mask (the fixed sink and slidingwindow positions of Streaming-Aware Indexing together with the indexer’s Top-K choice), the full MLA attention weights (p in Eq. (4)), and the resulting sparse MLA weights (the full weights after applying the selection mask). Several of these quantities are produced by a semantically equivalent reference implementation for analysis, rather than by the efficient fused-kernel implementation used in actual training and inference. The latter never instantiates a softmax over the indexer score, an explicit selection mask, or the full MLA weight matrix; we recompute all of them here purely for visualization, and normalize the indexer score by a softmax to make it comparable to the attention weights. For a CLI reuse layer, the indexer score and selection mask are copied directly from its group owner; for space we show two CLI groups, one from the middle and one from the tail of the network (layers 12/13 and 26/27). Full MLA weights, and sparse MLA weights are all clipped at a maximum of 10−3 for display. Several patterns stand out. First, the indexer score closely tracks the full MLA weights, indicating that the KL distillation during training lets the indexer faithfully reproduce the core attention distribution. Second, the sparse MLA weights closely match the full MLA weights, showing that the post-selection attention is a good approximation of full attention. Third, the selection mask reserves a wide bright band along the diagonal, the fixed SWA region. Finally, the indexer score, full MLA weights, and sparse MLA weights all exhibit bright regions along the first column and the diagonal, indicating that attention mass naturally concentrates on the initial tokens and the local neighborhood—exactly the structure that motivates the fixed sink and sliding-window (SWA) budgets of Streaming-Aware Indexing. The following two subsections examine the discrete selection (Appendix C.2) and the continuous attention weights (Appendix C.3) in detail.

LongCat Sparse Attention

Indexer Score (softmax normalized)

Selection Mask

Query position

Sparse MLA Weights (head-averaged w/ selection mask)

SWA boundary selected not selected

Layer 12

Full MLA Weights (head-averaged w/o selection mask)

0.0008

0.0006

3k 0.0004 4k 0.0002

0.0000

reuse indexer

SWA boundary selected not selected

Query position

Layer 13

0.0010

2k 3k 4k 5k

SWA boundary selected not selected

Layer 26

Query position

1k 2k 3k 4k 5k reuse indexer

SWA boundary selected not selected

Layer 27

Query position

1k 2k 3k 4k 5k 0

2k 3k 4k Key position

Figure 12: Overview of attention maps for the case-study sample at four layers (12, 13, 26, 27). Columns: (1) LSA indexer score (softmax-normalized), (2) selection mask (Top-K, K=2048; white = selected; red dashed lines mark sliding-window boundaries), (3) Full Attention (MLA, head-averaged), (4) LSA sparse attention (head-averaged, after selection).

LongCat Sparse Attention

C.2

Indexer Selection

We now focus on the discrete selection produced by the LSA indexer. Fig. 13 extracts the layer 26 selection mask from the Fig. 12, using the question tokens as queries. Beyond the bright diagonal band at the tail corresponding to the fixed SWA region, two sharply localized high-selection regions emerge in the key sequence: one near the beginning of the context and another around position 1K. These regions correspond to the task description and the target needle line, respectively, showing that the indexer effectively identifies the information-bearing tokens required to answer the query while filtering out irrelevant distractors. Fig. 14(a) quantifies this behavior by measuring the fraction of tokens selected from each line by question queries (excluding positions covered by SWA). The target needle line is selected substantially more frequently than other lines, achieving a selection ratio of 58%, compared with an average of 22% across all lines. Fig. 14(b) further decomposes selection within the needle line: compared with the surrounding text, the KEY and VALUE tokens, as well as the connecting token “is:”, receive significantly higher selection rates. Fig. 15 visualizes the same pattern at token granularity by highlighting selected tokens directly in the prompt.

Query position (final question)

task description

target needle line

question selected not selected SWA boundary

5900 5925 5950 0

Key position

Figure 13: LSA indexer selection mask for final-question query tokens at layer 26, whose indices are shared with layer 27 via Cross-Layer Indexing (N=2). White and black denote selected and unselected positions, respectively. Red dashed lines indicate the causal and SWA boundaries, while red brackets highlight the task-description span, target needle line, and question key span. (a) Per-line selection density: indexer concentrates budget on the needle line (averaged over the question tokens) avg. fraction of tokens selected per line (22%)

other 42%

KEY 67%

is: 88%

VALUE 52%

1.0 0.8

selection frequency

mean fraction of line tokens selected

1.0

(b) Zoom-in: per-token selection frequency within the target needle line

target needle line 58%

0.6

0.4

0.5

KEY ' is: ' VALUE other

0.2

0.0

1000

2000

3000

4000

Key token position (all lines except SWA)

0.0

1020

1040

1060

1080

Key token position (only target needle line)

Figure 14: (a) Per-line selection density at layer 26: the fraction of each line’s tokens selected by the indexer, averaged over the question tokens (lines fully covered by SWA are omitted). The target needle line is highlighted in red, and the dashed line denotes the average selection density across lines. (b) Per-token selection frequency within the target needle line, colored by segment (KEY / “ is: ” / VALUE / other). The needle line is selected far more often than distractor lines, and within it the KEY and VALUE tokens are preferentially selected—evidence that the indexer targets both the answer and its key.

LongCat Sparse Attention

A special magic uuid is hidden within the following text. Make sure to memorize it. I will quiz you about the uuid afterwards. One of the special magic uuids for fb62b35d-91f9-4c16-81b6-03bd48468d49 is: fb2d6908-388d-46c3-990e-f1eaf31b7c54. One of the special magic uuids for 1b20a57e-5e86-4a3d-87d7-12df0ef04ec3 is: b364c1fd-a0f0-442a-bd8b-6f649741c0fa. One of the special magic uuids for 73901c46-6ec4-40b2-973a-492abb00037f is: eac176f3-a9d5-4bcf-b067-8d81590de23b. One of the special magic uuids for 5d332535-5d99-4b1e-b54d-117766e24a94 is: e2f27f04-4bd6-407d-813e-e22dc73e77c7. One of the special magic uuids for 2f29ca3e-fd92-4a2a-9a0b-7ca42548f600 is: 161fc0eb-b972-4f40-a35d-e60372c0d42d. One of the special magic uuids for 677a21cf-63c4-422c-a7f3-b30374a7c15b is: 5c9379a6-a745-4323-82aa-fe29221deb4f. One of the special magic uuids for 424d3469-06a0-4fb4-9df0-fb2667b43715 is: 79f61159-020b-453b-9fbc-277061fdf942. One of the special magic uuids for 153fb118-0af7-41a9-a9d0-716a05c62ffa is: a7669cee-ffba-431a-9be8-d3ef4b4788c1. One of the special magic uuids for d6bdb561-1c79-4c85-8642-a480ebc56f09 is: d1836f31-fca0-4d72-814d-d11438f49961. One of the special magic uuids for f4aa1f32-a5f5-4ec4-825d-d0ea60c96560 is: 2e169e20-dc74-4ec7-abc5-4cdbf6054d3e. One of the special magic uuids for b34f9f51-c996-49e6-8a5f-dcff42bbba8c is: cedad670-c72c-4e50-b1f0-be7a830f3db8. One of the special magic uuids for 00e0d047-0151-42f1-a07f-b7511a447ac6 is: d00f207b-faff-492e-a35f-345cf7ea0599. One of the special magic uuids for 997929e3-8a9f-40e6-a8bf-e4fe696f0161 is: fcff9ed9-459e-441c-ac1c-e3b20f1a4334. One of the special magic uuids for e8ea1605-ea36-4b09-8695-0cd28545ce6d is: 9ac11a87-63f3-41a9-ad09-04225be0162e. One of the special magic uuids for 19aca52d-c27b-4491-8de0-81faf8d4b975 is: 2406225c-894a-4e18-a2b5-15247c616af7. One of the special magic uuids for 2b22d9cd-685b-48a9-b8c2-249a01f9214f is: 1c7f8315-b859-45b0-8062-72a131325047. One of the special magic uuids for caf13eb1-dec4-4905-99e6-8fe21f28b08d is: 3ab7e477-23d7-4f97-a94d-6e96a8db9b01. One of the special magic uuids for 2e54ca50-25e8-483d-8d5d-e8d9249031eb is: 48ae3a47-27af-4995-8e5a-5e092ccc181e. One of the special magic uuids for 78963d04-4718-46d9-893a-9348035f2c06 is: 86a883f2-27d1-4679-9dc7-af5953a55fb6. One of the special magic uuids for 9b228880-fffb-4041-9ba0-5fa5314e8bc7 is: 082ddc63-4c12-41b4-9872-643b974f65f5. One of the special magic uuids for 09fccdac-179e-4857-85de-0e0b005476f5 is: 449a1d03-8de2-4c8d-8055-ceb0b9ee2247. One of the special magic uuids for 1da6833f-cd72-4a7f-9af7-884873ce9813 is: c8211db0-e3d0-4f71-9f7e-9ce03a698c2f. One of the special magic uuids for ef9e83f4-1d8e-4dbc-b45c-25af1a0e3e79 is: 079a8055-aac1-4b42-98d6-e90d3f0bdbda. One of the special magic uuids for c638be20-f218-4b0e-9a5d-cf8dd3778853 is: f7993edc-5195-4d56-b1b9-b512507dbc11. One of the special magic uuids for ed7a2e7e-7168-440b-b194-f3d4c79eeb8e is: a4a411c9-c421-4a75-aa2b-e014254747cd. One of the special magic uuids for ec6cd6e0-2199-4b17-9790-808a87d63a5b is: edf27628-caa2-4c81-81f5-badfd35d46c4. One of the special magic uuids for a3b9fb3d-1e1e-4a47-8a0d-7cd86fca4dbf is: 5c009a69-aa37-4383-af9e-3c5cb340e35d. One of the special magic uuids for fb04f274-8e78-446b-bd47-af43f4dd5978 is: 76e06135-c259-45bf-994a-a04f0a7ab509. One of the special magic uuids for 20ac14c2-077a-4963-bd81-f28086315f2b is: d9fb128a-0466-4d04-8a5a-e470ed37e6c8. One of the special magic uuids for 4cd37b05-e51b-44b4-8613-252312a23aa7 is: e8f7f4ca-49a6-4acc-9f3f-4d911a321483. One of the special magic uuids for 54556afa-f177-4366-b658-f7a0dde17cd3 is: 9cfee5c9-13c3-46ce-8d20-d60f3e1a09ae. One of the special magic uuids for 2051c45f-c251-4191-9f20-53a3553a0159 is: c2cda60a-82b2-485a-8319-cd6fc9617519. One of the special magic uuids for a7e19fac-ee09-4932-9879-b203e8092f1a is: 8e40669a-7833-4ea1-89b4-8c9b433c2de9. One of the special magic uuids for 49afb037-7442-42f2-8cc4-1ce868f52184 is: 20ff1efc-a0f9-40b4-a23f-cbccbb966a3d. One of the special magic uuids for 15c2b871-2cdf-4e10-8952-56f88cf44c1c is: 8e32fd20-ac94-4700-828b-6347a43d3cc6.

... One of the special magic uuids for be734970-3d8a-4627-af26-cb42ace1415e is: 2c9266a4-4022-4192-a6a4-ee0e240dffae. One of the special magic uuids for 01387010-8325-4783-8d2d-e4cb366c6b8a is: 147bf656-461e-4f44-882b-3c90c44c8117. One of the special magic uuids for 4ecce990-2042-4893-b59d-bb5574518a2f is: 578dd65f-54bc-4607-b85b-81c21c50ff9e. One of the special magic uuids for 65f85154-df83-4859-b13a-5d893f21360a is: 88da0ce1-53df-4070-9ba6-b2ce99b0b51f. One of the special magic uuids for 48234c9a-7562-4bb8-90b9-286d5689909d is: ed9c52a7-95bf-40b2-b4e9-80e0d8894465. One of the special magic uuids for d58756bb-2c92-414d-a259-24b2075f0f97 is: 60fa95f1-cf1a-460b-9027-4b6a7fbda5de. One of the special magic uuids for 2a1f7077-470d-4b46-a982-c35f03d38568 is: 3d39b5bb-0fba-457b-b3c1-f6a25e71b662. One of the special magic uuids for 38bc4ce7-3680-4804-834c-afdfff2af7ad is: cb79dcb5-9ede-407c-a4f7-53a3a83b9212. One of the special magic uuids for 4a869f03-998b-4dd0-acf7-d36b56830bf8 is: 38e4cbdf-ee50-4b8d-ae68-92435bdcb2a9. One of the special magic uuids for a3d9653a-5278-4aea-9523-1a90d92bba25 is: 23a921a6-cf78-49eb-abef-753189ae9834. One of the special magic uuids for 0bcc9032-39f2-47a3-a30a-3b56867e0a97 is: d400f069-e961-4a72-852f-7be7478f8cad. One of the special magic uuids for 646e6d6b-6f61-4f65-9ae7-e3fd23cb56b8 is: b0ae3b25-e1ba-406f-a7a2-5a162f0ee497. One of the special magic uuids for 714faa88-1734-4175-999f-c68a9e03383e is: 4ba548a8-7c41-46be-9965-befc286e3525. One of the special magic uuids for 238aa954-99a9-4fad-a82f-77e5f386fc84 is: 4d523294-06c4-44a2-a18a-a7c0d447c810. One of the special magic uuids for 8aa499b7-76c9-4825-a842-e7f5164679a4 is: e7b96323-2d44-450e-84e3-a5eded77ed37. One of the special magic uuids for 3da10020-af3b-42b9-9bf3-64121e856ded is: 67f77e75-e65c-4189-ad14-6d3997208a86. One of the special magic uuids for 798f0d6b-35bd-4f08-9aa7-d84c61e839c5 is: 0144aea1-d58e-48b9-bd92-9259dbc2959d. One of the special magic uuids for 204ec480-fc77-4405-b8a6-8341656e5a10 is: 0873cccf-c063-44ed-9ade-15d6d1c4a375. One of the special magic uuids for d6c8cdd4-1f01-4085-b727-a7a5f89f057b is: 8007519f-282a-49a1-961c-4ea63b0a71ed. One of the special magic uuids for c7bd7b63-8819-4b4c-8d54-7b960b8f743d is: 01f169fb-5bb9-4317-9242-43abf007eee0. One of the special magic uuids for 6b012fae-0b8a-43b1-a6b6-99a62f58a848 is: 0c7acf96-95c5-481b-abe2-78a8186050e9. What is the special magic uuid for e8ea1605-ea36-4b09-8695-0cd28545ce6d mentioned in the provided text? The special magic uuid for e8ea1605-ea36-4b09-8695-0cd28545ce6d mentioned in the provided text is

Figure 15: Per-token selection frequency (layer 26, shared with layer 27), averaged over the question queries, overlaid on the prompt text. Darker gray = more frequently selected. The middle distractor block is truncated (“...”) and the intro line is wrapped for layout; truncation does not affect the computation. Red/blue boxes mark the KEY/VALUE spans on the target needle line.

LongCat Sparse Attention

C.3

Attention Weights

Mirroring the analysis above, we next examine the continuous attention weights after sparse selection. Fig. 16 extracts the layer-26 sparse attention heatmap for question queries from the overview. In addition to the high-weight regions over the task description and target needle line already identified by the selection mask, the question span also emerges as a prominent attention region, which was previously hidden by the fixed SWA coverage. Fig. 17 quantifies how well sparse selection preserves dense attention. For each question query, we measure both the overlap between the selected tokens and the dense full-MLA Top-K set, and the coverage of the retained full-attention mass. Despite only moderate Top-K overlap (∼0.56–0.66), the coverage remains consistently high (∼0.95–0.98), indicating that sparse selection captures the few dominant keys that contribute most of the attention mass, even without exactly matching the dense Top-K set. This observation also supports CLI. Within each layer group (L12–L13 and L26–L27), the owner and reuse layers exhibit similar overlap (0.640 vs. 0.665; 0.621 vs. 0.562) and coverage (0.970 vs. 0.980; 0.949 vs. 0.979), demonstrating that a single shared index set can effectively serve both layers. Finally, Fig. 18 visualizes the sparse attention distribution at token granularity by overlaying attention weights on the prompt text, revealing concentrated attention on the task description, question span, and, most prominently, the needle’s KEY and VALUE tokens. task description

target needle line

question

Query position (final question)

0.0008

5900

0.0006

5920 0.0004

5940 0.0002

5960

sparse MLA attn weight

0.0010

5880

0.0000

Key position

1.0

1.00

0.8

0.95

Attn mass coverage

Selection overlap

Figure 16: LSA sparse attention weights for the final-question query tokens (tail rows, 88 tokens) at layer 26. Color is the per-(query, key) attention weight, linearly clipped to [0, 10−3 ] to reveal weak long-range dependencies (the true maximum is ∼0.26). Red brackets above the panel mark the key spans of the task description, the target needle line, and the question. The question queries attend strongly to the task description, the target needle line, and the question span itself.

0.6

0.4

Layer 12 (mean 0.640) Layer 13 (mean 0.665) Layer 26 (mean 0.621) Layer 27 (mean 0.562)

0.2

0.0

5880

5890

5900

5910

0.90

0.85

Layer 12 (mean 0.970) Layer 13 (mean 0.980) Layer 26 (mean 0.949) Layer 27 (mean 0.979)

0.80

5920

5930

5940

5950

0.75

5960

Query position (final question)

5880

5890

5900

5910

5920

5930

5940

5950

5960

Query position (final question)

Figure 17: Sparse selection quality over the question tokens (tail queries) for layers 12/13/26/27. Left: Selection overlap between the indexer selection and the full-MLA Top-2048 within the causal range. Right: coverage, the fraction of dense attention mass retained by the selection. Thin curves are per-query values; horizontal lines are per-layer means. Here Overlap = |St ∩ StMLA |/K with K = 2048 (St the indexer-selected set of Eq. (6) and StMLA the Top-K keys of P the full-MLA row within the causal range), and Coverage = s∈St αt,s with α the head-averaged attention weight.

LongCat Sparse Attention

Figure 18: Per-token LSA sparse attention (layer 26), averaged over the question queries (causal-aware), overlaid on the prompt text. Greener = higher weight (clipped at 10−3 ). The middle distractor block is truncated (“...”) and the intro line is wrapped for layout; truncation does not affect the computation. Red/blue boxes mark the KEY/VALUE spans on the target needle line. Despite attending to only a small fraction of the context, LSA concentrates its attention mass on the needle’s KEY and VALUE, confirming that the sparse indexer reliably locates the relevant tokens.

LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexing

Analyze & explore related research in Conceptio

Related documents