Official Review · ICLR 2026

This paper introduces CAB (Cross-architecture distillation via Attention Bridge), a framework for distilling knowledge from Transformer-based teacher models to state-space model (SSM) students, specifically Mamba. The core innovation is a lightweight, MLP-based "Attention Bridge" that aligns the implicit attention carriers in Mamba (the token-dependent projections $B$ and $C$) with the explicit key ($K$) and query ($Q$) representations from the Transformer. After the distillation process is complete, the MLP weights are discarded. The final product is a standalone Mamba model that has internalized the attention-based inductive biases from the Transformer teacher, with no inference-time overhead from the bridge components. The distillation process does not freeze the Mamba weights; instead, the gradients from the attention alignment loss and the standard output distillation loss are backpropagated through the MLP bridge and into all weights of the student model. This method enables fine-grained, token-level supervision without the quadratic memory cost of aligning full attention matrices. It also employs a flexible, proportional layer-wise alignment strategy to handle architectural depth mismatches. The authors demonstrate CAB's effectiveness and data efficiency through extensive experiments on both vision (ImageNet) and language (OpenWebText, C4, WikiText) tasks, showing consistent improvements over standard and recent cross-architecture distillation baselines, particularly in low-data regimes. * The formulation of the "Attention Bridge" is a creative and novel solution to the cross-architecture distillation problem. The insight to treat Mamba's $B$ and $C$ projections as analogous to the Transformer's $K$ and $Q$ under a linear attention approximation is theoretically grounded and is a natural extension of State-Space Duality. * The experimental evaluation is thorough, spanning multiple domains, model scales, and data regimes. The consistent and often substantial improvements over strong baselines is very convincing. The ablation studies are well-designed and provide crucial insights into the contribution of each component. * The paper is well-written. The problem is motivated, the method is explained with sufficient detail, and the results are presented logically and convincingly. * This work has high practical significance. By enabling data-efficient knowledge transfer from Transformers to Mamba, it lowers the barrier to adopting high-performance SSMs in real-world scenarios where data is limited and computational resources are constrained. * The authors use a 2-layer MLP with SiLU activations for the bridge modules ($\phi_B$) and ($\phi_C$). While effective, the paper does not ablate this choice against a simpler linear projection. A brief justification or experimental result showing the superiority of a non-linear transformation over a linear one would strengthen this design decision, especially given the focus on efficiency. * T