Official Review · NEURIPS 2023

State space models (SSMs) perform well on modeling long-range dependencies with good efficiency scaling, but on language modeling, transformers still outperforms SSMs. This paper tries to combine the best of both worlds and proposes a hybrid model, Block-State Transformer, which combine SSMs’ capacity on long range modeling and Transformer’s ability on modeling local context. The input sequence is split to multiple smaller segments. For each segment, transformer layer will do a self attention on this token embeddings and cross attention to the output of SSMs. Their experiments show that on language modeling, their approach achieve reasonable speedup with comparable performance to Transformers. 1. The proposed combination of SSMs and transformers allow the model to exploit advantages of two powerful methods while avoiding their drawbacks. 2. The SSMs used in the proposed method can be swapped to different SSMs making it possible to enjoy the advancement on SSMs field. 3. The proposed method give similar performance on language modeling compared to Transformers. 1. There was already existing work on SSMs that achieve similar performance on language modeling (https://arxiv.org/abs/2212.14052) compared to Transformers. The authors should include a discussion and comparison to the relevant work. 2. The evaluations are performed on language modeling for 4096 length sequences. On this setting, there are a lot of strong transformer baselines with efficient self-attention designs. It would be good if the authors can provide an empirical comparison with these baselines. 3. Perplexity is only one indicator of how the language models performance. To get more precise understanding of performance, it would be good to include a comparison on downstream tasks. 4. Code is not available. The main concerns are listed above. There are two more questions: 1. On line 235, the 6,966,499 English language words seems to be a typo according to dataset statistics on https://github.com/deepmind/pg19. 2. What are the latency of one forward step and what about memory consumption?