Official Review · NEURIPS 2023

This paper proposes block-state transformers, a method to combine state space models with transformers for language modeling. The paper evaluates block-state transformers on PG19 and arxiv math and finds promising results. Combining state space models and Transformers is an interesting idea worth exploring. The presentation of the paper is clear. The explanation of state space models, which can be quite complex, is very clear. The evaluation hits the right notes in terms of the major questions to ask. The evaluation is missing many recent works combining state space models and attention in various ways. The claim that SSMs do not match Transformers on language has not been true for a while. Most of these methods were released significantly before the NeurIPS deadline and are critical to compare against for evaluation. * Mega [1] combines attention and state space models. * BiGS [2] is a new SSM-based architecture that matches Transformers in language. * H3 [3] combines SSMs and attention in alternate layers. * Hyena [4] removes attention completely and replaces it with a convolution-based layer (similar to an SSM). Confusingly, many of these works are cited in the paper - and ideas from the papers are used extensively in the methods proposed (e.g., "BST:{SH,MF}" uses the structure from H3 and Hyena without comparing against those architectures as baselines). Using the ideas from these papers without comparing against them makes it difficult to understand how this method compares against previous methods and where the performance improvement comes from. Performance is also hard to evaluate compared to standard models such as Transformers (GPT-Neo [5], Pythia suite [6]). TransformerXL is an older model that is not trained as well as modern Transformer-based LLMs. [1] https://arxiv.org/abs/2209.10655 [2] https://arxiv.org/abs/2212.10544 [3] https://arxiv.org/abs/2212.14052 [4] https://arxiv.org/abs/2302.10866 [5] https://github.com/EleutherAI/gpt-neo [6] https://github.com/EleutherAI/pythia How does BST compare to the architectures listed in the weaknesses section? It is important to compare against the original architectures that inspired components of BST, as well as modern standard Transformers.