Official Review · ICLR 2025

The paper introduces BDIA-transformers, a novel approach to reversible transformers designed to reduce memory consumption during training without altering the architecture during inference. The approach leverages the Bidirectional Integration Approximation (BDIA) technique, which treats each transformer block as an Euler integration approximation for solving ordinary differential equations (ODEs). The experimental results show that BDIA-transformers outperform standard transformers in image classification and language translation tasks while reducing training memory requirements. For text prediction, BDIA-GPT2 prevents overfitting when trained on small datasets. - The paper addresses the “memory wall” challenge, a critical issue in deep learning in which training large models requires extensive memory to store intermediate activations alongside the model itself. Authors explore reversible deep learning to tackle this issue. Reversible architectures allow backpropagation without storing or with minimal storage of intermediate activations, thus offering significant memory savings. The authors propose a novel technique by treating each transformer block as an Ordinary Differential Equation (ODE) solver and applying the Bidirectional Integration Approximation (BDIA) to these blocks. - Experimental results show that BDIA-transformers improve performance across various tasks, including image classification and language translation. Authors claim that performance gains and memory efficiency are gained by implementing this method on Vision Transformers. - This approach has the potential to enable the training of larger, more complex models when memory is the bottleneck, but we have very strong computational power. - The paper is hard to follow, especially in the preliminary and method sections. It doesn’t clearly explain key ideas, like why transformers can be viewed as ODE solvers or how BDIA transformation is applied, which makes it confusing. - The related work section mainly lists past studies without explaining how the field has developed or why this approach is needed. This makes it hard to see where this paper fits in with previous work. - Some arguments supporting the method are vague. For instance, in the subsection "on similarity to dropout technique". Overall, a clearer, simpler structure and explanations would make the paper easier to understand, especially for readers unfamiliar with reversible deep learning. - The paper attributes BDIA-transformer's improved performance to the regularization effect of random variables, which is said to work similarly to dropout. However, it’s not clear if this is the only reason for the performance gains. The authors haven’t compared BDIA against other standard regularization methods on the baseline transformer, leaving open the chance that similar improvements could be achieved without the added complexity of BDIA. - For language models, benchmarks and perplexity provide more informative insigh