Official Review · NEURIPS 2023

The paper introduces Mnemosyne, a new learnable optimizer that can train neural networks without task-specific tuning. The authors claim that it employs minimal resources and handles larger parameter sets, outperforming LSTM optimizers and matching state-of-the-art ones. Its efficacy is demonstrated in the fine-tuning of Vision Transformers, BERT models, and large T5XXL models. In addition, it also provides the theoretical analysis of the compact associative memory used by the proposed Mnemosyne. （1）The proposed Mnemosyne introduces a novel approach by utilizing spatio-temporal low-rank implicit attention Transformers to create learnable optimizers. This innovative idea goes beyond traditional optimizers by eliminating the need for task-specific tuning in training neural networks. The idea seems interesting. （2）Mnemosyne has demonstrated superior performance in various use cases, such as fine-tuning Vision Transformers, pre-training BERT models, and soft prompt-tuning large T5XXL models. This shows its broad applicability and robustness (1) The method's effectiveness on broader model training remains unknown as it only trains or fine-tunes a subset of Transformer layers. Moreover, given the Transformer's huge data requirements, its performance on larger datasets is also unclear. (2) As the method only trains or fine-tunes some layers of certain Transformer architectures, its title's emphasis on training Transformers seems somewhat overstated. See the weakness.