Official Review · ICLR 2026

The paper proposes Diversity-Guided MLP Pruning (DGMP) for post-training pruning of vision transformers. The method prunes hidden neurons of MLP layers in transformers iteratively by ranking the neuron weight norms, while additionally applying Gram-Schmidt iteration to preserve the independent component of remaining neurons. The method is followed by a knowledge distillation stage to recover the performance. Experiments on several state-of-the-art vision transformersshow that the proposed method achieves more than 57% parameter and FLOPs reductions in a lossless manner. 1. The manuscript is easy to follow and well-organized. 2. The work shows solid results on vision transformers. 1. **Limited comparison with strong structured-pruning baselines.** The main concern of the paper is that there are many recent strong structured pruning baselines for general transformer pruning that are highly relevant and should be discussed or compared in dicussion or experiments, for example: - Ma, Xinyin, Gongfan Fang, and Xinchao Wang. "Llm-pruner: On the structural pruning of large language models." Advances in neural information processing systems 36 (2023): 21702-21720. - Wang, Yuxin, et al. "CFSP: An Efficient Structured Pruning Framework for LLMs with Coarse-to-Fine Activation Information." Proceedings of the 31st International Conference on Computational Linguistics. 2025. - Li, Guangyan, Yongqiang Tang, and Wensheng Zhang. "LoRAP: Transformer Sub-Layers Deserve Differentiated Structured Compression for Large Language Models." Forty-first International Conference on Machine Learning. - Ling, Gui, Ziyang Wang, and Qingwen Liu. "Slimgpt: Layer-wise structured pruning for large language models." Advances in Neural Information Processing Systems 37 (2024): 107112-107137. - An, Yongqi, et al. "Fluctuation-based adaptive structured pruning for large language models." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 10. 2024. To validate the performance of the proposed method, some of these methods should be included. 2. **Limited experients beyond vision-only cases.** Another concern is that the method is specifically designed and only tested on vision transformer. Since structured pruning is a general architecture-level technique, one would expect DGMP to also benefit other transformer modalities, such as pure text-based transformers. Even a small experiment on text-based LLM would significantly improve the paper’s general relevance and demonstrate that the proposed method is a universal principle rather than a ViT-specific heuristic. Can the authors provide detailed experiment/discussion comparison with the mentioned structured-pruning baselines?