Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Models

[2605.26895] Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Models Skip to main content Search Submit Donate Log in Computer Science > Machine Learning arXiv:2605.26895 (cs) [Submitted on 26 May 2026 ( v1 ), last revised 28 Aug 2026 (this version, v2)] Title: Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Models Authors: Mingze Wang , Shuchen Zhu , Yuxin Fang , Binghui Li , Kai Shen , Shu Zhong View PDF Abstract: Normalization layers in modern large language models (LLMs) consist of a deterministic normalization operation and a learnable scale vector. While the normalization operation has been extensively studied, the scale vector remains poorly understood despite its ubiquitous use. In this work, we present a systematic study of scale vectors in LLMs from the perspectives of expressivity, optimization, and architectural structure. First, we show empirically that although scale vectors constitute only a negligible fraction of model parameters, removing them substantially degrades LLM pre-training. Our theory further shows that, in Pre-Norm architectures, scale vectors do not increase expressivity; instead, they improve optimization through a self-amplifying preconditioning effect on subsequent linear mappings. Second, we investigate the role of weight decay for scale vectors. By distinguishing Input-Norm and Output-Norm layers, we theoretically show that weight decay is beneficial for the former but harmful for the latter, due to their distinct roles in optimization and expressivity. Third, motivated by this understanding, we propose three lightweight and complementary improvements to scale vectors: branch-specific heterogeneity, improved placement around linear mappings, and magnitude-direction reparameterization. Both theory and experiments show that each improvement yields consistent gains. Finally, we combine these improvements into a unified scale-vector strategy and evaluate it through extensive LLM pre-training experiments on dense and mixture-of-experts models ranging from 0.12B to 2B parameters, across multiple optimizers and learning rate schedules, under industrial-scale token budgets. The unified strategy consistently achieves lower terminal loss than well-tuned baselines and exhibits more favorable scaling behavior, while adding negligible parameter and computational overhead. Comments: 36 pages Subjects: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Machine Learning (stat.ML) Cite as: arXiv:2605.26895 [cs.LG] (or arXiv:2605.26895v2 [cs.LG] for this version) https://doi.org/10.48550/arXiv.2605.26895 Focus to learn more arXiv-issued DOI via DataCite Submission history From: Mingze Wang [ view email ] [v1] Tue, 26 May 2026 11:56:49 UTC (10,403 KB) [v2] Fri, 28 Aug 2026 05:14:58 UTC (10,404 KB) Full-text links: Access Paper: View PDF TeX Source view license Additional Features Audio Summary Current browse context: cs.LG < prev | next > new | recent | 2026-05 Change to browse by: cs cs.AI stat stat.ML References & Citations SciX Google Scholar Semantic Scholar export BibTeX citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer ( What is the Explorer? ) Connected Papers Toggle Connected Papers ( What is Connected Papers? ) Litmaps Toggle Litmaps ( What is Litmaps? ) scite.ai Toggle scite Smart Citations ( What are Smart Citations? ) Code, Data, Media Code, Data and Media Associated with this Article alphaXiv Toggle alphaXiv ( What is alphaXiv? ) Links to Code Toggle CatalyzeX Code Finder for Papers ( What is CatalyzeX? ) DagsHub Toggle DagsHub ( What is DagsHub? ) GotitPub Toggle Gotit.pub ( What is GotitPub? ) Huggingface Toggle Hugging Face ( What is Huggingface? ) ScienceCast Toggle ScienceCast ( What is ScienceCast? ) Demos Demos Replicate Toggle Replicate ( What is Replicate? ) Spaces Toggle Hugging Face Spaces ( What is Spaces? ) Spaces Toggle TXYZ.AI ( What is TXYZ.AI? ) Related Papers Recommenders and Search Tools Link to Influence Flower Influence Flower ( What are Influence Flowers? ) Core recommender toggle CORE Recommender ( What is CORE? ) IArxiv recommender toggle IArxiv Recommender ( What is IArxiv? ) Author Venue Institution Topic About arXivLabs arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs . Which authors of this paper are endorsers? | Disable MathJax ( What is MathJax? ) We gratefully acknowledge support from our major funders , member institutions , and all contributors. About · Help · Contact · Subscribe · Copyright · Privacy · Accessibility · Operational Status (opens in new tab)

Related documents

• Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Models#682694
• Large Language Models: A Mathematical Formulation#629914
• Large Language Models: A Mathematical Formulation#628052
• Exploring the role of large language models in the scientific method: from hypothesis to discovery#627385
• "Small" Large Language Models in the Hospital: Evaluation Study on Real-World Data in a Resource-Constrained Setting.#1091178