AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methods

[2402.11215] AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methods Skip to main content Search Submit Donate Log in Computer Science > Machine Learning arXiv:2402.11215 (cs) [Submitted on 17 Feb 2024 ( v1 ), last revised 25 Aug 2026 (this version, v4)] Title: AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methods Authors: Tim Tsz-Kit Lau , Han Liu , Mladen Kolar View PDF HTML (experimental) Abstract: The choice of batch size in minibatch stochastic gradient optimization is critical for both optimization and generalization performance in large-scale model training. Although large-batch training is arguably the dominant paradigm in large-scale deep learning because of hardware advances, model generalization often deteriorates relative to small-batch training, leading to the so-called "generalization gap." To mitigate this issue, we investigate adaptive batch size strategies derived from adaptive sampling methods, which were originally developed for stochastic gradient descent. Given the strong interplay between learning rates and batch sizes, together with the prevalence of adaptive gradient methods in deep learning, we emphasize the need for adaptive batch size strategies in these settings. We introduce AdAdaGrad and its scalar variant AdAdaGradNorm, which progressively increase batch sizes during training while performing updates with AdaGrad and AdaGradNorm, respectively. We prove that AdAdaGradNorm converges with high probability at a rate of $\mathscr{O}(1/K)$ to a first-order stationary point of a smooth nonconvex function within $K$ iterations. AdAdaGrad also exhibits similar convergence properties when combined with a novel coordinate-wise variant of our adaptive batch size strategy. We corroborate our theoretical claims with image-classification experiments that highlight the merits of the proposed schemes in terms of both training efficiency and model generalization. Our work highlights the potential of adaptive batch size strategies for adaptive gradient optimizers in large-scale model training. Subjects: Machine Learning (cs.LG) ; Optimization and Control (math.OC); Machine Learning (stat.ML) MSC classes: 90C15, 90C30 Cite as: arXiv:2402.11215 [cs.LG] (or arXiv:2402.11215v4 [cs.LG] for this version) https://doi.org/10.48550/arXiv.2402.11215 Focus to learn more arXiv-issued DOI via DataCite Related DOI : https://doi.org/10.1016/j.slads.2026.100020 Focus to learn more DOI(s) linking to related resources Submission history From: Tim Tsz-Kit Lau [ view email ] [v1] Sat, 17 Feb 2024 07:49:50 UTC (847 KB) [v2] Sat, 25 May 2024 05:57:29 UTC (2,403 KB) [v3] Tue, 28 May 2024 05:40:38 UTC (2,403 KB) [v4] Tue, 25 Aug 2026 02:40:05 UTC (3,716 KB) Full-text links: Access Paper: View PDF HTML (experimental) TeX Source view license Current browse context: cs.LG < prev | next > new | recent | 2024-02 Change to browse by: cs math math.OC stat stat.ML References & Citations NASA ADS Google Scholar Semantic Scholar export BibTeX citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer ( What is the Explorer? ) Connected Papers Toggle Connected Papers ( What is Connected Papers? ) Litmaps Toggle Litmaps ( What is Litmaps? ) scite.ai Toggle scite Smart Citations ( What are Smart Citations? ) Code, Data, Media Code, Data and Media Associated with this Article alphaXiv Toggle alphaXiv ( What is alphaXiv? ) Links to Code Toggle CatalyzeX Code Finder for Papers ( What is CatalyzeX? ) DagsHub Toggle DagsHub ( What is DagsHub? ) GotitPub Toggle Gotit.pub ( What is GotitPub? ) Huggingface Toggle Hugging Face ( What is Huggingface? ) ScienceCast Toggle ScienceCast ( What is ScienceCast? ) Demos Demos Replicate Toggle Replicate ( What is Replicate? ) Spaces Toggle Hugging Face Spaces ( What is Spaces? ) Spaces Toggle TXYZ.AI ( What is TXYZ.AI? ) Related Papers Recommenders and Search Tools Link to Influence Flower Influence Flower ( What are Influence Flowers? ) Core recommender toggle CORE Recommender ( What is CORE? ) IArxiv recommender toggle IArxiv Recommender ( What is IArxiv? ) Author Venue Institution Topic About arXivLabs arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs . Which authors of this paper are endorsers? | Disable MathJax ( What is MathJax? ) We gratefully acknowledge support from our major funders , member institutions , and all contributors. About · Help · Contact · Subscribe · Copyright · Privacy · Accessibility · Operational Status (opens in new tab) Major funding support from

Related documents

• AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methods#670271
• Long-Run Dynamics of AdaGrad-Norm: Trajectory Stability and Asymptotic Stationarity#793541
• Long-Run Dynamics of AdaGrad-Norm: Trajectory Stability and Asymptotic Stationarity#789537
• Can Adaptive Gradient Methods Converge under Heavy-Tailed Noise? A Case Study of AdaGrad#200482
• Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Staging#1142386