Online Covariance Matrix Estimation in Sketched Newton Methods

[2502.07114] Online Covariance Matrix Estimation in Sketched Newton Methods Skip to main content Search Submit Donate Log in Search arXiv Press Enter to search · Advanced search Statistics > Machine Learning arXiv:2502.07114 (stat) [Submitted on 10 Feb 2025 ( v1 ), last revised 11 Apr 2026 (this version, v2)] Title: Online Covariance Matrix Estimation in Sketched Newton Methods Authors: Wei Kuang , Mihai Anitescu , Sen Na View a PDF of the paper titled Online Covariance Matrix Estimation in Sketched Newton Methods, by Wei Kuang and 2 other authors View PDF HTML (experimental) Abstract: Given the ubiquity of streaming data, online algorithms have been widely used for parameter estimation, with second-order methods particularly standing out for their efficiency and robustness. In this paper, we study an online sketched Newton method that leverages a randomized sketching technique to perform an approximate Newton step in each iteration, thereby eliminating the computational bottleneck of second-order methods. While existing studies have established the asymptotic normality of sketched Newton methods, a consistent estimator of the limiting covariance matrix remains an open problem. We propose a fully online covariance matrix estimator that is constructed entirely from the Newton iterates and requires no matrix factorization. Compared to covariance estimators for first-order online methods, our estimator for second-order methods is batch-free. We establish the consistency and convergence rate of our estimator, and coupled with asymptotic normality results, we can then perform online statistical inference for the model parameters based on sketched Newton methods. We also discuss the extension of our estimator to constrained problems, and demonstrate its superior performance on regression problems as well as benchmark problems in the CUTEst set. Comments: 63 pages, 4 figures, 9 tables Subjects: Machine Learning (stat.ML) ; Machine Learning (cs.LG); Numerical Analysis (math.NA); Optimization and Control (math.OC); Computation (stat.CO) Cite as: arXiv:2502.07114 [stat.ML] (or arXiv:2502.07114v2 [stat.ML] for this version) https://doi.org/10.48550/arXiv.2502.07114 Focus to learn more arXiv-issued DOI via DataCite Submission history From: Wei Kuang [ view email ] [v1] Mon, 10 Feb 2025 23:11:02 UTC (1,734 KB) [v2] Sat, 11 Apr 2026 04:45:15 UTC (8,745 KB) Full-text links: Access Paper: View a PDF of the paper titled Online Covariance Matrix Estimation in Sketched Newton Methods, by Wei Kuang and 2 other authors View PDF HTML (experimental) TeX Source view license Current browse context: stat.ML < prev | next > new | recent | 2025-02 Change to browse by: cs cs.LG cs.NA math math.NA math.OC stat stat.CO References & Citations NASA ADS Google Scholar Semantic Scholar export BibTeX citation Loading... BibTeX formatted citation × loading... Data provided by: Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer ( What is the Explorer? ) Connected Papers Toggle Connected Papers ( What is Connected Papers? ) Litmaps Toggle Litmaps ( What is Litmaps? ) scite.ai Toggle scite Smart Citations ( What are Smart Citations? ) Code, Data, Media Code, Data and Media Associated with this Article alphaXiv Toggle alphaXiv ( What is alphaXiv? ) Links to Code Toggle CatalyzeX Code Finder for Papers ( What is CatalyzeX? ) DagsHub Toggle DagsHub ( What is DagsHub? ) GotitPub Toggle Gotit.pub ( What is GotitPub? ) Huggingface Toggle Hugging Face ( What is Huggingface? ) ScienceCast Toggle ScienceCast ( What is ScienceCast? ) Demos Demos Replicate Toggle Replicate ( What is Replicate? ) Spaces Toggle Hugging Face Spaces ( What is Spaces? ) Spaces Toggle TXYZ.AI ( What is TXYZ.AI? ) Related Papers Recommenders and Search Tools Link to Influence Flower Influence Flower ( What are Influence Flowers? ) Core recommender toggle CORE Recommender ( What is CORE? ) Author Venue Institution Topic About arXivLabs arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs . Which authors of this paper are endorsers? | Disable MathJax ( What is MathJax? ) We gratefully acknowledge support from our major funders , member institutions , , and all contributors. About · Help · Contact · Subscribe · Copyright · Privacy · Accessibility · Operational Status (opens in new tab) Major funding support from

Related documents

• Inference for Newton Methods with Accelerated Sketch-and-Project via Random Scaling#1000371
• Global Linear Convergence of the Proximal Bundle Method under Unknown Piecewise Smoothness and Quadratic Growth#999009
• Relative Affine Structure: Canonical Model for 3D from 2D Geometry and Applications#239463
• Lipschitz Extension Initialization for Moving Least Squares Reconstruction from Sparse Irregular Samples#925301
• Lipschitz Extension Initialization for Moving Least Squares Reconstruction from Sparse Irregular Samples#921065