A Complete Decomposition of KL Error using Refined Information and Mode Interaction Selection

[2410.11964] A Complete Decomposition of KL Error using Refined Information and Mode Interaction Selection Skip to main content Search Submit Donate Log in Search arXiv Press Enter to search · Advanced search Computer Science > Machine Learning arXiv:2410.11964 (cs) [Submitted on 15 Oct 2024 ( v1 ), last revised 11 Apr 2026 (this version, v2)] Title: A Complete Decomposition of KL Error using Refined Information and Mode Interaction Selection Authors: James Enouen , Mahito Sugiyama View a PDF of the paper titled A Complete Decomposition of KL Error using Refined Information and Mode Interaction Selection, by James Enouen and Mahito Sugiyama View PDF HTML (experimental) Abstract: The log-linear model has received a significant amount of theoretical attention in previous decades and remains the fundamental tool used for learning probability distributions over discrete variables. Despite its large popularity in statistical mechanics and high-dimensional statistics, the majority of related energy-based models only focus on the two-variable relationships, such as Boltzmann machines and Markov graphical models. Although these approaches have easier-to-solve structure learning problems and easier-to-optimize parametric distributions, they often ignore the rich structure which exists in the higher-order interactions between different variables. Using more recent tools from the field of information geometry, we revisit the classical formulation of the log-linear model with a focus on higher-order mode interactions, going beyond the 1-body modes of independent distributions and the 2-body modes of Boltzmann distributions. This perspective allows us to define a complete decomposition of the KL error. This then motivates the formulation of a sparse selection problem over the set of possible mode interactions. In the same way as sparse graph selection allows for better generalization, we find that our learned distributions are able to more efficiently use the finite amount of data which is available in practice. We develop an algorithm called MAHGenTa which leverages a novel Monte-Carlo sampling technique for energy-based models alongside a greedy heuristic for incorporating statistical robustness. On both synthetic and real-world datasets, we demonstrate our algorithm's effectiveness in maximizing the log-likelihood for the generative task and also the ease of adaptability to the discriminative task of classification. Subjects: Machine Learning (cs.LG) ; Machine Learning (stat.ML) Cite as: arXiv:2410.11964 [cs.LG] (or arXiv:2410.11964v2 [cs.LG] for this version) https://doi.org/10.48550/arXiv.2410.11964 Focus to learn more arXiv-issued DOI via DataCite Submission history From: James Enouen [ view email ] [v1] Tue, 15 Oct 2024 18:08:32 UTC (616 KB) [v2] Sat, 11 Apr 2026 02:29:42 UTC (514 KB) Full-text links: Access Paper: View a PDF of the paper titled A Complete Decomposition of KL Error using Refined Information and Mode Interaction Selection, by James Enouen and Mahito Sugiyama View PDF HTML (experimental) TeX Source view license Current browse context: cs.LG < prev | next > new | recent | 2024-10 Change to browse by: cs stat stat.ML References & Citations NASA ADS Google Scholar Semantic Scholar export BibTeX citation Loading... BibTeX formatted citation × loading... Data provided by: Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer ( What is the Explorer? ) Connected Papers Toggle Connected Papers ( What is Connected Papers? ) Litmaps Toggle Litmaps ( What is Litmaps? ) scite.ai Toggle scite Smart Citations ( What are Smart Citations? ) Code, Data, Media Code, Data and Media Associated with this Article alphaXiv Toggle alphaXiv ( What is alphaXiv? ) Links to Code Toggle CatalyzeX Code Finder for Papers ( What is CatalyzeX? ) DagsHub Toggle DagsHub ( What is DagsHub? ) GotitPub Toggle Gotit.pub ( What is GotitPub? ) Huggingface Toggle Hugging Face ( What is Huggingface? ) ScienceCast Toggle ScienceCast ( What is ScienceCast? ) Demos Demos Replicate Toggle Replicate ( What is Replicate? ) Spaces Toggle Hugging Face Spaces ( What is Spaces? ) Spaces Toggle TXYZ.AI ( What is TXYZ.AI? ) Related Papers Recommenders and Search Tools Link to Influence Flower Influence Flower ( What are Influence Flowers? ) Core recommender toggle CORE Recommender ( What is CORE? ) IArxiv recommender toggle IArxiv Recommender ( What is IArxiv? ) Author Venue Institution Topic About arXivLabs arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs . Which authors of this paper are endorsers? | Disable MathJax ( What is MathJax? ) We gratefully acknowledge support from our major funders , member institutions , , and all contributors. About · Help · Contact · Subscribe · Copyright · Privacy · Accessibility · Operational Status (opens in new tab) Major funding support from

Related documents

• A polynomial approximation scheme for nonlinear model reduction by moment matching#1036535
• New directions in dynamical expectation estimation#1013732
• Bayesian Analysis and Model Revision for k’th Order Markov Chains with Unknown k.#486203
• Uncertainty quantification for inverse problems#280334
• Non-intrusive Nonlinear Model Reduction via Machine Learning Approximations to Low-dimensional Operators#1145817