Stack Exchange
open access
CC BY-SA
Training on the full dataset after cross-validation?
TL:DR: Is it ever a good idea to train an ML model on all the data available before shipping it to production? Put another way, is it ever ok to train on all data available and not check if the model overfits, or get a final read of the expected performance of the model? Say I have a family of models parametrized by $\alpha$. I can do a search (e.g. a grid search) on $\alpha$ by, for example, running k-fold cross-validation for each candidate. The point of using cross-validation for choosing $\alpha$ is that I can check if a learned model $\beta_i$ for that particular $\alpha_i$ had e.g. overfit, by testing it on the "unseen data" in each CV iteration (a validation set). After iterating through all $\alpha_i$ 's, I could then choose a model $\beta_{\alpha^*}$ learned for the parameters $\alpha^*$ that seemed to do best on the grid search, e.g. on average across all folds. Now, say that after model selection I would like to use all the the data that I have available in an attempt to ship the best possible model in production. For this, I could use the parameters $\alpha^*$ that I chose via grid search with cross-validation, and then, after training the model on the full ( $F$ ) dataset, I would a get a single new learned model $\beta^{F}_{\alpha^*}$ The problem is that, if I use my entire dataset for training, I can't reliably check if this new learned model $\beta^{F}_{\alpha^*}$ overfits or how it may perform on unseen data. So is this at all good practice? What is a good way to think about this problem?
Conceptio Open Archive
Analyze & explore related research in Conceptio
Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.
This document is indexed with metadata only: full text is not available in the archive for this record. Open the official source ↗
Related documents
- SCLATE: a Substrate for Continual-Learning Agent Training and Evaluation#1116722
- Measuring Structured Predictability in Neural Training Dynamics: A Cross-Regime Study#616645
- Measuring Structured Predictability in Neural Training Dynamics: A Cross-Regime Study#616145
- Reliable Neural Collapse Approximation for Open-World Test-Time Adaptation#471261
- Human Induction in Machine Learning: A Survey of the Nexus#769694
Record · ID 787302
Retrieved via Conceptio: every document is proof-bundled with source, license, and retrieval metadata.
Retrieved via Conceptio: every document is proof-bundled with source, license, and retrieval metadata.
Bring this data into your agent: Connect via MCP or API · 20 free credits to start · no card required
Read the Docs → · View CLI source on GitHub · Conceptio Open Knowledge Archive
Read the Docs → · View CLI source on GitHub · Conceptio Open Knowledge Archive