arXiv CSopen access

Inductive Venn-Abers and related regressors

Inductive Venn–Abers and related regressors

arXiv:2605.06646v1 [cs.LG] 7 May 2026

Ivan Petej and Vladimir Vovk May 8, 2026 Abstract Venn–Abers predictors are probabilistic predictors that enjoy appealing properties of validity, but their major limitation is that they are applicable only to the case of binary classification, with a recent extension to bounded regression. We generalize them to the case of unbounded regression, which requires adding an element of conformal prediction. In our simulation and empirical studies we investigate the predictive efficiency of point regressors derived from Venn–Abers regressors and argue that they somewhat improve the predictive efficiency of standard regressors for larger training sets.

1

Introduction

This paper explores the problem of regression estimation as presented in, e.g., Vapnik (1998, Sect. 1.4). In the standard setting of statistical learning, where we observe an IID sequence of random pairs (X, Y ) consisting of objects X and their labels Y ∈ R, the (ideal) regression estimator (of Y given X) maps each object x to the expected value E(Y | X = x) of its label Y conditional on observing x. Vapnik lists regression estimation as one of three basic statistical problems (Vapnik, 1998, Sect. 1.2). We develop ideas of conformal prediction (Angelopoulos et al., 2026; Vovk et al., 2022) and introduce an algorithm for regression estimation that satisfies a natural distribution-free notion of validity. We are interested in the distribution-free setting, when nothing is known about the distribution generating one pair (X, Y ) and we are only given a training sequence of labelled objects and an unlabelled test object. In typical cases we cannot hope to find the true regression estimate E(Y | X = x) (even when it is well-defined), and so we lower the bar in three respects in our definition of a valid regression estimator. First, we allow estimators of the form E(Y | F) for some σ-algebra F; we will express this by saying that our regression estimator is auto-calibrated. Ideally, F should be close to the σ-algebra generated by the training set and test object. Second, we allow our algorithms to output intervals (ideally short “imprecise regression estimates”) containing E(Y | F). And third, we replace E(Y | F) by E(Y ′ | F ), where Y ′ is a “regularized” version of Y , but Y ′ = Y with high probability. (This is the element of conformal prediction

1

that we mentioned in the abstract; it is required in the absence of bounds on the test label.) As usual, we consider two principal requirements for our algorithms, validity (technically, auto-calibration) and efficiency. Validity (in this technical sense) will be guaranteed for our algorithms, but to achieve efficiency we will use existing regression algorithms that we believe to be efficient, although perhaps miscalibrated. We will develop methods for improving their calibration. These methods will be adaptations of the methods used in Vovk et al. (2015) (and presented in a much greater detail in Vovk et al. 2022, Chap. 6) in the context of binary classification, with bounded regression considered earlier by van der Laan and Alaa (2024). The problem of regression estimation considered in this paper is very different from the regression problems in conformal prediction, where the task is to output a prediction region (typically a prediction interval) for a test label with a pre-specified coverage probability. In regression estimation the task is to cover the expected label E(Y | F ) rather than the label Y itself. This will allow us to produce much shorter intervals (especially that we replace Y by Y ′ that is not guaranteed to coincide with Y ). The goal of the computational experiments reported in this paper is to demonstrate that application of our methods leads to an improvement in the performance of standard point regressors (we consider those implemented in scikit-learn). The improvement is not as significant as we had hoped and disappears for smaller datasets.

2

Comparisons with Literature

Results of this paper are not directly comparable with the existing literature because of a somewhat unusual notion of validity that we use. But there are several related approaches. A strand of research that is closest to what we do in this paper is conformal regression, already mentioned in the previous section. Important advances in conformal regression include Romano et al. (2019), offering very flexible methods based on quantile regression, and Gibbs et al. (2025), establishing conditional validity results. See, e.g., Angelopoulos et al. (2026) for a recent review of conformal prediction in general and conformal regression in particular. The goal of conformal regression is to produce provably valid, under the assumption of IID data, prediction intervals for the label of a test object. Since validity here means a guaranteed coverage probability, this is a much more ambitious problem than regression estimation dealt with in this paper, as mentioned earlier. Conformal predictive distributions (Vovk et al., 2022, Chap. 7) are different from conformal regression in that, instead of prediction intervals, they output full predictive distributions for future labels (assumed to be real-valued, as in conformal regression). To achieve validity these predictive distributions should be imprecise in a certain sense. Our current task is easier in that we only aim to

2

cover E(Y ′ | F ), but we will achieve another property of validity, namely autocalibration instead of calibration in probability (which is stronger in the binary case but not comparable in general). See Allen et al. (2025) for a recent analysis of achievable properties of validity for conformal predictive distributions. The property of validity used in this paper is inherited from Venn prediction, which is, however, typically applied to produce imprecise probability forecasts in classification problems. Venn prediction (including Venn–Abers prediction) is reviewed in Vovk et al. (2022, Chap. 6) and Venn–Abers prediction is reviewed in Angelopoulos et al. (2026, Sect. 12.4). Venn–Abers predictors were extended to the case of bounded regression by van der Laan and Alaa (2024), as mentioned earlier. Our bounded Venn–Abers regressor in the next section is just an inductive version of van der Laan and Alaa (2024, Algorithm 1). As the next step, van der Laan and Alaa develop an algorithm (van der Laan and Alaa, 2024, Algorithm 2) for “self-calibrating conformal prediction” based on their Algorithm 1. Finally, van der Laan and Alaa (2025) extend these results to general loss functions interpreting the usual regression setting as the case of squared error loss. A key step in our algorithm is replacing the true labels Y by their regularized versions Y ′ , as discussed in the previous section. We need it to avoid the vacuous regression interval (−∞, ∞) (see the end of Sect. 3). Assumptions made by van der Laan and Alaa (2024) and van der Laan and Alaa (2025) allow them to bypass this step, but we avoid making any assumptions on the data-generating distribution apart from the observations being IID. In general, the properties of validity of our procedures are sufficiently different from the properties of validity considered in literature to make direct comparison between them in empirical or simulation studies difficult or impossible. Therefore, in our experimental section (Sect. 8) we concentrate on evaluating the predictive efficiency of point regressors derived from imprecise regressors implementing our methods; namely, we compare the predictive performance of those point regressors with that of the base algorithms.

3

Inductive Venn–Abers Regressors

In this section we ignore the computational complexity of our methods (it will be the topic of Sect. 5). Fix a measurable space X (the object space) and set Z := X × R (this is the example space). Each example z = (x, y) ∈ Z consists of an object x ∈ X and its real-valued label y. We consider two settings (leading to formally different, albeit similar, prediction algorithms). In the setting of bounded regression we are given a finite interval [C∗ , C ∗ ] ⊆ R guaranteed to contain all labels, training and test. Otherwise, we have unbounded regression. In the former case, the algorithm and its property of validity are simpler, and we consider them separately even though we are not particularly interested in them per se as we would like to avoid any assumptions apart from IID observations. Notice that an algorithm for unbounded regression can also be applied in

3

the bounded setting, and it may well be preferable if the bounds C∗ and C ∗ are loose. This is another reason why our main interest is in unbounded regression.

3.1

Bounded regression

Fix any regression algorithm (such as a neural net) producing point predictions as the base algorithm. We are given a training set consisting of l > k examples (where k ∈ N := {1, 2, . . . } is typically large and l − k ∈ N is also large) and a test object x ∈ X; k is a parameter of our algorithm. In bounded regression, we are given an interval [C∗ , C ∗ ] guaranteed to contain all labels. The corresponding bounded inductive Venn–Abers regressor (bounded IVAR) produces the regression interval [ŷ∗ , ŷ ∗ ] := [f∗ (r), f ∗ (r)]

(1)

for the test label, where f ∗ , f∗ , and r are defined as follows: 1. Randomly split the training set of size l > k into two parts, a proper training set of size l − k and a calibration set z1 , . . . , zk of size k; for each i ∈ {1, . . . , k}, zi = (xi , yi ) consists of an object xi and its label yi . 2. Train the regression algorithm on the proper training set obtaining a prediction rule R : X → R (a measurable function) mapping objects to their predicted labels. 3. Find the prediction ri := R(xi ) (base prediction) for the calibration object xi , i = 1, . . . , k, and the base prediction r := R(x) for the test object x. 4. Fit isotonic regression to (r1 , y1 ), . . . , (rk , yk ), (r, C ∗ ) obtaining an isotonic calibrator f ∗ . 5. Set ŷ ∗ := f ∗ (r). 6. Fit isotonic regression to (r1 , y1 ), . . . , (rk , yk ), (r, C∗ ) obtaining an isotonic calibrator f∗ . 7. Set ŷ∗ := f∗ (r). The bounded IVAR is a simple modification of the inductive Venn–Abers predictor as defined in Vovk et al. (2015). First, we relax the condition yi ∈ {0, 1} of binary labels by allowing the labels to take intermediate values, yi ∈ [0, 1], and then we scale the interval [0, 1] to arbitrary [C∗ , C ∗ ]. As mentioned in the previous section, non-inductive Venn–Abers regressors were introduced by van der Laan and Alaa (2024).

3.2

Unbounded regression

We again fix a regression algorithm and are given a training set of size l > k. Another parameter of our algorithm is m ∈ {1, . . . , ⌊(k − 1)/2⌋}; we are 4

interested in a small m, such as 1. The corresponding inductive Venn–Abers regressor (IVAR) produces the following regression interval of the form (1) for the test label: 1. Randomly split the training set of size l > k into two parts, a proper training set of size l − k and a calibration set z1 , . . . , zk of size k, as before. 2. Train the regression algorithm on the proper training set obtaining a prediction rule R : X → R. 3. Find the base predictions ri := R(xi ), i = 1, . . . , k, and r := R(x). 4. Replace the m smallest calibration labels yi by the (m + 1)th smallest calibration label y∗ and replace the m − 1 largest calibration labels yi by the mth largest calibration label y ∗ . (Notice the asymmetry in the definitions of y∗ and y ∗ .) In other words, let the new calibration labels be   y∗ if yi < y∗ (2) yi′ := y ∗ if yi > y ∗   yi otherwise. (Notice that the recipe is still unambiguous when there are ties among yi .) 5. Fit isotonic regression to (r1 , y1′ ), . . . , (rk , yk′ ), (r, y ∗ ) obtaining an isotonic calibrator f ∗ . 6. Set ŷ ∗ := f ∗ (r). 7. Replace the m largest calibration labels yi by the (m + 1)th largest calibration label y ∗ and replace the m − 1 smallest calibration labels yi by the mth smallest calibration label y∗ . (Notice that these y∗ and y ∗ are different from those in item 4.) In other words, let the new calibration labels be defined as (2) for the new y∗ and y ∗ . 8. Fit isotonic regression to (r1 , y1′ ), . . . , (rk , yk′ ), (r, y∗ ) obtaining an isotonic calibrator f∗ . 9. Set ŷ∗ := f∗ (r). When fitting isotonic regression in steps 5 and 8 (and the analogous steps, 4 and 6, in the bounded IVAR), we always use the standard pool-adjacentviolators algorithm (PAVA; see, e.g., Barlow et al. 1972, Sect. 1.2). Notice that the new steps 4 and 7 in the IVAR as compared with the bounded IVAR are really needed: if we just set y ∗ := ∞ and y∗ := −∞, the PAVA will produce (−∞, ∞) as the regression interval.

5

4

Validity

We often write Z1 , . . . , Zk , where Zi = (Xi , Yi ), for calibration examples and (X, Y ) for the test example, in order to emphasize that they are considered as random elements. First we state a simpler property of validity, the one for bounded regression (including binary classification as special case). So we assume that the labels take values in [C∗ , C ∗ ]. Let us say that a random variable S is auto-calibrated as regression estimate of Y if S = E(Y | S) a.s. (This is often used in the case of binary classification and was referred to as perfect calibration in, e.g., Vovk et al. 2022, Sect. 6.2.1, van der Laan and Alaa 2024, and van der Laan and Alaa 2025; for a general definition, see, e.g., Krüger and Ziegel 2021, Definition 3.1.) Remark 1. An equivalent definition of auto-calibration is that a random variable S is said to be auto-calibrated as regression estimate of Y if S = E(Y | F) a.s. for some σ-algebra F (if which case we may also say that S is ideal relative to F). A selector for an IVAR is a random variable that always belongs to the regression interval output by the IVAR. Theorem 2. For any bounded IVAR, there is a selector S that is auto-calibrated for the test label, i.e., E(Y | S) = S a.s. Our interpretation of Theorem 2 is that the regression interval produced by the bounded IVAR is approximately auto-calibrated provided it is narrow enough. Now we consider the case of unbounded regression and continue to assume 2m < k. A selector is defined as before, and the Winsorized test label Y is defined by   if Y < Y(m) Y(m) ′ (3) Y := Y(k−m+1) if Y > Y(k−m+1)   Y otherwise, where Y(1) ≤ · · · ≤ Y(k) is the sequence Y1 , . . . , Yk of calibration labels sorted in the ascending order. We can regard (3) as a more feasible version of Y corrected for the possibility of Y being an outlier; we make it easier to predict by restricting it to be in the range of the calibration labels. Remark 3. See Dixon (1960, Sect. 1) and Tukey (1962, Sect. 14) for the original definition of Winsorization, which we use in this paper. This definition is sometimes modified, as in, e.g., Wilcox (2012, Sect. 2.2.2). While the original definition is about moderating a dataset, the modified definition is about moderating the population. 2m . For any IVAR, Theorem 4. We have Y = Y ′ with probability at least 1 − k+1 there is a selector S that is auto-calibrated for the Winsorized test label Y ′ , i.e., E(Y ′ | S) = S.

6

The obvious first statement of Theorem 4 says that the Winsorized label Y ′ is the same as the original label Y with high probability assuming m ≪ k. The second statement asserts the validity of the IVAR as regression function for the Winsorized label. See Appendix A for the proofs. An alternative parametrization of the IVAR is in terms of ϵ := 2m/(k + 1); we then have Y = Y ′ with probability at least 1 − ϵ. If a given ϵ ∈ [2/(k + 1), 1) is not of the form 2m/(k + 1) for an integer m, we decrease it as little as possible so that it takes this form.

5

Algorithm

In our description of the algorithm, we will just refer to Vovk et al. (2022, Chap. 6, especially Sect. 6.5.3) (which in turn follows Vovk et al. 2015). We will discuss computing f ∗ and f∗ , which are denoted by f 1 and f 0 , respectively, in Vovk et al. (2022, Sect. 6.5.3). Remember that we moderate the original labels yi to their less extreme versions yi′ ; after that, we forget the original labels and drop the primes. But it should always be remembered that yi stand for the moderated labels. First we see how to compute F 1 , which determines f 1 = f ∗ . We are given the base predictions ri and the moderated labels y1 , . . . , yk . Define k ′ , r1′ , . . . , rk′ ′ , w1 , . . . , wk′ , and y1′ , . . . , yk′ ′ as in Vovk et al. (2022, Sect. 6.5.3) (where base predictions were called scores and were denoted by si ). The CSD consisting of the points Pi , i ∈ {0, . . . , k ′ }, is defined as before, but now it is extended by adding the point P−1 := (−1, −y ∗ ); this corresponds to adding the test example to the calibration set assuming that the base prediction for it is smaller than the base prediction for any calibration example while its label is y ∗ (which is a most unusual combination). Algorithm 6.3 in Vovk et al. (2022, Sect. 6.5.3) will compute the corners of the resulting CSD P−1 , . . . , Pk′ and Algorithm 6.4 will do the rest. Computing F 0 , which determines f 0 = f∗ , is analogous. Now the CSD consisting of the points Pi , i ∈ {0, . . . , k ′ }, is extended by adding the point Pk′ +1 := Pk′ + (1, y∗ ); this corresponds to adding the test example to the calibration set assuming that its base prediction is larger than any calibration base prediction while its label is y∗ (another most unusual combination). Algorithm 6.5 in Vovk et al. (2022, Sect. 6.5.3) will compute the corners of the resulting CSD P0 , . . . , Pk′ +1 and Algorithm 6.6 will do the rest. Algorithm 6.7 in Vovk et al. (2022, Sect. 6.5.3) shows how to arrange the calibration set into a convenient binary search tree, and Algorithm 6.8 shows how to use this tree for computationally efficient prediction. The computational complexity of these procedures is summarized in the following theorem. Theorem 5. The computation time of our prediction algorithm is O(k log k) for preprocessing (O(k) apart from sorting the calibration set). Once preprocessing is completed, processing each test object can be done in time O(log k).

7

6

Merging a Regression Interval into a Single Value

The IVAR introduced in the previous section achieves our goal of producing provably valid regression intervals that have a potential to be predictively efficient for efficient base algorithms. However, in order to be able to compare the predictive efficiency of our methods with traditional regression algorithms, in this and following section we will define natural modifications of the IVAR that produce point predictions. In this section we see how to replace the regression intervals output by IVARs by point predictions, and in the following section we will see how we can combine several IVARs to achieve both predictive and computational efficiency. Given y∗ < ŷ∗ < ŷ ∗ < y ∗ , let us see how to replace the regression interval [ŷ∗ , ŷ ∗ ] with a single regression value ŷ. Following the minimax approach of Vovk et al. (2022, Sect. 6.4.3), we need to solve the equation 2

2

2

2

(ŷ − y∗ ) − (ŷ∗ − y∗ ) = (y ∗ − ŷ) − (y ∗ − ŷ ∗ ) .

(4)

It is clear that there is a unique solution ŷ in the interval [ŷ∗ , ŷ ∗ ], since, over that interval, the left-hand side of (4) increases in ŷ from 0 to a positive number, while the right-hand side decreases from a positive number to 0. After simplification (4) becomes a linear equation in ŷ with solution ŷ =

ŷ∗2 − ŷ ∗2 + 2ŷ ∗ y ∗ − 2ŷ∗ y∗ , 2(y ∗ − y∗ )

(5)

and there is only one solution (both in the interval [ŷ∗ , ŷ ∗ ] and overall). We can also solve (4) approximately obtaining a much more intuitive expression. Rewriting (4) as 2(ŷ − ŷ∗ )(ŷ∗ − y∗ ) + (ŷ − ŷ∗ )2 = 2(ŷ ∗ − ŷ)(y ∗ − ŷ ∗ ) + (ŷ ∗ − ŷ)2 , assuming ŷ∗ ≈ ŷ ∗ , and ignoring the quadratic terms, we obtain the approximate equation (ŷ − ŷ∗ )(ŷ∗ − y∗ ) = (ŷ ∗ − ŷ)(y ∗ − ŷ ∗ ). Regrouping its terms as ŷ(ŷ∗ − y∗ + y ∗ − ŷ ∗ ) = ŷ∗ (ŷ∗ − y∗ ) + ŷ ∗ (y ∗ − ŷ ∗ ), we can write its solution as the weighted average ŷ =

ŷ∗ − y∗ y ∗ − ŷ ∗ ŷ∗ + ŷ ∗ ∗ ∗ ŷ∗ − y∗ + y − ŷ ŷ∗ − y∗ + y ∗ − ŷ ∗

of ŷ∗ and ŷ ∗ .

8

(6)

7

Cross Venn–Abers Regressors

We define cross Venn–Abers regressors (CVARs) as in Vovk et al. (2022, Sect. 6.4.4) except that the regression interval coming from each fold is replaced by the regression estimate computed as described in Sect. 6, either as in (5) or as in (6) (in our experiments in the next section we use the latter). Namely, we divide the training set into K folds of approximately equal size, and use each fold in turn as the calibration set while the union of the remaining folds is used as the proper training set. The overall regression estimate is found as the arithmetic mean of the K regression estimates obtained by merging the regression intervals coming from the K IVARs corresponding to the K folds. In our experiments we set the parameter K to 10.

8

Experimental Results

As discussed at the end of Sect. 2, in our experimental results we only compare our point regressors, namely the CVARs as defined in the previous section. These point regressors no longer satisfy any formal properties of validity, but the hope is that the validity properties of IVARs will show in superior performance of CVARs as compared with standard point regressors and measured in a traditional way. We evaluate our approach on a suite of controlled synthetic regression benchmarks designed to isolate different statistical challenges. We generate datasets with n = 10000 examples under the following generative scenarios. Each dataset consists of n examples whose objects are vectors of d = 10 features. As before, the objects are denoted by xi and the corresponding labels by yi , i = 1, . . . , n. Across all datasets, the level of noise in the generated labels is parametrized by σ ∈ {1, 3}. The results in this section represent the more challenging random noise level σ = 3, and results for σ = 1, as well as for n = 1000, are included in Appendix B. Unless stated otherwise, all random draws are independent. Bounded Logistic dataset Each object is generated as xi ∼ N (0, I10 ), the weight vector as w ∼ N (0, I10 ), and the Gaussian noise variables as ξi ∼ N (0, σ 2 ). (We parametrize the Gaussian distribution by its mean and variance, or its covariance matrix in the multidimensional case.) The labels are then generated using the sigmoid function: yi :=

10 + ξi . 1 + e−w⊤ xi

Notice that while the conditional expectation of yi is bounded (belongs to (0, 10]), the labels are not bounded because of the Gaussian noise. Since yi are “almost bounded” (the Gaussian distribution having thin tails), this is the most benign case, close to the setting of Sect. 3.1, and our methods work best for it.

9

Linear Gaussian dataset We generate xi ∼ N (0, I10 ), w ∼ N (0, I10 ), and ξi ∼ N (0, σ 2 ) as before. The labels are then defined by yi := w⊤ xi + ξi . Nonlinear dataset Again xi ∼ N (0, I10 ) and w ∼ N (0, I10 ). The label of xi combines a linear contribution with smooth nonlinear components: yi := w⊤ xi + 2 sin(xi,1 ) + 12 x2i,2 − cos(2xi,3 ) + ξi , where ξi ∼ N (0, σ 2 ) and xi,j denotes the jth feature of xi . Heteroscedastic noise dataset We again draw xi ∼ N (0, I10 ) and w ∼ N (0, I10 ). However, the observation noise now depends on the objects. Specifically,
yi := w⊤ xi + ξi , ξi ∼ N 0, σi2 , where the variance increases with the magnitude of the first feature, σi := 0.5σ + |xi,1 |. This creates object-dependent uncertainty. Heavy-tailed noise dataset Here xi ∼ N (0, I10 ) and w ∼ N (0, I10 ) as before, but the additive noise follows a Student-t distribution with low degrees of freedom ν: yi := w⊤ xi + ξi , ξi ∼ tν , where we use ν := 3σ. This produces infrequent but very large deviations. Outlier contamination dataset We generate xi ∼ N (0, I10 ) and w ∼ N (0, I10 ), and define yi := w⊤ xi + ξi , where ξi follows a contamination model: ( N (0, σ 2 ), with probability 1 − p, ξi ∼ N (0, τ 2 ), with probability p, with p := 0.01 representing the outlier probability and τ ≫ 1 (we set τ := 10σ) meaning that the rare errors are extremely large. Sparse high-dimensional dataset We again draw xi ∼ N (0, I10 ), but the true vector of coefficients is sparse. Let k be the sparsity level (in our experiments we set k := 2); we randomly select a support set S ⊂ {1, . . . , 10} with |S| = k and define ( N (0, 1), j ∈ S, wj := 0, j∈ / S. The labels follow yi := w⊤ xi + ξi ,

ξi ∼ N (0, σ 2 ). 10

Covariate shift dataset Here the training and test distributions differ. The (train) training objects are sampled as xi ∼ N (0, I10 ), while the test objects are (test) drawn from a shifted distribution xi ∼ N (µ, Σ) (we set µ to the vector of 1s and Σ := I10 ). A single w ∼ N (0, I10 ) generates both training and test labels: yi := w⊤ xi + ξi ,

ξi ∼ N (0, σ 2 ).

Thus the conditional distribution p(y | x) is the same across splits, but the marginal object distribution p(x) differs. Friedman’s datasets We also show results for Friedman’s three synthetic datasets (Friedman, 1991) as numbered by Breiman (Breiman, 1996, Sect. 3 and Appendix B) and implemented in scikit-learn. We use the scikit-learn implementation with the default values of parameters except for the size (namely, the default number of features is 10 for Friedman 1). For each dataset we randomly split the data into 80% training and 20% testing, standardize the features based on training statistics, and repeat all experiments across multiple (namely, 100) random seeds to reduce variance. At the end of the section we also briefly discuss results for the other combinations of n ∈ {1000, 10000} and σ ∈ {1, 3}. We compare our methods against seven widely used regression baselines implemented in scikit-learn, including linear regressors (Linear Regression, Ridge, Lasso, Elastic Net), kernel-based methods (Support Vector Regression, SVR, with the RBF kernel), and tree-based ensembles (Random Forest and Gradient Boosting). Linear Regression in fact implements Least Squares, and Ridge (or Ridge Regression), Lasso, and Elastic Net add various elements of regularization. All models are trained with default or widely adopted hyperparameters to reflect typical practitioner usage. Performance is measured using root mean squared error (RMSE) on the held-out test set. We report mean RMSE across repeated trials for each scenario to assess their accuracy. Tables 1–11 show the average RMSE over 100 trials for each synthetic dataset with σ = 3 and n = 10000 examples and for each base algorithm without calibration (none) or calibrated using our CVAR method with 10 folds and parameters m ∈ {1, 10} (denoted CVAR1 and CVAR10, respectively, in the tables; using m = 100 never leads to improvements as compared with the smaller m in our experiments). Apart from the average RMSE we also report the standard error of the mean (SEM) obtained √ by dividing the sample standard deviation of the RMSE over the 100 trials by 100 = 10; therefore, each cell in our tables has the form average RMSE ± SEM. These intervals reassure us that most of our comparisons are not unduly affected by randomness, but in our summaries we only use the averages. The smallest average in each row of our tables is shown in boldface. For each table we also report the average of each column, which is the average of the average RMSE over the seven regression algorithms. 11

Table 1: Bounded Logistic dataset

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

3.809 ± 0.015 3.239 ± 0.010 4.001 ± 0.018 3.257 ± 0.012 3.231 ± 0.008 3.257 ± 0.012 3.143 ± 0.007

3.205 ± 0.007 3.144 ± 0.006 3.588 ± 0.017 3.011 ± 0.005 3.208 ± 0.007 3.011 ± 0.005 3.070 ± 0.005

3.206 ± 0.007 3.145 ± 0.006 3.589 ± 0.017 3.012 ± 0.005 3.209 ± 0.007 3.012 ± 0.005 3.071 ± 0.005

3.420

3.177

3.178

Table 2: Linear Gaussian dataset

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

3.510 ± 0.015 3.073 ± 0.006 3.707 ± 0.018 3.002 ± 0.005 3.138 ± 0.006 3.002 ± 0.005 3.087 ± 0.005

3.147 ± 0.006 3.083 ± 0.006 3.419 ± 0.014 3.019 ± 0.005 3.150 ± 0.007 3.019 ± 0.005 3.101 ± 0.006

3.155 ± 0.006 3.091 ± 0.006 3.424 ± 0.014 3.029 ± 0.005 3.158 ± 0.007 3.029 ± 0.005 3.108 ± 0.006

3.217

3.134

3.142

Remark 6. The results for Linear Regression and Ridge coincide in Tables 1– 11, but they are sometimes slightly different in Appendix B. The reason for this closeness is that our features are constructed as independent, so multicollinearity is unlikely, and the sample size is large relative to the feature count; this makes Least Squares stable, and so the default ridge penalty has relatively little impact. The last rows in Tables 1–11 show that both of our methods, CVAR1 and CVAR10, attain better RMSE scores on average when compared with uncalibrated algorithms, especially for the bounded logistic, linear Gaussian, nonlinear, covariate shift, Friedman 1, and Friedman 2 datasets (the results for the last dataset, however, are very irregular, and in two cases our methods significantly lower the quality of predictions). In many rows the base algorithm performs better, but in a typical row of a typical table either our algorithms perform better or they lose little. For the “almost bounded” Bounded Logistic dataset our methods improve the base predictions in all rows.

12

Table 3: Nonlinear dataset

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

3.718 ± 0.015 3.100 ± 0.006 3.904 ± 0.018 3.199 ± 0.005 3.194 ± 0.008 3.199 ± 0.005 3.139 ± 0.006

3.327 ± 0.006 3.112 ± 0.006 3.590 ± 0.014 3.211 ± 0.006 3.210 ± 0.008 3.211 ± 0.006 3.157 ± 0.006

3.336 ± 0.006 3.124 ± 0.007 3.596 ± 0.014 3.221 ± 0.006 3.220 ± 0.008 3.221 ± 0.006 3.167 ± 0.006

3.350

3.260

3.269

Table 4: Heteroscedastic noise dataset

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

4.654 ± 0.016 4.371 ± 0.012 4.804 ± 0.017 4.289 ± 0.011 4.430 ± 0.012 4.289 ± 0.011 4.358 ± 0.011

4.386 ± 0.011 4.367 ± 0.012 4.587 ± 0.015 4.295 ± 0.011 4.425 ± 0.012 4.295 ± 0.011 4.365 ± 0.011

4.392 ± 0.011 4.372 ± 0.012 4.590 ± 0.015 4.302 ± 0.011 4.429 ± 0.012 4.302 ± 0.011 4.369 ± 0.012

4.456

4.388

4.394

Table 5: Heavy-tailed noise dataset

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

5.444 ± 0.076 5.207 ± 0.077 5.578 ± 0.075 5.120 ± 0.078 5.280 ± 0.077 5.120 ± 0.078 5.173 ± 0.077

5.218 ± 0.077 5.212 ± 0.077 5.407 ± 0.075 5.137 ± 0.078 5.283 ± 0.077 5.137 ± 0.078 5.196 ± 0.077

5.210 ± 0.077 5.204 ± 0.077 5.391 ± 0.075 5.132 ± 0.078 5.267 ± 0.077 5.132 ± 0.078 5.184 ± 0.077

5.275

5.227

5.217

13

Table 6: Outlier contamination dataset

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

4.674 ± 0.040 4.389 ± 0.042 4.826 ± 0.039 4.290 ± 0.043 4.456 ± 0.042 4.290 ± 0.043 4.353 ± 0.042

4.401 ± 0.042 4.397 ± 0.042 4.617 ± 0.040 4.309 ± 0.043 4.456 ± 0.041 4.309 ± 0.043 4.378 ± 0.042

4.400 ± 0.042 4.388 ± 0.042 4.604 ± 0.040 4.311 ± 0.043 4.443 ± 0.041 4.311 ± 0.043 4.370 ± 0.042

4.468

4.409

4.404

Table 7: Sparse high-dimensional signals dataset

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

3.049 ± 0.007 3.009 ± 0.005 3.079 ± 0.009 2.995 ± 0.005 3.046 ± 0.005 2.995 ± 0.005 3.038 ± 0.005

2.999 ± 0.005 3.008 ± 0.005 3.026 ± 0.006 2.998 ± 0.005 3.027 ± 0.005 2.998 ± 0.005 3.025 ± 0.005

3.000 ± 0.005 3.008 ± 0.005 3.027 ± 0.006 2.998 ± 0.005 3.026 ± 0.005 2.998 ± 0.005 3.024 ± 0.005

3.030

3.012

3.012

Table 8: Covariate shift dataset

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

3.887 ± 0.050 3.213 ± 0.011 4.191 ± 0.065 3.007 ± 0.004 3.371 ± 0.019 3.007 ± 0.004 3.600 ± 0.043

3.352 ± 0.023 3.275 ± 0.019 3.809 ± 0.044 3.109 ± 0.018 3.426 ± 0.024 3.109 ± 0.018 3.658 ± 0.046

3.378 ± 0.025 3.306 ± 0.023 3.825 ± 0.045 3.143 ± 0.021 3.453 ± 0.027 3.143 ± 0.021 3.675 ± 0.048

3.468

3.391

3.417

14

Table 9: Friedman 1 dataset

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

4.383 ± 0.008 3.139 ± 0.006 4.353 ± 0.008 3.864 ± 0.007 3.276 ± 0.006 3.864 ± 0.007 3.226 ± 0.006

3.871 ± 0.007 3.173 ± 0.006 3.960 ± 0.007 3.858 ± 0.007 3.312 ± 0.006 3.858 ± 0.007 3.258 ± 0.006

3.879 ± 0.007 3.190 ± 0.006 3.967 ± 0.007 3.865 ± 0.007 3.328 ± 0.006 3.865 ± 0.007 3.273 ± 0.006

3.729

3.613

3.624

Table 10: Friedman 2 dataset

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF)

none

CVAR1

CVAR10

181.783 ± 0.340 15.432 ± 0.057 137.838 ± 0.233 137.820 ± 0.233 6.550 ± 0.019 137.820 ± 0.233 140.878 ± 0.536

82.684 ± 0.170 48.841 ± 0.119 82.678 ± 0.171 82.720 ± 0.170 47.315 ± 0.119 82.720 ± 0.170 105.137 ± 0.376

84.122 ± 0.168 51.088 ± 0.149 84.126 ± 0.169 84.167 ± 0.169 49.651 ± 0.151 84.167 ± 0.169 105.298 ± 0.397

108.303

76.013

77.517

average

Table 11: Friedman 3 dataset

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

3.022 ± 0.005 3.023 ± 0.005 3.022 ± 0.005 3.013 ± 0.005 3.111 ± 0.005 3.013 ± 0.005 3.023 ± 0.005

3.022 ± 0.005 3.020 ± 0.005 3.022 ± 0.005 3.016 ± 0.005 3.026 ± 0.005 3.016 ± 0.005 3.020 ± 0.005

3.022 ± 0.005 3.018 ± 0.005 3.022 ± 0.005 3.014 ± 0.005 3.024 ± 0.005 3.014 ± 0.005 3.018 ± 0.005

3.032

3.020

3.019

15

We report the results for σ ∈ {1, 3} and n = 1000 and for σ = 1 and n = 10000 settings in Appendix B. The results for n = 10000 and σ = 1 are similar to the results in this section, while the results for n = 1000 are mixed for σ = 3 and weaker for our methods in the case of low noise, σ = 1. It appears that our methods work well for larger datasets. In Appendix B we also give results for four real-life datasets. Our methods tend to improve the performance of the base algorithms unless the dataset is small, but the tendency is weak.

9

Conclusion

This paper presents new validity guarantees for regression estimation. Computational experiments show that this leads to a limited improvement in the performance of standard point regressors on large datasets. These are some directions of further research. • An alternative merging procedure to the ones described in Sect. 7 is to try and merge the regression intervals coming from the K folds directly (in a minimax manner, as in Vovk et al. 2022, Sect. 6.4.5) in order to obtain an overall regression estimate, without the intermediate step of merging each regression interval. It would be interesting to compare it experimentally with the procedure used in this paper. • In this paper we explore the predictive efficiency of the CVAR in simulation and empirical studies. Alternatively, we could try and prove theoretical results along the lines of the Burnaev–Wasserman programme, as described in Vovk et al. (2022, Sects. 2.5 and 2.9.7). As a first step, we may assume that the base predictions r coincide with the true regression function, r := E(Y | X = x). • Our methods tend to improve significantly the quality of predictions made by Lasso and Elastic Net. A very common pattern of the boldface (i.e., best) entries in our tables is “CVAR1, none, CVAR1, none, none, none, none” (from top to bottom); we can observe it in three tables (Tables 2, 3, and 8) in the main paper, and in more than half of the tables (20 out of 37) in Appendix B. For this pattern our methods help only for Lasso and Elastic Net. However, it can be argued that applying our methods destroys, at least partially, a valued property of Lasso and Elastic Net, automatic feature selection. Optimizing the number of features is another interesting desideratum, alongside predictive efficiency.

Acknowledgements We are grateful to anonymous referees for their advice.

16

References Sam Allen, Georgios Gavrilopoulos, Alexander Henzi, Gian-Reto Kleger, and Johanna Ziegel. In-sample calibration yields conformal calibration guarantees. Technical Report arXiv:2503.03841 [stat.ME], arXiv.org e-Print archive, March 2025. Anastasios N. Angelopoulos, Rina Foygel Barber, and Stephen Bates. Theoretical foundations of conformal prediction. Technical Report arXiv:2411.11824 [math.ST], arXiv.org e-Print archive, March 2026. Pre-publication version of a book to be published by Cambridge University Press. Richard E. Barlow, D. J. Bartholomew, J. M. Bremner, and H. Daniel Brunk. Statistical Inference under Order Restrictions: The Theory and Application of Isotonic Regression. Wiley, London, 1972. Leo Breiman. Bagging predictors. Machine Learning, 24:123–140, 1996. Thomas Brooks, D. Pope, and Michael Marcolini. Airfoil self-noise [dataset]. UCI Machine Learning Repository, 1989. DOI: https://doi.org/10.24432/ C5VW2C. Dongjin Cho, Cheolhee Yoo, Jungho Im, and Dong-Hyun Cha. Bias correction of numerical prediction model temperature forecast [dataset]. UCI Machine Learning Repository, 2020. DOI: https://doi.org/10.24432/C59K76. Paulo Cortez. Student performance [dataset]. UCI Machine Learning Repository, 2008. DOI: https://doi.org/10.24432/C5TG7T. Paulo Cortez, A. Cerdeira, F. Almeida, T. Matos, and J. Reis. Wine quality [dataset]. UCI Machine Learning Repository, 2009. DOI: https://doi.org/10. 24432/C56S3T. Wilfrid J. Dixon. Simplified estimation from censored normal samples. Annals of Mathematical Statistics, 31:385–391, 1960. Jerome H. Friedman. Multivariate adaptive regression splines (with discussion). Annals of Statistics, 19:1–141, 1991. Isaac Gibbs, John J. Cherian, and Emmanuel J. Candès. Conformal prediction with conditional guarantees. Journal of the Royal Statistical Society B, 87: 1100–1126, 2025. Fabian Krüger and Johanna F. Ziegel. Generic conditions for forecast dominance. Journal of Business and Economic Statistics, 39:972–983, 2021. Ivan Petej. Inductive Venn–Abers and related regressors benchmark experiments, 2024. URL https://github.com/ip200/ivar-experiments (based on the “Venn–Abers calibration” library https://github.com/ip200/venn-abers).

17

Yaniv Romano, Evan Patterson, and Emmanuel J. Candès. Conformalized quantile regression. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019), pages 3543–3553, 2019. John W. Tukey. The future of data analysis. Annals of Mathematical Statistics, 33:1–67, 1962. Lars van der Laan and Ahmed M. Alaa. Self-calibrating conformal prediction. In NeurIPS, 2024. Available on OpenReview. Lars van der Laan and Ahmed M. Alaa. Generalized Venn and Venn–Abers calibration with applications in conformal prediction. In ICML, 2025. Available on OpenReview. Vladimir N. Vapnik. Statistical Learning Theory. Wiley, New York, 1998. Vladimir Vovk, Ivan Petej, and Valentina Fedorova. Large-scale probabilistic predictors with and without guarantees of validity. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 892–900. Curran Associates, 2015. Full version: arXiv:1511.00213 [cs.LG]. Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World. Springer, Cham, second edition, 2022. Rand R. Wilcox. Introduction to Robust Estimation and Hypothesis Testing. Elsevier, Amsterdam, third edition, 2012.

A

Proofs

A.1

Proof of Theorem 2

The selector S that we use for demonstrating the theorem is a modification of items 4–7 in the description of the bounded IVAR corresponding to using the true test label. Namely, we fit isotonic regression to (r1 , y1 ), . . . , (rk , yk ), (r, y), where y is the true label of the test object, obtaining an isotonic calibrator f . Set S := f (r). First we need to check that S is indeed a selector, namely that f∗ (r) ≤ S ≤ f ∗ (r). This follows from C∗ ≤ y ≤ C ∗ and the monotonicity property of PAVA given by the following lemma. Lemma 7. If yi ≤ yi′ for all i ∈ {1, . . . , n}, then f ≤ f ′ , where f (resp. f ′ ) is the isotonic regression fitted to (r1 , y1 ), . . . , (rn , yn ) (resp. to (r1 , y1′ ), . . . , (rn , yn′ )). Proof. This is an immediate corollary of the max-min formulas (as given in, e.g., Barlow et al. 1972, p. 19).

18

Now let us check that E(Y | S) = S even conditionally on a bag *Z1 , . . . , Zk , Z+ of k calibration examples and a test example under the uniform probability measure on all (k + 1)! orderings of the bag (see Vovk et al. 2022, Lemma A.3). The value of S determines the solution block containing the test example. Therefore, both S and E(Y | S) will equal the arithmetic mean of the labels in that solution block.

A.2

Proof of Theorem 4

Fix an IVAR. The selector S is produced by the following ideal picture. Let y be the true label of the test object x. Then the selector S associated with the IVAR is defined by the following recipe applied after splitting the training set and training the base regression algorithm on the proper training set (which gives us a prediction rule). 1. Winsorize the labels, namely: • replace the m largest labels in the augmented calibration sequence (x1 , y1 ), . . . , (xk , yk ), (x, y) by the (m + 1)th largest label; • replace the m smallest labels in the augmented calibration sequence by the (m + 1)th smallest label. 2. Find the base predictions r1 , . . . , rk , r of the objects x1 , . . . , xk , x using the prediction rule. 3. Fit isotonic regression f to (r1 , y1 ), . . . , (rk , yk ), (r, y) (with the Winsorized labels). 4. Set S := f (r). Step 1 regularizes the labels moderating the 2m most extreme ones. The recipe treats all elements of the augmented calibration sequence symmetrically, which is essential in the proof of E(Y ′ | S) = S; let us call it the symmetric recipe. First let us check that S is indeed a selector for the IVAR. Consider three cases: • If y is one of the m largest labels in the augmented calibration sequence, the definition of IVAR and the symmetric recipe modify the labels in the same way; in particular, the m − 1 largest calibration labels and the test label are replaced by y ∗ , and the m smallest calibration labels are replaced by y∗ . In this case S = ŷ ∗ . • Analogously, if y is one of the m smallest labels in the augmented calibration sequence, we have S = ŷ∗ . • Otherwise, the monotonicity property of isotonic regression given in Lemma 7 above implies that we still have S ∈ [ŷ∗ , ŷ ∗ ]. 19

Table 12: Bounded Logistic dataset (σ = 1)

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

2.551 ± 0.021 1.532 ± 0.018 2.830 ± 0.024 1.596 ± 0.021 1.497 ± 0.017 1.596 ± 0.021 1.211 ± 0.009

1.506 ± 0.012 1.323 ± 0.009 2.201 ± 0.025 1.031 ± 0.002 1.463 ± 0.014 1.031 ± 0.002 1.093 ± 0.002

1.508 ± 0.012 1.326 ± 0.009 2.203 ± 0.025 1.035 ± 0.002 1.466 ± 0.014 1.035 ± 0.002 1.097 ± 0.002

1.830

1.378

1.382

In all three cases, S ∈ [ŷ∗ , ŷ ∗ ], which means that S is a selector. Remark 8. The first two cases considered in the proof demonstrate the tightness, in some sense, of the regression interval [ŷ∗ , ŷ ∗ ]. It remains to prove E(Y ′ | S) = S. We prove this equality conditionally on the set of all permutations (equiprobable) of a given augmented calibration sequence, where a permutation π of {1, . . . , k + 1} makes zπ−1 (k+1) the test example. The Winsorized test label (3) over the permutations is the same random variable as the test label produced according to the symmetric recipe. It remains to remember that the isotonic regression at the base prediction r for the test object is the mean of the labels in the augmented calibration sequence in the same solution block as the test example (Barlow et al., 1972, pp. 13–15). Solution blocks are level sets of the isotonic regression, which implies S being the conditional average of the Winsorized test label given S.

B

Further Experimental Results

In Tables 12–22 we show the experimental results in the case of σ = 1 parallel to those reported in Sect. 8 of the main paper; in particular, we still have n = 10000. Our comments in the main paper are also applicable here. In particular, it is still true that on average both CVAR1 and CVAR10 produce better results than the base algorithms in all 11 tables. The level of noise does not appear to be a crucial parameter. Tables 23–33 give results for n = 1000 examples and noise level σ = 3. For these smaller datasets the results are mixed, as we said in the main paper. In Tables 34–44 we have n = 1000 examples, but the noise level is smaller, σ = 1. The results for our methods appear even worse than for the more challenging case σ = 3; the base algorithms work best on average in nine cases out of eleven.

20

Table 13: Linear Gaussian dataset (σ = 1)

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

2.079 ± 0.024 1.155 ± 0.008 2.397 ± 0.026 1.001 ± 0.002 1.281 ± 0.013 1.001 ± 0.002 1.094 ± 0.004

1.395 ± 0.009 1.189 ± 0.008 1.921 ± 0.024 1.084 ± 0.005 1.312 ± 0.014 1.084 ± 0.005 1.155 ± 0.006

1.414 ± 0.010 1.213 ± 0.009 1.933 ± 0.024 1.110 ± 0.006 1.333 ± 0.014 1.110 ± 0.006 1.179 ± 0.007

1.430

1.306

1.327

Table 14: Nonlinear dataset (σ = 1)

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

2.417 ± 0.021 1.227 ± 0.009 2.695 ± 0.024 1.494 ± 0.003 1.417 ± 0.015 1.494 ± 0.003 1.202 ± 0.004

1.763 ± 0.007 1.260 ± 0.009 2.212 ± 0.020 1.539 ± 0.005 1.455 ± 0.015 1.539 ± 0.005 1.272 ± 0.006

1.781 ± 0.008 1.289 ± 0.010 2.223 ± 0.020 1.560 ± 0.005 1.479 ± 0.016 1.560 ± 0.005 1.301 ± 0.007

1.707

1.577

1.599

Table 15: Heteroscedastic noise dataset (σ = 1)

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

2.315 ± 0.022 1.550 ± 0.007 2.604 ± 0.025 1.430 ± 0.004 1.640 ± 0.010 1.430 ± 0.004 1.509 ± 0.005

1.722 ± 0.008 1.570 ± 0.007 2.173 ± 0.021 1.480 ± 0.005 1.662 ± 0.011 1.480 ± 0.005 1.546 ± 0.006

1.740 ± 0.009 1.591 ± 0.008 2.184 ± 0.021 1.504 ± 0.006 1.681 ± 0.011 1.504 ± 0.006 1.566 ± 0.007

1.782

1.662

1.681

21

Table 16: Heavy-tailed noise dataset (σ = 1)

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

2.512 ± 0.029 1.819 ± 0.026 2.785 ± 0.030 1.707 ± 0.026 1.901 ± 0.026 1.707 ± 0.026 1.770 ± 0.025

1.957 ± 0.025 1.830 ± 0.026 2.381 ± 0.029 1.742 ± 0.026 1.915 ± 0.026 1.742 ± 0.026 1.797 ± 0.025

1.981 ± 0.025 1.856 ± 0.026 2.395 ± 0.029 1.772 ± 0.026 1.938 ± 0.026 1.772 ± 0.026 1.822 ± 0.025

2.028

1.909

1.934

Table 17: Outlier contamination dataset (σ = 1)

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

2.330 ± 0.022 1.561 ± 0.013 2.620 ± 0.025 1.430 ± 0.014 1.656 ± 0.015 1.430 ± 0.014 1.500 ± 0.014

1.718 ± 0.014 1.573 ± 0.014 2.182 ± 0.022 1.473 ± 0.014 1.671 ± 0.015 1.473 ± 0.014 1.533 ± 0.014

1.748 ± 0.014 1.607 ± 0.014 2.199 ± 0.022 1.512 ± 0.014 1.700 ± 0.015 1.512 ± 0.014 1.567 ± 0.014

1.790

1.661

1.692

Table 18: Sparse high-dimensional signals dataset (σ = 1)

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

1.144 ± 0.012 1.004 ± 0.002 1.214 ± 0.018 0.998 ± 0.002 1.020 ± 0.002 0.998 ± 0.002 1.030 ± 0.002

1.016 ± 0.003 1.010 ± 0.002 1.085 ± 0.011 1.005 ± 0.002 1.022 ± 0.002 1.005 ± 0.002 1.029 ± 0.003

1.019 ± 0.003 1.012 ± 0.002 1.087 ± 0.011 1.008 ± 0.002 1.025 ± 0.003 1.008 ± 0.002 1.032 ± 0.003

1.058

1.025

1.027

22

Table 19: Covariate shift dataset (σ = 1)

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

2.619 ± 0.068 1.462 ± 0.024 3.047 ± 0.083 1.002 ± 0.001 1.778 ± 0.036 1.002 ± 0.001 1.888 ± 0.052

1.794 ± 0.040 1.590 ± 0.037 2.537 ± 0.063 1.312 ± 0.036 1.863 ± 0.042 1.312 ± 0.036 2.025 ± 0.060

1.841 ± 0.043 1.646 ± 0.042 2.563 ± 0.064 1.377 ± 0.041 1.908 ± 0.046 1.377 ± 0.041 2.058 ± 0.062

1.828

1.776

1.824

Table 20: Friedman 1 dataset (σ = 1)

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

3.344 ± 0.005 1.307 ± 0.004 3.305 ± 0.005 2.629 ± 0.005 1.498 ± 0.003 2.629 ± 0.005 1.309 ± 0.003

2.649 ± 0.005 1.418 ± 0.004 2.775 ± 0.005 2.628 ± 0.005 1.603 ± 0.003 2.628 ± 0.005 1.428 ± 0.003

2.658 ± 0.005 1.447 ± 0.004 2.783 ± 0.005 2.637 ± 0.005 1.628 ± 0.003 2.637 ± 0.005 1.458 ± 0.003

2.289

2.161

2.178

Table 21: Friedman 2 dataset (σ = 1)

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

181.762 ± 0.340 15.208 ± 0.061 137.802 ± 0.233 137.783 ± 0.233 5.791 ± 0.021 137.784 ± 0.233 140.800 ± 0.538

82.637 ± 0.170 48.795 ± 0.120 82.631 ± 0.171 82.675 ± 0.171 47.239 ± 0.119 82.675 ± 0.171 105.034 ± 0.376

84.073 ± 0.168 51.037 ± 0.150 84.076 ± 0.170 84.118 ± 0.170 49.570 ± 0.152 84.118 ± 0.170 105.190 ± 0.398

108.133

75.955

77.455

23

Table 22: Friedman 3 dataset (σ = 1)

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

1.050 ± 0.002 1.009 ± 0.002 1.050 ± 0.002 1.022 ± 0.002 1.042 ± 0.002 1.022 ± 0.002 1.012 ± 0.002

1.050 ± 0.002 1.009 ± 0.002 1.050 ± 0.002 1.014 ± 0.002 1.021 ± 0.002 1.014 ± 0.002 1.011 ± 0.002

1.050 ± 0.002 1.009 ± 0.002 1.050 ± 0.002 1.014 ± 0.002 1.020 ± 0.002 1.014 ± 0.002 1.010 ± 0.002

1.030

1.024

1.024

Table 23: Bounded Logistic dataset (n = 1000 and σ = 3)

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

3.831 ± 0.021 3.420 ± 0.019 4.035 ± 0.023 3.288 ± 0.019 3.449 ± 0.018 3.288 ± 0.019 3.345 ± 0.018

3.302 ± 0.018 3.408 ± 0.016 3.691 ± 0.023 3.120 ± 0.015 3.466 ± 0.017 3.120 ± 0.015 3.242 ± 0.016

3.417 ± 0.018 3.514 ± 0.016 3.773 ± 0.022 3.247 ± 0.015 3.564 ± 0.017 3.247 ± 0.015 3.358 ± 0.016

3.522

3.336

3.446

Table 24: Linear Gaussian dataset (n = 1000 and σ = 3)

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

3.549 ± 0.022 3.228 ± 0.016 3.755 ± 0.025 3.016 ± 0.015 3.289 ± 0.018 3.016 ± 0.015 3.256 ± 0.019

3.289 ± 0.019 3.342 ± 0.019 3.562 ± 0.023 3.173 ± 0.018 3.375 ± 0.022 3.173 ± 0.018 3.307 ± 0.020

3.431 ± 0.023 3.475 ± 0.024 3.661 ± 0.026 3.333 ± 0.022 3.502 ± 0.026 3.333 ± 0.022 3.438 ± 0.025

3.301

3.317

3.453

24

Table 25: Nonlinear dataset (n = 1000 and σ = 3)

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

3.749 ± 0.023 3.265 ± 0.017 3.941 ± 0.026 3.210 ± 0.014 3.365 ± 0.019 3.209 ± 0.014 3.365 ± 0.019

3.473 ± 0.019 3.418 ± 0.020 3.730 ± 0.023 3.366 ± 0.018 3.483 ± 0.022 3.366 ± 0.018 3.425 ± 0.020

3.626 ± 0.024 3.579 ± 0.025 3.841 ± 0.027 3.536 ± 0.023 3.634 ± 0.028 3.536 ± 0.023 3.580 ± 0.025

3.444

3.466

3.619

Table 26: Heteroscedastic noise dataset (n = 1000 and σ = 3)

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

4.677 ± 0.032 4.545 ± 0.031 4.837 ± 0.033 4.282 ± 0.029 4.522 ± 0.031 4.282 ± 0.029 4.479 ± 0.031

4.474 ± 0.031 4.580 ± 0.031 4.688 ± 0.033 4.382 ± 0.030 4.572 ± 0.032 4.382 ± 0.030 4.498 ± 0.031

4.599 ± 0.033 4.682 ± 0.033 4.771 ± 0.035 4.524 ± 0.032 4.675 ± 0.034 4.524 ± 0.032 4.608 ± 0.033

4.518

4.511

4.626

Table 27: Heavy-tailed noise dataset (n = 1000 and σ = 3)

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

5.266 ± 0.091 5.297 ± 0.095 5.409 ± 0.089 4.949 ± 0.093 5.237 ± 0.093 4.949 ± 0.093 5.084 ± 0.093

5.094 ± 0.092 5.208 ± 0.093 5.279 ± 0.091 5.018 ± 0.093 5.207 ± 0.094 5.018 ± 0.093 5.107 ± 0.093

5.188 ± 0.092 5.280 ± 0.093 5.336 ± 0.091 5.129 ± 0.093 5.277 ± 0.093 5.129 ± 0.093 5.191 ± 0.093

5.170

5.133

5.218

25

Table 28: Outlier contamination dataset (n = 1000 and σ = 3)

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

4.561 ± 0.102 4.488 ± 0.108 4.727 ± 0.099 4.171 ± 0.108 4.480 ± 0.107 4.171 ± 0.108 4.329 ± 0.106

4.358 ± 0.105 4.457 ± 0.106 4.591 ± 0.103 4.268 ± 0.107 4.479 ± 0.105 4.269 ± 0.107 4.374 ± 0.106

4.469 ± 0.103 4.542 ± 0.104 4.652 ± 0.101 4.401 ± 0.105 4.555 ± 0.103 4.401 ± 0.105 4.474 ± 0.104

4.418

4.399

4.499

Table 29: Sparse high-dimensional signals dataset (n = 1000 and σ = 3)

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

3.022 ± 0.018 3.106 ± 0.017 3.042 ± 0.018 2.978 ± 0.016 3.064 ± 0.017 2.978 ± 0.016 3.041 ± 0.017

2.982 ± 0.016 3.038 ± 0.017 3.009 ± 0.016 2.995 ± 0.016 3.030 ± 0.017 2.995 ± 0.016 3.030 ± 0.017

2.997 ± 0.017 3.038 ± 0.018 3.024 ± 0.017 3.004 ± 0.017 3.032 ± 0.019 3.004 ± 0.017 3.033 ± 0.019

3.033

3.011

3.019

Table 30: Covariate shift dataset (n = 1000 and σ = 3)

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

3.954 ± 0.064 3.490 ± 0.029 4.273 ± 0.080 3.040 ± 0.018 3.624 ± 0.045 3.040 ± 0.018 4.151 ± 0.086

3.693 ± 0.067 3.832 ± 0.069 4.054 ± 0.072 3.536 ± 0.067 3.881 ± 0.071 3.536 ± 0.067 4.179 ± 0.089

3.934 ± 0.082 4.051 ± 0.084 4.234 ± 0.085 3.808 ± 0.083 4.097 ± 0.085 3.808 ± 0.083 4.342 ± 0.099

3.653

3.816

4.039

26

Table 31: Friedman 1 dataset (n = 1000 and σ = 3)

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

4.385 ± 0.023 3.378 ± 0.018 4.365 ± 0.023 3.893 ± 0.021 3.568 ± 0.019 3.893 ± 0.021 3.746 ± 0.021

4.055 ± 0.021 3.660 ± 0.020 4.157 ± 0.021 4.043 ± 0.021 3.808 ± 0.020 4.043 ± 0.021 3.865 ± 0.021

4.269 ± 0.022 3.934 ± 0.022 4.359 ± 0.023 4.256 ± 0.022 4.061 ± 0.022 4.256 ± 0.022 4.104 ± 0.023

3.890

3.947

4.177

Table 32: Friedman 2 dataset (n = 1000 and σ = 3)

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF)

none

CVAR1

CVAR10

181.546 ± 1.247 23.936 ± 0.181 137.739 ± 0.784 137.743 ± 0.784 20.166 ± 0.202 137.746 ± 0.784 345.749 ± 2.419

151.625 ± 1.044 142.035 ± 0.940 151.686 ± 1.032 151.713 ± 1.033 141.653 ± 0.914 151.711 ± 1.033 170.491 ± 1.685

181.004 ± 1.604 171.828 ± 1.657 181.067 ± 1.600 181.083 ± 1.600 171.544 ± 1.636 181.082 ± 1.600 193.626 ± 2.036

140.661

151.559

180.176

average

Table 33: Friedman 3 dataset (n = 1000 and σ = 3)

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

3.019 ± 0.015 3.130 ± 0.015 3.019 ± 0.015 3.018 ± 0.015 3.146 ± 0.016 3.018 ± 0.015 3.054 ± 0.016

3.019 ± 0.015 3.039 ± 0.015 3.019 ± 0.015 3.036 ± 0.015 3.045 ± 0.015 3.036 ± 0.015 3.036 ± 0.015

3.020 ± 0.015 3.024 ± 0.015 3.020 ± 0.015 3.019 ± 0.015 3.028 ± 0.015 3.019 ± 0.015 3.022 ± 0.015

3.058

3.033

3.022

27

Table 34: Bounded Logistic dataset (n = 1000 and σ = 1)

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

2.580 ± 0.022 1.708 ± 0.021 2.877 ± 0.026 1.630 ± 0.023 1.870 ± 0.023 1.629 ± 0.023 1.578 ± 0.017

1.749 ± 0.016 1.770 ± 0.016 2.390 ± 0.026 1.380 ± 0.007 1.953 ± 0.019 1.380 ± 0.007 1.504 ± 0.008

1.849 ± 0.015 1.869 ± 0.015 2.454 ± 0.025 1.506 ± 0.007 2.042 ± 0.018 1.506 ± 0.007 1.619 ± 0.008

1.982

1.732

1.835

Table 35: Linear Gaussian dataset (n = 1000 and σ = 1)

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

2.141 ± 0.029 1.326 ± 0.015 2.471 ± 0.032 1.005 ± 0.005 1.566 ± 0.024 1.005 ± 0.005 1.418 ± 0.020

1.744 ± 0.024 1.676 ± 0.028 2.189 ± 0.030 1.529 ± 0.025 1.797 ± 0.033 1.529 ± 0.025 1.651 ± 0.027

1.956 ± 0.031 1.899 ± 0.035 2.335 ± 0.035 1.772 ± 0.032 2.001 ± 0.039 1.772 ± 0.032 1.879 ± 0.035

1.562

1.731

1.945

Table 36: Nonlinear dataset (n = 1000 and σ = 1)

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

2.469 ± 0.028 1.430 ± 0.016 2.755 ± 0.032 1.496 ± 0.008 1.735 ± 0.025 1.496 ± 0.008 1.622 ± 0.020

2.069 ± 0.023 1.835 ± 0.030 2.458 ± 0.030 1.896 ± 0.023 2.004 ± 0.033 1.896 ± 0.023 1.870 ± 0.028

2.284 ± 0.031 2.087 ± 0.036 2.613 ± 0.035 2.139 ± 0.031 2.230 ± 0.039 2.139 ± 0.031 2.118 ± 0.035

1.858

2.004

2.230

28

Table 37: Heteroscedastic noise dataset (n = 1000 and σ = 1)

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

2.379 ± 0.027 1.697 ± 0.014 2.683 ± 0.030 1.427 ± 0.010 1.875 ± 0.022 1.427 ± 0.010 1.777 ± 0.019

2.014 ± 0.023 1.977 ± 0.026 2.422 ± 0.029 1.826 ± 0.023 2.069 ± 0.030 1.826 ± 0.023 1.945 ± 0.026

2.221 ± 0.030 2.192 ± 0.033 2.564 ± 0.033 2.062 ± 0.030 2.267 ± 0.036 2.062 ± 0.030 2.163 ± 0.033

1.895

2.011

2.219

Table 38: Heavy-tailed noise dataset (n = 1000 and σ = 1)

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

2.492 ± 0.034 1.907 ± 0.032 2.783 ± 0.037 1.650 ± 0.031 2.043 ± 0.034 1.650 ± 0.031 1.932 ± 0.034

2.140 ± 0.033 2.122 ± 0.036 2.523 ± 0.036 1.970 ± 0.035 2.196 ± 0.039 1.970 ± 0.035 2.078 ± 0.036

2.335 ± 0.037 2.321 ± 0.040 2.657 ± 0.039 2.191 ± 0.038 2.384 ± 0.042 2.191 ± 0.038 2.279 ± 0.039

2.065

2.143

2.337

Table 39: Outlier contamination dataset (n = 1000 and σ = 1)

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

2.329 ± 0.032 1.684 ± 0.034 2.634 ± 0.034 1.390 ± 0.036 1.852 ± 0.035 1.390 ± 0.036 1.707 ± 0.034

1.952 ± 0.032 1.929 ± 0.036 2.375 ± 0.035 1.768 ± 0.035 2.020 ± 0.037 1.768 ± 0.035 1.882 ± 0.036

2.159 ± 0.034 2.138 ± 0.037 2.513 ± 0.036 2.004 ± 0.036 2.217 ± 0.039 2.004 ± 0.036 2.099 ± 0.037

1.855

1.956

2.162

29

Table 40: Sparse high-dimensional signals dataset (n = 1000 and σ = 1)

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

1.153 ± 0.016 1.039 ± 0.006 1.204 ± 0.018 0.993 ± 0.005 1.030 ± 0.006 0.993 ± 0.005 1.073 ± 0.009

1.062 ± 0.011 1.072 ± 0.011 1.134 ± 0.013 1.049 ± 0.011 1.070 ± 0.012 1.049 ± 0.011 1.091 ± 0.014

1.105 ± 0.016 1.112 ± 0.017 1.167 ± 0.017 1.093 ± 0.016 1.110 ± 0.017 1.093 ± 0.016 1.128 ± 0.019

1.069

1.075

1.115

Table 41: Covariate shift dataset (n = 1000 and σ = 1)

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

2.706 ± 0.080 1.717 ± 0.037 3.149 ± 0.095 1.013 ± 0.006 2.152 ± 0.060 1.013 ± 0.006 2.740 ± 0.101

2.373 ± 0.088 2.395 ± 0.090 2.918 ± 0.089 2.128 ± 0.091 2.582 ± 0.092 2.128 ± 0.091 2.936 ± 0.110

2.688 ± 0.105 2.705 ± 0.106 3.136 ± 0.104 2.491 ± 0.108 2.859 ± 0.107 2.491 ± 0.108 3.126 ± 0.120

2.070

2.494

2.785

Table 42: Friedman 1 dataset (n = 1000 and σ = 1)

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

3.330 ± 0.015 1.551 ± 0.009 3.301 ± 0.016 2.634 ± 0.014 2.039 ± 0.011 2.633 ± 0.014 2.172 ± 0.013

2.926 ± 0.014 2.243 ± 0.012 3.051 ± 0.016 2.907 ± 0.014 2.525 ± 0.012 2.907 ± 0.014 2.530 ± 0.014

3.175 ± 0.015 2.583 ± 0.015 3.289 ± 0.016 3.155 ± 0.015 2.830 ± 0.014 3.155 ± 0.015 2.831 ± 0.015

2.523

2.727

3.003

30

Table 43: Friedman 2 dataset (n = 1000 and σ = 1)

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF)

none

CVAR1

CVAR10

181.524 ± 1.246 23.714 ± 0.181 137.715 ± 0.782 137.720 ± 0.782 19.889 ± 0.206 137.723 ± 0.782 345.751 ± 2.424

151.614 ± 1.044 142.014 ± 0.949 151.681 ± 1.031 151.707 ± 1.032 141.642 ± 0.918 151.706 ± 1.032 170.534 ± 1.669

180.970 ± 1.603 171.767 ± 1.660 181.033 ± 1.599 181.048 ± 1.599 171.495 ± 1.637 181.049 ± 1.599 193.720 ± 2.034

140.576

151.557

180.155

average

Table 44: Friedman 3 dataset (n = 1000 and σ = 1)

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

1.048 ± 0.005 1.045 ± 0.005 1.048 ± 0.005 1.024 ± 0.005 1.054 ± 0.005 1.024 ± 0.005 1.035 ± 0.005

1.048 ± 0.005 1.029 ± 0.005 1.048 ± 0.005 1.021 ± 0.005 1.031 ± 0.005 1.021 ± 0.005 1.026 ± 0.005

1.048 ± 0.005 1.028 ± 0.005 1.048 ± 0.005 1.022 ± 0.005 1.030 ± 0.005 1.022 ± 0.005 1.026 ± 0.005

1.040

1.032

1.032

31

Table 45: Bias correction dataset

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

1.836 ± 0.003 1.208 ± 0.002 1.977 ± 0.004 1.463 ± 0.003 0.979 ± 0.002 1.463 ± 0.003 1.152 ± 0.003

1.590 ± 0.003 1.272 ± 0.003 1.738 ± 0.003 1.507 ± 0.003 1.078 ± 0.003 1.507 ± 0.003 1.235 ± 0.003

1.605 ± 0.003 1.292 ± 0.003 1.751 ± 0.003 1.523 ± 0.003 1.102 ± 0.003 1.523 ± 0.003 1.256 ± 0.003

1.440

1.418

1.436

Finally, in Tables 45–48 we show results for the following four real-life datasets: Bias correction (in full, Bias correction of numerical prediction model temperature forecast) (Cho et al., 2020) with 7750 examples and 7 features, Wine quality (Cortez et al., 2009) with 4898 examples and 12 features (including colour, red or white), Airfoil self-noise (Brooks et al., 1989) with 1503 examples and 5 features, and Student performance (Cortez, 2008) with 649 examples and 30 features. The results are mixed; in two cases, our methods slightly improve the performance of the base algorithms on average. (One of the datasets where our methods do not improve, and even slightly lower, the quality of predictions is the Wine quality dataset in Table 46; for this data set the label variable is bounded but with bounds that are far from typical values of the labels.)

C

Code Availability

Python code for reproducing our experiments in Sect. 8 and Appendix B is available at the GitHub repository https://github.com/ip200/ivar-experiments (Petej, 2024).

32

Table 46: Wine quality dataset

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

0.870 ± 0.001 0.681 ± 0.001 0.870 ± 0.001 0.733 ± 0.001 0.604 ± 0.002 0.733 ± 0.001 0.676 ± 0.001

0.870 ± 0.001 0.684 ± 0.001 0.870 ± 0.001 0.732 ± 0.001 0.618 ± 0.002 0.732 ± 0.001 0.677 ± 0.001

0.870 ± 0.001 0.684 ± 0.001 0.870 ± 0.001 0.732 ± 0.001 0.619 ± 0.002 0.732 ± 0.001 0.678 ± 0.001

0.738

0.740

0.741

Table 47: Airfoil self-noise dataset

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

5.616 ± 0.018 2.658 ± 0.016 5.541 ± 0.019 4.820 ± 0.021 1.767 ± 0.014 4.820 ± 0.020 3.797 ± 0.019 4.145

4.865 ± 0.020 3.146 ± 0.017 5.072 ± 0.022 4.670 ± 0.018 2.626 ± 0.016 4.670 ± 0.018 4.097 ± 0.018 4.164

4.992 ± 0.019 3.423 ± 0.017 5.182 ± 0.021 4.814 ± 0.018 2.957 ± 0.017 4.814 ± 0.018 4.299 ± 0.018 4.354

Table 48: Student performance dataset

Elastic Net Gradient Boosting Lasso Linear Regression Random Forest Ridge SVR (RBF) average

none

CVAR1

CVAR10

241.009 ± 0.626 231.824 ± 0.613 240.262 ± 0.633 241.279 ± 0.649 235.259 ± 0.691 240.821 ± 0.636 256.059 ± 0.745

241.617 ± 0.632 232.643 ± 0.638 240.960 ± 0.641 241.408 ± 0.635 234.641 ± 0.704 241.245 ± 0.634 245.013 ± 0.653

241.694 ± 0.640 232.996 ± 0.646 241.154 ± 0.648 241.560 ± 0.643 234.992 ± 0.702 241.422 ± 0.644 244.957 ± 0.655

240.931

239.647

239.825

33