Selective Time Series Forecasting via Metalearning Ricardo Inácio1[0009−0008−6435−5245] , Vitor Cerqueira2[0000−0002−9694−8423] , Marília Barandas3[0000−0002−9445−4809] and Carlos Soares1,3[0000−0003−4549−8917]
arXiv:2606.23448v1 [cs.LG] 22 Jun 2026
1
Faculdade de Engenharia da Universidade do Porto, Porto, Portugal {rcinacio,csoares}@fe.up.pt 2 University of Coimbra, Portugal [email protected] 3 Fraunhofer Portugal AICOS, Portugal [email protected]
Abstract. Deep learning methods have achieved state-of-the-art in time series forecasting, yet their accuracy varies considerably across samples, as some instances remain inherently difficult to predict. Reject option mechanisms, which allow models to abstain from high-risk predictions, are well established in classification and regression but underexplored in forecasting. Existing abstention strategies typically rely on proxies, such as the width of the prediction interval or learned confidence scores derived from forecasts. However, these approaches are inherently tied to the training domain, limiting their ability to generalize. We propose a selective forecasting framework that addresses this limitation by modeling the empirical percentile of forecasting errors, that is, a scale-invariant statistic, based on structural characteristics extracted from recent lags via metalearning. By decoupling the rejection decision from the forecast itself and grounding it in domain-agnostic features, the framework enables effective abstention transfer across heterogeneous time series. Experiments in both in-domain and transfer learning settings show that rejecting samples predicted as challenging consistently improves forecasting accuracy across coverage levels.
1
Introduction
Deep learning methods have become the dominant approach in time series forecasting, achieving state-of-the-art results across diverse domains [2]. Yet their accuracy is not uniform: certain samples remain inherently difficult to predict, and errors can vary substantially across a time series [1]. This motivates rejectoption mechanisms, which allow models to abstain when risk is high. Despite the non-uniform nature of forecasting errors, such mechanisms remain underexplored in time series compared with classification [9] and regression [24]. Effectively identifying hard-to-predict periods in advance would allow practitioners to selectively avoid unreliable predictions. Existing approaches to abstention in forecasting typically rely on proxy heuristics derived from forecasts, such as prediction interval width [26], nonconformity
rejected=4/32
2
92 19
94 19
96 19
R. Inácio et al.
94 19
98
20
00
96 19
20
02
Evaluation target time Actual
rejected=12/32
92 19
19
19
98
20
04
Accepted forecast
20
00
20
02
Evaluation target time
20
20
06
20
08
Rejected forecast
04
20
06
20
08
Fig. 1. Example application of the proposed method for selective forecasting. A metamodel scores each forecast origin,Actual accepting predictions expected to beRejected reliableforecast (green) Accepted forecast rejected=7/32 and rejecting those likely to incur large errors (red). Rejected forecasts occur during the most challenging periods, when predictions deviate substantially from observations.
scores [27,25], or learned confidence estimates [6]. However, these signals are inherently tied to the trained model and its domain: prediction intervals reflect 6 92 94 96 98 00 02 04 08 00 19 19 uncertainty 19 19 20 20 20 20 the model-specific calibration, and confidence scores are 2learned on targetAs time same data distribution used Evaluation for forecasting. a result, such rejection mechanisms do not generalize well when applied to new domains or time series with Actual Accepted forecast Rejected forecast different characteristics. rejected=6/32 To address this limitation, we propose a selective forecasting framework based on metalearning. The key idea is to model the empirical percentile of forecasting errors, that is, a scale-invariant statistic, based on structural features extracted from recent lags. Because these features summarize local patterns (e.g., trend, seasonality, lumpiness) rather than forecast outputs, and because percentile ranks are normalized across series,0 the rejection mechanism is decoupled 4 2 92 96 98 0 04 06 08 99 forecasting 19 both 1the 19 19 20 20 enables 20 the from model and20the data200scale. This design Evaluation time metamodel to transfer effectively acrosstarget heterogeneous domains. We train on a source domain by computing error scores and assigning them empirical percentile ranks, which serve as regression meta-targets. Time series features extracted from recent lags [4,7] serve as inputs. At inference, the metamodel outputs a continuous score for each forecast origin, rejecting those exceeding a predefined threshold q (Figure 1). We evaluate the framework in both in-domain and transfer learning settings, showing consistent forecasting accuracy gains across coverage levels. In summary, our contributions are: i) a metalearning approach for large forecasting errors estimation using structural properties of recent lags and scale-invariant targets; and ii) empirical evidence that the approach transfers effectively across domains, outperforming forecast-derived heuristics. Code and data are available online. 1
1
https://github.com/ricardoinaciopt/selective_forecasting_metalearning
Selective Time Series Forecasting via Metalearning
2
Related Work
2.1
Time Series Forecasting
3
i Let Y = {Y i }ni=1 denote a collection of n time series, where each Y i = {yji }nj=1 is a sequence of observations indexed by time j, and ni is the length of the i-th series. Global forecasting models train a single model on Y exploiting shared structure across series, often improving accuracy and data efficiency relative to local (i.e., per-series) models [16,13]. Using a context window of length p, each series is turned into input-output pairs via time-delay embedding [5]. At each time step t, the input is the window of lags Xti ∈ Rp and the output i are the subsequent h observations (i.e, the horizon) Yt+1:t+h ∈ Rh , following: i i i i i i Xt = {yt−p+1 , . . . , yt }, Yt+1:t+h = {yt+1 , . . . , yt+h }. Any time step t at which a forecast is made is denoted as forecast origin [14], where the next h points are predicted from their previous p lags. Accordingly, training amounts to fitting a global function fθ : Rp → Rh so that fθ (Xti ) = i Ŷt+1:t+h approximates the true future values across all series and time steps. Deep learning offers a wide variety of such approaches to time series forecasting, including recurrent, convolutional, attention-based, and feed-forward designs, such as NHITS [8] or KAN [17], which have been able to attain high performance and efficiency, leading to favorable results in forecasting benchmarks and competitions, for instance, GIFT-Eval [2].
2.2
Reject Option and Selective Predictions
Traditional Reject Option. Reject-option introduces an abstention mechanism into predictive systems, allowing the model to withhold predictions when the expected cost of making a decision is high. Chow’s rule formalizes this idea as an optimal decision strategy under a cost-sensitive loss: given posterior class probabilities, the classifier predicts the most probable class only when its confidence is sufficiently high, rejecting it otherwise [9]. This induces an explicit trade-off between coverage (i.e., the fraction of accepted predictions) and risk (i.e., the error on the accepted subset). This principle is commonly generalized under the selective prediction framework. Given a predictor f and a selection function s that outputs 1 (accept) or 0 (reject), the selective predictor is defined as: ( f (x), if s(x) = 1, (f, s)(x) = ⊥, if s(x) = 0, where ⊥ denotes abstention. Following this, coverage is the fraction of instances that are accepted, ϕ(s) = P[s(X) = 1], and risk is the average loss over the accepted instances, R(f, s) = E[ℓ(f (X), Y ) | s(X) = 1]. The goal is to minimize risk while maintaining sufficient coverage, or equivalently, to maximize coverage subject to a target risk [10,12].
4
R. Inácio et al.
In the forecasting setting of Section 2.1, the predictor is the global model fθ , each instance corresponds to a forecast origin t with input Xti and target i horizon Yt+1:t+h , where the selection function s(Xti ) decides whether to issue i the forecast Ŷt+1:t+h . The selective forecaster thus abstains at origins where the predicted error percentile is high, reducing average error at the cost of issuing fewer forecasts. Performance is typically evaluated using risk-coverage curves, which report the mean error on the accepted instances at each coverage level, as progressively more instances are rejected (i.e., with less coverage). Moreover, AUCO (Area Under the Confidence-Oracle Error), defined as the distance between a risk-coverage curve induced by a given model and the oracle curve obtained by ranking instances by their realized errors, and ErrDrop, defined as the ratio of the no-rejection risk to the risk at the most selective coverage level, are other usual metrics that succinctly describe the error reduction achieved [22]. Selective Classification and Regression. In classification settings, rejection is often driven by confidence scores, calibrated probabilities, ensemble disagreement, or learned selection functions [12]. In regression tasks, there is no discrete class decision, so the relevant quantity is the expected error. The loss is therefore continuous, and rejection corresponds to withholding predictions whose estimated conditional error, variance, or uncertainty is high [28,24]. A selective regression model can therefore be understood as a predictor equipped with a real-valued rejection score s(x), where larger values indicate higher expected error. For a threshold q, predictions are rejected when s(x) ≥ q. Stricter q thresholds reduce the number of issued predictions, but should also reduce the average error among the accepted predictions [12,28]. Selective Forecasting. Most forecasting methods issue predictions continuously, without mechanisms to abstain when confidence is low [11]. Some approaches provide uncertainty estimates, prediction intervals [26] or nonconformity scores [27,25], but these are designed for uncertainty quantification, not rejection. While interval width or nonconformity scores can serve as post-hoc rejection proxy heuristics, uncertainty is not equivalent to forecasting risk: wide intervals do not necessarily imply large errors, and narrow intervals may still occur in systematically missed regimes. Recent work has introduced explicit selective forecasting mechanisms. Feng et al. [11] propose rejection by ambiguity (based on error variance across training samples) and rejection by novelty (using VAEs to detect distribution shift). The Time-Energy Model [6] learns a confidence score jointly with the predictor to enable rejection under a risk-coverage trade-off. However, these methods rely on learned confidence or energy scores as proxies for risk rather than directly modeling the factors that cause large forecasting errors. Moreover, they remain tied to specific model architectures and training domains, limiting their applicability in transfer settings where rejections must generalize beyond the source data.
Selective Time Series Forecasting via Metalearning
3
5
Methodology
Our goal is to predict, before issuing a forecast, whether a global forecasting model is likely to incur a large error at a given origin. The methodology consists of two stages (Figure 2): i) performance estimation, which collects forecasting errors across origins in the source domain; and ii) metalearning, which models error percentiles as a function of structural features extracted from recent lags. By targeting a scale-invariant statistic (percentile rank) and using domain-agnostic inputs (time series features), the resulting rejection mechanism is designed to transfer effectively across domains and methods. for each series Sequential Window Extraction
Forecasting Model
Meta-feature Extraction
Performance Estimation
for each window in each series
Meta Model Fig. 2. Overview of the proposed methodology. From each time series, a sequence of time-ordered windows is extracted, each composed of p lags, the forecast origin, and the horizon h. Features are extracted from the lags, and performance estimates are obtained by measuring forecasting errors across each horizon. The metamodel is trained on the resulting predictors (M) and targets (E).
3.1
Rolling Origin Performance Estimation
The first stage of the methodology involves collecting performance estimates for metalearning. We accomplish this by conducting cross-validation over the source domain using a rolling origin evaluation procedure [14] (also referred to as time series cross-validation [15]). For each forecast origin t (Section 2.1), the forecasting model fθ is trained on all observations up to t and is used to predict the next h steps. The horizon error is computed as: i i ei,t = loss Ŷt+1:t+h , Yt+1:t+h , (1) i i where fθ (Xti ) = Ŷt+1:t+h is the forecast and Yt+1:t+h is the ground truth. Repeating this over all available series yields the set of performance estimates E = {ei,t : i = 1, . . . , n, t ∈ Ti }, where Ti denotes the valid origins for series i.
6
3.2
R. Inácio et al.
Metamodel Training
Meta-Targets. Raw performance estimates depend on problem complexity and data scale, making them unsuitable as regression targets across heterogeneous series. Instead, we transform each error into an empirical percentile rank computed within its own series: ui,t =
#{j ∈ Ti : j ̸= t, ei,j < ei,t } . |Ti | + 1
(2)
where Ti is the set of valid origins for series i. This per-series normalization maps errors to a bounded target in (0, 1), yielding a scale-invariant score that enables the metamodel to learn transferable rejection criteria across domains. Meta-Predictors. The metamodel input is derived from the lags Xti ∈ Rp preceding each forecast origin. Each lag window is transformed into a feature vector mi,t = g(Xti ), where g(·) is a feature extractor that summarizes structural characteristics such as trend, seasonality, and complexity [4,7]. The full set of meta-features is M = {mi,t : i = 1, . . . , n, t ∈ Ti }. The metamodel thus learns a mapping from structural features mi,t to error percentiles ui,t . Because both inputs and targets are domain-agnostic (features describe local patterns rather than absolute values, and percentiles are normalized within each series), the learned rejection function transfers across datasets without explicitly requiring retraining.
3.3
Inference
At inference, the metamodel scores each forecast origin before the forecast is issued. Given the lag window Xti , the meta-features mi,t = g(Xti ) are computed and passed to the metamodel, which outputs a predicted error percentile ûi,t . Because this score depends only on the input lags, not on the forecast itself, the rejection decision can be made before invoking the forecasting model, enabling pre-forecast screening, in contrast to other approaches. The rejection rule is defined by a threshold q ∈ (0, 1) corresponding to the target coverage level: origins with ûi,t ≥ q are rejected, and the remaining forecasts are issued. At coverage q, a fraction of approximately q forecasts are accepted, with the rejected corresponding to those predicted to incur the largest errors. The metamodel can be applied in two modes: (i) zero-shot, where the sourcetrained metamodel is applied directly to a new domain without modification, and (ii) domain-adapted, where the metamodel is fine-tuned on a small subset of labeled data from the target domain before deployment. The zero-shot setting tests whether the scale-invariant design transfers out of the box, and the adapted setting tests whether light supervision improves rejection quality when past target data is available.
Selective Time Series Forecasting via Metalearning
4
7
Experimental Setup
The experiments address three research questions: – RQ1: Can the proposed metamodel accurately rank forecast origins by expected error, and does rejecting those origins predicted to incur large errors improve average forecasting accuracy? – RQ2: Does the proposed approach outperform uncertainty-based and residualbased rejection baselines in both ranking quality and downstream forecasting accuracy? – RQ3: Does the scale-invariant design enable effective transfer across domains, and how does optional domain adaptation affect performance? 4.1
Datasets and Evaluation Protocol
Experiments use the widely employed M3 [19], M1 [18], and Tourism [3] univariate time series collections, restricted to monthly (M) and quarterly (Q) frequencies (Table 1). Each experiment pairs a source dataset for in-domain evaluation with a target dataset for transfer evaluation: M3→M1 and M1→Tourism. The forecast horizon h is set to 6 for monthly data and 4 for quarterly data, with a forecast input context size of p = 2 × h. Following the rolling origin procedure described in Section 3.1, each series yields multiple forecast origins. To ensure the forecasting model has sufficient training history at each origin, only the second half of each series is used for evaluation; the first half serves exclusively as warm-up data for model fitting. Figure 3 illustrates the evaluation protocol. In the source domain (left), the in-domain holdout consists of the last h observations of each series, corresponding to the final forecast horizon (solid region). All prior kept origins form the metatraining set (hatched region). In the target domain (right), the latest origins are held out for assessing transferability (solid region). In the zero-shot setting, the source-trained metamodel is applied directly to this holdout. In the domainadapted setting, earlier target origins (hatched region) are used to fine-tune (i.e., domain-adapt) the metamodel. This adaptation set can be further split to reserve a calibration subset for setting the rejection threshold q. Table 1. Summary statistics of the time series datasets used in the experiments. Dataset
Series Avg. len. Windows
M3 Monthly M3 Quarterly
1428 756
117 48
26126 8380
Tourism Monthly Tourism Quarterly
366 427
298 99
17750 9973
M1 Monthly M1 Quarterly
617 203
72 40
6698 1850
8
R. Inácio et al. domain-adapted zero-shot
transfer learning
metamodel train
in-domain hold out
domain-adaption (and calibration)
out-of-domain hold out
Fig. 3. Evaluation protocol. Left (source domain): the metamodel is trained on early origins (hatched) and evaluated on the final held-out horizon (solid). Right (target domain): zero-shot evaluation uses only the out-of-domain holdout (solid); domainadapted evaluation additionally fine-tunes on early target origins (hatched), which may include a calibration subset for threshold selection.
4.2
Models
We use two deep learning architectures for the forecasting models: KAN [17], based on Kolmogorov-Arnold Networks with learnable edge activation functions, and NHITS [8], which employs multi-rate signal processing and hierarchical interpolation for long-horizon forecasting. Both are implemented via NeuralForecast2 with automatic hyperparameter tuning on 10 random search trials. Errors ei,t are computed using sMAPE, following its use in forecasting competitions [19,20], although the proposed approach is agnostic to the metric choice. The metamodel is a CatBoostRegressor [21] that predicts empirical error percentiles ui,t from time series features mi,t . Features are extracted using TSFEL [4], comprising statistical, temporal, and spectral descriptors of the lag window preceding each forecast origin.3 Missing feature values are imputed with the median. Hyperparameters (tree depth, learning rate, L2 regularization, bagging temperature, border count, boosting iterations) are selected via random search over 30 configurations, using grouped cross-validation to keep all origins from the same series within the same fold and prevent leakage. 4.3
Baselines
We compare the metamodel against the following baselines: – PI-width: Inspired by conformal prediction [23], this baseline uses prediction interval width as the rejection score. Residuals from recent origins are normalized by the median absolute deviation (MAD), and the 90% quantile defines a per-step interval. The mean interval width across the forecasting horizon serves as the rejection score. 2 3
https://nixtlaverse.nixtla.io/neuralforecast/models.html https://tsfel.readthedocs.io/en/latest/descriptions/feature_list.html
Selective Time Series Forecasting via Metalearning
9
– Residual-scale: Measures the variability of the recent loss by computing the MAD of sMAPE values from the most recent forecast origins. Higher variability indicates less stable predictions. – Residual-variance: Pools pointwise residuals from recent origins to estimate variance, then converts this to a 90% Student-t interval width. Larger widths indicate greater uncertainty. – Random: Rejects windows uniformly at random, matching the rejection count of other methods at each coverage level. Serves as a lower bound. – Oracle: Ranks windows by their realized sMAPE and rejects the highesterror windows first. Serves as an upper bound, representing the best achievable performance when true errors are known.
5
Results
We evaluate the metamodel at two levels: (i) meta-level, assessing whether predicted risk scores rank forecast origins by actual error, and (ii) base-level, measuring downstream forecasting accuracy after rejection. 5.1
Meta-level Evaluation
Table 2 compares the predicted large error scores against the realized errors across source (S), zero-shot transfer (Z), and domain-adapted transfer (A) settings. Metrics include Spearman’s ρ (rank correlation between ûi,t and ui,t ), AUCO, and ErrDrop (Section 2.2). Metamodel performance is consistent across both forecasters, indicating method-agnostic behavior. In-domain Settings (S). Spearman correlation (ρ) is consistently high (i.e., between 0.71 and 0.90), indicating that the metamodel correctly assigns higher scores to windows that actually lead to the largest errors. This is further supported by the small AUCO values the metamodel attains, which are closest to the oracle across all settings. The ErrDrop metric also shows that selective rejection can substantially reduce the average error on accepted forecasts at the extremes, as the metamodel is always either the best or the second-best. Transfer Settings (Z/A). Zero-shot transfer degrades metamodel performance relative to source (S), as expected, due to domain shift, with lower ρ, larger AUCO, and generally smaller ErrDrop, though still outperforming most baselines. Domain adaptation using 30% of target origins substantially recovers performance: ρ and AUCO improve in all cases, and ErrDrop improves in all but one (M1-M→T-M). Bold values indicate the better transfer setting. The baselines only outperform the metamodel in isolated cases. Residual scale is occasionally stronger in zero-shot transfer for ρ and AUCO, but these gains are small and disappear after target adaptation. PI width and Error Variance win on ErrDrop only in a few source-domain cases, indicating that
10
R. Inácio et al.
they sometimes identify the most extreme errors, but do not provide reliable transfer rankings. A dash indicates that the statistic is undefined because the corresponding score is constant or fully tied over the evaluated windows. Table 2. Evaluation of the metamodel alongside NHITS and KAN forecasting models. Each cell reports values for source holdout (S), zero-shot transfer (Z), and domain adaptation (A). The best method in each setting is bolded, and the second best is underlined. Spearman (ρ) measures rank association, AUCO measures area under calibration-oracle error curve, and ErrDrop is the keep-all to most-selective risk ratio. A dash indicates that the corresponding metric is constant or fully tied over the evaluated windows. Source Target Models Method M3-M M1-M KAN
ρ ↑ (S; Z/A)
AUCO↓ (S; Z/A)
ErrDrop↑ (S; Z/A)
Ours 0.894; 0.628/0.820 0.007; 0.043/0.013 19.32; 10.06/14.23 PI width 0.687; 0.158/0.158 0.023; 0.124/0.124 11.50; 0.67/0.67 Res. scale 0.804; 0.536/0.536 0.012; 0.055/0.055 12.38; 1.71/1.71 Err. var. 0.818; 0.152/0.152 0.012; 0.122/0.122 15.97; 0.76/0.76
NHITS Ours 0.899; 0.571/0.812 0.006; 0.045/0.013 28.58; 19.18/15.39 PI width 0.671; 0.110/0.110 0.024; 0.140/0.140 10.78; 0.45/0.45 Res. scale 0.801; 0.536/0.536 0.012; 0.047/0.047 15.92; 1.27/1.27 Err. var. 0.815; 0.105/0.105 0.012; 0.134/0.134 19.17; 1.14/1.14 M3-Q M1-Q KAN
Ours 0.750; 0.529/0.755 0.008; 0.059/0.013 PI width 0.556; 0.220/0.220 0.016; 0.093/0.093 Res. scale 0.466; 0.535/0.535 0.022; 0.056/0.056 Err. var. 0.663; 0.311/0.311 0.012; 0.058/0.058
4.74; 9.38/11.77 6.24; 0.44/0.44 1.66; 0.78/0.78 4.23; 7.73/7.73
NHITS Ours 0.747; 0.515/0.734 0.007; 0.037/0.013 10.54; 11.66/15.55 7.75; 0.50/0.50 PI width 0.612; 0.165/0.165 0.013; 0.086/0.086 Res. scale 0.520; 0.537/0.537 0.018; 0.047/0.047 1.87; 0.84/0.84 Err. var. 0.695; 0.199/0.199 0.010; 0.068/0.068 5.90; 7.38/7.38 M1-M T-M
KAN
Ours 0.752; 0.494/0.717 0.016; 0.058/0.035 PI width 0.213; -0.272/-0.272 0.106; 0.180/0.180 Res. scale 0.485; 0.519/0.519 0.036; 0.053/0.053 Err. var. 0.210; -0.316/-0.316 0.092; 0.186/0.186
4.99; 2.58/2.44 0.27; 0.50/0.50 2.74; 2.44/2.44 1.21; 0.47/0.47
NHITS Ours 0.766; 0.559/0.741 0.014; 0.054/0.033 11.01; 2.41/2.91 PI width 0.209; -0.251/-0.251 0.087; 0.188/0.188 0.35; 0.52/0.52 Res. scale 0.482; 0.473/0.473 0.038; 0.067/0.067 1.92; 1.58/1.58 Err. var. 0.204; -0.288/-0.288 0.082; 0.195/0.195 0.62; 0.47/0.47 M1-Q T-Q
KAN
Ours 0.710; 0.544/0.764 0.015; 0.053/0.030 PI width 0.247; -0.016/-0.016 0.059; 0.125/0.125 Res. scale –; 0.255/0.255 0.049; 0.099/0.099 Err. var. 0.257; -0.014/-0.014 0.062; 0.123/0.123
2.88; 2.92/3.28 2.21; 1.33/1.33 1.07; 1.64/1.64 3.08; 1.80/1.80
NHITS Ours 0.733; 0.525/0.751 0.013; 0.060/0.035 10.85; 3.05/2.76 2.14; 1.28/1.28 PI width 0.247; 0.004/0.004 0.058; 0.136/0.136 Res. scale –; 0.287/0.287 0.050; 0.096/0.096 0.98; 1.48/1.48 Err. var. 0.269; 0.008/0.008 0.058; 0.135/0.135 2.81; 1.58/1.58
Figure 4 shows one example using M3 Monthly as the source and M1 Monthly as the target in risk-coverage curves. In the source domain (left), the metamodel (green diamond) is consistently closest to the oracle (orange square) across rejection levels, indicating that it is generally the best at predicting high-error forecasts. The other baselines, even though they leverage the actual residuals to compute risk scores, only become competitive at higher levels. As expected, the random baseline remains relatively unaltered because it leverages no information about which forecasts to reject. In the transfer domain plots (center and right), PI width and Residual-variance severely underperform, being worse
Selective Time Series Forecasting via Metalearning
11
Fig. 4. Risk-Coverage plots for the NHITS forecaster, using M3 Monthly as the source domain and M1 Monthly as the target domain. The metamodel (green diamond) is closest to the oracle in the source domain across most coverage levels, and it consistently becomes the closest after domain adaptation in the target domain.
than the random baseline around the 0.5 level. This indicates that past error variance becomes progressively decoupled from transfer learning forecasting risk as the rejection fraction increases. Only Residual-scale approximates the metamodel, though it becomes unreliable at the largest rejections. However, applying domain adaptation (right) immediately approximates the metamodel to the oracle upper bound, demonstrating that it is crucial for robustness when domain shift is present. 5.2
Base-level Evaluation
We simulate sequential deployment in the target domain to assess the impact of the metamodel at the base level. The metamodel is domain-adapted using the earlier target windows, and an abstention threshold is computed for several fixed rejection fractions q on the calibration windows. It is then applied to the held-out windows to assess whether the metamodel effectively reduces accepted-forecast error as the thresholds increase (i.e., risk, as defined in Section 2.2). Table 3 showcases results for both forecasters on all datasets, when no forecast is rejected, and across increasing rejection thresholds q, for all methods. We also show the average distance to the oracle upper bound at the end, representing the error if we actually rejected the top-q worst windows at each reported level. Once again, the similar improvements attained across both forecasters indicate that the metamodel is model-agnostic. We consistently improve over the Keep all baseline at each level, and improve over the other baselines in most (in bold). This is also evidenced by the average gap to the oracle across q, at which our method consistently achieves the lowest. Nonetheless, when our method is not the best, usually at lower q values, it is the second best, denoted by an underline. Moreover, a series-level bootstrap confirmed that the metamodel is significantly closer to the oracle than all baselines in all settings, except Residual-scale for M1-Q/NHITS (p = 0.191). Figure 5 showcases whether abstaining from forecasts rejected by the metamodel improves the average error on retained forecasts. The left plot reports the mean sMAPE across accepted forecasts for several rejection levels q and all
12
R. Inácio et al.
Table 3. Base-level results on the transfer target datasets across q rejection fractions. Keep all is target-domain sMAPE without rejection. Baseline lists the abstention methods used. Each q reports sMAPE on kept forecasts by each method. Avg. gap reports the average distance to the oracle across q. Lower values and smaller gaps are better. Target Models Keep all↓ Baseline T-M
0.288 Ours 0.278 PI width 0.287 Res. scale 0.278 Err. var. 0.290 0.259
0.241
0.214
0.191
0.172
0.000
NHITS
0.307 Ours 0.295 PI width 0.304 Res. scale 0.298 Err. var. 0.307
0.284 0.310 0.290 0.313
0.264 0.322 0.277 0.326
0.244 0.330 0.262 0.335
0.226 0.343 0.249 0.345
0.034 0.094 0.047 0.097
0.275
0.256
0.226
0.203
0.182
0.000
KAN
0.272 Ours 0.239 PI width 0.260 Res. scale 0.265 Err. var. 0.259
0.229 0.262 0.258 0.260
0.210 0.261 0.247 0.260
0.193 0.260 0.241 0.259
0.180 0.259 0.236 0.256
0.021 0.071 0.060 0.069
0.235
0.217
0.188
0.164
0.144
0.000
NHITS
0.304 Ours 0.278 PI width 0.295 Res. scale 0.291 Err. var. 0.292
0.266 0.295 0.281 0.294
0.245 0.295 0.267 0.294
0.229 0.293 0.259 0.293
0.211 0.293 0.253 0.290
0.025 0.073 0.050 0.072
Oracle
Oracle T-Q
q = 0.05 q = 0.10 q = 0.20 q = 0.30 q = 0.40 Avg. gap↓
KAN
Oracle
Oracle M1-Q KAN
0.249 0.304 0.255 0.309
0.233 0.313 0.241 0.316
0.215 0.325 0.228 0.329
0.033 0.089 0.039 0.093
0.268
0.250
0.220
0.195
0.171
0.000
0.143 Ours 0.124 PI width 0.139 Res. scale 0.125 Err. var. 0.139
0.115 0.134 0.110 0.138
0.061 0.133 0.096 0.140
0.050 0.131 0.091 0.144
0.042 0.131 0.084 0.139
0.019 0.074 0.042 0.081
Oracle NHITS
0.267 0.291 0.268 0.295
0.095
0.074
0.054
0.041
0.033
0.000
0.120 Ours 0.103 PI width 0.113 Res. scale 0.098 Err. var. 0.113
0.098 0.109 0.087 0.112
0.060 0.107 0.069 0.109
0.048 0.105 0.067 0.112
0.040 0.103 0.067 0.110
0.016 0.053 0.023 0.057
0.067
0.050
0.039
0.031
0.000
Oracle
0.084
methods. The no-abstention baseline (keep all) remains essentially unchanged, as does random rejection. Although the PI width and Error variance baselines slightly improve over random and no abstention, both remain constant across q, showcasing their limitations on attainable improvement. Both the Residual scale baseline and the metamodel rejection curves decrease as q increases, with the latter decreasing more prominently. The persistent gap between the metamodel and the oracle, which is nearly half of the Residual scale gap, indicates that, even across rejection thresholds, our method remains the most reliable. The right plot compares the error distributions for all, the retained, and the rejected forecasts by each method at the strictest q value (0.05). It is clear that forecasts rejected by the metamodel, Residual scale, and Error variance exhibit larger errors than those retained, indicating that they concentrate large
Selective Time Series Forecasting via Metalearning
13
Fig. 5. Impact of the selective forecasting mechanism at base level across q thresholds, from M1-T to T-Q. On the left, even at higher q (more rejections), the metamodel remains close to the oracle. The right plot shows the distributions of all errors across windows, along with those kept and rejected at q = 0.05 for each method.
errors in the rejected subset. Our method (green) is particularly effective, as its distribution of rejected forecasts is closest to the oracle distribution (orange).
6
Discussion, Limitations, and Conclusion
We present a selective forecasting framework that predicts empirical error percentiles from structural features of recent observations, enabling abstention before the forecast is issued. The scale-invariant targets and domain-agnostic features allow the mechanism to transfer across heterogeneous domains. Research Questions. Results show that the metamodel can identify instances associated with large forecasting errors by predicting empirical error percentiles, and that using these to selectively forecast improves downstream performance on the retained windows (RQ1). This indicates that the extracted meta-features contain useful information about forecasting difficulty, even before the forecast is produced. The fact that the metamodel learns patterns from several time series via cross-learning may explain its ability to capture general conditions for forecastability across heterogeneous windows. The proposed method is competitive with uncertainty-based and residualbased post-hoc baselines for identifying windows associated with large forecasting errors, both in-domain and in transfer learning settings (RQ2). Transfer learning results also show that the metamodel is robust across most settings, particularly when domain adaptation is applied. This indicates that forecastability information can generalize across datasets, although distribution shift can reduce metamodel performance when source meta-features are not representative of the target domain. Unlike methods that employ prediction intervals or residual behavior, which are only available after the forecast is produced and may require direct knowledge of recent errors, the proposed method operates ex ante. It only uses descriptors extracted from lags, making it suitable as a pre-forecast triage mechanism.
14
R. Inácio et al.
Results further show that abstaining from forecasts rejected by the metamodel improves average predictive performance on the retained forecasts, producing competitive selective-risk trade-offs relative to uncertainty-based and residual-based post-hoc baselines in transfer learning settings (RQ3). This suggests that the rejected windows correspond to conditions under which the forecaster is more likely to incur large errors, while the retained forecasts are more closely aligned with previously observed predictable conditions. Limitations. The method relies heavily on the representativeness of the metatraining data, as domains with shorter series can make the rejection mechanism less reliable. Moreover, the method estimates relative forecasting risk rather than calibrated probabilistic guarantees, and thus does not provide formal quantifications of uncertainty. Moreover, given the novelty of the approach, all comparative baselines are based on recently incurred residuals, which makes them harder to beat but may not be representative of other, narrower approaches in the literature (e.g., those applicable only to time series foundation models). Results suggest that forecastability can be treated as a structural property of recent observations, enabling risk-aware decisions before the forecast horizon is realized. Future directions include integrating the rejection mechanism with conformal prediction to achieve calibrated coverage guarantees, extending to multivariate and irregular time series, and exploring joint training of the forecaster and the rejection model. Acknowledgments. This work was partially funded by projects AISym4Med (101095387) supported by Horizon Europe Cluster 1: Health, ConnectedHealth (n.º 46858), supported by Competitiveness and Internationalisation Operational Programme (POCI) and Lisbon Regional Operational Programme (LISBOA 2020), under the PORTUGAL 2020 Partnership Agreement, through the European Regional Development Fund (ERDF) and Agenda “Center for Responsible AI”, nr. C645008882-00000055, investment project nr. 62, financed by the Recovery and Resilience Plan (PRR) and by European Union - NextGeneration EU, and also by FCT plurianual funding for 20202023 of LIACC (UIDB/00027/2020 UIDP/00027/2020);
References 1. Aiolfi, M., Timmermann, A.: Persistence in forecasting performance and conditional combination strategies. Journal of Econometrics 135(1-2), 31–53 (2006) 2. Aksu, T., Woo, G., Liu, J., Liu, X., Liu, C., Savarese, S., Xiong, C., Sahoo, D.: GIFT-eval: A benchmark for general time series forecasting model evaluation (2024) 3. Athanasopoulos, G., Hyndman, R.J., Song, H., Wu, D.C.: The tourism forecasting competition. International Journal of Forecasting 27(3) (2011) 4. Barandas, M., Folgado, D., Fernandes, L., Santos, S., Abreu, M., Bota, P., Liu, H., Schultz, T., Gamboa, H.: Tsfel: Time series feature extraction library. SoftwareX 11 (2020)
Selective Time Series Forecasting via Metalearning
15
5. Bontempi, G., Ben Taieb, S., Le Borgne, Y.A.: Machine learning strategies for time series forecasting. In: European big data management and analytics summer school, pp. 62–77. Springer (2012) 6. Brusokas, J., Tirupathi, S., Zhang, D., Pedersen, T.B.: The time-energy model: Selective time-series forecasting using energy-based models. Transactions on Machine Learning Research (2025) 7. Cerqueira, V., Moniz, N., Soares, C.: Vest: Automatic feature engineering for forecasting. Machine Learning 113(7), 4523–4545 (2024) 8. Challu, C., Olivares, K.G., Oreshkin, B.N., Ramirez, F.G., Canseco, M.M., Dubrawski, A.: Nhits: Neural hierarchical interpolation for time series forecasting. In: Proceedings of the AAAI conference on artificial intelligence. vol. 37, pp. 6989–6997 (2023) 9. Chow, C.: On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory 16(1), 41–46 (1970) 10. El-Yaniv, R., et al.: On the foundations of noise-free selective classification. Journal of Machine Learning Research 11(5) (2010) 11. Feng, N., Lai, S., Zhou, X., Yang, J., Feng, K., Yin, Z., Zhou, F., Hu, Z., Yue, Y., Liang, Y., et al.: Towards reliable time series forecasting under future uncertainty: Ambiguity and novelty rejection mechanisms (2025) 12. Geifman, Y., El-Yaniv, R.: Selective classification for deep neural networks. Advances in neural information processing systems 30 (2017) 13. Godahewa, R., Bandara, K., Webb, G.I., Smyl, S., Bergmeir, C.: Ensembles of localised models for time series forecasting. Knowledge-Based Systems 233, 107518 (2021) 14. Hewamalage, H., Ackermann, K., Bergmeir, C.: Forecast evaluation for data scientists: common pitfalls and best practices: H. hewamalage et al. Data Mining and Knowledge Discovery (2023) 15. Hyndman, R.J., Athanasopoulos, G.: Forecasting: principles and practice. OTexts (2018) 16. Januschowski, T., Gasthaus, J., Wang, Y., Salinas, D., Flunkert, V., BohlkeSchneider, M., Callot, L.: Criteria for classifying forecasting methods. international Journal of forecasting 36(1), 167–177 (2020) 17. Liu, Z., Wang, Y., Vaidya, S., Ruehle, F., Halverson, J., Soljacic, M., Hou, T., Tegmark, M.: Kan: Kolmogorov–arnold networks. In: International conference on learning representations. vol. 2025, pp. 70367–70413 (2025) 18. Makridakis, S., Andersen, A., Carbone, R., Fildes, R., Hibon, M., Lewandowski, R., Newton, J., Parzen, E., Winkler, R.: The accuracy of extrapolation (time series) methods: Results of a forecasting competition. Journal of forecasting (1982) 19. Makridakis, S., Hibon, M.: The m3-competition: results, conclusions and implications. International journal of forecasting 16(4), 451–476 (2000) 20. Makridakis, S., Spiliotis, E., Assimakopoulos, V.: The m4 competition: Results, findings, conclusion and way forward. International Journal of forecasting (2018) 21. Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A.V., Gulin, A.: Catboost: unbiased boosting with categorical features. Advances in neural information processing systems 31 (2018) 22. Scalia, G., Grambow, C.A., Pernici, B., Li, Y.P., Green, W.H.: Evaluating scalable uncertainty estimation methods for deep learning-based molecular property prediction. Journal of chemical information and modeling 60(6), 2697–2717 (2020) 23. Shafer, G., Vovk, V.: A tutorial on conformal prediction. Journal of machine learning research 9(3) (2008)
16
R. Inácio et al.
24. Shah, A., Bu, Y., Lee, J.K., Das, S., Panda, R., Sattigeri, P., Wornell, G.W.: Selective regression under fairness criteria. In: International Conference on Machine Learning. PMLR (2022) 25. Stankeviciute, K., M Alaa, A., Van der Schaar, M.: Conformal time-series forecasting. Advances in neural information processing systems 34, 6216–6228 (2021) 26. Szabadváry, J.H., Löfström, T., Johansson, U., Sönströd, C., Ahlberg, E., Carlsson, L.: Classification with reject option: Distribution-free error guarantees via conformal prediction. Machine Learning with Applications p. 100664 (2025) 27. Vovk, V., Gammerman, A., Shafer, G.: Algorithmic learning in a random world. Springer (2005) 28. Wiener, Y., El-Yaniv, R.: Pointwise tracking the optimal regression function. Advances in Neural Information Processing Systems 25 (2012)