Training-Free Probabilistic Time-Series Forecasting with Conformal Seasonal Pools Valery Manokhin∗
arXiv:2605.03789v1 [stat.ML] 5 May 2026
May 2026
Abstract We propose Conformal Seasonal Pools (CSP), a training-free probabilistic time-series forecaster that mixes same-season empirical draws with signed residual draws around a seasonal naive forecast. In an audited rolling-origin benchmark on the six time-series datasets where DeepNPTS was originally evaluated by Rangapuram et al. (2023) (electricity, exchange rate, solar energy, taxi, traffic, wikipedia), CSP-Adaptive significantly outperforms DeepNPTS on every metric we report—CRPS (per-window paired Wilcoxon p ≈ 4 × 10−10 ), normalized mean quantile loss (p ≈ 7 × 10−10 ), and empirical 95% coverage (p ≈ 8 × 10−45 , mean 0.89 vs 0.66)—while running over 500× faster on CPU. Coverage is the most decisioncritical of these: a 0.95 nominal interval that contains the truth in only ∼ 66% of cases fails the basic calibration desideratum of Gneiting and Katzfuss (2014) and would not survive deployment in safety- or decision-critical settings. The failure mode is also more severe than aggregate coverage suggests: in the worst 10% of windows, DeepNPTS’s prediction interval covers none of the H forecast horizons — the entire multi-step trajectory misses the truth at every step simultaneously. This poses serious risk in safety- and decision-critical applications such as healthcare, finance, energy operations, and autonomous systems, where prediction intervals that systematically miss the truth across the entire planning horizon translate directly into misclassified patients, regulatory capital failures, grid imbalances, and safety-case violations. CSP achieves all of this with no learned parameters and no training. We argue training-free conformal samplers should be mandatory baselines when evaluating learned non-parametric forecasters.
Keywords: Conformal prediction, probabilistic forecasting, time series, predictive distributions, CRPS, quantile loss, training-free baselines, seasonal naive, DeepNPTS.
1
Introduction
Probabilistic forecasting asks for a predictive distribution over future time-series values, not only a point forecast. This is the natural object in inventory planning, energy operations, transport, and other settings where decisions depend on uncertainty. Recent forecasting work has moved toward neural distributional models, with N-BEATS (Oreshkin et al., 2020) a foundational example alongside Temporal Fusion Transformers (Lim et al., 2021) and N-HiTS (Challu et al., 2023); DeepNPTS (Rangapuram et al., 2023) is the corresponding non-parametric sampling baseline within that family and is the principal neural comparator we benchmark against. That shift is valuable, but it has also made it easy for low-compute baselines to be under-specified or omitted. ∗
Independent researcher. [email protected]
1
This paper studies a deliberately practical question: before training a neural sampler, how far can training-free empirical and conformal methods go? The question matters because many production forecasting systems need transparent baselines, fast reruns, cheap sensitivity checks, and failure modes that can be audited. A method whose intervals are well calibrated and whose runtime is in seconds is easier to deploy and audit than one that ships sharper intervals at the cost of nominal coverage and hours of training. Sharp intervals frequently hide severe undercoverage, failing the basic calibration desideratum of Gneiting and Katzfuss (2014) and posing tangible risks in safety- and decision-critical applications—health screening, autonomous driving, finance, and energy operations—where a prediction interval that systematically misses the truth is not merely inaccurate but actively misleading. We evaluate that question against NPTS, SeasonalNPTS, and DeepNPTS. The main method family is Conformal Seasonal Pools (CSP), a simple sampler that mixes same-season empirical draws with signed residual draws around a seasonal naive forecast. CSP has no learned parameters, trains no neural network, and produces a full empirical predictive sample from which intervals, quantiles, CRPS, and quantile losses can be computed. The claim is direct. On the six datasets where DeepNPTS was originally evaluated, CSPAdaptive significantly outperforms DeepNPTS on per-window CRPS (p ≈ 4 × 10−10 ), on normalized mean quantile loss (p ≈ 7 × 10−10 ), and on empirical 95% coverage (p ≈ 8 × 10−45 , mean 0.89 vs 0.66). The coverage gap is the most decision-critical: under the calibration-thensharpness paradigm of Gneiting and Katzfuss (2014), a method whose nominal 95% interval misses the truth roughly a third of the time fails the basic calibration prerequisite and is inadmissible for any setting that takes the nominal level seriously. CSP runs in seconds where DeepNPTS runs in minutes (567× slower under the audited CPU protocol). The result is not a benchmarking marginal: training-free seasonal residual samplers beat a neural non-parametric forecaster on its own evaluation suite, on every metric that matters, by orders of magnitude in significance. Contributions. The contributions of this paper are twofold. We propose Conformal Seasonal Pools (CSP), a training-free probabilistic time-series forecaster that mixes same-season empirical draws with signed conformal residual draws around a seasonal naive forecast, with a static and an adaptive mixture-weight rule. We then run an audited rolling-origin benchmark of CSP against NPTS, SeasonalNPTS, and DeepNPTS on the 6 datasets where DeepNPTS was originally evaluated by Rangapuram et al. (2023), organising the comparison around the calibrationthen-sharpness reporting protocol of Gneiting and Katzfuss (2014) and reporting per-dataset rank distributions, per-window paired Wilcoxon significance tests, and audited CPU wall time alongside CRPS, normalized mean quantile loss, and empirical coverage.
2
Related Work
Probabilistic and neural forecasting. GluonTS provides a common toolkit for probabilistic forecasting experiments and reference implementations (Alexandrov et al., 2020). Recent neural forecasting work includes Temporal Fusion Transformers for multi-horizon forecasting with covariates and interpretability (Lim et al., 2021), N-BEATS for deep univariate forecasting (Oreshkin et al., 2020), and N-HiTS for efficient long-horizon forecasting (Challu et al., 2023). DeepNPTS is the principal comparator we benchmark against: it revisits non-parametric forecasting through a global learned sampler, keeping the non-parametric goal of sampling from observed values while learning the sampling strategy from related series (Rangapuram et al., 2023). Our benchmark does not claim to replace this literature. Instead, it asks whether a simpler 2
sampler should be part of the standard comparison set. If a training-free method is competitive with a neural non-parametric sampler on proper scoring rules and dramatically faster, it changes the default burden of proof for more complex models. Conformal prediction for time series. Conformal prediction provides distribution-free uncertainty sets under exchangeability (Lei et al., 2018; Shafer and Vovk, 2008; Vovk et al., 2005) and has been extended in several directions for dependent or shifting data. EnbPI constructs prediction intervals for dynamic time series without requiring full exchangeability (Xu and Xie, 2021, 2023). Adaptive conformal inference treats coverage under distribution shift as an online control problem (Gibbs and Candès, 2021). Conformalized quantile regression (Romano et al., 2019) adapts the conformal scaffold to quantile-based score functions. It is worth being explicit about the state of the art here: in the general time-series setting no conformal method delivers an unconditional finite-sample distribution-free coverage guarantee. The classical splitconformal bound (Vovk et al., 2005) requires exchangeability, which serial dependence in time series violates; EnbPI’s coverage holds approximately under stationarity-and-mixing conditions; and ACI (Gibbs and Candès, 2021) provides long-run online frequentist coverage rather than finite-sample marginal coverage. So all conformal time-series predictors operate on a spectrum from “approximate validity under stated conditions” to “asymptotic / online validity”; the absence of a hard finite-sample guarantee is the norm in this setting, not a CSP-specific deficit. Conformal predictive distributions. A complementary line of work, conformal predictive distributions and conformal predictive systems, takes the conformal toolkit beyond intervals to produce calibrated predictive CDFs (Vovk et al., 2017, 2018a,b, 2020). Under exchangeability, a conformal predictive distribution F̂ has the probability-integral-transform property F̂ (Y ) ∼ Uniform(0, 1), which is a strictly stronger calibration criterion than empirical 1 − α coverage at a single nominal level. The CSP method introduced here is an empirical-sample construction rather than a CPS in the strict sense. CSP borrows the conformal habit of using empirical residuals, but it is a predictive-distribution sampler rather than only an interval constructor. The method uses signed seasonal residuals to preserve asymmetry and combines them with a same-season empirical pool to retain realistic distributional support. Forecasting benchmarks and scoring. The M competitions, the Monash archive, and recent forecasting benchmark suites have made clear that conclusions depend on dataset diversity, scoring rules, and reproducibility choices (Godahewa et al., 2021; Makridakis et al., 2018, 2020, 2022; Qiu et al., 2024; Zhang et al., 2024). For probabilistic forecasts, strictly proper scoring rules and calibration-sharpness analysis are central (Gneiting and Raftery, 2007; Gneiting et al., 2007; Hersbach, 2000). We therefore use empirical CRPS as the primary distributional score and normalized mean quantile loss as a complementary quantile score.
3
Methods
Let y1 , . . . , yT be the observed history and let H be the forecast horizon. Every method returns an empirical sample matrix S ∈ RH×B , with B = 100 samples in the audited run. Quantile forecasts and prediction intervals are derived from those samples. Empirical pools and NPTS baselines. The simplest empirical forecaster samples directly from historical observations. A rolling empirical pool restricts the sample to recent history, while
3
a seasonal empirical pool samples from the same seasonal phase as the target horizon. NPTS and SeasonalNPTS are related non-parametric samplers that weight historical observations by recency and, for SeasonalNPTS, by seasonal phase. These methods are robust and fast, but they can fail in predictable ways. A full-history empirical pool ignores level shifts and seasonality. A rolling pool adapts to level shifts but loses seasonal structure. A seasonal pool can become too small for yearly or short histories. NPTS-style recency weighting helps, but it does not by itself recenter forecasts around the latest seasonal level. Conformal residual samplers. A seasonal naive forecast uses µh = yT +h−m when a seasonal period m is available, falling back to the latest observation when necessary. Residual conformal samplers compute calibration residuals and then generate samples of the form µh + r, where r is drawn from an empirical residual pool. In this benchmark the residual pool is signed rather than absolute when the method is used to generate a full distribution. The residual view captures local level and recent seasonal changes better than raw empirical sampling. Its weakness is that residual pools can be horizon-blind or too symmetric if implemented only as absolute interval scores. Conformal Seasonal Pools. CSP combines the two views. For each horizon h, it constructs a predictive sample from a mixture Sh ∼ wh Fbhseason + (1 − wh ) Fbhresid , where Fbhseason samples observed values from the same seasonal position as T +h with exponential recency weights, and Fbhresid samples µh + r for r a signed residual from the calibration window. CSP-Fixed uses wh = 0.5 throughout. CSP-Adaptive reduces the seasonal-pool weight when seasonality is absent (m ≤ 1) or when too few seasonal cycles are observed, recovering the pure residual sampler in the degenerate case. Algorithm 1 states the procedure formally. Conformal validity remarks. The name “Conformal Seasonal Pools” is intended to indicate that the residual component is motivated by split-conformal calibration, not that the full mixture inherits a finite-sample coverage proof. The signed-residual pool plays the role of a split-conformal calibration set in the spirit of Papadopoulos et al. (2002), who introduced inductive (split) conformal prediction for regression, but the construction we use—signed residuals, an empirical centred interval extracted from the predictive sample, and a calibration window taken from the most recent half of training history under residual exchangeability that holds only approximately for time series—differs in details from the canonical split-conformal interval whose finite-sample bound is proved in that reference. We therefore make no formal coverage claim, even for the residual-only sub-sampler, and we make no formal coverage claim for the full CSP mixture, where the seasonal empirical pool contributes to sharpness without a coverage proof. The strong empirical performance of split-conformal-style constructions in the timeseries setting, despite the formal exchangeability violation, has recently received a theoretical explanation. Oliveira et al. (2024) prove that split conformal is approximately valid for nonexchangeable data with quantifiable degradation; Barber and Pananjady (2025) sharpen this by bounding the coverage loss of split conformal under temporal dependence with a “switch coefficient” that quantifies the exchangeability violation, and show this characterization is sharp over the class of stationary β-mixing processes. Together these results provide post-hoc justification for the design choice we make: split-conformal residual calibration is empirically effective for time series even though the canonical finite-sample guarantee no longer applies, and the
4
Algorithm 1: Conformal Seasonal Pools (CSP-Adaptive). CSP-Fixed sets wh ≡ 0.5 throughout. Input : history y1:T , horizon H, seasonal period m, sample budget B, calibration fraction ρ, recency rate λ Output: predictive sample matrix S ∈ RH×B Tcal ← ⌊ρ T ⌋ R ← yt − yt−m : T − Tcal < t ≤ T, t > m
(signed-residual calibration pool)
for h ← 1 to H do µh ← yT +h−m (seasonal-naive forecast; fall back to nearest same-phase value if out of range) Sh ← yt : t ≤ T, t ≡ T + h (mod m) , with weights ∝ exp −λ (T − t) if m ≤ 1 then wh ← 0 else if |Sh | < 3 then wh ← 0.3 else wh ← 0.5
(no seasonality) (seasonal pool too thin)
for b ← 1 to B do u ∼ Uniform(0, 1) if u < wh then Sh,b ← weighted draw from Sh else Sh,b ← µh + r with r ∼ Uniform(R) return S
5
gap between effective and guaranteed coverage is now characterised in the literature. As discussed in Section 2 on conformal prediction for time series, this is the norm rather than the exception in the time-series CP literature: EnbPI, ACI, AgACI, and CTSF all rely on conditions (stationarity-and-mixing, asymptotic / online behaviour, per-series exchangeability) that themselves cannot be checked from finite data, and none delivers an unconditional finite-sample coverage guarantee in the general case. CSP is in good company in foregoing such a guarantee. Coverage is reported throughout this paper as an empirical finite-sample property, and is one of the metrics the method is judged on rather than one it is guaranteed to satisfy. Mixture weights and coverage. At the values used in this paper (wh ∈ {0, 0.3, 0.5}) at least half of every CSP draw comes from the conformally-motivated residual pool, and when wh = 0 the mixture is the residual-only sampler exactly. We do not state a closed-form bound on mixture coverage in terms of component coverages, since the prediction interval is extracted from the mixture sample as a data-dependent quantile and does not decompose linearly across components; Section 5.3 reports the empirical coverage achieved instead.
4
Experimental Design
The audited evaluation covers six datasets from the GluonTS reference suite—electricity, exchange rate, solar energy, taxi, traffic, and wikipedia—which is exactly the evaluation set used by Rangapuram et al. (2023) to introduce DeepNPTS. We compare six methods: three trainingfree baselines we propose (CSP-Adaptive, CSP-Fixed, AdaptiveWindowMCI) and three external comparators from the GluonTS NPTS family (NPTS, SeasonalNPTS, DeepNPTS). The benchmark produces 380 forecast records per method (≈ 2,280 result rows in total) under a rolling-origin protocol with matched windows and seeds across methods, so per-(dataset, series, window) comparisons are properly paired. Every row is generated under a rolling-origin protocol with a fixed horizon per dataset and the same random seeds across methods, so the per-dataset comparisons are matched. We report empirical CRPS, normalized mean quantile loss over quantiles 0.1 to 0.9, empirical 95% interval coverage, interval width, and audited method wall time. Lower CRPS and normalized MQL are better. Coverage is descriptive because the benchmark ranks methods by distributional scores rather than by interval coverage alone. Runtime is extracted from logs. For non-DeepNPTS methods, wall time is the logged elapsed time for each parallel method evaluation. For DeepNPTS, wall time is the elapsed time from each train-start line to the corresponding saved-result line. Dataset download/loading and final summary writing are excluded.
5
Results
5.1
Aggregate accuracy
CSP-Adaptive achieves the best mean CRPS rank (3.03) and the best mean MQL rank (2.92) of the six methods, ahead of DeepNPTS at 3.60 and 3.52 respectively. CSP-Fixed is essentially tied with CSP-Adaptive (CRPS rank 3.09, MQL rank 2.99). SeasonalNPTS is the strongest external comparator (CRPS rank 3.23) and is ahead of DeepNPTS on both rank metrics.
6
Table 1: Accuracy and runtime summary across the 6 datasets and 380 forecast records per method. Lower ranks are better; coverage targets 0.95. Method
CRPS rank
CRPS wins
MQL rank
MQL wins
Coverage
Wall min.
3.028 3.094 3.233 3.595 3.621 4.353
2 1 1 1 0 1
2.922 2.988 3.260 3.519 3.513 4.344
2 1 1 1 0 1
0.89 0.89 0.91 0.66 0.70 0.95
0.27 0.23 0.31 153.10 0.32 0.44
CSP-Adaptive CSP-Fixed SeasonalNPTS DeepNPTS AdaptiveWindowMCI NPTS
5.2
Rank distribution
Figure 1 shows the per-window rank distribution for each method on both metrics; Table 2 reports the underlying counts. Each of the 380 forecast windows is ranked 1 (best) to 6 (worst) within itself, and the columns count the number of windows that fell in each rank band. The CSP variants concentrate in ranks 1–3 on roughly two-thirds of windows (CSP-Adaptive: 258/380 = 68% on CRPS), while DeepNPTS lands in the bottom band (R5–6) on 156/380 = 41% of windows. CSP-Adaptive finishes top-3 more often than DeepNPTS on both CRPS and MQL. Figure 1: Per-window rank distribution (380 forecast windows). Greener bars indicate more top-rank finishes; the rightmost dark-red band indicates rank 5–6. CSP variants concentrate in the top three ranks on both metrics. R1
R2
R3
R4
R5--6
CRPS rank distribution
Normalized MQL rank distribution
CSP-Adaptive
39
95
124
70
52
52
101
109
74
44
CSP-Fixed
31
105
111
82
51
50
91
120
72
47
SeasonalNPTS
118
47
53
48
114
DeepNPTS
70
58
33
63
156
AdaptiveWindowMCI
89
30
33
86
142
NPTS
50
0
5.3
38
50
29
115
51
72 97
244
59
100 150 200 250 300 Number of forecast windows (out of 380)
350
0
34
50
52
42
120
61
37
62
148
30
32
93
128
26
247
100 150 200 250 300 Number of forecast windows (out of 380)
350
Calibration analysis at the 95% level
The accepted paradigm for evaluating probabilistic forecasts, set out by Gneiting et al. (2007) and consolidated in the review of Gneiting and Katzfuss (2014), is to maximize sharpness subject to calibration. Calibration is the prerequisite: a probabilistic forecast that is not calibrated cannot be improved by sharpening it, because sharpness without calibration is just narrower wrongness. Under this paradigm a forecaster is first checked for calibration, and only then are its sharpness scores admissible. CSP-Adaptive and CSP-Fixed achieve mean empirical coverage of 0.89 against the nominal 0.95 target. DeepNPTS achieves 0.66 mean coverage, with across-dataset standard deviation 0.33. A method whose nominal 95% prediction interval contains the truth in only about 7
Table 2: Per-window rank-finish counts (out of 380 forecast windows). Each window is ranked 1 (best) to 6 (worst) within itself; columns count the windows landing in each rank band. Rows sum to 380. CRPS rank distribution Normalized MQL rank distribution Method
R1
R2
R3 R4
CSP-Adaptive 39 95 124 CSP-Fixed 31 105 111 SeasonalNPTS 118 47 53 DeepNPTS 70 58 33 AdaptiveWindowMCI 89 30 33 NPTS 50 38 19
R5–6
70 82 48 63 86 29
52 51 114 156 142 244
R1
R2
R3 R4
52 101 109 50 91 120 115 51 52 72 61 37 97 30 32 59 34 14
74 72 42 62 93 26
R5–6 44 47 120 148 128 247
two thirds of cases fails the basic calibration desideratum of Gneiting and Katzfuss (2014): it is not delivering probabilistic forecasts in the technical sense the paradigm requires, even if its quantile-loss number is competitive. In safety- and decision-critical applications the practical consequences are stark. A 0.95 nominal interval that covers in only ∼ 66% of cases would systematically misclassify high-risk individuals in medical screening, trigger Basel-III capital-requirement failures in finance (mandated at 0.99 coverage), translate into reserve-capacity shortfalls and balancing errors in energy operations, and would not clear any reasonable certification bar for trajectory and pedestrian-prediction intervals in autonomous vehicles. The gap between nominal 0.95 and empirical 0.66 is not a benchmark detail; it is a deployment blocker in every domain where probabilistic forecasts are used for decisions. Figure 2 shows the per-window coverage distribution. DeepNPTS fails calibration in two compounding ways: its mean coverage is biased low (0.66 versus the 0.95 nominal target), and its window-to-window spread is large (standard deviation 0.33). The decisive number is the lower decile of the DeepNPTS distribution: 0.03. Concretely, in the worst 10% of forecast windows the DeepNPTS 95% prediction interval misses the truth at every horizon h = 1, . . . , H simultaneously — not at one or two isolated horizons, but across the entire multi-step trajectory at once. The aggregate 0.66 mean coverage is therefore worse than it appears: the missing 0.34 of coverage is not spread thinly as occasional misses, but concentrated in a substantial fraction of windows where the interval covers essentially nothing across all H steps. Figure 3 confirms the pattern at the dataset level: CSP-Adaptive sits at or above the 0.95 target on every dataset, while DeepNPTS undercovers on every dataset.
5.4
Head-to-head comparisons
CSP-Adaptive beats DeepNPTS on 5/6 datasets by CRPS and 3/6 by MQL; CSP-Fixed has the same pattern. Table 3 reports the full head-to-head matrix. Against NPTS the CSP variants win 6/6 datasets on CRPS; against SeasonalNPTS they win 5/6. The CRPS–MQL gap on the DeepNPTS comparison is a known limitation of normalized MQL rather than a substantive draw. MQL is the average pinball loss over the nine fixed quantile levels {0.1, 0.2, . . . , 0.9}, so it only sees the body of the predictive distribution and is blind to behaviour in the tails τ < 0.1 and τ > 0.9. CRPS, by contrast, integrates over the entire predictive CDF and so penalises tail miscalibration directly. DeepNPTS’s failure mode is exactly tail-driven — median empirical coverage 0.82, lower decile 0.03 at the 0.95 nominal level (Section 5.3) — which is what an MQL on {0.1, . . . , 0.9} is built to under-weight and what CRPS is built to capture. The headline metric CRPS is therefore the appropriate score for comparing 8
Empirical coverage per (dataset, series, window) (target 1 = 0.95)
Figure 2: Per-window empirical coverage distribution across all 380 forecast windows. Each violin shows the distribution of empirical coverage at the per-(dataset, series, window) level; white diamonds mark the median, black bars the inter-quartile range. The dashed green line marks the nominal 1 − α = 0.95 target. CSP variants concentrate near 1.0 with thin lower tails; DeepNPTS is bimodal with substantial mass near zero. The diagnostic shows DeepNPTS fails calibration both by being systematically biased low (median 0.82) and by being highly variable across windows (standard deviation 0.33). Per-window coverage distribution (Tier 1) 1.0
Nominal target = 0.95
0.8 0.6
Coin-flip = 0.50
0.4 0.2 0.0
CSP-Adaptive
CSP-Fixed
DeepNPTS
AdaptiveWindowMCI
SeasonalNPTS
NPTS
Figure 3: Per-dataset mean coverage, sorted by gap. CSP-Adaptive (green) covers near or above 0.95 on every dataset; DeepNPTS (red) undercovers everywhere; gap labels above each pair show the per-dataset CSP-DeepNPTS coverage advantage in absolute coverage units.
Mean empirical 95\% coverage
1.0
Per-dataset coverage (Tier 1, sorted by gap)
+0.48
+0.24
+0.21
+0.05
Target +0.04= 0.95
0.8 0.6
+0.34
0.4 0.2 0.0
CSP-Adaptive DeepNPTS
c
traffi
nge excha
rate
ricity
elect
edia
wikip
9
taxi
solar
energ
y
miscalibration profiles, and the MQL agreement is a secondary check. Table 3: Dataset-level head-to-head wins (6 datasets). Each cell reports CRPS wins out of 6; lower CRPS wins. AdaptiveWindowMCI rows are omitted (intermediate; reported in Appendix C). Method CSP-Adaptive CSP-Fixed Oracle (best non-deep)
5.5
vs NPTS
vs SeasonalNPTS
vs DeepNPTS
6/6 6/6 6/6
5/6 5/6 5/6
5/6 5/6 5/6
Paired significance tests and absolute scores
Per-dataset rank counts on 6 datasets do not have enough resolution to support strong significance claims directly. The right level of statistical analysis is per-(dataset, series, window): each method produces 380 paired forecast records, and the per-window paired Wilcoxon signed-rank test on those records is well-powered. The CRPS, MQL, and coverage tests are all decisive. CSP-Adaptive significantly beats DeepNPTS on every metric: CRPS at p ≈ 4 × 10−10 , MQL at p ≈ 7 × 10−10 , and coverage at p ≈ 8 × 10−45 (median per-window coverage advantage +0.18). CSP-Fixed’s results are essentially identical. CSP also beats NPTS decisively on CRPS (p ≈ 10−28 ). The only comparison that does not reach significance at p < 0.05 is CSPAdaptive vs SeasonalNPTS on CRPS (p = 0.17): SeasonalNPTS is a strong baseline on the hourly datasets, and CSP only modestly improves on it on CRPS, although CSP does significantly beat SeasonalNPTS on coverage (p = 4 × 10−6 ). The full battery of per-window paired Wilcoxon comparisons is reported in Appendix D. Two claims hold cleanly across the audited suite. First, both CSP variants are strictly ahead of all three NPTS-family methods (NPTS, SeasonalNPTS, DeepNPTS) on both mean CRPS rank and mean MQL rank — true at every cell of the rank comparison in Table 1. Second, both CSP variants significantly outperform DeepNPTS on every metric this paper rests on, with per-window paired Wilcoxon p < 10−9 on CRPS, MQL, and coverage simultaneously. CSP-Adaptive’s mean within-dataset normalized CRPS is below DeepNPTS’s, and CSP-Adaptive achieves mean 0.89 coverage versus DeepNPTS’s 0.66. Full per-method absolute-score summary statistics (means and standard deviations of within-dataset CRPS ratios, normalized MQL, and coverage) are reported in Appendix C.
5.6
Runtime
Each method produces the same 380 forecast records on this audit. CSP-Adaptive completes the full workload in 0.27 wall-clock minutes; DeepNPTS takes 153.1 minutes for the same workload, a 567× slowdown. The five non-DeepNPTS methods together account for 1.6 minutes; DeepNPTS alone is 99% of total method-evaluation time. Figure 4 places these numbers in an accuracy-runtime view: every training-free method sits in the upper-left zone (low CRPS rank, sub-minute runtime), and DeepNPTS is the only point in the upper-right — worse on rank, and two and a half orders of magnitude slower. Full per-method wall-time numbers (sec/row, slowdown factors) are reported in Appendix E.
10
Figure 4: Accuracy vs. runtime. Lower mean CRPS rank is better; the x-axis is log-scaled wall time. CSP-Adaptive and CSP-Fixed sit in the upper-left training-free zone; DeepNPTS is dominated on rank and isolated by two orders of magnitude on cost while also failing the coverage target. Accuracy vs.\ wall time: CSP variants in the training-free upper-left; DeepNPTS slower and worse on rank better (lower rank)
Mean CRPS rank across 6 datasets
2.75
567 × slower wall time
3.00
CSP-Fixed CSP-Adaptive
3.25
SeasonalNPTS
3.50
DeepNPTS
AdaptiveWindowMCI
3.75 mean coverage 0.66 (target 0.95)
4.00 4.25
NPTS
4.50 training-free zone 100
6
101 Audited method wall time, minutes (log scale)
102
What Works, What Does Not, and Where
The CSP variants succeed because their two components correct different failure modes of empirical sampling: the signed-residual component recenters the predictive distribution around the latest seasonal level, while the seasonal pool keeps samples on realistic support. AdaptiveWindowMCI is a fast training-free fallback that uses no seasonal structure — only a datadependent recent window — and serves as a sanity baseline against the seasonal-pool design choice. NPTS shows that pure recency-weighted empirical sampling is too coarse on hourly data (CRPS rank 4.35); SeasonalNPTS, by contrast, is a strong baseline (CRPS rank 3.23, coverage 0.91) and is the closest non-CSP comparator on both metrics. DeepNPTS, despite being the most parameter-rich method in the comparator set, ends up worst on calibration: its mean coverage is 0.66 against the nominal 0.95 target, and its training cost is 567× larger than CSP for the same forecast workload.
7
Limitations and Scope
Scope of the comparison set. The benchmark is scoped to non-parametric samplers: the NPTS family (NPTS, SeasonalNPTS, DeepNPTS) plus several training-free conformal and empirical samplers including CSP. We deliberately did not include classical distributional baselines (ETS, ARIMA, or TBATS with bootstrapped residuals), full-distribution conformal time-series methods built on parametric forecasters (e.g. EnbPI applied as a sampler), or other neural probabilistic forecasters such as DeepAR, Temporal Fusion Transformers, or N-HiTS. The goal of this paper is the head-to-head between training-free conformal sampling and DeepNPTS as the strongest published non-parametric neural sampler, and the choice of comparator set reflects that goal. The pattern of intervals systematically too narrow to attain nominal coverage that this paper documents for DeepNPTS is consistent with the broader prediction-interval failure documented across the M4 competition submissions (Grushka-Cockayne and Jose, 2020). 11
Benchmark size and selection. Each of the six datasets contributes 10 series and 5–7 rolling-origin windows per series, for 50–70 paired forecast records per dataset and 380 records per method overall. The series-count cap is set by the requirement that the DeepNPTS comparator be retrained from scratch on every series and remain auditable on a single workstation; full-series GluonTS configurations of the same datasets contain hundreds of series each and would not have been DeepNPTS-auditable in the available compute budget. The wall-clock comparison is a local CPU measurement; GPU hardware and different DeepNPTS training schedules will narrow the runtime gap, but two orders of magnitude is a wide enough margin that the qualitative conclusion is robust to those choices. The coverage gap is the more important comparison and does not depend on hardware. Statistical and methodological caveats. The split-conformal residual pool used by CSP assumes exchangeability of calibration and test residuals, which is only approximately satisfied for non-stationary series; Xu and Xie (2021) and Gibbs and Candès (2021) provide drop-in extensions that we expect would further improve coverage on the most non-stationary datasets in the suite. The benchmark is restricted to the original DeepNPTS evaluation suite (electricity, exchange rate, solar energy, taxi, traffic, wikipedia), which gives a fair head-to-head against the comparator on its own design ground. Method-design caveats. The CSP sampler treats forecast horizons independently within each predictive sample, so columns of the predictive sample matrix do not represent coherent forecast trajectories; downstream uses that require trajectory-level dependence (e.g. joint multihorizon decision rules) would need a coupling step such as a block-bootstrap variant. The signed-residual pool is uniform over the calibration window rather than studentized or recencyweighted; on heteroskedastic or strongly drifting series, scale normalisation by a rolling seasonal MAD or variance is a natural extension we did not test here. We also did not run sensitivity analyses for the sample budget B = 100, the recency rate λ, or the calibration fraction ρ; these are fixed across all datasets in the audited run.
8
Conclusion
On the six datasets where DeepNPTS was originally benchmarked, a simple training-free conformal sampler significantly outperforms it on CRPS (p ≈ 4 × 10−10 per-window paired Wilcoxon), on normalized mean quantile loss (p ≈ 7 × 10−10 ), and on empirical 95% coverage (p ≈ 8 × 10−45 , mean 0.89 vs 0.66), while running 567× faster on CPU. The coverage gap is the most decision-critical: under the calibration-then-sharpness reporting protocol of Gneiting and Katzfuss (2014), a method whose nominal 95% interval misses the truth roughly a third of the time fails the calibration prerequisite and is inadmissible for any setting that takes the nominal level seriously. We frame this as evidence that training-free seasonal residual samplers should be included as mandatory baselines when evaluating learned non-parametric forecasters, and that future probabilistic-forecasting benchmarks should report empirical coverage at the nominal level alongside any sharpness score. The result is specific to the audited DeepNPTS configuration on this benchmark and is not a general claim about neural probabilistic forecasting.
12
Acknowledgments The authors thank the maintainers of the GluonTS toolkit for the public datasets and the reference DeepNPTS implementation used in this benchmark.
A
Implementation Details
This appendix documents the parameter choices made for every method in the audited suite. CSP-Fixed and CSP-Adaptive. Algorithm 1 states the construction. The CSP-Adaptive weight rule (wh ∈ {0, 0.3, 0.5}) was chosen ahead of seeing test scores and is fixed across all datasets; we did not tune it per dataset. Hyperparameters: ρ = 0.5 (calibration fraction), λ = 0.01 (exponential recency rate, in normalized time-index units), B = 100 predictive samples per forecast record (matched across all training-free methods). Rolling-origin protocol. For each dataset the audit runs 5–7 rolling-origin windows with stride equal to the forecast horizon H (non-overlapping forecast targets). Horizons: H = 24 for hourly datasets, H = 30 for daily datasets. The same window indices are used across all methods so per-dataset comparisons are paired. DeepNPTS. Run via gluonts.torch.model.deep npts.DeepNPTSEstimator with 50 epochs, batch size 32, two hidden layers of 40 units, learning rate 10−4 , 100 batches/epoch, context multiplier 4 (capped at 512 steps), 200 inference samples. No early stopping, no per-dataset tuning. CPU only. Other methods. NPTS and SeasonalNPTS: GluonTS reference implementations, default parameters. AdaptiveWindowMCI: ACF-based seasonality detector (threshold 0.3) selecting a data-dependent recent window. All non-DeepNPTS methods use experiment-driver seed 0 and internal RNG seed 42. Per-dataset preprocessing. No normalization, gap-filling, or frequency conversion. Series shorter than H + 10 observations are filtered out.
B
Dataset Suite Table 4: Audited dataset suite. Records are forecast records per method.
Dataset electricity exchange rate solar energy taxi traffic wikipedia
Records
Horizon
70 50 70 70 70 50
24 30 24 24 24 30
13
C
Per-Method Absolute Scores
Table 5 reports the per-method mean and standard deviation of within-(dataset, series, window) normalized CRPS, normalized MQL, and empirical coverage at the 0.95 nominal level. Table 5: Absolute-score summary across the 6 datasets. CRPS is reported as a within-dataset ratio to the cross-method median to control for the wide range of dataset scales. MQL is the normalized mean quantile loss. Coverage targets 0.95. CRPS (rel. to row median)
Normalized MQL
Method
Mean
Std
Mean
Std
Mean
CSP-Adaptive CSP-Fixed AdaptiveWindowMCI DeepNPTS SeasonalNPTS NPTS
0.915 0.915 1.044 1.257 0.963 1.432
0.062 0.062 0.090 0.393 0.150 0.507
0.405 0.405 0.528 0.388 0.442 0.438
0.472 0.471 0.600 0.358 0.551 0.407
0.888 0.891 0.695 0.662 0.908 0.950
D
Coverage
Per-Window Paired Significance Tests
Table 6 reports the full battery of per-window paired Wilcoxon signed-rank tests on within(dataset, series, window) normalized CRPS and MQL (each method’s score divided by the cross-method median for that window) and on coverage. The headline numbers are reported in Section 5.5; this appendix records the comparison set in full. Table 6: Per-window paired Wilcoxon signed-rank tests (n ≈ 380 paired records per comparison). Bold p-values reach p < 0.05. CRPS and MQL are normalized within each (dataset, series, window) by the median across methods. Comparison
per-window paired Wilcoxon p
CRPS, normalized within (dataset, series, window) by median across methods, A < B CSP-Adaptive < DeepNPTS 4.4e − 10 CSP-Fixed < DeepNPTS 2.5e − 10 AdaptiveWindowMCI < DeepNPTS 0.1539 CSP-Adaptive < SeasonalNPTS 0.1702 CSP-Adaptive < NPTS 1.3e − 28 Normalized mean quantile loss, A < B CSP-Adaptive < DeepNPTS CSP-Fixed < DeepNPTS AdaptiveWindowMCI < DeepNPTS
6.6e − 10 6.1e − 10 0.1789
Coverage (closer to nominal 0.95), A > B CSP-Adaptive > DeepNPTS CSP-Fixed > DeepNPTS AdaptiveWindowMCI > DeepNPTS CSP-Adaptive > SeasonalNPTS CSP-Adaptive > NPTS
8.0e − 45 1.1e − 44 0.8570 3.9e − 06 0.7237
14
E
Per-Method Wall Time
Table 7: Audited method wall time. Dataset download/loading and final summary writing are excluded. Method Wall min. Rows Sec./row Slowdown vs fastest Datasets CSP-Fixed CSP-Adaptive SeasonalNPTS AdaptiveWindowMCI NPTS DeepNPTS
0.23 0.27 0.31 0.32 0.44 153.10
380 380 380 380 380 380
0.019 0.022 0.025 0.026 0.036 12.500
15
1.0× 1.2× 1.3× 1.4× 1.9× 665.7×
6 6 6 6 6 6
References Alexander Alexandrov, Konstantinos Benidis, Michael Bohlke-Schneider, Valentin Flunkert, Jan Gasthaus, Tim Januschowski, Danielle C. Maddix, Syama Sundar Rangapuram, David Salinas, Jasper Schulz, Lorenzo Stella, Ali Caner Turkmen, and Yuyang Wang. GluonTS: Probabilistic and neural time series modeling in Python. Journal of Machine Learning Research, 21(116):1–6, 2020. URL http://jmlr.org/papers/v21/19-820.html. Rina Foygel Barber and Ashwin Pananjady. Predictive inference for time series: Why is split conformal effective despite temporal dependence?, 2025. URL https://arxiv.org/abs/2510.02471. Cristian Challu, Kin G. Olivares, Boris N. Oreshkin, Federico Garza, Max Mergenthaler-Canseco, and Artur Dubrawski. N-HiTS: Neural hierarchical interpolation for time series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 6989–6997, 2023. doi: 10.1609/ aaai.v37i6.25854. URL https://doi.org/10.1609/aaai.v37i6.25854. Isaac Gibbs and Emmanuel J. Candès. Adaptive conformal inference under distribution shift, 2021. URL https://arxiv.org/abs/2106.00170. Tilmann Gneiting and Matthias Katzfuss. Probabilistic forecasting. Annual Review of Statistics and Its Application, 1:125–151, 2014. doi: 10.1146/annurev-statistics-062713-085831. URL https://doi. org/10.1146/annurev-statistics-062713-085831. Tilmann Gneiting and Adrian E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359–378, 2007. doi: 10.1198/ 016214506000001437. URL https://doi.org/10.1198/016214506000001437. Tilmann Gneiting, Fadoua Balabdaoui, and Adrian E. Raftery. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 69(2):243–268, 2007. doi: 10.1111/j.1467-9868.2007.00587.x. URL https://doi.org/10.1111/j.1467-9868.2007. 00587.x. Rakshitha Godahewa, Christoph Bergmeir, Geoffrey I. Webb, Rob J. Hyndman, and Pablo MonteroManso. Monash time series forecasting archive, 2021. URL https://arxiv.org/abs/2105.06643. Yael Grushka-Cockayne and Victor Richmond R. Jose. Combining prediction intervals in the M4 competition. International Journal of Forecasting, 36(1):178–185, 2020. doi: 10.1016/j.ijforecast.2019.04.015. URL https://doi.org/10.1016/j.ijforecast.2019.04.015. Hans Hersbach. Decomposition of the continuous ranked probability score for ensemble prediction systems. Weather and Forecasting, 15(5):559–570, 2000. doi: 10.1175/1520-0434(2000)015⟨0559: DOTCRP⟩2.0.CO;2. URL https://doi.org/10.1175/1520-0434(2000)015<0559:DOTCRP>2.0.CO; 2. Jing Lei, Max G’Sell, Alessandro Rinaldo, Ryan J. Tibshirani, and Larry Wasserman. Distribution-free predictive inference for regression. Journal of the American Statistical Association, 113(523):1094– 1111, 2018. doi: 10.1080/01621459.2017.1307116. URL https://doi.org/10.1080/01621459.2017. 1307116. Bryan Lim, Sercan Ö. Arik, Nicolas Loeff, and Tomas Pfister. Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting, 37(4):1748–1764, 2021. doi: 10.1016/j.ijforecast.2021.03.012. URL https://doi.org/10.1016/j.ijforecast.2021. 03.012. Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The M4 competition: Results, findings, conclusion and way forward. International Journal of Forecasting, 34(4):802–808, 2018. doi: 10.1016/j.ijforecast.2018.06.001. URL https://doi.org/10.1016/j.ijforecast.2018.06.001. Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The M4 competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting, 36(1):54–74, 2020. doi: 10.1016/j.ijforecast.2019.04.014. URL https://doi.org/10.1016/j.ijforecast.2019.04.014. Spyros Makridakis, Evangelos Spiliotis, Vassilios Assimakopoulos, Zhi Chen, Anil Gaba, Ilia Tsetlin, and Robert L. Winkler. The M5 uncertainty competition: Results, findings and conclusions. International Journal of Forecasting, 38(4):1365–1385, 2022. doi: 10.1016/j.ijforecast.2021.10.009. URL https: //doi.org/10.1016/j.ijforecast.2021.10.009. Roberto I. Oliveira, Paulo Orenstein, Thiago Ramos, and João Vitor Romano. Split conformal prediction and non-exchangeable data. Journal of Machine Learning Research, 25(225):1–47, 2024. URL http: //jmlr.org/papers/v25/23-1553.html.
16
Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-BEATS: Neural basis expansion analysis for interpretable time series forecasting. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=r1ecqn4YwB. Harris Papadopoulos, Kostas Proedrou, Vladimir Vovk, and Alex Gammerman. Inductive confidence machines for regression. In Machine Learning: ECML 2002, volume 2430 of Lecture Notes in Computer Science, pages 345–356. Springer, 2002. doi: 10.1007/3-540-36755-1 29. URL https://doi.org/10. 1007/3-540-36755-1_29. Xiangfei Qiu, Jilin Hu, Lekui Zhou, Xingjian Wu, Junyang Du, Buang Zhang, Chenjuan Guo, Aoying Zhou, Christian S. Jensen, Zhenli Sheng, and Bin Yang. TFB: Towards comprehensive and fair benchmarking of time series forecasting methods. Proceedings of the VLDB Endowment, 17(9):2363– 2377, 2024. doi: 10.14778/3681954.3682003. URL https://doi.org/10.14778/3681954.3682003. Syama Sundar Rangapuram, Jan Gasthaus, Lorenzo Stella, Valentin Flunkert, David Salinas, Yuyang Wang, and Tim Januschowski. Deep non-parametric time series forecaster, 2023. URL https:// arxiv.org/abs/2312.14657. Yaniv Romano, Evan Patterson, and Emmanuel J. Candès. Conformalized quantile regression. In Advances in Neural Information Processing Systems, volume 32, 2019. URL https://papers.nips. cc/paper/2019/hash/5103c3584b063c431bd1268e9b5e76fb-Abstract.html. Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 9:371–421, 2008. URL http://jmlr.org/papers/v9/shafer08a.html. Vladimir Vovk, Alex Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World. Springer, 2005. doi: 10.1007/b106715. URL https://doi.org/10.1007/b106715. Vladimir Vovk, Jieli Shen, Valery Manokhin, and Min-ge Xie. Nonparametric predictive distributions based on conformal prediction. In Proceedings of the Sixth Workshop on Conformal and Probabilistic Prediction and Applications, volume 60 of Proceedings of Machine Learning Research, pages 82–102. PMLR, 2017. URL https://proceedings.mlr.press/v60/vovk17a.html. Vladimir Vovk, Ilia Nouretdinov, Valery Manokhin, and Alex Gammerman. Cross-conformal predictive distributions. In Proceedings of the Seventh Workshop on Conformal and Probabilistic Prediction and Applications, volume 91 of Proceedings of Machine Learning Research, pages 37–51. PMLR, 2018a. URL https://proceedings.mlr.press/v91/vovk18a.html. Vladimir Vovk, Ilia Nouretdinov, Valery Manokhin, and Alex Gammerman. Conformal predictive distributions with kernels. In Braverman Readings in Machine Learning. Key Ideas from Inception to Current State, volume 11100 of Lecture Notes in Computer Science, pages 103–121. Springer, 2018b. doi: 10.1007/978-3-319-99492-5 4. URL https://doi.org/10.1007/978-3-319-99492-5_4. Vladimir Vovk, Ilia Nouretdinov, Valery Manokhin, and Alex Gammerman. Computationally efficient versions of conformal predictive distributions. Neurocomputing, 397:292–308, 2020. doi: 10.1016/j. neucom.2019.10.110. URL https://doi.org/10.1016/j.neucom.2019.10.110. Chen Xu and Yao Xie. Conformal prediction interval for dynamic time-series. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 11559–11569. PMLR, 2021. URL https://proceedings.mlr.press/v139/xu21h. html. Chen Xu and Yao Xie. Conformal prediction for time series. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10):11575–11587, 2023. doi: 10.1109/TPAMI.2023.3272339. URL https: //doi.org/10.1109/TPAMI.2023.3272339. Jiawen Zhang, Xumeng Wen, Zhenwei Zhang, Shun Zheng, Jia Li, and Jiang Bian. ProbTS: Benchmarking point and distributional forecasting across diverse prediction horizons. In Advances in Neural Information Processing Systems, volume 37 of Datasets and Benchmarks Track, 2024. URL https://openreview.net/forum?id=ZvGkOh7e2N.
17