Leave a Window Out: Modifying the Jackknife for Predictive Inference in Time Series Hanyang Jiang⋆ , Rina Foygel Barber‡ , Ashwin Pananjady⋆,† , Yao Xie⋆
arXiv:2605.30292v1 [stat.ML] 28 May 2026
Schools of Industrial and Systems Engineering⋆ and Electrical and Computer Engineering† , Georgia Tech Department of Statistics, University of Chicago‡ May 29, 2026
Abstract Conformal prediction methods enjoy strong theoretical and empirical predictive inference performance, provided the data is exchangeable, and predictors are trained in a memoryless fashion. However, these assumptions and constraints are impractical in many real-data settings, such as time series (where temporal dependence violates exchangeability, and where memoryless predictors will inevitably have poor predictive accuracy). Recent work shows that the split conformal prediction method is robust to these issues of memory-based predictors and deviations from exchangeability that are common features of time-series data. However, since using sample splitting can lead to lower accuracy, this motivates asking whether other predictive inference methods (that do not rely on data splitting) could also be reliably used in the time series setting. In this work, we show that the vanilla leave-one-out jackknife can suffer an arbitrary loss of coverage even in canonical time series models with mild temporal dependence. As a remedy, we propose a careful modification tailored to such settings, which we term the leave-a-window-out (LWO) method, and show that it can achieve valid coverage provided that the model-fitting procedure satisfies mild stability properties. Our proofs are based on quantifying the degree to which the data departs from cyclic exchangeability, and we introduce new coefficients to measure the extent of this departure. Experiments on time series data demonstrate that our LWO method often enjoys valid coverage when the vanilla jackknife fails to cover, while producing much narrower intervals than split conformal prediction.
1
Introduction
Quantifying the uncertainty of predictions is a fundamental challenge in data science, including in time-series analysis and sequential decision-making. As machine learning models are increasingly deployed for forecasting in high-stakes domains such as financial markets (Sezer et al., 2020), climate change (Rolnick et al., 2022), and energy applications (Hong et al., 2016), it is essential to provide valid prediction intervals around their outputs. In general, uncertainty quantification can be performed by making assumptions about how the data is generated. On the one hand, classical time-series models are explicitly designed to accommodate dependent data, but provide valid uncertainty quantification only under strong parametric assumptions on the model. However, these parametric assumptions are often violated when using modern black-box predictors like neural networks. On the other hand, “distribution-free” predictive inference provides a suite of wrapper techniques that provide uncertainty quantification for predictions arising from any underlying model (Vovk et al., 2005; Shafer and Vovk, 2008), and the validity of these methods relies only on exchangeability of data. However, dependent data (e.g., in time series and other sequential scenarios) is not exchangeable. 1
This drawback notwithstanding, distribution-free predictive inference methods still enjoy widespread use across applications in which exchangeability may be violated. In fact, a particular such method called split conformal prediction has recently been shown to exhibit automatic robustness to violations of exchangeability that are often observed in time series (Oliveira et al., 2024; Barber and Pananjady, 2026). While split conformal is conceptually simple and easy to implement, it involves sample-splitting and can sacrifice statistical efficiency. In contrast, predictive inference methods based on the leave-one-out principle—colloquially referred to as variations of the jackknife (Tukey, 1958; Stone, 1974; Efron and Gong, 1983)—are appealing since they avoid sample-splitting and make full use of the dataset. While they can be computationally more intensive than conformal methods based on data splitting, leave-one-out methods offer greater statistical efficiency and are preferable in low-data regimes. However, the behavior of leave-one-out methods in settings with dependent data is not well understood. Are these methods also robust to similar violations of exchangeability—or are modifications of these methods needed in order to obtain reasonable coverage properties in a time series setting? This forms the motivating question of our paper.
1.1
Using the jackknife for predictive inference
To make our discussion concrete, let us formally introduce the jackknife method for memoryless predictors. Suppose we have a time series of covariate-response data (X1 , Y1 ), . . . , (Xn+1 , Yn+1 ), with (Xi , Yi ) ∈ X × Y representing the i-th data point. The data point at index n + 1 is considered to be the “test point”, with Xn+1 observed but Yn+1 unobserved, while the points indexed 1, . . . , n bn (Xn+1 ) ⊆ Y based on the serve as the training data. Our goal is to construct a prediction set C training data (X1 , Y1 ), . . . , (Xn , Yn ) and Xn+1 such that this set contains the unknown response Yn+1 with a pre-specified probability: bn (Xn+1 )) ≥ 1 − α. P(Yn+1 ∈ C
(1)
Here, α ∈ (0, 1) denotes the target miscoverage level. The jackknife, or leave-one-out, method produces a prediction interval by repeatedly fitting a model on n − 1 out of the n training data points and constructing conformity scores on the point that is left out. Concretely, suppose our model-fitting algorithm is memoryless (in that it produces a map X → Y) and that conformity is measured by the absolute residual. For each1 i ∈ [n], the jackknife computes the predictor fb−i by training our model-fitting algorithm on all data points except the i-th one. Then, the i-th leave-one-out conformity score is computed on the training point that was left out, with sloo = |Yi − fb−i (Xi )|. Letting fb denote the predictor obtained by i training on all n points, the jackknife prediction interval is then constructed by centering at the point prediction fb(Xn+1 ), and inflating it by the quantile of the leave-one-out scores. Specifically,2 h i bn (Xn+1 ) = fb(Xn+1 ) − Quantile1−α (sloo , . . . , sloo ), fb(Xn+1 ) + Quantile1−α (sloo , . . . , sloo ) . C 1 n 1 n In the setting of exchangeable data, existing theoretical results (Barber et al., 2021; Steinberger and Leeb, 2023) show that, if the model-fitting algorithm satisfies a mild form of out-of-sample stability with respect to leaving one data point out of the training set, then the jackknife achieves the desired coverage guarantee (1) for any sample size n up to a small additive correction to the miscoverage level α. But how robust are these conclusions in the time series setting, where the data can be far from exchangeable? 1
Throughout, we use the shorthand notation [k] := {1, . . . , k}. For a vector v ∈ Rm , the scalar Quantile1−α (v) is the (1 − α) empirical quantile of the entries that comprise v, i.e. the order statistic v(k) for k = ⌈(1 − α)m⌉. 2
2
5.0
0.8
4.5
Scaled Radius
Empirical Coverage
1.0 0.6 0.4
Nominal coverage (0.9) Split CP Jackknife LWO jackknife
0.2 0.0
1
20
40
60
Dimension (d)
80
4.0 3.5 3.0
Split CP Jackknife LWO jackknife
2.5
100
1
20
40
60
Dimension (d)
80
100
Figure 1. (Left) Empirical coverage on the multidimensional MA(1) process using 2-nearest neighbors as the base predictor. Coverage is averaged over 1000 independent trials for split CP, the vanilla jackknife, and the LWO method. The nominal coverage level is 90% throughout. (Right) Average radius √ of the prediction regions produced by split CP, the vanilla jackknife, and LWO, scaled by 1/ d to account for the response dimension d.
1.2
A motivating numerical experiment
To simulate a canonical time series, we generate our data (Xi , Yi )i≥1 according to a moving average (MA) process in various dimensions d and assess the coverage of various predictive inference methods. To be concrete, fix some value of dimension d ∈ N and suppose the covariate and response space are given by X = Y = Rd . Let {ωi }i≥0 be i.i.d. standard Gaussians drawn from the distribution N (0, Id ). Construct d-dimensional covariates {Xi }n+1 i=1 according to the MA(1) process Xi = ωi−1 + ωi ,
with Yi = Xi+1
for all i = 1, . . . , n + 1.
Our dataset is then given by (Xi , Yi )n+1 i=1 . In our experiment, we use n = 200. We use 2-nearest neighbors as the base predictor for training, and (since the response Y is multivariate) we extend the definition of the jackknife to construct a ball, rather than an interval, around the prediction. Concretely, for each i ∈ [n], let fb−i denote the predictor trained on all training points except i, and define the leave-one-out score b sloo i := Yi − f−i (Xi ) 2 . Let fb be the predictor trained on all n training points, and define the leave-one-out quantile loo qloo := Quantile1−α sloo 1 , . . . , sn . The vanilla jackknife prediction set is the ℓ2 -ball centered at fb(Xn+1 ), viz. n o bnjackknife (Xn+1 ) = y ∈ Rd : ∥y − fb(Xn+1 )∥2 ≤ qloo . C When d = 1, this reduces to the interval described in Section 1.1. For comparison, we also include split conformal prediction (CP), which fits a 2-nearest-neighbor predictor fbsp on the first n/2 points (i.e. using 50% of the dataset for training) and computes the empirical (1 − α)-quantile qsp of scores on the remaining n/2 calibration points. The prediction set is then given by an ℓ2 -ball of radius qsp centered at the point prediction fbsp (Xn+1 ). Figure 1(left) reports the empirical coverage of split CP and the vanilla jackknife. Even in this simple MA(1) setting, the vanilla jackknife can severely undercover, with the degradation becoming more pronounced in higher dimensions. By contrast, split CP maintains coverage close 3
to the nominal level across dimensions, as predicted by prior theory (Oliveira et al., 2024; Barber and Pananjady, 2026). In the same figure, we also plot the coverage of the “leave-a-windowout” (LWO) method, which is the main methodological contribution of this paper. Evidently, the coverage behavior of this method is similar to that of split CP. In Figure 1(right), we plot the radius of the prediction set for the three methods. Across dimensions, LWO produces consistently smaller prediction regions than split CP. At the same time, the size of the prediction region of LWO is similar to that of the vanilla jackknife when the latter exhibits reasonable coverage (i.e., for small dimensions).
1.3
Related work
For general background on conformal prediction, we refer the reader to the monographs (Vovk et al., 2005; Shafer and Vovk, 2008; Angelopoulos and Bates, 2023; Angelopoulos et al., 2024) for introductions to both practical and theoretical aspects of the subject. In this section, we discuss two lines of work that are most closely related to our contributions. Predictive inference beyond data splitting. Split conformal prediction (Papadopoulos et al., 2002; Lei et al., 2018) is typically the most computationally tractable among CP methods. However, it suffers from a widely-recognized “split tax”: partitioning the data into a proper training set and a calibration set reduces the effective sample size available for model fitting, which can lead to unnecessarily wide prediction intervals and higher variability, especially in data-scarce regimes (see, e.g., Figure 1(right)). A large body of work has therefore sought to reduce or eliminate this statistical inefficiency by using the data more effectively for both training and calibration. Popular approaches in this direction include cross-conformal prediction (Vovk, 2015) and related aggregation methods such as the jackknife, jackknife+, jackknife-minmax, and CV+ methods of Barber et al. (2021). Broadly speaking, these methods are based on leave-one-out or leave-fold-out residuals and are attractive because they often produce substantially tighter intervals than split CP while retaining valid coverage under exchangeability. More recent variants include multi-split CP (Solari and Djordjilović, 2022), out-of-bag CP (Linusson et al., 2020), nested conformal and quantile out-of-bag methods (Gupta et al., 2022), and bootstrap-based procedures such as the jackknife+-after-bootstrap (Kim et al., 2020). The jackknife validity result of Barber et al. (2021) relies on an out-of-sample stability condition, reflecting the fact that the leave-one-out residuals should not change too much when one training observation is removed. Stability-based arguments have also been used to obtain conditional or training-conditional forms of predictive validity (Steinberger and Leeb, 2023; Liang and Barber, 2025) and to make conformal procedures computationally efficient (Ndiaye, 2022). Recent work shows that bagging can provide assumption-free stability even when the underlying base learner is unstable (Soloff et al., 2024). More broadly, algorithmic stability has long played a central role in learning theory (Bousquet and Elisseeff, 2002), including for dependent observations (Mohri and Rostamizadeh, 2010). Conformal prediction under dependence and nonexchangeability. Classical validity theory for conformal prediction relies on exchangeability, but many modern applications involve temporal dependence, distribution shift, adaptive data collection, or other violations of exchangeability. Several works have developed general frameworks for conformal prediction beyond exchangeability (e.g. Tibshirani et al., 2019; Podkopaev and Ramdas, 2021; Barber et al., 2023; Prinster et al., 2024). These works provide useful tools and perspectives for understanding validity when calibration and test points are not exchangeable, but they do not specifically address the failure of leave-one-out methods under local temporal dependence. 4
In the specific setting of dependent data exhibiting Markovian and/or mixing structure, Chernozhukov et al. (2018) propose conformal methods based on approximate block exchangeability and Nettasinghe et al. (2023) obtain exact validity for hidden Markov models on finite sample spaces. Other papers (Zheng and Proutiere, 2024; Oliveira et al., 2024) analyze the robustness of split conformal prediction to violations of exchangeability observed in dependent data streams. Recent work by Barber and Pananjady (2026) sharpens these guarantees and introduces the notion of switch coefficients to measure departures from exchangeability. A distinct line of work develops conformal methods specifically for online prediction (Gibbs and Candes, 2021), distribution drift (Gibbs and Candès, 2024), and time-series forecasting (Zaffran et al., 2022; Angelopoulos et al., 2023). Other time-series conformal methods, such as EnbPI (Xu and Xie, 2021), SPCI (Xu and Xie, 2023) and KOWCPI (Lee et al., 2025) exploit the temporal structure of the residuals or conformity scores more directly. These insights have recently been extended to multistep and multidimensional responses for time-series (Sun and Yu, 2024; Xu et al., 2024), and spatial dependence structures (Jiang and Xie, 2026). While many of these methods are designed for the online (or streaming) setting, our focus (like that of e.g., Chernozhukov et al. (2018); Oliveira et al. (2024); Barber and Pananjady (2026)) is on the setting in which the dataset (Xt , Yt )n+1 t=1 is a fixed batch but may exhibit temporal dependence.
2
General setting and our method
In Section 1.1, we introduced the problem of predictive inference using memoryless predictors and the absolute residual score function, but given that memory-based prediction is the de facto method in time series analysis and conformity can be measured in various ways, we now present the more general setting that accommodates arbitrary score functions and predictors with memory.
2.1
Memory-based prediction
A predictor with memory L produces at each time t ≥ 1 a map from the L-sized window of past observations to Y, with the goal of approximating the response Yt . For convenience of notation, we therefore assume that the given data is indexed starting at −(L − 1), so that we have access to3 the time series of n+1+L data points (Xi , Yi )n+1 i=−(L−1) . As before, the (n+1)-th point should be viewed as the test point. Now for each t ∈ [n + 1], define the lifted covariate Xt = ((Xk , Yk )t−1 k=t−L , Xt ). Any memory-L predictor f : (X × Y)L × X → Y produces a mapping Xt 7→ f (Xt ); let F denote the space of all such predictors. The closeness of this prediction to the desired response Yt is measured in terms of a conformity score function score : Y × Y → R. For training, we have access to some k S prediction algorithm Alg : k∈N (X × Y)L × X → F which takes as input k samples of training data (for any natural number k) and outputs a particular memory L predictor which is the best fit according to some criterion. Typically, one should think of Alg as being symmetric in its inputs.4 Example: Exogenous Time Series Regression. In this setting, the real response Yt depends on a separate vector of concurrent, exogenous covariates Xt . For example, we might wish to predict electricity consumption Yt at time t based on observed weather conditions such as temperature, humidity, and other covariates, which we collect in Xt . It is reasonable to assume temporal dependence in this process—i.e., that Yt depends not only on contemporaneous exogenous information Note that this is simply a matter of indexing. If our data points (Xi , Yi )n+1 i=1 are indexed starting at 1, then we ′ +1 ′ may re-index as (Xi , Yi )n , where n = n − L. i=−(L−1) 4 Symmetry is assessed after lifting the covariates, so memory-based predictors can still be symmetric in this sense. 3
5
Pre-window training set X1 Y1
X2 Y2
···
Xk−1 Yk−1
Post-window training set Xk Yk
Proxy test point
Xk+1 Yk+1
···
Xk+τ Xk+τ +1 Yk+τ Yk+τ +1
···
Xn Yn
Window left out
Figure 2. Illustration of how the LWO score sk is computed. We leave out a window of length τ starting at (and including) (Xk+1 , Yk+1 ), which is denoted by the white blocks. The predictor fbk is trained on the green points (i.e. all remaining data except (Xk , Yk )), and the score sk is obtained by evaluating the trained predictor fbk on (Xk , Yk ).
Xt but also on covariates and responses in the L past time instants t − L, . . . , t − 1. Thus, any t−1 predictor f for Yt will map ({Xk , Yk }t−1 ♣ k=t−L , Xt ) 7→ f ({Xk , Yk }k=t−L , Xt ). Example: Autoregressive Forecasting. In many forecasting tasks, external covariates are unavailable, and prediction relies solely on past responses. In other words, the sequence is given by (Y−(L−1) , . . . , Yt ) and the goal is to predict Yt from a window of past responses Yt−L , . . . , Yt−1 . In order to write this in the notation of covariate-response pairs, we can follow the convention that X = {x0 } for some fixed x0 , and any predictor for Yt may once again be viewed as a map from (X × Y)L × X → Y (where the predictor simply ignores x0 ). ♣ Having given two canonical examples of memory-based prediction, we are now ready to define our predictive inference method.
2.2
The leave-a-window-out (LWO) method
Our leave-a-window-out (or LWO) method is given by the following steps: 1. For each k = 1, . . . , n: • Train the predictor fbk on the leave-window-out subsequence of all data points excluding the window of indices k, . . . , min{k + τ, n}. Specifically, compute h i fbk =Alg (X1 , Y1 ), . . . , (Xk−1 , Yk−1 ), (Xk+τ +1 , Yk+τ +1 ), . . . , (Xn , Yn ) , | {z } | {z } Pre-window sequence
(2a)
Post-window sequence
where the post-window sequence is empty for k ≥ n − τ . • Compute the score sk = score(Yk , fbk (Xk )).
(2b)
. 2. Let fb = Alg[(X1 , Y1 ), . . . , (Xn , Yn )] denote the predictor obtained by training on all n training data points. We then output the prediction set n o Cαlwo = y ∈ Y : score(y, fb(Xn+1 )) ≤ Quantile1−α (s1 , . . . , sn ) . 6
(3)
To describe our procedure in words, note that score sk is computed by treating the data at time index k as a proxy for the test point: it is generated by comparing the prediction fbk (Xk ) to the corresponding output Yk , just like in the vanilla jackknife. The main difference is in how we compute the predictor fbk in Eq. (2a). Unlike in the vanilla jackknife, our LWO predictor is trained on all the data except for a window of points corresponding to time indices k, . . . , k + τ . Figure 2 illustrates how the score is computed; note that the training set now has both a “pre-window” and “post-window” component. The post-window component may still use lagged information from the window (through the lifted covariates). Note that the typical amount of computation needed for LWO is comparable to the vanilla jackknife; indeed, we recover the vanilla jackknife described in Section 1.1 when τ = 0 (if we use the residual score, score(y, fb(Xn+1 )) = |y − fb(Xn+1 )|). Note that our algorithm involves a hyperparameter, which is the window size τ . A natural heuristic to choose τ is to have it be at least as large as the effective dependence range that we expect in the problem. In practice, this can be guided by domain knowledge, autocorrelation diagnostics, or validation-based selection over a small grid of candidate window lengths.
2.3
Overview of remaining contributions
As we illustrated in Figure 1, standard leave-one-out conformal methods can fail under temporal dependence, but the LWO method from Section 2.2 can maintain validity despite such dependence. In the rest of the paper, we provide theoretical results that rigorously establish this validity. Specifically: 1. We develop an analysis framework for LWO showing that its coverage loss is bounded under a natural stability assumption; we introduce this assumption along with some other useful notions in Section 3.1. Later in Section 3, we introduce the cyclic embedding coefficient of a process to measure how close the data sequence (X1 , Y1 ), . . . , (Xn+1 , Yn+1 ) is to a longer cyclically exchangeable sequence, and prove two finite-sample coverage bounds for LWO in terms of this coefficient. Our first result, Theorem 1, is based on embedding the original data sequence, and our second result, Theorem 2, is based on embedding a carefully modified version of the data sequence. 2. We bound the cyclic embedding coefficient under standard dependence conditions. We first study approximately Markovian processes in Section 3.2 and control the cyclic embedding coefficient using mixing properties of the process. Then, we study general (not necessarily Markovian) mixing processes in Section 3.3 and relate the resulting cyclic embedding coefficient to the switch coefficients of the process. Together, these results, presented in Corollaries 1 and 2 respectively, show that LWO produces valid prediction intervals despite natural notions of temporal dependence. 3. In Section 4, we demonstrate the empirical benefits of LWO on simulated and real time series. The experiments illustrate that LWO achieves valid coverage under dependence even when the vanilla jackknife fails, while remaining substantially more data-efficient than split conformal prediction. Section 5 collects proofs of all our main results, with technical lemmas postponed to Appendix A. We conclude with a discussion of open questions in Section 6, and present additional experimental results in Appendix B. 7
3
Theoretical results
In this section, we state theoretical results for LWO under natural dependence assumptions. Along the way, we introduce new coefficients to measure the extent to which cyclic exchangeability is violated by a dependent sequence.
3.1
Preliminary setup
Before we present our theoretical results, we present a useful reduction, some notation, and a natural stability assumption. 3.1.1
Reduction to zero memory predictors, dummy data points, and masking
Note that our setting can be equivalently described as follows. Let X := (X × Y)L × X . We have the stochastic process Z = (Z1 , . . . , Zn+1 ), where Zt = (Xt , Yt ) is a random variable taking values in Z := X × Y. Each memory-L predictor can be viewed as a function f : X → Y that produces, for each t ≥ 1, a prediction f (Xt ) for Yt . It thus suffices to consider memoryless prediction (i.e., standard regression functions Xt 7→ f (Xt )) for the stochastic process (Zt ). Given this reduction, the setting for the rest of the paper is as follows. Given Z1 , . . . , Zn and Xn+1 , our goal is to construct a prediction interval for the test response Yn+1 with coverage assessed marginally over Z. It is useful to also define a notion of dummy data points. A dummy data point is denoted by ⋆, and we define the augmented data space Z⋆ := Z ∪ {⋆}. Analogously, the augmented covariate space is given by X∗ = X ∪ ⋆ and the augmented response space by Y∗ = Y ∪ ⋆. We abuse notation slightly and say that if Z = (X, Y ) is a dummy data point, then X = Y = ⋆. We adopt the following conventions for dummy points. First, any predictor satisfies f (⋆) = ⋆, and the S F⋆ denotes k augmented space of predictors. Second, the training algorithm is extended to Alg : k≥1 (Z⋆ ) → F⋆ with the convention that dummy points in the training input are ignored by the training algorithm, and Alg always outputs a dummy point if it is trained only on dummy points. Finally, the score function is extended so that any comparison involving a dummy point contributes zero score, i.e. score(⋆, f (⋆)) = 0 for any f ∈ F⋆ . Having dummy data points then allows us to define a masking operation. Definition 1 (The masking operation). Fix any m ≥ 1 and any τ ∈ {0, . . . , m − 1}, and let w = (w1 , . . . , wm ) be any vector. For each k ∈ {−τ, . . . , m − 1}, we define a masked version of the vector w as ( ⋆, i ∈ {k + 1, . . . , k + τ }, Mk,τ (w) i = wi , i ∈ [m] \ {k + 1, . . . , k + τ }. In other words, the masked vector Mk,τ (w) is obtained from w by replacing entries wi , for any i = k + 1, . . . , k + τ , with ⋆. In particular, for 1 ≤ k ≤ m − τ − 1, the masked vector can be written as Mk,τ (w) = w1 , . . . , wk , ⋆, . . . , ⋆, wk+τ +1 , . . . , wm , | {z } τ times
for m − τ ≤ k ≤ m − 1 as Mk,τ (w) = w1 , . . . , wk , ⋆, . . . , ⋆ | {z }
(m−k) times
8
.
and for −τ ≤ k ≤ 0, Mk,τ (w) =
⋆, . . . , ⋆ , wk+τ +1 , . . . , wm . | {z }
(k+τ ) times
By this convention, we have M−τ,τ (w) = w. Note that masking can be applied in composition. In particular, Mk′ ,τ ′ Mk,τ (w) is the vector obtained by masking the two blocks beginning at indices k + 1 and k ′ + 1. In particular, once an entry is masked, it remains masked under any subsequent masking operation. Therefore, the composition is invariant to the order in which the two masking operations are applied. With this notation in hand, we can now write the LWO method as applied directly to the sequence Z = (Z1 , . . . , Zn+1 ) with Zt = (Xt , Yt ) ∈ Z. Fix a window length τ ≥ 0. For each k = 1, . . . , n, we train a predictor by leaving out (or equivalently, masking) the block of indices k, . . . , min{k + τ, n}, namely fbk = Alg Mk−1,τ +1 (Z1 , . . . , Zn ) , and compute the corresponding score sk = score(Yk , fbk (Xk )). Let fb = Alg[Z1 , . . . , Zn ] denote the predictor trained on all n training points. The LWO prediction set for the test response at time n + 1 is then o n Cαlwo [Z1 , . . . , Zn ](Xn+1 ) = y ∈ Y : score(y, fb(Xn+1 )) ≤ Quantile1−α (s1 , . . . , sn ) , which reduces to the vanilla (leave-one-out) jackknife when τ = 0. We also define an inflated prediction interval o n Cαlwo,t [Z1 , . . . , Zn ](Xn+1 ) = y ∈ Y : score(y, fb(Xn+1 )) ≤ Quantile1−α (s1 , . . . , sn ) + t , where the quantile is inflated by t ≥ 0. We often suppress the dependence on Z1 , . . . , Zn and use the shorthand Cαlwo (Xn+1 ) and Cαlwo,t (Xn+1 ) for these prediction sets. As a final piece of notation, let us introduce our convention for subvectors. If w = (w1 , . . . , wm ) is a length m vector and 1 ≤ a ≤ b ≤ m, we let wa:b = (wa , . . . , wb ) denote the subvector whose indices range from (and include) a to b. For example, the vector Z1:n denotes the training sequence. Since the method involves computing each score by leaving out (or masking) a τ -length block from the training data, the LWO method is particularly suited for problems in which the (score, Alg) pair satisfies a natural notion of stability. We introduce this notion next. 3.1.2
Out-of-sample stability to masking a random training block
The following definition is a general notion of out-of-sample (OOS) stability, and is stated for a general random sequence Z′ ∈ Z n+1 : Definition 2. For a pair of scalars (ν, t) and a natural number ℓ, the pair (score, Alg) is said to be OOS stable with respect to masking a random training block of length ℓ from Z′ = (X′i , Yi′ )n+1 i=1 if for K ∼ Unif({1 − ℓ, . . . , n − ℓ}), we have ( ) P
′ ′ score(Yn+1 , Alg[MK,ℓ (Z1′ , . . . , Zn′ )](X′n+1 )) − score(Yn+1 , Alg[Z1′ , . . . , Zn′ ](X′n+1 )) ≤ t
9
≥ 1 − ν.
In other words, the score computed on the (n + 1)-th (test) data point is stable to leaving out a uniformly random block of length ℓ from the n (training) data points.5 Note that this is an out-of-sample (or test-time) notion, since stability is always measured with respect to the test-point prediction and not the training point predictions. Other notions of stability have been defined for algorithms on dependent data (Mohri and Rostamizadeh, 2010). One such definition is uniform stability (Mohri and Rostamizadeh, 2010), which measures in-sample stability under deletion of any single training point. Definition 2 is weaker, since it is both an out-of-sample notion of stability, and averaged over the deletion of all contiguous blocks of length ℓ. Examples of stable algorithms. To be concrete, let us give some concrete examples of stable algorithms when the score function is given by the absolute residual score(y, yb) = |y − yb|. First, it is known that ridge regression, kernel ridge regression, and elastic net satisfy in-sample uniform stability under standard boundedness and strong-convexity assumptions (Bousquet and Elisseeff, 2002; Mohri and Rostamizadeh, 2010). For these algorithms, it can be verified that the (score, Alg) pair is OOS stable with respect to any Z in the sense of Definition 2 with
ν=0
and
t=C·
ℓ−1 X r=0
1 n ≤ C log , n−r n−ℓ
(4)
where C > 0 depends on the boundedness and strong convexity parameters. A second example (which is not in-sample stable) is the k-nearest neighbor algorithm, a particular instance of which was used in our experiments in Figure 1. In this case, the prediction at the test point changes only if the deleted block intersects the k nearest neighbors of the test covariate, and so OOS stability in the sense of Definition 2 is satisfied with ν=
kℓ n
and
t = 0.
(5)
To give a third example, we note that if Alg is obtained by bagging bounded base predictors, then OOS stability in the sense of Definition 2 is satisfied even when the underlying base learner is unstable. In particular, the guarantees of Soloff et al. (2024), which show OOS stability to leaving one training point out, can be straightforwardly extended via application of the triangle inequality to show stability of bagged predictors in the sense of Definition 2. Having introduced our method as well as a suitable stability definition on the (score, Alg) pair, we are now in a position to provide theoretical guarantees under certain assumptions on the process Z.
3.2
Guarantees when Z is approximately Markovian and mixing
A canonical class of time series models can be described as Markov chains—these include autoregressive processes and state space models, among others. In this section, we show that LWO attains approximate coverage for this class of processes provided that the process is suitably mixing and the (score, Alg) pair is OOS stable with respect to Z. To arrive at this conclusion, we introduce a new conceptual tool: embedding the given time series Z into a longer cyclically exchangeable sequence. 5
By the definition of the masking operation, we also include cases in which blocks of size ≤ ℓ are left out at the start of the training sequence, corresponding to when K takes negative values.
10
3.2.1
Coverage of LWO when Z is (approximately) cyclically embeddable
To understand cyclical embeddings of Z, we first introduce cyclic exchangeability. We use the d notation = to denote equality in distribution. e = (Z e1 , . . . , Zem ) ∈ Z∗m is said to satisfy cyclic Definition 3 (Cyclic exchangeability). A sequence Z exchangeability if d e e= em , Ze1 , . . . , Z ei−1 ) Z (Zi , . . . , Z for all i ∈ [m]. Let S(m) denote the collection of all cyclically exchangeable sequences of length m. Our first result is that if Z can be embedded into a cyclically exchangeable sequence of length n + 1 + τ , then we have approximate validity for the inflated LWO prediction set with window size τ . d
e ∈ S(n + 1 + τ ) such that Z = Z e 1:(n+1) . If Alg is symmetric Proposition 1. Suppose there exists Z and the pair (score, Alg) is OOS stable with respect to Z in the sense of Definition 2 with parameters (ν, t, τ + 1), then √ τ +1 P Yn+1 ∈ Cαlwo,t (Xn+1 ) ≥ 1 − α − 2 ν − . n We prove Proposition 1 in Section 5.1. Notably, if Z is an exchangeable sequence, then it is itself cyclically exchangeable, and can therefore be trivially embedded into a cyclically exchangeable sequence of length n+1. Proposition 1 then implies that the vanilla jackknife (or equivalently, LWO with τ = 0) satisfies approximate validity, thereby recovering the result of Barber et al. (2021). Another example of a cyclically embeddable process is a moving average process of memory q, as simulated in Figure 1. If Z is drawn from a stationary MA(q) process, then it is embeddable into a cyclically exchangeable process of length at least n + 1 + q. Proposition 1 then guarantees that LWO with a window size of at least q will achieve approximate validity. As a related example, for the process in Figure 1 (which is cyclically embeddable6 into a sequence of length n + 1 + q for any q ≥ 2), we used τ = 5 when implementing LWO; clearly, it remains valid for this choice of window size. While Proposition 1 covers exact cyclical embedding, it can be naturally extended to the situation in which such an embedding can only be performed approximately, as articulated by the following coefficient. Definition 4 (Cyclic embedding coefficient). For a nonnegative integer a, define the cyclic embedding coefficient of Z with lengthening a as ρa (Z) :=
inf
e 1:(n+1) ). dTV (Z, Z
e Z∈S(n+1+a)
In Section 5.2, we combine the definition of this coefficient with Proposition 1 to arrive at the following theorem. Theorem 1. If Alg is symmetric and the pair (score, Alg) is OOS stable with respect to Z in the sense of Definition 2 with parameters (ν, t, τ + 1), then p τ +1 P Yn+1 ∈ Cαlwo,t (Xn+1 ) ≥ 1 − α − ρτ (Z) − − 2 ν + ρτ (Z). n 6
Even though the covariates follow MA(1), the response adds a point in the future, thereby increasing memory by one.
11
Theorem 1 is proved in Section 5.2. While Theorem 1 shows that the loss of coverage can be bounded by the cyclic embedding coefficient ρτ (Z), we would ideally be able to show that the cyclic embedding coefficient is related to standard mixing coefficients, in order to be able to interpret the implications of this result. We relate these two notions in the next section under the additional assumption that Z is approximately Markovian. 3.2.2
Relating the cyclic embedding coefficient to mixing under Markov property
The following definition is a standard measure of dependence in a time series (e.g. Doukhan, 1994). Definition 5 (β-mixing coefficient). For a sequence Z ∈ Z n+1 , the β-mixing coefficient with lag τ is defined as ′ ′ β(τ ) := max dTV (Z1 , . . . , Zk , Zk+τ +1 , . . . , Zn+1 ) , Z1 , . . . , Zk , Zk+τ , +1 , . . . , Zn+1 1≤k≤n−τ
′ ) ∈ Z n+1 denotes an i.i.d. copy of Z. where Z′ = (Z1′ , . . . , Zn+1
As mentioned before, we will also consider sequences that are approximately Markovian by measuring dependence in terms of the following β-conditional-mixing coefficient: Definition 6 (β-conditional-mixing coefficient). For a sequence Z ∈ Z n+1 , the β-conditionalmixing coefficient with lag τ is defined as ′′ ′′ , β ∗ (τ ) := max dTV Z, Z1′ , . . . , Zk′ , Zk+1 , . . . , Zk+τ , Zk+τ +1 , . . . , Zn+1 1≤k≤n−τ
where (Z1′ , . . . , Zk′ ) is sampled from the distribution P(Z1 ,...,Zk )|(Zk+1 ,...,Zk+τ ) ( · |Zk+1 , . . . , Zk+τ ), and ′′ ′′ (Zk+τ +1 , . . . , Zn+1 ) is sampled from the distribution P(Zk+τ +1 ,...,Zn+1 )|(Zk+1 ,...,Zk+τ ) ( · |Zk+1 , . . . , Zk+τ ). Our definition of a conditional β-mixing coefficient does not seem to have appeared in the literature, although related notions (e.g., by Prakasa Rao (2009)) are common. To develop intuition for this definition, suppose Z is such that for any decomposition Z = (ZA , ZB , ZC ) where the block B has length τ , we have the Markovian relation ZA ⊥ ⊥ ZC | ZB . In this case, we would have ∗ β (τ ) = 0. Consequently, any mixing Markov chain has both β(τ ) and β ∗ (τ ) small provided τ is larger than the mixing time of the chain. The following proposition then shows that the cyclic coefficient ρτ (Z) can be bounded by combining the above mixing coefficients. Proposition 2. Suppose Z is a stationary sequence, with β-conditional-mixing coefficient β ∗ (τ ) and β-mixing coefficient β(τ ). Then ρτ (Z) ≤ 2β(τ ) + 2β ∗ (τ ) +
4τ . n+τ +1
Proposition 2 is proved in Section 5.3. Combining Theorem 1 and Proposition 2 immediately yields the following corollary. Corollary 1. If Alg is symmetric and the pair (score, Alg) is OOS with parameters (ν, t, τ + 1), and Z is stationary, β-conditional mixing and β-mixing, then r 5τ + 1 P Yn+1 ∈ Cαlwo,t (Xn+1 ) ≥ 1 − α − 3 ν + 2β(τ ) + 2β ∗ (τ ) + . n 12
Taking stock, Corollary 1 shows that the validity of the LWO method is governed by two sources of dependence at scale τ . The first source β ∗ (τ ) measures how close to Markovian the sequence is when we allow ourselves to condition on blocks of length τ , while the second source β(τ ) measures whether the sequence is rapidly mixing. In particular, if τ is chosen large enough that both quantities are small, then the LWO method achieves approximate coverage. As previously mentioned, the prototypical example where this is possible for small τ is a fast mixing Markov chain. While this is an important class of stochastic processes, the next section provides a result for a larger class of sequences.
3.3
Guarantees when Z is only mixing
In practice, it may be the case that Z is a mixing process without being approximately Markovian. Is the LWO method still an approximately valid procedure in such a setting? In this section, we answer this question in the affirmative provided the (score, Alg) pair satisfies a slightly stronger stability condition. In order to prove our eventual result (given in Corollary 2), we proceed via cyclically embedding masked versions of Z instead of Z itself. 3.3.1
Coverage of LWO when masked Z is cyclically embeddable
To state our result, we require the notion of switch coefficients, which were introduced in Barber and Pananjady (2026) as a measure of departure from exchangeability; our definition is a slight modification of theirs. Definition 7 (Switch coefficients). For each k ∈ {−τ, . . . , n}, define (Z1 , . . . , Zk , Zk+τ +1 , . . . , Zn+1 ), ∆0k,τ (Z) := (Z1 , . . . , Zk ), (Zk+τ +1 , . . . , Zn+1 ),
1 ≤ k ≤ n − τ, n − τ + 1 ≤ k ≤ n, −τ ≤ k ≤ 0,
and (Z , . . . , Zn+1 , Z1 , . . . , Zn−k−τ +1 ), n+2−k ∆1k,τ (Z) := (Zn+2−k , . . . , Zn+1 ), (Z1 , . . . , Zn−k−τ +1 ),
1 ≤ k ≤ n − τ, n − τ + 1 ≤ k ≤ n, −τ ≤ k ≤ 0.
The k-th switch coefficient of Z with lag τ is defined as Ψk,τ (Z) := dTV ∆0k,τ (Z), ∆1k,τ (Z) . The average switch coefficient is given by n
Ψτ (Z) :=
X 1 Ψk,τ (Z). n+τ +1 k=−τ
Our main result of this section expresses the loss of coverage of the LWO method using the coefficient ρτ (MK,τ (Z)) and the switch coefficient Ψτ (Z). Recall our notion of OOS stability from Definition 2. 13
Theorem 2. Suppose Alg is symmetric and let K ∼ Unif({−τ, . . . , n}) be drawn independently of the data. Now suppose that the pair (score, Alg) is OOS stable with respect to Z with parameters (ν, t, τ ), OOS stable with respect to MK,τ (Z) with parameters (ν, t, τ + 1), and OOS stable with respect to MK,τ +1 (Z) with parameters (ν, t, τ ). Then we have r P(Yn+1 ∈ Cαlwo,3t (Xn+1 )) ≥ 1 − α − 3
2τ + 2 ν+ + ρτ (MK,τ (Z)) − 2 n
r ν+
τ n+τ +1 + · Ψτ (Z). n n
We provide the proof of Theorem 2 in Section 5.4. The result of Proposition 3 to follow yields that ρτ (MK,τ (Z)) ≲ Ψτ (Z) ≲ ρτ (Z), so ignoring absolute constants, we see that Theorem 2 provides a stronger guarantee than its counterpart Theorem 1. On the other hand, Theorem 2 relies on a stronger stability assumption than the one made in Theorem 1. In particular, we now require that the (score, Alg) is leave-one-block-out stable not just with respect to the given sequence Z but also with respect to its masked counterparts MK,τ (Z) and MK,τ +1 (Z). This stability condition can be ensured if the (score, Alg) pair is OOS stable to leaving two random blocks—of sizes τ and τ + 1—out of the training data, as opposed to a single block (as required by Theorem 1). 3.3.2
Relating cyclic coefficient under masking to mixing
Since our eventual goal is to show a coverage result for mixing sequences, we now relate our cyclic embedding coefficient ρτ (MK (Z)) to the switch coefficients from Definition 7. Proposition 3. We have Ψτ (Z) ≤ 2ρτ (Z).
(6a)
Moreover, for K ∼ Unif({−τ, . . . , n}) drawn independently of the data, we have ρτ (MK,τ (Z)) ≤ Ψτ (Z) + Ψ0,τ (Z).
(6b)
We prove Proposition 3 in Section 5.5. Note that if Z is stationary, then Ψ0,τ (Z) = 0. Moreover, the results of Barber and Pananjady (2026) show that in this case7 n+τn+1 · Ψτ (Z) ≤ 2β(τ ), with no dependence on the conditional mixing coefficient. Combining these pieces then immediately yields the following corollary when Z is stationary and mixing. Corollary 2. Suppose Alg is symmetric and K ∼ Unif({−τ, . . . , n}) is drawn independently of the data. Also suppose that the pair (score, Alg) is OOS stable with respect to Z with parameters (ν, t, τ ), OOS stable with respect to MK,τ (Z) with parameters (ν, t, τ + 1), and OOS stable with respect to MK,τ +1 (Z) with parameters (ν, t, τ ). If Z is stationary and β-mixing, then r 2τ + 2 P Yn+1 ∈ Cαlwo,3t (Xn+1 ) ≥ 1 − α − 5 ν + + 2β(τ ). n
Corollary 2 should be viewed as complementary to Corollary 1. While Corollary 1 is useful for mixing sequences that are also approximately Markovian (having small β ∗ (τ ) and β(τ ) for an appropriately small τ ), Corollary 2 does not require any Markovian structure—the β-mixing condition alone suffices. However, we now need an additional stability assumption with respect to the masked sequences MK,τ (Z) and MK,τ +1 (Z). 7
When Z is stationary, note that our Ψ(Z) is equal to that of Barber and Pananjady (2026) up to a multiplicative normalization factor.
14
4
Experiments
In this section, we compare LWO with split CP (which, as discussed before, offers robustness to temporal dependence but may lead to larger prediction sets), and the vanilla jackknife (JK) (which may lead to undercoverage in a time series setting) on both synthetic and real data. These two baselines serve to illustrate the main trade-off between coverage and interval length, and our goal is to see if LWO navigates this trade-off favorably.8
4.1
Simulated data
We use our simulation setup from Section 1.2, which involves multidimensional responses. We also include a simulation experiment with univariate responses in Appendix B. 4.1.1
Data generation
As in the setup of Section 1.2, we generate our covariates as a sequence of random vectors Xt ∈ Rd according to i.i.d.
ωt ∼ N (0, Id ),
Xt = ωt + ωt−1 ,
in dimension d = 50. The response at time t is given by Yt = Xt+1 . We consider the memoryless setting with Xt = Xt . The prediction problem is thus to use the current observation Xt to forecast the next vector Xt+1 . Our MA(1) process Z exhibits mild dependence. It can be verified that ρτ (Z) = 0 for all τ ≥ 2, and its β-mixing coefficients satisfy β(τ ) = 0 for all τ ≥ 2. 4.1.2
Predictors and baseline methods
We compare five different predictors. None of the predictors has an internal recurrent or sequential state, and the only memory (if any) enters through the covariates. • Ridge: Linear regression with ℓ22 -regularization parameter λ = 1.0. As discussed in Eq. (4), this predictor satisfies OOS stability in the sense of Definition 2. • KNN: k-nearest-neighbor regression with k = 10. As discussed in Eq. (5), this predictor is also OOS stable in the sense of Definition 2. • MLP: A multilayer perceptron with one hidden layer of width 20 and ReLU activations. We include MLP as a flexible nonlinear predictor, but this predictor is not verifiably stable in the sense of Definition 2. • Decision Tree (DT): A regression tree with maximum depth 5 and minimum leaf size 2. Decision trees are typically unstable under small perturbations of the training set, so they represent a deliberately less stable predictor. • Kernel Regression (KR): Gaussian kernel regression with bandwidth 0.5. KR ought to be stable when the kernel weights are sufficiently spread out. We use the target coverage level 1 − α = 0.9 and set n = 200. We compare the following predictive inference procedures: 8
Code for our experiments is available at https://github.com/scotthanyang/LWO.
15
Split CP
Jackknife
0.90 0.85 0.80
10 5
0.75 0.70 Ridge KNN
LWO
15
Radius
Empirical coverage
0.95
MLP
DT
0 Ridge KNN
KR
MLP
DT
KR
Figure 3. Empirical performance on the multidimensional MA(1) process across five base predictors. (Left) Empirical coverage of Split CP, Jackknife, and LWO, with the dashed horizontal line marking the nominal 90% coverage level. (Right) Average prediction region radius for the same methods and predictors. Bars show means across 500 repeated trials, and error bars indicate ±1 standard error.
• Split CP: standard split conformal prediction. We split the sample evenly into a training set (50%) and a calibration set (50%). • Jackknife (JK): the vanilla leave-one-out procedure. • Leave-a-Window-Out (LWO): our proposed method with window size τ = 5. 4.1.3
Evaluation
We report empirical coverage and mean prediction-set size in each setting, averaged over 500 independent trials. Prediction set size is measured by the average radius of the Euclidean prediction ball. In the figures, bars show trial averages and error bars indicate ±1 standard error over replications. Figure 3 shows that LWO is well calibrated on the MA(1) process, with empirical coverage close to the nominal 0.9 level across all five predictors. By contrast, the vanilla jackknife undercovers for several predictors, especially Ridge, KNN, and KR. Split CP exhibits close-to-nominal coverage for all five predictors, but returns significantly larger prediction sets than the other methods. Overall, Figure 3 corroborates our theory: LWO preserves near-nominal coverage (comparable to split CP) while improving efficiency. To develop some intuition for the coverage of JK and LWO in this example, note that in the vanilla jackknife, there is an asymmetry between the test point and the leave-one-out proxy test points. At test time, predicting Yn+1 = Xn+2 = ωn+1 + ωn+2 is intrinsically “one-sided”: the training data contains information about ωn+1 , but does not contain information about the new innovation ωn+2 . In contrast, when an interior point t is used as a proxy test point in the vanilla jackknife, the leave-one-out training set still contains observations on both sides of t, carrying information about both ωt and ωt+1 when predicting Yt = Xt+1 = ωt + ωt+1 . Thus, the jackknife calibrates on easier, “two-sided” prediction problems (leading to small prediction regions) and so undercovers at test time. LWO breaks this asymmetry by removing the local future window around each proxy test point, preventing the calibration score from using observations that reveal the future innovation. As a result, its calibration residuals better mimic the information structure of the residual at the final test point. 16
4.2
Real data
In this section, we evaluate LWO on two publicly available real time-series datasets. 4.2.1
Datasets
We consider the following two benchmark datasets. • Traffic. This dataset contains hourly road occupancy rates (values between 0 and 1) collected from freeway sensors in the San Francisco Bay Area during 2015–2016. • Solar Energy. This dataset contains solar power production measurements sampled every 10 minutes from 137 photovoltaic plants in Alabama during 2006. On both datasets, we construct a univariate forecasting task by choosing a single coordinate of the multivariate data at each time index. Let Wt denote the resulting scalar time series. We consider one-step-ahead prediction with lag length L = 24, so that the covariate at time t is Xt = (Wt−L+1 , . . . , Wt ), and the response is Yt = Wt+1 . Thus, each predictor takes the most recent 24 observations as input and predicts the next one. To obtain multiple replicates of data from a single long series, we partition each dataset into disjoint local trajectories (“chunks”). Each chunk provides one forecasting problem of size n, and consecutive chunks are separated by a gap of 48 time steps. We use n = 100 for the Traffic dataset and n = 300 for the Solar Energy dataset. For both datasets, we fix the LWO window length at τ = 20. These choices (L = 24, τ = 20, and chunk gap 48) are held fixed across all predictors and datasets, and we do not tune them separately for different methods. 4.2.2
Predictors and baseline methods
We compare the same five predictors (with the same hyperparameters) and three baseline methods described in Section 4.1.2. As described above, for our LWO method, we use a window size τ = 20 for both real datasets. 4.2.3
Evaluation
The real-data experiments in Figure 4 reveal that JK can still undercover when temporal dependence interacts unfavorably with the predictor. This failure is most pronounced on the Solar Energy dataset with KNN, where JK falls well below nominal coverage, while LWO recovers coverage close to the target. A milder version of the same phenomenon appears on the Traffic dataset, where JK is generally competitive but can undercover for predictors such as DT. In contrast, LWO is more consistent across predictors, typically maintaining near-nominal coverage. Figure 4 also shows the expected efficiency trade-off. Split CP is often well calibrated, but it tends to produce wider intervals because it trains on only part of the data. LWO avoids much of this split tax while still guarding against the local-dependence failures that affect leave-oneout methods. Overall, these real-data results suggest that LWO transfers beyond the synthetic examples: it is generally more robust than JK under dependence, and often more efficient than Split CP. 17
Jackknife
Width
0.90 0.85
1.00
0.15 0.10 0.05
MLP
DT
0.00 Ridge KNN
KR
Split CP
Jackknife
MLP
DT
KR
MLP
DT
KR
LWO
12 10
0.95 0.90 0.85 0.80 0.75 Ridge KNN
LWO
0.20
0.95
0.80 Ridge KNN
Empirical coverage
Split CP
Width
Empirical coverage
1.00
8 6 4 2
MLP
DT
0 Ridge KNN
KR
Figure 4. Empirical performance on two real data benchmarks. (Top) Traffic dataset. (Bottom) Solar Energy dataset. In each row, the left panel reports empirical coverage, and the right panel reports average prediction-set size across base predictors. Bars show means across repeated trials, and error bars indicate ±1 standard error. The dashed horizontal line marks nominal 90% coverage.
5
Proofs of main results
In this section, we present proofs of all our main results. Before proceeding, we extend the definition of the masking operation Mk,τ (w) applied to a vector w of length m to indices k ∈ / {−τ, . . . , m − 1}, by interpreting such indices modulo m + τ . More precisely, if k ∈ / {−τ, . . . , m − 1} then define k ′ := (k + τ ) mod (m + τ ) − τ , and define the corresponding masking operation as Mk,τ (w) := Mk′ ,τ (w),
(7)
where the RHS should be interpreted according to Definition 1, since k ′ ∈ {−τ, . . . , m − 1}.
5.1
Proof of Proposition 1
e ∈ S(n + τ + 1) be the cyclically exchangeable sequence into which Z embeds, so that Let Z d e ⟳ n+τ +1 , as follows. For k ∈ [n + τ + 1], Z=Z 1:(n+1) . Define the vector of “circular scores” s ∈ R write e e e e s⟳ k = score(Yk , Alg[Zk+τ +1 , . . . , Zk−1 ](Xk )) with the convention that the index that follows n + τ + 1 is 1. In words, one should think of first e on a circle of length n + τ + 1. To evaluate the k-th score, we leave out the placing the process Z 18
After rotation: score at Zen+1
ek Before rotation: score at Z e1 en+τ +1 Z Z
e1 en+τ +1 Z Z
···
···
··· Equivalent by cyclic rotation ⟳
en+1 Z
···
en+2 Z en+1 Z ···
ek Z ek+1 Z ···
···
···
···
ek+τ Z
ek+τ +1 Z
Evaluation point
···
Training point
LWO block
Figure 5. Illustration of the circular embedding used to compute the scores s⟳ . We place the e = (Ze1 , . . . , Z en+τ +1 ) on a circle by gluing the end of the sequence cyclically exchangeable sequence Z e e1 . Black points indicate observations used for back to its beginning, so that Zn+τ +1 is followed by Z training, the blue point is the evaluation point, and the omitted LWO block is indicated by a red dashed arc together with red points marking the specific omitted observations. (Left) To compute e s⟳ k , the score is evaluated at Zk , and the following τ points on the circle are omitted. (Right) By e cyclic exchangeability of Z, this is equivalent to evaluating the score at Zen+1 , with the LWO block en+τ +1 . shifted to Zen+2 , . . . , Z
τ points appearing after index k on the circle and use the remaining points to train the predictor. See Figure 5(left) for an illustration. Note that by construction, we have d be b e s⟳ n+1 = score(Yn+1 , f (Xn+1 )) = score(Yn+1 , f (Xn+1 )), d
e 1:(n+1) . where equality in distribution holds since Z = Z e and by symmetry of Alg, the scores s⟳ , . . . , s⟳ By cyclic exchangeability of Z 1 n+τ +1 are exchangeable, and consequently, ⟳ ⟳ ′ P s⟳ n+1 ≤ Quantile1−α′ (s1 , . . . , sn+τ +1 ) ≥ 1 − α , for any choice of α′ . From Lemma 3 in Appendix A.3, it holds that ⟳ ⟳ ⟳ Quantile1−α′ (s⟳ 1 , . . . , sn+τ +1 ) ≤ Quantile(1−α′ )(1+ τ +1 ) (s1 , . . . , sn ), n
so that ⟳ ⟳ P s⟳ ≤ Quantile (s , . . . , s ) ≥ 1 − α′ . τ +1 ′ n n+1 (1−α )(1+ ) 1
(8)
n
⟳ It remains to relate the quantile of the scores s⟳ 1 , . . . , sn to the quantile of the scores s1 , . . . , sn d e computed by the LWO algorithm when applied to Z. Since Z = Z 1:(n+1) , we can think of sk e en ). On the other hand, computed with Alg applied to the masked sequence Mk−1,τ +1 (Z1 , . . . , Z ⟳ en+τ +1 , Z e1 , . . . , Z ek−1 ). By cyclic the scores s are computed with Alg applied to (Zek+τ +1 , . . . , Z k
19
exchangeability and symmetry, we therefore have that for all k ∈ [n], |sk − s⟳ k| d e k) en )](X = score Yek , Alg[Mk−1,τ +1 (Ze1 , . . . , Z e k) e1 , . . . , Zek−1 , Zek+τ +1 , . . . , Z en+τ +1 ](X − score Yek , Alg[Z d e n+1 ) − score Yen+1 , Alg[Z e n+1 ) en )](X e1 , . . . , Zen ](X = score Yen+1 , Alg[Mn−k−τ,τ +1 (Ze1 , . . . , Z (9) d = score Yn+1 , Alg[Mn−k−τ,τ +1 (Z1 , . . . , Zn )](Xn+1 ) − score Yn+1 , Alg[Z1 , . . . , Zn ](Xn+1 ) . (10) d e Eq. (10) holds because Z = Z 1:(n+1) ; let us explain the relation (9) in more detail. Note that we have d e n+1 ) = e k) e1 , . . . , Zen ](X ek−1 , Z ek+τ +1 , . . . , Zen+τ +1 ](X score Yen+1 , Alg[Z score Yek , Alg[Ze1 , . . . , Z
as illustrated in Figure 5(right), and equality in distribution of the first term in (9) follows by a similar argument (where there are two red blocks left out instead of just one). Now by assumption, the (score, Alg) pair satisfies OOS stability with respect to Z with parameters (ν, t, τ + 1). Thus, the RHS of Eq. (10) averaged over k ∈ [n] is bounded above by ν. Consequently, we have n
1X P |s⟳ k − sk | > t ≤ ν. n
(11)
k=1
Applying Lemma 4 from Appendix A.3 then yields √ ⟳ , . . . , s ) ≤ Quantile (s , . . . , s ) + t ≥ 1 − ν, P Quantile(1−α′ )(1+ τ +1 ) (s⟳ 1 n 1−α n 1 n
if we define α′ to satisfy 1 − α′ =
√ 1−α− ν . 1 + τ +1 n
√ √ √ ν Putting together the pieces, we see that the coverage is at least 1−α− − ν ≥ 1 − α − 2 ν − τ +1 n , 1+ τ +1 n
as claimed.
5.2
Proof of Theorem 1
e ∈ S(n + τ + 1), noting First, we repeat the steps of the proof of Proposition 1 for an arbitrary Z that Eq. (8) still holds. Reproducing it below, we have ⟳ ⟳ P s⟳ ≤ Quantile (s , . . . , s ) ≥ 1 − α′ . τ +1 ′ n n+1 (1−α )(1+ ) 1 n
⟳ The difference from this point on is in how we relate the scores s⟳ 1 , . . . , sn to the scores s1 , . . . , sn computed by the LWO algorithm when applied to Z. Define sek as the LWO score computed using e and symmetry of e1 , . . . , Z en ). By cyclic exchangeability of Z the predictor trained on Mk−1,τ +1 (Z Alg, we have that for all k ∈ [n],
d e n+1 ) − score Yen+1 , Alg[Z e n+1 ) . e1 , . . . , Zen )](X e1 , . . . , Zen ](X |s⟳ ek | = score Yen+1 , Alg[Mn−k−τ,τ +1 (Z k −s 20
Now, for each k ∈ [n], we may bound the tail probability of the RHS above as ( ) e n+1 )) − score(Yen+1 , Alg[Z e n+1 )) ≥ t e1 , . . . , Zen )](X e1 , . . . , Zen ](X P score(Yen+1 , Alg[Mn−k−τ,τ +1 (Z ( ≤P
) score(Yn+1 , Alg[Mn−k−τ,τ +1 (Z1 , . . . , Zn )](Xn+1 )) − score(Yn+1 , Alg[Z1 , . . . , Zn ](Xn+1 )) ≥ t
e 1:(n+1) ), + dTV (Z, Z e 1:(n+1) in the probability by Z and pay an additive price of the TV distance where we replace Z between these two sequences. Furthermore, by assumption, the pair (score, Alg) is OOS stable with respect to Z with parameters (ν, t, τ + 1). Putting this together with the above display, we have ⟳ 1 Pn e P |s − s e | > t ≤ ν + d TV (Z, Z1:(n+1) ). Applying Lemma 4 from Appendix A.3 then yields k k=1 k n q ⟳ e 1:(n+1) ), P Quantile(1−α′ )(1+ τ +1 ) (s⟳ ) ≤ Quantile (e s , . . . , s e ) + t ≥ 1 − ν + dTV (Z, Z , . . . , s n 1−α 1 n 1 n
if we define α′ to satisfy 1 − α′ =
1−α−
q e 1:(n+1) ) ν + dTV (Z, Z 1 + τ +1 n
.
Furthermore, by definition, we have s⟳ en+1 . Putting together the pieces then yields n+1 = s s1 , . . . , sen ) + t ≥ P sen+1 ≤ Quantile1−α (e
q e 1:(n+1) ) 1 − α − ν + dTV (Z, Z
−
q e 1:(n+1) ) ν + dTV (Z, Z
1 + τ +1 n q e 1:(n+1) ) − τ + 1 . ≥ 1 − α − 2 ν + dTV (Z, Z n
Since the LHS of the above expression can be expressed as the expectation of a bounded function e 1:(n+1) , an analogous bound holds for the same function applied to Z up to an additive term of Z e 1:(n+1) ). Concretely, we have dTV (Z, Z q e 1:(n+1) ) − τ + 1 − dTV (Z, Z e 1:(n+1) ). P sn+1 ≤ Quantile1−α (s1 , . . . , sn ) + t ≥ 1 − α − 2 ν + dTV (Z, Z n e ∈ S(n + τ + 1) then yields the claimed result. Taking an infimum of the TV over Z
5.3
Proof of Proposition 2
We will use notation such as P(Z1 ,...,Zτ ) to denote marginal distributions of subvectors of Z, and notation such as P(Zτ +1 ,...,Z2τ )|(Z1 ,...,Zτ ,Z2τ +1 ,...,Z3τ ) to denote conditional distributions of subvectors of Z. First we define a distribution that satisfies cyclic exchangeability. Sample (Zn+2 , . . . , Zn+τ +1 ) | Z ∼ P(Zτ +1 ,...,Z2τ )|(Z1 ,...,Zτ ,Z2τ +1 ,...,Z3τ ) · | Zn+2−τ , . . . , Zn+1 , Z1 , . . . , Zτ . 21
e as a rotation of Next, draw K ∼ Unif([n + τ + 1]) independently of (Z1 , . . . , Zn+τ +1 ), and define Z (Z1 , . . . , Zn+τ +1 ) by K positions: e = (ZK , . . . , Zn+τ +1 , Z1 , . . . , ZK−1 ). Z e ∈ S(n+τ +1) and by definition of the cyclic exchangeability coefficient, By construction, we have Z e1 , . . . , Zen+1 ) . we have ρτ (Z) ≤ dTV Z, (Z For each k ∈ [n + τ + 1], let P{k} denote the distribution of Z{k} = (Zk , . . . , Zn+τ +1 , Z1 , . . . , Zk−1 ).
(12)
e can be written as the mixture Then the distribution of Z e∼ Z
n+τ X+1 1 P{k} . n+τ +1 k=1
Therefore, since total variation distance is convex in each argument (in the space of measures), we have n+τ X+1 1 {k} {k} en+1 ) ≤ dTV Z, (Z1 , . . . , Zn+1 ) . dTV Z, (Ze1 , . . . , Z n+τ +1 k=1
Consequently, ρτ (Z) ≤
n+τ X+1 1 {k} {k} dTV Z, (Z1 , . . . , Zn+1 ) n+τ +1 k=1
(i)
≤
n−τ X+1 1 4τ {k} {k} dTV Z, (Z1 , . . . , Zn+1 ) + n+τ +1 n+τ +1 k=2τ +2
(ii)
≤
4τ + 2β(τ ) + 2β ∗ (τ ). n+τ +1 {k}
Here step (i) follows because for k = 1, we have Z1:(n+1) = Z, and for k ∈ {2, . . . , 2τ + 1} ∪ {n − τ + 2, . . . , n + τ + 1} we have bounded the total variation distance by 1. On the other hand, step (ii) follows from Lemma 1 from Appendix A.1.
5.4
Proof of Theorem 2
Throughout, suppose K is drawn uniformly at random from the set {−τ, . . . , n} independently of the data. For any k, k ′ ∈ {−τ, . . . , n}, we define the following shorthand for masked sequences: Z(k) = Mk,τ (Z), ′
Z(k,k ) = Mk,τ ◦ Mk′ ,τ (Z). Also recall our convention (7) for masking Mk,τ (Z) when k ∈ / {−τ, . . . , n}. We also define Yk = Xk = ⋆ for k ∈ / [n + 1], so that for such k, we have score(Yk , f (Xk )) = 0 for any function f . We present the proof in two steps. We first apply Theorem 1 to obtain a guarantee for LWO if applied to the masked sequence Z(K) . Next, we relate the coverage event of LWO on the masked sequence Z(K) to the coverage event of LWO on the true sequence Z. 22
(K) n+1 Step 1: Constructing coverage guarantee for algorithm applied on Z(K) . Let sk k=1 denote the scores computed by LWO on Z(K) . From the definition, the scores can be written as (K)
sk
(K)
= score(Yk
(K)
(K)
, Alg[Mk−1,τ +1 (Z1:n )](Xk )).
By assumption, the (score, Alg) pair is OOS stable with respect to Z(K) with parameters (ν, t, τ + 1). Thus, applying Theorem 1 to Z(K) , we have q τ +1 (K) (K) − ρτ (Z(K) ). P sn+1 ≤ Quantile1−α (s1 , . . . , s(K) ) + t ≥ 1 − α − 2 ν + ρτ (Z(K) ) − n n Step 2: Connecting coverage events on original and masked sequence. {−τ, . . . , n}, we begin by rewriting the LWO scores as follows: (K)
sk
(K,k)
= score(Yk
(K,k)
(K,k)
, Alg[Mk−1,τ +1 (Z1:n )](Xk
For each k ∈
)),
and (k)
(k)
(k)
sk = score(Yk , Alg[Mk−1,τ +1 (Z1:n )](Xk )). Note that if k ∈ {−τ, . . . , 0}, then we have score(Yk , f (Xk )) = 0 and so by our convention both scores defined above evaluate to zero. events on Z and Z(K) , we bound the two probabilities In order to relate the coverage P (K) (K) P |sn+1 − sn+1 | ≥ t and n1 nk=1 P(|sk − sk | ≥ t). Starting with the first term, we have (K) P |sn+1 − sn+1 | ≥ t " # X X 1 (k) (k) = P |sn+1 − sn+1 | ≥ t + P |sn+1 − sn+1 | ≥ t n+τ +1 k∈{−τ,...,n−τ −1}
k∈{n−τ,...,n}
(i)
τ +1 nν + n+τ +1 n+τ +1 τ +1 . (13) ≤ν+ n In step (i), we have used the fact that (score, Alg) pair satisfies OOS stability with respect to Z with parameters (ν, t, τ ) to bound the first term, and the fact that probabilities are bounded above by 1 to control the second term. P (K) We now turn to bounding the term n1 nk=1 P(|sk − sk | ≥ t), which ought to be small when the predictor satisfies stability when evaluated at a random training point (as opposed to the test point). But Lemma 2 in Appendix A.2 relates these two notions of stability to each other using the switch coefficients. Applying it, we have ≤
n
1X τ n+τ +1 (K) P(|sk − sk | ≥ t) ≤ ν + + · Ψτ (Z). n n+τ +1 n k=1
Having bounded the two probabilities of interest, we now proceed to relating the coverage events of LWO on Z(K) and Z. Applying Lemma 4 from Appendix A.3, we can relate the quantile of the LWO scores on Z(K) to the quantile of the LWO scores on Z via (K) (K) q P Quantile (s , . . . , s ) ≤ Quantile (s , . . . , s ) + t n +1 1−α 1 n 1−α− ν+ n+ττ +1 + n+τ Ψτ (Z) 1 n r τ n+τ +1 ≥1− ν+ + · Ψτ (Z). (14) n+τ +1 n 23
Thus, the coverage event on Z satisfies the chain of bounds P sn+1 ≤ Quantile1−α (s1 , . . . , sn ) + 3t (i) τ +1 (K) ≥ P sn+1 ≤ Quantile1−α (s1 , . . . , sn ) + 2t − ν − n (ii) (K) (K) (K) q ≥ P sn+1 ≤ Quantile (s , . . . , sn ) + t +1 1−α− ν+ n+ττ +1 + n+τ ·Ψτ (Z) 1 n r τ +1 τ n+τ +1 −ν− − ν+ + · Ψτ (Z), n n+τ +1 n where step (i) follows from Eq. (13) and step (ii) from Eq. (14). Using the definition of the quantile and performing some algebra, then yields P sn+1 ≤ Quantile1−α (s1 , . . . , sn ) + 3t r q τ n+τ +1 ≥1−α−ν−2 ν+ + · Ψτ (Z) − 2 ν + ρτ (Z(K) ) n+τ +1 n 2τ + 2 − − ρτ (Z(K) ) n r r 2τ + 2 τ n+τ +1 (K) ≥1−α−3 ν+ + ρτ (Z ) − 2 ν + + · Ψτ (Z), n n n as claimed.
5.5
Proof of Proposition 3
We prove the two claimed inequalities separately. 5.5.1
Proof of Ineq. (6a)
e ∈ S(n + τ + 1). For each k ∈ {−τ, . . . , n}, we Fix an arbitrary cyclically exchangeable sequence Z have Ψk,τ (Z) (i)
e 1:(n+1) ) + dTV ∆1 (Z e 1:(n+1) ), ∆1 (Z) e 1:(n+1) ) + dTV ∆0 (Z e 1:(n+1) ), ∆1 (Z ≤ dTV ∆0k,τ (Z), ∆0k,τ (Z k,τ k,τ k,τ k,τ
(ii)
e 1:(n+1) ) + dTV ∆0 (Z e 1:(n+1) ), ∆1 (Z e 1:(n+1) ) ≤ 2dTV (Z, Z k,τ k,τ
(iii)
e 1:(n+1) ). = 2dTV (Z, Z
Here step (i) follows by triangle inequality and step (ii) follows from the data processing inequality, since ∆0k,τ and ∆1k,τ are measurable maps of the length-(n + 1) sequence. Step (iii) follows by cyclic d e whereby ∆0 (Z e 1:(n+1) ) = e 1:(n+1) ). exchangeability of Z, ∆1k,τ (Z k,τ Averaging the above inequality over all k ∈ {−τ, . . . , n} yields e 1:(n+1) . Ψτ (Z) ≤ 2 dTV Z, Z e ∈ S(n + τ + 1), taking the infimum over Z e yields Ψτ (Z) ≤ 2ρτ (Z), as Since this holds for every Z claimed. 24
5.5.2
Proof of Ineq. (6b)
We prove a stronger, two-sided version of this result. Define the switch coefficients between uniform mixtures of deleted data as Ψτ (Z) := dTV ∆0K,τ (Z), ∆1K,τ (Z) for K ∼ Unif({−τ, . . . , n}) drawn independently of everything else. We now prove the following two-sided bound for such K ∼ Unif({−τ, . . . , n}): (a) (b) 1 Ψτ (Z) ≤ ρτ (MK,τ (Z)) ≤ Ψτ (Z) + Ψ0,τ (Z). 2
(15)
Notice that Ψτ (Z) ≤ Ψτ (Z) due to the convexity of the TV distance (in the space of measures), so we recover Ineq. (6b) from Ineq. (15), part (b). e ∈ S(n + τ + 1) denote a cyclically exchangeable seProof of Ineq. (15), part (a): Let Z e a:b = (Z ea , . . . , Zen+τ +1 , Z e1 , . . . , Zeb ) if a > b. Drawing K ∼ quence. We will use the shorthand Z Unif({−τ, . . . , n}) independently of everything else, we have Ψτ (Z) = dTV ∆0K,τ (Z), ∆1K,τ (Z) e 1:(n+1) ) + dTV ∆0 (Z e 1:(n+1) ), ∆1 (Z e 1:(n+1) ) ≤ dTV ∆0K,τ (Z), ∆0K,τ (Z K,τ K,τ e 1:(n+1) ), ∆1 (Z) + dTV ∆1K,τ (Z K,τ (i) e 1:(n+1) + dTV Z e 1:(n+1) , Z e (n+2−K):(n−K−τ +1) + dTV Z e 1:(n+1) , Mn−K−τ +1,τ (Z) ≤ dTV MK,τ (Z), Z (ii) e 1:(n+1) + dTV Z e 1:(n+1) , Mn−K−τ +1,τ (Z) ≤ dTV MK,τ (Z), Z (iii) e 1:(n+1) . = 2dTV MK,τ (Z), Z Step (i) holds by the marginalization property of TV distance: because the deletion function ∆0 extracts subsets of the underlying sequences, the distance between these marginals is upper-bounded e is cyclically exchangeable, by the distance between the full sequences. Step (ii) holds because Z d
and step (iii) holds because MK,τ (Z) = Mn−K−τ +1,τ (Z) when K ∼ Unif({−τ, . . . , n}), where we e to conclude. recall our convention (7) for masking. We may take an infimum of the RHS over Z Proof of Ineq. (15), part (b): The sequence Z and M0,τ (Z) can be divided into three blocks as follows: Z = (Z1 , . . . , Zτ , Zτ +1 , . . . , Zn−τ +1 , Zn−τ +2 , . . . , Zn+1 ), | {z } | {z } | {z } τ
n+1−2τ
τ
and M0,τ (Z) = (⋆, . . . , ⋆, Zτ +1 , . . . , Zn−τ +1 , Zn−τ +2 , . . . , Zn+1 ). | {z } | {z } | {z } τ
n+1−2τ
τ
e from M0,τ (Z). First, sample We now construct a cyclically exchangeable sequence Z Zn+2 , . . . , Zn+τ +1 | M0,τ (Z) ∼ P(Zn−τ +2 ,...,Zn+1 )|(Z1 ,...,Zn−τ +1 ) ( · |Zτ +1 , . . . , Zn+1 ). 25
Then draw K ∼ Unif([n + τ + 1]) independently of Z and define e = (ZK , . . . , Zn+τ +1 , ⋆, . . . , ⋆, Zτ +1 , . . . , ZK−1 ). Z e ∈ S(n+τ +1). Moreover, the conditional distribution of Zn+2 , . . . , Zn+τ +1 By construction, we have Z given Zτ +1 , . . . , Zn+1 is the same as the conditional distribution of Zn−τ +2 , . . . , Zn+1 given Z1 , . . . , Zn−τ +1 . Thus, the laws of Z and Z(τ +1):(n+τ +1) only differ through the marginal laws of Zτ +1 , . . . , Zn+1 and Z1 , . . . , Zn−τ +1 , respectively. Consequently, by the data processing inequality, we have dTV Z, Z(τ +1):(n+τ +1) ≤ dTV ((Z1 , . . . , Zn−τ +1 ), (Zτ +1 , . . . , Zn+1 )) = Ψ0,τ (Z), where the equality follows from the definition of the switch coefficient. For K ∼ Unif([n + τ + 1]) drawn independently of Z, we have e 1:(n+1) , Mn+τ +2−K,τ (Z) dTV Z = dTV (ZK , . . . , Zn+τ +1 , ⋆, . . . , ⋆, Zτ +1 , . . . , ZK−τ −1 ), (Z1 , . . . , Zn+τ +2−K , ⋆, . . . , ⋆, Zn+2τ +3−K , . . . , Zn+1 ) = dTV (ZK , . . . , Zn+τ +1 , Zτ +1 , . . . , ZK−τ −1 ), (Z1 , . . . , Zn+τ +2−K , Zn+2τ +3−K , . . . , Zn+1 ) = dTV ∆1n+τ +2−K (Z(τ +1):(n+τ +1) ), ∆0n+τ +2−K (Z) (i) ≤ dTV ∆1n+τ +2−K (Z), ∆0n+τ +2−K (Z) + Ψ0,τ (Z) ≤ Ψτ (Z) + Ψ0,τ (Z), where step (i) holds because we replace Z(τ +1):(n+τ +1) with Z by paying an additive price of TV distance between the two sequences. But by our convention (7) for masking, for K ′ ∼ Unif({−τ, . . . , n}) drawn independently of d
everything else, we have Mn+τ +2−K,τ (Z) = MK ′ ,τ (Z). Thus ρτ MK ′ ,τ (Z) ≤ Ψτ (Z) + Ψ0,τ (Z),
(16)
as claimed.
6
Discussion
Motivated by the question of whether leave-one-out predictive inference methods remain reliable under temporal dependence, we showed that the vanilla jackknife can suffer severe coverage loss even in simple time-series models exhibiting mild dependence. As a remedy, we proposed the LWO method, which modifies the leave-one-out calibration mechanism of the jackknife by removing a window of training data points after each proxy test point. Under an out-of-sample stability condition, this modification yields provable approximate coverage while retaining the data efficiency of jackknife-type methods. Besides this methodological contribution, the main conceptual contribution of our work is a viewpoint based on approximately embedding a temporally dependent stochastic process into a cyclically exchangeable sequence. The cyclic embedding coefficient that we define is a distributional property of the stochastic process; our results connect this coefficient to classical dependence notions, including mixing and approximate Markovianity, and after a suitable masking operation, to switch coefficients. We expect that cyclic embedding will prove useful as a tool in studying other statistical problems involving temporal dependence. 26
Several open questions remain, and we elaborate on three of them to conclude. First, our validity guarantees for LWO rely on an out-of-sample stability condition. In the exchangeable setting, jackknife+ and CV+ avoid such stability assumptions by modifying the leave-one-out construction, at the cost of a slightly weaker coverage guarantee. It is natural to ask whether an analogous modification exists for the LWO method, which can achieve valid coverage without requiring stability assumptions. Second, it is interesting to explore if our tool of leaving a window out can be combined with cross-conformal or cross-validation style methods. Developing such versions of cross-conformal prediction, CV+, or jackknife+-after-bootstrap for dependent data would clarify how broadly the LWO principle extends beyond the jackknife in prediction problems. We note that similar principles have recently been used in estimating (random) functionals of multinomial distributions (Pananjady et al., 2024; Nakul et al., 2025). Third, the choice of the window size τ remains an important practical question: our theory suggests that τ should exceed the effective dependence range, but we do not provide a fully data-driven choice of τ with valid coverage guarantees and believe this would be an interesting future direction.
Acknowledgments HJ and YX were partially supported by National Science Foundation grants DMS-2134037 and CMMI-2112533, Emory Healthcare grant A1230749, and the Coca-Cola Foundation. RFB was partially supported by the National Science Foundation via grant DMS-2023109, and by the Office of Naval Research via grant N00014-24-1-2544. AP was supported in part by National Science Foundation grant CCF-2107455 and a Google Research Scholar award.
References Anastasios N Angelopoulos and Stephen Bates. Conformal prediction: A gentle introduction. Foundations and Trends in Machine Learning, 16(4):494–591, 2023. Anastasios N. Angelopoulos, Emmanuel J. Candès, and Ryan J. Tibshirani. Conformal PID control for time series prediction. In Advances in Neural Information Processing Systems, volume 36, pages 23047– 23074. Curran Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper_files/paper/ 2023/hash/47f2fad8c1111d07f83c91be7870f8db-Abstract-Conference.html. Anastasios N Angelopoulos, Rina Foygel Barber, and Stephen Bates. Theoretical foundations of conformal prediction. arXiv preprint arXiv:2411.11824, 2024. Rina Foygel Barber and Ashwin Pananjady. Predictive inference for time series: why is split conformal effective despite temporal dependence? In Matus Telgarsky and Jonathan Ullman, editors, Proceedings of The 37th International Conference on Algorithmic Learning Theory, volume 313 of Proceedings of Machine Learning Research, pages 1–24. PMLR, 23–26 Feb 2026. URL https://proceedings.mlr.press/v313/ barber26a.html. Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, and Ryan J Tibshirani. Predictive inference with the jackknife+. The Annals of Statistics, 49(1):486–507, 2021. Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, and Ryan J Tibshirani. Conformal prediction beyond exchangeability. The Annals of Statistics, 51(2):816–845, 2023. Olivier Bousquet and André Elisseeff. Stability and generalization. Journal of Machine Learning Research, 2:499–526, 2002. Victor Chernozhukov, Kaspar Wüthrich, and Zhu Yinchu. Exact and robust conformal inference methods for predictive machine learning with dependent data. In Conference On learning theory, pages 732–749. PMLR, 2018.
27
Paul Doukhan. Mixing: Properties and Examples, volume 85 of Lecture Notes in Statistics. Springer-Verlag, New York, 1994. ISBN 0-387-94214-9. doi: 10.1007/978-1-4612-2642-0. URL https://doi.org/10. 1007/978-1-4612-2642-0. Bradley Efron and Gail Gong. A leisurely look at the bootstrap, the jackknife, and cross-validation. The American Statistician, 37(1):36–48, 1983. Isaac Gibbs and Emmanuel Candes. Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems, 34:1660–1672, 2021. Isaac Gibbs and Emmanuel J Candès. Conformal inference for online prediction with arbitrary distribution shifts. Journal of Machine Learning Research, 25(162):1–36, 2024. Chirag Gupta, Arun K. Kuchibhotla, and Aaditya K. Ramdas. Nested conformal prediction and quantile out-of-bag ensemble methods. Pattern Recognition, 127:108496, 2022. doi: 10.1016/j.patcog.2021.108496. Tao Hong, Pierre Pinson, Shu Fan, Hamidreza Zareipour, Alberto Troccoli, and Rob J Hyndman. Probabilistic energy forecasting: Global energy forecasting competition 2014 and beyond. International Journal of forecasting, 32(3):896–913, 2016. Hanyang Jiang and Yao Xie. Spatial conformal inference through localized quantile regression. International Conference on Machine Learning (ICML), 2026. Byol Kim, Chen Xu, and Rina Foygel Barber. Predictive inference is free with the jackknife+-after-bootstrap. In Advances in Neural Information Processing Systems, volume 33, pages 4138–4149, 2020. URL https:// proceedings.neurips.cc/paper/2020/hash/2b346a0aa375a07f5a90a344a61416c4-Abstract.html. Jonghyeok Lee, Chen Xu, and Yao Xie. Kernel-based optimally weighted conformal time-series prediction. In The Thirteenth International Conference on Learning Representations, 2025. Jing Lei, Max G’Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. Distribution-free predictive inference for regression. Journal of the American Statistical Association, 113(523):1094–1111, 2018. Ruiting Liang and Rina Foygel Barber. Algorithmic stability implies training-conditional coverage for distribution-free prediction methods. The Annals of Statistics, 53(4):1457–1482, 2025. doi: 10.1214/ 25-AOS2510. URL https://arxiv.org/abs/2311.04295. Henrik Linusson, Ulf Norinder, Henrik Boström, Ulf Johansson, and Tuve Löfström. Efficient conformity calibration of random forests. Expert Systems with Applications, 154:113335, 2020. Mehryar Mohri and Afshin Rostamizadeh. Stability bounds for stationary φ-mixing and β-mixing processes. Journal of Machine Learning Research, 11(2), 2010. Milind Nakul, Vidya Muthukumar, and Ashwin Pananjady. Estimating stationary mass, frequency by frequency. In The Thirty Eighth Annual Conference on Learning Theory, pages 4359–4359. PMLR, 2025. Eugene Ndiaye. Stable conformal prediction sets. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 16462–16479. PMLR, 2022. URL https://proceedings.mlr.press/v162/ndiaye22a.html. Buddhika Nettasinghe, Samrat Chatterjee, Ramakrishna Tipireddy, and Mahantesh M. Halappanavar. Extending conformal prediction to hidden Markov models with exact validity via de Finetti’s theorem for Markov chains. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 25890–25903. PMLR, 2023. URL https://proceedings.mlr.press/v202/nettasinghe23a.html.
28
Roberto I. Oliveira, Paulo Orenstein, Thiago Ramos, and João Vitor Romano. Split conformal prediction and non-exchangeable data. Journal of Machine Learning Research, 25(225):1–38, 2024. URL https: //jmlr.org/papers/v25/23-1553.html. Ashwin Pananjady, Vidya Muthukumar, and Andrew Thangaraj. Just Wing It: Near-optimal estimation of missing mass in a Markovian sequence. Journal of Machine Learning Research, 25(312):1–43, 2024. Harris Papadopoulos, Kostas Proedrou, Vladimir Vovk, and Alex Gammerman. Inductive confidence machines for regression. In European Conference on Machine Learning, pages 345–356. Springer, 2002. Aleksandr Podkopaev and Aaditya Ramdas. Distribution-free uncertainty quantification for classification under label shift. In Uncertainty in artificial intelligence, pages 844–853. PMLR, 2021. Bhagavatula LS Prakasa Rao. Conditional independence, conditional mixing and conditional association. Annals of the Institute of Statistical Mathematics, 61(2):441–460, 2009. Drew Prinster, Samuel Don Stanton, Anqi Liu, and Suchi Saria. Conformal validity guarantees exist for any data distribution (and how to find them). In International Conference on Machine Learning, pages 41086–41118. PMLR, 2024. David Rolnick, Priya L Donti, Lynn H Kaack, Kelly Kochanski, Alexandre Lacoste, Kris Sankaran, Andrew Slavin Ross, Nikola Milojevic-Dupont, Natasha Jaques, Anna Waldman-Brown, et al. Tackling climate change with machine learning. ACM Computing Surveys (CSUR), 55(2):1–96, 2022. Omer Berat Sezer, Mehmet Ugur Gudelek, and Ahmet Murat Ozbayoglu. Financial time series forecasting with deep learning: A systematic literature review: 2005–2019. Applied soft computing, 90:106181, 2020. Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 9(3), 2008. Aldo Solari and Vera Djordjilović. Multi-split conformal prediction. Statistics and Computing, 32(2):1–13, 2022. Jake A Soloff, Rina Foygel Barber, and Rebecca Willett. Bagging provides assumption-free stability. Journal of Machine Learning Research, 25(131):1–35, 2024. Lukas Steinberger and Hannes Leeb. Conditional predictive inference for stable algorithms. The Annals of Statistics, 51(1):290–311, 2023. doi: 10.1214/22-AOS2250. Mervyn Stone. Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society: Series B (Methodological), 36(2):111–133, 1974. Sophia Huiwen Sun and Rose Yu. Copula conformal prediction for multi-step time series prediction. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/ forum?id=ojIJZDNIBj. Ryan J Tibshirani, Rina Foygel Barber, Emmanuel Candes, and Aaditya Ramdas. Conformal prediction under covariate shift. Advances in neural information processing systems, 32, 2019. John Tukey. Bias and confidence in not quite large samples. Ann. Math. Statist., 29:614, 1958. Vladimir Vovk. Cross-conformal predictors. Annals of Mathematics and Artificial Intelligence, 74(1-2):9–28, 2015. Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Springer, 2005.
Algorithmic learning in a random world.
Chen Xu and Yao Xie. Conformal prediction interval for dynamic time-series. In International Conference on Machine Learning, pages 11559–11569. PMLR, 2021.
29
Chen Xu and Yao Xie. Sequential predictive conformal inference for time series. In International Conference on Machine Learning (ICML), pages 38707–38727. PMLR, 2023. Chen Xu, Hanyang Jiang, and Yao Xie. Conformal prediction for multi-dimensional time series by ellipsoidal sets. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 55076–55099. PMLR, 2024. URL https://proceedings.mlr. press/v235/xu24m.html. Margaux Zaffran, Olivier Feron, Yannig Goude, Julie Josse, and Aymeric Dieuleveut. Adaptive conformal predictions for time series. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 25834–25866. PMLR, 2022. URL https: //proceedings.mlr.press/v162/zaffran22a.html. Frédéric Zheng and Alexandre Proutiere. Conformal predictions under Markovian data. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 61470–61497. PMLR, 2024. URL https://proceedings.mlr.press/v235/zheng24j. html.
Appendix A
Technical lemmas and their proofs
In this section, we collect our technical lemmas and their proofs.
A.1
Supporting lemma for proof of Proposition 2
Recall the stochastic process Z{k} from Eq. (12). The following lemma bounds the total variation {k} distance between Z1:(n+1) and Z for a range of k. Lemma 1. If 2τ + 2 ≤ k ≤ n − τ + 1, then we have {k} dTV Z, Z1:(n+1) ≤ 2β(τ ) + 2β ∗ (τ ). Proof. First define subsets of indices: A = {1, . . . , n − τ − k + 2}, B = {n − τ − k + 3, . . . , n − k + 2}, C = {n − k + 3, . . . , n − k + τ + 2}, D = {n − k + τ + 3, . . . , n − k + 2τ + 2}, E = {n − k + 2τ + 3, . . . , n + 1}. To sample Z = (ZA , ZB , ZC , ZD , ZE ), we can equivalently follow the hierarchical sampling strategy (ZB , ZC , ZD ) ∼ P(ZB ,ZC ,ZD ) , ZA ∼ PZA |(ZB ,ZC ,ZD ) · | ZB , ZC , ZD , ZE ∼ PZE |(ZA ,ZB ,ZC ,ZD ) · | ZA , ZB , ZC , ZD . By definition of conditional-β-mixing, this distribution is at most 2β ∗ (τ ) away in total variation distance from the joint distribution (ZB , ZC , ZD ) ∼ P(ZB,ZC ,ZD ) , ZA ∼ PZA |ZB · | ZB , ZE ∼ PZE |ZD · | ZD . 30
This distribution can be equivalently written as (ZB , ZD ) ∼ P(ZB ,ZD ) , Z ∼ P A ZA |ZB · | ZB , Z ∼ P · | Z , E D Z |Z E D Z ∼ P C ZC |(ZB ,ZD ) · | ZB , ZD . Changing only the first line, and applying the definition of β-mixing, this is at most β(τ ) away in total variation distance from the joint distribution (ZB , ZD ) ∼ PZB × PZD , Z ∼ P A ZA |ZB · | ZB , Z ∼ P · | Z , E D Z |Z E D Z ∼ P C ZC |(ZB ,ZD ) · | ZB , ZD . This can equivalently be written as ( (ZA , ZB , ZD , ZE ) ∼ P(ZA ,ZB ) × P(ZD ,ZE ) , ZC ∼ PZC |(ZB ,ZD ) · | ZB , ZD . By stationarity, this is equivalent to ( (ZD , ZE , ZA , ZB ) ∼ P(Z1 ,...,Zk−τ −1 ) × P(Zk ,...,Zn+1 ) , ZC ∼ PZC |(ZB ,ZD ) · | ZB , ZD . Again applying the definition of β-mixing, this is at most β(τ ) away in total variation distance from the joint distribution ( (ZD , ZE , ZA , ZB ) ∼ P(Z1 ,...,Zk−τ −1 ,Zk ,...,Zn+1 ) , ZC ∼ PZC |(ZB ,ZD ) · | ZB , ZD . But, by construction, this is exactly the distribution of {k}
{k}
{k}
{k}
{k}
{k}
{k}
(Z1 , . . . , Zn+1 ) = (ZA , ZB , ZC , ZD , ZE ). {k}
Taking stock, we have transformed Z into Z1:(n+1) up to a total variation loss of 2β(τ ) + 2β ∗ (τ ), as claimed.
A.2
Transferring OOS stability from test point to a random training point
Recall that for any k, we have Z(k) = Mk,τ (Z). Also recall that the LWO scores on Z and Z(K) are given by sk = score(Yk , Alg[Mk−1,τ +1 (Z1:n )](Xk )), (K) (K) (K) (K) sk = score(Yk , Alg[Mk−1,τ +1 (Z1:n )](Xk ))
with the convention that if k ∈ {−τ, . . . , 0}, then we have score(Yk , f (Xk )) = 0 and so both scores defined above evaluate to zero. We define the following notion of OOS stability evaluated at a random training point. 31
Definition 8. For a pair of scalars (ν, t), the pair (score, Alg) is said to be OOS stable when evaluated at a random training point with respect to leaving a random block of length τ + 1 out of Z = (Xi , Yi )n+1 i=1 if for K ∼ Unif({−τ, . . . , n}), we have n 1X (K) P sk − s k > t ≤ ν. n k=1
The following lemma shows that if OOS stability in the sense of Definition 2 holds (evaluated at the test point), then OOS stability in the sense of Definition 8 holds (evaluated at a random training point). Lemma 2. Suppose the pair (score, Alg) is OOS stable with respect to MK0 ,τ +1 (Z) with parameters (ν, t, τ ), where K0 ∼ Unif({−τ, . . . , n}). Then this pair is OOS (random training point) with respect to Z in the sense of Definition 8 with parameters n+τ +1 τ + Ψτ (Z), t . ν+ n+τ +1 n Proof. As shorthand, define the set I := {−τ, .. . , n}, and let K ∼ Unif(I) be chosen independently of Z. For each fixed k ∈ [n], define Lk := (K − k) mod (n + τ + 1) − τ. By the definition (K)
of the deletion operation, the event |sk − sk | > t is a measurable function of ∆0k,τ (Z) and the independent index K. By definition of total variation distance, viewing this event as a function of (∆1k,τ (Z), K) instead can inflate the probability by at most Ψk,τ (Z). Concretely, we have (K) P |sk − sk | > t ≤ P
score Yn+1 , Alg[Mn−k−τ,τ +1 (Z1:n )](Xn+1 ) (17)
! (L ) (L ) (Lk ) − score Yn+1k , Alg[Mn−k−τ,τ +1 (Z1:nk )](Xn+1 )
>t
+ Ψk,τ (Z).
Now split the first term on the RHS into two parts, and note that P(Lk ∈ {n + 1 − τ, . . . , n}) = (L ) τ /(n + τ + 1). Also note that if Lk ∈ / {n + 1 − τ, . . . , n}, then Yn+1k = Yn+1 . Therefore, ! P
score Yn+1 , Alg[Mn−k−τ,τ +1 (Z1:n )](Xn+1 )
≤P
(L ) (L ) (Lk ) − score Yn+1k , Alg[Mn−k−τ,τ +1 (Z1:nk )](Xn+1 )
score Yn+1 , Alg[Mn−k−τ,τ +1 (Z1:n )](Xn+1 )
>t
!
(L )
− score Yn+1 , Alg[Mn−k−τ,τ +1 (Z1:nk )](Xn+1 )
> t, Lk ∈ / {n + 1 − τ, . . . , n}
+
τ . n+τ +1 (18)
We now claim that n
1X P n
score Yn+1 , Alg[Mn−k−τ,τ +1 (Z1:n )](Xn+1 )
k=1
(19)
! (L ) − score Yn+1 , Alg[Mn−k−τ,τ +1 (Z1:nk )](Xn+1 ) 32
> t, Lk ∈ / {n + 1 − τ, . . . , n}
≤ ν.
We prove this claim shortly. Taking it as given for the moment, averaging Ineqs. (17) and (18) over k = 1, . . . , n, and then applying claim (19) yields n
n
k=1
k=1
τ 1X 1X (K) P |sk − sk | > t ≤ ν + + Ψk,τ (Z). n n+τ +1 n Finally, by the definition of the average switch coefficient, we have n
1X n+τ +1 Ψk,τ (Z) ≤ · Ψτ (Z). n n k=1
Therefore,
n
1X τ n+τ +1 (K) P |sk − sk | > t ≤ ν + + · Ψτ (Z), n n+τ +1 n k=1
as desired. It remains to prove claim (19). Proof of claim (19). Recall that Lk ∼ Unif(I) with I = {−τ }∪{1−τ, . . . , n−τ }∪{n+1−τ, . . . , n}. Let U ∼ Unif({1 − τ, . . . , n − τ }) be a random variable chosen independently of Z. Note that for each fixed k ∈ [n], we have P
score Yn+1 , Alg[Mn−k−τ,τ +1 (Z1:n )](Xn+1 ) ! (L ) − score Yn+1 , Alg[Mn−k−τ,τ +1 (Z1:nk )](Xn+1 ) > t, Lk ∈ / {n + 1 − τ, . . . , n} =
n P n+τ +1
score Yn+1 , Alg[Mn−k−τ,τ +1 (Z1:n )](Xn+1 )
!
(U ) − score Yn+1 , Alg[Mn−k−τ,τ +1 (Z1:n )](Xn+1 )
>t , (20)
= −τ , then Z(Lk ) = Z. We now rewrite the rescaled average of
where we have also used that if Lk the RHS above as a conditional probability over the masking index K0 ∼ Unif(I), as follows: n
1X P n
! score Yn+1 , Alg[Mn−k−τ,τ +1 (Z1:n )](Xn+1 )
(U ) − score Yn+1 , Alg[Mn−k−τ,τ +1 (Z1:n )](Xn+1 )
>t
k=1
=P
score Yn+1 , Alg[MK0 ,τ +1 (Z1:n )](Xn+1 )
! − score Yn+1 , Alg[MU,τ (MK0 ,τ +1 (Z1:n ))](Xn+1 ) > t K0 ∈ {−τ, . . . , n − τ − 1} =
n+τ +1 ·P n
score Yn+1 , Alg[MK0 ,τ +1 (Z1:n )](Xn+1 ) ! − score Yn+1 , Alg[MU,τ (MK0 ,τ +1 (Z1:n ))](Xn+1 ) > t, K0 ∈ {−τ, . . . , n − τ − 1} . (21) 33
But P
score Yn+1 , Alg[MK0 ,τ +1 (Z1:n )](Xn+1 )
! − score Yn+1 , Alg[MU,τ (MK0 ,τ +1 (Z1:n ))](Xn+1 ) > t, K0 ∈ {−τ, . . . , n − τ − 1} ≤P
score (MK0 ,τ +1 (Y))n+1 , Alg[(MK0 ,τ +1 (Z))1:n ]((MK0 ,τ +1 (X))n+1 ) ! (i) − score (MK0 ,τ +1 (Y))n+1 , Alg[MU,τ ((MK0 ,τ +1 (Z))1:n )]((MK0 ,τ +1 (X))n+1 ) > t ≤ ν,
where step (i) follows since by assumption, we have OOS stability (in the sense of Definition 2) with respect to MK0 ,τ +1 (Z) with parameters (ν, t, τ ). Putting together the pieces, we have n
1X P n
score Yn+1 , Alg[Mn−k−τ,τ +1 (Z1:n )](Xn+1 )
k=1
! (U )
− score Yn+1 , Alg[Mn−k−τ,τ +1 (Z1:n )](Xn+1 )
>t
(22) n+τ +1 ≤ · ν. n
Averaging Ineq. (20) over k ∈ [n] and using Ineq. (22) yields the claimed bound.
A.3
Ancillary results on quantile stability
Lemma 3. If A and B are two (unordered) sets of real numbers such that |A| = k, |B| = m and A ⊆ B, then for all a ∈ (0, 1), we have Quantile1−a (A) ≥ Quantile(1−a)· k (B) m
Quantile1−a (B) ≥ Quantile
1−a· m k
(A) .
Proof. Both bounds follow immediately from the definition of the quantile function. Lemma 4. Let U = (U1 , . . . , Uk ) and V = (V1 , . . . , Vk ) denote two vectors of real-valued random variables. If there exists a bijection π : [k] → [k] such that k
1X P |Ui − Vπ(i) | ≥ t ≤ ν, k i=1
then for any a ∈ [0, 1], we have n o √ P Quantile1−a (U) ≤ Quantile1−a+√ν (V) + t ≥ 1 − ν. Proof. We bound the probability of the complementary event. If Quantile1−a (U) > Quantile1−a+√ν (V) + t, 34
then for all permutations (i.e. bijections) σ : [k] → [k], we must have √ √ #{i : Ui > Vσ(i) + t} ≥ ⌈(1 − a + ν)k⌉ − (⌈(1 − a)k⌉ − 1) ≥ νk. Indeed, if there exists a permutation such that this is not true, then it is impossible for the quantile to move by more than t. Thus, for the particular permutation π, we must have ( k ) o n X √ P Quantile1−a (U) > Quantile1−a+√ν (V) + t ≤ P I Ui > Vπ(i) + t ≥ νk i=1 (i)
≤
Pk
i=1 P
|Ui − Vπ(i) | > t √ νk
(ii) √
≤
ν.
Step (i) is due to Markov’s inequality, and step (ii) follows from the assumption on π.
B
Additional experiments
In this section, we construct a one-dimensional stochastic process on which the vanilla jackknife drastically undercovers. Concretely, we consider the following epoch-based Markov chain. At each time t ≥ 1, let (Tt , Kt , Xt , Yt ) ∈ N × N × R × R, where Kt is latent while (Xt , Yt ) are observed. The chain evolves at time t + 1 as follows: • If Tt = Kt , a new epoch starts: Tt+1 = 1,
Kt+1 ∼ Geom(ρ),
Xt+1 ∼ N (0, 1),
Yt+1 ∼ N (Kt+1 , 1).
• If Tt < Kt , the current epoch continues: Tt+1 = Tt + 1,
Kt+1 = Kt ,
Xt+1 = Xt ,
Yt+1 ∼ N (Kt , 1).
Thus Xt is constant within each epoch, while Yt has mean equal to the latent epoch length Kt . Since new epoch values of Xt are drawn from a continuous distribution, two time points have the same raw covariate value almost surely only when they belong to the same epoch. We set ρ = 0.05. The predictor does not use the raw covariate Xt directly. Instead, the fitted algorithm is a composition of a data-dependent feature map and a chosen base predictor, both constructed from the training data. Specifically, given a training set D of covariate-response pairs, we first learn the feature map X ϕbD (x) := 1{Xi = x}. (Xi ,Yi )∈D
This map counts how many training observations share the same raw covariate value as x. We then transform each training point (Xi , Yi ) ∈ D into ϕbD (Xi ), Yi , and fit a base regression model gbD on these transformed data points. The final predictor fb trained on D is given by gbD ◦ ϕbD . In particular, its evaluation at any point x is given by fbD (x) = gbD ϕbD (x) . 35
Split CP
Jackknife
0.6 0.4
30 20 10
0.2 0.0 Ridge KNN
LWO
40
0.8
Radius
Empirical coverage
1.0
MLP
DT
0 Ridge KNN
KR
MLP
DT
KR
Figure 6. Empirical performance on the sticky Markov-chain process across five base predictors. (Left) Empirical coverage of Split CP, Jackknife, and LWO, with the dashed horizontal line marking the nominal 90% coverage level. (Right) Average prediction radius for the same methods and predictors. Bars show means across 500 repeated trials, and error bars indicate ±1 standard error.
We use no additional lagged history, i.e., L = 0. Note that the same learned feature map ϕbD is used to featurize any calibration or test point evaluated by this fitted model. Since both ϕbD and gbD depend on the training dataset D, they are computed differently in the different predictive inference methods. For the jackknife D can be any set of n − 1 data points. In split conformal, D is the proper training split. In an LWO fit, D is such that it excludes the local window after the proxy test point. Note that for this stochastic process, the epoch lengths are geometric with parameter ρ, the probability that dependence persists across a gap of length τ decays as (1 − ρ)τ . Consequently, the process is geometrically β-mixing, with mixing coefficients β(τ ) = O((1 − ρ)τ ). Evaluation Figure 6 shows that the sticky Markov chain is particularly challenging for leave-oneout methods. The vanilla jackknife produces much narrower intervals, but these intervals are overly optimistic and lead to severe undercoverage for most predictors. The failure mechanism comes from the feature construction. Recall that ϕbD (x) counts how many training observations have the same covariate value as x, which, in this process, is essentially the number of observed points from the same latent epoch. For interior proxy points used by the jackknife, the surrounding training data typically contain almost the entire epoch, both before and after the proxy point. Thus ϕbD (Xk ) is a good proxy for the latent epoch length Kk , making these jackknife proxy points relatively easy to predict and producing small calibration scores. The final test point is different. At the end of the observed sequence, the current epoch is right-censored: we only observe the part of the epoch that has occurred so far, and do not observe future points from the same epoch. Consequently, ϕbD (Xn+1 ) tends to underestimate the true epoch length. Since Yn+1 has mean tied to the latent epoch length, this makes the fitted predictor systematically less accurate at the actual test point than at the interior jackknife proxy points. This mismatch causes the jackknife calibration scores to be too small and leads to undercoverage. LWO mitigates this asymmetry by removing a local window after each proxy point. This censors the proxy point’s epoch in the same way the final test epoch is censored, preventing the fitted model from using nearby future observations that would not be available at test time. As a result, the LWO calibration scores reflect the difficulty of predicting the final test point.
36