Nonparametric Goodness-of-fit Testing under Covariate Shift

Nonparametric Goodness-of-fit Testing under Covariate Shift Zhen Hou and Dong Xia∗ Department of Mathematics Hong Kong University of Science and Technology

(August 6, 2026)

arXiv:2608.04860v1 [stat.ME] 5 Aug 2026

Abstract This paper develops procedures for nonparametric goodness-of-fit testing under covariate shift, where labelled data are drawn from a source population but goodness-of-fit is evaluated for a target population. The distribution mismatch is quantified by either a bounded moment condition or a sub-exponential tail condition on the target-to-source density ratio. Our method combines truncated importance-weighting kernel ridge regression with a multiplier bootstrap to construct confidence sets for the regression function. The truncation stabilizes the importanceweighting kernel ridge regression as well as the bootstrap calibration, making our approach applicable even when the density ratio has heavy tails. We prove nonasymptotic validity and sharpness of the resulting confidence sets under suitable operator compatibility conditions, and establish explicit error rates for coverage probability under specific conditions on the targetto-source density ratio and on the spectral decay of the kernel integral operator. Numerical experiments corroborate our theoretical findings.

Keywords: Covariate shift, kernel ridge regression, multiplier bootstrap, goodness-of-fit, importance weighting.

1

Introduction

Covariate shift arises when labelled data are collected from a source population but scientific conclusions are sought for a target population with a different covariate distribution. This problem has become increasingly prominent as machine learning methods trained on labelled source data are deployed in specialized target domains [Guan and Liu, 2022], where target-domain labels are often scarce or expensive [Wang et al., 2024]. For instance, when general-purpose language models are deployed in specialized biomedical and legal domains, the prompts, documents, and questions can differ drastically from the broad corpora used for pretraining or instruction tuning [Gururangan et al., 2020]. These applications raise important statistical questions, for example, which statistical methods perform well under target distributions and are robust to target-to-source mismatch? and is it possible to evaluate the performance of these methods under the target distribution even when they are trained under the source distribution? While the first question has been extensively studied in the literature, the second is far less understood. This paper provides an affirmative answer through a ∗

The research was partially supported by Hong Kong RGC Grant GRF 16303224.

1

general inference framework based on truncated importance-weighted kernel ridge regression (KRR) and the multiplier bootstrap [Bickel and Freedman, 1981, Efron and Tibshirani, 1994, Hall, 2013]. Let P and Q denote the source and target distributions, respectively, on the joint space X × R. For simplicity, we assume that both PX and QX admit known densities with respect to a common dominating measure, and that QX is absolutely continuous with respect to PX . Our methods and main theorems hold as long as the density ratio ρ(x) = (dQX /dPX )(x) is known. Moreover, ρ(x) can be estimated from unlabelled source and target covariates by direct importance-estimation or distribution-matching methods, such as those in Sugiyama et al. [2008], Huang et al. [2007]. See further discussion in Section 7. We assume that the marginal distributions of the covariates may differ, PX ̸= QX , whereas the conditional distributions of the response remain the same, PY |X = QY |X . With a slight abuse of notation, we shall write P and Q for the covariate marginal distributions PX and QX , respectively, whenever no ambiguity arises. Suppose that an i.i.d. sample D = {(Xi , Yi )}ni=1 ⊂ X × R drawn from the source distribution P is available, where the response and covariates satisfy the regression model f∗ (x) = E[Y |X = x], where f∗ (·) : X 7→ R is an unknown regression function. Our goal is to evaluate f∗ under the target covariate distribution. More specifically, given a candidate function f0 , we aim to test H0 : ∥f0 − f∗ ∥L2 (Q) = 0 v.s. H1 : ∥f0 − f∗ ∥L2 (Q) > 0,

(1.1)

R where the L2 (Q)-norm is defined by ∥g∥2L2 (Q) := g(x)2 dQ(x). Equivalently, the null hypothesis
states that QX {x : f∗ (x) = f0 (x)} = 1. For ease of exposition, we focus on the simple test, but our framework can be easily generalized to composite tests and two-sample tests. See numerical experiments in Section 6.3 and the Appendix. The seminal work Shimodaira [2000] proposed importance weighting (IW) for estimating the target function f∗ (x) under covariate shift in parametric likelihood models, showing that the asymptotically optimal weighting function is the target-to-source density ratio ρ(x). Non-asymptotic analyses of importance weighting have also attracted considerable attention across diverse fields and applications, including the generalization bounds for statistical learning with function classes of finite pseudo-dimension [Cortes et al., 2010], local convergence rates for learning Lipschitz functions under local doubling conditions [Schmidt-Hieber and Zamolodtchikov, 2024], minimax-optimal rates for learning Hölder-continuous functions under covariate shift [Pathak et al., 2022], and kernel-based regression and classification [Ma et al., 2023, Wang, 2026]. Importance weighting is generally necessary for unbiased estimation of the target risk under covariate shift, but it may be unnecessary for identifying the Bayes-optimal predictor when the conditional distribution is invariant and the model is correctly specified. See, for example, Gogolashvili et al. [2023] and the references therein. However, importance weighting can degrade finite-sample performance when the density ratio is unbounded or highly variable. A common remedy is to truncate the density ratio, replacing ρ(x) by wn (x) := min{ρ(x), τn } where τn > 0 is a pre-specified truncation level. Ma et al. [2023] showed that the importance weighted kernel ridge regression (IWKRR) estimator achieves the minimax optimal convergence rate under covariate shift using a truncated importance weight for variance reduction. Figure 1 illustrates the stabilizing effect of truncation when the density ratio is large in target-relevant regions. Indeed, a small number of source observations may receive extremely large importance weights, causing the IWKRR estimator to be highly variable. Capping the weights can therefore substantially stabilize the L2 (Q) error. 2

IWKRR vs. Truncated IWKRR: High Importance Ratio Case Mean

L 2 error

8 6 4 2 0

KRR L 2(Q) med=2.157, IQR=3.279 out=6

IWKRR L 2(Q) med=0.978, IQR=2.936 out=27

(a) L2 (Q)-norm error rates: IWKRR with truncation.

Truncated IWKRR L 2(Q) med=0.706, IQR=1.176 out=23

KRR, IWKRR, and

(b) IWKRR L2 (Q)-norm error distributions, without and with truncation.

Figure 1: truncating large density-ratio weights ρ(x) := (dQ/dP)(x) stabilizes the finite-sample performance of the importance-weighting kernel ridge regression (IWKRR). Source and target distributions are truncated normal TN(−1, 0.25; [−2, 2]) and TN(1, 0.25; [−2, 2]), respectively. Panel (a) summarizes the validation-tuned errors of KRR, IWKRR, and truncated IWKRR. The true regression function is f∗ (x) = |x| . Panel (b) shows the Monte Carlo densities of the L2 (Q) errors over 200 repetitions, using raw importance weights which differ only in truncation. The aforementioned works primarily focus on efficient and robust estimation under covariate shift. While they provide practical guidance and theoretical justification for statistical methods that improve predictive performance under target distributions, it remains largely unclear how to evaluate the performance of these methods under the target distribution and how to quantify the uncertainty of the resulting estimators. For instance, in studies of conditional average treatment effects, one may wish not only to estimate the treatment-effect function but also to test whether treatment effects are constant over the target population [Crump et al., 2008, Ding et al., 2016]. These inferential conclusions depend critically on the target and source distribution mismatch: heterogeneity that is negligible under the source distribution may be substantial under the target distribution if the regions in which the effect varies have substantially different probability densities under the two distributions. In this paper, we develop statistical procedures to quantify the uncertainty of the truncated IWKRR estimator and to evaluate its performance under the target covariate distribution. More specifically, we construct confidence sets for f∗ with respect to the L2 (Q)-norm distance based on the truncated IWKRR estimator, establish non-asymptotic bounds on the absolute coverage error of these confidence sets, and study the goodness-of-fit testing problem in (1.1). The main challenge is that the IWKRR estimator, the inferential loss, and the key technical tools naturally live in different geometries. The IWKRR estimator is governed by the truncated weighted-source risk, whereas coverage of the confidence sets is evaluated in the target L2 (Q) metric. On the technical side, the covariance operator of the stochastic fluctuation must be analyzed in the target L2 (Q) geometry, whereas its algebraic structure is most transparent through RKHS operators. We address this mismatch using unitary isomorphisms generated by the square roots of the target and weightedsource integral operators. These maps realize the relevant L2 geometries inside the RKHS while preserving Hilbert-space norms and inner products, thereby allowing the Gaussian and bootstrap 3

approximations to be compared within a common RKHS operator calculus. This operator-theoretic treatment captures two effects of covariate shift: the bias of the truncated IWKRR estimator and its covariance structure relevant for coverage guarantees of the proposed confidence sets. Our contributions are as follows. First, based on the truncated IWKRR estimator, trained on finite-sample data from source distribution, and multiplier bootstrap, we propose an inference procedure to construct confidence sets of the unknown regression function in the target distribution. The procedure is computationally efficient where computing each bootstrap statistics mainly involves matrix and vector multiplications. Second, we provide a general theoretical framework establishing the coverage guarantee of the bootstrap confidence sets, where the error derives from the bias, Gaussian approximation, and bootstrap approximation. The target-to-source distribution mismatch, kernel integrals, and truncation level jointly determine the validity of multiplier bootstrap method. We identify the regularized transfer coverage condition ensuring that the source risk is transferable to the target risk and the non-degenerate covariance condition that prevents the Gaussian covariance from degenerating in the target geometry. Third, we establish explicit error rates for the coverage probability under distinct density ratio and kernel spectral conditions, covering the cases of sub-exponential type density ratio and polynomially decaying kernel eigenvalues. Finally, our inference methods yield interesting findings in real data experiments.

1.1

Related work

Covariate shift. The seminal work Shimodaira [2000] initiated this field by analyzing weighted likelihood methods for parametric predictive inference under covariate shift and identifying importance weighting as the asymptotically optimal weighting scheme. Finite-sample generalization bounds for importance weighting were developed by Cortes et al. [2008], covering kernel-based algorithms and function classes with finite pseudo-dimension. The unknown density ratio, as well as its variants, can be estimated using unlabelled samples from source and target populations such as the Kullback– Leibler importance estimation [Sugiyama et al., 2008] and kernel mean matching [Huang et al., 2007]. The γ-transfer exponent was introduced by Kpotufe and Martinet [2018] as a measure of source and target similarity. A systematic review of machine learning methods under covariate shift is provided by Sugiyama and Kawanabe [2012]. More recently, Zamolodtchikov [2026] established the minimax rates for nonparametric regression with both source and target samples, characterized through the notion of a transfer function. Model selection and cross validation under covariate shift have also gained considerable attention. An information criterion was derived in Shimodaira [2000] for model selection under covariate shift. Sugiyama et al. [2007] showed that the ordinary cross-validation can be biased under covariate shift and proposed importance-weighted cross-validation. The instability of importance-weighted cross-validation and its robust variants were studied by Kouw et al. [2019]. Nonparametric inference. Chernozhukov et al. [2014b, 2017, 2014a] established nonasymptotic Gaussian and bootstrap approximations for suprema of empirical processes, with applications to confidence bands. Zaitsev [1987] and Buzun et al. [2022] proved Gaussian approximation results for sums of independent random vectors under bounded and sub-Gaussian conditions, respectively. Yang et al. [2017] studied posterior coverage and sup-norm credible sets in Gaussian process regression. Singh and Vijaykumar [2023] developed inference procedures for KRR without covariate shift. Similarly, Shang et al. [2025] studied data integration based on KRR and constructed adaptive confidence intervals for f∗ (x) via local and global Bootstrap procedures. Constructing uniformly 4

valid pointwise confidence intervals is generally more challenging than testing a global null such as H0 : ∥f∗ − f0 ∥L2 (P) = 0. The former requires uniform control of the estimation error, typically in sup norm, and is subject to well-known impossibility and lower-bound phenomena in nonparametric models. See, e.g., Low [1997], Genovese and Wasserman [2008], Giné and Nickl [2010], Cai and Low [2006], Juditsky and Lambert-Lacroix [2003], Li [1989], Robins and van der Vaart [2006], Spokoiny [1996] on the methods and challenges of constructing honest and adaptive non-parametric confidence sets. Predictive intervals. A related but distinct line of work studies predictive intervals and sets under covariate shift. Tibshirani et al. [2019] developed weighted conformal prediction when the likelihood ratio between the target and source covariate distributions is known or can be accurately estimated, and Barber et al. [2023] further studied conformal prediction beyond exchangeability. These methods provide prediction sets for a future response and are largely distribution-free. By contrast, our goal is function-level inference: we construct a confidence ball for the regression function f∗ , with a radius calibrated in L2 (Q)-norm, and therefore must control the bias, covariance structure, and bootstrap approximation of the IWKRR estimator with truncated weights.

1.2

Notations

Throughout, H denotes the RKHS associated with a kernel function k(·, ·), L2 (Q) denotes the L2 space with respect to the target covariate distribution QX , and we write kx = k(x, ·). For a Hilbert space G, let ⟨·, ·⟩G and ∥ · ∥G denote its inner product and norm, respectively; for a, b ∈ G, a ⊗G b denotes the rank-one operator (a ⊗G b)c = ⟨b, c⟩G a for c ∈ G. For a bounded linear operator A : G → S, ∥A∥G→S denotes its operator norm and ∥A∥HS,G→S its Hilbert-Schmidt norm, when finite; if A : G → G is trace class, trG (A) denotes its trace; for self-adjoint operators A, B on G, we write A ⪯G B if B−A is positive semidefinite. When the subscript is omitted, inner products, norms, tensor products, operator norms, Hilbert-Schmidt norms, traces, and Loewner order are understood in H. For a compact positive self-adjoint operator A on G, (µk (A, G), ϕk (A, G)) denotes its kth eigenpair, with eigenvalues arranged in nonincreasing order and counted with multiplicity; we use the shorthand (µk , ϕk ) := (µk (LQ , L2 (Q)), ϕk (LQ , L2 (Q))), define the spectral tail σG2 (m, A) := P∞ 2 j=m+1 µj (A, G) for m ≥ 0, and also write σ (m, A, G) to specify the ambient space. For a positive trace-class operator C on G, NG (0, C) denotes the centered Gaussian element in G with covariance operator C. Finally, we write a ≲ b if a ≤ Cb for a universal constant C > 0, and a ≍ b if both P a ≲ b and b ≲ a hold; the empirical average is denoted by En [g] = EPn [g] = n−1 ni=1 gi .

2

Problem Formulation and Preliminaries

2.1

Confidence balls in target distribution

A pair of random variables (X, Y ) ∈ X × R satisfies Y = f∗ (X) + ε,

(2.1)

where X and Y denote the covariate and response, respectively, and the regression function f∗ is unknown. The random error ε carries no systematic information in the sense that E[ε|X] = 0.

5

Let (X, Y ), {(Xi , Yi )}ni=1 ∈ X × R be i.i.d. observations, where the covariate X is drawn from a source distribution P on X. We aim to construct a confidence set for f∗ in the L2 (Q) geometry. Given an L2 (Q)-norm consistent estimator fb, a natural confidence set is the L2 (Q)-norm ball centered at fb, with a radius ĉα,n calibrated using the quantiles of ∥fb − f∗ ∥L2 (Q) . More precisely, the inferential goal is to construct the 100(1 − α)% confidence set n o CSα (fb) = f : ∥fb − f ∥L2 (Q) ≤ ĉα,n , (2.2)
with asymptotically valid coverage probability; that is, P f∗ ∈ CSα (fb) → 1 − α as n → ∞. We suppress the dependence of fb and CSα (fb) on the sample size n for simplicity. Once a valid confidence set is available, testing the hypothesis in (1.1) is straightforward. For example, one can reject the null hypothesis at significance level α if f0 ∈ / CSα (fb). We emphasize that the choice of metric is part of the inferential goal, rather than merely a technical convention. A confidence set contains precisely those candidate functions that are close to the point estimator in the chosen geometry. Consequently, different metrics may lead to different confidence sets. There are three natural metrics in KRR: the reproducing kernel Hilbert space (RKHS) norm, the L2 (P)-norm induced by the source distribution, and the L2 (Q)-norm induced by the target distribution. The RKHS norm measures discrepancy in the smoothness geometry imposed by the kernel, whereas the L2 (P)- and L2 (Q)-norms weight pointwise errors according to the source and target distributions, respectively. Under covariate shift, regions that are important under Q may have little mass under P. Hence, a function can be close to f∗ in L2 (P) but not in L2 (Q).

2.2

Reproducing kernel Hilbert space and integral operators

Let H be a reproducing kernel Hilbert space (RKHS, Schölkopf and Smola [2002]) over X with positive definite kernel k(·, ·) : X × X 7→ R. Throughout, we assume that X is compact. More precisely,  H is the Hilbert space of functions on X obtained as the completion of the linear span of kx := P k(x, ·)}x∈X . For any {xi }pi=1 , {x̃j }qj=1 ⊂ X , consider functions of the form f (·) = pi=1 ai k(xi , ·) Pq Pp Pq and g(·) = j=1 bj k(x̃j , ·). Their inner product in H is given by ⟨f, g⟩H = i=1 j=1 ai bj k(xi , x̃j ). For any f ∈ H, the reproducing property holds: f (x) = ⟨f, k(x, ·)⟩H , for ∀x ∈ X. Throughout the paper, we assume that the kernel k(·, ·) is bounded, namely, supx∈X k(x, x) ≤ κ2 for some constant κ > 1. This immediately implies that point evaluation is bounded by the RKHS norm. Indeed, for any f ∈ H, we have supx |f (x)| = supx ⟨f, k(x, ·)⟩H ≤ ∥f ∥H · supx ∥k(x, ·)∥H ≤ κ∥f ∥H . The kernel integral operator is central to the analysis of kernel methods, as its spectral structure encodes both the geometry of the data distribution and the effective complexity of the induced feature space. Given the probability measure Q, define the integral operator LQ : L2 (Q) → L2 (Q) by Z LQ f =

k(x, ·)f (x)dQ(x),

∀f ∈ L2 (Q).

X

Note that the integral operator LQ is closely related to the RKHS covariance operator ΣQ = EQ [kX ⊗H kX ] : H → H, where, for g, h ∈ H, the rank-one operator g ⊗H h : H → H is defined by (g ⊗H h)f = ⟨h, f ⟩H g, ∀f ∈ H. R By the reproducing property, we have ΣQ f = X k(x, ·)f (x)dQ(x), ∀f ∈ H. Under the boundedkernel assumption, both LQ and ΣQ are bounded, positive, self-adjoint, and trace-class operators. 6

Since these two operators are given by the same integral expression but act on different spaces, we abuse notation and write LQ for both whenever the underlying domain is clear from context. The spectral decomposition of LQ is often used to characterize the kernel k(·, ·), the distribution Q, and the associated RKHS H. Lemma 2.1. [Mercer [1909]] Suppose that X is compact, k(·, ·) is continuous, symmetric, and positive definite, Q is a Borel probability measure on X, and supp(Q) = X. Then there exist nonnegative eigenvalues {µj }j≥1 and corresponding eigenfunctions {ϕj }j≥1 ⊂ L2 (Q), with ⟨ϕi , ϕj ⟩L2 (Q) = δij and P k(x, y) = ∞ j=1 µj ϕj (x)ϕj (y), x, y ∈ X. The convergence is absolute and uniform on X × X. Mercer’s theorem immediately implies several useful identities. For µi , µj > 0, ⟨ϕi , ϕj ⟩H =  1/2 P ∞ −1 µi δij . Thus, µj ϕj : µj > 0 j=1 is an orthonormal basis of H. In particular, if f = ∞ j=1 aj ϕj , P −1 2 2 then ∥f ∥H = j:µj >0 aj µj . The fractional powers of LQ are defined via the spectral theorem. For r > 0, define the operator LrQ : L2 (Q) → L2 (Q) by LrQ f =

X

µrj ⟨f, ϕj ⟩L2 (Q) ϕj ,

f ∈ L2 (Q).

j≥1 1/2

Although LQ is defined as an operator on L2 (Q), its range admits a natural RKHS interpreP P 2 −2 < ∞ . Indeed, if f ∈ Ran(L ), tation. Define Ran(LQ ) = Q j:µj >0 aj µj j:µj >0 aj ϕj : P 2 1/2 1/2 2 2 then LQ f H = j:µj >0 ⟨f, ϕj ⟩L2 (Q) = ∥f ∥L2 (Q) . Hence LQ can be viewed as an isometry from Ran(LQ ) to H. 1/2

Lemma 2.2. The operator LQ : Ran(LQ ) → H is a unitary isomorphism onto H. In particular, 1/2

1/2

it preserves inner products: ⟨f, g⟩L2 (Q) = ⟨LQ f, LQ g⟩H , for f, g ∈ Ran(LQ ). We can similarly define the integral operator and covariance operator by the probability measure P on X, denoted both by LP for notational simplicity. These notations can be generalized to the sampling distribution Pn , defined from the i.i.d. sample X1 , · · · , Xn ∼ P. For instance, we write the integral operator n 1X LPn f = f (Xi )kXi , f ∈ L2 (P), n i=1 P which may also represent the sample covariance operator f 7→ n−1 ni=1 (kXi ⊗H kXi )f, ∀f ∈ H. Let w : X → (0, ∞) be a weight function. Define the weighted measure Pw on X by Pw (A) = R A w(x) dP(x), A ⊆ X. For instance, if w is the target-to-source density ratio ρ(x) = (dQ/dP)(x), then Pw = Q, assuming that Q ≪ P. The measure Pw and its empirical counterpart induce the following integral operators: n

Z LP w f :=

kx f (x)w(x) dP(x), X

2.3

and LPnw f :=

1X w(Xi )kXi f (Xi ). n i=1

Importance weighted kernel ridge regression

Given the i.i.d. sample D = {(Xi , Yi )}ni=1 ⊂ X × R, the kernel ridge regression (KRR) seeks
2 P a function f ∈ H that minimizes the empirical risk n−1 ni=1 f (Xi ) − Yi with an RKHS-norm 7

penalization λn ∥f ∥2H , where λn > 0 is the regularization parameter controlling the trade-off between fitting the data and model complexity. The regularization prevents KRR from overfitting, especially when H is dense in the space of continuous functions. The error rates and optimality of KRR are well understood. See Wahba [1990], Caponnetto and De Vito [2007], Vapnik [2013] and references therein. Importance weighting (IW, Sugiyama et al. [2007]) is a technique for unbiased estimation of target risk under covariate shift. In particular, the importance-weighted kernel ridge regression (IWKRR) estimator is defined by 1 fb := arg min n f ∈H

n X

wn (Xi ){Yi − f (Xi )}2 + λn ∥f ∥2H ,

(2.3)

i=1

where wn : X → (0, ∞) is a weight function. If equipped with a proper weight function, IW transports the source risk to the target risk. For instance, using the density ratio weight wn (x) = ρ(x), we get EP [wn (X)(Y − f (X))2 ] = EQ [(Y − f (X))2 ]. This density-ratio weighting thus corrects the population mismatch at the level of the squared-error risk. For notational simplicity, we may write λ and w(·) and suppress their dependence on n. However, the IWKRR estimator can have large finite-sample variances if the target-to-source density ratio is unbounded or has heavy tails, in which case a few observations carry excessively large weights and destabilize the estimator. A common remedy is to use the truncated density ratio wn (x) := min{ρ(x), τn }, where τn > 0 is the truncation level. The truncation introduces a controlled bias but reduces variance. Under suitable density ratio and spectral conditions, with appropriate choices of regularization parameter λn and truncation parameter τn , the truncated IWKRR attains the minimax optimal rate for the target L2 (Q)-norm risk. See Ma et al. [2023] and Section 4 for detailed discussions. In this paper, we aim to construct the confidence ball of f∗ with respect to the L2 (Q)-norm distance based on the truncated IWKRR estimator (2.3). We begin with presenting the explicit matrix form of the IWKRR estimator. Denote the n × n matrix K with its (i, j)-th entry k(Xi , Xj ),
the weight matrix W = diag wn (X1 ), · · · , wn (Xn ) , and y = (Y1 , · · · , Yn )⊤ . Then, the IWKRR estimator can be compactly written as −1 ⊤ −1 −1 fb = k⊤ (2.4) x (WK + nλn I) Wy = kx (K + nλn W ) y,
⊤ where the vector function kx = k(X1 , ·), · · · , k(Xn , ·) and the second equation holds provided that W is nonsingular. The IWKRR estimator is biased from two sources: the RKHS-norm regularization and the truncated density-ratio weight. The population IWKRR estimator is   (2.5) fP w = arg min EP wn (X)(f (X) − Y )2 + λn ∥f ∥2H = (LP w + λn I)−1 LP w f∗ . f ∈H

If the weight wn (x) = (dQ/dP)(x) without truncation, the population IWKRR estimator reduces to   fQ = arg min EQ (f (X) − Y )2 + λn ∥f ∥2H = (LQ + λn I)−1 LQ f∗ , (2.6) f ∈H

which minimizes the regularized risk in target distribution. The explicit formulas in (2.5) and (2.6) follow from the first-order optimality conditions for the two population objectives. Finally, we re
mark that the IWKRR estimator (2.4) can be explicitly written as fb = (LPnw +λn I)−1 n−1 k⊤ x Wy . 8

3

Inference Procedure

3.1

Bias-variance decomposition and Bahadur representation

We begin with the bias-variance decomposition of fb and derive its Bahadur representation. Notice √ √ √ that n(fb− f∗ ) = n(fP w − f∗ ) + n(fb− fP w ), where the first and second terms capture the bias and variance, respectively. Moreover, the bias term can be explicitly written as √ √ n(fP w − f∗ ) = − nλn (LP w + λn I)−1 f∗ , √ implying that the bias term can be estimated by − nλn (LPnw + λn I)−1 fb. However, we will show in Section 4 that the bias term is asymptotically negligible if λn is small, i.e., under-smoothing √ IWKRR. Our bootstrap inference procedure therefore focuses on the variance term n(fb − fP w ). Nevertheless, we also provide a bias-corrected inference procedure and numerical experiments in Appendix B. √ The second term, n(fb − fP w ), captures the stochastic fluctuation of the IWKRR estimator around its population counterpart. We show in Section 4 that its leading term, in L2 (Q)-norm, admits the Bahadur representation: √

n

1 X n(fb − fP w ) ≈L2 (Q) √ Ui , n

(3.1)

i=1

where Ui := (LP w + λn I)−1 wn (Xi )εi kXi + wn (Xi )(kXi ⊗H kXi ) − LP w (f∗ − fP w ) . It suggests √ that n(fb−fP w ) can be approximated by a centered Gaussian process in L2 (Q) with the covariance operator V = E[Ui ⊗L2 (Q) Ui ], and Gaussian multiplier bootstrap can be used to calibrate the radius of L2 (Q)-norm confidence balls for f∗ .

3.2

Multiplier bootstrap

The random element Ui is not directly applicable because it relies on the unknown f∗ and the population IWKRR estimator fP w . Nevertheless, the Bahadur representation (3.1) motivates us to consider the oracle bootstrap function, defined by the anti-symmetric multiplier process: Bora :=

1 n

X 1≤i̸=j≤n

hij ·

Ui − U j Vi − V j 1 X √ = hij · √ , n 2 2 1≤i̸=j≤n

i.i.d.

where hij ∼ N (0, 1) are independent of the data D and
Vi := Ui + (LP w + λn I)−1 LP w (f∗ − fP w ) = wn (Xi ) Yi − fP w (Xi ) (LP w + λn I)−1 kXi . Conditioned on D, the oracle bootstrap function Bora is a centered Gaussian process in L2 (Q) with bora = EP [Ui ⊗L2 (Q) Ui ] − EP [Ui ] ⊗L2 (Q) EP [Ui ], which is the sample version of covariance operator V n n n the covariance operator V = E[Ui ⊗L2 (Q) Ui ] defined after (3.1). The conditional covariance operator bora = EP [Vi ⊗L2 (Q) Vi ] − EP [Vi ] ⊗L2 (Q) EP [Vi ]. of Bora can be equivalently written as V n n n The oracle bootstrap function Bora remains unavailable in practice, due to the unknown function fP w . By plugging in the empirical counterparts, we define
Vbi := wn (Xi ) Yi − fb(Xi ) (LPnw + λn I)−1 kXi = wn (Xi )b εi (LPnw + λn I)−1 kXi , 9

which leads to the computable multiplier bootstrap function: B :=

1 n

X 1≤i̸=j≤n

hij ·

√ Vbi − Vbj −1 −1 √ = n k⊤ ε)z, x (K + nλn W ) diag(b 2

(3.2)

with h := (hij )n×n and z := (2n)−1/2 (h − h⊤ )1n ∼ N (0, In − 11⊤ /n). Similarly, conditioned on b := EP [Vbi ⊗L2 (Q) Vbi ] − data, B is a centered Gaussian process in L2 (Q) with covariance operator V n EPn [Vbi ] ⊗L2 (Q) EPn [Vbi ]. Fast computation of L2 (Q)-norm. We use bootstrap percentiles of n−1/2 ∥B∥L2 (Q) to calibrate the radius of confidence balls. It is computationally expensive to evaluate these norms by definition. Fortunately, B is a linear combination of functions kXi ’s, which permits fast computation of ∥B∥2L2 (Q) . √ −1 −1 Indeed, given the sampled vector z, we have B = n k⊤ ε)z. x a with a = (K + nλn W ) diag(b 2 ⊤ 2 ⊤ Then, ∥B∥L2 (Q) = n∥kx a∥L2 (Q) = na KQ a, where KQ is an n × n matrix with its (i, j)-th entry being (KQ )ij = ⟨kXi , kXj ⟩L2 (Q) . Since the matrix KQ is fixed throughout bootstrap repetitions, computing ∥B∥2L2 (Q) amounts to evaluating a quadratic form. Algorithm 1 Multiplier bootstrap for L2 (Q)-norm confidence balls  n Require: Data D = (Xi , Yi ) i=1 , kernel k(·, ·), density ratio ρ(x) = q(x)/p(x), weight wn (x) = min{ρ(x), τn }, confidence level 1 − α, bootstrap repetitions M , regularization parameter λn ;
1: Form the matrix K = k(Xi , Xj ) n×n , the weight matrix W = diag(wn (X1 ), . . . , wn (Xn )), the
⊤ vector y = (Y1 , . . . , Yn )⊤ , and the vector function kx = k(X1 , ·), · · · , k(Xn , ·) ; −1 −1 2: Compute the IWKRR estimator fb = k⊤ x (K + nλn W ) y;
⊤ 3: Compute the residual vector b ε = Y1 − fb(X1 ), . . . , Yn − fb(Xn ) ; 4: Form the n × n matrix KQ with its (i, j)-th entry being (KQ )ij = ⟨kXi , kXj ⟩L2 (Q) ; 5: for b = 1, . . . , M do 6: Sample z(b) ∼ N (0, In − 11⊤ /n). 7: Set the coefficient vector a(b) = (K + nλn W−1 )−1 diag(b ε)z(b) .
(b) (b) ⊤ (b) 1/2 . 8: Compute the bootstrap statistic T (b) = ∥k⊤ x a ∥L2 (Q) = (a ) KQ a 9: end for 10: Let ĉα,n be the empirical (1 − α)%-percentile of {T (b) }M b=1 .  11: Define the confidence set CSα (fb) = f ∈ H : ∥fb − f ∥L2 (Q) ≤ ĉα,n . Ensure: the IWKRR fb and the radius ĉα,n . The multiplier bootstrap procedure is detailed in Algorithm 1. Let fb and ĉα,n be the IWKRR estimator and radius output by Algorithm 1. The resulting 100(1 − α)% confidence ball is n o CSα (fb) = f ∈ H : ∥fb − f ∥L2 (Q) ≤ ĉα,n . (3.3) The computational cost of (IW)KRR grows fast when the sample size n increases. Fortunately, the multiplier bootstrap Algorithm 1 requires solving IWKRR only once, and the subsequent bootstrap repetitions involve just matrix-vector multiplications. We remark that the residual bootstrap [Hall, 2013] is also applicable for constructing the L2 (Q)-norm confidence balls. See Appendix C for details.

10

4

Theoretical Guarantees

This section establishes the asymptotic validity of the bootstrap confidence balls (3.3). The proof mainly involves four components. First, we introduce key covariate-shift compatibility conditions connecting the target L2 (Q) geometry with the regularized truncated weighted-source geometry L2 (Pw ). Second, we provide a Bahadur representation of the IWKRR estimator fb and use Gaussian coupling to approximate the stochastic fluctuation of fb by a Gaussian element in L2 (Q). Third, a multiplier bootstrap function is developed to couple with the same Gaussian element. Finally, we establish the coverage-error bounds for the bootstrap confidence balls using Gaussian approximation and anti-concentration.

4.1

Assumptions and compatibility conditions

4.1.1

Statistical and spectral assumptions

We first collect the statistical and spectral assumptions needed for Bahadur representation and Gaussian approximation under covariate shift and truncated importance weighting. Assumption 4.1 (uniformly bounded noise). There exist constants 0 < σ ≤ σ < ∞ such that |ε| ≤ σ

and

E[ε2 | X] ≥ σ 2

almost surely.

For technical convenience, we assume the random error is uniformly bounded. It is possible to generalize the main theorems to sub-Gaussian noise, but we leave it for future work. The major challenge is to control the bias and variance induced by truncated importance weighting. Assumption 4.2 (source condition). There exist s ∈ [0, 1], τ0 > 0, and g∗ ∈ H such that    f∗ = LsP w0 (g∗ ), where LP w0 (g) = EP w0 (X)g(X)kX with w0 (X) = min ρ(X), τ0 . The source condition is typical for KRR that characterizes the smoothness of f∗ relative to the kernel integral operator. Assumption 4.2 differs slightly from the standard ones in KRR literature due to the truncated importance weighting. See, e.g., Caponnetto and De Vito [2007], Ma et al. [2023]. The larger smoothness parameter s ensures sharper control on the L2 (Q)-norm bias and the RKHS-norm quantities appearing in the bootstrap approximation. Note that we assume the truncation level used in Algorithm 1 satisfies τn ≥ τ0 so that LP w0 ⪯H LP w . Assumption 4.3 (effective dimension). There exists d ∈ [0, 1] such that n
1/2 o
Ed := max 1, supλ∈(0,1] Nλ λd < ∞, where Nλ := trH (LQ + λI)−1 LQ . Assumption 4.3 always holds with d = 1 under the bounded-kernel condition, since trH (LQ ) = 2 µ i i ≤ κ by Lemma 2.1. If d ∈ (0, 1), Assumption 4.3 holds if the eigenvalues of LQ decrease fast enough, for instance, µi = O(i−1/d ); see Caponnetto and De Vito [2007] for more discussions. P

11

4.1.2

Compatibility conditions under covariate shift

The covariate shift and truncated importance weighting require compatibility conditions on the integral operators defined on the measure Q and Pw . The two compatibility conditions play distinct roles. The first compatibility condition ensures the regularized transfer coverage so that the source-side risk and RKHS-norm regularization control the target-side risk. The second compatibility condition guarantees a non-degenerate covariance in the target distribution, which essentially prevents the Gaussian covariance from degenerating in target geometry and is generally necessary for anti-concentration properties. Assumption 4.4 (regularized transfer coverage). There exists a constant C1 ≥ 1, independent of 1/2 n, such that LQ (LP w + λn I)−1/2 H→H ≤ C1 , for the chosen τn and λn . Equivalently, Assumption 4.4 requires the regularized operator domination: LQ ⪯H C21 (LP w +
λn I), or, in quadratic-form notation, ∥f ∥2L2 (Q) ≤ C21 EP [wn (X)f (X)2 ] + λn ∥f ∥2H for every f ∈ H. Moreover, by Lemma 2.2, the operator norm in Assumption 4.4 admits the equivalent variational representation 2 1/2 LQ (LP w + λn I)−1/2 H→H =

∥f ∥2L2 (Q)

sup 2 . 2 f ∈H, f ̸=0 EP [wn (X)f (X) ] + λn ∥f ∥H

Therefore, Assumption 4.4 essentially states that the weighted source distribution Pw covers the target distribution Q in all kernel directions, up to the regularization level λn . If the constant C1 is large, there exist directions that have small regularized size in the weighted source distribution but large size in the target distribution, which may lead to unstable transfer from source distribution to the target geometry. The constant C1 measures the residual mismatch after importance weighting, truncation, and regularization; it is therefore determined jointly by the distribution shift, the kernel geometry, and the choices of τn and λn , rather than by the density ratio alone. Remark 1 (sufficient conditions of Assumption 4.4). The following three typical density-ratio conditions are sufficient to ensure Assumption 4.4. Recall the truncated weight function wn (x) = min{ρ(x), τn }. (i) Bounded density ratio. Suppose that ρ(X) ≤ ρ P-almost surely. If τn ≤ ρ, then ρ(x) ≤ (ρ̄/τn )wn (x). Therefore, LQ ⪯H (ρ̄/τn )LP w ⪯H (ρ̄/τn )(LP w + λI), and Assumption 4.4 holds with C1 ≤ (ρ̄/τn )1/2 . If τn > ρ, then wn (x) = ρ(x). Then, LP w = LQ , and Assumption 4.4 holds  with C1 = 1. Combining the two regimes gives C1 ≤ max 1, (ρ̄/τn )1/2 . (ii) Sub-exponential density ratio. The bounded density ratio condition is stringent. Assumption 4.4 still holds if the density ratio has light tails. Definition 4.1 (Bernstein-type moment condition). There exist γ ≥ 1, W > 0, and σ > 0 such  
1/γ that, for every integer m ≥ 2, EQ ρ(X)γ(m−1) ≤ (m!/2)W m−2 σ 2 . The density ratio may be unbounded, but has light tails under Definition 4.1. Nevertheless, √ Lemma F.29 in the Appendix verifies Assumption 4.4 with C1 ≤ 2, if the truncation level   1 (1+d)γ m −1 − τn ≥ 22γ−1 κ2γ Ed2γ m0 !W m0 −2 σ 2 0 λ m0 −1 , where m0 ≥ 2 is some integer and Ed is the effective dimension defined in Assumption 4.3. 12

(iii) Heavy-tailed density ratio. Even if the density ratio has heavy tails, Assumption 4.4 still holds if the truncation level is properly chosen. Definition 4.2 (Bounded θ-th moment). For a θ ≥ 2, there exists some Ω < ∞ such that   EP ρ(X)θ ≤ Ω. If θ = 2, then χ2 (Q ∥ P) ≤ Ω − 1. √ (θ−1) Lemma F.31 in the Appendix verifies Assumption 4.4 with C1 ≤ 2 for θ > 2, if τn ≥ 2 2 −(1+d) 4κ ΩEd λ , where Ed is the effective dimension defined in Assumption 4.3. Note that similar comparison inequality between integral operators already appeared in the literature to quantify the information transferred from the source distribution to the target task. In particular, Wang [2026] proposes an effective sample size defined by n o n neff := sup t ≤ n : LQ ⪯H LP + λ̃I , t with a prescribed regularization level λ̃. Equivalently, n/neff is the smallest c ≥ 1 such that LQ ⪯H c(LP + λ̃I). Suppose that λ̃ = λn , the truncation wn (X) ≥ ρ for some ρ ∈ (0, 1] P-almost surely, and τn ≥ 1. Since ρ(LP + λn I) ⪯H LP w + λn I ⪯H τn (LP + λn I), our Assumption 4.4 holds with C1 satisfying n/(neff τn ) ≤ C21 ≤ n/(neff ρ). The effective sample size neff measures sourceto-target compatibility in the regularized unweighted-source geometry, whereas C1 measures the remaining mismatch after incorporating truncated importance weighting. Remark 2 (other distribution shift measures). There exist other measures of source–target mismatch serving different purposes, which do not directly supply the domination inequality in Assumption 4.4. Hypothesis-class discrepancies compare distributions through the largest difference of expected losses over a chosen class; for example, the discrepancy distance of Mansour et al. [2009] is used to establish generalization bounds for domain adaptation. While the loss-reliant discrepancies provide convenient risk transfer from source to target distributions, they cannot reflect the impact of the truncated weights or the regularization level. Distribution-matching measures such as kernel mean matching [Huang et al., 2007] choose weights to match the kernel mean embeddings in target and weighted source distributions, i.e., controlling the first-order differences of the form EQ f (X) − EP [wn (X)f (X)]. Local nonparametric measures, including the transfer exponent of Kpotufe and Martinet [2018] and the ball-probability ratio measure of Pathak et al. [2022], compare how source and target mass scale on metric balls and are designed to characterize local transfer rates for Lipschitz or Hölder function classes. In contrast, Assumption 4.4 goes beyond the source and target distributions: it is a kernel-, truncation, and regularization-dependent domination condition for the specific RKHS geometry in which the IWKRR estimator resides. In addition, we require a non-degenerate covariance condition for valid inference, expressed as compatibility between integral operators. This lower bound condition on the variance is typical in high-dimensional statistical inference where estimators are usually biased. See, e.g., Assumption 4 in Xia and Yuan [2021]. This non-degenerate covariance condition is imposed to establish anticoncentration of the norm of Gaussian element in L2 (Q). Assumption 4.5 (non-degenerate covariance operator). There exists a constant ρ > 0 such that ρ(X) ≥ ρ P-almost surely and τn ≥ ρ. Moreover, there exists a constant C2 ≥ 1 such that (LP w + −1 −1 for the chosen τ , λ . λn I)−1 LP w (LP w + λn I)−1 ⪰H C−2 n n 2 (LQ + λn I) LQ (LQ + λn I) 13

The uniform lower bound on the density ratio is operationally less restrictive than a uniform upper bound, especially when source data are abundant. For instance, source observations in the low-ratio region Aδ = {x : ρ(x) < δ} may be screened out entirely for a small δ > 0. This   region has target probability QX (Aδ ) = EPX ρ(X)1{X ∈ Aδ } ≤ δ. Alternatively, we can randomly retain source observations in this region with probability aδ (x) := min{1, ρ(x)/δ}. Denote by (δ) PX the covariate distribution of the retained source sample. The resulting density ratio satisfies
(δ)
(δ) dQX /dPX (x) = Zδ ρ(x)/aδ (x) ≥ Zδ δ, PX -almost surely, where Zδ := EPX [aδ (X)]. Thus, partial thinning of source-rich yet target-irrelevant regions can enforce a uniform lower bound while retaining some observations from those regions. By contrast, a uniform upper bound on ρ(x) rules out target-important regions that are poorly represented by the source distribution, which is a more consequential restriction for target-domain inference. Assumption 4.5 lower bounds the regularized covariance operator (LP w + λn I)−1 LP w (LP w + λn I)−1 by (LQ + λn I)−1 LQ (LQ + λn I)−1 , which prevents the Gaussian approximation from becoming degenerate in the target geometry and thereby supports anti-concentration. In contrast, the regularized transfer coverage condition in Assumption 4.4 enables transfer of estimation and bootstrap errors to the target norm. Neither condition implies the other in general.

4.2

General coverage guarantee of bootstrap confidence balls

√ √ √ Recall the bias-variance decomposition of IWKRR estimator n(fb−f∗ ) = n(fb−fP w )+ n(fP w − f∗ ). The bootstrap confidence balls constructed by Algorithm 1 are valid if the quantiles of ∥B∥2L2 (Q) , conditioned on D, approximate the quantiles of n∥fb− f∗ ∥2 2 . Broadly speaking, we need to study L (Q)

three approximations to establish the connection between these two quantiles: - Bias control of IWKRR. There exists ∆bias > 0 such that the deterministic bias satisfies √

n∥fP w − f∗ ∥L2 (Q) ≤ ∆bias .

(4.1)

- Gaussian approximation of IWKRR. For any η ∈ (0, 1), there exist ∆G > 0 and Z ∼ NL2 (Q) (0, V) such that  √  P n(fb − fP w ) − Z L2 (Q) ≤ ∆G ≥ 1 − η. (4.2) - Bootstrap approximation. For any η ∈ (0, 1), there exist ∆B > 0 and Z ′ ∼ NL2 (Q) (0, V), conditionally coupled with B, such that, with probability at least 1 − η over the data D,
P ∥B − Z ′ ∥L2 (Q) ≤ ∆B D ≥ 1 − η.

(4.3)

The above approximation error terms ∆G , ∆B may depend on the probability tolerance η. The total approximation error is ∆ := ∆G + ∆B + ∆bias , which plays the primary role in the coverage guarantee of bootstrap confidence balls. For ease of exposition, we first present the coverage guarantees assuming that the inequalities (4.1)-(4.3) hold. Explicit bounds for ∆G , ∆B , and ∆bias will be established in Section 4.3 for specific source-target and kernel spectral conditions. Note that if the quantiles of ∥fb − f∗ ∥2L2 (Q) is discontinuous, establishing the L2 (Q)-norm confidence balls may be infeasible. To rule out these ill-posed cases, we require that the distribution of

14

∥fb − f∗ ∥2L2 (Q) does not concentrate too sharply near its quantiles. Due to the Gaussian approximation (4.2), it suffices to show that the density function of ∥Z∥L2 (Q) is uniformly bounded. Recall the covariate operator V defined in Section 3.1. Lemma 4.1. Suppose Assumptions 4.1 - 4.5 hold, τn ≥ τ0 , 0 < λn ≤ 1, and LQ has either polynomial-decay eigenvalues µi ≍ i−β for some β > 1 or exponential-decay eigenvalues µi ≍ exp(−βiγ ) for some β, γ > 0. Let Z ∼ NL2 (Q) (0, V) with V = E[Ui ⊗L2 (Q) Ui ]. Then the density of 1/2

∥Z∥L2 (Q) is uniformly upper bounded by C1 τn , where the constant C1 > 0 only depend on the noise and source-target distributions, kernel spectral condition, and compatibility constants. The general coverage guarantee of bootstrap confidence balls is presented in the following theorem. Theorem 4.1 (general coverage guarantee). Let α ∈ (0, 1) be a miscoverage level. Suppose the conditions of Lemma 4.1 hold and inequalities (4.1) - (4.3) hold with a probability tolerance satisfying 0 < η < min{α, 1 − α}. Denote by t̂α the bootstrap quantile such that P (∥B∥L2 (Q) ≥ t̂α | D) = α, and set ĉα,n = t̂α /n1/2 . The miscoverage of the confidence ball CSα (fb) defined in (3.3) satisfies
√ P f∗ ∈ / CSα (fb) − α ≲ η + τn ∆. √ Therefore, if ηn → 0 and τn {∆G,n + ∆B,n + ∆bias,n } → 0 as n → ∞, then the coverage probability of the bootstrap confidence balls converges to the nominal level.

4.3

Finite-sample bounds for coverage probability

We establish explicit bounds for the terms ∆G , ∆B , and ∆bias , which characterize the coverage guarantee of confidence balls. 4.3.1

Bias control of IWKRR

Proposition 4.1. Suppose that Assumptions 4.2 and 4.4 hold with τn ≥ τ0 . Then, √ √ n∥fP w − f∗ ∥L2 (Q) ≤ ∆bias := max{κ2s−1 , 1} C1 n λmin{s+1/2,1} ∥g∗ ∥H . n Thus the bias bound inequality (4.1) holds with above choice of ∆bias , which diminishes asympmin{2s+1,2} totically if nλn → 0 as n → ∞. 4.3.2

Gaussian approximation of IWKRR

To establish the inequality (4.2), we must: (i) bound the approximation error of Bahadur representation (3.1); (ii) bound the Gaussian coupling of the normalized sum of the influence elements. The Gaussian coupling proof uses techniques similar to those in Singh and Vijaykumar [2023], adapted to the target L2 (Q) geometry and the covariance operator induced by truncated importance weighting. Proposition 4.2 (Bahadur representation). Suppose Assumptions 4.1, 4.2, and 4.4 hold with τn ≥  τ0 , and τn → ∞, λn → 0 as n → ∞. If n > max (τn κ2 + 1) log(e + Nλn )/(C2 λn ), τn /(λn Nλn ) , where C2 is a constant depending on log(1/η), then with probability at least 1 − η, s   n X
√ 2 1 N log(e + N ) λ λ n n 3 2 log n(fb − fP w ) − √ Ui ≲ ∆G,Bah := C1 τn σ κ 1 + ∥g∗ ∥H λn n η n L2 (Q) i=1

15

where Ui = (LP w + λn I)−1 wn (Xi )εi kXi + (wn (Xi )kXi ⊗ kXi − LP w )(f∗ − fP w ) . Moreover, −1/2

∥Ui ∥L2 (Q) ≤ Ū := τn C1 λn

(κ2 ∥f∗ ∥H + κσ̄).

We note that the upper bound Ū for ∥Ui ∥L2 (Q) will appear frequently in the subsequent theorems. The Bahadur representation error ∆G,Bah diminishes asymptotically if nλn ≫ τn2 Nλn log(e + Nλn ). P 2 2 Recall the spectral tail σL22 (Q) (m, V) = ∞ j=m+1 µj (V, L (Q)), where µj (V, L (Q)) denotes the j-th eigenvalue of V in the L2 (Q) space. The spectral tail characterizes the smallest error of approximating V by a finite-rank operator. Proposition 4.3 (Gaussian coupling in target L2 (Q)). There exists Z ∼ NL2 (Q) (0, V) such that, with probability at least 1 − η, n

1 X √ Ui − Z ≲ ∆G,coup := inf m≥1 n L2 (Q)

 log

i=1

1/2

m2 Ū log(m2 /η) √ (6/η)σL2 (Q) (m, V) + n



The Gaussian coupling error ∆G,coup is determined by the decay rate of the eigenvalues of V in L2 (Q). By Propositions 4.2 and 4.3, the Gaussian approximation inequality (4.2) holds with ∆G = ∆G,Bah + ∆G,coup . 4.3.3

Bootstrap approximation

To establish (4.3), we must: (i) bound the difference between the oracle bootstrap function Bora and the computable bootstrap function B in L2 (Q)-norm; (ii) bound the Gaussian coupling of Bora . The regularized transfer coverage Assumption 4.4 is required in the first step because both the residual and the empirical regularized operator must be transferred from the weighted-source geometry to target L2 (Q) geometry. Proposition 4.4 (difference between oracle and computable bootstrap functions). Suppose Assumptions 4.1, 4.2, and 4.4 hold with τn ≥ τ0 , and τn → ∞, λn → 0 as n → ∞. If n >  max (τn κ2 + 1) log(e + Nλn )/(C2 λn ), τn /(λn Nλn ) , where C2 is a constant depending on log(1/η), then there exists a data event E with P(E) ≥ 1 − η such that, conditioned on E and data D, with probability at least 1 − η,
τn Nλ ∥B − Bora ∥L2 (Q) ≲ ∆B,feas := C1 σ κ2s+4 1 + ∥g∗ ∥H √ n · log3 (2/η), nλn where B=

1 n

X

hij ·

1≤i̸=j≤n

Vbi − Vbj √ 2

and

Bora =

1 n

X 1≤i̸=j≤n

hij ·

Vi − V j √ , 2

with Vi = wn (Xi ) Yi −fP w (Xi ) (LP w +λn I)−1 kXi and Vbi = wn (Xi ) Yi − fb(Xi ) (LPnw +λn I)−1 kXi . bora ) and B | D ∼ NL2 (Q) (0, V), b where V bora = EP [Vi ⊗L2 (Q) Vi ] − Moreover, Bora | D ∼ NL2 (Q) (0, V n b b b b b EPn [Vi ] ⊗L2 (Q) EPn [Vi ] and V = EPn [Vi ⊗L2 (Q) Vi ] − EPn [Vi ] ⊗L2 (Q) EPn [Vi ], respectively.

The error ∆B,feas diminishes asymptotically if nλn ≫ τn2 Nλ2n . Denote the trace trL2 (Q) (V) = P∞ 2 j=1 µj (V, L (Q)).

σL22 (Q) (0, V) =

16

Proposition 4.5 (bootstrap coupling in L2 (Q)). Let g be the isonormal Gaussian process in L2 (Q). d. b 1/2 Then there exists a random element G = V ora g D such that, conditioned on D, with total probability at least 1 − 6η,   2   Ū trL2 (Q) (V) Ū 4 1/4 1/4 ≲ ∆B,coup := inf m V g−G 2 + σL2 (Q) (m, V) log3/2 (2/η), + 2 m≥1 n n L (Q)   where the covariance operator V = E Ui ⊗L2 (Q) Ui . 1/2

Putting together Propositions 4.4 and 4.5, we can couple the Bootstrap function B with a Gaussian element in L2 (Q), conditioned on data D, which is formalized in the following corollary. Corollary 4.1 (bootstrap approximation in L2 (Q)). Suppose the conditions of Propositions 4.4 d.

and 4.5 hold. There exist a universal constant c > 0 and a random element Z ′ = NL2 (Q) (0, V) D, conditionally coupled with B, such that with probability at least 1 − η over the data,
P ∥B − Z ′ ∥L2 (Q) ≤ ∆B,feas + ∆B,coup D ≥ 1 − η, where ∆B,feas and ∆B,coup denote the bounds in Propositions 4.4 and 4.5, respectively, evaluated with tolerance ηtol = cη 2 . By Corollary 4.1, the bootstrap approximation inequality (4.3) holds with ∆B = ∆B,feas + ∆B,coup , where ∆B,feas and ∆B,coup are evaluated at tolerance cη 2 . 4.3.4

Examples

We now provide explicit error rates for the coverage probability of bootstrap confidence sets. By Propositions 4.1-4.5 and Corollary 4.1, the key roles in the error rates are played by: (i) target-source distribution mismatch; (ii) the spectral decay rate of the covariance operator V, or equivalently, of the integral operator LQ . For ease of exposition, we ignore the constants C1 , C2 , ρ, and assume that Assumptions 4.1 - 4.3 hold with s ≥ 1/2 and τn ≥ τ0 . Similarly, the quantities γ, W, σ, ρ̄, and Ω, appearing in Remark 1, will also be treated as constants. The probability parameter is fixed at η ≍ n−1 so that log(1/η) ≍ log n. Recall that µk denotes the k-th eigenvalue of LQ . Corollary 4.2 (bounded density ratio). Suppose that ρ(X) ≤ ρ P-almost surely, and let τn = C0 for a sufficiently large fixed constant C0 ≥ max{ρ, τ0 }. 2(β+1)

− 3(β+3)

(1) if µk ≍ k −β for some β > 5 and all k ≥ 1, by choosing the regularization parameter λn ≍ n we get (β−5)(β+10)
− P f∗ ∈ / CSα (fb) − α ≲ n 6(β+3)(2β+5) log3 n;

,

(2) if µk ≍ exp(−ck ζ ) for some c, ζ > 0 and all k ≥ 1, by choosing the regularization parameter λn ≍ n−3/5 , we get
3 +1 P f∗ ∈ / CSα (fb) − α ≲ n−1/10 (log n) 2 2ζ ; Corollary 4.3 (sub-exponential tail). Suppose that the density ratio satisfies the sub-exponential tail condition in Definition 4.1. 17

β−5

(1) if µk ≍ k −β for some β > 5 and all k ≥ 1, by choosing the truncation level τn ≍ n 9(β+3)(2β+5) and 2(β+1)

− 3(β+3)

the regularization parameter λn ≍ n

, we get

(β−5)(β+9)
− P f∗ ∈ / CSα (fb) − α ≲ n 6(β+3)(2β+5) log3 n;

(2) if µk ≍ exp(−ck ζ ) for some c, ζ > 0 and all k ≥ 1, by choosing the truncation level τn ≍ n1/150 and the regularization parameter λn ≍ n−3/5 , we get
9 P f∗ ∈ / CSα (fb) − α ≲ n− 100 log3 n; If the density ratio ρ(X) has heavy tails in P, our methods can still provide valid inference as long as the eigenvalues of LQ decay sufficiently fast and the density ratio has a bounded higher moment. For ease of exposition, we only present the theoretical guarantees for the case when LQ has exponentially decaying eigenvalues. If LQ has polynomial-decay eigenvalues, the explicit error rate for coverage probability is complicated (see Appendix D for more details). Corollary 4.4 (bounded θ-th moment). Suppose that EP [ρ(X)θ ] ≤ Ωθ < ∞ for some θ > 7 as in Definition 4.2. If µk ≍ exp(−ck ζ ) for some c, ζ > 0 and all k ≥ 1, then, by choosing 3(θ−1)

λ ≍ n− 5θ−2 we obtain

4.4

and

3

1

τn ≍ n 5θ−2 (log n) ζ(θ−1) ,

3 5
− θ−7 +1+ P f∗ ∈ / CSα (fb) − α ≲ n 2(5θ−2) (log n) 2 2ζ 4ζ(θ−1) .

Hypothesis tests and power analysis

For a prescribed function f0 , we reject the null hypothesis in (1.1) whenever f0 ∈ / CSα (fb), or equivalently whenever ∥fb− f0 ∥L2 (Q) > ĉα,n . The coverage guarantees in Sections 4.2-4.3 control the size of the inverted test under the null hypothesis. We now provide the power guarantee under fixed alternatives. The key observation is that, under H1 , the target-domain separation δ := ∥f∗ − f0 ∥L2 (Q) is strictly positive. Therefore, the inverted test is powerful whenever the estimation error and the scaled critical value are both negligible compared with this fixed separation. Theorem 4.2 (power guarantee). Suppose H1 holds with a fixed separation δ = ∥f∗ − f0 ∥L2 (Q) > 0 P in the target domain. If ∥fb − f∗ ∥L2 (Q) + ĉα,n − → 0 as n → ∞, then
P ∥fb − f0 ∥L2 (Q) > ĉα,n → 1,

5

Numerical Experiments

5.1

Simulation design

as n → ∞.

We numerically examine the finite-sample properties of our proposed method from three perspectives: (i) the calibration of bootstrap radius; (ii) the empirical coverage of the resulting confidence ball; (iii) the power of the induced test under fixed alternatives. The source covariates 18

are generated from N (0, 4), while the target distribution is N (1, 1). The true regression function is f∗ (x) = exp(−0.5x2 ), and the independent noise εi ∼ N (0, 0.09). We use the Gaussian radial kernel with the deterministic choices λn = n−0.6 and τn = n0.06 to examine the performance of bootstrap radius calibration and coverage guarantee. For power evaluation, we compare three validation-tuned kernel functions with a fixed length scale ℓ = 1. These kernels are √ √ kG (x, x′ ) = exp(−r2 /2), k3/2 (x, x′ ) = (1 + 3r) exp(− 3r), k1/2 (x, x′ ) = exp(−r), with r = |x − x′ |, where the latter two are the Matérn kernels with smoothness parameters ν = 3/2 and ν = 1/2, respectively. Note that k1/2 (·, ·) is also known as the Laplacian kernel. All three kernels satisfy k(x, x) = 1. The target-domain norm ∥ · ∥L2 (Q) is approximated by Monte Carlo integration using 1200 target samples. The IWKRR estimator uses normalized importance weights and truncated weights wn (x) = min{ρ(x), τn }. Tuning the regularization and truncation parameters. For each b = 1, · · · , 20, we independently draw a source-labelled tuning sample Dbtune and a source-labelled validation sample Dbval , each of size 300. The parameters are chosen from λ ∈ 300−0.6 {1/128, 1/64, 1/32, 1/16, 1/8, 1/4, 1/2, 1, 2, 4},

τ ∈ {1, 1.25, 1.5, 1.75, 2, 2.5}.

For each pair (λ, τ ), the candidate model is fitted on Dbtune , whereas its target prediction risk is estimated on Dbval using the self-normalized, untruncated importance-weighted criterion P300 bval (λ, τ ) = R b,k

val 2 val val b i=1 ρ(Xb,i ){Yb,i − fb,k,λ,τ (Xb,i )} . P300 val i=1 ρ(Xb,i )

bb,k , τbb,k ) is held fixed If numerical ties occur, we prefer larger λ and smaller τ . The selected pair (λ across repetitions.

5.2

Bootstrap calibration

We use 1000 independent draws to simulate the bootstrap distribution. Figure 2 compares the conditional bootstrap distribution with the empirical distribution of the true scaled target error. The true distribution is estimated using Monte Carlo simulation from 250 independent training samples, each with a sample size n = 300. The bootstrap distribution is computed conditional on one representative fitted data set. Table 1: Bootstrap estimates of upper quantiles at α ∈ {0.10, 0.05} and n = 300. α

Pooled sampling quantile

Pooled bootstrap-mixture quantile

Relative error

0.10 0.05

1.194 1.379

1.196 1.331

0.002 -0.035

Table 1 reports the retained 250-draw calibration summary, whereas the smoother display in Figure 2 uses 1000 conditional bootstrap draws. The table’s bootstrap quantiles are close to the empirical true quantiles at both levels, with absolute relative errors between two and three percent. 19

1.6

Monte Carlo sampling distribution Conditional bootstrap distribution

1.4 1.2

90% quantiles Monte Carlo: 1.268 Bootstrap: 1.248

Density

1.0 0.8 0.6 0.4 0.2 0.0 0.00

0.25

0.50

0.75 1.00 1.25 Scaled L 2(Q) error

1.50

1.75

2.00

Figure 2: Bootstrap distribution of the L2 (Q) error. The figure compares the true sampling distri√ bution of n∥fb − f∗ ∥L2 (Q) with the conditional bootstrap distribution of ∥B∥L2 (Q) .

5.3

Empirical coverage and size

We evaluate the empirical coverage of the feasible confidence ball CSα (fb) with the critical value ĉα,n estimated from 250 multiplier-bootstrap repetitions. Under the null hypothesis f0 = f∗ , we record the relative frequency of rejecting the null hypothesis using the inverted test 1{∥fb−f0 ∥L2 (Q) > ĉα,n }. For each sample size, the rejection probability is estimated from 60 independent repetitions. Table 2 shows finite-sample over-rejection and under-coverage of the constructed confidence balls. This indicates that the bootstrap radius is well calibrated at the level of marginal quantiles in Table 1, but the coverage performance remains sensitive to finite-sample variability and to the random seed when using only a moderate number of repetitions . Table 2: Empirical size of the inverted test and coverage of the confidence ball under H0 : f0 = f∗ .

n

α

Empirical size

Empirical coverage

Cluster SE

95% MC interval

200 200 300 300

0.10 0.05 0.10 0.05

0.148 0.092 0.112 0.061

0.852 0.908 0.888 0.939

0.011 0.009 0.013 0.007

[0.126, 0.170] [0.073, 0.111] [0.086, 0.138] [0.045, 0.077]

5.4

Power performance

We evaluate the empirical power of the induced test under fixed alternatives. Let f0,δ (x) = f∗ (x) + δ h(x) with h(x) = g(x)∥g∥−1 , where g(x) := kG (x, 1)−kG (x, 0) = exp{−(x−1)2 /2}−exp(−x2 /2) L2 (Q) and ∥g∥L2 (Q) = 3−1/4 (1 − e−1/3 )1/2 . Since g is a difference of Gaussian kernel sections and is a 20

Schwartz function, it belongs to each of the three RKHSs considered. Similarly, f0,δ belongs to all three RKHSs and ∥f0,δ − f∗ ∥L2 (Q) = δ. The significance level is fixed at α = 0.10 with n = 300. The critical value ĉα,n is estimated using 300 bootstrap draws. For each separation δ and kernel function, the empirical rejection-probability is estimated from 100 independent test repetitions, whose empirical variance is evaluated via 20 independent experiments. The results are displayed in Figure 3. The rejection probabilities increase with the target-domain separation for all three scenarios, reaching approximately one at δ = 0.12. At δ = 0, the average rejection probabilities for the Gaussian, Matérn ν = 3/2, and Matérn ν = 1/2 scenarios are 0.138, 0.121, and 0.149, respectively. These values are empirical sizes, which indicate finite-sample over-rejection relative to the nominal level.

Estimated rejection probability

Mean

Nominal level

Matérn ν = 3/2

Gaussian (RBF)

Matérn ν = 1/2 (Laplacian)

1.0 0.8 0.6 0.4 0.2 0.0 0.00

0.03

0.06

0.09

0.12

0.00

0.03

0.06

0.09

0.12

0.00

0.03

0.06

0.09

0.12

Separation δ = ‖f0, δ − f * ‖L (ℚ) 2

Figure 3: Empirical power under fixed alternatives ∥f0,δ − f∗ ∥L2 (Q) = δ for three kernel functions. From left to right, the panels correspond to the Gaussian RBF, Matérn ν = 3/2, and Matérn ν = 1/2 (Laplacian) kernels, all with length scale ℓ = 1. The parameters (λ, τ ) are tuned using independent source-labelled and validation samples of size 300, which are held fixed across all simulation experiments. The rejection critical values are estimated using 300 multiplier-bootstrap draws. The boxplot summarizes the average power, its standard deviation and interquartile range. The dashed horizontal line marks the nominal level α = 0.10.

6

Application to Survey-Weighted Household Finance

6.1

Data and survey weights

The Survey of Consumer Finances (SCF) is a triennial cross-sectional survey of U.S. families that records income, balance sheets, credit use, and demographic characteristics. Our experiment focuses on the implicate 1 of 2022 public summary extract released by Board of Governors of the Federal Reserve System [2023], which contains 4, 595 records representing distinct families. Because the SCF records are not randomly sampled, each observation comes with a survey weight indicating how many U.S. families that observation represents. These survey weights are normalized to have P mean one, denoted by ρi satisfying n−1 ni=1 ρi = 1, which help convert averages over the unweighted records into averages for the full U.S. family population represented by the SCF. We regard the unweighted records as a sample from the source distribution P and the survey-weighted records as a sample from the target distribution Q, under which the financial conclusions are sought. 21

Record Records

CV = 0.881 | Weight-only ESS = 2,587 (56.3% of n)

Records

Records

Central 80%: 0.073-2.246 | Median: 0.704 | Mean: 1 (normalized) 500 CV = 0.881 | Weight-only ESS = 2,587 (56.3% of n) 500 0 10 1 10 3 10 2 100 0 Normalized 10 1i (log scale) 10 3 10 2 released weight 100 Normalized released weight (log scale) i C. Life-cycle class B. the Work status distribution of the normalized weights Figure 4 illustrates empirical and the effect of reweight<55, unpartnered, +7.2 pp +2.3 of pp B. Work status C. Life-cyclethe class Works for someone else ing on two specific covariates. Two measures are provided panel A describing variation noinchildren <55, unpartnered, +7.2 pp +2.3 pp
P Works for someone else <55, partnered, 1/2 n 2 +2.3 pp children normalized weights, where the coefficient of variation CV and effective nono children i=1 (ρi − 1) /n <55,:= partnered, Self-employed/
PReleased-weight +2.3 pp -10.6 pp n <55, partnered, A. distribution 2 2 +0.8 pp children in Appendix E.1. partnership size size Self-employed/ ESS = n / are no reported -10.6 statistics pp i=1 ρi . More summary children <55, partnered, +0.8 pp partnership Central 80%: 0.073-2.246 | Median: 0.704 | Mean: 1 (normalized) <55, unpartnered, children pp Our experiments (i) how recorded affect CV = does 0.881 |the Weight-only ESS characteristics = 2,587 (56.3% of n)+0.3 Retired/disabled or primarily study four questions: +3.5 pp children <55, unpartnered, +0.3 pp 65+ other nonworker 500 Retired/disabled or +3.5 the household net worth; (ii) is the probability ofpp holding stock nondecreasing in usual income; children -9.0 pp 55+,equity working 65+ other nonworker -9.0 pp 55+, working (iii) Other do households to stock equity; (iv) does the late-payment +3.4 pp nonworker0 allocate the same share0.0ofppwealth 55+, not working +3.4 pp Other change nonworker 55+, working 1 2 pp probability across years? constraints, wenot present the studies in this 30 section 10 10 3 Due 100 20 0 20 40 to page 60100.0 0 two 10 0 Appendix. 20Share (%) 40 Normalized 60 0 10 Share20(%) 30 released weight i (log scale) and other results in the Share (%) Share (%) B. Work status C. Life-cycle class class D. Education E. Race/ethnicity <55, unpartnered, D. Education E. Race/ethnicity classpp +7.20.0pppp +2.3 Nosomeone HS diploma/ Works for else A. Released-weight distribution +7.0 pp non-Hispanic no children No HS diploma/ 0.0 pp White +7.0 pp GEDGED White non-Hispanic <55, partnered, +2.3 pp Central 80%: 0.073-2.246 | Median:+4.2 0.704 | Mean: 1 no (normalized) children CV = partnered, 0.881 | Weight-only ESS = 2,587 (56.3% of n) -10.6+4.2 pppppp HSSelf-employed/ diploma/GED <55, +0.8 pp HS diploma/GED partnership children 500 Some college/ +5.7 pppp <55, unpartnered, +0.3 pp Some college/ Retired/disabled or +5.7 +3.5 pp associate children associate 65+ other nonworker -9.0 pp-7.0 pp 55+,orworking Bachelor's 0 degree -9.9 Hispanic Bachelor's degree 3 -9.9pppp Nonwhite -7.0 pp Nonwhite or Hispanic 1 2 0 or higher 10 10 40 60 or higher +3.4 pp 80 Other nonworker 0 0 2020 404010 0.0 pp 6060 55+, not10working 00 20 20 40 60 80 0 20 Share 40Normalized 20 (%) (%)(%) 60 released weight i (log0scale)10 Share (%)30 Share Share

Share (%) B. WorkF.status F.Marital Maritalstatus status

Share (%) C. Life-cycle class G. G. Age Age bands

<55, unpartnered, <35 E. Race/ethnicity class+2.3 +7.2 pp -5.9 pppp +6.7 pp -5.9pppp +6.7 nonon-Hispanic children <35 0.0 pp +7.0 pp White +0.4 pp <55, partnered,35-44 +0.4 35-44 +2.3 pppp no children Self-employed/ +4.2 pp -1.4 pp -10.6 pp -1.4 pp HS diploma/GED 45-54 <55, partnered,45-54 +0.8 pp partnership children55-64 -3.9pp pp -3.9 Some college/ +5.7 pp <55, unpartnered,55-64 +0.3 pp Retired/disabled or associate -2.7 pp -2.7 pp 65-74 +3.5 pp children65-74 65+ other nonworker Bachelor's degree Neither married -9.9 pp -7.0 pp Neither married Hispanic75+ +5.9ppppNonwhite +1.0 -9.0 pppp +5.9 +1.0 pp 75+ 55+, orworking orpartnered higher nornor partnered 00 0 2020 40 60 0 20 40 60 80 20 40 60 80 0 10 20 40 0.060pp 80 55+, not working 0 Other nonworker Share (%) Share (%) (%) +3.4 pp Share Share Share (%)(%) 0 20F. Marital 40 status 60 sample 0 G. 10bands20 30 Unweighted sample Released-weight target Unweighted Released-weight target Age (%) +6.7 pp Married/living Share (%) -5.9 pp <35or complex-designShare with partner Weight-only is descriptive does incorporatereplicate replicateweights weights variance. Weight-only ESSESS is descriptive andand does notnotincorporate or complex-design variance. +0.4 pp 35-44E. Race/ethnicity class D. Education 4: Panel A shows the normalized released weights on a logarithmic scale; the shaded region No HSFigure diploma/ -1.4 pp pp 0.0 pp +7.0 45-54 White non-Hispanic spans GED the 10th through 90th percentiles, and the solid and dashed vertical lines mark the-3.9median pp 55-64 and normalized mean, respectively.+4.2 Panels unweighted and weighted-2.7sample pp D and G compare HS diploma/GED pp 65-74 Neither married percentages for education and age bands,+5.9 respectively. pp +1.0 pp 75+ Some college/ nor partnered +5.7 pp associate 0 20 40 60 80 0 10 20 Share (%)-9.9 pp Share (%) Bachelor's degree -7.0 pp or Hispanic target Unweighted sampleNonwhiteReleased-weight or higher 6.2 Study 1: predicting household net worth 0 20 40 60 0 20 40 60 80

Married/living Married/livingD. Education Works for someone No else HS diploma/ with partner with partner GED

Weight-onlyShare ESS is descriptive replicate weights or complex-design variance.
(%) and does not incorporate Share (%) The response is Y = asinh NETWORTH × 10−5 and target regression function f∗ (x) = E(Y | X = F. Marital status G. Age bands yielding a 33x). Numerical variables are standardized, and categorical variables are encoded, Married/living -5.9 pp +6.7 pp <35 dimensional We with partner covariate variable X. The variable construction details are reported in Appendix. +0.4 pp 35-44 propose four candidate functions for f0 and test against H0 : ∥f∗ − f0 ∥L2 (Q) = 0, among which the -1.4 pp 45-54 ordinary KRR f0,KRR trained on P, i.e., unweighted household records, is of primary interest. The data is randomly split into three disjoint subsets with55-64 size: |D0 | = 1,400, |DI | = 1,800,-3.9 |DQpp| = pp 1,395. The subset D0 is reserved for training the candidate 65-74 function. Three constant-valued-2.7 candi
P P P Neither married +1.0≡pp0. date functions are f0,tgt (x) ≡ i∈D0+5.9 ρi Yipp / ≡ i∈D0 Yi /|D0 |, and f0,zero (x) 75+ i∈D0 ρi , f0,src (x)

nor partnered

0

20

40 60 Share (%)

80

Unweighted sample

0

Released-weight target 22

10 20 Share (%)

Weight-only ESS is descriptive and does not incorporate replicate weights or complex-design variance.

For f0,KRR , we use the Gaussian RBF kernel with γ = 1/33 with λn tuned by cross validation. The subset DI is used for fitting IWKRR fb and implementing the multiplier bootstrap. The tuning parameters λn and τn are selected by cross validation using D0 . The subset DQ is used for evaluation,
P P based on which we define the empirical L2 (Q)-norm: ∥g∥2b = i∈DQ ρi g 2 (Xi )/ ρ . i i∈D Q Q By B = 5,000 multiplier-bootstrap draws, we get the 90%-percentile ĉ0.10 = 0.1165 as an estimate of the quantile for ∥fb − f∗ ∥L2 (Q) . By the triangle inequality ∥f∗ − f0 ∥Qb − ∥fb − f0 ∥Qb ≤ ∥fb − f∗ ∥ b , we construct the 90% confidence interval for ∥f∗ − f0 ∥L2 (Q) by ∥fb − f0 ∥ b ± ĉ0.10 , where Q

Q

the left endpoint is rounded to zero if negative. The confidence interval is conservative by nature. The bootstrap distribution, test values ∥fb − f0 ∥Qb for four candidate functions, and the corresponding 90% confidence intervals are illustrated in Figure 5. All four intervals exclude zero. In particular, the source-only KRR candidate f0,KRR is rejected at the nominal level. Further details are provided in Appendix E.2.

Zero function 1.8867

200

Source mean 1.3457

300

Target-weighted mean 1.0072

Observed frozen-test values

Source KRR (primary) 0.5907

A. Conditional bootstrap errors 90% shared radius 0.1165

Bootstrap draws

400

100 0

0.06

0.08

Frozen source KRR (primary)

0.10

0.12

0.14 0.16 0.75 1.00 1.25 1.50 1.75 2.00 2 Unscaled empirical L ( ) norm

B. Frozen benchmark comparisons under one shared confidence set

Frozen target-weighted mean Frozen source mean Zero function 0.00

0.25 0.50 0.75 1.00 1.25 1.50 1.75 Distance and simultaneous interval induced by the confidence set

2.00

Families: benchmark design 1400, inference 1800, target geometry 1395

Figure 5: testing candidate functions for estimating household net worth based on IWKRR fb. The four candidate functions include the source-only KRR f0,KRR , the target-weighted mean f0,tgt , the source mean f0,src , and the trivial function f0,zero . Panel A displays the bootstrap distribution over 5,000 draws, 90% confidence-set radius ĉ0.1 , and the test values ∥fb− f0 ∥ of four candidate functions. Panel B shows the 90% CI for ∥f∗ − f0 ∥L2 (Q) for four candidate functions, based on which all are rejected at the nominal level α = 0.10.

23

6.3

Study 2: does higher income increase the probability of holding equity?

We are interested in the relationship between household income and the probability of holding stock equity. Let the response variable be Y = 1{HEQUITY = 1}, the single covariate Z = log{1+NORMINC}, and unknown true regression function f∗ (z) = Q(Y = 1 | Z = z), where NORMINC denotes usual or normal income. In particular, we want to test whether f∗ (z) is monotonic in z. Let M↑ be the class of [0, 1]-valued nondecreasing functions on the income domain, where functions equal Q-almost surely are identified. The class M↑ is a closed subset of L2 (Q). We aim to test against H0 : f∗ ∈ M↑ . This is a composite test. Alternatively, we test against H0 : dQ (f∗ , M↑ ) = 0, where the projection distance dQ (h, M↑ ) := inf g∈M↑ ∥h − g∥L2 (Q) . We random split the data into the training set Dt of size 1,800 and the evaluation set De of size 2,795. The IWKRR estimator fb is learned using the Gaussian RBF kernel on the training set where the parameters (λn , τn ) are selected by cross validation. The empirical target distribution P P b is computed using the evaluation set by Q(Z b Q ≤ z) := i∈De ρi 1(Zi ≤ z)/ i∈De ρi . The b b empirical distance is d(f , M↑ ) = inf g∈M↑ ∥f − g∥L2 (Q) b = 0.0151, and the 90%-percentile based on B = 5, 000 multiplier bootstrap is ĉ0.10 = 0.0252. Note that ĉ0.10 is an estimate of the 90%-quantile of ∥fb − f∗ ∥L2 (Q) . By the inequality dQ (fb, M↑ ) − dQ (f∗ , M↑ ) ≤ ∥fb − f∗ ∥L2 (Q) , we construct the 90% confidence interval for dQ (f∗ , M↑ ) by dQ (fb, M↑ ) ± ĉ0.10 , resulting into the interval [0, 0.0403], where the left endpoint is rounded to zero if negative. Since zero is included in the interval, we do not reject the null hypothesis at the significance level α = 0.10. Figure 6 reports the multiplierbootstrap distribution, the empirical distance d(fb, M↑ ), and the critical value ĉ0.1 . More details can be found in the Appendix.

300 200

90% radius 0.0256

Bootstrap error-norm distribution Observed distance 0.0190

Bootstrap draws

400

100 0 0.00

0.01

0.02 0.03 Unscaled empirical L 2( Z, E) norm

0.04

0.05

Figure 6: testing whether the probability of holding stock equity is monotone with respect to the usual income. The histogram plot shows the empirical bootstrap distribution based on 5,000 draws. The solid orange line is the observed distance d(fb, M↑ ) = 0.0151, and the dashed green line is the 90% bootstrap percentile ĉ0.10 = 0.0252. The null hypothesis is not rejected at level 0.10.

7

Discussion

Unknown density ratio. For simplicity and technical convenience, we assumed known density ratio 24

ρ(x) throughout the paper. While this assumption may hold in semi-supervised learning where unlabelled datasets are abundant, it can be restrictive in many applications. There are several existing methods to estimate the density ratio if it is unknown. Suppose that a sample of source i.i.d. e1 , · · · , X eN i.i.d. ∼ Q are available. For example, covariates X1 , · · · , Xn ∼ P and target covariates X the least squares density-ratio estimator [Kanamori et al., 2009] approximates ρ(x) by minimizing EPn [ρ2 (X)]−2EQN [ρ(X)]; the Kullback-Leibler importance estimation procedure (KLIEP, Sugiyama et al. [2008]) maximizes the target likelihood EQN log ρ(X) subject to the normalization constraint EPn ρ(X) = 1; the kernel mean matching (KMM, Huang et al. [2007], Yu and Szepesvári [2012]) approach aims to minimize the discrepancy between mean feature maps ∥EPn ρ(X)ϕ(X) − EQN ϕ(X)∥H where ϕ denotes the canonical feature map. These empirical estimates of ρ(x) are applicable to our procedures. However, their effect on inference remains unclear and challenging to characterize, because the main issue is not only consistency of ρb, but also its impact to the first-order approximation, bootstrap approximation, as well as the finite-sample coverage guarantees. We shall leave this in potential future work. Sub-Gaussian kernel function. The main theorems are established for the bounded kernel where supx∈X k(x, x) ≤ κ2 . Our methods and theories are still valid if the kernel evaluating function kX = k(X, ·) is sub-Gaussian as an H-valued element in the source distribution. In Appendix A, we provide the theoretical guarantees for the Gaussian and bootstrap approximation under subGaussian kernel functions [Wang, 2026]. Density ratio moment condition and kernel spectral decay rate. The explicit error rates for the coverage probability established in Corollary 4.2 - 4.4 require a bounded high-order moment of the density ratio and sufficiently fast-decaying eigenvalues of the integral operator LQ . These conditions are stronger than the typical ones in existing literature on the consistency of (IW)KRR. The stronger conditions arise because valid inference procedures require accurate distributional approximation √ such as the Bahadur representation involving n-scale linearization, Gaussian and bootstrap approximations, and anti-concentration for coverage guarantee. Higher moments of the density ratio make the truncation-induced discrepancy between the target and weighted-source integral operators decay faster. Moreover, faster spectral decay reduces the effective dimension and the spectral-tail error in the infinite-dimensional Gaussian and bootstrap approximations. Together, these conditions place the truncation level, effective dimension, and spectral tail in a joint regime under which all inferential remainders are asymptotically negligible, thereby yielding valid coverage. It is possible that the moment order and spectral-decay requirements in those corollaries are improvable, which we leave in potential future work.

References Rina Foygel Barber, Emmanuel J. Candes, Aaditya Ramdas, and Ryan J. Tibshirani. Conformal prediction beyond exchangeability. The Annals of Statistics, 51(2):816–845, 2023. doi: 10.1214/ 23-AOS2276. Peter J Bickel and David A Freedman. Some asymptotic theory for the bootstrap. The annals of statistics, 9(6):1196–1217, 1981.

25

Board of Governors of the Federal Reserve System. 2022 Survey of Consumer Finances. Data set, 2023. URL https://www.federalreserve.gov/econres/scfindex.htm. Nazar Buzun, Nikolay Shvetsov, and Dmitry V Dylov. Strong gaussian approximation for the sum of random vectors. In Conference on Learning Theory, pages 1693–1715. PMLR, 2022. T. Tony Cai and Mark G. Low. Adaptive confidence balls. The Annals of Statistics, 34(1):202 – 228, 2006. Andrea Caponnetto and Ernesto De Vito. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics, 7(3):331–368, 2007. Victor Chernozhukov, Denis Chetverikov, and Kengo Kato. Anti-concentration and honest, adaptive confidence bands. The Annals of Statistics, 42(5), October 2014a. Victor Chernozhukov, Denis Chetverikov, and Kengo Kato. Gaussian approximation of suprema of empirical processes. The Annals of Statistics, 42(4):1564 – 1597, 2014b. Victor Chernozhukov, Denis Chetverikov, and Kengo Kato. Central limit theorems and bootstrap in high dimensions. The Annals of Probability, 45(4):2309 – 2352, 2017. Heinz Otto Cordes. Spectral Theory of Linear Differential Operators and Comparison Algebras. London Mathematical Society Lecture Note Series. Cambridge University Press, 1987. Corinna Cortes, Mehryar Mohri, Michael Riley, and Afshin Rostamizadeh. Sample selection bias correction theory. In Algorithmic Learning Theory, volume 5254 of Lecture Notes in Computer Science, pages 38–53. Springer, 2008. doi: 10.1007/978-3-540-87987-9_8. Corinna Cortes, Yishay Mansour, and Mehryar Mohri. Learning bounds for importance weighting. Advances in neural information processing systems, 23, 2010. Richard K. Crump, V. Joseph Hotz, Guido W. Imbens, and Oscar A. Mitnik. Nonparametric tests for treatment effect heterogeneity. Review of Economics and Statistics, 90(3):389–405, 2008. Peng Ding, Avi Feller, and Luke Miratrix. Randomization inference for treatment effect variation. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 78(3):655–671, 2016. Bradley Efron and Robert J Tibshirani. An introduction to the bootstrap. Chapman and Hall/CRC, 1994. Christopher Genovese and Larry Wasserman. Adaptive confidence bands. The Annals of Statistics, 36(2):875 – 905, 2008. Evarist Giné and Richard Nickl. Confidence bands in density estimation. The Annals of Statistics, 38(2):1122 – 1170, 2010. Davit Gogolashvili, Matteo Zecchin, Motonobu Kanagawa, Marios Kountouris, and Maurizio Filippone. When is importance weighting correction needed for covariate shift adaptation? arXiv preprint arXiv:2303.04020, 2023.

26

Friedrich Götze, Alexey Naumov, Vladimir Spokoiny, and Vladimir Ulyanov. Large ball probabilities, gaussian comparison and anti-concentration. Bernoulli, 25(4A):2538–2563, 2019. Hao Guan and Mingxia Liu. Domain adaptation for medical image analysis: A survey. IEEE Transactions on Biomedical Engineering, 69(3):1173–1185, March 2022. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360, 2020. Peter Hall. The bootstrap and Edgeworth expansion. Springer Science & Business Media, 2013. Jiayuan Huang, Alexander J. Smola, Arthur Gretton, Karsten M. Borgwardt, and Bernhard Schölkopf. Correcting sample selection bias by unlabeled data. In Advances in Neural Information Processing Systems, volume 19, pages 601–608, 2007. Anatoli Juditsky and Sophie Lambert-Lacroix. Nonparametric confidence set estimation. Mathematical Methods of Statistics, 12(4):410–428, 2003. Takafumi Kanamori, Shohei Hido, and Masashi Sugiyama. A least-squares approach to direct importance estimation. The Journal of Machine Learning Research, 10:1391–1445, 2009. Wouter M Kouw, Jesse H Krijthe, and Marco Loog. Robust importance-weighted cross-validation under sample selection bias. In 2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP), pages 1–6. IEEE, 2019. Samory Kpotufe and Guillaume Martinet. Marginal singularity, and the benefits of labels in covariate-shift. In Conference On Learning Theory, pages 1882–1886. PMLR, 2018. Ker-Chau Li. Honest confidence regions for nonparametric regression. The Annals of Statistics, 17 (3):1001–1008, 1989. Mark G Low. On nonparametric confidence intervals. The Annals of Statistics, 25(6):2547–2554, 1997. Cong Ma, Reese Pathak, and Martin J Wainwright. Optimally tackling covariate shift in rkhs-based nonparametric regression. The Annals of Statistics, 51(2):738–761, 2023. Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation: Learning bounds and algorithms. arXiv preprint arXiv:0902.3430, 2009. James Mercer. Functions of positive and negative type, and their connection with the theory of integral equations. Philosophical Transactions of the Royal Society of London. Series A, 209: 415–446, 1909. Stanislav Minsker. On some extensions of bernstein’s inequality for self-adjoint operators. Statistics & Probability Letters, 127:111–119, 2017.

27

Ditlev Monrad and Walter Philipp. Nearby variables with nearby conditional laws and a strong approximation theorem for hilbert space valued martingales. Probability Theory and Related Fields, 88(3):381–404, 1991. doi: 10.1007/BF01418867. Reese Pathak, Cong Ma, and Martin Wainwright. A new similarity measure for covariate shift with applications to nonparametric regression. In International Conference on Machine Learning, pages 17517–17530. PMLR, 2022. I. F. Pinelis and A. I. Sakhanenko. Remarks on inequalities for large deviation probabilities. Theory of Probability and Its Applications, 30(1):143–148, 1986. doi: 10.1137/1130013. James Robins and Aad van der Vaart. Adaptive nonparametric confidence sets. The Annals of Statistics, 34(1):229 – 253, 2006. Johannes Schmidt-Hieber and Petr Zamolodtchikov. Local convergence rates of the nonparametric least squares estimator with applications to transfer learning. Bernoulli, 30(3):1845–1877, 2024. Bernhard Schölkopf and Alexander J Smola. Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT press, 2002. Zuofeng Shang, Peijun Sang, and Chong Jin. Bootstrap nonparametric inference under data integration. arXiv preprint arXiv:2501.01610, 2025. Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the loglikelihood function. Journal of statistical planning and inference, 90(2):227–244, 2000. Rahul Singh and Suhas Vijaykumar. arXiv:2302.06578, 2023.

Kernel ridge regression inference.

arXiv preprint

Vladimir G Spokoiny. Adaptive hypothesis testing using wavelets. The Annals of Statistics, 24(6): 2477–2498, 1996. Masashi Sugiyama and Motoaki Kawanabe. Machine Learning in Non-Stationary Environments: Introduction to Covariate Shift Adaptation. The MIT Press, 2012. Masashi Sugiyama, Matthias Krauledat, and Klaus-Robert Müller. Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research, 8:985–1005, 2007. Masashi Sugiyama, Taiji Suzuki, Shinichi Nakajima, Hisashi Kashima, Paul von Bünau, and Motoaki Kawanabe. Direct importance estimation for covariate shift adaptation. Annals of the Institute of Statistical Mathematics, 60(4):699–746, 2008. Ryan J. Tibshirani, Rina Foygel Barber, Emmanuel J. Candes, and Aaditya Ramdas. Conformal prediction under covariate shift. In Advances in Neural Information Processing Systems, volume 32, 2019. Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 2013. Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018. 28

Grace Wahba. Spline models for observational data. SIAM, 1990. Haoran Wang, Qiuye Jin, Shiman Li, Siyu Liu, Manning Wang, and Zhijian Song. A comprehensive survey on deep active learning in medical image analysis. Medical Image Analysis, 95:103201, 2024. Kaizheng Wang. Pseudo-labeling for kernel ridge regression under covariate shift. The Annals of Statistics, 54(1):252–276, 2026. Dong Xia and Ming Yuan. Statistical inferences of linear forms for noisy matrix completion. Journal of the Royal Statistical Society Series B: Statistical Methodology, 83(1):58–77, 2021. Yun Yang, Anirban Bhattacharya, and Debdeep Pati. Frequentist coverage and sup-norm convergence rate in gaussian process regression. arXiv preprint arXiv:1708.04753, 2017. Yaoliang Yu and Csaba Szepesvári. Analysis of kernel mean matching under covariate shift. arXiv preprint arXiv:1206.4650, 2012. A. Yu. Zaitsev. Estimates of the lévy–prokhorov distance in the multivariate central limit theorem for random variables with finite exponential moments. Theory of Probability & Its Applications, 31(2):203–220, 1987. Petr Zamolodtchikov. A minimax theory of nonparametric regression under covariate shift. arXiv preprint arXiv:2603.05897, 2026.

29

A

Extension I: sub-Gaussian Kernel Function

In addition to the bounded-kernel condition assumed throughout the paper, this section considers instead the sub-Gaussian feature regularity and derives the corresponding Gaussian and bootstrap coupling bounds and their specific spectral-rate consequences. Definition A.1 (Sub-Gaussian in Hilbert Space). For a square-integrable random element X in a Hilbert space G, with its expectation understood as a Bochner mean, we say X is G-b-sub-Gaussian if  b2  log EP [exp ⟨X − EX, t⟩G ] ≤ EP ⟨X − EX, t⟩2G , ∀t ∈ G. 2 Assumption A.1 (Sub-Gaussian Feature). For some b0 ≥ 1, the feature element kX = k(X, ·) is H-b0 -sub-Gaussian. Equivalently, for every t ∈ H, log EP [exp ⟨kX − EP kX , t⟩H ] = log EP [exp{t(X) − EP [t(X)]}]  b2  ≤ 0 EP {t(X) − EP [t(X)]}2 . 2 For τ ≥ τ0 , let wτ (x) := min{ρ(x), τ }. Since importance weighting is used, we additionally assume that, for every τ ≥ τ0 and t ∈ H, log EP [exp ⟨wτ (X)kX − EP [wτ (X)kX ], t⟩H ] = log EP [exp{wτ (X)t(X) − EP [wτ (X)t(X)]}]  b2  ≤ 0 EP wτ (X)2 t(X)2 . 2 The weighted-feature condition is imposed on the truncation family and does not follow from subGaussianity of kX alone. We first verify that the influence elements Ui in the Bahadur representation (3.1) satisfy the required sub-Gaussian condition. Lemma A.1. Under Assumptions 4.1, 4.2, and A.1, if τn ≥ τ0 , then Ui is L2 (Q)-b-sub-Gaussian with (κ∥f∗ ∥ + σ) 3 b = Csg · b0 , σ where Csg is a universal constant as in Vershynin [2018]. The sub-Gaussian condition yields the following new Gaussian and bootstrap coupling bounds, whereas the Bahadur representation and feasible-bootstrap approximation remain as in the main analysis. Proposition A.1 (Gaussian coupling under sub-Gaussian regularity). Suppose U1 , . . . , Un are i.i.d., centered, L2 (Q)-b-sub-Gaussian random elements with covariance operator V. For any η ∈ (0, 1) and any integer m ∈ {1, . . . , n}, if n ≥ max{2, log(2/η)}, then there exists Z ∼ NL2 (Q) (0, V) such that, with probability at least 1 − η, s   ! ( ) n 3/2 log(n)2 1 1 X 6 m √ √ Ui − Z ≲ bσL2 (Q) (m, V) log + ∥V∥L22 (Q) b3 (3/η)1/ log(mn) . η n n 2 i=1

L (Q)

30

Proposition A.2 (Bootstrap coupling under sub-Gaussian regularity). Suppose U1 , . . . , Un are i.i.d., centered, L2 (Q)-b-sub-Gaussian random elements. Let g be the isonormal Gaussian process in d. b 1/2 L2 (Q). Then there exists a random element G = V ora g D such that, conditioned on D, with total probability at least 1 − 6η,  V

1/2

g−G

L2 (Q)

≲ inf m1/4

( 4 4 )1/4 b σL2 (Q) (0, V) n

m≥1



  3/2 2  . + σL2 (Q) (m, V) log η

where the covariance operator V = E[Ui ⊗L2 (Q) Ui ]. Combining these two coupling bounds with the unchanged bias, Bahadur, feasible-bootstrap, and anti-concentration arguments from the main analysis yields sub-Gaussian counterparts of Corollaries 4.2–4.4. Throughout the following corollaries, fix α ∈ (0, 1), take η ≍ n−1 , and let n → ∞. Suppose Assumptions 4.1, 4.2, 4.3, 4.5, and A.1 hold, with source exponent s ≥ 1/2 and τn ≥ τ0 for all sufficiently large n. In each corollary, the stated density-ratio and truncation conditions verify Assumption 4.4 with C1 bounded uniformly in n. All other constants in these assumptions are independent of n. Corollary A.1 (bounded density ratio under sub-Gaussian regularity). Suppose that ρ(X) ≤ ρ P-almost surely, and let τn = C0 for a sufficiently large fixed constant C0 ≥ max{ρ, τ0 }. (1) If µk ≍ k −β for some β > 3 and all k ≥ 1, then, by choosing 2β − 3β+2

λn ≍ n we obtain

,

− β−3 P f∗ ∈ / CSα (fb) − α ≲ n 2(3β+2) log3 n.

(2) If µk ≍ exp(−ck ζ ) for some c, ζ > 0 and all k ≥ 1, then, by choosing λn ≍ n−2/3 , we obtain
P f∗ ∈ / CSα (fb) − α ≲ n−1/6 (log n)3+1/ζ . Corollary A.2 (sub-exponential density ratio under sub-Gaussian regularity). Suppose that the density ratio satisfies the sub-exponential tail condition in Definition 4.1. (1) If µk ≍ k −β for some β > 3 and all k ≥ 1, then, by choosing β−3

2β − 3β+2

τn ≍ n 6(3β+2) , we obtain

λn ≍ n

,

− β−3 P f∗ ∈ / CSα (fb) − α ≲ n 4(3β+2) log3 n.

(2) If µk ≍ exp(−ck ζ ) for some c, ζ > 0 and all k ≥ 1, then, by choosing λn ≍ n−2/3 ,

τn ≍ n1/150 , we obtain

P f∗ ∈ / CSα (fb) − α ≲ n−47/300 (log n)3+1/ζ . 31

Corollary A.3 (bounded θ-th moment under sub-Gaussian regularity). Suppose that EP [ρ(X)θ ] ≤ Ωθ < ∞ for some θ > 5. If µk ≍ exp(−ck ζ ) for some c, ζ > 0 and all k ≥ 1, then, by choosing −

λn ≍ n we obtain

B

8(θ 2 −1) 3(4θ 2 +3θ−3)

19θ+1

τn ≍ n 6(4θ2 +3θ−3) ,

,

(θ−5)(8θ+1)
− P f∗ ∈ / CSα (fb) − α ≲ n 12(4θ2 +3θ−3) (log n)3+1/ζ .

Extension II: Bias-Corrected Multiplier Bootstrap

The asymptotic validity of the confidence ball constructed by Algorithm 1 relies on undersmoothing to render the deterministic bias √

√ n(fP w − f∗ ) = − n λn (LP w + λn I)−1 f∗

negligible in L2 (Q). The procedure developed here instead estimates the corresponding biascorrection term by replacing (LP w , f∗ ) with (LPnw , fb), yielding, with α b := (K + nλn W−1 )−1 y, bb :=

√

n λn (LPnw + λn I)−1 fb =

√

−1 −1 −1 n k⊤ x (K + nλn W ) nλn W α̂.

(B.1)

√ √ As bb estimates the negative of n(fP w − f∗ ), the bias-adjusted analogue of n(fb − f∗ ) is B − bb. With db := nλn W−1 α b, equations (3.2) and (B.1) give n o √ −1 −1 diag(b ε)z − db . B − bb = n k⊤ x (K + nλn W ) Algorithm 2 summarizes this practical bias-corrected variant of Algorithm 1, which may improve finite-sample calibration accuracy.

C

Extension III: Residual Bootstrap

The residual bootstrap resamples with replacement from the empirical distribution of the centered (b) fitted residuals. Conditional on the observed covariates, it generates bootstrap responses Yi,res = (b) fb(Xi ) + εb , i = 1, . . . , n, and refits IWKRR with the same wn (x) and λn . i,res

Step 1: Gaussian and bootstrap approximation. The residual-bootstrap coverage guarantee can be established using the same proof framework as the multiplier bootstrap. Because the point estimator is unchanged, it suffices to replace the bootstrap approximation in (4.3) by its residual√ (b) bootstrap analogue: conditionally on D, n(fbres − fb) can be approximated by the same Gaussian element as in (4.2). If the corresponding error is ∆B,res , the proof of Theorem 4.1 yields the same coverage bound with ∆B replaced by ∆B,res . To identify the required bootstrap approximation, let ψn (x) := wn (x)(LP w + λn I)−1 kx ,

ψbi,n := wn (Xi )(LPnw + λn I)−1 kXi .

The leading noise component of the influence function is Ui,ε := εi ψn (Xi ). 32

Algorithm 2 Bias-corrected multiplier bootstrap for L2 (Q)-norm confidence balls Require: Data D = {(Xi , Yi )}ni=1 , kernel k(·, ·), density ratio ρ(x) = q(x)/p(x), weight wn (x) = min{ρ(x), τn }, confidence level 1 − α, bootstrap repetitions M , regularization parameter λn ; 1: Form K, W, y, kx , and KQ as in Algorithm 1. 2: Compute α b = (K + nλn W−1 )−1 y and fb = k⊤ b. xα b 3: Compute the residual vector b ε = (Y1 − f (X1 ), . . . , Yn − fb(Xn ))⊤ . 4: Form the dual bias-correction vector db = nλn W−1 α b. 5: for b = 1, . . . , M do 6: Sample z(b) ∼ N (0, In − 11⊤ /n). 7: Set the bias-corrected coefficient vector (b)

abc =

8:

√

b n (K + nλn W−1 )−1 {diag(b ε)z(b) − d}.

Compute the bootstrap statistic (b) (b) Tbc = ∥k⊤ x abc ∥L2 (Q) =

q (b) (b) (abc )⊤ KQ abc .

9: end for (b)

M bc be the empirical (1 − α)100%-percentile of {T 10: Let r̂α bc }b=1 .

b 11: Define CSbc α (f ) = {f ∈ H :

√

n ∥fb − f ∥L2 (Q) ≤ r̂αbc }. Ensure: the IWKRR fb and the bias-corrected critical value r̂αbc . The normal equations yield the exact decompositions 1 fb − f∗ = n

n X

εi ψbi,n − λn (LPnw + λn I)−1 f∗ ,

(C.1)

i=1 n

1 X (b) b (b) εbi,res ψi,n − λn (LPnw + λn I)−1 fb. fbres − fb = n

(C.2)

i=1

The same Bahadur, undersmoothing, and operator-approximation methods used in the main analysis can be applied to the exact decompositions above to obtain the first-order representations √

n

1 X n(fb − f∗ ) ≈L2 (Q) √ εi ψn (Xi ), n i=1

n √ 1 X (b) b (b) b b n(fres − f ) ≈L2 (Q) √ εbi,res ψi,n , n i=1

where we ignore the bias fP w − f∗ for simplicity. Starting from these representations, the bootstrapapproximation and coverage arguments in the main text can be adapted to the residual bootstrap, subject to the covariance-consistency condition below. P Step 2: covariance comparison. The population covariance operator of √1n ni=1 εi ψn (Xi ) is     Vε,n := E Ui,ε ⊗L2 (Q) Ui,ε = E E[ε2 | X] ψn (X) ⊗L2 (Q) ψn (X) . 33

(C.3)

Algorithm 3 Residual bootstrap for L2 (Q)-norm confidence balls Require: Data D = {(Xi , Yi )}ni=1 , kernel k(·, ·), density ratio ρ(x) = q(x)/p(x), weight wn (x) = min{ρ(x), τn }, confidence level 1 − α, bootstrap repetitions M , regularization parameter λn . 1: Form K, W, y, kx , and KQ as in Algorithm 1. 2: Compute α b = (K + nλn W−1 )−1 y and fb = k⊤ b. xα b 3: Compute the residual vector b ε = (Y1 − f (X1 ), . . . , Yn − fb(Xn ))⊤ . 4: Center the fitted residuals by setting n

1X εb := εbj , n

εb◦j := εbj − εb.

j=1

5: for b = 1, . . . , M do 6: 7:

(b)

(b)

Sample εb1,res , . . . , εbn,res independently with replacement from {b ε◦1 , . . . , εb◦n }. Form the bootstrap-response vector
⊤ (b) (b) yres := fb(X1 ) + εb1,res , . . . , fb(Xn ) + εb(b) . n,res

8:

Refit IWKRR with the same regularized inverse: (b) −1 −1 (b) fbres := k⊤ x (K + nλn W ) yres .

9:

Define the coefficient perturbation
−1 −1 (b) a(b) yres −y , res := (K + nλn W )

10:

(b) (b) fbres − fb = k⊤ x ares .

Compute the bootstrap statistic n o1/2 (b) (b) (b) (b) ⊤ (b) Tres := ∥fbres − fb∥L2 (Q) = ∥k⊤ a ∥ = (a ) K a . 2 Q res x res L (Q) res

11: end for (b)

M 12: Let ĉres α,n be the empirical (1 − α)100%-percentile of {Tres }b=1 . res b 13: Define CSres α (f ) by replacing ĉα,n with ĉα,n in (3.3).

b Ensure: The residual-bootstrap confidence ball CSres α (f ).

34

Conditional on D, the covariance operator of the leading term generated by Algorithm 3 is h i  ◦ 2 2 2 bres := σ V bres En ψbi,n ⊗L2 (Q) ψbi,n , σ bres := En (b εi ) .

(C.4)

Under conditional homoskedasticity, E[ε2 | X] = E[ε2 ] almost surely, (C.3) factorizes as Vε,n = E[ε2 ] E[ψn (X) ⊗L2 (Q) ψn (X)]. Write h i   Abn := En ψbi,n ⊗L2 (Q) ψbi,n , An := E ψn (X) ⊗L2 (Q) ψn (X) . Covariance consistency is therefore ensured by 2 σ bres − E[ε2 ]

Abn

HS,L2 (Q)

+ E[ε2 ] Abn − An

P

HS,L2 (Q)

− → 0,

as n → ∞.

Step 3: comparison with the multiplier bootstrap. Notice that Vbi = εbi ψbi,n . The conditional covariance operators of the multiplier and residual bootstraps are, respectively, h
i 2 bres = σ b = En εb2i ψbi,n ⊗L2 (Q) ψbi,n − En [Vbi ] ⊗L2 (Q) En [Vbi ], V V bres En [ψbi,n ⊗L2 (Q) ψbi,n ]. Thus, the multiplier bootstrap weights each ψbi,n ⊗L2 (Q) ψbi,n by its paired squared residual εb2i , whereas 2 for all i. The multiplier bootstrap can therefore the residual bootstrap uses the common factor σ bres accommodate conditional heteroskedasticity, whereas the residual bootstrap requires the covariancematching condition above.

D

Extension IV: mis-coverage rate for heavy-tailed density ratio

For ease of exposition, the mis-coverage rate is presented in Corollary 4.4 only for the case when LQ has exponential-decay eigenvalues. The following corollary showcases the mis-coverage rate if LQ has polynomial-decay eigenvalues. Corollary D.1 (Polynomial spectral decay under a bounded θ-th moment). Under the assumptions of Section 4.3.4, suppose that EP [ρ(X)θ ] ≤ Ωθ < ∞ for some θ > 2. If µk ≍ k −β for some β > 5 and all k ≥ 1, assume that 4(β + 1)2 (2β + 5) θ−1> . β(β − 5)(β + 10) Take d = 1/β in Assumption 4.3 and choose δ=

β(θ − 1)(β − 5)(β + 10) − 4(β + 1)2 (2β + 5) . 3(2β + 5)(β + 10){3β(θ − 1)(β + 3) + 2(β + 1)2 }

By choosing −

λn ≍ n

τn ≍ n we obtain

2β(θ−1)(β+1) 3β(θ−1)(β+3)+2(β+1)2

,

β(θ−1)(β−5)(β+10)+2(3β+28)(β+1)2 (2β+5) 3(2β+5)(β+10){3β(θ−1)(β+3)+2(β+1)2 }

,

(β+9){β(θ−1)(β−5)(β+10)−4(β+1)2 (2β+5)}
− b P f∗ ∈ / CSα (f ) − α ≲ n 2(β+10)(2β+5){3β(θ−1)(β+3)+2(β+1)2 } log3 n.

In particular, the displayed rate converges to zero under any of the following sufficient regimes: 144 θ ≥ 9, β ≥ 113; θ > 9, β > 5 + ; θ ≥ 10, β ≥ 20. 5(θ − 9) Another sufficient regime is θ ≥ 11 and β ≥ 15. 35

E

Additional Results of SCF Data Analysis

This appendix provides additional data, implementation, and diagnostic details for Studies 1 and 2 reported in Section 6, and presents additional results from Studies 3 and 4.

E.1

Data description.

Studies 1–3 use the 2022 implicate-1 sample described in Section 6.1. Study 4 additionally uses implicate 1 of the 2019 public summary extract, comprising 5,777 distinct families. To illustrate how the released weights change covariate distribution, we partition the 33 scalar coordinates into a collection B of 12 demographic and household covariate blocks. For a coordinate P Z observed on the n records, define the unweighted moments µ bP (Z) := n−1 ni=1 Zi and σ bP2 (Z) := P P 2 (Z) := n−1 ni=1 {Zi − µ bP (Z)}2 , and the released-weight moments µ bQ (Z) := n−1 ni=1 ρi Zi and σ bQ P n−1 ni=1 ρi {Zi − µ bQ (Z)}2 . The coordinate-level signed standardized difference and the score of each block b ∈ B are D(Z) :=

µ bQ (Z) − µ bP (Z) , 2 2 {[b σP (Z) + σ bQ (Z)]/2}1/2

Sb := max|D(Z)|. Z∈b

We rank the 12 block scores Sb , and Figure 7 displays the six highest-ranked blocks.

E.2

Study 1: predicting household net worth

Variables and preprocessing. Table 3 describes the variables used in the 33-dimensional design summarized in Section 6.2. The work-status block is given by the four-level public-file variable OCCAT1. Table 3: Additional details of variables used for Study 1. Object

SCF variable

Meaning

Processing

Response

NETWORTH

Total family net worth

asinh(NETWORTH/100000), truncated to zero for negative net worth.

Target weight

WGT

# of U.S. families represented

normalized to mean one

Continuous coordinates

AGE KIDS

Age of the reference person; Number of children

standardized.

Ordered block

EDCL

Four levels of education

1{EDCL ≥ k}, k = 2, 3, 4.

Unordered blocks

HHSEX MARRIED LF LIFECL FAMSTRUCT RACECL OCCAT1 INDCAT HHOUSES

Sex of the reference person; Married or not; Working versus not working; Six categories of life cycle; Five-class family structure; (non)White, (non)Hispanic; Four-category work status; Three industry classes; own residence or not

one-hot encodings yielding 28 coordinates, three education indicators, and two continuous coordinates; total number: 33.

36

Records

A. Released-weight distribution Central 80%: 0.073-2.246 | Median: 0.704 | Mean: 1 (normalized) CV = 0.881 | Weight-only ESS = 2,587 (56.3% of n)

500 0

Works for someone else

10 3

B. Work status

Self-employed/ partnership Retired/disabled or 65+ other nonworker Other nonworker

No HS diploma/ GED HS diploma/GED Some college/ associate Bachelor's degree or higher Married/living with partner

Neither married nor partnered

10 1 10 2 Normalized released weight i (log scale)

0.0 pp

<55, unpartnered, no children <55, partnered, no children <55, partnered, children <55, unpartnered, children 55+, working 55+, not working

0.0 pp

White non-Hispanic

+7.2 pp -10.6 pp +3.5 pp

0

20

40 Share (%)

D. Education

60

100

C. Life-cycle class

0

10

+2.3 pp +2.3 pp +0.8 pp +0.3 pp -9.0 pp +3.4 pp

20 30 Share (%)

E. Race/ethnicity class

+7.0 pp

+4.2 pp +5.7 pp

0

20

40 Share (%)

F. Marital status

0

20

40 60 Share (%)

-9.9 pp

Nonwhite or Hispanic

-5.9 pp

<35 35-44 45-54 55-64 65-74 75+

60

+5.9 pp

80

Unweighted sample

0

20

40 60 Share (%)

G. Age bands

0

Released-weight target

10 20 Share (%)

-7.0 pp

80

+6.7 pp +0.4 pp -1.4 pp -3.9 pp -2.7 pp +1.0 pp

Weight-only ESS is descriptive and does not incorporate replicate weights or complex-design variance.

Figure 7: Distribution of survey weights and their impact in 2022 implicate-1 records. Panel A summarizes the normalized weights; the shaded band is the 10–90% percentile range, and the vertical lines mark the median and normalized mean. Panels B–G compare unweighted and weighted shares for the six blocks with the largest Sb ; labels are target-minus-sample percentage-point differences.

37

KRR and parameters tuning on D0 . Within the 980/420 inner split of D0 , the 980 training families determine the continuous-variable standardization constants and categorical encoding. This transformation is applied directly to the 420 validation families. Writing ntr = 980, the ordinary KRR search uses λn (a) = an−0.6 tr ,

a ∈ Aλ := {0.0625, 0.125, 0.25, 0.5, 1, 1.5, 2, 3, 4},

and are compared by unweighted validation mean squared error. The selected multiplier a = 0.0625 is evaluated at n = |D0 | = 1,400, giving λKRR = 0.000809475. IWKRR and multiplier bootstrap on DI . The same 980/420 split of D0 is used to tune the IWKRR parameters. Writing ntr = 980, the joint search uses λn (a) = an−0.6 tr ,

a ∈ Aλ ,

τn (b) = bn0.06 tr ,

b ∈ Aτ := {0.25, 0.5, 1, 1.5, 2},

together with τ = 10 as the truncation cap. After validation, the selected multipliers are fixed at a = 0.25 and b = 1. Substituting n = |DI | = 1,800 into the two sequences gives λIW = 0.002784688 and τn = 1.567893. For the final IWKRR fit, the weights are wi = ρi ∧ τn ,

|DI |wi , w̄i = P ℓ∈DI wℓ

and the final inference-sample fit solves   X 1 fbIW = arg min  w̄i {Yi − f (Xi )}2 + λIW ∥f ∥2H  . f ∈H |DI | i∈DI

The multiplier bootstrap is then carried out using all 1,800 records in DI . Confidence set construction and p-values. The confidence-set radius in Section 6.2 corresponds √ to the 90% critical value 4.942249 on the 1,800 scale. Let Rb denote the target-norm bootstrap error in draw b. For a candidate f0 , the corresponding plus-one bootstrap p-value is n o P b 1 R ≥ ∥ f − f ∥ 1+ B 0 IW b b b=1 Q pb(f0 ) = . B+1 All bootstrap statistics are smaller than the four observed distances, so pb(f0 ) = 1/(B + 1) ≈ 0.0002 for all the four proposed candidate f0 . Response distribution. Figure 8 compares the empirical distribution of household net worth with and without re-weighting. Covariate distributions before and after weighting. Figure 9 reports the leading change of coordinates from each of the 12 blocks. For example, the label “Work status: Self-employed/partnership” means that, among the one-hot coordinates in the work-status block, the self-employed/partnership indicator has the largest |D(Z)|. Figure 10 reports |D(Z)| for every coordinate. These standardized differences illustrate the change of covariates’ distribution under reweighting.

E.3

Study 2: does higher income increase the probability of holding equity?

The training set Dt is divided into a fitting subset Dt,fit of size 1,260 and a validation subset Dt,val of size 540 for parameter tuning. The RBF kernel with γ = 1 is used after standardizing the income variable. 38

A. Net-worth response distribution

Survey-weighted mean response

Cumulative share

1.0 0.8 0.6 0.4 0.2 0.0

Unweighted sample Survey-weighted target

2.5 0.0 2.5 5.0 7.5 10.0 Response: asinh(NETWORTH/100000)

B. Descriptive household pattern Does not own primary residence Owns primary residence

2.5 2.0 1.5 1.0 0.5

<35 35-44 45-54 55-64 65-74 75+ Age of reference person

Figure 8: empirical distribution of household net worth with and without re-weighting in Study 1, where the full-sample means are 2.497 and 1.587, respectively.

All 12 Study 1 covariate blocks Work status: Self-employed/partnership

-0.290

Life-cycle class: Age 55 or older, working

-0.209

Education: Bachelor's degree or higher

-0.199 +0.146

Race/ethnicity class: White non-Hispanic Marital status: Married/living with partner

-0.122

Age

-0.119

Family structure: Partnered, no children

-0.097

Sex: Male

-0.094

Labor-force participation: Not working

+0.078

Industry class: Not working

+0.074

Homeownership: Does not own primary residence

+0.035 -0.026

Number of children

Top-six block leader Other coordinate

0.3 0.2 0.1 0.0 0.1 0.2 0.3 Standardized difference (released-weight target - sample) Figure 9: block-level source–target comparison of covariates used in Study 1. The coordinate with the largest |D(Z)| within each block is presented. The top-six covariates are already reported in Figure 7.

39

All 33 Study 1 design coordinates

Ranks 1-17 by absolute difference occupation_class=2

-0.290

Ranks 18-33 by absolute difference

labor_force_status=1

-0.078

life_cycle_class=5

-0.209

life_cycle_class=6

+0.078

education_class>=4

-0.199

industry_class=4

+0.074

family_structure=2

+0.068

life_cycle_class=1

+0.068

industry_class=2

-0.065

homeownership_indicator=0

+0.035

+0.146

race_class=1 race_class=2

-0.146

occupation_class=1

+0.145

married=2

+0.122

married=1

-0.122

age

-0.119

family_structure=5

-0.097

hhsex=2

+0.094

hhsex=1

-0.094

education_class>=3

homeownership_indicator=1

-0.035

family_structure=4

-0.032

kids

-0.026

life_cycle_class=3

+0.019

-0.090

family_structure=1

+0.014

family_structure=3

+0.087

life_cycle_class=4

+0.013

occupation_class=3

+0.078

industry_class=1

-0.006

life_cycle_class=2

+0.078

occupation_class=4

-0.002

labor_force_status=0

+0.078

education_class>=2

0.2 0.0 0.2 Standardized difference (released-weight target - sample) Top-six block leader

+0.001

0.2 0.0 0.2 Standardized difference (released-weight target - sample)

Other coordinate

Figure 10: Coordinate-level source–target comparison for all 33 covariates used in Study 1, ordered by |D(Z)|. The top-six covariates are already reported in Figure 7.

40

Parameter tuning. Writing nt = |Dt | = 1,800, the validation search uses λn (a) = an−0.6 , t

a ∈ Aλ := {0.015625, 0.03125, 0.0625, 0.125, 0.25, 0.5, 1, 1.5, 2, 3, 4},

τn (b) = bn0.06 , t

b ∈ Aτ := {0.0625, 0.125, 0.25, 0.5, 1, 1.5, 2},

as well as the weak-truncation candidate τn = 10. The final selection is (λn , τn ) = (0.005569, 0.1960), corresponding to (a, b) = (0.5, 0.125). There are 1,480 records in Dt have ρi > τn , which carry 98.9% of the total weight in Dt if no truncation. Projection and evaluation. Let QZ denote the marginal law of Z under Q, and let Ie denote the index set of the 2,795 records in De . The empirical target measure uses the untruncated ρi values, normalized to sum to one over Ie : X ρi b := , i ∈ Ie , Q πi δZi . πi := P j∈Ie ρj i∈Ie

Let z(1) < · · · < z(K) be the distinct values in {Zi : i ∈ Ie }; here K = 802. For Ik = {i ∈ Ie : Zi = z(k) }, define X X Wk = πi , f¯k = W −1 πi fb(Zi ). k

i∈Ik

i∈Ik

The pool-adjacent-violators algorithm (PAVA) computes the bounded weighted projection onto the nondecreasing class: K X (b g1 , . . . , gbK ) = arg min Wk (f¯k − gk )2 , 0≤g1 ≤···≤gK ≤1

k=1

and assigns gb(Zi ) = gbk for every i ∈ Ik . The fitted IWKRR and its monotonic projection are presented in Figure 11. Among the 5,000 multiplier bootstrap draws, 2,697 exceed the observed distance, yielding the p-value 0.5395.

E.4

Study 3: equity-share allocation over financial wealth

We are interested in whether, among families holding stock equity (HEQUITY = 1) and having positive financial assets (FIN > 0), the share of financial assets invested in stock equity (EQUITY/FIN) is constant across levels of financial wealth (FIN). Define

Y =

S = 1{HEQUITY = 1, FIN > 0},

Qalloc = Q( · | S = 1),

EQUITY , FIN

f ∗ (z) = EQalloc (Y | Z = z).

Z = log{1 + FIN},

Let Qalloc denote the marginal law of Z under Qalloc and Z C = {gc : gc (z) ≡ c, c ∈ [0, 1]} be the class of constant-valued functions. We test the composite null H0 : f ∗ ∈ C, Qalloc Z -almost surely. Equivalently, H0 : dQalloc (f ∗ , C) = 0, dQalloc (h, C) := inf ∥h − g∥L2 (Qalloc ) . g∈C

41

Z

d2mono share (\%)

Participation probability

A. IWKRR center and closest nondecreasing projection 1.0

P1 $11k

Usual income (fit uses log{1 + NORMINC}) P50 $75k

P10 $22k

P90 $216k

P99 $996k

dmono = 0.0235, r0.10 = 0.0431 dmono/r0.10 = 0.545; working rule not triggered

0.8 0.6 0.4 0.2 0.0

B. Contributions to the global shape distance

100 89.6%

blue/orange: positive/negative mean deviation; gray: balanced

50 0

8.9%

0

10 IWKRR center

20

30 40 50 60 70 Target-weighted percentile of usual income

Monotone projection

Observed decile share

80

90

100

Outside development support (n=9)

Figure 11: probability of holding stock-equity as a function of usual income. Panel A shows the IWKRR center and its closest nondecreasing projection. Horizontal segments and open circles give the income range and observed participation mean, respectively. Panel B partitions the difference between fb and its monotonic projection into 20 intervals, each containing approximately 5% of the evaluation target weight.

42

There are 2,895 eligible records. A random permutation assigns 800 records to the training set and the remaining to the held-out evaluation set. After tuning, we select λn = 0.001132, τn = 10, and refit the IWKRR on the full training set. b alloc denote the normalized released-weight On the held-out survey-weighted evaluation set, let Q empirical measure of Z. Projecting the resulting IWKRR fb onto C gives the nearest constant 0.4432, and d(fb, C) = 0.08703, ĉ0.10 = 0.043620. The constructed confidence interval is d(f ∗ , C) ∈ [0.043406, 0.130646], and the bootstrap p-value is 0.0002, leading to the rejection of the null hypothesis. The empirical bootstrap distribution is displayed in Figure 12. The fitted IWKRR fb is illustrated in Figure 13.

Observed distance 0.0884

Confidence-ball bootstrap error distribution 90% radius 0.0425

Bootstrap draws

400 300 200 100

0 0.00

0.02

0.04 0.06 Unscaled empirical L 2( +Z ) norm

0.08

Figure 12: empirical bootstrap distribution for Study 3 based on 5,000 bootstrap draws. The dashed green line is the 90% percentile ĉ0.10 = 0.043620, and the orange line is the observed statistic d(fb, C) = 0.087026, yielding the plus-one p-value 0.0002. We reject the null hypothesis that households allocate the same share of wealth to equity stocks.

E.5

Study 4: does late-payment occurrence change across years?

We study whether late-payment occurrence in the full target population changed between the 2019 and 2022 SCF waves. Let Q2019 and Q2022 denote the survey-weighted target distributions in the two waves. For each t ∈ {2019, 2022}, the SCF public summary extract contains the Federal Reserve–constructed binary late-payment variable Yt = LATEt = 1{X3004t = 5}. Here, Yt = 1 indicates that the family sometimes fell behind or missed payments during the preceding year. The value Yt = 0 includes both families with X3004t = 1, indicating payments made on or ahead of schedule, and families with X3004t = 0, for whom the question was inapplicable because they held none of the listed loan types. For each t ∈ {2019, 2022}, let Xt denote the wave-t vector of the same twelve SCF variables used in Study 1. Let Qt,X denote the marginal distribution of Xt under Qt , and define the target 43

Equity-share allocation versus financial wealth 2022 SCF stock-equity participants, implicate 1 exploratory pilot

(a)

Stock-equity share of financial assets

̂

0.60

dflat = 0.0884 r0.10 = 0.0425 (B = 4, 999) Working distance interval: [0.0460, 0.1309] Working exact-flatness rule: triggered Shading shows deviation from the constant, not uncertainty.

IWKRR center Closest bounded constant Observed share (10 equal-target-mass bins) Outside development support (0.18% target mass)

0.45

0.30

0.15

Target mass per bin (%)

0.00 10

(b)

5 0 $10

$100

$1k $10k $100k $1m $10m Financial assets, FIN (2022 dollars; logarithmic scale)

$100m

$1b

Figure 13: equity-share allocation over financial wealth. The upper panel shows the IWKRR fb, its closest constant-valued function, and descriptive equity shares in ten approximately equal-targetmass wealth groups. The lower panel shows the held-out target mass over financial wealth.

44

regression function ft∗ (x) = Qt (Yt = 1 | Xt = x). We test against the null hypothesis ∗ ∗ H0 : ∥f2022 − f2019 ∥L2 (Q2022,X ) = 0.

b 2022,E be the normalized released-weight Let fbt be the wave-specific IWKRR estimator, and let Q empirical covariate measure on a held-out 2022 evaluation set. Define b = ∥fb2022 − fb2019 ∥ 2 b D . L (Q2022,E ) The triangle inequality gives ∗ ∗ b b b − ∥f ∗ − f ∗ ∥ 2 b D b 2022,E ) . 2022 2019 L (Q2022,E ) ≤ ∥(f2022 − f2022 ) − (f2019 − f2019 )∥L2 (Q (b)

For b = 1, . . . , B, independently generate centered-Gaussian residual-multiplier perturbations ebt of the two fitted functions and compute (b)

(b)

Rb = ∥b e2022 − eb2019 ∥L2 (Qb 2022,E ) . ∗ ∗ If b cjoint is the (1−α)100%-percentile of R1 , . . . , RB , the corresponding interval for ∥f2022 −f2019 ∥L2 (Qb 2022,E ) α is h i b −b b +b {D cjoint }+ , D cjoint , α α

and its plus-one upper-tail p-value is pb =

1+

PB

b=1 1{Rb ≥ D}

b

B+1

.

In each wave, 1,200 records form the wave-specific training set. The parameters (λn , τn ) are tuned separately within each wave, using the Gaussian RBF kernel with γ = 1/33. The final selections are: (λn , τn ) = (0.003551659, 1.530209) for 2019,

(λn , τn ) = (0.028413272, 0.765105) for 2022.

The joint multiplier bootstrap uses B = 5,000 draws. The fitted distance and 90th-percentile are b = 0.046569, D

b cjoint 0.10 = 0.049029.

The corresponding interval is [0, 0.095598], and the plus-one upper-tail p-value is 0.1562, leading to the non-rejection of the null hypothesis. Figure 14 showcases the survey-weighted distribution of the difference fb2022 − fb2019 on the held-out 2022 evaluation set.

F

Proofs of Main Results

We collect the proofs of main theorems and propositions that appear in the main text and Appendix A. Proofs of auxiliary results are collected in Appendix G. For notational simplicity, we may write λ and w(x) in short for λn and wn (x), respectively. The notation ∥ · ∥ is reserved for ∥ · ∥H . 45

Distribution of fitted 2022-2019 late-payment risk changes q10 -6.65

median +0.80

q90 +5.51

10 8

Mean +0.14 pp

2022 target mass per bin (%)

12

6 4 2 Lower in 2022 0 15

10

Higher in 2022

5 0 5 10 Estimated 2022-2019 late-payment risk change (percentage points)

15

Figure 14: survey-weighted distribution of the difference fb2022 − fb2019 on the held-out 2022 evaluation set.

F.1

Proof of Lemma 4.1

Lemma F.1. For fixed k ∈ N+ and sufficiently small λ > 0, k ∞  ∞ X µi 1 X k ≍ s∗ + k µi , µi + λ λ ∗ i=1

i=s +1

where s∗ = min{s : µs ≤ λ}. Lemma F.2. For fixed k ∈ N+ and sufficiently small λ > 0, the following rates hold. If µi ≍ i−β for some β > 1, then k ∞  X µi ≍ λ−1/β . µi + λ i=1

If µi

≍ exp(−βiγ ) for some β > 0 and γ > 0, then ∞  X i=1

µi µi + λ

k

≍ {log(1/λ)}1/γ .

Lemma F.3. Under Assumption 4.1, for λ > 0, nh i h io V ⪰L2 (Q) σ 2 E (LP w + λI)−1 w(Xi )kXi ⊗L2 (Q) (LP w + λI)−1 w(Xi )kXi . Lemma F.4.
     = (LP w + λI)−1 LP w (LP w + λI)−1 LQ E w(Xi ) (LP w + λI)−1 kXi ⊗L2 (Q) (LP w + λI)−1 kXi Lemma F.5. Let T : H → H be a bounded positive self-adjoint operator in H. If T LQ : L2 (Q) → 1/2 1/2 L2 (Q) is a compact operator in L2 (Q) and LQ T LQ : H → H is a compact operator in H, then their nonzero eigenvalues, counted with multiplicity, coincide. In particular, after padding a finite nonzero spectrum by zeros, 1/2

1/2

µk (T LQ , L2 (Q)) = µk (LQ T LQ , H), 46

k ≥ 1.

Lemma F.6. Under Assumption 4.2, when τn ≥ τ0 , we have ∥fP w − f∗ ∥ ≤ max{κ2s−1 , 1}λmin{s,1/2} ∥g∗ ∥. Lemma F.7. Under Assumptions 4.1, 4.2, and 4.4, when τn ≥ τ0 , we have  
2
trL2 (Q) (V) ≤ τn σ 2 + κ2 max{κ2s−1 , 1}2 λ2 min{s,1/2} ∥g∗ ∥2 C1 trH (LP w + λI)−1 LP w  
2 ≤ τn σ 2 + κ2 max{κ2s−1 , 1}2 λ2 min{s,1/2} ∥g∗ ∥2 C1 Nλ ,  
2 ∥V∥L2 (Q) ≤ τn σ 2 + κ2 max{κ2s−1 , 1}2 λ2 min{s,1/2} ∥g∗ ∥2 C1 . Lemma F.8. Under Assumptions 4.1 and 4.5, writing λ = λn , we have V ⪰L2 (Q) C2

−2

ρσ 2 (LQ + λI)−1 LQ (LQ + λI)−1 LQ .

Lemma F.9 (Theorem 2.6 of Götze et al. [2019]). Let ξ ∼ NG (0, Vξ ), where G is a separable Hilbert space. Denoting the probability density function of ∥ξ∥2G by p∥ξ∥2 , we have G

sup p∥ξ∥2 (t) ≲ κ(Vξ ) ≲ (Λ1ξ Λ2ξ )−1/2 G

t≥0

where

 −1  Λ1ξ , 

if 3λ21ξ < Λ21ξ ;

(λ1ξ Λ2ξ )−1/2 , if 3λ21ξ ≥ Λ21ξ , 3λ22ξ < Λ22ξ ;   (λ λ )−1/2 , if 3λ2 ≥ Λ2 , 3λ2 ≥ Λ2 . 1ξ 2ξ 2ξ 2ξ 1ξ 1ξ P ∞ with λiξ = λi (Vξ , G) and Λ2kξ = i=k λ2iξ for k = 1, 2. κ(Vξ ) =

Remark 3. Lemma F.9 states a slightly corrected version of Theorem 2.6 in Götze et al. [2019], which does not affect any of the ≲-order consequences used in this proof. Proof of Lemma 4.1. Let (νj , ej )j≥1 be the L2 (Q)-eigenpairs of V, ordered so that ν1 ≥ ν2 ≥ · · ·, and define ∞ X 2 ΛkZ := νj2 , k = 1, 2. j=k

We first prove the general bound sup p∥Z∥L2 (Q) (t) ≲ t≥0

q trL2 (Q) (V)(Λ1Z Λ2Z )−1/2 .

Let T := ∥Z∥L2 (Q) . By Lemma F.9, applied with G = L2 (Q) and covariance V, the random variable T 2 has a density satisfying sup pT 2 (u) ≲ (Λ1Z Λ2Z )−1/2 . u≥0

For t > 0, the density of T satisfies pT (t) = 2t pT 2 (t2 ). 47

It remains to turn this pointwise bound into a bound that is uniform in t. For a finite-rank truncation Zm =

m X √

νj gj ej ,

j=1

the distribution function of Tm = ∥Zm ∥L2 (Q) can be written as P(Tm ≤ t) = γm (tKm ), where γm is a centered Gaussian measure on span{e1 , . . . , em } and Km is the unit ball in that subspace. Since Gaussian measures are log-concave and a(tKm ) + (1 − a)(sKm ) = {at + (1 − a)s}Km ,

a ∈ [0, 1],

the distribution function FTm (t) = P(Tm ≤ t) is log-concave on [0, ∞). Moreover, Tm → T almost surely. Since T has a density, FTm (t) → FT (t) for every t, and hence FT is also log-concave. Put AZ := (Λ1Z Λ2Z )−1/2 , mZ := {trL2 (Q) (V)}1/2 . The Gaussian fourth-moment identity gives ET 4 = {trL2 (Q) (V)}2 + 2∥V∥2HS,L2 (Q) ≤ 3m4Z . Applying the Paley–Zygmund inequality to T 2 therefore yields   mZ 1 mZ √ P T ≥ ≥ , ET ≥ √ . 12 2 12 2 If 0 < t ≤ 4ET , the squared-norm density bound gives pT (t) = 2tpT 2 (t2 ) ≲ AZ mZ . If t > 4ET , Markov’s inequality gives FT (t/2) ≥ 1/2. At every differentiability point of log FT , its concavity implies pT (t) log FT (t) − log FT (t/2) 2 log 2 = {log FT }′ (t) ≤ ≤ . FT (t) t/2 t Thus pT (t) ≲ m−1 Z for almost every t in this second regime. Finally, m2Z = trL2 (Q) (V) ≥ Λ1Z ≥ (Λ1Z Λ2Z )1/2 , so that m−1 Z ≤ AZ mZ . Choosing the density version satisfying these almost-everywhere bounds and combining the two regimes proves q sup p∥Z∥L2 (Q) (t) ≲ trL2 (Q) (V)(Λ1Z Λ2Z )−1/2 . t≥0

We now bound the two covariance quantities in the preceding display. By Lemma F.7 and its proof, ∞ X
µi trL2 (Q) (V) ≲ τn trH (LQ + λI)−1 LQ = τn Nλ = τn , µi + λ i=1

48

where fixed constants depending on the noise, source condition, and compatibility constant are absorbed into ≲. On the other hand, Lemma F.8 gives V ⪰L2 (Q) C2

−2

ρσ 2 (LQ + λI)−1 LQ (LQ + λI)−1 LQ .

By monotonicity of eigenvalues under the Loewner order for positive compact operators, the ordered eigenvalues satisfy  2 µi νi ≳ , i ≥ 1, µi + λ because the eigenvalues of (LQ + λI)−1 LQ (LQ + λI)−1 LQ are (µi /(µi + λ))2 . Consequently, for k = 1, 2, 4 ∞  X µi 2 . ΛkZ ≳ µi + λ i=k

Under either two-sided spectral condition, Lemma F.2 shows that the full sum on the right-hand side diverges as λ → 0. Since each summand is at most one, deleting its first term does not change its order for all sufficiently small λ. On any remaining compact subinterval of (0, 1], the ratio of the tail sum to the full sum is continuous and strictly positive. Thus, after enlarging the implicit constant, for k = 1, 2, (∞  4 )1/2 X µi ΛkZ ≳ . µi + λ i=1

Combining the preceding bounds yields sup p∥Z∥L2 (Q) (t) ≲ t≥0

√

1/2  P∞ i=1 µi /(µi + λ) P τn . ∞ 4 i=1 {µi /(µi + λ)}

Under either the two-sided polynomial or two-sided exponential spectral decay regime, Lemma F.2, after the same compact-interval adjustment over 0 < λ ≤ 1, ∞ X i=1

∞ 

X µi ≍ µi + λ i=1

µi µi + λ

4 ;

as λ → 0, both sums are of order λ−1/β in the polynomial case and {log(1/λ)}1/γ in the exponential case, up to constants depending only on the spectral parameters. Therefore sup p∥Z∥L2 (Q) (t) ≲

√

τn .

t≥0

F.2

Proof of Theorem 4.1

Proof of Theorem 4.1. Let T :=

√

n∥fˆ − f∗ ∥L2 (Q) ,

S := ∥Z∥L2 (Q) ,

49

an := sup p∥Z∥L2 (Q) (t), t≥0

and let tβ denote the deterministic upper β-quantile of S. By the Gaussian approximation and the bias bound, for every deterministic t, P (T ≥ t) ≤ P (S ≥ t − ∆G − ∆bias ) + η, P (T ≥ t) ≥ P (S ≥ t + ∆G + ∆bias ) − η. Let EB be the data event on which the conditional bootstrap approximation (4.3) holds. By the statement preceding (4.3), P (EB ) ≥ 1 − η. On EB , the conditional coupling of B and Z ′ gives, for every s ≥ 0,
P (S ≥ s + ∆B ) ≤ P ∥B∥L2 (Q) ≥ s | D + η,
P ∥B∥L2 (Q) ≥ s + ∆B | D ≤ P (S ≥ s) + η, because Z ′ | D has the same law as Z. Taking s = t̂α in the first inequality and s = tα−η in the second yields tα+η − ∆B ≤ t̂α ≤ tα−η + ∆B . Consequently, P (T ≥ t̂α ) ≤ P (EBc ) + P (T ≥ tα+η − ∆B ) ≤ α + 3η + an ∆, where the last step uses the density bound to compare the Gaussian upper-tail probabilities at tα+η − ∆ and tα+η . Similarly, P (T ≥ t̂α ) ≥ P (T ≥ tα−η + ∆B ) − P (EBc ) ≥ α − 3η − an ∆. Lemma 4.1 gives an ≲

√

τn , which proves the result.

Corollary F.1 (Asymptotic coverage). Under the conditions of Theorem 4.1, let the tuning parameters and probability tolerance depend on n. If ηn → 0

and

√

τn {∆G,n + ∆B,n + ∆bias,n } → 0,

then, for every fixed α ∈ (0, 1),   P f∗ ∈ CSα (fˆ) → 1 − α. Corollary F.1 follows immediately from Theorem 4.1.

F.3

Proof of Proposition 4.1

Lemma F.10. Under Assumption 4.2, when τn ≥ τ0 , we have 1

1

∥(LP w + λI)− 2 f∗ ∥ ≤ max{κ2s−1 , 1} λmin{s− 2 ,0} ∥g∗ ∥.

50

Proof of Proposition 4.1. Applying Lemma F.10 with λ = λn , we obtain √

n ∥fP w − f∗ ∥L2 (Q) =

√

1

2 n λn LQ (LP w + λn I)−1 f∗

√ 1 ≤ C1 n λn (LP w + λn I)− 2 f∗  √ min{s+ 12 ,1} ≤ max κ2s−1 , 1 C1 n λn ∥g∗ ∥H = ∆bias . This proves the proposition.

F.4

Proof of Proposition 4.2

Let ε = (ε1 , . . . , εn )⊤ . The following extension of Bernstein’s inequality for Hilbert–Schmidt operators is useful. Lemma F.11 (Minsker [2017], Theorem 3.1). Let G be a separable Hilbert space, let R > 0, and let {ξi }ni=1 be independent, self-adjoint Hilbert–Schmidt random operators on G such that E[ξi ] = 0

∥ξi ∥ ≤ R

and

a.s.

Suppose there exist V, W > 0 such that n h X 2 i E ξi ≤V i=1 n h X 2 i Tr E ξi

! ≤VW

i=1

Then ∀ t ≥ V 1/2 + R/3, P

n X

! ξi ≥ t

i=1

t2
≤ 4W exp − 2 V + Rt/3

! .

Lemma F.12. For 0 < η < 1 and 0 < λ ≤ 1, with probability at least 1 − η,
1 1 (LP w + λI)− 2 LPnw − LP w (LP w + λI)− 2   s

2 2 τn κ + 1 log(e + Nλ ) τn κ + 1 log(e + Nλ )  log 2 . <C1  + nλ nλ η If further τn → ∞ and λ → 0 as n → ∞, and
τn κ2 + 1 log(e + Nλ ) < 1, nλ then with probability at least 1 − η, s (LP w + λI)

− 12

LPnw − LP w (LP w + λI)

− 12

51

≲

τn κ2 + 1 log(e + Nλ ) 2 log . nλ η

Lemma F.13. For 0 < η < 1 and 0 < λ ≤ 1, if ) ( (τn κ2 + 1) log(e + Nλ ) 1 < C2 := min 1,
2 nλ 4C1 log η2 where C1 is the constant in Lemma F.12, then on the same 1 − η probability event of Lemma F.12, 1

1

(LP w + λI) 2 (LPnw + λI)−1 (LP w + λI) 2 ≤ 2. Lemma F.14. Under Assumption 4.1, for 0 < η < 1 and λ > 0, with probability at least 1 − η, r     1 1 2 τn Nλ 2τn κ ∥(LP w + λI)− 2 max{ k⊤ , √ } x Wε ∥ ≲ 2σ log n η n n λ Proof of Proposition 4.2. For brevity, write λ = λn , w = wn , and Nλ = Nλn throughout this proof. 

 1 ⊤ k Wy − (LP w + λI)−1 LP w f∗ n x   1 ⊤ kx Wε = (LPnw + λI)−1 n

fˆ − fP w = (LPnw + λI)−1

+ (LPnw + λI)−1 LPnw f∗ − (LP w + λI)−1 LP w f∗   1 ⊤ −1 = (LPnw + λI) k Wε n x + (LPnw + λI)−1 LPnw f∗ − (LPnw + λI)−1 LP w f∗   + (LPnw + λI)−1 − (LP w + λI)−1 (LP w f∗ )   1 ⊤ kx Wε = (LPnw + λI)−1 n + (LPnw + λI)−1 (LPnw − LP w )f∗ − (LPnw + λI)−1 (LPnw − LP w )fP w    1 ⊤ −1 k Wε + (LPnw − LP w )(f∗ − fP w ) = (LP w + λI) n x  h i  1  ⊤ −1 −1 kx Wε + (LPnw − LP w )(f∗ − fP w ) + (LPnw + λI) − (LP w + λI) n Denote h

u = (LPnw + λI)

−1

− (LP w + λI)

−1

i  1 n



k⊤ x Wε

 + (LPnw − LP w )(f∗ − fP w ) .

Since 

 n 1 ⊤ 1X kx Wε = w(Xi )εi kXi , n n

LPnw − LP w =

i=1

n
1X w(Xi )kXi ⊗ kXi − LP w , n i=1

P the first term in the preceding decomposition is n−1 ni=1 Ui . Thus 1 fˆ − fP w = n 52

n X i=1

Ui + u.

With the bound

1

1

2 (L w + λI)− 2 ≤ C , LQ 1 P

we can bound ∥u∥L2 (Q) 1 2u ∥u∥L2 (Q) = LQ 1

≤ C1 · (LP w + λI) 2 u . i h (LPnw + λI)−1 − (LP w + λI)−1 = (LPnw + λI)−1 (LP w + λI) − I (LP w + λI)−1 = (LPnw + λI)−1 (LP w − LPnw )(LP w + λI)−1 = (LP w + λI)−1 (LP w − LPnw )(LPnw + λI)−1 . Combining this identity with f∗ − fP w = f∗ − (LP w + λI)−1 LP w f∗ = λ(LP w + λI)−1 f∗ , rewrite 1 2

(LP w + λI) u = (LP w + λI)

− 12

(LP w − LPnw )(LPnw + λI)

−1



 1 ⊤ k Wε n x

1

− (LP w + λI)− 2 (LPnw − LP w )(LPnw + λI)−1 (LPnw − LP w )(f∗ − fP w ) h i 1 1 = (LP w + λI)− 2 (LP w − LPnw )(LP w + λI)− 2 h i 1 1 · (LP w + λI) 2 (LPnw + λI)−1 (LP w + λI) 2  i h 1 1 ⊤ − · (LP w + λI) 2 k Wε n x i h 1 1 − λ (LP w + λI)− 2 (LPnw − LP w )(LP w + λI)− 2 h i 1 1 · (LP w + λI) 2 (LPnw + λI)−1 (LP w + λI) 2 i h 1 1 · (LP w + λI)− 2 (LPnw − LP w )(LP w + λI)− 2 h i 1 · (LP w + λI)− 2 f∗ . Apply the stochastic bounds below with suitably split failure probabilities; this changes their logarithmic factors only by universal constants. Combining Lemmas F.12, F.13, F.14, and F.10, and then using both sample-size conditions in Proposition 4.2, gives, with probability at least 1 − η, s  
1 Nλn log(e + Nλn ) 3 2 2 log ∥u∥L2 (Q) ≲ C1 τn σ κ 1 + ∥g∗ ∥H . n λn η

53

Consequently, √

n

1 X n(fˆ − fP w ) − √ Ui n i=1

=

√

n ∥u∥L2 (Q)

L2 (Q)

s 3

≲ C1 τn σ κ 1 + ∥g∗ ∥H

  Nλn log(e + Nλn ) 2 2 = ∆G,Bah . log λn n η −1/2

Finally, using the definition of Ui , the transfer bound, and ∥(LP w + λn I)−1/2 ∥ ≤ λn

, we obtain

1/2

∥Ui ∥L2 (Q) = ∥LQ Ui ∥
τn C1 2 ≤ √ κ ∥f∗ ∥H + κσ = Ū , λn which is the asserted uniform bound.

F.5

Proof of Proposition 4.3

Let {(νj , ej )}j≥1 be an eigensystem of V in L2 (Q), ordered so that ν1 ≥ ν2 ≥ · · ·, and define Πm :=

m X

ej ⊗L2 (Q) ej .

j=1

Proof of Proposition 4.3. It suffices to prove the bound inside the braces for an arbitrary fixed m, with a Gaussian element Zm that may depend on m. The displayed right-hand side is attained over m ≥ 1; indeed, if Ū > 0, the finite-dimensional coupling term diverges as m → ∞, while if Ū = 0, any m is admissible. Applying the resulting construction at a minimizer gives the stated infimum. The proof follows the same projection-coupling strategy as in Singh and Vijaykumar [2023], with all projections and norms taken in L2 (Q). Let Am : L2 (Q) → Rm ,

Am f = (⟨f, e1 ⟩L2 (Q) , . . . , ⟨f, em ⟩L2 (Q) ),

Pm ∗ so that A∗m z = s=1 zs es and Am Am = Πm . The covariance of Am Ui is the diagonal matrix diag(ν1 , . . . , νm ), and ∥Am Ui ∥Rm ≤ ∥Ui ∥L2 (Q) ≤ Ū . By the finite-dimensional Gaussian coupling theorem of Zaitsev [1987], there exist independent Gaussian vectors ζi ∈ Rm , each having the same covariance as Am Ui , such that   2 log m2 n Ū m X η 1 √ √ {Πm Ui − A∗m ζi } ≲ n n 2 i=1

L (Q)

with probability at least 1 − η/3. Define Zi = A∗m ζi +

∞ X √

νs hi,s es .

s=m+1

Then the Zi ’s are independent centered Gaussian elements in L2 (Q) with covariance V. Set Z = P n−1/2 ni=1 Zi . Then Z ∼ NL2 (Q) (0, V). 54

It remains to control the two tails. Since EUi = 0, n

1 X E √ (I − Πm )Ui n i=1

2

= trL2 (Q) {(I − Πm )V} = σL22 (Q) (m, V). L2 (Q)

The boundedness of Ui and the Hilbert-space Bernstein inequality [Pinelis and Sakhanenko, 1986, Corollary 1] imply, with probability at least 1 − η/3,   s   6 n Ū log X η 1 6 √ √ + (I − Πm )Ui ≲ σL2 (Q) (m, V) log . η n n 2 i=1

L (Q)

Similarly, by Borell’s inequality for the Gaussian element √1n is σL22 (Q) (m, V), with probability at least 1 − η/3,

i=1

i=1 (I −Πm )Zi , whose second moment

s

n

1 X √ (I − Πm )Zi n

Pn

L2 (Q)

  6 . ≲ σL2 (Q) (m, V) log η

Combining these bounds by the triangle inequality and a union bound gives the claim, since  three √ 6 the term Ū log η / n is absorbed by the displayed finite-dimensional coupling term for m ≥ 1.

F.6

Proof of Proposition 4.4

Let

Vi = w(Xi ) Yi − fP w (Xi ) (LP w + λI)−1 kXi ,
V̂i = w(Xi ) Yi − fˆ(Xi ) (LP w + λI)−1 kX . i

n

Let

n

n

V̂i − V̂j 1 XX hij √ , B= n 2 i=1 j=1

Boracle =

n n 1 XX

n

hij

i=1 j=1

Vi − Vj √ . 2

For notational convenience, write LPiw := w(Xi ) kXi ⊗ kXi , LP w ,λ := LP w + λI, LPnw ,λ := LPnw + λI, w

ϵPi := Yi − fP w (Xi ) ϵ̂i := Yi − fˆ(Xi ). Then

w

Vi = w(Xi ) ϵPi L−1 P w ,λ kXi V̂i = w(Xi ) ϵ̂i L−1 P w ,λ kXi . n

Let Eh [·] be the expectation with respect to the random variables {hij }ni,j=1 under the conditional law given D = {(Xi , Yi )}ni=1 . Note that the randomness comes from data and {hij }ni,j=1 . We first find a bound under P(· | D), and then bound it further. 55

Lemma F.15. If we denote  1  w w αij = √ w(Xi ) ϵPi kXi − w(Xj ) ϵPj kXj , 2  1  α̂ij = √ w(Xi ) εbi kXi − w(Xj ) εbj kXj , 2 then B − Bora = ∆inv + ∆res , where


−1  ∆inv = L−1 P w ,λ − LP w ,λ n

  ∆res = L−1 P w ,λ

n n 1 XX

n

n n 1 XX

n

 hij α̂ij  ,

i=1 j=1

 hij (α̂ij − αij ) .

i=1 j=1

Lemma F.16. For 0 < η < 1, under Assumptions 4.1 and 4.2, suppose τn ≥ τ0 for all sufficiently large n, τn → ∞, and λ = λn → 0. If, for all sufficiently large n,   (τn κ2 + 1) log(e + Nλ ) τn , n > max C2 λ λNλ where C2 is the constant in Lemma F.13 after replacing its failure probability by the fixed fraction of η used in the union bound, then, for all sufficiently large n, with probability at least 1 − η, r
2 τn Nλ ˆ ∥f − fP w ∥ ≲ σ κ2s+1 1 + ∥g∗ ∥H log( ), nλ η "r #
1 τn Nλ 2 ∥fˆ − f∗ ∥ ≲ + λmin{s, 2 } σ κ2s+1 1 + ∥g∗ ∥H log( ). nλ η Lemma F.17. For 0 < η < 1, under Assumptions 4.1, 4.2, and 4.4, suppose τn ≥ τ0 for all sufficiently large n, τn → ∞, and λ = λn → 0. If, for all sufficiently large n,   (τn κ2 + 1) log(e + Nλ ) τn n > max , C2 λ λNλ where C2 is the constant in Lemma F.13 after replacing its failure probability by the fixed fraction of η used in the union bound, then, for all sufficiently large n, with probability at least 1 − η over the data, conditionally on the data with probability at least 1 − η, 2s+4

∥∆inv ∥L2 (Q) ≲ C1 σ κ

τn Nλ 1 + ∥g∗ ∥H √ nλ

  3 2 log . η

Lemma F.18. For 0 < η < 1, under Assumptions 4.1, 4.2, and 4.4, suppose τn ≥ τ0 for all sufficiently large n, τn → ∞, and λ = λn → 0. If, for all sufficiently large n,   (τn κ2 + 1) log(e + Nλ ) τn n > max , C2 λ λNλ 56

where C2 is the constant in Lemma F.13 after replacing its failure probability by the fixed fraction of η used in the union bound, then, for all sufficiently large n, with probability at least 1 − η over the data, conditionally on the data with probability at least 1 − η,   2
τn Nλ 2 ∥∆res ∥L2 (Q) ≲ C1 σ κ2s+3 1 + ∥g∗ ∥H √ log . η nλ Proof of Proposition 4.4. Conditional on D, the processes Bora and B are centered Gaussian elements in L2 (Q), because the multipliers hij are independent standard Gaussian variables. The anti-symmetric covariance calculation gives the covariance operators V̂oracle and V̂feas displayed in Proposition 4.4. The high-probability bound for ∥B −Bora ∥L2 (Q) follows by combining Lemmas F.15, F.17, and F.18. In these lemmas, the data-dependent events are intersected first, and the conditional multiplier events are then intersected on that data event. Replacing η in each constituent bound by a fixed fraction of η and applying the union bound changes only the universal constants inside the logarithms.

F.7

Proof of Proposition 4.5

Lemma F.19 (Conditional Strassen–Dudley coupling lemma). The following simplified form is a direct consequence of the conditional Strassen–Dudley theorem of Monrad and Philipp [1991, Theorem 4]. Let (S, d) be a complete separable metric space, let A be a countably generated sub-σ-field, and suppose the underlying probability space supports a [0, 1]-uniform random variable independent of A ∨ σ(X). If X, X ′ , Y ′ are S-valued random elements such that d

X | A = X ′ | A,

P {d(X ′ , Y ′ ) > r} ≤ ε

for some r, ε ≥ 0, then there exists an S-valued random element Y such that d

Y |A=Y′ |A

P {d(X, Y ) > r} ≤ ε.

and

Let {(νj , ej )}j≥1 be the eigensystem of V introduced above, and retain the notation Πm = Pm j=1 ej ⊗L2 (Q) ej . Proof of Proposition 4.5. The proof uses the covariance-comparison and anti-symmetric multiplierbootstrap techniques of Singh and Vijaykumar [2023], but all covariance operators, projections, Hilbert-Schmidt norms, and Gaussian elements below are taken in L2 (Q). First, the anti-symmetric multiplier process has the desired empirical covariance. Since Vi = Ui + µ, where µ = (LP w + λI)−1 LP w (f∗ − fP w ), we have Vi − Vj = Ui − Uj . Conditional on the data, n

n

Vi − Vj 1 XX hij √ n 2 i=1 j=1

is a centered Gaussian element in L2 (Q). Its conditional covariance is     X X Ui − Uj  1 1 Uk − U ℓ   Eh  hij √ ⊗L2 (Q)  hkℓ √ = En [Ui ⊗L2 (Q) Ui ] − En [Ui ] ⊗L2 (Q) En [Ui ] n n 2 2 i,j k,ℓ bora . =V 57

1/2

bora g. Thus the conditional law of the anti-symmetric multiplier process is the same as that of V Since V is trace class, writing gi := g(ei ) for the coordinates of the isonormal process in Proposition 4.5 gives V 1/2 g =

∞ X √

νi gi ei ∈ L2 (Q) a.s.,

h

i E V 1/2 g ⊗L2 (Q) V 1/2 g = V.

i=1

bora . Fix m ≥ 1, and write We next compare the Gaussian elements with covariances V and V Π⊥ m = I − Πm . Define bora − V∥HS,L2 (Q) , ∆HS = ∥V

⊥ b ∆tail = trL2 (Q) {Π⊥ m (Vora − V)Πm } .

The covariance-comparison argument gives a random element G, with the same conditional distri1/2 bora bution as V g, such that conditional on the data, with probability at least 1 − 3η, s   n o 1 1/2 1/2 1/4 1/2 ∥V g − G∥L2 (Q) ≤ {1 + 2 log } m ∆HS + ∆tail + 2σL2 (Q) (m, V) . (F.1) η Indeed, decompose V 1/2 g into its Πm and Π⊥ m parts. On the leading m-dimensional subspace, the Powers–Størmer inequality gives bora Πm )1/2 ∥HS,L2 (Q) ≤ m1/4 ∥V − V bora ∥1/2 2 . ∥(Πm VΠm )1/2 − (Πm V HS,L (Q) ⊥ Borell’s inequality then controls the corresponding Gaussian norm. The Π⊥ m VΠm tail contributes ⊥ b b 1/2 σL2 (Q) (m, V). The remaining Π⊥ m Vora Πm tail is coupled to the full Vora g by Lemma F.19 and is bounded by Borell’s inequality through 1/2

⊥ 1/2 b ∥(Π⊥ ∥HS,L2 (Q) ≤ ∆tail + σL2 (Q) (m, V). m Vora Πm )

This proves (F.1). It remains to bound ∆HS and ∆tail under the target-norm bound on Ui . Hilbert-space Bernstein bounds (Lemma F.11) for the empirical covariance operator imply, with probability at least 1 − 3η,   !1/2   2  Ū 2 σ 2 2 (0, V) 2 Ū  L2 (Q) ∆HS ≲ log + ,  η n n  and

  !1/2    Ū 2 σ 2 2 Ū  2 L2 (Q) (m, V) ∆tail ≲ log + . η  n n 

Substituting these bounds into (F.1), using  ( 2 2 )1/4    1/2 4 Ū σ (0, V) 2 2 Ū L (Q) 1/2 σL2 (Q) (m, V) + , ∆tail ≲ log + 2 η n n   3/2 and absorbing logarithmic factors into log η2 , gives the claim after taking the infimum over m ≥ 1. The total probability is at least 1 − 6η by a union bound over the covariance-estimation event and the conditional Gaussian-coupling event. 58

F.8

Proof of Corollary 4.1

Proof of Corollary 4.1. Fix η ∈ (0, 1), and let ηtol = cη 2 , where c > 0 is a sufficiently small universal constant. In this proof, ∆B,feas and ∆B,coup are evaluated with tolerance ηtol . Write W = Bora ,

W ′ = B,

rfeas = ∆B,feas ,

rcoup = ∆B,coup .

c )≤η , Let Efeas be the data event from Proposition 4.4, evaluated with tolerance ηtol . Then P (Efeas tol and, on Efeas , the conditional failure probability is at most ηtol . Consequently,

 c P ∥W ′ − W ∥L2 (Q) > rfeas ≤ P (Efeas ) + E 1Efeas P ∥W ′ − W ∥L2 (Q) > rfeas D

≤ 2ηtol . Next, the identity Vi = Ui + µ, with µ = (LP w + λI)−1 LP w (f∗ − fP w ), implies Vi − Vj = Ui − Uj . Hence, conditional on D, the oracle anti-symmetric bootstrap W is centered Gaussian with covariance bora = En [Ui ⊗L2 (Q) Ui ] − En [Ui ] ⊗L2 (Q) En [Ui ]. V 1/2

bora g. By Proposition 4.5, applied with tolerance ηtol , there is Thus W | D has the same law as V a Gaussian element Z0 ∼ NL2 (Q) (0, V) and a random element G, with G | D distributed as W | D, such that
P ∥Z0 − G∥L2 (Q) > rcoup ≤ 6ηtol . By Lemma F.19, applied to the regular conditional laws given D, we may replace the copy G by the actual oracle bootstrap W : there exists a copy Z with Z | D ∼ NL2 (Q) (0, V) such that
P ∥Z − W ∥L2 (Q) > rcoup ≤ 6ηtol . Gluing this coupling with the original pair (W, W ′ ) and using the triangle inequality gives
P ∥Z − W ′ ∥L2 (Q) > rcoup + rfeas ≤ 8ηtol . Let
A(D) = P ∥Z − W ′ ∥L2 (Q) > rcoup + rfeas D . Then EA(D) ≤ 8ηtol . Markov’s inequality gives P {A(D) > η} ≤

8ηtol . η

Choosing c ≤ 1/8 yields 8ηtol /η ≤ η. Therefore, with probability at least 1 − η over the data,
P ∥B − Z∥L2 (Q) ≤ ∆B,feas + ∆B,coup D ≥ 1 − η. Renaming Z as Z ′ proves the corollary. 59

F.9

Proof of Corollary 4.2

Lemma F.20. Suppose that the eigenvalues of LQ satisfy µk ≤ αk −β for all k ≥ 1, where α > 0 and β > 1. Then, for every λ > 0, Nλ ≤ α1/β

β λ−1/β . β−1

Lemma F.21. Suppose that the eigenvalues of LQ satisfy µk ≤ αk −β for all k ≥ 1, where α > 0 and β > 1. Then, for every m ∈ N+ ,  σ(m, LQ ) ≤

α β−1

1/2

m1/2−β/2 .

Lemma F.22. For β > 0 and γ > 0, R∞

exp(−βxγ ) dx = 1. a→∞ exp(−βaγ )/(βγaγ−1 ) lim

a

Lemma F.23. Suppose that the eigenvalues of LQ satisfy µk ≤ α exp(−βk γ ) for all k ≥ 1, where α > 0, β > 0, and γ > 0. Then there exist λ0 ∈ (0, 1) and Cα,β,γ > 0 such that, for every 0 < λ ≤ λ0 , Nλ ≤ Cα,β,γ {log(1/λ)}1/γ . Lemma F.24. Suppose that the eigenvalues of LQ satisfy µk ≤ α exp(−βk γ ) for all k ≥ 1, where α > 0, β > 0, and γ > 0. Then there exists Cα,β,γ > 0 such that, for every m ∈ N+ , σ(m, LQ ) ≤ Cα,β,γ m1/2−γ/2 exp(−βmγ /2). Lemma F.25. Under Assumptions 4.1 and 4.2, when τn ≥ τ0 , for every m ∈ N+ and λ > 0, we have  τn  2 σL22 (Q) (m, V) ≤ σ + κ2 max{κ2s−1 , 1}2 λ2 min{s,1/2} ∥g∗ ∥2 σ 2 (m, LQ ). λ Proposition F.1. Suppose µk ≤ αk −β for some α > 0 and β > 1. Under Assumptions 4.1, 4.2, and 4.4, when τn ≥ τ0 , if τn → ∞, λ → 0, and   (τn κ2 + 1) log(e + Nλ ) τn n > max , , C2 λ λNλ where C2 is the constant in Lemma F.13 after replacing its failure probability by the fixed fraction of η used in the union bound, then with probability at least 1 − η, there exist i.i.d. Gaussian elements Zi ∼ NL2 (Q) (0, V) such that n √ 1 X n(fˆ − fP w ) − √ Zi n i=1 L2 (Q)     2 r   β−1   τn −1/2−1/(2β) 2 1 2 τn τn 2(β+3) ≲√ λ log log + log(n) log . λ η λ n η n

60

Proof of Proposition F.1. Apply Propositions 4.2 and 4.3 with a fixed fraction of the tolerance η. By Lemma F.20, Nλ ≲ λ−1/β . Consequently, the Bahadur remainder satisfies     2 1 2 τn − 12 − 2β 1 log . ∆G,Bah ≲ √ λ log λ η n Lemmas F.25 and F.21 give r σL2 (Q) (m, V) ≲

τn − β−1 m 2 . λ

√ Moreover, Proposition 4.2 gives Ū ≲ τn / λ. Hence Proposition 4.3 yields r ∆G,coup ≲ inf

m≥1

τn − β−1 m 2 λ

Take



 1/2  2 2 τn m 2 log + √ m log . η η nλ

& m=

n τn

1/(β+3) ' .

The sample-size condition implies n/τn → ∞, and substitution gives r   β−1   τn τn 2(β+3) 2 ∆G,coup ≲ log(n) log . λ n η The triangle inequality and a union bound combine the Bahadur and coupling events. Finally, P the coupled Gaussian element can be represented as n−1/2 ni=1 Zi by adjoining an independent Gaussian bridge, with the Zi ’s i.i.d. with covariance V. This proves the claim. Proposition F.2. Suppose µk ≤ αk −β for some α > 0 and β > 1. Under Assumptions 4.1, 4.2, and 4.4, together with Definition 4.1, when τn ≥ τ0 , if τn → ∞, λ → 0, and   (τn κ2 + 1) log(e + Nλ ) τn n > max , , C2 λ λNλ where C2 is the constant in Lemma F.13 evaluated at the component tolerance ηtol = cη 2 used in Corollary 4.1, then, on a data event of probability at least 1 − η, there exists a coupling of B and Z, conditionally on D, such that Z | D ∼ NL2 (Q) (0, V). On this event, let  3 2 τ √n λ−1/2−1/β RB,poly := log η n ! β−1   3/2 r 2 τn τn λ1−1/β 2(2β−1) + log . η λ n 

Then  P ∥Z − B∥L2 (Q) ≲ RB,poly D ≥ 1 − η.

61

Proof of Proposition F.2. Apply Corollary 4.1. By Proposition 4.4 and Lemma F.20,   3 1 1 τ 2 √n λ− 2 − β . ∆B,feas ≲ log η n Also, Proposition 4.2 and Lemma F.7 imply Ū 2 trL2 (Q) (V) Ū 4 τ 3 Nλ τ4 + 2 ≲ n + 2n 2 . n n nλ n λ The condition n > τn /(λNλ ) ensures that the second term is bounded by the first. Therefore, Proposition 4.5 and Lemmas F.25, F.20, and F.21 give !1/4 r     3/2 3 λ−1/β τ τn − β−1 2 n 1/4 2 inf m + . ∆B,coup ≲ log m m≥1 η nλ λ Choose

& m=

n

1/(2β−1) ' .

τn λ1−1/β

Both terms inside the braces are then bounded by r

τn λ

τn

λ1−1/β n

!

β−1 2(2β−1)

.

Combining the feasible and coupling bounds in Corollary 4.1, with tolerances adjusted by universal constants, proves the stated outer and conditional probability claim. Proposition F.3. Suppose µk ≤ α exp(−βk γ ) for some α, β, γ > 0. Under Assumptions 4.1, 4.2, and 4.4, when τn ≥ τ0 , if τn → ∞, λ → 0, and   (τn κ2 + 1) log(e + Nλ ) τn n > max , , C2 λ λNλ where C2 is the constant in Lemma F.13 after replacing its failure probability by the fixed fraction of η used in the union bound, then with probability at least 1 − η, there exist i.i.d. Gaussian elements Zi ∼ NL2 (Q) (0, V) such that n

√

1 X n(fˆ − fP w ) − √ Zi n i=1 L2 (Q)        #  " τn 1 n n 1/(2γ) 1 2/γ 2 2 ≲ √ log log log log + log log log . η λ λ τ τ nλ n n

Proof of Proposition F.3. Apply Propositions 4.2 and 4.3 with a fixed fraction of η. Lemma F.23 gives   1/γ 1 Nλ ≲ log , λ and hence  ∆G,Bah ≲

 2   1/(2γ)   2 τn 1 1 √ log log log log . η λ λ nλ 62

Lemmas F.25 and F.24 imply r σL2 (Q) (m, V) ≲

  τn 1 − γ βmγ . m 2 2 exp − λ 2

√ Together with Ū ≲ τn / λ, this yields r  2   τn 1 − γ −βmγ /2 τn m 2 2 ∆G,coup ≲ inf + √ m log log . m2 2 e m≥1 λ η η nλ Take

& m=

 1/γ ' 1 n . log β τn

The sample-size condition makes this choice well defined for all sufficiently large n. Since m1/2−γ/2 ≤ 1 + m2 , substitution gives   2/γ     τn n 2 n ∆G,coup ≲ √ log log . log log τn τn η nλ The triangle inequality, a union bound, and the Gaussian-bridge representation used in the polynomial case complete the proof. Proposition F.4. Suppose µk ≤ α exp(−βk γ ) for some α, β, γ > 0. Under Assumptions 4.1, 4.2, and 4.4, together with Definition 4.1, when τn ≥ τ0 , if τn → ∞, λ → 0, and   (τn κ2 + 1) log(e + Nλ ) τn n > max , , C2 λ λNλ where C2 is the constant in Lemma F.13 evaluated at the component tolerance ηtol = cη 2 used in Corollary 4.1, then, on a data event of probability at least 1 − η, there exists a coupling of B and Z, conditionally on D, such that Z | D ∼ NL2 (Q) (0, V) and (   3   1/γ 2 τn 1 √ P ∥Z − B∥L2 (Q) ≲ log log η λ nλ )   3/2  3 1/4   1/(4γ)   1/(4γ) 2 τn 1 n + log log log D ≥ 1 − η. η nλ λ τn λ Proof of Proposition F.4. Apply Corollary 4.1. Proposition 4.4 and Lemma F.23 give   3   1/γ 2 τn 1 √ ∆B,feas ≲ log . log η λ nλ As in the polynomial case, the sample-size condition and Lemma F.7 imply Ū 2 trL2 (Q) (V) Ū 4 τ 3 Nλ + 2 ≲ n . n n nλ Thus Proposition 4.5 and Lemmas F.25, F.23, and F.24 yield   3/2   3 1/4   1/(4γ) 2 τn 1 ∆B,coup ≲ log inf m1/4 log m≥1 η nλ λ r  τn 1 − γ −βmγ /2 + m2 2 e . λ 63

For a sufficiently large fixed constant C > 1/(2β), take & 1/γ '  n . m= C log τn λ The first term in braces is bounded by 

τn3 nλ

1/4   1/(4γ)   1/(4γ) 1 n log log . λ τn λ

Because Cβ/2 > 1/4, the exponential factor in the spectral-tail term absorbs its polynomial factor, so that term is bounded by the same display for all sufficiently large n. Combining the coupling and feasible-bootstrap bounds proves the result. Proof of Corollary 4.2. Let ρ̄ < ∞ be an almost-sure upper bound for the density ratio and choose C0 ≥ max{ρ̄, τ0 }. Then wn = ρ, LP w = LQ , and Assumption 4.4 holds with C1 = 1. A bounded density ratio also satisfies Definition 4.1: one may take γ = 1 and W = σ 2 = max{ρ̄, 1}. We next justify the use of the spectral approximation bounds with a fixed envelope. Under either two-sided spectral condition in the corollary, Nλ → ∞ as λ → 0. Since τn = C0 , the first, fully nonasymptotic inequality in Lemma F.12 implies the same simplified Bahadur bound used in Proposition 4.2: the linear term is absorbed by the square-root term under the stated sample-size condition, while τn κ2 + 1 ≲ C0 and log(e + Nλ ) ≍ log Nλ for all sufficiently large n. The remaining Gaussian and bootstrap arguments use only this bound, the fixed truncated-weight envelope, and their displayed sample-size conditions. Thus their nonasymptotic conclusions, and hence the specialized spectral bounds, remain valid for τn = C0 . First consider polynomial spectral decay. At η ≍ n−1 , Proposition 4.1 and Propositions F.1 and F.2 give
∆ ≲ log3 n ∆0 + ∆1 + ∆2 + ∆3 , (F.2) where ∆0 =

√

n λ,

β−1 − 2(β+3)

∆1 = n

1

1

− 12 − β1

∆2 = n− 2 λ

β−1 − 2(2β−1)

∆3 = n

β+1

λ− 2 τnβ+3 , τn , 2

β +β−1 − 2β(2β−1)

λ

3β−2

τn2(2β−1) .

Here the Gaussian Bahadur term τn n−1/2 λ−1/2−1/(2β) is absorbed by ∆2 , since their ratio is λ1/(2β) → 0. The factor log3 n in (F.2) is obtained directly from the component bounds: the Gaussian Bahadur and coupling terms contribute at most log3 n and log2 n, respectively, whereas the feasible-bootstrap and bootstrap-coupling terms contribute at most log3 n and log3/2 n, respectively. Write λ = n−a . Since τn = C0 , the four powers of n in (F.2) are   1 β−1 a 1 1 1 β−1 β2 + β − 1 δ0 = − a, δ1 = − + , δ2 = − + a + , δ3 = − +a . 2 2(β + 3) 2 2 2 β 2(2β − 1) 2β(2β − 1) The exponent δ0 is decreasing in a, whereas δ1 , δ2 , δ3 are increasing. The balance δ0 = δ2 gives a = 2β/(3β + 2), but at this value δ1 > δ0 and δ3 > δ0 , so it cannot be optimal. 64

Balancing δ0 = δ1 gives a01 =

2(β + 1) , 3(β + 3)

A01 =

−β + 5 . 6(β + 3)

At this value, δ2 < A01 , and δ1 − δ 3 = −

(β − 1)(β 2 − 12β + 2) . 6β(2β − 1)(β + 3)

Balancing δ0 = δ3 gives a03 =

β(3β − 2) , 5β 2 − β − 1

A03 =

−β 2 + 3β − 1 . 2(5β 2 − β − 1)

At this value, δ2 < A03 , and (β − 1)(−β 2 + 12β − 2) . 2(β + 3)(5β 2 − β − 1)

δ1 − δ 0 = Consequently, with β0 = 6 +

√

34, the sharp optimizer is   2(β + 1) β − 5   , 5 < β ≤ β0 ,  3(β + 3) 6(β + 3) , ♯ ♯  (aβ , rβ ) =   β 2 − 3β + 1 β(3β − 2)   , , β > β0 . 5β 2 − β − 1 2(5β 2 − β − 1)

The switch occurs when β 2 − 12β + 2 = 0. Thus the sharper, piecewise choice a = a♯β yields ♯

∆ ≲ n−rβ log3 n and, by Theorem 4.1, ♯

P{f∗ ∈ / CSα (fb)} − α ≲ n−1 + n−rβ log3 n. For the non-piecewise statement in the corollary, instead set e aβ =

2(β + 1) , 3(β + 3)

reβ =

(β − 5)(β + 10) . 6(β + 3)(2β + 5)

At a = e aβ , the four exponents simplify to δ0 = δ1 = −

β−5 , 6(β + 3)

and δ3 = −

δ2 = −

(β − 1)(β + 4) , 6β(β + 3)

(β − 2)(β 2 + 4β − 1) . 6β(β + 3)(2β − 1)

Writing x = β − 5 > 0, direct calculation gives −δ0 − reβ = −δ1 − reβ =

65

(β − 5)2 ≥ 0, 6(β + 3)(2β + 5)

x3 + 21x2 + 192x + 540 > 0, 6β(β + 3)(2β + 5)

−δ2 − reβ = and −δ3 − reβ =

97x2 + 879x + 1980 > 0. 6β(β + 3)(2β − 1)(2β + 5)

Consequently, δj ≤ −e rβ for j = 0, 1, 2, 3, and hence ∆ ≲ n−erβ log3 n. √ Because τn = O(1), Theorem 4.1 with η ≍ n−1 proves part (1). Under exponential spectral decay, Proposition 4.1 and Propositions F.3 and F.4 give, after suppressing only their displayed logarithmic and tolerance factors, the polynomial-order envelope ∆≲

√

τn nλ + √ + nλ



τn3 nλ

1/4 .

(F.3)

Since τn = C0 , its three polynomial-order contributions are √

nλ,



1 √ , nλ

1 nλ

1/4 .

The active balance is between the first and third terms, giving λ ≍ n−3/5 and the power n−1/10 . At η ≍ n−1 , the logarithmic multiplier of the active bootstrap coupling term in Proposition F.4 is 3

(log n)3/2 (log(1/λ))1/(4ζ) (log(n/λ))1/(4ζ) ≲ (log n) 2

1 + 2ζ

.

The bias term has the same power without this logarithmic multiplier, and all remaining stochastic terms have a strictly smaller power of n. A final application of Theorem 4.1 proves part (2).

F.10

Proof of Corollary 4.3

Lemma F.26 (Cordes inequality; see Cordes [1987]). Let A and B be bounded, self-adjoint, positive semidefinite operators on a Hilbert space G. Then, for every s ∈ [0, 1], ∥As B s ∥G ≤ ∥AB∥sG . The positive-semidefinite case follows by applying the strictly positive version to A + εI and B + εI and letting ε ↓ 0. Lemma F.27 (Gogolashvili et al. [2023], Lemma 21). Under Definition 4.1, for every integer m ≥ 2 and τn > 0,    1/γ 1 −(m−1) w(X) 2 EQ 1 − ≤ τn m!W m−2 σ 2 . ρ(X) 2 Lemma F.28 (Gogolashvili et al. [2023], Lemma 22). Under Definition 4.1, for every integer m ≥ 2, τn > 0, and λ > 0, −1

(LQ − LP w )(LQ + λI)

−1/2

≤ κλ

1/2 Nλ

66



1 −(m−1) τ m!W m−2 σ 2 2 n

1/(2γ) .

Lemma F.29 (Gogolashvili et al. [2023], Lemma 23). Under Assumption 4.3 and Definition 4.1, if, for some integer m0 ≥ 2 and 0 < λ ≤ 1,  1/(m0 −1) (1+d)γ − λ m0 −1 , τn ≥ 22γ−1 κ2γ Ed2γ m0 !W m0 −2 σ 2 then, as operators on H, 1. ∥(LQ − LP w )(LQ + λI)−1 ∥ ≤ 1/2; 2. ∥LQ (LP w + λI)−1 ∥ ≤ 2; 1/2

3. ∥LQ (LP w + λI)−1/2 ∥ ≤

√

2.

Proof of Corollary 4.3. Let τn = nδ for a fixed δ > 0, and consider a tuning parameter λ = n−a . The polynomial spectral condition verifies Assumption 4.3 with d = 1/β, while the exponential condition does so with any fixed d ∈ (0, 1]. Choose a fixed integer m0 ≥ 2 such that m0 − 1 > It follows that λ

(1+d)γ 0

− m −1

a(1 + d)γ . δ

a(1+d)γ

= n m0 −1 = o(nδ ).

√ Consequently, for all sufficiently large n, Lemma F.29 verifies Assumption 4.4 with C1 ≤ 2. We may also write τn = λ−ϱ , where ϱ = δ/a. Because the Bernstein-type condition holds for every integer m0 ≥ 2, this argument is available for every fixed δ > 0. Taking δ sufficiently small also ensures all sample-size restrictions in the relevant spectral approximation propositions, including a + δ < 1. For polynomial spectral decay, take a=e aβ =

2(β + 1) , 3(β + 3)

reβ =

(β − 5)(β + 10) . 6(β + 3)(2β + 5)

Restoring τn = nδ in the four terms of (F.2), their powers of n are δ0 ,

δ1 + δ

β+1 , β+3

δ2 + δ,

δ3 + δ

3β − 2 , 2(2β − 1)

where the preceding proof shows that δj ≤ −e rβ for j = 0, 1, 2, 3. Both fractional truncation coefficients in the display are smaller than one. Hence ∆ ≲ n−erβ +δ log3 n. The factor nδ is a uniform upper bound for the truncation loss of every term; the active terms have a strictly smaller truncation power. Thus any remaining fixed logarithmic factors are absorbed by this polynomial slack. Theorem 4.1 now yields 3 P{f∗ ∈ / CSα (fb)} − α ≲ n−1 + n−erβ + 2 δ log3 n.

For part (1), choose δ=

β−5 , 9(β + 3)(2β + 5) 67

β−5

τn = n 9(β+3)(2β+5) .

Then

3 (β − 5)(β + 9) reβ − δ = , 2 6(β + 3)(2β + 5)

and hence the preceding display gives the bound in part (1). For exponential spectral decay, take a = 3/5. Substituting λ = n−3/5 and τn = nδ into √ (F.3), and then multiplying the approximation error by the anti-concentration factor τn , gives the conservative bound 3

1

P{f∗ ∈ / CSα (fb)} − α ≲ n−1 + n− 10 + 2 δ log3 n. √ Indeed, after multiplication by τn , the active coupling term has truncation power 5δ/4, leaving a polynomial margin of δ/4 that absorbs the fixed ζ-dependent logarithmic factors. Substituting δ=

1 , 150

τn = n1/150 ,

and using 1 3 9 − δ= 10 2 100 proves part (2). The two selected values of δ satisfy the sufficient positivity conditions 0<δ<

(β − 5)(β + 10) 9(β + 3)(2β + 5)

and

0<δ<

1 , 15

respectively. These upper bounds are also smaller than 1 − a for the corresponding choices of a, and hence imply a + δ < 1.

F.11

Proof of Corollary D.1

Definition F.1 (Bounded ϑ-th moment condition). There exist ϑ > 2 and Ωϑ < ∞ such that i i h h ϑ−1 ϑ ≤ Ωϑ . EP ρ(X) = EQ ρ(X) Lemma F.30. Under Definition F.1, for τn > 0 and λ > 0, 1/2

1/2

(LQ − LP w )(LQ + λI)−1 ≤ κλ−1/2 Nλ τn−(ϑ−1)/2 Ωϑ . Lemma F.31. Under Assumption 4.3 and Definition F.1, for 0 < λ ≤ 1, if  1/(ϑ−1) τn ≥ 4κ2 Ωϑ Ed2 λ−(1+d) , then, as operators on H, 1. ∥(LQ − LP w )(LQ + λI)−1 ∥ ≤ 1/2; 2. ∥LQ (LP w + λI)−1 ∥ ≤ 2; 1/2

3. ∥LQ (LP w + λI)−1/2 ∥ ≤

√

2.

68

(F.4)

The following compatibility reduction is common to Corollaries D.1 and 4.4. Set qθ = θ − 1. Then qθ > 1. Since dQ = ρ dP, the moment condition in either corollary is Definition F.1 with ϑ = θ. Hence, by Lemma F.30, 1/2

(LQ − LP w )(LQ + λI)−1 ≲ λ−1/2 Nλ τn−qθ /2 . Thus the perturbation proof of Lemma F.31 verifies Assumption 4.4 whenever τn ≳ (λ−1 Nλ )1/qθ .

(F.5)

Once this compatibility condition holds, the proofs of the Gaussian and bootstrap approximation propositions use only the bounded truncated-weight envelope and the standing assumptions. In particular, the Bernstein-type condition appearing in the statements of the specialized bootstrap propositions is not used in their proofs. Repeating those proofs from Propositions 4.4 and 4.5, together with the corresponding effective-dimension and spectral-tail lemmas, yields the same spectral envelopes under (F.5). Proof of Corollary D.1. Suppose that µk ≍ k −β . Lemma F.20 gives Nλ ≲ λ−1/β , so we may take d = 1/β in Assumption 4.3. Set β+1 vβ,θ = . βqθ For λ = n−a , because nδ → ∞, the truncation level τn ≍ navβ,θ +δ satisfies (F.5) for all sufficiently large n. Setting δ = 0, after multiplying the four terms in (F.2) by √ the anti-concentration factor τn , their powers of n are avβ,θ 1 −a+ , 2 2   β−1 a β+1 1 (0) e1 (a) = − + + avβ,θ + , 2(β + 3) 2 β+3 2   1 1 1 3 (0) + e2 (a) = − + a + avβ,θ , 2 2 β 2   2 β−1 β +β−1 3β − 2 1 (0) e3 (a) = − +a + avβ,θ + . 2(2β − 1) 2β(2β − 1) 2(2β − 1) 2 (0)

e0 (a) =

For reference, the exact optimizer of these four exponents remains piecewise. Let √ β0 = 6 + 34, D1 = 3βqθ (β + 3) + 2(β + 1)2 , D2 = qθ (5β 2 − β − 1) + (β + 1)(3β − 2). Then

   2βqθ (β + 1) , 5 < β ≤ β0 ,  D1 ♯ aβ,θ = βq (3β − 2)    θ , β > β0 , D2 69

 2(β + 1)2    , 5 < β ≤ β0 , D1 t♯β,θ = a♯β,θ vβ,θ = (β + 1)(3β − 2)    , β > β0 , D2  βqθ (β − 5) − 4(β + 1)2   , 5 < β ≤ β0 ,  2D1 ♯ rβ,θ =  q (β 2 − 3β + 1) − 2(β + 1)(3β − 2)   θ , β > β0 . 2D2 (0)

(0)

(0)

(0)

Indeed, the first branch balances e0 = e1 , the second balances e0 = e3 , and the crossover equation is β 2 − 12β + 2 = 0. These sharper piecewise quantities are retained only for comparison and are not needed in Corollary D.1. For the non-piecewise statement, put D = 3βqθ (β + 3) + 2(β + 1)2 ,

aM =

2βqθ (β + 1) . D

Thus

2(β + 1)2 + δ, D which is the truncation power used in Corollary D.1. At δ = 0, define λ = n−aM ,

τn ≍ nup ,

r01 = r03 =

up = aM vβ,θ + δ =

βqθ (β − 5) − 4(β + 1)2 , 2D

qθ (β 3 + 2β 2 − 9β + 2) − 4(β + 1)2 (2β − 1) 2(2β − 1)D

and rM =

βqθ (β − 5)(β + 10) − 4(β + 1)2 (2β + 5) . 2(2β + 5)D

Direct substitution gives (0)

(0)

e0 (aM ) = e1 (aM ) = −r01 ,

(0)

(0)

e2 (aM ) − e0 (aM ) = −

2qθ (2β − 1) < 0, D

(0)

and e3 (aM ) = −r03 . Moreover, r01 − rM = r03 − rM =

qθ β(β − 5)2 > 0, 2(2β + 5)D

qθ (97β 2 − 91β + 10) > 0. 2(2β − 1)(2β + 5)D

Hence every exponent is at most −rM . Restoring the factor nδ in τn adds, respectively, δ , 2

3β + 5 δ, 2(β + 3)

3δ , 2

5β − 3 δ 2(2β − 1)

to the four coverage exponents. All four coefficients are at most 3/2, so Theorem 4.1 gives P{f∗ ∈ / CSα (fb)} − α ≲ n−1 + n−{rM −3δ/2} log3 n. 70

The displayed expression for rM is the polynomial rate before fixing the truncation slack. Its positivity is equivalent to 4(β + 1)2 (2β + 5) qθ > , β(β − 5)(β + 10) which is assumed in Corollary D.1. Now choose δ=

2rM βqθ (β − 5)(β + 10) − 4(β + 1)2 (2β + 5) = . 3(2β + 5)(β + 10)D 3(β + 10)

Since β + 10 > 1, this choice lies strictly between zero and the admissible upper bound 2rM /3. Substitution gives up =

βqθ (β − 5)(β + 10) + 2(3β + 28)(β + 1)2 (2β + 5) 3(2β + 5)(β + 10)D

and, writing the resulting coverage exponent as rp , rp := rM −

3δ β+9 (β + 9){βqθ (β − 5)(β + 10) − 4(β + 1)2 (2β + 5)} = rM = , 2 β + 10 2(β + 10)(2β + 5)D

which are the truncation power and coverage exponent stated in Corollary D.1. Moreover, the assumed moment threshold and the definition of D imply 0 < rp <

(β + 9)(β − 5) < 1. 6(2β + 5)(β + 3)

Consequently, the n−1 term in the preceding coverage bound is absorbed by n−rp log3 n. It remains to verify the listed transparent sufficient regimes. The numerator of rM can be written as (θ − 9)β(β − 5)(β + 10) + 4(β 2 − 112β − 5). It is therefore positive when θ ≥ 9 and β ≥ 113. For the flexible regime, put R(β) =

4(β + 1)2 (2β + 5) . β(β − 5)(β + 10)

For every β > 5, (β − 5){R(β) − 8} =

4(−β 2 + 112β + 5) 144 ≤ ; β(β + 10) 5

the rational function in the middle is decreasing on (5, ∞). Consequently, θ > 9 and β > 5 + 144/{5(θ − 9)} imply R(β) < θ − 1. Finally, the same numerator is increasing in both parameters on each of {θ ≥ 10, β ≥ 20} and {θ ≥ 11, β ≥ 15}, and its values at the respective lower corners are 1620 and 1660. This proves the remaining two sufficient regimes. Finally, direct substitution of the selected polynomial slack gives 1 − a M − up =

2βqθ (β + 10)(3β 2 + 28β + 55) + 4(β + 1)2 (2β + 5) > 0. 3(2β + 5)(β + 10)D

Under the assumed two-sided polynomial spectral decay, Lemma F.2 gives Nλ ≍ λ−1/β . Together with the strict margin above, this implies τn /(nλNλ ) → 0; the remaining fixed logarithmic factors are absorbed by the same margin. Thus all sample-size conditions in the relevant approximation propositions hold for sufficiently large n, completing the proof. 71

F.12

Proof of Corollary 4.4

Proof. Suppose that µk ≍ exp(−ck ζ ), and write qθ = θ − 1,

a=

3qθ , 5qθ + 3

λ = n−a .

By Lemma F.23, Nλ ≲ {log(1/λ)}1/ζ . Choose a sufficiently large fixed constant C > 0 and set h i1/qθ τn = C λ−1 {log(1/λ)}1/ζ . Then (F.5) holds and 1

3

τn ≍ n 5θ−2 (log n) ζ(θ−1) , as stated in the corollary. √ After multiplying the three terms in (F.3) by the anti-concentration factor τn , their polynomial powers of n are e0 =

1 a −a+ , 2 2qθ

1 a 3a e1 = − + + , 2 2 2qθ

Define rθ =

e2 =

a−1 5a + . 4 4qθ

qθ − 6 θ−7 = . 2(5qθ + 3) 2(5θ − 2)

Direct substitution gives e0 = e2 = −rθ ,

e 2 − e1 =

qθ > 0. 2(5qθ + 3)

Thus the bias and bootstrap-coupling terms have the same polynomial order, while the middle term is polynomially smaller. The bias term carries the factor (log n)1/{2ζqθ } . The bootstrap-coupling term carries 3 +1+ 5 (log n) 2 2ζ 4ζqθ , 5/4

where the final exponent is contributed by the factor τn . Hence the bootstrap-coupling term dominates logarithmically, and the fixed logarithmic factors of the polynomially smaller terms are absorbed. Theorem 4.1 consequently yields 3

1

5

3

1

5

+ + + + P{f∗ ∈ / CSα (fb)} − α ≲ n−1 n−rθ (log n) 2 2ζ 4ζqθ ≲ n−rθ (log n) 2 2ζ 4ζqθ ,

where 0 < rθ < 1 because θ > 7. Finally, 1−a−

a 2qθ = > 0. qθ 5qθ + 3

Hence both sample-size restrictions in the relevant approximation propositions hold for all sufficiently large n; the strict polynomial margin absorbs the additional logarithmic factors in τn and in the failure-probability tolerances. This completes the proof. 72

F.13

Proof of Theorem 4.2

Proof of Theorem 4.2. The type II error probability is   √  t̂ α . P n∥fˆ − f0 ∥L2 (Q) < t̂α = P ∥fˆ − f0 ∥L2 (Q) < √ n By the triangle inequality, ∥f∗ − f0 ∥L2 (Q) ≤ ∥fˆ − f0 ∥L2 (Q) + ∥fˆ − f∗ ∥L2 (Q) . Hence 

t̂α ∥fˆ − f0 ∥L2 (Q) < √ n



 ⊆

t̂α δ < ∥fˆ − f∗ ∥L2 (Q) + √ n

 .

Since δ > 0 is fixed and t̂α P ∥fˆ − f∗ ∥L2 (Q) + √ − → 0, n the probability of the event on the right-hand side tends to zero. Therefore √  P n∥fˆ − f0 ∥L2 (Q) < t̂α → 0, which is equivalent to P

F.14

√

 n∥fˆ − f0 ∥L2 (Q) > t̂α → 1.

Proof of Proposition A.1

Proof. Fix η ∈ (0, 1) and an integer m ∈ {1, . . . , n}. Again let Am : L2 (Q) → Rm be the coordinate projection onto the first m eigenfunctions of V. Since Ui is L2 (Q)-b-sub-Gaussian, Am Ui is b-subGaussian in Rm , and its covariance is bounded in operator norm by ∥V∥L2 (Q) . If this covariance is singular, apply the finite-dimensional sub-Gaussian coupling theorem of Buzun et al. [2022] on the range of Πm VΠm , whose dimension is at most m and whose covariance is nonsingular. Every zerovariance coordinate of the centered vector Am Ui vanishes almost surely, so the resulting Gaussian vector can be extended by zero on the orthogonal complement. Thus, in either case, the theorem gives a Gaussian vector ζ ∈ Rm with covariance diag(ν1 , . . . , νm ) such that, with probability at least 1 − η/3, n 1 X m3/2 log(n)2 1/2 √ √ Πm Ui − A∗m ζ ≲ ∥V∥L2 (Q) b3 (3/η)1/ log(mn) . n n 2 i=1

L (Q)

Let Z = A∗m ζ +

∞ X √

νs hs es ,

s=m+1

where (hs )s>m are independent standard Gaussian variables, independent of the data and of ζ. Then Z ∼ NL2 (Q) (0, V). For the empirical tail, the L2 (Q)-sub-Gaussianity of Ui gives the moment bound n o1/p √ E∥(I − Πm )Ui ∥pL2 (Q) ≲ b p σL2 (Q) (m, V), 73

p ≥ 2.

The Hilbert-space Bernstein inequality [Pinelis and Sakhanenko, 1986, Corollary 1] therefore yields, with probability at least 1 − η/3,   s   6 n bσ (m, V) log 2 L (Q) η 1 X 6 √ √ (I − Πm )Ui ≲ bσL2 (Q) (m, V) log . + η n n 2 i=1

L (Q)

The last term is dominated by the first when n ≥ log

  2 η

. Finally, Borell’s inequality applied to

(I − Πm )Z, whose second moment is σL22 (Q) (m, V), gives s ∥(I − Πm )Z∥L2 (Q) ≲ σL2 (Q) (m, V) log

  6 η

with probability at least 1 − η/3. A union bound and the triangle inequality complete the proof.

F.15

Proof of Proposition A.2

bora , and the Proof. The anti-symmetric multiplier identity gives the oracle empirical covariance V covariance-comparison bound (F.1) applies with this empirical covariance. Since adding a common deterministic element does not change an empirical covariance, ! ! n n n X X X 1 1 1 bora − V = V (Ui ⊗L2 (Q) Ui − V) − Ui ⊗L2 (Q) Ui . n n n i=1

i=1

i=1

We first make the covariance-estimation step explicit. For every orthogonal projection P on L2 (Q) and every p ≥ 2, the standard moment bound for a Hilbert-space sub-Gaussian random element gives n o1/p √  1/2 . E∥P Ui ∥pL2 (Q) ≲ b p trL2 (Q) (P VP ) This follows by applying the sub-Gaussian increment bound to finite-dimensional projections and then passing to the limit by monotone convergence. Indeed, on each finite-dimensional range, Gaussian integration and Definition A.1 give E exp{θ∥P Ui ∥2L2 (Q) } ≤ Eg exp{θb2 ⟨g, P VP g⟩L2 (Q) }, for every 0 < θ < {2b2 ∥P VP ∥L2 (Q) }−1 , where g is standard Gaussian on that range. The Gaussian quadratic-form bound yields the displayed moment estimate, and increasing finite-dimensional projections give the general case. Because ∥u ⊗L2 (Q) u∥HS,L2 (Q) = ∥u∥2L2 (Q) and ∥V∥HS,L2 (Q) ≤ trL2 (Q) (V), it follows that n o E∥Ui ⊗L2 (Q) Ui − V∥pHS,L2 (Q)

1/p

≲ b2 p σL22 (Q) (0, V).

The preceding moment estimate satisfies the Bernstein moment condition with variance proxy of order b4 σL42 (Q) (0, V) and scale of order b2 σL22 (Q) (0, V). Hence the Hilbert-space Bernstein inequality [Pinelis and Sakhanenko, 1986, Corollary 1], applied in the Hilbert space of Hilbert–Schmidt operators, yields (r ) n 1X log(2/η) log(2/η) (Ui ⊗L2 (Q) Ui − V) ≲ b2 σL22 (Q) (0, V) + . n n n 2 i=1

HS,L (Q)

74

P Independence and Definition A.1 also show directly that n−1 ni=1 Ui is L2 (Q)-b-sub-Gaussian with covariance V/n. Applying the preceding norm-moment bound with p = max{2, log(2/η)} and then Markov’s inequality gives r n log(2/η) 1X Ui ≲ bσL2 (Q) (0, V) . n n 2 i=1

L (Q)

Combining the last three displays and enlarging the logarithmic factor gives   2 b2 σ 2 2 L2 (Q) (0, V) √ . ∆HS ≲ log η n For the tail, take P = Π⊥ m . The centering identity above gives n

bora − V)P } = trL2 (Q) {P (V

o 1 Xn ∥P Ui ∥2L2 (Q) − E∥P Ui ∥2L2 (Q) n i=1 ! 2 n 1X Ui . − P n 2 i=1

L (Q)

Moreover, n p o1/p ≲ b2 p σL22 (Q) (m, V). E ∥P Ui ∥2L2 (Q) − E∥P Ui ∥2L2 (Q) Scalar Bernstein controls the first term in the tail identity. The same sub-Gaussian closure argument P applied to P (n−1 ni=1 Ui ), whose covariance trace is σL22 (Q) (m, V)/n, controls the second term by b2 σL22 (Q) (m, V) log(2/η)/n. After replacing η by a fixed fraction and taking a union bound, these bounds hold simultaneously with probability at least 1 − 3η, and   b2 σ 2 2 L2 (Q) (m, V) √ ∆tail ≲ log . η n Substitution into (F.1) yields   ( 4 4 )1/4   3/2 b σ (0, V) 2 2 L (Q) m1/4 + σL2 (Q) (m, V) . ∥V 1/2 g − G∥L2 (Q) ≲ log η n Here the empirical-tail contribution b n−1/4 σL2 (Q) (m, V) is absorbed by the first term because m ≥ 1 and σL2 (Q) (m, V) ≤ σL2 (Q) (0, V). Taking the infimum over m ≥ 1 proves the claim.

F.16

Proof of Corollary A.1

Polynomial spectral decay. Proposition F.5. Suppose µk ≤ αk −β for some α > 0 and β > 1. Under Assumptions 4.1, 4.2, 4.4, and A.1, when τn ≥ τ0 , if τn → ∞ and λ → 0 and   (τn κ2 + 1) log(e + Nλ ) τn n > max , C2 λ λNλ 75

where C2 is the constant in Lemma F.13 after replacing its failure probability by the fixed fraction of η used in the union bound, then with probability at least 1 − η, there exists Z ∼ NL2 (Q) (0, V) such that   1     2 r   3 1 1 √ 3 log(n) τ 1 2 τ n 2(2+β) n n − − 2 n(fˆ − fP w ) − Z 2 ≲ √ λ 2 2β log log log(n) . + λ η n λ η n L (Q) Proof of Proposition F.5. Lemmas F.7, F.25, F.20, and F.21 give trL2 (Q) (V) ≲ τn λ

−1/β

r ,

∥V∥L2 (Q) ≲ τn ,

σL2 (Q) (m, V) ≲

τn −(β−1)/2 m . λ

By Lemma A.1, the influence elements are uniformly L2 (Q)-b-sub-Gaussian. Apply Proposition A.1 with     n 1/(β+2) . m = min n, λ Both coupling terms are then bounded, up to their displayed logarithmic and tolerance factors, by r   3 τn n 2(β+2) . n λ The sample-size condition and τn → ∞ imply nλ → ∞; hence the untruncated ceiling is at most n for all sufficiently large n. Moreover, (3/η)1/ log(mn) ≤ (3/η)1/ log n . Proposition 4.2 and Nλ ≲ λ−1/β give     2 1 τn − 12 − 2β 1 2 ∆G,Bah ≲ √ λ log log . λ η n Combining the Bahadur and coupling events after dividing η by a fixed constant proves the result. Proposition F.6. Suppose µk ≤ αk −β for some α > 0 and β > 1. Under Assumptions 4.1, 4.2, 4.4, and A.1, when τn ≥ τ0 , if τn → ∞ and λ → 0 and   (τn κ2 + 1) log(e + Nλ ) τn , n > max C2 λ λNλ where C2 is the constant in Lemma F.13 evaluated at the component tolerance ηtol = cη 2 used in Corollary 4.1, then, on a data event of probability at least 1 − η, there exists a coupling of B and Z, conditionally on D, such that Z | D ∼ NL2 (Q) (0, V) and (   3 1 1 2 τ √n λ− 2 − β P ∥Z − B∥L2 (Q) ≲ log η n β−1 ! 2β−1 )    3   1 1− 2 2 2 τn 2 τn λ β √ + log D ≥ 1 − η. η λ n Proof of Proposition F.6. The spectral bounds used in the preceding proof imply r τn −(β−1)/2 2 −1/β σL2 (Q) (0, V) ≲ τn λ , σL2 (Q) (m, V) ≲ m . λ 76

In Proposition A.2, take (  m = max 1,

√

&



n

2 2β−1

)' .

τn λ1−2/β

The spectral-tail term is bounded by r

τn λ1−2/β √ n

τn λ

! β−1

2β−1

.

The finite-dimensional term obeys the same bound because the ratio of that term to the preceding −1/2 display is at most τn λ1/(2β) ≤ 1. If the expression inside the maximum is smaller than one, the choice m = 1 gives the same conclusion. On the other hand, Proposition 4.4 and Nλ ≲ λ−1/β give   3 1 1 2 τ √n λ− 2 − β . ∆B,feas ≲ log η n The conditional Strassen–Dudley argument in Lemma F.19, followed by the same two-layer union bound as in Corollary 4.1, couples the oracle bootstrap to Z and then combines it with the feasiblebootstrap bound. The triangle inequality proves the assertion. Exponential spectral decay. Proposition F.7. Suppose µk ≤ α exp(−βk γ ) for some α, β, γ > 0. Under Assumptions 4.1, 4.2, 4.4, and A.1, when τn ≥ τ0 , if τn → ∞ and λ → 0 and   (τn κ2 + 1) log(e + Nλ ) τn n > max , C2 λ λNλ where C2 is the constant in Lemma F.13 after replacing its failure probability by the fixed fraction of η used in the union bound, then with probability at least 1 − η, there exists Z ∼ NL2 (Q) (0, V) such that    1   2   2γ √ τ 1 2 1 n √ log n(fˆ − fP w ) − Z 2 ≲ log log log η λ λ L (Q) nλ   1 r   n  2γ3 1 log n τn 1 + log (log(n))2 . η n β λ Proof of Proposition F.7. Lemmas F.7, F.25, F.23, and F.24 imply   1/γ 1 trL2 (Q) (V) ≲ τn log , λ and

r σL2 (Q) (m, V) ≲

τn (1−γ)/2 m exp(−βmγ /2). λ

Apply Proposition A.1, using Lemma A.1, with ( & m = min n,

∥V∥L2 (Q) ≲ τn ,

n 1 log β λ

77

1/γ ') .

γ

Since e−βm /2 ≲ (n/λ)−1/2 whenever the ceiling is at most n, and m(1−γ)/2 ≤ 1 + m3/2 , both coupling terms are bounded by r   1/ log n  n 3/(2γ) τn 1 1 2 log log (n) . n β λ η The sample-size condition and τn → ∞ imply nλ → ∞; hence the untruncated ceiling is at most n for all sufficiently large n. Finally, Proposition 4.2 and the exponential effective-dimension bound give   2   1/(2γ)   2 τn 1 1 √ . ∆G,Bah ≲ log log log log η λ λ nλ A union bound completes the proof. Proposition F.8. Suppose µk ≤ α exp(−βk γ ) for some α, β, γ > 0. Under Assumptions 4.1, 4.2, 4.4, and A.1, when τn ≥ τ0 , if τn → ∞ and λ → 0 and   (τn κ2 + 1) log(e + Nλ ) τn n > max , C2 λ λNλ where C2 is the constant in Lemma F.13 evaluated at the component tolerance ηtol = cη 2 used in Corollary 4.1, then, on a data event of probability at least 1 − η, there exists a coupling of B and Z, conditionally on D, such that Z | D ∼ NL2 (Q) (0, V) and (   3    1 γ 2 τn 1 √ P ∥Z − B∥L2 (Q) ≲ log log η λ nλ )    3 √    1    1 2 2γ τn n 4γ 2 1 log + log log D ≥ 1 − η. 1 η λ λ n4 Proof of Proposition F.8. The exponential spectral bounds give   1/γ 1 2 σL2 (Q) (0, V) ≲ τn log λ and

r σL2 (Q) (m, V) ≲

τn (1−γ)/2 −βmγ /2 m e . λ

In Proposition A.2, take & m=

n 1 log β λ

1/γ ' .

The finite-dimensional term is then bounded by √   1/(2γ)   1/(4γ) τn 1 n log log . 1/4 λ λ n The spectral-tail term is at most r

τn   n (1−γ)/(2γ) log , n λ 78

which is smaller than the preceding display for all sufficiently large n, because a polynomial factor n−1/4 dominates every fixed power of the logarithm. Proposition 4.4 and the exponential effective-dimension bound give   3   1/γ 2 τn 1 √ ∆B,feas ≲ log log . η λ nλ Combining the sub-Gaussian oracle coupling with the feasible bootstrap through Lemma F.19, and applying the two-layer union bound from Corollary 4.1, proves the result. Proof of Corollary A.1. Choose C0 ≥ max{ρ, τ0 }. Then wn = ρ, LP w = LQ , and Assumption 4.4 holds with C1 = 1. We first establish the required spectral approximation bounds directly for the fixed envelope τn = C0 . Because wn = ρ ≤ ρ ≤ C0 , every occurrence of the truncation envelope in the nonasymptotic Bahadur, covariance, and feasible-bootstrap bounds is controlled by C0 . Repeating the proofs of the four sub-Gaussian spectral approximation propositions with this fixed envelope, rather than invoking their asymptotic statements, gives the same displayed approximation bounds under the corresponding sample-size conditions. Thus τn → ∞ is not required in the bounded-ratio case. The corresponding sample-size conditions hold for all sufficiently large n under each tuning choice below. √ We next establish the polynomial result. Since s ≥ 1/2, Proposition 4.1 gives ∆bias ≲ nλ. At η ≍ n−1 , the fixed-envelope polynomial bounds just established yield sg
sg sg ∆ ≲ log3 n ∆sg 0 + ∆1 + ∆2 + ∆3 ,

where

∆sg 0 =

(F.6)

√

nλ, r   3 τn n 2(β+2) , ∆sg 1 = n λ τn − 12 − β1 ∆sg , 2 = √ λ n ! β−1 r τn τn λ1−2/β 2β−1 sg √ . ∆3 = λ n

1/ log n = O(1). Here the Bahadur term is absorbed by ∆sg 2 , and (3/η) Before deriving the simpler rate stated in the corollary, we record the sharper optimization underlying (F.6). With τn = C0 and λ = n−a , the four powers of n are   1 1 3(1 + a) 1 1 1 β−1 5β − 4 − a, − + , − +a + , − +a . 2 2 2(β + 2) 2 2 β 2(2β − 1) 2β(2β − 1)

For β > 3, their sharp balance is   √ β(3β − 2) 2β 2 − 7β + 4   ,   4β 2 + 3β − 4 2(4β 2 + 3β − 4) , 3 < β ≤ 3 + 5, (a♯β , rβ♯ ) =    √ 2β β−2   , , β > 3 + 5.  3β + 2 2(3β + 2)

79

(F.7)

sg sg sg Indeed, the first branch balances ∆sg 0 and ∆3 , whereas the second balances ∆0 and ∆2 ; the two √ formulas meet at β = 3 + 5, and direct substitution shows that the remaining terms are smaller. Consequently, ♯ ∆ ≲ n−rβ log3 n. sg More generally, if τn = nt and the same a♯β is used, the four powers of τn in ∆sg 0 , . . . , ∆3 are

1 , 2

0,

4β − 3 < 1. 2(2β − 1)

1,

Thus, whenever a♯β + t < 1, ♯

∆ ≲ n−rβ +t log3 n,

√

♯

τn ∆ ≲ n−rβ +3t/2 log3 n.

For the non-piecewise statement of the corollary, set e aβ =

2β , 3β + 2

reβ =

β−3 . 2(3β + 2)

At a = e aβ , the negatives of the four powers of n are β−2 , 2(3β + 2)

3β 2 − 7β − 2 , 2(β + 2)(3β + 2)

β−2 , 2(3β + 2)

(β − 3)(3β − 2) . 2(3β + 2)(2β − 1)

Each is at least reβ when β > 3. Therefore ∆ ≲ n−erβ log3 n. √ Since τn = O(1), Theorem 4.1 proves part (1). Under exponential spectral decay, the fixed-envelope exponential bounds just established give the polynomial-order envelope r √ √ τn τn τn ∆ ≲poly nλ + √ + + 1/4 , (F.8) n n nλ where ≲poly suppresses only the displayed logarithmic and tolerance factors. With fixed τn and λ = n−a , the potentially active powers are 1 − a, 2

a−1 , 2

1 − . 4

The first two are balanced at the sharp polynomial-order choice a = 2/3, where they equal −1/6, while the last equals −1/4. More generally, with λ = n−2/3 and τn = nt , ∆ ≲ n−1/6+t (log n)3+1/ζ ,

√

τn ∆ ≲ n−1/6+3t/2 (log n)3+1/ζ .

For t = 0, the active stochastic term is the feasible-bootstrap term, whose logarithmic factor is (log n)3+1/ζ ; all other stochastic terms have strictly smaller powers of n, so their fixed logarithmic factors are absorbed. Theorem 4.1 now proves part (2).

80

F.17

Proof of Corollary A.2

Proof. Let τn = nt and λ = n−a . As in the proof of Corollary 4.3, polynomial spectral decay verifies Assumption 4.3 with d = 1/β, while exponential spectral decay does so with any fixed d ∈ (0, 1]. Under Definition 4.1, we may choose a fixed integer m0 sufficiently large that a(1 + d)γ . t √ Lemma F.29 then verifies Assumption 4.4 with C1 ≤ 2 for all sufficiently large n. For polynomial spectral decay, use m0 − 1 >

a=e aβ =

2β , 3β + 2

t=

reβ β−3 = . 6(3β + 2) 3

By the calculation following (F.7), restoring τn = nt enlarges the approximation error by at most nt , and Theorem 4.1 contributes one further factor nt/2 . Consequently, β−3

− P{f∗ ∈ / CSα (fb)} − α ≲ n−1 + n−erβ +3t/2 log3 n ≲ n 4(3β+2) log3 n.

The selected powers satisfy a+t < 1, so the sample-size conditions in the polynomial approximation propositions hold for all sufficiently large n. This proves part (1). For exponential spectral decay, take a = 2/3 and t = 1/150. The calculation following (F.8) gives ∆ ≲ n−1/6+t (log n)3+1/ζ = n−4/25 (log n)3+1/ζ . √ After multiplication by the anti-concentration factor τn , P{f∗ ∈ / CSα (fb)} − α ≲ n−1 + n−1/6+3t/2 (log n)3+1/ζ ≲ n−47/300 (log n)3+1/ζ . Here a + t < 1, and the strictly smaller polynomial powers of the remaining terms absorb their fixed logarithmic factors. This proves part (2).

F.18

Proof of Corollary A.3

Proof. By the common moment-compatibility reduction in (F.5), it is sufficient that τn ≳ (λ−1 Nλ )1/(θ−1) . The proof of Proposition F.8 uses only this compatibility condition, the bounded truncated-weight envelope, and the standing assumptions; the Bernstein-type condition is not used. Write λ = n−a and τn = nt . Under exponential spectral decay, Nλ ≲ {log(1/λ)}1/ζ . With a=

8(θ2 − 1) , 3(4θ2 + 3θ − 3)

we have t−

t=

19θ + 1 , 6(4θ2 + 3θ − 3)

a θ−5 = > 0. 2 θ−1 2(4θ + 3θ − 3)

Thus the strict polynomial slack absorbs the logarithmic effective-dimension factor and verifies (F.5). 81

After suppressing logarithmic factors, the four approximation terms in (F.8) have powers 1 − a, 2

1 a − + + t, 2 2

1 t − + , 2 2

1 t − + . 4 2

The feasible-bootstrap term is the largest. Indeed, its excess over the bias term equals −1 +

θ−5 3a +t= > 0, 2 6(4θ2 + 3θ − 3)

and its excesses over the final two terms are, respectively, (a + t)/2 > 0 and −1/4 + (a + t)/2 > 0. Its negative exponent is 1 a 2θ2 − 5θ − 1 − −t= . 2 2 3(4θ2 + 3θ − 3) Therefore −

∆≲n Multiplying by

√

2θ 2 −5θ−1 3(4θ 2 +3θ−3)

(log n)3+1/ζ .

τn gives 2θ2 − 5θ − 1 t (θ − 5)(8θ + 1) − = , 2 3(4θ + 3θ − 3) 2 12(4θ2 + 3θ − 3)

which is positive for θ > 5. The active feasible-bootstrap term has logarithmic factor (log n)3+1/ζ , while all remaining terms have strictly smaller powers of n. Finally, 8θ2 − θ − 3 1−a−t= > 0, 6(4θ2 + 3θ − 3) so both sample-size restrictions hold for all sufficiently large n. Since the displayed coverage exponent is smaller than one, the n−1 term in Theorem 4.1 is absorbed. This proves the corollary.

G

Proofs of Technical Lemmas

This appendix collects auxiliary proofs and the derivation of the bootstrap matrix representation.

G.1

Additional lemmas

Lemma G.1. Let T : H → H be a bounded linear self-adjoint operator on H. Then T LQ : L2 (Q) → L2 (Q) is a self-adjoint operator in L2 (Q). Lemma G.2. For 0 < η < 1 and 0 < λ ≤ 1, if (τn κ2 + 1) log(e + Nλ ) < C2 nλ where C2 is the constant in Lemma F.13, then, on the same event of probability at least 1 − η as in Lemma F.12, √ 1 1 (LPnw + λI)− 2 (LP w + λI) 2 ≤ 2.

82

G.2

Proof of Lemma 2.2

Proof. Let f =

P

µj >0 aj ϕj ∈ Ran(LQ ). Then 1/2

1/2

X

LQ f =

µj a j ϕ j .

µj >0

By Lemma 2.1, this element belongs to H, and for g = D

1/2

1/2

LQ f, LQ g

E H

=

X

P

µj >0 bj ϕj ∈ Ran(LQ ),

aj bj = ⟨f, g⟩L2 (Q) .

µj >0

1/2

Thus LQ preserves inner products on Ran(LQ ). It remains to note that it is onto H: if h = P µj >0 hj ϕj ∈ H, then X hj fh := ϕ 1/2 j µj >0 µj P 1/2 2 2 belongs to Ran(LQ ), since ∥fh ∥2L2 (Q) = µj >0 hj /µj = ∥h∥H < ∞, and LQ fh = h. Hence 1/2

LQ : Ran(LQ ) → H is a unitary isomorphism.

G.3

Proof of Lemma F.1

Proof. Since the eigenvalues are ordered non-increasingly, the definition of s∗ implies that µi > λ for i < s∗ and µi ≤ λ for i ≥ s∗ . For sufficiently small λ, s∗ ≥ 2. Hence, for i < s∗ ,  k µi −k 2 ≤ ≤ 1, µi + λ and therefore

∗ −1  sX

µi µi + λ

k

≍ s∗ .

i=1 ∗ On the other hand, for i > s we have µi ≤ λ, so

µk 2−k ki ≤ λ Consequently,

∞  X i=s∗ +1



µi µi + λ

k

k

∞ 1 X k µi . λk ∗

µi µi + λ

≍

≤

µki . λk

i=s +1

The remaining term i = s∗ is bounded above by one and below by zero, and is therefore absorbed by the s∗ term. Combining the preceding two displays gives the claimed equivalence.

G.4

Proof of Lemma F.2

Proof. In the polynomial case, s∗ ≍ λ−1/β and λ−k

∞ X

µki ≍ λ−k (s∗ )1−kβ ≍ λ−1/β .

i=s∗ +1

83

Lemma F.1 gives the first rate. In the exponential case, s∗ ≍ {log(1/λ)}1/γ , while an integral comparison gives ∞ X λ−k µki ≲ s∗ . i=s∗ +1

Lemma F.1 then gives the second rate, since the leading s∗ term also provides the matching lower bound.

G.5

Proof of Lemma F.3

Proof. Decompose the influence element Ui in (3.1) as Ui,1 + Ui,2 , where
−1
−1

Ui,1 = LP w + λI w(Xi ) εi kXi , Ui,2 = LP w + λI w(Xi )kXi ⊗ kXi − LP w f∗ − fP w . As E[εi | Xi ] = 0, we have   V = E Ui ⊗L2 (Q) Ui     = E Ui,1 ⊗L2 (Q) Ui,1 + E Ui,2 ⊗L2 (Q) Ui,2   ⪰L2 (Q) E Ui,1 ⊗L2 (Q) Ui,1 n o  
−1
−1 w(Xi )kXi . ⪰L2 (Q) σ 2 E LP w + λI w(Xi )kXi ⊗L2 (Q) LP w + λI

G.6

Proof of Lemma F.4

Proof. For any f ∈ L2 (Q),    
 E w(Xi ) (LP w + λI)−1 kXi ⊗L2 (Q) (LP w + λI)−1 kXi f h i =E w(Xi ) (LP w + λI)−1 kXi , f L2 (Q) (LP w + λI)−1 kXi   =E w(Xi ) (LP w + λI)−1 kXi , LQ f (LP w + λI)−1 kXi   =E w(Xi ) (LP w + λI)−1 kXi , (LP w + λI)−1 LQ f kXi =(LP w + λI)−1 LP w (LP w + λI)−1 LQ f.

G.7

Proof of Lemma F.5

Proof. Let 1/2

HQ := Ran(LQ ) and U := LQ : HQ → H. By Lemma 2.2, U is a unitary isomorphism. Moreover, L2 (Q) = HQ ⊕ ker(LQ ), and T LQ vanishes on ker(LQ ). For f, g ∈ HQ , D E 1/2 1/2 ⟨f, T LQ g⟩L2 (Q) = U f, LQ T LQ U g . H

1/2

1/2

Since T is positive self-adjoint, LQ T LQ is positive self-adjoint. The preceding identity therefore shows that the restriction of T LQ to HQ is also positive self-adjoint. Its nonzero eigenvalues are precisely those of T LQ on L2 (Q). 84

We may now apply the min–max theorem and Lemma 2.2: µk (T LQ , L2 (Q)) = =

max

min

⟨f, T LQ f ⟩L2 (Q)

max

min

⟨LQ f, LQ T LQ LQ f ⟩H

V ⊂HQ f ∈V dim(V )=k ∥f ∥L2 (Q) =1

1/2

1/2

1/2

1/2

V ⊂HQ f ∈V dim(V )=k ∥L1/2 f ∥H =1 Q

=

max

1/2

1/2

min ⟨g, LQ T LQ g⟩H

g∈W W ⊂H dim(W )=k ∥g∥H =1 1/2

1/2

= µk (LQ T LQ , H).

G.8

Proof of Lemma F.10

Proof. If 0 ≤ s ≤ 12 in Assumption 4.2, LP w + λI

− 1

2

− 1

f∗ =

LP w + λI

≤

LP w0 + λI

2

LsP w0 g∗

− 1

2

LsP w0 g∗

1

≤λ− 2 +s ∥g∗ ∥. If 12 < s ≤ 1 in Assumption 4.2, s− 1

1

(LP w + λI)− 2 f∗ ≤ LP w20 g∗ s− 1

≤ LQ 2 g∗ ≤ κ2s−1 ∥g∗ ∥ . In conclusion, 1

1

(LP w + λI)− 2 f∗ ≤ max{κ2s−1 , 1} λmin{s− 2 ,0} ∥g∗ ∥.

G.9

Proof of Lemma F.6

Proof. Since fP w − f∗ = −λ(LP w + λI)−1 f∗ , we have ∥fP w − f∗ ∥ = λ (LP w + λI)−1 f∗ ≤ λ1/2 (LP w + λI)−1/2 f∗ ≤ max{κ2s−1 , 1}λ1/2+min{s−1/2,0} ∥g∗ ∥ = max{κ2s−1 , 1}λmin{s,1/2} ∥g∗ ∥. 85

The first inequality follows from λ1/2 (LP w + λI)−1/2 ≤ 1, and the second follows from Lemma F.10.

G.10

Proof of Lemma F.7

Proof. With the same notation as in the proof of Lemma F.3, V = E[Ui,1 ⊗L2 (Q) Ui,1 ] + E[Ui,2 ⊗L2 (Q) Ui,2 ] =: V1 + V2     (LP w + λI)−1 w(Xi )εi kXi ⊗L2 (Q) (LP w + λI)−1 w(Xi )εi kXi  ⪯L2 (Q) τn σ 2 E w(Xi )[(LP w + λI)−1 kXi ] ⊗L2 (Q) [(LP w + λI)−1 kXi ]

V1 = E By Lemma F.4,

V1 ⪯L2 (Q) τn σ 2 · (LP w + λI)−1 LP w (LP w + λI)−1 LQ . 1

1

2 2 (LP w + λI)−1 LP w (LP w + λI)−1 LQ . Denote T1 = (LP w + λI)−1 LP w (LP w + λI)−1 LQ , T2 = LQ With Lemma F.5,
2 (G.1) ∥T1 ∥L2 (Q) = ∥T2 ∥H ≤ C1 .

Then,
2 ∥V1 ∥L2 (Q) ≤ τn σ 2 ∥T1 ∥L2 (Q) ≤ τn σ 2 C1 . ∀f ∈ L2 (Q), D E 1/2 1/2 ⟨f, V1 f ⟩L2 (Q) ≤ f, τn σ 2 T1 f L2 (Q) = τn σ 2 LQ f, T2 LQ f . Similarly with Lemma F.5, trL2 (Q) (V1 ) =

∞ X

µk (V1 , L2 (Q))

k=1

≤ τn σ 2

∞ X

µk (T1 , L2 (Q))

k=1

= τn σ 2

∞ X

µk (T2 , H)

(G.2)

k=1 2

= τn σ trH (T2 )

2 ≤ τn σ 2 C1 trH (LP w + λI)−1 LP w
2 ≤ τn σ 2 C1 Nλ . The last inequality follows from the domination of the truncated weighted-source operator by the target operator. Indeed, since w ≤ ρ, for every h ∈ H, Z ⟨h, (LQ − LP w )h⟩H = (ρ(x) − w(x))|h(x)|2 dP(x) ≥ 0, X

86

and hence LP w ⪯H LQ . The min–max principle and the monotonicity of t 7→ t/(t + λ) on [0, ∞) therefore give

trH (LP w + λI)−1 LP w ≤ trH (LQ + λI)−1 LQ = Nλ .

nh i
V2 = E (LP w + λI)−1 w(Xi ) kXi ⊗ kXi − LP w (f∗ − fP w ) ⊗L2 (Q) h io
(LP w + λI)−1 w(Xi ) kXi ⊗ kXi − LP w (f∗ − fP w ) nh i
⪯L2 (Q) E (LP w + λI)−1 w(Xi ) kXi ⊗ kXi (f∗ − fP w ) ⊗L2 (Q) h io
(LP w + λI)−1 w(Xi ) kXi ⊗ kXi (f∗ − fP w ) nh i
= E w(Xi ) f∗ (Xi ) − fP w (Xi ) (LP w + λI)−1 kXi ⊗L2 (Q) h io
w(Xi ) f∗ (Xi ) − fP w (Xi ) (LP w + λI)−1 kXi n h i h io
2 = E w(Xi ) f∗ (Xi ) − fP w (Xi ) w(Xi ) (LP w + λI)−1 kXi ⊗L2 (Q) (LP w + λI)−1 kXi ⪯L2 (Q) τn κ2 max{κ2s−1 , 1}2 λ2 min{s,1/2} ∥g∗ ∥2 n h i h io × E w(Xi ) (LP w + λI)−1 kXi ⊗L2 (Q) (LP w + λI)−1 kXi = τn κ2 max{κ2s−1 , 1}2 λ2 min{s,1/2} × ∥g∗ ∥2 T1 . where we use Lemma F.6 to get the inequality f∗ (Xi ) − fP w (Xi ) = ⟨f∗ − fP w , kXi ⟩ ≤ κ∥f∗ − fP w ∥ ≤ κ max{κ2s−1 , 1}λmin{s,1/2} ∥g∗ ∥. Combining this bound with (G.1) and (G.2) proves the result.

G.11

Proof of Lemma F.8

Proof. With Lemma F.3 and F.4, nh i h io V ⪰L2 (Q) σ 2 E (LP w + λI)−1 w(Xi )kXi ⊗L2 (Q) (LP w + λI)−1 w(Xi )kXi ⪰L2 (Q) ρσ 2 (LP w + λI)−1 LP w (LP w + λI)−1 LQ . For any f ∈ L2 (Q), ⟨f, Vf ⟩L2 (Q) ≥ ρσ 2 ⟨f, (LP w + λI)−1 LP w (LP w + λI)−1 LQ f ⟩L2 (Q) = ρσ 2 ⟨LQ f, (LP w + λI)−1 LP w (LP w + λI)−1 LQ f ⟩H
−2 2 ρσ ⟨LQ f, (LQ + λI)−1 LQ (LQ + λI)−1 LQ f ⟩H ≥ C2
−2 2 = C2 ρσ ⟨f, (LQ + λI)−1 LQ (LQ + λI)−1 LQ f ⟩L2 (Q) .

87

G.12

Proof of Lemma F.12

Proof. Let 1

1

η(x) = (LP w + λI)− 2 w(x) (kx ⊗ kx )(LP w + λI)− 2 ∀h ∈ H, 1

1

η(x)h = w(x) (LP w + λI)− 2 ⟨kx , (LP w + λI)− 2 h⟩ kx 1

1

[η(x)]2 h = w2 (x) (LP w + λI)− 2 ⟨kx , (LP w + λI)−1 ⟨kx , (LP w + λI)− 2 h⟩kx ⟩ kx 1

1

= w2 (x) ⟨kx , (LP w + λI)−1 kx ⟩ (LP w + λI)− 2 (kx ⊗ kx )(LP w + λI)− 2 h. Thus, 1

1

[η(x)]2 = w2 (x) ⟨kx , (LP w + λI)−1 kx ⟩(LP w + λI)− 2 (kx ⊗ kx )(LP w + λI)− 2 . As

⟨kx , (LP w + λI)−1 kx ⟩ = tr (LP w + λI)−1 kx ⊗ kx

≤ λ−1 κ2  1 1 τn κ2 E w(x)(LP w + λI)− 2 (kx ⊗ kx )(LP w + λI)− 2 λ 1 1 τn κ2 = (LP w + λI)− 2 LP w (LP w + λI)− 2 λ τn κ2 ≤ . λ Since wn ≤ ρ, we have LP w ⪯ LQ . The map A 7→ A(A + λI)−1 is operator monotone on the cone of positive operators. Therefore,   Tr (LP w + λI)−1 LP w ≤ Tr (LQ + λI)−1 LQ = Nλ , E[(η(x))2 ] ≤

and hence    Tr E (η(x))2 ≤ τn κ2 λ−1 Nλ . Let ξ(x) = η(x) − E[η(x)]. ∥ξ(x)∥ ≤ ∥η(x)∥ + ∥E[η(x)]∥ ≤

τn κ2 + 1. λ

n n hX 2 i X E ξ(Xi ) = E[ξ(Xi )2 ] = n Eη 2 − (Eη)2 i=1

i=1

 τ κ2  n ≤ n Eη 2 + n (Eη)2 ≤ n +1 . λ n  hX 2 i     Tr E ξ(Xi ) = n Tr E (η)2 − (Eη)2 i=1

≤ n Tr E[(η)2 ] ≤ nτn κ2 λ−1 Nλ   τ κ2 n + 1 Nλ . ≤n λ 88

Consider 0 < λ ≤ 1. Applying Lemma F.11 with R=

τn κ2 + 1 , λ

V =n r

∀t ≥

τn κ2 + 1 , λ

W = e + Nλ

τn κ2 + 1 τn κ2 + 1 + nλ 3nλ

  1 1 P (LP w + λI)− 2 (LPnw − LP w )(LP w + λI)− 2 ≥ t n  1X  =P ξ(Xi ) ≥ t n i=1   λ n t2 ≤4(e + Nλ ) exp − . 2t(τn κ2 + 1) 2 (τn κ2 + 1) + 3 Let ϵ > 0 be the positive solution of   4(e + Nλ ) nλ ϵ2
= log . η 2(τn κ2 + 1) 1 + 3ϵ p √ Solving this quadratic equation and using x2 + y ≤ x + y for x, y ≥ 0 give   τn κ2 + 1 4(e + Nλ ) ϵ= log 3nλ η s  2   τn κ2 + 1 4(e + Nλ ) 2(τn κ2 + 1) 4(e + Nλ ) + log + log 3nλ η nλ η s     4(e + Nλ ) 4(e + Nλ ) 2(τn κ2 + 1) 2(τn κ2 + 1) log + log . ≤ 3nλ η nλ η

Since log{4(e + Nλ )/η} > 1, this choice also satisfies the applicability threshold of Lemma F.11; indeed, r τn κ2 + 1 τn κ2 + 1 ϵ≥ + . nλ 3nλ Consequently,   1 1 P (LP w + λI)− 2 (LPnw − LP w )(LP w + λI)− 2 ≥ ϵ    4(e + Nλ ) ≤ 4(e + Nλ ) exp − log = η. η Moreover, because log(e + Nλ ) ≥ 1 and log(2/η) ≥ log 2,       4 2 4(e + Nλ ) log = log(e + Nλ ) + log ≲ log(e + Nλ ) log . η η η Together with {log(2/η)}1/2 ≲ log(2/η), substituting this comparison into the preceding upper bound for ϵ proves the first assertion after increasing C1 by a universal constant. Under the additional condition in the second assertion, the nonnegative ratio satisfies s

τn κ2 + 1 log(e + Nλ ) τn κ2 + 1 log(e + Nλ ) ≤ . nλ nλ 89

Thus, the second assertion follows from the first after absorbing the factor 2C1 into the implicit constant.

G.13

Proof of Lemma F.13

Proof. With Lemma F.12, when ) ( (τn κ2 + 1) log(e + Nλ ) 1 < min 1,
2 , nλ 4C1 log η2 1 1 1 (LP w + λI)− 2 (LP w − LPnw )(LP w + λI)− 2 ≤ , 2

then 1

1

(LP w + λI) 2 (LPnw + λI)−1 (LP w + λI) 2 h i 1 1 =(LP w + λI) 2 (LPnw + λI)−1 − (LP w + λI)−1 (LP w + λI) 2 + I 1

1

1

1

=I + (LP w + λI)− 2 (LP w − LPnw )(LP w + λI)− 2 (LP w + λI) 2 (LPnw + λI)−1 (LP w + λI) 2 . Thus, 1

1

(LP w + λI) 2 (LPnw + λI)−1 (LP w + λI) 2 1 1 1 ≤1 + (LP w + λI) 2 (LPnw + λI)−1 (LP w + λI) 2 , 2 and hence 1

1

(LP w + λI) 2 (LPnw + λI)−1 (LP w + λI) 2 ≤ 2.

G.14

Proof of Lemma F.14

Proof. (LP w + λI)



− 12

 n 1 1X 1 ⊤ kx Wε = (LP w + λI)− 2 w(Xi )εi kXi n n i=1

− 12

E[(LP w + λI)

w(Xi )εi kXi ] = 0.

1 κ·σ ∥(LP w + λI)− 2 w(Xi )εi kXi ∥ ≤ τn √ λ 1

E∥(LP w + λI)− 2 w(Xi )εi kXi ∥2 1

≤ σ 2 · E∥(LP w + λI)− 2 w(Xi )kXi ∥2 1

1

= σ 2 · Etr([(LP w + λI)− 2 w(Xi )kXi ] ⊗ [(LP w + λI)− 2 w(Xi )kXi ]) ≤ τn · σ 2 · Nλ For 1

Zi := (LP w + λI)− 2 w(Xi )εi kXi ,

90

the preceding bounds imply n X

τn κσ ∥Zi ∥ ≤ √ λ

E∥Zi ∥2 ≤ nτn σ 2 Nλ ,

i=1

a.s.

Therefore, the Hilbert-space Bernstein inequality [Pinelis and Sakhanenko, 1986, Corollary 1] yields, with probability at least 1 − η, "s    # n τn κ 1X 2 2 τn Nλ + √ log . Zi ≲ σ log n n η η n λ i=1 Since

p log(2/η) ≲ log(2/η) and a + b ≤ 2 max{a, b}, the asserted bound follows.

G.15

Proof of Lemma F.15

Proof. n n n n Vbi − Vbj Vi − V j 1 XX 1 XX √ hij − hij √ B − Bora = n n 2 2 i=1 j=1

=

=

G.16

n n 1 XX

n 1 n

hij L−1 P w ,λ α̂ij − n

i=1 j=1 n X n X i=1 j=1

i=1 j=1 n X n X

1 n

hij L−1 P w ,λ αij

i=1 j=1

n n

1 XX −1 hij L−1 − L α̂ + hij L−1 w ,λ α̂ij − αij . ij Pnw ,λ P w ,λ P n i=1 j=1

Proof of Lemma F.16

Lemma G.3. Under Assumption 4.2, when τn ≥ τ0 , if 1

1

(LP w + λI)− 2 (LPnw − LP w )(LP w + λI)− 2 < δ ≤ and 1

(LP w + λI)− 2 then



1 2

 1 ⊤ < γ, kx Wε n

 1 2  ∥fˆ − fP w ∥ ≤ √ γ + δ · max{κ2s−1 , 1} · λmin{s+ 2 ,1} · ∥g∗ ∥ , λ   1 2 ∥fˆ − f∗ ∥ ≤ √ γ + max{κ2s−1 , 1} · λmin{s+ 2 ,1} · ∥g∗ ∥ . λ

Proof of Lemma G.3. The Bahadur representation gives    1 ⊤ −1 ˆ f − fP w = (LP w + λI) k Wε + (LPnw − LP w )(f∗ − fP w ) + u n x    w w w = L−1 P w ,λ En w(Xi )kXi εi + (LPn − LP )(f∗ − fP ) + u.

91

Here u satisfies   1 1 ∥(LP w + λI) 2 u∥ ≤ 2δ γ + δ · max{κ2s−1 , 1} · λmin{s+ 2 ,1} ∥g∗ ∥ . Then, 1 1 (LP w + λI) 2 (fˆ − fP w ) ≤ γ + δ · (LP w + λI) 2 (f∗ − fP w )   1 + 2δ γ + δ · (LP w + λI) 2 (f∗ − fP w )   1 ≤ 2 γ + δ · max{κ2s−1 , 1} · λmin{s+ 2 ,1} · ∥g∗ ∥ .

  1 1 (LP w + λI) 2 (fˆ − f∗ ) ≤ 2 γ + δ · max{κ2s−1 , 1} · λmin{s+ 2 ,1} · ∥g∗ ∥ 1

+ max{κ2s−1 , 1} · λmin{s+ 2 ,1} · ∥g∗ ∥   1 ≤ 2 γ + max{κ2s−1 , 1} · λmin{s+ 2 ,1} · ∥g∗ ∥ .

Proof of Lemma F.16. Apply Lemmas F.12, F.13, and F.14 with suitably split failure probabilities. The second sample-size condition implies (r ) r τn Nλ 2τn κ τn Nλ max , √ ≲κ . n n n λ The sample-size condition is meaningful only if Nλ > 0, and hence LQ ̸= 0. Let µ1 > 0 be a positive eigenvalue of LQ . Since λ → 0, for all sufficiently large n, Nλ ≥

µ1 1 ≥ . µ1 + λ 2

It follows that log(e + Nλ ) ≲ Nλ . Hence Lemma F.12 and the first sample-size condition give r   τn Nλ 2 −1/2 −1/2 (LP w + λI) (LPnw − LP w )(LP w + λI) ≲κ log , nλ η whereas Lemma F.14 gives 1 √ (LP w + λI)−1/2 λ



r    1 ⊤ τn Nλ 2 ≲ σκ kx Wε log . n nλ η

Substituting these two bounds into Lemma G.3, and using max{κ2s−1 , 1} ≤ κ2s , 0 < λ ≤ 1, and ∥g∗ ∥H ≤ 1 + ∥g∗ ∥H , proves both assertions.

92

G.17

Proof of Lemma F.17

−1 Denote ∆inv = (L−1 P w ,λ − LP w ,λ )u, where n

n

n

1 XX u= hij α̂ij . n i=1 j=1

Lemma G.4. Under Assumption 4.4, writing λ = λn , we have −1/2

−1/2

1/2

1/2

−1/2

w w ∥∆inv ∥L2 (Q) ≤ C1 ∥LP w ,λ L−1 P w ,λ LP w ,λ ∥∥LP w ,λ (LP − LPn )LP w ,λ ∥∥LP w ,λ u∥. n

Proof of Lemma G.4. 1/2

∥∆inv ∥L2 (Q) = LQ ∆inv 1/2

−1 = LQ (L−1 P w ,λ − LP w ,λ )u n

= =

1/2 −1 w w LQ L−1 Pnw ,λ (LP − LPn )LP w ,λ u



−1/2

1/2

LQ LP w ,λ



1/2

1/2

LP w ,λ L−1 P w ,λ LP w ,λ n

h i −1/2 −1/2 −1/2 LP w ,λ (LP w − LPnw )LP w ,λ LP w ,λ u

1/2 1/2 −1/2 −1/2 −1/2 w w ≤ C1 ∥LP w ,λ L−1 Pnw ,λ LP w ,λ ∥∥LP w ,λ (LP − LPn )LP w ,λ ∥∥LP w ,λ u∥.

Lemma G.5. For 0 < η < 1 and λ > 0, conditionally on D, with probability at least 1 − η, v s  ou X n n u4 n 1 4 X −1/2 2 2 −1/2 −1/2 t LP w ,λ u ≤ 1 + 2 log w(Xi ) εi LP w ,λ kXi + LP w ,λ LPiw (f∗ − fˆ) . η n n i=1

i=1

−1/2

Proof of Lemma G.5. LP w ,λ u | D is Gaussian. By Borell’s inequality under P(· | D), with probability 1 − η, s  oq n 1 2 −1/2 −1/2 LP w ,λ u ≤ 1 + 2 log Eh LP w ,λ u . η 1 2 −1/2 Eh LP w ,λ u = 2 n

n X n X

−1/2

LP w ,λ α̂ij

2

.

i=1 j=1

1 2 −1/2 −1/2 w(Xi )b εi LP w ,λ kXi − w(Xj )b εj LP w ,λ kXj 2 2 2 −1/2 −1/2 ≤ w(Xi )b εi LP w ,λ kXi + w(Xj )b εj LP w ,λ kXj .   w(Xi )b εi kXi = w(Xi ) Yi − fˆ(Xi ) kXi h i = w(Xi ) (Yi − f∗ (Xi )) + (f∗ (Xi ) − fˆ(Xi )) kXi
= w(Xi )εi kXi + w(Xi ) f∗ (Xi ) − fˆ(Xi ) kXi = w(Xi )εi kX + LP w (f∗ − fˆ). −1/2

LP w ,λ α̂ij

2

=

i

i

93

Then, 2 2 −1/2 Eh LP w ,λ u ≤ n ≤

n X

2

i=1

n 4X

n

−1/2

w(Xi )b εi LP w ,λ kXi

i=1

4 2 −1/2 w(Xi )εi LP w ,λ kXi + n

n X

2 −1/2 LP w ,λ LPiw (f∗ − fˆ) .

i=1

Lemma G.6. Under Assumption 4.1, for 0 < η < 1 and λ > 0, with probability at least 1 − η,  ( 2 2 2 r 4 2 3 ) n σ κ τn σ κ τn Nλ 1X 2 2 −1/2 2 w(Xi )εi LP w ,λ kXi ≤ τn σ Nλ + 2 log . + n η nλ nλ i=1

Proof of Lemma G.6. Let −1/2

ξi := w(Xi )εi LP w ,λ kXi Then ξi ≤

2

.

τn2 σ 2 κ2 . λ

Moreover, −1/2

2

Eξi = E w(Xi )εi LP w ,λ kXi h i 2 −1/2 ≤ τn σ 2 E w(Xi ) LP w ,λ kXi i h −1/2 −1/2 = τn σ 2 E w(Xi ) LP w ,λ kXi , LP w ,λ kXi   w = τn σ 2 tr L−1 L w P P ,λ ≤ τn σ 2 Nλ . The last inequality follows from w ≤ ρ, which implies LP w ⪯H LQ , and the operator monotonicity of x 7→ x/(x + λ). Then, τ 3 σ 4 κ2 Nλ . Eξi2 ≤ n λ Therefore, by Bernstein’s inequality, with probability 1 − η,  ( 2 2 2 r 3 4 2 ) τn σ κ Nλ τn σ κ 2 En ξi ≤ Eξi + 2 log + . η nλ nλ

Lemma G.7. For 0 < η < 1 and λ > 0, with probability at least 1 − η,  ( 4 2 r 6 3 ) n 1 X −1/2 2 2κ τn κ τn Nλ 2 2 LP w ,λ LPiw HS ≤ τn κ Nλ + 2 log + . n η nλ nλ i=1

Proof of Lemma G.7. Let −1/2

2

ξi := LP w ,λ LPiw HS 2

= (LP w + λI)−1/2 w(Xi ) kXi ⊗ kXi HS . 94

Then 0 ≤ ξi ≤

1 2 4 τ κ . λ n

Moreover, h i 2 Eξi ≤ τn E w(Xi ) (LP w + λI)−1/2 (kXi ⊗ kXi ) HS   ∞ X 2 = τn Ew(Xi ) (LP w + λI)−1/2 (kXi ⊗ kXi )ej  j=1

 = τn Ew(Xi )

∞ X

 ⟨kXi , ej ⟩ (LP w + λI)−1/2 kXi

2

j=1

 = τn Ew(Xi ) (LP w + λI)−1/2 kXi

∞ X 2

 ⟨kXi , ej ⟩2 

j=1

i h 2 = τn E w(Xi ) (LP w + λI)−1/2 kXi ∥kXi ∥2 ≤ τn κ2 Nλ . Thus, 1 3 6 τ κ Nλ . λ n By Bernstein’s inequality, with probability 1 − η, ) r  ( 4 κ6 Nλ 2 2 2κ 3 τn + τn . En ξi ≤ Eξi + 2 log η nλ nλ E ξi2 ≤

Lemma G.8. For 0 < η < 1, under Assumptions 4.1 and 4.2, suppose τn ≥ τ0 for all sufficiently large n, τn → ∞, and λ = λn → 0. If, for all sufficiently large n,   (τn κ2 + 1) log(e + Nλ ) τn n > max , C2 λ λNλ where C2 is the constant in Lemma F.13 after replacing its failure probability by the fixed fraction of η used in the union bound, then, for all sufficiently large n, with probability at least 1 − η over the data, conditionally on the data with probability at least 1 − η, !    r 2
p τn Nλ 2 −1/2 2s+3 ∥LP w ,λ u∥ ≲ σ κ 1 + ∥g∗ ∥H τn Nλ 1 + log . nλ η Proof of Lemma G.8. Apply Lemmas G.6, G.7, and F.16 with suitably split failure probabilities. The second sample-size condition implies   n 1X 2 −1/2 2 2 2 ∥w(Xi )εi LP w ,λ kXi ∥ ≲ σ κ τn Nλ log , n η i=1   n 2 1 X −1/2 4 2 ∥LP w ,λ LPiw ∥HS ≲ κ τn Nλ log . n η i=1

95

Furthermore,

n

n

i=1

i=1

1 X −1/2 1 X −1/2 ∥LP w ,λ LPiw (f∗ − fˆ)∥2 ≤ ∥f∗ − fˆ∥2 ∥LP w ,λ LPiw ∥2HS . n n Substituting these bounds into Lemma G.5, followed by the second bound in Lemma F.16, yields
p −1/2 ∥LP w ,λ u∥ ≲ σ κ2s+3 1 + ∥g∗ ∥H τn Nλ " #    r 2 τn Nλ 2 min{s,1/2} × 1+ +λ log . nλ η Since 0 < λ ≤ 1 for all sufficiently large n, the last λ-term is absorbed by 1, proving the stated bound. Replacing η by fixed fractions of η in the data and conditional multiplier bounds changes only universal constants inside the logarithm. Proof of Lemma F.17. Apply Lemmas F.12, F.13, G.4, and G.8, with suitably split failure probabilities. This gives r
Nλ log(e + Nλ ) 2s+4 ∥∆inv ∥L2 (Q) ≲ C1 σ κ 1 + ∥g∗ ∥H τn !   nλ r 3 τn Nλ 2 × 1+ log . nλ η By the argument in the proof of Lemma F.16, log(e + Nλ ) ≲ Nλ for all sufficiently large n. Hence s log(e + Nλ ) ≲ 1. Nλ The first sample-size condition also gives s r r √ log(e + Nλ ) τn Nλ τn log(e + Nλ ) C2 = ≤ ≤ 1. Nλ nλ nλ κ √ Both terms are therefore bounded by τn Nλ / nλ, which proves the claim.

G.18

Proof of Lemma F.18

Lemma G.9. Under Assumption 4.4, writing λ = λn , for 0 < η < 1, conditionally on D, with probability at least 1 − η, v s   u X n
1 u −1 2 t2 ∥∆res ∥L2 (Q) ≤ C1 ∥fP w − fˆ∥ 1 + 2 log LP w2 ,λ LPiw HS . η n i=1

Proof of Lemma G.9.   1  1  w w α̂ij − αij = √ w(Xi )b εi kXi − w(Xj )b εj kXj − √ w(Xi )ϵPi kXi − w(Xj )ϵPj kXj 2 2 i

1 h = √ w(Xi ) fP w (Xi ) − fˆ(Xi ) kXi − w(Xj ) fP w (Xj ) − fˆ(Xj ) kXj 2 i 1 h = √ LPiw (fP w − fˆ) − LPjw (fP w − fˆ) 2
1 = √ LPiw − LPjw (fP w − fˆ). 2 96

Hence,   ∆res = L−1 P w ,λ



n n 1 XX

n

1 hij √ LPiw − LPjw (fP w − fˆ) . 2 i=1 j=1

1

− ∥∆res ∥L2 (Q) ≤ C1 ∥fP w − fˆ∥ LP w2 ,λ u HS ,

where u=

n n
1 XX 1 hij √ LPiw − LPjw . n 2 i=1 j=1

By Borell’s inequality under P(· | D), with probability at least 1 − η, s   r 1
− 21 −1 2 LP w ,λ u HS ≤ 1 + 2 log Eh LP w2 ,λ u HS . η And

n n 1 XX 1 − 12 2 Eh LP w ,λ u HS = 2 n 2 i=1 j=1 n X n  X

1 ≤ 2 n

−1

−1

2

LP w2 ,λ LPiw − LP w2 ,λ LPjw HS −1 2 LP w2 ,λ LPiw HS +

−1 2 LP w2 ,λ LPjw HS



i=1 j=1 n 2 X − 12 2 = LP w ,λ LPiw HS . n i=1

Proof of Lemma F.18. On the intersection of the data events in Lemmas F.16 and G.7, the second sample-size condition gives v s   u n p u 2 X −1/2 2 t ∥LP w ,λ LPiw ∥2HS ≲ κ2 τn Nλ log . n η i=1

Conditionally on the data, Lemma G.9 and the first bound in Lemma F.16 therefore imply   2
τn Nλ 2 ∥∆res ∥L2 (Q) ≲ C1 σ κ2s+3 1 + ∥g∗ ∥H √ log . η nλ Using fixed fractions of η for the two data events and the conditional multiplier event gives the asserted two-layer probability statement without changing the displayed logarithmic order.

G.19

Proof of Lemma F.19

Proof. Let H(· | A) be a regular conditional distribution of the pair (X ′ , Y ′ ) given A. Define its two marginal kernels by F (C | A) = H(C × S | A),

G(C | A) = H(S × C | A),

for Borel C ⊂ S. Then G(· | A) is a regular conditional distribution of Y ′ given A, and, because d

X | A = X ′ | A, F (· | A) is also a version of the regular conditional distribution of X given A. 97

The conditional Strassen–Dudley theorem of Monrad and Philipp [1991, Theorem 4] gives a random element Y , with conditional law G(· | A) given A, provided that E sup [F (C | A) − G(C r | A)] ≤ ε, C

where the supremum is over Borel sets C ⊂ S and C r = {s ∈ S : inf d(s, t) ≤ r}. t∈C

The extra uniform random variable in the assumptions is the randomization needed in that theorem to realize the resulting conditional transition kernel on the original probability space. It remains to check this displayed condition. Let Dr = {(s, t) ∈ S × S : d(s, t) ≤ r}. For any Borel set C ⊂ S, the set inclusion (C × S) ∩ Dr ⊂ S × C r holds by the definition of C r . Hence C × S ⊂ (S × C r ) ∪ Drc . Applying the conditional probability kernel H(· | A) to this inclusion gives, outside a fixed null set in the underlying probability space and simultaneously for all Borel C, F (C | A) − G(C r | A) ≤ H(Drc | A) = P {d(X ′ , Y ′ ) > r | A}. The right-hand side does not depend on C, and therefore the same bound holds after taking the supremum over Borel C. Hence E sup [F (C | A) − G(C r | A)] ≤ EP {d(X ′ , Y ′ ) > r | A} C

= P {d(X ′ , Y ′ ) > r} ≤ ε. The conditional Strassen–Dudley theorem applies and yields an S-valued random element Y satisd fying Y | A = Y ′ | A and P {d(X, Y ) > r} ≤ ε.

G.20

Proof of Lemma F.27

Proof. Because w(X) = min{ρ(X), τn }, w(X) 1− = ρ(X)

  τn 1− 1{ρ(X) ≥ τn }. ρ(X)

In particular, 

w(X) 1− ρ(X)

2 ≤ 1{ρ(X) ≥ τn }. 98

Let pm = (m − 1)/γ. Since γ ≥ 1, Lyapunov’s inequality and Definition 4.1 give 1/γ n o1/γ 2  1 m−2 2 pm γ(m−1) m!W σ . EQ ρ(X) ≤ EQ ρ(X) ≤ 2 Consequently, Markov’s inequality yields   w(X) 2 EQ 1 − ≤ Q{ρ(X) ≥ τn } ρ(X) ≤ τn−pm EQ ρ(X)pm  1 γ − m−1 1 γ m−2 2 ≤ τn m!W σ 2  1 γ 1 −(m−1) m−2 2 = τn m!W σ . 2

G.21

Proof of Lemma F.28

Proof. Since q = ρp and w(x) = min{ρ(x), τn }, Z Z w(x) w LP f = w(x)kx f (x) p(x) dx = kx f (x) q(x) dx. ρ(x) Hence, with a(x) := 1 − w(x)/ρ(x), Z LQ − LP w =

a(x) kx ⊗H kx Q(dx).

For Aλ = (LQ + λI)−1 , self-adjointness gives (kx ⊗H kx )Aλ = kx ⊗H Aλ kx . Thus (LQ − L

Pw

−1

)(LQ + λI)

Z =

a(x) kx ⊗H Aλ kx Q(dx).

Using ∥T ∥ ≤ ∥T ∥HS , Jensen’s inequality in Hilbert–Schmidt norm, and Cauchy–Schwarz, Z −1 w ∥(LQ − LP )(LQ + λI) ∥ ≤ |a(x)| ∥kx ⊗H Aλ kx ∥HS Q(dx) ≤ EQ a(X)2

1/2

EQ ∥kX ⊗H Aλ kX ∥2HS

1/2

.

For a rank-one operator, ∥u ⊗ v∥HS = ∥u∥ ∥v∥, and ∥kX ∥ ≤ κ. Therefore, EQ ∥kX ⊗H Aλ kX ∥2HS ≤ κ2 EQ ∥Aλ kX ∥2
= κ2 trH A2λ LQ ≤ κ2 λ−1 trH (Aλ LQ ) = κ2 λ−1 Nλ , because Aλ ⪯ λ−1 I. Combining this with Lemma F.27 yields  1 2γ 1/2 1 −(m−1) −1 −1/2 m−2 2 (LQ − LP w )(LQ + λI) ≤ κλ Nλ τn m!W σ . 2

99

G.22

Proof of Lemma F.29

Proof. Let ∆λ := (LQ − LP w )(LQ + λI)−1 . By Lemma F.28 with m = m0 , 1 2γ 1 −(m0 −1) m0 −2 2 ∥∆λ ∥ ≤ κλ τn m0 !W σ 2  1 2γ 1 −(m0 −1) − 1+d m0 −2 2 2 ≤ κEd λ , τn m0 !W σ 2 −1/2

1/2 Nλ



where the second inequality follows from Nλ ≤ Ed2 λ−d . The lower bound imposed on τn is exactly the condition that the last display is at most 1/2. This proves the first claim. For the second claim, use the factorization LP w + λI = LQ + λI − (LQ − LP w )
= I − ∆λ (LQ + λI). Since ∥∆λ ∥ ≤ 1/2, the operator I − ∆λ is invertible and ∥(I − ∆λ )−1 ∥ ≤

1 ≤ 2. 1 − ∥∆λ ∥

Consequently, ∥LQ (LP w + λI)−1 ∥ = LQ (LQ + λI)−1 (I − ∆λ )−1 ≤ ∥LQ (LQ + λI)−1 ∥∥(I − ∆λ )−1 ∥ ≤ 2, because 0 ⪯ LQ (LQ + λI)−1 ⪯ I. It remains to prove the third claim. By the Cordes inequality in Lemma F.26, applied with A = LQ , B = (LP w + λI)−1 , and s = 1/2, 1/2

∥LQ (LP w + λI)−1/2 ∥ ≤ ∥LQ (LP w + λI)−1 ∥1/2 ≤

G.23

Proof of Lemma G.10

Lemma G.10. Under Definition 4.2 with θ = 2, for τn > 0,  EQ

w(X) 1− ρ(X)

2

≤ τn−1 Ω.

Proof.  EQ

w(X) 1− ρ(X)

2

 Z  w(x) 2 = Q(dx) 1− ρ(x)   Z τn 2 = 1− Q(dx) ρ(x) ρ≥τn 100

√

2.

Z ≤

1 Q(dx) ρ≥τn

= Q(ρ(X) ≥ τn ) ≤ τn−1 EQ ρ(X) ≤ τn−1 Ω.

G.24

Proof of Lemma G.11

Lemma G.11. Under Definition 4.2 with θ = 2, for τn > 0 and λ > 0, 1/2

(LQ − LP w )(LQ + λI)−1 ≤ κλ−1/2 Nλ τn−1/2 Ω1/2 . Proof. The proof is identical to the Hilbert–Schmidt argument in Lemma F.28; the only input that changes is the moment bound for the truncated density ratio. Indeed, the same calculation gives " (LQ − LP w ) (LQ + λI)

−1

−1/2

≤ κλ

1/2 Nλ



EQ

w(X) 1− ρ(X)

2 #1/2 .

Applying Lemma G.10 yields 1/2

(LQ − LP w ) (LQ + λI)−1 ≤ κλ−1/2 Nλ τn−1/2 Ω1/2 .

G.25

Proof of Lemma G.12

Lemma G.12. Under Assumption 4.3 and Definition 4.2 with θ = 2, for 0 < λ ≤ 1, if τn ≥ 4κ2 ΩEd2 λ−(1+d) , then, as operators on H, 1. ∥(LQ − LP w )(LQ + λI)−1 ∥ ≤ 1/2; 2. ∥LQ (LP w + λI)−1 ∥ ≤ 2; 1/2

3. ∥LQ (LP w + λI)−1/2 ∥ ≤

√

2.

Proof. Let again ∆λ := (LQ − LP w )(LQ + λI)−1 . By Lemma G.11 and Nλ ≤ Ed2 λ−d , 1/2

∥∆λ ∥ ≤ κλ−1/2 Nλ τn−1/2 Ω1/2 1+d

≤ κEd Ω1/2 λ− 2 τn−1/2 . Condition (G.3) implies that the right-hand side is at most 1/2, proving the first claim.

101

(G.3)

The remaining two claims follow from the same deterministic perturbation argument as in Lemma F.29. Namely, ∥(I − ∆λ )−1 ∥ ≤ 2.

LP w + λI = (I − ∆λ )(LQ + λI), Therefore,

∥LQ (LP w + λI)−1 ∥ ≤ ∥LQ (LQ + λI)−1 ∥∥(I − ∆λ )−1 ∥ ≤ 2, and Lemma F.26 gives 1/2

∥LQ (LP w + λI)−1/2 ∥ ≤ ∥LQ (LP w + λI)−1 ∥1/2 ≤

G.26

√

2.

Proof of Lemma G.13

Lemma G.13. Under Definition F.1, for τn > 0,   w(X) 2 ≤ τn−(ϑ−1) Ωϑ . EQ 1 − ρ(X) Proof. Since w(X) = min{ρ(X), τn },  EQ

w(X) 1− ρ(X)

2 ≤ Q{ρ(X) ≥ τn } i h ≤ τn−(ϑ−1) EQ ρ(X)ϑ−1 ≤ τn−(ϑ−1) Ωϑ ,

where the second inequality follows from Markov’s inequality.

G.27

Proof of Lemma F.30

Proof. The Hilbert–Schmidt argument used in Lemma F.28 gives " 1/2

(LQ − LP w )(LQ + λI)−1 ≤ κλ−1/2 Nλ



EQ

w(X) 1− ρ(X)

The claim follows by applying Lemma G.13.

G.28

Proof of Lemma F.31

Proof. Let ∆λ := (LQ − LP w )(LQ + λI)−1 . By Lemma F.30 and Nλ ≤ Ed2 λ−d , ϑ−1 1/2 − 2

1/2

∥∆λ ∥ ≤ κλ−1/2 Nλ τn

Ωϑ

1+d

− ϑ−1 2

1/2

≤ κEd Ωϑ λ− 2 τn 102

.

2 #1/2 .

Condition (F.4) makes the last display at most 1/2, proving the first claim. The remaining claims follow from the deterministic perturbation argument in Lemma F.29. In particular, LP w + λI = (I − ∆λ )(LQ + λI), ∥(I − ∆λ )−1 ∥ ≤ 2. Consequently, ∥LQ (LP w + λI)−1 ∥ ≤ 2, and Lemma F.26 yields 1/2

∥LQ (LP w + λI)−1/2 ∥ ≤ ∥LQ (LP w + λI)−1 ∥1/2 ≤

G.29

Proof of Lemma F.20

Proof. Nλ = ≤ =

∞ X i=1 ∞ X i=1 ∞ X

µi µi + λ αi−β αi−β + λ α α + λiβ

Zi=1∞ ≤ 0

α dx α + λxβ

Let t = (λ/α)1/β x. Then  α 1/β Z 1

1 dt + Nλ ≤ β λ 0 1+t   α 1/β  1 1+ ≤ λ β−1 β = α1/β λ−1/β . β−1

G.30

Z ∞ 1

1 dt 1 + tβ

Proof of Lemma F.21

Proof. σ 2 (m, LQ ) =

∞ X

µi

i=m+1

Z ∞

α · t−β dt   1 1−β ∞ =α t 1−β m α 1−β = m . β−1 ≤

m

Taking square roots proves the claim. 103



√

2.

G.31

Proof of Lemma F.22

Proof. Let u = βxγ . Then Z ∞ e

−βxγ

a

1 dx = γβ 1/γ

Z ∞

u1/γ−1 e−u du.

βaγ

The standard incomplete-gamma tail estimate Z ∞ uq−1 e−u du ∼ tq−1 e−t ,

t → ∞,

t

with q = 1/γ, yields Z ∞

γ

γ

e−βx dx ∼

a

G.32

1 e−βa γ 1/γ−1 −βaγ (βa ) e = . βγaγ−1 γβ 1/γ

Proof of Lemma F.23

Proof. Nλ =

∞ X i=1

≤

∞ X i=1

=

∞ X

µi µi + λ α exp(−βiγ ) α exp(−βiγ ) + λ α α + λ exp(βiγ )

Zi=1∞ ≤

α dx α + λ exp(βxγ )

0 h i1 Z 1 ln( α ) γ Z ∞  β λ = + h 0

 

1 ln α β λ

i1 

( )

γ

α dx α + λ exp(βxγ )

1 Z α ∞ 1 α γ γ ≤ ln + h i 1 exp(−βx ) dx β λ λ 1 ln( α ) γ β λ  h
i 1 1 α     exp −β ln γ β λ 1 α α ≲ ln + · i γ−1 h
β λ λ γ βγ β1 ln αλ 1  1  1 α γ 1 1  α  γ −1 = ln + · ln β λ βγ β λ  1 1 γ ≲ log . λ 

The last comparison holds for all sufficiently small λ > 0, which gives constants λ0 ∈ (0, 1) and Cα,β,γ > 0 as stated. 104

G.33

Proof of Lemma F.24

Proof. σ 2 (m, LQ ) =

∞ X

µi

i=m+1 Z ∞

≤

α · exp(−βtγ ) dt

m 2 ≤ Cα,β,γ m1−γ exp(−βmγ ).

Here Lemma F.22 gives the last inequality for all sufficiently large m; increasing Cα,β,γ covers the finitely many remaining positive integers. Taking square roots proves the claim.

G.34

Proof of Lemma F.25

Proof. As in the proof of Lemma F.7, V = V1 + V2 , V1 ⪯L2 (Q) τn σ 2 · T1 , V2 ⪯L2 (Q) τn κ2 max{κ2s−1 , 1}2 λ2 min{s,1/2} ∥g∗ ∥2 T1 ,   V1 + V2 ⪯L2 (Q) τn σ 2 + κ2 max{κ2s−1 , 1}2 λ2 min{s,1/2} ∥g∗ ∥2 T1 . where T1 = (LP w + λI)−1 LP w (LP w + λI)−1 LQ . ∀ f ∈ L2 (Q), ⟨f, T1 f ⟩L2 (Q) = f, (LP w + λI)−1 LP w (LP w + λI)−1 LQ f L2 (Q) = LQ f, (LP w + λI)−1 LP w (LP w + λI)−1 LQ f   1 ≤ LQ f, LQ f λ 1 = ⟨f, LQ f ⟩L2 (Q) . λ Thus, 1 LQ . λ By the min–max eigenvalue monotonicity for compact positive operators and the identity T1 ⪯L2 (Q)

σL22 (Q) (m, LQ ) = σ 2 (m, LQ ), we have σL22 (Q) (m, V) ≤

 τn  2 σ + κ2 max{κ2s−1 , 1}2 λ2 min{s,1/2} ∥g∗ ∥2 σ 2 (m, LQ ). λ

105

G.35

Proof of Lemma A.1

Proof. h i
ei = (LP w + λI)−1 w(Xi ) εi kX + w(Xi ) kX ⊗ kX (f∗ − fP w ) U i i i
= w(Xi ) εi + f∗ (Xi ) − fP w (Xi ) (LP w + λI)−1 kXi . Fix t ∈ L2 (Q). Since (LP w + λI)−1 LQ t ∈ H is deterministic, the weighted-feature part of Assumption A.1 and [Vershynin, 2018, Proposition 2.5.2] imply   w(Xi )kXi , (LP w + λI)−1 LQ t − E w(Xi )kXi , (LP w + λI)−1 LQ t ψ2  h i1/2 2 . ≤ Csg b0 E w(Xi )kXi , (LP w + λI)−1 LQ t Here the condition applies with w = wn = wτn , since τn ≥ τ0 . The same bound holds for the uncentered scalar projection after changing the universal constant, because b0 ≥ 1 and the absolute mean is bounded by the square root of the second moment. ei , LQ t⟩ ei , t⟩L2 (Q) = ⟨U ⟨U ≤ (σ + κ∥f∗ ∥) ⟨w(Xi ) (LP w + λI)−1 kXi , LQ t⟩ = (σ + κ∥f∗ ∥) ⟨w(Xi )kXi , (LP w + λI)−1 LQ t⟩ . ei , t⟩L2 (Q) ⟨U ≤ (σ + κ∥f∗ ∥) ⟨w(Xi )kXi , (LP w + λI)−1 LQ t⟩ ψ2 ψ2  h i1/2 2 ≤ Csg · b0 (σ + κ∥f∗ ∥) E w(Xi )kXi , (LP w + λI)−1 LQ t io1/2 σ + κ∥f∗ ∥ n h E ⟨Ui , t⟩2L2 (Q) . ≤ Csg · b0 σ where we used Lemma F.3. ei − EU ei , centering changes only the universal constant. Hence Since Ui = U i1/2 σ + κ∥f∗ ∥  h 2 ⟨Ui , t⟩L2 (Q) ψ2 ≤ Csg · b0 E ⟨Ui , t⟩2L2 (Q) . σ 3 b Again by [Vershynin, 2018, Proposition 2.5.2], Ui is L2 (Q)-Csg 0

G.36

σ + κ∥f∗ ∥ -sub-Gaussian. σ

Proof of Lemma G.1

Proof. By the spectral decomposition of LQ , L2 (Q) = Ran(LQ ) ⊕⊥ L2 (Q) ker(LQ ),

Ran(LQ ) = span{ϕj : µj > 0}.

Moreover, when H is viewed as a subspace of L2 (Q), Lemma 2.1 gives H ⊂ Ran(LQ ). We first record the identity ⟨u, h⟩L2 (Q) = ⟨LQ u, h⟩H , u ∈ L2 (Q), h ∈ H. (G.4) P P Indeed, writing u = j uj ϕj in L2 (Q) and h = µj >0 hj ϕj ∈ H, the kernel component of u is orthogonal to h, and X µj uj hj X ⟨LQ u, h⟩H = = uj hj = ⟨u, h⟩L2 (Q) . µj µj >0

µj >0

106

Now take f, g ∈ L2 (Q) and decompose f = fr + f0 , g = gr + g0 , with fr , gr ∈ Ran(LQ ) and f0 , g0 ∈ ker(LQ ). Since LQ f0 = LQ g0 = 0 and T LQ fr , T LQ gr ∈ H ⊂ Ran(LQ ), the kernel components do not contribute to the L2 (Q) inner products. Hence, by (G.4) and the self-adjointness of T on H, ⟨f, T LQ g⟩L2 (Q) = ⟨fr , T LQ gr ⟩L2 (Q) = ⟨LQ fr , T LQ gr ⟩H = ⟨T LQ fr , LQ gr ⟩H = ⟨T LQ fr , gr ⟩L2 (Q) = ⟨T LQ f, g⟩L2 (Q) . Thus T LQ is self-adjoint in L2 (Q).

G.37

Proof of Lemma G.2

Proof. As LP w and LPnw are self-adjoint, we have 1

1

2

(LPnw + λI)− 2 (LP w + λI) 2 D E 1 1 1 1 = sup (LPnw + λI)− 2 (LP w + λI) 2 f, (LPnw + λI)− 2 (LP w + λI) 2 f ∥f ∥=1

= sup

D

∥f ∥=1

1

1

f, (LP w + λI) 2 (LPnw + λI)−1 (LP w + λI) 2 f 1

E

1

= (LP w + λI) 2 (LPnw + λI)−1 (LP w + λI) 2 . Combining this with Lemma F.13, we have the desired result.

G.38

Derivation of Equation (3.2)

Starting from the first representation of B in (3.2), we have n n Vbi − Vbj 1 XX hij √ n 2 i=1 j=1   n n n n X X X X 1 1 =√ ·  Vbi hij − Vbj hij  2 n i=1 j=1 j=1 i=1   n n n n X X 1 1 X b X Vi hij − Vbi hji  =√ · 2 n

B=

i=1

n 1X

1 =√ · 2 n i=1

j=1

i=1

j=1

n X Vbi (hij − hji ) j=1

1 = √ vx⊤ diag(b ε)(h − h⊤ )1. 2 where



1 1 w(X1 )(LPnw + λI)−1 kX1 , . . . , w(Xn )(LPnw + λI)−1 kXn n n 1 = (LPnw + λI)−1 k⊤ x W. n

vx⊤ =

107



As

(LPnw + λI)k⊤ x = [(LPnw + λI)kX1 , . . . , (LPnw + λI)kXn ] n 1X LPnw kXj = w(Xi )k(Xi , Xj )kXi n i=1

1 = (k⊤ WK)j , n x we have ⊤ (LPnw + λI)k⊤ x = kx ⊤ (LPnw + λI)−1 k⊤ x = kx



1 WK + λI n





1 WK + λI n

−1 .

Then, −1 ⊤ −1 −1 vx⊤ = k⊤ x (WK + nλI) W = kx (K + nλW ) .

108

Related documents

• A Reproducing-Kernel-Based Nonparametric Test for Conditional Independence of Functional Data#974353
• Distribution-free testing of linear type#799309
• Distribution-free testing of linear type#797886
• Estimation and goodness-of-fit testing for non-negative random variables with explicit Laplace transform#1116057
• Compound decisions and empirical Bayes via Bayesian nonparametrics#996882