Transfer Learning in Nonparametric Regression with Deep ReLU Networks

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

Junpeng Ren 1 Carlos Misael Madrid Padilla 2 Yanzhen Chen 3 Oscar Hernan Madrid Padilla 1

arXiv:2608.20255v1 [stat.ML] 20 Aug 2026

Abstract

empirical success in fields such as natural language processing (Daumé III, 2007; Howard & Ruder, 2018), computer vision (Gong et al., 2012; Tzeng et al., 2017), bioinformatics (Schweikert et al., 2008), transportation (Lu et al., 2019), and epidemiology (Apostolopoulos & Mpesiana, 2020). Consequently, enhancing nonparametric regression through transfer learning becomes a meaningful problem. Meanwhile, the growing complexity of modern data is also reflected in higher dimensionality, as exemplified by text and image data. Deep neural networks (DNNs), as a cornerstone class of nonparametric estimators, have demonstrated outstanding empirical performance on such tasks (Krizhevsky et al., 2012; Vaswani et al., 2017; Radford et al., 2018) and have been theoretically shown to possess strong advantages in modeling such high-dimensional nonlinear structures (Schmidt-Hieber, 2020; Kohler & Langer, 2021; Chen et al., 2022). Therefore, studying how transfer learning can further improve neural network estimators provides a practically meaningful instantiation of this problem. Motivated by these developments, this work investigates nonparametric regression under the transfer learning framework, focusing on deep neural networks as the primary estimator.

This paper develops a general transfer learning framework for nonparametric regression with data consisting of multiple groups. Under the assumption that groups share a common structure along with group-specific deviations in additive form, the proposed method employs a two-stage offset learning procedure: the first stage pools data from all groups to estimate an overall mean function, and the second stage estimates offsets for each group, yielding final group-level estimators through additive combination. Upper bounds on the L2 error are established for the proposed framework, covering a broad class of nonparametric estimators under mild complexity and noise conditions. When instantiated with deep ReLU networks, explicit convergence rates are derived under hierarchical composition models, demonstrating the ability to overcome the curse of dimensionality. Conditions that enable positive transfer with faster rates are considered, including learning with simpler functions and data augmentation through pooling samples across groups. Various simulations and real-data experiments further validate the effectiveness of the proposed method.

We formalize nonparametric regression with data composed of distinct groups that share structural similarities. Given n independent copies of (X, Y, Z), denoted by {(xi , yi , zi )}ni=1 , where xi ∈ X ⊂ Rp represents covariates, yi ∈ R is the response, and zi ∈ {1, . . . , L} indexes the group membership, we assume the data are generated from the model

1. Introduction Nonparametric regression is a flexible modeling paradigm that captures complex data relationships without assuming specific functional forms. With the increasing diversity of modern data, transfer learning provides an effective framework for leveraging information from related sources to improve performance on a target, achieving remarkable

yi = f0 (xi ) + f0,zi (xi ) + ϵi ,

(1)

where ϵi are independent errors with E(ϵi | xi , zi ) = 0. Here, f0 represents a shared function common to all groups, while f0,ℓ denotes the group-specific deviation for group ℓ ∈ {1, . . . , L}. Our goal is to estimate the group-specific conditional mean function for each group:

1

Department of Statistics and Data Science, University of California, Los Angeles, California, USA 2 Department of Statistics and Data Science, Washington University in St. Louis, Missouri, USA 3 Department of Information Systems, Business Statistics and Operations Management, Hong Kong University of Science and Technology, Hong Kong, China. Correspondence to: Oscar Hernan Madrid Padilla <oscar.madrid@stat.ucla.edu>.

g0,ℓ (x) := E(Y | X = x, Z = ℓ) = f0 (x) + f0,ℓ (x). (2) To tackle this problem, we propose a two-stage offset transfer learning framework inspired by pretraining. In the first stage, we pool data from all L groups to estimate the overall conditional mean function, defined as the average of the conditional mean functions across all groups. In the second stage, for each target group ℓ, we use its group-specific data

Proceedings of the 43 rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026. Copyright 2026 by the author(s).

1

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

to estimate the offset between the overall mean function and the group-specific conditional mean. The final estimator combines the overall mean function with the estimated group-specific offset in an additive form.

Feng, 2023; Li et al., 2024), functional linear regression (Lin & Reimherr, 2022), and graphical models (Li et al., 2023). Various regularization schemes have also been explored to facilitate transfer, including ℓ1 (Craig et al., 2026), ℓ2 (Duan & Wang, 2023), and graph-based penalties (Dinh et al., 2022).

Our main contributions are summarized as follows: General Theoretical Framework. We study a general transfer learning framework for nonparametric regression. In Theorem 3.2, we derive an L2 error upper bound for a broad class of nonparametric estimators under mild complexity and noise conditions that accommodate subexponential noise. We additionally study the variant where the two stages are estimated on independent data via sample splitting, and provide a complementary general bound (Theorem 3.7) that yields a tighter rate. Notably, our framework accommodates the regime in which the number of groups L grows with the total sample size n, capturing the essence of modern machine learning on data of growing size and diversity, in contrast to existing transfer learning results that focus on a fixed number of groups. As a direct consequence of the generality of our framework, we yield the first convergence guarantees for trend filtering (Tibshirani, 2014) in a transfer learning setting (Appendix D.1), and recover the existing convergence rates established by (Wang et al., 2016) for orthogonal series regression using Sobolev sieves up to logarithmic factors (Appendix D.2).

In the nonparametric regime, (Cai & Wei, 2021) investigates transfer learning for classification, while a series of works (Wang & Schneider, 2015; Wang et al., 2016; Du et al., 2017; Lin & Reimherr, 2024; Cai & Pu, 2024) focus on nonparametric regression with kernel-based estimators. Specifically, (Wang & Schneider, 2015) introduces a twostage kernel ridge regression (KRR) framework that models the target function as an additive combination of a source function and an offset, demonstrating that transfer learning can improve estimation when the offset is smoother than the target. (Wang et al., 2016) further formalizes this framework under Sobolev smoothness assumptions. Subsequent work (Du et al., 2017) generalizes it via a nonlinear transformation linking the source and target functions, and (Lin & Reimherr, 2024) develops adaptive KRR algorithms that automatically adjust to unknown smoothness levels. Beyond KRR, (Cai & Pu, 2024) considers transfer learning using local polynomial estimators. In contrast to these studies, our theory establishes convergence rates for general nonparametric estimators and achieves faster rates that overcome the curse of dimensionality for hierarchically composed function classes by leveraging deep neural networks.

Transfer Learning with Dense ReLU Networks. Building on the general theory, in Theorem 3.4 and Corollary 3.8 we derive explicit upper bounds for transfer learning with dense ReLU neural networks under hierarchical composition models (Kohler & Langer, 2021), showing that the ability to overcome the curse of dimensionality is preserved within the transfer learning framework. As emphasized in (Cagnetta et al., 2024; Danhofer et al., 2025), such property remains critical for explaining the empirical success of DNNs, even as dataset sizes have scaled dramatically in modern architectures such as ImageNet (Deng et al., 2009), ResNet (He et al., 2016), and GPT (Brown et al., 2020). We further identify regimes in which transfer learning with DNNs achieves strictly faster convergence rates than singlegroup estimation, providing theoretical justification for the empirical success of pretrained models. Through extensive simulations across diverse scenarios and two real-data applications, we demonstrate that the proposed estimators consistently outperform a range of competitors.

Another important line of research studies transfer learning through shared representations, where source and target tasks are assumed to depend on a common low-dimensional structure along with task-specific prediction functions (Maurer et al., 2016; Du et al., 2020; Tripuraneni et al., 2020; 2021; Xu & Tewari, 2021; Tian et al., 2023). However, most previous studies in this line discuss neural networks under parametric formulations, including the analyses in (Tripuraneni et al., 2020; Xu & Tewari, 2021). A recent work (Jiao et al., 2024) studies nonparametric transfer with deep ReLU networks under a representation learning framework. Our work differs by adopting an additive formulation and offering relaxed conditions on noise and network constraints, establishing rates that overcome the curse of dimensionality. Transfer learning with shared representations is also closely tied to multi-task learning, which jointly learns multiple related tasks in parallel to improve performance, often by assuming shared structures across tasks, with a rich body of early work (Caruana, 1997; Baxter, 2000; Evgeniou & Pontil, 2004; Argyriou et al., 2008). The multi-task learning setting typically treats tasks symmetrically and aims to estimate functions for all tasks jointly without distinguishing source and target, in contrast to the transfer learning formulation of leveraging a data-rich source for a data-scarce

1.1. Related Literature Transfer Learning. There are vast works of transfer learning having been discussed, including but not limited to transfer learning in models like linear regression (Chen et al., 2015), high dimensional linear regression (Gross & Tibshirani, 2016; Bastani, 2021; Li et al., 2022; Lai et al., 2024), high dimensional generalized linear model (Tian & 2

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

K ⊆ X and r S > 0. A subset C ⊂ X is an r-external cover of K if K ⊆ x∈C Br (x, d), where Br (x, d) denotes the ball of radius r centered at x. The external covering number N (r, K, d) is the minimal cardinality among all r-external covers of K. For any index set I ⊆ {1, . . . , n}, we use ∥f ∥I to denote the empirical L2 -norm computed over the subset of data points {xi : i ∈ I}. For a function f : Rd → R and An > 0, let fAn denote the truncation of f at level An , defined as fAn (x) = max{−An , min{f (x), An }}.

target. However, the distinction between transfer learning and multi-task learning is not sharply drawn in the literature; for instance, the works cited above (Tripuraneni et al., 2020; Jiao et al., 2024) use transfer learning terminology while adopting symmetric multi-task learning setups, or analyze transfer to a data-scarce target within a multi-task framework. We retain the terminology of transfer learning throughout this work. Nevertheless, our framework accommodates both regimes within a unified theory: the target group proportion over the whole dataset is allowed to vanish as the total sample size n grows (transfer learning with data-scarce target), while our algorithm simultaneously produces estimators for all groups on equal footing (multi-task learning).

1.3. Outline The remainder of this paper is organized as follows. Section 2 introduces the general methodology of the proposed transfer learning framework. Section 3.1 establishes general theoretical upper bounds for our estimator, covering a broad class of nonparametric estimators. Section 3.2 provides concrete upper bounds specialized to deep ReLU networks, with discussion on conditions for positive transfer. Section 3.3 further provides theoretical results for the independent two-stage transfer learning estimator based on data splitting. Section 4 presents simulation studies and real-data experiments, comparing the proposed transfer learning approach with alternative training strategies and estimators in both low- and high-dimensional settings. Finally, Section 5 concludes the paper and discusses potential extensions.

Deep Neural Networks as Nonparametric Estimators. A fundamental challenge in nonparametric estimation is the curse of dimensionality (Donoho et al., 2000). Recent theoretical work shows that deep ReLU networks can mitigate this issue under structured function classes, such as hierarchical composition and low-dimensional manifold assumptions (Schmidt-Hieber, 2020; Kohler & Langer, 2021; Chen et al., 2022; Padilla et al., 2022; Jiao et al., 2023; Padilla et al., 2024b;a). In particular, hierarchical composition has been identified as a natural and practically relevant assumption for explaining the empirical success of deep networks (Cagnetta et al., 2024). Building on this line of research, we study transfer learning with deep ReLU networks under hierarchical composition models and establish explicit convergence rates within our two-stage framework.

2. Methodology We formalize our transfer learning framework based on a two-stage offset learning procedure, which shares similarities with the approaches in (Wang & Schneider, 2015; Wang et al., 2016; Du et al., 2017; Lin & Reimherr, 2024). Under model (1), we define the overall mean function f¯(x) := E(Y | X = x), which can be written as

1.2. Notation The following notation is used throughout the article: Z+ denotes the set of positive integers, and N denotes the set of natural numbers. For two positive sequences an and bn , let an = O(bn ) or an ≲ bn when an ≤ Cbn for some constant C > 0 which is independent of n, and an = Θ(bn ) or an ≍ bn when an = O(bn ) and bn = O(an ). Besides, we write an = O(poly(log n)) if there exists a polynomial h such that an = O(h(log n)). For a sequence of random variables Xn and a positive sequence an , we write Xn = OP (an ) if for any ε > 0, there exist constants M > 0 and N > 0 such that P(|Xn | > M an ) < ε for all n > N . The L∞ -norm of a function f (·) : Rd → R is defined by ∥f ∥∞ = supx∈Rd |f (x)|. Given the probability distribution PX of the random vector X over X , define the
1/2 R L2 (PX )-norm as ∥f ∥L2 (PX ) := X f 2 (x)PX (dx) =  2 1/2 n := E f (X) . Besides, given n random samples x1 {xi }ni=1 independently and identically distributed according to PX , the corresponding Pn empirical probability measure is Pn (xn1 ) := n1 i=1 δxi (x). Define the empir
1/2 Pn ical L2 -norm as ∥f ∥L2 (Pn ) := n1 i=1 f 2 (xi ) =
1/2 R 2 f (x)Pn (dx) . For a metric space (X , d), let X

f¯(x) = f0 (x) +

L X ℓ=1

P(Z = ℓ | X = x) f0,ℓ (x).

In the first stage, we use a nonparametric estimator to estimate f¯ based on data pooled from all groups. Let F denote the function class of the nonparametric estimator. We construct the estimator as ( n ) X 2 ˆ f := arg min (yi − f (xi )) , (3) f ∈F

i=1

and consider its clipped version fˆAn with An > 0. Next, in the second stage, for ℓ = 1, . . . , L, we construct groupspecific estimators for the offset between the overall mean E(Y |X = x) and the group-specific mean function g0,ℓ as ( ) X 2 ˆ ˆ fℓ := arg min (yi − fA (xi ) − f (xi )) , (4) f ∈Fℓ

3

n

i : zi =ℓ

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

where Fℓ denotes the function class for the group-specific nonparametric estimator. The group data used in this stage may be either identical or independent of the data used in the first stage. As in the first stage, we consider the truncated version of the second-stage estimator, denoted by fˆℓ,Bn . The final estimator, which is our main object of theoretical interest, is then constructed as the sum of the two estimators: ĝℓ (x) := fˆAn (x) + fˆℓ,Bn (x).

is obtained. Leveraging Theorem 1 of (Padilla et al., 2024a), under mild complexity conditions on F, we obtain ∥f¯ − fˆAn ∥2L2 = OP (rn ), where the rate rn consists of an approximation error term ¯ ˜ ϕn , defined √ via the existence of f ∈ F satisfying ∥f − ˜ f ∥∞ ≤ ϕn , and an estimation error term determined by the complexity of F (see Theorem B.1 in the Appendix for a formal statement).

(5)

2.1. Background of Deep ReLU Neural Networks as Nonparametric Estimators

In the second stage, for each group ℓ ∈ {1, . . . , L}, we use a separate group-specific nonparametric function class Fℓ to estimate the offset between overall mean and group mean, which is

We now introduce the basic notation for deep ReLU neural networks, which serve as the primary class of nonparametric estimators in this paper. Let τ (x) = max{0, x} denote the ReLU activation function, applied componentwise to vectors. A fully connected feedforward neural network with M hidden layers defines a function

Gℓ (x) := f0,ℓ (x) −

L X

f0,k (x) P(Z = k|X = x). (6)

k=1

This results in the estimator fˆℓ,Bn in (4). Building upon the first-stage bound, we now establish bounds for the final group-specific estimator ĝℓ .

f = ΦM ◦ τ ◦ ΦM −1 ◦ · · · ◦ τ ◦ Φ1 , where each Φs : Rws−1 → Rws is an affine map of the form Φs (x) = Ws x + bs . Here w0 = d, wM = 1, Ws ∈ Rws ×ws−1 , and bs ∈ Rws .

Theorem 3.2. Under the conditions of Theorem B.1, and where fˆ has been constructed as in (3). Suppose for ℓ ∈ e ℓ ∈ Fℓ such that {1, . . . , L} there exists G p e ℓ ∥∞ ≤ ϕℓ,n , ∥Gℓ − G

Following (Kohler & Langer, 2021; Padilla et al., 2024a), we restrict attention to dense architectures in which all hidden layers have the same width ν. We denote by F(M, ν) the corresponding class of deep ReLU networks. Under the proposed transfer learning framework, the first-stage estimator fˆ is obtained by empirical risk minimization over F(M, ν), while the second-stage group-specific offset estimator fˆℓ is obtained from an analogous class F(Mℓ , νℓ ).

where ϕℓ,n denotes the approximation error. Let ηℓ,n : R+ → R+ for ℓ = 1, . . . , L be functions such that ∀ δ ∈ (0, 1), max

nπ ℓ 2 ≤ nℓ ≤ 2nπ ℓ

sup {xi }i∈Iℓ

log N (δ, Fℓ,Bn , ∥·∥Iℓ ) ≤ ηℓ,n (δ),

(7) where nℓ = |Iℓ |, where Fℓ,Bn := {fBn /(2Bn ) : f ∈ Fℓ }, and Bn is chosen to be sufficiently large.

3. Theory In this section, we provide theoretical guarantees for the two-stage offset learning framework introduced in Section 2. Suppose the covariate space is X = [0, 1]d , which is standard in nonparametric theory (Györfi et al., 2002). We begin with a mild overlap assumption requiring the group membership probabilities to be bounded away from 0 and 1 given the covariates, similar to the overlap condition commonly assumed in causal inference (Rosenbaum & Rubin, 1983).

Suppose the function classes Fℓ,Bn and FAn satisfy appropriate complexity conditions (see Theorem B.2 for a formal statement), and P(∥ϵ∥∞ > Un ) → 0 as n → ∞, for some Un > 0. Then, for δ ∈ (0, 1),   Un2 ηℓ,n (δ) 2 2 2 ∥g0,ℓ −ĝℓ ∥L2 = OP ϕℓ,n + Bn δ + + rn nπ ℓ (8) provided that δ 2 nπ ℓ → ∞.

Assumption 3.1. For every ℓ ∈ {1, . . . , L} there exists π ℓ , π ℓ such that π ℓ ≍ π ℓ and ∀x ∈ X ,

Theorem 3.2 establishes a general upper bound for twostage transfer learning, where the convergence rate in (8) decomposes into the first-stage error rn and the other terms as errors from the second-stage estimation of Gℓ , with ϕℓ,n and Bn2 δ 2 + Un2 ηℓ,n (δ)/(nπ ℓ ) denoting the approximation and estimation errors, respectively. The result holds under δ 2 nπ ℓ → ∞, a mild condition which implies that the expected number of observations in group ℓ grows to ∞. In

0 < π ℓ < P(Z = ℓ|X = x) < π ℓ < 1. 3.1. General Result Our general theoretical analysis proceeds by first establishing a bound for the first-stage estimator fˆAn and then leveraging it to provide an upper bound for the final estimator ĝℓ in (5). Let F denote a generic function class from which fˆ 4

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

addition to complexity conditions on Fℓ similar in form to those in Theorem B.1, the two-stage estimation procedure requires two additional localized complexity constraints, namely conditions (19) and (20) in the detailed version of Theorem 3.2, imposed on the function classes F and Fℓ with respect to group ℓ. These complexity conditions are mild, and in Section 3.2 we demonstrate that they are satisfied by standard nonparametric function classes, such as deep ReLU neural networks. Moreover, in Section 3.3, we show that sample splitting can further relax these constraints and potentially accelerate the convergence rate.

deep neural networks. Theorem 3.4. Suppose that the conditions of Theorem B.3 hold. Let −2p (10) ϕℓ,n = max (nπ ℓ ) (2p+K) . (p,K)∈Pℓ

Then there exists sufficiently large positive constants c3 and c4 such that if Mℓ and νℓ are appropriately chosen (see Theorem B.4 for a formal statement), then ĝℓ as defined in (5) with Fℓ := F(Mℓ , νℓ ), satisfies, 
∥g0,ℓ −ĝℓ ∥2L2 = OP ϕℓ,n max{Un2 , Bn2 } log4 Bn2 nπ ℓ
 3 2 2 + ϕn π −1 . max{A , U } log (n) log A nπ ℓ n n n ℓ (11) provided that log n ≲ nπ ℓ , and Bn is chosen to satisfy (18). Remark 3.5. Suppose that the following quantity

3.2. Theory for Transfer Learning with Deep ReLU Networks Our proposed general theorems serve as powerful tools for studying the convergence rates of nonparametric estimators within a two-stage transfer learning framework. In this section, we focus on deep ReLU networks as nonparametric estimators in both stages to further investigate the properties of the proposed framework. Throughout, we assume that the overall mean and offset functions satisfy a hierarchical composited structure (Kohler & Langer, 2021). Assumption 3.3. Assume the overall mean f¯ and offset Gℓ for ℓ ∈ {1, . . . , L} satisfy the hierarchical compositional structure (see Appendix A for a formal statement), with f¯ ∈ H(l0 , P0 ) and Gℓ ∈ H(Rℓ , Pℓ ). Furthermore, assume that each component function m in the definition of f¯ or Gℓ can have different smoothness pm = qm + sm , for qm ∈ N, sm ∈ (0, 1], and of potentially different input dimension Km , so that (pm , Km ) ∈ P0 ∪ (∪ℓ Pℓ ). Let Kmax be the largest input dimension and pmax the largest smoothness of any of the functions m. Suppose that all the partial derivatives of order less than or equal to qm are uniformly bounded by constant c2 , and each function m is Lipschitz continuous with Lipschitz constant CLip ≥ 1. Also, assume that max{pmax , Kmax } = O(1).

max{Un , An , Bn , ∥f¯∥∞ , max ∥f0,ℓ ∥∞ , ∥Gℓ ∥∞ }, ℓ=1,...,L

(12) grows at most in O(poly(log n)). This is a general setting that includes the case where the error distribution can be sub-Gaussian or even sub-exponential. Let us now compare the result in Theorem 3.4 (based on pretraining) with the alternative approach of estimating each g0,ℓ in (2) separately: X
2 ĥℓ := arg min yi − g(xi ) . (13) g∈Fℓ

The convergence rate of ĥℓ depends on the approximation error of the class Fℓ relative to g0,ℓ (x) = f0 (x) + f0,ℓ (x) as well as the estimation error (e.g., for ReLU neural networks) based on approximately nπ ℓ observations, assuming π ℓ ≍ π ℓ . In contrast, under Theorem 3.4—ignoring logarithmic factors and assuming π ℓ ≍ 1—the rate for our estimator ĝℓ becomes ϕn + ϕℓ,n , where ϕn , as defined in (9), is the rate for estimating f¯ based on n samples, and ϕℓ,n is the convergence rate for estimating Gℓ (x) based on approximately nπ ℓ ≍ n samples.

With Assumption 3.3 in place, we are ready to present our results. Specifically, the convergence rate for the first-stage estimator fˆAn , which is estimated by deep ReLU networks class F(M, ν) with appropriately chosen depth and width, is established by leveraging Theorem 2 of (Padilla et al., 2024a). Consequently, it holds that ∥f¯− fˆAn ∥2L2 = OP (rn ), where the rate rn depends only on ϕn =

max n−2p/(2p+K) ,

(p,K)∈P0

i:zi =ℓ

Thus, when the functions f¯ and Gℓ are less complex than g0,ℓ , pretraining can yield better performance than estimating each g0,ℓ separately via ĥℓ . This scenario is reasonable: f¯ involves a weighted average of the difference functions f0,k , which can mitigate the influence of extreme values associated with a particular f0,k , while Gℓ may be small or close to zero in regions where the functions f0,k take similar values.

(9)

the approximation error of f¯ by F(M, ν), with An and Un chosen appropriately. We refer readers to Theorem B.3 for a formal statement.

As another example, consider the case where ϕn ≲ πℓ , ϕℓ,n

Building upon this first-stage result, we present the upper bound for the final transfer-learning estimator ĝℓ of the group-specific conditional mean, also constructed using

for some fixed ℓ ∈ {1, . . . , L}, which can allow for π ℓ → 0. Then the convergence rate of our estimator depends only 5

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

on estimating the function Gℓ (x) using a neural network class and a sample size of the same order as that required for estimating g0,ℓ with a neural network class. In this setting, the comparison between our pretraining-based estimator ĝℓ and the naive estimator ĥℓ reduces to determining which function is easier to estimate with the same number of samples using neural networks. When g0,ℓ is not substantially different from the functions {g0,k }, the function Gℓ is typically easier to estimate than g0,ℓ , as in problems with shared information across groups (see 6)—precisely the type of setting in which pretraining is most beneficial. Remark 3.6. Our upper bounds demonstrate two key points: (1) the convergence rate of transfer learning with deep ReLU networks can overcome the curse of dimensionality, and (2) the benefits of transfer learning. However, we do not claim optimality. We emphasize that this is not a limitation of our work. In fact, lower bounds for the hierarchical function class are only known in very specific cases (see Remark 2 in (Kohler & Langer, 2021)).

ϕn + ϕℓ,n . This rate naturally decomposes into two components: the rate for estimating f¯ based on n samples, plus the rate for estimating Gℓ based on nπ ℓ samples. The key difference from Theorem 3.4 is that here we are able to remove the factor π ℓ from the denominator on the right-hand side of (11). This distinction matters primarily when π ℓ → 0. If the latter holds, the first stage estimation is done with significantly more samples than if the estimation was done only using the data from the ℓth group (n vs nπ ℓ ), while the second stage estimation is done with the same number of samples (roughly nπ ℓ ) though potentially for estimating a simpler function. We conjecture that the appearance of π ℓ in Theorem 3.4 is an artifact of the proof. Indeed, in practice we observe that the estimator performs better without sample splitting—that is, in the setting considered in Theorem 3.4—than with sample splitting, as in Corollary 3.8. That said, both results coincide and yield the same rate when π ℓ ≍ 1. Remark 3.10. We illustrate the benefit of transfer learning with DNNs in comparison with classical estimators whose rates depend on the ambient dimension and therefore suffer from the curse of dimensionality, such as transfer learning by kernel smoothing (Du et al., 2017), through a simple example applying Corollary 3.8. Let X ∈ [0, 1]10 and ℓ ∈ {1, 2, 3} index three groups. For simplicity, suppose that π̄ℓ = π ℓ = πℓ for all ℓ. Consider g0,ℓ (x) = f0 (x) + f0,ℓ (x) with f0 (x) = |x1 + · · · + x10 − 1/2| and f0,ℓ (x) = ℓ · |x1 + x2 + x3 + x4 − 1/2|3/2 . Then g0,ℓ and f¯ has smoothness p = 1 on the full K = 10 ambient space, while the Stage-2 offset G1 (x) = (1 − π1 − 2π2 − 3π3 ) |x1 + x2 + x3 + x4 − 1/2|3/2 for group 1 is much simpler, with smoothness p = 3/2 and intrinsic dimension K = 4.1 Table 1 provides the convergence rates of four estimators across regimes. As π1 shrinks, classical TL estimator degrades to (nπ1 )−3/13 , while NN with TL remains at n−1/6 until π1 ≪ n−11/18 and then transitions to (nπ1 )−3/7 , strictly faster than classical TL. This example clearly illustrates how a small πℓ can drastically worsen the rate of transfer learning using classical estimators while NN with TL remains robust. We refer readers to Appendix E for a detailed explanation of this example and other concrete examples illustrating the rate ϕn + ϕℓ,n using DNNs.

3.3. Independent Two Stages Estimator In this subsection, we assume that the estimator fˆ in (3) is computed using data independent of that in (4), e.g., via sample splitting. Theorem 3.7. Consider the conditions and notations from Theorem 3.2. However, instead of assuming that (20) holds, suppose that fˆ is computed with data independent of that used in (4). Then ∥g0,ℓ − ĝℓ ∥2L2 is of order   Un2 ηℓ,n (δ) A2n log n 2 2 OP ϕℓ,n + rn + + + Bn δ . nπ ℓ nπ ℓ It is important to note that if fˆ is computed using data independent of that used in (4), the conclusion of Theorem 3.2 continues to hold under its original assumptions. The main difference between the upper bound in Theorem 3.7 and that in Theorem 3.2 is the relaxation of the complexity constraints (specifically, the replacement of condition (20)) with an additional term A2n log n/(nπ ℓ ). The potential advantage of Theorem 3.7 is that the additional term may yield a faster convergence rate than that in Theorem 3.2. We illustrate this next. Corollary 3.8. Consider the conditions of Theorems B.3 and 3.4, with the only modification that fˆ is independent of the data used in (4). Then, the ReLU neural network pretraining estimator ĝℓ satisfies 
∥g0,ℓ − ĝℓ ∥2L2 = OP ϕℓ,n max{Un2 , Bn2 , A2n } log4 Bn2 nπ ℓ  + ϕn log3 (n) log(An ) max{An , Un2 } .

4. Experiments 4.1. Numerical Experiments We conduct experiments in diverse simulation settings to evaluate our proposed transfer learning strategy (denoted as 2-Stage). For deep ReLU networks (NN), we compare 1 We note that the same function may admit different hierarchical compositions, and the values (p, K) we chosen here are for illustrative purposes. In fact, our upper bounds apply at the function-class level rather than to a specific function instance.

Remark 3.9. Corollary 3.8 shows that the ReLU pretraining estimator (ignoring logarithmic factors) attains the rate 6

Transfer Learning in Nonparametric Regression with Deep ReLU Networks Table 1. Convergence rates of four estimators under different regimes of π1 . CLS and NN stand for classical estimators and deep ReLU networks, respectively. Regime of π1

CLS-no-TL NN-no-TL

5 n− 18 11 n− 18

≪ π1 ≪ 1

5

≲ π1 ≲ n− 18 11

n−1 ≪ π1 ≪ n− 18

1

h1 (q) =

NN-TL

n− 6

n− 6

1

n− 6

1 (nπ1 )− 6 1 (nπ1 )− 6 1 (nπ1 )− 6

1 (nπ1 )− 6 1 (nπ1 )− 6 1 (nπ1 )− 6

1 n− 6

(nπ1 )− 13

1 n− 6 1 n− 6

1

(nπ1 )− 13

(nπ1 )− 7

3 3

10 X

qi2 ,

h2 (q) =

i=1

n− 6

1

π1 ≍ 1

CLS-TL

2. We define four intermediate quantities:

h3 (q; z) =

5 X i=1

3

10 X i=1

qi2 − qz2 ,

h4 (q; z) =

5 X i=1

|qi |, |qi | − |qz |.

The shared and group-specific components are given by: f0 (q) = log (1 + h1 (q) · h2 (q)), p f0,z (q) = 1 + |h3 (q; z) + h4 (q; z)|,

2-Stage with several training strategies, including Pooled, Separate, Pool-w-L, and Top-FT. Specifically, the Pooled strategy trains a single model fˆ on all data while ignoring group labels, following (3), and also serves as the first stage of our procedure. We further consider Separate, which trains independent models for each group; Pool-w-L, which pools the data and appends group labels as additional inputs to train a single model; and Top-FT, which fine-tunes only the output layer of the pooled NN for each group. The method of (Jiao et al., 2024), conceptually similar to TopFT, was also considered but omitted as it did not show advantages in most of our simulations. Besides NN, we compare with Random Forest (denoted as RF, (Breiman, 2001)), implementing the same strategies except for Top-FT. The ptLasso (Craig et al., 2026) is also considered in the high-dimensional setting as a parametric competitor.

and the response is generated according to the additive model y = f0 (x) + f0,z (x) + ϵ. Scenario 2. We consider a low-dimensional setting with a nonlinear mixture of shared and group-specific components, without an explicit additive structure. Based on the four intermediate quantities defined in Scenario 1, we further define the following compositions: p h5 (q; z) = h1 (q) h2 (q) + h3 (q; z) h4 (q; z), h6 (q; z) = (h1 (q) + h3 (q; z))2 /(h2 (q) + h4 (q; z))2 . The group-specific conditional mean is then given by fz (x) = h5 (q; z) · h6 (q; z), and the response is generated with y = fz (x) + ϵ.

We generate n ∈ {5000, 10000, 30000, 50000} independent samples for each scenario. Gaussian noise is used in all scenarios. In the low-dimensional settings, we control the noise level via a signal-to-noise ratio (SNR), defined as the ratio between the empirical variance of the noiseless signal computed within each generated dataset and Var(ϵ), with SNR ∈ {2, 5, 10}. Each sample is assigned to one of L = 5 groups, where group sizes are unbalanced and proportional to the group index z ∈ {1, . . . , 5}. In the high-dimensional scenarios, the noise variance is set to Var(ϵ) ∈ {0.12 , 1}. To mimic settings with a large number of different groups, each sample is assigned to one of L = 30 groups with equal group sizes. For each generated dataset, 15% of the samples within each group are randomly held out as a test set. All experiments are repeated over 50 Monte Carlo replications, and performance is evaluated using the mean squared error (MSE) on the test set. Due to space limitations, we present two low-dimensional scenarios with SNR = 5 and two high-dimensional scenario with Var(ϵ) = 1, and refer readers to Appendix F for other simulation scenarios and detailed experimental configurations.

Scenario 3. We consider a high-dimensional setting with shifted latent representations generated from a lowdimensional latent factor model. Let the latent variable be u = (u1 , . . . , u10 ) ∼ N (0, I10 ). The response is generP10 ated directly from the latent variable as y = j=1 u2j + ϵ. However, the latent variable u is unobserved; instead, for each group z ∈ {1, . . . , L}, the observed covariates q ∈ R100 are obtained via a group-specific nonlinear transformation of u. Specifically, each group is randomly assigned parameters wz ∈ {1/2, 1/3, . . . , 1/10}, νz ∈ {1/5, 1/4, 1/3, 1/2, 1, 2, 3, 4, 5}, and ψz ∈ {2π/k : k = 1, . . . , 6}. We define a shifted latent vector ũ(z) : ( uj , if j ∈ {1, . . . , 5}, (z) ũj = uj + wz · sin(ψz + νz · uj ), if j ∈ {6, . . . , 10}. The observed covariates are then generated from ũ(z) via a two-layer neural network mapping T : R10 → R100 , defined as q = T (ũ(z) ) = V tanh(W ũ(z) + b), where W ∈ R50×10 , b ∈ R50 , and V ∈ R100×50 are fixed across
i.i.d. all groups and randomly initialized as Wij ∼ N 0, 10−1 ,
i.i.d. i.i.d. bj ∼ N (0, 0.12 ), and Vij ∼ N 0, 50−1 . Under this construction, the estimation target fz (q) = E[y | q, z] is a highly nonlinear and implicit function of the observed q.

Scenario 1. We consider a low-dimensional setting with an additive structure. Let the input vector be q = i.i.d. (q1 , . . . , q10 ) ∼ Unif([0, 1]10 ), holding for Scenario 1 and 7

Transfer Learning in Nonparametric Regression with Deep ReLU Networks Table 3. Average MSE for Scenario 3 and 4 with Var(ϵ) = 1 over 50 independent trials. Random forest–based estimators are omitted due to high computational cost and poor performance. For each sample size, the lowest MSE is highlighted in bold.

Scenario 4. This scenario features a shared lowdimensional latent structure with sparse group-specific signals. Let u = (u1 , . . . , u10 ) ∼ N (0, I10 ) be the shared latent variable. The observed covariates q ∈ R500 consist of two parts: the first 410 coordinates arise from a shared nonlinear transformation qshare = V tanh(W u + b), where W ∈ R50×10 , b ∈ R50 , V ∈ R410×50 are i.i.d. i.i.d. randomly initialized as Wij ∼ N (0, 10−1 ), bj ∼

Model / Size

Pooled (NN) 2-Stage (NN) Separate (NN) Top-FT (NN) Pool-w-L (NN) ptLasso

i.i.d.

10000

30000

50000

Pooled (NN) 2-Stage (NN) Separate (NN) Top-FT (NN) Pool-w-L (NN) ptLasso

2.34 (0.7) 0.34 (0.1) 0.60 (0.1) 1.14 (0.2) 1.03 (1.0) 6.55 (0.2) 2.52 (0.1) 5.32 (0.2) 6.37 (0.2)

1.93 (0.3) 0.27 (0.1) 0.49 (0.1) 0.92 (0.2) 0.48 (0.2) 6.69 (0.1) 2.36 (0.1) 5.00 (0.1) 6.52 (0.1)

3.67 (1.1) 1.44 (0.1) 2.00 (0.1) 1.98 (0.2) 2.53 (1.2) 10.87 (0.3) 5.12 (0.2) 10.31 (0.3) 10.78 (0.3)

2.77 (0.2) 1.09 (0.1) 1.75 (0.2) 1.64 (0.2) 1.51 (0.2) 11.03 (0.3) 4.76 (0.2) 9.57 (0.2) 10.97 (0.3)

2.13 (0.2) 0.64 (0.1) 0.97 (0.4) 1.77 (0.2) 0.74 (0.2) 6.73 (0.3) 3.37 (0.2) 6.50 (0.3) 6.56 (0.3)

3.97 (0.5) 2.75 (0.3) 8.84 (1.4) 3.52 (0.3) 3.52 (0.7) 12.37 (1.0) 7.96 (0.8) 15.70 (1.3) 12.26 (1.0)

3.51 (0.3) 2.09 (0.2) 3.75 (0.8) 2.92 (0.3) 2.73 (0.5) 11.43 (0.6) 6.58 (0.5) 13.12 (0.7) 11.33 (0.6)

3.22 (0.4) 2.97 (0.3) 16.84 (1.1) 3.05 (0.3) 3.19 (0.3) 19.14 (1.2)

2.63 (0.4) 2.12 (0.2) 8.01 (0.6) 2.26 (0.3) 2.34 (0.3) 18.76 (0.8)

2.12 (0.3) 1.77 (0.2) 5.70 (0.4) 1.87 (0.2) 1.85 (0.2) 18.51 (0.7)

9.13 (0.9) 9.08 (0.9) 22.33 (1.5) 9.12 (0.9) 9.21 (0.9) 21.76 (1.7)

6.46 (0.6) 6.33 (0.6) 21.59 (0.9) 6.40 (0.6) 6.47 (0.6) 19.75 (1.0)

4.40 (0.5) 3.89 (0.3) 15.16 (0.6) 4.11 (0.3) 4.59 (0.7) 19.31 (0.8)

3.56 (0.3) 3.19 (0.2) 13.50 (0.7) 3.44 (0.3) 3.51 (0.3) 19.20 (0.8)

Experiments on the UTKFace Dataset. We conduct realdata experiments to image data for facial age estimation, where the goal is to estimate age conditioned on a facial image input. In this task, shared visual attributes such as skin texture and wrinkles provide universal cues for age estimation, while facial morphology varies across different ethnic groups, making it a natural setting for transfer learning.

Scenario 2 (MSE ×102 ) Pooled (NN) 2-Stage (NN) Separate (NN) Top-FT (NN) Pool-w-L (NN) Pooled (RF) 2-Stage (RF) Separate (RF) Pool-w-L (RF)

4.08 (0.4) 3.95 (0.4) 18.52 (1.4) 3.97 (0.4) 4.37 (0.5) 20.14 (1.4)

We conduct two real-data experiments: a low-dimensional study based on the Beijing PM2.5 dataset (Chen, 2015) and a high-dimensional study based on the UTKFace dataset (Zhang et al., 2017). In this section, we present partial results for UTKFace and refer readers to Appendix G for the complete results across all experiments.

Scenario 1 (MSE ×10 ) 2.28 (0.2) 0.93 (0.2) 6.64 (1.2) 2.12 (0.2) 1.03 (0.2) 7.08 (0.4) 4.10 (0.3) 7.75 (0.5) 6.92 (0.4)

50000

4.2. Real Data Experiments

2

Pooled (NN) 2-Stage (NN) Separate (NN) Top-FT (NN) Pool-w-L (NN) Pooled (RF) 2-Stage (RF) Separate (RF) Pool-w-L (RF)

30000

Scenario 4 (MSE)

Table 2. Average MSE for Scenario 1 and 2 with SNR = 5 over 50 independent trials. For each sample size, the lowest MSE is highlighted in bold. 5000

10000 Scenario 3 (MSE)

N (0, 0.12 ), Vij ∼ N (0, 50−1 ); the remaining 90 coordinates qtail ∼ N (0, I90 ) form group-specific sparse signals. Each group z is assigned three disjoint indices from qtail , each associated with a randomly selected function from {sin(νz ·), tanh(νz ·), (1 + exp(νz ·))−1 } with a randomly chosen parameter νz ∈ {1/5, 1/4, 1/3, 1/2, 1, 2, 3, 4, 5}. P10 2 P3 The response is y = j=1 uj + k=1 θz,k (qsz,k ) + ϵ, where θz,k denotes the function for group z at index sz,k .

Model / Size

5000

Specifically, experiments are conducted on the UTKFace dataset (Zhang et al., 2017), which contains 23,705 facial images annotated with ages ranging from 0 to 116, along with ethnicity labels across five groups: White (43%), Black (19%), Asian (14%), Indian (17%), and Other (7%). We treat ethnicity as the group label and perform neural network–based transfer learning across groups. Following (Baumann et al., 2021), we adopt a more practical approach in which rather than learning directly from raw pixels, we first extract 512-dimensional latent features for each image using a pretrained FaceNet model (Schroff et al., 2015), with weights obtained from an open-source implementation trained on the VGGFace2 dataset (Cao et al., 2018). These features then serve as input covariates for both our proposed framework and competing methods.

Based on the numerical simulations, the deep neural network estimator generally outperforms the other considered estimators, demonstrating the advantages of NN as a nonparametric estimator in handling complex functional composite structures across various dimensions. Notably, the NN trained with the proposed two-stage transfer learning strategy consistently outperforms both alternative training strategies and other estimators, highlighting the advantage of the proposed strategy. Moreover, in low-dimensional settings within RF–based estimators, the two-stage transfer learning strategy achieves the lowest MSE, demonstrating the generalizability of our framework to other estimators.

We evaluate the five learning strategies from the previous section, focusing exclusively on DNN estimators. For data partitioning, we randomly split the data while maintaining 8

Transfer Learning in Nonparametric Regression with Deep ReLU Networks Table 4. Overall and group-wise average MSE of age regression models over 20 independent trials. Random forest–based estimators are omitted due to poor performance. The best performance within each group is bolded. Model

Overall

White

Black

Asian

Indian

Pooled 2-Stage Separate Top-FT Pool-w-L

61.8 (1.7) 59.6 (1.9) 63.1 (2.5) 61.4 (1.7) 60.7 (2.5)

72.2 (2.7) 68.9 (3.2) 72.4 (3.8) 71.8 (2.5) 71.1 (3.9)

60.1 (3.8) 59.1 (4.1) 65.4 (6.0) 59.8 (3.7) 59.5 (3.4)

47.9 (7.8) 46.3 (8.4) 48.2 (8.6) 47.8 (7.9) 46.7 (8.4)

49.2 (2.9) 48.3 (2.7) 49.6 (3.6) 48.7 (3.0) 47.5 (2.7)

Several directions remain open for future work. First, our framework formally assumes an additive decomposition; while our simulations do not require this explicit form to hold, extending this to other coupling forms remains interesting. Second, negative transfer is theoretically possible in our setting when the offset Gℓ is highly complex or when the first-stage aggregation produces a complicated f¯. Such a case may arise when the groups are substantially different; in practice, however, groups in many real-world datasets exhibit strong cross-group similarity (e.g., our real-data experiments), and formal detection of negative transfer is left as future work. Third, our two-stage offset framework is conceptually related to the practically popular last-layer fine-tuning strategy, and developing analogous theory for fine-tuning is an open and important topic. Beyond these, aligning our theory with current practice motivates extensions to more advanced deep architectures, such as convolutional neural networks (LeCun et al., 2002; Krizhevsky et al., 2012) and Transformers (Vaswani et al., 2017). Finally, while this work assumes discrete group differences, a natural extension would be to model continuous variation across smoothly related tasks.

the ethnic group proportions across training (70%), validation (15%), and test (15%) sets. Each neural network instance is a four-layer MLP with 64 hidden units per layer and ReLU activations, trained with early stopping on the validation set. Additional configurations are provided in Section G.2. Table 4 reports the average MSE over 20 random trials on the test set, excluding samples labeled as “Other” ethnicity due to their ambiguous demographic attribution. From Table 4, the three hybrid strategies combining common and group-specific information consistently outperform the simple Pooled and Separated baselines, demonstrating the value of both leveraging shared patterns across groups and accommodating differences among groups in age estimation. Among all strategies, the proposed two-stage transfer learning achieves the lowest overall MSE, attaining optimal performance on White, Black, and Asian subgroups while yielding substantial improvements on the Indian group relative to Pooled. These results validate the effectiveness of our proposed method. For additional results, including analyses that incorporate gender as an additional grouping factor and experiments using raw image inputs trained directly with MLPs, we refer readers to Appendix G.3.

Software and Data Implementation of this work is available at https:// github.com/RenJump/Twostage-Trans-DNN.

Acknowledgements We greatly thank the anonymous reviewers for their helpful comments and suggestions, which have greatly helped us improve the quality of this work. In preparing the final revision of the paper, OHMP was partially supported by the NSF CAREER Award (No. DMS-2541747).

5. Conclusion

Impact Statement

This paper investigates transfer learning for nonparametric regression, using fully connected deep neural networks with ReLU activation as the estimator of interest. We assume that each group-specific conditional mean consists of a shared component common across all groups and a group-specific deviation. Our approach employs a two-stage offset learning framework: in the first stage, the overall conditional mean is estimated using pooled data from all groups to capture the shared structure; in the second stage, a group-specific offset is estimated to learn group-level characteristics, which are then combined to form the final estimator. We further discuss scenarios where transfer learning with deep neural networks can be beneficial, including cases where averaging similar groups yields a simpler overall function, where the offset function is simpler than the group-specific conditional mean, and data augmentation with all groups’ data improves the estimation of the overall conditional mean.

This paper studies transfer learning for nonparametric regression with data from multiple groups and proposes a general two-stage framework with theoretical guarantees. While the proposed methods may have potential applications in areas such as scientific data analysis and other domains involving grouped data, the primary contributions of this work are focuses on improving statistical efficiency and convergence properties. As such, the societal impact of this work is expected to be indirect, mainly by advancing the theoretical understanding of transfer learning and nonparametric regression with deep neural networks, and by informing future research in related areas.

9

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

References

Caruana, R. Multitask learning. Machine learning, 28(1): 41–75, 1997.

Apostolopoulos, I. D. and Mpesiana, T. A. Covid-19: automatic detection from x-ray images utilizing transfer learning with convolutional neural networks. Physical and engineering sciences in medicine, 43(2):635–640, 2020.

Chen, A., Owen, A. B., and Shi, M. Data enriched linear regression. 2015. Chen, M., Jiang, H., Liao, W., and Zhao, T. Nonparametric regression on low-dimensional manifolds using deep relu networks: Function approximation and statistical recovery. Information and Inference: A Journal of the IMA, 11 (4):1203–1253, 2022.

Argyriou, A., Evgeniou, T., and Pontil, M. Convex multitask feature learning. Machine learning, 73(3):243–272, 2008. Bartlett, P. L., Harvey, N., Liaw, C., and Mehrabian, A. Nearly-tight vc-dimension and pseudodimension bounds for piecewise linear neural networks. Journal of Machine Learning Research, 20(63):1–17, 2019.

Chen, S. Beijing PM2.5. UCI Machine Learning Repository, 2015. DOI: https://doi.org/10.24432/C5JS49.

Bastani, H. Predicting with proxies: Transfer learning in high dimension. Management Science, 67(5):2964–2984, 2021.

Craig, E., Pilanci, M., Le Menestrel, T., Narasimhan, B., Rivas, M. A., Gullaksen, S.-E., Dehghannasiri, R., Salzman, J., Taylor, J., and Tibshirani, R. Pretraining and the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 88(1):261–281, 2026.

Baumann, P. F., Hothorn, T., and Rügamer, D. Deep conditional transformation models. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 3–18. Springer, 2021.

Danhofer, D. A., D’Ascenzo, D., Dubach, R., and Poggio, T. A. Position: A theory of deep learning must include compositional sparsity. In International Conference on Machine Learning, pp. 81199–81210. PMLR, 2025.

Baxter, J. A model of inductive bias learning. Journal of artificial intelligence research, 12:149–198, 2000.

Daumé III, H. Frustratingly easy domain adaptation. In Proceedings of the 45th annual meeting of the association of computational linguistics, pp. 256–263, 2007.

Bradski, G. The opencv library. Dr. Dobb’s Journal: Software Tools for the Professional Programmer, 25(11):120– 123, 2000.

Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. Ieee, 2009.

Breiman, L. Random forests. Machine learning, 45(1): 5–32, 2001.

Dinh, C. T., Vu, T. T., Tran, N. H., Dao, M. N., and Zhang, H. A new look and convergence rate of federated multitask learning with laplacian regularization. IEEE Transactions on Neural Networks and Learning Systems, 35(6):8075– 8085, 2022.

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901, 2020.

Donoho, D. L. et al. High-dimensional data analysis: The curses and blessings of dimensionality. AMS math challenges lecture, 1(2000):32, 2000.

Cagnetta, F., Petrini, L., Tomasini, U. M., Favero, A., and Wyart, M. How deep neural networks learn compositional data: The random hierarchy model. Physical Review X, 14(3):031001, 2024.

Du, S. S., Koushik, J., Singh, A., and Póczos, B. Hypothesis transfer learning via transformation functions. Advances in neural information processing systems, 30, 2017.

Cai, T. T. and Pu, H. Transfer learning for nonparametric regression: Non-asymptotic minimax analysis and adaptive procedure. arXiv preprint arXiv:2401.12272, 2024.

Du, S. S., Hu, W., Kakade, S. M., Lee, J. D., and Lei, Q. Few-shot learning via learning the representation, provably. arXiv preprint arXiv:2002.09434, 2020.

Cai, T. T. and Wei, H. Transfer learning for nonparametric classification. The Annals of Statistics, 49(1):100–128, 2021.

Duan, Y. and Wang, K. Adaptive and robust multi-task learning. The Annals of Statistics, 51(5):2015–2039, 2023.

Cao, Q., Shen, L., Xie, W., Parkhi, O. M., and Zisserman, A. Vggface2: A dataset for recognising faces across pose and age. In 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018), pp. 67–74. IEEE, 2018.

Evgeniou, T. and Pontil, M. Regularized multi–task learning. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 109–117, 2004. 10

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

Gong, B., Shi, Y., Sha, F., and Grauman, K. Geodesic flow kernel for unsupervised domain adaptation. In 2012 IEEE conference on computer vision and pattern recognition, pp. 2066–2073. IEEE, 2012.

Li, S., Zhang, L., Cai, T. T., and Li, H. Estimation and inference for high-dimensional generalized linear models with knowledge transfer. Journal of the American Statistical Association, 119(546):1274–1285, 2024.

Gross, S. M. and Tibshirani, R. Data shared lasso: A novel tool to discover uplift. Computational statistics & data analysis, 101:226–235, 2016.

Liang, X., Zou, T., Guo, B., Li, S., Zhang, H., Zhang, S., Huang, H., and Chen, S. X. Assessing beijing’s pm2. 5 pollution: severity, weather impact, apec and winter heating. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 471(2182):20150257, 2015.

Györfi, L., Kohler, M., Krzyżak, A., and Walk, H. A distribution-free theory of nonparametric regression. Springer, 2002.

Lin, H. and Reimherr, M. On transfer learning in functional linear regression. CoRR, 2022.

He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.

Lin, H. and Reimherr, M. Smoothness adaptive hypothesis transfer learning. In International Conference on Machine Learning, pp. 30286–30316. PMLR, 2024.

Howard, J. and Ruder, S. Universal language model finetuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 328–339, 2018.

Loshchilov, I. and Hutter, F. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016.

Jiao, Y., Shen, G., Lin, Y., and Huang, J. Deep nonparametric regression on approximate manifolds: Nonasymptotic error bounds with polynomial prefactors. The Annals of Statistics, 51(2):691–716, 2023.

Lu, C., Hu, F., Cao, D., Gong, J., Xing, Y., and Li, Z. Transfer learning for driver model adaptation in lane-changing scenarios using manifold alignment. IEEE transactions on intelligent transportation systems, 21(8):3281–3293, 2019.

Jiao, Y., Lin, H., Luo, Y., and Yang, J. Z. Deep transfer learning: Model framework and error analysis. arXiv preprint arXiv:2410.09383, 2024.

Madrid Padilla, O. H., Sharpnack, J., Chen, Y., and Witten, D. M. Adaptive nonparametric regression with the knearest neighbour fused lasso. Biometrika, 107(2):293– 310, 2020.

Kohler, M. and Langer, S. On the rate of convergence of fully connected deep neural network regression estimates. The Annals of Statistics, 49(4):2231–2249, 2021.

Mammen, E. and Van De Geer, S. Locally adaptive regression splines. The Annals of Statistics, 25(1):387–413, 1997.

Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.

Maurer, A., Pontil, M., and Romera-Paredes, B. The benefit of multitask representation learning. Journal of Machine Learning Research, 17(81):1–32, 2016.

Lai, D., Padilla, O. H. M., and Gu, T. Bayesian transfer learning for enhanced estimation and inference. arXiv preprint arXiv:2412.02986, 2024.

Padilla, C. M. M., Padilla, O. H. M., Kei, Y. L., Zhang, Z., and Chen, Y. Confidence interval construction and conditional variance estimation with dense relu networks. arXiv preprint arXiv:2412.20355, 2024a.

LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradientbased learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 2002.

Padilla, C. M. M., Zhang, Z., Luo, X., Wang, D., and Padilla, O. H. M. Dense relu neural networks for temporal-spatial model. arXiv preprint arXiv:2411.09961, 2024b.

Li, S., Cai, T. T., and Li, H. Transfer learning for highdimensional linear regression: Prediction, estimation and minimax optimality. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(1):149–173, 2022.

Padilla, O. H. M., Tansey, W., and Chen, Y. Quantile regression with relu networks: Estimators and minimax rates. Journal of Machine Learning Research, 23(247):1–42, 2022.

Li, S., Cai, T. T., and Li, H. Transfer learning in largescale gaussian graphical models with false discovery rate control. Journal of the American Statistical Association, 118(543):2171–2183, 2023.

Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al. Improving language understanding by generative pre-training. 2018. 11

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

Rosenbaum, P. R. and Rubin, D. B. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41–55, 1983.

Wang, X. and Schneider, J. G. Generalization bounds for transfer learning under model shift. In UAI, pp. 922–931, 2015.

Sadhanala, V. and Tibshirani, R. J. Additive models with trend filtering. The Annals of Statistics, 47(6):3032–3068, 2019.

Wang, X., Oliva, J. B., Schneider, J. G., and Póczos, B. Nonparametric risk and stability analysis for multi-task learning problems. In IJCAI, pp. 2146–2152, 2016.

Schmidt-Hieber, J. Nonparametric regression using deep neural networks with relu activation function. 2020.

Wasserman, L. All of nonparametric statistics. Springer, 2006.

Schroff, F., Kalenichenko, D., and Philbin, J. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 815–823, 2015.

Xu, Z. and Tewari, A. Representation learning beyond linear prediction functions. Advances in Neural Information Processing Systems, 34:4792–4804, 2021. Zhang, Z., Song, Y., and Qi, H. Age progression/regression by conditional adversarial autoencoder. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5810–5818, 2017.

Schweikert, G., Rätsch, G., Widmer, C., and Schölkopf, B. An empirical analysis of domain adaptation algorithms for genomic sequence analysis. Advances in neural information processing systems, 21, 2008. Tian, Y. and Feng, Y. Transfer learning under highdimensional generalized linear models. Journal of the American Statistical Association, 118(544):2684–2697, 2023. Tian, Y., Gu, Y., and Feng, Y. Learning from similar linear representations: Adaptivity, minimaxity, and robustness. arXiv preprint arXiv:2303.17765, 2023. Tibshirani, R. J. Adaptive piecewise polynomial estimation via trend filtering. 2014. Tripuraneni, N., Jordan, M., and Jin, C. On the theory of transfer learning: The importance of task diversity. Advances in neural information processing systems, 33: 7852–7862, 2020. Tripuraneni, N., Jin, C., and Jordan, M. Provable metalearning of linear representations. In International conference on machine learning, pp. 10434–10443. PMLR, 2021. Tzeng, E., Hoffman, J., Saenko, K., and Darrell, T. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7167–7176, 2017. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017. Vershynin, R. High-Dimensional Probability: An Introduction with Applications in Data Science. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge, 2 edition, 2026. 12

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

A. Formal Definitions of Hierarchical Composition Models Definition A.1 ((p, C)-Smoothness). Let p = q + s for some q ∈ N = Z+ ∪ {0} and 0 < s ≤ 1. We say that a function Pd g : Rd → R is called (p, C)-smooth, if for every α = (α1 , . . . , αd ) ∈ Nd , with d ∈ Z+ , where j=1 αj = q the partial αd 1 derivative ∂ q g/ (∂aα 1 . . . ∂ad ) exists and satisfies ∂q g

∂q g

αd (a) − αd (b) 1 1 ∂aα ∂aα 1 . . . ∂ad 1 . . . ∂ad

≤ C∥a − b∥s

for all a, b ∈ Rd . Building on the definition of (p, C)-smoothness, we now introduce the class of hierarchical composition models. Definition A.2 (Space of Hierarchical Composition Models, (Kohler & Langer, 2021)). For l = 1 and smoothness constraint P ⊆ (0, ∞) × N the space of hierarchical composition models is given as 
H(1, P) := h : Rd → R : h(a) = m a(π(1)) , . . . , a(π(K)) , where m : RK → R is (p, C)-smooth for some (p, K) ∈ P and π : {1, . . . , K} → {1, . . . , d}}. For l > 1, we recursively construct  H(l, P) := h : Rd → R : h(a) = m (f1 (a), . . . , fK (a)) , where m : RK → R is (p, C)-smooth for some (p, K) ∈ P and fi ∈ H(l − 1, P)} . Figure 1 gives an illustration of a hierarchical composition model from the class H(2, P), where a complex multivariate function is constructed via a two-level composition of low-dimensional smooth functions acting on subsets of the 10dimensional input variables.

(2)

g1 (u1 , u2 , u3 ) = u21 + u22 + u23

(1)

(1)

g1 (u1 , u2 ) = u1 u2

xπ(1)

x1

xπ(2)

x2

(1)

g2 (u1 , u2 , u3 ) = log(1 + u1 + u2 + u3 )

xπ(3)

x3

x4

xπ(4)

x5

g3 (u1 , u2 ) = sin(u1 u2 )

xπ(5)

x6

x7

xπ(6)

x8

x9

xπ(7)

x10

Figure 1. Illustration of a hierarchical composition model of the class H(2, P).

B. Formal Statements of Theorems Theorem B.1. [General Error Bound for First-Stage Estimation (Theorem 1 from (Padilla et al., 2024a)]). Let Un > 0 and suppose that f˜ ∈ F is such that p ∥f¯ − f˜∥∞ ≤ ϕn , so that ϕn is the approximating error. Suppose that An is chosen to satisfy p An ≥ 8 max{∥f¯∥∞ + Un , 8∥f¯∥∞ , 8 ϕn }. 13

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

Moreover, let FAn := {fAn /(2An ) : f ∈ F} and assume that sup x1 ,...,xn ∈[0,1]d

log N (δ, FAn , ∥·∥n ) ≤ ηn (δ)

for some decreasing function ηn : (0, 1) → R≥0 . If "∞ ∞ # ∞   XX X
′ lim exp −C1 ηn (2−k−k −1 ) + exp −C2 ηn (2−k−1 ) + P(∥ϵ̄∥∞ > Un ) = 0, n→∞

k=0 k′ =1

for some constants C1 , C2 > 0 and

′ ∞ X ηn (2−k−k )

sup k∈N

then

(14)

k=0

k′ =1

22k′ ηn (2−k )

≤ 1,

(15)

  Un2 ηn (δn ) 2 2 2 ¯ ˆ ¯ ˆ max{∥f − fAn ∥n , ∥f − fAn ∥L2 } = OP ϕn + + An δn , n

(16)

where δn is a critical radius of FAn , see Definition C.1. Theorem B.2. [Detailed Statement of Theorem 3.2]. With the notation and conditions from Theorem B.1, let rn be the rate of convergence of fˆAn towards f¯, namely rn := ϕn +

Un2 ηn (δn ) + An δn2 , n

e ℓ ∈ Fℓ such that and where fˆ has been constructed as in (3). Suppose for ℓ ∈ {1, . . . , L} there exists G p e ℓ ∥∞ ≤ ϕℓ,n , ∥Gℓ − G where ϕℓ,n denotes the approximation error. Let ηℓ,n : R+ → R+ , be functions such that max

sup

nπ ℓ 2 ≤ nℓ ≤ 2nπ ℓ

{xi }i∈Iℓ ⊂[0,1]d

log N (δ, Fℓ,Bn , ∥ · ∥Iℓ ) ≤ ηℓ,n (δ) ∀ δ ∈ (0, 1),

(17)

where nℓ = |Iℓ |, where Fℓ,Bn := {fBn /(2Bn ) : f ∈ Fℓ }, and Bn is chosen such that Bn ≥ max{Un + ∥f¯∥∞ + An + 2 max ∥f0,ℓ ∥∞ , ∥Gℓ ∥∞ + ℓ=1,...,L

p ϕℓ,n }.

(18)

Suppose that δ ∈ (0, 1) satisfies (

Z δ

(

Z δ

sup

max

m:

1 √ m

nπ ℓ m d 2 ≤m≤2nπ ℓ {xj }j=1 ⊂[0,1]

q

δ 2 /48

) δ p log N (t/4, Fℓ,Bn , ∥·∥m )dt + √ c1 log(48/δ 2 ) ≲ δ 2 , m

(19)

) δ p log N (t/4, FAn , ∥·∥m )dt + √ c1 log(48/δ 2 ) ≲ δ 2 , m

(20)

and max

m:

sup

nπ ℓ m d 2 ≤m≤2nπ ℓ {xj }j=1 ⊂[0,1]

then

1 √ m

δ 2 /48

q

  U 2 ηℓ,n (δn ) ∥g0,ℓ − ĝℓ ∥2L2 = OP ϕℓ,n + rn + n + Bn2 δn2 nπ ℓ

provided that nπ 3ℓ /π 2ℓ → ∞, δn2 nπ ℓ → ∞, and "∞ ∞ # ∞   XX X
−k−k′ −1 −k−1 lim exp −C1 ηℓ,n (2 ) + exp −C2 ηℓ,n (2 ) + P(∥ϵ∥∞ > Un ) = 0 n→∞

k=0 k′ =1

(21)

(22)

k=0

for some constants C1 , C2 > 0, and sup k∈N

′ ∞ X ηℓ,n (2−k−k )

k′ =1

22k′ ηℓ,n (2−k ) 14

≤ 1.

(23)

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

Theorem B.3. [Upper Bound for First-Stage Estimation with Deep ReLU Network (Theorem 2 in (Padilla et al., 2024a))]. Suppose that f¯ ∈ H(l0 , P0 ) for some l0 ∈ N and P0 ⊂ [1, ∞) × N. Furthermore, assume that each function m in the definition of f¯ can have different smoothness pm = qm + sm , for qm ∈ N, sm ∈ (0, 1], and of potentially different input dimension Km , so that (pm , Km ) ∈ P0 . Let Kmax be the largest input dimension and pmax the largest smoothness of any of the functions m. Suppose that all the partial derivatives of order less than or equal to qm are uniformly bounded by constant c2 , and each function m is Lipschitz continuous with Lipschitz constant CLip ≥ 1. Also, assume that max{pmax , Kmax } = O(1). Let −2p ϕn = max n (2p+K) . (24) (p,K)∈P0

Then there exists sufficiently large positive constants c3 and c4 such that if   K M = ⌈c3 log n⌉ and ν = c4 max n 2(2p+K) (p,K)∈P0

or

 M = c3

(25)



K

max n 2(2p+K) log n

(p,K)∈P0

ν = ⌈c4 ⌉ ,

and

(26)

then fˆAn , with fˆ as defined in (3) with F := F (M, ν), satisfies, ∥f¯ − fˆAn ∥2L2 = OP (rn ) where rn :=

(27)

max{An , Un2 } log n + ϕn log3 (n) log(An ) max{An , Un2 }, n

(28)

provided that lim P(∥ϵ∥∞ > Un ) = 0

n→∞

and

p An ≥ 8 max{∥f¯∥∞ + Un , 8∥f¯∥∞ , 8 ϕn }

hold. Theorem B.4. [Detailed Statement of Theorem 3.4.] Suppose that the conditions of Theorem B.3 hold. Also, for ℓ ∈ {1, . . . , L}, let L X Gℓ (x) := f0,ℓ (x) − f0,k (x) P(Z = k|X = x). k=1

Suppose that Gℓ ∈ H(Rℓ , Pℓ ) for some Rℓ ∈ N and Pℓ ⊂ [1, ∞) × N. Furthermore, assume that each function m in the definition of Gℓ can have different smoothness pm = qm + sm , for qm ∈ N, sm ∈ (0, 1], and of potentially different input dimension Km , so that (pm , Km ) ∈ Pℓ . Let Kℓ,max be the largest input dimension and pℓ,max the largest smoothness of any of the functions m. Suppose that all the partial derivatives of order less than or equal to qm are uniformly bounded by constant c2 , and each function m is Lipschitz continuous with Lipschitz constant CLip ≥ 1. Also, assume that max{pℓ,max , Kℓ,max } = O(1). Let ϕℓ,n =

−2p

max (nπ ℓ ) (2p+K) .

(29)

(p,K)∈Pℓ

Then there exists sufficiently large positive constants c3 and c4 such that if   K Mℓ = ⌈c3 log(nπ ℓ )⌉ and νℓ = c4 max (nπ ℓ ) 2(2p+K) (p,K)∈Pℓ

or

 Mℓ = c3

max (nπ ℓ )

K 2(2p+K)

(p,K)∈Pℓ

 log(nπ ℓ )

and

νℓ = ⌈c4 ⌉ ,

then ĝℓ as defined in (5) with Fℓ := F(Mℓ , νℓ ), satisfies,   ϕn max{An , Un2 } log3 (n) log(A2n nπ ℓ ) 4 2 2 2 2 + ϕℓ,n max{Un , Bn } log (Bn nπ ℓ ) ∥g0,ℓ − ĝℓ ∥L2 = OP πℓ provided that nπ 3ℓ /π 2ℓ → ∞, Bn is chosen to satisfy (18). 15

(30)

(31)

(32)

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

C. Proofs C.1. Preliminaries Definition C.1. Define B∞ (1) := {f : [0, 1]d → R : ∥f ∥∞ ≤ 1}. Given a function class F with F ⊂ B∞ (1), we call δn > 0 a critical radius for F if ! n 1X E sup ξi f (xi ) {xi }ni=1 ≤ δn2 , f ∈star(F ) : ∥f ∥n ≤δn n i=1 where ξ1 , . . . , ξn are independent Rademacher variables independent of {xi }ni=1 and where star(F) is defined as star(F) := {λf : λ ∈ [0, 1], f ∈ F}. C.2. Auxiliary Lemmas Lemma C.2. Let ỹi = yi − fˆAn (xi ) for i = 1, . . . , n. Then ỹi = f0,zi (xi ) −

L X ℓ=1

f0,ℓ (xi ) P(Z = ℓ|X = xi ) + (f¯(xi ) − fˆAn (xi )) + ϵi ,

for i = 1, . . . , n. Proof. Notice that ỹi

= yi − fˆAn (xi ) = yi + (f¯(xi ) − fˆAn (xi )) − f¯(xi ) = f0 (xi ) + f0,zi (xi ) + ϵi + (f¯(xi ) − fˆAn (xi )) − f¯(xi ) = =

f0 (xi ) + f0,zi (xi ) + ϵi + (f¯(xi ) − fˆAn (xi )) − [f0 (xi ) + f0,zi (xi ) −

L X ℓ=1

L X

f0,ℓ (xi ) P(Z = ℓ|X = xi ).]

ℓ=1

f0,ℓ (xi ) P(Z = ℓ|X = xi ) + (f¯(xi ) − fˆAn (xi )) + ϵi .

Lemma C.3. Suppose that the event Ω1 = {∥ϵ∥∞ ≤ Un } holds and Bn satisfies Bn ≥ Un + ∥f¯∥∞ + An + 2 max ∥f0,ℓ ∥∞ . ℓ=1,...,L

(33)

Then ∥ỹ∥∞ ≤ Bn . Proof. Notice that by Lemma C.2, |ỹi |

≤

∥f0,zi ∥∞ +

max ∥f0,ℓ ∥∞ + ∥f¯∥∞ + An + Un

ℓ=1,...,L

and the claim follows.

Lemma C.4. Let Gℓ (x) := f0,zi (x) −

L X

f0,k (x) P(Z = k|X = x)

k=1

p e ℓ ∈ Fℓ be such that ∥Gℓ − G e ℓ ∥∞ ≤ ϕℓ,n . Such G e ℓ exists by Theorem 3 in (Kohler & Langer, 2021). Then if and G p eℓ = G e ℓ,B := (G e ℓ )B . Bn ≥ ∥Gℓ ∥∞ + ϕℓ,n we have that G n n 16

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

Proof. Simply observe that e ℓ ∥∞ ≤ ∥G e ℓ − Gℓ ∥∞ + ∥Gℓ ∥∞ ≤ ∥G

p

ϕℓ,n + ∥Gℓ ∥∞ ≤ Bn

and the claim follows. Lemma C.5. Suppose that Assumption 3.1 holds. Then letting nℓ = |{i ∈ [n] : zi = ℓ}|, for ℓ ∈ {1, . . . , L} the event n nπ o ℓ Ω2 (ℓ) := ≤ nℓ ≤ 2nπ ℓ 2 happens with probability at least nπ 3ℓ 1 − exp − 27π 2ℓ 

 .

Proof. First, notice that nℓ is Binomial(n, pℓ ) where Z pℓ = P(zi = ℓ|xi = x)dFX (dx) ∈ [π ℓ , π ℓ ], by Assumption 3.1. Hence, by the binomial concentration inequality, for instance see Lemma 5 in (Madrid Padilla et al., 2020), letting κℓ = π ℓ /(3π ℓ ) ∈ [0, 1], P(|npℓ − nℓ | ≥ κℓ nπ ℓ ) ≤ P(|np ℓ − 2nℓ | ≥κℓ npℓ ) π npℓ ≤ exp − ℓ 2  27π3 ℓ nπ ℓ . ≤ exp − 27π 2ℓ However, |npℓ − nℓ | < κℓ nπ ℓ holds if and only if npℓ − κℓ nπ ℓ < nℓ < npℓ + κℓ nπ ℓ which implies

nπ ℓ < nℓ < 2nπ ℓ 2

given our choice of κℓ . The claim then follows. Lemma C.6. Let F be a function class such that |f | ≤ M for all f ∈ F . For a set S = {a1 , . . . , aW } ⊂ [0, 1]d , we write v uW uX ∥f ∥S := t f (aw )2 . w=1

We also write ∥f ∥ℓ,L2 :=

sZ

f (x)2 dPX|Z=ℓ (x)

with PX|Z=ℓ the distribution of X conditioning on Z = ℓ. Then for any δ > 0 we have that (Z ) r r δ log N (t/4, F, ∥·∥S ) log(48/δ 2 ) sup dt + δ ≲ δ2 , nπ ℓ W W 2 W d δ /48 S={aw } ⊂[0,1] : ≤ W ≤ 2nπ ℓ w=1

2

implies that the event ( Eℓ (F) :=

sup g∈F

|∥g∥2ℓ,n − ∥g∥2ℓ,L2 | 1 δ2 2 2 ∥g∥ℓ,L2 + 2

) ≤1

happens with probability at least    nπ 3ℓ [1 − c1 exp(−c3 nπ ℓ δ 2 )] · 1 − exp − . 27π 2ℓ 17

(34)

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

Proof. We notice that P(Eℓ (F))

≥

X W:

nπ ℓ 2 ≤ W ≤ 2nπ ℓ

P(Eℓ (F) | nℓ = W ) · P(nℓ = W ).

(35)

Let {ξi }ni=1 be Radamacher variables, and observe that 1 X ξi f (xi ) nℓ = W E sup f ∈star(F ) : ∥f ∥ℓ,n ≤δ nℓ

!

i : zi =ℓ

1 X =E E sup ξi f (xi ) {xi , zi }ni=1 , nℓ = W f ∈star(F ) : ∥f ∥ℓ,n ≤δ nℓ

!! .

i : zi =ℓ

Hence, proceeding as in the proof Lemma 6 in (Padilla et al., 2024a), based on our choice of δ, we obtain that P(Eℓ (F) | nℓ = W ) ≥ 1 − c1 exp(−c2 W δ 2 ) for some constants c1 , c2 > 0. Therefore, from (35), for some constant c3 > 0, we have that P(Eℓ (F))

nπ [1 − c1 exp(−c3 nπ ℓ δ 2 )] · P( ℓ ≤ W ≤ 2nπ ℓ )  2   nπ 3ℓ ≥ [1 − c1 exp(−c3 nπ ℓ δ 2 )] · 1 − exp − 27π 2ℓ

≥

where the last inequality follows from Lemma C.5. The claim then follows.

Lemma C.7. Suppose that Assumption 3.1 holds. There exists constants c1 , c2 > 0 such that c1 ∥f ∥2ℓ,L2 ≤ ∥f ∥2L2 ≤ c2 ∥f ∥2ℓ,L2 for any measurable function f with ∥f ∥2L2 < ∞. Proof. By Assumption 3.1, there exist 0 < π ℓ ≤ π ℓ < 1 such that π ℓ ≤ P(Z = ℓ | X = x) ≤ π ℓ for all x. Hence π ℓ ≤ P(Z = ℓ) =

Z

P(Z = ℓ | X = x) dPX (x) ≤ π ℓ .

Moreover, consider the Radon–Nikodym derivative dPX|Z=ℓ P(Z = ℓ | X = x) (x) = . dPX P(Z = ℓ) Therefore,

dPX|Z=ℓ πℓ πℓ ≤ (x) ≤ . πℓ dPX πℓ

Thus, for any nonnegative measurable function g, Z Z Z πℓ πℓ g(x) dPX (x) ≤ g(x) dPX|Z=ℓ (x) ≤ g(x) dPX (x). πℓ πℓ Equivalently, πℓ πℓ

Z

g(x) dPX|Z=ℓ (x) ≤

Z

g(x) dPX (x) ≤ 18

πℓ πℓ

Z g(x) dPX|Z=ℓ (x).

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

Taking g(x) = f 2 (x), we obtain πℓ πℓ ∥f ∥2ℓ,L2 . ∥f ∥2ℓ,L2 ≤ ∥f ∥2L2 ≤ πℓ πℓ Hence the result holds with c1 =

πℓ , πℓ

c2 =

πℓ . πℓ

C.3. Proof of Theorem B.2 Proof. For a fixed ℓ ∈ {1, . . . , L} let Iℓ := {i ∈ [n] : zi = ℓ} and nℓ := |Iℓ |, and we condition on z being in the set {z ∈ {1, . . . , L}n :

nπ ℓ ≤ |{i : zi = ℓ}| ≤ 2nπ ℓ }. 2

(36)

By Lemma C.5 this happens with probability at least   nπ 3ℓ 1 − exp − . 27π 2ℓ Let δ > 0 to be defined later. Suppose that the event Ω1 defined in Lemma C.3 holds. Then, given our choice of Bn , from eℓ = G e ℓ,B and ∥ỹ∥∞ ≤ Bn . Hence, Lemmas C.3 and C.4 we have G n 1 X 1 X (ỹi − fˆℓ,Bn (xi ))2 ≤ (ỹi − fˆℓ (xi ))2 nℓ nℓ i∈Iℓ

i∈Iℓ

where fˆl,Bn = (fˆl )Bn . Also, by the basic inequality, X 1 X 1 X e ℓ (xi ))2 = 1 e ℓ,B (xi ))2 . (ỹi − fˆℓ (xi ))2 ≤ (ỹi − G (ỹi − G n nℓ nℓ nℓ i∈Iℓ

i∈Iℓ

i∈Iℓ

As a result, 1 X 1 X e ℓ,B (xi ))2 . (ỹi − fˆℓ,Bn (xi ))2 ≤ (ỹi − G n nℓ nℓ i∈Iℓ

∈Iℓ

However, by Lemma C.2 and the notation of Lemma C.4, ỹi = Gℓ (xi ) + (f¯(xi ) − fˆAn (xi )) + ϵi for i with zi = ℓ, then 1 X 1 X e ℓ,B + (f¯(xi ) − fˆA (xi )) + ϵi ]2 [Gℓ (xi ) − fˆℓ,Bn (xi ) + (f¯(xi ) − fˆAn (xi )) + ϵi ]2 ≤ [Gℓ (xi ) − G n n nℓ nℓ i∈Iℓ

i∈Iℓ

which implies 1 X (Gℓ (xi ) − fˆℓ,Bn (xi ))2 nℓ i∈Iℓ

≤

≤

X 1 X e ℓ,B (xi ))2 + 2 e ℓ,B (xi )) + (Gℓ (xi ) − G ϵi (fˆℓ,Bn (xi ) − G n n nℓ nℓ i∈Iℓ i∈Iℓ 2 X ˆ e ℓ,B (xi ))(f¯(xi ) − fˆA (xi )) (fℓ,Bn (xi ) − G n n nℓ i∈Iℓ X e ℓ − Gℓ ∥2∞ + 2 e ℓ,B (xi )) + ∥G ϵi (fˆℓ,Bn (xi ) − G n nℓ i∈Iℓ 2 X ˆ e ℓ,B (xi ))(f¯(xi ) − fˆA (xi )) (fℓ,Bn (xi ) − G n n nℓ i∈Iℓ

19

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

2 X e ℓ,B (xi )) + ϕℓ,n + ϵi (fˆℓ,Bn (xi ) − G n nℓ i∈Iℓ " # X 2 1 X ˆ 2 2 e ℓ,B (xi )) + 8 (fℓ,Bn (xi ) − G (f¯(xi ) − fˆAn (xi )) n nℓ 32 i∈Iℓ i∈Iℓ X 2 X e ℓ,B (xi )) + 1 ϵi (fˆℓ,Bn (xi ) − G (Gℓ (xi ) − fˆℓ,Bn (xi ))2 ≤ ϕℓ,n + n nℓ 8nℓ i∈Iℓ i∈Iℓ 16 X ¯ 1 X 2 ˆ e (Gℓ (xi ) − Gℓ (xi )) + (f (xi ) − fAn (xi ))2 8nℓ nℓ i∈Iℓ i∈Iℓ 9ϕℓ,n 2 X 1 X ˆ e ≤ + ϵi (fℓ,Bn (xi ) − Gℓ,Bn (xi )) + (Gℓ (xi ) − fˆℓ,Bn (xi ))2 8 nℓ 8nℓ i∈Iℓ i∈Iℓ 16 X ¯ 2 ˆ (f (xi ) − fAn (xi )) . nℓ

≤

i∈Iℓ

Therefore, 7 X (Gℓ (xi ) − fˆℓ,Bn (xi ))2 8nℓ i∈Iℓ

≤

X 2 X 9ϕℓ,n e ℓ,B (xi )) + 16 + ϵi (fˆℓ,Bn (xi ) − G (f¯(xi ) − fˆAn (xi ))2 . n 8 nℓ nℓ i∈Iℓ i∈Iℓ (37)

Next let Hℓ := {(f − g)/2 : f, g ∈ Fℓ,Bn }. Then f ∈ Hℓ implies ∥f ∥∞ ≤ 1 and log N (δ, Hℓ , ∥ · ∥Iℓ ) ≤ 2 log N (δ/2, Fℓ,Bn , ∥ · ∥Iℓ ) ≤ 2ηℓ,n (δ/2),

(38)

where the second inequality follows since we are conditioning on z satisfiying (36) and due to our assumption that (17) holds. Hence, by Lemma 4 from (Padilla et al., 2024a), for some positive constant C > 0, e ℓ,B (xi )) 2Bn X (fˆℓ,Bn (xi ) − G n ϵi nℓ 2Bn i∈Iℓ

e ℓ,B )/(2Bn )∥I · 2CBn Un ∥(fˆℓ,Bn − G n ℓ q e ℓ,B )/(4Bn )∥I )/nℓ ηℓ,n (∥(fˆℓ,Bn − G n ℓ q e ℓ,B ∥I · ηℓ,n (∥(fˆℓ,B − G e ℓ,B )/(4Bn )∥I )/nℓ ≤ CUn ∥fˆℓ,Bn − G n n n ℓ ℓ

≤

with probability at least 1−4

∞ X ∞ X k=0 k′ =1

∞   X
′ exp −C1 ηℓ,n (2−k−k −1 ) − 4 exp −C2 ηℓ,n (2−k−1 ) − P(∥ϵ∥∞ > Un ). k=0

Define Ω3 , the event such that (39) holds and let δ > 0. From now on suppose that Ω1 ∩ Ω3 holds. If e ℓ,B )/(4Bn )∥I ≤ δ ∥(fˆℓ,Bn − G n ℓ we obtain e ℓ,B )∥2I ≤ 16Bn2 δ 2 , ∥(fˆℓ,Bn − G n ℓ which implies ∥fˆℓ,Bn − Gℓ ∥2Iℓ ≤ 32Bn2 δ 2 + 2ϕℓ,n , Suppose now that e ℓ,B )/(4Bn )∥I > δ. ∥(fˆℓ,Bn − G n ℓ 20

(39)

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

Then (37) and (39) imply 7 X (Gℓ (xi ) − fˆℓ,Bn (xi ))2 8nℓ

9ϕℓ,n 16 X ¯ + (f (xi ) − fˆAn (xi ))2 8 nℓ i∈Iℓ q e ℓ,B ∥I · ηℓ,n (δ)/nℓ + 2CUn ∥fˆℓ,Bn − G n ℓ 9ϕℓ,n 16 X ¯ + (f (xi ) − fˆAn (xi ))2 + 8 nℓ i∈Iℓ e ℓ,B ∥2 ∥fˆℓ,Bn − G 4C 2 Un2 ηℓ,n (δ) n Iℓ + 4 nℓ 9ϕℓ,n 16 X ¯ ˆ + (f (xi ) − fAn (xi ))2 + 8 nℓ i∈Iℓ e ℓ,B ∥2 ∥Gℓ − G ∥fˆℓ,Bn − Gℓ ∥2Iℓ 4C 2 Un2 ηℓ,n (δ) n Iℓ + + . 2 2 nℓ

≤

i∈Iℓ

≤

≤

(40)

Hence, 3 ∥Gℓ − fˆℓ,Bn ∥2Iℓ 8

≤ ≤ ≤

17ϕℓ,n 4C 2 Un2 ηℓ,n (δ) + 16∥f¯ − fˆAn ∥2Iℓ + 8 nℓ 17ϕℓ,n 4C 2 Un2 ηℓ,n (δ) 2 + 32∥f˜An − fˆAn ∥Iℓ + 32∥f¯ − f˜∥2Iℓ + 8 nℓ 17ϕℓ,n 4C 2 Un2 ηℓ,n (δ) 2 ˜ ˆ + 32∥fAn − fAn ∥Iℓ + 32ϕn + 8 nℓ

(41)

with f˜ and ϕn as in the notation of Theorem B.1. Moreover, by choosing δ to satisfy ( sup

max

m:

nπ ℓ m d 2 ≤m≤2nπ ℓ {xj }j=1 ⊂[0,1]

1 √ m

Z δ

q

δ 2 /48

) δ p 2 log N (t/4, Fℓ,Bn , ∥·∥m )dt + √ c1 log(48/δ ) ≲ δ 2 , m

(42)

Lemma C.6 implies that the event ( Ω4 :=

sup h∈Hℓ

|∥h∥2Iℓ − ∥h∥2ℓ,L2 | 1 δ2 2 2 ∥h∥ℓ,L2 + 2

) ≤1

holds with probability at least    nπ 3ℓ [1 − c2 exp(−c1 δ 2 nπ ℓ )] · 1 − exp − 27π 2ℓ for constants c1 , c2 > 0. Therefore, from (41) and Lemma C.7, in the event Ω1 ∩ Ω3 ∩ Ω4 , we obtain that ∥Gℓ − fˆℓ,Bn ∥2L2

e ℓ ∥2ℓ,L + 2∥G e ℓ − fˆℓ,B ∥2ℓ,L ≲ 2∥Gℓ − G n 2 2 e ℓ ∥2∞ + 2[2∥G e ℓ − fˆℓ,B ∥2I + δ 2 ] ≤ 2∥Gℓ − G n ℓ 2 2   e ℓ ∥2∞ + 32 17ϕℓ,n + 32∥f˜A − fˆA ∥2I + 32ϕn + 4C Un ηℓ,n (δ) + 2δ 2 ≤ 2∥Gℓ − G n n ℓ 3 8 nℓ Un2 ηℓ,n (δ) 2 2 ˜ ˆ ≲ ϕℓ,n + ∥fAn − fAn ∥Iℓ + ϕn + +δ . nℓ 2 U ηℓ,n (δ) ≲ ϕℓ,n + ∥f˜An − fˆAn ∥2Iℓ + ϕn + n + δ2 . nπ ℓ

Furthermore, let FAn := {fAn /(2An ) : f ∈ F} and H := {f − g : f, g ∈ FAn }. 21

(43)

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

Then, by choosing δ to satisfy ( max

m:

sup

nπ ℓ m d 2 ≤m≤2nπ ℓ {xj }j=1 ⊂[0,1]

1 √ m

Z δ

q

δ 2 /48

) δ p log N (t/4, FAn , ∥·∥m )dt + √ c1 log(48/δ 2 ) ≲ δ 2 , m

(44)

Lemma C.7 also implies that the event ( Ω5 :=

|∥h∥2Iℓ − ∥h∥2ℓ,L2 |

sup

1 δ2 2 2 ∥h∥ℓ,L2 + 2

h∈H

) ≤1

holds with probability at least   nπ 3ℓ [1 − c2 exp(−c1 δ nπℓ )] · 1 − exp − . 27π 2ℓ 

2

As a result, from (43), in the event Ω1 ∩ Ω3 ∩ Ω4 ∩ Ω5 , we have that ∥Gℓ − fˆℓ,Bn ∥2L2

Un2 ηℓ,n (δ) + δ2 nℓ U 2 ηℓ,n (δ) + δ2 ≲ ϕℓ,n + ∥f˜An − fˆAn ∥2ℓ,L2 + ϕn + n nℓ U 2 ηℓ,n (δ) ≲ ϕℓ,n + ∥f˜An − fˆAn ∥2L2 + ϕn + n + δ2 nℓ U 2 ηℓ,n (δ) + δ2 . ≲ ϕℓ,n + rn + n nπ ℓ

≲

ϕℓ,n + ∥f˜An − fˆAn ∥2Iℓ + ϕn +

(45)

Finally, the claim follows from the inequality ∥g0,ℓ − ĝℓ ∥2L2 ≲ ∥Gℓ − fˆℓ,Bn ∥2L2 + ∥f¯ − fˆAn ∥2L2 .

C.4. Proof of Theorem B.4 We apply Theorem B.2 and proceed as in the proof of Theorem 2 in (Padilla et al., 2024a). Specifically, let FBn (Mℓ , νℓ ) := {fBn : f ∈ F (Mℓ , νℓ )}. Then, by Theorem 6 from (Bartlett et al., 2019), it holds that the VC dimension of C(Mℓ , νℓ ) satisfies VC(F (Mℓ , νℓ )) ≲ Mℓ2 νℓ2 log(Mℓ νℓ ). As a result, by Lemma 9.2 and Theorem 9.4 from (Györfi et al., 2002), for any δ ∈ (0, 1) and for any {xj }m j=1 with induced norm ∥ · ∥m , it holds that   log N (δ, FBn (Mℓ , νℓ ), ∥ · ∥m ) ≲ M 2 ν 2 log(Mℓ νℓ ) · log(Bn2 δ −2 ) + log log(Bn2 δ −2 ) ≲ Mℓ2 νℓ2 log(Mℓ νℓ ) log(Bn2 δ −1 ) (46) ≤ C0 (nπ ℓ )ϕℓ,n log3 (nπ ℓ ) log(Bn2 δ −1 ) where C0 > 0 is a constant, and the last inequality holds from our choice of Mℓ and νℓ . Here, m ∈ N is arbitray. Therefore, with Fℓ := F(Mℓ , νℓ ), we have that max

nπ ℓ 2 ≤ nℓ ≤ 2nπ ℓ

sup {xi }i∈Iℓ ⊂[0,1]d

log N (δ, Fℓ,Bn , ∥ · ∥Iℓ ) ≤ ηℓ,n (δ) ∀ δ ∈ (0, 1),

for ηℓ,n (δ) := C0 (nπ ℓ )ϕℓ,n log3 (nπ ℓ ) log(Bn2 δ −1 ). Thus, (17) holds with ηℓ,n (δ) as in (47). 22

(47)

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

Furthermore, for any positive constants C1 and C2 , we have that ∞ X ∞ X

∞   X
′ exp −C1 ηℓ,n (2−l−l −1 ) + exp −C2 ηℓ,n (2−l−1 )

l=0 l′ =1 ∞ X ∞ X

=

l=0 l′ =1 ∞ X

+

l=0

l=0

exp −C1 C0 (l + l′ + 1)nπ ℓ ϕℓ,n log3 (nπ ℓ ) log(2Bn2 )

exp −C2 C0 nπ ℓ ϕℓ,n log3 (nπ ℓ ) log(2Bn2 )

(48)



2 + = exp −C1 C0 nπ ℓ ϕℓ,n log3 (nπ ℓ ) log(2Bn2 ) / 1 − exp −C1 C0 nπ ℓ ϕℓ,n log3 (nπ ℓ ) log(2Bn2 )

 3 3 2 2 exp −C2 C0 nπ ℓ ϕℓ,n log (nπ ℓ ) log(2Bn ) / 1 − exp −C2 C0 nπ ℓ ϕℓ,n log (nπ ℓ ) log(2Bn ) → 0,

n→∞

since lim nπ ℓ ϕℓ,n log3 (nπ ℓ ) log(2Bn2 ) = ∞

n→∞

holds provided that lim nπ ℓ = ∞,

n→∞

which holds by assumption. Thus, we have verified (22). Next, notice that

′ ∞ X ηℓ,n (2−k−k )

sup k∈N

2k′

k′ =1

2

ηℓ,n (2−k )

≤ sup k∈N

∞ X (k + k ′ ) k′ =1

22k′ k

≤ 1,

which verifies (23). Now, it remains to verify (19) and (20) for an appropriate δ > 0. Towards that end, notice that by the proof of Lemma 6 in (Padilla et al., 2024a), (19) holds if δ is chosen to satisfy "r # r log N (1/(48m), Fℓ,Bn , ∥·∥m ) log m δ ≥ max c1 + (49) nπ m m m : 2 ℓ ≤m≤2nπ ℓ for a positive constant c1 . However, by (46), we can take s q log(nπ ℓ ) + ϕℓ,n log3 (nπ ℓ ) log(Bn2 nπ ℓ ) ≲ δ nπ ℓ to ensure that (19) holds. Similarly, by the proof of Lemma 6 in (Padilla et al., 2024a), (20) holds if δ is chosen to satisfy "r # r log N (1/(48m), FAn , ∥·∥m ) log m δ ≥ max c1 + (50) nπ m m m : 2 ℓ ≤m≤2nπ ℓ which holds if we take s

log(nπ ℓ ) + nπ ℓ

s

ϕn log3 (n) log(A2n nπ ℓ ) ≲ δ, πℓ

by our choice of F in Theorem B.2. Therefore, taking s s q log(nπ ℓ ) ϕn log3 (n) log(A2n nπ ℓ ) + + ϕℓ,n log3 (nπ ℓ ) log(Bn2 nπ ℓ ) ≲ δ, nπ ℓ πℓ it follows that (19) and (20) both hold. 23

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

C.5. Proof of Theorem 3.7 Proof. Proceeding as in the proof of Theorem B.2, supposing that e ℓ,B )/(4Bn )∥I > δ, ∥(fˆℓ,Bn − G n ℓ we obtain that, in the event Ω1 ∩ Ω3 ∩ Ω4 , ∥Gℓ − fˆℓ,Bn ∥2L2

2

≲

U ηℓ,n (δ) + δ2 . ϕℓ,n + ∥f˜An − fˆAn ∥2Iℓ + ϕn + n nπ ℓ

(51)

Next, notice that since fˆAn is independent of {(xi )}i∈Iℓ , we obtain that E(∥f˜An − fˆAn ∥2Iℓ fˆAn ) = ∥f˜An − fˆAn ∥2ℓ,L2 . Furthermore, for all i ∈ Iℓ , we have 0 ≤ qi := Also,

(f˜An (xi ) − fˆAn (xi ))2 ≤ 1. 4A2n

E(qi2 | zi = ℓ, f˜An , fˆAn )

≤ =

E(qi | zi = ℓ, f˜An , fˆAn ) ∥f˜An − fˆAn ∥2ℓ,L2 4A2n

Hence, conditionining on Ω2 (ℓ) by Bernstein’s inequality and Lemma C.7, s ∥f˜An − fˆAn ∥2ℓ,L2 log(2n) 4A2n log(2n) ∥f˜An − fˆAn ∥2Iℓ ≤ ∥f˜An − fˆAn ∥2ℓ,L2 + 2An + nℓ 3nℓ 2 log(2n) A ≤ 2∥f˜An − fˆAn ∥2ℓ,L2 + n nℓ 2 log(2n) A n 2 , ≲ ∥f˜An − fˆAn ∥L2 + nπ ℓ

(52)

with probability at least 1 − 1/n.

Let Ω5 be the event that (52) holds. Then with Ω1 , . . . , Ω4 as in the proof of Theorem B.2, we obtain that in Ω1 ∩ Ω2 ∩ Ω3 ∩ Ω 4 ∩ Ω 5 , ∥Gℓ − fˆℓ,Bn ∥2L2

≲ ≲

log n U 2 ηℓ,n (δ) + ϕn + n + δ2 nπ ℓ nℓ U 2 ηℓ,n (δ) log n + n + δ2 . ϕℓ,n + rn + A2n nπ ℓ nπ ℓ

ϕℓ,n + ∥f˜An − fˆAn ∥2L2 + A2n

(53)

Finally, the claim follows as in the proof of Theorem B.2 and the inequality ∥g0,ℓ − ĝℓ ∥2L2 ≲ ∥Gℓ − fˆℓ,Bn ∥2L2 + ∥f¯ − fˆAn ∥2L2 .

C.6. Proof of Corollary 3.8 Proof. The proof follows as that of Theorem B.4. The only difference is that we do not need to enforce (20). Hence, δ can be chosen simply to satisfy (50) which as in the proof of Theorem B.4 can be done with δ satisfying s q log(nπ ℓ ) + ϕℓ,n log3 (nπ ℓ ) log(Bn2 nπ ℓ ) ≲ δ. nπ ℓ 24

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

Then the conclusion follows from Theorem B.2 and by noticing that log n << ϕℓ,n . nπ ℓ

D. Applications to Other Nonparametric Estimators The general theoretical framework developed in Section 3.1 applies to a broad class of nonparametric estimators beyond deep ReLU networks. In this section, we present two additional examples: orthogonal series regression and trend filtering. For trend filtering, we instantiate Theorem B.2 (our general result under shared two-stage estimation), and to the best of our knowledge, the transfer learning setting has not been previously studied for this estimator, with our results yielding the first convergence guarantees in this regime. For orthogonal series regression, we instantiate Theorem 3.7 (our result under independent two-stage estimation) and recover, up to logarithmic factors, the existing convergence rates established in (Wang et al., 2016), providing a unified theoretical perspective on offset-based transfer learning. D.1. Trend Filtering We now apply Theorem B.1 to total variation–based estimators, commonly referred to as trend filtering (Mammen & Van De Geer, 1997; Tibshirani, 2014). Trend filtering estimates a univariate regression function by penalizing the discrete total variation of its rth-order derivative, and is known to adapt to spatially varying smoothness. To the best of our knowledge, the pretraining setting has not been previously studied for this class of estimators. Let S be a space of univariate functions invariant to scalar multiplication and R : S → R a regularizer that acts in S and it is a seminorm. Assumption D.1. Suppose that there exists constants K > 0 and 0 < w ≤ 1 such that for all m ∈ N and δ ∈ (0, 1), it holds that sup log N (δ, BR (1) ∩ B∞ (1), ∥ · ∥m ) ≤ Kδ −w , x1 ,...,xm ⊂[0,1]

where ∥ · ∥m is the empirical norm induced by x1 , . . . , xm , and BR (1) is the unit ball defined by R: BR (1) := {f ∈ S : R(f ) ≤ 1} and B∞ (1) = {f ∈ S : ∥f ∥∞ ≤ 1}. Assumption D.2. Suppose that P(Z = ℓ|X = x) = πℓ ∈ (0, 1) for all x ∈ [0, 1] and ℓ ∈ {1, . . . , L}. Assumption D.2 restricts the probabilities to be constant for notational simplicity. Consider the model in (1) with d = 1 and, for simplicity, with f0 = 0. Our goal is to estimate the functions f0,1 , . . . , f0,L which now coincide with g0,1 , . . . , g0,L , respectively. As an estimator we consider ĝℓ defined in (3) and (4), and with F := {f ∈ S : R(f ) ≤ V } and Fℓ := {f ∈ S : R(f ) ≤ Vℓ }.

(54)

Corollary D.3. Suppose that Assumptions D.1 and D.2 hold and V ≥ V0 and Vℓ ≥ V0,ℓ where ! ! L L X X V0 := R πk g0,k , V0,ℓ := R g0,ℓ − πk g0,k , , k=1

and min{V0 , V0,ℓ } ≳ log n/n

k=1

1/2−1/(2+w)

. Suppose that An is chosen to satisfy An → ∞ and p An ≥ 8 max{∥f¯∥∞ + Un , 8∥f¯∥∞ , 8 ϕn , 1}.

Moreover, assume that Bn is taken to satisfy Bn and (18). Then the estimator ĝℓ defined in (5) satisfies 2w/(2+w)

∥g0,ℓ − ĝℓ ∥2L2 = OP

2w/(2+w)

max{Un2 , An , Bn2 }(V 2w/(2+w) An + Vℓ (nπℓ )2/(2+w)

provided that nπℓ → ∞. 25

2w/(2+w)

Bn

)

! (55)

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

n o Remark D.4. In the natural setting where max Un , An , Bn , ∥f¯∥∞ , maxℓ=1,...,L ∥f0,ℓ ∥∞ , ∥Gℓ ∥∞ = O(poly(log n)), and ignoring logarithmic factors, the convergence rate in (55) simplifies to 2w/(2+w) 2w/(2+w)
(nπℓ )−2/(2+w) , V0 + V0,ℓ provided that V ≍ V0 and Vℓ ≍ V0,ℓ . In contrast, by Theorem B.1, the naive estimator ĥℓ —which only uses data from the ℓth group and is defined as in (13) with Fℓ from (54)—satisfies ! e 2w/(2+w) V ℓ ∥g0,ℓ − ĥℓ ∥2L2 = OP (nπℓ )2/(2+w) ignoring logarithmic factors, and where Veℓ ≍ R(g0,ℓ ). Thus, the pretraining estimator ĝℓ will achieve a faster rate of convergence than the naive estimator ĥℓ provided that ! ! L L X X R πk g0,k + R g0,ℓ − πk g0,k << R(g0,ℓ ), k=1

k=1

which can occur when the functions g0,k are relatively similar, and the averaging of these functions results in a representation with lower complexity, in terms of R, than that of g0,ℓ . We now apply Corollary D.3 to total variation–based estimators, commonly referred to as trend filtering (see (Mammen & Van De Geer, 1997; Tibshirani, 2014)). Towards that end, for a function f : [0, 1] → R, we define its total variation as TV(f ) =

sup

TV(f, M ),

(56)

M ∈N : M ≥1

where TV(f, M ) =

M −1 X

sup 0 ≤ a1 ≤ ... ≤aM ≤1

j=1

|f (aj ) − f (aj+1 )|.

Then, we let S be the class of functions S := {f : [0, 1] → R : TV(f (r−1) ) < ∞} where r ∈ N\{0} is fixed and f (r−1) denotes the (r − 1)th weak derivative of f . With this notation, we are now ready to state our next result. Corollary D.5. Let the (r − 1)th order total variation of function be defined as R(f ) = TV(f (r−1) ) and the corresponding function classes F and Fℓ as in (54). Suppose that the notation and conditions of Corollary D.3 hold. Then the estimator ĝℓ defined in (5) satisfies ! 2/(2r+1) 2/(2r+1) 2/(2r+1) + Vℓ Bn ) max{Un2 , An , Bn2 }(V 2/(2r+1) An 2 ∥g0,ℓ − ĝℓ ∥L2 = OP (57) (nπℓ )2r/(2r+1) n o Remark D.6. Let max Un , An , Bn , ∥f¯∥∞ , maxℓ=1,...,L ∥f0,ℓ ∥∞ , ∥Gℓ ∥∞ = O(poly(log n)) just as in Remark D.4, then the rate achieved by trend filtering with pretraining—namely, the estimator ĝℓ —is 2/(2r+1) 2/(2r+1)
V0 + V0,ℓ (nπℓ )−2r/(2r+1) , while the rate attained by the usual trend filtering estimator is (r−1)
2/(2r+1) TV(g0,ℓ ) (nπℓ )−2r/(2r+1) . Therefore, the pretraining estimator achieves a faster convergence rate than the usual trend filtering estimator whenever the functions L L X X πk g0,k and g0,ℓ − πk g0,k k=1

k=1

are substantially smoother than g0,ℓ , in terms of their total variation. 26

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

Proof of Corollary D.3 Proof. With the notation from Theorem B.1, we let FAn := {fAn /(2An ) : f ∈ F}. Then log N (δ, FAn , ∥·∥n )

≤ log N (δ/V, FAn /V, ∥·∥n ) ≤ log N (δ/V, BR (1) ∩ B∞ (1), ∥·∥n ) ≤ KV w δ −w w −w ≤ KAw nV δ

(58)

Hence, we define w −w ηn (δ) := KAw , nV δ

and so by Assumption D.2 we proceed to check the conditions of Theorem B.1. To verify (14), we observe that ∞ X ∞ X

∞   X
′ exp −C1 ηn (2−l−l −1 ) + exp −C2 ηn (2−l−1 )

l=0 l′ =1 ∞ X ∞ X

=

l=0 l′ =1

≤ =

∞ ∞ X X

l=0 l′ =1

∞  X l=0 ∞ X l=0

l=0



exp −C1 K

w

w (l+l′ +1)w Aw nV 2



+

∞ X

l=0 ∞ X

w ′ exp (−C1 K w Aw n V (l + l + 1)w) +

exp (−C1 K

w

w Aw n V (l + 1)w) ·

  w (l+1)w exp −C2 K w Aw nV 2

∞ X l′ =1

l=0

w exp (−C2 K w Aw n V (l + 1)w) l′

w [exp (−C1 K w Aw n V w)]

 +

(59)

l+1

w [exp (−C2 K w Aw n V w)]

2

w w w w = [exp (−C1 K w Aw n V w) / [1 − exp (−C1 K An V /w)]] + w w w w w w exp (−C2 K An V1 w) / [1 − exp (−C2 K An V w)] → 0, n→∞

since by construction An → ∞. Thus, (14) holds. To verify (15), notice that sup k∈N

′ ∞ X ηn (2−k−k )

k′ =1

22k′ ηn (2−k )

∞ X

1 ≤ 1. (2−w)k′ 2 k∈N ′ k =1

= sup

(60)

Next, we construct a δ that is a critical radius. Towards that end, we notice that as in Lemma 6 of (Padilla et al., 2024a), it is enough to have δ satisfy Z δ q 1 δ p √ log N (t/2, FAn , ∥·∥n )dt + √ log(48/δ 2 ) ≲ δ 2 . (61) n δ2 /4 n However, 1 √ n

Z δ q

δ p log N (t/2, FAn , ∥·∥n )dt + √ log(48/δ 2 ) n δ 2 /4 Z δq 1 δn p ≤√ log N (t/2, FAn , ∥·∥n )dt + √ log(48/δ 2 ) n Z0 n δp 1 δ p w −w dt + √ ≲√ Aw log(48/δ 2 ) nV t n 0 n w/2 V w/2 An δ 1−w/2 δ p √ ≲ + √ log(48/δ 2 ). n n

(62)

Hence, (61) holds for δ > 0 satisfying δ 2 ≍ n−2/(2+w) V 2w/(2+w) A2w/(2+w) . n 27

(63)

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

Thus, δ is a critical radius for FAn . Moreover, in this case, we also have that ηn (δ) ≲ δ2 . n Therefore, from Theorem B.1, we obtain that max{∥f¯ − fˆAn ∥2n , ∥f¯ − fˆAn ∥2L2 } = OP (rn ) , where

2w/(2+w)

rn :=

max{Un2 , An }An n2/(2+w)

V 2w/(2+w)

(64)

.

Next, we analyze the second stage estimator using Theorem B.2. Let Fℓ,Bn := {fBn /(2Bn ) : f ∈ Fℓ }. Then, taking ηℓ,n (δ) := KBnw Vℓw δ −w , proceeding as in (58), we obtain that max

nπℓ 2 ≤ nℓ ≤ 2nπℓ

sup {xi }i∈Iℓ ⊂[0,1]d

log N (δ, Fℓ,Bn , ∥ · ∥Iℓ ) ≤ ηℓ,n (δ).

which shows (17). Moreover, we can verify (22) proceeding as in (59), and (23) as in (60). Next, we proceed to find δ for which (19) and (20) hold. However, based on our previous calculations (19) and (20) are equivalent to Z δ q 1 δ p Bnw Vℓw t−w dt + √ log(48/δ 2 ) ≲ δ 2 , √ nπℓ δ2 /48 nπℓ and √

1 nπℓ

Z δ δ 2 /48

p

w −w dt + √ Aw nV t

δ p log(48/δ 2 ) ≲ δ 2 , nπℓ

respectively. Hence, (19) and (20) hold for a choice of δ satisfying h i 2w/(2+w) 2w/(2+w) δ 2 ≍ (nπℓ )−2/(2+w) V 2w/(2+w) A2w/(2+w) + Vℓ Bn . n Therefore, the claim follows from Theorem B.2.

Proof of Corollary D.5 Proof. The statement of Corollary D.5 follows from Corollary D.3, since Assumption D.1 is satisfied for the trend filtering regularizer. In particular, by Corollary 1 of (Sadhanala & Tibshirani, 2019), Assumption D.1 holds with w = 1/r for R(f ) = TV(f (r−1) ), corresponding to k = r − 1 in their notation. D.2. Orthogonal Series Regression We now apply Theorem 3.7 to orthogonal series estimator, instantiating the function classes F and Fℓ as Sobolev ellipsoids on [0, 1]d . This setting recovers, up to logarithmic factors, the two-task transfer learning rate established in (Wang et al., 2016), whose offset-based decomposition (Model 4.1 of (Wang et al., 2016)) parallels our two-stage procedure. Setup. Following the set up in (Wang et al., 2016), we specialize the model in (1) to L = 2 groups, with group ℓ = 1 playing the role of the target. The data consist of n i.i.d. copies of (X, Y, Z), partitioned into the target group I1 := {i : zi = 1} of size n1 := |I1 | and the source group I2 := {i : zi = 2} of size n − n1 . Following (2), the target-group conditional mean function decomposes as g0,1 (x) := E[Y | X = x, Z = 1] = f¯(x) + G1 (x), 28

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

where

f¯(x) := E[Y | X = x]

G1 (x) := g0,1 (x) − f¯(x)

and

denote the overall mean and the target-specific offset, respectively, as introduced in Section 2. Throughout this subsection, we work under the independent two-stage estimation setting of Theorem 3.7: the Stage-1 estimator fˆ is constructed using a sample independent of the one used in Stage 2. d d According to (Wang et al., R 2016), let L2 ([0, 1] ) denote the space of square-integrable functions on [0, 1] equipped with the inner product ⟨f, g⟩ = [0,1]d f (x)g(x) dx. Let {φj }j∈Z be an orthonormal basis of L2 ([0, 1]) with supj ∥φj ∥∞ ≤ Cφ for some constant Cφ > 0. For example, we can consider the cosine basis in the following: ( 1, j = 0, φj (x) = √ 2 cos(jπx), j ≥ 1, √ where the index j orders the basis functions by frequency and supj ∥φj ∥∞ ≤ Cφ with Cφ = 2. The tensor product basis is defined as {φα }α∈Zd , where

φα (x) =

d Y

α = (α1 , . . . , αd ) ∈ Zd ,

φαw (xw ),

w=1

forms an orthonormal basis of L2 ([0, 1]d ), so every f ∈ L2 ([0, 1]d ) admits the expansion f = aα (f ) := ⟨φα , f ⟩.

P

α aα (f ) φα with

Definition D.7 (Sobolev Ellipsoid). Fix smoothness s > 0, radius A > 0, and uniform bound fmax > 0, where A and fmax are taken to be common across all smoothness levels s for simplicity. The (isotropic) Sobolev ellipsoid on [0, 1]d is     X W s := f ∈ L2 ([0, 1]d ) : aα (f )2 κ2s (α) ≤ A2 , ∥f ∥∞ ≤ fmax ,   d α∈Z

where κ2s (α) :=

d X w=1

|αw |2s .

P The quantity κ2s (α) assigns larger weights to higher-frequency basis functions. Hence the constraint α∈Zd aα (f )2 κ2s (α) ≤ A2 forces the coefficients aα (f ) corresponding to high-frequency basis functions to decay sufficiently fast. In this sense, larger values of s correspond to smoother functions. For each smoothness level s > 0 and threshold t > 0, define the set n o Ms (t) := α ∈ Zd : κ2s (α) ≤ t , which collects all multi-indices whose Sobolev weight does not exceed t. We then write J := |Ms (t)| for the cardinality of Ms (t), which stands for the number of basis functions retained at threshold t. Function Classes. We assume that both the overall mean function and the target offset belong to Sobolev ellipsoids: f¯ ∈ W τ ,

G1 ∈ W ν .

The smoothness levels τ, ν > 0 are allowed to differ. In particular, ν > τ corresponds to the regime in which the offset G1 is smoother than the overall mean f¯, which is the setting where transfer learning is expected to be most beneficial. Now consider the Stage-1 estimator. Define the Stage-1 threshold t0 > 0 and let J0 := |Mτ (t0 )| denote the number of basis functions retained. For each α ∈ Mτ (t0 ), let θα ∈ R be the scalar coefficient associated with the basis function φα ; collecting these J0 coefficients into the vector θ = (θα )α∈Mτ (t0 ) ∈ RJ0 , we estimate the Stage-1 target function f¯ using fˆ ∈ Ft0 , where n X o Ft0 := θα φα : ∥θ∥2 ≤ R0 , α∈Mτ (t0 )

29

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

with a fixed R0 ∈ R+ , and fˆ := arg min

f ∈Ft0

n X i=1

2 yi − f (xi ) .

Here Ft0 consists of all linear combinations of the J0 lowest-frequency tensor-product basis functions selected under the Sobolev weight corresponding to the smoothness τ . Analogously, for the Stage-2 estimator, let t1 > 0 and J1 := |Mν (t1 )|. For each α ∈ Mτ (t1 ), let γα ∈ R be the scalar coefficient associated with the basis function φα ; collecting these J1 coefficients into the vector γ = (γα )α∈Mτ (t1 ) ∈ RJ1 , we estimate the Stage-2 target function G1 using fˆ1 ∈ Gt1 , where n X o Gt1 := γα φα : ∥γ∥2 ≤ R1 , α∈Mν (t1 )

with a fixed R1 ∈ R+ , and

fˆ1 := arg min

f ∈Gt1

X i: zi =1

2 yi − fˆAn (xi ) − f (xi ) ,

where we focus on group 1 for simplicity. The final estimator would be ĝ1 (x) := fˆAn (x) + fˆ1,Bn (x). Now we are ready to state our results. Corollary D.8. Suppose that P(Z = ℓ | X = x) = πℓ ∈ (0, 1) for all x ∈ [0, 1] and ℓ ∈ {1, 2}, that f¯ ∈ W τ and G1 ∈ W ν . Assume further that the two stages are estimated on independent samples as in Theorem 3.7, with truncation levels An = Bn = fmax . Suppose that Un grows at most in O(poly(log n)). With  d/(2τ +d)  d/(2ν+d) n n1 J0 ≍ , J1 ≍ , log n log n and n1 ≍ nπ1 , the transfer learning estimator ĝ1 defined in (5) satisfies   2ν 2τ ∥g0,1 − ĝ1 ∥2L2 = OP n− 2τ +d + n1 − 2ν+d , up to logarithmic factors. Remark D.9. The two terms in the rate correspond, respectively, to the Stage-1 error of estimating f¯ from the full sample of size n, and the Stage-2 error of estimating the offset G1 from the target-group sample of size n1 . To compare with the simplified rate of (Wang et al., 2016), parameterize n = nλ1 for some λ ≥ 1 — in their two-task setting, λ captures the relative abundance of source data, with λ = 1 corresponding to balanced source and target sample sizes and λ → ∞ corresponding to a much larger source pool. Substituting yields   2λτ 2ν − 2τ − 2ν+d +d 2 ∥g0,1 − ĝ1 ∥L2 = OP n1 + n1 up to logarithmic factors, which matches the simplified rate stated immediately after Theorem 4.3 of (Wang et al., 2016). Consistent with Remark 3.5, the pretraining estimator achieves a faster rate than the naive estimator using only target-group data — whose rate is (log n1 /n1 )2σ/(2σ+d) when g0,1 ∈ W σ — whenever f¯ and G1 are smoother than g0,1 itself (i.e., τ, ν > σ, with λ large enough). D.3. Proof of Corollary D.8 We first establish the approximation rate for d-dimensional Sobolev ellipsoids by sieve classes. At the beginning, we state a standard result for approximating a generic function f in a one-dimensional Sobolev ellipsoid. The purpose of the lemma is to bound the projection error incurred by keeping only the first J low-frequency basis functions. Lemma D.10 (One-Dimensional Approximation by Sobolev Sieves (Lemma 8.4 in (Wasserman, 2006))). Consider f ∈ W s . Let d = 1, so that the multi-index α reduces to a scalar index α ∈ Z, and κ2s (α) = |α|2s . Let Ms (t) ⊂ Z denote the indices of the J one-dimensional basis functions having J := |Mτ (t)|, and define the projection X fJ := aα (f )φα . α∈Ms (t)

30

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

If

2s 2 α∈Z |α| aα (f ) ≤ C, then

P

∥f − fJ ∥2L2 =

X

aα (f )2 ≤ CJ −2s .

α∈M / s (t)

Now we extend this result to d-dimension. Proposition D.11. Let f : [0, 1]d → R be a function in W s , and let Ms (t) be the set of indices defined earlier, which contains the J multi-indices with the Sobolev weight κ2s (α) truncated by t. For all J sufficiently large, it holds that X inf ∥f − f˜∥2L2 = aα (f )2 ≤ C · J −2s/d , f˜∈Ft

α∈M / s (t)

with α ∈ Zd for some constant C not depending on J. Proof. For α ∈ / Ms (t), κ2s (α) > t, so X α∈M / s (t)

aα (f )2 ≤ t−1

X α∈Zd

aα (f )2 κ2s (α) ≤

A2 . t

(65)

Pd 2s where Ms (t) = {α : ≤ t}, which forces |αw | ≤ t1/(2s) for each dimension w. Hence J ≤ w=1 |αw |

d d 2⌊t1/(2s) ⌋ + 1 ≤ 3t1/(2s) for t1/(2s) ≥ 1, giving t≥

J 2s/d . 32s

(66)

The case t1/(2s) < 1 is degenerate: since κ2s (α) ≥ 1 for every nonzero integer multi-index α, it forces Ms (t) = {(0, . . . , 0)} and J = 1, which is excluded in the regime of interest where J → ∞. Combining (65) and (66) yields the claim. d/(2s) Remark D.12. The proof of Proposition D.11 establishes the upper bound J ≲ td/(2s) . A matching lower   bound J d≳ t 1/(2s) follows analogously by counting the integer points in a box contained in Ms (t). Let Λ := (t/d) . If α ∈ Z satisfies |αw | ≤ Λ for all w = 1, . . . , d, then d X t |αw |2s ≤ dΛ2s ≤ d · = t. d w=1

Hence {−Λ, . . . , Λ}d ⊆ Ms (t), and therefore  j k d J = |Ms (t)| ≥ (2Λ + 1)d = 2 (t/d)1/(2s) + 1 . For (t/d)1/(2s) ≥ 1, this implies J ≳ td/(2s) . Combining the two bounds gives J ≍ td/(2s) . For simplicity we track the dimension J rather than the truncation t. Approximation Error.

By Proposition D.11, −2τ /d

ϕn := inf ∥f¯ − f ∥2L2 ≲ J0 f ∈Ft0

−2ν/d

ϕ1,n := inf ∥G1 − g∥2L2 ≲ J1

,

g∈Gt1

.

Empirical Covering Entropy. Since Ft0 and Gt1 are finite-dimensional linear spans of bounded basis functions, each function is uniquely parameterized by its coefficient vector. Moreover, for any sample {xi }ni=1 and any coefficient vectors θ, θ′ , n

∥fθ − fθ′ ∥2n =

1 X n i=1

X

n 2 1 X (θα − θα′ )φα (xi ) ≤ n i=1

α∈Mτ (t0 )

X

(θα − θα′ )2

α∈Mτ (t0 )



X α∈Mτ (t0 )

 φα (xi )2 ≤ J0 Cφ2d ∥θ − θ′ ∥22 ,

where the first inequality follows from the Cauchy–Schwarz inequality and the second uses the uniform bound supj ∥φj ∥∞ ≤ √ Cφ . Equivalently, ∥fθ − fθ′ ∥m ≤ Cφ2d J0 ∥θ − θ′ ∥2 . The same bound holds for Gt1 with J1 in place of J0 . Consequently, 31

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

√ any (δ/(Cφ2d J0 ))-cover of the Euclidean ball B2J0 (R0 ) := {θ ∈ RJ0 : ∥θ∥2 ≤ R0 } in RJ0 induces a δ-cover of Ft0 under ∥ · ∥n :   p N (δ, Ft0 , ∥ · ∥n ) ≤ N δ/(Cφ2d J0 ), B2J0 (R0 ), ∥ · ∥2 . Combined with the standard covering bound N (η, B2J (R), ∥ · ∥2 ) ≤ (3R/η)J (Corollary 4.2.11 of (Vershynin, 2026)), we obtain p p log N (δ, Ft0 , ∥ · ∥m ) ≲ J0 log(C J0 /δ), log N (δ, Gt1 , ∥ · ∥m ) ≲ J1 log(C J1 /δ). (67) Verification of Conditions (14) and (15) for Stage 1. By the entropy bound established above, we may take the majorant p ηn (δ) := c0 J0 log(C J0 /δ), under our assumption An = O(1), which satisfies log N (δ, Ft0 ,An , ∥ · ∥n ) ≤ ηn (δ) as required by Theorem B.1. √ √ Step 1: Verification of (14). Under increasing J0 with n, log(C J0 ) > 0, so for any k ≥ 0 and k ′ ≥ 1, p   ′ ηn (2−k−k −1 ) = c0 J0 (k + k ′ + 1) log 2 + log(C J0 ) ≥ c0 J0 (k + k ′ + 1) log 2. Hence for any C1 > 0, ∞ X ∞ X k=0 k′ =1

∞ X ∞
X ′ ′ exp −C1 ηn (2−k−k −1 ) ≤ ρ k+k +1 = ρ · k=0 k′ =1

1 ρ ρ2 · = →0 1−ρ 1−ρ (1 − ρ)2

as J0 → ∞, where ρ := exp(−C1 c0 J0 log 2) ∈ (0, 1). The second summation in (14) is bounded analogously: ∞ X k=0

∞
X exp −C2 ηn (2−k−1 ) ≤ ρ′ k+1 = k=0

ρ′ → 0, 1 − ρ′

where ρ′ := exp(−C2 c0 J0 log 2) ∈ (0, 1). Together with the noise tail assumption P(∥ϵ∥∞ > Un ) → 0, condition (14) is verified. √ Step 2: Verification of (15). For k ≥ 1 and k ′ ≥ 1, since log(C J0 ) > 0, √ ′ (k + k ′ ) log 2 + log(C J0 ) k + k′ ηn (2−k−k ) √ = ≤ . −k ηn (2 ) k k log 2 + log(C J0 ) Therefore sup k≥1

′ ∞ X ηn (2−k−k )

k′ =1

22k′ ηn (2−k )

≤ sup k≥1

∞ X k + k′ k′ =1

" ∞ #   ∞ X 1 1 X k′ 1 4 7 = sup + = sup + = < 1, 22k′ k 4k ′ k ′ 4k′ 9k 9 k≥1 ′ k≥1 3

where we used the geometric-series identities identically. This verifies (15).

k =1

P

k′ ≥1 4

−k′

= 1/3 and

k =1

P

k′ ≥1 k

′

′

· 4−k = 4/9. The case k = 0 is handled

Verification of Condition (19) for Stage 2. Under the independent two-stage estimation setting of Theorem 3.7, condition (20) is replaced by the independence assumption between the data used in Stages 1 and 2, so only condition (19) on the Stage-2 requires verification. Consider m ∈ [nπℓ /2, 2nπℓ ]. By the entropy bound for Gt1 established above, q q p log N (t/4, Gt1 ,Bn , ∥ · ∥m ) ≲ J1 log(C J1 /t). Therefore, the Stage-2 complexity integral satisfies 1 √ m

Z δ δ 2 /48

q

√ Z δ q p J1 log N (t/4, Gt1 ,Bn , ∥ · ∥m ) dt ≲ √ log(C J1 /t) dt. m δ2 /48 32

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

p √ √ Since t 7→ log(C J1 /t) is decreasing on (0, C J1 ), Z δ q q p p log(C J1 /t) dt ≲ δ log(C J1 /δ 2 ). δ 2 /48

Hence Z δ

q  p p δ p δ 2 2 2 log N (t/4, Gt1 ,Bn , ∥ · ∥m ) dt + √ c1 log(48/δ ) ≲ √ J1 log(C J1 /δ ) + log(1/δ ) . m m δ 2 /48 √ Thus condition (19) holds whenever δ 2 ≳ J1 log(C J1 /δ 2 )/(nπℓ ). Under n1 ≍ nπ1 , the choice 1 √ m

q

δ2 ≍

J1 log(J1 · n) n1

suffices. Combining the Two Stages.

By Theorem B.1, applied to the Stage-1 sieve class Ft0 , the Stage-1 error satisfies −2τ /d

rn ≲ J0

+

J0 log(J0 n) n −2ν/d

up to polylogarithmic factors. Applying Theorem 3.7 with ϕ1,n ≲ J1 and effective target sample size n1 ≍ nπ1 ,   J1 log(J1 n) J0 log(J0 n) −2ν/d −2τ /d + J1 + ∥g0,1 − ĝ1 ∥2L2 = OP J0 + n n1 up to polylogarithmic factors. Balancing bias and variance in each stage yields the choices of J0 and J1 stated in Corollary D.8, and substituting back gives the stated rate.

E. Illustrative Examples to Interpret the Theoretical Results We provide three illustrative examples to complement the theoretical results in the main text. The first example shows that the two-stage transfer learning procedure can yield estimation targets that are considerably simpler than direct estimation of the group-specific functions. The second example instantiates the convergence rate ϕn + ϕℓ,n under a concrete compositional structure with 10-dimensional input. The third example compares convergence rates across four estimators under different regimes of the group proportion πℓ , illustrating the advantage of NN with TL over classical methods. E.1. Two-Stage Targets Can Be Simpler Than Direct Estimation We provide a concrete example illustrating that, under the hierarchical compositional structure, both the shared and groupspecific functions can individually be simple, yet their sum is complex, thus yielding simpler estimation targets in the two-stage transfer learning procedure. Consider L = 2 groups with proportions π1 + π2 = 1 and group-specific regression functions g0,1 (x1 , . . . , x5 ) = f0 (x1 , x2 ) + f0,1 (x3 , x4 , x5 ),

g0,2 (x1 , . . . , x5 ) = f0 (x1 , x2 ) + f0,2 (x3 , x4 , x5 ).

Each g0,ℓ depends on all 5 variables, so direct estimation incurs rates governed by dimension 5. Now suppose the group deviation functions satisfy π1 f0,1 (x3 , x4 , x5 ) + π2 f0,2 (x3 , x4 , x5 ) ≈ c for some constant c. Then the first-stage target

f¯ = f0 (x1 , x2 ) + π1 f0,1 + π2 f0,2 ≈ f0 (x1 , x2 ) + c effectively depends only on 2 variables, yielding faster first-stage convergence rates. The second-stage offset for group 2, G2 = π2 (f0,2 − f0,1 ), depends on 3 variables and becomes nearly zero as π2 → 0, making second-stage estimation easy. 33

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

E.2. Concrete Instantiation of Convergence Rates We provide a concrete example to illustrate the final convergence rate ϕn + ϕℓ,n under specific settings. Consider L = 2 groups with proportions π1 + π2 = 1 and 10-dimensional input x ∈ R10 . Both groups share the same compositional (1) (1) structure as illustrated in Figures 2 and 3, where the shared components h1 , h2 have input dimension K = 2, 3, 3 and (1) smoothness p = 1.5, 1.5, 2.5, respectively. The two groups differ only in the group-specific component h3,groupℓ (in red color), with K = 2, p = 1.5 for both groups. (2)

h1 (u1 , u2 , u3 ) = |u1 − 1|2.5 + |u2 − 1|2.5 + |u3 − 1|2.5

(1)

xπ(1)

xπ(3)

x2

h3, group1 (u1 , u2 ) = |u1 |1.5 + |u2 |1.5

h2 (u1 , u2 , u3 ) = |u1 |1.5 + |u2 |1.5 + |u3 |1.5

xπ(2)

x1

(1)

(1)

h1 (u1 , u2 ) = |u1 |1.5 + |u2 |1.5

x3

x4

xπ(4)

xπ(5)

x5

x6

x7

xπ(6)

x8

x9

xπ(7)

x10

Figure 2. Illustration of the estimated group mean function g0,1 (x) for Group 1, where x ∈ R10 .

(2)

h1 (u1 , u2 , u3 ) = |u1 − 1|2.5 + |u2 − 1|2.5 + |u3 − 1|2.5

(1)

xπ(1)

xπ(3)

x2

h3, group2 (u1 , u2 ) = 2(|u1 |1.5 + |u2 |1.5 )

h2 (u1 , u2 , u3 ) = |u1 |1.5 + |u2 |1.5 + |u3 |1.5

xπ(2)

x1

(1)

(1)

h1 (u1 , u2 ) = |u1 |1.5 + |u2 |1.5

x3

x4

xπ(4)

x5

xπ(5)

x6

x7

xπ(6)

x8

x9

xπ(7)

x10

Figure 3. Illustration of the estimated group mean function g0,2 (x) for Group 2, where x ∈ R10 .

Estimating f¯ = π1 g0,1 + π2 g0,2 in the first stage preserves the same compositional structure as g0,1 and g0,2 , giving an upper bound ϕn = max n−2p/(2p+K) = n−1/2 . (p,K)∈{(1.5, 2), (1.5, 3), (2.5, 3)}

In the second stage, the offset G1 = π2 (g0,1 − g0,2 ) cancels all shared components, reducing to a 2-layer model with a single branch, giving an upper bound ϕ1,n =

max

(p,K)∈{(1.5, 2), (2.5, 1)}

(nπ1 )−2p/(2p+K) = (nπ1 )−3/5 .

The overall rate n−1/2 + (nπ1 )−3/5 depends only on the intrinsic dimensions of the compositional structure, not on the ambient dimension d = 10. In contrast, standard estimators that treat all 10 dimensions yield the slower rate n−3/13 + (nπ1 )−3/13 , demonstrating the advantage of our method in overcoming the curse of dimensionality. 34

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

E.3. Benefit of Transfer Learning with DNNs over Classical Estimators We illustrate the benefit of transfer learning (TL) with DNNs in comparison with classical estimators, such as transfer learning with kernel smoothing (Du et al., 2017), through a simple example applying Corollary 3.8. Let X ∈ [0, 1]10 and ℓ ∈ {1, 2, 3} denote the group index. For simplicity, suppose that π̄ℓ = π ℓ = πℓ for all ℓ. We observe Yi = g0,ℓ (Xi ) + εℓ,i , and consider the additive decomposition g0,ℓ (x) = f0 (x) + f0,ℓ (x),

ℓ = 1, 2, 3,

with shared component f0 (x) = |x1 + · · · + x10 − 1/2| and group-specific components f0,1 (x) = |x1 +x2 +x3 +x4 −1/2|3/2 ,

f0,2 (x) = 2|x1 +x2 +x3 +x4 −1/2|3/2 ,

f0,3 (x) = 3|x1 +x2 +x3 +x4 −1/2|3/2 . q (ℓ) (ℓ) Each group may further have its own noise distribution; for example, εℓ,i ∼ N (0, ℓ · Xi ), where Xi denotes the ℓ-th coordinate of Xi . Since each g0,ℓ and the distribution of εℓ can differ across groups. We now use the example above to compare convergence rates under different πℓ with directly comparable classical estimators under the similar offset TL framework, and illustrate the novelty of our results.

(a) Without Transfer Learning. In this example, the smoothness of g0,ℓ is p = 1 with intrinsic dimensionalities K = 10. Training each group separately on nπℓ samples, classical estimators must use all d = 10 dimensions, yielding rate 2p

(nπℓ )− 2p+d = (nπℓ )−1/6 . Direct NN estimation without transfer learning exploits intrinsic dimensionality K = 10, also giving 2p

(nπℓ )− 2p+K = (nπℓ )−1/6 . This is a deliberately chosen example where direct NN estimation without TL cannot improve upon the classical rate, specifically constructed to motivate the benefit of TL with NN in the subsequent comparison. In most settings, NN estimation does outperform classical methods by exploiting intrinsic dimensionality; in particular, when d > 10 but K = 10, the NN achieves a strictly faster rate by overcoming the curse of dimensionality. (We note that the same function may admit different hierarchical compositions with different intrinsic dimensionalities, since our upper bounds apply at the function class level rather than to a specific function instance. The composition with intrinsic dimension K = 10 chosen here is just for illustrative purposes.) (b) With Transfer Learning. The first-stage target is f¯(x) = |x1 + · · · + x10 − 1/2| + (π1 + 2π2 + 2π3 ) · |x1 + x2 + x3 + x4 − 1/2|3/2 , which has smoothness p = 1 and intrinsic dimension K = 10. The second-stage offset (taking group 1 as an example) is G1 (x) = (1 − π1 − 2π2 − 3π3 ) · |x1 + x2 + x3 + x4 − 1/2|3/2 , which has smoothness p = 3/2 and intrinsic dimension K = 4, hence simpler than g0,ℓ and f¯. Under this setting, classical estimators, which require all 10 dimensions for both f¯ and G1 , yield rate n−1/6 + (nπ1 )−3/13 . Our Theorem 3.7, by exploiting the intrinsic dimensionality of G1 , yields the faster rate n−1/6 + (nπ1 )−3/7 . The four regimes of convergence rates for all four estimators as π1 varies are summarized in the table below. Specifically: 1. When π1 ≍ 1, all methods achieve n−1/6 and TL provides no additional gain. 35

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

2. When n−5/18 ≪ π1 ≪ 1, both classical and NN methods with TL achieve the faster rate n−1/6 , while classical and NN without TL are slower at (nπ1 )−1/6 . 3. When n−11/18 ≲ π1 ≲ n−5/18 , NN with TL still achieves n−1/6 while the classical TL rate degrades to (nπ1 )−3/13 ; for instance, at π1 ≍ n−11/18 , our rate is n−1/6 versus n−7/78 for classical TL.

4. When n−1 ≪ π1 ≪ n−11/18 , NN with TL at (nπ1 )−3/7 remains strictly faster than classical TL at (nπ1 )−3/13 . The condition n−1 ≪ π1 follows from the group probability requirement in Theorem B.2.

This example clearly illustrates how a small π1 can drastically worsen the classical rate while NN with TL remains robust.

π1 ≍ 1 5 n− 18 ≪ π1 ≪ 1 11 5 n− 18 ≲ π1 ≲ n− 18 11 n−1 ≪ π1 ≪ n− 18

Classical w/o TL

NN w/o TL

Classical w/ TL

NN w/ TL

− 16

− 16

− 61

n− 6 1 n− 6 1 n− 6 3 (nπ1 )− 7

n 1 (nπ1 )− 6 1 (nπ1 )− 6 1 (nπ1 )− 6

n 1 (nπ1 )− 6 1 (nπ1 )− 6 1 (nπ1 )− 6

n 1 n− 6 3 (nπ1 )− 13 3 (nπ1 )− 13

1

Table 5. Convergence rates under different regimes of π1 .

F. Details of Numerical Experiments F.1. Complete Numerical Results In Section 4 of the main text, we selectively present four scenarios from our simulation experiments. Scenarios 1 and 2 correspond to low-dimensional settings, while Scenarios 3 and 4 are high-dimensional. In this supplementary section, we report results for four additional low-dimensional scenarios (Extra Scenarios 1–4) to provide a more comprehensive evaluation of method performance in low-dimensional settings. Extra Scenario 1. This scenario features a shared additive model with simple group-specific shifts. Let the input vector i.i.d. L−z+1 be q = (q1 , . . . , q10 ) ∼ Unif([0, 1]10 ), holding for Extra Scenario 1–4. With a group-varying scale wz = L(L+1)/2 for group z, the shifts are applied to the last five dimensions as follows: ( qj if j = 5 + (z mod 5) or j = 6 + (z mod 5) (z) q̃j = qj + wz · sin(z · qj ) if j ∈ {6, 7, 8, 9, 10} \ {5 + (z mod 5), 6 + (z mod 5)}. P5 P10 (z) (z) Define y = gz (q) + ϵ, where gz (q) = f0 (q) + f0,z (q) = j=1 θj (qj ) + j=6 θj (q̃j ) as the group-specific conditional mean function. The component functions {θj : R → R}10 j=1 are defined as follows: q θ1 (q1 ) = sin(tanh(q12 )), θ6 (q6 ) = cos( 1 + q62 ), θ2 (q2 ) = log(1 + |q2 |), q θ3 (q3 ) = exp(− 1 + q32 ),

θ7 (q7 ) = − log(1 + tanh(q72 )),

θ4 (q4 ) = cos(log(1 + q42 )),

θ9 (q9 ) = cos(tanh(q92 )), q 2 ). θ10 (q10 ) = − exp(− 1 + q10

θ5 (q5 ) = − sin(tanh(|q5 |)),

θ8 (q8 ) = − sin(log(1 + q82 )),

Extra Scenario 2. This scenario combines covariate shifts with mild concept shifts. The shared function remains identical P5 to that in Extra Scenario 1, given by f0 (q) = j=1 θj (qj ). The group-specific effects introduce covariate shifts through the group scale wz , defined as ( qj if j = 5 + z (z) q̃j = qj + wz · sin(z · qj ) if j ∈ {6, 7, 8, 9, 10} \ {5 + z}. The deviation function f0,z further incorporates concept shift by applying a group-specific scaling f0,z (q) = (1 + wz ) · P10 (z) j=6 θj (q̃j ), where θ6 , . . . , θ10 are defined as in Extra Scenario 1. 36

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

Extra Scenario 3. This scenario introduces a two-layer hierarchical structure where inputs are first mapped to intermediate representations before final transformation. With the scaling factor wz , let ( qj if j = 5 + (z mod 5) or j = 6 + (z mod 5) (z) q̃j = qj + wz · sin(z · qj ) if j ∈ {6, 7, 8, 9, 10} \ {5 + (z mod 5), 6 + (z mod 5)} The intermediate transformations are: h1 (q) = sin(q1 ) · q22 + exp(q3 ) − q4 · q5 , h2 (q) = cos(q2 ) + q3 · tanh(q4 ) + q53 ,

h3 (q; z) = log (1 + q̃6 · q̃7 − sin(q̃8 )) − tanh(q̃9 · q̃10 ),

2 −1 h4 (q; z) = exp(−|q̃8 |) + q̃9 · (1 + q̃10 ) .

The shared and group-specific functions are constructed as: p p f0 (q) = 1 + h1 (q)2 + 1 + h2 (q)2

and

f0,z (q) = |h3 (q; z) − h4 (q; z)| ,

and the final response is generated according to y = f0 (q) + f0,z (q) + ϵ. Extra Scenario 4. This scenario extends Extra Scenario 3 beyond the additive model by introducing more complex group effects through mixed dimensional shifts. Specifically, for even indices j ∈ {2, 4, . . . , 10}, we define (z)

q̃j

= qj + wz · sin(z qj ).

Group effects are then incorporated by modifying the even-indexed dimensions in the following components: h1 (q; z) = sin(q1 ) · q̃22 + exp(q3 ) − q̃4 · q5 , h2 (q; z) = cos(q̃2 ) + q3 · tanh(q̃4 ) + q53 ,

h3 (q; z) = log (1 + q̃6 · q7 − sin(q̃8 )) − tanh(q9 · q̃10 ),

2 −1 h4 (q; z) = exp(−|q̃8 |) + q9 · (1 + q̃10 ) .

The final conditional mean for group z is defined as q 2 fz (q) = |h1 (q; z) · h3 (q; z)| + (h2 (q; z) + h4 (q; z)) , and the response is generated according to y = fz (q) + ϵ. In the following, we present the complete set of experimental results, including all signal-to-noise ratio (SNR) settings, as well as group-level results for SNR = 5 in the low-dimensional settings. The results are organized from low-dimensional settings (Scenarios 1 and 2, and Extra Scenarios 1–4) to high-dimensional settings (Scenarios 3 and 4). In most cases, the proposed two-stage transfer learning method achieves the best performance as the sample size increases.

37

Transfer Learning in Nonparametric Regression with Deep ReLU Networks Table 6. Average MSE for Scenario 1 with different SNR levels over 50 independent trials. The best performance in terms of lower MSE within each SNR group is bolded. SNR

Model / Sample size

1000

5000

10000

30000

50000

10

Pooled (NN) 2-Stage (NN) Separate (NN) Top-FT (NN) Pool-w-L (NN) Pooled (RF) 2-Stage (RF) Separate (RF) Pool-w-L (RF)

0.0352 (0.0356) 0.0175 (0.0064) 0.8039 (0.0735) 0.0315 (0.0249) 0.0177 (0.0037) 0.0957 (0.0109) 0.0675 (0.0085) 0.1300 (0.0198) 0.0957 (0.0109)

0.0214 (0.0016) 0.0067 (0.0012) 0.0673 (0.0130) 0.0188 (0.0022) 0.0069 (0.0011) 0.0721 (0.0037) 0.0410 (0.0026) 0.0785 (0.0046) 0.0707 (0.0035)

0.0197 (0.0011) 0.0045 (0.0007) 0.0079 (0.0031) 0.0154 (0.0017) 0.0055 (0.0016) 0.0690 (0.0028) 0.0338 (0.0016) 0.0661 (0.0026) 0.0674 (0.0028)

0.0209 (0.0056) 0.0025 (0.0003) 0.0041 (0.0005) 0.0093 (0.0020) 0.0069 (0.0061) 0.0675 (0.0019) 0.0255 (0.0011) 0.0548 (0.0015) 0.0658 (0.0019)

0.0184 (0.0016) 0.0020 (0.0002) 0.0034 (0.0005) 0.0074 (0.0015) 0.0036 (0.0016) 0.0691 (0.0012) 0.0242 (0.0008) 0.0518 (0.0011) 0.0675 (0.0012)

5

Pooled (NN) 2-Stage (NN) Separate (NN) Top-FT (NN) Pool-w-L (NN) Pooled (RF) 2-Stage (RF) Separate (RF) Pool-w-L (RF)

0.0364 (0.0355) 0.0222 (0.0078) 0.8183 (0.0788) 0.0331 (0.0249) 0.0219 (0.0044) 0.0952 (0.0105) 0.0680 (0.0085) 0.1318 (0.0195) 0.0945 (0.0106)

0.0228 (0.0017) 0.0093 (0.0016) 0.0664 (0.0116) 0.0212 (0.0018) 0.0103 (0.0021) 0.0708 (0.0039) 0.0410 (0.0028) 0.0775 (0.0047) 0.0692 (0.0038)

0.0213 (0.0022) 0.0064 (0.0009) 0.0097 (0.0035) 0.0177 (0.0018) 0.0074 (0.0017) 0.0673 (0.0028) 0.0337 (0.0016) 0.0650 (0.0028) 0.0656 (0.0028)

0.0234 (0.0068) 0.0034 (0.0005) 0.0060 (0.0007) 0.0114 (0.0019) 0.0103 (0.0097) 0.0655 (0.0018) 0.0252 (0.0011) 0.0532 (0.0015) 0.0637 (0.0018)

0.0193 (0.0029) 0.0027 (0.0004) 0.0049 (0.0008) 0.0092 (0.0017) 0.0048 (0.0023) 0.0669 (0.0012) 0.0236 (0.0008) 0.0500 (0.0011) 0.0652 (0.0012)

2

Pooled (NN) 2-Stage (NN) Separate (NN) Top-FT (NN) Pool-w-L (NN) Pooled (RF) 2-Stage (RF) Separate (RF) Pool-w-L (RF)

0.0395 (0.0350) 0.0346 (0.0100) 0.8097 (0.0875) 0.0370 (0.0244) 0.0306 (0.0081) 0.0970 (0.0106) 0.0738 (0.0089) 0.1401 (0.0216) 0.0960 (0.0105)

0.0248 (0.0018) 0.0151 (0.0032) 0.0685 (0.0112) 0.0239 (0.0017) 0.0151 (0.0024) 0.0703 (0.0041) 0.0446 (0.0028) 0.0793 (0.0050) 0.0683 (0.0038)

0.0243 (0.0027) 0.0110 (0.0019) 0.0115 (0.0011) 0.0212 (0.0014) 0.0120 (0.0026) 0.0658 (0.0032) 0.0366 (0.0021) 0.0655 (0.0033) 0.0636 (0.0031)

0.0271 (0.0094) 0.0059 (0.0010) 0.0087 (0.0008) 0.0155 (0.0018) 0.0146 (0.0105) 0.0625 (0.0019) 0.0267 (0.0013) 0.0520 (0.0015) 0.0603 (0.0018)

0.0212 (0.0046) 0.0045 (0.0008) 0.0078 (0.0015) 0.0129 (0.0019) 0.0071 (0.0030) 0.0631 (0.0012) 0.0243 (0.0010) 0.0480 (0.0011) 0.0611 (0.0012)

Table 7. Average per-group MSE for Scenario 1 over 50 independent trials at an SNR of 5 using neural networks. The best performance in terms of lower MSE is bolded. n

1000

5000

10000

30000

50000

Group 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5

Pooled 0.0367 (0.0296) 0.0441 (0.0444) 0.0359 (0.0325) 0.0373 (0.0426) 0.0326 (0.0351) 0.0302 (0.0058) 0.0266 (0.0039) 0.0248 (0.0036) 0.0219 (0.0027) 0.0191 (0.0026) 0.0285 (0.0039) 0.0254 (0.0036) 0.0231 (0.0028) 0.0206 (0.0031) 0.0176 (0.0026) 0.0308 (0.0070) 0.0280 (0.0069) 0.0251 (0.0071) 0.0224 (0.0073) 0.0197 (0.0068) 0.0263 (0.0039) 0.0235 (0.0036) 0.0209 (0.0033) 0.0189 (0.0029) 0.0156 (0.0032)

2-Stage 0.0435 (0.0341) 0.0302 (0.0302) 0.0222 (0.0117) 0.0205 (0.0151) 0.0157 (0.0051) 0.0164 (0.0078) 0.0107 (0.0031) 0.0088 (0.0029) 0.0087 (0.0024) 0.0082 (0.0020) 0.0090 (0.0028) 0.0069 (0.0012) 0.0067 (0.0019) 0.0061 (0.0016) 0.0057 (0.0010) 0.0045 (0.0013) 0.0040 (0.0012) 0.0036 (0.0008) 0.0031 (0.0007) 0.0032 (0.0007) 0.0040 (0.0012) 0.0030 (0.0006) 0.0030 (0.0008) 0.0026 (0.0006) 0.0024 (0.0005)

38

Separate 0.8045 (0.3100) 0.8241 (0.2425) 0.8402 (0.2157) 0.8012 (0.1283) 0.8198 (0.1723) 0.8092 (0.1464) 0.0209 (0.0421) 0.0151 (0.0212) 0.0104 (0.0016) 0.0094 (0.0016) 0.0224 (0.0510) 0.0104 (0.0018) 0.0089 (0.0015) 0.0083 (0.0012) 0.0085 (0.0014) 0.0091 (0.0014) 0.0075 (0.0012) 0.0062 (0.0016) 0.0053 (0.0013) 0.0053 (0.0014) 0.0089 (0.0015) 0.0061 (0.0014) 0.0053 (0.0020) 0.0042 (0.0017) 0.0039 (0.0013)

Top-FT 0.0426 (0.0378) 0.0417 (0.0380) 0.0342 (0.0303) 0.0328 (0.0272) 0.0272 (0.0198) 0.0294 (0.0064) 0.0259 (0.0043) 0.0232 (0.0043) 0.0205 (0.0030) 0.0170 (0.0027) 0.0261 (0.0041) 0.0214 (0.0038) 0.0184 (0.0036) 0.0167 (0.0030) 0.0150 (0.0027) 0.0206 (0.0053) 0.0154 (0.0037) 0.0124 (0.0033) 0.0105 (0.0031) 0.0079 (0.0023) 0.0187 (0.0044) 0.0124 (0.0037) 0.0096 (0.0029) 0.0084 (0.0025) 0.0063 (0.0020)

Pool-w-L 0.0353 (0.0154) 0.0283 (0.0100) 0.0206 (0.0077) 0.0208 (0.0075) 0.0180 (0.0059) 0.0171 (0.0059) 0.0118 (0.0033) 0.0105 (0.0029) 0.0094 (0.0023) 0.0090 (0.0027) 0.0103 (0.0032) 0.0083 (0.0022) 0.0076 (0.0023) 0.0069 (0.0019) 0.0066 (0.0023) 0.0122 (0.0094) 0.0109 (0.0107) 0.0107 (0.0104) 0.0099 (0.0110) 0.0097 (0.0088) 0.0060 (0.0026) 0.0055 (0.0032) 0.0048 (0.0029) 0.0045 (0.0023) 0.0044 (0.0025)

Transfer Learning in Nonparametric Regression with Deep ReLU Networks Table 8. Average MSE for Scenario 2 with different SNR levels over 50 independent trials. The best performance in terms of lower MSE within each SNR group is bolded. SNR

Model / Sample size

1000

5000

10000

30000

50000

10

Pooled (NN) 2-Stage (NN) Separate (NN) Top-FT (NN) Pool-w-L (NN) Pooled (RF) 2-Stage (RF) Separate (RF) Pool-w-L (RF)

0.0592 (0.0098) 0.0537 (0.0097) 0.6172 (0.0661) 0.0579 (0.0103) 0.0817 (0.0191) 0.1841 (0.0239) 0.1339 (0.0198) 0.2683 (0.0426) 0.1851 (0.0240)

0.0358 (0.0049) 0.0220 (0.0020) 0.0759 (0.0129) 0.0303 (0.0027) 0.0271 (0.0026) 0.1249 (0.0103) 0.0770 (0.0077) 0.1566 (0.0124) 0.1241 (0.0105)

0.0331 (0.0039) 0.0169 (0.0011) 0.0273 (0.0053) 0.0252 (0.0027) 0.0218 (0.0039) 0.1161 (0.0066) 0.0642 (0.0047) 0.1315 (0.0067) 0.1152 (0.0066)

0.0355 (0.0132) 0.0109 (0.0010) 0.0160 (0.0014) 0.0166 (0.0018) 0.0190 (0.0068) 0.1112 (0.0031) 0.0504 (0.0020) 0.1045 (0.0026) 0.1106 (0.0031)

0.0263 (0.0026) 0.0083 (0.0005) 0.0140 (0.0015) 0.0135 (0.0011) 0.0112 (0.0020) 0.1134 (0.0033) 0.0475 (0.0019) 0.0978 (0.0024) 0.1129 (0.0033)

5

Pooled (NN) 2-Stage (NN) Separate (NN) Top-FT (NN) Pool-w-L (NN) Pooled (RF) 2-Stage (RF) Separate (RF) Pool-w-L (RF)

0.0779 (0.0146) 0.0774 (0.0162) 0.6209 (0.0756) 0.0782 (0.0155) 0.1094 (0.0195) 0.1828 (0.0245) 0.1366 (0.0205) 0.2722 (0.0436) 0.1835 (0.0245)

0.0397 (0.0051) 0.0275 (0.0025) 0.0884 (0.0138) 0.0352 (0.0026) 0.0352 (0.0071) 0.1237 (0.0100) 0.0796 (0.0078) 0.1570 (0.0129) 0.1226 (0.0100)

0.0351 (0.0033) 0.0209 (0.0015) 0.0375 (0.0076) 0.0292 (0.0026) 0.0273 (0.0054) 0.1143 (0.0062) 0.0658 (0.0045) 0.1312 (0.0067) 0.1133 (0.0061)

0.0367 (0.0105) 0.0144 (0.0011) 0.0200 (0.0011) 0.0198 (0.0021) 0.0253 (0.0121) 0.1087 (0.0032) 0.0512 (0.0021) 0.1031 (0.0027) 0.1078 (0.0032)

0.0277 (0.0021) 0.0109 (0.0008) 0.0175 (0.0019) 0.0164 (0.0015) 0.0151 (0.0021) 0.1103 (0.0032) 0.0476 (0.0019) 0.0957 (0.0024) 0.1097 (0.0032)

2

Pooled (NN) 2-Stage (NN) Separate (NN) Top-FT (NN) Pool-w-L (NN) Pooled (RF) 2-Stage (RF) Separate (RF) Pool-w-L (RF)

0.1080 (0.0168) 0.1187 (0.0214) 0.6185 (0.0803) 0.1101 (0.0169) 0.1423 (0.0272) 0.1878 (0.0260) 0.1508 (0.0225) 0.2912 (0.0448) 0.1883 (0.0261)

0.0513 (0.0075) 0.0433 (0.0059) 0.1192 (0.0148) 0.0463 (0.0045) 0.0565 (0.0100) 0.1244 (0.0107) 0.0892 (0.0085) 0.1625 (0.0138) 0.1228 (0.0104)

0.0420 (0.0056) 0.0298 (0.0030) 0.0615 (0.0108) 0.0369 (0.0025) 0.0395 (0.0082) 0.1131 (0.0061) 0.0729 (0.0046) 0.1346 (0.0071) 0.1116 (0.0061)

0.0460 (0.0172) 0.0202 (0.0018) 0.0307 (0.0038) 0.0254 (0.0021) 0.0291 (0.0111) 0.1046 (0.0030) 0.0548 (0.0021) 0.1033 (0.0031) 0.1034 (0.0030)

0.0342 (0.0076) 0.0165 (0.0014) 0.0249 (0.0028) 0.0217 (0.0017) 0.0209 (0.0040) 0.1047 (0.0029) 0.0496 (0.0019) 0.0940 (0.0022) 0.1037 (0.0029)

Table 9. Average per-group MSE for Scenario 2 over 50 independent trials at an SNR of 5 using neural networks. The best performance in terms of lower MSE is bolded. n

1000

5000

10000

30000

50000

Group 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5

Pooled 0.0768 (0.0401) 0.0804 (0.0265) 0.0861 (0.0384) 0.0811 (0.0211) 0.0698 (0.0201) 0.0467 (0.0128) 0.0450 (0.0087) 0.0421 (0.0083) 0.0393 (0.0070) 0.0350 (0.0054) 0.0424 (0.0089) 0.0406 (0.0064) 0.0374 (0.0061) 0.0340 (0.0038) 0.0309 (0.0048) 0.0441 (0.0133) 0.0416 (0.0119) 0.0387 (0.0107) 0.0352 (0.0108) 0.0331 (0.0105) 0.0359 (0.0047) 0.0328 (0.0044) 0.0293 (0.0030) 0.0265 (0.0035) 0.0242 (0.0033)

2-Stage 0.0852 (0.0420) 0.0843 (0.0322) 0.0862 (0.0416) 0.0770 (0.0212) 0.0680 (0.0194) 0.0398 (0.0123) 0.0302 (0.0061) 0.0281 (0.0054) 0.0261 (0.0051) 0.0246 (0.0041) 0.0275 (0.0068) 0.0235 (0.0049) 0.0212 (0.0031) 0.0199 (0.0031) 0.0192 (0.0027) 0.0182 (0.0032) 0.0163 (0.0027) 0.0145 (0.0022) 0.0139 (0.0019) 0.0131 (0.0016) 0.0147 (0.0028) 0.0126 (0.0016) 0.0110 (0.0012) 0.0104 (0.0012) 0.0097 (0.0011)

39

Separate 0.6384 (0.2559) 0.6267 (0.1901) 0.6510 (0.1942) 0.6245 (0.1700) 0.5940 (0.1198) 0.6086 (0.1238) 0.0865 (0.0717) 0.0523 (0.0174) 0.0461 (0.0100) 0.0389 (0.0077) 0.1050 (0.1076) 0.0456 (0.0092) 0.0346 (0.0054) 0.0298 (0.0059) 0.0285 (0.0062) 0.0353 (0.0068) 0.0231 (0.0035) 0.0198 (0.0031) 0.0182 (0.0034) 0.0172 (0.0024) 0.0284 (0.0057) 0.0206 (0.0045) 0.0182 (0.0049) 0.0156 (0.0027) 0.0152 (0.0034)

Top-FT 0.0820 (0.0420) 0.0810 (0.0262) 0.0871 (0.0407) 0.0807 (0.0224) 0.0691 (0.0197) 0.0435 (0.0103) 0.0408 (0.0069) 0.0379 (0.0075) 0.0344 (0.0058) 0.0304 (0.0036) 0.0373 (0.0079) 0.0349 (0.0064) 0.0308 (0.0060) 0.0277 (0.0044) 0.0255 (0.0038) 0.0301 (0.0068) 0.0219 (0.0040) 0.0208 (0.0035) 0.0185 (0.0033) 0.0174 (0.0023) 0.0256 (0.0048) 0.0193 (0.0037) 0.0170 (0.0024) 0.0153 (0.0020) 0.0139 (0.0019)

Pool-w-L 0.1501 (0.0883) 0.1269 (0.0490) 0.1133 (0.0402) 0.1038 (0.0327) 0.0957 (0.0258) 0.0507 (0.0131) 0.0418 (0.0102) 0.0368 (0.0089) 0.0336 (0.0103) 0.0298 (0.0072) 0.0382 (0.0101) 0.0311 (0.0068) 0.0273 (0.0057) 0.0251 (0.0066) 0.0253 (0.0063) 0.0322 (0.0170) 0.0280 (0.0132) 0.0247 (0.0114) 0.0236 (0.0141) 0.0245 (0.0128) 0.0218 (0.0043) 0.0171 (0.0036) 0.0152 (0.0030) 0.0142 (0.0032) 0.0136 (0.0029)

Transfer Learning in Nonparametric Regression with Deep ReLU Networks Table 10. Average MSE for Extra Scenario 1 with different SNR levels over 50 independent trials. The best performance in terms of lower MSE within each SNR group is bolded. SNR

Model / Sample size

1000

5000

10000

30000

50000

10

Pooled (NN) 2-Stage (NN) Separate (NN) Top-FT (NN) Pool-w-L (NN) Pooled (RF) 2-Stage (RF) Separate (RF) Pool-w-L (RF)

0.0146 (0.0021) 0.0109 (0.0016) 0.4149 (0.0643) 0.0114 (0.0020) 0.0115 (0.0019) 0.0405 (0.0056) 0.0279 (0.0039) 0.0640 (0.0092) 0.0402 (0.0055)

0.0106 (0.0009) 0.0051 (0.0007) 0.0367 (0.0080) 0.0060 (0.0006) 0.0059 (0.0008) 0.0278 (0.0017) 0.0157 (0.0010) 0.0326 (0.0019) 0.0271 (0.0017)

0.0096 (0.0009) 0.0035 (0.0003) 0.0059 (0.0005) 0.0044 (0.0003) 0.0042 (0.0008) 0.0255 (0.0013) 0.0124 (0.0007) 0.0261 (0.0012) 0.0247 (0.0012)

0.0100 (0.0024) 0.0021 (0.0002) 0.0032 (0.0003) 0.0030 (0.0002) 0.0043 (0.0026) 0.0247 (0.0006) 0.0093 (0.0004) 0.0201 (0.0006) 0.0240 (0.0006)

0.0085 (0.0007) 0.0016 (0.0001) 0.0026 (0.0003) 0.0026 (0.0002) 0.0019 (0.0006) 0.0254 (0.0005) 0.0086 (0.0003) 0.0188 (0.0004) 0.0247 (0.0005)

5

Pooled (NN) 2-Stage (NN) Separate (NN) Top-FT (NN) Pool-w-L (NN) Pooled (RF) 2-Stage (RF) Separate (RF) Pool-w-L (RF)

0.0170 (0.0027) 0.0140 (0.0028) 0.4180 (0.0605) 0.0140 (0.0026) 0.0148 (0.0028) 0.0412 (0.0060) 0.0294 (0.0043) 0.0653 (0.0096) 0.0408 (0.0059)

0.0121 (0.0011) 0.0068 (0.0008) 0.0365 (0.0099) 0.0074 (0.0007) 0.0078 (0.0011) 0.0277 (0.0017) 0.0164 (0.0011) 0.0330 (0.0021) 0.0269 (0.0016)

0.0106 (0.0011) 0.0047 (0.0005) 0.0079 (0.0020) 0.0054 (0.0005) 0.0059 (0.0012) 0.0252 (0.0013) 0.0130 (0.0008) 0.0263 (0.0013) 0.0243 (0.0013)

0.0119 (0.0039) 0.0028 (0.0003) 0.0045 (0.0006) 0.0035 (0.0003) 0.0050 (0.0023) 0.0241 (0.0006) 0.0097 (0.0004) 0.0200 (0.0006) 0.0233 (0.0006)

0.0089 (0.0011) 0.0022 (0.0002) 0.0037 (0.0005) 0.0030 (0.0002) 0.0026 (0.0007) 0.0246 (0.0005) 0.0088 (0.0004) 0.0185 (0.0004) 0.0239 (0.0005)

2

Pooled (NN) 2-Stage (NN) Separate (NN) Top-FT (NN) Pool-w-L (NN) Pooled (RF) 2-Stage (RF) Separate (RF) Pool-w-L (RF)

0.0216 (0.0044) 0.0220 (0.0060) 0.4198 (0.0632) 0.0197 (0.0042) 0.0207 (0.0036) 0.0444 (0.0071) 0.0350 (0.0055) 0.0696 (0.0104) 0.0440 (0.0068)

0.0143 (0.0017) 0.0101 (0.0017) 0.0411 (0.0119) 0.0097 (0.0009) 0.0108 (0.0017) 0.0290 (0.0018) 0.0196 (0.0014) 0.0353 (0.0020) 0.0279 (0.0017)

0.0128 (0.0013) 0.0074 (0.0011) 0.0113 (0.0019) 0.0074 (0.0007) 0.0084 (0.0015) 0.0259 (0.0013) 0.0156 (0.0009) 0.0280 (0.0014) 0.0248 (0.0013)

0.0122 (0.0030) 0.0045 (0.0006) 0.0070 (0.0010) 0.0047 (0.0004) 0.0069 (0.0028) 0.0235 (0.0007) 0.0112 (0.0005) 0.0210 (0.0007) 0.0226 (0.0007)

0.0098 (0.0010) 0.0037 (0.0004) 0.0057 (0.0007) 0.0040 (0.0003) 0.0040 (0.0009) 0.0236 (0.0005) 0.0099 (0.0004) 0.0189 (0.0005) 0.0226 (0.0005)

Table 11. Average per-group MSE for Extra Scenario 1 over 50 independent trials at an SNR of 5 using neural networks. The best performance in terms of lower MSE is bolded. n

1000

5000

10000

30000

50000

Group 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5

Pooled 0.0254 (0.0121) 0.0271 (0.0116) 0.0152 (0.0051) 0.0110 (0.0035) 0.0170 (0.0035) 0.0213 (0.0064) 0.0180 (0.0046) 0.0101 (0.0026) 0.0069 (0.0010) 0.0131 (0.0049) 0.0207 (0.0053) 0.0166 (0.0040) 0.0088 (0.0027) 0.0057 (0.0009) 0.0112 (0.0049) 0.0221 (0.0093) 0.0181 (0.0079) 0.0101 (0.0059) 0.0071 (0.0036) 0.0124 (0.0115) 0.0184 (0.0050) 0.0146 (0.0046) 0.0067 (0.0031) 0.0042 (0.0014) 0.0098 (0.0040)

2-Stage 0.0224 (0.0112) 0.0199 (0.0085) 0.0136 (0.0050) 0.0118 (0.0036) 0.0117 (0.0030) 0.0112 (0.0034) 0.0085 (0.0024) 0.0075 (0.0017) 0.0064 (0.0012) 0.0053 (0.0014) 0.0067 (0.0019) 0.0056 (0.0011) 0.0052 (0.0011) 0.0045 (0.0007) 0.0038 (0.0009) 0.0039 (0.0007) 0.0038 (0.0006) 0.0034 (0.0007) 0.0028 (0.0005) 0.0020 (0.0005) 0.0035 (0.0006) 0.0030 (0.0005) 0.0025 (0.0004) 0.0021 (0.0003) 0.0016 (0.0003)

40

Separate 0.4225 (0.2332) 0.5008 (0.1793) 0.4046 (0.1851) 0.3859 (0.0916) 0.4167 (0.0992) 0.4110 (0.1464) 0.0135 (0.0027) 0.0107 (0.0018) 0.0085 (0.0011) 0.0074 (0.0010) 0.0178 (0.0281) 0.0093 (0.0014) 0.0083 (0.0013) 0.0066 (0.0011) 0.0060 (0.0012) 0.0085 (0.0014) 0.0058 (0.0009) 0.0050 (0.0010) 0.0038 (0.0010) 0.0034 (0.0013) 0.0078 (0.0012) 0.0048 (0.0010) 0.0039 (0.0009) 0.0033 (0.0011) 0.0025 (0.0008)

Top-FT 0.0225 (0.0106) 0.0227 (0.0096) 0.0128 (0.0044) 0.0108 (0.0036) 0.0119 (0.0025) 0.0157 (0.0042) 0.0114 (0.0026) 0.0073 (0.0011) 0.0060 (0.0008) 0.0053 (0.0010) 0.0121 (0.0029) 0.0088 (0.0020) 0.0053 (0.0007) 0.0044 (0.0006) 0.0036 (0.0005) 0.0076 (0.0017) 0.0055 (0.0009) 0.0038 (0.0004) 0.0030 (0.0003) 0.0021 (0.0003) 0.0064 (0.0012) 0.0050 (0.0009) 0.0033 (0.0003) 0.0027 (0.0003) 0.0017 (0.0002)

Pool-w-L 0.0253 (0.0109) 0.0208 (0.0088) 0.0151 (0.0056) 0.0123 (0.0037) 0.0120 (0.0035) 0.0131 (0.0032) 0.0100 (0.0022) 0.0088 (0.0019) 0.0070 (0.0014) 0.0060 (0.0011) 0.0094 (0.0028) 0.0074 (0.0019) 0.0065 (0.0015) 0.0052 (0.0012) 0.0047 (0.0016) 0.0073 (0.0032) 0.0061 (0.0028) 0.0052 (0.0024) 0.0047 (0.0025) 0.0043 (0.0029) 0.0040 (0.0010) 0.0033 (0.0011) 0.0029 (0.0010) 0.0023 (0.0008) 0.0021 (0.0008)

Transfer Learning in Nonparametric Regression with Deep ReLU Networks Table 12. Average MSE for Extra Scenario 2 with different SNR levels over 50 independent trials. The best performance in terms of lower MSE within each SNR group is bolded. SNR

Model / Sample size

1000

5000

10000

30000

50000

10

Pooled (NN) 2-Stage (NN) Separate (NN) Top-FT (NN) Pool-w-L (NN) Pooled (RF) 2-Stage (RF) Separate (RF) Pool-w-L (RF)

0.0223 (0.0040) 0.0163 (0.0036) 0.4959 (0.0843) 0.0181 (0.0038) 0.0157 (0.0026) 0.0518 (0.0078) 0.0358 (0.0055) 0.0773 (0.0103) 0.0510 (0.0076)

0.0163 (0.0014) 0.0065 (0.0008) 0.0482 (0.0115) 0.0082 (0.0011) 0.0076 (0.0010) 0.0355 (0.0023) 0.0199 (0.0014) 0.0398 (0.0023) 0.0338 (0.0022)

0.0153 (0.0016) 0.0045 (0.0004) 0.0079 (0.0007) 0.0065 (0.0008) 0.0054 (0.0009) 0.0331 (0.0015) 0.0159 (0.0008) 0.0316 (0.0014) 0.0308 (0.0014)

0.0147 (0.0021) 0.0028 (0.0003) 0.0043 (0.0004) 0.0039 (0.0004) 0.0059 (0.0046) 0.0318 (0.0010) 0.0117 (0.0006) 0.0240 (0.0007) 0.0291 (0.0009)

0.0133 (0.0011) 0.0020 (0.0002) 0.0037 (0.0004) 0.0031 (0.0005) 0.0024 (0.0006) 0.0325 (0.0007) 0.0108 (0.0004) 0.0222 (0.0005) 0.0295 (0.0006)

5

Pooled (NN) 2-Stage (NN) Separate (NN) Top-FT (NN) Pool-w-L (NN) Pooled (RF) 2-Stage (RF) Separate (RF) Pool-w-L (RF)

0.0248 (0.0049) 0.0200 (0.0044) 0.4885 (0.0793) 0.0212 (0.0046) 0.0200 (0.0038) 0.0524 (0.0076) 0.0376 (0.0058) 0.0793 (0.0115) 0.0513 (0.0076)

0.0180 (0.0018) 0.0088 (0.0010) 0.0504 (0.0136) 0.0103 (0.0013) 0.0098 (0.0014) 0.0357 (0.0021) 0.0209 (0.0014) 0.0404 (0.0024) 0.0338 (0.0020)

0.0159 (0.0013) 0.0060 (0.0006) 0.0099 (0.0007) 0.0077 (0.0008) 0.0078 (0.0019) 0.0329 (0.0016) 0.0167 (0.0009) 0.0320 (0.0014) 0.0306 (0.0015)

0.0157 (0.0027) 0.0037 (0.0003) 0.0059 (0.0006) 0.0048 (0.0005) 0.0065 (0.0029) 0.0312 (0.0009) 0.0122 (0.0006) 0.0239 (0.0007) 0.0284 (0.0009)

0.0141 (0.0012) 0.0029 (0.0003) 0.0049 (0.0006) 0.0040 (0.0004) 0.0039 (0.0013) 0.0317 (0.0007) 0.0110 (0.0005) 0.0219 (0.0005) 0.0287 (0.0007)

2

Pooled (NN) 2-Stage (NN) Separate (NN) Top-FT (NN) Pool-w-L (NN) Pooled (RF) 2-Stage (RF) Separate (RF) Pool-w-L (RF)

0.0306 (0.0067) 0.0296 (0.0067) 0.5007 (0.0691) 0.0285 (0.0065) 0.0281 (0.0058) 0.0563 (0.0088) 0.0439 (0.0075) 0.0851 (0.0128) 0.0554 (0.0088)

0.0208 (0.0023) 0.0129 (0.0016) 0.0556 (0.0130) 0.0138 (0.0015) 0.0140 (0.0017) 0.0374 (0.0022) 0.0249 (0.0017) 0.0434 (0.0029) 0.0353 (0.0022)

0.0188 (0.0020) 0.0096 (0.0013) 0.0147 (0.0031) 0.0106 (0.0013) 0.0111 (0.0016) 0.0338 (0.0017) 0.0199 (0.0012) 0.0344 (0.0017) 0.0314 (0.0016)

0.0191 (0.0054) 0.0055 (0.0006) 0.0091 (0.0008) 0.0065 (0.0005) 0.0095 (0.0042) 0.0307 (0.0009) 0.0141 (0.0006) 0.0252 (0.0009) 0.0279 (0.0009)

0.0152 (0.0017) 0.0045 (0.0005) 0.0076 (0.0009) 0.0055 (0.0007) 0.0060 (0.0024) 0.0306 (0.0008) 0.0123 (0.0005) 0.0226 (0.0006) 0.0277 (0.0007)

Table 13. Average per-group MSE for Extra Scenario 2 over 50 independent trials at an SNR of 5 using neural networks. The best performance in terms of lower MSE is bolded. n

1000

5000

10000

30000

50000

Group 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5

Pooled 0.0463 (0.0228) 0.0631 (0.0245) 0.0168 (0.0058) 0.0141 (0.0044) 0.0176 (0.0040) 0.0393 (0.0091) 0.0465 (0.0112) 0.0100 (0.0022) 0.0097 (0.0022) 0.0136 (0.0037) 0.0396 (0.0075) 0.0428 (0.0068) 0.0084 (0.0020) 0.0077 (0.0020) 0.0114 (0.0032) 0.0380 (0.0077) 0.0418 (0.0122) 0.0082 (0.0040) 0.0079 (0.0038) 0.0115 (0.0052) 0.0366 (0.0051) 0.0398 (0.0075) 0.0062 (0.0021) 0.0064 (0.0020) 0.0101 (0.0029)

2-Stage 0.0426 (0.0209) 0.0372 (0.0186) 0.0183 (0.0089) 0.0144 (0.0044) 0.0137 (0.0037) 0.0198 (0.0088) 0.0127 (0.0034) 0.0086 (0.0022) 0.0077 (0.0018) 0.0059 (0.0011) 0.0111 (0.0028) 0.0085 (0.0021) 0.0063 (0.0012) 0.0053 (0.0009) 0.0043 (0.0008) 0.0067 (0.0010) 0.0055 (0.0008) 0.0038 (0.0005) 0.0035 (0.0006) 0.0025 (0.0004) 0.0053 (0.0008) 0.0041 (0.0008) 0.0031 (0.0006) 0.0028 (0.0006) 0.0019 (0.0003)

41

Separate 0.5862 (0.3274) 0.6185 (0.2614) 0.4944 (0.1956) 0.4476 (0.1183) 0.4433 (0.1004) 0.5857 (0.2020) 0.0201 (0.0038) 0.0131 (0.0023) 0.0101 (0.0017) 0.0084 (0.0013) 0.0217 (0.0049) 0.0146 (0.0023) 0.0102 (0.0015) 0.0081 (0.0012) 0.0069 (0.0011) 0.0127 (0.0022) 0.0084 (0.0012) 0.0069 (0.0017) 0.0049 (0.0009) 0.0037 (0.0007) 0.0118 (0.0029) 0.0069 (0.0019) 0.0052 (0.0011) 0.0040 (0.0010) 0.0034 (0.0014)

Top-FT 0.0436 (0.0225) 0.0505 (0.0221) 0.0162 (0.0059) 0.0134 (0.0037) 0.0135 (0.0039) 0.0259 (0.0095) 0.0216 (0.0057) 0.0082 (0.0012) 0.0069 (0.0010) 0.0066 (0.0012) 0.0203 (0.0054) 0.0159 (0.0029) 0.0065 (0.0010) 0.0051 (0.0006) 0.0047 (0.0008) 0.0134 (0.0031) 0.0094 (0.0019) 0.0044 (0.0006) 0.0034 (0.0004) 0.0026 (0.0004) 0.0108 (0.0022) 0.0083 (0.0020) 0.0035 (0.0005) 0.0029 (0.0003) 0.0021 (0.0004)

Pool-w-L 0.0459 (0.0173) 0.0373 (0.0180) 0.0175 (0.0059) 0.0154 (0.0050) 0.0124 (0.0034) 0.0205 (0.0049) 0.0160 (0.0032) 0.0099 (0.0023) 0.0081 (0.0016) 0.0064 (0.0013) 0.0146 (0.0050) 0.0121 (0.0031) 0.0082 (0.0021) 0.0067 (0.0023) 0.0054 (0.0019) 0.0105 (0.0041) 0.0084 (0.0038) 0.0065 (0.0031) 0.0056 (0.0028) 0.0056 (0.0035) 0.0067 (0.0024) 0.0054 (0.0021) 0.0043 (0.0014) 0.0035 (0.0013) 0.0027 (0.0013)

Transfer Learning in Nonparametric Regression with Deep ReLU Networks Table 14. Average MSE for Extra Scenario 3 with different SNR levels over 50 independent trials. The best performance in terms of lower MSE within each SNR group is bolded. SNR

Model / Sample size

1000

5000

10000

30000

50000

10

Pooled (NN) 2-Stage (NN) Separate (NN) Top-FT (NN) Pool-w-L (NN) Pooled (RF) 2-Stage (RF) Separate (RF) Pool-w-L (RF)

0.0536 (0.0095) 0.0393 (0.0090) 1.4760 (0.1806) 0.0393 (0.0082) 0.0480 (0.0100) 0.0875 (0.0139) 0.0631 (0.0107) 0.1239 (0.0140) 0.0843 (0.0138)

0.0375 (0.0031) 0.0164 (0.0020) 0.1328 (0.0264) 0.0191 (0.0018) 0.0181 (0.0021) 0.0624 (0.0044) 0.0346 (0.0029) 0.0652 (0.0049) 0.0567 (0.0042)

0.0345 (0.0026) 0.0112 (0.0011) 0.0200 (0.0018) 0.0150 (0.0013) 0.0142 (0.0025) 0.0578 (0.0035) 0.0276 (0.0019) 0.0503 (0.0026) 0.0510 (0.0029)

0.0374 (0.0114) 0.0068 (0.0007) 0.0112 (0.0012) 0.0106 (0.0010) 0.0143 (0.0062) 0.0543 (0.0016) 0.0195 (0.0007) 0.0366 (0.0009) 0.0460 (0.0013)

0.0298 (0.0025) 0.0050 (0.0004) 0.0091 (0.0010) 0.0086 (0.0008) 0.0064 (0.0014) 0.0535 (0.0013) 0.0172 (0.0006) 0.0331 (0.0008) 0.0450 (0.0011)

5

Pooled (NN) 2-Stage (NN) Separate (NN) Top-FT (NN) Pool-w-L (NN) Pooled (RF) 2-Stage (RF) Separate (RF) Pool-w-L (RF)

0.0610 (0.0100) 0.0500 (0.0117) 1.4668 (0.1573) 0.0472 (0.0091) 0.0546 (0.0099) 0.0910 (0.0142) 0.0688 (0.0118) 0.1296 (0.0150) 0.0873 (0.0138)

0.0407 (0.0032) 0.0210 (0.0027) 0.1385 (0.0261) 0.0223 (0.0019) 0.0240 (0.0037) 0.0641 (0.0044) 0.0376 (0.0030) 0.0684 (0.0051) 0.0584 (0.0043)

0.0376 (0.0042) 0.0144 (0.0015) 0.0262 (0.0019) 0.0175 (0.0016) 0.0181 (0.0042) 0.0588 (0.0037) 0.0301 (0.0020) 0.0530 (0.0028) 0.0519 (0.0031)

0.0406 (0.0103) 0.0088 (0.0006) 0.0148 (0.0014) 0.0121 (0.0011) 0.0174 (0.0066) 0.0542 (0.0016) 0.0209 (0.0008) 0.0380 (0.0010) 0.0458 (0.0014)

0.0312 (0.0038) 0.0068 (0.0006) 0.0119 (0.0012) 0.0099 (0.0008) 0.0098 (0.0036) 0.0530 (0.0014) 0.0182 (0.0007) 0.0339 (0.0009) 0.0444 (0.0012)

2

Pooled (NN) 2-Stage (NN) Separate (NN) Top-FT (NN) Pool-w-L (NN) Pooled (RF) 2-Stage (RF) Separate (RF) Pool-w-L (RF)

0.0720 (0.0121) 0.0649 (0.0166) 1.4359 (0.1835) 0.0596 (0.0119) 0.0654 (0.0114) 0.1018 (0.0160) 0.0853 (0.0142) 0.1460 (0.0177) 0.0982 (0.0151)

0.0483 (0.0051) 0.0302 (0.0037) 0.1539 (0.0312) 0.0294 (0.0035) 0.0354 (0.0054) 0.0701 (0.0051) 0.0473 (0.0041) 0.0796 (0.0057) 0.0642 (0.0049)

0.0422 (0.0035) 0.0214 (0.0023) 0.0370 (0.0031) 0.0220 (0.0018) 0.0255 (0.0043) 0.0624 (0.0040) 0.0372 (0.0023) 0.0615 (0.0031) 0.0555 (0.0035)

0.0449 (0.0160) 0.0131 (0.0012) 0.0222 (0.0025) 0.0155 (0.0014) 0.0260 (0.0153) 0.0556 (0.0019) 0.0254 (0.0010) 0.0433 (0.0011) 0.0471 (0.0017)

0.0344 (0.0043) 0.0105 (0.0011) 0.0185 (0.0024) 0.0128 (0.0010) 0.0145 (0.0043) 0.0532 (0.0015) 0.0215 (0.0009) 0.0377 (0.0011) 0.0445 (0.0013)

Table 15. Average per-group MSE for Extra Scenario 3 over 50 independent trials at an SNR of 5 using neural networks. The best performance in terms of lower MSE is bolded. n

1000

5000

10000

30000

50000

Group 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5

Pooled 0.2043 (0.0972) 0.0747 (0.0239) 0.0646 (0.0187) 0.0401 (0.0114) 0.0392 (0.0078) 0.2000 (0.0467) 0.0496 (0.0129) 0.0421 (0.0116) 0.0203 (0.0049) 0.0203 (0.0029) 0.1944 (0.0449) 0.0456 (0.0153) 0.0386 (0.0161) 0.0178 (0.0073) 0.0179 (0.0041) 0.2025 (0.0765) 0.0503 (0.0297) 0.0395 (0.0333) 0.0210 (0.0169) 0.0207 (0.0101) 0.1874 (0.0393) 0.0393 (0.0140) 0.0316 (0.0165) 0.0123 (0.0079) 0.0116 (0.0039)

2-Stage 0.0975 (0.0573) 0.0681 (0.0414) 0.0468 (0.0169) 0.0395 (0.0105) 0.0426 (0.0124) 0.0556 (0.0215) 0.0249 (0.0051) 0.0199 (0.0054) 0.0168 (0.0032) 0.0165 (0.0033) 0.0283 (0.0075) 0.0181 (0.0039) 0.0140 (0.0027) 0.0119 (0.0016) 0.0124 (0.0020) 0.0161 (0.0030) 0.0108 (0.0017) 0.0088 (0.0015) 0.0078 (0.0012) 0.0073 (0.0008) 0.0128 (0.0020) 0.0082 (0.0014) 0.0068 (0.0011) 0.0063 (0.0010) 0.0053 (0.0007)

42

Separate 1.8615 (0.9071) 1.6906 (0.5349) 1.3591 (0.4425) 1.3825 (0.3397) 1.4208 (0.3058) 1.6041 (0.3878) 0.0507 (0.0119) 0.0368 (0.0076) 0.0272 (0.0043) 0.0260 (0.0047) 0.0599 (0.0170) 0.0378 (0.0082) 0.0247 (0.0043) 0.0204 (0.0041) 0.0204 (0.0030) 0.0342 (0.0048) 0.0205 (0.0036) 0.0140 (0.0026) 0.0119 (0.0026) 0.0116 (0.0025) 0.0284 (0.0049) 0.0156 (0.0025) 0.0123 (0.0030) 0.0096 (0.0020) 0.0088 (0.0023)

Top-FT 0.1011 (0.0584) 0.0609 (0.0212) 0.0452 (0.0140) 0.0373 (0.0107) 0.0390 (0.0084) 0.0699 (0.0161) 0.0272 (0.0050) 0.0191 (0.0037) 0.0157 (0.0020) 0.0179 (0.0027) 0.0569 (0.0129) 0.0210 (0.0033) 0.0143 (0.0023) 0.0120 (0.0013) 0.0145 (0.0018) 0.0455 (0.0090) 0.0152 (0.0021) 0.0094 (0.0014) 0.0083 (0.0010) 0.0088 (0.0011) 0.0394 (0.0076) 0.0123 (0.0016) 0.0076 (0.0010) 0.0069 (0.0008) 0.0069 (0.0007)

Pool-w-L 0.1085 (0.0573) 0.0675 (0.0227) 0.0503 (0.0186) 0.0473 (0.0115) 0.0459 (0.0113) 0.0528 (0.0175) 0.0295 (0.0061) 0.0235 (0.0046) 0.0201 (0.0045) 0.0193 (0.0038) 0.0375 (0.0106) 0.0237 (0.0075) 0.0170 (0.0046) 0.0152 (0.0037) 0.0150 (0.0038) 0.0304 (0.0089) 0.0207 (0.0086) 0.0162 (0.0075) 0.0154 (0.0072) 0.0157 (0.0067) 0.0189 (0.0049) 0.0123 (0.0052) 0.0094 (0.0042) 0.0090 (0.0040) 0.0079 (0.0035)

Transfer Learning in Nonparametric Regression with Deep ReLU Networks Table 16. Average MSE for Extra Scenario 4 with different SNR levels over 50 independent trials. The best performance in terms of lower MSE within each SNR group is bolded. SNR

Model / Sample size

1000

5000

10000

30000

50000

10

Pooled (NN) 2-Stage (NN) Separate (NN) Top-FT (NN) Pool-w-L (NN) Pooled (RF) 2-Stage (RF) Separate (RF) Pool-w-L (RF)

0.0159 (0.0026) 0.0144 (0.0029) 0.3787 (0.0438) 0.0145 (0.0025) 0.0192 (0.0049) 0.0323 (0.0039) 0.0253 (0.0034) 0.0501 (0.0064) 0.0322 (0.0041)

0.0089 (0.0009) 0.0058 (0.0007) 0.0354 (0.0068) 0.0066 (0.0007) 0.0068 (0.0008) 0.0217 (0.0013) 0.0135 (0.0009) 0.0285 (0.0019) 0.0214 (0.0013)

0.0076 (0.0010) 0.0042 (0.0005) 0.0077 (0.0006) 0.0052 (0.0006) 0.0054 (0.0015) 0.0193 (0.0009) 0.0105 (0.0006) 0.0229 (0.0011) 0.0190 (0.0009)

0.0075 (0.0017) 0.0025 (0.0002) 0.0043 (0.0004) 0.0035 (0.0004) 0.0052 (0.0032) 0.0179 (0.0006) 0.0079 (0.0003) 0.0174 (0.0005) 0.0176 (0.0006)

0.0060 (0.0008) 0.0019 (0.0001) 0.0037 (0.0004) 0.0028 (0.0003) 0.0027 (0.0009) 0.0179 (0.0005) 0.0073 (0.0003) 0.0159 (0.0004) 0.0176 (0.0005)

5

Pooled (NN) 2-Stage (NN) Separate (NN) Top-FT (NN) Pool-w-L (NN) Pooled (RF) 2-Stage (RF) Separate (RF) Pool-w-L (RF)

0.0202 (0.0047) 0.0194 (0.0049) 0.3739 (0.0474) 0.0191 (0.0048) 0.0244 (0.0047) 0.0332 (0.0042) 0.0270 (0.0037) 0.0514 (0.0066) 0.0331 (0.0043)

0.0101 (0.0011) 0.0076 (0.0008) 0.0381 (0.0071) 0.0080 (0.0008) 0.0092 (0.0014) 0.0219 (0.0012) 0.0145 (0.0010) 0.0292 (0.0021) 0.0216 (0.0012)

0.0087 (0.0010) 0.0056 (0.0006) 0.0101 (0.0009) 0.0063 (0.0007) 0.0067 (0.0009) 0.0195 (0.0009) 0.0114 (0.0007) 0.0235 (0.0011) 0.0191 (0.0009)

0.0089 (0.0029) 0.0034 (0.0003) 0.0057 (0.0006) 0.0041 (0.0004) 0.0061 (0.0029) 0.0177 (0.0006) 0.0083 (0.0003) 0.0176 (0.0005) 0.0173 (0.0006)

0.0066 (0.0009) 0.0026 (0.0002) 0.0048 (0.0006) 0.0034 (0.0003) 0.0032 (0.0006) 0.0176 (0.0005) 0.0075 (0.0003) 0.0159 (0.0004) 0.0173 (0.0005)

2

Pooled (NN) 2-Stage (NN) Separate (NN) Top-FT (NN) Pool-w-L (NN) Pooled (RF) 2-Stage (RF) Separate (RF) Pool-w-L (RF)

0.0266 (0.0051) 0.0280 (0.0053) 0.3740 (0.0515) 0.0259 (0.0052) 0.0302 (0.0045) 0.0362 (0.0053) 0.0321 (0.0049) 0.0571 (0.0074) 0.0359 (0.0055)

0.0132 (0.0016) 0.0114 (0.0015) 0.0448 (0.0090) 0.0112 (0.0012) 0.0140 (0.0025) 0.0234 (0.0014) 0.0178 (0.0012) 0.0324 (0.0022) 0.0230 (0.0014)

0.0110 (0.0016) 0.0081 (0.0010) 0.0153 (0.0016) 0.0083 (0.0009) 0.0099 (0.0016) 0.0206 (0.0010) 0.0142 (0.0008) 0.0256 (0.0012) 0.0201 (0.0011)

0.0103 (0.0033) 0.0050 (0.0004) 0.0087 (0.0009) 0.0055 (0.0005) 0.0093 (0.0056) 0.0178 (0.0006) 0.0098 (0.0003) 0.0189 (0.0006) 0.0174 (0.0006)

0.0079 (0.0016) 0.0040 (0.0003) 0.0070 (0.0007) 0.0046 (0.0004) 0.0052 (0.0014) 0.0173 (0.0005) 0.0086 (0.0003) 0.0169 (0.0004) 0.0169 (0.0005)

Table 17. Average per-group MSE for Extra Scenario 4 over 50 independent trials at an SNR of 5 using neural networks. The best performance in terms of lower MSE is bolded. n

1000

5000

10000

30000

50000

Group 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5

Pooled 0.0294 (0.0171) 0.0337 (0.0126) 0.0190 (0.0071) 0.0151 (0.0046) 0.0173 (0.0048) 0.0160 (0.0045) 0.0193 (0.0046) 0.0083 (0.0019) 0.0065 (0.0014) 0.0090 (0.0022) 0.0146 (0.0039) 0.0178 (0.0045) 0.0070 (0.0021) 0.0053 (0.0010) 0.0076 (0.0022) 0.0141 (0.0042) 0.0164 (0.0061) 0.0066 (0.0039) 0.0058 (0.0033) 0.0086 (0.0054) 0.0122 (0.0025) 0.0149 (0.0034) 0.0049 (0.0020) 0.0034 (0.0011) 0.0057 (0.0025)

2-Stage 0.0317 (0.0186) 0.0298 (0.0134) 0.0190 (0.0074) 0.0154 (0.0049) 0.0161 (0.0043) 0.0148 (0.0046) 0.0120 (0.0024) 0.0074 (0.0016) 0.0060 (0.0010) 0.0058 (0.0011) 0.0102 (0.0028) 0.0090 (0.0016) 0.0053 (0.0009) 0.0046 (0.0008) 0.0044 (0.0010) 0.0068 (0.0011) 0.0053 (0.0008) 0.0033 (0.0004) 0.0027 (0.0003) 0.0025 (0.0003) 0.0050 (0.0008) 0.0040 (0.0006) 0.0026 (0.0003) 0.0021 (0.0004) 0.0020 (0.0003)

43

Separate 0.4555 (0.2110) 0.4091 (0.1315) 0.3641 (0.1010) 0.3518 (0.0806) 0.3653 (0.0886) 0.3776 (0.1010) 0.0251 (0.0064) 0.0152 (0.0033) 0.0116 (0.0027) 0.0094 (0.0019) 0.0240 (0.0064) 0.0163 (0.0033) 0.0100 (0.0013) 0.0076 (0.0011) 0.0068 (0.0012) 0.0129 (0.0025) 0.0088 (0.0011) 0.0061 (0.0009) 0.0043 (0.0011) 0.0038 (0.0009) 0.0110 (0.0021) 0.0071 (0.0009) 0.0050 (0.0012) 0.0036 (0.0009) 0.0036 (0.0013)

Top-FT 0.0294 (0.0169) 0.0306 (0.0126) 0.0183 (0.0065) 0.0149 (0.0046) 0.0158 (0.0044) 0.0147 (0.0046) 0.0150 (0.0032) 0.0071 (0.0014) 0.0058 (0.0009) 0.0062 (0.0010) 0.0124 (0.0032) 0.0121 (0.0025) 0.0053 (0.0009) 0.0043 (0.0005) 0.0050 (0.0007) 0.0093 (0.0016) 0.0082 (0.0014) 0.0035 (0.0005) 0.0028 (0.0003) 0.0030 (0.0005) 0.0084 (0.0014) 0.0070 (0.0013) 0.0027 (0.0003) 0.0022 (0.0003) 0.0024 (0.0004)

Pool-w-L 0.0454 (0.0236) 0.0342 (0.0103) 0.0255 (0.0080) 0.0205 (0.0065) 0.0183 (0.0051) 0.0178 (0.0062) 0.0142 (0.0030) 0.0096 (0.0019) 0.0074 (0.0013) 0.0066 (0.0015) 0.0122 (0.0028) 0.0106 (0.0024) 0.0067 (0.0013) 0.0055 (0.0012) 0.0050 (0.0010) 0.0095 (0.0036) 0.0091 (0.0043) 0.0060 (0.0026) 0.0052 (0.0030) 0.0049 (0.0028) 0.0062 (0.0015) 0.0051 (0.0009) 0.0033 (0.0007) 0.0026 (0.0007) 0.0024 (0.0006)

Transfer Learning in Nonparametric Regression with Deep ReLU Networks Table 18. Average MSE for Scenario 3 (high-dimensional) with different SNR levels over 50 independent trials. The best performance in terms of lower MSE within each SNR group is bolded. Std

Model / Sample size

5000

10000

30000

50000

0.1

Pooled (NN) 2-Stage (NN) Separate (NN) Top-FT (NN) Pool-w-L (NN) ptLasso

3.5772 (0.4119) 3.4045 (0.3687) 18.4700 (1.3653) 3.4864 (0.3970) 3.7586 (0.4649) 19.2066 (1.1992)

2.8049 (0.3413) 2.5317 (0.2908) 16.7562 (1.0588) 2.6559 (0.3170) 2.6799 (0.3126) 18.1132 (1.1836)

2.4122 (0.3941) 1.7826 (0.2254) 7.3540 (0.6464) 2.0301 (0.2903) 2.0207 (0.3539) 17.7140 (0.8321)

1.8712 (0.2405) 1.4377 (0.1909) 5.1880 (0.3722) 1.6499 (0.2149) 1.5020 (0.2305) 17.5210 (0.6882)

1

Pooled (NN) 2-Stage (NN) Separate (NN) Top-FT (NN) Pool-w-L (NN) ptLasso

4.0795 (0.4407) 3.9457 (0.4051) 18.5199 (1.4433) 3.9666 (0.4319) 4.3708 (0.4846) 20.1427 (1.3865)

3.2150 (0.3589) 2.9710 (0.3132) 16.8376 (1.1317) 3.0476 (0.3328) 3.1855 (0.3481) 19.1391 (1.2099)

2.6265 (0.3937) 2.1172 (0.2096) 8.0144 (0.6275) 2.2643 (0.2707) 2.3404 (0.2989) 18.7576 (0.8002)

2.1231 (0.2551) 1.7750 (0.1908) 5.7036 (0.4192) 1.8705 (0.2096) 1.8470 (0.2286) 18.5122 (0.7150)

Table 19. Average MSE for Scenario 4 (high-dimensional) with different SNR levels over 50 independent trials. The best performance in terms of lower MSE within each SNR group is bolded. Std

Model / Sample size

5000

10000

30000

50000

0.1

Pooled (NN) 2-Stage (NN) Separate (NN) Top-FT (NN) Pool-w-L (NN) ptLasso

8.5086 (0.8878) 8.4782 (0.8924) 22.0682 (1.5491) 8.5038 (0.8945) 8.5776 (0.8611) 21.7040 (1.6185)

5.8025 (0.5703) 5.6875 (0.5791) 21.4115 (1.0980) 5.7637 (0.5768) 5.8783 (0.5824) 19.7228 (0.9760)

3.8477 (0.6878) 3.2175 (0.3005) 14.8534 (0.6357) 3.5023 (0.3075) 3.8614 (0.7056) 19.2816 (0.8314)

2.9454 (0.3160) 2.5256 (0.2120) 13.0925 (0.7355) 2.8225 (0.2286) 2.9559 (0.2845) 19.2234 (0.7995)

1

Pooled (NN) 2-Stage (NN) Separate (NN) Top-FT (NN) Pool-w-L (NN) ptLasso

9.1348 (0.9145) 9.0841 (0.9062) 22.3329 (1.5473) 9.1156 (0.9069) 9.2112 (0.9200) 21.7648 (1.6522)

6.4555 (0.6181) 6.3287 (0.5981) 21.5863 (0.9121) 6.4022 (0.5908) 6.4724 (0.6281) 19.7493 (0.9892)

4.3988 (0.5348) 3.8919 (0.3088) 15.1587 (0.5992) 4.1114 (0.3084) 4.5905 (0.6670) 19.3060 (0.8403)

3.5638 (0.2752) 3.1943 (0.2400) 13.5028 (0.7384) 3.4415 (0.2636) 3.5074 (0.3140) 19.2003 (0.8071)

F.2. Detailed Configurations for Numerical Simulation This section presents the detailed configurations used in our numerical simulations. Regarding the neural network setups, the Pooled, Pool-w-L, and Top-FT strategies each employ a single neural network instance, whereas the 2-Stage strategy involves two networks: one trained on the full dataset and another trained separately for each group. Within each scenario, all NN instances share the same network architecture. For Scenarios 1,2 and Extra Scenarios 1–4, which correspond to the low-dimensional settings, each NN is implemented as a four-layer MLP with 64 hidden units per layer, using ReLU activations after the first three layers. For the high-dimensional settings, we adopt a five-layer MLP with 128 hidden units per layer for Scenario 3, and an eight-layer MLP with 256 hidden units per layer for Scenario 4. All neural networks are trained using the Adam optimizer with a learning rate of 10−3 and a batch size of 256. For each setting, data are randomly partitioned using a group-wise stratified split into training (70%), validation (15%), and testing (15%) subsets. Early stopping is applied with a patience of 10 epochs, based on validation performance. For the competing methods, random forests are implemented with 100 tree estimators and a maximum tree depth of 10, while other hyperparameters follow the default settings from the sklearn.ensemble package. For ptLasso, we perform 10-fold cross-validation over α ∈ {0, 0.25, 0.5, 0.75, 1}, keeping other parameters at their default values. F.3. Experiments with Highly Unbalanced Groups To further investigate the robustness of our proposed framework under highly unbalanced groups, we examine the six low-dimensional scenarios previously introduced with unbalanced group assignment proportions of [0.02, 0.25, 0.25, 0.24, 0.24], where the first group represents only 2% of the total sample. Here we focus on the DNN realization and present 44

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

our experimental results at SNR=5 across total sample sizes n ∈ {5000, 10000, 30000, 50000}. Table 20 summarizes the MSE performance of the unbalanced group (Group 1) over 50 independent Monte Carlo trials across the six simulation scenarios. The results demonstrate that as the sample size increases, the proposed two-stage transfer learning method yields increasingly pronounced performance improvements for the extremely low-proportion group, achieving the best performance in most cases. Table 20. Average MSE for the unbalanced group (Group 1) across six low-dimensional scenarios at SNR = 5 over 50 independent trials. The best performance for each sample size is highlighted in bold. Model / Sample size

5000

10000

30000

50000

Scenario 1

Pooled (NN) 2-Stage (NN) Separate (NN) Top-FT (NN) Pool-w-L (NN)

0.0339 (0.0164) 0.0327 (0.0179) 0.6937 (0.4308) 0.0236 (0.0099) 0.0334 (0.0149)

0.0333 (0.0113) 0.0184 (0.0061) 0.7003 (0.3028) 0.0232 (0.0099) 0.0184 (0.0050)

0.0297 (0.0047) 0.0088 (0.0018) 0.0236 (0.0047) 0.0132 (0.0039) 0.0155 (0.0076)

0.0308 (0.0065) 0.0060 (0.0009) 0.0176 (0.0020) 0.0131 (0.0041) 0.0108 (0.0038)

Scenario 2

Pooled (NN) 2-Stage (NN) Separate (NN) Top-FT (NN) Pool-w-L (NN)

0.0488 (0.0211) 0.0498 (0.0210) 0.6019 (0.2203) 0.0514 (0.0257) 0.0918 (0.0427)

0.0456 (0.0144) 0.0415 (0.0199) 0.5966 (0.1470) 0.0410 (0.0121) 0.0594 (0.0255)

0.0437 (0.0124) 0.0250 (0.0074) 0.1108 (0.0858) 0.0358 (0.0073) 0.0413 (0.0223)

0.0375 (0.0086) 0.0185 (0.0050) 0.0571 (0.0147) 0.0312 (0.0067) 0.0286 (0.0069)

Extra Scenario 1

Pooled (NN) 2-Stage (NN) Separate (NN) Top-FT (NN) Pool-w-L (NN)

0.0206 (0.0103) 0.0160 (0.0078) 0.4917 (0.2299) 0.0156 (0.0069) 0.0200 (0.0096)

0.0186 (0.0061) 0.0112 (0.0038) 0.4940 (0.1773) 0.0146 (0.0051) 0.0128 (0.0047)

0.0169 (0.0061) 0.0053 (0.0020) 0.0145 (0.0032) 0.0099 (0.0021) 0.0095 (0.0050)

0.0163 (0.0049) 0.0042 (0.0011) 0.0113 (0.0016) 0.0085 (0.0020) 0.0060 (0.0018)

Extra Scenario 2

Pooled (NN) 2-Stage (NN) Separate (NN) Top-FT (NN) Pool-w-L (NN)

0.0344 (0.0184) 0.0355 (0.0156) 0.6453 (0.3275) 0.0255 (0.0147) 0.0321 (0.0112)

0.0337 (0.0119) 0.0167 (0.0064) 0.6625 (0.3514) 0.0187 (0.0086) 0.0182 (0.0035)

0.0297 (0.0079) 0.0089 (0.0015) 0.0228 (0.0029) 0.0133 (0.0028) 0.0116 (0.0044)

0.0291 (0.0043) 0.0064 (0.0016) 0.0177 (0.0032) 0.0120 (0.0029) 0.0098 (0.0038)

Extra Scenario 3

Pooled (NN) 2-Stage (NN) Separate (NN) Top-FT (NN) Pool-w-L (NN)

0.1963 (0.0784) 0.0696 (0.0393) 1.9146 (0.7185) 0.0747 (0.0376) 0.0710 (0.0376)

0.1868 (0.0610) 0.0545 (0.0231) 1.8240 (0.4825) 0.0703 (0.0223) 0.0562 (0.0214)

0.2073 (0.0686) 0.0231 (0.0061) 0.0644 (0.0147) 0.0534 (0.0134) 0.0407 (0.0142)

0.2012 (0.0375) 0.0182 (0.0049) 0.0505 (0.0119) 0.0494 (0.0102) 0.0267 (0.0074)

Extra Scenario 4

Pooled (NN) 2-Stage (NN) Separate (NN) Top-FT (NN) Pool-w-L (NN)

0.0149 (0.0070) 0.0174 (0.0098) 0.4438 (0.2062) 0.0155 (0.0076) 0.0283 (0.0128)

0.0143 (0.0062) 0.0124 (0.0060) 0.4219 (0.1147) 0.0123 (0.0050) 0.0180 (0.0068)

0.0121 (0.0036) 0.0081 (0.0020) 0.0263 (0.0059) 0.0084 (0.0017) 0.0122 (0.0047)

0.0100 (0.0017) 0.0065 (0.0013) 0.0197 (0.0063) 0.0078 (0.0015) 0.0083 (0.0023)

G. Details of Real Data Experiments G.1. Beijing PM2.5 Dataset In this section, we evaluate our framework on an air quality prediction task using an open-source dataset provided by (Chen, 2015) and following the study of (Liang et al., 2015). This dataset records hourly meteorological data from Beijing, China over several years, with particulate matter with a diameter of 2.5 micrometers (PM2.5), a central index for air pollution levels, as the predicted response variable. Specifically, the dataset spans January 1, 2010 to December 31, 2014, containing 43824 hourly records after removing missing values. Features include six numerical meteorological covariates (dew point, temperature, pressure, cumulated wind speed, and cumulated hours of snow and rain) and one categorical variable for dominant wind direction with four categories: northeast (NE), northwest (NW), south (S), and calm (CV). Due to Beijing’s geography with mountains to the north and west and industrial zones to the south and east, wind direction significantly influences air quality by determining whether clean air flows in or pollutants become trapped (Liang et al., 2015). This makes wind direction a natural group label, as the PM2.5-meteorological relationship varies substantially across wind directions. We conduct transfer learning based on these groups, with 4997, 14150, 9387, and 15290 samples for NE, NW, CV, and S winds, respectively. We specify a similar autoregressive model to that stated in (Liang et al., 2015) equation (4.3) for predicting PM2.5 as 45

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

follows: yt = fzt (xt , yt−1 ) + ϵt , where yt is the PM2.5 concentration at hour t, and yt−1 is the lag-1 PM2.5 value to capture temporal dependencies. The covariate vector xt includes the six numerical meteorological variables, along with four additional variables from sine and cosine transformations of the hour-of-day and month-of-year to help capture diurnal and seasonal patterns. The group label zt corresponds to one of the four dominant wind directions. We follow a similar setting to the low-dimensional numerical experiments, where we compare DNN and RF as estimators across various learning strategies. To prevent potential data leakage risks in time series prediction tasks, we use data from 2014 as the hold-out test set. We repeat the entire procedure for 20 independent trials using different random seeds and report the average test MSE across trials. For the neural network implementations, each model instance is a five-layer MLP with 64 neurons per layer, using ReLU activation functions after the first four layers. All networks are trained with the Adam optimizer at a learning rate of 10−3 , a batch size of 256, and early stopping with a patience of 10 epochs based on validation performance. For the random forest models, we use 100 decision trees with a maximum depth of 10, while other hyperparameters remain at their default settings. All input features are standardized to have zero mean and unit variance. Table 21. Overall and per-group average MSE for PM2.5 models over 20 independent trials. The best performance in terms of lower MSE within each group is bolded. Model Pooled (NN) 2-Stage (NN) Separate (NN) Top-FT (NN) Pool-w-L (NN) Pooled (RF) 2-Stage (RF) Separate (RF) Pool-w-L (RF)

Overall

NE

NW

S

CV

0.0555 (0.0023) 0.0519 (0.0007) 0.0541 (0.0009) 0.0529 (0.0011) 0.0538 (0.0023) 0.0556 (0.0002) 0.0534 (0.0003) 0.0580 (0.0002) 0.0542 (0.0002)

0.0509 (0.0024) 0.0486 (0.0016) 0.0555 (0.0039) 0.0481 (0.0017) 0.0521 (0.0020) 0.0514 (0.0004) 0.0456 (0.0004) 0.0502 (0.0005) 0.0503 (0.0004)

0.0566 (0.0014) 0.0548 (0.0024) 0.0561 (0.0027) 0.0561 (0.0036) 0.0557 (0.0031) 0.0581 (0.0005) 0.0555 (0.0005) 0.0602 (0.0005) 0.0558 (0.0005)

0.0617 (0.0030) 0.0573 (0.0009) 0.0591 (0.0010) 0.0581 (0.0010) 0.0585 (0.0022) 0.0626 (0.0004) 0.0606 (0.0006) 0.0643 (0.0004) 0.0610 (0.0003)

0.0469 (0.0046) 0.0418 (0.0007) 0.0434 (0.0011) 0.0435 (0.0019) 0.0452 (0.0039) 0.0437 (0.0004) 0.0438 (0.0005) 0.0494 (0.0010) 0.0435 (0.0005)

From Table 21, DNN trained with two-stage transfer learning achieves the lowest overall prediction error and wins in most subgroup metrics, highlighting the effectiveness of our proposed training strategy in low-dimensional settings. Notably, the two-stage training strategy also performs best when applied to RF, demonstrating the potential applicability of our framework and theorems to other nonparametric estimators. G.2. Detailed Configurations for UTKFace Experiment Using Pretrained Model As required by the pretrained face feature extraction model, all images were resized to 160 × 160 × 3 resolution using OpenCV (Bradski, 2000), with pixel values normalized to the range [0, 1] prior to feature extraction. For the neural networks used for age regression based on the extracted features, in addition to the MLP architecture described in the main text, each MLP instance was trained using the Adam optimizer with an initial learning rate of 10−4 . The model was trained under a cosine annealing learning rate schedule (Loshchilov & Hutter, 2016), where the rate smoothly decreased to 10−6 over the course of training. Early stopping was applied with a patience of 5 epochs based on the validation loss. G.3. Complementary Experiments on the UTKFace Dataset Joint Grouping by Ethnicity and Gender. In addition to ethnicity, gender is another potential factor that may influence facial characteristics and consequently affect age estimation. To further investigate this, we conducted complementary experiments where both gender and ethnicity were considered simultaneously, resulting in 10 subgroups. The following table presents the results, with all other configurations kept consistent with those in the experiments that considered only ethnicity. As shown in the table, the proposed two-stage transfer learning method improves models’ performance on each subgroup in comparison with the pooled training, achieving the lowest average MSE both overall and across most subgroups. 46

Transfer Learning in Nonparametric Regression with Deep ReLU Networks Table 22. Overall and per-group average MSE for age regression models over 20 independent trials, where both ethnicity and gender are used as grouping factors. Standard deviation is computed across seeds. Group

Pooled

2-Stage

Separate

Top-FT

Pool-w-L

Overall White-M White-F Black-M Black-F Asian-M Asian-F Indian-M Indian-F

59.1 (2.6) 63.8 (4.8) 77.2 (5.0) 58.0 (5.8) 57.9 (6.8) 49.1 (12.6) 34.7 (8.4) 52.5 (3.7) 43.1 (6.4)

56.4 (2.1) 61.2 (4.2) 71.7 (3.3) 56.4 (6.0) 57.1 (6.6) 47.4 (12.7) 32.0 (8.2) 51.0 (4.1) 41.0 (5.3)

63.4 (3.2) 65.0 (6.6) 78.5 (8.2) 62.3 (7.0) 68.0 (5.5) 60.0 (12.1) 39.4 (9.8) 58.0 (6.3) 50.0 (9.9)

58.8 (2.5) 63.6 (4.7) 76.7 (4.8) 58.0 (5.8) 57.7 (6.8) 48.8 (12.5) 34.4 (8.2) 52.2 (3.9) 43.1 (6.2)

57.0 (1.8) 62.0 (4.3) 73.7 (4.3) 56.5 (6.0) 56.3 (5.5) 48.3 (12.2) 33.7 (8.4) 50.2 (4.0) 40.8 (5.8)

Direct Modeling with MLP. As discussed in the main text, we can also directly employ MLPs to perform age estimation from raw image inputs. As a toy experiment, all images are resized to a resolution of 32 × 32 × 3 using OpenCV, with pixel values normalized to the range [0, 1]. Each image is then flattened into a 3,072-dimensional feature vector, which serves as the input xi . We compare five learning strategies using MLPs as nonparametric estimator, following the same data-splitting setting as before. For each neural network instance, we employ an MLP with six layers and ReLU activation functions. Specifically, the input vector has 3,072 dimensions for all strategies except Pooled-with-Label, where we concatenate the one-hot encoded ethnicity feature with the image vector, resulting in a 3,077-dimensional input. This input passes through layers with 2048, 1024, 512, 256, 128, and 64 units, respectively, and the network concludes with a scalar output for age prediction. We adopt the AdamW optimizer (lr = 10−3 , weight decay = 10−4 ) and train the model under a cosine annealing learning rate schedule. Early stopping is applied with a patience of 10 epochs based on validation loss. In Table 23, we report the average MSE over 20 random trials, where the two-stage transfer learning achieves the lowest overall MSE and wins in the most subgroups. Table 23. Overall and per-group average MSE for age regression models over 20 independent trials. Model Pooled (NN) 2-Stage (NN) Pool-w-L (NN) Separate (NN) Top-FT (NN)

Overall

White

Black

Asian

Indian

140.8 (13.7) 136.7 (12.1) 139.3 (12.5) 161.9 (14.8) 140.3 (13.5)

165.9 (13.0) 161.2 (11.3) 164.4 (10.6) 180.0 (15.2) 165.2 (13.0)

135.2 (14.3) 129.9 (12.3) 134.0 (12.2) 157.3 (16.1) 135.1 (13.5)

112.6 (16.4) 110.9 (15.7) 111.8 (18.8) 147.5 (17.0) 112.8 (16.8)

107.8 (12.0) 104.6 (9.8) 105.4 (10.3) 133.9 (14.3) 107.0 (11.6)

47

Related documents

• Hybrid Quantum-Classical Generative Adversarial Networks with Transfer Learning#139208
• A comparative analysis of deep learning and deep transfer learning approaches for identification of rice varieties.#489720
• QUARTET: Quad-branch cross-Attention and Random-walk Traces for Enhancing Transformers on Relational Graphs#1063066
• Efficient generative adversarial networks using linear additive-attention Transformers#1013372
• Learning to Select a Pre-trained Deep-Learning Model for Transfer Learning on Multivariate Time-Series Data.#489696