Nonlinear Data Integration via Kernel Methods for Data Collaboration Analysis Yamato Suetakea , Yuta Kawakamia , Shunnosuke Ikedab , Yuichi Takanob
arXiv:2605.27219v1 [cs.LG] 26 May 2026
a
Graduate School of Science and Technology, University of Tsukuba, Tsukuba, Ibaraki, Japan b Institute of Systems and Information Engineering, University of Tsukuba, Tsukuba, Ibaraki, Japan
Abstract Collaborative analysis of decentralized confidential datasets is important, but direct sharing of original datasets is often restricted by privacy and institutional constraints. Data collaboration (DC) analysis transforms each dataset into privacy-preserving intermediate representations via party-specific obfuscation functions and integrates them into common collaboration representations using an anchor dataset. However, many existing DC analysis methods rely on linear transformations for data obfuscation and integration, which may increase reconstruction risk. Although nonlinear dimensionality reduction can mitigate this risk, conventional linear integration methods cannot accurately align intermediate representations produced by nonlinear transformations. Moreover, existing integration methods mainly minimize discrepancies among parties and do not explicitly incorporate geometric or target-variable information useful for downstream analysis. To overcome these limitations, we first formulate linear kernel integration (LKI) as a linear integration method and then kernelize it to obtain nonlinear kernel integration (NKI). NKI admits a globally optimal solution via kernel ridge regression and an eigenvalue problem. We also introduce graph regularization and a centering constraint so that the target representation can capture geometric and target-variable information useful for downstream analysis. Experiments on image classification tasks demonstrate that NKI improves classification accuracy over existing linear integration methods under nonlinear dimensionality reduction, with further gains from target-variable-aware Email address: [email protected] (Yamato Suetake)
graph regularization and centering. The results also show that dimensionality reduction choices substantially affect both classification accuracy and reconstruction risk. Keywords: Data collaboration (DC) analysis, Privacy-preserving machine learning, Kernel method, Nonlinear integration, Distributed learning 1. Introduction In recent years, the collaborative use of decentralized datasets held by multiple parties has become increasingly important in domains such as healthcare, finance, manufacturing, and public administration. Such collaboration can increase the effective sample size, incorporate diverse data distributions, and improve prediction and estimation accuracy beyond what is possible for a single party. In practice, however, original datasets often contain confidential information such as personal data and trade secrets, making central analysis—which aggregates all original datasets at a single location—legally, ethically, and practically challenging [1–3]. This has motivated privacypreserving machine learning (PPML), which aims to enable collaborative analysis while protecting the confidentiality of original datasets [4]. A wide variety of PPML frameworks have been proposed [5–14]. Although these methods provide ways to analyze decentralized datasets without sharing original datasets, many tightly couple the privacy mechanism with a specific downstream analysis algorithm, restricting the reusability of the protected data. Practical decentralized analysis therefore calls for a framework in which the privacy-preserving transformation performed by each party is independent of the downstream analysis performed by the analyst. Data collaboration (DC) analysis has been proposed as such a framework [15, 16]. In DC analysis, each party independently applies its own obfuscation function to transform its original dataset into intermediate representations and provides only these representations to an analyst. The analyst constructs integration functions using a shared anchor dataset and integrates the intermediate representations into collaboration representations in a common feature space. Because each party’s obfuscation process and the analyst’s downstream analysis are completely decoupled, the resulting integrated dataset can be analyzed with any method without modifying the upstream obfuscation. However, existing DC analysis methods still face important limitations 2
in the design of integration functions and obfuscation functions. First, no existing linear integration method guarantees a globally optimal solution without either retaining unnecessary degrees of freedom, imposing strong structural assumptions, or relying on rank conditions that may be violated. Second, linear integration functions can effectively integrate only intermediate representations from low-confidentiality linear obfuscation and cannot align representations produced by more secure nonlinear dimensionality reduction. Third, existing integration methods cannot incorporate information useful for downstream analysis into the integration process. To address these issues, we propose a kernel-based integration framework for DC analysis consisting of two methods. We first propose linear kernel integration (LKI), which overcomes the limitations of existing linear integration methods by admitting an analytically derived globally optimal solution without unnecessary degrees of freedom, strong structural assumptions, or rank conditions. We then kernelize LKI to obtain nonlinear kernel integration (NKI), which overcomes the limitation that linear integration functions cannot align representations from more secure nonlinear dimensionality reduction, while retaining an analytically derived globally optimal solution. We further extend NKI with graph embedding and a centering constraint to incorporate information useful for downstream analysis into the target representation. Our contribution The main goal of this paper is to construct integration methods for DC analysis that retain an analytically tractable optimization structure and, in particular, can handle intermediate representations obtained via nonlinear dimensionality reduction. To this end, we summarize our main contributions as follows: • First, we propose LKI as a principled linear integration method for DC analysis and show that an analytically derived globally optimal target representation of anchor data can be obtained from an eigenvalue problem. • Second, we propose NKI by kernelizing the integration function of LKI, thereby enabling DC analysis with intermediate representations obtained via nonlinear dimensionality reduction. For a fixed target representation, we show that each integration function is obtained from 3
a finite-dimensional kernel ridge regression-type problem and that the target representation is estimated by solving an eigenvalue problem. • Third, we introduce graph embedding to incorporate geometric structure and target-variable information into the target representation, and we add a centering constraint to suppress degenerate constant solutions. • Finally, through numerical experiments on image classification tasks, we evaluate analytical accuracy, computation time, anchor dataset design, and reconstruction risk, and show that NKI improves prediction accuracy under nonlinear dimensionality reduction. Structure of this paper This paper is structured as follows. Section 2 reviews privacy-preserving machine learning and related integration methods for DC analysis. Section 3 describes the DC analysis framework considered in this study. Section 4 formulates the proposed linear integration, nonlinear kernel integration, graph embedding, and centering constraint. Section 5 reports numerical experiments on analytical accuracy, computation time, anchor dataset design, and reconstruction risk. Finally, Section 6 concludes this paper and discusses future work. 2. Related Work This section reviews studies related to privacy-preserving decentralized data analysis. We first summarize representative PPML frameworks and data obfuscation methods, clarifying why they are not direct comparison targets in this study. We then review DC analysis and existing integration methods, discussing their limitations and how they motivate the proposed LKI, NKI, and graph embedding extensions. 2.1. Privacy-preserving Machine Learning and Data Obfuscation A wide variety of PPML techniques have been proposed to protect confidential information in decentralized data environments. Representative approaches include differential privacy [5–7], secure computation based on cryptography [8, 17–20], homomorphic encryption [9, 10], and federated learning [11–14]. Differential privacy provides a rigorous privacy definition by controlling the influence of individual records on analytical results, but stronger privacy guarantees often require larger amounts of noise and can degrade 4
analytical accuracy and data utility [7]. Secure computation and homomorphic encryption enable computation on encrypted data and provide strong security guarantees; however, they incur substantially higher computational costs than plaintext computation, particularly for large-scale datasets or complex machine learning models [8, 9]. Federated learning enables collaborative model training without directly sharing original datasets by exchanging model updates among parties. However, it requires iterative communication, can increase communication costs and runtime, and may leak information through gradients or model updates unless additional mechanisms such as secure aggregation are employed [21–25]. Furthermore, because aggregation in federated learning is tightly coupled with a specific learning algorithm, it is difficult to construct a reusable integrated dataset and then freely select a downstream analysis method. These PPML techniques address different aspects of privacy-preserving analysis and are therefore not direct competitors of the integration methods studied in this paper. Differential privacy focuses on a privacy definition, secure computation and homomorphic encryption focus on protecting the computation process, and federated learning focuses on a collaborative model-training protocol. In contrast, this study focuses on integration methods that construct collaboration representations from party-specific intermediate representations after obfuscation. Accordingly, our experimental comparisons focus on integration methods within DC analysis rather than on PPML frameworks with different objectives and assumptions. Data obfuscation-based methods, also referred to as data perturbation, are closely related to this study because they transform original datasets before sharing and can decouple data aggregation from downstream analysis without iterative communication [26–28]. A representative example is multiplicative data perturbation using a common random matrix [29]. However, when a shared linear transformation is employed, a structural vulnerability arises: if correspondences between original and transformed data are partially leaked, the transformation matrix can be estimated and other parties’ datasets may be reconstructed [30]. Private obfuscation mitigates this issue by allowing each party to apply an independently generated transformation [31]. In this setting, however, party-specific transformations map datasets onto different bases or coordinate systems, and naive aggregation of the resulting intermediate representations tends to cause misalignment in the feature space. These two limitations—reconstruction risk from shared transformations and misalignment from private transformations—indicate that 5
privacy-preserving data obfuscation requires not only secure transformations of original datasets but also an integration mechanism that aligns partyspecific intermediate representations. 2.2. DC Analysis and Limitations of Existing Methods Data collaboration (DC) analysis is a framework for decentralized data analysis that integrates party-specific intermediate representations into collaboration representations using an anchor dataset [15, 16]. Each party applies its own dimensionality reduction or obfuscation function to its original dataset and provides only the resulting intermediate representations to an analyst. The analyst constructs integration functions by referencing the intermediate representations of the anchor dataset from each party and analyzes the resulting collaboration representations in a common feature space. Imakura et al. [32] theoretically demonstrated that intermediate representations in DC analysis can protect original datasets against both internal and external attacks. Furthermore, DC analysis can be combined with differential privacy and federated learning, and such combinations have been reported to complement the limitations of each approach [33, 34]. Owing to its computational efficiency and versatility, DC analysis has been applied to a wide range of decentralized data analysis tasks, including predictive modeling and causal inference [35, 36], survival analysis [37], explainable machine learning [38], recommender systems [39], and clustering [40]. Privacy-oriented variants of DC analysis have also been proposed to make correspondence with original datasets more difficult to establish [41]. These studies indicate that DC analysis is a practical framework for decentralized data analysis, but its performance and confidentiality depend strongly on how party-specific intermediate representations are integrated. As a representative integration method in DC analysis, Imakura et al. [16] proposed the minimum perturbation problem (MPP), which minimizes discrepancies among the collaboration representations of the anchor dataset. MPP has been widely adopted as a standard integration method in DC analysis studies [33–43]. However, its formulation retains an unnecessary degree of freedom: any right-side invertible transformation of the collaboration representations can yield the same objective value, and this ambiguity can affect downstream performance. Orthogonal data collaboration (ODC) addresses this instability by restricting integration functions to orthogonal transformations [44]. This restriction removes unnecessary transformation degrees of freedom and stabi6
lizes the scale and inner-product structure of the collaboration representations. However, ODC requires the dimensionality reduction or obfuscation performed by each party to be compatible with an orthogonal transformation, which may be too restrictive in general decentralized settings. In addition, if the anchor dataset or its correspondences are leaked, the obfuscation function may be estimated using the orthogonal structure as a clue, increasing the risk of reconstructing original datasets. A generalized eigenvalue problem (GEP)-based integration method has also been proposed [45]. This method avoids both the unnecessary transformation degree of freedom in MPP and the strong orthogonality assumption in ODC. However, it relies on rank conditions for the intermediate representations of the anchor dataset, which may be violated depending on the anchor dataset size, the intermediate dimension, or the dimensionality reduction method. Furthermore, the relationship between the obtained solution and a globally optimal solution to a clearly defined optimization problem is not explicitly characterized. Therefore, DC analysis requires a principled linear integration method that has a clear optimization objective, avoids unnecessary degrees of freedom and strong structural assumptions, and admits an analytically derived globally optimal solution. This requirement motivates LKI in this study. Existing DC analysis methods often employ linear dimensionality reduction methods, particularly principal component analysis (PCA), as obfuscation functions. Linear dimensionality reduction is theoretically tractable and simplifies the construction of integration functions. However, it has a structural vulnerability: although the analyst normally receives only the intermediate representations of the anchor dataset, if the original anchor dataset— which is commonly held across all parties—is also accessed by the analyst, the linear obfuscation function can be estimated by applying the pseudoinverse of the anchor data to its intermediate representations. In particular, if a party colludes with the analyst such that the number of leaked original anchor samples substantially exceeds the feature dimensionality, the linear obfuscation function can be completely identified and the original datasets of other parties reconstructed [29, 44]. This limitation motivates the use of nonlinear dimensionality reduction as an obfuscation function. Nonlinear dimensionality reduction is a promising way to mitigate reconstruction risk because a single global linear inverse mapping is generally insufficient for recovering original datasets. However, simply replacing linear dimensionality reduction with nonlinear dimensionality reduction is not 7
enough. Previous attempts to introduce nonlinear dimensionality reduction into DC analysis reported degradation in analytical accuracy compared with linear methods [46, 47]. A possible reason is that nonlinear dimensionality reduction changes local coordinate structures in a party-specific manner, and a single linear integration function cannot generally align the resulting nonlinear intermediate representations. Therefore, a nonlinear integration method is needed to construct collaboration representations from nonlinear intermediate representations. This requirement motivates NKI in this study. Most existing integration methods mainly minimize discrepancies among parties and do not explicitly incorporate information that may be useful for downstream analysis. In particular, the geometric structure of the anchor dataset and the relationship between anchor samples and target variables are often not directly reflected in the target representation. As a result, even if the collaboration representations are aligned across parties, they do not necessarily preserve structures that are useful for prediction or classification. This gap motivates the graph embedding and centering constraint extensions proposed in this study. 3. Data Collaboration (DC) Analysis This section describes the DC analysis framework considered in this study. We first define the original datasets, anchor dataset, and obfuscation functions for each party. We then introduce intermediate representations, integration functions, collaboration representations, and the target representation of anchor data. Finally, we describe how the analyst constructs a single integrated dataset for downstream analysis. Fig. 1 provides an overview of the DC analysis framework considered in this study. Throughout this paper, we denote by [n] the set of positive integers {1, 2, . . . , n}. We consider K parties, where each party k ∈ [K] holds a d-dimensional feature matrix X (k) ∈ Rn(k)×d and a target variable vector y (k) ∈ Rn(k) for n(k) samples. In DC analysis, an anchor dataset A ∈ Rna ×d is introduced to provide a common reference for integrating datasets across parties; it is shared among parties but is not disclosed to the analyst. The anchor dataset is pseudodata shared by all parties; it has the same feature dimension as the original datasets but an arbitrary number of rows, and is artificially generated using, for example, random sampling or oversampling of publicly shareable data. 8
Each Party X (1) f 1 A X (2) f 2 A
...
X̃ (1) Ã(1) X̃ (2) Ã(2)
...
X (K) f K A
Analyst X̃ (1) g1 Ã(1) X̃ (2) g2 Ã(2)
...
X̃ (K) Ã(K)
X̂ (1) Â(1)
X̂ (2) .. .
X̂ (2) Â(2)
h
X̂ (K)
...
X̃ (K) gK Ã(K)
y (1)
X̂ (1)
X̂ (K) Â(K)
y (2) .. . y (K)
Z
g1 , g 2 , . . . , g K , h
Figure 1: Workflow of data collaboration (DC) analysis.
To protect data confidentiality, each party k ∈ [K] constructs its own ˜ obfuscation function fk : R1×d → R1×d(k) . We denote the i-th row vector of a matrix X ∈ Rn×d by Xi,: ∈ R1×d . The action of fk on a matrix is defined as follows: fk (X1,: ) ˜ .. n×d(k) fk (X) := . (1) ∈R . fk (Xn,: ) Applying fk to both the original dataset and the anchor dataset yields the following intermediate representations: ˜
X̃ (k) := fk (X (k) ) ∈ Rn(k)×d(k) , ˜
Ã(k) := fk (A) ∈ Rna ×d(k) .
(2)
Here, X̃ (k) and Ã(k) denote the intermediate representations of the original dataset and the anchor dataset, respectively. Each party keeps its original dataset, anchor dataset, and obfuscation function confidential, and provides only the intermediate representations together with the target variable vector y (k) to the analyst. This restriction is important because simultaneous access to the anchor dataset and its intermediate representations could allow the analyst to estimate the obfuscation function. ˜ ˆ The analyst constructs an integration function gk : R1×d(k) → R1×d that maps party-specific intermediate representations into a common feature space. The corresponding collaboration representations are defined as 9
follows:
ˆ
X̂ (k) := gk (X̃ (k) ) ∈ Rn(k)×d , ˆ
Â(k) := gk (Ã(k) ) ∈ Rna ×d .
(3)
The integration function gk is not chosen arbitrarily; it must align the intermediate representations, which differ across parties, into a common feature space. Ideally, we would like the composed functions across parties to approximately coincide: g k ◦ fk ≈ g k ′ ◦ fk ′ ,
k, k ′ ∈ [K].
(4)
When Eq. (4) holds, essentially the same feature transformation is applied to the original datasets of all parties, and the collaboration representations can be treated as coherent observations in a common feature space. However, the analyst cannot directly observe the obfuscation function fk itself; the analyst observes only its output on the anchor dataset, Ã(k) = fk (A). Therefore, Eq. (4) cannot be directly enforced. Instead, we adopt concordance on the anchor dataset as a surrogate objective and design the integration functions to satisfy: ′
gk (Ã(k) ) ≈ gk′ (Ã(k ) ),
k, k ′ ∈ [K].
(5)
In practice, rather than handling Eq. (5) for each pair of parties, we inˆ troduce a target representation Z ∈ Rna ×d of anchor data in the common feature space. Each integration function is then trained so that the collaboration representations of the anchor dataset approach this common target: gk (Ã(k) ) ≈ Z,
k ∈ [K].
(6)
Eq. (6) aligns the intermediate representations of all parties through the common target representation. Once the integration function gk for each party is obtained, the analyst applies it to the intermediate representations X̃ (k) to obtain the collaboration representations X̂ (k) . Since these collaboration representations lie in the same common feature space, the analyst vertically concatenates them to construct the following single integrated dataset: y (1) X̂ (1) ŷ = ... . (7) X̂ = ... , y (K) X̂ (K) 10
The analyst then performs regression or classification using the single integrated dataset in Eq. (7) and trains a machine learning model h on the collaboration representations. After training, the analyst returns the integration function gk and the trained model h to each party k, as indicated by the feedback arrow in Fig. 1. By composing these with its own obfuscation function fk , each party obtains a model applicable directly to its original dataset: h ◦ g k ◦ fk . Through this procedure, each party can perform analysis and prediction on unseen data independently, without any additional communication with the analyst. 4. Proposed Method This section presents the proposed integration methods for DC analysis. We first formulate linear kernel integration (LKI) as a linear integration problem and give its globally optimal solution. We then extend LKI to nonlinear kernel integration (NKI) by kernelizing the integration function. Finally, we introduce graph embedding and a centering constraint to incorporate geometric and target-variable information into the target representation. 4.1. Linear Kernel Integration Motivated by Eq. (6), we first consider the case where the integration function is a linear map. For each party k ∈ [K], the integration function is defined as follows: ˜
ˆ
G(k) ∈ Rd(k)×d ,
gk (x) = xG(k) ,
(8)
˜
where x ∈ R1×d(k) and G(k) is a coefficient matrix. Then, the collaboration representations of the anchor dataset are given by gk (Ã(k) ) = Ã(k) G(k) .
(9) ˆ
To align Eq. (9) with a common target representation Z ∈ Rna ×d , a naive formulation is written as follows: min
Z, {G(k) }K k=1
K X
Ã(k) G(k) − Z
k=1
11
2
. F
(10)
However, the problem in Eq. (10) admits the trivial optimal solution Z = 0 and G(k) = 0. To eliminate such degenerate solutions and fix the scale of each feature axis, we impose the orthonormality constraint Z ⊤ Z = Idˆ.
(11)
As a result, we formulate the following linear integration problem: K X
min
Ã(k) G(k) − Z
k=1 ˜
ˆ
s. t. G(k) ∈ Rd(k)×d
2 F
(k ∈ [K]),
(12)
Z ⊤ Z = Idˆ, ˆ
Z ∈ Rn a ×d . The objective in Eq. (12) measures the discrepancy between the collaboration representations of the anchor dataset and the target representation. The constraint in Eq. (11) prevents the zero solution and avoids selecting the same direction multiple times. The problem in Eq. (12) can be solved by a two-step reduction. First, for fixed Z, the inner least-squares problem with respect to G(k) is solved by the Moore–Penrose pseudoinverse. Second, substituting this solution into Eq. (12) reduces the problem over Z to a constrained trace maximization. The following theorem summarizes the resulting globally optimal solution. Theorem 1 (Global Optimal Solution of the Linear Integration Problem). Assume that dˆ ≤ na . For each k ∈ [K], let Q(k) ∈ Rna ×r(k) be an orthonormal basis for Col(Ã(k) ), where r(k) = rank(Ã(k) ). Define WQ = [Q(1) , . . . , Q(K) ],
(13)
ˆ and let UQdˆ ∈ Rna ×d be one choice of the dˆ left singular vectors of WQ ˆ dˆ ˆ := {O ∈ Rd× corresponding to the dˆ largest singular values. Let O(d) | ˆ a globally optimal solution to Eq. (12) O ⊤ O = Idˆ}. Then, for any O ∈ O(d), is given by
Z ⋆ = UQdˆO,
G(k)⋆ = (Ã(k) )† Z ⋆
(k ∈ [K]).
(14)
For each fixed Z ⋆ , the coefficient matrix G(k)⋆ in Eq. (14) is the minimumnorm least-squares solution. The arbitrary orthogonal matrix O reflects the 12
ˆ selected rotational indeterminacy of the target representation. When the d-th singular value is separated from the next one, this form characterizes all globally optimal Z ⋆ up to an orthogonal rotation. The proof is provided in Appendix A. Theorem 1 shows that Z ⋆ is obtained from the dominant left singular subspace of WQ in Eq. (13). Thus, Z ⋆ can be interpreted as an orthonormal representation of the directions most consistently present across parties. We refer to this linear integration method as Linear Kernel Integration (LKI). 4.2. Nonlinear Kernel Integration The linear formulation in Eq. (12) clarifies the optimization structure of the target representation. From a data confidentiality perspective, however, it is desirable to integrate intermediate representations obtained via nonlinear dimensionality reduction. We therefore extend LKI to a nonlinear integration method by kernelizing the integration function. For each party k ∈ [K], let Hk denote the RKHS associated with the reproducing kernel ˜ ˜ (15) κk : R1×d(k) × R1×d(k) → R. ˆ Since the collaboration representations are d-dimensional, we consider the vector-valued function space K X 2 (k) 2 gk (Ã ) − Z + λkgk kHdˆ min F
k=1 ˆ
s. t. gk ∈ Hkd
(k ∈ [K]),
k
(16)
Z ⊤ Z = Idˆ, ˆ
Z ∈ Rn a ×d . where λ ∈ R+ := {x ∈ R | x > 0} is a regularization parameter controlling the RKHS norm penalty. The problem in Eq. (16) is solved by an analogous two-step reduction. First, for fixed Z, each integration function is obtained by solving a finitedimensional kernel ridge regression-type problem. Second, substituting the optimal integration functions into Eq. (16) reduces the problem over Z to a constrained trace minimization. To state this reduction, define the kernel matrix (k) (k) (Kk )ii′ = κk (Ãi,: , Ãi′ ,: ), Kk ∈ Rna ×na , (17) 13
˜
for each k ∈ [K]. For any x ∈ R1×d(k) , define
(k) 1×na . κk (x) = κk (x, Ã1,: ), . . . , κk (x, Ã(k) na ,: ) ∈ R
Finally, let −1
Sk := (Kk + λIna ) ,
Mλ := λ
K X
Sk .
(18)
(19)
k=1
Theorem 2 (Reduction of the Nonlinear Kernel Integration Problem). Assume that dˆ ≤ na . The problem in Eq. (16) reduces to min tr(Z ⊤ Mλ Z) s. t.
Z ⊤ Z = Idˆ, Z∈R
(20)
na ×dˆ
.
Moreover, for fixed Z, the optimal integration function is given by gk⋆ (x) = κk (x)Sk Z.
(21)
The proof is provided in Appendix B. By Theorem 2, the problem in Eq. (16) reduces to the constrained trace minimization in Eq. (20). Since each Kk is symmetric positive semidefinite and λ > 0, Sk in Eq. (19) is symmetric positive definite. Therefore, Mλ is also symmetric positive definite. By the Ky Fan minimum principle [48], a globally optimal solution to Eq. (20) is obtained by taking the eigenvectors associated with the dˆ smallˆ est eigenvalues of Mλ . Let Udˆ ∈ Rna ×d be the matrix composed of these ˆ eigenvectors. Then, for any O ∈ O(d), Z ⋆ = UdˆO
(22)
is a globally optimal solution to Eq. (20). The arbitrary orthogonal matrix O reflects the rotational indeterminacy of the target representation. When ˆ selected eigenvalue is separated from the next one, this form charthe d-th acterizes all globally optimal Z ⋆ up to an orthogonal rotation. Substituting Eq. (22) into Eq. (21), we obtain gk⋆ (x) = κk (x)Sk Z ⋆ .
(23)
Since Z ⋆ is constructed from the bottom-dˆ eigenspace of Mλ , it represents the subspace that minimizes the regularized discrepancy among parties in the RKHS-induced common feature space. We refer to this nonlinear integration method as Nonlinear Kernel Integration (NKI). 14
4.3. Graph Embedding The target representation Z ⋆ obtained by NKI aligns the intermediate representations of the anchor dataset across parties, but it does not explicitly exploit relational structure among anchor samples. To incorporate such structure, we introduce graph embedding into the estimation of the target representation. Following the graph embedding framework [49], we use two graphs: an intrinsic graph and a penalty graph. Let i, i′ ∈ [na ] denote indices of anchor samples. For each pair (i, i′ ), let (B) (C) wii′ and wii′ be the weights of the intrinsic graph and the penalty graph, respectively. Specific constructions of these weights are given in Section 4.4. The corresponding weight matrices are defined as follows: W (B) = [wii′ ] ∈ Rna ×na ,
W (C) = [wii′ ] ∈ Rna ×na .
(B)
(C)
(24)
Let D (B) and D (C) be the degree matrices corresponding to the weight matrices in Eq. (24). Using these matrices, we define the intrinsic-graph and penalty-graph Laplacians as follows: B = D (B) − W (B) ,
C = D (C) − W (C) .
(25)
Here, B, C ∈ Rna ×na . Hereafter, we refer to B and C as the intrinsic-graph Laplacian and the penalty-graph Laplacian, respectively. Using the Laplacians in Eq. (25) and writing Z = [z1⊤ , . . . , zn⊤a ]⊤ , the Laplacian quadratic forms can be rewritten as follows: 1 X (B) tr(Z ⊤ BZ) = w ′ kzi − zi′ k22 , 2 i,i′ ii (26) 1 X (C) 2 ⊤ w ′ kzi − zi′ k2 . tr(Z CZ) = 2 i,i′ ii Eq. (26) is a standard identity in Laplacian Eigenmaps and LPP [50, 51]. Therefore, the first equality in Eq. (26) shows that minimizing tr(Z ⊤ BZ) encourages pairs connected by the intrinsic graph to be close in the target representation. Similarly, the second equality in Eq. (26) shows that C controls the scale or separation induced by the penalty graph. Assume that dˆ ≤ na . Using the graph Laplacian matrices B and C defined in Eq. (25), we extend Eq. (20) by adding the graph regularization term motivated by Eq. (26) to the objective and replacing the orthonormality 15
constraint with a penalty-graph-weighted constraint. Specifically, we add µtr(Z ⊤ BZ) and impose Z ⊤ CZ = Idˆ, where µ ≥ 0 controls the strength of graph regularization. Since these modifications depend only on Z, the inner minimization over {gk } for fixed Z is unchanged. Thus, by the same argument as in Theorem 2, the optimal form of gk⋆ in Eq. (21) remains valid, and the reduced problem becomes min tr Z ⊤ (Mλ + µB)Z s. t. Z ⊤ CZ = Idˆ,
(27)
ˆ
Z ∈ Rn a × d . By the Lagrange multiplier method, the problem in Eq. (27) reduces to the following generalized eigenvalue problem: (Mλ + µB)u = γ Cu.
(28)
If C in Eq. (27) is singular, we replace it with Cε = C + εIna for ε > 0 and solve the corresponding regularized problem. By the Ky Fan minimum principle, a globally optimal solution to Eq. (27), or to the regularized problem when Cε is used, is obtained by taking the generalized eigenvectors corresponding to the dˆ smallest generalized eigenvalues of Eq. (28) with C ˆ replaced by Cε if necessary. Let Udˆ ∈ Rna ×d be the matrix composed of ˆ Z ⋆ = U ˆO the selected generalized eigenvectors. Then, for any O ∈ O(d), d gives a globally optimal solution to the corresponding reduced problem. The arbitrary orthogonal matrix O reflects the rotational indeterminacy of the ˆ selected generalized eigenvalue is septarget representation. When the d-th arated from the next one, this form characterizes all globally optimal Z ⋆ up to an orthogonal rotation. Substituting this Z ⋆ into Eq. (21) yields the graph-regularized integration function. In practice, Mλ and B may differ in scale. We therefore normalize both matrices to have unit trace by replacing Mλ with Mλ /tr(Mλ ) and B with B/tr(B). Under this normalization, µ can be interpreted as a dimensionless parameter controlling the relative importance of integration error and graph regularization. NKI is recovered from Eq. (27) by setting µ = 0 and C = Ina . For clarity, we refer to the graph-regularized variant introduced in this section as NKI+Graph. In the experiments, this abstract variant is instantiated by concrete graph choices, including GL and TSL, i.e., NKI+GL and NKI+TSL. 16
4.4. Graph Construction This subsection describes how to construct the intrinsic-graph weights (B) (C) wii′ and penalty-graph weights wii′ used in Eq. (24). Because the analyst cannot observe the anchor dataset A directly and has access only to the intermediate representations of the anchor dataset Ã(k) , the weights are constructed from the intermediate representations of each party and then aggregated across parties. For each party k ∈ [K] and each pair i, i′ ∈ [na ], define per-party raw (Bk) (Ck) weights ŵii′ and ŵii′ . To make each Laplacian symmetric, the weights are symmetrized as follows: 1 (Bk) 1 (Ck) (Bk) (Ck) (Ck) (Bk) ŵii′ + ŵi′ i , wii′ := ŵii′ + ŵi′ i . (29) wii′ := 2 2 They are then averaged across parties: 1 X (Bk) w ′ , K k=1 ii K
(B) wii′ :=
1 X (Ck) w ′ . K k=1 ii K
(C) wii′ :=
(30)
The average in Eq. (30) treats each party equally, suppresses party-specific distortions, and extracts relational structures that are consistently present across parties. The Laplacians B and C are then constructed from the aggregated weight matrices according to Eq. (25). (k) Let Nknn (i) denote the knn -nearest-neighbor set of anchor sample i in Ã(k) . Let yai denote the target variable assigned to anchor sample i, when such target information is available for the anchor dataset. In the experiments, these anchor target variables are assigned according to the anchor construction procedure described in Section 5. The graph construction therefore uses only target variables assigned to anchor samples and does not require access to original private datasets. We define three types of Laplacians: the geometric Laplacian (GL), target-similarity Laplacian (TSL), and target-dissimilarity Laplacian (TDL). GL and TSL are candidates for the intrinsic-graph Laplacian B in the objective tr(Z ⊤ BZ) and define pairs to be kept close in the target representation. TDL is a candidate for the penalty-graph Laplacian C and defines pairs used for scale normalization or separation in the constraint. TSL and TDL are used only when target variables are assigned to the anchor dataset. When only B is specified and C is not separately defined, we set C = Ina . Specific combinations are defined in Section 5. 17
(1) Geometric Laplacian (GL). For GL, attraction weights are defined only by nearest-neighbor relationships: ( (k) 1 if i′ ∈ Nknn (i), (Bk) ŵii′ = (31) 0 otherwise. (2) Target-Similarity Laplacian (TSL). For classification tasks, TSL uses nearest-neighbor pairs with the same target variable: ( (k) 1 if i′ ∈ Nknn (i), yai = yai′ , (Bk) ŵii′ = (32) 0 otherwise. For regression tasks, TSL uses a smooth similarity based on target values: 2 (k) exp − (yai − yai′ ) if i′ ∈ Nknn (i), (Bk) 2 σy (33) ŵii′ = 0 otherwise, where σy > 0 is a scale parameter for the difference in target values. (3) Target-Dissimilarity Laplacian (TDL). For classification tasks, TDL uses nearest-neighbor pairs with different target variables: ( (k) 1 if i′ ∈ Nknn (i), yai 6= yai′ , (Ck) ŵii′ = (34) 0 otherwise. For regression tasks, TDL uses a smooth dissimilarity based on target values: 2 (k) 1 − exp − (yai − yai′ ) if i′ ∈ Nknn (i), (Ck) 2 σy ŵii′ = (35) 0 otherwise. Combining TSL and TDL as B and C, respectively, encourages the target representation to reflect discriminative structures associated with the target variable.
18
4.5. Centering Constraint The constraint Z ⊤ CZ = Idˆ in Eq. (27) fixes the scale and orthogonality of the target representation but does not directly constrain its column mean 1⊤ na Z. Consequently, a redundant constant-direction component can remain in the solution. This component corresponds to a degree of freedom in the origin of the embedding and may reduce the interpretability and comparability of the target representation. To remove this component, we impose the following centering constraint: 1⊤ na Z = 01×dˆ.
(36)
For the centered formulation, we assume that dˆ ≤ na − 1 because the feasible subspace orthogonal to 1na has dimension na − 1. The problem in Eq. (27) is then extended as follows: min tr Z ⊤ (Mλ + µB)Z s. t. Z ⊤ CZ = Idˆ,
(37)
1⊤ na Z = 01×dˆ, ˆ
Z ∈ Rn a × d . The second constraint in Eq. (37), namely Eq. (36), is equivalent to requiring that each column of Z belongs to U = x ∈ Rn a | 1 ⊤ (38) na x = 0 . Let T ∈ Rna ×(na −1) be a matrix whose columns form an orthonormal basis ⊤ for U , i.e., 1⊤ na T = 0 and T T = Ina −1 . Then any Z feasible for Eq. (37) can be represented as Z = TY ,
ˆ
Y ∈ R(na −1)×d .
(39)
Substituting Eq. (39) into Eq. (37), we obtain ⊤ min tr Y M̃ Y s. t. Y ⊤ C̃Y = Idˆ, ˆ
Y ∈ R(na −1)×d . 19
(40)
where, using the graph Laplacian matrices B and C defined in Eq. (25), the reduced matrices in Eq. (40) are defined as M̃ = T ⊤ (Mλ + µB)T ,
C̃ = T ⊤ CT .
(41)
By the Lagrange multiplier method, the problem in Eq. (40) reduces to the generalized eigenvalue problem M̃ u = γ C̃u.
(42)
If C̃ in Eq. (40) is singular, we replace it with C̃ε = C̃ + εIna −1 for ε > 0 and solve the corresponding regularized problem. By the Ky Fan minimum principle, a globally optimal solution to Eq. (40), or to the regularized problem when C̃ε is used, is obtained by taking the generalized eigenvectors corresponding to the dˆ smallest generalized eigenvalues of Eq. (42) with C̃ ˆ replaced by C̃ε if necessary. Let Udˆ ∈ R(na −1)×d be the matrix composed of ˆ Y ⋆ = U ˆO the selected generalized eigenvectors. Then, for any O ∈ O(d), d gives a globally optimal solution to the corresponding reduced problem. The arbitrary orthogonal matrix O reflects the rotational indeterminacy of the ˆ selected generalized eigenvalue centered target representation. When the d-th is separated from the next one, this form characterizes all globally optimal Y ⋆ up to an orthogonal rotation. Substituting this solution into Eq. (39), the globally optimal target representation for the centered problem is given by Z ⋆ = T Y ⋆. (43) Finally, substituting Eq. (43) into Eq. (21) yields the centered integration function gk⋆ . We refer to the centered variant without graph regularization, i.e., µ = 0 and the penalty-graph Laplacian is set to C = Ina , as NKI+Center. We refer to the centered graph-regularized variant as NKI+Graph+Center. Illustrative visualization of collaboration representations. To illustrate the effects of graph embedding and the centering constraint on collaboration representations, Fig. 2 shows a toy example with three classes. The color of each point indicates its target label. Fig. 2(a) shows an example of intermediate representations of anchor samples for one party. Figs. 2(b)–(d) show collaboration representations obtained by NKI, NKI+TSL, and NKI+TSL+Center, respectively. NKI nonlinearly integrates intermediate representations into a common feature space, but it does not directly use neighborhood relationships or target-variable information; therefore, geometric or class structures 20
(a) Intermediate representations
(b) NKI
(c) NKI+TSL
(d) NKI+TSL+Center
Figure 2: Illustrative visualization of NKI-based collaboration representations.
may not be sufficiently preserved in the resulting collaboration representations. In contrast, Fig. 2(c) shows that adding TSL as graph regularization tends to place neighboring samples with the same target label close to each other in the collaboration representation. Furthermore, Fig. 2(d) shows that the centering constraint removes a redundant constant-direction component and can yield a collaboration representation based on relative arrangements. This figure is a schematic visualization for explaining the roles of the proposed extensions and is not used for quantitative evaluation. 4.6. Algorithm and Computational Complexity This subsection summarizes the algorithmic procedure for NKI-based methods and analyzes their computational complexity. Table 1 lists the main steps for constructing the target representation and the integration functions. We analyze the complexity under the following assumptions: (i) each (k) Ã has full column rank, and (ii) the intermediate dimension is common ˜ ˆ across parties and equals the integrated dimension, i.e., d(k) = d˜ = d. Here, integration-function construction means constructing all party-wise integration functions {gk }K k=1 , whereas collaboration-representation construction means computing gk (x), i.e., constructing the collaboration representa˜ tion of one new sample x ∈ R1×d of party k. For existing methods (MPP, GEP, and ODC), we follow the complexity reported in prior work [44]. For the proposed methods, the procedure is summarized in Table 1, and the overall comparison is reported in Table 2. For LKI, the dominant cost of integration-function construction is the ˜ singular value decomposition of WQ ∈ Rna ×K d , which is ˜ 2 , n2 K d} ˜ . O min{na (K d) a 21
Table 1: Algorithm for constructing NKI-based integration functions.
Input Output Step 1 Step 2 Step 3
Step 4
K ˆ {Ã(k) }K k=1 , kernels {κk }k=1 , λ > 0, d, (optional) graph settings for B, C, and (optional) centering constraint Z ⋆ , {gk⋆ }K k=1
Construct kernel matrices Kk ∈ Rna ×na for all k ∈ [K] according to Eq. (17). ComputePSk = (Kk + λIna )−1 for all k ∈ [K] and construct Mλ = λ K k=1 Sk . Compute Z ⋆ by solving the corresponding reduced problem: NKI uses Eq. (20); NKI+Graph uses Eq. (27); NKI+Graph+Center uses Eq. (37). For each k ∈ [K], define gk⋆ (x) = κk (x)Sk Z ⋆ .
This order coincides with MPP and GEP. For collaboration-representation construction, LKI computes the linear map x 7→ xG(k) , resulting in O(d˜2 ) ˆ under d˜ = d. Next, we analyze NKI-based methods according to Table 1. In Step 1, 2˜ constructing all kernel matrices {Kk }K k=1 costs O(Kna d) for standard kernel evaluation. In Step 2, computing Sk = (Kk + λIna )−1 for all k ∈ [K] costs O(Kn3a ), and constructing Mλ is lower-order. In Step 3, computing Z ⋆ requires an eigendecomposition or generalized eigendecomposition of size na , which costs O(n3a ). Thus, the dominant cost is governed by Step 2. In Step 4, defining each gk⋆ from precomputed factors is lowerorder than matrix inversion and eigendecomposition. Therefore, the dominant complexity of integration-function construction for NKI, NKI+Graph, and NKI+Graph+Center is ˜ O(Kn3a + Kn2a d). The graph Laplacian matrices B, C and the centering-related transformations add preprocessing or lower-order terms and do not change the leading asymptotic order. For collaboration-representation construction, once Sk and Z ⋆ are precomputed, NKI-based methods evaluate Eq. (23). The per-sample dominant 22
Table 2: Computational complexity of integration methods.
Method MPP [16] GEP [45]
Function ˜ O min{K 2 na d˜2 , Kn2a d} ˜ O min{K 2 na d˜2 , Kn2 d} a
O(Kna d˜2 ) ˜ LKI O min{K 2 na d˜2 , Kn2a d} ˜ NKI O(Kn3a + Kn2a d) ˜ NKI + Graph O(Kn3a + Kn2a d) ˜ NKI + Graph + Center O(Kn3a + Kn2a d) ODC [44]
Representation O(d˜2 ) O(d˜2 ) O(d˜2 ) O(d˜2 ) ˜ O(na d) ˜ O(na d) ˜ O(na d)
cost is therefore ˜ O(na d). Table 2 summarizes these results. While NKI-based methods are computationally heavier than linear methods in integration-function construction, they enable integration after nonlinear dimensionality reduction. 5. Experiments This section evaluates the proposed method through four research questions. • RQ1: Does NKI improve classification accuracy compared with existing DC integration methods? • RQ2: How do the proposed extensions, namely graph regularization and centering, contribute to classification accuracy? • RQ3: How do anchor size and anchor quality affect classification accuracy and computation time? • RQ4: How do dimensionality reduction methods and intermediate dimensions affect the trade-off between classification accuracy and reconstruction risk? The following experiments focus on classification tasks. As a supplementary evaluation, Appendix C reports regression experiments on two additional datasets. 23
5.1. Experimental Setup 5.1.1. Default Settings Unless otherwise stated, all experiments use the default settings shown in Table 3. RQ-specific differences are described in the RQ-specific setting paragraphs. 5.1.2. Datasets and Partitioning We use MNIST [52] and Fashion-MNIST [53], both of which are 10-class classification datasets consisting of 28 × 28 grayscale images with input dimension d = 784. Input features were normalized to [0, 1] by dividing each pixel value by 255. We set the number of training samples for party k ∈ [K] to a common value n, i.e., n(k) = n, ∀k ∈ [K]. Test data are shared across all parties with a total size of nK. Training samples are randomly allocated to K parties in each trial. Hyperparameters for integration function construction are fixed throughout, and no validation data are used. 5.1.3. Dimensionality Reduction Settings Unless otherwise stated, each party uses UMAP [54] for dimensionality ˜ to a common value d˜ for reduction. We set the intermediate dimension d(k) ˜ = d˜ for all k ∈ [K]. To represent inter-party variation all parties, i.e., d(k) in the obfuscation function, the distance metric, number of neighbors, and minimum-distance parameter of UMAP were varied per party. The distance metric was selected from correlation, cosine, and euclidean. The number of neighbors (n_neighbors) was sampled uniformly from {2, . . . , 7}, and the minimum-distance parameter (min_dist) was sampled uniformly from [0.0, 0.8). 5.1.4. Compared Methods We compare the following methods. • Central: All party data are centrally aggregated to train a single model; this method ignores privacy and serves as an idealized upper bound. • Local: Each party trains a separate model on its own data only; this serves as a non-collaborative baseline. • MPP: Integration based on the minimum perturbation problem [16]. 24
• GEP: Integration based on the generalized eigenvalue problem [45]. • ODC: Integration based on the orthogonal Procrustes problem [44]. • LKI: The proposed linear kernel integration based on Eq. (12). • NKI: The proposed nonlinear kernel integration based on Eq. (16). • NKI+Center: NKI with the centering constraint (Eq. (36)). • NKI+GL: NKI with GL weights (Eq. (31)). • NKI+GL+Center: NKI+GL with the centering constraint (Eq. (36)). • NKI+TSL: NKI with TSL weights (Eq. (32) for classification). • NKI+TSL+Center: NKI+TSL with the centering constraint (Eq. (36)). ˜ For For all integration methods, the integrated dimension is set to dˆ = d. the NKI extensions, the penalty-graph Laplacian C defined in Eq. (25) is set to C = Ina unless otherwise stated. We also conducted preliminary experiments with the Target-Dissimilarity Laplacian (TDL), but no consistent improvement over TSL was observed; TDL is therefore excluded from the main figures and tables to improve readability. 5.1.5. Anchor Data and Target Variables Following the prior report that using SMOTE for anchor dataset generation can improve DC analysis performance [55], we adopt this approach. denotes the number of real data points used as SMOTE source Here, nsmote a samples, and na denotes the total size of the anchor dataset. In our setup, real data points disjoint from training and test data are used as source samples; these real samples are themselves included in the anchor set and augmented by SMOTE to reach the target anchor size. The number of neighbors used for SMOTE generation and graph construction is set to a common value knn . Note that the n_neighbors parameter of UMAP is configured separately to represent inter-party variation in the dimensionality reduction mapping. The anchor dataset is class-balanced, with na /10 samples per class. For RQ1–RQ3, SMOTE is applied only between source samples sharing the same target variable. Real source samples retain their original target variables, and SMOTE-generated samples are assigned the target variable of their corresponding source sample. 25
5.1.6. Random Seed Policy The random seed for each trial was set to the trial index r. This seed was applied to data partitioning, UMAP embedding, SMOTE-based anchor generation, and Random Forest classifier training. We conducted Nseed independent trials per condition; the specific number of trials for each RQ is stated in the RQ-specific settings. 5.1.7. Evaluation Analytical performance is evaluated as the mean test accuracy across all parties, where each party trains a classifier on its own collaboration representations. For Central, a single classifier is trained on the aggregated training data from all parties. We use RandomForestClassifier with default settings from scikit-learn (version 1.6.1) [56]. Results are reported as the mean and 95% confidence interval over independent trials. 5.1.8. Integration Function Parameters Default parameters for integration function construction follow Table 3. As the integration kernel κk in NKI, we use the RBF kernel κk (x, x′ ) = exp −γk kx − x′ k22 . The RBF kernel parameter is shared across all parties, i.e., γk = γ for all k ∈ [K]. The regularization parameter λ, the graph regularization parameter µ, and the RBF kernel parameter γ are fixed throughout all trials without validation-based tuning. 5.1.9. Computation Environment All experiments were conducted on a MacBook Air with Apple M4 (32 GB memory, macOS 15.7.2, arm64). All methods were implemented in Python 3.9.6 with NumPy 1.26.4, pandas 2.3.2, SciPy 1.13.1, scikit-learn 1.6.1, and umap-learn 0.5.9.post2. To eliminate variability from parallelism and ensure fair runtime comparison across methods, BLAS was fixed to single-threaded execution (OMP_NUM_THREADS=OPENBLAS_NUM_THREADS=MKL_NUM_THREADS=1). 5.1.10. RQ1 and RQ2 Specific Settings For RQ1 and RQ2, the number of parties K is varied over K = {2, 4, 8, 16, 32, 64}. RQ1 compares NKI against existing integration methods, and RQ2 compares the contributions of the proposed extensions GL, TSL, and Center. The number of trials per condition is Nseed = 100. 26
Table 3: Default experimental settings.
Item
Default setting
Datasets Task Feature dimension Feature normalization Number of parties Training samples per party Test samples Dimensionality reduction Intermediate dimension Integrated dimension Anchor size SMOTE source samples Anchor class balance Classifier Kernel Regularization SMOTE/graph neighborhood size
MNIST, Fashion-MNIST 10-class classification d = 784 Pixel values scaled to [0, 1] K = 10 n = 100 nK common samples UMAP d˜ = 10 dˆ = d˜ na = 1000 nsmote = 100 a na /10 samples per class RandomForestClassifier default settings RBF kernel with γ = 1 λ = 1, µ = 1 knn = 10
5.1.11. RQ3 Specific Settings Anchor Conditions. RQ3 uses MNIST only. To examine the effect of anchor quantity, we fix nsmote = 100 and vary na ∈ {100, 200, 400, 800, 1600}. To a ∈ examine the effect of anchor quality, we fix na = 1000 and vary nsmote a {10, 30, 100, 300, 1000}. The number of trials per condition is Nseed = 30. Computation Time Evaluation. For RQ3, we evaluate the following two types of computation time. • Integration-function construction time [ms]: time to construct the integration functions {gk }K k=1 for all parties (abbreviated as Function in figures and tables). • Collaboration-representation construction time [ms]: time to generate collaboration representations using the constructed integration functions (abbreviated as Representation in figures and tables). 27
Collaboration-representation construction time is defined as the total time to transform nK test samples from intermediate representations to collaboration representations across all parties. This total time differs from the per-sample collaboration-representation construction time shown in Table 2 by a constant factor depending on n and K, which are fixed in RQ3. Computation times are reported as the mean over independent trials for each condition. 5.1.12. RQ4 Specific Settings Dimensionality Reduction Conditions. For RQ4, we vary the dimensionality reduction method (PCA, Kernel PCA, UMAP) and the intermediate dimen˜ sion d(k) ∈ {4, 16, 64} to evaluate the trade-off between analysis accuracy and reconstruction risk. The RBF kernel bandwidth for Kernel PCA is determined by the median heuristic applied to standardized data. The datasets used are MNIST and Fashion-MNIST. Anchor Data and Attacker Model. While RQ1–RQ3 use SMOTE-augmented anchors, RQ4 assumes a stronger attacker and uses real-data anchors exclusively (na = nsmote = 1000). The anchor dataset consists of 100 samples a per label, for a total of 1000 samples; the number of trials per condition is Nseed = 30. The attacker is assumed to have obtained, for the target party k, a set of row indices I ⊂ [na ] of the anchor matrix A along with the corresponding original data AI,: ∈ R|I|×d and their intermediate representations ˜ (k) fk (AI,: ) = ÃI,: ∈ R|I|×d(k) . Specifically, for MNIST we assume that 300 anchor pairs (100 per label) corresponding to digits 0, 1, 2 have been leaked, and we evaluate reconstruction risk for the remaining digits 3–9. For Fashion-MNIST, we assume that 300 anchor pairs (100 per label) corresponding to labels 0 (T-shirt/top), 1 (Trouser), and 2 (Pullover) have been leaked, and we evaluate reconstruction risk for the remaining labels 3–9. This setting is designed to examine whether leakage of a subset of labels enables reconstruction of samples from non-leaked labels. Reconstruction Attack Methods. The attacker is assumed to apply multiple reconstruction attacks and adopt the one that achieves the highest reconstruction accuracy (defined below). All attack methods estimate a recon(k) struction mapping using only the leaked anchor pairs (AI,: , ÃI,: ). We compare the following three methods. 28
• LR: (centered) least-squares linear regression from the intermediate representation to the original data. • MLP: a multilayer perceptron with one hidden layer of 128 units and ReLU activations, optimized with Adam, with a maximum of 600 iterations and early stopping with a 20% validation split. • PINV: (centered) least-squares estimation of the linear mapping from original data to intermediate representations, followed by approximate reconstruction via the pseudoinverse and mean correction. For each condition, we report the maximum reconstruction accuracy among the three methods and the corresponding attack method. Reconstruction Evaluation Metric. Reconstruction accuracy is defined as the label-match rate when reconstructed images are passed to a classifier trained independently on 5000 original data points. A higher value indicates higher reconstruction risk and greater privacy leakage. We use RandomForestClassifier with default settings, the same classifier used for classification accuracy evaluation. Reconstruction accuracy for each condition is computed over 50 reconstructed samples per evaluation label (350 samples in total under this setting). 5.2. Results 5.2.1. RQ1: Comparison with Existing Integration Methods We first evaluate whether the proposed NKI outperforms existing linear integration methods. Fig. 3 shows the classification accuracy as a function of the number of parties K. For comparison, we include the existing methods MPP, GEP, and ODC; the proposed linear integration method LKI; and the non-collaborative baseline Local. Central is also evaluated as an idealized upper bound but is excluded from Fig. 3 to preserve readability. The mean accuracy of Central ranged from 0.680 to 0.941 on MNIST and from 0.656 to 0.844 on Fashion-MNIST as K increased from 2 to 64. As shown in Fig. 3, NKI generally achieved higher classification accuracy than the existing linear integration methods on both MNIST and FashionMNIST in most conditions, with the advantage becoming more pronounced at larger K. In contrast, linear integration methods (MPP, GEP, ODC, and LKI) underperformed NKI in most conditions.
29
(a) MNIST
(b) Fashion-MNIST
Figure 3: Classification accuracy for existing methods in RQ1.
Among the linear integration methods, LKI and GEP performed comparably and tended to rank highest in accuracy, followed by MPP and ODC. The fact that linear integration methods frequently underperformed Local is likely because these methods cannot sufficiently align the nonlinearly transformed intermediate representations produced by UMAP, and the cost of misalignment and information loss from dimensionality reduction outweighs the benefit of integration. These results suggest that, when nonlinear dimensionality reduction is used, employing an integration function that accounts for the nonlinearity of intermediate representations is more effective than directly applying linear integration methods. 5.2.2. RQ2: Contribution of Proposed Extensions For RQ2, we compare the contributions of TSL (which uses target-variable information), GL (which captures geometric structure only), and the centering constraint (Center). Fig. 4 shows the classification accuracy as a function of K for the proposed extensions (GL, TSL, and Center). On both MNIST and Fashion-MNIST, NKI+TSL and NKI+TSL+Center consistently achieved higher accuracy than NKI without extensions. Overall, NKI+TSL+Center tended to rank highest, followed by NKI+TSL. These results suggest that, whereas GL uses only geometry-based neighborhood relationships without considering target-variable information, TSL additionally draws together anchor pairs sharing the same target variable, enabling the collaboration representation to more directly reflect class structure. Furthermore, combining Center removes constant-direction components from the 30
(a) MNIST
(b) Fashion-MNIST
Figure 4: Classification accuracy for the proposed extensions in RQ2.
collaboration representations, which may be one reason why the class structure captured by TSL is reflected more stably. These results suggest that, in settings where classification performance is prioritized, combining TSL with the centering constraint is an effective strategy. 5.2.3. RQ3: Effect of Anchor Data Quantity and Quality on Analytical Performance and Computation Time This section examines the effect of anchor dataset design on classification accuracy and computational cost. We separate anchor dataset design into two aspects: quantity and quality. Quantity refers to the total anchor dataset size na , and quality is approximated by the number of SMOTE source ; a larger number of source samples is expected to produce ansamples nsmote a chor data that more diversely and representatively reflects the original data distribution. Details of experimental conditions follow the Experimental Setup. For = 100 and vary na ; for anchor quality, we fix anchor quantity, we fix nsmote a smote na = 1000 and vary na . In the former setting, we additionally evaluate computation time as a function of na , which directly affects computational cost. Classification accuracy for these settings is shown in Fig. 5. Analytical Performance. First, regarding the effect of anchor quantity, classification accuracy improved overall as na increased. The improvement was particularly large for NKI-based methods (NKI, NKI+TSL), indicating that adding more anchor data is especially beneficial for these methods. Next, regarding the effect of anchor quality, the accuracy improvement from increasing nsmote was more pronounced than that from increasing na . a 31
(a) Anchor size
(b) Anchor quality
Figure 5: Classification accuracy for the anchor-size and anchor-quality experiments on MNIST.
(a) Function construction
(b) Representation construction
Figure 6: Computation time on MNIST as a function of anchor size.
This tendency was especially strong for NKI-based methods, suggesting that these methods are more sensitive to the diversity and representativeness of the SMOTE source data than to the sheer quantity of anchor data. Computation Time. The definition of computation time follows the metrics described in the Experimental Setup. Fig. 6 shows the computation time as a function of na (corresponding to the anchor quantity experiment), displayed on a log-log scale. The left subfigure shows integration-function construction time and the right shows collaboration-representation construction time. Note that because na is fixed in the anchor quality experiment, computation time evaluation applies to the anchor quantity experiment only. To verify correspondence with the theoretical complexity, Table 4 reports the local 32
Table 4: Theoretical and empirical scaling exponents for computation time.
Function Method MPP GEP ODC LKI NKI NKI+TSL
Representation
Theory Empirical Theory Empirical 1 1 1 1 3 3
0.7455 0.8273 0.3528 0.9317 2.8156 2.6308
0 0 0 0 1 1
0.0105 0.0801 0.1152 -0.0248 1.2898 1.2315
log-log slope (empirical scaling exponent) of each computation time curve. According to Table 2, the theoretical complexity for linear integration methods (GEP, MPP, ODC, LKI) is approximately O(na ) for function construction and O(1) for representation construction. NKI-based methods (NKI, NKI+TSL) have theoretical complexity O(n3a ) and O(na ), respectively. Because fixed overhead dominates at small na , we compute the local loglog slope at the large-scale end (na = 800 → 1600). As shown in Table 4, the empirical scaling exponents are broadly consistent with the theoretical complexities. Specifically, linear integration methods (GEP, MPP, ODC, LKI) exhibited sub-linear growth in function construction time and near-zero growth in representation construction time, consistent with the theoretical O(na ) and O(1), respectively. NKI-based methods showed near-cubic growth in function construction time and near-linear growth in representation construction time, consistent with the theoretical O(n3a ) and O(na ). Regarding function construction time, NKI+TSL was consistently slower than NKI due to the additional cost of graph construction. However, as na increased, the shared computation of Sk = (Kk + λIna )−1 became dominant, and the relative gap between NKI and NKI+TSL tended to shrink. Summary of RQ3. These results indicate that increasing na improves classification accuracy but substantially increases computation time, particularly for NKI-based methods. Moreover, even with the total anchor size fixed, inyielded more pronounced accuracy improvements. Therefore, creasing nsmote a in practice, it is important not only to increase the number of anchor data points but also to ensure sufficient diversity and representativeness of the source data used for anchor generation. 33
5.2.4. RQ4: Effect of Dimensionality Reduction Method and Intermediate Dimension on the Accuracy–Reconstruction Trade-off This section presents results when varying the dimensionality reduction ˜ ∈ method (PCA, Kernel PCA, UMAP) and the intermediate dimension d(k) {4, 16, 64}. Because RQ4 evaluates reconstruction risk, we use real-data anchors, assuming a stronger attacker than in RQ1–RQ3. Details of the evaluation setup (attacker model, reconstruction attack methods, and evaluation metrics) are given in Section 5.1.12. Table 5 summarizes the quantitative results for all conditions. Each cell shows the mean value with the 95% confidence interval in parentheses. For analysis accuracy, the highest value among {LKI, NKI, NKI+TSL} in each dataset–intermediate-dimension–dimensionality-reduction combination is shown in bold. For reconstruction accuracy, the lowest value among {PCA, Kernel PCA, UMAP} in each dataset–intermediate-dimension combination is shown in bold. Ties are all shown in bold. Attack denotes the reconstruction method achieving the highest reconstruction accuracy for each condition. In Table 5, K-PCA denotes Kernel PCA. As shown in Table 5, a trade-off tendency between analysis accuracy and reconstruction risk is observed. Analysis accuracy tended to be highest for PCA, followed by Kernel PCA and UMAP; reconstruction risk generally ˜ followed a similar ordering. As the intermediate dimension d(k) increased, analysis accuracy improved, but reconstruction risk also increased for PCA and Kernel PCA. In contrast, under UMAP, reconstruction risk remained relatively low even in conditions where analysis accuracy improved, suggesting that UMAP provides a favorable balance between utility and privacy. This ˜ = 64 tendency is also visible in the qualitative reconstruction results at d(k) shown in Fig. 7. In Fig. 7, the first row for each dataset shows original images and subsequent rows show the best reconstruction results for each dimensionality reduction method; the attack method achieving the highest reconstruction accuracy is shown in parentheses. For Fashion-MNIST, reconstructed images tend to resemble specific training samples rather than the original input. This may be because UMAP constructs a latent space that separates clusters, making it easier for the inverse mapping to pull an input point toward typical samples in the nearest learned cluster. Comparing integration methods, under the no-regularization condition (NKI vs. LKI), LKI outperformed NKI under PCA, whereas NKI outperformed LKI under the nonlinear transformations Kernel PCA and UMAP.
34
This result further suggests the effectiveness of NKI for nonlinear intermediate representations. Adding TSL regularization improved accuracy in most conditions, with particularly large gains under PCA where NKI+TSL approached or surpassed LKI. One possible reason is that PCA relatively well preserves distance relationships, enabling the graph Laplacian based on anchor intermediate representations to function effectively. Regarding the reconstruction attack methods, PINV tended to be most effective under PCA, whereas LR was most effective in many conditions under Kernel PCA and UMAP. This tendency suggests that, under PCA, the obfuscation function fk of party k can be approximated by a linear mapping matrix F (k) as X̃ (k) ≃ X (k) F (k) , making reconstruction via the pseudoinverse (F (k) )† effective. In contrast, since the mappings in Kernel PCA and UMAP are nonlinear, pseudoinverse reconstruction based on a single linear mapping is less effective. These results demonstrate that the choice of dimensionality reduction method and intermediate dimension substantially affects both analysis accuracy and reconstruction risk. Therefore, in practice, it is important to select the dimensionality reduction method and intermediate dimension not solely based on analysis accuracy but by also considering reconstruction risk within the acceptable performance range.
35
Table 5: Analysis and reconstruction results under different dimensionality reduction methods. Analysis accuracy Dataset
˜ d(k)
DimRed PCA
4 K-PCA UMAP
MNIST
PCA 16 K-PCA UMAP PCA 64 K-PCA UMAP PCA 4 K-PCA UMAP
Fashion-MNIST
PCA 16 K-PCA UMAP PCA 64 K-PCA UMAP
Reconstruction accuracy
LKI
NKI
NKI+TSL
Score
.408 (±.007) .346 (±.006) .349 (±.006)
.336 (±.011) .320 (±.006) .425 (±.008)
.394 (±.008) .329 (±.005) .457 (±.008)
.267 (±.010) .188 (±.007) .143 (±.000)
.755 (±.007) .636 (±.007) .512 (±.007)
.517 (±.012) .673 (±.006) .567 (±.005)
.824 (±.004) .685 (±.006) .594 (±.005)
.632 (±.007) .540 (±.014) .159 (±.017)
.751 (±.013) .635 (±.014) .554 (±.007)
.673 (±.008) .717 (±.009) .586 (±.005)
.859 (±.004) .753 (±.006) .598 (±.005)
.834 (±.005) .684 (±.010) .153 (±.021)
.538 (±.008) .476 (±.008) .416 (±.005)
.502 (±.005) .484 (±.007) .509 (±.008)
.573 (±.004) .489 (±.007) .527 (±.007)
.211 (±.007) .142 (±.007) .151 (±.029)
.721 (±.006) .664 (±.007) .544 (±.008)
.631 (±.007) .681 (±.007) .598 (±.007)
.709 (±.005) .684 (±.006) .607 (±.006)
.557 (±.011) .326 (±.011) .220 (±.033)
.725 (±.010) .656 (±.012) .575 (±.007)
.672 (±.007) .709 (±.008) .609 (±.007)
.720 (±.006) .717 (±.007) .617 (±.007)
.687 (±.015) .454 (±.013) .189 (±.021)
36
Attack PINV LR PINV PINV LR LR PINV LR LR PINV LR MLP PINV LR LR PINV LR LR
˜ = 64. Figure 7: Qualitative reconstruction results for MNIST and Fashion-MNIST at d(k)
37
6. Conclusion In this study, we addressed the integration problem in DC analysis by first formulating a linear integration problem, characterizing its globally optimal solution, and then extending it to a nonlinear setting through kernelization. This nonlinear extension enables the integration of intermediate representations obtained through nonlinear dimensionality reduction. In the proposed method, the estimation of each integration function for a fixed target representation is reduced to a finite-dimensional kernel ridge regression-type problem, and the estimation of the target representation is reduced to an eigenvalue problem. This formulation makes nonlinear integration tractable through eigendecomposition. We also introduced graph regularization and a centering constraint to incorporate geometric structure and target-variable information of the anchor dataset into the target representation. Numerical experiments showed that, under nonlinear dimensionality reduction, the proposed NKI-based methods tended to achieve higher analytical performance than existing linear integration methods. The results also showed that combining TSL using target-variable information with the centering constraint improved classification performance, and that not only the quantity of anchor data but also the distributional characteristics of the source data used to generate anchors substantially affected analytical performance. Furthermore, the choice of dimensionality reduction method and intermediate dimension affected both analytical performance and reconstruction risk. In particular, the results suggest that UMAP provides a favorable balance between analytical performance and reconstruction risk. These results indicate that, in practice, integration and dimensionality reduction methods should be selected by jointly considering analytical accuracy, anchor data design, computational cost, and reconstruction risk. Future work includes extending the proposed method to large-scale datasets using kernel approximation and constructing a DC framework that can adjust dimensionality reduction methods and intermediate dimensions according to party-specific confidentiality and accuracy requirements. Additionally, evaluating the privacy guarantees of the proposed method against stronger reconstruction attack models remains an important open problem.
38
Appendix A. Proof of Theorem 1 Proof. Firstly, we fix Z. Then, the inner minimization in Eq. (12) with respect to G(k) can be written as follows: min Ã(k) G(k) − Z G(k)
2
(A.1) F
for each k ∈ [K]. Eq. (A.1) is a least-squares problem, and its minimum-norm solution is given by G(k)⋆ = (Ã(k) )† Z, (A.2) where (·)† denotes the Moore–Penrose pseudoinverse. Using Eq. (A.2), we obtain (A.3) Ã(k) G(k)⋆ = P (k) Z, P (k) := Ã(k) (Ã(k) )† , where P (k) is the orthogonal projection matrix onto Col(Ã(k) ). Substituting Eq. (A.3) into Eq. (12), the problem with respect to Z can be rewritten as K X
min
2
(Ina − P (k) )Z F
k=1
(A.4)
s. t. Z ⊤ Z = Idˆ, ˆ
Z ∈ Rn a × d . Since P (k) is an orthogonal projection matrix, it satisfies P (k)⊤ = P (k) and (P (k) )2 = P (k) . Therefore, the objective function in Eq. (A.4) can be transformed as follows: K X k=1
(Ina − P
(k)
2
)Z F =
K X
tr Z ⊤ (Ina − P (k) )Z
k=1
= K dˆ − tr Z ⊤
K X
! P (k)
! Z
(A.5) .
k=1
Here, the first term K dˆ does not depend on Z. As a result, Eq. (A.4) can
39
be reformulated as the following maximization problem: ! ! K X P (k) Z max tr Z ⊤ k=1
(A.6)
s. t. Z ⊤ Z = Idˆ, ˆ
Z ∈ Rn a ×d . Now, let Q(k) be an orthonormal basis for Col(Ã(k) ). Then, P (k) = Q(k) Q(k)⊤ ,
(A.7)
and therefore, K X
P (k) = WQ WQ⊤ ,
WQ = [Q(1) , . . . , Q(K) ].
(A.8)
k=1
Substituting Eq. (A.8) into Eq. (A.6), Eq. (12) can be reduced to max tr Z ⊤ WQ WQ⊤ Z s. t. Z ⊤ Z = Idˆ,
(A.9)
ˆ
Z ∈ Rn a ×d . By the Ky Fan maximum principle, the maximum value of Eq. (A.9) is given by the sum of the dˆ largest eigenvalues of WQ WQ⊤ . Consequently, a globally optimal solution to Eq. (A.9) can be obtained from an orthonormal basis of the eigenspace associated with these eigenvalues. The left singular vectors of WQ can be chosen as eigenvectors of WQ WQ⊤ . Let UQdˆ be a matrix whose columns are the dˆ selected left singular vectors of WQ . Then, ˆ Z ⋆ = U ˆO, O ∈ O(d), (A.10) Qd
gives a globally optimal solution to Eq. (12). Here, the arbitrary orthogonal matrix O represents the rotational indeterminacy of the target representaˆ selected singular value is separated from the next one, tion. When the d-th this form characterizes all globally optimal Z ⋆ up to an orthogonal rotation. Finally, the corresponding coefficient matrix is obtained by substituting Eq. (A.10) into the minimum-norm solution in Eq. (A.2) as follows: G(k)⋆ = (Ã(k) )† Z ⋆ = (Ã(k) )† UQdˆO.
(A.11)
This is the minimum-norm solution of the inner least-squares problem for the fixed Z ⋆ . 40
Appendix B. Proof of Theorem 2 ˆ
Proof. Firstly, we fix Z ∈ Rna ×d . Then, the inner minimization in Eq. (16) can be written as the following multi-output kernel ridge regression-type problem for the fixed Z: min
K X
ˆ
{gk ∈Hkd }K k=1
kgk (Ã(k) ) − Zk2F + λkgk k2Hdˆ .
(B.1)
k
k=1
Eq. (B.1) is separable with respect to k ∈ [K]. Therefore, each subproblem ˆ can be solved independently. Since Hkd is a product RKHS, the representer theorem applies coordinate-wise to each component of gk . By the representer theorem [57], an optimal solution for each k ∈ [K] can be represented using ˆ a coefficient matrix Γk ∈ Rna ×d as follows: gk⋆ (x) = κk (x)Γk .
(B.2)
Using this representation, the function values and the RKHS norm are written as follows: gk⋆ (Ã(k) ) = Kk Γk ,
kgk⋆ k2Hdˆ = tr(Γ⊤ k Kk Γk ).
(B.3)
k
Consequently, by Eqs. (B.2) and (B.3), Eq. (B.1) is decomposed into the following matrix optimization problem for each k ∈ [K]: min
Γk
∈Rna ×dˆ
kKk Γk − Zk2F + λ tr(Γ⊤ k Kk Γk ).
(B.4)
Next, to solve Eq. (B.4), we compute the first-order condition with respect to Γk . The optimality condition can be written as follows: ∇Γk kKk Γk − Zk2F + λ tr(Γ⊤ k Kk Γk ) = 0 ⇐⇒ Kk {(Kk + λIna )Γk − Z} = 0. (B.5) Since Kk is positive semidefinite and λ > 0, Kk + λIna is positive definite and hence invertible. Therefore, we define Sk := (Kk + λIna )−1
(B.6)
and consider the following candidate solution: Γ⋆k = Sk Z. 41
(B.7)
This candidate solution satisfies Eq. (B.5), as shown by Eqs. (B.6) and (B.7): Kk {(Kk + λIna )Γ⋆k − Z} = Kk {(Kk + λIna )Sk Z − Z} = Kk (Z − Z) (by Eq. (B.6)) = 0. Therefore, Eq. (B.7) satisfies Eq. (B.5). Moreover, because Kk is positive semidefinite and λ > 0, the objective function in Eq. (B.4) is a differentiable convex function with respect to Γk . Therefore, Γ⋆k satisfying the first-order condition in Eq. (B.5) is a globally optimal coefficient matrix. Consequently, ˜ by Eqs. (B.2) and (B.7), for any x ∈ R1×d(k) , we obtain gk⋆ (x) = κk (x)Sk Z.
(B.8)
Next, we compute the optimal objective value for the fixed Z using this solution. Since Kk is symmetric positive semidefinite, Sk is also symmetric. From Eq. (B.6), the following identity holds: Kk Sk − Ina = −λSk .
(B.9)
Therefore, by Eqs. (B.7) and (B.9), we have Kk Γ⋆k − Z = (Kk Sk − Ina )Z = −λSk Z.
(by Eq. (B.7))
(B.10)
Using Eq. (B.10), the optimal objective value of Eq. (B.4) can be computed as follows: kKk Γ⋆k − Zk2F + λ tr(Γ⋆k ⊤ Kk Γ⋆k ) = λ2 tr(Z ⊤ Sk2 Z) + λ tr(Z ⊤ Sk Kk Sk Z) (by Eqs. (B.7) and (B.10)) = λ tr Z ⊤ Sk (λIna + Kk )Sk Z = λ tr(Z ⊤ Sk Z) (by Eq. (B.6)). (B.11) Summing Eq. (B.11) over k ∈ [K] yields K X
kKk Γ⋆k − Zk2F + λ tr(Γ⋆k ⊤ Kk Γ⋆k )
k=1
= tr Z ⊤ λ
K X
! Sk Z
k=1
42
(B.12) = tr(Z ⊤ Mλ Z).
As a result, Eq. (16) reduces to the following problem with respect to Z: min tr(Z ⊤ Mλ Z) s. t. Z ⊤ Z = Idˆ, ˆ
Z ∈ Rn a × d . This coincides with Eq. (20). The corresponding function gk⋆ is given by Eq. (B.8), which is identical to Eq. (21). Appendix C. Additional Regression Evaluation To examine whether the proposed methods are also effective for regression tasks, we conducted additional experiments using UJIIndoorLoc [58] and Relative location of CT slices on axial axis [59] (hereafter Slice localization) from the UCI Machine Learning Repository. For UJIIndoorLoc, we used only latitude as the target variable and treated the task as a singleoutput regression problem. For each dataset, we compared dimensionality reduction methods (PCA, Kernel PCA, and UMAP) and integration methods (NKI, NKI+GL, NKI+TSL, LKI, GEP, MPP, and ODC). Except for the datasets, prediction target, and regression metric, the experimental conditions were the same as those in the main experiments. As the regression model, we used RandomForestRegressor with default settings, corresponding to the Random Forest classifier used in the classification experiments. Both explanatory and target variables were standardized to zero mean and unit variance before evaluation. Accordingly, the scale parameter in the TSL weight in Eq. (33) was set to σy2 = 1. Each condition was repeated 20 times, and Table C.6 reports the mean RMSE and 95% confidence interval. In each cell, the mean value is shown in the upper row and the 95% confidence interval is shown in parentheses in the lower row. Bold values indicate the lowest RMSE among the integration methods shown in the table; ties are all shown in bold. For reference, Local and Central achieved RMSEs of 13.02 (±0.08) and 6.41 (±0.07) on Slice localization, and 28.54 (±0.25) and 13.43 (±0.26) on UJIIndoorLoc, respectively. In Table C.6, K-PCA denotes Kernel PCA.
43
Table C.6: Regression results for the additional evaluation. Dataset
DimRed
Slice localization
PCA
Slice localization
K-PCA
Slice localization
UMAP
UJIIndoorLoc
PCA
UJIIndoorLoc
K-PCA
UJIIndoorLoc
UMAP
NKI
NKI+GL
NKI+TSL
LKI
GEP
MPP
ODC
14.62 (±0.40) 9.76 (±0.20) 12.18 (±0.28) 27.93 (±0.96) 43.73 (±1.05) 23.20 (±0.53)
13.88 (±0.56) 9.73 (±0.25) 12.05 (±0.29) 25.11 (±0.91) 44.63 (±1.36) 22.85 (±0.48)
11.75 (±0.41) 9.47 (±0.24) 12.10 (±0.24) 24.47 (±0.83) 43.77 (±0.87) 22.95 (±0.52)
8.91 (±0.25) 9.92 (±0.26) 12.90 (±0.23) 18.04 (±0.36) 45.05 (±2.00) 26.53 (±0.63)
8.91 (±0.25) 9.92 (±0.26) 12.91 (±0.23) 18.04 (±0.36) 45.05 (±2.00) 26.61 (±0.66)
8.95 (±0.23) 9.77 (±0.30) 13.01 (±0.31) 18.11 (±0.34) 50.15 (±1.77) 27.34 (±0.76)
8.96 (±0.21) 9.83 (±0.26) 12.89 (±0.24) 18.04 (±0.24) 46.49 (±1.30) 27.45 (±0.76)
As shown in Table C.6, under nonlinear transformations using Kernel PCA and UMAP, NKI-based methods achieved RMSE comparable to or lower than that of linear integration methods (LKI, GEP, MPP, and ODC) in many cases. In particular, under UMAP, NKI-based methods achieved lower RMSE than linear integration methods on both datasets. For Slice localization under PCA, however, linear integration methods (LKI and GEP) substantially outperformed NKI, suggesting that NKI is most beneficial when the intermediate representations are nonlinear, whereas linear integration can remain preferable under linearly structured representations. Overall, these results suggest that nonlinear integration can also be effective for regression tasks with nonlinear intermediate representations, which is consistent with the trend observed in the classification experiments. References [1] B. Liu, M. Ding, S. Shaham, W. Rahayu, F. Farokhi, Z. Lin, When machine learning meets privacy: A survey and outlook, ACM Computing Surveys (CSUR) 54 (2) (2021) 1–36. [2] B. C. M. Fung, K. Wang, R. Chen, P. S. Yu, Privacy-preserving data publishing: A survey of recent developments, ACM Comput. Surv. 42 (4) (Jun. 2010). [3] C. J. Hoofnagle, B. van der Sloot, F. Z. Borgesius, The European Union general data protection regulation: What it is and what it means, Information & Communications Technology Law 28 (1) (2019) 65–98. 44
[4] R. Xu, N. Baracaldo, J. Joshi, Privacy-preserving machine learning: Methods, challenges and directions, arXiv preprint arXiv:2108.04417 (2021). [5] C. Dwork, A. Roth, The algorithmic foundations of differential privacy, Found. Trends Theor. Comput. Sci. 9 (34) (2014) 211407. [6] C. Dwork, Differential privacy: A survey of results, in: International Conference on Theory and Applications of Models of Computation, Springer, 2008, pp. 1–19. [7] B. Balle, Y.-X. Wang, Improving the Gaussian mechanism for differential privacy: Analytical calibration and optimal denoising, in: Proceedings of the 35th International Conference on Machine Learning, Vol. 80 of Proceedings of Machine Learning Research, PMLR, 2018, pp. 394– 403. [8] A. C. Yao, Protocols for secure computations, in: 23rd Annual Symposium on Foundations of Computer Science (SFCS 1982), IEEE, 1982, pp. 160–164. [9] P. Martins, L. Sousa, A. Mariano, A survey on fully homomorphic encryption: An engineering perspective, ACM Computing Surveys (CSUR) 50 (6) (2017) 1–33. [10] A. Acar, H. Aksu, A. S. Uluagac, M. Conti, A survey on homomorphic encryption schemes: Theory and implementation, ACM Computing Surveys (CSUR) 51 (4) (2018) 1–35. [11] B. McMahan, E. Moore, D. Ramage, S. Hampson, B. A. y Arcas, Communication-efficient learning of deep networks from decentralized data, in: Artificial Intelligence and Statistics, PMLR, 2017, pp. 1273– 1282. [12] J. Konečnỳ, H. B. McMahan, F. X. Yu, P. Richtárik, A. T. Suresh, D. Bacon, Federated learning: Strategies for improving communication efficiency, arXiv preprint arXiv:1610.05492 (2016). [13] Q. Li, Z. Wen, Z. Wu, S. Hu, N. Wang, Y. Li, X. Liu, B. He, A survey on federated learning systems: Vision, hype and reality for data privacy
45
and protection, IEEE Transactions on Knowledge and Data Engineering 35 (4) (2023) 3347–3366. [14] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings, et al., Advances and open problems in federated learning, Foundations and Trends® in Machine Learning 14 (1–2) (2021) 1–210. [15] A. Imakura, T. Sakurai, Data collaboration analysis framework using centralization of individual intermediate representations for distributed data sets, ASCE-ASME Journal of Risk and Uncertainty in Engineering Systems, Part A: Civil Engineering 6 (2) (2020) 04020018. [16] A. Imakura, X. Ye, T. Sakurai, Collaborative data analysis: Non-model sharing-type machine learning for distributed data, in: Knowledge Management and Acquisition for Intelligent Systems, Springer, 2021, pp. 14–29. [17] X. Wang, S. Ranellucci, J. Katz, Global-scale secure multiparty computation, in: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, 2017, pp. 39–56. [18] A. Ben-Efraim, Y. Lindell, E. Omri, Optimizing semi-honest secure multiparty computation for the internet, in: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, 2016, pp. 578–590. [19] A. Gascon, P. Schoppmann, B. Balle, M. Raykova, J. Doerner, S. Zahur, D. Evans, Privacy-preserving distributed linear regression on highdimensional data, Cryptology ePrint Archive, Paper 2016/892 (2016). [20] B. D. Rouhani, M. S. Riazi, F. Koushanfar, Deepsecure: Scalable provably-secure deep learning, Cryptology ePrint Archive, Paper 2017/502 (2017). [21] Q. Li, Y. Diao, Q. Chen, B. He, Federated learning on non-iid data silos: An experimental study, in: 2022 IEEE 38th International Conference on Data Engineering (ICDE), IEEE, 2022, pp. 965–978. [22] X. Li, K. Huang, W. Yang, S. Wang, Z. Zhang, On the convergence of fedavg on non-iid data, arXiv preprint arXiv:1907.02189 (2019). 46
[23] T. Li, A. K. Sahu, A. Talwalkar, V. Smith, Federated learning: Challenges, methods, and future directions, IEEE Signal Processing Magazine 37 (3) (2020) 50–60. [24] L. Zhu, Z. Liu, S. Han, Deep leakage from gradients, in: Advances in Neural Information Processing Systems, Vol. 32, 2019, pp. 14774–14784. [25] K. Bonawitz, V. Ivanov, B. Kreuter, A. Marcedone, H. B. McMahan, S. Patel, D. Ramage, A. Segal, K. Seth, Practical secure aggregation for privacy-preserving machine learning, in: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, 2017, pp. 1175–1191. [26] R. Mendes, J. P. Vilela, Privacy-preserving data mining: Methods, metrics, and applications, IEEE Access 5 (2017) 10562–10582. [27] E. Bingham, H. Mannila, Random projection in dimensionality reduction: Applications to image and text data, in: Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2001, pp. 245–250. [28] K. Chen, L. Liu, Privacy preserving data classification with rotation perturbation, in: Fifth IEEE International Conference on Data Mining (ICDM’05), 2005, pp. 589–592. [29] K. Liu, H. Kargupta, J. Ryan, Random projection-based multiplicative data perturbation for privacy preserving distributed data mining, IEEE Transactions on Knowledge and Data Engineering 18 (1) (2006) 92–106. [30] K. Liu, C. Giannella, H. Kargupta, A survey of attack techniques on privacy-preserving data perturbation methods, in: C. C. Aggarwal, P. S. Yu (Eds.), Privacy-Preserving Data Mining: Models and Algorithms, Springer US, Boston, MA, 2008, pp. 359–381. [31] L. Jiang, R. Tan, X. Lou, G. Lin, On lightweight privacy-preserving collaborative learning for internet-of-things objects, in: Proceedings of the International Conference on Internet of Things Design and Implementation, 2019, pp. 70–81.
47
[32] A. Imakura, A. Bogdanova, T. Yamazoe, K. Omote, T. Sakurai, Accuracy and privacy evaluations of collaborative data analysis, arXiv preprint arXiv:2101.11144 (2021). [33] H. Yamashiro, K. Omote, A. Imakura, T. Sakurai, Toward the application of differential privacy to data collaboration, IEEE Access 12 (2024) 63292–63301. [34] A. Imakura, T. Sakurai, FedDCL: a federated data collaboration learning as a hybrid-type privacy-preserving framework based on federated learning and data collaboration, arXiv preprint arXiv:2409.18356 (2024). [35] Y. Kawamata, R. Motai, Y. Okada, A. Imakura, T. Sakurai, Collaborative causal inference on distributed data, Expert Systems with Applications 244 (2024) 123024. [36] T. Nakayama, Y. Kawamata, A. Toyoda, A. Imakura, R. Kagawa, M. Sanuki, R. Tsunoda, K. Yamagata, T. Sakurai, Y. Okada, Data collaboration for causal inference from limited medical testing and medication data, Scientific Reports 15 (1) (2025) 9827. [37] A. Imakura, R. Tsunoda, R. Kagawa, K. Yamagata, T. Sakurai, DCCOX: Data collaboration cox proportional hazards model for privacypreserving survival analysis on multiple parties, Journal of Biomedical Informatics 137 (2023) 104264. [38] A. Bogdanova, A. Imakura, T. Sakurai, DC-SHAP method for consistent explainability in privacy-preserving distributed machine learning, Human-Centric Intelligent Systems 3 (3) (2023) 197–210. [39] T. Yanagi, S. Ikeda, N. Sukegawa, Y. Takano, Privacy-preserving recommender system using the data collaboration analysis for distributed datasets, PLoS ONE 20 (4) (2025) e0319954. [40] Y. Kawamata, K. Kamijo, M. Kihira, A. Toyoda, T. Nakayama, A. Imakura, T. Sakurai, Y. Okada, A new type of federated clustering: A non-model-sharing approach (2025). [41] A. Imakura, T. Sakurai, Y. Okada, T. Fujii, T. Sakamoto, H. Abe, Non-readily identifiable data collaboration analysis for multiple datasets including personal information, Information Fusion 98 (2023) 101826. 48
[42] A. Bogdanova, A. Nakai, Y. Okada, A. Imakura, T. Sakurai, Federated learning system without model sharing through integration of dimensional reduced data representations (2020). [43] A. Imakura, H. Inaba, Y. Okada, T. Sakurai, Interpretable collaborative data analysis on distributed data, Expert Systems with Applications 177 (2021) 114891. [44] K. Nosaka, Y. Suetake, Y. Takano, A. Yoshise, Data collaboration analysis with orthonormal basis selection and alignment, Computers and Electrical Engineering 135 (2026) 111192. [45] Y. Kawakami, Y. Takano, A. Imakura, New solutions based on the generalized eigenvalue problem for the data collaboration analysis, Information Sciences 723 (2025) 122642. [46] S. Mashiko, Y. Kawamata, T. Nakayama, T. Sakurai, Y. Okada, Anomaly detection in double-entry bookkeeping data by federated learning system with non-model sharing approach, Scientific Reports 15 (1) (2025) 42208. [47] G. E. Hinton, R. R. Salakhutdinov, Reducing the dimensionality of data with neural networks, Science 313 (5786) (2006) 504507. [48] K. Fan, On a theorem of Weyl concerning eigenvalues of linear transformations I, Proceedings of the National Academy of Sciences 35 (11) (1949) 652–655. [49] S. Yan, D. Xu, B. Zhang, H.-J. Zhang, Q. Yang, S. Lin, Graph embedding and extensions: A general framework for dimensionality reduction, IEEE Transactions on Pattern Analysis and Machine Intelligence 29 (1) (2007) 40–51. [50] M. Belkin, P. Niyogi, Laplacian eigenmaps for dimensionality reduction and data representation, Neural Computation 15 (6) (2003) 13731396. [51] X. He, P. Niyogi, Locality preserving projections, in: Advances in Neural Information Processing Systems, Vol. 16, 2003, pp. 153–160. [52] L. Deng, The mnist database of handwritten digit images for machine learning research [best of the web], IEEE signal processing magazine 29 (6) (2012) 141–142. 49
[53] H. Xiao, K. Rasul, R. Vollgraf, Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms, arXiv preprint arXiv:1708.07747 (2017). [54] L. McInnes, J. Healy, J. Melville, UMAP: Uniform manifold approximation and projection for dimension reduction (2020). [55] A. Imakura, M. Kihira, Y. Okada, T. Sakurai, Another use of smote for interpretable data collaboration analysis, Expert Systems with Applications 228 (2023) 120385. [56] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, et al., Scikit-learn: Machine learning in python, Journal of Machine Learning Research 12 (2011) 2825–2830. [57] B. Schölkopf, R. Herbrich, A. J. Smola, A generalized representer theorem, in: Proceedings of the 14th Annual Conference on Computational Learning Theory and 5th European Conference on Computational Learning Theory, 2001, pp. 416–426. [58] J. Torres-Sospedra, R. Montoliu, A. Martínez-Usó, T. Arnau, J. Avariento, UJIIndoorLoc, UCI Machine Learning Repository (2014). [59] F. Graf, H.-P. Kriegel, M. Schubert, S. Pölsterl, A. Cavallaro, Relative location of CT slices on axial axis, UCI Machine Learning Repository (2011).
50