1
Fourier Preconditioning for Neural Feature Learning
arXiv:2607.02199v1 [eess.SP] 2 Jul 2026
Preston Pitzer, Anish Pradhan, Member, IEEE, Harpreet S. Dhillon, Fellow, IEEE Abstract—Mutual information (MI)-inspired feature learning techniques are capable of generating low-dimensional embeddings that retain nonlinear dependence structures, but direct estimations of MI suffer from noisy probability distribution estimates in the low-data regime. The H-Score objective, computed from second-order statistics, provides a practical proxy metric for training feature extraction networks. We prove that H-Score is invariant to invertible transformations in the unrestricted functional setting, but becomes sensitive to input basis rotations under constrained approximation classes. Consequently, we study unitary preconditioning for H-Score networks and show that selecting an appropriate basis rotation reduces finite-width truncation error by concentrating predictive dependence into fewer dominant modes. We identify the fast Fourier transform (FFT) as an effective data-independent, low-cost preconditioner for approximately stationary processes, where spectral structure induces concentration of the cross-covariance singular value spectrum. We introduce training-free metrics based on spectral entropy and cumulative dependence energy to quantify basis suitability and predict downstream inference gains prior to network training. Experiments across eight multivariate datasets demonstrate that FFT preconditioning is particularly useful in resource-constrained regimes, achieving up to 50% normalized mean squared error (NMSE) reduction, while the proposed metrics correlate with observed performance gains and correctly identify cases where spectral preconditioning is detrimental. Index Terms—Machine Learning, Hilbert Space, Fourier Transforms, Open RAN, Eigenvalues and Eigenfunctions
I. I NTRODUCTION
M
ACHINE learning has enabled the study of complex nonlinear relationships across high-dimensional data. However, practical hardware limitations necessitate the use of low-dimensional embedding techniques that retain dependence structures while reducing computational cost. Feature extraction networks based on the MI-inspired H-Score objective use second-order statistics to generate such embeddings [1]. Theoretically, we show these objectives are invariant to invertible transformations of the input space. Practically, finite-width network effects and limited training data introduce approximation constraints that make the learned representation sensitive to the input basis. Existing work, which typically treats the input basis as fixed, focuses on network architecture rather than the structure of input data [2]. We propose a novel preconditioning step for H-Score networks that minimizes error associated with these finite-width effects by compacting dependence structure into a minimal set of modes. Although principal component analysis (PCA) and canonical correlation analysis (CCA) The authors are with Wireless@VT, Department of ECE, Virginia Tech, Blacksburg, VA. Email: {n3kos, pradhananish1, hdhillon}@vt.edu. This material is based upon work supported by the National Institute of Standards and Technology (NIST), U.S. Department of Commerce, through the Public Wireless Supply Chain Innovation Fund administered by the National Telecommunications and Information Administration (NTIA), under award number 51-60-IF012. Any opinions, findings, conclusions, or recom- mendations expressed in this material are those of the authors and do not necessarily reflect the views of NIST, NTIA, or the U.S. Department of Commerce
achieve optimal linear compaction under ideal covariance estimation, empirical covariance matrices are unreliable under data constraints. For approximately stationary processes, the FFT produces an approximately diagonal covariance structure, concentrating predictive dependence into a smaller number of spectral modes. Rather than assuming this property holds for the input data, we define metrics based on spectral entropy of the cross-covariance spectrum that allow for principled a priori determination of input basis, rather than heuristic selection. A. Prior Art The challenge of extracting informative features from highdimensional data has historically been addressed through linear methods like PCA and CCA [3]. While these methods provide optimal basis rotations for linearly dependent processes, they rely on empirical covariance estimates that become unreliable and often noise-dominated in resourceconstrained regimes [4]. Furthermore, they struggle to capture the nonlinear dependencies of interest in complex modern datasets. To overcome these limitations, information-theoretic representation learning has emerged, leveraging surrogate objectives like the H-Score to capture nonlinear relationships through neural mappings [5], [6]. However, the transition from classical linear algebra to neural learning introduces a new bottleneck: truncation error. Because H-Score networks approximate the Hilbert-Schmidt decomposition using finitewidth layers, they are most effective when the underlying dependence energy is concentrated into a small number of dominant modes. Prior work has noted that MI estimation is significantly more efficient when dependence structures are spectrally localized or when the underlying process is approximately stationary [7], [8]. While MI in frequency has been shown to be an accurate and efficient estimator [8]–[10], its role as a structural preconditioner to alleviate truncation error in feature learning networks remains largely unexplored. B. Contributions We show that although the H-Score objective is theoretically invariant to invertible transformations, finite-width approximation induces basis-dependent truncation error. This motivates the study of input preconditioning for practical feature learning. We identify the FFT as an effective fixed preconditioner for processes that are approximately stationary. To evaluate if a dataset is well suited to FFT preconditioning, we introduce training-free spectral metrics derived from the singular value spectrum of empirical cross-covariance operators, enabling quantitative prediction of basis effects on downstream inference performance. We empirically validate the utility of this approach by measuring inference accuracy effects on FFTpreconditioned and unconditioned datasets.
2
II. H-S CORE I N F REQUENCY We first briefly describe how the H-Score objective follows from the goal of maximizing MI between two random variables, by maximizing the captured Hilbert-Schmidt (HS) dependence energy in the low-rank approximation of their true HS decomposition, the Canonical Dependence Kernel (CDK). We then describe how taking the Fourier transform of the input data can modify the structure of the cross-covariance, which leads the network to learn more informative dependence modes in the frequency domain.
The proof follows from bijectivity of composition. Because T is invertible and F is closed under composition with T and T −1 , the map f → f ◦ T is a bijection on F. Hence the supremum over F is invariant under a change of variables X → TX Remark 1. This invariance does not hold practically for restricted function classes, like those that appear in finitewidth neural networks, which explains empirically observed performance differences.
A. A Computable Approximation of Mutual Information
C. Basis Transformation and Low-Rank Approximation
Consider two random variables X and Y , which for the sake of inference may be considered as current and future data respectively. We compute point-wise MI (PMI) as the ratio of their joint distribution PXY and their marginal distributions PX , P Y : PXY (x, y) . (1) PMI(x, y) := log2 PX (x)PY (y)
We now formalize the restriction finite-width neural networks impose on the HS Decomposition. First, let HX = L2 (PX ) and HY = L2 (PY ) be the Hilbert spaces of squareintegrable functions of X and Y . The CDK iX;Y is a member of tensor product space HX ⊗ HY . Then the approximation manifold Fk is:
These distributions are typically intractable to compute directly, so we instead look to the modal decomposition of MI as the sum of orthogonal dependence modes [11], which follows from the Schmidt decomposition of Hilbert spaces generated from PX , PY (assuming square integrability and absolute continuity of PX PY ) ! K X ∗ ∗ σi fi (x)gi (y) . (2) PX,Y (x, y) = PX (x)PY (y) 1 + i=1
From this decomposition we characterize the statistical dependence between X and Y using the CDK iX;Y [1]: iX;Y =
K X i=1
σi (fi∗ ⊗ gi∗ ),
(3)
where ∥iX;Y ∥HS = 0 implies independence of X and Y . Just as the true distributions on X and Y are incomputable, the true full-rank decomposition of iX;Y is equivalently unknowable. We then seek a metric for computing the approximation error of a lower-rank subset of orthogonal modes. This metric, proposed in [12] is known as H-Score (H) 1 H(f, g) := E[f ⊺ (X)g(Y )]−(E[f (X)])⊺ E[g(Y )]− ·tr(Λf Λg ) 2 Λf = E[f (X)f ⊺ (X)] Λg = E[g(Y )g ⊺ (Y )]. This metric has the advantage of being computable with training data, since the expected value can be computed as the empirical mean [1]. B. Objective Equivalence We observe the effect (or lack thereof) of preconditioning input data on the H-Score objective in the full-rank case. Theorem 1. Let T : Rdx → Rdx be invertible. Suppose the function class F is closed under composition with T and T −1 . Then sup H(f (X), g(Y )) = sup H(f (T X), g(Y )).
f,g∈F
f,g∈F
(4)
Fk :=
( k X i=1
) wi (ψi ⊗ ϕi ) | ψi ∈ VX , ϕi ∈ VY , wi ∈ R , (5)
where VX ⊂ HX and VY ⊂ HY are the functional subspaces representable by the finite-width network. To understand the effect of basis rotations on decompositions that lie within the manifold, we first consider their effect on the original tensorproduct space. Let X ∈ RL×dx and Y ∈ RL×dy be centered random variables with cross-covariance CXY = E[X ⊺ Y ]. Consider a generic unitary transformation U ∈ CL×L acting on the input space such that XU = U X. The transformed cross-covariance follows as CXU Y = E[(U X)⊺ Y ] = U CXY .
(6)
Because U is unitary, the HS norm remains invariant, which implies that the total dependence energy is preserved. However, the alignment of the operator’s singular values relative to Fk is not preserved. Remark 2. (Basis-Dependent Approximation): Let Fk be a class of rank-k operators. While the singular values σi of CXY are invariant under U , the projection error ∥CU − PFk ∥HS depends on how well U aligns the dominant singular values with directions favored by the approximation class Fk . In H-Score networks, projection into the approximation manifold PFk is truncation to the k highest energy modes. Therefore an ideal U maximizes energy concentrated in these k modes, minimizing truncation error. The DFT, defined as a unitary matrix F ∈ CL×L , is a specific case of this transformation. We define the frequency domain crosscovariance CXf Y = F CXY . When the underlying process is approximately stationary, the DFT is known to approximately diagonalize the covariance [7], which aligns the dominant singular values of CXf Y in a way that minimizes projection error at minimal computational cost.
3
III. BASIS S ELECTION Although domain knowledge may suggest the use of a particular basis for low-rank projection, a numerical approach may identify more subtle dependence structures. We evaluate the effect of basis rotations on properties of the singular value spectra by using representative samples of historical data. We cover both a generic and FFT-specific method for quantifying these effects to facilitate a comparative basis selection process. A. Whitened Cross-Covariance Operator Let X ∈ RN ×dx and Y ∈ RN ×dy denote centered observations. The cross-covariance matrix is defined as CXY = 1 ⊤ N X Y . To ensure scale invariance, we analyze the whitened cross-covariance operator: eXY = C −1/2 CXY C −1/2 , C XX YY
(7)
where CXX = N1 X ⊤ X and CY Y = N1 Y ⊤ Y . The singular eXY , denoted σt,i , represent the energy associated values of C with dependence structures in the temporal domain. Practically, these quantities are computed using low-rank PCA approximations for numerical efficiency.
For H-Score network parameterization, we define the Cumulative Energy Rank as the minimum integer k required to retain a threshold α of the total energy: ) ( Pk 2 i=1 σi ≥α . (11) CERα = min k ∈ Z : PK 2 i=1 σi This serves as a heuristic estimate for the latent dimension of the feature extractor. Additionally, the Effective Rank, computed via the participation ratio [13] or as the exponential of the entropy [14], provides a real-valued approximation for the smallest viable latent dimension. IV. N UMERICAL R ESULTS A. Case Study Datasets We evaluate the proposed framework on eight public multivariate datasets spanning traffic, solar power, weather, wireless networking, cloud gaming, and tabular regression tasks [2], [15]–[18]. The datasets exhibit varying levels of temporal and spectral structure, allowing evaluation of when FFT preconditioning improves low-rank feature learning. The wine quality datasets [19] serve as a negative control due to the lack of temporal dependence.
B. Frequency-Resolved Cross-Covariance For temporal data X ∈ RN ×L×dx , the unitary DFT applied along the time axis yields Xf = F X. The frequency-resolved cross-covariance at frequency ω is defined as: 1 Xf (ω)† Y. (8) N The predictive energy at each frequency is the squared Frobenius norm σω = ∥Cω ∥2F . This energy distribution across frequencies provides a computable proxy for the singular value decay of the full operator CXf Y described in Section II-C. Cω =
C. Spectral Metrics for Basis Evaluation To provide a principled criterion for basis selection, we derive comparative statistics from the singular value spectra computed in the temporal and Fourier domains. We quantify the concentration of predictive information by treating the normalized singular values (or P frequency energies) as a probability distribution, pi = σi / σj . The normalized spectral entropy is: H=−
n 1 X pi log pi , log n i=1
(9)
where n denotes the number of components. We denote the entropy in the temporal and frequency domains as Ht and Hf . To quantify the impact of spectral preconditioning, we define the Entropy Ratio: ER =
Hf . Ht
(10)
Values of ER < 1 indicate CXf Y has a more concentrated singular value spectrum than CXY . From Remark 2, we know that this corresponds with a reduction in finite-width approximation error, and practically, more accurate prediction.
B. Experimental Design We evaluate the relationship of Fourier Score with Mean Squared Error (MSE) reduction associated with preconditioning the input using the FFT. The input time series is segmented into sliding windows of length L = 29. For each window, the first L − 1 = 28 samples constitute the input X ∈ R28×dx and the final sample is the prediction target Y ∈ Rdy . The choice of L is a trade-off between FFT-induced spectral leakage effects, and retaining temporal information. Leakage leads to a higher entropy singular value spectrum by spreading energy over many bins. This effect can be mitigated by making L arbitrarily large, but this in turn destroys temporal information and increases processing time. The fence effect, which also arises from limited FFT resolution, can split energy from a dominant mode across adjacent bins when the true modal frequency is not represented by the discrete FFT grid; unlike leakage, this redistribution is localized and therefore has a smaller effect on retained energy. For each dataset, we compute the dataset-level metrics, including the spectral entropy and CER95 in the temporal and frequency domains. These are evaluated prior to model training and are used for correlation analysis. The feature extraction model consists of mappings fθ (·) and gϕ (·), trained jointly to maximize H-Score. The f network is an Echo State Network (ESN) with ReLU activations, and the g network is an MLP with two hidden layers (16 and 32 neurons) with a ReLU activation function. The ESN may initially appear to be an odd choice when considering frequency domain inputs, as the reservoir is intended to learn temporal dependencies by considering previous input samples. However, we can exploit this structure in the frequency domain by using the reservoir to capture interfrequency coupling, which substantially expands the learnable dynamics to include harmonic relationships, coupling factors,
4
Traffic
Wine Quality (Red)
Climate
Wine Quality (White)
Entropy Ratio vs FFT Gain
Solar
O-RAN (Video)
O-RAN (Web Browsing)
Entropy Ratio vs Random Gain
Cloud VR Gaming
Low-Data (5%): Entropy Ratio vs FFT Gain
30 % NMSE Improvement
50
50
r = −0.49, p = 0.215
20
25
0
0
10 −50
−25
0
−100
−50 −75
r = −0.69, p = 0.058 0.5
0.6
0.7 0.8 Entropy Ratio
0.9
1.0
−10
r = −0.80, p = 0.018 0.5
0.6
0.7 0.8 Entropy Ratio
0.9
1.0
0.5
0.6
0.7 0.8 Entropy Ratio
0.9
1.0
Fig. 1: Numeric evaluation of entropy ratio prediction accuracy O-RAN Web Browsing - Average NMSE vs. Dimension 1.0
Hf
Ht
O-RAN 0 (Video) O-RAN 1 (Web Browsing) Cloud VR Gaming Traffic Wine Quality (Red) Wine Quality (White) Weather Solar
14 4 2 7 9 8 2 2
11 8 14 26 5 9 8 19
0.98 0.44 0.60 0.72 0.97 0.97 0.63 0.55
0.98 0.91 0.81 0.92 0.96 0.93 0.90 0.86
TABLE I: Structural metrics for case study datasets and the underlying power spectral density function of the data by presenting each frequency bin in monotonic order. The dimension of the output layer for each network is fixed at 8, which functionally caps the number of learnable dependence modes. The feature mapping network trains with a learning rate of 0.005 for 50 epochs, with a batch size of 128. Following training, the learned feature representation fθ (X) is used as the input to a downstream low-dimensional MLP. This predictor is trained on MSE loss to estimate the target Y . Model performance is evaluated on data retained from traintest splits of [0.05, 0.3, 0.6, 0.95]. Performance is quantified as the NMSE of the inference, evaluated using either no basis rotation, a random basis rotation, or the FFT with real and imaginary parts concatenated. We present the mean and 95% confidence interval across 10 random seeds. C. Spectral Transformation Results The results in Fig. 1 show a statistically significant relationship between ER and NMSE improvements in the time and frequency domains in the low-data (5% training split) case, and a near-significant relationship in the unrestricted data case. The difference between frequency and time-domain conditions is especially stark in the low-data regime, with a noticeable decline in metric accuracy and performance improvement when data availability is unrestricted. The randomized rotation condition shows a near-zero insignificant (p = 0.12) improvement over no rotation, reinforcing the need for numeric evaluation of dependence structures before arbitrarily applying a rotation. The aperiodic “wine” datasets demonstrate a case where the metric accurately predicts a negative consequence of spectral pre-conditioning, further emphasizing the need for metric computation prior to using the FFT. In absolute terms, we observe NMSE decreases by as much as 50% when minimal data is available for training. Smaller margins in the higher-data regime are expected, as any differences between function-space mappings are learned by the supervised model
Time 95%: k=8 100 Freq 95%: k=4 90
0.8
80 0.6
Frequency Domain Time Domain FFT (Low Data Mean)
0.4
No FFT (Low Data Mean)
70 60
Retained Energy %
(time) CER0.95
Average NMSE (MSE)
Dataset
(freq) CER0.95
50 0.2 40 0.0
30 2
3
4
5
6
7
8
9
Dimension
Fig. 2: Multi-domain average NMSE trend (full opacity) and per-seed (reduced opacity) NMSE vs. latent dimension given sufficient data, eventually resulting in similar outcomes. Fig. 2 shows the relationship between empirically observed NMSE differences across dimensions and domains for the Web Browsing dataset. The results of three random seeds are shown in low opacity, with a smoothed trend to indicate the average result. Retained energy, acting as a proxy for truncation error, appears predictive of inference accuracy across latent dimensions. CER, with an appropriate choice of α, may be useful for dataset-level cost-accuracy tuning. V. C ONCLUSION This letter identified input preconditioning as a method for optimizing H-Score feature extraction networks. We showed how the basis dependence of these networks is a result of finite-width network effects, and developed proxy metrics for evaluating the impact of a chosen basis on the resulting truncation error. The FFT appears to be a strong choice for such a preconditioner in theory, as it produces a concentrated basis without data dependence and for a minimal computational cost. Small window lengths, necessary for rapid FFT computation, also introduce spectral leakage. This leakage increases spectral entropy, degrading performance, but can be mitigated using windowing methods like Hann windowing. Numeric results validate both the magnitude of the preconditioning effect and the accuracy of the proxy metrics, measured using NMSE of a downstream inference model. Results in the lowdata regime show a stronger effect, enhancing the already strong low-data performance of H-Score networks [2]. Further, we found that CER is weakly predictive of the optimal latent dimension using O-RAN data as a case study. Although the frequency domain input representation produced a more accurate inference model in highly compressed representations, the empirically optimal dimension remained the same in both domains. Future work may consider studying optimal latent dimension prediction, preconditioning alternative network architectures or window length effects.
5
R EFERENCES [1] X. Xu and L. Zheng, “Neural feature learning in function space,” Journal of Machine Learning Research, vol. 25, no. 142, pp. 1–76, 2024. [2] J. Dai, R. Zhao, F. Rezazadeh, L. Zheng, H. Wang, and L. Liu, “Learning low-dimensional representation for O-RAN testing via transformerESN,” in IEEE MASS, 2025, pp. 127–133. [3] M. Greenacre, P. J. F. Groenen, T. Hastie, A. I. D’Enza, A. I. Markos, and E. Tuzhilina, “Principal component analysis,” Nature Reviews Methods Primers, vol. 2, 2022. [4] F. L. Gewers, G. R. Ferreira, H. F. D. Arruda, F. N. Silva, C. H. Comin, D. R. Amancio, and L. D. F. Costa, “Principal component analysis: A natural approach to data exploration,” ACM Computing Surveys, vol. 54, no. 4, pp. 1–34, 2021. [5] X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel, “InfoGAN: interpretable representation learning by information maximizing generative adversarial nets,” in NIPS, 2016, pp. 2180– 2188. [6] R. D. Hjelm, A. Fedorov, S. Lavoie-Marchildon, K. Grewal, P. Bachman, A. Trischler, and Y. Bengio, “Learning deep representations by mutual information estimation and maximization,” in ICLR, 2019. [7] A. G. Marques, S. Segarra, G. Leus, and A. Ribeiro, “Stationary graph processes and spectral estimation,” IEEE Trans. on Signal Processing, vol. 65, no. 22, pp. 5911–5926, 2017. [8] J. Öfverstedt, J. Lindblad, and N. Sladoje, “Fast computation of mutual information in the frequency domain with applications to global multimodal image alignment,” Pattern Recognition Letters, vol. 159, pp. 196–203, 2022. [9] A. Pradhan, L. Liu, and H. S. Dhillon, “Mutual information-driven visualization and clustering for core KPI selection in O-RAN testing,” IEEE Wireless Communications, pp. 1–8, 2026. [10] R. Malladi, D. H. Johnson, G. P. Kalamangalam, N. Tandon, and B. Aazhang, “Mutual information in frequency and its application to measure cross-frequency coupling in epilepsy,” IEEE Trans. on Signal Processing, vol. 66, no. 11, pp. 3008–3023, 2018. [11] A. Makur, G. W. Wornell, and L. Zheng, “On estimation of modal decompositions,” in IEEE ISIT, 2020, pp. 2717–2722. [12] X. Xu and S.-L. Huang, “Maximal correlation regression,” IEEE Access, vol. 8, pp. 26 591–26 601, 2020. [13] Y. Liu, P. Sierant, P. Stornati, M. Lewenstein, and M. Podzie, “Quantum algorithms for inverse participation ratio estimation in multiqubit and multiqudit systems,” Phys. Rev. A, vol. 111, p. 052614, May 2025. [14] F. Bunea and L. Xiao, “On the sample covariance matrix estimator of reduced effective rank population matrices, with applications to fPCA,” Bernoulli, vol. 21, no. 2, May 2015. [15] G. Lai, W.-C. Chang, Y. Yang, and H. Liu, “Modeling long and shortterm temporal patterns with deep neural networks,” in SIGIR, 2018, pp. 95–104. [16] J. Dai, R. Zhao, and L. Liu, “Video streaming network KPIs for O-RAN testing,” 2025. [Online]. Available: https://dx.doi.org/10.21227/v8p4v974 [17] P. Graff, X. Marchal, T. Cholez, S. Tuffin, B. Mathieu, and O. Festor, “An Analysis of Cloud Gaming Platforms Behavior under Different Network Constraints,” in Proc. 17th Int. Conf. Netw. Serv. Manag, 2021, p. 7. [18] M. P. I. for Biogeochemistry, “Jena climate dataset,” 2021. [Online]. Available: https://www.kaggle.com/datasets/mnassrib/jena-climate [19] P. Cortez, A. Cerdeira, F. Almeida, T. Matos, and J. Reis, “Wine quality,” UCI Machine Learning Repository, 2009.