Spectral Stability of Pseudoinverse-Based Extreme Learning Machine Nguyen Bich Van1,∗ , Khong Ngoc Anh2 1 2 , : Institute for Artificial Intelligence, VNU University of Engineering and Technology, Vietnam 1 : [email protected], 2 : [email protected]
Abstract
arXiv:2607.08581v1 [cs.LG] 9 Jul 2026
Extreme Learning Machine (ELM) computes output weights analytically using the Moore–Penrose pseudoinverse. Although this leads to fast training, its numerical stability depends strongly on the conditioning of the hidden-layer matrix. This paper studies pseudoinverse-based ELM from a spectral perspective. We show that the smallest singular value governs perturbation amplification in the output weights, while the condition number provides a quantitative measure of hidden-layer instability. We compare SVD-based pseudoinverse computation with iterative hyperpower methods and discuss width-dependent conditioning through a random feature interpretation. Experiments on synthetic matrices and ELM benchmarks show that SVD-based methods remain the most reliable under ill-conditioning, while iterative methods are more sensitive to spectral properties. The results suggest that ELM stability is fundamentally governed by the singular-value structure of the hidden-layer matrix.
Keywords: Extreme Learning Machine, Moore–Penrose inverse, numerical stability, singular values, conditioning
1.
Introduction
Related work on ELM has emphasized fast analytical training and classification performance [3]. The theory of
Extreme Learning Machine (ELM), originally introduced
generalized inverses and least-squares solutions is classi-
by Huang et al. [3], is a single-hidden-layer feedforward
cal [1], while SVD-based conditioning and stability analysis
neural network in which hidden-layer parameters are ran-
are standard topics in numerical linear algebra [2, 6]. Itera-
domly assigned and output weights are computed analyt-
tive inverse and pseudoinverse methods, including Newton–
ically. Given a training set {(xj , tj )}N j=1 , the hidden-layer
Schulz and hyperpower iterations, may be computation-
matrix is
ally attractive but can be sensitive to initialization and illHji = g(wiT xj + bi ),
(1)
conditioning. In parallel, random matrix theory provides use-
where wi , bi , and g denote the hidden weights, biases, and
ful estimates for the singular values of random feature ma-
activation function, respectively. The output weights are ob-
trices [4, 5]. The contribution of this paper is a compact spectral inter-
tained from Hβ = T,
β = H + T,
pretation of pseudoinverse-based ELM stability. We connect:
(2)
hidden-layer spectrum
where H
+
is the Moore–Penrose pseudoinverse.
→
conditioning
→
pseudoinverse sensitivity → learning stability.
This closed-form training is computationally attractive,
We also summarize experiments showing how singular val-
but it is also sensitive to the spectral properties of H. If the
ues, condition numbers, and solver behavior are related in
hidden-layer matrix is ill-conditioned, small singular values
synthetic matrices and ELM benchmarks.
may amplify perturbations in the targets, numerical round-
2.
off, or solver approximation errors. Thus, the stability of
Let H ∈ RN ×L be the hidden-layer matrix of an ELM and
ELM is not only a learning problem but also a numerical
suppose its compact singular value decomposition is
linear algebra problem. ∗
Spectral Stability Analysis
H = Ur Σr VrT ,
Corresponding author
1
(3)
where r = rank(H), Ur ∈ RN ×r , Vr ∈ RL×r have orthonor-
convergence may slow down, stagnate, or become sensitive
mal columns, Σr = diag(σ1 , . . . , σr ) and σmax (H) ≥ σ1 ≥
to finite-precision errors.
· · · ≥ σr = σmin (H) > 0 are positive singular values of H.
The hidden-layer matrix in ELM may be viewed as a random feature matrix because
The Moore–Penrose pseudoinverse is T H + = Vr Σ−1 r Ur .
Hji = g(wiT xj + bi )
(4)
(10)
is generated from random hidden parameters such as hid-
Hence, 1 ∥H + ∥2 = . σmin (H)
den weights wi and bias bi . If the normalized entries of H
(5)
are independent, centered, sub-Gaussian random variables
Equation (5) shows that the smallest singular value is the key
with unit variance, following a standard theorem in random
spectral quantity controlling pseudoinverse amplification.
matrix theory [4, 5] we have
The condition number of H is σmax (H) κ(H) = . σmin (H)
√ √ σmin (H) ≥ c( N − L)
(11)
(6) with probability at least 1 − 2 exp(−C(N − L)) for some
It measures how unevenly H stretches different directions. If
constants c, C > 0. Consequently, ∥H + ∥2 ≤ c(√N1−√L) .
κ(H) is large, at least one direction is nearly collapsed by H,
Although practical ELM matrices are not fully indepen-
making the least-squares solution sensitive to perturbations.
dent random matrices, this estimate motivates a width-
Consider a perturbation T 7→ T + ∆T . The corresponding
dependent interpretation: larger sample size tends to improve
perturbation in the ELM output weights is ∆β = H + ∆T.
stability, while excessively large hidden width may deteriorate conditioning.
(7)
4.
Experimental Results
Therefore, ∥∆β∥2 ≤ ∥H + ∥2 ∥∆T ∥2 =
∥∆T ∥2 . σmin (H)
We conduct two groups of experiments. The first group (8)
uses synthetic matrices with prescribed singular-value spectra to compare SVD-based and iterative pseudoin-
Thus, small σmin (H) or large κ(H) implies stronger pertur-
verse solvers. The second group evaluates pseudoinverse-
bation amplification. This provides the main theoretical ex-
based ELM on benchmark classification datasets, including
planation for instability in pseudoinverse-based ELM.
MNIST, Fashion-MNIST, and ISOLET.
3.
The main evaluation metrics are least-squares residual,
Solvers and Width-Dependent Conditioning
Penrose residuals, runtime, convergence success, classification accuracy, the smallest singular value σmin (H), and the
SVD-based pseudoinverse computation directly uses the
condition number κ(H).
singular values of H and is typically backward stable because
Table 1 summarizes the convergence behavior of iterative
it relies on orthogonal transformations [2, 6] which preserve
pseudoinverse methods under different spectral conditions.
Euclidean norm. It also allows small singular directions to
The results show that iterative methods are reliable for well-
be detected and truncated if necessary.
conditioned and moderately ill-conditioned spectra, but fail
Iterative methods approximate the pseudoinverse through
completely for severely ill-conditioned spectra. In particu-
repeated matrix updates. A representative example is the
lar, all iterative runs succeeded in the well-conditioned case,
Newton–Schulz iteration
whereas no iterative method succeeded in the severely ill-
Xk+1 = Xk (2I − HXk ).
conditioned case. This confirms that convergence reliability
(9)
is strongly controlled by the singular-value structure of the matrix.
Such methods can be efficient in favorable settings, but their convergence depends on the initialization, scaling, stopping
The detailed residual summaries further support this ob-
criteria, and the spectrum of H. When H is ill-conditioned,
servation. In the well-conditioned case, all methods achieved 2
tion in the output weights, while the condition number pro-
Table 1. Convergence behavior of iterative pseudoinverse methods under different spectral conditions. 10
−10
−7
vides a compact measure of hidden-layer ill-conditioning.
and 10
The comparison between SVD-based and iterative methods
are thresholds.
indicates that SVD remains the most reliable solver under seSpectrum
10−10
10−7
vere ill-conditioning, whereas hyperpower methods are more
Well-conditioned
36/36
36/36
sensitive to spectral properties and initialization.
Moderately ill-conditioned
36/36
36/36
The width-dependent interpretation further suggests that
Severely ill-conditioned
0/36
0/36
hidden-layer stability depends on the balance between sam-
Rank deficient
10/36
36/36
ple size and hidden width. Future work includes regularized
Clustered small
24/36
36/36
ELM, randomized SVD, improved stopping rules for iterative solvers, and larger-scale GPU implementations.
Reference [1] A. Ben-Israel and T. N. E. Greville, Generalized Inverses: Theory and Applications, Springer, 2003. [2] G. H. Golub and C. F. Van Loan, Matrix Computations, 4th ed., Johns Hopkins University Press, 2013. [3] G.-B. Huang, Q.-Y. Zhu, and C.-K. Siew, “Extreme Learning Machine: Theory and Applications,” Neurocomputing, vol. 70, no. 1–3, pp. 489–501, 2006. Figure 1. Mean smallest singular value of the ELM hidden-
[4] M. Rudelson and R. Vershynin, “The Smallest Singular
layer matrix for different training sample sizes N and hidden
Value of a Random Rectangular Matrix,” Communica-
widths L. Larger L generally reduces σmin (H), indicating
tions on Pure and Applied Mathematics, vol. 62, no. 12,
poorer conditioning and stronger pseudoinverse sensitivity.
pp. 1707–1739, 2009. [5] R. Vershynin, High-Dimensional Probability: An Intro-
successful runs with residuals close to machine precision. For
duction with Applications in Data Science, Cambridge
example, Newton–Schulz achieved a median least-squares
University Press, 2018. [6] L. N. Trefethen and D. Bau, Numerical Linear Algebra,
residual of 1.21×10−15 under εconv = 10−10 . In contrast, un-
SIAM, 1997.
der the severely ill-conditioned spectrum, all iterative methods failed, while SVD-based methods retained successful runs. Figure 1 reports the mean smallest singular value of the ELM hidden-layer matrix under different sample sizes and hidden widths. The results show a consistent decrease of σmin (H) as the hidden width L increases. This behavior supports the width-dependent spectral interpretation: increasing L may improve representational capacity, but it can also worsen conditioning and increase the sensitivity of the pseudoinverse solution.
5.
Conclusion and Future Work This paper studied pseudoinverse-based ELM from a spec-
tral stability perspective. The analysis showed that the smallest singular value determines perturbation amplifica3