Residual-Guided Randomized Neural Networks

arXiv:2608.28267v1 [cs.LG] 28 Aug 2026

Residual-Guided Randomized Neural Networks Mushir Akhtar

M. Tanveer

Mohd. Arshad

Department of Mathematics Indian Institute of Technology Indore phd2101241004@iiti.ac.in

Department of Mathematics Indian Institute of Technology Indore mtanveer@iiti.ac.in

Department of Mathematics Indian Institute of Technology Indore arshad@iiti.ac.in

Abstract—Randomized neural networks enable fast and analytically tractable training by fixing the input to hidden layer parameters at random and learning the output weights in closed form; however, their performance critically depends on a single uninformed draw of hidden units. This one shot and task uninformed feature construction often leads to redundant representations and suboptimal utilization of model capacity. To address this limitation, we propose a simple and broadly applicable residual guided procedure that greedily constructs the hidden layer using a closed form residual decrease criterion. At each stage, we (i) generate a pool of random candidate units, (ii) score each candidate by the exact reduction it induces in the ridge regularized objective, (iii) select the top k units, and (iv) refit the readout in closed form using the standard design with direct input links. This procedure yields a progressive training process with a guaranteed monotonic decrease of the training objective. The method is model agnostic: only the candidate generation is architecture specific, while the scoring selection refitting loop is shared across models. Extensive experiments on 71 benchmark datasets from the UCI repository, covering both binary and multiclass classification tasks, demonstrate that the proposed residual-guided models consistently outperform their baseline counterparts in terms of accuracy, stability, and overall ranking performance. Index Terms—Randomized neural networks, residual guided learning, closed form training, random feature construction, greedy incremental learning.

I. I NTRODUCTION

R

ANDOMIZED neural networks (RaNNs) have attracted significant attention as efficient alternatives to deep learning models, particularly in scenarios involving limited data, strict computational constraints, or real-time requirements [1, 2, 3]. Unlike deep architectures trained via iterative backpropagation, RaNNs fix the input-to-hidden layer parameters at random and learn only the output-layer weights, typically using closed-form. This decoupling eliminates gradientbased optimization, substantially reduces training complexity, and yields analytically tractable learning procedures while retaining universal approximation capability under mild conditions [4, 5]. Within this paradigm, several representative RaNN architectures have been proposed, differing primarily in how random features are constructed and incorporated into the predictive model. The Random Vector Functional Link (RVFL) network [1, 6] is a widely adopted formulation that combines nonlinear random hidden features with direct input-to-output links, Accepted at the 2026 IEEE World Congress on Computational Intelligence (WCCI 2026).

enabling the preservation of linear input structure alongside nonlinear representations. The Extreme Learning Machine (ELM) [7] can be viewed as a special case of RVFL in which the direct input connections are removed, further simplifying the architecture while maintaining fast closed-form training. To enhance representational diversity without increasing depth, the Broad Learning System (BLS) [8, 9] was proposed as an alternative that expands the network horizontally by incrementally adding feature and enhancement nodes, enabling efficient incremental learning without retraining from scratch. Building upon these foundational architectures, extensive research has focused on improving RVFL, ELM, and BLS models in terms of stability, robustness, and generalization [6, 10]. Representative efforts include architectural refinements [11, 12], regularization strategies [13, 14], ensemble formulations [15, 16], and incremental learning mechanisms [17, 18], many of which report enhanced empirical performance across diverse learning scenarios. Recently, copula-aligned weight initialization (CAWI) has been proposed to incorporate dependency-aware copula modeling into weight initialization, aligning randomized weights with the intrinsic structure of the input data and thereby improving representation quality and learning performance [19]. Robust objectives and samplereweighting mechanisms provide complementary directions: the HawkEye loss combines boundedness, smoothness, and an insensitive zone for robust regression [20], while its integration with RVFL yields a noise- and outlier-resistant classifier [21]. For class-imbalanced settings, a location-aware slack-factor fuzzy SVM improves the assignment of sample memberships and reduces minority-class misclassification [22]. Despite this progress, most existing extensions share a common structural limitation: hidden representations are still generated through fixed, randomly sampled feature mappings. Consequently, increasing the number of hidden nodes or feature groups often introduces redundancy, leading to inefficient utilization of model capacity and limited adaptability to the data distribution. In particular, indiscriminate expansion with additional random features provides no guarantee that newly introduced representations contribute meaningfully to reducing the supervised residual error. These observations motivate the need for randomized learning frameworks that retain closed-form training while enabling principled and data-driven feature expansion. In this work, we propose a residual-guided randomized neural network framework that incrementally constructs hidden representations by

explicitly leveraging the current residual of the model. At each stage, randomly generated candidate features are evaluated based on their closed-form contribution to residual reduction, and only the most effective features are retained. This residualguided strategy enables adaptive and efficient feature construction while preserving the computational efficiency and analytical tractability of randomized neural networks. Importantly, the proposed framework is model-agnostic and can be seamlessly integrated into a broad class of RaNNs, including RVFL, ELM, BLS, and their variants, without modifying their fundamental training procedures. The main contributions of this work are summarized as follows: 1) We propose a general residual-guided learning framework for randomized neural networks that enables adaptive, data-driven hidden feature expansion while retaining closed-form output weight learning. 2) The proposed framework is model-agnostic and can be seamlessly integrated into a broad class of randomized neural networks, including RVFL, ELM, BLS, and their variants, without altering their core architectures or training paradigms. 3) We provide a theoretical monotonicity guarantee, establishing that the ridge-regularized training objective is non-increasing with the progressive addition of residualguided hidden features, which is further validated through empirical analysis. 4) Extensive experimental evaluations on 71 benchmark datasets covering both binary and multiclass classification tasks demonstrate that the proposed residualguided models consistently outperform their baseline counterparts. The remainder of this paper is organized as follows. Section II presents the preliminaries. Section III provides the motivation and problem statement. Section IV introduces the proposed framework. Section V reports experimental results. Finally, Section VI concludes the paper. II. P RELIMINARIES This section establishes the notation and reviews the Random Vector Functional-Link (RVFL) network, one of the standard randomized neural network. A. Notation Let U ∈ Rn×p denote the input data matrix with n samples and p features, and let T ∈ Rn×q denote the corresponding target matrix, where q is the output dimension (for classification, T is typically one-hot encoded). Randomized hiddenlayer activations are collected in G ∈ Rn×h , generated via random affine transformations of U followed by an elementwise nonlinearity ϕ(·). Each hidden unit is parameterized by a random weight vector w ∈ Rp and a bias β ∈ R, yielding the random weight matrix W ∈ Rp×h . The augmented feature matrix is defined as A = [ U G ] ∈ Rn×(p+h) . The outputlayer parameter matrix is denoted by Ω ∈ R(p+h)×q ; when an explicit bias term is included, A is augmented with a column

Fig. 1: Architecture of a standard RVFL network. of ones. The transpose operator is denoted by (·)⊤ , en ∈ Rn denotes the vector of ones, and Ik denotes the k × k identity matrix. B. Random Vector Functional-Link Network (RVFL) RVFL is a shallow feedforward architecture consisting of input, hidden, and output layers, distinguished by the presence of direct links from the input layer to the output layer. These direct links enable RVFL to jointly exploit linear structure from the raw inputs and nonlinear structure from randomized hidden features, often improving stability and generalization. The standard RVFL architecture is illustrated in Fig. 1. In RVFL, the hidden-layer parameters are randomly sampled once and kept fixed throughout training. Given W ∈ Rp×h and bias β ∈ R, the hidden-layer activations for input matrix U ∈ Rn×p are computed as
G = ϕ UW + en β ∈ Rn×h . The output layer operates on the augmented representation A = [ U G ] ∈ Rn×(p+h) and learns the readout matrix Ω by solving the ridge-regularized least-squares problem Ω⋆ = arg min ∥AΩ − T∥2F + λ∥Ω∥2F , Ω

λ > 0.

(1)

This problem admits a closed-form solution, which can be expressed in either primal or dual form depending on the relative dimensions of A, ( ⊤ (A A + λIp+h )−1 A⊤ T, if (p + h) ≤ n, ⋆ Ω = (2) A⊤ (AA⊤ + λIn )−1 T, otherwise. III. M OTIVATION & P ROBLEM S TATEMENT RaNNs are attractive since they eliminate backpropagation while enhancing linear models with nonlinear features. However, their performance critically depends on a single, task-uninformed draw of randomized hidden features. This section clarifies why such a design is fragile and formulates the precise optimization problem addressed in this work. Note: For clarity, we present the subsequent discussion in the RVFL setting, a representative RaNN architecture, and note that the proposed framework extends naturally to other

h hidden features from S that minimizes the ridge-regularized training loss. Formally, the design problem is

RaNNs. Setting: Let U ∈ Rn×p denote the input data matrix and T ∈ Rn×q the corresponding target matrix. A randomized mapping generates hidden-layer activations G ∈ Rn×h , which are concatenated with the inputs to form the augmented matrix A = [ U G ] ∈ Rn×(p+h) . The output layer uses a linear readout Ω ∈ R(p+h)×q , learned via ridge-regularized least squares, J (G) ≜

min Ω∈R(p+h)×q

2

[ U G ] Ω − T F + λ∥Ω∥2F . (3)

For any fixed G, the minimizer of (3) is available in closed form. Consequently, the effective design problem is to construct G so as to minimize J (G) under a prescribed hidden-unit budget. A. Core Limitations In the standard RVFL formulation, G is generated once by sampling a random affine map and applying a nonlinearity elementwise. This one-shot, label-uninformed construction leads to several practical shortcomings: 1) Feature misalignment and inefficiency. The randomly generated columns of G are not tailored to the supervised task and need not align with discriminative directions in the data. Consequently, many hidden units may exhibit weak correlation with the supervised residual after fitting the linear readout, leading to inefficient use of model capacity. 2) Lack of adaptive feature selection. Once G is fixed, ridge regularization can only uniformly shrink the output weights; it cannot selectively suppress or discard uninformative or noisy hidden features. This restriction yields a suboptimal bias-variance trade-off and can degrade generalization when h is constrained. 3) Sensitivity to random initialization and lack of progressive guarantees. Performance is highly sensitive to the random seed, as the entire representation depends on a single draw of G. Moreover, there is no principled mechanism for progressive model building: intermediate subsets of hidden units are not guaranteed to yield monotonic improvement in the training objective or predictive performance as additional hidden units are added.

min

G=[ g1 ··· gh ]

J (G)

s.t.

gj ∈ S ∀j, G ∈ Rn×h .

Equivalently, (5) seeks a size-r multiset of hidden features drawn from S such that, when concatenated with the direct input links U, the resulting augmented representation minimizes the ridge objective in (3). This formulation defines a combinatorial feature selection problem over a large and typically continuous set of admissible features, rendering exact optimization computationally intractable. Desired Properties: A practical alternative to one-shot RVFL should: 1) preserve closed-form training at the output layer (no backpropagation); 2) adapt hidden-unit selection to the supervised signal by prioritizing features that most reduce the current objective; 3) admit an anytime training procedure with a monotonic decrease of the objective as hidden nodes are added. Our Approach: We address (5) via a stagewise, residualguided construction. At each iteration, a manageable candidate pool is sampled from S. Each candidate is scored by its closedform reduction in (3) relative to the current model, the most effective units are selected, and the readout Ω is refit exactly on the updated augmented matrix. This procedure directly targets the limitations of standard RVFL while retaining its hallmark efficiency and closed-form training. IV. M ETHOD In this section, we propose a residual-guided construction for randomized neural networks that retains the key advantages of fixed randomized hidden mappings and closedform training while overcoming the limitations of one-shot hidden-layer sampling. The core idea is to replace the single, task-uninformed draw of hidden features with a supervised, stagewise selection procedure guided by the current residual error. A. Training Objective Given an input matrix U ∈ Rn×p and a hidden-feature matrix G ∈ Rn×h , RVFL forms the augmented representation A = [ U G ] and learns a linear readout Ω ∈ R(p+h)×q via ridge-regularized least squares,

B. Problem Statement

Ω⋆ (A, λ) = arg min ∥AΩ−T∥2F +λ∥Ω∥2F ,

Let S denote the set of admissible hidden features generated by the randomized mapping,

The resulting training objective value is

n

S = {g ∈ R | g = ϕ(Uw + en β) , (w, β) ∼ P} ,

Ω

(4)

where P denotes a prescribed sampling distribution over Rp × R. Given a hidden-unit budget h ∈ N and a regularization parameter λ > 0, the objective is to select a collection of

(5)

λ > 0. (6)

J (G) = min ∥ [ U G ]Ω − T∥2F + λ∥Ω∥2F . Ω

(7)

Since the minimizer in (6) is available in closed form, learning in RVFL reduces to constructing a hidden-feature matrix G, under a prescribed hidden-unit budget h, that yields a small value of J (G). This observation motivates a direct focus on how hidden features are generated and selected.

Algorithm 1 Residual-Guided Stagewise Algorithm

B. Residual-Guided Stagewise Construction Rather than generating all h hidden features in a single step, the proposed method constructs the hidden-feature matrix G incrementally in a stagewise manner. Let Gt ∈ Rn×ht denote the hidden-feature matrix after stage t, with h0 = 0, and define the corresponding augmented representation as At = [ U Gt ]. At each stage, the current model is first fitted exactly by solving the ridge regression problem, yielding the optimal readout and the associated residual Ωt = Ω⋆ (At , λ),

Rt = At Ωt − T.

(8)

The residual Rt represents the portion of the supervised signal that is not yet captured by the current representation. Since the training objective J (Gt ) depends directly on this residual, reducing ∥Rt ∥F leads to a decrease in the objective value. Each stage of the algorithm then proceeds by (i) generating a temporary pool of candidate random hidden features, (ii) evaluating each candidate according to its ability to reduce the current residual, and (iii) selecting a small subset of the most effective features to be permanently added to the model. After feature selection, the readout is refitted exactly using the augmented representation, and the process is repeated until the hidden-unit budget h is reached or a stopping criterion is satisfied. The individual steps of candidate generation, residual-based scoring, and selection are detailed below. 1) Candidate generation: At stage t, a temporary pool of M candidate hidden features is generated. Specifically, random affine parameters are drawn from the sampling distribution P and transformed using the activation function ϕ(·),

Require: Data U ∈ Rn×p , targets T ∈ Rn×q ; hidden nodes budget h; pool size M ; block size k; regularization λ > 0; sampling distribution P; activation ϕ(·). ′ Ensure: Hidden features G ∈ Rn×h with h′ ≤ h and readout ′ Ω ∈ R(p+h )×q . 1: Initialize G0 ← [ ], h0 ← 0, t ← 0. 2: while ht < h and stopping criterion not met do 3: At ← [ U Gt ]. 4: Ωt ← Ω⋆ (At , λ) ▷ closed-form ridge solution 5: Rt ← At Ωt − T. 6: for j = 1, . . . , M do ▷ temporary candidate pool 7: Sample (w(j) , β (j) ) ∼ P. 8: g(j) ← ϕ(Uw(j) + en β (j) ) ∈ Rn . ∥g(j)⊤ Rt ∥2F ▷ score ∆(g(j) ) 9: s(j) ← ∥g(j) ∥22 + λ 10: end for 11: Let I be indices of the top-k scores {s(j) }M j=1 (or fewer if ht + k > h). (j) ]j∈I . 12: Gsel t ← [g 13: Gt+1 ← [ Gt Gsel ht+1 ← ht + |I|. t ], 14: t ← t + 1. 15: end while 16: A ← [ U Gt ], Ω ← Ω⋆ (A, λ). 17: return Gt , Ω.

and append them to the current hidden-feature matrix, Gt+1 = [ Gt Gsel t ],

At+1 = [ U Gt+1 ].

The linear readout is then refitted exactly on the updated representation by solving (6), yielding Ωt+1 . This stagewise process g(j) = ϕ(Uw(j) +en β (j) ), (w(j) , β (j) ) ∼ P, j = 1, . . . , M. of candidate generation, residual-based scoring, selection, and (9) refitting is repeated until the hidden-unit budget h is reached The resulting vectors {g(j) }M form a candidate pool that is or a validation criterion terminates training. j=1 used only at the current stage, from which a small number of The aforementioned steps complete the description of the informative features will be selected and permanently added proposed residual-guided stagewise construction. For clarity to the model. and completeness, the entire procedure is summarized in 2) Residual-based scoring: Each candidate hidden feature Algorithm 1. is evaluated based on its ability to reduce the current training objective. For a candidate g, we consider augmenting the C. Monotonicity Guarantee current representation At with this feature and optimizing only In this subsection, we establish a key theoretical property of its associated coefficient, while keeping the previously learned the proposed residual-guided stagewise method. Specifically, readout parameters fixed. Under this setting, the optimal coefwe show that the ridge-regularized training objective is nonficient and the resulting exact decrease in the ridge-regularized increasing across stages, ensuring that the algorithm makes objective are given by consistent progress as additional hidden features are incorpo⊤ 2 ⊤ ∥g R ∥ g R rated. t F t , ∆(g) = . (10) Ω⋆g = 2 2 ∥g∥2 + λ ∥g∥2 + λ Lemma IV.1. Let J (G) be defined as in (7). At each stage of The score ∆(g) therefore quantifies the immediate reduction the residual-guided procedure, the following inequality holds: in the training objective that would be achieved by adding J (Gt+1 ) ≤ J (Gt ). g alone, making it a supervised and scale-aware criterion for greedy feature selection. Proof. Let A = [ U G ] and A = [U G ], where t

3) Selection and refitting: After scoring all M candidates, we select the top-k features with the largest values of ∆(g)

t

t+1

t+1

Gt+1 = [ Gt Gsel t ]. By construction, At is a columnsubmatrix of At+1 .

Let Ω⋆t denote an optimal solution attaining J (Gt ) in (7). e for At+1 by assigning Ω⋆ We construct a coefficient matrix Ω t to the columns corresponding to At and setting the coefficients of the newly added columns to zero. Then, e = At Ω⋆ , At+1 Ω t

TABLE I: Average accuracy (%), average standard deviation, and average rank of baseline and residual-guided models on 32 binary and 39 multiclass datasets. Task

RVFL

RG-RVFL†

ELM

RG-ELM†

BLS

RG-BLS†

80.92 8.91 1.97

82.92 7.88 1.03

80.82 8.36 1.88

82.76 7.76 1.13

81.65 6.24 1.91

82.78 5.75 1.09

Accuracy ↑ 73.33 Std. Dev. ↓ 10.43 Rank ↓ 1.96 † denotes the proposed counterpart.

75.94 9.11 1.04

73.34 10.12 1.94

75.35 8.99 1.06

74.96 7.98 1.86

76.20 7.17 1.14

e F = ∥Ω⋆ ∥F . ∥Ω∥ t Binary (32)

e achieves the same objective value on At+1 Consequently, Ω ⋆ as Ωt does on At . Since J (Gt+1 ) is defined as the minimum of the ridge-regularized objective over all coefficient matrices associated with At+1 , it cannot exceed this value. Therefore,

Method Comparison

Metric

Multiclass (39)

Accuracy ↑ Std. Dev. ↓ Rank ↓

J (Gt+1 ) ≤ J (Gt ), which completes the proof. D. Computational Complexity At each stage, generating a pool of M candidate hidden features requires O(npM ) operations, corresponding to the evaluation of random affine mappings and activation functions. Scoring the candidates incurs an additional cost of O(nM q) to compute the inner products Gpool⊤ Rt , along t with O(nM ) operations to evaluate the associated ℓ2 norms. Refitting the linear readout after stage t using a naive approach costs O(n(p + ht )2 ) when using the primal formulation, or O(n2 (p + ht )) when using the dual formulation. However, since only a small block of k hidden features is added at each stage, the readout update can be performed incrementally by exploiting the resulting low-rank modification of the augmented matrix. This

reduces the refitting cost to O nk(p+ht ) +O k 2 (p+ht ) . Let T denote the total number of stages. Since T ≈ ⌈h/k⌉, the overall computational cost of the proposed method grows approximately linearly with the hidden-unit budget h for fixed pool size M and block size k. V. E XPERIMENTAL R ESULTS This section evaluates the proposed residual-guided framework by integrating it into three RaNN architectures, namely RVFL, ELM, and BLS, yielding the corresponding residualguided models RG-RVFL, RG-ELM, and RG-BLS. The proposed models are compared with their baseline counterparts to assess the effectiveness of residual-guided construction. Performance is evaluated using classification accuracy, average rank, and standard deviation. Experiments are conducted on a total of 71 benchmark datasets, comprising both binary and multiclass classification problems, downloaded from the UCI repository [23]. The detailed experimental setup along with the hyperparametr configuration is provided in Section S.I of the supplementary file. A. Performance Analysis Table I summarizes the average classification accuracy, standard deviation, and average rank of the baseline and residualguided models over 32 binary and 39 multiclass datasets. Overall, the results consistently demonstrate the effectiveness of the proposed residual-guided learning framework across all three randomized architectures, namely RVFL, ELM, and BLS.

For binary classification tasks, the residual-guided variants achieve clear and consistent accuracy gains over their corresponding baselines. In particular, RG-RVFL, RG-ELM, and RG-BLS improve the average accuracy from 80.92% to 82.92%, from 80.82% to 82.76%, and from 81.65% to 82.78%, respectively. Similar trends are observed for multiclass datasets, where the proposed models yield notable improvements, with average accuracy increasing from 73.33% to 75.94% for RVFL, from 73.34% to 75.35% for ELM, and from 74.96% to 76.20% for BLS. These results indicate that residual-guided construction consistently enhances the representational capacity of the underlying randomized learners. Beyond accuracy, the proposed framework also exhibits improved stability. As reflected by the lower average standard deviation values, all residual-guided models demonstrate reduced performance variability across datasets. For binary problems, the standard deviation decreases from 8.91 to 7.88 for RVFL, from 8.36 to 7.76 for ELM, and from 6.24 to 5.75 for BLS. A similar reduction is observed in multiclass settings, confirming that the residual-guided strategy not only improves mean performance but also yields more reliable and robust predictions across diverse data distributions. The superiority of the proposed approach is further reinforced by the average rank analysis. Across both binary and multiclass tasks, the residual-guided variants consistently achieve nearoptimal average ranks, all close to one, indicating dominant performance among the compared methods. In contrast, the baseline models exhibit noticeably higher ranks, reflecting inferior relative performance. This consistent ranking advantage highlights the effectiveness of the proposed residualguided mechanism independent of the specific randomized architecture employed. Collectively, these results demonstrate that incorporating residual-guided learning into RVFL, ELM, and BLS models leads to systematic improvements in accuracy, stability, and overall ranking performance. Importantly, the gains are observed across both binary and multiclass classification tasks, underscoring the generality of the proposed framework. Detailed dataset-wise results for all evaluated models are provided in Tables S.I and S.II of the supplementary material. To empirically validate the monotonicity property established in Section IV-C, we analyze the stagewise behavior of the proposed residual-guided framework. The detailed analysis is provided in Section S.II of the supplementary.

VI. C ONCLUSIONS In this work, we introduced a residual-guided learning framework for randomized neural networks that addresses a fundamental limitation of existing RaNN architectures, namely the reliance on a single, task-uninformed draw of hidden features. By replacing one-shot random feature construction with a supervised, stagewise selection mechanism guided by the current residual, the proposed framework enables adaptive and efficient hidden-layer expansion while preserving the hallmark advantages of randomized learning, including closedform training and computational efficiency. We establish a monotonicity guarantee showing that the ridge-regularized training objective is non-increasing as residual-guided hidden features are progressively added. This property provides a foundation for progressive model construction and stands in contrast to conventional randomized networks, where intermediate representations offer no such guarantees. Extensive experimental results on 71 benchmark datasets from the UCI repository, spanning both binary and multiclass classification tasks, confirm the practical effectiveness of the proposed framework. The results highlight the generality of the framework and its ability to enhance diverse randomized models without altering their core training paradigms. In future, one can extend the proposed framework for regression, multilabel, and semi-supervised tasks. R EFERENCES [1] Y.-H. Pao, G.-H. Park, and D. J. Sobajic, “Learning and generalization characteristics of the random vector functional-link net,” Neurocomputing, vol. 6, no. 2, pp. 163–180, 1994. [2] W. Cao et al., “A review on neural networks with random weights,” Neurocomputing, vol. 275, pp. 278–287, 2018. [3] P. N. Suganthan and R. Katuwal, “On the origins of randomization-based feedforward neural networks,” Applied Soft Computing, vol. 105, p. 107239, 2021. [4] B. Igelnik and Y.-H. Pao, “Stochastic choice of basis functions in adaptive function approximation and the functional-link net,” IEEE Transactions on Neural Networks, vol. 6, no. 6, pp. 1320–1329, 1995. [5] D. Needell et al., “Random vector functional link networks for function approximation on manifolds,” Frontiers in Applied Mathematics and Statistics, vol. 10, p. 1284706, 2024. [6] A. K. Malik et al., “Random vector functional link network: recent developments, applications, and future directions,” Applied Soft Computing, vol. 143, p. 110377, 2023. [7] G.-B. Huang, Q.-Y. Zhu, and C.-K. Siew, “Extreme learning machine: theory and applications,” Neurocomputing, vol. 70, no. 1-3, pp. 489–501, 2006. [8] C. P. Chen and Z. Liu, “Broad learning system: An effective and efficient incremental learning system without the need for deep architecture,” IEEE Transactions on Neural Networks and Learning Systems, vol. 29, no. 1, pp. 10–24, 2017.

[9] ——, “Broad learning system: A new learning paradigm and system without going deep,” in 2017 32nd Youth Academic Annual Conference of Chinese Association of Automation (YAC). IEEE, 2017, pp. 1271–1276. [10] X. Gong et al., “Research review for broad learning system: Algorithms, theory, and applications,” IEEE Transactions on Cybernetics, vol. 52, no. 9, pp. 8922– 8950, 2022. [11] S. Feng and C. P. Chen, “Fuzzy broad learning system: A novel neuro-fuzzy model for regression and classification,” IEEE transactions on cybernetics, vol. 50, no. 2, pp. 414–424, 2018. [12] A. K. Malik et al., “Alzheimer’s disease diagnosis via intuitionistic fuzzy random vector functional link network,” IEEE Transactions on Computational Social Systems, vol. 11, no. 4, pp. 4754–4765, 2024. [13] J.-W. Jin and C. P. Chen, “Regularized robust broad learning system for uncertain data modeling,” Neurocomputing, vol. 322, pp. 58–69, 2018. [14] M. Akhtar et al., “Towards robust and inversion-free randomized neural networks: The XG-RVFL framework,” Pattern Recognition, vol. 172, 4 2026. [15] Q. Shi et al., “Random vector functional link neural network based ensemble deep learning,” Pattern Recognition, vol. 117, 9 2021. [16] M. Sajid, M. Tanveer, and P. N. Suganthan, “Ensemble deep random vector functional link neural network based on fuzzy inference system,” IEEE Transactions on Fuzzy Systems, pp. 1–12, 2024. [Online]. Available: 10.1109/TFUZZ.2024.3411614 [17] C. Liu et al., “Domain-incremental learning without forgetting based on random vector functional link networks,” Pattern Recognition, vol. 151, p. 110430, 2024. [Online]. Available: https://www.sciencedirect. com/science/article/pii/S003132032400181X [18] L. Zhong et al., “Robust incremental broad learning system for data streams of uncertain scale,” IEEE Transactions on Neural Networks and Learning Systems, vol. 36, no. 4, pp. 7580–7593, 2025. [19] M. Akhtar, M. Tanveer, and M. Arshad, “CAWI: Copula-aligned weight initialization for randomized neural networks,” in The 29th International Conference on Artificial Intelligence and Statistics, 2026. [Online]. Available: https://openreview.net/forum?id=fwqydIjLn6 [20] ——, “HawkEye: A robust loss function for regression with bounded, smooth, and insensitive zone characteristics,” Applied Soft Computing, vol. 176, p. 113118, 2025. [21] M. Akhtar, R. Mishra, M. Tanveer, and M. Arshad, “Advancing RVFL networks: Robust classification with the HawkEye loss function,” in Neural Information Processing: 31st International Conference, ICONIP 2024, Proceedings, Part III, ser. Lecture Notes in Computer Science, vol. 15288. Singapore: Springer, 2025, pp. 226–240. [22] M. Tanveer, A. Tiwari, M. Akhtar, and C.-T. Lin, “Enhancing imbalance learning: A novel slack-factor fuzzy

SVM approach,” IEEE Transactions on Emerging Topics in Computational Intelligence, vol. 9, no. 4, pp. 3112– 3121, 2025. [23] D. Dua and C. Graff, “UCI machine learning repository,” URL http://archive. ics. uci. edu/ml, vol. 7, no. 1, p. 62, 2017.

Supplementary Material Residual-Guided Randomized Neural Networks S.I. E XPERIMENTAL S ETUP AND H YPERPARAMETER C ONFIGURATION All experiments are conducted using MATLAB R2023a on a system equipped with an 11th-generation Intel Core i7-11700 processor running at 2.50 GHz, 16 GB of RAM, and a Windows 11 operating system. To ensure a fair and statistically reliable evaluation, all models are assessed using a 5-fold cross-validation protocol combined with grid-based hyperparameter search. For each dataset, the samples are partitioned into five mutually exclusive folds. For a given hyperparameter configuration, the model is trained on four folds and evaluated on the remaining fold. This process is repeated until each fold has served once as the test set. The classification accuracy obtained across the five folds is averaged, and the configuration achieving the highest mean accuracy is reported as the final performance. In addition to the mean accuracy, the standard deviation across the five folds is also reported to reflect performance stability. For RVFL, ELM, and their residual-guided counterparts, the regularization parameter λ is selected from the set {10−5 , 10−4 , . . . , 105 }, and the number of hidden nodes is tuned within the range 5:10:205. For BLS and its residualguided variant, the regularization parameter λ is chosen from {10−6 , 10−4 , . . . , 106 }. The number of feature node windows is selected from 1:2:21, the number of feature nodes per window from 5:5:50, and the number of enhancement nodes from 5:10:105. For the proposed residual-guided framework, additional hyperparameters control the stagewise feature selection process. In particular, the block size k denotes the number of hidden features added at each stage, while the pool size M specifies the number of candidate random hidden nodes generated at that stage. The choice of k and M is adapted based on the number of training samples n and the input dimensionality p. When n ≤ 2000 or p ≤ 50, we consider k ∈ {1, 2} and M ∈ {50, 100}. For intermediatescale settings where 2000 < n ≤ 20000 or 50 < p ≤ 300, the search space is expanded to k ∈ {1, 2, 4} and M ∈ {100, 200}. For larger-scale problems, we use k ∈ {2, 4} and M ∈ {100, 200}. In addition, the total number of hidden features is constrained by a predefined budget, which is set as max(50, min(1000, ⌊0.2 n⌋ , 5p)). This constraint ensures a balance between model expressiveness and computational efficiency while remaining consistent with the stagewise feature expansion procedure. S.II. E MPIRICAL I LLUSTRATION OF M ONOTONICITY To empirically validate the monotonicity property established in Section IV.C of the main script, we analyze the stagewise behavior of the proposed residual-guided framework. We consider the breast cancer dataset and track the evolution of the ridge-regularized training objective J (Gt ), the number of selected hidden features ht , and the residual energy ∥Rt ∥2F

across successive stages of the algorithm. Fig. S.1 illustrates these quantities as the residual-guided construction proceeds. As additional hidden features are incrementally selected based on residual-driven utility, the training objective exhibits a strictly non-increasing trend with respect to both the stage index and the hidden-feature budget. This behavior is consistent with the theoretical guarantee and confirms that each stage makes concrete progress toward reducing the overall objective. Moreover, the residual energy decreases steadily across stages, indicating that newly added features effectively capture previously unexplained components of the supervised signal. These results provide clear empirical evidence that the proposed residual-guided strategy enables stable and consistent optimization behavior, in contrast to one-shot random feature construction.

TABLE S.I: Dataset-wise results on 32 binary datasets. Dataset

RVFL

RG-RVFL†

ELM

RG-ELM†

BLS

RG-BLS†

acute inflammation acute nephritis molec biol promoter parkinsons pittsburg bridges T OR D bank blood breast cancer breast cancer wisc breast cancer wisc diag chess krvkp congressional voting cylinder bands echocardiogram heart hungarian hepatitis horse colic ionosphere mammographic monks 1 monks 2 monks 3 musk 1 oocytes merluccius nucleus 4d oocytes trisopterus nucleus 2f pima spect spectf statlog australian credit statlog german credit statlog heart tic tac toe

100 100 71.8182 81.0256 87.2381 89.6925 76.6380 70.1754 88.4173 93.8457 81.1655 63.2184 65.6349 84.6724 72.4547 85.1613 85.5942 89.1911 79.9196 84.3292 82.3457 91.3382 67.2763 82.2898 78.8314 73.5736 68.3019 79.3431 68.2609 77.2000 81.4815 89.0303

100 100 75.5411 82.0513 89.2381 90.0022 76.9047 70.5263 90.1357 95.2554 88.8291 63.6782 68.7626 85.4416 76.1777 85.8065 86.6864 92.0362 80.0248 93.1532 89.5014 92.7846 73.1206 83.9517 79.8181 74.4877 68.6792 80.4612 68.5507 77.7000 81.8519 92.3653

100 100 68.9610 80.5128 87.1429 89.5378 76.6389 70.1754 88.2785 94.5552 79.4141 63.2184 68.1915 84.6724 76.8615 84.5161 85.6016 90.3380 79.6098 83.4299 82.5193 91.5152 67.2456 82.1908 77.5206 72.6594 67.5472 79.3431 68.1159 76.4000 80.7407 88.7178

100 100 77.3593 81.0256 88.1429 89.8032 77.0398 70.1754 89.5642 95.0784 87.9228 63.4483 69.1357 86.2108 76.1835 85.8065 87.2195 91.4728 79.5056 91.8951 90.8375 93.1466 71.2368 84.1502 80.0390 74.4843 67.9245 79.7135 68.9855 77.9000 80.7407 92.0561

100 100 83.9827 81.0256 89.2381 89.7368 77.3047 70.9074 87.9908 94.2028 84.3870 63.4483 69.5393 83.1624 78.5973 87.0968 85.8793 88.3461 79.0895 76.0505 73.1915 86.8272 76.2654 82.7800 79.0554 72.5236 69.0566 79.7135 68.4058 76.9000 82.2222 96.0143

100 100 85.8874 82.0256 90.1905 89.7589 78.1047 71.5910 88.9928 95.3240 83.7923 63.9080 70.1275 85.1624 77.9018 88.2258 86.0648 89.1952 79.1899 81.9932 74.1942 88.9894 78.8969 84.2688 80.2728 74.0049 71.9434 79.7205 68.7159 77.9000 84.2222 98.3273

Average Accuracy Average Std. Dev. Average Rank

80.92 8.91 1.97

82.92 7.88 1.03

80.82 8.36 1.88

82.76 7.76 1.13

81.65 6.24 1.91

82.78 5.75 1.09

†

denotes the proposed counterpart.

TABLE S.II: Dataset-wise result on 39 multi-class datasets. Dataset

RVFL

RG-RVFL†

ELM

RG-ELM†

BLS

RG-BLS†

abalone annealing arrhythmia audiology std cardiotocography 10clases cardiotocography 3clases contrac energy y1 energy y2 flags glass hayes roth heart cleveland heart switzerland heart va image segmentation iris letter low res spect lung cancer lymphography molec biol splice nursery optical page blocks pendigits pittsburg bridges MATERIAL pittsburg bridges TYPE post operative seeds semeion soybean statlog landsat statlog vehicle teaching vertebral column 3clases wall following wine zoo

63.4179 89.1924 65.4823 69.2949 69.6156 85.2331 40.5239 89.1902 90.3599 52.1188 38.1506 61.2500 59.6995 45.6000 39.5000 87.8355 75.3333 80.3300 87.7588 50.9524 86.4828 51.8809 70.5324 96.2100 95.2491 98.4443 75.8442 42.8571 72.2222 87.6190 84.4321 89.4450 81.5385 81.0846 70.7097 64.8387 77.2369 97.2222 95.0000

63.8732 91.1962 71.0208 69.8718 71.3099 86.2683 41.6098 91.5338 91.9277 54.1430 43.2780 62.5000 60.0437 47.3000 41.0000 90.2597 76.0000 85.4300 89.2647 66.1905 88.4598 58.0251 75.2546 98.1139 95.7974 99.0448 81.5152 49.5238 73.3333 89.5238 91.0227 90.6172 84.6775 83.6881 72.0215 65.4839 79.8034 96.6349 95.0000

63.3465 89.6356 65.7192 67.7821 69.2395 85.3743 40.8666 89.3116 89.9728 52.1053 39.0808 60.0000 58.6721 47.3000 39.0000 86.9697 76.0000 80.0900 87.7605 53.8095 85.8161 51.8809 70.2623 96.4769 95.2490 98.4807 77.8788 42.8571 72.2222 87.1429 84.5565 88.8622 81.7249 81.3227 70.7097 65.1613 77.1454 95.5238 95.0000

63.8729 91.0838 68.3663 69.8333 70.7451 86.2683 41.2729 90.8777 91.5338 53.1309 42.8128 63.1250 59.0601 48.9000 41.0000 90.3030 78.0000 84.7800 89.0760 53.3333 88.4828 57.0219 73.7809 97.9181 95.6696 98.9902 82.4675 50.4762 72.2222 89.0476 89.9552 89.5942 84.5532 82.8625 71.3548 65.1613 80.4447 96.0952 95.0000

63.4657 89.6344 64.1587 69.8333 65.5708 85.9390 45.0847 88.4135 90.1070 53.1309 40.0111 65.6250 59.6885 47.2667 41.0000 88.7446 77.3333 84.7350 86.2511 71.9048 85.7701 68.0564 70.7022 96.6370 95.3771 98.9811 73.9827 42.8571 70.0000 89.5238 87.1925 89.0103 82.3621 81.3241 73.3763 67.4194 78.1901 97.7778 97.0000

63.9856 90.6400 65.2576 70.8718 66.4182 86.8921 45.3691 89.5392 91.3667 53.6302 40.9524 67.7500 59.7104 45.6667 42.0000 90.8596 79.3333 87.9345 88.3560 72.3810 86.0575 69.4679 73.8426 97.1174 96.7607 99.1630 75.1602 49.5238 70.3333 91.9048 92.1602 88.8643 84.8562 80.8514 72.7097 69.6774 80.0443 97.2222 97.0000

Average Accuracy Average Std. Dev. Average Rank

73.33 10.43 1.96

75.94 9.11 1.04

73.34 10.12 1.94

75.35 8.99 1.06

74.96 7.98 1.86

76.20 7.17 1.14

†

denotes the proposed counterpart.

(a)

(b)

(c)

Fig. S.1: Stagewise behavior of the proposed residual-guided framework on the breast cancer dataset, illustrating the monotonic decrease of the ridge-regularized training objective and residual energy as hidden features are incrementally added.

Related documents

• Residual-Guided Randomized Neural Networks#792288
• Residual-Guided Randomized Neural Networks#788288
• Beyond the universal law of robustness: Sharper laws for random features and neural tangent kernels#1158453
• Scale-invariant Gaussian derivative residual networks#1104974
• Randomized subspace correction methods for convex optimization#173322