Convex losses and their applications to SVM, SVR, and Shallow Neural Networks

Convex losses and their applications to SVM, SVR, and Shallow Neural Networks Filippo Portera∗

arXiv:2608.14288v1 [cs.LG] 14 Aug 2026

August, 2026

Abstract We propose multiple new convex losses for SVM and Neural Networks, applied to binary classification tasks. While there are practical limitations in exploiting them with the dual SVM models, we are able to use them with SVM primal formulation and Neural Networks. In detail, the primal SVM problem with the modified losses has been solved with the Particle Swarm Optimization algorithm. We prove that the proposed losses are a generalization of the standard loss, and we experiment them with several small data-sets. This preliminary study shows that using pattern correlations inside the loss function could in theory enhance the generalization performances on some data-sets. To evaluate the performance of each loss, we adopt a Nested Cross-Validation procedure. Results show that generalization measures are the same with or without the new losses.

1

Introduction

The effort here exposed consists in measuring the generalization performances of several Machine Learning algorithms through a set of novel loss functions. We apply a modification of the generalized quadratic loss described initially in [F P04]. The convex loss developed here, theoretically, will perform equal or better than the standard loss, since it is a generalization. We wrote the mathematical steps to obtain two dual objectives functions that can be computed using kernel functions for SVM and SVR. The proposed losses were then applied to some Shallow and Deep Learning tasks, where it was more challenging to measure given its computational cost. A set of different radial basis functions is evaluated. The significance of this work consists on the results obtained with these losses and several settings and data-sets. Code is available at Floss on GitHub. ∗ DAIS, Università ippo.portera@unive.it

degli

studi

”Ca’

Foscari”

1

di

Venezia,

Italy,

Email:

fil-

2

Related work

This study is partially focused on the works [A J02] and [Ron05] for the modified SVM and SVR problems. The introduction of SVM has been done by [C C95]. Concerning shallow Neural Networks, the evaluation of these losses have been possible because of autograd. In fact, the gradients of arbritrary loss functions could be automatically computed due to this procedure (see [MDA15]). Previously, the study of generalized quadratic losses where based on the con√ sideration of ξi variables, while in this study we concentrate on evaluating ξi variables in conjunction with a pattern correlation matrix F . This prospective raises analytical sound dual problems, but their numerical solution is not well understood at present. In addition, we evaluate those losses with Neural Networks on some small UCI data-sets for binary classification tasks. In the next section we present the mathematical details in order to set up the modified SVM dual problems, and describe how to plug the generalized loss inside Neural Networks.

3

Some Machine Learning models for binary classification and regression

In the following, we show the mathematical details of plugging the new convex losses into SVM and SVR models. At last, we describe how to use these losses with Shallow (up to 4 layers). Studying SVM and SVR models, we were unable to find a numerical solution of the emerging problems, since the dual formulation requires also primal variables.

3.1

Problems statement for SVM and SVR

Given the samples x⃗i ∈ Rd , i ∈ [1, .., l] where l is the number of training samples, d is the number of input features, the related target values yi ∈ {−1, 1}, and, for example some RBF functions, the l × l matrix defined below: ′

2

′

′

Fi,j = e−γF ||⃗xi −⃗xj || = e−γF (⃗xi ⃗xi −2⃗xi ⃗xj +⃗xj ⃗xj )

(1)

Where i ∈ [1, .., l], j ∈ [1, .., l], and γF is another hyper-parameter, we would like to solve, for example, the following binary classification problem: l X l p X p 1 2 || w|| ⃗ + C w ⃗ ∗ , b∗ = argminw,b ξi ξj Fi,j ⃗ 2 i=1 j=1

yi (w ⃗ ′ x⃗i + b) ≥ 1 − ξi , ξi ≥ 0,

i ∈ [1, .., l]

i ∈ [1, .., l] 2

(2) (3) (4)

In this setting, for sufficiently big γF , the F matrix becomes the identity matrix, and the problem resorts to the standard Support Vector Classification problem. We add that any set of A = {F1 , . . . , Fn } matrices can be used, as long as the identity matrix I is contained in A, since from the identity matrix it follows the standard case.

3.2

Dual SVC

We derived the relative dual problem through the KKT conditions. The Lagrangian of (2) is: α ⃗ ∗ , ⃗η ∗ , w ⃗ ∗ , b∗ = argmaxα⃗ ,⃗η argminw,b ⃗ L=

(5)

l p l l l X X X X p 1 ξi ξj Fi,j − αi (yi (w ⃗ ′ x⃗i + b) − 1 + ξi ) − ηi ξi (6) ||w|| ⃗ 2+C 2 i=1 i=1 i=1 j=1

With: ξi ≥ 0,

i ∈ [1, .., l]

(7)

αi ≥ 0,

i ∈ [1, .., l]

(8)

i ∈ [1, .., l]

(9)

And: ηi ≥ 0, ∗

∗

∗

∗

The possible saddle points (⃗ α , ⃗η , w ⃗ , b ) are the optimal solutions. The KKT optimality condition on the variable ξi is: l l l p p X ∂L ∂L X p X p =C ( ξj ξk Fj,k + ξi ξk Fj,i ) − αi − ηi = 0 (10) ∂ξi ∂ξi j=1,j̸=i

k=1

k=1

Thus: l X p ∂L ∂L p =C (2 ξi ξj Fi,j + ξi Fi,i ) − αi − ηi = 0 ∂ξi ∂ξi

(11)

l X p 1 C(2 √ ξj Fi,j + Fi,i ) = αi + ηi 2 ξi j=1,j̸=i

(12)

l X p 1 C( √ ξj Fi,j ) = αi + ηi − CFi,i ξi j=1,j̸=i

(13)

j=1,j̸=i

3

Pl C

j=1,j̸=i

√

l X p

C

p ξj Fi,j

ξi

ξj Fi,j =

= αi + ηi − CFi,i

p

ξi (αi + ηi − CFi,i )

(14)

(15)

j=1,j̸=i

The KKT optimality condition on w ⃗ is: l l X X ∂L =w ⃗− αi yi x⃗i = 0 ⇒ w ⃗= αi yi x⃗i ∂w ⃗ i=1 i=1

(16)

The KKT optimality condition on b is: l X

αi yi = 0

(17)

i=1 2

Here K(x⃗i , x⃗j ) = e−γK ||⃗xi −⃗xj || is the dot product in a RKHS parameterised by γK , and the simplified Lagrangian is: α ⃗ ∗ , ⃗η ∗ = argmaxα⃗ ,⃗η

l X i=1

+C

l

l

αi −

1 XX αi yi αj yj K(x⃗i , x⃗j )+ 2 i=1 j=1

l X l p l X X p ξi (αi + ηi ) ξi ξj Fi,j − i=1 j=1

(18)

(19)

i=1

Subject to: ξi ≥ 0, l X

i ∈ [1, .., l]

(20)

αi yi = 0

(21)

i=1

Pl 0 ≤ αi ≤ C

p

j=1,j̸=i

√

ξj Fi,j

ξi

ηi ≥ 0,

! + Fi,i

,

i ∈ [1, .., l]

i ∈ [1, .., l]

(22) (23)

Since the part of the objective function containing the ξ⃗ vanishes, this becomes: α ⃗ ∗ , ⃗η ∗ = argmaxα⃗ ,⃗η

l X

l

αi −

i=1

l

1 XX αi yi αj yj K(x⃗i , x⃗j ) 2 i=1 j=1

(24)

Subject to: l X

αi yi = 0

i=1

4

(25)

Pl

p

j=1,j̸=i

√

0 ≤ αi ≤ C

ξj Fi,j

ξi

! + Fi,i

,

i ∈ [1, .., l]

(26)

With: w ⃗=

l X

αi yi x⃗i

(27)

i=1

and the estimator function can be expressed as: f (⃗x) =

l X

αi yi K(x⃗i , ⃗x) + b

(28)

i=1

Where b can be determined with SVM b. This dual problem still depends on the dual variables ξ⃗ and it is not easy to solve it.

3.3

Dual SVR

The Support Vector Regression primal problem is therefore: l X l q X p 1 2 ∗ || w|| ⃗ + C ξ + ξ ξj + ξj∗ Fi,j w ⃗ ∗ , b∗ = argminw,b i ⃗ i 2 i=1 j=1

(29)

Subject to: yi − w ⃗ ′ ⃗x − b ≤ ϵ + ξi , ′

w ⃗ ⃗x + b ≤ ϵ + ξi∗ ,

i ∈ [1, .., l] i ∈ [1, .., l]

(30) (31)

ξi ≥ 0,

i ∈ [1, .., l]

(32)

ξi∗ ≥ 0,

i ∈ [1, .., l]

(33)

Then, the relative Lagrangian is: ∗

−

l X

∗

α ⃗ ∗, α ⃗ ∗ , ⃗η ∗ , ⃗η ∗ , w ⃗ ∗ , b∗ = argmaxα⃗ ,⃗α∗ ,⃗η,⃗η∗ argminw,b ⃗ L=

(34)

l X l p q X 1 ||w|| ⃗ 2+C ξi + ξ∗ ξj + ξj∗ Fi,j + 2 i=1 j=1

(35)

αi (ϵ + ξi − yi + w ⃗ ′ x⃗i + b) −

i=1

l X

αi∗ (ϵ + ξi∗ + yi − w ⃗ ′ x⃗i − b)+

(36)

i=1

−

l X

ηi ξi −

i=1

l X i=1

With: 5

ηi∗ ξi∗

(37)

ξi ≥ 0,

i ∈ [1, .., l]

(38)

ξi∗ ≥ 0,

i ∈ [1, .., l]

(39)

αi ≥ 0,

i ∈ [1, .., l]

(40)

αi∗ ≥ 0,

i ∈ [1, .., l]

(41)

ηi ≥ 0,

i ∈ [1, .., l]

(42)

ηi∗ ≥ 0,

i ∈ [1, .., l]

(43)

Applying the KKT optimality conditions: l l l X X X ∂L =w ⃗− αi x⃗i + αi∗ x⃗i = 0 ⇒ w ⃗= (αi − αi∗ )x⃗i ∂w ⃗ i=1 i=1 i=1 l l l X X X ∂L (αi − αi∗ ) = 0 αi∗ = 0 ⇒ αi + =− ∂b i=1 i=1 i=1   p l ∗F X ξ + ξ ∂L j j i,j p + Fp,p  = αp + ηp =C ∂ξp ξ + ξp∗ p j=1,j̸=p

 l X ∂L  = C ∂ξp∗

 ξj + ξj∗ Fi,j p + Fp,p  = αp∗ + ηp∗ ξp + ξp∗

(44)

(45)

(46)

p

j=1,j̸=p

(47)

Thus, the dual is: l l l l X X 1 XX (αi − αi∗ )(αj − αj∗ )Ki,j − (αi + αi∗ )ϵ (αi − αi∗ )yi − 2 i=1 j=1 i=1 i=1

(48)

Subject to: l X (αi − αi∗ ) = 0

(49)

i=1



l X

0 ≤ αi ≤ C 

j=1,j̸=i

 0 ≤ αi∗ ≤ C 

l X j=1,j̸=i

 ξj + ξj∗ Fi,j p + Fi,i  , ξi + ξi∗

i ∈ [1, .., l]

(50)

 ξj + ξj∗ Fi,j p + Fi,i  , ξi + ξi∗

i ∈ [1, .., l]

(51)

p

p

And the estimator function is:

6

f (⃗x) =

l X

(αi − αi∗ )K(x⃗i , ⃗x) + b

(52)

i=1

This dual problem still depends on the dual variables ξ⃗ and ξ⃗∗ and, again, we don’t know how to solve it.

3.4

Particle Swarm Optimization for SVM and the new losses

Using the PSO algorithm proposed in [KE95] in combination with a dual solver for standard SVM, we are able to solve the primal SVM problem 2. The dual optimizer is used to pretrain the PSO solver with an optimal standard SVM solution, and then we run PSO to solve the problems with the new losses. In order to obtain a solution with PSO, we used the predictor function 28, and we optimized the variables α ⃗ and b. The number of particles is 20, while the PSO termination criteria is set to 10 iterations of the velocity’s update. The inertia weight W is set to 0.7. At last the acceleration constants are so defined: φp = φg = 1.0 . We have tested this algorithm with several losses and various F matrices.

3.5

Shallow Neural Networks

We try to test the proposed losses with several Neural Networks. For shallow neural networks we pre-compute the F matrix, and then use it on the whole training set. We used models proposed by a LLM, for each specific data-set. In general, for UCI data-sets, they are shallow (max 4 layers) Neural Networks with DropOut and Batch Normalisation layers. We adopted PyTorch as a framework for implementing and optimise those nets. The optimiser chosen is Adam.

4

Data-sets

The binary classification data-set used are taken from UCI Data-sets. All datasets are described in table 1.

5

Method

In order to define these new losses we split the problem into 2 dimensions: how to weight errors and how to define error weighting matrices. The first dimension is explored in the following subsection. While the latter is described in the second subsection.

7

Data-set Sonar Haberman Heart Iono WDBC Breast German

Examples 205 285 304 350 569 699 1000

Features 60 3 13 32 30 9 24

Table 1: UCI Binary Classification data-sets dimensions

5.1

Proposed losses

⃗ the Binary Cross-Entropy with logits We evaluate the following losses. Let β errors on some model output and target values: P

i βi

L0 =

Pl L1 =

i|yi =0

j|yj =0

√ p √ p Pl Pl βi βj Fi,j + i|yi =1 j|yj =1 βi βj Fi,j l

Pl L2 =

Pl

(53)

l

i=1|yi =0

Pl

j=1

√ p Pl Pl √ p βi βj Fi,j + i=1|yi =1 j=1 βi βj Fi,j l √

Pl

i=1

L3 =

i|yi =0

L4 = L1 +

Pl L6 =

j=1 Fi,j

Pl

j|yj =1

√ p βi βj Fi,j

l Pl

i=1

i=1 βi +

Pl

j=1

√ p βi βj Fi,j l

Pl

i|yi =0

Pl

j|yj =1

l

Note that L0 is the standard BCE loss.

8

(55)

Pl

l

Pl

L5 =

βi

(54)

√ p βi βj Fi,j

(56)

(57)

(58)

(59)

5.2

Loss F matrices

We decide to test several F matrices that could take into account pattern similarities or dissimilarities (all of them are parameterised by the γF hyperparameter): Ri,j = e−γF ||⃗xi −⃗xj ||

Qi,j =

Mi,j =

2

(60)

1 1 + γF ||⃗xi − ⃗xj ||2

(61)

1 γF ||⃗xi − ⃗xj ||2

(62)

1+

p

Li,j = e−γF ||⃗xi −⃗xj ||1

(63)

Here || ||1 is the Manhattan distance. Each F matrices can be normalised in order to have a norm that is equal to √ the norm of the identity matrix ( l), with the following formula: √ l F̄ = F (64) ||F || For SVM and PSO, a Nested-Cross-Validation procedure (NCV, [Sto74]) is used, involving 5 outer loops and 3 inner loops for model validation. The hyper-parameters that have been tuned are C, γK and γS . For shallow Neural Networks, we carry on a Nested Cross-Validation procedure involving 5 outer loops and 3 inner loops for models validation, where each NCV has been repeated 10 times to mitigate Neural Networks random initialization effects. The parameter grids is made of the hidden number of neurons, the learning rate and γF . They can be found on the GitHub repository, one for each data-set. The number of the maximim iteration for the scikit-learn SVC solver have to be set (to 100000) in order to prevent optimizations that do not stop. We used a PC equipped with 32 GB of RAM, 11th Gen Intel(R) Core(TM) i7-11700 @ 2.50GHz with 16 cores (hyper-trading on) and a GPU.

6

Results

The results of the application of our losses with several matrices to several data-sets can be found in the following sections. With SVM we describe the SVM algorithm implemented in scikit-learn package. With Adaboost we recall the AdaboostClassifier algorithm implemented in scikit-learn package. The other algorithms refer to Neural Netoworks with out losses. For example: with SVM L 1 R n, we refer to the SVM PSO learning algorithm that uses loss L 1, 9

F = R with no F normalisation. While, with NN L 3 R y, we refer to the Neural Networks learning algorithm that uses loss L 3, F = R with F normalisation.

6.1

Complexity analysis

We discuss some time and space complexity measures about the proposed algorithms. It is well known ([RR07]) that for SVM and data-sets with l samples it is necessary to handle an l×l Gram matrix, hence requiring O(l2 ) evaluation of the kernel and an O(l2 ) memory cost. Therefore they suffer a scalability drawback compared to Neural Networks. We observed also that, for SVM, time complexity depends on l, but also on C, the trade-off constant, and on the other hyperparameters γK and γF . We note that with SVM, for certain hyper-parameter configurations, the training could not converge at all and a maximum number of iterations threshold has been introduced. We used Adaboost with decision trees and its time complexity is O(m d l log (l) 2k )

(65)

where m is the number of boosting iterations, d is the number of features, l is the number of training samples, k is the maximum depth of each decision tree. Its space complexity is O(l d + m 2k ). The PSO algorithm complexity is: O(p i l2 )

(66)

where p is the number of particles, i the number of iterations, and l is the number of training examples. On the Neural Network experiments, the time complexity is mainly driven by the number of epochs e, the number of examples l, the number of features d of each example, the number of network weights w, the batch-size b, and the type of loss adopted. The algorithm that uses a standard loss has 4 distinct functions that are called for each epoch and batch: 1. Batch model computation: O(w b) 2. Criterion computation: O(b) 3. Back-propagation: O(w b) 4. Optimisation(weights update): O(w b) . Therefore, the complexity of the standard loss Neural Network algorithms is O(e bl w b) = O(e l w). The algorithm that uses L(γF ) loss has still 4 distinct functions that are called for each epoch and batch: 1. Batch model computation: O(w b) 10

2. Criterion computation: O(b2 d) 3. Back-propagation: O(w b) 4. Optimisation(weights update): O(w b) . So, comparing each of the 4 functions described above, the second function is smaller than the other 3 functions. Since the search of the best γF value takes g steps (usually g ∈ {3, 4, 5}), the whole model selection algorithm has a time complexity of l w b) = O(g e l w) (67) b About space complexity, the whole F matrix has to be computed once for small data-sets and shallow networks (l × l float numbers). This is feasible as long as a F = (l × l) tensor fits into memory, in combination with the network weights. Nevertheless, for bigger problems, we resorted to use F = (b × b) tensors allocated once per loss computation. O(ge

6.2

Small UCI data-sets with SVM, Adaboost and Neural Networks

The basic SVM algorithm is evaluated with linear, polynomial and RBF kernels, while Adaboost uses DecisionTree classifiers with different estimator maximum depth, estimators number, and learning rate. The PSO optimizer for modified SVM is based on an number of particles equals to 20 and an inertia weight equal to 0.7, while φp = φg = 1. The tables below show the mean accuracy and their standard deviation on 10 different runs of NCV. We show only the top performing 4 algorithms for our losses. We reported also the recorded running time of each algorithm. In order to measure how these new losses behaves w.r.t. to the baseline (our standard SVM implementation and Neural Networks with BCE loss), we report also the results obtained with the standard Binary Cross Entropy with Logits loss. We underscore that the times reported have been measured with the time GNU/Linux command, recording the ’elapsed’ field. The results are described in tables 2, 3, 4, 5, 6, 7, and 8.

7

Conclusions

We introduce several novel loss functions for Machine Learning problems. Concerning the theory behind SVM, we resorted to use Particle Swarm Optimization, and we think it is possibe to apply the same procedure to SVR. Results with these losses on Neural Networks confirm the theory: the generalized losses presents performances that were rarely worse and in general better than the standard case, although we tested it mostly on small problems.

11

Algorithm SVM Adaboost Baseline NN NN L6 R y NN L2 L y NN L6 R n NN L5 L y

Data-set Sonar Sonar Sonar Sonar Sonar Sonar Sonar

Mean accuracy 0.855900 0.839988 0.800058 0.821173 0.823461 0.824971 0.825981

σ 0.021871 0.018232 0.014619 0.017558 0.013800 0.018317 0.021100

Time 3:13a.19 34:58s.84 16:36s.54 24:50s.5 26:00s.9 24:50s.5 20:51s.7

Table 2: Results with Sonar data-set Algorithm SVM Adaboost Baseline NN NN L0 M n NN L0 Q n NN L2 L n NN L6 M y

Data-set Haberman Haberman Haberman Haberman Haberman Haberman Haberman

Mean accuracy 0.72810 0.733379 0.732718 0.738265 0.738265 0.738265 0.744194

σ 0.009135 0.014566 0.008481 0.008556 0.008556 0.006932 0.013071

Time 3:18s.95 17:17s.75 19:06s.80 27:03s.45 27:55s.11 42:55s.24 36:43s.89

Table 3: Results with Haberman data-set Algorithm SVM Adaboost Baseline NN NN L0 R n NN L2 M y NN L4 R y NN L4 L n

Data-set Heart Heart Heart Heart Heart Heart Heart

Mean accuracy 0.842295 0.821454 0.810191 0.825388 0.825388 0.825656 0.828842

σ 0.004432 0.020263 0.011785 0.010731 0.010752 0.006299 0.014692

Time 1:25s.47 13:01.40 10:47s.85 36:10s.67 44:50.75s 1:06:27s 1:07:59s

Table 4: Results with Heart data-set Algorithm SVM Adaboost Baseline NN NN L0 R y NN L2 L y NN L2 L n NN L0 M y

Data-set Iono Iono Iono Iono Iono Iono Iono

Mean accuracy

σ

Time

0.927356 0.850419 0.878893 0.880632 0.881783 0.881799

0.006537 0.010944 0.010237 0.008437 0.013360 0.015062

20:44s.97 19:23s.86 40:46s.75 1:03:26s 1:03:18s 39:52s.06

Table 5: Results with Iono data-set

12

Algorithm SVM Adaboost Baseline NN NN L2 Q n NN L1 M y NN L3 Q n NN L6 L n

Data-set WDBC WDBC WDBC WDBC WDBC WDBC WDBC

Mean accuracy

σ

Time

0.968893 0.974850 0.976642 0.977507 0.977667 0.977680

0.003240 0.004153 0.003436 0.004284 0.003339 0.004365

41:19s.69 20:46s.51 1:00:21s 1:07:19s 59:55s.35 43:51s.30

Table 6: Results with WDBC data-set

Algorithm SVM Adaboost Baseline NN NN L1 R y NN L1 Q n NN L1 M n NN L1 M y

Data-set Breast Breast Breast Breast Breast Breast Breast

Mean accuracy

σ

Time

0.739855 0.676939 0.683206 0.686382 0.688457 0.696467

0.018226 0.027589 0.025450 0.009336 0.008466 0.009899

19:33s.66 1:06:49s 3:08:45s 3:09:45s 3:10:11s 3:10:40s

Table 7: Results with Breast data-set

Algorithm Baseline SVM SVM SVM SVM SVM SVM Adaboost Baseline NN L0 L y NN L1 L n NN L1 L y NN L5 Q n

Data-set

Mean accuracy

σ

Time

German German German German German German German

0.726500 0.728500 0.736900 0.737100 0.737100 0.737900

0.009490 0.010679 0.004949 0.004930 0.005856 0.007077

27:12s.45 2:47:55s 8:32:44s 8:51:40s 8:51:47s 8:31:18s

Table 8: Results with German data-set

13

While SVM reaches better generalization performances than Neural Networks, in 3 of 7 data-sets, in a reduced amount of time, Adaboost scored the first place on the Breast data-set, again in an amount of time that is negligible w.r.t. the time spent to train Neural Networks. However, what we would like to show is that performance with these losses with the described matrices are comparable if not better than the Neural Network baseline. In fact, while an exception is the Iono data-set, where the standard BCE loss gave the best result, with the other data-sets, the expected behaviour is observed. A strategy to build more efficiently and more effective F matrices for Deep Neural Networks should be investigated, in conjunction with bigger data-sets. Next, the Muon optimizer should be used in order to exploit second order derivatives of the loss function and obtain faster a possible minimum ([She+26]). Furthermore, Anisotropic Kernels developed in [F A14] could be evaluated in order to refine the quality of the similarity matrices F . As shown by the experiments, it should be also considered the fact concerning the unique sample distribution of each data-set, so every generalization scheme have to be tested to obtain better performances. Otherwise an algorithm that determines a priori which is the best scheme should be devised. At last, in a graphs binary classification setting, given distinct graph kernels and using these losses we will, hopefully, improve the generalization performances.

8

Acknowledgements

We thank Professor Alessandro Sperduti for helpful suggestions on the theoretical part this work. We thank Professor Gavin C. Cawley for helpful suggestions on the theoretical and experimental part of this paper. We thank Professor Richard W. Hass for helpful comments on an earlier version of this manuscript. We thank Dr. Stephan Kolassa for insightful suggestions on the losses definition.

References [Sto74]

M. Stone. “Cross-validatory choice and assessment of statistical predictions”. In: Journal of the Royal Statistical Society, Series B (Statistical Methodology) (1974).

[C C95]

V. Vapnik C. Cortes. “Support-Vector Networks”. In: Machine Learning (1995).

[KE95]

J. Kennedy and R. Eberhart. “Particle swarm optimization”. In: Proceedings of ICNN’95 - International Conference on Neural Networks. Vol. 4. 1995, 1942–1948 vol.4. doi: 10 . 1109 / ICNN . 1995 . 488968.

14

[A J02]

B.Scholkopf A. J. Smola. Learning with kernels: support vector machines, regularization, optimization, and beyond. Ed. by MIT Press. 2002.

[F P04]

A. Sperduti F. Portera. “A generalized quadratic loss for Support Vector Machines”. In: Proceedings of the 2004 ECAI conference. ECAI. 2004.

[Ron05]

Chih-Jen Lin Rong-En Fan Pai-Hsuen Chen. “Working Set Selection Using Second Order Information for Training Support Vector Machines”. In: Journal of Machine Learning Research (2005).

[RR07]

A. Rahimi and B. Recht. “Random features for large-scale kernel machines”. In: Advances in Neural Information Processing Systems (2007).

[F A14]

M. Donini F. Aiolli. “Learning Anisotropic RBF Kernels”. In: (2014). Ed. by Springer International Publishing.

[MDA15]

Dougal Maclaurin, David Duvenaud, and Ryan P Adams. “Autograd: Effortless gradients in numpy”. In: ICML 2015 AutoML Workshop. Vol. 238. 2015, p. 5.

[She+26]

W. Shen et al. “On the Convergence Analysis of Muon”. In: Transactions on Machine Learning Research (2026).

15

Related documents

• Bias-variance decompositions: the exclusive privilege of Bregman divergences#1075972
• On the Convergence of a Noisy Gradient Method for Non-convex Distributed Resource Allocation: Saddle Point Escape#670365
• On the Convergence of a Noisy Gradient Method for Non-convex Distributed Resource Allocation: Saddle Point Escape#668300
• Computation and Estimation for Neural Networks via Log-Concave Coupling#410666
• Optimization of randomized neural networks for transfer operator approximation#222596