Learning in Curved Weight Space:Exponential-Linear Weight Reparameterization for Improved Optimization

Many neural networks operations have a multiplicative nature rather than additive: halving or doubling a norm are analogous relatively but require unequal optimization distances when taking linear steps. Adaptive optimizers such as Adam normalize updates per coordinate, but update steps remain addit