About this question
I was reading the paper Weight Normalisation: A Simple Reparameterization to Accelerate Training of Deep Neural Networks about improving the learning of an ANN using weight normalisation.
They consider standard artificial neural networks where the computation of each neuron consists in taking a weighted sum of input features, followed by an elementwise nonlinearity
y=ϕ(x⋅w+b)where w is a k-dimensional weight vector, b is a scalar bias term, x is a k-dimensional vector of input features, ϕ(⋅) denotes an elementwise nonlinearity and y denotes the the scalar output of the neuron.
They then propose to reparameterize each weight vector w in terms of a parameter vector v and a scalar parameter g and to perform stochastic gradient descent with respect to those parameters instead.
w=g∥v∥vwhere v is a k-dimensional vector, g is a scalar, and ∥v∥ denotes the Euclidean norm of v. They call this parameterization weight normalisation. What is this scalar g used for, and where does it come from? Is w is the normalized weight? In general, how does weight normalization work? What is the intuition behind it?
