Every layer of a neural network is a matrix multiplication followed by a nonlinearity. Understanding what these matrices do geometrically — how they stretch, rotate, and project — explains why deep learning works.
Activation norms, gradient norms, weight norms — what they tell you, why they explode or vanish, and every practical cause from hyperparameters to data bugs.
Activation norms, gradient norms, weight norms — what they tell you, why they explode or vanish, and every practical cause from hyperparameters to data bugs.