On the Spectral Bias of Neural Networks

src: OR

The theory of Implicit Regularisation is used to explain why overparameterised neural networks don't overfit even when no explicit regularisation techniques are used. One of the key points is that we're working in the overparameterised regime – you have multiple interpolating solutions, but during training, neural networks tend to converge to the "simpler" solutions