Overparameterized Regression

src: (Arora et al., 2018)^Arora:2018vn, section 3 on $l_p$ regression.

Summary

They show that in the simple scalar linear regression problem, i.e. loss function

if you overparameterize ever so slightly to

then gradient descent turns into an accelerated version.

Thoughts

I remember Arora presenting on this particular result a few years back (or something like this), and finding it very intriguing. I think this goes nicely with the theme of [[statistics-vs-ml]], though it's a slightly different angle. Essentially: we statisticians are afraid of overparameterization, because it removes specificity/leads to ambiguity. But actually, when it comes to these gradient methods, it actually helps to overparameterize!