Click here to flash read.
We prove closed-form equations for the exact high-dimensional asymptotics of
a family of first order gradient-based methods, learning an estimator (e.g.
M-estimator, shallow neural network, ...) from observations on Gaussian data
with empirical risk minimization. This includes widely used algorithms such as
stochastic gradient descent (SGD) or Nesterov acceleration. The obtained
equations match those resulting from the discretization of dynamical mean-field
theory (DMFT) equations from statistical physics when applied to gradient flow.
Our proof method allows us to give an explicit description of how memory
kernels build up in the effective dynamics, and to include non-separable update
functions, allowing datasets with non-identity covariance matrices. Finally, we
provide numerical implementations of the equations for SGD with generic
extensive batch-size and with constant learning rates.
No creative common's license