Event Date
Speaker: Jiaqi Li, William Kruskal Instructor, University of Chicago
Title: "Mathematical Foundations for Machine Learning from a Nonlinear Time Series Perspective"
Abstract: Modern machine learning (ML) algorithms achieve remarkable empirical success, yet providing rigorous statistical guarantees remains a major challenge, particularly in distributional theory and online inference methods. In this talk, we will introduce a novel framework to provide mathematical foundations for ML by bringing powerful tools in nonlinear time series. First, we focus on the stochastic gradient descent (SGD) with constant learning rates. By interpreting the SGD sequence as a nonlinear AR(1) process, we can establish the geometric moment contraction (GMC) for SGD regardless of initializations. By this GMC property, we can derive refined asymptotic theory of SGD and its averaging variant, including general moment convergence, quenched central limit theorems, quenched invariance principles, and sharp Berry-Esseen bounds. Then, we extend this theoretical framework to SGD with dropout regularization, a widely used but theoretically underexplored technique in deep learning. By establishing GMC under explicit learning-rate and dimensional scaling regimes, we obtain asymptotic normality and invariance principles for dropout SGD and its averaged version. These results enable online inference, for which we introduce a fully recursive estimator of the long-run covariance matrix appearing in the limiting distributions. The proposed online confidence intervals with asymptotically correct coverage can be generalized to many other ML algorithms. Overall, viewing online learning algorithms as nonlinear time series provides a powerful toolkit for deriving statistical guarantees in modern ML, with implications for high-dimensional stochastic optimization and real-time uncertainty quantification.
This talk is part of the STA 290 Seminar series.