Course: STAT 946 -- Adv Topics: Mathematical Foundations of Deep Learning


Blurb: The goal of this course is to introduce and explore some of the recent theoretical advances that aim to understand modern deep learning methods and training regimes.

Topics may include: Universal approximations, Uniform convergence, Benign overparamterization, functional limit theory/scaling limits, NTK, comparison to kernel methods, Training dynamics and related phenomenology (Implicit regularization, Training regimes, sample complexity, Mean-Field/hydrodynamic limits, DMFT/Effective Dynamics), Loss landscapes and geometry, transformers, diffusion models.

There will be a heavy focus on the analysis of training dynamics (sample complexity and scaling limits). The course will be interspersed with primers on important tools and techniques from probability theory such as concentration of measure, random matrix theory, stochastic analysis, and related ideas inspired by statistical physics.

This will be a fast-paced, research-level, seminar-style course. We will be learning as a group. Students will play an active role in the course: as the course progresses students will pick up and explore these or other topics to catch us all up!

Lectures

Week 1: Approximation theory

• Lec 1: Universal Approximation and Depth Separation
• Reading: Telgarsky's Deep Learning Theory notes, Chap 1,4

Week 2: Uniform convergence

• Lec 2: Generalization and Rademacher Complexity
• Lec 3: Vacuous bounds and the need for a new approach
• Reading:
• P. Bartlett, A Montanari, and A Rakhlin, Deep Learning: A statistical view point Sec 2
• M. Mohri, Foundations of Machine Learning Chap 3
• M. Wainwright, High-dimensional Statistics Chap 4
• V. Nagarajan and J. Zico Kolter, Uniform convergence may be unable to explain generalization in deep learning, NeurIPS 2019

Week 3: Benign overparamterization and Double descent

• Lec 4: TBA
• Lec 5: TBA
• Reading: TBA

Week 4: TBA
Week 5: TBA
Week 6: TBA
Week 7: Reading Week
Week 8: TBA
Week 9: TBA
Week 10: CANCELLED
Week 11: TBA
Week 12: TBA
Week 13: TBA
Week 14: TBA