skip to main content

Mathematics & Machine Learning Seminar

Tuesday, October 31, 2023
2:00pm to 3:00pm
Add to Cal
East Bridge 114
Mildly Overparameterized ReLU Networks Have a Favorable Loss Landscape
Guido Montufar, Departments of Mathematics and Statistics & Data Science, UCLA,

We study the loss landscape of two-layer mildly overparameterized ReLU neural networks on a generic finite input dataset for the squared error loss. Our approach involves bounding the dimension of the sets of local and global minima using the rank of the Jacobian of the parametrization map. Using results on random binary matrices, we show most activation patterns correspond to parameter regions with no bad differentiable local minima. Furthermore, for one-dimensional input data, we show most activation regions realizable by the network contain a high dimensional set of global minima and no bad local minima. We experimentally confirm these results by finding a phase transition from most regions having full rank to many regions having deficient rank depending on the amount of overparameterization. This is work with Kedar Karhadkar, Michael Murray, and Hanna Tseran.

For more information, please contact Math Department by phone at 626-395-4335 or by email at [email protected].