.The Spectrum of Learning: Efficient, Fast, and Robust Training of Massively Overparameterized Deep Neural Networks

Abstract: Modern neural networks, particularly large language models, thrive in the overparameterized regime: scaling capacity enables gradient-based optimization to discover simple, structured solutions that generalize remarkably well. Yet, this scale has turned optimization itself into a prohibitive bottleneck. Today, the primary memory and compute costs stem not from the model weights themselves, but from the gradients and optimizer states maintained alongside them. In this talk, I show that examining the spectral structure of the gradient provides a unifying framework to resolve this tension between scale and efficiency. As training progresses, learning signals concentrate into a surprisingly low-dimensional subspace. I will demonstrate that this concentration is a provable theoretical property across a broad class of architectures, and that exploiting it allows large models to be trained at a fraction of standard memory budgets without degrading quality. Crucially, restricting updates to this compact subspace fundamentally alters computational feasibility. Curvature- and geometry-aware steps, which were originally designed to accelerate training but remain typically computationally prohibitive in the overparameterized regime, can now be computed exactly. This stabilizes convergence while circumventing the pitfalls of popular adaptive optimizers, which often inadvertently amplify noise and mislabeled data. Furthermore, this structural perspective naturally guides optimization toward flatter, more generalizable minima on complex reasoning tasks, and extends directly to pruning redundant components for accelerated inference. Together, these results show that the intrinsic geometry of learning is both a resource to exploit and a signal to respect. Ultimately, this framework establishes principled foundations for designing the next generation of machine learning pipelines: systems that are memory-efficient, fast, and robust by design. Short bio: Yehonathan Refael completed his Ph.D. in Electrical Engineering, under the supervision of Prof. Wasim Huleihel and Dr. Ofir Lindenbaum, specializing in deep learning training and inference optimization, deep learning theory, and privacy in AI. He has an accomplished track record of publications in top-tier venues (such as NeurIPS, ICLR, ICML, JMLR, TMLR, AISTATS, ICANN, and IEEE), alongside multiple patents and industry experience at Amazon, Google, Microsoft, and Samsung.