Abstract: In this talk, I will take you on a tour of large language models, tracing their evolution from Recurrent Neural Networks (RNNs) to the Transformer architecture. We will explore how Transformers elegantly sidestep the vanishing and exploding gradient issues that plagued RNNs. I will introduce neural scaling laws—empirical relationships reminiscent of scaling behaviors common in physics—that predict how model performance improves with increased computational investment. We will also discuss different training paradigms and the key stages a model undergoes, from pretraining through deployment. I will illustrate some of the primary complexities encountered when scaling up large-model training, focusing on performance, resilience, and correctness. Finally, zooming out, I will share my personal perspective on our trajectory toward artificial general intelligence (AGI) and what to expect in the near term.
The Physics Frontiers Centers (PFC) program supports university-based centers and institutes where the collective efforts of a larger group of individuals can enable transformational advances in the most promising research areas. The program is designed to foster major breakthroughs at the intellectual frontiers of physics by providing needed resources such as combinations of talents, skills, disciplines, and/or specialized infrastructure, not usually available to individual investigators or small groups, in an environment in which the collective efforts of the larger group can be shown to be seminal to promoting significant progress in the science and the education of students. PFCs also include creative, substantive activities aimed at enhancing education, broadening participation of traditionally underrepresented groups, and outreach to the scientific community and general public.