We explore the fascinating evolution of Large Language Models (LLMs) and their transformative journey through the lenses of computation and optimization. We begin by tracing the origins of LLMs, highlighting how advances in computation and optimization were pivotal in their development. We then delve into the key optimizations that have achieved a staggering 1,000x cost reduction, making LLMs widely accessible even on portable devices. Moving forward, we address the limitations of human-generated data and introduce the concept of constructive hallucination in LLMs. This technique allows for the generation of new hypotheses and their validation through reasoning chains, pushing the boundaries of knowledge creation. Next, we provide an overview of the technology fundamentals and early successes of reasoning models, such as OpenAI's o1 and o3 preview. These models, while significantly enhancing computational capabilities, also exponentially increase computational demands. Finally, we conclude by presenting our ambitious Ultra Ethernet effort, which aims to establish the interconnect standard for future AI workloads. This initiative is crucial in meeting the growing demands at the system level, ensuring seamless and efficient operation in the age of reasoning models.

One point of intersection between Bill Gropp and me is the dear, late, and great Hormozd Gahvari, who was a classmate of mine at Berkeley before leaving to do his Ph.D. at UIUC under Bill. In 2010, Hormozd and Bill published a paper at IPDPS, one that I continually return to as I reflect on prior supercomputer architectures and wonder about their future. ("Wonder" here meaning in the admittedly cartoon-ish way an algorithmic person, as opposed to a systems person, does.) In this talk, I'll reflect on their work and trace how its ideas shaped my own research and views on the machines we have built, the ones we seem likely to build, and what we leave "on the table" when we pursue designs that favor one class of systems over others.

High Performance Computing application codes often contain many tuning parameters and require extensive computational resources. It is desirable to have a machine leaning framework to automate the parameter optimization process with minimal code executions on the actual HPC systems. Another desideratum is a trustworthy digital twin for a physical system, by which we want to quantify the uncertainties of the simulation model with respect to the physical phenomena. For these optimization and uncertainty quantification purposes, a Bayesian statistical learning framework is a powerful tool that can treat the application as a black-box function and use Gaussian Process regression to compute the mean function and the variance in distribution. To this end, we have been developing a public domain software framework called GPTune. For parameter tuning we implemented a number of learning methods including multi-task learning, transfer learning, multi-objective tuning, and multi-fidelity tuning. For uncertainty quantification, we have been developing novel kernels to make them scientific-domain aware, fast linear algebra algorithms to enable large scale GP with millions of data points, and novel Gaussian Process to handle uncertainties in both input data and modeling. We illustrate the versatility of the GPTune software when it is applied to HPC codes ranging from mathematical libraries to complex simulation codes, as well as large-scale scientific apparatuses and instruments.

Scientific computing has undergone a fundamental transformation over the past decade. GPU-accelerated computing, which in 2015 was still at the fringes of supercomputing, has become the norm today. The recent advances — indeed, the revolution — in AI, together with the broader digital transformation of science, have placed HPC at the very center of scientific discovery. No AI-for-Science project can succeed without it. This tremendous opportunity, however, also poses a threat to HPC centers that fail to adapt to the needs of scientists. The success of AI in science depends not only on access to sufficient compute power but equally on data — and data requires curation and domain-centric workflows. Traditional HPC operational models are ill-suited for this. In a recent publication [1], Hoefler et al. introduced the concept of Acceleration as a Service (XaaS), a step in the right direction that leverages container-based deployment of HPC workloads. In this presentation, I will go a step further and demonstrate how supercomputing infrastructures can evolve into service-oriented architectures. Drawing on principles from cloud computing and exploiting features of modern high-performance networks, we can simultaneously serve operational weather prediction, climate simulation, AI, traditional HPC, and large-scale experimental and observational instruments — all through specialized, elastic platforms that do not compromise scalability or performance. [1] T. Hoefler et al., “XaaS: Acceleration as a Service,” arXiv preprint, arXiv:2401.04552, Jan. 2024.

We focus on the scientific and engineering advances being driven by HPC together with advanced statistical methods featuring artificial intelligence/deep learning/machine learning (AI/DL/ML). Accelerated progress in delivering scientific discovery via predictions with these methods guided by Validation, Verification & Uncertainty Quantification (VVUQ) are enabling data-driven discoveries essential for realizing the grand challenge potential of fusion energy. The best validated fusion devices are tokamaks, which are vulnerable to large scale plasma instabilities called “disruptions” that can stop the reaction and damage the device. Reliably predicting and avoiding these damaging events is essential, and recent advances feature successful capabilities involving the deployment of recurrent and convolutional neural networks in Princeton's Deep Learning Code – FRNN – for carrying out efficient transfer learning across very different tokamak devices. Essential validation vs. a huge data base is documented and highlights the applicability of FRNN to both the large JET and smaller DIIID. It is further shown that this AI/DL capability can provide not only the “disruption score,” as an indicator of the probability of an imminent disruption but also a “sensitivity score” in real-time to indicate the underlying reasons for the predicted event. Consequently, this deep learning capability provides the detailed information for a plasma control system (PCS) that can help improve disruption avoidance in real-time and to thereby optimize plasma performance. Application of such AI/DL methods for real-time prediction and control has been further advanced with the innovative introduction of a surrogate model simulator (SGTC). These capabilities are now leading to exciting avenues for moving from passive prediction to active control and ultimately, to the optimization of the design for a first-of-a-kind fusion pilot plant.

The end of Dennard or frequency scaling in computer processors nearly twenty years ago has caused a transformation in computing. Innovations in computer architecture have enabled continued improvements in performance, but at the cost of increasing software and algorithmic complexity. Software has also undergone major transformations, making it both easier and harder to exploit the changes in hardware. This talk will provide some background on the transformations in computing over the last two decades, how the National Center for Supercomputing Applications has adapted to these changes, and how performance engineering can help guide the developments of algorithms and applications in this rapidly changing environment.

High-performance computing (HPC)—including modeling and simulation, data analytics, machine learning, and AI—is a prevalent means of discovery and innovation in essentially all areas of science, engineering, technology, and societal advancement. This presentation will introduce the history of computing at Argonne National Laboratory, including projects where Bill Gropp’s fundamental contributions have led to advances in algorithms and software that power discovery across the globe. We’ll also discuss challenges in next-generation ecosystems for scientific computing, including opportunities for harnessing community, software, and AI for advances in cross-disciplinary team science.

Bill Gropp's technical work spans many topical areas and demonstrates his diverse expertise. In mimicry, I will present three mini-talks on three recent topics I have been involved with, alongside a diverse group of co-authors. The use of MPI-based mathematical libraries from pure OpenMP codes has been neglected; I will present some recent work in that area with PETSc linear solvers. Secondly, I will provide some technical background on recent work to improve LLM responses to PETSc-related questions. Finally, I will present work on updating the automatic generation of full Fortran 2018 bindings for PETSc, as Bill provided the initial Fortran stub generation for both PETSc and MPICH.

Over its 37-year history, the Gordon Bell Prize has strongly influenced the development of high performance simulation and big data statistics in mostly positive, but also in some negative, ways, with implications for the operations of leading supercomputer centers. We briefly discuss the evolution of the prize in its several awarded categories then focus on a handful of submissions of influence. For example, driven by insights from Bill Gropp, the 1999 Gordon Bell Special Prize paper that first addressed an unstructured grid PDE application is presented as a pre-cursor of the 2009 roofline model now universally used to assess whether an application is compute- or bandwidth-bound, or even instruction-bound at a finer examination. The flop/s rate orientation of the Prize has encouraged wasteful flops, where a smaller number would provide more cost-effective delivery of quantities of scientific interest to working accuracy; we provide examples from four of our own recent submissions. How to pilot the Prize at the confluence of the roaring rivers of simulation and learning, with the tributary of quantum computing about to join, is the final topic of discussion. As energy costs now dominate what can be simulated and what can be learned or inferred, the elusive metric of “science per Joule” is proposed as one of several aspects to be addressed in the future evolution of this inspiring legacy.