The computational performance of leading AI supercomputers has doubled every nine months

We measure computational performance as the peak number of 16-bit floating-point operations per second (FLOP/s) a supercomputer could theoretically perform based on reported hardware specs. Performance values were either taken directly from reported data or calculated using processor specifications and quantity. After excluding supercomputers without known 16-bit performance data, we retained 482 observations. From these, we identify 57 “leading” AI supercomputers, defined as those that ranked among the top 10 most powerful supercomputers at the time they first became operational. We then run log-linear regressions to estimate annual growth rates in computational performance, number of chips, and per-chip performance for these leading supercomputers. We estimate confidence intervals from the standard errors of each slope.

Our results are as follows:

Note that AI training used different numerical precisions throughout our study period. While 32-bit was still common in 2019, most training likely happened in 16-bit format by the early 2020s. By 2024, some AI training workloads had begun to move to 8-bit precisions. When considering the highest performance available across these three precisions, we find the following trends:

Find an explanation and discussion of numerical precision in Appendix B.8.

We chose to focus on AI supercomputers that became operational in 2019 or later. This date cutoff was predetermined prior to data collection in order to cover the time period with best data availability about supercomputers, and modern machine learning practices. Some large clusters, used for traditional scientific research rather than AI computing, existed prior to 2019, and would appear among the top 10 if the start date were earlier. We assume these are not indicative of current growth trends, but if they are included, the growth rate results would likely be slower, especially in the first half of the study period.