AI capabilities progress has sped up

We use data from the Epoch Capabilities Index (ECI), which itself builds off the paper “A Rosetta Stone for AI Benchmarks”. The current implementation of ECI filters out all models released before 2023, and further filters any models with less than four ECI-included benchmark scores. These requirements were relaxed for the purpose of improving this retrospective analysis’ scope, allowing earlier models with sparser data to be included.

Specifically, models before 2023 were permitted to have as few as three ECI-included benchmark scores, though all models estimated to be at the frontier had at least 4 benchmarks. This left only two points in the 2019-2020 range, so the data cutoff was extended by only a year to preserve continuity. The analysis spans 149 models over four years, ranging from December 2021 to December 2025.

We found that re-fitting the ECI with extended data changed the models’ rankings very little despite extra data; the largest jump was Claude 3 Haiku (From 97th to 102nd place) and the third-largest jump was a movement by one place. That is, while some granular score shifts may have happened, we are confident that general trends did not change.

Upon obtaining the newly fitted ECI scores, we keep a running maximum in order to determine the model with the highest ECI at each timepoint. This yields 17 frontier points.

Next, we fit a two-segment linear model to frontier ECI vs time, with a single breakpoint and a continuity constraint at the breakpoint. Dates are converted to numeric days for regression purposes, and 5000 breakpoint candidates are chosen between the 10th and 90th percentile of dates (assuming there are no breaks near the edges).

For each candidate breakpoint, we fit a single continuous piecewise‑linear model by OLS using a hinge basis. This jointly optimizes both segments under the continuity constraint (the right‑hand intercept is implied, not fit separately). Candidates are scored by residual sum of squares, and the breakpoint with the lowest RSS is selected.
April 8, 2024 was chosen as the best breakpoint candidate. Our fit has an R2 of 0.9653, with a pre-breakpoint slope of 8.337 ECI/yr, and a post-breakpoint slope of 15.459 ECI/yr. The post-breakpoint slope is 1.85x as large, and can be interpreted as a marked change in the rate of AI progress. We compare this breakpoint model to a simple linear OLS regression in the table below:

We find that the AIC and BIC are both lower in the two-segment model, implying an Akaike Evidence Ratio of 13.5 and a Bayes factor of 5.7. Across 2000 resampled frontier datasets (same size, sampled with replacement), the two-segment model beat the single-line fit on AIC 90% of the time and on BIC about 80% of the time. This reinforces our belief that the breakpoint signal is fairly robust.

We also obtain 90% confidence intervals via a nonparametric bootstrap, resampling the dataset with replacement and refitting the piecewise model with new frontier points each time.

ECI is a composite index constructed from many underlying benchmarks. While we are confident that it tracks a single underlying factor of capabilities progress, interpreting that factor is a bit difficult. For more information, see our ECI FAQ section.

Several data points in our extended ECI dataset feature very wide confidence intervals for capability estimates. Caution should be used when interpreting point estimates for data before 2023.

While we compare piecewise linear models against a simple linear OLS fit, we do not compare against a model of smooth exponential growth. We leave this for future analysis.

Previously, we enforced continuity in the segmented model by adjusting the right intercept such that the two lines meet at the breakpoint. However, it was suggested that this method would make goodness of fit worse on the second line—that is, we could find a better overall fit by optimizing both segments simultaneously. Following this feedback, we edited the methodology. This resulted in negligible (~0.1) differences in our estimates of the ECI progress slopes, and the optimal breakpoint shifted from April 9, 2024 to April 8, 2024. METR progress slopes did not change, and an identical breakpoint was chosen by this method.