Epoch’s Capabilities Index stitches together benchmarks across a wide range of difficulties

The ECI is calculated using data from Epoch’s Benchmarking Hub. We utilize both internally run evaluations, as well as benchmark creator- and model developer-reported evaluations. Currently, the ECI uses 1123 distinct evaluations, covering 147 models and 39 underlying benchmarks. More details on the methodology used for the ECI can be found here.

ECI is based on a statistical model that estimates three types of parameters. Each AI model is given an estimated capability (which we use to produce the ECI), and each benchmark is given location and slope parameters, indicating the model’s overall difficulty and the range of difficulties among its constituent problems, respectively.

These parameters are estimated by fitting the observed data to the following formula:

$$ \textrm{performance}(m,b) = \sigma(\alpha_b [C_m - D_b]) $$

Where \(C_m\) is capability for model \(m\) (i.e. ECI), \(D_b\) is overall difficulty for benchmark \(b\), and \(\alpha_b\) is the “slope” for benchmark \(b\).

To back out the expected performance on each benchmark across a range of ECI scores, we simply plug in the estimated \(\alpha_b\) and \(D_b\) values for each benchmark, and plot how performance changes as \(C_m\) varies.

We calculate 90% confidence intervals by bootstrap resampling from our observed scores, fitting a new model for each resample, and calculating the 5th and 95th percentile for expected scores across samples.

This analysis focuses on the expected score on several benchmarks for a given ECI score. The plotted 90% confidence intervals represent a confidence interval over the expected performance, not a prediction interval for the performance obtainable for a given ECI. In practice, individual models may perform above or below the 90% confidence interval.

More information on the technical details and limitations of ECI can be found on the ECI tab of the AI Benchmarking hub.