The Epoch Capabilities Index (ECI) combines scores from many different AI benchmarks into a single “general capability” scale, allowing comparisons between models even over timespans long enough for single benchmarks to reach saturation.
ECI is a composite metric which uses scores from 37 distinct benchmarks to generate a single, general capability scale. At a high level, ECI stitches together its component benchmarks, determining their relative difficulty by making comparisons wherever models are evaluated on multiple benchmarks. Individual models obtain higher ECI scores if they perform better on harder benchmarks.
We give an overview of our methodology below; further technical details will be available in our paper, A Rosetta Stone for AI Benchmarks, which was funded by Google DeepMind, and written in collaboration with researchers from their AGI Safety & Alignment team. However, the ECI is an independent Epoch AI product that Epoch has full rights over.
The technical foundation for the ECI comes from Item Response Theory (IRT), a statistical framework originally developed for educational testing. IRT enables comparisons between students, even when they took tests from different years, and one test might be harder than another.
The core of our model is a simple logistic function:
\(\textrm{performance}(m,b) = \sigma(\alpha_b [C_m - D_b])\)
Here, \(\sigma\) represents the logistic function, \(C_m\) is the model’s capability, \(D_b\) is the benchmark’s difficulty, and \(\alpha_b\) is a slope parameter related to the distribution of difficulty across questions within the benchmark. The formula says that a model’s performance on a given benchmark is dependent on how capable it is relative to the difficulty of the benchmark, and how “steep” the benchmark is. Higher \(\alpha_b\) values correspond to “steeper” benchmarks, where individual questions have a narrower range of difficulties, and there is no long tail of much harder questions. These benchmarks tend to saturate quickly as soon as models are able to gain some headway.
We fit model capability (\(C_m\)), benchmark difficulty (\(D_b\)), and benchmark slope (\(\alpha_b\)) parameters that best explain the full set of observed scores. We do not assume any relationship between capability and time or compute inside the model.
We fit the model via non-linear least-squares estimation, using a ridge regularization penalty to discourage overfitting. The scale of the resultant values is arbitrary; we currently rescale so that Claude 3.5 Sonnet is fixed at 130, and GPT-5 is fixed at 150, in order to allow for consistent scoring over recent models in a way that balances communicating our uncertainty with providing detailed information.
We fit the model to benchmark scores covering 42 benchmarks spanning 2023–present, drawn from the Epoch Benchmarking Hub. We use the following benchmarks:
Internal evaluations
Chess Puzzles, FrontierMath Tiers 1-3, FrontierMath Tier 4, GPQA Diamond, MATH Level 5, OTIS Mock AIME 2024-2025, SimpleQA Verified
External benchmark leaderboards
Aider polyglot, APEX-Agents, ARC-AGI-2, BALROG, DeepResearch Bench, Fiction.liveBench, GeoBench, GSO, HLE, SimpleBench, SWE-bench (Bash Only), Terminal-Bench, The Agent Company, VPCT, WeirdML V2
Developer reported scores
ANLI, ARC AI2, ARC-AGI, BIG-Bench Hard, CADEval, CSQA2, Cybench, GSM8K, HellaSwag, LAMBADA, Lech Mazur Writing, MMLU, OpenBookQA, OSWorld, PIQA, ScienceQA, SuperGLUE, TriviaQA, Video MME, WinoGrande
To be used in our methodology, benchmarks need to be scored on a 0-1 scale (or equivalently, 0% to 100%). For benchmarks where random guessing would score above 0 (e.g. those with multiple choice responses), we rescale so that random guessing performance is scaled to zero.
In order to capture the upper end of each model’s capabilities, we aggregate across model evaluation settings (e.g. thinking effort and inference provider), taking the highest score for each benchmark. We only aggregate over models released on the same day with the same name (e.g. we do not aggregate across versions of GPT-4o released on different days). We drop any models with fewer than 4 benchmark scores from our model fit, to avoid low-certainty estimates. We also exclude models released before 2023 due to data sparsity; we hope to increase coverage in the future.
Similar to an IQ test, ECI is designed to capture a broad, underlying capability useful across many tasks, rather than performance on specific skills. ECI summarizes a model’s capability across benchmarks it has been evaluated on, giving more weight to the benchmarks that carry the most signal.
ECI scores are abstract values that can’t be interpreted in isolation, but they can be compared across models to judge meaningful differences in underlying capabilities. See the next question for more details.
Absolute ECI values are meaningless by themselves, but meaningful comparisons can be made between models. The ECI scale is linear; in theory, a 10 point jump should be equally “impressive” when moving from 100 to 110 as it is going from 140 to 150. That said, just because ECI is linear doesn’t mean that other things we care about are linear in ECI units. For instance, at the time of ECI’s launch, a 5 point gain in ECI appeared to roughly correspond to a doubling of the METR Time Horizon. Similar principles hold when looking at trends in ECI; absolute slope values are meaningless, but relative changes in slope indicate faster or slower progress.
The scale of ECI scores is arbitrary. To put values in a convenient range, we have scaled the raw scores so that Claude 3.5 Sonnet = 130 and GPT-5 = 150. As future models and benchmarks are incorporated, these calibration points may be revisited. Note that as with Elo scores, there is no maximum achievable ECI.
Values are not linearly related to benchmark accuracy, since benchmarks differ in both difficulty and slope.
ECI reflects how well a model performs across many benchmarks. Performance on harder benchmarks is more informative about general capability, with benchmark difficulty inferred statistically from overlapping model results (not set by hand).
Models which are highly specialized may receive low ECI scores, despite being very capable within their domain.
As a general principle, Epoch aims to collect benchmarks that are diverse, economically valuable, unsaturated, and widely used. We supplemented our existing set of internally run and benchmark-developer reported scores with a set of older benchmarks, with scores obtained from developer reports (i.e. model cards and technical reports). Having a wider range of benchmark difficulties allows us to obtain more precise ECI estimates.
Benchmarks need to be scored between 0 and 1; for this reason, we left out “ELO” style benchmarks like WebDev Arena.
The model used to produce ECI scores is fit jointly across all data. As new models and new benchmark evaluations are obtained, values may shift slightly even for models whose data have not changed.
It is true that evaluations from model developer reports might be cherry-picked in order to make the model look better. Relying on these evaluations alone could lead to bias in our estimation. We mitigate against this by running models on our own internal evaluations, and by collecting evaluations from independently-run leaderboards. Since these sources are reported regardless of outcome, they reduce the impact of cherry-picking.
Although our approach inherits the benefits of benchmarks, it also inherits its limitations. One shortcoming is that model developers can optimize for high performance on certain benchmarks, so that we overestimate the capabilities of some models.
We strive to cover as many models as possible, with particular focus on plausibly-frontier level models. However, we require a minimum of 4 benchmark evaluations for any model to ensure ECI scores are both stable and fair. If a model you care about doesn’t yet have an ECI score, we are most likely trying to obtain more benchmark scores for it.
See “How do you decide which benchmarks to use?”. We are actively expanding the number of benchmarks we support while balancing considerations like breadth of coverage, relevance, and feasibility of maintenance.
This work was based on research conducted with support from Google DeepMind, and thus draws directly on the methodology introduced in our paper, A Rosetta Stone for AI Benchmarks. However, the ECI is an independent Epoch AI product that Epoch has full rights over. We thank Rohin Shah, Samuel Albanie, Anna Wang, Eli Lifland, Nate Rush, Ezra Edelman, and Isabel Juniewicz.
Have a question? Noticed something wrong? Let us know.