50 exceptionally difficult research-level math problems.
XX models evaluated models300 expert-written math problems covering advanced undergrad through early career research.
XX models evaluated modelsGitHub issues from real-world Python repos, testing whether models can implement valid code fixes.
XX models evaluated models100 novel puzzles, generated programmatically with a chess engine. Each puzzle has a single best next move.
XX models evaluated models1,000 factoid questions about politics, science and technology, art, sports, geography, music, and more.
XX models evaluated modelsA challenging multiple-choice question set in biology, chemistry, and physics, authored by PhD-level experts.
XX models evaluated models45 competition-style math problems from OTIS, harder than MATH Level 5 but easier than FrontierMath.
XX models evaluated modelsThe hardest tier of problems from the MATH dataset, drawn from competitions like the AMC 10, AMC 12, and AIME.
XX models evaluated models500 GitHub issues from real-world Python repos, testing whether models can implement valid code fixes.
XX models evaluated modelsA benchmark that evaluates models’ performance on a set of challenging programming problems from Exercism, an online programming education platform.
XX models evaluated models