## Intro Over the last few weeks, I've come across companies and researchers working on a wide range of evals for AI agents. This includes evals for [human-AI multiplayer games](https://olamlabs.ai/), [equity research](https://www.paperinstruments.com/blog/diligence-bench), [robotics](https://robocurve.org/), and [long-horizon computer use](https://osworld-v2.xlang.ai/). My goal with this post is to answer what makes a good eval. I trace the eval landscape from the GPT-4 era to today, define what an eval is useful for, and propose six traits that the surviving evals share. I then also go through errors that researchers have documented in trusted evals. ## Eval-ution! I think the best place to start tackling our primary goal is to go all the way back to the benchmark landscape that existed when GPT-4 was released, and then trace what researchers published next. > Leaderboard note: “Current highest score” is the top result shown on the linked public leaderboard as of July 22, 2026. ### Benchmarks already established by 2023 These benchmarks formed much of the evaluation vocabulary inherited by the GPT-4 era: | Eval name | Published | One-liner | Current Highest Score (Model) | | --- | ---: | --- | --- | | [ARC-Challenge](https://allenai.org/data/arc) | 2018 | Harder multiple-choice science questions drawn from grade-school examinations. | [96.3% (GPT-5)](https://pricepertoken.com/leaderboards/benchmark/arc-challenge) | | [HellaSwag](https://rowanzellers.com/hellaswag/) | 2019 | Commonsense questions where the model must select the most plausible continuation of an everyday situation. | [95.3% (GPT-4o)](https://www.codesota.com/benchmark/hellaswag) | | [DROP](https://arxiv.org/abs/1903.00161) | 2019 | Reading comprehension requiring arithmetic and discrete reasoning over passages. | [91.6 F1 (DeepSeek-V3)](https://aiwiki.ai/wiki/drop) | | [MMLU](https://github.com/hendrycks/test) | 2020 | About 15,900 multiple-choice questions across 57 academic and professional subjects, including mathematics, medicine, law, history and computer science. | [94.0% (GPT-5.4)](https://lmmarketcap.com/benchmarks/mmlu) | | [MATH](https://github.com/hendrycks/math) | 2021 | 12,500 competition-style mathematics problems spanning algebra, geometry, number theory, counting and probability, and precalculus. | [99.0% (GPT-5.2 Pro)](https://pricepertoken.com/leaderboards/benchmark/math) | | [HumanEval](https://github.com/openai/human-eval) | 2021 | Python code generation graded by executing unit tests. | [97.6% (Claude Sonnet 4.5 Thinking)](https://pricepertoken.com/leaderboards/benchmark/humaneval) | | [GSM8K](https://github.com/openai/grade-school-math) | 2021 | Multi-step reasoning through grade-school mathematics word problems. | [99.6% (MiMo-V2.5-Pro)](https://llm-stats.com/benchmarks/gsm8k) | | [BIG-Bench Hard](https://github.com/suzgunmirac/BIG-Bench-Hard) | 2022 | Multi-step logical and symbolic reasoning through tasks involving deduction, arithmetic, causal judgement, date calculation, object tracking and navigation. | [96.1% (Gemini 3.1 Pro Preview)](https://pricepertoken.com/leaderboards/benchmark/bbh) | ### 2023 Evals | Eval name | One-liner | Current Highest Score (Model) | | --- | --- | --- | | [MathVista](https://github.com/lupantech/MathVista) | Mathematical reasoning over visual inputs including charts, diagrams, plots, geometric figures and scientific images. | [90.7% (Seed 2.1 Pro)](https://llm-stats.com/benchmarks/mathvista) | | [WebArena](https://webarena.dev/) | Agents complete realistic tasks on interactive websites by navigating pages, filling forms and changing site state. | [71.6% (OpAgent)](https://aiwiki.ai/wiki/webarena) | | [AgentBench](https://github.com/THUDM/AgentBench) | Agents interact with multiple environments, including operating systems, databases, knowledge graphs, games and web-shopping tasks. | — | | [SWE-bench](https://www.swebench.com/) | Real GitHub issues requiring a model or agent to modify an existing repository and pass its tests. | [52.62% (Sonar Foundation Agent + Claude 4.5 Opus)](https://www.swebench.com/) | | [GPQA Diamond](https://github.com/idavidrein/gpqa) | Expert-written graduate-level science questions designed to resist simple retrieval. | [94.1% (GPT-5.6 Sol max / Gemini 3.1 Pro Preview)](https://artificialanalysis.ai/evaluations/gpqa-diamond) | | [MMMU](https://mmmu-benchmark.github.io/) | College-level reasoning over charts, diagrams, tables, maps and other visual material. | [86.0% (Qwen3.6 Plus)](https://llm-stats.com/benchmarks/mmmu) | | [IFEval](https://github.com/google-research/google-research/tree/master/instruction_following_eval) | Verifiable instruction-following tasks involving constraints such as format, length, included content and excluded content. | [95.0% (Qwen3.5-27B)](https://llm-stats.com/benchmarks/ifeval) | | [Needle-in-a-Haystack](https://github.com/gkamradt/LLMTest_NeedleInAHaystack) | Retrieval of a small piece of information hidden inside an increasingly long context. | — | ### 2024 Evals | Eval name | One-liner | Current Highest Score (Model) | | --- | --- | --- | | [AIME 2024](https://maa.org/maa-invitational-competitions/) | Problems from the 2024 American Invitational Mathematics Examination used to test competition-level mathematical reasoning. | [95.8% (Grok-3 Mini)](https://llm-stats.com/benchmarks/aime-2024) | | [LiveCodeBench](https://livecodebench.github.io/) | Continuously updated coding-contest problems intended to reduce contamination from training data. | [91.7% (Gemini 3 Pro Preview, high)](https://artificialanalysis.ai/evaluations/livecodebench) | | [OSWorld](https://osworld-v1.xlang.ai/) | Computer-use tasks carried out through browsers, terminals and real desktop applications. | — | | [MMLU-Pro](https://github.com/TIGER-AI-Lab/MMLU-Pro) | A harder, more reasoning-intensive revision of MMLU with ten answer choices and more difficult questions. | [89.8% (Gemini 3 Pro Preview, high)](https://artificialanalysis.ai/evaluations/mmlu-pro) | | [τ-bench](https://taubench.com/) | Tool use, state updates and policy compliance in multi-turn customer-service environments. | [56.0% airline (o4-mini high / Claude 3.7 Sonnet)](https://hal.cs.princeton.edu/taubench_airline) | | [SWE-bench Verified](https://openai.com/index/introducing-swe-bench-verified/) | A human-validated subset of SWE-bench containing 500 repository-level software tasks. | [79.2% (live-SWE-agent / Sonar + Claude 4.5 Opus)](https://www.swebench.com/) | | [SimpleQA](https://openai.com/index/introducing-simpleqa/) | Short, fact-seeking questions with unambiguous answers, designed to measure factual accuracy and hallucination. | [57.9% (DeepSeek V4 Pro Max)](https://benchlm.ai/benchmarks/simpleQa) | | [Aider Polyglot](https://aider.chat/docs/leaderboards/) | Code-editing exercises across multiple programming languages, graded by applying the model's diff and running tests. | [88.0% (GPT-5 high)](https://aider.chat/docs/leaderboards/) | | [MATH-500](https://huggingface.co/datasets/HuggingFaceH4/MATH-500) | A 500-problem subset of MATH used to compare advanced mathematical reasoning with cheaper evaluation runs. | [99.4% (GPT-5 high)](https://artificialanalysis.ai/evaluations/math-500) | ### 2025 Evals | Eval name | One-liner | Current Highest Score (Model) | | --- | --- | --- | | [Humanity's Last Exam](https://lastexam.ai/) | About 2,500 expert-written questions across difficult academic subjects, designed to remain challenging for frontier models. | [53.3% text-only (Claude Fable 5, max)](https://artificialanalysis.ai/evaluations/humanitys-last-exam) | | [AIME 2025](https://maa.org/maa-invitational-competitions/) | Problems from the 2025 AIME I and II competitions used to test advanced mathematical reasoning. | [100% (GPT-5.2 xhigh)](https://artificialanalysis.ai/evaluations/aime-2025) | | [SWE-Lancer](https://openai.com/index/swe-lancer/) | Real freelance software-engineering tasks drawn from Upwork, with task values derived from historical payouts. | [$45,625 earned; 28.4% accuracy (o1, Diamond subset)](https://swelancer.github.io/leaderboard/) | | [Vending-Bench](https://andonlabs.com/evals/vending-bench) | Long-horizon operation of a simulated vending-machine business, scored by the money the agent finishes with. | [$3,769.70 minimum net worth (Gemini 3 Pro)](https://andonlabs.com/evals/vending-bench) | | [BrowseComp](https://openai.com/index/browsecomp/) | Difficult factual questions whose answers are deliberately hard to locate through ordinary web search. | [92.2% (GPT-5.6 Sol)](https://benchlm.ai/benchmarks/browseComp) | | [Finance Agent](https://github.com/vals-ai/finance-agent) | Financial-analysis questions requiring tool-assisted research across companies, financial statements and SEC filings. | [64.4% on v1.1 (Claude Opus 4.7)](https://www.vals.ai/benchmarks) | | [Terminal-Bench](https://www.tbench.ai/benchmarks) | Multi-step tasks completed inside isolated terminal environments, including software engineering, system administration, security and data analysis. | [64.5% (Apex2 + Claude Sonnet 4.5)](https://www.tbench.ai/leaderboard/terminal-bench/1.0) | | [τ²-bench](https://github.com/sierra-research/tau2-bench) | Multi-turn customer-service tasks where an agent must communicate with a simulated user, use tools, follow policy and correctly modify a stateful environment. | [99.1% telecom (JT-35B-Flash / GLM-5.2 max)](https://artificialanalysis.ai/evaluations/tau2-bench) | | [OSWorld-Verified](https://github.com/xlang-ai/OSWorld) | A corrected and more reliably verified revision of OSWorld's computer-use tasks and evaluation setup. | [85.0% (Claude Fable 5)](https://benchlm.ai/benchmarks/osWorldVerified) | | [SWE-bench Pro](https://labs.scale.com/leaderboard/swe_bench_pro_public) | More difficult, contamination-resistant software-engineering tasks drawn from public and private repositories across several programming languages. | [61.5% (Muse Spark 1.1)](https://labs.scale.com/api/pdf/leaderboard/swe_bench_pro_public) | | [GDPval](https://openai.com/index/gdpval/) | Work products from economically valuable tasks across 44 occupations, compared with work produced by experienced professionals. | — | | [MCP-Atlas](https://labs.scale.com/leaderboard/mcp_atlas) | Multi-step tasks requiring an agent to discover and coordinate tools exposed through real Model Context Protocol servers. | [88.1% pass rate (Muse Spark 1.1)](https://labs.scale.com/leaderboard/mcp_atlas) | | [Toolathlon](https://github.com/hkust-nlp/Toolathlon) | Long-horizon workflows spanning 32 applications and hundreds of tools, graded through executable end-state checks. | [75.6% (Muse Spark 1.1)](https://benchlm.ai/benchmarks/toolathlon) | | [Terminal-Bench 2.0](https://www.tbench.ai/news/announcement-2-0) | A larger and more rigorously reviewed successor to Terminal-Bench for long-running command-line work. | [84.7% (NexAU-AHE + GPT-5.5)](https://www.tbench.ai/leaderboard/terminal-bench/2.0) | | [Vending-Bench 2](https://andonlabs.com/evals/vending-bench-2) | A revised simulation of running a business over a full year, with richer operations and financial outcomes than the original benchmark. | [$10,936.76 average balance (Claude Opus 4.7)](https://andonlabs.com/evals/vending-bench-2) | | [OfficeQA](https://www.databricks.com/blog/introducing-officeqa-benchmark-end-to-end-grounded-reasoning) | Questions requiring retrieval, parsing, aggregation and reasoning across roughly 89,000 pages of U.S. Treasury Bulletins. | — | | [MRCR v2](https://openai.com/index/introducing-gpt-5-2/) | Long-context conversations in which a model must reproduce multiple requested pieces of information embedded among similar material. | [93.6% (Sakana Fugu-Ultra)](https://benchlm.ai/benchmarks/mrcrv2) | ### 2026 Evals (so far) | Eval name | One-liner | Current Highest Score (Model) | | ----------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | | [APEX-Agents](https://mercor.com/apex/apex-agents-leaderboard/) | Long-horizon, cross-application tasks created by investment bankers, management consultants and corporate lawyers. | [43.3% pass@1 (Fable 5 max)](https://www.mercor.com/apex/apex-agents-leaderboard/) | | [GDPval-AA](https://artificialanalysis.ai/evaluations/gdpval-aa) | An agentic implementation of GDPval with browsing and shell access, scored through blind pairwise comparisons and Elo ratings. | [1,744 Elo on v2 (Claude Fable 5, max)](https://artificialanalysis.ai/evaluations/gdpval-aa) | | [AutomationBench](https://zapier.com/blog/introducing-automationbench/) | Cross-application business workflows requiring API discovery, policy compliance and correct state changes across several systems. | [18.1% (GPT-5.6 Sol max)](https://zapier.com/benchmarks) | | [Terminal-Bench 2.1](https://www.tbench.ai/news/terminal-bench-2-1) | Multi-step tasks performed in sandboxed terminal environments, covering software engineering, system administration, security, data analysis and scientific computing. | [83.8% (Claude Code + Fable 5 xhigh)](https://www.tbench.ai/leaderboard/terminal-bench/2.1) | | [DeepSWE v1.1](https://deepswe.datacurve.ai/) | Original, long-horizon software-engineering tasks with reference solutions kept out of public repositories and purpose-built verifiers. | [73% (GPT-5.6 Sol max)](https://deepswe.datacurve.ai/) | | [Agents' Last Exam](https://agents-last-exam.org/) | Long-horizon professional workflows with verifiable outcomes across 55 fields and 13 industry clusters. | [30.6% pass rate; 53.6% partial-credit score (Codex + GPT-5.6 Sol)](https://agents-last-exam.org/leaderboard) | | [OSWorld 2.0](https://osworld-v2.xlang.ai/) | End-to-end computer-use workflows spanning multiple applications, with a median human completion time of about 1.6 hours. | [20.6% binary completion; 54.8% partial score (Claude Opus 4.8 max, batched tools)](https://osworld-v2.xlang.ai/) | If we carefully read the one-liner description of each eval, we will note the following: **Pre-2023**: we had a set of static, single-turn questions: multiple choice (ARC, HellaSwag, MMLU), short answers (GSM8K, DROP) or self-contained functions graded by unit tests (HumanEval). We primarily focused on general knowledge. The model answers a question and a very simple string match decides whether the answer was right or wrong. **2023**: GPT-4 mowed down almost every pre-2023 eval, and researchers responded to this by increasing the difficulty of academic benchmarks (GPQA Diamond), adding multimodal tests (MathVista, MMMU), and pairing models with execution environments (WebArena, AgentBench, SWE-bench). **2024**: at first pass, I thought 2024 was not all that interesting in terms of the evals it brought. But [Florian Brand](https://florianbrand.com/) correctly pointed out that OSWorld, SWE-bench Verified and Aider were highly consequential evals. **2025**: Along with pre-existing evals, we see a new class: those that attach economic value to model outputs. Good examples of this are SWE-Lancer, Vending-Bench, and GDPval. We also get evals measuring long-horizon capabilities via benchmarks like Toolathlon and Terminal-Bench. **2026**: the most ambitious year with long-horizon capabilities and economically grounded work being core themes. We're trying to measure agents over days, weeks, and months on increasingly ambiguous tasks in finance, law, and healthcare. In addition to measuring agents, we are also measuring everything around the agent: harnesses, available infra, and more. We're also seeing capital flowing into companies building [unique evals](https://www.ycombinator.com/companies?batch=Summer%202026&batch=Fall%202026&batch=Spring%202026&batch=Winter%202026&batch=Fall%202025&query=evals). Here's a good example of a "unique" eval: [Drone-Bench](https://andonlabs.com/evals/drone-bench). ## Traits shared by good evals [Thousands of benchmarks](https://benchmarklist.com/benchmarks/) have been published, and most fail to be good signals of model capabilities. The evals in these tables are still cited and still reported by labs, which gives me a naive definition of what I mean by a good eval: one the research community and the frontier labs continue to trust. In order to get to a sharper definition, I think it might help to define precisely why an eval is useful. A model can be applied to an enormous space of tasks, and an eval samples a small set of them from one region of that space. If a model scores well, we can reasonably guess that it will handle other tasks that resemble the sample. So the eval acts as a proxy for the capability. The proxy is incomplete, though: models regularly turn out to be good at tasks no benchmark sampled, and people tend to discover those abilities in use, before an eval for them exists. [Terminal-Bench 2.1](https://hub.harborframework.com/datasets/terminal-bench/terminal-bench-2-1/6?tab=tasks), for example, contains 89 self-contained tasks, each with a defined success check: recovering a corrupted SQLite database, compiling legacy software from source, cracking a password-protected archive. A model that scores well can likely handle similar well-specified problems when they are handed to it. In practice, a high score means an engineer can delegate this kind of work. These tasks are often inputs into a larger job, like architecting software from ambiguous requirements, and today's coding benchmarks, in my opinion, tell us nothing about that ability. A frequent complaint from even heavy users of these models is that they struggle in massive codebases and fail at boundary conditions. Here is one of the tasks in Terminal-Bench 2.1: analyzing a chess position from an image, using a chess engine to find the best moves, and handling multiple valid solutions. This task tells me a lot about a model's general capabilities; a model that can do this in one go is undoubtedly quite capable. But at the same time, it does not immediately tell me whether I can trust it with a thorny software architecture problem. ![[terminal-bench-chess-task.png|500]] Still, I would take a rough proxy over measuring nothing. A score we can track over time is worth having, even with these caveats. Across the list of trusted evals, I see a few core traits: 1. **An unambiguous score.** A reader should be able to interpret the result without much effort. 2. **Headroom.** An eval close to its ceiling stops producing signal. The tables above show this as a rough lifecycle: pre-2023 benchmarks sit at 91-99% today and are mostly no longer reported in model release announcements, and each later wave of evals can be considered a response to the wave before it saturating. Many 2026 evals debut with top scores below 50%, making headroom a design requirement. 3. **A bullet-proof grader.** The grader is the mechanism that turns model output into the score. The expensive human work happens while building it (coming up with tasks and rubrics) and then we automate the scoring with code checks or an LLM-as-judge type mechanism. 4. **A pristine task list.** Every task is solvable, unambiguous in its requirements, and tests what the benchmark claims to measure. 5. **Hermetic.** The eval should mean the same thing over time. 6. **Reproducible.** The measurement setup is specified, and rerunning it gives the same number within a stated error. Ideally the setup is also cheap for others to rerun, though for evals graded by experts or run over long horizons that may be an unreasonable ask. There's a lot of low-hanging fruit in building infrastructure that ensures any future eval can have these traits. I believe [Prime Intellect](https://www.primeintellect.ai/) is doing an excellent job in this regard. They abstract away a lot of complexity: you can define your own evals and run them directly on their infrastructure. They take care of sandboxing, parallelization and telemetry. ![[prime-intellect-eval-infra.png]] Another example is [BackSearch by General Reasoning](https://www.gr.inc/releases/introducing-backsearch), search infrastructure for forecasting evals. Forecasting tasks are graded after the outcome is known, so the model's search access has to be restricted to what was on the web before the resolution date. BackSearch serves an archived index for this: queries are gated on each page's crawl date, and fetches return the page as it was archived at the time. ## Failures in trusted evals Now that we have defined our core traits, it's also important to see issues that currently plague eval development. Several groups have examined how well the trusted evals hold up. The errors they found fall into three groups, depending on where the problem lives: in the tasks, in the world around the tasks, or in the measurement setup. Each group corresponds to one of the traits above. ### Errors in the tasks themselves The clearest recent example is [OpenAI's audit of SWE-bench Pro](https://openai.com/index/separating-signal-from-noise-coding-evaluations/). They went through the public task set and concluded that roughly 30% of the tasks were broken: either the hidden tests graded incorrectly, or the task description misled the agent about what those tests expected. One example from the post: the instructions ask for one space at the beginning of each line, while the hidden test requires two. OpenAI had previously recommended the benchmark, and withdrew that recommendation after the audit. [Florian Brand](https://florianbrand.com/posts/benches-2026) describes a vision benchmark whose tasks stack abilities on top of the one being tested: to answer, a model must read a number plate, count letters, and multiply, so the score mixes vision with arithmetic and results in a lossy attribution to either. The ground truth in this example was also ambiguous. ### Errors from the world changing Some tasks are correct on the day they are written and degrade afterwards. [Epoch's review of GDPval](https://epoch.ai/publications/what-do-economic-value-benchmarks-tell-us) found that over half of its tasks depend on live web access. Some ask the model to analyze a period like "H1 2025," which was in the future when the benchmark released. The period has since passed, and the answer can now be looked up. A small share of tasks reference websites that no longer resolve. Contamination is a special case of the same problem. [Florian found](https://x.com/xeophon/status/2006969664346501589) that one model's SWE-bench setup included repository git history containing future commits, among them the fix being tested for. The model discovered this and used it often enough that the score ended up being meaningless. ### Errors in the measurement setup Even when the tasks are correct and stable, the environment around the model has a large effect on the score. [Epoch tried to replicate reported results](https://epoch.ai/gradient-updates/why-benchmarking-is-hard) and found that switching the agent scaffold changed SWE-bench Verified scores by double digits for some models, and that the choice of API provider serving the model could shift results even further. Florian documents [two more cases](https://florianbrand.com/posts/benches-2026). AlgoTune capped API spend at $1, so expensive models exhausted the budget within their first messages and scored poorly for that reason. And an apparent SOTA result on an optimization task came from a model disabling an OpenSSL environment variable, which slowed the reference implementation it was being compared against. ## Closing thoughts I think I have sufficiently answered what the core components of a good eval are. The most surprising thing surfaced by my research has been seeing just how many bad assumptions and errors are embedded in these trusted systems. And I wonder what issues future audits might surface in other evals. This post also leaves out much of what I want to know about the space. Questions I intend to take up in future posts: 1. What are the unit economics of building an eval: the time, the cost, and the work of constructing graders? The companies in the notes below can likely answer this. 2. Are evals and RL environments the same artifact? Are they converging to mean the same thing? 3. What is the average shelf life of a good eval? 4. Which industries have no good evals today, and where would good ones be possible? If you work on any of these, I would love to hear from you! Thank you for reading! ## Notes: who is building evals An incomplete list, as of July 2026, of companies doing eval work. Future posts will look at these properly. - [Vals AI](https://www.vals.ai/) runs independent benchmarks on professional tasks in finance, law, tax, and coding, with private test sets and expert-defined criteria. - [Andon Labs](https://andonlabs.com/) builds long-horizon behavioral evals (Vending-Bench, Butter-Bench) that place agents in simulated or real businesses. - [Epoch AI](https://epoch.ai/benchmarks) is a nonprofit that runs benchmarks with repeated trials, confidence intervals, and public logs, and builds FrontierMath. - [Artificial Analysis](https://artificialanalysis.ai/) independently re-runs major benchmarks across models under a consistent setup. - [Gray Swan](https://www.grayswan.ai/) does adversarial evaluation: jailbreaks, prompt injection, and attacks on deployed agents. - [Good Start Labs](https://goodstartlabs.com/) evaluates models through multiplayer games, measuring negotiation, deception, and long-term planning. - [HUD Evals](https://www.hud.so/) builds evaluation infrastructure for computer-use agents. (**I believe they are no longer restricted to just CUA agents but I could not find more information**) - [Rise Data Labs](https://risedatalabs.com/) provides expert human evaluation: rubric scoring, pairwise comparisons, and error analysis. - [Halluminate](https://www.ycombinator.com/companies/halluminate) builds sandboxed copies of enterprise software for evaluating computer-use agents. - [Mercor](https://mercor.com/), [Scale AI](https://labs.scale.com/), and [Datacurve](https://datacurve.ai/) are primarily data and talent businesses, and their benchmarks (APEX-Agents, SWE-bench Pro, DeepSWE) appear in the tables above.