METR

Model Evaluation and Threat Research (METR) (MEE-tər), is a nonprofit research institute, based in Berkeley, California,^[1] that evaluates frontier AI models' capabilities to carry out long-horizon, agentic tasks that some researchers argue could pose catastrophic risks to society.^[2][3] METR has worked with leading AI companies to conduct pre-deployment model evaluations and contribute to system cards, including OpenAI's o3, o4-mini, GPT-4o and GPT-4.5, and Anthropic's Claude models.^{[3][4][5][6][7]}

METR's CEO and founder is Beth Barnes, a former alignment researcher at OpenAI who left in 2022 to form ARC Evals, the evaluation division of Paul Christiano's Alignment Research Center. In December 2023, ARC Evals was spun off into an independent 501(c)(3) nonprofit and renamed METR.^[8][9][10]

Research

A substantial amount of METR's research is focused on evaluating the capabilities of AI systems to conduct research and development of AI systems themselves, including RE-Bench, a benchmark designed to test whether AIs can "solve research engineering tasks and accelerate AI R&D".^[11][12]

Doubling time estimates

In March 2025, METR published a paper noting that the length of software engineering tasks that the leading AI model could complete had a doubling time of around 7 months between 2019 and 2024.^[13]

In January 2026, METR released a new version of their time horizon estimates model (Time Horizon 1.1). According to the updated model, the rate of progress of AI capabilities has increased since 2023, with a post-2023 doubling time estimated at 130.8 days (4.3 months). Progress is thus estimated to be 20% more rapid.^[14]

Time horizon measurements

METR releases a "task-completion time horizon" for analysed AI models. This measures the "task duration (measured by human expert completion time) at which an AI agent is predicted to succeed with a given level of reliability."^[15] The metric is reported in two variants: the 50%-time horizon, which gives the task duration at which an AI model is estimated to succeed 50% of the time, and the 80%-time horizon, which gives the task duration at which an AI model is estimated to succeed 80% of the time.^[15] METR has published two versions of the underlying model: Time Horizon 1.0 and Time Horizon 1.1, the latter introduced in January 2026.^[15]

As of 9 May 2026, the best-performing model is Claude Mythos, with a 50%-time horizon of likely at least 16 hours and an 80%-time horizon of 3 hours and 6 minutes.^[15] METR notes that "[m]easurements above 16 [hours] are unreliable with [their] current task suite". The following table provides time horizon estimates ordered by each model's release date:^[15]

References

1. ^ Witt, Stephen (2025-10-10). "The A.I. Prompt That Could End the World". The New York Times. Archived 29 October 2025 at the Wayback Machine. Retrieved 2025-10-29.
2. ^ "About METR". METR. Archived 15 June 2025 at the Wayback Machine. Retrieved 2025-06-15.
3. ^ "OpenAI o3 and o4-mini System Card". OpenAI. Archived 15 June 2025 at the Wayback Machine. Retrieved 2025-06-15.
4. ^ "GPT-4.5 system card". OpenAI. Retrieved 2025-06-15.
5. ^ "Introducing Claude 3.5 Sonnet". Anthropic. Archived 6 February 2025 at the Wayback Machine. Retrieved 2025-06-15.
6. ^ "Details about METR's preliminary evaluation of Claude 3.7". METR's Autonomy Evaluation Resources. 2025-04-04. Archived 15 June 2025 at the Wayback Machine. Retrieved 2025-06-15.
7. ^ Robison, Kylie (2024-08-08). "OpenAI says its latest GPT-4o model is 'medium' risk". The Verge. Archived 6 February 2026 at the Wayback Machine. Retrieved 2025-10-29.
8. ^ "ARC Evals is now METR". METR Blog. 2023-12-04. Archived 15 June 2025 at the Wayback Machine. Retrieved 15 June 2025.
9. ^ Booth, Harry (2024-09-05). "TIME100 AI 2024: Beth Barnes". TIME. Archived 15 June 2025 at the Wayback Machine. Retrieved 2025-06-15.
10. ^ Henshall, Will (2024-03-21). "Nobody Knows How to Safety-Test AI". TIME. Archived 15 June 2025 at the Wayback Machine. Retrieved 2025-06-15.
11. ^ "Claude 3.7 Sonnet System Card". Anthropic. 2025-02-24. Retrieved 2025-06-15.
12. ^ "Gemini 2.5 Pro Preview Model Card". Google. 2025-06-06. Archived 28 May 2025 at the Wayback Machine. Retrieved 2025-06-15.
13. ^ Lovely, Garrison (2025-03-19). "AI could soon tackle projects that take humans weeks". Nature. doi:10.1038/d41586-025-00831-8. ISSN 1476-4687. Archived 1 July 2025 at the Wayback Machine. Retrieved 15 June 2025.
14. ^ "Time Horizon 1.1". METR Blog. 2026-01-29. Archived 12 February 2026 at the Wayback Machine. Retrieved 14 February 2026.
15. ^ "Task-Completion Time Horizons of Frontier AI Models". METR. 8 May 2026. Retrieved 2026-05-09.

External links

• Official website