Entity Profile: METR (Model Evaluation and Threat Research)
METR (pronounced "meter"), formerly known as ARC Evals, is a high-impact 501(c)(3) nonprofit research institute based in Berkeley, California. The organization serves as a critical third-party evaluator for frontier artificial intelligence systems. METR focuses on developing the scientific rigorousness required to assess whether advanced AI models?such as those developed by OpenAI, Anthropic, and Google DeepMind?pose "catastrophic risks" to society, particularly through autonomous capabilities or specialized technical knowledge.
Core Mission and Research Philosophy
The primary objective of METR is to provide empirical, technical evidence regarding AI capabilities before and after model deployment. The organization operates under the belief that as AI systems transition from text-prediction tools to autonomous agents, current safety benchmarks are insufficient.
The "Time Horizon" Metric
A hallmark of METR?s research is the shift from measuring accuracy on static exams to measuring the Time Horizon of AI ability.
* Long-Horizon Tasks: METR evaluates AI performance based on the length of time an AI agent can work toward a goal without human intervention (measured in the number of hours or days it would take a human expert).
* Forecasting Doubling Times: Their research indicates that the "time horizon" of frontier models has historically doubled approximately every 7 months, suggesting a rapid trajectory toward autonomous week-long task completion.
Elicitation Research
METR specializes in "elicitation," which involves attempting to draw out the maximum possible capability of a model through advanced prompting, fine-tuning, and scaffolding. This ensures that safety evaluations reflect the model's true potential rather than just its default "out-of-the-box" behavior.
Products and Services
METR provides a suite of open-source tools and high-level advisory services designed to standardize how the industry measures AI risk.
1. Evaluation Infrastructure
METR develops the technical frameworks used by leading labs to run safety tests.
* Inspect AI Infrastructure: A software framework for running large-scale evaluations on frontier models.
* Vivaria: An open-source platform designed for hosting and managing complex, multi-step evaluation tasks in secure, sandboxed environments.
2. The METR Task Suites
The organization maintains several "Task Suites" that are used to benchmark the autonomous capabilities of models.
* Public Task Suite: A collection of real-world tasks (e.g., software engineering, data analysis) that are manually vetted to ensure they test core reasoning and error-recovery rather than rote memorization.
* RE-Bench (Research Engineering Benchmark): Specifically designed to test whether an AI can "accelerate AI R&D" by automating the tasks of machine learning engineers, such as reproducing research papers or optimizing neural architectures.
3. Pre-Deployment Evaluations
METR acts as an independent "red team" for frontier AI developers. They conduct extensive private evaluations of unreleased models (such as GPT-4, GPT-4.5, and various Claude iterations).
* Dangerous Capabilities Testing: Specialized probing for capabilities in cyber-offense (e.g., identifying zero-day vulnerabilities) and assistance in the development of chemical, biological, radiological, and nuclear (CBRN) weapons.
* Sabotage and Subversion Tests: Evaluating if a model can detect when it is being monitored and whether it attempts to subvert or bypass its own safety protocols.
4. Policy and Standards Advisory
METR serves as a technical bridge between AI labs and government regulators.
* Safety Policy Review: They analyze and provide feedback on the Responsible Scaling Policies (RSPs) of major AI companies.
* Government Consultation: METR provides technical input to domestic and international government agencies (such as the U.S. AI Safety Institute) to help shape national AI action plans and safety standards.
Organizational Structure and Leadership
METR was originally established in 2022 as ARC Evals, a division of Paul Christiano?s Alignment Research Center. In December 2023, it spun off as an independent nonprofit to ensure its evaluations remain unbiased and objective.
Beth Barnes (CEO & Founder): Formerly an alignment researcher at OpenAI and DeepMind, Barnes founded the organization to address the lack of standardized safety testing in the industry.
Funding: As a 501(c)(3), METR is funded primarily through philanthropic grants. Crucially, the organization does not accept monetary compensation from the AI companies it evaluates to maintain its status as an impartial "third-party" auditor.
Please note: This information is generated by AI. For legal use, verify with company sources.
