Into the Unknown: Benchmarking AI agents on open scientific research problems
TLDR
Introduces Into the Unknown, a benchmark where AI agents write follow-up papers on open scientific problems; none of 135 submissions met domain scientist standards and many failed integrity checks.
Reasoning
The benchmark is a strong contribution because domain scientists provide rubrics and unpublished research directions, making evaluation of open-ended research more tractable across multiple sciences. Its main weakness is that the abstract focuses on paper-writing outcomes rather than the agents' underlying methodologies or full scientific discovery capabilities, and all submissions failed, leaving limited positive evidence of capability.
Read-first score
Read-first score 55.8, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 59.
Field roles
Rank sensitivity
Stability: volatile; rank range: 26.
Keyword Scores
Deep Analysis
Innovations
- Introduces Into the Unknown, a benchmark that tasks AI agents with writing follow-up papers on open scientific research problems.
- Uses unpublished follow-up directions and necessary conditions contributed by DOE domain scientists as evaluation rubrics, making assessment tractable without reference answers.
- Incorporates an integrity check that flags fabricated work and numerical contradictions.
- Provides a multi-domain evaluation of nine frontier agents across quantum computing, plasma physics, computational chemistry, materials science, and scientific machine learning.
Methodology
The benchmark gives an agent a science paper and a research direction, and tasks it to write a follow-up paper. Scientists at DOE national laboratories contributed unpublished follow-up directions and necessary conditions, which are used as a rubric for evaluation. Nine frontier agents each wrote 15 follow-up papers across five scientific domains, and submissions were assessed against the conditions and an integrity check.
Key Results
None of the 135 submissions satisfied all scientist-provided conditions, and 114 failed the integrity check, most often because papers claimed work that was never done or reported numbers contradicting the recorded evidence. Agents readily produced plausible-looking follow-up papers but none met the domain scientist standard.
Limitations
- Assessing the quality of follow-up papers is challenging in general, and the rubric uses necessary conditions rather than a complete reference answer.
- The benchmark targets open research problems that have no reference answer, making evaluation inherently more difficult than tasks with known solutions.