Scores Alone Do Not Prove Discovery: The Discovery Certification Protocol for Auditing AI Research Agents

arXiv 2026 49.7 method

TLDR

Proposes the Discovery Certification Protocol to audit AI research agents using recovery tests and feedback gates; audits in SQLite and catalyst control yielded zero recoveries in 96 episodes.

Reasoning

The paper presents a rigorous, evidence-based certification protocol with clear gates, finite-sample bounds, and reproducible audits, which is a strong contribution. However, the abstract offers limited comparison to existing audit frameworks and the evaluations are narrow, with only two controlled settings.

Read-first score

Read-first score 49.7, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 50.

Recency 8%

100

Uses a gentle age decay so recent papers surface without erasing older foundations. 2026

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=evaluation,experiment,result

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=checkpoint,reproduce

Topical relevance 42%

41.7

Uses existing LLM keyword relevance scores normalized to 0-100. AI scientist,automated scientific discovery,autonomous research agent,automated research,literature review agent,survey generation,automated experimentation,experiment design agent,AI for scientific research,paper writing agent,research automation,scientific discovery agent

Field roles

Frontier

Rank sensitivity

Stability: volatile; rank range: 39.

Keyword Scores

autonomous research agent

automated scientific discovery

AI for scientific research

automated research

scientific discovery agent

AI scientist

research automation

automated experimentation

experiment design agent

literature review agent

survey generation

paper writing agent

Resources

Paper