What Can a Leaderboard Certify? Compositional Controllability for Fair Evaluation and Training of Biomedical Literature-Review Agents

arXiv 2026 40.9 method

TLDR

Introduces compositional controllability for fair evaluation of biomedical literature-review agents, with a new benchmark and tests showing many leaderboard comparisons are inadmissible.

Reasoning

The paper's strength is a rigorous framework for controlled comparisons and a new benchmark. Its weakness is the narrow focus on literature-review agents, limiting generalizability to broader scientific discovery.

Read-first score

Read-first score 40.9, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 29.

Recency 8%

100

Uses a gentle age decay so recent papers surface without erasing older foundations. 2026

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=analysis,benchmark,evaluation,result

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=none

Topical relevance 42%

24.2

Uses existing LLM keyword relevance scores normalized to 0-100. AI scientist,automated scientific discovery,autonomous research agent,automated research,literature review agent,survey generation,automated experimentation,experiment design agent,AI for scientific research,paper writing agent,research automation,scientific discovery agent

Field roles

Frontier

Rank sensitivity

Stability: volatile; rank range: 39.

Keyword Scores

literature review agent

AI for scientific research

research automation

automated research

autonomous research agent

survey generation

paper writing agent

AI scientist

automated scientific discovery

automated experimentation

experiment design agent

scientific discovery agent

Resources

Paper