SciΠvot: Benchmarking Principal-Investigator Judgment at Pivotal Moments of Scientific AutoResearch Agents
TLDR
Introduces SciΠvot, a 248-case benchmark evaluating LLM agents' scientific judgment by comparing original plans to human corrections during autonomous research; frontier models often underperform chance.
Reasoning
The benchmark uses real longitudinal records from 26 projects and multiple scientific fields, providing a realistic evaluation of research judgment. However, the abstract is truncated and gives limited detail on case selection, model robustness, and the practical significance of the accuracy gaps.
Read-first score
Read-first score 57.4, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 62.
Field roles
Rank sensitivity
Stability: volatile; rank range: 33.
Keyword Scores
Deep Analysis
Innovations
- SciΠvot: a 248-case benchmark of scientific judgment built from human corrections to LLM agent research decisions.
- Framing scientific judgment evaluation as a forced choice between the original LLM plan and the human-corrected alternative using only the pre-correction project context.
- Systematic comparison of reasoning effort, access to project record, and reconsideration as factors in LLM scientific judgment.
Methodology
SciΠvot draws on six months of records from senior researchers running LLM agents on 26 projects across six scientific fields, with repeated human corrections. Each of the 248 cases asks an agent to choose between the original LLM plan and the human-corrected alternative using only the project record available before the correction. The study compares LLM and human performance and examines the effects of reasoning effort, project-record access, and reconsideration.
Key Results
At medium reasoning effort with the project record, models systematically prefer the original LLM plan over the human-corrected alternative, and four of five frontier models perform below chance on the selected cases. The highest accuracy in the project condition is 63.5%, compared with 73.4% for fresh researchers; at medium effort, every model is less accurate with the project record than without it, and reconsideration does not consistently improve overall accuracy.
Limitations
- The benchmark is based on selected human-correction cases from a specific six-month period, 26 projects, and six scientific fields.
- Scientific judgment evaluation is inherently difficult because research decisions are open-ended and depend on project context.
- Reported below-chance performance applies to the selected cases and tested frontier models, not necessarily all scientific decisions or models.