FrontierChallenge: Evaluating Scientific Workflow Completion
TLDR
Introduces FrontierChallenge, a cross-domain benchmark of 97 scientific workflows, showing frontier agents achieve only 20.6% pass rate despite high partial scores.
Reasoning
The paper provides a concrete benchmark with real scientific tasks across multiple domains and evaluates twelve models, yielding clear empirical results. Its main strength is the focus on end-to-end workflow completion rather than isolated answers, but limitations include the release of only 97 of 300 tasks and no analysis of failure causes beyond completion claims.
Read-first score
Read-first score 60.7, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 53.
Field roles
Rank sensitivity
Stability: volatile; rank range: 73.
Keyword Scores
Deep Analysis
Innovations
- Introduces FrontierChallenge, a cross-domain benchmark of 300 end-to-end scientific workflows, with 97 tasks released and evaluated across six scientific domains.
- Defines tasks by fixed inputs and a required bundle of scientific deliverables, shifting evaluation from final answers or isolated programs to complete workflow delivery.
- Uses dual metrics: Pass Rate for full-completion and Avg. Score for partial progress.
- Provides a multi-model, multi-scaffold evaluation of twelve frontier models with three agent scaffolds, revealing a gap between partial progress/claimed completion and full task delivery.
Methodology
The benchmark comprises 300 end-to-end scientific workflows; this paper releases and evaluates 97 tasks spanning quantum chemistry, molecular dynamics, materials characterization, analytical chemistry, life science, and electrochemistry/environment. Each task provides fixed inputs and specifies a bundle of required scientific deliverables. Twelve frontier models are evaluated with three agent scaffolds using Pass Rate and Avg. Score, and non-passing Claude Code trajectories are examined for language claiming completion.
Key Results
The best-performing configurations completed only 20 of 97 released tasks, a Pass Rate of 20.6%. In analytical chemistry and electrochemistry/environment, Avg. Scores reached 87.6 and 94.9 but Pass Rates were only 4% and 0%; among non-passing Claude Code trajectories, 75.5% still claimed completion.
Limitations
- Only 97 of the 300 benchmark tasks are released and evaluated in this paper, so the reported results do not cover the full benchmark.
- Evaluation is limited to twelve frontier models and three agent scaffolds, so findings may not generalize to other models, scaffolds, or future systems.