⌁ Awesome Auto Research Hub Papers · Datasets · Projects
中文 GitHub
← Back to papers

ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery

arXiv 2024 56.4 method

TLDR

A benchmark for rigorously assessing language agents on data-driven scientific discovery tasks from real publications.

Reasoning

The paper's strength lies in its rigorous methodology: extracting tasks from peer-reviewed papers, expert validation, and contamination mitigation. Weaknesses include low agent success rates (32.4%) and focus only on data-driven tasks, limiting generalizability.

Read-first score

Read-first score 56.4, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 45.

Methodology quality 25%
100

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=benchmark,evaluation,metric,result,validation

Recency 8%
75.1

Uses a gentle age decay so recent papers surface without erasing older foundations. 2024

Reproducibility 25%
38

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=code

Topical relevance 42%
37.5

Uses existing LLM keyword relevance scores normalized to 0-100. AI scientist,automated scientific discovery,autonomous research agent,automated research,literature review agent,survey generation,automated experimentation,experiment design agent,AI for scientific research,paper writing agent,research automation,scientific discovery agent

Field roles

BridgeMethodology anchor

Rank sensitivity

Stability: volatile; rank range: 80.

Keyword Scores

AI for scientific research
7
automated scientific discovery
6
research automation
6
scientific discovery agent
6
autonomous research agent
5
automated research
5
AI scientist
3
automated experimentation
2
experiment design agent
2
literature review agent
1
survey generation
1
paper writing agent
1

Deep Analysis

Innovations

  • Creation of ScienceAgentBench, a benchmark with 102 tasks extracted from 44 peer-reviewed publications across four disciplines and validated by nine subject matter experts.
  • Unified task output as self-contained Python programs, enabling evaluation of generated code, execution results, and computational cost.
  • Multi-round manual validation by annotators and experts to ensure annotation quality and scientific plausibility.
  • Two strategies to mitigate data contamination concerns in benchmark design.

Methodology

The benchmark comprises 102 data-driven scientific discovery tasks sourced from 44 peer-reviewed publications and validated by subject matter experts. Each task requires generating a self-contained Python program, evaluated on code correctness, execution results, and cost. Five open-weight and proprietary LLMs were tested with three frameworks (direct prompting, OpenHands CodeAct, self-debug) over three attempts per task, along with OpenAI o1-preview using direct prompting and self-debug.

Key Results

The best-performing agent solved only 32.4% of tasks independently and 34.3% with expert-provided knowledge; OpenAI o1-preview reached 42.2% but incurred over 10 times the cost of other LLMs, highlighting significant performance gaps.

Limitations

  • Current language agents still fail to reliably generate correct code for data-driven scientific tasks, with top performance at 42.2%.
  • The benchmark covers only individual tasks, not end-to-end scientific workflows, limiting claims about full automation.
  • High inference cost of advanced models like o1-preview (over 10x) may hinder practical deployment.
  • Task scope is limited to 102 instances from four disciplines, which may not represent all scientific domains.

Resources

Paper

Tags

CLAILG