⌁ Awesome Auto Research Hub Papers · Datasets · Projects
中文 GitHub
← Back to papers

How Is Automated Research Evaluated? A Survey of Benchmarks and Evaluation Practices

arXiv 2026 67.5 method

TLDR

A survey of benchmarks and evaluation practices for automated research systems across six targets, comparing task construction, evidence, evaluators, and scoring to inform evaluation design.

Reasoning

The paper provides a structured synthesis of evaluation designs for automated research systems, with clear categorization across targets and practical recommendations. Its main limitation is that, as a survey, it contributes no new empirical experiments or datasets of its own.

Read-first score

Read-first score 67.5, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 71.

Methodology quality 25%
100

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=benchmark,dataset,evaluation,experiment,result

Recency 8%
100

Uses a gentle age decay so recent papers surface without erasing older foundations. 2026

Topical relevance 42%
59.2

Uses existing LLM keyword relevance scores normalized to 0-100. AI scientist,automated scientific discovery,autonomous research agent,automated research,literature review agent,survey generation,automated experimentation,experiment design agent,AI for scientific research,paper writing agent,research automation,scientific discovery agent

Reproducibility 25%
38

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=dataset

Field roles

FrontierMethodology anchor

Rank sensitivity

Stability: volatile; rank range: 28.

Keyword Scores

automated research
10
research automation
9
AI for scientific research
7
automated scientific discovery
6
literature review agent
6
automated experimentation
6
paper writing agent
6
AI scientist
5
autonomous research agent
5
experiment design agent
5
scientific discovery agent
5
survey generation
1

Deep Analysis

Innovations

  • Surveys automated research evaluation across six targets: literature synthesis, research ideation, executable workflows, scholarly writing and communication, automatic peer review, and end-to-end research.
  • Compares evaluation designs along task construction, evidence sources, evaluators, and scoring procedures.
  • Synthesizes three recurring evaluation lessons: output checks, process checks, and human studies are complementary; evaluator calibration is property-specific; resource budgets and attempt selection affect performance interpretation.
  • Identifies diagnostic evaluation designs and translates findings into reporting and audit recommendations for specific evaluation settings.

Methodology

This is a survey paper that reviews and organizes existing benchmarks and studies on evaluating automated research systems. It examines six research task areas and compares their evaluation designs in terms of task construction, evidence sources, evaluators, and scoring procedures. The synthesis focuses on evaluation design and evidence rather than introducing new experimental results or datasets.

Key Results

The survey highlights three recurring lessons: output checks, process checks, and human studies provide complementary information; evaluator calibration is specific to the property being assessed; and resource budgets and attempt selection are integral to interpreting performance comparisons. It also identifies diagnostic evaluation designs and documented gaps in supporting evidence.

Limitations

  • The abstract mentions documented gaps in supporting evidence but does not enumerate them.
  • No explicit limitations of the survey methodology or the reviewed evaluations are stated in the abstract.

Resources

Paper

Tags