AblationBench: Evaluating Automated Planning of Ablations in Empirical AI Research

arXiv 2025 73.9 method

TLDR

Introduces AblationBench, a benchmark for evaluating LM agents on planning ablation experiments in AI research, finding current LMs underperform humans.

Reasoning

Strengths: Provides a concrete benchmark with two tasks (AuthorAblation, ReviewerAblation) and automatic evaluation, revealing clear limitations of current LMs. Weaknesses: Focuses narrowly on ablation planning, not broader scientific discovery; the benchmark's generalizability to other research domains is unclear.

Read-first score

Read-first score 73.9, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 69.

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=ablation,benchmark,evaluation,experiment

Recency 8%

86.7

Uses a gentle age decay so recent papers surface without erasing older foundations. 2025

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=True; dataset=False; markers=code,github

Topical relevance 42%

57.5

Uses existing LLM keyword relevance scores normalized to 0-100. AI scientist,automated scientific discovery,autonomous research agent,automated research,literature review agent,survey generation,automated experimentation,experiment design agent,AI for scientific research,paper writing agent,research automation,scientific discovery agent

Field roles

FrontierMethodology anchorReproducibility anchor

Rank sensitivity

Stability: volatile; rank range: 37.

Keyword Scores

experiment design agent

automated experimentation

autonomous research agent

AI for scientific research

research automation

AI scientist

automated research

scientific discovery agent

automated scientific discovery

literature review agent

paper writing agent

survey generation

Deep Analysis

Innovations

Introduction of AblationBench, a benchmark suite for evaluating automated planning of ablation experiments in empirical AI research.
Two complementary tasks: AuthorAblation (proposing ablations from a method section) and ReviewerAblation (finding missing ablations in a full paper).
LM-based judges for automatic evaluation of ablation planning.
Finding that chain-of-thought prompting outperforms an agent-based approach on these tasks.
Observation of an inverse performance trend between author and reviewer tasks, attributed to differences in model grounding.

Methodology

AblationBench consists of 83 AuthorAblation instances and 350 ReviewerAblation instances, evaluated with LM-based judges. Frontier language models are tested on both tasks, comparing chain-of-thought prompting against an agent-based setup.

Key Results

The best-performing LM system identifies only 45% of the original ablations on average, below human-level performance. Chain-of-thought prompting outperforms the agent-based approach, and an inverse performance trend between author and reviewer tasks is observed.

Limitations

Current LMs achieve only 45% recall on ablation identification, remaining below human-level performance.
Inverse performance between author and reviewer tasks indicates grounding differences that limit consistent model behavior.

Resources

Paper Code