BoxingGym: Benchmarking Progress in Automated Experimental Design and Model Discovery

arXiv 2025 62.2 method

TLDR

Introduces BoxingGym, a benchmark with 10 environments for evaluating LLM-based agents on experimental design and model discovery using probabilistic models from real-world domains.

Reasoning

Strengths: Clear focus on a novel benchmark for automated scientific discovery, with quantitative metrics (EIG) and diverse domains. Weaknesses: Limited to simulated probabilistic models; no real-world experimental validation or comparison to human scientists.

Read-first score

Read-first score 62.2, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 72.

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=benchmark,evaluation,experiment,metric,result

Recency 8%

86.7

Uses a gentle age decay so recent papers surface without erasing older foundations. 2025

Topical relevance 42%

Uses existing LLM keyword relevance scores normalized to 0-100. AI scientist,automated scientific discovery,autonomous research agent,automated research,literature review agent,survey generation,automated experimentation,experiment design agent,AI for scientific research,paper writing agent,research automation,scientific discovery agent

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=none

Field roles

FrontierMethodology anchor

Rank sensitivity

Stability: volatile; rank range: 37.

Keyword Scores

automated scientific discovery

automated experimentation

experiment design agent

scientific discovery agent

autonomous research agent

AI for scientific research

AI scientist

automated research

research automation

literature review agent

survey generation

paper writing agent

Deep Analysis

Innovations

Introduction of BoxingGym, a benchmark with 10 environments for evaluating automated experimental design and model discovery.
Use of generative probabilistic models from real-world scientific domains as interactive environments.
Quantitative evaluation of experimental design via expected information gain (EIG).
Evaluation of model discovery through explanation-based prediction by another agent and standard prediction error metrics.
Finding that LLMs struggle and augmenting with explicit statistical models does not reliably help.

Methodology

BoxingGym consists of 10 environments implemented as generative probabilistic models from domains like psychology and ecology. Agents interact with these environments to design experiments and propose/revise scientific models. Experimental design is evaluated using expected information gain (EIG), while model discovery is assessed by having the agent explain its model and then measuring how well another agent can predict using that explanation, along with standard prediction error metrics.

Key Results

Current LLMs such as GPT-4o struggle with both experimental design and model discovery; augmenting the LLM-based agent with an explicit statistical model does not reliably improve performance.

Resources

Paper