AutoDataBench: A Data-centric Testbed for Accelerating Auto Research
TLDR
AutoDataBench is a controlled data-centric testbed isolating data intelligence in auto-research, evaluating frontier LLMs' data diagnosis, organization, construction, and effect-understanding via iterative experimentation.
Reasoning
The paper presents a focused benchmark that cleanly isolates data-related factors in auto-research evaluation, with a novel emphasis on whether LLMs understand data interventions. However, the abstract omits detailed results, baselines, and limitations, making it hard to assess robustness and generalizability.
Read-first score
Read-first score 58.7, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 47.
Field roles
Rank sensitivity
Stability: volatile; rank range: 115.
Keyword Scores
Deep Analysis
Innovations
- Introduces AutoDataBench, a data-centric testbed that isolates data intelligence from confounding auto-research factors such as training frameworks, hyperparameters, compute budgets, and data.
- Proposes a conceptual framework of data intelligence spanning data diagnosis, data organization, and data construction.
- Instantiates the framework through three curated optimization tasks across tool use, retrieval, and knowledge injection while holding non-data factors fixed.
- Evaluates data-effect reasoning by comparing LLMs' pre-training predictions of data intervention outcomes with observed results, going beyond trial-and-error.
- Shows that reusing AutoDataBench trajectories for mid-training improves downstream coding performance.
Methodology
AutoDataBench is a controlled testbed that evaluates frontier LLMs on three curated data optimization tasks (tool use, retrieval, and knowledge injection) under task-specific resource budgets with iterative experimentation. It compares predictions made before training with observed outcomes to probe data-effect reasoning, explores whether iterative feedback improves understanding of data interventions, and reuses trajectories for mid-training.
Key Results
The abstract reports that reusing AutoDataBench trajectories for mid-training improves downstream coding performance. It does not state specific quantitative results for the data intelligence evaluations or prediction comparisons.
Limitations
- The testbed holds non-data factors fixed, so it does not evaluate interactions between data interventions and other factors such as hyperparameters or compute budgets.
- Evaluation is instantiated through three curated tasks, limiting the breadth of data intelligence behaviors assessed.
- Data-effect reasoning and iterative feedback are described as exploratory; the abstract does not state whether they produced confirmed improvements.