⌁ Awesome Auto Research Hub Papers · Datasets · Projects
中文 GitHub
← Back to papers

Automating ML for Science: Can Frontier Agents Climb Scientific Hills in the Wild?

arXiv 2026 65.6 method

TLDR

Introduces Agent4Sci, a 20-benchmark, 11-domain testbed showing frontier agents improve published ML-for-science results mostly via engineering tricks; plug-in skills increase gains.

Reasoning

The paper provides a broad, multi-domain benchmark and careful trajectory analysis, which are strengths. However, the abstract reveals that current agents achieve improvements mainly through ML-engineering tricks rather than genuine scientific insight, limiting the claim of autonomous scientific discovery.

Read-first score

Read-first score 65.6, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 57.

Methodology quality 25%
100

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=analysis,benchmark,dataset,evaluation,experiment

Recency 8%
100

Uses a gentle age decay so recent papers surface without erasing older foundations. 2026

Reproducibility 25%
50

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=code,dataset,github

Topical relevance 42%
47.5

Uses existing LLM keyword relevance scores normalized to 0-100. AI scientist,automated scientific discovery,autonomous research agent,automated research,literature review agent,survey generation,automated experimentation,experiment design agent,AI for scientific research,paper writing agent,research automation,scientific discovery agent

Field roles

FrontierMethodology anchor

Rank sensitivity

Stability: volatile; rank range: 90.

Keyword Scores

autonomous research agent
8
AI for scientific research
8
automated scientific discovery
7
automated research
7
research automation
7
AI scientist
6
scientific discovery agent
6
automated experimentation
4
experiment design agent
4
literature review agent
0
survey generation
0
paper writing agent
0

Deep Analysis

Innovations

  • Introduces Agent4Sci, a long-horizon testbed spanning 20 scientific ML benchmarks across 11 domains for evaluating automated improvement of published best models.
  • Shifts evaluation from leaderboard ranking to analyzing what kinds of progress agents make under in-the-wild conditions and where they fall short.
  • Uses trajectory analysis to identify recurring agent limitations: novelty miscalibration, resource under-utilization, and rapid collapse into local engineering sweeps.
  • Proposes lightweight plug-in skills for research ideation, experiment management, and scheduled checklists that improve frontier agent performance and intervention diversity.

Methodology

Agent4Sci evaluates frontier coding agents on improving published best models across 20 scientific ML benchmarks in 11 domains under in-the-wild conditions. The study measures mean relative improvement and academic misconduct flags, and analyzes agent trajectories to characterize progress and failures. It also tests the effect of adding plug-in skills for research ideation, experiment management, and scheduled checklists, comparing Claude Opus and Codex.

Key Results

Frontier agents improved the published best on every benchmark, with Claude Opus reaching 10.4% mean relative improvement and no run flagged for academic misconduct, though gains came almost entirely from ML-engineering tricks. Plug-in skills raised Claude Opus from 10.4% to 20.1% and Codex from 4.4% to 8.1%, with more diverse, domain-relevant interventions.

Limitations

  • Gains come almost entirely from ML-engineering tricks rather than research-level insight.
  • Trajectory analysis shows novelty miscalibration.
  • Trajectory analysis shows resource under-utilization.
  • Trajectory analysis shows rapid collapse into local engineering sweeps.

Resources

Paper

Tags