AI Scientist via Synthetic Task Scaling
TLDR
A pipeline to generate synthetic ML research tasks for training agents, improving performance on MLGym benchmark via teacher-student training.
Reasoning
The paper introduces a novel synthetic task generation pipeline for training AI agents in ML research, with grounding in real datasets and self-debugging. Strengths include a principled training approach and empirical gains on MLGym; weaknesses include limited scope to ML tasks and lack of evaluation on broader scientific discovery.
Read-first score
Read-first score 63.5, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 68.
Field roles
Rank sensitivity
Stability: volatile; rank range: 23.
Keyword Scores
Deep Analysis
Innovations
- Synthetic environment generation pipeline for machine learning agents that automatically synthesizes ML challenges (topic sampling, dataset proposal, code generation) compatible with SWE-agent.
- Grounding synthetic tasks in real ML datasets via verification against the Huggingface API.
- Self-debugging loop to ensure higher quality of generated tasks.
- Teacher-student training paradigm using trajectories from GPT-5 on synthetic tasks to improve smaller models (Qwen3-4B, Qwen3-8B) on MLGym benchmark.
Methodology
The paper proposes a pipeline that automatically synthesizes machine learning challenges compatible with the SWE-agent framework, covering topic sampling, dataset proposal, and code generation. The tasks are grounded in real datasets by verifying proposals against the Huggingface API and refined via a self-debugging loop. Trajectories from a teacher model (GPT-5) on these synthetic tasks are used to train student models (Qwen3-4B and Qwen3-8B), which are then evaluated on the MLGym benchmark using the AUP metric.
Key Results
Training with synthetic tasks improved the AUP metric on MLGym by 9% for Qwen3-4B and 12% for Qwen3-8B.