Scaling Automatic Research Agents via World Models
TLDR
Proposes World Model RL to overcome environment execution bottlenecks in automatic research agents, using debiasing and denoising to accelerate training by 3-4x while improving performance.
Reasoning
The paper identifies a real scalability bottleneck in RL for AutoResearch agents and proposes a principled world-model approach with theoretical convergence guarantees and promising empirical speedups. Weaknesses include an incomplete abstract and limited visibility into benchmark details, task diversity, and comparison baselines.
Read-first score
Read-first score 57.7, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 63.
Field roles
Rank sensitivity
Stability: volatile; rank range: 33.
Keyword Scores
Deep Analysis
Innovations
- Identifies a fundamental scaling tension in RL for AutoResearch agents: agent generation shares compute through batching while environment execution requires exclusive sandbox and real machine time, making execution the dominant training bottleneck.
- Proposes World Model RL (WMRL), which replaces environment execution with a world model to remove the execution bottleneck.
- Introduces Online Debiasing to offset bias in world-model rewards.
- Introduces Inverse-Variance Denoising to suppress noise in world-model rewards.
- Provides theoretical proof that Online Debiasing and Inverse-Variance Denoising strictly improve the convergence guarantee.
Methodology
WMRL post-trains AutoResearch agents with RL by replacing real environment execution with a world model, while Online Debiasing and Inverse-Variance Denoising correct the imperfect world-model rewards. The method is evaluated on various tasks at different agent scales against standard RL baselines, with post-trained 4B and 9B agents tested on held-out benchmarks. WMRL is also transferred to post-training embodied VLA policies to test generality.
Key Results
WMRL accelerates training by 3-4x on various tasks at different agent scales while exceeding standard RL baselines. Post-trained 4B and 9B agents outperform much larger open-weight 48B and 120B agents on held-out benchmarks, and the method transfers to embodied VLA post-training.
Limitations
- The world model can be imperfect, with rewards corrupted by bias and noise, requiring additional mitigation via Online Debiasing and Inverse-Variance Denoising.