Language Models Learn to Improve Language Models
TLDR
Proposes LM2LM, training LLMs with RL using a frozen research world model as reward to generate LM improvement proposals, validated by real-execution evaluations.
Reasoning
The paper addresses cost and reliability challenges in using LLMs for automated LM research, combining surrogate rewards with periodic real executions. Strengths include a practical RL framework and multi-environment evaluation; weaknesses include reliance on a frozen reward model and focus on LM improvement rather than broader scientific discovery.
Read-first score
Read-first score 52.4, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 55.
Field roles
Rank sensitivity
Stability: volatile; rank range: 42.
Keyword Scores
Deep Analysis
Innovations
- LM2LM framework that trains LLMs with reinforcement learning to generate effective language-model improvement proposals.
- Uses a research world model (RWM), a frozen LLM that predicts proposal gains from real experimental records in context, as a scalable surrogate reward model.
- Periodically executes high-ranked current-policy proposals in real environments and adds their measured outcomes to the RWM context to maintain feedback reliability under distribution shift.
Methodology
LM2LM trains an LLM policy with RL to propose LM improvements, using a frozen RWM to predict proposal gains from real experimental records as reward instead of running full model training for every proposal. To counter reward-model unreliability, high-ranked current-policy proposals are periodically executed and their measured outcomes are added to the RWM context. The method is evaluated across eight autoregressive and diffusion LM development environments with real-execution evaluations on held-out tasks.
Key Results
Across eight autoregressive and diffusion LM development environments, LM2LM enables Qwen3.8-27B to outperform GPT-6-sol even when GPT-6-sol is given 300 real experimental records in context, while requiring 33% lower data-generation cost. LM2LM also generalizes to a model family unseen during training.
Limitations
- Training with real feedback is expensive: RL requires evaluating tens of thousands of proposals, potentially consuming hundreds of thousands of GPU-hours if each requires a full model-training run.
- Surrogate feedback can become unreliable as policy optimization favors overestimated proposals when the proposal distribution shifts.