The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvements
TLDR
Introduces a benchmark to evaluate AI agents' ability to reproduce NanoGPT speedrun improvements, finding current LLMs struggle even with detailed hints.
Reasoning
Strengths: The benchmark is well-motivated, accessible, and realistic, targeting a necessary skill for autonomous research. Weaknesses: The abstract does not discuss limitations or generalizability beyond reproduction tasks, and the benchmark may not capture full scientific discovery.
Read-first score
Read-first score 51.1, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 43.
Field roles
Rank sensitivity
Stability: volatile; rank range: 45.
Keyword Scores
Deep Analysis
Innovations
- Introduction of the Automated LLM Speedrunning Benchmark, a new benchmark for evaluating AI agents' ability to reproduce research results using the NanoGPT speedrun competition.
- Design of 19 tasks that provide agents with a previous record's training script and one of three hint formats (pseudocode, description, paper-like) to reimplement known improvements.
- Demonstration that the benchmark is non-saturated, as current reasoning LLMs with state-of-the-art scaffolds fail to reproduce the improvements even with detailed hints.
Methodology
The benchmark consists of 19 tasks derived from the NanoGPT speedrun competition, where each task gives an AI agent the previous record's training script and optionally a hint (pseudocode, description, or paper-like) about the improvement. The agent must modify the script to reproduce the speedup, with tasks covering diverse code-level changes from algorithmic to hardware-aware optimizations. Recent reasoning LLMs with state-of-the-art scaffolds are evaluated on their ability to reimplement these known innovations.
Key Results
Recent reasoning LLMs combined with state-of-the-art scaffolds struggle to reimplement already-known innovations in the benchmark, even when provided with detailed hints, indicating the benchmark is a non-saturated measure of scientific reproduction ability.
Limitations
- The benchmark only measures reproduction of existing results, which is a necessary but not sufficient skill for autonomous research.
- Tasks are limited to NanoGPT training optimizations, potentially not capturing the full diversity of scientific reproduction challenges.
- The hint formats may not fully represent the information available in real-world scientific reproduction scenarios.