WhatWorkedBench: Benchmarking Experimental Understanding in AI Agents
TLDR
Introduces WhatWorkedBench, a benchmark measuring AI agents' experimental understanding by predicting effects of component changes from code and measurements, across multiple tasks and workflows.
Reasoning
The paper presents a novel benchmark with exhaustive CPU-executed reference effects and evaluates agent measurement strategies, showing improvements via Gaussian process fitting and configuration equivalence encoding. The abstract is strong on empirical evaluation but does not discuss limitations or compare with broader autonomous discovery benchmarks.
Read-first score
Read-first score 60.4, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 65.
Field roles
Rank sensitivity
Stability: volatile; rank range: 20.
Keyword Scores
Deep Analysis
Innovations
- Introduces WhatWorkedBench, a benchmark for measuring experimental understanding in AI agents as the accuracy of predictions about component changes after budgeted experimentation.
- Defines a response-surface evaluation protocol where agents inspect code, select measurements, and predict scores for every configuration of component settings.
- Provides exhaustive CPU-derived reference effects for component changes across 36 tasks, 30 data sources, 8 workflow types, and 1,248 configuration records.
- Evaluates inference methods including pair-effect ridge, Gaussian process fitting on agent observations, and code-equivalence encoding.
Methodology
Agents inspect task code, choose measurements under a budget, and submit a response surface predicting scores for all component settings. Exhaustive CPU execution supplies reference effects for changing each component while holding others fixed, across 36 tasks from 30 data sources and 8 workflow types. Evaluation uses 4,206 numerical-control records across eight families and 108 agent episodes on the original six, with metrics such as effect error, accuracy relative to true effect magnitude, family-macro recovery, and GP recovery.
Key Results
Pair-effect ridge selects an optimum on 15 of 22 sources and limits every effect error to 10% of score range on three. Fitting a Gaussian process to agent observations raises effect recovery from 0.632 to 0.698 in the original Flash cohort and from 0.621 to 0.720 in an additional cohort, while same-observation GP raises family-macro recovery from 0.303 to 0.455 on six completed submissions and code-equivalence encoding raises GP recovery from 0.248 to 0.462 on six workflows.