EvalResearchBench: Can AI Agents Build Their Own Feedback for Recursive Self-Improvement?
TLDR
Introduces EvalResearchBench, a benchmark for autonomous evaluation research, testing whether AI agents can design executable evaluators for coding, co-work, and reasoning tasks under fixed budgets and pilot feedback.
Reasoning
The paper addresses a well-motivated problem with a broad empirical study across multiple agents, models, and target benchmarks, and it honestly reports limitations such as human baseline competitiveness and evaluation failures. However, the abstract provides limited detail on task construction and metric definitions, making the exact novelty and reliability of the benchmark harder to fully assess.
Read-first score
Read-first score 59.6, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 54.
Field roles
Rank sensitivity
Stability: volatile; rank range: 78.
Keyword Scores
Deep Analysis
Innovations
- Introduces EvalResearchBench (ERB), a benchmark for autonomous evaluation research in recursive self-improvement settings.
- Frames evaluation design as an agentic pipeline where an agent selects or synthesizes tasks, implements graders, and revises them through pilot tests under fixed time and API budgets.
- Evaluates AI-generated evaluators against 14 target benchmarks using score concordance and pairwise agreement metrics.
- Benchmarks 9 researcher configurations and 13 candidate models, including comparison to a human-designed public-task baseline and sealed target analysis.
Methodology
Given target materials, development references, candidate APIs, and fixed time and API budgets, an agent researcher constructs an executable evaluator for coding, co-work, and reasoning by selecting or synthesizing tasks and implementing graders, with pilot-test revisions before freezing. The study examines 9 researchers and 13 candidate models, comparing each frozen evaluator with 14 target benchmarks on score concordance and pairwise agreement. It also compares against a human-designed sample of public tasks and uses sealed targets hidden from the researcher to test generalization.
Key Results
The best generated evaluators order about 75% of candidate pairs as the targets do, below the 91% ceiling set by disagreements among the targets. No researcher leads on every metric, the best evaluator on development targets is not best on sealed targets, a human-designed sample remains a strong baseline, and the evaluator with the lowest recorded execution cost achieves the highest pairwise agreement.
Limitations
- Best evaluators order only about 75% of candidate pairs as targets, below the 91% ceiling set by target disagreements.
- No researcher leads on every metric.
- The best evaluator on development targets is not the best on sealed targets hidden from the researcher.
- A human-designed sample of public tasks remains a strong baseline.
- Generated evaluators can still truncate answers, exhaust the evaluation budget, or let a few questions dominate a domain score.