Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents
TLDR
AutoSciRub is an evaluation-first framework that induces task-specific executable rubrics before research execution to guide autonomous agents, improving performance by 2-3 points on ResearchClawBench.
Reasoning
The paper addresses the important problem of underspecified research tasks and proposes a novel rubric-induction method to make implicit requirements explicit. Its strengths include a clear framework and consistent empirical gains across multiple LLMs and harnesses. Limitations include reliance on a single benchmark and partial results on a 20-task subset.
Read-first score
Read-first score 47.7, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 66.
Field roles
Rank sensitivity
Stability: volatile; rank range: 45.
Keyword Scores
Deep Analysis
Innovations
- Proposes AutoSciRub, an evaluation-first framework that induces a task-specific executable rubric before research execution.
- Decomposes underspecified research instructions into atomic scientific goals grounded in relevant literature and task-visible data.
- Synthesizes specific, actionable, and verifiable criteria to make implicit experimental and evidential requirements explicit.
- Uses rubric-guided criterion-level verification and iterative revision to identify unmet criteria and refine the research report and supporting artifacts.
Methodology
AutoSciRub first decomposes an underspecified instruction into atomic scientific goals, grounds them in literature and task-visible data, and synthesizes verifiable criteria into an executable rubric. The rubric then guides research execution, criterion-level verification, and iterative revision of the report and artifacts. The approach is evaluated on ResearchClawBench and a randomly sampled 20-task subset of AstaBench E2E Discovery across multiple backbone LLMs and agent harnesses, using average score gains and task completion as metrics.
Key Results
On ResearchClawBench, AutoSciRub improved all tested configurations, with average gains of 2.08 points across three backbone LLMs under a fixed Codex harness and 2.95 points across three agent harnesses using a fixed DeepSeek-V4-Flash backbone. On a 20-task AstaBench E2E Discovery subset, it achieved an average improvement of 16.8 points across three agent harnesses while maintaining or increasing the number of successfully completed tasks.
Limitations
- The AstaBench E2E Discovery evaluation is based on a randomly sampled 20-task subset rather than the full benchmark.