Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents
TLDR
AutoSciRub is an evaluation-first framework that induces task-specific executable rubrics before research execution to guide autonomous agents, improving performance by 2-3 points on ResearchClawBench.
评分理由
The paper addresses the important problem of underspecified research tasks and proposes a novel rubric-induction method to make implicit requirements explicit. Its strengths include a clear framework and consistent empirical gains across multiple LLMs and harnesses. Limitations include reliance on a single benchmark and partial results on a 20-task subset.
Read-first 评分解释
综合优先阅读分 47.7，由主题、引用、图谱、方法、可复现性和近期性等信号加权得到。 原始总分保留为 66。
研究版图角色
排序敏感性
稳定性：volatile；排名波动范围：45。
关键词评分
深度分析
创新点
- Proposes AutoSciRub, an evaluation-first framework that induces a task-specific executable rubric before research execution.
- Decomposes underspecified research instructions into atomic scientific goals grounded in relevant literature and task-visible data.
- Synthesizes specific, actionable, and verifiable criteria to make implicit experimental and evidential requirements explicit.
- Uses rubric-guided criterion-level verification and iterative revision to identify unmet criteria and refine the research report and supporting artifacts.
方法
AutoSciRub first decomposes an underspecified instruction into atomic scientific goals, grounds them in literature and task-visible data, and synthesizes verifiable criteria into an executable rubric. The rubric then guides research execution, criterion-level verification, and iterative revision of the report and artifacts. The approach is evaluated on ResearchClawBench and a randomly sampled 20-task subset of AstaBench E2E Discovery across multiple backbone LLMs and agent harnesses, using average score gains and task completion as metrics.
关键结果
On ResearchClawBench, AutoSciRub improved all tested configurations, with average gains of 2.08 points across three backbone LLMs under a fixed Codex harness and 2.95 points across three agent harnesses using a fixed DeepSeek-V4-Flash backbone. On a 20-task AstaBench E2E Discovery subset, it achieved an average improvement of 16.8 points across three agent harnesses while maintaining or increasing the number of successfully completed tasks.
局限性
- The AstaBench E2E Discovery evaluation is based on a randomly sampled 20-task subset rather than the full benchmark.