OpenBioRQ: Unsolved Biomedical Research Questions for Agents
TLDR
Introduces OpenBioRQ, a benchmark of 12,553 unsolved biomedical questions to test agentic citation faithfulness and abstention, revealing high citation resolution but 15.9% wrong links.
Reasoning
Strengths: Novel benchmark addressing a critical failure mode (citation misattribution) with real-world verification and empirical difficulty anchoring. Weaknesses: Limited to biomedical domain; no direct evaluation of full scientific discovery pipeline (e.g., experimentation or paper writing).
Read-first score
Read-first score 51, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 38.
Field roles
Rank sensitivity
Stability: volatile; rank range: 72.
Keyword Scores
Deep Analysis
Innovations
- Introduces OpenBioRQ, a retrieval-grounded agentic benchmark of 12,553 unsolved biomedical research questions across 12 domains, designed as a faithfulness-and-abstention probe with no answer key.
- First biomedical benchmark to combine an agentic setting (multiple tool calls) with unsolved questions, where openness is verified against real follow-up evidence rather than parametric knowledge.
- Empirical difficulty anchoring: questions are selected based on failure of three open-weight reference models, avoiding subjective hardness labels.
- Discovery of agentic collapse on the hardest questions, where agents stop using tools, and demonstration that blocking tools barely changes scores for the most collapse-prone model.
- Use of a frozen per-question checklist to raise inter-judge agreement from Spearman 0.35 to 0.82.
Methodology
The benchmark comprises 12,553 unsolved biomedical questions requiring agents to perform multiple tool calls in a retrieval-grounded setting. Difficulty is determined empirically by selecting questions that three open-weight reference models fail to answer. Evaluation uses a frozen per-question checklist to judge answer correctness, with openness verified against real follow-up evidence.
Key Results
On the hardest subset, held-out models from the same lineage solve ~17%, while frontier agents (Gemini-3-Pro, Opus-4.7, GPT-5.5) achieve 29-60%, leaving 33-40% unsolved for the best agent. Agentic collapse occurs on the hardest questions, and for the most collapse-prone model, blocking tool access barely changes its score.
Limitations
- Benchmark limited to biomedical domain, potentially not generalizing to other fields.
- Difficulty anchoring relies on three specific open-weight models, which may bias the hardest subset and not reflect universal hardness.
- Agentic collapse observation is model-specific; extent of collapse may vary across different agent architectures.
- Checklist-based evaluation, while improving agreement, may still miss nuanced correctness or introduce checklist-specific biases.
- Unsolved nature means no ground-truth answers exist, so verification against follow-up evidence could be imperfect.