RATIO: A Benchmark for Retrieval Across Typed Ideation Operations in Scientific Literature

arXiv 2026 32.3 method

TLDR

Introduces RATIO, a benchmark for retrieving scientific literature by ideation operations (Address, Broaden, Specify), built via distant supervision and human/LLM vetting.

Reasoning

The paper presents a novel, large-scale benchmark with clear methodology and empirical evaluation, showing fine-tuning improves retrievers. However, its focus is on retrieval operations, not full automated discovery or experimentation, limiting relevance to broader AI-scientist keywords.

Read-first score

Read-first score 32.3, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 28.

Recency 8%

100

Uses a gentle age decay so recent papers surface without erasing older foundations. 2026

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=benchmark,evaluation,experiment

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=none

Topical relevance 42%

9.5

Matches configured research keywords against title, abstract, tags, and analysis text. matched=2

Field roles

Frontier

Rank sensitivity

Stability: volatile; rank range: 29.

Keyword Scores

AI for scientific research

research automation

automated research

AI scientist

automated scientific discovery

autonomous research agent

literature review agent

scientific discovery agent

survey generation

automated experimentation

experiment design agent

paper writing agent

Resources

Paper