IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications

arXiv 2026 34.6 method

TLDR

Introduces IdeaAMBIG, a benchmark of 660 evidence-grounded instances for evaluating whether research-method specifications are sufficiently detailed for implementation by coding agents.

Reasoning

The paper presents a novel benchmark combining real-world and synthetic gaps, evaluating LLMs on codification-readiness, defect localization, and clarification generation. Strengths include evidence-grounded construction and multi-model evaluation; weaknesses are not fully visible from the abstract, though the low defect localization rates suggest this capability remains a key bottleneck.

Read-first score

Read-first score 34.6, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 18.

Recency 8%

100

Uses a gentle age decay so recent papers surface without erasing older foundations. 2026

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=artifact,code,github

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=benchmark

Topical relevance 42%

Uses existing LLM keyword relevance scores normalized to 0-100. AI scientist,automated scientific discovery,autonomous research agent,automated research,literature review agent,survey generation,automated experimentation,experiment design agent,AI for scientific research,paper writing agent,research automation,scientific discovery agent

Field roles

Frontier

Rank sensitivity

Stability: volatile; rank range: 21.

Keyword Scores

AI for scientific research

research automation

automated research

autonomous research agent

AI scientist

automated scientific discovery

scientific discovery agent

literature review agent

survey generation

automated experimentation

experiment design agent

paper writing agent

Resources

Paper