Can LLMs Generate Novel Research Ideas? A Large Scale Human Study with 100+ NLP Researchers

arXiv '24 2024 18.3 benchmark

Read-first score

Read-first score 18.3, weighted from topical fit, citation, graph, method, reproducibility, and recency signals.

Recency 11%

75.1

Uses a gentle age decay so recent papers surface without erasing older foundations. 2024

Reproducibility 33%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=none

Topical relevance 56%

Matches configured research keywords against title, abstract, tags, and analysis text. matched=0

Candidate

Stability: stable; rank range: 1.