FirstResearch: Auditable Question Formation for LLM Scientific Discovery Agents

arXiv 2026 70.1 method

TLDR

FirstResearch introduces a structured Research Question Certificate for LLM scientific discovery agents, enabling auditable question formation and outperforming baselines in preliminary evaluations.

Reasoning

The paper presents a novel framework for making research questions inspectable, with clear methodology and comparative evaluation. Strengths include the certificate design and strong preliminary results; weaknesses are the limited scope (only question formation, not full discovery) and reliance on LLM-based judges without real-world validation.

Read-first score

Read-first score 70.1, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 52.

Recency 8%

100

Uses a gentle age decay so recent papers surface without erasing older foundations. 2026

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=ablation,baseline,experiment,result

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=True; dataset=False; markers=artifact,checkpoint,code,github

Topical relevance 42%

43.3

Uses existing LLM keyword relevance scores normalized to 0-100. AI scientist,automated scientific discovery,autonomous research agent,automated research,literature review agent,survey generation,automated experimentation,experiment design agent,AI for scientific research,paper writing agent,research automation,scientific discovery agent

Field roles

FrontierMethodology anchorReproducibility anchor

Rank sensitivity

Stability: volatile; rank range: 94.

Keyword Scores

AI for scientific research

scientific discovery agent

AI scientist

autonomous research agent

automated scientific discovery

automated research

research automation

experiment design agent

automated experimentation

literature review agent

survey generation

paper writing agent

Deep Analysis

Innovations

FirstResearch: a first-principles research-question formation framework for scientific LLM agents that produces auditable questions
Research Question Certificate: a structured artifact recording primitive definitions, assumptions, mechanism, tension, falsifiable hypothesis, test, and failure update rule to enable explicit inspection

Methodology

The framework is evaluated on ten LLM-agent research topics, comparing against prompt-level baselines inspired by AI co-scientist, Agent Laboratory, and AI Scientist-v2. Quality is judged via a primary DeepSeek-blind-judge protocol and an independent Gemini-2.5-Flash rescore, with an ablation study isolating the certificate component.

Key Results

FirstResearch scores 4.86/5 versus 4.38/5 for the strongest baseline under Gemini rescore, with Pearson agreement 0.865. Ablation shows certificate-only scoring reaches 4.90/5 (DeepSeek) and 4.88/5 (Gemini), while removing certificates drops below 1/5.

Limitations

Results are preliminary and rely on LLM judges rather than human domain experts

Resources

Paper Code