PaperScorer: How Predictable Is Paper Acceptance?

arXiv 2026 30.3 method

TLDR

PaperScorer predicts paper acceptance from text or page images, showing vision models outperform text and frontier reasoning models on ICLR and arXiv papers.

Reasoning

The paper presents a novel approach to modeling peer review decisions using page images, with systematic evaluation across datasets and strong calibration analysis. However, it only predicts accept/reject decisions and does not address broader research evaluation criteria, limiting its scope.

Read-first score

Read-first score 30.3, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 13.

Recency 8%

100

Uses a gentle age decay so recent papers surface without erasing older foundations. 2026

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=benchmark,evaluation

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=none

Topical relevance 42%

10.8

Uses existing LLM keyword relevance scores normalized to 0-100. AI scientist,automated scientific discovery,autonomous research agent,automated research,literature review agent,survey generation,automated experimentation,experiment design agent,AI for scientific research,paper writing agent,research automation,scientific discovery agent

Field roles

Frontier

Rank sensitivity

Stability: volatile; rank range: 38.

Keyword Scores

AI for scientific research

research automation

automated research

AI scientist

automated scientific discovery

scientific discovery agent

autonomous research agent

literature review agent

survey generation

automated experimentation

experiment design agent

paper writing agent

Resources

Paper