Research Design Tracking and Assessment for the Social Sciences

arXiv 2026 37.2 method

TLDR

Introduces ARDTrA, an expert-annotated task and RAG pipeline for automatically detecting and assessing causal research designs in social science papers, finding passage length drives performance.

Reasoning

The paper contributes a novel annotated dataset and systematically evaluates LLM-based RAG pipelines across multiple configurations, which is a strength. Its main limitation is the narrow focus on social science research designs, and the assessment of quality may depend heavily on expert annotations.

Read-first score

Read-first score 37.2, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 27.

Recency 8%

100

Uses a gentle age decay so recent papers surface without erasing older foundations. 2026

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=analysis,dataset

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=dataset

Topical relevance 42%

22.5

Uses existing LLM keyword relevance scores normalized to 0-100. AI scientist,automated scientific discovery,autonomous research agent,automated research,literature review agent,survey generation,automated experimentation,experiment design agent,AI for scientific research,paper writing agent,research automation,scientific discovery agent

Field roles

Frontier

Rank sensitivity

Stability: volatile; rank range: 48.

Keyword Scores

AI for scientific research

automated research

research automation

experiment design agent

AI scientist

automated scientific discovery

autonomous research agent

literature review agent

automated experimentation

survey generation

paper writing agent

scientific discovery agent

Resources

Paper