(Human) Attention Is (Still) All You Need: Human oversight makes AI-assisted social science reliable

arXiv 2026 54.3 method

TLDR

Human oversight via HLER architecture reduces AI-assisted social science research failures from 72% to 16%.

Reasoning

The paper presents a well-designed experiment with strong empirical evidence (Fisher's test p<0.001) and a real-world historical dataset. Its strength lies in demonstrating that reliability depends on structuring human-machine cognitive labor, not just model capability. A weakness is the narrow focus on social science and the specific HLER architecture, limiting generalizability.

Read-first score

Read-first score 54.3, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 33.

Methodology quality 25%

100

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=ablation,baseline,dataset,evaluation,experiment

Recency 8%

100

Uses a gentle age decay so recent papers surface without erasing older foundations. 2026

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=dataset

Topical relevance 42%

27.5

Uses existing LLM keyword relevance scores normalized to 0-100. AI scientist,automated scientific discovery,autonomous research agent,automated research,literature review agent,survey generation,automated experimentation,experiment design agent,AI for scientific research,paper writing agent,research automation,scientific discovery agent

Field roles

FrontierMethodology anchor

Rank sensitivity

Stability: volatile; rank range: 133.

Keyword Scores

AI for scientific research

automated experimentation

research automation

automated scientific discovery

automated research

experiment design agent

AI scientist

scientific discovery agent

autonomous research agent

literature review agent

survey generation

paper writing agent

Deep Analysis

Innovations

Human-in-the-Loop Economic Research (HLER) decision architecture incorporating pre-commitment, decision sequencing, accountability, and attention allocation
Three architectural commitments: LLMs reason but do not execute data work, data and estimation are handled deterministically, and three human decision gates bind the workflow
Conceptualization of the system as a research harness rather than an autonomous AI scientist, making residual weaknesses visible and preventing unreliable claims from advancing

Methodology

A pre-specified 2×4 factorial experiment with 280 complete research runs across four datasets compared an unconstrained multi-agent baseline to HLER, using the same underlying model, agent decomposition, and identical prompts for shared reasoning agents. Failure rates were compared with Fisher's exact test, and an 80-run ablation assessed the independent contributions of deterministic computation and human gates.

Key Results

The unconstrained baseline produced critical failures in 72% of runs, while HLER reduced the failure rate to 16% (p<0.001). Reliability gains were largest on the least publicly represented dataset (Qing-dynasty population register), and the ablation suggested independent contributions of deterministic computation and human gates with exploratory evidence of complementarity.

Limitations

A residual failure rate of 16% remains even with HLER
Evidence for complementarity between deterministic computation and human gates is exploratory
Evaluation is limited to four datasets and a single underlying model architecture
Human decision gates introduce human effort and potential bottlenecks not quantified in the abstract

Resources

Paper