AutoSOTA: An End-to-End Automated Research System for State-of-the-Art AI Model Discovery

arXiv 2026 67.6 method

TLDR

AutoSOTA is an end-to-end multi-agent system that automates replication and optimization of SOTA AI models, discovering 105 new SOTA models across top conferences.

Reasoning

The paper presents a novel multi-agent architecture for automated research, with strong empirical results (105 new SOTA models) and evaluation across diverse domains. However, it relies on code availability and execution cost filters, and the abstract does not detail limitations or failure cases, which may affect generalizability.

Read-first score

Read-first score 67.6, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 80.

Recency 8%

100

Uses a gentle age decay so recent papers surface without erasing older foundations. 2026

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=evaluation,experiment,result

Topical relevance 42%

66.7

Uses existing LLM keyword relevance scores normalized to 0-100. AI scientist,automated scientific discovery,autonomous research agent,automated research,literature review agent,survey generation,automated experimentation,experiment design agent,AI for scientific research,paper writing agent,research automation,scientific discovery agent

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=code,reproduce

Field roles

FrontierMethodology anchor

Rank sensitivity

Stability: volatile; rank range: 21.

Keyword Scores

automated research

research automation

automated scientific discovery

automated experimentation

AI for scientific research

autonomous research agent

scientific discovery agent

experiment design agent

AI scientist

literature review agent

survey generation

paper writing agent

Deep Analysis

Innovations

End-to-end automated research system that reproduces and improves state-of-the-art AI models from top-tier papers
Multi-agent architecture with eight specialized agents for paper-to-code grounding, environment repair, experiment tracking, idea generation, and validity supervision
Three-stage problem formulation: resource preparation and goal setting, experiment evaluation, and reflection and ideation

Methodology

AutoSOTA uses a multi-agent system with eight agents that ground papers to code, initialize and repair execution environments, track long-horizon experiments, generate and schedule optimization ideas, and supervise validity. It is evaluated on recent papers from eight top-tier AI conferences, filtered for code availability and execution cost, measuring end-to-end replication and subsequent optimization.

Key Results

The system discovered 105 new SOTA models surpassing original reported results, averaging about five hours per paper, and demonstrated improvements beyond hyperparameter tuning, including architectural innovations and algorithmic redesigns across LLM, NLP, computer vision, time series, and optimization domains.

Limitations

Evaluation limited to papers with available code and manageable execution cost
May not generalize to papers without code or requiring prohibitive computational resources
Optimization ideation is constrained by the predefined capabilities of the agents

Resources

Paper