⌁ Awesome Auto Research Hub Papers · Datasets · Projects
中文 GitHub
← Back to papers

CodeScientist: End-to-End Semi-Automated Scientific Discovery with Code-based Experimentation

arXiv 2025 64.1 method

TLDR

CodeScientist uses genetic search over articles and codeblocks for semi-automated scientific discovery, yielding 19 discoveries with multi-faceted evaluation.

Reasoning

The paper presents a novel ASD system that combines genetic search with code-based experimentation, addressing limitations of prior work by exploring broader design spaces and using multi-faceted evaluation including code review and replication. However, the system is semi-automated and limited to the domain of agents and virtual environments, and the evaluation still relies on human judgment.

Read-first score

Read-first score 64.1, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 66.

Methodology quality 25%
90

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=benchmark,evaluation,experiment,metric

Recency 8%
86.7

Uses a gentle age decay so recent papers surface without erasing older foundations. 2025

Topical relevance 42%
55

Uses existing LLM keyword relevance scores normalized to 0-100. AI scientist,automated scientific discovery,autonomous research agent,automated research,literature review agent,survey generation,automated experimentation,experiment design agent,AI for scientific research,paper writing agent,research automation,scientific discovery agent

Reproducibility 25%
46

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=artifact,code

Field roles

FrontierMethodology anchor

Rank sensitivity

Stability: volatile; rank range: 33.

Keyword Scores

automated scientific discovery
9
automated experimentation
8
scientific discovery agent
8
automated research
7
AI for scientific research
7
research automation
7
AI scientist
6
autonomous research agent
5
experiment design agent
5
literature review agent
2
survey generation
1
paper writing agent
1

Deep Analysis

Innovations

  • Framing ideation and experiment construction as genetic search over combinations of research articles and codeblocks defining common actions in a domain.
  • Multi-faceted evaluation beyond conference-style review, including code review and replication attempts.
  • Generating discoveries that span new tasks, agents, metrics, and data, moving beyond benchmark optimization.

Methodology

CodeScientist uses a genetic search paradigm that jointly combines research articles and codeblocks (defining common actions like prompting a language model) to generate ideas and construct experiments. It conducted hundreds of automated experiments on machine-generated ideas in the domain of agents and virtual environments.

Key Results

The system returned 19 discoveries, 6 of which were judged at least minimally sound and incrementally novel after external review, code review, and replication attempts. These discoveries spanned new tasks, agents, metrics, and data.

Limitations

  • Only 6 out of 19 discoveries were judged sound and incrementally novel, indicating many outputs were not sound or novel.
  • The system is semi-automated, not fully autonomous.
  • Evaluation limited to the domain of agents and virtual environments.

Resources

Paper

Tags

AICL