⌁ Awesome Auto Research Hub Papers · Datasets · Projects
中文 GitHub
← Back to papers

AutoEmpirical: LLM-Based Automated Research for Empirical Software Fault Analysis

arXiv 2025 51.4 method

TLDR

LLMs automate empirical software fault analysis, reducing processing time from weeks to two hours on 3,829 faults.

Reasoning

Strengths: addresses labor-intensive fault analysis, shows significant efficiency gains with LLMs, uses real-world data. Weaknesses: only initial exploration, not fully automated end-to-end, limited to software faults, lacks comparison with other methods.

Read-first score

Read-first score 51.4, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 35.

Methodology quality 25%
90

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=analysis,dataset,evaluation,result

Recency 8%
86.7

Uses a gentle age decay so recent papers surface without erasing older foundations. 2025

Reproducibility 25%
38

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=dataset

Topical relevance 42%
29.2

Uses existing LLM keyword relevance scores normalized to 0-100. AI scientist,automated scientific discovery,autonomous research agent,automated research,literature review agent,survey generation,automated experimentation,experiment design agent,AI for scientific research,paper writing agent,research automation,scientific discovery agent

Field roles

FrontierMethodology anchor

Rank sensitivity

Stability: volatile; rank range: 93.

Keyword Scores

research automation
7
automated research
6
AI for scientific research
5
automated scientific discovery
3
automated experimentation
3
AI scientist
2
autonomous research agent
2
experiment design agent
2
scientific discovery agent
2
literature review agent
1
survey generation
1
paper writing agent
1

Deep Analysis

Innovations

  • Decomposition of empirical software fault study into three phases: research objective definition, data preparation, and fault analysis.
  • Application of LLMs to automate fault analysis on a large-scale dataset of 3,829 software faults.
  • Demonstration of significant efficiency gains, reducing analysis time from weeks to about two hours.

Methodology

The study decomposes empirical fault analysis into three phases and applies LLMs to analyze 3,829 software faults sourced from a prior high-quality empirical study, measuring processing time against traditional manual effort.

Key Results

LLMs completed fault analysis in about two hours on average, compared to weeks of manual effort, demonstrating substantial efficiency improvement.

Limitations

  • The approach is an initial exploration and not yet a fully automated, end-to-end system.
  • Open challenges remain to achieve complete automation of the entire empirical fault study process.

Resources

Paper

Tags

SEAI