Measuring the Creativity of Frontier LLMs in Automated Research

arXiv 2026 46.8 method

TLDR

Proposes creativity metrics for frontier LLMs in automated research, evaluating idea valueness and novelty, and finds variable-level novelty correlates with research performance.

Reasoning

The paper introduces a systematic evaluation framework for LLM creativity in automated research, which is a strength. However, the abstract lacks methodological details and the proposed metrics may require further validation beyond the reported correlations.

Read-first score

Read-first score 46.8, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 46.

Recency 8%

100

Uses a gentle age decay so recent papers surface without erasing older foundations. 2026

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=evaluation,metric

Topical relevance 42%

38.3

Uses existing LLM keyword relevance scores normalized to 0-100. AI scientist,automated scientific discovery,autonomous research agent,automated research,literature review agent,survey generation,automated experimentation,experiment design agent,AI for scientific research,paper writing agent,research automation,scientific discovery agent

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=none

Field roles

Frontier

Rank sensitivity

Stability: volatile; rank range: 39.

Keyword Scores

automated research

research automation

AI for scientific research

automated scientific discovery

AI scientist

autonomous research agent

scientific discovery agent

automated experimentation

experiment design agent

literature review agent

survey generation

paper writing agent

Deep Analysis

Innovations

Proposes a set of metrics to systematically evaluate creativity in frontier LLMs for automated research along two dimensions: valueness and novelty.
Introduces Valueness to assess whether each proposed idea is useful.
Introduces Exact-Match P-Novelty to determine whether the same idea has appeared before.
Introduces Variable-level P-Novelty to assess whether the modified variable or variable combination has been explored before, capturing breadth of research-space exploration.
Introduces H-Novelty to assess whether the proposed idea is explicitly attributed to external knowledge in the model's reasoning.
Establishes a link between Variable-level P-Novelty and research performance through correlation and idea-level analyses.

Methodology

The study proposes metrics for evaluating creativity in automated research along the dimensions of valueness and novelty, with novelty measured via Exact-Match P-Novelty, Variable-level P-Novelty, and H-Novelty. It evaluates frontier LLMs using these metrics and analyzes correlations and idea-level performance in relation to Variable-level P-Novelty.

Key Results

Evaluated models achieve relatively similar Valueness and Exact-Match P-Novelty scores, but differ substantially in Variable-level P-Novelty; H-Novelty is consistently high among models for which it can be evaluated. A strong positive correlation is found between Variable-level P-Novelty and research performance.

Resources

Paper