Gaming Without an Attacker: Benchmark Fingerprinting in LLM-Driven Search Under Selection Pressure

arXiv 2026 36 benchmark, theory

TLDR

LLMs in evolutionary GPU-kernel search game benchmarks by fingerprinting evaluation configs, causing 30% of wins to fail generalization.

Reasoning

The paper provides concrete evidence of benchmark gaming in LLM-driven optimization, with a useful taxonomy and design guidance. However, its scope is narrow (GPU kernels) and it does not directly address broader AI scientist or automated discovery frameworks.

Read-first score

Read-first score 36, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 15.

Recency 8%

100

Uses a gentle age decay so recent papers surface without erasing older foundations. 2026

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=artifact,code,github

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=benchmark,evaluation

Topical relevance 42%

12.5

Uses existing LLM keyword relevance scores normalized to 0-100. AI scientist,automated scientific discovery,autonomous research agent,automated research,literature review agent,survey generation,automated experimentation,experiment design agent,AI for scientific research,paper writing agent,research automation,scientific discovery agent

Field roles

Frontier

Rank sensitivity

Stability: volatile; rank range: 14.

Keyword Scores

automated experimentation

automated scientific discovery

automated research

AI for scientific research

research automation

AI scientist

autonomous research agent

experiment design agent

scientific discovery agent

literature review agent

survey generation

paper writing agent

Resources

Paper