TruthInsightBench: An Evidence-Grounded Benchmark for Automated Evaluation of Open-Ended Scientific Discovery Agents
TLDR
TruthInsightBench benchmarks open-ended scientific discovery agents with 40 blind tasks from peer-reviewed studies, using an automated LLM judge to evaluate evidentiary maturity; four coding agents plateau, lacking scientific judgment.
Reasoning
The benchmark innovatively targets discovery rather than reproduction, using blinded tasks and automated, deterministic scoring without per-instance human grading. However, evaluation is limited to one frozen base model and four coding agents, and the visible abstract does not provide detailed validity or reliability analyses.
Read-first score
Read-first score 77.2, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 65.
Field roles
Rank sensitivity
Stability: volatile; rank range: 38.
Keyword Scores
Deep Analysis
Innovations
- Introduces TruthInsightBench, a benchmark configured for scientific discovery rather than reproduction of hidden target studies.
- Uses 40 blind tasks from 40 peer-reviewed studies across 10 scientific domains, exposing only a neutral objective and frozen data while withholding source conclusions, expected values, and analysis paths.
- Employs a fixed LLM-based judge that scores evidentiary maturity along six dimensions operationalized as 29 artifact-grounded items.
- Provides automated, deterministic aggregation with no per-instance human grading, enabling repeatable evaluation as agents evolve.
Methodology
TruthInsightBench comprises 40 blind tasks drawn from 40 peer-reviewed studies across 10 scientific domains, giving agents a neutral scientific objective and frozen data while withholding source conclusions, expected values, and analysis paths. Agents must determine what claim the data support, and a fixed LLM-based judge scores the evidentiary maturity of their claims along six dimensions using 29 artifact-grounded items. Scoring is aggregated automatically and deterministically without per-instance human grading.
Key Results
On one frozen base model, four coding agents scored between 58.4 and 60.3 out of 100 with no statistically reliable pairwise separation. The agents executed and documented analyses competently and showed comparatively strong evidence auditability and novelty, but largely lacked controls, robustness, falsifiability, and cross-dataset generalization.
Limitations
- Reported agent performance is based on only one frozen base model, so generalizability across base models is not established.
- The evaluated coding agents largely lack the discriminating acts needed for trustworthy claims, including controls, robustness, falsifiability, and cross-dataset generalization.
- Genuine scientific discovery remains out of reach for the evaluated agents, indicating a wide remaining gap rather than demonstrated discovery capability.