Predictive Credit: Measuring What Scientific Explanations Add to Experimental Forecasts
TLDR
Introduces a protocol measuring predictive credit of research-agent explanations via paired experimental forecasts, tested on controlled learning, Tox21, and OpenML tasks.
Reasoning
The paper presents a rigorous, multi-check evaluation framework for assessing explanation value in scientific forecasting, with extensive empirical tests. However, results are largely inconclusive or negative for predictive gains, limiting support for its core claims. Strengths include preregistration and diverse benchmarks; weaknesses include unmet primary tests and reliance on specific model replays.
Read-first score
Read-first score 47, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 32.
Field roles
Rank sensitivity
Stability: volatile; rank range: 72.
Keyword Scores
Deep Analysis
Innovations
- Introduces predictive credit as a measure of what scientific explanations add to experimental forecasts.
- Uses paired forecasts that share an intervention, forecaster, and outcome while varying description, matched explanation, and donor context.
- Defines five checks tracking commitment, delivery, predictive gain, alignment, and known-signal uptake.
- Applies the protocol across controlled learning, Tox21 endpoints, and OpenML tasks with preregistered interval-score harm and equivalence tests.
Methodology
The study compares paired forecasts that share an intervention, forecaster, and outcome while varying whether the forecaster receives a description, matched explanation, or donor context. It applies five checks tracking commitment, delivery, predictive gain, alignment, and known-signal uptake across 336 prospective controlled-learning states, 12 Tox21 endpoints, and 24 OpenML tasks. Metrics include ROC AUC interval-score harm tests, point MAE, coverage intervals, and preregistered formation, point-equivalence, and repeatability rules.
Key Results
The v5 frozen credit decision was inconclusive; Tox21's preregistered ROC AUC interval-score harm test was unmet (D-M=-.0026, 95% interval [-.0174, .0104]) and OpenML's joint formation, point-equivalence, and repeatability rule was unmet. Under DeepSeek V4 Pro, matched and donor cards reduced secondary Tox21 drift by 64.5% and 59.1%, but Flash replay raised matched point MAE from .01823 to .02020, OpenML full-card intervals widened by 21% with 49.3% coverage versus 51.4%, and only a researcher-authored mechanism positive control lowered point MAE by 2.60 percentage points versus description.
Limitations
- The v5 frozen credit decision was inconclusive.
- Natural-explanation credit remained unconfirmed at the tested donor resolutions.
- Matched point-accuracy gains over description remained unconfirmed, and Tox21/OpenML seed-donor intervals spanned zero.
- Tox21's preregistered ROC AUC interval-score harm test was unmet, and OpenML's joint formation, point-equivalence, and repeatability rule was unmet.
- DeepSeek V4 Flash replay raised matched point MAE and missed matched-donor interval-score equivalence.
- OpenML full-card assignment widened nominal 80% intervals and showed lower coverage than description/content.
- Direct-text Flash delivered all 144 notes without detectable matched point-accuracy gain.