Predictive Credit: Measuring What Scientific Explanations Add to Experimental Forecasts
TLDR
Introduces a protocol measuring predictive credit of research-agent explanations via paired experimental forecasts, tested on controlled learning, Tox21, and OpenML tasks.
评分理由
The paper presents a rigorous, multi-check evaluation framework for assessing explanation value in scientific forecasting, with extensive empirical tests. However, results are largely inconclusive or negative for predictive gains, limiting support for its core claims. Strengths include preregistration and diverse benchmarks; weaknesses include unmet primary tests and reliance on specific model replays.
Read-first 评分解释
综合优先阅读分 47，由主题、引用、图谱、方法、可复现性和近期性等信号加权得到。 原始总分保留为 32。
研究版图角色
排序敏感性
稳定性：volatile；排名波动范围：72。
关键词评分
深度分析
创新点
- Introduces predictive credit as a measure of what scientific explanations add to experimental forecasts.
- Uses paired forecasts that share an intervention, forecaster, and outcome while varying description, matched explanation, and donor context.
- Defines five checks tracking commitment, delivery, predictive gain, alignment, and known-signal uptake.
- Applies the protocol across controlled learning, Tox21 endpoints, and OpenML tasks with preregistered interval-score harm and equivalence tests.
方法
The study compares paired forecasts that share an intervention, forecaster, and outcome while varying whether the forecaster receives a description, matched explanation, or donor context. It applies five checks tracking commitment, delivery, predictive gain, alignment, and known-signal uptake across 336 prospective controlled-learning states, 12 Tox21 endpoints, and 24 OpenML tasks. Metrics include ROC AUC interval-score harm tests, point MAE, coverage intervals, and preregistered formation, point-equivalence, and repeatability rules.
关键结果
The v5 frozen credit decision was inconclusive; Tox21's preregistered ROC AUC interval-score harm test was unmet (D-M=-.0026, 95% interval [-.0174, .0104]) and OpenML's joint formation, point-equivalence, and repeatability rule was unmet. Under DeepSeek V4 Pro, matched and donor cards reduced secondary Tox21 drift by 64.5% and 59.1%, but Flash replay raised matched point MAE from .01823 to .02020, OpenML full-card intervals widened by 21% with 49.3% coverage versus 51.4%, and only a researcher-authored mechanism positive control lowered point MAE by 2.60 percentage points versus description.
局限性
- The v5 frozen credit decision was inconclusive.
- Natural-explanation credit remained unconfirmed at the tested donor resolutions.
- Matched point-accuracy gains over description remained unconfirmed, and Tox21/OpenML seed-donor intervals spanned zero.
- Tox21's preregistered ROC AUC interval-score harm test was unmet, and OpenML's joint formation, point-equivalence, and repeatability rule was unmet.
- DeepSeek V4 Flash replay raised matched point MAE and missed matched-donor interval-score equivalence.
- OpenML full-card assignment widened nominal 80% intervals and showed lower coverage than description/content.
- Direct-text Flash delivered all 144 notes without detectable matched point-accuracy gain.