Auto-MD-bench: Evaluating Autoresearch Agents for Real-World Medical Tasks
TLDR
Introduces Auto-MD-bench, a benchmark of 15 real-world medical tasks evaluating four autonomous ML agents, showing primary metric performance doesn't guarantee calibration or robustness.
Reasoning
The paper's strength is its realistic benchmark of clinical tasks and multi-agent evaluation, revealing that AUROC alone is insufficient for medical deployment. A weakness is that the abstract is truncated, leaving some metric and baseline details unclear.
Read-first score
Read-first score 62.5, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 51.
Field roles
Rank sensitivity
Stability: volatile; rank range: 118.
Keyword Scores
Deep Analysis
Innovations
- Introduces Auto-MD-bench, a benchmark of 15 real-world medical tasks from 10 clinical datasets with external validation across cohorts, hospitals, and time periods.
- Evaluates four autonomous research agents (AIDE, AI Scientist v2, Codex, and Autoresearch) on real-world medical ML tasks spanning survival analysis, classification, and sequential prediction.
- Shows that primary task performance alone is insufficient for assessing autonomous clinical modeling and that healthcare-relevant metrics such as calibration and robustness across population shifts must also be considered.
Methodology
Auto-MD-bench comprises 15 real-world medical tasks from 10 clinical datasets spanning survival analysis, classification, and sequential prediction, with external validation across cohorts, hospitals, and time periods. Four autonomous agents (AIDE, AI Scientist v2, Codex, and Autoresearch) are evaluated on primary predictive performance and additional healthcare-relevant metrics including calibration and robustness to population shifts.
Key Results
The strongest agents achieve competitive primary scores, but their predicted risks are less calibrated than simple baseline models, and models with nearly identical primary scores can show markedly different calibration and substantial performance variation across population shifts.