ScienceIDE: Turning World's Scientific Codebase into Agent Learnable Environments
TLDR
ScienceIDE converts scientific code repositories into executable environments for training and evaluating scientific agents, yielding model families with improved code repair and general reasoning.
Reasoning
The paper presents a novel infrastructure and trained models, with concrete benchmarks and code release. However, the abstract lacks detailed evaluation methodology and limitations, making the strength of the reported gains difficult to fully assess.
Read-first score
Read-first score 54.3, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 49.
Field roles
Rank sensitivity
Stability: volatile; rank range: 52.
Keyword Scores
Deep Analysis
Innovations
- Introduces ScienceIDE, infrastructure for turning scientific code repositories into programmable environments for scientific agents.
- Identifies and frames the 'scientific experience bottleneck' arising from fragmented toolchains, implicit domain conventions, and specialized correctness criteria.
- Uses expert-defined scientific cases and acceptance criteria to guide agents in transforming repositories into executable environments supporting task generation, execution, and scientific verification.
- Creates a shared foundation for supervised fine-tuning, reinforcement learning, and evaluation using verified interaction trajectories.
- Trains the PhAI-IDE model family at 72B, 9B, and 4B scales and reports transfer from scientific experience to broader code, reasoning, and knowledge capabilities.
Methodology
ScienceIDE uses expert-defined scientific cases and acceptance criteria to guide agents in converting scientific repositories into executable environments that support task generation, execution, and scientific verification. Verified interaction trajectories from these environments are then used for supervised fine-tuning, reinforcement learning, and evaluation. The authors train PhAI-IDE-72B, PhAI-IDE-9B, and PhAI-IDE-4B on this basis.
Key Results
The PhAI-IDE model family shows gains in held-out scientific-code repair and across selected general-purpose benchmarks in code, reasoning, and knowledge. The authors interpret these gains as evidence of positive transfer from scientific experience to broader capabilities.
Limitations
- Evaluation is reported on held-out scientific-code repair and selected general-purpose benchmarks, so coverage beyond these benchmarks is not established.
- The broader transfer claim is framed as 'providing evidence' rather than definitive proof.
- The abstract notes that fragmented toolchains, implicit domain conventions, and specialized correctness criteria make converting scientific code into reliable learning experiences difficult.