OSWorld-Science: A Benchmark of Computer Use Agents for Learning and Using Scientific Software
TLDR
Introduces OSWorld-Science, a benchmark and evaluation environment for computer-using agents on scientific software tasks, with 12 VLMs and 146 tasks across domains.
Reasoning
The benchmark is well-designed with artifact-based evaluation and expert-AI co-design, providing strong empirical results. However, the abstract focuses on task coverage and harness rather than broader claims of autonomous discovery, limiting relevance for discovery-focused keywords.
Read-first score
Read-first score 52.8, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 36.
Field roles
Rank sensitivity
Stability: volatile; rank range: 139.
Keyword Scores
Deep Analysis
Innovations
- Introduces OSWorld-Science, a benchmark and evaluation environment for computer-using agents in scientific software workflows.
- Provides 146 scientifically meaningful tasks across multiple scientific domains and software configurations, developed through expert proposals and iterative human-AI co-design.
- Uses artifact-based, execution-based evaluation with task-specific evaluators that inspect application states and generated artifacts and award partial credit.
- Integrates a specialized agent harness with model adapters, interaction-loop control, and trajectory logging to compare models and interaction strategies.
- Analyzes 12 VLMs across multiple linguistic, reasoning effort, context length, and other factors to derive directions for future development.
Methodology
The benchmark contains 146 tasks across domains such as molecular drawing and retrosynthesis, pathology image analysis, statistical computing, and physical simulation. Tasks are developed through expert proposals and iterative human-AI co-design, with selection guided by scientific value and difficulty. Evaluation uses task-specific execution-based evaluators that inspect application states and generated artifacts, including molecular structures, segmentation masks, plots, and numerical results, and award partial credit; a custom harness integrates model adapters, interaction-loop control, and trajectory logging to support comparisons of 12 VLMs and interaction strategies.
Key Results
Current state-of-the-art VLMs, even with a strong harness, still face challenges in addressing key questions in scientific domains. The paper also reports analyses across multi-linguistics, reasoning effort, context length, and other factors to derive conclusions and future directions.
Limitations
- The abstract does not state specific limitations of the benchmark or study beyond the broad finding that current VLMs still face challenges in scientific software workflows.