1GC-7RC: One Graphic Card -- Seven Research Challenges! How Good Are AI Agents at Doing Your Job?
TLDR
Introduces 1GC-7RC benchmark evaluating AI coding agents on seven ML tasks with constraints, revealing performance differences.
Reasoning
Strengths include a novel benchmark with diverse tasks, realistic constraints, and public artifacts. Weaknesses are limited scope to coding agents and lack of broader scientific discovery tasks; evaluation on proprietary agents may hinder reproducibility.
Read-first score
Read-first score 65.9, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 47.
Field roles
Rank sensitivity
Stability: volatile; rank range: 105.
Keyword Scores
Deep Analysis
Innovations
- Introduces 1GC-7RC, the first standardized benchmark for evaluating autonomous AI coding agents on end-to-end ML model design, implementation, and training from scratch across diverse domains.
- Benchmark design enforces realistic constraints: single GPU, no internet, no pretrained weights (with one controlled exception), and strict wall-clock budgets, testing implicit ML knowledge, planning, and time management.
- Modular and extensible platform that can be adapted to new tasks, GPU budgets, and multi-agent settings for future research on autonomous research agents.
Methodology
The benchmark comprises seven ML tasks (language modeling, image classification, semantic segmentation, graph learning, tabular prediction, time-series forecasting, text classification) with locked data-preparation and evaluation scripts plus a baseline training script. Agents may only modify the training code, have no internet access, no pretrained weights (except one controlled exception for segmentation), and must complete each task within a 40–120 minute wall-clock budget on a single GPU. Seven coding agents (five proprietary, two open-source) are evaluated over five runs per agent-task pair.
Key Results
Substantial performance differences are observed across agents, revealing varying levels of implicit ML knowledge, planning ability, and time-budget management.
Limitations
- Limited to seven tasks, which may not cover all ML subfields or real-world complexity.
- Single-GPU and time-budget constraints restrict applicability to larger-scale projects.
- No internet access prevents agents from leveraging external resources, which may not reflect typical development workflows.
- The controlled exception allowing pretrained weights for semantic segmentation introduces inconsistency across tasks.