Dataset zoo
Datasets grouped by representation and task.
Autonomous coding agents are increasingly proposed as AI-scientist systems that conduct analyses and write research reports, but executing a prescribed analysis is not the same as making a discovery. Existing benchmarks are configured for r...
AI scientist systems are increasingly deployed for autonomous research, yet their academic integrity has never been systematically evaluated. We introduce SCIINTEGRITY-BENCH, the first benchmark designed around a dilemmatic evaluation parad...
LLMs have made significant progress in complex but easy-to-verify problems, yet they still struggle with discovering the unknown. In this paper, we present \textbf{AlphaResearch}, an autonomous research agent designed to discover new algori...
Automated research systems support literature synthesis, ideation, experiments, writing, and peer review, but their evaluation is dispersed across tasks, benchmarks, and studies that are difficult to compare directly. We review this literat...
The automation of scientific discovery has been a long-standing goal within the research community, driven by the potential to accelerate knowledge creation. While significant progress has been made using commercial large language models (L...
Pursuing artificial intelligence for biomedical science, a.k.a. AI Scientist, draws increasing attention, where one common approach is to build a copilot agent driven by Large Language Models (LLMs). However, to evaluate such systems, peopl...
Scientific progress increasingly relies on effective collaboration among researchers, a dynamic that large language models (LLMs) have only begun to emulate. While recent LLM-based scientist agents show promise in autonomous scientific disc...
Introduces Agent4Sci, a testbed for evaluating frontier agents on scientific ML benchmarks; agents improve results via engineering tricks, not research insight, but plug-in skills boost performance.
Agentic AI "scientists" now use language models to search the literature, run analyses, and generate hypotheses. We evaluate KOSMOS, an autonomous AI scientist, on three problems in radiation biology using simple random-gene null benchmarks...
As foundation models advance and agent scaffolding becomes increasingly sophisticated, agents have demonstrated remarkable proficiency in complex, long-horizon coding tasks and even autonomous experiment execution. Despite their evolution f...
Apodex Discovery introduces benchmarks and environments for discoverative AI, with a heavy-duty solver and HDS6 evaluation, surpassing baselines in AAV capsid design and drug repurposing.
Introduces Auto-MD-bench, a benchmark of 15 real-world medical tasks evaluating four autonomous ML agents, showing primary metric performance doesn't guarantee calibration or robustness.
The rapid development of automated scientific survey generation technology has made it increasingly important to establish a comprehensive benchmark to evaluate the quality of generated surveys.Nearly all existing evaluation benchmarks rely...
AutoDataBench is a controlled data-centric testbed isolating data intelligence in auto-research, evaluating frontier LLMs' data diagnosis, organization, construction, and effect-understanding via iterative experimentation.
Introduces MechHypoBench, a benchmark evaluating AI agents' ability to generate mechanistic hypotheses from real-world datasets across 14 fields, revealing a gap between generated and true mechanisms.
Scientific reasoning is critical for developing AI scientists and supporting human researchers in advancing the frontiers of natural science discovery. However, the open-source community has primarily focused on mathematics and coding while...
Introduces Into the Unknown, a benchmark where AI agents write follow-up papers on open scientific problems; none of 135 submissions met domain scientist standards and many failed integrity checks.
Search agents are language models (LMs) that reason and search knowledge bases (or the web) to answer questions; recent methods supervise only the final answer accuracy using reinforcement learning with verifiable rewards (RLVR). Most RLVR ...
Automatic Survey Generation (ASG) aims to produce comprehensive literature surveys by retrieving, organizing, and synthesizing academic papers. Despite rapid progress in specialized ASG frameworks and Deep Research agents, existing evaluati...
We introduce MLRC-Bench, a benchmark designed to quantify how effectively language agents can tackle challenging Machine Learning (ML) Research Competitions, with a focus on open research problems that demand novel methodologies. Unlike pri...
Introduces AutoWorldModel-Bench, a closed-loop benchmark where frontier coding agents autonomously improve world-model starters across eight game environments, showing research-style modifications dominate.
Introduces OSWorld-Science, a benchmark and evaluation environment for computer-using agents on scientific software tasks, with 12 VLMs and 146 tasks across domains.
SDABench evaluates LLMs on six scientific data analysis capabilities across five domains, finding strengths in descriptive analysis but weaknesses in assumption selection and mechanistic reasoning.
Large Language Models (LLMs) are being explored for applications in scientific research, including their capabilities to synthesize literature, answer research questions, generate research ideas, and even conduct computational experiments. ...
Recent advances in large language models (LLMs) have fueled the vision of automated scientific discovery, often called AI Co-Scientists. To date, prior work casts these systems as generative co-authors responsible for crafting hypotheses, s...
Self-driving laboratories (SDLs) have shown promise to accelerate materials discovery by integrating machine learning with automated experimental platforms. However, errors in the capture of input parameters may corrupt the features used to...
Presents InternReviewer and InternAdvocate, scholarly agents for peer review and rebuttal trained via agentic RL with retrieval and citation verification to improve reasoning and accuracy.
Rapid advancements in large language models (LLMs) have the potential to assist in scientific progress. A critical capability toward this endeavor is the ability to reproduce existing work. To evaluate the ability of AI agents to reproduce ...
Despite rapid advances in autonomous AI scientists powered by language models, generating publication-ready illustrations remains a labor-intensive bottleneck in the research workflow. To lift this burden, we introduce PaperBanana, an agent...
Discovering governing equations directly from observational data is a key step towards interpretable scientific machine learning. Current data-driven approaches typically operate on a single dataset, inherently limiting their performance wh...
Existing benchmarks for autonomous AI scientists evaluate only final outputs---generated code, hypotheses, or papers---yet discard the reasoning process by which those outputs were obtained. This makes it impossible to audit scientific meth...
Partial differential equations (PDEs) are essential for modeling complex physical systems, and neural solvers have recently emerged as powerful data-driven tools for numerically solving them. However, existing neural solvers struggle with d...
AI-powered scientific research tools are rapidly being integrated into research workflows, yet the field lacks a clear lens into how researchers use these systems in real-world settings. We present and analyze the Asta Interaction Dataset, ...
Would experience designing faster GPU kernels also help close in on a long-standing open mathematical conjecture? Large Language Models (LLMs) integrated into evolutionary search have recently produced state-of-the-art solutions on optimiza...
Introduces compositional controllability for fair evaluation of biomedical literature-review agents, with a new benchmark and tests showing many leaderboard comparisons are inadmissible.
The materials science literature encodes decades of experimental knowledge in figures, yet this visual record remains locked away and inaccessible to AI at scale. The core difficulty is structural: most scientific figures are compound, with...
Derived from paper: DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis
Derived from paper: Probing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows
Generative molecular design is shaped by simple proxy benchmarks for drug-like properties and models pretrained on large pharmaceutical datasets. This combination yields strong benchmark metrics but limits transferability to domains structu...
The rigorous evaluation of the novelty of a scientific paper is, even for human scientists, a challenging task. With the increasing interest in AI scientists and AI involvement in scientific idea generation and paper writing, it also become...
Reliable assessment of causal research designs in the social sciences is critical for evidence-based policy-making, yet has so far relied entirely on manual expert analysis. We introduce Automated Research Design Tracking and Assessment (AR...
Derived from paper: AI Idea Bench 2025: AI Research Idea Generation Benchmark
Derived from paper: ReportBench: Evaluating Deep Research Agents via Academic Survey Tasks
Derived from paper: SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers
LLMs in evolutionary GPU-kernel search game benchmarks by fingerprinting evaluation configs, causing 30% of wins to fail generalization.
As agents rapidly evolve, existing benchmarks can become saturated, limiting their ability to distinguish capabilities and reveal remaining failure modes. Particularly in scientific domains, constructing and updating benchmarks requires sub...
What makes great scientists great? Even as AI systems start to make progress on open problems, scientists remain far ahead of them at sensing which prior idea, buried in an ever-growing archive of research, a new problem needs. To study thi...
Derived from paper: PostTrainBench: Can LLM Agents Automate LLM Post-Training?
Derived from paper: DiscoveryBench: Towards Data-Driven Discovery with Large Language Models
Derived from paper: MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation
Can LLM agents make contributions to frontier computer science R&D? We introduce FrontierCS 2, a benchmark of 48 open-ended tasks. We take a paper and the code its authors released and delete the paper's contribution. The deleted code becom...
DianShi-RxnDB is a large-scale organic reaction data platform with ~24M instances, built via automated extraction from patents, and offering web/MCP access for researchers and AI agents.
Derived from paper: AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery
Derived from paper: Process-Oriented Evaluation of AI-Assisted Scientific Writing
Derived from paper: ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research
Derived from paper: SciFlow-Bench: Evaluating Structure-Aware Scientific Diagram Generation via Inverse Parsing
Derived from paper: SciNetBench: A Relation-Aware Benchmark for Scientific Literature Retrieval Agents
Derived from paper: ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition
Derived from paper: ClaimCheck: How Grounded are LLM Critiques of Scientific Papers?
Derived from paper: FrontierScience: Evaluating AI's Ability to Perform Expert-Level Scientific Tasks
Derived from paper: LiveIdeaBench: Evaluating LLMs' Scientific Creativity and Idea Generation with Minimal Context
Derived from paper: PseudoBench: Measuring How Agentic Auto-Research Fuels Pseudoscience
Derived from paper: ScholarGym: Benchmarking Large Language Model Capabilities in the Information-Gathering Stage of Deep Research
Derived from paper: Let's Use ChatGPT To Write Our Paper! Benchmarking LLMs To Write the Introduction of a Research Paper
Derived from paper: Can LLMs Generate Novel Research Ideas? A Large Scale Human Study with 100+ NLP Researchers
Hugging Face dataset: alexshengzhili/ai-scientist-blog-assets
Corral – Rise of AI Scientists Bibliometric evidence for the rise of AI scientists in chemistry and materials science relative to general AI for chemistry 📋 Dataset Summary This dataset is part of the Corral collection accompanying the paper AI scientists produce results without reasoning scientifically. It contains the bibliometric evidence for the growing relevance and impact of AI scientists in chemistry and materials science compared to general AI… See the full description on the dataset page: https://huggingface.co/datasets/jablonkagroup/rise_ai_scientists.
AI Scientist trajectories Private Poke & Wiggle agent trajectories. Each run contains the original main JSONL and all subagent JSONLs in a separate directory. Credential values are redacted. Study Run Main agent Subagents ActionMap: scarce data cff59152 Trace Traces ActionMap: full data ac77530f Trace Traces DSD 4c6d8d5a Trace Traces FACT 0a082af0 Trace Traces ManiFlow c2e8e274 Trace Traces Patch Policy vs OpenVLA-OFT 8ca0c7b0 Trace Traces Patch Policy:… See the full description on the dataset page: https://huggingface.co/datasets/pokeandwiggle/ai-scientist-trajectories.
Dataset Card for "llama2_7b-arc_hard" More Information needed
Dataset Card for "llama2_7b-arc_hard-results_playing" More Information needed
Dataset Card for "boolq" More Information needed
Dataset Card for "gpt2-winogrande_base" More Information needed
Dataset Card for "gpt2-winogrande_inverted_option" More Information needed
Hugging Face dataset: automated-research-group/llama2_7b_chat-agieval-results
Hugging Face dataset: automated-research-group/llama2_7b_chat-arc_challenge-results
Hugging Face dataset: automated-research-group/llama2_7b_chat-arc_easy-results
Dataset Card for "llama2_7b_chat-boolq" More Information needed
Dataset Card for "llama2_7b_chat-boolq-results" More Information needed
Dataset Card for "llama2_7b_chat-boolq-results_jacksee" More Information needed
Hugging Face dataset: automated-research-group/llama2_7b_chat-commonsense_qa-results
Hugging Face dataset: automated-research-group/llama2_7b_chat-hellaswag_0_label
Hugging Face dataset: automated-research-group/llama2_7b_chat-hellaswag_0_label-results
Hugging Face dataset: automated-research-group/llama2_7b_chat-hellaswag-results
Hugging Face dataset: automated-research-group/llama2_7b_chat-openbookqa-results
Hugging Face dataset: automated-research-group/llama2_7b_chat-piqa-results
Hugging Face dataset: automated-research-group/llama2_7b_chat-siqa
Hugging Face dataset: automated-research-group/llama2_7b_chat-siqa-results
Hugging Face dataset: automated-research-group/llama2_7b_chat-winogrande-results
Dataset Card for "phi-boolq-results" More Information needed
Dataset Card for "phi-boolq-results_playing" More Information needed
Dataset Card for "phi-winogrande_base" More Information needed
Dataset Card for "phi-winogrande_inverted_option" More Information needed
Dataset Card for "phi-winogrande_inverted_option-results" More Information needed
Hugging Face dataset: automated-research-group/winogrande
Dataset Card for "winogrande_inverted_option" More Information needed