Aggregate analysis

Research Analysis

Cross-paper synthesis of shared research patterns, differences, mainstream directions, and trend evolution.

341 papers

AI Scientist Systems

End-to-end autonomous research agents that combine ideation, experimentation, analysis, and manuscript generation.

55 Methodology quality 93 Recency 42 Reproducibility 39 Topical relevance

Literature review synthesis

Research Lines

Fully automated discovery pipelines

Chains idea generation, code implementation, experiment execution, analysis, and paper writing into a single automated workflow, reducing human effort per research cycle.

Open edge: Output quality is inconsistent; only a fraction of generated papers pass peer review, and the pipeline often fails outside the domain it was tuned for.

Verification and self-correction mechanisms

Detects and corrects hallucinated results or silent failures through internal logic checks, physics-based visual inspection, or verifiable report synthesis, increasing trustworthiness of generated claims.

Open edge: Verification coverage is incomplete—e.g., a vision-language gate missed 2 of 16 planted failures—and the verifier itself can be fooled by plausible but incorrect outputs.

Discipline-specific adaptation

Tailors the generic AI scientist to a particular domain (CFD, applied mathematics, biology) by embedding domain knowledge, physical constraints, or specialized toolchains, enabling defensible results in those fields.

Open edge: Domain knowledge injection still relies on human curation; open-ended discovery of new physical principles remains an unsolved challenge.

Benchmarking and evaluation frameworks

Provides standardized tasks, metrics, and human baselines (e.g., DISCOVERYWORLD, BAISBench) to compare AI scientist systems quantitatively and expose their weaknesses.

Open edge: Benchmark tasks simplify real scientific complexity; proxy metrics like task completion do not yet align with genuine scientific insight or long-term impact.

Shared Direction

All systems use LLMs as the central reasoning engine for scientific workflows.
Every system includes some form of result validation or evaluation to counteract hallucination.
Current AI scientist performance is insufficient for fully reliable autonomous discovery; human involvement or verification remains necessary.
End-to-end paper generation is a common end goal, but the quality of generated manuscripts varies widely.

Key Differences

Verification strategy: logic-based self-audit (ReasFlow) vs. physics-aware vision gate (AI CFD Scientist) vs. verifiable report synthesis (AutoResearchClaw).
Evaluation target: automated reviewer scores (The AI Scientist series) vs. standardized benchmark tasks (DISCOVERYWORLD, BAISBench) vs. real peer review acceptance (AI Scientist-v2).
Domain generality: build a generic framework and then apply it (The AI Scientist, AutoResearchClaw) vs. construct a domain-specific system from the start (AI CFD Scientist, ReasFlow).
Human collaboration model: PI–student metaphor with high-level guidance (ReasFlow) vs. multi-mode intervention with seven granularity levels (AutoResearchClaw).

Open Questions

Can verification gates be designed to catch all critical failure modes without blocking novel but unconventional valid discoveries?
How well do benchmark scores and automated reviewer metrics predict the actual scientific value or reproducibility of a generated study?
What is the minimal set of domain-specific components needed to make a generic AI scientist competitive with a purpose-built one in a new field?
At which stages of the research cycle does human intervention provide the largest marginal benefit over full autonomy?

AILGCLmtrl-sciMACYDLIR

284 papers

Evaluation, Reliability & Governance

Evaluation, reproducibility, safety, epistemic reliability, governance, and benchmark infrastructure for automated research.

55 Methodology quality 92 Recency 42 Reproducibility 37 Topical relevance

Literature review synthesis

Research Lines

End-to-end autonomous research system infrastructure

Integrating LLMs into agents that execute full research cycles (idea generation, code implementation, experiment execution, paper writing) to reduce manual effort.

Open edge: Cross-domain generalization, long-horizon memory and context fidelity, and the reproducibility of agent-generated experiments under varying conditions.

Benchmarking and evaluation design for scientific discovery agents

Providing standardized tasks, difficulty levels, and automatic or human evaluation metrics to enable comparable performance assessment of autonomous research agents.

Open edge: The alignment between proxy metrics (task completion, reviewer scores) and genuine scientific impact; the coverage and diversity of benchmark tasks relative to real-world research.

Failure attribution and reliability improvement

Identifying recurring failure modes in autonomous research pipelines and applying structured diagnosis or verification gates to increase the trustworthiness of outputs.

Open edge: Causal validation of attribution mechanisms; residual false-negative rates in verification gates (e.g., 2/16 silent failures missed); applicability to complex physical or biological domains.

Shared Direction

Current LLM-based agents can partially automate research workflows but remain unstable for fully autonomous end-to-end scientific discovery.
External verification gates such as automated review or physics-based checks are necessary to catch obvious errors, yet they still miss certain failure cases.
Standardized benchmarks are crucial for measuring progress, but existing ones cover limited domains and do not yet guarantee alignment with real scientific value.
Self-correction and multi-agent collaboration are promising directions for improving robustness and output quality.

Key Differences

Evaluation methodology: some works rely on automatic reviewer scores, others employ human expert judgment or real peer review, leading to different standards for what constitutes acceptable research quality.
Domain specificity: general-purpose ML research systems contrast with domain-embedded systems (e.g., CFD, omics) that incorporate physics or biological priors, making their failure modes and generalization claims incomparable without careful normalization.
Correction depth: the spectrum ranges from simple reflection loops to multi-hypothesis causal failure attribution, with differing granularity and interpretability of the recovery process.
Interaction mode: fully autonomous operation versus interactive dashboards that allow human intervention, creating a divergence in how “autonomy” is defined and evaluated.

Open Questions

What unified cross-domain benchmark can measure autonomous research capability while predicting real-world scientific utility?
How can the reliability of verification gates be quantified and their false-negative rate reduced below operationally meaningful thresholds?
What are the failure boundaries of autonomous systems when faced with out-of-distribution tasks or data, and how can those boundaries be detected online?
How can long-horizon research agents maintain memory and context coherence to avoid cascading errors?
What is the causal effect of multi-agent role design and communication protocols on the final scientific quality of the output?

AICLLGMAmtrl-sciIRDLCY

252 papers

Literature Intelligence

Automated literature search, paper reading, citation analysis, literature review, survey synthesis, and research gap discovery.

51 Methodology quality 91 Recency 41 Reproducibility 35 Topical relevance

Literature review synthesis

Research Lines

Scientific knowledge representation and retrieval for literature synthesis

Turning unstructured paper collections into structured, retrievable knowledge that guides generation.

Open edge: Whether retrieved knowledge is sufficient for novel insight beyond training data, and how to handle contradictions across sources.

Agentic planning and iterative refinement for automated literature review

Producing high-quality, coherent survey papers by orchestrating multiple specialized agents with feedback loops.

Open edge: The extent to which planning mechanisms can improve scientific depth versus surface-level formatting, and scalability to long-horizon tasks without memory degradation.

Benchmarking and evaluation design for literature intelligence systems

Providing standardized tasks and metrics (cell annotation, multiple-choice questions, rubric scores) to compare AI scientist capabilities.

Open edge: Coverage of diverse scientific literature tasks beyond narrow domains, and the gap between proxy metrics and genuine scientific contribution.

End-to-end agentic systems for literature synthesis and survey generation

Turning isolated model ideas into runnable, reproducible pipelines that can autonomously generate literature.

Open edge: Standardized validation across different scientific fields and ensuring reproducibility of agent outputs given non-deterministic LLM behaviors.

Shared Direction

Multi-agent LLM systems can automate parts of the literature review process, but fully autonomous literature intelligence remains unsolved.
Iterative refinement (via rubrics, verification, or evolutionary steps) is essential for improving output quality.
Human evaluation or expert-designed benchmarks are necessary to assess output quality.

Key Differences

Some works focus on generating full survey papers with rubric-based quality assessment, while others integrate literature synthesis as a sub-step in broader scientific discovery workflows.
Evaluation targets differ: domain-specific scientific discovery tasks (e.g., cell annotation) versus general survey quality judged by LLM rubrics.
Verification strategies vary: internal logical verification loops versus external reviewer agents scoring against rubrics.
Domain scope ranges from domain-agnostic frameworks to systems tailored to applied mathematics or omics.

Open Questions

How to verify the factual accuracy of AI-generated literature reviews, especially when handling contradictory or outdated sources.
Whether LLM-based evaluation rubrics can replace human peer review for assessing scientific quality.
The scalability of these agentic systems to very large literature corpora and long-horizon research projects without memory degradation.
The ethical implications of fully automated scientific literature generation, including plagiarism and accountability.

AICLLGIRDLMAmtrl-sciCY

251 papers

Writing & Communication

Automatic paper writing, manuscript drafting, scientific reporting, peer review assistance, and research communication.

45 Methodology quality 92 Recency 42 Reproducibility 34 Topical relevance

Literature review synthesis

Research Lines

End-to-end autonomous research and writing systems

Automates the full scientific pipeline—hypothesis generation, code execution, result visualization, and manuscript drafting—to reduce human effort in routine research and paper writing.

Open edge: Reliability across diverse domains, consistent factual accuracy, and the ability to produce manuscripts that meet rigorous human peer-review standards without extensive human intervention.

Evaluation and quality assessment of generated papers

Provides metrics, automated reviewers, and benchmarks (e.g., LLM-based rubrics, blind scoring, workshop acceptance thresholds) to measure the quality and completeness of AI-generated scientific writing.

Open edge: Alignment between automated metrics and true scientific contribution; generalizability of evaluation methods beyond the specific ML subfields or curated benchmarks used in development.

Failure analysis and self-correction mechanisms

Identifies and classifies failure modes (e.g., implementation drift, hallucinated numbers, memory degradation) and designs self-correction strategies such as multi-hypothesis attribution or internal verification loops to improve research robustness.

Open edge: Whether these mechanisms can prevent all critical failure types in open-ended scientific exploration; how to handle complex, multi-step failures that involve intertwined logical and experimental errors.

Human-AI collaboration modalities

Explores different intervention granularities (from full autonomy to step-by-step oversight) to leverage human expertise at high-leverage decision points while maintaining automation efficiency.

Open edge: Optimal interaction design that balances cognitive load, scalability, and writing quality; empirical validation of which collaboration mode benefits which type of scientific writing task.

Shared Direction

Large language models can generate coherent research papers when combined with code execution environments and automated review loops.
Automated verification—whether through logic checks, physical simulations, or fact grounding—is essential to reduce hallucinated or fabricated results in generated manuscripts.
Current systems still frequently fail in long-horizon tasks, requiring either human intervention or iterative self-correction to produce defensible scientific output.
Workshop-level acceptance is emerging as a meaningful milestone, but conference-ready autonomous paper generation remains an unsolved challenge.

Key Differences

The appropriate degree of autonomy: some works advocate fully autonomous pipelines (AI Scientist line), while others demonstrate that targeted human collaboration at key decision points outperforms both full autonomy and exhaustive oversight (AutoResearchClaw).
The nature of verification: approaches range from internal logical coherence auditing (ReasFlow) and physics-based vision gates (AI CFD Scientist) to fact-grounded report generation that constrains claims to measured values (SAGE).
The role of failure handling: some systems use a single-stage reflection loop, whereas SAGE proposes a multi-hypothesis causal attribution that deterministically routes corrections to the correct pipeline stage, implying a divergence in assumed failure complexity.
Domain specificity: general machine learning frameworks (AI Scientist) assume broad applicability, while domain-specific agents (ReasFlow for applied math, AI CFD Scientist for fluid dynamics) argue that deep verification requires domain-aware gates and knowledge retrieval.

Open Questions

What verification methods can guarantee that an AI-generated paper contains no fabricated experimental data or hallucinated citations in open-ended scientific tasks?
How can we build evaluation benchmarks that measure the true scientific contribution of an AI-written paper, beyond surface-level writing quality and metric-bearing output?
What is the upper bound on self-correction when the underlying model lacks domain intelligence or scientific taste, and how can human feedback be integrated without becoming a bottleneck?
Can the design principles derived from failure analyses (e.g., Why LLMs Aren't Scientists Yet) be systematically translated into architectural constraints that prevent the most common failure modes?

AILGCLMAmtrl-sciDLCYIR

220 papers

Hypothesis & Idea Generation

Systems that propose hypotheses, research ideas, experimental directions, or scientific questions.

54 Methodology quality 92 Recency 41 Reproducibility 37 Topical relevance

Literature review synthesis

Research Lines

Action-conditioned control and planning

Connects hypothesis generation, experiment design, and execution in a closed loop, enabling agents to plan interventions, react to experimental outcomes, and improve decision quality over cycles.

Open edge: Whether the improvement stems from the agent's planning logic rather than the underlying LLM or task-specific priors, and how well the loop handles distributional shifts outside the training domain.

Evaluation and evidence design

Provides standardized benchmarks (e.g., ARC-Bench, BAISBench) with tasks, metrics, and human baselines to make claims about AI scientist capabilities comparable and to surface failure modes.

Open edge: Coverage of diverse scientific domains and alignment between proxy metrics (benchmark scores) and real-world scientific discovery utility remain limited.

Systems and reproducible infrastructure

Turns isolated research agents into runnable, customizable platforms (e.g., multi-agent laboratories, code harnesses) with artifacts, monitoring, and reusable workflows, lowering the barrier to build and reproduce AI scientist experiments.

Open edge: Standardized external validation across hardware, datasets, and user settings is missing; most platforms are evaluated only on internal case studies.

Conceptual taxonomy and theory

Organizes fragmented AI scientist systems into categories (literature review, hypothesis generation, experimentation, analysis) and identifies common challenges, metrics, and frameworks, enabling a structured understanding of the field.

Open edge: The proposed taxonomies lack direct empirical validation that the categories predict agent behavior, and the field still lacks a unified theoretical framework for autonomous scientific reasoning.

Shared Direction

Reliable scientific discovery requires structured verification beyond raw LLM outputs; all systems incorporate some form of validity gate or verification step.
Multi-agent or modular architectures are the dominant design pattern for decomposing the research workflow into manageable sub-tasks (e.g., ideation, execution, analysis).
Current AI scientist agents are not yet capable of fully autonomous, end-to-end conference-ready paper generation; human intervention or collaboration remains necessary for quality and correctness.

Key Differences

The role of formal verification: some systems rely on machine-checked proofs and statement audits, while others depend on empirical validation or vision-language gates, creating a divide between formal and data-driven verification.
Human collaboration mode: approaches range from fully autonomous execution to targeted human-in-the-loop at high-leverage decision points, with no consensus on the optimal level or timing of human involvement.
Domain specificity: some works are domain-agnostic foundational frameworks, while others are tightly integrated with domain-specific tools and physics models, raising a tension between generality and depth.

Open Questions

How can we close the gap between formal or empirical verification and the actual scientific value of generated hypotheses?
To what extent do improvements in benchmark scores translate to more effective AI scientists in real-world labs?
What are the failure modes of self-evolving agents when they operate outside the distribution of tasks seen during their design and evaluation?

AICLLGMAIRmtrl-sciDLCE

158 papers

Experiment Automation

Agents that design, execute, monitor, analyze, or iterate experiments, including code-based and lab-based experimentation.

61 Methodology quality 92 Recency 42 Reproducibility 43 Topical relevance

Literature review synthesis

Research Lines

End-to-end autonomous research agent systems

Automating the complete scientific discovery pipeline from idea generation through experiment execution to manuscript writing, reducing human effort per discovery cycle.

Open edge: Reliability under rigorous peer review (only one workshop paper accepted), generalization beyond ML or applied math, and handling of long-horizon safety and ethical review.

Experiment planning and execution optimization

Improving the efficiency of individual experiment steps by learning from past failures or by planning ablations systematically, aiming to reduce wasted computation and token usage.

Open edge: LMs still identify only 45% of human-expected ablations; negative knowledge is shown only on PDE and ScienceAgentBench tasks, with unclear benefit on broader experiment types.

Benchmarks for scientific discovery agents

Providing standardized tasks and metrics to compare agent capabilities in hypothesis testing, experiment design, and discovery, thus highlighting current limitations.

Open edge: Coverage of diverse scientific practices beyond text-based or simulated environments; agreement between automatic proxy metrics and real-world scientific utility.

Multi-agent collaborative research platforms

Enabling customizable role-based collaboration among agents to handle complex research workflows, with human oversight via dashboards and artifact inspection.

Open edge: Lack of systematic validation of the added value of multi-agent coordination over single-agent baselines; platform maturity and reproducibility across different hardware and user settings.

Shared Direction

LLM-based reasoning and code generation are the central engines for automating scientific experiments.
Decomposing the research loop into hypothesis, experiment, and analysis steps is a common architectural pattern.
Automated evaluation (simulated reviewers or benchmark tasks) is necessary for scalable assessment, but its correlation with human scientific judgment remains uncertain.
Current systems are most effective in domains where the experimental feedback loop can be fully simulated or executed in code (ML, math).

Key Differences

The level of autonomy: some systems aim for fully automated paper generation with no human intervention, while others position the human as a principal investigator overseeing agent execution.
Evaluation philosophy: one line uses simulated peer review (LLM judges) to gate system output, while another uses task completion metrics in synthetic environments, and a third submits to real workshops, creating tension between proxy and real-world validity.
Failure handling: negative knowledge memory banks are contrasted with tree-search-based iterative refinement and with simple retry mechanisms, affecting memory footprint and exploration strategy.
Domain specificity: systems like ReasFlow are designed for theory-heavy mathematical reasoning, whereas AI Scientist targets empirical ML; this specialization impacts the required verification and grounding components.

Open Questions

How can autonomous agents handle domains where experiments are costly, physical, or safety-critical, beyond purely simulated or code-based environments?
What is the minimal set of human feedback signals needed to keep an autonomous research agent aligned with scientific rigor over long discovery trajectories?
Can negative knowledge or similar failure-memory approaches generalize to capture common experimental pitfalls across different scientific fields?
How should the community standardize evaluation so that improvements in benchmark scores translate to higher-quality peer-reviewed scientific output?

AILGCLmtrl-sciMACYCEIR

40 papers

Survey Generation

Papers and resources related to Survey Generation.

52 Methodology quality 88 Recency 43 Reproducibility 34 Topical relevance

Literature review synthesis

Research Lines

Automated survey generation pipelines

End-to-end synthesis of a literature survey from paper retrieval through drafting and revision, using iterative refinement and multi-agent orchestration to improve coherence, coverage, and citation accuracy.

Open edge: Generalization across diverse disciplines without task-specific tuning; scalability to very large paper sets without loss of citation recall; and alignment of pipeline-generated quality with expert human judgment in real-world scholarly workflows.

Evaluation benchmarks and metrics for survey quality

Providing standardized testbeds and metrics (e.g., TaxoBench, Survey-Arena, 12-dimensional evaluation) to quantitatively compare automated survey generation systems on retrieval, organization, content depth, and citation fidelity.

Open edge: The gap between proxy metrics and true scholarly utility; the reliance on a single expert taxonomy as reference risks penalizing valid alternative organizations; and limited coverage of non-CS or non-English survey tasks.

Taxonomy and knowledge organization

Constructing hierarchical or flat taxonomies of research topics from reference papers to structure a survey, enabling clear paper-to-category mapping and topical navigation.

Open edge: Whether automatically generated taxonomies are preferable to human-curated ones in terms of utility for readers; verifying that taxonomy-based organization improves downstream comprehension rather than just structural metrics.

Shared Direction

LLM-based agents are the core engine for survey generation, and simple single-pass generation is insufficient; iterative refinement is necessary.
Citation accuracy and content coverage are the primary quality dimensions, and systems must incorporate explicit mechanisms to verify claims against source papers.
Structural coherence and logical organization of the survey are as important as factual correctness, leading to outline planning and taxonomy-guided sectioning.

Key Differences

Systems adopt either a single multi-stage pipeline (e.g., AutoSurvey2) or a multi-agent collaboration (e.g., ARISE, Agentic AutoSurvey, DeepSurvey), with no consensus on which paradigm yields better reliability–cost trade-offs.
Evaluation approaches diverge: some rely on custom benchmarks with expert annotations (TaxoBench, CS-TaxoBench), while others build ad-hoc human evaluations or use LLM-as-judge; there is no standardized external benchmark accepted across all works.
The depth of source analysis varies: most systems operate on abstracts and metadata, but DeepSurvey pushes full-text and code-repository analysis, raising open questions about the necessary and sufficient evidence for high-quality synthesis.
The role of human oversight differs: NVAITC AI Scientist emphasizes governance and scientist-in-the-loop, whereas many other systems target fully autonomous generation, leaving the boundary between assistance and automation unsettled.

Open Questions

How can we verify that a citation in a generated survey truly supports the attributed claim, beyond checking keyword overlap?
Do automated surveys improve researcher efficiency or comprehension in real-world workflows, and what experiment would demonstrate this?
What are the failure modes of multi-agent iterative refinement when applied to rapidly evolving preprint literature with incomplete or contradictory claims?
Can taxonomy-based evaluation metrics be designed to be reference-free yet still aligned with human preferences for survey organization?

CLAIIRDLHCLG

37 papers

Scientific Discovery Agents

Papers and resources related to Scientific Discovery Agents.

30 Methodology quality 92 Recency 48 Reproducibility 22 Topical relevance

Literature review synthesis

Research Lines

Science discovery environments and benchmarks

Provides standardized tasks, interaction protocols, and metrics to evaluate AI agents on end-to-end scientific discovery, from hypothesis formation to experiment and conclusion.

Open edge: Task coverage is limited to certain domains; LLM-based judges show only moderate agreement with human experts; outcome metrics cannot detect incorrect reasoning that yields correct results.

Agent infrastructure and tool augmentation

Automates parts of the discovery pipeline—code generation, wet-lab execution, training, safety checks—to improve agent efficiency, reproducibility, and task completion rate.

Open edge: Generalization across scientific domains is unverified; most systems are evaluated in isolation or with simulated tasks, not in real-world lab settings; long-term autonomy and reliability remain open.

Faithful reasoning and safety mechanisms

Integrates explicit checks, structured certificates, or safety loops to ensure agents' reasoning is consistent with evidence and free from dangerous actions or tool-chain escapes.

Open edge: Current safety reasoning is evaluated on limited tasks; detecting all compositional risks without stifling exploration is hard; mechanism-fidelity checks often require known ground-truth, limiting use in open-ended discovery.

Shared Direction

All works adopt a multi-stage experiment loop (hypothesis, experiment, analysis) as the agent's decision cycle.
Current LLM-based agents are insufficient for robust scientific discovery and require additional scaffolding, such as tools, memory, or safety modules.
Evaluation should go beyond task completion and include process quality, safety, or reasoning faithfulness.

Key Differences

Environment type: some use fully text-based virtual labs, others rely on code execution with real software constraints, and a few simulate physical wet-lab operations.
Safety integration: one approach embeds safety reasoning into every agent reasoning step, while another applies external safety gates after trajectory completion.
Question formation: one line advocates structured, auditable research question certificates; others keep question generation as a free-form prompt step without explicit traceability.
Evaluation judges: one study reports moderate agreement between LLM-as-a-judge and domain experts, while others use LLM judges without human calibration, raising concerns about proxy reliability.

Open Questions

How can we build a single agent infrastructure that generalizes across diverse scientific fields without per-domain retuning?
What automatic methods can detect 'correct answer, wrong mechanism' failures when the ground-truth mechanism is unknown?
How can safety mechanisms be designed to prevent compositional tool-chain risks without over-restricting the agent's ability to explore novel hypotheses?
What is the minimal set of reality-grounded tasks and metrics needed to validate that a simulated discovery agent will succeed in a real laboratory?

AILGCLcomp-phmtrl-scichem-ph

34 papers

Hypothesis Generation

Papers and resources related to Hypothesis Generation.

66 Methodology quality 94 Recency 39 Reproducibility 47 Topical relevance

Literature review synthesis

Research Lines

Multi-agent hypothesis generation workflows

Automating the full scientific discovery cycle by distributing tasks—hypothesis formation, data analysis, literature search, and experimental design—across specialized agents that iteratively refine candidate hypotheses with feedback.

Open edge: Ensuring long-horizon hypothesis quality without error propagation, preventing hallucinated claims, and verifying that the final output meets scientific rigor beyond surface-level accuracy metrics.

Structured search and evolutionary hypothesis generation

Exploring large hypothesis spaces efficiently by combining tree-based backtracking, evolutionary pressure, and code-level optimization so that the system can discover equations, algorithms, or novel mechanisms without exhaustive enumeration.

Open edge: Balancing exploration breadth with novelty, defining reliable novelty scores, and demonstrating that the proposed hypotheses generalize beyond the training distribution of the search algorithm.

Human-in-the-loop hypothesis co-creation

Mitigating the brittleness of fully autonomous systems by allowing human researchers to intervene at high-leverage decision points, preserving scientific intuition while reducing exhaustive oversight.

Open edge: Optimal placement of human intervention, measuring the cognitive load on the researcher, and determining whether the collaboration scales to multiple domains without retraining the interaction policy.

Benchmarking and evaluation frameworks for hypothesis generation

Providing standardized tasks and metrics (e.g., ARC-Bench, LABBench2) to compare hypothesis quality, execution correctness, and end-to-end discovery performance across AI scientist systems.

Open edge: Alignment between proxy metrics and real-world publication- or patent-level impact, coverage of diverse scientific disciplines, and the construction of negative controls that robustly reject weak hypotheses.

Shared Direction

All systems treat large language models as the core reasoning engine for hypothesis generation and refinement.
Discovery is structured as an iterative closed loop: hypothesize, execute, analyze, and update, mirroring the scientific method.
Multi-agent or modular decomposition is preferred over monolithic architectures to handle the complexity of research tasks.
Reproducibility, evidence traceability, and the risk of fabricated claims are recognized as central challenges.
Evaluation is moving toward realistic, multi-step benchmarks that test integrated capabilities rather than isolated component accuracy.

Key Differences

Hypothesis generation mechanisms differ: some employ structured multi-agent debate, others use evolutionary search with backtracking, and a few rely on programmatic optimization with reinforcement learning.
Human involvement ranges from fully autonomous operation to seven distinct intervention modes; there is no agreement on the optimal level or timing of human input.
Evaluation targets are domain-specific: biology, chemistry, materials science, combinatorial optimization, and equation discovery, each with incompatible success criteria.
Architectural granularity varies: some systems share a structured world model across agents, while others keep agents independent with minimal shared state.
Assessment of scientific validity is split between automated metrics and human expert judgment, with no consensus on how to weight accuracy, novelty, and reproducibility.

Open Questions

Can generated hypotheses maintain 100% factual accuracy over long autonomy spans, or will error compounding always require human correction?
How should we design a unified evaluation protocol that spans biology, chemistry, and algorithm discovery while capturing real-world scientific utility?
What is the minimal set of human interventions needed to guarantee reliability without sacrificing the speed of autonomous discovery?
Do current taxonomies of autonomous research workflows have predictive power over system failure modes, and can they guide the design of safer, more robust agents?

AICLLGGNMADLCENE

25 papers

Research Agent Benchmarks

Papers and resources related to Research Agent Benchmarks.

8 Methodology quality 92 Recency 41 Reproducibility 6 Topical relevance

Literature review synthesis

Research Lines

Action-conditioned control and planning

Enables LLM agents to execute multi-step research actions—querying literature, generating hypotheses, running experiments, and self-correcting—through closed-loop decision-making and feedback.

Open edge: Robustness of agent decisions in open-ended, long-horizon tasks; whether improvements come from better planning, larger models, or domain-specific heuristics; and how to prevent error cascades.

Evaluation and evidence design

Standardizes comparison of research agents through curated benchmarks, metrics (e.g., accuracy, MAE improvement), and controlled task designs (coding problems, survey generation, held-out validation).

Open edge: Coverage of diverse scientific domains, alignment between proxy metrics and real-world research utility, and the lack of consensus on what constitutes a valid test of autonomous research capability.

Systems and reproducible infrastructure

Packages agent workflows into auditable, executable pipelines that produce reusable code and artifacts, enabling verification across different environments.

Open edge: Standardized external validation across hardware, data distributions, and user settings; most systems lack comprehensive reproducibility artifacts beyond the reported experiments.

Shared Direction

LLMs serve as the core reasoning and coding engine for research agents.
The research process is broken into discrete steps (search, plan, code, verify) with feedback signals to guide the agent.
Evaluation should go beyond toy tasks and move toward real scientific problem-solving, though implementations differ.
Auditability and reproducibility are emerging as desired properties of research agent systems.

Key Differences

Evaluation approach: static benchmarks with pre-defined metrics versus dynamic held-out task transfer that measures generalization to unseen data.
Domain scope: some works target specific scientific fields (materials, mathematics), while others aim for general-purpose research agents.
Agent architecture: recursive self-improvement via reinforcement learning against predefined search space decomposition with inner-fold validation.
Reproducibility granularity: some systems provide full code and auditable logs, while others only describe the methodology without released artifacts.

Open Questions

Do current benchmarks (e.g., academic survey tasks, coding problems) genuinely measure progress toward autonomous research, or do they reward narrow capabilities?
How can agent decision quality be validated in open-ended research where ground truth is unknown or evolves over time?
What is the minimum set of evidence (code, data, logs) required to trust a research agent's findings, and how can this be standardized across domains?

23 papers

Automated Experimentation

Papers and resources related to Automated Experimentation.

42 Methodology quality 86 Recency 40 Reproducibility 28 Topical relevance

Literature review synthesis

Research Lines

End-to-end autonomous research labs

Automating the complete research pipeline, from ideation and literature analysis to experimental design, execution, and manuscript drafting, reducing human involvement.

Open edge: Lack of implemented prototypes with empirical validation; dependence on community consensus and resource investment; generalization beyond recommender systems or specific molecular dynamics tasks.

Digital-twin-driven autonomous decision-making

Predicting experimental outcomes, uncertainties, and risks to guide autonomous operations in microscopy and self-driving labs, enabling open decision-making and collaborative optimization.

Open edge: Validation limited to specific microscopy modalities or simple color-mixing tasks; noise dynamics and missing physical mechanisms still cause prediction residuals; scalability to complex scientific problems not demonstrated.

LLM-based reasoning and multi-agent workflows

Using LLM agents to propose hypotheses, edit code, assess novelty, and reason about experimental data, operationalizing serendipity and improving sampling efficiency.

Open edge: Trustworthiness of LLM reasoning and physical admissibility under distribution shift; overhead of multi-agent coordination; lack of standardized benchmarks for scientific reasoning.

Data quality and experimental infrastructure

Ensuring data reliability and reproducibility in self-driving labs through noise detection and correction (kNN) and FAIR data pipelines with automatic indexing.

Open edge: Methods are sensitive to feature distribution and noise type; not validated across diverse instrument modalities; reliance on specific infrastructure (nanoHUB) limits adoption.

Shared Direction

Integrating physical constraints or domain knowledge is essential for reliable autonomous experimentation.
LLMs and foundation models are becoming central components in automating scientific reasoning and experiment planning.
Data quality, FAIR principles, and reproducibility are critical bottlenecks for deploying self-driving labs in practice.
Standardized benchmarks and evaluation protocols are needed to compare different autonomous experimentation approaches.

Key Differences

Scope of automation: some works target full end-to-end autonomy, while others keep a human-in-the-loop for critical decisions or serendipity.
Decision mechanism: explicit world models (digital twins) vs. implicit LLM reasoning without explicit state representation.
Application domain: materials science imposes physical constraints and multi-modal data, whereas recommender systems face fairness and governance challenges.
Role of human guidance: SciLink actively integrates real-time expert feedback, whereas MLIPilot and AutoRecLab aim to minimize human intervention.

Open Questions

How can a unified evaluation framework balance scientific validity, computational cost, and reproducibility across diverse autonomous experimentation systems?
Are LLM reasoning trajectories reliable when encountering novel physical phenomena outside the training distribution?
What mechanisms ensure that autonomous labs produce repeatable and verifiable scientific results?
How should data ownership, privacy, and governance be managed in distributed collaborative self-driving laboratories?
Can noise detection and imputation methods generalize to highly heterogeneous and out-of-distribution experimental data?

LGAImtrl-sciCLdata-anchem-phCEIR

17 papers

Literature Review Agents

Papers and resources related to Literature Review Agents.

9 Methodology quality 87 Recency 37 Reproducibility 10 Topical relevance

Literature review synthesis

Research Lines

Retrieval-augmented knowledge layers for agent access

Upgrades literature databases into AI-agent-accessible knowledge layers that can decompose natural language queries, execute live searches, and extract evidence, improving retrieval accuracy and downstream task performance.

Open edge: Generalization beyond the curated search engine (e.g., Europe PMC) and the brittleness of relying on a single LLM orchestrator; no evidence on how the system behaves with ambiguous or cross-domain queries.

Multi-agent literature review generation

Automates the production of structured literature reviews by emulating a human workflow with specialized agents for outlining, writing, editing, and reviewing, thereby improving readability, citation quality, and overall coherence.

Open edge: Factual accuracy and depth compared to expert-written reviews; the evaluation relies on surface-level similarity and writing quality metrics, not on downstream utility or expert verification of the synthesized claims.

Automated peer review and robustness analysis

Studies the behavior and vulnerability of LLM agents in peer review, revealing that surface-level revisions can manipulate review outcomes and that adversarial attacks can break reviewer models.

Open edge: Building defenses against such manipulations, ensuring fairness and consistency of automated reviewers, and validating whether simulated review dynamics reflect real human review processes.

Iterative and hierarchical literature organization

Provides structured or iterative organization of scientific papers to support researchers in navigating and synthesizing large bodies of literature, often through hierarchical clustering or iterative retrieval loops.

Open edge: Lack of detailed methodology and evaluation evidence in the available data; the assumption that better organization directly translates to higher-quality literature reviews remains untested.

Shared Direction

LLM-based agents are a viable backbone for automating literature review tasks, with retrieval and generation steps being orchestrated through multi-step or multi-agent workflows.
Factual accuracy and reliability remain major concerns across all systems, and current evaluation metrics only partially capture the quality of a generated review.
Human review behavior (e.g., iterative refinement, organization, and critical appraisal) serves as a design template for agent workflows.

Key Differences

The primary target of automation differs: some works aim to improve the retrieval interface for agents (information access), while others focus on generating the final review document (content creation) or on auditing the review process itself (meta-evaluation).
Evaluation practices diverge—citation F1, writing quality, similarity to human reviews, and robustness under attack are used in isolation, with no agreed-upon benchmark that covers all aspects.
Interaction mode varies: real-time live search versus offline batch generation, single-agent orchestration versus multi-agent collaboration, and human-in-the-loop versus fully autonomous operation.

Open Questions

How can we verify that a claim in an automatically generated review is supported by the cited evidence without relying on heuristics that are vulnerable to presentation-only manipulations?
What defenses can make automated review agents robust against adversarial papers, and how do we design such defenses without leaking review criteria?
Is the improvement in retrieval or generation quality observed on existing benchmarks retained when the system is deployed in a different scientific domain or with a different underlying search index?
What evaluation protocol would capture both the utility of a generated review for a human researcher and the factual trustworthiness of its claims, beyond writing style and citation overlap?

13 papers

Paper Writing Agents

Papers and resources related to Paper Writing Agents.

1 Methodology quality 90 Recency 47 Reproducibility 6 Topical relevance

Literature review synthesis

Research Lines

Automated paper drafting and generation

Producing a complete research paper from a high-level idea or outline, reducing manual writing effort.

Open edge: Quality, originality, and factual reliability of generated content; lack of standardized evaluation against human-written papers.

Interactive writing assistance and tutoring

Providing real-time feedback, editing suggestions, and writing guidance within an editor to improve draft quality.

Open edge: Adaptation to diverse writing styles and disciplines; measurable impact on final paper acceptance or readability.

Paper review and critique

Generating or evaluating critiques of scientific papers, aiming to support or automate peer review.

Open edge: Groundedness of critiques beyond surface claims; coverage of nuanced methodological flaws and integration into real review workflows.

Multimodal paper transformation

Converting paper content into other formats such as posters, slides, or visual summaries.

Open edge: Design quality, content selection accuracy, and user customization; evaluation beyond aesthetic appeal.

Shared Direction

All works leverage large language models as the core reasoning engine for paper-related tasks.
Multi-agent architectures are widely adopted to decompose complex writing or analysis workflows.
The research is concentrated in 2025–2026, indicating a nascent but rapidly growing area.
There is a shared emphasis on integrating tools into existing academic environments (e.g., Overleaf) or providing reproducible artifacts.

Key Differences

Task scope: some systems target end-to-end paper generation, while others focus on editing, review, poster creation, or experiment reproduction.
Autonomy level: fully automated generation versus human-in-the-loop tutoring or assistance.
Interaction mode: in-editor plugins, standalone frameworks, or benchmark suites with no direct user interaction.
Evaluation target: output text quality, critique groundedness, reproduction success rate, or search recall/precision.

Open Questions

How can we evaluate the novelty and scientific contribution of automatically generated papers beyond surface-level metrics?
What mechanisms ensure factual accuracy, proper citation, and avoidance of plagiarism in generated content?
How do these agents perform on papers requiring deep domain expertise or novel experimental designs?
What are the ethical implications and potential for misuse in academic publishing, and how can they be mitigated?