Aggregate analysis

Research Analysis

Cross-paper synthesis of shared research patterns, differences, mainstream directions, and trend evolution.

502 papers

AI Scientist Systems

End-to-end autonomous research agents that combine ideation, experimentation, analysis, and manuscript generation.

58 Methodology quality 95 Recency 41 Reproducibility 38 Topical relevance

Literature review synthesis

Research Lines

End-to-end autonomous discovery pipelines

Close the loop from hypothesis generation through code execution, experiment management, visualization, and manuscript writing without human-coded templates.

Open edge: Output is inconsistent under peer review; reported success is often workshop-level or rubric-level, not full conference-quality discovery, and limitations are frequently unspecified.

Domain-grounded verification and validity gates

Detect logical or physical silent failures before a claim is written by auditing reasoning or inspecting rendered domain artifacts.

Open edge: Gates still miss planted failures and have only been shown in selected domains; transfer to new reasoning domains or simulators is not established.

Evidence-oriented discovery benchmarks

Turn scientific discovery into comparable tasks with automatic metrics for task completion, action relevance, explicable knowledge, or evidence maturity.

Open edge: Current agents plateau near 60/100 and lack controls, robustness, falsifiability, and cross-dataset generalization; judge metrics are not calibrated to real discovery utility.

Collaborative multi-agent research workflows

Coordinate knowledge retrieval, derivation, experiment management, and manuscript drafting under either a human principal investigator or an experiment-manager agent.

Open edge: Human–agent contribution boundaries remain unclear and evaluations rely on LLM rubrics or single-domain case studies rather than reproducible multi-domain evidence.

Shared Direction

The discovery loop is commonly decomposed into ideation, experiment execution, analysis, paper writing, and automated review.
Automated evaluation or review is used as a scalable substitute for human assessment, but it still inherits LLM-judge limitations.
Current systems execute and document analyses competently, but they do not yet reliably produce trustworthy scientific claims with controls and robustness.
Verification or validity gates improve claim quality, yet published gate miss rates and benchmark plateaus show they remain partial.

Key Differences

Evaluation target: simulated virtual tasks versus blind evidence-from-data tasks versus domain-specific solver discovery versus conference-style peer review.
Verification mechanism: logical coherence auditing versus vision-language inspection of rendered flow fields versus a general automated reviewer without domain-specific validity gates.
Deployment setting: fully autonomous generation versus human PI–agent collaboration; some systems output manuscripts, others stop at analysis actions.
Domain and supervision: theory-driven applied mathematics, CFD physics, ML subfields, or multi-domain blind studies; supervision ranges from open-ended hypothesis search to frozen data from peer-reviewed studies.
Accepted evidence of success: rubric score, task completion, RMSE improvement on a particular simulation, or peer-review acceptance, making cross-system comparison difficult.

Open Questions

Can agents move from competent analysis execution to producing controls, robustness checks, falsification, and cross-dataset generalization?
How do logic-based and physics-based verification gates transfer to unseen domains or solvers when they already miss planted failures?
Are LLM-judge scores or workshop peer-review acceptance calibrated to real scientific contribution and reproducibility?
How does discovery-agent performance generalize across base models when the benchmark evidence uses only one frozen base model?
What is the actual human expert contribution in PI–agent workflows and how should autonomy be measured?
Which missing baselines, datasets, limitations, or code signals in the evidence packet create verification gaps rather than confirmed weaknesses?

AILGCLmtrl-sciMACYDLIR

436 papers

Evaluation, Reliability & Governance

Evaluation, reproducibility, safety, epistemic reliability, governance, and benchmark infrastructure for automated research.

59 Methodology quality 95 Recency 41 Reproducibility 36 Topical relevance

Literature review synthesis

Research Lines

End-to-end autonomous discovery workflow systems

Turns LLM-driven idea generation, code writing, experiment execution, visualization, paper writing, and simulated review into a continuous research pipeline.

Open edge: Workflow completeness does not guarantee defensible claims; peer-review pass rates remain inconsistent and verification gates are missing or domain-specific.

Blind and peer-reviewed-derived benchmark environments for discovery agents

Provides tasks from real or simulated scientific studies with hidden conclusions, fixed scorers, and task-completion or evidence-maturity metrics for repeatable agent evaluation.

Open edge: Task coverage and difficulty do not yet show that metric improvements correspond to real scientific utility; base-model generalization is not established.

Reliability and verification controls in agentic research systems

Adds physics-aware visual gates, deterministic program authority, or source-edit boundaries to detect silent failures and constrain autonomous modifications.

Open edge: Verification is imperfect and may depend on domain-specific priors; validation across hardware, datasets, and user settings remains a gap.

Human-aligned and review-simulated evaluation

Uses automated reviewers or graduate-level human baselines to calibrate generated paper scores and task performance.

Open edge: Automated and human-aligned scores may not capture controls, robustness, falsifiability, or long-term validity of scientific claims.

Shared Direction

Autonomous execution and documentation alone are not enough; the field increasingly requires evidence, verification, or review gates before claims are considered defensible.
Blind tasks drawn from peer-reviewed work or large virtual task suites help scale evaluation, but they mostly show that current agents plateau in scientific judgment.
Current agents can perform competent analyses and produce audit-friendly records, yet they systematically lack controls, robustness, falsifiability, and cross-dataset generalization.
Verification and reproducibility signals are treated as part of trustworthiness, although the way they are measured varies across systems.

Key Differences

Supervision differs: some rely on a fixed LLM evidence-maturity judge, others on deterministic program authority, vision-language physics gates, human baselines, or simulated conference review.
Deployment setting differs: text-based simulated discovery, computational fluid dynamics, single-cell transcriptomics, time-series forecasting, and broader machine learning domains shape different failure modes.
Evaluation target differs: one line targets complete open-ended discovery, another targets blind evidence claims, another targets task-specific architecture improvement, and another targets human-aligned annotation and discovery.
Success criterion differs: peer-review acceptance, benchmark score, RMSE improvement, implementation success and token-time cost, or human-relative performance.

Open Questions

Which verification or authority mechanism generalizes across base models and domains without overfitting to a narrow evaluation task?
How can proxy metrics such as documentation quality or task completion be validated against the epistemic quality of scientific claims?
Why do otherwise competent coding agents plateau at similar evidence-maturity scores and lack discriminating acts such as controls and robustness checks?
How should external validation be standardized across datasets, simulators, and user settings when artifacts and baselines are not yet fully reported?

AICLLGMAmtrl-sciIRDLCY

370 papers

Writing & Communication

Automatic paper writing, manuscript drafting, scientific reporting, peer review assistance, and research communication.

51 Methodology quality 95 Recency 40 Reproducibility 34 Topical relevance

Literature review synthesis

Research Lines

End-to-end autonomous research and manuscript generation

Covers idea generation, code/experiment execution, result visualization, and full paper drafting in a single agent pipeline.

Open edge: Cross-domain consistency and whether generated manuscripts are accepted beyond workshop or rubric-level evaluations.

Verification and grounded reporting for generated artifacts

Detects or prevents invalid results, hallucinated numbers/citations, and silent experimental failures before paper acceptance.

Open edge: Verification gates can miss domain-specific failures; evidence is limited to a small number of planted or benchmark failure cases.

Human-AI collaboration and intervention design

Allocates human oversight to high-leverage decision points instead of fully manual or fully autonomous workflows.

Open edge: Optimal intervention modes and their transferability across disciplines and task lengths are not established.

Automated review and evaluation

Provides scalable LLM/VLM-based scoring of generated papers and research artifacts, including rubric, acceptance threshold, and blind comparison signals.

Open edge: Agreement with human peer review and coverage of rigorous theoretical, empirical, and domain-specific work is incomplete.

Shared Direction

End-to-end pipelines now couple experiment execution with manuscript drafting and review, moving beyond isolated text generation.
Verification and grounded reporting are treated as necessary for trustworthy generated papers, not as optional add-ons.
Automated LLM/VLM reviewers or rubrics are used as scalable evaluation proxies, often compared to human acceptance thresholds or blind scores.
Selective human oversight can improve robustness, and full autonomy remains inconsistent across runs.
Reproducibility signals such as open-source code or artifacts are present but uneven, especially for systems evaluated on proprietary or domain-specific setups.

Key Differences

Supervision level: systems differ between full autonomy, structured human-in-the-loop intervention, and PI-graduate-student collaboration.
Verification anchor: logical/theorem audits, vision-language physical validity gates, grounded numeric/citation reporting, and causal failure attribution target different failure modes.
Evaluation target: acceptance thresholds, rubric scores, benchmark success rates, artifact quality, and blind review scores are not directly comparable.
Domain scope: general ML workspace systems contrast with applied mathematics and computational fluid dynamics systems that embed domain-specific knowledge/validation.
Reported capability: some claim workshop-level peer review acceptance; others report only rubric or benchmark improvements, leaving conference-level generalization unresolved.

Open Questions

How well do LLM/VLM reviewers align with human peer review outside narrow workshop or rubric evaluations?
Which verification mechanisms can catch the silent failures that current gates miss, such as 2 of 16 planted failures in one study?
Can these systems produce full conference or journal papers with reproducible code, data, baselines, and limitations, not just workshop-level acceptance?
What is the minimally sufficient human intervention pattern across disciplines and long-horizon research?
How can domain-specific knowledge be incorporated without losing transfer across new research areas?

AILGCLMAmtrl-sciDLCYIR

355 papers

Literature Intelligence

Automated literature search, paper reading, citation analysis, literature review, survey synthesis, and research gap discovery.

55 Methodology quality 94 Recency 40 Reproducibility 35 Topical relevance

Literature review synthesis

Research Lines

Autonomous multi-agent research generation with verification and knowledge retrieval, e.g., ReasFlow

End-to-end literature synthesis, derivation, experimentation, and paper generation in reasoning-centric domains under human principal-investigator supervision.

Open edge: Evaluation relies on curated LLM review rubrics; no external human validity evidence or limitation analysis is reported, leaving cross-domain trust uncertain.

Evidence-grounded evaluation of open-ended discovery agents, e.g., TruthInsightBench and BAISBench

Uses blind tasks, frozen data, and peer-reviewed-study-derived questions to compare whether agents support claims with controls, robustness, falsifiability, and auditability.

Open edge: Coverage is limited to a single frozen base model or a specific biological data setting; cross-base-model and cross-domain generalization remain unverified.

Failure-case and design-principle analysis for autonomous research pipelines, e.g., Why LLMs Aren't Scientists Yet

Identifies recurring failure modes such as implementation drift, memory degradation, overexcitement, insufficient domain intelligence, and weak scientific taste.

Open edge: Evidence comes from four end-to-end attempts in ML research, so it does not establish which design interventions generalize to other pipelines or fields.

Self-evolving domain-agnostic agent frameworks, e.g., EvoMaster

Provides reusable infrastructure for iterative hypothesis refinement, self-critique, and knowledge accumulation across scientific disciplines.

Open edge: Reported benchmark scores are task-proxy results; missing limitation details and external validation across hardware, datasets, and user settings leave practical reliability open.

Shared Direction

Autonomous discovery agents are still treated as assistants rather than fully independent scientists, with human oversight or human baselines remaining central.
Verification loops, self-critique, knowledge retrieval, and evidence auditing are repeated mechanism choices for reducing unchecked model output.
Evaluations increasingly use frozen data, automated judges, and peer-reviewed-study-derived tasks to make open-ended claims comparable.
Current systems can execute and document analyses competently, but they largely lack controls, robustness, falsifiability, and cross-domain generalization.
Literature review and synthesis automation is repeatedly identified as an unresolved challenge across system-building, benchmarking, and survey evidence.

Key Differences

Evaluation target differs: one line scores generated paper quality with an LLM rubric, another scores empirical claim evidence maturity, another scores domain-specific biological discovery, and another aggregates general benchmark performance.
Supervision and interaction differ: human-as-PI and agent-as-graduate-student collaboration contrasts with autonomous self-evolving pipelines and with blind evaluation of coding agents.
Performance interpretation conflicts: some evidence reports highest LLM-rubric scores, while other evidence finds no statistically reliable separation among coding agents and documents multiple failure modes.
Task scope differs: theory-driven mathematical derivation, open-ended peer-reviewed data analysis, single-cell omics, and general science benchmark tasks impose different notions of valid evidence.

Open Questions

Do LLM-judge and evidence-maturity scores predict human research utility or publication quality when independent expert review is missing from the evidence packet?
Can the observed gaps in controls, robustness, falsifiability, and cross-domain generalization be reduced by stronger verification loops, or are they inherent limits of a frozen base model?
How far do findings transfer beyond the limited systems, domains, and one successful pipeline attempt represented in this evidence?
What external validation standard would make autonomous literature synthesis and discovery claims trustworthy across different scientific tasks?

AICLLGIRDLMAmtrl-sciCY

318 papers

Hypothesis & Idea Generation

Systems that propose hypotheses, research ideas, experimental directions, or scientific questions.

56 Methodology quality 94 Recency 40 Reproducibility 37 Topical relevance

Literature review synthesis

Research Lines

Autonomous hypothesis generation with verification-gated execution

Turns open-ended hypotheses or research directions into executable experiments whose results are checked before acceptance, reducing silent failures and unsupported claims.

Open edge: Verification gates still miss some planted failures and are validated mainly in narrow domains; generalization to unseen failure modes is not established.

Structured failure attribution and self-correction

Converts failed experiments into evidence-grounded causal diagnoses routed to hypothesis, design, or implementation fixes, improving metrics-bearing artifacts.

Open edge: Fully autonomous scientific writing and conference-ready reporting remain unsolved, and evaluation is concentrated on a small benchmark.

Human-in-the-loop multi-agent research orchestration

Coordinates agent roles, workflows, and selective human interventions at high-leverage points, with reporting checks against fabricated numbers or hallucinated citations.

Open edge: Evidence is limited to small internal case studies and expert preference judgments; optimal intervention modes across domains are unresolved.

Benchmarks and testbeds for AI-driven scientific discovery

Provides comparable tasks and baselines for cell annotation, biology discovery questions, and improving published ML models across domains.

Open edge: Current gains often reflect ML-engineering tricks rather than research insight, and proxy benchmark performance is not yet tied to real-world discovery utility.

Shared Direction

Autonomous research systems converge on a loop of hypothesis generation, executable experiment, verification, and result-grounded reporting.
Reliability is treated as a first-class design goal through verification gates, deterministic authority separation, or failure attribution rather than raw model generation.
Selective human collaboration at high-leverage decision points tends to outperform both full autonomy and exhaustive step-by-step oversight in reported ablations.
Benchmark and expert-judge evaluations are supplementing leaderboard metrics, but observed improvements often concentrate in code execution and engineering changes.

Key Differences

Verification mechanism differs: physics-specific visual inspection, deterministic source-edit authority, structured debate, and multi-hypothesis causal diagnosis encode trust differently.
Interaction mode differs: some systems target full autonomy, others emphasize human-in-the-loop checkpoints, and one reports optimal performance with targeted rather than exhaustive oversight.
Evaluation target differs: single-cell biology discovery, computational fluid dynamics, time-series forecasting, and cross-domain ML model improvement impose different success criteria.
Measurement focus differs: implementation success and token cost, benchmark accuracy, expert preference, or metrics-bearing artifact rate are not interchangeable evidence.

Open Questions

Can autonomous research agents distinguish a novel scientific hypothesis from an ML-engineering trick, especially on long-horizon improvement tasks?
Do verification and validity gates transfer to domains or failure modes not represented in the ablation set?
Are current AI scientist benchmark improvements predictive of downstream discovery utility in real laboratories?
What is the minimal and most effective human intervention schedule across research workflows beyond small case studies?
Can these systems reliably automate literature grounding without hallucinated citations or fabricated numerical results?

AICLLGMAIRmtrl-sciDLCE

235 papers

Experiment Automation

Agents that design, execute, monitor, analyze, or iterate experiments, including code-based and lab-based experimentation.

63 Methodology quality 95 Recency 41 Reproducibility 41 Topical relevance

Literature review synthesis

Research Lines

End-to-end autonomous research agents

Automating ideation, code or experiment execution, visualization, and manuscript generation in closed scientific discovery loops.

Open edge: Reliability across runs and venues, human-expert agreement, and whether generated papers correspond to reproducible results.

Scientific discovery task environments

Providing controllable multi-domain tasks and metrics for hypothesis formation, experiment design, analysis, and conclusion use.

Open edge: Whether task completion and proxy metrics capture research-level insight rather than surface behavior or engineering tricks.

Failure-aware memory for research agents

Converting failed attempts into reusable negative evidence to improve retry and cross-task experimentation while reducing token cost.

Open edge: Granularity of typed failure records, transfer outside PDE and ScienceAgentBench settings, and integration with larger research workflows.

Interactive multi-agent research infrastructure

Providing customizable teams, role-based workflows, code harnesses, dashboards, monitoring, and rollback to make autonomous research controllable.

Open edge: Maturity, standardization, and external validation across hardware, datasets, teams, and research domains.

Shared Direction

Autonomous systems can cover idea generation, experiment execution, analysis, and manuscript writing as a continuous loop, but human oversight or verification remains common.
Agents benefit from structured memory, verification, or failure records rather than treating each run as isolated from past attempts.
Evaluation of generated scientific work relies on automated reviewers, expert preference, task-success metrics, or workshop peer review, but these proxies are not fully aligned with reproducible scientific value.
Current scientific agent environments show that strong general-purpose agents struggle or make progress mainly through implementation-level tricks, revealing a gap between engineering and discovery.

Key Differences

Task and evaluation target differs: full manuscript generation and venue acceptance versus virtual discovery tasks versus improving published scientific ML models versus solving PDE problems.
Interaction and supervision differ: fully automated execution, principal-investigator and graduate-student guidance, human dashboard rollback, and curator and researcher shared-memory architectures.
Control mechanism differs: progressive agentic tree search, role-specialized multi-agent teams, failure-record adoption, and code-harness execution loops.
Validation strength differs: some evidence uses LLM review rubrics, some uses AI expert judging on case studies, one uses workshop peer review acceptance, and task suites use completion metrics.
Deployment settings differ: text-based simulated lab tasks, local ML codebases, applied mathematics derivation and theory, and scientific ML benchmarks across multiple domains.

Open Questions

How can automated research agents move beyond local engineering sweeps and produce genuine domain-level research insights?
Do automated reviewer rubrics and expert-preference comparisons predict human peer review, reproducibility, and long-term scientific impact?
Can negative knowledge captured from failures transfer robustly across scientific domains, tasks, and experimental infrastructures beyond the exhibited PDE cases?
What evidence is needed to validate autonomous research infrastructure outside small internal case studies and curated task suites?
What is the value of end-to-end paper generation when limitations, code, datasets, and external reproducibility details are missing or weakly evidenced?

AILGCLmtrl-sciMACYCEIR

49 papers

Hypothesis Generation

Papers and resources related to Hypothesis Generation.

65 Methodology quality 96 Recency 39 Reproducibility 45 Topical relevance

Literature review synthesis

Research Lines

Autonomous hypothesis-to-result pipelines

Turns an open-ended objective and dataset into iterative data analysis, literature search, hypothesis generation, and experiment execution, producing traceable reports.

Open edge: Long-horizon accuracy and factuality are incomplete; reported statement accuracy is below full reliability, and scaling was tested only up to a bounded number of cycles.

Structured search and evolutionary research trajectories

Explores hypothesis, algorithm, or equation spaces through branching, backtracking, or evolutionary selection, yielding more controlled and inspectable research trajectories.

Open edge: Scaling beyond tested combinatorial optimization, driving, or forecasting cases and isolating search gains from model or task-specific priors remain unresolved.

Human-AI collaboration and oversight protocols

Allocates human intervention to high-leverage decision points to improve reliability without requiring exhaustive step-by-step oversight.

Open edge: Which decision points justify intervention and how autonomy boundaries generalize beyond the reported ablations lack systematic external validation.

Evaluation benchmarks and evidence design

Measures real-world research capabilities with realistic tasks and public evaluation harnesses, quantifying difficulty shifts across model generations.

Open edge: The alignment between benchmark metrics and true discovery novelty, repeatability, or downstream utility is not established, and domain coverage remains limited.

Shared Direction

Systems converge on iterative cycles of analysis, literature search, hypothesis generation, and experimental validation, with language agents writing or executing code to test hypotheses.
Evidence grounding and traceability are treated as core reliability mechanisms: reports or results should cite code, data, or prior literature, or pass deterministic authority checks.
Evaluation relies on benchmark tasks and baseline comparison, but reported accuracy or benchmark performance still leaves residual error or untested edge cases.
Autonomy is viewed as domain-conditioned: credible in structured, executable, rapidly verifiable settings, while embodied, delayed, heterogeneous, or ethically accountable settings remain limited.

Key Differences

Interaction mode differs: some systems rely on long-horizon fully autonomous operation, while others show that targeted human collaboration at high-leverage points outperforms full autonomy or exhaustive oversight.
Exploration mechanism differs: structured tree or evolutionary search uses branching and backtracking to control trajectory, whereas parallel iterative loops rely on large-scale agent rollouts and information sharing.
Authority design differs: some separate cognition from deterministic program authority for source edits and promotion, while others depend on agent self-verification and traceable reporting to prevent fabricated outputs.
Evaluation target differs: biology research benchmarks, scientific equation discovery, combinatorial optimization, and real-world forecasting cases are not directly comparable because they assess different research capabilities.

Open Questions

How do hypothesis quality and factual accuracy change beyond the tested number of cycles or agent rollouts? Existing evidence covers only bounded runs and specific benchmarks.
Do proxy research benchmarks such as ARC-Bench and LABBench2 correlate with real scientific discovery value, novelty, reproducibility, or downstream utility?
Under what conditions is human intervention necessary, and which decision points produce the largest reliability gain per unit of human effort?
Can systems reject weak directions early and avoid fabricated numbers or hallucinated citations in open, non-code-executable research environments?
How much of the reported improvement comes from hypothesis generation itself, versus task-specific priors, baselines, or execution and controller design?

AICLLGGNMADLCENE

47 papers

Scientific Discovery Agents

Papers and resources related to Scientific Discovery Agents.

36 Methodology quality 94 Recency 47 Reproducibility 23 Topical relevance

Literature review synthesis

Research Lines

Complete discovery-loop benchmarking

Assesses whether agents can form hypotheses, design experiments, analyze results, and act in controlled discovery environments.

Open edge: Automatic task-completion metrics may not capture scientific validity, and task diversity beyond curated environments is unproven.

Evidence-grounded open-ended evaluation

Blindly exposes peer-reviewed data to agents and scores whether claims are supported by evidence maturity rather than reproduction.

Open edge: Only one frozen base model is reported; LLM judge alignment with human experts and cross-domain generalization are not established.

Auditable research question formation

Structures assumptions, mechanisms, falsifiable hypotheses, tests, and update rules before experimentation.

Open edge: Evidence is preliminary and judge-based; downstream effect on actual discovery outcomes is not shown.

Constrained end-to-end coding evaluation

Tests autonomous model design, implementation, and training under single-GPU, no-internet, and time-budget limits.

Open edge: Only seven ML tasks, and constraints may not represent real scientific software workflows or larger projects.

Shared Direction

Agents remain far from trustworthy autonomous discovery; strong baselines struggle or plateau on existing benchmarks.
Evaluation is increasingly organized around open-ended scientific goals, evidence maturity, safety, and reproducibility rather than simple answer matching.
Agents can execute and document analyses or code, but lack controls, robustness, falsifiability, and cross-domain generalization.
Automated LLM judges are common but their reliability as proxies for domain expertise is unresolved.

Key Differences

Evaluation targets differ: discovery-loop task completion, evidence maturity, constrained coding performance, wet-lab safety, or auditable question quality.
Interaction settings differ: text-based simulated environments, blind peer-reviewed datasets, single-GPU code-only tasks, and physical or virtual execution loops.
Evidence sources differ: automatic metrics, LLM judges, domain expert checks, or execution success and safety-gate intercepts.
Representations and scaffolds differ: structured research-question certificates, distilled code libraries, episodic memory, or trajectory-aware risk state.

Open Questions

How well do automated LLM judges align with human domain experts across scientific fields?
Can agents produce novel, generalizable claims rather than rederiving or reproducing hidden conclusions?
Do safety, auditability, and evidence-maturity improvements transfer across base models, domains, and deployment settings?
Which evaluation constraints preserve scientific validity while still reflecting real discovery workflows?
Where does the remaining gap lie between competent analysis/documentation and trustworthy, falsifiable scientific discovery?

AILGCLcomp-phmtrl-scichem-ph

45 papers

Survey Generation

Papers and resources related to Survey Generation.

54 Methodology quality 89 Recency 42 Reproducibility 35 Topical relevance

Literature review synthesis

Research Lines

Agentic iterative survey generation

Automates retrieval, outline expansion, section drafting, and rubric/reviewer-driven refinement to produce organized, long-form literature surveys.

Open edge: Many evaluations use proxy coverage/structure/citation metrics; matched-backend end-to-end comparisons and rigorous cross-domain external validation are missing.

Structured knowledge and taxonomy organization

Replaces flat paper/document views with methods-entity relations or hierarchical taxonomies to improve retrieval, outlining, and topic organization.

Open edge: Semantic relation support is incomplete, with relation types such as extends weakest; alignment to a single expert taxonomy remains low and assumes that reference taxonomy is valid.

Evidence-gated citation and synthesis verification

Uses full-text extraction, citation graph expansion, hybrid filtering, semantic evidence gates, and multi-agent verification to reduce unsupported claims.

Open edge: Audits still report partial or unsupported citations and semantic edges; evidence gates achieve moderate AUC and do not ensure statement-level correctness across all topics.

Evaluation and benchmarking for survey quality

Provides benchmarks, taxonomies, pairwise arenas, and metrics for retrieval, taxonomy structure, content coverage, and citation quality comparisons.

Open edge: Reference-based alignment scores disagree with reference-free capability gaps; benchmarks often depend on single expert-authored taxonomies or narrow proxy metrics.

Shared Direction

LLM-based agentic systems are the dominant way to automate survey generation, with retrieval, iterative planning, and multi-stage refinement as core components.
Survey quality is evaluated through coverage, structure, citation fidelity, and taxonomy or outline alignment rather than any single surface-level score.
Citation reliability and retrieval quality are treated as primary failure modes, prompting evidence gates, citation filters, and paper-level grounding.
Structured organization, especially hierarchical taxonomy or method-entity relation modeling, is considered central to producing expert-like surveys.
Automated metrics and LLM/human judgment are used together, but results show expert alignment remains difficult.

Key Differences

Representation differs: some systems operate over flat retrieved documents while others build methods-entity knowledge graphs or expert-like taxonomies.
Evaluation target differs: rubric-aligned quality, pairwise human-preference comparisons, expert taxonomy alignment, and citation precision/recall are optimized by different lines of work.
Interaction mode differs: fully autonomous generation pipelines contrast with governed scientist-in-the-loop or aggregate-only research orchestration.
Task scope differs: some work targets end-to-end survey drafting, others focus narrowly on taxonomy organization, retrieval evaluation, or research workflow orchestration.
Evidence enforcement differs: some systems use semantic evidence gates and citation graph constraints, while others rely on iterative LLM reviewer or rubric feedback without statement-level evidence gating.

Open Questions

Why does the best deep research agent retrieve only 20.92% of expert-cited papers, and how much of that is retrieval failure versus legitimate expert curation mismatch?
Can methods-entity knowledge graphs improve weak semantic relation types such as extends, which were supported at only 52.8% in human adjudication?
Do rubric scores, semantic path similarity, and citation metrics predict actual researcher utility or expert acceptance of a generated survey?
How do these systems perform in matched-backend end-to-end comparisons rather than bounded-context concurrent-system comparisons?
Can governed end-to-end research systems generalize from a hypertension GWAS case to diverse survey-generation topics while preserving institutional privacy and reproducibility?

CLAIIRDLHCLG

36 papers

Research Agent Benchmarks

Papers and resources related to Research Agent Benchmarks.

29 Methodology quality 95 Recency 40 Reproducibility 16 Topical relevance

Literature review synthesis

Research Lines

Research-state and experience representation

Turns long experiment histories into durable compact state such as harness-owned search state, reusable experience libraries, or repository-derived skills so agents avoid history replay, duplicated work, and excessive interaction cost.

Open edge: Whether the represented state or skills transfer outside the benchmark families and domains in the visible evidence is unclear; several records provide no explicit limitation details.

Closed-loop search and feedback allocation

Connects prediction to interventions by assigning experiments, selecting pilot or high-fidelity evaluations, and rewriting optimizers under token, oracle-query, or force-call budgets.

Open edge: How much gain comes from search and allocation policy versus fixed backbone, harness, or task-specific prior remains difficult to attribute; domain coverage is narrow.

Benchmark and held-out evaluation design

Makes claims comparable through fixed budgets, sealed or held-out targets, oracle query counts, baseline comparisons, and target-disagreement ceilings.

Open edge: Proxy metrics may not align with real utility; development-best evaluators do not always remain best on sealed targets, and human-designed baselines remain strong.

Resource-aware research infrastructure

Distills repositories into verified skills, builds executable evaluators or harnesses, and reduces token, oracle-query, or force-call costs across multiple suites.

Open edge: Visible reproducibility evidence is sparse: code, dataset, artifact, and limitation fields are often missing; standardized cross-framework external validation is absent.

Shared Direction

Long-horizon autonomous research is usually measured by final task improvement and resource cost together, not only by single-point accuracy.
Feedback design is central: held-out or sealed evaluation, sparse high-fidelity labels, pilot tests, and explicit budgets are used to test generalization beyond training/search.
Separating durable research state or external knowledge from the agent's immediate context improves efficiency and reuse.
Short development evaluation can mislead, so several studies use sealed/hold-out tests and target-disagreement ceilings to correct overoptimistic conclusions.

Key Differences

Supervision signal differs: one line uses target benchmark agreement, another uses held-out metric reductions, another uses sparse high-fidelity oracle labels, and another uses downstream benchmark score gains.
Evaluation target differs: code and ML benchmark suites, materials endpoints such as Matbench, molecular optimizer force calls, and evaluator-design tasks each serve as the primary testbed in different strands.
Interaction mode differs: a stateless harness owns shared search state, an experience library retrieves prior evidence, repository-derived skills provide operational knowledge, and online surrogate repair interleaves high-fidelity feedback.
Deployment setting differs: software/ML research, materials science, molecular geometry optimization, and autonomous evaluation research define the boundary of a research agent differently.

Open Questions

Do gains on development targets transfer to sealed or held-out tasks when target disagreement ceilings are high?
How much of reported improvement comes from agent search policy versus fixed backbone, harness, judge, or task-specific priors?
Are token, oracle-query, and force-call efficiency metrics comparable across frameworks when budgets and baselines differ?
Several visible records lack code, datasets, artifacts, or explicit limitations; can standardized external validation close these verification gaps?

25 papers

Automated Experimentation

Papers and resources related to Automated Experimentation.

44 Methodology quality 87 Recency 39 Reproducibility 29 Topical relevance

Literature review synthesis

Research Lines

End-to-end autonomous research-lab and pipeline automation

Framing entire research cycles, including ideation, literature analysis, experimental design, execution, interpretation, manuscript drafting, and provenance logging, as automatable workflows.

Open edge: Governance, community coordination, implemented prototypes, and empirical validation are missing or unresolved.

Agentic experiment campaign orchestration

Maintaining persistent optimization state across computational or physical campaigns while separating LLM reasoning from repeated Bayesian optimization or code-editing loops.

Open edge: Inference cost is substantial, domain coverage remains limited, and comparisons with human-led campaign efficiency are not standardized.

Predictive digital twins for instruments and samples

Forecasting outcomes, uncertainty, and risk of candidate operations by pairing sample-state representations with instrument or compute-infrastructure models.

Open edge: Validation is modality-specific, instrument-model mismatch remains, and transfer beyond the demonstrated microscopy or color-mixing settings is not shown.

Noisy-feature detection and recovery for self-driving labs

Detecting and correcting noisy features in experimental datasets by analyzing dataset size, noise intensity, noise type, and feature distribution with kNN-based workflows.

Open edge: Low-intensity noise, discrete or narrow feature distributions, and generalization beyond DFT and self-driving-lab datasets remain unresolved.

Shared Direction

Multiple systems agree on closing an experiment loop that includes proposing changes, executing or simulating them, evaluating by constraints or fitness, and updating persistent state.
There is a shared shift from hyperparameter optimization toward higher-level experiment design, interpretation, and model or campaign management.
Physical and digital evaluations are used to demonstrate autonomy, but evidence quality varies and often lacks standardized baselines or released artifacts.
Human oversight is recognized as important for guidance, review, or governance, even in systems described as autonomous.

Key Differences

Interaction modes differ: some propose fully autonomous research labs, while others keep human experts in the loop for novelty assessment, feedback, or campaign supervision.
Representations differ: language agents that edit code or manage campaigns, learned predictive twins, and nearest-neighbor data correction rely on different assumptions about where intelligence and predictive modeling are needed.
Evaluation targets differ: recommender-research automation, interatomic-potential training, microscopy, flow chemistry, color mixing, and data-cleaning benchmarks are not directly comparable.
Deployment assumptions differ: HPC job management, nanoHUB infrastructure, physical instruments, and purely digital tasks impose different reproducibility and scaling requirements.

Open Questions

Which evaluation protocols would certify that an autonomous experiment system produces reproducible, governable, and scientifically meaningful results beyond task-specific scorecards?
How do predictive twins and agentic controllers transfer across instrument modalities, material systems, and lab infrastructures?
Can noise detection and closed-loop optimization remain reliable with small, narrow, or low-intensity-noise datasets?
What is the cost-benefit boundary for LLM inference in long-running physical campaigns, and when is human supervision necessary?

LGAImtrl-sciCLdata-anchem-phCEIR

20 papers

Literature Review Agents

Papers and resources related to Literature Review Agents.

17 Methodology quality 89 Recency 36 Reproducibility 14 Topical relevance

Literature review synthesis

Research Lines

Retrieval-grounded literature and evidence access

Converts literature databases into queryable evidence layers so review agents can retrieve, verify, and cite source material without relying only on parametric knowledge or open web search.

Open edge: Coverage beyond life-sciences corpora and whether improved citation-level metrics generalize to broader expert review quality is not established.

Multi-agent and iterative review generation workflows

Automates structuring, writing, editing, and review using specialized or iterative agent roles to improve readability, citation quality, and similarity to human surveys.

Open edge: Evidence is tied to limited benchmark datasets and baselines; the contribution of each agent stage and robustness across domains remain underexamined.

Evaluation and leaderboard certification for review agents

Defines fair comparison conditions and compositional controllability for biomedical literature-review agents, exposing when leaderboard comparisons are admissible.

Open edge: The provided packet lacks detailed methodology and results, so whether the controllability framework transfers beyond biomedical reviews and which metrics certify real utility is unknown.

Conceptual surveys and scientific-agent integration

Frames literature review as one component inside broader AI scientist systems, using historical systems such as Adam and Eve as an organizing lens for automation and integration challenges.

Open edge: Integration of literature evidence with hypothesis-experiment loops remains unsolved; claims are largely qualitative and long-term.

Shared Direction

Recent systems treat literature review as a compound task requiring retrieval, source grounding, structured organization, drafting, and evaluation rather than a single LLM call.
They emphasize citation-level or expert-aligned evidence as a central quality target, even when exact metrics differ.
There is convergence on using LLM agents as orchestrators: decomposing queries, calling live databases or search, and aggregating evidence.

Key Differences

One line grounds review quality in live literature database retrieval and expert-consensus verification, while another focuses on role-specialized multi-agent writing workflows judged by similarity and citation quality.
Evaluation designs differ: some use fixed review benchmarks with established baselines, while leaderboard certification work questions whether many comparisons are admissible due to controllability constraints.
Scope differs between domain-specific biomedical/life-science systems and general literature-review automation, with different assumptions about domain-specific metadata and evaluation.

Open Questions

Does improving Citation F1 and expert consensus on literature-probing benchmarks translate into more reliable and useful full literature reviews for researchers?
Which review-agent components, such as retrieval, planning, source extraction, writing, and editing, causally drive gains, and what failure modes remain outside current benchmarks?
What evaluation conditions make literature-review agent comparisons fair and externally valid beyond the biomedical/life-science settings represented in the evidence?

13 papers

Paper Writing Agents

Papers and resources related to Paper Writing Agents.

1 Methodology quality 90 Recency 47 Reproducibility 6 Topical relevance

Literature review synthesis

Research Lines

Automated paper drafting and generation

Producing a complete research paper from a high-level idea or outline, reducing manual writing effort.

Open edge: Quality, originality, and factual reliability of generated content; lack of standardized evaluation against human-written papers.

Interactive writing assistance and tutoring

Providing real-time feedback, editing suggestions, and writing guidance within an editor to improve draft quality.

Open edge: Adaptation to diverse writing styles and disciplines; measurable impact on final paper acceptance or readability.

Paper review and critique

Generating or evaluating critiques of scientific papers, aiming to support or automate peer review.

Open edge: Groundedness of critiques beyond surface claims; coverage of nuanced methodological flaws and integration into real review workflows.

Multimodal paper transformation

Converting paper content into other formats such as posters, slides, or visual summaries.

Open edge: Design quality, content selection accuracy, and user customization; evaluation beyond aesthetic appeal.

Shared Direction

All works leverage large language models as the core reasoning engine for paper-related tasks.
Multi-agent architectures are widely adopted to decompose complex writing or analysis workflows.
The research is concentrated in 2025–2026, indicating a nascent but rapidly growing area.
There is a shared emphasis on integrating tools into existing academic environments (e.g., Overleaf) or providing reproducible artifacts.

Key Differences

Task scope: some systems target end-to-end paper generation, while others focus on editing, review, poster creation, or experiment reproduction.
Autonomy level: fully automated generation versus human-in-the-loop tutoring or assistance.
Interaction mode: in-editor plugins, standalone frameworks, or benchmark suites with no direct user interaction.
Evaluation target: output text quality, critique groundedness, reproduction success rate, or search recall/precision.

Open Questions

How can we evaluate the novelty and scientific contribution of automatically generated papers beyond surface-level metrics?
What mechanisms ensure factual accuracy, proper citation, and avoidance of plagiarism in generated content?
How do these agents perform on papers requiring deep domain expertise or novel experimental designs?
What are the ethical implications and potential for misuse in academic publishing, and how can they be mitigated?