BioKGBench: A Knowledge Graph Checking Benchmark of AI Agent for Biomedical Science

arXiv 2024 66.6 benchmark

TLDR

BioKGBench benchmarks AI agents for biomedical science by evaluating literature understanding via claim verification and KGQA, introducing KGCheck task to identify factual errors in knowledge graphs.

Reasoning

Strengths: novel benchmark addressing evaluation gap for AI scientists, disentangles literature understanding into atomic abilities, includes real-world data. Weaknesses: limited to biomedical domain, only evaluates literature understanding not other scientific abilities.

Read-first score

Read-first score 66.6, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 58.

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=True; dataset=False; markers=code,github

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=baseline,benchmark,evaluation,experiment

Recency 8%

75.1

Uses a gentle age decay so recent papers surface without erasing older foundations. 2024

Topical relevance 42%

48.3

Uses existing LLM keyword relevance scores normalized to 0-100. AI scientist,automated scientific discovery,autonomous research agent,automated research,literature review agent,survey generation,automated experimentation,experiment design agent,AI for scientific research,paper writing agent,research automation,scientific discovery agent

Field roles

Methodology anchorReproducibility anchor

Rank sensitivity

Stability: volatile; rank range: 87.

Keyword Scores

AI scientist

AI for scientific research

autonomous research agent

literature review agent

automated scientific discovery

automated research

scientific discovery agent

research automation

survey generation

automated experimentation

experiment design agent

paper writing agent

Deep Analysis

Innovations

Introduction of BioKGBench, a benchmark for evaluating biomedical AI agents from an AI Scientist perspective focusing on literature understanding.
Disentangling 'Understanding Literature' into two atomic abilities: scientific claim verification on unstructured text and Knowledge-Graph Question-Answering (KGQA) as literature grounding.
Formulation of a novel agent task, KGCheck, that uses KGQA and domain-based Retrieval-Augmented Generation (RAG) to identify factual errors in knowledge graph databases.
Proposal of a simple yet effective baseline agent, BKGAgent, for the benchmark.

Methodology

The benchmark comprises two atomic tasks (scientific claim verification and KGQA) with over two thousand data points, and an agent task (KGCheck) with 225 high-quality annotated instances. State-of-the-art agents from daily and biomedical domains are evaluated, and a baseline BKGAgent is introduced to perform error detection in knowledge graphs using KGQA and RAG.

Key Results

State-of-the-art agents fail or show inferior performance on BioKGBench; the proposed BKGAgent discovers over 90 factual errors in a widely used knowledge graph, demonstrating the benchmark's utility for agent evaluation and discovery.

Resources

Paper Code