Can Language Models Serve as Text-Based World Simulators?

ACL 2024 67.8 benchmark

TLDR

LLMs are tested as text-based world simulators using a new benchmark; GPT-4 proves unreliable, highlighting limitations.

Reasoning

The paper introduces a novel benchmark (ByteSized32-State-Prediction) to directly quantify LLM performance as world simulators, which is a strength. However, it only tests GPT-4 and focuses on text-based environments, limiting generalizability. The empirical evaluation provides concrete evidence for its claims.

Read-first score

Read-first score 67.8, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 40.

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=True; dataset=False; markers=dataset,github

Recency 8%

75.1

Uses a gentle age decay so recent papers surface without erasing older foundations. 2024

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=benchmark,dataset

Topical relevance 42%

57.1

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Field roles

Methodology anchorReproducibility anchor

Rank sensitivity

Stability: volatile; rank range: 252.

Keyword Scores

world simulator

world dynamics prediction

world model

interactive world model

model-based reinforcement learning world model

generative world model

video world model

Deep Analysis

Innovations

Introduction of ByteSized32-State-Prediction, a new benchmark for quantifying LLMs as text-based world simulators
First direct quantification of how well LLMs can serve as text-based world simulators
Insights into GPT-4's capabilities and weaknesses as a world simulator

Methodology

The authors constructed a new benchmark, ByteSized32-State-Prediction, consisting of a dataset of text game state transitions and accompanying game tasks. They used this benchmark to evaluate GPT-4's ability to predict state changes, directly quantifying its performance as a text-based world simulator.

Key Results

GPT-4, despite impressive performance, is still an unreliable world simulator without further innovations.

Limitations

GPT-4 is unreliable as a world simulator, indicating current LLMs are insufficient without further innovations

Resources

Paper Code