⌁ Awesome World Model Hub Papers · Datasets · Projects
中文 GitHub
← Back to papers

PragWorld: A Benchmark Evaluating LLMs' Local World Model under Minimal Linguistic Alterations and Conversational Dynamics

arXiv 25.11 2025 47.6 benchmark

TLDR

Evaluates LLMs' local world model robustness in conversations under minimal linguistic alterations, proposes interpretability and fine-tuning methods.

Reasoning

Strengths include a novel benchmark and interpretability framework for analyzing LLM internal representations. Weaknesses are the narrow focus on dyadic conversations and minimal alterations, limiting generalizability.

Read-first score

Read-first score 47.6, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 15.

Recency 8%
86.7

Uses a gentle age decay so recent papers surface without erasing older foundations. 2025

Methodology quality 25%
80

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=analysis,benchmark,dataset

Reproducibility 25%
46

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=code,dataset

Topical relevance 42%
21.4

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Field roles

FrontierMethodology anchor

Rank sensitivity

Stability: volatile; rank range: 495.

Keyword Scores

world model
9
interactive world model
2
world dynamics prediction
2
world simulator
1
generative world model
1
video world model
0
model-based reinforcement learning world model
0

Deep Analysis

Innovations

  • PragWorld benchmark for evaluating LLMs' local world model under minimal linguistic alterations and conversational dynamics
  • Dual-perspective interpretability framework identifying transformer layers that are useful or harmful for tracking entities under linguistic alterations
  • Two layer-regularization based fine-tuning strategies that suppress the effect of harmful layers

Methodology

The authors construct two benchmarks by applying seven minimal linguistic alterations to conversations sourced from popular datasets, then create yes-no questions to test LLMs' ability to encode and update their internal world model. They evaluate a wide range of open and closed source LMs on these benchmarks and analyze performance. They also propose a dual-perspective interpretability framework to identify useful and harmful transformer layers, and introduce two layer-regularization fine-tuning strategies to mitigate the impact of harmful layers.

Key Results

LLMs struggle to maintain robust accuracy on the benchmarks, particularly in memorizing crucial details such as tracking entities under linguistic alterations to conversations.

Limitations

  • Benchmark is limited to dyadic conversations and yes-no questions, which may not capture the full complexity of pragmatic understanding
  • Only seven minimal linguistic alterations are applied, potentially missing other important pragmatic phenomena
  • The proposed fine-tuning strategies are introduced but their effectiveness is not demonstrated in the abstract
  • The benchmark is constructed from popular datasets, which may introduce dataset-specific biases

Resources

Paper

Tags