Higher Embedding Dimension Creates a Stronger World Model for a Simple Sorting Task

arXiv 25.10 2025 32.1 theory

TLDR

Larger embedding dimensions improve the faithfulness, consistency, and interpretability of internal world models in transformers trained on a sorting task.

Reasoning

The paper provides clear quantitative evidence and mechanistic insights from hundreds of experiments, but its focus on a simple algorithmic task limits generalizability to complex real-world scenarios.

Read-first score

Read-first score 32.1, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 19.

Recency 6%

86.7

Uses a gentle age decay so recent papers surface without erasing older foundations. 2025

Methodology quality 18%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=experiment,metric,result

Reproducibility 18%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=code

Topical relevance 29%

27.1

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Citation impact 18%

Uses OpenAlex-shaped citation metadata as a bibliometric attention signal, separate from paper quality. cited_by_count=0

Citation velocity 12%

Citation velocity estimates citations per publication-year to reduce old-paper bias. velocity=0.00

Field roles

FrontierMethodology anchor

Rank sensitivity

Stability: volatile; rank range: 124.

Keyword Scores

world model

world dynamics prediction

model-based reinforcement learning world model

world simulator

generative world model

interactive world model

video world model

Deep Analysis

Innovations

Investigation of how embedding dimension affects the emergence of an internal world model in a transformer trained with reinforcement learning for a bubble-sort sorting task
Discovery of two consistent mechanisms: (1) the last row of the attention weight matrix monotonically encodes the global ordering of tokens; (2) the selected transposition aligns with the largest adjacent difference of these encoded values

Methodology

A transformer is trained with reinforcement learning to perform bubble-sort-style adjacent swaps. The embedding dimension is varied across hundreds of experiments to study its effect on internal representations. Models are evaluated on accuracy, faithfulness, consistency, and robustness of internal representations.

Key Results

Models achieve high accuracy even with very small embedding dimensions, but larger dimensions yield more faithful, consistent, and robust internal representations, leading to better interpretability.

Resources

Paper