LLM-as-a-Judge: Toward World Models for Slate Recommendation Systems

arXiv 25.11 2025 42.7 method

TLDR

LLMs can act as world models for slate recommendation via pairwise reasoning, with empirical results across tasks and datasets.

Reasoning

The paper provides empirical evidence for using LLMs as world models in a specific domain (slate recommendation), which is a strength. However, it is limited to this narrow application and does not address broader world model capabilities or real-world deployment.

Read-first score

Read-first score 42.7, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 10.

Recency 8%

86.7

Uses a gentle age decay so recent papers surface without erasing older foundations. 2025

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=analysis,dataset,result

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=dataset

Topical relevance 42%

14.3

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Field roles

FrontierMethodology anchor

Rank sensitivity

Stability: volatile; rank range: 397.

Keyword Scores

world model

world simulator

generative world model

interactive world model

video world model

world dynamics prediction

model-based reinforcement learning world model

Deep Analysis

Innovations

LLM-as-a-Judge approach for slate recommendation systems
Pairwise reasoning over slates to model user preferences
Empirical analysis of LLM performance on three slate recommendation tasks

Methodology

The authors conduct an empirical study using several Large Language Models (LLMs) on three tasks spanning different datasets. They evaluate the LLMs' ability to act as world models of user preferences through pairwise reasoning over slates, analyzing relationships between task performance and properties of the preference function.

Key Results

The study reveals relationships between task performance and properties of the preference function captured by LLMs, indicating areas for improvement and highlighting the potential of LLMs as world models in recommender systems.

Limitations

The study is exploratory and does not provide a complete world model for slate recommendation
Performance of LLMs varies depending on preference function properties, indicating need for further improvement
Limited to three tasks and datasets; generalizability across domains is not established

Resources

Paper