Thinking by Doing: Building Efficient World Model Reasoning in LLMs via Multi-turn Interaction
TLDR
WMAct improves LLM world model reasoning via reward rescaling and interaction annealing, enabling single-turn task resolution.
Reasoning
The paper introduces novel mechanisms (reward rescaling and interaction frequency annealing) that effectively reduce redundant interactions and internalize environmental dynamics. Strengths include clear methodology and empirical validation on multiple environments, but the abstract lacks details on limitations and generalizability beyond the tested tasks.
Read-first score
Read-first score 48.2, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 31.
Field roles
Rank sensitivity
Stability: volatile; rank range: 406.
Keyword Scores
Deep Analysis
Innovations
- World-model internalization through efficient interaction and active reasoning (WMAct)
- Reward rescaling mechanism adjusting outcome reward based on action efficacy to incentivize redundancy reduction and purposeful interaction
- Interaction frequency annealing strategy to progressively reduce the maximum allowed interaction turns, compelling the model to condense learning and internalize environmental dynamics
Methodology
WMAct uses multi-turn interaction with two key mechanisms: a reward rescaling mechanism that adjusts outcome reward based on action efficacy to incentivize redundancy reduction and purposeful interaction, and an interaction frequency annealing strategy that progressively reduces the maximum allowed interaction turns, compelling the model to condense learning and internalize environmental dynamics. Experiments are conducted on Sokoban, Maze, and Taxi environments.
Key Results
WMAct yields effective world model reasoning capable of resolving tasks in a single turn that previously required multiple interactions, and fosters strong transferability to complex environments, improving performance on a suite of reasoning benchmarks.