VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agents
TLDR
VAGEN uses RL to train VLM agents with explicit visual state reasoning, decomposing it into state estimation and transition modeling, achieving 3x improvement.
Reasoning
The paper presents a novel RL-based approach for world model reasoning in VLM agents, with clear methodology and strong empirical results across multiple benchmarks. However, the abstract lacks details on generalization to other modalities or tasks, and the reliance on specific reasoning formats may limit applicability.
Read-first score
Read-first score 53.3, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 33.
Field roles
Rank sensitivity
Stability: volatile; rank range: 324.
Keyword Scores
Deep Analysis
Innovations
- Architecturally enforcing and rewarding the agent's reasoning process via reinforcement learning (RL) for VLM agents
- Formulating the problem as a Partially Observable Markov Decision Process (POMDP)
- Decomposing reasoning into State Estimation and Transition Modeling
- World Modeling Reward providing dense, turn-level supervision for accurate state prediction
- Bi-Level General Advantage Estimation (Bi-Level GAE) for turn-aware credit assignment
Methodology
The paper proposes VAGEN, a scalable framework for training multi-turn VLM agents using reinforcement learning. The agent's reasoning is decomposed into State Estimation and Transition Modeling, and a World Modeling Reward is designed to provide dense supervision. Bi-Level GAE is introduced for turn-aware credit assignment. The method is evaluated across five diverse agent benchmarks using a 3B-parameter model.
Key Results
A 3B-parameter model achieves a score of 0.82 across five benchmarks, a 3× improvement over its untrained counterpart (0.21) and outperforming proprietary models like GPT-5 (0.75), Gemini 2.5 Pro (0.67), and Claude 4.5 (0.62).