GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Control
TLDR
GEM is a multimodal world model for controllable future frame prediction with ego-motion, object dynamics, and human pose control, using a large real-world dataset.
Reasoning
The paper presents a novel world model with fine-grained control and long-horizon generation, supported by a large multimodal dataset and a new controllability metric. However, reliance on pseudo-labels and lack of explicit comparison to baselines may limit validation of its generalizability.
Read-first score
Read-first score 71.3, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 53.
Field roles
Rank sensitivity
Stability: volatile; rank range: 38.
Keyword Scores
Deep Analysis
Innovations
- Predicts future frames using reference frame, sparse features, human poses, and ego-trajectories for fine-grained control over object dynamics, ego-agent motion, and human poses
- Generates paired RGB and depth outputs for richer spatial understanding
- Introduces autoregressive noise schedules to enable stable long-horizon generations
- New Control of Object Manipulation (COM) metric to assess controllability
- Large-scale multimodal dataset of 4000+ hours across autonomous driving, egocentric human activities, and drone flights
Methodology
GEM is a multimodal world model that takes a reference frame, sparse features, human poses, and ego-trajectories as inputs to predict future frames. It generates paired RGB and depth outputs using autoregressive noise schedules for stable long-horizon generation. The model is trained on a dataset of 4000+ hours of multimodal data with pseudo-labels for depth, ego-trajectories, and human poses, and evaluated using a comprehensive framework including the COM metric.
Key Results
Experiments show GEM excels at generating diverse, controllable scenarios and maintains temporal consistency over long generations.