TesserAct: Learning 4D Embodied World Models
TLDR
Learns 4D embodied world models from RGB-DN videos, enabling spatial-temporal coherent scene prediction and improved policy learning.
Reasoning
The paper introduces a novel approach to learning 4D world models that incorporate depth and normal information, achieving spatial and temporal consistency. Strengths include a clear methodology and demonstrated policy improvement; weaknesses are reliance on off-the-shelf models for data annotation and limited discussion of limitations.
Read-first score
Read-first score 61.6, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 51.
Field roles
Rank sensitivity
Stability: volatile; rank range: 300.
Keyword Scores
Deep Analysis
Innovations
- Learning 4D embodied world models that predict dynamic evolution of 3D scenes over time with spatial and temporal consistency
- Using RGB-DN (RGB, Depth, Normal) videos as training data, surpassing traditional 2D models
- Extending existing robotic manipulation video datasets with depth and normal information using off-the-shelf models
- Fine-tuning a video generation model to jointly predict RGB, Depth, and Normal for each frame
- Algorithm to convert generated RGB-DN videos into high-quality 4D scenes
Methodology
The approach first extends existing robotic manipulation video datasets with depth and normal information using off-the-shelf models. Then, a video generation model is fine-tuned on this annotated dataset to jointly predict RGB, Depth, and Normal for each frame. Finally, an algorithm converts the generated RGB-DN videos into a high-quality 4D scene of the world, ensuring temporal and spatial coherence.
Key Results
The method enables novel view synthesis for embodied environments and facilitates policy learning that significantly outperforms those derived from prior video-based world models.