Latent Spatial Memory for Video World Models
TLDR
Introduces latent spatial memory for video world models, storing 3D scene info in diffusion latent space, achieving faster generation and reduced memory.
Reasoning
The paper presents a novel method (Mirage) that avoids pixel-space reconstruction, showing significant speed and memory improvements. Strengths include clear methodology and strong empirical results on benchmarks; weaknesses are limited scope to video generation without explicit interaction or RL context.
Read-first score
Read-first score 56.6, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 39.
Field roles
Rank sensitivity
Stability: volatile; rank range: 440.
Keyword Scores
Deep Analysis
Innovations
- Latent spatial memory that stores scene information directly in diffusion latent space, avoiding pixel-space reconstruction and its associated information loss and computational cost.
- Depth-guided back-projection to lift latent tokens into 3D and direct latent-space warping for novel view synthesis, forming a unified latent-space spatial memory framework (Mirage).
- Elimination of the round trip through pixel space, reducing both information loss and the computational burden of repeated rendering and VAE encoding.
Methodology
Mirage constructs a persistent 3D cache by lifting latent tokens from a diffusion model into 3D using depth-guided back-projection, and queries this memory by synthesizing novel views through direct latent-space warping. This avoids explicit point cloud memory in RGB space and the associated rendering and encoding steps.
Key Results
Latent spatial memory achieves up to 10.57× faster end-to-end video generation and 55× reduction in memory footprint relative to explicit 3D baselines. Mirage attains state-of-the-art performance on WorldScore and strong reconstruction quality on RealEstate10K.