Whole-Body Conditioned Egocentric Video Prediction

Innovations

• Conditioning egocentric video prediction on whole-body kinematic pose trajectories structured by the joint hierarchy of the human body.
• Auto-regressive conditional diffusion transformer for first-person video prediction from actions.
• Hierarchical evaluation protocol with increasingly challenging tasks for embodied prediction and control analysis.

Methodology

The authors train an auto-regressive conditional diffusion transformer on the Nymeria dataset, which contains real-world egocentric video and corresponding 3D body pose captures. The model takes past video and a relative 3D body pose trajectory as input, and predicts future video frames autoregressively, with the pose conditioning structured by the joint hierarchy.

Key Results

The model demonstrates the ability to simulate how physical human actions shape the environment from a first-person perspective, as evaluated on the Nymeria dataset using a hierarchical protocol. No quantitative results are reported in the abstract.

Limitations

• The work is described as an initial attempt, implying limited scope and unresolved challenges in modeling complex real-world environments.
• The evaluation is confined to the Nymeria dataset, so generalization to other environments or actions is not established.
• Autoregressive prediction may suffer from error accumulation over long horizons.
• The method requires full 3D body pose as input, which may not be readily available in many real-world scenarios.