⌁ Awesome World Model Hub Papers · Datasets · Projects
中文 GitHub
← Back to papers

AVID: Adapting Video Diffusion Models to World Models

arXiv 24.10 2024 75.2 method

TLDR

AVID adapts pretrained video diffusion models to action-conditioned world models using a learned mask, without accessing model parameters.

Reasoning

The paper presents a novel method for adapting closed-source video diffusion models to world models, addressing data scarcity in robotics. Strengths include leveraging unlabelled videos and evaluating on real-world data; weaknesses involve reliance on small domain-specific datasets and potential generalization limits.

Read-first score

Read-first score 75.2, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 44.

Methodology quality 25%
90

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=baseline,dataset,evaluation,result

Reproducibility 25%
81

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=True; dataset=False; markers=dataset,github

Recency 8%
75.1

Uses a gentle age decay so recent papers surface without erasing older foundations. 2024

Topical relevance 42%
62.9

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Field roles

Methodology anchorReproducibility anchor

Rank sensitivity

Stability: volatile; rank range: 114.

Keyword Scores

world model
9
video world model
8
world dynamics prediction
7
generative world model
6
model-based reinforcement learning world model
6
world simulator
5
interactive world model
3

Deep Analysis

Innovations

  • Adapting pretrained video diffusion models to action-conditioned world models without accessing the model parameters
  • Using a learned mask to modify intermediate outputs of the pretrained model for action conditioning
  • Demonstrating that pretrained video models can be effectively leveraged for embodied AI with only a small domain-specific action-labelled dataset

Methodology

AVID trains an adapter on a small domain-specific dataset of action-labelled videos. The adapter uses a learned mask to modify the intermediate outputs of a closed-source pretrained video diffusion model, enabling action-conditioned video generation without finetuning the original model. Evaluation is performed on video game and real-world robotics data, comparing against existing baselines for diffusion model adaptation.

Key Results

AVID outperforms existing baselines for diffusion model adaptation on both video game and real-world robotics data, showing that pretrained video diffusion models can be adapted to serve as accurate world models for decision-making.

Limitations

  • Requires a small domain-specific dataset of action-labelled videos for training the adapter, which may still be scarce in some applications
  • Relies on a pretrained video diffusion model that may not be publicly available or may have inherent biases and limitations
  • Evaluation is limited to video game and real-world robotics domains; generalization to other sequential decision-making tasks is not demonstrated

Resources

PaperCode

Tags