SlowFast-VGen: Slow-Fast Learning for Action-Driven Long Video Generation

arXiv 2024 59.2 method

TLDR

SlowFast-VGen introduces dual-speed learning combining slow world dynamics and fast episodic memory for action-driven long video generation.

Reasoning

Strengths include a novel dual-speed learning framework, a large-scale dataset, and empirical improvements over baselines. Weaknesses are the narrow focus on action-driven video generation and lack of explicit interactive or RL components.

Read-first score

Read-first score 59.2, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 36.

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=baseline,dataset,experiment,metric

Recency 8%

75.1

Uses a gentle age decay so recent papers surface without erasing older foundations. 2024

Topical relevance 42%

51.4

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=dataset,github

Field roles

Methodology anchor

Rank sensitivity

Stability: volatile; rank range: 202.

Keyword Scores

world model

video world model

world dynamics prediction

generative world model

interactive world model

world simulator

model-based reinforcement learning world model

Deep Analysis

Innovations

Dual-speed learning system: Slow learning via masked conditional video diffusion model for world dynamics, and fast learning via inference-time temporal LoRA for episodic memory storage.
Slow-fast learning loop algorithm that integrates the inner fast learning loop into the outer slow learning loop, enabling recall of multi-episode experiences for context-aware skill learning.
Large-scale dataset of 200K videos with language action annotations for slow pre-training of an approximate world model.

Methodology

SlowFast-VGen combines a masked conditional video diffusion model pre-trained on a large dataset for slow learning of world dynamics with an inference-time fast learning strategy using a temporal LoRA module. The fast learning updates LoRA parameters based on local inputs and outputs to store episodic memory, and a slow-fast learning loop algorithm integrates these updates into the outer training loop for context-aware video generation.

Key Results

SlowFast-VGen achieves an FVD score of 514 (vs. 782 baseline) and maintains consistency in longer videos with an average of 0.37 scene cuts (vs. 0.89). The slow-fast learning loop also significantly improves long-horizon planning task performance.

Resources

Paper