Audio-Visual World Models: Towards Multisensory Imagination in Sight and Sound

arXiv 25.12 2025 45.9 method

TLDR

Proposes Audio-Visual World Models (AVWM) integrating binaural audio and visual dynamics, with a benchmark and a diffusion transformer model for multimodal prediction.

Reasoning

The paper introduces a novel formulation and benchmark for audio-visual world modeling, addressing a gap in multisensory simulation. Strengths include a unified framework and a new dataset; weaknesses are that the abstract cuts off before full results and practical validation details are limited.

Read-first score

Read-first score 45.9, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 55.

Recency 6%

86.7

Uses a gentle age decay so recent papers surface without erasing older foundations. 2025

Topical relevance 29%

78.6

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Methodology quality 18%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=benchmark,evaluation,experiment

Reproducibility 18%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=none

Citation impact 18%

Uses OpenAlex-shaped citation metadata as a bibliometric attention signal, separate from paper quality. cited_by_count=0

Citation velocity 12%

Citation velocity estimates citations per publication-year to reduce old-paper bias. velocity=0.00

Field roles

FrontierMethodology anchor

Rank sensitivity

Stability: volatile; rank range: 493.

Keyword Scores

world model

generative world model

world dynamics prediction

world simulator

interactive world model

video world model

model-based reinforcement learning world model

Deep Analysis

Innovations

Unified formulation of Audio-Visual World Models (AVWM) as a partially observable Markov decision process with synchronized audio-visual observations.
Construction of AVW-4k, a controlled benchmark comprising 30 hours of binaural audio-visual trajectories with action annotations across 76 indoor environments.
Proposal of AV-CDiT, an Audio-Visual Conditional Diffusion Transformer with a novel modality expert architecture and three-stage training strategy for effective multimodal integration.

Methodology

The paper presents a unified formulation for audio-visual world modeling under low-level action control, casting multimodal environment simulation as a POMDP with synchronized audio-visual observations. They construct the AVW-4k benchmark with 30 hours of binaural audio-visual trajectories and action annotations across 76 indoor environments. They propose AV-CDiT, a conditional diffusion transformer with a modality expert architecture, optimized through a three-stage training strategy for multimodal integration. Evaluation is performed on the benchmark and in embodied navigation tasks.

Key Results

AV-CDiT achieves high-fidelity multimodal prediction across visual and auditory modalities on the AVW-4k benchmark. It also improves a vision-language-model-guided agent in continuous audio-visual navigation.

Resources

Paper