⌁ Awesome World Model Hub Papers · Datasets · Projects
中文 GitHub
← Back to papers

RenderWorld: World Model with Self-Supervised 3D Label

arXiv 24.9 2024 46 method, system, application

TLDR

RenderWorld uses self-supervised 3D labels and Gaussian Splatting for vision-only autonomous driving, achieving SOTA in occupancy forecasting and planning via world model.

Reasoning

The paper introduces a novel self-supervised method for generating 3D occupancy labels and leverages Gaussian Splatting for efficient scene representation, outperforming NeRF-based approaches. However, the abstract lacks explicit mention of real-world datasets or benchmarks, and the framework is narrowly focused on autonomous driving, limiting generalizability.

Read-first score

Read-first score 46, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 34.

Recency 8%
75.1

Uses a gentle age decay so recent papers surface without erasing older foundations. 2024

Topical relevance 42%
48.6

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Methodology quality 25%
40

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=none

Reproducibility 25%
38

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=code

Field roles

Candidate

Rank sensitivity

Stability: volatile; rank range: 311.

Keyword Scores

world model
10
world dynamics prediction
8
generative world model
6
model-based reinforcement learning world model
4
video world model
3
world simulator
2
interactive world model
1

Deep Analysis

Innovations

  • Self-supervised gaussian-based Img2Occ Module for generating 3D occupancy labels from 2D images
  • AM-VAE encoding air and non-air separately for fine-grained scene representation
  • Use of Gaussian Splatting for 3D scene representation and rendering 2D images, improving segmentation accuracy and reducing GPU memory compared to NeRF-based methods
  • Autoregressive world model for 4D occupancy forecasting and motion planning

Methodology

RenderWorld is a vision-only end-to-end autonomous driving framework. It first generates 3D occupancy labels using a self-supervised gaussian-based Img2Occ Module, then encodes these labels with AM-VAE (which encodes air and non-air separately), and finally uses an autoregressive world model for forecasting and planning. The framework employs Gaussian Splatting to represent 3D scenes and render 2D images.

Key Results

RenderWorld achieves state-of-the-art performance in both 4D occupancy forecasting and motion planning. It also demonstrates improved segmentation accuracy and reduced GPU memory consumption compared to NeRF-based methods.

Resources

Paper

Tags