WM-Craftnet: World Synesthesia Model for Generalizable and Robust Dexterous In-Hand Manipulation

arXiv 2026 34.6 method, application

TLDR

WM-Craftnet uses a World Synesthesia Model as task context for robust dexterous in-hand manipulation, improving generalization and sim-to-real transfer.

Reasoning

The paper introduces a clever use of a world model as recurrent context rather than for policy optimization, with strong ablations and multi-object generalization. Weaknesses include limited abstract-level detail on real-world evaluation and no direct comparison to state-of-the-art world simulators.

Read-first score

Read-first score 34.6, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 35.

Recency 5%

100

Uses a gentle age decay so recent papers surface without erasing older foundations. 2026

Methodology quality 15%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=ablation,baseline,metric

Topical relevance 25%

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Reproducibility 15%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=code

Graph prestige 15%

5.7

PageRank-style prestige over the local citation graph built from available OpenAlex references. local_pagerank=0.0016

Citation impact 15%

Uses OpenAlex-shaped citation metadata as a bibliometric attention signal, separate from paper quality. cited_by_count=0

Citation velocity 10%

Citation velocity estimates citations per publication-year to reduce old-paper bias. velocity=0.00

Field roles

FrontierMethodology anchor

Rank sensitivity

Stability: volatile; rank range: 338.

Keyword Scores

world model

world dynamics prediction

interactive world model

generative world model

model-based reinforcement learning world model

world simulator

video world model

Deep Analysis

Innovations

Proposes WM-Craftnet, a world-model-conditioned framework using a learned World Synesthesia Model (WSM) as recurrent task context for an asymmetric actor-critic policy, rather than for latent imagination or policy optimization.
Learns compact action-conditioned latent dynamics from proprioception, depth, tactile sensing, and actions, supervised by multimodal reconstruction and reward prediction.
Trains WSM to reconstruct clean depth targets from noisy depth inputs, providing a denoised geometric state for real-robot deployment.
Uses a WSM pretrained on nine z-axis objects as a reusable prior for 49-object downstream policy learning.
Demonstrates via ablations that predictive world modeling, clean-depth supervision, and tactile contact cues shape the learned state.

Methodology

WM-Craftnet learns a World Synesthesia Model (WSM) that encodes action-conditioned latent dynamics from proprioception, depth, tactile sensing, and actions, with supervision from multimodal reconstruction and reward prediction. The WSM is used as recurrent task context for an asymmetric actor-critic policy, and a WSM pretrained on nine z-axis objects is transferred to 49-object downstream policy learning. Ablations compare recurrent baselines, auxiliary heads, tactile masking, and WSM modality heads.

Key Results

The WSM context improves multi-object rotation, and ablations show that predictive world modeling, clean-depth supervision, and tactile contact cues all contribute to the learned state. Quantitative and qualitative evidence is reported for unseen-object, perturbation-recovery, and sim-to-real transfer.

Resources

Paper