SWEET: Sparse World Modeling with Image Editing for Embodied Task Execution
TLDR
Proposes SWEET, a sparse visual world model using image editing to generate task-relevant keyframes for robot manipulation, reducing cost vs video generation.
Reasoning
Strengths include a novel sparse world modeling approach that reduces computational cost compared to dense video generation, with empirical comparisons and a mixed-training strategy to mitigate domain mismatch. Weaknesses include reliance on keyframe prediction rather than full dynamics, and limited evaluation to specific robotic datasets without broader generalization tests.
Read-first score
Read-first score 57.5, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 40.
Field roles
Rank sensitivity
Stability: volatile; rank range: 314.
Keyword Scores
Deep Analysis
Innovations
- Using image editing models as sparse world models for robot manipulation, replacing dense video generation.
- SWEET: a one-shot sparse visual planning framework that progressively generates task-relevant manipulation keyframes via successive image editing conditioned on language and optional arrow-based spatial guidance.
- Mixed-training strategy with filtered edited targets to reduce mismatch between real and edited visual subgoals.
- Goal-conditioned diffusion action predictor that converts adjacent imagined keyframes into executable action chunks.
Methodology
The paper first conducts a controlled comparison between the video generation model Wan2.2 and the image editing model FLUX-Kontext under the same robotic data setting. It then proposes SWEET, which uses successive image editing to generate a sequence of task-relevant keyframes conditioned on language instructions and optional arrow-based spatial guidance. A goal-conditioned diffusion action predictor converts adjacent keyframes into executable action chunks, and a mixed-training strategy with filtered edited targets is introduced to reduce the mismatch between real and edited subgoals. Experiments are performed on DROID and RoboMimic datasets.
Key Results
Image editing produces more reliable task-level keyframes with better visual fidelity and substantially lower inference cost than video generation. SWEET improves keyframe prediction across seen and unseen scenes and enables a full pipeline from sequential keyframe planning to executable robot actions.
Limitations
- The edited visual subgoals may not perfectly match real observations, requiring a mixed-training strategy with filtered edited targets to mitigate the mismatch.
- Evaluation is limited to DROID and RoboMimic datasets; generalization to other domains or tasks is not demonstrated.
- The framework relies on image editing models, which may have inherent failure modes such as inconsistency across successive edits.