Reimagination with Test-time Observation Interventions: Distractor-Robust World Model Predictions for Visual Model Predictive Control
TLDR
World models enable robots to "imagine" future observations given current observations and planned actions, and have been increasingly adopted as generalized dynamics models to facilitate robot learning.
Reasoning
Fallback reasoning generated from available title and abstract metadata: World models enable robots to "imagine" future observations given current observations and planned actions, and have been increasingly adopted as generalized dynamics models to facilitate robot learning. Despite their promise, these models remain brittle when encountering novel visual...
Read-first score
Read-first score 45.8, weighted from topical fit, citation, graph, method, reproducibility, and recency signals.
Field roles
Rank sensitivity
Stability: volatile; rank range: 370.
Deep Analysis
Innovations
- Test-time observation intervention strategy (ReOI) that detects visual distractors by identifying elements degrading in physically implausible ways during world model prediction
- Modification of current observation to remove distractors and bring it closer to training distribution before reimagining future outcomes
- Post-hoc reintroduction of distractors to preserve visual consistency for downstream planning and verification
Methodology
ReOI operates at test time: first, it detects visual distractors by analyzing which scene elements degrade in physically implausible ways during world model prediction. Then, it modifies the current observation to remove these distractors, bringing the observation closer to the training distribution. Finally, it reimagines future outcomes using the modified observation and reintroduces the distractors post-hoc to maintain visual consistency. The approach is validated on robotic manipulation tasks in the context of action verification, where a verifier selects desired action plans based on world model predictions.
Key Results
ReOI is robust to both in-distribution and out-of-distribution visual distractors, improving task success rates by up to 3x in the presence of novel distractors compared to action verification relying on world model predictions without imagination interventions.
Limitations
- Only validated on a suite of robotic manipulation tasks in the context of action verification, so generalization to other tasks or planning paradigms is not established
- The detection of distractors relies on physical implausibility during prediction, which may not capture all types of novel distractors (e.g., those that degrade plausibly)
- Post-hoc reintroduction of distractors may introduce visual artifacts or inconsistencies that could affect downstream planning or verification