Stealthy World Model Manipulation via Data Poisoning
TLDR
Proposes SWAAP, a two-stage data poisoning framework to stealthily manipulate learned world models in model-based RL, causing performance degradation.
Reasoning
The paper introduces a novel attack framework with clear methodology and evaluation across continuous-control tasks, demonstrating effectiveness and stealthiness. However, it focuses only on fine-tuning attacks and may not address broader training scenarios or defenses comprehensively.
Read-first score
Read-first score 54.5, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 35.
Field roles
Rank sensitivity
Stability: volatile; rank range: 377.
Keyword Scores
Deep Analysis
Innovations
- First two-stage data poisoning framework for learned world models (SWAAP)
- First stage uses first-order bilevel optimization with a transition-gradient theorem to identify a harmful target world model that induces low-return behavior while staying close to clean dynamics
- Second stage employs stealth-constrained gradient matching to modify fine-tuning transition targets, with a prediction-error regularizer to keep poisoned targets close to the world model's natural approximation error
- Evaluation of attack stealthiness across three stages of the poisoning pipeline: pre-training detection, robust training during fine-tuning, and test-time monitoring of the resulting world model
Methodology
SWAAP operates in two stages: first, it identifies a harmful target world model via first-order bilevel optimization enabled by a transition-gradient theorem, ensuring the target induces low-return behavior under planning while remaining close to clean dynamics. Second, it realizes this target through stealth-constrained gradient matching, modifying only a limited fraction of fine-tuning transition targets so that training gradients steer the victim model toward the adversarial target, while a prediction-error regularizer keeps poisoned targets close to the world model's natural approximation error. The attack is evaluated on diverse continuous-control tasks against non-adaptive residual, CUSUM, and TRIM-style defenses across three detection stages.
Key Results
SWAAP causes substantial performance degradation in downstream planning while keeping poisoned transitions close to clean data and evading the evaluated non-adaptive residual/CUSUM/TRIM-style defenses, revealing a practical vulnerability in world-model adaptation pipelines.
Limitations
- Only non-adaptive defenses (residual, CUSUM, TRIM-style) were evaluated; adaptive defenses may detect the attack
- The attack assumes access to the fine-tuning data and the ability to modify transition targets, which may not always be feasible
- The framework may require knowledge of the victim's world model architecture or training procedure, though not explicitly stated
- Computational cost of the bilevel optimization in the first stage is not discussed and could be a practical limitation