AutoGUIWorld: Image Generators as Visual World Models for GUI Agent

arXiv 2026 36.5 method

TLDR

AutoGUIWorld uses image generators as visual world models to synthesize GUI trajectories without running software, improving GUI agent performance on OSWorld and ScienceBoard benchmarks.

Reasoning

The paper presents a clever data generation framework that leverages image generators and planners to create diverse GUI interaction data, with strong empirical gains on real benchmarks. However, the reliance on generated trajectories may raise questions about fidelity and generalization, and the abstract does not detail limitations or failure cases.

Read-first score

Read-first score 36.5, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 48.

Recency 5%

100

Uses a gentle age decay so recent papers surface without erasing older foundations. 2026

Topical relevance 25%

68.6

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Methodology quality 15%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=evaluation,result

Reproducibility 15%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=none

Graph prestige 15%

5.7

PageRank-style prestige over the local citation graph built from available OpenAlex references. local_pagerank=0.0016

Citation impact 15%

Uses OpenAlex-shaped citation metadata as a bibliometric attention signal, separate from paper quality. cited_by_count=0

Citation velocity 10%

Citation velocity estimates citations per publication-year to reduce old-paper bias. velocity=0.00

Field roles

Frontier

Rank sensitivity

Stability: volatile; rank range: 416.

Keyword Scores

world model

generative world model

world simulator

interactive world model

world dynamics prediction

video world model

model-based reinforcement learning world model

Deep Analysis

Innovations

Introduces AutoGUIWorld, a data generation framework that synthesizes GUI interaction trajectories without deploying or running the corresponding software environments.
Combines visual priors of image generators with task knowledge of a planner, using image generators as visual world models that iteratively edit screenshots based on planned atomic actions and intended visual consequences.
Generates 79,266 spatially annotated step-level training samples across Ubuntu, Windows, macOS, and Chrome using action grounding and transition-level quality filtering.

Methodology

AutoGUIWorld samples initial GUI scenes from structured specifications of operating-system context, visual appearance, and interface state, then generates tasks conditioned on those scenes. A planner specifies atomic actions and their intended visual consequences, while an image generator iteratively edits the current screenshot to produce subsequent observations. After action grounding and transition-level quality filtering, the resulting trajectories are used to fine-tune Qwen3.5-35B-A3B, with evaluation on OSWorld and ScienceBoard.

Key Results

Fine-tuning Qwen3.5-35B-A3B on AutoGUIWorld trajectories improves mean task score on OSWorld from 33.0% to 40.8% and task success rate on ScienceBoard from 14.0% to 32.2%.

Resources

Paper