CausalARC: Abstract Reasoning with Causal World Models

arXiv 25.9 2025 40.7 benchmark

TLDR

Introduces CausalARC, a testbed for AI reasoning using causal world models, evaluated on language models.

Reasoning

Strengths: novel integration of causal world models with abstract reasoning tasks. Weaknesses: synthetic benchmark without real-world validation; performance varies heavily across models.

Read-first score

Read-first score 40.7, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 10.

Recency 8%

86.7

Uses a gentle age decay so recent papers surface without erasing older foundations. 2025

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=benchmark,evaluation,experiment

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=none

Topical relevance 42%

14.3

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Field roles

FrontierMethodology anchor

Rank sensitivity

Stability: volatile; rank range: 366.

Keyword Scores

world model

world simulator

generative world model

interactive world model

video world model

world dynamics prediction

model-based reinforcement learning world model

Deep Analysis

Innovations

Introduces CausalARC, a testbed for AI reasoning in low-data and out-of-distribution regimes, modeled after the Abstraction and Reasoning Corpus (ARC).
Each reasoning task is sampled from a fully specified causal world model expressed as a structural causal model (SCM).
Principled data augmentations provide observational, interventional, and counterfactual feedback as few-shot in-context learning demonstrations.
Proposes four evaluation settings: abstract reasoning with test-time training, counterfactual reasoning with in-context learning, program synthesis, and causal discovery with logical reasoning.

Methodology

CausalARC constructs reasoning tasks by sampling from fully specified structural causal models (SCMs). It uses principled data augmentations to generate observational, interventional, and counterfactual feedback, which are presented as few-shot in-context learning demonstrations. The testbed is evaluated across four language model settings: abstract reasoning with test-time training, counterfactual reasoning with in-context learning, program synthesis, and causal discovery with logical reasoning.

Key Results

Within- and between-model performance varied heavily across tasks, indicating significant room for improvement in language model reasoning.

Limitations

The work is presented as a proof-of-concept, not a fully scaled benchmark.
Performance varied heavily across tasks, suggesting current models struggle with the causal reasoning demands.
Limited data and distribution shift are inherent challenges, but the paper does not demonstrate robust solutions beyond the testbed design.

Resources

Paper