Can LLMs Uncover Hidden Physical Laws through Free Exploration?
TLDR
Introduces LongChang, an automated framework with simulated altered-physics worlds to test whether LLMs can discover hidden laws through free exploration; only GPT-6-Astra achieves 16.7%.
Reasoning
The paper proposes a novel automated benchmark for LLM-driven scientific discovery, using multi-agent review and scene evolution. However, the simulated environments limit real-world applicability, and the near-zero success rates across models highlight significant limitations in autonomous hypothesis generation.
Read-first score
Read-first score 29.3, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 67.
Field roles
Rank sensitivity
Stability: volatile; rank range: 57.
Keyword Scores
Deep Analysis
Innovations
- Introduces LongChang, an automated framework that creates simulated worlds with altered physics to evaluate LLM-driven scientific discovery.
- Enables LLMs to act as explorers that formulate and investigate their own questions without predefined research tasks or hints about the altered laws.
- Uses separate LLM agents to generate and review scenes, review tool designs, and evaluate reports for correctness and experimental support against the altered physical laws and exploration records.
- Closes the exploration loop without human intervention by using exploration records to guide scene evolution for further investigation.
Methodology
LongChang creates simulated worlds where one or more physical laws differ from ordinary physics. Evaluated LLMs act as explorers, manipulating objects and building tools to investigate their own questions, while separate LLM agents generate and review scenes, review tool designs, and evaluate reports using the altered laws and exploration records. These records guide scene evolution, allowing the cycle to run without human intervention across 12 LLMs.
Key Results
Among 12 evaluated LLMs, GPT-6-Astra achieved a discovery rate of 16.7%, while the other 11 LLMs scored 0%. Exploration records revealed uninvestigated anomalies and only partial identification of the altered physical laws.
Limitations
- Only one of 12 LLMs achieved any discoveries; the other 11 scored 0%.
- Even the best model achieved only a 16.7% discovery rate.
- Exploration records showed uninvestigated anomalies.
- Models only partially identified the altered physical laws.