Pre-registered tests of solid-state-physics-inspired LLM compression: a cluster-level negative result at small-language-model scale
TLDR
An autonomous research agent pre-registered and tested five solid-state-physics-inspired LLM compression methods, finding mostly negative results and falsifying expected Kohn-nearsighted behavior.
Reasoning
The paper's strengths include rigorous pre-registration, explicit falsification criteria, and honest reporting of negative results across multiple compression mappings. Weaknesses are the small-language-model scale, many shelved or falsified pilots, and limited evidence for any positive compression outcome.
Read-first score
Read-first score 63.2, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 73.
Field roles
Rank sensitivity
Stability: volatile; rank range: 35.
Keyword Scores
Deep Analysis
Innovations
- Pre-registration + 3-sigma gate + cluster-framing + append-only-catalogue discipline for autonomous LLM compression research, with predictions committed to git before pilot data.
- Systematic testing of five solid-state-physics-inspired compression mappings organized by distance face (P001 Wannier, P002 tight-binding) and rank face (P003 DMRG-truncated MLPs, P005 Wilson-RG, P011 tensor-train embeddings) under an area-law/Kohn-nearsighted anchor.
- Four pre-registered negative results with full data release at small language-model scale.
- Empirical inversion of the pre-registered Kohn-nearsighted insulator prediction for attention heads at the tested scale, pointing instead to critical, glassy, or heavy-tailed regimes.
- Cluster-level negative-result catalogue reporting eighteen concluded studies, seventeen negative.
Methodology
A three-month autonomous research-agent program tested five solid-state-physics-inspired compression mappings on pretrained language models, with predictions committed to git before pilot data and a 3-sigma gate deciding PASS or SHELVE. The study used models such as GPT-2-medium and Pythia-160M, with metrics including perplexity, sparsity, and correlation, and baselines including PCA, random-Haar, and identity.
Key Results
P005 was pre-empted at Phase 1 and three of four Phase-3 pilots were falsified; tight-binding cutoff cost +96% perplexity on GPT-2-medium, Wannier sparsity on Pythia-160M (0.054 +/- 0.004) was indistinguishable from PCA, random-Haar, and identity baselines, and tensor-train bond dimension did not track surprisal (r = 0.016 vs pre-registered 0.65) and inflated rather than compressed. Attention-versus-distance was best fit by a stretched exponential in 12 of 16 median-layer heads once probe padding was excluded; P003 was mixed, with scaling r = -0.434 shelved, MPO premise failing at stage-0, and a cross-paper r = 0.523 collapsing to 0.047 under correction, inverting the insulator prediction at the tested scale while rank-face no-gain held to 7-8B.
Limitations
- The inversion of the Kohn-nearsighted insulator prediction is specific to the ~350M scale tested.
- The enforcement gate was designed but not deployed.
- Most findings are negative at small language-model scale, so generality beyond the tested models and scales is not established.
- P003 is mixed rather than cleanly falsified, with the scaling claim shelved and the MPO premise failing at stage-0.
- The attention-versus-distance stretched-exponential result depends on excluding probe padding, suggesting sensitivity to probe/tokenization artifacts.
- The rank-face no-gain result held to 7-8B, but other results were obtained at smaller model scale.