LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architectures
TLDR
Proposes LLM-JEPA, a JEPA-based training objective for LLMs that outperforms standard methods across models and datasets.
Reasoning
Strengths include a novel application of JEPA to language and strong empirical results across multiple models and datasets. Weaknesses are that the paper focuses solely on language tasks and does not address world modeling or dynamics, limiting relevance to the specified keywords.
Read-first score
Read-first score 48.5, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 0.
Field roles
Rank sensitivity
Stability: volatile; rank range: 749.
Keyword Scores
Deep Analysis
Innovations
- First application of Joint Embedding Predictive Architecture (JEPA) to language models, bridging a gap between vision and language training paradigms.
- Development of a JEPA-based training objective for LLMs that works for both finetuning and pretraining, outperforming standard input-space reconstruction objectives.
Methodology
The paper proposes LLM-JEPA, which replaces the standard input-space reconstruction objective in LLM training with an embedding-space predictive objective inspired by vision JEPAs. The method is evaluated across multiple datasets (NL-RX, GSM8K, Spider, RottenTomatoes) and model families (Llama3, OpenELM, Gemma2, Olmo) by comparing performance against standard LLM training objectives.
Key Results
LLM-JEPA outperforms standard LLM training objectives by a significant margin across all tested models and datasets, while also demonstrating robustness to overfitting.
Limitations
- The results are presented as a first step and may be preliminary; the method's effectiveness on a wider range of tasks and larger scales is not yet established.
- The abstract does not specify the exact architecture or training details, leaving potential implementation challenges unaddressed.