Grounding LLMs with JEPA-based world models trained in simulation — has this been tried? [D]

Excerpt

LLMs describe physics well but don't "understand" it in any grounded sense — they've learned statistical relationships between tokens like "falls" and "gravity", not actual physical intuition. This is basically the Mary's Room problem: Mary knows every physical fact about color but has never seen one. LLMs are Mary. The idea I've been thinking about: Train a JEPA-style model inside a physics simulation (think MuJoCo or a simple 2D env). Instead of predicting pixels or tokens, the model predicts