HeadlinesBriefing favicon HeadlinesBriefing.com

Le Mario: JEPA on Super Mario Bros

Hacker News •
×

I rebuilt the Le Mario Joint-Embedding Predictive Architecture (JEPA) from scratch and trained it on Super Mario Bros. The model learns world dynamics from pixels and vlieg actions, originally applied to reward‑free planning in Push‑T.

The architecture compresses each frame into a 192‑dimensional latent via a vision encoder, and each 5‑frame action sequence into another 192‑vector. A six‑block transformer predicts future latents, with Adaptive Layer‑Norm‑Zero injecting action controls (shift, scale, gate). Training uses a prediction loss plus 0.1×SIGReg1 to avoid representation collapse.

Le Mario was trained on 737,134 frames from 280 episodes across 32 levels. On held‑out episodes it beat a persistence baseline: five‑step error 0.077 versus 0.142, and shuffled‑action error 0.115. The model thus captured short‑horizon Mario dynamics conditioned on player input.

For planning, I applied Cross‑Entropy Method (CEM) to search imagined futures, scoring them against a goal image. A probe recovered Mario’s horizontal position from the latent with MAE 9.30 px, enabling CEM to move Mario from x=40 to x≈72, then to x=176 with local replanning.