LeCun and Others Propose H-JEPA Hierarchical World Model for End-to-End Learning of Visual Planning

Golden Finance reported on October 12 that, according to a paper on arXiv, researchers Wancong Zhang, Basile Terver, Michael Rabbat, Yann LeCun, and Randall Balestriero proposed H-JEPA, a scheme for end-to-end training of a hierarchical action-conditioned JEPA world model. Each level predicts further into the future within its own learned latent space. Planning is conducted top-down: the top layer optimizes goal progression, while each layer's predictions become sub-goals for the lower layers. When factors in the data evolve at separate time scales, higher layers discard fast, unpredictable details and retain slower, task-relevant states. In four simulated navigation and manipulation environments, hierarchical planning outperformed flat JEPA; in the Visual AntMaze task, the three-layer structure increased success rates from 18% to 73% while requiring less computational power for planning. Ablation experiments showed that the gains stemmed from both temporal decomposition and high-level goal representation. Under inverse dynamics supervision, this method can be extended to real robot videos in DROID.