DreamX: Video Diffusion Model is a Universal Latent Action Learner
arXiv cs.AI · 4 Nov 2026, 04:59 AEST
DreamX is a new model that learns universal action representations from video. It effectively captures action-relevant dynamics while minimizing the influence of visual variations.
Key facts
- Learning a universal action representation from video requires modeling dynamics that are consistent across different executions of the same action yet invariant to changes in embodiment and appearance.
- We introduce DreamX, which repurposes a pretrained video diffusion model as a universal latent action learner.
- The resulting latent space serves as a shared action interface across embodiments (human and robot), environments, and action parameterizations.
- On LARY-Bench, DreamX achieves state-of-the-art performance among latent action models.
- Furthermore, this shared interface also enables policy learning in a compact latent action space, reducing the computational cost of video-diffusion-based policies.
Read aloud
DreamX is a new model that learns universal action representations from video. It effectively captures action-relevant dynamics while minimizing the influence of visual variations. Learning a universal action representation from video requires modeling dynamics that are consistent across different executions of the same action yet invariant to changes in embodiment and appearance. We introduce DreamX, which repurposes a pretrained video diffusion model as a universal latent action learner. The resulting latent space serves as a shared action interface across embodiments human and robot, environments, and action parameterizations. On LARY-Bench, DreamX achieves state-of-the-art performance among latent action models. Furthermore, this shared interface also enables policy learning in a compact latent action space, reducing the computational cost of video-diffusion-based policies.
Sources in this cluster
- DreamX: Video Diffusion Model is a Universal Latent Action Learner
arXiv cs.AI · 4 Nov 2026, 04:59 AEST