World Embedding Benchmark
arXiv cs.CL · 3 Oct 2026, 03:24 AEST
The World Embedding Benchmark introduces a dataset of 8,000 controlled simulation cases to enhance understanding of physical information in video representations. It supports tasks like text-video retrieval and highlights the trade-offs in aligning physical properties with video generation.
Key facts
- We introduce the World Embedding Benchmark, comprising 8,000 controlled simulation cases from 80 families spanning fluid mechanics, solid mechanics, dynamics, and optics & electromagnetism.
- Each case pairs a rendered video with simulation-derived physical annotations, supporting three complementary tasks: text-video retrieval, physical-property regression, and multiple-choice video-description pair classification.
- Evaluated pre-trained omnimodal embedding models show weak retrieval and near-chance within-family pair classification, while lightweight probes recover useful physical information from frozen video embeddings.
- Continual contrastive training with physics-specific video-text pairs improves retrieval and pair classification but degrades physical-property regression, revealing a trade-off between alignment and quantitative information recoverability.
Read aloud
The World Embedding Benchmark introduces a dataset of eight thousand controlled simulation cases to enhance understanding of physical information in video representations. It supports tasks like text-video retrieval and highlights the trade-offs in aligning physical properties with video generation. We introduce the World Embedding Benchmark, comprising eight thousand controlled simulation cases from eighty families spanning fluid mechanics, solid mechanics, dynamics, and optics & electromagnetism. Each case pairs a rendered video with simulation-derived physical annotations, supporting three complementary tasks: text-video retrieval, physical-property regression, and multiple-choice video-description pair classification. Evaluated pre-trained omnimodal embedding models show weak retrieval and near-chance within-family pair classification, while lightweight probes recover useful physical information from frozen video embeddings. Continual contrastive training with physics-specific video-text pairs improves retrieval and pair classification but degrades physical-property regression, revealing a trade-off between alignment and quantitative information recoverability.
Sources in this cluster
- World Embedding Benchmark
arXiv cs.CL · 3 Oct 2026, 03:24 AEST