WorldSonus: Bringing Sound to Worlds
arXiv cs.AI · 7 Oct 2026, 03:50 AEST
WorldSonus is an interactive video-to-audio framework designed for real-time spatial sound synthesis in world models. It addresses challenges such as real-time generation, interactive control, and spatially aligned stereo sound.
Key facts
- Bringing sound to world models poses three core challenges: real-time generation to keep pace with interactive video streams, interactive control to respond to mid-stream sound instructions, and spatially aligned stereo to reflect scene geometry and camera motion.
- For real-time generation, WorldSonus employs a streaming causal autoregressive diffusion architecture that synthesizes audio chunks at a low real-time factor (RTF) of 0.41.
- For interactive control, we incorporate an audio-centric captioning pipeline with chunk-indexed prompt scheduling, enabling dynamic manipulation of sound events during generation.
- Extensive experiments demonstrate that while tailored for world models, WorldSonus generalizes effectively to open-domain video-to-audio benchmarks, matching or outperforming state-of-the-art bidirectional models in both acoustic quality and spatial alignment.
Read aloud
WorldSonus is an interactive video-to-audio framework designed for real-time spatial sound synthesis in world models. It addresses challenges such as real-time generation, interactive control, and spatially aligned stereo sound. Bringing sound to world models poses three core challenges: real-time generation to keep pace with interactive video streams, interactive control to respond to mid-stream sound instructions, and spatially aligned stereo to reflect scene geometry and camera motion. For real-time generation, WorldSonus employs a streaming causal autoregressive diffusion architecture that synthesizes audio chunks at a low real-time factor RTF of zero point four one. For interactive control, we incorporate an audio-centric captioning pipeline with chunk-indexed prompt scheduling, enabling dynamic manipulation of sound events during generation. Extensive experiments demonstrate that while tailored for world models, WorldSonus generalizes effectively to open-domain video-to-audio benchmarks, matching or outperforming state-of-the-art bidirectional models in both acoustic quality and spatial alignment.
Sources in this cluster
- WorldSonus: Bringing Sound to Worlds
arXiv cs.AI · 7 Oct 2026, 03:50 AEST