DepthWorld: 3D World Model for Robot Manipulation
arXiv cs.AI · 7 Oct 2026, 03:59 AEST
DepthWorld introduces a 3D world model for robot manipulation that enhances traditional simulators. It combines learned stereo depth with a calibration pipeline to improve the accuracy of 3D geometry in robotics.
Key facts
- World models offer a data-driven alternative to traditional simulators for robotics, with applications spanning policy evaluation, improvement, and planning.
- Closing this gap requires progress on two fronts: large-scale 3D supervision for manipulation, and an architecture that can absorb it without disturbing strong pretrained video priors.
- Applied to the DROID dataset, this yields DROID-3D, a calibrated 3D dataset providing dense metric depth and recalibrated multi-view extrinsics (achieving <0.7 px reprojection error on 90% of episodes for external cameras).
- Depth supervision improves RGB prediction itself by +1.48 dB PSNR over an identical RGB-only baseline at equal training budget, while simultaneously yielding accurate metric depth for downstream geometric reasoning.
Read aloud
DepthWorld introduces a 3D world model for robot manipulation that enhances traditional simulators. It combines learned stereo depth with a calibration pipeline to improve the accuracy of 3D geometry in robotics. World models offer a data-driven alternative to traditional simulators for robotics, with applications spanning policy evaluation, improvement, and planning. Closing this gap requires progress on two fronts: large-scale 3D supervision for manipulation, and an architecture that can absorb it without disturbing strong pretrained video priors. Applied to the DROID dataset, this yields DROID-3D, a calibrated 3D dataset providing dense metric depth and recalibrated multi-view extrinsics achieving <zero point seven px reprojection error on ninety percent of episodes for external cameras. Depth supervision improves RGB prediction itself by +one point four eight dB PSNR over an identical RGB-only baseline at equal training budget, while simultaneously yielding accurate metric depth for downstream geometric reasoning.
Sources in this cluster
- DepthWorld: 3D World Model for Robot Manipulation
arXiv cs.AI · 7 Oct 2026, 03:59 AEST