Rephrase Before You Act: Characterizing and Mitigating Language Sensitivity in Vision-Language-Action Models
arXiv cs.CL · 8 Oct 2026, 03:57 AEST
Vision-language-action models (VLAs) are highly sensitive to instruction phrasing, lacking the robustness of their vision-language counterparts. A systematic approach to rephrasing can significantly improve performance on various tasks without the need for retraining.
Key facts
- Vision-language-action models (VLAs) are strikingly sensitive to instruction phrasing and do not inherit the language robustness of the vision-language models they are built on.
- A one-word edit can move success by tens of points: $π_{0.5}$ turns on a LIBERO stove 100% of the time for 'switch on the stove' and 2% for 'switch on the hot plate'.
- We characterize this sensitivity with statistically tested single-edit swings and an oracle phrase search, which shows that phrasing alone nearly closes the 21-point gap between in-distribution and out-of-distribution tasks.
- The rules improve the frozen $π_0$ by 16 to 27% relative on twelve held-out tasks across adversarial, VLM-generated, and human-generated phrasings, with gains concentrated on out-of-distribution tasks.
- The method requires no retraining and no per-step verification, and applies zero-shot to unseen tasks and instructions.
Read aloud
Vision-language-action models VLAs are highly sensitive to instruction phrasing, lacking the robustness of their vision-language counterparts. A systematic approach to rephrasing can significantly improve performance on various tasks without the need for retraining. Vision-language-action models VLAs are strikingly sensitive to instruction phrasing and do not inherit the language robustness of the vision-language models they are built on. A one-word edit can move success by tens of points: $π zero point five $ turns on a LIBERO stove one hundred percent of the time for 'switch on the stove' and two percent for 'switch on the hot plate'. We characterize this sensitivity with statistically tested single-edit swings and an oracle phrase search, which shows that phrasing alone nearly closes the twenty one-point gap between in-distribution and out-of-distribution tasks. The rules improve the frozen $π 0$ by sixteen to twenty seven percent relative on twelve held-out tasks across adversarial, VLM-generated, and human-generated phrasings, with gains concentrated on out-of-distribution tasks. The method requires no retraining and no per-step verification, and applies zero-shot to unseen tasks and instructions.
Sources in this cluster
- Rephrase Before You Act: Characterizing and Mitigating Language Sensitivity in Vision-Language-Action Models
arXiv cs.CL · 8 Oct 2026, 03:57 AEST