VocalGrad: Evaluating Acoustic Perception in Audio Language Models
arXiv cs.CL · 17 Oct 2026, 02:54 AEST
Recent research introduces VocalGrad, a benchmark for evaluating the acoustic perception capabilities of large audio language models (LALMs). The findings indicate a significant gap in performance between human annotators and LALMs in recognizing low-level acoustic variations.
Key facts
- Recent large audio language models (LALMs) have extended their capabilities beyond lexical speech understanding to tasks requiring paralinguistic and non-linguistic audio understanding, such as emotion recognition.
- To address this issue, we introduce VocalGrad, a benchmark designed to evaluate LALMs' ability to perceive low-level acoustic variations.
- Our benchmark results reveal a substantial gap in acoustic perception between human annotators and state-of-the-art LALMs: while human annotators achieve high accuracy, LALMs perform at near-chance levels.
- Our analysis reveals two distinct failure modes: for some categories, task-relevant information is preserved in internal representations but not utilized by the language model head; for others, the information becomes progressively less linearly decodable across model layers.
Read aloud
Recent research introduces VocalGrad, a benchmark for evaluating the acoustic perception capabilities of large audio language models LALMs. The findings indicate a significant gap in performance between human annotators and LALMs in recognizing low-level acoustic variations. Recent large audio language models LALMs have extended their capabilities beyond lexical speech understanding to tasks requiring paralinguistic and non-linguistic audio understanding, such as emotion recognition. To address this issue, we introduce VocalGrad, a benchmark designed to evaluate LALMs' ability to perceive low-level acoustic variations. Our benchmark results reveal a substantial gap in acoustic perception between human annotators and state-of-the-art LALMs: while human annotators achieve high accuracy, LALMs perform at near-chance levels. Our analysis reveals two distinct failure modes: for some categories, task-relevant information is preserved in internal representations but not utilized by the language model head; for others, the information becomes progressively less linearly decodable across model layers.
Sources in this cluster
- VocalGrad: Evaluating Acoustic Perception in Audio Language Models
arXiv cs.CL · 17 Oct 2026, 02:54 AEST