The Missing Minimal Pair: Stereotype Evaluation in LLMs
arXiv cs.CL · 7 Oct 2026, 03:41 AEST
A new study critiques the reliability of traditional methods for measuring bias in Large Language Models. It proposes a dual minimal pair setup and a data-augmentation framework to enhance stereotype evaluation across multiple languages.
Key facts
- A common approach to measuring bias in Large Language Models is to compare the log-likelihoods of two contrastive stereotype sentences.
- We argue that such single-pair comparisons are often unreliable: simply rewriting the same stereotype with an alternative attribute can yield logically inconsistent preferences.
- To address this, we propose a dual minimal pair setup that introduces two axes of comparison for robust stereotype evaluation.
- We apply our framework on a set of English, Russian, Spanish and Chinese stereotypes.
- One of these metrics provides a new perspective on bias by modeling the mutual information (MI) between social groups and stereotyped attributes.
Read aloud
A new study critiques the reliability of traditional methods for measuring bias in Large Language Models. It proposes a dual minimal pair setup and a data-augmentation framework to enhance stereotype evaluation across multiple languages. A common approach to measuring bias in Large Language Models is to compare the log-likelihoods of two contrastive stereotype sentences. We argue that such single-pair comparisons are often unreliable: simply rewriting the same stereotype with an alternative attribute can yield logically inconsistent preferences. To address this, we propose a dual minimal pair setup that introduces two axes of comparison for robust stereotype evaluation. We apply our framework on a set of English, Russian, Spanish and Chinese stereotypes. One of these metrics provides a new perspective on bias by modeling the mutual information MI between social groups and stereotyped attributes.
Sources in this cluster
- The Missing Minimal Pair: Stereotype Evaluation in LLMs
arXiv cs.CL · 7 Oct 2026, 03:41 AEST