AI TrackerFrontier desk · AEST
Back to news
normalaitech

The Missing Minimal Pair: Stereotype Evaluation in LLMs

arXiv cs.CL · 7 Oct 2026, 03:41 AEST

A new study critiques the reliability of traditional methods for measuring bias in Large Language Models. It proposes a dual minimal pair setup and a data-augmentation framework to enhance stereotype evaluation across multiple languages.

Key facts

Read aloud

A new study critiques the reliability of traditional methods for measuring bias in Large Language Models. It proposes a dual minimal pair setup and a data-augmentation framework to enhance stereotype evaluation across multiple languages. A common approach to measuring bias in Large Language Models is to compare the log-likelihoods of two contrastive stereotype sentences. We argue that such single-pair comparisons are often unreliable: simply rewriting the same stereotype with an alternative attribute can yield logically inconsistent preferences. To address this, we propose a dual minimal pair setup that introduces two axes of comparison for robust stereotype evaluation. We apply our framework on a set of English, Russian, Spanish and Chinese stereotypes. One of these metrics provides a new perspective on bias by modeling the mutual information MI between social groups and stereotyped attributes.

Sources in this cluster