Validity Without Ground Truth: What Stated-Preference Economics Offers the Evaluation of Language Models
arXiv cs.AI · 8 Oct 2026, 03:51 AEST
Stated-preference economics offers a framework for evaluating language models in the absence of correct answers. This approach assesses validity through concepts like reliability and incentive compatibility, demonstrating its effectiveness with a water-quality valuation survey.
Key facts
- Many of the questions now put to large language models have no correct answer to score against: what a policy is worth, which option a user should choose, how to weigh competing values.
- Stated-preference economics has faced this problem for decades.
- We demonstrate the approach using a published water-quality stated preference economic valuation survey (Vossler et al. 2023) administered to six models.
- Two older models fail the most basic test at a household income level of $75,000, and the two newest pass every test of theoretical validity we can score, but diverge on convergent validity.
Read aloud
Stated-preference economics offers a framework for evaluating language models in the absence of correct answers. This approach assesses validity through concepts like reliability and incentive compatibility, demonstrating its effectiveness with a water-quality valuation survey. Many of the questions now put to large language models have no correct answer to score against: what a policy is worth, which option a user should choose, how to weigh competing values. Stated-preference economics has faced this problem for decades. We demonstrate the approach using a published water-quality stated preference economic valuation survey Vossler et al. twenty twenty three administered to six models. Two older models fail the most basic test at a household income level of seventy five thousand dollars and the two newest pass every test of theoretical validity we can score, but diverge on convergent validity.
Sources in this cluster
- Validity Without Ground Truth: What Stated-Preference Economics Offers the Evaluation of Language Models
arXiv cs.AI · 8 Oct 2026, 03:51 AEST