PersonaChatBench: Evaluating Chat Assistants with Diverse Simulated Users
arXiv cs.CL · 31 Oct 2026, 03:23 AEST
PersonaChatBench is a new framework designed to evaluate chat assistants' performance with diverse simulated users. It highlights the variability in task performance and user need fulfillment across different personas and domains.
Key facts
- Conversational assistants are expected to help people with diverse backgrounds, preferences, and communication styles.
- We introduce PersonaChatBench, a framework for evaluating task competence and responsiveness to diverse user needs.
- Our results show that task performance and need fulfillment vary across personas and domains.
- Strong task performance does not necessarily translate into satisfying individual needs, and models' relative strengths in meeting those needs are not consistent across domains.
- PersonaChatBench makes these differences measurable, revealing gaps obscured by aggregate scores and helping identify which users an assistant serves well and where adaptation is needed.
Read aloud
PersonaChatBench is a new framework designed to evaluate chat assistants' performance with diverse simulated users. It highlights the variability in task performance and user need fulfillment across different personas and domains. Conversational assistants are expected to help people with diverse backgrounds, preferences, and communication styles. We introduce PersonaChatBench, a framework for evaluating task competence and responsiveness to diverse user needs. Our results show that task performance and need fulfillment vary across personas and domains. Strong task performance does not necessarily translate into satisfying individual needs, and models' relative strengths in meeting those needs are not consistent across domains. PersonaChatBench makes these differences measurable, revealing gaps obscured by aggregate scores and helping identify which users an assistant serves well and where adaptation is needed.
Sources in this cluster
- PersonaChatBench: Evaluating Chat Assistants with Diverse Simulated Users
arXiv cs.CL · 31 Oct 2026, 03:23 AEST