Same Question, Different Answers: Auditing Persona-Dependent Variation in Six LLMs on Smoking-Cessation Advice
arXiv cs.AI · 31 Oct 2026, 03:03 AEST
A study audits six large language models (LLMs) on smoking cessation advice, revealing that responses vary significantly based on user personas. The findings highlight the need for tailored interventions to ensure consistent and appropriate health information delivery.
Key facts
- We audit six widely deployed LLMs from four providers on smoking cessation, a domain with codified clinical guidance and substantial public-health stakes, by putting the same four cessation questions to twelve user personas that differ in life stage, self-described expertise, and smoking history, for a total of 2,880 responses.
- Every model named FDA-approved prescription cessation medications less often to teen personas than to adult personas, by 32 to 67 percentage points, and in five of six models a persona presenting health-care expertise received fewer facts than a persona presenting none.
- The prompt narrows persona differences in every model, but its effect on overall fact coverage varies in both magnitude and direction across models, so a prompt validated on one system cannot be assumed to transfer to another.
Read aloud
A study audits six large language models L L M s on smoking cessation advice, revealing that responses vary significantly based on user personas. The findings highlight the need for tailored interventions to ensure consistent and appropriate health information delivery. We audit six widely deployed L L M s from four providers on smoking cessation, a domain with codified clinical guidance and substantial public-health stakes, by putting the same four cessation questions to twelve user personas that differ in life stage, self-described expertise, and smoking history, for a total of two thousand eight hundred eighty responses. Every model named FDA-approved prescription cessation medications less often to teen personas than to adult personas, by thirty two to sixty seven percentage points, and in five of six models a persona presenting health-care expertise received fewer facts than a persona presenting none. The prompt narrows persona differences in every model, but its effect on overall fact coverage varies in both magnitude and direction across models, so a prompt validated on one system cannot be assumed to transfer to another.
Sources in this cluster
- Same Question, Different Answers: Auditing Persona-Dependent Variation in Six LLMs on Smoking-Cessation Advice
arXiv cs.AI · 31 Oct 2026, 03:03 AEST