When Both Sides Adapt: The Challenges of Evaluating Interactive LLM Systems
arXiv cs.AI · 13 Nov 2026, 04:26 AEST
The evaluation of interactive LLM systems is complicated by user adaptivity, where users modify their prompting strategies based on model responses. This study defines two classes of estimands for comparing models and highlights the implications for decision-making and model ranking.
Key facts
- In interactive systems where users engage with LLMs, users are adaptive and, after observing LLM responses, can change their prompting strategy, as well as the type and difficulty of subsequent tasks.
- We define two classes of estimands for comparing models: the common-distribution effect, which compares model performance on the same distribution of prompts, and the induced-distribution effect, which compares each model on the prompt distribution generated through user interaction.
- We show that offline benchmark evaluation and live A/B testing target different quantities, with the standard difference-in-means estimators for the two methods converging to the common-distribution effect and the induced-distribution effect, respectively.
- This distinction can affect decision-making: the two comparisons can rank models differently and lead to discrepancies in deployment recommendations.
- Together, this work highlights the necessity of measuring both model performance as well as changes in the user behavior that the model induces.
Read aloud
The evaluation of interactive L L M systems is complicated by user adaptivity, where users modify their prompting strategies based on model responses. This study defines two classes of estimands for comparing models and highlights the implications for decision-making and model ranking. In interactive systems where users engage with L L M s, users are adaptive and, after observing L L M responses, can change their prompting strategy, as well as the type and difficulty of subsequent tasks. We define two classes of estimands for comparing models: the common-distribution effect, which compares model performance on the same distribution of prompts, and the induced-distribution effect, which compares each model on the prompt distribution generated through user interaction. We show that offline benchmark evaluation and live A/B testing target different quantities, with the standard difference-in-means estimators for the two methods converging to the common-distribution effect and the induced-distribution effect, respectively. This distinction can affect decision-making: the two comparisons can rank models differently and lead to discrepancies in deployment recommendations. Together, this work highlights the necessity of measuring both model performance as well as changes in the user behavior that the model induces.
Sources in this cluster
- When Both Sides Adapt: The Challenges of Evaluating Interactive LLM Systems
arXiv cs.AI · 13 Nov 2026, 04:26 AEST