Same Query, Different Answer: Run-to-Run Variance in Semantic Query Processing
arXiv cs.AI · 10 Oct 2026, 02:34 AEST
A study on semantic query processing reveals significant run-to-run variance in outputs from large language models. The research highlights that between 0.07% and 10.2% of rows can change their operator output across multiple executions, with implications for benchmarking and engine design.
Key facts
- We measure how much another execution changes.
- At temperature 0, between 0.07% and 10.2% of rows change their operator output over five executions, depending on the model, and up to 24.0% on one task.
- In SemBench's published results, 21% of text results change quality between executions, including on engines whose plans are deterministic.
- Voting reduces the variance but does not remove it, and a calibrated cascade keeps its guarantee while the rows it returns change.
Read aloud
A study on semantic query processing reveals significant run-to-run variance in outputs from large language models. The research highlights that between zero point zero seven percent and ten point two percent of rows can change their operator output across multiple executions, with implications for benchmarking and engine design. We measure how much another execution changes. At temperature zero, between zero point zero seven percent and ten point two percent of rows change their operator output over five executions, depending on the model, and up to twenty four point zero percent on one task. In SemBench's published results, twenty one percent of text results change quality between executions, including on engines whose plans are deterministic. Voting reduces the variance but does not remove it, and a calibrated cascade keeps its guarantee while the rows it returns change.
Sources in this cluster
- Same Query, Different Answer: Run-to-Run Variance in Semantic Query Processing
arXiv cs.AI · 10 Oct 2026, 02:34 AEST