Before You Continue: Evaluating Time-Sensitive Safety Awareness in Conversational Assistants
arXiv cs.CL · 5 Nov 2026, 03:54 AEST
Large language model assistants are increasingly used for tasks with time-dependent constraints. A new benchmark, TASC, evaluates their ability to recognize and adjust advice based on these constraints.
Key facts
- To evaluate such capability, we introduce TASC (Temporal Action Safety in Conversation), a benchmark that evaluates whether LLMs apply time-dependent constraints when responding to conversational requests.
- TASC consists of 3,418 inputs derived from 128 templates, along with a separate set of 33 scenarios designed to compare models' ability to calculate elapsed time with their ability to apply that information.
- Across 13 models, assistants account for time-dependent constraints in only 30.5% of cases where the proposed action violates a rule, while rarely warning users unnecessarily.
- More importantly, models fail to apply these constraints in 62.1% of cases despite correctly calculating the elapsed time.
Read aloud
Large language model assistants are increasingly used for tasks with time-dependent constraints. A new benchmark, TASC, evaluates their ability to recognize and adjust advice based on these constraints. To evaluate such capability, we introduce TASC Temporal Action Safety in Conversation, a benchmark that evaluates whether L L M s apply time-dependent constraints when responding to conversational requests. TASC consists of three thousand four hundred eighteen inputs derived from one hundred twenty eight templates, along with a separate set of thirty three scenarios designed to compare models' ability to calculate elapsed time with their ability to apply that information. Across thirteen models, assistants account for time-dependent constraints in only thirty point five percent of cases where the proposed action violates a rule, while rarely warning users unnecessarily. More importantly, models fail to apply these constraints in sixty two point one percent of cases despite correctly calculating the elapsed time.
Sources in this cluster
- Before You Continue: Evaluating Time-Sensitive Safety Awareness in Conversational Assistants
arXiv cs.CL · 5 Nov 2026, 03:54 AEST