Fast Rates for Offline Q-Learning with KL Regularization
arXiv cs.AI · 14 Oct 2026, 03:56 AEST
This paper investigates offline Q-learning with Kullback-Leibler (KL) regularization. It proposes KL-regularized Q-Learning and KL-regularized Pessimistic Q-Learning, achieving improved sample complexity in offline decision-making problems.
Key facts
- This paper investigates offline Q-learning for finite-horizon Markov decision processes with Kullback-Leibler (KL) regularization.
- We first propose KL-regularized Q-Learning (KLQ), a simple variant of classical Q-learning.
- We prove that KLQ achieves a sample complexity of $ ilde{O}(1/ε)$ without explicit pessimism.
- To address this limitation, we further propose KL-regularized Pessimistic Q-Learning (KLPQ), which incorporates pessimism into KLQ.
- We prove that KLPQ retains the same $ ilde{O}(1/ε)$ rate while removing the explicit exponential dependence.
Read aloud
This paper investigates offline Q-learning with Kullback-Leibler KL regularization. It proposes KL-regularized Q-Learning and KL-regularized Pessimistic Q-Learning, achieving improved sample complexity in offline decision-making problems. This paper investigates offline Q-learning for finite-horizon Markov decision processes with Kullback-Leibler KL regularization. We first propose KL-regularized Q-Learning KLQ, a simple variant of classical Q-learning. We prove that KLQ achieves a sample complexity of $ ilde O one/ε $ without explicit pessimism. To address this limitation, we further propose KL-regularized Pessimistic Q-Learning KLPQ, which incorporates pessimism into KLQ. We prove that KLPQ retains the same $ ilde O one/ε $ rate while removing the explicit exponential dependence.
Sources in this cluster
- Fast Rates for Offline Q-Learning with KL Regularization
arXiv cs.AI · 14 Oct 2026, 03:56 AEST