FERPO: Forward Entropy-Regularized Policy Optimization
arXiv cs.AI · 2 Oct 2026, 03:59 AEST
Several state-of-the-art methods for online reinforcement learning in continuous control improve policies using action gradients of a learned critic. However, critics are typically trained to predict returns, and accurate value predictions do not necessarily yield accurate action derivatives, potentially leading to unreliable policy updates.
Key facts
- Several state-of-the-art methods for online reinforcement learning in continuous control improve policies using action gradients of a learned critic.
- However, critics are typically trained to predict returns, and accurate value predictions do not necessarily yield accurate action derivatives, potentially leading to unreliable policy updates.
- We propose Forward Entropy-Regularized Policy Optimization (FERPO), an on-policy maximum entropy reinforcement learning algorithm that performs policy improvement using critic values without differentiating the.
- FERPO derives an optimal target action distribution from a policy-improvement objective regularized by entropy and Kullback-Leibler (KL) divergence.
- We then fit the actor to this target by minimizing a forward-KL objective, estimated using self-normalized importance sampling (SNIS) with actions drawn from the rollout policy.
Read aloud
Several state-of-the-art methods for online reinforcement learning in continuous control improve policies using action gradients of a learned critic. However, critics are typically trained to predict returns, and accurate value predictions do not necessarily yield accurate action derivatives, potentially leading to unreliable policy updates. Several state-of-the-art methods for online reinforcement learning in continuous control improve policies using action gradients of a learned critic. However, critics are typically trained to predict returns, and accurate value predictions do not necessarily yield accurate action derivatives, potentially leading to unreliable policy updates. We propose Forward Entropy-Regularized Policy Optimization FERPO, an on-policy maximum entropy reinforcement learning algorithm that performs policy improvement using critic values without differentiating the. FERPO derives an optimal target action distribution from a policy-improvement objective regularized by entropy and Kullback-Leibler KL divergence. We then fit the actor to this target by minimizing a forward-KL objective, estimated using self-normalized importance sampling SNIS with actions drawn from the rollout policy.
Sources in this cluster
- FERPO: Forward Entropy-Regularized Policy Optimization
arXiv cs.AI · 2 Oct 2026, 03:59 AEST