AI TrackerFrontier desk · AEST
Back to news
normalai

FERPO: Forward Entropy-Regularized Policy Optimization

arXiv cs.AI · 2 Oct 2026, 03:59 AEST

Several state-of-the-art methods for online reinforcement learning in continuous control improve policies using action gradients of a learned critic. However, critics are typically trained to predict returns, and accurate value predictions do not necessarily yield accurate action derivatives, potentially leading to unreliable policy updates.

Key facts

Read aloud

Several state-of-the-art methods for online reinforcement learning in continuous control improve policies using action gradients of a learned critic. However, critics are typically trained to predict returns, and accurate value predictions do not necessarily yield accurate action derivatives, potentially leading to unreliable policy updates. Several state-of-the-art methods for online reinforcement learning in continuous control improve policies using action gradients of a learned critic. However, critics are typically trained to predict returns, and accurate value predictions do not necessarily yield accurate action derivatives, potentially leading to unreliable policy updates. We propose Forward Entropy-Regularized Policy Optimization FERPO, an on-policy maximum entropy reinforcement learning algorithm that performs policy improvement using critic values without differentiating the. FERPO derives an optimal target action distribution from a policy-improvement objective regularized by entropy and Kullback-Leibler KL divergence. We then fit the actor to this target by minimizing a forward-KL objective, estimated using self-normalized importance sampling SNIS with actions drawn from the rollout policy.

Sources in this cluster