Proximal Policy Optimization

Proximal Policy Optimization (PPO)

Proximal Policy Optimization (PPO) is a state-of-the-art reinforcement-learning algorithm introduced by OpenAI in 2017. Designed as a policy gradient method, PPO strikes a balance between ease of implementation, sample efficiency, and overall training stability. It has become the default choice for various complex tasks, notably in training Large Language Models (LLMs) via reinforcement-learning-from-human-feedback (RLHF).

---

Core Concept: Policy Update Stability

In standard policy-gradient-methods, executing large updates to policy parameters ($\theta$) can lead to “performance collapse,” where the model’s behavior deteriorates significantly and fails to recover. PPO addresses this instability by constraining the step size of policy updates during each iteration, ensuring that the new policy does not deviate too far from the old policy.

---

Key Features

1. Clipped Surrogate Objective The defining feature of the standard variant (PPO-Clip) is the use of a clipped objective function. Instead of complex mathematical constraints, PPO clips the probability ratio between the new and old policy, ensuring updates remain within a safe “trust region”.

2. First-Order Optimization Unlike its predecessor, trust-region-policy-optimization (TRPO), which requires complex second-order derivative calculations (such as computing the inverse Hessian matrix), PPO achieves similar stability using simple first-order methods like Stochastic Gradient Descent (SGD) or Adam.

3. Actor-Critic Architecture PPO typically implements an actor-critic-methods framework:

   • Actor: Responsible for learning the policy (selecting actions).
   • Critic: Estimates the value function (evaluating action outcomes) to reduce the variance of policy gradient updates.

4. On-Policy Learning PPO is an on-policy algorithm, meaning it updates the policy using data collected by the most recent version of that same policy.

5. Generalized Advantage Estimation (GAE) PPO frequently leverages GAE to calculate the advantage function, optimizing the trade-off between bias and variance during training.

---

The Clipped Surrogate Objective Equation

The objective function $L^{CLIP}(\theta )$ is defined as:

$L^{CLIP}(\theta )={\hat{\mathbb{E}}}_t\left[\min \left(r_t(\theta ){\hat{A}}_t,\text{clip}(r_t(\theta ),1-ϵ,1+ϵ){\hat{A}}_t\right)\right]$

Component Breakdown:

• ${\hat{\mathbb{E}}}_t$: The expectation over a batch of sampled transitions.
• $r_t(\theta )$: The probability ratio of the action under the new policy vs. the old policy: $r_t(\theta )=\frac{{\pi }_{\theta }(a_t|s_t)}{{\pi }_{{\theta }_{\text{old}}}(a_t|s_t)}$
• ${\hat{A}}_t$: The estimated advantage function, indicating how much better the chosen action is compared to the baseline expectation in state $s_t$.
• $ϵ$: A hyperparameter (typically set to $0.1$ or $0.2$) controlling the clipping threshold.
• $\min$ function: Establishes a pessimistic lower bound, ignoring updates that would make the policy change too drastically when the reward or penalty is high.

---

Related Topics

• reinforcement-learning
• policy-gradient-methods
• trust-region-policy-optimization (TRPO)
• actor-critic-methods
• reinforcement-learning-from-human-feedback (RLHF)