Vanilla Policy Gradient (VPG)

1. Introduction

Vanilla Policy Gradient (VPG) is a foundational algorithm that directly optimizes a policy ${\pi }_{\theta }$ by maximizing the expected return via gradient ascent. Instead of learning a Q-function to derive a policy, VPG simply updates $\theta$ in the direction that increases the probability of rewarding trajectories.

1a. Why VPG?

• Direct approach: No need for a separate value-based step or complex argmax computations.
• Flexible: Works in continuous action spaces where Q-learning might struggle.

2. What VPG Offers

1. Simple Conceptual Flow:
   • Collect trajectories, compute returns, update the policy in proportion to how good those returns are.
2. Baseline for Variance Reduction:
   • Incorporating a value function baseline (as discussed above) can significantly reduce gradient variance.
3. Policy-Centric:
   • The algorithm focuses purely on improving ${\pi }_{\theta }$, which can be beneficial in complicated environments like Lunar Lander.

3. Algorithm Steps

Initialization:

• Policy parameters $\theta$.
• Baseline parameters (value function) $\varphi$.

Loop (until convergence):

1. Trajectory Collection: Roll out episodes $\{{\tau }_i\}$ under the current policy ${\pi }_{\theta }$.
2. Returns & Advantages: For each time step $t$, compute:
   $$R_t=\sum _{k=t}^{T-1}{\gamma }^{k-t}{r}_k,$$ $$A_t=R_t-V_{\varphi }(s_t).$$
3. Baseline Update: Fit $V_{\varphi }$ to the returns: $$\underset{\varphi }{min}\sum _t(V_{\varphi }(s_t)-R_t{)}^2.$$
4. Policy Update$${\mathrm{\nabla }}_{\theta }J(\theta )=\sum _t{\mathrm{\nabla }}_{\theta }\log {\pi }_{\theta }(a_t\mid s_t)(A_t).$$
   Perform gradient ascent on $\theta$.

4. Practical Considerations

1. High Variance: VPG can be noisy; that’s why a good baseline or advanced advantage methods are key.
2. Sample Efficiency: Must gather new trajectories from the current policy each iteration (on-policy).
3. Lunar Lander Example:
   • You gather a few episodes using the current ${\pi }_{\theta }$.
   • Many landings might fail at first, but you compute returns anyway.
   • Over time, you see improvements as the policy learns to control thrusters more precisely, using the advantage to adjust the policy parameters.

5. Summary

• VPG is a direct, relatively simple approach to on-policy RL.
• It forms the basis for many advanced methods (e.g., PPO, A2C).
• Key to success: variance-reducing measures like a well-trained value function baseline.