Actor-Critic Methods

Actor-Critic methods represent a prominent class of algorithms in Reinforcement Learning (RL) that combine the advantages of both policy-based and value-based reinforcement learning approaches.

---

1. Core Architecture

An Actor-Critic agent is divided into two separate, cooperating components:

1. The Actor (Policy): Decides which action $a$ to take in a given state $s$. It is represented as a parameterized policy ${\pi }_{\theta }(a|s)$. The actor learns and improves the policy parameters $\theta$ to maximize expected future rewards.
2. The Critic (Value Function): Evaluates the selected action. It estimates the state-value function $V_w(s)$ or action-value function $Q_w(s,a)$ with parameters $w$. It provides a feedback signal (usually the TD error or advantage) to indicate how much better or worse the chosen action was compared to expectations.

---

2. Mathematical Framework & Learning Loop

At each time step $t$, the actor and critic interact with the environment in the following sequence:

1. Action Selection: The actor selects an action $a_t\sim {\pi }_{\theta }(a|s_t)$ based on the current state $s_t$.
2. Environment Transition: The agent executes $a_t$, transitioning to state $s_{t+1}$ and receiving reward $r_{t+1}$.
3. Critic Evaluation (TD Error): The critic computes the Temporal Difference (TD) error ${\delta }_t$, which serves as the feedback signal: ${\delta }_t=r_{t+1}+\gamma V_w(s_{t+1})-V_w(s_t)$ where $\gamma \in [0,1]$ is the discount factor.
4. Actor Update (Policy Gradient): The actor’s policy parameters $\theta$ are updated to increase the probability of actions that yield positive TD errors: $\theta \leftarrow \theta +{\alpha }_{\theta }{\delta }_t{\nabla }_{\theta }\ln {\pi }_{\theta }(a_t|s_t)$ where ${\alpha }_{\theta }$ is the actor’s learning rate.
5. Critic Update (Value Function): The critic’s value parameters $w$ are updated to minimize the value prediction error: $w\leftarrow w+{\alpha }_w{\delta }_t{\nabla }_w{V}_w(s_t)$ where ${\alpha }_w$ is the critic’s learning rate.

---

3. Key Features & Advantages

• Variance Reduction: Pure policy gradient methods (e.g., REINFORCE) suffer from high variance because policy updates rely on cumulative rewards from entire trajectories. Actor-Critic methods use the critic’s value function as a baseline or advantage function ($A(s,a)=Q(s,a)-V(s)$), drastically reducing variance and accelerating convergence.
• Continuous Action Spaces: Value-based algorithms (like Q-learning or Deep Q-Networks) require finding the maximizing action $\arg {\max }_{a}Q(s,a)$, which is computationally intractable in continuous action spaces. Actor-Critic methods circumvent this because the actor directly represents the policy distribution (e.g., outputs the mean and variance of a Gaussian distribution).
• Online and Bootstrapping Capability: Unlike Monte Carlo methods, Actor-Critic algorithms can update parameters after every single step using Temporal Difference Learning (bootstrapping), making them highly suitable for online learning and continuous environments.

---

4. Prominent Algorithms

• A2C / A3C (Advantage Actor-Critic / Asynchronous Advantage Actor-Critic): Standard actor-critic frameworks that use the advantage function to update policies. A3C parallelizes training asynchronously across multiple worker threads to break correlation and stabilize deep networks.
• DDPG (Deep Deterministic Policy Gradient): An off-policy actor-critic algorithm designed specifically for continuous action spaces, using a deterministic policy.
• PPO (Proximal Policy Optimization): An on-policy method that utilizes a clipped surrogate objective function to ensure updates do not change the policy too drastically.
• SAC (Soft Actor-Critic): An off-policy actor-critic algorithm that optimizes for maximum entropy, encouraging the agent to explore more diverse action strategies.

---

5. Related Topics

• Reinforcement Learning
• Policy Gradient Methods
• Temporal Difference Learning
• Value Function Approximation
• Markov Decision Processes

---

6. References & Sources

1. Barto, A. G., Sutton, R. S., and Anderson, C. W. (1983). Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics.
2. Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. (2016). Asynchronous Methods for Deep Reinforcement Learning. International Conference on Machine Learning (ICML).
3. Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction. MIT Press.