Lesson 1: Learning Parameterized Policies

Learning Objectives

Understand how to define policies as parameterized functions #
Define one class of parameterized policies based on the softmax function #
Understand the advantages of using parameterized policies over action-value based methods #

Lesson 2: Policy Gradient for Continuing Tasks

Learning Objectives

Describe the objective for policy gradient algorithms #
Describe the results of the policy gradient theorem #
Understand the importance of the policy gradient theorem #

Lesson 3: Actor-Critic for Continuing Tasks

Learning Objectives

Derive a sample-based estimate for the gradient of the average reward objective #
Describe the actor-critic algorithm for control with function approximation, for continuing tasks #

Lesson 4: Policy Parameterizations

Learning Objectives

Derive the actor-critic update for a softmax policy with linear action preferences #
Implement this algorithm #
Design concrete function approximators for an average reward actor-critic algorithm #
Analyze the performance of an average reward agent #
Derive the actor-critic update for a gaussian policy #
Apply average reward actor-critic with a gaussian policy to a particular task with continuous actions #

Discussion prompt

Are tasks really ever continuing? Everything eventually breaks or dies. It’s clear that individual people do not learn from death, but we don’t live forever. Why might the continuing problem formulation be a reasonable model for long-lived agents?

References

Sutton, R. S., and A. G. Barto. 2018. Reinforcement Learning, Second Edition: An Introduction. Adaptive Computation and Machine Learning Series. MIT Press. http://incompleteideas.net/book/RLbook2020.pdf.

Reuse

CC SA BY-NC-ND

Citation

BibTeX citation:

@online{bochman2024,
  author = {Bochman, Oren},
  title = {Policy {Gradient}},
  date = {2024-04-04},
  url = {https://orenbochman.github.io/notes/RL/c3-w4.html},
  langid = {en}
}

For attribution, please cite this work as:

Bochman, Oren. 2024. “Policy Gradient.” April 4, 2024. https://orenbochman.github.io/notes/RL/c3-w4.html.