Theory of reinforcement learning (RL), with a focus on sample complexity analyses.
Previously taught as CS598: F20, S19, F18
Canvas will be used as the main platform for announcements, discussions, and homework submissions. It is important that you join the Canvas space for this course and turn on notifications to receive announcements in a timely manner. If you enroll in the course after 08/18 or are auditing, please contact the TA (Jiawei Huang, jiaweih@illinois.edu) to add you to Canvas.
| Date | Lecture | Comments |
|---|---|---|
| 08/25 | Overview, logistics, and MDP basics | slides |
| 08/27 | MDP basics | blackboard (updated: 09/01), reading homework |
| 09/01 | Bellman equations | note1 (updated: 09/13) |
| 09/03 | Value Iteration | blackboard (updated: 09/08) |
| 09/08 | VI | |
| 09/10 | Policy Iteration | blackboard, HW2 available on Canvas |
| 09/15 | LP, concentration inequalities | blackboard, see updated note1 |
| 09/17 | concentration and union bound | blackboard, note2, reading |
| 09/22 | Certainty equivalence | blackboard (updated: 09/24), note3 |
| 09/24 | CE, abstractions | note4 |
| 09/29 | Abstractions | slides (updated: 10/01) |
| 10/01 | Abstractions | hw3 available |
| 10/06 | Fitted-Q Iteration | slides |
| 10/08 | FQI | annotated slides (updated: 10/15) |
| 10/13 | FQI proof | note5 |
| 10/15 | FQI, Importance Sampling | note6, hw3 due |
| 10/20 | IS | blackboard (updated: 10/22) |
| 10/22 | PG, MIS | Project proposal due |
| 10/27 | MIS | blackboard, hw4 available |
| 10/29 | Rmax exploration | blackboard (updated: 11/5), note7 |
| 11/03 | Cancelled | |
| 11/05 | Linear MDP | blackboard (updated: 11/10) |
| 11/10 | Linear MDP | |
| 11/12 | Bellman rank | slides (updated: 11/17) |
| 11/17 | Bellman rank | |
| 11/19 | Office hour (no class) | |
| 11/24 | Fall break (no class) | |
| 11/26 | Fall break (no class) | |
| 12/01 | Partial observability | slides |
| 12/03 | PSRs | |
| 12/08 | NeurIPS (no class) | |
| 12/10 | NeurIPS (no class) | |
| 12/12 | Project report due EOD |
Time & Location
Wed & Fri, 12:30-01:45pm. Zoom link.
Recordings will be uploaded to MediaSpace; please subscribe if you want to be notified of new video uploadings.
TA
Jiawei Huang (please contact the TA on Canvas)
Office Hours
TBA
Prerequisites
Linear algebra, probability & statistics, and basic calculus. Experience with machine learning (e.g., CS 446), and preferably reinforcement learning. It is also recommended that the students are familiar with stochastic processes and numerical analysis.
Coursework & Grading
Homework may be assigned on an ad hoc basis to help students digest particular material. The main assignment will be a course project that involves literature review, reproduction of theoretical analyses in existing work, and original research (see details below). No exams. Grades decomposition: tentatively 40% homework and 60% project; subject to +-10% changes depending on the actual number of homework assignments (expected: 3).
Topics Covered in Lectures
Course Project
You will work individually. You can choose one of the following three types of projects:
Reproduce the proofs of existing paper(s). You must fully understand the proofs and rewrite them in your own words. Sometimes a paper considers a relatively general setting and the analysis can be quite complicated. In this case you should aim at scrutinizing the results and presenting them in the cleanest possible way. Ask yourself: What’s the most essential part of the analysis? Can you introduce simplification assumptions to simplify the proofs sigificantly without trivializing the results?
Novel research Pick a new research topic and work on it. Be sure to discuss with me before you settle on the topic. The project must contain a significant theoretical component.
Something between 1 & 2 I would encourage most of you to start in this category. The idea is to reproduce the proofs of existing results and see if you can extend the analysis to a more challenging and/or interesting setting. This way, even if you do not get the new results before the end of semester, your project will just fall back to category 1.
See the link at the top of this page for potential topics. You are expected to submit a short project proposal in the middle of the semester. The proposal should consist of a short paragraph describing your project topic, the papers you plan to work on, and the original research question (if applicable).
Resources
Useful inequalities cheat sheet (by László Kozma)
Concentration of measure (by John Lafferty, Han Liu, and Larry Wasserman)
We will not follow a specific textbook, but here are some good books that you can consult:
Alekh Agarwal, Sham Kakade, Wen Sun, and I also have a draft monograph which contained some of the lecture notes from this course.
There are also many related courses whose material is available online. Here is an incomplete list (not in any particular order; list from 2019 and has not been updated since then):