Professors: GERGELY NEU 

Course description
Reinforcement learning (RL) is a model-based theory of sequential decision-making under uncertainty. It is currently a dominant theoretical framework for understanding and building autonomous agents that can learn and act in the environment on their own. The objective of this course is to introduce students to the main challenges and techniques of modern RL, particularly focusing on computational aspects of dealing with the dynamic nature of the RL problem, and on the statistical challenges posed by the uncertainty of the environment. On both fronts, the goal is to provide a strong understanding of the most common methods and provide a basic algorithmic toolbox for building RL systems. The course puts a strong emphasis on the mathematical foundations of reinforcement learning, and complements it with hands-on lab sessions whose main objective is to deepen the understanding of the fundamental concepts.

Prerequisites
The students are expected to have a strong mathematical background (particularly in probability theory, linear algebra, and multivariate calculus), good programming skills, and some familiarity with machine learning in general.

Format
The majority of the material will be presented in 8 lectures which will be complemented by 2 lab sessions. The evaluation will be based on 5 problem sets handed out during the trimester and a final project which can be done in groups of 2 students.

Contents

1. An introduction to Reinforcement Learning

2. Dynamic Programming pt I

3. Dynamic Programming pt II / Lab session I

4. Basic RL algorithms: Monte Carlo and Temporal Difference Learning

5. Approximate Dynamic Programming methods

6. Policy Optimization methods

7. Deep RL / Lab session II

8. Primal-Dual methods

9. Multi-armed bandits and contextual bandits

10. Exploration and exploitation in Reinforcement Learning

Teaching Methods
The course will consist of 8 lectures exposing the foundations of reinforcement learning, which are complemented by 2 lab sessions. The lectures present the mathematical foundations of the RL problem and develop algorithms from first principles, aiming to highlight the key challenges faced in this unique learning setting and to introduce the most important solution concepts. Besides providing the theoretical  foundations, the lectures also illustrate the concepts on a number of examples and exercises. The lab sessions serve to further familiarize the students with the most important techniques by giving them hands-on experience in implementing some key RL methods.

Evaluation
Evaluation will be based on the problem sets and the final projects handed in by the students, with the final grade produced as "Final grade = 0.6 * problem sets + 0.4 * final project". To pass the course, the students need to hand in all the problem sets and a final project, and need to reach a grade of at least 5 in both parts.

Bibliography and information resources

1. R. Sutton & A. Barto (2018): Reinforcement Learning: An Introduction (2nd edition).

2. Cs. Szepesvári (2010): Algorithms for Reinforcement Learning.

3. D. Bertsekas (2017): Dynamic Programming and Optimal Control, volume 1 (4th edition).