About

I am currently studying an MSc in ML at UCL, taking modules in bioinformatics, robotics, statistical learning theory, kernel methods and many more. Prior to this I achieved a first class Natural Sciences MSci in Mathematics and Computer Science at Durham University. Here are some of the projects that I have done in my spare time.

Contact

louis.robinson.20@ucl.ac.uk

CV

RL ⇨ Approximate Solution Methods

This second section is focused on extending the tabular methods presented in the first part, to problems with arbitrarily large/continuous state spaces (and in chapter 13, continuous action spaces). Generally we cannot find an optimal policy/optimal value function; instead we aim to find a good approximate solution. To leverage computational resources and make sensible decisions for states in such large state spaces we need to generalize between states; have some function that computes how similar states are. Supervised Learning offers many function approximate techniques.

GitHub Repo

Value-function Approximation
The Prediction Objective (VE)
Stochastic-gradient and Semi-gradient Methods
Linear Methods
Feature Construction for Linear Methods
Selecting Step-Size Parameters Manually
Nonlinear Function Approximation: Artificial Neural Networks
Least-Squares TD
Memory-based, Kernel-based Function Approximation

The most important takeaway from this chapter is understanding how we will no longer parameterise our state-value function as a table, but instead as a function. So the value function under policy \(v_{\pi}(s) \approx \hat v(s, \textbf{w})\) where \(\textbf{w}\) is a weight vector, which defines the families of functions defined by \(v\).

Value-function Approximation

The Prediction Objective (\(\overline{VE}\))

Stochastic-gradient and Semi-gradient Methods

State Aggregation on the 1000-state Random Walk:

True and approximate value functions by state aggregation on the 1000-state random walk task, using the gradient Monte Carlo algorithm.

State distribution, \(\mu\).

Linear Methods

Bootstrapping on the 1000-state Random Walk

final value function learned by the semi-gradient TD(0)

How error varies with \(\alpha\) and n

Feature Construction for Linear Methods

Polynomials

Fourier Basis

Learning curves comparing the Fourier and polynomial bases on the 1000-state random walk example. In general, we do not recommend using polynomials for online learning.

Shown are learning curves on the 1000-state random walk example for the gradient Monte Carlo algorithm with a single tiling and with multiple tilings.

Coarse Coding

Selecting Step-Size Parameters Manually

Nonlinear Function Approximation: Artificial Neural Networks

Least-Squares TD

Memory-based, Kernel-based Function Approximation

Episodic Semi-gradient Control
Semi-gradient n-step Sarsa
Average Reward: A New Problem Setting for Continuing Tasks
Deprecating the Discounted Setting
Differential Semi-gradient n-step Sarsa

In this chapter we extend the methods of chapter 9 to the control problem, with parametric approximation of the action-value function \(\hat q(s, a, \textbf{w})\), as well as extending to the continuous case (non-episodic).

Episodic Semi-gradient Control

Mountain Car Task

A semi-gradient Sarsa agents policy on the Mountain Car task after 9000 episodes of learning.

Semi-gradient Sarsa learning curves on the Mountain Car task.

Semi-gradient n-step Sarsa

Performance of one-step vs 8-step semi-gradient Sarsa on the Mountain Car task. Good step sizes were used: α=0.5/8 for n=1 and α=0.3/8 for n=8.

Effect of the α and n on early performance of n-step semi-gradient Sarsa and tile-coding function approximation on the Mountain Car task. As usual, an intermediate level of bootstrapping (n = 4) performed best.