RL0 - Introduction to Reinforcement Learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**Class 0: Introduction to Reinforcement Learning**\n",
    "\n",
    "Contents of the class\n",
    "1. [The mad hatter's casino](#mad)\n",
    "2. [References](#refs)\n",
    "3. [Ruining the suspense with a general definition](#def)\n",
    "4. [Examples of RL problems](#examples)\n",
    "5. [Course syllabus](#syllabus)\n",
    "6. [OpenAI Gym](#gym)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id=\"gpi\"></a>The mad hatter's casino\n",
    "\n",
    "Getting the main intuitions in 30 minutes by playing a fun game!\n",
    "<img src=\"images/madhatter.png\"></img>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id=\"refs\"></a>References\n",
    "<table>\n",
    "<tr>\n",
    "<td><img src=\"images/book_szepesvari.jpg\" style=\"width: 200px;\"></td>\n",
    "<td><b>Algorithms for Reinforcement Learning</b><br>Csaba Szepesvari<br>2010.<br>The essentials in 104 pages. A bit mathematical.<br>PDF available <a href=\"https://sites.ualberta.ca/~szepesva/RLBook.html\">here</a> (last update in 2017).</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td><img src=\"images/book_sutton.jpg\" style=\"width: 200px;\"></td>\n",
    "<td><b>Reinforcement Learning: an introduction</b><br>Richard Sutton and Andrew Barto<br>1998.<br>The Reinforcement Learning bible. Both complete and didactical.<br>2nd edition scheduled in 2018, available as an online draft.<br><a href=\"http://incompleteideas.net/book/the-book.html\">Online versions</a> of the <a href=\"http://incompleteideas.net/book/the-book-1st.html\">1st</a> and <a href=\"http://incompleteideas.net/book/the-book-2nd.html\">2nd</a> editions.</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td><img src=\"images/web_silver.png\" style=\"width: 200px;\"></td>\n",
    "<td><b>David Silver's UCL course on RL</b><br>10 video lectures + presentation PDFs.<br>2015.<br><a href=\"http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html\">Available here</a>.</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td><img src=\"images/web_sigaud.png\" style=\"width: 200px;\"></td>\n",
    "<td><b>Olivier Sigaud's Sorbonne University course on RL</b><br>8 video lectures + presentation PDFs and notebooks.<br>2018.<br><a href=\"http://pages.isir.upmc.fr/~sigaud/teach/english.html\">Available here</a>.</td>\n",
    "</tr>\n",
    "</table>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id=\"def\"></a>Ruining the suspense with a general definition\n",
    "\n",
    "What is Reinforcement Learning about?\n",
    "\n",
    "It is about controlling dynamic systems.\n",
    "<img src=\"images/dynamic.png\" style=\"width: 400px;\"></img>\n",
    "Dynamic systems? **dynamic** evolution of $s$ and $o$ under $\\pi$.\n",
    "\n",
    "Our object of study:<br>\n",
    "We want to find a control policy $\\pi$ (with $u = \\pi(o)$) such that the system $\\Sigma$ behaves as we desire."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id=\"examples\"></a>Examples of RL problems\n",
    "\n",
    "<table>\n",
    "<tr>\n",
    "  <td><img src=\"images/spiral.jpg\" style=\"width: 200px;\"></td>\n",
    "  <td>Exiting a spiral</td>\n",
    "</tr>\n",
    "<tr>\n",
    "  <td><img src=\"images/tests.jpg\" style=\"width: 200px;\"></td>\n",
    "  <td>Dynamic treatment regimes for HIV patients</td>\n",
    "</tr>\n",
    "<tr>\n",
    "  <td><img src=\"images/pend.png\" style=\"width: 200px;\"></td>\n",
    "  <td>Cart-pole balancing</td>\n",
    "</tr>\n",
    "<tr>\n",
    "  <td><img src=\"images/waiting.jpg\" style=\"width: 200px;\"></td>\n",
    "  <td>Queueing problems</td>\n",
    "</tr>\n",
    "<tr>\n",
    "  <td><img src=\"images/market.jpg\" style=\"width: 200px;\"></td>\n",
    "  <td>Portfolio management</td>\n",
    "</tr>\n",
    "<tr>\n",
    "  <td><img src=\"images/dam.jpg\" style=\"width: 200px;\"></td>\n",
    "  <td>Hydroelectric production</td>\n",
    "</tr>\n",
    "</table>\n",
    "\n",
    "But also:\n",
    "- Elevator scheduling\n",
    "- Ship steering\n",
    "- Bioreactor control\n",
    "- Aerobatics helicopter control\n",
    "- Airport departures scheduling\n",
    "- Airlines scheduling\n",
    "- Robocup soccer\n",
    "- Video game playing (Quake, CS, Starcraft...)\n",
    "- Game of Go\n",
    "- ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id=\"syllabus\"></a>Course syllabus\n",
    "\n",
    "This course is organized as separate and almost independent notebooks.\n",
    "- RL1 - Markov Decision Processes and model-based policy search\n",
    "- RL2 - Online Value Function Prediction\n",
    "- RL3 - Control Problems, model-free Policy Optimization\n",
    "- RL4 - Deep Reinforcement Learning\n",
    "- RL5 - Monte Carlo Tree Search\n",
    "\n",
    "The final evaluation is made on a challenge that will be revealed near the middle of the course, for which you will be asked to hand in your commented code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id=\"gym\"></a>OpenAI Gym\n",
    "\n",
    "This class requires a recent version of Python 3 and scikit-learn (available in the <a href=\"https://www.anaconda.com/download\">Anaconda distribution</a>).\n",
    "\n",
    "You will need standard elements of Anaconda (numpy, matplotlib, scikit-learn, scikit-image) and graphviz.\n",
    "```sh\n",
    "conda install graphviz\n",
    "conda install python-graphviz\n",
    "```\n",
    "\n",
    "It also require that you install the <a href=\"https://github.com/openai/gym\">OpenAi Gym</a> collection of Reinforcement Learning environments.\n",
    "\n",
    "Installation instructions for Gym (11/2017) - note that you can also follow the steps in the link above.\n",
    "```sh\n",
    "pip install gym[all]\n",
    "conda install libgcc\n",
    "```\n",
    "\n",
    "If needed, upgrade any previous installation:\n",
    "```sh\n",
    "conda update anaconda\n",
    "pip install gym[all] --upgrade\n",
    "conda update libgcc\n",
    "```\n",
    "\n",
    "Test your installation (if the code below runs fine, you're sorted)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This should display a 4x4 grid of letters and open a window of the Breakout game.\n",
    "# Don't close the window yourself (it shouldn't work anyway)\n",
    "import gym\n",
    "env0 = gym.make('FrozenLake-v0')\n",
    "env0.render()\n",
    "env1 = gym.make('Breakout-v0')\n",
    "env1.render()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This should close the Breakout window\n",
    "env1.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": false,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": false,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}