Gradient Descent.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------------------------------\n",
    "# Gadient Descent\n",
    "> Prerequisites to increase your understanding\n",
    "> - Programming knowledge\n",
    "> - Mathematical background\n",
    "> - Jupyter notebook knowledge\n",
    "\n",
    "I will try to explain gradient descent a little in this assignment. I am going to abbreviate gradient descent as GD in this notebook now on. \n",
    "\n",
    "GD is an algorithm that is used to solve the machine learning problems such as linear regression.\n",
    "\n",
    "Simply, the goal of linear regression is to fit a line of corresponding given dataset or points. \n",
    "\n",
    "Let's consider that we want to fit a line for the points showed with the graphic below.\n",
    "\n",
    "| x (input)| y (output)|\n",
    "| :----- | :----- |\n",
    "| 10.0   | 8.04   |\n",
    "| 8.0    | 6.95   |\n",
    "| 13.0   | 7.58   |\n",
    "| 9.0    | 8.81   |\n",
    "| 11.0   | 8.33   |\n",
    "| 14.0   | 9.96   |\n",
    "| 6.0    | 7.24   |\n",
    "| 4.0    | 4.26   |\n",
    "| 12.0   | 10.84  |\n",
    "| 7.0    | 4.82   |\n",
    "| 5.0    | 5.68   |\n",
    "\n",
    "<img src=\"datasetgraph.png\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# dataset\n",
    "y = [8.04, 6.95, 7.58, 8.81, 8.33, 9.96, 7.24, 4.26, 10.84, 4.82, 5.68]\n",
    "x = [10.0, 8.00, 13.0, 9.00, 11.0, 14.0, 6.00, 4.00, 12.00, 7.00, 5.00]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can guess, we are talking about a line and a line has an equation like $y = mx + b$. Here the coefficient of $x$, $m$ is the slope of the line and $b$ is y-intersect. We can simply say that the fittest line (the solution of linear regression) is strictly related to the values of $m$ and $b$. That's why it is obvious that we have to find the best pair of $m$ and $b$ values.\n",
    "\n",
    "The common approach to finding the best values is to use a function named _error function_ or _cost function_ to measure how fit the line is.\n",
    "\n",
    "\\begin{equation*}\n",
    "Error_{m, b} = \\frac{1}{N} \\sum_{i=1}^N \\left( y_i - \\left(m x_i + b \\right) \\right)^2\n",
    "\\end{equation*}\n",
    "\n",
    "The squared error function makes us enable to calculate the error value positive and differentiate the error function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total error: 3.3391545454545453\n"
     ]
    }
   ],
   "source": [
    "def computeError(m, b):\n",
    "    # initialize total error\n",
    "    error_m_b = 0.0\n",
    "    \n",
    "    # loop within the dataset\n",
    "    for i in range(len(x)):\n",
    "        x_i = x[i]\n",
    "        y_i = y[i]\n",
    "        \n",
    "        # compute the error for a single point\n",
    "        # and add it to the total error\n",
    "        error_m_b += (y_i - (m * x_i + b)) ** 2\n",
    "    # end of for loop\n",
    "    \n",
    "    # compute the mean of the error for all dataset\n",
    "    error_m_b = error_m_b / float(len(x))\n",
    "    return error_m_b\n",
    "# end of def computeError\n",
    "\n",
    "# compute error for the line corresponded by given m and b values\n",
    "print(\"total error:\", computeError(0.9, 0.1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is simple to compute the total error for a certain line. But we need to find the direction of total error to determine what we are going to do. Are we going to reduce the slope and y-intersect or raise them? Is total error increasing or decreasing? So we have to calculate the changes of the total error. As you know this means the first derivative of the error function.\n",
    "\n",
    "<img src=\"x2graph.png\" style=\"height:300px;width:550px\"/>\n",
    "\n",
    "However, the error function has two separate parameters. Which parameter are we going to differentiate it with respect to? We are going to differentiate it with respect to two parameters separately by the partial derivative. Partial derivative means that simply we approach the other parameter (variable) as a constant (for more information [wiki](https://en.wikipedia.org/wiki/Partial_derivative)). When we differentiate the error function with respect to two separate parameters partially, we will have two equations such below.\n",
    "\n",
    "\\begin{equation}\n",
    "\\frac{\\partial E}{\\partial m} = \\frac{2}{N} \\sum_{i=1}^N -x_i \\left( y_i - \\left( m x_i + b \\right) \\right)\n",
    "\\end{equation}\n",
    "\n",
    "----------------------\n",
    "\\begin{equation}\n",
    "\\frac{\\partial E}{\\partial b} = \\frac{2}{N} \\sum_{i=1}^N - \\left( y_i - \\left( m x_i + b \\right) \\right)\n",
    "\\end{equation}\n",
    "\n",
    "Let's look at how the code of these equations looks like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gradient_m:  36.98181818181817\n",
      "gradient_b:  2.998181818181818\n"
     ]
    }
   ],
   "source": [
    "# definition of the function to compute the gradient of m.\n",
    "def gradientM(m, b):\n",
    "    # setting default value of gradient\n",
    "    gradient_m = 0.0\n",
    "    \n",
    "    # loop within the dataset\n",
    "    for i in range(len(x)):\n",
    "        x_i = x[i]\n",
    "        y_i = y[i]\n",
    "        \n",
    "        # compute the first derivative of the error funtion\n",
    "        # with respect to m\n",
    "        gradient_m += -x_i * (y_i - (m * x_i + b))\n",
    "    # end of for loop\n",
    "    \n",
    "    # mean of total error\n",
    "    gradient_m = 2 * gradient_m / float(len(x))\n",
    "    return gradient_m\n",
    "# end of def gradientM\n",
    "\n",
    "\n",
    "# definition of the function to compute the gradient of b.\n",
    "def gradientB(m, b):\n",
    "    # setting default value of gradient\n",
    "    gradient_b = 0.0\n",
    "    \n",
    "    # loop within the dataset\n",
    "    for i in range(len(x)):\n",
    "        x_i = x[i]\n",
    "        y_i = y[i]\n",
    "        \n",
    "        # compute the first derivative of the error funtion\n",
    "        # with respect to b\n",
    "        gradient_b += -(y_i - (m * x_i + b))\n",
    "    # end of for loop\n",
    "    \n",
    "    # mean of total error\n",
    "    gradient_b = 2 * gradient_b / float(len(x))\n",
    "    return gradient_b\n",
    "# end of def gradientB\n",
    "\n",
    "print(\"gradient_m: \", gradientM(1, 0))\n",
    "print(\"gradient_b: \", gradientB(1, 0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have enough information how to compute the gradient of the line and determine how fit it is. So what is next that we are going to do? We are going to use these gradient results that we computed to determine the next values of $m$ and $b$. That's why we need a new equation to determine new values of $m$ and $b$ parameters.\n",
    "\n",
    "\\begin{align}\n",
    "& m_0 = \\text{initial value} \\\\\n",
    "& \\text{for} \\; \\tau = 1, 2,\\dots \\\\\n",
    "& m_{\\tau} = m_{\\tau-1} - \\eta \\nabla E(m_{\\tau-1})\n",
    "\\end{align}\n",
    "\n",
    "-------------------------------\n",
    "\\begin{align}\n",
    "& b_0 = \\text{initial value} \\\\\n",
    "& \\text{for} \\; \\tau = 1, 2,\\dots \\\\\n",
    "& b_{\\tau} = b_{\\tau-1} - \\eta \\nabla E(b_{\\tau-1})\n",
    "\\end{align}\n",
    "\n",
    "* $\\eta$: Learning rate is a parameter of the algorithm\n",
    "* $m$ and $b$: the variable is the parameters of the Model. _They represent scalar variables._\n",
    "* $y$: targets (outputs)\n",
    "* $x$: inputs\n",
    "\n",
    "Yes, above you can see the calculation methods for new $m$ and $b$ values. As you know $\\nabla E(b_{\\tau-1})$ and $\\nabla E(m_{\\tau-1})$ mean derivative. Thus, we can write these equations like;\n",
    "\n",
    "\\begin{align}\n",
    "\\nabla E(m_{\\tau-1}) = \\frac{2}{N} \\sum_{i=1}^N -x_i \\left( y_i - \\left( m_{\\tau-1} x_i + b_{\\tau-1} \\right) \\right)\n",
    "\\end{align}\n",
    "\n",
    "-------------------------------\n",
    "\\begin{align}\n",
    "\\nabla E(b_{\\tau-1}) = \\frac{2}{N} \\sum_{i=1}^N - \\left( y_i - \\left( m_{\\tau-1} x_i + b_{\\tau-1} \\right) \\right)\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 \t error: 60.01569999999999\n",
      "10 \t error: 3.8288460786805696\n",
      "20 \t error: 2.159837161759056\n",
      "30 \t error: 2.0758079629206465\n",
      "40 \t error: 2.039328445552182\n",
      "50 \t error: 2.005681412537383\n",
      "60 \t error: 1.9735062215911647\n",
      "70 \t error: 1.942704101310751\n",
      "80 \t error: 1.9132154562486745\n",
      "90 \t error: 1.884984247675222\n",
      "100 \t error: 1.8579568559211126\n"
     ]
    }
   ],
   "source": [
    "def gradientStep(eta, m, b):\n",
    "    # compute the derivate with respect to m and b\n",
    "    dM = gradientM(m, b)\n",
    "    dB = gradientB(m, b)\n",
    "    \n",
    "    # compute new value of m and b\n",
    "    m = m - eta * dM\n",
    "    b = b - eta * dB\n",
    "    \n",
    "    return m, b\n",
    "# end of def gradientStep\n",
    "\n",
    "# iteration count\n",
    "N = 101\n",
    "\n",
    "# learning rate\n",
    "eta = 0.01\n",
    "\n",
    "# initial values\n",
    "m = b = 0\n",
    "\n",
    "# Let's start iterating\n",
    "for i in range(N):\n",
    "    if i % 10 == 0:\n",
    "        print(i, \"\\t error:\", computeError(m, b))\n",
    "    m, b = gradientStep(eta, m, b)\n",
    "# end of for loop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see that we accomplished finding the fittest line out. Let's how it looks like :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parameters of the fittest line:\n",
      "m: 0.7327729535801762\n",
      "b: 0.6502302349598857\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEICAYAAABPgw/pAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VOXd/vHPl90AAgqoqMmg4I64xA03MKgsrq11KbYu\nbWOt1lqrFowiWINptS0ufaw87hqtv8fd4gLGBeqCgorBFZckAiJQJYoBCcn390eGIRMCWWY5s1zv\n14tXuO9MZq6h9uLmzDn3MXdHRETSX4egA4iISHyo0EVEMoQKXUQkQ6jQRUQyhApdRCRDqNBFRDKE\nCl1SmpnlmtkqM+sYdJbNMbPDzeyjoHNIdlOhS0owswozWx0u7/W/Brh7lbv3cPe68ONeMrNfNvlZ\nN7NBccgwyczub0XOkU3n3X22u+8aawaRWHQKOoBII8e7+/NBhxBJV1qhS0ozs1B4Bd7JzIqBw4Fb\nwiv4W8xsVvih88Nzp4V/7jgze8fMVprZq2a2d6Pn/KOZLTaz78zsIzMrMLNRwBXAaeHnmd/GnMPN\nbFGjcYWZXWpm75pZtZk9ZGbdGn1/k/lE2ksrdEkb7l5kZocC97v77evnzcyBoe7+SXi8H3AncDww\nFzgTeNLMdgVCwIXAAe6+xMxCQEd3/9TMpgCD3P3MOEU+FRgFrAFeAc4G/rm5fO7+Q5xeW7KQVuiS\nSh4Pr1hXmtnjMTzPr4Db3H2Ou9e5+z3AD8DBQB3QFdjDzDq7e4W7fxqH7M25yd2XuPvXwFPAPq3I\nJ9JuKnRJJSe5e+/wr5NieJ484A+N/nJYCewIDAiv4i8GJgHLzOxfZjYg9ujNWtro9zVAj5byJSiH\nZAkVuqSb1mwP+gVQ3Ogvh97unuPuDwK4+wPufhgNxerAn9vw3PGw2Xwi7aVCl3TzFbBTC3P/C/za\nzA6yBt3NbKyZ9TSzXc3sKDPrSsOx7dU0HIZZ/zwhM2vp/xedzaxbo19t/Sxqk/na+DwiUVTokm5u\nBE4xs2/M7Kbw3CTgnvDhi1PdfS4Nx6lvAb4BPqHhA0loOH5eAqyg4ZBIfxrObgH4v/DX/5rZW5vJ\n8DQNfxGs/zWpLW+ghXwi7Wa6wYWISGbQCl1EJEOo0EVEMoQKXUQkQ6jQRUQyRFIv/e/bt6+HQqFk\nvqSISNqbN2/eCnfv19LjklrooVCIuXPnJvMlRUTSnplVtuZxOuQiIpIhVOgiIhlChS4ikiFaLHQz\nu9PMlpnZgkZzPzGz98ys3szyExtRRERaozUr9Ltp2KS/sQXAj4BZGz1aREQC0eJZLu4+K3xXl8Zz\nHwCYWWJSiYhImyX8GLqZFZrZXDObu3z58kS/nIhI1kp4obv7NHfPd/f8fv1aPC9eRDJYaXkpoakh\nOkzuQGhqiNLy0qAjJdxny1dxc9lCauvqE/5aukm0iCRFaXkphU8VUlNbA0BldSWFTxUCMG7IuCCj\nJYS7c8EDb/F0ecOdCE/eb3t26JOT0NfUaYsikhRFZUWRMl+vpraGorKigBIlTvmiagZOeDpS5n8/\nbWjCyxxasUI3sweB4UBfM1sEXA18DdwM9AOmm9k77n5sIoOKSHqrqq5q03w6qq93fnLba8yr/AaA\nvj268sr4EXTt1DEpr9+as1zO2MS3HotzFhHJYLm9cqms3nhLktxeuQGkib9XPlnBuNvnRMZ3n3MA\nw3ftn9QMOoYuIklRXFAcdQwdIKdzDsUFxQGmil1tXT3Dr3+JxStXA7DngC158sLD6Ngh+ad1q9BF\nJCnWf/BZVFZEVXUVub1yKS4oTusPRKe/+yUXPLDhfuKP/mYY++X2CSxPUm8SnZ+f79o+V0TSXc3a\ndQydPIPauob+PGq3/txxVn7CLrY0s3nu3uI2K1qhi4i0wX2vV3LV45GtrZj5+yMYvE3PABNtoEIX\nEWmFb75fy75/mhkZn37AjpT8eO8AE21MhS4i0oKpz3/M1OcXRsavjD+K7XtvEWCi5qnQRUQ2YcnK\n1QwreSEyvqhgMJccvUuAiTZPhS4i0owrHy/n/tc3XPT01lVHs1X3LgEmapkKXUSkkU+WfcfIv224\n1cPkE/bkrGGh4AK1gQpdRISGzbQK75vHzPe/AsAMFkw6lu5d06cm0yepiEiCvPPFSk76xyuR8U1n\n7MsJQwcEmKh9VOgikrXq6p2T/vEK5YurAdiuVzdevmwEXTql50a0KnQRyUovf7ycs+58IzK+7xcH\ncvjg9L4JjwpdRLLK2nX1HPbnF1j23Q8A7LNjbx49fxgdAthMK95U6CKSNZ6cv4SLHnw7Mn78gkPZ\nZ8feASaKLxW6iGS8Zd+t4cDissj42D234Z9n7p+wzbSCokIXkYx21A0v8dmK7yPj5y85kkH9ewSY\nKHFU6CKSkZpeIARQUTI2oDTJ0Zp7it4JHAcsc/e9wnNbAQ8BIaACONXdv0lcTBGR1guNnx41DvrG\nE8nSmpMt7wZGNZkbD5S5+2CgLDwWEQnUG59/HVXmnToYFSVjAy3z0vJSQlNDdJjcgdDUEKXlpQl7\nrdbcJHqWmYWaTJ8IDA///h7gJeCPccwlItImTVflL106nFDf7gGlaVBaXhp1H9XK6koKnyoESMit\n99p7OdQ27v4lQPjrJm9tbWaFZjbXzOYuX768nS8nItK86e9+GVXme2y3JRUlYwMvc2i4f2rjm2ID\n1NTWUFRWlJDXS/iHou4+DZgGDfcUTfTriUh2cHcGTng6am7elSPZukfXgBJtrKq6qk3zsWrvCv0r\nM9sOIPx1WfwiiYhs3u2zP4sq8+P23o6KkrEpVeYAub1y2zQfq/au0J8EzgJKwl+fiFsiEZFNWLuu\nnl2ufCZq7oNrRrFFl44BJdq84oLiqGPoADmdcyguKE7I67W4QjezB4HXgF3NbJGZ/YKGIj/azBYC\nR4fHIiIJM/GJBVFlfuGIQVSUjE3ZMoeGDz6nHT+NvF55GEZerzymHT8tIR+IAph78g5r5+fn+9y5\nc5P2eiKS/r5bU8uQSTOi5j6dMoaOGbCZVmuZ2Tx3z2/pcbpSVERS1s/umMPshSsi4+t+NIQzDkzM\n8edMoEIXkZSztHoNB19XFjX3+XVjMm4zrXhToYtIShl2XRlLqtdExnedfQAjdtvkpS7SiApdRFLC\nh0u/ZdTU2VFzmb6ZVryp0EUkcE0v23/qwsMYskOvgNKkLxW6iATm1U9W8NPb50TGPbt2onzysQEm\nSm8qdBEJRNNV+ezLR7DjVjkBpckMKnQRSarH317MxQ+9Exnvl9ubR39zaICJMocKXUSSoq7e2fmK\n6M203pl4NL1zugSUKPOo0EUk4c6+6w1e+mjD9tn5eX14+PxhASbKTCp0EUmYmrXr2GPic1Fz700+\nlu5dVT2JoD9VEUmI/GtnsmLV2sj4sEF9uf+XBwWYKPOp0EUkrpZ9u4YDp0Rftp9tm2kFRYUuInHT\n9FTEXxw2kKuO2yOgNNlHhS4iMfvgy28ZfaMu2w+aCl1EYtJ0VV588l6MOygvoDTZTYUuIu3y4kfL\nOOeuN6PmtCoPlgpdRNqs6ar83nMP5Ihd+gWURtZr8Z6im2NmvzOzBWb2npldHK9QIpKa7n2tYqMy\nrygZqzJPEe1eoZvZXsCvgAOBtcCzZjbd3RfGK5yIpI6mRT7j90ewyzY9A0ojzYnlkMvuwOvuXgNg\nZi8DJwN/iUcwEUkNk558j7tfrYia07Hy1BRLoS8Ais1sa2A1MAaY2/RBZlYIFALk5urmriLpYl1d\nPYOKnomae7NoJP16dg0okbSk3YXu7h+Y2Z+BmcAqYD6wrpnHTQOmAeTn53t7X09Ekue0215jzudf\nR8bbbtmN168oCDCRtEZMH4q6+x3uvp+7HwF8Dej4uaSV0vJSQlNDdJjcgdDUEKXlpUFHCtSqH9YR\nGj89qsw/uGaUyjxNxHTaopn1d/dlZpYL/Ag4JD6xRBKvtLyUwqcKqamtAaCyupLCpwoBGDdkXJDR\nArH7Vc+yurYuMh65+zbcflZ+gImkrWI9D/2R8DH0WuACd/8mDplEkqKorChS5uvV1NZQVFaUVYW+\nZOVqhpW8EDX32ZQxdNBmWmknpkJ398PjFUQk2aqqq9o0n4manop4wYiduezY3QJKI7HSlaKStXJ7\n5VJZXdnsfKYrX1TN8bf8J2pOpyKmPxW6ZK3iguKoY+gAOZ1zKC4oDjBV4jVdld/wk6Gcsv8OAaWR\neFKhS9Zaf5y8qKyIquoqcnvlUlxQnLHHz2e8t5TC++ZFzWlVnlnMPXmnhufn5/vcuRtdeyQiCdZ0\nVf7grw7mkJ23DiiNtJWZzXP3Fk850gpdJIPdPvszrp3+QdScVuWZS4UukoHcnYETno6aK/vDkezc\nr0dAiSQZVOgiGWb8I+/yrze/iJrTqjw7qNBFMkRtXT2Dm2ym9dZVR7NV9y4BJZJkU6GLZIATbvkP\n7y6qjox36tedF/4wPLhAEggVukgaq15dy9DJM6LmPvzTKLp17rjJnyktL82aUzWzjQpdJE01PRXx\n+KEDuPmMfTf7M9qQLLPFtH2uiCTfF1/XbFTmn183psUyh81vSCbpTyt0kTTStMgvOXoXLioY3Oqf\n14ZkmU2FLpIG5lV+w49vfTVqrj2nImbzhmTZQIUukuKarspvPH0fTtxn+3Y9V7ZuSJYtVOgiKerf\n7y7hwgfejpqL9QKhbNuQLNtocy6RFNR0Vf7I+Yewf95WAaWRoGlzLpEU19z54CtXHMT1z30U9Thd\nti+tFetNon8P/BJwoBw4x93XxCOYSCbb6HzwlZUUlfYGNpT5y5cNJ2/r7gEllHTU7vPQzWx74CIg\n3933AjoCp8crmEgma3w+eN+1l5K35t9R368oGasylzaL9ZBLJ2ALM6sFcoAlsUcSyXxV1VXgnclb\n81jU/BfdTqdu0ncBpZJ01+5Cd/fFZnYDUAWsBma4+4ymjzOzQqAQIDdX57qKAOSufhToHBn/YJ+w\ntNvF5PXKCy6UpL1YDrn0AU4EBgIDgO5mdmbTx7n7NHfPd/f8fv36tT+pSAZYseqH8BksG8q8sttJ\nLO12sc4Hl5jFcshlJPC5uy8HMLNHgWHA/fEIJpJpmp6KOGi7Wj7xi7HqOnJ75el8cIlZLIVeBRxs\nZjk0HHIpAHSSuUgTHy39jmOnzoqa+/y6MZgZcFIwoSQjxXIMfY6ZPQy8BawD3gamxSuYSCZouio/\n78idmDB694DSSKaL6SwXd78auDpOWUQyxosfLeOcu96MmtMFQpJoulJUJM6arspv+MlQTtl/h4DS\nSDZRoYvEyd2vfM6kp96PmtOqXJJJhS4SB01X5f/vvEM4cKA200oV2XIfVRW6SAyueKycB+ZE3+1H\nq/LUkk33UVWhi7RDfb2z0xVPR81pM63UtLn7qKrQRbLc8Tf/h/LF1VFzWpWnrmy6j6oKXaSVVq+t\nY/eJz0bNzb/6GHpt0XkTPyGpIJvuo9ruvVxEsklo/PSoMt+uVzcqSsaqzNNAcUExOZ1zouYydd8c\nrdBFNuOrb9dw0JSyqLmFxaPp3FFroXSRTfdR1T1FRTah6amIx+65Dbf9rMXbOorEne4pKtJOCxZX\nc9zN/4ma27CZlkjq0r8bU1BpeSmhqSE6TO5AaGqI0vLSoCNljdD46VFlfvHIwVSUjFWZS1rQCj3F\nZNNFEKnkufeWct5986LmdCqipBsVeorJposgUkXTY+U3n7Evxw8dEFAakfZToaeYbLoIImhPl3/J\nb0rfiprTqlzSmQo9xWTTRRBBaroqf/6SIxnUv0dAaUTiQx+KpphsuggiCNNmfRpV5t06d6CiZKzK\nXDKCVugpJpsugkim5jbTmnvlSPr26BpQIpH4a/eFRWa2K/BQo6mdgInuPnVTP6MLiyQIEx4t58E3\nNnwGsX9eHx45f1iAiUTaJuEXFrn7R8A+4RfrCCwGHmvv80l2S8QNCJrbTOvDP42iW+eOMT2vSKqK\n1yGXAuBTd9/40zyRFiTi3Puf/PNV3qz4JjI+48Adue5He8ceViSFxWUvFzO7E3jL3W9p5nuFQCFA\nbm7u/pWV6nyJFpoaavbMnrxeeVRcXNGm51qx6gfyr30+au6zKWPo0EFXekr6StpeLmbWBTgBmNDc\n9919GjANGo6hx/p6knnide797lc9y+raush4wujdOO/InWPKJpJO4nHIZTQNq/Ov4vBckoViPff+\n0+WrKPjry1FzukBIslE8Cv0M4ME4PI9kqeKC4qhj6ND6c++bXiD0P+P2Y8yQ7eKeUSQdxFToZpYD\nHA2cF584ko3ac+79nM/+y2nTXo+a06pcsp1ucCFpp+mq/JHzh7F/Xp+A0ogknm5wIRnniXcW87t/\nvRM1p1W5yAYqdEl57s7ACdGX7b982XDytu4eUCKR1KRCl5R2c9lC/jrz48h46+5dmHfV0QEmEkld\nKnRJSXX1zs5NNtN6+6qj6dO9S0CJRFKfCl1SziUPvcOjby+OjA8b1Jf7f3lQgIlE0oMKXVJGzdp1\n7DHxuag5baYl0noqdEkJt7ywkBtmbDhWfvawEJNO2DPARCLpR4Uugfp2TS17T5oRNafNtETaR4Uu\ngWl644kHfnkQwwb1DTCRSHpToUvSffXtGg6aUhYZ987pzDsTjwkwkUhmUKFLUv3sjjnMXrgiMn76\nosPZY8CWASYSyRwqdEmKT5Z9x8i/zYqM983tzWO/OTTARCKZR4UuCXfk9S9S+d8NW+POvnwEO26V\nE2AikcykQpeEmVf5NT++9bXI+IShA7jpjH0DTCSS2VToEnfNbaaly/ZFEk+FLnE18/2v+NW9G/a8\nP3/4zvxx1G4BJhLJHip0iYvmNtP64JpRbNFFl+2LJIsKXWL2wJwqrnisPDK+5sQ9+fkhoeACiWSp\nWO8p2hu4HdgLcOBcd39t8z8lmWJNbR27XfVs1NwnxaPp1LFDQIlEslusK/QbgWfd/RQz6wLoXLQs\n8bcZH3HTC59ExreO24/RQ7YLMJGItLvQzWxL4AjgbAB3XwusjU8sSVXVNbUMvSZ6M63PrxuDmTbT\nEglaLCv0nYDlwF1mNhSYB/zO3b9v/CAzKwQKAXJzc2N4OQnapf83n4fnLYqMHyo8mIN22jrARCLS\nWCwHOzsB+wG3uvu+wPfA+KYPcvdp7p7v7vn9+vWL4eUkKEtWriY0fnqkzLfZsisVJWNV5iIpJpYV\n+iJgkbvPCY8fpplCl/R22m2vMefzryPj5y4+gl237RlgIhHZlHYXursvNbMvzGxXd/8IKADej180\nCdLHX33HMX/fsJnWQQO34qHzDgkwkYi0JNazXH4LlIbPcPkMOCf2SBK0g6eUsfTbNZHxq+OPYkDv\nLQJMJCKtEdMJw+7+Tvj4+N7ufpK7fxOvYKmgtLyU0NQQHSZ3IDQ1RGl5adCREmrOZ/8lNH56pMxP\n2X8HKkrGqsxF0oSuFN2E0vJSCp8qpKa2YdvXyupKCp8qBGDckHFBRou75jbTmj/xGHrldA4okYi0\nhy7p24SisqJIma9XU1tDUVlRQIkS45nyL6PK/KKCwVSUjFWZi6QhrdA3oaq6qk3z6WZdXT2Dip6J\nmvvwT6Po1lmbaYmkK63QNyG3V/MXQW1qPp3c+1pFVJlPOXkIFSVjVeYiaU4r9E0oLiiOOoYOkNM5\nh+KC4gBTxWb12jp2nxi9mdanU8bQsYMu2xfJBCr0TVj/wWdRWRFV1VXk9sqluKA4bT8QLXnmQ/75\n8qeR8bSf7c8xe24bYCIRiTdz96S9WH5+vs+dO7flB0rcfPP9Wvb908yoOW2mJZJezGyeu+e39Dit\n0DPYRQ++zZPzl0TGj5w/jP3z+gSYSEQSSYWegb74uobD//JiZJy7VQ6zLh8RYCIRSQYVeoY56R+v\n8M4XKyPj5y85kkH9ewSYSESSRYWeId5f8i1jbpodGR8+uC/3/eKgABOJSLKp0DPAPtfMYGVNbWQ8\n54oCttmyW4CJRCQIKvQ09sonKxh3+5zI+KcH5TLl5CEBJhKRIKnQ01Bzm2m9O+kYtuym/VdEspkK\nPc08OX8JFz34dmR86TG7cOFRgwNMJCKpQoWeJmrr6hncZDOtj64dRddO2n9FRBqo0NPA7bM/49rp\nH0TGfzllb07N3zHARCKSimIqdDOrAL4D6oB1rbk0VVrv+x/WsefVz0XNfTZlDB20mZaINCMeK/QR\n7r4iDs8jjVz77/e5/T+fR8Z3nX0AI3brH2AiEUl1OuSSYlas+oH8a5+PjDt3ND6+drQ20xKRFsVa\n6A7MMDMHbnP3aXHIlLXOv38ezyxYGhk/fsGh7LNj7wATiUg6ibXQD3X3JWbWH5hpZh+6+6zGDzCz\nQqAQIDc3/e/2kwiV//2eI69/KTIe3L8HMy85MrhAIpKWYip0d18S/rrMzB4DDgRmNXnMNGAaNOyH\nHsvrZaLRN87mgy+/jYxfvHQ4A/t2DzCRiKSrdt9T1My6m1nP9b8HjgEWxCtYpitfVE1o/PRImY/c\nvT8VJWMDLfPS8lJCU0N0mNyB0NQQpeWlgWURkbaLZYW+DfBY+MO6TsAD7v7s5n9EAHa/6llW19ZF\nxm8UFdC/Z7CbaZWWl0bdQ7WyupLCpwoB0va2eyLZRregS6JZHy/n53e+ERmfdUgek0/cK8BEG4Sm\nhqisrtxoPq9XHhUXVyQ/kIhE6BZ0KaS+3tnpiujNtBZMPpYeXVPnj7+quqpN8yKSetp9DF1a59G3\nFkWV+fjRu1FRMjalyhwgt1fzZyBtal5EUk9qtUoGWbuunl2ujN5M6+NrR9OlU2r+HVpcUBx1DB0g\np3MOxQXFAaYSkbZQoSfArS99yp+f/TAy/vtpQzl53x0CTNSy9R98FpUVUVVdRW6vXIoLivWBqEga\n0YeicbTqh3Xspc20RCTO9KFokl39xALueW3DWSL3nnsgR+zSL8BEIpJtVOgxWvbdGg4sLouMc7p0\n5P1rRgWYSESylQo9Bufe/SYvfLgsMv73bw9jr+17BZhIRLKZCr0dPlu+iqP++nJkvOeALZl+0eEB\nJhIRUaG32ci/vcwny1ZFxrMuG0Hu1jkBJhIRaaBCb6W3q77h5P95NTIevde23Hrm/gEmEhGJpkJv\ngbszqOgZ6uo3nN4578qRbN2ja4CpREQ2lpqXLTYS5JauL360jIETno6U+a8OH0hFyViVuYikpJRe\noQe1pWt9vXPqba8xt/KbyNz71xxLTpeU/uMSkSyX0iv0orKiqL1FAGpqaygqK0rYa7766Qp2uuLp\nSJn/46f7UVEyVmUuIikvpVsqmVu61tbVc9RfX+KLr1cDsNu2PZl+0eF01GX7IpImUrrQc3vlNnvT\nhXhv6fpM+ZecX/pWZPzwrw8hP7RVXF9DRCTRUrrQE72l6+q1dQy9ZgZr19UDcOQu/bj7nAMI31ZP\nRCStxFzoZtYRmAssdvfjYo+0QSK3dH1gThVXPFYeGT938RHsum3PmJ9XRCQo8Vih/w74ANgyDs+1\nkXFDxsX1jJaVNWvZ55qZkfGp+Tvwl1OGxu35RUSCElOhm9kOwFigGLgkLokS6Kayhfxt5seR8ezL\nR7DjVrpsX0QyQ6wr9KnA5UBKH6tYWr2Gg6/bsMXtBSN25rJjdwswkYhI/LW70M3sOGCZu88zs+Gb\neVwhUAiQm5v8Gw5PfGIB9za68YQu2xeRTBXLCv1Q4AQzGwN0A7Y0s/vd/czGD3L3acA0aLgFXQyv\n1yafLl9FQaMtbicetwfnHjYwWS8vIpJ07S50d58ATAAIr9AvbVrmQXB3fn3/PJ5776vI3ILJx9Kj\na0qfoSkiErOMarn5X6zkxH+8EhnfePo+nLjP9gEmEhFJnrgUuru/BLwUj+dqj/p65+RbX2X+FysB\n6N+zK7P/OIKunToGFUlEJOnSfoU+e+FyfnbHG5Hx3eccwPBd+weYSEQkGGlb6GvX1XPk9S/yZfUa\nAIZs34vHLzhUm2mJSNZKy0J/av4Sfvvg25Hxo78Zxn65fQJMJCISvLQq9O9/WMeQSc+x/m5wI3fv\nz//+PF+baYmIkEaFfu9rFUx84r3I+PlLjmBQ/5S+QFVEJKnSotAferMqUuZnHJjLdT8aEnAiEZHU\nkxaFvss2Pdk/rw83n7EvA3pvEXQcEZGUlBaFvm9uHx45f1jQMUREUlpK3yRaRERaT4UuIpIhVOgi\nIhlChS4ikiFU6CIiGUKFLiKSIVToIiIZQoUuIpIhzD1pt/nEzJYDlS0+sHl9gRVxjJMO9J6zg95z\ndojlPee5e7+WHpTUQo+Fmc119/ygcyST3nN20HvODsl4zzrkIiKSIVToIiIZIp0KfVrQAQKg95wd\n9J6zQ8Lfc9ocQxcRkc1LpxW6iIhshgpdRCRDpEWhm1lHM3vbzP4ddJZkMbPeZvawmX1oZh+Y2SFB\nZ0okM/u9mb1nZgvM7EEz6xZ0pkQwszvNbJmZLWg0t5WZzTSzheGvfYLMGE+beL/Xh/+7ftfMHjOz\n3kFmjLfm3nOj711qZm5mfRPx2mlR6MDvgA+CDpFkNwLPuvtuwFAy+P2b2fbARUC+u+8FdARODzZV\nwtwNjGoyNx4oc/fBQFl4nCnuZuP3OxPYy933Bj4GJiQ7VILdzcbvGTPbETgaqErUC6d8oZvZDsBY\n4PagsySLmW0JHAHcAeDua919ZbCpEq4TsIWZdQJygCUB50kId58FfN1k+kTgnvDv7wFOSmqoBGru\n/br7DHdfFx6+DuyQ9GAJtIn/jQH+DlwOJOxMlJQvdGAqDX8I9UEHSaKdgOXAXeFDTbebWfegQyWK\nuy8GbqBh5fIlUO3uM4JNlVTbuPuXAOGv/QPOk0znAs8EHSLRzOwEYLG7z0/k66R0oZvZccAyd58X\ndJYk6wTsB9zq7vsC35NZ/wyPEj5mfCIwEBgAdDezM4NNJYlmZkXAOqA06CyJZGY5QBEwMdGvldKF\nDhwKnGBmFcC/gKPM7P5gIyXFImCRu88Jjx+moeAz1Ujgc3df7u61wKPAsIAzJdNXZrYdQPjrsoDz\nJJyZnQUcB4zzzL8YZmcaFivzw122A/CWmW0b7xdK6UJ39wnuvoO7h2j4kOwFd8/4lZu7LwW+MLNd\nw1MFwPvy0qiSAAAAx0lEQVQBRkq0KuBgM8sxM6Ph/Wbsh8DNeBI4K/z7s4AnAsyScGY2CvgjcIK7\n1wSdJ9Hcvdzd+7t7KNxli4D9wv8/j6uULvQs91ug1MzeBfYBpgScJ2HC/xJ5GHgLKKfhv8uMvDTc\nzB4EXgN2NbNFZvYLoAQ42swW0nAWREmQGeNpE+/3FqAnMNPM3jGzfwYaMs428Z6T89qZ/68dEZHs\noBW6iEiGUKGLiGQIFbqISIZQoYuIZAgVuohIhlChi4hkCBW6iEiG+P/Aq/W735B3XQAAAABJRU5E\nrkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1fc17a04710>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# python library for plotting\n",
    "import matplotlib.pylab as plt\n",
    "\n",
    "# iteration count\n",
    "N = 100\n",
    "\n",
    "# learning rate\n",
    "eta = 0.01\n",
    "\n",
    "# initial values\n",
    "m = b = 0\n",
    "\n",
    "# lists to keep y-values of line\n",
    "line = []\n",
    "\n",
    "# Let's start iterating\n",
    "for i in range(N):\n",
    "    m, b = gradientStep(eta, m, b)\n",
    "# end of for loop\n",
    "\n",
    "for i in range(len(x)):\n",
    "    line.append(m * x[i] + b)\n",
    "# end of for loop\n",
    "\n",
    "print(\"Parameters of the fittest line:\")\n",
    "print(\"m:\", m)\n",
    "print(\"b:\", b)\n",
    "\n",
    "# drawing of graph.\n",
    "plt.title(\"Fittest Line\")\n",
    "plt.plot(x, y, \"go\")\n",
    "plt.plot(x, line)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how the error level changes through the iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYXHWd7/H3t7p6S7o7ne500p2EELKQEJEQaZCAMCji\noMjmgCt3ouLAOOrgdh30eh91rqM446COjjioaNwQBkQQ0QHBCIhAGgJhSUJWQtbu7N2dpZf63j/O\nqaTS9FJJuvpU1fm8nqeeOvv5njrp+uRsvzJ3R0RE4isRdQEiIhItBYGISMwpCEREYk5BICIScwoC\nEZGYUxCIiMScgkCGnZm5mc0Iu79nZv93mJY7xcw6zKwk7F9kZh8ajmWHy/udmS0YruVlLLfSzH5j\nZrvN7L+He/nHwszOMbMVEdfwOTP7QZQ1xJ2CoICZ2Toz2xd+OaZf34m6rkzu/vfu/v+Gmi7cljcP\nsaz17l7l7r3HWpeZfdHMftZn+W9194XHuux+XAFMAOrd/cpjXZiZzTGzFjPbGb7+YGZzMsabmX3N\nzLaHr381M+tvWe7+iLvPyph3yP1wjLWfZ2Yb+tTwFXcftkCXI6cgKHwXh1+O6ddH+5vIzJLZDBvM\nkU4/nKJc9zA4HnjJ3XuOdMYBtnsTQbjUAeOAe4BfZoy/BrgMmAucArwduPZI132kwgDSd0ohcne9\nCvQFrAPePMC49wN/Br4B7AC+PMCwBPB54GWgFfgJMCZcxlTAgauB9cDDA6zrfwObCb6gPhjOMyMc\n92Pgy2H3OOBeYFe4/kfC9f8USAH7gA7gM/2tO2NYMlzeIuCrwJPAbuBuoC4cdx6wob/PC7gQ6AK6\nw/U9m7G8D4Xd2XwuC8LatgH/Z4DP5kt91nX1cHzmGctPAh8B9mYMewy4JqP/auDxAeY/+Dn1tx/C\n4WeGy9wFPAuclzH/IuBfCP5d7QNmAB8AlgHtwBrg2nDa0eE0qXD5HcBE4IvAzzKWeQnwQri+RcBJ\nffbhp4Gl4T6/DaiI+m+x0F+RF6DXMey8oYOgB/hY+GVROcCwDwKrgGlAFfAr4KfhMtJfSj8J/4gr\n+1nPhcBW4ORwml8wcBB8FfgeUBq+zgGsv23pb930HwQbM9Z9Z/oLhUGCIOw+7MsnY3npIMjmc/l+\nWNdc4EDmF1af5fb9ojumzzxjObvC/ZkCPp8xfDfw+oz+ZqB9gGUc9jn1sx8mAduBtxEE2AVhf0PG\nZ7YeeA3Bv6lS4CJgOmDAXwF7gdcNsl8Ofj7AiUBnuJ5Sgv8UrALKMup7kiBA6ggC5++j/lss9JcO\n4wrfr81sV8br7zLGbXL3b7t7j7vvG2DY+4Ab3X2Nu3cAnwXe3eeUxBfdvTNjGZneCfzI3Z93906C\nP+qBdANNwPHu3u3B+emhGrsabN0QfIGm1/1/gXemLyYfo2w+ly+5+z53f5bgf8pzh3HZQ2037l4L\njAE+CizJGFVFEAZpu4Gqga4TDOEq4D53v8/dU+7+ANBCEAxpP3b3F8J/U93u/lt3X+2BPwH3E4R+\nNt4F/NbdH3D3buDrBGF7VsY0/+Hum9x9B/Ab4NSj2C7JoCAofJe5e23G6/sZ417pZ/q+wyYSnKJI\ne5ngf3YThlhO5vyZ418eaELg3wj+d3e/ma0xs+sHmTabdfcd/zLB/yLHZbHcoWTzuWzJ6N5L8AU8\nXMsearsBCAPwe8BPzGx8OLgDqMmYrAboyCJ0+3M8cGXmfzaANxAEer+1mtlbzexxM9sRTv82st8n\nh3027p4Klz8pY5qj/dxlAAqC4tbfH37fYZsI/tjTphCcbtg6xHLSNgPH9Zm//2Lc2939U+4+DbgY\n+KSZnT/EOob68uq77m6Cc/adwKj0iPAooeEIlpvN53K0jvUz7ytBsK3pL8sXOPzoZG44LBt91/sK\nwVFX5n82Rrv7Df3NY2blBKfovg5MCI9a7iM4TdTf8vs67LMJj2KOIzgFKDmiIJBbgU+Y2QlmVgV8\nBbjNs7/D5Xbg/eEtjaOALww0oZm93cxmhH/ce4De8AXBl+C0o6j/qox1/zNwhwe3l74EVJjZRWZW\nSnBxtjxjvq3A1EHucjnWz2Uwx7RsM7vAzOaZWYmZ1QA3AjsJzpdDcH3hk2Y2ycwmAp8iuFaTjb77\n4WfAxWb21+H6KsJbQCcPMH8ZwefcBvSY2VuBt/RZfr2ZjRlg/tuBi8zs/HC/fYrg+stjWdYvR0FB\nUPh+0+c5gruOcP5bCO4WeRhYC+wnuJicFXf/HfBN4CGC0z4PDTL5TOAPBKcu/gJ8190XheO+Cnw+\nPP3w6SOo/6cEX3JbgArgH8O6dgP/APyA4H+TnUDm/evpB7u2m9nT/Sz3mD6XIRzrsmsJwmQ3sJrg\nTp0L3X1/OP6/CM6dPwc8D/w2HJaNw/aDu78CXAp8juDL/RWCu8T6/e5w93aCfXA7QTi9l+D21vT4\n5WHta8J1TOwz/wqC6xLfJjiyu5jgFumuLOuXo2BHd9pQRESKhY4IRERiTkEgIhJzCgIRkZhTEIiI\nxFxBNOQ1btw4nzp1atRliIgUlKeeemqbuzcMNV1BBMHUqVNpaWmJugwRkYJiZoM96X+QTg2JiMSc\ngkBEJOYUBCIiMacgEBGJOQWBiEjM5TQIzKzWzO4ws+VmtszM5ptZnZk9YGYrw/exuaxBREQGl+sj\ngm8Bv3f32QRtoi8DrgcedPeZwINhv4iIRCRnQRC2k34u8EMAd+9y910ETdouDCdbCFyWqxrufmYj\nP3s8q9toRURiK5dHBNMI2i//kZktMbMfmNlogl8t2gwQvo/vb2Yzu8bMWsyspa2t7agK+N1zW7jl\n0bVHWb6ISDzkMgiSwOuAm9x9HsEPg2R9Gsjdb3b3ZndvbmgY8gnpfs1qrGbt9k72dfUOPbGISEzl\nMgg2ABvc/Ymw/w6CYNhqZk0A4Xtrrgo4qakad1jZ2p6rVYiIFLycBYG7bwFeMbNZ4aDzgRcJfrZu\nQThsAXB3rmqY3VgDwPLNCgIRkYHkutG5jwE/N7MyYA3wAYLwud3MrgbWA1fmauVT6kZRWVrCsi17\ncrUKEZGCl9MgcPdngOZ+Rp2fy/WmJRLGiY3VOiIQERlE0T9ZfFJjNcu37MHdoy5FRCQvFX0QzG6s\nZufebtraD0RdiohIXir6IJgVXjBetkWnh0RE+lP0QTC7sRqAFbpgLCLSr6IPgrGjy2isqdAFYxGR\nARR9EEDwhLFODYmI9C8WQTC7qZrVrR1096aiLkVEJO/EIghOaqyhqzfF2m2dUZciIpJ3YhEEs8IL\nxss264KxiEhfsQiC6Q1VJBPGCl0nEBF5lVgEQVkywYzxVSxXEIiIvEosggCC00PLdWpIRORVYhME\nsxtr2LR7P7v3dUddiohIXolPEDSlnzDW6SERkUzxCYLwzqHlampCROQwsQmCxpoKxlSWskxNTYiI\nHCY2QWBmzG6sVuNzIiJ9xCYIgDAI2kml9CM1IiJp8QqCpho6u3rZsHNf1KWIiOSNeAWBLhiLiLxK\nrILgxAnVmKEnjEVEMsQqCEaXJ5lSN0pHBCIiGWIVBBCcHtIRgYjIITEMghrWbetkX1dv1KWIiOSF\nGAZBNSmHla06KhARgRwHgZmtM7PnzOwZM2sJh9WZ2QNmtjJ8H5vLGvqa3VQD6IKxiEjaSBwRvNHd\nT3X35rD/euBBd58JPBj2j5gpdaOoLC1huZqaEBEBojk1dCmwMOxeCFw2kisvSRgnTqjSnUMiIqFc\nB4ED95vZU2Z2TThsgrtvBgjfx/c3o5ldY2YtZtbS1tY2rEXNbqxh+ZZ23NXUhIhIroPgbHd/HfBW\n4CNmdm62M7r7ze7e7O7NDQ0Nw1rU7KZqdnR20dZxYFiXKyJSiHIaBO6+KXxvBe4CzgC2mlkTQPje\nmssa+jMr3dSErhOIiOQuCMxstJlVp7uBtwDPA/cAC8LJFgB356qGgcxuTN85pOsEIiLJHC57AnCX\nmaXX8wt3/72ZLQZuN7OrgfXAlTmsoV91o8uYUFOuW0hFRMhhELj7GmBuP8O3A+fnar3ZmtVYo1ND\nIiLE8MnitJMaq1nV2kF3byrqUkREIhXbIJjdVE1Xb4p12zqjLkVEJFKxDYJZE4ILxst0nUBEYi62\nQTB9/GiSCWP5Zt05JCLxFtsgKE+WML2hihU6IhCRmIttEEDwYJluIRWRuIt1EMxuqmbjrn3s3tcd\ndSkiIpGJdRCcFD5hrNNDIhJnsQ6C2U1Bm0Mr1NSEiMRYrIOgsaaCmoqkbiEVkViLdRCYGbObanQL\nqYjEWqyDAIKmJl7a2kEqpR+pEZF4in0QzGqsoeNADxt37Yu6FBGRSMQ+CNIXjJfp9JCIxFTsg2DW\nhPSdQ7pgLCLxFPsgGF2eZErdKD1hLCKxFfsgAJjdWM0yPUsgIjGlIABmN9Wwblsn+7t7oy5FRGTE\nKQgIjghSDi9t1ekhEYkfBQHw2kljAHj65Z0RVyIiMvIUBMBxdaOYVFvJX9Zsj7oUEZERpyAIzZ9e\nzxNrd+gJYxGJHQVBaP60enbt7dbdQyISOwqC0Pzp9QD8ZbVOD4lIvCgIQhNrK5laP4rHdZ1ARGIm\n50FgZiVmtsTM7g37TzCzJ8xspZndZmZlua4hW+nrBL26TiAiMTISRwTXAcsy+r8GfMPdZwI7gatH\noIasnDmtnvb9PbywaXfUpYiIjJicBoGZTQYuAn4Q9hvwJuCOcJKFwGW5rOFIzJ+m6wQiEj+5PiL4\nJvAZIBX21wO73L0n7N8ATMpxDVkbX1PB9IbRep5ARGIlZ0FgZm8HWt39qczB/Uza7wl5M7vGzFrM\nrKWtrS0nNfZn/vR6Fq/dQXdvauiJRUSKQC6PCM4GLjGzdcAvCU4JfROoNbNkOM1kYFN/M7v7ze7e\n7O7NDQ0NOSzzcPOnjaOzq5fnNuo6gYjEQ86CwN0/6+6T3X0q8G7gIXd/H/BH4IpwsgXA3bmq4Wic\nOa0O0HUCEYmPKJ4j+Cfgk2a2iuCawQ8jqGFA9VXlzJpQrecJRCQ2kkNPcuzcfRGwKOxeA5wxEus9\nWvOn13Pb4lfo6klRltQzdyJS3PQt148zp9Wzr7uXZzfsiroUEZGcUxD048xpdZjpOoGIxIOCoB+1\no8qY01TDY6u3RV2KiEjOKQgGMH9aPU+v36XfMRaRoqcgGMD86fV09aR4er1+vlJEipuCYACnn1BH\nwuBxXScQkSKnIBhATUUpr500Ru0OiUjRUxAM4szp9Tzzyi72dek6gYgULwXBIOZPq6e712l5eUfU\npYiI5IyCYBCnT60jmTA9TyAiRU1BMIjR5UlOmazrBCJS3BQEQ5g/vZ6lG3bTcaBn6IlFRAqQgmAI\n86eNozflLF6n6wQiUpwUBEM47fixlJaYnicQkaKlIBhCZVkJ86aM1XUCESlaQwaBmZWY2b+NRDH5\nav60ep7fuJs9+7ujLkVEZNgNGQTu3gucZmb9/fB8LMyfXk/K4ck1uk4gIsUn218oWwLcbWb/DXSm\nB7r7r3JSVZ6ZN6WW8mSCv6zZzpvnTIi6HBGRYZVtENQB24E3ZQxzIBZBUJ4s4bTjx+rBMhEpSlkF\ngbt/INeF5Lv50+q58Q8vsbOzi7Gjy6IuR0Rk2GR115CZTTazu8ys1cy2mtmdZjY518Xlk/nT63GH\nJ9bqqEBEiku2t4/+CLgHmAhMAn4TDouNUybXUllaotNDIlJ0sg2CBnf/kbv3hK8fAw05rCvvlCUT\nNE/V8wQiUnyyDYJtZnZV+ExBiZldRXDxOFbmT6/npa0dbOs4EHUpIiLDJtsg+CDwTmALsBm4IhwW\nK/On1QPwuI4KRKSIZPVkMfA37n6Juze4+3h3v8zdXx5ivgoze9LMnjWzF8zsS+HwE8zsCTNbaWa3\nmVnB3ILz2kljqCpP6jqBiBSVbJ8svvQoln0AeJO7zwVOBS40szOBrwHfcPeZwE7g6qNYdiSSJQnO\nnFbHohVtpFIedTkiIsMi21NDfzaz75jZOWb2uvRrsBk80BH2loYvJ3go7Y5w+ELgsqMpPCoXz53I\nxl37eFLNUotIkcj2yeKzwvd/zhiW/lIfUHha6SlgBvCfwGpgl7unf+VlA8HtqAXjgjkTGFVWwq+X\nbOTM8JqBiEghy+YaQQK4yd3f2Oc1aAhAcFrJ3U8FJgNnACf1N9kA673GzFrMrKWtrW2oVY2YUWVJ\nLjy5kd8+t5n93b1RlyMicsyyuUaQAj56LCtx913AIuBMoNbM0kcik4FNA8xzs7s3u3tzQ0N+PbLw\njnmTad/fw4PLWqMuRUTkmGV7jeABM/u0mR1nZnXp12AzmFmDmdWG3ZXAm4FlwB8Jbj8FWADcfZS1\nR2b+9Hom1JRz15KNUZciInLMsr1GkH5m4CMZwxyYNsg8TcDC8DpBArjd3e81sxeBX5rZlwmat/7h\nEdYcuZKEcempk7jl0bXs6OyiTo3QiUgBy7b10ROOdMHuvhSY18/wNQTXCwra5fMmcfPDa7h36Sb+\ndv7UqMsRETlqg54aMrPPZHRf2WfcV3JVVCE4qamG2Y3VOj0kIgVvqGsE787o/myfcRcOcy0F5/J5\nk1iyfhdrt3UOPbGISJ4aKghsgO7++mPn0lMnYYaOCkSkoA0VBD5Ad3/9sdM4poKzp4/j10s24h77\nj0NECtRQQTDXzPaYWTtwStid7n/tCNSX9y6bN4n1O/by9PqdUZciInJUBg0Cdy9x9xp3r3b3ZNid\n7i8dqSLz2YUnN1JRmuBXT+v0kIgUpmwfKJMBVJUn+evXNHLv0s0c6FGTEyJSeBQEw+CyeZPYva+b\nRSvyp00kEZFsKQiGwTkzxjGuqpy7dHpIRAqQgmAYJEsSXDJ3Ig8tb2X33u6oyxEROSIKgmFy+bxJ\ndPWmuPe5fhtTFRHJWwqCYXLypBpmjK/i13q4TEQKjIJgmJgZl8+bxOJ1O3llx96oyxERyZqCYBhd\neupEQE1OiEhhURAMo8ljR/H6E+rU5ISIFBQFwTB7x+smsWZbJ89u2B11KSIiWVEQDLMLT26iLJng\nrqc3RF2KiEhWFATDbExlKRecNIHfLN1Md28q6nJERIakIMiBy+dNYkdnFw+/pCYnRCT/KQhy4NwT\nGxg7qpRf6e4hESkACoIcKEsmuHjuRB54cSt79qvJCRHJbwqCHLnitMl09aS49Yn1UZciIjIoBUGO\nnDK5lnNPbOB7f1pNx4GeqMsRERmQgiCHPnXBiezc282PHl0bdSkiIgNSEOTQ3ONquWDOBG5+ZI2a\npxaRvKUgyLFPXnAi7ft7+P4ja6IuRUSkXzkLAjM7zsz+aGbLzOwFM7suHF5nZg+Y2crwfWyuasgH\nJzXV8PZTmrjlz2vZ3nEg6nJERF4ll0cEPcCn3P0k4EzgI2Y2B7geeNDdZwIPhv1F7eNvPpH93b18\n70+roy5FRORVchYE7r7Z3Z8Ou9uBZcAk4FJgYTjZQuCyXNWQL2aMr+LyeZP5yV9eZuue/VGXIyJy\nmBG5RmBmU4F5wBPABHffDEFYAOMHmOcaM2sxs5a2tsJvquG682fSm3L+84+roi5FROQwOQ8CM6sC\n7gQ+7u57sp3P3W9292Z3b25oaMhdgSNkSv0o3nn6cdz65Ho27NQvmIlI/shpEJhZKUEI/NzdfxUO\n3mpmTeH4JqA1lzXkk4+9aQZmxrcf1FGBiOSPXN41ZMAPgWXufmPGqHuABWH3AuDuXNWQb5rGVPK+\n10/hjqc3sHZbZ9TliIgAuT0iOBv4X8CbzOyZ8PU24AbgAjNbCVwQ9sfGh8+bTllJgm/94aWoSxER\nASCZqwW7+6OADTD6/FytN9+Nr65gwVlT+a+HV/MPb5zBiROqoy5JRGJOTxZH4NpzpzG6LMk3HtBR\ngYhET0EQgbGjy7j6DSfwu+e38PxG/ci9iERLQRCRq885gTGVpdyoowIRiZiCICI1FaVc+1fTeGh5\nK0+9vDPqckQkxhQEEXr/WVMZV1XGjQ+siLoUEYkxBUGERpUl+fB5M/jzqu08tnpb1OWISEwpCCL2\nvtdPobGmgq/ct4yunlTU5YhIDCkIIlZRWsIXL5nD8xv38PX7dYpIREaegiAPXHhyE1edOYWbH17D\nH1fEpuklEckTCoI88fmL5jC7sZpP3f6sfrNAREaUgiBPVJSW8J33zmNfVy+fuO0ZelMedUkiEhMK\ngjwyY3w1X7rkNTy2ejs3LVJT1SIyMhQEeebK5slcMnci3/jDShav2xF1OSISAwqCPGNm/MvlJzOp\ntpLrbl3Crr1dUZckIkVOQZCHqitK+c5759HWcYDP3LEUd10vEJHcURDkqVMm1/JPF87m/he38tPH\nX466HBEpYgqCPPbBs0/gjbMa+PK9y3hhk5qrFpHcUBDksUTC+PqVc6kdVcrHbl1C54GeqEsSkSKk\nIMhz9VXlfPPdp7J2WydfuOeFqMsRkSKkICgAZ00fx8feOIM7ntrAXUs2RF2OiBQZBUGB+MfzZ3L6\n1LF8/q7neXHTnqjLEZEioiAoEMmSBN969zxqKkt5z/cf55lXdkVdkogUCQVBAZlYW8nt185nTGUp\nV/3gCZ5Ysz3qkkSkCCgICsxxdaO4/dr5TKgpZ8GPnuThl9qiLklECpyCoAA1jqngtmvnc8K4Kj60\nsIX7X9gSdUkiUsByFgRmdouZtZrZ8xnD6szsATNbGb6PzdX6i924qnJu/bvXc9LEGj7886e559lN\nUZckIgUql0cEPwYu7DPseuBBd58JPBj2y1GqHVXGz64+g9OOH8t1v1zC7YtfibokESlAOQsCd38Y\n6NuO8qXAwrB7IXBZrtYfF9UVpSz8wBm8YcY4PnPnUhY+ti7qkkSkwIz0NYIJ7r4ZIHwfP9CEZnaN\nmbWYWUtbmy6IDqayrIQfLGjmgjkT+MI9L3DTotVRlyQiBSRvLxa7+83u3uzuzQ0NDVGXk/fKkyV8\n932v4+K5E/na75dz4/0r1Hy1iGQlOcLr22pmTe6+2cyagNYRXn9RKy1J8M13nUplaYL/eGgV2zu7\n+PxFc6gsK4m6NBHJYyN9RHAPsCDsXgDcPcLrL3olCeOGd5zCtedO4+dPrOei/3iEp17eGXVZIpLH\ncnn76K3AX4BZZrbBzK4GbgAuMLOVwAVhvwyzRML47NtO4hcfej0HelJc+b3H+Op9y9jf3Rt1aSKS\nh6wQziM3Nzd7S0tL1GUUpI4DPXzlvmX84on1TG8YzdevnMu8KXp8QyQOzOwpd28earq8vVgsw6Oq\nPMlXLn8tP/ngGezr6uVvbnqMr/1+OQd6dHQgIgEFQUyce2IDv//EuVx52nHctGg1F3/7UZZuUAum\nIqIgiJWailK+dsUp/OgDp7N7XzeXf/cx/v3+FXT1pKIuTUQipCCIoTfOGs/9H/8rLjt1Et9+aBWX\nfOdRHlq+Vc8diMSUgiCmxowq5d/fOZcfLmimfX8PH/xxC2/91iPc/cxGenp1hCASJ7prSOjuTXHP\nM5u46U+rWdXawZS6UVxz7jSuOG0yFaV6GE2kUGV715CCQA5KpZwHlm3lu4tW8+wruxhXVc7VbziB\nq86cQnVFadTlicgRUhDIUXN3/rJmOzctWs0jK7dRXZHkb+cfzwfOPoFxVeVRlyciWVIQyLBYumEX\nNy1aze9f2EJZSYJLT53IRadM5Kzp9ZSW6BKTSD5TEMiwWtXawfcfXsO9SzfR2dVL7ahS3jJnAm97\nbRNnzxinUBDJQwoCyYn93b08/FIb9z23mT8sa6XjQA9jKsNQOKWJs6ePoyypUBDJBwoCybn93b08\nsnIbv3tuMw+8uJX2Az3UVCR5y2saedtrGznjhHqqyke6pXMRSVMQyIg60NPLoyu38dt0KOzvIWEw\nZ2INzcfX0Tx1LM3H19E4piLqUkViQ0EgkTnQ08uTa3eweN1OWtbtYMn6XewLm8CePLaS06ceCoaZ\n46tIJCziikWKU7ZBoON2GXblyRLOmdnAOTODnxjt7k2xbPMeFq/byVMv7+DRVdu4a8lGAGoqkpwy\nuZYZ46uYOaGKmeOrmTm+irGjy6LcBJFY0RGBjDh3Z/2OvbSs20nLyzt4cdMeVrZ2sLfrUNPY46rK\ngnAYX83MCVXMGB+8GqrKMdMRhEg2dEQgecvMOL5+NMfXj+ZvTpsMBE81b96zn5Vb21nV2sHKrR2s\nbG3n10s20n6g5+C8ZckEE8dU0DSmkom1lUyqraCpNqN7TCWjdYFa5IjoL0byQiJhTKqtZFJtJefN\nGn9wuLvT2n6AlVs7WNXazqbd+9m0ax+bdu3jsdXb2LpnP6k+B7VjKksZX11O3egy6qvKqBtdRt3o\ncupGlVJXVU796GBY/egyakeV6XZXiT0FgeQ1M2NCTQUTaip4w8xxrxrf3Zuitf3AwXDYtCsIirb2\nA+zo7GLFlnZ2dHaxa183A50FrShNUF1RSnVFkuqKUmoqklSVJw/2V4f9VeVJKstKqCwtYVRZksqy\nBBXp7tKSg+NKS0ynr6SgKAikoJWWJA4eSQymN+Xs2tvFjs4utnceet/Z2UX7/m46DvSwZ38P7ft7\naN/fzZbd+w92d3Yd2c96liSM8mSCsmTi4HtZSYLyZMlhw4J+I5lIkCwxykqC92QiQWmJUVqSIFmS\noDRhJEsSJBNGScJIloTvCSNh6f5D40vMSCQgYYf6Ld0dDk+PS2RMm7AgeNPdCTPMgu0xgmGkh3No\nvIXv6eFmYKTHZXQTTpueRmGZNxQEEgslCaO+qpz6qnJmHuG8vSmnY38PnV097OvuZV9X78H3vV29\n7O8O3oNhwTQHulN09abo6klxoCf93suBsL99fw/berro7k3R05uiu9fpSQXvwbBD/cWu35AgDImD\n4w4N6zv94dMdmiYcHPZbRnfGTBnLO3y+V49LL6e/+vt2Z87ft47M/r49/a3rlgWnM6V+1KvWO5wU\nBCJDKEkYY0aVMmbUyDfF7e70pPxgMKRS0JNK0ZsKhveGr3R3d28Kd+j1oD/lTirl9LqTShG+B8OD\n8cE6Uk4wrTt+sJuD87qDEwzDD5/PwzrT84WT4ITzHZyfw5cTbOCrpz80CicY4BmfR+ay4NB8mZ9Z\nMLzPcg7KyBXbAAAGgUlEQVSbh4z+PuMGWu5h++XQNH0nyKz11evqu5xDfYdFfkbPSFzDUhCI5DEz\nC08TAehHgiQ3dLuEiEjMKQhERGIukiAwswvNbIWZrTKz66OoQUREAiMeBGZWAvwn8FZgDvAeM5sz\n0nWIiEggiiOCM4BV7r7G3buAXwKXRlCHiIgQTRBMAl7J6N8QDjuMmV1jZi1m1tLW1jZixYmIxE0U\nQdDf44SvemrG3W9292Z3b25oaBiBskRE4imKINgAHJfRPxnYFEEdIiJCBL9HYGZJ4CXgfGAjsBh4\nr7u/MMg8bcDLR7nKccC2o5w3XxXbNml78l+xbVOxbQ/0v03Hu/uQp1RG/Mlid+8xs48C/0PwqOQt\ng4VAOM9Rnxsys5ZsfpihkBTbNml78l+xbVOxbQ8c2zZF0sSEu98H3BfFukVE5HB6slhEJObiEAQ3\nR11ADhTbNml78l+xbVOxbQ8cwzYVxI/Xi4hI7sThiEBERAahIBARibmiDoJia+XUzNaZ2XNm9oyZ\ntURdz9Ews1vMrNXMns8YVmdmD5jZyvB9bJQ1HokBtueLZrYx3E/PmNnboqzxSJjZcWb2RzNbZmYv\nmNl14fBC3kcDbVNB7iczqzCzJ83s2XB7vhQOP8HMngj30W1mVpb1Mov1GkHYyulLwAUETzMvBt7j\n7i9GWtgxMLN1QLO7F+yDMGZ2LtAB/MTdTw6H/Suww91vCAN7rLv/U5R1ZmuA7fki0OHuX4+ytqNh\nZk1Ak7s/bWbVwFPAZcD7Kdx9NNA2vZMC3E8W/JjxaHfvMLNS4FHgOuCTwK/c/Zdm9j3gWXe/KZtl\nFvMRgVo5zUPu/jCwo8/gS4GFYfdCgj/SgjDA9hQsd9/s7k+H3e3AMoJGIQt5Hw20TQXJAx1hb2n4\ncuBNwB3h8CPaR8UcBFm1clpgHLjfzJ4ys2uiLmYYTXD3zRD80QLjI65nOHzUzJaGp44K5jRKJjOb\nCswDnqBI9lGfbYIC3U9mVmJmzwCtwAPAamCXu/eEkxzR910xB0FWrZwWmLPd/XUEP+rzkfC0hOSf\nm4DpwKnAZuDfoy3nyJlZFXAn8HF33xN1PcOhn20q2P3k7r3ufipBo51nACf1N1m2yyvmICi6Vk7d\nfVP43grcRfAPoBhsDc/jps/ntkZczzFx963hH2oK+D4Ftp/C8853Aj9391+Fgwt6H/W3TYW+nwDc\nfRewCDgTqA0b9YQj/L4r5iBYDMwMr6SXAe8G7om4pqNmZqPDC12Y2WjgLcDzg89VMO4BFoTdC4C7\nI6zlmKW/MEOXU0D7KbwQ+UNgmbvfmDGqYPfRQNtUqPvJzBrMrDbsrgTeTHDd44/AFeFkR7SPivau\nIYDwdrBvcqiV03+JuKSjZmbTCI4CIGgs8BeFuD1mditwHkGTuVuBLwC/Bm4HpgDrgSvdvSAuwA6w\nPecRnG5wYB1wbfr8er4zszcAjwDPAalw8OcIzqkX6j4aaJveQwHuJzM7heBicAnBf+Zvd/d/Dr8j\nfgnUAUuAq9z9QFbLLOYgEBGRoRXzqSEREcmCgkBEJOYUBCIiMacgEBGJOQWBiEjMKQgkVsysI3yf\nambvHeZlf65P/2PDuXyRXFEQSFxNBY4oCMIWbQdzWBC4+1lHWJNIJBQEElc3AOeE7dB/ImzE69/M\nbHHYCNm1AGZ2XtiW/S8IHkjCzH4dNvz3QrrxPzO7AagMl/fzcFj66MPCZT9vwe9JvCtj2YvM7A4z\nW25mPw+fghUZUcmhJxEpStcDn3b3twOEX+i73f10MysH/mxm94fTngGc7O5rw/4PuvuO8PH+xWZ2\np7tfb2YfDRsC6+sdBE+wziV4AnmxmT0cjpsHvIagXZg/A2cTtC8vMmJ0RCASeAvwt2HTvk8A9cDM\ncNyTGSEA8I9m9izwOEHDhjMZ3BuAW8MGzrYCfwJOz1j2hrDhs2cITlmJjCgdEYgEDPiYu//PYQPN\nzgM6+/S/GZjv7nvNbBFQkcWyB5LZFkwv+puUCOiIQOKqHajO6P8f4MNhc8WY2YlhK699jQF2hiEw\nm6D537Tu9Px9PAy8K7wO0QCcCzw5LFshMgz0vw+Jq6VAT3iK58fAtwhOyzwdXrBto/+f+vs98Pdm\nthRYQXB6KO1mYKmZPe3u78sYfhcwH3iWoKXLz7j7ljBIRCKn1kdFRGJOp4ZERGJOQSAiEnMKAhGR\nmFMQiIjEnIJARCTmFAQiIjGnIBARibn/DyfFtGbFx1ukAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1fc17207908>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# python library for plotting\n",
    "import matplotlib.pylab as plt\n",
    "\n",
    "# iteration count\n",
    "N = 30\n",
    "\n",
    "# learning rate\n",
    "eta = 0.01\n",
    "\n",
    "# initial values\n",
    "m = b = 0\n",
    "\n",
    "# lists to keep the error and the iteration values.\n",
    "errors = []\n",
    "Ns = []\n",
    "\n",
    "# Let's start iterating\n",
    "for i in range(N):\n",
    "    errors.append(computeError(m, b))\n",
    "    Ns.append(i)\n",
    "    m, b = gradientStep(eta, m, b)\n",
    "# end of for loop\n",
    "\n",
    "# drawing of graph.\n",
    "plt.title(\"Error distribution for %s iteration\" % N)\n",
    "plt.ylabel(\"Error\")\n",
    "plt.xlabel(\"Iteration\")\n",
    "plt.plot(Ns, errors)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is obvious that we can carry out a few optimization for the GD algorithm. For example, extending the steps of GD. This idea leads us to the notion of momentum.\n",
    "\n",
    "## Gradient Descent with momentum\n",
    "\n",
    "We are going to denote $\\eta$ as $\\alpha$ now on and for momentum calculation, we have to do a few modification in GD equation.\n",
    "\n",
    "$p$ will denote the momentum;\n",
    "\n",
    "\\begin{align}\n",
    "& p_{\\tau} = \\nabla E(m_{\\tau-1}) + \\beta p_{\\tau-1} \\\\\n",
    "& m_{\\tau} = m_{\\tau-1} - \\alpha p_{\\tau} \\\\\n",
    "& m_{\\tau} = m_{\\tau-1} - \\alpha \\left( \\nabla E(m_{\\tau-1}) + \\beta p_{\\tau-1} \\right)\n",
    "\\end{align}\n",
    "\n",
    "------------------------------------------\n",
    "\\begin{align}\n",
    "& p_{\\tau} = \\nabla E(b_{\\tau-1}) + \\beta p_{\\tau-1} \\\\\n",
    "& b_{\\tau} = b_{\\tau-1} - \\alpha p_{\\tau} \\\\\n",
    "& b_{\\tau} = b_{\\tau-1} - \\alpha \\left( \\nabla E(b_{\\tau-1}) + \\beta p_{\\tau-1} \\right) \\\\\n",
    "\\end{align}\n",
    "\n",
    "$\\beta$ will be our momentum coefficient. When $\\beta=0$, we recover GD."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 \t error: 60.01569999999999\n",
      "10 \t error: 32.94127110737379\n",
      "20 \t error: 14.501623737861456\n",
      "30 \t error: 5.786070865606438\n",
      "40 \t error: 2.5509966153706385\n",
      "50 \t error: 1.5774154763500476\n",
      "60 \t error: 1.338840949345472\n",
      "70 \t error: 1.288118708474226\n",
      "80 \t error: 1.2736507445172964\n",
      "90 \t error: 1.2651044588940537\n",
      "100 \t error: 1.2590759027871392\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztvXmcHHd55/9++j7m0jG6LckG2cYGbIO4STAYBxMg9muX\nGxID3jWbhSS7gSWQZBPIsgkkvySQLAvrYLAhhGAcjIFwOQbjcBlkY/mShS/JlnWNjrn77uf3R32r\nuqqne2YkTc9oup/36zWvrvp2Hd/qkupTz/F9vqKqGIZhGL1LbKk7YBiGYSwtJgSGYRg9jgmBYRhG\nj2NCYBiG0eOYEBiGYfQ4JgSGYRg9jgmBccqIiIrIU93yp0Tkfy7QcTeLyKSIxN36bSLynxbi2O54\n3xKRKxfqeKHjZkXk6yIyJiJfXujjd5KFvH+n0If7ReTipexDr2FCsIwQkT0iUnAPR//v/yx1v8Ko\n6n9R1f8113buWl4+x7EeV9U+Va2dar9E5IMi8o9Nx3+lql5/qsduwWuBtcAqVX3dqR5MRFIicqP7\nzbT5ISkeHxWRo+7vL0VEQt9fIyK7RaQuIm+b7Vzh+yciF4vIvlPt/2yIyHUi8uGmPpyvqrd18rxG\nFBOC5cdr3MPR/3t3q41EJDGfttk40e0XkqU89wKwBfilqlZPdMdZrvuHwFuBgy2+uxq4ArgAeCbw\nauCdoe93Av8VuOtE+3MqLPN72Fuoqv0tkz9gD/DyNt+9DfgR8LfAMeDDbdpiwB8De4HDwOeAQXeM\nrYACVwGPA7e3Odf/AA4A+4F3uH2e6r67DviwW14NfAMYdef/d3f+zwN1oABMAu9rde5QW8Id7zbg\nL4CfAWPAzcBK993FwL5WvxdwGVAGKu58O0PH+09ueT6/y5Wub0eAP2rz23yo6VxXLcRvHjr+PuDi\nprYfA1eH1q8Cftpi3x8Cb5vj+Ne5fyd5d3/q7jomgQ3uWt4PPAIcBW4I3YOW1wJ8GU/Axtx9Pd+1\nX+1+p7I7/teb/50DaeBjeP/W9rvldPieA+9xv+sB4O1L/f90Of6ZRdBdPA94FFgD/O82bW9zfy8F\nzgL6gGb30kuApwGvaD6BiFwGvBe4FNiG96Btx3vw/qMO47lK/hBQVf1NvAeFb9385XzO7fgtPPHZ\nAFSBv5vl/OCd8NvAnwNfcue7oMVmb2Pu3+XFwDnAJcCfiMjTWpzrT5vOde08jz3Xdc/G+Xhv/T47\nXdtJo6pTwCuB/dqwPvcDv4tnfbwE7x4cBz7RtHvztXwL79/KGjyr5AvuHNe45b90x39Ni678EfB8\n4EI8i+e5eKLqsw4YBDbiCdAnRGTFKVx6T2JCsPz4qoiMhv7+c+i7/ar696paVdVCm7a3AH+jqo+q\n6iTwAeCNTWb8B1V1KnSMMK8HPquq97mHxQdn6WsFWA9sUdWKqv67ule5WZjt3ACfD537fwKv94PJ\np8h8fpcPqWpBVXfiPWxbCcrJHnuu656NPry3bZ8xoC8cJ1hA3olnDe1T1RLe/X/tbNeiqp9R1YnQ\n9heIyOA8z/cW4M9U9bCqjuBZXL8Z+r7ivq+o6jfxLItzTuUCexETguXHFao6FPr7h9B3T7TYvrlt\nA56LwmcvkMB7Y5/tOOH9w9/vbbch8FfAw8B3ReRREXn/LNvO59zN3+8FknguqFNlPr9L2D8/jfcA\nXqhjz3XdszEJDITWB4DJeYjuybAFuMl/EQF2ATXaXIuIxEXkIyLyiIiM47l9YP73rNVvtyG0flSj\nsZgTuS+Gw4Sgu2j1H7+5bT/ef2afzXgulkNzHMfnAHBG0/6tO+O9Bb5HVc8CXgP8vohcMsc55np4\nNZ+7gueznwJy/hfOShg+gePO53c5WU71N5+L+4laJxe4tlOlVZ+eAF7Z9DKSUdUn2+z3ZuByPBfi\nIF4cAUBabNuKVr/d/nn235gnJgS9xxeB/y4iZ4pIHw1/9nwzXG4A3iYi54lIDvjTdhuKyKtF5KnO\nRTGO9+bop4IewvOXnyhvDZ37z4Ab1Usv/SWQEZFXiUgSz4+cDu13CNgqIu3+zZ/q7zIbp3xsEUmL\nSMatpkQkE3L9fA5PZDeKyAa82Mx1oX1Tbl8Bkm7f+fzfPwSsanLjfAr43yKyxR17WEQun+UY/UAJ\nL7Ccw7v25nPM9u/gi8Afu/OsBv4E+MdZtjdOAhOC5cfXm8YR3HSC+38GL2vnduAxoAj8znx3VtVv\n4WVufA/P7fO9WTbfBvwbnuviJ8D/1UZ++F/g/QcfFZH3nkD/P4/3kDsIZPCCl6jqGF6K5KeBJ/Es\nhHAOvD+w66iItEqjPKXfZQ4W4ti78bJ4NgLfccv+m/L/A74O3AvcB/yra/P5rtv+hcA1bvlX5zqh\nqj6I9yB+1N2nDcDHga/hufsmgJ/iJSS043N47pwngQfc9mGuBc5zx/9qi/0/DOwA7nHXd5drMxYQ\n6Ywb0TAMw1gumEVgGIbR45gQGIZh9DgmBIZhGD2OCYFhGEaPsyyKQq1evVq3bt261N0wDMNYVtx5\n551HVHV4ru2WhRBs3bqVHTt2LHU3DMMwlhUiMtvI/wBzDRmGYfQ4JgSGYRg9jgmBYRhGj2NCYBiG\n0eOYEBiGYfQ4HRUCERlyk24/KCK7ROQFIrJSRG4RkYfcp80mZBiGsYR02iL4OPBtVT0Xr0b6Lrz5\nTm9V1W3ArW7dMAzDWCI6JgQiMoBX6vZaAFUtq+oo3iQV17vNrseb/7Qj3PSLffzjT+eVRmsYhtGz\ndNIiOAsYAT4rIr8QkU+LSB5Yq6oHANznmlY7i8jVIrJDRHaMjIycVAe+sfMAX/zZ4yfZfcMwjN6g\nk0KQAJ4FfFJVL8KbKGTebiBVvUZVt6vq9uHhOUdItySfTjBdrs29oWEYRg/TSSHYB+xT1Tvc+o14\nwnBIRNYDuM/DnepAPp1gsrQQMw0ahmF0Lx0TAlU9CDwhIue4pkvwpqr7GnCla7sSuLlTfcin4kyZ\nEBiGYcxKp4vO/Q7wBRFJAY8Cb8cTnxtE5CrgceB1nTq57xqq15VYTObewTAMowfpqBCo6t3A9hZf\nXdLJ8/rk03EApis1+tLLotCqYRjGotPVI4vz7uE/be4hwzCMtnS3EKQ8IbCAsWEYRnu6WwicRTBV\nshRSwzCMdnS3EKS8GMFU2SwCwzCMdnS3EAQWgQmBYRhGO3pDCGx0sWEYRlu6XAica8gsAsMwjLZ0\nuRCYa8gwDGMuuloIcknfIjDXkGEYRju6WggS8RiZZMyyhgzDMGahq4UAvEFl5hoyDMNoT/cLQdqE\nwDAMYza6XghyqfiM9NFP/eARrrru50vUI8MwjNOLrheCvhYWwd2Pj7Jz39gS9cgwDOP0ouuFoJVr\naLRQplSxTCLDMAzoCSGY6Roana5QMCEwDMMAekEIWmQNjRUqVOtKtVZfol4ZhmGcPnS/ELRyDU1X\nAChWTQgMwzB6QAg815CqAlCs1AK3UNHcQ4ZhGN0vBLlUglpdKbm3//FCJfjOhMAwDKMHhKCvqfDc\naEQIzDVkGIbR9UKQS0ULz/nxATCLwDAMA3pACAKLwBWeGzPXkGEYRoSuF4LmOQlGp8vBd+YaMgzD\n6Akh8Cew997+zSIwDMOIkujkwUVkDzAB1ICqqm4XkZXAl4CtwB7g9ap6vFN9mGkRhISgakJgGIax\nGBbBS1X1QlXd7tbfD9yqqtuAW916x8inmrOGzDVkGIYRZilcQ5cD17vl64ErOnmyVhZBv2sz15Bh\nGEbnhUCB74rInSJytWtbq6oHANznmlY7isjVIrJDRHaMjIycdAeC9NFQjGDNQBowITAMw4DOC8GL\nVPVZwCuBd4nIr853R1W9RlW3q+r24eHhk+5AOhEjEZOIRbB+MAuYEBiGYUCHhUBV97vPw8BNwHOB\nQyKyHsB9Hu5kH0QkUnhutFBmdV+KmFiMwDAMAzooBCKSF5F+fxn4NeA+4GvAlW6zK4GbO9UHn3xo\nusrR6QpDuRSZZHyGRXDDjie44hM/6nR3DMMwTis6mT66FrhJRPzz/JOqfltEfg7cICJXAY8Dr+tg\nH4BGKepaXZkoVhnKJT0haEoffWD/ODv3jaKquH4bhmF0PR0TAlV9FLigRftR4JJOnbcVuXSCyVI1\nqDw6lE2SScRmuIamSlVUoVJTUgkTAsMweoOuH1kM0JeOM12uBZVHfddQ83SV0859VLKBZoZh9BA9\nIQQ5N12lX2doMJcknYzPmMB+2hWmK9nMZYZh9BA9IQR96QRT5WpgEQxmk2STLVxDgUVgQmAYRu/Q\nE0KQS8WZKtUYmw7FCFpkDQUWgY0vMAyjh+gJIehLR11DQfpoUyxguuTPZWwWgWEYvUNPCEE+naBU\nrXN0yhOCgUyCTEvXkB8jMIvAMIzeoSeEwK839OTxAv2ZBIl4jEwiTqHcLmvILALDMHqHnhACf7rK\nJ0cLDOWSAF7WUOjNX1VNCAzD6El6QghyTgj2jxUYyqYAyCbjEddQqVqnVldv2YLFhmH0ED0hBH1u\nusoDo8XAIvBiBI0H/nTITWQWgWEYvURPCEHOzVJWrSuDWV8I4lTrSrXmPfT91FEwITAMo7foCSHw\nYwRASAi8Sy9WfSEIWwTmGjIMo3foCSHws4aAkGvIa/Mzh/z5CgBKNo7AMIweoieEIGwR+MHiTMIT\nAj9OYDECwzB6lZ4QgnzYNeRbBM5K8N1AEYvAXEOGYfQQPSEE2WTINeTHCBIuRuDcQOGS1M0jjiu1\nOk+OFjrdTcMwjCWhJ4QgFhPyzgIYyjnXUDLqGpoqtQ8Wf+Wufbz8r38wYySyYRhGN9ATQgCNQWXN\nwWL/7d9PH00lYjNiBIfGSxQqtaAWkWEYRjfRM0LgB4yHmtJHC00WwcpcakbWkL+NBZENw+hGekYI\n/BTSgWyzReCyhipV0okYuVR8hmvIdwmVTQgMw+hCekYI8mmv9LQvANlmISjVyKcTLV1DxcAisBiB\nYRjdR88IQV86EYwhAEg3jSyeKlfJpeKuKmkb15ANNDMMowtJzL1Jd/DsLSuC+AA0XEOlkEWQS8VJ\nJ2Izqo8WrDy1YRhdTM8Iwbte+tTI+oyRxZUauVSCTDLOuJvk3qdgriHDMLqYjruGRCQuIr8QkW+4\n9TNF5A4ReUhEviQiqbmO0QmScSEmofTRUpV82lkE7WIE5hoyDKMLWYwYwe8Bu0LrHwX+VlW3AceB\nqxahDzMQETLJeCN9tOxZBC1dQ5Y+ahhGF9NRIRCRTcCrgE+7dQFeBtzoNrkeuKKTfZiNTDIeKjpX\nJZ+Kk060CBaXzTVkGEb30ukYwceA9wH9bn0VMKqq/hDdfcDGDvehLeHpKqdKNbKpBCIzH/j+NjaO\nwDCMbqRjFoGIvBo4rKp3hptbbKpt9r9aRHaIyI6RkZGO9DGdjFF0D/1CYBHEbGSxYRg9RSddQy8C\nfkNE9gD/jOcS+hgwJCK+JbIJ2N9qZ1W9RlW3q+r24eHhjnQwk4hTqtSo19XLGkonzDVkGEbP0TEh\nUNUPqOomVd0KvBH4nqq+Bfg+8Fq32ZXAzZ3qw1x4E9jXKVZrqBJYBOVanXrdM1RU1QaUGYbR1SzF\nyOI/AH5fRB7GixlcuwR9AAiyhvyCc7l0IhhxXHaT2ofnJjDXkGEY3ciiDChT1duA29zyo8BzF+O8\nc5FJxhkrVALXTy4ZDwLCpUo9kl4K5hoyDKM76ZlaQ63IuvRRf56BfDoelKf2H/pRIYhaBPW6svfo\n1CL11jAMozP0tBCkXYzAn5TGG1Dmz2XsprAMzUrWnD76vQcP87K//gGHx4uL1GPDMIyFp6eFIJP0\n5h7wYwR+iQloWATFWSyCwxMlanXl+HS0NpFhGMZyoreFIBGPWATZZCIQglaT2s8caGZppYZhLH96\nWwiSMQqVGtPlkEXgl6cOBpqFhMAGmhmG0YX0uBDEqdWVMVd22i86B42Hvv+wzyStKqlhGN1JTwuB\nP13lsaky0Bwj8McReA/7oWyqrWuoWDHXkGEYy5eeFgI/VfTYVBkRL2bQyBqKuoaGcsn2U1iaa8gw\njGVMTwtBOmQRZJNxYjEJRhYH6aPuYT+YTc6MEZTd4DMLFhuGsYzpaSHw5y0+OlUml/IGWbeLEQzl\nkkHZCR+/cqlZBIZhLGd6WwjcQ//oZIl82hOFZtdQsVxDBPozyRkzlxX9qqQWIzAMYxnT20IQcg35\nFkGmhWsom4y3zBqyGIFhGN1ATwtBNuUJwWihQi7VbBFEhaDVPAVFEwLDMLqAnhaCjHvoqxIIQTIu\n3nSVFT9ryKtCmk7EZgSFCxULFhuGsfzpbSFINi4/71xDIkI6EaMYGkeQdZPaV2pKrd6YWbMUjCMw\ni8AwjOVLjwtBPFjOpRvLaTeFJYRcQ/6ENSE3UMFqDRmG0QXMKQQiEheRv1qMziw26RYWAeDcQI0y\n1NnkzKqkgE1haRhGVzCnEKhqDXi2iMgi9GdRiVgEqZBFEMoQKlRqZFJxUomZFkG7YHG9rty66xCq\nimEYxunOfF1DvwBuFpHfFJH/4P91smOLQTYiBGGLIB6ZjyCbjM3IJqrXNYgNNLuGfvrYUa66fgc7\n9411tP+GYRgLwXznLF4JHAVeFmpT4CsL3qNFJBmPEY8JtboGA8rAuYYqzemjUddQ2ApotgjGC978\nBhNFm7DGMIzTn3kJgaq+vdMdWSoyiRhT5VqTRdAUI0jFZ0xYE5m5rNJ6fIFlExmGsRyYl2tIRDaJ\nyE0iclhEDonIv4jIpk53bjHw4wRhi8CfwhJcjCAZnrBm5sxlxRnjC6w8tWEYy4f5xgg+C3wN2ABs\nBL7u2pY9vhCE4wXzyRoqzGIR+KWrbcSxYRjLgfkKwbCqflZVq+7vOmC4g/1aNPwU0ny6KVhcqVOp\n1anWtUkIoq6hZFxajDg2i8AwjOXDfIXgiIi81Y0piIvIW/GCx8sev8xEc/posVoLHujZFumjwcxl\nuVTbGkQmBIZhLAfmKwTvAF4PHAQOAK91bW0RkYyI/ExEdorI/SLyIdd+pojcISIPiciXRCR1Khdw\nqviF56IWgZc15JeZziTjM9JH/UDwYLbFzGXmGjIMYxkxr5HFwH9U1d9Q1WFVXaOqV6jq3jl2LQEv\nU9ULgAuBy0Tk+cBHgb9V1W3AceCqU7yGU8KvNxSNEXjB4sAiCLuGgmJ0/lzGLeYp8FNMzSIwDGMZ\nMN+RxZef6IHVY9KtJt2f4o1FuNG1Xw9ccaLHXkh819AMi6Baj7iGZp3CcoZF4KwGswgMw1gGzNc1\n9CMR+T8i8isi8iz/b66dXDzhbuAwcAvwCDCqqlW3yT68LKRW+14tIjtEZMfIyMg8u3ni+FlDrUpM\n+G/92ZauIScEblL7cDkJixEYhrGcmO/I4he6zz8Ltflv921x1sSFIjIE3AQ8rdVmbfa9BrgGYPv2\n7R0r2pNOeqOLfdcPeK6hWl2ZLHl6FR5QFi49ATCUTbn2eiAqljVkGMZyYk4hEJEY8ElVveFkT6Kq\noyJyG/B8YEhEEs4q2ATsP9njLgTZZJxcKk64pp7/0B+drgTbzDapPTQJgQWLDcNYRswnRlAH3n2i\nBxaRYWcJICJZ4OXALuD7eFlHAFcCN5/osReSNz9vM3/6mvMjbYEQFJwQOKFIhQaahbOGoHV5arMI\nDMNYDszXNXSLiLwX+BIw5Teq6rFZ9lkPXO+yjmLADar6DRF5APhnEfkwXlXTa0+u6wvD+RsGOX/D\nYKTNf7Mfmy4DjYyidDwWjCMoVGqk4rEgthAeXWy1hgzDWE7MVwj8MQPvCrUpcFa7HVT1HuCiFu2P\nAs+dbweXAj9DyHcN+cLgBZEb6aPpZGxGDSJoHyw+NlXmdZ/6MZ9667PZtra/sxdhGIYxT+ZbffTM\nTnfkdMLPEAq7hvx2/4Ffqs5j5rKmGMGeo1M8MjLFroMTJgSGYZw2zBojEJH3hZZf1/Tdn3eqU0tN\nI1jsuYYybr25GF2mRQ0i/zuYaRH4I5X9T8MwjNOBuYLFbwwtf6Dpu8sWuC+nDYFFMF0hFY+RiHs/\nUyoRmzmpfaJVjMC3GqIWwbQvBDbZvWEYpxFzCYG0WW613jUEMYJCJShB4bXHI1lDmdCIY//hXq3V\nKdecEDRZBIWm8hSGYRinA3MJgbZZbrXeNYTHEWRTzfMUhCasScRmjC8Il5VoLjHhC0DB0koNwziN\nmCtYfIGIjOO9/WfdMm4909GeLSG+u2esUGbjUDbUHgtGG5cqNVbkU0FGUTibyN+2OUYwXfb2tbRS\nwzBOJ2YVAlWNz/Z9t+K/5VdqGjzo/fZjU41xBBtmmbBmKJdkZKKEqgajlgst5js2DMNYauZbdK6n\nSIfiAlHXUDxSfbT1PAXeQ35FLkVdoVpveNAKziKwGIFhGKcTJgQt8EtTQ6u5jBujhr1J7ZvmKWiq\nQRR++w9KT1jWkGEYpxEmBC2IWATJpvLUvnunXCOTjM1wDflv+ytyXlXScDzATx81i8AwjNMJE4IW\npOKNnyUzi2som4wH256IRWBZQ4ZhnE6YELQgEY+RiHkB3lauoUqtTrWuZJNeVdJ0pCppY1J7aD3i\nuGRZQ4ZhnEaYELTBd/nMFILGFJZ+RlEmGbUUAFa0sAimbRyBYRinISYEbfCrioazhlKJGKowWfSy\nfzJBMbpwVVJPEBozl83tGpouV3nxR7/Hjx4+0olLMQzDmBUTgja0tgj8gWaVyHeRIHJoLmNvfe5i\ndEcmyuw7XuDBgxMLfh2GYRhzYULQhkAImia1h/A8BX5V0lauofYWQbMQTAXjC6oLexGGYRjzwISg\nDf7bf3OMALzSE+HvouMLasQE+jMJtz7TImhOH/VLT0xbWqlhGEuACUEb/Lf/2VxDmYgQNMYRZJPx\n4Lt26aOqjRHHvgCYEBiGsRSYELTBf/vPpGZaBDOmsEzEgwd+oVIjm4oHbqNw+qj/5l9Xr46Rz1TJ\nBpoZhrF0mBC0wX/IN48shpnB4kwyNqMGkV+mwheIel0pVur0pxPBdj6Ba8jSSg3DWAJMCNowW9bQ\naKFFsDiUNZSJuIb8eQpcEDmfCrbzmQpiBxYsNgxj8TEhaEMQLE41fqJUoskiSIXSR0PF6MKT2vsP\nfN//30oIChYsNgxjCTEhaEMQI2iRNTTePI6gRbA4FhNS8diMYnSrnBCEXUN+jMCEwDCMpcCEoA2z\nZQ21ChZHYgQhS6HYNJrYH18QDgxP2zwFhmEsISYEbWi4hlpkDblxBP56OhELqo8WKzWyyYY10TyF\n5cr8zBHHfoxgumIxAsMwFp+OCYGInCEi3xeRXSJyv4j8nmtfKSK3iMhD7nNFp/pwKrQMFvtZQ9MV\nMslYMAVluilrKOwy8h/404EQpIHmGEH79NFP//ujPHx4cuEuzDAMo4lOWgRV4D2q+jTg+cC7ROQ8\n4P3Araq6DbjVrZ92tI4ReMvjxeoMl1G1rlRrdS9GkApXJY2WlfAtgmiMoHWwuFip8eF/3cXNdz+5\noNdmGIYRZtbJ608FVT0AHHDLEyKyC9gIXA5c7Da7HrgN+INO9eNkefl5aylW64EgAJHlsBCEB4/5\n4wj89maLoDFzWevy1OHJ7iedQPifhmEYnaBjQhBGRLYCFwF3AGudSKCqB0RkTZt9rgauBti8efNi\ndDPCMzcN8cxNQ5G2VEgIWlkKpWo9GEcA3tzHjfRR72G+slXWkPtO1aWfOovCtxSmTAgMw+ggHQ8W\ni0gf8C/Af1PV8fnup6rXqOp2Vd0+PDzcuQ6eAImY4CYua5lWOlWqUqlppDx1sRJ1DfnjCMLxgFYZ\nRAATRV8ILJvIMIzO0VEhEJEkngh8QVW/4poPich69/164HAn+7CQeNNStsgmald6IpRWGgSLW0xh\nORV6+IfjBFPmGjIMYxHoZNaQANcCu1T1b0JffQ240i1fCdzcqT50gnSQGhqOHTRVJQ0Fi5vHEQxk\nk8RjErUCSrVgastWLqNpKz1hGEYH6aRF8CLgN4GXicjd7u/XgY8Al4rIQ8Clbn3Z0G4uY2gMNIu6\nhhoji9OJGPGYkEnEZjzwV/d5aaVhi8B3DU2aa8gwjA7SyayhHwLS5utLOnXeTuO//adbBIuPTzdP\nWBMdcey7k7KphqVQc1VJV/WleOhw9O3fjw1YsNgwjE5iI4tPkFYWQaY5RpBquI9KoaJzuWTDZdQ8\nkb1vEYRdRpMl73gmBIZhdBITghNk9hpEnkXgz0WQScaD8tOFcqMGUTYUO5h2D3lfCKYiQlBznyYE\nhmF0DhOCEyQVbxEsbrIIgmBxIk6lptTqSqFSIxcJInsuI//BP9zvWwRh15C3XKrWqdYaWUYAH/32\ng3z/wWWTcGUYxmmMCcEJMtuk9q2CxQClao3pcqMsRTYZD1xAfkzAL08dDhZPFkOi0FR+4rM/eoxv\n3Xdgga7KMIxexoTgBAnSR1MtXEMzxhH4k9PUKVTqZFOJYN9C04Q1rbKGJltYBwDlap1ipR5kFRmG\nYZwKJgQnSFCMLjHTIhibjs5c1piuskahXG0EixONEcf+A35FPklMosHi8MM/vDxRrLhPEwLDME4d\nE4ITZLaRxf48BZlkCyFokz7qP/hzqQS5VKKtaygcMB537b4gGIZhnAomBCdIozz1zJHFx5tjBIlQ\nVdJQeepsKH3U9/3nUwmyqTiFSvThP5j1RhxHB5qZRWAYxsJhQnCCtEofjceEZFwoV+vEBJJxbxxd\n2CKYLjcmrMm0CBbn0nFyqXi01lC5yrqBDNBkERS85XGzCAzDWABMCE6QdCLq9mluzybjkZnLwA8W\nN6WPuhHH/ujhfCpBNhmf4RpaM+DGF7SIEYybRWAYxgJgQnCCpFrMXAahEceh2IG/zVihgmpjPZuM\nU67WvfEF5Soinqspl4o3BYtrrOnPuOVwjMATgnK1HsyAZhiGcbKYEJwgrUpMhNtbzVPgjzhuWAS+\npVBjypWeEBEXLPYHkdUo1+qsG/QsgnDhuXBsoDlO8PDhCQ6MFU7xKg3D6CVMCE6QVllD0ChCF61B\nFA0i51LRfb3YQZVcOhG0+64h32W0Kp8mJtFidOOFRmygWQje/U+/4M+/+eCpXKJhGD3GokxV2U20\nGkcQbm/GUX3ZAAAbTElEQVTlGvKrkjanlRYqNaZKjdhBLjTQzHcF9WcS5FOJlumjMDOF9NB4kSE3\nt4FhGMZ8MIvgBNkwlCWbjDOUjz5sZ3MNHZ/yXUOJyDZ+NpHfHs4a8t/0+9IJ8ulEyxgBNDKIAOp1\nZaxQCUpdGIZhzAcTghPkFeev5Y4/uoSBTJMQzMM1lG3aplipM12ukg/GFySCYLE/O1lfJkEuHY/M\nWzxeqBJ3kyeHLYKJYpW6YkJgGMYJYUJwgojIDBGA1gPN/FpDwYQ1qagQFPxgcTpsEVRR1cAVlE8n\n6EtHXUMTxUowviAcI/BHNvufhmEY88GEYIFoVZU0EY+RiMmMmct8sSiUa0yXGjWIsqk4dfVGIk+G\nXUOhbCLwYgQbh7JuufH271sexUo9KGFhGIYxFyYEC0Qw4rg5mygRC1w1uRbF6KbLNXLpeOT7QrkW\nxAT8GEE0fbTC+qGZFoEvOGDuIcMw5o8JwQLRKljsrzePI/DFouDSR/OhYDHAdKXW5BqKR4PFhQor\ncinyqXhECMZCD/9m99DBsSIHx4qnfqGGYXQdlj66QLRyDYEnBHV1yy0sgqmQReDPV1AoVxtCkIqT\nC2UN1ete/KA/k6A/k2xyDbW3CP7HjTsB+PxVzzv1izUMo6swIVgg2o44DgWPm7OGJks1ytU6uaSz\nCFz7tHMNZZNxEvFYJFg8VfYygwYySQayiUjWUPjhPzodtQiePG6jjQ3DaI0JwQKRSUbdPj6+pZCM\nC8l4VCyOTZUAyDfFCKbLnmso77KJ8qlEMG+xP5hsIOtZBJGsoVksgiOTJXQBrtMwjO7DYgQLhG8R\npGe4hmZaCv62RyejA82yoWDxZKlGf8YJgROKqXItsAD6M0n6M4mm9NEKw/3pYNmnVK0xXqwyUaxa\nkTrDMGbQMSEQkc+IyGERuS/UtlJEbhGRh9znik6df7FpNU8BNEpRhC2FWExIJ2IccULQsAi8B7/v\nGvLb+5xlMFWqBiOJBzJJZxFE00c3DGZIxWMRi+DYVLnlsmEYBnTWIrgOuKyp7f3Araq6DbjVrXcF\n7YPF3k/sP+R9sql44Bry92m4hqpMFkPZRCEhaFgEiRkWwdh0maFcisFcMuImOjLRetkwDAM6KASq\nejtwrKn5cuB6t3w9cEWnzr/YNIrOxZraWwtENhkP3s7z6SbXkEsf9V1DfWk/uFwNsoQGsg3XkKrn\n/T8+XWEol2Qom4xYBEec4DQvA/zy0AQ/e6z5NhmG0UssdoxgraoeAHCfaxb5/B2j/TiC1gPNMsl4\nKEYwM1g8VY4Gi/123wLozyQYyCQp1+qU3Gxno9NlVuRSrMilIuMIjkyUWi4D/NV3dvPeL+882cs2\nDKMLOG2DxSJytYjsEJEdIyMjS92dOWlVdA4awpBrIQQTLiU0qEqaCGUNFUNC4D4nS9VgLgJPCLz2\n8WIlyCgazCada6hhERwNxQWONsUI9o8WODBWoF63nCLD6FUWWwgOich6APd5uN2GqnqNqm5X1e3D\nw8OL1sGTZeuqHOlEjA2uBpBP8xwEPtnQ+AJfJGIxIZuMBwPK+tO+aygcI6iSScZIJ+L0u+J3E8Vq\nkFa6opVraKJENhknk4xxdDJqERwaL1KpKcemLXZgGL3KYgvB14Ar3fKVwM2LfP6O8cxNQ+z+8CtZ\n66qC+vguo1YWgY//xu9v56V51oN2f+TxlIsR+ALgxxAmitVgVPGKfIqhXDLiGjo6VWZ1f4rVfekg\nUwm8tFJ/3cpPGEbv0sn00S8CPwHOEZF9InIV8BHgUhF5CLjUrXc17VxG4fWwSGRTcUacH7+vySKY\nLNUYL1QDl1DDImhMRjOYTTKUS0UqkB6ZLLEqn2ZVX5ojIYvg8Hhj+YAJgWH0LB0bWayqb2rz1SWd\nOufpSNtgse8OkobVAJ4ojExGhSCbjAfzFoctgoFswyIou4DxilwqmKpyrFAhk4xzZLLMxqEsqhp5\n4IeXDzZNeP+DX47w88eO8d5XnHOKv4BhGKc7p22wuFvwA8AzXEOuPZ9KICJBezaVCN7UfdeQiATz\nFo8XqwxkfdeQ9zleqARzEXjpoymgUYTuyGSJ1X2+a6hhBRwcby0KAF/e8QT/97aHA4ExDKN7MSHo\nMO1GHPvjDXz/v08uGQ8e1n2ZUOzAlaKeKFaC2EA4RuAPIBsKWQSj0xXqdeXYVJnVfWlW9aU4NlUO\nMoR8K2Awm5wRI3j82DR1hSdHrVidYXQ7JgQdplFiomlkcbJhEYTJpeJU3YO6Lx0NKE8FMQLvQd+X\nSiDSiBHEBPrTiYgQjBYq1OrKqr4Uq/rSVOsaDEo7MFakL53gqWv6ItYBwN6j0+5zakF+B8MwTl9M\nCDpMZo7xBc2xg/B6OJvIL0U9UawEweJYTOhLJRgvVhkteOUlYjFhKOe5hkany0G66Oq+NKv7vHbf\n4jg4VmTdYIZ1g5mIRTA2XWHMjVd4/Nj0Kf4ChmGc7pgQdJhGraHWQtDKIvDpCwlBPpVgdLpMqVoP\nYgRAUGbi+HSFIdfuf44WKkHgeZWLEQBByuiBsSLrBjKsG8hwYKwYlKrYe6xhBew5EhWCr+/czys/\n/u9UaxY7MIxuwYSgw/i1hmYOKHNB5OYYQSpqBfjk04kgoDsQih0MZL0KpGOuzpB3jDjJuDA6XQnK\nWAz3pUNC4InDoXHPIlg/mKFQqQWVTX23UCoR4/FjUdfQ93cfZteBcfaYy8gwugYTgg6zZsB7+K4b\njA40a2cRtHMN5dONtFI/W8hbTjBerHDcVR4FL8toKJdirFAOHvqrXLAYvHkQqrU6hydKrHeuIYAD\n415g2HcHPXfrykAUfHYfnADgQfdpGMbyx4Sgw5y9tp8f/sFLufCMoUi7nzXUHCPwp6tMJ2LBjGbg\niYLz3ATjB4BglrLRkEUAnnvo+JRnEcRjwlA2yYpcipjA0ckSRybL1OoaWATQSCF9/Og0q/vSnLuu\n38secsHraq3OQ4cngYYgGIax/DEhWAQ2rcjNaGtkDbUOFofdQs3rzRaBnz66wlkEQFBm4shkiZV5\nL4gcjwkr8ylGJssccKmj6wYyrBv06iMdckKw99gUW1bl2LI6T6nqWQ4Ae45OB+MKdh2ICsEtDxzi\ngg99l+M28Y1hLDtMCJaIdBAjaA4Wu7ISmWh72IU00CQEx6bKTJVrQZAYYDCbYnS6wpHJchAbAFiV\nT3N0shRkCa0bzLCmP41I1CLYsjLHlpWegPnxAN8KOHN1nt2HxiP9+7cHDjFWqPDzPTa3gWEsN0wI\nloh2FoGfNdQcO8iHgsr9mah1MOnKWQ/lGxbBilySsUIlGFXss6ovxdGpcjBuYP1glmQ8xuq+NAfH\nipSqNQ6MF9m8KseWVZ4QPO7iBLsPjhMTeNUz1vPEsUJwXoAde4+5z+Mn83MYhrGEmBAsEdlkm4Fm\nvmuo2SJIRzOFguVMNC4QLOeSHJ8uc3SqFLEI/DITB8eKpBIxVri4wvrBDAfGizxxrIAqbFmVY8NQ\nlnhMgnTSBw9OsHV1ngtcvMO3EI5PlXlkxNtmR5NF8MjIJG+65qczyl8bhnH6YEKwRGTmsAiaYwS+\nEMQkuk/YOojGCLwKpIfGSqzKN1kEk2UOjBVZP5gJ6hytG8hwcKwQpItuXpknGY+xcSgbZA7tPjTB\nuev6OXddv7fuhOCuxz0r4KLNQ9z75FhQ9RTghh1P8JNHj/LNew+c2A9kGMaiYUKwRGxakeWCM4Z4\n5qZoNpEfI8jPCBZ7D//+TDJSpC4sBOGsoUFnHZRrdVb3Ry2CyVKVPUenInMnrB/0BpX5D33fLbRl\nVY69R6eZLld5/Ng056wdYNOKLH3pBLsPenGCHXuPk4gJ73jRmVRqyj37xoLj3vLAIQC+6z7DPDIy\nGQxiMwxj6TAhWCLy6QQ3v+tFnLdhINLe1iJwAhFOHYUm11BICMLWQdgi8OMFDx6cCNJGAdYNZpko\nVnnwwAT5VDzYxxOCKR46NIkqnLOuHxHh7LV97HIWwZ17jnP+xkFe/NTVAEHA+JGRSR4dmWK4P81P\nHjkalK0AuG33YS756x/wzXsPzv1jGYbRUUwITjMaQhB1GfkWQn86GWmPWgTR9FGfsEWwKu8tl6v1\nyCC3dYNe+x2PHWXzqnxgdWxZmWe8WOVnj3kPd98tdO76AXYfnKBcrbNz3yjbt6xgRT7FU9f0cacL\nGPvWwJ+8+jyqdeW23Y2ZST/zoz3u87EZv8FUKAhtGEbnMSE4zfCDxc2uIX99hkXgXEDJuERiB4Oh\nwPHqfMg1FBKF9SHX0LoBbyzBHpc66rPZuYi+c/9BMskYm913567rZ6xQ4dZdhyhV62zfsgKA7VtW\nsGPPMep15ZYHDnHe+gFe9Yz1rO5LB8LwyMgkt/9yhDNX57lz73Hu2TcanO+ux49z0f+6hc/9ZM+c\nv5VhGAuDCcFpxkAmyUAmEfjoffKhGEEY3yIYzKYisYOoRdDaTeQPJAMibqLwuf3lOx8/ztlr+4nF\nvHOcs9azDL5wx+MAPNsXgq0rGS9W+emjR7nr8eNcet5aYjHh0vPWcNvuEUrVGp//yV6SceHaK7eT\nT8W57sd7AG/k8h/fdB/lap2//PZuDoVKY6sqX7lrXzAQzjCMhcOE4DQjk4zz4w9cwuUXbIy0+zGD\ngRlC4K2vyEXbwzGClZEYQcMiiLqGGstnhC0Ct6zaePgDnLvOi2388OEjnLEyyxpnXfiWwUe/sxtV\nuPS8tQD82nnrmCxV+bcHDnPjnft41TPWc9ZwH6999ia+sfMARyZL/ONP9/LAgXHed9k5lGt1Pvyv\nu4LzfeRbD/L7N+zkzf9wx4xU1KOTJSpWDdUwThoTgtOQvnQiePP28ect7p8x4thrDz/4oVGBdCCT\nCCqggud68l1IYSsgk4wHYhK2CHKpBGucO+mcdQ0hGMwlg/23b1kZtG9ZlWN1X5qdT4yycSjL+S4Y\n/oKnrCKXivMnN9/HZKnKlS/cCsBvvXAr5Vqdv7v1If76u7/kV7at5rdf8hR++yVP4es79/Ojh4/w\nD7c/yv+7/VF+7by17B8t8I7rdzBdrlKrK5+87RFe8Bff4zV//0N+eSha9mKiWKFUrWEYxux0bPJ6\nY2EREX7/0rN5kcvMCbf3pRMMNlkEIsJgNhUpWe2zqi9NcbQQsQ7AcxUdn66wZWU+0r5lVY7DE6XA\nCvA5Z10/B8aKgVvIP+/2LSv49v0HefnT1gTuqkwyzsXnDPPNew9ywaZBLtrs7fOU4T5+9exhPveT\nvaTiMf7s8qcjIvz2xU/hq3c/ye9+8RccnSrzqmes5+/edBHfe/Aw7/z8Dt75+TuZLte4c+9xLj5n\nmPueHOM1f/9DPvDKc1k3mOXGO/dx2+7D9GUSvH77GbzleZvZOJRl3/ECjx2ZIpWI8fQNgzN+N8Po\nRUwIlhHvftm2lu1PXdPHtjV9M9qHcklWNlkK4KWQVmt14k1Wx7qBNA8dEjYMRUtmb16Z5+d7jkcs\nAvDcQ7ftHmH71hWR9u1bPSG49Lx1kfZXnL+Ob957MLAGfN7+wq3c/ssR3vmSszhztSdCmWScD/7G\n+bz9sz/nRU9dxd+84QLiMeHS89by4SuewR/edC8DmQQfe8OFXH7hBo5MlnnfjTv54NcfAGC4P83b\nXriV/WMFrv3hY1xz+6Mk40KlFh23sHlljg1DGQRBBJLxGPl0nHzKm/Jz7UCGtQOZYIa4qVKVRDwW\n/Ob5tFcG/MBokWKlxtlr+yMVZet1ZbpSm5EObBinE/avswu44Z0vICYyo/1Nz91Mf4sH0HO2rmRk\nYmbJhxc8ZRXxmJCIRz2Gv/6MddTqdYb7oxbE67ZvIh6Ds9dEBeINzzmDbCrOC5+yKtL+6mduIJdK\ncMm5ayLtF58zzBf/8/N5TpOgvPScNdz8rhdx9tr+iHvrzc/bzNZVOZ6ypi8YFDfcn+Yzb3sO37n/\nEOlEjF/Ztjq4joNjRW688wkmSzXOGs5z1uo8hUqNe58c4959YxydLKN4MYapUpUnR2tMlaocm/Jm\nhJuNXCrOdLnhforHhG1r+ti0IssTxwo8dnSKcrXOGSuzXHjGCs5cnWffsWkeHpnk4FiRs9f2c8EZ\ng2xb08/+sQIPH5pk3/ECW1bleMamQc5e28/IRIlHRiZ54liB9YMZnrZ+gLPX9jFaqPDYkSn2HZtm\nZT7FtrX9nDWcZ7pcY9/xAk8eL9CXSbB1VY4tq/JUa3UOjBU5MFYknYixcUWWjUNewsDIRImRyRIx\nEdYOeJMYxUUYL1Y4NlWmrt4LxEAmSSwmlKreREZ1VQYySTLJGCKCqlKs1KnU6+RTicjLhqpSq+uM\nf1/G0iPLYWTn9u3bdceOHUvdDaPHUFXGChUOjheZLntv9fl0glKlxkOHJ3no0ARHp8qsH8y44n3C\n/fvHuWffGAfGCmxemees4TwDmQT37x9n5xOj7HfTg25b28dwf5rdByd48OAENTfnw/rBDBuGsjx2\nZIpjoZLeIrCmPx3MI9FpRECA5lPFY0IyLhQrUYFMxWOkEzGmytXIPrlUnFQiRrFSC/ZJxIRsKk46\nEaNUrVOu1qnW1TtGMkYqHqNaVyquPREXUnFvfo6aExPfok3EYyRiQl2VWt27ZyJCwpVdB6irUldF\nEGLizfXd/NgTIXiZUhpf+pZiy98o2Lf1BtJ2pX1zq2N95srnBGncJ4qI3Kmq2+faziwCw2iDP9Pb\nUAv32lnDfbzi/HUz2i97+vpZj1mq1iLWDUCxUmPv0WnWD2WCrDBV5cnRAg8fnmTtQIYzV+fJJOMU\nKzUePjzJQ4cnGMqlOHNVno0rshyfKvPQ4UkeHZkkn06waUWOjSuyTBa9ciJ7j06RisdYN5hl/WCG\nUrXOvuPTPHm8QCwm3lSm/SlU4dB4iYPjRep1ZWU+xYp8EkE4OlXm2FSJSk0ZyCQYzHrWwXihymih\nTLnqWQH5dIJETJgqV5ksVilV62RTcTLJOMmYUKjUmC7XKNfqwcM/EfPcdsVKjUqtTiLmPfjjMajW\nlXK1TsV/+MdixGPiCUK9TrWmxESIxbwHfV09l1y1ru4B7z3QAWqq3kRL0njIq3oP/3pdgwexAOru\ngzLzge1LRbv36HBzu5ftGa3qf2jQX/CmjO00S2IRiMhlwMeBOPBpVf3IbNubRWAYhnHizNciWHRn\nnYjEgU8ArwTOA94kIuctdj8MwzAMj6WI2jwXeFhVH1XVMvDPwOVL0A/DMAyDpRGCjcATofV9ri2C\niFwtIjtEZMfIyMiidc4wDKPXWAohaBU/nxk3Ub1GVber6vbh4eFF6JZhGEZvshRCsA84I7S+Cdi/\nBP0wDMMwWBoh+DmwTUTOFJEU8Ebga0vQD8MwDIMlGEegqlUReTfwHbz00c+o6v2L3Q/DMAzDY0kG\nlKnqN4FvLsW5DcMwjCjLosSEiIwAe09y99XAkQXsznLArrk3sGvufk71ereo6pzZNstCCE4FEdkx\nn5F13YRdc29g19z9LNb1WhlAwzCMHseEwDAMo8fpBSG4Zqk7sATYNfcGds3dz6Jcb9fHCAzDMIzZ\n6QWLwDAMw5gFEwLDMIwep6uFQEQuE5HdIvKwiLx/qfuz0IjIGSLyfRHZJSL3i8jvufaVInKLiDzk\nPlfMdazlhojEReQXIvINt36miNzhrvlLrnxJ1yAiQyJyo4g86O73C7r9PovIf3f/ru8TkS+KSKbb\n7rOIfEZEDovIfaG2lvdVPP7OPc/uEZFnLVQ/ulYIemQCnCrwHlV9GvB84F3uGt8P3Kqq24Bb3Xq3\n8XvArtD6R4G/ddd8HLhqSXrVOT4OfFtVzwUuwLv2rr3PIrIR+F1gu6o+Ha8czRvpvvt8HXBZU1u7\n+/pKYJv7uxr45EJ1omuFgB6YAEdVD6jqXW55Au/hsBHvOq93m10PXLE0PewMIrIJeBXwabcuwMuA\nG90mXXXNIjIA/CpwLYCqllV1lC6/z3glcLIikgBywAG67D6r6u3Asabmdvf1cuBz6vFTYEhEZp8k\ne550sxDMawKcbkFEtgIXAXcAa1X1AHhiAaxZup51hI8B7wPqbn0VMKqqVbfebff6LGAE+Kxzh31a\nRPJ08X1W1SeB/w94HE8AxoA76e777NPuvnbsmdbNQjCvCXC6ARHpA/4F+G+qOr7U/ekkIvJq4LCq\n3hlubrFpN93rBPAs4JOqehEwRRe5gVrh/OKXA2cCG4A8nmukmW66z3PRsX/n3SwEPTEBjogk8UTg\nC6r6Fdd8yDcZ3efhpepfB3gR8BsisgfP3fcyPAthyLkQoPvu9T5gn6re4dZvxBOGbr7PLwceU9UR\nVa0AXwFeSHffZ59297Vjz7RuFoKunwDH+cavBXap6t+EvvoacKVbvhK4ebH71ilU9QOquklVt+Ld\n0++p6luA7wOvdZt12zUfBJ4QkXNc0yXAA3TxfcZzCT1fRHLu37l/zV17n0O0u69fA37LZQ89Hxjz\nXUinjKp27R/w68AvgUeAP1rq/nTg+l6MZxreA9zt/n4dz2d+K/CQ+1y51H3t0PVfDHzDLZ8F/Ax4\nGPgykF7q/i3wtV4I7HD3+qvAim6/z8CHgAeB+4DPA+luu8/AF/FiIBW8N/6r2t1XPNfQJ9zz7F68\njKoF6YeVmDAMw+hxutk1ZBiGYcwDEwLDMIwex4TAMAyjxzEhMAzD6HFMCAzDMHocEwKjpxCRSfe5\nVUTevMDH/sOm9R8v5PENo1OYEBi9ylbghITAVbSdjYgQqOoLT7BPhrEkmBAYvcpHgF8Rkbtd3fu4\niPyViPzc1Xp/J4CIXOzmfPgnvEE8iMhXReROVyv/atf2EbxKmXeLyBdcm299iDv2fSJyr4i8IXTs\n20LzDHzBjaI1jEUlMfcmhtGVvB94r6q+GsA90MdU9TkikgZ+JCLfdds+F3i6qj7m1t+hqsdEJAv8\nXET+RVXfLyLvVtULW5zrP+CNDL4AWO32ud19dxFwPl7NmB/h1VL64cJfrmG0xywCw/D4Nbw6Lnfj\nlfJehTcBCMDPQiIA8LsishP4KV4RsG3MzouBL6pqTVUPAT8AnhM69j5VreOVCNm6IFdjGCeAWQSG\n4SHA76jqdyKNIhfjlX0Or78ceIGqTovIbUBmHsduRym0XMP+TxpLgFkERq8yAfSH1r8D/LYr642I\nnO0mf2lmEDjuROBcvClCfSr+/k3cDrzBxSGG8WYb+9mCXIVhLAD29mH0KvcAVefiuQ5vTuCtwF0u\nYDtC62kQvw38FxG5B9iN5x7yuQa4R0TuUq80ts9NwAuAnXjVYt+nqgedkBjGkmPVRw3DMHoccw0Z\nhmH0OCYEhmEYPY4JgWEYRo9jQmAYhtHjmBAYhmH0OCYEhmEYPY4JgWEYRo/z/wOx4ed3Dk+IZAAA\nAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1fc17207470>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def momentumM(m, b, p_m, beta):\n",
    "    gradient_m = gradientM(m, b)\n",
    "    momentum_m = gradient_m + beta * p_m\n",
    "    return momentum_m\n",
    "# end of def momentumM\n",
    "\n",
    "def momentumB(m, b, p_b, beta):\n",
    "    gradient_b = gradientB(m, b)\n",
    "    momentum_b = gradient_b + beta * p_b\n",
    "    return momentum_b\n",
    "# end of def momentumB\n",
    "\n",
    "def gradientStepWithMomentum(alpha, m, b, p_m0, p_b0, beta):\n",
    "    # compute the momentum values of m and b\n",
    "    p_m = momentumM(m, b, p_m0, beta)\n",
    "    p_b = momentumB(m, b, p_b0, beta)\n",
    "    \n",
    "    # compute new values of m and b\n",
    "    m = m - alpha * p_m\n",
    "    b = b - alpha * p_b\n",
    "    \n",
    "    return m, b, p_m, p_b\n",
    "# end of def gradientStepWithMomentum\n",
    "\n",
    "# iteration count\n",
    "N = 101\n",
    "\n",
    "# learning rate\n",
    "alpha = 0.01\n",
    "\n",
    "# momentum coefficient\n",
    "beta = 0.9\n",
    "\n",
    "# initial values\n",
    "m = b = 0\n",
    "p_m = p_b = 1\n",
    "\n",
    "# lists to keep the error and the iteration values.\n",
    "errors = []\n",
    "Ns = []\n",
    "\n",
    "# Let's start iterating\n",
    "for i in range(N):\n",
    "    # append these value for plotting grapghic\n",
    "    errors.append(computeError(m, b))\n",
    "    Ns.append(i)\n",
    "    \n",
    "    if i % 10 == 0:\n",
    "        print(i, \"\\t error:\", computeError(m, b))\n",
    "        \n",
    "    m, b, p_m, p_b = gradientStepWithMomentum(alpha, m, b, p_m, p_b, beta)\n",
    "# end of for loop\n",
    "\n",
    "# drawing of graph.\n",
    "plt.title(\"Error distribution for %s iteration\" % N)\n",
    "plt.ylabel(\"Error\")\n",
    "plt.xlabel(\"Iteration\")\n",
    "plt.plot(Ns, errors)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient Descent with adaptive alpha"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 \t error: 60.01569999999999\n",
      "10 \t error: 3.934350308414748\n",
      "20 \t error: 6.90880324724523\n",
      "30 \t error: 7.393997601591168\n",
      "40 \t error: 4.72413001453322\n",
      "50 \t error: 3.129228958275091\n",
      "60 \t error: 2.5057963081608374\n",
      "70 \t error: 2.0123751845498736\n",
      "80 \t error: 1.7734996712392657\n",
      "90 \t error: 1.76418748811383\n",
      "100 \t error: 1.6437117355804072\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmcXHWZ6P/PU3tX7+l09o1AWBJ2wyIiMgIjIgq/O+6K\nUXEY73WbUa8XHWd01JnB8Y6jzjB6ERBUxAVFEEVFkB0CYQuEkITsSac7ne5O713r8/vjnOpUd9fa\n3ZVO6jzv16tfXXWqzjnf05Wcp57vKqqKMcYY7/LNdAGMMcbMLAsExhjjcRYIjDHG4ywQGGOMx1kg\nMMYYj7NAYIwxHmeBwEyZiKiIHOc+/p6I/MM0HXeJiAyIiN99/qCIfGQ6ju0e714RWTNdx8s6bo2I\n/EZEekXkF9N9/Eqazs9vCmXYICIXzmQZvMYCwVFERHaIyLB7c8z8/NdMlyubqn5UVb9a7H3utVxc\n5Fi7VLVOVVNTLZeIfFlEfjzu+G9W1Vuneuwc3g7MBVpU9R1TPZiIhETkDvdvpuNvkuL4uoh0uT//\nJiKS9foNIrJJRNIi8sFC58r+/ETkQhHZM9XyFyIit4jI18aVYZWqPljJ85qxLBAcfd7q3hwzPx/P\n9SYRCZSyrZBy3z+dZvLc02ApsFlVk+XuWOC6HwXeD7TneO0a4ErgNOBU4HLgb7JefwH4X8Cz5ZZn\nKo7yz9BbVNV+jpIfYAdwcZ7XPgg8BvwH0A18Lc82H/BFYCewH/gh0OgeYxmgwNXALuDhPOf638A+\noA34sLvPce5rtwBfcx/PBu4BDrrnf8Q9/4+ANDAMDACfy3XurG0B93gPAv8KPAX0AncBs9zXLgT2\n5Pp7AZcCcSDhnu+FrON9xH1cyt9ljVu2A8Df5/nb/NO4c109HX/zrOPvAS4ct+1x4Jqs51cDT+bY\n91Hgg0WOf4v776TW/XzS7nUMAAvca7kW2Ap0AT/P+gxyXgvwC5wA1ut+rqvc7de4f6e4e/zfjP93\nDoSBb+H8W2tzH4ezP3PgM+7fdR/woZn+f3o0/lhGUF3OAbYBc4B/zrPtg+7PXwDLgTpgfPXSG4CT\ngDeNP4GIXAp8FrgEWIFzo83nMzj/UVtxqkq+AKiqXoVzo8hkN/9WyrldH8AJPguAJPCdAucH54S/\nB/4F+Jl7vtNyvO2DFP+7nA+cAFwE/KOInJTjXF8ad66bSjx2sesuZBXOt/6MF9xtk6aqg8CbgTY9\nlH22AZ/EyT7egPMZ9ADXj9t9/LXci/NvZQ5OVnKbe44b3Mf/5h7/rTmK8vfAucDpOBnP2ThBNWMe\n0AgsxAlA14tI8xQu3ZMsEBx9fi0iB7N+/jrrtTZV/U9VTarqcJ5t7wO+qarbVHUA+Dzw7nFp/JdV\ndTDrGNneCfxAVV9ybxZfLlDWBDAfWKqqCVV9RN2vcgUUOjfAj7LO/Q/AOzONyVNUyt/ln1R1WFVf\nwLnZ5gookz12sesupA7n23ZGL1CX3U4wjf4GJxvao6oxnM//7YWuRVVvVtX+rPefJiKNJZ7vfcBX\nVHW/qnbiZFxXZb2ecF9PqOrvcDKLE6ZygV5kgeDoc6WqNmX9fD/rtd053j9+2wKcKoqMnUAA5xt7\noeNk75/9+s58bwS+AbwK/FFEtonItQXeW8q5x7++EwjiVEFNVSl/l+z6+SGcG/B0HbvYdRcyADRk\nPW8ABkoIupOxFLgz80UE2AikyHMtIuIXketEZKuI9OFU+0Dpn1muv92CrOddOrYtppzPxbgsEFSX\nXP/xx29rw/nPnLEEp4qlo8hxMvYBi8ftn7swzrfAz6jqcuCtwKdF5KIi5yh28xp/7gROnf0gEM28\n4GYJrWUct5S/y2RN9W9ezAbGZienudumKleZdgNvHvdlJKKqe/Ps917gCpwqxEacdgQAyfHeXHL9\n7dpKLL8pkQUC77kd+DsROUZE6jhUn11qD5efAx8UkZUiEgW+lO+NInK5iBznVlH04XxzzHQF7cCp\nLy/X+7PO/RXgDnW6l24GIiLyFhEJ4tQjh7P26wCWiUi+f/NT/bsUMuVji0hYRCLu05CIRLKqfn6I\nE2QXisgCnLaZW7L2Dbn7ChB09y3l/34H0DKuGud7wD+LyFL32K0ickWBY9QDMZyG5SjOtY8/R6F/\nB7cDX3TPMxv4R+DHBd5vJsECwdHnN+PGEdxZ5v434/TaeRjYDowAnyh1Z1W9F6fnxgM41T4PFHj7\nCuBPOFUXTwD/rYf6h/8rzn/wgyLy2TLK/yOcm1w7EMFpvERVe3G6SN4I7MXJELL7wGcGdnWJSK5u\nlFP6uxQxHcfehNOLZyHwB/dx5pvy/wN+A7wIvAT81t2W8Uf3/ecBN7iPLyh2QlV9BedGvM39nBYA\n3wbuxqnu6weexOmQkM8Pcapz9gIvu+/PdhOw0j3+r3Ps/zVgHbDevb5n3W1mGkllqhGNMcYcLSwj\nMMYYj7NAYIwxHmeBwBhjPM4CgTHGeNxRMSnU7NmzddmyZTNdDGOMOao888wzB1S1tdj7jopAsGzZ\nMtatWzfTxTDGmKOKiBQa+T/KqoaMMcbjLBAYY4zHWSAwxhiPs0BgjDEeZ4HAGGM8rqKBQESa3EW3\nXxGRjSLyWhGZJSL3icgW97etJmSMMTOo0hnBt4Hfq+qJOHOkb8RZ7/R+VV0B3O8+N8YYM0MqFghE\npAFnqtubAFQ1rqoHcRapuNV92604659WxJ3P7eG2tSV1ozXGGM+qZEawHOgEfiAiz4nIjSJSC8xV\n1X0A7u85uXYWkWtEZJ2IrOvs7JxUAe55YR8/WbtrksU3xhhvqGQgCABnAt9V1TNwFgopuRpIVW9Q\n1dWqurq1tegI6Zyi4QBD8VTxNxpjjIdVMhDsAfao6lr3+R04gaFDROYDuL/3V6oAtSE/g7HpWGnQ\nGGOqV8UCgaq2A7tF5AR300U4S9XdDaxxt60B7qpUGaIhywiMMaaYSk869wngNhEJAduAD+EEn5+L\nyNXALuAdlTp5bdjPYDyJqnJonW9jjDHZKhoIVPV5YHWOly6q5HkzoqEAqjCSSFMT8h+OUxpjzFGn\nqkcW14adm/9g3NoJjDEmn6oOBNGQk/AMxaydwBhj8qnqQFAbsozAGGOKqepAEA27GYEFAmOMyauq\nA8FoRmBVQ8YYk1dVB4LRNgLLCIwxJq+qDgSjvYYsIzDGmLyqOhBYRmCMMcVVdSA4NI7AMgJjjMmn\nqgNBJOBHBIZs4jljjMmrqgOBzydEg37LCIwxpoCqDgSQWZPAMgJjjMmn6gOBsyaBZQTGGJNP1QcC\nZ00CywiMMSafqg8EtWHLCIwxppCqDwSWERhjTGFVHwicVcosIzDGmHyqPhBEQwEbR2CMMQVUfSCo\nDVlGYIwxhVR9ILBxBMYYU1jVB4LakJ9ESokn0zNdFGOMOSJVfSCwGUiNMaawqg8ENgOpMcYUVvWB\nYDQjsJ5DxhiTU9UHAssIjDGmsEAlDy4iO4B+IAUkVXW1iMwCfgYsA3YA71TVnkqVwTICY4wp7HBk\nBH+hqqer6mr3+bXA/aq6ArjffV4xtW4gsIzAGGNym4mqoSuAW93HtwJXVvJkUbdqyHoNGWNMbpUO\nBAr8UUSeEZFr3G1zVXUfgPt7Tq4dReQaEVknIus6OzsnXYDRjMBmIDXGmJwq2kYAvE5V20RkDnCf\niLxS6o6qegNwA8Dq1at1sgWwjMAYYwqraEagqm3u7/3AncDZQIeIzAdwf++vZBmiQbfXkGUExhiT\nU8UCgYjUikh95jHwl8BLwN3AGvdta4C7KlUGgIDfRzjgs4zAGGPyqGTV0FzgThHJnOcnqvp7EXka\n+LmIXA3sAt5RwTIAUBsOMGiBwBhjcqpYIFDVbcBpObZ3ARdV6ry5REN+hqxqyBhjcqr6kcXg9Byy\njMAYY3LzRCCIhv0M2YAyY4zJyROBoDYUYNCmmDDGmJw8EQiiIcsIjDEmH08EAus1ZIwx+XkiEFiv\nIWOMyc8TgcAyAmOMyc8TgSAa8jOSSJNKT3rKImOMqVqeCAS1toC9Mcbk5YlAkJmBdNh6DhljzASe\nCAS2SpkxxuTniUAQDWWmoraqIWOMGc8TgaA2nGkjsIzAGGPG80QgGM0IrLHYGGMm8EQgGM0IbFCZ\nMcZM4IlAYBmBMcbk54lAMDqOwBqLjTFmAk8Egsw4Aus+aowxE3kiEIT8PgI+sZHFxhiTgycCgYgQ\nDfkZtMZiY4yZwBOBAJyeQ5YRGGPMRJ4JBNGQ39oIjDEmB88EgtpwwHoNGWNMDp4JBJYRGGNMbp4J\nBLUhayMwxphcKh4IRMQvIs+JyD3u82NEZK2IbBGRn4lIqNJlAIjYusXGGJPT4cgIPgVszHr+deA/\nVHUF0ANcfRjKQCTgZyRhgcAYY8araCAQkUXAW4Ab3ecCvBG4w33LrcCVlSxDRiToYySZPhynMsaY\no0qlM4JvAZ8DMnfgFuCgqmYq6/cAC3PtKCLXiMg6EVnX2dk55YJEgpYRGGNMLhULBCJyObBfVZ/J\n3pzjrZprf1W9QVVXq+rq1tbWKZcnEvQxkkihmvN0xhjjWYEKHvt1wNtE5DIgAjTgZAhNIhJws4JF\nQFsFyzAqEvCTVkimlaA/VzwyxhhvqlhGoKqfV9VFqroMeDfwgKq+D/gz8Hb3bWuAuypVhmyRoDMD\nqVUPGWPMWDMxjuD/AJ8WkVdx2gxuOhwnjQSdSx1JWIOxMcZkq2TV0ChVfRB40H28DTj7cJw3W9gy\nAmOMyckzI4szVUOxpAUCY4zJ5p1AELCqIWOMycU7gcCqhowxJicPBoLyM4KugRh3v3BYerkaY8xh\n55lAEB6tGio/I7j9qV188vbn2N83Mt3FMsaYGeeZQDCaEUyisXhn1xAA2w4MTmuZjDHmSOChQDD5\nxuJd3U4g2GGBwBhThTwUCCbfWLynZxiA7V0WCIwx1cc7gSCQGUdQXkYQT6bZ1+sEAssIjDHVyDOB\nIBycXGNx28Fh0goisOPAUCWKZowxM8o7gSDgQwRiZQaC3T3Ozf+UhY3s6BoknbZprI0x1cUzgUBE\nCAfKX6Vsd7dTLXTBilZiyTTt1oXUGFNlPBMIYHKrlO3qHiLoF85ZPguwdgJjTPXxViCYxAL2u3uG\nWNQcZXlrHWA9h4wx1cdbgSDoK3scwe7uIRY11zC/IUI44LOMwBhTdYoGAhHxi8g3DkdhKm0yVUO7\nu4dYMiuKzycsbYmy3XoOGWOqTNFAoKop4DUictQv9BsO+stqLO4fSdAzlGDxrCgAy1pq2WFVQ8aY\nKlPqCmXPAXeJyC+A0Tuhqv6qIqWqkEjAV1ZGkOkxtLjZCQTHzK7lwU2dpNKK33fUx0VjjAFKDwSz\ngC7gjVnbFDiqAkE46Kd3KF7y+zNjCJZkMoLZtcRTadoODo9mCcYYc7QrKRCo6ocqXZDDIRLw0VFG\nY/Fud7K5xbNqAKdqCGBH16AFAmNM1Sip15CILBKRO0Vkv4h0iMgvRWRRpQs33SJBf1nTUO/uHqI+\nEqCxJgg4VUNgYwmMMdWl1O6jPwDuBhYAC4HfuNuOKpGgj1g5GUHPMIubo2Tayec2hKkJ+q3nkDGm\nqpQaCFpV9QeqmnR/bgFaK1iuiig3I9jVPTRaLQTONBVLW6LWc8gYU1VKDQQHROT97pgCv4i8H6fx\n+KhSzjgCVWVPz9BoQ3HGMbNrrWrIGFNVSg0EHwbeCbQD+4C3u9uOKk730TSqxWcQ7RyIMZJIT2gU\nXja7ll3dQyRT5a90ZowxR6KivYZExA/8laq+rZwDi0gEeBgIu+e5Q1W/JCLHAD/F6ZL6LHCVqpbe\np3MKwsFDi9NkVizL51CPobGBYHFzlGRa6RyIMb+xJteuxhhzVCl1ZPEVkzh2DHijqp4GnA5cKiLn\nAl8H/kNVVwA9wNWTOPakZG7+pTQYjx9MlpHpQdQ3nJzm0hljzMwotWroMRH5LxF5vYicmfkptIM6\nBtynQfdHcQal3eFuvxW4cjIFn4zRBexLaDDee9AJBIuax37rb6hxkqi+kcQ0l84YY2ZGqSOLz3N/\nfyVrW+amnpdbrfQMcBxwPbAVOKiqma/Te3C6o+ba9xrgGoAlS5aUWMzCMusWl9Jg3DecIBzwTahC\naogER183xphqUEobgQ/4rqr+vNyDu9VKp4tIE3AncFKut+XZ9wbgBoDVq1dPy/qQmZt6KVNRD8SS\n1Ecm/nkaMlVDlhEYY6pEKW0EaeDjUzmJqh4EHgTOBZpEJHOHXQS0TeXY5YiUsYD9YCxJbThHIHCD\nQ/+ItREYY6pDqW0E94nIZ0VksYjMyvwU2kFEWt1MABGpAS4GNgJ/xul+CrAGuGuSZS/boYygeCAY\niCWpDU0MBPVWNWSMqTKlthFkxgx8LGubAssL7DMfuNVtJ/ABP1fVe0TkZeCnIvI1nOmtbyqzzJN2\nqLG4tKqhuhwZQSjgoybop88yAmNMlSh19tFjyj2wqq4HzsixfRtwdrnHmw7hMhqLB2MpZteFcr5W\nHwlYRmCMqRoFq4ZE5HNZj98x7rV/qVShKiWTEcRKyAjytRGA02BsjcXGmGpRrI3g3VmPPz/utUun\nuSwVV05GkK/XEDgNxjagzBhTLYoFAsnzONfzI96hkcUl9hrK0VgMlhEYY6pLsUCgeR7nen7EO9R9\ntHDVUDqtDMZT+auGIkHrPmqMqRrFGotPE5E+nG//Ne5j3OeRipasAkrtPjoYd27yuXoNgTPNhDUW\nG2OqRcFAoKqFp+g8ygT9Pvw+KTrX0GDMeb1QRtA3kkBVR1cvM8aYo1WpA8qqRmZNgkIGYk5GUBvO\nHQfrI0ESKS1pqgpjjDnSeS8QlLBK2aAbCPL2GrIZSI0xVcSjgaDwN/lMIMjba8immTDGVBHPBYJw\n0Fe0jaB/tGoof/dRsIzAGFMdPBcIIgF/0XEEmYwgb6+hSKZqyLqQGmOOft4LBMHijcWDpWYEVjVk\njKkCHgwEfmJFqoYG3O6j+TKCessIjDFVxHOBIFxC99HBWBKfHBqJPN5UGov394+UvY8xxlSS5wJB\nKd1HM2sR5BssFgn6CQV8ZTcW7zgwyDn/cj+3rd1Z1n7GGFNJ3gwERauGci9Kk60hEix7BtJtBwZQ\nhevufYXO/lhZ+xpjTKV4MBCUVjWUr6E4o6EmUHZG0N7r3PwHYkn+9Xcby9rXGGMqxXOBIBworWqo\naCCYxAyk7X0jiMA1FyznV8/t5cltXWXtb4wxleC5QBAJ+omVkBEUrRqqCZbdWNzRO8LsujB/e9Hx\nLGqu4Yu/fol4CaulGWNMJXkwEPiIp9Kk0vmXUxiMpfJOOJdRH5lE1VDfCPMbI9SE/Hzprat4df8A\n973cUdYxjDFmunkwELirlBVoMHYai4MFjzOZxuL23hHmNjjLOJy7fBYAew8OlXUMY4yZbt4LBIHi\nq5Q5gaBwRjCpxuK+Eea5gaAuHCAS9LG/z3oPGWNmlvcCQZFVylS1tF5DkSDxZLpow3PGSCJF73CC\neY1OIBAR5tRH6BywQGCMmVkWCMaJJdMk01pC99HyZiBt73VGFGeqhgBa68OWERhjZpwHA0HhqqFi\nM49mZGYgLbULaXufEwjmNx4KBHPqw5YRGGNmXMUCgYgsFpE/i8hGEdkgIp9yt88SkftEZIv7u7lS\nZcglXKSxuNh6xRnlzkDa0ZcvI7C5h4wxM6uSGUES+IyqngScC3xMRFYC1wL3q+oK4H73+WETCWSq\nhnJnBP0x58ZeakZQ6gyk+9yqoXnjMoK+kWTJ7QzGGFMJFQsEqrpPVZ91H/cDG4GFwBXAre7bbgWu\nrFQZchmtGiqSEZQy1xCUnhG0945QFw6MOW5rfRjA5h0yxsyow9JGICLLgDOAtcBcVd0HTrAA5uTZ\n5xoRWSci6zo7O6etLGE3I8i3StmhRWmKdR8tr7G4o2+EuQ3hMdvm1DvZgbUTGGNmUsUDgYjUAb8E\n/lZV+0rdT1VvUNXVqrq6tbV12spTrLF4oOTG4kxGUHpjcXa1EBzKCKznkDFmJlU0EIhIECcI3Kaq\nv3I3d4jIfPf1+cD+SpZhvGLdR4stU3noOD6Cfik9I+gdYV5DzZhtczJVQ5YRGGNmUCV7DQlwE7BR\nVb+Z9dLdwBr38RrgrkqVIZdigWCgxEAgIu4MpMUDQSqt7O+PMa9xbNXQrNoQItBpPYeMMTOo8N1u\nal4HXAW8KCLPu9u+AFwH/FxErgZ2Ae+oYBkmONRYPLWqIXAnniuhaqhrIEYyraPTS2QE/D5aam0s\ngTFmZlUsEKjqo0DutR7hokqdt5hD3UfzVw3VBP34ffmKfkhDTbCkqqH2HGMIMmx0sTFmpnluZLHP\nJ4T8+VcpG4ililYLZTgzkJYQCHKMIchotdHFxpgZ5rlAABAO+gpmBMVmHs1wZiAtXjWUGVU8vmoI\nnAZjywiMMTPJk4EgEvQXmGKi+MyjGSVnBH0jBHxCS114wmut9WEODMRIF1goJ5eugZiNSDbGTAuP\nBoL8VUP95QSCmtLWLd7XO8Kc+nDOdoc59WGSaaVnKF7SOcGZKvut//ko//zbjSXvY4wx+XgzEAQK\nZwT1JWcEAYYTqaLrDnf0jTA3R/sAZE0zUUY7QWd/jLbeEX6zvo1EytY8NsZMjTcDQdBfcBrqUjOC\nend0cbGxBO29IznbB+DQNBPltBNs6ugH4OBQgkdfPVDyfsYYk4tHA0H+xuKyeg3VOO/rLdJO0NEX\ny9l1FCY38dymdicQ1AT9/Hb9vpL3M8aYXDwaCPzT0muoKRoCoGcofyAYiCUZiCXHLEiTbTLTTGzp\nGKClNsRlp8znDxva81ZzGWNMKTwZCMKB3I3FqbQynCg9I2h2A0HvcP6G3kJjCMCZyiIa8pddNXT8\n3HouP20+/SNJHtls1UPGmMnzZiAI+nOuR1DO9BIATe5U1D2D+TOC/f1OIMhUAeVSzpKVqsqWjn5O\nmFfP+cfNprEmyG9ftOohY8zkeTIQ1AT9jMQnBoJS1yvOaB6tGsqfEWSCREtt/kBQzpKVew8OMxhP\ncfzceoJ+H5eumsd9L3fYmAJjzKR5MhA01gRzjggudQrqjPpIAJ84vXfy6XaDRHNtMO975tRHSs4I\nNrs9ho6fWwfA5afNZyCW5KHN07d4jzHGWzwZCJpqggzEkhP64JdbNeTzCU3RUJGMwA0EbvaQS2t9\nmM4S2wg2tQ8AsGJuPQCvXd5CUzTIAxsP67IOxpgq4s1AEHW+nY/v9plZr7jUjCBzrIMFuo92D8ap\njwQI+vP/qVvrw/THkgznqK4ab3NHP/MbIzS67RMBv49jW+vY1T1UcpmNMSabJwNBo/vtfHyVzkCJ\n6xVna46GOFgoIxiKM6s2fzYA5Y0l2Oz2GMq2oKmGtt7hEkprjDETeTIQZHr7jO/2WW7VUOZYhXoN\ndQ/GC1YLQfY0E4UbjFNpZcv+gdH2gYyFTTXsOzhS9sR1xhgDXg0EbtXQ+Iyg3F5DzrGmnhHMKXER\n+51dg8ST6QkZwcKmCPFUmgO2roExZhK8GQhqilUNlR4ImqPBgiOLewYTJWcE+4tUDW3ucBqKT5g3\nLhA01wCw56BVDxljyufJQNCYyQgmNBYnCfiEcKD0P0tzbYjhRCpvP/7uwTizCnQdBWeMgU+KtxFs\n7uhHBI6bM7ZqaEGTEwjaLBAYYybBk4GgPhxABHrHVen0DMVprg0hUny94ox8PZAAhuMphhMpmotU\nDfl9wuy68Ogo5Hw2dfSzuDlKNDQ2Y1noBoK9PRYIjDHl82Qg8PmExpqJ3T4PDMRpKXLTHq/Q6OLM\ntllFqoYA5jSEi1cNtU/sMQTOdNj1kYBlBMaYSfFkIACnt8/4NoKugRizcywnWew4kHu+oe7MYLIS\ngsuc+kjBxuJ0WtnRNTihWihjYVMNey0QGGMmwbOBoDEampARdA3GaakrLyNoGh2TUCAjKCEQtNYV\nnnjuwGCMREpZ0JR7FlMnEJQ2X9F41u3UGG/zbCBoqglOaCPoGogXnBwul8wcQrl6DnWXML1ExpyG\nMF0DMVJ5bsodvU6QyLfAzcLmGvb2lDe6eP2eg1z6rYd5z/efLGs/Y0x1qVggEJGbRWS/iLyUtW2W\niNwnIlvc382VOn8x46eGGEmkGIgly84ICrYRDJaeEcypD5NWp3oql3Z3dtJ8C9wsaKqhbyRZdNlM\ncK71untf4crrH2NzRz9rt3fnPa8xpvpVMiO4Bbh03LZrgftVdQVwv/t8RoxvI8gMxppdZiCIBP1E\ngr6cvYa6hxKIMDovUCGtmbWL8zQYZwJBvrWPF452IS1ePfSPd73E9x7ayjtes5gffOhsAJ7c1l10\nP2NMdapYIFDVh4Hxd5crgFvdx7cCV1bq/MU0RkP0jSRGq2K6Bpxv7+U2FoMzQC3z7T9bz2Ccppog\nfl/x7qhzGgrPN9TeO4zfJ7TkKV85YwnWbu/m0lXz+PrbT+V1x7ZQFw7w+FZb5cwYrzrcbQRzVXUf\ngPt7zmE+/6immiCqjFaldA06N+B8N9qCx8ozurjbHZdQita6zOji3N/o23tjzKkP5w0qi0ocXTwU\nT7Kre4iVCxoAZ/bSs4+ZxRNbu0oqpzGm+hyxjcUico2IrBORdZ2d07/oyvj5hg64GUG54wgg/wyk\nPYPxksYQQNY0E3m6kHb0jeRtKAYnkAT9UjQj2NIxgCpjxiOcd2wL2w4Mjq6vbIzxlsMdCDpEZD6A\n+zvvaiqqeoOqrlbV1a2trdNekKZx00xkqobKbSwGp+dQrsbi7sHSM4JI0E9jTbBgG0G+9gFwBsnN\nb6wpOrp4U7uzwtmJWfMVvfbYFgCe2GbVQ8Z40eEOBHcDa9zHa4C7DvP5RzXWjO3/3zUQIxryT5i+\noRRN0VDOxuKeodIzAnB6DuWvGhphXp4eQxkLmiJFM4JX2vuJBH0snhUd3XbSvAaaokEef9Wqh4zx\nokp2H70deAI4QUT2iMjVwHXAJSKyBbjEfT4jxs8RNJnBZBnNUacHkuqhMQCq6sw8WkZV05yGcM7G\n4oFYkoFYsmggWNgULRoIMgvbZLc1+HzCuce08PjWrjHXYIzxhkr2GnqPqs5X1aCqLlLVm1S1S1Uv\nUtUV7u+tVizFAAAXCElEQVQZ67OYmRriUBtBrOzBZIeOFSKZVvrdaawBBuMp4ql00ZlHs7XW5Z5v\nKFN3X6hqCJx1Cdr7RiasxZztlTzzFZ13XAt7Dw6zu7v0aSrSabVpLYypAkdsY3GlNU4IBPGyxxBk\njLY3ZM03VMqi9ePNaYiwvz824Vt5hzuGoFBjMTiji9N66P3jdQ3EODAQG9M+kHHeJNoJ/t/D2zj/\n6w9w+1O7St7HGHPk8WwgCPh91IcDHBw+1EYwmTEEkHt0cXcZo4oz5tSHiSfT9A0nx2wfzQiKthEU\nno56U4fTUJwrIzi2tY7W+jCPl9iNVFX5xTO7EeDzv3qRWx7bXtJ+xpgjj2cDAUBDTZDeoQTptNI9\nlTaC0fmGsgLBUOkzj2bkW7u42KjijNHRxXkWst+co8dQhojw2uUtPLmttECwoa2PbZ2D/OPlK/nL\nlXP58m9e5nsPbS1pX2PMkcXTgSAz31DfSIJkWiffRuBmBNk9h0bnGSqr15A7zcS4sQTtvSM01gSp\nCfkL7l9KRtAUDY4GnPFOX9xER18sb9VStrtfaCPoF644fSHXv+9MLjtlHtfd+wrbDwwW3dcYc2Sx\nQDAUPzSYbNK9htyqocGJVUPl9hqCifMNFRtDkBEJ+pldF8rbgLupvZ8T5tbnXYHt1EWNAKzf01vw\nPOm0cvfzbbzh+Faaa0ME/T6uvfQkAB7clHdoiDHmCOXtQFDj9P/vGp1wbnIZQUPEGXuQPc1Ez1Ac\nv09GXyvFoUXsx34j7+gbYW6R9oGMpS21bO2c+K1cVdncMTBh4ftsqxY04hN4cc/Bgud4akc37X0j\nvO30haPblrREOWZ2LQ9tnv5R4MaYyvJ0IGiMBp1AMDi1jCDg99EQCYyZZqJ7MEFztLz1j+vDASJB\n34SxBPt6R5jXUFqQWjm/gY1tfRMWm9nTM8xALFkwENSE/Bw/t571ewtnBHc9v5doyM/FJ42dKuoN\nx7fy5LYuRhKpkspqjDkyeDoQZKaizkxBPdk2AnCqgMZkBIPxssYQgNNgO6c+MqZqKJFKc2AgVlLV\nEMCqBQ30x5LsHrdIzeaO/A3F2U5Z2MiLe3rzDiyLJVP87sV23rRq3oRR2G84oZWRRJqnttuU1sYc\nTbwdCKJBkmllZ9cQIs4I4ckfKzSh11A5Ywgy5tSHxzQWd/bHUIV5jTUl7Z+ZVfTltr4x219xewyt\nyNF1NNupi5voGoznbWd4ePMBeocTvO30BRNeO/eYFkIBHw9uKl49lE4rv1i3m2t/uZ4rrn+M13z1\nPpsK25gZ4u1A4M43tLVzgOZoiIB/8n+OZreaKcPJCCYRCBrGzjc02nW0sbRsJTN9xIZxgWBzRz8L\nm2poiBQOdqcudBqMX8zTYPynlzuojwQ4/7jZE16rCfk555hZPLS5eIPxb9a38b/vWM/vN7QTDfoJ\n+IUv3bWBZIFR0caYyvB0IGh0M4CtnQOTHlWc0TwuI+gpYy2CbK11Y+cb6ugtbVRxRiTo57jWOl7e\ndygQqCrP7OwZzRYKOXF+PUG/5GwnUFUeffUA5x3bQjBP0LzwhDls7Rxkd3fh9ZN//OROlrZEefaL\nl3D7NefyT287mS37B2yUsjEzwNOBIDPf0J6e4Sm1D4AzZUVmiol0WukZSpQ1hiBjTkOEvpHkaIPr\nvhLnGcq2akEDG9oO3ci3HxhkT88wF6yY+C1+vHDAz4nzGlifo+fQzq4h9h4c5vwV+acFf8PxzmsP\nb8lfPbSpvZ+nd/TwvnOW4HMnv3vTqrm8dnkL37xvM705FvkpZGfXIAOxZPE3GmNy8nYgcG/UqpPv\nMZTRHA3RH0uSSKXpH0mSSuvkMoL6sUtWdvSNEPL7yqpmWrmggY6+2Ggj+MNul84Lji9tXYdTFjWy\nPkeD8aOvOnX4uaqFMo5trWVhU03BdoLb1u4kFPDx9tcsHt0mIvzD5Ss5OJzgOw9sKamcAA+80sFF\n//4Qf/nNh3ikQPAxxuTn8UBwqL58smMIMjLTTBwcSoxOL1FuryFwGovh0FiC9r4R5jSEy+qGOr7B\n+OEtB1jaEmVpS21J+5+6sJH+kSQ7u8ZW7zy65QALm2pY1hLNs6dzQ7/whFYef/UA8eTE+v7BWJJf\nPbuXt5wyf0JwW7mggXeftZhbH9/Bts6BouV87NUDfPTHz3L83HpqQn6uuukp/v7OFxm07MCYsng6\nEGRmIIXJLVGZLdND6L3ff5LP/uKFMdvKkZlmIpMRtPeOML/EwWQZq+Y7Db4b2vqIJVM8sbWLCwpU\n54x36qImAF7Iqh5KpZXHtx7g/ONmFw1KF580l8F4ivs3dkx47e4X2hiIJXn/uUty7vvpS04g6Pfx\nnw+8WvAcT+/o5iO3ruOYllpu+8g5/PaTr+eaC5bzk6d28amfPm/rKhhTBk8HgkjQTyTo/Akms2h9\ntgtWtPLB85axZFaU4XiK5a21nDS/eOPseJlpJm5+bAd/3rSffb2F1yrOpTEaZGFTDS/v6+OZHT0M\nJ1IlVwsBrJhbRzjgG9Nz6KW9vfSNJHldCe0MFxzfyqLmGm55fMeY7arKj5/cyYnz6jlzSXPOfVvr\nw1z12qXc9fzevPMWdfbHuPqWp5nfGOFHHzmb5toQkaCfL1x2EtdeeiJ/2tjBHzZMDELjDcdTfOwn\nz/LG//sgn/7589y2dqet22w8ydOBAA51IZ1qG0FjNMiX37aKmz54Fr/71Ot54DMXln0DBycz+eRF\nK9i6f4AP/eBpdnUPldVQnJFpMH5oSycBn4yuS1yKoN/HygUNY3oOZdoHzivhOH6fcNW5S1m7vZuN\nWb2XntzWzYa2Pt537tKCWcVfv345Qb+P6/+cOyv49z9uYiie4vtrVo9mUBkfPv8YTprfwJfv3kD/\nSP5G58FYkg/d8hS/e3Efi2ZFeWhTJ39/50tc9p1HeHV/8WopY6qJ5wNBpnpoqt1Hp4uI8OlLjueJ\nz1/E9e89k8tPnc+bT5lf9nFWLmhg+4FB7tvQwWuWNlMXLm8t5rOXzWLdjm5+9+I+wGkfWDm/oeS2\nlHedtZhwwMcPn9gBON++v3DniyxqruGvzlxYcN/W+jDvPWcJdz63l13j2ile2tvLz9btZs15yzi2\ntW7CvkG/j3/5/06mo3+Ef//j5pzH7xtJ8IGbn+LpHT18612n88MPn826L17MPZ84H58IH7hpbdEl\nP+PJNN/4wyu8+duP8NEfPcM3/vAKf3q5w6qkzFHJAkE0EwimVjU03UIBH285dT7/9d4zec3S3NUo\nhaxa0IgqbDswyBtOKL1aKONTF6/gzCXNfPL25/jNC208s7OH80uoFspoioa48vSF3PncXnqHEnzz\nvk1sPzDIv/3VqROmpsjlo284Fr9P+O8HD2UFqspX7nmZ5qiTNeVzxpJmrjp3Kbc+sYPndvWMee3g\nUJyrblzLC7sP8p/vOYMr3InzRISTFzZy64fPon8kyVU3rR2dQXa8Te39XHH9Y1z/563UhwNs3t/P\n9x7axkd+uI6//dnzRRurU2nl/o0dfOmul/jRkzt5ua2PVNoCiJk55X1NrEKZsQRTbSM40mQPHiun\noTgjGgrwgw+dxftveopP3P4cAK8r0G00lw+ct5SfrdvNP9z1Evesb+O95yzhvBKPMbchwrvPWsxP\n1u7ijSfO4exjZvH41i6e2t7N1648eUxDfy6ffdMJ/HFDBx+4+Sm+8fZTufTk+RwYiPH+G9eyrXOQ\n777/NVyycu6E/VYtaOTGNav5wM1P8e4bnuCb7zydk93R1rFkihsf2c63/7SFhpoAN35gNRe7x4gn\n09zw8Fa+ed9mNrT18d33nTlhOo/+kQQ/e3o3P3xiJ7u6hwj5fcTdkdQNkQB//frlfOT1y3OuO6Gq\nbGjr46W9vbzS3s+OrkFOnNfAJSvncsbiptHxGMZMhhwNqezq1at13bp1FTn25+54gbueb+OVr15a\nVhfNI52qcsZX7yPgE576wsWTvlH0DiV4z/efZGfXIOu+eEnRxXHGe8f3HufpHT0saIzwh7+7gPoi\nU1xkazs4zGXfeWR0XemQ38fy1lru+cT5JU0Hsrt7iI//5Fle2NPL+85Zwtrt3ezpGeKGq1YXbTx/\ndMsBPv3z5+kajPM/33Aspy1u4p9/+zI7uoa47JR5fPWKk3N+eXj81QN88qfP0Tuc4NKT5/Pes5dw\n0vx6bnl8Bz94bAe9wwnOWtbMmvOW8aZV82jvHeGZnT389sV93PdyB3Mbwvzdxcdz3rGzmd/ktH/8\n7sV93PjIdl5022yiIT+Lm6Ns7RwgmVZm14W57JR5XHnGQs5Y3DT67ziWTPHszoM8sqWTx7Z20Tec\nIOATAu7f8fXHzeb8FbNZ1DyxO/BgLDk6gHBvzxBphSWzoiyeFWVpS5RIsPC/A1Wtqv9PRysReUZV\nVxd9n9cDwVPbu3l+dw/XXHBsRY4/k6679xXqIwE+9hfHTek4Q/EkB/rjLCkwfiCfP73cwf/6ybN8\n/wOrR0cdl2MglmT97oM8t/sgG/f18devX85pi5tK3j+eTHPdva9w82PbqQ35ufmDZ3HO8tIaznuH\nEnzlnpf55bN7AFjeWsuX37qqaBDZ3zfCfz+4lV89u4e+kSQ+gbTCJSvn8ok3HjfaPXe8p3d087Xf\nbuSF3U63XZ9AbShAfyzJ8tm1fPj8Y7hghdMjy+cTeocTPLhpP3/c0MGfNnYQS6ZZ2hKlPhIYHVCo\nCgGfcMaSJuY2REimlHgqzYa2XjrcyQ1b68Msc8eZDCdSbGzrY3vXIPluDT5x1rheuaCBhU01xJJp\nYskUPUMJ9nQPsbtnmO7BOCLOuSNBJ3AtbYmyqLmGxpogDTVBQn4fXYNx9veN0DUYZyieYjieIpFK\n01gTpLk2REttiJa6EC21YZprgyRSykgiRSyRZiCWpH8kyVA8ic8nhAM+wgE/DTUBmqMhmqJBwgEf\nIoJPhJFEioGRJAMxZ8CnCPhEqAn5aYgEaawJEgn6CPh8BPxCMqUMxpMMxVOkVfGJjF5PXSRAfSRA\nOOBDFdKqpNJKIqUkU2kURssTDvhmLGOzQGCOGMPxVNmZxHR7clsXs2pDHF9k9tVcHt7cyc6uQd51\n1hJCgdKb1UYSKX67fh8b2vp4x+pFJXUnVlXW7exhe+cgu3uG6OyPcfFJc3njiXMK3kz6RxLc+1I7\n97qN+3MbIsxpiHDKwkbOXT5rQiamqry6f4BHthxgk1vVtKNrkFDAx6r5jaxc0MBxc+pY2FTDwmZn\n5tvd3UPs6h5i6/4BXt7Xx4a2Pvb3x4gEfISDfhoiARY1R1k8q4bWujBphZQqg7Eku7uH2Nk9RNvB\nYUYSYwca1kcCzK4LEw35iYb8BP0+eocTdA/G6RqM5xyYmC1zM44fwRMWBv1CJOAfDTDJtBM4FOf+\nKziBLOT+BPziBCSfcNOasyb1JQwsEBhjjlCxZIp+dz6tltpwwS8JqspgPEXXQIyeocToN/JI0Ed9\nOEht2D9aTZhOK7Fkmt7hBD1DcQ4OJUik0qRUUVUiAeebfG04QMAno9/kh+Ip+kYS9A0niCXTo9/q\ng34f0ZCfmpCfgM9HMp0mmXLO0T+SYCCWJJZM4xNBBPwiBP1C0P2yEE+miSXTTgbj/k6mlIBfCPp9\no/sJTsYYT6aJp5xsJ5V2gkUyneZLb101qa7oUHog8HxjsTHm8AoH/ITrSssQRYS6cIC6cIClRWr0\nfD6nmqcm5GdemaPxvW5Guo+KyKUisklEXhWRa2eiDMYYYxyHPRCIiB+4HngzsBJ4j4isPNzlMMYY\n45iJjOBs4FVV3aaqceCnwBUzUA5jjDHMTCBYCOzOer7H3TaGiFwjIutEZF1np80zb4wxlTITgSBX\nH7gJXZdU9QZVXa2qq1tby+9/bowxpjQzEQj2AIuzni8C2magHMYYY5iZQPA0sEJEjhGREPBu4O4Z\nKIcxxhhmYByBqiZF5OPAHwA/cLOqbjjc5TDGGOM4KkYWi0gnsHOSu88GDkxjcY4Gds3eYNdc/aZ6\nvUtVtWgj61ERCKZCRNaVMsS6mtg1e4Ndc/U7XNfr+YVpjDHG6ywQGGOMx3khENww0wWYAXbN3mDX\nXP0Oy/VWfRuBMcaYwryQERhjjCnAAoExxnhcVQeCal/3QEQWi8ifRWSjiGwQkU+522eJyH0issX9\n3TzTZZ1uIuIXkedE5B73+TEista95p+5o9arhog0icgdIvKK+3m/tto/ZxH5O/ff9UsicruIRKrt\ncxaRm0Vkv4i8lLUt5+cqju+497P1InLmdJWjagOBR9Y9SAKfUdWTgHOBj7nXeC1wv6quAO53n1eb\nTwEbs55/HfgP95p7gKtnpFSV823g96p6InAazrVX7ecsIguBTwKrVfVknFkI3k31fc63AJeO25bv\nc30zsML9uQb47nQVomoDAR5Y90BV96nqs+7jfpybw0Kc67zVfdutwJUzU8LKEJFFwFuAG93nArwR\nuMN9S1Vds4g0ABcANwGoalxVD1LlnzPOFDg1IhIAosA+quxzVtWHge5xm/N9rlcAP1THk0CTiMyf\njnJUcyAoad2DaiEiy4AzgLXAXFXdB06wAObMXMkq4lvA54C0+7wFOKiqSfd5tX3Wy4FO4AduddiN\nIlJLFX/OqroX+L/ALpwA0As8Q3V/zhn5PteK3dOqORCUtO5BNRCROuCXwN+qat9Ml6eSRORyYL+q\nPpO9Ocdbq+mzDgBnAt9V1TOAQaqoGigXt178CuAYYAFQi1M1Ml41fc7FVOzfeTUHAk+seyAiQZwg\ncJuq/srd3JFJGd3f+2eqfBXwOuBtIrIDp7rvjTgZQpNbhQDV91nvAfao6lr3+R04gaGaP+eLge2q\n2qmqCeBXwHlU9+ecke9zrdg9rZoDQdWve+DWjd8EbFTVb2a9dDewxn28BrjrcJetUlT186q6SFWX\n4XymD6jq+4A/A29331Zt19wO7BaRE9xNFwEvU8WfM06V0LkiEnX/nWeuuWo/5yz5Pte7gQ+4vYfO\nBXozVUhTpqpV+wNcBmwGtgJ/P9PlqcD1nY+TGq4Hnnd/LsOpM78f2OL+njXTZa3Q9V8I3OM+Xg48\nBbwK/AIIz3T5pvlaTwfWuZ/1r4Hmav+cgX8CXgFeAn4EhKvtcwZux2kDSeB847863+eKUzV0vXs/\nexGnR9W0lMOmmDDGGI+r5qohY4wxJbBAYIwxHmeBwBhjPM4CgTHGeJwFAmOM8TgLBMZTRGTA/b1M\nRN47zcf+wrjnj0/n8Y2pFAsExquWAWUFAndG20LGBAJVPa/MMhkzIywQGK+6Dni9iDzvznvvF5Fv\niMjT7lzvfwMgIhe6az78BGcQDyLyaxF5xp0r/xp323U4M2U+LyK3udsy2Ye4x35JRF4UkXdlHfvB\nrHUGbnNH0RpzWAWKv8WYqnQt8FlVvRzAvaH3qupZIhIGHhORP7rvPRs4WVW3u88/rKrdIlIDPC0i\nv1TVa0Xk46p6eo5z/Q+ckcGnAbPdfR52XzsDWIUzZ8xjOHMpPTr9l2tMfpYRGOP4S5x5XJ7Hmcq7\nBWcBEICnsoIAwCdF5AXgSZxJwFZQ2PnA7aqaUtUO4CHgrKxj71HVNM4UIcum5WqMKYNlBMY4BPiE\nqv5hzEaRC3Gmfc5+fjHwWlUdEpEHgUgJx84nlvU4hf2fNDPAMgLjVf1AfdbzPwD/053WGxE53l38\nZbxGoMcNAifiLBGakcjsP87DwLvcdohWnNXGnpqWqzBmGti3D+NV64GkW8VzC86awMuAZ90G205y\nL4P4e+CjIrIe2IRTPZRxA7BeRJ5VZ2rsjDuB1wIv4MwW+zlVbXcDiTEzzmYfNcYYj7OqIWOM8TgL\nBMYY43EWCIwxxuMsEBhjjMdZIDDGGI+zQGCMMR5ngcAYYzzu/wf4DUxTDOW1KgAAAABJRU5ErkJg\ngg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1fc17207400>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import math\n",
    "\n",
    "def momentumM(m, b, p_m, beta):\n",
    "    gradient_m = gradientM(m, b)\n",
    "    momentum_m = gradient_m + beta * p_m\n",
    "    return momentum_m\n",
    "# end of def momentumM\n",
    "\n",
    "def momentumB(m, b, p_b, beta):\n",
    "    gradient_b = gradientB(m, b)\n",
    "    momentum_b = gradient_b + beta * p_b\n",
    "    return momentum_b\n",
    "# end of def momentumB\n",
    "\n",
    "def gradientStepWithMomentum(alpha, m, b, p_m0, p_b0, beta):\n",
    "    # compute the momentum values of m and b\n",
    "    p_m = momentumM(m, b, p_m0, beta)\n",
    "    p_b = momentumB(m, b, p_b0, beta)\n",
    "    \n",
    "    # compute new values of m and b\n",
    "    m = m - alpha * p_m\n",
    "    b = b - alpha * p_b\n",
    "    \n",
    "    return m, b, p_m, p_b\n",
    "# end of def gradientStepWithMomentum\n",
    "\n",
    "def adaptAlpha(currentError, prevError, alpha):\n",
    "    newAlpha = alpha\n",
    "    if currentError > prevError:\n",
    "        newAlpha = alpha * 0.9\n",
    "    else:\n",
    "        newAlpha = alpha * 1.1\n",
    "    return newAlpha\n",
    "# end of def adaptAlpha\n",
    "\n",
    "# iteration count\n",
    "N = 101\n",
    "\n",
    "# learning rate\n",
    "alpha = 0.001\n",
    "\n",
    "# momentum coefficient\n",
    "beta = 0.9\n",
    "\n",
    "# initial values\n",
    "m = b = 0\n",
    "p_m = p_b = 1\n",
    "\n",
    "# lists to keep the error and the iteration values.\n",
    "errors = []\n",
    "Ns = []\n",
    "\n",
    "# keeps previous error value.\n",
    "prevError = math.inf\n",
    "\n",
    "# Let's start iterating\n",
    "for i in range(N):\n",
    "    # compute current error value\n",
    "    currentError = computeError(m, b)\n",
    "    \n",
    "    # append these value for plotting grapghic\n",
    "    errors.append(currentError)\n",
    "    Ns.append(i)\n",
    "    \n",
    "    # adapt alpha\n",
    "    if i > 0:\n",
    "        alpha = adaptAlpha(currentError, prevError, alpha)\n",
    "    prevError = currentError\n",
    "    \n",
    "    if i % 10 == 0:\n",
    "        print(i, \"\\t error:\", currentError)\n",
    "        \n",
    "    m, b, p_m, p_b = gradientStepWithMomentum(alpha, m, b, p_m, p_b, beta)\n",
    "# end of for loop\n",
    "\n",
    "# drawing of graph.\n",
    "plt.title(\"Error distribution for %s iteration\" % N)\n",
    "plt.ylabel(\"Error\")\n",
    "plt.xlabel(\"Iteration\")\n",
    "plt.plot(Ns, errors)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References\n",
    "* [github notes](https://github.com/atcemgil/notes/blob/master/GradientDescent.ipynb), Ali Taylan Cemgil, Bogaziçi University, İstanbul, Turkey"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}