DL01-cell08-explanations.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Python for deep learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cell 8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from IPython.display import SVG\n",
    "from keras.utils import model_to_dot\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(N*N-1, input_dim=(N*N), activation='relu'))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "model.summary()\n",
    "SVG(model_to_dot(model, show_shapes=True, show_layer_names=True, dpi=65).create(prog='dot', format='svg'))\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create our neural network using a sequential model. In a sequential model we have a number of sequential layers of neurons where neurons in the layer N are conneted to neurons in the layer N+1.  \n",
    "\n",
    "An alternative way of building a neural network is using the functional API, which allows to create the network as an arbitrary directed, acyclic graph. However we will only use the sequential model here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create a sequential model by instanciating the class Sequential of keras. We can then add step by step the different layers of neurons. Keras makes this easy for us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "model = Sequential()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The network is empty at the beginning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding the first layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We do not explicitly add a layer of input neurons, instead we specify the size of the input when we add the first inner (hidden) layer of neurons. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_1 (Dense)              (None, 8)                 80        \n",
      "=================================================================\n",
      "Total params: 80\n",
      "Trainable params: 80\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from keras.layers import Dense\n",
    "N = 3\n",
    "model.add(Dense(N*N-1, input_dim=(N*N), activation='relu'))\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create the first hidden layer as a densly connected layer and add it to the model. The layer is created by instanciating the class Dense of keras. The parameters are the number of neurons in the layer, the size of the input vector and the name of the activation function for the neurons in the layer.\n",
    "\n",
    "We choose an input vector of size 9. The first layer has 8 neurons. Densly connected means that each of the 9 inputs is connected to each of the 8 neurons in the layer. That means we have $9\\cdot 8 = 72$ connections between the input neurons and the neurons in the first layer. Each connection has a weight associated. Initially the weights are choosen randomly. The values of the weights are a part of the parameters of the network, that will be modified during the training phase. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Variable 'dense_1/kernel:0' shape=(9, 8) dtype=float32, numpy=\n",
       " array([[ 0.00978899, -0.20192972,  0.4986899 , -0.5173455 ,  0.08104658,\n",
       "         -0.5685644 ,  0.38113755,  0.40224344],\n",
       "        [ 0.15906155,  0.14552766,  0.55346584,  0.27251434, -0.4772766 ,\n",
       "         -0.5497159 ,  0.1846388 , -0.48060942],\n",
       "        [-0.53819686, -0.19008508,  0.25113803,  0.40805364,  0.07280511,\n",
       "          0.530421  , -0.3660394 ,  0.43846226],\n",
       "        [-0.11456609,  0.11884028,  0.27335978,  0.23939878, -0.2645831 ,\n",
       "         -0.22316438, -0.30544972,  0.40835178],\n",
       "        [ 0.01483798, -0.3248926 ,  0.38018656, -0.11941361,  0.5011437 ,\n",
       "          0.51028264, -0.34542328, -0.07835621],\n",
       "        [-0.16129255,  0.05881137, -0.22097373, -0.12685889,  0.35467833,\n",
       "          0.14034063, -0.40189725, -0.45697933],\n",
       "        [-0.24265003, -0.4117571 ,  0.42562306,  0.5620301 , -0.00448835,\n",
       "          0.09999239,  0.2108289 , -0.19211608],\n",
       "        [ 0.18542421,  0.3062997 , -0.5674551 ,  0.20257986, -0.23946407,\n",
       "         -0.4206264 ,  0.00809097,  0.4450109 ],\n",
       "        [ 0.08557087, -0.3063395 , -0.3297199 ,  0.44342327,  0.02854562,\n",
       "          0.07078958,  0.4225552 , -0.01507592]], dtype=float32)>,\n",
       " <tf.Variable 'dense_1/bias:0' shape=(8,) dtype=float32, numpy=array([0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32)>]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The summary of the model tells us that there are 80 parameters. 72 parameters are the weights of the connections. What are the remaining 8 parameters? Each neuron has an additional parameter, the bias. The output of a neuron is calculated by applying the activation function on the input values multiplied by the corresponding weights plus the bias. The summary shows that the biases of the eight neurons are initially 0. Like the weights, the biases will be modified in the training phase. \n",
    "\n",
    "The activation function we selected is ``relu``, which stands for `Rectified Linear Unit`. The ``relu`` function is defined as:\n",
    "\n",
    "  $relu(x) = max(x,0)$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deZwU9ZnH8c/jcIkgCAynIIeAIHIMoxCvYNQIXiSuRg41JiasHMpG1ytq1nN1TdwYA2JcY4zhigpRDJhgQmLIYlQYGMbhHJBjBGG4kZuZZ//oItsOwzAzdE31dH/fr9e86K7z29VNP131q6qfuTsiIpK+Too6gIiIREuFQEQkzakQiIikORUCEZE0p0IgIpLmVAhERNKcCoFUGzN7xMwmJkGOfDMbEMJy25nZF2aWEcKyLzCzlcHyv5Ho5Zez3ovMbHl1rU+ioUIgKc3MXjWzJ+KHufvZ7v7XBCx7jZldFrfcde7ewN2LT3TZZXgMGBcs/60Qlg+AmbmZnXnkubvPdfeuYa1PkoMKgSSEmdWKOkOKOwPIjzqEpCYVAqmy4BfxfWa2GNhjZrXMrLWZTTOzIjP71MzuPMa8A8yssIzlXXaM6a8ys4VmtsvM1pvZI6XGX2hm88xsRzD+VjMbAQwH7g0OqbwTv54g6z4zaxK3nD5mtsXMaptZJzObY2Zbg2GTzKxxMN1vgHbAO8Gy7zWz9sEv6lrBNK3NbIaZbTOzAjP7ftx6HjGz183sNTPbHRyuyj7Ga18FdIxbV93S2yr+sFtcjm+b2bog+4Nx02aY2Q/NbFWw7gVm1tbM/hZMkhus58bS75OZdTOzvwbbOd/Mro0b96qZjTezmcFyPzSzTmW9JkkuKgRyooYCVwGNgRLgHSAXaANcCvybmV2RgPXsAW4J1nMVMPLIsXIzawe8C/wcyAR6A4vc/SVgEvBMcEjlmvgFuvsG4APgX+IGDwPedPdDgAFPAa2BbkBb4JFg3puBdcA1wbKfKSPzFKAwmP964D/N7NK48dcCU4PXNAMYV9YLd/dOpdZ1oPxN9U8XAl2JvQ8/MrNuwfC7iL1vVwKnAt8F9rr7xcH4XsF6fhu/MDOrTez9nQ00B+4AJplZ/KGjocCjwGlAAfBkBbNKhFQI5EQ97+7r3X0fcC6Q6e6PuftBd18N/A8w5ERX4u5/dfc8dy9x98XEvmS/GoweDvzJ3ae4+yF33+ruiyq46MnEvrwwMwuyTg7WWeDu77n7AXcvAv47bp3lMrO2xL6I73P3/UGel4Gb4yb7u7vPCtoUfgP0qmDminrU3fe5ey6x4nxk+d8DHnL35R6T6+5bK7C8/kAD4Ong/Z0D/J5g+wWmu/tH7n6YWBHunbiXI2HRcV05UevjHp8BtDazHXHDMoC5J7oSM+sHPA30AOoAdYE3gtFtgVVVXPSbwM/NrDXQGXCCvGbWHHgeuAhoSOyH0/YKLrc1sM3dd8cNWwvEH/75PO7xXqCemdUKvkQTofTyGwSPq7q9WgPr3b0kbthaYnt/x1unJDHtEciJir997XrgU3dvHPfX0N2vLGO+PUD9I0+CUy4zy1nPZGKHT9q6eyPgRWKHbo6s91jHosu9va677yB2qONbxA4LTfH/vyXvU8H8Pd39VOCmuHUeb9kbgCZm1jBuWDvgs/LyVMKXth/QshLzlre9yrMBaGtm8d8biXxNEhEVAkmkj4BdQQPyyUGjZA8zO7eMaVcQ+wV8VXDs+SFiv/KPpSGxX9j7zew8Yl/aR0wCLjOzbwUN1k3N7MghiU3EGlrLM5lY+8O/BI/j1/kFsMPM2gD3lJrvmMt29/XAPOApM6tnZj2B24KsibAIGBI0amcTa4OoqJeBx82ss8X0NLOmwbjytteHxArQvcF6BwDXEGvnkBpMhUASJjjWfQ2x48KfAluIfek0KmPancCoYPxnxL5gCktPF2cU8JiZ7QZ+BLwet6x1xBo+7wa2EfuSPHI8/JdA9+Asl2Odfz+D2GGhTcHx9CMeBbKAncBMYHqp+Z4CHgqW/e9lLHco0J7YL+nfAf/h7u+V8xor42Fiv+q3Bzknlz/5l/w3se03G9hFbBudHIx7BPh18Jq+FT+Tux8k1sA9iNh7+wJwi7svq/rLkGRg6phGRCS9aY9ARCTNqRCIiKQ5FQIRkTSnQiAikuZq3AVlzZo18/bt20cdQ0SkRlmwYMEWdy/zWp0aVwjat2/P/Pnzo44hIlKjmNnaY43ToSERkTSnQiAikuZUCERE0lxohcDMXjGzzWb2yTHGm5k9H3TYsdjMssLKIiIixxbmHsGrwMByxg8idn+XzsAIYEKIWURE5BhCKwTu/jdiNwA7lsHAa0HHGP8AGptZq7DyiIhI2aJsI2jDlzs1KeTLHVz8k5mNMLP5Zja/qKioWsKJiKSLKAuBlTGszFuhuvtL7p7t7tmZmeX1XSIikpqe+9MK8jfsDGXZUV5QVkisy7wjTid233YREYkzbUEhz/1pJYeKSzi79VHde5ywKPcIZgC3BGcP9Qd2uvvGCPOIiCSdZZ/v4sG38ujXoQk/uKxLKOsIbY/AzKYAA4BmZlYI/AdQG8DdXwRmEetVqoBYJ9ffCSuLiEhN9MWBw4yamEPDerX5+bA+1MoI57d7aIXA3YceZ7wDo8Nav4hITebu3DdtMWu27mHy9/vTvGG90NalK4tFRJLQq/PWMHPxRu654iz6d2wa6rpUCEREksyCtdt5cuZSLuvWnH+9uGPo61MhEBFJItv2HGTM5BxaNqrHszf05qSTyjrTPrFqXH8EIiKpqrjEGTt1IVu/OMi0kefTqH7talmvCoGISJIYN6eAuSu38J/fPIdzTk/89QLHokNDIiJJYO7KIp778wqu69OGoee1Pf4MCaRCICISsY079zF26iI6N2/AE9/sgVn47QLxVAhERCJ0qLiE0ZNyOHComAk39aV+neo/Yq82AhGRCD397jJy1u3g50P70CmzQSQZtEcgIhKRd/M28su/f8qt57fnml6tI8uhQiAiEoFPt+zhnjcX07ttY354ZbdIs6gQiIhUs30Hixk5cQG1Mozxw7OoUyvar2K1EYiIVCN35+G3P2H5pt386tZzadP45KgjaY9ARKQ6vT5/PW8uKOSOS85kQNfmUccBVAhERKpN/oadPPx2Phee2YyxIXUyUxUqBCIi1WDnvkOMmpRDk/p1+NmQ3mRUw83kKkptBCIiIXN37nkjl8+272PqiP40bVA36khfoj0CEZGQvTz3U2Yv2cT9g84iu32TqOMcRYVARCREH6/ZxtN/WMbAs1ty24Udoo5TJhUCEZGQFO0+wOhJObQ97WSeuaFntd9MrqLURiAiEoIjnczs3HeIV79zHqfWq55OZqpChUBEJATP/WkF81Zt5Znre9K99alRxymXDg2JiCTYX5Zv5udzCvhW9ul8K7t6O5mpChUCEZEEKty+lx/8dhFntWzIY4N7RB2nQlQIREQS5MDhYkZPyqG42Hnxpr7Uq50RdaQKURuBiEiCPDlzKbmFO3nxpr60b3ZK1HEqTHsEIiIJ8Paiz3jtg7V8/6IODOzRMuo4laJCICJyggo27+aB6Xlkn3Ea9w48K+o4laZCICJyAvYcOMztE3M4uXYG44ZlUTuj5n2tqo1ARKSK3J0Hf5fHqqIvmHhbP1o2qhd1pCqpeaVLRCRJTPpwHW8t2sBdl3XhgjObRR2nykItBGY20MyWm1mBmd1fxvhGZvaOmeWaWb6ZfSfMPCIiibK4cAePvbOEAV0zGX3JmVHHOSGhFQIzywDGA4OA7sBQM+tearLRwBJ37wUMAJ41szphZRIRSYQdew8ycmIOzRrU4aff6s1JSdTJTFWEuUdwHlDg7qvd/SAwFRhcahoHGlrslnwNgG3A4RAziYickJIS5+7Xc9m8ez/jh2dx2ik1/7drmIWgDbA+7nlhMCzeOKAbsAHIA8a6e0npBZnZCDObb2bzi4qKwsorInJcE95fxZ+Xbeahq7rTp91pUcdJiDALQVn7Sl7q+RXAIqA10BsYZ2ZH3abP3V9y92x3z87MzEx8UhGRCpi3agvPzl7ONb1ac8tXzog6TsKEWQgKgfjb7p1O7Jd/vO8A0z2mAPgUqHlXY4hIytu0az93TllIh2an8NR15yRtJzNVEWYh+BjobGYdggbgIcCMUtOsAy4FMLMWQFdgdYiZREQq7XBxCXdMXsieA8VMuKkvDeqm1iVYob0adz9sZmOAPwIZwCvunm9mtwfjXwQeB141szxih5Luc/ctYWUSEamKH89ezkdrtvHcjb3p0qJh1HESLtSy5u6zgFmlhr0Y93gD8PUwM4iInIjZ+Z/zi/dXM7xfO77Rp/T5LqlBVxaLiBzDuq17ufuNXM5p04iHry59GVTqUCEQESnD/kPFjJy0AANeGJ5VYzqZqYrUavEQEUmQR99ZQv6GXbx8SzZtm9SPOk6otEcgIlLKtAWFTPloHSMHdOKy7i2ijhM6FQIRkTjLPt/Fg2/l0a9DE+6+vEvUcaqFCoGISOCLA4cZNTGHhvVq8/NhfahVAzuZqQq1EYiIEOtk5r43F7N2214mf68fzRvWzE5mqiI9yp2IyHG8Om8NM/M2cs8VXenXsWnUcaqVCoGIpL0Fa7fz5MylXNatBSMu6hh1nGqnQiAiaW3bnoOMmZxDq8b1ePaGXjW+k5mqUBuBiKSt4hJn7NSFbN1zkOkjz6dR/dpRR4qE9ghEJG2Nm1PA3JVbeOSas+nRplHUcSKjQiAiaWnuyiKe+/MKruvThqHntT3+DClMhUBE0s7GnfsYO3URnZs34Ilv9kipTmaqQoVARNLKoeISRk/K4cChWCcz9euoqVRbQETSytPvLiNn3Q7GDetDp8wGUcdJCtojEJG08W7eRn7590+59fz2XN2zddRxkoYKgYikhdVFX3DPm4vp3bYxP7yyW9RxkooKgYikvH0Hixk1KYfaGcb44VnUqaWvvnhqIxCRlObuPPTWJyzftJtf3XoubRqfHHWkpKOyKCIp7fX565mWU8gdX+vMgK7No46TlFQIRCRl5W/YycNv53Phmc0Ye2nnqOMkLRUCEUlJO/cdYtSkHJrUr8PPhvQmIw1vJldRaiMQkZTj7tzzRi6fbd/H1BH9adqgbtSRkpr2CEQk5bw891NmL9nE/YPOIrt9k6jjJD0VAhFJKR+v2cbTf1jGoB4tue3CDlHHqRFUCEQkZRTtPsDoSTm0a1KfZ67vmfY3k6soFQIRSQlHOpnZue8QLwzPomG99OxkpirUWCwiKeGn761g3qqt/Pj6nnRrdWrUcWoU7RGISI03Z9kmxv2lgBuz23JDdnp3MlMVKgQiUqMVbt/LD36bS7dWp/Lo4LOjjlMjhVoIzGygmS03swIzu/8Y0wwws0Vmlm9m74eZR0RSy4HDxYyelENJiTNheBb1amdEHalGCq2NwMwygPHA5UAh8LGZzXD3JXHTNAZeAAa6+zoz041ARKTCnpy5lNzCnbx4U1/aNzsl6jg1Vph7BOcBBe6+2t0PAlOBwaWmGQZMd/d1AO6+OcQ8IpJC3l70Ga99sJbvX9SBgT1aRh2nRguzELQB1sc9LwyGxesCnGZmfzWzBWZ2S1kLMrMRZjbfzOYXFRWFFFdEaoqCzbt5YHoe2Wecxr0Dz4o6To0XZiEo60oOL/W8FtAXuAq4AnjYzLocNZP7S+6e7e7ZmZmZiU8qIjXGngOHuX1iDifXzmDcsCxqZ+iclxMV5nUEhUD8eVynAxvKmGaLu+8B9pjZ34BewIoQc4lIDeXuPPi7PFYXfcFvbutHy0b1oo6UEsIspR8Dnc2sg5nVAYYAM0pN8zZwkZnVMrP6QD9gaYiZRKQGm/ThOt5atIG7Lu/CBWc2izpOyghtj8DdD5vZGOCPQAbwirvnm9ntwfgX3X2pmf0BWAyUAC+7+ydhZRKRmmtx4Q4ee2cJA7pmMmrAmVHHSSnmXvqwfXLLzs72+fPnRx1DRKrRjr0Huer5vwPw+zsu5LRT6kScqOYxswXunl3WON1rSESSWkmJc9fruWzevZ83bj9fRSAEFS4EZnYSsYbc1sA+IN/dN4UVTEQEYML7q5izbDOPDT6b3m0bRx0nJR23EJhZJ+A+4DJgJVAE1AO6mNle4BfAr929JMygIpJ+5q3awrOzl3NNr9bc3P+MqOOkrIrsETwBTAD+1Us1KAS3hBgG3Az8OvHxRCRdbdq1nzunLKRDs1N46rpz1MlMiI5bCNx9aDnjNgPPJTSRiKS9w8Ul3DF5IXsOFDP5+/1pUFfNmWGq8HUEZva4mdWKe36qmf0qnFgiks5+PHs5H63ZxlPXnUOXFg2jjpPyKnNBWS3gQzPraWZfJ3bB2IJwYolIupqd/zm/eH81w/u14xt9St+eTMJQ4f0td3/AzP4MfAhsBy5294LQkolI2lm3dS93v5FLz9Mb8aNrukcdJ21U5tDQxcDPgMeAvwLjzKx1SLlEJM3sP1TMyEkLOMmM8cOyqFtLncxUl8q0wPwEuOFIxzJmdh0wB9A9YEXkhD36zhLyN+zil9/Opm2T+lHHSSuVKQRfcffiI0/cfbq6lhSRRJi2oJApH61j1IBOXNqtRdRx0s5xDw2Z2U1mdlJ8ETjC3beaWSczuzCceCKS6pZ9vosH38qjf8cm3HX5Ud2RSDWoyB5BU2ChmS0gdpbQkSuLzwS+CmwByuyYXkSkPLv3H2LkxBwa1qvN80P7UEudzESiIheU/czMxgFfAy4AehK719BS4OYj/Q2LiFSGu3P/tDzWbdvL5O/1o3lDdTITlQq1EQSHhd4L/kRETtir89YwM28j9w86i34dm0YdJ61V5u6jv+LoPodx9+8mNJGIpLwFa7fz5MylXNatBSMu6hh1nLRXmbOGfh/3uB7wTY7ug1hEpFzb9hxkzOQcWjWux7M39OKkk3QzuahV5sriafHPzWwK8KeEJxKRlFVc4oydupCtew4yfeT5NKpfO+pIwol1Xt8ZaJeoICKS+sbNKWDuyi08eu3Z9GjTKOo4EqhMG8FuYm0EFvz7ObEOa0REjmvuyiKe+/MKrstqw5Bz20YdR+JU5tCQ7gUrIlWycec+xk5dRJfmDXniGz3UyUySqUhXlVnljXf3nMTFEZFUc6i4hNGTcjhwqJgXbsqifh11MpNsKvKOPFvOOCd2oZmISJmemrWMnHU7GDesD50yG0QdR8pQkSuLL6mOICKSemblbeSV//2UW89vz9U9ddf6ZFWZ/gjqm9lDZvZS8LyzmV0dXjQRqclWF33BvW8upnfbxvzwym5Rx5FyVOb00V8BB4Hzg+eFwBMJTyQiNd6+g8WMmpRD7Qxj/PAs6tTSzeSSWWXenU7u/gxwCMDd9xE7lVRE5J/cnYfe+oTlm3bz0xt706bxyVFHkuOoTCE4aGYnE9xvyMw6AQdCSSUiNdbr89czLaeQO77WmQFdm0cdRyqgQudxWeyk3xeBPwBtzWwSsVtS3xpeNBGpafI37OTht/O5qHMzxl7aOeo4UkEVvQ21m9lY4OtAf2KHhMa6+5Yww4lIzbFz3yFGTcqhSf06PHdjbzJ0M7kaozJXdvwD6OjuM8MKIyI1k7tzzxu5fLZ9H7/91/40bVA36khSCZVpI7gE+MDMVpnZYjPLM7PF5c1gZgPNbLmZFZjZMbuzNLNzzazYzK6vRB4RSRIvz/2U2Us28cCV3eh7RpOo40glVWaPYFBlFmxmGcB44HJip5p+bGYz3H1JGdP9F/DHyixfRJLDx2u28fQfljGoR0u+e0H7qONIFVTmpnNrK7ns84ACd18NYGZTgcHAklLT3QFMA86t5PJFJGJFuw8welIO7ZrU55nre+pmcjVUmFd5tAHWxz0vDIb9k5m1IdbT2YvlLcjMRpjZfDObX1RUlPCgIlJ5xSXOnVMWsnPfIV4YnkXDeupkpqYKsxCU9dOgdJ/HzwH3uXtxeQty95fcPdvdszMzMxMWUESq7qfvreCD1Vt54hs96Nbq1KjjyAkI836whUB87xOnc3Qfx9nA1GB3shlwpZkddve3QswlIidozrJNjPtLATdmt+WGbHUyU9OFWQg+BjqbWQfgM2AIMCx+AnfvcOSxmb0K/F5FQCS5FW7fyw9+m0u3Vqfy6OCzo44jCRBaIXD3w2Y2htjZQBnAK+6eb2a3B+PLbRcQkeRz4HAxoyflUFLiTBieRb3aGVFHkgQItasgd58FzCo1rMwC4O63hplFRE7ckzOXklu4k1/c3Jf2zU6JOo4kiO4NKyIV8vaiz3jtg7WMuLgjV5zdMuo4kkAqBCJyXAWbd/PA9DzObX8a91zRNeo4kmAqBCJSrj0HDnP7xBzq18lg3LAsamfoayPVhNpGICI1m7vz4O/yWF30Bb+5rR8tTq0XdSQJgUq7iBzTxA/X8daiDdx1eRcuOLNZ1HEkJCoEIlKm3PU7ePydJQzomsmoAWdGHUdCpEIgIkfZsfcgoyblkNmwLj/9Vm9OUiczKU1tBCLyJSUlzl2v57J5937euP18TjulTtSRJGTaIxCRL5nw/irmLNvMw1d3p3fbxlHHkWqgQiAi/zRv1Raenb2ca3u15ub+Z0QdR6qJCoGIALBp137unLKQjpkNeOq6c9TJTBpRG4GIcLi4hDsmL2TPgWKmfD+LU+rqqyGd6N0WEX48ezkfrdnGz4b0pnOLhlHHkWqmQ0MiaW52/uf84v3V3NS/HYN7tzn+DJJyVAhE0ti6rXu5+41cep7eiIev7h51HImICoFImtp/qJiRkxZwkhnjh2VRt5Y6mUlXaiMQSVOPvpNP/oZd/PLb2bRtUj/qOBIh7RGIpKE3FxQy5aP1jBrQiUu7tYg6jkRMhUAkzSz7fBcPvZVH/45NuOvyLlHHkSSgQiCSRnbvP8TIiTmcWq82zw/tQy11MiOojUAkbbg790/LY922vUz+Xj+aN1QnMxKjnwMiaeLVeWuYmbeRe6/oSr+OTaOOI0lEhUAkDSxYu50nZy7l8u4tGHFxx6jjSJJRIRBJcdv2HGTM5BxaNa7HT27opZvJyVHURiCSwopLnLFTF7J1z0GmjzyfRifXjjqSJCHtEYiksHFzCpi7cguPXns2Pdo0ijqOJCkVApEUNXdlEc/9eQXXZbVhyLlto44jSUyFQCQFbdixj7FTF9GleUOe+EYPtQtIuVQIRFLMwcMljJmcw4FDxbxwUxb166gpUMqnT4hIinn63WXkrNvB+GFZdMpsEHUcqQG0RyCSQmblbeSV//2U71zQnqt6too6jtQQoRYCMxtoZsvNrMDM7i9j/HAzWxz8zTOzXmHmEUllq4u+4N43F9OnXWMeGNQt6jhSg4RWCMwsAxgPDAK6A0PNrHQXSJ8CX3X3nsDjwEth5RFJZfsOFjNqUg61M2KdzNSppZ19qbgwPy3nAQXuvtrdDwJTgcHxE7j7PHffHjz9B3B6iHlEUpK789Bbn7B8026eG9KH1o1PjjqS1DBhFoI2wPq454XBsGO5DXi3rBFmNsLM5pvZ/KKiogRGFKn5Xp+/nmk5hdz5tc58tUtm1HGkBgqzEJR14rKXOaHZJcQKwX1ljXf3l9w9292zMzP1QRc5In/DTh5+O5+LOjfjzks7Rx1HaqgwTx8tBOIvZzwd2FB6IjPrCbwMDHL3rSHmEUkpO/cdYtSkHJrUr8NzN/Ym4yRdNCZVE+YewcdAZzPrYGZ1gCHAjPgJzKwdMB242d1XhJhFJKW4O/e8kctn2/cxfngfmjaoG3UkqcFC2yNw98NmNgb4I5ABvOLu+WZ2ezD+ReBHQFPgheAS+MPunh1WJpFU8T9zVzN7ySYevro7fc9oEnUcqeFCvbLY3WcBs0oNezHu8feA74WZQSTVfPTpNv7rD8sZ1KMl372gfdRxJAXoZGORGqRo9wHGTM6hXZP6PHN9T91MThJChUCkhiguce6cspBd+w8x4aYsGtZTJzOSGLrpnEgN8dP3VvDB6q385IZenNXy1KjjSArRHoFIDTBn2SbG/aWAIee25fq+ugBfEkuFQCTJFW7fyw9+m0v3VqfyyLVnRx1HUpAKgUgSO3C4mNGTcihxZ8JNWdSrnRF1JElBaiMQSWJPzlxKbuFOfnFzX85oekrUcSRFaY9AJEm9vegzXvtgLSMu7sgVZ7eMOo6kMBUCkSRUsHk3D0zP49z2p3HPFV2jjiMpToVAJMnsOXCY2yfmUL9OBuOGZVE7Q/9NJVxqIxBJIu7OA9PzWF30BRNv60eLU+tFHUnSgH5qiCSRiR+uY0buBu7+elfOP7NZ1HEkTagQiCSJ3PU7ePydJVzSNZORX+0UdRxJIyoEIklgx96DjJqUQ2bDuvz0xt6cpE5mpBqpjUAkYiUlzl2v57J5937evP18GtevE3UkSTPaIxCJ2IT3VzFn2WZ+dHV3erVtHHUcSUMqBCIRmrdqC8/OXs61vVpzU/8zoo4jaUqFQCQim3bt584pC+mY2YCnrjtHncxIZNRGIBKBw8Ul3DF5IXsOFDPl+1mcUlf/FSU6+vSJRODHs5fz0Zpt/GxIbzq3aBh1HElzOjQkUs1m53/OL95fzU392zG4d5uo44ioEIhUp7Vb93D3G7n0PL0RD1/dPeo4IoAKgUi12X+omJETczjJjPHDsqhbS53MSHJQG4FINXn0nXyWbNzFK7dm07ZJ/ajjiPyT9ghEqsGbCwqZ8tF6Rl/Sia+d1SLqOCJfokIgErJln+/iobfy+ErHpvzgsi5RxxE5igqBSIh27z/EyIk5nFqvNj8b2pta6mRGkpDaCERC4u7cPy2Pddv2Mvl7/WjeUJ3MSHLSzxORkLw6bw0z8zZy7xVd6dexadRxRI5JhUAkBAvWbufJmUu5vHsLRlzcMeo4IuVSIRBJsG17DjJmcg6tGtfjJzf00s3kJOmFWgjMbKCZLTezAjO7v4zxZmbPB+MXm1lWmHlEwlZc4oydupCtew4yYXhfGp1cO+pIIscVWiEwswxgPDAI6A4MNbPS19QPAjoHfyOACWHlEakOP5+zkrkrt/DYtWfTo02jqOOIVEiYZw2dBxS4+2oAM5sKDAaWxE0zGHjN3R34h5k1NrNW7r4x0WHeX1HEE79fckT5wVYAAAcASURBVPwJRarIgVVFX/AvWadz47lto44jUmFhFoI2wPq454VAvwpM0wb4UiEwsxHE9hho165dlcI0qFuLzi0aVGlekYq68Mxm3DfwLLULSI0SZiEo63+CV2Ea3P0l4CWA7Ozso8ZXRN8zTqPvGX2rMquISEoLs7G4EIjfPz4d2FCFaUREJERhFoKPgc5m1sHM6gBDgBmlppkB3BKcPdQf2BlG+4CIiBxbaIeG3P2wmY0B/ghkAK+4e76Z3R6MfxGYBVwJFAB7ge+ElUdERMoW6r2G3H0WsS/7+GEvxj12YHSYGUREpHy6slhEJM2pEIiIpDkVAhGRNKdCICKS5izWXltzmFkRsLaKszcDtiQwTiIlazblqpxkzQXJm025Kqequc5w98yyRtS4QnAizGy+u2dHnaMsyZpNuSonWXNB8mZTrsoJI5cODYmIpDkVAhGRNJduheClqAOUI1mzKVflJGsuSN5sylU5Cc+VVm0EIiJytHTbIxARkVJUCERE0lzKFQIzu8HM8s2sxMyOeYqVmQ00s+VmVmBm98cNb2Jm75nZyuDf0xKU67jLNbOuZrYo7m+Xmf1bMO4RM/ssbtyVichV0WzBdGvMLC9Y//zKzh9GLjNra2Z/MbOlwfs+Nm5cQrfZsT4zcePNzJ4Pxi82s6yKzhtyruFBnsVmNs/MesWNK/M9raZcA8xsZ9z786OKzlsN2e6Jy/WJmRWbWZNgXCjbzMxeMbPNZvbJMcaH9/ly95T6A7oBXYG/AtnHmCYDWAV0BOoAuUD3YNwzwP3B4/uB/0pQrkotN8j4ObGLQAAeAf49pG1WoWzAGqDZib62ROYCWgFZweOGwIq49zJh26y8z0zcNFcC7xLrea8/8GFF5w051/nAacHjQUdylfeeVlOuAcDvqzJv2NlKTX8NMKcattnFQBbwyTHGh/b5Srk9Andf6u7LjzPZeUCBu69294PAVGBwMG4w8Ovg8a+BbyQoWmWXeymwyt2rehV1ZZzoa45sm7n7RnfPCR7vBpYS6/c60cr7zMTnfc1j/gE0NrNWFZw3tFzuPs/dtwdP/0GsJ8CwnchrDnN7VWX5Q4EpCVx/mdz9b8C2ciYJ7fOVcoWggtoA6+OeF/L/Xx4tPOglLfi3eYLWWdnlDuHoD9+YYJfwlUQdfqlkNgdmm9kCMxtRhfnDygWAmbUH+gAfxg1O1DYr7zNzvGkqMm+YueLdRuxX5RHHek+rK9dXzCzXzN41s7MrOW/Y2TCz+sBAYFrc4LC22fGE9vkKtWOasJjZn4CWZYx60N3frsgiyhh2wufRlperksupA1wLPBA3eALwOLGcjwPPAt+t5mwXuPsGM2sOvGdmy4JfMVWWwG3WgNh/1n9z913B4BPaZqVXUcaw0p+ZY00TyuftOOs8ekKzS4gVggvjBif8Pa1Erhxihz6/CNpv3gI6V3DesLMdcQ3wv+4e/0s9rG12PKF9vmpkIXD3y05wEYVA27jnpwMbgsebzKyVu28Mdrs2JyKXmVVmuYOAHHffFLfsfz42s/8Bfl/RXInK5u4bgn83m9nviO2S/o2It5mZ1SZWBCa5+/S4ZZ/QNiulvM/M8aapU4F5w8yFmfUEXgYGufvWI8PLeU9DzxVXsHH3WWb2gpk1q8i8YWeLc9SeeYjb7HhC+3yl66Ghj4HOZtYh+PU9BJgRjJsBfDt4/G2gInsYFVGZ5R51TDL4Ijzim0CZZxaElc3MTjGzhkceA1+PyxDZNjMzA34JLHX3/y41LpHbrLzPTHzeW4KzO/oDO4NDWhWZN7RcZtYOmA7c7O4r4oaX955WR66WwfuHmZ1H7Ptoa0XmDTtbkKkR8FXiPnchb7PjCe/zleiW76j/iP2HLwQOAJuAPwbDWwOz4qa7ktgZJquIHVI6Mrwp8GdgZfBvkwTlKnO5ZeSqT+w/Q6NS8/8GyAMWB29yqwRus+NmI3ZGQm7wl58s24zYYQ4Ptsui4O/KMLZZWZ8Z4Hbg9uCxAeOD8XnEnbV2rM9bgrbT8XK9DGyP2z7zj/eeVlOuMcF6c4k1Yp9fHdurItmC57cCU0vNF9o2I/bjbyNwiNh32G3V9fnSLSZERNJcuh4aEhGRgAqBiEiaUyEQEUlzKgQiImlOhUBEJM2pEIiIpDkVAhGRNKdCIHKCzOzc4MZ29YIrT/PNrEfUuUQqSheUiSSAmT0B1ANOBgrd/amII4lUmAqBSAIE93j5GNhP7FYJxRFHEqkwHRoSSYwmQANivaTViziLSKVoj0AkAcxsBrGeoToQu7ndmIgjiVRYjeyPQCSZmNktwGF3n2xmGcA8M/uau8+JOptIRWiPQEQkzamNQEQkzakQiIikORUCEZE0p0IgIpLmVAhERNKcCoGISJpTIRARSXP/B8s1F5xH7K59AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "import numpy as np\n",
    "X = np.arange(-1, 1, 0.01)\n",
    "Y = np.array([max(x, 0) for x in X])\n",
    "plt.plot(X, Y)\n",
    "plt.xlabel(\"x\")\n",
    "plt.ylabel(\"relu(x)\")\n",
    "plt.title('relu activation function')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding the second layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The second and last layer contains one neuron that calculates the output of the network. The output is the probability of an input to belong to the foreground. By thresholding the probability map a binary classification output is obtained.\n",
    "\n",
    "The layer is densly connected to the previous layer. This means all units in the previous layer are connected to the only unit in this layer. Since the previous layer has 8 units, this adds 9 more parameters to the model. 8 for the 8 weights of the connections and one for the bias of the output neuron."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_1 (Dense)              (None, 8)                 80        \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 1)                 9         \n",
      "=================================================================\n",
      "Total params: 89\n",
      "Trainable params: 89\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.add(Dense(1, activation='sigmoid'))\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As activation function we use a sigmoid function this time. Sigmoid functions have an S like shape and map arbitrary ranges into the interval [0, 1]. An example of a sigmoid function is the logistic function:\n",
    "\n",
    "$f(x) = \\frac{1}{1+e^{-x}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dd3yV9fn/8dcV9t6bQACRIZsQcOOqOFrrBkEBUWpbO2xrtdXaqv1W/bVa26pVRBQHOHHUgYqKuBhh773CJmGHkHX9/jgHe0wTcgg5uZOc9/PxyINzn/s+933dJ+S8z/353Pf9MXdHRETiV0LQBYiISLAUBCIicU5BICIS5xQEIiJxTkEgIhLnFAQiInFOQSAlYmbtzOygmVUpT9s1sz+Z2YtlUMfvzWx8jNb9gZmNjMF6a5nZf8xsn5m9VtrrL2bbS81scFluU6JXNegCpGJy901A3XjYbvgD7EV3bxtRx19Kad1/Ak5y9xER676oNNZdiKuAFkATd8+N0TYws+eANHe/++hz7n5KrLYnJ05HBCLxoz2wKpYhIBWUu+snzn6AO4G1wAFgGXD5MZZNAVKB/cAO4JHw80mAA1XD0x2AGeF1TgMeJ/QtOnLZ0cBmYA9wCzAAWATsBR6L2GYCcDewEdgJPA80OMZ2Pw9v92PgsaPbLWRfGgHvArvCNbwLtI2Y3xh4Ftganv8WUAc4DOQDB8M/rYE/RezfVODWAttaCFwRfvyP8H7vB+YCZ4afHwJkAznh9S4MPz8duOk43ouRwCZgN3BXEft+b4FtjYnchyLe2+nA/cBX4ff3I6BpxPJnAF+Hf3+bgVHA2PA2ssPb+U942Q3A+eHHNYBHw+/z1vDjGuF5g4E04Nfh/d0GjA76b6ay/wRegH4C+KXD1eEPswTgWuAQ0KqIZb8Brg8/rgsMCj8u+KHxDfA3oHr4A2I//xsETwI1ge8BWYQ+aJsDbcJ/9GeHl78RWAN0DG9zCvDCMbb7SPjD5azwB1ZRQdAEuBKoDdQDXgPeipj/HvAKocCoFlHPYEJNHZHr+vZDFLgB+CpiXvfwh+PRD7cR4W1XDX/AbQdqFlxPxOun898giOa9eBqoBfQGjgDditj/72yrkOmC7+10Ql8YTg6vfzrwYHheu/B7PSz8XjUB+oTnPQf8ucC2N/DfILgPmBn+3TcjFCb3R7zXueFlqgEXA5lAo6D/birzj5qG4pC7v+buW909391fAVYT+uZfmBzgJDNr6u4H3X1mwQXMrB2hb/f3uHu2u38JvFPIuu539yx3/4hQ+Ex2953uvgX4AugbXm44oSOPde5+EPgdMNTMvtOnFbHdP7j7EXefAfznGPud7u5vuHumux8A/g84O7yuVsBFwC3uvsfdc9z986LWVcCbQB8zax9R/xR3PxLe7ovhbee6+8OEQqtLlOuO5r24190Pu/tCQkcivaNcdzSedfdV7n4YeBXoE1HXNHefHH6v0t19QZTrHA7cF/7d7yJ0tHJ9xPyc8Pwcd3+f0JFFtO+XlICCIA6Z2Q1mtsDM9prZXqAH0LSIxccQ+ka4wszmmNmlhSzTGshw98yI5zYXstyOiMeHC5k+2gncmlBTyFEbCX2bblHIdve4+6ECyxbKzGqb2VNmttHM9hNqymoYPgMpMbwPe4p6fVHCofIeMDT81FDgpYjt/trMlofP1tkLNKDo97ugaN6L7RGPMyndzvSi1p1I6GihJArbp9YR0+n+3X6M0t4nKUBBEGfC31qfBm4ldPZIQ2AJYIUt7+6r3X0YocP4h4DXzaxOgcW2AY3NrHbEc4knUOZWQh2bR7Uj1Fywo8By24BGBeppd4z1/prQN8uB7l6fUFMShPZ9M6F9aFjI66K5Re9kYJiZnUqoGeUzADM7E7gDuIZQ80ZDYB//fb+LW3e070VJHCLUTHZUy+N47WagUxHzSrJPW49j21LKFATxpw6hP9RdAGY2mtARQaHMbISZNXP3fELt3gB5kcu4+0ZCHcp/MrPq4Q/D759AjZOB28ysg5nVBf4CvFLgW2Lkdu8Nb/eMYrZbj9CRx14zawz8MWJd24APgCfMrJGZVTOzo0GxA2hiZg2Ose73CX243ReuNT9im7mE3u+qZnYPUD/idTuAJDMr6m8xqveihBYAZ4WvzWhAqNkpWi8B55vZNWZW1cyamNnRZqMdhPo0ijIZuNvMmplZU+AeIObXfkjRFARxxt2XAQ8T6mTdAfQkdFZIUYYAS83sIKGzX4a6e1Yhyw0HTgXSgT8T6nQ9UsIyJwAvEGq6WU+oY/lnRSx7HTAQyCD0wf78Mdb7KKFv67sJdVZOLTD/ekLt0ysIdV7/EsDdVxD68FoXbk5rXeB1hPsDpgDnA5MiZn1IKGBWEWoCyeK7zWZHL+xKN7N5hdR8PO/FcXH3jwn9nhYROpvp3eN47SZCHbm/JvTeL+C/fRPPAN3D79Vbhbz8z4QCfBGwGJgXfk4CYu4amEZKn5m9Aqxw9z8Wu7CIBEpHBFIqzGyAmXUyswQzGwJcRuj0UBEp53SLCSktLQk1jTQhdEHQj919frAliUg01DQkIhLn1DQkIhLnKlzTUNOmTT0pKSnoMkREKpS5c+fudvdmhc2rcEGQlJREampq0GWIiFQoZlbkVfdqGhIRiXMKAhGROKcgEBGJcwoCEZE4pyAQEYlzCgIRkTinIBARiXMV7joCEZF4si8zhyVb97EobR892zTgjM7RDm4XPQWBiEg5kZOXz/Jt+5m7cQ/zNu1lUdpeNqb/dwTYHw/upCAQEalMMg5lM2/jHuZu2sPcjXtYlLaXrJzQ4HYt69ekT2JDrklOpFfbBvRs04CGtavHpA4FgYhIGdlzKJuZ69L5em0636xLZ83OgwBUTTBOadOAYSnt6N++Ef3aNaJ1w1plVpeCQEQkRg5k5TBrXQbfhD/8l2/bD0Dt6lVI6dCYK/u1pX/7RvRq24Ca1aoEVqeCQESklLg7q3ce5LMVO/ls5U5SN+whN9+pUTWB5KRG/OZ7J3Nqp6b0atuAalXKz0mbCgIRkROQmZ3L12vS+WzlTqav3MWWvYcB6NqyHjef1ZEzOzelX7tGgX7jL46CQETkOO3PyuGT5Tv4YPF2Pl+1iyO5+dSpXoXTT2rKreeexOAuzWjVoOza+E+UgkBEJAoZh7L5eNl2Pliyna/W7CYnz2lRvwZDByRyQfeWDOjQiBpVy++3/mNREIiIFCEzO5ePlu5gyvwtfLVmN3n5TmLjWow+vQNDerSkT9uGJCRY0GWeMAWBiEiE3Lx8vlqbzlvzt/Dh0u1kZufRpmEtfnRWRy7u2YpTWtfHrOJ/+EdSEIiIAKt3HOCVOZt5e+FWdh04Qv2aVbmsTxsu79uG5PaNKsU3/6IoCEQkbmXl5PH+4m1Mnr2JORv2UK2KcW7X5lzetw3ndG1eYdv8j5eCQETizqodB5g0axNT5qWxPyuXDk3r8PuLu3Jlv7Y0qVsj6PLKnIJAROJCfr7z6YqdPPPler5Zl071Kglc2KMlw1ISObVjk0rX7n88FAQiUqkdPJLL66mbee7rDWxIz6RVg5r8dkgXrk1OjMtv/4VREIhIpbRt32EmfLmel+ds5kBWLn3bNeTX3+vCkB4ty9XtHcoDBYGIVCobdh/iyc/X8sa8NPIdLu7ZitGnJ9GvXaOgSyu3FAQiUims3H6AJ6av4T8Lt1K1SgJDB7Rj7FkdSWxcO+jSyj0FgYhUaEu27OMfn6zm42U7qF29Cjed2ZGbzuhA8/o1gy6twlAQiEiFtHbXQR75eBXvLdpG/ZpV+fl5nRl9WhKN6sRmFK/KTEEgIhVK2p5M/vnJal6fm0bNalX42bkncdOZHWlQq1rQpVVYCgIRqRB2HzzCY5+uYdKsTQCMOq0DPzmnE011CugJUxCISLl2JDeP577awGOfriEzJ4+r+7flZ+d1pk0Zjulb2SkIRKRccnemLtnOAx+sYFNGJud2bc7vL+7GSc3rBl1apaMgEJFyZ8mWfdz37jJmr8/g5BZ1ef7GFM46uVnQZVVaCgIRKTf2Zebwt49W8uKsjTSqXZ0//7AHQwckUlVXAseUgkBEAufuvDl/C395fzkZh7IZeWoSt11wss4EKiMKAhEJ1OodB7j7rSXMWp9Bn8SGPDc6hR5tGgRdVlxREIhIILJy8vjXp6t56vN11KlRlb9c3pOhAxIr9Uhg5ZWCQETK3NyNe/jt6wtZu+sQV/Rrw10Xd9MtoQMU0yAwsyHAP4AqwHh3f7DA/AbAi0C7cC1/c/dnY1mTiATncHYef/toJRO+Wk+r+jWZeGMKZ+tsoMDFLAjMrArwOHABkAbMMbN33H1ZxGI/BZa5+/fNrBmw0sxecvfsWNUlIsH4Zm06d05ZxMb0TEYMascdQ7pSr6Y6g8uDWB4RpABr3H0dgJm9DFwGRAaBA/UsNEZcXSADyI1hTSJSxrJy8nho6gqe/WoD7ZvUZvLNgzi1U5Ogy5IIsQyCNsDmiOk0YGCBZR4D3gG2AvWAa909v+CKzGwsMBagXbt2MSlWRErfsq37+eUr81m14yAjT23PnRd1o1b1KkGXJQXEMggK6/r3AtMXAguAc4FOwMdm9oW77//Oi9zHAeMAkpOTC65DRMqZvHxn/Bfr+NtHK2lYu7r6Asq5WAZBGpAYMd2W0Df/SKOBB93dgTVmth7oCsyOYV0iEkNb9h7mV68sYNb6DIac0pK/XNGTxhojoFyLZRDMATqbWQdgCzAUuK7AMpuA84AvzKwF0AVYF8OaRCSGPly6ndtfW0hevvP/rurF1f3bEuoClPIsZkHg7rlmdivwIaHTRye4+1IzuyU8/0ngfuA5M1tMqCnpDnffHauaRCQ2snPzeeCD5Tz71QZ6tmnAY9f1pX2TOkGXJVGK6XUE7v4+8H6B556MeLwV+F4saxCR2Nqckcmtk+axMG0fo05L4ncXd6VGVXUIVyS6slhESmzqku3c/vpCAJ4c0Y8hPVoFXJGUhIJARI5bTl4+f3k/1BTUu20DHruuH4mNawddlpSQgkBEjsuuA0f46UvzmL0hg9GnJ/G7i7pRvarGC6jIFAQiErUFm/dyywtz2Xs4m38M7cNlfdoEXZKUAgWBiETllTmb+MNbS2levwZTfnw63VvXD7okKSUKAhE5puzcfO79z1JemrWJMzs35Z9D+9JIF4hVKgoCESnS7oNHuOWFuaRu3MOPzu7I7d/rovGDKyEFgYgUauX2A9z43BzSDx3hX8P68v3erYMuSWJEQSAi/2P6yp3cOmk+tapX4ZWxp9I7sWHQJUkMKQhE5Dsmfr2Be/+zlK4t6zN+ZDKtG9YKuiSJMQWBiACQm5fPfe8u4/lvNnJ+txb8Y2gf6tTQR0Q80G9ZRDh4JJefvjSPz1ftYuxZHbljSFeqJOiuofFCQSAS53YeyGL0s3NYsf0AD1zRk2EpGgUw3igIROLYul0HuWHCbDIOZTN+ZDLndGkedEkSAAWBSJyat2kPY56bQ4IZk28epDOD4piCQCQOfbJ8Bz+dNI8W9WsycXQKSU01iEw8UxCIxJmXZ2/i928upkebBkwYNYCmdWsEXZIELKogMLNk4EygNXAYWAJMc/eMGNYmIqXI3fnnJ2v4+7RVnH1yM54Y3k+nhwoAx7xpiJmNMrN5wO+AWsBKYCdwBvCxmU00M51iIFLOuTv3v7ucv09bxRX92jB+ZLJCQL5V3P+EOsDp7n64sJlm1gfoDGwq7cJEpHTk5Tu/m7KIV1PTGHVaEvdc2p0EXSMgEY4ZBO7+eFHzzKy6uy8o/ZJEpLRk5+Zz2ysLeG/xNn5+XmduO78zZgoB+a6o7idrZtPNLCliOgWYE6OaRKQUHM7O4+bnU3lv8TbuvqQbv7rgZIWAFCraRsIHgKlm9k+gDXARMDpmVYnICdmflcNNz6UyZ2MGD17Rk6G6WliOIaogcPcPzewW4GNgN9DX3bfHtDIRKZGMQ9ncMGEWK7Yd4F/D+nJpL40jIMcW7emjfwCuAc4CegHTzezX7v5eLIsTkeOz80AWw5+exaaMTJ6+IZlzuuqWEVK8aJuGmgIp4bOHvjGzqcB4QEEgUk7s2J/FsKdnsn1fFhNvTGFQxyZBlyQVRLRNQ78oML0RuCAmFYnIcdu27zDXPT2LnftDITAgqXHQJUkFUtwFZePMrGcR8+qY2Y1mNjw2pYlINLbsPcy1T81k14EjPD9moEJAjltxRwRPAH8Ih8ESYBdQk9BFZPWBCcBLMa1QRIq0OSOTYU/PZN/hHF4Yk0Lfdo2CLkkqoOIuKFsAXGNmdYFkoBWhew0td/eVZVCfiBRhU3ooBA5k5fDSTQPp1Va3kZaSibaP4CAwPbaliEi0Nuw+xHVPzyQzJ49JNw+iR5sGQZckFVi0p4+eDvwJaB9+jQHu7h1jV5qIFGb97kMMGzeTI7l5TLppEN1b1w+6JKngoj199BngNmAukBe7ckTkWDalZzJs3Eyy8/KZPHYQXVsqBOTERRsE+9z9g5hWIiLHlLYn1CeQlZvH5JsVAlJ6og2Cz8zsr8AU4MjRJ919XkyqEpHvOHqdwIGsHCbdPIhurRQCUnqiDYKB4X+TI55z4NxjvcjMhgD/AKoA4939wUKWGQw8ClQDdrv72VHWJBIXdu7P4rqnZ5FxKJsXbxqojmEpddGeNXTO8a7YzKoAjxO6AjkNmGNm77j7sohlGhK6VmGIu28yM90YRSTC7oNHuG78LHbsz+KFMSn0SdQpolL6oh2PoIGZPWJmqeGfh82suK8lKcAad1/n7tnAy8BlBZa5Dpji7psA3H3n8e6ASGWVcSibEeNnkbYnk2dHDaB/e10xLLERVRAQuoL4AKE7kF4D7AeeLeY1bYDNEdNp4ecinQw0Cg98M9fMbihsRWY29mgI7dq1K8qSRSquvZmhEFi/+xDPjBzAQN1ATmIo2j6CTu5+ZcT0vWZW3DCVhQ2F5IVsvz9wHlCL0J1NZ7r7qu+8yH0cMA4gOTm54DpEKpX9WTncMGE2a3YeZNwN/Tn9pKZBlySVXLRHBIfN7IyjE+ELzAod0D5CGpAYMd0W2FrIMlPd/ZC77wZmAL2jrEmk0jl0JJdRE2azbOt+nhjej8Fd1G0msRftEcGPgYnhfgEDMoBRxbxmDtDZzDoAW4ChhPoEIr0NPGZmVYHqhM5O+nuUNYlUKlk5eYx9IZWFaft4/Lq+nN+9RdAlSZyI9qyhBUBvM6sfnt4fxWtyzexW4ENCp49OcPel4SEvcfcn3X15eJCbRUA+oVNMl5RwX0QqrNy8fH42eT5frUnnkWt6M6RHq6BLkjhi7kU3uZvZCHd/0cx+Vdh8d38kZpUVITk52VNTU8t6syIxk5/v/Oa1hUyZv4V7f3AKI09LCrokqYTMbK67Jxc2r7gjgjrhf+sVMk+dtiInyN3503+WMmX+Fn7zvZMVAhKI4sYjeCr8cJq7fxU5L9xhLCIn4OGPVvH8NxsZe1ZHfnrOSUGXI3Eq2rOG/hXlcyISpac+X8tjn61h6IBEfndRV8wKO+NaJPaOeURgZqcCpwHNCvQT1CfUASwiJTB59iYe+GAFl/Rqxf9d3lMhIIEqro+gOlA3vFxkP8F+4KpYFSVSmf1n4VZ+/+ZiBndpxt+v6UOVBIWABKu4PoLPgc/N7Dl331hGNYlUWp+t2MltryxgQFJj/j28P9WrRts6KxI70f4vHB++UygAZtbIzD6MUU0ildLMdenc8uJcurWqzzMjk6lVXa2rUj5EGwRN3X3v0Ql33wPo2neRKC1K28tNE1NJbFybiTemUK9mtaBLEvlWtEGQb2btjk6YWXt0HYFIVFbvOMDICbNpWLsaL44ZSOM61YMuSeQ7or3X0F3Al2b2eXj6LGBsbEoSqTw2Z2Qy4plZVK2SwItjBtKyQc2gSxL5H9Hea2iqmfUDBhG66dxt4buFikgRdu7PYvj4WWTl5PPKjwaR1LRO8S8SCUC0RwQANQjddbQq0N3McPcZsSlLpGLbcyibEc/MIv3gEV66eRBdW2qweSm/ogoCM3sIuBZYSuguoRDqI1AQiBRw8Eguo56bw4b0TJ4bPUDjDEu5F+0RwQ+BLu5+JJbFiFR0WTl53DRxDku27OPJEf05rZNGF5PyL9qzhtYBOt9N5Bhy8vK5ddI8Zq3P4OGre3OBBpaRCiLaI4JMYIGZfQJ8e1Tg7j+PSVUiFczRMQWmLd/J/Zedwg/7tgm6JJGoRRsE74R/RKQAd+eed5bw9oKt3H5hF64/NSnokkSOS7Snj06MdSEiFdVfP1zJizM38aOzO/KTwZ2CLkfkuEV71tB6CrmS2N07lnpFIhXIv6ev5Ynpa7luYDvuHKIxBaRiirZpKHKcy5rA1UDj0i9HpOJ4adZGHpq6gh/0bs39l/VQCEiFFdVZQ+6eHvGzxd0fBc6NcW0i5dbbC7Zw91tLOLdrcx6+prfGFJAKLdqmoX4RkwmEjhAKG9BepNKbtmwHv3p1IQM7NOaJ4f2oVkVjCkjFFm3T0MMRj3OB9cA1pV+OSPn29Zrd/GTSPHq0rs/4kQOoWU1jCkjFF+1ZQ+fEuhCR8m7+pj3c9HwqSU1q89zoFOrWOJ5bdYmUX8c8pjWz5yIej4x5NSLl1Irt+xn17Bya1q3Bi2MG0khjCkglUlzjZu+Ix7+IZSEi5dWG3YcYMX42Nasl8NJNA2leX2MKSOVSXBBoFDKJa9v2HWb4+Fnk5efz4piBJDauHXRJIqWuuEbOtmb2T0KD0Rx9/C3da0gqs/SDRxgxfhb7D+cw6eZBdG6hE+WkciouCG6PeJway0JEypP9WTncMGE2aXsO88KYgfRs2yDokkRi5phBoHsMSTw6nJ3HmOfmsGrHAcbdkExKB11EL5VbVFfCmNnHZtYwYrqRmX0Yu7JEgnEkN48fvTiXuRv38Oi1fTmnS/OgSxKJuWgviWzm7nuPTrj7HkB/IVKp5Obl88uXFzBj1S4evKIXl/RqFXRJImUi2iDIM7N2RyfMrD06o0gqkfx8584pi/lgyXbuvqQb1wxIDLokkTIT7aWRdwFfmtnn4emzgLGxKUmkbLk7f3h7Ca/PTeOX53fmpjN1d3WJL9HeYmJq+MZzgwidSnqbu++OaWUiZcDdue/dZbw0axM/HtyJX5zXOeiSRMpccbeY6Br+tx/QDtgKbAHaFbgjaVGvH2JmK81sjZndeYzlBphZnplddXzli5Scu/PQ1JU8+9UGRp+exG8v7KIxBSQuFXdE8CtCTUAPFzLPOcaYBGZWBXgcuABIA+aY2TvuvqyQ5R4CdBaSlKl/fLKaJz9fy/CB7bjn0u4KAYlbxV1HcLQf4CJ3z4qcZ2bF3XAlBVjj7uvCy78MXAYsK7Dcz4A3gAHRFi1yov49fS2PTlvNVf3banQxiXvRnjX0dZTPRWoDbI6YTgs/9y0zawNcDjx5rBWZ2VgzSzWz1F27dkVRrkjRJny5noemruD7vVvz0JW9SNDoYhLnjnlEYGYtCX141zKzvoQ6igHqA8Xdfauwv66Cp5w+Ctzh7nnH+kbm7uOAcQDJyck6bVVK7MWZG7nv3WUMOaUlj2iISRGg+D6CC4FRQFtC/QRH/2oOAL8v5rVpQOTJ2G0JdTZHSgZeDodAU+BiM8t197eKrVzkOL2WuvnbcYb/OayvhpgUCYvmXkMTzexKd3/jONc9B+hsZh0InWk0FLiuwPo7HH0cHgTnXYWAxMJb87dwxxuLOOOkpjwxvB/VqyoERI6K9q+hrZnVt5DxZjbPzL53rBe4ey5wK6GzgZYDr7r7UjO7xcxuOcG6RaL25vw0fvXqAlI6NObpG5I1zrBIAdFeWXyju//DzC4kdI+h0cCzwEfHepG7vw+8X+C5QjuG3X1UlLWIRG3KvDR+/dpCBnVowoRRA6hVXSEgUlC0RwRH+wYuBp5194UU3hksUm68MTcUAqd2VAiIHEu0QTDXzD4iFAQfmlk9ID92ZYmcmNfnpvGb1xdyWqcmPDNSISByLNE2DY0B+gDr3D3TzJoQah4SKXdeS93Mb99YxOmdmvL0DckKAZFiFHcdQVd3X0EoBAA66gpMKc9eTd387dlB6hgWiU7M7jUkUtYUAiIlE9W9htz9nLIpR6RkXvhmA394eylndlYIiByvqPoIzOyKQp7eByx2952lW5LI8Xnq87U88MEKzu/Wgseu66sQEDlOx9NZfCrwWXh6MDATONnM7nP3F2JQm8gxuTt/n7aaf36ymkt7teLv1/bRbSNESiDaIMgHurn7DgAzawH8GxgIzAAUBFKm3J0/v7ecZ75czzXJbXngil66gZxICUUbBElHQyBsJ3Cyu2eYWU4M6hIpUn6+c9dbS5g8exOjTkvinku761bSIicg2iD4wszeBV4LT18FzDCzOsDemFQmUojcvHxuf30Rb87fwk8Gd+J2DS8pcsKiDYKfAlcAZxC6tcRE4A13d0BnFEmZyMrJ4xcvz+fDpTu4/cIu/PSck4IuSaRSiCoI3N3N7Esgm9D1A7PDISBSJvZn5TD2+VRmrsvgj9/vzujTOxT/IhGJSlSnWJjZNcBsQk1C1wCzzOyqWBYmctTO/Vlc+9RMUjfs4dFr+ygEREpZtE1DdwEDjl4zYGbNgGnA67EqTARgw+5DXD9hFukHs3lm1ADOPrlZ0CWJVDrRBkFCgQvH0on+zqUiJbI4bR+jnp1NvjuTbh5En8SGQZckUilFGwRTzexDYHJ4+loKDDgjUpq+XL2bH72QSsPa1Xl+TAqdmtUNuiSRSivazuLbzexK4HRCZw2Nc/c3Y1qZxK23F2zhN68tpGPTuky8MYWWDWoGXZJIpRbtEQHhweuPdwB7kai5O09MX8tfP1xJSlJjnh6ZTINa1YIuS6TSK248ggOEThf9n1mEziqtH5OqJO7k5OVz15uLeTU1jcv6tOb/XdWLGlV18ziRslDcbajrlVUhEr/2Hc7hJy/N5as16fz83JO47YKTdbWwSBmKumlIJBY2Z2Ry43Nz2JB+iL9d3Zur+rcNuiSRuKMgkMAs2LyXmybOITs3n4k3pnBap6ZBlyQSlxQEEog356dxxxuLaVG/Bi+PHcRJzdUKKRIUBYGUqbx856GpK2KsinkAAAurSURBVBg3Yx2DOjbmieH9aVynetBlicQ1BYGUmX2ZOdw6eR5frN7NyFPbc/el3TWimEg5oCCQMrFm5wFumpjKlr2HeeCKngxLaRd0SSISpiCQmJu2bAe/fGUBNaslMPnmQSQnNQ66JBGJoCCQmMnNy+eRj1fxxPS19GhTn3HXJ9O6Ya2gyxKRAhQEEhM7D2Tx88nzmbkug2Epifzx+6dQs5quFBYpjxQEUupmrkvnZ5PncyArRxeJiVQACgIpNfn5zlMz1vHXD1eQ1KQOL4xJoWtL3Y5KpLxTEEipyDiUzW9fX8i05Tu5pFcrHryiJ/Vq6s6hIhWBgkBO2Jerd/OrVxewNzOHP32/OyNPS9JN40QqkJhezWNmQ8xspZmtMbM7C5k/3MwWhX++NrPesaxHSld2bj4PvL+cEc/Mon6tarz109MZdXoHhYBIBROzIwIzqwI8DlwApAFzzOwdd18Wsdh64Gx332NmFwHjgIGxqklKz9pdB/nFy/NZsmU/wwe24+5LulOrus4KEqmIYtk0lAKscfd1AGb2MnAZ8G0QuPvXEcvPBHR6STnn7kyavYk/v7ucGtUSeOr6/lx4SsugyxKRExDLIGgDbI6YTuPY3/bHAB/EsB45QVv2HuaO1xfx5ZrdnH5SEx6+uo/GExapBGIZBIU1FBc27CVmdg6hIDijiPljgbEA7drpHjVlzd15NXUz97+7nHx3/vzDHgwf2E59ASKVRCyDIA1IjJhuC2wtuJCZ9QLGAxe5e3phK3L3cYT6D0hOTi40TCQ2tu07zJ1vLObzVbsY1LExf72qN4mNawddloiUolgGwRygs5l1ALYAQ4HrIhcws3bAFOB6d18Vw1rkOOXnh/oCHpq6gtw8594fnML1g9qTkKCjAJHKJmZB4O65ZnYr8CFQBZjg7kvN7Jbw/CeBe4AmwBPhZoZcd0+OVU0SnZXbD/D7Nxczd+MeTu3YhAeu6ElS0zpBlyUiMWLuFaulJTk52VNTU4Muo1LKysnjn5+sZtyMddSrWZW7LunOlf3aqC9ApBIws7lFfdHWlcUCwIxVu/jD20vYmJ7Jlf3actcl3TSEpEicUBDEuY3ph7j/3eVMW76DDk3rMOmmgZx2UtOgyxKRMqQgiFOHjuTy+GdrGP/FeqpVMe4Y0pUbz0iiRlVdHSwSbxQEcSY/33l74RYe/GAFO/Yf4Yp+bbhjSFda1NeFYSLxSkEQR75as5sHP1jB4i376NW2Af8e0Z9+7RoFXZaIBExBEAeWbt3HQ1NXMmPVLto0rMXDV/fm8r5tdE2AiAAKgkptc0YmD3+0krcWbKVh7WrcfUk3Rgxqr7GDReQ7FASV0OaMTB7/bA2vz02jSoLx48GduOXsTjSopRHDROR/KQgqkc0ZmTz26RremJdGghnXDWzHTwafpDuEisgxKQgqgU3pmTz22WqmzNtCQoIxfGA7bhnciVYNagVdmohUAAqCCmxR2l7GzVjH+4u3UbVKAiMGtefHgzvpVFAROS4KggomP9+Zvmon42asY+a6DOrVqMrNZ3bkxjM6KABEpEQUBBVEVk4e7yzYytNfrGP1zoO0blCTuy/pxrUDEqlXU53AIlJyCoJybsPuQ7w0ayOvpqax73AO3VrV59Fr+3BJr1ZUq5IQdHkiUgkoCMqhvHznsxU7eX7mRmas2kXVBOPCHi0ZMbA9gzo21m2hRaRUKQjKkU3pmbw+L4035qaxZe9hWtSvwW3nn8zQlES1/4tIzCgIAnboSC7vL97G63PTmLU+AzM446Sm/OHSbpzXrYWaf0Qk5hQEAcjNy+ebdem8vWAr7y/eRmZ2HklNanP7hV24vG8bWjfU+f8iUnYUBGUkL9+ZvT6DdxdtZeqS7aQfyqZujap8v1drrk5uS//2jdT2LyKBUBDEUF6+M2/THt5btI33F29j54Ej1KpWhfO6NefSXq0Z3KWZbgAnIoFTEJSyzOxcZqzazbTlO/h0xU4yDmVTvWoC53RpxqW9WnNet+bUrq63XUTKD30ilYKtew8zfeUupi3fwZdrdpOdm0/9mlU5p2tzLujegrNPbqaLvkSk3FIQlMChI7nMXJfOF6t388XqXazddQiAxMa1GDGwPed3b86ApMY640dEKgQFQRSyc/NZvGUf36zdzRerdzNv0x5y8pya1RIY2KEJw1LacWbnZpzcoq46fEWkwlEQFOLQkVzmb9rL7A0ZzF6fzoLNe8nKyQfglNb1GXNGR87s3JT+7Rups1dEKry4DwJ3J23PYRam7WXh5r3M3rCHJVv2kZfvJBh0b12fYSntGNihMQOSGtOkbo2gSxYRKVVxFwTpB4+wKG3ftx/8C9P2kXEoG4DqVRLok9iQW87uyICkxvRv30idvCJS6cVNEHy6Ygf3vL2UtD2HATCDzs3rcl7X5vRKbEiftg3p0rIe1auqg1dE4kvcBEGzujXp3bYh1w9qT+/EhvRo04C6NeJm90VEihQ3n4Q92zbg8eH9gi5DRKTcUTuIiEicUxCIiMQ5BYGISJxTEIiIxDkFgYhInFMQiIjEOQWBiEicUxCIiMQ5c/egazguZrYL2FjClzcFdpdiOUHSvpRP2pfyp7LsB5zYvrR392aFzahwQXAizCzV3ZODrqM0aF/KJ+1L+VNZ9gNity9qGhIRiXMKAhGROBdvQTAu6AJKkfalfNK+lD+VZT8gRvsSV30EIiLyv+LtiEBERApQEIiIxLm4CwIzu9/MFpnZAjP7yMxaB11TSZnZX81sRXh/3jSzhkHXVFJmdrWZLTWzfDOrcKf6mdkQM1tpZmvM7M6g6zkRZjbBzHaa2ZKgazkRZpZoZp+Z2fLw/61fBF1TSZlZTTObbWYLw/tyb6muP976CMysvrvvDz/+OdDd3W8JuKwSMbPvAZ+6e66ZPQTg7ncEXFaJmFk3IB94CviNu6cGXFLUzKwKsAq4AEgD5gDD3H1ZoIWVkJmdBRwEnnf3HkHXU1Jm1gpo5e7zzKweMBf4YUX8vZiZAXXc/aCZVQO+BH7h7jNLY/1xd0RwNATC6gAVNgnd/SN3zw1PzgTaBlnPiXD35e6+Mug6SigFWOPu69w9G3gZuCzgmkrM3WcAGUHXcaLcfZu7zws/PgAsB9oEW1XJeMjB8GS18E+pfXbFXRAAmNn/mdlmYDhwT9D1lJIbgQ+CLiJOtQE2R0ynUUE/cCorM0sC+gKzgq2k5MysipktAHYCH7t7qe1LpQwCM5tmZksK+bkMwN3vcvdE4CXg1mCrPbbi9iW8zF1ALqH9Kbei2ZcKygp5rsIeaVY2ZlYXeAP4ZYEWgQrF3fPcvQ+hI/8UMyu1ZruqpbWi8sTdz49y0UnAe8AfY1jOCSluX8xsJHApcJ6X8w6f4/i9VDRpQGLEdFtga0C1SIRwe/obwEvuPiXoekqDu+81s+nAEKBUOvQr5RHBsZhZ54jJHwArgqrlRJnZEOAO4Afunhl0PXFsDtDZzDqYWXVgKPBOwDXFvXAH6zPAcnd/JOh6ToSZNTt6VqCZ1QLOpxQ/u+LxrKE3gC6EzlDZCNzi7luCrapkzGwNUANIDz81swKfAXU58C+gGbAXWODuFwZbVfTM7GLgUaAKMMHd/y/gkkrMzCYDgwnd8ngH8Ed3fybQokrAzM4AvgAWE/p7B/i9u78fXFUlY2a9gImE/n8lAK+6+32ltv54CwIREfmuuGsaEhGR71IQiIjEOQWBiEicUxCIiMQ5BYGISJxTEIiIxDkFgYhInFMQiJwgMxsQHhOippnVCd8vvsLevlnijy4oEykFZvZnoCZQC0hz9wcCLkkkagoCkVIQvsfQHCALOM3d8wIuSSRqahoSKR2NgbpAPUJHBiIVho4IREqBmb1DaGSyDoSGRyzX41yIRKqU4xGIlCUzuwHIdfdJ4fGLvzazc93906BrE4mGjghEROKc+ghEROKcgkBEJM4pCERE4pyCQEQkzikIRETinIJARCTOKQhEROLc/wcTm3z5QCRI4wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import math\n",
    "def logisticFunction(x):\n",
    "    return 1 / (1 + math.exp(-x))\n",
    "\n",
    "X = np.arange(-3, 3, 0.01)\n",
    "Y = np.array([logisticFunction(x) for x in X])\n",
    "plt.plot(X, Y)\n",
    "plt.xlabel(\"x\")\n",
    "plt.ylabel(\"logisticFunction(x)\")\n",
    "plt.title('a sigmoid activation function')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The advantage of using a sigmoid function for the output units in classification problems is that it yields a result between 0 and 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compiling the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the compilation step we configure the model for the training, by choosing the loss-function, the optimizer and the metrics to be computed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The loss function is the function that is minimized by the optimizer. It calculates the loss from the predictions and the true classification results. We pass in the name of one of the loss funcions from the module keras.losses. We use binary crossentropy because we are dealing with a binary classification problem.\n",
    "\n",
    "The optimizer implements the optimization algorithm. We pass in the name of one of the optimizers from the module keras.optimizers. See [[1]](https://arxiv.org/abs/1609.04747) for an overview of gradient descent optimization algorithms. We use ``rmsprop``. We could also use ``Adam``, but rmsprop is a little bit faster and the performances are similar on this problem.\n",
    "\n",
    "The third parameter is a list of metrics that will be calculatated during training and validation. We calculate the accuracy. Depending on the loss function, accuracy is calculated in different ways. Since we have a binary classification, the binary accuracy is automatically used. It is calculated as the number of correct predictions divided by the number of all predictions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Displaying the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The method summary of the model, displays the architecture of the module in text mode."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_1 (Dense)              (None, 8)                 80        \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 1)                 9         \n",
      "=================================================================\n",
      "Total params: 89\n",
      "Trainable params: 89\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The non-trainable parameters in the summary are weights and biases that are kept constant. They are not modified during the training. We can make weights and biases of a layer non-trainable before compiling the model.\n",
    "\n",
    "```python\n",
    "    model.get_layer(layerName).trainable = False\n",
    "```\n",
    "\n",
    "This is useful if we want to reuse a pre-trained network for a similiar problem. A keras model has methods to save and load the trained model:\n",
    "```python\n",
    "    from keras.models import load_model\n",
    "    model.save(\"mymodel.h5\")\n",
    "    model = load_model(\"mymodel.h5\")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can create a graphical representation using the function model_to_dot. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"200pt\" viewBox=\"0.00 0.00 295.00 221.00\" width=\"266pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g class=\"graph\" id=\"graph0\" transform=\"scale(.9028 .9028) rotate(0) translate(4 217)\">\n",
       "<title>G</title>\n",
       "<polygon fill=\"#ffffff\" points=\"-4,4 -4,-217 291,-217 291,4 -4,4\" stroke=\"transparent\"/>\n",
       "<!-- 139698011328784 -->\n",
       "<g class=\"node\" id=\"node1\">\n",
       "<title>139698011328784</title>\n",
       "<polygon fill=\"none\" points=\"0,-166.5 0,-212.5 287,-212.5 287,-166.5 0,-166.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"81.5\" y=\"-185.8\">dense_1_input: InputLayer</text>\n",
       "<polyline fill=\"none\" points=\"163,-166.5 163,-212.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"190.5\" y=\"-197.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"163,-189.5 218,-189.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"190.5\" y=\"-174.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"218,-166.5 218,-212.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"252.5\" y=\"-197.3\">(None, 9)</text>\n",
       "<polyline fill=\"none\" points=\"218,-189.5 287,-189.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"252.5\" y=\"-174.3\">(None, 9)</text>\n",
       "</g>\n",
       "<!-- 139696219299088 -->\n",
       "<g class=\"node\" id=\"node2\">\n",
       "<title>139696219299088</title>\n",
       "<polygon fill=\"none\" points=\"30.5,-83.5 30.5,-129.5 256.5,-129.5 256.5,-83.5 30.5,-83.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"81.5\" y=\"-102.8\">dense_1: Dense</text>\n",
       "<polyline fill=\"none\" points=\"132.5,-83.5 132.5,-129.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"160\" y=\"-114.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"132.5,-106.5 187.5,-106.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"160\" y=\"-91.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"187.5,-83.5 187.5,-129.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"222\" y=\"-114.3\">(None, 9)</text>\n",
       "<polyline fill=\"none\" points=\"187.5,-106.5 256.5,-106.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"222\" y=\"-91.3\">(None, 8)</text>\n",
       "</g>\n",
       "<!-- 139698011328784&#45;&gt;139696219299088 -->\n",
       "<g class=\"edge\" id=\"edge1\">\n",
       "<title>139698011328784-&gt;139696219299088</title>\n",
       "<path d=\"M143.5,-166.3799C143.5,-158.1745 143.5,-148.7679 143.5,-139.8786\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"147.0001,-139.784 143.5,-129.784 140.0001,-139.784 147.0001,-139.784\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 139696164853456 -->\n",
       "<g class=\"node\" id=\"node3\">\n",
       "<title>139696164853456</title>\n",
       "<polygon fill=\"none\" points=\"30.5,-.5 30.5,-46.5 256.5,-46.5 256.5,-.5 30.5,-.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"81.5\" y=\"-19.8\">dense_2: Dense</text>\n",
       "<polyline fill=\"none\" points=\"132.5,-.5 132.5,-46.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"160\" y=\"-31.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"132.5,-23.5 187.5,-23.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"160\" y=\"-8.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"187.5,-.5 187.5,-46.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"222\" y=\"-31.3\">(None, 8)</text>\n",
       "<polyline fill=\"none\" points=\"187.5,-23.5 256.5,-23.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"222\" y=\"-8.3\">(None, 1)</text>\n",
       "</g>\n",
       "<!-- 139696219299088&#45;&gt;139696164853456 -->\n",
       "<g class=\"edge\" id=\"edge2\">\n",
       "<title>139696219299088-&gt;139696164853456</title>\n",
       "<path d=\"M143.5,-83.3799C143.5,-75.1745 143.5,-65.7679 143.5,-56.8786\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"147.0001,-56.784 143.5,-46.784 140.0001,-56.784 147.0001,-56.784\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import SVG\n",
    "from keras.utils import model_to_dot\n",
    "\n",
    "SVG(model_to_dot(model, show_shapes=True, show_layer_names=True, dpi=65).create(prog='dot', format='svg'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a graphical representation of the network created with [NN-SVG](http://alexlenail.me/NN-SVG/index.html). Each circle represents a unit. In the first column are the input units. In the second column are the units of the first hidden layer. All units in a layer are connected to all units in the following layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img \n",
    "    src=\"./pics/nn.svg\" \n",
    "    alt=\"network\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Literature"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[1] Ruder, S. (2017). [An overview of gradient descent optimization algorithms](https://arxiv.org/abs/1609.04747). ArXiv:1609.04747 [Cs].\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}