python_advanced_features.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "70880e68",
   "metadata": {},
   "source": [
    "\n",
    "<a id='python-advanced-features'></a>\n",
    "<div id=\"qe-notebook-header\" align=\"right\" style=\"text-align:right;\">\n",
    "        <a href=\"https://quantecon.org/\" title=\"quantecon.org\">\n",
    "                <img style=\"width:250px;display:inline;\" width=\"250px\" src=\"https://assets.quantecon.org/img/qe-menubar-logo.svg\" alt=\"QuantEcon\">\n",
    "        </a>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4be02ee",
   "metadata": {},
   "source": [
    "# More Language Features"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "088bf407",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "With this last lecture, our advice is to **skip it on first pass**, unless you have a burning desire to read it.\n",
    "\n",
    "It’s here\n",
    "\n",
    "1. as a reference, so we can link back to it when required, and  \n",
    "1. for those who have worked through a number of applications, and now want to learn more about the Python language  \n",
    "\n",
    "\n",
    "A variety of topics are treated in the lecture, including iterators, decorators and descriptors, and generators."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "daf52080",
   "metadata": {},
   "source": [
    "## Iterables and Iterators\n",
    "\n",
    "\n",
    "<a id='index-0'></a>\n",
    "We’ve [already said something](https://python-programming.quantecon.org/python_essentials.html#iterating-version-1) about iterating in Python.\n",
    "\n",
    "Now let’s look more closely at how it all works, focusing in Python’s implementation of the `for` loop.\n",
    "\n",
    "\n",
    "<a id='iterators'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b788d853",
   "metadata": {},
   "source": [
    "### Iterators\n",
    "\n",
    "\n",
    "<a id='index-1'></a>\n",
    "Iterators are a uniform interface to stepping through elements in a collection.\n",
    "\n",
    "Here we’ll talk about using iterators—later we’ll learn how to build our own.\n",
    "\n",
    "Formally, an *iterator* is an object with a `__next__` method.\n",
    "\n",
    "For example, file objects are iterators .\n",
    "\n",
    "To see this, let’s have another look at the [US cities data](https://python-programming.quantecon.org/python_essentials.html#us-cities-data),\n",
    "which is written to the present working directory in the following cell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9743de2a",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "%%file us_cities.txt\n",
    "new york: 8244910\n",
    "los angeles: 3819702\n",
    "chicago: 2707120\n",
    "houston: 2145146\n",
    "philadelphia: 1536471\n",
    "phoenix: 1469471\n",
    "san antonio: 1359758\n",
    "san diego: 1326179\n",
    "dallas: 1223229"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76d055ed",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "f = open('us_cities.txt')\n",
    "f.__next__()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80f62973",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "f.__next__()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3c8d1cf",
   "metadata": {},
   "source": [
    "We see that file objects do indeed have a `__next__` method, and that calling this method returns the next line in the file.\n",
    "\n",
    "The next method can also be accessed via the builtin function `next()`,\n",
    "which directly calls this method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b2cd510f",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "next(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac64f59a",
   "metadata": {},
   "source": [
    "The objects returned by `enumerate()` are also iterators"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ac4c36c",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "e = enumerate(['foo', 'bar'])\n",
    "next(e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07d5f6aa",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "next(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6422c5d6",
   "metadata": {},
   "source": [
    "as are the reader objects from the `csv` module .\n",
    "\n",
    "Let’s create a small csv file that contains data from the NIKKEI index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b5f6e65",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "%%file test_table.csv\n",
    "Date,Open,High,Low,Close,Volume,Adj Close\n",
    "2009-05-21,9280.35,9286.35,9189.92,9264.15,133200,9264.15\n",
    "2009-05-20,9372.72,9399.40,9311.61,9344.64,143200,9344.64\n",
    "2009-05-19,9172.56,9326.75,9166.97,9290.29,167000,9290.29\n",
    "2009-05-18,9167.05,9167.82,8997.74,9038.69,147800,9038.69\n",
    "2009-05-15,9150.21,9272.08,9140.90,9265.02,172000,9265.02\n",
    "2009-05-14,9212.30,9223.77,9052.41,9093.73,169400,9093.73\n",
    "2009-05-13,9305.79,9379.47,9278.89,9340.49,176000,9340.49\n",
    "2009-05-12,9358.25,9389.61,9298.61,9298.61,188400,9298.61\n",
    "2009-05-11,9460.72,9503.91,9342.75,9451.98,230800,9451.98\n",
    "2009-05-08,9351.40,9464.43,9349.57,9432.83,220200,9432.83"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "188fc1c5",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "from csv import reader\n",
    "\n",
    "f = open('test_table.csv', 'r')\n",
    "nikkei_data = reader(f)\n",
    "next(nikkei_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e8e7bf3c",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "next(nikkei_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6931c590",
   "metadata": {},
   "source": [
    "### Iterators in For Loops\n",
    "\n",
    "\n",
    "<a id='index-2'></a>\n",
    "All iterators can be placed to the right of the `in` keyword in `for` loop statements.\n",
    "\n",
    "In fact this is how the `for` loop works:  If we write"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5423e4c",
   "metadata": {
    "hide-output": false
   },
   "source": [
    "```python3\n",
    "for x in iterator:\n",
    "    <code block>\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "053b81f1",
   "metadata": {},
   "source": [
    "then the interpreter\n",
    "\n",
    "- calls `iterator.___next___()` and binds `x` to the result  \n",
    "- executes the code block  \n",
    "- repeats until a `StopIteration` error occurs  \n",
    "\n",
    "\n",
    "So now you know how this magical looking syntax works"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b94eeac6",
   "metadata": {
    "hide-output": false
   },
   "source": [
    "```python3\n",
    "f = open('somefile.txt', 'r')\n",
    "for line in f:\n",
    "    # do something\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5c14232",
   "metadata": {},
   "source": [
    "The interpreter just keeps\n",
    "\n",
    "1. calling `f.__next__()` and binding `line` to the result  \n",
    "1. executing the body of the loop  \n",
    "\n",
    "\n",
    "This continues until a `StopIteration` error occurs."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7cd392fc",
   "metadata": {},
   "source": [
    "### Iterables\n",
    "\n",
    "\n",
    "<a id='index-3'></a>\n",
    "You already know that we can put a Python list to the right of `in` in a `for` loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6bc5e81c",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "for i in ['spam', 'eggs']:\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38969f8c",
   "metadata": {},
   "source": [
    "So does that mean that a list is an iterator?\n",
    "\n",
    "The answer is no"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4be652d8",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "x = ['foo', 'bar']\n",
    "type(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "abf99ad5",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "next(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "198ea26a",
   "metadata": {},
   "source": [
    "So why can we iterate over a list in a `for` loop?\n",
    "\n",
    "The reason is that a list is *iterable* (as opposed to an iterator).\n",
    "\n",
    "Formally, an object is iterable if it can be converted to an iterator using the built-in function `iter()`.\n",
    "\n",
    "Lists are one such object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb5e5305",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "x = ['foo', 'bar']\n",
    "type(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73192972",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "y = iter(x)\n",
    "type(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f2703b2",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "next(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "711df779",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "next(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4e1434e",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "next(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "450b5647",
   "metadata": {},
   "source": [
    "Many other objects are iterable, such as dictionaries and tuples.\n",
    "\n",
    "Of course, not all objects are iterable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a4e7890",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "iter(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f7334e3",
   "metadata": {},
   "source": [
    "To conclude our discussion of `for` loops\n",
    "\n",
    "- `for` loops work on either iterators or iterables.  \n",
    "- In the second case, the iterable is converted into an iterator before the loop starts.  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fe0e678",
   "metadata": {},
   "source": [
    "### Iterators and built-ins\n",
    "\n",
    "\n",
    "<a id='index-4'></a>\n",
    "Some built-in functions that act on sequences also work with iterables\n",
    "\n",
    "- `max()`, `min()`, `sum()`, `all()`, `any()`  \n",
    "\n",
    "\n",
    "For example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15b68d67",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "x = [10, -10]\n",
    "max(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "631ce08d",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "y = iter(x)\n",
    "type(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f55a708",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "max(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "782fdb94",
   "metadata": {},
   "source": [
    "One thing to remember about iterators is that they are depleted by use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8520a447",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "x = [10, -10]\n",
    "y = iter(x)\n",
    "max(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b138eda",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "max(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "532c2d00",
   "metadata": {},
   "source": [
    "## `*` and `**` Operators\n",
    "\n",
    "`*` and `**` are convenient and widely used tools to unpack lists and tuples and to allow users to define functions that take arbitrarily many arguments as input.\n",
    "\n",
    "In this section, we will explore how to use them and distinguish their use cases."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "504cc3cf",
   "metadata": {},
   "source": [
    "### Unpacking Arguments\n",
    "\n",
    "When we operate on a list of parameters, we often need to extract the content of the list as individual arguments instead of a collection when passing them into functions.\n",
    "\n",
    "Luckily, the `*` operator can help us to unpack lists and tuples into [*positional arguments*](https://python-programming.quantecon.org/functions.html#pos-args) in function calls.\n",
    "\n",
    "To make things concrete, consider the following examples:\n",
    "\n",
    "Without `*`, the `print` function prints a list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "286f83af",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "l1 = ['a', 'b', 'c']\n",
    "\n",
    "print(l1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e1ec51d",
   "metadata": {},
   "source": [
    "While the `print` function prints individual elements since `*` unpacks the list into individual arguments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b5128736",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "print(*l1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59785922",
   "metadata": {},
   "source": [
    "Unpacking the list using `*` into positional arguments is equivalent to defining them individually when calling the function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0fdf0f89",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "print('a', 'b', 'c')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62039d7a",
   "metadata": {},
   "source": [
    "However, `*` operator is more convenient if we want to reuse them again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "531901d6",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "l1.append('d')\n",
    "\n",
    "print(*l1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e06854f",
   "metadata": {},
   "source": [
    "Similarly, `**` is used to unpack arguments.\n",
    "\n",
    "The difference is that `**` unpacks *dictionaries* into *keyword arguments*.\n",
    "\n",
    "`**` is often used when there are many keyword arguments we want to reuse.\n",
    "\n",
    "For example, assuming we want to draw multiple graphs using the same graphical settings,\n",
    "it may involve repetitively setting many graphical parameters, usually defined using keyword arguments.\n",
    "\n",
    "In this case, we can use a dictionary to store these parameters and use `**` to unpack dictionaries into keyword arguments when they are needed.\n",
    "\n",
    "Let’s walk through a simple example together and distinguish the use of `*` and `**`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a79af198",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Set up the frame and subplots\n",
    "fig, ax = plt.subplots(2, 1)\n",
    "plt.subplots_adjust(hspace=0.7)\n",
    "\n",
    "# Create a function that generates synthetic data\n",
    "def generate_data(β_0, β_1, σ=30, n=100):\n",
    "    x_values = np.arange(0, n, 1)\n",
    "    y_values = β_0 + β_1 * x_values + np.random.normal(size=n, scale=σ)\n",
    "    return x_values, y_values\n",
    "\n",
    "# Store the keyword arguments for lines and legends in a dictionary\n",
    "line_kargs = {'lw': 1.5, 'alpha': 0.7}\n",
    "legend_kargs = {'bbox_to_anchor': (0., 1.02, 1., .102), \n",
    "                'loc': 3, \n",
    "                'ncol': 4,\n",
    "                'mode': 'expand', \n",
    "                'prop': {'size': 7}}\n",
    "\n",
    "β_0s = [10, 20, 30]\n",
    "β_1s = [1, 2, 3]\n",
    "\n",
    "# Use a for loop to plot lines\n",
    "def generate_plots(β_0s, β_1s, idx, line_kargs, legend_kargs):\n",
    "    label_list = []\n",
    "    for βs in zip(β_0s, β_1s):\n",
    "    \n",
    "        # Use * to unpack tuple βs and the tuple output from the generate_data function\n",
    "        # Use ** to unpack the dictionary of keyword arguments for lines\n",
    "        ax[idx].plot(*generate_data(*βs), **line_kargs)\n",
    "\n",
    "        label_list.append(f'$β_0 = {βs[0]}$ | $β_1 = {βs[1]}$')\n",
    "\n",
    "    # Use ** to unpack the dictionary of keyword arguments for legends\n",
    "    ax[idx].legend(label_list, **legend_kargs)\n",
    "\n",
    "generate_plots(β_0s, β_1s, 0, line_kargs, legend_kargs)\n",
    "\n",
    "# We can easily reuse and update our parameters\n",
    "β_1s.append(-2)\n",
    "β_0s.append(40)\n",
    "line_kargs['lw'] = 2\n",
    "line_kargs['alpha'] = 0.4\n",
    "\n",
    "generate_plots(β_0s, β_1s, 1, line_kargs, legend_kargs)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "462c2f3f",
   "metadata": {},
   "source": [
    "In this example, `*` unpacked the zipped parameters `βs` and the output of `generate_data` function stored in tuples,\n",
    "while `**` unpacked graphical parameters stored in `legend_kargs` and `line_kargs`.\n",
    "\n",
    "To summarize, when `*list`/`*tuple` and `**dictionary` are passed into *function calls*, they are unpacked into individual arguments instead of a collection.\n",
    "\n",
    "The difference is that `*` will unpack lists and tuples into *positional arguments*, while `**` will unpack dictionaries into *keyword arguments*."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c921f0d",
   "metadata": {},
   "source": [
    "### Arbitrary Arguments\n",
    "\n",
    "When we *define* functions, it is sometimes desirable to allow users to put as many arguments as they want into a function.\n",
    "\n",
    "You might have noticed that the `ax.plot()` function could handle arbitrarily many arguments.\n",
    "\n",
    "If we look at the [documentation](https://github.com/matplotlib/matplotlib/blob/v3.6.2/lib/matplotlib/axes/_axes.py#L1417-L1669) of the function, we can see the function is defined as"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b9c30c39",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "Axes.plot(*args, scalex=True, scaley=True, data=None, **kwargs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1489c92",
   "metadata": {},
   "source": [
    "We found `*` and `**` operators again in the context of the *function definition*.\n",
    "\n",
    "In fact, `*args` and `**kargs` are ubiquitous in the scientific libraries in Python to reduce redundancy and allow flexible inputs.\n",
    "\n",
    "`*args` enables the function to handle *positional arguments* with a variable size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47256abb",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "l1 = ['a', 'b', 'c']\n",
    "l2 = ['b', 'c', 'd']\n",
    "\n",
    "def arb(*ls):\n",
    "    print(ls)\n",
    "\n",
    "arb(l1, l2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6625caef",
   "metadata": {},
   "source": [
    "The inputs are passed into the function and stored in a tuple.\n",
    "\n",
    "Let’s try more inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cbfb9e0c",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "l3 = ['z', 'x', 'b']\n",
    "arb(l1, l2, l3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "625baf04",
   "metadata": {},
   "source": [
    "Similarly, Python allows us to use `**kargs` to pass arbitrarily many *keyword arguments* into functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17bebe63",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "def arb(**ls):\n",
    "    print(ls)\n",
    "\n",
    "# Note that these are keyword arguments\n",
    "arb(l1=l1, l2=l2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "961e482b",
   "metadata": {},
   "source": [
    "We can see Python uses a dictionary to store these keyword arguments.\n",
    "\n",
    "Let’s try more inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d7f300a",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "arb(l1=l1, l2=l2, l3=l3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e702ae4",
   "metadata": {},
   "source": [
    "Overall, `*args` and `**kargs` are used when *defining a function*; they enable the function to take input with an arbitrary size.\n",
    "\n",
    "The difference is that functions with `*args` will be able to take *positional arguments* with an arbitrary size, while `**kargs` will allow functions to take arbitrarily many *keyword arguments*."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "feb7218d",
   "metadata": {},
   "source": [
    "## Decorators and Descriptors\n",
    "\n",
    "\n",
    "<a id='index-6'></a>\n",
    "Let’s look at some special syntax elements that are routinely used by Python developers.\n",
    "\n",
    "You might not need the following concepts immediately, but you will see them\n",
    "in other people’s code.\n",
    "\n",
    "Hence you need to understand them at some stage of your Python education."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea088961",
   "metadata": {},
   "source": [
    "### Decorators\n",
    "\n",
    "\n",
    "<a id='index-7'></a>\n",
    "Decorators are a bit of syntactic sugar that, while easily avoided, have turned out to be popular.\n",
    "\n",
    "It’s very easy to say what decorators do.\n",
    "\n",
    "On the other hand it takes a bit of effort to explain *why* you might use them."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b53c0211",
   "metadata": {},
   "source": [
    "#### An Example\n",
    "\n",
    "Suppose we are working on a program that looks something like this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5c42560",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def f(x):\n",
    "    return np.log(np.log(x))\n",
    "\n",
    "def g(x):\n",
    "    return np.sqrt(42 * x)\n",
    "\n",
    "# Program continues with various calculations using f and g"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac7586b5",
   "metadata": {},
   "source": [
    "Now suppose there’s a problem: occasionally negative numbers get fed to `f` and `g` in the calculations that follow.\n",
    "\n",
    "If you try it, you’ll see that when these functions are called with negative numbers they return a NumPy object called `nan` .\n",
    "\n",
    "This stands for “not a number” (and indicates that you are trying to evaluate\n",
    "a mathematical function at a point where it is not defined).\n",
    "\n",
    "Perhaps this isn’t what we want, because it causes other problems that are hard to pick up later on.\n",
    "\n",
    "Suppose that instead we want the program to terminate whenever this happens, with a sensible error message.\n",
    "\n",
    "This change is easy enough to implement"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3de2c833",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def f(x):\n",
    "    assert x >= 0, \"Argument must be nonnegative\"\n",
    "    return np.log(np.log(x))\n",
    "\n",
    "def g(x):\n",
    "    assert x >= 0, \"Argument must be nonnegative\"\n",
    "    return np.sqrt(42 * x)\n",
    "\n",
    "# Program continues with various calculations using f and g"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40b8a8f0",
   "metadata": {},
   "source": [
    "Notice however that there is some repetition here, in the form of two identical lines of code.\n",
    "\n",
    "Repetition makes our code longer and harder to maintain, and hence is\n",
    "something we try hard to avoid.\n",
    "\n",
    "Here it’s not a big deal, but imagine now that instead of just `f` and `g`, we have 20 such functions that we need to modify in exactly the same way.\n",
    "\n",
    "This means we need to repeat the test logic (i.e., the `assert` line testing nonnegativity) 20 times.\n",
    "\n",
    "The situation is still worse if the test logic is longer and more complicated.\n",
    "\n",
    "In this kind of scenario the following approach would be neater"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37e5830e",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def check_nonneg(func):\n",
    "    def safe_function(x):\n",
    "        assert x >= 0, \"Argument must be nonnegative\"\n",
    "        return func(x)\n",
    "    return safe_function\n",
    "\n",
    "def f(x):\n",
    "    return np.log(np.log(x))\n",
    "\n",
    "def g(x):\n",
    "    return np.sqrt(42 * x)\n",
    "\n",
    "f = check_nonneg(f)\n",
    "g = check_nonneg(g)\n",
    "# Program continues with various calculations using f and g"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a74e8110",
   "metadata": {},
   "source": [
    "This looks complicated so let’s work through it slowly.\n",
    "\n",
    "To unravel the logic, consider what happens when we say `f = check_nonneg(f)`.\n",
    "\n",
    "This calls the function `check_nonneg` with parameter `func` set equal to `f`.\n",
    "\n",
    "Now `check_nonneg` creates a new function called `safe_function` that\n",
    "verifies `x` as nonnegative and then calls `func` on it (which is the same as `f`).\n",
    "\n",
    "Finally, the global name `f` is then set equal to `safe_function`.\n",
    "\n",
    "Now the behavior of `f` is as we desire, and the same is true of `g`.\n",
    "\n",
    "At the same time, the test logic is written only once."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2086422",
   "metadata": {},
   "source": [
    "#### Enter Decorators\n",
    "\n",
    "\n",
    "<a id='index-8'></a>\n",
    "The last version of our code is still not ideal.\n",
    "\n",
    "For example, if someone is reading our code and wants to know how\n",
    "`f` works, they will be looking for the function definition, which is"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ade3298",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "def f(x):\n",
    "    return np.log(np.log(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfaa0e2f",
   "metadata": {},
   "source": [
    "They may well miss the line `f = check_nonneg(f)`.\n",
    "\n",
    "For this and other reasons, decorators were introduced to Python.\n",
    "\n",
    "With decorators, we can replace the lines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc3f368a",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "def f(x):\n",
    "    return np.log(np.log(x))\n",
    "\n",
    "def g(x):\n",
    "    return np.sqrt(42 * x)\n",
    "\n",
    "f = check_nonneg(f)\n",
    "g = check_nonneg(g)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52da08c0",
   "metadata": {},
   "source": [
    "with"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28b7902b",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "@check_nonneg\n",
    "def f(x):\n",
    "    return np.log(np.log(x))\n",
    "\n",
    "@check_nonneg\n",
    "def g(x):\n",
    "    return np.sqrt(42 * x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "272293c0",
   "metadata": {},
   "source": [
    "These two pieces of code do exactly the same thing.\n",
    "\n",
    "If they do the same thing, do we really need decorator syntax?\n",
    "\n",
    "Well, notice that the decorators sit right on top of the function definitions.\n",
    "\n",
    "Hence anyone looking at the definition of the function will see them and be\n",
    "aware that the function is modified.\n",
    "\n",
    "In the opinion of many people, this makes the decorator syntax a significant improvement to the language.\n",
    "\n",
    "\n",
    "<a id='descriptors'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb1f5fa7",
   "metadata": {},
   "source": [
    "### Descriptors\n",
    "\n",
    "\n",
    "<a id='index-9'></a>\n",
    "Descriptors solve a common problem regarding management of variables.\n",
    "\n",
    "To understand the issue, consider a `Car` class, that simulates a car.\n",
    "\n",
    "Suppose that this class defines the variables `miles` and `kms`, which give the distance traveled in miles\n",
    "and kilometers respectively.\n",
    "\n",
    "A highly simplified version of the class might look as follows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "83e8a382",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "class Car:\n",
    "\n",
    "    def __init__(self, miles=1000):\n",
    "        self.miles = miles\n",
    "        self.kms = miles * 1.61\n",
    "\n",
    "    # Some other functionality, details omitted"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "039b707d",
   "metadata": {},
   "source": [
    "One potential problem we might have here is that a user alters one of these\n",
    "variables but not the other"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e35e2ca",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "car = Car()\n",
    "car.miles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f246ab8c",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "car.kms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "268861a3",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "car.miles = 6000\n",
    "car.kms"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1bdbc847",
   "metadata": {},
   "source": [
    "In the last two lines we see that `miles` and `kms` are out of sync.\n",
    "\n",
    "What we really want is some mechanism whereby each time a user sets one of these variables, *the other is automatically updated*."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad9cddd5",
   "metadata": {},
   "source": [
    "#### A Solution\n",
    "\n",
    "In Python, this issue is solved using *descriptors*.\n",
    "\n",
    "A descriptor is just a Python object that implements certain methods.\n",
    "\n",
    "These methods are triggered when the object is accessed through dotted attribute notation.\n",
    "\n",
    "The best way to understand this is to see it in action.\n",
    "\n",
    "Consider this alternative version of the `Car` class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c519b83",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "class Car:\n",
    "\n",
    "    def __init__(self, miles=1000):\n",
    "        self._miles = miles\n",
    "        self._kms = miles * 1.61\n",
    "\n",
    "    def set_miles(self, value):\n",
    "        self._miles = value\n",
    "        self._kms = value * 1.61\n",
    "\n",
    "    def set_kms(self, value):\n",
    "        self._kms = value\n",
    "        self._miles = value / 1.61\n",
    "\n",
    "    def get_miles(self):\n",
    "        return self._miles\n",
    "\n",
    "    def get_kms(self):\n",
    "        return self._kms\n",
    "\n",
    "    miles = property(get_miles, set_miles)\n",
    "    kms = property(get_kms, set_kms)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "524aedb4",
   "metadata": {},
   "source": [
    "First let’s check that we get the desired behavior"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f223e109",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "car = Car()\n",
    "car.miles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d7b7d8d",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "car.miles = 6000\n",
    "car.kms"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71cab6e0",
   "metadata": {},
   "source": [
    "Yep, that’s what we want — `car.kms` is automatically updated."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2c4e0f6",
   "metadata": {},
   "source": [
    "#### How it Works\n",
    "\n",
    "The names `_miles` and `_kms` are arbitrary names we are using to store the values of the variables.\n",
    "\n",
    "The objects `miles` and `kms` are *properties*, a common kind of descriptor.\n",
    "\n",
    "The methods `get_miles`, `set_miles`, `get_kms` and `set_kms` define\n",
    "what happens when you get (i.e. access) or set (bind) these variables\n",
    "\n",
    "- So-called “getter” and “setter” methods.  \n",
    "\n",
    "\n",
    "The builtin Python function `property` takes getter and setter methods and creates a property.\n",
    "\n",
    "For example, after `car` is created as an instance of `Car`, the object `car.miles` is a property.\n",
    "\n",
    "Being a property, when we set its value via `car.miles = 6000` its setter\n",
    "method is triggered — in this case `set_miles`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9778590",
   "metadata": {},
   "source": [
    "#### Decorators and Properties\n",
    "\n",
    "\n",
    "<a id='index-11'></a>\n",
    "These days its very common to see the `property` function used via a decorator.\n",
    "\n",
    "Here’s another version of our `Car` class that works as before but now uses\n",
    "decorators to set up the properties"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38cc0ae7",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "class Car:\n",
    "\n",
    "    def __init__(self, miles=1000):\n",
    "        self._miles = miles\n",
    "        self._kms = miles * 1.61\n",
    "\n",
    "    @property\n",
    "    def miles(self):\n",
    "        return self._miles\n",
    "\n",
    "    @property\n",
    "    def kms(self):\n",
    "        return self._kms\n",
    "\n",
    "    @miles.setter\n",
    "    def miles(self, value):\n",
    "        self._miles = value\n",
    "        self._kms = value * 1.61\n",
    "\n",
    "    @kms.setter\n",
    "    def kms(self, value):\n",
    "        self._kms = value\n",
    "        self._miles = value / 1.61"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "390eb006",
   "metadata": {},
   "source": [
    "We won’t go through all the details here.\n",
    "\n",
    "For further information you can refer to the [descriptor documentation](https://docs.python.org/3/howto/descriptor.html).\n",
    "\n",
    "\n",
    "<a id='paf-generators'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c11b4415",
   "metadata": {},
   "source": [
    "## Generators\n",
    "\n",
    "\n",
    "<a id='index-12'></a>\n",
    "A generator is a kind of iterator (i.e., it works with a `next` function).\n",
    "\n",
    "We will study two ways to build generators: generator expressions and generator functions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "106d4ef9",
   "metadata": {},
   "source": [
    "### Generator Expressions\n",
    "\n",
    "The easiest way to build generators is using *generator expressions*.\n",
    "\n",
    "Just like a list comprehension, but with round brackets.\n",
    "\n",
    "Here is the list comprehension:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fbe6ca5e",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "singular = ('dog', 'cat', 'bird')\n",
    "type(singular)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f7d2509",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "plural = [string + 's' for string in singular]\n",
    "plural"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e87cbf09",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "type(plural)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fccf8277",
   "metadata": {},
   "source": [
    "And here is the generator expression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "212c147a",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "singular = ('dog', 'cat', 'bird')\n",
    "plural = (string + 's' for string in singular)\n",
    "type(plural)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1f70449",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "next(plural)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35fb1fab",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "next(plural)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b43acc3c",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "next(plural)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ad8c449",
   "metadata": {},
   "source": [
    "Since `sum()` can be called on iterators, we can do this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4981b88c",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "sum((x * x for x in range(10)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1cbf00be",
   "metadata": {},
   "source": [
    "The function `sum()` calls `next()` to get the items, adds successive terms.\n",
    "\n",
    "In fact, we can omit the outer brackets in this case"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de1b16e4",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "sum(x * x for x in range(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7afd2a7",
   "metadata": {},
   "source": [
    "### Generator Functions\n",
    "\n",
    "\n",
    "<a id='index-13'></a>\n",
    "The most flexible way to create generator objects is to use generator functions.\n",
    "\n",
    "Let’s look at some examples."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c7f3080",
   "metadata": {},
   "source": [
    "#### Example 1\n",
    "\n",
    "Here’s a very simple example of a generator function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b7bc506",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "def f():\n",
    "    yield 'start'\n",
    "    yield 'middle'\n",
    "    yield 'end'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77f9ba6b",
   "metadata": {},
   "source": [
    "It looks like a function, but uses a keyword `yield` that we haven’t met before.\n",
    "\n",
    "Let’s see how it works after running this code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2894321b",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "type(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94a067ab",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "gen = f()\n",
    "gen"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c25584c1",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "next(gen)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d94b46ab",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "next(gen)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "279e2cce",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "next(gen)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f516a31b",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "next(gen)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "614fd5f2",
   "metadata": {},
   "source": [
    "The generator function `f()` is used to create generator objects (in this case `gen`).\n",
    "\n",
    "Generators are iterators, because they support a `next` method.\n",
    "\n",
    "The first call to `next(gen)`\n",
    "\n",
    "- Executes code in the body of `f()` until it meets a `yield` statement.  \n",
    "- Returns that value to the caller of `next(gen)`.  \n",
    "\n",
    "\n",
    "The second call to `next(gen)` starts executing *from the next line*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "951e8137",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "def f():\n",
    "    yield 'start'\n",
    "    yield 'middle'  # This line!\n",
    "    yield 'end'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "263a7872",
   "metadata": {},
   "source": [
    "and continues until the next `yield` statement.\n",
    "\n",
    "At that point it returns the value following `yield` to the caller of `next(gen)`, and so on.\n",
    "\n",
    "When the code block ends, the generator throws a `StopIteration` error."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6788ba73",
   "metadata": {},
   "source": [
    "#### Example 2\n",
    "\n",
    "Our next example receives an argument `x` from the caller"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "86e4bfa5",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "def g(x):\n",
    "    while x < 100:\n",
    "        yield x\n",
    "        x = x * x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e847aea",
   "metadata": {},
   "source": [
    "Let’s see how it works"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89394296",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "g"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04c21c3e",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "gen = g(2)\n",
    "type(gen)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50a08bf1",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "next(gen)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b9c2ebb5",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "next(gen)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4833253e",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "next(gen)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b27facc8",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "next(gen)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3da32755",
   "metadata": {},
   "source": [
    "The call `gen = g(2)` binds `gen` to a generator.\n",
    "\n",
    "Inside the generator, the name `x` is bound to `2`.\n",
    "\n",
    "When we call `next(gen)`\n",
    "\n",
    "- The body of `g()` executes until the line `yield x`, and the value of `x` is returned.  \n",
    "\n",
    "\n",
    "Note that value of `x` is retained inside the generator.\n",
    "\n",
    "When we call `next(gen)` again, execution continues *from where it left off*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "08e5d84f",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "def g(x):\n",
    "    while x < 100:\n",
    "        yield x\n",
    "        x = x * x  # execution continues from here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "412545c4",
   "metadata": {},
   "source": [
    "When `x < 100` fails, the generator throws a `StopIteration` error.\n",
    "\n",
    "Incidentally, the loop inside the generator can be infinite"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "34f7ad62",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "def g(x):\n",
    "    while 1:\n",
    "        yield x\n",
    "        x = x * x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a2a128b",
   "metadata": {},
   "source": [
    "### Advantages of Iterators\n",
    "\n",
    "What’s the advantage of using an iterator here?\n",
    "\n",
    "Suppose we want to sample a binomial(n,0.5).\n",
    "\n",
    "One way to do it is as follows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "842690c7",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "import random\n",
    "n = 10000000\n",
    "draws = [random.uniform(0, 1) < 0.5 for i in range(n)]\n",
    "sum(draws)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e167ebc",
   "metadata": {},
   "source": [
    "But we are creating two huge lists here,  `range(n)` and `draws`.\n",
    "\n",
    "This uses lots of memory and is very slow.\n",
    "\n",
    "If we make `n` even bigger then this happens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0a14434d",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "n = 100000000\n",
    "draws = [random.uniform(0, 1) < 0.5 for i in range(n)]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f22c234",
   "metadata": {},
   "source": [
    "We can avoid these problems using iterators.\n",
    "\n",
    "Here is the generator function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c59a778",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "def f(n):\n",
    "    i = 1\n",
    "    while i <= n:\n",
    "        yield random.uniform(0, 1) < 0.5\n",
    "        i += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f9ddc1c",
   "metadata": {},
   "source": [
    "Now let’s do the sum"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "458d63f6",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "n = 10000000\n",
    "draws = f(n)\n",
    "draws"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32e78dd7",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "sum(draws)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54205b69",
   "metadata": {},
   "source": [
    "In summary, iterables\n",
    "\n",
    "- avoid the need to create big lists/tuples, and  \n",
    "- provide a uniform interface to iteration that can be used transparently in `for` loops  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd11da82",
   "metadata": {},
   "source": [
    "## Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7c27b41",
   "metadata": {},
   "source": [
    "## Exercise 21.1\n",
    "\n",
    "Complete the following code, and test it using [this csv file](https://raw.githubusercontent.com/QuantEcon/lecture-python-programming/master/source/_static/lecture_specific/python_advanced_features/test_table.csv), which we assume that you’ve put in your current working directory"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7b1f1a6",
   "metadata": {
    "hide-output": false
   },
   "source": [
    "```python3\n",
    "def column_iterator(target_file, column_number):\n",
    "    \"\"\"A generator function for CSV files.\n",
    "    When called with a file name target_file (string) and column number\n",
    "    column_number (integer), the generator function returns a generator\n",
    "    that steps through the elements of column column_number in file\n",
    "    target_file.\n",
    "    \"\"\"\n",
    "    # put your code here\n",
    "\n",
    "dates = column_iterator('test_table.csv', 1)\n",
    "\n",
    "for date in dates:\n",
    "    print(date)\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d4d7dfc",
   "metadata": {},
   "source": [
    "## Solution to[ Exercise 21.1](https://python-programming.quantecon.org/#paf_ex1)\n",
    "\n",
    "One solution is as follows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "710fb8fd",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "def column_iterator(target_file, column_number):\n",
    "    \"\"\"A generator function for CSV files.\n",
    "    When called with a file name target_file (string) and column number\n",
    "    column_number (integer), the generator function returns a generator\n",
    "    which steps through the elements of column column_number in file\n",
    "    target_file.\n",
    "    \"\"\"\n",
    "    f = open(target_file, 'r')\n",
    "    for line in f:\n",
    "        yield line.split(',')[column_number - 1]\n",
    "    f.close()\n",
    "\n",
    "dates = column_iterator('test_table.csv', 1)\n",
    "\n",
    "i = 1\n",
    "for date in dates:\n",
    "    print(date)\n",
    "    if i == 10:\n",
    "        break\n",
    "    i += 1"
   ]
  }
 ],
 "metadata": {
  "date": 1732936631.8544333,
  "filename": "python_advanced_features.md",
  "kernelspec": {
   "display_name": "Python",
   "language": "python3",
   "name": "python3"
  },
  "title": "More Language Features"
 },
 "nbformat": 4,
 "nbformat_minor": 5
}