04. exercise-the-sliding-window.ipynb

{"metadata":{"jupytext":{"formats":"md,ipynb","split_at_heading":true},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.7.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"**This notebook is an exercise in the [Computer Vision](https://www.kaggle.com/learn/computer-vision) course.  You can reference the tutorial at [this link](https://www.kaggle.com/ryanholbrook/the-sliding-window).**\n\n---\n","metadata":{}},{"cell_type":"markdown","source":"# Introduction #\n\nIn these exercises, you'll explore the operations a couple of popular convnet architectures use for feature extraction, learn about how convnets can capture large-scale visual features through stacking layers, and finally see how convolution can be used on one-dimensional data, in this case, a time series.\n\nRun the cell below to set everything up.","metadata":{}},{"cell_type":"code","source":"# Setup feedback system\nfrom learntools.core import binder\nbinder.bind(globals())\nfrom learntools.computer_vision.ex4 import *\n\nimport tensorflow as tf\nimport matplotlib.pyplot as plt\nimport learntools.computer_vision.visiontools as visiontools\n\n\nplt.rc('figure', autolayout=True)\nplt.rc('axes', labelweight='bold', labelsize='large',\n       titleweight='bold', titlesize=18, titlepad=10)\nplt.rc('image', cmap='magma')","metadata":{"lines_to_next_cell":2,"execution":{"iopub.status.busy":"2022-07-18T18:46:02.804425Z","iopub.execute_input":"2022-07-18T18:46:02.804911Z","iopub.status.idle":"2022-07-18T18:46:13.411238Z","shell.execute_reply.started":"2022-07-18T18:46:02.804817Z","shell.execute_reply":"2022-07-18T18:46:13.410002Z"},"trusted":true},"execution_count":1,"outputs":[]},{"cell_type":"markdown","source":"# (Optional) Experimenting with Feature Extraction #\n\nThis exercise is meant to give you an opportunity to explore the sliding window computations and how their parameters affect feature extraction. There aren't any right or wrong answers -- it's just a chance to experiment!\n\nWe've provided you with some images and kernels you can use. Run this cell to see them.","metadata":{}},{"cell_type":"code","source":"from learntools.computer_vision.visiontools import edge, blur, bottom_sobel, emboss, sharpen, circle\n\nimage_dir = '../input/computer-vision-resources/'\ncircle_64 = tf.expand_dims(circle([64, 64], val=1.0, r_shrink=4), axis=-1)\nkaggle_k = visiontools.read_image(image_dir + str('k.jpg'), channels=1)\ncar = visiontools.read_image(image_dir + str('car_illus.jpg'), channels=1)\ncar = tf.image.resize(car, size=[200, 200])\nimages = [(circle_64, \"circle_64\"), (kaggle_k, \"kaggle_k\"), (car, \"car\")]\n\nplt.figure(figsize=(14, 4))\nfor i, (img, title) in enumerate(images):\n    plt.subplot(1, len(images), i+1)\n    plt.imshow(tf.squeeze(img))\n    plt.axis('off')\n    plt.title(title)\nplt.show();\n\nkernels = [(edge, \"edge\"), (blur, \"blur\"), (bottom_sobel, \"bottom_sobel\"),\n           (emboss, \"emboss\"), (sharpen, \"sharpen\")]\nplt.figure(figsize=(14, 4))\nfor i, (krn, title) in enumerate(kernels):\n    plt.subplot(1, len(kernels), i+1)\n    visiontools.show_kernel(krn, digits=2, text_size=20)\n    plt.title(title)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-18T18:46:26.712883Z","iopub.execute_input":"2022-07-18T18:46:26.713627Z","iopub.status.idle":"2022-07-18T18:46:28.084615Z","shell.execute_reply.started":"2022-07-18T18:46:26.713584Z","shell.execute_reply":"2022-07-18T18:46:28.083392Z"},"trusted":true},"execution_count":2,"outputs":[]},{"cell_type":"markdown","source":"To choose one to experiment with, just enter it's name in the appropriate place below. Then, set the parameters for the window computation. Try out some different combinations and see what they do!","metadata":{}},{"cell_type":"code","source":"# YOUR CODE HERE: choose an image\nimage = circle_64\n\n# YOUR CODE HERE: choose a kernel\nkernel = bottom_sobel\n\nvisiontools.show_extraction(\n    image, kernel,\n\n    # YOUR CODE HERE: set parameters\n    conv_stride=1,\n    conv_padding='valid',\n    pool_size=2,\n    pool_stride=2,\n    pool_padding='same',\n    \n    subplot_shape=(1, 4),\n    figsize=(14, 6),\n)","metadata":{"lines_to_next_cell":0,"execution":{"iopub.status.busy":"2022-07-18T18:46:43.239632Z","iopub.execute_input":"2022-07-18T18:46:43.240049Z","iopub.status.idle":"2022-07-18T18:46:44.955233Z","shell.execute_reply.started":"2022-07-18T18:46:43.240015Z","shell.execute_reply":"2022-07-18T18:46:44.954424Z"},"trusted":true},"execution_count":3,"outputs":[]},{"cell_type":"markdown","source":"# The Receptive Field #\n\nTrace back all the connections from some neuron and eventually you reach the input image. All of the input pixels a neuron is connected to is that neuron's **receptive field**. The receptive field just tells you which parts of the input image a neuron receives information from.\n\nAs we've seen, if your first layer is a convolution with $3 \\times 3$ kernels, then each neuron in that layer gets input from a $3 \\times 3$ patch of pixels (except maybe at the border).\n\nWhat happens if you add another convolutional layer with $3 \\times 3$ kernels? Consider this next illustration:\n\n<figure>\n<img src=\"https://i.imgur.com/HmwQm2S.png\" alt=\"Illustration of the receptive field of two stacked convolutions.\" width=250>\n</figure>\n\nNow trace back the connections from the neuron at top and you can see that it's connected to a $5 \\times 5$ patch of pixels in the input (the bottom layer): each neuron in the $3 \\times 3$ patch in the middle layer is connected to a $3 \\times 3$ input patch, but they overlap in a $5 \\times 5$ patch. So that neuron at top has a $5 \\times 5$ receptive field.","metadata":{}},{"cell_type":"markdown","source":"# 1) Growing the Receptive Field #\n\nNow, if you added a *third* convolutional layer with a `(3, 3)` kernel, what receptive field would its neurons have? Run the cell below for an answer. (Or see a hint first!)","metadata":{}},{"cell_type":"code","source":"# View the solution (Run this code cell to receive credit!)\nq_1.check()","metadata":{"execution":{"iopub.status.busy":"2022-07-18T18:46:55.897026Z","iopub.execute_input":"2022-07-18T18:46:55.897400Z","iopub.status.idle":"2022-07-18T18:46:55.906913Z","shell.execute_reply.started":"2022-07-18T18:46:55.897368Z","shell.execute_reply":"2022-07-18T18:46:55.905825Z"},"trusted":true},"execution_count":4,"outputs":[]},{"cell_type":"code","source":"# Lines below will give you a hint\n#q_1.hint()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So why stack layers like this? Three `(3, 3)` kernels have 27 parameters, while one `(7, 7)` kernel has 49, though they both create the same receptive field. This stacking-layers trick is one of the ways convnets are able to create large receptive fields without increasing the number of parameters too much. You'll see how to do this yourself in the next lesson!","metadata":{}},{"cell_type":"markdown","source":"# (Optional) One-Dimensional Convolution #\n\nConvolutional networks turn out to be useful not only (two-dimensional) images, but also on things like time-series (one-dimensional) and video (three-dimensional).\n\nWe've seen how convolutional networks can learn to extract features from (two-dimensional) images. It turns out that convnets can also learn to extract features from things like time-series (one-dimensional) and video (three-dimensional).\n\nIn this (optional) exercise, we'll see what convolution looks like on a time-series.\n\nThe time series we'll use is from [Google Trends](https://trends.google.com/trends/). It measures the popularity of the search term \"machine learning\" for weeks from January 25, 2015 to January 15, 2020.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\n\n# Load the time series as a Pandas dataframe\nmachinelearning = pd.read_csv(\n    '../input/computer-vision-resources/machinelearning.csv',\n    parse_dates=['Week'],\n    index_col='Week',\n)\n\nmachinelearning.plot();","metadata":{"execution":{"iopub.status.busy":"2022-07-18T18:47:00.691127Z","iopub.execute_input":"2022-07-18T18:47:00.691531Z","iopub.status.idle":"2022-07-18T18:47:01.052454Z","shell.execute_reply.started":"2022-07-18T18:47:00.691497Z","shell.execute_reply":"2022-07-18T18:47:01.051579Z"},"trusted":true},"execution_count":5,"outputs":[]},{"cell_type":"markdown","source":"What about the kernels? Images are two-dimensional and so our kernels were 2D arrays. A time-series is one-dimensional, so what should the kernel be? A 1D array! Here are some kernels sometimes used on time-series data:","metadata":{}},{"cell_type":"code","source":"detrend = tf.constant([-1, 1], dtype=tf.float32)\n\naverage = tf.constant([0.2, 0.2, 0.2, 0.2, 0.2], dtype=tf.float32)\n\nspencer = tf.constant([-3, -6, -5, 3, 21, 46, 67, 74, 67, 46, 32, 3, -5, -6, -3], dtype=tf.float32) / 320","metadata":{"execution":{"iopub.status.busy":"2022-07-18T18:47:04.921142Z","iopub.execute_input":"2022-07-18T18:47:04.921569Z","iopub.status.idle":"2022-07-18T18:47:04.936422Z","shell.execute_reply.started":"2022-07-18T18:47:04.921531Z","shell.execute_reply":"2022-07-18T18:47:04.935244Z"},"trusted":true},"execution_count":6,"outputs":[]},{"cell_type":"markdown","source":"Convolution on a sequence works just like convolution on an image. The difference is just that a sliding window on a sequence only has one direction to travel -- left to right -- instead of the two directions on an image. And just like before, the features picked out depend on the pattern on numbers in the kernel.\n\nCan you guess what kind of features these kernels extract? Uncomment one of the kernels below and run the cell to see!","metadata":{}},{"cell_type":"code","source":"# UNCOMMENT ONE\nkernel = detrend\n# kernel = average\n# kernel = spencer\n\n# Reformat for TensorFlow\nts_data = machinelearning.to_numpy()\nts_data = tf.expand_dims(ts_data, axis=0)\nts_data = tf.cast(ts_data, dtype=tf.float32)\nkern = tf.reshape(kernel, shape=(*kernel.shape, 1, 1))\n\nts_filter = tf.nn.conv1d(\n    input=ts_data,\n    filters=kern,\n    stride=1,\n    padding='VALID',\n)\n\n# Format as Pandas Series\nmachinelearning_filtered = pd.Series(tf.squeeze(ts_filter).numpy())\n\nmachinelearning_filtered.plot();","metadata":{"lines_to_next_cell":0,"execution":{"iopub.status.busy":"2022-07-18T18:47:07.977117Z","iopub.execute_input":"2022-07-18T18:47:07.977541Z","iopub.status.idle":"2022-07-18T18:47:08.229097Z","shell.execute_reply.started":"2022-07-18T18:47:07.977503Z","shell.execute_reply":"2022-07-18T18:47:08.227918Z"},"trusted":true},"execution_count":7,"outputs":[]},{"cell_type":"markdown","source":"In fact, the `detrend` kernel filters for *changes* in the series, while `average` and `spencer` are both \"smoothers\" that filter for low-frequency components in the series.\n\nIf you were interested in predicting the future popularity of search terms, you might train a convnet on time-series like this one. It would try to learn what features in those series are most informative for the prediction.\n\nThough convnets are not often the best choice on their own for these kinds of problems, they are often incorporated into other models for their feature extraction capabilities.","metadata":{}},{"cell_type":"markdown","source":"# Conclusion #\n\nThis lesson ends our discussion of feature extraction. Hopefully, having completed these lessons, you've gained some intuition about how the process works and why the usual choices for its implementation are often the best ones.","metadata":{}},{"cell_type":"markdown","source":"# Keep Going #\n\nIn the next lesson, [**Lesson 5**](https://www.kaggle.com/ryanholbrook/custom-convnets), you'll learn how to compose the `Conv2D` and `MaxPool2D` layers to build your own convolutional networks from scratch.","metadata":{}},{"cell_type":"markdown","source":"---\n\n\n\n\n*Have questions or comments? Visit the [course discussion forum](https://www.kaggle.com/learn/computer-vision/discussion) to chat with other learners.*","metadata":{}}]}