02. exercise-convolution-and-relu.ipynb

{"metadata":{"jupytext":{"formats":"md,ipynb"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.7.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"**This notebook is an exercise in the [Computer Vision](https://www.kaggle.com/learn/computer-vision) course.  You can reference the tutorial at [this link](https://www.kaggle.com/ryanholbrook/convolution-and-relu).**\n\n---\n","metadata":{}},{"cell_type":"markdown","source":"# Introduction #\n\nIn this exercise, you'll work on building some intuition around feature extraction. First, we'll walk through the example we did in the tutorial again, but this time, with a kernel you choose yourself. We've mostly been working with images in this course, but what's behind all of the operations we're learning about is mathematics. So, we'll also take a look at how these feature maps can be represented instead as arrays of numbers and what effect convolution with a kernel will have on them.\n\nRun the cell below to get started!","metadata":{}},{"cell_type":"code","source":"# Setup feedback system\nfrom learntools.core import binder\nbinder.bind(globals())\nfrom learntools.computer_vision.ex2 import *\n\nimport numpy as np\nimport tensorflow as tf\nimport matplotlib.pyplot as plt\n\nplt.rc('figure', autolayout=True)\nplt.rc('axes', labelweight='bold', labelsize='large',\n       titleweight='bold', titlesize=18, titlepad=10)\nplt.rc('image', cmap='magma')\n\ntf.config.run_functions_eagerly(True)","metadata":{"execution":{"iopub.status.busy":"2022-07-18T00:28:27.267756Z","iopub.execute_input":"2022-07-18T00:28:27.268605Z","iopub.status.idle":"2022-07-18T00:28:36.714933Z","shell.execute_reply.started":"2022-07-18T00:28:27.268481Z","shell.execute_reply":"2022-07-18T00:28:36.713962Z"},"trusted":true},"execution_count":1,"outputs":[]},{"cell_type":"markdown","source":"# Apply Transformations #\n\nThe next few exercises walk through feature extraction just like the example in the tutorial. Run the following cell to load an image we'll use for the next few exercises.","metadata":{}},{"cell_type":"code","source":"image_path = '../input/computer-vision-resources/car_illus.jpg'\nimage = tf.io.read_file(image_path)\nimage = tf.io.decode_jpeg(image, channels=1)\nimage = tf.image.resize(image, size=[400, 400])\n\nimg = tf.squeeze(image).numpy()\nplt.figure(figsize=(6, 6))\nplt.imshow(img, cmap='gray')\nplt.axis('off')\nplt.show();","metadata":{"execution":{"iopub.status.busy":"2022-07-18T00:28:40.942104Z","iopub.execute_input":"2022-07-18T00:28:40.942788Z","iopub.status.idle":"2022-07-18T00:28:41.431908Z","shell.execute_reply.started":"2022-07-18T00:28:40.942749Z","shell.execute_reply":"2022-07-18T00:28:41.430904Z"},"trusted":true},"execution_count":2,"outputs":[]},{"cell_type":"markdown","source":"You can run this cell to see some standard kernels used in image processing.","metadata":{}},{"cell_type":"code","source":"import learntools.computer_vision.visiontools as visiontools\nfrom learntools.computer_vision.visiontools import edge, bottom_sobel, emboss, sharpen\n\nkernels = [edge, bottom_sobel, emboss, sharpen]\nnames = [\"Edge Detect\", \"Bottom Sobel\", \"Emboss\", \"Sharpen\"]\n\nplt.figure(figsize=(12, 12))\nfor i, (kernel, name) in enumerate(zip(kernels, names)):\n    plt.subplot(1, 4, i+1)\n    visiontools.show_kernel(kernel)\n    plt.title(name)\nplt.tight_layout()","metadata":{"execution":{"iopub.status.busy":"2022-07-18T00:28:54.535257Z","iopub.execute_input":"2022-07-18T00:28:54.535673Z","iopub.status.idle":"2022-07-18T00:28:55.583713Z","shell.execute_reply.started":"2022-07-18T00:28:54.535638Z","shell.execute_reply":"2022-07-18T00:28:55.582446Z"},"trusted":true},"execution_count":3,"outputs":[]},{"cell_type":"markdown","source":"# 1) Define Kernel #\n\nUse the next code cell to define a kernel. You have your choice of what kind of kernel to apply. One thing to keep in mind is that the *sum* of the numbers in the kernel determines how bright the final image is. Generally, you should try to keep the sum of the numbers between 0 and 1 (though that's not required for a correct answer).\n\nIn general, a kernel can have any number of rows and columns. For this exercise, let's use a $3 \\times 3$ kernel, which often gives the best results. Define a kernel with `tf.constant`.","metadata":{}},{"cell_type":"code","source":"# YOUR CODE HERE: Define a kernel with 3 rows and 3 columns.\nkernel = tf.constant([\n    [-2, -1, 0],\n    [-1, 1, 1],\n    [0, 1, 2],\n])\n# Uncomment to view kernel\n# visiontools.show_kernel(kernel)\n\n# Check your answer\nq_1.check()","metadata":{"lines_to_next_cell":0,"execution":{"iopub.status.busy":"2022-07-18T00:30:21.970767Z","iopub.execute_input":"2022-07-18T00:30:21.971596Z","iopub.status.idle":"2022-07-18T00:30:21.982028Z","shell.execute_reply.started":"2022-07-18T00:30:21.971555Z","shell.execute_reply":"2022-07-18T00:30:21.981154Z"},"trusted":true},"execution_count":5,"outputs":[]},{"cell_type":"code","source":"# Lines below will give you a hint or solution code\n#q_1.hint()\n#q_1.solution()","metadata":{"execution":{"iopub.status.busy":"2022-07-18T00:30:24.590585Z","iopub.execute_input":"2022-07-18T00:30:24.591855Z","iopub.status.idle":"2022-07-18T00:30:24.596835Z","shell.execute_reply.started":"2022-07-18T00:30:24.591792Z","shell.execute_reply":"2022-07-18T00:30:24.595569Z"},"trusted":true},"execution_count":6,"outputs":[]},{"cell_type":"markdown","source":"Now we'll do the first step of feature extraction, the filtering step. First run this cell to do some reformatting for TensorFlow.","metadata":{}},{"cell_type":"code","source":"# Reformat for batch compatibility.\nimage = tf.image.convert_image_dtype(image, dtype=tf.float32)\nimage = tf.expand_dims(image, axis=0)\nkernel = tf.reshape(kernel, [*kernel.shape, 1, 1])\nkernel = tf.cast(kernel, dtype=tf.float32)","metadata":{"execution":{"iopub.status.busy":"2022-07-18T00:30:27.491604Z","iopub.execute_input":"2022-07-18T00:30:27.491991Z","iopub.status.idle":"2022-07-18T00:30:27.504862Z","shell.execute_reply.started":"2022-07-18T00:30:27.491961Z","shell.execute_reply":"2022-07-18T00:30:27.504061Z"},"trusted":true},"execution_count":7,"outputs":[]},{"cell_type":"markdown","source":"# 2) Apply Convolution #\n\nNow we'll apply the kernel to the image by a convolution. The *layer* in Keras that does this is `layers.Conv2D`. What is the *backend function* in TensorFlow that performs the same operation?","metadata":{}},{"cell_type":"code","source":"# YOUR CODE HERE: Give the TensorFlow convolution function (without arguments)\nconv_fn = tf.nn.conv2d\n\n# Check your answer\nq_2.check()","metadata":{"lines_to_next_cell":0,"execution":{"iopub.status.busy":"2022-07-18T00:31:11.357219Z","iopub.execute_input":"2022-07-18T00:31:11.357591Z","iopub.status.idle":"2022-07-18T00:31:11.366628Z","shell.execute_reply.started":"2022-07-18T00:31:11.357561Z","shell.execute_reply":"2022-07-18T00:31:11.365537Z"},"trusted":true},"execution_count":11,"outputs":[]},{"cell_type":"code","source":"# Lines below will give you a hint or solution code\n#q_2.hint()\n#q_2.solution()","metadata":{"execution":{"iopub.status.busy":"2022-07-18T00:31:01.322800Z","iopub.execute_input":"2022-07-18T00:31:01.323519Z","iopub.status.idle":"2022-07-18T00:31:01.327597Z","shell.execute_reply.started":"2022-07-18T00:31:01.323474Z","shell.execute_reply":"2022-07-18T00:31:01.326472Z"},"trusted":true},"execution_count":9,"outputs":[]},{"cell_type":"markdown","source":"Once you've got the correct answer, run this next cell to execute the convolution and see the result!","metadata":{}},{"cell_type":"code","source":"image_filter = conv_fn(\n    input=image,\n    filters=kernel,\n    strides=1, # or (1, 1)\n    padding='SAME',\n)\n\nplt.imshow(\n    # Reformat for plotting\n    tf.squeeze(image_filter)\n)\nplt.axis('off')\nplt.show();","metadata":{"execution":{"iopub.status.busy":"2022-07-18T00:31:13.459845Z","iopub.execute_input":"2022-07-18T00:31:13.460252Z","iopub.status.idle":"2022-07-18T00:31:13.716069Z","shell.execute_reply.started":"2022-07-18T00:31:13.460219Z","shell.execute_reply":"2022-07-18T00:31:13.715247Z"},"trusted":true},"execution_count":12,"outputs":[]},{"cell_type":"markdown","source":"Can you see how the kernel you chose relates to the feature map it produced?\n\n# 3) Apply ReLU #\n\nNow detect the feature with the ReLU function. In Keras, you'll usually use this as the activation function in a `Conv2D` layer. What is the *backend function* in TensorFlow that does the same thing?","metadata":{}},{"cell_type":"code","source":"# YOUR CODE HERE: Give the TensorFlow ReLU function (without arguments)\nrelu_fn = tf.nn.relu\n\n# Check your answer\nq_3.check()","metadata":{"lines_to_next_cell":0,"execution":{"iopub.status.busy":"2022-07-18T00:31:34.189391Z","iopub.execute_input":"2022-07-18T00:31:34.190278Z","iopub.status.idle":"2022-07-18T00:31:34.198399Z","shell.execute_reply.started":"2022-07-18T00:31:34.190239Z","shell.execute_reply":"2022-07-18T00:31:34.197574Z"},"trusted":true},"execution_count":15,"outputs":[]},{"cell_type":"code","source":"# Lines below will give you a hint or solution code\n#q_3.hint()\n#q_3.solution()","metadata":{"execution":{"iopub.status.busy":"2022-07-18T00:31:32.808377Z","iopub.execute_input":"2022-07-18T00:31:32.808791Z","iopub.status.idle":"2022-07-18T00:31:32.813572Z","shell.execute_reply.started":"2022-07-18T00:31:32.808758Z","shell.execute_reply":"2022-07-18T00:31:32.812639Z"},"trusted":true},"execution_count":14,"outputs":[]},{"cell_type":"markdown","source":"Once you've got the solution, run this cell to detect the feature with ReLU and see the result!\n\nThe image you see below is the feature map produced by the kernel you chose. If you like, experiment with some of the other suggested kernels above, or, try to invent one that will extract a certain kind of feature.\n","metadata":{}},{"cell_type":"code","source":"image_detect = relu_fn(image_filter)\n\nplt.imshow(\n    # Reformat for plotting\n    tf.squeeze(image_detect)\n)\nplt.axis('off')\nplt.show();","metadata":{"execution":{"iopub.status.busy":"2022-07-18T00:31:37.022354Z","iopub.execute_input":"2022-07-18T00:31:37.022922Z","iopub.status.idle":"2022-07-18T00:31:37.231758Z","shell.execute_reply.started":"2022-07-18T00:31:37.022870Z","shell.execute_reply":"2022-07-18T00:31:37.230913Z"},"trusted":true},"execution_count":16,"outputs":[]},{"cell_type":"markdown","source":"In the tutorial, our discussion of kernels and feature maps was mainly visual. We saw the effect of `Conv2D` and `ReLU` by observing how they transformed some example images.\n\nBut the operations in a convolutional network (like in all neural networks) are usually defined through mathematical functions, through a computation on numbers. In the next exercise, we'll take a moment to explore this point of view.\n\nLet's start by defining a simple array to act as an image, and another array to act as the kernel. Run the following cell to see these arrays.","metadata":{}},{"cell_type":"code","source":"# Sympy is a python library for symbolic mathematics. It has a nice\n# pretty printer for matrices, which is all we'll use it for.\nimport sympy\nsympy.init_printing()\nfrom IPython.display import display\n\nimage = np.array([\n    [0, 1, 0, 0, 0, 0],\n    [0, 1, 0, 0, 0, 0],\n    [0, 1, 0, 0, 0, 0],\n    [0, 1, 0, 0, 0, 0],\n    [0, 1, 0, 1, 1, 1],\n    [0, 1, 0, 0, 0, 0],\n])\n\nkernel = np.array([\n    [1, -1],\n    [1, -1],\n])\n\ndisplay(sympy.Matrix(image))\ndisplay(sympy.Matrix(kernel))\n# Reformat for Tensorflow\nimage = tf.cast(image, dtype=tf.float32)\nimage = tf.reshape(image, [1, *image.shape, 1])\nkernel = tf.reshape(kernel, [*kernel.shape, 1, 1])\nkernel = tf.cast(kernel, dtype=tf.float32)","metadata":{"execution":{"iopub.status.busy":"2022-07-18T00:31:45.390787Z","iopub.execute_input":"2022-07-18T00:31:45.391158Z","iopub.status.idle":"2022-07-18T00:31:46.162355Z","shell.execute_reply.started":"2022-07-18T00:31:45.391128Z","shell.execute_reply":"2022-07-18T00:31:46.160942Z"},"trusted":true},"execution_count":17,"outputs":[]},{"cell_type":"markdown","source":"# 4) Observe Convolution on a Numerical Matrix #\n\n\nWhat do you see? The image is simply a long vertical line on the left and a short horizontal line on the lower right. What about the kernel? What effect do you think it will have on this image? After you've thought about it, run the next cell for the answer.","metadata":{}},{"cell_type":"code","source":"# View the solution (Run this code cell to receive credit!)\nq_4.check()","metadata":{"lines_to_next_cell":2,"execution":{"iopub.status.busy":"2022-07-18T00:31:51.923473Z","iopub.execute_input":"2022-07-18T00:31:51.924663Z","iopub.status.idle":"2022-07-18T00:31:51.935101Z","shell.execute_reply.started":"2022-07-18T00:31:51.924622Z","shell.execute_reply":"2022-07-18T00:31:51.934045Z"},"trusted":true},"execution_count":18,"outputs":[]},{"cell_type":"markdown","source":"Now let's try it out. Run the next cell to apply convolution and ReLU to the image and display the result.","metadata":{}},{"cell_type":"code","source":"image_filter = tf.nn.conv2d(\n    input=image,\n    filters=kernel,\n    strides=1,\n    padding='VALID',\n)\nimage_detect = tf.nn.relu(image_filter)\n\n# The first matrix is the image after convolution, and the second is\n# the image after ReLU.\ndisplay(sympy.Matrix(tf.squeeze(image_filter).numpy()))\ndisplay(sympy.Matrix(tf.squeeze(image_detect).numpy()))","metadata":{"execution":{"iopub.status.busy":"2022-07-18T00:31:54.735751Z","iopub.execute_input":"2022-07-18T00:31:54.736164Z","iopub.status.idle":"2022-07-18T00:31:54.859164Z","shell.execute_reply.started":"2022-07-18T00:31:54.736121Z","shell.execute_reply":"2022-07-18T00:31:54.857987Z"},"trusted":true},"execution_count":19,"outputs":[]},{"cell_type":"markdown","source":"Is the result what you expected?\n\n# Conclusion #\n\nIn this lesson, you learned about the first two operations a convolutional classifier uses for feature extraction: **filtering** an image with a **convolution** and **detecting** the feature with the **rectified linear unit**. \n\n# Keep Going #\n\nMove on to [**Lesson 3**](https://www.kaggle.com/ryanholbrook/maximum-pooling) to learn the final operation: **condensing** the feature map with **maximum pooling**!","metadata":{}},{"cell_type":"markdown","source":"---\n\n\n\n\n*Have questions or comments? Visit the [course discussion forum](https://www.kaggle.com/learn/computer-vision/discussion) to chat with other learners.*","metadata":{}}]}