exercise-deep-neural-networks.ipynb

{"metadata":{"jupytext":{"cell_metadata_filter":"-all","formats":"ipynb"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":1480608,"sourceType":"datasetVersion","datasetId":829369}],"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"**This notebook is an exercise in the [Intro to Deep Learning](https://www.kaggle.com/learn/intro-to-deep-learning) course.  You can reference the tutorial at [this link](https://www.kaggle.com/ryanholbrook/deep-neural-networks).**\n\n---\n","metadata":{}},{"cell_type":"markdown","source":"# Introduction #\n\nIn the tutorial, we saw how to build deep neural networks by stacking layers inside a `Sequential` model. By adding an *activation function* after the hidden layers, we gave the network the ability to learn more complex (non-linear) relationships in the data.\n\nIn these exercises, you'll build a neural network with several hidden layers and then explore some activation functions beyond ReLU. Run this next cell to set everything up!","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\n\n# Setup plotting\nimport matplotlib.pyplot as plt\n\nplt.style.use('seaborn-whitegrid')\n# Set Matplotlib defaults\nplt.rc('figure', autolayout=True)\nplt.rc('axes', labelweight='bold', labelsize='large',\n       titleweight='bold', titlesize=18, titlepad=10)\n\n# Setup feedback system\nfrom learntools.core import binder\nbinder.bind(globals())\nfrom learntools.deep_learning_intro.ex2 import *","metadata":{"execution":{"iopub.status.busy":"2023-11-23T22:22:10.343927Z","iopub.execute_input":"2023-11-23T22:22:10.344321Z","iopub.status.idle":"2023-11-23T22:22:23.581773Z","shell.execute_reply.started":"2023-11-23T22:22:10.344288Z","shell.execute_reply":"2023-11-23T22:22:23.580603Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the *Concrete* dataset, your task is to predict the compressive strength of concrete manufactured according to various recipes.\n\nRun the next code cell without changes to load the dataset.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\n\nconcrete = pd.read_csv('../input/dl-course-data/concrete.csv')\nconcrete.head()","metadata":{"execution":{"iopub.status.busy":"2023-11-23T22:22:23.583901Z","iopub.execute_input":"2023-11-23T22:22:23.585135Z","iopub.status.idle":"2023-11-23T22:22:23.635777Z","shell.execute_reply.started":"2023-11-23T22:22:23.585090Z","shell.execute_reply":"2023-11-23T22:22:23.634751Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1) Input Shape #\n\nThe target for this task is the column `'CompressiveStrength'`. The remaining columns are the features we'll use as inputs.\n\nWhat would be the input shape for this dataset?","metadata":{}},{"cell_type":"code","source":"# YOUR CODE HERE\ninput_shape =[8]\n\n# Check your answer\nq_1.check()","metadata":{"lines_to_next_cell":2,"execution":{"iopub.status.busy":"2023-11-23T11:58:19.795192Z","iopub.execute_input":"2023-11-23T11:58:19.796428Z","iopub.status.idle":"2023-11-23T11:58:19.805545Z","shell.execute_reply.started":"2023-11-23T11:58:19.796377Z","shell.execute_reply":"2023-11-23T11:58:19.804297Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Lines below will give you a hint or solution code\n# q_1.hint()\n# q_1.solution()","metadata":{"execution":{"iopub.status.busy":"2023-11-23T11:57:56.722393Z","iopub.execute_input":"2023-11-23T11:57:56.722821Z","iopub.status.idle":"2023-11-23T11:57:56.738159Z","shell.execute_reply.started":"2023-11-23T11:57:56.722787Z","shell.execute_reply":"2023-11-23T11:57:56.736876Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2) Define a Model with Hidden Layers #\n\nNow create a model with three hidden layers, each having 512 units and the ReLU activation.  Be sure to include an output layer of one unit and no activation, and also `input_shape` as an argument to the first layer.","metadata":{}},{"cell_type":"code","source":"from tensorflow import keras\nfrom tensorflow.keras import layers\n\n# YOUR CODE HERE\nmodel = keras.Sequential([\n    layers.Dense(units=512,activation='relu',input_shape=[8]),\n    layers.Dense(units=512,activation='relu'),\n    layers.Dense(units=512,activation='relu'),\n    # output layer\n    layers.Dense(units=1)\n])\n\n# Check your answer\nq_2.check()","metadata":{"lines_to_next_cell":0,"execution":{"iopub.status.busy":"2023-11-23T22:34:27.338546Z","iopub.execute_input":"2023-11-23T22:34:27.339617Z","iopub.status.idle":"2023-11-23T22:34:27.410756Z","shell.execute_reply.started":"2023-11-23T22:34:27.339580Z","shell.execute_reply":"2023-11-23T22:34:27.409563Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Lines below will give you a hint or solution code\n#q_2.hint()\n#q_2.solution()","metadata":{"execution":{"iopub.status.busy":"2023-11-23T22:34:08.071802Z","iopub.execute_input":"2023-11-23T22:34:08.072162Z","iopub.status.idle":"2023-11-23T22:34:08.075979Z","shell.execute_reply.started":"2023-11-23T22:34:08.072136Z","shell.execute_reply":"2023-11-23T22:34:08.075176Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3) Activation Layers #\n\nLet's explore activations functions some.\n\nThe usual way of attaching an activation function to a `Dense` layer is to include it as part of the definition with the `activation` argument. Sometimes though you'll want to put some other layer between the `Dense` layer and its activation function. (We'll see an example of this in Lesson 5 with *batch normalization*.) In this case, we can define the activation in its own `Activation` layer, like so:\n\n```\nlayers.Dense(units=8),\nlayers.Activation('relu')\n```\n\nThis is completely equivalent to the ordinary way: `layers.Dense(units=8, activation='relu')`.\n\nRewrite the following model so that each activation is in its own `Activation` layer.","metadata":{}},{"cell_type":"code","source":"### YOUR CODE HERE: rewrite this to use activation layers\nmodel = keras.Sequential([\n    layers.Dense(32, input_shape=[8]),\n    layers.Activation('relu'),\n    layers.Dense(32),\n    layers.Activation('relu'),\n    layers.Dense(1),\n])\n\n# Check your answer\nq_3.check()","metadata":{"lines_to_next_cell":0,"execution":{"iopub.status.busy":"2023-11-23T22:40:00.555662Z","iopub.execute_input":"2023-11-23T22:40:00.556067Z","iopub.status.idle":"2023-11-23T22:40:00.610797Z","shell.execute_reply.started":"2023-11-23T22:40:00.556038Z","shell.execute_reply":"2023-11-23T22:40:00.609780Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Lines below will give you a hint or solution code\n#q_3.hint()\n#q_3.solution()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Optional: Alternatives to ReLU #\n\nThere is a whole family of variants of the `'relu'` activation -- `'elu'`, `'selu'`, and `'swish'`, among others -- all of which you can use in Keras. Sometimes one activation will perform better than another on a given task, so you could consider experimenting with activations as you develop a model. The ReLU activation tends to do well on most problems, so it's a good one to start with.\n\nLet's look at the graphs of some of these. Change the activation from `'relu'` to one of the others named above. Then run the cell to see the graph. (Check out the [documentation](https://www.tensorflow.org/api_docs/python/tf/keras/activations) for more ideas.)","metadata":{}},{"cell_type":"code","source":"# YOUR CODE HERE: Change 'relu' to 'elu', 'selu', 'swish'... or something else\nactivation_layer = layers.Activation('gelu')\n\nx = tf.linspace(-3.0, 3.0, 100)\ny = activation_layer(x) # once created, a layer is callable just like a function\n\nplt.figure(dpi=100)\nplt.plot(x, y)\nplt.xlim(-3, 3)\nplt.xlabel(\"Input\")\nplt.ylabel(\"Output\")\nplt.show()","metadata":{"lines_to_next_cell":0,"execution":{"iopub.status.busy":"2023-11-23T22:43:24.727318Z","iopub.execute_input":"2023-11-23T22:43:24.727717Z","iopub.status.idle":"2023-11-23T22:43:25.100264Z","shell.execute_reply.started":"2023-11-23T22:43:24.727675Z","shell.execute_reply":"2023-11-23T22:43:25.099190Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Keep Going #\n\nNow move on to Lesson 3 and [**learn how to train neural networks**](https://www.kaggle.com/ryanholbrook/stochastic-gradient-descent) with stochastic gradient descent.","metadata":{}},{"cell_type":"markdown","source":"---\n\n\n\n\n*Have questions or comments? Visit the [course discussion forum](https://www.kaggle.com/learn/intro-to-deep-learning/discussion) to chat with other learners.*","metadata":{}}]}