16 Final_Project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"text-align:center\">\n",
    "    <a href=\"https://skills.network/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMSkillsNetworkBD0231ENCoursera2789-2023-01-01\">\n",
    "    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/assets/logos/SN_web_lightmode.png\" width=\"200\" alt=\"Skills Network Logo\">\n",
    "    </a>\n",
    "</p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Final Project - Build an ML Pipeline for Airfoil noise prediction\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Estimated time needed: **90** minutes\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scenario\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You are a data engineer at an aeronautics consulting company. Your company prides itself in being able to efficiently design airfoils for use in planes and sports cars. Data scientists in your office need to work with different algorithms and data in different formats. While they are good at Machine Learning, they count on you to be able to do ETL jobs and build ML pipelines. In this project you will use the modified version of the NASA Airfoil Self Noise dataset. You will clean this dataset, by dropping the duplicate rows, and removing the rows with null values. You will create an ML pipe line to create a model that will predict the SoundLevel based on all the other columns. You will evaluate the model and towards the end you will persist the model.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objectives\n",
    "\n",
    "In this 4 part assignment you will:\n",
    "\n",
    "- Part 1 Perform ETL activity\n",
    "  - Load a csv dataset\n",
    "  - Remove duplicates if any\n",
    "  - Drop rows with null values if any\n",
    "  - Make transformations\n",
    "  - Store the cleaned data in parquet format\n",
    "- Part 2 Create a  Machine Learning Pipeline\n",
    "  - Create a machine learning pipeline for prediction\n",
    "- Part 3 Evaluate the Model\n",
    "  - Evaluate the model using relevant metrics\n",
    "- Part 4 Persist the Model \n",
    "  - Save the model for future production use\n",
    "  - Load and verify the stored model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Datasets\n",
    "\n",
    "In this lab you will be using dataset(s):\n",
    "\n",
    " - The original dataset can be found here NASA airfoil self noise dataset. https://archive.ics.uci.edu/dataset/291/airfoil+self+noise\n",
    " \n",
    " - This dataset is licensed under a Creative Commons Attribution 4.0 International (CC BY 4.0) license.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Diagram of an airfoil. - For informational purpose\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Airfoil with flow](https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMSkillsNetwork-BD0231EN-Coursera/images/Airfoil_with_flow.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Diagram showing the Angle of attack. - For informational purpose\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Airfoil angle of attack](https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMSkillsNetwork-BD0231EN-Coursera/images/Airfoil_angle_of_attack.jpg)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Before you Start\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before you start attempting this project it is highly recommended that you finish the practice project.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this lab, we will be using the following libraries:\n",
    "\n",
    "*   [`PySpark`](https://spark.apache.org/docs/latest/api/python/index.html?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMSkillsNetworkBD0231ENCoursera2789-2023-01-01) for connecting to the Spark Cluster\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installing Required Libraries\n",
    "\n",
    "Spark Cluster is pre-installed in the Skills Network Labs environment. However, you need libraries like pyspark and findspark to\n",
    " connect to this cluster.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following required libraries are __not__ pre-installed in the Skills Network Labs environment. __You will need to run the following cell__ to install them:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install pyspark==3.1.2 -q\n",
    "!pip install findspark -q"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importing Required Libraries\n",
    "\n",
    "_We recommend you import all required libraries in one place (here):_\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# You can also use this section to suppress warnings generated by your code:\n",
    "def warn(*args, **kwargs):\n",
    "    pass\n",
    "import warnings\n",
    "warnings.warn = warn\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "# FindSpark simplifies the process of using Apache Spark with Python\n",
    "\n",
    "import findspark\n",
    "findspark.init()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1 - Perform ETL activity\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 1 - Import required libraries\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#your code goes here\n",
    "\n",
    "from pyspark.sql import SparkSession\n",
    "from pyspark.ml import Pipeline\n",
    "from pyspark.ml.pipeline import PipelineModel\n",
    "from pyspark.ml.feature import VectorAssembler\n",
    "from pyspark.ml.regression import LinearRegression\n",
    "from pyspark.ml.feature import StringIndexer\n",
    "from pyspark.ml.feature import StandardScaler"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 2 - Create a spark session\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "24/01/09 04:50:17 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n",
      "Setting default log level to \"WARN\".\n",
      "To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n"
     ]
    }
   ],
   "source": [
    "#Create a SparkSession\n",
    "\n",
    "spark = SparkSession.builder.appName(\"ML Pipeline for Airfoil noise prediction\").getOrCreate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 3 - Load the csv file into a dataframe\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download the data file.\n",
    "\n",
    "NOTE : Please ensure you use the dataset below and not the original dataset mentioned above.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2024-01-09 04:50:47--  https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMSkillsNetwork-BD0231EN-Coursera/datasets/NASA_airfoil_noise_raw.csv\n",
      "Resolving cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud (cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud)... 169.63.118.104, 169.63.118.104\n",
      "Connecting to cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud (cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud)|169.63.118.104|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 60682 (59K) [text/csv]\n",
      "Saving to: ‘NASA_airfoil_noise_raw.csv’\n",
      "\n",
      "NASA_airfoil_noise_ 100%[===================>]  59.26K  --.-KB/s    in 0.002s  \n",
      "\n",
      "2024-01-09 04:50:47 (33.2 MB/s) - ‘NASA_airfoil_noise_raw.csv’ saved [60682/60682]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMSkillsNetwork-BD0231EN-Coursera/datasets/NASA_airfoil_noise_raw.csv\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the dataset into the spark dataframe\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Load the dataset that you have downloaded in the previous task\n",
    "\n",
    "df = spark.read.csv(\"NASA_airfoil_noise_raw.csv\", header=True, inferSchema=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 4 - Print top 5 rows of the dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(Frequency=800, AngleOfAttack=0.0, ChordLength=0.3048, FreeStreamVelocity=71.3, SuctionSideDisplacement=0.00266337, SoundLevel=126.201),\n",
       " Row(Frequency=1000, AngleOfAttack=0.0, ChordLength=0.3048, FreeStreamVelocity=71.3, SuctionSideDisplacement=0.00266337, SoundLevel=125.201),\n",
       " Row(Frequency=1250, AngleOfAttack=0.0, ChordLength=0.3048, FreeStreamVelocity=71.3, SuctionSideDisplacement=0.00266337, SoundLevel=125.951),\n",
       " Row(Frequency=1600, AngleOfAttack=0.0, ChordLength=0.3048, FreeStreamVelocity=71.3, SuctionSideDisplacement=0.00266337, SoundLevel=127.591),\n",
       " Row(Frequency=2000, AngleOfAttack=0.0, ChordLength=0.3048, FreeStreamVelocity=71.3, SuctionSideDisplacement=0.00266337, SoundLevel=127.461)]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#your code goes here\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 6 - Print the total number of rows in the dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1522\n"
     ]
    }
   ],
   "source": [
    "#your code goes here\n",
    "rowcount1 = df.count()\n",
    "print(rowcount1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 7 - Drop all the duplicate rows from the dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df = df.dropDuplicates()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 8 - Print the total number of rows in the dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Stage 7:===================================================>   (189 + 9) / 200]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1503\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "#your code goes here\n",
    "\n",
    "rowcount2 = df.count()\n",
    "print(rowcount2)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 9 - Drop all the rows that contain null values from the dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df=df.dropna()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 10 - Print the total number of rows in the dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Stage 10:===================================================>  (191 + 8) / 200]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1499\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "#your code goes here\n",
    "\n",
    "rowcount3 = df.count()\n",
    "print(rowcount3)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 11 - Rename the column \"SoundLevel\" to \"SoundLevelDecibels\"Drop\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# your code goes here\n",
    "\n",
    "df = df.withColumnRenamed(\"SoundLevel\",\"SoundLevelDecibels\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 12 - Save the dataframe in parquet formant, name the file as \"NASA_airfoil_noise_cleaned.parquet\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Stage 13:>                                                       (0 + 8) / 200]24/01/09 04:56:48 WARN hadoop.MemoryManager: Total allocation exceeds 95.00% (906,992,014 bytes) of heap memory\n",
      "Scaling row group sizes to 96.54% for 7 writers\n",
      "24/01/09 04:56:48 WARN hadoop.MemoryManager: Total allocation exceeds 95.00% (906,992,014 bytes) of heap memory\n",
      "Scaling row group sizes to 84.47% for 8 writers\n",
      "24/01/09 04:56:49 WARN hadoop.MemoryManager: Total allocation exceeds 95.00% (906,992,014 bytes) of heap memory\n",
      "Scaling row group sizes to 96.54% for 7 writers\n",
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "# your code goes here\n",
    "df.write.parquet(\"NASA_airfoil_noise_cleaned.parquet\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Part 1 - Evaluation\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Run the code cell below.<br>\n",
    "Use the answers here to answer the final evaluation quiz in the next section.<br>\n",
    "If the code throws up any errors, go back and review the code you have written.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Part 1 - Evaluation\n",
      "Total rows =  1522\n",
      "Total rows after dropping duplicate rows =  1503\n",
      "Total rows after dropping duplicate rows and rows with null values =  1499\n",
      "New column name =  SoundLevelDecibels\n",
      "NASA_airfoil_noise_cleaned.parquet exists : True\n"
     ]
    }
   ],
   "source": [
    "print(\"Part 1 - Evaluation\")\n",
    "\n",
    "print(\"Total rows = \", rowcount1)\n",
    "print(\"Total rows after dropping duplicate rows = \", rowcount2)\n",
    "print(\"Total rows after dropping duplicate rows and rows with null values = \", rowcount3)\n",
    "print(\"New column name = \", df.columns[-1])\n",
    "\n",
    "import os\n",
    "\n",
    "print(\"NASA_airfoil_noise_cleaned.parquet exists :\", os.path.isdir(\"NASA_airfoil_noise_cleaned.parquet\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part - 2 Create a  Machine Learning Pipeline\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 1 - Load data from \"NASA_airfoil_noise_cleaned.parquet\" into a dataframe\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#your code goes here\n",
    "\n",
    "df = spark.read.parquet(\"NASA_airfoil_noise_cleaned.parquet\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 2 - Print the total number of rows in the dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Stage 15:>                                                         (0 + 8) / 8]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1499\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "#your code goes here\n",
    "\n",
    "rowcount4 = df.count()\n",
    "print(rowcount4)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 3 - Define the VectorAssembler pipeline stage\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stage 1 - Assemble the input columns into a single column \"features\". Use all the columns except SoundLevelDecibels as input features.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- Frequency: integer (nullable = true)\n",
      " |-- AngleOfAttack: double (nullable = true)\n",
      " |-- ChordLength: double (nullable = true)\n",
      " |-- FreeStreamVelocity: double (nullable = true)\n",
      " |-- SuctionSideDisplacement: double (nullable = true)\n",
      " |-- SoundLevelDecibels: double (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#your code goes here\n",
    "assembler = VectorAssembler(inputCols=['Frequency','AngleOfAttack','ChordLength','FreeStreamVelocity','SuctionSideDisplacement'], outputCol=\"features\")\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 4 - Define the StandardScaler pipeline stage\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stage 2 - Scale the \"features\" using standard scaler and store in \"scaledFeatures\" column\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#your code goes here\n",
    "\n",
    "scaler = StandardScaler(inputCol=\"features\", outputCol=\"scaledFeatures\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 5 - Define the StandardScaler pipeline stage\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stage 3 - Create a LinearRegression stage to predict \"SoundLevelDecibels\"\n",
    "\n",
    "**Note:You need to use the scaledfeatures retreived in the previous step.**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#your code goes here\n",
    "\n",
    "lr = LinearRegression(featuresCol=\"features\", labelCol=\"SoundLevelDecibels\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 6 - Build the pipeline\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Build a pipeline using the above three stages\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#your code goes here\n",
    "\n",
    "pipeline = Pipeline(stages=[assembler, scaler, lr])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 7 - Split the data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Split the data into training and testing sets with 70:30 split.\n",
    "# set the value of seed to 42\n",
    "# the above step is very important. DO NOT set the value of seed to any other value other than 42.\n",
    "\n",
    "#your code goes here\n",
    "\n",
    "(trainingData, testingData) = df.randomSplit([0.7, 0.3], seed=42)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 8 - Fit the pipeline\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "24/01/09 05:05:04 WARN util.Instrumentation: [539279fe] regParam is zero, which might cause numerical instability and overfitting.\n",
      "[Stage 20:>                                                         (0 + 8) / 8]24/01/09 05:05:06 WARN netlib.BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS\n",
      "24/01/09 05:05:06 WARN netlib.BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS\n",
      "24/01/09 05:05:06 WARN netlib.LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeSystemLAPACK\n",
      "24/01/09 05:05:06 WARN netlib.LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeRefLAPACK\n",
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "# Fit the pipeline using the training data\n",
    "# your code goes here\n",
    "\n",
    "pipelineModel = pipeline.fit(trainingData)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Part 2 - Evaluation\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Run the code cell below.<br>\n",
    "Use the answers here to answer the final evaluation quiz in the next section.<br>\n",
    "If the code throws up any errors, go back and review the code you have written.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Part 2 - Evaluation\n",
      "Total rows =  1499\n",
      "Pipeline Stage 1 =  VectorAssembler\n",
      "Pipeline Stage 2 =  StandardScaler\n",
      "Pipeline Stage 3 =  LinearRegression\n",
      "Label column =  SoundLevelDecibels\n"
     ]
    }
   ],
   "source": [
    "print(\"Part 2 - Evaluation\")\n",
    "print(\"Total rows = \", rowcount4)\n",
    "ps = [str(x).split(\"_\")[0] for x in pipeline.getStages()]\n",
    "\n",
    "print(\"Pipeline Stage 1 = \", ps[0])\n",
    "print(\"Pipeline Stage 2 = \", ps[1])\n",
    "print(\"Pipeline Stage 3 = \", ps[2])\n",
    "\n",
    "print(\"Label column = \", lr.getLabelCol())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 3 - Evaluate the Model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 1 - Predict using the model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Make predictions on testing data\n",
    "# your code goes here\n",
    "\n",
    "predictions = pipelineModel.transform(testingData)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 2 - Print the MSE\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Stage 29:>                                                         (0 + 8) / 8]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MSE : 22.5937540713487\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "#your code goes here\n",
    "\n",
    "#TODO\n",
    "from pyspark.ml.evaluation import RegressionEvaluator\n",
    "\n",
    "evaluator = RegressionEvaluator(predictionCol=\"prediction\", labelCol=\"SoundLevelDecibels\", metricName=\"mse\")\n",
    "mse = evaluator.evaluate(predictions)\n",
    "print('MSE :',mse)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 3 - Print the MAE\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Stage 31:>                                                         (0 + 8) / 8]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAE : 3.7336902294630927\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "#your code goes here\n",
    "\n",
    "#TODO\n",
    "evaluator = RegressionEvaluator(predictionCol=\"prediction\", labelCol=\"SoundLevelDecibels\", metricName=\"mae\")\n",
    "mae = evaluator.evaluate(predictions)\n",
    "print('MAE :',mae)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 4 - Print the R-Squared(R2)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Stage 33:==================================================>       (7 + 1) / 8]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "R-Squared (R2) : 0.542601650868908\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "#your code goes here\n",
    "\n",
    "#TODO\n",
    "evaluator = RegressionEvaluator(predictionCol=\"prediction\", labelCol=\"SoundLevelDecibels\", metricName=\"r2\")\n",
    "r2 = evaluator.evaluate(predictions)\n",
    "print('R-Squared (R2) :',r2)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Part 3 - Evaluation\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Run the code cell below.<br>\n",
    "Use the answers here to answer the final evaluation quiz in the next section.<br>\n",
    "If the code throws up any errors, go back and review the code you have written.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Part 3 - Evaluation\n",
      "Mean Squared Error =  22.59\n",
      "Mean Absolute Error =  3.73\n",
      "R Squared =  0.54\n",
      "Intercept =  132.6\n"
     ]
    }
   ],
   "source": [
    "print(\"Part 3 - Evaluation\")\n",
    "\n",
    "print(\"Mean Squared Error = \", round(mse,2))\n",
    "print(\"Mean Absolute Error = \", round(mae,2))\n",
    "print(\"R Squared = \", round(r2,2))\n",
    "\n",
    "lrModel = pipelineModel.stages[-1]\n",
    "\n",
    "print(\"Intercept = \", round(lrModel.intercept,2))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 4 - Persist the Model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 1 - Save the model to the path \"Final_Project\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "# Save the pipeline model as \"Final_Project\"\n",
    "# your code goes here\n",
    "pipelineModel.write().save(\"Airfoil_Project\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 2 - Load the model from the path \"Final_Project\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Load the pipeline model you have created in the previous step\n",
    "loadedPipelineModel = PipelineModel.load(\"Airfoil_Project\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 3 - Make predictions using the loaded model on the testdata\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Use the loaded pipeline model and make predictions using testingData\n",
    "predictions = loadedPipelineModel.transform(testingData)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 4 - Show the predictions\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Stage 87:>                                                         (0 + 1) / 1]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------------+------------------+\n",
      "|SoundLevelDecibels|        prediction|\n",
      "+------------------+------------------+\n",
      "|           127.315|123.64344009624709|\n",
      "|           119.975| 123.4869578861485|\n",
      "|           121.783|124.38983849684239|\n",
      "|           127.224|121.44706993294264|\n",
      "|           122.229|125.68312652454149|\n",
      "+------------------+------------------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "#show top 5 rows from the predections dataframe. Display only the label column and predictions\n",
    "#your code goes here\n",
    "predictions.select(\"SoundLevelDecibels\",\"prediction\").show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Part 4 - Evaluation\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Run the code cell below.<br>\n",
    "Use the answers here to answer the final evaluation quiz in the next section.<br>\n",
    "If the code throws up any errors, go back and review the code you have written.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Part 4 - Evaluation\n",
      "Number of stages in the pipeline =  3\n",
      "Coefficient for Frequency is -0.0013\n",
      "Coefficient for AngleOfAttack is -0.4111\n",
      "Coefficient for ChordLength is -36.1575\n",
      "Coefficient for FreeStreamVelocity is 0.1019\n",
      "Coefficient for SuctionSideDisplacement is -135.2067\n"
     ]
    }
   ],
   "source": [
    "print(\"Part 4 - Evaluation\")\n",
    "\n",
    "loadedmodel = loadedPipelineModel.stages[-1]\n",
    "totalstages = len(loadedPipelineModel.stages)\n",
    "inputcolumns = loadedPipelineModel.stages[0].getInputCols()\n",
    "\n",
    "print(\"Number of stages in the pipeline = \", totalstages)\n",
    "for i,j in zip(inputcolumns, loadedmodel.coefficients):\n",
    "    print(f\"Coefficient for {i} is {round(j,4)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stop Spark Session\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "spark.stop()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Authors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Ramesh Sannareddy](https://www.linkedin.com/in/rsannareddy/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMBD0231ENSkillsNetwork866-2023-01-01)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other Contributors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Change Log\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|Date (YYYY-MM-DD)|Version|Changed By|Change Description|\n",
    "|-|-|-|-|\n",
    "|2023-05-26|0.1|Ramesh Sannareddy|Initial Version Created|\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright © 2023 IBM Corporation. All rights reserved.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python",
   "language": "python",
   "name": "conda-env-python-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}