07 Regression_using_SparkML.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"text-align:center\">\n",
    "    <a href=\"https://skills.network/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMSkillsNetworkBD0231ENCoursera2789-2023-01-01\">\n",
    "    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/assets/logos/SN_web_lightmode.png\" width=\"200\" alt=\"Skills Network Logo\">\n",
    "    </a>\n",
    "</p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regression using SparkML\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Estimated time needed: **30** minutes\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style='color: red'>The purpose of this lab is to show you how to use SparkML to predict the mileage of a car.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## __Table of Contents__\n",
    "\n",
    "<ol>\n",
    "  <li>\n",
    "    <a href=\"#Objectives\">Objectives\n",
    "    </a>\n",
    "  </li>\n",
    "  <li>\n",
    "    <a href=\"#Datasets\">Datasets\n",
    "    </a>\n",
    "  </li>\n",
    "  <li>\n",
    "    <a href=\"#Setup\">Setup\n",
    "    </a>\n",
    "    <ol>\n",
    "      <li>\n",
    "        <a href=\"#Installing-Required-Libraries\">Installing Required Libraries\n",
    "        </a>\n",
    "      </li>\n",
    "      <li>\n",
    "        <a href=\"#Importing-Required-Libraries\">Importing Required Libraries\n",
    "        </a>\n",
    "      </li>\n",
    "    </ol>\n",
    "  </li>\n",
    "  <li>\n",
    "    <a href=\"#Examples\">Examples\n",
    "    </a>\n",
    "    <ol>\n",
    "    <li>\n",
    "      <a href=\"#Task-1---Create-a-spark-session\">Task 1 - Create a spark session\n",
    "      </a>\n",
    "    </li>\n",
    "    <li>\n",
    "      <a href=\"#Task-2---Load-the-data-in-a-csv-file-into-a-dataframe\">Task 2 - Load the data in a csv file into a dataframe\n",
    "      </a>\n",
    "    </li>\n",
    "    <li>\n",
    "      <a href=\"#Task-3---Identify-the-label-column-and-the-input-columns\">Task 3 - Identify the label column and the input columns\n",
    "      </a>\n",
    "    </li>\n",
    "    <li>\n",
    "      <a href=\"#Task-4---Split-the-data\">Task 4 - Split the data\n",
    "      </a>\n",
    "    </li>\n",
    "    <li>\n",
    "      <a href=\"#Task-5---Build-and-Train-a-Linear-Regression-Model\">Task 5 - Build and Train a Linear Regression Model\n",
    "      </a>\n",
    "    </li>\n",
    "    <li>\n",
    "      <a href=\"#Task-6---Evaluate-the-model\">Task 6 - Evaluate the model\n",
    "      </a>\n",
    "    </li>\n",
    "    </ol>\n",
    "  </li>\n",
    "  <li>\n",
    "    <a href=\"#Exercises\">Exercises\n",
    "    </a>\n",
    "  </li>\n",
    "  <ol>\n",
    "    <li>\n",
    "      <a href=\"#Exercise-1---Create-a-spark-session\">Exercise 1 - Create a spark session\n",
    "      </a>\n",
    "    </li>\n",
    "    <li>\n",
    "      <a href=\"#Exercise-2---Load-the-data-in-a-csv-file-into-a-dataframe\">Exercise 2 - Load the data in a csv file into a dataframe\n",
    "      </a>\n",
    "    </li>\n",
    "    <li>\n",
    "      <a href=\"#Exercise-3---Identify-the-label-column-and-the-input-columns\">Exercise 3 - Identify the label column and the input columns\n",
    "      </a>\n",
    "    </li>\n",
    "    <li>\n",
    "      <a href=\"#Exercise-4---Split-the-data\">Exercise 4 - Split the data\n",
    "      </a>\n",
    "    </li>\n",
    "    <li>\n",
    "      <a href=\"#Exercise-5---Build-and-Train-a-Linear-Regression-Model\">Exercise 5 - Build and Train a Linear Regression Model\n",
    "      </a>\n",
    "    </li>\n",
    "    <li>\n",
    "      <a href=\"#Exercise-6---Evaluate-the-model\">Exercise 6 - Evaluate the model\n",
    "      </a>\n",
    "    </li>\n",
    "  </ol>\n",
    "</ol>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objectives\n",
    "\n",
    "After completing this lab you will be able to:\n",
    "\n",
    " - Use PySpark to connect to a spark cluster.\n",
    " - Create a spark session.\n",
    " - Read a csv file into a data frame.\n",
    " - Split the dataset into training and testing sets.\n",
    " - Use VectorAssembler to combine multiple columns into a single vector column\n",
    " - Use Linear Regression to build a prediction model.\n",
    " - Use metrics to evaluate the model.\n",
    " - Stop the spark session\n",
    " \n",
    " \n",
    " \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Datasets\n",
    "\n",
    "In this lab you will be using dataset(s):\n",
    "\n",
    " - Modified version of car mileage dataset. Original dataset available at https://archive.ics.uci.edu/ml/datasets/auto+mpg \n",
    " - Modified version of diamonds dataset. Original dataset available at https://www.openml.org/search?type=data&sort=runs&id=42225&status=active\n",
    " \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this lab, we will be using the following libraries:\n",
    "\n",
    "*   [`PySpark`](https://spark.apache.org/docs/latest/api/python/index.html?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMSkillsNetworkBD0231ENCoursera2789-2023-01-01) for connecting to the Spark Cluster\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installing Required Libraries\n",
    "\n",
    "Spark Cluster is pre-installed in the Skills Network Labs environment. However, you need libraries like pyspark and findspark to connect to this cluster.\n",
    "\n",
    "If you wish to download this jupyter notebook and run on your local computer, follow the instructions mentioned <a href=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMSkillsNetwork-BD0231EN-Coursera/labs/Connecting_to_spark_cluster_using_Skills_Network_labs.ipynb\">here.</a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following required libraries are __not__ pre-installed in the Skills Network Labs environment. __You will need to run the following cell__ to install them:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install pyspark==3.1.2 -q\n",
    "!pip install findspark -q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# You can use this section to suppress warnings generated by your code:\n",
    "def warn(*args, **kwargs):\n",
    "    pass\n",
    "import warnings\n",
    "warnings.warn = warn\n",
    "warnings.filterwarnings('ignore')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importing Required Libraries\n",
    "\n",
    "_We recommend you import all required libraries in one place (here):_\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# FindSpark simplifies the process of using Apache Spark with Python\n",
    "\n",
    "import findspark\n",
    "findspark.init()\n",
    "\n",
    "from pyspark.sql import SparkSession\n",
    "\n",
    "#import functions/Classes for sparkml\n",
    "\n",
    "from pyspark.ml.feature import VectorAssembler\n",
    "from pyspark.ml.regression import LinearRegression\n",
    "\n",
    "# import functions/Classes for metrics\n",
    "from pyspark.ml.evaluation import RegressionEvaluator\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 1 - Create a spark session\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "24/01/08 05:07:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n",
      "Setting default log level to \"WARN\".\n",
      "To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n"
     ]
    }
   ],
   "source": [
    "#Create SparkSession\n",
    "#Ignore any warnings by SparkSession command\n",
    "\n",
    "spark = SparkSession.builder.appName(\"Regressing using SparkML\").getOrCreate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 2 - Load the data in a csv file into a dataframe\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download the data file\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-BD0231EN-SkillsNetwork/datasets/mpg.csv\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the dataset into the spark dataframe\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# using the spark.read.csv function we load the data into a dataframe.\n",
    "# the header = True mentions that there is a header row in out csv file\n",
    "# the inferSchema = True, tells spark to automatically find out the data types of the columns.\n",
    "\n",
    "# Load mpg dataset\n",
    "mpg_data = spark.read.csv(\"mpg.csv\", header=True, inferSchema=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the schema of the dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- MPG: double (nullable = true)\n",
      " |-- Cylinders: integer (nullable = true)\n",
      " |-- Engine Disp: double (nullable = true)\n",
      " |-- Horsepower: integer (nullable = true)\n",
      " |-- Weight: integer (nullable = true)\n",
      " |-- Accelerate: double (nullable = true)\n",
      " |-- Year: integer (nullable = true)\n",
      " |-- Origin: string (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "mpg_data.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "show top 5 rows from the dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----+---------+-----------+----------+------+----------+----+--------+\n",
      "| MPG|Cylinders|Engine Disp|Horsepower|Weight|Accelerate|Year|  Origin|\n",
      "+----+---------+-----------+----------+------+----------+----+--------+\n",
      "|15.0|        8|      390.0|       190|  3850|       8.5|  70|American|\n",
      "|21.0|        6|      199.0|        90|  2648|      15.0|  70|American|\n",
      "|18.0|        6|      199.0|        97|  2774|      15.5|  70|American|\n",
      "|16.0|        8|      304.0|       150|  3433|      12.0|  70|American|\n",
      "|14.0|        8|      455.0|       225|  3086|      10.0|  70|American|\n",
      "+----+---------+-----------+----------+------+----------+----+--------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "mpg_data.show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 3 - Identify the label column and the input columns\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We ask the VectorAssembler to group a bunch of inputCols as single column named \"features\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Prepare feature vector\n",
    "assembler = VectorAssembler(inputCols=[\"Cylinders\", \"Engine Disp\", \"Horsepower\", \"Weight\", \"Accelerate\", \"Year\"], outputCol=\"features\")\n",
    "mpg_transformed_data = assembler.transform(mpg_data)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the assembled \"features\" and the label column \"MPG\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+----+\n",
      "|            features| MPG|\n",
      "+--------------------+----+\n",
      "|[8.0,390.0,190.0,...|15.0|\n",
      "|[6.0,199.0,90.0,2...|21.0|\n",
      "|[6.0,199.0,97.0,2...|18.0|\n",
      "|[8.0,304.0,150.0,...|16.0|\n",
      "|[8.0,455.0,225.0,...|14.0|\n",
      "|[8.0,350.0,165.0,...|15.0|\n",
      "|[8.0,307.0,130.0,...|18.0|\n",
      "|[8.0,454.0,220.0,...|14.0|\n",
      "|[8.0,400.0,150.0,...|15.0|\n",
      "|[8.0,307.0,200.0,...|10.0|\n",
      "|[8.0,383.0,170.0,...|15.0|\n",
      "|[8.0,318.0,210.0,...|11.0|\n",
      "|[8.0,360.0,215.0,...|10.0|\n",
      "|[8.0,429.0,198.0,...|15.0|\n",
      "|[6.0,200.0,85.0,2...|21.0|\n",
      "|[8.0,302.0,140.0,...|17.0|\n",
      "|[8.0,304.0,193.0,...| 9.0|\n",
      "|[8.0,340.0,160.0,...|14.0|\n",
      "|[6.0,198.0,95.0,2...|22.0|\n",
      "|[8.0,440.0,215.0,...|14.0|\n",
      "+--------------------+----+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "mpg_transformed_data.select(\"features\",\"MPG\").show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 4 - Split the data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We split the data set in the ratio of 70:30. 70% training data, 30% testing data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Split data into training and testing sets\n",
    "(training_data, testing_data) = mpg_transformed_data.randomSplit([0.7, 0.3], seed=42)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The random_state variable \"seed\" controls the shuffling applied to the data before applying the split. Pass the same integer for reproducible output across multiple function calls\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 5 - Build and Train a Linear Regression Model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a LR model and train the model using the training data set\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "24/01/08 05:08:58 WARN util.Instrumentation: [6a4e0416] regParam is zero, which might cause numerical instability and overfitting.\n",
      "24/01/08 05:08:59 WARN netlib.BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS\n",
      "24/01/08 05:08:59 WARN netlib.BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS\n",
      "24/01/08 05:08:59 WARN netlib.LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeSystemLAPACK\n",
      "24/01/08 05:08:59 WARN netlib.LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeRefLAPACK\n"
     ]
    }
   ],
   "source": [
    "# Train linear regression model\n",
    "# Ignore any warnings\n",
    "\n",
    "lr = LinearRegression(featuresCol=\"features\", labelCol=\"MPG\")\n",
    "model = lr.fit(training_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 6 - Evaluate the model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Your model is now trained. We use the testing data to make predictions.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Make predictions on testing data\n",
    "predictions = model.transform(testing_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### R Squared\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "R Squared = 0.7685170910359864\n"
     ]
    }
   ],
   "source": [
    "#R-squared (R2): R2 is a statistical measure that represents the proportion of variance\n",
    "#in the dependent variable (target) that is explained by the independent variables (features).\n",
    "#Higher values indicate better performance.\n",
    "\n",
    "evaluator = RegressionEvaluator(labelCol=\"MPG\", predictionCol=\"prediction\", metricName=\"r2\")\n",
    "r2 = evaluator.evaluate(predictions)\n",
    "print(\"R Squared =\", r2)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Root Mean Squared Error\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE = 3.7870898850307\n"
     ]
    }
   ],
   "source": [
    "#Root Mean Squared Error (RMSE): RMSE is the square root of the average of the squared differences\n",
    "#between the predicted and actual values. It measures the average distance between the predicted\n",
    "#and actual values, and lower values indicate better performance.\n",
    "\n",
    "evaluator = RegressionEvaluator(labelCol=\"MPG\", predictionCol=\"prediction\", metricName=\"rmse\")\n",
    "rmse = evaluator.evaluate(predictions)\n",
    "print(\"RMSE =\", rmse)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Mean Absolute Error\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAE = 2.990444141294136\n"
     ]
    }
   ],
   "source": [
    "#Mean Absolute Error (MAE): MAE is the average of the absolute differences between the predicted and\n",
    "#actual values. It measures the average absolute distance between the predicted and actual values, and\n",
    "#lower values indicate better performance.\n",
    "\n",
    "evaluator = RegressionEvaluator(labelCol=\"MPG\", predictionCol=\"prediction\", metricName=\"mae\")\n",
    "mae = evaluator.evaluate(predictions)\n",
    "print(\"MAE =\", mae)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stop Spark Session\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "spark.stop()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercises\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1 - Create a spark session\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a spark session with appname \"Diamond Price Prediction\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "spark = SparkSession.builder.appName(\"Diamond Price Prediction\").getOrCreate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for a Hint</summary>\n",
    "    \n",
    "Use the SparkSession.builder\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```python\n",
    "spark = SparkSession.builder.appName(\"Diamond Price Prediction\").getOrCreate()\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2 - Load the data in a csv file into a dataframe\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download the data set\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-BD0231EN-SkillsNetwork/datasets/diamonds.csv\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the dataset into a spark dataframe\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "diamond_data = spark.read.csv(\"diamonds.csv\", header=True, inferSchema=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for a Hint</summary>\n",
    "    \n",
    "Use the spark.read.csv function\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```python\n",
    "diamond_data = spark.read.csv(\"diamonds.csv\", header=True, inferSchema=True)\n",
    "```\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display sample data from dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---+-----+-------+-----+-------+-----+-----+-----+----+----+----+\n",
      "|  s|carat|    cut|color|clarity|depth|table|price|   x|   y|   z|\n",
      "+---+-----+-------+-----+-------+-----+-----+-----+----+----+----+\n",
      "|  1| 0.23|  Ideal|    E|    SI2| 61.5| 55.0|  326|3.95|3.98|2.43|\n",
      "|  2| 0.21|Premium|    E|    SI1| 59.8| 61.0|  326|3.89|3.84|2.31|\n",
      "|  3| 0.23|   Good|    E|    VS1| 56.9| 65.0|  327|4.05|4.07|2.31|\n",
      "|  4| 0.29|Premium|    I|    VS2| 62.4| 58.0|  334| 4.2|4.23|2.63|\n",
      "|  5| 0.31|   Good|    J|    SI2| 63.3| 58.0|  335|4.34|4.35|2.75|\n",
      "+---+-----+-------+-----+-------+-----+-----+-----+----+----+----+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#TO DO\n",
    "diamond_data.show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```python\n",
    "diamond_data.show(5)\n",
    "```\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 3 - Identify the label column and the input columns\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " - use the price column as label column\n",
    " - use the columns carat,depth and table as features\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assemble the columns columns carat,depth and table into a single column named \"features\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "assembler = VectorAssembler(inputCols=[\"carat\", \"depth\", \"table\"], outputCol=\"features\")\n",
    "diamond_transformed_data = assembler.transform(diamond_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for a Hint</summary>\n",
    "    \n",
    "Refer to Task4\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```python\n",
    "assembler = VectorAssembler(inputCols=[\"carat\", \"depth\", \"table\"], outputCol=\"features\")\n",
    "diamond_transformed_data = assembler.transform(diamond_data)\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the vectorized features and label columns\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----------------+-----+\n",
      "|        features|price|\n",
      "+----------------+-----+\n",
      "|[0.23,61.5,55.0]|  326|\n",
      "|[0.21,59.8,61.0]|  326|\n",
      "|[0.23,56.9,65.0]|  327|\n",
      "|[0.29,62.4,58.0]|  334|\n",
      "|[0.31,63.3,58.0]|  335|\n",
      "|[0.24,62.8,57.0]|  336|\n",
      "|[0.24,62.3,57.0]|  336|\n",
      "|[0.26,61.9,55.0]|  337|\n",
      "|[0.22,65.1,61.0]|  337|\n",
      "|[0.23,59.4,61.0]|  338|\n",
      "| [0.3,64.0,55.0]|  339|\n",
      "|[0.23,62.8,56.0]|  340|\n",
      "|[0.22,60.4,61.0]|  342|\n",
      "|[0.31,62.2,54.0]|  344|\n",
      "| [0.2,60.2,62.0]|  345|\n",
      "|[0.32,60.9,58.0]|  345|\n",
      "| [0.3,62.0,54.0]|  348|\n",
      "| [0.3,63.4,54.0]|  351|\n",
      "| [0.3,63.8,56.0]|  351|\n",
      "| [0.3,62.7,59.0]|  351|\n",
      "+----------------+-----+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#TO DO\n",
    "diamond_transformed_data.select(\"features\",\"price\").show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for a Hint</summary>\n",
    "    \n",
    "Refer to Task3\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```python\n",
    "diamond_transformed_data.select(\"features\",\"price\").show()\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 4 - Split the data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Split the dataset into training and testing sets in the ratio of 70:30.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "(training_data, testing_data) = diamond_transformed_data.randomSplit([0.7, 0.3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for a Hint</summary>\n",
    "    \n",
    "use the randomSplit method\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```\n",
    "(training_data, testing_data) = diamond_transformed_data.randomSplit([0.7, 0.3])\n",
    "\n",
    "\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 5 - Build and Train a Linear Regression Model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Build a linear regression and train it\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "24/01/08 05:11:35 WARN util.Instrumentation: [53daae87] regParam is zero, which might cause numerical instability and overfitting.\n",
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "lr = LinearRegression(featuresCol=\"features\", labelCol=\"price\")\n",
    "model = lr.fit(training_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for a Hint</summary>\n",
    "    \n",
    "use the fit method of LinearRegression\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```\n",
    "lr = LinearRegression(featuresCol=\"features\", labelCol=\"price\")\n",
    "model = lr.fit(training_data)\n",
    "\n",
    "\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Predict the values using the test data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 6 - Evaluate the model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Your model is now trained. Make the model predict on testing_data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "predictions = model.transform(testing_data)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for a Hint</summary>\n",
    "    \n",
    "use the transform method of the model\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```python\n",
    "predictions = model.transform(testing_data)\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the metrics :\n",
    "- R squared\n",
    "- mean absolute error\n",
    "- root mean squared error\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "R Squared = 0.8529811288635093\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAE = 998.0289483200453\n",
      "RMSE = 1543.074083645224\n"
     ]
    }
   ],
   "source": [
    "#your code goes here\n",
    "evaluator = RegressionEvaluator(labelCol=\"price\", predictionCol=\"prediction\", metricName=\"r2\")\n",
    "r2 = evaluator.evaluate(predictions)\n",
    "print(\"R Squared =\", r2)\n",
    "\n",
    "\n",
    "evaluator = RegressionEvaluator(labelCol=\"price\", predictionCol=\"prediction\", metricName=\"mae\")\n",
    "mae = evaluator.evaluate(predictions)\n",
    "print(\"MAE =\", mae)\n",
    "\n",
    "\n",
    "evaluator = RegressionEvaluator(labelCol=\"price\", predictionCol=\"prediction\", metricName=\"rmse\")\n",
    "rmse = evaluator.evaluate(predictions)\n",
    "print(\"RMSE =\", rmse)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for a Hint</summary>\n",
    "    \n",
    "use the RegressionEvaluator </details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```python\n",
    "evaluator = RegressionEvaluator(labelCol=\"price\", predictionCol=\"prediction\", metricName=\"r2\")\n",
    "r2 = evaluator.evaluate(predictions)\n",
    "print(\"R Squared =\", r2)\n",
    "\n",
    "\n",
    "evaluator = RegressionEvaluator(labelCol=\"price\", predictionCol=\"prediction\", metricName=\"mae\")\n",
    "mae = evaluator.evaluate(predictions)\n",
    "print(\"MAE =\", mae)\n",
    "\n",
    "\n",
    "evaluator = RegressionEvaluator(labelCol=\"price\", predictionCol=\"prediction\", metricName=\"rmse\")\n",
    "rmse = evaluator.evaluate(predictions)\n",
    "print(\"RMSE =\", rmse)\n",
    "\n",
    "\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations you have completed this lab.<br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Authors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Ramesh Sannareddy](https://www.linkedin.com/in/rsannareddy/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMBD0231ENSkillsNetwork866-2023-01-01)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other Contributors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Change Log\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|Date (YYYY-MM-DD)|Version|Changed By|Change Description|\n",
    "|-|-|-|-|\n",
    "|2023-04-25|0.1|Ramesh Sannareddy|Initial Version Created|\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright © 2023 IBM Corporation. All rights reserved.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python",
   "language": "python",
   "name": "conda-env-python-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}