13 ML_Pipelines_using_SparkML.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"text-align:center\">\n",
    "    <a href=\"https://skills.network/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMSkillsNetworkBD0231ENCoursera2789-2023-01-01\">\n",
    "    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/assets/logos/SN_web_lightmode.png\" width=\"200\" alt=\"Skills Network Logo\">\n",
    "    </a>\n",
    "</p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ML Pipelines using SparkML\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Estimated time needed: **30** minutes\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style='color: red'>The purpose of this lab is to show you how to use SparkML to create a machine learning pipelines.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## __Table of Contents__\n",
    "\n",
    "<ol>\n",
    "  <li>\n",
    "    <a href=\"#Objectives\">Objectives\n",
    "    </a>\n",
    "  </li>\n",
    "  <li>\n",
    "    <a href=\"#Datasets\">Datasets\n",
    "    </a>\n",
    "  </li>\n",
    "  <li>\n",
    "    <a href=\"#Setup\">Setup\n",
    "    </a>\n",
    "    <ol>\n",
    "      <li>\n",
    "        <a href=\"#Installing-Required-Libraries\">Installing Required Libraries\n",
    "        </a>\n",
    "      </li>\n",
    "      <li>\n",
    "        <a href=\"#Importing-Required-Libraries\">Importing Required Libraries\n",
    "        </a>\n",
    "      </li>\n",
    "    </ol>\n",
    "  </li>\n",
    "  <li>\n",
    "    <a href=\"#Examples\">Examples\n",
    "    </a>\n",
    "    <ol>\n",
    "    <li>\n",
    "      <a href=\"#Task-1---Load-data-set\">Task 1 - Load data set\n",
    "      </a>\n",
    "    </li>\n",
    "    <li>\n",
    "      <a href=\"#Task-2---Define-pipeline-stages\">Task 2 - Define pipeline stages\n",
    "      </a>\n",
    "    </li>\n",
    "    <li>\n",
    "      <a href=\"#Task-3---Build-the-pipeline\">Task 3 - Build the pipeline\n",
    "      </a>\n",
    "    </li>\n",
    "    <li>\n",
    "      <a href=\"#Task-4---Split-the-data\">Task 4 - Split the data\n",
    "      </a>\n",
    "    </li>\n",
    "    <li>\n",
    "      <a href=\"#Task-5---Fit-the-pipeline\">Task 5 - Fit the pipeline\n",
    "      </a>\n",
    "    </li>\n",
    "    <li>\n",
    "      <a href=\"#Task-6---Evaluate-the-model\">Task 6 - Evaluate the model\n",
    "      </a>\n",
    "    </li>\n",
    "    </ol>\n",
    "  </li>\n",
    "  <li>\n",
    "    <a href=\"#Exercises\">Exercises\n",
    "    </a>\n",
    "  </li>\n",
    "  <ol>\n",
    "    <li>\n",
    "      <a href=\"#Exercise-1---Load-data-set\">Exercise 1 - Load data set\n",
    "      </a>\n",
    "    </li>\n",
    "    <li>\n",
    "      <a href=\"#Exercise-2---Define-pipeline-stages\">Exercise 2 - Define pipeline stages\n",
    "      </a>\n",
    "    </li>\n",
    "    <li>\n",
    "      <a href=\"#Exercise-3---Build-the-pipeline\">Exercise 3 - Build the pipeline\n",
    "      </a>\n",
    "    </li>\n",
    "    <li>\n",
    "      <a href=\"#Exercise-4---Split-the-data\">Exercise 4 - Split the data\n",
    "      </a>\n",
    "    </li>\n",
    "    <li>\n",
    "      <a href=\"#Exercise-5---Fit-the-pipeline\">Exercise 5 - Fit the pipeline\n",
    "      </a>\n",
    "    </li>\n",
    "    <li>\n",
    "      <a href=\"#Exercise-6---Evaluate-the-model\">Exercise 6 - Evaluate the model\n",
    "      </a>\n",
    "    </li>\n",
    "  </ol>\n",
    "</ol>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objectives\n",
    "\n",
    "After completing this lab you will be able to:\n",
    "\n",
    " - Create a machine learning pipeline.\n",
    " - Add stages to the pipeline.\n",
    " - Run the pipeline.\n",
    " - Create a machine learning pipeline for regression.\n",
    " - Create a machine learning pipeline for classification.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Datasets\n",
    "\n",
    "In this lab you will be using dataset(s):\n",
    "\n",
    " - Modified version of car mileage dataset. Original dataset available at https://archive.ics.uci.edu/ml/datasets/auto+mpg \n",
    "  - Modified version of iris dataset. Original dataset available at https://archive.ics.uci.edu/ml/datasets/Iris \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this lab, we will be using the following libraries:\n",
    "\n",
    "*   [`PySpark`](https://spark.apache.org/docs/latest/api/python/index.html?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMSkillsNetworkBD0231ENCoursera2789-2023-01-01) for connecting to the Spark Cluster\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installing Required Libraries\n",
    "\n",
    "Spark Cluster is pre-installed in the Skills Network Labs environment. However, you need libraries like pyspark and findspark to connect to this cluster.\n",
    "\n",
    "If you wish to download this jupyter notebook and run on your local computer, follow the instructions mentioned <a href=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMSkillsNetwork-BD0231EN-Coursera/labs/Connecting_to_spark_cluster_using_Skills_Network_labs.ipynb\">here.</a>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following required libraries are __not__ pre-installed in the Skills Network Labs environment. __You will need to run the following cell__ to install them:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install pyspark==3.1.2 -q\n",
    "!pip install findspark -q"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importing Required Libraries\n",
    "\n",
    "_We recommend you import all required libraries in one place (here):_\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# You can also use this section to suppress warnings generated by your code:\n",
    "def warn(*args, **kwargs):\n",
    "    pass\n",
    "import warnings\n",
    "warnings.warn = warn\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "# FindSpark simplifies the process of using Apache Spark with Python\n",
    "\n",
    "import findspark\n",
    "findspark.init()\n",
    "\n",
    "#import functions/Classes for sparkml\n",
    "\n",
    "from pyspark.ml.regression import LinearRegression\n",
    "from pyspark.ml.classification import LogisticRegression\n",
    "\n",
    "from pyspark.ml.feature import VectorAssembler\n",
    "from pyspark.ml.feature import StandardScaler\n",
    "from pyspark.ml.feature import StringIndexer\n",
    "\n",
    "from pyspark.sql import SparkSession\n",
    "\n",
    "\n",
    "# import functions/Classes for pipeline creation\n",
    "\n",
    "from pyspark.ml import Pipeline\n",
    "\n",
    "# import functions/Classes for metrics\n",
    "from pyspark.ml.evaluation import RegressionEvaluator\n",
    "from pyspark.ml.evaluation import MulticlassClassificationEvaluator\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Examples\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 1 - Load data set\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create SparkSession\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "24/01/08 11:27:25 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n",
      "Setting default log level to \"WARN\".\n",
      "To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n"
     ]
    }
   ],
   "source": [
    "#Ignore any warnings by SparkSession command\n",
    "\n",
    "spark = SparkSession.builder.appName(\"ML Pipeline Example\").getOrCreate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download the data file\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-BD0231EN-SkillsNetwork/datasets/mpg.csv\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the dataset into the spark dataframe\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "# using the spark.read.csv function we load the data into a dataframe.\n",
    "# the header = True mentions that there is a header row in out csv file\n",
    "# the inferSchema = True, tells spark to automatically find out the data types of the columns.\n",
    "\n",
    "# Load mpg dataset\n",
    "mpg_data = spark.read.csv(\"mpg.csv\", header=True, inferSchema=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the schema of the dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- MPG: double (nullable = true)\n",
      " |-- Cylinders: integer (nullable = true)\n",
      " |-- Engine Disp: double (nullable = true)\n",
      " |-- Horsepower: integer (nullable = true)\n",
      " |-- Weight: integer (nullable = true)\n",
      " |-- Accelerate: double (nullable = true)\n",
      " |-- Year: integer (nullable = true)\n",
      " |-- Origin: string (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "mpg_data.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show top 5 rows from the dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----+---------+-----------+----------+------+----------+----+--------+\n",
      "| MPG|Cylinders|Engine Disp|Horsepower|Weight|Accelerate|Year|  Origin|\n",
      "+----+---------+-----------+----------+------+----------+----+--------+\n",
      "|15.0|        8|      390.0|       190|  3850|       8.5|  70|American|\n",
      "|21.0|        6|      199.0|        90|  2648|      15.0|  70|American|\n",
      "|18.0|        6|      199.0|        97|  2774|      15.5|  70|American|\n",
      "|16.0|        8|      304.0|       150|  3433|      12.0|  70|American|\n",
      "|14.0|        8|      455.0|       225|  3086|      10.0|  70|American|\n",
      "+----+---------+-----------+----------+------+----------+----+--------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "mpg_data.show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 2 - Define pipeline stages\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Stage 1 - assemble the input columns into a single vector \n",
    "vectorAssembler = VectorAssembler(inputCols=[\"Weight\", \"Horsepower\", \"Engine Disp\"], outputCol=\"features\")\n",
    "# Stage 2 - scale the features using standard scaler\n",
    "scaler = StandardScaler(inputCol=\"features\", outputCol=\"scaledFeatures\")\n",
    "# Stage 3 - create a linear regression instance\n",
    "lr = LinearRegression(featuresCol=\"scaledFeatures\", labelCol=\"MPG\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 3 - Build the pipeline\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Build the pipeline\n",
    "# All the stages of the pipeline are mentioned in the order of execution.\n",
    "pipeline = Pipeline(stages=[vectorAssembler, scaler, lr])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 4 - Split the data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Split the data into training and testing sets\n",
    "(trainingData, testData) = mpg_data.randomSplit([0.7, 0.3], seed=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 5 - Fit the pipeline\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "24/01/08 11:28:30 WARN util.Instrumentation: [2c90b73e] regParam is zero, which might cause numerical instability and overfitting.\n",
      "24/01/08 11:28:30 WARN netlib.BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS\n",
      "24/01/08 11:28:30 WARN netlib.BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS\n",
      "24/01/08 11:28:30 WARN netlib.LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeSystemLAPACK\n",
      "24/01/08 11:28:30 WARN netlib.LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeRefLAPACK\n"
     ]
    }
   ],
   "source": [
    "# Fit the pipeline to the training data\n",
    "# ignore any warnings. The warnings are due to the simplified settings and the security settings of the lab\n",
    "\n",
    "model = pipeline.fit(trainingData)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 6 - Evaluate the model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make predictions on the testing data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "\n",
    "predictions = model.transform(testData)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the rmse value\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Root Mean Squared Error (RMSE) = 4.69281610922322\n"
     ]
    }
   ],
   "source": [
    "\n",
    "evaluator = RegressionEvaluator(labelCol=\"MPG\", predictionCol=\"prediction\", metricName=\"rmse\")\n",
    "rmse = evaluator.evaluate(predictions)\n",
    "print(\"Root Mean Squared Error (RMSE) =\", rmse)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stop Spark Session\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "spark.stop()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercises\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1 - Load data set\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create SparkSession with appname \"ML Pipeline Exercise\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#Ignore any warnings by SparkSession command\n",
    "\n",
    "spark = SparkSession.builder.appName(\"ML Pipeline Exercise\").getOrCreate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for a Hint</summary>\n",
    "    \n",
    "Use the SparkSession.builder\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```python\n",
    "spark = SparkSession.builder.appName(\"ML Pipeline Exercise\").getOrCreate()\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download the iris data set\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-BD0231EN-SkillsNetwork/datasets/iris.csv\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the dataset into the spark dataframe\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "\n",
    "iris_data = spark.read.csv(\"iris.csv\", header=True, inferSchema=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for a Hint</summary>\n",
    "    \n",
    "Use the spark.read.csv method\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```python\n",
    "iris_data = spark.read.csv(\"iris.csv\", header=True, inferSchema=True)\n",
    "\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the schema of the dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- SepalLengthCm: double (nullable = true)\n",
      " |-- SepalWidthCm: double (nullable = true)\n",
      " |-- PetalLengthCm: double (nullable = true)\n",
      " |-- PetalWidthCm: double (nullable = true)\n",
      " |-- Species: string (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "iris_data.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#Notice that the \"Species\" column is a string column\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show top 5 rows from the dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------+------------+-------------+------------+-----------+\n",
      "|SepalLengthCm|SepalWidthCm|PetalLengthCm|PetalWidthCm|    Species|\n",
      "+-------------+------------+-------------+------------+-----------+\n",
      "|          5.1|         3.5|          1.4|         0.2|Iris-setosa|\n",
      "|          4.9|         3.0|          1.4|         0.2|Iris-setosa|\n",
      "|          4.7|         3.2|          1.3|         0.2|Iris-setosa|\n",
      "|          4.6|         3.1|          1.5|         0.2|Iris-setosa|\n",
      "|          5.0|         3.6|          1.4|         0.2|Iris-setosa|\n",
      "+-------------+------------+-------------+------------+-----------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "iris_data.show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2 - Define pipeline stages\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stage 1 - Create an indexer stage using StringIndexer that will convert the Species column into a numeric column named \"label\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "indexer = StringIndexer(inputCol=\"Species\", outputCol=\"label\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for a Hint</summary>\n",
    "    \n",
    "Use the StringIndexer with inputcol as species and outcol as label\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```python\n",
    "indexer = StringIndexer(inputCol=\"Species\", outputCol=\"label\")\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stage 2 - Create a vectorAssembler stage that creates a feature vector named features using \"SepalLengthCm\", \"SepalWidthCm\", \"PetalLengthCm\",\"PetalWidthCm\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "\n",
    "vectorAssembler = VectorAssembler(inputCols=[\"SepalLengthCm\", \"SepalWidthCm\", \"PetalLengthCm\",\"PetalWidthCm\"], outputCol=\"features\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for a Hint</summary>\n",
    "    \n",
    "Use the VectorAssembler\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```python\n",
    "vectorAssembler = VectorAssembler(inputCols=[\"SepalLengthCm\", \"SepalWidthCm\", \"PetalLengthCm\",\"PetalWidthCm\"], outputCol=\"features\")\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stage 3 - Create a scaler stage that scales the features using standard scaler, name the output columns as scaledFeatures\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "\n",
    "scaler = StandardScaler(inputCol=\"features\", outputCol=\"scaledFeatures\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for a Hint</summary>\n",
    "    \n",
    "Use the StandardScaler\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```python\n",
    "scaler = StandardScaler(inputCol=\"features\", outputCol=\"scaledFeatures\")\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stage 4 - Create a logistic regression stage using featuresCol=\"scaledFeatures\", labelCol=\"label\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "\n",
    "classifier = LogisticRegression(featuresCol=\"scaledFeatures\", labelCol=\"label\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for a Hint</summary>\n",
    "    \n",
    "Use the SparkSession.builder\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```python\n",
    "classifier = LogisticRegression(featuresCol=\"scaledFeatures\", labelCol=\"label\")\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 3 - Build the pipeline\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Build a pipeline with all the four stages created earlier. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "pipeline = Pipeline(stages=[indexer,vectorAssembler, scaler, classifier])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for a Hint</summary>\n",
    "    \n",
    "Build the pipeline using the 4 stages created earlier\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```python\n",
    "pipeline = Pipeline(stages=[indexer,vectorAssembler, scaler, classifier])\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 4 - Split the data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Split the data into training and testing sets\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "(trainingData, testData) = iris_data.randomSplit([0.7, 0.3], seed=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 5 - Fit the pipeline\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fit the pipeline to the training data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model = pipeline.fit(trainingData)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for a Hint</summary>\n",
    "    \n",
    "Use the fit method of the pipeline\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```python\n",
    "model = pipeline.fit(trainingData)\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 6 - Evaluate the model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make predictions on the testing data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "\n",
    "predictions = model.transform(testData)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for a Hint</summary>\n",
    "    \n",
    "Use the transform method of the model\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```python\n",
    "predictions = model.transform(testData)\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluate model performance\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the RMSE value\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy = 0.9722222222222222\n"
     ]
    }
   ],
   "source": [
    "evaluator = MulticlassClassificationEvaluator(labelCol=\"label\", predictionCol=\"prediction\", metricName=\"accuracy\")\n",
    "accuracy = evaluator.evaluate(predictions)\n",
    "print(\"Accuracy =\", accuracy)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stop Spark Session\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "spark.stop()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations you have completed this lab.<br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Authors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Ramesh Sannareddy](https://www.linkedin.com/in/rsannareddy/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMBD0231ENSkillsNetwork866-2023-01-01)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other Contributors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Change Log\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|Date (YYYY-MM-DD)|Version|Changed By|Change Description|\n",
    "|-|-|-|-|\n",
    "|2023-05-04|0.1|Ramesh Sannareddy|Initial Version Created|\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright © 2023 IBM Corporation. All rights reserved.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python",
   "language": "python",
   "name": "conda-env-python-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}