12 Feature_Extraction_and_Transformation_using_Spark.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"text-align:center\">\n",
    "    <a href=\"https://skills.network/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMSkillsNetworkBD0231ENCoursera2789-2023-01-01\">\n",
    "    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/assets/logos/SN_web_lightmode.png\" width=\"200\" alt=\"Skills Network Logo\">\n",
    "    </a>\n",
    "</p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Extraction and Transformation using Spark\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Estimated time needed: **30** minutes\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style='color: red'>The purpose of this lab is to show you how to use Spark to extract and transform features.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## __Table of Contents__\n",
    "\n",
    "<ol>\n",
    "  <li>\n",
    "    <a href=\"#Objectives\">Objectives\n",
    "    </a>\n",
    "  </li>\n",
    "  <li>\n",
    "    <a href=\"#Datasets\">Datasets\n",
    "    </a>\n",
    "  </li>\n",
    "  <li>\n",
    "    <a href=\"#Setup\">Setup\n",
    "    </a>\n",
    "    <ol>\n",
    "      <li>\n",
    "        <a href=\"#Installing-Required-Libraries\">Installing Required Libraries\n",
    "        </a>\n",
    "      </li>\n",
    "      <li>\n",
    "        <a href=\"#Importing-Required-Libraries\">Importing Required Libraries\n",
    "        </a>\n",
    "      </li>\n",
    "    </ol>\n",
    "  </li>\n",
    "  <li> \n",
    "    <a href=\"#Examples\">Examples\n",
    "    </a>\n",
    "    <ol>\n",
    "    <li>\n",
    "      <a href=\"#Task-1---Tokenizer\">Task 1 - Tokenizer\n",
    "      </a>\n",
    "    </li>\n",
    "    <li>\n",
    "      <a href=\"#Task-2---CountVectorizer\">Task 2 - CountVectorizer\n",
    "      </a>\n",
    "    </li>\n",
    "    <li>\n",
    "      <a href=\"#Task-3---TF-IDF\">Task 3 - TF-IDF\n",
    "      </a>\n",
    "    </li>\n",
    "    <li>\n",
    "      <a href=\"#Task-4---StopWordsRemover\">Task 4 - StopWordsRemover\n",
    "      </a>\n",
    "    </li>\n",
    "    <li>\n",
    "      <a href=\"#Task-5---StringIndexer\">Task 5 - StringIndexer\n",
    "      </a>\n",
    "    </li>\n",
    "    <li>\n",
    "      <a href=\"#Task-6---StandardScaler\">Task 6 - StandardScaler\n",
    "      </a>\n",
    "    </li>\n",
    "    </ol>\n",
    "  </li>\n",
    "  <li>\n",
    "    <a href=\"#Exercises\">Exercises\n",
    "    </a>\n",
    "  </li>\n",
    "  <ol>\n",
    "    <li>\n",
    "      <a href=\"#Exercise-1---Tokenizer\">Exercise 1 - Tokenizer\n",
    "      </a>\n",
    "    </li>\n",
    "    <li>\n",
    "      <a href=\"#Exercise-2---CountVectorizer\">Exercise 2 - CountVectorizer\n",
    "      </a>\n",
    "    </li>\n",
    "    <li>\n",
    "      <a href=\"#Exercise-3---StringIndexer\">Exercise 3 - StringIndexer\n",
    "      </a>\n",
    "    </li>\n",
    "    <li>\n",
    "      <a href=\"#Exercise-4---StandardScaler\">Exercise 4 - StandardScaler\n",
    "      </a>\n",
    "    </li>\n",
    "  </ol>\n",
    "</ol>\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objectives\n",
    "\n",
    "After completing this lab you will be able to:\n",
    "\n",
    " - Use the feature extractor CountVectorizer\n",
    " - Use the feature extractor TF-IDF\n",
    " - Use the feature transformer Tokenizer\n",
    " - Use the feature transformer StopWordsRemover\n",
    " - Use the feature transformer StringIndexer\n",
    " - Use the feature transformer StandardScaler\n",
    " \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Datasets\n",
    "\n",
    "In this lab you will be using dataset(s):\n",
    "\n",
    " - Modified version of car mileage dataset. Original dataset available at https://archive.ics.uci.edu/ml/datasets/auto+mpg \n",
    " \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this lab, we will be using the following libraries:\n",
    "\n",
    "*   [`PySpark`](https://spark.apache.org/docs/latest/api/python/index.html?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMSkillsNetworkBD0231ENCoursera2789-2023-01-01) for connecting to the Spark Cluster\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installing Required Libraries\n",
    "\n",
    "Spark Cluster is pre-installed in the Skills Network Labs environment. However, you need libraries like pyspark and findspark to connect to this cluster.\n",
    "\n",
    "If you wish to download this jupyter notebook and run on your local computer, follow the instructions mentioned <a href=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMSkillsNetwork-BD0231EN-Coursera/labs/Connecting_to_spark_cluster_using_Skills_Network_labs.ipynb\">here.</a>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following required libraries are __not__ pre-installed in the Skills Network Labs environment. __You will need to run the following cell__ to install them:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install pyspark==3.1.2 -q\n",
    "!pip install findspark -q"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importing Required Libraries\n",
    "\n",
    "_We recommend you import all required libraries in one place (here):_\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# You can also use this section to suppress warnings generated by your code:\n",
    "def warn(*args, **kwargs):\n",
    "    pass\n",
    "import warnings\n",
    "warnings.warn = warn\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "# FindSpark simplifies the process of using Apache Spark with Python\n",
    "\n",
    "import findspark\n",
    "findspark.init()\n",
    "\n",
    "from pyspark.sql import SparkSession\n",
    "from pyspark.sql.functions import rand"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "24/01/08 10:21:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n",
      "Setting default log level to \"WARN\".\n",
      "To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n"
     ]
    }
   ],
   "source": [
    "#Create SparkSession\n",
    "#Ignore any warnings by SparkSession command\n",
    "\n",
    "spark = SparkSession.builder.appName(\"Feature Extraction and Transformation using Spark\").getOrCreate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 1 - Tokenizer\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A tokenizer is used to break a sentence into words.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#import tokenizer\n",
    "from pyspark.ml.feature import Tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#create a sample dataframe\n",
    "sentenceDataFrame = spark.createDataFrame([\n",
    "    (1, \"Spark is a distributed computing system.\"),\n",
    "    (2, \"It provides interfaces for multiple languages\"),\n",
    "    (3, \"Spark is built on top of Hadoop\")\n",
    "], [\"id\", \"sentence\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---+---------------------------------------------+\n",
      "|id |sentence                                     |\n",
      "+---+---------------------------------------------+\n",
      "|1  |Spark is a distributed computing system.     |\n",
      "|2  |It provides interfaces for multiple languages|\n",
      "|3  |Spark is built on top of Hadoop              |\n",
      "+---+---------------------------------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#display the dataframe\n",
    "sentenceDataFrame.show(truncate = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#create tokenizer instance.\n",
    "#mention the column to be tokenized as inputcol\n",
    "#mention the output column name where the tokens are to be stored.\n",
    "tokenizer = Tokenizer(inputCol=\"sentence\", outputCol=\"words\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#tokenize\n",
    "token_df = tokenizer.transform(sentenceDataFrame)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---+---------------------------------------------+----------------------------------------------------+\n",
      "|id |sentence                                     |words                                               |\n",
      "+---+---------------------------------------------+----------------------------------------------------+\n",
      "|1  |Spark is a distributed computing system.     |[spark, is, a, distributed, computing, system.]     |\n",
      "|2  |It provides interfaces for multiple languages|[it, provides, interfaces, for, multiple, languages]|\n",
      "|3  |Spark is built on top of Hadoop              |[spark, is, built, on, top, of, hadoop]             |\n",
      "+---+---------------------------------------------+----------------------------------------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#display the tokenized data\n",
    "token_df.show(truncate=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 2 - CountVectorizer\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CountVectorizer is used to convert text into numerical format. It gives the count of each word in a given document.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#import CountVectorizer\n",
    "from pyspark.ml.feature import CountVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---+-------------------------------------------------+\n",
      "|id |words                                            |\n",
      "+---+-------------------------------------------------+\n",
      "|1  |[I, love, Spark, Spark, provides, Python, API]   |\n",
      "|2  |[I, love, Python, Spark, supports, Python]       |\n",
      "|3  |[Spark, solves, the, big, problem, of, big, data]|\n",
      "+---+-------------------------------------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#create a sample dataframe and display it.\n",
    "textdata = [(1, \"I love Spark Spark provides Python API \".split()),\n",
    "            (2, \"I love Python Spark supports Python\".split()),\n",
    "            (3, \"Spark solves the big problem of big data\".split())]\n",
    "\n",
    "textdata = spark.createDataFrame(textdata, [\"id\", \"words\"])\n",
    "\n",
    "textdata.show(truncate=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Create a CountVectorizer object\n",
    "# mention the column to be count vectorized as inputcol\n",
    "# mention the output column name where the count vectors are to be stored.\n",
    "cv = CountVectorizer(inputCol=\"words\", outputCol=\"features\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "# Fit the CountVectorizer model on the input data\n",
    "model = cv.fit(textdata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Transform the input data to bag-of-words vectors\n",
    "result = model.transform(textdata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---+-------------------------------------------------+----------------------------------------------------+\n",
      "|id |words                                            |features                                            |\n",
      "+---+-------------------------------------------------+----------------------------------------------------+\n",
      "|1  |[I, love, Spark, Spark, provides, Python, API]   |(13,[0,1,2,3,5,7],[2.0,1.0,1.0,1.0,1.0,1.0])        |\n",
      "|2  |[I, love, Python, Spark, supports, Python]       |(13,[0,1,2,3,12],[1.0,2.0,1.0,1.0,1.0])             |\n",
      "|3  |[Spark, solves, the, big, problem, of, big, data]|(13,[0,4,6,8,9,10,11],[1.0,2.0,1.0,1.0,1.0,1.0,1.0])|\n",
      "+---+-------------------------------------------------+----------------------------------------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# display the dataframe\n",
    "result.show(truncate=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 3 - TF-IDF\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Term Frequency-Inverse Document Frequency is used to quantify the importance of a word in a document. TF-IDF is computed by multiplying the number of times a word occurs in a document by the inverse document frequency of the word.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#import necessary classes for TF-IDF calculation\n",
    "from pyspark.ml.feature import HashingTF, IDF, Tokenizer\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---+---------------------+\n",
      "|id |sentence             |\n",
      "+---+---------------------+\n",
      "|1  |Spark supports python|\n",
      "|2  |Spark is fast        |\n",
      "|3  |Spark is easy        |\n",
      "+---+---------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#create a sample dataframe and display it.\n",
    "sentenceData = spark.createDataFrame([\n",
    "        (1, \"Spark supports python\"),\n",
    "        (2, \"Spark is fast\"),\n",
    "        (3, \"Spark is easy\")\n",
    "    ], [\"id\", \"sentence\"])\n",
    "\n",
    "sentenceData.show(truncate = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---+---------------------+-------------------------+\n",
      "|id |sentence             |words                    |\n",
      "+---+---------------------+-------------------------+\n",
      "|1  |Spark supports python|[spark, supports, python]|\n",
      "|2  |Spark is fast        |[spark, is, fast]        |\n",
      "|3  |Spark is easy        |[spark, is, easy]        |\n",
      "+---+---------------------+-------------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#tokenize the \"sentence\" column and store in the column \"words\"\n",
    "tokenizer = Tokenizer(inputCol=\"sentence\", outputCol=\"words\")\n",
    "wordsData = tokenizer.transform(sentenceData)\n",
    "wordsData.show(truncate = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---+---------------------+-------------------------+--------------------------+\n",
      "|id |sentence             |words                    |rawFeatures               |\n",
      "+---+---------------------+-------------------------+--------------------------+\n",
      "|1  |Spark supports python|[spark, supports, python]|(10,[4,5,9],[1.0,1.0,1.0])|\n",
      "|2  |Spark is fast        |[spark, is, fast]        |(10,[1,3,5],[1.0,1.0,1.0])|\n",
      "|3  |Spark is easy        |[spark, is, easy]        |(10,[0,1,5],[1.0,1.0,1.0])|\n",
      "+---+---------------------+-------------------------+--------------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Create a HashingTF object\n",
    "# mention the \"words\" column as input\n",
    "# mention the \"rawFeatures\" column as output\n",
    "\n",
    "hashingTF = HashingTF(inputCol=\"words\", outputCol=\"rawFeatures\", numFeatures=10)\n",
    "featurizedData = hashingTF.transform(wordsData)\n",
    "\n",
    "featurizedData.show(truncate = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "# Create an IDF object\n",
    "# mention the \"rawFeatures\" column as input\n",
    "# mention the \"features\" column as output\n",
    "\n",
    "idf = IDF(inputCol=\"rawFeatures\", outputCol=\"features\")\n",
    "idfModel = idf.fit(featurizedData)\n",
    "tfidfData = idfModel.transform(featurizedData)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---------------------+---------------------------------------------------------+\n",
      "|sentence             |features                                                 |\n",
      "+---------------------+---------------------------------------------------------+\n",
      "|Spark supports python|(10,[4,5,9],[0.6931471805599453,0.0,0.6931471805599453]) |\n",
      "|Spark is fast        |(10,[1,3,5],[0.28768207245178085,0.6931471805599453,0.0])|\n",
      "|Spark is easy        |(10,[0,1,5],[0.6931471805599453,0.28768207245178085,0.0])|\n",
      "+---------------------+---------------------------------------------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#display the tf-idf data\n",
    "tfidfData.select(\"sentence\", \"features\").show(truncate=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 4 - StopWordsRemover\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "StopWordsRemover is a transformer that filters out stop words like \"a\",\"an\" and \"the\".\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#import StopWordsRemover\n",
    "from pyspark.ml.feature import StopWordsRemover"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---+------------------------------------------------------------+\n",
      "|id |sentence                                                    |\n",
      "+---+------------------------------------------------------------+\n",
      "|1  |[Spark, is, an, open-source, distributed, computing, system]|\n",
      "|2  |[IT, has, interfaces, for, multiple, languages]             |\n",
      "|3  |[It, has, a, wide, range, of, libraries, and, APIs]         |\n",
      "+---+------------------------------------------------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#create a dataframe with sample text and display it\n",
    "textData = spark.createDataFrame([\n",
    "    (1, ['Spark', 'is', 'an', 'open-source', 'distributed', 'computing', 'system']),\n",
    "    (2, ['IT', 'has', 'interfaces', 'for', 'multiple', 'languages']),\n",
    "    (3, ['It', 'has', 'a', 'wide', 'range', 'of', 'libraries', 'and', 'APIs'])\n",
    "], [\"id\", \"sentence\"])\n",
    "\n",
    "textData.show(truncate = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# remove stopwords from \"sentence\" column and store the result in \"filtered_sentence\" column\n",
    "remover = StopWordsRemover(inputCol=\"sentence\", outputCol=\"filtered_sentence\")\n",
    "textData = remover.transform(textData)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---+------------------------------------------------------------+----------------------------------------------------+\n",
      "|id |sentence                                                    |filtered_sentence                                   |\n",
      "+---+------------------------------------------------------------+----------------------------------------------------+\n",
      "|1  |[Spark, is, an, open-source, distributed, computing, system]|[Spark, open-source, distributed, computing, system]|\n",
      "|2  |[IT, has, interfaces, for, multiple, languages]             |[interfaces, multiple, languages]                   |\n",
      "|3  |[It, has, a, wide, range, of, libraries, and, APIs]         |[wide, range, libraries, APIs]                      |\n",
      "+---+------------------------------------------------------------+----------------------------------------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# display the dataframe\n",
    "textData.show(truncate = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 5 - StringIndexer\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "StringIndexer converts a column of strings into a column of integers.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#import StringIndexer\n",
    "from pyspark.ml.feature import StringIndexer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---+------+\n",
      "| id| color|\n",
      "+---+------+\n",
      "|  0|   red|\n",
      "|  1|   red|\n",
      "|  2|  blue|\n",
      "|  3|yellow|\n",
      "|  4|yellow|\n",
      "|  5|yellow|\n",
      "+---+------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#create a dataframe with sample text and display it\n",
    "colors = spark.createDataFrame(\n",
    "    [(0, \"red\"), (1, \"red\"), (2, \"blue\"), (3, \"yellow\" ), (4, \"yellow\"), (5, \"yellow\")],\n",
    "    [\"id\", \"color\"])\n",
    "\n",
    "colors.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "# index the strings in the column \"color\" and store their indexes in the column \"colorIndex\"\n",
    "indexer = StringIndexer(inputCol=\"color\", outputCol=\"colorIndex\")\n",
    "indexed = indexer.fit(colors).transform(colors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---+------+----------+\n",
      "| id| color|colorIndex|\n",
      "+---+------+----------+\n",
      "|  0|   red|       1.0|\n",
      "|  1|   red|       1.0|\n",
      "|  2|  blue|       2.0|\n",
      "|  3|yellow|       0.0|\n",
      "|  4|yellow|       0.0|\n",
      "|  5|yellow|       0.0|\n",
      "+---+------+----------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# display the dataframe\n",
    "indexed.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 6 - StandardScaler\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "StandardScaler transforms the data so that it has a mean of 0 and a standard deviation of 1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#import StandardScaler\n",
    "from pyspark.ml.feature import StandardScaler\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---+------------------+\n",
      "| id|          features|\n",
      "+---+------------------+\n",
      "|  1| [70.0,170.0,17.0]|\n",
      "|  2| [80.0,165.0,25.0]|\n",
      "|  3|[65.0,150.0,135.0]|\n",
      "+---+------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Create a sample dataframe and display it\n",
    "from pyspark.ml.linalg import Vectors\n",
    "data = [(1, Vectors.dense([70, 170, 17])),\n",
    "        (2, Vectors.dense([80, 165, 25])),\n",
    "        (3, Vectors.dense([65, 150, 135]))]\n",
    "df = spark.createDataFrame(data, [\"id\", \"features\"])\n",
    "\n",
    "df.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Define the StandardScaler transformer\n",
    "scaler = StandardScaler(inputCol=\"features\", outputCol=\"scaledFeatures\", withStd=True, withMean=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Fit the transformer to the dataset\n",
    "scalerModel = scaler.fit(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Scale the data\n",
    "scaledData = scalerModel.transform(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---+------------------+------------------------------------------------------------+\n",
      "|id |features          |scaledFeatures                                              |\n",
      "+---+------------------+------------------------------------------------------------+\n",
      "|1  |[70.0,170.0,17.0] |[-0.218217890235993,0.8006407690254366,-0.6369487984517485] |\n",
      "|2  |[80.0,165.0,25.0] |[1.0910894511799611,0.32025630761017515,-0.5156252177942725]|\n",
      "|3  |[65.0,150.0,135.0]|[-0.8728715609439701,-1.120897076635609,1.152574016246021]  |\n",
      "+---+------------------+------------------------------------------------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Show the scaled data\n",
    "scaledData.show(truncate = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stop Spark Session\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "spark.stop()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercises\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create Spark Session\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#Create SparkSession\n",
    "#Ignore any warnings by SparkSession command\n",
    "\n",
    "spark = SparkSession.builder.appName(\"Exercises - Feature Extraction and Transformation using Spark\").getOrCreate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create Dataframes\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2024-01-08 10:50:20--  https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMSkillsNetwork-BD0231EN-Coursera/datasets/proverbs.csv\n",
      "Resolving cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud (cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud)... 169.63.118.104, 169.63.118.104\n",
      "Connecting to cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud (cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud)|169.63.118.104|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 846 [text/csv]\n",
      "Saving to: ‘proverbs.csv’\n",
      "\n",
      "proverbs.csv        100%[===================>]     846  --.-KB/s    in 0s      \n",
      "\n",
      "2024-01-08 10:50:20 (2.62 MB/s) - ‘proverbs.csv’ saved [846/846]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMSkillsNetwork-BD0231EN-Coursera/datasets/proverbs.csv\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Load proverbs dataset\n",
    "textdata = spark.read.csv(\"proverbs.csv\", header=True, inferSchema=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---+-----------------------------------------------------------+\n",
      "|id |text                                                       |\n",
      "+---+-----------------------------------------------------------+\n",
      "|1  |When in Rome do as the Romans do.                          |\n",
      "|2  |Do not judge a book by its cover.                          |\n",
      "|3  |Actions speak louder than words.                           |\n",
      "|4  |A picture is worth a thousand words.                       |\n",
      "|5  |If at first you do not succeed try try again.              |\n",
      "|6  |Practice makes perfect.                                    |\n",
      "|7  |An apple a day keeps the doctor away.                      |\n",
      "|8  |When the going gets tough the tough get going.             |\n",
      "|9  |All is fair in love and war.                               |\n",
      "|10 |Too many cooks spoil the broth.                            |\n",
      "|11 |You can not make an omelette without breaking eggs.        |\n",
      "|12 |The early bird catches the worm.                           |\n",
      "|13 |Better late than never.                                    |\n",
      "|14 |Honesty is the best policy.                                |\n",
      "|15 |A penny saved is a penny earned.                           |\n",
      "|16 |Two wrongs do not make a right.                            |\n",
      "|17 |The grass is always greener on the other side of the fence.|\n",
      "|18 |Do not count your chickens before they're hatched.         |\n",
      "|19 |Laughter is the best medicine.                             |\n",
      "|20 |Rome wasn't built in a day.                                |\n",
      "+---+-----------------------------------------------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# display dataframe\n",
    "textdata.show(truncate = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2024-01-08 10:57:31--  https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMSkillsNetwork-BD0231EN-Coursera/datasets/mpg.csv\n",
      "Resolving cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud (cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud)... 169.63.118.104, 169.63.118.104\n",
      "Connecting to cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud (cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud)|169.63.118.104|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 13891 (14K) [text/csv]\n",
      "Saving to: ‘mpg.csv.1’\n",
      "\n",
      "mpg.csv.1           100%[===================>]  13.57K  --.-KB/s    in 0.001s  \n",
      "\n",
      "2024-01-08 10:57:31 (20.5 MB/s) - ‘mpg.csv.1’ saved [13891/13891]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMSkillsNetwork-BD0231EN-Coursera/datasets/mpg.csv\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Load mpg dataset\n",
    "mpgdata = spark.read.csv(\"mpg.csv\", header=True, inferSchema=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----+---------+-----------+----------+------+----------+----+--------+\n",
      "| MPG|Cylinders|Engine Disp|Horsepower|Weight|Accelerate|Year|  Origin|\n",
      "+----+---------+-----------+----------+------+----------+----+--------+\n",
      "|15.0|        8|      390.0|       190|  3850|       8.5|  70|American|\n",
      "|21.0|        6|      199.0|        90|  2648|      15.0|  70|American|\n",
      "|18.0|        6|      199.0|        97|  2774|      15.5|  70|American|\n",
      "|16.0|        8|      304.0|       150|  3433|      12.0|  70|American|\n",
      "|14.0|        8|      455.0|       225|  3086|      10.0|  70|American|\n",
      "|15.0|        8|      350.0|       165|  3693|      11.5|  70|American|\n",
      "|18.0|        8|      307.0|       130|  3504|      12.0|  70|American|\n",
      "|14.0|        8|      454.0|       220|  4354|       9.0|  70|American|\n",
      "|15.0|        8|      400.0|       150|  3761|       9.5|  70|American|\n",
      "|10.0|        8|      307.0|       200|  4376|      15.0|  70|American|\n",
      "|15.0|        8|      383.0|       170|  3563|      10.0|  70|American|\n",
      "|11.0|        8|      318.0|       210|  4382|      13.5|  70|American|\n",
      "|10.0|        8|      360.0|       215|  4615|      14.0|  70|American|\n",
      "|15.0|        8|      429.0|       198|  4341|      10.0|  70|American|\n",
      "|21.0|        6|      200.0|        85|  2587|      16.0|  70|American|\n",
      "|17.0|        8|      302.0|       140|  3449|      10.5|  70|American|\n",
      "| 9.0|        8|      304.0|       193|  4732|      18.5|  70|American|\n",
      "|14.0|        8|      340.0|       160|  3609|       8.0|  70|American|\n",
      "|22.0|        6|      198.0|        95|  2833|      15.5|  70|American|\n",
      "|14.0|        8|      440.0|       215|  4312|       8.5|  70|American|\n",
      "+----+---------+-----------+----------+------+----------+----+--------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# display dataframe\n",
    "mpgdata.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1 - Tokenizer\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---+-----------------------------------------------------------+\n",
      "|id |text                                                       |\n",
      "+---+-----------------------------------------------------------+\n",
      "|1  |When in Rome do as the Romans do.                          |\n",
      "|2  |Do not judge a book by its cover.                          |\n",
      "|3  |Actions speak louder than words.                           |\n",
      "|4  |A picture is worth a thousand words.                       |\n",
      "|5  |If at first you do not succeed try try again.              |\n",
      "|6  |Practice makes perfect.                                    |\n",
      "|7  |An apple a day keeps the doctor away.                      |\n",
      "|8  |When the going gets tough the tough get going.             |\n",
      "|9  |All is fair in love and war.                               |\n",
      "|10 |Too many cooks spoil the broth.                            |\n",
      "|11 |You can not make an omelette without breaking eggs.        |\n",
      "|12 |The early bird catches the worm.                           |\n",
      "|13 |Better late than never.                                    |\n",
      "|14 |Honesty is the best policy.                                |\n",
      "|15 |A penny saved is a penny earned.                           |\n",
      "|16 |Two wrongs do not make a right.                            |\n",
      "|17 |The grass is always greener on the other side of the fence.|\n",
      "|18 |Do not count your chickens before they're hatched.         |\n",
      "|19 |Laughter is the best medicine.                             |\n",
      "|20 |Rome wasn't built in a day.                                |\n",
      "+---+-----------------------------------------------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#display the dataframe\n",
    "textdata.show(truncate = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Write code to tokenize the \"text\" column of the \"textdata\" dataframe and store the tokens in the column \"words\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# your code goes here\n",
    "from pyspark.ml.feature import Tokenizer\n",
    "\n",
    "tokenizer = Tokenizer(inputCol=\"text\", outputCol=\"words\")\n",
    "\n",
    "textdata = tokenizer.transform(textdata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for a Hint</summary>\n",
    "    \n",
    "Refer to Task 1\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```python\n",
    "from pyspark.ml.feature import Tokenizer\n",
    "\n",
    "tokenizer = Tokenizer(inputCol=\"text\", outputCol=\"words\")\n",
    "\n",
    "textdata = tokenizer.transform(textdata)\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---+------------------------------------------------------------------------+\n",
      "|id |words                                                                   |\n",
      "+---+------------------------------------------------------------------------+\n",
      "|1  |[when, in, rome, do, as, the, romans, do.]                              |\n",
      "|2  |[do, not, judge, a, book, by, its, cover.]                              |\n",
      "|3  |[actions, speak, louder, than, words.]                                  |\n",
      "|4  |[a, picture, is, worth, a, thousand, words.]                            |\n",
      "|5  |[if, at, first, you, do, not, succeed, try, try, again.]                |\n",
      "|6  |[practice, makes, perfect.]                                             |\n",
      "|7  |[an, apple, a, day, keeps, the, doctor, away.]                          |\n",
      "|8  |[when, the, going, gets, tough, the, tough, get, going.]                |\n",
      "|9  |[all, is, fair, in, love, and, war.]                                    |\n",
      "|10 |[too, many, cooks, spoil, the, broth.]                                  |\n",
      "|11 |[you, can, not, make, an, omelette, without, breaking, eggs.]           |\n",
      "|12 |[the, early, bird, catches, the, worm.]                                 |\n",
      "|13 |[better, late, than, never.]                                            |\n",
      "|14 |[honesty, is, the, best, policy.]                                       |\n",
      "|15 |[a, penny, saved, is, a, penny, earned.]                                |\n",
      "|16 |[two, wrongs, do, not, make, a, right.]                                 |\n",
      "|17 |[the, grass, is, always, greener, on, the, other, side, of, the, fence.]|\n",
      "|18 |[do, not, count, your, chickens, before, they're, hatched.]             |\n",
      "|19 |[laughter, is, the, best, medicine.]                                    |\n",
      "|20 |[rome, wasn't, built, in, a, day.]                                      |\n",
      "+---+------------------------------------------------------------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#display the tokenized data\n",
    "textdata.select(\"id\",\"words\").show(truncate=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2 - CountVectorizer\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CountVectorize the column \"words\" of the \"textdata\" dataframe and store the result in the column \"features\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# your code goes here\n",
    "from pyspark.ml.feature import CountVectorizer\n",
    "\n",
    "cv = CountVectorizer(inputCol=\"words\", outputCol=\"features\")\n",
    "model = cv.fit(textdata)\n",
    "textdata = model.transform(textdata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for a Hint</summary>\n",
    "    \n",
    "Refer to Task 2\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```python\n",
    "from pyspark.ml.feature import CountVectorizer\n",
    "\n",
    "cv = CountVectorizer(inputCol=\"words\", outputCol=\"features\")\n",
    "\n",
    "model = cv.fit(textdata)\n",
    "\n",
    "textdata = model.transform(textdata)\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------------------------------------------------------------------+----------------------------------------------------------------------------+\n",
      "|words                                                                   |features                                                                    |\n",
      "+------------------------------------------------------------------------+----------------------------------------------------------------------------+\n",
      "|[when, in, rome, do, as, the, romans, do.]                              |(99,[0,4,5,6,17,38,78,95],[1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0])                |\n",
      "|[do, not, judge, a, book, by, its, cover.]                              |(99,[1,3,4,19,21,22,62,70],[1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0])               |\n",
      "|[actions, speak, louder, than, words.]                                  |(99,[9,16,58,66,89],[1.0,1.0,1.0,1.0,1.0])                                  |\n",
      "|[a, picture, is, worth, a, thousand, words.]                            |(99,[1,2,16,61,79,83],[2.0,1.0,1.0,1.0,1.0,1.0])                            |\n",
      "|[if, at, first, you, do, not, succeed, try, try, again.]                |(99,[3,4,11,13,26,40,49,69,75],[1.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0])       |\n",
      "|[practice, makes, perfect.]                                             |(99,[23,24,52],[1.0,1.0,1.0])                                               |\n",
      "|[an, apple, a, day, keeps, the, doctor, away.]                          |(99,[0,1,7,33,41,46,73,91],[1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0])               |\n",
      "|[when, the, going, gets, tough, the, tough, get, going.]                |(99,[0,12,14,17,20,60,88],[2.0,2.0,1.0,1.0,1.0,1.0,1.0])                    |\n",
      "|[all, is, fair, in, love, and, war.]                                    |(99,[2,5,27,36,74,77,97],[1.0,1.0,1.0,1.0,1.0,1.0,1.0])                     |\n",
      "|[too, many, cooks, spoil, the, broth.]                                  |(99,[0,32,34,35,37,59],[1.0,1.0,1.0,1.0,1.0,1.0])                           |\n",
      "|[you, can, not, make, an, omelette, without, breaking, eggs.]           |(99,[3,7,13,15,44,55,56,68,93],[1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0])       |\n",
      "|[the, early, bird, catches, the, worm.]                                 |(99,[0,29,43,50,54],[2.0,1.0,1.0,1.0,1.0])                                  |\n",
      "|[better, late, than, never.]                                            |(99,[9,31,42,64],[1.0,1.0,1.0,1.0])                                         |\n",
      "|[honesty, is, the, best, policy.]                                       |(99,[0,2,10,39,98],[1.0,1.0,1.0,1.0,1.0])                                   |\n",
      "|[a, penny, saved, is, a, penny, earned.]                                |(99,[1,2,8,18,81],[2.0,1.0,2.0,1.0,1.0])                                    |\n",
      "|[two, wrongs, do, not, make, a, right.]                                 |(99,[1,3,4,15,57,67,84],[1.0,1.0,1.0,1.0,1.0,1.0,1.0])                      |\n",
      "|[the, grass, is, always, greener, on, the, other, side, of, the, fence.]|(99,[0,2,30,45,53,65,71,85,90,94],[3.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0])|\n",
      "|[do, not, count, your, chickens, before, they're, hatched.]             |(99,[3,4,47,48,51,63,80,96],[1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0])              |\n",
      "|[laughter, is, the, best, medicine.]                                    |(99,[0,2,10,25,87],[1.0,1.0,1.0,1.0,1.0])                                   |\n",
      "|[rome, wasn't, built, in, a, day.]                                      |(99,[1,5,6,28,72,82],[1.0,1.0,1.0,1.0,1.0,1.0])                             |\n",
      "+------------------------------------------------------------------------+----------------------------------------------------------------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Show the resulting dataframe\n",
    "textdata.select(\"words\",\"features\").show(truncate=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 3 - StringIndexer\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convert the string column \"Origin\" to a numeric column \"OriginIndex\" in the dataframe \"mpgdata\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# your code goes here\n",
    "from pyspark.ml.feature import StringIndexer\n",
    "indexer = StringIndexer(inputCol=\"Origin\", outputCol=\"OriginIndex\")\n",
    "indexed = indexer.fit(mpgdata).transform(mpgdata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for a Hint</summary>\n",
    "    \n",
    "Refer to Task 5\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```python\n",
    "from pyspark.ml.feature import StringIndexer\n",
    "indexer = StringIndexer(inputCol=\"Origin\", outputCol=\"OriginIndex\")\n",
    "indexed = indexer.fit(mpgdata).transform(mpgdata)\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----+---------+-----------+----------+------+----------+----+--------+-----------+\n",
      "| MPG|Cylinders|Engine Disp|Horsepower|Weight|Accelerate|Year|  Origin|OriginIndex|\n",
      "+----+---------+-----------+----------+------+----------+----+--------+-----------+\n",
      "|17.5|        6|      258.0|        95|  3193|      17.8|  76|American|        0.0|\n",
      "|14.5|        8|      351.0|       152|  4215|      12.8|  76|American|        0.0|\n",
      "|27.0|        4|      101.0|        83|  2202|      15.3|  76|European|        2.0|\n",
      "|14.0|        8|      351.0|       153|  4154|      13.5|  71|American|        0.0|\n",
      "|23.0|        8|      350.0|       125|  3900|      17.4|  79|American|        0.0|\n",
      "|17.0|        8|      260.0|       110|  4060|      19.0|  77|American|        0.0|\n",
      "|26.4|        4|      140.0|        88|  2870|      18.1|  80|American|        0.0|\n",
      "|36.1|        4|       91.0|        60|  1800|      16.4|  78|Japanese|        1.0|\n",
      "|37.3|        4|       91.0|        69|  2130|      14.7|  79|European|        2.0|\n",
      "|30.0|        4|       88.0|        76|  2065|      14.5|  71|European|        2.0|\n",
      "|31.5|        4|       89.0|        71|  1990|      14.9|  78|European|        2.0|\n",
      "|39.1|        4|       79.0|        58|  1755|      16.9|  81|Japanese|        1.0|\n",
      "|27.9|        4|      156.0|       105|  2800|      14.4|  80|American|        0.0|\n",
      "|28.0|        4|      107.0|        86|  2464|      15.5|  76|European|        2.0|\n",
      "|20.0|        6|      232.0|       100|  2914|      16.0|  75|American|        0.0|\n",
      "|23.2|        4|      156.0|       105|  2745|      16.7|  78|American|        0.0|\n",
      "|26.6|        8|      350.0|       105|  3725|      19.0|  81|American|        0.0|\n",
      "|15.0|        8|      400.0|       150|  3761|       9.5|  70|American|        0.0|\n",
      "|19.2|        8|      267.0|       125|  3605|      15.0|  79|American|        0.0|\n",
      "|21.0|        6|      231.0|       110|  3039|      15.0|  75|American|        0.0|\n",
      "+----+---------+-----------+----------+------+----------+----+--------+-----------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#show the dataframe\n",
    "\n",
    "indexed.orderBy(rand()).show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 4 - StandardScaler\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a single column named \"feaures\" using the columns \"Cylinders\", \"Engine Disp\", \"Horsepower\", \"Weight\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----+------------------------+\n",
      "|MPG |features                |\n",
      "+----+------------------------+\n",
      "|15.0|[8.0,390.0,190.0,3850.0]|\n",
      "|21.0|[6.0,199.0,90.0,2648.0] |\n",
      "|18.0|[6.0,199.0,97.0,2774.0] |\n",
      "|16.0|[8.0,304.0,150.0,3433.0]|\n",
      "|14.0|[8.0,455.0,225.0,3086.0]|\n",
      "|15.0|[8.0,350.0,165.0,3693.0]|\n",
      "|18.0|[8.0,307.0,130.0,3504.0]|\n",
      "|14.0|[8.0,454.0,220.0,4354.0]|\n",
      "|15.0|[8.0,400.0,150.0,3761.0]|\n",
      "|10.0|[8.0,307.0,200.0,4376.0]|\n",
      "|15.0|[8.0,383.0,170.0,3563.0]|\n",
      "|11.0|[8.0,318.0,210.0,4382.0]|\n",
      "|10.0|[8.0,360.0,215.0,4615.0]|\n",
      "|15.0|[8.0,429.0,198.0,4341.0]|\n",
      "|21.0|[6.0,200.0,85.0,2587.0] |\n",
      "|17.0|[8.0,302.0,140.0,3449.0]|\n",
      "|9.0 |[8.0,304.0,193.0,4732.0]|\n",
      "|14.0|[8.0,340.0,160.0,3609.0]|\n",
      "|22.0|[6.0,198.0,95.0,2833.0] |\n",
      "|14.0|[8.0,440.0,215.0,4312.0]|\n",
      "+----+------------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from pyspark.ml.feature import VectorAssembler\n",
    "assembler = VectorAssembler(inputCols=[\"Cylinders\", \"Engine Disp\", \"Horsepower\", \"Weight\"], outputCol=\"features\")\n",
    "\n",
    "mpg_transformed_data = assembler.transform(mpgdata)\n",
    "\n",
    "#show the dataframe\n",
    "mpg_transformed_data.select(\"MPG\",\"features\").show(truncate = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use StandardScaler to scale the \"features\" column of the dataframe \"mpg_transformed_data\" and save the scaled data into the \"scaledFeatures\" column.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# your code goes here\n",
    "from pyspark.ml.feature import StandardScaler\n",
    "\n",
    "scaler = StandardScaler(inputCol=\"features\", outputCol=\"scaledFeatures\", withStd=True, withMean=True)\n",
    "\n",
    "scalerModel = scaler.fit(mpg_transformed_data)\n",
    "\n",
    "scaledData = scalerModel.transform(mpg_transformed_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for a Hint</summary>\n",
    "    \n",
    "Refer to Task 6\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```python\n",
    "from pyspark.ml.feature import StandardScaler\n",
    "\n",
    "scaler = StandardScaler(inputCol=\"features\", outputCol=\"scaledFeatures\", withStd=True, withMean=True)\n",
    "\n",
    "scalerModel = scaler.fit(mpg_transformed_data)\n",
    "\n",
    "scaledData = scalerModel.transform(mpg_transformed_data)\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------------------+-----------------------------------------------------------------------------------+\n",
      "|features                |scaledFeatures                                                                     |\n",
      "+------------------------+-----------------------------------------------------------------------------------+\n",
      "|[8.0,390.0,190.0,3850.0]|[1.48205302652896,1.869079955831451,2.222084561602166,1.027093462353608]           |\n",
      "|[6.0,199.0,90.0,2648.0] |[0.3095711165403583,0.043843985634147174,-0.37591456792553746,-0.38801882543985255]|\n",
      "|[6.0,199.0,97.0,2774.0] |[0.3095711165403583,0.043843985634147174,-0.1940546288585982,-0.2396792678175763]  |\n",
      "|[8.0,304.0,150.0,3433.0]|[1.48205302652896,1.0472459587792617,1.1828849097910845,0.5361601645084557]        |\n",
      "|[8.0,455.0,225.0,3086.0]|[1.48205302652896,2.4902335582546176,3.131384256936862,0.12763773200901246]        |\n",
      "|[8.0,350.0,165.0,3693.0]|[1.48205302652896,1.4868315851095026,1.57258477922024,0.8422576643639463]          |\n",
      "|[8.0,307.0,130.0,3504.0]|[1.48205302652896,1.0759145865834079,0.6632850838855439,0.619748327930532]         |\n",
      "|[8.0,454.0,220.0,4354.0]|[1.48205302652896,2.480677348986569,3.001484300460477,1.6204516928427128]          |\n",
      "|[8.0,400.0,150.0,3761.0]|[1.48205302652896,1.964642048511938,1.1828849097910845,0.9223139335569208]         |\n",
      "|[8.0,307.0,200.0,4376.0]|[1.48205302652896,1.0759145865834079,2.481884474554936,1.646352250522793]          |\n",
      "|[8.0,383.0,170.0,3563.0]|[1.48205302652896,1.80218649095511,1.7024847356966253,0.689208914436201]           |\n",
      "|[8.0,318.0,210.0,4382.0]|[1.48205302652896,1.1810328885319439,2.7416843875077066,1.6534160389809966]        |\n",
      "|[8.0,360.0,215.0,4615.0]|[1.48205302652896,1.5823936777899896,2.871584343984092,1.9277264907745708]         |\n",
      "|[8.0,429.0,198.0,4341.0]|[1.48205302652896,2.241772117285351,2.4299244919643823,1.6051468178499384]         |\n",
      "|[6.0,200.0,85.0,2587.0] |[0.3095711165403583,0.053400194902195885,-0.5058145244019226,-0.4598340080982561]  |\n",
      "|[8.0,302.0,140.0,3449.0]|[1.48205302652896,1.0281335402431644,0.9230849968383142,0.5549969337303321]        |\n",
      "|[8.0,304.0,193.0,4732.0]|[1.48205302652896,1.0472459587792617,2.300024535487997,2.0654703657095417]         |\n",
      "|[8.0,340.0,160.0,3609.0]|[1.48205302652896,1.3912694924290154,1.442684822743855,0.7433646259490956]         |\n",
      "|[6.0,198.0,95.0,2833.0] |[0.3095711165403583,0.03428777636609846,-0.24601461144915227,-0.17021868131190726] |\n",
      "|[8.0,440.0,215.0,4312.0]|[1.48205302652896,2.3468904192338864,2.871584343984092,1.5710051736352875]         |\n",
      "+------------------------+-----------------------------------------------------------------------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Show the scaled data\n",
    "scaledData.select(\"features\",\"scaledFeatures\").show(truncate = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stop Spark Session\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "spark.stop()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations you have completed this lab.<br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Authors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Ramesh Sannareddy](https://www.linkedin.com/in/rsannareddy/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMBD0231ENSkillsNetwork866-2023-01-01)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other Contributors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Change Log\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|Date (YYYY-MM-DD)|Version|Changed By|Change Description|\n",
    "|-|-|-|-|\n",
    "|2023-05-14|0.1|Ramesh Sannareddy|Initial Version Created|\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright © 2023 IBM Corporation. All rights reserved.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python",
   "language": "python",
   "name": "conda-env-python-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}