01_About_the_Datasets.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "01 About the Datasets.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyOl5gA/AbqjUBi/xVY0e+ZQ",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/sandipanpaul21/EDA-in-Python/blob/main/01_About_the_Datasets.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2sHRkUwXV0hw"
      },
      "source": [
        "**Here we will load and understand 3 Datasets, which will be used in Exploratory Data Analysis**\n",
        "\n",
        "1. **Classification Dataset -** IRIS Dataset\n",
        "2. **Regression Dataset -**\n",
        "  \n",
        "  1. House Price Dataset\n",
        "  2. Airlines Dataset "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "u92VtX5hPh-T"
      },
      "source": [
        "# Libraries\n",
        "from sklearn import datasets\n",
        "import pandas as pd\n",
        "import warnings\n",
        "warnings.filterwarnings(\"ignore\")"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JrEIsrBxXhdp"
      },
      "source": [
        "**REGRESSION DATASET**\n",
        "1. Boston Dataset\n",
        "2. Airlines Dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uJTgA3OnqX4J",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "df18216a-717d-416a-a067-9c40b3ac8c35"
      },
      "source": [
        "# Loading Dataset\n",
        "boston_dataset = datasets.load_boston()\n",
        "\n",
        "# Dataset Description\n",
        "print(boston_dataset.DESCR) "
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            ".. _boston_dataset:\n",
            "\n",
            "Boston house prices dataset\n",
            "---------------------------\n",
            "\n",
            "**Data Set Characteristics:**  \n",
            "\n",
            "    :Number of Instances: 506 \n",
            "\n",
            "    :Number of Attributes: 13 numeric/categorical predictive. Median Value (attribute 14) is usually the target.\n",
            "\n",
            "    :Attribute Information (in order):\n",
            "        - CRIM     per capita crime rate by town\n",
            "        - ZN       proportion of residential land zoned for lots over 25,000 sq.ft.\n",
            "        - INDUS    proportion of non-retail business acres per town\n",
            "        - CHAS     Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)\n",
            "        - NOX      nitric oxides concentration (parts per 10 million)\n",
            "        - RM       average number of rooms per dwelling\n",
            "        - AGE      proportion of owner-occupied units built prior to 1940\n",
            "        - DIS      weighted distances to five Boston employment centres\n",
            "        - RAD      index of accessibility to radial highways\n",
            "        - TAX      full-value property-tax rate per $10,000\n",
            "        - PTRATIO  pupil-teacher ratio by town\n",
            "        - B        1000(Bk - 0.63)^2 where Bk is the proportion of black people by town\n",
            "        - LSTAT    % lower status of the population\n",
            "        - MEDV     Median value of owner-occupied homes in $1000's\n",
            "\n",
            "    :Missing Attribute Values: None\n",
            "\n",
            "    :Creator: Harrison, D. and Rubinfeld, D.L.\n",
            "\n",
            "This is a copy of UCI ML housing dataset.\n",
            "https://archive.ics.uci.edu/ml/machine-learning-databases/housing/\n",
            "\n",
            "\n",
            "This dataset was taken from the StatLib library which is maintained at Carnegie Mellon University.\n",
            "\n",
            "The Boston house-price data of Harrison, D. and Rubinfeld, D.L. 'Hedonic\n",
            "prices and the demand for clean air', J. Environ. Economics & Management,\n",
            "vol.5, 81-102, 1978.   Used in Belsley, Kuh & Welsch, 'Regression diagnostics\n",
            "...', Wiley, 1980.   N.B. Various transformations are used in the table on\n",
            "pages 244-261 of the latter.\n",
            "\n",
            "The Boston house-price data has been used in many machine learning papers that address regression\n",
            "problems.   \n",
            "     \n",
            ".. topic:: References\n",
            "\n",
            "   - Belsley, Kuh & Welsch, 'Regression diagnostics: Identifying Influential Data and Sources of Collinearity', Wiley, 1980. 244-261.\n",
            "   - Quinlan,R. (1993). Combining Instance-Based and Model-Based Learning. In Proceedings on the Tenth International Conference of Machine Learning, 236-243, University of Massachusetts, Amherst. Morgan Kaufmann.\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vK7WZRSkzHhQ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ec25835d-fdb6-4f4a-cbb2-5629a9b8193d"
      },
      "source": [
        "# Type of Dataset\n",
        "print(\"Dataset Type : \", type(boston_dataset))"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Dataset Type :  <class 'sklearn.utils.Bunch'>\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J9JvMvGnzUO2",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ff3c2bc3-76c7-490c-a7fd-42fb74ba5efb"
      },
      "source": [
        "# Dataset is a dictionary which holds key and value for each key\n",
        "# We can view the keys() methods\n",
        "print(\"Dataset is in Dictionary form, so the key present are as follow \\n\",boston_dataset.keys())"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Dataset is in Dictionary form, so the key present are as follow \n",
            " dict_keys(['data', 'target', 'feature_names', 'DESCR', 'filename', 'data_module'])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PQM7fhIwzjIa",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "78e45ea9-ab59-42de-db73-bdb9b3bd9463"
      },
      "source": [
        "# To check Number of Column and Rows are\n",
        "print(\"INDEPENDENT COLUMNS : Number of Columns and Rows are \",boston_dataset.data.shape)\n",
        "print(\"DEPENDENT COLUMNS : Number of Column and Rows are \", boston_dataset.target.shape)"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "INDEPENDENT COLUMNS : Number of Columns and Rows are  (506, 13)\n",
            "DEPENDENT COLUMNS : Number of Column and Rows are  (506,)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Fj_yHTXTz3rX",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "24addbe6-586f-41bc-ca54-ef79cb679354"
      },
      "source": [
        "# To check feature names\n",
        "print(\"Indepedent columns, feature names are : \\n\",boston_dataset.feature_names)"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Indepedent columns, feature names are : \n",
            " ['CRIM' 'ZN' 'INDUS' 'CHAS' 'NOX' 'RM' 'AGE' 'DIS' 'RAD' 'TAX' 'PTRATIO'\n",
            " 'B' 'LSTAT']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0dMDWJ9Pqmdn",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "87df9d6f-400b-4e13-f485-6cc4c6df788c"
      },
      "source": [
        "# Type of Dataset\n",
        "print(\"Present Dataset Type : \",type(boston_dataset))\n",
        "\n",
        "# Coverting to Data Frame\n",
        "print(\"\\nConverting to Dataframe.. \\n\")\n",
        "boston_pd = pd.DataFrame(boston_dataset.data)\n",
        "print(\"Converted Dataset Type : \",type(boston_pd))"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Present Dataset Type :  <class 'sklearn.utils.Bunch'>\n",
            "\n",
            "Converting to Dataframe.. \n",
            "\n",
            "Converted Dataset Type :  <class 'pandas.core.frame.DataFrame'>\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WMHtPydHrznw",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d2109cd4-f28b-464c-9a58-b4b77f472c17"
      },
      "source": [
        "# Top 10 Rows of the Dataset\n",
        "print(\"Column names are missing !! \\n\",boston_pd.head())"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Column names are missing !! \n",
            "         0     1     2    3      4   ...   8      9     10      11    12\n",
            "0  0.00632  18.0  2.31  0.0  0.538  ...  1.0  296.0  15.3  396.90  4.98\n",
            "1  0.02731   0.0  7.07  0.0  0.469  ...  2.0  242.0  17.8  396.90  9.14\n",
            "2  0.02729   0.0  7.07  0.0  0.469  ...  2.0  242.0  17.8  392.83  4.03\n",
            "3  0.03237   0.0  2.18  0.0  0.458  ...  3.0  222.0  18.7  394.63  2.94\n",
            "4  0.06905   0.0  2.18  0.0  0.458  ...  3.0  222.0  18.7  396.90  5.33\n",
            "\n",
            "[5 rows x 13 columns]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oZAdUegDsb9n",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cf2f0da3-26fb-4bdf-cb00-1a21d2532362"
      },
      "source": [
        "# All Column Name\n",
        "boston_pd.columns = boston_dataset.feature_names\n",
        "\n",
        "print(\"Top 5 Rows of the dataset \\n\",boston_pd.head())"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Top 5 Rows of the dataset \n",
            "       CRIM    ZN  INDUS  CHAS    NOX  ...  RAD    TAX  PTRATIO       B  LSTAT\n",
            "0  0.00632  18.0   2.31   0.0  0.538  ...  1.0  296.0     15.3  396.90   4.98\n",
            "1  0.02731   0.0   7.07   0.0  0.469  ...  2.0  242.0     17.8  396.90   9.14\n",
            "2  0.02729   0.0   7.07   0.0  0.469  ...  2.0  242.0     17.8  392.83   4.03\n",
            "3  0.03237   0.0   2.18   0.0  0.458  ...  3.0  222.0     18.7  394.63   2.94\n",
            "4  0.06905   0.0   2.18   0.0  0.458  ...  3.0  222.0     18.7  396.90   5.33\n",
            "\n",
            "[5 rows x 13 columns]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kbE39U4ntbHn",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6fc064bf-9983-4777-ae41-28a8c85b2b33"
      },
      "source": [
        "# Checking number of Rows and Columns in both the dataset\n",
        "print(\"Number of Columns and Rows in previous dataset : \", boston_dataset.data.shape)\n",
        "print(\"Number of Rows in the dataset : \",len(boston_pd))\n",
        "print(\"Number of Column in the dataset : \",len(boston_pd.columns))\n",
        "\n",
        "print(\"\\nNOTE : Target Column is MISSING !!!\")"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of Columns and Rows in previous dataset :  (506, 13)\n",
            "Number of Rows in the dataset :  506\n",
            "Number of Column in the dataset :  13\n",
            "\n",
            "NOTE : Target Column is MISSING !!!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YNQttrVzuAmp",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "dd88a48a-0bef-49d7-a22b-948b35dcc4ed"
      },
      "source": [
        "# Target Column \n",
        "boston_pd[\"MEDV\"] = boston_dataset.target\n",
        "\n",
        "print(\"Number of columns in dataset : \",len(boston_pd.columns))\n",
        "print(\"Top 5 Rows of the dataset \\n\", boston_pd.head())"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of columns in dataset :  14\n",
            "Top 5 Rows of the dataset \n",
            "       CRIM    ZN  INDUS  CHAS    NOX  ...    TAX  PTRATIO       B  LSTAT  MEDV\n",
            "0  0.00632  18.0   2.31   0.0  0.538  ...  296.0     15.3  396.90   4.98  24.0\n",
            "1  0.02731   0.0   7.07   0.0  0.469  ...  242.0     17.8  396.90   9.14  21.6\n",
            "2  0.02729   0.0   7.07   0.0  0.469  ...  242.0     17.8  392.83   4.03  34.7\n",
            "3  0.03237   0.0   2.18   0.0  0.458  ...  222.0     18.7  394.63   2.94  33.4\n",
            "4  0.06905   0.0   2.18   0.0  0.458  ...  222.0     18.7  396.90   5.33  36.2\n",
            "\n",
            "[5 rows x 14 columns]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jnu20RtX5rAh",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9a28ff77-ef23-4e03-f413-a15b61640531"
      },
      "source": [
        "# Air Passenger Dataset\n",
        "\n",
        "# Installing libraries\n",
        "# !pip install nycflights13\n",
        "\n",
        "from nycflights13 import flights\n",
        "print(\"Number of Column are \",len(flights.columns))\n",
        "print(\"Number of Rows are \", len(flights))\n",
        "print(\"Top 6 Rows are \", flights.head())"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of Column are  19\n",
            "Number of Rows are  336776\n",
            "Top 6 Rows are     year  month  day  dep_time  ...  distance  hour  minute             time_hour\n",
            "0  2013      1    1     517.0  ...      1400     5      15  2013-01-01T10:00:00Z\n",
            "1  2013      1    1     533.0  ...      1416     5      29  2013-01-01T10:00:00Z\n",
            "2  2013      1    1     542.0  ...      1089     5      40  2013-01-01T10:00:00Z\n",
            "3  2013      1    1     544.0  ...      1576     5      45  2013-01-01T10:00:00Z\n",
            "4  2013      1    1     554.0  ...       762     6       0  2013-01-01T11:00:00Z\n",
            "\n",
            "[5 rows x 19 columns]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0ixBwRWnXuwO"
      },
      "source": [
        "**CLASSIFICATION DATASET** : IRIS DATASET\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T9AOiOYXrTb7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5cbcab69-c89f-4127-cc10-a4d9e299d4d0"
      },
      "source": [
        "# For IRIS Dataset (For Classification Dataset)\n",
        "\n",
        "# Load the dataset\n",
        "iris_dataset = datasets.load_iris()\n",
        "\n",
        "# Iris Dataset Information\n",
        "print(iris_dataset.DESCR)\n",
        "\n",
        "# Converting to DataFrame\n",
        "iris_pd = pd.DataFrame(iris_dataset.data)\n",
        "\n",
        "# Independent Column Name\n",
        "iris_pd.columns = iris_dataset.feature_names\n",
        "\n",
        "# Dependent Column Name\n",
        "iris_pd[\"Class\"] = iris_dataset.target\n",
        "\n",
        "# Features of IRIS Dataframe\n",
        "print(\"Column in IRIS Dataframe :\",len(iris_pd.columns))\n",
        "print(\"Rows in IRIS Dataframe :\",len(iris_pd))\n",
        "print(\"Top 6 Rows of the IRIS Dataframe \\n\",iris_pd.head())"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            ".. _iris_dataset:\n",
            "\n",
            "Iris plants dataset\n",
            "--------------------\n",
            "\n",
            "**Data Set Characteristics:**\n",
            "\n",
            "    :Number of Instances: 150 (50 in each of three classes)\n",
            "    :Number of Attributes: 4 numeric, predictive attributes and the class\n",
            "    :Attribute Information:\n",
            "        - sepal length in cm\n",
            "        - sepal width in cm\n",
            "        - petal length in cm\n",
            "        - petal width in cm\n",
            "        - class:\n",
            "                - Iris-Setosa\n",
            "                - Iris-Versicolour\n",
            "                - Iris-Virginica\n",
            "                \n",
            "    :Summary Statistics:\n",
            "\n",
            "    ============== ==== ==== ======= ===== ====================\n",
            "                    Min  Max   Mean    SD   Class Correlation\n",
            "    ============== ==== ==== ======= ===== ====================\n",
            "    sepal length:   4.3  7.9   5.84   0.83    0.7826\n",
            "    sepal width:    2.0  4.4   3.05   0.43   -0.4194\n",
            "    petal length:   1.0  6.9   3.76   1.76    0.9490  (high!)\n",
            "    petal width:    0.1  2.5   1.20   0.76    0.9565  (high!)\n",
            "    ============== ==== ==== ======= ===== ====================\n",
            "\n",
            "    :Missing Attribute Values: None\n",
            "    :Class Distribution: 33.3% for each of 3 classes.\n",
            "    :Creator: R.A. Fisher\n",
            "    :Donor: Michael Marshall (MARSHALL%PLU@io.arc.nasa.gov)\n",
            "    :Date: July, 1988\n",
            "\n",
            "The famous Iris database, first used by Sir R.A. Fisher. The dataset is taken\n",
            "from Fisher's paper. Note that it's the same as in R, but not as in the UCI\n",
            "Machine Learning Repository, which has two wrong data points.\n",
            "\n",
            "This is perhaps the best known database to be found in the\n",
            "pattern recognition literature.  Fisher's paper is a classic in the field and\n",
            "is referenced frequently to this day.  (See Duda & Hart, for example.)  The\n",
            "data set contains 3 classes of 50 instances each, where each class refers to a\n",
            "type of iris plant.  One class is linearly separable from the other 2; the\n",
            "latter are NOT linearly separable from each other.\n",
            "\n",
            ".. topic:: References\n",
            "\n",
            "   - Fisher, R.A. \"The use of multiple measurements in taxonomic problems\"\n",
            "     Annual Eugenics, 7, Part II, 179-188 (1936); also in \"Contributions to\n",
            "     Mathematical Statistics\" (John Wiley, NY, 1950).\n",
            "   - Duda, R.O., & Hart, P.E. (1973) Pattern Classification and Scene Analysis.\n",
            "     (Q327.D83) John Wiley & Sons.  ISBN 0-471-22361-1.  See page 218.\n",
            "   - Dasarathy, B.V. (1980) \"Nosing Around the Neighborhood: A New System\n",
            "     Structure and Classification Rule for Recognition in Partially Exposed\n",
            "     Environments\".  IEEE Transactions on Pattern Analysis and Machine\n",
            "     Intelligence, Vol. PAMI-2, No. 1, 67-71.\n",
            "   - Gates, G.W. (1972) \"The Reduced Nearest Neighbor Rule\".  IEEE Transactions\n",
            "     on Information Theory, May 1972, 431-433.\n",
            "   - See also: 1988 MLC Proceedings, 54-64.  Cheeseman et al\"s AUTOCLASS II\n",
            "     conceptual clustering system finds 3 classes in the data.\n",
            "   - Many, many more ...\n",
            "Column in IRIS Dataframe : 5\n",
            "Rows in IRIS Dataframe : 150\n",
            "Top 6 Rows of the IRIS Dataframe \n",
            "    sepal length (cm)  sepal width (cm)  ...  petal width (cm)  Class\n",
            "0                5.1               3.5  ...               0.2      0\n",
            "1                4.9               3.0  ...               0.2      0\n",
            "2                4.7               3.2  ...               0.2      0\n",
            "3                4.6               3.1  ...               0.2      0\n",
            "4                5.0               3.6  ...               0.2      0\n",
            "\n",
            "[5 rows x 5 columns]\n"
          ]
        }
      ]
    }
  ]
}