Webflow_Initial_TechScreen.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Webflow_Initial_TechScreen.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "mount_file_id": "1fkBfJnHSRvwV838UcUxadTN_KuUH2aTu",
      "authorship_tag": "ABX9TyPK+yfJiZEUvBNwjqiM4W/l",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/mindyng/Projects/blob/master/Webflow_Initial_TechScreen.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qkoH5VXleJ3h",
        "outputId": "4108e0d4-8eee-43d6-b9ec-db6eef37165f"
      },
      "source": [
        "!pip install chart_studio"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting chart_studio\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ca/ce/330794a6b6ca4b9182c38fc69dd2a9cbff60fd49421cb8648ee5fee352dc/chart_studio-1.1.0-py3-none-any.whl (64kB)\n",
            "\r\u001b[K     |█████                           | 10kB 17.0MB/s eta 0:00:01\r\u001b[K     |██████████▏                     | 20kB 18.4MB/s eta 0:00:01\r\u001b[K     |███████████████▎                | 30kB 10.4MB/s eta 0:00:01\r\u001b[K     |████████████████████▍           | 40kB 7.9MB/s eta 0:00:01\r\u001b[K     |█████████████████████████▍      | 51kB 4.3MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████▌ | 61kB 4.4MB/s eta 0:00:01\r\u001b[K     |████████████████████████████████| 71kB 3.5MB/s \n",
            "\u001b[?25hRequirement already satisfied: retrying>=1.3.3 in /usr/local/lib/python3.6/dist-packages (from chart_studio) (1.3.3)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from chart_studio) (2.23.0)\n",
            "Requirement already satisfied: plotly in /usr/local/lib/python3.6/dist-packages (from chart_studio) (4.4.1)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from chart_studio) (1.15.0)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->chart_studio) (1.24.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->chart_studio) (2020.11.8)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->chart_studio) (2.10)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->chart_studio) (3.0.4)\n",
            "Installing collected packages: chart-studio\n",
            "Successfully installed chart-studio-1.1.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QzKsOeqC20Tt"
      },
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "import seaborn as sns\n",
        "import matplotlib.pyplot as plt\n",
        "sns.set_style(\"darkgrid\")\n",
        "from statsmodels.stats.outliers_influence import variance_inflation_factor\n",
        "\n",
        "from datetime import datetime\n",
        "import chart_studio.plotly as py\n",
        "import plotly.graph_objs as go\n",
        "import cufflinks as cf\n",
        "from plotly import figure_factory as FF"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LSS1ooi43Dd7",
        "outputId": "6cb1cee4-38b1-4579-8804-a2694df623f2"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gBWKFvdhCkMK"
      },
      "source": [
        "## For Single csv file"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TA3MpUy12ttY"
      },
      "source": [
        "# Byteboard Prompt: Production Optimization.\n",
        "\n",
        "Reference file: https://docs.google.com/document/d/1-qmYa3O9CSHtQT8SxaRO0yamtT_-5olcgwJgXhcmXV4/edit\n",
        "\n",
        "Using given csv files for widget creation and testing. Please do an analysis to optimize production from creation to testing.\n",
        "\n",
        "Understanding Prompt:\n",
        "1. Production optimization usually looks at minimizing: \n",
        "\n",
        "* Production Volume: Track the quantities that you are able to produce\n",
        "* Production Downtime: Analyze and optimize your maintenance\n",
        "* Production Costs: Monitor the costs implied in the production\n",
        "* Overall Operations Effectiveness (OOE): Evaluate your operational efficiency\n",
        "* Overall Equipment Effectiveness (OEE): Assess the scheduled efficiency\n",
        "* Total Effective Equipment Performance (TEEP): Track overall effectiveness\n",
        "* Capacity Utilization: Maximize the use of your capacities\n",
        "* Defect Density: Track the damaged items right away\n",
        "* Rate of Return: Measure how many items are sent back\n",
        "* On-time Delivery: Ensure your products are delivered on time\n",
        "* Right First Time: Understand the performance of your production process\n",
        "* Asset Turnover: Acknowledge your assets in relation to your revenue\n",
        "* Unit Costs: Track and optimize your units costs over time\n",
        "* Return on Assets: See how profitable your business is relative to its assets\n",
        "* Maintenance Costs: Evaluate your equipment costs in the long run\n",
        "* Revenue Per Employee: Measure the success of your workforce\n",
        "\n",
        "Given data, what can I analyze:\n",
        "\n",
        "*  Production Volume\n",
        "*  Operational Efficiency\n",
        "\n",
        "\n",
        "Approach:\n",
        "\n",
        "1. For Production Volume, see what is created and made to testing. Not all created widgets are made to testing (Assumption is that some widgets just are bad for some reason before testing occurs). So good to examine distribution of widgets created intially and those that make it to testing. Do percent of tested/created.\n",
        "\n",
        "2. For Operational Efficiency, look at time from creation to test. See what the mean is as baseline. Then if there is left skew, look at what caused smaller time intervals. And optimize process for that."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AKfR8p-O3GoF",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "b23e760f-d1a9-4a52-d49a-ce8de10d2947"
      },
      "source": [
        "df = pd.read_csv('/content/sample_data/widgets_created.csv')\n",
        "df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Unnamed: 0</th>\n",
              "      <th>uid</th>\n",
              "      <th>widget_type</th>\n",
              "      <th>created_ts</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>3fc760e3-afc7-4f3b-8b17-813de9ac3e7e</td>\n",
              "      <td>A</td>\n",
              "      <td>2018-06-01 06:05:49</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>86ba8559-fb1c-4178-b835-1dc8ad8be2a1</td>\n",
              "      <td>A</td>\n",
              "      <td>2018-06-01 06:11:47</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2</td>\n",
              "      <td>76131121-b9ba-4d17-a600-374d5b46414a</td>\n",
              "      <td>A</td>\n",
              "      <td>2018-06-01 06:17:32</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3</td>\n",
              "      <td>b74b4756-8837-450c-9dc1-d4a7d78e01ea</td>\n",
              "      <td>A</td>\n",
              "      <td>2018-06-01 06:23:36</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4</td>\n",
              "      <td>d54e5b6b-1bcd-4c23-95c5-f5743497f1a3</td>\n",
              "      <td>A</td>\n",
              "      <td>2018-06-01 06:29:30</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Unnamed: 0  ...           created_ts\n",
              "0           0  ...  2018-06-01 06:05:49\n",
              "1           1  ...  2018-06-01 06:11:47\n",
              "2           2  ...  2018-06-01 06:17:32\n",
              "3           3  ...  2018-06-01 06:23:36\n",
              "4           4  ...  2018-06-01 06:29:30\n",
              "\n",
              "[5 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JerfuPZpU6P4",
        "outputId": "35d1e5f5-72ff-4a4d-ddd9-f2e675ac6f4b"
      },
      "source": [
        "df.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(32435, 4)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "G_3WT2c9VHD2",
        "outputId": "33d492d5-4f37-48db-d5c5-3d71bdb4c13b"
      },
      "source": [
        "df.info()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 32435 entries, 0 to 32434\n",
            "Data columns (total 4 columns):\n",
            " #   Column       Non-Null Count  Dtype \n",
            "---  ------       --------------  ----- \n",
            " 0   Unnamed: 0   32435 non-null  int64 \n",
            " 1   uid          32435 non-null  object\n",
            " 2   widget_type  32434 non-null  object\n",
            " 3   created_ts   32434 non-null  object\n",
            "dtypes: int64(1), object(3)\n",
            "memory usage: 1013.7+ KB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tFZm_dOIVSr0",
        "outputId": "791d4d7d-bc2a-45ef-bda9-0ee5f353ab50"
      },
      "source": [
        "df.nunique()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Unnamed: 0     32435\n",
              "uid            32435\n",
              "widget_type        1\n",
              "created_ts     32434\n",
              "dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-eV9uV9EVoXs"
      },
      "source": [
        "## Production Volume Optimization\n",
        "\n",
        "Based on preliminary examination of created df, see that only 1 Type of Widget produced: Type A and 48567 different ones are produced at majority different times. Time stamps are mostly unique but there seems to be a duplicate based on output of unique values in created_ts.\n",
        "\n",
        "So there are a total of **48567** total Widget Type A produced. Let us now see how many are sent over to testing by doing an inner join between created and tested DF's. Assumption is that uid is primary key from created and foreign key in tested."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "id": "0tfRztxrf-rK",
        "outputId": "c8abb68f-81a8-4e54-f61a-f7e68e0dc1d2"
      },
      "source": [
        "df2 = pd.read_csv('/content/sample_data/widgets_tested.csv')\n",
        "df2.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Unnamed: 0</th>\n",
              "      <th>uid</th>\n",
              "      <th>test_ts</th>\n",
              "      <th>test_status</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>3fc760e3-afc7-4f3b-8b17-813de9ac3e7e</td>\n",
              "      <td>2018-06-01 06:08:11</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>86ba8559-fb1c-4178-b835-1dc8ad8be2a1</td>\n",
              "      <td>2018-06-01 06:13:41</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2</td>\n",
              "      <td>76131121-b9ba-4d17-a600-374d5b46414a</td>\n",
              "      <td>2018-06-01 06:19:43</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3</td>\n",
              "      <td>b74b4756-8837-450c-9dc1-d4a7d78e01ea</td>\n",
              "      <td>2018-06-01 06:25:40</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4</td>\n",
              "      <td>d54e5b6b-1bcd-4c23-95c5-f5743497f1a3</td>\n",
              "      <td>2018-06-01 06:31:21</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Unnamed: 0  ... test_status\n",
              "0           0  ...        True\n",
              "1           1  ...        True\n",
              "2           2  ...        True\n",
              "3           3  ...        True\n",
              "4           4  ...        True\n",
              "\n",
              "[5 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "v4WIY_RhXDNc",
        "outputId": "875e1aae-fa43-443f-b24f-72c8d56c5fce"
      },
      "source": [
        "df2.shape #Bigger dataset then created dataset"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(46357, 4)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dDXrsDqFXC-j",
        "outputId": "54e7b646-0776-4683-effc-b26d816f3927"
      },
      "source": [
        "df2.info()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 46357 entries, 0 to 46356\n",
            "Data columns (total 4 columns):\n",
            " #   Column       Non-Null Count  Dtype \n",
            "---  ------       --------------  ----- \n",
            " 0   Unnamed: 0   46357 non-null  int64 \n",
            " 1   uid          46357 non-null  object\n",
            " 2   test_ts      46356 non-null  object\n",
            " 3   test_status  46356 non-null  object\n",
            "dtypes: int64(1), object(3)\n",
            "memory usage: 1.4+ MB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "q_CQqRg2XCz8",
        "outputId": "59959272-ca3b-4187-9724-a66f037888be"
      },
      "source": [
        "df2.nunique()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Unnamed: 0     46357\n",
              "uid            46353\n",
              "test_ts        46356\n",
              "test_status        2\n",
              "dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-PjvWTMmX2Uh"
      },
      "source": [
        "This tested dataset is much bigger than the created dataset so this dataset was sourced from production of other widgets. Also, can tell there are repeated widgets logged into tested (tested multiple times). There are repeated time stamps as well, which means either different widgets got tested at the same time, same widgets got tested multiple times or both. Now, let's merge created and tested to see which produced widgets got tested."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 606
        },
        "id": "ieUj0CXChAlJ",
        "outputId": "c85489b0-1b77-4088-fbea-5235b17a0341"
      },
      "source": [
        "df3 = df.merge(df2, on='uid', how='inner')\n",
        "df3"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Unnamed: 0_x</th>\n",
              "      <th>uid</th>\n",
              "      <th>widget_type</th>\n",
              "      <th>created_ts</th>\n",
              "      <th>Unnamed: 0_y</th>\n",
              "      <th>test_ts</th>\n",
              "      <th>test_status</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>3fc760e3-afc7-4f3b-8b17-813de9ac3e7e</td>\n",
              "      <td>A</td>\n",
              "      <td>2018-06-01 06:05:49</td>\n",
              "      <td>0</td>\n",
              "      <td>2018-06-01 06:08:11</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>86ba8559-fb1c-4178-b835-1dc8ad8be2a1</td>\n",
              "      <td>A</td>\n",
              "      <td>2018-06-01 06:11:47</td>\n",
              "      <td>1</td>\n",
              "      <td>2018-06-01 06:13:41</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2</td>\n",
              "      <td>76131121-b9ba-4d17-a600-374d5b46414a</td>\n",
              "      <td>A</td>\n",
              "      <td>2018-06-01 06:17:32</td>\n",
              "      <td>2</td>\n",
              "      <td>2018-06-01 06:19:43</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3</td>\n",
              "      <td>b74b4756-8837-450c-9dc1-d4a7d78e01ea</td>\n",
              "      <td>A</td>\n",
              "      <td>2018-06-01 06:23:36</td>\n",
              "      <td>3</td>\n",
              "      <td>2018-06-01 06:25:40</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4</td>\n",
              "      <td>d54e5b6b-1bcd-4c23-95c5-f5743497f1a3</td>\n",
              "      <td>A</td>\n",
              "      <td>2018-06-01 06:29:30</td>\n",
              "      <td>4</td>\n",
              "      <td>2018-06-01 06:31:21</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>32393</th>\n",
              "      <td>32429</td>\n",
              "      <td>36a6affd-1006-4647-ae42-26410330a637</td>\n",
              "      <td>A</td>\n",
              "      <td>2019-01-26 16:08:47</td>\n",
              "      <td>32393</td>\n",
              "      <td>2019-01-26 16:10:31</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>32394</th>\n",
              "      <td>32430</td>\n",
              "      <td>9cb1a8b0-ef47-4e16-a054-b5cc3f99a798</td>\n",
              "      <td>A</td>\n",
              "      <td>2019-01-26 16:14:00</td>\n",
              "      <td>32394</td>\n",
              "      <td>2019-01-26 16:15:48</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>32395</th>\n",
              "      <td>32431</td>\n",
              "      <td>0485fe6c-4cd1-4787-8992-e0497a816a57</td>\n",
              "      <td>A</td>\n",
              "      <td>2019-01-26 16:18:53</td>\n",
              "      <td>32395</td>\n",
              "      <td>2019-01-26 16:20:50</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>32396</th>\n",
              "      <td>32432</td>\n",
              "      <td>2a440628-ad63-4b26-951b-5d5cda64de16</td>\n",
              "      <td>A</td>\n",
              "      <td>2019-01-26 16:24:05</td>\n",
              "      <td>32396</td>\n",
              "      <td>2019-01-26 16:26:28</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>32397</th>\n",
              "      <td>32433</td>\n",
              "      <td>4b16fbcb-2cdd-4fe6-b18d-011f51643ed2</td>\n",
              "      <td>A</td>\n",
              "      <td>2019-01-26 16:29:39</td>\n",
              "      <td>32397</td>\n",
              "      <td>2019-01-26 16:31:27</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>32398 rows × 7 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "       Unnamed: 0_x  ... test_status\n",
              "0                 0  ...        True\n",
              "1                 1  ...        True\n",
              "2                 2  ...        True\n",
              "3                 3  ...        True\n",
              "4                 4  ...        True\n",
              "...             ...  ...         ...\n",
              "32393         32429  ...        True\n",
              "32394         32430  ...        True\n",
              "32395         32431  ...        True\n",
              "32396         32432  ...        True\n",
              "32397         32433  ...        True\n",
              "\n",
              "[32398 rows x 7 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 36
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PAr0WyLCaIVS",
        "outputId": "79e1c9a7-97f1-4ed0-9fc3-56832f9f535b"
      },
      "source": [
        "df3.shape #Looks like majority of created got tested "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(32398, 7)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OpbcbBdbaIN_",
        "outputId": "8be554a2-47f4-4fe6-bf67-681fe88ea0a2"
      },
      "source": [
        "df3.info()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "Int64Index: 32398 entries, 0 to 32397\n",
            "Data columns (total 7 columns):\n",
            " #   Column        Non-Null Count  Dtype \n",
            "---  ------        --------------  ----- \n",
            " 0   Unnamed: 0_x  32398 non-null  int64 \n",
            " 1   uid           32398 non-null  object\n",
            " 2   widget_type   32398 non-null  object\n",
            " 3   created_ts    32398 non-null  object\n",
            " 4   Unnamed: 0_y  32398 non-null  int64 \n",
            " 5   test_ts       32398 non-null  object\n",
            " 6   test_status   32398 non-null  object\n",
            "dtypes: int64(2), object(5)\n",
            "memory usage: 2.0+ MB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qfKyoxhjaID1",
        "outputId": "644c19df-e94f-42a7-fbe2-ce2d5ee87821"
      },
      "source": [
        "df3.nunique()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Unnamed: 0_x    32395\n",
              "uid             32395\n",
              "widget_type         1\n",
              "created_ts      32395\n",
              "Unnamed: 0_y    32398\n",
              "test_ts         32398\n",
              "test_status         2\n",
              "dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 39
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jh_SAdshbqM6"
      },
      "source": [
        "Looks like there are repeated widgets."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "75YltCRHdmQk",
        "outputId": "f3be0548-bf81-4b53-d7d5-1721c41f9af3"
      },
      "source": [
        "df3.test_status.value_counts()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True     29198\n",
              "False     3200\n",
              "Name: test_status, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 40
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 297
        },
        "id": "3IGo7qQ6f6WB",
        "outputId": "451be837-3b08-4b6f-f30b-a3f629bba3f6"
      },
      "source": [
        "sns.countplot(x=\"test_status\", data=df3)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.axes._subplots.AxesSubplot at 0x7fe261c59fd0>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 41
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAEHCAYAAABvHnsJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAW60lEQVR4nO3df7SdVX3n8ff1En7JxQt4pDSBQkv67UQZo2SQlpkWbMVg7QSVoaBCUEadAqMsOx3RxQIbmBlYIq6MVbrkhySOGilqYSwaMwyODiPCRKnhR7+diBES+RFIAqEohMuZP5598XDJDTc7Oefcm/t+rXVWnud7nh/7ZJ2cT569n7PPQLvdRpKkGi/rdwMkSVOXISJJqmaISJKqGSKSpGqGiCSpmiEiSaq2W78b0GvPPfdce2TE25olaXvMmDH4KNAaW+9aiETEnsB3gT3Kea7PzAsj4jBgGXAAsBI4LTOfiYg9gKXAkcBjwJ9m5ppyrI8CZwIjwAczc3mpzwcWA4PAVZl5yUu1a2SkzaZNT+3U1ypJu7pWa+hnW6t3szvraeCNmflaYC4wPyKOBi4FPpWZhwMbacKB8ufGUv9U2Y6ImAOcArwamA98NiIGI2IQ+AxwAjAHOLVsK0nqka6FSGa2M/PJsjqjPNrAG4HrS30JcGJZXlDWKc//YUQMlPqyzHw6M38KrAaOKo/VmXlfZj5Dc3WzoFuvR5L0Yl0dWC9XDHcCjwArgJ8AmzLz2bLJWmBmWZ4JPABQnn+cpsvr+fqYfcarS5J6pKsD65k5AsyNiGHg68DvdPN8EzE4OMDw8N79boYk7RJ6cndWZm6KiFuA3wWGI2K3crUxC1hXNlsHHAysjYjdgFfQDLCP1kd17jNefVwOrEvS9mu1hrZa71p3VkS0yhUIEbEX8CbgXuAW4KSy2ULghrJ8Y1mnPP8/M7Nd6qdExB7lzq7ZwO3AHcDsiDgsInanGXy/sVuvR5L0Yt0cEzkIuCUifkzzgb8iM78BfAT4cESsphnzuLpsfzVwQKl/GDgPIDPvBq4D7gG+BZydmSPlSuYcYDlNOF1XtpUk9cjAdPs9kS1bRtp2Z0nS9mm1hlYC88bWp9031qVd2f6vmMHg7nv2uxmaZEae+SUbHt/SlWMbItIuZHD3Pbl/0RH9boYmmUMuWAV0J0ScgFGSVM0QkSRVM0QkSdUMEUlSNUNEklTNEJEkVTNEJEnVDBFJUjVDRJJUzRCRJFUzRCRJ1QwRSVI1Q0SSVM0QkSRVM0QkSdUMEUlSNUNEklTNEJEkVTNEJEnVDBFJUjVDRJJUzRCRJFUzRCRJ1QwRSVK13bp14Ig4GFgKHAi0gc9l5uKI+DjwPmB92fRjmXlT2eejwJnACPDBzFxe6vOBxcAgcFVmXlLqhwHLgAOAlcBpmflMt16TJOmFunkl8izw55k5BzgaODsi5pTnPpWZc8tjNEDmAKcArwbmA5+NiMGIGAQ+A5wAzAFO7TjOpeVYhwMbaQJIktQjXQuRzHwwM39YljcD9wIzt7HLAmBZZj6dmT8FVgNHlcfqzLyvXGUsAxZExADwRuD6sv8S4MTuvBpJ0tb0ZEwkIg4FXgf8oJTOiYgfR8Q1EbFfqc0EHujYbW2pjVc/ANiUmc+OqUuSeqRrYyKjImIf4KvAuZn5RERcAVxEM05yEfBJ4L3dbseowcEBhof37tXpJGlS6NbnXldDJCJm0ATIFzPzawCZ+XDH81cC3yir64CDO3afVWqMU38MGI6I3crVSOf24xoZabNp01N1L0ia5FqtoX43QZPUjn7ujffe6lp3VhmzuBq4NzMv76gf1LHZ24C7yvKNwCkRsUe562o2cDtwBzA7Ig6LiN1pBt9vzMw2cAtwUtl/IXBDt16PJOnFunklcgxwGrAqIu4stY/R3F01l6Y7aw3wAYDMvDsirgPuobmz6+zMHAGIiHOA5TS3+F6TmXeX430EWBYRFwM/ogktSVKPDLTb7X63oae2bBlp252lXVWrNcT9i47odzM0yRxywSrWr9+8Q8dotYZWAvPG1v3GuiSpmiEiSapmiEiSqhkikqRqhogkqZohIkmqZohIkqoZIpKkaoaIJKmaISJJqmaISJKqGSKSpGqGiCSpmiEiSapmiEiSqhkikqRqhogkqZohIkmqZohIkqoZIpKkaoaIJKmaISJJqmaISJKqGSKSpGqGiCSpmiEiSapmiEiSqu3WrQNHxMHAUuBAoA18LjMXR8T+wFeAQ4E1wMmZuTEiBoDFwFuAp4AzMvOH5VgLgfPLoS/OzCWlfiRwLbAXcBPwocxsd+s1SZJeqJtXIs8Cf56Zc4CjgbMjYg5wHnBzZs4Gbi7rACcAs8vj/cAVACV0LgTeABwFXBgR+5V9rgDe17Hf/C6+HknSGF0Lkcx8cPRKIjM3A/cCM4EFwJKy2RLgxLK8AFiame3MvA0YjoiDgDcDKzJzQ2ZuBFYA88tz+2bmbeXqY2nHsSRJPdCTMZGIOBR4HfAD4MDMfLA89RBNdxc0AfNAx25rS21b9bVbqUuSeqRrYyKjImIf4KvAuZn5REQ8/1xmtiOip2MYg4MDDA/v3ctTSlLfdetzr6shEhEzaALki5n5tVJ+OCIOyswHS5fUI6W+Dji4Y/dZpbYOOHZM/TulPmsr22/TyEibTZue2v4XI00BrdZQv5ugSWpHP/fGe291rTur3G11NXBvZl7e8dSNwMKyvBC4oaN+ekQMRMTRwOOl22s5cHxE7FcG1I8HlpfnnoiIo8u5Tu84liSpB7p5JXIMcBqwKiLuLLWPAZcA10XEmcDPgJPLczfR3N67muYW3/cAZOaGiLgIuKNstygzN5Tls/jVLb7fLA9JUo8MtNvT62sVW7aMtO3O0q6q1Rri/kVH9LsZmmQOuWAV69dv3qFjtFpDK4F5Y+t+Y12SVM0QkSRVM0QkSdUMEUlSNUNEklTNEJEkVTNEJEnVDBFJUjVDRJJUzRCRJFUzRCRJ1QwRSVI1Q0SSVM0QkSRVM0QkSdUMEUlStQmFSETcPJGaJGl62ebP40bEnsDewCvL75sPlKf2BWZ2uW2SpEnupX5j/QPAucCvAyv5VYg8AfxVF9slSZoCthkimbkYWBwR/z4zP92jNkmSpoiXuhIBIDM/HRG/BxzauU9mLu1SuyRJU8CEQiQivgD8FnAnMFLKbcAQkaRpbEIhAswD5mRmu5uNkSRNLRP9nshdwK91syGSpKlnolcirwTuiYjbgadHi5n5r7vSKknSlDDREPl4NxshSZqaJnp31v/a3gNHxDXAW4FHMvM1pfZx4H3A+rLZxzLzpvLcR4EzaQbuP5iZy0t9PrAYGASuysxLSv0wYBlwAM13WE7LzGe2t52SpHoTvTtrM83dWAC7AzOAf8rMfbex27U0X0gcewfXpzLzsjHHnwOcArya5ouN/yMifrs8/RngTcBa4I6IuDEz7wEuLcdaFhF/TRNAV0zk9UiSdo4JDaxn5lBm7ltCYy/gHcBnX2Kf7wIbJtiOBcCyzHw6M38KrAaOKo/VmXlfucpYBiyIiAHgjcD1Zf8lwIkTPJckaSfZ7ll8M7OdmX8LvLnynOdExI8j4poyHxc083A90LHN2lIbr34AsCkznx1TlyT10ES7s97esfoymu+N/LLifFcAF9F0jV0EfBJ4b8Vxqg0ODjA8vHcvTylJfdetz72J3p31Jx3LzwJraLqgtktmPjy6HBFXAt8oq+uAgzs2nVVqjFN/DBiOiN3K1Ujn9ts0MtJm06antrfp0pTQag31uwmapHb0c2+899ZE7856zw6dvYiIgzLzwbL6NpovMQLcCHwpIi6nGVifDdxOM2vw7HIn1jqawfd3ZmY7Im4BTqIZJ1kI3LAz2ihJmriJdmfNAj4NHFNK3wM+lJlrt7HPl4FjaX6LZC1wIXBsRMyl6c5aQzPVPJl5d0RcB9xDc6VzdmaOlOOcAyynucX3msy8u5ziI8CyiLgY+BFw9QRfsyRpJxlot196OqyIWAF8CfhCKb0beFdmvqmLbeuKLVtG2nZnaVfVag1x/6Ij+t0MTTKHXLCK9es379AxWq2hlTTj4S8w0TGRVmZ+vmP92og4d4daJEma8iYaIo9FxLuBL5f1U2kGtyVJ09hEvyfyXuBk4CHgQZoB7TO61CZJ0hQx0SuRRcDCzNwIEBH7A5fR4+94SJIml4leifzz0QAByMwNwOu60yRJ0lQx0RB5WccUJaNXIhO9ipEk7aImGgSfBL4fEX9T1v8N8J+60yRJ0lQx0Vl8lwJvBx4uj7dn5he2vZckaVc34S6p8hse93SxLZKkKWa7p4KXJGmUISJJqmaISJKqGSKSpGqGiCSpmiEiSapmiEiSqhkikqRqhogkqZohIkmqZohIkqoZIpKkaoaIJKmaISJJqmaISJKqGSKSpGqGiCSpmiEiSao24Z/H3V4RcQ3wVuCRzHxNqe0PfAU4FFgDnJyZGyNiAFgMvAV4CjgjM39Y9lkInF8Oe3FmLin1I4Frgb2Am4APZWa7W69HkvRi3bwSuRaYP6Z2HnBzZs4Gbi7rACcAs8vj/cAV8HzoXAi8ATgKuDAi9iv7XAG8r2O/seeSJHVZ10IkM78LbBhTXgAsKctLgBM76kszs52ZtwHDEXEQ8GZgRWZuyMyNwApgfnlu38y8rVx9LO04liSpR7rWnTWOAzPzwbL8EHBgWZ4JPNCx3dpS21Z97VbqL2lwcIDh4b23v+WSNIV163Ov1yHyvMxsR0TPxzBGRtps2vRUr08r9USrNdTvJmiS2tHPvfHeW72+O+vh0hVF+fORUl8HHNyx3axS21Z91lbqkqQe6nWI3AgsLMsLgRs66qdHxEBEHA08Xrq9lgPHR8R+ZUD9eGB5ee6JiDi63Nl1esexJEk90s1bfL8MHAu8MiLW0txldQlwXUScCfwMOLlsfhPN7b2raW7xfQ9AZm6IiIuAO8p2izJzdLD+LH51i+83y0OS1EMD7fb0+mrFli0jbcdEtKtqtYa4f9ER/W6GJplDLljF+vWbd+gYrdbQSmDe2LrfWJckVTNEJEnVDBFJUjVDRJJUzRCRJFUzRCRJ1QwRSVI1Q0SSVM0QkSRVM0QkSdUMEUlSNUNEklTNEJEkVTNEJEnVDBFJUjVDRJJUzRCRJFUzRCRJ1QwRSVI1Q0SSVM0QkSRVM0QkSdUMEUlSNUNEklTNEJEkVTNEJEnVduvHSSNiDbAZGAGezcx5EbE/8BXgUGANcHJmboyIAWAx8BbgKeCMzPxhOc5C4Pxy2Iszc0kPX4YkTXv9vBI5LjPnZua8sn4ecHNmzgZuLusAJwCzy+P9wBUAJXQuBN4AHAVcGBH79bD9kjTtTaburAXA6JXEEuDEjvrSzGxn5m3AcEQcBLwZWJGZGzJzI7ACmN/rRkvSdNavEGkD346IlRHx/lI7MDMfLMsPAQeW5ZnAAx37ri218eqSpB7py5gI8C8zc11EvApYERH/0PlkZrYjot2NEw8ODjA8vHc3Di1Jk1a3Pvf6EiKZua78+UhEfJ1mTOPhiDgoMx8s3VWPlM3XAQd37D6r1NYBx46pf+elzj0y0mbTpqd2+DVIk1GrNdTvJmiS2tHPvfHeWz3vzoqIl0fE0OgycDxwF3AjsLBsthC4oSzfCJweEQMRcTTweOn2Wg4cHxH7lQH140tNktQj/RgTORD43xHx98DtwN9l5reAS4A3RcT/A/6orAPcBNwHrAauBM4CyMwNwEXAHeWxqNQkST0y0G53Zehh0tqyZaRtd5Z2Va3WEPcvOqLfzdAkc8gFq1i/fvMOHaPVGloJzBtbn0y3+EqSphhDRJJUzRCRJFUzRCRJ1QwRSVI1Q0SSVM0QkSRVM0QkSdX6NQHjlLXPvnuy1x4z+t0MTTK/eHoLTz7xy343Q+o5Q2Q77bXHDI78i6X9boYmmZWfOJ0nMUQ0/didJUmqZohIkqoZIpKkaoaIJKmaISJJqmaISJKqGSKSpGqGiCSpmiEiSapmiEiSqhkikqRqhogkqZohIkmqZohIkqoZIpKkaoaIJKmaISJJqjblf9kwIuYDi4FB4KrMvKTPTZKkaWNKX4lExCDwGeAEYA5wakTM6W+rJGn6mNIhAhwFrM7M+zLzGWAZsKDPbZKkaWOqd2fNBB7oWF8LvGFbO8yYMfhoqzX0sx056cpPnL4ju2sX1WoN9bsJABxywap+N0GT0E54f/7G1opTPURqtPrdAEnaVUz17qx1wMEd67NKTZLUA1P9SuQOYHZEHEYTHqcA7+xvkyRp+pjSVyKZ+SxwDrAcuBe4LjPv7m+rJGn6GGi32/1ugyRpiprSVyKSpP4yRCRJ1ab6wLp2sogYATq/aHBiZq4ZZ9snM3OfnjRMAiLiAODmsvprwAiwvqwfVb50rB4yRDTWLzJzbr8bIW1NZj4GzAWIiI8DT2bmZaPPR8Ru5YYb9Yghom2KiH2AG4D9gBnA+Zl5w5htDgK+AuxL8576s8z8XkQcD/wlsAfwE+A9mflkL9uvXV9EXAv8EngdcGtEPEFHuETEXcBbM3NNRLwb+CCwO/AD4KzMHOlPy3cNjolorL0i4s7y+DrNP863ZebrgeOAT0bEwJh93gksL1cwrwXujIhXAucDf1T2/b/Ah3v3MjTNzAJ+LzPHfY9FxD8D/hQ4prxXR4B39ah9uyyvRDTWC7qzImIG8J8j4veB52jmKzsQeKhjnzuAa8q2f5uZd0bEH9DMrHxrREDzP7/v9+g1aPr5mwlcUfwhcCRwR3lP7gU80u2G7eoMEb2Ud9HMN3ZkZm6JiDXAnp0bZOZ3S8j8MXBtRFwObARWZOapPW6vpqd/6lh+lhf2soy+XweAJZn50Z61ahqwO0sv5RXAIyVAjmMrM3lGxG8AD2fmlcBVwOuB24BjIuLwss3LI+K3e9huTV9raN6DRMTrgcNK/WbgpIh4VXlu//Le1Q4wRPRSvgjMi4hVwOnAP2xlm2OBv4+IH9H0OS/OzPXAGcCXI+LHNF1Zv9OTFmu6+yqwf0TcTTMt0j8CZOY9NON03y7vyRXAQX1r5S7CaU8kSdW8EpEkVTNEJEnVDBFJUjVDRJJUzRCRtiEihiPirMp9z42IvSv2OyMifn1nbSd1kyEibdswUBUiwLnAdocIza3REwmHiW4ndY23+ErbEBHLgAVA0nyv4BHgZJpJJb+emRdGxMuB62jmbxoELqKZGuayst+jmXncVo49CFwNzAPawDXAA8C1wDrgF8DvAn8B/AnNNB3/B/gA8I6tbHcvMC8zH42IecBlmXlsmYJmcTltG/j9zNy8c/6GNN15JSJt23nAT8p8YiuA2cBRNNORH1mme5kP/DwzX5uZrwG+lZn/Ffg5cNzWAqSYC8zMzNdk5hHA5zPzeprJKt+VmXMz8xfAX2XmvyjH3otmRtqtbTee/wCcXV7Dv6IJHWmncO4saeKOL48flfV9aELlezSzG18KfCMzvzfB490H/GZEfBr4O+Db42x3XET8R5qusf2Bu4H/vh3tvhW4PCK+CHwtM9dux77SNnklIk3cAPBfyv/852bm4Zl5dWb+I81cTauAiyPigokcLDM30kyd/x3g39HMO/YCEbEn8FngpHK1ciVjJsDs0Dnx4PPbZOYlwL+luYq5NSKcfkY7jSEibdtmYKgsLwfeW36oi4iYGRGvKndIPZWZ/w34BGXyvzH7vkj5zZWXZeZXaeZ02tp+o2HwaDnvSeO0DZqJB48sy+/oOM9vZeaqzLyUZtp+Q0Q7jd1Z0jZk5mMRcWv5dbxvAl8Cvl9+j+JJ4N3A4cAnIuI5YAvwZ2X3zwHfioifjzMuMhP4fESM/mdudIrya4G/jojRAfMrgbtofsPljo79x273l8DVEXERzdXNqHPLDMzP0XSFfbPir0LaKu/OkiRVsztLklTN7iypByLiBzTfLel0Wmau6kd7pJ3F7ixJUjW7syRJ1QwRSVI1Q0SSVM0QkSRVM0QkSdUMEUlStf8P5YmxAEkPZFoAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UW7P24uvdxlC"
      },
      "source": [
        "So most got sent to testing, but not all got tested at the time this dataset got created. \n",
        "\n",
        "## Production Volume % Tested:  43723/48567= 0.9002614944303745 = 90%"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FlAiYU9TgoCf"
      },
      "source": [
        "# Operational Efficiency \n",
        "Look at time from creation to test. See what the mean is as baseline. Then if there is left skew, look at what caused smaller time intervals. And optimize process for that."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XaaBUgBUlUIq"
      },
      "source": [
        "df3['created_ts'] =  pd.to_datetime(df3['created_ts'],\n",
        "                              format='%Y-%m-%d %H:%M:%S')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sRc4fMr7kel_"
      },
      "source": [
        "df3['test_ts'] =  pd.to_datetime(df3['test_ts'],\n",
        "                              format='%Y-%m-%d %H:%M:%S')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "beypf5BahYTY"
      },
      "source": [
        "df3['time_diff'] = df3['test_ts'] - df3['created_ts']"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 306
        },
        "id": "D9dRXHhFYmuy",
        "outputId": "509fd2f7-37d2-4c3f-e30f-fb2ccfcca156"
      },
      "source": [
        "df3.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Unnamed: 0_x</th>\n",
              "      <th>uid</th>\n",
              "      <th>widget_type</th>\n",
              "      <th>created_ts</th>\n",
              "      <th>Unnamed: 0_y</th>\n",
              "      <th>test_ts</th>\n",
              "      <th>test_status</th>\n",
              "      <th>time_diff</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>3fc760e3-afc7-4f3b-8b17-813de9ac3e7e</td>\n",
              "      <td>A</td>\n",
              "      <td>2018-06-01 06:05:49</td>\n",
              "      <td>0</td>\n",
              "      <td>2018-06-01 06:08:11</td>\n",
              "      <td>True</td>\n",
              "      <td>0 days 00:02:22</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>86ba8559-fb1c-4178-b835-1dc8ad8be2a1</td>\n",
              "      <td>A</td>\n",
              "      <td>2018-06-01 06:11:47</td>\n",
              "      <td>1</td>\n",
              "      <td>2018-06-01 06:13:41</td>\n",
              "      <td>True</td>\n",
              "      <td>0 days 00:01:54</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2</td>\n",
              "      <td>76131121-b9ba-4d17-a600-374d5b46414a</td>\n",
              "      <td>A</td>\n",
              "      <td>2018-06-01 06:17:32</td>\n",
              "      <td>2</td>\n",
              "      <td>2018-06-01 06:19:43</td>\n",
              "      <td>True</td>\n",
              "      <td>0 days 00:02:11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3</td>\n",
              "      <td>b74b4756-8837-450c-9dc1-d4a7d78e01ea</td>\n",
              "      <td>A</td>\n",
              "      <td>2018-06-01 06:23:36</td>\n",
              "      <td>3</td>\n",
              "      <td>2018-06-01 06:25:40</td>\n",
              "      <td>True</td>\n",
              "      <td>0 days 00:02:04</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4</td>\n",
              "      <td>d54e5b6b-1bcd-4c23-95c5-f5743497f1a3</td>\n",
              "      <td>A</td>\n",
              "      <td>2018-06-01 06:29:30</td>\n",
              "      <td>4</td>\n",
              "      <td>2018-06-01 06:31:21</td>\n",
              "      <td>True</td>\n",
              "      <td>0 days 00:01:51</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Unnamed: 0_x  ...       time_diff\n",
              "0             0  ... 0 days 00:02:22\n",
              "1             1  ... 0 days 00:01:54\n",
              "2             2  ... 0 days 00:02:11\n",
              "3             3  ... 0 days 00:02:04\n",
              "4             4  ... 0 days 00:01:51\n",
              "\n",
              "[5 rows x 8 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 45
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "--R0uB0-hpvx"
      },
      "source": [
        "df3['time_diff'] = df3['time_diff'].astype(int)/ 10**9 #number of seconds to the unix epoch start (1970-01-01 00:00:00)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        },
        "id": "ckjdKxpqbt6Z",
        "outputId": "9fcff7cd-514b-4814-ae1c-78fdf2e1fd22"
      },
      "source": [
        "df3['time_diff'].hist(bins = 100)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.axes._subplots.AxesSubplot at 0x7fe2599ecda0>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 60
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAD4CAYAAAAO9oqkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAeA0lEQVR4nO3df2zTdeLH8WfpnN+Z/Sjg2oq3w+ChRxC2XI7D3eY4S9oBozJwy+WSM2Gn8SIL3ERJnEZ+TvxxnEd0CdluUTHxLgLSkaOeDIqwVfHw124HpxeJt7gltDVz60CEsfL5/jHXL/syhHXtVs7X46/P3m0/e33e7T6v9vNpO5NhGAYiIvK9NmG8A4iIyPhTGYiIiMpARERUBiIigspARESAlPEOEKsLFy4QiQy8EcpsNkWXk0my5gJli1WyZkvWXKBssUhkruuuMw87fs2WQSRi0NNzBgCL5YbocjJJ1lygbLFK1mzJmguULRaJzJWdnTHsuA4TiYiIykBERK6iDKqrq8nPz2fx4sWXXPbSSy9x++2389VXXwFgGAY1NTU4nU7cbjfHjx+PXtfj8eByuXC5XHg8nuj4sWPHcLvdOJ1Oampq0AeiRUTG3hXLYNmyZTQ0NFwyfvLkSd555x2mTJkSHWtubqa9vZ2mpiY2bdrE+vXrAejp6aG2tpYdO3awc+dOamtrCYfDAKxfv55NmzbR1NREe3s7zc3Ncdo0ERG5Wlcsgzlz5pCVlXXJ+NNPP82aNWswmUzRMZ/PR2lpKSaTiby8PHp7ewmFQvj9fgoKCrBYLGRlZVFQUEBLSwuhUIjTp0+Tl5eHyWSitLQUn88X3y0UEZEriundRAcOHMBqtfLjH/94yHgwGMRut0d/ttvtBIPBS8ZtNtuw44PXvxpmswmL5YZvlydEl5NJsuYCZYtVsmZL1lygbLEYj1wjLoNvvvmGuro6XnrppUTkuWp6a+noKFtskjVbsuYCZYvFNfHW0i+++ILOzk6WLFmCw+EgEAiwbNkyvvzyS2w2G4FAIHrdQCCAzWa7ZDwYDA47Pnh9EREZWyMug9tvv50jR45w8OBBDh48iN1uZ/fu3WRnZ+NwOGhsbMQwDFpbW8nIyMBqtVJYWIjf7yccDhMOh/H7/RQWFmK1WklPT6e1tRXDMGhsbGT+/PmJ2E4REfkOVzxMtHr1ao4ePUp3dzdFRUWsXLmS8vLyYa87b948Dh8+jNPpJC0tjc2bNwNgsVhYsWIFZWVlAFRWVmKxWABYt24d1dXVnD17lqKiIoqKiuK1bSOWnplG2vUDU/LNuX5O934zbllERMaS6Vr9T2fnz0fifs4gOzuDWx7zAtD+TAlffnlqVOtL1uORoGyxStZsyZoLlC0W18Q5AxER+e+jMhAREZWBiIioDEREBJWBiIigMhAREVQGIiKCykBERFAZiIgIKgMREUFlICIiqAxERASVgYiIoDIQERFUBiIigspARERQGYiICCoDERFBZSAiIqgMREQElYGIiHAVZVBdXU1+fj6LFy+Ojj377LMsWLAAt9tNZWUlvb290cvq6upwOp0UFxfT0tISHW9ubqa4uBin00l9fX10vKOjg/LycpxOJ1VVVfT19cVr20RE5CpdsQyWLVtGQ0PDkLGCggL27t3LX//6V2655Rbq6uoAOHHiBF6vF6/XS0NDAxs2bCASiRCJRNi4cSMNDQ14vV727t3LiRMnANiyZQvLly9n//79ZGZmsmvXrgRspoiIfJcrlsGcOXPIysoaMlZYWEhKSgoAeXl5BAIBAHw+HyUlJaSmppKTk8PUqVNpa2ujra2NqVOnkpOTQ2pqKiUlJfh8PgzD4L333qO4uBiApUuX4vP54r2NIiJyBSmjXcEbb7zBwoULAQgGg+Tm5kYvs9lsBINBAOx2+5DxtrY2uru7yczMjBaL3W6PXv9KzGYTFssN3y5PiC7H02jXmahc8aBssUnWbMmaC5QtFuORa1RlsG3bNsxmM/fcc0+88ly1SMSgp+cMMLDTHlweqfTMNNKuH34aYl3noNHkSjRli02yZkvWXKBssUhkruzsjGHHYy6D3bt3c+jQIV555RVMJhMw8Ix/8JARDLxSsNlsAMOOT5w4kd7eXvr7+0lJSSEQCESvP1bSrk/hlse8ALQ/UzKmv1tEJFnE9NbS5uZmGhoa2LZtG2lpadFxh8OB1+ulr6+Pjo4O2tvbmT17NrNmzaK9vZ2Ojg76+vrwer04HA5MJhNz585l3759AHg8HhwOR3y2TERErtoVXxmsXr2ao0eP0t3dTVFREStXrqS+vp6+vj4qKioAyM3NZePGjUyfPp2FCxeyaNEizGYza9euxWw2A7B27VoeeOABIpEI9957L9OnTwdgzZo1PPzww2zdupUZM2ZQXl6ewM0VEZHhmAzDMMY7RCzOn4/E5ZxBdnbGkMNEFy9/+eWpUWVM1uORoGyxStZsyZoLlC0W43HOQJ9AFhERlYGIiKgMREQElYGIiKAyEBERVAYiIoLKQEREUBmIiAgqAxERQWUgIiKoDEREBJWBiIigMhAREVQGIiKCykBERFAZiIgIKgMREUFlICIiqAxERASVgYiIoDIQERGuogyqq6vJz89n8eLF0bGenh4qKipwuVxUVFQQDocBMAyDmpoanE4nbreb48ePR2/j8XhwuVy4XC48Hk90/NixY7jdbpxOJzU1NRiGEc/tExGRq3DFMli2bBkNDQ1Dxurr68nPz6epqYn8/Hzq6+sBaG5upr29naamJjZt2sT69euBgfKora1lx44d7Ny5k9ra2miBrF+/nk2bNtHU1ER7ezvNzc1x3kQREbmSK5bBnDlzyMrKGjLm8/koLS0FoLS0lAMHDgwZN5lM5OXl0dvbSygUwu/3U1BQgMViISsri4KCAlpaWgiFQpw+fZq8vDxMJhOlpaX4fL4EbKaIiHyXlFhu1NXVhdVqBSA7O5uuri4AgsEgdrs9ej273U4wGLxk3GazDTs+eP2rYTabsFhu+HZ5QnQ5nka7zkTligdli02yZkvWXKBssRiPXDGVwcVMJhMmkykeWUYkEjHo6TkDDOy0B5dHKjs747KXxbrOQaPJlWjKFptkzZasuUDZYpHIXJfb58X0bqLJkycTCoUACIVCTJo0CRh4xh8IBKLXCwQC2Gy2S8aDweCw44PXFxGRsRVTGTgcDhobGwFobGxk/vz5Q8YNw6C1tZWMjAysViuFhYX4/X7C4TDhcBi/309hYSFWq5X09HRaW1sxDGPIukREZOxc8TDR6tWrOXr0KN3d3RQVFbFy5UoefPBBqqqq2LVrF1OmTGHr1q0AzJs3j8OHD+N0OklLS2Pz5s0AWCwWVqxYQVlZGQCVlZVYLBYA1q1bR3V1NWfPnqWoqIiioqJEbauIiFzGFcvg+eefH3Z8+/btl4yZTCbWrVs37PXLysqiZXCxWbNmsXfv3ivFEBGRBNInkEVERGUgIiIqAxERQWUgIiKoDEREBJWBiIigMhAREVQGIiKCykBERFAZiIgIKgMREUFlICIiqAxERASVgYiIoDIQERFUBiIigspARERQGYiICCoDERFBZSAiIqgMRESEUZbBK6+8QklJCYsXL2b16tWcO3eOjo4OysvLcTqdVFVV0dfXB0BfXx9VVVU4nU7Ky8vp7OyMrqeurg6n00lxcTEtLS2j2yIRERmxmMsgGAzy6quv8sYbb7B3714ikQher5ctW7awfPly9u/fT2ZmJrt27QJg586dZGZmsn//fpYvX86WLVsAOHHiBF6vF6/XS0NDAxs2bCASicRn60RE5KqM6pVBJBLh7Nmz9Pf3c/bsWbKzs3nvvfcoLi4GYOnSpfh8PgAOHjzI0qVLASguLubIkSMYhoHP56OkpITU1FRycnKYOnUqbW1to9wsEREZiZRYb2iz2fjNb37D3XffzfXXX09BQQEzZ84kMzOTlJSB1drtdoLBIDDwSuKmm24a+KUpKWRkZNDd3U0wGCQ3N3fIegdv813MZhMWyw3fLk+ILsfTaNeZqFzxoGyxSdZsyZoLlC0W45Er5jIIh8P4fD58Ph8ZGRn87ne/G9Pj/ZGIQU/PGWBgpz24PFLZ2RmXvSzWdQ4aTa5EU7bYJGu2ZM0FyhaLROa63D4v5sNE7777Lj/4wQ+YNGkS1113HS6Xi48++oje3l76+/sBCAQC2Gw2YOAZ/8mTJwHo7+/n1KlTTJw4EZvNRiAQiK43GAxGbyMiImMj5jKYMmUK//jHP/jmm28wDIMjR47wox/9iLlz57Jv3z4APB4PDocDAIfDgcfjAWDfvn3ceeedmEwmHA4HXq+Xvr4+Ojo6aG9vZ/bs2XHYNBERuVoxHybKzc2luLiYpUuXkpKSwowZM/jlL3/JL37xCx5++GG2bt3KjBkzKC8vB6CsrIw1a9bgdDrJysrij3/8IwDTp09n4cKFLFq0CLPZzNq1azGbzfHZOhERuSomwzCM8Q4Ri/PnI3E7Z3DLY14A2p8pGbL85ZenRpUxWY9HgrLFKlmzJWsuULZYXFPnDERE5L+HykBERFQGIiKiMhAREVQGIiKCykBERFAZiIgIKgMREUFlICIiqAxERASVgYiIoDIQERFUBiIigspARERQGYiICCoDERFBZSAiIqgMREQElYGIiKAyEBERVAYiIsIoy6C3t5dVq1axYMECFi5cyMcff0xPTw8VFRW4XC4qKioIh8MAGIZBTU0NTqcTt9vN8ePHo+vxeDy4XC5cLhcej2d0WyQiIiM2qjJ46qmnuOuuu3jrrbfYs2cPt956K/X19eTn59PU1ER+fj719fUANDc3097eTlNTE5s2bWL9+vUA9PT0UFtby44dO9i5cye1tbXRAhERkbERcxmcOnWK999/n7KyMgBSU1PJzMzE5/NRWloKQGlpKQcOHACIjptMJvLy8ujt7SUUCuH3+ykoKMBisZCVlUVBQQEtLS1x2DQREblaKbHesLOzk0mTJlFdXc2nn37KzJkzeeKJJ+jq6sJqtQKQnZ1NV1cXAMFgELvdHr293W4nGAxeMm6z2QgGg7HGEhGRGMRcBv39/fzrX//iySefJDc3l5qamughoUEmkwmTyTTqkMMxm01YLDd8uzwhuhxPo11nonLFg7LFJlmzJWsuULZYjEeumMvAbrdjt9vJzc0FYMGCBdTX1zN58mRCoRBWq5VQKMSkSZOAgWf8gUAgevtAIIDNZsNms3H06NHoeDAY5Gc/+9kVf38kYtDTcwYY2GkPLo9UdnbGZS+LdZ2DRpMr0ZQtNsmaLVlzgbLFIpG5LrfPi/mcQXZ2Nna7nc8//xyAI0eOcOutt+JwOGhsbASgsbGR+fPnA0THDcOgtbWVjIwMrFYrhYWF+P1+wuEw4XAYv99PYWFhrLFERCQGMb8yAHjyySd59NFHOX/+PDk5OTz99NNcuHCBqqoqdu3axZQpU9i6dSsA8+bN4/DhwzidTtLS0ti8eTMAFouFFStWRE9EV1ZWYrFYRrlZIiIyEqMqgxkzZrB79+5Lxrdv337JmMlkYt26dcOup6ysLFoGIiIy9vQJZBERURmIiIjKQEREUBmIiAgqAxERQWUgIiKoDEREBJWBiIigMhAREVQGIiKCykBERFAZiIgIKgMREUFlICIiqAxERASVgYiIoDIQERFUBiIigspARERQGYiICCoDEREhDmUQiUQoLS3lt7/9LQAdHR2Ul5fjdDqpqqqir68PgL6+PqqqqnA6nZSXl9PZ2RldR11dHU6nk+LiYlpaWkYbSURERmjUZfDqq69y6623Rn/esmULy5cvZ//+/WRmZrJr1y4Adu7cSWZmJvv372f58uVs2bIFgBMnTuD1evF6vTQ0NLBhwwYikchoY4mIyAiMqgwCgQCHDh2irKwMAMMweO+99yguLgZg6dKl+Hw+AA4ePMjSpUsBKC4u5siRIxiGgc/no6SkhNTUVHJycpg6dSptbW2jiSUiIiM0qjLYvHkza9asYcKEgdV0d3eTmZlJSkoKAHa7nWAwCEAwGOSmm24CICUlhYyMDLq7uwkGg9jt9ug6bTZb9DYiIjI2UmK94dtvv82kSZO44447+Pvf/x7PTFfFbDZhsdzw7fKE6HI8jXadicoVD8oWm2TNlqy5QNliMR65Yi6Djz76iIMHD9Lc3My5c+c4ffo0Tz31FL29vfT395OSkkIgEMBmswEDz/hPnjyJ3W6nv7+fU6dOMXHiRGw2G4FAILreYDAYvc13iUQMenrOAAM77cHlkcrOzrjsZbGuc9BociWassUmWbMlay5QtlgkMtfl9nkxHyZ65JFHaG5u5uDBgzz//PPceeed/OEPf2Du3Lns27cPAI/Hg8PhAMDhcODxeADYt28fd955JyaTCYfDgdfrpa+vj46ODtrb25k9e3assUREJAZx/5zBmjVrePnll3E6nfT09FBeXg5AWVkZPT09OJ1OXn75ZR599FEApk+fzsKFC1m0aBEPPPAAa9euxWw2xzuWiIh8h5gPE11s7ty5zJ07F4CcnJzo20kvdv311/PCCy8Me/uHHnqIhx56KB5RREQkBvoEsoiIqAxERERlICIiqAxERASVgYiIoDIQERFUBiIigspARERQGYiICCoDERFBZSAiIqgMREQElYGIiKAyEBERVAYiIoLKQEREUBmIiAgqAxERQWUgIiKoDEREBJWBiIigMhAREUZRBidPnuS+++5j0aJFlJSUsH37dgB6enqoqKjA5XJRUVFBOBwGwDAMampqcDqduN1ujh8/Hl2Xx+PB5XLhcrnweDyj3CQRERmpmMvAbDbz2GOP8eabb/L666/z5z//mRMnTlBfX09+fj5NTU3k5+dTX18PQHNzM+3t7TQ1NbFp0ybWr18PDJRHbW0tO3bsYOfOndTW1kYLRERExkbMZWC1Wpk5cyYA6enpTJs2jWAwiM/no7S0FIDS0lIOHDgAEB03mUzk5eXR29tLKBTC7/dTUFCAxWIhKyuLgoICWlpa4rBpIiJytVLisZLOzk4++eQTcnNz6erqwmq1ApCdnU1XVxcAwWAQu90evY3dbicYDF4ybrPZCAaDV/ydZrMJi+WGb5cnRJfjabTrTFSueFC22CRrtmTNBcoWi/HINeoy+Prrr1m1ahWPP/446enpQy4zmUyYTKbR/ophRSIGPT1ngIGd9uDySGVnZ1z2sljXmZ6ZRtr1A1N77rzB6d5vYlpPIo1mzhJN2UYuWXOBssUikbkut88b1buJzp8/z6pVq3C73bhcLgAmT55MKBQCIBQKMWnSJGDgGX8gEIjeNhAIYLPZLhkPBoPYbLbRxBp3adencMtjXm55zBstBRGRZBZzGRiGwRNPPMG0adOoqKiIjjscDhobGwFobGxk/vz5Q8YNw6C1tZWMjAysViuFhYX4/X7C4TDhcBi/309hYeEoN0tEREYi5qetH374IXv27OG2225jyZIlAKxevZoHH3yQqqoqdu3axZQpU9i6dSsA8+bN4/DhwzidTtLS0ti8eTMAFouFFStWUFZWBkBlZSUWi2W02yUiIiMQcxn89Kc/5d///vewlw1+5uBiJpOJdevWDXv9srKyaBmIiMjY0wHtOLn4pLGIyLVGe6/LOHs+Ej3r/s25/iu+I2jwpDFA+zMlCc8nIhJPKoPL+J/rzEN27qfHOY+ISCLpi+pERERlICIiKgMREUFlICIiqAxERASVgYiIoDIQERFUBiIigspARERQGYiICPo6ioQb6XcciYiMB5VBguk7jkTkWvC9LAN93bSIyFDfy3MGF/+PYhER+Z6WgYiIDKUyEBERlYGIiHxPTyDHi05Ei8h/i6R5ZdDc3ExxcTFOp5P6+vrxjnNVdCJaRP5bJEUZRCIRNm7cSENDA16vl71793LixInxjiUi8r2RFGXQ1tbG1KlTycnJITU1lZKSEnw+33jHirvBTyNnZ2eQnpk23nFERKJMhmEY4x3irbfeoqWlhaeeegqAxsZG2traWLt27TgnExH5fkiKVwYiIjK+kqIMbDYbgUAg+nMwGMRms41jIhGR75ekKINZs2bR3t5OR0cHfX19eL1eHA7HeMcSEfneSIo3yaekpLB27VoeeOABIpEI9957L9OnTx/vWCIi3xtJcQJZRETGV1IcJhIRkfGlMhARkWu7DMb7KyxOnjzJfffdx6JFiygpKWH79u0AvPjii9x1110sWbKEJUuWcPjw4eht6urqcDqdFBcX09LSktB8DocDt9vNkiVLWLZsGQA9PT1UVFTgcrmoqKggHA4DYBgGNTU1OJ1O3G43x48fT0imzz//PDovS5Ys4Sc/+QmvvPLKuM1ZdXU1+fn5LF68ODoWyxx5PB5cLhculwuPx5OwbM8++ywLFizA7XZTWVlJb28vAJ2dncyePTs6fxd/RufYsWO43W6cTic1NTXE48jwcNliuQ/j/Tc8XK6qqqpoJofDwZIlS4Cxn7PL7S+S5fGGcY3q7+835s+fb3zxxRfGuXPnDLfbbXz22WdjmiEYDBrHjh0zDMMwTp06ZbhcLuOzzz4zXnjhBaOhoeGS63/22WeG2+02zp07Z3zxxRfG/Pnzjf7+/oTlu/vuu42urq4hY88++6xRV1dnGIZh1NXVGc8995xhGIZx6NAh4/777zcuXLhgfPzxx0ZZWVnCcg3q7+83fv7znxudnZ3jNmdHjx41jh07ZpSUlETHRjpH3d3dhsPhMLq7u42enh7D4XAYPT09CcnW0tJinD9/3jAMw3juueei2To6OoZc72L33nuv8fHHHxsXLlww7r//fuPQoUMJyTbS+zARf8PD5brY008/bbz44ouGYYz9nF1uf5Esj7dr9pVBMnyFhdVqZebMmQCkp6czbdo0gsHgZa/v8/koKSkhNTWVnJwcpk6dSltb21jFjWYoLS0FoLS0lAMHDgwZN5lM5OXl0dvbSygUSmiWI0eOkJOTw8033/ydeRM5Z3PmzCErK+uS3zmSOfL7/RQUFGCxWMjKyqKgoCAur2CGy1ZYWEhKysCbAPPy8oZ8Pmc4oVCI06dPk5eXh8lkorS0NC5/J8Nlu5zL3YeJ+Bv+rlyGYfC3v/1tyKuG4SRqzi63v0iWx9s1WwbBYBC73R792WazfeeOONE6Ozv55JNPyM3NBeC1117D7XZTXV0dfdk3Hpnvv/9+li1bxuuvvw5AV1cXVqsVgOzsbLq6uobNZrfbE57N6/UO+cNMljkb6RyN12PxjTfeoKioKPpzZ2cnpaWl/PrXv+aDDz74zsyJMpL7cKzn7YMPPmDy5Mnccsst0bHxmrOL9xfJ8ni7ZssgmXz99desWrWKxx9/nPT0dH71q1+xf/9+9uzZg9Vq5ZlnnhmXXH/5y1/weDz86U9/4rXXXuP9998fcrnJZMJkMo1Ltr6+Pg4ePMiCBQsAkmbO/r/xnKPvsm3bNsxmM/fccw8w8Kzz7bffprGxkccee4xHHnmE06dPj2mmZL0PB+3du3fIk4/xmrP/v7+42Hg+3q7ZMkiWr7A4f/48q1atwu1243K5ALjxxhsxm81MmDCB8vJy/vnPf45L5sF1T548GafTSVtbG5MnT44e/gmFQkyaNGnYbIFAIKHZmpubmTlzJjfeeCOQPHMGjHiOxjrj7t27OXToEFu2bInuOFJTU5k4cSIAd9xxBz/84Q/5z3/+M6b360jvw7Gct/7+fvbv38+iRYuiY+MxZ8PtL5Ll8XbNlkEyfIWFYRg88cQTTJs2jYqKiuj4xcfaDxw4EP00tcPhwOv10tfXR0dHB+3t7cyePTsh2c6cORN9lnPmzBneeecdpk+fjsPhoLGxERj4dtj58+dHszU2NmIYBq2trWRkZERfuiaC1+ulpKQk+nMyzNmgkc5RYWEhfr+fcDhMOBzG7/dTWFiYkGzNzc00NDSwbds20tL+72vQv/rqKyKRCEB0nnJycrBaraSnp9Pa2ophGEO2J95Geh+O5d/wu+++y7Rp04YcXhnrObvc/iJZHm9J8XUUsUiGr7D48MMP2bNnD7fddlv07WqrV69m7969fPrppwDcfPPNbNy4EYDp06ezcOFCFi1ahNlsZu3atZjN5oRk6+rqorKyEhj450GLFy+mqKiIWbNmUVVVxa5du5gyZQpbt24FYN68eRw+fBin00laWhqbN29OSC4YKKd33303Oi8Av//978dlzlavXs3Ro0fp7u6mqKiIlStX8uCDD45ojiwWCytWrKCsrAyAyspKLBZLQrLV19fT19cX3Znk5uayceNG3n//fV544QVSUlKYMGECGzZsiGZYt24d1dXVnD17lqKioiHnGeKZ7ejRoyO+D+P9NzxcrvLyct58880hTz6AMZ+zy+0vkuXxpq+jEBGRa/cwkYiIxI/KQEREVAYiIqIyEBERVAYiIoLKQEREUBmIiAjwv4tfNmh3tUYdAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IL7B9K26hYYV",
        "outputId": "d43e2873-c2b8-44a1-8bab-c8979fecb5a1"
      },
      "source": [
        "df3['time_diff'].value_counts()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "119.0    1101\n",
              "121.0    1014\n",
              "122.0    1009\n",
              "120.0     981\n",
              "124.0     971\n",
              "         ... \n",
              "157.0       1\n",
              "194.0       1\n",
              "187.0       1\n",
              "195.0       1\n",
              "193.0       1\n",
              "Name: time_diff, Length: 189, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 61
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yAuiLfRagpFR"
      },
      "source": [
        "We can see here that most of the lag between creation and tested is 119 seconds or around 2 minutes. At the same time there are some values that are less that. These would be the target values to optimize production on.   "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QqYAj6a0d31N"
      },
      "source": [
        "________________________________________________________________________________"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "trcz955gd8vd"
      },
      "source": [
        "# *Below is data pipeline used to analyze numerical and categorical variables as well as any associations between them.*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nI3_6V5i75-6"
      },
      "source": [
        "# For zip files."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jAiHO2Nx4Ys2",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e2bce8bb-2831-469a-d816-2f1d10f4d596"
      },
      "source": [
        "!wget http://files.grouplens.org/datasets/movielens/ml-latest.zip"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2020-11-23 23:15:49--  http://files.grouplens.org/datasets/movielens/ml-latest.zip\n",
            "Resolving files.grouplens.org (files.grouplens.org)... 128.101.65.152\n",
            "Connecting to files.grouplens.org (files.grouplens.org)|128.101.65.152|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 277113433 (264M) [application/zip]\n",
            "Saving to: ‘ml-latest.zip.1’\n",
            "\n",
            "ml-latest.zip.1     100%[===================>] 264.28M  16.9MB/s    in 17s     \n",
            "\n",
            "2020-11-23 23:16:07 (15.7 MB/s) - ‘ml-latest.zip.1’ saved [277113433/277113433]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TVOMLDBA9F_j",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ab3f5273-4cdc-47b8-ead8-66cf66df5399"
      },
      "source": [
        "!unzip ml-latest.zip"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Archive:  ml-latest.zip\n",
            "replace ml-latest/links.csv? [y]es, [n]o, [A]ll, [N]one, [r]ename: "
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1kAWMlQP9SFS"
      },
      "source": [
        "!ls"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-aY0NVLs9Vt8"
      },
      "source": [
        "df2 = pd.read_csv('ml-latest/ratings.csv')\n",
        "df2.head()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ajc8NBm94n_2"
      },
      "source": [
        "# EDA Pipeline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aX349exLDNkc"
      },
      "source": [
        "## Numerical Variables"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uPUoQ95tDGcl"
      },
      "source": [
        "For numerical features, we are always concerned about the distribution of these features, including the statistical characteristics of these columns e.g mean, median, mode. Hence we will usually use Distribution plot to visualize their data distribution. Boxplots are also commonly used to unearth the statistical characteristics of each feature. More often than not, we use it to look for any outliers that we might need to filter out later on."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Nq565si3-DgE"
      },
      "source": [
        "numeric_ = df.select_dtypes(exclude=['object']).copy()\n",
        "numeric_.columns"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_F7uZS_G46hT"
      },
      "source": [
        "disc_num = ['Postal Code', 'Row ID']"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m2ze0sQX3xew"
      },
      "source": [
        "cont_num = []\n",
        "for i in numeric_.columns:\n",
        "    if i not in disc_num:\n",
        "        cont_num.append(i)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tyJGe6my3oGY"
      },
      "source": [
        "cat = df.select_dtypes(include=['object']).copy()\n",
        "cat.columns"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9Vr6pDlC3p6C"
      },
      "source": [
        "#Continuous Variables\n",
        "\n",
        "fig = plt.figure(figsize=(18,16))\n",
        "for index,col in enumerate(cont_num):\n",
        "    plt.subplot(6,4,index+1)\n",
        "    sns.distplot(numeric_.loc[:,col].dropna(), kde=False)\n",
        "fig.tight_layout(pad=1.0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bcjBd57jDXeq"
      },
      "source": [
        "Variables with mostly or 1 as their value:\n",
        "\n",
        "1. Customers\n",
        "2. Number of Records\n",
        "2. Profit\n",
        "\n",
        "INFORMATIVE? \n",
        "These are not the most informative, but good to note thier lack of variance.\n",
        "\n",
        "SKEWNESS? \n",
        "Also, there is some skew in Quantity."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MzM1DgEs4Shg"
      },
      "source": [
        "#Continuous Variables\n",
        "\n",
        "fig = plt.figure(figsize=(14,15))\n",
        "\n",
        "for index,col in enumerate(cont_num):\n",
        "    plt.subplot(6,4,index+1)\n",
        "    sns.boxplot(y=col, data=numeric_.dropna())\n",
        "fig.tight_layout(pad=1.0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ahwtwBRAEAt_"
      },
      "source": [
        "A lot of 0's and near 0 values in Profit and Sales.\n",
        "\n",
        "Skewness in upper region.\n",
        "\n",
        "Most normal looking distribution is Profit."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pONiL6Mb4U_9"
      },
      "source": [
        "#Discrete Number\n",
        "\n",
        "sns.countplot(x='Postal Code', data=df)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RKd9fvvV4m3R"
      },
      "source": [
        "# Discrete Variables\n",
        "\n",
        "# sns.countplot(x='Row ID', data=df)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q19GBCzi4sVk"
      },
      "source": [
        "# Categorical Variables"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MVZqwRhAE2jB"
      },
      "source": [
        "In the case of categorical features, we will often use countplots to visualize the count of each distinct value within each features. We can see that categorical feature Country/Region consist of mainly one value, which is not very useful information."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xWJWmQJ85s2V"
      },
      "source": [
        "list(cat)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CE0xEpTJ5B1n"
      },
      "source": [
        "sns.countplot(x=\"Category\", data=cat)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_d7TWrXX5zwA"
      },
      "source": [
        "sns.countplot(x=\"City\", data=cat)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AOivEL9f50HY"
      },
      "source": [
        "sns.countplot(x=\"Country/Region\", data=cat)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KSrYcNkn5z67"
      },
      "source": [
        "sns.countplot(x=\"Product Name\", data=cat)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eFVuJdG95zNx"
      },
      "source": [
        "sns.countplot(x=\"Region\", data=cat)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0bci0QQt5zB4"
      },
      "source": [
        "sns.countplot(x=\"Segment\", data=cat)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kBGt8WH76NV3"
      },
      "source": [
        "sns.countplot(x=\"Ship Mode\", data=cat)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Dk9SZaj06Qnm"
      },
      "source": [
        "sns.countplot(x=\"State\", data=cat)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c05IH-zr6Qdi"
      },
      "source": [
        "sns.countplot(x=\"Sub-Category\", data=cat)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R42wm2C8GBqh"
      },
      "source": [
        "Univariate Analysis helps us to understand all the features better, on an individual scale. To further deepen our insights and uncover potential pattern in the data, we will also need to find out more about the relationship between all these features with one another, which brings us to our next step in our analysis - Bivariate Analysis"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X_k9Pspa4JCT"
      },
      "source": [
        "# Bi-Variate Analysis"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z2HqaFlFGMJx"
      },
      "source": [
        "Bi-variate analysis looks at 2 different features to identify any possible relationship or distinctive patterns between the 2 features. One of the commonly used technique is through the Correlation Matrix. Correlation matrix is an effective tool to uncover linear relationship (Correlation) between any 2 continuous features. Correlation not only allow us to determine which features are important to Sales, but also as a means to investigate any multicollinearity between our independent predictors.\n",
        "Multicollinearity happens when 2 or more independent variables are highly correlated with one another. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yG-tW77D3-x1"
      },
      "source": [
        "sns.set(font_scale=1.1)\n",
        "correlation_df = df.corr()\n",
        "mask = np.triu(correlation_df.corr())\n",
        "plt.figure(figsize=(20, 20))\n",
        "sns.heatmap(correlation_df,\n",
        "            annot=True,\n",
        "            fmt='.1f',\n",
        "            cmap='coolwarm',\n",
        "            square=True,\n",
        "            mask=mask,\n",
        "            linewidths=1,\n",
        "            cbar=False)\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3N3Zcslt7IIk"
      },
      "source": [
        "Highly correlated variables:\n",
        "\n",
        "1. Sales and Profit\n",
        "2. Quantity and Sales\n",
        "3. Quantity and Profit\n",
        "\n",
        "RECLASSIFY SIMILAR VARIABLES? At this point, can also determine similar variables that could be reclassified into one to avoid redundancy in dataset as well as possible multicollinearity."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_eRn15E5Gk7y"
      },
      "source": [
        "## Identifying Relationship between Variables (Ind and Dep (Sales))\n",
        "\n",
        "INFORMING BUSINESS STRATEGY:\n",
        "\n",
        "Below, we sorted the strength of linear relationship between Sales and other variables. Overall, Profit and Quantity have the strongest linear relationship with Sales."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tLIpfB4f60Qw"
      },
      "source": [
        "numeric_ = df.select_dtypes(exclude=['object'])\n",
        "correlation = numeric_.corr()\n",
        "correlation[['Sales']].sort_values(['Sales'], ascending=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gD_lnC8gHYty"
      },
      "source": [
        "## Scatterplots\n",
        "Using scatterplot can also help us to identify potential linear relationship between Numerical features. Although scatterplot does not provide quantitative evidence on the strength of linear relationship between our features, it is useful in helping us to visualize any sort of relationship that correlation matrix could not calculate. E.g Quadratic, Exponential relationships."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wfzr8yAP8BfN"
      },
      "source": [
        "fig = plt.figure(figsize=(20,20))\n",
        "for index in range(len(numeric_.columns)):\n",
        "    plt.subplot(10,5,index+1)\n",
        "    sns.scatterplot(x=numeric_.iloc[:,index], y='Sales', data=numeric_.dropna())\n",
        "fig.tight_layout(pad=1.0)"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}