2.4 Deteccion outlieres Azure- Función.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicción de la producción de activos industriales utilizando el algoritmo de detección de anomalías de Azure"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Cargar librerías"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import math\n",
    "\n",
    "from azure.cognitiveservices.anomalydetector import AnomalyDetectorClient\n",
    "from azure.cognitiveservices.anomalydetector.models import Request, Point, Granularity, \\\n",
    "    APIErrorException\n",
    "from msrest.authentication import CognitiveServicesCredentials\n",
    "import os\n",
    "\n",
    "from sklearn.metrics import mean_squared_error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Conexión con Azure y autentificación del cliente"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# URLs para la detección de anomalías\n",
    "batch_detection_url = \"/anomalydetector/v1.0/timeseries/entire/detect\"\n",
    "\n",
    "# Se debe crear la variable de entorno y el endpoint correspondiente según la sinstrucciones de Azure\n",
    "endpoint = os.environ[\"ANOMALY_DETECTOR_ENDPOINT\"]\n",
    "subscription_key = os.environ[\"ANOMALY_DETECTOR_KEY\"]\n",
    "\n",
    "#Autentificación del cliente\n",
    "client=AnomalyDetectorClient(endpoint, CognitiveServicesCredentials(subscription_key))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Lectura de datos y preparación"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SAMPLE_ID</th>\n",
       "      <th>PRODUCTION_GROUP_2</th>\n",
       "      <th>PRODUCTION_GROUP_3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>3850220.0</td>\n",
       "      <td>9019860.320</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>3793240.0</td>\n",
       "      <td>8568867.304</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>3870570.0</td>\n",
       "      <td>8549258.912</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>3801380.0</td>\n",
       "      <td>8451216.952</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>3752540.0</td>\n",
       "      <td>8323762.404</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>100</td>\n",
       "      <td>3984530.0</td>\n",
       "      <td>9372811.376</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <td>101</td>\n",
       "      <td>3976390.0</td>\n",
       "      <td>9588503.688</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>101</th>\n",
       "      <td>102</td>\n",
       "      <td>3903130.0</td>\n",
       "      <td>9598307.884</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102</th>\n",
       "      <td>103</td>\n",
       "      <td>3927550.0</td>\n",
       "      <td>9745370.824</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>103</th>\n",
       "      <td>104</td>\n",
       "      <td>3951970.0</td>\n",
       "      <td>9578699.492</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>104 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     SAMPLE_ID  PRODUCTION_GROUP_2  PRODUCTION_GROUP_3\n",
       "0            1           3850220.0         9019860.320\n",
       "1            2           3793240.0         8568867.304\n",
       "2            3           3870570.0         8549258.912\n",
       "3            4           3801380.0         8451216.952\n",
       "4            5           3752540.0         8323762.404\n",
       "..         ...                 ...                 ...\n",
       "99         100           3984530.0         9372811.376\n",
       "100        101           3976390.0         9588503.688\n",
       "101        102           3903130.0         9598307.884\n",
       "102        103           3927550.0         9745370.824\n",
       "103        104           3951970.0         9578699.492\n",
       "\n",
       "[104 rows x 3 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y_train=pd.read_csv(('Datos/train_y.csv'))\n",
    "Y_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>asset_id</th>\n",
       "      <th>group_id</th>\n",
       "      <th>week</th>\n",
       "      <th>weekday</th>\n",
       "      <th>t1</th>\n",
       "      <th>t2</th>\n",
       "      <th>t3</th>\n",
       "      <th>t4</th>\n",
       "      <th>total_t</th>\n",
       "      <th>week_day</th>\n",
       "      <th>date</th>\n",
       "      <th>day</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>19.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1_1</td>\n",
       "      <td>2018-01-01</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1_2</td>\n",
       "      <td>2018-01-02</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>9.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1_3</td>\n",
       "      <td>2018-01-03</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1_4</td>\n",
       "      <td>2018-01-04</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1_5</td>\n",
       "      <td>2018-01-05</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60419</th>\n",
       "      <td>83</td>\n",
       "      <td>3</td>\n",
       "      <td>104</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>48.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>4</td>\n",
       "      <td>104_3</td>\n",
       "      <td>2019-12-25</td>\n",
       "      <td>724</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60420</th>\n",
       "      <td>83</td>\n",
       "      <td>3</td>\n",
       "      <td>104</td>\n",
       "      <td>4</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>37.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4</td>\n",
       "      <td>104_4</td>\n",
       "      <td>2019-12-26</td>\n",
       "      <td>725</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60421</th>\n",
       "      <td>83</td>\n",
       "      <td>3</td>\n",
       "      <td>104</td>\n",
       "      <td>5</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>48.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>4</td>\n",
       "      <td>104_5</td>\n",
       "      <td>2019-12-27</td>\n",
       "      <td>726</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60422</th>\n",
       "      <td>83</td>\n",
       "      <td>3</td>\n",
       "      <td>104</td>\n",
       "      <td>6</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>53.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>4</td>\n",
       "      <td>104_6</td>\n",
       "      <td>2019-12-28</td>\n",
       "      <td>727</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60423</th>\n",
       "      <td>83</td>\n",
       "      <td>3</td>\n",
       "      <td>104</td>\n",
       "      <td>7</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>61.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4</td>\n",
       "      <td>104_7</td>\n",
       "      <td>2019-12-29</td>\n",
       "      <td>728</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>60424 rows × 12 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       asset_id  group_id  week  weekday   t1   t2    t3    t4  total_t  \\\n",
       "0             1         2     1        1  NaN  NaN  19.0   0.0        2   \n",
       "1             1         2     1        2  NaN  NaN   8.0   0.0        2   \n",
       "2             1         2     1        3  NaN  NaN   9.0   0.0        2   \n",
       "3             1         2     1        4  NaN  NaN   6.0   0.0        2   \n",
       "4             1         2     1        5  NaN  NaN   6.0   2.0        2   \n",
       "...         ...       ...   ...      ...  ...  ...   ...   ...      ...   \n",
       "60419        83         3   104        3  1.0  0.0  48.0   8.0        4   \n",
       "60420        83         3   104        4  0.0  0.0  37.0   1.0        4   \n",
       "60421        83         3   104        5  1.0  0.0  48.0   9.0        4   \n",
       "60422        83         3   104        6  2.0  0.0  53.0  11.0        4   \n",
       "60423        83         3   104        7  1.0  0.0  61.0   0.0        4   \n",
       "\n",
       "      week_day        date  day  \n",
       "0          1_1  2018-01-01    1  \n",
       "1          1_2  2018-01-02    2  \n",
       "2          1_3  2018-01-03    3  \n",
       "3          1_4  2018-01-04    4  \n",
       "4          1_5  2018-01-05    5  \n",
       "...        ...         ...  ...  \n",
       "60419    104_3  2019-12-25  724  \n",
       "60420    104_4  2019-12-26  725  \n",
       "60421    104_5  2019-12-27  726  \n",
       "60422    104_6  2019-12-28  727  \n",
       "60423    104_7  2019-12-29  728  \n",
       "\n",
       "[60424 rows x 12 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "data=pd.read_csv('Output/X_train_formateado.csv')  #Los datos originados fueron preprocesados en el archivo 1. Detección\n",
    "                                                   #de outiliers Holt-Winter\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>asset_id</th>\n",
       "      <th>group_id</th>\n",
       "      <th>week</th>\n",
       "      <th>weekday</th>\n",
       "      <th>t1</th>\n",
       "      <th>t2</th>\n",
       "      <th>t3</th>\n",
       "      <th>t4</th>\n",
       "      <th>total_t</th>\n",
       "      <th>week_day</th>\n",
       "      <th>date</th>\n",
       "      <th>day</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>19.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1_1</td>\n",
       "      <td>2018-01-01 00:00:00+00:00</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1_2</td>\n",
       "      <td>2018-01-02 00:00:00+00:00</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>9.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1_3</td>\n",
       "      <td>2018-01-03 00:00:00+00:00</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1_4</td>\n",
       "      <td>2018-01-04 00:00:00+00:00</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1_5</td>\n",
       "      <td>2018-01-05 00:00:00+00:00</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60419</th>\n",
       "      <td>83</td>\n",
       "      <td>3</td>\n",
       "      <td>104</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>48.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>4</td>\n",
       "      <td>104_3</td>\n",
       "      <td>2019-12-25 00:00:00+00:00</td>\n",
       "      <td>724</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60420</th>\n",
       "      <td>83</td>\n",
       "      <td>3</td>\n",
       "      <td>104</td>\n",
       "      <td>4</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>37.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4</td>\n",
       "      <td>104_4</td>\n",
       "      <td>2019-12-26 00:00:00+00:00</td>\n",
       "      <td>725</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60421</th>\n",
       "      <td>83</td>\n",
       "      <td>3</td>\n",
       "      <td>104</td>\n",
       "      <td>5</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>48.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>4</td>\n",
       "      <td>104_5</td>\n",
       "      <td>2019-12-27 00:00:00+00:00</td>\n",
       "      <td>726</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60422</th>\n",
       "      <td>83</td>\n",
       "      <td>3</td>\n",
       "      <td>104</td>\n",
       "      <td>6</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>53.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>4</td>\n",
       "      <td>104_6</td>\n",
       "      <td>2019-12-28 00:00:00+00:00</td>\n",
       "      <td>727</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60423</th>\n",
       "      <td>83</td>\n",
       "      <td>3</td>\n",
       "      <td>104</td>\n",
       "      <td>7</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>61.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4</td>\n",
       "      <td>104_7</td>\n",
       "      <td>2019-12-29 00:00:00+00:00</td>\n",
       "      <td>728</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>60424 rows × 12 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       asset_id  group_id  week  weekday   t1   t2    t3    t4  total_t  \\\n",
       "0             1         2     1        1  NaN  NaN  19.0   0.0        2   \n",
       "1             1         2     1        2  NaN  NaN   8.0   0.0        2   \n",
       "2             1         2     1        3  NaN  NaN   9.0   0.0        2   \n",
       "3             1         2     1        4  NaN  NaN   6.0   0.0        2   \n",
       "4             1         2     1        5  NaN  NaN   6.0   2.0        2   \n",
       "...         ...       ...   ...      ...  ...  ...   ...   ...      ...   \n",
       "60419        83         3   104        3  1.0  0.0  48.0   8.0        4   \n",
       "60420        83         3   104        4  0.0  0.0  37.0   1.0        4   \n",
       "60421        83         3   104        5  1.0  0.0  48.0   9.0        4   \n",
       "60422        83         3   104        6  2.0  0.0  53.0  11.0        4   \n",
       "60423        83         3   104        7  1.0  0.0  61.0   0.0        4   \n",
       "\n",
       "      week_day                      date  day  \n",
       "0          1_1 2018-01-01 00:00:00+00:00    1  \n",
       "1          1_2 2018-01-02 00:00:00+00:00    2  \n",
       "2          1_3 2018-01-03 00:00:00+00:00    3  \n",
       "3          1_4 2018-01-04 00:00:00+00:00    4  \n",
       "4          1_5 2018-01-05 00:00:00+00:00    5  \n",
       "...        ...                       ...  ...  \n",
       "60419    104_3 2019-12-25 00:00:00+00:00  724  \n",
       "60420    104_4 2019-12-26 00:00:00+00:00  725  \n",
       "60421    104_5 2019-12-27 00:00:00+00:00  726  \n",
       "60422    104_6 2019-12-28 00:00:00+00:00  727  \n",
       "60423    104_7 2019-12-29 00:00:00+00:00  728  \n",
       "\n",
       "[60424 rows x 12 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['borrar']='T00:00:00Z'\n",
    "data['date']=data['date']+data['borrar']\n",
    "data['date']=pd.to_datetime(data['date'])\n",
    "data=data.drop('borrar', axis=1)\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>asset_id</th>\n",
       "      <th>nominalc_week</th>\n",
       "      <th>nominalc_day</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>132000.0</td>\n",
       "      <td>18857.142857</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>93000.0</td>\n",
       "      <td>13285.714286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>238600.0</td>\n",
       "      <td>34085.714286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>206000.0</td>\n",
       "      <td>29428.571429</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>310000.0</td>\n",
       "      <td>44285.714286</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   asset_id  nominalc_week  nominalc_day\n",
       "0         1       132000.0  18857.142857\n",
       "1         2        93000.0  13285.714286\n",
       "2         3       238600.0  34085.714286\n",
       "3         4       206000.0  29428.571429\n",
       "4         5       310000.0  44285.714286"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Leer datos de capacidad nominal por asset por semana\n",
    "\n",
    "capacity=pd.read_csv('Datos/assets.csv', header=0, names=['asset_id', 'nominalc_week'],\n",
    "                     dtype={'asset_id':int, 'nominalc_week':float} )\n",
    "\n",
    "#agregar columna de capacidad por día\n",
    "capacity['nominalc_day']=capacity['nominalc_week']/7     \n",
    "capacity.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>asset_id</th>\n",
       "      <th>group_id</th>\n",
       "      <th>week</th>\n",
       "      <th>weekday</th>\n",
       "      <th>t1</th>\n",
       "      <th>t2</th>\n",
       "      <th>t3</th>\n",
       "      <th>t4</th>\n",
       "      <th>total_t</th>\n",
       "      <th>week_day</th>\n",
       "      <th>date</th>\n",
       "      <th>day</th>\n",
       "      <th>nominalc_day</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>19.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1_1</td>\n",
       "      <td>2018-01-01 00:00:00+00:00</td>\n",
       "      <td>1</td>\n",
       "      <td>18857.142857</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1_2</td>\n",
       "      <td>2018-01-02 00:00:00+00:00</td>\n",
       "      <td>2</td>\n",
       "      <td>18857.142857</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>9.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1_3</td>\n",
       "      <td>2018-01-03 00:00:00+00:00</td>\n",
       "      <td>3</td>\n",
       "      <td>18857.142857</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1_4</td>\n",
       "      <td>2018-01-04 00:00:00+00:00</td>\n",
       "      <td>4</td>\n",
       "      <td>18857.142857</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1_5</td>\n",
       "      <td>2018-01-05 00:00:00+00:00</td>\n",
       "      <td>5</td>\n",
       "      <td>18857.142857</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   asset_id  group_id  week  weekday  t1  t2    t3   t4  total_t week_day  \\\n",
       "0         1         2     1        1 NaN NaN  19.0  0.0        2      1_1   \n",
       "1         1         2     1        2 NaN NaN   8.0  0.0        2      1_2   \n",
       "2         1         2     1        3 NaN NaN   9.0  0.0        2      1_3   \n",
       "3         1         2     1        4 NaN NaN   6.0  0.0        2      1_4   \n",
       "4         1         2     1        5 NaN NaN   6.0  2.0        2      1_5   \n",
       "\n",
       "                       date  day  nominalc_day  \n",
       "0 2018-01-01 00:00:00+00:00    1  18857.142857  \n",
       "1 2018-01-02 00:00:00+00:00    2  18857.142857  \n",
       "2 2018-01-03 00:00:00+00:00    3  18857.142857  \n",
       "3 2018-01-04 00:00:00+00:00    4  18857.142857  \n",
       "4 2018-01-05 00:00:00+00:00    5  18857.142857  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data=data.merge(capacity[['asset_id','nominalc_day']])\n",
    "data.head()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Definición de funciones"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    " \n",
    "def ClasificacionDeObservaciones(df):  \n",
    "    '''FUNCIÓN PARA CLASIFICAR ANOMALÍAS MEDIANTE TRANSFORMADA DE WAVELET PARA CADA UNA \n",
    "    DE LAS SERIES CORREPONDIENTES A LAS MEDICIONES HECHAS EN CADA ASSET\n",
    "    df: pandas data frame\n",
    "    wavelet_family:  'haar', 'db2', o cualquier otra\n",
    "    '''\n",
    "\n",
    "    if df.total_t.mean() == 2:\n",
    "        mediciones=[3,4]\n",
    "    else:\n",
    "        mediciones=[1,2,3,4]\n",
    "    for x in mediciones:  #para cada una de las mediciones\n",
    "        t='t'+str(x)\n",
    "        if df[t].unique().sum()!=0:\n",
    "            series=[]\n",
    "            for index, row in df.iterrows():\n",
    "                series.append(Point(timestamp=row['date'].isoformat(),value=row[t]))\n",
    "            request= Request(series=series, granularity=Granularity.daily, period=7, sensitivity=20)\n",
    "            try:\n",
    "                response = client.entire_detect(request)\n",
    "            except Exception as e:\n",
    "                if isinstance(e, APIErrorException):\n",
    "                    print('Error code: {}'.format(e.error.code),\n",
    "                          'Error message: {}'.format(e.error.message))\n",
    "                else:\n",
    "                    print(e)\n",
    "            respuesta=response.as_dict()\n",
    "            #Crear df anomlias con resultados en el diccionario respuesta\n",
    "            for key, value in respuesta.items():\n",
    "                anomalias=pd.DataFrame(respuesta)\n",
    "            #Agregar columna a asset\n",
    "            df['outlier_'+t]=anomalias['is_positive_anomaly'].astype(int)\n",
    "        continue\n",
    "    return df       "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def resultados(dict_df):\n",
    "    '''FUNCION PARA CALCULAR EL ERROR ASIGNANDO EL VALOR DE PRODUCCION NOMINAL DIARIO A LOS DÍAS EN QUE NO HUBO \n",
    "    NINGUNA ANOMALÍAS Y 0 EN CASO CONTRARIO \n",
    "    \n",
    "    dict_df: diccionario resultante con la detección de anomalías con un dataframe por cada asset\n",
    "\n",
    "    '''\n",
    "    \n",
    "    X=pd.concat(dict_df.values())\n",
    "    X=X.reset_index(drop=True)\n",
    "    \n",
    "    X=X[['asset_id', 'group_id', 'week', 'weekday','total_t','nominalc_day', 'outlier_t1','outlier_t2','outlier_t3','outlier_t4']]\n",
    "    \n",
    "\n",
    "    X['produccion']=np.where(\n",
    "        ( ( X['outlier_t1']==1 ) | ( X['outlier_t2']==1 ) | (X['outlier_t3']==1)  | ( X['outlier_t4']==1 ) ),\n",
    "        0, X['nominalc_day'] )\n",
    "   \n",
    "    #Agregamos los datos por grupo y semana para obtener Y_predicted\n",
    "\n",
    "    ProduccionGrupoSemana=X.groupby(['group_id','week'])\n",
    "    Y_predicted=ProduccionGrupoSemana.agg({'produccion':'sum'})\n",
    "    Y_predicted=Y_predicted.reset_index()\n",
    "\n",
    "    # Configuramos el formato  Y_predicted para tener el mismo que en Y_train\n",
    "    Y_predicted=pd.pivot_table(Y_predicted,index=['week'], columns=['group_id'], values=['produccion'])\n",
    "    Y_predicted=Y_predicted.reset_index()\n",
    "    Y_predicted.columns.set_levels(['production_group_2', 'production_group_3', ''], level=1, inplace=True)\n",
    "    Y_predicted.columns=list(map(\"\".join, Y_predicted))\n",
    "    Y_predicted.columns=Y_train.columns\n",
    "\n",
    "    # Calculamos el error para cada guropo\n",
    "    MSE2=mean_squared_error(Y_train.PRODUCTION_GROUP_2, Y_predicted.PRODUCTION_GROUP_2)\n",
    "    MSE3=mean_squared_error(Y_train.PRODUCTION_GROUP_3, Y_predicted.PRODUCTION_GROUP_3)\n",
    "\n",
    "    \n",
    "    #Calculamos el error total\n",
    "    Y_train_tot = Y_train.PRODUCTION_GROUP_2.to_list() + Y_train.PRODUCTION_GROUP_3.to_list()\n",
    "    Y_predicted_tot = Y_predicted.PRODUCTION_GROUP_2.to_list() +  Y_predicted.PRODUCTION_GROUP_3.to_list() \n",
    "    ErrorTotal = mean_squared_error(Y_train_tot, Y_predicted_tot)\n",
    "    \n",
    "\n",
    "    return Y_predicted, ErrorTotal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Clasificar observaciones, identificando anomalías para cada uno de las series de tiempo de los assets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Crear objeto GroupBy de Pandas para iterar despues en las tablas de cada asset\n",
    "data=data.set_index('asset_id')\n",
    "porAsset=data.groupby(data.index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--- 91.46268129348755 seconds ---\n",
      "Wall time: 1min 31s\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "# 1 min 41 seg aproximadamente\n",
    "\n",
    "#Pasamos el subconjunto de datos de cada asset por la función de deteccion de anomalías \n",
    "#y guardamos cada uno de los dataframes resultantes con la clasificacione de las observaciones en el diccionario new_data\n",
    "import time\n",
    "start_time=time.time()\n",
    "new_data={}\n",
    "for ass in range(1,84):\n",
    "    asset = porAsset.get_group(ass).reset_index()\n",
    "    new_data[ass] = ClasificacionDeObservaciones(asset)\n",
    "print(\"--- %s seconds ---\"%(time.time()-start_time))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Resultados"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "error = 2.0547269315712854e+12\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SAMPLE_ID</th>\n",
       "      <th>PRODUCTION_GROUP_2</th>\n",
       "      <th>PRODUCTION_GROUP_3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>3.577851e+06</td>\n",
       "      <td>7.410714e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>3.444366e+06</td>\n",
       "      <td>6.366856e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>3.717009e+06</td>\n",
       "      <td>5.904107e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>3.395196e+06</td>\n",
       "      <td>6.046025e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>3.752381e+06</td>\n",
       "      <td>5.958516e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>100</td>\n",
       "      <td>3.135859e+06</td>\n",
       "      <td>7.265531e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <td>101</td>\n",
       "      <td>3.039880e+06</td>\n",
       "      <td>7.447671e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>101</th>\n",
       "      <td>102</td>\n",
       "      <td>3.311820e+06</td>\n",
       "      <td>7.743620e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102</th>\n",
       "      <td>103</td>\n",
       "      <td>3.757094e+06</td>\n",
       "      <td>7.912013e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>103</th>\n",
       "      <td>104</td>\n",
       "      <td>3.712752e+06</td>\n",
       "      <td>8.213165e+06</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>104 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     SAMPLE_ID  PRODUCTION_GROUP_2  PRODUCTION_GROUP_3\n",
       "0            1        3.577851e+06        7.410714e+06\n",
       "1            2        3.444366e+06        6.366856e+06\n",
       "2            3        3.717009e+06        5.904107e+06\n",
       "3            4        3.395196e+06        6.046025e+06\n",
       "4            5        3.752381e+06        5.958516e+06\n",
       "..         ...                 ...                 ...\n",
       "99         100        3.135859e+06        7.265531e+06\n",
       "100        101        3.039880e+06        7.447671e+06\n",
       "101        102        3.311820e+06        7.743620e+06\n",
       "102        103        3.757094e+06        7.912013e+06\n",
       "103        104        3.712752e+06        8.213165e+06\n",
       "\n",
       "[104 rows x 3 columns]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_predicted, error= resultados(new_data)\n",
    "print('error =', np.format_float_scientific(error))\n",
    "y_predicted"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Referencias\n",
    "\n",
    "\n",
    "1. aahill. Quickstart: Use the Anomaly Detector client library - Azure Cognitive Services [Internet]. [citado 29 de julio de 2020]. Disponible en: https://docs.microsoft.com/en-us/azure/cognitive-services/anomaly-detector/quickstarts/client-libraries\n",
    "\n",
    "\n",
    "2. Hou X, Zhang L. Saliency detection: A spectral residual approach. En: In IEEE Conference on Computer Vision and Pattern Recognition (CVPR07) IEEE Computer Society. 2007. p. 1–8. Disponible en: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.125.5641&rep=rep1&type=pdf\n",
    "\n",
    "\n",
    "3. Ren H, Xu B, Wang Y, Yi C, Huang C, Kou X, et al. Time-Series Anomaly Detection Service at Microsoft. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining [Internet]. 25 de julio de 2019 [citado 30 de julio de 2020];3009-17. Disponible en: http://arxiv.org/abs/1906.03821\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}