complete_pipeline.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(c) Copyright 2023 Enzo Alexander Cording - https://github.com/EnzoCording - GNU GPL v3.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This pipeline walks through the entire functionalities of FleetRL\n",
    "\n",
    "1) Creating a custom use-case\n",
    "    - Updating your data path\n",
    "    - Changing environment settings if needed\n",
    "    - Generating your own vehicle schedules\n",
    "2) Training an RL agent\n",
    "3) Building benchmark charging strategies\n",
    "4) Comparing the RL agent to the benchmarks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This code could also be run in a .py file. Then, the code should be wrapped in:\n",
    "\n",
    "    if __name__ == \"__main__\":\n",
    "        #code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Importing dependencies**"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "collapsed": true,
    "ExecuteTime": {
     "end_time": "2024-08-12T20:42:29.660148Z",
     "start_time": "2024-08-12T20:42:23.970853Z"
    }
   },
   "source": [
    "import datetime as dt\n",
    "import numpy as np\n",
    "import math\n",
    "import matplotlib.pyplot as plt\n",
    "from typing import Literal\n",
    "import pandas as pd\n",
    "import time\n",
    "import os\n",
    "\n",
    "from fleetrl.fleet_env.fleet_environment import FleetEnv\n",
    "from fleetrl.benchmarking.benchmark import Benchmark\n",
    "from fleetrl.benchmarking.uncontrolled_charging import Uncontrolled\n",
    "from fleetrl.benchmarking.distributed_charging import DistributedCharging\n",
    "from fleetrl.benchmarking.night_charging import NightCharging\n",
    "from fleetrl.benchmarking.linear_optimization import LinearOptimization\n",
    "\n",
    "from fleetrl.agent_eval.evaluation import Evaluation\n",
    "from fleetrl.agent_eval.basic_evaluation import BasicEvaluation\n",
    "\n",
    "from stable_baselines3.common.vec_env import VecNormalize, SubprocVecEnv\n",
    "from stable_baselines3.common.env_util import make_vec_env\n",
    "from stable_baselines3 import PPO\n",
    "from stable_baselines3.common.callbacks import EvalCallback, ProgressBarCallback, BaseCallback\n",
    "from stable_baselines3.common.logger import HParam\n",
    "\n",
    "from pink import PinkActionNoise\n",
    "from stable_baselines3.common.noise import OrnsteinUhlenbeckActionNoise, NormalActionNoise"
   ],
   "outputs": [],
   "execution_count": 1
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Creating a custom use-case**\n",
    "Go through this step by step and check the documentation if needed. The docs specify what type of input data is required, what format it should be in, etc.\n",
    "The code below is commented to provide the most essential information.\n",
    "\n",
    "Docs: fleetrl.readthedocs.io"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**General settings**\n",
    "Under general settings, you can adjust how many vehicles to optimize for, whether you would like to create new schedules how long the episodes should be, etc.\n",
    "There is a pre-trained agent for the 1-EV environments. So you can give this a try before training your own agents and just benchmark performances and evaluate your environment. Once you are sure everything is set up correctly, increase the number of EVs and train your own agents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2024-04-29T22:13:05.753934Z",
     "end_time": "2024-04-29T22:13:05.854806Z"
    }
   },
   "outputs": [],
   "source": [
    "# define fundamental parameters\n",
    "# data path to inputs folder with schedule csv, prices, load, pv, etc.\n",
    "input_data_path: str = \"inputs\"  # NOTE: either use \"/\" (linux) or \"\\\\\" (windows)\n",
    "run_name: str = \"Test_run_custom\"  # Change this name or make it dynamic, e.g. with a timestamp\n",
    "n_train_steps = 48  # number of hours in a training episode\n",
    "n_eval_steps = 48  # number of hours in one evaluation episode\n",
    "n_eval_episodes = 1  # number of episodes for evaluation\n",
    "n_evs = 1  # number of evs\n",
    "n_envs = 2  # number of envs parallel - has to be equal to 1, if train_freq = (1, episode) or default setting\n",
    "time_steps_per_hour = 4  # temporal resolution\n",
    "use_case: str = \"custom\"  # for file name - lmd=last mile delivery, by default can insert \"lmd\", \"ct\", \"ut\", \"custom\"\n",
    "custom_schedule_name = \"1_lkw.csv\"  # name for custom schedule if you have generated one. If you want to generate one this time, this field will be ignored\n",
    "scenario: Literal[\"arb\", \"tariff\"] = \"tariff\"  # arbitrage or tariff. Arbitrage allows for bidirectional spot trading, no fees. Tariff models commercial tariff\n",
    "gen_new_schedule = True  # generate a new schedule - refer to schedule generator documentation and adjust statistics in config.json\n",
    "gen_new_test_schedule = True  # generate a new schedule for agent testing\n",
    "\n",
    "real_time = False  # Experimental - leave False for now but in the future FleetRL will be able to handle real-time data with arbitrary time resolution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Training settings**\n",
    "These more low-level settings allow you to change training-related parameters. Refer to the documentation of FleetRL and stable-baselines3 for further details. Observations are by default normalized within SB3, due to their rolling average normalization. You can also conduct absolute normalization via FleetRL.\n",
    "\n",
    "Adapt total training steps and saving interval for a full run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2024-04-29T22:13:07.130126Z",
     "end_time": "2024-04-29T22:13:07.134093Z"
    }
   },
   "outputs": [],
   "source": [
    "# training parameters\n",
    "norm_obs_in_env = False  # normalize observations within FleetRL (max, min normalization)\n",
    "vec_norm_obs = True  # normalize observations in SB3 (rolling normalization)\n",
    "vec_norm_rew = True  # normalize rewards in SB3 (rolling normalization)\n",
    "\n",
    "# Total steps should be sep to 1e6 or 5e6 for a full run. Check tensorboard for stagnating reward signal and stop training at some point to avoid overfit\n",
    "total_steps = int(1e3)  # total training time steps\n",
    "\n",
    "# Specifies how often you want to make an intermediate artifact. For a full run, I recommend every 50k - 100k steps, so you can backtrack for best model\n",
    "saving_interval = 5e2  # interval for saving the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Parameters for environment object creation**\n",
    "Further settings can be adjusted below, view the comments and docs for more detailed explanations.\n",
    "Most important:\n",
    "- Episode length: How long an episode is in hours - at least 36 hours are recommended so the agent always sees one passage of a night for night charging\n",
    "- include_building, include_pv, include_price: These adjust the shape of the observations to make the problem simpler or more complex\n",
    "- price_lookahead, bl_pv_lookahead: These dictate how much knowledge into the future the agent has on price, building load and PV in hours\n",
    "- Time picker: Use random during training: This way, a new episode always starts at a random point in the dataframe\n",
    "- Deg_emp: For simple degradation, set to True\n",
    "- Ignore_x_reward: Set accordingly with Include_x... or deactivate certain parts of the reward function to adjust problem complexity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2024-04-29T22:13:08.462790Z",
     "end_time": "2024-04-29T22:13:08.576467Z"
    }
   },
   "outputs": [],
   "source": [
    "# environment arguments - adjust settings if necessary\n",
    "# additional settings can be changed in the config files\n",
    "env_config = {\"data_path\": input_data_path,\n",
    "              # Specify file names: there is a naming convention for default files, otherwise, custom name is used\n",
    "              \"schedule_name\": (str(n_evs) + \"_\" + str(use_case) + \".csv\") if use_case != \"custom\" else custom_schedule_name,\n",
    "              \"building_name\": \"load_\" + str(use_case) + \".csv\" if use_case != \"custom\" else \"load_lmd.csv\",\n",
    "              \"pv_name\": None,  # if separate file for PV inputs, specify here, otherwise, uses \"PV\" column in building_name\n",
    "              # Define use case\n",
    "              \"use_case\": use_case,\n",
    "              # Change observation space\n",
    "              \"include_building\": True,  # False removes building load from Observation\n",
    "              \"include_pv\": True,  # False removes PV from Observation\n",
    "              \"include_price\": True,  # False removes electricity prices from Observation\n",
    "              \"price_lookahead\": 8,  # Hours seen into the future\n",
    "              \"bl_pv_lookahead\": 4,  # Hours seen into the future\n",
    "              \"time_steps_per_hour\": 4,  # Time resolution\n",
    "              # Specify time picker: \"eval\", \"static\", or \"random\" are implemented\n",
    "              \"time_picker\": \"random\",  # Pick a random starting day in the schedule dataframe\n",
    "              # Pick degradation methodology: True sets empirical degradation from experimental degradation\n",
    "              \"deg_emp\": False,  # empirical degradation calculation\n",
    "              # Shape reward function\n",
    "              \"ignore_price_reward\": False,  # True sets price-related reward coefficient to 0\n",
    "              \"ignore_invalid_penalty\": False,  # True ignores penalties on invalid actions (charging an empty spot)\n",
    "              \"ignore_overcharging_penalty\": False,  # True ignores penalties on charging signals above target SOC\n",
    "              \"ignore_overloading_penalty\": False,  # True ignores grid connection overloading penalty\n",
    "              # Set episode length during training\n",
    "              \"episode_length\": n_train_steps,  # in hours\n",
    "              # Additional parameters\n",
    "              \"normalize_in_env\": norm_obs_in_env,  # Conduct normalization within FleetRL.\n",
    "              \"verbose\": 0,  # Print statements, can slow down FPS\n",
    "              \"aux\": True,  # Include auxiliary data (recommended). Check documentation for more information.\n",
    "              \"log_data\": False,  # Log data (Makes most sense for evaluation runs)\n",
    "              \"calculate_degradation\": True,  # Calculate SOH degradation (Can slow down FPS)\n",
    "              # Target SOC\n",
    "              \"target_soc\": 0.85,  # Signals that would charge above target SOC are clipped.\n",
    "              # settings regarding the generation of evs\n",
    "              \"gen_schedule\": gen_new_schedule,  # generate a new schedule\n",
    "              \"gen_start_date\": \"2021-01-01 00:00\",  # if new schedule, start date\n",
    "              \"gen_end_date\": \"2021-12-31 23:59:59\",  # if new schedule, end date\n",
    "              \"gen_name\": \"my_custom_schedule.csv\",  # name of newly generated schedule\n",
    "              \"gen_n_evs\": 1,  # number of EVs in new schedule, per EV it takes ca. 10-20 min.\n",
    "              # seed for random number generation\n",
    "              \"seed\": 42,  # Seed for RNG - can be set to None so always random\n",
    "              # flag to optionally use real-time functions in FleetRL: no resampling of data, taking it in as is and\n",
    "              # using the event manager module to decide whether to perform an update or not\n",
    "              \"real_time\": real_time,\n",
    "              # if you are comparing cars with different bess sizes, use this to norm their reward function range\n",
    "              \"max_batt_cap_in_all_use_cases\": 600,\n",
    "              \"init_battery_cap\": 600,\n",
    "              # initial state of health of the battery\n",
    "              \"init_soh\": 1.0,\n",
    "              \"min_laxity\": 1.75,\n",
    "              \"obc_max_power\": 250,\n",
    "              # custom schedule timing settings, mean and standard deviation\n",
    "              \"custom_weekday_departure_time_mean\": 7,\n",
    "              \"custom_weekday_departure_time_std\": 1,\n",
    "              \"custom_weekday_return_time_mean\": 19,\n",
    "              \"custom_weekday_return_time_std\": 1,\n",
    "              \"custom_weekend_departure_time_mean\": 9,\n",
    "              \"custom_weekend_departure_time_std\": 1.5,\n",
    "              \"custom_weekend_return_time_mean\": 17,\n",
    "              \"custom_weekend_return_time_std\": 1.5,\n",
    "              \"custom_earliest_hour_of_departure\": 3,\n",
    "              \"custom_latest_hour_of_departure\": 11,\n",
    "              \"custom_earliest_hour_of_return\": 12,\n",
    "              \"custom_latest_hour_of_return\": 23,\n",
    "              # custom distance settings\n",
    "              \"custom_weekday_distance_mean\": 300,\n",
    "              \"custom_weekday_distance_std\": 25,\n",
    "              \"custom_weekend_distance_mean\": 150,\n",
    "              \"custom_weekend_distance_std\": 25,\n",
    "              \"custom_minimum_distance\": 20,\n",
    "              \"custom_max_distance\": 400,\n",
    "              # custom consumption data for vehicle\n",
    "              \"custom_consumption_mean\": 1.3,\n",
    "              \"custom_consumption_std\": 0.167463672468669,\n",
    "              \"custom_minimum_consumption\": 0.3,\n",
    "              \"custom_maximum_consumption\": 2.5,\n",
    "              \"custom_maximum_consumption_per_trip\": 500,\n",
    "              # custom ev-related settings\n",
    "              \"custom_ev_charger_power_in_kw\": 120,\n",
    "              \"custom_ev_battery_size_in_kwh\": 600,\n",
    "              \"custom_grid_connection_in_kw\": 500\n",
    "              }\n",
    "\n",
    "# commercial tariff scenario, fixed fee on spot price (+10 ct/kWh, and a 50% mark-up)\n",
    "# Feed-in tariff orientates after PV feed-in, with 25% deduction\n",
    "if scenario == \"tariff\":\n",
    "    env_config[\"spot_markup\"] = 10\n",
    "    env_config[\"spot_mul\"] = 1.5\n",
    "    env_config[\"feed_in_ded\"] = 0.25\n",
    "    env_config[\"price_name\"] = \"spot_2021_new.csv\"\n",
    "    env_config[\"tariff_name\"] = \"fixed_feed_in.csv\"\n",
    "\n",
    "# arbitrage scenario, up and down prices are spot price, no markups or taxes\n",
    "elif scenario == \"arb\":\n",
    "    env_config[\"spot_markup\"] = 0\n",
    "    env_config[\"spot_mul\"] = 1\n",
    "    env_config[\"feed_in_ded\"] = 0\n",
    "    env_config[\"price_name\"] = \"spot_2021_new.csv\"\n",
    "    env_config[\"tariff_name\"] = \"spot_2021_new_tariff.csv\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Environment object creation**\n",
    "Vec_Env are created to enable multi-processing.\n",
    "\n",
    "Train_vec_env: For agent training\n",
    "Eval_vec_env: For agent evaluation during training on same csv file (70% training data, 30% evaluation data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2024-04-29T22:13:09.935503Z",
     "end_time": "2024-04-29T22:13:16.746862Z"
    }
   },
   "outputs": [],
   "source": [
    "env_kwargs = {\"env_config\": env_config}\n",
    "\n",
    "train_vec_env = make_vec_env(FleetEnv,\n",
    "                             n_envs=n_envs,\n",
    "                             vec_env_cls=SubprocVecEnv,\n",
    "                             env_kwargs=env_kwargs,\n",
    "                             seed=env_config[\"seed\"])\n",
    "\n",
    "train_norm_vec_env = VecNormalize(venv=train_vec_env,\n",
    "                                  norm_obs=vec_norm_obs,\n",
    "                                  norm_reward=vec_norm_rew,\n",
    "                                  training=True,\n",
    "                                  clip_reward=10.0)\n",
    "\n",
    "env_config[\"time_picker\"] = \"eval\"\n",
    "\n",
    "if gen_new_schedule:\n",
    "    env_config[\"gen_schedule\"] = False\n",
    "    env_config[\"schedule_name\"] = env_config[\"gen_name\"]\n",
    "\n",
    "env_kwargs = {\"env_config\": env_config}\n",
    "\n",
    "eval_vec_env = make_vec_env(FleetEnv,\n",
    "                            n_envs=n_envs,\n",
    "                            vec_env_cls=SubprocVecEnv,\n",
    "                            env_kwargs=env_kwargs,\n",
    "                            seed=env_config[\"seed\"])\n",
    "\n",
    "eval_norm_vec_env = VecNormalize(venv=eval_vec_env,\n",
    "                                  norm_obs=vec_norm_obs,\n",
    "                                  norm_reward=vec_norm_rew,\n",
    "                                  training=True,\n",
    "                                  clip_reward=10.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "If you want to conduct simple analytics on a FleetEnv object, it is recommended to create it directly instead over make_vec_env (which makes debugging more difficult due to sub-processes)\n",
    "\n",
    "FleetEnv objects can be created with a one-liner via specifying where the config file is located."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# As alternative, FleetEnv objects can be created via config.json files.\n",
    "test_env = FleetEnv(\"config.json\")"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2024-04-29T22:13:17.331333Z",
     "end_time": "2024-04-29T22:13:17.996518Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This creates a schedule for testing the trained agents on unseen data.\n",
    "It is recommended to generate a testing schedule along with a newly generated training schedule."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2024-04-29T22:13:45.713996Z",
     "end_time": "2024-04-29T22:13:49.383095Z"
    }
   },
   "outputs": [],
   "source": [
    "if gen_new_test_schedule:\n",
    "    # generate an evaluation schedule\n",
    "    test_sched_name = env_config[\"gen_name\"]\n",
    "    if not test_sched_name.endswith(\".csv\"):\n",
    "        test_sched_name = test_sched_name + \"_test\" + \".csv\"\n",
    "    else:\n",
    "        test_sched_name = test_sched_name.strip(\".csv\")\n",
    "        test_sched_name = test_sched_name + \"_test\" + \".csv\"\n",
    "\n",
    "    env_config[\"gen_schedule\"] = True\n",
    "    env_config[\"gen_name\"] = test_sched_name\n",
    "\n",
    "    env_kwargs = {\"env_config\": env_config}\n",
    "\n",
    "    test_vec_env = make_vec_env(FleetEnv,\n",
    "                                n_envs=1,\n",
    "                                vec_env_cls=SubprocVecEnv,\n",
    "                                env_kwargs=env_kwargs,\n",
    "                                seed=env_config[\"seed\"])\n",
    "\n",
    "    env_config[\"gen_schedule\"] = False\n",
    "    env_config[\"schedule_name\"] = test_sched_name\n",
    "\n",
    "    env_kwargs = {\"env_config\": env_config}\n",
    "\n",
    "test_vec_env = make_vec_env(FleetEnv,\n",
    "                            n_envs=n_envs,\n",
    "                            vec_env_cls=SubprocVecEnv,\n",
    "                            env_kwargs=env_kwargs,\n",
    "                            seed=env_config[\"seed\"])\n",
    "\n",
    "test_norm_vec_env = VecNormalize(venv=test_vec_env,\n",
    "                                 norm_obs=vec_norm_obs,\n",
    "                                 norm_reward=vec_norm_rew,\n",
    "                                 training=True,\n",
    "                                 clip_reward=10.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Callbacks are regularly called during training and enable useful functionalities such as logging or progress reporting. View SB3 docs for further information. Note that wandb callbacks are possible with SB3.\n",
    "\n",
    "Eval callback triggers an evaluation at fixed intervals\n",
    "HyperParamCallback logs hyperparameters, also visible in TensorBoard\n",
    "ProgressBar indicated progress of an epoch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2024-04-29T22:13:50.540610Z",
     "end_time": "2024-04-29T22:13:50.614502Z"
    }
   },
   "outputs": [],
   "source": [
    "eval_callback = EvalCallback(eval_env=eval_norm_vec_env,\n",
    "                             warn=True,\n",
    "                             verbose=1,\n",
    "                             deterministic=True,\n",
    "                             eval_freq=max(10000 // n_envs, 1),\n",
    "                             n_eval_episodes=5,\n",
    "                             render=False,\n",
    "                             )\n",
    "\n",
    "class HyperParamCallback(BaseCallback):\n",
    "    \"\"\"\n",
    "    Saves hyperparameters and metrics at start of training, logging to tensorboard\n",
    "    \"\"\"\n",
    "\n",
    "    def _on_training_start(self) -> None:\n",
    "        hparam_dict = {\n",
    "            \"algorithm\": self.model.__class__.__name__,\n",
    "            \"learning rate\": self.model.learning_rate,\n",
    "            \"gamma\": self.model.gamma,\n",
    "        }\n",
    "\n",
    "        metric_dict = {\n",
    "            \"rollout/ep_len_mean\": 0,\n",
    "            \"train/value_loss\": 0.0,\n",
    "        }\n",
    "\n",
    "        self.logger.record(\n",
    "            \"hparams\",\n",
    "            HParam(hparam_dict, metric_dict),\n",
    "            exclude=(\"stdout\", \"log\", \"json\", \"csv\")\n",
    "        )\n",
    "\n",
    "    def _on_step(self) -> bool:\n",
    "        return True\n",
    "\n",
    "progress_bar = ProgressBarCallback()\n",
    "\n",
    "## wandb callback possible, check documentation of SB3 and wandb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2024-04-29T22:13:51.259140Z",
     "end_time": "2024-04-29T22:13:51.264428Z"
    }
   },
   "outputs": [],
   "source": [
    "hyperparameter_callback = HyperParamCallback()"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "If you use TD3, pink action noise is said to improve performance\n",
    "If you use PPO, this is not used"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2024-04-29T22:13:51.664688Z",
     "end_time": "2024-04-29T22:13:51.773582Z"
    }
   },
   "outputs": [],
   "source": [
    "# model-related settings\n",
    "n_actions = train_norm_vec_env.action_space.shape[-1]\n",
    "param_noise = None\n",
    "noise_scale = 0.1\n",
    "seq_len = n_train_steps * time_steps_per_hour\n",
    "action_noise = PinkActionNoise(noise_scale, seq_len, n_actions)"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "You can choose to create your own agent of load an existing one. A pretrained PPO agent exists for a 1 EV environment."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "model = PPO.load(path=\"./rl_agents/trained_agents/LMD_arbitrage_1e6_steps_example_agent/PPO-fleet_LMD_2021_arbitrage_PPO_mul3.zip\", env=train_norm_vec_env)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2024-04-29T22:13:52.752534Z",
     "end_time": "2024-04-29T22:13:52.926496Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "If you want to use a loaded agent, you can skip the next 5 cells below"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "model = PPO(policy=\"MlpPolicy\",\n",
    "            verbose=1, # setting verbose to 0 can introduce performance increases in jupyterlab environments\n",
    "            env=train_norm_vec_env,\n",
    "            tensorboard_log=\"./rl_agents/trained_agents/tb_log\")\n",
    "\n",
    "# might introduce performance increases\n",
    "            # gamma=0.99,\n",
    "            # learning_rate=0.0005,\n",
    "            # batch_size=128,\n",
    "            # n_epochs=8,\n",
    "            # gae_lambda=0.9,\n",
    "            # clip_range=0.2,\n",
    "            # clip_range_vf=None,\n",
    "            # normalize_advantage=True,\n",
    "            # ent_coef=0.0008,\n",
    "            # vf_coef=0.5,\n",
    "            # max_grad_norm=0.5,\n",
    "            # n_steps=2048)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2024-04-28T14:21:15.694637Z",
     "end_time": "2024-04-28T14:21:15.718353Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# NOTE: make the Notebook trusted\n",
    "%reload_ext tensorboard\n",
    "%tensorboard --logdir ./rl_agents/trained_agents/tb_log --port 6006 --bind_all"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2024-04-28T14:21:16.480099Z",
     "end_time": "2024-04-28T14:21:19.006576Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# NOTE: make the Notebook trusted\n",
    "%reload_ext tensorboard\n",
    "%tensorboard --logdir ./rl_agents/trained_agents/tb_log --port 6006 --bind_all"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2024-04-28T14:21:19.330313Z",
     "end_time": "2024-04-28T14:21:19.350393Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2024-04-28T14:21:22.533644Z",
     "end_time": "2024-04-28T14:21:22.539732Z"
    }
   },
   "outputs": [],
   "source": [
    "comment = run_name\n",
    "time_now = int(time.time())\n",
    "trained_agents_dir = f\"./rl_agents/trained_agents/vec_PPO_{time_now}_{run_name}\"\n",
    "logs_dir = f\"{trained_agents_dir}/logs/\"\n",
    "\n",
    "if not os.path.exists(trained_agents_dir):\n",
    "    os.makedirs(trained_agents_dir)\n",
    "\n",
    "if not os.path.exists(logs_dir):\n",
    "    os.makedirs(logs_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2024-04-28T14:21:23.975640Z",
     "end_time": "2024-04-28T14:23:29.219714Z"
    }
   },
   "outputs": [],
   "source": [
    "# model training\n",
    "# models are saved in a specified interval: once with unique step identifiers\n",
    "# model and the normalization metrics are saved as well, overwriting the previous file every time\n",
    "for i in range(0, int(total_steps / saving_interval)):\n",
    "    model.learn(total_timesteps=saving_interval,\n",
    "                reset_num_timesteps=False,\n",
    "                tb_log_name=f\"PPO_{time_now}_{comment}\",\n",
    "                callback=[eval_callback, hyperparameter_callback, progress_bar])\n",
    "\n",
    "    model.save(f\"{trained_agents_dir}/{saving_interval * i}\")\n",
    "\n",
    "    # Don't forget to save the VecNormalize statistics when saving the agent\n",
    "    tmp_dir = f\"{trained_agents_dir}/tmp/\"\n",
    "    model_path = tmp_dir + f\"PPO-fleet_{comment}_{time_now}\"\n",
    "    model.save(model_path)\n",
    "    stats_path = os.path.join(tmp_dir, f\"vec_normalize-{comment}_{time_now}.pkl\")\n",
    "    train_norm_vec_env.save(stats_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "Agent evaluation\n",
    "- Sets the time_picker to static, so an evaluation always starts at the same point in time\n",
    "- Usually at the beginning of the year and then runs for a long time, e.g. a whole year to test the agent's ability to handle large amounts of unseen data\n",
    "- Make sure that the environment passed to the trained agent includes the unseen schedule\n",
    "- For that you should use test_norm_vec_env with the _test.csv schedule\n",
    "\n",
    "- Data is logged to be used for data analytics later. Every datapoint during the evaluation is tracked at every timestep"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2024-04-29T22:13:57.834957Z",
     "end_time": "2024-04-29T22:13:57.916943Z"
    }
   },
   "outputs": [],
   "source": [
    "# environment arguments for evaluation\n",
    "env_config[\"time_picker\"] = \"static\"  # Pick a random starting day in the schedule dataframe\n",
    "env_config[\"log_data\"] = True,  # Log data (Makes most sense for evaluation runs)\n",
    "\n",
    "env_kwargs = {\"env_config\": env_config}"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "Creation of Evaluation object"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2024-04-29T22:13:58.044070Z",
     "end_time": "2024-04-29T22:13:58.183420Z"
    }
   },
   "outputs": [],
   "source": [
    "eval: Evaluation = BasicEvaluation(n_steps=n_eval_steps,\n",
    "                                   n_evs=n_evs,\n",
    "                                   n_episodes=n_eval_episodes,\n",
    "                                   n_envs=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "- Use this is you are using a loaded agent. Otherwise, skip this cell\n",
    "- If you have just trained an agent, stats_path and model_path have been automatically defined"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "stats_path = \"./rl_agents/trained_agents/LMD_arbitrage_1e6_steps_example_agent/vec_normalize-LMD_2021_arbitrage_PPO_mul3.pkl\"\n",
    "model_path = \"./rl_agents/trained_agents/LMD_arbitrage_1e6_steps_example_agent/PPO-fleet_LMD_2021_arbitrage_PPO_mul3.zip\""
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2024-04-29T22:13:58.326190Z",
     "end_time": "2024-04-29T22:13:58.520667Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Saves the log from agent evaluation"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2024-04-29T22:13:59.998797Z",
     "end_time": "2024-04-29T22:14:13.328289Z"
    }
   },
   "outputs": [],
   "source": [
    "rl_log = eval.evaluate_agent(env_kwargs=env_kwargs, norm_stats_path=stats_path, model_path=model_path, seed=env_config[\"seed\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "The cells below define benchmarks and run them, saving the logs in the same way as RL evaluation\n",
    "- Uncontrolled charging: Plug in on arrival\n",
    "- LP: Linear-programming model - requires an LP solver (on Windows you might run into trouble when doing pip install glpk or similar due to C++ or MSVC. In that case you can try using a Conda environment for the repo and it should work.)\n",
    "  - Gurobi requires a license to be specified\n",
    "- Dist: Distributed charging, charges such that the car is completely filled up the moment before it leaves, charging the whole time with evenly distributed power\n",
    "- Night: Charging at night only at set times, e.g. from 1am to 5am"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2024-04-29T22:14:23.366811Z",
     "end_time": "2024-04-29T22:14:31.929957Z"
    }
   },
   "outputs": [],
   "source": [
    "uncontrolled_charging: Benchmark = Uncontrolled(n_steps=n_eval_steps,\n",
    "                                                n_evs=n_evs,\n",
    "                                                n_episodes=n_eval_episodes,\n",
    "                                                n_envs=1,\n",
    "                                                time_steps_per_hour=time_steps_per_hour)\n",
    "\n",
    "uc_log = uncontrolled_charging.run_benchmark(env_kwargs=env_kwargs, use_case=use_case, seed=env_config[\"seed\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "To try out linear optimisation, glpk must be installed. Alternatively, you can use your gurobi license. Simply swap out \"glpk\" for \"gurobi\" in linear_optimization.py in line 224"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "lp: Benchmark = LinearOptimization(n_steps=n_eval_steps,\n",
    "                                   n_evs=n_evs,\n",
    "                                   n_episodes=n_eval_episodes,\n",
    "                                   n_envs=1,\n",
    "                                   time_steps_per_hour=time_steps_per_hour)\n",
    "\n",
    "lp_log = lp.run_benchmark(env_kwargs=env_kwargs, use_case=use_case, seed=env_config[\"seed\"])"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2024-04-29T22:15:05.922758Z",
     "end_time": "2024-04-29T22:15:18.815682Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "dist: Benchmark = DistributedCharging(n_steps=n_eval_steps, n_evs=n_evs, n_episodes=n_eval_episodes, n_envs=1, time_steps_per_hour=time_steps_per_hour)\n",
    "\n",
    "dist_log = dist.run_benchmark(env_kwargs=env_kwargs, use_case=use_case, seed=env_config[\"seed\"])"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2024-04-29T22:15:19.515685Z",
     "end_time": "2024-04-29T22:15:31.833791Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "night: Benchmark = NightCharging(n_steps=n_eval_steps, n_evs=n_evs, n_episodes=n_eval_episodes, n_envs=1, time_steps_per_hour=time_steps_per_hour)\n",
    "\n",
    "night_log = night.run_benchmark(env_kwargs=env_kwargs, use_case=use_case, seed=env_config[\"seed\"])"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2024-04-29T22:15:31.833566Z",
     "end_time": "2024-04-29T22:15:43.552608Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "You can plot the average actions of the benchmarks. Just provide the log dataframes"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "lp.plot_benchmark(lp_log)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2024-04-29T22:15:43.551404Z",
     "end_time": "2024-04-29T22:15:43.951379Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "uncontrolled_charging.plot_benchmark(uc_log)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2024-04-29T22:15:43.962576Z",
     "end_time": "2024-04-29T22:15:44.328831Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "This showcases some evaluation methods that have already been written for easy comparison\n",
    "You can, of course, create new analytics from the log dataframes"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2024-04-29T22:15:45.612607Z",
     "end_time": "2024-04-29T22:15:47.927971Z"
    }
   },
   "outputs": [],
   "source": [
    "eval.compare(rl_log=rl_log, benchmark_log=uc_log)\n",
    "eval.compare(rl_log=rl_log, benchmark_log=lp_log)\n",
    "eval.plot_soh(rl_log=rl_log, benchmark_log=uc_log)\n",
    "eval.plot_soh(rl_log=rl_log, benchmark_log=dist_log)\n",
    "eval.plot_soh(rl_log=rl_log, benchmark_log=night_log)\n",
    "eval.plot_violations(rl_log=rl_log, benchmark_log=uc_log)\n",
    "eval.plot_action_dist(rl_log=rl_log, benchmark_log=uc_log)"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "The detailed plot shows what is happening during certain days of the simulation. You can add as many benchmark logs as you have compiled, the figure will automatically adjust based on number of EVs, number of logs, etc."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "detailed_fig = eval.plot_detailed_actions(start_date='2021-01-01 00:00',\n",
    "                                          end_date='2021-01-04 00:10',\n",
    "                                          rl_log=rl_log,\n",
    "                                          uc_log=uc_log,\n",
    "                                          dist_log=dist_log,\n",
    "                                          night_log=night_log)\n",
    "\n",
    "detailed_fig.show()"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2024-04-29T22:35:44.546801Z",
     "end_time": "2024-04-29T22:35:44.895840Z"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}