03_model_playground.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model Playground\n",
    "\n",
    "This notebook is intended as a playground to test out different models with the data prepared from the previous notebook.\n",
    "\n",
    "## Import Modules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.model_selection import train_test_split\n",
    "from keras.models import Sequential, load_model\n",
    "from keras import layers\n",
    "import tensorflow as tf\n",
    "import datetime\n",
    "import cv2"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CNN Model Training\n",
    "This code trains a convolutional neural network to perform obstacle detection. Here is a breakdown of the implementation:\n",
    "\n",
    "- The model architecture is defined using the Sequential API of Keras. The model consists of a series of convolutional layers with ReLU activation and pooling layers, followed by two fully connected layers. The architecture is chosen such that the spatial dimensions of the feature maps are progressively reduced by the pooling layers.\n",
    "- 5 pooling layers with a pool size of 2x2 are used to progressively reduce the spatial dimensions of the feature maps.\n",
    "- The binary_crossentropy loss function is commonly used in binary classification tasks as it is well suited for computing the cross-entropy between two probability distributions.\n",
    "- The Adam optimizer is commonly used in deep learning as it adapts its learning rate during training, making it more efficient and effective compared to other optimization algorithms.\n",
    "- In obstacle detection, it's important to have a metric that evaluates the model's ability to detect obstacles accurately. F1 score is a good metric to use because it takes both precision and recall into account, which makes it more reliable when the class distribution is imbalanced."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "17/17 [==============================] - 41s 2s/step - loss: 29.5220 - accuracy: 0.7008 - val_loss: 0.5572 - val_accuracy: 0.8161\n",
      "Epoch 2/10\n",
      "17/17 [==============================] - 41s 2s/step - loss: 0.4680 - accuracy: 0.8185 - val_loss: 0.4184 - val_accuracy: 0.8161\n",
      "Epoch 3/10\n",
      "17/17 [==============================] - 40s 2s/step - loss: 0.4131 - accuracy: 0.8263 - val_loss: 0.3885 - val_accuracy: 0.8296\n",
      "Epoch 4/10\n",
      "17/17 [==============================] - 49s 3s/step - loss: 0.3894 - accuracy: 0.8320 - val_loss: 0.3800 - val_accuracy: 0.8251\n",
      "Epoch 5/10\n",
      "17/17 [==============================] - 42s 2s/step - loss: 0.3815 - accuracy: 0.8340 - val_loss: 0.3894 - val_accuracy: 0.8430\n",
      "Epoch 6/10\n",
      "17/17 [==============================] - 42s 2s/step - loss: 0.3612 - accuracy: 0.8668 - val_loss: 0.4245 - val_accuracy: 0.8341\n",
      "Epoch 7/10\n",
      "17/17 [==============================] - 42s 2s/step - loss: 0.3947 - accuracy: 0.8533 - val_loss: 0.3640 - val_accuracy: 0.8386\n",
      "Epoch 8/10\n",
      "17/17 [==============================] - 47s 3s/step - loss: 0.3491 - accuracy: 0.8552 - val_loss: 0.3626 - val_accuracy: 0.8341\n",
      "Epoch 9/10\n",
      "17/17 [==============================] - 55s 3s/step - loss: 0.3574 - accuracy: 0.8649 - val_loss: 0.3761 - val_accuracy: 0.8475\n",
      "Epoch 10/10\n",
      "17/17 [==============================] - 45s 3s/step - loss: 0.3266 - accuracy: 0.8707 - val_loss: 0.3515 - val_accuracy: 0.8520\n",
      "7/7 [==============================] - 4s 510ms/step - loss: 0.3515 - accuracy: 0.8520\n",
      "Test Accuracy: 0.8520179390907288\n",
      "Model: \"sequential_5\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " conv2d_25 (Conv2D)          (None, 382, 286, 32)      1184      \n",
      "                                                                 \n",
      " max_pooling2d_25 (MaxPoolin  (None, 191, 143, 32)     0         \n",
      " g2D)                                                            \n",
      "                                                                 \n",
      " conv2d_26 (Conv2D)          (None, 189, 141, 64)      18496     \n",
      "                                                                 \n",
      " max_pooling2d_26 (MaxPoolin  (None, 94, 70, 64)       0         \n",
      " g2D)                                                            \n",
      "                                                                 \n",
      " conv2d_27 (Conv2D)          (None, 92, 68, 128)       73856     \n",
      "                                                                 \n",
      " max_pooling2d_27 (MaxPoolin  (None, 46, 34, 128)      0         \n",
      " g2D)                                                            \n",
      "                                                                 \n",
      " conv2d_28 (Conv2D)          (None, 44, 32, 128)       147584    \n",
      "                                                                 \n",
      " max_pooling2d_28 (MaxPoolin  (None, 22, 16, 128)      0         \n",
      " g2D)                                                            \n",
      "                                                                 \n",
      " conv2d_29 (Conv2D)          (None, 20, 14, 256)       295168    \n",
      "                                                                 \n",
      " max_pooling2d_29 (MaxPoolin  (None, 10, 7, 256)       0         \n",
      " g2D)                                                            \n",
      "                                                                 \n",
      " flatten_5 (Flatten)         (None, 17920)             0         \n",
      "                                                                 \n",
      " dense_10 (Dense)            (None, 256)               4587776   \n",
      "                                                                 \n",
      " dense_11 (Dense)            (None, 1)                 257       \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 5,124,321\n",
      "Trainable params: 5,124,321\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Define model\n",
    "model_name = \"obstacle_detection_v3\"\n",
    "\n",
    "# Define checkpoint callback\n",
    "checkpoint_dir = f\"checkpoint/{model_name}\"\n",
    "os.makedirs(checkpoint_dir, exist_ok=True)\n",
    "checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(\n",
    "filepath=os.path.join(\"./checkpoint/\", \"ckpt_{epoch}\"),\n",
    "save_weights_only=True,\n",
    "save_freq=\"epoch\")\n",
    "\n",
    "#Define logger callback\n",
    "log_dir = f\"logs/{model_name}/{datetime.datetime.now().strftime('%Y%m%d-%H%M%S')}\"\n",
    "tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)\n",
    "\n",
    "#Load Data\n",
    "X = np.load(\"data/train_x.npy\", allow_pickle=True)\n",
    "y = np.load(\"data/train_y_v2.npy\", allow_pickle=True) #Labels from Algorithm 2\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=7)\n",
    "\n",
    "#Define model\n",
    "model = tf.keras.models.Sequential([\n",
    "    tf.keras.layers.Conv2D(32, (3,3), activation='relu', input_shape=X_train.shape[1:]),\n",
    "    tf.keras.layers.MaxPooling2D((2,2)),\n",
    "    tf.keras.layers.Conv2D(64, (3,3), activation='relu'),\n",
    "    tf.keras.layers.MaxPooling2D((2,2)),\n",
    "    tf.keras.layers.Conv2D(128, (3,3), activation='relu'),\n",
    "    tf.keras.layers.MaxPooling2D((2,2)),\n",
    "    tf.keras.layers.Conv2D(128, (3,3), activation='relu'),\n",
    "    tf.keras.layers.MaxPooling2D((2,2)),\n",
    "    tf.keras.layers.Conv2D(256, (3,3), activation='relu'),\n",
    "    tf.keras.layers.MaxPooling2D((2,2)),\n",
    "    tf.keras.layers.Flatten(),\n",
    "    tf.keras.layers.Dense(256, activation='relu'),\n",
    "    #representing the binary classification output (obstacle or no obstacle)\n",
    "    tf.keras.layers.Dense(1, activation='sigmoid')\n",
    "])\n",
    "\n",
    "# Compile model\n",
    "model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])\n",
    "\n",
    "csv_logger = tf.keras.callbacks.CSVLogger('training.log', separator=',', append=False)\n",
    "\n",
    "# Train model with CSVLogger callback\n",
    "model.fit(X_train, y_train, epochs=10, batch_size=32,\n",
    "          callbacks=[checkpoint_callback, tensorboard_callback, csv_logger],\n",
    "          validation_data=(X_test, y_test))\n",
    "\n",
    "# Evaluate the model on the testing set\n",
    "test_loss, test_acc = model.evaluate(X_test, y_test)\n",
    "print('Test Accuracy:', test_acc)\n",
    "\n",
    "#Save the model\n",
    "model.save(f\"{model_name}.h5\")\n",
    "weights = model.get_weights()\n",
    "np.savetxt(f\"{model_name}_weights.txt\", np.concatenate([w.flatten() for w in weights]))\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/1 [==============================] - 0s 359ms/step\n",
      "File ground_truth.png: predicted class 0\n",
      "1/1 [==============================] - 0s 43ms/step\n",
      "File no_obstacle.png: predicted class 0\n",
      "1/1 [==============================] - 0s 39ms/step\n",
      "File side_obstacle1.png: predicted class 0\n",
      "1/1 [==============================] - 0s 27ms/step\n",
      "File side_obstacle2.png: predicted class 0\n",
      "True labels: [1 0 1 1]\n",
      "Predicted labels: [0. 0. 0. 0.]\n",
      "F1 score: 0.0\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import f1_score\n",
    "\n",
    "model = load_model('obstacle_detection_v3.h5')\n",
    "\n",
    "# Define the input directory and true labels\n",
    "input_dir = 'data/validate'\n",
    "true_labels = np.load('data/test_y_v2.npy')\n",
    "\n",
    "# Get a list of all image files in the directory and sort them alphabetically\n",
    "image_files = sorted([f for f in os.listdir(input_dir) if f.endswith('.png')])\n",
    "\n",
    "# Initialize empty arrays to store predicted and true labels\n",
    "predicted_labels = np.zeros(len(image_files))\n",
    "\n",
    "# Loop over each image file and make predictions\n",
    "for i, filename in enumerate(image_files):\n",
    "    # Load the image and preprocess it as needed\n",
    "    image_path = os.path.join(input_dir, filename)\n",
    "    image = cv2.imread(image_path, cv2.IMREAD_UNCHANGED)\n",
    "    data = np.array(image)\n",
    "    data = np.expand_dims(data, axis=0)  # shape: (1, 384, 288, 4)\n",
    "\n",
    "    # Make predictions on the image\n",
    "    predictions = model.predict(data)\n",
    "\n",
    "    # Print the predicted classes\n",
    "    predicted_class = np.argmax(predictions, axis=1)[0]\n",
    "    print(f\"File {filename}: predicted class {predicted_class}\")\n",
    "    \n",
    "    # Store predicted label\n",
    "    predicted_labels[i] = predicted_class\n",
    "\n",
    "# Calculate F1 score\n",
    "f1 = f1_score(true_labels, predicted_labels)\n",
    "\n",
    "print(f\"True labels: {true_labels}\")\n",
    "print(f\"Predicted labels: {predicted_labels}\")\n",
    "print(f\"F1 score: {f1}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "se15",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "82d40570fe7a8cef5127558dff9f999d665f8f3c4074b1ee889e6c0afa23c999"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}