04-QDA.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linear and quadratic discriminant analysis\n",
    "In todays lecture we will cover linear (LDA) and quadratic discriminant analysis (QDA) and how we can use them for classification.\n",
    "But before we dive in deeper think about what we actually trying to achieve with this approaches.\n",
    "\n",
    "We have learned that we use so-called discriminants to represent classifiers. For a classifier with $\\omega_1,...,\\omega_n$ classes, we have a set of discriminant functions $g_i(\\mathbf{x})$ with $i \\in \\{1,...,c\\}$.    \n",
    "*Task*: Give some examples for discriminants with a minimal error rate!    \n",
    "* Todo           \n",
    "\n",
    "*Task*: What happens with the feature space $\\mathbf{R}^d$ when we use such discriminants/decission rules?\n",
    "* Todo        \n",
    "\n",
    "*Task*: How do we decide to which class $\\omega_i$ a feature vetor $\\mathbf{x}$ belongs?\n",
    "* Todo        \n",
    "\n",
    "Now that we have an idea about discriminants, we can dive deeper into our LDA and QDA classifiers.    \n",
    "*Task*: We want to calculate the probability that a featire vector $\\mathbf{x}$ belongs to a class $\\mathbf{\\omega_i}$. What can we use to calculate $P(\\omega_i|\\mathbf{x})$?    \n",
    "* $P(\\omega_i|\\mathbf{x}) = $ Todo    \n",
    "\n",
    "*Task*: What kind of assumption do we have to make when we use an LDA/QDA?\n",
    "* Todo    \n",
    "\n",
    "*Task*: What is the difference between a LDA and QDA?\n",
    "* Todo    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1\n",
    "A fisherman needs your help in classifying fish. He recently caught the following fish:\n",
    "\n",
    "| Length (m)    | Species          | \n",
    "| ------------- |-------------  |\n",
    "| 1.3           | Seabass       |\n",
    "| 0.7           | Salmon       |\n",
    "| 0.62           | Salmon      |\n",
    "| 0.9           | Salmon       |\n",
    "| 0.91          | Seabass       |\n",
    "| 0.31          | Herring       |\n",
    "| 0.26           | Herring       |\n",
    "\n",
    "* Calculate the priors $p(\\omega_i)$ for each fish species\n",
    "* What is the formula for calculating the parameters $\\mu$ and $\\sigma^2$ for the likelihoods?\n",
    "* Calculate the parameters $\\mu$ and $\\sigma^2$ for the likelihoods $p(\\mathbf{x}|\\omega_i)$.\n",
    "* The fisherman catches a new fish with length $x = 0.82 m$. Calculate the posterior probability $p(\\omega_i|\\mathbf{x})$ for each class. How is the fish classified?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 2\n",
    "Implement a function `priors(classes)` that outputs the prior $p(\\omega)$ for each class for a vector of class labels.\n",
    "The input should be an array of classes (e.g. `np.array([\"stand\", \"sit\", \"sit\", \"stand\"])`). The output should be a data frame with the columns `class` and `prior`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "def priors(classes):\n",
    "    'Implement me!'\n",
    "    \n",
    "pp = priors(np.array([\"stand\",\"sit\",\"sit\",\"sit\",\"stand\"]))\n",
    "print(pp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 3\n",
    "Implement a function `likelihood(data)` that approximates the likelihood $p(\\omega_i|\\mathbf{x})$ for each class $\\omega_i$ with a normal distribution for a data frame consisting of a column $y$ and a column $x$, i.e. a mean value and a variance are to be output for each class.\n",
    "The output should therefore have the columns `class`, `mean` and `variance`.\n",
    "\n",
    "Plot the likelihood for each class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.io import arff\n",
    "\n",
    "def likelihood(data):\n",
    "    'Implement me!'\n",
    "    \n",
    "data = arff.loadarff('features1.arff')\n",
    "df = pd.DataFrame(data[0])\n",
    "\n",
    "dat = df.loc[:, [\"AccX_mean\",\"class\"]]\n",
    "dat.columns = [\"x\",\"class\"]\n",
    "lik = likelihood(dat)\n",
    "lik"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 4\n",
    "Implement a function myqda(newdat,lik,priors) that returns the most probable class for a new observation `newdat`.\n",
    "\n",
    "Test your implementation on the dataset `features1.arff`. \"Train\" the QDA (i.e. calculate likelihood and prior), and then perform classification on the same data. How good is the classification?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.io import arff\n",
    "import scipy.stats\n",
    "\n",
    "def myqda(newdat,lik,prior):\n",
    "    'Implement me!'\n",
    "\n",
    "\n",
    "data = arff.loadarff('features1.arff')\n",
    "df = pd.DataFrame(data[0])\n",
    "\n",
    "dat = df.loc[:, [\"AccX_mean\",\"class\"]]\n",
    "dat.columns = [\"x\",\"class\"]\n",
    "\n",
    "lik = likelihood(dat)\n",
    "prior = priors(dat[\"class\"])\n",
    "\n",
    "nc = myqda(dat[\"x\"][1:100],lik,prior) \n",
    "print(sum(nc == dat[\"class\"][1:100])/100) # compute fraction of correct classified data points"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}