HMM_POS_Tagger.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hidden Markov Model for Part-of-Speech Tagging "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This Jupyter notebook illustrates how to use the developed module in this repository. The goal is to train a HMM-based POS tagger on German data. The developed code provides clean and elegant solutions to deal with common issues in this approach such as dealing with zero-value emission and transition probabilities as well as unknown words handling.  Lets start. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# general imports\n",
    "import nltk\n",
    "from nltk.corpus.reader import ConllCorpusReader\n",
    "\n",
    "# import tagger module\n",
    "from HMMTagger import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Indicate the directory of the data and some other metadata. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "directory    = 'data/'\n",
    "train_fileid = 'de-train.tt'\n",
    "test_fileid  = 'de-test.t'\n",
    "columntypes  = ['words', 'pos'] "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (1) Navigating through the HMM probabilisitc components"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets create an object of the HMMTager class and check the POS tags it can recognize."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# create a CoNLL corpus reader object\n",
    "train_corpus = nltk.corpus.ConllCorpusReader(\n",
    "    directory, train_fileid, columntypes, tagset='universal', encoding='utf8'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The constructor of the class HMMTagger takes as an input an object of ConllCorpusReader, which is a class to systematically process files in the CoNLL format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'.',\n",
       " 'ADJ',\n",
       " 'ADP',\n",
       " 'ADV',\n",
       " 'CONJ',\n",
       " 'DET',\n",
       " 'NOUN',\n",
       " 'NUM',\n",
       " 'PRON',\n",
       " 'PRT',\n",
       " 'VERB',\n",
       " 'X'}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "POSTagger = HMMTagger(train_corpus, smoothing='add_one')\n",
    "\n",
    "POSTagger.tagset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we choose to use add-one (a.k.a Laplace) smoothing in order to avoid zero-value probabilities. Thus, the parameters of the HMM components (that is; initial, transition, and emission probabilities) will be smoothed by adding one count to each of the events. That is, we will pretend that each event occurs one more time than its actual occurance count.  \n",
    "\n",
    "Moreover, the tagger can recognize 12 POS tags, which were used in the training data. \n",
    "\n",
    "Now, lets check some of the HMM parameters in depths. Lets start with the initial probabilities, or the probability of the tag being in the start position of the sentence. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "P( PRON|<S>) = 0.1429582449\n",
      "P(  ADV|<S>) = 0.1176928521\n",
      "P(  NUM|<S>) = 0.0288747346\n",
      "P( NOUN|<S>) = 0.1462137297\n",
      "P(  PRT|<S>) = 0.0015569710\n",
      "P(  DET|<S>) = 0.2613588110\n",
      "P( VERB|<S>) = 0.0148619958\n",
      "P(    .|<S>) = 0.0037508846\n",
      "P(    X|<S>) = 0.0009200283\n",
      "P(  ADJ|<S>) = 0.0428874735\n",
      "P(  ADP|<S>) = 0.2121019108\n",
      "P( CONJ|<S>) = 0.0268223638\n"
     ]
    }
   ],
   "source": [
    "for tag in POSTagger.tagset:\n",
    "    print(\"P({0:>5}|<S>) = {1:.10f}\".format(tag, POSTagger.initials.P[tag]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that P( DET |< S >) is the highest probobility, which is quite reasonable since it very common to start a sentence with a determiner in German. It is also common in German to start a sentence with a preposition, noun or pronoun. \n",
    "\n",
    "The initial probabilities come from a single distribution, therefore if the implementation is correct, these probability values should sum up to one. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(POSTagger.initials.P[tag] for tag in POSTagger.tagset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great! That means everything regarding initial probabilities (including smoothing) has been done correctly. \n",
    "\n",
    "Now lets move to peek into the transition probabilities. HMM transitions are conditioned on the POS tags. That is, we can think of the transition probabilities as 12 different independent distributions (one per each tag), where the event space for each distribution is the POS tags.  \n",
    "\n",
    "Lets check the transition probabilities for the determiner tag 'DET'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "P( PRON|DET) = 0.0055608624\n",
      "P(  ADV|DET) = 0.0118381550\n",
      "P(  NUM|DET) = 0.0107123362\n",
      "P( NOUN|DET) = 0.7300764192\n",
      "P(  PRT|DET) = 0.0004776201\n",
      "P(  DET|DET) = 0.0008528930\n",
      "P( VERB|DET) = 0.0002046943\n",
      "P(    .|DET) = 0.0084606987\n",
      "P(    X|DET) = 0.0002046943\n",
      "P(  ADJ|DET) = 0.2179312227\n",
      "P(  ADP|DET) = 0.0134415939\n",
      "P( CONJ|DET) = 0.0002388100\n"
     ]
    }
   ],
   "source": [
    "for tag in POSTagger.tagset:\n",
    "    print(\"P({0:>5}|DET) = {1:.10f}\".format(tag, POSTagger.transitions['DET'].P[tag]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is not surprising the conditional probability P(NOUN|DET) is the highest, since it is very natural that a noun would follow a determiner. \n",
    "\n",
    "Lets check the transition probabilities for a the tag 'VERB'. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "P( PRON|VERB) = 0.1428773969\n",
      "P(  ADV|VERB) = 0.0838266048\n",
      "P(  NUM|VERB) = 0.0190692234\n",
      "P( NOUN|VERB) = 0.0504377415\n",
      "P(  PRT|VERB) = 0.0101017262\n",
      "P(  DET|VERB) = 0.1250132917\n",
      "P( VERB|VERB) = 0.0805656967\n",
      "P(    .|VERB) = 0.3084393719\n",
      "P(    X|VERB) = 0.0002835572\n",
      "P(  ADJ|VERB) = 0.0323964130\n",
      "P(  ADP|VERB) = 0.1088505299\n",
      "P( CONJ|VERB) = 0.0381384468\n"
     ]
    }
   ],
   "source": [
    "for tag in POSTagger.tagset:\n",
    "    print(\"P({0:>5}|VERB) = {1:.10f}\".format(tag, POSTagger.transitions['VERB'].P[tag]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, the conditional probability P(.|VERB) is the highest, which might be surprising. Perhaps the reason is that many German verbs end up positioned at the end of the sentence, before the final period. \n",
    "\n",
    "Now lets move to emission probabilities, which estimat the probability of an observation being emitted from each state. In POS tagging, the observations are the words and the states are the POS tags. Like transition probabilities, we can think of it as 12 different probability distributions per each POS tags. But the difference is that the event space for the emission probabilities is the set of all possible observations, which is the word vocabulary in our case.  \n",
    "\n",
    "Lets check the probability of the word 'der' being emitted from the tag 'DET'. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "P('der'|DET) = 0.0920676830\n"
     ]
    }
   ],
   "source": [
    "print(\"P('der'|DET) = {0:.10f}\".format(POSTagger.emissions['DET'].P['der']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This value is actually quite high if we consider that the event space is very large (the size of the word vocabulary of the training data). If we check the probability of another word being emitted from the tag 'DET', it would be much smaller. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "P('klein'|DET) = 0.0000126937\n"
     ]
    }
   ],
   "source": [
    "print(\"P('klein'|DET) = {0:.10f}\".format(POSTagger.emissions['DET'].P['klein']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is certainly the case because the word 'klein' has never been tagged as a 'DET' in the training data. However, our add-one smoothing technique has prevented a zero-value emission probability here, which is very desirable to avoid run time errors. Lets see what value we get for the word 'klein' given the tag 'ADJ', or adjective. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "P('klein'|ADJ) = 0.0001885999\n"
     ]
    }
   ],
   "source": [
    "print(\"P('klein'|ADJ) = {0:.10f}\".format(POSTagger.emissions['ADJ'].P['klein']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It makes sense! The word 'klein' is actually an adjective, thus the probability P('klein'|ADJ) should be way higher than  P('klein'|DET).\n",
    "\n",
    "Another source for zero-value emission probabilies is the unknown words. These words were never observed in the training data thus their emission probability with each tag is zero. To solve this problem, the HMMTagger class has a procedure that adds a pseudo word, '< UNK >', to the word vocabulary before applying smoothing.\n",
    "\n",
    "To make sure that the pseodu-word '< UNK >' is actually in the vocab, lets do the following."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'<UNK>' in POSTagger.word_vocab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " During decoding, every unknown word will be replace by this ' < UNK >' token. Therefore, all emission probabilities for an unknown word will the value has been allocated to the token '< UNK >'. Lets check the emission probability of '< UNK >' being emitted from the tag 'DET'. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "P('<UNK>'|DET) = 0.0000126937\n"
     ]
    }
   ],
   "source": [
    "print(\"P('<UNK>'|DET) = {0:.10f}\".format(POSTagger.emissions['DET'].P['<UNK>']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It can be observed that the probabilities P('< UNK >'|DET) and P('klein'|DET)  have equal values. This can be justified by the fact that both the word 'klein' and the pseudo word '< UNK >', which has been added to the word vocabulary before smoothing, have never been observed with the tag 'DET' in the training data. Thus, it makes sense that these two probability values are equal, since they both receive the probability mass allocated for unseen events. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (2) Decoding word sequences "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can use the POS Tagger that has been trained to tag actual sentences. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "die             DET  \n",
      "Hölle           NOUN \n",
      "ist             VERB \n",
      "leer            VERB \n",
      ",               .    \n",
      "alle            PRON \n",
      "Teufel          NOUN \n",
      "sind            VERB \n",
      "hier.           VERB \n",
      ".               .    \n"
     ]
    }
   ],
   "source": [
    "words = 'die Hölle ist leer , alle Teufel sind hier.  .'.split()\n",
    "tags = POSTagger.decode(words)\n",
    "\n",
    "for word, tag in zip(words, tags):\n",
    "    print(\"{0:15} {1:5}\".format(word, tag))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's cool. But it seems that the tagger makes some errors. Lets investigate the taggers's performance on another sentence. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Im              ADP  \n",
      "Juli            NOUN \n",
      "2012            NUM  \n",
      "erhielt         VERB \n",
      "GitHub          ADP  \n",
      "eine            DET  \n",
      "Investition     NOUN \n",
      "von             ADP  \n",
      "100             NUM  \n",
      "Millionen       NOUN \n",
      ".               .    \n"
     ]
    }
   ],
   "source": [
    "words = 'Im Juli 2012 erhielt GitHub eine Investition von 100 Millionen .'.split()\n",
    "tags = POSTagger.decode(words)\n",
    "\n",
    "for word, tag in zip(words, tags):\n",
    "    print(\"{0:15} {1:5}\".format(word, tag))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems like all words have been tagged correctly, except for the proper noun 'GitHub', which has been incorrectly tagged as 'ADP, or adposition (this categroy group postpositions and prepositions).  \n",
    "\n",
    "Why was this word confusing? We can check if the word 'GitHub' is actually an unknown word. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'Github' in POSTagger.word_vocab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The word 'Github' is indeed an unknown word, emission probabilities won't help much in deciding the correct tag here.\n",
    "\n",
    "However, add-one smoothing is not an optimal smoothing technique. A better smoothing technique is the so-called absolute discounting. The developed HMMTagger class enables absolute discounting as a smoothing technique as well, but we need to decide the discounting parameters d, which a floating-point value between [0, 1]. Lets create another POS tagger with absolute discounting as a smoothing technique.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "POSTagger_disc = HMMTagger(train_corpus, smoothing='abs_disc', d=0.4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use the new tagger object to tag the same sentece as before and display the tags"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Im              ADP  \n",
      "Juli            NOUN \n",
      "2012            NUM  \n",
      "erhielt         VERB \n",
      "GitHub          NOUN \n",
      "eine            DET  \n",
      "Investition     NOUN \n",
      "von             ADP  \n",
      "100             NUM  \n",
      "Millionen       NOUN \n",
      ".               .    \n"
     ]
    }
   ],
   "source": [
    "tags_disc = POSTagger_disc.decode(words)\n",
    "\n",
    "for word, tag in zip(words, tags_disc):\n",
    "    print(\"{0:15} {1:5}\".format(word, tag))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time the word 'GitHub' has been tagged correctly as a 'NOUN'. Therefore, it seems that smoothing is a very crucial aspect with respect to the performance of the HMM-based POS tagger.  Lets check our first sentence with the new absolute discounting-based tagger. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "die             DET  \n",
      "Hölle           NOUN \n",
      "ist             VERB \n",
      "leer            ADJ  \n",
      ",               .    \n",
      "alle            PRON \n",
      "Teufel          NOUN \n",
      "sind            VERB \n",
      "hier            ADV  \n",
      ".               .    \n"
     ]
    }
   ],
   "source": [
    "words = 'die Hölle ist leer , alle Teufel sind hier  .'.split()\n",
    "tags = POSTagger_disc.decode(words)\n",
    "\n",
    "for word, tag in zip(words, tags):\n",
    "    print(\"{0:15} {1:5}\".format(word, tag))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Much better indeed! Absolute discounting as a smoothing technique might take some time to understand. But it really works very well compared to add-one smoothing. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (3) POS Taggers Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have two POS taggers, one uses add-one smoothing and the other uses absolute discounting. We can quantitatively evaluate the performace of the two taggers using an disjoint test set. Lets read the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test_corpus = nltk.corpus.ConllCorpusReader(\n",
    "    directory, test_fileid, ['words'], tagset='universal', encoding='utf8'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the tagger on the test data, and save the tagged words in file. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with open('tagged_add_one.tt', 'w') as op_file:\n",
    "    for sent in test_corpus.sents():\n",
    "        predicted_tags = POSTagger.decode(sent)\n",
    "\n",
    "        for w, tag in zip(sent, predicted_tags):\n",
    "            op_file.write(w + '\\t' + tag + '\\n')\n",
    "\n",
    "        op_file.write('\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To evaluate the POS tagger, the evaluation script can be used as follows. The file 'data/de-eval.tt' the gold annotations of the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Comparing gold file \"data/de-eval.tt\" and system file \"tagged_add_one.tt\"\n",
      "\n",
      "Precision, recall, and F1 score:\n",
      "\n",
      "  DET 0.7190 0.9808 0.8297\n",
      " NOUN 0.8934 0.8442 0.8681\n",
      " VERB 0.8613 0.8630 0.8621\n",
      "  ADP 0.8598 0.9852 0.9182\n",
      "    . 0.9243 0.9803 0.9515\n",
      " CONJ 0.9464 0.8652 0.9040\n",
      " PRON 0.8111 0.7862 0.7985\n",
      "  ADV 0.8847 0.7506 0.8121\n",
      "  ADJ 0.7809 0.6111 0.6857\n",
      "  NUM 1.0000 0.5593 0.7173\n",
      "  PRT 0.9468 0.8111 0.8737\n",
      "\n",
      "Accuracy: 0.8608\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%run eval.py data/de-eval.tt tagged_add_one.tt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The tagger based on add-one smoothing gives about 86.1% accuracy, which not impressive for the POS tagging problem. Now lets check the performance of the tagger with absolute discounting as smoothing. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with open('tagged_abs_disc.tt', 'w') as op_file:\n",
    "    for sent in test_corpus.sents():\n",
    "        predicted_tags = POSTagger_disc.decode(sent)\n",
    "\n",
    "        for w, tag in zip(sent, predicted_tags):\n",
    "            op_file.write(w + '\\t' + tag + '\\n')\n",
    "\n",
    "        op_file.write('\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And there we go .. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Comparing gold file \"data/de-eval.tt\" and system file \"tagged_abs_disc.tt\"\n",
      "\n",
      "Precision, recall, and F1 score:\n",
      "\n",
      "  DET 0.9092 0.9761 0.9415\n",
      " NOUN 0.8476 0.9835 0.9105\n",
      " VERB 0.9605 0.8712 0.9137\n",
      "  ADP 0.9632 0.9762 0.9697\n",
      "    . 0.9983 0.9992 0.9987\n",
      " CONJ 0.9544 0.8974 0.9250\n",
      " PRON 0.9391 0.8309 0.8817\n",
      "  ADV 0.9234 0.7893 0.8511\n",
      "  ADJ 0.7993 0.6485 0.7160\n",
      "  NUM 0.9906 0.7778 0.8714\n",
      "  PRT 0.8730 0.8730 0.8730\n",
      "    X 0.2000 0.0909 0.1250\n",
      "\n",
      "Accuracy: 0.9136\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%run eval.py data/de-eval.tt tagged_abs_disc.tt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Et voila!!! Absolute discounting is way better. The accuracy went up to 91.4%. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (4) Final remarks "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this Jupyter notebook I shared some ideas about the HMM-based POS tagger I developed for German. There are a few things that I am willing to investigate or improve in the tagger.  \n",
    "\n",
    "* Investigate the impact of the discounting parameter on the performance. \n",
    "* Develope a trigram HMM. That is, transition probabilities should be conditioned on the two previous states instead of a single previous state.\n",
    "* Visualise the Viterbi matrix.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}