Chapter 5.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "import datetime\n",
    "from nltk.corpus import brown"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**1. Search the web for \"spoof newspaper headlines\", to find such gems as: ** *British Left Waffles on Falkland Islands* **, and ** *Juvenile Court to Try Shooting Defendant* **. Manually tag these headlines to see if knowledge of the part-of-speech tags removes the ambiguity.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Juvenile', 'NOUN'),\n",
       " ('Court', 'NOUN'),\n",
       " ('to', 'PRT'),\n",
       " ('Try', 'VERB'),\n",
       " ('Shooting', 'ADJ'),\n",
       " ('Defendant', 'NOUN')]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# here TRY means to examine evidence in court and decide whether sb is innocent or guilty\n",
    "headline = 'Juvenile/NOUN Court/NOUN to/PRT Try/VERB Shooting/ADJ Defendant/NOUN'\n",
    "[nltk.tag.str2tuple(t) for t in headline.split()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**2. Working with someone else, take turns to pick a word that can be either a noun or a verb (e.g. ** *contest* **); the opponent has to predict which one is likely to be the most frequent in the Brown corpus; check the opponent's prediction, and tally the score over several turns.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Omitted."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**3. Tokenize and tag the following sentence: ** *They wind back the clock, while we chase after the wind.* ** What different pronunciations and parts of speech are involved?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('They', 'PRP'),\n",
       " ('wind', 'VBP'),\n",
       " ('back', 'RB'),\n",
       " ('the', 'DT'),\n",
       " ('clock', 'NN'),\n",
       " (',', ','),\n",
       " ('while', 'IN'),\n",
       " ('we', 'PRP'),\n",
       " ('chase', 'VBP'),\n",
       " ('after', 'IN'),\n",
       " ('the', 'DT'),\n",
       " ('wind', 'NN'),\n",
       " ('.', '.')]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sent = 'They wind back the clock, while we chase after the wind.'\n",
    "nltk.pos_tag(nltk.word_tokenize(sent))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**4. Review the mappings in 3.1. Discuss any other examples of mappings you can think of. What type of information do they map from and to?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| More Linguistic Object | Maps From | Maps To |\n",
    "| --- | --- | --- |\n",
    "| Word Frequency | Word | Number of occurrences in a text |\n",
    "| Word Prounciation | Word | List of the word's prounciation |\n",
    "| Abbreviation | Acronym | List of the full name |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**5. Using the Python interpreter in interactive mode, experiment with the dictionary examples in this chapter. Create a dictionary ** `d` **, and add some entries. What happens if you try to access a non-existent entry, e.g. ** `d['xyz']` **?** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "Traceback (most recent call last):\n",
    "  File \"<stdin>\", line 1, in <module>\n",
    "KeyError: 'xyz'\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**6. Try deleting an element from a dictionary d, using the syntax ** `del d['abc']` **. Check that the item was deleted.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Omitted."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**7. Create two dictionaries, ** `d1` ** and ** `d2` **, and add some entries to each. Now issue the command ** `d1.update(d2)` **. What did this do? What might it be useful for?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'hello': 1, 'language': 4, 'natural': 3, 'processing': 5, 'world': 2}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d1 = {'hello': 1, 'world': 2, 'natural': 0}\n",
    "d2 = {'natural': 3, 'language': 4, 'processing': 5}\n",
    "d1.update(d2)\n",
    "d1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Update the dictionary with the key/value pairs from other, overwriting existing keys.  \n",
    "Useful when merging two dictionaries."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**8. Create a dictionary ** `e` **, to represent a single lexical entry for some word of your choice. Define keys like ** `headword, part-of-speech, sense, ` ** and ** `example` **, and assign them suitable values.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "e = {}\n",
    "e['headword'] = ['NOUN', 'a word or term placed at the beginning (as of a chapter or an entry in an encyclopedia)']\n",
    "e['part-of-speech'] = ['PHRASE', 'a traditional class of words distinguished according to the kind of idea denoted and the function performed in a sentence']\n",
    "e['sense'] = ['NOUN', 'a meaning conveyed or intended']\n",
    "e['example'] = ['NOUN', 'one that serves as a pattern to be imitated or not to be imitated']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**9. Satisfy yourself that there are restrictions on the distribution of ** *go* ** and ** *went* **, in the sense that they cannot be freely interchanged in the kinds of contexts illustrated in (3d) in 7.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "'We *went* on the excursion.' means the tense is past.  \n",
    "'We *go* on the excursion.' well, is it unlikely used in daily life?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**10. Train a unigram tagger and run it on some new text. Observe that some words are not assigned a tag. Why not?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('hello', None),\n",
       " ('world', 'NN'),\n",
       " ('natural', 'JJ'),\n",
       " ('language', 'NN'),\n",
       " ('processing', 'NN')]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "brown_tagged_sents = brown.tagged_sents(categories='news')\n",
    "unigram_tagger = nltk.UnigramTagger(brown_tagged_sents)\n",
    "\n",
    "test_text = ['hello', 'world', 'natural', 'language', 'processing']\n",
    "unigram_tagger.tag(test_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The words doesn't appear in the training text, and therefore the tagger can't speculate the word's tag."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**11. Learn about the affix tagger (type ** `help(nltk.AffixTagger)` **). Train an affix tagger and run it on some new text. Experiment with different settings for the affix length and the minimum word length. Discuss your findings.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Experiment', 'NN-TL'),\n",
       " ('with', None),\n",
       " ('different', 'JJ'),\n",
       " ('settings', 'NN'),\n",
       " ('for', None),\n",
       " ('the', None),\n",
       " ('affix', None),\n",
       " ('length', None),\n",
       " ('and', None),\n",
       " ('the', None),\n",
       " ('minimum', 'NNS'),\n",
       " ('word', None),\n",
       " ('length', None)]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# help(nltk.AffixTagger)\n",
    "affix_tagger = nltk.AffixTagger(brown_tagged_sents, affix_length=3, min_stem_length=4)\n",
    "test_text = 'Experiment with different settings for the affix length and the minimum word length'.split()\n",
    "affix_tagger.tag(test_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**12. Train a bigram tagger with no backoff tagger, and run it on some of the training data. Next, run it on some new data. What happens to the performance of the tagger? Why?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Temporarily omitted."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**13. We can use a dictionary to specify the values to be substituted into a formatting string. Read Python's library documentation for formatting strings http://docs.python.org/lib/typesseq-strings.html** *(404 NOT FOUND)* ** and use this method to display today's date in two different formats.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2018-08-31'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datetime.datetime.today().strftime(\"%Y-%m-%d\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**14. Use ** `sorted()` ** and ** `set()` ** to get a sorted list of tags used in the Brown corpus, removing duplicates.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "list_of_tags = sorted(set([tag for (_, tag) in brown.tagged_words()]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**15. Write programs to process the Brown Corpus and find answers to the following questions:**  \n",
    "a. **Which nouns are more common in their plural form, rather than their singular form? (Only consider regular plurals, formed with the ** *-s* ** suffix.)**  \n",
    "b. **Which word has the greatest number of distinct tags. What are they, and what do they represent?**  \n",
    "c. **List tags in order of decreasing frequency. What do the 20 most frequent tags represent?**  \n",
    "d. **Which tags are nouns most commonly found after? What do these tags represent?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "brown_tagged = brown.tagged_words()\n",
    "cfd = nltk.ConditionalFreqDist(brown_tagged)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Which nouns are more common in their plural form, rather than their singular form? \n",
    "# (Only consider regular plurals, formed with the -s suffix.)\n",
    "\n",
    "common_plural = set()\n",
    "for word in set(brown.words()):\n",
    "    if cfd[word+'s']['NNS'] > cfd[word]['NN']:\n",
    "        common_plural.add(word)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Which word has the greatest number of distinct tags. What are they, and what do they represent?\n",
    "\n",
    "tag_dict = {k:len(cfd[k]) for k in cfd}\n",
    "greatest = max(tag_dict, key=lambda key: tag_dict[key])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*that*  \n",
    "CS, CS-HL, CS-NC, DT, DT-NC, NIL, QL, WPO, WPO-NC, WPS, WPS-HL, WPS-NC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('NN', 152470),\n",
       " ('IN', 120557),\n",
       " ('AT', 97959),\n",
       " ('JJ', 64028),\n",
       " ('.', 60638),\n",
       " (',', 58156),\n",
       " ('NNS', 55110),\n",
       " ('CC', 37718),\n",
       " ('RB', 36464),\n",
       " ('NP', 34476),\n",
       " ('VB', 33693),\n",
       " ('VBN', 29186),\n",
       " ('VBD', 26167),\n",
       " ('CS', 22143),\n",
       " ('PPS', 18253),\n",
       " ('VBG', 17893),\n",
       " ('PP$', 16872),\n",
       " ('TO', 14918),\n",
       " ('PPSS', 13802),\n",
       " ('CD', 13510)]"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# List tags in order of decreasing frequency. What do the 20 most frequent tags represent?\n",
    "\n",
    "helper_list = [t for (_, t) in brown_tagged]    # extract the tags to a list \n",
    "fd = nltk.FreqDist(helper_list)\n",
    "fd.most_common(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['IN', '.', ',', 'CC', 'NN', 'NNS', 'VBD', 'CS', 'MD', 'BEZ']"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Which tags are nouns most commonly found after? What do these tags represent?\n",
    "\n",
    "word_tag_pairs = nltk.bigrams(brown_tagged)\n",
    "noun_after = [b[1] for (a, b) in word_tag_pairs if a[1].startswith('NN')]\n",
    "fdist = nltk.FreqDist(noun_after)\n",
    "[tag for (tag, _) in fdist.most_common(10)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**16. Explore the following issues that arise in connection with the lookup tagger:**  \n",
    "a. **What happens to the tagger performance for the various model sizes when a backoff tagger is omitted?**  \n",
    "b. **Consider the curve in 4.2; suggest a good size for a lookup tagger that balances memory and performance. Can you come up with scenarios where it would be preferable to minimize memory usage, or to maximize performance with no regard for memory usage?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When a backoff tagger is omitted, with the increase of model sizes, the tagger performance would be improved since there would be less UNKNOWN words.  \n",
    "If the memory usage is limited, then a 90% performance is advisable(about 8000 in Figure 4.2). Use as large model size as possible with no regard for memory usage.(Well, take overfitting and calculating time into consideration as well =D)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**17. What is the upper limit of performance for a lookup tagger, assuming no limit to the size of its table? (Hint: write a program to work out what percentage of tokens of a word are assigned the most likely tag for that word, on average.)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The word's most possible tag's proportion of all that word's tags?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**18. Generate some statistics for tagged data to answer the following questions:**  \n",
    "a. **What proportion of word types are always assigned the same part-of-speech tag?**  \n",
    "b. **How many words are ambiguous, in the sense that they appear with at least two tags?**  \n",
    "c. **What percentage of word ** *tokens* ** in the Brown Corpus involve these ambiguous words?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "brown_tag = brown.tagged_words(tagset='universal')\n",
    "cfd = nltk.ConditionalFreqDist(brown_tag)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "proportion = sum(1 for word in cfd if len(cfd[word]) == 1) / len(cfd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "ambiguous = sum(1 for word in cfd if len(cfd[word]) > 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**19. The ** `evaluate()` ** method works out how accurately the tagger performs on this text. For example, if the supplied tagged text was ** `[('the', 'DT'), ('dog', 'NN')]` ** and the tagger produced the output ** `[('the', 'NN'), ('dog', 'NN')]` **, then the score would be ** `0.5` **. Let's try to figure out how the evaluation method works:**  \n",
    "a. **A tagger ** `t` ** takes a list of words as input, and produces a list of tagged words as output. However,** ` t.evaluate()` ** is given correctly tagged text as its only parameter. What must it do with this input before performing the tagging?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}