p.01 Tokenizing words and Sentences.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# nltk.download()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### tokenizing\n",
    "- word tokenizers \n",
    "- sentence tokenizers\n",
    "\n",
    "#### lexicon and corporas\n",
    "\n",
    "1. corpora - body of text. eg - presedential speeches, medical journals, english language\n",
    "2. lexicon - words and their meanings \n",
    "    \n",
    "    - investor-speak\n",
    "    - regular english-speak\n",
    "    \n",
    "    E.g:\n",
    "        - investor-speak \"bull\": someone who is positive about the market.\n",
    "        - regular english-speak \"bull\":scary animal you do not want running at you.\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.tokenize import word_tokenize, sent_tokenize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sentence list:\n",
      "\n",
      "['\\nContrary to popular belief, Lorem Ipsum is not simply random text.', 'It has roots in a piece of classical Latin literature from 45 BC, making it over 2000 years old.', 'Richard McClintock, a Latin professor at Hampden-Sydney College in Virginia, looked up one of the more obscure Latin words, consectetur, from a Lorem Ipsum passage, and going through the cites of the word in classical literature, discovered the undoubtable source.', 'Lorem Ipsum comes from sections 1.10.32 and 1.10.33 of \"de Finibus Bonorum et Malorum\" (The Extremes of Good and Evil) by Cicero, written in 45 BC.', 'This book is a treatise on the theory of ethics, very popular during the Renaissance.', 'The first line of Lorem Ipsum, \"Lorem ipsum dolor sit amet..\", comes from a line in section 1.10.32.', 'The standard chunk of Lorem Ipsum used since the 1500s is reproduced below for those interested.', 'Sections 1.10.32 and 1.10.33 from \"de Finibus Bonorum et Malorum\" by Cicero are also reproduced in their exact original form, accompanied by English versions from the 1914 translation by H. Rackham.']\n",
      "\n",
      "word list:\n",
      "\n",
      "['Contrary', 'to', 'popular', 'belief', ',', 'Lorem', 'Ipsum', 'is', 'not', 'simply', 'random', 'text', '.', 'It', 'has', 'roots', 'in', 'a', 'piece', 'of', 'classical', 'Latin', 'literature', 'from', '45', 'BC', ',', 'making', 'it', 'over', '2000', 'years', 'old', '.', 'Richard', 'McClintock', ',', 'a', 'Latin', 'professor', 'at', 'Hampden-Sydney', 'College', 'in', 'Virginia', ',', 'looked', 'up', 'one', 'of', 'the', 'more', 'obscure', 'Latin', 'words', ',', 'consectetur', ',', 'from', 'a', 'Lorem', 'Ipsum', 'passage', ',', 'and', 'going', 'through', 'the', 'cites', 'of', 'the', 'word', 'in', 'classical', 'literature', ',', 'discovered', 'the', 'undoubtable', 'source', '.', 'Lorem', 'Ipsum', 'comes', 'from', 'sections', '1.10.32', 'and', '1.10.33', 'of', '``', 'de', 'Finibus', 'Bonorum', 'et', 'Malorum', \"''\", '(', 'The', 'Extremes', 'of', 'Good', 'and', 'Evil', ')', 'by', 'Cicero', ',', 'written', 'in', '45', 'BC', '.', 'This', 'book', 'is', 'a', 'treatise', 'on', 'the', 'theory', 'of', 'ethics', ',', 'very', 'popular', 'during', 'the', 'Renaissance', '.', 'The', 'first', 'line', 'of', 'Lorem', 'Ipsum', ',', '``', 'Lorem', 'ipsum', 'dolor', 'sit', 'amet', '..', \"''\", ',', 'comes', 'from', 'a', 'line', 'in', 'section', '1.10.32', '.', 'The', 'standard', 'chunk', 'of', 'Lorem', 'Ipsum', 'used', 'since', 'the', '1500s', 'is', 'reproduced', 'below', 'for', 'those', 'interested', '.', 'Sections', '1.10.32', 'and', '1.10.33', 'from', '``', 'de', 'Finibus', 'Bonorum', 'et', 'Malorum', \"''\", 'by', 'Cicero', 'are', 'also', 'reproduced', 'in', 'their', 'exact', 'original', 'form', ',', 'accompanied', 'by', 'English', 'versions', 'from', 'the', '1914', 'translation', 'by', 'H.', 'Rackham', '.']\n"
     ]
    }
   ],
   "source": [
    "example_text = '''\n",
    "Contrary to popular belief, Lorem Ipsum is not simply random text. It has roots in a piece of classical Latin literature from 45 BC, making it over 2000 years old. Richard McClintock, a Latin professor at Hampden-Sydney College in Virginia, looked up one of the more obscure Latin words, consectetur, from a Lorem Ipsum passage, and going through the cites of the word in classical literature, discovered the undoubtable source. Lorem Ipsum comes from sections 1.10.32 and 1.10.33 of \"de Finibus Bonorum et Malorum\" (The Extremes of Good and Evil) by Cicero, written in 45 BC. This book is a treatise on the theory of ethics, very popular during the Renaissance. The first line of Lorem Ipsum, \"Lorem ipsum dolor sit amet..\", comes from a line in section 1.10.32.\n",
    "\n",
    "The standard chunk of Lorem Ipsum used since the 1500s is reproduced below for those interested. Sections 1.10.32 and 1.10.33 from \"de Finibus Bonorum et Malorum\" by Cicero are also reproduced in their exact original form, accompanied by English versions from the 1914 translation by H. Rackham.'''\n",
    "print(\"sentence list:\\n\")\n",
    "print(sent_tokenize(example_text))\n",
    "print()\n",
    "print(\"word list:\\n\")\n",
    "print(word_tokenize(example_text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Contrary\n",
      "to\n",
      "popular\n",
      "belief\n",
      ",\n",
      "Lorem\n",
      "Ipsum\n",
      "is\n",
      "not\n",
      "simply\n",
      "random\n",
      "text\n",
      ".\n",
      "It\n",
      "has\n",
      "roots\n",
      "in\n",
      "a\n",
      "piece\n",
      "of\n",
      "classical\n",
      "Latin\n",
      "literature\n",
      "from\n",
      "45\n",
      "BC\n",
      ",\n",
      "making\n",
      "it\n",
      "over\n",
      "2000\n",
      "years\n",
      "old\n",
      ".\n",
      "Richard\n",
      "McClintock\n",
      ",\n",
      "a\n",
      "Latin\n",
      "professor\n",
      "at\n",
      "Hampden-Sydney\n",
      "College\n",
      "in\n",
      "Virginia\n",
      ",\n",
      "looked\n",
      "up\n",
      "one\n",
      "of\n",
      "the\n",
      "more\n",
      "obscure\n",
      "Latin\n",
      "words\n",
      ",\n",
      "consectetur\n",
      ",\n",
      "from\n",
      "a\n",
      "Lorem\n",
      "Ipsum\n",
      "passage\n",
      ",\n",
      "and\n",
      "going\n",
      "through\n",
      "the\n",
      "cites\n",
      "of\n",
      "the\n",
      "word\n",
      "in\n",
      "classical\n",
      "literature\n",
      ",\n",
      "discovered\n",
      "the\n",
      "undoubtable\n",
      "source\n",
      ".\n",
      "Lorem\n",
      "Ipsum\n",
      "comes\n",
      "from\n",
      "sections\n",
      "1.10.32\n",
      "and\n",
      "1.10.33\n",
      "of\n",
      "``\n",
      "de\n",
      "Finibus\n",
      "Bonorum\n",
      "et\n",
      "Malorum\n",
      "''\n",
      "(\n",
      "The\n",
      "Extremes\n",
      "of\n",
      "Good\n",
      "and\n",
      "Evil\n",
      ")\n",
      "by\n",
      "Cicero\n",
      ",\n",
      "written\n",
      "in\n",
      "45\n",
      "BC\n",
      ".\n",
      "This\n",
      "book\n",
      "is\n",
      "a\n",
      "treatise\n",
      "on\n",
      "the\n",
      "theory\n",
      "of\n",
      "ethics\n",
      ",\n",
      "very\n",
      "popular\n",
      "during\n",
      "the\n",
      "Renaissance\n",
      ".\n",
      "The\n",
      "first\n",
      "line\n",
      "of\n",
      "Lorem\n",
      "Ipsum\n",
      ",\n",
      "``\n",
      "Lorem\n",
      "ipsum\n",
      "dolor\n",
      "sit\n",
      "amet\n",
      "..\n",
      "''\n",
      ",\n",
      "comes\n",
      "from\n",
      "a\n",
      "line\n",
      "in\n",
      "section\n",
      "1.10.32\n",
      ".\n",
      "The\n",
      "standard\n",
      "chunk\n",
      "of\n",
      "Lorem\n",
      "Ipsum\n",
      "used\n",
      "since\n",
      "the\n",
      "1500s\n",
      "is\n",
      "reproduced\n",
      "below\n",
      "for\n",
      "those\n",
      "interested\n",
      ".\n",
      "Sections\n",
      "1.10.32\n",
      "and\n",
      "1.10.33\n",
      "from\n",
      "``\n",
      "de\n",
      "Finibus\n",
      "Bonorum\n",
      "et\n",
      "Malorum\n",
      "''\n",
      "by\n",
      "Cicero\n",
      "are\n",
      "also\n",
      "reproduced\n",
      "in\n",
      "their\n",
      "exact\n",
      "original\n",
      "form\n",
      ",\n",
      "accompanied\n",
      "by\n",
      "English\n",
      "versions\n",
      "from\n",
      "the\n",
      "1914\n",
      "translation\n",
      "by\n",
      "H.\n",
      "Rackham\n",
      ".\n"
     ]
    }
   ],
   "source": [
    "for word in word_tokenize(example_text):\n",
    "    print(word)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### There are nltk trainers using which you can build your own language!!!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}