Text_Summarization_Using_NLTK.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text_Summarization_Using_NLTK\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 01\n",
    "#### Convert the input text to a list of sentences. Then, compute the number of sentences in the given Text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Task 01 - Convert the input text to a list of sentences. Then, compute the number of sentences in the given Text. \n",
      "\n",
      "79\n"
     ]
    }
   ],
   "source": [
    "print(\"Task 01 - Convert the input text to a list of sentences. Then, compute the number of sentences in the given Text. \\n\")\n",
    "\n",
    "import nltk.data\n",
    "from nltk import tokenize\n",
    "import string\n",
    "\n",
    "with open('Article.txt', 'r') as file:\n",
    "    text = file.read().replace('Â', '').replace('â€”','').replace('\\xa0',' ')\n",
    "text_sentences=tokenize.sent_tokenize(text)\n",
    "sentence_count=len(text_sentences)\n",
    "print(sentence_count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 02\n",
    "#### Calculate the frequency of words in each sentence: the output should be a dictionary where each key is a sentence and the value is also a dictionary of word frequency."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Task 02 - Calculate the frequency of words in each sentence: the output should be a dictionary where each key is a sentence and the value is also a dictionary of word frequency. \n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"Task 02 - Calculate the frequency of words in each sentence: the output should be a dictionary where each key is a sentence and the value is also a dictionary of word frequency. \\n\")\n",
    "\n",
    "wordFrequencyAllSentences = {} #dictionary to store dictionary of  sentences and wordFrequencyPerSentence\n",
    "\n",
    "for sentence in text_sentences:\n",
    "    wordFrequencyPerSentence = {} # dictionary to store word and its count\n",
    "    word_list=sentence.split()\n",
    "    \n",
    "    for word in word_list:        \n",
    "        wordFrequencyPerSentence[word]=word_list.count(word) # repeating words will just overwrite the count of words so no problem\n",
    "        wordFrequencyAllSentences[sentence] = wordFrequencyPerSentence\n",
    "\n",
    "# wordFrequencyAllSentences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 03\n",
    "#### Calculate Term frequency for each word in a sentence.\n",
    "#### TF(word) = (Number of times term \"word\" appears in a sentence)  /  (Total number of terms in the sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Task 03 - Calculate Term frequency for each word in a sentence. \n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"Task 03 - Calculate Term frequency for each word in a sentence. \\n\")\n",
    "\n",
    "from nltk import word_tokenize\n",
    "\n",
    "TFwordAllSentences = {} #dictionary to store dictionary of  sentences and TFWordPerSentence\n",
    "\n",
    "for sentence in text_sentences:\n",
    "    TFWordPerSentence = {} # dictionary to store word and TF(word)\n",
    "#     word_list=sentence.split()\n",
    "    word_list=word_tokenize(sentence.lower())\n",
    "    word_list=[word.lower() for word in word_list if word.isalpha()] # removing punctuations\n",
    "    \n",
    "    for word in word_list:        \n",
    "        TFWordPerSentence[word]=word_list.count(word) / len(word_list)\n",
    "        TFwordAllSentences[sentence] = TFWordPerSentence\n",
    "\n",
    "# TFwordAllSentences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 04\n",
    "#### Create a matrix termFrequency: The termFrequency matrix should be a dictionary where each key is a sentence and the value is also a dictionary of word frequency."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Task 04 - Create a matrix termFrequency: The termFrequency matrix should be a dictionary where each key is a sentence and the value is also a dictionary of word frequency. \n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"Task 04 - Create a matrix termFrequency: The termFrequency matrix should be a dictionary where each key is a sentence and the value is also a dictionary of word frequency. \\n\")\n",
    "\n",
    "# already calculated in 1.2\n",
    "termFrequencyMatrix = wordFrequencyAllSentences; \n",
    "# termFrequencyMatrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 05\n",
    "#### For each word compute how many sentences contain that word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Task 05 - For each word compute how many sentences contain that word. \n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"Task 05 - For each word compute how many sentences contain that word. \\n\")\n",
    "\n",
    "from nltk import word_tokenize\n",
    "allWords = word_tokenize(text.lower())\n",
    "\n",
    "allWords=[word.lower() for word in allWords if word.isalpha()] # removing punctuations\n",
    "\n",
    "allWords=set(allWords) # getting unique words in text\n",
    "\n",
    "word_count= dict()\n",
    "\n",
    "for word in allWords:\n",
    "    count=0\n",
    "    for sentence in text_sentences:\n",
    "        if(word in sentence.lower()): # check if word is present in sentence            \n",
    "            count=count+1 # count sentene\n",
    "    word_count[word]=count\n",
    "# word_count"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 06\n",
    "#### Calculate IDF for each word in a sentence.\n",
    "#### IDF(word)  =  log e(Total number of sentences  /   number of sentences with term word in it)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Task 06 - Calculate IDF for each word in a sentence. \n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"Task 06 - Calculate IDF for each word in a sentence. \\n\")\n",
    "\n",
    "import math\n",
    "\n",
    "IDFwordsAllSentences = {} #dictionary to store dictionary of  sentences and IDFwordPerSentence\n",
    "\n",
    "totalSentences=len(text_sentences) ## for using in formula\n",
    "\n",
    "for sentence in text_sentences:\n",
    "    IDFwordPerSentence = {} # dictionary to store word and its IDF\n",
    "\n",
    "    word_list=word_tokenize(sentence.lower())\n",
    "    word_list=[word.lower() for word in word_list if word.isalpha()] # removing punctuations\n",
    "        \n",
    "    for word in word_list:\n",
    "        if(word_count[word]==0):\n",
    "            pass\n",
    "        else:\n",
    "            IDFwordPerSentence[word]= math.log10(totalSentences / word_count[word]) # repeating words will just overwrite the count of words so no problem\n",
    "            IDFwordsAllSentences[sentence] = IDFwordPerSentence\n",
    "# IDFwordsAllSentences\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 07\n",
    "#### Compute the TF-IDF for each word in each sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Task 07 - Compute the TF-IDF for each word in each sentence. \n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"Task 07 - Compute the TF-IDF for each word in each sentence. \\n\")\n",
    "\n",
    "TF_IFDwordAllSentences=TFwordAllSentences\n",
    "\n",
    "for key1, value1 in TFwordAllSentences.items():\n",
    "        for key2, value2 in value1.items():\n",
    "            TF_IFDwordAllSentences[key1][key2]=TFwordAllSentences[key1][key2] * IDFwordsAllSentences[key1][key2]\n",
    "            break\n",
    "# print(TF_IFDwordAllSentences)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 08\n",
    "#### Use the TF-IDF computed in (7) and give a weight for each sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Task 08 - Use the TF-IDF computed in (7) and give a weight for each sentence. \n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"Task 08 - Use the TF-IDF computed in (7) and give a weight for each sentence. \\n\")\n",
    "\n",
    "weight_sentences={} # for storing weights of all sentences\n",
    "\n",
    "for key1, value1 in TF_IFDwordAllSentences.items():\n",
    "    x=0 # initial score for a sentence\n",
    "    for key2, value2 in value1.items():\n",
    "        x=x+TF_IFDwordAllSentences[key1][key2] # weight of a sentence is the sum of tf_idf / length of sentence\n",
    "    weight_sentences[key1]=x / len(key1)\n",
    "\n",
    "# weight_sentences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 09\n",
    "#### Threshold: compute the average sentence weight"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Task 09 - Threshold: compute the average sentence weight \n",
      "\n",
      "0.010140202742075824\n"
     ]
    }
   ],
   "source": [
    "print(\"Task 09 - Threshold: compute the average sentence weight \\n\")\n",
    "\n",
    "averageSentenceWeight = sum(list(weight_sentences.values())) / len(weight_sentences)\n",
    "print(averageSentenceWeight)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 10\n",
    "#### Generate the summary : select a sentence for summarization if the weight of the sentence exceeds the threshold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Task 10 - Generate the summary : select a sentence for summarization if the weight of the sentence exceeds the threshold. \n",
      "\n",
      "If you haven't subscribed yet, you can do that by clicking here.\n",
      "Will it help if everyone wears a mask?\n",
      "Is keeping everyone two metres apart far enough?\n",
      "Here's a closer look at that research and what it might reveal.\n",
      "How do scientists think COVID-19 is transmitted?\n",
      "The WHO says it's important to stay \"more than one metre away\" from a person who is sick.\n",
      "Why is 2 metres the recommended distance for preventing transmission?\n",
      "Then, in 1934, W.F.\n",
      "Wells proposed that could explain how diseases are transmitted.\n",
      "Why don't experts think the virus is airborne?\n",
      "Loeb said that's just a \"signal\" that part of the virus was there.\n",
      "\"Does it mean that COVID-19 is spreading from person to person through aerosols?\n",
      "I would say definitively not,\" Loeb said.\n",
      "Is there evidence the virus could be spread farther than 2 metres?\n",
      "Within the chamber, droplets remained suspended for up to three minutes.\n",
      "What does that say about the 2-metre guideline?\n",
      "Mubareka stands by the two-metre guideline despite the findings of her cough chamber study.\n",
      "Second OpinionShould masks be mandatory in public to stop the spread of COVID-19?\n",
      "\"And that's really the key variable  that's what really determines your risk,\" she said.\n",
      "\"Those are the kinds of things we haven't been able to measure.\"\n",
      "\"They're basically saying what's theoretically possible,\" he said.\n",
      "But are coughing and sneezing all we need to worry about?\n",
      "\"It's just that maybe the dispersion is a little bit more limited.\"\n",
      "It found the louder someone spoke, the more droplets were emitted.\n",
      "So what about using masks to curb the spread of COVID-19?\n",
      "Some does leak out the sides, top and bottom, but without much momentum.\n",
      "Many governments haven't waited.\n"
     ]
    }
   ],
   "source": [
    "print(\"Task 10 - Generate the summary : select a sentence for summarization if the weight of the sentence exceeds the threshold. \\n\")\n",
    "\n",
    "# print([key for key, value in weight_sentences.items() if value>averageSentenceWeight])\n",
    "\n",
    "for key, value in weight_sentences.items():\n",
    "    if(value>averageSentenceWeight):\n",
    "        print(key)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.7.7 64-bit",
   "language": "python",
   "name": "python37764bit65e12efdfba64cce8bf5305c88373520"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}