NLPTextPreProcessing.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "NLPTextPreProcessing.ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyOf3BWvj8ao5ArTAiShJYZf",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Neel7317/NLP/blob/main/NLPTextPreProcessing.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O0znuZwm3ryY"
      },
      "source": [
        "import nltk"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eZvG6XdS4ELm"
      },
      "source": [
        "paragraph = \"\"\"I have three visions for India. In 3000 years of our history, people from all over \r\n",
        "               the world have come and invaded us, captured our lands, conquered our minds. \r\n",
        "               From Alexander onwards, the Greeks, the Turks, the Moguls, the Portuguese, the British,\r\n",
        "               the French, the Dutch, all of them came and looted us, took over what was ours. \r\n",
        "               Yet we have not done this to any other nation. We have not conquered anyone. \r\n",
        "               We have not grabbed their land, their culture, \r\n",
        "               their history and tried to enforce our way of life on them. \r\n",
        "               Why? Because we respect the freedom of others.That is why my \r\n",
        "               first vision is that of freedom. I believe that India got its first vision of \r\n",
        "               this in 1857, when we started the War of Independence. It is this freedom that\r\n",
        "               we must protect and nurture and build on. If we are not free, no one will respect us.\r\n",
        "               My second vision for India’s development. For fifty years we have been a developing nation.\r\n",
        "               It is time we see ourselves as a developed nation. We are among the top 5 nations of the world\r\n",
        "               in terms of GDP. We have a 10 percent growth rate in most areas. Our poverty levels are falling.\r\n",
        "               Our achievements are being globally recognised today. Yet we lack the self-confidence to\r\n",
        "               see ourselves as a developed nation, self-reliant and self-assured. Isn’t this incorrect?\r\n",
        "               I have a third vision. India must stand up to the world. Because I believe that unless India \r\n",
        "               stands up to the world, no one will respect us. Only strength respects strength. We must be \r\n",
        "               strong not only as a military power but also as an economic power. Both must go hand-in-hand. \r\n",
        "               My good fortune was to have worked with three great minds. Dr. Vikram Sarabhai of the Dept. of \r\n",
        "               space, Professor Satish Dhawan, who succeeded him and Dr. Brahm Prakash, father of nuclear material.\r\n",
        "               I was lucky to have worked with all three of them closely and consider this the great opportunity of my life. \r\n",
        "               I see four milestones in my career\"\"\"\r\n",
        "\r\n",
        "\r\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "L2Wn6q2u4X6_",
        "outputId": "37aab4ba-bfb7-4bf7-e812-901f9fe9d25c"
      },
      "source": [
        "#Tokenization\r\n",
        "nltk.download('punkt')\r\n",
        "sentence=nltk.sent_tokenize(paragraph)#convert paragraph to sentence"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vjK-jKVY5K4J",
        "outputId": "2219d671-64a0-42fe-a7f2-e4105869a8ad"
      },
      "source": [
        "sentence"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['I have three visions for India.',\n",
              " 'In 3000 years of our history, people from all over \\n               the world have come and invaded us, captured our lands, conquered our minds.',\n",
              " 'From Alexander onwards, the Greeks, the Turks, the Moguls, the Portuguese, the British,\\n               the French, the Dutch, all of them came and looted us, took over what was ours.',\n",
              " 'Yet we have not done this to any other nation.',\n",
              " 'We have not conquered anyone.',\n",
              " 'We have not grabbed their land, their culture, \\n               their history and tried to enforce our way of life on them.',\n",
              " 'Why?',\n",
              " 'Because we respect the freedom of others.That is why my \\n               first vision is that of freedom.',\n",
              " 'I believe that India got its first vision of \\n               this in 1857, when we started the War of Independence.',\n",
              " 'It is this freedom that\\n               we must protect and nurture and build on.',\n",
              " 'If we are not free, no one will respect us.',\n",
              " 'My second vision for India’s development.',\n",
              " 'For fifty years we have been a developing nation.',\n",
              " 'It is time we see ourselves as a developed nation.',\n",
              " 'We are among the top 5 nations of the world\\n               in terms of GDP.',\n",
              " 'We have a 10 percent growth rate in most areas.',\n",
              " 'Our poverty levels are falling.',\n",
              " 'Our achievements are being globally recognised today.',\n",
              " 'Yet we lack the self-confidence to\\n               see ourselves as a developed nation, self-reliant and self-assured.',\n",
              " 'Isn’t this incorrect?',\n",
              " 'I have a third vision.',\n",
              " 'India must stand up to the world.',\n",
              " 'Because I believe that unless India \\n               stands up to the world, no one will respect us.',\n",
              " 'Only strength respects strength.',\n",
              " 'We must be \\n               strong not only as a military power but also as an economic power.',\n",
              " 'Both must go hand-in-hand.',\n",
              " 'My good fortune was to have worked with three great minds.',\n",
              " 'Dr. Vikram Sarabhai of the Dept.',\n",
              " 'of \\n               space, Professor Satish Dhawan, who succeeded him and Dr. Brahm Prakash, father of nuclear material.',\n",
              " 'I was lucky to have worked with all three of them closely and consider this the great opportunity of my life.',\n",
              " 'I see four milestones in my career']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "avhu5P3c4bUM"
      },
      "source": [
        "word=nltk.word_tokenize(paragraph)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1_OyWb635Cmp"
      },
      "source": [
        "word"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OYQxsdB55DyR"
      },
      "source": [
        "#Stemming and Lemmatization with stopwords"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2V4E-8ho5co1",
        "outputId": "411b8143-c4a9-4f59-bc13-5d4e576d3ecd"
      },
      "source": [
        "from nltk.stem import PorterStemmer\r\n",
        "from nltk.corpus import stopwords\r\n",
        "nltk.download('stopwords')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/stopwords.zip.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2Y9hVun75wPA"
      },
      "source": [
        "stemm=PorterStemmer()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2GmfwUjE51W3"
      },
      "source": [
        "for i in range(len(sentence)):\r\n",
        "  words=nltk.word_tokenize(sentence[i])\r\n",
        "  word=[stemm.stem(word)for word in words if word not in set(stopwords.words('english'))]\r\n",
        "  sentence[i]=' '.join(word)\r\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HIY21RH_7Hqg",
        "outputId": "6e74e8d9-550a-44c8-ba13-8c235ceecd4d"
      },
      "source": [
        "sentence"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['I three vision india .',\n",
              " 'In 3000 year histori , peopl world come invad u , captur land , conquer mind .',\n",
              " 'alexand onward , greek , turk , mogul , portugues , british , french , dutch , came loot u , took .',\n",
              " 'yet done nation .',\n",
              " 'We conquer anyon .',\n",
              " 'We grab land , cultur , histori tri enforc way life .',\n",
              " 'whi ?',\n",
              " 'becaus respect freedom others.that first vision freedom .',\n",
              " 'I believ india got first vision 1857 , start war independ .',\n",
              " 'It freedom must protect nurtur build .',\n",
              " 'If free , one respect u .',\n",
              " 'My second vision india ’ develop .',\n",
              " 'fifti year develop nation .',\n",
              " 'It time see develop nation .',\n",
              " 'We among top 5 nation world term gdp .',\n",
              " 'We 10 percent growth rate area .',\n",
              " 'poverti level fall .',\n",
              " 'achiev global recognis today .',\n",
              " 'yet lack self-confid see develop nation , self-reli self-assur .',\n",
              " '’ incorrect ?',\n",
              " 'I third vision .',\n",
              " 'india must stand world .',\n",
              " 'becaus I believ unless india stand world , one respect u .',\n",
              " 'onli strength respect strength .',\n",
              " 'We must strong militari power also econom power .',\n",
              " 'must go hand-in-hand .',\n",
              " 'My good fortun work three great mind .',\n",
              " 'dr. vikram sarabhai dept .',\n",
              " 'space , professor satish dhawan , succeed dr. brahm prakash , father nuclear materi .',\n",
              " 'I lucki work three close consid great opportun life .',\n",
              " 'I see four mileston career']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CxIrUa-Q7OyF"
      },
      "source": [
        "#Lemmatization"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1pftXtt29Cj8",
        "outputId": "f2ede2eb-fb45-4d02-cb5a-257ada72510b"
      },
      "source": [
        "from nltk.stem import WordNetLemmatizer\r\n",
        "nltk.download('wordnet')\r\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data]   Package wordnet is already up-to-date!\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 31
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZfyPpwyY9VZD"
      },
      "source": [
        "lemma=WordNetLemmatizer()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oMQbil4Q-guu"
      },
      "source": [
        "sentences=nltk.sent_tokenize(paragraph)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "l-YbCKhS9ZCp"
      },
      "source": [
        "for i in range(len(sentences)):\r\n",
        "  words=nltk.word_tokenize(sentences[i])\r\n",
        "  word=[lemma.lemmatize(word)for word in words if word not in set(stopwords.words('english'))]\r\n",
        "  sentences[i]=' '.join(word)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oEvYwJl997A6",
        "outputId": "b6670b41-f27b-4992-da98-7e3bff355630"
      },
      "source": [
        "sentences#now after Lemmatization it gives results in meaningful words in sentences.."
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['I three vision India .',\n",
              " 'In 3000 year history , people world come invaded u , captured land , conquered mind .',\n",
              " 'From Alexander onwards , Greeks , Turks , Moguls , Portuguese , British , French , Dutch , came looted u , took .',\n",
              " 'Yet done nation .',\n",
              " 'We conquered anyone .',\n",
              " 'We grabbed land , culture , history tried enforce way life .',\n",
              " 'Why ?',\n",
              " 'Because respect freedom others.That first vision freedom .',\n",
              " 'I believe India got first vision 1857 , started War Independence .',\n",
              " 'It freedom must protect nurture build .',\n",
              " 'If free , one respect u .',\n",
              " 'My second vision India ’ development .',\n",
              " 'For fifty year developing nation .',\n",
              " 'It time see developed nation .',\n",
              " 'We among top 5 nation world term GDP .',\n",
              " 'We 10 percent growth rate area .',\n",
              " 'Our poverty level falling .',\n",
              " 'Our achievement globally recognised today .',\n",
              " 'Yet lack self-confidence see developed nation , self-reliant self-assured .',\n",
              " 'Isn ’ incorrect ?',\n",
              " 'I third vision .',\n",
              " 'India must stand world .',\n",
              " 'Because I believe unless India stand world , one respect u .',\n",
              " 'Only strength respect strength .',\n",
              " 'We must strong military power also economic power .',\n",
              " 'Both must go hand-in-hand .',\n",
              " 'My good fortune worked three great mind .',\n",
              " 'Dr. Vikram Sarabhai Dept .',\n",
              " 'space , Professor Satish Dhawan , succeeded Dr. Brahm Prakash , father nuclear material .',\n",
              " 'I lucky worked three closely consider great opportunity life .',\n",
              " 'I see four milestone career']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qBKLiqMs-Cx-"
      },
      "source": [
        "#create bag of word implementation with sklearn countVectoried method\r\n",
        "import nltk"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Vk6wrYj9O0GJ"
      },
      "source": [
        "paragraph = \"\"\"I have three visions for India. In 3000 years of our history, people from all over \r\n",
        "               the world have come and invaded us, captured our lands, conquered our minds. \r\n",
        "               From Alexander onwards, the Greeks, the Turks, the Moguls, the Portuguese, the British,\r\n",
        "               the French, the Dutch, all of them came and looted us, took over what was ours. \r\n",
        "               Yet we have not done this to any other nation. We have not conquered anyone. \r\n",
        "               We have not grabbed their land, their culture, \r\n",
        "               their history and tried to enforce our way of life on them. \r\n",
        "               Why? Because we respect the freedom of others.That is why my \r\n",
        "               first vision is that of freedom. I believe that India got its first vision of \r\n",
        "               this in 1857, when we started the War of Independence. It is this freedom that\r\n",
        "               we must protect and nurture and build on. If we are not free, no one will respect us.\r\n",
        "               My second vision for India’s development. For fifty years we have been a developing nation.\r\n",
        "               It is time we see ourselves as a developed nation. We are among the top 5 nations of the world\r\n",
        "               in terms of GDP. We have a 10 percent growth rate in most areas. Our poverty levels are falling.\r\n",
        "               Our achievements are being globally recognised today. Yet we lack the self-confidence to\r\n",
        "               see ourselves as a developed nation, self-reliant and self-assured. Isn’t this incorrect?\r\n",
        "               I have a third vision. India must stand up to the world. Because I believe that unless India \r\n",
        "               stands up to the world, no one will respect us. Only strength respects strength. We must be \r\n",
        "               strong not only as a military power but also as an economic power. Both must go hand-in-hand. \r\n",
        "               My good fortune was to have worked with three great minds. Dr. Vikram Sarabhai of the Dept. of \r\n",
        "               space, Professor Satish Dhawan, who succeeded him and Dr. Brahm Prakash, father of nuclear material.\r\n",
        "               I was lucky to have worked with all three of them closely and consider this the great opportunity of my life. \r\n",
        "               I see four milestones in my career\"\"\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wdKvpzjUO6lx"
      },
      "source": [
        "senten=nltk.sent_tokenize(paragraph)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZM3Gbq3PPOJ6"
      },
      "source": [
        "from nltk.stem import WordNetLemmatizer\r\n",
        "from nltk.stem import PorterStemmer\r\n",
        "from nltk.corpus import stopwords\r\n",
        "import re"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dp3-C8mqPv-f"
      },
      "source": [
        "ps=PorterStemmer()\r\n",
        "ls=WordNetLemmatizer()\r\n",
        "corpus=[]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "r4fAVqedP2l9"
      },
      "source": [
        "\r\n",
        "for i in range(len(senten)):\r\n",
        "  word=re.sub('[^a-zA-Z]',' ',senten[i])\r\n",
        "  word=word.lower()\r\n",
        "  word=word.split()\r\n",
        "  word=[ls.lemmatize(w) for w in word if w  not in set(stopwords.words('english'))]\r\n",
        "  word=' '.join(word)\r\n",
        "  corpus.append(word)\r\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1-7RBIdqS6q_",
        "outputId": "6986189d-cd9b-496a-c874-c53b62381059"
      },
      "source": [
        "corpus"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['three vision india',\n",
              " 'year history people world come invaded u captured land conquered mind',\n",
              " 'alexander onwards greek turk mogul portuguese british french dutch came looted u took',\n",
              " 'yet done nation',\n",
              " 'conquered anyone',\n",
              " 'grabbed land culture history tried enforce way life',\n",
              " '',\n",
              " 'respect freedom others first vision freedom',\n",
              " 'believe india got first vision started war independence',\n",
              " 'freedom must protect nurture build',\n",
              " 'free one respect u',\n",
              " 'second vision india development',\n",
              " 'fifty year developing nation',\n",
              " 'time see developed nation',\n",
              " 'among top nation world term gdp',\n",
              " 'percent growth rate area',\n",
              " 'poverty level falling',\n",
              " 'achievement globally recognised today',\n",
              " 'yet lack self confidence see developed nation self reliant self assured',\n",
              " 'incorrect',\n",
              " 'third vision',\n",
              " 'india must stand world',\n",
              " 'believe unless india stand world one respect u',\n",
              " 'strength respect strength',\n",
              " 'must strong military power also economic power',\n",
              " 'must go hand hand',\n",
              " 'good fortune worked three great mind',\n",
              " 'dr vikram sarabhai dept',\n",
              " 'space professor satish dhawan succeeded dr brahm prakash father nuclear material',\n",
              " 'lucky worked three closely consider great opportunity life',\n",
              " 'see four milestone career']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 57
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "55ZovSJ4RkmG"
      },
      "source": [
        "#convert to bag of word\r\n",
        "from sklearn.feature_extraction.text import CountVectorizer"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1xQGRNXySPTm"
      },
      "source": [
        "cv=CountVectorizer()\r\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ydKmH1BGScO6"
      },
      "source": [
        "X=cv.fit_transform(corpus).toarray()# Bag of word"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ys1tKBneSgRY",
        "outputId": "7dd94c37-c1b8-47cc-dc40-fc9e526b9e06"
      },
      "source": [
        "X"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0, 0, 0, ..., 0, 0, 0],\n",
              "       [0, 0, 0, ..., 1, 1, 0],\n",
              "       [0, 1, 0, ..., 0, 0, 0],\n",
              "       ...,\n",
              "       [0, 0, 0, ..., 0, 0, 0],\n",
              "       [0, 0, 0, ..., 0, 0, 0],\n",
              "       [0, 0, 0, ..., 0, 0, 0]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 56
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WRPrk9tegAtz"
      },
      "source": [
        "# **#Convert paragraph to Tf-Idf**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LbgvuiEvSqSg"
      },
      "source": [
        "paragraph = \"\"\"I have three visions for India. In 3000 years of our history, people from all over \r\n",
        "               the world have come and invaded us, captured our lands, conquered our minds. \r\n",
        "               From Alexander onwards, the Greeks, the Turks, the Moguls, the Portuguese, the British,\r\n",
        "               the French, the Dutch, all of them came and looted us, took over what was ours. \r\n",
        "               Yet we have not done this to any other nation. We have not conquered anyone. \r\n",
        "               We have not grabbed their land, their culture, \r\n",
        "               their history and tried to enforce our way of life on them. \r\n",
        "               Why? Because we respect the freedom of others.That is why my \r\n",
        "               first vision is that of freedom. I believe that India got its first vision of \r\n",
        "               this in 1857, when we started the War of Independence. It is this freedom that\r\n",
        "               we must protect and nurture and build on. If we are not free, no one will respect us.\r\n",
        "               My second vision for India’s development. For fifty years we have been a developing nation.\r\n",
        "               It is time we see ourselves as a developed nation. We are among the top 5 nations of the world\r\n",
        "               in terms of GDP. We have a 10 percent growth rate in most areas. Our poverty levels are falling.\r\n",
        "               Our achievements are being globally recognised today. Yet we lack the self-confidence to\r\n",
        "               see ourselves as a developed nation, self-reliant and self-assured. Isn’t this incorrect?\r\n",
        "               I have a third vision. India must stand up to the world. Because I believe that unless India \r\n",
        "               stands up to the world, no one will respect us. Only strength respects strength. We must be \r\n",
        "               strong not only as a military power but also as an economic power. Both must go hand-in-hand. \r\n",
        "               My good fortune was to have worked with three great minds. Dr. Vikram Sarabhai of the Dept. of \r\n",
        "               space, Professor Satish Dhawan, who succeeded him and Dr. Brahm Prakash, father of nuclear material.\r\n",
        "               I was lucky to have worked with all three of them closely and consider this the great opportunity of my life. \r\n",
        "               I see four milestones in my career\"\"\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_TNrnk2hf-ah"
      },
      "source": [
        "sentenn=nltk.sent_tokenize(paragraph)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ch_fjsehgmRR"
      },
      "source": [
        "corpuss=[]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dVwdjtWags1V"
      },
      "source": [
        "for i in range(len(sentenn)):\r\n",
        "  review=re.sub('[^a-zA-Z]',' ',sentenn[i])\r\n",
        "  review=review.lower()\r\n",
        "  review=review.split()\r\n",
        "  review=[lemma.lemmatize(word) for word in review if word not in set(stopwords.words('english'))]\r\n",
        "  review=' '.join(review)\r\n",
        "  corpuss.append(review)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lkd--X9Uhz8x"
      },
      "source": [
        "from sklearn.feature_extraction.text import TfidfVectorizer"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7Hy8NO-qh1_v"
      },
      "source": [
        "vc=TfidfVectorizer()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WEROITAfiJWy"
      },
      "source": [
        "X=vc.fit_transform(corpuss).toarray()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "frQb-KpgiSrm",
        "outputId": "ecbf0983-e4fd-43f2-8de3-bd36c1120a50"
      },
      "source": [
        "X"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.        , 0.        , 0.        , ..., 0.        , 0.        ,\n",
              "        0.        ],\n",
              "       [0.        , 0.        , 0.        , ..., 0.25883507, 0.30512561,\n",
              "        0.        ],\n",
              "       [0.        , 0.28867513, 0.        , ..., 0.        , 0.        ,\n",
              "        0.        ],\n",
              "       ...,\n",
              "       [0.        , 0.        , 0.        , ..., 0.        , 0.        ,\n",
              "        0.        ],\n",
              "       [0.        , 0.        , 0.        , ..., 0.        , 0.        ,\n",
              "        0.        ],\n",
              "       [0.        , 0.        , 0.        , ..., 0.        , 0.        ,\n",
              "        0.        ]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 67
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b65n4mkEFe8J"
      },
      "source": [
        "## Word2vec"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VAQ3V0jjiTfP"
      },
      "source": [
        "# Preprocessing the data\r\n",
        "text = re.sub(r'\\[[0-9]*\\]',' ',paragraph)\r\n",
        "text = re.sub(r'\\s+',' ',text)\r\n",
        "text = text.lower()\r\n",
        "text = re.sub(r'\\d',' ',text)\r\n",
        "text = re.sub(r'\\s+',' ',text)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-r-TXUHUFyCq"
      },
      "source": [
        "# Preparing the dataset\r\n",
        "sentences = nltk.sent_tokenize(text)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-cAPuUwLF1ki"
      },
      "source": [
        "sentences = [nltk.word_tokenize(sentence) for sentence in sentences]\r\n",
        "\r\n",
        "for i in range(len(sentences)):\r\n",
        "    sentences[i] = [word for word in sentences[i] if word not in stopwords.words('english')]\r\n",
        "    \r\n",
        "    \r\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AKI4oEdbF5my"
      },
      "source": [
        "from gensim.models import Word2Vec"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GtzKhOnpGDz4"
      },
      "source": [
        "# Training the Word2Vec model\r\n",
        "model = Word2Vec(sentences, min_count=1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VygGZNB8GHGT"
      },
      "source": [
        "words = model.wv.vocab"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Khus4wMUGMMP"
      },
      "source": [
        "# Finding Word Vectors\r\n",
        "vector = model.wv['war']\r\n",
        "\r\n",
        "# Most similar words\r\n",
        "similar = model.wv.most_similar('vikram')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XqhmcjaYGO-Y",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "dbeec9d0-2e8d-4f68-cc5a-3a39261cb053"
      },
      "source": [
        "vector"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([ 1.2328625e-03, -1.5694118e-03,  3.4692043e-03, -1.0177789e-03,\n",
              "       -3.2810331e-03, -2.0171034e-03, -4.2416500e-03,  2.2702734e-04,\n",
              "        4.4105225e-03, -3.2633482e-03, -4.0658615e-03,  5.0367974e-04,\n",
              "        2.4579882e-03, -1.5390347e-03,  1.1413589e-03,  4.4950354e-04,\n",
              "       -2.1837333e-03,  4.1792155e-03, -9.9586637e-04,  4.4658519e-03,\n",
              "        8.5416378e-04, -3.7313469e-03, -1.5122493e-03,  4.6049468e-03,\n",
              "        2.9618009e-03,  2.3597806e-05, -4.0003592e-03,  1.4211597e-03,\n",
              "       -2.1682032e-03,  3.1778603e-03,  2.9145007e-03,  2.4083620e-03,\n",
              "        4.0524290e-03, -2.5524583e-03,  1.8572254e-03, -3.8750421e-03,\n",
              "       -2.6117375e-03, -2.8877496e-03,  3.8559267e-03, -4.2663468e-03,\n",
              "        6.5602089e-04, -4.5060893e-03,  2.0708577e-03, -6.0694834e-04,\n",
              "        4.1832761e-03, -1.8703985e-03,  2.9064966e-03,  3.6410897e-03,\n",
              "        2.2387182e-04, -7.9084357e-04, -1.6781577e-03,  4.9741701e-03,\n",
              "       -5.9208658e-04,  5.0202414e-04,  2.4707608e-03, -5.1191036e-04,\n",
              "       -2.0426321e-03,  1.4920123e-03,  3.6426117e-03, -1.9151272e-03,\n",
              "        3.8996309e-03,  4.4403938e-03,  2.7454987e-03,  4.1151592e-03,\n",
              "        3.9721644e-03, -2.5669239e-03, -2.5421178e-03, -4.3010249e-04,\n",
              "       -2.3097068e-03, -2.4407546e-03, -4.1141529e-03, -4.2791590e-03,\n",
              "       -2.8412074e-03,  4.8802793e-03, -3.0221429e-03, -9.2223648e-04,\n",
              "       -3.5159881e-03,  1.3421115e-03, -4.5968625e-03, -3.9611445e-03,\n",
              "        4.2287433e-03, -3.4224188e-03,  1.9993535e-03,  4.6724809e-04,\n",
              "        2.9090690e-04,  1.2234231e-03,  2.8629196e-03,  4.5680357e-03,\n",
              "       -3.9389208e-03,  4.2321174e-03, -3.3202914e-03, -1.1940532e-03,\n",
              "       -4.8154681e-03, -3.6963448e-03,  4.3663313e-03,  2.1042007e-03,\n",
              "        2.2366664e-03,  1.1136098e-03,  3.5251987e-03,  2.0250359e-03],\n",
              "      dtype=float32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 78
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZYxP3g3QGQhy",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6b55897e-970a-4a77-d736-ebc373591002"
      },
      "source": [
        "similar"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('land', 0.2741365432739258),\n",
              " ('yet', 0.27055177092552185),\n",
              " ('stands', 0.22241505980491638),\n",
              " ('power', 0.22081242501735687),\n",
              " ('hand-in-hand', 0.2011113166809082),\n",
              " ('top', 0.1960967630147934),\n",
              " ('space', 0.19049623608589172),\n",
              " ('dept', 0.18042442202568054),\n",
              " ('captured', 0.16446073353290558),\n",
              " ('started', 0.1551610827445984)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 79
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2T7TRFahGTak"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}