4_Text_Processing.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "private_outputs": true,
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "1. Listen the ENTIRE Speech (text to audio file) -- using gtts,playsound and pyttsx3\n",
        "\n",
        "2. Text Summarization (shorten the passage) -- from nltk.tokenize import word_tokenize; using spacy\n",
        "\n",
        "3. #Listen the Summary -- - using gtts,playsound and pyttsx3\n",
        "\n",
        "4. FAQ/QnA on the speech (Questions generation) -- using summarizers\n",
        "\n",
        "5. Word Cloud (Word diagram) -- using wordcloud\n",
        "\n",
        "6. Sentiment Analysis -- using textblob, TFAutoModelForSequenceClassification model and  AutoTokenizer from transformers \n",
        "\n",
        "7_1. POS Tagging (Parts of Speech Tagging) --  using pos_tag from nltk\n",
        "\n",
        "7. NER (Named Entity Recognizer) -- importing displacy from spacy, importing ne_chunk from nltk\n",
        "\n",
        "8. Extract Audio file from the youtube Video : importing Model ('deepspeech-0.9.3-models.pbmm', 'deepspeech-0.9.3-models.scorer') from deepspeech; importig YouTubeVideo, Audio, Image from IPython.display\n",
        "\n",
        "8_1. Speech (youtube video) to Text Transcription using AssemblyAI -- \n",
        "\n",
        "9. Dictionary -- from nltk wordnet package synsets function\n",
        "\n",
        "9_1 Correct the spelling --  . from textblob import TextBlob, Word\n",
        "\n",
        "10. Get similar sentences from his speech --  Importing PegasusForConditionalGeneration, PegasusTokenizer from transformers.Used ''tuner007/pegasus_paraphrase'' model. \n",
        "\n",
        "11. ANY language to ENGLISH translation -- from transformers import MBartForConditionalGeneration, MBart50TokenizerFast. Used facebook/mbart-large-50-many-to-many-mmt model. \n",
        "\n",
        "12. Next word /Sentence prediction"
      ],
      "metadata": {
        "id": "3EC0X90aBiKN"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "aZ4jR_KUqS0_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# ngram"
      ],
      "metadata": {
        "id": "Z3AQtu7sY7fN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#An n-gram is a contiguous sequence of n items from a given sequence of text or speech. \n",
        "#The items can be syllables, letters, words or base pairs according to the application. \n",
        "import nltk\n",
        "nltk.download('punkt')\n",
        "from nltk import word_tokenize\n",
        "from nltk.util import ngrams\n",
        "\n",
        "text = \"I need to write a program in NLTK that breaks a corpus (a large collection of txt files) into unigrams, bigrams, trigrams, fourgrams and fivegrams.\"\n",
        "token = nltk.word_tokenize(text)\n",
        "bigrams = ngrams(token,2) #3 for trigrams\n",
        "for grams in bigrams:print(grams)\n"
      ],
      "metadata": {
        "id": "9um33kK1Cu2t"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "GMO9oO-cFJik"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')\n",
        "%cd \"/content/drive/My Drive/Colab Notebooks/POC_Projects/\""
      ],
      "metadata": {
        "id": "TJ0zFeNUep50"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "with open('/content/drive/My Drive/Colab Notebooks/POC_Projects/PM’s Speech on 75th Independence Day.txt', 'r') as file:#\n",
        "    data = file.read().replace('\\n', '')\n",
        "\n",
        "fhand = data.replace(\"\\ufeff\", \"\")\n",
        "print(fhand)"
      ],
      "metadata": {
        "id": "uIFGarcPetZT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Tokenizing sentences"
      ],
      "metadata": {
        "id": "vVmKUrK3g9Xf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Tokenizing sentences\n",
        "import nltk\n",
        "nltk.download('punkt')\n",
        "sentences = nltk.sent_tokenize(fhand)\n",
        "print(sentences)"
      ],
      "metadata": {
        "id": "f0pabK1Cetch"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Tokenizing words\n",
        "words = nltk.word_tokenize(fhand)\n",
        "print(words)"
      ],
      "metadata": {
        "id": "gi9Z5GPdetgB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#stemming"
      ],
      "metadata": {
        "id": "wD7DoclhhBIY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#stemming\n",
        "from nltk.stem import PorterStemmer\n",
        "from nltk.corpus import stopwords\n",
        "nltk.download('stopwords')\n",
        "stemmer = PorterStemmer()\n",
        "for i in range(len(sentences)):\n",
        "  words =  nltk.word_tokenize(sentences[i])\n",
        "  words = [stemmer.stem(word) for word in words if word not in  set(stopwords.words('english'))]\n",
        "  sentences[i] = ' '.join(words)\n",
        "print(sentences)"
      ],
      "metadata": {
        "id": "xWrSxI4getog"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#lemmitization\n",
        "from nltk.stem import WordNetLemmatizer\n",
        "from nltk.corpus import stopwords\n",
        "nltk.download('wordnet')\n",
        "nltk.download('omw-1.4')\n",
        "\n",
        "\n",
        "sentences = nltk.sent_tokenize(fhand)\n",
        "lemmatizer = WordNetLemmatizer()\n",
        "\n",
        "for i in range(len(sentences)):\n",
        "  words =  nltk.word_tokenize(sentences[i])\n",
        "  words = [lemmatizer.lemmatize(word) for word in words if word not in  set(stopwords.words('english'))]\n",
        "  sentences[i] = ' '.join(words)\n",
        "print(sentences)"
      ],
      "metadata": {
        "id": "_t0k6qRc8eU4"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#bag of words - after lemmitization\n",
        "import re\n",
        "\n",
        "\n",
        "for i in range(len(sentences)):\n",
        "  words = re.sub('[^a-zA-Z]',' ',sentences[i])\n",
        "  words = words.lower()\n",
        "  words = words.split()\n",
        "\n",
        "  words_lemmitization = [lemmatizer.lemmatize(word) for word in words if word not in  set(stopwords.words('english'))]\n",
        "  sentences[i] = ' '.join(words_lemmitization)\n",
        "\n",
        "print(sentences)"
      ],
      "metadata": {
        "id": "r8mk4Gwx8ea5"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#bag of words - after stemming\n",
        "import re\n",
        "\n",
        "\n",
        "for i in range(len(sentences)):\n",
        "  words = re.sub('[^a-zA-Z]',' ',sentences[i])\n",
        "  words = words.lower()\n",
        "  words = words.split()\n",
        "\n",
        "  words_stemming = [stemmer.stem(word) for word in words if word not in  set(stopwords.words('english'))]\n",
        "  sentences[i] = ' '.join(words_stemming)\n",
        "\n",
        "print(sentences)"
      ],
      "metadata": {
        "id": "0hRpdftM8eX9"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Count Vectorizer"
      ],
      "metadata": {
        "id": "rOLuOr0UhHnl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Count Vectorizer_1\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer\n",
        "import numpy as np\n",
        "\n",
        "vectorizer=CountVectorizer()\n",
        "countvec=vectorizer.fit_transform(sentences)\n",
        "print(countvec.A)"
      ],
      "metadata": {
        "id": "tHSApA1H8edj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Count Vectorizer_2\n",
        "vectorizer=TfidfVectorizer(use_idf=False, norm=None)\n",
        "tfvec=vectorizer.fit_transform(sentences)\n",
        "print(tfvec.A)\n",
        "print(vectorizer.get_feature_names_out())"
      ],
      "metadata": {
        "id": "JOr80qCib6GR"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Term Frequency (TF) \n",
        "vectorizer=TfidfVectorizer(use_idf=False, norm='l1')\n",
        "tfvec=vectorizer.fit_transform(sentences)\n",
        "print(tfvec.A)\n",
        "print(vectorizer.get_feature_names_out())"
      ],
      "metadata": {
        "id": "6AoLVMO9b6M7"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#IDF\n",
        "vectorizer_idf=TfidfVectorizer(smooth_idf=False)\n",
        "tfidfvec=vectorizer_idf.fit_transform(sentences)\n",
        "print(vectorizer_idf.idf_)\n",
        "print(vectorizer_idf.get_feature_names_out())"
      ],
      "metadata": {
        "id": "124SJIMj8egM"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "dxnqagKio_hu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Naive Bayes classifier "
      ],
      "metadata": {
        "id": "hru1tRHIsezj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Naive Bayes classifier \n",
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "vectorizer = CountVectorizer(max_features=2500)\n",
        "X = vectorizer.fit_transform(sentences).toarray()"
      ],
      "metadata": {
        "id": "JSFuhqbjgfab"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "y=pd.get_dummies(sentences)\n",
        "y=y.iloc[:,1].values"
      ],
      "metadata": {
        "id": "jck5sNVJgfdi"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Train Test Split\n",
        "from sklearn.model_selection import train_test_split\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.20, random_state = 0)"
      ],
      "metadata": {
        "id": "G6Yp-yfTgfgf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Training model using Naive bayes classifier\n",
        "from sklearn.naive_bayes import MultinomialNB\n",
        "spam_detect_model = MultinomialNB().fit(X_train, y_train)\n",
        "\n",
        "y_pred=spam_detect_model.predict(X_test)"
      ],
      "metadata": {
        "id": "NbEGRZbmgfj-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.metrics import confusion_matrix\n",
        "confusion_m = confusion_matrix(y_test,y_pred )\n",
        "print(confusion_m)"
      ],
      "metadata": {
        "id": "6D7LKJpop5nc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.metrics import accuracy_score\n",
        "accuracy_s = accuracy_score(y_test,y_pred )\n",
        "print(accuracy_s)"
      ],
      "metadata": {
        "id": "_KHh97lup5qw"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Word2vec"
      ],
      "metadata": {
        "id": "aVBCPz8o3GLI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import nltk\n",
        "nltk.download('stopwords')"
      ],
      "metadata": {
        "id": "RjR0M4FXtcVL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import nltk\n",
        "from gensim.models import Word2Vec\n",
        "from nltk.corpus import stopwords\n",
        "import re\n",
        "\n",
        "sentences = nltk.sent_tokenize(fhand)\n",
        "sentences = [nltk.word_tokenize(sentence) for sentence in sentences]\n",
        "for i in range(len(sentences)):\n",
        "  sentences[i]=[word for word in sentences[i] if word not in stopwords.words('english')]\n",
        "\n",
        "print(sentences)"
      ],
      "metadata": {
        "id": "CKBo48S6tcYT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Training the Word2Vec model\n",
        "model = Word2Vec(sentences, min_count=1)\n",
        "word_stemming = model.wv.vocab\n",
        "print(word_stemming)"
      ],
      "metadata": {
        "id": "vmhIDlFvtcbL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Finding Word Vectors\n",
        "vector = model.wv['Best']\n",
        "print(vector)"
      ],
      "metadata": {
        "id": "6Amr89X7y9HI"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Most similar words\n",
        "similar = model.wv.most_similar('Best')\n",
        "print(similar)"
      ],
      "metadata": {
        "id": "B6udeA6Ky9Jy"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "Y-A8l97Ky9Ul"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# RNN"
      ],
      "metadata": {
        "id": "8BOQFeAeZHXV"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#RNN\n",
        "X_train, X_test, y_train, y_test = train_test_split(X,y, test_size=0.2)"
      ],
      "metadata": {
        "id": "MZONggZpZCI0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Prepare Data for Modelling\n",
        "# Install keras\n",
        "!pip install -U keras"
      ],
      "metadata": {
        "id": "AKI-R3VXZCL-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install tensorflow"
      ],
      "metadata": {
        "id": "n6EyenypZCPL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Import the tools we will need from keras\n",
        "from keras.preprocessing.text import Tokenizer\n",
        "from keras_preprocessing.sequence import pad_sequences\n"
      ],
      "metadata": {
        "id": "YtPEcm9gCu5r"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Initialize and fit the tokenizer\n",
        "tokenizer = Tokenizer()\n",
        "tokenizer.fit_on_texts(X_train)"
      ],
      "metadata": {
        "id": "S-LoAgA4Cu8i"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Use that tokenizer to transform the text messages in the training and test sets\n",
        "X_train_seq = tokenizer.texts_to_sequences(X_train)\n",
        "X_test_seq = tokenizer.texts_to_sequences(X_test)"
      ],
      "metadata": {
        "id": "qLun89Qyy9Xc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# What do these sequences look like?\n",
        "X_train_seq[0]"
      ],
      "metadata": {
        "id": "ViDzFSK2y9aJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Pad the sequences so each sequence is the same length\n",
        "X_train_seq_padded = pad_sequences(X_train_seq, 50)\n",
        "X_test_seq_padded = pad_sequences(X_test_seq, 50)"
      ],
      "metadata": {
        "id": "BI5c96uBbdTA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# What do these padded sequences look like?\n",
        "X_train_seq_padded[0]"
      ],
      "metadata": {
        "id": "viqYp-FXbdWL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Build Model\n",
        "# Import the tools needed from keras and define functions to calculate recall and precision\n",
        "import keras.backend as K\n",
        "from keras.layers import Dense, Embedding, LSTM\n",
        "from keras.models import Sequential\n",
        "\n",
        "def recall_m(y_true, y_pred):\n",
        "        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n",
        "        possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))\n",
        "        recall = true_positives / (possible_positives + K.epsilon())\n",
        "        return recall\n",
        "\n",
        "def precision_m(y_true, y_pred):\n",
        "        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n",
        "        predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))\n",
        "        precision = true_positives / (predicted_positives + K.epsilon())\n",
        "        return precision"
      ],
      "metadata": {
        "id": "BgTxhGFAbdZR"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Construct a simple RNN model\n",
        "model = Sequential()\n",
        "\n",
        "model.add(Embedding(len(tokenizer.index_word)+1, 32))\n",
        "model.add(LSTM(32, dropout=0, recurrent_dropout=0))\n",
        "model.add(Dense(32, activation='relu'))\n",
        "model.add(Dense(1, activation='sigmoid'))\n",
        "model.summary()"
      ],
      "metadata": {
        "id": "2DP0EvTKbdcb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Compile the model\n",
        "model.compile(optimizer='adam',\n",
        "              loss='binary_crossentropy',\n",
        "              metrics=['accuracy', precision_m, recall_m])"
      ],
      "metadata": {
        "id": "d2odqSB_bdf_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Fit the RNN model\n",
        "history = model.fit(X_train_seq_padded, y_train, \n",
        "                    batch_size=32, epochs=10,\n",
        "                    validation_data=(X_test_seq_padded, y_test))"
      ],
      "metadata": {
        "id": "7IGTiCd7tcg7"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "ecoyIZADb0g2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Plot the evaluation metrics by each epoch for the model to see if we are over or underfitting\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "for i in ['accuracy', 'precision_m', 'recall_m']:\n",
        "    acc = history.history[i]\n",
        "    val_acc = history.history['val_{}'.format(i)]\n",
        "    epochs = range(1, len(acc) + 1)\n",
        "\n",
        "    plt.figure()\n",
        "    plt.plot(epochs, acc, label='Training Accuracy')\n",
        "    plt.plot(epochs, val_acc, label='Validation Accuracy')\n",
        "    plt.title('Results for {}'.format(i))\n",
        "    plt.legend()\n",
        "    plt.show()"
      ],
      "metadata": {
        "id": "fLo3JO_Gb0kT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "FBhQRtqJfCvs"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Word Embedding Represntation"
      ],
      "metadata": {
        "id": "zsYh8ryVc0OF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#https://www.youtube.com/watch?v=TsXR7_vtusQ&list=PLZoTAELRMXVMdJ5sqbCK2LiM0HhQVWNzm&index=21\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive')\n",
        "%cd \"/content/drive/My Drive/Colab Notebooks/POC_Projects/\"\n",
        "\n",
        "with open('/content/drive/My Drive/Colab Notebooks/POC_Projects/PM’s Speech on 75th Independence Day.txt', 'r') as file:#\n",
        "    data = file.read().replace('\\n', '')\n",
        "\n",
        "fhand = data.replace(\"\\ufeff\", \"\")\n",
        "print(fhand)\n",
        "\n",
        "#Tokenizing sentences\n",
        "import nltk\n",
        "nltk.download('punkt')\n",
        "sentences = nltk.sent_tokenize(fhand)\n",
        "print(sentences)\n",
        "\n",
        "#Tokenizing words\n",
        "words = nltk.word_tokenize(fhand)\n",
        "print(words)"
      ],
      "metadata": {
        "id": "t4tCUdBWb0qr"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "##tensorflow >2.0\n",
        "from tensorflow.keras.preprocessing.text import one_hot #imp\n",
        "\n",
        "### Vocabulary size\n",
        "voc_size=10000\n",
        "onehot_repr=[one_hot(words,voc_size)for words in sentences] \n",
        "print(onehot_repr)"
      ],
      "metadata": {
        "id": "RdNAneZOdFHB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Word Embedding Represntation\n",
        "from tensorflow.keras.layers import Embedding\n",
        "from tensorflow.keras.preprocessing.sequence import pad_sequences #IMP\n",
        "from tensorflow.keras.models import Sequential"
      ],
      "metadata": {
        "id": "1a3SDgp9b0tc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np"
      ],
      "metadata": {
        "id": "I1niaO-Jb0wb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "sent_length=8\n",
        "embedded_docs=pad_sequences(onehot_repr,padding='pre',maxlen=sent_length)\n",
        "print(embedded_docs)"
      ],
      "metadata": {
        "id": "XyrYac92b0zX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "dim=10 #number of features"
      ],
      "metadata": {
        "id": "aLXfoj2ab02T"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model=Sequential()\n",
        "model.add(Embedding(voc_size,10,input_length=sent_length))\n",
        "model.compile('adam','mse')"
      ],
      "metadata": {
        "id": "faU6P3FVb05o"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model.summary()"
      ],
      "metadata": {
        "id": "8bawlL3Bb080"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(model.predict(embedded_docs))"
      ],
      "metadata": {
        "id": "Iax3cbmUb0_7"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "embedded_docs[0]"
      ],
      "metadata": {
        "id": "o0fP0C7xb1C_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(model.predict(embedded_docs)[0])"
      ],
      "metadata": {
        "id": "UNHtnFBKeR_n"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "0xVX5oHAeSZt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "0D3bQ2w0qi_x"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "bIfFmMsBtlWn"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "C3pl394StlZg"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "pG6EdvqWetxC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "NLP Tasks (Stnd. areas)\n",
        "1. Fill-Mask: 2. Question Answering 3. Sentence Similarity 4. Summarization:\n",
        "\n",
        "5. Text Classification 6. Text Generation: 7. Token Classification \n",
        "8. Translation"
      ],
      "metadata": {
        "id": "dGeAJccKoCOw"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QxzIUqXZlc_J"
      },
      "source": [
        "1. Listen the ENTIRE speech"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wiXqy1HclJT9"
      },
      "source": [
        "!pip install playsound  \n",
        "!pip install pyttsx3  \n",
        "!pip install gTTS\n",
        "import gtts  \n",
        "from playsound import playsound \n",
        "\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive')\n",
        "%cd \"/content/drive/My Drive/Colab Notebooks/POC_Projects/\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WjntvZPYlJaU"
      },
      "source": [
        "with open('/content/drive/My Drive/Colab Notebooks/POC_Projects/PM’s Speech on 75th Independence Day.txt', 'r') as file:#\n",
        "    data = file.read().replace('\\n', '')\n",
        "\n",
        "fhand = data.replace(\"\\ufeff\", \"\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6aL9W1z77vYI"
      },
      "source": [
        "fhand[0:100]\n",
        "t1 = gtts.gTTS(fhand,lang = 'hi')\n",
        "\n",
        "# save the audio file  \n",
        "t1.save(\"welcome.mp3\")#takes 4 to 5 mins\n",
        "\n",
        "from IPython.display import Audio\n",
        "Audio('welcome.mp3')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8YAn2LNslSkP"
      },
      "source": [
        "\n",
        "2. Text Summarization"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Summarization is the task of producing a shorter version of a document while preserving its important information. Some models can extract text from the original input, while other models can generate entirely new text."
      ],
      "metadata": {
        "id": "qprVU4vQ5rR7"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1kWWk4lrlJw1"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')\n",
        "\n",
        "with open('/content/drive/My Drive/Colab Notebooks/POC_Projects/PM’s Speech on 75th Independence Day.txt', 'r') as file:\n",
        "    data = file.read().replace('\\n', '')\n",
        "\n",
        "fhand = data.replace(\"\\ufeff\", \"\")\n",
        "from nltk.tokenize import word_tokenize\n",
        "import nltk\n",
        "nltk.download('punkt')\n",
        "words = word_tokenize(fhand)\n",
        "nltk.download('stopwords')\n",
        "from nltk.corpus import stopwords\n",
        "stopwordlist = list(stopwords.words('english'))\n",
        "\n",
        "import spacy\n",
        "from spacy.lang.en.stop_words import STOP_WORDS\n",
        "from string import punctuation\n",
        "\n",
        "stopwords = list(STOP_WORDS)\n",
        "stopwords = stopwordlist.extend (['(',')','-',':',',',\"'s\",'!',':',\"'\",\"''\",'--','.',':','?',';''[',']','``','o','’','“','”','”','[',';'])\n",
        "\n",
        "nlp = spacy.load('en_core_web_sm')\n",
        "doc = nlp(fhand)\n",
        "\n",
        "tokens = [token.text for token in doc]\n",
        "\n",
        "punctuation = punctuation + '\\n'\n",
        "word_frequencies = {} #making a dictionary\n",
        "for word in doc:\n",
        "  if word.text.lower() not in punctuation:\n",
        "    if word.text not in word_frequencies.keys():\n",
        "      word_frequencies[word.text] =1\n",
        "    else:\n",
        "      word_frequencies[word.text] +=1 #if any word is present more than 1 time\n",
        "\n",
        "max_frequency = max(word_frequencies.values())\n",
        "\n",
        "for word in word_frequencies.keys():\n",
        "  word_frequencies[word] = word_frequencies[word]/max_frequency\n",
        "\n",
        "  sentence_tokens = [sent for sent in doc.sents]\n",
        "\n",
        "sentence_scores = {} #create dictionary\n",
        "for sent in sentence_tokens:\n",
        "  for word in sent:\n",
        "    if word.text.lower() in word_frequencies.keys():\n",
        "      if sent not in sentence_scores.keys():\n",
        "        sentence_scores[sent] = word_frequencies[word.text.lower()]\n",
        "      else:\n",
        "\n",
        "        sentence_scores[sent] += word_frequencies[word.text.lower()]\n",
        "\n",
        "from heapq import nlargest\n",
        "select_length = int(len(sentence_tokens)*.3) #selecting only 10% of the sentences\n",
        "summary = nlargest(select_length,sentence_scores,key = sentence_scores.get)\n",
        "final_summary = [word.text for word in summary]\n",
        "summary = ' '.join(final_summary)\n",
        "\n",
        "# summary\n",
        "\n",
        "fhand = open('summary.txt','w')\n",
        "fhand.write(summary) # Erass all, already writtena and write what has been passed\n",
        "fhand.close()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "UxynPGPCdaDA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#text summary: 2. Controllable Text Summarization CtrlSum Huggingface\n",
        "#https://www.youtube.com/watch?v=P8CqGqR1Zr4\n",
        "#https://colab.research.google.com/drive/17zmc7jXFkc4o4bXXQ65DdC0ZVWmhqQpy?usp=sharing#scrollTo=6dENbUQK5ouv\n",
        "#IMP: https://github.com/hyunwoongko/summarizers\n",
        "#https://github.com/salesforce/ctrl-sum/tree/master/ctrlsum/token-classification\n",
        "\n",
        "!pip install summarizers -q\n",
        "\n",
        "from summarizers import Summarizers\n",
        "summ = Summarizers(type = 'normal', device='cuda') #If you want GPU acceleration, set param device='cuda'. #type = 'paper', type = 'patent'\n"
      ],
      "metadata": {
        "id": "mkTmXSVUdaNH"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "text = \"Recent state-of-the-art approaches to summarization utilize large pre-trained Transformer models. Distilling these models to smaller student models has become critically important for practical use; however there are many different distillation methods proposed by the NLP literature. Recent work on distilling BERT for classification and regression tasks shows strong performance using direct knowledge distillation. Alternatively, machine translation practitioners distill using pseudo-labeling, where a small model is trained on the translations of a larger model. A third, simpler approach is to 'shrink and fine-tune' (SFT), which avoids any explicit distillation by copying parameters to a smaller student model and then fine-tuning. We compare these three approaches for distillation of Pegasus and BART, the current and former state of the art, pre-trained summarization models, and find that SFT outperforms knowledge distillation and pseudo-labeling on the CNN/DailyMail dataset, but under-performs pseudo-labeling on the more abstractive XSUM dataset. PyTorch Code and checkpoints of different sizes are available through Hugging Face transformers here this http URL.\"\n",
        "# print(text)\n",
        "print(summ(text, query = 'approaches of distilling'))"
      ],
      "metadata": {
        "id": "ONKEnf8i8eAz"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "text2 = '''\n",
        "\tWhen I was young, the kitchen had an invisible 'stop' sign put there by my mother. “No you cannot enter here!” she always said to me. I don't blame her as it is no place for kids. Fire and sharp stuff are present there.\n",
        "\t\n",
        "\tI alwayswonder the beautiful experience of cooking. This is an art and our mothers are artist, very perfect in making the food delicious. I wanted to cook and my school gave me the project to make a dish of potato and lady finger. This was a golden chance to get in the kitchen and get my apron messy. I called my mother to help as this was my first time cooking and I am not that old to cook by my self. I told my mother about the task and we began.\n",
        "\n",
        "\tShe thought for a while and suggested some dishes. I picked my favourite one. She instructed me and I followed. I was wearing my apron and so was mom. She explained me so sweetly and nicely that I know the recipe by heart. The using of stove and knives were done by her. We were cooking it in the evening for the dinner of the family. \n",
        "\n",
        "\tMy father and my brother also came to help us . We prepared the dish by 8 o'clock and mom gave me to taste some of it. It was delicious and so finely prepared. That day I got to know the work required for making the food that we simply eat it watching television. I ate the food and felt all the spices blend in my mouth. It was a deling for me and my tongue. It was a wonderful experience.\n",
        "\t\t\t\t         \n",
        "                                                                     \" A recipe has no soul. You, as a cook, must bring the soul to the recipe.\"   '''\n",
        "print(summ(text2))\n",
        "\n",
        "print(summ(text2, query = \"What was I cooking ?\" ))\n",
        "#print(summ(text2, query = \"What was I cooking ?\", question_detection=False ))\n",
        "#print(summ(text2, prompt=\"Q:Where is India? A:\"))"
      ],
      "metadata": {
        "id": "TrQWm6Gf8eD-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# text2 = \"India (Hindi: Bhārat), officially the Republic of India (Hindi: Bhārat Gaṇarājya),[23] is a country in South Asia. It is the second-most populous country, the seventh-largest country by land area, and the most populous democracy in the world. Bounded by the Indian Ocean on the south, the Arabian Sea on the southwest, and the Bay of Bengal on the southeast, it shares land borders with Pakistan to the west;[f] China, Nepal, and Bhutan to the north; and Bangladesh and Myanmar to the east. In the Indian Ocean, India is in the vicinity of Sri Lanka and the Maldives; its Andaman and Nicobar Islands share a maritime border with Thailand, Myanmar and Indonesia.Modern humans arrived on the Indian subcontinent from Africa no later than 55,000 years ago.[24] Their long occupation, initially in varying forms of isolation as hunter-gatherers, has made the region highly diverse, second only to Africa in human genetic diversity.[25] Settled life emerged on the subcontinent in the western margins of the Indus river basin 9,000 years ago, evolving gradually into the Indus Valley Civilisation of the third millennium BCE.[26] By 1200 BCE, an archaic form of Sanskrit, an Indo-European language, had diffused into India from the northwest,[27] unfolding as the language of the Rigveda, and recording the dawning of Hinduism in India.[28][disputed – discuss] The Dravidian languages of India were supplanted in the northern and western regions.[29] By 400 BCE, stratification and exclusion by caste had emerged within Hinduism,[30] and Buddhism and Jainism had arisen, proclaiming social orders unlinked to heredity.[31] Early political consolidations gave rise to the loose-knit Maurya and Gupta Empires based in the Ganges Basin.[32] Their collective era was suffused with wide-ranging creativity,[33] but also marked by the declining status of women,[34] and the incorporation of untouchability into an organised system of belief.[g][35] In South India, the Middle kingdoms exported Dravidian-languages scripts and religious cultures to the kingdoms of Southeast Asia.[36]\"\n",
        "\n",
        "print(summ(text2, query = \"What was I cooking ?\", question_detection=False ))"
      ],
      "metadata": {
        "id": "CemsRsAp8eGt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(summ(text2, prompt=\"Q:Where is India? A:\"))"
      ],
      "metadata": {
        "id": "weuguBkv8eJT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(summ(text2, query=\"Where is India?\", prompt=\"India is\")) \n",
        "#Generating text is the task of producing new text. These models can, for example, fill in incomplete text or paraphrase (Casual Language Modelling)"
      ],
      "metadata": {
        "id": "an3uvV2U8wFj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "summ(\n",
        "...     contents=text2,\n",
        "...     num_beams=10,\n",
        "...     top_k=30,\n",
        "...     top_p=0.85,\n",
        "...     no_repeat_ngram_size=3,      \n",
        "...     length_penalty=1.2,\n",
        "... )"
      ],
      "metadata": {
        "id": "k7NagDHD8wIl"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "3nO4bUmj8wLP"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "_b6mJRN98eMY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ujIRli9UsjmY"
      },
      "source": [
        "3. Listen the Summary"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8l1NHcZMlKAe"
      },
      "source": [
        "with open('/content/drive/My Drive/Colab Notebooks/POC_Projects/summary.txt', 'r') as file:\n",
        "    data = file.read().replace('\\n', '')\n",
        "\n",
        "fhand = data.replace(\"\\ufeff\", \"\")\n",
        "fhand[0:100]\n",
        "t1 = gtts.gTTS(fhand,lang = 'hi')\n",
        "\n",
        "# save the audio file  \n",
        "t1.save(\"summary.mp3\") #will take 4 to 5 mins\n",
        "\n",
        "from IPython.display import Audio\n",
        "Audio('summary.mp3')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qwWzihbVuoAa"
      },
      "source": [
        "4. FAQ/QnA on the speech"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Question Answering models can retrieve the answer to a question from a given text, which is useful for searching for an answer in a document. Some question answering models can generate answers without context!"
      ],
      "metadata": {
        "id": "g8hDxw5UN2uk"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JMQU1r5SnQJ5"
      },
      "source": [
        "# !pip install summarizers -q\n",
        "\n",
        "# from summarizers import Summarizers\n",
        "# summ = Summarizers(type = 'normal', device='cuda') #If you want GPU acceleration, set param device='cuda'.\n",
        "# # summ = Summarizers(type = 'paper', device='cuda') #If you want GPU acceleration, set param device='cuda'.\n",
        "# # summ = Summarizers(type = 'patent', device='cuda') #If you want GPU acceleration, set param device='cuda'.\n",
        "\n",
        "# text = fhand\n",
        "# text"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7ZeVOc4pnQPy"
      },
      "source": [
        "# print(summ(text, query = \"whom I salute?\", question_detection=False  ))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bMXDPejF9TIB"
      },
      "source": [
        "# print(summ(text, prompt=\"Q:whom I salute? A:\"))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MXdat1Xb9TLX"
      },
      "source": [
        "# print(summ(text, query=\"Whom I salute?\", prompt=\"I salute countless\"))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_4RmrX2ZnQST"
      },
      "source": [
        "# summ(\n",
        "# ...     contents=text,\n",
        "# ...     num_beams=10,\n",
        "# ...     top_k=30,\n",
        "# ...     top_p=0.85,\n",
        "# ...     no_repeat_ngram_size=3,      \n",
        "# ...     length_penalty=1.2,\n",
        "# ... )"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "G7B4l8YqjQ92"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "kT-wvpeRjRAE"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "jmdKkFA4irbd"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dpfLeX_ZDf1d"
      },
      "source": [
        "5. Word Cloud"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "347LsquGnQbT"
      },
      "source": [
        "import pathlib \n",
        "from wordcloud import WordCloud, STOPWORDS, ImageColorGenerator \n",
        "import matplotlib.pyplot as plt \n",
        "import numpy as np \n",
        "from PIL import Image \n",
        "%matplotlib inline"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IIgNcmmGDjUB"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')\n",
        "path = pathlib.Path.cwd() /'files'/ '/content/drive/My Drive/Colab Notebooks/POC_Projects/PM’s Speech on 75th Independence Day.txt'\n",
        "print(path)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pPQVTTMPDjaU"
      },
      "source": [
        "text2 = path.read_text()\n",
        "wordcloud = WordCloud(max_font_size=50, max_words=1000).generate(text2) #,background_color=\"white\"\n",
        "plt.imshow(wordcloud,interpolation='bilinear')\n",
        "plt.axis('off')\n",
        "plt.savefig('/content/drive/My Drive/Colab Notebooks/POC_Projects/ML1.png',dpi = 600, facecolor = 'w',format = 'png')\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "uYLh-A6ERGFA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oVw53rPRlRcm"
      },
      "source": [
        "6. Sentiment Analysis"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Text classification: Text Classification is the task of assigning a label or class to a given text. Some use cases are sentiment analysis, natural language inference, and assessing grammatical correctness."
      ],
      "metadata": {
        "id": "m95LcHbe6O0P"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "47aPjPfdLfHt"
      },
      "source": [
        "#Way 1\n",
        "#Sentiment Analysis: TextBlob returns polarity and subjectivity of a sentence. \n",
        "#Polarity lies between [-1,1], -1 defines a negative sentiment and 1 defines a positive sentiment. Negation words reverse the polarity.\n",
        "#Subjectivity lies between [0,1]. Subjectivity quantifies the amount of personal opinion and factual information contained in the text. \n",
        "#The higher subjectivity means that the text contains personal opinion rather than factual information.\n",
        "\n",
        "\n",
        "\n",
        "# !pip install textblob\n",
        "from textblob import TextBlob\n",
        "\n",
        "feedback1 = fhand\n",
        "\n",
        "blob1=TextBlob(feedback1)\n",
        "\n",
        "print(blob1.sentiment)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MtBtXhJ0l4A_"
      },
      "source": [
        "#Way 2\n",
        "from transformers import AutoTokenizer, TFAutoModelForSequenceClassification\n",
        "model_name = \"nlptown/bert-base-multilingual-uncased-sentiment\"\n",
        "# This model only exists in PyTorch, so we use the `from_pt` flag to import that model in TensorFlow.\n",
        "model = TFAutoModelForSequenceClassification.from_pretrained(model_name, from_pt=True)\n",
        "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
        "classifier = pipeline('sentiment-analysis', model=model, tokenizer=tokenizer)\n",
        "\n",
        "results = classifier('/content/drive/My Drive/Colab Notebooks/POC_Projects/PM’s Speech on 75th Independence Day.txt')\n",
        "results"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a-SdjNxXl4Dz"
      },
      "source": [
        "#Way 3\n",
        "#! pip install transformers\n",
        "from transformers import pipeline\n",
        "classifier = pipeline('sentiment-analysis')\n",
        "print(classifier('We are very happy to show you the 🤗 Transformers library.'))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zV1rSw6LS2dd"
      },
      "source": [
        "7_1 POS Tagging"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L0XHKCavS1kO"
      },
      "source": [
        "#6 POS Tagging\n",
        "nltk.download('averaged_perceptron_tagger')\n",
        "sent= open('/content/drive/My Drive/Colab Notebooks/POC_Projects/PM’s Speech on 75th Independence Day.txt','r')\n",
        "sent2 = sent.read()\n",
        "sent_tag = nltk.word_tokenize(sent2)\n",
        "\n",
        "for token in sent_tag:\n",
        "  print(nltk.pos_tag([token]))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EZ8O7nbaqQ8a"
      },
      "source": [
        "7. NER (Named Entity Recognizer)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Token Classification: Token classification is a natural language understanding task in which a label is assigned to some tokens in a text. Some popular token classification subtasks are Named Entity Recognition (NER) and Part-of-Speech (PoS) tagging. NER models could be trained to identify specific entities in a text, such as dates, individuals and places; and PoS tagging would identify, for example, which words in a text are verbs, nouns, and punctuation marks."
      ],
      "metadata": {
        "id": "_eGhRvLp6d0l"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4Or0tNbxl4R7"
      },
      "source": [
        "import spacy\n",
        "from spacy import displacy\n",
        "nlp=spacy.load('en_core_web_sm')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "meBva2KqqYJR"
      },
      "source": [
        "doc = nlp(fhand)\n",
        "for ent in doc.ents:\n",
        "  print(ent.text, ent.start_char, ent.end_char, ent.label_)\n",
        "displacy.render(nlp(doc.text),style='ent', jupyter=True) "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "un4SryS4qYS_"
      },
      "source": [
        "#7: NER\n",
        "from nltk import ne_chunk\n",
        "nltk.download('maxent_ne_chunker')\n",
        "nltk.download('words')\n",
        "ne_sent= open('/content/drive/My Drive/Colab Notebooks/POC_Projects/PM’s Speech on 75th Independence Day.txt','r')\n",
        "ne_sent2 = ne_sent.read()\n",
        "\n",
        "ne_token =word_tokenize(ne_sent2)\n",
        "ne_tags = nltk.pos_tag(ne_token)\n",
        "\n",
        "ne_ner = ne_chunk(ne_tags)\n",
        "print(ne_ner)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L3-QY-tux6V6"
      },
      "source": [
        "8. Extract Audio file from the youtube Video"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RCL9LNHByL1n"
      },
      "source": [
        "!pip install wget \n",
        "!pip install easyocr \n",
        "!pip install deepspeech-gpu==0.9.3 \n",
        "!pip install pafy \n",
        "!pip install youtube-dl "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6_t6ynbPymIj"
      },
      "source": [
        "!wget https://github.com/mozilla/DeepSpeech/releases/download/v0.9.3/deepspeech-0.9.3-models.pbmm \n",
        "!wget https://github.com/mozilla/DeepSpeech/releases/download/v0.9.3/deepspeech-0.9.3-models.scorer"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UKjkbBBfqYWJ"
      },
      "source": [
        "from deepspeech import Model \n",
        "import numpy as np \n",
        "import os \n",
        "import wave \n",
        "# import easyocr \n",
        "import pafy \n",
        "from IPython.display import Audio, Image \n",
        "from IPython.display import YouTubeVideo \n",
        "\n",
        "model_file_path = 'deepspeech-0.9.3-models.pbmm' \n",
        "lm_file_path = 'deepspeech-0.9.3-models.scorer' \n",
        "beam_width = 500  #more beam width, more accuracy but it will take more time also\n",
        "lm_alpha = 0.93 \n",
        "lm_beta = 1.18 \n",
        "model = Model(model_file_path) #will take 4 to 5 mins\n",
        "model.enableExternalScorer(lm_file_path) \n",
        "model.setScorerAlphaBeta(lm_alpha, lm_beta) \n",
        "model.setBeamWidth(beam_width) \n",
        "YOUTUBE_ID = 'mkVjrB8g6mM' #https://www.youtube.com/watch?v=mkVjrB8g6mM #t23x9gcbamA\n",
        "YouTubeVideo(YOUTUBE_ID)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Zvc2iePiqYZR"
      },
      "source": [
        "URL='https://www.youtube.com/watch\\?v\\='+ YOUTUBE_ID \n",
        "!youtube-dl --extract-audio --audio-format wav --output \"YahiSamayhaiSahiSamayHai.%(ext)s\" $URL "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "54eWog41qYcj"
      },
      "source": [
        "!ffmpeg -i YahiSamayhaiSahiSamayHai.wav -vn -ar 16000 -ac 1 YahiSamayhaiSahiSamayHai.wav #16000 is the framerate\n",
        "Audio('YahiSamayhaiSahiSamayHai.wav')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "K0m5yc45qYfr"
      },
      "source": [
        "stream = model.createStream()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kZg16Ks8qYil"
      },
      "source": [
        "def read_wav_file(filename):\n",
        "  with wave.open(filename,'rb') as w:\n",
        "    rate = w.getframerate()\n",
        "    frames = w.getnframes()\n",
        "    buffer = w.readframes(frames)\n",
        "  return buffer, rate  "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TSC7fjc5qYlx"
      },
      "source": [
        "from IPython.display import clear_output \n",
        "def transcribe_streaming(audio_file): \n",
        "  buffer, rate = read_wav_file(audio_file) \n",
        "  offset=0 \n",
        "  batch_size=65536 \n",
        "  text=''\n",
        "\n",
        "  while offset < len(buffer): \n",
        "    end_offset=offset+batch_size \n",
        "    chunk=buffer[offset:end_offset]\n",
        "    datal6 = np.frombuffer(chunk, dtype=np.int16)\n",
        "\n",
        "\n",
        "    stream.feedAudioContent(datal6) \n",
        "    text=stream.intermediateDecode() \n",
        "    #clear output(wait=True) \n",
        "    print(text) \n",
        "    offset=end_offset \n",
        "  return True "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zLvZ6VZgqYo3"
      },
      "source": [
        "transcribe_streaming('YahiSamayhaiSahiSamayHai.wav')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QdK_mrCXqYsa"
      },
      "source": [
        "def transcribe(audio_file):\n",
        "  buffer, rate =read_wav_file(audio_file)\n",
        "  data16 = np.frombuffer(buffer,dtype = np.int16)\n",
        "  return model.sttWithMetadata(data16)\n",
        "\n",
        "  transcribe('YahiSamayhaiSahiSamayHai.wav')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QW1XUCkol4Uq"
      },
      "source": [
        "!pip install easyocr\n",
        "import easyocr \n",
        "reader = easyocr.Reader(['en'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "O6eBkMy79AZ9"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "8_1. Speech (youtube video) to Text Transcription using AssemblyAI"
      ],
      "metadata": {
        "id": "Xm2ZHEWr-Nxt"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "api_key = '4f8acc9d2aed40e89dd5f3754dca0f92' # get API key from: https://app.assemblyai.com/\n",
        "\n",
        "!pip install pytube\n",
        "\n",
        "# 2. Retrieving audio file from YouTube video\n",
        "from pytube import YouTube\n",
        "video = YouTube(\"https://www.youtube.com/watch?v=mkVjrB8g6mM\")#\n",
        "yt = video.streams.get_audio_only()\n",
        "\n",
        "yt.download()\n",
        "\n",
        "import os\n",
        "current_dir = os.getcwd()\n",
        "\n",
        "for file in os.listdir(current_dir):\n",
        "    if file.endswith(\".mp4\"):\n",
        "        mp4_file = os.path.join(current_dir, file)\n",
        "        print(mp4_file)\n",
        "\n",
        "# 3. Upload YouTube audio file to AssemblyAI\n",
        "\n",
        "import sys\n",
        "import time\n",
        "import requests\n",
        "\n",
        "filename = mp4_file\n",
        "\n",
        "def read_file(filename, chunk_size=5242880):\n",
        "    with open(filename, 'rb') as _file:\n",
        "        while True:\n",
        "            data = _file.read(chunk_size)\n",
        "            if not data:\n",
        "                break\n",
        "            yield data\n",
        " \n",
        "headers = {'authorization': api_key}\n",
        "response = requests.post('https://api.assemblyai.com/v2/upload',\n",
        "                         headers=headers,\n",
        "                         data=read_file(filename))\n",
        "\n",
        "audio_url = response.json()['upload_url']\n",
        "\n",
        "# 4. Transcribe uploaded audio file\n",
        "\n",
        "import requests\n",
        "\n",
        "endpoint = \"https://api.assemblyai.com/v2/transcript\"\n",
        "\n",
        "json = {\n",
        "  \"audio_url\": audio_url\n",
        "}\n",
        "\n",
        "headers = {\n",
        "    \"authorization\": api_key,\n",
        "    \"content-type\": \"application/json\"\n",
        "}\n",
        "\n",
        "transcript_input_response = requests.post(endpoint, json=json, headers=headers)\n",
        "\n",
        "\n",
        "# 5. Extract transcript ID\n",
        "\n",
        "transcript_id = transcript_input_response.json()[\"id\"]\n",
        "\n",
        "\n",
        "\n",
        "# 6. Retrieve transcription results\n",
        "endpoint = f\"https://api.assemblyai.com/v2/transcript/{transcript_id}\"\n",
        "headers = {\n",
        "    \"authorization\": api_key,\n",
        "}\n",
        "\n",
        "transcript_output_response = requests.get(endpoint, headers=headers)\n",
        "\n",
        "print('6. Retrieve transcription results: ',transcript_output_response)\n",
        "\n",
        "transcript_output_response.json()['status']\n",
        "\n",
        "from time import sleep\n",
        "\n",
        "while transcript_output_response.json()['status'] != 'completed':\n",
        "  sleep(5)\n",
        "  print('Transcription is processing ...')\n",
        "  transcript_output_response = requests.get(endpoint, headers=headers)\n",
        "\n",
        "\n",
        "# 7. Print transcribed text\n",
        "print('Output:\\n')\n",
        "print(transcript_output_response.json())\n",
        "\n",
        "# 8. Save transcribed text to file\n",
        "\n",
        "# Save as TXT file\n",
        "yt_txt = open('yt.txt', 'w')\n",
        "yt_txt.write(transcript_output_response.json()[\"text\"])\n",
        "yt_txt.close()\n",
        "\n",
        "\n",
        "# Save as SRT file\n",
        "srt_endpoint = endpoint + \"/srt\"\n",
        "srt_response = requests.get(srt_endpoint, headers=headers)\n",
        "\n",
        "with open(\"yt.srt\", \"w\") as _file:\n",
        "    _file.write(srt_response.text)"
      ],
      "metadata": {
        "id": "5K8oK-vq6kfV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "ECT2Gzx66kvt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f5mpctOPGUpo"
      },
      "source": [
        "9. Dictionary"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OfGYXQORFYD8"
      },
      "source": [
        "import nltk\n",
        "nltk.download('wordnet')\n",
        "nltk.download('omw-1.4')\n",
        "from nltk.corpus import wordnet as wn\n",
        "#n --> noun\n",
        "# v verb\n",
        "#a or s Adjective\n",
        "#r adverb"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wnbJ8lMyFYJv"
      },
      "source": [
        "sense_word = input('Enter a word (e.g. cricket): ') #e.g. cricket and then cricket.n.01; day and day.n.04\n",
        "synsets = wn.synsets(sense_word)\n",
        "print('All possible senses for', sense_word,  'is: ',synsets )\n",
        "\n",
        "for sense_word in synsets:\n",
        "    print(\"\\nSense: \", sense_word.name())\n",
        "    print(\"Synonyms: \" , [lemma.name() for lemma in sense_word.lemmas()])\n",
        "    print (\"Anotonyms: \", [lemma.name() for lemma in sense_word.lemmas()[0].antonyms()])\n",
        "    \n",
        "#    print (\"Anotonyms: \", [a.name()for a in wn.synset('day.n.04').lemmas()[0].antonyms()]) \n",
        "    print(\"Gloss Definition: \" + sense_word.definition())\n",
        "    print(\"Example Sentemces: \" + str(sense_word.examples()))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "84vTBlMCfXfD"
      },
      "source": [
        "9_1 Correct the spelling"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ei6K5PYlFYcD"
      },
      "source": [
        "#10 Usage: Spelling Correction: deletion / insertion / substitution / transposition\n",
        "# !pip install textblob\n",
        "from textblob import TextBlob\n",
        "b = TextBlob(input('Enter word with incorrect spelling: '))\n",
        "print('Correct sentence is:', b.correct())#responsable, sily, arr"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from textblob import TextBlob, Word\n",
        "for text in ('how arr you doing','another sily mistake','I am responsable'):\n",
        "  print(TextBlob(text).correct())\n",
        "\n",
        "w = Word('sily')\n",
        "w.spellcheck()"
      ],
      "metadata": {
        "id": "yrdBYyMHFFOT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1tmndxZM-nma"
      },
      "source": [
        "10. Get similar sentences from his speech"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Sentence Similarity: Sentence Similarity is the task of determining how similar two texts are. Sentence similarity models convert input texts into vectors (embeddings) that capture semantic information and calculate how close (similar) they are between them. This task is particularly useful for information retrieval and clustering/grouping."
      ],
      "metadata": {
        "id": "OAnQ5Nti69jE"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dP-IPtOwgmeY"
      },
      "source": [
        "! pip install sentence-splitter\n",
        "! pip install transformers\n",
        "! pip install SentencePiece\n",
        "\n",
        "import torch\n",
        "from transformers import PegasusForConditionalGeneration, PegasusTokenizer\n",
        "\n",
        "model_name = 'tuner007/pegasus_paraphrase'\n",
        "torch_device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
        "tokenizer = PegasusTokenizer.from_pretrained(model_name)\n",
        "model = PegasusForConditionalGeneration.from_pretrained(model_name).to(torch_device)\n",
        "\n",
        "def get_response(tgt_text,num_return_sequences,num_beams): #def get_response(input_text,num_return_sequences,num_beams):\n",
        "  batch = tokenizer.prepare_seq2seq_batch(tgt_text,truncation=True,padding='longest',max_length=60, return_tensors=\"pt\").to(torch_device)\n",
        "\n",
        "  translated = model.generate(**batch,max_length=60,num_beams=10, num_return_sequences=num_return_sequences, temperature=1.5)\n",
        "  tgt_text = tokenizer.batch_decode(translated, skip_special_tokens=True)\n",
        "  return tgt_text\n",
        "\n",
        "num_return_sequences = 10\n",
        "num_beams = 10\n",
        "\n",
        "tgt_text = \"I have faith in the youth of my country, I trust the sisters of the country, the daughters of the country, the farmers of the country, and the professionals of the country.\"\n",
        "#tgt_text = \"The ultimate test of your knowledge is your capacity to convey it to another\"\n",
        "get_response(tgt_text,num_return_sequences,num_beams)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2OtZQCmlnhWN"
      },
      "source": [],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zF7uQD3Oroqh"
      },
      "source": [
        "## 11. ANY language to ENGLISH translation"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Translation: Translation is the task of converting text from one language to another."
      ],
      "metadata": {
        "id": "2Jjf8YTi6xvV"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "08Qry1J5nhgH"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')\n",
        "%cd \"/content/drive/My Drive/Colab Notebooks/POC_Projects/\"\n",
        "!pip install -q git+https://github.com/huggingface/transformers.git\n",
        "!pip install -q sentencepiece\n",
        "\n",
        "!wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip\n",
        "!unzip ngrok-stable-linux-amd64.zip\n",
        "\n",
        "get_ipython().system_raw('./ngrok http 8501 &')\n",
        "\n",
        "#ORIGINALS \n",
        "!pip install transformers"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9ukdu8HrrsSN"
      },
      "source": [
        "from transformers import MBartForConditionalGeneration, MBart50TokenizerFast\n",
        "model = MBartForConditionalGeneration.from_pretrained(\"facebook/mbart-large-50-many-to-many-mmt\") \n",
        "tokenizer = MBart50TokenizerFast.from_pretrained(\"facebook/mbart-large-50-many-to-many-mmt\", src_lang=\"hi_IN\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ktVf4EhdtHQR"
      },
      "source": [
        "article_en = \"पपीता\"\n",
        "#प्याज,आलू,आलु,\n",
        "model_inputs = tokenizer(article_en, return_tensors=\"pt\")\n",
        "\n",
        "generated_tokens = model.generate(\n",
        "    **model_inputs,\n",
        "    forced_bos_token_id=tokenizer.lang_code_to_id[\"en_XX\"]#hi_IN\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RSE1oQE6rsVt"
      },
      "source": [
        "translation = tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)\n",
        "translation"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6fdlHQ74rs6w"
      },
      "source": [],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "6txEtap2jzgZ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "12: Next word /Sentence prediction"
      ],
      "metadata": {
        "id": "HUXrXLX5t__m"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "from tensorflow.keras.optimizers import RMSprop,Adam\n",
        "\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense, Dropout, LSTM\n",
        "\n",
        "from tensorflow.keras.optimizers import RMSprop\n",
        "import numpy as np\n",
        "import random\n",
        "import sys"
      ],
      "metadata": {
        "id": "u2yk5L81jzju"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')\n",
        "filename = open('/content/drive/My Drive/Colab Notebooks/POC_Projects/PM’s Speech on 75th Independence Day.txt', \"r\", encoding = \"utf8\").read().replace('\\n', '')\n",
        "filename = open('/content/drive/My Drive/Colab Notebooks/POC_Projects/PM’s Speech on 75th Independence Day.txt', \"r\", encoding = \"utf8\").read().replace(\"\\ufeff\", \"\")\n",
        "\n",
        "raw_text = filename\n",
        "raw_text = raw_text.lower()\n",
        "# print(raw_text[0:1000])\n"
      ],
      "metadata": {
        "id": "cy1gcGYXuGKy"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "from tensorflow.keras.preprocessing.text import Tokenizer\n",
        "tokenizer = Tokenizer()\n",
        "corpus = raw_text.lower().split(\"\\n\")\n",
        "\n",
        "tokenizer.fit_on_texts(corpus)\n",
        "total_words = len(tokenizer.word_index) + 1\n",
        "total_words"
      ],
      "metadata": {
        "id": "cilA6iJViLif"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# tokenizer.word_index"
      ],
      "metadata": {
        "id": "8jn-Yv0LiqL_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#CLEAN TEXT\n",
        "#Remove numbers\n",
        "raw_text = ''.join(c for c in raw_text if not c.isdigit())\n",
        "\n",
        "#How many total characters do we have in our training text?\n",
        "chars = sorted(list(set(raw_text))) #List of every character\n",
        "\n",
        "#Character sequences must be encoded as integers. \n",
        "#Each unique character will be assigned an integer value. \n",
        "#Create a dictionary of characters mapped to integer values\n",
        "char_to_int = dict((c, i) for i, c in enumerate(chars))\n",
        "\n",
        "#Do the reverse so we can print our predictions in characters and not integers\n",
        "int_to_char = dict((i, c) for i, c in enumerate(chars))\n",
        "\n",
        "# summarize the data\n",
        "n_chars = len(raw_text)\n",
        "n_vocab = len(chars)\n",
        "# print(\"Total Characters in the text; corpus length: \", n_chars)\n",
        "# print(\"Total Vocab: \", n_vocab)"
      ],
      "metadata": {
        "id": "D4bv1BTOuGQi"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "seq_length = 60  #Length of each input sequence\n",
        "step = 10   #Instead of moving 1 letter at a time, try skipping a few. \n",
        "sentences = []    # X values (Sentences)\n",
        "next_chars = []   # Y values. The character that follows the sentence defined as X\n",
        "for i in range(0, n_chars - seq_length, step):  #step=1 means each sentence is offset just by a single letter\n",
        "    sentences.append(raw_text[i: i + seq_length])  #Sequence in\n",
        "    next_chars.append(raw_text[i + seq_length])  #Sequence out\n",
        "n_patterns = len(sentences)    \n",
        "# print('Number of sequences:', n_patterns)\n",
        "\n",
        "#Have a look at sentences and next_chars to see the continuity..."
      ],
      "metadata": {
        "id": "yBdCddiMuGdD"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "x = np.zeros((len(sentences), seq_length, n_vocab), dtype=np.bool)\n",
        "y = np.zeros((len(sentences), n_vocab), dtype=np.bool)\n",
        "for i, sentence in enumerate(sentences):\n",
        "    for t, char in enumerate(sentence):\n",
        "        x[i, t, char_to_int[char]] = 1\n",
        "    y[i, char_to_int[next_chars[i]]] = 1\n",
        "    \n",
        "# print(x.shape)"
      ],
      "metadata": {
        "id": "KjjSHlEqwMMt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "##################################################\n",
        "#Basic model with one LSTM\n",
        "# build the model: a single LSTM\n",
        "model = Sequential()\n",
        "model.add(LSTM(128, input_shape=(seq_length, n_vocab)))\n",
        "model.add(Dense(n_vocab, activation='softmax'))\n",
        "\n",
        "optimizer = RMSprop(lr=0.01)\n",
        "model.compile(loss='categorical_crossentropy', optimizer=optimizer,run_eagerly=None)\n",
        "# model.summary()"
      ],
      "metadata": {
        "id": "gFcfloQKwMVf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "###############\n",
        "# define the checkpoint\n",
        "from keras.callbacks import ModelCheckpoint\n",
        "\n",
        "filepath=\"saved_weights/saved_weights-{epoch:02d}-{loss:.4f}.hdf5\"\n",
        "checkpoint = ModelCheckpoint(filepath, monitor='loss', verbose=1, save_best_only=True, mode='min')\n",
        "\n",
        "callbacks_list = [checkpoint]"
      ],
      "metadata": {
        "id": "cz8yvfv4wMYa"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Fit the model\n",
        "history = model.fit(x, y,\n",
        "          batch_size=128,\n",
        "          epochs=50,   \n",
        "          callbacks=callbacks_list)\n",
        "\n",
        "model.save('my_saved_weights_jungle_book_50epochs.h5')"
      ],
      "metadata": {
        "id": "DOn5MwmswdgN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def sample(preds):\n",
        "    preds = np.asarray(preds).astype('float64')\n",
        "    preds = np.log(preds)\n",
        "    exp_preds = np.exp(preds) #exp of log (x), isn't this same as x??\n",
        "    preds = exp_preds / np.sum(exp_preds)\n",
        "    probas = np.random.multinomial(1, preds, 1) \n",
        "    return np.argmax(probas)"
      ],
      "metadata": {
        "id": "GM16cxoxwdms"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Prediction\n",
        "# load the network weights\n",
        "filename = \"my_saved_weights_jungle_book_50epochs.h5\"\n",
        "model.load_weights(filename)"
      ],
      "metadata": {
        "id": "H9e7Ag3Bwdpr"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#IMP: Pick a random sentence from the text as seed.\n",
        "# start_index = random.randint(0, n_chars - seq_length - 1)\n",
        "\n",
        "start_index = 377\n",
        "# print(start_index)\n",
        "range(start_index)"
      ],
      "metadata": {
        "id": "5e8mRBKSwdst"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "Qg5kZLE6CI33"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "L77gLKLlCJAt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Initiate generated text and keep adding new predictions and print them out\n",
        "generated = ''\n",
        "sentence = raw_text[start_index: start_index + seq_length]\n",
        "generated += sentence"
      ],
      "metadata": {
        "id": "kUu71Y-4wdyl"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print('Predicted text: \"' + sentence + '\"')\n",
        "#sys.stdout.write(generated)"
      ],
      "metadata": {
        "id": "U65mcWh4wd1m"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "SNNpfJ10weEt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "gotgvGG-weIC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "8ZICEwT9wMbS"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}