TopicModelTuning.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "TopicModelTuning_Working.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "cNBEw7aCwZoT"
      ],
      "mount_file_id": "1fN35KBnFAhmFZ--lyR7j3Rz7TfO2ucds",
      "authorship_tag": "ABX9TyPUtWu47zXTXg2wuSq1H2PD",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/drob-xx/TopicModelTuning/blob/main/TopicModelTuning.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "This notebook has code that parrallels the article [*Using Metrics to Determine  The Right LDA Topic Model Size*](https://drob707.medium.com/use-metrics-to-determine-lda-topic-model-size-1a1feaa1ff3c?source=friends_link&sk=27e152470f064c27fe28abd348b18c9b). Users can run the notebook and step-by-step re-create the procedures described in the article.\n",
        "\n",
        "To run the code presented here, follow this outline (details in the cells below):\n",
        "\n",
        "1.   Download two csv files from the GitHub repository into a directory accessible to the notebook.\n",
        "1.   Download the text DB csv file from Kaggle.\n",
        "1.   Assign the global directory value to the location of the above files.\n",
        "1. Install the required packages.\n",
        "1. Execute the imports.\n",
        "1. Run the cells containing Python function definitions used in the notebook.\n",
        "1. Generate the six models used in the evaluation. This shold take about 15 minutes on a standard Google Colab account. You can save the models for later use if desired.\n",
        "1. Run the evaluation code.\n"
      ],
      "metadata": {
        "id": "IhrtRhSU3Fi5"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Download CSV Files\n",
        "There are three csv files that are needed to run this notebook:\n",
        "\n",
        "In the [GitHub repository](https://github.com/drob-xx/TopicModelTuning):\n",
        "- ExcludelistDF.csv\n",
        "- ModelRunMetrics.csv\n",
        "\n",
        "On Kaggle\n",
        "- [NewsDF.csv](https://www.kaggle.com/datasets/danrobinson707/newsdf) \n",
        "\n",
        "***ExcludelistDF*** is a list of stop words which can be used when building models based on the sample text.\n",
        "\n",
        "***ModelRunMetrics*** are the metrics from 90 runs of the LDA and can be used to re-create and explore the data from the article.\n",
        "\n",
        "***NewsDF*** is a copy of the 30,000 article DB that has both the original text as well as pre-processed versions of the articles. You will need this if you want to run your own models AND if you want to explore the text that the models are built on.\n",
        "\n",
        "It is recommended that you place all of these files in a location accessible to the Colab notebook and referenced in the DATA_DIR variable"
      ],
      "metadata": {
        "id": "bXSmg4tgTBIW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Set DATA_DIR to a string corresponding to the location of the above datafiles\n",
        "DATA_DIR = <your data dir>"
      ],
      "metadata": {
        "id": "Yp5Ow9H4V5GA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Installs"
      ],
      "metadata": {
        "id": "hckCqoM93fj5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install kneed"
      ],
      "metadata": {
        "id": "aKYJu0rXGJI3"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Imports"
      ],
      "metadata": {
        "id": "vHuhFhhZ8sHD"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "import plotly.graph_objects as go\n",
        "from  plotly.subplots import make_subplots\n",
        "import collections\n",
        "import textwrap\n",
        "import pickle\n",
        "\n",
        "\n",
        "\n",
        "from kneed import KneeLocator"
      ],
      "metadata": {
        "id": "9A1zZ08OYr2X"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Defs"
      ],
      "metadata": {
        "id": "APOh9MkOZDW8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def plotKnee(x, y) :\n",
        "  k1 = KneeLocator(x, y, curve='concave' )\n",
        "  print(round(k1.knee, 3))\n",
        "  k1.plot_knee()\n",
        "\n",
        "def run_Topic_Metrics(DF, metricsToPlot, mode='markers', rows=2, cols=3) :\n",
        "  traces = []\n",
        "  fig = make_subplots(rows=rows, cols=cols)\n",
        "  xval = 1\n",
        "  yval = 1\n",
        "  for metric in metricsToPlot :\n",
        "      fig.append_trace(go.Scatter(x=DF['num_topics'], y=DF[metric], name=metric, mode=mode), yval, xval)\n",
        "      xval += 1\n",
        "      if xval > 3 :\n",
        "        xval = 1\n",
        "        yval += 1\n",
        "  return fig"
      ],
      "metadata": {
        "id": "O5d7lyZEd5KF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Model Monster"
      ],
      "metadata": {
        "id": "1iw25jowZNrV"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pickle\n",
        "import collections\n",
        "import textwrap\n",
        "\n",
        "import gensim.corpora as corpora\n",
        "from gensim.models.ldamodel import LdaModel\n",
        "\n",
        "class ModelMonster :\n",
        "  \n",
        "  # ModelsDict key = String Identifyer, Topic Size\n",
        "  def __init__(self, ModelsDict, CorpusText, StopWords=[], ContribMinThreshold=0.005) :\n",
        "    self.modelsdict = ModelsDict\n",
        "    self.corpustext = CorpusText\n",
        "    self.stopwords = StopWords\n",
        "    self.corpustext = CorpusText\n",
        "    self.models = {}\n",
        "    self.pctContribDFs = {}\n",
        "    self.topicIDDFs = {}\n",
        "    self.numdocspertopic = {}\n",
        "    self.corpus = None\n",
        "\n",
        "    self.contrib_min_threshold = ContribMinThreshold\n",
        "    self.num_topics_to_retrieve = 20\n",
        "\n",
        "    self._GenerateModels()\n",
        "    self._GenerateMatricies()\n",
        "\n",
        "    print('Models Complete')\n",
        "\n",
        "  def save(self, filepath):\n",
        "      with open(filepath, 'wb') as f:\n",
        "          pickle.dump(self, f)\n",
        "\n",
        "  @classmethod\n",
        "  def load(cls, filepath):\n",
        "      with open(filepath, 'rb') as f:\n",
        "          return pickle.load(f)\n",
        "\n",
        "  def _GenerateModels(self) :\n",
        "    if self.corpus == None :\n",
        "        self.docs = []\n",
        "        templist = [docs for docs in [lines.split() for lines in self.corpustext]]\n",
        "        for doc in templist :\n",
        "            tempdoc = []\n",
        "            if len(self.stopwords) > 0 :\n",
        "              for word in doc :\n",
        "                if word not in self.stopwords :\n",
        "                  tempdoc.append(word)\n",
        "              self.docs.append(tempdoc)\n",
        "        print('Generating Dictionary')\n",
        "        self.id2word = corpora.Dictionary(self.docs)\n",
        "        print('Generating Corpus')\n",
        "        self.corpus = [self.id2word.doc2bow(word) for word in self.docs] \n",
        "\n",
        "    for key, _ in self.modelsdict.items() :\n",
        "        print('Generating ', key, ' LDA model')\n",
        "        self.models[key] = LdaModel(self.corpus, num_topics=self.modelsdict[key])\n",
        "\n",
        "  def _GenerateMatricies(self) :\n",
        "      for key, _ in self.modelsdict.items() :\n",
        "        pctContribution = []\n",
        "        topicID = []\n",
        "        for _, row in enumerate( (self.models[key])[self.corpus] ) :\n",
        "          row = sorted(row, key=lambda x: (x[1]), reverse=True)\n",
        "          tempNum = []\n",
        "          tempVal = []\n",
        "          for _, tupe in enumerate(row) :\n",
        "            tempNum.append(tupe[0])\n",
        "            tempVal.append(tupe[1])\n",
        "          topicID.append(tempNum)\n",
        "          pctContribution.append(tempVal)\n",
        "        pctDF = pd.DataFrame(pctContribution)\n",
        "        # Set low scores to 0\n",
        "        self.pctContribDFs[key] = pctDF * (pctDF > self.contrib_min_threshold)\n",
        "        # Topic IDs are 0 based, increase all topic nums by 1, \n",
        "        #   mask out any where there are no corresponding pct contrib values\n",
        "        #   reset topicIDs by subtracting 1\n",
        "        self.topicIDDFs[key] = (((pd.DataFrame(topicID) + 1) * (self.pctContribDFs[key] > 0))) - 1\n",
        " \n",
        "  # Returns a dict where each key is the Model's name and the data is reverse \n",
        "  #    sorted OrderedDict of TopicIDs, # of docs where that TopicID is dominant\n",
        "  def GetNumDocsPerTopic(self, modelKeys = []) :\n",
        "    if type(modelKeys) == dict :\n",
        "      modelKeys = list(modelKeys.keys())\n",
        "    elif type(modelKeys) == str :\n",
        "      modelKeys = list([modelKeys])\n",
        "    elif ((type(modelKeys) == list) & (len(modelKeys) == 0)):\n",
        "      # run all models\n",
        "      modelKeys = list(self.modelsdict.keys())\n",
        "    elif type(modelKeys) != list :\n",
        "      print('ERROR - keys must be dict(keys), a single string, or list of strings resolving to a valid model name')\n",
        "\n",
        "    returnDict = {}\n",
        "    for modelKey in modelKeys :\n",
        "      returnDict[modelKey] = self._GetNumDocsPerTopic(modelKey)\n",
        "\n",
        "    return returnDict\n",
        "\n",
        "  # returns Ordered Dict of number of documents a topicID contributes to\n",
        "  def _GetNumDocsPerTopic(self, modelKey) :\n",
        "    topicIDDF = self.topicIDDFs[modelKey]\n",
        "    # For each topic, mask frame T/F for topicID, \n",
        "    #   then sum and filter out < 1, \n",
        "    #   resulting shape[0] is number of docs containing that topicID\n",
        "    TopicOccurence = {topicID: topicIDDF[pd.Series((topicIDDF == topicID).sum(axis=1))>0].shape[0] for topicID in range(self.modelsdict[modelKey])}\n",
        "    return collections.OrderedDict(sorted(TopicOccurence.items(), key=lambda item: item[1], reverse=True))\n",
        "\n",
        "  # Returns a dict where each key is the Model's name and the data is reverse \n",
        "  #    sorted OrderedDict of total of all percentage contributions that\n",
        "  #    Topic makes to all of its dominant topic documents (potentially useful\n",
        "  #    when comparing the weight of each topic to its documents)\n",
        "  def GetTotContribPerTopic(self, modelKeys = []) :\n",
        "    if type(modelKeys) == dict :\n",
        "      modelKeys = list(modelKeys.keys())\n",
        "    elif type(modelKeys) == str :\n",
        "      modelKeys = list([modelKeys])\n",
        "    elif ((type(modelKeys) == list) & (len(modelKeys) == 0)):\n",
        "      # run all models\n",
        "      modelKeys = list(self.modelsdict.keys())\n",
        "    elif type(modelKeys) != list :\n",
        "      print('ERROR - keys must be dict(keys), a single string, or list of strings resolving to a valid model name')\n",
        "      return\n",
        "\n",
        "    returnDict = {}\n",
        "    for modelKey in modelKeys :\n",
        "      returnDict[modelKey] = self._GetTotContribPerTopic(modelKey)\n",
        "\n",
        "    return returnDict\n",
        "\n",
        "  def _GetTotContribPerTopic(self, modelKey) :\n",
        "    topicIDDF = self.topicIDDFs[modelKey]\n",
        "    pctContribDF = self.pctContribDFs[modelKey]\n",
        "    pctContrib = {topicID - 1 : ((pctContribDF * ((topicIDDF + 1) == topicID)).sum()).sum() for topicID in range(1, self.modelsdict[modelKey]+1)}\n",
        "    return collections.OrderedDict(sorted(pctContrib.items(), key=lambda item: item[1], reverse=True))\n",
        "\n",
        "  # Returns an ordered dict where the values are reverse sorted by the \n",
        "  #   average stringl length of each topic keyword.\n",
        "  def GetAverageTopicWordLength(self, modelKeys = []) :\n",
        "    if type(modelKeys) == dict :\n",
        "      modelKeys = list(modelKeys.keys())\n",
        "    elif type(modelKeys) == str :\n",
        "      modelKeys = list([modelKeys])\n",
        "    elif ((type(modelKeys) == list) & (len(modelKeys) == 0)):\n",
        "      # run all models\n",
        "      modelKeys = list(self.modelsdict.keys())\n",
        "    elif type(modelKeys) != list :\n",
        "      print('ERROR - keys must be dict(keys), a single string, or list of strings resolving to a valid model name')\n",
        "      return\n",
        "\n",
        "    returnDict = {}\n",
        "    for modelKey in modelKeys :\n",
        "      returnDict[modelKey] = self._GetAverageTopicWordLength(modelKey)\n",
        "\n",
        "    return returnDict\n",
        "\n",
        "  def _GetAverageTopicWordLength(self, modelKey) :\n",
        "\n",
        "    model = self.models[modelKey]\n",
        "\n",
        "    topicWords = [[topic[0] for topic in model.show_topic(topicnum, topn=self.num_topics_to_retrieve)]  for topicnum in range(len(model.get_topics()))]\n",
        "    avgWordLengths = []\n",
        "    for topic in topicWords :\n",
        "      totalLen = 0\n",
        "      for word in topic :\n",
        "        totalLen += len(word)\n",
        "      avgWordLengths.append((totalLen / len(topic)))\n",
        "\n",
        "    avgTopicWordLengths = {topicNum: avg for topicNum, avg in enumerate(avgWordLengths) }\n",
        "    return collections.OrderedDict(sorted(avgTopicWordLengths.items(), key=lambda item: item[1]))\n",
        "\n",
        "  \n",
        "  # Returns a list of words that make up a topic definition\n",
        "  def GetTopics(self, modelKeys, topN=10) :\n",
        "    if type(modelKeys) == dict :\n",
        "      modelKeys = list(modelKeys.keys())\n",
        "    elif type(modelKeys) == str :\n",
        "      modelKeys = list([modelKeys])\n",
        "    elif ((type(modelKeys) == list) & (len(modelKeys) == 0)):\n",
        "      # run all models\n",
        "      modelKeys = list(self.modelsdict.keys())\n",
        "    elif type(modelKeys) != list :\n",
        "      print('ERROR - keys must be dict(keys), a single string, or list of strings resolving to a valid model name')\n",
        "      return\n",
        "\n",
        "    returnDict = {}\n",
        "    for modelKey in modelKeys :\n",
        "      returnDict[modelKey] = self._GetTopics(self.models[modelKey], [*range(self.modelsdict[modelKey])], topN)\n",
        "\n",
        "    return returnDict\n",
        "    \n",
        "    \n",
        "  def _GetTopics(self, model, listOfTopics, topN=10) :\n",
        "    vals = [[int(topic[0]) for topic in model.show_topic(topicnum, topn=topN)] for topicnum in listOfTopics]\n",
        "    return [[self.id2word[topic] for topic in topiclist] for topiclist in vals]\n",
        "\n",
        "  \n",
        "  # Prints a descending sorted list of the most to least represented topic. \n",
        "  #   Includes number of documents the topic is dominant for, the topic id\n",
        "  #   and the words that comprise that topic\n",
        "  def PrintTopicsCount(self, modelKeys) :\n",
        "    if type(modelKeys) == dict :\n",
        "      modelKeys = list(modelKeys.keys())\n",
        "    elif type(modelKeys) == str :\n",
        "      modelKeys = list([modelKeys])\n",
        "    elif ((type(modelKeys) == list) & (len(modelKeys) == 0)):\n",
        "      # run all models\n",
        "      modelKeys = list(self.modelsdict.keys())\n",
        "    elif type(modelKeys) != list :\n",
        "      print('ERROR - keys must be dict(keys), a single string, or list of strings resolving to a valid model name')\n",
        "      return\n",
        "\n",
        "    for modelKey in modelKeys :\n",
        "      self._PrintTopicsCount(modelKey)\n",
        "\n",
        "  def _PrintTopicsCount(self, modelkey) :\n",
        "    counterDict = collections.OrderedDict(sorted(collections.Counter(self.topicIDDFs[modelkey][0]).items(), key=lambda x: x[1], reverse=True))\n",
        "    print('')\n",
        "    print('==============================  ' + modelkey + '  ===================================')\n",
        "    numHD, topicHD, txtHD, underscore = 'Count', 'TopicID', 'Text', '-------'\n",
        "    print(f'{numHD:^8} {topicHD:^10} {txtHD}')\n",
        "    print(f'{underscore:^8} {underscore:^10} {underscore}')\n",
        "    allTopics = self.GetTopics(modelkey)[modelkey]\n",
        "    for topicID, count in counterDict.items()  :\n",
        "        topiclst = ' '.join(allTopics[topicID])\n",
        "        print(f'{count: ^ 8} {topicID: ^ 10} {topiclst:^}')\n",
        "\n",
        "  # Prints n randomly selected sample text for a given model where the dominant\n",
        "  #   topic for that text is within a range of likely presence in that Document\n",
        "  #   Prints the model name, the document's id, the dominant topic, the percent\n",
        "  #   liklihood that the topic is present in the document, the topic key word\n",
        "  #   list and n lines of the sample text.\n",
        "  def PrintSampleText(self, modelKeys, newsDF, MinRelevance=0.2, MaxRelevance=0.6, SampleSize=5, LinesToPrint=5) :\n",
        "    if type(modelKeys) == dict :\n",
        "      modelKeys = list(modelKeys.keys())\n",
        "    elif type(modelKeys) == str :\n",
        "      modelKeys = list([modelKeys])\n",
        "    elif ((type(modelKeys) == list) & (len(modelKeys) == 0)):\n",
        "      # run all models\n",
        "      modelKeys = list(self.modelsdict.keys())\n",
        "    elif type(modelKeys) != list :\n",
        "      print('ERROR - keys must be dict(keys), a single string, or list of strings resolving to a valid model name')\n",
        "      return\n",
        "\n",
        "    for modelKey in modelKeys :\n",
        "      self._PrintSampleText(modelKey, newsDF, MinRelevance, MaxRelevance, SampleSize, LinesToPrint )\n",
        "\n",
        "  def _PrintSampleText(self, modelkey, newsDF, MinRelevance, MaxRelevance, SampleSize, LinesToPrint) :\n",
        "    textSamples = pd.DataFrame()\n",
        "    textSamples['pctContribution'] = self.pctContribDFs[modelkey][0]\n",
        "    textSamples['topics'] = self.topicIDDFs[modelkey][0]\n",
        "    textSamples['text'] = newsDF['Content']\n",
        "\n",
        "    samplesDF = textSamples[(textSamples['pctContribution'] >= MinRelevance) & (textSamples['pctContribution'] <= MaxRelevance )].sample(n=SampleSize, replace=True)\n",
        "\n",
        "    topicslist = self.GetTopics(modelkey)[modelkey]\n",
        "\n",
        "    for row in samplesDF.iterrows() :\n",
        "        print('')\n",
        "        print('************************')\n",
        "        print('Model: ', modelkey)\n",
        "        print('Document ID: ', row[0])\n",
        "        print('Topic: ', row[1]['topics'])\n",
        "        print('Contribution: ', row[1]['pctContribution'])\n",
        "        print('')\n",
        "        print(' '.join(topicslist[row[1]['topics']]))\n",
        "        print('')\n",
        "        for line in textwrap.wrap(row[1]['text'])[:LinesToPrint] :\n",
        "          print(line)\n",
        "    print('')\n",
        "    print('')"
      ],
      "metadata": {
        "id": "BLr7uzASZNFK"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Load Data "
      ],
      "metadata": {
        "id": "lbnbnKEXepZn"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "NewsDF = pd.read_csv(DATA_DIR + '/NewsDF.csv')\n",
        "StopWordsDF = pd.read_csv(DATA_DIR + '/ExcludelistDF.csv')\n",
        "ModelRunMetricsDF = pd.read_csv(DATA_DIR + '/ModelRunMetrics.csv')\n",
        "\n",
        "# A dictionary that can be used to intialize the ModelMonster.\n",
        "ModelsToEval = {'Five': 5, 'Ten': 10, 'Twenty': 20, 'Thirtyfive' : 35, 'Fifty' : 50, 'Eighty' : 80}"
      ],
      "metadata": {
        "id": "to4oYLbE8v-x"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Run this to generate the ModelMonster. Will run for approx. 15 minutes in Colab\n",
        "MM = ModelMonster(ModelsToEval, NewsDF['Clean_Text'], StopWords=list(StopWordsDF['0']))"
      ],
      "metadata": {
        "id": "8KOgbOvANaY6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Save the ModelMonster for later use if you wish\n",
        "MM.save(DATA_DIR + '/MM.pck')"
      ],
      "metadata": {
        "id": "S8LPC6aoc1w9"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Read a previously saved version of the ModelMonster\n",
        "MM = ModelMonster.load(DATA_DIR + '/MM.pck')"
      ],
      "metadata": {
        "id": "jVAdzMRddIc4"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Analyze"
      ],
      "metadata": {
        "id": "X_RyAJI4ZqND"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# List of metrics to be graphed.\n",
        "metrics = ['KLBac', 'KLDiv',  'KLUni', 'KLVac', 'JaccardSim', 'InvertedRBO', 'Topic_Diversity', 'NPMI', 'C_V']"
      ],
      "metadata": {
        "id": "YMC1PDCn3VXc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "fig = run_Topic_Metrics(ModelRunMetricsDF, metrics, rows=3)\n",
        "fig.show()\n"
      ],
      "metadata": {
        "id": "zggtgmLpTBMW",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 542
        },
        "outputId": "33771ce5-7a15-4cfd-9726-a637bb2bc2e5"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<html>\n",
              "<head><meta charset=\"utf-8\" /></head>\n",
              "<body>\n",
              "    <div>            <script src=\"https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-AMS-MML_SVG\"></script><script type=\"text/javascript\">if (window.MathJax) {MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}</script>                <script type=\"text/javascript\">window.PlotlyConfig = {MathJaxConfig: 'local'};</script>\n",
              "        <script src=\"https://cdn.plot.ly/plotly-2.8.3.min.js\"></script>                <div id=\"31e6c71b-23b0-43fa-a533-a298eacc966e\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"31e6c71b-23b0-43fa-a533-a298eacc966e\")) {                    Plotly.newPlot(                        \"31e6c71b-23b0-43fa-a533-a298eacc966e\",                        [{\"mode\":\"markers\",\"name\":\"KLBac\",\"x\":[25,25,20,15,10,5,25,20,15,10,5,20,15,10,5,55,50,45,40,35,30,55,50,45,40,35,30,55,50,45,40,35,30,150,145,140,135,130,125,120,115,110,105,100,95,90,85,80,75,70,65,60,150,145,140,135,130,125,120,115,110,105,100,95,90,85,80,75,70,65,60,150,145,140,135,130,125,120,115,110,105,100,95,90,85,80,75,70,65,60],\"y\":[1.4406151876784974,1.4117390486100652,1.3103425499423984,1.1433720061259678,0.9979720220987,0.7071381351777807,1.4163456684631126,1.3389260766135824,1.1725780634476275,1.0104470064815785,0.7478745764007432,1.371785034626659,1.240118861770167,1.0100703988197772,0.724634469237413,1.73092181049817,1.7378664822866678,1.6543187747942465,1.584332580149609,1.5853598648008478,1.5241190785308982,1.6878152666993993,1.7241230640810212,1.6617582508966695,1.5838650581284175,1.5502372544052856,1.4492643387929482,1.7630329188495686,1.7239769394009938,1.628544139200448,1.5984228599412262,1.5198430279359467,1.4965241985900344,2.087006738862004,2.0784831896179043,2.034391999137058,2.0276788852859435,2.012375299982982,2.002357352004281,1.9798469472555555,1.9508853038651337,1.928321954042796,1.9525837236970407,1.932793257308664,1.9011823379181976,1.906904148211984,1.8802472807002164,1.858126166773168,1.806521034662789,1.839078199318109,1.822288259312309,1.746524168288438,2.0346155756167,2.058507489310803,2.032990716183985,2.0240225274263626,2.028210375043152,2.004960910666464,1.9790825909916632,1.9535680027387,1.9580448701255444,1.930870894501056,1.915561098274682,1.911510931281903,1.894076074920791,1.8785931141045904,1.8829748121832937,1.8514215569473784,1.8253216815443964,1.7965352393114904,1.75283110311125,2.0825959022049974,2.0467394887262924,2.0446426167443,2.021097489973984,2.0128785285143462,1.9849538493325247,1.9741437495651588,1.958428441765654,1.9370259395482616,1.916523060462085,1.9037731099935584,1.8965354808549053,1.8668221747967384,1.86867566822582,1.8636106162018264,1.859517348242352,1.8332538906836964,1.801699096032209,1.787520734453444],\"type\":\"scatter\",\"xaxis\":\"x\",\"yaxis\":\"y\"},{\"mode\":\"markers\",\"name\":\"KLDiv\",\"x\":[25,25,20,15,10,5,25,20,15,10,5,20,15,10,5,55,50,45,40,35,30,55,50,45,40,35,30,55,50,45,40,35,30,150,145,140,135,130,125,120,115,110,105,100,95,90,85,80,75,70,65,60,150,145,140,135,130,125,120,115,110,105,100,95,90,85,80,75,70,65,60,150,145,140,135,130,125,120,115,110,105,100,95,90,85,80,75,70,65,60],\"y\":[1.8587314514319104,1.806380603114764,1.64452488140056,1.3853696295193263,1.1673060046301948,0.8846262633800507,1.8320488427082693,1.7381305142452843,1.4730048826762607,1.220043749279446,0.9652930080890656,1.6986537171037577,1.4716825127601625,1.2228915201293098,0.88687943816185,2.672506839699216,2.510964993408748,2.2946511583496827,2.225433561817193,2.1751065001768226,1.9650111317634584,2.4429924693171827,2.65629723505098,2.45508439980372,2.178145446991309,2.2814835062547894,1.931535260841764,2.6423487744347414,2.5281999190486206,2.395028109442104,2.162179326476195,2.001599586911562,2.045093860160345,3.847523535284687,3.815627264342774,3.699000420011695,3.747574828315532,3.736844310272856,3.474399191687184,3.5404413495744977,3.315629678678185,3.266839605236769,3.3092159033694983,3.206373856296443,3.1417005151677104,3.1516687629523497,3.011792732921301,3.074654648443566,2.836033590376914,2.9412445869751846,2.7952013235252635,2.611579189260127,3.733656235839963,3.712082541502755,3.67232309180443,3.5638318187166407,3.562035159113582,3.532992926897541,3.484309395321277,3.402169984138076,3.341035989963382,3.207685341645073,3.1701028105466054,3.2364124006279904,3.07187960859244,3.071542152837545,2.911236487876011,2.9218866111781145,2.9008045252559103,2.8844053644686936,2.5704164270290546,3.8206950735619136,3.8428701961120426,3.659023386135866,3.541162516057261,3.6976382585103553,3.6093258509712833,3.3881225161191795,3.271126305975139,3.2245747543952183,3.160697481047103,3.1951490306131767,3.127497163598508,3.029740284385157,2.999767637152632,3.03857302263945,3.01302525842512,2.874175628668033,2.741394539005481,2.7373498566069845],\"type\":\"scatter\",\"xaxis\":\"x2\",\"yaxis\":\"y2\"},{\"mode\":\"markers\",\"name\":\"KLUni\",\"x\":[25,25,20,15,10,5,25,20,15,10,5,20,15,10,5,55,50,45,40,35,30,55,50,45,40,35,30,55,50,45,40,35,30,150,145,140,135,130,125,120,115,110,105,100,95,90,85,80,75,70,65,60,150,145,140,135,130,125,120,115,110,105,100,95,90,85,80,75,70,65,60,150,145,140,135,130,125,120,115,110,105,100,95,90,85,80,75,70,65,60],\"y\":[2.8831140343100063,2.937642753888671,2.8290677720307764,2.6894116656315257,2.5947780486625622,2.3159791525018125,2.9104289291981136,2.8539858080349307,2.7316541623259374,2.580613428496691,2.3381061906557115,2.8044763348566475,2.6644448130337266,2.592696656419988,2.322350477350414,3.366703841599047,3.27249817826246,3.18049394740006,3.14650176144741,3.058339621429777,3.005957888295272,3.25835881068414,3.3241966393517,3.2047506241457464,3.125671665484737,3.061200150767972,2.981963874859173,3.2967902086558776,3.2498708640931544,3.217590437055966,3.1424022152600672,3.0512668173457684,3.005334171746491,4.08813943902169,4.012204281744139,3.961141410198677,3.97624554005552,3.917658638939789,3.842041490013572,3.851590343310397,3.7604782287354808,3.754500536250324,3.738617069641193,3.6759073207959214,3.6385712032070936,3.6280637069347166,3.609952756070864,3.605462373202966,3.481083070990071,3.4856835366930468,3.4153072647807114,3.4010448340128323,3.997626066824248,3.993226437450284,3.926484289198857,3.9197999901074176,3.92513903624741,3.875605250766449,3.824017988047722,3.763420086259072,3.76816941227883,3.7353396568056487,3.657590792920293,3.66025054286694,3.606862570288016,3.5864839186510427,3.5522495738287576,3.525452446606634,3.485320032975066,3.437038346617015,3.354784299544688,4.0275516016400665,4.033248871992066,3.9367430052451886,3.903800300898775,3.943700916411024,3.8727908059501495,3.7801523957244014,3.7666853842762498,3.721024299722633,3.668341568019472,3.664576912596282,3.631026908985935,3.582689268353218,3.5751249261693308,3.562290204817927,3.534039187074868,3.495658553754799,3.435918033083013,3.353480721208288],\"type\":\"scatter\",\"xaxis\":\"x3\",\"yaxis\":\"y3\"},{\"mode\":\"markers\",\"name\":\"KLVac\",\"x\":[25,25,20,15,10,5,25,20,15,10,5,20,15,10,5,55,50,45,40,35,30,55,50,45,40,35,30,55,50,45,40,35,30,150,145,140,135,130,125,120,115,110,105,100,95,90,85,80,75,70,65,60,150,145,140,135,130,125,120,115,110,105,100,95,90,85,80,75,70,65,60,150,145,140,135,130,125,120,115,110,105,100,95,90,85,80,75,70,65,60],\"y\":[0.9816380681891632,0.921020502315864,0.8305538795022421,0.6486913427600386,0.5321428650679539,0.3312874016640081,0.9204274906773952,0.8615159788453566,0.6953420784975378,0.5331172789923322,0.3554002615482277,0.8603662638510583,0.7360699047322897,0.538204706261909,0.339743636004128,1.4983885653282705,1.4476290496329682,1.312924079375067,1.259399073128843,1.1935661530634984,1.0676995558208122,1.3821340657052092,1.5400857089334716,1.387613877560682,1.1863573680073372,1.2439940441585202,1.0040922779865398,1.5599437511493934,1.4282150821883786,1.3722631445722606,1.2051429991544456,1.0881935570528647,1.0960340542768223,2.46186299406944,2.4131415913021166,2.350198106140637,2.379644725822232,2.339382612581656,2.144156918953396,2.187231084920348,2.0329294158877658,2.034521722532625,2.0445745889732807,1.9661297652837169,1.9516223876434184,1.9042747539668243,1.8685599294473143,1.8709809218360207,1.662836674769617,1.7534671849254513,1.6823144625026467,1.5274324267895276,2.377733936331575,2.3793711635175416,2.3289549219479846,2.281869802998008,2.2342754675015124,2.2062466702539654,2.162560157971319,2.03566727685237,2.030213322153256,1.970701029251624,1.952298169774625,1.974684259576536,1.8521066764322616,1.8329349500952583,1.787611958411078,1.7458869150686114,1.7749096065702623,1.6888342726433188,1.4397415007286476,2.42254815150352,2.459407063456037,2.322149766513327,2.197503511431594,2.348587219299626,2.257592408238988,2.103137971373729,2.001948701872934,2.009066395423264,1.9628108529127024,1.9818213377153115,1.9066765812938369,1.7804711368540438,1.8052512083751069,1.8383124471346044,1.8369834100290856,1.740818778791407,1.63702019424778,1.623381551289728],\"type\":\"scatter\",\"xaxis\":\"x4\",\"yaxis\":\"y4\"},{\"mode\":\"markers\",\"name\":\"JaccardSim\",\"x\":[25,25,20,15,10,5,25,20,15,10,5,20,15,10,5,55,50,45,40,35,30,55,50,45,40,35,30,55,50,45,40,35,30,150,145,140,135,130,125,120,115,110,105,100,95,90,85,80,75,70,65,60,150,145,140,135,130,125,120,115,110,105,100,95,90,85,80,75,70,65,60,150,145,140,135,130,125,120,115,110,105,100,95,90,85,80,75,70,65,60],\"y\":[0.0113582158009402,0.0078362573099415,0.0097007223942208,0.0152005831572394,0.0168940870695256,0.0680426556587547,0.0102832244008714,0.0117683269060163,0.0158689206676822,0.0196460650537017,0.043859649122807,0.0116144334003222,0.0183334152374399,0.0196689982035699,0.060921912624699,0.0044726012562269,0.0045867469794935,0.0046278922697911,0.0050333853739426,0.0061971537841142,0.0085730440593256,0.006311886205247,0.0050571807670436,0.0041724433881296,0.0065502809310858,0.0048292588449208,0.0106395976418395,0.0046450052642003,0.0063951896548092,0.0056852494674992,0.0076970486800208,0.0059676295698878,0.0065383201072322,0.0014493827673539,0.0020263664382201,0.002004102158344,0.0020677771559585,0.0017368373873521,0.0022024035420619,0.0019291548748842,0.0028491814780966,0.0023686749621932,0.0022986702553265,0.0029849231913215,0.0020830709054215,0.0032138619295539,0.0026567765048915,0.0032962502122767,0.0040522256002132,0.003127882223319,0.003679447487497,0.0044964035767939,0.0017009624996825,0.0016924401003259,0.0014981734205735,0.0017447449143458,0.0019578510038586,0.0024076145455352,0.0018299665927602,0.0025585943291095,0.0028508305711702,0.0026793825942432,0.0023864201005583,0.002329069385362,0.0032797195799711,0.0031569697022255,0.0038150492264416,0.0035302795055116,0.0027955296026368,0.003411857320526,0.0057559776809921,0.0015299968737747,0.0013778012381264,0.0017900584441778,0.0024773221383051,0.001854754392734,0.0017732170400701,0.0023162963155678,0.0022002179436416,0.0024449052114005,0.0025634725325127,0.0027588926763333,0.0024663023300516,0.0038547661656159,0.0030943130606215,0.0030160293075702,0.0025675210814529,0.0030271401721125,0.0038279616839988,0.0028479862556579],\"type\":\"scatter\",\"xaxis\":\"x5\",\"yaxis\":\"y5\"},{\"mode\":\"markers\",\"name\":\"InvertedRBO\",\"x\":[25,25,20,15,10,5,25,20,15,10,5,20,15,10,5,55,50,45,40,35,30,55,50,45,40,35,30,55,50,45,40,35,30,150,145,140,135,130,125,120,115,110,105,100,95,90,85,80,75,70,65,60,150,145,140,135,130,125,120,115,110,105,100,95,90,85,80,75,70,65,60,150,145,140,135,130,125,120,115,110,105,100,95,90,85,80,75,70,65,60],\"y\":[0.9866557174978096,0.990317785855738,0.9890269482043608,0.9825338053755782,0.9773760431977778,0.9113763480914284,0.9888890504022144,0.9840392072663158,0.9812671820438776,0.9769493886228572,0.9447922923657144,0.9859527763785338,0.9810361163781632,0.9743701287442856,0.9317072307957144,0.9944128376708802,0.9943601745588456,0.994372742056645,0.9938191085876557,0.9915315398267948,0.9896352179295238,0.9927694349922896,0.99477833186228,0.9939778851824892,0.9923652970320604,0.994560867609748,0.9886464960938588,0.9947873486198028,0.9921637199113528,0.9931616891098628,0.9909988331713828,0.993115506032401,0.9921850352726108,0.998225671836964,0.9974264618248836,0.9975859403079672,0.9977021609269604,0.9978682630141708,0.9974124012343356,0.997630911745048,0.9968048131464738,0.9972401731446516,0.9972866600476662,0.9963347311788876,0.9975286327845976,0.9962363677884892,0.996573065057271,0.9958477056832684,0.9953865174587156,0.9961736116407246,0.9957108771745158,0.994607046182292,0.9980186405581132,0.9979155393895636,0.9982152193722044,0.9978598616455112,0.9975818092507786,0.997129309871364,0.997960795255058,0.996899664604118,0.996629031367698,0.9967997669460192,0.997244276991733,0.9972384573776452,0.9959444946638006,0.996326768950104,0.9955570192823532,0.9958728600169472,0.9968905274382877,0.9956795186302988,0.9933942710838256,0.9981311425697073,0.9983786538082868,0.9977978783880128,0.99713952869501,0.9977256175830564,0.9979290945352904,0.9973885268832952,0.9974183162314132,0.9971012083074908,0.996925620745068,0.9965472685678484,0.997110990318973,0.995541483239194,0.9962193933237916,0.9966071396120728,0.9970190967542936,0.9964034169265276,0.9957149763049828,0.996326505199435],\"type\":\"scatter\",\"xaxis\":\"x6\",\"yaxis\":\"y6\"},{\"mode\":\"markers\",\"name\":\"Topic_Diversity\",\"x\":[25,25,20,15,10,5,25,20,15,10,5,20,15,10,5,55,50,45,40,35,30,55,50,45,40,35,30,55,50,45,40,35,30,150,145,140,135,130,125,120,115,110,105,100,95,90,85,80,75,70,65,60,150,145,140,135,130,125,120,115,110,105,100,95,90,85,80,75,70,65,60,150,145,140,135,130,125,120,115,110,105,100,95,90,85,80,75,70,65,60],\"y\":[0.824,0.852,0.855,0.8466666666666667,0.88,0.82,0.848,0.845,0.84,0.87,0.84,0.845,0.8466666666666667,0.86,0.8,0.8363636363636363,0.852,0.8577777777777778,0.86,0.8542857142857143,0.83,0.8090909090909091,0.86,0.8622222222222222,0.8325,0.8857142857142857,0.8166666666666667,0.8418181818181818,0.796,0.8488888888888889,0.8175,0.86,0.8633333333333333,0.866,0.8379310344827586,0.8557142857142858,0.8414814814814815,0.8638461538461538,0.8384,0.8566666666666667,0.8043478260869565,0.8409090909090909,0.8523809523809524,0.829,0.868421052631579,0.8322222222222222,0.8505882352941176,0.83,0.824,0.86,0.8523076923076923,0.8366666666666667,0.8453333333333334,0.8496551724137931,0.86,0.8637037037037038,0.8476923076923077,0.8272,0.8633333333333333,0.8252173913043478,0.82,0.839047619047619,0.853,0.8589473684210527,0.8344444444444444,0.8364705882352941,0.81125,0.84,0.8685714285714285,0.8492307692307692,0.7966666666666666,0.8486666666666667,0.8724137931034482,0.85,0.8133333333333334,0.8569230769230769,0.8656,0.8375,0.8408695652173913,0.8354545454545454,0.84,0.842,0.8442105263157895,0.7955555555555556,0.8305882352941176,0.85125,0.8733333333333333,0.8571428571428571,0.8338461538461538,0.88],\"type\":\"scatter\",\"xaxis\":\"x7\",\"yaxis\":\"y7\"},{\"mode\":\"markers\",\"name\":\"NPMI\",\"x\":[25,25,20,15,10,5,25,20,15,10,5,20,15,10,5,55,50,45,40,35,30,55,50,45,40,35,30,55,50,45,40,35,30,150,145,140,135,130,125,120,115,110,105,100,95,90,85,80,75,70,65,60,150,145,140,135,130,125,120,115,110,105,100,95,90,85,80,75,70,65,60,150,145,140,135,130,125,120,115,110,105,100,95,90,85,80,75,70,65,60],\"y\":[0.0489365524349263,0.0893486011441021,0.0868149730189454,0.0703479741449751,0.0625288871042963,0.0267792660606761,0.0754743842846321,0.0833656384893754,0.068218648110596,0.0590325173321602,0.030410639170017,0.0767667255935054,0.0553503980463805,0.0556769355239681,0.0324710024446915,0.0719468539250655,0.0569129622212347,0.0561116210939235,0.0590201380527626,0.0572603000542404,0.086274295804389,0.0350067589386663,0.0328754832160175,0.0506236466759616,0.0717143104818161,0.0405225542193326,0.0819817849942592,0.0197640509644611,0.0467443578094415,0.0650056852024533,0.0744652284395143,0.0829826666489235,0.0636665840876857,-0.0154330874029314,-0.0149914489143367,-0.0231064316936348,-0.020091798613567,-0.0185253066710766,0.0123942369101477,0.0024590548092713,0.0004841525813294,0.0215934277519974,0.008176687874129,0.0249198100338251,0.0097946270972635,0.0336442845948459,0.0242131015714447,0.0341962996616098,0.0470409147309825,0.0416912910318515,0.0361823831746424,0.0705578573252095,-0.0172494781802177,-0.0170056395358779,-0.0143966528444905,-0.0065312794728697,0.0036017316418377,-0.0003786080273187,0.0099580321111553,0.0180578971806999,0.0194935461070774,0.0261574739580419,0.0042674648167049,0.0070683447073858,0.0174713565519456,0.0262703111345701,0.0449011245298355,0.0395504216122182,0.0422905610074432,0.0371750402072305,0.067836863101004,-0.0082618963395579,-0.0214447709279194,-0.0100695660571762,0.0134603398663487,-0.0240903345315909,-0.021923827386958,0.0008707651583927,0.0281115205084529,0.0085360560857349,0.0034890680807317,-0.0047910854878809,0.0258305350864486,0.0457415204201334,0.0317850638865991,0.0235062138144199,0.0257644426670136,0.0468010926030784,0.056059932879963,0.0366334650875383],\"type\":\"scatter\",\"xaxis\":\"x8\",\"yaxis\":\"y8\"},{\"mode\":\"markers\",\"name\":\"C_V\",\"x\":[25,25,20,15,10,5,25,20,15,10,5,20,15,10,5,55,50,45,40,35,30,55,50,45,40,35,30,55,50,45,40,35,30,150,145,140,135,130,125,120,115,110,105,100,95,90,85,80,75,70,65,60,150,145,140,135,130,125,120,115,110,105,100,95,90,85,80,75,70,65,60,150,145,140,135,130,125,120,115,110,105,100,95,90,85,80,75,70,65,60],\"y\":[0.5289685719892782,0.5941950218889887,0.5921110446794593,0.5534631915022196,0.5432795933870139,0.451385413895763,0.5702453675083589,0.5855115516900498,0.5638815221001541,0.5296766846851726,0.4726645356869396,0.5922598056429407,0.5286883366025444,0.5279520276405786,0.4741337383973179,0.5895677273355556,0.579874989688445,0.5707468175193412,0.5690183972599536,0.5663014717091652,0.5882057019465595,0.5290887591457037,0.5349164590277871,0.5660839818307669,0.5664666799030125,0.5546661649994953,0.5857942286198352,0.5311032176426115,0.5536066231100416,0.57107946596485,0.5776379837232788,0.5937524386759189,0.5725402884387055,0.5112383792060785,0.5219264738489469,0.4944106058803199,0.4954765570792035,0.5045003082996763,0.5368919143858857,0.5191961473314568,0.5178589262869558,0.5447057469533764,0.5293509003365706,0.5490179074091524,0.5367501839222653,0.5539798891523724,0.5425925291541726,0.551226406740421,0.5504584311464026,0.5701981966419993,0.5621843503580798,0.5765860061622249,0.5032865661982063,0.5080832046277625,0.5111396625573784,0.508135408876484,0.5163889103984155,0.5172641390364202,0.524611848793938,0.5384556221205968,0.5301076068284795,0.5387128050580442,0.513417842321774,0.5212531997963012,0.5355261824890882,0.5365284552158187,0.5676222486239096,0.5522790356583979,0.5709835551520756,0.5531356694471649,0.5798309460299209,0.5186273450878476,0.4973319091543689,0.5203252032087347,0.5281539192106836,0.4925987912919308,0.5029607259596761,0.5253125140534344,0.5526680991659491,0.5226989048611959,0.5132266294772612,0.5131941453336981,0.5441533568747304,0.5453583407688907,0.5471058612614635,0.5494386289935222,0.5453528616751668,0.5648425089670366,0.5681182093806101,0.5603666908498041],\"type\":\"scatter\",\"xaxis\":\"x9\",\"yaxis\":\"y9\"}],                        {\"template\":{\"data\":{\"bar\":[{\"error_x\":{\"color\":\"#2a3f5f\"},\"error_y\":{\"color\":\"#2a3f5f\"},\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"baxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"type\":\"carpet\"}],\"choropleth\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"type\":\"choropleth\"}],\"contour\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"contour\"}],\"contourcarpet\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"type\":\"contourcarpet\"}],\"heatmap\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"heatmap\"}],\"heatmapgl\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"heatmapgl\"}],\"histogram\":[{\"marker\":{\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"histogram\"}],\"histogram2d\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"histogram2d\"}],\"histogram2dcontour\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"histogram2dcontour\"}],\"mesh3d\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"type\":\"mesh3d\"}],\"parcoords\":[{\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"parcoords\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}],\"scatter\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatter\"}],\"scatter3d\":[{\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatter3d\"}],\"scattercarpet\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scattercarpet\"}],\"scattergeo\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scattergeo\"}],\"scattergl\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scattergl\"}],\"scattermapbox\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scattermapbox\"}],\"scatterpolar\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatterpolar\"}],\"scatterpolargl\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatterpolargl\"}],\"scatterternary\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatterternary\"}],\"surface\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"surface\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"#EBF0F8\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"#C8D4E3\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}]},\"layout\":{\"annotationdefaults\":{\"arrowcolor\":\"#2a3f5f\",\"arrowhead\":0,\"arrowwidth\":1},\"autotypenumbers\":\"strict\",\"coloraxis\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"colorscale\":{\"diverging\":[[0,\"#8e0152\"],[0.1,\"#c51b7d\"],[0.2,\"#de77ae\"],[0.3,\"#f1b6da\"],[0.4,\"#fde0ef\"],[0.5,\"#f7f7f7\"],[0.6,\"#e6f5d0\"],[0.7,\"#b8e186\"],[0.8,\"#7fbc41\"],[0.9,\"#4d9221\"],[1,\"#276419\"]],\"sequential\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"sequentialminus\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]},\"colorway\":[\"#636efa\",\"#EF553B\",\"#00cc96\",\"#ab63fa\",\"#FFA15A\",\"#19d3f3\",\"#FF6692\",\"#B6E880\",\"#FF97FF\",\"#FECB52\"],\"font\":{\"color\":\"#2a3f5f\"},\"geo\":{\"bgcolor\":\"white\",\"lakecolor\":\"white\",\"landcolor\":\"#E5ECF6\",\"showlakes\":true,\"showland\":true,\"subunitcolor\":\"white\"},\"hoverlabel\":{\"align\":\"left\"},\"hovermode\":\"closest\",\"mapbox\":{\"style\":\"light\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"#E5ECF6\",\"polar\":{\"angularaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"bgcolor\":\"#E5ECF6\",\"radialaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"gridwidth\":2,\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\"},\"yaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"gridwidth\":2,\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\"},\"zaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"gridwidth\":2,\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\"}},\"shapedefaults\":{\"line\":{\"color\":\"#2a3f5f\"}},\"ternary\":{\"aaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"baxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"bgcolor\":\"#E5ECF6\",\"caxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"title\":{\"x\":0.05},\"xaxis\":{\"automargin\":true,\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"zerolinewidth\":2},\"yaxis\":{\"automargin\":true,\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"zerolinewidth\":2}}},\"xaxis\":{\"anchor\":\"y\",\"domain\":[0.0,0.2888888888888889]},\"yaxis\":{\"anchor\":\"x\",\"domain\":[0.7333333333333333,1.0]},\"xaxis2\":{\"anchor\":\"y2\",\"domain\":[0.35555555555555557,0.6444444444444445]},\"yaxis2\":{\"anchor\":\"x2\",\"domain\":[0.7333333333333333,1.0]},\"xaxis3\":{\"anchor\":\"y3\",\"domain\":[0.7111111111111111,1.0]},\"yaxis3\":{\"anchor\":\"x3\",\"domain\":[0.7333333333333333,1.0]},\"xaxis4\":{\"anchor\":\"y4\",\"domain\":[0.0,0.2888888888888889]},\"yaxis4\":{\"anchor\":\"x4\",\"domain\":[0.36666666666666664,0.6333333333333333]},\"xaxis5\":{\"anchor\":\"y5\",\"domain\":[0.35555555555555557,0.6444444444444445]},\"yaxis5\":{\"anchor\":\"x5\",\"domain\":[0.36666666666666664,0.6333333333333333]},\"xaxis6\":{\"anchor\":\"y6\",\"domain\":[0.7111111111111111,1.0]},\"yaxis6\":{\"anchor\":\"x6\",\"domain\":[0.36666666666666664,0.6333333333333333]},\"xaxis7\":{\"anchor\":\"y7\",\"domain\":[0.0,0.2888888888888889]},\"yaxis7\":{\"anchor\":\"x7\",\"domain\":[0.0,0.26666666666666666]},\"xaxis8\":{\"anchor\":\"y8\",\"domain\":[0.35555555555555557,0.6444444444444445]},\"yaxis8\":{\"anchor\":\"x8\",\"domain\":[0.0,0.26666666666666666]},\"xaxis9\":{\"anchor\":\"y9\",\"domain\":[0.7111111111111111,1.0]},\"yaxis9\":{\"anchor\":\"x9\",\"domain\":[0.0,0.26666666666666666]}},                        {\"responsive\": true}                    ).then(function(){\n",
              "                            \n",
              "var gd = document.getElementById('31e6c71b-23b0-43fa-a533-a298eacc966e');\n",
              "var x = new MutationObserver(function (mutations, observer) {{\n",
              "        var display = window.getComputedStyle(gd).display;\n",
              "        if (!display || display === 'none') {{\n",
              "            console.log([gd, 'removed!']);\n",
              "            Plotly.purge(gd);\n",
              "            observer.disconnect();\n",
              "        }}\n",
              "}});\n",
              "\n",
              "// Listen for the removal of the full notebook cells\n",
              "var notebookContainer = gd.closest('#notebook-container');\n",
              "if (notebookContainer) {{\n",
              "    x.observe(notebookContainer, {childList: true});\n",
              "}}\n",
              "\n",
              "// Listen for the clearing of the current output cell\n",
              "var outputEl = gd.closest('.output');\n",
              "if (outputEl) {{\n",
              "    x.observe(outputEl, {childList: true});\n",
              "}}\n",
              "\n",
              "                        })                };                            </script>        </div>\n",
              "</body>\n",
              "</html>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "wWP1pw4k3NTY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Get the mean of each run and graph that\n",
        "\n",
        "ninetyRunsSummaryDF = ModelRunMetricsDF.groupby(['num_topics']).mean()\n",
        "ninetyRunsSummaryDF['num_topics'] = ninetyRunsSummaryDF.index\n",
        "\n",
        "fig = run_Topic_Metrics(ninetyRunsSummaryDF, metrics, rows=3)\n",
        "fig.show()\n"
      ],
      "metadata": {
        "id": "3UQ_o12_LZcT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Print out the knee for each metric\n",
        "\n",
        "for metric in metrics :\n",
        "  k1 = KneeLocator(ninetyRunsSummaryDF['num_topics'], ninetyRunsSummaryDF[metric], curve='concave' )\n",
        "  print(metric+':', k1.knee)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gP9TLtu34g-8",
        "outputId": "fa0e066d-4710-45e4-d638-5273ab202bcc"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "KLBac: 50\n",
            "KLDiv: 50\n",
            "KLUni: 80\n",
            "KLVac: 50\n",
            "JaccardSim: 5\n",
            "InvertedRBO: 35\n",
            "Topic_Diversity: 10\n",
            "NPMI: 20\n",
            "C_V: 20\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Graph the knee for each metric\n",
        "for metric in metrics :\n",
        "  plotKnee(ninetyRunsSummaryDF['num_topics'], ninetyRunsSummaryDF[metric])"
      ],
      "metadata": {
        "id": "GUmTwYF0VmvK"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Generate histogram showing the number of times each topic is dominant for \n",
        "#   a document per the article.\n",
        "\n",
        "fig = make_subplots(rows=2, cols=3)\n",
        "traces = []\n",
        "xval = 1\n",
        "yval = 1\n",
        "for modelkey in ModelsToEval.keys() :\n",
        "  xvals = MM.topicIDDFs[modelkey][0]\n",
        "  traces.append(go.Histogram(x=xvals, name=modelkey))\n",
        "for index, trace in enumerate(traces) :\n",
        "  fig.append_trace(trace, yval, xval)\n",
        "  xval += 1\n",
        "  if xval > 3 :\n",
        "    xval = 1\n",
        "    yval += 1\n",
        "fig.update_layout(title='Histogram - Count of Dominant Topics')\n",
        "fig.update_xaxes(type='category', categoryorder='total descending')\n",
        "fig.show()\n"
      ],
      "metadata": {
        "id": "atXz9rDEPYjG"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Generate box plots showing the profile of each topic  \n",
        "#   per the article.\n",
        "\n",
        "fig = make_subplots(rows=6, cols=1)\n",
        "traces = []\n",
        "xval = 1\n",
        "yval = 1\n",
        "for modelname in ModelsToEval.keys() :\n",
        "  pctevalDF = MM.pctContribDFs[modelname]\n",
        "  topicIDDF = MM.topicIDDFs[modelname]\n",
        "  fig.append_trace(go.Box(y=pctevalDF[0], x=topicIDDF[0], name=modelname), yval, xval)\n",
        "  yval += 1\n",
        "\n",
        "fig.update_layout(width=1000, height=1800)\n",
        "fig.update_yaxes(range=[0.0, 1.1], tickvals=[0, .25, .5, .75, 1])\n",
        "\n",
        "fig.show()\n"
      ],
      "metadata": {
        "id": "LUPlS64EQMTZ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Print out the topics for the Five and Ten models\n",
        "MM.PrintTopicsCount(['Five', 'Ten'])"
      ],
      "metadata": {
        "id": "iJza16erW_o0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Print sample text for the Five and Ten models\n",
        "MM.PrintSampleText(['Five', 'Ten'], NewsDF)"
      ],
      "metadata": {
        "id": "ZizNhNAGPtOo"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}