copycat-spark/src/main/jupyter/copycat-on-robust04.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CopyCat: Example Usage on Robust04 Runs\n",
    "\n",
    "This notebook assumes that you have a jupyter notebook with CopyCat installed.\n",
    "To start a jupyter notebook with CopyCat installed and your local directory mounted, run:\n",
    "```\n",
    "docker run --rm -ti -v ${PWD}:/home/jovyan -p 8888:8888 webis/chatnoir-copycat:1.0-jupyter\n",
    "```\n",
    "\n",
    "\n",
    "Then, we use the CopyCat cli to deduplicate run files submitted to Robust04."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Download the Robust04 anserini index\n",
    "\n",
    "The following code downloads the Robust04 index for Anserini that we use for the deduplication analysis on the robust04 corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Robust04 index is already downloaded.\n"
     ]
    }
   ],
   "source": [
    "# Download the robust04 anserini index\n",
    "import os.path\n",
    "ROBUST_INDEX='index-robust04-20191213/'\n",
    "\n",
    "if not os.path.isfile('robust04-index.tar.gz') or not os.path.isdir(ROBUST_INDEX):\n",
    "     #cf.: https://github.com/castorini/pyserini/blob/7e03ecaef7514fa53e491786a073eabfff373a74/pyserini/prebuilt_index_info.py#L6\n",
    "    !wget -O robust04-index.tar.gz 'https://git.uwaterloo.ca/jimmylin/anserini-indexes/raw/master/index-robust04-20191213.tar.gz'\n",
    "    !tar -xvzf robust04-index.tar.gz\n",
    "else:\n",
    "    print('Robust04 index is already downloaded.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 2: Verify the Download; Double Check the Document Preprocessing\n",
    "\n",
    "We have double-checked the preprocessing for the ClueWebs and CommonCrawls by many unit and integration tests.\n",
    "Since the Robust04 dataset is not included in these tests, we have to double-check the preprocessing using some examples. (Please [see here](https://github.com/chatnoir-eu/copycat/blob/master/copycat-modules/document-preprocessing/README.md) for an overview of preprocessing options.)\n",
    "\n",
    "To verify the preprocessing, we check a few documents manually."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bfn articl samuel blixen excerpt approxim month ago interior minist instal anti mafia unit try through intellig work break up connect between polic offic uruguayan crimin organ organ ar part intern mafia network dedic white slave trade drug traffick classifi inform handl interior ministri reveal mafia infiltr variou domest circl ha affect alarm number middl high rank polic offic ar numer indic organ dedic white slave trade drug traffick have develop support structur neighborhood level sport recreat organ mafia have enrol onli polic offic polic congress judici document indic mafia have also ensur complic sport entrepreneur profess sportsmen peopl entertain busi report even includ name deputi evid degr develop impun crimin organ intern connect ha alarm peopl highest level believ mafia ar try creat social support base like those brazil bolivia colombia countri drug traffick control entir neighborhood well entir line product activ nation mafia combin violenc coercion corrupt protect impun system ha been charact shameless thi respect total ineffici polic organ respon repress white slave trade try stem constant export prostitut consid kei factor grow power garner uruguayan mafia some legisl think absolut impun enjoi pimp dealer fiolo intermediari contact organ manag intern crimin trade constitut threat institut power creation anti mafia unit consequ fear concern express polit circl over increa level corrupt provok organ consid intern econom power group deriv from crimin activ consid intern scourg inform collect dure investig some concret case ha confirm uruguai interact between drug traffick white slave trade contraband brecha learn anti mafia unit ha begun process inform from variou sourc ha launch investig origin capit permit some individu enjoi lifestyl accord offici incom passag omit war mafia offici inabl act reiter classifi report need face power mafia ha forc human right committ take initi coordin effort thi area begin march wa estim situat wa extrem seriou out control committ member held sever meet interior minist raul iturria interior under secretari juan lui storac demand effect offen against crimin structur polic protect thei enjoi deci creat anti mafia unit wa result close door meet wa tacit agreement between ministri committ hinder investig matter who involv unit made up three high rank offici whose background have assur satisfi legisl unit directli subordin interior minist high rank offici oper administr field second respon intern relat third move field polic intellig brecha omit name offici name individu suspect involv illeg activ wit those who denounc illeg act obviou reason so hinder investig under wai first wa said govern deci might impli action similar clean hand oper now under wai itali wa allegedli initi idea minist iturria decid keep low profil minist told brecha despit exist element implic polic offic mafia difficult obtain proof through administr investig besid minist intend act cautiou order affect entir polic institut which indispen ani administr offici minist iturria prefer face offen against organ crime reorgan place investig repress so call intern crime under hi portfolio jurisdict express reorgan deci move intellig narcot interpol depart from under jurisdict montevideo polic depart other offici think interior ministri abl destroi mafia organ without depth attack against polic corrupt otherwi those involv white slave trade drug traffick continu act impun deputi told brecha deputi said he wa wait reason time result befor propo other measur\r\n"
     ]
    }
   ],
   "source": [
    "# Use CopyCat to check the preprocessing for a single document\n",
    "\n",
    "!copy-cat \\\n",
    "    --retrieveDocId FBIS4-56982 \\\n",
    "    --documents AnseriniIndex \\\n",
    "    --anseriniIndex $ROBUST_INDEX \\\n",
    "    --keepStopwords True \\\n",
    "    --output a --input a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The preprocessed Robust04 document FBIS4-56982 looks good (stemming as expected, no problems with encoding, etc.)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Deduplicate all run files submitted to Robust04 Tracks\n",
    "\n",
    "Now that we have double-checked that the document preprocessing works as expected for Robust04, we can deduplicate the run files and inspect the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The specified output 'copycat-on-robust-data/trec13-robust-run-files-deduplicated/run-apl04rsDw-deduplication-top10.jsonl' exists.\n",
      "Skip...\n",
      "Process topic 301\n",
      "Process topic 302\n",
      "Process topic 303\n",
      "Process topic 304\n",
      "Process topic 305\n",
      "Process topic 306\n",
      "Process topic 307\n",
      "Process topic 308\n",
      "Process topic 309\n",
      "Process topic 310\n",
      "Process topic 311\n",
      "Process topic 312\n",
      "Process topic 313\n",
      "Process topic 314\n",
      "Process topic 315\n",
      "Process topic 316\n",
      "Process topic 317\n",
      "Process topic 318\n",
      "Process topic 319\n",
      "Process topic 320\n",
      "Process topic 321\n",
      "Process topic 322\n",
      "Process topic 323\n",
      "Process topic 324\n",
      "Process topic 325\n",
      "Process topic 326\n",
      "Process topic 327\n",
      "Process topic 328\n",
      "Process topic 329\n",
      "Process topic 330\n",
      "Process topic 331\n",
      "Process topic 332\n",
      "Process topic 333\n",
      "Process topic 334\n",
      "Process topic 335\n",
      "Process topic 336\n",
      "Process topic 337\n",
      "Process topic 338\n",
      "Process topic 339\n",
      "Process topic 340\n",
      "Process topic 341\n",
      "Process topic 342\n",
      "Process topic 343\n",
      "Process topic 344\n",
      "Process topic 345\n",
      "Process topic 346\n",
      "Process topic 347\n",
      "Process topic 348\n",
      "Process topic 349\n",
      "Process topic 350\n",
      "Process topic 351\n",
      "Process topic 352\n",
      "Process topic 353\n",
      "Process topic 354\n",
      "Process topic 355\n",
      "Process topic 356\n",
      "Process topic 357\n",
      "Process topic 358\n",
      "Process topic 359\n",
      "Process topic 360\n",
      "Process topic 361\n",
      "Process topic 362\n",
      "Process topic 363\n",
      "Process topic 364\n",
      "Process topic 365\n",
      "Process topic 366\n",
      "Process topic 367\n",
      "Process topic 368\n",
      "Process topic 369\n",
      "Process topic 370\n",
      "Process topic 371\n",
      "Process topic 372\n",
      "Process topic 373\n",
      "Process topic 374\n",
      "Process topic 375\n",
      "Process topic 376\n",
      "Process topic 377\n",
      "Process topic 378\n",
      "Process topic 379\n",
      "Process topic 380\n",
      "Process topic 381\n",
      "Process topic 382\n",
      "Process topic 383\n",
      "Process topic 384\n",
      "Process topic 385\n",
      "Process topic 386\n",
      "Process topic 387\n",
      "Process topic 388\n",
      "Process topic 389\n",
      "Process topic 390\n",
      "Process topic 391\n",
      "Process topic 392\n",
      "Process topic 393\n",
      "Process topic 394\n",
      "Process topic 395\n",
      "Process topic 396\n",
      "Process topic 397\n",
      "Process topic 398\n",
      "Process topic 399\n",
      "Process topic 400\n",
      "Process topic 401\n",
      "Process topic 402\n",
      "Process topic 403\n",
      "Process topic 404\n",
      "Process topic 405\n",
      "Process topic 406\n",
      "Process topic 407\n",
      "Process topic 408\n",
      "Process topic 409\n",
      "Process topic 410\n",
      "Process topic 411\n",
      "Process topic 412\n",
      "Process topic 413\n",
      "Process topic 414\n",
      "Process topic 415\n",
      "Process topic 416\n",
      "Process topic 417\n",
      "Process topic 418\n",
      "Process topic 419\n",
      "Process topic 420\n",
      "Process topic 421\n",
      "Process topic 422\n",
      "Process topic 423\n",
      "Process topic 424\n",
      "Process topic 425\n",
      "Process topic 426\n",
      "Process topic 427\n",
      "Process topic 428\n",
      "Process topic 429\n",
      "Process topic 430\n",
      "Process topic 431\n",
      "Process topic 432\n",
      "Process topic 433\n",
      "Process topic 434\n",
      "Process topic 435\n",
      "Process topic 436\n",
      "Process topic 437\n",
      "Process topic 438\n",
      "Process topic 439\n",
      "Process topic 440\n",
      "Process topic 441\n",
      "Process topic 442\n",
      "Process topic 443\n",
      "Process topic 444\n",
      "Process topic 445\n",
      "Process topic 446\n",
      "Process topic 447\n",
      "Process topic 448\n",
      "Process topic 449\n",
      "Process topic 450\n",
      "Process topic 601\n",
      "Process topic 602\n",
      "Process topic 603\n",
      "Process topic 604\n",
      "Process topic 605\n",
      "Process topic 606\n",
      "Process topic 607\n",
      "Process topic 608\n",
      "Process topic 609\n",
      "Process topic 610\n",
      "Process topic 611\n",
      "Process topic 612\n",
      "Process topic 613\n",
      "Process topic 614\n",
      "Process topic 615\n",
      "Process topic 616\n",
      "Process topic 617\n",
      "Process topic 618\n",
      "Process topic 619\n",
      "Process topic 620\n",
      "Process topic 621\n",
      "Process topic 622\n",
      "Process topic 623\n",
      "Process topic 624\n",
      "Process topic 625\n",
      "Process topic 626\n",
      "Process topic 627\n",
      "Process topic 628\n",
      "Process topic 629\n",
      "Process topic 630\n",
      "Process topic 631\n",
      "Process topic 632\n",
      "Process topic 633\n",
      "Process topic 634\n",
      "Process topic 635\n",
      "Process topic 636\n",
      "Process topic 637\n",
      "Process topic 638\n",
      "Process topic 639\n",
      "Process topic 640\n",
      "Process topic 641\n",
      "Process topic 642\n",
      "Process topic 643\n",
      "Process topic 644\n",
      "Process topic 645\n",
      "Process topic 646\n",
      "Process topic 647\n",
      "Process topic 648\n",
      "Process topic 649\n",
      "Process topic 650\n",
      "Process topic 651\n",
      "Process topic 652\n",
      "Process topic 653\n",
      "Process topic 654\n",
      "Process topic 655\n",
      "Process topic 656\n",
      "Process topic 657\n",
      "Process topic 658\n",
      "Process topic 659\n",
      "Process topic 660\n",
      "Process topic 661\n",
      "Process topic 662\n",
      "Process topic 663\n",
      "Process topic 664\n",
      "Process topic 665\n",
      "Process topic 666\n",
      "Process topic 667\n",
      "Process topic 668\n",
      "Process topic 669\n",
      "Process topic 670\n",
      "Process topic 671\n",
      "Process topic 672\n",
      "Process topic 673\n",
      "Process topic 674\n",
      "Process topic 675\n",
      "Process topic 676\n",
      "Process topic 677\n",
      "Process topic 678\n",
      "Process topic 679\n",
      "Process topic 680\n",
      "Process topic 681\n",
      "Process topic 682\n",
      "Process topic 683\n",
      "Process topic 684\n",
      "Process topic 685\n",
      "Process topic 686\n",
      "Process topic 687\n",
      "Process topic 688\n",
      "Process topic 689\n",
      "Process topic 690\n",
      "Process topic 691\n",
      "Process topic 692\n",
      "Process topic 693\n",
      "Process topic 694\n",
      "Process topic 695\n",
      "Process topic 696\n",
      "Process topic 697\n",
      "Process topic 698\n",
      "Process topic 699\n",
      "Process topic 700\n",
      "Process topic 301\n",
      "Process topic 302\n",
      "Process topic 303\n",
      "Process topic 304\n",
      "Process topic 305\n",
      "Process topic 306\n",
      "Process topic 307\n",
      "Process topic 308\n",
      "Process topic 309\n",
      "Process topic 310\n",
      "Process topic 311\n",
      "Process topic 312\n",
      "Process topic 313\n",
      "Process topic 314\n",
      "Process topic 315\n",
      "Process topic 316\n",
      "Process topic 317\n",
      "Process topic 318\n",
      "Process topic 319\n",
      "Process topic 320\n",
      "Process topic 321\n",
      "Process topic 322\n",
      "Process topic 323\n",
      "Process topic 324\n",
      "Process topic 325\n",
      "Process topic 326\n",
      "Process topic 327\n",
      "Process topic 328\n",
      "Process topic 329\n",
      "Process topic 330\n",
      "Process topic 331\n",
      "Process topic 332\n",
      "Process topic 333\n",
      "Process topic 334\n",
      "Process topic 335\n",
      "Process topic 336\n",
      "Process topic 337\n",
      "Process topic 338\n",
      "Process topic 339\n",
      "Process topic 340\n",
      "Process topic 341\n",
      "Process topic 342\n",
      "Process topic 343\n",
      "Process topic 344\n",
      "Process topic 345\n",
      "Process topic 346\n",
      "Process topic 347\n",
      "Process topic 348\n",
      "Process topic 349\n",
      "Process topic 350\n",
      "Process topic 351\n",
      "Process topic 352\n",
      "Process topic 353\n",
      "Process topic 354\n",
      "Process topic 355\n",
      "Process topic 356\n",
      "Process topic 357\n",
      "Process topic 358\n",
      "Process topic 359\n",
      "Process topic 360\n",
      "Process topic 361\n",
      "Process topic 362\n",
      "Process topic 363\n",
      "Process topic 364\n",
      "Process topic 365\n",
      "Process topic 366\n",
      "Process topic 367\n",
      "Process topic 368\n",
      "Process topic 369\n",
      "Process topic 370\n",
      "Process topic 371\n",
      "Process topic 372\n",
      "Process topic 373\n",
      "Process topic 374\n",
      "Process topic 375\n",
      "Process topic 376\n",
      "Process topic 377\n",
      "Process topic 378\n",
      "Process topic 379\n",
      "Process topic 380\n",
      "Process topic 381\n",
      "Process topic 382\n",
      "Process topic 383\n",
      "Process topic 384\n",
      "Process topic 385\n",
      "Process topic 386\n",
      "Process topic 387\n",
      "Process topic 388\n",
      "Process topic 389\n",
      "Process topic 390\n",
      "Process topic 391\n",
      "Process topic 392\n",
      "Process topic 393\n",
      "Process topic 394\n",
      "Process topic 395\n",
      "Process topic 396\n",
      "Process topic 397\n",
      "Process topic 398\n",
      "Process topic 399\n",
      "Process topic 400\n",
      "Process topic 401\n",
      "Process topic 402\n",
      "Process topic 403\n",
      "Process topic 404\n",
      "Process topic 405\n",
      "Process topic 406\n",
      "Process topic 407\n",
      "Process topic 408\n",
      "Process topic 409\n",
      "Process topic 410\n",
      "Process topic 411\n",
      "Process topic 412\n",
      "Process topic 413\n",
      "Process topic 414\n",
      "Process topic 415\n",
      "Process topic 416\n",
      "Process topic 417\n",
      "Process topic 418\n",
      "Process topic 419\n",
      "Process topic 420\n",
      "Process topic 421\n",
      "Process topic 422\n",
      "Process topic 423\n",
      "Process topic 424\n",
      "Process topic 425\n",
      "Process topic 426\n",
      "Process topic 427\n",
      "Process topic 428\n",
      "Process topic 429\n",
      "Process topic 430\n",
      "Process topic 431\n",
      "Process topic 432\n",
      "Process topic 433\n",
      "Process topic 434\n",
      "Process topic 435\n",
      "Process topic 436\n",
      "Process topic 437\n",
      "Process topic 438\n",
      "Process topic 439\n",
      "Process topic 440\n",
      "Process topic 441\n",
      "Process topic 442\n",
      "Process topic 443\n",
      "Process topic 444\n",
      "Process topic 445\n",
      "Process topic 446\n",
      "Process topic 447\n",
      "Process topic 448\n",
      "Process topic 449\n",
      "Process topic 450\n",
      "Process topic 601\n",
      "Process topic 602\n",
      "Process topic 603\n",
      "Process topic 604\n",
      "Process topic 605\n",
      "Process topic 606\n",
      "Process topic 607\n",
      "Process topic 608\n",
      "Process topic 609\n",
      "Process topic 610\n",
      "Process topic 611\n",
      "Process topic 612\n",
      "Process topic 613\n",
      "Process topic 614\n",
      "Process topic 615\n",
      "Process topic 616\n",
      "Process topic 617\n",
      "Process topic 618\n",
      "Process topic 619\n",
      "Process topic 620\n",
      "Process topic 621\n",
      "Process topic 622\n",
      "Process topic 623\n",
      "Process topic 624\n",
      "Process topic 625\n",
      "Process topic 626\n",
      "Process topic 627\n",
      "Process topic 628\n",
      "Process topic 629\n",
      "Process topic 630\n",
      "Process topic 631\n",
      "Process topic 632\n",
      "Process topic 633\n",
      "Process topic 634\n",
      "Process topic 635\n",
      "Process topic 636\n",
      "Process topic 637\n",
      "Process topic 638\n",
      "Process topic 639\n",
      "Process topic 640\n",
      "Process topic 641\n",
      "Process topic 642\n",
      "Process topic 643\n",
      "Process topic 644\n",
      "Process topic 645\n",
      "Process topic 646\n",
      "Process topic 647\n",
      "Process topic 648\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Process topic 649\n",
      "Process topic 650\n",
      "Process topic 651\n",
      "Process topic 652\n",
      "Process topic 653\n",
      "Process topic 654\n",
      "Process topic 655\n",
      "Process topic 656\n",
      "Process topic 657\n",
      "Process topic 658\n",
      "Process topic 659\n",
      "Process topic 660\n",
      "Process topic 661\n",
      "Process topic 662\n",
      "Process topic 663\n",
      "Process topic 664\n",
      "Process topic 665\n",
      "Process topic 666\n",
      "Process topic 667\n",
      "Process topic 668\n",
      "Process topic 669\n",
      "Process topic 670\n",
      "Process topic 671\n",
      "Process topic 672\n",
      "Process topic 673\n",
      "Process topic 674\n",
      "Process topic 675\n",
      "Process topic 676\n",
      "Process topic 677\n",
      "Process topic 678\n",
      "Process topic 679\n",
      "Process topic 680\n",
      "Process topic 681\n",
      "Process topic 682\n",
      "Process topic 683\n",
      "Process topic 684\n",
      "Process topic 685\n",
      "Process topic 686\n",
      "Process topic 687\n",
      "Process topic 688\n",
      "Process topic 689\n",
      "Process topic 690\n",
      "Process topic 691\n",
      "Process topic 692\n",
      "Process topic 693\n",
      "Process topic 694\n",
      "Process topic 695\n",
      "Process topic 696\n",
      "Process topic 697\n",
      "Process topic 698\n",
      "Process topic 699\n",
      "Process topic 700\n",
      "Process topic 301\n",
      "Process topic 302\n",
      "Process topic 303\n",
      "Process topic 304\n",
      "Process topic 305\n",
      "Process topic 306\n",
      "Process topic 307\n",
      "Process topic 308\n",
      "Process topic 309\n",
      "Process topic 310\n",
      "Process topic 311\n",
      "Process topic 312\n",
      "Process topic 313\n",
      "Process topic 314\n",
      "Process topic 315\n",
      "Process topic 316\n",
      "Process topic 317\n",
      "Process topic 318\n",
      "Process topic 319\n",
      "Process topic 320\n",
      "Process topic 321\n",
      "Process topic 322\n",
      "Process topic 323\n",
      "Process topic 324\n",
      "Process topic 325\n",
      "Process topic 326\n",
      "Process topic 327\n",
      "Process topic 328\n",
      "Process topic 329\n",
      "Process topic 330\n",
      "Process topic 331\n",
      "Process topic 332\n",
      "Process topic 333\n",
      "Process topic 334\n",
      "Process topic 335\n",
      "Process topic 336\n",
      "Process topic 337\n",
      "Process topic 338\n",
      "Process topic 339\n",
      "Process topic 340\n",
      "Process topic 341\n",
      "Process topic 342\n",
      "Process topic 343\n",
      "Process topic 344\n",
      "Process topic 345\n",
      "Process topic 346\n",
      "Process topic 347\n",
      "Process topic 348\n",
      "Process topic 349\n",
      "Process topic 350\n",
      "Process topic 351\n",
      "Process topic 352\n",
      "Process topic 353\n",
      "Process topic 354\n",
      "Process topic 355\n",
      "Process topic 356\n",
      "Process topic 357\n",
      "Process topic 358\n",
      "Process topic 359\n",
      "Process topic 360\n",
      "Process topic 361\n",
      "Process topic 362\n",
      "Process topic 363\n",
      "Process topic 364\n",
      "Process topic 365\n",
      "Process topic 366\n",
      "Process topic 367\n",
      "Process topic 368\n",
      "Process topic 369\n",
      "Process topic 370\n",
      "Process topic 371\n",
      "Process topic 372\n",
      "Process topic 373\n",
      "Process topic 374\n",
      "Process topic 375\n",
      "Process topic 376\n",
      "Process topic 377\n",
      "Process topic 378\n",
      "Process topic 379\n",
      "Process topic 380\n",
      "Process topic 381\n",
      "Process topic 382\n",
      "Process topic 383\n",
      "Process topic 384\n",
      "Process topic 385\n",
      "Process topic 386\n",
      "Process topic 387\n",
      "Process topic 388\n",
      "Process topic 389\n",
      "Process topic 390\n",
      "Process topic 391\n",
      "Process topic 392\n",
      "Process topic 393\n",
      "Process topic 394\n",
      "Process topic 395\n",
      "Process topic 396\n",
      "Process topic 397\n",
      "Process topic 398\n",
      "Process topic 399\n",
      "Process topic 400\n",
      "Process topic 401\n",
      "Process topic 402\n",
      "Process topic 403\n",
      "Process topic 404\n",
      "Process topic 405\n",
      "Process topic 406\n",
      "Process topic 407\n",
      "Process topic 408\n",
      "Process topic 409\n",
      "Process topic 410\n",
      "Process topic 411\n",
      "Process topic 412\n",
      "Process topic 413\n",
      "Process topic 414\n",
      "Process topic 415\n",
      "Process topic 416\n",
      "Process topic 417\n",
      "Process topic 418\n",
      "Process topic 419\n",
      "Process topic 420\n",
      "Process topic 421\n",
      "Process topic 422\n",
      "Process topic 423\n",
      "Process topic 424\n",
      "Process topic 425\n",
      "Process topic 426\n",
      "Process topic 427\n",
      "Process topic 428\n",
      "Process topic 429\n",
      "Process topic 430\n",
      "Process topic 431\n",
      "Process topic 432\n",
      "Process topic 433\n",
      "Process topic 434\n",
      "Process topic 435\n",
      "Process topic 436\n",
      "Process topic 437\n",
      "Process topic 438\n",
      "Process topic 439\n",
      "Process topic 440\n",
      "Process topic 441\n",
      "Process topic 442\n",
      "Process topic 443\n",
      "Process topic 444\n",
      "Process topic 445\n",
      "Process topic 446\n",
      "Process topic 447\n",
      "Process topic 448\n",
      "Process topic 449\n",
      "Process topic 450\n",
      "Process topic 601\n",
      "Process topic 602\n",
      "Process topic 603\n",
      "Process topic 604\n",
      "Process topic 605\n",
      "Process topic 606\n",
      "Process topic 607\n",
      "Process topic 608\n",
      "Process topic 609\n",
      "Process topic 610\n",
      "Process topic 611\n",
      "Process topic 612\n",
      "Process topic 613\n",
      "Process topic 614\n",
      "Process topic 615\n",
      "Process topic 616\n",
      "Process topic 617\n",
      "Process topic 618\n",
      "Process topic 619\n",
      "Process topic 620\n",
      "Process topic 621\n",
      "Process topic 622\n",
      "Process topic 623\n",
      "Process topic 624\n",
      "Process topic 625\n",
      "Process topic 626\n",
      "Process topic 627\n",
      "Process topic 628\n",
      "Process topic 629\n",
      "Process topic 630\n",
      "Process topic 631\n",
      "Process topic 632\n",
      "Process topic 633\n",
      "Process topic 634\n",
      "Process topic 635\n",
      "Process topic 636\n",
      "Process topic 637\n",
      "Process topic 638\n",
      "Process topic 639\n",
      "Process topic 640\n",
      "Process topic 641\n",
      "Process topic 642\n",
      "Process topic 643\n",
      "Process topic 644\n",
      "Process topic 645\n",
      "Process topic 646\n",
      "Process topic 647\n",
      "Process topic 648\n",
      "Process topic 649\n",
      "Process topic 650\n",
      "Process topic 651\n",
      "Process topic 652\n",
      "Process topic 653\n",
      "Process topic 654\n",
      "Process topic 655\n",
      "Process topic 656\n",
      "Process topic 657\n",
      "Process topic 658\n",
      "Process topic 659\n",
      "Process topic 660\n",
      "Process topic 661\n",
      "Process topic 662\n",
      "Process topic 663\n",
      "Process topic 664\n",
      "Process topic 665\n",
      "Process topic 666\n",
      "Process topic 667\n",
      "Process topic 668\n",
      "Process topic 669\n",
      "Process topic 670\n",
      "Process topic 671\n",
      "Process topic 672\n",
      "Process topic 673\n",
      "Process topic 674\n",
      "Process topic 675\n",
      "Process topic 676\n",
      "Process topic 677\n",
      "Process topic 678\n",
      "Process topic 679\n",
      "Process topic 680\n",
      "Process topic 681\n",
      "Process topic 682\n",
      "Process topic 683\n",
      "Process topic 684\n",
      "Process topic 685\n",
      "Process topic 686\n",
      "Process topic 687\n",
      "Process topic 688\n",
      "Process topic 689\n",
      "Process topic 690\n",
      "Process topic 691\n",
      "Process topic 692\n",
      "Process topic 693\n",
      "Process topic 694\n",
      "Process topic 695\n",
      "Process topic 696\n",
      "Process topic 697\n",
      "Process topic 698\n",
      "Process topic 699\n",
      "Process topic 700\n",
      "Process topic 301\n",
      "Process topic 302\n",
      "Process topic 303\n",
      "Process topic 304\n",
      "Process topic 305\n",
      "Process topic 306\n",
      "Process topic 307\n",
      "Process topic 308\n",
      "Process topic 309\n",
      "Process topic 310\n",
      "Process topic 311\n",
      "Process topic 312\n",
      "Process topic 313\n",
      "Process topic 314\n",
      "Process topic 315\n",
      "Process topic 316\n",
      "Process topic 317\n",
      "Process topic 318\n",
      "Process topic 319\n",
      "Process topic 320\n",
      "Process topic 321\n",
      "Process topic 322\n",
      "Process topic 323\n",
      "Process topic 324\n",
      "Process topic 325\n",
      "Process topic 326\n",
      "Process topic 327\n",
      "Process topic 328\n",
      "Process topic 329\n",
      "Process topic 330\n",
      "Process topic 331\n",
      "Process topic 332\n",
      "Process topic 333\n",
      "Process topic 334\n",
      "Process topic 335\n",
      "Process topic 336\n",
      "Process topic 337\n",
      "Process topic 338\n",
      "Process topic 339\n",
      "Process topic 340\n",
      "Process topic 341\n",
      "Process topic 342\n",
      "Process topic 343\n",
      "Process topic 344\n",
      "Process topic 345\n",
      "Process topic 346\n",
      "Process topic 347\n",
      "Process topic 348\n",
      "Process topic 349\n",
      "Process topic 350\n",
      "Process topic 351\n",
      "Process topic 352\n",
      "Process topic 353\n",
      "Process topic 354\n",
      "Process topic 355\n",
      "Process topic 356\n",
      "Process topic 357\n",
      "Process topic 358\n",
      "Process topic 359\n",
      "Process topic 360\n",
      "Process topic 361\n",
      "Process topic 362\n",
      "Process topic 363\n",
      "Process topic 364\n",
      "Process topic 365\n",
      "Process topic 366\n",
      "Process topic 367\n",
      "Process topic 368\n",
      "Process topic 369\n",
      "Process topic 370\n",
      "Process topic 371\n",
      "Process topic 372\n",
      "Process topic 373\n",
      "Process topic 374\n",
      "Process topic 375\n",
      "Process topic 376\n",
      "Process topic 377\n",
      "Process topic 378\n",
      "Process topic 379\n",
      "Process topic 380\n",
      "Process topic 381\n",
      "Process topic 382\n",
      "Process topic 383\n",
      "Process topic 384\n",
      "Process topic 385\n",
      "Process topic 386\n",
      "Process topic 387\n",
      "Process topic 388\n",
      "Process topic 389\n",
      "Process topic 390\n",
      "Process topic 391\n",
      "Process topic 392\n",
      "Process topic 393\n",
      "Process topic 394\n",
      "Process topic 395\n",
      "Process topic 396\n",
      "Process topic 397\n",
      "Process topic 398\n",
      "Process topic 399\n",
      "Process topic 400\n",
      "Process topic 401\n",
      "Process topic 402\n",
      "Process topic 403\n",
      "Process topic 404\n",
      "Process topic 405\n",
      "Process topic 406\n",
      "Process topic 407\n",
      "Process topic 408\n",
      "Process topic 409\n",
      "Process topic 410\n",
      "Process topic 411\n",
      "Process topic 412\n",
      "Process topic 413\n",
      "Process topic 414\n",
      "Process topic 415\n",
      "Process topic 416\n",
      "Process topic 417\n",
      "Process topic 418\n",
      "Process topic 419\n",
      "Process topic 420\n",
      "Process topic 421\n",
      "Process topic 422\n",
      "Process topic 423\n",
      "Process topic 424\n",
      "Process topic 425\n",
      "Process topic 426\n",
      "Process topic 427\n",
      "Process topic 428\n",
      "Process topic 429\n",
      "Process topic 430\n",
      "Process topic 431\n",
      "Process topic 432\n",
      "Process topic 433\n",
      "Process topic 434\n",
      "Process topic 435\n",
      "Process topic 436\n",
      "Process topic 437\n",
      "Process topic 438\n",
      "Process topic 439\n",
      "Process topic 440\n",
      "Process topic 441\n",
      "Process topic 442\n"
     ]
    }
   ],
   "source": [
    "# this helper function executes copycat on the passed run file with the double-checked document preprocessing\n",
    "def deduplicate_run_file(run_name, ranks):\n",
    "    input_file = 'copycat-on-robust-data/trec13-robust-run-files/input.' + run_name + '.gz'\n",
    "    output_file = 'copycat-on-robust-data/trec13-robust-run-files-deduplicated/run-' + run_name + '-deduplication-top' + str(ranks) + '.jsonl'\n",
    "    \n",
    "    !copy-cat \\\n",
    "        --output $output_file \\\n",
    "        --input  $input_file \\\n",
    "        --similarities \"s3\" \\\n",
    "        --s3Threshold 0.8 \\\n",
    "        --threads 10 \\\n",
    "        --ranks $ranks \\\n",
    "        --documents AnseriniIndex \\\n",
    "        --keepStopwords True \\\n",
    "        --anseriniIndex $ROBUST_INDEX\n",
    "\n",
    "# Preprocess all runs\n",
    "RUNS = !ls copycat-on-robust-data/trec13-robust-run-files/ |grep input|grep gz\n",
    "for run in RUNS:\n",
    "    for depth in [10]:\n",
    "        deduplicate_run_file(run.replace('input.', '').replace('.gz', ''), depth)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "RUNS = !ls copycat-on-robust-data/trec13-robust-run-files/ |grep input|grep gz\n",
    "for run in RUNS:\n",
    "    for depth in [100]:\n",
    "        deduplicate_run_file(run.replace('input.', '').replace('.gz', ''), depth)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import pandas as pd\n",
    "\n",
    "def eval_with_threshold(threshold, run_file_name):\n",
    "    rows = []\n",
    "    with open(run_file_name) as jsonl_file:\n",
    "        for jsonl in jsonl_file:\n",
    "            dedup_data = json.loads(jsonl)\n",
    "            docs_to_remove = []\n",
    "            \n",
    "            for sim in dedup_data['similarities']:\n",
    "                if sim['similarities']['s3'] >=  threshold:\n",
    "                    docs_to_remove += [sim['secondId']]\n",
    "\n",
    "                \n",
    "            rows += [{\n",
    "                    'topic': dedup_data['topic'],\n",
    "                    'duplicates': len(set(docs_to_remove)),\n",
    "                    'docs': dedup_data['docs'],\n",
    "                }]\n",
    "\n",
    "    return rows\n",
    "\n",
    "def eval_runs_with_threshold(threshold, run_files):\n",
    "    rows = []\n",
    "    for r in run_files:\n",
    "        rows += eval_with_threshold(threshold, r)\n",
    "    \n",
    "    return pd.DataFrame(rows)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "DEDUP_TARGET_DIR='copycat-on-robust-data/trec13-robust-run-files-deduplicated/'\n",
    "ALL_DIRS=!ls $DEDUP_TARGET_DIR\n",
    "ALL_DIRS = [DEDUP_TARGET_DIR + i for i in ALL_DIRS if '.jsonl' in i]\n",
    "\n",
    "df = eval_runs_with_threshold(0.82, ALL_DIRS)\n",
    "df['redundancy'] = df['duplicates']/df['docs']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>redundancy</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>docs</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>101</th>\n",
       "      <td>0.018693</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1000</th>\n",
       "      <td>0.020308</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      redundancy\n",
       "docs            \n",
       "101     0.018693\n",
       "1000    0.020308"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[['docs', 'redundancy']].groupby('docs').mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<seaborn.axisgrid.FacetGrid at 0x7fe4803def10>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAFuCAYAAAChovKPAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAycUlEQVR4nO3deXxcdb3/8ddnsjRbkzRNmq4p3WlLS4HQArJboIqACEJFLnjhiiKIAioov2uFC173iwrKoiAILiwWCyKVfZNCQyld6UqXpFvapOmSZpv5/v440yaTTNJJm8nkTN7PxyOPzPme75l8Auk7J99zzvdrzjlERMQ/AokuQEREOkfBLSLiMwpuERGfUXCLiPiMgltExGdSE11AV5kxY4Z74YUXEl2GiEhXsmiNSXPGvX379kSXICLSLZImuEVEegsFt4iIzyi4RUR8RsEtIuIzCm4REZ9RcIuI+IyCW0TEZ+Ia3GY2w8xWmNlqM7s1yv6bzGyZmS0ys5fNbHiLfUEzWxj+mBPPOkVE/CRuT06aWQpwL3AWUA7MN7M5zrllLbp9AJQ652rN7FrgJ8Cl4X37nHNT4lWfiIhfxfOMeyqw2jm31jnXAPwFuKBlB+fcq8652vDmPGBoHOsREUkK8ZyrZAiwscV2OTCtg/5XA/9ssZ1hZmVAE/Aj59wzrQ8ws2uAawBKSkoOt96ksXLrbl5YsoWqvfUMzs/k3MmDGZKfmeiyRKSL9IhJpszscqAUOK1F83DnXIWZjQReMbPFzrk1LY9zzj0APABQWlqqNdiA11dWctXD7xFs8V/jVy+v5m9fO4mxxX0TV5iIdJl4DpVUAMNabA8Nt0Uws+nAbcD5zrn6/e3OuYrw57XAa8Axcaw1aTz4xtqI0AbYU9/EH99Zn5iCRKTLxTO45wNjzGyEmaUDM4GIu0PM7BjgfrzQ3taivZ+Z9Qm/LgQ+AbS8qCntcET/w6O9dhHxn7gFt3OuCbgemAssB55wzi01szvM7Pxwt58COcCTrW77Gw+UmdmHwKt4Y9wK7hj818kjCbSawTcrPYXLTxge/QAR8R1zLjnOxEpLS11ZWVmiy+gRlm/exQtLNrN9TwND+mVy3uTBDCvISnRZItJ5URdS6BEXJ6VrjR+Uy/hBuYkuQ0TiRI+8i4j4jIJbRMRnFNwiIj6j4BYR8RkFt4iIzyi4RUR8RsEtIuIzCm4REZ9RcIuI+IyCW0TEZxTcIiI+o+AWEfEZBbeIiM8ouEVEfEbBLSLiMwpuERGfUXCLiPiMgltExGcU3CIiPqPgFhHxGQW3iIjPKLhFRHxGwS0i4jMKbhERn1Fwi4j4jIJbRMRnFNwiIj6j4BYR8RkFt4iIzyi4RUR8RsEtIuIzCm4REZ9RcIuI+IyCW0TEZxTcIiI+o+AWEfEZBbeIiM8ouEVEfEbBLSLiMwpuERGfUXCLiPiMgltExGcU3CIiPqPgFhHxGQW3iIjPKLhFRHxGwS0i4jMKbhERn1Fwi4j4TFyD28xmmNkKM1ttZrdG2X+TmS0zs0Vm9rKZDW+x70ozWxX+uDKedYqI+EncgtvMUoB7gU8BE4AvmNmEVt0+AEqdc5OBp4CfhI8tAGYB04CpwCwz6xevWkVE/CSeZ9xTgdXOubXOuQbgL8AFLTs45151ztWGN+cBQ8OvzwFedM5VOeeqgReBGXGsVUTEN+IZ3EOAjS22y8Nt7bka+GdnjjWza8yszMzKKisrD7NcERF/6BEXJ83scqAU+GlnjnPOPeCcK3XOlRYVFcWnOBGRHiaewV0BDGuxPTTcFsHMpgO3Aec75+o7c6yISG8Uz+CeD4wxsxFmlg7MBOa07GBmxwD344X2tha75gJnm1m/8EXJs8NtIiK9Xmq83tg512Rm1+MFbgrwkHNuqZndAZQ55+bgDY3kAE+aGcAG59z5zrkqM/sfvPAHuMM5VxWvWkVE/MScc4muoUuUlpa6srKyRJchItKVLFpjj7g4KSIisVNwi4j4jIJbRMRnFNwiIj6j4BYR8RkFt7B9Tz1zPtzEsk27ItqDIcfrKyt5eflWGppCCapORFqL233c4g/z11Vxxe/fY19jEIAbzhzNTWePo74pyKX3z2Phxp0AjBmQw9NfO4ncjLQEVisioDPuXu9XL686ENoA972+lpraRl5YsuVAaAOs2raH2Qs064BIT6Dg7uV27WuM2G4IhtjXGGRXXVObvjWt+opIYii4e7mZU0sits8YV8TAvAw+ddRA8rOah0Wy0lO4YMrg7i5PRKLQGHcv94WpJfTPTufl5dsYPSCHy0/wVo8rzOnDnOtO5rF319MYDHHZ1BKG989OcLUiApqrRESkJ9NcJSIiyUDBLSLiMwpuERGfUXCLiPiMgltExGcU3CIiPqPgFhHxGQW3iIjPKLhFRHxGwS0i4jMKbhERn1Fwi4j4jIJbRMRnFNwiIj6j4BYR8RkFt4iIzyi4RUR8RkuXJanqvQ3c8vQiXltRydiBOfzvhZOZNDSvU++xetsebn16EYvKa5g2soCfXnw0A/My4lSxiMRKZ9xJ6q7nl/OvZVtpCIZYUrGLr/3pfUKhzi1Td+NfF1K2vpqGYIg3V23nttmL41StiHSGgjtJvb++OmJ7Y9U+tu6ui/n4+qYgiytqItrmr6vqktpE5PAouJPUsSX9IraH5GdS3Df2YY4+qSlMHJwb0Xbc8H7t9BaR7qTgTlK3nTue6eMHkBIwxg/K5TdfPJZAIOqC0e365cwpTBmWT0rAOGlUf+66cFKcqhWRzjDnOjfu2VOVlpa6srKyRJchItKVop5t6YxbRMRnFNwiIj6j4BYR8RkFt4iIzyi4RUR8RsEtIuIzCm4REZ9RcIuI+IyCW0TEZxTcIiI+o+AWEfEZBbeIiM8ouEVEfEbBLSLiMwpuERGfiWtwm9kMM1thZqvN7NYo+081swVm1mRmF7faFzSzheGPOfGsU0TET+K2yruZpQD3AmcB5cB8M5vjnFvWotsG4EvAt6K8xT7n3JR41Sci4ldxC25gKrDaObcWwMz+AlwAHAhu59y68L5QHOvolbbtquOxeevZVdfExccN5agheW36bNhRy+Pvrcc5mHn8MEYW5SSgUhHprHgG9xBgY4vtcmBaJ47PMLMyoAn4kXPumdYdzOwa4BqAkpKSQ680yexrCHLhb/5Nxc59APzp3Q387WsnRYT3tt11nH/vW+ysbQTgz+9u4IUbT2VIfmZCahaR2PXki5PDnXOlwGXA3WY2qnUH59wDzrlS51xpUVFR91fYQ72+ctuB0AZoCIZ46v3yiD7PL9p8ILQBdtc3MWfhpm6rUUQOXTyDuwIY1mJ7aLgtJs65ivDntcBrwDFdWVwyy+mT1qatb0bkH1c5GW375GTE8w8wEekq8Qzu+cAYMxthZunATCCmu0PMrJ+Z9Qm/LgQ+QYuxcenYSaP6c/LowgPbg/MyuPyE4RF9zp00iKOG5B7YHlucw4XHDOm2GkXk0Jlz7uCdzFKcc8FOv7nZp4G7gRTgIefcXWZ2B1DmnJtjZscDs4F+QB2wxTk30cxOAu4HQni/XO52zv2+o69VWlrqysrKOlti0gqFHG+v2c7uuiZOH1dEVnrbs+nGYIg3VlYScnDa2CLSU3vyyJlIr2RRG2MM7rXA08DDrW7n6zEU3CKShKIGd6ynWEcDK4Hfmdk8M7vGzHIPdpCIiHS9mILbObfbOfegc+4k4BZgFrDZzB4xs9FxrVBERCLEFNxmlmJm55vZbLwx658DI4FngefjV56IiLQW6/1fq4BXgZ865/7dov0pMzu168sSEZH2xBrck51ze6LtcM7d0IX1iIjIQcR6cfJeM8vfvxG+z/qh+JQkIiIdiTW4Jzvndu7fcM5VoycZRUQSItbgDphZv/0bZlZAfCeoEhGRdsQavj8H3jGzJ/FuCL8YuCtuVYmISLtiCm7n3KNm9j5wRrjpcz31CUoRkWTXmeGOj4Dq/ceYWYlzbkNcqhIRkXbFFNxm9nW8pyW3AkG84RIHTI5faSIiEk2sZ9zfAMY553bEsxgRSUJN9bBvJ/QtTnQlSSPW4N4I1MSzEBFJQoufgn/cDHU7YehUmPk45AxIdFW+F2twrwVeM7N/APX7G51zv4hLVSLif3W7YM4N0LjX2y5/D165E87/VWLrSgKxBveG8Ed6+ENEpGM71zeH9n7bliemliQT6+2At8e7EOk+jcEQSypqGNovi6K+fdrtt3LrbgIGowf07cbqJGkUjYe8YVCzsbltzFmJqyeJxHpXSRHwHWAikLG/3Tl3ZpzqkjhZvW0PV/z+XTbV1JGWYvy/cydw5UlHRPRpaApxzR/LeG1FJQDTxxdz3+XHkpqipc2kE1JS4bIn4KVZULUWxp8PJ9+Y6KqSQqz/Eh/Hu497BHA7sA5vMWDxmV+8uIJNNXUANAYdP3x+OTX7GiP6/GPxpgOhDfDS8q3MXbq1W+uUJFE8Ab74JHz9fZg+C1LSEl1RUog1uPuHF+ttdM697py7CtDZtg9V7KyL2K5vCrFjT31kn+p9UY6rjWtdIhK7WIN7/ynZZjM718yOAQriVJPE0XmTB0VsTxycy8iinIi2cyYOJC2leY3S9NQAZ08Y2C31icjBxXpXyZ1mlgfcDPwayAU0WOVDV588gvTUAP9aupWRRdlcf2bbJUPHFPflj1dP4+G3PyZgxtUnj+CIwuwEVCsi0ZhzLtE1dInS0lJXVlaW6DJERLqSRWvs8IzbzH6NNydJVFq2TESk+x1sjLsMeB/vFsBj8RYNXgVMQQ/iiEhX2boUVr0IjW0vjEtbHZ5xO+ceATCza4GTnXNN4e37gDfjX56IJL3nvwPv3e+97jsI/vN5KBiZ2Jp6uFjvKumHd0Fyv5xwm4jIoduxpjm0AXZvhrfuTlg5fhHrXSU/Aj4ws1fxBstPBX4Qr6JEpJeojTJT9N7t3V+Hz8Q6V8nDZvZPYFq46Rbn3Jb4lSXd5d21O3h1RSVjBuRwwZTBBx5r37Cjlr99UE5WegqfP24Y/bJ1SUPiYMhxUDgOtq9obpvyhcTV4xOdWbosBagMHzPWzMY6596IT1nSHWZ/UM6Nf/3wwPabqyq5e+YxfLx9L+f9+i321DcB8Ni8DbzwzVPISu/Mj4tIDAIp8KXn4J17YNcmmPR5GHtOoqvq8WKdZOrHwKXAUiAUbnaAgtvHHn57XcT23z/cxH9/ZgJPlm08ENoAG6pqeXn5Ns47enA3Vyi9Qs4AOOuORFfhK7GeQn0Wb+my+oN1FP9IbzXbX4oZqYEAaVFmAUxP1cyAIj1FrP8a1wKa1ivJXHfGaFIDzQ9mXXnSEeRlpfGFqSUMaDFP96QheZwxTstNifQUMT3ybmZPA0cDLxO5dFmPeXJSj7wfmo+37+WNlZWMKc7hpFGFB9prahuZu3QLmekpnDWhmIy0lARWKdJrdf6R9xbmhD8kyYwozGZElAmk8rLSuOT4YQmoSEQOJtbbAR+JdyEiIhKbWO8q+Zgok0055/RcqohIN4t1qKS0xesM4PNoIQURkYSI6a4S59yOFh8Vzrm7gXPjW5qIiEQT61DJsS02A3hn4HqMTkQkAWIN35+3eN2Et8r7JV1ejSTEKx9t5Z5XVlPfFOJLJx3B50uH8eAba3nq/XL656Rz89ljOW64RsYkBhXvwz9uhu2rYOwM+Mz/QUbuwY9rbdGT8OqdUL8bSq+CM24Di3pnXK+kpct6uXXb9zL9F6/TFGr+OfjqaaO47/U1B7b79knl7e+eSW6GnsGSDgSb4JeTYVdFc9vxX4Zzf9b+MWUPeSGdOwhOuxWKxkLVWvj1ceBCzf0+9yBM7pXnioe0dNlNHe13zv3icCqSxHtzVWVEaAP8a2nkxI+765soW1fFmUcWd2dp4jc710eGNsD6f7fff+Gf4LkbI/t+40PY8G5kaAOsf7u3BndUB7s42Tf8UQpcCwwJf3wVbykz8bmxxX3btI0pzonYDhiMGdC2n0iE/BLIafXLfehx7fdf1uqZvt2boXw+DC2lzYnmkFKkWYfB7Zy73Tl3OzAUONY5d7Nz7mbgOKCkOwqU+Jo2sj9fOW0kaSlGwODCY4bw44smc9rYIgAy01L43qfHM6wgK8GVSo+XkgYXPwz9xwDmjXFPv739/gUjIrctAPnDoXAMnHc3ZBVCaiZM+ypM+WI8K/edWOcqWQFM3j87oJn1ARY558bFub6YaYz78OyqayQYdBELJlTurie7T4rm4ZbOC4UgcJA/6Pdsg8c+B1sWQyANzvgunHJzZB/nevtFycOaq+RR4D0zmx3e/iygx+CTSLQLj0UtZggU6ZSDhTZ483B/9S3YthyyiyC7sG2f3h3a7Yp1rpK7wkuXnRJu+k/n3AfxK0tEeo0B4xNdge90Znb8LGCXc+6XQLmZjTjYASIi0vViCm4zmwXcAnw33JQGPBavokREpH2xnnFfCJwP7AVwzm3Cu02wQ2Y2w8xWmNlqM7s1yv5TzWyBmTWZ2cWt9l1pZqvCH1fGWKeI+MHmRd7tgHU1ia7El2K9ONngnHNm5gDMrO3M+62YWQpwL3AWUA7MN7M5zrllLbptAL4EfKvVsQXALLz7xx3wfvjY6hjrlRjsrmtk6656RhVl4xys3b6HwfmZuotE4mvubd6q7gAZeXDlczBocmJr8pmD/gs1MwOeM7P7gXwz+zJwFfDgQQ6dCqx2zq0Nv89fgAuAA8HtnFsX3tfqMSnOAV50zlWF978IzAD+HMP3JDH46/wN/GDOMvY1BikpyKIxGGJzTR19+6Tyk4sn86lJgxJdoiSjXZtg3m+at+tq4M2fwyW6Sa0zDjpU4rwbvT8PPAU8DYwDvu+c+/VBDh0CbGyxXR5ui0VMx5rZNWZWZmZllZWVMb617KprZNacpexrDAKwoaqWzTV1gPd4+23PLKGhqfXvUpEuULer7ePs+6oSU4uPxTrGvQDY6Zz7tnPuW865F+NZVKyccw8450qdc6VFRUWJLsc3ttTUUdfYfjBX7W2gZl9jN1YkvcaAI2Ho1Mi2Y3UJq7NiHcycBnzRzNYTvkAJ4JzraGCqAmi52uzQcFssKoDTWx37WozHykGMLsphZGE2a7fvjbr/mJJ8PXwj8XP5U/DuA1C9DiacD2PPSXRFvhNrcB/Kf9n5wJjw/d4VwEzgshiPnQv80Mz6hbfPpvlWRDlMgYDxh/+cys9fXOFN6zqhmLrGEG+uqmRccV++fU6PmclAklFGHpz27URX4WtxnY/bzD4N3A2kAA+Fn8C8Ayhzzs0xs+OB2UA/oA7Y4pybGD72KuB74be6yzn3cEdfS3OViEgSivrMvxZSEBHpuaIGd2ceeRcRkR5AwS0i4jMKbhERn1Fwi4j4jIJbRMRnFNwiIj6j4BYR8RkFt4h0P+fgxe/DD4fCz8bCgkcTXZGvKLhFJHZN9fDS7XDfKTD7Wti1+dDeZ/GT8PYvoWE37NkKc26AyhVdW2sS04z5IhK7l+9oXgRhyyLYsQr+66XOv8/Gd1s1OCifD0WaJycWOuMWkdh99Fzkdvl82HMIc+EPO6FVg7Wd7lXapeAWkdj1HxO5nVUImfmdf59JF8PJN0GfPMgdAhfcA0Vju6TE3kBDJSISu3Pu8oZHqtd507OedzekpHX+fcxg+izvQzpNwS0isSsaB1//AKrWQN5QSMtMdEW9koJbRDonEIDCMQfvJ3GjMW4REZ/RGXcvs3rbbn74/Ecs3LiTzLQUvjB1GNeePpqUgDdf++aaffxs7krWVO6huG8fduxtoCA7nRs+OYajhuQluHoRAQV3r1LfFOSyB99l2+76A20/+9dKgiH4xnTvT9+r/lDG8s272hw7b+0O3rr1THIzDuFClIh0KQ2V9CILN+yMCO395i7dAsDGqtqooQ2wq66JeWt2xLU+EYmNgrsXGVaQRSDKCnYjCrMB6J+TTk6f9v8IOyLcTySukmQd3HhScPcig/MzuWXGkQfGswGG9cvk2+d4jxlnpadyxwUTyUpPAaBPqvfjkRowrj9jNGOL+3Z/0dJ71FbBn2bCHQVwz/Gw/p1EV9RjaZX3Xqh6bwMbqmrJTE9hdFEOgVan4bvrGtlcU8foohzWV9XSNyOVwpw+CapWeo05X4+cJTBnINy45NAe8EkeUVd518XJXqhfdjr9stPb3d83I42+4YuQIzQ8It2lvNWJ154tULMRCkYmpp4eTEMlItIzlJwYuZ07FPKHJ6aWHk5n3CLSM0yfBXU7YcUL3pOZn/kFBFISXVWPpOAWkZ4hIw8ufijRVfiChkpERHxGwS1R7axtYG9904HtbbvqaAyGEliRiOynoRKJ0NAU4uYnP+S5RZtITwlw+QklvPdxNYsraijMSedHn5vM9AnFiS5TpFfTGbdEeKJsI89+uAnnoL4pxO/fWsfiihoAtu9p4NtPfUh9UzDBVYr0bgpuibBiy+4O91fXNrJtV9v5TkSk+yi4JcLp44oitlvPbTKqKJuh/bTqiUgiaYxbInxyfDH/89mjeOyd9WT3SeHa00czb+0OXl6+ldED+nLbueMxi/oUroh0E81VIiLSc0U9S9JQiYj0DMEmeOVOuPcE+PNlULky0RX1WBoqEZHE2FMJ8+6FmgqYdDFsXgRv/NTbV7kcti31VpQP6PyyNQW3iHStnRth2TOQWQBHfQ7SolzMDoXgkfO8gAZY/AQUjIrsU70OKj+C4gnxrth3FNwi0nW2fQS/mw4N4dtKy34PV7/U9qy54v3m0N6vqdVtpmnZkDc0frX6mP4GEZGuM/93zaENXkCve6Ntv6yCtm3DpsGgKd7rPnlw3t2QkRuPKn1PZ9xJaF9DkIfe/piPK/dw6fElHD/C+0finOPNVdvZWF3LuOK+LN+ymzEDcpgyLJ8Xl21lY1UtCzZUU15dyydGFXLt6aN4e80O6hqDvLmykg8rajh30mC+c864NqvmiLQr2p1r/UfB1K/Ae/d72zkD4czbvPadGyG7MPoQiwC6HTDp7Klv4oyfvUZli9Xcb/v0eL586khuemIhf1tQ0eaYgux0qvY2tGk3i/5vbli/TN685cwurVuSxLbl4aGSPd724GPhv15qf17tLUtg1yYYcYqCOjotXdYb/GPRpojQBvjly6s4Z+LAqKENRA1taH+x7Y3V+1hTuYdRRTmHVaskoQHj4WvvwNLZ4YuTF3W8GMLAo7wP6RQFd5Kpb2o79WpjMERDsGsnhqpv1BSv0o78EvjENxJdRVLTxckkc+6kQWSlR57hXD6thNED+nLy6MKox2Skde7HIDcjlQmDddFIJFE0xp2Etu6q465/LKOieh8XHTeUy6Z5C67WNQZ56v1yNlbXMqxfFhuqahk9IIdTRhfytw8qWL65hsXlNezc18SEQX256ayxzFtbRVVtA699tI2tu+s5emgeD31pKpnpWgtQpBtEHeNWcIvI4Wuqh5R074q2dCXNVSIiXWxfNTx+CdxZDHdPglUvJrqiXkEXJ0Xk0L36Q1g113tdsxGeuAIGTIBgPUy9Bo69IrH1JSkFt4gcuooFkduNtVARHrKc83XIGwajzuj+upKchkpE5NCNOKXj/Ro6iYu4BreZzTCzFWa22sxujbK/j5n9Nbz/XTM7Itx+hJntM7OF4Y/74lmniBxEbRV89A+oWhvZftotcNyXIKs/FI5re9yA8d1SXm8Tt6ESM0sB7gXOAsqB+WY2xzm3rEW3q4Fq59xoM5sJ/Bi4NLxvjXNuSrzqSwbLN+/igTfWsqe+icumlbB9dz3/WLyZIfmZXHfGaAbnRz5CPPuDcv6+cBOD8jK57oxRDO2X1eY9//TuBuYu3cLw/llcf8ZoBuRmROz/n2eX8czCCnIz07jjgomcMqaozXtIkln3NvzpkvBj7Abn3AUnXuddmGxqgPN+6X0AvHQ7vHMPhJrgqIvh6JmH/nX3bvc+Z0d//qA3i9vtgGZ2IvAD59w54e3vAjjn/rdFn7nhPu+YWSqwBSgChgPPOedifha2t90OWL23gVN/+iq765qi7h9ZmM1LN512YDKo2R+Uc+NfPzywv6Qgi1duPo3UlOY/uh759zpmzVl6YPvIgX355zdOObDG5I/+uZz7Xm8+4woYzL9tOv1z+nTp9yY9zMOfhvVvN2+nZcNxV8J7D0KoESZcAJ/7HaSme/vrdnnBHW0GwFiEQvDcN+CDx7ztYy6Hz/yyty6o0O23Aw4BNrbYLg+3Re3jnGsCaoD+4X0jzOwDM3vdzKIOpJnZNWZWZmZllZWVXVt9D/faym3thjbA2u17WbKp5sD2sx9ujti/oaqWD8t3RrQ9++GmiO2Ptuxm9bY9B7b/vjByf8jB4+9u6Gzp4jf7z3z3a9wL837jhTbAsr/Dwseb92fkHnpoA6x4HhY8Ci7kfSx41GuTA3rqr7DNQIlz7hjgJuBPZtbmGWvn3APOuVLnXGlRUe/6k31QXsczqaUEjOIWwxyD8iKHPAJGxH6AQa2GVtJTAhS2OJsu6tv2zHpscd+YaxafOuaLkdvFUf4QrlzR/LqmHJ65Dh6aAf++xzuD7ozKj2Jr68XiGdwVwLAW20PDbVH7hIdK8oAdzrl659wOAOfc+8AaYGwca/WdE0b256Jjm1cHOXJQX4aEgzdg8M1PjokI5uvOGM0R/bMO7L/+zDFtxrhvnD6GweGATw0Y3z5nHP2y0w/s//FFk0lvMbQyYXAuM44a2PXfnPQsn/gGXHi/N2b9ye/DxQ95T0m2NPYc77Nz8NjFsPAx2PAO/Os2b8y7M8acDdYimizgtckB8RzjTgVWAp/EC+j5wGXOuaUt+lwHTHLOfTV8cfJzzrlLzKwIqHLOBc1sJPBmuF9Ve1+vt41x77e2cg9764NMGppHUzDEh+U1DM7PiHpGHgw5PizfSXFuxoGQb60xGGJR+U6G9stqc0YO0NgU5JkPNlHSP4tpI/tHeQfpFda8Cm/+HBr2wtQvw5TLvPbKFXDv1Mi+Q0rhyy937v2XPwv//rX3+qSvw/jzDr9mf+r+uUrM7NPA3UAK8JBz7i4zuwMoc87NMbMM4I/AMUAVMNM5t9bMLgLuABqBEDDLOfdsR1+rtwa3SLdqaoDNC6HfEZAzoO3+fTvh50dC077mtqMuhot/300FJh1NMiUih6FyBfzxQthVAYE077bAaV9p26/sIXjhu9BUBwUj4fKnvc9yKBTcInIY/vofsHxO83ZqBnxrJWTkte1bW+UF/ICJvfU2vq6ipctE5DDUlEduN9V5twpGC+6sgsO7JVA6pF+FIhKboy6K3B50tLcq++HYvRXm3gZ/vRyWPH1479WL6Iw7CTQGQ9Q1BOmbmQZAQ1OIgEFqSoDa+iaCIUd1bQOD8jIIOXBARpq3gk19U5CmphCNoRBpKSmkBCAtJYX6xiB76pvITA2QnZGGmbee5f7j6hqD9EkNHHiqUnqBE6+D1D7enCWFY+CUb0FNBSx+AtKyYPIlkNkv9vfbugwePR/2hh+eW/4sBBsP7zH5XkJj3D7mnOOWpxfxZFk5DuifncapYwfw/OLNmHn3Yu+pj75I8LQRBeRnpTF36daDfp20AORlpbN9TwOlw/uRlhLgnbU7GJKfyZ0XHsUZ46LcXSDJr3o93H8q1O30tgtGwlffgvTsgx+77m0vtEOtnv4deQZc8UxXV+pnWgEn2cxdupUnwqENsGNvI7M/qKC+KURdY6jd0AZ49+OqmEIboDEE2/c0AFC2vpp31u4AoGLnPr75l4Xsa+jaFeTFJxY+3hza4M0cuOKfsR373v1tQxugb6sHuhr3wY413oM9coCC28cWV+xMdAnU7GtkQ1VtosuQRLAoC0ZbjJESrV9WoTf8st/yZ+Hn4+DXx8I9pbB99aHVmYQU3D524sjET3dZnNuHkUUx/GksyefY/4DsFsNkAybAuE/HduyJ10Nqi6d3hx4PNy2DwtHedlMDzLkB6sITpe1YDS/+d9fUnQR0cdLHTh5TyM1nj+U3r66mvinE2OK+nD2xmL8t8KaECRhsqNoXcUxqANJTU5gxcSD9c9L54zvrqWtqngQoJXxRs75FW7/MVAblZ7FlVx2njCkkLSXAqx9t44jCbGadN4G0FP3+75VyB8PX5sGy2d7FyQkXQFrbaRIO2LIEnr7Ku62weBJc9U9Y/w7kDfFC/M8zvSGRE6+H4omwr9UMF9tXxvf78RFdnBSR+AkFvXm1P34Nlv4dXIvrIYVj4fr5sG053Hdy85h3IBW+8iY8c633eP1+J30dzr6zO6vvCfQAjogchvo9sOAR2LIYjjwXxp178KciX7gV3nsg+r7tK6F+Nyx/LvJCZajJm3975uPeijrblsOY6XD697rue/E5BbeIHNy+nfDbE2FXeDGND/8MEy6ES/7Q/jGh8CII7UnL8j6WPNV2X8FIyBsKFz14OFUnLQ1OisjBLX6yObT3Wza77eLBLQXrO76ne9q1sP7fbRdJKBgF488/9Fp7AZ1xi8jBBRtjb3fOGyIpe6jj+6///UvYtKBt+8jTIEXR1BGdcYvIwQ0YD4FW920P/wQUjWvbd/kcePc+CDY0r0t5wnVt+4WaYO2rkW2BVDjuS11ScjLTrzUfW7B+B7f+bQlba/Zx1JBcAoEAS8pr6J/TBzM4fkQB/bPT2bG7gdTUACMKs3hz5XbWVO5mc00dZsbt5x3J0k172bmvkdPHFYEZW3bWMW/tdgIBY+LgXDJSU/hg407yMtMozuvD/I+rqa1v4v99Zjwnj9Hj7kkv2Aizv+LdIbLfyNPhC3+J3n/zorZteUO85cdW/avjrzXmLG/yKumQbgf0qZ+98BH3vLYm0WXwHyeU8D+fnZToMiSeNn/ozUnSUk6xNxd3NB+/CY98pkWDwVffhPzh8IdzYUuUYN/vs/fBlC8cdslJRHOVJItgyPHb1zu4KNSNHpu3IdElSLzlD2/btq+6/f4jToHP/J93XN8hMOPHMHASrHnZu5WwPcVHKbRjpKESHwqGHMEe8pdSz6hC4iozHzILIp9kzC9pfl1TDtUbYOBRkJHb3L5zA+Bg7ve8oZJ599HhT8yoM7u48OSlM24fSk8NcOLInrG6yMRBuQfvJP536ncityd+DoJN3mPq/zcR/vAp+PERUPawNxb+0u0cCGnX5D0FGejgPLFPXvQLmBKVgtun/nj1NM6bPJBAeAQsALS85h8AcjNSGdE/k8KcdAbnZTB2QDYZaZH/y/tlpTI4L4N+WWkcNzyfsyYUM6Iwi8y0ABmpAYbmZVBSkEl2nxT6ZqQwMDed1ICREoDTxhTy3A0nd9e3LIm05uXI7Td/Dv87JHIaVxeE57/jrTdZvzuyf/0uqOrgmkx9DWxfEdm2bA7cPRnuHAjP3dj+LYm9kIZKfCo1JcCvLzuOXye6EEl+y/4OH78R2eaC0BRlHvZQA+zZAv2OaBvUuzc3v84phj2t5oN/5lrvUfoz/9tbz/Lpq71bCsG7J7xgFJx0/WF/O8lAZ9wi0r6N78ETV3pPQcZq9StwxEH+EqvfTZsbJnZVePOaPHsDbFrYHNoHank39hqSnIJbRNq34nk6fQl65T9hwkEeWc8phiNOjb7vo+dh8DGQkh7ZPmxa5+pIYgpuEWlf/9ExdGoVI/1Hwejp8MnvQ1pm9EOqP4Z1r0ff13cgmMFFv4e8EkjpA5M+D9O+0qnSk5mCW0TaN+kSGH9ex30KRjS/DqTBlMu916fcHLnKzYE+B7m0tnM9/GIC7NzojaUH62HxU94FUQEU3CLSkdR0uPQxGH1W9P3ZAyIvQoYaYeFj3uu37m67ig1EXyS4tWA9vHKHN+4NgIPXfwK7Nnd4WG+hu0p8avPOfZz7qzepqm2+RSpg3uWekGt/VNKAUUXZFGT3YUtNLRuq6yKOJ3w8QFrAGJyfyb7GRrbtjrwVK2AwND+DWecfxSfHF3fdNyY90ye+4U0ItT90i8Z7QyFzvt62b005bF0KL806vK/ZVBe57YKwdxvkDjq8900CCm6fOvWnr9IYjIznUAzXkBywunIvVO5ts6/18Y0hx/p2VnAPOdhQXcfVj5Tx4o2nMqa4b6yli9+8/yi88WNIzfAuGGb2856kLH/PW2ChtQFHwdz/1/V1FI2HgZO7/n19SMHtQx9trmkT2on029fW8ItLpyS6DImHj9+AZ1udVe+r8oZHNrxN2zmQDNa+AtuWdW0d+SVwxTPeRUvRGLcfFWT3SXQJEQbld7Cyt/jbO785SIdWJxCZBV0f2gCTL/XuNhFAwe1LA3IzmDi4ZwxN9MtK44YzxyS6DImXlneMxKKjWQMPVf4R8Ilvdv37+pjm4/ax5xdv4kfPf0RtQxOThuSSl5lOSsAIOcfG6joqdtbSFHIMyc+goSnItl0NlA4v4KLjhpKXlcaqrXtYUl7F3z7YRGZ6KledPIKPK/eyfMsuMlMDHDkolzPHF1NbH+TP721gQ1UtBdmp5GamMTA3k+OPKOAL06JM+SnJo36vNydJoueBvPRxGP+Zg/dLPlHHhhTcItKxP830noaMmXHQoM8qhNrtsb9lfgl8s4O5vJOXFlIQkUMwfRbt5Ec7wqGdOzT67vRsb4myztipBTtaUnCLSMfWvcUhDZXsKo/e3new9yRkZ1V93PljkpSCW0Q6digh25Edq5pXf++M1ivC92IKbhHpWOvpVROl9eIMvZiCW0Q6FtMMgd1gydOJrqDH0JOTPvXPJZu59rEFiS6DiYP68o9vtDOvsiSH9W8lugKPxrgP0Bm3T/WE0AZYunk3//t8HJ6Uk56jbleiK/Ck6gnd/RTcPrS7rmctmvr26h2JLkHiJdgEoVCiq/BkFyW6gh5Dwe1DfTPSEl1ChBNH9U90CRIvVWugKfoMkd2upp3bC3shBbdP/eayYxNdAgATBuVw27kTEl2GxEt+CaT0kBOFzixYnOR0cdKnPj15EOsmn5voMiTZpWV682BvWZToSryFFarXQb8jEl1JwumMW0Ta59zhPW6ekd9lpQDewsGi4BaRDrhQ2yXEOqO+K+9I0SIK+ym4RaR9gRQ49spDP75Lx8cd1GzswvfzL41xi0jHZvwIBh0N5fO97aq1UPE+NOxp/5iMPPjUT2D1y7D4ia6pI68EBveMi/KJpuAWkY4FAnDMF72P/e4s7viYuhoYery35JgLwqoXvUWGT/0OfPwa7FgDGNTt9C44umDzsZn9Wq2kY94iCtNvhxRFFsQ5uM1sBvBLIAX4nXPuR6329wEeBY4DdgCXOufWhfd9F7gaCAI3OOfmxrNWP3pl+VZum72YHXvraQx6415BvM/9c9JoaAqxtz5IYXYaexuC7G0IkZEGIWc4HOagIQhpKTAkP5PtextoaAgSBDLTAvRJDbCztomQg/RUb4SxrgnSUyAtJcBZE4r56eenkJKiEbdeJ3sA1HRw0bJPHvQd5C3ue/FDkfuOvbz59aqX4PGLIvePOcdbX3LBo5BVAJ+cBRPO77rak0DcVsAxsxRgJXAWUA7MB77gnFvWos/XgMnOua+a2UzgQufcpWY2AfgzMBUYDLwEjHWu5a/lSL1tBZyLf/tvytbHYX2/Q7Dgv6f3uAWMJc42vgePnNd84XLypbB1KWxd4g2TfOZuOOpzB3+fTR/AA6dHtp14PZxzV1dX7FdRr8jG84x7KrDaObcWwMz+AlwAtJzY4gLgB+HXTwH3mJmF2//inKsHPjaz1eH3eyeO9fpKTwltgFueXsSDVxyf6DKkOw2bCrdtgY3vQuFY78wYoHo95Azw7v+OxeBjYNIlzePgeSVwwtfiU3MSiWdwDwFaXgIuB6a118c512RmNUD/cPu8VscOaf0FzOwa4BqAkpKSLitcOqe8el+iS5BEMIOSEyLb+h3C4tEXPQgnfg32bocRp0Kq/no7GF+P9DvnHgAeAG+oJMHldKv0FKMh2DO+ZT3yLodt8DGJrsBX4nlVqQIY1mJ7aLgtah8zSwXy8C5SxnJsr7bw+2eRm5H437u3zhjHyaMLE12GSK8Sz3/584ExZjYCL3RnApe16jMHuBJv7Ppi4BXnnDOzOcCfzOwXeBcnxwDvxbFW38nqk8aiH5yT6DJEJAHiFtzhMevrgbl4twM+5JxbamZ3AGXOuTnA74E/hi8+VuGFO+F+T+BdyGwCruvojhIRkd4kbrcDdrfedjugiPQKUW8H1JMTIiI+o+AWEfEZBbeIiM8ouEVEfEbBLSLiMwpuERGfUXCLiPiMgltExGcU3CIiPpM0T06aWSWwPtF1+FAhsD3RRUjS08/ZodnunJvRujFpglsOjZmVOedKE12HJDf9nHUtDZWIiPiMgltExGcU3PJAoguQXkE/Z11IY9wiIj6jM24REZ9RcIuI+IyCO4mZ2UNmts3MlrRoKzCzF81sVfhzv3D7kWb2jpnVm9m3Ele1+EUnf77MzH5lZqvNbJGZHdvimCvD/VeZ2ZWJ+F78RsGd3P4AtL55/1bgZefcGODl8DZ4a37eAPys26oTv/sDsf98fQpv0e8xwDXAb8ELemAWMA2YCszaH/bSPgV3EnPOvYEXyC1dADwSfv0I8Nlw323OuflAY7cVKL7WmZ+vcPujzjMPyDezQcA5wIvOuSrnXDXwIm1/GUgrCu7ep9g5tzn8egtQnMhiJOm09/M1BNjYol95uK29dumAgrsXc969oLofVOJCP1/xo+DufbaG/0Ql/HlbguuR5NLez1cFMKxFv6HhtvbapQMK7t5nDrD/yv2VwN8TWIskn/Z+vuYAV4TvLjkBqAkPqcwFzjazfuGLkmeH26QDenIyiZnZn4HT8abU3Ip39f4Z4AmgBG8a3Eucc1VmNhAoA3KBELAHmOCc29X9lYsfdPLny4B78C481gL/6ZwrC7/PVcD3wm97l3Pu4W78NnxJwS0i4jMaKhER8RkFt4iIzyi4RUR8RsEtIuIzCm4REZ9RcItEYWY/0CyJ0lMpuEVEfEbBLRJmZreZ2UozewsYF26bYmbzwnNIz24xv/RoM3vJzD40swVmNsrMBpnZG2a20MyWmNkpCf2GJGkpuEUAMzsOmAlMAT4NHB/e9Shwi3NuMrAY7+lAgMeBe51zRwMnAZuBy4C5zrkpwNHAwm4qX3qZ1EQXINJDnALMds7VApjZHCAbyHfOvR7u8wjwpJn1BYY452YDOOfqwsfMBx4yszTgGefcwm7+HqSX0Bm3SBcJLyxwKt7sdn8wsysSXJIkKQW3iOcN4LNmlhk+oz4P2AtUtxir/g/gdefcbqDczD4LYGZ9zCzLzIYDW51zDwK/A45t81VEuoAmmRIJM7Pb8KYi3QZsABYALwH3AVnAWrxZ7arNbAxwP97MeI3A5/GGW74d3t4DXOGc+7i7vw9JfgpuERGf0VCJiIjPKLhFRHxGwS0i4jMKbhERn1Fwi4j4jIJbRMRnFNwiIj7z/wEMOFR9TUdg8wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import seaborn as sns\n",
    "sns.catplot(data=df, x='docs', y='redundancy', hue='docs')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}