RAKE.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": [
        "K_pa4ekXK_8F"
      ]
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# RAKE implementation and Document linking in Python\n",
        "\n",
        "This notebook aims to implement the Rapid Automatic Keyword Extraction (RAKE) method to extract keywords of a given corpus. Based on *Automatic keyword extraction from individual documents, 2010* (Stuart Rose, Dave Engel, Nick Cramer and Wendy Cowley).\n",
        "\n",
        "Then, a graph-matrix is computed to link documents if they share keywords."
      ],
      "metadata": {
        "id": "sjzCEe-qYRqQ"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 74,
      "metadata": {
        "id": "4zetPYXdhBvP"
      },
      "outputs": [],
      "source": [
        "# List of stop words\n",
        "stopList = [\"0o\", \"0s\", \"3a\", \"3b\", \"3d\", \"6b\", \"6o\", \"a\", \"a1\", \"a2\", \"a3\", \"a4\", \"ab\", \"able\", \"about\", \"above\", \"abst\", \"ac\", \"accordance\", \"according\", \"accordingly\", \"across\", \"act\", \"actually\", \"ad\", \"added\", \"adj\", \"ae\", \"af\", \"affected\", \"affecting\", \"affects\", \"after\", \"afterwards\", \"ag\", \"again\", \"against\", \"ah\", \"ain\", \"ain't\", \"aj\", \"al\", \"all\", \"allow\", \"allows\", \"almost\", \"alone\", \"along\", \"already\", \"also\", \"although\", \"always\", \"am\", \"among\", \"amongst\", \"amoungst\", \"amount\", \"an\", \"and\", \"announce\", \"another\", \"any\", \"anybody\", \"anyhow\", \"anymore\", \"anyone\", \"anything\", \"anyway\", \"anyways\", \"anywhere\", \"ao\", \"ap\", \"apart\", \"apparently\", \"appear\", \"appreciate\", \"appropriate\", \"approximately\", \"ar\", \"are\", \"aren\", \"arent\", \"aren't\", \"arise\", \"around\", \"as\", \"a's\", \"aside\", \"ask\", \"asking\", \"associated\", \"at\", \"au\", \"auth\", \"av\", \"available\", \"aw\", \"away\", \"awfully\", \"ax\", \"ay\", \"az\", \"b\", \"b1\", \"b2\", \"b3\", \"ba\", \"back\", \"bc\", \"bd\", \"be\", \"became\", \"because\", \"become\", \"becomes\", \"becoming\", \"been\", \"before\", \"beforehand\", \"begin\", \"beginning\", \"beginnings\", \"begins\", \"behind\", \"being\", \"believe\", \"below\", \"beside\", \"besides\", \"best\", \"better\", \"between\", \"beyond\", \"bi\", \"bill\", \"biol\", \"bj\", \"bk\", \"bl\", \"bn\", \"both\", \"bottom\", \"bp\", \"br\", \"brief\", \"briefly\", \"bs\", \"bt\", \"bu\", \"but\", \"bx\", \"by\", \"c\", \"c1\", \"c2\", \"c3\", \"ca\", \"call\", \"came\", \"can\", \"cannot\", \"cant\", \"can't\", \"cause\", \"causes\", \"cc\", \"cd\", \"ce\", \"certain\", \"certainly\", \"cf\", \"cg\", \"ch\", \"changes\", \"ci\", \"cit\", \"cj\", \"cl\", \"clearly\", \"cm\", \"c'mon\", \"cn\", \"co\", \"com\", \"come\", \"comes\", \"con\", \"concerning\", \"consequently\", \"consider\", \"considering\", \"contain\", \"containing\", \"contains\", \"corresponding\", \"could\", \"couldn\", \"couldnt\", \"couldn't\", \"course\", \"cp\", \"cq\", \"cr\", \"cry\", \"cs\", \"c's\", \"ct\", \"cu\", \"currently\", \"cv\", \"cx\", \"cy\", \"cz\", \"d\", \"d2\", \"da\", \"date\", \"dc\", \"dd\", \"de\", \"definitely\", \"describe\", \"described\", \"despite\", \"detail\", \"df\", \"di\", \"did\", \"didn\", \"didn't\", \"different\", \"dj\", \"dk\", \"dl\", \"do\", \"does\", \"doesn\", \"doesn't\", \"doing\", \"don\", \"done\", \"don't\", \"down\", \"downwards\", \"dp\", \"dr\", \"ds\", \"dt\", \"du\", \"due\", \"during\", \"dx\", \"dy\", \"e\", \"e2\", \"e3\", \"ea\", \"each\", \"ec\", \"ed\", \"edu\", \"ee\", \"ef\", \"effect\", \"eg\", \"ei\", \"eight\", \"eighty\", \"either\", \"ej\", \"el\", \"eleven\", \"else\", \"elsewhere\", \"em\", \"empty\", \"en\", \"end\", \"ending\", \"enough\", \"entirely\", \"eo\", \"ep\", \"eq\", \"er\", \"es\", \"especially\", \"est\", \"et\", \"et-al\", \"etc\", \"eu\", \"ev\", \"even\", \"ever\", \"every\", \"everybody\", \"everyone\", \"everything\", \"everywhere\", \"ex\", \"exactly\", \"example\", \"except\", \"ey\", \"f\", \"f2\", \"fa\", \"far\", \"fc\", \"few\", \"ff\", \"fi\", \"fifteen\", \"fifth\", \"fify\", \"fill\", \"find\", \"fire\", \"first\", \"five\", \"fix\", \"fj\", \"fl\", \"fn\", \"fo\", \"followed\", \"following\", \"follows\", \"for\", \"former\", \"formerly\", \"forth\", \"forty\", \"found\", \"four\", \"fr\", \"from\", \"front\", \"fs\", \"ft\", \"fu\", \"full\", \"further\", \"furthermore\", \"fy\", \"g\", \"ga\", \"gave\", \"ge\", \"get\", \"gets\", \"getting\", \"gi\", \"give\", \"given\", \"gives\", \"giving\", \"gj\", \"gl\", \"go\", \"goes\", \"going\", \"gone\", \"got\", \"gotten\", \"gr\", \"greetings\", \"gs\", \"gy\", \"h\", \"h2\", \"h3\", \"had\", \"hadn\", \"hadn't\", \"happens\", \"hardly\", \"has\", \"hasn\", \"hasnt\", \"hasn't\", \"have\", \"haven\", \"haven't\", \"having\", \"he\", \"hed\", \"he'd\", \"he'll\", \"hello\", \"help\", \"hence\", \"her\", \"here\", \"hereafter\", \"hereby\", \"herein\", \"heres\", \"here's\", \"hereupon\", \"hers\", \"herself\", \"hes\", \"he's\", \"hh\", \"hi\", \"hid\", \"him\", \"himself\", \"his\", \"hither\", \"hj\", \"ho\", \"home\", \"hopefully\", \"how\", \"howbeit\", \"however\", \"how's\", \"hr\", \"hs\", \"http\", \"hu\", \"hundred\", \"hy\", \"i\", \"i2\", \"i3\", \"i4\", \"i6\", \"i7\", \"i8\", \"ia\", \"ib\", \"ibid\", \"ic\", \"id\", \"i'd\", \"ie\", \"if\", \"ig\", \"ignored\", \"ih\", \"ii\", \"ij\", \"il\", \"i'll\", \"im\", \"i'm\", \"immediate\", \"immediately\", \"importance\", \"important\", \"in\", \"inasmuch\", \"inc\", \"indeed\", \"index\", \"indicate\", \"indicated\", \"indicates\", \"information\", \"inner\", \"insofar\", \"instead\", \"interest\", \"into\", \"invention\", \"inward\", \"io\", \"ip\", \"iq\", \"ir\", \"is\", \"isn\", \"isn't\", \"it\", \"itd\", \"it'd\", \"it'll\", \"its\", \"it's\", \"itself\", \"iv\", \"i've\", \"ix\", \"iy\", \"iz\", \"j\", \"jj\", \"jr\", \"js\", \"jt\", \"ju\", \"just\", \"k\", \"ke\", \"keep\", \"keeps\", \"kept\", \"kg\", \"kj\", \"km\", \"know\", \"known\", \"knows\", \"ko\", \"l\", \"l2\", \"la\", \"largely\", \"last\", \"lately\", \"later\", \"latter\", \"latterly\", \"lb\", \"lc\", \"le\", \"least\", \"les\", \"less\", \"lest\", \"let\", \"lets\", \"let's\", \"lf\", \"like\", \"liked\", \"likely\", \"line\", \"little\", \"lj\", \"ll\", \"ll\", \"ln\", \"lo\", \"look\", \"looking\", \"looks\", \"los\", \"lr\", \"ls\", \"lt\", \"ltd\", \"m\", \"m2\", \"ma\", \"made\", \"mainly\", \"make\", \"makes\", \"many\", \"may\", \"maybe\", \"me\", \"mean\", \"means\", \"meantime\", \"meanwhile\", \"merely\", \"mg\", \"might\", \"mightn\", \"mightn't\", \"mill\", \"million\", \"mine\", \"miss\", \"ml\", \"mn\", \"mo\", \"more\", \"moreover\", \"most\", \"mostly\", \"move\", \"mr\", \"mrs\", \"ms\", \"mt\", \"mu\", \"much\", \"mug\", \"must\", \"mustn\", \"mustn't\", \"my\", \"myself\", \"n\", \"n2\", \"na\", \"name\", \"namely\", \"nay\", \"nc\", \"nd\", \"ne\", \"near\", \"nearly\", \"necessarily\", \"necessary\", \"need\", \"needn\", \"needn't\", \"needs\", \"neither\", \"never\", \"nevertheless\", \"new\", \"next\", \"ng\", \"ni\", \"nine\", \"ninety\", \"nj\", \"nl\", \"nn\", \"no\", \"nobody\", \"non\", \"none\", \"nonetheless\", \"noone\", \"nor\", \"normally\", \"nos\", \"not\", \"noted\", \"nothing\", \"novel\", \"now\", \"nowhere\", \"nr\", \"ns\", \"nt\", \"ny\", \"o\", \"oa\", \"ob\", \"obtain\", \"obtained\", \"obviously\", \"oc\", \"od\", \"of\", \"off\", \"often\", \"og\", \"oh\", \"oi\", \"oj\", \"ok\", \"okay\", \"ol\", \"old\", \"om\", \"omitted\", \"on\", \"once\", \"one\", \"ones\", \"only\", \"onto\", \"oo\", \"op\", \"oq\", \"or\", \"ord\", \"os\", \"ot\", \"other\", \"others\", \"otherwise\", \"ou\", \"ought\", \"our\", \"ours\", \"ourselves\", \"out\", \"outside\", \"over\", \"overall\", \"ow\", \"owing\", \"own\", \"ox\", \"oz\", \"p\", \"p1\", \"p2\", \"p3\", \"page\", \"pagecount\", \"pages\", \"par\", \"part\", \"particular\", \"particularly\", \"pas\", \"past\", \"pc\", \"pd\", \"pe\", \"per\", \"perhaps\", \"pf\", \"ph\", \"pi\", \"pj\", \"pk\", \"pl\", \"placed\", \"please\", \"plus\", \"pm\", \"pn\", \"po\", \"poorly\", \"possible\", \"possibly\", \"potentially\", \"pp\", \"pq\", \"pr\", \"predominantly\", \"present\", \"presumably\", \"previously\", \"primarily\", \"probably\", \"promptly\", \"proud\", \"provides\", \"ps\", \"pt\", \"pu\", \"put\", \"py\", \"q\", \"qj\", \"qu\", \"que\", \"quickly\", \"quite\", \"qv\", \"r\", \"r2\", \"ra\", \"ran\", \"rather\", \"rc\", \"rd\", \"re\", \"readily\", \"really\", \"reasonably\", \"recent\", \"recently\", \"ref\", \"refs\", \"regarding\", \"regardless\", \"regards\", \"related\", \"relatively\", \"research\", \"research-articl\", \"respectively\", \"resulted\", \"resulting\", \"results\", \"rf\", \"rh\", \"ri\", \"right\", \"rj\", \"rl\", \"rm\", \"rn\", \"ro\", \"rq\", \"rr\", \"rs\", \"rt\", \"ru\", \"run\", \"rv\", \"ry\", \"s\", \"s2\", \"sa\", \"said\", \"same\", \"saw\", \"say\", \"saying\", \"says\", \"sc\", \"sd\", \"se\", \"sec\", \"second\", \"secondly\", \"section\", \"see\", \"seeing\", \"seem\", \"seemed\", \"seeming\", \"seems\", \"seen\", \"self\", \"selves\", \"sensible\", \"sent\", \"serious\", \"seriously\", \"seven\", \"several\", \"sf\", \"shall\", \"shan\", \"shan't\", \"she\", \"shed\", \"she'd\", \"she'll\", \"shes\", \"she's\", \"should\", \"shouldn\", \"shouldn't\", \"should've\", \"show\", \"showed\", \"shown\", \"showns\", \"shows\", \"si\", \"side\", \"significant\", \"significantly\", \"similar\", \"similarly\", \"since\", \"sincere\", \"six\", \"sixty\", \"sj\", \"sl\", \"slightly\", \"sm\", \"sn\", \"so\", \"some\", \"somebody\", \"somehow\", \"someone\", \"somethan\", \"something\", \"sometime\", \"sometimes\", \"somewhat\", \"somewhere\", \"soon\", \"sorry\", \"sp\", \"specifically\", \"specified\", \"specify\", \"specifying\", \"sq\", \"sr\", \"ss\", \"st\", \"still\", \"stop\", \"strongly\", \"sub\", \"substantially\", \"successfully\", \"such\", \"sufficiently\", \"suggest\", \"sup\", \"sure\", \"sy\", \"system\", \"sz\", \"t\", \"t1\", \"t2\", \"t3\", \"take\", \"taken\", \"taking\", \"tb\", \"tc\", \"td\", \"te\", \"tell\", \"ten\", \"tends\", \"tf\", \"th\", \"than\", \"thank\", \"thanks\", \"thanx\", \"that\", \"that'll\", \"thats\", \"that's\", \"that've\", \"the\", \"their\", \"theirs\", \"them\", \"themselves\", \"then\", \"thence\", \"there\", \"thereafter\", \"thereby\", \"thered\", \"therefore\", \"therein\", \"there'll\", \"thereof\", \"therere\", \"theres\", \"there's\", \"thereto\", \"thereupon\", \"there've\", \"these\", \"they\", \"theyd\", \"they'd\", \"they'll\", \"theyre\", \"they're\", \"they've\", \"thickv\", \"thin\", \"think\", \"third\", \"this\", \"thorough\", \"thoroughly\", \"those\", \"thou\", \"though\", \"thoughh\", \"thousand\", \"three\", \"throug\", \"through\", \"throughout\", \"thru\", \"thus\", \"ti\", \"til\", \"tip\", \"tj\", \"tl\", \"tm\", \"tn\", \"to\", \"together\", \"too\", \"took\", \"top\", \"toward\", \"towards\", \"tp\", \"tq\", \"tr\", \"tried\", \"tries\", \"truly\", \"try\", \"trying\", \"ts\", \"t's\", \"tt\", \"tv\", \"twelve\", \"twenty\", \"twice\", \"two\", \"tx\", \"u\", \"u201d\", \"ue\", \"ui\", \"uj\", \"uk\", \"um\", \"un\", \"under\", \"unfortunately\", \"unless\", \"unlike\", \"unlikely\", \"until\", \"unto\", \"uo\", \"up\", \"upon\", \"ups\", \"ur\", \"us\", \"use\", \"used\", \"useful\", \"usefully\", \"usefulness\", \"uses\", \"using\", \"usually\", \"ut\", \"v\", \"va\", \"value\", \"various\", \"vd\", \"ve\", \"ve\", \"very\", \"via\", \"viz\", \"vj\", \"vo\", \"vol\", \"vols\", \"volumtype\", \"vq\", \"vs\", \"vt\", \"vu\", \"w\", \"wa\", \"want\", \"wants\", \"was\", \"wasn\", \"wasnt\", \"wasn't\", \"way\", \"we\", \"wed\", \"we'd\", \"welcome\", \"well\", \"we'll\", \"well-b\", \"went\", \"were\", \"we're\", \"weren\", \"werent\", \"weren't\", \"we've\", \"what\", \"whatever\", \"what'll\", \"whats\", \"what's\", \"when\", \"whence\", \"whenever\", \"when's\", \"where\", \"whereafter\", \"whereas\", \"whereby\", \"wherein\", \"wheres\", \"where's\", \"whereupon\", \"wherever\", \"whether\", \"which\", \"while\", \"whim\", \"whither\", \"who\", \"whod\", \"whoever\", \"whole\", \"who'll\", \"whom\", \"whomever\", \"whos\", \"who's\", \"whose\", \"why\", \"why's\", \"wi\", \"widely\", \"will\", \"willing\", \"wish\", \"with\", \"within\", \"without\", \"wo\", \"won\", \"wonder\", \"wont\", \"won't\", \"words\", \"world\", \"would\", \"wouldn\", \"wouldnt\", \"wouldn't\", \"www\", \"x\", \"x1\", \"x2\", \"x3\", \"xf\", \"xi\", \"xj\", \"xk\", \"xl\", \"xn\", \"xo\", \"xs\", \"xt\", \"xv\", \"xx\", \"y\", \"y2\", \"yes\", \"yet\", \"yj\", \"yl\", \"you\", \"youd\", \"you'd\", \"you'll\", \"your\", \"youre\", \"you're\", \"yours\", \"yourself\", \"yourselves\", \"you've\", \"yr\", \"ys\", \"yt\", \"z\", \"zero\", \"zi\", \"zz\"]\n",
        "\n",
        "# Items to split a document into sentences\n",
        "phraseDelimiters = ['.', '!', '?', ',', ':', '; ', '(', ')']\n",
        "\n",
        "# Items to split a document into words\n",
        "wordDelimiters = [' ', '[', ']']\n",
        "\n",
        "# Items that represent the end of a word\n",
        "wordEnds = ['[']\n",
        "\n",
        "documents = [\n",
        "    \"Artificial intelligence (AI) is the intelligence of machines or software, as opposed to the intelligence of humans or other animals. It is a field of study in computer science that develops and studies intelligent machines. Such machines may be called AIs. AI technology is widely used throughout industry, government, and science. Some high-profile applications are: advanced web search engines (e.g., Google Search), recommendation systems (used by YouTube, Amazon, and Netflix), understanding human speech (such as Google Assistant, Siri, and Alexa), self-driving cars (e.g., Waymo), generative and creative tools (ChatGPT and AI art), and superhuman play and analysis in strategy games (such as chess and Go).[1] Alan Turing was the first person to conduct substantial research in the field that he called Machine Intelligence.[2] Artificial intelligence was founded as an academic discipline in 1956.[3] The field went through multiple cycles of optimism[4][5] followed by disappointment and loss of funding.[6][7] Funding and interest vastly increased after 2012 when deep learning surpassed all previous AI techniques,[8] and after 2017 with the transformer architecture.[9] This led to the AI spring of the early 2020s, with companies, universities, and laboratories overwhelmingly based in the United States pioneering significant advances in artificial intelligence.[10] The various sub-fields of AI research are centered around particular goals and the use of particular tools. The traditional goals of AI research include reasoning, knowledge representation, planning, learning, natural language processing, perception, and support for robotics.[a] General intelligence (the ability to complete any task performable by a human) is among the field's long-term goals.[11] To solve these problems, AI researchers have adapted and integrated a wide range of problem-solving techniques, including search and mathematical optimization, formal logic, artificial neural networks, and methods based on statistics, operations research, and economics.[b] AI also draws upon psychology, linguistics, philosophy, neuroscience and other fields.[12] Goals The general problem of simulating (or creating) intelligence has been broken into sub-problems. These consist of particular traits or capabilities that researchers expect an intelligent system to display. The traits described below have received the most attention and cover the scope of AI research.[a] Reasoning, problem-solving Early researchers developed algorithms that imitated step-by-step reasoning that humans use when they solve puzzles or make logical deductions.[13] By the late 1980s and 1990s, methods were developed for dealing with uncertain or incomplete information, employing concepts from probability and economics.[14]\",\n",
        "    \"Natural language processing (NLP) is an interdisciplinary subfield of computer science and linguistics. It is primarily concerned with giving computers the ability to support and manipulate human language. It involves processing natural language datasets, such as text corpora or speech corpora, using either rule-based or probabilistic (i.e. statistical and, most recently, neural network-based) machine learning approaches. The goal is a computer capable of understanding the contents of documents, including the contextual nuances of the language within them. The technology can then accurately extract information and insights contained in the documents as well as categorize and organize the documents themselves.Challenges in natural language processing frequently involve speech recognition, natural-language understanding, and natural-language generation.History[edit]Natural language processing has its roots in the 1950s. Already in 1950, Alan Turing published an article titled Computing Machinery and Intelligence which proposed what is now called the Turing test as a criterion of intelligence, though at the time that was not articulated as a problem separate from artificial intelligence. The proposed test includes a task that involves the automated interpretation and generation of natural language.Symbolic NLP (1950s – early 1990s)[edit]The premise of symbolic NLP is well-summarized by John Searle's Chinese room experiment: Given a collection of rules (e.g., a Chinese phrasebook, with questions and matching answers), the computer emulates natural language understanding (or other NLP tasks) by applying those rules to the data it confronts.1950s: The Georgetown experiment in 1954 involved fully automatic translation of more than sixty Russian sentences into English. The authors claimed that within three or five years, machine translation would be a solved problem.[1] However, real progress was much slower, and after the ALPAC report in 1966, which found that ten-year-long research had failed to fulfill the expectations, funding for machine translation was dramatically reduced. Little further research in machine translation was conducted in America (though some research continued elsewhere, such as Japan and Europe[2]) until the late 1980s when the first statistical machine translation systems were developed.1960s: Some notably successful natural language processing systems developed in the 1960s were SHRDLU, a natural language system working in restricted blocks worlds with restricted vocabularies, and ELIZA, a simulation of a Rogerian psychotherapist, written by Joseph Weizenbaum between 1964 and 1966. Using almost no information about human thought or emotion, ELIZA sometimes provided a startlingly human-like interaction. When the patient exceeded the very small knowledge base, ELIZA might provide a generic response, for example, responding to My head hurts with Why do you say your head hurts?. Ross Quillian's successful work on natural language was demonstrated with a vocabulary of only twenty words, because that was all that would fit in a computer memory at the time.[3]1970s: During the 1970s, many programmers began to write conceptual ontologies, which structured real-world information into computer-understandable data. Examples are MARGIE (Schank, 1975), SAM (Cullingford, 1978), PAM (Wilensky, 1978), TaleSpin (Meehan, 1976), QUALM (Lehnert, 1977), Politics (Carbonell, 1979), and Plot Units (Lehnert 1981). During this time, the first chatterbots were written (e.g., PARRY).1980s: The 1980s and early 1990s mark the heyday of symbolic methods in NLP. Focus areas of the time included research on rule-based parsing (e.g., the development of HPSG as a computational operationalization of generative grammar), morphology (e.g., two-level morphology[4]), semantics (e.g., Lesk algorithm), reference (e.g., within Centering Theory[5]) and other areas of natural language understanding (e.g., in the Rhetorical Structure Theory). Other lines of research were continued, e.g., the development of chatterbots with Racter and Jabberwacky. An important development (that eventually led to the statistical turn in the 1990s) was the rising importance of quantitative evaluation in this period.[6]Statistical NLP (1990s–2010s)[edit]Up until the 1980s, most natural language processing systems were based on complex sets of hand-written rules. Starting in the late 1980s, however, there was a revolution in natural language processing with the introduction of machine learning algorithms for language processing. This was due to both the steady increase in computational power (see Moore's law) and the gradual lessening of the dominance of Chomskyan theories of linguistics (e.g. transformational grammar), whose theoretical underpinnings discouraged the sort of corpus linguistics that underlies the machine-learning approach to language processing.[7]1990s: Many of the notable early successes on statistical methods in NLP occurred in the field of machine translation, due especially to work at IBM Research, such as IBM alignment models. These systems were able to take advantage of existing multilingual textual corpora that had been produced by the Parliament of Canada and the European Union as a result of laws calling for the translation of all governmental proceedings into all official languages of the corresponding systems of government. However, most other systems depended on corpora specifically developed for the tasks implemented by these systems, which was (and often continues to be) a major limitation in the success of these systems. As a result, a great deal of research has gone into methods of more effectively learning from limited amounts of data.2000s: With the growth of the web, increasing amounts of raw (unannotated) language data has become available since the mid-1990s. Research has thus increasingly focused on unsupervised and semi-supervised learning algorithms. Such algorithms can learn from data that has not been hand-annotated with the desired answers or using a combination of annotated and non-annotated data. Generally, this task is much more difficult than supervised learning, and typically produces less accurate results for a given amount of input data. However, there is an enormous amount of non-annotated data available (including, among other things, the entire content of the World Wide Web), which can often make up for the inferior results if the algorithm used has a low enough time complexity to be practical\",\n",
        "    \"Deep learning is the subset of machine learning methods based on artificial neural networks with representation learning. The adjective deep refers to the use of multiple layers in the network. Methods used can be either supervised, semi-supervised or unsupervised.[2]Deep-learning architectures such as deep neural networks, deep belief networks, recurrent neural networks, convolutional neural networks and transformers have been applied to fields including computer vision, speech recognition, natural language processing, machine translation, bioinformatics, drug design, medical image analysis, climate science, material inspection and board game programs, where they have produced results comparable to and in some cases surpassing human expert performance.[3][4][5]Artificial neural networks (ANNs) were inspired by information processing and distributed communication nodes in biological systems. ANNs have various differences from biological brains. Specifically, artificial neural networks tend to be static and symbolic, while the biological brain of most living organisms is dynamic (plastic) and analog.[6][7] ANNs are generally seen as low quality models for brain function.[8]Definition[edit]Deep learning is a class of machine learning algorithms that[9]: 199–200  uses multiple layers to progressively extract higher-level features from the raw input. For example, in image processing, lower layers may identify edges, while higher layers may identify the concepts relevant to a human such as digits or letters or faces.From another angle to view deep learning, deep learning refers to computer-simulate or automate human learning processes from a source (e.g., an image of dogs) to a learned object (dogs). Therefore, a notion coined as deeper learning or deepest learning[10] makes sense. The deepest learning refers to the fully automatic learning from a source to a final learned object. A deeper learning thus refers to a mixed learning process: a human learning process from a source to a learned semi-object, followed by a computer learning process from the human learned semi-object to a final learned object.Overview[edit]Most modern deep learning models are based on multi-layered artificial neural networks such as convolutional neural networks and transformers, although they can also include propositional formulas or latent variables organized layer-wise in deep generative models such as the nodes in deep belief networks and deep Boltzmann machines.[11]In deep learning, each level learns to transform its input data into a slightly more abstract and composite representation. In an image recognition application, the raw input may be a matrix of pixels; the first representational layer may abstract the pixels and encode edges; the second layer may compose and encode arrangements of edges; the third layer may encode a nose and eyes; and the fourth layer may recognize that the image contains a face. Importantly, a deep learning process can learn which features to optimally place in which level on its own. This does not eliminate the need for hand-tuning; for example, varying numbers of layers and layer sizes can provide different degrees of abstraction.[12][13]The word deep in deep learning refers to the number of layers through which the data is transformed. More precisely, deep learning systems have a substantial credit assignment path (CAP) depth. The CAP is the chain of transformations from input to output. CAPs describe potentially causal connections between input and output. For a feedforward neural network, the depth of the CAPs is that of the network and is the number of hidden layers plus one (as the output layer is also parameterized). For recurrent neural networks, in which a signal may propagate through a layer more than once, the CAP depth is potentially unlimited.[14] No universally agreed-upon threshold of depth divides shallow learning from deep learning, but most researchers agree that deep learning involves CAP depth higher than 2. CAP of depth 2 has been shown to be a universal approximator in the sense that it can emulate any function.[15] Beyond that, more layers do not add to the function approximator ability of the network. Deep models (CAP > 2) are able to extract better features than shallow models and hence, extra layers help in learning the features effectively.Deep learning architectures can be constructed with a greedy layer-by-layer method.[16] Deep learning helps to disentangle these abstractions and pick out which features improve performance.[12]For supervised learning tasks, deep learning methods enable elimination of feature engineering, by translating the data into compact intermediate representations akin to principal components, and derive layered structures that remove redundancy in representation.Deep learning algorithms can be applied to unsupervised learning tasks. This is an important benefit because unlabeled data are more abundant than the labeled data. Examples of deep structures that can be trained in an unsupervised manner are deep belief networks.[12][17]Machine learning models are now adept at identifying complex patterns in financial market data. Due to the benefits of artificial intelligence, investors are increasingly utilizing deep learning techniques to forecast and analyze trends in stock and foreign exchange markets.[18]\"\n",
        "    ]"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Extracting words from a text"
      ],
      "metadata": {
        "id": "52UCBX_9K9Hd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def extractWords(text, textId):\n",
        "  \"\"\"\n",
        "  Returns all words of a text as a list where each word is a\n",
        "  tuple (word, textId)\n",
        "\n",
        "  Parameters:\n",
        "  text (string): text to split into words\n",
        "  textId (int): id to affect to each word\n",
        "\n",
        "  Returns:\n",
        "  list[(string, int)]: list of tuples (words, textId)\n",
        "  \"\"\"\n",
        "  words = []\n",
        "  for w in text.split(' '):\n",
        "    word = w\n",
        "\n",
        "    buffer = ''\n",
        "    for letter in word:\n",
        "      if letter in wordEnds or letter in wordDelimiters:\n",
        "        word = buffer\n",
        "        break\n",
        "      else:\n",
        "        buffer += letter\n",
        "\n",
        "    word = ''.join((x for x in word.lower() if not x.isdigit()))\n",
        "    if (len(word) > 1): words.append((word, textId))\n",
        "\n",
        "  return words"
      ],
      "metadata": {
        "id": "lXSF20g0haoI"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Keywords stats extraction\n",
        "\n",
        "Create a list of sentences based on delimiters as '.' that were not removed from the words.\n",
        "\n",
        "A matrix of co-occurrences is also created: if two words occur in the same sentence, 1 is added to the count of their co-occurrence. This value can be red in the line corresponding to one word and the column of the other.\n",
        "To know which line/colum correponds to which word, a word-index list is created simultaneously: the ith line/column of the matrix correspond to the ith word of the word-index list."
      ],
      "metadata": {
        "id": "K_pa4ekXK_8F"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def extractInfoKeywords(words):\n",
        "  \"\"\"\n",
        "  Create a co-occurence matrix with a corresponding word-index list and a list\n",
        "  of all sentences.\n",
        "\n",
        "  Parameters:\n",
        "  words (list) : list of tuple (word, textId)\n",
        "\n",
        "  Returns:\n",
        "  list[(list[string], int)]: list of sentences (=lists of tuples): (words, textId)\n",
        "  list[list[int]]: matrix of co-occurrences\n",
        "  list[string]: index of the matrix\n",
        "  \"\"\"\n",
        "  sentences = []          # list of tuples : (keyword - a list of words, textId)\n",
        "  matrixCooc = []         # matrix of co-occurrences\n",
        "  wordIndexingMatrix = [] # list of words = index of matrix\n",
        "  sentence = []\n",
        "  for word, docId in words:\n",
        "    endOfSentence = False # True if a word is the last of a sentence\n",
        "\n",
        "    for delimiter in phraseDelimiters: # Checking if 'word' is the last one\n",
        "      if delimiter in word:\n",
        "        word = word.replace(delimiter, '')\n",
        "        endOfSentence = True\n",
        "        # no break as it is possible to encounter '!?'\n",
        "\n",
        "    # Next, the word is added to the sentence if it is not a stop word.\n",
        "    if not word in stopList:\n",
        "      sentence.append(word)\n",
        "      if not word in wordIndexingMatrix:\n",
        "        # In that case, this is the first seeing this word in that document.\n",
        "        # It is thus needed to append a new line and a new column to the matrix\n",
        "        # as well as a new entry in the word-indexing matrix.\n",
        "        wordIndexingMatrix.append(word)\n",
        "\n",
        "        if len(matrixCooc) > 0: # If the matrix already exist, expand it.\n",
        "          for line in matrixCooc:\n",
        "            line.append(0)\n",
        "          matrixCooc.append([0 for _ in range(len(matrixCooc[-1]))])\n",
        "        else: # If the matrix does not exist yet, create the first entry.\n",
        "          matrixCooc = [[0]]\n",
        "\n",
        "      # As the word is not a stop word, a loop is made on all words of the\n",
        "      # sentence to update the co-occurrences matrix: the entry of the current\n",
        "      # word is added one for each words of the same (current) sentence.\n",
        "      line = wordIndexingMatrix.index(word)\n",
        "      for wordInKeyword in sentence:\n",
        "        col = wordIndexingMatrix.index(wordInKeyword)\n",
        "        matrixCooc[line][col] += 1\n",
        "        if line != col: matrixCooc[col][line] += 1\n",
        "\n",
        "    # If the word is a stop word or that a phrase delimiters matched,\n",
        "    # the current sentence is added to the sentences list.\n",
        "    if word in stopList or endOfSentence:\n",
        "      if len(sentence) > 0: sentences.append((sentence, docId))\n",
        "      sentence = []\n",
        "\n",
        "  return sentences, matrixCooc, wordIndexingMatrix"
      ],
      "metadata": {
        "id": "BnMmhA0Z3PUZ"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Keyword significance measurement\n",
        "\n",
        "As described in *Automatic keyword extraction from individual documents, 2010*, several measurement methods can be created to evaluate the relevance of a keywords.\n",
        "\n",
        "In this section,\n",
        "- rdf(k) means number of doc in which the keyword occurred as candidate\n",
        "- edf(k) means number of doc from which the keyword was extracted"
      ],
      "metadata": {
        "id": "-9bBohqKtBKZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# @title Scoring\n",
        "\n",
        "def deg(matrix, index):\n",
        "  \"\"\"\n",
        "  Count the words appearing in the same sentence of a given word.\n",
        "\n",
        "  Parameters:\n",
        "  matrix (list[list[int]]): matrix of co-occurrences\n",
        "  index[int]: index of the word\n",
        "\n",
        "  Returns:\n",
        "  int: number of words appearing in the same sentence of a given word.\n",
        "  \"\"\"\n",
        "  return sum([i for i in matrix[index] if i != 0])\n",
        "\n",
        "def freq(matrix, index):\n",
        "  \"\"\"\n",
        "  Count the number of time a word appeared.\n",
        "\n",
        "  Parameters:\n",
        "  matrix (list[list[int]]): matrix of co-occurrences\n",
        "  index[int]: index of the word\n",
        "\n",
        "  Returns:\n",
        "  int: number of time the word appeared.\n",
        "  \"\"\"\n",
        "  return matrix[index][index]\n",
        "\n",
        "def score(matrix, sentences, wordIndexingMatrix):\n",
        "  \"\"\"\n",
        "  Associate each keywords to its score given by deg/freq\n",
        "\n",
        "  Parameters:\n",
        "  matrix (list[list[int]]): matrix of co-occurrences\n",
        "  list[(list[string], int)]: list of sentences (=lists of tuples): (words, textId)\n",
        "  list[string]: index of the matrix\n",
        "\n",
        "  Returns\n",
        "  dict{string: float}: dict that maps each keywords to its score\n",
        "  \"\"\"\n",
        "  kwAndScores = {}\n",
        "\n",
        "  for keyword_list, textId in sentences:\n",
        "    keyword = ' '.join(keyword_list)\n",
        "\n",
        "    kwAndScores[keyword] = 0\n",
        "    for word in keyword_list:\n",
        "      index = wordIndexingMatrix.index(word)\n",
        "      kwAndScores[keyword] += deg(matrix, index) / freq(matrix, index)\n",
        "\n",
        "  return kwAndScores"
      ],
      "metadata": {
        "id": "If14PPfqa6P9"
      },
      "execution_count": 51,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# @title exclusivity\n",
        "def exclusivity(keywordsOfEachDoc):\n",
        "  \"\"\"\n",
        "  Computes the list of exclusivity score for each keywords.\n",
        "  Exclusivity is computed as edf / rdf\n",
        "\n",
        "  Parameters:\n",
        "  keywordsOfEachDoc (list[(keyword, textId)]): list of tuples (kw and their textId)\n",
        "\n",
        "  Returns:\n",
        "  list[(string, float, int)]: list of exclusitity scores as (kw, score, textId)\n",
        "  \"\"\"\n",
        "\n",
        "  # Creating a list of keywords\n",
        "  allKeywords = []\n",
        "  for kws, textId in keywordsOfEachDoc:\n",
        "    allKeywords += kws\n",
        "\n",
        "  # List of exclusivity score as (kw, score, textId)\n",
        "  excs = []\n",
        "\n",
        "  for kw in allKeywords:\n",
        "    exc, rdf, edf = 0, 0, 1\n",
        "    # a kw was at least in one document.\n",
        "    # So edf is forced to 1, because we are removing part of kw as '(' or ')'\n",
        "    for i in range(len(documents)):\n",
        "      doc = documents[i]\n",
        "      # Computes the edf and rdf scores\n",
        "      if kw in doc.lower():\n",
        "        edf += 1\n",
        "      if kw in keywordsOfEachDoc[i]:\n",
        "        rdf += 1\n",
        "\n",
        "    # Computes exclusivity score and add it to the list\n",
        "    if rdf > 0 :excs.append((kw, edf/rdf, textId))\n",
        "    else: excs.append((kw, -1, textId))\n",
        "\n",
        "  return excs"
      ],
      "metadata": {
        "cellView": "form",
        "id": "qTQmv3vkz293"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# @title generality\n",
        "def generality(allKeywords, topKeywords):\n",
        "  \"\"\"\n",
        "  Computes the list of generality score for each keywords.\n",
        "  Generality is computed as rdf * (1 - exc) = rdf * (1 - edf(k) / rdf(k))\n",
        "\n",
        "  Parameters:\n",
        "  allKeywords (dict{string: [list[int], float]}):\n",
        "    the dict of all keywords as that maps each keyword to its info list as\n",
        "    info_list[ list_of_doc_in_which_the_kw_occurred[], score_given_by_score_func ]\n",
        "\n",
        "  topKeywords (dict{string: [list[int], float]}):\n",
        "    the dict of TOP keywords as that maps each keyword to its info list as\n",
        "    info_list[ list_of_doc_in_which_the_kw_occurred[], score_given_by_score_func ]\n",
        "\n",
        "  Returns:\n",
        "  list[(string, float, list[list[int], score)]: list of generality score as (kw, gen_score, info)\n",
        "  \"\"\"\n",
        "  gens = []\n",
        "\n",
        "  for kw, infos in topKeywords.items():\n",
        "    edf = len(infos[0]) # infos[0] is the list of the documents in which the kw appears\n",
        "    rdf = len(allKeywords[kw][0])\n",
        "\n",
        "    # Computes exclusivity score used in generality score\n",
        "    if rdf > 0 :exc = edf/rdf\n",
        "    else: exc = 0\n",
        "\n",
        "    gens.append((kw, rdf * (1-exc), infos))\n",
        "\n",
        "  return gens"
      ],
      "metadata": {
        "cellView": "form",
        "id": "uA_W7DyC7KNA"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#  @title essentiality\n",
        "def essentiality(allKeywords, topKeywords):\n",
        "  \"\"\"\n",
        "  Computes the list of essentiality score for each keywords.\n",
        "  Essentiality is computed as exc(k) x edf(k)\n",
        "\n",
        "  Parameters:\n",
        "  allKeywords (dict{string: [list[int], float]}):\n",
        "    the dict of all keywords as that maps each keyword to its info list as\n",
        "    info_list[ list_of_doc_in_which_the_kw_occurred[], score_given_by_score_func ]\n",
        "\n",
        "  topKeywords (dict{string: [list[int], float]}):\n",
        "    the dict of TOP keywords as that maps each keyword to its info list as\n",
        "    info_list[ list_of_doc_in_which_the_kw_occurred[], score_given_by_score_func ]\n",
        "\n",
        "  Returns:\n",
        "  list[(string, float, list[list[int], score])]: list of essentiality score as (kw, ess_score, infos)\n",
        "  \"\"\"\n",
        "  ess = []\n",
        "\n",
        "  for kw, infos in topKeywords.items():\n",
        "    edf = len(infos[0]) # infos[0] is the list of the documents in which the kw appears\n",
        "    rdf = len(allKeywords[kw][0])\n",
        "\n",
        "    # Computes excelusivity score used in essentiality score\n",
        "    if rdf > 0 :exc = edf/rdf\n",
        "    else: exc = 0\n",
        "\n",
        "    ess.append((kw, exc * edf, infos))\n",
        "  return ess"
      ],
      "metadata": {
        "cellView": "form",
        "id": "fSabIoco_X4a"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Keywords retrieval\n",
        "\n",
        "Global function to exract the keywords of a corpus, using all previous function."
      ],
      "metadata": {
        "id": "V2BT7lsFxdCz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def listToDict(listToConvert):\n",
        "  resultingDict = {}\n",
        "  for item in listToConvert:\n",
        "    resultingDict[item[0]] = [item[1], item[2]]\n",
        "  return resultingDict\n",
        "\n",
        "def getKeywordsFromCorpus(corpus):\n",
        "  \"\"\"\n",
        "  Extract the most general and most essential keywords from a corpus.\n",
        "\n",
        "  Parameters:\n",
        "  corpus (list[string]): list of text documents\n",
        "\n",
        "  Returns:\n",
        "  dict{string: list[float, list[set(int), float]]}: dict of kw as {kw: infos}\n",
        "  \"\"\"\n",
        "  keys_dict = {}           # dict of all keywords\n",
        "  topKeys_dict = {}        # dict of top keywords\n",
        "  maxKeywords = 100\n",
        "\n",
        "  for textId in range(len(corpus)):\n",
        "    text = corpus[textId]\n",
        "\n",
        "    # Extract matrix and indexes of keywords for the current text\n",
        "    k, m, i = extractInfoKeywords(extractWords(text, textId))\n",
        "    # k = list of keywords\n",
        "    # m = matrices\n",
        "    # i= list of string (word-indexing list)\n",
        "\n",
        "    # Get, for each keyword, its score.\n",
        "    bestKeywordsForThisDoc = sorted(\n",
        "            score(m, k, i).items(),\n",
        "            key=lambda item: item[1],\n",
        "            reverse=True)\n",
        "\n",
        "    # Create the all keywords dict and top-keywords dict\n",
        "    numberOfKeywordsPassed = 0\n",
        "    for kw, score_value in bestKeywordsForThisDoc:\n",
        "      kw_string = ''.join(kw)\n",
        "      if kw_string in list(keys_dict.keys()):\n",
        "        if not textId in list(keys_dict[kw_string]):\n",
        "          keys_dict[kw_string][0].add(textId)\n",
        "          # Add the kw to top list if not full\n",
        "          if kw_string in topKeys_dict:\n",
        "            if numberOfKeywordsPassed < maxKeywords:\n",
        "              topKeys_dict[kw_string][0].add(textId)\n",
        "            else:\n",
        "              topKeys_dict[kw_string] = [set([textId]), score_value]\n",
        "            numberOfKeywordsPassed+=1\n",
        "\n",
        "      else:\n",
        "        keys_dict[kw_string] = [set([textId]), score_value]\n",
        "        # Add the kw to top list if not full\n",
        "        if kw_string in topKeys_dict:print(kw_string)\n",
        "        if numberOfKeywordsPassed < maxKeywords:\n",
        "          topKeys_dict[kw_string] = [set([textId]), score_value]\n",
        "          numberOfKeywordsPassed+=1\n",
        "\n",
        "  # Compute the generality score and essentiality score\n",
        "  genKeywords = generality(keys_dict, topKeys_dict)\n",
        "  essKeywords = essentiality(keys_dict, topKeys_dict)\n",
        "\n",
        "  # Gets the best overall keywords\n",
        "  #finalKeywords = bestKeywords(sorted(essKeywords, key=lambda item: item[2][1], reverse=True), sorted(genKeywords, key=lambda item: item[2][1], reverse=True))\n",
        "\n",
        "  return listToDict(sorted(genKeywords+essKeywords, key=lambda item:(item[1], item[2][1]), reverse = True)[:maxKeywords])\n",
        ""
      ],
      "metadata": {
        "id": "M1mLn7s5C7sG"
      },
      "execution_count": 55,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Final retrieval"
      ],
      "metadata": {
        "id": "YwK1KrE2mL7i"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "keywords = getKeywordsFromCorpus(documents)\n",
        "for k, v in keywords.items():\n",
        "  print(f\"doc {v[1][0]} : {k} {v}\")"
      ],
      "metadata": {
        "id": "Q_9qFcB-FPlg"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Relations of keywords (WIP)\n",
        "\n",
        "Two keywords are in relation if they appear in the same sentence.\n",
        "\n",
        "*For now, only works with one word keywords.*"
      ],
      "metadata": {
        "id": "4jueeYhhMmCC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import re\n",
        "\n",
        "def linkKeywords(keywords, phraseDelimiters, wordDelimiters):\n",
        "  \"\"\"\n",
        "  Links keywords if they appear in the same phrase.\n",
        "  A keyword is linked to the keywords list in the first element of the tuple,\n",
        "  which is a set.\n",
        "\n",
        "  Parameters:\n",
        "  keywords: dict{string: list[float, list[list[int], float]]}\n",
        "  phraseDelimiters: list[string]\n",
        "  wordDelimiters: list[string]\n",
        "\n",
        "  Returns:\n",
        "  dict{string: (set{string}, list[float, list[set[int]; float]])}\n",
        "  \"\"\"\n",
        "\n",
        "  conceptualLinks = {}\n",
        "  sentence = []\n",
        "  for document in documents:\n",
        "\n",
        "    # Regex pattern splits on word delimiters\n",
        "    for word in re.split('| '.join(wordDelimiters), document):\n",
        "      if any(ext in word for ext in phraseDelimiters):\n",
        "        # End of phrase => link all keywords of this phrase\n",
        "        for kw in sentence:\n",
        "          if not kw in keywords:\n",
        "            print(f\"no {kw} in dict\")\n",
        "            continue\n",
        "          copiedSentence = [_ for _ in sentence]\n",
        "          copiedSentence.remove(kw)\n",
        "          conceptualLinks[kw] = (set(copiedSentence), keywords[kw])\n",
        "      else:\n",
        "        if word in keywords.keys(): sentence.append(word)\n",
        "\n",
        "  return conceptualLinks\n",
        "\n",
        "keywords = getKeywordsFromCorpus(documents)\n",
        "keywordsLinks = linkKeywords(keywords, phraseDelimiters=phraseDelimiters, wordDelimiters=wordDelimiters)\n",
        "for k, v in keywordsLinks.items():\n",
        "  print(k, \" is linked to \", v[0])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-dcg9SRDMore",
        "outputId": "c01ae77a-1ed0-4ac0-fc88-4408557eefb7"
      },
      "execution_count": 110,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "methods  is linked to  {'methods'}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Relations of documents\n",
        "\n",
        "Two documents are in relation if they share keywords. This information is retrieved from the set that represents in which documents a keyword occurred."
      ],
      "metadata": {
        "id": "AG4RcRoFSXaj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import re\n",
        "\n",
        "def documentsLinking(keywords, numberOfDocuments):\n",
        "  \"\"\"\n",
        "  Links document if they share keywords. The result is a weigthed matrix.\n",
        "\n",
        "  Parameters:\n",
        "  keywords: dict{string: list[float, list[set[int], float]]}\n",
        "  numberOfDocuments: int\n",
        "\n",
        "  Returns:\n",
        "  list[list[int]]: weighted matrix of dimensions numberOfDocuments,numberOfDocuments\n",
        "  \"\"\"\n",
        "  documentsLinks = [[0 for _ in range(numberOfDocuments)] for _ in range(numberOfDocuments)]\n",
        "\n",
        "  for kw, infos in keywords.items():\n",
        "    source_doc = list(infos[1][0])[0]\n",
        "    for dest_doc in list(infos[1][0]): # add [1:] to remove the source doc\n",
        "      documentsLinks[source_doc][dest_doc] += 1\n",
        "  return documentsLinks\n",
        "\n",
        "  return documentsLinks"
      ],
      "metadata": {
        "id": "1VwQS4mLNG_B"
      },
      "execution_count": 105,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "keywords = getKeywordsFromCorpus(documents)\n",
        "documentsLinking(keywords, 3)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "d4_oomtdg2tk",
        "outputId": "924f7a7c-0efa-4610-bf10-ed8c3133b765"
      },
      "execution_count": 106,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[[13, 2, 2], [0, 32, 3], [0, 0, 55]]"
            ]
          },
          "metadata": {},
          "execution_count": 106
        }
      ]
    }
  ]
}