4-Vocabulary_Overlap.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Processing for Arabic - Vocabulary Overlap"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this exercise, we present a code snippet that reads two files and uses the vocabulary overlap to measure their similarity.\n",
    "\n",
    "The metric we are using is the weighted Jaccard similarity index, which is a variation of the Intersection over Union metric. We follow the formula described [here](https://en.wikipedia.org/wiki/Jaccard_index#Weighted_Jaccard_similarity_and_distance).\n",
    "\n",
    "If we have the following two files:\n",
    "\n",
    "File 1:\n",
    "```\n",
    "العقل السليم في الجسم السليم\n",
    "```\n",
    "\n",
    "File 2:\n",
    "```\n",
    "يستخدم سليم فرشاة الأسنان بالشكل السليم مرتين في اليوم\n",
    "```\n",
    "\n",
    "Applying the wighted Jaccard similarity index on the untokenized words:\n",
    "\n",
    "The word list:\n",
    "```\n",
    "['العقل','السليم','في','الجسم','يستخدم','سليم','فرشاة','الأسنان','بالشكل','مرتين','اليوم']\n",
    "```\n",
    "The word list frequencies in the first file:\n",
    "```\n",
    "[1,2,1,1,0,0,0,0,0,0,0]\n",
    "```\n",
    "The word list frequencies in the second file:\n",
    "```\n",
    "[0,1,1,0,1,1,1,1,1,1,1]\n",
    "```\n",
    "\n",
    "$Jaccard_{Weighted} = \\frac{\\sum minFreq_{(file1,file2)}}{\\sum maxFreq_{(file1,file2)}} = \\frac{0 + 1 + 1 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0}{1 + 2 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1} = \\frac{2}{12} = 16.67\\%$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "\n",
    "from collections import Counter \n",
    "from camel_tools.utils.normalize import normalize_unicode\n",
    "import csv\n",
    "\n",
    "# function to get a list of sentences from a raw file\n",
    "def get_sentences_from_raw(filename):\n",
    "    sentences = []\n",
    "\n",
    "    # Open the file for reading, assuming it is UTF-8 encoded\n",
    "    with open(filename, mode='r', encoding='utf8') as input_file:\n",
    "\n",
    "        # Iterate through every line in the file\n",
    "        for line in input_file:\n",
    "\n",
    "            # Normalize unicode characters\n",
    "            normalized_sentence = normalize_unicode(line)\n",
    "            \n",
    "            # Remove spaces/tabs/newlines at the beginning and end of the sentence\n",
    "            stripped_line = normalized_sentence.strip()\n",
    "\n",
    "            # Add the sentence to the existing list of sentences\n",
    "            sentences.append(stripped_line)\n",
    "\n",
    "    return sentences\n",
    "\n",
    "# function to generate different statistics\n",
    "def vocabulary_frequency(sentences):\n",
    "    \n",
    "    # initialize an empty list to store the word tokens\n",
    "    list_of_tokens = []\n",
    "    \n",
    "    for sentence in sentences:\n",
    "        \n",
    "        # for every sentence in the list, extract the words and append them to the list \n",
    "        list_of_tokens.extend(sentence.split())\n",
    "\n",
    "    # number of sentences\n",
    "    num_of_sentences = len(sentences) \n",
    "    \n",
    "    # number of words\n",
    "    num_of_tokens = len(list_of_tokens)\n",
    "    \n",
    "    # generate a histogram from the complete list of words (the frequency for each unique word)\n",
    "    histogram = Counter(list_of_tokens)\n",
    "    \n",
    "    # sort the histogram according to the highest frequency\n",
    "    sorted_histogram = {k: v for k, v in sorted(histogram.items(), key=lambda item: item[1], reverse=2)}\n",
    "    \n",
    "    return sorted_histogram, len(histogram)\n",
    "\n",
    "\n",
    "# This function calculates the vocabulary overlap between two files\n",
    "# It takes two files for input, and prints a short report\n",
    "def vocabulary_overlap(file_1, file_2):\n",
    "    \n",
    "    # Read the sentences from each of the files\n",
    "    file_1_sentences = get_sentences_from_raw(file_1)\n",
    "    file_2_sentences = get_sentences_from_raw(file_2)\n",
    "    \n",
    "    # Calculate the frequencies of words in each of the files \n",
    "    file_1_frequency, file_1_size = vocabulary_frequency(file_1_sentences)\n",
    "    file_2_frequency, file_2_size = vocabulary_frequency(file_2_sentences)\n",
    "            \n",
    "    min_sum = 0.0\n",
    "    max_sum = 0.0\n",
    "    \n",
    "    overlap_histogram = {}\n",
    "    file_1_unique_tokens = {}\n",
    "    file_2_unique_tokens = {}\n",
    "    \n",
    "    \n",
    "    # For every word that appears in the first file, calculate its intersection with the second file\n",
    "    # Simultaneously, calculate the union from all the words that appear in the first file\n",
    "    for word in file_1_frequency:\n",
    "        if word in file_2_frequency:\n",
    "            min_sum += min(file_1_frequency[word], file_2_frequency[word])\n",
    "            overlap_histogram[word] = min(file_1_frequency[word], file_2_frequency[word])\n",
    "            max_sum += max(file_1_frequency[word], file_2_frequency[word])\n",
    "        else:\n",
    "            max_sum += file_1_frequency[word]\n",
    "            file_1_unique_tokens[word] = file_1_frequency[word]\n",
    "    \n",
    "    # Add to the union the words that only appear in the second file\n",
    "    for word in file_2_frequency:\n",
    "        if word not in file_1_frequency:\n",
    "            max_sum += file_2_frequency[word]\n",
    "            file_2_unique_tokens[word] = file_2_frequency[word]\n",
    "\n",
    "    # Calculate the wighted Jaccard similarity\n",
    "    jaccard_similarity = (min_sum / max_sum) * 100\n",
    "\n",
    "    \n",
    "    print('File 1: ' + file_1)\n",
    "    print('# of lines in the first file = ' + str(len(file_1_sentences)))\n",
    "    print('# of tokens in the first file = ' + str(file_1_size))\n",
    "    print('# of unique tokens in the first file = ' + str(len(file_1_frequency)) + '\\n')\n",
    "\n",
    "    print('File 2: ' + file_2)\n",
    "    print('# of lines in the second file = ' + str(len(file_2_sentences)))\n",
    "    print('# of tokens in the second file = ' + str(file_2_size))\n",
    "    print('# of unique tokens in the second file = ' + str(len(file_2_frequency)) + '\\n')\n",
    "                             \n",
    "    print('Overlap similarity between the two files = ' + \"{0:.2f}\".format(jaccard_similarity) + '%')\n",
    "    \n",
    "    sorted_overlap_histogram = {k: v for k, v in sorted(overlap_histogram.items(), key=lambda item: item[1], reverse=2)}\n",
    "    print('\\nTop 10 most shared tokens:')\n",
    "    for word in list(sorted_overlap_histogram)[0:10]:\n",
    "        print('\\t{}\\t{}'.format(word, sorted_overlap_histogram[word]))\n",
    "    \n",
    "    # write histogram to file:\n",
    "    with open(file_1 + '_' + file_2.split('/')[-1] +'.overlap.tsv', 'w', encoding='utf-8', newline='') as csvfile:\n",
    "        \n",
    "        # create a writer object\n",
    "        row_writer = csv.writer(csvfile, dialect='excel-tab')\n",
    "        \n",
    "        # write the header of the table\n",
    "        row_writer.writerow(['Token', 'Freq'])\n",
    "        for word in sorted_overlap_histogram:\n",
    "            # write the rows (row by row)\n",
    "            row_writer.writerow([word, sorted_overlap_histogram[word]])\n",
    "    print('A complete historgram of the overlapped words is written to \\'{}.overlap.tsv\\''.format(file_1 + '_' + file_2.split('/')[-1]))\n",
    "        \n",
    "        \n",
    "    sorted_file_1_unique_list = {k: v for k, v in sorted(file_1_unique_tokens.items(), key=lambda item: item[1], reverse=2)}\n",
    "    print('\\nTop 10 most frequent unique tokens in file 1:')\n",
    "    for word in list(sorted_file_1_unique_list)[0:10]:\n",
    "        print('\\t{}\\t{}'.format(word, sorted_file_1_unique_list[word]))\n",
    "        \n",
    "    \n",
    "    # write histogram to file:\n",
    "    with open(file_1 +'.unique.tsv', 'w', encoding='utf-8', newline='') as csvfile:\n",
    "        \n",
    "        # create a writer object\n",
    "        row_writer = csv.writer(csvfile, dialect='excel-tab')\n",
    "        \n",
    "        # write the header of the table\n",
    "        row_writer.writerow(['Token', 'Freq'])\n",
    "        for word in sorted_file_1_unique_list:\n",
    "            # write the rows (row by row)\n",
    "            row_writer.writerow([word, sorted_file_1_unique_list[word]])\n",
    "    print('A complete historgram of the unique words in file 1 is written to \\'{}.unique.tsv\\''.format(file_1))\n",
    "        \n",
    "    \n",
    "    sorted_file_2_unique_list = {k: v for k, v in sorted(file_2_unique_tokens.items(), key=lambda item: item[1], reverse=2)}\n",
    "    print('\\nTop 10 most frequent unique tokens in file 2:')\n",
    "    for word in list(sorted_file_2_unique_list)[0:10]:\n",
    "        print('\\t{}\\t{}'.format(word, sorted_file_2_unique_list[word]))\n",
    "    \n",
    "    \n",
    "    # write histogram to file:\n",
    "    with open(file_2 +'.unique.tsv', 'w', encoding='utf-8', newline='') as csvfile:\n",
    "        \n",
    "        # create a writer object\n",
    "        row_writer = csv.writer(csvfile, dialect='excel-tab')\n",
    "        \n",
    "        # write the header of the table\n",
    "        row_writer.writerow(['Token', 'Freq'])\n",
    "        for word in sorted_file_2_unique_list:\n",
    "            # write the rows (row by row)\n",
    "            row_writer.writerow([word, sorted_file_2_unique_list[word]])\n",
    "    print('A complete historgram of the unique words in file 2 is written to \\'{}.unique.tsv\\''.format(file_2))\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To run this code, you need to call the function `vocabulary_overlap` with two paramters that are the two files that you are comparing. Try to call the function with different files."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vocabulary Overlap: Gigaword_AR D3 and ATB tokenizations as an example:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The call to the function is as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Choose two files you want to compare, and change the parameters to be those files\n",
    "file_1_name = 'Results/Gigaword_AR/gigaword_tiny_cleaned.txt.D3.tok'\n",
    "file_2_name = 'Results/Gigaword_AR/gigaword_tiny_cleaned.txt.ATB.tok'\n",
    "\n",
    "vocabulary_overlap(file_1_name, file_2_name)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}