COVID19_voc_processing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# COVID-19 Variant of Concern (VOC) Lab Results\n",
    "Written by: Branson Chen, Sina Brar <br> \n",
    "Last modified: 20210429"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of Contents\n",
    "\n",
    "<a href='#Overview'>Overview</a><br>\n",
    "<a href='#Input-variables'>Input variables</a><br>\n",
    "<a href='#Importing-data'>Importing data</a><br>\n",
    "<a href='#Text-analysis'>Text analysis</a><br>\n",
    "\n",
    "- <a href='#Algorithm-description'>Algorithm description</a><br>\n",
    "- <a href='#Initial-processing'>Initial processing</a><br>\n",
    "- <a href='#Assign-results'>Assign results</a><br>\n",
    "\n",
    "<a href='#Final-output'>Final output</a><br>\n",
    "<a href='#Roll-Up'>Roll Up</a><br>\n",
    "<a href='#Manual-review'>Manual review</a><br>\n",
    "<a href='#Testing-and-validation'>Testing and validation</a><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- This script first imports a SAS dataset based on the input variables provided, and then fields are decoded/renamed.\n",
    "- <strong> The SAS dataset is created by taking any records under two TR Codes: TR12952-8 (VOC screening), TR12953-6 (VOC sequencing) and any records that contain the strings: '(VOC', ' VOC', or both 'VARIANT' and 'SARS' </strong>\n",
    "- Next, the text is cleaned (clean function) and then tokenized (tokenize function).\n",
    "- Relevant labels are then assigned to the tokens (assign_labels function).\n",
    "- The labelled tokens are then interpreted using an in-house algorithm (interpret function).\n",
    "- All of the information from the previous step is then collapsed to give one result per virus per test (process_result function), and unidentified virus/test types are filled in based on observation codes and testrequest codes.\n",
    "- The results are converted to a single character per test type (char_output function) and then output in a csv.\n",
    "\n",
    "\n",
    "- The second part of the script collapses the final output from the previous section into TESTING EPISODES (unique patientid+observationdate). \n",
    "- First, exclusions are applied to remove observations with resultstatus = N/X/W and observations with missing patientid.\n",
    "- Records with no results are removed and they are identified by the temporary scr_flag and seq_flag columns.\n",
    "- Flags (scr_test, seq_test) are created for each test type to identify records where there is at least one clear result (P, N, I, D).\n",
    "- Observations (with a patientid) are rolled-up to TEST REQUESTS and then TESTING EPISODES:\n",
    "    - For each TEST REQUEST among each result column, select the result by prioritizing clear results (P,N,I,D), then the latest release timestamp, and then the result hierarchy (P>N>I>D>C>R>blank).\n",
    "    - For each TESTING EPISODE among each result column, select the result by using the result hierarchy (P>N>I>D>C>R>blank).\n",
    "- For each test type, an observation release date (observationreleasedate_scr, observationreleasedate_seq) is specified by taking the latest release date with a clear result."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Input variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#input path and filename (should be .sas7bdat file)\n",
    "input_path = '//'\n",
    "\n",
    "#Change the date of input file if needed\n",
    "input_filename = '.sas7bdat'\n",
    "\n",
    "#name of patientid variable in input dataset, will be renamed as 'patientid'\n",
    "input_patientid_var = 'ikn'\n",
    "\n",
    "#output additional columns\n",
    "#1 = with key columns, 2 = with ALL columns\n",
    "output_flag = 1\n",
    "\n",
    "#output filename\n",
    "output_filename = 'output_voc'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "#import sas file (COMPRESS=BINARY MAY NOT WORK WITH READ_SAS; COMPRESS=YES|CHAR DOES NOT WORK WITH READ_SAS)\n",
    "df_raw=pd.read_sas(input_path+input_filename)\n",
    "\n",
    "#decode strings (np objects)\n",
    "df_raw.loc[:, df_raw.dtypes == np.object] = df_raw.loc[:, df_raw.dtypes == np.object].apply(lambda x: x.str.decode('UTF-8'))\n",
    "df_raw.fillna('', inplace=True)\n",
    "print('# of records:',len(df_raw))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "df = df_raw.copy(deep = True)\n",
    "\n",
    "#rename variables\n",
    "df = df.rename(columns={input_patientid_var:'patientid','fillerordernumber':'fillerordernumberid',\n",
    "                       'observationvalue':'value','observationsubid':'subid'})\n",
    "#keep key cols\n",
    "key_cols = ['patientid', 'ordersid', 'fillerordernumberid', \n",
    "            'reportinglaborgname', 'performinglaborgname', 'observationdatetime','specimenreceiveddatetime',\n",
    "            'testrequestcode', 'observationcode', 'observationreleasets', \n",
    "            'observationresultstatus', 'subid', 'value']\n",
    "df = df[key_cols]\n",
    "\n",
    "#set exclude_flag based on observationresultstatus = W\n",
    "df_W = df.loc[df['observationresultstatus'] == 'W', ['ordersid', 'observationcode', 'value']]\n",
    "df_excl = df[['ordersid', 'observationcode', 'value']].reset_index().merge(df_W, how='inner').set_index('index')\n",
    "df['exclude_flag'] = 'N'\n",
    "df.loc[df.index.isin(df_excl.index),['exclude_flag']] = 'Y'\n",
    "print(df['exclude_flag'].value_counts())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "#determine which observations need to be concatenated\n",
    "group_cols = ['ordersid', 'fillerordernumberid', 'reportinglaborgname', \n",
    "              'testrequestcode', 'observationcode', 'observationreleasets', 'observationresultstatus']\n",
    "df_gp_subid = df.reset_index().groupby(group_cols).agg({'index':tuple, 'subid':tuple}).reset_index()\n",
    "df_gp_subid = df_gp_subid.rename(columns={'index':'original_indexes'})\n",
    "\n",
    "#only concatenate ones where there are more than two subids, all the subids are numbers and contains 1\n",
    "df_to_concat = df_gp_subid[df_gp_subid['subid'].apply(lambda x: all([subid.isdigit() for subid in x]) and len(x) > 2 and '1' in x)]\n",
    "concat_indexes = [i for tup in df_to_concat['original_indexes'] for i in tup]\n",
    "\n",
    "#concatenate based on subid\n",
    "df_gp_concat = df[df.index.isin(concat_indexes)].reset_index()\n",
    "df_gp_concat['subid'] = df_gp_concat['subid'].apply(int)\n",
    "df_gp_concat = df_gp_concat.sort_values(by = group_cols+['subid']).groupby(group_cols)\n",
    "df_gp_concat = df_gp_concat.agg({'index': tuple,\n",
    "                   'value': lambda x: ' '.join(map(str, x))}).reset_index()\n",
    "\n",
    "#add on records that were not concatenated\n",
    "df_gp = df.loc[~df.index.isin(concat_indexes), group_cols+['value']].reset_index()\n",
    "df_gp['index'] = df_gp['index'].apply(lambda x: (x,))\n",
    "df_gp = pd.concat([df_gp_concat, df_gp], sort=False).rename(columns={'index':'original_indexes'})\n",
    "\n",
    "#narrow down columns of df\n",
    "df_cols = ['patientid','ordersid','fillerordernumberid',\n",
    "           'observationdatetime','specimenreceiveddatetime','testrequestcode',\n",
    "           'observationcode','observationreleasets', 'observationresultstatus','exclude_flag']\n",
    "df = df[df_cols]\n",
    "\n",
    "print('# of TEST RESULTS:', len(df_gp))\n",
    "\n",
    "#cleanup\n",
    "del df_W\n",
    "del df_excl\n",
    "del df_gp_subid\n",
    "del df_to_concat\n",
    "del concat_indexes\n",
    "del df_gp_concat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#clean punctuation, xml field, numbers, other text\n",
    "puncs = [';', ':', ',', '.', '-', '_', '/', '(', ')', '[', ']', '{', '}', '<', '>', '*', '#', '?', '.', '+', \n",
    "        'br\\\\', '\\\\br', '\\\\e\\\\', '\\\\f\\\\', '\\\\t\\\\', '\\\\r\\\\', '\\\\', \"'\", '\"', '=']\n",
    "terms_to_space = ['detected', 'by', 'positive', 'parainfluenza', 'accession']\n",
    "nums_following = ['date', 'telephone', 'tel', 'phone', 'received', 'collected',  \n",
    "                 'result', 'on', 'at', '@', 'approved', 'final', 'time', 'number']\n",
    "strings_to_replace = {'non detected':'not detected', 'npot detected':'not detected', \n",
    "                      'nor detected':'not detected', 'mot detected':'not detected', \n",
    "                      'n0t detected':'not detected', 'nit detected':'not detected',\n",
    "                      'covid 19 virus not interpretation detected':'covid 19 virus interpretation not detected',\n",
    "                      'presumptive interpretation':'interpretation presumptive',\n",
    "                      'preliminary interpretation':'interpretation preliminary',\n",
    "                      'covid 19 not detected and covid 19 detected':'covid 19 detected and covid 19 not detected',\n",
    "                      'virusnot':'virus not', 'prevuous':'previous',\n",
    "                      'mutaion': 'mutation', 'muation':'mutation','dectection':'detection',\n",
    "                      'sars coc': 'sars cov', 'cov @': 'cov 2', 'cov2': 'cov 2', '2voc': '2 voc',\n",
    "                      'covid 19': 'sars cov 2',\n",
    "                      'u k':'uk', '2uk': '2 uk', \n",
    "                      'e4874k': 'e484k', 'e484 k': 'e484k', ' 484k s gene': ' e484k s gene',\n",
    "                      'b 1 1 7':'b117', 'b 1 351':'b1351',\n",
    "                      'p 1':'p1','p 2':'p2',\n",
    "                      'n501 y':'n501y', 'n5oy 1':'n501y', 'n5oy1':'n501y', '5o':'50', 'n51y':'n501y', \n",
    "                      'n5g1y':'n501y', ' 501y s gene':' n501y s gene',\n",
    "                      '501y v2':'501yv2', '20i 501y v1':'20i501yv1','20b 501y v3':'20b501yv3',\n",
    "                      'voc202012 01':'voc20201201', 'voc202012 02':'voc20201202', 'voc202101 02':'voc20210102',\n",
    "                      'voc 202012 01':'voc20201201', 'voc 202012 02':'voc20201202', 'voc 202101 02':'voc20210102',\n",
    "                      'varieant':'variant', 'vriant':'variant',\n",
    "                      'wtih ':'with ', 'wih ':'with ', 'associated wit ':'associated with ',\n",
    "                      'dtecteion':'detection', 'dtected':'detected',\n",
    "                      'deteected':'detected', 'detetcetd':'detected', 'dotected':'detected'\n",
    "            \n",
    "                     }\n",
    "\n",
    "date_id_patterns = [r'\\d{2,4} \\d{2} \\d{2,4} ', r'\\d{4} \\d{2} ', r'\\d{4}h ', \n",
    "                   r' \\d{0,2}[a-z]{0,2}\\d{5,}[a-z]{0,1}', r' [a-z]{0,2}\\d{1,3}[a-z]{1,3}\\d{4,}[a-z]{0,1}',\n",
    "                   r' \\d{2}[a-z]{1}\\d{3}[a-z]{2}\\d{4}', r' [a-z]{4,}\\d{7,}']\n",
    "\n",
    "def clean(value):\n",
    "    cleaned = value.lower()\n",
    "\n",
    "    #clean xml field, only keep text field surrounded with 'p1 text'\n",
    "    pattern = r'(<p1:microorganism xmlns)(.+)(<p1:text>.+</p1:text>)(.+)(</p1:microorganism>)'\n",
    "    while re.search(pattern, cleaned):\n",
    "        cleaned = re.sub(pattern, r'\\g<3>', cleaned)\n",
    "    \n",
    "    #surround terms with spaces (some terms found stuck together)\n",
    "    for t in terms_to_space:\n",
    "        cleaned = cleaned.replace(t, ' ' + t + ' ')\n",
    "    \n",
    "    #replace punctuation with space\n",
    "    for punc in puncs:\n",
    "        cleaned = cleaned.replace(punc, ' ')\n",
    "\n",
    "    #remove consecutive spaces\n",
    "    while '  ' in cleaned:\n",
    "        cleaned = cleaned.replace('  ', ' ')\n",
    "    \n",
    "    cleaned = cleaned.strip()     \n",
    "    \n",
    "    #remove numbers after certain terms\n",
    "    for term in nums_following:\n",
    "        pattern = term + r' \\d{1,4}'\n",
    "        \n",
    "        while re.search(pattern, cleaned):\n",
    "            cleaned = re.sub(pattern, term, cleaned)\n",
    "\n",
    "    #fix certain strings\n",
    "    for k, v in strings_to_replace.items():\n",
    "        cleaned = cleaned.replace(k, v)       \n",
    "        \n",
    "    #remove more dates and ids\n",
    "    for pattern in date_id_patterns:\n",
    "        while re.search(pattern, cleaned):\n",
    "            cleaned = re.sub(pattern, '', cleaned)\n",
    "    \n",
    "    #remove numbers at the end, but exclude variant terms\n",
    "#     while len(cleaned) > 0 and (cleaned[-1].isdigit() or cleaned[-1] == ' ') and \\\n",
    "#         not (cleaned[-4:] == 'b117' or cleaned[-5:] == 'b1351' or cleaned[-6:] in ('501yv1','501yv2','501yv3')\n",
    "#                 or cleaned[-4:] in ('voc20201201','voc20201202','voc20210102') ) :\n",
    "#         cleaned = cleaned[:-1]\n",
    "    \n",
    "    #remove \"no\" at the end\n",
    "    while cleaned.endswith(' no') or cleaned == 'no':\n",
    "        cleaned = cleaned[:-3]\n",
    "    \n",
    "    return cleaned"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#tokenize values using nltk\n",
    "def tokenize(value):\n",
    "    tokenized = nltk.word_tokenize(value)\n",
    "   \n",
    "    return tokenized"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#assign labels for useful tokens based on some dictionaries and exclusions\n",
    "voc_dict = {'v_sgene_n501y': ['n501y'],\n",
    "            'v_sgene_e484k': ['e484k'],\n",
    "            'v_sgene_k417n': ['k417n'],\n",
    "            'v_sgene_k417t': ['k417t'],\n",
    "            'v_voc_b117': ['uk', 'b117', 'voc20201201', '20i501yv1'],\n",
    "            'v_voc_b1351': ['south', 'africa', 'african','b1351','501yv2','voc20201202'],\n",
    "            'v_voc_p1': ['p1','brazil', 'brazilian','voc20210102','20b501yv3'],\n",
    "            'v_voc_p2': [],\n",
    "             }\n",
    "\n",
    "indirect_matches_dict = {'r_pos': ['posi','pos1','covpos'], \n",
    "                         'r_neg': ['neg'],  \n",
    "                         'r_ind': ['indeter', 'eterminate', 'inconclu', 'inderter', 'unable', 'inval',\n",
    "                                   'equivocal', 'unresolved'],\n",
    "                         'r_can': ['cancel', 'incorrect', 'duplicate', 'mislabel', 'recollect','mistaken','redirected'],\n",
    "                         'r_rej': ['reject', 'leak', 'insuffic', \n",
    "                                   'spill', 'inapprop', 'nsq', 'poor', 'uninterpret'],\n",
    "                         'presumptive': ['presump', 'prelim', 'possi']}\n",
    "direct_matches_dict = {'r_pos': ['detected', 'pos', 'deteced', 'postive', 'organism','isolated', 'evidence'],\n",
    "                       'r_neg': ['no', 'not'],\n",
    "                       'r_ind': ['ind'],\n",
    "                       'r_pen': ['pending','progress','follow','ordered','reordered','referred'], #'sent', 'send', 'forward'\n",
    "                       'presumptive': ['possible', 'probable'],\n",
    "#                        'xml': ['p1'], \n",
    "                       'stop': ['specific', 'required', 'error', 'copy', 'see', 'laboratory',\n",
    "                                'note', 'notes', 'stability', 'changed', 'recollect', 'moh', 'if'],\n",
    "                       'final': ['interpretation', 'interpetation', 'interp', 'pretation', 'interpretive',\n",
    "                                 'final', 'overall', 'corrected', 'proved', 'correct','current'],\n",
    "                       'connecting': ['presence', 'as',\n",
    "                                      'is', 'of', 'in', '1', '2', '3', '4', 'a', 'b', 'c',\n",
    "                                      '229e', 'nl63', 'hku1', 'oc43', '2019', 'low',\n",
    "                                      'biosafety', 'hazard', 'has', 'been', 'for', 'changed', 'identified', \n",
    "                                      'result', 'other', 'using', 'to', 'from', 'tested',\n",
    "                                      'phl', 'phol', 'phlo', 'new', 'request', 'lab', 'will',\n",
    "                                      'panel', 'seasonal', 'human', 'report', 'said', 'updated',\n",
    "                                      'associated', 'with', 'associate', 'vocs', 'entry',\n",
    "                                      'voc','variant','concern', 'mutation','detection','characterization',\n",
    "                                      'complete', 'rt','time','pcr'] #'lineage?','emerged','analysis'\n",
    "                                      } \n",
    "test_type_dict = {'t_scr':['snp', 'screening','screen','screened','screens'], #rt,pcr, 'seegene', 'allplex',\n",
    "                  't_seq':['genomic', 'analysis', 'sequencing','sequenced','sequence']} #genome\n",
    "\n",
    "def assign_labels(tokenized):\n",
    "    tokenized_length = len(tokenized)\n",
    "    useful = [None]*tokenized_length #store same list length of tokens and update each accordingly\n",
    "    \n",
    "    for counter, token in enumerate(tokenized):\n",
    "        #skip if already assigned\n",
    "        if useful[counter]:\n",
    "            continue\n",
    "        \n",
    "        ###easy viruses dictionary (non-exact matching)\n",
    "        ## mutation/voc dict\n",
    "        for virus, patterns in voc_dict.items():\n",
    "            if any([pattern in token for pattern in patterns]):\n",
    "                useful[counter] = virus\n",
    "                break\n",
    "        \n",
    "        # VOC/variant of concern at beginning or preceded by interpretation, treat as virus term\n",
    "        if token in ('variant','variants') and (tokenized_length > counter+3)\\\n",
    "        and tokenized[counter+1:counter+3] == ['of','concern']\\\n",
    "        and (counter == 0 or tokenized[counter-1] in ('interpretation','covid19')):\n",
    "            useful[counter:counter+3] = ['v_voc_general', 'connecting', 'connecting']\n",
    "                  \n",
    "        # elif useful[counter].startwith('v_sgene')  \n",
    "        elif token in ('n501y','e484k','k417n','k417t') and (tokenized_length > counter+3)\\\n",
    "        and tokenized[counter+1:counter+3] in (['s','gene'],\n",
    "                                               ['spike','gene'],\n",
    "                                               ['spike','s']):\n",
    "            useful[counter+1:counter+3] = ['connecting']*2  \n",
    "        elif token in ('n501y','e484k','k417n','k417t') and (tokenized_length > counter+3)\\\n",
    "        and tokenized[counter+1:counter+3] in (['uk','variant'],\n",
    "                                               ['b117','variant'],\n",
    "                                               ['brazil', 'variant'],\n",
    "                                               ['brazilian', 'variant'],\n",
    "                                               ['b1351', 'variant'],\n",
    "                                               ['south', 'africa'],\n",
    "                                               ['south', 'african']\n",
    "                                               ):\n",
    "            useful[counter+1:counter+3] = ['connecting']*2\n",
    "        \n",
    "        # sgene\n",
    "        elif token in ('s', 'spike') and (tokenized_length > counter+1)\\\n",
    "        and tokenized[counter+1] == 'gene':\n",
    "            useful[counter:counter+2] = ['v_sgene_mutation', 'connecting']\n",
    "        elif token in ('sars') and (tokenized_length > counter+5)\\\n",
    "        and tokenized[counter+1:counter+5] in (['cov', '2', 'gene', 'mutation'],):\n",
    "            useful[counter:counter+5] = ['v_sgene_mutation','connecting', 'connecting','connecting','connecting']\n",
    "        \n",
    "        #SARS COV 2\n",
    "        elif tokenized[counter:counter+7] == ['sars','cov','2','n501y','single','nucleotide','polymorphism']:\n",
    "            useful[counter:counter+7] = ['connecting']*7\n",
    "        elif tokenized[counter:counter+4] == ['sars','cov','2','virus']:\n",
    "            useful[counter:counter+4] = ['v_covid']*4\n",
    "        elif tokenized[counter:counter+3] == ['sars','cov','2']:\n",
    "            useful[counter:counter+3] = ['v_covid']*3\n",
    "            \n",
    "        # SARS COV 2 VARIANT(S)/VOC/VARIANT(S) OF CONCERN\n",
    "        elif token in ('variant','voc','variants','vocs') and tokenized[counter-3:counter] == ['sars','cov','2']\\\n",
    "        and (tokenized_length > counter+1) and tokenized[counter+1] not in ('s','testing') \\\n",
    "        and tokenized[counter-5:counter-3] != ['associated','with']:\n",
    "            useful[counter-3:counter]=['v_voc_general','connecting','connecting']\n",
    "        \n",
    "        # p1 voc emerged in the uk \n",
    "        elif token in ('p1',) and (tokenized_length > counter+6)\\\n",
    "        and tokenized[counter+1:counter+6] in (['voc','emerged', 'in', 'the', 'uk'],):\n",
    "            useful[counter:counter+5] = ['v_voc_p1','connecting', 'connecting','connecting','connecting','connecting']\n",
    "        \n",
    "        elif tokenized[counter:counter+2] == ['p1','text']:\n",
    "            useful[counter:counter+2] = [None]*2\n",
    "            \n",
    "        elif tokenized[counter:counter+2] == ['this','specimen']:\n",
    "            useful[counter:counter+2] = ['stop']*2\n",
    "        \n",
    "        elif token in ('results','specimen'):\n",
    "            useful[counter] = 'v_unk'\n",
    "            \n",
    "    # loop over the record again\n",
    "    for counter, token in enumerate(tokenized):\n",
    "        #skip if already assigned\n",
    "        if useful[counter]:\n",
    "            continue\n",
    "        \n",
    "        #two mutations\n",
    "        elif token == 'or' and (tokenized[counter-5:counter-2] == ['unable','to','screen'] \n",
    "                                and useful[counter-1] is not None\n",
    "                                and useful[counter-1].startswith('v_')):\n",
    "            useful[counter] = 'r_ind'\n",
    "        elif token == 'and' and ('detected' in tokenized[counter+10:counter+12]+tokenized[counter+17:counter+19]\n",
    "                                 and 'not' not in tokenized[counter+9:counter+11] +tokenized[counter+16:counter+18]\n",
    "                                 and useful[counter+1] is not None\n",
    "                                 and useful[counter+1].startswith('v_')):\n",
    "            useful[counter] = 'r_pos'\n",
    "        elif token == 'and' and ('detected' in tokenized[counter-4:counter-2]\n",
    "                                 and useful[counter+1] is not None\n",
    "                                 and useful[counter+1].startswith('v_')):\n",
    "            useful[counter] = 'r_pos'    \n",
    "        elif token == 'and' and (tokenized[counter+11:counter+13] == ['not','detected']\n",
    "                                 and useful[counter+1] is not None\n",
    "                                 and useful[counter+1].startswith('v_')):  \n",
    "            useful[counter] = 'r_neg'\n",
    "            \n",
    "        #two VOCs\n",
    "        elif token == 'or' and ('presumptively' in tokenized[counter+4:counter+6]\n",
    "                                and useful[counter-1] is not None\n",
    "                                and useful[counter-1].startswith('v_')):\n",
    "            useful[counter:counter+2] = ['presumptive','r_pos']\n",
    "            \n",
    "        #condition for mention of pos/neg\n",
    "        elif token in ('negative','neg','positive','pos','organism','detected')\\\n",
    "        and (tokenized_length > counter+1)\\\n",
    "        and ((tokenized[counter-1] in ('a','original','or','level','of','the','tested','was','false')\n",
    "              and tokenized[counter+1] in ('test','result','covid','new','at'))\n",
    "             or tokenized[counter+1] in ('or','swab','to','contact','workers','retest',\n",
    "                                         'son','person','patients','travel','individual','undergo')):\n",
    "            useful[counter-1:counter+2] = [None]*3\n",
    "        elif token in ('negative','neg','positive','pos','organism','detected','posivtive')\\\n",
    "        and (tokenized_length > 1)\\\n",
    "        and (tokenized[counter-2] in ('previous','previously','contact','worker','depot','targets',\n",
    "                                      'being','unless','patient','law','due','exposure','needs','if',\n",
    "                                      'swab','who')\n",
    "             or tokenized[counter-1] in ('previous','previously','known','unit','first','second',\n",
    "                                         'needs','need','requires','considered','swab','if',\n",
    "                                         'depot','employee','gram','cx','member','coworker','shows',\n",
    "                                         'father','contact','both','and')):\n",
    "            useful[counter-1:counter+1] = [None]*2\n",
    "        elif token in ('negative','neg','positive','pos','organism','detected')\\\n",
    "        and (tokenized_length > 2)\\\n",
    "        and (tokenized[counter-3] in ('mom','him','father')):\n",
    "            useful[counter-2:counter+1] = [None]*3 \n",
    "            \n",
    "        #condition for word before no\n",
    "        elif token == 'no' and (tokenized[counter-1] in ('as','by','lab','specimen','accession',\n",
    "                                                         'sample','order','please','phl')\n",
    "                                or any([pattern in tokenized[counter-1] for pattern in ('out','break','inv')]))\\\n",
    "        and tokenized[counter+1:counter+2] != ['virus'] and counter > 0:\n",
    "            useful[counter-1:counter+1] = [None]*2\n",
    "        \n",
    "        #condition for word after no (cancel)\n",
    "        elif token == 'no' and (tokenized_length > counter+1)\\\n",
    "        and tokenized[counter+1] in ('specimen','reportable','done','gene','result',\n",
    "                                     'media','liquid','sample','swab','nasopharyngeal','record','fluid',\n",
    "                                     'patient','second','results','testing','eluate','option','chose',\n",
    "                                     'speicmen','label','validated','culture',):\n",
    "            useful[counter] = 'r_can'\n",
    "\n",
    "        #condition for due to\n",
    "        elif tokenized[counter:counter+2] == ['due','to'] and 'new' not in tokenized[counter+2:counter+4]:\n",
    "            useful[counter:counter+2] = ['stop']*2\n",
    "                \n",
    "        #condition for word after not (cancel)\n",
    "        elif token == 'not' and (tokenized_length > counter+1) and \\\n",
    "        tokenized[counter+1] in ('tested','tessted','perform','performed','process','processed', \n",
    "                                 'transmit','suitable','done','doen','reported','received',\n",
    "                                 'match','needed','labelled','available','symptomatic','forwared',\n",
    "                                 'met','specified','indicated','returned','sufficient',\n",
    "                                 'valid','required','able','needed','contain','ordered','recieved',\n",
    "                                 'labeled','a','provided','appropriate','sent','send','remove',\n",
    "                                 'report','rapid','found','applicable','rec','used','order',\n",
    "                                 'matched','labled','proccessed','accepted','receivd','completed',\n",
    "                                 'recollect','preformed','appearing','in','collected','obtained',\n",
    "                                 'reported','successfully'):\n",
    "            useful[counter:counter+2] = ['r_can']*2\n",
    "        \n",
    "        #condition for word before not\n",
    "        elif token == 'not' and tokenized[counter-1] in ('does','did','please','done','over','swab','but','do','has'):\n",
    "            useful[counter-1:counter+1] = ['reset']*2\n",
    "        \n",
    "        #condition for errors\n",
    "        elif tokenized[counter:counter+3] in (['ordered', 'in', 'error'], ['no', 'covid', 'result']):\n",
    "            useful[counter:counter+3] = ['r_can']*3\n",
    "\n",
    "        # condition for note\n",
    "        elif tokenized[counter:counter+2] in (['note','specimens'],):\n",
    "            useful[counter:counter+2] = ['end']*2        \n",
    "        \n",
    "        #condition for previous\n",
    "        elif 'previous' in token and ('reported' in tokenized[counter+1:counter+3] or\n",
    "                                      'specimen' in tokenized[counter+1:counter+2] or\n",
    "                                      (tokenized[counter+1:counter+3] == ['report','covid'] and\n",
    "                                           tokenized[counter-1] == 'the') or\n",
    "                                      tokenized[counter+1:counter+3] in (['report','of'],\n",
    "                                                                         ['reports','of'],\n",
    "                                                                         ['reportof','covid'],\n",
    "                                                                         ['result','of'],\n",
    "                                                                         ['covid','19'],\n",
    "                                                                         ['entered','covid'],\n",
    "                                                                         ['report','as'],\n",
    "                                                                         ['result','was'],\n",
    "                                                                         ['report','that'])):\n",
    "            useful[counter:counter+2] = ['end']*2\n",
    "                \n",
    "        #unable and no evidence - indeterminate\n",
    "        elif tokenized[counter:counter+3] in (['unable','to','complete'],\n",
    "                                              ['unable','to','be'],\n",
    "                                              ['not','be','performed']):\n",
    "            useful[counter:counter+3] = ['r_ind','v_unk','connecting']\n",
    "        elif tokenized[counter:counter+4] in (['unable','to','generate','sars'],):\n",
    "            useful[counter:counter+4] = ['r_ind','connecting','t_seq','v_voc_general']    \n",
    "        \n",
    "        elif tokenized[counter:counter+6] == ['as','well','as','wild','type','virus']:\n",
    "            useful[counter:counter+6] = ['connecting']*6\n",
    "        \n",
    "        #SNP screen\n",
    "        elif tokenized[counter:counter+3] in (['single','nucleotide','polymorphism'],\n",
    "                                              ['mutation','associated','with'],\n",
    "                                              ['mutations','associated','with']\n",
    "                                             ):\n",
    "            useful[counter:counter+3] = ['t_scr']*3\n",
    "        \n",
    "        #sequencing instead of screening\n",
    "        elif tokenized[counter:counter+7] == ['variant','screening','was','performed','by','sanger','sequencing'] \\\n",
    "                    and (tokenized_length > counter+1):\n",
    "            useful[counter+1] = 'connecting'\n",
    "        \n",
    "        #will be screened\n",
    "        elif tokenized[counter:counter+4] == ['will','be','screened','at']:\n",
    "            useful[counter:counter+4] = ['r_pen','connecting','t_scr','connecting']\n",
    "            \n",
    "        #will be sequenced\n",
    "        elif tokenized[counter:counter+6] == ['will','undergo','genome','sequencing','for','voc']:\n",
    "            useful[counter:counter+6] = ['r_pen','connecting','t_seq','t_seq','connecting','v_voc_general']\n",
    "        \n",
    "        else:\n",
    "            #indirect_matches dictionary\n",
    "            for term, patterns in indirect_matches_dict.items():\n",
    "                if any([pattern in token for pattern in patterns]):\n",
    "                    useful[counter] = term\n",
    "                    break\n",
    "                    \n",
    "            #direct_matches dictionary\n",
    "            for term, patterns in direct_matches_dict.items():\n",
    "                if any([pattern == token for pattern in patterns]):\n",
    "                    useful[counter] = term\n",
    "                    break\n",
    "                    \n",
    "            #test_type dictionary\n",
    "            for test, patterns in test_type_dict.items():\n",
    "                if any([pattern == token for pattern in patterns]):\n",
    "                    useful[counter] = test\n",
    "                    break\n",
    "        \n",
    "    return useful"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Algorithm description"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the useful_tokens field, this interpret function sequentially \"reads\" the terms. It picks up virus/result/test terms and they are held in a \"bundle\" (virus, result, test). There are also multiple modifiers that affect the way that the algorithm processes the terms. These modifiers are: final (flag to take highest priority later on), presumptive (change pos to pre), and end (end reading early or skip the next save). Any time a bundle is saved, the bundle (except for test type) and the final/presumptive modifiers are cleared. If a save occurs with incomplete information, the virus defaults to an unknown virus, result defaults to negative, and test defaults to unknown test. Whenever a save happens, all of the previous tokens+labels that were read are considered to be a \"segment\".\n",
    "<br>\n",
    "- The algorithm will go through the labelled tokens one by one. There are different conditions for storing terms and saving the bundle when encountering a virus, a result, a special term, or an irrelevant (unlabelled) term.\n",
    "    - Viruses: A relevant virus is always kept. If the virus switches, save the bundle. If the same virus is read, save the bundle only if there is a result as well. An unknown virus is only kept if there is no current virus.\n",
    "    - Results: A clear result (ind, neg, pos) is kept with hierarchy ind > neg > pos such that a neg/ind can overwrite a positive if it's close together (e.g., \"not detected\" becomes a neg). An unclear result (rej, can, pen) is only kept if there is no current result with hierarchy rej > can > pen. If there is already a previous result and a neg/ind is encounter, save the bundle.\n",
    "    - Special terms:\n",
    "        - Final: Modifier to add flag when saving to specify whether it is a final result, which takes higher priority over all others in the process_result function. Save if there is a current virus and current result. Clear the current result.\n",
    "        - Presumptive: Modifier to change positive (r_pos) into presumptive-positive (r_pre).\n",
    "        - End: Modifier to skip the next save. Save if there is a current virus and current result. Stop the reading if there are any results.\n",
    "        - Reset: Clear bundle without saving.\n",
    "        - Stop: Save if there is a current virus and current result. Clear the bundle.        \n",
    "    - Irrelevant terms: If two irrelevant terms (Nones) are read in a row, save the bundle if there is both a current result and virus. Also save the bundle if there is a virus and the past segment had another virus (virus_counter > 1; normally viruses tested are listed in a mpx or pcr assay). Otherwise, clear the bundle without saving and reset all the counter variables (i.e., start a new segment).\n",
    "    - If the sentence ends before hitting two Nones, save any result and save the bundle if there is a virus and the past segment had another virus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#interpret text to get initial results\n",
    "def interpret(useful):\n",
    "    \n",
    "    def save(b):\n",
    "        #presumptive modifier\n",
    "        if b[1] == 'r_pos' and modifier[1]:\n",
    "            b[1] = 'r_pre'\n",
    "\n",
    "        #end modifier (skips a save)\n",
    "        if not modifier[2] or modifier[0]:\n",
    "            output.append([b[0] if b[0] else 'v_unk', \n",
    "                           b[1] if b[1] else 'r_neg', \n",
    "                           b[2] if b[2] else 't_unk', \n",
    "                           modifier[0]]) #final modifier\n",
    "        \n",
    "        b[0] = None\n",
    "        b[1] = None\n",
    "        modifier[0:4] = [False, False, False, False]\n",
    "        return\n",
    "    \n",
    "    sentence = useful[:]\n",
    "    output = []\n",
    "    \n",
    "    #bundle for current virus/result/test\n",
    "    #0 = virus, 1 = result, 2 = test\n",
    "    bundle = [None, None, None]\n",
    "    \n",
    "    #modifiers\n",
    "    #0 = final, 1 = presumptive, 2 = end, 3 = skip\n",
    "    modifier = [False, False, False, False]\n",
    "    \n",
    "    none_counter = 0 #counter for hitting consecutive irrelevant words\n",
    "    virus_counter = 0 #counter for different viruses in same segment\n",
    "    \n",
    "    #xml field processing\n",
    "#     xml_pos = [i for i, x in enumerate(sentence) if x == 'xml']\n",
    "#     num = len(xml_pos)//2\n",
    "#     for i in range(num):\n",
    "#         xml_start_pos = xml_pos[i*2]\n",
    "#         xml_end_pos = xml_pos[i*2+1]\n",
    "#         for j in range(xml_start_pos, xml_end_pos + 1):\n",
    "#             if sentence[j] and sentence[j].startswith('v_') and sentence[j] != 'v_unk':\n",
    "#                 bundle[0] = sentence[j]\n",
    "#                 bundle[1] = 'r_pos'\n",
    "#                 save(bundle)\n",
    "\n",
    "    #loop on words in sentence\n",
    "    for word in sentence:\n",
    "        \n",
    "        if word: #relevant term\n",
    "            none_counter = 0 #restart counter\n",
    "            \n",
    "            #set current virus             \n",
    "            if word.startswith('v_'):\n",
    "                #different virus\n",
    "                if word != 'v_unk' and word != bundle[0]:\n",
    "                    #save current result if hitting a different virus\n",
    "                    if bundle[0] and bundle[0] != 'v_unk' and bundle[1]:\n",
    "                        save(bundle)\n",
    "                    bundle[0] = word\n",
    "                #same virus\n",
    "                elif word != 'v_unk' and word == bundle[0]:\n",
    "                    #save current result if there is one\n",
    "                    if bundle[1]:\n",
    "                        save(bundle)\n",
    "                    bundle[0] = word\n",
    "                #only set to general virus if there's no current virus\n",
    "                elif word == 'v_unk' and not bundle[0]:\n",
    "                    bundle[0] = word\n",
    "            \n",
    "            #set current result\n",
    "            elif word.startswith('r_'):\n",
    "                if word == 'r_ind':\n",
    "                    if bundle[1]: \n",
    "                        save(bundle)\n",
    "                    bundle[1] = word\n",
    "                elif word == 'r_neg' and bundle[1] not in ('r_ind',):\n",
    "                    if bundle[1]: \n",
    "                        save(bundle)\n",
    "                    bundle[1] = word\n",
    "                elif word == 'r_pos' and bundle[1] not in ('r_ind', 'r_neg'):\n",
    "                    bundle[1] = word\n",
    "\n",
    "                elif word in ('r_rej', 'r_can', 'r_pen') and bundle[1] not in ('r_ind', 'r_neg', 'r_pos'):\n",
    "                    if word == 'r_rej':\n",
    "                        bundle[1] = word\n",
    "                    elif word == 'r_can' and bundle[1] not in ('r_rej',):\n",
    "                        bundle[1] = word\n",
    "                    elif word == 'r_pen' and bundle[1] not in ('r_rej', 'r_can'):\n",
    "                        bundle[1] = word\n",
    "                \n",
    "            #set current test\n",
    "            elif word.startswith('t_'):\n",
    "                #make sure sequencing doesn't affect previous screening result\n",
    "                if (bundle[0] and bundle[1] and word == 't_seq'):\n",
    "                    save(bundle)\n",
    "                bundle[2] = word\n",
    "            \n",
    "            #final modifier\n",
    "            elif word == 'final':\n",
    "                if bundle[1]:\n",
    "                    save(bundle)\n",
    "                modifier[0] = True\n",
    "                bundle[1] = None #reset result\n",
    "            \n",
    "            #presumptive modifier\n",
    "            elif word == 'presumptive':\n",
    "                modifier[1] = True\n",
    "            \n",
    "            #end modifier/word\n",
    "            elif word == 'end':\n",
    "                #end early only if there is already result\n",
    "                if bundle[0] and bundle[1]:\n",
    "                    save(bundle)\n",
    "                modifier[0:4] = [False, False, True, False] #end modifier skips next save\n",
    "                #end early only if there is already result\n",
    "                if len(output) > 0:\n",
    "                    return output        \n",
    "            \n",
    "            #stop word\n",
    "            elif word == 'stop':\n",
    "                if bundle[0] and bundle[1]:\n",
    "                    save(bundle)\n",
    "                modifier[0:4] = [False, False, False, False]\n",
    "                bundle[0] = None\n",
    "                bundle[1] = None           \n",
    "                \n",
    "            #reset word\n",
    "            elif word == 'reset':\n",
    "                modifier[0:4] = [False, False, False, False]\n",
    "                bundle[0] = None\n",
    "                bundle[1] = None\n",
    "            \n",
    "        else: #word is None\n",
    "            none_counter += 1\n",
    "            \n",
    "            if none_counter == 2: #can change threshold\n",
    "                #save if there is current virus and result\n",
    "                if bundle[0] and bundle[1]:\n",
    "                    save(bundle)\n",
    "                #reset\n",
    "                none_counter = 0 \n",
    "                virus_counter = 0\n",
    "                bundle[0] = None\n",
    "                bundle[1] = None\n",
    "                modifier[0:4] = [False, False, False, False]\n",
    "                \n",
    "    #if there is still a remaining result\n",
    "    if bundle[1]: \n",
    "        save(bundle)\n",
    "            \n",
    "    return output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#using reference excel to assign LOINCs to virus and test type\n",
    "xlsx_filename = 'COVID19_VOC_codes_20210422.xls'\n",
    "mappings = {'--':'unk'}\n",
    "\n",
    "df_loincs = pd.read_excel(xlsx_filename, sheet_name='VOC_LOINCs')\n",
    "\n",
    "#cleaning the categories to match previously defined ones\n",
    "df_loincs = df_loincs.replace(mappings)\n",
    "df_loincs['Virus_to_assign'] = df_loincs['Virus_to_assign'].apply(lambda x: 'v_' + x)\n",
    "df_loincs['Test_to_assign'] = df_loincs['Test_to_assign'].apply(lambda x: 't_' + x)\n",
    "\n",
    "#assign LOINCs to virus and test type\n",
    "loincs_by_v = {}\n",
    "loincs_by_t = {}\n",
    "for index, row in df_loincs.iterrows():\n",
    "    loincs_by_v.setdefault(row['Virus_to_assign'], [])\n",
    "    loincs_by_v[row['Virus_to_assign']].append(row['LOINCs'])\n",
    "    loincs_by_t.setdefault(row['Test_to_assign'], [])\n",
    "    loincs_by_t[row['Test_to_assign']].append(row['LOINCs'])\n",
    "\n",
    "#remove the unk ones\n",
    "del loincs_by_v['v_unk']\n",
    "del loincs_by_t['t_unk']\n",
    "\n",
    "#use reference excel to assign TR codes to virus and test type\n",
    "df_tr_codes = pd.read_excel(xlsx_filename, sheet_name='VOC_TRs')\n",
    "\n",
    "#cleaning the categories to match previously defined ones\n",
    "df_tr_codes = df_tr_codes.replace(mappings)\n",
    "df_tr_codes['Virus_to_assign'] = df_tr_codes['Virus_to_assign'].apply(lambda x: 'v_' + x)\n",
    "df_tr_codes['Test_to_assign'] = df_tr_codes['Test_to_assign'].apply(lambda x: 't_' + x)\n",
    "\n",
    "#assign LOINCs to virus and test type\n",
    "tr_codes_by_v = {}\n",
    "tr_codes_by_t = {}\n",
    "for index, row in df_tr_codes.iterrows():\n",
    "    tr_codes_by_v.setdefault(row['Virus_to_assign'], [])\n",
    "    tr_codes_by_v[row['Virus_to_assign']].append(row['TRs'])\n",
    "    tr_codes_by_t.setdefault(row['Test_to_assign'], [])\n",
    "    tr_codes_by_t[row['Test_to_assign']].append(row['TRs'])\n",
    "    \n",
    "#remove the unk ones\n",
    "del tr_codes_by_v['v_unk']\n",
    "# del tr_codes_by_t['t_unk']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# assign more details to v_unk or t_unk based on LOINC and TR code\n",
    "# group by test type and then type of virus, remove duplicates\n",
    "loinc_exclusions = ['10219-4','10182-4','11329-0','14869-2','21026-0','22633-2','22634-0','22635-7','22636-5','22637-3',\n",
    "                    '22638-1','22639-9','31208-2','3150-0','33882-2','35265-8','41000-1','47526-9','49049-0','55752-0','56816-2','59465-5',\n",
    "                    '59466-3','664-3','66746-9','76425-8','XON10007-3','XON10011-5','XON10313-5','XON10315-0','XON10316-8',\n",
    "                    'XON10337-4','XON11913-1','XON12721-7','XON12875-1','XON13543-4','XON13544-2','XON13545-9',\n",
    "                    '94558-4','94661-6']\n",
    "tr_exclusions = ['TR12942-9']\n",
    "\n",
    "def process_result(tokens, testrequestcode, observationcode, results, reportinglaborgname):\n",
    "    dd = {}\n",
    "    \n",
    "    #LOINC/TR exclusions\n",
    "    if (observationcode in loinc_exclusions) or (testrequestcode in tr_exclusions):\n",
    "        return dd \n",
    "    \n",
    "    ###extra conditions\n",
    "              \n",
    "    ###determine virus or test based on LOINC or TR\n",
    "    v_from_loinc = [loinc_vir for loinc_vir, loincs in loincs_by_v.items() if observationcode in loincs]\n",
    "    v_from_tr = [tr_codes_vir for tr_codes_vir, tr_codes in tr_codes_by_v.items() if testrequestcode in tr_codes]\n",
    "    t_from_loinc = [loinc_test for loinc_test, loincs in loincs_by_t.items() if observationcode in loincs]\n",
    "    t_from_tr = [tr_codes_test for tr_codes_test, tr_codes in tr_codes_by_t.items() if testrequestcode in tr_codes]\n",
    "    \n",
    "    #determine if there are any final/interpretation results\n",
    "    viruses_with_final = [(v,t) for (v,r,t,f) in results if r in ('r_pos', 'r_pre', 'r_ind', 'r_neg', 'r_rej') and f]\n",
    "    results_final = results\n",
    "    #remove the non-final/interpretation results for viruses with final/interpretation\n",
    "    for vf,tf in viruses_with_final:\n",
    "        results_final = [(v,r,t,f) for (v,r,t,f) in results if not (v == vf and t == tf and not f)]\n",
    "    \n",
    "    #add each result into the output dictionary of results per virus per test type\n",
    "    for v, r, t, f in results_final:\n",
    "        #fill in unknown virus\n",
    "        if v == 'v_unk':\n",
    "            if len(v_from_loinc) > 0:\n",
    "                v = v_from_loinc[0]\n",
    "            elif len(v_from_tr) > 0:\n",
    "                v = v_from_tr[0]\n",
    "            \n",
    "        ## if any variant term present excluding voc_gen, assign t_seq\n",
    "        if v.startswith('v_voc') and v != 'v_voc_general':\n",
    "            t = 't_seq'\n",
    "        \n",
    "        #If test is unknown and any voc virus present, assign t_seq\n",
    "        if t == 't_unk' and any([v.startswith('v_voc') and r in ('r_pos','r_neg') for v,r,t,f in results]):\n",
    "            t = 't_seq'\n",
    "        \n",
    "        #fill in unknown test\n",
    "        if t == 't_unk':\n",
    "            if len(t_from_loinc) > 0:\n",
    "                t = t_from_loinc[0]\n",
    "            elif len(t_from_tr) > 0:\n",
    "                t = t_from_tr[0]\n",
    "        \n",
    "        #assume screen or sequence based on tokens or mutations/vocs\n",
    "        if t == 't_unk' and (any(['screen' in t for t in tokens]) \n",
    "                             or v in ('v_sgene_mutation','v_sgene_n501y','v_sgene_e484k')):\n",
    "            t = 't_scr'\n",
    "        elif t == 't_unk' and any(['sequenc' in t for t in tokens]): \n",
    "            t = 't_seq'\n",
    "        elif t == 't_unk' and any([v.startswith('v_voc') for v,r,t,f in results]):\n",
    "            t = 't_seq'\n",
    "   \n",
    "        # replace all v_sgene_mutation for t_scr with v_voc_general\n",
    "        if v == 'v_sgene_mutation' and t == 't_scr':\n",
    "            v ='v_voc_general'\n",
    "\n",
    "        #remove unknown virus results\n",
    "        if v != 'v_unk' and t != 't_unk':\n",
    "            v, r, t = v[2:], r[2:], t[2:]\n",
    "            dd.setdefault(t, [])\n",
    "            \n",
    "            #compiling results with hierarchy: S (presumptive positive) > P (positive) > N (negative)\n",
    "            #                                  >  I (indeterminate) > D (pending) > R (invalid) > C (cancelled) \n",
    "            same_vir = False\n",
    "            for i in range(len(dd[t])):\n",
    "                if v == dd[t][i][0]:\n",
    "                    same_vir = True\n",
    "                    if r == 'pre':\n",
    "                        dd[t][i] = (v,r)\n",
    "                    elif r == 'pos' and dd[t][i][1] not in ('pre',):\n",
    "                        dd[t][i] = (v,r)\n",
    "                    elif r == 'neg' and dd[t][i][1] not in ('pre', 'pos'):\n",
    "                        dd[t][i] = (v,r)\n",
    "                    elif r == 'ind' and dd[t][i][1] not in ('pre', 'pos', 'neg'):\n",
    "                        dd[t][i] = (v,r)\n",
    "                    elif r == 'pen' and dd[t][i][1] not in ('pre', 'pos', 'neg', 'ind'):\n",
    "                        dd[t][i] = (v,r)\n",
    "                    elif r == 'rej' and dd[t][i][1] not in ('pre', 'pos', 'neg', 'ind', 'pen'):\n",
    "                        dd[t][i] = (v,r)\n",
    "                    elif r == 'can':\n",
    "                        pass\n",
    "            if not same_vir:\n",
    "                dd[t].append((v,r))\n",
    "        \n",
    "    return dd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#create output as character value for each virus and test type\n",
    "result_char = {'pre':'S', 'pos': 'P', 'neg':'N', 'ind':'I', 'pen':'D', 'can':'C', 'rej':'R'}\n",
    "\n",
    "def char_output(results, ind):\n",
    "\n",
    "    #loop through each test type and virus\n",
    "    for t, pairs in results.items(): \n",
    "            for v, r in pairs:\n",
    "                if v.startswith('voc'):\n",
    "                    if '_general' in v:\n",
    "                        df_results.at[ind, t+'_voc'] = result_char[r]\n",
    "                    elif '_b117' in v:\n",
    "                        df_results.at[ind, t+'_voc_b117'] = result_char[r]\n",
    "                    elif '_b1351' in v:\n",
    "                        df_results.at[ind, t+'_voc_b1351'] = result_char[r]\n",
    "                    elif '_p1' in v:\n",
    "                        df_results.at[ind, t+'_voc_p1'] = result_char[r]\n",
    "                    elif '_p2' in v:\n",
    "                        df_results.at[ind, t+'_voc_p2'] = result_char[r]\n",
    "                    \n",
    "                elif v.startswith('sgene'):\n",
    "                    if '_n501y' in v:\n",
    "                        df_results.at[ind, t+'_sgene_n501y'] = result_char[r]\n",
    "                    elif '_e484k' in v:\n",
    "                        df_results.at[ind, t+'_sgene_e484k'] = result_char[r]\n",
    "                    elif '_k417n' in v:\n",
    "                        df_results.at[ind, t+'_sgene_k417n'] = result_char[r]\n",
    "                    elif '_k417t' in v:\n",
    "                        df_results.at[ind, t+'_sgene_k417t'] = result_char[r]\n",
    "                   \n",
    "    return"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initial processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "#make copy of df\n",
    "df_unique = df_gp.copy(deep = True)\n",
    "\n",
    "#clean text\n",
    "df_unique[\"cleaned_value\"] = df_unique[\"value\"].apply(clean)\n",
    "\n",
    "#group by unique records (org, TR code, Obs code, cleaned text) and store original indexes as tuple\n",
    "df_unique = df_unique.reset_index()\n",
    "groupby_vars = ['reportinglaborgname', 'testrequestcode', 'observationcode', 'cleaned_value']\n",
    "df_unique = df_unique.groupby(groupby_vars).agg({'value': 'count', \n",
    "                                                 'original_indexes': lambda x: tuple([i for tup in x for i in tup])}).reset_index()\n",
    "df_unique = df_unique.rename(columns={'value':'count'})\n",
    "\n",
    "df_unique = df_unique.sort_values(by=['count'], ascending=False).reset_index(drop=True)\n",
    "print('unique records after cleaning:', len(df_unique))\n",
    "\n",
    "#tokenize\n",
    "df_unique[\"cleaned_tokenized_value\"] = df_unique[\"cleaned_value\"].apply(tokenize)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Assign results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#assign labels using dictionary\n",
    "df_unique[\"useful_tokens\"] = df_unique[\"cleaned_tokenized_value\"].apply(assign_labels)\n",
    "\n",
    "#interpret the labelled tokens\n",
    "df_unique[\"initial_results\"] = df_unique[\"useful_tokens\"].apply(interpret)\n",
    "\n",
    "# #fill in unknown viruses based on LOINC or TR code, roll up results to one test type\n",
    "final_results = []\n",
    "for i in range(len(df_unique)):\n",
    "    final_results.append(process_result(df_unique[\"cleaned_tokenized_value\"][i],\n",
    "                                        df_unique[\"testrequestcode\"][i],df_unique[\"observationcode\"][i], \n",
    "                                        df_unique[\"initial_results\"][i],df_unique[\"reportinglaborgname\"][i]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#translate results to 1-character format\n",
    "# change output cols\n",
    "result_cols = ['scr_voc','scr_sgene_n501y','scr_sgene_e484k','seq_sgene_n501y',\n",
    "               'seq_sgene_e484k','seq_sgene_k417n','seq_sgene_k417t',\n",
    "               'seq_voc', 'seq_voc_b117', 'seq_voc_b1351','seq_voc_p1', 'seq_voc_p2']\n",
    "\n",
    "#create empty df to fill in results\n",
    "df_results = pd.DataFrame(index=np.arange(len(df_unique)), columns=['original_indexes']+result_cols)\n",
    "df_results['original_indexes'] = df_unique['original_indexes']\n",
    "\n",
    "#fill in results\n",
    "for i in range(len(df_unique)):\n",
    "    char_output(final_results[i], i)\n",
    "\n",
    "#fill in seq_voc/scr_voc if there are any specific vocs\n",
    "# take max of set of columns\n",
    "result_mappings = {'S':0,'P':1,'N':2,'I':3,'D':4,'C':5,'R':6, np.nan:100}\n",
    "result_mappings_output = {0:'S',1:'P',2:'N',3:'I',4:'D',5:'C',6:'R',100:np.nan}\n",
    "\n",
    "for c in result_cols:\n",
    "    df_results[c] = df_results[c].map(result_mappings)\n",
    "    \n",
    "df_results['scr_voc'] = df_results.apply(lambda row: min([row[c] for c in result_cols if c.startswith('scr')]),axis=1)\n",
    "df_results['seq_voc'] = df_results.apply(lambda row: min([row[c] for c in result_cols if c.startswith('seq_voc')]) ,axis=1)\n",
    "\n",
    "for c in result_cols:\n",
    "    df_results[c] = df_results[c].map(result_mappings_output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Final output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#order results based on original_indexes\n",
    "output = [None]*len(df)\n",
    "\n",
    "# drop any columns not in result_cols - should not happen\n",
    "for c in df_results.columns:\n",
    "    if c not in result_cols and c!='original_indexes':\n",
    "        print('Missing column:',c)\n",
    "        print(df_results[~df_results[c].isna()][c])\n",
    "        df_results.drop(columns=c,inplace=True)\n",
    "        \n",
    "for row in df_results.itertuples():\n",
    "    for i in row[1]: #original_indexes\n",
    "        output[i] = tuple(row[2:])\n",
    "        \n",
    "if output_flag == 1:                \n",
    "    df_output = pd.concat([df, pd.DataFrame(output, columns=result_cols)], axis=1)\n",
    "elif output_flag == 2: \n",
    "    df_output = df_raw.join(df[['exclude_flag']].join(pd.DataFrame(output, columns=result_cols)))\n",
    "    \n",
    "else:\n",
    "    print('PLEASE ENTER ONE OF THE FOLLOWING OPTIONS FOR OUTPUT_FLAG IN THE FIRST CELL: 1, 2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#FINAL DATASET TO OUTPUT\n",
    "df_output.to_csv(output_filename+'.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df_output['scr_voc'].value_counts()\n",
    "# df_output.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Roll Up \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "proper_TRCs = False\n",
    "output_filename_episodes = 'episodes_voc'\n",
    "df_output2 = df_output.copy()\n",
    "\n",
    "#run only on proper TRCs\n",
    "if proper_TRCs:\n",
    "    output_filename_episodes += '_tr'\n",
    "    df_output2 = df_output2[df_output2.testrequestcode.isin(['TR12952-8','TR12953-6'])]\n",
    "\n",
    "#remove observations based on observationresultstatus\n",
    "print('Number of records removed with result status N/X:',str(sum(df_output2['observationresultstatus'].isin(('N','X')))))\n",
    "df_clean = df_output2[~df_output2['observationresultstatus'].isin(('N','X'))]\n",
    "\n",
    "print('Number of records removed with result status W (exclude_flag):',str(sum(df_clean['exclude_flag'] == 'Y')))\n",
    "df_clean = df_clean[df_clean['exclude_flag'] == 'N']\n",
    "\n",
    "print('Number of records removed with blank patientid:',str(sum(df_clean['patientid'] == '')))\n",
    "df_clean = df_clean[df_clean['patientid'] != '']\n",
    "\n",
    "print('Number of records remaining: '+str(len(df_clean)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hierarchy: P > N > I > D > C > R \n",
    "#P = Positive,  N = Negative, I = Indeterminate, D = penDing, C = Cancelled, R = Rejected/invalid\n",
    "result_mappings = {'P':1,'N':2,'I':3,'D':4,'C':5,'R':6, np.nan:100,'':100}\n",
    "result_mappings_output = {1:'P',2:'N',3:'I',4:'D',5:'C',6:'R',100:''}\n",
    "result_tf_output = {1:'T',0:'F'}\n",
    "\n",
    "#assign S (presumptive-positive) as P (positive)\n",
    "#convert result variable (from previous script) to number for hierarchy\n",
    "for c in result_cols:\n",
    "#     df_clean.loc[df_clean[c] == 'S', c] = 'P'\n",
    "    df_clean.loc[df_clean[c] == 'S', c] = np.nan\n",
    "    df_clean[c] = df_clean[c].map(result_mappings)\n",
    "\n",
    "blank_col_count = 100*len([c for c in result_cols if c.startswith('seq')])\n",
    "\n",
    "# scr/seq flag\n",
    "df_clean['scr_flag'] = df_clean.apply(lambda row: 1 if row['scr_voc'] < 100 else 0, axis=1)\n",
    "df_clean['seq_flag'] = df_clean.apply(lambda row: 1 if sum([row[c] for c in result_cols if c.startswith('seq')]) < blank_col_count else 0, axis=1)\n",
    "\n",
    "# keep rows with any result\n",
    "df_clean = df_clean[(df_clean.scr_flag == 1) | (df_clean.seq_flag == 1)]\n",
    "\n",
    "# sqr/seq completed: observations that have a clear covid result (P, N, I, D) \n",
    "df_clean['scr_test'] = df_clean['scr_voc'].apply(lambda row: 1 if row in (1,2,3,4) else 0)\n",
    "df_clean['seq_test'] = df_clean.apply(lambda row: 1 if any([row[c] in (1,2,3,4) for c in result_cols if c.startswith('seq')]) else 0,axis=1)\n",
    "\n",
    "#date versions of datetime\n",
    "df_clean['specimenreceiveddate'] = df_clean['specimenreceiveddatetime'].apply(lambda x: np.datetime64(x, 'D'))\n",
    "df_clean['observationdate'] = df_clean['observationdatetime'].apply(lambda x: np.datetime64(x, 'D'))\n",
    "df_clean['observationreleasedate'] = df_clean['observationreleasets'].apply(lambda x: np.datetime64(x, 'D'))\n",
    "\n",
    "#drop used columns\n",
    "df_clean.drop(['scr_flag','seq_flag','observationresultstatus','exclude_flag',\n",
    "               'observationcode','observationdatetime','specimenreceiveddatetime'], 1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### ROLL UP to TEST REQUEST and then to TESTING EPISODE\n",
    "\n",
    "# for episode, each record with latest obsreleasets scr/seq completed\n",
    "df_episodes = df_clean.groupby(['patientid','observationdate']).agg({'seq_test':'max','scr_test':'max'}).reset_index()\n",
    "\n",
    "# for each result column, take highest priority result (clear result > latest release timestamp > result hierarchy)\n",
    "# also collect ordersids where results were taken\n",
    "for c in result_cols:\n",
    "    t = c + '_test'\n",
    "    df_temp = df_clean[['patientid','observationdate','ordersid','fillerordernumberid','observationreleasets',c]].copy()\n",
    "    df_temp[t] = df_temp[c].apply(lambda x: 1 if x in (1,2,3,4) else 0)\n",
    "    \n",
    "    #group to test request level\n",
    "    df_temp = df_temp.sort_values(['ordersid','fillerordernumberid',t,'observationreleasets',c], \n",
    "                           ascending=[True,True,False,False,True]).\\\n",
    "                groupby(['ordersid','fillerordernumberid']).first().reset_index()\n",
    "    \n",
    "    #group to episode level\n",
    "    df_temp = df_temp.sort_values(['patientid','observationdate',c,'ordersid'], \n",
    "                           ascending=[True,True,True,True]).\\\n",
    "                groupby(['patientid','observationdate']).first().reset_index()\n",
    "    \n",
    "    df_temp.rename(columns={'ordersid':'ordersid_'+c},inplace=True)\n",
    "    df_temp.drop(columns=['observationreleasets',t,'fillerordernumberid'],inplace=True)\n",
    "    df_episodes = df_episodes.merge(df_temp,how='left',on=['patientid','observationdate'])\n",
    "\n",
    "df_episodes['final_result_ordersids'] = df_episodes.apply(lambda x: ','.join(sorted(list(set([str(int(x[c])) for c in df_episodes.columns \\\n",
    "                                                                     if c.startswith('ordersid_')])))),axis=1)\n",
    "\n",
    "df_episodes.drop(columns=[x for x in df_episodes.columns if x.startswith('ordersid_')], inplace=True)\n",
    "\n",
    "#taking the max observationreleasedate/specimenreceiveddate for each test type\n",
    "for c in ['observationreleasedate','specimenreceiveddate']:\n",
    "    for t in ['scr','seq']:\n",
    "        df_temp = df_clean[df_clean[t+'_test']==1].groupby(['patientid','observationdate']).\\\n",
    "                            agg({c:'max'}).\\\n",
    "                            rename(columns={c:c+'_'+t}).\\\n",
    "                            reset_index()\n",
    "        df_episodes = df_episodes.merge(df_temp,how='left',on=['patientid','observationdate'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Map numeric results back to text\n",
    "for c in result_cols:\n",
    "    df_episodes[c] = df_episodes[c].map(result_mappings_output)\n",
    "    \n",
    "df_episodes['scr_test']=df_episodes['scr_test'].map(result_tf_output)\n",
    "df_episodes['seq_test']=df_episodes['seq_test'].map(result_tf_output)\n",
    "\n",
    "print(df_episodes['scr_voc'].value_counts())\n",
    "print(df_episodes['seq_voc'].value_counts())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Output file\n",
    "df_episodes.to_csv(output_filename_episodes+'.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Manual review"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#tracker for unique records (some records may be marked as new if clean function changes)\n",
    "\n",
    "#initialize tracker\n",
    "try:\n",
    "    f = open('record_tracker.pkl')\n",
    "    f.close()\n",
    "except FileNotFoundError:\n",
    "    df_tracker = pd.DataFrame(columns=['filename', 'reportinglaborgname', 'testrequestcode', 'observationcode', 'cleaned_value'])\n",
    "    df_tracker.to_pickle(\"./record_tracker.pkl\")\n",
    "    print('CREATING RECORD TRACKER FILE')\n",
    "    \n",
    "#read tracker\n",
    "df_tracker = pd.read_pickle('./record_tracker.pkl')\n",
    "\n",
    "#RESET TRACKER\n",
    "#df_tracker = df_tracker.iloc[0:0]\n",
    "\n",
    "df_tracker_orig = df_tracker[['reportinglaborgname', 'testrequestcode', 'observationcode', 'cleaned_value']].copy(deep = True)\n",
    "df_tracker_delta = df_unique[['reportinglaborgname', 'testrequestcode', 'observationcode', 'cleaned_value']].copy(deep = True)\n",
    "\n",
    "#set difference\n",
    "df_tracker_delta = pd.concat([df_tracker_delta, df_tracker_orig, df_tracker_orig], ignore_index=True).drop_duplicates(keep=False)\n",
    "print('Original tracker length:', len(df_tracker_orig))\n",
    "print('Delta tracker length:', len(df_tracker_delta))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#intermediate output for checking results\n",
    "int_output_cols = ['count', 'reportinglaborgname', 'testrequestcode', 'observationcode', 'cleaned_value']\n",
    "df_unique[int_output_cols].join(df_results.drop(columns=['original_indexes'])).to_csv('intermediate_output.csv')\n",
    "df_unique[int_output_cols][df_unique.index.isin(df_tracker_delta.index)].join(df_results.drop(columns=['original_indexes'])).to_csv('intermediate_output_delta.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "#FINALIZE THE RECORD TRACKER (only run when you are satisfied with the review process)\n",
    "#add filename\n",
    "df_tracker_delta['filename'] = input_filename\n",
    "\n",
    "#add the delta\n",
    "df_tracker = pd.concat([df_tracker, df_tracker_delta], sort=False, ignore_index=True)\n",
    "\n",
    "#save file\n",
    "df_tracker.to_pickle(\"./record_tracker.pkl\")\n",
    "print('Records in tracker:', len(df_tracker))\n",
    "\n",
    "#cleanup\n",
    "# del df\n",
    "# del df_unique\n",
    "# del df_output\n",
    "# del df_output2\n",
    "# del df_tracker\n",
    "# del df_tracker_orig\n",
    "# del df_tracker_delta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # % positivity numbers\n",
    "\n",
    "# # patient level - SCREENING\n",
    "# df_pat = df_episodes[['patientid','scr_voc','scr_test']]\n",
    "# df_pat = df_pat.groupby(['patientid','scr_test'],as_index=False).agg({'scr_voc':'min'})\n",
    "\n",
    "# print(\"PATIENT LEVEL SCREENING\")\n",
    "# print(sum(df_pat.scr_voc==1))\n",
    "# print(sum(df_pat.scr_test==1))\n",
    "# print(sum(df_pat.scr_voc==1)/sum(df_pat.scr_test==1))\n",
    "\n",
    "# # patient level - SCREENING or SEQUENCING\n",
    "# df_pat = df_episodes[['patientid','scr_voc','scr_test','seq_voc','seq_test']]\n",
    "# df_pat = df_pat.groupby(['patientid','scr_test','seq_test'],as_index=False).agg({'scr_voc':'min','seq_voc':'min'})\n",
    "\n",
    "# print(\"PATIENT LEVEL SCREENING OR SEQUENCING\")\n",
    "# print(sum((df_pat.scr_voc==1)|(df_pat.seq_voc==1)))\n",
    "# print(sum((df_pat.scr_test==1)|(df_pat.seq_test==1)))\n",
    "# print(sum((df_pat.scr_voc==1)|(df_pat.seq_voc==1))/sum((df_pat.scr_test==1)|(df_pat.seq_test==1)))\n",
    "\n",
    "# # episodes level - SCREENING\n",
    "# print(\"EPISODE LEVEL SCREENING\")\n",
    "# print(sum(df_episodes.scr_voc==1))\n",
    "# print(sum(df_episodes.scr_test==1))\n",
    "# print(sum(df_episodes.scr_voc==1)/sum(df_episodes.scr_test==1))\n",
    "\n",
    "# # episodes level - SCREENING or SEQUENCING\n",
    "# print(\"EPISODE LEVEL SCREENING OR SEQUENCING\")\n",
    "# print(sum((df_episodes.scr_voc==1)|(df_episodes.seq_voc==1)))\n",
    "# print(sum((df_episodes.scr_test==1)|(df_episodes.seq_test==1)))\n",
    "# print(sum((df_episodes.scr_voc==1)|(df_episodes.seq_voc==1))/sum((df_episodes.scr_test==1)|(df_episodes.seq_test==1)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing and validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#test a string\n",
    "test_string = r'''\n",
    "\n",
    "'''\n",
    "\n",
    "test_clean = clean(test_string)\n",
    "print('----', test_clean)\n",
    "test_useful = assign_labels(tokenize(test_clean))\n",
    "print('----', test_useful)\n",
    "test_interpret = interpret(test_useful)\n",
    "print('----', test_interpret)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}