process

Sample Source List:

1. From UD Treebanks, generate a list of nouns, using extract_dict. The files *.NOUN are of no use, so can be discarded.
2. The different *.most_common are combined using process_sample's "combine" switch. The output is saved as sample.most_common
3. sample.most_common contains the frequency count of the most reliable nouns.

********************
Sample_count = 2287
*********************

Corpus Analysis:

1. Get IIT-B Parallel Corpus
2. Process both sides through UDPipe
3. generate a list of nouns, using extract_dict. The file *.NOUN is again of no use, and can be discarded.
4. Check the number of values in hi.most_common.

************************
Corpus_Count = 159 447
************************

5. Check the number of values in Sample_count occuring in Corpus_count, using process_sample's "coverage" switch.

***********************************
Coverage = 7632/8829    86.4424 %
***********************************

########################################################################################################################

Elimination of nouns from sample.most_common:

1. Remove all the cases where the word can be used for male/female occurrences.
    An example is of words which require 'male' or 'female' as a prefix to dismabiguate between the two.
    could be using 'nar' or 'maada'
    or maybe no prefix and 'mahila'

English(with case)  Hindi   Frequency   Line_no in sample.most_common

List of words removed:



Doubtful:



Words without meaning, with numerals or incorrect spelling:



Process pipeline:

1. Filter the nouns as per the elimination from sample.most_common
    4000 done
2. get morphological rules for these nouns, and the pair of allowed values. Consult wikipedia on this.
3. get a trie model

Type- counting each type of context for each noun only once, regardless how many times that context and word form
        appears in corpus with a given noun.