Codebase for Dasgupta et al. 2018, https://arxiv.org/abs/1802.04302
An updated version of this work is Dasgupta, Ishita, et al. "Analyzing machine-learned representations: A natural language case study." arXiv preprint arXiv:1909.05885 (2019). Linked here: https://arxiv.org/abs/1909.05885
Code to generate Compositional dataset based on comparisons, SNLI data analysis and scripts for augmented training is in the training-experiments branch.
Dataset used in the paper is here: https://github.com/ishita-dg/ScrambleTests/tree/training-experiment/testData/new
Code in main branch generates a smaller but more general dataset, sets up classifiers, downloads and tokenizes data.
In the Downloads folder, run:
./get_data.bash
Requires 7za to unzip downloaded files, download and install from https://sourceforge.net/projects/p7zip/files/p7zip/
Path to sed tokenizer might need to be adjusted.
Run: python main.py, with toy = True.
This should run through training the classifier and test code on toy data sets (provided).
Setting it False will run the true classifier and take a long time, and very high memory (~150+ GB) for InferSent embeddings.
Set useCudaReg = True in main.py
The logistic regression models (in ./models/) as well as their outputs on the true scramble-test results (in ./regout/) are provided. So you can run the analysis script directly.
In AnalyseTests.ipynb, setting Scram = True runs tests for Scramble test data, Scram = False runs it for te SNLI test and/or dev sets.
Produces the plots (inline as well as in ./figures/), and displays high-margin BOW misclassifications (inline).