Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: allenai/bi-att-flow
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: master
Choose a base ref
...
head repository: carlosscastro/bi-att-flow
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: master
Choose a head ref
Able to merge. These branches can be automatically merged.

Commits on Mar 25, 2018

  1. Copy the full SHA
    9b5dee4 View commit details
  2. Copy the full SHA
    7ed0cf0 View commit details
  3. add samples and results from almost state of the art results on squad…

    … dataset
    carlosscastro committed Mar 25, 2018
    Copy the full SHA
    d0bbb83 View commit details

Commits on Mar 28, 2018

  1. adding report draft

    carlosscastro committed Mar 28, 2018
    Copy the full SHA
    5bef09d View commit details
  2. preprocess and marco skeleton. Not working yet, need to concatenate m…

    …arco entries and tweak the schema to make it look like squad in terms of output data structure
    carlosscastro committed Mar 28, 2018
    Copy the full SHA
    f6185db View commit details
  3. improving report

    carlosscastro committed Mar 28, 2018
    Copy the full SHA
    4e5014f View commit details
  4. Copy the full SHA
    b9a03f7 View commit details
  5. improving report

    carlosscastro committed Mar 28, 2018
    Copy the full SHA
    8242a85 View commit details
  6. improving report

    carlosscastro committed Mar 28, 2018
    Copy the full SHA
    7eb667f View commit details

Commits on Apr 15, 2018

  1. Copy the full SHA
    c8e4544 View commit details
  2. Copy the full SHA
    67d8516 View commit details
  3. Copy the full SHA
    6545b5b View commit details
  4. Copy the full SHA
    22dee54 View commit details
  5. Copy the full SHA
    5ce4dd3 View commit details
  6. Copy the full SHA
    a62f14d View commit details
  7. Copy the full SHA
    bf75e78 View commit details
  8. Copy the full SHA
    ab70f11 View commit details

Commits on Apr 16, 2018

  1. Copy the full SHA
    6690fa4 View commit details
  2. Copy the full SHA
    d4d9c3f View commit details
  3. Copy the full SHA
    2c64a4e View commit details

Commits on Apr 21, 2018

  1. add marco convert tool which converts the marco dataset format to squ…

    …ad format and also add presentation slides
    carlosscastro committed Apr 21, 2018
    Copy the full SHA
    fca5f54 View commit details
  2. Copy the full SHA
    2da2aba View commit details
  3. update report

    carlosscastro committed Apr 21, 2018
    Copy the full SHA
    36c1b72 View commit details
  4. Copy the full SHA
    de408c0 View commit details

Commits on Apr 22, 2018

  1. Copy the full SHA
    7d839f7 View commit details
Showing with 6,772 additions and 27 deletions.
  1. +19 −26 README.md
  2. BIN docs/Transfer learning for Machine reading comprehension.pdf
  3. BIN docs/Transfer learning for Machine reading comprehension.pptx
  4. +74 −0 docs/acl2018.bib
  5. +539 −0 docs/acl2018.sty
  6. +1,976 −0 docs/acl_natbib.bst
  7. BIN docs/bidaf_architecture.JPG
  8. BIN docs/ccastro_w266_bidaf_transferlearning.pdf
  9. BIN docs/ccastro_w266_bidaf_transferlearning.synctex.gz
  10. +262 −0 docs/ccastro_w266_bidaf_transferlearning.tex
  11. +29 −0 download-marco.sh
  12. BIN marco-data/marco-data-v2.zip
  13. BIN marco-data/marco-data.zip
  14. 0 marco-preprocess/__init__.py
  15. +157 −0 marco-preprocess/aug_squad.py
  16. +271 −0 marco-preprocess/eda_aug_dev.ipynb
  17. +314 −0 marco-preprocess/eda_aug_train.ipynb
  18. +94 −0 marco-preprocess/evaluate-v1.1.py
  19. +94 −0 marco-preprocess/evaluate.py
  20. +242 −0 marco-preprocess/prepro.py
  21. +183 −0 marco-preprocess/prepro_aug.py
  22. +112 −0 marco-preprocess/utils.py
  23. 0 marco/__init__.py
  24. +109 −0 marco/cli.py
  25. +106 −0 marco/ensemble.py
  26. +39 −0 marco/ensemble_fast.py
  27. +422 −0 marco/evaluator.py
  28. +79 −0 marco/graph_handler.py
  29. +230 −0 marco/main.py
  30. +425 −0 marco/model.py
  31. +316 −0 marco/read_data.py
  32. +29 −0 marco/run_ensemble.sh
  33. +27 −0 marco/run_single.sh
  34. +76 −0 marco/templates/visualizer.html
  35. +73 −0 marco/trainer.py
  36. +138 −0 marco/visualizer.py
  37. +1 −0 samples/evaluate_squad.txt
  38. +1 −0 samples/marco_dev_sample.txt
  39. +1 −0 samples/marco_train_sample.txt
  40. +1 −0 samples/squad_train_sample.txt
  41. +3 −1 squad/utils.py
  42. +25 −0 tools/MarcoToSquadConverter/MarcoToSquadConverter.sln
  43. +6 −0 tools/MarcoToSquadConverter/MarcoToSquadConverter/App.config
  44. +63 −0 tools/MarcoToSquadConverter/MarcoToSquadConverter/MarcoToSquadConverter.csproj
  45. +14 −0 tools/MarcoToSquadConverter/MarcoToSquadConverter/MsMarcoPassage.cs
  46. +18 −0 tools/MarcoToSquadConverter/MarcoToSquadConverter/MsMarcoPassageQuestionAndAnswer.cs
  47. +104 −0 tools/MarcoToSquadConverter/MarcoToSquadConverter/Program.cs
  48. +36 −0 tools/MarcoToSquadConverter/MarcoToSquadConverter/Properties/AssemblyInfo.cs
  49. +12 −0 tools/MarcoToSquadConverter/MarcoToSquadConverter/SquadAnswer.cs
  50. +10 −0 tools/MarcoToSquadConverter/MarcoToSquadConverter/SquadDataset.cs
  51. +12 −0 tools/MarcoToSquadConverter/MarcoToSquadConverter/SquadEntry.cs
  52. +12 −0 tools/MarcoToSquadConverter/MarcoToSquadConverter/SquadParagraph.cs
  53. +14 −0 tools/MarcoToSquadConverter/MarcoToSquadConverter/SquadQuestionAndAnswers.cs
  54. +4 −0 tools/MarcoToSquadConverter/MarcoToSquadConverter/packages.config
45 changes: 19 additions & 26 deletions README.md
Original file line number Diff line number Diff line change
@@ -29,6 +29,7 @@ python -m squad.prepro
```

## 2. Training

The model has ~2.5M parameters.
The model was trained with NVidia Titan X (Pascal Architecture, 2016).
The model requires at least 12GB of GPU RAM.
@@ -56,8 +57,8 @@ Note that during the training, the EM and F1 scores from the occasional evaluati
The printed scores are not official (our scoring scheme is a bit harsher).
To obtain the official number, use the official evaluator (copied in `squad` folder, `squad/evaluate-v1.1.py`). For more information See 3.Test.

## 3. Test on the SQuAD dataset

## 3. Test
To test, run:
```
python -m basic.cli
@@ -99,43 +100,35 @@ If you are unfamiliar with CodaLab, follow these simple steps (given that you me
```
If you want to run on GPU, you should run the script sequentially by removing '&' in the forloop, or you will need to specify different GPUs for each run of the for loop.

## Results

### Dev Data
## 4. Train over MS-MARCO

Note these scores are from the official evaluator (copied in `squad` folder, `squad/evaluate-v1.1.py`). For more information See 3.Test.
The scores appeared during the training could be lower than the scores from the official evaluator.
To train over MS-MARCO, copy the documents in `marco-data` within the repository to `$HOME/data/marco`. These documents were created from running the MArcoToSquadConverter tool under `tools` upon the original downloaded MS-MARCO dataset. This filters the questions that we want to study, where the answer is a subspan of the passage, and furthermore modifies the format so it matches that of the SQuAD dataset, and fits the input of the bi-directional attention flow implementation.

| | EM (%) | F1 (%) |
| -------- |:------:|:------:|
| single | 67.7 | 77.3 |
| ensemble | 72.6 | 80.7 |
Preprocess the MS-Marco data:

### Test Data
```
python -m marco.prepro
```

| | EM (%) | F1 (%) |
| -------- |:------:|:------:|
| single | 68.0 | 77.3 |
| ensemble | 73.3 | 81.1 |
Before training, it is recommended to first try the following code to verify everything is okay and memory is sufficient:
```
python -m marco.cli --mode train --debug
```

Refer to [our paper][paper] for more details.
See [SQuAD Leaderboard][squad] to compare with other models.
Then, train the existing model. It is important to not specify `--noload` so that it will load the network we trained with the SQuAD dataset:

```
python -m marco.cli --mode train --debug
```

<!--
## Using Pre-trained Model
To test on MS-MARCO, you can run:

If you would like to use pre-trained model, it's very easy!
You can download the model weights [here][save] (make sure that its commit id matches the source code's).
Extract them and put them in `$PWD/out/basic/00/save` directory, with names unchanged.
Then do the testing again, but you need to specify the step # that you are loading from:
```
python -m basic.cli --mode test --batch_size 8 --eval_num_batches 0 --load_step ####
python -m marco.cli --len_opt --cluster
```
-->

## 5. Multi-GPU Training & Testing

## Multi-GPU Training & Testing
Our model supports multi-GPU training.
We follow the parallelization paradigm described in [TensorFlow Tutorial][multi-gpu].
In short, if you want to use batch size of 60 (default) but if you have 3 GPUs with 4GB of RAM,
Binary file not shown.
Binary file not shown.
74 changes: 74 additions & 0 deletions docs/acl2018.bib
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
@article{conneau:2017,
author = {A. Conneau and others},
year = "2017",
title = {Supervised Learning of Universal Sentence Representations from Natural Language Inference Data}
}

@article{bidaf:2017,
author = {Minjoon Seo and others},
year = "2017",
title = {Bi-directional attention flow for machine comprehension}
}

@article{imagenet,
author = {Jia Deng and others},
title = {ImageNet: A Large-Scale Hierarchical Image Database}
}



@article{transfertextclassification,
author = {Chuong Do and Andrew Ng},
title = {Transfer learning for text classification}
}


@article{msmarco:2016,
author = {Tri Nguyen and others},
year = "2016",
title = {MS MARCO: A Human Generated MAchine Reading COmprehension Dataset}
}


@article{domainadaption,
author = {John Blitzer and others},
title = {Domain Adaptation with Structural Correspondence Learning}
}


@article{deepcontextualizedwr,
author = {Matthew Peters and others},
title = {Deep contextualized word representations}
}


@article{squad:2016,
author = {Pranav Rajpurkar and others},
year = "2016",
title = {SQuAD: 100,000+ Questions for Machine Comprehension of Text}
}


@article{nliwithlstm,
author = {Shuohang Wang and others},
year = "2016",
title = {Learning Natural Language Inference with LSTM}
}


@article{rnet,
author = {Natural Language Computing Group, Microsoft Research Asia},
title = {R-NET: Machine reading comprehension with self-matching networks}
}


@article{matchlstm,
author = {Wang and others},
year = "",
title = {Machine comprehension using match-LSTM and answer pointer}
}

@article{surveytransferlearning,
author = {Sinno Jialin Pan and Qiang Yang},
title = {A Survey on Transfer Learning}
}
Loading