Skip to content

Latest commit

 

History

History
91 lines (63 loc) · 4.32 KB

regressions-msmarco-passage-deepimpact.md

File metadata and controls

91 lines (63 loc) · 4.32 KB

Anserini: Regressions for DeepImpact on MS MARCO Passage

This page documents regression experiments for DeepImpact on the MS MARCO Passage Ranking Task, which is integrated into Anserini's regression testing framework. DeepImpact is described in the following paper:

Antonio Mallia, Omar Khattab, Nicola Tonellotto, and Torsten Suel. Learning Passage Impacts for Inverted Indexes. SIGIR 2021.

For more complete instructions on how to run end-to-end experiments, refer to this page.

The exact configurations for these regressions are stored in this YAML file. Note that this page is automatically generated from this template as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead.

Indexing

Typical indexing command:

nohup sh target/appassembler/bin/IndexCollection -collection JsonVectorCollection \
 -input /path/to/msmarco-passage-deepimpact \
 -index indexes/lucene-index.msmarco-passage-deepimpact.raw \
 -generator DefaultLuceneDocumentGenerator \
 -threads 16 -impact -pretokenized -storeRaw \
  >& logs/log.msmarco-passage-deepimpact &

The directory /path/to/msmarco-passage-deepimpact/ should be a directory containing the compressed jsonl files that comprise the corpus. See this page for additional details.

For additional details, see explanation of common indexing options.

Retrieval

Topics and qrels are stored in src/main/resources/topics-and-qrels/. The regression experiments here evaluate on the 6980 dev set questions; see this page for more details.

After indexing has completed, you should be able to perform retrieval as follows:

nohup target/appassembler/bin/SearchCollection -index indexes/lucene-index.msmarco-passage-deepimpact.raw \
 -topicreader TsvInt -topics src/main/resources/topics-and-qrels/topics.msmarco-passage.dev-subset.deepimpact.tsv.gz \
 -output runs/run.msmarco-passage-deepimpact.deepimpact.topics.msmarco-passage.dev-subset.deepimpact.tsv.gz \
 -impact -pretokenized &

Evaluation can be performed using trec_eval:

tools/eval/trec_eval.9.0.4/trec_eval -m map -c -m recip_rank -c -m recall.1000 -c src/main/resources/topics-and-qrels/qrels.msmarco-passage.dev-subset.txt runs/run.msmarco-passage-deepimpact.deepimpact.topics.msmarco-passage.dev-subset.deepimpact.tsv.gz

Effectiveness

With the above commands, you should be able to reproduce the following results:

MAP DeepImpact
MS MARCO Passage: Dev 0.3334
MRR DeepImpact
MS MARCO Passage: Dev 0.3386
R@1000 DeepImpact
MS MARCO Passage: Dev 0.9476

The above runs are in TREC output format and evaluated with trec_eval. In order to reproduce results reported in the paper, we need to convert to MS MARCO output format and then evaluate:

python tools/scripts/msmarco/convert_trec_to_msmarco_run.py \
   --input runs/run.msmarco-passage-deepimpact.deepimpact.topics.msmarco-passage.dev-subset.deep-impact.tsv.gz \
   --output runs/run.msmarco-passage-deepimpact.deepimpact.topics.msmarco-passage.dev-subset.deep-impact.tsv.gz.msmarco --quiet

python tools/scripts/msmarco/msmarco_passage_eval.py \
   collections/msmarco-passage/qrels.dev.small.tsv \
   runs/run.msmarco-passage-deepimpact.deepimpact.topics.msmarco-passage.dev-subset.deep-impact.tsv.gz.msmarco

The results should be as follows:

#####################
MRR @10: 0.3252764133351524
QueriesRanked: 6980
#####################

The final evaluation metric is very close to the one reported in the paper (0.326).