ir_metadata: An Extensible Metadata Schema for Information Retrieval Experiments

<p>This dataset accompanies our work that introduces a metadata schema for TREC run files based on the PRIMAD model. PRIMAD considers essential components of computational experiments that possibly can affect reproducibility on a conceptual level. We propose to align the metadata annotations to the PRIMAD components. In order to demonstrate the potential of metadata annotations, we curated a dataset with run files derived from experiments with different instantiations of PRIMAD components and annotated these with the corresponding metadata. With this work, we hope to stimulate IR researchers to annotate run files and improve the reuse value of experimental artifacts even further.</p> <p>&nbsp;</p> <p>This archive contains the following data:</p> <ul> <li> <p><strong>demo.tar.xz</strong> : Selected annotated runs files that are used in the Colab demonstration.</p> </li> <li> <p><strong>metadata.zip</strong> : YAML files containing only the metadata annotations for each run.</p> </li> <li> <p><strong>runs.zip</strong> : The entire set of run files with annotations.</p> </li> </ul> <p>&nbsp;</p> <p>The annotated runs result from the following experiments:</p> <ul> <li> <p>Grossman and Cormack @ TREC Common Core 2017 <a href="https://trec.nist.gov/pubs/trec26/papers/MRG_UWaterloo-CC.pdf">Paper</a> |&nbsp;<a href="https://trec.nist.gov/">Source</a></p> </li> <li> <p>Grossman and Cormack @ TREC Common Core 2018 <a href="https://trec.nist.gov/pubs/trec27/papers/MRG_UWaterloo-CC.pdf">Paper</a> | <a href="https://trec.nist.gov/">Source</a></p> </li> <li> <p>Yu et al. @ TREC Common Core 2018 <a href="https://trec.nist.gov/pubs/trec27/papers/h2oloo-CC.pdf">Paper</a> | <a href="https://github.com/castorini/Anserini/blob/master/docs/runbook-trec2018-h2oloo.md">Source</a></p> </li> <li> <p>Yu et al. @ ECIR 2019 <a href="https://link.springer.com/chapter/10.1007/978-3-030-15712-8_26">Paper</a> | <a href="https://github.com/castorini/anserini/blob/master/docs/runbook-ecir2019-ccrf.md">Source</a></p> </li> <li> <p>Breuer et al. @ SIGIR 2020 <a href="https://dl.acm.org/doi/10.1145/3397271.3401036">Paper</a> | <a href="https://zenodo.org/record/3856042">Source</a></p> </li> <li> <p>Breuer et al. @ CLEF 2021 <a href="https://link.springer.com/chapter/10.1007/978-3-030-85251-1_5">Paper</a> | <a href="https://zenodo.org/record/4105885">Source</a></p> </li> </ul>