Crowdsourcing StoryLines with CrowdTruth

<p>Crowdsourced ground truth dataset for 1,204 sentences and 7,778 event pairs covering 22 news topics. The corpus was created by using the CrowdTruth methodology, as described in the following paper:</p> <ul> <li>Tommaso Caselli and Oana Inel: Crowdsourcing StoryLines: Harnessing the Crowd for Causal Relation Annotation. Events and Stories in the News Workshops, COLING 2018</li> </ul> <p>If you find this data useful in your research, please consider citing:</p> <pre>@inproceedings{caselli2018crowdsourcing, &nbsp; title={Crowdsourcing StoryLines: Harnessing the Crowd for Causal Relation Annotation}, &nbsp; author={Caselli, Tommaso and Inel, Oana}, &nbsp; booktitle={Proceedings of the Workshop Events and Stories in the News 2018}, &nbsp; pages={44--54}, &nbsp; year={2018} } </pre> <p>Crowdsourcing results and evaluation against expert data are available in folder:&nbsp;<code>|--data/results/</code></p> <p>Expert ground truth data is available in folder:&nbsp;<code>|--data/ground_truth/</code></p> <p>Aggregated raw crowdsourcig data is available in folder:&nbsp;<code>|--data/aggregated_input/</code></p> <p>Raw crowdsourcig data is available in folder:&nbsp;<code>|--data/input/</code></p> <p>&nbsp;</p> <p><strong>Running the notebooks</strong></p> <p>To run and regenerate the results, you need to install the stable version of the&nbsp;<em><strong>crowdtruth==2.0</strong></em>&nbsp;package from PyPI using:</p> <p>pip install crowdtruth==2.0</p> <pre>&nbsp;</pre>