TweetNERD - End to End Entity Linking Benchmark for Tweets

<p>TweetNERD - End to End Entity Linking Benchmark for Tweets</p> <p><a href="https://arxiv.org/abs/2210.08129">Paper</a>&nbsp;- <a href="https://www.youtube.com/watch?v=H5ypIHterWQ">Video</a>&nbsp;- <a href="https://neurips.cc/virtual/2022/poster/55647">Neurips Page</a></p> <p>This is the dataset described in the paper <a href="https://arxiv.org/abs/2210.08129"><strong>TweetNERD - End to End Entity Linking Benchmark for Tweets</strong></a> (accepted to <a href="https://datasets-benchmarks-proceedings.neurips.cc/paper/2022">Thirty-sixth Conference on Neural Information Processing Systems (Neurips) Datasets and Benchmarks Track</a>).</p> <blockquote> <p>Named Entity Recognition and Disambiguation (NERD) systems are foundational for information retrieval, question answering, event detection, and other natural language processing (NLP) applications. We introduce TweetNERD, a dataset of 340K+ Tweets across 2010-2021, for benchmarking NERD systems on Tweets. This is the largest and most temporally diverse open sourced dataset benchmark for NERD on Tweets and can be used to facilitate research in this area.</p> </blockquote> <p><strong>UPDATE: The new version contains an additional ~125K Tweets leading to a total dataset size of ~465K Tweets.</strong></p> <p>TweetNERD dataset is released under <a href="https://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution 4.0 International (CC BY 4.0)</a> LICENSE.</p> <p>The license only applies to the data files present in this dataset. See <strong>Data usage policy</strong> below.</p> <p>Check out more details at <a href="https://github.com/twitter-research/TweetNERD">https://github.com/twitter-research/TweetNERD</a></p> <p><strong>Usage</strong></p> <p>We provide the dataset split across the following tab seperated files:</p> <ul> <li><strong>OOD.public.tsv</strong>: OOD split of the data in the paper.</li> <li><strong>Academic.public.tsv</strong>: Academic split of the data described in the paper.</li> <li><code>part_*.public.tsv</code>: Remaining data split into parts in no particular order.</li> </ul> <p>Each file is tab separated and has has the following format:</p> <table> <thead> <tr> <th>tweet_id</th> <th>phrase</th> <th>start</th> <th>end</th> <th>entityId</th> <th>score</th> </tr> </thead> <tbody> <tr> <td>22</td> <td>twttr</td> <td>20</td> <td>25</td> <td>Q918</td> <td>3</td> </tr> <tr> <td>21</td> <td>twttr</td> <td>20</td> <td>25</td> <td>Q918</td> <td>3</td> </tr> <tr> <td>1457198399032287235</td> <td>Diwali</td> <td>30</td> <td>38</td> <td>Q10244</td> <td>3</td> </tr> <tr> <td>1232456079247736833</td> <td>NO_PHRASE</td> <td>-1</td> <td>-1</td> <td>NO_ENTITY</td> <td>-1</td> </tr> </tbody> </table> <p>For tweets which don&#39;t have any entity, their column values for <code>phrase, start, end, entityId, score</code> are set <code>NO_PHRASE, -1, -1, NO_ENTITY, -1</code> respectively.</p> <p>Description of file columns is as follows:</p> <table> <thead> <tr> <th>Column</th> <th>Type</th> <th>Missing Value</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td>tweet_id</td> <td>string</td> <td>&nbsp;</td> <td>ID of the Tweet</td> </tr> <tr> <td>phrase</td> <td>string</td> <td>NO_PHRASE</td> <td>entity phrase</td> </tr> <tr> <td>start</td> <td>int</td> <td>-1</td> <td>start offset of the phrase in text using <code>UTF-16BE</code> encoding</td> </tr> <tr> <td>end</td> <td>int</td> <td>-1</td> <td>end offset of the phrase in the text using <code>UTF-16BE</code> encoding</td> </tr> <tr> <td>entityId</td> <td>string</td> <td>NO_ENTITY</td> <td>Entity ID. If not missing can be NOT FOUND, AMBIGUOUS, or Wikidata ID of format Q{numbers}, e.g. Q918</td> </tr> <tr> <td>score</td> <td>int</td> <td>-1</td> <td>Number of annotators who agreed on the phrase, start, end, entityId information</td> </tr> </tbody> </table> <p>In order to use the dataset you need to utilize the <code>tweet_id</code> column and get the Tweet text using the <a href="https://developer.twitter.com/en/docs/twitter-api">Twitter API</a> (See <strong>Data usage policy</strong> section below).</p> <p>Data stats</p> <table> <thead> <tr> <th>Split</th> <th>Number of Rows</th> <th>Number unique tweets</th> </tr> </thead> <tbody> <tr> <td>OOD</td> <td>34102</td> <td>25000</td> </tr> <tr> <td>Academic</td> <td>51685</td> <td>30119</td> </tr> <tr> <td>part_0</td> <td>11830</td> <td>10000</td> </tr> <tr> <td>part_1</td> <td>35681</td> <td>25799</td> </tr> <tr> <td>part_2</td> <td>34256</td> <td>25000</td> </tr> <tr> <td>part_3</td> <td>36478</td> <td>25000</td> </tr> <tr> <td>part_4</td> <td>37518</td> <td>24999</td> </tr> <tr> <td>part_5</td> <td>36626</td> <td>25000</td> </tr> <tr> <td>part_6</td> <td>34001</td> <td>24984</td> </tr> <tr> <td>part_7</td> <td>34125</td> <td>24981</td> </tr> <tr> <td>part_8</td> <td>32556</td> <td>25000</td> </tr> <tr> <td>part_9</td> <td>32657</td> <td>25000</td> </tr> <tr> <td>part_10</td> <td>32442</td> <td>25000</td> </tr> <tr> <td>part_11</td> <td>32033</td> <td>24972</td> </tr> <tr> <td>part_12</td> <td>76559</td> <td>25000</td> </tr> <tr> <td>part_13</td> <td>67240</td> <td>24920</td> </tr> <tr> <td>part_14</td> <td>67745</td> <td>25000</td> </tr> <tr> <td>part_15</td> <td>67652</td> <td>25000</td> </tr> <tr> <td>part_16</td> <td>65739</td> <td>25000</td> </tr> </tbody> </table> <p>Data usage policy</p> <p>Use of this dataset is subject to you obtaining lawful access to the <a href="https://developer.twitter.com/en/docs/twitter-api">Twitter API</a>, which requires you to agree to the <a href="https://developer.twitter.com/en/developer-terms/">Developer Terms Policies and Agreements</a>.</p> <p>Please cite the following if you use TweetNERD in your paper:</p> <pre>@dataset{TweetNERD_Zenodo_2022_6617192, author = {Mishra, Shubhanshu and Saini, Aman and Makki, Raheleh and Mehta, Sneha and Haghighi, Aria and Mollahosseini, Ali}, title = {{TweetNERD - End to End Entity Linking Benchmark for Tweets}}, month = jun, year = 2022, note = {{Data usage policy Use of this dataset is subject to you obtaining lawful access to the [Twitter API](https://developer.twitter.com/en/docs /twitter-api), which requires you to agree to the [Developer Terms Policies and Agreements](https://developer.twitter.com/en /developer-terms/).}}, publisher = {Zenodo}, version = {0.0.0}, doi = {10.5281/zenodo.6617192}, url = {https://doi.org/10.5281/zenodo.6617192} } @inproceedings{TweetNERDNeurips2022, author = {Mishra, Shubhanshu and Saini, Aman and Makki, Raheleh and Mehta, Sneha and Haghighi, Aria and Mollahosseini, Ali}, booktitle = {Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks}, pages = {}, title = {TweetNERD - End to End Entity Linking Benchmark for Tweets}, volume = {2}, year = {2022}, eprint = {arXiv:2210.08129}, doi = {10.48550/arXiv.2210.08129} } </pre>