unarXive: All arXiv Publications Pre-Processed for NLP, Including Structured Full-Text and Citation Network (full)

<h2><strong>Description</strong></h2><p>unarXive is a scholarly data set containing publications' structured full-text, annotated in-text citations, linked non-text content (mathematical notation, figure/table captions) and a citation network.</p><p>The data is generated from all LaTeX sources on <a href="https://arxiv.org/">arXiv</a> and therefore of higher quality than data generated from PDF files.</p><p>Typical uses are</p><ul><li>Training of ML models (citation recommendation, summarization, LLMs)</li><li>Citation context analysis</li><li>Bibliographic analyses</li></ul><h2><strong>Access</strong></h2><p>┏━━━━━━━━━━━━━━━━━━━━━━━━━━┓<br>┃ &nbsp;<a href="https://github.com/IllDepence/unarXive/raw/master/doc/unarXive_data_sample.tar.gz"><strong>D O W N L O A D &nbsp; S A M P L E</strong></a> &nbsp; ┃<br>┗━━━━━━━━━━━━━━━━━━━━━━━━━━┛</p><p>To download the whole data set send an access request and note the following:</p><blockquote><p><strong>Note</strong>: this Zenodo record is the "full" version of unarXive, which was generated from all of arXiv.org <i>including non-permissively licensed papers</i>. Make sure that your use of the data is compliant with the paper's licensing terms.¹<br>Alternatively you can use the <a href="https://doi.org/10.5281/zenodo.7752615">unarXive open subset</a>.</p><p>¹ For information on papers' licenses use <a href="https://info.arxiv.org/help/bulk_data/index.html">arXiv's bulk metadata access</a>.</p></blockquote><p>The code for generating the data set is <a href="https://github.com/IllDepence/unarXive">publicly available</a>.</p>