The Software Heritage Graph Dataset

<p>Software Heritage is the largest existing public archive of software source<br> code and accompanying development history: it currently spans more than five<br> billion unique source code files and one billion unique commits, coming from<br> more than 80 million software projects.</p> <p>This is the Software Heritage graph dataset: a fully-deduplicated<br> Merkle DAG representation of the Software Heritage archive. The dataset links<br> together file content identifiers, source code directories, Version Control<br> System (VCS) commits tracking evolution over time, up to the full states of VCS<br> repositories as observed by Software Heritage during periodic crawls. The<br> dataset&rsquo;s contents come from major development forges (including GitHub and<br> GitLab), FOSS distributions (e.g., Debian), and language-specific package<br> managers (e.g., PyPI). &nbsp;Crawling information is also included, providing<br> timestamps about when and where all archived source code artifacts have been<br> observed in the wild.</p> <p>The Software Heritage graph dataset is available in multiple formats, including<br> downloadable CSV dumps and Apache Parquet files for local use, as well as a<br> public instance on Amazon Athena interactive query service for ready-to-use<br> powerful analytical processing.</p> <p>By accessing the dataset, you agree with the Software Heritage&nbsp;<a href="https://www.softwareheritage.org/legal/users-ethical-charter/">Ethical Charter<br> for using the archive data</a>, and the&nbsp;<a href="https://www.softwareheritage.org/legal/bulk-access-terms-of-use/">terms of use for bulk access</a>.</p> <p>If you use this dataset for research purposes, please cite the following paper:</p> <ul> <li>Antoine Pietri, Diomidis Spinellis, Stefano Zacchiroli.&nbsp;<br> <em>The Software Heritage Graph Dataset: Public software development under one roof</em>.&nbsp;<br> In proceedings of&nbsp;<a href="http://2019.msrconf.org/">MSR 2019</a>: The 16th International Conference on Mining Software Repositories, May 2019, Montreal, Canada. Co-located with&nbsp;<a href="https://2019.icse-conferences.org/">ICSE 2019</a>.&nbsp;<br> <a href="https://upsilon.cc/~zack/research/publications/msr-2019-swh.pdf">preprint</a>,&nbsp;<a href="https://upsilon.cc/~zack/research/publications/msr-2019-swh.bib">bibtex</a></li> </ul> <p>You can also refer to the above paper for more information the dataset and sample queries.</p>