Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
6
datasets available to search
ShareScore release 0.9.0
Dataset results
6 results for “software heritage”
Cartolabe Software Heritage
<p>The Software Heritage dataset shows projects that are collected by the <a href="https://www.softwareheritage.org/">Software Heritage</a> archive initiative. Text data for each software comes from the project's description on github.</p>
The Software Heritage Graph Dataset
<p>Software Heritage is the largest existing public archive of software source<br> code and accompanying development history: it currently spans more than five<br> billion unique source code files and one billion unique commits, coming from<br> more than 80 million software projects.</p> <p>This is the Software Heritage graph dataset: a fully-deduplicated<br> Merkle DAG representation of the Software Heritage archive. The dataset links<br> together file content identifiers, source code directories, Version Control<br> System (VCS) commits tracking evolution over time, up to the full states of VCS<br> repositories as observed by Software Heritage during periodic crawls. The<br> dataset’s contents come from major development forges (including GitHub and<br> GitLab), FOSS distributions (e.g., Debian), and language-specific package<br> managers (e.g., PyPI). Crawling information is also included, providing<br> timestamps about when and where all archived source code artifacts have been<br> observed in the wild.</p> <p>The Software Heritage graph dataset is available in multiple formats, including<br> downloadable CSV dumps and Apache Parquet files for local use, as well as a<br> public instance on Amazon Athena interactive query service for ready-to-use<br> powerful analytical processing.</p> <p>By accessing the dataset, you agree with the Software Heritage <a href="https://www.softwareheritage.org/legal/users-ethical-charter/">Ethical Charter<br> for using the archive data</a>, and the <a href="https://www.softwareheritage.org/legal/bulk-access-terms-of-use/">terms of use for bulk access</a>.</p> <p>If you use this dataset for research purposes, please cite the following paper:</p> <ul> <li>Antoine Pietri, Diomidis Spinellis, Stefano Zacchiroli. <br> <em>The Software Heritage Graph Dataset: Public software development under one roof</em>. <br> In proceedings of <a href="http://2019.msrconf.org/">MSR 2019</a>: The 16th International Conference on Mining Software Repositories, May 2019, Montreal, Canada. Co-located with <a href="https://2019.icse-conferences.org/">ICSE 2019</a>. <br> <a href="https://upsilon.cc/~zack/research/publications/msr-2019-swh.pdf">preprint</a>, <a href="https://upsilon.cc/~zack/research/publications/msr-2019-swh.bib">bibtex</a></li> </ul> <p>You can also refer to the above paper for more information the dataset and sample queries.</p>
Mapping of swMATH ids and Software Heritage ids
<p>We present the mapping from swMATH identifiers and software heritage identifiers for selected git repositories, with the following fields:</p> <p><strong>swmathid</strong>: Unique identifier from swMATH. Prefix with https://swmath.org/software/ to visit resolve. For example, 4503 is associated with https://swmath.org/software/4503 <br><strong>cvs</strong>: The link to the git repository<br><strong>swhid</strong>: Unique software heritage identifer. Prefix with <code>https://archive.softwareheritage.org/</code> to resolve. For example, swh:1:snp:e9e31cc35f3677801b896fc4d84f1c26fd3df3d2 is assocated with <code>https://archive.softwareheritage.org/</code>swh:1:snp:e9e31cc35f3677801b896fc4d84f1c26fd3df3d2 </p> <p><span>The following software versions were used</span></p> <ol> <li><span><a href="https://archive.softwareheritage.org/swh:1:cnt:3c525f7ef4feb933692f2d33d8186f69481d248b;origin=https:/github.com/MaRDI4NFDI/swMATH4EOSC;visit=swh:1:snp:ef1fa467e1a15ad613cf328b4b9bfe6db5be7f9d;anchor=swh:1:rev:8d7031d00664ca66e9b50ea4631cf297d1f16dc6;path=/swMATH_analysis/swMATH_no_empty_source_code.csv"><span>swh:1:cnt:3c525f7ef4feb933692f2d33d8186f69481d248b</span></a> to get the swmath and git repositories</span></li> <li><span><a href="https://archive.softwareheritage.org/swh:1:cnt:3aea0d0f97d9276282ee1de5ebc5945cc66e77b5;origin=https:/phabricator.wikimedia.org/diffusion/EMAS/extension-mathsearch.git;visit=swh:1:snp:2ba33a0d4adb14c00a63ab4cfb595412c3e47cbc;anchor=swh:1:rev:4f2f4b12f4763a817e8533e2997ece9407cd0b04;path=/maintenance/AddSwhids.php" target="_blank" rel="noopener"><span>swh:1:cnt:3aea0d0f97d9276282ee1de5ebc5945cc66e77b5</span></a> to get the corresponding software heritage identifiers</span></li> </ol> <p> </p> <p>Then the following Query was used to retrieve the uploaded file.</p> <p>SPARQL Query (<a href="https://query.portal.mardi4nfdi.de/#PREFIX%20wdt%3A%20%3Chttps%3A%2F%2Fportal.mardi4nfdi.de%2Fprop%2Fdirect%2F%3E%0A%0ASELECT%20%3Fswmathid%20%20%28str%28%3Frepo%29%20as%20%3Fcvs%29%20%3Fswhid%0AWHERE%20%7B%0A%20%20%3Fitem%20wdt%3AP13%20%3Fswmathid.%0A%20%20%3Fitem%20wdt%3AP339%20%3Frepo.%0A%20%20%3Fitem%20wdt%3AP1454%20%3Fswhid%0A%7D" target="_blank" rel="noopener">try it out</a>)</p> <pre><code>PREFIX wdt: <https://portal.mardi4nfdi.de/prop/direct/> SELECT ?swmathid (str(?repo) as ?cvs) ?swhid WHERE { ?item wdt:P13 ?swmathid. ?item wdt:P339 ?repo. ?item wdt:P1454 ?swhid }<br><br></code>For more details see our upcoming paper.</pre>
Index of software revisions/commits archived by Software Heritage
<p><a href="https://www.softwareheritage.org">Software Heritage</a> is a comprehensive archive of publicly available software source code together with its development history, as captured by modern version control system.</p> <p>This dataset provides a full index of the revision (AKA "commits") identifiers archived by Software Heritage at a given point in time. Each revision identifier can be used to fetch complete revision information via either the Software Heritage API from the <a href="https://archive.softwareheritage.org">main archive</a> or as a <a href="https://docs.softwareheritage.org/devel/swh-model/persistent-identifiers.html">persistent identifier</a> to retrieve the same information from any existing Software Heritage mirrors.</p> <p>Also, revision identifiers are compatible with the identifiers used by the Git version control system, so that they can be cross-referenced with any existing Git repository.</p> <p><strong>Note</strong>: the authors of the dataset are currently anonymous for double-blind review needs.</p>
The Software Heritage License Dataset (2022 Edition)
<p>This dataset contains all “license files” extracted from a snapshot of the <a href="https://www.softwareheritage.org">Software Heritage</a> archive taken on 2022-04-25. (Other, possibly more recent, versions of the datasets can be found at <a href="https://annex.softwareheritage.org/public/dataset/license-blobs/">https://annex.softwareheritage.org/public/dataset/license-blobs/</a>).</p> <p>In this context, a <em>license file</em> is a unique file content (or “blob”) that appeared in a software origin archived by Software Heritage as a file whose name is often used to ship licenses in software projects. Some name examples are: <code>COPYING</code>, <code>LICENSE</code>, <code>NOTICE</code>, <code>COPYRIGHT</code>, etc. The exact file name pattern used to select the blobs contained in the dataset can be found in the SQL query file <code>01-select-blobs.sql</code>. Note that the file name was not expected to be at the project root, because project subdirectories can contain different licenses than the top-level one, and we wanted to include those too.</p> <p><strong>Format</strong></p> <p>The dataset is organized as follows:</p> <ul> <li> <p><code>blobs.tar.zst</code>: a <a href="http://facebook.github.io/zstd/">Zst</a>-compressed tarball containing deduplicated license blobs, one per file. The tarball contains 6’859’189 blobs, for a total uncompressed size on disk of 66 GiB.</p> <p>The blobs are organized in a sharded directory structure that contains files named like <code>blobs/86/24/8624bcdae55baeef00cd11d5dfcfa60f68710a02</code>, where:</p> <ul> <li> <p><code>blobs/</code> is the root directory containing all license blobs</p> </li> <li> <p><code>8624bcdae55baeef00cd11d5dfcfa60f68710a02</code> is the SHA1 checksum of a specific license blobs, a copy of the GPL3 license in this case. Each license blob is ultimately named with its SHA1:</p> <pre><code>$ head -n 3 blobs/86/24/8624bcdae55baeef00cd11d5dfcfa60f68710a02 GNU GENERAL PUBLIC LICENSE Version 3, 29 June 2007 $ sha1sum blobs/86/24/8624bcdae55baeef00cd11d5dfcfa60f68710a02 8624bcdae55baeef00cd11d5dfcfa60f68710a02 blobs/86/24/8624bcdae55baeef00cd11d5dfcfa60f68710a02</code></pre> </li> <li> <p><code>86</code> and <code>24</code> are, respectively, the first and second group of two hex digits in the blob SHA1</p> </li> </ul> <p>One blob is missing, because its size (313MB) prevented its inclusion; (it was originally a tarball containing source code):</p> <pre><code>swh:1:cnt:61bf63793c2ee178733b39f8456a796b72dc8bde,1340d4e2da173c92d432026ecdc54b4859fe9911,"AUTHORS"</code></pre> </li> <li> <p><code>blobs-sample20k.tar.zst</code>: analogous to <code>blobs.tar.zst</code>, but containing “only” 20’000 randomly selected license blobs</p> </li> <li> <p><code>license-blobs.csv.zst</code> a <a href="http://facebook.github.io/zstd/">Zst</a>-compressed <a href="https://en.wikipedia.org/wiki/Comma-separated_values">CSV</a> index of all the blobs in the dataset. Each line in the index (except the first one, which contains column headers) describes a license blob and is in the format <code>SWHID,SHA1,NAME</code>, for example:</p> <pre><code> swh:1:cnt:94a9ed024d3859793618152ea559a168bbcbb5e2,8624bcdae55baeef00cd11d5dfcfa60f68710a02,"COPYING" swh:1:cnt:94a9ed024d3859793618152ea559a168bbcbb5e2,8624bcdae55baeef00cd11d5dfcfa60f68710a02,"COPYING.GPL3" swh:1:cnt:94a9ed024d3859793618152ea559a168bbcbb5e2,8624bcdae55baeef00cd11d5dfcfa60f68710a02,"COPYING.GLP-3"</code></pre> <p>where:</p> <ul> <li> <p><strong>SWHID:</strong> the <a href="https://docs.softwareheritage.org/devel/swh-model/persistent-identifiers.html">Software Heritage persistent identifier</a> of the blob. It can be used to retrieve and cross-reference the license blob via the Software Heritage archive, e.g., at: <a href="https://archive.softwareheritage.org/swh:1:cnt:94a9ed024d3859793618152ea559a168bbcbb5e2">https://archive.softwareheritage.org/swh:1:cnt:94a9ed024d3859793618152ea559a168bbcbb5e2</a></p> </li> <li> <p><strong>SHA1:</strong> the blob SHA1, that can be used to cross-reference blobs in the <code>blobs/</code> directory</p> </li> <li> <p><strong>NAME:</strong> <em>a</em> file name given to the license blob in a given software origin. As the same license blob can have different names in different contexts, the index contain multiple entries for the same blob with different names, as it is the case in the example above (yes, one of those has a typo in it, but it’s an <em>original</em> typo from some repository!).</p> </li> </ul> </li> <li> <p><code>blobs-fileinfo.csv.zst</code> a <a href="http://facebook.github.io/zstd/">Zst</a>-compressed <a href="https://en.wikipedia.org/wiki/Comma-separated_values">CSV</a> mapping from blobs to basic file information in the format: <code>SHA1,MIME_TYPE,ENCODING,LINE_COUNT,WORD_COUNT,SIZE</code>, where:</p> <ul> <li><strong>SHA1:</strong> blob SHA1</li> <li><strong>MIME_TYPE:</strong> blob MIME type, as detected by <a href="https://pubs.opengroup.org/onlinepubs/9699919799/utilities/file.html">libmagic</a></li> <li><strong>ENCODING:</strong> blob character encoding, as detected by <a href="https://pubs.opengroup.org/onlinepubs/9699919799/utilities/file.html">libmagic</a></li> <li><strong>LINE_COUNT:</strong> number of lines in the blob (only for textual blobs with UTF8 encoding)</li> <li><strong>WORD_COUNT:</strong> number of words in the blob (only for textual blobs with UTF8 encoding)</li> <li><strong>SIZE:</strong> blob size in bytes</li> </ul> </li> <li> <p><code>blobs-scancode.csv.zst</code> a <a href="http://facebook.github.io/zstd/">Zst</a>-compressed <a href="https://en.wikipedia.org/wiki/Comma-separated_values">CSV</a> mapping from blobs to software license detected in them by <a href="https://www.aboutcode.org/projects/scancode.html">ScanCode</a>, in the format: <code>SHA1,LICENSE,SCORE</code>, where:</p> <ul> <li><strong>SHA1:</strong> blob SHA1</li> <li><strong>LICENSE:</strong> license detected in the blob, as an <a href="https://spdx.dev/">SPDX</a> identifier (or <a href="https://scancode-licensedb.aboutcode.org/">ScanCode identifier</a> for non-SPDX-indexed licenses)</li> <li><strong>SCORE:</strong> confidence score in the result, as a decimal number between 0 and 100</li> </ul> <p>There may be zero or arbitrarily many lines for each blob.</p> </li> <li> <p><code>blobs-scancode.ndjson.zst</code> a <a href="http://facebook.github.io/zstd/">Zst</a>-compressed <a href="http://ndjson.org/">line-delimited JSON</a>, containing a superset of the information in <code>blobs-scancode.csv.zst</code>. Each line is a JSON dictionary with three keys:</p> <ul> <li><strong>sha1</strong>: blob SHA1</li> <li><strong>licenses</strong>: output of <code>scancode.api.get_licenses(..., min_score=0)</code></li> <li><strong>copyrights</strong>: output of <code>scancode.api.get_copyrights(...)</code></li> </ul> <p>There is exactly one line for each blob. <code>licenses</code> and <code>copyrights</code> keys are omitted for files not detected as plain text.</p> </li> <li> <p><code>blobs-origins.csv.zst</code> a <a href="http://facebook.github.io/zstd/">Zst</a>-compressed <a href="https://en.wikipedia.org/wiki/Comma-separated_values">CSV</a> mapping of where license blobs come from. Each line in the index associate a license blob to one of its origins in the format <code>SWHID<TAB>URL</code>, for example:</p> <pre><code> swh:1:cnt:94a9ed024d3859793618152ea559a168bbcbb5e2 https://github.com/pombreda/Artemis</code></pre> <p>Note that a license blob can come from many different places, only an arbitrary (and somewhat random) one is listed in this mapping.</p> <p>If no origin URL is found in the Software Heritage archive, then a blank is used instead. This happens when they were either being loaded when the dataset was generated, or the loader process crashed before completing the blob’s origin’s ingestion.</p> </li> <li> <p><code>blobs-nb-origins.csv.zst</code> a <a href="http://facebook.github.io/zstd/">Zst</a>-compressed <a href="https://en.wikipedia.org/wiki/Comma-separated_values">CSV</a> mapping of how many origins of this blob are known to Software Heritage. Each line in the index associate a license blob to this count in the format <code>SWHID<TAB>NUMBER</code>, for example:</p> <pre><code> swh:1:cnt:94a9ed024d3859793618152ea559a168bbcbb5e2 2822260</code></pre> <p>Two blobs are missing because the computation crashes:</p> <pre><code> swh:1:cnt:e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 swh:1:cnt:8b137891791fe96927ad78e64b0aad7bded08bdc</code></pre> <p>This issue will be fixed in a future version of the dataset</p> </li> <li> <p><code>blobs-earliest.csv.zst</code> a <a href="http://facebook.github.io/zstd/">Zst</a>-compressed <a href="https://en.wikipedia.org/wiki/Comma-separated_values">CSV</a> mapping from blobs to information about their (earliest) known occurence(s) in the archive. Format: <code>SWHID<TAB>EARLIEST_SWHID<TAB>EARLIEST_TS<TAB>OCCURRENCES</code>, where:</p> <ul> <li><strong>SWHID:</strong> blob SWHID</li> <li><strong>EARLIEST_SWHID:</strong> SWHID of the earliest known commit containing the blob</li> <li><strong>EARLIEST_TS:</strong> timestamp of the earliest known commit containing the blob, as a <a href="https://en.wikipedia.org/wiki/Unix_time">Unix time</a> integer</li> <li><strong>OCCURRENCES:</strong> number of known commits containing the blob</li> </ul> </li> <li> <p><code>replication-package.tar.gz</code>: code and scripts used to produce the dataset</p> </li> <li> <p><code>licenses-annotated-sample.tar.gz</code>: ground truth, i.e., manually annotated random sample of license blobs, with details about the kind of information they contain.</p> </li> </ul> <p><strong>Changes since the 2021-03-23 dataset</strong></p> <ul> <li> <p>More input data, due to the SWH archive growing: more origins in supported forges and package managers; and support for more forges and package managers. See the <a href="https://docs.softwareheritage.org/devel/archive-changelog.html">SWH Archive Changelog</a> for details.</p> </li> <li> <p>Values in the <strong>NAME</strong> column of <code>license-blobs.csv.zst</code> are quoted, as some file names now contain commas.</p> </li> <li> <p>Replication package now contains all the steps needed to reproduce all artefacts including the <code>licenseblobs/fetch.py</code> script.</p> </li> <li> <p><code>blobs-nb-origins.csv.zst</code> is added.</p> </li> <li> <p><code>blobs-origins.csv.zst</code> is now generated using the first origin returned by swh-graph’s <code>leaves</code> endpoint, instead of its <code>randomwalk</code> endpoint. This should have no impact on the result, other than a different distribution of “random” origins being picked.</p> </li> <li> <p><code>blobs-origins.csv.zst</code> was missing ~10% of its results in previous versions of the dataset, due to errors and/or timeouts in its generation, this is now down to 0.02% (1254 of the 6859445 unique blobs). Blobs with no known origins are now present, with a blank instead of URL.</p> </li> <li> <p><code>blobs-earliest.csv.zst</code> was missing ~10% of its results in previous versions of the dataset. It is complete now.</p> </li> <li> <p><code>blobs-scancode.csv.zst</code> is generated with a newer scancode-toolkit version (31.2.1)</p> </li> <li> <p><code>blobs-scancode.ndjson.zst</code> is added.</p> </li> </ul> <p><strong>Errata</strong></p> <p>A file name <code>.tmp_1340d4e2da173c92d432026ecdc54b4859fe9911</code> was present in the initial version of the dataset (published on 2022-11-07). It was removed on 2022-11-09 using these two commands:</p> <pre><code>pv blobs-fileinfo.csv.zst | zstdcat | grep -v "\.tmp" | zstd -19 pv blobs.tar.zst| zstdcat | tar --delete blobs/13/40/.tmp_1340d4e2da173c92d432026ecdc54b4859fe9911 | zstd -19 -T12</code></pre> <p>The total uncompressed size was announced as 84 GiB based on the physical size on ext4, but it is actually 66 GiB.</p> <p><strong>Citation</strong></p> <p>If you use this dataset for research purposes, please acknowledge its use by citing one or both of the following papers:</p> <ul> <li> <p>[<a href="https://arxiv.org/abs/2308.11258">pdf</a>, <a href="https://dblp.uni-trier.de/rec/journals/ese/GonzalezBarahonaLRZ23.html?view=bibtex">bib</a>] Jesús M. González-Barahona, Sergio Raúl Montes León, Gregorio Robles, Stefano Zacchiroli. <a href="https://doi.org/10.1007/s10664-023-10377-w"><em>The software heritage license dataset (2022 edition)</em></a>. Empirical Software Engineering, Volume 28, Number 6, Article number 147 (2023).</p> </li> <li>[<a href="https://arxiv.org/pdf/2204.00256.pdf">pdf</a>, <a href="https://dblp.uni-trier.de/rec/conf/msr/Zacchiroli22.html?view=bibtex">bib</a>] Stefano Zacchiroli. <a href="https://doi.org/10.1145/3524842.3528491"><em>A Large-scale Dataset of (Open Source) License Text Variants</em></a>. In proceedings of the <a href="https://conf.researchr.org/home/msr-2022">2022 Mining Software Repositories Conference (MSR 2022)</a>. 23-24 May 2022 Pittsburgh, Pennsylvania, United States. ACM 2022.</li> </ul> <p><strong>References</strong></p> <p>The dataset has been built using primarily the data sources described in the following papers:</p> <ul> <li> <p>[<a href="https://phaidra.univie.ac.at/open/o:931064">pdf</a>, <a href="https://dblp.uni-trier.de/rec/conf/ipres/CosmoZ17.html?view=bibtex">bib</a>] Roberto Di Cosmo, Stefano Zacchiroli. <a href="https://hdl.handle.net/11353/10.931064">Software Heritage: Why and How to Preserve Software Source Code</a>. In Proceedings of iPRES 2017: 14th International Conference on Digital Preservation, Kyoto, Japan, 25-29 September 2017.</p> </li> <li> <p>[<a href="https://upsilon.cc/~zack/research/publications/msr-2019-swh.pdf">pdf</a>, <a href="https://dblp.uni-trier.de/rec/conf/msr/PietriSZ19.html?view=bibtex">bib</a>] Antoine Pietri, Diomidis Spinellis, Stefano Zacchiroli. <a href="https://doi.org/10.1109/MSR.2019.00030">The Software Heritage Graph Dataset: Public software development under one roof</a>. In proceedings of MSR 2019: The 16th International Conference on Mining Software Repositories, May 2019, Montreal, Canada. Pages 138-142, IEEE 2019.</p> </li> </ul> <p><strong>Errata (v2, 2024-01-09)</strong></p> <p><code>licenses-annotated-sample.tar.gz</code>: some comments not intended for publication were removed, and 4 files incorrectly labeled were fixed.</p> <p> </p>
Preprocessed data of Software Heritage Graph Dataset
<p>Preprocessed data of Software Heritage Graph Dataset</p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.