Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
468
datasets available to search
ShareScore release 0.9.0
Dataset results
468 results for “dump”
DOIBoost Dataset Dump
<p>Research in information science and scholarly communication strongly relies on the availability of openly accessible datasets of metadata and, where possible, their relative payloads. To this end, CrossRef plays a pivotal role by providing free access to its entire metadata collection, and allowing other initiatives to link and enrich its information. Therefore, a number of key pieces of information result scattered across diverse datasets and resources freely available online. As a result of this fragmentation, researchers in this domain end up struggling with daily integration problems producing a plethora of ad-hoc datasets, therefore incurring in a waste of time, resources, and infringing open science best practices. </p> <p>The latest DOIBoost release is a metadata collection that enriches CrossRef (October 2019 release: 108,048,986 publication records) with inputs from Microsoft Academic Graph (October 2019 release: 76,171,072 publication records), ORCID (October 2019 release: 12,642,131 publication records), and Unpaywall (August 2019 release: 26,589,869 publication records) for the purpose of supporting high-quality and robust research experiments. As a result of DOIBoost, CrossRef records have been "boosted" as follows:</p> <ul> <li>47,254,618 CrossRef records have been enriched with an abstract from MAG;</li> <li>33,279,428 CrossRef records have been enriched with an affiliation from MAG and/or ORCID;</li> <li>509,588 CrossRef records have been enriched with an ORCID identifier from ORCID.</li> </ul> <p>This entry consists of two files: <strong>doiboost_dump-2019-11-27.tar </strong>(contains a set of <strong>partXYZ.gz</strong> files, each one containing the JSON files relative to the enriched CrossRef records), a <strong>schemaAndSample.zip</strong>, and <strong>termsOfUse.doc </strong>(contains details on the terms of use of DOIBoost).</p> <p>Note that this records comes with two relationships to other results of this experiment: </p> <ol> <li>link to the data paper: for more information on how the dataset is (and can be) generated;</li> <li>link to the software: to repeat the experiment</li> </ol>
Cleaned LargeRDFBench dumps
<p>Dumps for each of the <a href="https://github.com/dice-group/LargeRDFBench">LargeRDFBench</a> datasets in two formats:</p> <ol> <li>A single N-Triples (no prefixes, no unquoted numbers/booleans) file compressed with <a href="https://github.com/facebook/zstd">zstd</a></li> <li>A <a href="https://www.rdfhdt.org/">HDT</a> file with sidecar index file (.hdt.index.v1-1) for faster querying.</li> <li>.mark files, which are JSON files storing the SHA-256 hashes of the above files and the input dump file from the original LargeRDFBench.</li> </ol> <p>In addition to individual datasets, there is <strong>LargeRDFBench-all.hdt</strong>, containing the union of all triples in all datasets.</p> <p>The files in this dataset were generated using the "fix" subcommand of the <a href="https://github.com/alexishuf/freqel-benchmark">freqel-driver</a> command-line utility. The files in this Zenodo dataset were generated from commit 47cea26 of said repository. Nearly all of the cleanup code, however, is from <a href="https://github.com/lapesd/rdfit">rdfit</a> version 1.0.6, which is available from maven central.</p> <p>There are four reasons to use this dataset as a substitute for the <a href="https://github.com/dice-group/LargeRDFBench">original</a>:</p> <ul> <li>Flatter file structure: there is a single file per dataset</li> <li>All data is in N-Triples (no RDF/XML or Turtle syntax in <code>.nt </code>named files)</li> <li>Valid IRIs and valid N-Triples syntax (no parsers errors, at most warnings)</li> <li>Provided .hdt files are directly queryable</li> </ul> <p>Since the original dumps have syntax errors and invalid IRIs, there are multiple ways to handle such issues and this dataset is one set of choices for handling them. For example, the Virtuoso endpoints of the original (as of commit <a href="https://github.com/dice-group/LargeRDFBench/commit/49d1401d70673a8054ddb2cba185e387e7a0e81c">49d1401</a>) ingest and expose invalid IRIs and langtags without complaining. Thus, there are SPARQL queries for which the results obtained using this cleaned version and the original Virtuoso endpoint bundles will differ. As far as we know, such possibility does not apply to the LargeRDFBench SPARQL queries. No triples were discarded in the cleaning process, rather triples with invalid IRIs (as per <a href="https://tools.ietf.org/html/rfc3987">RFC 3987</a>) and invalid language tags are mapped to valid counterparts. Literals are mostly unaffected, except for one particular syntax violation in the Affymetrix dataset: non-escaped null characters (U+0000) in lexical forms were replaced with spaces (U+0020) to make HDT files possible. The syntax fixes were made using RIt.tolerant() functionality of the <a href="https://github.com/lapesd/rdfit">rdfit</a> library, version 1.0.6. The list of transformations (beyond flattening the file structure and storing as N-Ttiples and HDT) was:</p> <ul> <li>Percent-encode characters not allowed at their current position in the IRI by RFC 3987. <ul> <li>If percent-encoding is not allowed at that position by RFC 3987 (e.g., port rule), the character will be erased</li> </ul> </li> <li>Erase invalid character encodings (when the binary representation is so messed up it does not appear as the wrong character but is straight up invalid UTF-8)</li> <li>Replace '_' in language tags with '-' (e.g., en_US becomes en-US)</li> <li>For NT/Turtle, \-escape occurrences of \r (0x0D) and \n (0x0A) inside single-quoted lexical forms.</li> <li>For NT/Turtle, replace \ with \\ in any \x-escape where x is not in tbnrf"' (see <a href="https://www.w3.org/TR/turtle/#grammar-production-ECHAR">ECHAR</a>).</li> <li>For NT/Turtle, identify <a href="https://www.w3.org/TR/turtle/#grammar-production-UCHAR">UCHAR</a>) escape sequences that represent an UTF-8 encoding instead of an unicode code point. Such sequences are composed of only byte-sized code points, which value sequence correspond to a valid UTF-8 sequence and where at least one such byte has a value that is the code point of a control character. Given such conditions, the sequence of UCHARs is replaced by a single UCHAR for the character encoded in UTF-8. Example: \x00C3\x0085, which corresponds to Å in UTF-8 becomes \u00C5 since U+0085 is a control character.</li> <li>For NT/Turtle, @PREFIX and @BASE are rewritten to @prefix and @base</li> <li>For NT/Turtle, literals true and false with any variation in case (e.g., True) are replaced</li> <li>with the standard true and false.</li> <li>For NT/Turtle, a lexical form followed by an <IRI> without space or with a number of ^</li> <li>characters different from 2 is replaced with ^^<IRI></li> <li>For NT/Turtle, replace invalid unquoted plain literals with plain string literals. For this, the code assumes the invalid unquoted literal has no spaces (i.e., whitespace is a separator and never part of the invalid literal). Examples of this fix in action: <ul> <li><strong><an-iri> t:chromosome X</strong> becomes <strong><an-iri> t:chromosome "X"</strong>, <strong><s> <p> 2e-3.4</strong> becomes <strong><s> <p> "2e-3.4"</strong> (exponent must be an integer) and <strong><s> <p> falseful</strong> becomes <strong><s> <p> "falseful"</strong></li> <li>Strip leading whitespace, %20, %09, %0A %0D and strip underlines at any position from IRI schemes. Affymetrix and Jamendo are affected</li> <li>For Turtle/NT/TriG, replace NULL characters (U+0000) in string literals with (U+0020). Use case: only Affymetrix</li> </ul> </li> </ul> <p><strong>Changelog</strong></p> <ul> <li><strong>1.0.1</strong>: Re-generated <strong>LMDB.index.v1-1</strong> to fix wrong results on queries with unbound subject, <strong>owl:sameAs</strong> predicate and bound object.</li> <li>1.0.2: Added <strong>LargeRDFBench-all.hdt</strong> and sidecar index file</li> </ul>
EOSC Providers and Resources data-dump
<p><span><span>The EOSC Providers and Resources dataset contains the metadata descriptions (EOSC Profiles) of the EOSC Providers and the Resources (e.g</span>. catalogues, services, data sources, training material and interoperability guidelines) they onboarded to the EOSC Catalogue and Marketplace during the <a href="https://eoscfuture.eu">EOSC Future project</a>.</span></p> <p><span>The dataset is based on a data extraction, provided by the ATHENA Research Center, using the public API of the EOSC Service Registry, which is part of the EOSC Resource Catalogue. The information provided here is a snapshot: a historical record of (a subset of) the information about various providers and resources recorded within the EOSC Catalogue and Marketplace at the end of the EOSC Future project in April 2024.</span></p> <p><span>A curation process, using a <a href="https://gitlab.desy.de/paul.millar/eosc-datadump">publicly accessible data curation workflow</a>, designed and implemented at DESY, was used to remove all known personal or sensitive data in line with GDPR and the <a href="https://eoscfuture.eu/privacy-policy/">EOSC Future Privacy Policy</a>. This process is described in the <code>methodology.md</code> file within this dataset.</span></p> <p><span>From April 2024, the EOSC Portal was phased out. On 24th April 2024, the European Commission <a href="https://open-science-cloud.ec.europa.eu/news/commission-announces-eosc-eu-nodes-web-presence">announced</a> the next phase of EOSC with the launch of the initial web presence of the <a href="https://open-science-cloud.ec.europa.eu">EOSC EU Node</a>. The goal of publishing this dataset is to enable other EOSC-related projects such as <a href="https://oscars-project.eu">OSCARS</a>, <a href="https://www.eosc-beyond.eu">EOSC Beyond</a> as well as researchers more broadly, to both reuse and build further on this work.<br></span></p>
OpenEBench Software Observatory data dump
<p>Data dump of the OpenEBench Software Observatory, consisting in metadata of software in the Life Sciences domain extracted from several sources. All collections contain the same information at different points in the processing pipeline:</p> <p>- <em>alambique</em> collection contains metadata as extracted from sources.</p> <p>- <em>pretools</em> is the harmonized version of such metadata.</p> <p>- <em>tools </em>is the final integrated collection of software metadata used to identify trends and perform FAIRness evaluations. </p> <p><strong>Repository</strong> containing the code used to generate the datasets: <a href="https://gitlab.bsc.es/inb/elixir/software-observatory/FAIRsoft_ETL">https://gitlab.bsc.es/inb/elixir/software-observatory/FAIRsoft_ETL </a></p>
Generated Wikidata Subset for Taxons based on dump: 20201102-all
<p>Source file: GeneTaxon_wikidata-20201102-all.ttl.gz</p> <p>ShEx: https://github.com/kg-subsetting/paper-wikidata-subsetting-2023/blob/master/flexibility-experiments/genes%2Btaxons/GeneTaxon.shex</p> <p>More information: https://www.semantic-web-journal.net/content/wikidata-subsetting-approaches-tools-and-evaluation</p>
Generated Wikidata Subset for Taxons based on dump: GeneTaxon_wikidata-20190121-all
<p>Source file: GeneTaxon_wikidata-20190121-all.ttl.gz</p> <p>ShEx: https://github.com/kg-subsetting/paper-wikidata-subsetting-2023/blob/master/flexibility-experiments/genes%2Btaxons/GeneTaxon.shex</p> <p>More information: https://www.semantic-web-journal.net/content/wikidata-subsetting-approaches-tools-and-evaluation</p>
Generated Wikidata Subset for Taxons based on dump: GeneTaxon_wikidata-20180115-all
<p>Source file: GeneTaxon_wikidata-20180115-all.ttl.gz</p> <p>ShEx: https://github.com/kg-subsetting/paper-wikidata-subsetting-2023/blob/master/flexibility-experiments/genes%2Btaxons/GeneTaxon.shex</p> <p>More information: https://www.semantic-web-journal.net/content/wikidata-subsetting-approaches-tools-and-evaluation</p>
Generated Wikidata Subset for Taxons based on dump: 20170821-all
<p>Source file: GeneTaxon_wikidata-20170821-all.ttl.gz</p> <p>ShEx: https://github.com/kg-subsetting/paper-wikidata-subsetting-2023/blob/master/flexibility-experiments/genes%2Btaxons/GeneTaxon.shex</p> <p>More information: https://www.semantic-web-journal.net/content/wikidata-subsetting-approaches-tools-and-evaluation</p>
Generated Wikidata Subset for Taxons based on dump: wikidata-20150601-all
<p>Source file: GeneTaxon_wikidata-20150601-all.ttl.gz</p> <p>ShEx: https://github.com/kg-subsetting/paper-wikidata-subsetting-2023/blob/master/flexibility-experiments/genes%2Btaxons/GeneTaxon.shex</p> <p>More information: https://www.semantic-web-journal.net/content/wikidata-subsetting-approaches-tools-and-evaluation</p>
Generated Wikidata Subset for Taxons based on dump: 20160613-all
<p>Source file: GeneTaxon_wikidata-20160613-all.ttl.gz</p> <p>ShEx: https://github.com/kg-subsetting/paper-wikidata-subsetting-2023/blob/master/flexibility-experiments/genes%2Btaxons/GeneTaxon.shex</p> <p>More information: https://www.semantic-web-journal.net/content/wikidata-subsetting-approaches-tools-and-evaluation</p>
Global Earth Mineral Inventory Data Dump
<p>Contains data dumps of various versions of the Global Earth Mineral Inventory (GEMI) - One of the Deep carbon observatory data legacies. </p>
Generated Wikidata Subset for Taxons based on dump: 20220630-all
<p>Source file: GeneTaxon_wikidata-20220630-all.ttl.gz</p> <p>ShEx: https://github.com/kg-subsetting/paper-wikidata-subsetting-2023/blob/master/flexibility-experiments/genes%2Btaxons/GeneTaxon.shex</p> <p>More information: https://www.semantic-web-journal.net/content/wikidata-subsetting-approaches-tools-and-evaluation</p>
Generated Wikidata Subset for Taxons based on dump: 20210531-all
<p>Source file: GeneTaxon_wikidata-20210531-all.ttl.gz</p> <p>ShEx: https://github.com/kg-subsetting/paper-wikidata-subsetting-2023/blob/master/flexibility-experiments/genes%2Btaxons/GeneTaxon.shex</p> <p>More information: https://www.semantic-web-journal.net/content/wikidata-subsetting-approaches-tools-and-evaluation</p>
OpenAIRE ScholeXplorer Service: Scholix JSON Dump
<p>This dataset contains the GZ-compressed dump of the Scholix links (<a href="https://doi.org/10.5281/zenodo.6351557">schema Version 4</a>) exposed by the OpenAIRE ScholeXplorer service. It consists of 417+Mi bi-directional links (i.e. 975+Mi directed links) between literature-dataset and dataset-dataset involving 24+ Mi literature objects and 37+ Mi datasets (showing an increase of around 160Mi links wrt the previous release). Links are collected from publishers (CrossRef, EventData), data centers (DataCite and data centers), institutional/thematic repositories (OpenAIRE), life-science databases (EMBL-EBI), and inferred by OpenAIRE via text-mining around 14Mi publication's PDFs. The dataset is structured in 30 compressed files, each of at most ~10 Gb, for a total of ~328GB.</p> <p><strong>Note that the dataset matches a new version of the schema (</strong><a href="https://doi.org/10.5281/zenodo.6351557">schema Version 4</a>). Changes are minor, backward compatible, and regard optional fields and extensions of vocabularies. The <strong>readme.doc</strong> file includes a description of the schema changes and statistics about the dataset.</p>
MongoDB database dump for the analysis of the current sustainability state of research software
<p>This data set is the MongoDB dump (bson files) of the data created and analyzed with the rsps framework. In the first step, a research subject is assigned to the research software repositories. Afterwards, the current sustainability state is evaluated. The data set comprises the following six bson files:</p> <p><strong>repositories: </strong>metadata, received from the GitHub REST API, for repositories containing the search terms "doi+10" or "doi+10+in:readme", additional information are the request date, the contained search term, and the repository hosting service, in this case for all repositories "github". For repositories the Readme files are available.</p> <p><strong>publications:</strong> metadata of publications, published on arXiv and ACM, that contain the search term "github.com".</p> <p><strong>rs_repositories:</strong> research software candidates containing a DOI or that are referenced by the publications contained in the publications data set.</p> <p><strong>rs_artifacts: </strong>research software artifacts that are referenced in the harvested GitHub repositories by a DOI and the harvested publications.</p> <p><strong>publication_subjects:</strong> All Science Journal Classification (ASJC) of Scopus combined with the Scopus source list and Scopus book title list (https://www.scopus.com/home.uri)</p> <p><strong>arxiv_subjects:</strong> arXiv taxonomy complemented with the ASJC research subject.</p> <p> </p>
Synergy database dump
<p>An SQL dump of the Synergy database. Synergy was first published in 2014, but the associated application has now reached the end of its life. This database contains the data that was presented in the publication, and that all tools in the web application were using.</p>
A word2vec model file built from the French Wikipedia XML Dump using gensim.
<p>A word2vec model file built from the French Wikipedia XML dump using gensim. The data published here includes three model files (you need all three of them in the same folder) as well as the Python script used to build the model (for documentation). The Wikipedia dump was downloaded on October 7, 2016 from https://dumps.wikimedia.org/. Before building the model, plain text was extracted from the dump. The size of that dataset is about 500 million words or 3.6 GB of plain text. The principal parameters for building the model were the following: no lemmatization was performed, tokenization was done using the "\W" regular expression (any non-word character splits tokens), and the model was built with 500 dimensions.</p>
PostgreSQL dump with DCSim simulations and raw simulation data
<p>The archives contain postgreSQL dump with DCSim simulations and raw simulation data<br><br>Archive</p> <ul> <li>simulations_dump.zip</li> </ul> <p>contains the PostgreSQL 15 dump, with schema and data, and a restore command.</p> <p> </p> <p>Archives:</p> <ul> <li>first_phase_archive.zip</li> <li>second_phase_archive.zip</li> <li>third_phase_archive.zip</li> <li>third_phase_archive_extrapolation_x5.zip</li> <li>third_phase_archive_extrapolation_x10.zip</li> <li>third_phase_archive_extrapolation_x20.zip</li> <li>fourth_phase_archive.zip </li> <li>fourth_phase_archive_extrapolation.zip</li> </ul> <p>contain the DCSim configurations, output, slurm job description and slurm output, and a CSV file with simulation results for each simulation.</p>
Machine Learning Assisted SSH Keys Extraction From The Heap Dump
<p>This dataset contains heap dump of OpenSSH that contains session keys.</p> <p>On the performance test data, we also include the PCAP file that contains the encrypted SSH network traffic. With the correct session keys, it can be decrypted.</p>
Papyrus dataset postgres dump
<p>The dataset that the database dump was created from is described here: <a href="https://doi.org/10.33774/chemrxiv-2021-1rxhk">10.33774/chemrxiv-2021-1rxhk</a><br> <br> A dump of the postgres database created from the code in the 'postgres' directory (<a href="https://github.com/reskyner/Papyrus-scripts">Papyrus-scripts</a>/<a href="https://github.com/reskyner/Papyrus-scripts/tree/master/src">src</a>/<a href="https://github.com/reskyner/Papyrus-scripts/tree/master/src/papyrus_scripts">papyrus_scripts</a>/<strong>postgres</strong>/) of Rachael Skyner's fork (https://github.com/reskyner/Papyrus-scripts) of Oliver Bequignon's Papyrus-scripts github (https://github.com/OlivierBeq/Papyrus-scripts). The database was created by:</p> <p>1. Download the papyrus csv files from Oliver's code using the download functionality</p> <p>2. Spin up a 'papyrus' container using the docker-compose.yml file in Rachael's fork (running on a machine with access to the postgres instance you want to add the database to)</p> <p>3. Start a shell in the papyrus container with docker exec -it papyrus /bin/bash</p> <p>4. Start a jupyter notebook server with jupyter notebook --ip 0.0.0.0 --allow-root --no-browser</p> <p>5. Run the two notebooks (<a href="https://github.com/reskyner/Papyrus-scripts/blob/master/src/papyrus_scripts/postgres/notebooks/1-insert_molecule_data.ipynb">1-insert_molecule_data.ipynb</a> and <a href="https://github.com/reskyner/Papyrus-scripts/blob/master/src/papyrus_scripts/postgres/notebooks/2-insert_activities.ipynb">2-insert_activities.ipynb</a>) in order</p> <p>6. Create a dump of the database</p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.