Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
75
datasets available to search
ShareScore release 0.9.0
Dataset results
75 results for “Document Datasets”
DocMine: A Software Documentation-Related Dataset of 950 GitHub Repositories
<p>DocMine dataset consists of textual information collated from multiple software artifacts, across 950 GitHub Repositories. It also consists of probable percentage contribution of text in each software artifact towards different documentation types in each repository, accompanied by metadata information about the repository such as stargazer count, number of pull requests, commits, issues and other files analyzed.</p>
ICDAR 2021 Historical Document Classification Dataset for Task 2 - Dating
<p>Dataset for the localization classification task of the ICDAR 2021 Competition on Historical Document Classification competition. The tar.gz file contains the images and ground truth file (CSV format). The metadata csv file contains information, such as where the image comes from.</p>
Long document similarity dataset, Wikipedia excerptions for movies collections
<p>Movies-related articles extracted from Wikipedia.</p> <p>For all articles, the figures and tables have been filtered out, as well as the categories and "see also" sections.</p> <p>The article structure, and particularly the sub-titles and paragraphs are kept in these datasets</p> <p> </p> <p><strong>Movies</strong></p> <p>The Wikipedia Movies dataset consists of 100,371 articles describing various movies. Each article may consist of text passages describing the plot, cast, production, reception, soundtrack, and more.</p>
Open-source Software Governance Documentation Dataset on GitHub
<p>This dataset contains 710 GitHub-hosted OSS projects, which contain a governance file in the root directory of the project. It also contains commits, issues, and comments on each project.</p>
Line Segment in Document Images Datasets
<p>Those datasets are related to the accepted article to ICDAR 2023: "Linear Object Detection in Document Images<br> using Multiple Object Tracking" by Bernet et al.</p> <p>The official github repository is : https://github.com/EPITAResearchLab/bernet.23.icdar</p> <p>.png ground truth are labelled images where white pixels correspond to the background.</p> <p>.csv ground truth are using the following format:<br> ```<br> x,y,w,h # First line is the size of the interest area in the image<br> x1,y1,x2,y2 # Following lines are the coordinates of the extremities of the line<br> ...<br> xn,yn,xn+1,yn+1<br> ```</p>
Dataset for Paper: Text Line Detection and Recognition of Greek Polytonic Documents
<p>Dataset for Paper: Text Line Detection and Recognition of Greek Polytonic Documents, P. Kaddas, B. Gatos, K. Palaiologos, K. Christopoulou and K. Kritsis, 4th Workshop on Machine Learning (WML), San Jose, California, USA</p> <p>We introduce a new dataset, named GTLD-small dataset, with annotated text line quadrilateral polygons of 1.642 documents, including annotations on 3 datasets (Tobacco-3482, PIOP and ShakeIT dataset)</p> <table> <caption>Overview of the datasets included in this work and the number of images used for training, validation and testing.</caption> <thead> <tr> <th scope="col">Collection</th> <th scope="col">#Total</th> <th scope="col">#train</th> <th scope="col">#val</th> <th scope="col">#test</th> </tr> </thead> <tbody> <tr> <td>PIOP-small</td> <td>950</td> <td>672</td> <td>90</td> <td>188</td> </tr> <tr> <td>ShakeIT-small</td> <td>357</td> <td>264</td> <td>27</td> <td>66</td> </tr> <tr> <td>Tobacco-3482-small</td> <td>335</td> <td>240</td> <td>30</td> <td>65</td> </tr> </tbody> </table> <p> </p>
Dataset of fake identity documents for research purposes
<p>Dataset of fake Spanish ID documents to train fake ID detectors on. Base material taken from MIDV 2020 dataset.</p>
Pairwise Multi-Class Document Classification for Semantic Relations between Wikipedia Articles (Dataset, Models & Code)
<p>Many digital libraries recommend literature to their users considering the similarity between a query document and their repository. However, they often fail to distinguish what is the relationship that makes two documents alike. In this paper, we model the problem of finding the relationship between two documents as a pairwise document classification task. To find the semantic relation between documents, we apply a series of techniques, such as GloVe, Paragraph-Vectors, BERT, and XLNet under different configurations (e.g., sequence length, vector concatenation scheme), including a Siamese architecture for the Transformer-based systems. We perform our experiments on a newly proposed dataset of 32,168 Wikipedia article pairs and Wikidata properties that define the semantic document relations. Our results show vanilla BERT as the best performing system with an F1-score of 0.93,<br> which we manually examine to better understand its applicability to other domains. Our findings suggest that classifying semantic relations between documents is a solvable task and motivates the development of recommender systems based on the evaluated techniques. The discussions in this paper serve as first steps in the exploration of documents through SPARQL-like queries such that one could find documents that are similar in one aspect but dissimilar in another.</p> <p>Additional information can be found on <a href="https://github.com/malteos/semantic-document-relations/">GitHub</a>.</p> <p>The following data is supplemental to the experiments described in our research paper. The data consists of:</p> <ul> <li>Datasets (articles, class labels, cross-validation splits)</li> <li>Pretrained models (Transformers, GloVe, Doc2vec)</li> <li>Model output (prediction) for the best performing models</li> </ul> <p><strong>Dataset</strong></p> <p>The Wikipedia article corpus is available in <code>enwiki-20191101-pages-articles.weighted.10k.jsonl.bz2</code>. The original data have been downloaded as <a href="https://dumps.wikimedia.org/enwiki/">XML dump</a>, and the corresponding articles were extracted as plain-text with <a href="https://radimrehurek.com/gensim/scripts/segment_wiki.html">gensim.scripts.segment_wiki</a>. The archive contains only articles that are available in training or test data.</p> <p>The actual dataset is provided as used in the stratified k-fold with <code>k=4</code> in <code>train_testdata__4folds.tar.gz</code>.</p> <pre><code>├── 1 │ ├── test.csv │ └── train.csv ├── 2 │ ├── test.csv │ └── train.csv ├── 3 │ ├── test.csv │ └── train.csv └── 4 ├── test.csv └── train.csv 4 directories, 8 files </code></pre> <p>Pretrained models</p> <p>PyTorch: vanilla and Siamese BERT + XLNet</p> <p>Pretrained model for each fold is available in the corresponding model archives:</p> <pre><code># Vanilla model_wiki.bert_base__joint__seq512.tar.gz model_wiki.xlnet_base__joint__seq512.tar.gz # Siamese model_wiki.bert_base__siamese__seq512__4d.tar.gz model_wiki.xlnet_base__siamese__seq512__4d.tar.gz </code></pre>
Dataset for "Understanding Performance Concerns in the API Documentation of Data Science Libraries"
<p>Dataset for the manuscript "Understanding Performance Concerns in the API Documentation of Data Science Libraries", including the results of knowledge classification, consistency analysis, and evolution analysis on the API documentation data.</p>
EPARCHOS - Historical Greek handwritten document dataset
<p>The dataset originates from a Greek handwritten codex that dates from around 1500-1530. This is the subset of the codex British Museum Addit. 6791, written by two hands, one by Antonius Eparchos and the other by Camillos Zanettus (ff. 104r-174v) and delivers texts by Hierocles (In Aureum carmen), Matthaeus Blastares (Collectio alphabetica) and, notably, texts by Michael Psellos (De omnifaria doctrina). The writing delivers the most important abbreviations, logograms and conjunctions, which are cited in virtually every Greek minuscule handwritten codex from the years of the manuscript transliteration and the prevalence of the minuscule script (9th century) to the post-Byzantine years. This dataset consists of 120 scanned handwritten text pages, containing 9285 lines of text, 18809 words (6787 unique words). For each page, a PageXML is provided containing the following groundtruth:</p> <ol> <li>Text region polygon coordinates</li> <li>Text line polygon coordinates with the corresponding transcription text</li> <li>Word polygon coordinated with the corresponding transcription text</li> </ol>
Datasets for "Discourse-Aware Unsupervised Summarization for Long Scientific Documents"
<p>Datasets used in <a href="https://aclanthology.org/2021.eacl-main.93.pdf">Discourse-Aware Unsupervised Summarization for Long Scientific Documents</a></p>
Annual Health Survey Clean Datasets and Documentation
<p>Between July of 2010 and May of 2013, the Government of India conducted the Annual Health Survey in the Empowered Action Group states of north India. The purpose of the survey was to study the impact of reforms to maternal and child health services in the nine high-fertility, high-mortality states that comprise the EAG.</p> <p>The AHS consists of a three-round panel that interviewed over 4 million households in each round, as well as a one-time Clinical, Anthropometric, and Bio-Chemical Survey. These data were released to the public in 2015 as a set of .csv files. Between December of 2019 and May of 2021, a team of researchers from The University of Texas at Austin assembled the publicly available microdata, constructed the necessary identifiers, cleaned the data, and released user-friendly versions of the constituent datasets on the University of Michigan’s Inter-university Consortium for Political and Social Research.</p> <p>The documentation, code, and data shared here describes how the final datasets were constructed from the original files released by the Government of India’s Ministry of Health and Family Welfare, what information the final datasets contain, how to use the panel structure of the data, and how to link the different datasets together. It also discusses some of the limitations of the data.</p> <p>The original files released by the Government of India can be accessed here: <a href="https://zenodo.org/record/6062984#.YiZTG0ntwdU">https://zenodo.org/record/6062984#.YiZTG0ntwdU</a></p>
Underlying dataset to the Report on the Open Consultation of the NPOS2030 Ambition Document
<p>This is the underlying dataset to the <a href="https://zenodo.org/record/6106850">Report on the Open Consultation</a> on the NPOS2030 Ambition Document.</p> <p>This dataset contains the original responses to the online Open Consultation that was open between November 22<sup>nd</sup> and December 22<sup>nd</sup> 2021, aiming to give all Netherlands stakeholders the opportunity to reflect on the NPOS2030 Ambition Document of the Netherlands National Programme Open Science (<a href="http://openscience.nl/">NPOS</a>).</p> <p>Out of a total of 78 respondents, 54 institutions, networks, initiatives and persons gave their consent to publish their response including their name. They are listed below. 11 Responses are included on a basis of anonymity: this was sometimes caused by having to ask too many people in a network to give their consent. 13 participants decided to not disclose their responses.</p> <p>NB: Some respondents did say that their responses to the questions about the level of support for each of the four sections of the Ambition Document paint a too negative picture: due to the nature of scoring on a scale of 1-5, they did not see a better way than to give a low score to bring across that they always see ways to improve.</p> <p>The institutions, networks, initiatives and persons gave their consent to publish their response:</p> <ol> <li>4TU.ResearchData</li> <li>Adviescollege Open Science van de hogescholen</li> <li>Amsterdam University of Applied Sciences</li> <li>Avans Hogeschool</li> <li>brief consultation of five senior researchers</li> <li>Centre for Science and Technology Studies (CWTS), Leiden University</li> <li>Dutch Research Council NWO</li> <li>Eindhoven University of Technology</li> <li>Erasmus University Rotterdam </li> <li>Hogeschool Leiden</li> <li>KB, National Library of the Netherlands</li> <li>Maastricht University Recognition & Rewards Programme Team</li> <li>Media voor Vak en Wetenschap (MVW)</li> <li>Medical libraries of the STZ Samenwerkende Topklinische ziekenhuizen</li> <li>Netherlands eScience Center</li> <li>Network of Dutch Open Science Communities: the OSC-NL Board and 60 individual researchers who signed this response</li> <li>NL-RSE (Netherlands Research Software Engineers network)</li> <li>NRO (Nationaal Regieorgaan Onderwijsonderzoek)</li> <li>NWO Institute Organisation</li> <li>ODISSEI (social sciences)</li> <li>Open Science programme of the University of Groningen</li> <li>Open Science Public Engagement Fellow Network, Utrecht University</li> <li>Pedagogische en Onderwijswetenschappen, Faculteit der Maatschappij en Gedragswetenschappen, UvA</li> <li>Promovendi Netwerk Nederland (PNN)</li> <li>PULSAQUA</li> <li>Radboud University</li> <li>Radboudumc Nijmegen</li> <li>Rathenau Instituut</li> <li>Regieorgaan SIA</li> <li>Samenwerkingsverband Universiteitsbibliotheken en Koninklijke Bibliotheek (UKB)</li> <li>SciComNL (Science Communication Association Netherlands)</li> <li>Springer Nature</li> <li>Stuurgroep DCC-PO</li> <li>SURF</li> <li>The Dutch Parkinson's Association</li> <li>The Open Science Community Amsterdam (OSCA). OSCA consists of members from the Amsterdam University of Applied Sciences (HvA), the University of Amsterdam (UvA), the Vrije Universiteit Amsterdam (VU) and the Student Initiative for Open Science (SIOS).</li> <li>Three open science specialists at Utrecht University Library: Bianca Kramer, Jeroen Sondervan and Jeroen Bosman</li> <li>Tilburg University Open Science Focus Group</li> <li>TU Delft</li> <li>Universiteit Leiden</li> <li>University of Amsterdam</li> <li>University of Twente</li> <li>Utrecht University Freudenthal Institute Science Communication and Public Engagement group</li> <li>Vrije Universiteit Amsterdam</li> <li>Zone "Towards digital (open) educational resources" from the Dutch Acceleration Plan</li> <li>Prof.dr. Karin Pfeffer</li> <li>Drs Marina Noordegraaf</li> <li>Mr. Niels van Tol</li> <li>Dr. Egon Willighagen</li> <li>Drs (Msc) Gaby Lutgens</li> <li>Jeroen Jansen</li> <li>Dr. Miaomiao Zhou</li> <li>Prof.dr. René Bekkers</li> <li>Dr.ir. Rolf Hut</li> </ol>
Dataset for documents referenced as part of my research on immigration in the German context
<p>A combination of legal documents, surveys and reports (published by organisations affiliated to the EU or the German state and non-governmental agencies/institutes. These are open access documents downloaded either directly from the owner's website or other online platforms.</p> <p><strong>Declaration: I am not the author or owner of any of the documents uploaded here.</strong></p> <p>They have been uploaded here as part of the Marie Curie grant stipulations to make the research data open access.</p> <p>Documents are in German and English.</p> <p>MSCA Project: RE-NUP: Spousal Reunification and Integration Laws in Europe</p> <p>Grant agreement No. 890826</p>
Original dataset of :"First pre-Miocene paleomagnetic data from the Calabrian block document a 160° post-late Jurassic CCW rotation as a consequence of left-lateral shear along Alpine Tethys"
<p>In this table the original paleomagnetic dataset related to the research article :"First pre-Miocene paleomagnetic data from the Calabrian block document a 160° post-late Jurassic CCW rotation as a consequence of left-lateral shear along Alpine Tethys" is published</p>
Datasets for "Reading Order Independent Metrics for Information Extraction in Handwritten Documents"
<p>This repository includes the five datasets used for our paper entitled <em>Reading Order Independent Metrics for Information Extraction in Handwritten Documents</em>, in which we compare various metrics to evaluate end-to-end information extraction from scanned documents.</p> <h2>Datasets</h2> <p>Five datasets are released following the BIO format:</p> <ul> <li>IAM</li> <li>Simara</li> <li>POPP</li> <li>Esposalles</li> <li>French Military Records</li> </ul> <p>For each dataset, we provide the following data (on test sets):</p> <ul> <li>Ground truth annotations (<code>gt/</code>)</li> <li>Automatic predictions (<code>dan/</code>)</li> <li>Automatic predictions with entities appearing in random order (<code>dan_shuffled/</code>)</li> </ul> <p>The data is organized as follows:</p> <p><code>├── Dataset name/</code><br><code>│ ├── gt/</code><br><code>│ ├── dan/</code><br><code>│ └── dan_shuffled/</code></p> <h2>Metrics</h2> <p>To install the <a href="https://pypi.org/project/ie-eval/"><code>ie-eval</code></a> package, run <code>pip install ie-eval</code>.</p> <p>To compute all metrics on a specific dataset, run:<br><br><code>ie-eval all --label-dir IAM_paragraph/gt/ --prediction-dir IAM_paragraph/dan/</code><br><br></p> <p>To learn more about the various options, use the <code>--help</code> argument or read the <a href="https://ie-eval-ner-metrics-050f40e80b04480e2310d39ad338de778f6bec80e18.pages.teklia.com/">documentation</a>.</p> <p> </p>
Wikipedia Dataset for "Structure Tree-LSTM: Structure-aware Attentional Document Encoders"
<p>The Wikipedia dataset collected for the paper "Structure Tree-LSTM: Structure-aware Attentional Document Encoders".</p> <p>Paper link: <a href="https://arxiv.org/abs/1902.09713">https://arxiv.org/abs/1902.09713</a></p>
MultiEURLEX - A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer
<p>The dataset is published with:<br> <br> <em>MultiEURLEX - A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer. Ilias Chalkidis, Manos Fergadiotis, and Ion Androutsopoulos. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021. Punta Cana, Dominican Republic.</em><br> <br> <strong>Documents: </strong>MultiEURLEX comprises 65k EU in 23 official EU languages. Each EU law has been annotated with EUROVOC concepts (labels) by the Publication Office of EU. Each EUROVOC label ID is associated with a Label descriptor, e.g., [60, `agri-foodstuffs'], [6006, `plant product'], [1115, `fruit']. The descriptors are also available in 23 languages. Chalkidis et al. (2019) published a monolingual (English) version of this dataset, called EURLEX57K, comprising 57k EU laws with the originally assigned gold labels.</p> <p><strong>Languages: </strong>MultiEURLEX covers 23 languages from 7 families. EU laws are published in all official EU languages, except for Irish for resource-related reasons (Read more: https://europa.eu/european-union/about-eu/eu-languages_en). This wide coverage makes the dataset a valuable testbed for cross-lingual transfer. All languages use the Latin script, except for Bulgarian (Cyrillic script) and Greek.</p> <p><strong>Multi-granular Labeling: </strong>EUROVOC<strong> </strong>has eight levels of concepts. Each document is assigned one or more concepts (labels). If a document is assigned a concept, the ancestors and descendants of that concept are typically not assigned to the same document. The documents were originally annotated with concepts from levels 3 to 8. We created three alternative sets of labels per document, by replacing each assigned concept by its ancestor from levels 1, 2, or 3, respectively. Thus, we provide four sets of gold labels per document, one for each of the first three levels of the hierarchy, plus the original sparse label assignment.</p> <p><strong>Supported Tasks: </strong>Similarly to EURLEX (Chalkidis et al., 2019), MultiEURLEX can be used for legal topic classification, a multi-label classification task where legal documents need to be assigned concepts (in our case, from EUROVOC) reflecting their topics. Unlike EURLEX57K, however, MultiEURLEX supports labels from three different granularities (EUROVOC levels). More importantly, apart from monolingual (one-to-one) experiments, it can be used to study cross-lingual transfer scenarios, including one-to-many (systems trained in one language and used in other languages with no training data), and many-to-one or many-to-many (systems jointly trained in multiple languages and used in one or more other languages).</p> <p><strong>Data Split and Concept Drift: </strong>MultiEURLEX is chronologically split in training (55k, 1958-2010), development (5k, 2010-2012), test (5k, 2012-2016) subsets, using the English documents. The test subset contains the same 5k documents in all 23 languages. The development subset also contains the same 5k documents in 23 languages, except Croatian. Croatia is the most recent EU member (2013); older laws are gradually translated. For the official languages of the seven oldest member countries, the same 55k training documents are available; for the other languages, only a subset of the 55k training documents is available. Compared to EURLEX57K (Chalkidis et al., 2019), MultiEURLEX is not only larger (8k more documents) and multilingual; it is also more challenging, as the chronological split leads to temporal real-world concept drift across the training, development, test subsets, i.e., differences in label distribution and phrasing, representing a realistic temporal generalization problem (Huang and Paul, 2019; Lazaridou et al., 2021). Recently, Søgaard et al. (2021) showed this setup is more realistic, as it does not overestimate real performance, contrary to random splits (Gorman and Bedrick, 2019).</p>
Documentation datasets for APAV
<p>Datasets used in documentation for APAV version 1.4.0. All datasets are from APT experiments of a Ag-doped GdBa2Cu3O7-x superconductor.</p>
ICDAR 2021 Historical Document Classification Test Dataset for Task 3 - Location
<p>Test set for the location classification task of the ICDAR 2021 Historical Document Classification.</p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.