Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
9
datasets available to search
ShareScore release 0.9.0
Dataset results
9 results for “Text Corpora”
The Curated Courier: Digital Text Corpora from the UNESCO Courier (1948–2020)
<p>Founded in 1948 as the official magazine of the United Nations Educational, Scientific and Cultural Organization, <i>The UNESCO Courier</i> represents an extraordinary resource for research on global themes in the humanities. The complete <a href="https://en.unesco.org/courier/archives">archive of the magazine</a> is available in PDF form through UNESCO. These files make it possible for users anywhere to read individual issues, but it does not allow for full-text searching, much less any of the computational text analysis methods that have recently made important advances in humanities research.</p><p>The Curated Courier 1.0 is a package of digital text corpora, text analysis tools, and supplementary materials that makes the complete archive of <i>The UNESCO Courier</i> from 1948 to 2020 machine-readable, accessible, and reusable for digital text analysis. </p><p>Here on Zenodo we publish two <i>Courier</i> corpora. The first corpus (curated_courier_article_corpus) consists of the texts of all articles published in the English-language edition of <i>The UNESCO Courier</i> between 1948 and 2020. For this corpus we have extracted and reconstructed the complete text of all articles, for example by pulling together non-contiguous pages where necessary and by removing non-article text (masthead, photo captions, letters to the editor, and so on). We have linked each article to a comprehensive curated metadata index, included in the download (document_index.csv).</p><p>The second corpus (curated_issues) compiles the complete text of all <i>Courier</i> issues (English-language edition), 1948-2020. To prepare this corpus we extracted text from <a href="https://en.unesco.org/courier/archives">the PDFs that UNESCO has made available</a>, used multiple modes of OCR, and rendered each issue as a simple text file. Our test of the OCR quality finds an average error rate of 0.7 %, which should be considered good quality.</p><p>Working data from the process can be found in our <a href="https://github.com/inidun/tagged_courier">GitHub repository "tagged Courier."</a> The products, text analysis tools, and additional documentation are in the <a href="https://github.com/inidun/curated_courier">repository "Curated Courier."</a></p><p>The text of <i>The UNESCO Courier</i> is <a href="https://courier.unesco.org/en/about">available in Open Access</a> under the Attribution-ShareAlike 3.0 IGO (CC-BY-SA 3.0 IGO) license, in the context of <a href="https://en.unesco.org/open-access/">UNESCO's open access publications policy</a>. This dataset is published under the most recent version of the same license: Attribution-ShareAlike 4.0 International (<a href="https://creativecommons.org/licenses/by-sa/4.0/deed.en">CC BY-SA 4.0 Deed</a>).</p><p>These datasets was developed as part of the research project "International Ideas at UNESCO: Digital Approaches to Global Conceptual History" (INIDUN), led by Benjamin G. Martin at Uppsala University and funded by a grant from the Swedish Research Council (Vetenskapsrådet), 2020-2024. For more information, see: <a href="https://inidun.github.io">https://inidun.github.io</a>, as well as the<a href="https://github.com/inidun"> project repository on GitHub</a>, which includes documentation and files related to the curating process.</p>
Accelerating Digital Skills for Music Researchers - Processing Text-Based Corpora for Musical Discourse Analysis - Episode 5
<p>Dataset containing four .xlsx and .csv files for the exercises in Episode 5 of the <a href="https://acceleratingdigitalskills.github.io/Processing-Text-Based-Corpora/">Processing Text-Based Corpora for Musical Discourse Analysis</a> lesson of the <a href="https://acceleratingdigitalskills.org/">Accelerating Digital Skills for Music Researchers</a> project. The original data was collected from <a href="https://boomkat.com/">Boomkat.com</a> with permission.</p>
Accelerating Digital Skills for Music Researchers - Processing Text-Based Corpora for Musical Discourse Analysis - Episode 2
<p>Dataset containing three subgenre-specific .xlsx files for the exercises in Episode 2 of the <a href="https://acceleratingdigitalskills.github.io/Processing-Text-Based-Corpora/">Processing Text-Based Corpora for Musical Discourse Analysis</a> lesson of the <a href="https://acceleratingdigitalskills.org/">Accelerating Digital Skills for Music Researchers</a> project. The original data was collected from <a href="https://boomkat.com/">Boomkat.com</a> with permission.</p>
AQL queries and benchmark results from PhD thesis "ANNIS: A graph-based query system for deeply annotated text corpora"
<p>These are the queries, the benchmark results and the evaluation scripts of the thesis "ANNIS: A graph-based query system for deeply annotated text corpora" (Thomas Krause 2018, Humboldt-Universität zu Berlin)</p> <p><strong>diss_2018-01-12_v0.5.0.csv </strong><br> Results of all configurations of executed benchmarks for graphANNIS and also the baseline times of relANNIS.</p> <p><strong>queries.zip</strong><br> Contains folders for each corpus containing all queries used for the benchmark. Each file-name begins with the ID of the query. The extension denotes the type, which can be one of the following:</p> <ul> <li><em>".</em>aql" contains the original AQL (ANNIS query language) query which was collected</li> <li>".json" is JSON representation of the parsed AQL query</li> <li>".count" is the number of matches a query should have</li> <li>".time" is the average time in milliseconds that was needed to execute the query in relANNIS on the benchmark system</li> <li>".corpora" contains the name of the corpus the query belongs to (should be only one corpus and the same as the folder name in the selection of queries in this data set)</li> <li>".relplan" contains the PostgreSQL plan for the query</li> <li>".graphplan" contains the graphANNIS plan for the query</li> </ul> <p><strong>evaluation-scripts.py/evaluation-scripts.ipynb</strong><br> Python scripts to perform the evaluation and generate the images. This are both a Python-file and the original notebook file that can be used with the Jupyter Notebook application.</p> <p><strong>relannis_benchmark_scripts.zip </strong><br> The files in this zip-file can be used to execute the benchmarks in the relANNIS system by piping the into the "annis.sh" command line tool of relANNIS</p>
Hierarchical Text Classification corpora
<p>A set of 3 datasets for Hierarchical Text Classification (HTC), with samples divided into training and testing splits. The hierarchies of labels within all datasets have depth 2.</p> <ul> <li>The <strong>Amazon5x5</strong> dataset contains 500,000 user reviews tagged with the reviewed product's categories. There are 5 product categories with 100,000 examples each, and each category has 5 sub-categories.</li> <li>The <strong>Bugs</strong> dataset contains 30,050 bugs of the Linux kernel, labeled with exactly two categories identifying the affected component.</li> <li>Finally, the <strong>Web Of Science</strong> dataset contains 46,960 abstracts of scientific papers, labeled the article's domain (see <a href="https://data.mendeley.com/datasets/9rw3vkcfy4/6">original repo</a> for more details).</li> </ul> <p>Datasets are published in JSONL format, where each line is a string formatted as a JSON, like in the example below.</p> <pre><code>{ "text": <article text>, "labels": [<label1>, <label2>, ...] }</code></pre> <p>The <em>hierarchical structure</em> of labels in each dataset is documented in <a href="https://gitlab.com/distration/dsi-nlp-publib/-/tree/main/htc-survey-24/data/taxonomies">this repository</a>.</p> <p> </p> <p>These datasets have been presented in this paper:</p> <ul> <li>"Hierarchical Text Classification and its Foundations: a Review of Current Research" - DOI: <a href="https://doi.org/10.3390/electronics13071199">10.3390/electronics13071199</a></li> </ul> <p>Some of these datasets have also been used in:</p> <ul> <li>"Ticket Automation: an Insight into Current Research with Applications to Multi-level Classification Scenarios" - DOI: <a href="https://doi.org/10.1016/j.eswa.2023.119984">10.1016/j.eswa.2023.119984</a></li> <li>"A multi-level approach for hierarchical Ticket Classification", accepted at WNUT 2022 - <a href="https://aclanthology.org/2022.wnut-1.22/">link</a></li> </ul> <p> </p> <p>These datasets are partially derived from previous work, namely:</p> <ul> <li>[Amazon] J. Ni, J. Li, J. McAuley, "Justifying Recommendations using Distantly-Labeled Reviews and Fine-Grained Aspects", EMNLP 2019, doi: <a href="http://dx.doi.org/10.18653/v1/D19-1018">10.18653/v1/D19-1018</a></li> <li>[WOS] K. Kowsari, D. E. Brown, M. Heidarysafa, K. Jafari Meimandi, M. S. Gerber and L. E. Barnes, "HDLTex: Hierarchical Deep Learning for Text Classification," 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA), 2017, pp. 364-371, doi: <a href="http://doi.org/10.1109/ICMLA.2017.0-134">10.1109/ICMLA.2017.0-134</a></li> <li>[Linux Bugs] V. Lyubinets, T. Boiko and D. Nicholas, "Automated Labeling of Bugs and Tickets Using Attention-Based Mechanisms in Recurrent Neural Networks," <em>2018 IEEE Second International Conference on Data Stream Mining & Processing (DSMP)</em>, 2018, pp. 271-275, doi: <a href="http://doi.org/10.1109/DSMP.2018.8478511">10.1109/DSMP.2018.8478511</a></li> </ul>
Appendix B: Language Corpora Available for Text Mining
<p>Appendix B is associated with <em><strong>Chapter 3: Text Pre-Processing</strong></em> of the book -- Manika Lamba and Margam Madhusudhan (2021) Text Mining for Information Professionals: An Uncharted Territory, SpringerNature.</p>
Sparse Machine Learning Methods for Understanding Large Text Corpora
Sparse machine learning has recently emerged as powerful tool to obtain models of high-dimensional data with high degree of interpretability, at low computational cost. This paper posits that these methods can be extremely useful for understanding large collections of text documents, without requiring user expertise in machine learning. Our approach relies on three main ingredients: (a) multi-document text summarization and (b) comparative summarization of two corpora, both using parse regression or classifi cation; (c) sparse principal components and sparse graphical models for unsupervised analysis and visualization of large text corpora. We validate our approach using a corpus of Aviation Safety Reporting System (ASRS) reports and demonstrate that the methods can reveal causal and contributing factors in runway incursions. Furthermore, we show that the methods automatically discover four main tasks that pilots perform during flight, which can aid in further understanding the causal and contributing factors to runway incursions and other drivers for aviation safety incidents. Citation: L. El Ghaoui, G. C. Li, V. Duong, V. Pham, A. N. Srivastava, and K. Bhaduri, “Sparse Machine Learning Methods for Understanding Large Text Corpora,” Proceedings of the Conference on Intelligent Data Understanding, 2011.
Restricted corpora used in the benchmarks of the PhD thesis "ANNIS: A graph-based query system for deeply annotated text corpora"
<p>Corpora used in the evaluation of the PhD thesis "ANNIS: A graph-based query system for deeply annotated text corpora" (Thomas Krause 2018, Humboldt-Universität zu Berlin) which access is limited to academic research purposes. This includes the following corpora in the relANNIS format:</p> <ul> <li> <p>Kobalt-DaF-Korpus (not published under an open license yet, https://www.linguistik.hu-berlin.de/de/institut/professuren/korpuslinguistik/forschung/kobalt-daf)</p> </li> <li> <p>TIGER (http://www.ims.uni-stuttgart.de/forschung/ressourcen/korpora/tiger.html)</p> </li> <li> <p>TüBa-D/Z (http://www.sfs.uni-tuebingen.de/ascl/ressourcen/corpora/tueba-dz.html)</p> </li> </ul> <p>The original TIGER corpus can be downloaded from http://www.ims.uni-stuttgart.de/forschung/ressourcen/korpora/TIGERCorpus/license/htmlicense.html and the TüBa-D/Z is available after manually requesting a download-link at http://www.sfs.uni-tuebingen.de/ascl/ressourcen/corpora/tueba-dz.html</p> <p>Access to both the TIGER and TüBa-D/Z corpora is restricted by the original authors and you need to accept licenses that you are solely using the corpus for academic research. Since the original providers don't include versions in the relANNIS format, it has been made available as restricted download here.</p> <p> </p> <p> </p>
Corpora used in the benchmarks of the PhD thesis "ANNIS: A graph-based query system for deeply annotated text corpora"
<p>Corpora used in the evaluation of the PhD thesis "ANNIS: A graph-based query system for deeply annotated text corpora" (Thomas Krause 2018, Humboldt-Universität zu Berlin) which have some form of license that allows their redistribution. See LICENSE.txt in each folder for the actual license of each corpus and the original authors.</p> <p>Since each corpus has a different license, the access is restricted.</p> <p><strong>If using one of the corpora please cite the original authors mentioned in the LICENSE.txt!</strong></p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.