Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
22
datasets available to search
ShareScore release 0.9.0
Dataset results
22 results for “language resources”
LRRo: A Lip Reading Data Set for the Under-resourced Romanian Language
<p>Two distinct collections are presented in this repository:</p> <p>(i) wild LRRo data is designed for an Internet in-the-wild, ad-hoc scenario, coming with more than 35 different speakers, 1.1k words, a vocabulary of 21 words, and more than 20 hours;</p> <p>(ii) lab LRRo data, addresses a lab controlled scenario for more accurate data, coming with 19 different speakers, 6.4k words, a vocabulary of 48 words, and more than 5 hours.</p>
Machine Assisted Translation of Wikipedia Articles into Low Resource Languages
<p><strong>Wikipedia is the largest encyclopedia ever assembled with the vision of enabling every human being to freely share in the sum of all knowledge. Wikipedia currently has a total of more than six million articles and over 17 billion words in its English edition. Unfortunately, millions of people cannot access this resource because it’s not available in their language. For instance, at the moment there are only 218 Tigrinya Wikipedia and 15,018 Amharic Wikipedia articles.</strong></p> <p><strong>In this project, we investigate the problem of translating Wikipedia articles from a high resource language into low resource languages using human-in-the-loop MT systems. In particular, we investigate different approaches to translate a sample of English Wikipedia articles into Tigrinya and Amharic. Currently, this repository contains 100k English Wikipeida abstracts translated using Lesan (https://lesan.ai) into Amharic and Tigrinya.</strong><br> <br> </p> <p><strong>Structure of data directory:</strong></p> <p><strong>data<br> ├── human<br> └── mt<br> ├── google<br> ├── lesan<br> │ ├── am.txt<br> │ ├── en.txt<br> │ └── ti.txt<br> └── microsoft</strong><br> </p>
Griots Interviews, Bambara Language WAV, 30 hours, Recorded 2022, Cultural and ASR Training Resource
<p><strong>Source material to this project:</strong></p> <ul> <li> <p>Addition to 200,000 lines Bambara-French clean synchronized corpus</p> </li> <li> <p>Co-project with Google, recorded 30 hours video interviews with Griots</p> </li> <li> <p>30 hours manually transcribed and translated to French</p> </li> <li> <p>10 hours used in training ASR system and MT transformer</p> </li> <li> <p>100% Open Sourced</p> </li> <li> <p>Cultural/Technical Exhibition to be hosted online and in the National Museum of Mali</p> </li> <li> <p>Record, preserve, and share Malian culture with the world</p> </li> <li> <p>Contribute to the science of low-resource language NLP</p> </li> <li> <p>Reinforce the development of written Bambara</p> </li> <li> <p>Enable Bambara to reach status as a “first-class internet language”</p> </li> </ul> <p>Corresponding transcribed data can be found at the following <a href="https://github.com/robotsmali-ai/jeli-asr">Github repository</a></p>
German-Swiss survey on the use of (digital) language resources
<p>This survey on the use of (digital) language resources in German-speaking Switzerland complements the Europe-wide survey conducted by the European Network of e-Lexicography (ENeL) and was carried out in close cooperation with the Leibniz Institute for the German Language (IDS) in Mannheim. The aim was to obtain information on the use of language resources from the user's perspective.</p>
Dutch language resources for spaCy
<p>Version 0.1.0 of the Dutch language resources for spaCy</p> <p>spaCy is a library for advanced natural language processing in Python and Cython. See also https://spacy.io/ and https://github.com/explosion/spaCy</p> <p>The Dutch language resources were trained with data from Universal Dependencies (UD_Dutch: https://github.com/UniversalDependencies/UD_Dutch and UD_Dutch-LassySmall: https://github.com/UniversalDependencies/UD_Dutch-LassySmall ) and CoNLL 2002 (http://www.cnts.ua.ac.be/conll2002/ner.tgz ; http://www.cnts.ua.ac.be/conll2002/ner/).</p> <p>More information about this data set can be found at https://github.com/nlesc-sherlock/spacy-dutch</p>
Human review for post-training improvement of low-resource language performance in large language models
<p>Large language models (LLMs) have significantly improved natural language processing, holding the potential to support health workers and their clients directly. Unfortunately, there is a substantial and variable drop in performance for low-resource languages. Here we present results from an exploratory case study in Malawi, aiming to enhance the performance of LLMs in Chichewa through innovative prompt engineering techniques. By focusing on practical evaluations over traditional metrics, we assess the subjective utility of LLM outputs, prioritizing end-user satisfaction. Our findings suggest that tailored prompt engineering may improve LLM utility in underserved linguistic contexts, offering a promising avenue to bridge the language inclusivity gap in digital health interventions.</p>
Comparing the Use of Research Resource Identifiers and Natural Language Processing for Citation of Databases, Software and Other Digital Artifacts
<p><strong>The Research Resource Identifier was introduced in biomedicine in 2014 to more precisely identify the reagents and tools used in published biomedical research and to track use of tools across the breadth of the biomedical literature. The current RRID specification covers key biological and digital resources. Authors are instructed to include an RRID after the first mention of any resource used. RRIDs are designed to be easy to find using a full text search search engine. </strong></p> <p><strong>The published data sets were used in our comparative study where comparing the output of our RRID curation workflow with the outputs of automated text mining systems that have been used to identify mentions of resources in the text of publications. All files in tab-separated format (tsv). </strong></p> <p><strong>Scibot.tsv: Records of the RRID curation workflow using SciBot. </strong></p> <p>Each record shows that a resource RRID was identified in paper PMID with curator tags (Tag1, Tag2, both optional)</p> <p><strong> </strong>PMID: Pubmed ID</p> <p> RRID: Research Resource Identifier</p> <p> Tag1: Curator tags (optional)</p> <p> Tag2: Additional curator tags (optional)</p> <p><strong>rdwsorted.tsv: Records of the output from RDW, a text mining software. </strong></p> <p>RDW identifies mentions of research resources in papers. Each record shows that a resource RRID was identified in paper PMID.</p> <p><strong> </strong>PMID: Pubmed ID</p> <p> RRID: Research Resource Identifier</p> <p><strong>rridbyrdw05282019.tsv: Records of the output of the RRID-by-RDW in RDW. </strong></p> <p>RRID-by-RDW is a component in RDW that identifies mentions of research resources in papers by matching patterns of RRID specifications. Each record shows that a resource RRID was identified in paper PMID.</p> <p><strong> </strong>PMID: Pubmed ID</p> <p> RRID: Research Resource Identifier</p> <p> Context: Snippet where the RRID was found</p> <p><strong>resource_metadata20190418.tsv: Metadata of RRIDs</strong></p> <p>This file contains metadata of resources and their RRIDs. See file header for column definitions.</p> <p><strong>RRIDCUR-definitions.tsv: Definitions of curator tags used in Scibot.tsv.</strong></p> <p><strong> </strong>tag: Tag name</p> <p> definition: Definition of the tag</p>
Lesan: Machine Translation for Low Resource Languages
<p>Human evaluation dataset to evaluate machine translation systems to and from Amharic, English and Tigrinya.</p>
Open Language Access (OLA): Democratising Linguistic Resources
<p>The audio file is a podcast introducing the Open Language Access (OLA) project.</p> <p>OLA is a collaborative digital archive that brings together researchers from all over the world. Launched in 2019 in the context of linguistic research, OLA is not just an archive but, above all, a platform that aims at providing tools for the creation, collaborative editing, exploration and publication of textual archives. In today's society, language stands as a central tool for change, and at the same time as a register and trace of the change itself. Hence the need to create a digital and accessible linguistic archive (open source and open access) with the aim of changing the paradigms of society's observation by providing an accessible tool to anyone who wants to look at change through language.</p>
Human review for post-training improvement of low-resource language performance in large language models
Open the record for dataset details and reuse information.
Charitable objectives or donor benefits? What sponsor language reveals about donor-advised fund priorities and resource flows
Open the record for dataset details and reuse information.
Resources for the paper "Teaching a Multilingual Large Language Model to Understand Multilingual Speech via Multi-Instructional Training"
Open the record for dataset details and reuse information.
Tools & Resources for the Armenian language
<p>Different tools and resources for Classical Armenian, Eastern Armenian and Western Armenian.</p>
Datasets for "Exploring the potential of neural machine translation for cross-language clinical NLP resource generation through annotation projection"
<p>This repository contains the data and additional resources used for the paper:</p> <p>"Exploring the Potential of Neural Machine Translation for Cross-Language Clinical NLP Resource Generation through Annotation Projection. Rodriguez Miret et al. Information (2024)".</p> <p>There are four different datasets included, namely:</p> <ul> <li>The (1) <strong><a href="https://temu.bsc.es/distemist" target="_blank" rel="noopener">DisTEMIST</a>, </strong>(2) <strong> <a href="https://temu.bsc.es/multicardioner" target="_blank" rel="noopener">DrugTEMIST</a> </strong>and<strong> </strong>(3) <strong><a href="https://temu.bsc.es/meddoprof" target="_blank" rel="noopener">MEDDOPROF</a> Spanish corpora and corresponding versions in 10 different languages</strong>, created through Machine Translation and annotation projection techniques. The Catalan annotations, used in the paper's experiments, were validated by bilingual expert annotators, who also provided alternative translations for the annotated terms in case they were wrongly translated. Thus, for Catalan we provide two different versions of the data: (i) the output of the annotation projection process as is, without any further validation, and (ii) the validated version of the data. For the rest of the languages (with the exception of the DrugTEMIST English and Italian data, used for the MultiCardioNER shared task), only an unvalidated version is provided.<strong><br></strong></li> <li>The (4) <strong>Catalan Clinical Case Corpus (CataCCC)</strong>, a collection of <em>200 clinical case reports in originally written in Catalan </em>covering a variety of clinical specialties. This corpus includes manually validated annotations for diseases, medications and professions created by the experts who annotated the corpora mentioned above, using the same guidelines and annottaion criteria. It can this be considered the first clinical Gold Standard corpus for diseases, medications and processions in Catalan.</li> </ul> <p>It is noteworthy that the MEDDOPROF-related data includes annotations for two labels, PROFESION and SITUACION_LABORAL, but only the former was used for training and evaluation in the paper.</p> <p>These are the <strong>10 languages</strong> included in the repository (along with their language codes):</p> <ul> <li><strong>Spanish</strong> (`es-gs`, with `gs` standing for Gold Standard)</li> <li><strong>Catalan</strong> (`cat`)</li> <li><strong>English</strong> (`en`)</li> <li><strong>French</strong> (`fr`)</li> <li><strong>Italian</strong> (`it`)</li> <li><strong>Dutch</strong> (`nl`)</li> <li><strong>Portuguese</strong> (`pt`)</li> <li><strong>Romanian</strong> (`ro`)</li> <li><strong>Swedish</strong> (`sv`)</li> <li><strong>Czech</strong> (`cz`)</li> </ul> <h2><strong>Related Links</strong></h2> <ul> <li><a href="../doi/10.5281/zenodo.13151039" target="_blank" rel="noopener">Validation and Correction Guidelines for the Multilingual Annotation Projection of Gold Standard Corpora</a></li> <li><a href="https://temu.bsc.es/multicardioner/" target="_blank" rel="noopener">MultiCardioNER Shared Task</a> </li> </ul> <h2><strong>License</strong></h2> <p>This work is licensed under a <a href="http://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution 4.0 International License</a>.</p> <h2><strong>Contact</strong></h2> <p>If you have any questions or suggestions, please contact us at:</p> <p>- Salvador Lima-López (<salvador [dot] limalopez [at] gmail [dot] com>)<br>- Martin Krallinger (<krallinger [dot] martin [at] gmail [dot] com>)</p>
[Replication Package] Enhancing Code Generation for Low-Resource Languages: No Silver Bullet
<p>This repository contains scripts and results related to the work <em>"Enhancing Code Generation for Low-Resource Languages: No Silver Bullet".<br><br></em><strong>Link to the GitHub repository: <a href="https://github.com/Devy99/low-resource-study">https://github.com/Devy99/low-resource-study</a></strong><em><br></em></p>
Chat-GPT MT: Competitive for High- (but not Low-) Resource Languages
<p>System outputs for the paper Chat-GPT MT: Competitive for High- (but not Low-) Resource Languages. Contains outputs for ChatGPT zero-shot and five-shot, GPT-4 five-shot, and Google Translate API</p>
European Language Resources Initiatives – Infrastructural and Policy Issues
<p>3<sup>rd </sup>Lecture</p>
SLURP: A Spoken Language Understanding Resource Package
<p>This resource contains supplementary audio material used in our EMNLP 2020 paper where we release SLURP, a new SLU package containing the following: (1) A new challenging dataset in English spanning 18 domains, which is substantially bigger and linguistically more diverse than existing datasets; (2) Competitive baselines based on state-of-the-art NLU and ASR systems; (3) A new transparent metric for entity labelling which enables a detailed error analysis for identifying potential areas of improvement. Textual annotations are released via the corresponding Github repository (for ease of maintenance).</p>
An explorative Investigation into Neural Machine Translation: the Case of Low-Resource Language Pairs in Burkina Faso
<p>We explore the caveats of a promising research field, namely neural machine translation, in the context of preserving indigenous languages in Africa. We face the challenge of dealing with low-resource language pairs. Methodically, we employ some literature approaches and frameworks to learn translation models from Bible data in Moore (a major language in Burkina Faso) and French. Our experiments indeed confirm previous findings in the literature that vanilla neural machine translation models are ineffective for low resource language pairs. Surprisingly, however, we also found that we are not able to even remotely match performance recorded by the state of the art adapted methods for low-resource language pair in the literature. Nevertheless, although<br> word alignment, Byte Pair Encoding and Adam optimization did not successfully bring reasonable performance, we note that there are many remaining insightful approaches for low-resource language pairs.</p>
Afro-MNIST: Synthetic generation of MNIST-style datasets for low-resource languages
<p>We present Afro-MNIST, a set of synthetic MNIST-style datasets for four orthographies used in Afro-Asiatic and Niger-Congo languages: Ge`ez (Ethiopic), Vai, Osmanya, and N'Ko.<br> These datasets serve as ``drop-in'' replacements for MNIST. We hope that MNIST-style datasets will be developed for other numeral systems, and that these datasets vitalize machine learning education in underrepresented nations in the research community.</p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.