Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
31
datasets available to search
ShareScore release 0.9.0
Dataset results
31 results for “machine translation”
Machine Translation Evaluation Dataset for Amharic
<p><strong>Machine Translation Evaluation Dataset for Amharic </strong></p> <p>The dataset contains source sentences in Amharic and English and their corresponding reference translations that were collected using crowd sourcing. These ground-truth sentences are from across different domains such as news headlines, social media, Wikipedia and everyday conversation.</p> <p><br> <strong>Metadata of files in the dataset</strong></p> <p>amen.tsv<br> - Domain: news | wiki | twitter | convo<br> - Source Sentence: Amharic sentence<br> - Reference Translation: English translation<br> - Google Translate: output of Google Translate<br> - Yandex Translate: output of Yandex Translate</p> <p><br> enam.tsv<br> - Domain: news | wiki | twitter | convo<br> - Source Sentence: English sentence<br> - Reference Translation: Amharic translation<br> - Google Translate: output of Google Translate<br> - Yandex Translate: output of Yandex Translate</p> <p><br> <strong>Amharic source and reference translations across domains:</strong></p> <p><em>News: </em>These are news headlines from Ethiopian news websites.<br> <em>Wikipedia: </em>A random sample of sentences from the Amharic Wikipedia.<br> <em>Twitter: </em>Amharic Twitter posts on consumer products.<br> <em>Conversational: </em>Everyday conversational expressions from Amharic native speakers.</p> <p> </p> <p><strong>English source and reference translations across domains:</strong></p> <p><em>News: </em>These are news headlines from <a href="https://en.wikipedia.org/wiki/Portal:Current_events">Wikipedia current events portal</a>.<br> <em>Wikipedia: </em>A random sample of sentences from the English Wikipedia.<br> <em>Twitter: </em>English Twitter posts on global events from Wikipedia current events portal.<br> <em>Conversational: </em>Everyday conversational expressions from English native speakers.</p> <p> </p> <p><strong>Evaluation of two systems that provide Amharic translation</strong></p> <p>The dataset also contains evaluation of two commercial systems: [Google<br> Translate](https://translate.google.com/) and [Yandex<br> Translate](https://translate.yandex.com/). Both systems provide free APIs that<br> users can sign up and get access keys to. The translations for Amharic to English were generated on 14th<br> February 2020. The translations for English to Amharic were generated on 30th March 2020.</p>
Glossed Hittite Texts with German Translation for Machine Learning
<p>This dataset contains 7,099 processed Hittite texts from 143 CTH numbers, sourced with permission from the <a href="https://www.hethport.uni-wuerzburg.de/HPM/index.php" target="_blank" rel="noopener"><strong>Hethitologie Portal Mainz (HPM)</strong></a>, which is the comprehensive resource of Hittite texts and culture (modern Turkey, c. 1,650 - 1,200 BCE). These texts have been converted from XML format to a tabular structure for computational and linguistic analysis, as well as machine learning applications, while attempting to preserve the full complexity of the source material.</p> <p><strong>Key Features:</strong></p> <ul> <li><strong>txtid</strong>: Identifier for the specific text (e.g., "IBoT 1.30+"), referencing entries in the HPM Konkordanz (S. Košak, hethiter.net/: hetkonk (2.plus)).</li> <li><strong>lnr</strong>: Surface, column (if extant), and line number within the text.</li> <li><strong>cth_number</strong>: <em>Catalogue des textes hittites </em>(CTH) classification, organizing texts by genre and content (S. Košak – G.G.W. Müller – S. Görke – Ch.W. Steitler, hethiter.net/: CTH (2022-10-26)).</li> <li><strong>word</strong>: Orthographic form of the text as presented in the HPM, not in cuneiform but in a standard Latin script representation.</li> <li><strong>translit</strong>: Detailed transliteration, preserving all nuances such as diacritics, broken parts, and editorial markings to reflect the original condition on the text.</li> <li><strong>gloss</strong>: Linguistic glosses, including grammatical, morphological, and semantic information.</li> <li><strong>trans_de</strong>: German translation of the word or phrase.</li> </ul>
HornMT – Machine Translation Benchmark Dataset for Languages in the Horn of Africa
<p>The <strong>HornMT</strong> repository contains data and the associated metadata for the project <a href="https://lesan.ai/benchmark">Machine Translation Benchmark Dataset for Languages in the Horn of Africa</a>. It is a multi-way parallel corpus that will serve as a benchmark to accelerate progress in machine translation research and production systems for languages in the Horn of Africa.</p> <p>Supported Languages</p> <table> <tbody> <tr> <td> <p>Language</p> </td> <td> <p>ISO 639-3 code</p> </td> </tr> </tbody> <tbody> <tr> <td> <p>Afar</p> </td> <td> <p>aaf</p> </td> </tr> <tr> <td> <p>Amharic</p> </td> <td> <p>amh</p> </td> </tr> <tr> <td> <p>English</p> </td> <td> <p>eng</p> </td> </tr> <tr> <td> <p>Oromo</p> </td> <td> <p>orm</p> </td> </tr> <tr> <td> <p>Somali</p> </td> <td> <p>som</p> </td> </tr> <tr> <td> <p>Tigrinya</p> </td> <td> <p>tir</p> </td> </tr> </tbody> </table> <p><strong> </strong></p> <p>data/ contains one text file per language and each file contains news snippets in the same order for each language.</p> <p>data<br> ├── aar.txt<br> ├── amh.txt<br> ├── eng.txt<br> ├── orm.txt<br> ├── som.txt<br> └── tir.txt</p> <p>metadata.tsv contains tab separated data describing each news snippet. The metadata contains the following fields.</p> <ul> <li> <p><strong>Scope</strong> - describes whether the news is global or local. It takes two values: Global news and Local news.</p> </li> <li> <p><strong>Category</strong> - News category covering the following 12 topics</p> <ul> <li> <p>Art and Culture</p> </li> <li> <p>Business and Economy</p> </li> <li> <p>Conflicts and Attacks</p> </li> <li> <p>Disaster and Accidents</p> </li> <li> <p>Entertainment</p> </li> <li> <p>Environment</p> </li> <li> <p>Health</p> </li> <li> <p>International Relations</p> </li> <li> <p>Law and Crime</p> </li> <li> <p>Politics</p> </li> <li> <p>Science and Technology</p> </li> <li> <p>Sport</p> </li> </ul> </li> <li> <p><strong>Source</strong> - List of one or more URLs from which the news content is extracted or based on.</p> </li> <li> <p><strong>Domain</strong> - TLD corresponding to the URL(s) in Source.</p> </li> <li> <p><strong>Date</strong> - The publication date of the source article. The format is yyyy-mm-dd.</p> </li> </ul> <p>Other formats</p> <p>All the data and associated metadata together in one file is also available in other file formats.</p> <p><strong>HornMT.xlsx</strong> - data and associated metadata in xlsx format.</p> <p><strong>HornMT.json</strong> - data and associated metadata in json format.</p> <p>Below is an example row.</p> <pre><code class="language-javascript">{ "data":{ "eng":"The World Meteorological Organisation reports that the ozone layer is damaged to its worst extent ever in the Arctic.", "aaf":"Baad Metrolojih Eglali Areketekeh Addal Ozonih qelu faxe waktik lafetle calat biyakisem xayose.", "amh":"የአለም የአየር ንብረት ድርጅት በአርክቲክ አካባቢ ያለው የኦዞን ምንጣፍ ከፍተኛ ጉዳት እንደደረሰበት አስታወቀ፡፡", "orm":"Dhaabbanni Meetiroolojii Addunyaa baqqaanni oozonii Arkiitik keessatti gara sadarkaa isa hamaa haga ammaatti akka miidhame gabaase.", "som":"Ururka Saadaasha Hawada Adduunka ayaa ku warramaya in lakabka ozoneka ee Ka koreeya dhulka baraflayda uu waxyeelladii abid ugu darnaa soo gaadhay.", "tir":"ውድብ ሜትሮሎጂ ዓለም ኣብ ኣርክቲክ ዝርከብ ናሕሲ ኦዞን ኣዝዩ ብዝኸፍአ ደረጃ ከምዝተጎድአ ሓቢሩ፡፡" }, "metadata":{ "scope":"Global", "category":"Science and Technology", "source":"https://www.independent.co.uk/environment/climate-change/ozone-layer-damaged-by-unusually-harsh-winter-2263653.html", "domain":"www.independent.co.uk", "date":"2011-04-05" } }</code></pre> <p><strong>Team</strong></p> <p>Afar</p> <ul> <li> <p>Mohammed Deresa</p> </li> <li> <p>Yasin Nur</p> </li> </ul> <p>Amharic</p> <ul> <li> <p>Tigist Taye</p> </li> <li> <p>Selamawit Hailemariam</p> </li> <li> <p>Wako Tilahun</p> </li> </ul> <p>Oromo</p> <ul> <li> <p>Gemechis Melkamu</p> </li> <li> <p>Galata Girmaye</p> </li> </ul> <p>Somali</p> <ul> <li> <p>Abdiselam Mohamed</p> </li> <li> <p>Beshir Abdi</p> </li> </ul> <p>Tigrinya</p> <ul> <li> <p>Berhanu Abadi Weldegiorgis</p> </li> <li> <p>Michael Minassie</p> </li> <li> <p>Nureddin Mohammedshiek</p> </li> </ul> <p><strong>Project Leaders</strong></p> <ul> <li> <p>Asmelash Teka Hadgu <a href="mailto:asme@lesan.ai">asme@lesan.ai</a></p> </li> <li> <p>Gebrekirstos G. Gebremeskel <a href="mailto:gebrekirstos.gebremeskel@ru.nl">gebrekirstos.gebremeskel@ru.nl</a></p> </li> <li> <p>Abel Aregawi <a href="mailto:abel@lesan.ai">abel@lesan.ai</a></p> </li> </ul> <p><strong>License</strong></p> <p>Shield: <a href="http://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a></p> <p>This work is licensed under a<br> <a href="http://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution 4.0 International License</a>.</p>
Accompanying dataset for: A Monte Carlo Method for Metamorphic Testing of Machine Translation Services
<p>This dataset includes enhanced analysis of the machine translation data. The original dataset has been reported in [1], where white spaces were used to separate words in different languages. This is however not the best method of analyzing some Asian languages such as the Chinese language. In the present analysis, we used a character-based approach to separating the Chinese and Japanese results, hence obtaining a different set of BLEU and Cosine Similarity scores. These new scores are given in the present dataset.</p> <p>[1] Daniel Pesu, Zhi Quan Zhou, Jingfeng Zhen, & Dave Towey. (2018). Accompanying dataset for: A Monte Carlo Method for Metamorphic Testing of Machine Translation Services (Version 1.0) [Data set]. Zenodo. http://doi.org/10.5281/zenodo.1194560</p>
Nos_Machine translation test suite English-Galician
<p>364 english-galician parallel sentences for the evaluation of machine translation. Sentences are classified in categories based on challenging linguistic phenomena.</p>
Nos_Machine translation test suite Spanish-Galician
<p>334 spanish-galician parallel sentences for the evaluation of machine translation. Sentences are classified in categories based on challenging linguistic phenomena.</p>
Nos_Machine translation gold standard English-Galician 1
<p>1777 carefully curated Galician-English parallel sentences for the evaluation of automatic machine translation. Sentence pairs are classified according to whether the translations are highly literal or not.</p>
Nos_Machine translation gold standard Spanish-Galician 2
<p>1998 carefully curated Galician-Spanish parallel sentences for the evaluation of automatic machine translation. Sentence pairs are classified according to whether the translations are highly literal or not.</p>
Nos_Machine translation gold standard English-Galician 2
<p>1777 carefully curated Galician-English parallel sentences for the evaluation of automatic machine translation. Sentence pairs are classified according to whether the translations are highly literal or not.</p>
Nos_Machine translation gold standard Spanish-Galician 1
<p>1998 carefully curated Galician-Spanish parallel sentences for the evaluation of automatic machine translation. Sentence pairs are classified according to whether the translations are highly literal or not.</p>
Spanish abstracts from PubMed (machine-translated from English)
<p>Spanish abstracts from PubMed (machine-translated from English)</p> <p>A state-of-the-art, domain-specific Neural Machine Translation system has been used to automatically translate a large number of articles (titles and abstracts) from PubMed, from English to Spanish. This dataset, in JSON format, provides MeSH terms and DeCS codes (if available), as well as other fields such as the year of publication for all articles.</p> <p>Copyright (c) 2020 Secretaría de Estado de Digitalización e Inteligencia Artificial</p>
Adapting Phrase-based Machine Translation to Normalise Medical Terms in Social Media Messages
<p>Data and supplementary information for the paper entitled "Adapting Phrase-based Machine Translation to Normalise Medical Terms in Social Media Messages" to be published at EMNLP 2015: Conference on Empirical Methods in Natural Language Processing — September 17–21, 2015 — Lisboa, Portugal.</p> <p>ABSTRACT: Previous studies have shown that health reports in social media, such as DailyStrength and Twitter, have potential for monitoring health conditions (e.g. adverse drug reactions, infectious diseases) in particular communities. However, in order for a machine to understand and make inferences on these health conditions, the ability to recognise when laymen's terms refer to a particular medical concept (i.e. text normalisation) is required. To achieve this, we propose to adapt an existing phrase-based machine translation (MT) technique and a vector representation of words to map between a social media phrase and a medical concept. We evaluate our proposed approach using a collection of phrases from tweets related to adverse drug reactions. Our experimental results show that the combination of a phrase-based MT technique and the similarity between word vector representations outperforms the baselines that apply only either of them by up to 55%.</p>
Machine Assisted Translation of Wikipedia Articles into Low Resource Languages
<p><strong>Wikipedia is the largest encyclopedia ever assembled with the vision of enabling every human being to freely share in the sum of all knowledge. Wikipedia currently has a total of more than six million articles and over 17 billion words in its English edition. Unfortunately, millions of people cannot access this resource because it’s not available in their language. For instance, at the moment there are only 218 Tigrinya Wikipedia and 15,018 Amharic Wikipedia articles.</strong></p> <p><strong>In this project, we investigate the problem of translating Wikipedia articles from a high resource language into low resource languages using human-in-the-loop MT systems. In particular, we investigate different approaches to translate a sample of English Wikipedia articles into Tigrinya and Amharic. Currently, this repository contains 100k English Wikipeida abstracts translated using Lesan (https://lesan.ai) into Amharic and Tigrinya.</strong><br> <br> </p> <p><strong>Structure of data directory:</strong></p> <p><strong>data<br> ├── human<br> └── mt<br> ├── google<br> ├── lesan<br> │ ├── am.txt<br> │ ├── en.txt<br> │ └── ti.txt<br> └── microsoft</strong><br> </p>
Machine-readable Northern Karelian Proper-Livvi bilingual translation dictionary
<p>This machine readable bilingual translation dictionary of Northern Karelian Proper (ISO-639: krl) to Livvi aka Olonets-Karelian (ISO-639 olo) was created by Timo Rantakaulio during Google Summer of Code 2021 at Apertium. His work was facilitated through an online back-end dictionary editing tool `https://akusanat.com/verdd' with translation suggestions generated by Khalid Alnajjar and Mika Hämäläinen developers of the Verdd editor. Workflow, coordination, instruction and advice were given by Flammie A. Pirinen and Jack Rueter.</p>
Machine-readable Finnish-Livvi bilingual translation dictionary
<p>This machine readable bilingual translation dictionary of Finnish to Livvi aka Olonets-Karelian (ISO-396: olo) was proofread and extended by Timo Rantakaulio during Google Summer of Code 2021 at Apertium. His work was facilitated through an online back-end dictionary editing tool `https://akusanat.com/verdd' with translation suggestions generated by Khalid Alnajjar and Mika Hämäläinen developers of the Verdd editor. Workflow, coordination, instruction and advice were given by Flammie A. Pirinen and Jack Rueter.</p>
Machine-readable Finnish-Karelian bilingual translation dictionary
<p>This machine readable bilingual translation dictionary of Finnish to Northern Karelian Proper (ISO-396: krl) was proofread and extended by Timo Rantakaulio during Google Summer of Code 2021 at Apertium. His work was facilitated through an online back-end dictionary editing tool `https://akusanat.com/verdd' with translation suggestions generated by Khalid Alnajjar and Mika Hämäläinen developers of the Verdd editor. Workflow, coordination, instruction and advice were given by Flammie A. Pirinen and Jack Rueter.</p>
Accompanying dataset for: A Monte Carlo Method for Metamorphic Testing of Machine Translation Services
<p>This is the original dataset, where white spaces are used to separate words of all languages. This is however not the best method of analyzing some Asian languages. Please refer to the following new version for enhanced, character-based analysis results:</p> <p>Zhi Quan Zhou. (2018). Accompanying dataset for: A Monte Carlo Method for Metamorphic Testing of Machine Translation Services (Version 2.0) [Data set]. Zenodo. http://doi.org/10.5281/zenodo.1230139</p>
Neural Machine Translation for the Biomedical Domain - WMT19
<p>This package contains the files needed to use the Neural Machine Translation (NMT) system for the Biomedical Domain.</p> <p>The available language directions for translation are:</p> <ul> <li>English to Spanish</li> <li>Spanish to English</li> <li>English to Portuguese</li> <li>Portuguese to English</li> <li>Spanish to Portuguese</li> <li>Portuguese to Spanish</li> </ul> <p>The code for using the translation files is in <a href="https://github.com/PlanTL-SANIDAD/Medical-Translator-WMT19">https://github.com/PlanTL-SANIDAD/Medical-Translator-WMT19</a></p> <p> </p> <p>Copyright (c) 2019 Secretaría de Estado para el Avance Digital</p>
Bugsplainer: Explaining Software Bugs Leveraging Code Structures in Neural Machine Translation
<p>Software bugs claim approximately 50% of development time and cost the global economy billions of dollars. Once a bug is reported, the assigned developer attempts to identify and understand the source code responsible for the bug and then corrects the code. Over the last five decades, there has been significant research on automatically finding or correcting software bugs. However, there has been little research on automatically explaining the bugs to the developers, which is essential but a highly challenging task. In this paper, we propose Bugsplainer, a transformer-based generative model, that generates natural language explanations for software bugs by learning from a large corpus of bug-fix commits. Bugsplainer can leverage structural information and buggy patterns from the source code to generate an explanation for a bug. Our evaluation using three performance metrics shows that Bugsplainer can generate understandable and good explanations according to Google's standard, and can outperform multiple baselines from the literature. We also conduct a developer study involving 20 participants where the explanations from Bugsplainer were found to be more accurate, more precise, more concise and more useful than the baselines.</p>
machine translation bias
<p>Dataset and code for study investigating bias in machine translation</p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.