HornMT – Machine Translation Benchmark Dataset for Languages in the Horn of Africa

<p>The <strong>HornMT</strong> repository contains data and the associated metadata for the project <a href="https://lesan.ai/benchmark">Machine Translation Benchmark Dataset for Languages in the Horn of Africa</a>. It is a multi-way parallel corpus that will serve as a benchmark to accelerate progress in machine translation research and production systems for languages in the Horn of Africa.</p> <p>Supported Languages</p> <table> <tbody> <tr> <td> <p>Language</p> </td> <td> <p>ISO 639-3 code</p> </td> </tr> </tbody> <tbody> <tr> <td> <p>Afar</p> </td> <td> <p>aaf</p> </td> </tr> <tr> <td> <p>Amharic</p> </td> <td> <p>amh</p> </td> </tr> <tr> <td> <p>English</p> </td> <td> <p>eng</p> </td> </tr> <tr> <td> <p>Oromo</p> </td> <td> <p>orm</p> </td> </tr> <tr> <td> <p>Somali</p> </td> <td> <p>som</p> </td> </tr> <tr> <td> <p>Tigrinya</p> </td> <td> <p>tir</p> </td> </tr> </tbody> </table> <p><strong>&nbsp;</strong></p> <p>data/ contains one text file per language and each file contains news snippets in the same order for each language.</p> <p>data<br> ├── aar.txt<br> ├── amh.txt<br> ├── eng.txt<br> ├── orm.txt<br> ├── som.txt<br> └── tir.txt</p> <p>metadata.tsv contains tab separated data describing each news snippet. The metadata contains the following fields.</p> <ul> <li> <p><strong>Scope</strong> - describes whether the news is global or local. It takes two values: Global news and Local news.</p> </li> <li> <p><strong>Category</strong> - News category covering the following 12 topics</p> <ul> <li> <p>Art and Culture</p> </li> <li> <p>Business and Economy</p> </li> <li> <p>Conflicts and Attacks</p> </li> <li> <p>Disaster and Accidents</p> </li> <li> <p>Entertainment</p> </li> <li> <p>Environment</p> </li> <li> <p>Health</p> </li> <li> <p>International Relations</p> </li> <li> <p>Law and Crime</p> </li> <li> <p>Politics</p> </li> <li> <p>Science and Technology</p> </li> <li> <p>Sport</p> </li> </ul> </li> <li> <p><strong>Source</strong> - List of one or more URLs from which the news content is extracted or based on.</p> </li> <li> <p><strong>Domain</strong> - TLD corresponding to the URL(s) in Source.</p> </li> <li> <p><strong>Date</strong> - The publication date of the source article. The format is yyyy-mm-dd.</p> </li> </ul> <p>Other formats</p> <p>All the data and associated metadata together in one file is also available in other file formats.</p> <p><strong>HornMT.xlsx</strong> - data and associated metadata in xlsx format.</p> <p><strong>HornMT.json</strong> - data and associated metadata in json format.</p> <p>Below is an example row.</p> <pre><code class="language-javascript">{ "data":{ "eng":"The World Meteorological Organisation reports that the ozone layer is damaged to its worst extent ever in the Arctic.", "aaf":"Baad Metrolojih Eglali Areketekeh Addal Ozonih qelu faxe waktik lafetle calat biyakisem xayose.", "amh":"የአለም የአየር ንብረት ድርጅት በአርክቲክ አካባቢ ያለው የኦዞን ምንጣፍ ከፍተኛ ጉዳት እንደደረሰበት አስታወቀ፡፡", "orm":"Dhaabbanni Meetiroolojii Addunyaa baqqaanni oozonii Arkiitik keessatti gara sadarkaa isa hamaa haga ammaatti akka miidhame gabaase.", "som":"Ururka Saadaasha Hawada Adduunka ayaa ku warramaya in lakabka ozoneka ee Ka koreeya dhulka baraflayda uu waxyeelladii abid ugu darnaa soo gaadhay.", "tir":"ውድብ ሜትሮሎጂ ዓለም ኣብ ኣርክቲክ ዝርከብ ናሕሲ ኦዞን ኣዝዩ ብዝኸፍአ ደረጃ ከምዝተጎድአ ሓቢሩ፡፡" }, "metadata":{ "scope":"Global", "category":"Science and Technology", "source":"https://www.independent.co.uk/environment/climate-change/ozone-layer-damaged-by-unusually-harsh-winter-2263653.html", "domain":"www.independent.co.uk", "date":"2011-04-05" } }</code></pre> <p><strong>Team</strong></p> <p>Afar</p> <ul> <li> <p>Mohammed Deresa</p> </li> <li> <p>Yasin Nur</p> </li> </ul> <p>Amharic</p> <ul> <li> <p>Tigist Taye</p> </li> <li> <p>Selamawit Hailemariam</p> </li> <li> <p>Wako Tilahun</p> </li> </ul> <p>Oromo</p> <ul> <li> <p>Gemechis Melkamu</p> </li> <li> <p>Galata Girmaye</p> </li> </ul> <p>Somali</p> <ul> <li> <p>Abdiselam Mohamed</p> </li> <li> <p>Beshir Abdi</p> </li> </ul> <p>Tigrinya</p> <ul> <li> <p>Berhanu Abadi Weldegiorgis</p> </li> <li> <p>Michael Minassie</p> </li> <li> <p>Nureddin Mohammedshiek</p> </li> </ul> <p><strong>Project Leaders</strong></p> <ul> <li> <p>Asmelash Teka Hadgu <a href="mailto:asme@lesan.ai">asme@lesan.ai</a></p> </li> <li> <p>Gebrekirstos G. Gebremeskel <a href="mailto:gebrekirstos.gebremeskel@ru.nl">gebrekirstos.gebremeskel@ru.nl</a></p> </li> <li> <p>Abel Aregawi <a href="mailto:abel@lesan.ai">abel@lesan.ai</a></p> </li> </ul> <p><strong>License</strong></p> <p>Shield: <a href="http://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a></p> <p>This work is licensed under a<br> <a href="http://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution 4.0 International License</a>.</p>