PuoBERTa + PuoBERTaJW300 Setswana Language Models

<p><strong>PuoBERTa +&nbsp;PuoBERTaJW300: Setswana Language Models</strong></p><p>A Roberta-based language model specially designed for Setswana, using the new PuoData dataset (PuoBERTa) and PuoData + JW300 TSN (PuoBERTaJW300)</p><p><strong>Cite&nbsp;</strong></p><p>@inproceedings{marivate2023puoberta, title &nbsp; = {PuoBERTa: Training and evaluation of a curated language model for Setswana}, author &nbsp;= {Vukosi Marivate and Moseli Mots'Oehli and Valencia Wagner and Richard Lastrucci and Isheanesu Dzingirai}, year &nbsp; &nbsp;= {2023}, booktitle= {Artificial Intelligence Research. SACAIR 2023. Communications in Computer and Information Science}, url= {https://link.springer.com/chapter/10.1007/978-3-031-49002-6_17}, keywords = {NLP}, preprint_url = {https://arxiv.org/abs/2310.09141}, dataset_url = {https://github.com/dsfsi/PuoBERTa}, software_url = {https://huggingface.co/dsfsi/PuoBERTa} }</p><p><strong>Model Details</strong></p><p>Model Description</p><p>This is a masked language model trained on Setswana corpora, making it a valuable tool for a range of downstream applications from translation to content creation. It's powered by the PuoData dataset to ensure accuracy and cultural relevance.</p><ul><li><strong>Developed by:</strong>&nbsp;Vukosi Marivate (<a href="https://huggingface.co/@vukosi">@vukosi</a>), Moseli Mots'Oehli (<a href="https://huggingface.co/@MoseliMotsoehli">@MoseliMotsoehli</a>) , Valencia Wagner, Richard Lastrucci and Isheanesu Dzingirai</li><li><strong>Model type:</strong>&nbsp;RoBERTa Model</li><li><strong>Language(s) (NLP):</strong>&nbsp;Setswana</li><li><strong>License:</strong>&nbsp;CC BY 4.0</li></ul><p>Usage</p><p>Use this model filling in masks or finetune for downstream tasks. Here's a simple example for masked prediction:</p><p>from transformers import RobertaTokenizer, RobertaModel # Load model and tokenizer model = RobertaModel.from_pretrained('dsfsi/PuoBERTa') tokenizer = RobertaTokenizer.from_pretrained('dsfsi/PuoBERTa')</p><p>&nbsp;</p><p>Downstream Use</p><p>Downstream Performance</p><p><i><strong>MasakhaPOS</strong></i></p><p>Performance of models on the MasakhaPOS downstream task.</p><p>Model Test Performance&nbsp;</p><p><strong>Multilingual Models</strong></p><p>AfroLM</p><p>83.8</p><p>AfriBERTa</p><p>82.5</p><p>AfroXLMR-base</p><p>82.7</p><p>AfroXLMR-large</p><p>83.0</p><p><strong>Monolingual Models</strong></p><p>NCHLT TSN RoBERTa</p><p>82.3</p><p>PuoBERTa</p><p><strong>83.4</strong></p><p>PuoBERTa+JW300</p><p><strong>84.1</strong></p><p>&nbsp;</p><p><i><strong>MasakhaNER</strong></i></p><p>Performance of models on the MasakhaNER downstream task.</p><p>Model Test Performance (f1 score)&nbsp;</p><p><strong>Multilingual Models</strong></p><p>AfriBERTa</p><p>83.2</p><p>AfroXLMR-base</p><p>87.7</p><p>AfroXLMR-large</p><p>89.4</p><p><strong>Monolingual Models</strong></p><p>NCHLT TSN RoBERTa</p><p>74.2</p><p>PuoBERTa</p><p><strong>78.2</strong></p><p>PuoBERTa+JW300</p><p><strong>80.2</strong></p><p>&nbsp;</p><p><strong>Dataset</strong></p><p>We used the PuoData dataset, a rich source of Setswana text, ensuring that our model is well-trained and culturally attuned.</p><p>Citation Information</p><p><strong>Bibtex Reference</strong></p><p>@inproceedings{marivate2023puoberta, title &nbsp; = {PuoBERTa: Training and evaluation of a curated language model for Setswana}, author &nbsp;= {Vukosi Marivate and Moseli Mots'Oehli and Valencia Wagner and Richard Lastrucci and Isheanesu Dzingirai}, year &nbsp; &nbsp;= {2023}, booktitle= {Artificial Intelligence Research. SACAIR 2023. Communications in Computer and Information Science}, url= {https://link.springer.com/chapter/10.1007/978-3-031-49002-6_17}, keywords = {NLP}, preprint_url = {https://arxiv.org/abs/2310.09141}, dataset_url = {https://github.com/dsfsi/PuoBERTa}, software_url = {https://huggingface.co/dsfsi/PuoBERTa} }</p><p>Contributing</p><p>Your contributions are welcome! Feel free to improve the model.</p><p>Model Card Authors</p><p>Vukosi Marivate</p><p>Model Card Contact</p><p>For more details, reach out or check our&nbsp;<a href="https://dsfsi.github.io/">website</a>.</p><p>Email:&nbsp;<a href="mailto:vukosi.marivate@cs.up.ac.za">vukosi.marivate@cs.up.ac.za</a></p><p><strong>Enjoy exploring Setswana through AI!</strong></p>