Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
27
datasets available to search
ShareScore release 0.9.0
Dataset results
27 results for “language classification”
CLDF dataset derived from Valenzuela, Pilar and Roberto Zariquiey's " Language classification in Western Amazonia: advances in favor of the Pano-Takana Hypothesis" from 2023
<p>Cite the source of the dataset as:</p> <blockquote> <p>Valenzuela, Pilar and Zariquiey, Roberto (2023). Language classification in Western Amazonia: advances in favor of the Pano-Takana Hypothesis. LIAMES: Línguas Indígenas Americanas, Campinas, SP. https://doi.org/10.20396/liames.v23i00.8670150</p> </blockquote>
Language intensity classification and Neuronal Networks
<p>Kohonen Self-Organizing Maps (SOM) are a particular type of artificial neural network created by Teuvo Kohonen. Their unsupervised learning makes them suitable for application, among other things, to grouping tasks. Occupations have been grouped into this analysis considering the similarity in value of the variables that define this occupation in terms of language proficiency requirements. The 8 variables used were: v1-speaking skills, v2-writing skills, v3-speech clarity, v4-speech recognition, v5-English knowledge, v6-speaking ability, v7-communication with people outsiders, v8-communication with superiors, equals or subordinates. After applying this authomatic classification technique, the SOC occupations are classified according to five distinc groups with decreasing linguistic intensity:</p> <p>Class 1: High linguistic intenisty requirements<br> Class 2: Medium-high linguistic intenisty requirements<br> Class 3: Medium linguistic intenisty requirements<br> Class 4: Medium-low linguistic intenisty requirements<br> Class 5: Low linguistic intenisty requirements</p>
CLDF dataset derived from Blum et al.'s "A phylolinguistic classification of the Quechua language family" from 2023
<p>Cite the source of the dataset as:</p> <blockquote> <p>Blum, Frederic, Carlos Barrientos, Adriano Ingunza & Zoe Poirier. 2023. A phylolinguistic classification of the Quechua language family. INDIANA - Anthropological Studies on Latin America and the Caribbean 40(1). 29–-54. DOI: https://doi.org/10.18441/IND.V40I1.29-54.</p> </blockquote>
CLDF dataset derived from Peiros' "Genetic classification of Austro-Asiatic languages" from 2004
<p>Cite the source of the dataset as:</p> <blockquote> <p>Peiros, I. I. (2004): Genetičeskaja klassifikacija avstroaziatskix jazykov [Genetic classification of Austro-Asiatic languages]. Russian State University for the Humanities, Russian State University for the Humanities, Moscow.</p> </blockquote>
CLDF dataset derived from Robinson and Holton's "Internal Classification of the Alor-Pantar Language Family" from 2012
<p>Cite the source of the dataset as:</p> <blockquote> <p>Robinson, Laura C. and Holton, Gary (2012): Internal Classification of the Alor-Pantar Language Family Using Computational Methods Applied to the Lexicon. Language Dynamics and Change 2.2. 123-149.</p> </blockquote>
CLDF dataset derived from Gerardi and Reichert's "The Tupí-Guaraní Language Family: A Phylogenetic Classification" from 2021
<p>Cite the source of the dataset as:</p> <blockquote> <p>Ferraz Gerardi, Fabrício and Reichert, Stanislav (2021) The Tupí-Guaraní Language Family: A Phylogenetic Classification. Diachronica 38(2). 151--188. DOI: https://doi.org/10.1075/dia.18032.fer.</p> </blockquote>
Report on Transformers interpretability for Natural Language Processing: A case study on Technical Debt classification
<p>Transformer models have significantly advanced the field of natural language processing (NLP), achieving exceptional results in various tasks. However, these models are often seen as "black boxes", providing limited insight into the factors influencing their predictions. It has become crucial to develop and utilise methods for interpreting and explaining these models to uncover their complex inner workings. This report discusses the latest techniques and tools that aid in a more profound understanding of transformer models within NLP. Additionally, it explores a vital industrial use case: Technical Debt (TD) classification. In this context, the report leverages transformer model interpretability tools and Retrieval Augmented Generation (RAG) to analyse and understand the characteristics of text in Github issues, distinguishing between TD and non-TD.</p> <p>This report thoroughly outlines an approach to improve the transparency and reproducibility of machine learning models, with a special emphasis on TD classification. It integrates the RAG approach and exploits feature attribution techniques, presenting a route to create AI systems that are not only high-performing but also demonstrably trustworthy and comprehensible. Through a detailed examination of word patterns in TD classification and the innovative use of the RAG approach, the research highlights a strong dedication to promoting transparency and responsibility in AI systems, potentially ushering in a new phase in machine learning research that focuses on clarity and dependability.</p>
Technical Debt Classification in Issue Trackers using Natural Language Processing based on Transformers
<p>In order to ensure transparency and reproducibility, we have made everything available publicly here, including the Code, Models, Datasets and more. All the files and their functionality used in this paper are explained clearly in the <strong>README.md</strong> file.</p> <p>Background: Technical Debt (TD) needs to be controlled and tracked during software development. Support to automatically track TD in issue trackers is limited. </p> <p>Aim: We explore the usage of a large dataset of developer-labeled TD issues in combination with cutting-edge Natural Language Processing (NLP) approaches to automatically classify TD in issue trackers.</p> <p>Method: We mine and analyze more than 160GB of textual data from GitHub projects, collecting over 55,600 TD issues and consolidating them into a large dataset (GTD dataset). We use such datasets to train and test Transformer ML models. Then we test the model's generalization ability by testing them on six unseen projects. Finally, we re-train the models including part of the TD issues from the target project to test their adaptability. </p> <p>Results and Conclusion: (i) We create and release the GTD dataset, a comprehensive dataset including TD issues from 6,401 public repositories with various contexts; (ii) By training Transformers using the GTD dataset, we achieve performance metrics that are promising; (iii) Our results are a significant step forward towards supporting the automatic classification of TD in issue trackers, especially when the models are adapted to the context of unseen projects after fine-tuning.</p>
Large Language Model-Based Classification of Flash Flood Impacts Across the United States
<p>This repository contains the data sets used for the publication of the journal article titled <em>Large Language Model-Based Classification of Flash Flood Impacts Across the United States</em>.</p> <p>This is the first release of the data with a Zenodo DOI attached to the README.md file. </p> <p>Further information about the data can be found in the GitHub repository's README.md file.</p>
Phlorest phylogeny derived from Robinson and Holton 2012 'Internal Classification of the Alor-Pantar Language Family Using Computational Methods Applied to the Lexicon'
<p>Cite the source of the dataset as:</p> <blockquote> <p>Robinson, L. C., & Holton, G. (2012). Internal Classification of the Alor-Pantar Language Family Using Computational Methods Applied to the Lexicon. Language Dynamics and Change, 2(2), 123-149. doi:10.1163/22105832-20120201</p> </blockquote>
SpokeN-100: A Cross-Lingual Benchmarking Dataset for The Classification of Spoken Numbers in Different Languages
<div> <div>SpokeN-100 is a novel, entirely artificially generated benchmarking dataset tailored for speech recognition, representing a core challenge in the field of tiny deep learning. SpokeN-100 consists of spoken numbers from 0 to 99 spoken by 32 different speakers in four different languages, namely English, Mandarin, German and French, resulting in 12,800 audio samples.</div> </div>
Complementary Dataset for Semitic Languages Classification
<p>The dataset was first presented at the 14TH ITALIAN MEETING OF AFRO-ASIATIC LINGUISTICS conference in Torino (2011) and is used in an article Zemánek 2017.</p> <p>The dataset is constructed as a complement to the datasets contained in Faber 1997 and Gai 1994 in order to study possibilities of enlargement of the list of grammatical features usable in the classification of the Semitic languages. The aim of the dataset was to be comparable in size with the two other datasets because of the balance requirements.</p>
CLDF Dataset derived from Syrjänen et al.'s "Shedding more light on language classification" from 2013
<p>Cite the source of the dataset as:</p> <blockquote> <p>Syrjänen, K.; Honkola, T.; Korhonen, K.; Lehtinen, J.; Vesakoski, O. &amp; Wahlber, N. Shedding more light on language classification using basic vocabularies and phylogenetic methods. Diachronica, 2013, 30, 323-352</p> </blockquote>
CLDF dataset derived from Oskolskaya et al.'s "A Bayesian approach to the classification of Tungusic languages" from 2021
<p>Cite the source of the dataset as:</p> <blockquote> <p>Oskolskaya, S., Koile, E., & Robbeets, M. (2021). A Bayesian approach to the classification of Tungusic languages. Diachronica, 39(1), 128–158. https://doi.org/10.1075/dia.20010.osk</p> </blockquote>
Classification of Mobile Application Reviews using Deep Language Models
<p>supplementary material for ASE 2022 "Classification of Mobile Application Reviews using Deep Language Models"</p>
The classification of Cosao: a Lolo-Burmese language of China and Laos
<p>Abstract</p> <p>Cosao is a Tibeto-Burman language of the Lolo-Burmese branch spoken in three villages that straddle the border China-Laos border. It is spoken by about 150 people in the village of Man'gang in Mengban Town, Mengla County, Yunnan Province, China, as well as by approximately 400 speakers in two villages across the border in Phongsaly Province, Laos. Lexical data of Cosao (autonym: tsho55 sɔ55) has only recently been published in Bai Bibo (2015). Bai (2015) recognizes that Cosao is a Lolo-Burmese language, but provides no arguments about where exactly Cosao fits within the Lolo-Burmese branch.</p> <p>Using comparative lexical evidence, this talk explains how Cosao is a Southern Loloish language most closely related to the Khir language of Laos, and also shows how Sida, Sila, Phusang, Khir, and Cosao all constitute a cluster of closely related languages within the Southern Loloish branch. Computational phylogenetics is also used to show the position of this cluster of languages within Southern Loloish.</p>
The classification of Na Meo, a Hmong-Mien language of Vietnam
<p>Abstract</p> <p>Na Meo is a language spoken in a cluster of villages encompassing the northern Vietnam provinces of Lang Son, Cao Bang, and Bac Kan, as well as a single village in Tuyen Quang province (Nguyen 2007). Its existence as a divergent Hmong-Mien language has been noted by the Vietnamese government since 1975. However, Na Meo has remained very poorly documented, and is currently still listed as an unclassified language in Ethnologue (ISO 639-3 code [neo]) due to the lack of published data. The lengthiest published word list to date is that of Nguyen (2007), which has 67 Na Meo words in non-IPA orthography.</p> <p>Within Tuyên Quang province, Na Meo is spoken only in the single village of Khuân Hẻ, Kim Quan commune, Yên Sơn district. In February 2014, I had personally collected nearly 400 lexical items from Na Meo of Khuân Hẻ. This Na Meo variety was previously undocumented, and the commune location is known only from Nguyen (2007). This variety is very similar to, and likely mutually intelligible with, the Na Meo data in Jerold Edmondson (n.d.). Edmondson (n.d.), a handwritten manuscript of about 500 lexical items of Na Meo, was the result of a field expedition in Cao Bang province during the late 1990’s, and the data in there has remained unpublished to date.</p> <p>Following suggestions from Edmondson (p.c.) and Nguyen (2007) that Na Meo may likely be a Qiandong Miao (Central Miao) language belonging to the Hmongic branch of Hmong-Mien, I will support these claims by demonstrating lexical isoglosses and phonological features that are shared exclusively among Na Meo and Qiandong Miao languages, but not in other branches of Hmongic. Since Qiandong Miao includes diverse lects spoken mainly in Guizhou but also in neighboring Hunan and Guangxi provinces, I will then further compare Tuyen Quang and Cao Bang Na Meo with the Qiandong Miao dialects covered in Li (2000) and Chen (2013) to determine where in China the Na Meo speakers’ ancestors had most likely migrated from. Comparison shows that Na Meo is most similar to Southern Qiandong Miao dialects spoken in Guangxi province, a result of one of the many migrations that had brought various ethnic groups from southern China to northern Vietnam during the turbulent Qing Dynasty period.</p>
Source Code Classifications: Code Dataset of Programming Languages
Open the record for dataset details and reuse information.
Examining the Influence of Political Bias on Large Language Model Performance in Stance Classification
<p>Code and dataset for paper "Examining the Influence of Political Bias on Large Language Model Performance in Stance Classification". ICWSM 2025</p> <p>Preprint: https://arxiv.org/abs/2407.17688</p> <p>Citation: </p> <p>@misc{ng2024examininginfluencepoliticalbias,<br> title={Examining the Influence of Political Bias on Large Language Model Performance in Stance Classification}, <br> author={Lynnette Hui Xian Ng and Iain Cruickshank and Roy Ka-Wei Lee},<br> year={2024},<br> eprint={2407.17688},<br> archivePrefix={arXiv},<br> primaryClass={cs.CL},<br> url={https://arxiv.org/abs/2407.17688}, <br>}</p>
Combining Natural Language and Images for Garbage Classification: A Public Benchmark
Open the record for dataset details and reuse information.
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.