Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
187
datasets available to search
ShareScore release 0.9.0
Dataset results
187 results for “Language Data”
Data from: Natural language processing systems for pathology parsing in limited data environments with uncertainty estimation
Open the record for dataset details and reuse information.
A machine readable collection of lexical data on the Burmish languages
<p>This dataset includes lexical lists, mostly semantically normalized to WordNet, that brings together with near comprehensiveness the published data on Burmish languages. It was compiled by the ERC project 'Asia: Beyond Boundaries' in collaboration with the Center for Research in Computational Linguistics.</p> <p>The Bibtex references can be found at this Zenodo deposit--</p> <p>Hill, Nathan, List, Johann-Mattis, & Gong, Xun. (2020). Asia.bib: A bibtex bibliography for Asian Historical Linguistics [Data set]. Zenodo. http://doi.org/10.5281/zenodo.3759114</p>
Domain-specific Language for Data-driven Design Time Analyses and Result Mappings for Logic Programs - Data Set
<p>This data set contains additional information to the master thesis of Sebastian Hahner. This includes the prototypical implementation of the domain-specific language (DSL) as well as sample scenarios and mapping results.</p>
Data from: The appropriateness of language found in research consent form templates: a computational linguistic analysis
Background: To facilitate informed consent, consent forms should use language below the grade eight level. Research Ethics Boards (REBs) provide consent form templates to facilitate this goal. Templates with inappropriate language could promote consent forms that participants find difficult to understand. However, a linguistic analysis of templates is lacking. Methods: We reviewed the websites of 124 REBs for their templates. These included English language medical school REBs in Australia/New Zealand (n=23), Canada (n=14), South Africa (n=8), the United Kingdom (n=34), and a geographically-stratified sample from the United States (n=45). Template language was analyzed using Coh-Metrix linguistic software (v.3.0, Memphis, USA). We evaluated the proportion of REBs with five key linguistic outcomes at or below grade eight. Additionally, we compared quantitative readability to the REBs' own readability standards. To determine if the template's country of origin or the presence of a local REB readability standard influenced the linguistic variables, we used a MANOVA model. Results: Of the REBs who provided templates, 0/94 (0%, 95% CI=0-3.9%) provided templates with all linguistic variables at or below the grade eight level. Relaxing the standard to a grade 12 level did not increase this proportion. Further, only 2/22 (9.1%, 95% CI= 2.5-27.8) REBs met their own readability standard. The country of origin (DF= 20, 177.5, F=1.97, p=0.01), but not the presence of an REB-specific standard (DF=5, 84, F=0.73, p=0.60), influenced the linguistic variables. Conclusions: Inappropriate language in templates is an international problem. Templates use words that are long, abstract, and unfamiliar. This could undermine the validity of participant informed consent. REBs should set a policy of screening templates with linguistic software.
Data from: Oceanic barriers promote language diversification in the Japanese Islands
Good barriers make good languages. Scholars have long speculated that geographical barriers impede linguistic contact between speech communities, and promote language diversification in a manner similar to the process of allopatric speciation. This hypothesis, however, has seldom been tested systematically and quantitatively. Here we adopt methods from evolutionary biology and attempt to quantify the influence of oceanic barriers on the degree of lexical diversity in the Japanese Islands. Measuring the degree of beta diversity from basic vocabularies, we find that geographical proximity and, more importantly, isolation by surrounding ocean, independently explains a significant proportion of lexical variation across Japonic languages. Further analyses indicate that our results are neither a byproduct of using a distance matrix derived from a Bayesian language phylogeny nor an epiphenomenon of accelerated evolutionary rates in languages spoken by small communities. Moreover, we find that the effect of oceanic barriers is reproducible with the Ainu languages, indicating that our analytic approach as well as the results can be generalized beyond Japonic language family. The findings we report here are the first quantitative evidence that physical barriers formed by ocean can influence language diversification, and points to an intriguing common mechanism between linguistic and biological evolution.
Data and code for "Large language models identify causal genes in complex trait GWAS"
<p><span>This file contains the data and scripts for the preprint "Large language models identify causal genes in complex trait GWAS"</span></p>
Supplementary Data for "Exploring the Integration of Large Language Models in Industrial Test Maintenance Processes"
<p>This package contains supplementary data not directly included in the paper, including per-commit results for each prototype and the prompts used in the proof-of-concept implementations.</p>
Connecting the Dots Variables of Literary History and Emotions in German-language Poetry - Code and Data
<p><strong>Usage</strong></p><p>Experiments can be replicated by running run.py. Please take a look at run.py, since there are some choices to be made within the code. Since pymc4 and gpu usage is under heavy development at the time, we advise to stick to the excact package versions from requirements.txt.</p><p><strong>Content</strong></p><ul><li>data<ul><li>input.tsv - tab-separated table containing corpus metadata (including emotions) </li></ul></li><li>factors<ul><li>model.py - declaration the HGLM in pymc3</li><li>preprocessing.py - code to transform the data in input.tsv to fit the HGLM</li></ul></li><li>plots</li><li>requirements.txt - required python version and packages</li><li>run.py - wrapper to start the experiments (priors and targeted emotions need to be decalered in the code)</li></ul>
Data belonging to the thesis intitled "Prosody in Language Contact. French and Vietnamese"
<p>This dataset belongs to our doctoral thesis intitled "Prosody in Language Contact. French and Vietnamese". The dataset contains recordings of oral data as well as tables of written data, transcriptions and annotations. It also contains metadata about oral and written data. Detailed information about this data and how to understand it can be taken from our thesis.</p> <p>In the thesis, we are dealing with prosodic language contact between the languages Vietnamese and French. Our research is devoted to different language contact situations as well as to both directions of language contact. The starting point of our experimental research is the observation of French loanwords in Vietnamese. In this context, we examine prosodic adaptation patterns that speakers have undertaken to adapt the loanwords to Vietnamese. The focus is on the repair of structures that are illicit in Vietnamese: Consonants in certain positions in the syllable are replaced or deleted, consonant clusters are dissolved by epenthesis or by deletion of one of the two consonants, syllable boundaries are shifted, consonants or consonant slots in certain syllabic structures are doubled, and tones are assigned to syllables according to certain patterns.</p> <p>The aim of our experimental investigations then is to find out whether similar or different patterns occur in an instantaneous situation of language contact. Monolingual speakers of Vietnamese are exposed to French stimuli and asked to reproduce them in three different conditions. We have additionally conducted the same experiment with learners of French whose first language is Vietnamese. The experimental data show many similar patterns to the loanword data. However, the data from monolingual speakers in particular display much more variability. Finally, we reverse the direction of language contact: Native speakers of French are asked to reproduce Vietnamese stimuli. In this case, the question is whether certain patterns can be reversed in the opposite direction, which can partly be observed.</p> <p>With the help of the present work, we can gain a deep understanding of systematicity and variability in borrowing and second language acquisition. We conclude that from a phonological perspective, there are many similarities between the two fields of prosodic language contact, and we suggest, for the phenomena under consideration, to understand some aspects in their complexity in a gradual rather than a categorical way.</p>
Source concepts for language data
<p>An Excel spreadsheet containing the data for the paper "Mouths, tongues, and ears: Source concepts for 'language' across Africa".</p>
Text simplification in second language: process and product data
<p>This folder contains data on text simplification collected from an experimental study with second-language university students. We adopted a pre-test and post-test design, and randomly divided participants into experimental and control group. In the pre-test, participants were given an extract of a corporate report dealing with sustainability and were asked to revise it to make it easier to read for a lay customer. Subsequently, they took part in training. The experimental group received training on both plain language and sustainability, while the control group received training exclusively on the topic of sustainability. In the post-test session (2-3 days after the pre-test), all participants were assigned a second extract of a corporate report dealing with sustainability, and were asked again to make it easier to read for a lay customer by applying what they had learned from their respective training. This design allowed us to examine the impact of plain language training on text simplification (revision) tasks. The texts were in English while the participants were native speakers of other languages (mainly Dutch), so the text simplification took place in their second language.</p> <p>This project (PLanTra) has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 888918.</p> <p>Please see "readme" file for additional information.</p>
Processed data for the paper "Evaluating natural language processing models with generalization metrics that do not need access to any training or testing data"
<p>This is the data used to reproduce the results from "Evaluating natural language processing models with generalization metrics that do not need access to any training or testing data".</p>
Scatter plots for the paper "Evaluating natural language processing models with generalization metrics that do not need access to any training or testing data"
<p>This file contains the test-score-vs-metric plots generated by the paper "Evaluating natural language processing models with generalization metrics that do not need access to any training or testing data".</p>
Generalization metrics for the paper "Evaluating natural language processing models with generalization metrics that do not need access to any training or testing data"
<p>This file contains all the generalization metrics that can be used to reproduce the results of "Evaluating natural language processing models with generalization metrics that do not need access to any training or testing data".</p>
Rank correlation results for the paper "Evaluating natural language processing models with generalization metrics that do not need access to any training or testing data"
<p>This file contains the rank correlation results from the paper "Evaluating natural language processing models with generalization metrics that do not need access to any training or testing data".</p>
Open Language Archive Community (OLAC) Nightly Data Dump (XML) from 12 September 2015
<p>Open Language Archive Community (OLAC) Nightly Data Dump (XML) from 12 September 2015</p>
WildCLIP: Scene and animal attribute retrieval from camera trap data with domain-adapted vision-language models
<p>#############</p> <h1>WildCLIP: Scene and animal attribute retrieval from camera trap data with domain-adapted vision-language models</h1> <p>#############</p> <p>Authors: Valentin Gabeff, Marc Russwurm, Devis Tuia & Alexander Mathis</p> <p>Affiliation: EPFL</p> <p>Date: January, 2024</p> <p>Link to the article: <a href="https://link.springer.com/article/10.1007/s11263-024-02026-6">https://link.springer.com/article/10.1007/s11263-024-02026-6</a></p> <p>--------------------------------</p> <p>WildCLIP is a fine-tuned CLIP model that allows to retrieve camera-trap events with natural language from the Snapshot Serengeti dataset. This project intends to demonstrate how vision-language models may assist the annotation process of camera-trap datasets.</p> <p>Here we provide the processed Snapshot Serengeti data used to train and evaluate WildCLIP, along with two versions of WildCLIP (model weights).</p> <p>Details on how to run these models can be found in the project <a href="https://github.com/amathislab/wildclip">github repository</a>.</p> <h2>Provided data (images and attribute annotations): </h2> <p>The data consists of 380 x 380 image crops corresponding to the MegaDetector output of Snapshot Serengeti with a confidence threshold above 0.7. We considered only camera trap images containing single individuals.</p> <p>A description of the original data can be found on LILA <a href="https://lila.science/datasets/snapshot-serengeti">here</a>, released under the <a href="https://cdla.dev/permissive-1-0/" rel="nofollow">Community Data License Agreement (permissive variant)</a>.</p> <p>We warmly thank the authors of LILA for making the MegaDetector outputs publicly available, as well as for structuring the dataset and facilitating its access.</p> <h2>Adapted CLIP model (model weights): </h2> <p>WildCLIP models provided:</p> <ul> <li><strong>[New] WildCLIP_vitb16_t1.pth: </strong>CLIP model with the ViT-B/16 visual backbone trained on data with captions following template 1. Trained on both base and novel vocabulary (see paper for details).</li> <li><strong>[New] WildCLIP_vitb16_t1_lwf.pth: </strong>CLIP model with the ViT-B/16 visual backbone trained on data with captions following template 1, and with the additional VR-LwF loss. Trained on both base and novel vocabulary (see paper for details).</li> <li><strong>WildCLIP_vitb16_t1_base.pth:</strong> CLIP model with the ViT-B/16 visual backbone trained on data with captions following template 1. Model used for evaluation and trained on base vocabulary only. (previously named <em>WildCLIP_vitb16_t1.pth</em>)</li> <li><strong>WildCLIP_vitb16_t1t7_lwf_base.pth</strong>: CLIP model with the ViT-B/16 visual backbone trained on data with captions following templates 1 to 7, and with the additional VR-LwF loss. Model used for evaluation and trained on base vocabulary only. (previously named <em>WildCLIP_vitb16_t1t7_lwf.pth</em>)</li> </ul> <p>We also provide the CSV files containing the train / val / test splits. The train / test splits follow camera split from LILA (https://lila.science/datasets/snapshot-serengeti). The validation split is custom, and also at the camera level.</p> <ul> <li><strong>train_dataset_crops_single_animal_template_captions_T1T7_ID.csv</strong>: Train set with captions from templates 1 through 7 (column "all captions") or template 1 only (column "template 1")</li> <li><strong>val_dataset_crops_single_animal_template_captions_T1T7_ID.csv</strong>: Validation set with captions from templates 1 through 7 (column "all captions") or template 1 only (column "template 1")</li> <li><strong>test_dataset_crops_single_animal_template_captions_T1T8T10.csv</strong>: Test set with captions from templates 1, 8, 9 and 10 (columns "all captions")</li> </ul> <p>Details on how the models were trained can be found in the associated <a href="https://link.springer.com/article/10.1007/s11263-024-02026-6" target="_blank" rel="noopener">publication</a>.</p> <h2>References: </h2> <p>If you find our code, or weights, please cite:</p> <pre>@article{gabeff2024wildclip, title={WildCLIP: Scene and animal attribute retrieval from camera trap data with domain-adapted vision-language models}, author={Gabeff, Valentin and Ru{\ss}wurm, Marc and Tuia, Devis and Mathis, Alexander}, journal={International Journal of Computer Vision}, pages={1--17}, year={2024}, publisher={Springer} }</pre> <p>If you use the adapted Snapshot Serengeti data please also cite their article:</p> <pre>@article{swanson2015snapshot, title={Snapshot Serengeti, high-frequency annotated camera trap images of 40 mammalian species in an African savanna}, author={Swanson, Alexandra and Kosmala, Margaret and Lintott, Chris and Simpson, Robert and Smith, Arfon and Packer, Craig}, journal={Scientific data}, volume={2}, number={1}, pages={1--14}, year={2015}, publisher={Nature Publishing Group} }</pre>
Jargon: A Suite of Language Models and Evaluation Tasks for French Specialized Domains: Data
<p>This repository contains the data for the following paper:</p> <p><span>Vincent Segonne, Aidan Mannion, Laura Cristina Alonzo Canul, Alexandre Daniel Audibert, Xingyu Liu, Cécile Macaire, Adrien Pupier, Yongxin Zhou, Mathilde Aguiar, Felix E. Herron, Magali Norré, Massih R Amini, Pierrette Bouillon, Iris Eshkol-Taravella, Emmanuelle Esperança-Rodier, Thomas François, Lorraine Goeuriot, Jérôme Goulian, Mathieu Lafourcade, et al.. 2024. <a href="https://aclanthology.org/2024.lrec-main.827">Jargon: A Suite of Language Models and Evaluation Tasks for French Specialized Domains</a>. In <em>Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)</em>, pages 9463–9476, Torino, Italia. ELRA and ICCL.</span></p> <p>1) pretraining data for the Jargon specialized language models</p> <p>2) ECTHR_FR dataset for text classification in the French legal domain</p> <p> </p>
Data for: Unraveling the influence of essential climatic factors on the number of tones through an extensive database of languages in China
Open the record for dataset details and reuse information.
Replication data for the paper "Leveraging Large Language Models for Comprehensive Psychological Analysis: Insights from Four Theoretical Frameworks"
<p>This is a replication data for the paper titled "Leveraging Large Language Models for Comprehensive Psychological Analysis: Insights from Four Theoretical Frameworks" submitted for a blind review.</p> <p>Abstract</p> <p>The rapid advancement of generative Artificial Intelligence (AI) has significantly transformed various research domains. This paper introduces a novel, fully automated methodology for applying Large Language Models (LLMs) to psychological text analysis. The approach includes prompt design for zero-shot and few-shot learning, model internal consistency analysis, autonomous machine evaluation, and additional human validation. Applied to four psychological theories—Self-Determination Theory, the Big Five Personality Traits, Psychological Well-being, and Cognitive Behavioral Therapy—this methodology is tested on a dataset of 25,780 emails written by a senior executive (called Person X) over 16 years. The analysis involves extracting psychological characteristics from the emails and regressing these characteristics against personal, professional, and environmental factors. The results demonstrate that the methodology provides unique insights into the examined psychological theories, offering a detailed understanding of how various factors influence psychological states and traits over time. This research highlights the potential of LLMs in capturing and analyzing complex psychological patterns in large text corpora, contributing a robust framework for future studies and practical applications in psychological assessment and intervention. The findings underscore the transformative impact of generative AI in psychological research, opening new avenues for understanding human behavior through advanced language models.</p> <p>The zipped file contains five csv files:</p> <ol> <li>Email_classification-csv: LLM (GPT-3.5 Turbo) classification of 25,780 emails for four psychological theories: SDT, Big Five, PWB and CBT.</li> <li>SDT_regression_data.csv</li> <li>Big_Five_regression_data.csv</li> <li>PWB_regression_data.csv</li> <li>CBT_regression_data.csv</li> </ol> <p>For 2-5 files the dependent variable is monthy percentage share of emails the were assigned a given value for categories of one of the four psychological theories analyzed. </p> <p>Linear regression model has been applied, where dependent variable is the percentage of emails in a specified category that assigned a specific value in this category. For example in Big Five Traits Model, for the Openness category, for each month we calculated percentage of emails that exhibit <em>High</em> or <em>Low</em> openness, or <em>None</em> if the content of the email does not provide enough information to assess whether the specific need is relevant. Two dependent variables were created: <em>Openness-high</em> and <em>Openness-low</em> and regressed on all independent variables. Regressions were not run for the <em>None</em> values.</p> <p>Descriptions of independent variables:</p> <p>- <em>income_index</em>: Person X salary income and consulting fees in a given month, normalized to [0,1].</p> <p>- <em>card_spending</em>: Person X credit card expenditures in a given month, normalized to [0,1].</p> <p>- <em>abroad_far</em>: dummy variable set to 1 for months when Person X worked in Central Asia</p> <p>- <em>abroad_near</em>: dummy variable set to 1 when Person X worked in other EU country</p> <p>- <em>death_1_war</em>: variable set to 1 in a month when Person X’ farther in law passed away. In the same month Russia invaded Ukraine. The variable was set to .75 in the following month, and to .5 in the month after that.</p> <p>- <em>death_2</em>: variable set to 1 in a month when Person X’ mother passed away. The variable was set to .75 in the following month, and to .5 in the month after that.</p> <p>- <em>court_case</em>: dummy variable set to 1 for months with the emotionally engaging inheritance court case involving other family members.</p> <p>- <em>BIG4_partner</em>: dummy variable set to 1 for months when Person X worked as a partner in BIG4 accounting firm, which resulted in adopting a professional activity sharply different from the usual Person X habits.</p> <p>- <em>AI_company</em>: dummy variable set to 1 for months when Person X worked as C-level executive at a company specializing in artificial intelligence.</p> <p>- <em>elections</em>: dummy variable set to 1 for months when Person X unsuccessfully run in parliamentary elections</p> <p>- <em>covid_lockdown</em>: dummy variable set to 1 for month where Polish government imposed tough measures during two covid lockdowns.</p> <p>- <em>no_receive</em>: number of different email recipients each month, normalized to [0,1].</p> <p>- <em>avg_length</em>: average number of words in emails sent each month, normalized to [0,1].</p> <p> While the email data was collected for January 2008 – March 2014 period, financial data was available from October 2009. There were some months where no emails with more than 10 words were sent, yielding 166 monthly observations used for regressions, before removing outliers.</p> <p> Independent variables were tested for multicollinearity, outlier months were removed, regressions were estimated with robust standard errors, and a range of standard tests were conducted for normality and autocorrelation of residuals, confirming good statistical properties of estimated models.</p> <p>Due to privacy concerns, the email texts cannot be publicly shared. However, the classifications of psychological categories derived from the email texts, along with all other relevant data, are made publicly available in this open access repository, with the consent of email author.</p> <p> </p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.