Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
187
datasets available to search
ShareScore release 0.9.0
Dataset results
187 results for “Language Data”
language data
<p>language data</p>
Data from: Markovian language model of the DNA and its information content
This work proposes a Markovian memoryless model for the DNA that simplifies enormously the complexity of it. We encode nucleotide sequences into symbolic sequences, called words, from which we establish meaningful length of words and groups of words that share symbolic similarities. Interpreting a node to represent a group of similar words and edges to represent their functional connectivity allows us to construct a network of the grammatical rules governing the appearance of groups of words in the DNA. Our model allows us to predict the transition between groups of words in the DNA with unprecedented accuracy, and to easily calculate many informational quantities to better characterize the DNA. In addition, we reduce the DNA of known bacteria to a network of only tens of nodes, show how our model can be used to detect similar (or dissimilar) genes in different organisms, and which sequences of symbols are responsible for most of the information content of the DNA. Therefore, the DNA can indeed be treated as a language, a Markovian language, where a 'word' is an element of a group, and its grammar represents the rules behind the probability of transitions between any two groups.
Data and code from: Large language models can segment narrative events similarly to humans.
<p>Data and code supporting the paper: Large language models can segment narrative events similarly to humans.</p>
Data Supporting High-Level to Low-Level Requirements Coverage Reviewing with Large Language Models
Open the record for dataset details and reuse information.
Measurement Data for "LIRPA: Parallelized List Reductions for Language Agnostic Test Case Reducers"
Open the record for dataset details and reuse information.
Data to accompany dissertation: Geographic, Cultural, and Ecological Correlations with Indigenous Language Vitality in North America
<p>Text files:</p> <ul> <li>readme_general.txt contains a brief description of files included.</li> <li>readme_modeldata.txt contains a metadata description of model_data.csv.</li> <li>readme_languageslandNorthAmerica.txt contains a metadata description of Languages_land_NorthAmerica.csv.</li> <li>readme_languagerevitalizationdatabase.txt contains a metadata description of Language_revitalization_database.csv.</li> </ul> <p>CSV files:</p> <ul> <li>Languages_land_NorthAmerica.csv is a version of the Languages of Government-Recognized Native Land Areas in the Continental United States database. It includes data from the US Census 2017 TIGER/Line AIANNH shapefile with one row per Native land area and additional columns for associated information that was coded and calculated for this dissertation as discussed in Section 3.3.1.</li> <li>Language_revitalization_database.csv is the Language Revitalization Database. It contains the master language list used for this dissertation and columns created while coding data for the language revitalization variable, as discussed in Section 3.3.2.</li> <li>model_data.csv contains data for all variables used in the analysis and is the .csv file needed to run LanguageVitalityModels.R.</li> </ul> <p>R scripts:</p> <ul> <li>LanguageVitalityModels.R is the R script for the main part of the dissertation analysis.</li> </ul>
Excel file with data from the implementation of "Using a Digital Assistant for learning English language and Geography in Primary Education"
Open the record for dataset details and reuse information.
Automated electrosynthesis reaction mining with multimodal large language models - raw data and prompts
<p>Compilation of the prompts used, together with the raw response files from the tested MLMM models. </p>
Data for "No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models"
<p>Datasets to reproduce the experiments associated with the paper: https://doi.org/10.48550/arXiv.2307.06440</p> <p>The readme contains instructions for how to use them: https://github.com/JeanKaddour/NoTrainNoGain/blob/main/bert/README.md</p> <p>c4-subset-random.tar.bz2 is a subset of the C4 dataset (https://arxiv.org/abs/1910.10683), licensed under ODC-BY 1.0.</p>
Data for article entitled 'Prediction in SVO and SOV languages: Processing and typological considerations', published in Linguistics
<p>see the publication for the description of the data and analysis</p>
Data and code for, "Large language models design sequence-defined macromolecules via evolutionary optimization"
<div> <pre># Codes and data for "Large language models design sequence-defined macromolecules via evolutionary optimization"<br><br>Note this repository contains codes and data files for the manuscript. This is a snapshot of the repository, frozen at the time of submission.<br><br># Codes<br><br>## LLM codes<br>- `run_claude.py` - the routine for performing LLM-based rollouts; intended for command line execution using argparse<br>- `message_utils.py` - utilities for constructing and parsing messages for LLM I/O<br>- `model_utils.py` - lightweight utilities for retrieving formatted predictions from the RNN ensemble<br>- `target_defs.py` - defines the sequence, locations, and natural language descriptions of the target structures<br>- `ask_about_oracle.ipynb` - asks the LLM to speculate about the nature of the optimization task<br><br>## other algorithms<br>- `active_learning.ipynb` - use EI acquisition with RF surrogate to label new sequences; includes an unused tokenization scheme<br>- `evolutionary_algorithm.ipynb` - use DEAP library to perform evolutionary optimization<br>- `random_sampling.ipynb` - sample sequences randomly from all possible sequences<br><br>## postprocessing<br>- `process_aggregated_logs.py` - reads data from the raw log files and prepares them for visualization<br>- `process_sample_rollouts.py` - reads data from the raw log files and prepares individual rollouts<br><br>## visualization<br>- `figure1b.ipynb` - renders panel b of Fig. 1<br>- `figure1efg.ipynb` - renders the last row of Fig. 1 (panels e-g)<br>- `figure2.ipynb` - renders all of Fig. 2<br>- `figure_si.ipynb` - renders Figs. S1 and S2<br>- `figure_md_validation.ipynb` - renders Fig. S3<br><br># Data files<br><br>- `prompts/`<br> - `prompt-scientific-v4.4.yml` - the full text of the scientific prompt, to be read by `run_claude.py`<br> - `prompt-oracle-v4.4.yml` - the full text of the oracle prompt, to be read by `run_claude.py`<br>- `models/` - the TorchScript RNN models used to make predictions<br>- `data/`<br> - `embeddings` - calculated embeddings for a collection of sequences from our prior work<br> - `llm-logs` - the raw logs obtained from the Claude 3.5 Sonnet LLM (other algorithms made to look like the LLM logs after the fact)<br> - `llm-logs-opus` - the raw logs obtained from the Claude 3.0 Opus LLM (used in the first draft of the article, replaced by Claude 3.5 Sonnet) <br> - `all-rollouts-kltd.csv` - postprocessed logs for all the rollouts using the "top $k < d^*$" metric<br> - `all-rollouts-topkd.csv` - postprocessed logs for all the rollouts using the "mean $d$ for top $k$" metric<br> - `sample-rollout-membranes-x-3.csv` - postprocessed logs for a single rollout replica, `x` = each algorithm type<br> - `snapshots` - png snapshots of MD simulation results at different locations in the manifold</pre> </div>
Data from: What is in a general plan? using natural language processing to read 461 California city general plans
Open the record for dataset details and reuse information.
Data from: Cultural and climatic changes shape the evolutionary history of the Uralic languages
Open the record for dataset details and reuse information.
Data from: Markovian language model of the DNA and its information content
Open the record for dataset details and reuse information.
Data from: Social media enhances languages differentiation: a mathematical description
Open the record for dataset details and reuse information.
Data from: Natural language processing and recurrent network models for identifying genomic mutation-associated cancer treatment change from patient progress notes
Open the record for dataset details and reuse information.
Data from: Language structure is influenced by the number of speakers but seemingly not by the proportion of non-native speakers
Open the record for dataset details and reuse information.
Data from: Bringing the nonlinearity of the movement system to gestural theories of language use: multifractal structure of spoken English supports the compensation for coarticulation in human speech perception
Open the record for dataset details and reuse information.
Data from: Robust clustering of languages across Wikipedia growth
Open the record for dataset details and reuse information.
Data from: On the links between nature's values and language
Open the record for dataset details and reuse information.
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.