Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
296
datasets available to search
ShareScore release 0.9.0
Dataset results
296 results for “language models”
Data from: Markovian language model of the DNA and its information content
This work proposes a Markovian memoryless model for the DNA that simplifies enormously the complexity of it. We encode nucleotide sequences into symbolic sequences, called words, from which we establish meaningful length of words and groups of words that share symbolic similarities. Interpreting a node to represent a group of similar words and edges to represent their functional connectivity allows us to construct a network of the grammatical rules governing the appearance of groups of words in the DNA. Our model allows us to predict the transition between groups of words in the DNA with unprecedented accuracy, and to easily calculate many informational quantities to better characterize the DNA. In addition, we reduce the DNA of known bacteria to a network of only tens of nodes, show how our model can be used to detect similar (or dissimilar) genes in different organisms, and which sequences of symbols are responsible for most of the information content of the DNA. Therefore, the DNA can indeed be treated as a language, a Markovian language, where a 'word' is an element of a group, and its grammar represents the rules behind the probability of transitions between any two groups.
CodeEval: Pedagogy Based Benchmark Dataset for Evaluation Of Large Language Models Trained On Code
<p>Categories has been modified to 21 from 26.</p>
Data and code from: Large language models can segment narrative events similarly to humans.
<p>Data and code supporting the paper: Large language models can segment narrative events similarly to humans.</p>
GPTCloneBench: A comprehensive benchmark of semantic clones and cross-language clones using GPT-3 model and SemanticCloneBench
<p>This is the full dataset of GPTCloneBench (version 2)</p>
Data Supporting High-Level to Low-Level Requirements Coverage Reviewing with Large Language Models
Open the record for dataset details and reuse information.
CodeQual: A dataset for fine-tuning Large Language Models for code quality assessment task
Open the record for dataset details and reuse information.
Large Language Models and prostate cancer
Open the record for dataset details and reuse information.
Open Large Language Model Framework
Open the record for dataset details and reuse information.
Are Large Language Models Good Software Librarians?
Open the record for dataset details and reuse information.
Deepurify: a multi-modal deep language model to remove contamination from metagenome-assembled genomes
<p>The SIM2 testing set.</p>
Automated electrosynthesis reaction mining with multimodal large language models - raw data and prompts
<p>Compilation of the prompts used, together with the raw response files from the tested MLMM models. </p>
Improving antibody language models with native pairing
<p><strong>Motivation.</strong> Existing large language models designed to predict antibody structure and function have been trained exclusively with unpaired antibody sequences. This is a substantial drawback, as each antibody represents a unique pairing of heavy and light chains that both contribute to antigen recognition. The cost of generating large datasets of natively paired antibody sequences is orders of magnitude higher than the cost of unpaired sequences, and the paucity of available paired antibody sequence datasets precludes training a state-of-the-art language model using only paired training data. Here, we sought to determine whether and to what extent natively paired training data improves model performance.</p> <p><strong>Results.</strong> Using a unique and recently reported dataset of approximately 1.6 x 10<sup>6</sup> natively paired human antibody sequences, we trained two baseline antibody language model (BALM) variants: BALM-paired and BALM-unpaired. We quantify the superiority of BALM-paired over BALM-unpaired, and we show that BALM-paired's improved performance can be attributed at least in part to its ability to learn cross-chain features that span natively paired heavy and light chains. Additionally, we fine-tuned the general protein language model ESM-2 using these paired antibody sequences and report that the fine-tuned model, but not base ESM-2, demonstrates a similar understanding of cross-chain features.</p> <p><strong>Files.</strong> The following files are included in this repository:</p> <ul> <li><em><strong>BALM-paired.tar.gz</strong></em>: Model weights for the BALM-paired model.</li> <li><em><strong>BALM-shuffled.tar.gz</strong></em>: Model weights for the BALM-shuffled model.</li> <li><em><strong>BALM-unpaired.tar.gz</strong></em>: Model weights for the BALM-unpaired model.</li> <li><em><strong>ESM2-650M_paired-fine-tuned.tar.gz</strong></em>: Model weights for the 650M-parameter ESM-2 model after fine-tuning with natively paired antibody sequences.</li> <li><em><strong>jaffe-paired-dataset_airr-annotation.tar.gz</strong></em>: All natively paired antibody sequences from the <a href="https://www.nature.com/articles/s41586-022-05371-z">Jaffe dataset</a> were annotated with <a href="https://github.com/briney/abstar">abstar</a> and subsequently filtered to remove duplicates or unproductive sequences. The annotated sequences are provided in an <a href="https://docs.airr-community.org/en/stable/datarep/rearrangements.html">AIRR-compliant format</a>.</li> <li><em><strong>test-dataset_annotated.tar.gz</strong></em>: Two csv files, both with sequences annotated in an <a href="https://docs.airr-community.org/en/stable/datarep/rearrangements.html">AIRR-compliant format</a>. <em>lc-coherence_test-unique_annotated.csv</em> contains all sequences from the test dataset and <em>fig3-20kembeddings_annotated.csv</em> contains the 20k sequences from the test used for the Figure 2 UMAP embeddings. For both datasets, the sequences can be paired together based on their pair_id.</li> <li><em><strong>train-test-eval_paired.tar.gz</strong></em>: Datasets used to train, test, and evaluate the BALM-paired model. Compressed folder containing three files: <em>train.txt</em>,<em> test.txt</em>, and <em>eval.txt</em>. Each file has one input sequence per line. This dataset was also used to fine-tune the 650M-parameter ESM-2 variant.</li> <li><em><strong>train-test-eval_shuffled.tar.gz</strong></em>: Datasets used to train, test, and evaluate the BALM-shuffled model. Compressed folder containing three csv files, with two columns for the heavy and light chains.</li> <li><em><strong>train-test-eval_unpaired.tar.gz</strong></em>: Datasets used to train, test, and evaluate the BALM-unpaired model. Compressed folder containing three files: <em>train.txt</em>, <em>test.txt</em>, and <em>eval.txt</em>. Each file has one input sequence per line.</li> <li><em><strong>classification-datasets.tar.gz</strong></em>: Three classification datasets used to train classification models in Figure 5. The datasets are: <em>flu-0_cov-1.csv</em>, <em>hd-0_cov-1.csv</em>, and <em>hd-0_flu-1_cov-2.csv</em>. CoV antibody sequences were obtained from <a href="https://doi.org/10.1093/bioinformatics/btaa739">CoV-AbDab</a>, Flu antibody sequences were obtained from <a href="https://doi.org/10.1101/2023.09.11.557288">Wang et al.</a>, and healthy donor antibody sequences were obtained from <a href="https://doi.org/10.1016/j.celrep.2024.114307">Hurtado et al</a>.</li> </ul> <p><strong>Code:</strong> All code used for model training, testing, and figure generation is available under the MIT license on <a href="https://www.github.com/briney/BALM-paper">GitHub</a>. An archived version of the GitHub repository (from the time of manuscript publication) is included here as <em><strong>code-archive.zip</strong></em>.</p> <p> </p>
S-Cypher: A Temporal Query Language on the Temporal Property Graph Model
Open the record for dataset details and reuse information.
Evaluating Large Language Models' Proficiency in Answering Arabic GAT Exam Questions
Open the record for dataset details and reuse information.
An Architecture for Integrating Large Language Models into Metamodeling Platforms: The Example of MM-AR
Open the record for dataset details and reuse information.
The best fitting tree model for Japonic languages
<p>Data from Lee, S., & Hasegawa, T. (2011). Bayesian phylogenetic analysis supports an agricultural origin of Japonic languages. <em>Proceedings of the Royal Society of London B: Biological Sciences</em>, rspb20110518.</p>
Quantity doesn't buy quality syntax with neural language models
<p>This repository contains the 125 LSTM models analyzed in <a href="https://www.aclweb.org/anthology/D19-1592/">van Schijndel, Mueller, and Linzen (2019) "Quantity doesn't buy quality syntax with neural language models"</a>. Each archive contains 25 models trained on a specific number of training tokens. All models were trained to use the vocabulary in vocab.txt.</p> <p>The naming convention for each model is:<br> LSTM_[Hidden Units]_[Training Tokens]_[Training Partition]_[Random Seed]-d[Dropout Rate].pt</p> <p>Hidden Units: The number of hidden units per layer (there are two layers in each model) {100, 200, 400, 800, 1600}<br> Training Tokens: The number of tokens used to train each model {2m, 10m, 20m, 40m, 80m}<br> Training Partition: Five distinct training partitions were created for each amount of training data {a, b, c, d, e}<br> Random Seed: The random seed used to train each model*<br> Dropout Rate: All models used a dropout rate of 0.2</p> <p>*A scripting bug led to a random seed of 0 for all models trained on less than 40 million tokens. This does not substantively affect the analyses since each model is distinct in terms of the model configuration or training data, so we opted to not retrain the models with unique random seeds to save time and computational resources.</p>
Supplementary Material for Paper "Distilling Event Sequence Knowledge From Large Language Models"
<p>Supplementary Material for Paper:<br>Distilling Event Sequence Knowledge From Large Language Models<br>Somin Wadhwa, Oktie Hassanzadeh, Debarun Bhattacharjya, Ken Barker, and Jian Ni</p> <p>Appendix:<br>- <code>Appendix.pdf</code>: contains our prompts and a description of our human evaluation details.</p> <p>Data:<br>- <code>base_kg_v7.jsonl</code>: Our Wikidata-based Event Causal Knowledge Graph.</p> <p>Outputs:<br>- <code>sample_new_patterns_discovered.txt</code>: examples of observed new patters through application of sequential pattern mining algorithms, described in section 3.<br>- <code>precision_eval_sample.txt:</code> examples of output evaluated with a precision-evaluator model. <br>- <code>bsumm_output.txt</code>: sample outputs of identified influencing events through the application of binary summary markov model, described in section 5.2.</p> <p>Code:<br>- <code>src/generator.py</code>: ingests ICL prompts and generates requisite event sequences.<br>- <code>src/benchmarking.py</code>: ingests a _trained_ Flan-style seq2seq model to evaluate precision, and recall from the base KG.<br>- <code>src/utils.py</code>: utilities for generator and benchmarking.</p> <p>To cite:</p> <pre><code>@inproceedings{WadhwaHBBN24, author = {Somin Wadhwa and Oktie Hassanzadeh and Debarun Bhattacharjya and Ken Barker and Jian Ni}, title = {Distilling Event Sequence Knowledge From Large Language Models}, booktitle = {The Semantic Web - 23rd International Conference, {ISWC} 2024}, series = {Lecture Notes in Computer Science}, publisher = {Springer}, year = {2024}, }</code></pre>
HiPHD: Hierarchical Classification for Protein Remote Homology Detection using Graph Neural Networks and Language Models
Open the record for dataset details and reuse information.
Evaluating Inductive Reasoning Capabilities of Large Language Models With The One Dimensional Abstract Reasoning Corpus
Open the record for dataset details and reuse information.
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.