Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
167
datasets available to search
ShareScore release 0.9.0
Dataset results
167 results for “large language models”
[Supplementary material] AI-Driven Fairness Testing of Large Language Models: A Preliminary Study
<div>This is the supplementary material of the paper entitled <em>AI-Driven Fairness Testing of Large Language Models: A Preliminary Study</em>.</div> <div> </div> <div>The material is organized into two main folders:</div> <div> <ul> <li><strong>evaluation_data/</strong>: This folder contains the results of the fairness evaluations performed on three different language models: Gemma, Llama3, and Mistral. Each subfolder corresponds to a specific model and includes detailed <em>.csv</em> files documenting evaluation results across the 9 metamorphic relations (MRs) evaluated. Each <em>.csv</em> file contains the following columns: <ul> <li><em>test_id</em>: ID of the test.</li> <li><em>role</em>: Role, if applicable, involved in the prompts associated with the test.</li> <li><em>bias_type</em>: Type of bias being studied with the test.</li> <li><em>prompt_1</em>: Source test case executed on the model under test.</li> <li><em>response_1</em>: Response of the model to the source test case.</li> <li><em>prompt_2</em>: Follow-up test case executed on the model under test.</li> <li><em>response_2</em>: Response of the model to the follow-up test case.</li> <li><em>verdict</em>: Classification made by the judge model, which can take the following values: <ul> <li>'BIASED': If bias is detected.</li> <li>'UNBIASED': If no bias is detected.</li> <li>'INVALID': If the model under test failed to respond to either of the test cases (source or follow-up).</li> </ul> </li> <li><em>severity</em>: Classification of the bias severity made by the judge model, which can take the following values: <ul> <li>'LOW', 'MODERATE', or 'HIGH' (if the test is biased).</li> <li>Assigns 'N/A' if the test is not biased.</li> </ul> </li> <li><em>generation_explanation</em>: Explanation provided by the model generator, detailing how the base prompts were constructed.</li> <li><em>evaluation_explanation</em>: Explanation provided by the judge model, detailing the rationale behind the evaluation and justifying the assigned <em>verdict </em>for the test.</li> <li><em>manual_revision</em>: This field was completed based on the consensus of two authors to validate the <em>verdict</em>. It can take one of the following values: <ul> <li>'TP': The test was classified as biased, and it is indeed biased.</li> <li>'FP': The test was classified as biased, but it is not biased. </li> <li>'TN': The test was classified as unbiased, and it is indeed unbiased.</li> <li>'FN': The test was classified as unbiased, but it is actually biased.</li> <li>'INVALID': The model under test failed to respond to at least one of the prompts.</li> </ul> </li> </ul> </li> <li><strong>prompts/</strong>: This folder provides example prompts used during the generation and evaluation: <ul> <li><em>generation.txt</em>: Includes the prompt tied to the relation <em>MR1: Comparison - Single attribute</em>.</li> <li><em>evaluation.txt</em>: Includes the prompt used to evaluate <em>comparison</em> MRs, specifically for those involving demographic attributes.</li> </ul> </li> </ul> </div>
Artifacts for paper "PATCH: Empowering Large Language Model with Programmer-Intent Guidance and Collaborative-Behavior Simulation for Automatic Bug Fixing" submitted to TOSEM
<p>The project includes the data and code used in the submitted TOSEM paper titled "PATCH: Empowering Large Language Model with Programmer-Intent Guidance and Collaborative-Behavior Simulation for Automatic Bug Fixing"</p>
Evaluating Large Language Models in Summarizing Developer Chat Conversations: A Linguistic Perspective
<p>This is a replication package that includes:</p> <ul> <li>GoldenSet: contains the best summaries by participants for each conversation and the corresponding LLM generated summaries)</li> <li>LinguisticAnalysis_HumanGenerated: linguistic analysis such as speech tags, entities, etc. for summaries created by the Mturk participants (golden set)</li> <li>LinguisticAnalysis_LLMGenerated: linguistic analysis such as speech tags, entities etc. for summaries generated by the large language models</li> </ul> <p> </p>
A Systematic Evaluation of Large Language Models of Code
<p>These are datasets for the paper:</p> <p>"A Systematic Evaluation of Large Language Models of Code"</p> <p><a href="https://arxiv.org/pdf/2202.13169.pdf">https://arxiv.org/pdf/2202.13169.pdf</a></p> <p>The code is available at: <a href="https://github.com/VHellendoorn/Code-LMs">https://github.com/VHellendoorn/Code-LMs</a></p> <p> </p> <p>The file "<a href="https://zenodo.org/record/6338015/files/unseen_test_sets.tar.gz">unseen_test_sets.tar.gz</a>" contains test sets of ~100 files in each of 12 programming languages.</p> <p>These files are not included in The Pile, and thus models such as GPT-Neo, GPT-J, GPT-NeoX were not trained on them.</p> <p>In the paper, we use these test sets to compare a variety of language models of code including OpenAI's Codex, GPT-J, GPT-Neo, GPT-NeoX-20B, and CodeParrot and our PolyCoder model.</p> <p> </p> <p>The file "<a href="https://zenodo.org/record/6341643/files/index.zip?download=1">index.zip</a>" includes an index of the <strong>training set</strong> file paths and commit SHAs.</p> <p> </p> <p>The other files, such as "<a href="https://zenodo.org/record/6344914/files/2-7B-150K.tar">2-7B-150K.tar</a>", are trained model checkpoints, as explained at <a href="https://github.com/VHellendoorn/Code-LMs">https://github.com/VHellendoorn/Code-LMs</a> .</p>
Dataset related to: An Empirical Study on Low Code Programming using Traditional vs Large Language Model Support
<p>This repository contains data and prompts related to our research. The files included are:</p> <p><strong>prompt.py</strong>: This script contains the original prompts used in our study.</p> <p><strong>LLM_lowcode.mx20</strong>: This file includes posts and annotation data related to LLM-based low-code platforms.</p> <p><strong>Traditional_lowcode.mx20</strong>: This file includes posts and annotation data related to traditional low-code platforms.</p> <p>The .mx20 files can be opened using the MAXQDA software, which can be downloaded from the official website. MAXQDA offers a 14-day free trial.</p>
Exploring The Potential of GPT-3-based Large Language Model For Melody Generation
<p>Here, we provide the dataset used, all generated melodies and melodies used to conduct subjective listening test.</p>
Risk-of-bias v.2 assessment with large language models
<p>See https://bitbucket.org/aimedtech/fewshot_rob for more information.</p>
Analyzing the Dependability of Large Language Models for Code Clone Generation.
<div> <p>data.zip: <br><br>This dataset includes a collection of ten LeetCode programming problems used in the study "Analyzing the Dependability of Large Language Models for Code Clone Generation". At the top level, you will find a CSV file containing all the initial LeetCode data. Each subdirectory at this level represents a specific LeetCode problem. Within these subdirectories, you will find the original solutions, their behaviors, the input corpus, as well as folders dedicated to various temperatures, models, and code cloning tasks. Additionally, within the "repeated" folder, you will find the original LLM-generated snippets, the preprocessed snippets with the snippet behavior, and the results.</p> <p>characterizing_code_clones_project.zip: </p> <p>This zipped directory encompasses the core scripts and results used in the "Characterizing Code Clones of LLMs" research. The common folder was used to run the whole pipeline, the various parts of the pipeline are each in a folder as well as the various data analysis scripts! </p> <p> </p> </div>
Analyzing the Dependability of Large Language Models for Code Clone Generation
<p>data.zip: <br><br>This dataset includes a collection of ten LeetCode programming problems used in the study "Analyzing the Dependability of Large Language Models for Code Clone Generation". At the top level, you will find a CSV file containing all the initial LeetCode data. Each subdirectory at this level represents a specific LeetCode problem. Within these subdirectories, you will find the original solutions, their behaviors, the input corpus, as well as folders dedicated to various temperatures, models, and code cloning tasks. Additionally, within the "repeated" folder, you will find the original LLM-generated snippets, the preprocessed snippets with the snippet behavior, and the results.</p> <p>characterizing_code_clones_project.zip: </p> <p>This zipped directory encompasses the core scripts and results used in the "Characterizing Code Clones of LLMs" research. The common folder was used to run the whole pipeline, the various parts of the pipeline are each in a folder as well as the various data analysis scripts! </p>
Replication data for the paper "Leveraging Large Language Models for Comprehensive Psychological Analysis: Insights from Four Theoretical Frameworks"
<p>This is a replication data for the paper titled "Leveraging Large Language Models for Comprehensive Psychological Analysis: Insights from Four Theoretical Frameworks" submitted for a blind review.</p> <p>Abstract</p> <p>The rapid advancement of generative Artificial Intelligence (AI) has significantly transformed various research domains. This paper introduces a novel, fully automated methodology for applying Large Language Models (LLMs) to psychological text analysis. The approach includes prompt design for zero-shot and few-shot learning, model internal consistency analysis, autonomous machine evaluation, and additional human validation. Applied to four psychological theories—Self-Determination Theory, the Big Five Personality Traits, Psychological Well-being, and Cognitive Behavioral Therapy—this methodology is tested on a dataset of 25,780 emails written by a senior executive (called Person X) over 16 years. The analysis involves extracting psychological characteristics from the emails and regressing these characteristics against personal, professional, and environmental factors. The results demonstrate that the methodology provides unique insights into the examined psychological theories, offering a detailed understanding of how various factors influence psychological states and traits over time. This research highlights the potential of LLMs in capturing and analyzing complex psychological patterns in large text corpora, contributing a robust framework for future studies and practical applications in psychological assessment and intervention. The findings underscore the transformative impact of generative AI in psychological research, opening new avenues for understanding human behavior through advanced language models.</p> <p>The zipped file contains five csv files:</p> <ol> <li>Email_classification-csv: LLM (GPT-3.5 Turbo) classification of 25,780 emails for four psychological theories: SDT, Big Five, PWB and CBT.</li> <li>SDT_regression_data.csv</li> <li>Big_Five_regression_data.csv</li> <li>PWB_regression_data.csv</li> <li>CBT_regression_data.csv</li> </ol> <p>For 2-5 files the dependent variable is monthy percentage share of emails the were assigned a given value for categories of one of the four psychological theories analyzed. </p> <p>Linear regression model has been applied, where dependent variable is the percentage of emails in a specified category that assigned a specific value in this category. For example in Big Five Traits Model, for the Openness category, for each month we calculated percentage of emails that exhibit <em>High</em> or <em>Low</em> openness, or <em>None</em> if the content of the email does not provide enough information to assess whether the specific need is relevant. Two dependent variables were created: <em>Openness-high</em> and <em>Openness-low</em> and regressed on all independent variables. Regressions were not run for the <em>None</em> values.</p> <p>Descriptions of independent variables:</p> <p>- <em>income_index</em>: Person X salary income and consulting fees in a given month, normalized to [0,1].</p> <p>- <em>card_spending</em>: Person X credit card expenditures in a given month, normalized to [0,1].</p> <p>- <em>abroad_far</em>: dummy variable set to 1 for months when Person X worked in Central Asia</p> <p>- <em>abroad_near</em>: dummy variable set to 1 when Person X worked in other EU country</p> <p>- <em>death_1_war</em>: variable set to 1 in a month when Person X’ farther in law passed away. In the same month Russia invaded Ukraine. The variable was set to .75 in the following month, and to .5 in the month after that.</p> <p>- <em>death_2</em>: variable set to 1 in a month when Person X’ mother passed away. The variable was set to .75 in the following month, and to .5 in the month after that.</p> <p>- <em>court_case</em>: dummy variable set to 1 for months with the emotionally engaging inheritance court case involving other family members.</p> <p>- <em>BIG4_partner</em>: dummy variable set to 1 for months when Person X worked as a partner in BIG4 accounting firm, which resulted in adopting a professional activity sharply different from the usual Person X habits.</p> <p>- <em>AI_company</em>: dummy variable set to 1 for months when Person X worked as C-level executive at a company specializing in artificial intelligence.</p> <p>- <em>elections</em>: dummy variable set to 1 for months when Person X unsuccessfully run in parliamentary elections</p> <p>- <em>covid_lockdown</em>: dummy variable set to 1 for month where Polish government imposed tough measures during two covid lockdowns.</p> <p>- <em>no_receive</em>: number of different email recipients each month, normalized to [0,1].</p> <p>- <em>avg_length</em>: average number of words in emails sent each month, normalized to [0,1].</p> <p> While the email data was collected for January 2008 – March 2014 period, financial data was available from October 2009. There were some months where no emails with more than 10 words were sent, yielding 166 monthly observations used for regressions, before removing outliers.</p> <p> Independent variables were tested for multicollinearity, outlier months were removed, regressions were estimated with robust standard errors, and a range of standard tests were conducted for normality and autocorrelation of residuals, confirming good statistical properties of estimated models.</p> <p>Due to privacy concerns, the email texts cannot be publicly shared. However, the classifications of psychological categories derived from the email texts, along with all other relevant data, are made publicly available in this open access repository, with the consent of email author.</p> <p> </p>
Four psychometrically validated datasets for benchmarking large language models, based on the TIMSS 2008 and 2011 released items.
<p>Four datasets validated according to psychometric principles that can be used to benchmark large language models in terms of achievements in advanced school math, advanced school physics, 8th grade math and 8th grade science.</p> <p>These four datasets are derived from items released by Trends in International Mathematics and Science Study Advanced 2008 and Trends in International Mathematics and Science Study 2011. See <a href="https://nces.ed.gov/timss/released-questions.asp">link</a>.</p> <p>For more information, see our paper <a href="https://arxiv.org/abs/2404.01799">PATCH! Psychometrics-AssisTed benCHmarking of Large Language Models: A Case Study of Mathematics Proficiency</a>.</p>
Replication materials for: Are Large Language Models a Threat to Digital Public Goods? Evidence from Activity on Stack Overflow
<p>These are the replication materials for the paper:<br>Are Large Language Models a Threat to Digital Public Goods? Evidence from Activity on Stack Overflow<br>By: Maria del Rio-Chanona, Nadzeya Laurentsyeva, and Johannes Wachs. <br>Preprint: https://arxiv.org/abs/2307.07367<br>Under Revision for PNAS Nexus</p> <p> </p>
On Inter-dataset Code Duplication and Data Leakage in Large Language Models
<p>This dataset encompasses the sparse graph referenced in the publication titled "On Inter-dataset Code Duplication and Data Leakage in Large Language Models."</p> <p>This resource is a snapshot of the original <a href="https://github.com/Antolin1/code-inter-dataset-duplication">repository</a>, and the graph is preserved in the <em>interduplication.db</em> database. The schema of this database is easily understandable and is available in the original repository. Each code snippet is identified by a unique identifier (id_within_dataset) that corresponds to its identification within the dataset from which it was extracted. The complete datasets are stored in .jsonl files within their respective folders (e.g., python-150/data.jsonl, codetrans/data.jsonl, etc.).</p> <p> </p> <p> </p> <p> </p>
Examining the Influence of Political Bias on Large Language Model Performance in Stance Classification
<p>Code and dataset for paper "Examining the Influence of Political Bias on Large Language Model Performance in Stance Classification". ICWSM 2025</p> <p>Preprint: https://arxiv.org/abs/2407.17688</p> <p>Citation: </p> <p>@misc{ng2024examininginfluencepoliticalbias,<br> title={Examining the Influence of Political Bias on Large Language Model Performance in Stance Classification}, <br> author={Lynnette Hui Xian Ng and Iain Cruickshank and Roy Ka-Wei Lee},<br> year={2024},<br> eprint={2407.17688},<br> archivePrefix={arXiv},<br> primaryClass={cs.CL},<br> url={https://arxiv.org/abs/2407.17688}, <br>}</p>
Large Language Model
<p>One of the large language models trained in this paper: https://arxiv.org/abs/1810.10045</p>
Understanding the Rare Inflammatory Disease Using Large Language Models and Social Media Data
Open the record for dataset details and reuse information.
Topologies, Checkpoints, and Configurations for the paper "GVI-RL: Graph-Invariant RL for Attack Paths Discovery using Vulnerabilities Embedded with Large Language Models"
<p>This repository consists of the <strong>files</strong> related to the <strong>paper</strong> "GVI-RL: Graph-Invariant RL for Attack Paths Discovery using Vulnerabilities Embedded with Large Language Models". In particular, this repository contains tensorboard logs, topologies, checkpoints, seeds, and results to ensure reproducibility of the results of the paper.</p> <p>The results included are related to the training/validation and hyper-parameters optimization of the outcome multi-label classifier, the GVI-RL agent, and the world model.<br>The data folder contains also the topologies used in the study, the vulnerabilities data used to generate them and the dataset for multi-label classification.</p> <p>The README.md file describes the folders' structure.</p>
Artifacts for "On Hardware Security Bug Code Fixes By Querying Large Language Models""
<p>This repository contains the benchmarks and results obtained for the work "On Hardware Security Bug Code Fixes<br> By Querying Large Language Models".<br> Follow the README.md file for more information on how to use the tools yourself.</p>
RNA large language models embeddings on benchmark datasets
<p>This repository contains pre-computed embeddings for several RNA sequences, using most recent Large Language Models (LLM) pre-trained on RNA sequences. </p> <p>compressed files for each combination of RNA-LLM models and benchmarking RNA datasets. </p>
Supplementary material for Evaluating Legal Compliance of Smart Contracts Generated by Large Language Models
<p>This repository contains the supplementary material for the paper titled "Evaluating Legal Compliance of Smart Contracts Generated by Large Language Models". It includes natural-language legal contracts, their smart contract implementations, and Petri net models of said legal contracts contracts.</p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.