Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
36
datasets available to search
ShareScore release 0.9.0
Dataset results
36 results for “Language Models for Code”
Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code
<p>Artifact repository for the paper <a href="http://arxiv.org/abs/2308.03109" rel="nofollow"><em>Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code</em></a>, accepted at <em>ICSE 2024</em>, Lisbon, Portugal. Authors are <a href="https://rangeetpan.github.io/" rel="nofollow">Rangeet Pan</a>* <a href="https://alirezai.cs.illinois.edu/" rel="nofollow">Ali Reza Ibrahimzada</a>*, <a href="http://rkrsn.us/" rel="nofollow">Rahul Krishna</a>, Divya Sankar, Lambert Pougeum Wassi, Michele Merler, Boris Sobolev, Raju Pavuluri, Saurabh Sinha, and <a href="https://reyhaneh.cs.illinois.edu/index.htm" rel="nofollow">Reyhaneh Jabbarvand</a>.</p> <h3>Install</h3> <p>This repository contains the source code for reproducing the results in our paper. Please start by cloning this repository:</p> <div> <pre><code>git clone https://github.com/Intelligent-CAT-Lab/PLTranslationEmpirical </code></pre> </div> <p>We recommend using a virtual environment for running the scripts. Please download <code>conda 23.11.0</code> from this <a href="https://docs.conda.io/projects/miniconda/en/latest/miniconda-other-installer-links.html" rel="nofollow">link</a>. You can create a virtual environment using the following command:</p> <div> <pre><code>conda create -n plempirical python=3.10.13 </code></pre> </div> <p>After creating the virtual environment, you can activate it using the following command:</p> <div> <pre><code>conda activate plempirical </code></pre> </div> <p>You can run the following command to make sure that you are using the correct version of Python:</p> <div> <pre><code>python3 --version && pip3 --version </code></pre> </div> <h3>Dependencies</h3> <p>To install all software dependencies, please execute the following command:</p> <div> <pre><code>pip3 install -r requirements.txt </code></pre> </div> <p>As for hardware dependencies, we used 16 NVIDIA A100 GPUs with 80GBs of memory for inferencing models. The models can be inferenced on any combination of GPUs as long as the reader can properly distribute the model weights across the GPUs. We did not perform weight distribution since we had enough memory (80 GB) per GPU.</p> <p>Moreover, for compiling and testing the generated translations, we used Python 3.10, g++ 11, GCC Clang 14.0, Java 11, Go 1.20, Rust 1.73, and .Net 7.0.14 for Python, C++, C, Java, Go, Rust, and C#, respectively. Overall, we recommend using a machine with Linux OS and at least 32GB of RAM for running the scripts.</p> <p>For running scripts of alternative approaches, you need to make sure you have installed <a href="https://github.com/immunant/c2rust">C2Rust</a>, <a href="https://github.com/gotranspile/cxgo">CxGO</a>, and <a href="https://github.com/paulirwin/JavaToCSharp">Java2C#</a> on your machine. Please refer to their repositories for installation instructions. For Java2C#, you need to create a <code>.csproj</code> file like below:</p> <div> <pre><code><Project Sdk="Microsoft.NET.Sdk"> <PropertyGroup> <OutputType>Exe</OutputType> <TargetFramework>net7.0</TargetFramework> <ImplicitUsings>enable</ImplicitUsings> <Nullable>enable</Nullable> </PropertyGroup> </Project> </code></pre> </div> <h3>Dataset</h3> <p>We uploaded the dataset we used in our empirical study to <a href="../doi/10.5281/zenodo.8190051" rel="nofollow">Zenodo</a>. The dataset is organized as follows:</p> <ol> <li><a href="https://github.com/IBM/Project_CodeNet">CodeNet</a></li> <li><a href="https://github.com/wasiahmad/AVATAR">AVATAR</a></li> <li><a href="https://github.com/evalplus/evalplus">Evalplus</a></li> <li><a href="https://github.com/apache/commons-cli">Apache Commons-CLI</a></li> <li><a href="https://github.com/pallets/click">Click</a></li> </ol> <p>Please download and unzip the <code>dataset.zip</code> file from Zenodo. After unzipping, you should see the following directory structure:</p> <div> <pre><code>PLTranslationEmpirical ├── dataset ├── codenet ├── avatar ├── evalplus ├── real-life-cli ├── ... </code></pre> </div> <p>The structure of each dataset is as follows:</p> <p>1. CodeNet & Avatar: Each directory in these datasets correspond to a source language where each include two directories <code>Code</code> and <code>TestCases</code> for code snippets and test cases, respectively. Each code snippet has an <code>id</code> in the filename, where the <code>id</code> is used as a prefix for test I/O files.</p> <p>2. Evalplus: The source language code snippets follow a similar structure as CodeNet and Avatar. However, as a one time effort, we manually created the test cases in the target Java language inside a maven project, <code>evalplus_java</code>. To evaluate the translations from an LLM, we recommend moving the generated Java code snippets to the <code>src/main/java</code> directory of the maven project and then running the command <code>mvn clean test surefire-report:report -Dmaven.test.failure.ignore=true</code> to compile, test, and generate reports for the translations.</p> <p>3. Real-life Projects: The <code>real-life-cli</code> directory represents two real-life CLI projects from Java and Python. These datasets only contain code snippets as files and no test cases. As mentioned in the paper, the authors manually evaluated the translations for these datasets.</p> <h3>Scripts</h3> <p>We provide bash scripts for reproducing our results in this work. First, we discuss the translation script. For doing translation with a model and dataset, first you need to create a <code>.env</code> file in the repository and add the following:</p> <div> <pre><code>OPENAI_API_KEY=<your openai api key> LLAMA2_AUTH_TOKEN=<your llama2 auth token from huggingface> STARCODER_AUTH_TOKEN=<your starcoder auth token from huggingface> </code></pre> </div> <p>1. Translation with GPT-4: You can run the following command to translate all <code>Python -> Java</code> code snippets in <code>codenet</code> dataset with the <code>GPT-4</code> while top-k sampling is <code>k=50</code>, top-p sampling is <code>p=0.95</code>, and <code>temperature=0.7</code>:</p> <div> <pre><code>bash scripts/translate.sh GPT-4 codenet Python Java 50 0.95 0.7 0 </code></pre> </div> <p>2. Translation with CodeGeeX: Prior to running the script, you need to clone the CodeGeeX repository from <a href="https://github.com/THUDM/CodeGeeX">here</a> and use the instructions from their artifacts to download their model weights. After cloning it inside <code>PLTranslationEmpirical</code> and downloading the model weights, your directory structure should be like the following:</p> <div> <pre><code>PLTranslationEmpirical ├── dataset ├── codenet ├── avatar ├── evalplus ├── real-life-cli ├── CodeGeeX ├── codegeex ├── codegeex_13b.pt # this file is the model weight ├── ... ├── ... </code></pre> </div> <p>You can run the following command to translate all <code>Python -> Java</code> code snippets in <code>codenet</code> dataset with the <code>CodeGeeX</code> while top-k sampling is <code>k=50</code>, top-p sampling is <code>p=0.95</code>, and <code>temperature=0.2</code> on GPU <code>gpu_id=0</code>:</p> <div> <pre><code>bash scripts/translate.sh CodeGeeX codenet Python Java 50 0.95 0.2 0 </code></pre> </div> <p>3. For all other models (StarCoder, CodeGen, LLaMa, TB-Airoboros, TB-Vicuna), you can execute the following command to translate all <code>Python -> Java</code> code snippets in <code>codenet</code> dataset with the <code>StarCoder|CodeGen|LLaMa|TB-Airoboros|TB-Vicuna</code> while top-k sampling is <code>k=50</code>, top-p sampling is <code>p=0.95</code>, and <code>temperature=0.2</code> on GPU <code>gpu_id=0</code>:</p> <div> <pre><code>bash scripts/translate.sh StarCoder codenet Python Java 50 0.95 0.2 0 </code></pre> </div> <p>4. For translating and testing pairs with traditional techniques (i.e., C2Rust, CxGO, Java2C#), you can run the following commands:</p> <div> <pre><code>bash scripts/translate_transpiler.sh codenet C Rust c2rust fix_report bash scripts/translate_transpiler.sh codenet C Go cxgo fix_reports bash scripts/translate_transpiler.sh codenet Java C# java2c# fix_reports bash scripts/translate_transpiler.sh avatar Java C# java2c# fix_reports </code></pre> </div> <p>5. For compile and testing of CodeNet, AVATAR, and Evalplus (Python to Java) translations from GPT-4, and generating fix reports, you can run the following commands:</p> <div> <pre><code>bash scripts/test_avatar.sh Python Java GPT-4 fix_reports 1 bash scripts/test_codenet.sh Python Java GPT-4 fix_reports 1 bash scripts/test_evalplus.sh Python Java GPT-4 fix_reports 1 </code></pre> </div> <p>6. For repairing unsuccessful translations of Java -> Python in CodeNet dataset with GPT-4, you can run the following commands:</p> <div> <pre><code>bash scripts/repair.sh GPT-4 codenet Python Java 50 0.95 0.7 0 1 compile bash scripts/repair.sh GPT-4 codenet Python Java 50 0.95 0.7 0 1 runtime bash scripts/repair.sh GPT-4 codenet Python Java 50 0.95 0.7 0 1 incorrect </code></pre> </div> <p>7. For cleaning translations of open-source LLMs (i.e., StarCoder) in codenet, you can run the following command:</p> <div> <pre><code>bash scripts/clean_generations.sh StarCoder codenet </code></pre> </div> <p>Please note that for the above commands, you can change the dataset and model name to execute the same thing for other datasets and models. Moreover, you can refer to <a href="https://github.com/Intelligent-CAT-Lab/PLTranslationEmpirical/blob/main/prompts/README.md"><code>/prompts</code></a> for different vanilla and repair prompts used in our study.</p> <h3>Artifacts</h3> <p>Please download the <code>artifacts.zip</code> file from our <a href="../doi/10.5281/zenodo.8190051" rel="nofollow">Zenodo</a> repository. We have organized the artifacts as follows:</p> <ol> <li>RQ1 - Translations: This directory contains the translations from all LLMs and for all datasets. We have added an excel file to show a detailed breakdown of the translation results.</li> <li>RQ2 - Manual Labeling: This directory contains an excel file which includes the manual labeling results for all translation bugs.</li> <li>RQ3 - Alternative Approaches: This directory contains the translations from all alternative approaches (i.e., C2Rust, CxGO, Java2C#). We have added an excel file to show a detailed breakdown of the translation results.</li> <li>RQ4 - Mitigating Translation Bugs: This directory contains the fix results of GPT-4, StarCoder, CodeGen, and Llama 2. We have added an excel file to show a detailed breakdown of the fix results.</li> </ol> <h3>Contact</h3> <p>We look forward to hearing your feedback. Please contact <a href="mailto:rangeet.pan@ibm.com">Rangeet Pan</a> or <a href="mailto:alirezai@illinois.edu">Ali Reza Ibrahimzada</a> for any questions or comments 🙏.</p>
Specification-Driven Code Translation By Large Language Models: How Far Are We?
<p>The artifacts and dataset for "Specification-Driven Code Translation By Large Language Models: How Far Are We?"</p>
Code and Dataset for "Examining Zero-Shot Vulnerability Repair with Large Language Models"
<p><strong>Code and Dataset for "Examining Zero-Shot Vulnerability Repair with Large Language Models"</strong></p> <p>The following Zenodo contains the resources associated with the S&P accepted paper ‘Examining Zero Shot Vulnerability Repair with Large Language Models’, https://arxiv.org/abs/2112.02125</p> <p>In this resource, you can find the following.</p> <p> - 'important_results' directory:<br> This directory is for containing the final raw results as generated by the framework, including a global CSV of all generations and an HTML file containing all of the diffs generated for the 'high-confidence' real-world scenarios.<br> - final_results.csv<br> - This contains the final results of all generated software patches.<br> - Note the nomenclature differences with the manuscript tables. These are explained in the README in the framework.<br> - Original vs LLM-Generated Vulnerability Fixes.html<br> - This contains all of the diffs for the real-world patches versus the canonical developer-provided patches.</p> <p> - 'framework' directory:<br> This directory contains the complete archive of the code framework and all results at the time of the paper’s submission. It contains every language model prompt, suggestions, assembled repair patch and analysis data. It contains every script used for generation and analysis. It is a large archive, and within it contains an included README describing how to understand and use it.<br> - For convenience, we include a copy of the README external to the zipped archive.</p> <p> - 'resources' directory: <br> This directory contains the resources used by our large associated tools, including:<br> - 'gpt2-csrc' subdirectory:<br> Everything to do with the gpt2-csrc model, including the trained files, training scripts, and training data.</p> <p> - 'ExtractFix' subdirectory:<br> A docker image containing all ExtractFix scenarios, even those we did not use. Provided for interest (not required for usage).</p>
Data and code from: Learning a deep language model for microbiomes: The power of large scale unlabeled microbiome data
<p>We use open source human gut microbiome data to learn a microbial "language" model by adapting techniques from Natural Language Processing (NLP). Our microbial "language" model is trained in a self-supervised fashion (i.e., without additional external labels) to capture the interactions among different microbial species and the common compositional patterns in microbial communities. The learned model produces contextualized taxa representations that allow a single bacteria species to be represented differently according to the specific microbial environment it appears in. The model further provides a sample representation by collectively interpreting different bacteria species in the sample and their interactions as a whole. We show that, compared to baseline representations, our sample representation consistently leads to improved performance for multiple prediction tasks including predicting Irritable Bowel Disease (IBD) and diet patterns. Coupled with a simple ensemble strategy, it produces a highly robust IBD prediction model that generalizes well to microbiome data independently collected from different populations with substantial distribution shift.</p> <p>We visualize the contextualized taxa representations and find that they exhibit meaningful phylum-level structure, despite never exposing the model to such a signal. Finally, we apply an interpretation method to highlight bacterial species that are particularly influential in driving our model's predictions for IBD.</p>
ThoughtSource: A central hub for large language model reasoning data (code snapshot)
<p><strong>ThoughtSource is a meta-dataset and software library for chain-of-thought reasoning in large language models (LLMs). This repository contains a snapshot of the associated GitHub repository.</strong></p>
Data and code on the Moral Machine experiment on large language models (LLMs)
<p>As large language models (LLMs) have become more deeply integrated into various sectors, understanding how they make moral judgments has become crucial, particularly in the realm of autonomous driving. This study utilized the Moral Machine framework to investigate the ethical decision-making tendencies of prominent LLMs, including GPT-3.5, GPT-4, PaLM 2, and Llama 2, to compare their responses to human preferences. While LLMs' and humans' preferences such as prioritizing humans over pets and favoring saving more lives are broadly aligned, PaLM 2 and Llama 2, especially, evidence distinct deviations. Additionally, despite the qualitative similarities between the LLM and human preferences, there are significant quantitative disparities, suggesting that LLMs might lean toward more uncompromising decisions, compared to the milder inclinations of humans. These insights elucidate the ethical frameworks of LLMs and their potential implications for autonomous driving.</p>
Data and code on the Moral Machine experiment on large language models (LLMs)
Open the record for dataset details and reuse information.
Data and code from: Learning a deep language model for microbiomes: The power of large scale unlabeled microbiome data
Open the record for dataset details and reuse information.
Artifacts for paper "Code Refinement with the Assistance of Conversation-Aided Large Language Models"
<p>All the conversation data collected and the source code for our method are inside the uploaded zip file.</p>
(supplementary material) Fine-Tuning and Prompt Engineering for Large Language Models-based Code Review Automation
<div> <div> <div> <div>Supplementary material for paper <strong>"Fine-Tuning and Prompt Engineering for Large Language Models-based Code Review Automation"</strong></div> <div> </div> <div> <div> <div>The script for the paper can be found in this GitHub repository: https://github.com/awsm-research/LLM-for-code-review-automatiton</div> </div> </div> </div> </div> </div>
Do Current Language Models Support Code Intelligence for R Programming Language?
<p>This is the dataset used in the paper: Do Current Language Models Support Code Intelligence for Programming Language?</p> <p> </p> <p>This dataset contains code snippets from R programming language repositories on GitHub, paired with their corresponding natural language (NL) descriptions. It was created for research in software engineering tasks like code summarization and code search. The data was collected using the GitHub REST API and includes over 1,500 public R repositories. To ensure quality, only active, well-structured R packages with proper documentation were included. Roxygen2, a popular documentation framework, was used to extract both the code and its matching NL descriptions.</p> <p>The dataset is organized into three parts: base R functions (Base), functions from the tidyverse (Tidy), and a combined set (RCombine). The dataset follows the CodeSearchNet format, with a split for training, validation, and testing data, ensuring no duplicate functions.</p>
Collu-Bench: A Benchmark for Predicting Language Model Hallucinations in Code
<p>Despite their success, large language models (LLMs) face the critical challenge of hallucinations, generating plausible but incorrect content. While much research has focused on hallucinations in multiple modalities including images and natural language text, less attention has been given to hallucinations in source code, which leads to incorrect and vulnerable code that causes significant financial loss. To pave the way for research in LLMs' hallucinations in code, we introduce Collu-Bench, a benchmark for predicting code hallucinations of LLMs across code generation (CG) and automated program repair (APR) tasks. Collu-Bench includes 13,234 code hallucination instances collected from five datasets and 11 diverse LLMs, ranging from open-source models to commercial ones. <br>To better understand and predict code hallucinations, Collu-Bench provides detailed features such as the per-step log probabilities of LLMs' output, token types, and the execution feedback of LLMs' generated code for in-depth analysis. In addition, we conduct experiments to predict hallucination on Collu-Bench, using both traditional machine learning techniques and neural networks, which achieves 22.03 -- 33.15% accuracy. Our experiments draw insightful findings of code hallucination patterns, reveal the challenge of accurately localizing LLMs' hallucinations, and highlight the need for more sophisticated techniques.</p>
Official Code and Dataset of Table Tennis Coaching System Based on a Multimodal Large Language Model with Knowledge Base
<p>Official Code and Dataset of Table Tennis Coaching System Based on a Multimodal Large Language Model with Knowledge Base</p>
Impact of Code Language Models on Automated Program Repair (Dataset)
<p>This is the fine-tuning dataset used in the paper Impact of Code Language Models on Automated Program Repair</p>
Integration of protein and coding sequences enables mutual augmentation of the language model
<p><strong>The file structure is as follows:</strong></p> <p>Project Root<br>├── TE_MRL<br>│ ├── MRL_dataset.zip<br>│ └── TE_dataset.zip<br>│<br>├── finetuned_model<br>│ ├── FoldP<br>│ ├── LocP<br>│ ├── SSP<br>│ └── SolP<br>│<br>├── tax_tsne<br>│ └── emb_3models.zip<br>│<br>└── training_data<br> ├── FoldP.csv<br> ├── LocP.csv<br> ├── SolP.csv<br> ├── SSP.pkl<br> └── pretrain_source_GCF.txt</p>
Data and code for "Large language models identify causal genes in complex trait GWAS"
<p><span>This file contains the data and scripts for the preprint "Large language models identify causal genes in complex trait GWAS"</span></p>
Characterizing Code Clones from Large Language Models Dataset and Scripts
<p>characterizing_code_clones_data.zip: <br><br>This dataset contains a collection of code snippets generated by Large Language Models (LLMs) such as GPT-3.5 and GPT-4 in response to specific programming prompts derived from LeetCode. Each sub-directory within the dataset corresponds to a particular LLM version and contains code snippets, preprocessed data, and SLACC input files. </p><p>characterizing_code_clones_project.zip: </p><p>This zipped directory encompasses the core scripts and results used in the "Characterizing Code Clones of LLMs" research. It features the Python script <strong>collect_samples.py</strong> for collecting LLM-generated code snippets, as well as a suite of scripts in the <strong>slacc_scripts</strong> sub-directory for processing and analyzing the data using SLACC. The directory also includes the results of the LeetCode test suites, providing insights into the correctness and efficiency of the code generated by GPT-3.5 and GPT-4. </p>
Leveraging Search-Based and Pre-Trained Code Language Models for Automated Program Repair
<p>This page serves as supplementary material for the article: <strong>Leveraging Search-Based and Pre-Trained Code Language Models for Automated Program Repair</strong>. Here, we provide the ARJACLM code utilized in the study, enabling other researchers to replicate the experiments and further develop the tool. </p>
A Systematic Evaluation of Large Language Models of Code
<p>These are datasets for the paper:</p> <p>"A Systematic Evaluation of Large Language Models of Code"</p> <p><a href="https://arxiv.org/pdf/2202.13169.pdf">https://arxiv.org/pdf/2202.13169.pdf</a></p> <p>The code is available at: <a href="https://github.com/VHellendoorn/Code-LMs">https://github.com/VHellendoorn/Code-LMs</a></p> <p> </p> <p>The file "<a href="https://zenodo.org/record/6338015/files/unseen_test_sets.tar.gz">unseen_test_sets.tar.gz</a>" contains test sets of ~100 files in each of 12 programming languages.</p> <p>These files are not included in The Pile, and thus models such as GPT-Neo, GPT-J, GPT-NeoX were not trained on them.</p> <p>In the paper, we use these test sets to compare a variety of language models of code including OpenAI's Codex, GPT-J, GPT-Neo, GPT-NeoX-20B, and CodeParrot and our PolyCoder model.</p> <p> </p> <p>The file "<a href="https://zenodo.org/record/6341643/files/index.zip?download=1">index.zip</a>" includes an index of the <strong>training set</strong> file paths and commit SHAs.</p> <p> </p> <p>The other files, such as "<a href="https://zenodo.org/record/6344914/files/2-7B-150K.tar">2-7B-150K.tar</a>", are trained model checkpoints, as explained at <a href="https://github.com/VHellendoorn/Code-LMs">https://github.com/VHellendoorn/Code-LMs</a> .</p>
Dataset related to: An Empirical Study on Low Code Programming using Traditional vs Large Language Model Support
<p>This repository contains data and prompts related to our research. The files included are:</p> <p><strong>prompt.py</strong>: This script contains the original prompts used in our study.</p> <p><strong>LLM_lowcode.mx20</strong>: This file includes posts and annotation data related to LLM-based low-code platforms.</p> <p><strong>Traditional_lowcode.mx20</strong>: This file includes posts and annotation data related to traditional low-code platforms.</p> <p>The .mx20 files can be opened using the MAXQDA software, which can be downloaded from the official website. MAXQDA offers a 14-day free trial.</p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.