Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
167
datasets available to search
ShareScore release 0.7.1
Dataset results
167 results for “Large Language Models”
Dataset for "Large Language Models as molecular design engines"
<ol> <li><strong>claude-gpt-paper.zip :</strong><br><br>This dataset contains data and results associated with the paper "Large Language Models as molecular design<br>engines" The paper investigates the use of large language models, specifically Claude 3 Opus, for generating and analyzing chemical structures based on various prompts from A-H (as mentioned in the manuscript), and guided design related to electron-withdrawing groups (EWG), electron-donating groups (EDG).</li> </ol> <p>The dataset includes:</p> <ol> <li>PM7 MOPAC energy calculations for generated molecules, along with their SMILES representations and molecule IDs.</li> <li>PM7-calculated charges for the generated molecules.</li> <li>Output files from the Claude 3 Opus language model for each prompt category along.</li> <li>Original dataset (subset of ZINC database) used to build common keys and the initial design space.</li> <li>JSON file containing common keys for featurizing unknown SMILES.</li> <li>PCA object to convert molecule embeddings to 3-dimensional embeddings.</li> </ol> <p>The data is organized into the following folders:</p> <ul> <li><code>pm7_charge_results</code>: Contains HOMO-LUMO energy differences for plotting.</li> <li><code>pm7_charge_calculation</code>: Contains PM7 MOPAC energy calculations and charges.</li> <li><code>out</code>: Contains output files from the Claude 3 Opus language model.</li> <li><code>fact-dropbox</code>: Contains the original dataset, common keys, and PCA object file.</li> </ul> <p>The data can be used to reproduce the results presented in the paper and serve as a foundation for further research in this area.</p> <p>For a detailed description of the folder structure and contents, please refer to the File_descriptions.md file included in the dataset.<br><br><br>2. llm-visulizer-dashapp.zip<br><br>This is the code for the visualizer app for viewing the molecules generated by the LLM. The README.md file has details about running the app.</p> <p>3. claude-gpt-paper-codes.zip </p> <p>This contains the notebook GPT_modification_just_plots.ipynb for plotting, and other codes. The README.md file has details about running the main notebook for getting the plots.</p>
Dataset for : A New Era in Software Security: Towards Self-Healing Software via Large Language Models and Formal Verification
<p>We present a novel solution combining Large Language Model (LLM) capabilities with Formal Verification strategies to falsify and automatically repair software vulnerabilities. Initially, we employ Bounded Model Checking (BMC) to locate the software vulnerability and derive a counterexample. Relying on mathematical proofs, counterexamples provide evidence that the system behaves incorrectly or contains a vulnerability, thereby preventing the generation of false positive alerts. The counterexample that has been detected, along with the source code, are provided to the LLM engine. Our approach involves establishing a specialized prompt language for conducting code debugging and generation to understand the vulnerability's root cause and repair the code. Finally, we use BMC to verify the corrected version of the code generated by the LLM. As a proof of concept, we create \esbmcai based on the Efficient SMT-based Context-Bounded Model Checker (ESBMC) and a pre-trained Transformer model, specifically gpt-3.5-turbo, to detect and fix errors in C programs. We generated a dataset comprising $1{,}000$ C code samples, each consisting of $20$ to $50$ lines of C code. Experimental results show that our proposed method achieved an impressive success rate of up to $80$\% in repairing vulnerable code, encompassing buffer overflow, arithmetic overflow, and pointer dereference failures. To our knowledge, \esbmcai represents the first proposal for a pioneering initiative to integrate a Large Language Model (LLM) with software model checking. We advocate that this automated approach has the potential to incorporate into the software development lifecycle's continuous integration and deployment (CI/CD) process. </p> <p> </p> <p>The uploaded dataset contains 1000 codes, each comprising 20 to 50 lines of C code generated with gpt-3.5-turbo. The material also consists of a version of ESBMC statically compiled with all dependencies, a classifier script, and the output file.</p> <p> </p> <p> </p>
FoodSky: A Food-oriented Large Language Model, and FoodEarth: A Foundamental Food Corpus and Instruction Dataset
<p>Food is the cornerstone of both survival and social life. With the increasing complexity of global dietary needs and preferences, there is a growing demand for food intelligence to enable tasks like recipe recommendation and diet-disease correlation discovery. To address this, we introduce the Food-oriented Large Language Model (LLM) FoodSky, which offers fine-grained perception and reasoning of food data. We constructed a food corpus, FoodEarth, from various authoritative sources to enhance FoodSky's knowledge. We also developed the Topic-based Selective State Space Model and Hierarchical Topic Retrieval Augmented Generation algorithms to improve FoodSky's ability to capture fine-grained food semantics and generate context-aware food-relevant text. Extensive experiments show that FoodSky outperforms general-purpose LLMs on the Chinese National Chef Exam and Dietetic Exam, achieving accuracies of 67.2% and 66.4%, respectively. FoodSky not only enhances culinary creativity and promotes healthier eating patterns but also establishes a new standard for domain-specific LLMs tackling real-world food-related issues.</p>
Replication Package for the paper "Conversing with business process-aware Large Language Models: the BPLLM framework"
<p>Replication Package for the research paper "<em>Conversing with business process-aware Large Language Models: the BPLLM framework</em>".</p> <p>The package includes the process models, the questions (and expected answers), the results of the qualitative evaluation, and the Hugging Face links to the fine-tuned versions of Llama 3.1 8B employed in the quantitative evaluation of the framework.</p> <p>In particular, the process models are:</p> <ul> <li>The natural language Directly-follows graph (DFG) of the Food Delivery process: <em>food_delivery_activities.txt</em> for the definition of the activities and <em>food_delivery_flow.txt</em> for the sequence flow.</li> <li>The BPMN model of the Food Delivery, E-commerce, and Reimbursement processes: <em>ecommerce.bpmn</em>, <em>food_delivery.bpmn</em>, and <em>reimbursement.bpmn</em>.</li> </ul> <p>The datasets with the questions and the expected answers are:</p> <ul> <li><em>1_questions_answers_not_refined_for_DFG.csv</em> ;</li> <li><em>1.1_questions_answers_refined_for_DFG.csv</em> ;</li> <li><em>2_questions_answers_not_refined.csv</em> ;</li> <li><em>3_questions_answers_refined.csv</em> ;</li> <li><em>4_questions_answers_different_processes.csv</em> ;</li> <li><em>5_questions_answers_similar_processes.csv</em> ;</li> <li><em>6_questions_answers_refined_ft.csv</em> .</li> </ul> <p>The complete results of the qualitative evaluation are contained in the file <em>qualitative_experiments_results.pdf</em>.</p> <p>The Hugging Face links to the fine-tuned versions of Llama 3.1 8B are reported in <em>hf_links_finetuned_models.pdf</em>.</p>
Large Language Models are Easily Confused: A Quantitative Metric, Security Implications and Typological Analysis
<p>This repository contain datasets and results for the paper:</p> <p><strong>Large Language Models are Easily Confused: A Quantitative Metric, Security Implications and Typological Analysis</strong></p> <p> </p> <p><strong>Github repository for the code: </strong></p> <p><a href="https://github.com/siebeniris/QuantifyingLanguageConfusion/tree/main">Quantifying Language Confusion GitHub repo</a></p> <p> </p> <p><strong>DATA</strong> include the following datasets:</p> <p>i) raw language graphs and</p> <p>ii) the calculated language similarities from the language graphs,</p> <p>iii) <strong>MTEI</strong>: the files from the <a href="https://github.com/siebeniris/vec2text_exp/tree/aaai">experimental results of multilingual inversion attacks</a>, and calculated language confusion entropy from the data;</p> <p>iv) <strong>LCB</strong>: the files from the <a href="https://github.com/for-ai/language-confusion?tab=Apache-2.0-1-ov-file#readme">language confusion benchmark</a> and calculated language confusion entropy from the data </p> <p> </p> <p><strong>Results</strong> include aggregated results for further analysis:</p> <p>i) <strong>inversion_language_confusion</strong>: results from MTEI</p> <p>ii) <strong>prompting_language_confusion</strong>: results from LCB</p> <p> </p> <p> </p>
Replication package for "Assessing the Latent Automated Program Repair Capabilities of Large Language Models using Round-Trip Translation"
<p>This repository contains the replication package for the paper "Assessing the Latent Automated Program Repair Capabilities of Large Language Models using Round-Trip Translation" by Fernando Vallecillos Ruiz, Anastasiia Grishina, Max Hort and Leon Moonen, accepted for publication in ACM Transactions on Software Engineering and Methodology on 2025-10-09.</p> <p>A preprint is deposited on arXiv with DOI: <a href="https://doi.org/10.48550/arXiv.2401.07994">10.48550/arXiv.2401.07994</a>.</p> <p>The replication package is archived on Zenodo with DOI: <a href="https://doi.org/10.5281/zenodo.10500593">10.5281/zenodo.10500593</a>. It is maintained on GitHub at <a href="https://github.com/secureIT-project/RTT_for_APR">https://github.com/secureIT-project/RTT_for_APR</a>.</p> <p>This project builds on code from the <a href="https://github.com/lin-tan/clm/">clm</a> project, which is (c) 2023, The ASSET research group led by Lin Tan, Purdue University, licensed under the BSD 3-Clause License (see jasper/LICENSE.BSD). All modifications and new contributions are (c) 2025 by the authors of this replication package and distributed under the MIT License (see LICENSE.MIT). The data, models and preprint are distributed under the CC BY 4.0 license.</p> <h2>Citation<code> </code></h2> <p>If you build on this data or code, please cite this work by referring to the paper:</p> <div> <pre><code>@article{ruiz2025:rtt, title = {Assessing the Latent Automated Program Repair Capabilities of Large Language Models using Round-Trip Translation}, author = {Vallecillos Ruiz, Fernando and Anastasiia Grishina and Max Hort and Leon Moonen}, journal = {ACM Transactions on Software Engineering and Methodology (TOSEM)}, year = {2025}, publisher = {{ACM}} }</code></pre> </div> <h2>Organization</h2> <p>The replication package is organized as follows:</p> <ul> <li>clm-apr <ul> <li>plbart: code to generate patches with PLBART models.</li> <li>codet5: code to generate patches with CodeT5 models.</li> <li>transcoder: code to generate patches with the TransCoder model.</li> <li>incoder: code to generate patches with InCoder models.</li> <li>santacoder: code to generate patches with the SantaCoder model.</li> <li>starcoder: code to generate patches with the StarCoderBase model.</li> <li>quixbugs: code to validate patches generated for the QuixBugs benchmark.</li> <li>defects4j: code to validate patches generated for any of the Defects4J benchmarks.</li> <li>humaneval: code to validate patches generated for the HumanEval-Java benchmark.</li> </ul> </li> <li>humaneval-java: the HumanEval-Java benchmark proposed by Jiang et al. 2023</li> <li>jasper: a Java tool to parse Java programs needed to preprocess input.</li> <li>model: folder to download the language models.</li> <li>analysis_wandb: data from WandB and Jupyter notebook to create graphs.</li> <li>tmp_benchmarks: folder for temporary files used in patch validation. The folder may contain pairs of `paralell’ folders src and src_org for each benchmark, used to replace buggy code with candidate patches.</li> </ul> <h2>Replication</h2> <h3>Prerequisites</h3> <ul> <li>Python version: 3.8—3.10.</li> <li><a href="https://git-lfs.com/">Git LFS</a> is required for model downloading.</li> </ul> <h4>Weight and Biases (WandB)</h4> <ol> <li>Create an account on <a href="https://wandb.ai/">Weights and Biases</a></li> <li>Install the <a href="https://docs.wandb.ai/ref/python">Weights and Biases</a> library</li> <li>Run <code>wandb login</code> and follow the instructions</li> </ol> <h4>Set up OpenAI access</h4> <p>OpenAI account is needed with access to <code>gpt-3.5-turbo</code> and <code>gpt-4</code> . The <code>OPENAI_API_KEY</code> environment variable should be set to your OpenAI API access token.</p> <h3>Dependencies</h3> <ul> <li><a href="https://github.com/rjust/defects4j">Defects4J</a> - To generate inputs for the Defects4J datasets or to validate them, you need to have installed <a href="https://github.com/rjust/defects4j">their tool</a>.</li> <li>Java 8</li> <li>Apache Maven</li> </ul> <h3>Setup</h3> <p>We recommend the use of the setup script:</p> <pre><code>setup.sh </code></pre> <p>which performs the following:</p> <ol> <li>Creates a virtual environment for Python and activate it.</li> <li>Install the packages in <code>requirements.txt</code>.</li> <li>Compiles Jasper.</li> <li>Downloads parsers.</li> <li>Check if the Defects4J installation is correct.</li> </ol> <h3>Download models</h3> <p>The following bash script contains the code to download all of the models used:</p> <pre><code>models/download_models.sh </code></pre> <p>We recommend downloading only the models you are going to use due to their size</p> <pre><code>cd models chmod +x download_models.sh ./download_models.sh </code></pre> <p>To run one specific model, for example, PLBART (C#), use the following commands:</p> <pre><code>cd models git lfs install git clone https://huggingface.co/uclanlp/plbart-java-cs git clone https://huggingface.co/uclanlp/plbart-cs-java cd ../.. </code></pre> <h3>Step 1: Preprocessing and Prompting:</h3> <p>Each script in each <code>clm-apr/[model]</code> folder connects one or more models with<br>one dataset. These scripts follow the template: [benchmark]_[model]_[technique].py.<br>The scripts first create an <code>[model]_input.json</code> file with the preprocessed<br>input. Then generate outputs based on that file with one or more models.<br>For example:</p> <pre><code>cd clm-apr/plbart python quixbugs_plbart_round.py # Generates input for QuixBugs and generate patches using Java<->C# RTT. python quixbugs_plbart_round_nl.py # Generates input for QuixBugs and generate patches using Java<->NL RTT. </code></pre> <p>Optionally, use argument <code>--device_map cpu</code> if you wish to run the script on<br>CPU, for example:</p> <pre><code>python quixbugs_plbart_round.py --device_map cpu </code></pre> <p>Otherwise, the script will be run on all available CUDA GPU’s.</p> <p>We have commented the generation of inputs in the scripts. Users are free to<br>uncomment this method and try for themselves. It is easily recognizable by<br>their name template <code>[model]_[benchmark]_input()</code>. In the previous case:</p> <pre><code>quixbugs_plbart_input() </code></pre> <h3>Step 2 and 3: Round Trip Translation and Postprocessing</h3> <p>These steps are also included in the [benchmark]_[model]_[technique].py<br>script mentioned above. They are modularized in the method recognizable by<br>their name template [model]_[benchmark]_output().<br>For example:</p> <pre><code>quixbugs_incoder_output() </code></pre> <p>This method:</p> <ol> <li>Reads the input json file.</li> <li>Generates outputs through the LLM.</li> <li>Postprocess the output (extract the patch, clean up extra token, etc.).</li> <li>Creates [model]_output_[technique]_[extra].json.</li> </ol> <p>The last 3 steps are repeated according to the number of runs set to performed<br>(10 in our experiments). Each run will produce a different file with the seed<br>used in its generation. For example, <code>quixbugs\_plbart\_round.py</code> and<br><code>quixbugs\_plbart\_round_nl.py</code> scripts create:</p> <pre><code>clm-apr/quixbugs/plbart_results/run_0/plbart_java_cs_java_output_round_csharp_batch.json clm-apr/quixbugs/plbart_results/run_0/plbart_java_nl_java_output_round_nl_batch.json </code></pre> <h3>Step 4: Evaluation of RTT Results:</h3> <p>The last step evaluates the generated outputs against the test-suites of each<br>benchmark. This script reads the previous outputs files and generates a new one<br>with the results of the test for one model. Furthermore, it connects with the<br><em>WandB</em> tool to calculate metrics and send them to analyze.</p> <p>Following the previous examples, to validate the results previously obtained,<br>we execute the following:</p> <pre><code>cd clm-apr/quixbugs python validate_quixbugs_parallel.py </code></pre> <p>Given the included JSON, this script would create:</p> <pre><code>clm-apr/quixbugs/plbart_results/run_0/plbart_java_cs_java_validate_round_csharp_batch.json </code></pre> <p>We have disabled <em>WandB</em> in the script to allow users to try the script first.<br>However, it can be easily activated by changing the parameter <code>mode="disabled"</code><br>to <code>mode="online"</code>.<br>We have set the variable <code>total_runs = 1</code>, as well as <code>input_file</code> and <code>output_file</code><br>to the results included. They should be modified accordingly to validate more runs<br>or to validate other files/models.</p> <h3>Included Results</h3> <p>We include two CSV files obtained through WandB.</p> <pre><code>'data_cleaned_grouped.csv': Aggregated metrics of the 25 outputs for all runs. 'full_data_all_runs.csv': All metrics for all outputs on all runs. </code></pre> <h2>Changelog</h2> <ul> <li>v1.0 - updates corresponding to the accepted version of the manuscript in TOSEM</li> <li>v0.1 - initial replication package corresponding to v1 of arXiv deposit: includes raw data, code, and example outputs.</li> </ul> <h2>References</h2> <p>Jiang, N.; Liu, K.; Lutellier, T.; and Tan, L. 2023. Impact of Code Language<br>Models on Automated Program Repair. In 45th International Conference on<br>Software Engineering (ICSE), 1430–1442. IEEE. ISBN 978-1-66545-701-9.</p> <div> </div>
Delphi Study: Exploring the Implications of Large Language Models on the Science System
<p><strong>Sample description:</strong> Our target audience consisted of researchers working in the fields of science, technology, and society with a specific interest in Large Language Models (LLMs).</p> <p><strong>Collection method: </strong>Participants were recruited through the professional and personal networks of the authors, as well as the Alexander von Humboldt Institute (HIIG), using a combination of generic emails via LimeSurvey and personal contacts.</p> <p><strong>Description. </strong>The aim of this study was to explore the impact of large language models, specifically ChatGPT, on scholarly practice and academic writing, targeting researchers and experts in the fields of artificial intelligence, science, and technology who publish their research and scientific work. The two-stage Delphi survey sought to identify and assess the potential opportunities and challenges associated with the use of ChatGPT in academic work and scientific writing, with a specific focus on research impact rather than university teaching. Phase 1 yielded 72 responses, while Phase 2 had 52 responses.</p> <p>To conduct our analysis, we developed two distinct codebooks (see Files ChatGPT Delphi Codebook Phase 1.csv and ChatGPT Delphi Codebook Phase 2.csv) for the Delphi study. The first codebook was created by examining approximately half of the responses, extracting relevant information, and generating codes through inductive reasoning. We then categorized and developed subcodes based on these initial codes, assigning them to each participant's answers using deductive reasoning. For example, when addressing the potential applications of ChatGPT and other language models (LLMs), we identified six subcategories with precise definitions and illustrative examples. The analysis in Phase 1 led to the formulation of ranking questions for Phase 2, focusing on determining the most frequently utilized applications of ChatGPT and other LLMs based on the established codes.</p> <p>During Phase 2, we introduced two additional open-ended questions to explore the impact of ChatGPT and LLMs on the scientific system and society, aiming to envision future scenarios. The analysis of these questions in the second codebook followed a similar approach to Phase 1, including inductive reasoning for code generation and deductive reasoning for assigning codes to the answers. We observed overlapping codes with the Phase 1 codebook and assigned them to the second codebook. Additionally, we noted a shift in the connotation of certain answers from neutral in Phase 1 to being perceived as either positive or negative consequences of ChatGPT and other LLMs. This observation prompted the bifurcation of specific codes to capture the nuanced perspectives. For instance, applications such as reducing administrative tasks initially seen as valuable aids for researchers were sometimes viewed as potential causes for job replacement, implying negative outcomes.</p> <p>For detailed information on the analytical approach employed, including references to these methodologies, please refer to the methodology chapter in the official publication.</p> <p><strong>Content</strong></p> <ol> <li> <p>Questionaire-ChatGPT-Delphi-Phase1-Limesurvey-Export.pdf – This file file is an exported version of the Phase 1 questionnaire from Limesurvey. It includes the description, socio demographic questions, content questions, and a request for participant naming.</p> </li> <li> <p>Questionaire-ChatGPT-Delphi-Phase2-Limesurvey-Export.pdf – This file file is an exported version of the Phase 2 questionnaire from Limesurvey. It includes the description, socio demographic questions, content questions, and a request for participant naming.</p> </li> <li> <p>ChatGPT Delphi - Results Phase 1.pdf – This file contains the responses and corresponding questions from Phase 1 of the Delphi study. The responses provided by the participants are in the form of open-ended answers. As part of this publication, we have ensured the anonymity of the participants.</p> </li> <li> <p>ChatGPT Delphi - Results Phase 2. pdf – This file contains the responses and corresponding questions from Phase 2 of the Delphi study. It encompasses the ranking answers provided by the participants, as well as two open-ended answers. To maintain anonymity consistently, all participants have been anonymized again in this publication of our results.</p> </li> <li> <p>ChatGPT Delphi Codebook Phase 1.pdf – This file contains the Phase 1 codebook, which presents the primary codes, their respective subcodes, detailed definitions, and noteworthy examples.</p> </li> <li> <p>ChatGPT Delphi Codebook Phase 2.pdf – This file contains the Phase 1 codebook, which provides a comprehensive overview of the primary codes within the given scenario. It includes their corresponding subcodes, detailed definitions, and notable examples to enhance understanding and interpretation.</p> </li> </ol>
Dataset Worldwide Survey on the Impact of AI Chatbots and Large Language Models in Dental Education: Insights from Dental Educators
<p><strong>This dataset contains responses from participants regarding their awareness, knowledge, and perceptions of AI-powered tools in dental education. The data was collected during May-June 2023 to investigate the potential enhancement that AI can bring to dental education. The dataset includes variables related to participants' demographics, experiences, perceptions, and opinions.</strong></p> <p><strong>Details in the published protocol by Uribe, S. E., & Maldupa, I. (2023, June 2). Chatbots In Dental Education - Research Protocol. https://doi.org/10.17605/OSF.IO/3BSG2</strong></p>
HoneyBee: Progressive Instruction Finetuning of Large Language Models for Materials Science
<p>We propose an instruction-based process for trustworthy data curation in materials science (MatSci-Instruct), which we then apply to finetune a LLaMa-based language model targeted for materials science (HoneyBee). MatSci-Instruct helps alleviate the scarcity of relevant, high-quality materials science textual data available in the open literature, and HoneyBee is the first billion-parameter language model specialized to materials science. In MatSci-Instruct we improve the trustworthiness of generated data by prompting multiple commercially available large language models for generation with an Instructor module (e.g. Chat-GPT) and verification from an independent Verifier module (e.g. Claude). Using MatSci-Instruct, we construct a dataset of multiple tasks and measure the quality of our dataset along multiple dimensions, including accuracy against known facts, relevance to materials science, as well as completeness and reasonableness of the data. Moreover, we iteratively generate more targeted instructions and instruction-data in a finetuning-evaluation-feedback loop leading to progressively better performance for our finetuned HoneyBee models. Our evaluation on the MatSci-NLP benchmark shows HoneyBee's outperformance of existing language models on materials science tasks and iterative improvement in successive stages of instruction-data refinement. We study the quality of HoneyBee's language modeling through automatic evaluation and analyze case studies to further understand the model's capabilities and limitations. Our code and relevant datasets are publicly available at https://github.com/BangLab-UdeM-Mila/NLP4MatSci-HoneyBee.</p>
Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code
<p>Artifact repository for the paper <a href="http://arxiv.org/abs/2308.03109" rel="nofollow"><em>Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code</em></a>, accepted at <em>ICSE 2024</em>, Lisbon, Portugal. Authors are <a href="https://rangeetpan.github.io/" rel="nofollow">Rangeet Pan</a>* <a href="https://alirezai.cs.illinois.edu/" rel="nofollow">Ali Reza Ibrahimzada</a>*, <a href="http://rkrsn.us/" rel="nofollow">Rahul Krishna</a>, Divya Sankar, Lambert Pougeum Wassi, Michele Merler, Boris Sobolev, Raju Pavuluri, Saurabh Sinha, and <a href="https://reyhaneh.cs.illinois.edu/index.htm" rel="nofollow">Reyhaneh Jabbarvand</a>.</p> <h3>Install</h3> <p>This repository contains the source code for reproducing the results in our paper. Please start by cloning this repository:</p> <div> <pre><code>git clone https://github.com/Intelligent-CAT-Lab/PLTranslationEmpirical </code></pre> </div> <p>We recommend using a virtual environment for running the scripts. Please download <code>conda 23.11.0</code> from this <a href="https://docs.conda.io/projects/miniconda/en/latest/miniconda-other-installer-links.html" rel="nofollow">link</a>. You can create a virtual environment using the following command:</p> <div> <pre><code>conda create -n plempirical python=3.10.13 </code></pre> </div> <p>After creating the virtual environment, you can activate it using the following command:</p> <div> <pre><code>conda activate plempirical </code></pre> </div> <p>You can run the following command to make sure that you are using the correct version of Python:</p> <div> <pre><code>python3 --version && pip3 --version </code></pre> </div> <h3>Dependencies</h3> <p>To install all software dependencies, please execute the following command:</p> <div> <pre><code>pip3 install -r requirements.txt </code></pre> </div> <p>As for hardware dependencies, we used 16 NVIDIA A100 GPUs with 80GBs of memory for inferencing models. The models can be inferenced on any combination of GPUs as long as the reader can properly distribute the model weights across the GPUs. We did not perform weight distribution since we had enough memory (80 GB) per GPU.</p> <p>Moreover, for compiling and testing the generated translations, we used Python 3.10, g++ 11, GCC Clang 14.0, Java 11, Go 1.20, Rust 1.73, and .Net 7.0.14 for Python, C++, C, Java, Go, Rust, and C#, respectively. Overall, we recommend using a machine with Linux OS and at least 32GB of RAM for running the scripts.</p> <p>For running scripts of alternative approaches, you need to make sure you have installed <a href="https://github.com/immunant/c2rust">C2Rust</a>, <a href="https://github.com/gotranspile/cxgo">CxGO</a>, and <a href="https://github.com/paulirwin/JavaToCSharp">Java2C#</a> on your machine. Please refer to their repositories for installation instructions. For Java2C#, you need to create a <code>.csproj</code> file like below:</p> <div> <pre><code><Project Sdk="Microsoft.NET.Sdk"> <PropertyGroup> <OutputType>Exe</OutputType> <TargetFramework>net7.0</TargetFramework> <ImplicitUsings>enable</ImplicitUsings> <Nullable>enable</Nullable> </PropertyGroup> </Project> </code></pre> </div> <h3>Dataset</h3> <p>We uploaded the dataset we used in our empirical study to <a href="../doi/10.5281/zenodo.8190051" rel="nofollow">Zenodo</a>. The dataset is organized as follows:</p> <ol> <li><a href="https://github.com/IBM/Project_CodeNet">CodeNet</a></li> <li><a href="https://github.com/wasiahmad/AVATAR">AVATAR</a></li> <li><a href="https://github.com/evalplus/evalplus">Evalplus</a></li> <li><a href="https://github.com/apache/commons-cli">Apache Commons-CLI</a></li> <li><a href="https://github.com/pallets/click">Click</a></li> </ol> <p>Please download and unzip the <code>dataset.zip</code> file from Zenodo. After unzipping, you should see the following directory structure:</p> <div> <pre><code>PLTranslationEmpirical ├── dataset ├── codenet ├── avatar ├── evalplus ├── real-life-cli ├── ... </code></pre> </div> <p>The structure of each dataset is as follows:</p> <p>1. CodeNet & Avatar: Each directory in these datasets correspond to a source language where each include two directories <code>Code</code> and <code>TestCases</code> for code snippets and test cases, respectively. Each code snippet has an <code>id</code> in the filename, where the <code>id</code> is used as a prefix for test I/O files.</p> <p>2. Evalplus: The source language code snippets follow a similar structure as CodeNet and Avatar. However, as a one time effort, we manually created the test cases in the target Java language inside a maven project, <code>evalplus_java</code>. To evaluate the translations from an LLM, we recommend moving the generated Java code snippets to the <code>src/main/java</code> directory of the maven project and then running the command <code>mvn clean test surefire-report:report -Dmaven.test.failure.ignore=true</code> to compile, test, and generate reports for the translations.</p> <p>3. Real-life Projects: The <code>real-life-cli</code> directory represents two real-life CLI projects from Java and Python. These datasets only contain code snippets as files and no test cases. As mentioned in the paper, the authors manually evaluated the translations for these datasets.</p> <h3>Scripts</h3> <p>We provide bash scripts for reproducing our results in this work. First, we discuss the translation script. For doing translation with a model and dataset, first you need to create a <code>.env</code> file in the repository and add the following:</p> <div> <pre><code>OPENAI_API_KEY=<your openai api key> LLAMA2_AUTH_TOKEN=<your llama2 auth token from huggingface> STARCODER_AUTH_TOKEN=<your starcoder auth token from huggingface> </code></pre> </div> <p>1. Translation with GPT-4: You can run the following command to translate all <code>Python -> Java</code> code snippets in <code>codenet</code> dataset with the <code>GPT-4</code> while top-k sampling is <code>k=50</code>, top-p sampling is <code>p=0.95</code>, and <code>temperature=0.7</code>:</p> <div> <pre><code>bash scripts/translate.sh GPT-4 codenet Python Java 50 0.95 0.7 0 </code></pre> </div> <p>2. Translation with CodeGeeX: Prior to running the script, you need to clone the CodeGeeX repository from <a href="https://github.com/THUDM/CodeGeeX">here</a> and use the instructions from their artifacts to download their model weights. After cloning it inside <code>PLTranslationEmpirical</code> and downloading the model weights, your directory structure should be like the following:</p> <div> <pre><code>PLTranslationEmpirical ├── dataset ├── codenet ├── avatar ├── evalplus ├── real-life-cli ├── CodeGeeX ├── codegeex ├── codegeex_13b.pt # this file is the model weight ├── ... ├── ... </code></pre> </div> <p>You can run the following command to translate all <code>Python -> Java</code> code snippets in <code>codenet</code> dataset with the <code>CodeGeeX</code> while top-k sampling is <code>k=50</code>, top-p sampling is <code>p=0.95</code>, and <code>temperature=0.2</code> on GPU <code>gpu_id=0</code>:</p> <div> <pre><code>bash scripts/translate.sh CodeGeeX codenet Python Java 50 0.95 0.2 0 </code></pre> </div> <p>3. For all other models (StarCoder, CodeGen, LLaMa, TB-Airoboros, TB-Vicuna), you can execute the following command to translate all <code>Python -> Java</code> code snippets in <code>codenet</code> dataset with the <code>StarCoder|CodeGen|LLaMa|TB-Airoboros|TB-Vicuna</code> while top-k sampling is <code>k=50</code>, top-p sampling is <code>p=0.95</code>, and <code>temperature=0.2</code> on GPU <code>gpu_id=0</code>:</p> <div> <pre><code>bash scripts/translate.sh StarCoder codenet Python Java 50 0.95 0.2 0 </code></pre> </div> <p>4. For translating and testing pairs with traditional techniques (i.e., C2Rust, CxGO, Java2C#), you can run the following commands:</p> <div> <pre><code>bash scripts/translate_transpiler.sh codenet C Rust c2rust fix_report bash scripts/translate_transpiler.sh codenet C Go cxgo fix_reports bash scripts/translate_transpiler.sh codenet Java C# java2c# fix_reports bash scripts/translate_transpiler.sh avatar Java C# java2c# fix_reports </code></pre> </div> <p>5. For compile and testing of CodeNet, AVATAR, and Evalplus (Python to Java) translations from GPT-4, and generating fix reports, you can run the following commands:</p> <div> <pre><code>bash scripts/test_avatar.sh Python Java GPT-4 fix_reports 1 bash scripts/test_codenet.sh Python Java GPT-4 fix_reports 1 bash scripts/test_evalplus.sh Python Java GPT-4 fix_reports 1 </code></pre> </div> <p>6. For repairing unsuccessful translations of Java -> Python in CodeNet dataset with GPT-4, you can run the following commands:</p> <div> <pre><code>bash scripts/repair.sh GPT-4 codenet Python Java 50 0.95 0.7 0 1 compile bash scripts/repair.sh GPT-4 codenet Python Java 50 0.95 0.7 0 1 runtime bash scripts/repair.sh GPT-4 codenet Python Java 50 0.95 0.7 0 1 incorrect </code></pre> </div> <p>7. For cleaning translations of open-source LLMs (i.e., StarCoder) in codenet, you can run the following command:</p> <div> <pre><code>bash scripts/clean_generations.sh StarCoder codenet </code></pre> </div> <p>Please note that for the above commands, you can change the dataset and model name to execute the same thing for other datasets and models. Moreover, you can refer to <a href="https://github.com/Intelligent-CAT-Lab/PLTranslationEmpirical/blob/main/prompts/README.md"><code>/prompts</code></a> for different vanilla and repair prompts used in our study.</p> <h3>Artifacts</h3> <p>Please download the <code>artifacts.zip</code> file from our <a href="../doi/10.5281/zenodo.8190051" rel="nofollow">Zenodo</a> repository. We have organized the artifacts as follows:</p> <ol> <li>RQ1 - Translations: This directory contains the translations from all LLMs and for all datasets. We have added an excel file to show a detailed breakdown of the translation results.</li> <li>RQ2 - Manual Labeling: This directory contains an excel file which includes the manual labeling results for all translation bugs.</li> <li>RQ3 - Alternative Approaches: This directory contains the translations from all alternative approaches (i.e., C2Rust, CxGO, Java2C#). We have added an excel file to show a detailed breakdown of the translation results.</li> <li>RQ4 - Mitigating Translation Bugs: This directory contains the fix results of GPT-4, StarCoder, CodeGen, and Llama 2. We have added an excel file to show a detailed breakdown of the fix results.</li> </ol> <h3>Contact</h3> <p>We look forward to hearing your feedback. Please contact <a href="mailto:rangeet.pan@ibm.com">Rangeet Pan</a> or <a href="mailto:alirezai@illinois.edu">Ali Reza Ibrahimzada</a> for any questions or comments 🙏.</p>
Automated Programming Exercise Generation in the Era of Large Language Models
<p>Lecturers are increasingly attempting to use large language models (LLMs) to simplify and make the creation of exercises for students more efficient. Efforts are also being made to automate the exercise creation process in software engineering (SE) education. This study explores the use of advanced LLMs, including GPT-4 and LaMDA, for automated programming exercise creation in higher education and compares the results with related work using GPT-3.5-turbo. Utilizing applications such as ChatGPT, Bing AI Chat, and Google Bard, we identify LLMs capable of initiating different exercise designs. However, manual refinement is crucial for accuracy. Common error patterns across LLMs highlight challenges in complex programming concepts, while specific strengths in various topics showcase model distinctions. This research underscores LLMs' value in exercise generation, emphasizing the critical role of human supervision in refining these processes. Our concise insights cater to educators, practitioners, and other researchers seeking to enhance SE education through LLM applications.</p>
Specification-Driven Code Translation By Large Language Models: How Far Are We?
<p>The artifacts and dataset for "Specification-Driven Code Translation By Large Language Models: How Far Are We?"</p>
USPTO-LLM: A Large Language Model-Assisted Information-enriched Chemical Reaction Dataset
<p>USPTO-LLM is an <strong>information-enriched chemical reaction dataset</strong> that provides more side information (reaction conditions and reaction steps division) for developing new reaction prediction and retrosynthesis methods and inspires new problems, such as reaction condition prediction. It comprises over <strong>247K chemical reactions</strong> extracted from the patent documents of USPTO (United States Patent and Trademark Office), encompassing abundant information on reaction conditions. </p> <p>We employ large language models to expedite the data collection procedures automatically with a reliable quality control process. The extracted chemical reactions are organized as <strong>heterogeneous directed graphs</strong>, allowing us to formulate a series of prediction tasks, such as reaction prediction, retrosynthesis, and reaction condition prediction, in a unified graph-filling framework.</p>
Paper information in the topic of large language models
<p>This dataset supports the findings in the preprint 'Academic collaboration on large language model studies increases overall but varies across disciplines.' The study aims to explore the application of large language models (LLMs) in scientific disciplines and their implications for interdisciplinary collaboration.</p> <p>To build LLM paper group, we start with a broad search using general terms related to LLMs and popular models based on the MMLU benchmark spanning from October 2018 to September 2024. We apply this search to the title and abstract to avoid excessive noise in the dataset and then undergo a series of filtering steps<br>to enhance relevance and remove duplicates. The resulting dataset contains 59,293 papers.</p> <p>In addition to the paper group in the topic of LLMs, we establish two control groups. The first control group focuses on machine learning (ML) papers. We select ML as a control because it is a well-established field from which LLM emerged as a subfield. To construct this group, we collect a random sampling of 70,945 papers containing the phrase ''machine learning'' in either their title or abstract. To provide an even broader perspective beyond AI-related fields, we create a second control group consisting of a random sample of 73,110 papers from all other research categories---specifically, papers that belong neither to the ML nor LLM categories. </p> <p>The three files below contain the cleaned samples collected from OpenAlex, which are derived from the original files. </p> <ul> <li>LLM: llm-cleaned-samples.csv</li> <li>ML: ml-cleaned-samples.csv</li> <li>Non-LLM/ML: non-llm-cleaned-samples.csv</li> </ul> <p>The three zip files below contain author affiliation information (including departmental discipline) extracted by GPT-4o-mini to support the departmental analysis in the paper:</p> <ul> <li>LLM: llm-author-affiliations.zip</li> <li>ML: ml-author-affiliations.zip</li> <li>Non-LLM/ML: non-llm-author-affiliations.zip</li> </ul> <p>The three files below contain the paper information used to support all the analysis in our paper:</p> <ul> <li>LLM: llm-information-entropy.csv</li> <li>ML: ml-information-entropy.csv</li> <li>Non-LLM/ML: non-llm-information-entropy.csv</li> </ul> <p>If you have any additional questions, please feel free to contact <a rel="noreferrer">lingyaol@umich.edu or lydinh@usf.edu.</a></p>
Distinguishing GUI Component States for Blind Users using Large Language Models
<p><strong># Data Code Repository</strong></p><p> </p><p>This repository contains open-source data code that provides utilities for the paper named "Here comes trouble! Distinguishing GUI Component States for Blind Users using Large Language Models". The code is designed to facilitate data-related tasks and promote reproducibility in research and data analysis projects.</p><p> </p><p><strong>## Features</strong></p><p> </p><p>- Attribute identification and extraction: Including real-time recognition and extraction of GUI components in the view type, resource-id, color, action of four attributes</p><p>- Components State Distinction: Provides the prompt needed for large language models, covering their specific design schemes and chain of thought reasoning processes as well as contextual learning content.</p><p>- Implementation: Offers specific methods to realize the process, including the setting of relevant parameters and the use of functions.</p><p> </p><p><strong>## Installation</strong></p><p> </p><p>To use the data code, you can down or clone the required code.</p><p>Notably, before using the code, make sure the necessary environment configuration is done.</p><p> </p><p><strong>## Dependencies</strong></p><p>The data code has the following dependencies:</p><p> </p><p>Python (version 3.6 or higher)</p><p>NumPy</p><p>Pandas</p><p>Seaborn</p><p>Scikit-learn</p><p>Openai</p><p>Android Studio (version 4.0)</p><p> </p><p>Install the required dependencies using pip:</p><p>pip install numpy..</p><p> </p><p><strong>##License</strong></p><p>This data code is distributed under the MIT License. See LICENSE for more information.</p><p> </p><p><strong>##Copyright</strong></p><p>All copyright of the tool is owned by the author of the paper.</p>
Code and Dataset for "Examining Zero-Shot Vulnerability Repair with Large Language Models"
<p><strong>Code and Dataset for "Examining Zero-Shot Vulnerability Repair with Large Language Models"</strong></p> <p>The following Zenodo contains the resources associated with the S&P accepted paper ‘Examining Zero Shot Vulnerability Repair with Large Language Models’, https://arxiv.org/abs/2112.02125</p> <p>In this resource, you can find the following.</p> <p> - 'important_results' directory:<br> This directory is for containing the final raw results as generated by the framework, including a global CSV of all generations and an HTML file containing all of the diffs generated for the 'high-confidence' real-world scenarios.<br> - final_results.csv<br> - This contains the final results of all generated software patches.<br> - Note the nomenclature differences with the manuscript tables. These are explained in the README in the framework.<br> - Original vs LLM-Generated Vulnerability Fixes.html<br> - This contains all of the diffs for the real-world patches versus the canonical developer-provided patches.</p> <p> - 'framework' directory:<br> This directory contains the complete archive of the code framework and all results at the time of the paper’s submission. It contains every language model prompt, suggestions, assembled repair patch and analysis data. It contains every script used for generation and analysis. It is a large archive, and within it contains an included README describing how to understand and use it.<br> - For convenience, we include a copy of the README external to the zipped archive.</p> <p> - 'resources' directory: <br> This directory contains the resources used by our large associated tools, including:<br> - 'gpt2-csrc' subdirectory:<br> Everything to do with the gpt2-csrc model, including the trained files, training scripts, and training data.</p> <p> - 'ExtractFix' subdirectory:<br> A docker image containing all ExtractFix scenarios, even those we did not use. Provided for interest (not required for usage).</p>
Data and code from: Learning a deep language model for microbiomes: The power of large scale unlabeled microbiome data
<p>We use open source human gut microbiome data to learn a microbial "language" model by adapting techniques from Natural Language Processing (NLP). Our microbial "language" model is trained in a self-supervised fashion (i.e., without additional external labels) to capture the interactions among different microbial species and the common compositional patterns in microbial communities. The learned model produces contextualized taxa representations that allow a single bacteria species to be represented differently according to the specific microbial environment it appears in. The model further provides a sample representation by collectively interpreting different bacteria species in the sample and their interactions as a whole. We show that, compared to baseline representations, our sample representation consistently leads to improved performance for multiple prediction tasks including predicting Irritable Bowel Disease (IBD) and diet patterns. Coupled with a simple ensemble strategy, it produces a highly robust IBD prediction model that generalizes well to microbiome data independently collected from different populations with substantial distribution shift.</p> <p>We visualize the contextualized taxa representations and find that they exhibit meaningful phylum-level structure, despite never exposing the model to such a signal. Finally, we apply an interpretation method to highlight bacterial species that are particularly influential in driving our model's predictions for IBD.</p>
CausalBench A Comprehensive Benchmark for Evaluating Causal Reasoning Capabilities of Large Language Models
<p>CausalBench is a comprehensive benchmark dataset designed to evaluate the causal reasoning capabilities of large language models. The primary uses of this dataset include, but are not limited to:</p> <p>- Testing the performance of large language models on causal reasoning tasks</p> <p>- Serving as a benchmark dataset for causal reasoning research</p> <p>- Improving and developing new causal reasoning algorithms and models</p>
Lost in Translation? Not for Large Language Models: Automated Divergent Thinking Scoring Performance Translates to Non-English Contexts (Datasets)
<p>Datasets for: Zielińska, A., Organisciak, P., Dumas, D., & Karwowski, M. (2023). Lost in translation? Not for large language models: Automated divergent thinking scoring performance translates to non-English contexts. <em>Thinking Skills and Creativity, 50</em>, 101414. https://doi.org/10.1016/j.tsc.2023.101414</p>
Large Language Model-Based Classification of Flash Flood Impacts Across the United States
<p>This repository contains the data sets used for the publication of the journal article titled <em>Large Language Model-Based Classification of Flash Flood Impacts Across the United States</em>.</p> <p>This is the first release of the data with a Zenodo DOI attached to the README.md file. </p> <p>Further information about the data can be found in the GitHub repository's README.md file.</p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.