Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
21
datasets available to search
ShareScore release 0.9.0
Dataset results
21 results for “Automated program repair”
Replication package for "Assessing the Latent Automated Program Repair Capabilities of Large Language Models using Round-Trip Translation"
<p>This repository contains the replication package for the paper "Assessing the Latent Automated Program Repair Capabilities of Large Language Models using Round-Trip Translation" by Fernando Vallecillos Ruiz, Anastasiia Grishina, Max Hort and Leon Moonen, accepted for publication in ACM Transactions on Software Engineering and Methodology on 2025-10-09.</p> <p>A preprint is deposited on arXiv with DOI: <a href="https://doi.org/10.48550/arXiv.2401.07994">10.48550/arXiv.2401.07994</a>.</p> <p>The replication package is archived on Zenodo with DOI: <a href="https://doi.org/10.5281/zenodo.10500593">10.5281/zenodo.10500593</a>. It is maintained on GitHub at <a href="https://github.com/secureIT-project/RTT_for_APR">https://github.com/secureIT-project/RTT_for_APR</a>.</p> <p>This project builds on code from the <a href="https://github.com/lin-tan/clm/">clm</a> project, which is (c) 2023, The ASSET research group led by Lin Tan, Purdue University, licensed under the BSD 3-Clause License (see jasper/LICENSE.BSD). All modifications and new contributions are (c) 2025 by the authors of this replication package and distributed under the MIT License (see LICENSE.MIT). The data, models and preprint are distributed under the CC BY 4.0 license.</p> <h2>Citation<code> </code></h2> <p>If you build on this data or code, please cite this work by referring to the paper:</p> <div> <pre><code>@article{ruiz2025:rtt, title = {Assessing the Latent Automated Program Repair Capabilities of Large Language Models using Round-Trip Translation}, author = {Vallecillos Ruiz, Fernando and Anastasiia Grishina and Max Hort and Leon Moonen}, journal = {ACM Transactions on Software Engineering and Methodology (TOSEM)}, year = {2025}, publisher = {{ACM}} }</code></pre> </div> <h2>Organization</h2> <p>The replication package is organized as follows:</p> <ul> <li>clm-apr <ul> <li>plbart: code to generate patches with PLBART models.</li> <li>codet5: code to generate patches with CodeT5 models.</li> <li>transcoder: code to generate patches with the TransCoder model.</li> <li>incoder: code to generate patches with InCoder models.</li> <li>santacoder: code to generate patches with the SantaCoder model.</li> <li>starcoder: code to generate patches with the StarCoderBase model.</li> <li>quixbugs: code to validate patches generated for the QuixBugs benchmark.</li> <li>defects4j: code to validate patches generated for any of the Defects4J benchmarks.</li> <li>humaneval: code to validate patches generated for the HumanEval-Java benchmark.</li> </ul> </li> <li>humaneval-java: the HumanEval-Java benchmark proposed by Jiang et al. 2023</li> <li>jasper: a Java tool to parse Java programs needed to preprocess input.</li> <li>model: folder to download the language models.</li> <li>analysis_wandb: data from WandB and Jupyter notebook to create graphs.</li> <li>tmp_benchmarks: folder for temporary files used in patch validation. The folder may contain pairs of `paralell’ folders src and src_org for each benchmark, used to replace buggy code with candidate patches.</li> </ul> <h2>Replication</h2> <h3>Prerequisites</h3> <ul> <li>Python version: 3.8—3.10.</li> <li><a href="https://git-lfs.com/">Git LFS</a> is required for model downloading.</li> </ul> <h4>Weight and Biases (WandB)</h4> <ol> <li>Create an account on <a href="https://wandb.ai/">Weights and Biases</a></li> <li>Install the <a href="https://docs.wandb.ai/ref/python">Weights and Biases</a> library</li> <li>Run <code>wandb login</code> and follow the instructions</li> </ol> <h4>Set up OpenAI access</h4> <p>OpenAI account is needed with access to <code>gpt-3.5-turbo</code> and <code>gpt-4</code> . The <code>OPENAI_API_KEY</code> environment variable should be set to your OpenAI API access token.</p> <h3>Dependencies</h3> <ul> <li><a href="https://github.com/rjust/defects4j">Defects4J</a> - To generate inputs for the Defects4J datasets or to validate them, you need to have installed <a href="https://github.com/rjust/defects4j">their tool</a>.</li> <li>Java 8</li> <li>Apache Maven</li> </ul> <h3>Setup</h3> <p>We recommend the use of the setup script:</p> <pre><code>setup.sh </code></pre> <p>which performs the following:</p> <ol> <li>Creates a virtual environment for Python and activate it.</li> <li>Install the packages in <code>requirements.txt</code>.</li> <li>Compiles Jasper.</li> <li>Downloads parsers.</li> <li>Check if the Defects4J installation is correct.</li> </ol> <h3>Download models</h3> <p>The following bash script contains the code to download all of the models used:</p> <pre><code>models/download_models.sh </code></pre> <p>We recommend downloading only the models you are going to use due to their size</p> <pre><code>cd models chmod +x download_models.sh ./download_models.sh </code></pre> <p>To run one specific model, for example, PLBART (C#), use the following commands:</p> <pre><code>cd models git lfs install git clone https://huggingface.co/uclanlp/plbart-java-cs git clone https://huggingface.co/uclanlp/plbart-cs-java cd ../.. </code></pre> <h3>Step 1: Preprocessing and Prompting:</h3> <p>Each script in each <code>clm-apr/[model]</code> folder connects one or more models with<br>one dataset. These scripts follow the template: [benchmark]_[model]_[technique].py.<br>The scripts first create an <code>[model]_input.json</code> file with the preprocessed<br>input. Then generate outputs based on that file with one or more models.<br>For example:</p> <pre><code>cd clm-apr/plbart python quixbugs_plbart_round.py # Generates input for QuixBugs and generate patches using Java<->C# RTT. python quixbugs_plbart_round_nl.py # Generates input for QuixBugs and generate patches using Java<->NL RTT. </code></pre> <p>Optionally, use argument <code>--device_map cpu</code> if you wish to run the script on<br>CPU, for example:</p> <pre><code>python quixbugs_plbart_round.py --device_map cpu </code></pre> <p>Otherwise, the script will be run on all available CUDA GPU’s.</p> <p>We have commented the generation of inputs in the scripts. Users are free to<br>uncomment this method and try for themselves. It is easily recognizable by<br>their name template <code>[model]_[benchmark]_input()</code>. In the previous case:</p> <pre><code>quixbugs_plbart_input() </code></pre> <h3>Step 2 and 3: Round Trip Translation and Postprocessing</h3> <p>These steps are also included in the [benchmark]_[model]_[technique].py<br>script mentioned above. They are modularized in the method recognizable by<br>their name template [model]_[benchmark]_output().<br>For example:</p> <pre><code>quixbugs_incoder_output() </code></pre> <p>This method:</p> <ol> <li>Reads the input json file.</li> <li>Generates outputs through the LLM.</li> <li>Postprocess the output (extract the patch, clean up extra token, etc.).</li> <li>Creates [model]_output_[technique]_[extra].json.</li> </ol> <p>The last 3 steps are repeated according to the number of runs set to performed<br>(10 in our experiments). Each run will produce a different file with the seed<br>used in its generation. For example, <code>quixbugs\_plbart\_round.py</code> and<br><code>quixbugs\_plbart\_round_nl.py</code> scripts create:</p> <pre><code>clm-apr/quixbugs/plbart_results/run_0/plbart_java_cs_java_output_round_csharp_batch.json clm-apr/quixbugs/plbart_results/run_0/plbart_java_nl_java_output_round_nl_batch.json </code></pre> <h3>Step 4: Evaluation of RTT Results:</h3> <p>The last step evaluates the generated outputs against the test-suites of each<br>benchmark. This script reads the previous outputs files and generates a new one<br>with the results of the test for one model. Furthermore, it connects with the<br><em>WandB</em> tool to calculate metrics and send them to analyze.</p> <p>Following the previous examples, to validate the results previously obtained,<br>we execute the following:</p> <pre><code>cd clm-apr/quixbugs python validate_quixbugs_parallel.py </code></pre> <p>Given the included JSON, this script would create:</p> <pre><code>clm-apr/quixbugs/plbart_results/run_0/plbart_java_cs_java_validate_round_csharp_batch.json </code></pre> <p>We have disabled <em>WandB</em> in the script to allow users to try the script first.<br>However, it can be easily activated by changing the parameter <code>mode="disabled"</code><br>to <code>mode="online"</code>.<br>We have set the variable <code>total_runs = 1</code>, as well as <code>input_file</code> and <code>output_file</code><br>to the results included. They should be modified accordingly to validate more runs<br>or to validate other files/models.</p> <h3>Included Results</h3> <p>We include two CSV files obtained through WandB.</p> <pre><code>'data_cleaned_grouped.csv': Aggregated metrics of the 25 outputs for all runs. 'full_data_all_runs.csv': All metrics for all outputs on all runs. </code></pre> <h2>Changelog</h2> <ul> <li>v1.0 - updates corresponding to the accepted version of the manuscript in TOSEM</li> <li>v0.1 - initial replication package corresponding to v1 of arXiv deposit: includes raw data, code, and example outputs.</li> </ul> <h2>References</h2> <p>Jiang, N.; Liu, K.; Lutellier, T.; and Tan, L. 2023. Impact of Code Language<br>Models on Automated Program Repair. In 45th International Conference on<br>Software Engineering (ICSE), 1430–1442. IEEE. ISBN 978-1-66545-701-9.</p> <div> </div>
The dataset of the APSEC'20 paper titled "Understanding the Non-Repairability Factors of Automated Program Repair Techniques"
<p>The file <strong>ExecutionLogs.zip</strong> includes all execution logs that we take into consideration in the study. </p> <p>The file <strong>Exception_Statistics.xlsx</strong> includes the distribution of each type of exception in our study. </p> <p> </p> <p>All logs are located in directories that are named by fault localization strategy and the corresponding automated program repair tool. Specifically, in the directories of <strong>ExecutionLogs</strong> there are four sub-directories: <em>GZ-0.1</em>, <em>GZ-1.7</em>, <em>No_FL_Required</em>, and <em>Perfect_FL</em>.</p> <ul> <li><em>GZ-0.1</em> means fault localization strategy with GZoltar-v0.1 (from the <a href="https://dl.acm.org/doi/10.1145/3338906.3338911">FSE19</a> paper).</li> <li><em>GZ-1.7</em> means fault localization strategy with GZoltar-v1.7 (from the <a href="https://zenodo.org/record/3678960#.XwcRNIgzaUk">ICSE20</a> paper).</li> <li><em>Perfect_FL</em> means directly feeding ground-truth bug positions to APR tools (from the <a href="https://zenodo.org/record/3678960#.XwcRNIgzaUk">ICSE20</a> paper).</li> <li><em>No_FL_Required</em> means that the tool does not require fault localization process (from <a href="https://dl.acm.org/doi/10.1145/3338906.3338911">FSE19</a> paper).</li> </ul> <p>For example, the log of bug <em>Chart-1</em> generated by <em>Cardumen</em> with <em>Gzoltar-0.1</em> are stored in the directory <strong>GZ-0.1/Chart/1/Cardumen/</strong>. </p> <p> </p> <p>In the file <strong>Exception_Statistics.xlsx</strong>, we demonstrate the detailed distribution of each type of exception.</p> <p><strong>Format of the content</strong></p> <p>We use format like <em>Fault_localization_strategy-project_id-version_id-tool_name</em> to represent each log.</p> <p>Take <em>GZ0.1-Chart-1-Cardumen</em> as an example, this log is generated by <em>Cardumen</em> on bug <em>Chart-1</em> with fault localization strategy <em>GZoltar-v0.1</em>.</p> <p><strong>About compliance level</strong></p> <p>Take the log of <em>GZ0.1-Mockito-1-Cardumen</em> as an example. We can get the compliance level from the parameter <em>javacompliancelevel</em>. For example, 4 corresponds to java 1.4, 5 corresponds to java 1.5 and so on. Hence, we can know what compliance level authors used during the original experiment.</p> <pre><code class="language-java">time java -Xmx4048m -cp /home/tdurieux/RepairThemAll/script/../libs/jtestex7.jar:/home/tdurieux/RepairThemAll/script/../repair_tools/astor.jar fr.inria.main.evolution.AstorMain \ -mode cardumen \ -location . \ -id Chart-1 \ -failing org.jfree.chart.renderer.category.junit.AbstractCategoryItemRendererTests \ -jvm4testexecution /home/tdurieux/openjdk-1.7.0/bin/ \ -jvm4evosuitetestexecution /home/tdurieux/openjdk-1.7.0/bin/ \ -maxgen 1000000 \ -maxtime 120 \ -stopfirst true \ -seed 0 \ -scope local \ -population 1 \ -javacompliancelevel 4 \ ......</code></pre> <p><strong>About the server</strong></p> <p>In the end of each log, it illustrates the brief information about the cloud node. Hence, we can know which logs are from the same node.</p> <pre><code class="language-java">...... Node: suno-2.sophia.grid5000.fr Date: Fri Dec 28 21:21:00 EST 2018</code></pre> <p>We next give introduction to the omitted three types of exceptions in the paper.</p> <ul> <li><strong>NumberFormat exception</strong></li> </ul> <p>Below is an example of this type of exception. As we have mentioned in the paper, due to the impossibility of integrating GZoltar-v1.7 into APR tools, researchers choose to read the FL results from independent files. From the content of the log, Astor needs to get suspicious line numbers during initialization (executing <em>createEngine</em>) but triggers an exception. This exception only occurs for Closure-76 and we have confirmed with the authors of reference[9] that it is due to the wrong content of the input file. We thus categorized it into <strong>Improper operation of replication study</strong>.</p> <pre><code class="language-java">Exception in thread "main" java.lang.NumberFormatException: For input string: "" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Integer.parseInt(Integer.java:592) at java.lang.Integer.parseInt(Integer.java:615) at fr.inria.main.evolution.AstorMain.createEngine(AstorMain.java:172) at ......</code></pre> <ul> <li><strong>IO exception</strong></li> </ul> <p>During the experiment, the generated output files (i.e., the execution logs and the generated patches) can sometimes be more than 10 trillions of bites. Meanwhile, the system may generate and store abundant template files such as the variants of the buggy program. If there is not enough space left on the machine, the execution logs will not be outputted successfully and an IO exception will occur as shown below. This indicates that these exceptions may not occur if the template files can be cleaned in time. We thus classified this type into <strong>Improper operation of replication study</strong> in that researchers ignore the storage state of the machine.</p> <pre><code class="language-java">Exception in thread "main" java.lang.RuntimeException: java.io.IOException: No space left on device at fr.inria.astor.core.setup.ProjectRepairFacade.setupWorkingDirectories(ProjectRepairFacade.java:46) at fr.inria.main.evolution.AstorMain.initProject(AstorMain.java:61) at fr.inria.main.evolution.AstorMain.run(AstorMain.java:239)</code></pre> <ul> <li><strong>NoClassDefFound exception</strong></li> </ul> <p>An example of this type of exception is illustrated below. We failed to reproduce this exception in our own machine. We further tried the docker file provided in the project page of reference[8] but still did not succeed. We noted that the experiment in reference[8] was performed on cloud instances whose experimental environment may be unstable. Bugs that throw this exception are consecutive (i.e., the IDs of these bugs are consecutive which are from Math 39 to Math 52 in the Defects4J benchmark), which further confirms our hypothesis that the environment might be problematic during these executions. We thus categorized this type into <strong>Unstable environment</strong>.</p> <pre><code class="language-java">Exception in thread "main" java.lang.NoClassDefFoundError: com/gzoltar/core/spectra/Spectra at com.gzoltar.core.GZoltar.run(GZoltar.java:50) at us.msu.cse.repair.core.faultlocalizer.GZoltarFaultLocalizer.<init>(GZoltarFaultLocalizer.java:42) at us.msu.cse.repair.core.AbstractRepairProblem.invokeFaultLocalizer(AbstractRepairProblem.java:298) at us.msu.cse.repair.core.AbstractRepairProblem.invokeModules(AbstractRepairProblem.java:269) at us.msu.cse.repair.core.AbstractRepairProblem.<init>(AbstractRepairProblem.java:244) at us.msu.cse.repair.algorithms.kali.Kali.<init>(Kali.java:37) at ......</code></pre> <p> </p>
Comparing developer-provided to user-provided tests for fault localization and automated program repair: Artifacts
<p>Artifacts for the paper <em>Comparing developer-provided to user-provided tests for fault localization and automated program repair.</em></p> <p>Note that the artifacts are maintained in the following repositories:</p> <ul> <li>https://github.com/rjust/defects4j</li> <li>https://bitbucket.org/rjust/tests-tested-data</li> <li>https://bitbucket.org/rjust/fault-localization-data</li> </ul>
Impact of Code Language Models on Automated Program Repair (Dataset)
<p>This is the fine-tuning dataset used in the paper Impact of Code Language Models on Automated Program Repair</p>
KNOD: Domain Knowledge Distilled Tree Decoder for Automated Program Repair (Dataset)
<p>This is the training dataset used in the paper KNOD: Domain Knowledge Distilled Tree Decoder for Automated Program Repair (Dataset)</p>
Automated Patch Assessment for Program Repair at Scale
<p>In this package, we share 257 patches from Dcorrect and 381 patches from Doverfitting.</p> <p>We also a script used in our experiment for doing sanity checks and automated patch assessments.</p> <p> </p> <p>If you use this dataset, please cite:</p> <pre><code>@Article{Ye2021EMSE, author = {Ye, He and Martinez, Matias and Monperrus, Martin}, title = "Automated Patch Assessment for Program Repair at Scale", journal="Empirical Software Engineering", volume = "26", issn = "1573-7616", doi = "https://doi.org/10.1007/s10664-020-09920-w", year = "2021" } </code></pre> <p> </p>
Leveraging Search-Based and Pre-Trained Code Language Models for Automated Program Repair
<p>This page serves as supplementary material for the article: <strong>Leveraging Search-Based and Pre-Trained Code Language Models for Automated Program Repair</strong>. Here, we provide the ARJACLM code utilized in the study, enabling other researchers to replicate the experiments and further develop the tool. </p>
HeteroGen: Transpiling C to Heterogeneous HLS Code with Automated Test Generation and Program Repair
<p>This artifact submission includes 1. an error study, 2. a fuzzing-based test generation tool, and 3. a code-editing tool for error removal.</p>
Replication Package for "Timing Side Channel Mitigation via Automated Program Repair"
<p><strong>Pendulum: Timing Side Channel Mitigation via Automated Program Repair</strong></p> <p>This is the replication artifact for the paper "Timing Side Channel Mitigation via Automated Program Repair". It includes:</p> <ul> <li>a <strong>Docker</strong> file for easy setup and execution of Pendulum,</li> <li>the <strong>setup</strong> instructions,</li> <li>the instructions for a <strong>quick trial</strong> of the setup,</li> <li>the <strong>workflow</strong> description incl. how to <strong>apply</strong> Pendulum to other programs, and</li> <li>detailed instructions about how to <strong>replicate our experiments</strong>.</li> </ul> <p>Please read our README for more information and explanations.</p>
Experiment data of Automated Program Repair from Fuzzing Perspective
<p>This dataset contains the result of experiment of the paper called Automated Program Repair from Fuzzing Perspective.</p> <p>Each zip file contains the results of each APR tool.</p> <p>Our artifact evaluation is in <a href="https://doi.org/10.5281/zenodo.7972926">https://doi.org/10.5281/zenodo.7972926</a>.</p>
The repository of the SANER'21 paper titled "On the Impact of Flaky Tests in Automated Program Repair"
<p>This repository contains the artefact of the paper “On the Impact of Flaky Tests in Automated Program Repair” under review by SANER2021.</p> <ul> <li>results: <ul> <li><strong>RQ1:</strong> This table contains all the statistical data of flaky tests we find out from Defects4J benchmark where each line illustrates the commit date, commit id, java file name, as well as all the flaky methods in this test file. Note that the commit id is empty if there is no update about the java file in that line since last version. <ul> <li>flaky_methods.xlsx</li> </ul> </li> <li><strong>RQ2:</strong> Here are the execution results of 10 times running of flaky tests under 4 different environment. Detailed execution log of each running are classified by project and named in the format <strong><em>test_log_[project_name].txt</em></strong>,the summarized data of each project are named in the format <strong><em>Flaky_test_[project_name].xlsx</em></strong>. The <strong><em>Flaky_test_all.xlsx</em></strong> contain the summarized data for all projects. <ul> <li>jdk1.7 + ubuntu16.04</li> <li>jdk1.7 + ubuntu18.04</li> <li>jdk1.8 + ubuntu16.04</li> <li>jdk1.8 + ubuntu18.04</li> </ul> </li> <li><strong>RQ3:</strong> This folder contains execution results of the fault localization tool <a href="https://github.com/GZoltar/gzoltar">GZoltar-V1.7</a> under two jdk versions. Result of each bug is named in the format <strong><em>[project_name]_[bug_id].csv</em></strong>. <ul> <li>gzoltar_reuslt_flaky_jdk1.7</li> <li>gzoltar_reuslt_flaky_jdk1.8</li> <li>gzoltar_reuslt_noFlaky_jdk1.7</li> <li>gzoltar_reuslt_noFlaky_jdk1.8</li> </ul> </li> <li><strong>RQ4:</strong> This folder contains repair results of different APR tools. For the 10 tools from <a href="https://github.com/program-repair/RepairThemAll">RepairThemAll</a> framework, each result is named in format <strong><em>validation_result_[APR_name].txt</em></strong> which indicates whether the previous patch can be generated this time. For <a href="https://github.com/SerVal-DTF/TBar">TBar</a>, we release all the generated patches including <em>fixed</em> and <em>partially fixed</em>. <ul> <li>RepairThemAll</li> <li>TBar</li> </ul> </li> </ul> </li> </ul>
Replication package for our paper "Search-based Automated Program Repair of CPS Controllers Modeled in Simulink-Stateflow"
Open the record for dataset details and reuse information.
On the acceptance by code reviewers of candidate security patches suggested by Automated Program Repair tools - Dataset
<p>Dataset of the empirical experiment presented in the paper On the acceptance by code reviewers of candidate security patches suggested by Automated Program Repair tools. The dataset includes the participants' responses regarding their background, and the responses of the tasks from the experiment. </p>
Automated Program Repair Cardumen Results
Open the record for dataset details and reuse information.
Automated Program Repair Term Project Cardumen Results Part 2
Open the record for dataset details and reuse information.
Automated Program Repair via Conversation: Fixing 162 out of 337 Bugs for $0.42 Each using ChatGPT
Open the record for dataset details and reuse information.
Automated Program Repair in the Era of Large Pre-trained Language Models
<p>Code used for the paper along with the generated outputs</p>
Artifact of the paper: On the Effectiveness of Automated Program Repair: An Extensive Study
<p>The pre-trained Edits model.</p>
Artifact of paper: On the Effectiveness of Automated Program Repair: An Extensive Study
<p>This is the artifact repo for paper: <strong>On the Effectiveness of Automated Program Repair: An Extensive Study</strong>. This repo contains large files, e.g., patches and compilation logs for this study.</p>
Less Training, More Repairing Please: Revisiting Automated Program Repair via Zero-shot Learning
<p>Code used for the paper along with the generated outputs</p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.