Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
144
datasets available to search
ShareScore release 0.9.0
Dataset results
144 results for “Test generation”
Empirical Study on Test Generation Using GitHub Copilot --- Replication Package
<p>This replication package contains the data and scripts used in the "Empirical Study on Test Generation Using GitHub Copilot" thesis. </p>
Paper data for DeepManeuver: Adversarial Test Generation for Trajectory Manipulation of Autonomous Vehicles
<p>This repo contains the study and appendix data for "DeepManeuver: Adversarial Test Generation for Trajectory Manipulation of Autonomous Vehicles". DOI 10.1109/TSE.2023.3301443.</p>
Replication Package for "Improving the Readability of Generated Tests Using GPT-4 and ChatGPT Code Interpreter"
<p>While automated test generation can decrease the human burden associated with testing, it does not eliminate this burden. Humans must still work with generated test cases to interpret testing results, debug the code, build and maintain a comprehensive test suite, and many other tasks. Therefore, a major challenge with automated test generation is understandability of generated test test cases. </p> <p>Large language models (LLMs), machine learning models trained on massive corpora of textual data - including both natural language and programming languages - are an emerging technology with great potential for performing language-related predictive tasks such as translation, summarization, and decision support. </p> <p>In this study, we are exploring the capabilities of LLMs with regard to improving test case understandability.</p> <p>This package contains the data produced during this exploration:</p> <ul> <li>The examples directory contains the three case studies we tested our transformation process on: <ul> <li>queue_example: Tests of a basic queue data structure</li> <li>httpie_sessions: Tests of the sessions module from the httpie project. </li> <li>string_utils_validation: Tests of the validation module from the python-string-utils project.</li> <li>Each directory contains the modules-under-test, the original test cases generated by Pynguin, and the transformed test cases. </li> <li>Two trials were performed per case example of the transformation technique to assess the impact of different results from the LLM.</li> </ul> </li> <li>The survey directory contains the survey that was sent to assess the impact of the transformation on test readability. <ul> <li>survey.pdf contains the survey questions.</li> <li>responses.xlsx contains the survey results.</li> </ul> </li> </ul>
Search-based Software Testing Driven by Automatically Generated and Manually Defined Fitness Functions - Dataset and Results
<p>This dataset contains the replication package for the paper: Federico Formica, Tony Fan, and Claudio Menghi. 2023. "Search-based Software Testing Driven by Automatically Generated and Manually Defined Fitness Functions".</p> <p>The dataset contains:</p> <ul> <li>The models used in the evaluation section.</li> <li>All the results obtained by running Athena-S on the models.</li> <li>The scripts that automatically analyze the results file and produce the tables and figures used in the paper.</li> </ul>
Automated Isolation for White-box Test Generation
<p>We present an approach for automated isolation in white-box test generation. This publication contains the data and the corresponding analysis script written in R.</p>
Test Suites from Test-Generation Tools (Test-Comp 2020)
<p>This archive contains the test suites that were generated during the<br> 2nd Competition on Software Testing (Test-Comp 2020)<br> <a href="https://test-comp.sosy-lab.org/2020/">https://test-comp.sosy-lab.org/2020/</a></p> <p>The competition was run by Dirk Beyer, LMU Munich, Germany.<br> More information is available in the following article:<br> Dirk Beyer. Second Competition on Software Testing: Test-Comp 2020.<br> In Proceedings of the 23rd International Conference on Fundamental Approaches to Software Engineering<br> (FASE 2020, Dublin, April 28-30), 2020. Springer.<br> <a href="https://doi.org/10.1007/978-3-030-45234-6_25">https://doi.org/10.1007/978-3-030-45234-6_25</a></p> <p>Copyright (C) Dirk Beyer<br> <a href="https://www.sosy-lab.org/people/beyer/">https://www.sosy-lab.org/people/beyer/</a></p> <p>SPDX-License-Identifier: CC-BY-4.0<br> <a href="https://spdx.org/licenses/CC-BY-4.0.html">https://spdx.org/licenses/CC-BY-4.0.html</a></p> <p> </p> <p>Contents:</p> <p>LICENSE.txt specifies the license<br> README.txt this file<br> witnessFileByHash/ This directory contains test suites (witnesses for coverage).<br> Each witness in this directory is stored in a file whose name is the SHA2 256-bit hash<br> of its contents followed by the filename extension .zip.<br> The format of each test suite is described on the format web page:<br> https://gitlab.com/sosy-lab/software/test-format<br> A test suite contains also metadata in order to relate it<br> to the test problem for which it was produced.<br> witnessInfoByHash/ This directory contains for each test suite (witness) in directory witnessFileByHash/<br> a record in JSON format (also using the SHA2 256-bit hash of the witness as filename,<br> with .json as filename extension) that contains the meta data.<br> witnessListByProgramHashJSON/<br> For convenient access to all test suites for a certain program, this directory represents<br> a function that maps each program (via its SHA2256-bit hash) to a set of test suites<br> (JSON records for test suites as described above) that the test tools have produced<br> for that program. For each program for which test suites exist, the directory contains<br> a JSON file (using the SHA2 256-bit hash of the program as filename, with .json as<br> filename extension) that contains all JSON records for test suites for that program.</p> <p>A similar data structure was used by SV-COMP and is described in the following article:<br> Dirk Beyer. A Data Set of Program Invariants and Error Paths.<br> In Proceedings of the 2019 IEEE/ACM 16th International Conference on Mining Software Repositories<br> (MSR 2019, Montreal, Canada, May 26-27), pages 111-115, 2019. IEEE.<br> <a href="https://doi.org/10.1109/MSR.2019.00026">https://doi.org/10.1109/MSR.2019.00026</a></p> <p> </p> <p>Overview over archives from Test-Comp 2020 that are available at Zenodo:</p> <p><a href="https://doi.org/10.5281/zenodo.3678275">https://doi.org/10.5281/zenodo.3678275</a> Witness store (containing the generated test suites)<br> <a href="https://doi.org/10.5281/zenodo.3678264">https://doi.org/10.5281/zenodo.3678264</a> Results (XML result files, log files, file mappings, HTML tables)<br> <a href="https://doi.org/10.5281/zenodo.3678250">https://doi.org/10.5281/zenodo.3678250</a> Test tasks, version testcomp20<br> <a href="https://doi.org/10.5281/zenodo.3574420">https://doi.org/10.5281/zenodo.3574420</a> BenchExec, version 2.5.1</p> <p>All benchmarks were executed<br> for Test-Comp 2020, <a href="https://test-comp.sosy-lab.org/2020/">https://test-comp.sosy-lab.org/2020/</a><br> by Dirk Beyer, LMU Munich<br> based on the components<br> git@github.com:sosy-lab/sv-benchmarks.git testcomp20-0-gd6cd3e5dd4<br> git@gitlab.com:sosy-lab/test-comp/bench-defs.git testcomp19-84-gac76836<br> git@github.com:sosy-lab/benchexec.git 2.5.1-0-gffad635</p> <p><br> Feel free to contact me in case of questions:<br> <a href="https://www.sosy-lab.org/people/beyer/">https://www.sosy-lab.org/people/beyer/</a></p>
Unit test generation for common and uncommon behaviors: dataset
<p>The data used for the 'Unit test generation for common and uncommon behaviors' Master's thesis.</p> <p>The folder structure used in the archives is the same as the one used by the linked evaluation tool.</p> <p>The archives contain the following:</p> <ul> <li>cubtg-eval-1.4.tar.gz: tests generated by EvoSuite</li> <li>stadard.log.tar.gz: EvoSuite log files created during test generation</li> <li>cubtg-eval-1.4-pit-1.6.tar.gz: result of running PIT on the generated tests</li> </ul>
Classifying the Correctness of Generated White-Box Tests: An Exploratory Study
<p>White-box test generator tools rely only on the code under test to select test inputs, and capture the implementation's output as assertions. If there is a fault in the implementation, it could get encoded in the generated tests. Tool evaluations usually measure fault-detection capability using the number of such fault-encoding tests. However, these faults are only detected, if the developer can recognize that the encoded behavior is faulty. We designed an exploratory study to investigate how developers perform in classifying generated white-box test as faulty or correct. We carried out the study in a laboratory setting with 54 graduate students. The tests were generated for two open-source projects with the help of the IntelliTest tool. The performance of the participants were analyzed using binary classification metrics and by coding their observed activities. The results showed that participants incorrectly classified a large number of both fault-encoding and correct tests (with median misclassification rate 33% and 25% respectively). Thus the real fault-detection capability of test generators could be much lower than typically reported, and we suggest to take this human factor into account when evaluating generated white-box tests.<br> <br> This material contains the dataset and the recorded videos of the study.</p>
[Supplementary material] Test Oracle Generation for REST APIs
<p>This is the supplementary material of the paper entitled <em>Test Oracle Generation for REST APIs</em></p> <p>To use the most up-to-date version of AGORA+, please refer to the official GitHub repository: https://github.com/isa-group/Beet</p> <p>The Ubuntu Virtual Machine containing all the projects configured can be found here: https://figshare.com/articles/dataset/AGORA_Virtual_Machine/28266926</p> <p> </p> <h1>How to Open the Replication Package (Multi-part ZIP Archive)</h1> <p><span lang="EN-US">The replication package is distributed as a multi-part ZIP archive. This is a standard way to split a large ZIP file into smaller chunks for easier uploading and downloading. The archive consists of the following files:</span></p> <ul> <li><span lang="EN-US">ReplicationPackage.z01 </span></li> <li><span lang="EN-US">ReplicationPackage.z02</span></li> <li><span lang="EN-US">ReplicationPackage.z03 </span></li> <li><span lang="EN-US">ReplicationPackage.z04 </span></li> <li><span lang="EN-US">ReplicationPackage.z05</span></li> <li><span lang="EN-US">ReplicationPackage.zip </span></li> </ul> <p><span lang="EN-US">These files <strong>must be in the same folder</strong> before unzipping.</span></p> <p><span lang="EN-US">Below are step-by-step instructions for <strong>Windows</strong> and <strong>Linux</strong> users.</span></p> <h1><span lang="EN-US">For Windows Users</span></h1> <h2><span lang="EN-US">Option 1: Using 7-Zip (Recommended)</span></h2> <ol> <li><span lang="EN-US">Download and install 7-Zip: <a href="https://www.7-zip.org/">https://www.7-zip.org/</a></span></li> <li><span lang="EN-US">Place all the files (.z01, .z02, ..., .z05, and .zip) in the <strong>same folder</strong>.</span></li> <li><span lang="EN-US">Right-click on ReplicationPackage.zip.</span></li> <li><span lang="EN-US">Choose <strong>7-Zip -></strong></span><strong><span lang="EN-US"> Extract Here</span></strong><span lang="EN-US"> or <strong>Extract to ReplicationPackage\</strong>.</span></li> </ol> <p><span lang="EN-US">7-Zip will automatically combine the parts and extract the full content.</span></p> <h2>Option 2: Using WinRAR</h2> <ol> <li><span lang="EN-US">Download and install <strong>WinRAR</strong>: <a href="https://www.rarlab.com/">https://www.rarlab.com/</a></span></li> <li><span lang="EN-US">Ensure all parts are in the same directory.</span></li> <li><span lang="EN-US">Right-click on ReplicationPackage.zip.</span></li> <li><span lang="EN-US">Select <strong>Extract Here</strong> or <strong>Extract to ReplicationPackage\</strong>.</span></li> </ol> <h1><span lang="EN-US">For Linux Users</span></h1> <ol> <li><span lang="EN-US">Make sure all the parts are in the same directory.</span></li> <li><span lang="EN-US">Install p7zip if not already installed:</span></li> </ol> <p><span lang="EN-US"> sudo apt install p7zip-full</span></p> <p><span lang="EN-US"> 3. Navigate to the folder containing the files:</span></p> <p><span lang="EN-US"> cd /path/to/your/downloaded/files</span></p> <p> 4. Run the following command:</p> <p> 7z x ReplicationPackage.zip</p> <p><span lang="EN-US">This will extract the full archive by automatically reading the .z01, .z02, etc.</span></p> <h1><span lang="EN-US">NOTES</span></h1> <ul> <li><strong><span lang="EN-US">Do not</span></strong><span lang="EN-US"> try to unzip the .z01, .z02, etc., individually. Only unzip the main .zip file (the one with the .zip extension).</span></li> <li><span lang="EN-US">If you see an error like <em>"not a valid zip file"</em>, it usually means:</span></li> <ul> <li><span lang="EN-US">Not all parts were downloaded correctly.</span></li> <li><span lang="EN-US">The files are not in the same folder.</span></li> <li><span lang="EN-US">The file names were modified (e.g., by the browser).</span></li> </ul> </ul>
PhiPiPi Dataset for our paper "Event Generation and Consistence Test for Physics with Sliced Wasserstein Distance"
<p>The PhiPiPi dataset is used to train and test our fast event simulation model for our paper "Event Generation and Consistence Test for Physics with Sliced Wasserstein Distance" in <a href="https://arxiv.org/abs/2310.17897">https://arxiv.org/abs/2310.17897</a>. The dataset is generated by BES III simulation framework to simulate phase-space Monte Carlo events, focusing on the production of psi(2S) in electron-positron annihilation. This intricate process entails the psi(2S) decay into a trio of particles: phi, pi+, and pi-, with the phi particle subsequently decaying into a K+ and K- pair.</p><p>We have meticulously partitioned the data into distinct sets for training, validation, and testing to facilitate a robust machine learning workflow. Each set is systematically saved in the convenient npy format for ease of access and integration.</p><p>The dataset contains the four-momentum components (px, py, pz, E) of the final-state particles: K+, K-, pi+, and pi-. This comprehensive inclusion ensures a detailed representation of the kinematic properties of the particles for accurate model training.</p><p>Researchers and practitioners are invited to use this dataset by incorporating it into our public code repository: <a href="https://github.com/caihao/SWD-EvtGen">https://github.com/caihao/SWD-EvtGen</a>. Doing so will enable the training of our fast event simulation model, enhancing the fidelity and efficiency of high energy physics simulations.</p>
Choosing the Right Test Generation Tool: A Guide for Software Practitioners
<p>Context: Software testing tools are essential during the processes of developing and maintaining quality digital applications in a scenario where they become increasingly complex. Problem: The market offers hundreds of test generation tools with different approaches and functionalities, it is important that developers and designers have knowledge of which tools are best suited to their work in order to avoid unnecessary expenses and easily avoidable frustrations. Solution: to identify the main tools available on the market in order to highlight their advantages and disadvantages, thus presenting a reference guide that will help developers choose<br>the most appropriate tool for each test situation. </p><p>Method In this work: a multivocal literature review has been carried out to identify works related to test generation tools, also identifying the most cited tools in white and gray literature. Furthermore, a survey was developed and conducted with professionals working in different areas of software development, in order to ascertain their familiarity with the testing tools identified in the white and gray literature, as well as the advantages and challenges that have more weight when deciding which tool to adopt.</p><p>Summarization of Results: The results showed that most of the tools found in the white literature, such as Monkey and Dynodroid, are not very well known and used by the participants and that they were more familiar with tools more mentioned in the gray literature, such as Postman and Selenium. In addition, a list of characteristics most desired and avoided by participants when needing to choose a new tool for a project was obtained. Impact on the IS field: This work contributes to IS field by providing a reference guide where practitioners can identify the most recommended tools according to a specific scenario, and then use a comparative table in order to evaluate the most appropriate tool for the project among those recommended. The Guide can be used by any testing practitioner.</p>
Learning by Viewing: Generating Test Inputs for Games by Integrating Human Gameplay Traces in Neuroevolution
<p>Replication package for the paper "Learning by Viewing: Generating Test Inputs for Games by Integrating Human Gameplay Traces in Neuroevolution" </p><p> </p><p>Although automated test generation is common in many programming domains, games still challenge test generators due to their heavy randomisation and hard-to-reach program states. Neuroevolution combined with search-based software testing principles has been shown to be a promising approach for testing games, but the co-evolutionary search for optimal network topologies and weights involves unreasonably long search durations. Humans, on the other hand, tend to be quick in picking up basic gameplay. In this paper, we therefore aim to improve the evolutionary search for game input generators by integrating knowledge about human gameplay behaviour. To this end, we propose a novel way of systematically recording human gameplay traces, and integrating these traces into the evolutionary search for networks using traditional gradient descent as a mutation operator. Experiments conducted on eight diverse Scratch games demonstrate that the proposed approach reduces the required search time from five hours down to only 30 minutes on average.</p>
Results of the mechanical characterization by tensile tests and the in vitro cell-biomaterial interaction analyses by WST-1 and Live/Dead of 3D-printed scaffolds generated by PLA, PCL, FF, FD and GelMA
<p>Dataset containing the quantitative results of the mechanical characterization, and the in vitro cell-biomaterial interaction analyses with neural cells after 72 hours and 7 days of cell culture of the following 3D printed scaffolds:</p> <ul> <li>Polylactic acid (PLA)</li> <li>Polycaprolactone (PCL)</li> <li>Conductive Filaflex (FF)</li> <li>Flexdym (FD)</li> <li>GelMA (G)</li> </ul> <p>The Live/Dead results were quantified using ImageJ software to determine area fractions corresponding to live (green) and dead (red) cells.</p>
Test models and test results to evaluate CAD assembly modules capabilities to generate component interfaces
<p>Set of 3D CAD assembly models in STEP AP 203 format.</p> <p>Assembly test models are devoted to evaluations of interfaces between components. The interfaces can be of type surface, rectilinear contacts, circular contacts, or point contacts.</p> <p>Test results obtained from some commercially available CAD assembly modules are given as a set of tables organized in accordancce with contact categories (surface, rectilinear, circular, point).</p> <p>The content and use of the test models are described into the pdf document: Test models and test results to evaluate CAD assembly modules capabilities to generate component interfaces.</p> <p> </p>
dataset of test trial with random pattern generator
<p>test session of a mems based random caustic-pattern generator. In this session we collected the experimental patterns that have helped us understand the physics and functioning of the device that we used, moreover these patterns are being used to train a neural network that will aid us to perform computational ghost imaging experiments with electrons.</p>
Test Suites from Test-Generation Tools (Test-Comp 2022)
<p>Test-Comp 2022</p> <p>Test Suites</p> <p>This file describes the contents of an archive of the 4th Competition on Software Testing (Test-Comp 2022).<br> <a href="https://test-comp.sosy-lab.org/2022/">https://test-comp.sosy-lab.org/2022/</a></p> <p>The competition was run by Dirk Beyer, LMU Munich, Germany.<br> More information is available in the following article:<br> Dirk Beyer. <em>Advances in Automatic Software Testing: Test-Comp 2022.</em> In Proceedings of the 25th International Conference on Fundamental Approaches to Software Engineering (FASE 2022, Munich, April 2 - 7), 2021. Springer.</p> <p>Copyright (C) Dirk Beyer<br> <a href="https://www.sosy-lab.org/people/beyer/">https://www.sosy-lab.org/people/beyer/</a></p> <p>SPDX-License-Identifier: CC-BY-4.0<br> <a href="https://spdx.org/licenses/CC-BY-4.0.html">https://spdx.org/licenses/CC-BY-4.0.html</a></p> <p>Contents</p> <ul> <li><code>LICENSE.txt</code>: specifies the license</li> <li><code>README.txt</code>: this file</li> <li><code>witnessFileByHash/</code>: This directory contains test suites (witnesses for coverage). Each test witness in this directory is stored in a file whose name is the SHA2 256-bit hash of its contents followed by the filename extension .zip. The format of each test suite is described on the format web page: <a href="https://gitlab.com/sosy-lab/software/test-format">https://gitlab.com/sosy-lab/software/test-format</a> A test suite contains also metadata in order to relate it to the test task for which it was produced.</li> <li><code>witnessInfoByHash/</code>: This directory contains for each test suite (witness) in directory witnessFileByHash/ a record in JSON format (also using the SHA2 256-bit hash of the witness as filename, with .json as filename extension) that contains the meta data.</li> <li><code>witnessListByProgramHashJSON/</code>: For convenient access to all test suites for a certain program, this directory represents a function that maps each program (via its SHA2256-bit hash) to a set of test suites (JSON records for test suites as described above) that the test-generation tools have produced for that program. For each program for which test suites exist, the directory contains a JSON file (using the SHA2 256-bit hash of the program as filename, with .json as filename extension) that contains all JSON records for test suites for that program.</li> </ul> <p>A similar data structure was used by SV-COMP and is described in the following article:<br> Dirk Beyer. <em>A Data Set of Program Invariants and Error Paths.</em> In Proceedings of the 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR 2019, Montreal, Canada, May 26-27), pages 111-115, 2019. IEEE.<br> <a href="https://doi.org/10.1109/MSR.2019.00026">https://doi.org/10.1109/MSR.2019.00026</a></p> <p>Other Archives</p> <p>Overview over archives from Test-Comp 2022 that are available at Zenodo:</p> <ul> <li><a href="https://doi.org/10.5281/zenodo.5831010">https://doi.org/10.5281/zenodo.5831010</a> Test Suites from Test-Comp 2022 Test-Generation Tools. Witness store (containing the generated test suites)</li> <li><a href="https://doi.org/10.5281/zenodo.5831012">https://doi.org/10.5281/zenodo.5831012</a> Results of the 4th Intl. Competition on Software Testing (Test-Comp 2022). Results (XML result files, log files, file mappings, HTML tables)</li> <li><a href="https://doi.org/10.5281/zenodo.5831003">https://doi.org/10.5281/zenodo.5831003</a> SV-Benchmarks: Benchmark Set of SV-COMP 2022 and Test-Comp 2022. Test tasks, version testcomp22</li> <li><a href="https://doi.org/10.5281/zenodo.5720267">https://doi.org/10.5281/zenodo.5720267</a> BenchExec, version 3.10. Benchmarking framework</li> </ul> <p>All benchmarks were executed for Test-Comp 2022 <a href="https://test-comp.sosy-lab.org/2022/">https://test-comp.sosy-lab.org/2022/</a><br> by Dirk Beyer, LMU Munich, based on the following components:</p> <ul> <li><a href="https://gitlab.com/sosy-lab/test-comp/archives-2022">https://gitlab.com/sosy-lab/test-comp/archives-2022</a> testcomp22 c211efbc</li> <li><a href="https://gitlab.com/sosy-lab/benchmarking/sv-benchmarks">https://gitlab.com/sosy-lab/benchmarking/sv-benchmarks</a> testcomp22 ad265d07</li> <li><a href="https://gitlab.com/sosy-lab/test-comp/bench-defs">https://gitlab.com/sosy-lab/test-comp/bench-defs</a> testcomp22 93f581b9</li> <li><a href="https://gitlab.com/sosy-lab/software/benchexec">https://gitlab.com/sosy-lab/software/benchexec</a> 3.10 4e8716bd</li> <li><a href="https://gitlab.com/sosy-lab/benchmarking/competition-scripts">https://gitlab.com/sosy-lab/benchmarking/competition-scripts</a> testcomp22 2022c698</li> <li><a href="https://gitlab.com/sosy-lab/test-comp/test-format">https://gitlab.com/sosy-lab/test-comp/test-format</a> testcomp22 2895be41</li> </ul> <p>Contact</p> <p>Feel free to contact me in case of questions: <a href="https://www.sosy-lab.org/people/beyer/">https://www.sosy-lab.org/people/beyer/</a></p>
TRAVEL: A Dataset with Toolchains for Test Generation and Regression Testing of Self-driving Cars Software
<p><strong>Introduction</strong></p> <p>This repository hosts the Testing Roads for Autonomous VEhicLes (TRAVEL) dataset. TRAVEL is an extensive collection of virtual roads that have been used for testing lane assist/keeping systems (i.e., driving agents) and data from their execution in state of the art, physically accurate driving simulator, called <a href="https://www.beamng.tech/">BeamNG.tech</a>. Virtual roads consist of sequences of <strong>road points</strong> interpolated using Cubic splines.</p> <p>Along with the data, this repository contains instructions on how to install the tooling necessary to generate new data (i.e., test cases) and analyze them in the context of test regression. We focus on test selection and test prioritization, given their importance for developing high-quality software following the DevOps paradigms.</p> <p>This dataset builds on top of our previous work in this area, including work on</p> <ul> <li>test generation (e.g., <a href="https://doi.org/10.1145/3293882.3330566">AsFault</a>, <a href="https://doi.org/10.1145/3368089.3409730">DeepJanus</a>, and <a href="https://doi.org/10.1145/3460319.3464811">DeepHyperion</a>) and the SBST CPS tool competition (<a href="https://doi.org/10.1109/SBST52555.2021.00011">SBST2021</a>),</li> <li>test selection: <a href="https://arxiv.org/abs/2111.04666">SDC-Scissor</a> and related <a href="https://github.com/ChristianBirchler/sdc-scissor">tool</a></li> <li>test prioritization: <a href="https://arxiv.org/abs/2107.09614">automated test cases prioritization work for SDCs</a>.</li> </ul> <p><strong>Dataset Overview</strong></p> <p>The TRAVEL dataset is available under the <code>data</code> folder and is organized as a set of <em>experiments</em> folders. Each of these folders is generated by running the <code>test-generator</code> (see below) and contains the configuration used for generating the data (<code>experiment_description.csv</code>), various statistics on generated tests (<code>generation_stats.csv</code>) and found faults (<code>oob_stats.csv</code>). Additionally, the folders contain the raw test cases generated and executed during each experiment (<code>test.<TEST_ID>.json</code>).</p> <p>The following sections describe what each of those files contains.</p> <p>Experiment Description</p> <p>The <code>experiment_description.csv</code> contains the settings used to generate the data, including:</p> <ul> <li><strong>Time budget</strong>. The overall generation budget in hours. This budget includes both the time to generate and execute the tests as driving simulations.</li> <li><strong>The size of the map</strong>. The size of the squared map defines the boundaries inside which the virtual roads develop in meters.</li> <li><strong>The test subject</strong>. The driving agent that implements the lane-keeping system under test. The TRAVEL dataset contains data generated testing the BeamNG.AI and the end-to-end Dave2 systems.</li> <li><strong>The test generator</strong>. The algorithm that generated the test cases. The TRAVEL dataset contains data obtained using various algorithms, ranging from naive and advanced random generators to complex evolutionary algorithms, for generating tests.</li> <li><strong>The speed limit</strong>. The maximum speed at which the driving agent under test can travel.</li> <li><strong>Out of Bound (OOB) tolerance</strong>. The test cases' oracle that defines the tolerable amount of the ego-car that can lie outside the lane boundaries. This parameter ranges between 0.0 and 1.0. In the former case, a test failure triggers as soon as any part of the ego-vehicle goes out of the lane boundary; in the latter case, a test failure triggers only if the entire body of the ego-car falls outside the lane.</li> </ul> <p>Experiment Statistics</p> <p>The <code>generation_stats.csv</code> contains statistics about the test generation, including:</p> <ul> <li><strong>Total number of generated tests</strong>. The number of tests generated during an experiment. This number is broken down into the number of <strong>valid tests</strong> and <strong>invalid tests</strong>. Valid tests contain virtual roads that do not self-intersect and contain turns that are not too sharp.</li> <li><strong>Test outcome</strong>. The test outcome contains the number of <strong>passed tests</strong>, <strong>failed tests</strong>, and <strong>test in error</strong>. Passed and failed tests are defined by the OOB Tolerance and an additional (implicit) oracle that checks whether the ego-car is moving or standing. Tests that did not pass because of other errors (e.g., the simulator crashed) are reported in a separated category.</li> </ul> <p>The TRAVEL dataset also contains statistics about the failed tests, including the overall number of failed tests (<strong>total oob</strong>) and its breakdown into OOB that happened while driving <strong>left</strong> or <strong>right</strong>. Further statistics about the diversity (i.e., <strong>sparseness</strong>) of the failures are also reported.</p> <p>Test Cases and Executions</p> <p>Each <code>test.<TEST_ID>.json</code> contains information about a test case and, if the test case is valid, the data observed during its execution as driving simulation.</p> <p>The data about the test case definition include:</p> <ul> <li><strong>The road points</strong>. The list of points in a 2D space that identifies the center of the virtual road, and their interpolation using cubic splines (<strong>interpolated_points</strong>)</li> <li><strong>The test ID</strong>. The unique identifier of the test in the experiment.</li> <li><strong>Validity flag and explanation</strong>. A flag that indicates whether the test is valid or not, and a brief message describing why the test is not considered valid (e.g., <code>the road contains sharp turns</code> or <code>the road self intersects</code>)</li> </ul> <p>The test data are organized according to the following JSON Schema and can be interpreted as <code>RoadTest</code> objects provided by the <a href="https://github.com/se2p/tool-competition-av/blob/main/code_pipeline/tests_generation.py">tests_generation.py</a> module.</p> <pre><code>{ "type": "object", "properties": { "id": { "type": "integer" }, "is_valid": { "type": "boolean" }, "validation_message": { "type": "string" }, "road_points": { §\label{line:road-points}§ "type": "array", "items": { "$ref": "schemas/pair" }, }, "interpolated_points": { §\label{line:interpolated-points}§ "type": "array", "items": { "$ref": "schemas/pair" }, }, "test_outcome": { "type": "string" }, §\label{line:test-outcome}§ "description": { "type": "string" }, "execution_data": { "type": "array", "items": { "$ref" : "schemas/simulationdata" } } }, "required": [ "id", "is_valid", "validation_message", "road_points", "interpolated_points" ] } </code></pre> <p>Finally, the <strong>execution data</strong> contain a list of timestamped state information recorded by the driving simulation. State information is collected at constant frequency and includes absolute position, rotation, and velocity of the ego-car, its speed in Km/h, and control inputs from the driving agent (steering, throttle, and braking). Additionally, execution data contain OOB-related data, such as the lateral distance between the car and the lane center and the OOB percentage (i.e., how much the car is outside the lane).</p> <p>The simulation data adhere to the following (simplified) JSON Schema and can be interpreted as Python objects using the <a href="https://github.com/se2p/tool-competition-av/blob/main/self_driving/simulation_data.py">simulation_data.py</a> module.</p> <pre><code>{ "$id": "schemas/simulationdata", "type": "object", "properties": { "timer" : { "type": "number" }, "pos" : { "type": "array", "items":{ "$ref" : "schemas/triple" } } "vel" : { "type": "array", "items":{ "$ref" : "schemas/triple" } } "vel_kmh" : { "type": "number" }, "steering" : { "type": "number" }, "brake" : { "type": "number" }, "throttle" : { "type": "number" }, "is_oob" : { "type": "number" }, "oob_percentage" : { "type": "number" } §\label{line:oob-percentage}§ }, "required": [ "timer", "pos", "vel", "vel_kmh", "steering", "brake", "throttle", "is_oob", "oob_percentage" ] } </code></pre> <p><strong>Dataset Content</strong></p> <p>The TRAVEL dataset is a lively initiative so the content of the dataset is subject to change. Currently, the dataset contains the data collected during the SBST CPS tool competition, and data collected in the context of our recent work on test selection (<a href="https://arxiv.org/abs/2111.04666">SDC-Scissor work</a> and <a href="https://github.com/ChristianBirchler/sdc-scissor">tool</a>) and test prioritization (<a href="https://arxiv.org/abs/2107.09614">automated test cases prioritization work for SDCs</a>).</p> <p>SBST CPS Tool Competition Data</p> <p>The data collected during the SBST CPS tool competition are stored inside <code>data/competition.tar.gz</code>. The file contains the test cases generated by <a href="https://github.com/mahshidhelali/tool-competition-av">Deeper</a>, <a href="https://github.com/ERATOMMSD/frenetic-sbst21">Frenetic</a>, <a href="https://github.com/TayYim/adafrenetic-sbst22">AdaFrenetic</a>, and <a href="https://github.com/alessiogambi/swat-sbst2021">Swat</a>, the <strong>open-source</strong> test generators submitted to the competition and executed against BeamNG.AI with an aggression factor of 0.7 (i.e., conservative driver).</p> <table> <thead> <tr> <th>Name</th> <th>Map Size (m x m)</th> <th>Max Speed (Km/h)</th> <th>Budget (h)</th> <th>OOB Tolerance (%)</th> <th>Test Subject</th> </tr> </thead> <tbody> <tr> <td>DEFAULT</td> <td>200 × 200</td> <td>120</td> <td>5 (real time)</td> <td>0.95</td> <td>BeamNG.AI - 0.7</td> </tr> <tr> <td>SBST</td> <td>200 × 200</td> <td>70</td> <td>2 (real time)</td> <td>0.5</td> <td>BeamNG.AI - 0.7</td> </tr> </tbody> </table> <p>Specifically, the TRAVEL dataset contains 8 repetitions for each of the above configurations for each test generator totaling 64 experiments.</p> <p>SDC Scissor</p> <p>With SDC-Scissor we collected data based on the <a href="https://github.com/ERATOMMSD/frenetic-sbst21">Frenetic</a> test generator. The data is stored inside <code>data/sdc-scissor.tar.gz</code>. The following table summarizes the used parameters.</p> <table> <thead> <tr> <th>Name</th> <th>Map Size (m x m)</th> <th>Max Speed (Km/h)</th> <th>Budget (h)</th> <th>OOB Tolerance (%)</th> <th>Test Subject</th> </tr> </thead> <tbody> <tr> <td>SDC-SCISSOR</td> <td>200 × 200</td> <td>120</td> <td>16 (real time)</td> <td>0.5</td> <td>BeamNG.AI - 1.5</td> </tr> </tbody> </table> <p>The dataset contains 9 experiments with the above configuration. For generating your own data with SDC-Scissor follow the instructions in its <a href="https://github.com/ChristianBirchler/sdc-scissor">repository</a>.</p> <p><strong>Dataset Statistics</strong></p> <p>Here is an overview of the TRAVEL dataset: generated tests, executed tests, and faults found by all the test generators grouped by experiment configuration. Some 25,845 test cases are generated by running 4 test generators 8 times in 2 configurations using the SBST CPS Tool Competition code pipeline (SBST in the table). We ran the test generators for 5 hours, allowing the ego-car a generous speed limit (120 Km/h) and defining a high OOB tolerance (i.e., 0.95), and we also ran the test generators using a smaller generation budget (i.e., 2 hours) and speed limit (i.e., 70 Km/h) while setting the OOB tolerance to a lower value (i.e., 0.85). We also collected some 5, 971 additional tests with SDC-Scissor (SDC-Scissor in the table) by running it 9 times for 16 hours using Frenetic as a test generator and defining a more realistic OOB tolerance (i.e., 0.50).</p> <p><strong>Generating new Data</strong></p> <p>Generating new data, i.e., test cases, can be done using the <a href="https://github.com/se2p/tool-competition-av">SBST CPS Tool Competition pipeline</a> and the driving simulator <a href="https://www.beamng.tech/">BeamNG.tech</a>.</p> <p>Extensive instructions on how to install both software are reported inside the SBST CPS Tool Competition pipeline <a href="https://github.com/se2p/tool-competition-av/blob/main/documentation/README.md">Documentation</a>; therefore, below we only summarize the overall installation process.</p> <p>Installation</p> <ol> <li> <p>Install Python 3.7. (We tested the code using 3.7.9, so we suggest to install that version of Python)</p> </li> <li> <p>Clone the <a href="https://github.com/se2p/tool-competition-av">SBST CPS Tool Competition pipeline</a></p> </li> <li> <p>Request a (free) copy of the driving simulator at <a href="https://register.beamng.tech/">https://register.beamng.tech</a> and, after receiving the registration key file (<code>tech.key</code>) and the download link, download and install the latest distribution of the software (<code>0.24</code>).</p> </li> <li> <p>Create a python virtual environment (<code>python -m venv .venv</code>) inside the root of the SBST CPS Tool Competition pipeline and activate it. Please, be sure to call the virtual environment exactly <code>.venv</code> and place it in the correct position as the scripts to collect new data assume this setup.</p> </li> <li> <p>Install inside the virtual environment all the SBST CPS Tool Competition pipeline requirements (<code>pip install -r requirements.txt</code>).</p> </li> </ol> <p>If you have successfully setup the python environment, activate the virtual environment, and run</p> <pre><code>python.exe competition.py --help </code></pre> <p>and check that the command outputs the <a href="https://github.com/se2p/tool-competition-av/blob/main/documentation/GUIDELINES.md#technical-considerations">usage</a> of the SBST CPS Tool Competition pipeline.</p> <p>Data Collection</p> <p>Collecting new data is as simple as starting the SBST CPS Tool Competition pipeline with suitable parameters for test generation (<code>--time-budget</code>, <code>--module-name</code>, <code>--module-path</code> and <code>--class-name</code>), test executor (<code>--executor</code>), and test definition (<code>--map-size</code>, <code>--speed-limit</code> and <code>--oob-tolerance</code>).</p> <p>To ease the collection of new data, we include the (PowerShell) scripts we used to create the current TRAVEL dataset. Those scripts can be found inside the <code>data-collection</code> folder and are named after the test generator and configuration they implement. For instance, <code>Frenetic-DEFAULT.ps1</code> will execute the Frenetic test generator in the DEFAULT configuration, whereas <code>Deeper-SBST.ps1</code> will execute the Deeper test generation in the SBST configuration. The description of these configurations is reported in the <a href="https://github.com/se2p/msr-travel-dataset#sbst-cps-tool-competition-data">table above</a>).</p> <p>To run those scripts, open a PowerShell, <code>cd</code> to the folder containing the script (<code>data-collection</code>), and invoke the script you want. Assuming you have configured the SBST CPS Tool Competition pipeline as described above, the scripts will start the code pipeline and the BeamNG.tech driving simulator.</p> <p>Once the run is finished, you will find the results under the <code>results</code> folder inside the SBST CPS Tool Competition pipeline project. The results of each experiment are stored in a uniquely named folder that clearly indicates the test generator used to produce them (see <a href="https://github.com/se2p/msr-travel-dataset#dataset-overview">above</a> for the detailed description of the produced results).</p> <p><strong>Data Usage</strong></p> <p>So far, the data contained in the TRAVEL dataset have been used for benchmarking test generators as well as for optimizing regression testing in the context of Self-driving car software.</p> <p>We used the TRAVEL to study the problem of test generation from different perspectives.</p> <p>In the last edition of the <a href="https://doi.org/10.1109/SBST52555.2021.00011">SBST Tool Challenge</a>, we answer the question <em>"How efficient and effective are the existing test generators for SDCs?"</em>. We assess test generation efficiency by counting how many tests are produced by each test generator within a given time budget. To assess test generation effectiveness, we count how many of the generated tests are valid and invalid.</p> <p>In a <a href="https://doi.org/10.1145/3460319.3464811">recent study</a>, we propose (along the DeepHyperion test generator) a novel approach to assess SDCs test suites adequacy by measuring the coverage of feature maps that they achieved. To do so, we extract structural and behavioral test case features from the test case descriptions and simulation data by looking at the geometrical properties describing the roads and the physical quantities describing the ego-car behavior (e.g., position, speed).</p> <p>We also answer the question <em>"How effective are the resulting tests in finding faults in SDCs?"</em>, by counting how many failures are triggered by the generated tests and how different those failures are (e.g., Left or Right OOBs). Specifically, we propose an approach to measure OOBs similarity by computing the edit distance of the road segments relevant to the corresponding failure.</p> <p><a href="https://github.com/ERATOMMSD/roadsearch">Castellano et al.</a> demonstrated that a dataset such as TRAVEL can be used to answer other interesting questions, such as <em>"Which road representation is the most suitable for generating SDCs test cases?"</em>.</p> <p>In the context of regression testing the TRAVEL dataset is also used to develop tools to select test cases that are likely to fail before executing them simulation (e.g., <a href="https://github.com/ChristianBirchler/sdc-scissor">SDC-Scissor</a>) and also to prioritize them in a single-objective and multi-objective approach by <a href="https://arxiv.org/abs/2107.09614">SDC-Prioritizer</a>.</p> <p>With SDC-Scissor we can select test cases that are more likely to fail. This allows us to answer the following research questions like "To what extent is it possible to identify safe and unsafe test scenarios for SDCs before executing them?" or "Does SDC-Scissor improve the cost-effectiveness of simulation-based testing of SDCs?" <a href="https://arxiv.org/abs/2111.04666">Khatiri et al.</a>. SDC-Scissor achieves a F1-score (between 47% and 90%) compared to a baseline in identifying failing tests and reduced the time spent running uninformative tests by 107% and 170%. <a href="https://christianbirchler.github.io/sdc_scissor_demonstration_paper.pdf">(Birchler et al.)</a></p> <p>Regarding test prioritization, the TRAVEL dataset allows us to answer the following questions: "To what extent is it possible to apply test prioritization strategies to prioritize the execution of safe and unsafe test scenarios for SDCs?", “What is the cost-effectiveness of SDC-Prioritizer compared to baseline approaches?” and “What is the overhead introduced by it?”. To address the former question, we compared the test prioritization approaches in terms of fault detection rate (i.e., how fast faults are detected during the test execution process) using Cost cognizant Average Percentage of Fault Detection (APFD𝑐) (<a href="https://doi.org/10.1145/2771783.2771788">Epitropakis et al.</a>, <a href="https://doi.org/10.1109/ICSM.1999.792604">Rothermel et al.</a>) and concluded that SDC-Prioritizer (with 82.5% APFD𝑐) significantly outperformed the baseline random and greedy test prioritization approaches <a href="https://arxiv.org/abs/2107.09614">Birchler et al.</a>. For the latter research question, we compared the time required by SDC-Prioritizer to sort tests against the time needed to run all of the tests and showed that, on average, SDC-Prioritizer needs less than 13 minutes to perform the test prioritization while running the SDC test suites takes between 16 to 106 hours. As explained in Section 3.2.3, SDC-Prioritizer uses the road features shared with TRAVEL to guide the search process towards generating test orders with the high diversity in the road shapes. The selection of best features for test prioritization was performed with the Principal component analysis (PCA) <a href="https://arxiv.org/abs/2107.09614">Birchler et al.</a>.</p>
Test Image Set for Pyramid Generation
<p>A set of 7 images for a test of pyramid generation of the OMERO.server</p> <p>Varying sizes and file formats</p>
Artefact to our paper "An Empirical Study of Automated Unit Test Generation for Python"
<p><strong>Artefact for “An Empirical Study of Automated Unit Test Generation for Python”</strong></p> <p>Together with our paper “An Empirical Study of Automated Unit Test Generation for Python”, we provide this artefact for future use.</p> <p><strong>Pynguin Version</strong></p> <p>We used Pynguin 0.25.2 for our experiments. The releases of Pynguin are archieved by Zenodo, too. Pynguin 0.25.2 is available under DOI <a href="https://doi.org/10.5281/zenodo.6836225">10.5281/zenodo.6836225</a>.</p> <p><strong>Preparation of the Environment</strong></p> <p>We use the <a href="https://python-poetry.org"><code>poetry</code></a> dependency-management tool to manage all dependencies for this artefact. Install this tool if you have not done yet. Furthermore, let <code>poetry</code> create a virtual environment for the experiment by execution <code>poetry install</code>.</p> <p><strong>Execution of the Experiment</strong></p> <p>The execution scripts make several assumptions that are based on our infrastructure. We maintain a SLURM cluster infrastructure that defines different <code>constraints</code> for different machines.</p> <p>Furthermore, we assume some paths to be present: we assume every computing machine to have writable mount points at <code>/local/${USER}</code> and <code>/local/hdd/${USER}</code>. On our machines, both are mount points on the local hard disk/SSD of the computing machines. Additionally, we have a shared mount <code>/scratch/${USER}</code>, which is mounted via NFS from a central file server. This mount point is also mounted on all computing machines.</p> <p>We assume the created and packaged Docker image to be located at <code>/scratch/lukasczy/pynguin.tar</code>. You can change this path by editing the XML files. These XML files contain the basic definitions of the jobs: they specify the SLURM constraint, the version of the Pynguin Docker container, the used Pynguin configurations as well as the modules used for the experiments. These modules have to reside under <code>projects</code>, as they come with this artefact.</p> <p>The Python script <code>execution.py</code> generates the actual run scripts from the XML file. It generates all scripts necessary to run a SLURM array job consisting of all runs for the experiment. Further general settings for the SLURM array job are present in this file.</p> <p>The Bash script <code>run_experiment.sh</code> executes the full execution pipeline; one has to specify the variable <code>EXPERIMENT_NAME</code> to match the name of the respective XML file who's defined experiment shall be executed.</p> <p><em>Important:</em> Executing the full experiment can take several days, depending on your computing infrastructure! We do therefore provide the raw result CSVs for further inspection.</p> <p><strong>Data Analysis</strong></p> <p>All raw data resides in the <code>data</code> folder:</p> <ul> <li><code>loc_data.csv</code> contains all information about the lines of code in each module. This file was created using the <code>extract_locs_and_types.py</code> script in the root folder. Please note that executing this script requires that the <code>cloc</code> utility tool is installed on your system's path.</li> <li><code>types.csv</code> and <code>types_per_module.csv</code> contain type information extract from the modules at different granularity level. They are also generated using the aforementioned script.</li> <li><code>results-assertion.csv.xz</code> contains the raw results from the experiment for RQ3 that evaluates the effectiveness of the assertions.</li> <li><code>results.csv.xz</code> contains the raw results from the experiment for RQ1 and RQ2.</li> </ul> <p>We provide the Jupyter Notebook that generated the plots, tables, and various LaTeX macros in the <code>notebooks</code> folder. Please note that if you want to reexecute this notebook, you might have to change the <code>PAPER_EXPORT_PATH</code> constant in cell <code>[2]</code> to a suitable location on your machine. Executing this notebook requires a installation of a TeX system to be available on your system because the plots are generated using <code>pdflatex</code> and <code>matplotlib</code>s pgf backend.</p> <p><strong>Further Data</strong></p> <p>The folder <code>projects</code> contains all the projects in the versions stated in our paper. The folder <code>run-logs</code> contains all the run logs from our experiment executions.</p>
Artefact for "Automated Test Generation for Scratch Programs"
<p><strong>Automated Test Generation for Scratch Programs</strong></p> <p>Replication package for our study on <a href="https://arxiv.org/pdf/2202.06274.pdf">Automated Test Generation for Scratch Programs</a>.</p> <pre><code>@article{Deiner2022AutomatedTG, title={Automated Test Generation for Scratch Programs}, author={Adina Deiner and Patric Feldmeier and Gordon Fraser and Sebastian Schweikl and Wengran Wang}, journal={ArXiv}, year={2022}, volume={abs/2202.06274} }</code></pre> <p>It contains:</p> <ul> <li>A comprehensive README,</li> <li>the source code of <a href="https://github.com/se2p/whisker/tree/emse22">Whisker</a> and our custom <a href="https://github.com/se2p/scratch-vm/tree/emse22">Scratch VM</a>,</li> <li>a docker image of Whisker for a controlled execution environment,</li> <li>the datasets (as <code>*.sb3</code> Scratch project files) used in the study,</li> <li>the Whisker configuration files we used to generate tests with, and</li> <li>all experimental data from the paper as CSV files, along with scripts to re-create the plots.</li> </ul> <p>In case of questions, <a href="mailto:Patric.Feldmeier@uni-passau.de,Sebastian.Schweikl@uni-passau.de">feel free to contact us</a>.</p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.