Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
102
datasets available to search
ShareScore release 0.9.0
Dataset results
102 results for “software testing”
ChatGPT Software Testing Study
<p>This repository contains the dataset and replication package for the TestEd'23 paper </p> <blockquote> <p>Sajed Jalil, Suzzana Rafi, Thomas LaToza, Kevin Moran, and Wing Lam, "ChatGPT and Software Testing Education: Promises & Perils", in Proceedings of the 2nd International Workshop on Software Testing Educaiton (co-located with ICST'23), Dublin Ireland</p> </blockquote> <p> </p>
Reference datasets for consistency tests of GENAPOPOP 1.0 software: a user-friendly software to analyse genetic diversity and structure in partially clonal and selfed polyploid organisms.
<p>Datasets companion of the manuscript entitled GenAPoPop 1.0: a user-friendly software to analyse genetic diversity and structure in partially clonal and selfed polyploid organisms, used to achieve consistency test with Spagedi 1.5 software, and used as reference datasets to demonstrate the new possibilities allowed by GenAPoPop software.</p> <p>Raw datasets used for testing GenAPoPop 1.0, A user-friendly software for easily compute genetic analyses of autopolyploid populations packaged for Linux, MacOS and Windows; Results obtained from Spagedi 1.5 (Hardy & Vekemans 2001) and GenAPoPop1.0.</p> <p>Four pseudo-observed genotyping autotetrapolyploid SNP datasets, corresponding respectively to panmictic (A), highly clonal (B), highly selfed (C) and half-clonal-half-selfed (D) reproductive mode scenario. In all these four scenarios, we simulated two populations of 100 individuals each, connected with a migration rate of 0.01 and mutating at a rate of 0.01, genotyped at 10 SNPs. Datasets were recorded 1000 generations after an initial randomly drawing population with equal allele frequencies.</p> <p>One SNP tetraploid genotyping dataset from two French <em>Ludwigia grandiflora subsp. hexapetala</em> populations (aquatic plant from the Angiosperm clade): two populations in which we collected 75 individuals, each genotyped with 36 SNPs using the Hiplex method allowing confident allele dosage (Delord et al. 2018).</p> <p>One microsatellite tetraploid genotyping dataset on two Aulactinia stella populations (sea-anemone from the Cnidaria phylum), sampled on the coast of the arctic ocean. One population of 21 individuals and one population of 15 individuals, both genotyped with 10 microsatellites.</p> <p>We also report here the consistency tests with GenAlex and Spagedi, results of analyses (GPP tab) on 6300 independant simulations and inferences of the quantitative reproductive modes using the bayesian method on CEMP tab made on 6300 another independant simulations.</p>
Search-based Software Testing Driven by Automatically Generated and Manually Defined Fitness Functions - Dataset and Results
<p>This dataset contains the replication package for the paper: Federico Formica, Tony Fan, and Claudio Menghi. 2023. "Search-based Software Testing Driven by Automatically Generated and Manually Defined Fitness Functions".</p> <p>The dataset contains:</p> <ul> <li>The models used in the evaluation section.</li> <li>All the results obtained by running Athena-S on the models.</li> <li>The scripts that automatically analyze the results file and produce the tables and figures used in the paper.</li> </ul>
Results of the 2nd International Competition on Software Testing (Test-Comp 2020)
<p>This archive contains the results of the<br> 2nd Competition on Software Testing (Test-Comp 2020)<br> <a href="https://test-comp.sosy-lab.org/2020/">https://test-comp.sosy-lab.org/2020/</a></p> <p>The competition was run by Dirk Beyer, LMU Munich, Germany.<br> More information is available in the following article:<br> Dirk Beyer. Second Competition on Software Testing: Test-Comp 2020.<br> In Proceedings of the 23rd International Conference on Fundamental Approaches to Software Engineering<br> (FASE 2020, Dublin, April 28-30), 2020. Springer.<br> <a href="https://doi.org/10.1007/978-3-030-45234-6_25">https://doi.org/10.1007/978-3-030-45234-6_25</a></p> <p>Copyright (C) Dirk Beyer<br> <a href="https://www.sosy-lab.org/people/beyer/">https://www.sosy-lab.org/people/beyer/</a></p> <p>SPDX-License-Identifier: CC-BY-4.0<br> <a href="https://spdx.org/licenses/CC-BY-4.0.html">https://spdx.org/licenses/CC-BY-4.0.html</a></p> <p> </p> <p>To browse the competition results with a web browser, there are two options:<br> - start a local web server using<br> php -S localhost:8000<br> in order to view the data in this archive, or<br> - browse <a href="https://test-comp.sosy-lab.org/2020/results/">https://test-comp.sosy-lab.org/2020/results/</a><br> in order to view the data on the Test-Comp web page.</p> <p><br> Contents:</p> <p>index.html directs to the overview web page<br> LICENSE.txt specifies the license<br> README.txt this file<br> results-validated/ results of coverage-validation runs<br> results-verified/ results of test-generation runs and aggregated results</p> <p><br> The folder results-validated/ contains the results from coverage-validation runs:</p> <p>- *.xml.bz2 XML results from BenchExec<br> - *.logfiles.zip output from tools<br> - *.json.gz mapping from files names to SHA 256 hashes for the file content</p> <p><br> The folder results-verified/ contains the results from test-generation runs and aggregated results:</p> <p>index.html overview web page with rankings and score table<br> design.css HTML style definitions<br> *.xml.bz2 XML results from BenchExec<br> *.merged.xml.bz2 XML results from BenchExec, status adjusted according to the validation results<br> *.logfiles.zip output from tools<br> *.json.gz mapping from files names to SHA 256 hashes for the file content<br> *.xml.bz2.table.html HTML views on the detailed results data as generated by BenchExec's table generator<br> *.All.table.html HTML views of the full benchmark set (all categories) for each tool<br> META_*.table.html HTML views of the benchmark set for each meta category for each tool, and over all tools<br> <category>*.table.html HTML views of the benchmark set for each category over all tools<br> iZeCa0gaey.html HTML views per tool</p> <p>quantilePlot-* score-based quantile plots as visualization of the results<br> quantilePlotShow.gp example Gnuplot script to generate a plot<br> score* accumulated score results in various formats</p> <p><br> The hashes of the file names (in the files *.json.gz) are useful for<br> - validating the exact contents of a file and<br> - accessing the files from the witness store.</p> <p> </p> <p>Overview over archives from Test-Comp 2020 that are available at Zenodo:</p> <p><a href="https://doi.org/10.5281/zenodo.3678275">https://doi.org/10.5281/zenodo.3678275</a> Witness store (containing the generated test suites)<br> <a href="https://doi.org/10.5281/zenodo.3678264">https://doi.org/10.5281/zenodo.3678264</a> Results (XML result files, log files, file mappings, HTML tables)<br> <a href="https://doi.org/10.5281/zenodo.3678250">https://doi.org/10.5281/zenodo.3678250</a> Test tasks, version testcomp20<br> <a href="https://doi.org/10.5281/zenodo.3574420">https://doi.org/10.5281/zenodo.3574420</a> BenchExec, version 2.5.1</p> <p>All benchmarks were executed<br> for Test-Comp 2020, <a href="https://test-comp.sosy-lab.org/2020/">https://test-comp.sosy-lab.org/2020/</a><br> by Dirk Beyer, LMU Munich<br> based on the components<br> git@github.com:sosy-lab/sv-benchmarks.git testcomp20-0-gd6cd3e5dd4<br> git@gitlab.com:sosy-lab/test-comp/bench-defs.git testcomp19-84-gac76836<br> git@github.com:sosy-lab/benchexec.git 2.5.1-0-gffad635</p> <p><br> Feel free to contact me in case of questions:<br> <a href="https://www.sosy-lab.org/people/beyer/">https://www.sosy-lab.org/people/beyer/</a></p> <p> </p>
Investigating Multi and Many-Objective Metaheuristics to Support Software Integration Testing
<p> In spite of the fact that Search-Based Software Testing (SBST) is a very appealing field today, there are few studies that deal with software integration testing and, even so, most of these works are not truly related to the generation of test cases to this testing level. In this paper, we present a method, InMeHy, which aims at investigating the use of metaheuristics to derive integration test cases based on C++ source code. A graph is created based on the code which represents the integration of several classes of the application. Multi and Many-Objective metaheuristics (Evolutionary Algorithms) were considered to generate integration test cases and were assessed via three quality indicators. Results show that the traditional Indicator-Based Evolutionary Algorithm (IBEA) turned out to be the best out of four algorithms evaluated, including newer Many-Objective strategies such as Nondominated Sorting Genetic Algorithm-III (NSGA-III).</p>
A dataset for testing a single-cell Hi-C softwares
Open the record for dataset details and reuse information.
Instructions for ContinuousFlex software installation and Tutorial for running MDSPACE and MDTOMO with test datasets
<p>Instructions for ContinuousFlex software installation and Tutorial for running MSPACE and MDTOMO methods of ContinuousFlex, together with test datasets. The ContinuousFlex installation instructions also include the instructions for installing Scipion, Xmipp, ChimeraX, and VMD (required for using ContinuousFlex). This distribution was prepared for the practical session of AlgoSB 2023 thematic school (Novembre 20 - 24, 2023, Cargese, France) on hybrid methods MDSPACE and MDTOMO for obtaining continuous conformational landscapes from cryo electron microscopy and cryo electron tomography data. Instructions for the software installation are in Instructions-Installation-ContinuousFlex.txt. The tutorial for the practical session is in MDSPACE_Tutorial_v4.docx and MDSPACE_Tutorial_v4.pdf.</p>
Choosing the Right Test Generation Tool: A Guide for Software Practitioners
<p>Context: Software testing tools are essential during the processes of developing and maintaining quality digital applications in a scenario where they become increasingly complex. Problem: The market offers hundreds of test generation tools with different approaches and functionalities, it is important that developers and designers have knowledge of which tools are best suited to their work in order to avoid unnecessary expenses and easily avoidable frustrations. Solution: to identify the main tools available on the market in order to highlight their advantages and disadvantages, thus presenting a reference guide that will help developers choose<br>the most appropriate tool for each test situation. </p><p>Method In this work: a multivocal literature review has been carried out to identify works related to test generation tools, also identifying the most cited tools in white and gray literature. Furthermore, a survey was developed and conducted with professionals working in different areas of software development, in order to ascertain their familiarity with the testing tools identified in the white and gray literature, as well as the advantages and challenges that have more weight when deciding which tool to adopt.</p><p>Summarization of Results: The results showed that most of the tools found in the white literature, such as Monkey and Dynodroid, are not very well known and used by the participants and that they were more familiar with tools more mentioned in the gray literature, such as Postman and Selenium. In addition, a list of characteristics most desired and avoided by participants when needing to choose a new tool for a project was obtained. Impact on the IS field: This work contributes to IS field by providing a reference guide where practitioners can identify the most recommended tools according to a specific scenario, and then use a comparative table in order to evaluate the most appropriate tool for the project among those recommended. The Guide can be used by any testing practitioner.</p>
Simulations of FLEXPART-WRF used for testing TRansport Of water VApor (TROVA) software (Part II)
<p>Necessary data for carrying out the tests using TRansport Of water VApor (TROVA) software for backward in time. The TROVA software, was developed in Python and Fortran for the study of moisture sources and sinks. <span><span><span>In addition, the Python code is provided for the representation of the results.</span></span></span></p>
Source data for testing results of the ADS software
<p>This is the source data for the testing results of the ADS software at https://zenodo.org/record/5579390</p> <p> </p> <p> </p>
TRAVEL: A Dataset with Toolchains for Test Generation and Regression Testing of Self-driving Cars Software
<p><strong>Introduction</strong></p> <p>This repository hosts the Testing Roads for Autonomous VEhicLes (TRAVEL) dataset. TRAVEL is an extensive collection of virtual roads that have been used for testing lane assist/keeping systems (i.e., driving agents) and data from their execution in state of the art, physically accurate driving simulator, called <a href="https://www.beamng.tech/">BeamNG.tech</a>. Virtual roads consist of sequences of <strong>road points</strong> interpolated using Cubic splines.</p> <p>Along with the data, this repository contains instructions on how to install the tooling necessary to generate new data (i.e., test cases) and analyze them in the context of test regression. We focus on test selection and test prioritization, given their importance for developing high-quality software following the DevOps paradigms.</p> <p>This dataset builds on top of our previous work in this area, including work on</p> <ul> <li>test generation (e.g., <a href="https://doi.org/10.1145/3293882.3330566">AsFault</a>, <a href="https://doi.org/10.1145/3368089.3409730">DeepJanus</a>, and <a href="https://doi.org/10.1145/3460319.3464811">DeepHyperion</a>) and the SBST CPS tool competition (<a href="https://doi.org/10.1109/SBST52555.2021.00011">SBST2021</a>),</li> <li>test selection: <a href="https://arxiv.org/abs/2111.04666">SDC-Scissor</a> and related <a href="https://github.com/ChristianBirchler/sdc-scissor">tool</a></li> <li>test prioritization: <a href="https://arxiv.org/abs/2107.09614">automated test cases prioritization work for SDCs</a>.</li> </ul> <p><strong>Dataset Overview</strong></p> <p>The TRAVEL dataset is available under the <code>data</code> folder and is organized as a set of <em>experiments</em> folders. Each of these folders is generated by running the <code>test-generator</code> (see below) and contains the configuration used for generating the data (<code>experiment_description.csv</code>), various statistics on generated tests (<code>generation_stats.csv</code>) and found faults (<code>oob_stats.csv</code>). Additionally, the folders contain the raw test cases generated and executed during each experiment (<code>test.<TEST_ID>.json</code>).</p> <p>The following sections describe what each of those files contains.</p> <p>Experiment Description</p> <p>The <code>experiment_description.csv</code> contains the settings used to generate the data, including:</p> <ul> <li><strong>Time budget</strong>. The overall generation budget in hours. This budget includes both the time to generate and execute the tests as driving simulations.</li> <li><strong>The size of the map</strong>. The size of the squared map defines the boundaries inside which the virtual roads develop in meters.</li> <li><strong>The test subject</strong>. The driving agent that implements the lane-keeping system under test. The TRAVEL dataset contains data generated testing the BeamNG.AI and the end-to-end Dave2 systems.</li> <li><strong>The test generator</strong>. The algorithm that generated the test cases. The TRAVEL dataset contains data obtained using various algorithms, ranging from naive and advanced random generators to complex evolutionary algorithms, for generating tests.</li> <li><strong>The speed limit</strong>. The maximum speed at which the driving agent under test can travel.</li> <li><strong>Out of Bound (OOB) tolerance</strong>. The test cases' oracle that defines the tolerable amount of the ego-car that can lie outside the lane boundaries. This parameter ranges between 0.0 and 1.0. In the former case, a test failure triggers as soon as any part of the ego-vehicle goes out of the lane boundary; in the latter case, a test failure triggers only if the entire body of the ego-car falls outside the lane.</li> </ul> <p>Experiment Statistics</p> <p>The <code>generation_stats.csv</code> contains statistics about the test generation, including:</p> <ul> <li><strong>Total number of generated tests</strong>. The number of tests generated during an experiment. This number is broken down into the number of <strong>valid tests</strong> and <strong>invalid tests</strong>. Valid tests contain virtual roads that do not self-intersect and contain turns that are not too sharp.</li> <li><strong>Test outcome</strong>. The test outcome contains the number of <strong>passed tests</strong>, <strong>failed tests</strong>, and <strong>test in error</strong>. Passed and failed tests are defined by the OOB Tolerance and an additional (implicit) oracle that checks whether the ego-car is moving or standing. Tests that did not pass because of other errors (e.g., the simulator crashed) are reported in a separated category.</li> </ul> <p>The TRAVEL dataset also contains statistics about the failed tests, including the overall number of failed tests (<strong>total oob</strong>) and its breakdown into OOB that happened while driving <strong>left</strong> or <strong>right</strong>. Further statistics about the diversity (i.e., <strong>sparseness</strong>) of the failures are also reported.</p> <p>Test Cases and Executions</p> <p>Each <code>test.<TEST_ID>.json</code> contains information about a test case and, if the test case is valid, the data observed during its execution as driving simulation.</p> <p>The data about the test case definition include:</p> <ul> <li><strong>The road points</strong>. The list of points in a 2D space that identifies the center of the virtual road, and their interpolation using cubic splines (<strong>interpolated_points</strong>)</li> <li><strong>The test ID</strong>. The unique identifier of the test in the experiment.</li> <li><strong>Validity flag and explanation</strong>. A flag that indicates whether the test is valid or not, and a brief message describing why the test is not considered valid (e.g., <code>the road contains sharp turns</code> or <code>the road self intersects</code>)</li> </ul> <p>The test data are organized according to the following JSON Schema and can be interpreted as <code>RoadTest</code> objects provided by the <a href="https://github.com/se2p/tool-competition-av/blob/main/code_pipeline/tests_generation.py">tests_generation.py</a> module.</p> <pre><code>{ "type": "object", "properties": { "id": { "type": "integer" }, "is_valid": { "type": "boolean" }, "validation_message": { "type": "string" }, "road_points": { §\label{line:road-points}§ "type": "array", "items": { "$ref": "schemas/pair" }, }, "interpolated_points": { §\label{line:interpolated-points}§ "type": "array", "items": { "$ref": "schemas/pair" }, }, "test_outcome": { "type": "string" }, §\label{line:test-outcome}§ "description": { "type": "string" }, "execution_data": { "type": "array", "items": { "$ref" : "schemas/simulationdata" } } }, "required": [ "id", "is_valid", "validation_message", "road_points", "interpolated_points" ] } </code></pre> <p>Finally, the <strong>execution data</strong> contain a list of timestamped state information recorded by the driving simulation. State information is collected at constant frequency and includes absolute position, rotation, and velocity of the ego-car, its speed in Km/h, and control inputs from the driving agent (steering, throttle, and braking). Additionally, execution data contain OOB-related data, such as the lateral distance between the car and the lane center and the OOB percentage (i.e., how much the car is outside the lane).</p> <p>The simulation data adhere to the following (simplified) JSON Schema and can be interpreted as Python objects using the <a href="https://github.com/se2p/tool-competition-av/blob/main/self_driving/simulation_data.py">simulation_data.py</a> module.</p> <pre><code>{ "$id": "schemas/simulationdata", "type": "object", "properties": { "timer" : { "type": "number" }, "pos" : { "type": "array", "items":{ "$ref" : "schemas/triple" } } "vel" : { "type": "array", "items":{ "$ref" : "schemas/triple" } } "vel_kmh" : { "type": "number" }, "steering" : { "type": "number" }, "brake" : { "type": "number" }, "throttle" : { "type": "number" }, "is_oob" : { "type": "number" }, "oob_percentage" : { "type": "number" } §\label{line:oob-percentage}§ }, "required": [ "timer", "pos", "vel", "vel_kmh", "steering", "brake", "throttle", "is_oob", "oob_percentage" ] } </code></pre> <p><strong>Dataset Content</strong></p> <p>The TRAVEL dataset is a lively initiative so the content of the dataset is subject to change. Currently, the dataset contains the data collected during the SBST CPS tool competition, and data collected in the context of our recent work on test selection (<a href="https://arxiv.org/abs/2111.04666">SDC-Scissor work</a> and <a href="https://github.com/ChristianBirchler/sdc-scissor">tool</a>) and test prioritization (<a href="https://arxiv.org/abs/2107.09614">automated test cases prioritization work for SDCs</a>).</p> <p>SBST CPS Tool Competition Data</p> <p>The data collected during the SBST CPS tool competition are stored inside <code>data/competition.tar.gz</code>. The file contains the test cases generated by <a href="https://github.com/mahshidhelali/tool-competition-av">Deeper</a>, <a href="https://github.com/ERATOMMSD/frenetic-sbst21">Frenetic</a>, <a href="https://github.com/TayYim/adafrenetic-sbst22">AdaFrenetic</a>, and <a href="https://github.com/alessiogambi/swat-sbst2021">Swat</a>, the <strong>open-source</strong> test generators submitted to the competition and executed against BeamNG.AI with an aggression factor of 0.7 (i.e., conservative driver).</p> <table> <thead> <tr> <th>Name</th> <th>Map Size (m x m)</th> <th>Max Speed (Km/h)</th> <th>Budget (h)</th> <th>OOB Tolerance (%)</th> <th>Test Subject</th> </tr> </thead> <tbody> <tr> <td>DEFAULT</td> <td>200 × 200</td> <td>120</td> <td>5 (real time)</td> <td>0.95</td> <td>BeamNG.AI - 0.7</td> </tr> <tr> <td>SBST</td> <td>200 × 200</td> <td>70</td> <td>2 (real time)</td> <td>0.5</td> <td>BeamNG.AI - 0.7</td> </tr> </tbody> </table> <p>Specifically, the TRAVEL dataset contains 8 repetitions for each of the above configurations for each test generator totaling 64 experiments.</p> <p>SDC Scissor</p> <p>With SDC-Scissor we collected data based on the <a href="https://github.com/ERATOMMSD/frenetic-sbst21">Frenetic</a> test generator. The data is stored inside <code>data/sdc-scissor.tar.gz</code>. The following table summarizes the used parameters.</p> <table> <thead> <tr> <th>Name</th> <th>Map Size (m x m)</th> <th>Max Speed (Km/h)</th> <th>Budget (h)</th> <th>OOB Tolerance (%)</th> <th>Test Subject</th> </tr> </thead> <tbody> <tr> <td>SDC-SCISSOR</td> <td>200 × 200</td> <td>120</td> <td>16 (real time)</td> <td>0.5</td> <td>BeamNG.AI - 1.5</td> </tr> </tbody> </table> <p>The dataset contains 9 experiments with the above configuration. For generating your own data with SDC-Scissor follow the instructions in its <a href="https://github.com/ChristianBirchler/sdc-scissor">repository</a>.</p> <p><strong>Dataset Statistics</strong></p> <p>Here is an overview of the TRAVEL dataset: generated tests, executed tests, and faults found by all the test generators grouped by experiment configuration. Some 25,845 test cases are generated by running 4 test generators 8 times in 2 configurations using the SBST CPS Tool Competition code pipeline (SBST in the table). We ran the test generators for 5 hours, allowing the ego-car a generous speed limit (120 Km/h) and defining a high OOB tolerance (i.e., 0.95), and we also ran the test generators using a smaller generation budget (i.e., 2 hours) and speed limit (i.e., 70 Km/h) while setting the OOB tolerance to a lower value (i.e., 0.85). We also collected some 5, 971 additional tests with SDC-Scissor (SDC-Scissor in the table) by running it 9 times for 16 hours using Frenetic as a test generator and defining a more realistic OOB tolerance (i.e., 0.50).</p> <p><strong>Generating new Data</strong></p> <p>Generating new data, i.e., test cases, can be done using the <a href="https://github.com/se2p/tool-competition-av">SBST CPS Tool Competition pipeline</a> and the driving simulator <a href="https://www.beamng.tech/">BeamNG.tech</a>.</p> <p>Extensive instructions on how to install both software are reported inside the SBST CPS Tool Competition pipeline <a href="https://github.com/se2p/tool-competition-av/blob/main/documentation/README.md">Documentation</a>; therefore, below we only summarize the overall installation process.</p> <p>Installation</p> <ol> <li> <p>Install Python 3.7. (We tested the code using 3.7.9, so we suggest to install that version of Python)</p> </li> <li> <p>Clone the <a href="https://github.com/se2p/tool-competition-av">SBST CPS Tool Competition pipeline</a></p> </li> <li> <p>Request a (free) copy of the driving simulator at <a href="https://register.beamng.tech/">https://register.beamng.tech</a> and, after receiving the registration key file (<code>tech.key</code>) and the download link, download and install the latest distribution of the software (<code>0.24</code>).</p> </li> <li> <p>Create a python virtual environment (<code>python -m venv .venv</code>) inside the root of the SBST CPS Tool Competition pipeline and activate it. Please, be sure to call the virtual environment exactly <code>.venv</code> and place it in the correct position as the scripts to collect new data assume this setup.</p> </li> <li> <p>Install inside the virtual environment all the SBST CPS Tool Competition pipeline requirements (<code>pip install -r requirements.txt</code>).</p> </li> </ol> <p>If you have successfully setup the python environment, activate the virtual environment, and run</p> <pre><code>python.exe competition.py --help </code></pre> <p>and check that the command outputs the <a href="https://github.com/se2p/tool-competition-av/blob/main/documentation/GUIDELINES.md#technical-considerations">usage</a> of the SBST CPS Tool Competition pipeline.</p> <p>Data Collection</p> <p>Collecting new data is as simple as starting the SBST CPS Tool Competition pipeline with suitable parameters for test generation (<code>--time-budget</code>, <code>--module-name</code>, <code>--module-path</code> and <code>--class-name</code>), test executor (<code>--executor</code>), and test definition (<code>--map-size</code>, <code>--speed-limit</code> and <code>--oob-tolerance</code>).</p> <p>To ease the collection of new data, we include the (PowerShell) scripts we used to create the current TRAVEL dataset. Those scripts can be found inside the <code>data-collection</code> folder and are named after the test generator and configuration they implement. For instance, <code>Frenetic-DEFAULT.ps1</code> will execute the Frenetic test generator in the DEFAULT configuration, whereas <code>Deeper-SBST.ps1</code> will execute the Deeper test generation in the SBST configuration. The description of these configurations is reported in the <a href="https://github.com/se2p/msr-travel-dataset#sbst-cps-tool-competition-data">table above</a>).</p> <p>To run those scripts, open a PowerShell, <code>cd</code> to the folder containing the script (<code>data-collection</code>), and invoke the script you want. Assuming you have configured the SBST CPS Tool Competition pipeline as described above, the scripts will start the code pipeline and the BeamNG.tech driving simulator.</p> <p>Once the run is finished, you will find the results under the <code>results</code> folder inside the SBST CPS Tool Competition pipeline project. The results of each experiment are stored in a uniquely named folder that clearly indicates the test generator used to produce them (see <a href="https://github.com/se2p/msr-travel-dataset#dataset-overview">above</a> for the detailed description of the produced results).</p> <p><strong>Data Usage</strong></p> <p>So far, the data contained in the TRAVEL dataset have been used for benchmarking test generators as well as for optimizing regression testing in the context of Self-driving car software.</p> <p>We used the TRAVEL to study the problem of test generation from different perspectives.</p> <p>In the last edition of the <a href="https://doi.org/10.1109/SBST52555.2021.00011">SBST Tool Challenge</a>, we answer the question <em>"How efficient and effective are the existing test generators for SDCs?"</em>. We assess test generation efficiency by counting how many tests are produced by each test generator within a given time budget. To assess test generation effectiveness, we count how many of the generated tests are valid and invalid.</p> <p>In a <a href="https://doi.org/10.1145/3460319.3464811">recent study</a>, we propose (along the DeepHyperion test generator) a novel approach to assess SDCs test suites adequacy by measuring the coverage of feature maps that they achieved. To do so, we extract structural and behavioral test case features from the test case descriptions and simulation data by looking at the geometrical properties describing the roads and the physical quantities describing the ego-car behavior (e.g., position, speed).</p> <p>We also answer the question <em>"How effective are the resulting tests in finding faults in SDCs?"</em>, by counting how many failures are triggered by the generated tests and how different those failures are (e.g., Left or Right OOBs). Specifically, we propose an approach to measure OOBs similarity by computing the edit distance of the road segments relevant to the corresponding failure.</p> <p><a href="https://github.com/ERATOMMSD/roadsearch">Castellano et al.</a> demonstrated that a dataset such as TRAVEL can be used to answer other interesting questions, such as <em>"Which road representation is the most suitable for generating SDCs test cases?"</em>.</p> <p>In the context of regression testing the TRAVEL dataset is also used to develop tools to select test cases that are likely to fail before executing them simulation (e.g., <a href="https://github.com/ChristianBirchler/sdc-scissor">SDC-Scissor</a>) and also to prioritize them in a single-objective and multi-objective approach by <a href="https://arxiv.org/abs/2107.09614">SDC-Prioritizer</a>.</p> <p>With SDC-Scissor we can select test cases that are more likely to fail. This allows us to answer the following research questions like "To what extent is it possible to identify safe and unsafe test scenarios for SDCs before executing them?" or "Does SDC-Scissor improve the cost-effectiveness of simulation-based testing of SDCs?" <a href="https://arxiv.org/abs/2111.04666">Khatiri et al.</a>. SDC-Scissor achieves a F1-score (between 47% and 90%) compared to a baseline in identifying failing tests and reduced the time spent running uninformative tests by 107% and 170%. <a href="https://christianbirchler.github.io/sdc_scissor_demonstration_paper.pdf">(Birchler et al.)</a></p> <p>Regarding test prioritization, the TRAVEL dataset allows us to answer the following questions: "To what extent is it possible to apply test prioritization strategies to prioritize the execution of safe and unsafe test scenarios for SDCs?", “What is the cost-effectiveness of SDC-Prioritizer compared to baseline approaches?” and “What is the overhead introduced by it?”. To address the former question, we compared the test prioritization approaches in terms of fault detection rate (i.e., how fast faults are detected during the test execution process) using Cost cognizant Average Percentage of Fault Detection (APFD𝑐) (<a href="https://doi.org/10.1145/2771783.2771788">Epitropakis et al.</a>, <a href="https://doi.org/10.1109/ICSM.1999.792604">Rothermel et al.</a>) and concluded that SDC-Prioritizer (with 82.5% APFD𝑐) significantly outperformed the baseline random and greedy test prioritization approaches <a href="https://arxiv.org/abs/2107.09614">Birchler et al.</a>. For the latter research question, we compared the time required by SDC-Prioritizer to sort tests against the time needed to run all of the tests and showed that, on average, SDC-Prioritizer needs less than 13 minutes to perform the test prioritization while running the SDC test suites takes between 16 to 106 hours. As explained in Section 3.2.3, SDC-Prioritizer uses the road features shared with TRAVEL to guide the search process towards generating test orders with the high diversity in the road shapes. The selection of best features for test prioritization was performed with the Principal component analysis (PCA) <a href="https://arxiv.org/abs/2107.09614">Birchler et al.</a>.</p>
Dataset for Cost-effective Simulation-based Test Selection in Self-driving Cars Software with SDC-Scissor
<p><strong>SDC-Scissor tool for Cost-effective Simulation-based Test Selection in Self-driving Cars Software</strong></p> <p>This dataset provides test cases for self-driving cars with the BeamNG simulator. Check out the repository and demo video to get started.</p> <p><strong>GitHub:</strong> <a href="https://github.com/ChristianBirchler/sdc-scissor">github.com/ChristianBirchler/sdc-scissor</a></p> <p>This project extends the tool competition platform from the <a href="https://github.com/se2p/tool-competition-av">Cyber-Phisical Systems Testing Competition</a> which was part of the <a href="https://sbst21.github.io/">SBST Workshop in 2021</a>.</p> <p><strong>Usage</strong></p> <p><strong>Demo</strong></p> <p> <a href="https://youtu.be/Cn8p648KnfQ">YouTube Link</a></p> <p><strong>Installation</strong></p> <p>The tool can either be run with <a href="https://docs.docker.com/get-docker/">Docker</a> or locally using <a href="https://python-poetry.org/docs/">Poetry</a>.</p> <p>When running the simulations a working installation of <a href="https://beamng.gmbh/research/">BeamNG.research</a> is required. Additionally, this simulation cannot be run in a Docker container but must run locally.</p> <p>To install the application use one of the following approaches:</p> <ul> <li>Docker: <code>docker build --tag sdc-scissor .</code></li> <li>Poetry: <code>poetry install</code></li> </ul> <p><strong>Using the Tool</strong></p> <p>The tool can be used with the following two commands:</p> <ul> <li>Docker: <code>docker run --volume "$(pwd)/results:/out" --rm sdc-scissor [COMMAND] [OPTIONS]</code> (this will write all files written to <code>/out</code> to the local folder <code>results</code>)</li> <li>Poetry: <code>poetry run python sdc-scissor.py [COMMAND] [OPTIONS]</code></li> </ul> <p>There are multiple commands to use. For simplifying the documentation only the command and their options are described.</p> <ul> <li>Generation of tests: <ul> <li><code>generate-tests --out-path /path/to/store/tests</code></li> </ul> </li> <li>Automated labeling of Tests: <ul> <li><code>label-tests --road-scenarios /path/to/tests --result-folder /path/to/store/labeled/tests</code></li> <li><em>Note:</em> This only works locally with BeamNG.research installed</li> </ul> </li> <li>Model evaluation: <ul> <li><code>evaluate-models --dataset /path/to/train/set --save</code></li> </ul> </li> <li>Split train and test data: <ul> <li><code>split-train-test-data --scenarios /path/to/scenarios --train-dir /path/for/train/data --test-dir /path/for/test/data --train-ratio 0.8</code></li> </ul> </li> <li>Test outcome prediction: <ul> <li><code>predict-tests --scenarios /path/to/scenarios --classifier /path/to/model.joblib</code></li> </ul> </li> <li>Evaluation based on random strategy: <ul> <li><code>evaluate --scenarios /path/to/test/scenarios --classifier /path/to/model.joblib</code></li> </ul> </li> </ul> <p>The possible parameters are always documented with <code>--help</code>.</p> <p><strong>Linting</strong></p> <p>The tool is verified the linters <a href="https://flake8.pycqa.org/en/latest/">flake8</a> and <a href="https://pylint.org/">pylint</a>. These are automatically enabled in <a href="https://code.visualstudio.com/">Visual Studio Code</a> and can be run manually with the following commands:</p> <pre>poetry run flake8 . poetry run pylint **/*.py</pre> <p><strong>License</strong></p> <p>The software we developed is distributed under GNU GPL license. See the <a href="https://github.com/ChristianBirchler/sdc-scissor/blob/main/LICENSE.md">LICENSE.md</a> file.</p> <p><strong>Contacts</strong></p> <p>Christian Birchler - Zurich University of Applied Science (ZHAW), Switzerland - <a href="mailto:birc@zhaw.ch">birc@zhaw.ch</a></p> <p>Nicolas Ganz - Zurich University of Applied Science (ZHAW), Switzerland - <a href="mailto:gann@zhaw.ch">gann@zhaw.ch</a></p> <p>Sajad Khatiri - Zurich University of Applied Science (ZHAW), Switzerland - <a href="mailto:mazr@zhaw.ch">mazr@zhaw.ch</a></p> <p>Dr. Alessio Gambi - Passau University, Germany - <a href="mailto:alessio.gambi@uni-passau.de">alessio.gambi@uni-passau.de</a></p> <p>Dr. Sebastiano Panichella - Zurich University of Applied Science (ZHAW), Switzerland - <a href="mailto:panc@zhaw.ch">panc@zhaw.ch</a></p> <p><strong>References</strong></p> <ul> <li>Christian Birchler, Nicolas Ganz, Sajad Khatiri, Alessio Gambi, and Sebastiano Panichella. 2022. Cost-effective Simulation-based Test Selection in Self-driving Cars Software with SDC-Scissor. In 2022 IEEE 29th International Conference on Software Analysis, Evolution and Reengineering (SANER), IEEE.</li> </ul> <p><strong>If you use this tool in your research, please cite the following papers:</strong></p> <pre><code>@INPROCEEDINGS{Birchler2022, author={Birchler, Christian and Ganz, Nicolas and Khatiri, Sajad and Gambi, Alessio, and Panichella, Sebastiano}, booktitle={2022 IEEE 29th International Conference on Software Analysis, Evolution and Reengineering (SANER), title={Cost-effective Simulationbased Test Selection in Self-driving Cars Software with SDC-Scissor}, year={2022}, }</code></pre>
Supplementary materials of the paper entitled: "Metamorphic Testing Meets Regression Testing: A Case Study of Scientific Software Maintenance"
<p>Supplementary materials of the paper entitled:</p> <p>“Metamorphic Testing Meets Regression Testing: A Case Study of Scientific Software Maintenance”</p> <p>file01 - source code for detecting and comparing output relations of a given metamorphic relation<br> file02 - interview transcripts and qualitative coding results<br> file03 - system script for compiling SWMM, creating sSWMM simulation outputs, and logging execution time<br> file04 - source code for SWMM 5.1.014<br> file05 - source code for SWMM 5.1.015<br> file06 - a suite of 40 regression tests (40 .inp files)<br> file07 - result table for 760 cases (19 MRs × 40 .inp files)<br> file08 - 491 automatically generated follow-up .inp files via reuse<br> file09 - result table for mutation analysis<br> file10 - output files for mutation analysis with metamorphic testing:<br> a. Original source output<br> b. Original follow-up output<br> c. Mutant source output<br> d. Mutant follow-up output</p>
Cloud computing is one of the most popular and sophisticated technologies adopted by organizations worldwide. Some world-leading organizations enhance their efficiency and effectiveness by using cloud computing technology. Working from home (WFH) has been a popular trend among organizations during the coronavirus (COVID-19) pandemic. The COVID-19 saw a breakthrough in work cultures and environments where working from home was a remarkable success in remote working environments, despite being a rare phenomenon in Sri Lanka. Yet, it is argued that the deployment of work from home has not been effective among Sri Lankan business organizations due to a lack of IT infrastructure, facilities, and knowledge. The purpose of the study is to investigate the impact of cloud computing, embracing the service models (Infrastructure as a Service, Platform as a Service, and Software as a Service) as theoretical lenses and testing the COVID-19 as the moderator. The study has been conducted based on a deductive approach and adopted a stratified random sampling method. The sample consisted of 384 IT employees among those who had experienced working from home. The study utilized multiple regression and found that cloud computing service models significantly impact work from home with the moderating effect of COVID-19.
<p>Cloud computing is one of the most popular and sophisticated technologies adopted by organizations worldwide. Some world-leading organizations enhance their efficiency and effectiveness by using cloud computing technology. Working from home (WFH) has been a popular trend among organizations during the coronavirus (COVID-19) pandemic. The COVID-19 saw a breakthrough in work cultures and environments where working from home was a remarkable success in remote working environments, despite being a rare phenomenon in Sri Lanka. Yet, it is argued that the deployment of work from home has not been effective among Sri Lankan business organizations due to a lack of IT infrastructure, facilities, and knowledge. The purpose of the study is to investigate the impact of cloud computing, embracing the service models (Infrastructure as a Service, Platform as a Service, and Software as a Service) as theoretical lenses and testing the COVID-19 as the moderator. The study has been conducted based on a deductive approach and adopted a stratified random sampling method. The sample consisted of 384 IT employees among those who had experienced working from home. The study utilized multiple regression and found that cloud computing service models significantly impact work from home with the moderating effect of COVID-19.</p>
An empirical study on the influence of developers' experience on software test code quality
<p>Software developers' engagement in open-source software projects lies in different levels of participation, e.g., core or peripheral developers. Recent studies have investigated the role of developers' contributions and their influence on software quality. However, few studies investigate the relationship between the developers' experience and test code quality in software projects. In this study, we aim to investigate the relationship between the developers' experience and the test code quality from the perspective of test smells. We performed an empirical study to analyze the insertion and removal of test smells in four open-source Java projects. We collected 18 test smells and calculated their authorship through the projects' Tags. The four software projects contain 386 test classes and 5,178 test smells. We found out that the insertion of 67.28\% of test smells occurs during the test class creation, and the removal of 20.88\% of test smells occurs during the evolution of projects. In addition, core developers are responsible for inserting 88.91\% and removing 89.82\% test smells. Core developers insert and remove more test smells than the peripheral developers. Most test smell removal is due to test code deletion, which may indicate that both core and peripheral developers are unaware of test smells in test code.</p>
Replication package for our paper entitled "Faster and Better Quantum Software Testing through Specification Reduction and Projective Measurements"
<p><br> -Experiment1<br> Contains the source files for experiment 1 and the study subjects.</p> <p> -Experiment2<br> Contains the source files for experiment 2 and a mutation generator.</p> <p> -results<br> Contains the results from experiment 1 and 2 and postprocessing files</p> <p> -RQ1,RQ2,RQ3<br> Contains source files for figures, tables and data for each research question<br> of the paper</p> <p><br>Please inspect the README.txt file for more details.</p>
A Survey of Assertions in Software Testing - Papers Classification
<p>The tabular listing of the classification of the collected papers for the literature survey of assertions in software testing.</p>
Survey Data for "Software Testing: Survey of the Industry Practices"
<p>Dataset for the surveys presented in our article "Software Testing: Survey of the Industry Practices."</p>
Artificial Software Testing Workload
<p>This is a small dataset representing an artificial software testing workload generated from the real one. It consists of instance files, each of which is structured as follows:</p> <pre><code>[number of test suites] [test suite uuid] [test suite priority (1>0)] [arrival time (s)] [number of test cases] [test case uuid] [test case type] [test case outcome (0 - pass)] [test case duration (s)] [test case uuid] [test case type] [test case outcome (0 - pass)] [test case duration (s)] (...) [test suite uuid] [test suite priority (1>0)] [arrival time (s)] [number of test cases] [test case uuid] [test case type] [test case outcome (0 - pass)] [test case duration (s)] [test case uuid] [test case type] [test case outcome (0 - pass)] [test case duration (s)] (...) (...)</code></pre>
3D cranial landmark coordinates from the Athens Collection. A modern Greek population reference dataset for testing 3D-ID software
<p>The present dataset comprises the landmark coordinates of 158 intact crania (80 males and 78 females) of adult individuals from the Athens Collection. The 3D coordinates of up to 34 landmarks have been extracted from high quality textured 3D models produced with photogrammetry. The dataset aims to evaluate the correct classification performance of 3D-ID software. Hence, the dataset contains the landmark 3D coordinates both in Meshlab's PickedPoints files (.pp) but also in 3D-ID's input text format (.3did). The dataset is accompanied by certain GNU Octave scripts and functions used for data conversion and integrity check. For more details see the Dataset Description pdf.</p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.