Test Suites from Test-Generation Tools (Test-Comp 2021)

<p>Test Suites</p> <p>This file describes the contents of an archive of the 3rd Competition on Software Testing (Test-Comp 2021).<br> <a href="https://test-comp.sosy-lab.org/2021/">https://test-comp.sosy-lab.org/2021/</a></p> <p>The competition was run by Dirk Beyer, LMU Munich, Germany.<br> More information is available in the following article:<br> Dirk Beyer. <em>Status Report on Software Testing: Test-Comp 2021.</em> In Proceedings of the 24th International Conference on Fundamental Approaches to Software Engineering (FASE 2021, Luxembourg, March 27 - April 1), 2021. Springer.</p> <p>Copyright (C) Dirk Beyer<br> <a href="https://www.sosy-lab.org/people/beyer/">https://www.sosy-lab.org/people/beyer/</a></p> <p>SPDX-License-Identifier: CC-BY-4.0<br> <a href="https://spdx.org/licenses/CC-BY-4.0.html">https://spdx.org/licenses/CC-BY-4.0.html</a></p> <p>Contents</p> <ul> <li><code>LICENSE.txt</code>: specifies the license</li> <li><code>README.txt</code>: this file</li> <li><code>witnessFileByHash/</code>: This directory contains test suites (witnesses for coverage). Each witness in this directory is stored in a file whose name is the SHA2 256-bit hash of its contents followed by the filename extension .zip. The format of each test suite is described on the format web page: <a href="https://gitlab.com/sosy-lab/software/test-format">https://gitlab.com/sosy-lab/software/test-format</a> A test suite contains also metadata in order to relate it to the test problem for which it was produced.</li> <li><code>witnessInfoByHash/</code>: This directory contains for each test suite (witness) in directory witnessFileByHash/ a record in JSON format (also using the SHA2 256-bit hash of the witness as filename, with .json as filename extension) that contains the meta data.</li> <li><code>witnessListByProgramHashJSON/</code>: For convenient access to all test suites for a certain program, this directory represents a function that maps each program (via its SHA2256-bit hash) to a set of test suites (JSON records for test suites as described above) that the test tools have produced for that program. For each program for which test suites exist, the directory contains a JSON file (using the SHA2 256-bit hash of the program as filename, with .json as filename extension) that contains all JSON records for test suites for that program.</li> </ul> <p>A similar data structure was used by SV-COMP and is described in the following article:<br> Dirk Beyer. <em>A Data Set of Program Invariants and Error Paths.</em> In Proceedings of the 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR 2019, Montreal, Canada, May 26-27), pages 111-115, 2019. IEEE.<br> <a href="https://doi.org/10.1109/MSR.2019.00026">https://doi.org/10.1109/MSR.2019.00026</a></p> <p>Other Archives</p> <p>Overview over archives from Test-Comp 2021 that are available at Zenodo:</p> <ul> <li><a href="https://doi.org/10.5281/zenodo.4459466">https://doi.org/10.5281/zenodo.4459466</a> Witness store (containing the generated test suites)</li> <li><a href="https://doi.org/10.5281/zenodo.4459470">https://doi.org/10.5281/zenodo.4459470</a> Results (XML result files, log files, file mappings, HTML tables)</li> <li><a href="https://doi.org/10.5281/zenodo.4459132">https://doi.org/10.5281/zenodo.4459132</a> Test tasks, version testcomp21</li> <li><a href="https://doi.org/10.5281/zenodo.4317433">https://doi.org/10.5281/zenodo.4317433</a> BenchExec, version 3.6</li> </ul> <p>All benchmarks were executed for Test-Comp 2021 <a href="https://test-comp.sosy-lab.org/2021/">https://test-comp.sosy-lab.org/2021/</a><br> by Dirk Beyer, LMU Munich, based on the following components:</p> <ul> <li><a href="https://gitlab.com/sosy-lab/test-comp/archives-2021">https://gitlab.com/sosy-lab/test-comp/archives-2021</a> testcomp21-0-gdacd4bf</li> <li><a href="https://gitlab.com/sosy-lab/software/sv-benchmarks">https://gitlab.com/sosy-lab/software/sv-benchmarks</a> testcomp21-0-gefea738258</li> <li><a href="https://gitlab.com/sosy-lab/software/benchexec">https://gitlab.com/sosy-lab/software/benchexec</a> 3.6-0-gb278ebbb</li> <li><a href="https://gitlab.com/sosy-lab/benchmarking/competition-scripts">https://gitlab.com/sosy-lab/benchmarking/competition-scripts</a> testcomp21-0-g8339740</li> <li><a href="https://gitlab.com/sosy-lab/test-comp/bench-defs">https://gitlab.com/sosy-lab/test-comp/bench-defs</a> testcomp21-0-g9d532c9</li> </ul> <p>Contact</p> <p>Feel free to contact me in case of questions: <a href="https://www.sosy-lab.org/people/beyer/">https://www.sosy-lab.org/people/beyer/</a></p>