Results of the 4th Intl. Competition on Software Testing (Test-Comp 2022)

<p>Test-Comp 2022</p> <p>Competition Results</p> <p>This file describes the contents of an archive of the 4th Competition on Software Testing (Test-Comp 2022).<br> <a href="https://test-comp.sosy-lab.org/2022/">https://test-comp.sosy-lab.org/2022/</a></p> <p>The competition was run by Dirk Beyer, LMU Munich, Germany.<br> More information is available in the following article:<br> Dirk Beyer. <em>Advances in Automatic Software Testing: Test-Comp 2022.</em> In Proceedings of the 25th International Conference on Fundamental Approaches to Software Engineering (FASE 2022, Munich, April 2 - 7), 2021. Springer.</p> <p>Copyright (C) Dirk Beyer<br> <a href="https://www.sosy-lab.org/people/beyer/">https://www.sosy-lab.org/people/beyer/</a></p> <p>SPDX-License-Identifier: CC-BY-4.0<br> <a href="https://spdx.org/licenses/CC-BY-4.0.html">https://spdx.org/licenses/CC-BY-4.0.html</a></p> <p>To browse the competition results with a web browser, there are two options:</p> <ul> <li>start a local web server using php -S localhost:8000 in order to view the data in this archive, or</li> <li>browse <a href="https://test-comp.sosy-lab.org/2022/results/">https://test-comp.sosy-lab.org/2022/results/</a> in order to view the data on the Test-Comp web page.</li> </ul> <p>Contents</p> <ul> <li><code>index.html</code>: directs to the overview web page</li> <li><code>LICENSE.txt</code>: specifies the license</li> <li><code>README.txt</code>: this file</li> <li><code>results-validated/</code>: results of validation runs</li> <li><code>results-verified/</code>: results of test-generation runs and aggregated results</li> </ul> <p>The folder <code>results-validated/</code> contains the results from validation runs:</p> <ul> <li><code>*.xml.bz2</code>: XML results from BenchExec</li> <li><code>*.logfiles.zip</code>: output from tools</li> <li><code>*.json.gz</code>: mapping from files names to SHA 256 hashes for the file content</li> </ul> <p>The folder <code>results-verified/</code> contains the results from test-generation runs and aggregated results:</p> <ul> <li> <p><code>index.html</code>: overview web page with rankings and score table</p> </li> <li> <p><code>design.css</code>: HTML style definitions</p> </li> <li> <p><code>*.xml.bz2</code>: XML results from BenchExec</p> </li> <li> <p><code>*.merged.xml.bz2</code>: XML results from BenchExec, status adjusted according to the validation results</p> </li> <li> <p><code>*.logfiles.zip</code>: output from tools</p> </li> <li> <p><code>*.json.gz</code>: mapping from files names to SHA 256 hashes for the file content</p> </li> <li> <p><code>*.xml.bz2.table.html</code>: HTML views on the detailed results data as generated by BenchExec&rsquo;s table generator</p> </li> <li> <p><code>*.All.table.html</code>: HTML views of the full benchmark set (all categories) for each tool</p> </li> <li> <p><code>META_*.table.html</code>: HTML views of the benchmark set for each meta category for each tool, and over all tools</p> </li> <li> <p><code>&lt;category&gt;*.table.html</code>: HTML views of the benchmark set for each category over all tools</p> </li> <li> <p><code>iZeCa0gaey.html</code>: HTML views per tool</p> </li> <li> <p><code>quantilePlot-*</code>: score-based quantile plots as visualization of the results</p> </li> <li> <p><code>quantilePlotShow.gp</code>: example Gnuplot script to generate a plot</p> </li> <li> <p><code>score*</code>: accumulated score results in various formats</p> </li> </ul> <p>The hashes of the file names (in the files <code>*.json.gz</code>) are useful for</p> <ul> <li>validating the exact contents of a file and</li> <li>accessing the files from the witness store.</li> </ul> <p>Other Archives</p> <p>Overview over archives from Test-Comp 2022 that are available at Zenodo:</p> <ul> <li><a href="https://doi.org/10.5281/zenodo.5831010">https://doi.org/10.5281/zenodo.5831010</a> Test Suites from Test-Comp 2022 Test-Generation Tools. Witness store (containing the generated test suites)</li> <li><a href="https://doi.org/10.5281/zenodo.5831012">https://doi.org/10.5281/zenodo.5831012</a> Results of the 4th Intl. Competition on Software Testing (Test-Comp 2022). Results (XML result files, log files, file mappings, HTML tables)</li> <li><a href="https://doi.org/10.5281/zenodo.5831003">https://doi.org/10.5281/zenodo.5831003</a> SV-Benchmarks: Benchmark Set of SV-COMP 2022 and Test-Comp 2022. Test tasks, version testcomp22</li> <li><a href="https://doi.org/10.5281/zenodo.5720267">https://doi.org/10.5281/zenodo.5720267</a> BenchExec, version 3.10. Benchmarking framework</li> </ul> <p>All benchmarks were executed for Test-Comp 2022 <a href="https://test-comp.sosy-lab.org/2022/">https://test-comp.sosy-lab.org/2022/</a> by Dirk Beyer, LMU Munich, based on the following components:</p> <ul> <li><a href="https://gitlab.com/sosy-lab/test-comp/archives-2022">https://gitlab.com/sosy-lab/test-comp/archives-2022</a> testcomp22 c211efbc</li> <li><a href="https://gitlab.com/sosy-lab/benchmarking/sv-benchmarks">https://gitlab.com/sosy-lab/benchmarking/sv-benchmarks</a> testcomp22 ad265d07</li> <li><a href="https://gitlab.com/sosy-lab/test-comp/bench-defs">https://gitlab.com/sosy-lab/test-comp/bench-defs</a> testcomp22 93f581b9</li> <li><a href="https://gitlab.com/sosy-lab/software/benchexec">https://gitlab.com/sosy-lab/software/benchexec</a> 3.10 4e8716bd</li> <li><a href="https://gitlab.com/sosy-lab/benchmarking/competition-scripts">https://gitlab.com/sosy-lab/benchmarking/competition-scripts</a> testcomp22 2022c698</li> <li><a href="https://gitlab.com/sosy-lab/test-comp/test-format">https://gitlab.com/sosy-lab/test-comp/test-format</a> testcomp22 2895be41</li> </ul> <p>Contact</p> <p>Feel free to contact me in case of questions: <a href="https://www.sosy-lab.org/people/beyer/">https://www.sosy-lab.org/people/beyer/</a></p>