Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
52
datasets available to search
ShareScore release 0.9.0
Dataset results
52 results for “Test Suites”
swmm-nrtestsuite: Regression Test Suite for OWA SWMM
<p>Open Water Analytics (OWA) Stormwater Management Model (SWMM)</p> <p>Project Link: <br> https://github.com/OpenWaterAnalytics/swmm-nrtestsuite</p> <p> </p>
Nos_Machine translation test suite English-Galician
<p>364 english-galician parallel sentences for the evaluation of machine translation. Sentences are classified in categories based on challenging linguistic phenomena.</p>
Nos_Machine translation test suite Spanish-Galician
<p>334 spanish-galician parallel sentences for the evaluation of machine translation. Sentences are classified in categories based on challenging linguistic phenomena.</p>
Test Suites from Test-Generation Tools (Test-Comp 2019)
<p>This file describes the contents of an archive of the<br> 1st Competition on Software Testing (Test-Comp 2019)<br> <a href="https://test-comp.sosy-lab.org/2019/">https://test-comp.sosy-lab.org/2019/</a></p> <p>The competition was run by Dirk Beyer, LMU Munich, Germany.<br> More information is available in the following article:<br> Dirk Beyer. First International Competition on Software Testing: Test-Comp 2019.<br> International Journal on Software Tools for Technology Transfer, 2020.</p> <p>Copyright (C) Dirk Beyer<br> <a href="https://www.sosy-lab.org/people/beyer/">https://www.sosy-lab.org/people/beyer/</a></p> <p>SPDX-License-Identifier: CC-BY-4.0<br> <a href="https://spdx.org/licenses/CC-BY-4.0.html">https://spdx.org/licenses/CC-BY-4.0.html</a></p> <p> </p> <p>Contents:</p> <p>LICENSE.txt specifies the license<br> README.txt this file<br> witnessFileByHash/ This directory contains test suites (witnesses for coverage).<br> Each witness in this directory is stored in a file whose name is the SHA2 256-bit hash<br> of its contents followed by the filename extension .zip.<br> The format of each test suite is described on the format web page:<br> https://gitlab.com/sosy-lab/software/test-format<br> A test suite contains also metadata in order to relate it<br> to the test problem for which it was produced.<br> witnessInfoByHash/ This directory contains for each test suite (witness) in directory witnessFileByHash/<br> a record in JSON format (also using the SHA2 256-bit hash of the witness as filename,<br> with .json as filename extension) that contains the meta data.<br> witnessListByProgramHashJSON/<br> For convenient access to all test suites for a certain program, this directory represents<br> a function that maps each program (via its SHA2 256-bit hash) to a set of test suites<br> (JSON records for test suites as described above) that the test tools have produced<br> for that program. For each program for which test suites exist, the directory contains<br> a JSON file (using the SHA2 256-bit hash of the program as filename, with .json as<br> filename extension) that contains all JSON records for test suites for that program.</p> <p>A similar data structure was used by SV-COMP and is described in the following article:<br> Dirk Beyer. A Data Set of Program Invariants and Error Paths.<br> In Proceedings of the 2019 IEEE/ACM 16th International Conference on Mining Software Repositories<br> (MSR 2019, Montreal, Canada, May 26-27), pages 111-115, 2019. IEEE.<br> <a href="https://doi.org/10.1109/MSR.2019.00026">https://doi.org/10.1109/MSR.2019.00026</a></p> <p> </p> <p>Overview over archives from Test-Comp 2019 that are available at Zenodo:</p> <p><a href="https://doi.org/10.5281/zenodo.3856669">https://doi.org/10.5281/zenodo.3856669</a> Witness store (containing the generated test suites)<br> <a href="https://doi.org/10.5281/zenodo.3856661">https://doi.org/10.5281/zenodo.3856661</a> Results (XML result files, log files, file mappings, HTML tables)<br> <a href="https://doi.org/10.5281/zenodo.3856478">https://doi.org/10.5281/zenodo.3856478</a> Test tasks, version testcomp19<br> <a href="https://doi.org/10.5281/zenodo.2561835">https://doi.org/10.5281/zenodo.2561835</a> BenchExec, version 1.18</p> <p>All benchmarks were executed<br> for Test-Comp 2019, <a href="https://test-comp.sosy-lab.org/2019/">https://test-comp.sosy-lab.org/2019/</a><br> by Dirk Beyer, LMU Munich<br> based on the components<br> git@github.com:sosy-lab/sv-benchmarks.git testcomp19-0-g6a770a9c1<br> git@gitlab.com:sosy-lab/test-comp/bench-defs.git testcomp19-0-g1677027<br> git@github.com:sosy-lab/benchexec.git 1.18-0-gff72868</p> <p><br> Feel free to contact me in case of questions:<br> <a href="https://www.sosy-lab.org/people/beyer/">https://www.sosy-lab.org/people/beyer/</a></p> <p> </p>
Test Suites from Test-Generation Tools (Test-Comp 2021)
<p>Test Suites</p> <p>This file describes the contents of an archive of the 3rd Competition on Software Testing (Test-Comp 2021).<br> <a href="https://test-comp.sosy-lab.org/2021/">https://test-comp.sosy-lab.org/2021/</a></p> <p>The competition was run by Dirk Beyer, LMU Munich, Germany.<br> More information is available in the following article:<br> Dirk Beyer. <em>Status Report on Software Testing: Test-Comp 2021.</em> In Proceedings of the 24th International Conference on Fundamental Approaches to Software Engineering (FASE 2021, Luxembourg, March 27 - April 1), 2021. Springer.</p> <p>Copyright (C) Dirk Beyer<br> <a href="https://www.sosy-lab.org/people/beyer/">https://www.sosy-lab.org/people/beyer/</a></p> <p>SPDX-License-Identifier: CC-BY-4.0<br> <a href="https://spdx.org/licenses/CC-BY-4.0.html">https://spdx.org/licenses/CC-BY-4.0.html</a></p> <p>Contents</p> <ul> <li><code>LICENSE.txt</code>: specifies the license</li> <li><code>README.txt</code>: this file</li> <li><code>witnessFileByHash/</code>: This directory contains test suites (witnesses for coverage). Each witness in this directory is stored in a file whose name is the SHA2 256-bit hash of its contents followed by the filename extension .zip. The format of each test suite is described on the format web page: <a href="https://gitlab.com/sosy-lab/software/test-format">https://gitlab.com/sosy-lab/software/test-format</a> A test suite contains also metadata in order to relate it to the test problem for which it was produced.</li> <li><code>witnessInfoByHash/</code>: This directory contains for each test suite (witness) in directory witnessFileByHash/ a record in JSON format (also using the SHA2 256-bit hash of the witness as filename, with .json as filename extension) that contains the meta data.</li> <li><code>witnessListByProgramHashJSON/</code>: For convenient access to all test suites for a certain program, this directory represents a function that maps each program (via its SHA2256-bit hash) to a set of test suites (JSON records for test suites as described above) that the test tools have produced for that program. For each program for which test suites exist, the directory contains a JSON file (using the SHA2 256-bit hash of the program as filename, with .json as filename extension) that contains all JSON records for test suites for that program.</li> </ul> <p>A similar data structure was used by SV-COMP and is described in the following article:<br> Dirk Beyer. <em>A Data Set of Program Invariants and Error Paths.</em> In Proceedings of the 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR 2019, Montreal, Canada, May 26-27), pages 111-115, 2019. IEEE.<br> <a href="https://doi.org/10.1109/MSR.2019.00026">https://doi.org/10.1109/MSR.2019.00026</a></p> <p>Other Archives</p> <p>Overview over archives from Test-Comp 2021 that are available at Zenodo:</p> <ul> <li><a href="https://doi.org/10.5281/zenodo.4459466">https://doi.org/10.5281/zenodo.4459466</a> Witness store (containing the generated test suites)</li> <li><a href="https://doi.org/10.5281/zenodo.4459470">https://doi.org/10.5281/zenodo.4459470</a> Results (XML result files, log files, file mappings, HTML tables)</li> <li><a href="https://doi.org/10.5281/zenodo.4459132">https://doi.org/10.5281/zenodo.4459132</a> Test tasks, version testcomp21</li> <li><a href="https://doi.org/10.5281/zenodo.4317433">https://doi.org/10.5281/zenodo.4317433</a> BenchExec, version 3.6</li> </ul> <p>All benchmarks were executed for Test-Comp 2021 <a href="https://test-comp.sosy-lab.org/2021/">https://test-comp.sosy-lab.org/2021/</a><br> by Dirk Beyer, LMU Munich, based on the following components:</p> <ul> <li><a href="https://gitlab.com/sosy-lab/test-comp/archives-2021">https://gitlab.com/sosy-lab/test-comp/archives-2021</a> testcomp21-0-gdacd4bf</li> <li><a href="https://gitlab.com/sosy-lab/software/sv-benchmarks">https://gitlab.com/sosy-lab/software/sv-benchmarks</a> testcomp21-0-gefea738258</li> <li><a href="https://gitlab.com/sosy-lab/software/benchexec">https://gitlab.com/sosy-lab/software/benchexec</a> 3.6-0-gb278ebbb</li> <li><a href="https://gitlab.com/sosy-lab/benchmarking/competition-scripts">https://gitlab.com/sosy-lab/benchmarking/competition-scripts</a> testcomp21-0-g8339740</li> <li><a href="https://gitlab.com/sosy-lab/test-comp/bench-defs">https://gitlab.com/sosy-lab/test-comp/bench-defs</a> testcomp21-0-g9d532c9</li> </ul> <p>Contact</p> <p>Feel free to contact me in case of questions: <a href="https://www.sosy-lab.org/people/beyer/">https://www.sosy-lab.org/people/beyer/</a></p>
TREXIO files used for the validation tests in the paper entitled 'TurboGenius: Python suite for high-throughput calculations of ab initio quantum Monte Carlo methods'.
<p>The TREXIO files used for the validation tests in the paper entitled TurboGenius: Python suite for high-throughput calculations of ab initio quantum Monte Carlo methods. The detail about the TREXIO library is described in the JCP article [J. Chem. Phys. 158, 174801 (2023)] and the GitHub repository [https://github.com/TREX-CoE/trexio]. The TREXIO files were generated using TREXIO version 2.3.2 (and the corresponding Python API version 1.3.2).</p>
Survey Protocol - A Metrics suite for End-to-End Microservice Test Coverage
<p>These survey is performed and reported in a paper titled "A Metrics suite for End-to-End Microservice Test Coverage".</p>
Earthquake Early Warning Global Test Suite
<p>This is the companion data to the JGR paper "Quantifying the Value of<br> Real-time Geodetic Constraints on Earthquake Early Warning using a<br> Global Seismic and Geodetic Dataset" by Ruhl et al. It contains both<br> strong motion and GNSS displacement waveforms. There is one folder per<br> event, and for each there is an "accel" and a "disp" file containing<br> each kind of data. There is a .chan channel file with station metadata.</p> <p>Changes from Version 1.0:</p> <p>The timing of seismic waveforms for Cascadia001300 were delayed by 1<br> minute and this has been corrected to match the geodetic data and the<br> origin time.</p> <p>Four of the Japanese events were mistakenly in GPS time and have now<br> been corrected into UTC time to match the seismic data. Affected events<br> include Tohoku2011, Miyagi2011B, E.Fukushima2011, and Kumamoto2016.</p>
Adequate vs. Inadequate Test Suite Reduction Approaches. Raw Data
<p>Context: Regression testing is an important activity that allows ensuring the correct behavior of a system after a change. As the system grows, the time and resources to perform regression testing increase. Test Suite Reduction (TSR) approaches aim to speed up regression testing by removing obsolete or redundant test cases. These approaches can be classified as adequate or inadequate. Adequate TSR approaches reduce test suites and completely preserve test requirements (e.g., covered statements) of the original test suites. Inadequate TSR approaches do not preserve test requirements. The percentage of satisfed test requirements indicates the inadequacy level.</p> <p>Objective: We compare some state-of-the art adequate and inadequate TSR approaches with respect to the size of the reduced test suites and their fault-detection capability. Specifcally, we aim to increase our body of knowledge on TSR approaches by performing the following comparisons: (i) well-known adequate TSR approaches; (ii) their inadequate variants; and (iii) several variants of a novel Clustering-Based (CB) approach for (adequate and inadequate) TSR.</p> <p>Method: We conducted an experiment to compare adequate and inadequate TSR approaches and this comparison is founded on a public dataset containing information on real faults.</p> <p>Results: The most important findings from our experiment can be summarized as follows: (i) there is not an inadequate TSR approach that performs better than others; (ii) some variants of the CB approach, and a few well-known inadequate approaches, outperform the adequate ones in terms of the reductions in test suite size with a negligible, or no, eect on fault-detection capability; and (iii) the CB approach is less sensitive than the other inadequate approaches, that is, variations in the inadequacy level have a small effect on the reduction in test suite size and on the loss in fault-detection capability.</p> <p>Conclusions: These findings imply that inadequate TSR approaches and especially the CB approach might be appealing because they lead to a greater reduction in test suite size (with respect to the adequate ones) at the expense of a small loss in fault-detection capability.</p>
Test Suites from Test-Generation Tools (Test-Comp 2020)
<p>This archive contains the test suites that were generated during the<br> 2nd Competition on Software Testing (Test-Comp 2020)<br> <a href="https://test-comp.sosy-lab.org/2020/">https://test-comp.sosy-lab.org/2020/</a></p> <p>The competition was run by Dirk Beyer, LMU Munich, Germany.<br> More information is available in the following article:<br> Dirk Beyer. Second Competition on Software Testing: Test-Comp 2020.<br> In Proceedings of the 23rd International Conference on Fundamental Approaches to Software Engineering<br> (FASE 2020, Dublin, April 28-30), 2020. Springer.<br> <a href="https://doi.org/10.1007/978-3-030-45234-6_25">https://doi.org/10.1007/978-3-030-45234-6_25</a></p> <p>Copyright (C) Dirk Beyer<br> <a href="https://www.sosy-lab.org/people/beyer/">https://www.sosy-lab.org/people/beyer/</a></p> <p>SPDX-License-Identifier: CC-BY-4.0<br> <a href="https://spdx.org/licenses/CC-BY-4.0.html">https://spdx.org/licenses/CC-BY-4.0.html</a></p> <p> </p> <p>Contents:</p> <p>LICENSE.txt specifies the license<br> README.txt this file<br> witnessFileByHash/ This directory contains test suites (witnesses for coverage).<br> Each witness in this directory is stored in a file whose name is the SHA2 256-bit hash<br> of its contents followed by the filename extension .zip.<br> The format of each test suite is described on the format web page:<br> https://gitlab.com/sosy-lab/software/test-format<br> A test suite contains also metadata in order to relate it<br> to the test problem for which it was produced.<br> witnessInfoByHash/ This directory contains for each test suite (witness) in directory witnessFileByHash/<br> a record in JSON format (also using the SHA2 256-bit hash of the witness as filename,<br> with .json as filename extension) that contains the meta data.<br> witnessListByProgramHashJSON/<br> For convenient access to all test suites for a certain program, this directory represents<br> a function that maps each program (via its SHA2256-bit hash) to a set of test suites<br> (JSON records for test suites as described above) that the test tools have produced<br> for that program. For each program for which test suites exist, the directory contains<br> a JSON file (using the SHA2 256-bit hash of the program as filename, with .json as<br> filename extension) that contains all JSON records for test suites for that program.</p> <p>A similar data structure was used by SV-COMP and is described in the following article:<br> Dirk Beyer. A Data Set of Program Invariants and Error Paths.<br> In Proceedings of the 2019 IEEE/ACM 16th International Conference on Mining Software Repositories<br> (MSR 2019, Montreal, Canada, May 26-27), pages 111-115, 2019. IEEE.<br> <a href="https://doi.org/10.1109/MSR.2019.00026">https://doi.org/10.1109/MSR.2019.00026</a></p> <p> </p> <p>Overview over archives from Test-Comp 2020 that are available at Zenodo:</p> <p><a href="https://doi.org/10.5281/zenodo.3678275">https://doi.org/10.5281/zenodo.3678275</a> Witness store (containing the generated test suites)<br> <a href="https://doi.org/10.5281/zenodo.3678264">https://doi.org/10.5281/zenodo.3678264</a> Results (XML result files, log files, file mappings, HTML tables)<br> <a href="https://doi.org/10.5281/zenodo.3678250">https://doi.org/10.5281/zenodo.3678250</a> Test tasks, version testcomp20<br> <a href="https://doi.org/10.5281/zenodo.3574420">https://doi.org/10.5281/zenodo.3574420</a> BenchExec, version 2.5.1</p> <p>All benchmarks were executed<br> for Test-Comp 2020, <a href="https://test-comp.sosy-lab.org/2020/">https://test-comp.sosy-lab.org/2020/</a><br> by Dirk Beyer, LMU Munich<br> based on the components<br> git@github.com:sosy-lab/sv-benchmarks.git testcomp20-0-gd6cd3e5dd4<br> git@gitlab.com:sosy-lab/test-comp/bench-defs.git testcomp19-84-gac76836<br> git@github.com:sosy-lab/benchexec.git 2.5.1-0-gffad635</p> <p><br> Feel free to contact me in case of questions:<br> <a href="https://www.sosy-lab.org/people/beyer/">https://www.sosy-lab.org/people/beyer/</a></p>
SBML Test Suite Syntactic Test Cases 3.2.0
<p>The SBML Test Suite is a conformance testing system. It allows developers and users to test the degree and correctness of the SBML support provided in a software package. A core part of the SBML Test Suite is the collection of test cases. There are 3 sets of tests: <strong>semantic</strong> (for deterministic simulation behavior), <strong>stochastic</strong>(for stochastic simulation behavior), and <strong>syntactic</strong> (for basic parsing).</p> <p>This is the version 3.2.0 release of the s<strong>yntactic</strong> test cases archive.</p> <p>For more information about SBML and the SBML Test Suite, please visit http://sbml.org.</p>
SBML Test Suite Semantic Test Cases 3.2.0
<p>The SBML Test Suite is a conformance testing system. It allows developers and users to test the degree and correctness of the SBML support provided in a software package. A core part of the SBML Test Suite is the collection of test cases. There are 3 sets of tests: <strong>semantic</strong> (for deterministic simulation behavior), <strong>stochastic</strong>(for stochastic simulation behavior), and <strong>syntactic</strong> (for basic parsing).</p> <p>This is the version 3.2.0 release of the <strong>semantic</strong> test cases archive.</p> <p>For more information about SBML and the SBML Test Suite, please visit http://sbml.org.</p>
SBML Test Suite Stochastic Test Cases 3.2.0
<p>The SBML Test Suite is a conformance testing system. It allows developers and users to test the degree and correctness of the SBML support provided in a software package. A core part of the SBML Test Suite is the collection of test cases. There are 3 sets of tests: <strong>semantic</strong> (for deterministic simulation behavior), <strong>stochastic</strong>(for stochastic simulation behavior), and <strong>syntactic</strong> (for basic parsing).</p> <p>This is the version 3.2.0 release of the <strong>stochastic</strong> test cases archive.</p> <p>For more information about SBML and the SBML Test Suite, please visit http://sbml.org.</p>
Test Suites from Test-Generation Tools (Test-Comp 2022)
<p>Test-Comp 2022</p> <p>Test Suites</p> <p>This file describes the contents of an archive of the 4th Competition on Software Testing (Test-Comp 2022).<br> <a href="https://test-comp.sosy-lab.org/2022/">https://test-comp.sosy-lab.org/2022/</a></p> <p>The competition was run by Dirk Beyer, LMU Munich, Germany.<br> More information is available in the following article:<br> Dirk Beyer. <em>Advances in Automatic Software Testing: Test-Comp 2022.</em> In Proceedings of the 25th International Conference on Fundamental Approaches to Software Engineering (FASE 2022, Munich, April 2 - 7), 2021. Springer.</p> <p>Copyright (C) Dirk Beyer<br> <a href="https://www.sosy-lab.org/people/beyer/">https://www.sosy-lab.org/people/beyer/</a></p> <p>SPDX-License-Identifier: CC-BY-4.0<br> <a href="https://spdx.org/licenses/CC-BY-4.0.html">https://spdx.org/licenses/CC-BY-4.0.html</a></p> <p>Contents</p> <ul> <li><code>LICENSE.txt</code>: specifies the license</li> <li><code>README.txt</code>: this file</li> <li><code>witnessFileByHash/</code>: This directory contains test suites (witnesses for coverage). Each test witness in this directory is stored in a file whose name is the SHA2 256-bit hash of its contents followed by the filename extension .zip. The format of each test suite is described on the format web page: <a href="https://gitlab.com/sosy-lab/software/test-format">https://gitlab.com/sosy-lab/software/test-format</a> A test suite contains also metadata in order to relate it to the test task for which it was produced.</li> <li><code>witnessInfoByHash/</code>: This directory contains for each test suite (witness) in directory witnessFileByHash/ a record in JSON format (also using the SHA2 256-bit hash of the witness as filename, with .json as filename extension) that contains the meta data.</li> <li><code>witnessListByProgramHashJSON/</code>: For convenient access to all test suites for a certain program, this directory represents a function that maps each program (via its SHA2256-bit hash) to a set of test suites (JSON records for test suites as described above) that the test-generation tools have produced for that program. For each program for which test suites exist, the directory contains a JSON file (using the SHA2 256-bit hash of the program as filename, with .json as filename extension) that contains all JSON records for test suites for that program.</li> </ul> <p>A similar data structure was used by SV-COMP and is described in the following article:<br> Dirk Beyer. <em>A Data Set of Program Invariants and Error Paths.</em> In Proceedings of the 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR 2019, Montreal, Canada, May 26-27), pages 111-115, 2019. IEEE.<br> <a href="https://doi.org/10.1109/MSR.2019.00026">https://doi.org/10.1109/MSR.2019.00026</a></p> <p>Other Archives</p> <p>Overview over archives from Test-Comp 2022 that are available at Zenodo:</p> <ul> <li><a href="https://doi.org/10.5281/zenodo.5831010">https://doi.org/10.5281/zenodo.5831010</a> Test Suites from Test-Comp 2022 Test-Generation Tools. Witness store (containing the generated test suites)</li> <li><a href="https://doi.org/10.5281/zenodo.5831012">https://doi.org/10.5281/zenodo.5831012</a> Results of the 4th Intl. Competition on Software Testing (Test-Comp 2022). Results (XML result files, log files, file mappings, HTML tables)</li> <li><a href="https://doi.org/10.5281/zenodo.5831003">https://doi.org/10.5281/zenodo.5831003</a> SV-Benchmarks: Benchmark Set of SV-COMP 2022 and Test-Comp 2022. Test tasks, version testcomp22</li> <li><a href="https://doi.org/10.5281/zenodo.5720267">https://doi.org/10.5281/zenodo.5720267</a> BenchExec, version 3.10. Benchmarking framework</li> </ul> <p>All benchmarks were executed for Test-Comp 2022 <a href="https://test-comp.sosy-lab.org/2022/">https://test-comp.sosy-lab.org/2022/</a><br> by Dirk Beyer, LMU Munich, based on the following components:</p> <ul> <li><a href="https://gitlab.com/sosy-lab/test-comp/archives-2022">https://gitlab.com/sosy-lab/test-comp/archives-2022</a> testcomp22 c211efbc</li> <li><a href="https://gitlab.com/sosy-lab/benchmarking/sv-benchmarks">https://gitlab.com/sosy-lab/benchmarking/sv-benchmarks</a> testcomp22 ad265d07</li> <li><a href="https://gitlab.com/sosy-lab/test-comp/bench-defs">https://gitlab.com/sosy-lab/test-comp/bench-defs</a> testcomp22 93f581b9</li> <li><a href="https://gitlab.com/sosy-lab/software/benchexec">https://gitlab.com/sosy-lab/software/benchexec</a> 3.10 4e8716bd</li> <li><a href="https://gitlab.com/sosy-lab/benchmarking/competition-scripts">https://gitlab.com/sosy-lab/benchmarking/competition-scripts</a> testcomp22 2022c698</li> <li><a href="https://gitlab.com/sosy-lab/test-comp/test-format">https://gitlab.com/sosy-lab/test-comp/test-format</a> testcomp22 2895be41</li> </ul> <p>Contact</p> <p>Feel free to contact me in case of questions: <a href="https://www.sosy-lab.org/people/beyer/">https://www.sosy-lab.org/people/beyer/</a></p>
LTM: Scalable and Black-box Similarity-based Test Suite Minimization based on Language Models - Replication Package
<p>LTM: Scalable and Black-box Similarity-based Test Suite Minimization based on Language Models</p> <p>This is the replication package associated with the paper "LTM: Scalable and Black-box Similarity-based Test Suite Minimization based on Language Models".</p> <p><strong>Replication Package Contents:</strong></p> <p>This replication package contains all the necessary data and code required to reproduce the results reported in the paper. We provide the results of the Fault Detection Rate (FDR), Total Minimization Time (MT), Time Saving Rate (TSR) , statistical tests for all the minimization budgets (i.e., 25%, 50%, and 75%), results for the preliminary study, results for UniXcoder/Cosine with preprocessed code on 16 projects.</p> <p><strong>Data:</strong></p> <p>We provide in the <em><strong>Data</strong></em> directory the data used in our experiments, which is the source code of test cases (Java test methods) of 17 projects collected from Defects4J.</p> <p><strong>Code:</strong></p> <p>We provide in the<em> <strong>Code</strong> </em>directory the code (Python) and bash files required to run the experiments and reproduce the results.</p> <p><strong>Results:</strong></p> <p>We provide in the<em> <strong>Results </strong></em>directory the detailed results for our approach (called LTM). We also provide the summarized results of LTM and a baseline (ATM) for comparison purposes. Additional technical details about ATM can be found at https://zenodo.org/record/7455766.</p> <p><strong>_________________________________</strong></p> <p><strong>LTM's Similarity Measurement:</strong></p> <p>The source code of this step is in the <strong><em>Code/LTM/Similarity</em></strong> directory.</p> <p><strong>Requirements:</strong></p> <p>To run this step, Python 3 is required (we used Python 3.10). Also, the required libraries in the <em><strong>Code/LTM/Similarity/requirements.txt</strong></em> file should be installed, as follows:</p> <p>cd Code/LTM/Similarity</p> <p>pip install -r requirements.txt</p> <p><strong>Input:</strong></p> <ul> <li>Data/LTM/TestMethods</li> </ul> <p><strong>Output:</strong></p> <ul> <li>Data/LTM/similarity_measurements</li> </ul> <p><strong>Running the experiment:</strong></p> <p>To measure the similarity between all pairs of test cases, the following bash script should be executed:</p> <p>bash measure_similarity.sh</p> <p>The source code of test methods of each project in the <strong><em>Data/LTM/TestMethods</em></strong> is parsed to generate pairs of test cases. This steps includes test methods tokenization, test methods embeddings extraction and similarity calculation. Then, all similarity scores are stored in <em><strong>Data/LTM/similarity_measurements</strong></em> folder. Due to the large size of the calculated similarity scores (60 GB), they were not uploaded on Zenodo, but they can be available upon request.</p> <p><strong>LTM's Test Suite Minimization:</strong></p> <p>The source code of this step is in the Code/LTM/Search directory.</p> <p><strong>Requirements:</strong></p> <p>To run this step, Python 3 is required (we used Python 3.10). Also, the required libraries in the <strong><em>Code/LTM/Search/requirements.txt</em></strong> file should be installed, as follows:</p> <p>cd Code/LTM/Search</p> <p>pip install -r requirements.txt</p> <p><strong>Input:</strong></p> <p>Data/LTM/similarity_measurements</p> <p><strong>Output:</strong></p> <p>Results/LTM/minimization_results</p> <p><strong>Running the experiments:</strong></p> <p>To minimize the test suite for each project version, the following bash script should be executed:</p> <p>bash minimize.sh</p> <p>The similarity scores of all test case pairs per project version are parsed by the search algorithm (Genetic Algorithm). Each experiment runs ten times using three minimization budgets (25%, 50%, and 75%). The results are stored in the <em><strong>Results/LTM/minimization_results</strong></em> directory.</p> <p><strong>LTM's Evaluation:</strong></p> <p>To evaluate the minimization results for each version and each project, the following bash script should be executed:</p> <p>cd Code/LTM/Evaluation</p> <p>bash evaluate_per_version.sh</p> <p>cd Code/LTM/Evaluation</p> <p>bash evaluate_per_project.sh</p> <p>This will evaluate the FDR, MT and TSR results for each version and each project for each minimization budget. These results are stored in the <em><strong>Results/LTM</strong></em> directory.</p> <p>Note that for each version, the FDR is either 1 or 0. For each project, the FDR ranges from 0 to 1.</p>
Openbenchmarking - Phoronix test suite dataset
<p>Data extracted from the openbenchmarking.org website, see https://openbenchmarking.org/</p> <p>Unzip the data.zip archive to get the raw data.</p> <p>The dataset is two-fold:</p> <p>- The performances of about 400 test profiles (or test suites) in the `suite` directory. Each suite has its own .csv file e.g. pts-ai-benchmark.csv related to the table of performance of this page : https://openbenchmarking.org/test/pts/ai-benchmark . We also provide a dataframe merge.csv with all the raw data,</p> <p>- Hardware metadata, in the `hardware` directory. Each hardware has its own directory, e.g. "AMDEPYC7F328-Core" for the AMD EPYC 7F32 8-Core Processor. This folder contains the output of the 'lscpu' and 'cat /proc/cpuinfo' commands, see the lscpu.txt and the catproc.txt files.</p>
Test Suites from Test-Generation Tools (Test-Comp 2023)
<p><strong>Test-Comp 2023</strong></p> <p><strong>Test Suites</strong></p> <p>This file describes the contents of an archive of the 5th Competition on Software Testing (Test-Comp 2023). <a href="https://test-comp.sosy-lab.org/2023/">https://test-comp.sosy-lab.org/2023/</a></p> <p>The competition was organized by Dirk Beyer, LMU Munich, Germany. More information is available in the following article: Dirk Beyer. <em>Software Testing: 5th Comparative Evaluation: Test-Comp 2023.</em> In Proceedings of the 26th International Conference on Fundamental Approaches to Software Engineering (FASE 2023, Paris, April 22 - 27), 2023. Springer.</p> <p>Copyright (C) Dirk Beyer <a href="https://www.sosy-lab.org/people/beyer/">https://www.sosy-lab.org/people/beyer/</a></p> <p>SPDX-License-Identifier: CC-BY-4.0 <a href="https://spdx.org/licenses/CC-BY-4.0.html">https://spdx.org/licenses/CC-BY-4.0.html</a></p> <p><strong>Contents</strong></p> <ul> <li><code>LICENSE.txt</code>: specifies the license</li> <li><code>README.txt</code>: this file</li> <li><code>witnessFileByHash/</code>: This directory contains test suites (witnesses for coverage). Each test witness in this directory is stored in a file whose name is the SHA2 256-bit hash of its contents followed by the filename extension .zip. The format of each test suite is described on the format web page: <a href="https://gitlab.com/sosy-lab/software/test-format">https://gitlab.com/sosy-lab/software/test-format</a> A test suite contains also metadata in order to relate it to the test task for which it was produced.</li> <li><code>witnessInfoByHash/</code>: This directory contains for each test suite (witness) in directory witnessFileByHash/ a record in JSON format (also using the SHA2 256-bit hash of the witness as filename, with .json as filename extension) that contains the meta data.</li> <li><code>witnessListByProgramHashJSON/</code>: For convenient access to all test suites for a certain program, this directory represents a function that maps each program (via its SHA2256-bit hash) to a set of test suites (JSON records for test suites as described above) that the test-generation tools have produced for that program. For each program for which test suites exist, the directory contains a JSON file (using the SHA2 256-bit hash of the program as filename, with .json as filename extension) that contains all JSON records for test suites for that program.</li> </ul> <p>A similar data structure was used by SV-COMP and is described in the following article: Dirk Beyer. <em>A Data Set of Program Invariants and Error Paths.</em> In Proceedings of the 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR 2019, Montreal, Canada, May 26-27), pages 111-115, 2019. IEEE. <a href="https://doi.org/10.1109/MSR.2019.00026">https://doi.org/10.1109/MSR.2019.00026</a></p> <p><strong>Other Archives</strong></p> <p>Overview of archives from Test-Comp 2023 that are available at Zenodo:</p> <ul> <li><a href="https://doi.org/10.5281/zenodo.7701126">https://doi.org/10.5281/zenodo.7701126</a> Test Suites from Test-Comp 2023 Test-Generation Tools. Store of coverage witnesses (containing the generated test suites)</li> <li><a href="https://doi.org/10.5281/zenodo.7701118">https://doi.org/10.5281/zenodo.7701118</a> Test-Suite Generators from Test-Comp 2023. Executable tools that were evaluated in the competition</li> <li><a href="https://doi.org/10.5281/zenodo.7701122">https://doi.org/10.5281/zenodo.7701122</a> Results of the 5th Intl. Competition on Software Testing (Test-Comp 2023). Results (XML result files, log files, file mappings, HTML tables)</li> <li><a href="https://doi.org/10.5281/zenodo.7627783">https://doi.org/10.5281/zenodo.7627783</a> SV-Benchmarks: Benchmark Set of SV-COMP 2023 and Test-Comp 2023. Verification tasks, version testcomp23</li> <li><a href="https://doi.org/10.5281/zenodo.7612021">https://doi.org/10.5281/zenodo.7612021</a> BenchExec, version 3.16. Benchmarking framework</li> <li><a href="https://doi.org/10.5281/zenodo.7635975">https://doi.org/10.5281/zenodo.7635975</a> CoVeriTeam, version 1.0. Remote execution and continuous integration of testers</li> </ul> <p>All benchmarks were executed for Test-Comp 2023 <a href="https://test-comp.sosy-lab.org/2023/">https://test-comp.sosy-lab.org/2023/</a> by Dirk Beyer, LMU Munich, based on the following components:</p> <ul> <li><a href="https://gitlab.com/sosy-lab/test-comp/archives-2023">https://gitlab.com/sosy-lab/test-comp/archives-2023</a> testcomp23 bd9d84151c52ba7ad549212a590e2b3a18c071bc</li> <li><a href="https://gitlab.com/sosy-lab/benchmarking/sv-benchmarks">https://gitlab.com/sosy-lab/benchmarking/sv-benchmarks</a> testcomp23 f87854c275a177e8dc3673fe6959cd89038652af</li> <li><a href="https://gitlab.com/sosy-lab/test-comp/bench-defs">https://gitlab.com/sosy-lab/test-comp/bench-defs</a> testcomp23 21a5ba27dc84c8e59fefe18369b5723550c96786</li> <li><a href="https://gitlab.com/sosy-lab/software/benchexec">https://gitlab.com/sosy-lab/software/benchexec</a> 3.16 b43e7384112770a41b571399135065daa7f387ab</li> <li><a href="https://gitlab.com/sosy-lab/benchmarking/competition-scripts">https://gitlab.com/sosy-lab/benchmarking/competition-scripts</a> testcomp23 e6f84b5a79626b4295108a0f96ea370edde1f023</li> <li><a href="https://gitlab.com/sosy-lab/test-comp/test-format">https://gitlab.com/sosy-lab/test-comp/test-format</a> testcomp23 e6b0eff82a5132a6e4e4b52f8fbfaa3553af3ebb</li> <li><a href="https://gitlab.com/sosy-lab/software/coveriteam">https://gitlab.com/sosy-lab/software/coveriteam</a> 1.0 78d9f9f9d5fefc4fdd934d1e0f43ddba4ec51711</li> </ul> <p><strong>Contact</strong></p> <p>Feel free to contact me in case of questions: <a href="https://www.sosy-lab.org/people/beyer/">https://www.sosy-lab.org/people/beyer/</a></p>
Small Test Suites for Active Automata Learning: Supplemental Material
<p>Supplemental material for the paper: Small Test Suites for Active Automata Learning.</p> <p>Submitted to TACAS 2024.</p>
Skeleton Test Suite and PRONOM Archive v73
<p>Skeleton Test Suite and PRONOM Archive v73</p>
The benchmark test suites of the FJSP-MW (MATLAB version)
<p>Two benchmark test suites are provided in this file.</p> <p>LLG-A1--LLGA20 (the multitasking level is less than 3)<br>LLG-B1--LLGB20 (the multitasking level is less than 5)</p> <p><br>------------------------------------------------------</p> <p>The meaning of the main variables:<br>1. dt: The delivery time of each job<br>2. Jm: The eligible machine set of each operation.<br>3. JmNumber: The total number of the machines<br>4. Jw: The eligible worker set of each operation.<br>5. multitask: The multitasking level of each worker (The multitasking level of the worker is 3 indicates that this worker can process three operations at the same time).<br>6. pjob: The priority of each job.<br>7. rt: The release time of each job.<br>8. T: The processing time of each operation.</p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.