ATM: Black-box Test Case Minimization based on Test Code Similarity and Evolutionary Search – Replication Package
<p>This is the replication package associated with the paper "<em>ATM: Black-box Test Case Minimization based on Test Code Similarity and Evolutionary Search</em>" accepted at the 45th IEEE/ACM International Conference on Software Engineering (ICSE 2023) – Technical Track. Cite this paper using the following:</p>
<p><em>@inproceedings{pan2023atm,<br>
title={ATM: Black-box Test Case Minimization based on Test Code Similarity and Evolutionary Search},<br>
author={Pan, Rongqi and Ghaleb, Taher A. and Briand, Lionel},<br>
booktitle={Proceedings of the 45th IEEE/ACM International Conference on Software Engineering},<br>
year={2023},<br>
pages={1--12}<br>
}</em></p>
<p><strong>Replication Package Contents:</strong><br>
The replication package contains all the necessary data and code required to reproduce the results reported in the paper. We also provide the results for other minimization budgets, and detailed <em>FDR,</em> execution time, and statistical test results. In addition, we provide the data and code required to reproduce the results of baselines techniques: FAST-R and random minimization.</p>
<p><strong>Data:</strong><br>
We provide in the <em><strong>Data</strong></em> directory the data used in our experiments, which is based on 16 projects from <a href="https://github.com/rjust/defects4j">Defects4J</a>, whose characteristics can be found in <em><strong>Data/subject_projects.csv</strong></em><em>.</em></p>
<p><strong>Code:</strong><br>
We provide in the <em><strong>Code</strong></em> directory the code and scripts (Java, Python, and Bash) required to run the experiments and reproduce the results.</p>
<p><strong>Results:</strong><br>
We provide in the <em><strong>Results</strong></em> directory the results for each technique independently, and also a summary of all results together for comparison purposes. The source code for this step is in the <em><strong>Code/ATM/CodeToAST</strong></em> directory. The source code for this step is in the <em><strong>Code/ATM/Similarity</strong></em> directory.</p>
<p><strong>_________________________________</strong></p>
<p><strong>ATM - Code to AST transformation:</strong></p>
<p><strong>Requirements:</strong><br>
* Eclipse IDE (we used 2021-12)<br>
* The libraries (the <em><strong>.jar</strong></em> files in the <em><strong>Code/ATM/CodeToAST/lib</strong></em> directory)</p>
<p><strong>Input:</strong><br>
All zipped data files should be unzipped before running each step.<br>
* Data/test_suites/all_test_cases.zip → Data/test_suites/all_test_cases<br>
* Data/test_suites/changed_test_cases.zip → Data/test_suites/changed_test_cases<br>
* Data/test_suites/relevant_test_cases.zip → Data/test_suites/relevant_test_cases</p>
<p><strong>Output:</strong><br>
* Data/ATM/ASTs/all_test_cases<br>
* Data/ATM/ASTs/changed_test_cases</p>
<p><strong>Running the experiment:</strong><br>
To generate ASTS for all test cases in the project test suites, the <em><strong>Code/ATM/CodeToAST/src/CodeToAST.java</strong></em> file should be compiled and run using the Eclipse IDE by including all the required <em><strong>.jar</strong></em> files in the <em><strong>Code/ATM/CodeToAST/lib</strong></em> directory as part of the classpath. A bash script is provided along with a pre-generated <em><strong>.jar</strong></em> file in the <em><strong>Code/ATM/CodeToAST/bin</strong></em> directory to run this step, as follows:</p>
<pre><code class="language-bash">cd Code/ATM/CodeToAST
bash transform_code_to_ast.sh</code></pre>
<p>Each test file in the <em><strong>Data/test_suites/all_test_cases</strong></em> and <em><strong>Data/test_suites/changed_test_cases</strong></em> directories is parsed to generate a corresponding AST for each test case method (saved in an XML format in <strong>Data/ATM/ASTs/all_test_cases</strong> and <em><strong>Data/ATM/ASTs/changed_test_cases</strong></em> for each project version)<br>
<strong>_________________________________</strong></p>
<p><strong>ATM - Similarity Measurement:</strong></p>
<p><strong>Requirements:</strong><br>
* Eclipse IDE (we used 2021-12)<br>
* The libraries (the <em><strong>.jar</strong></em><strong> </strong>files in the <em><strong>Code/ATM/Similarity/lib</strong></em> directory)<br>
<br>
<strong>Input:</strong><br>
* Data/test_suites/all_test_cases<br>
* Data/test_suites/changed_test_cases<br>
<br>
<strong>Output:</strong><br>
* Data/ATM/similarity_measurements<br>
<br>
<strong>Running the experiment:</strong><br>
To measure the similarity between each pair of test cases, the <em><strong>Code/ATM/Similarity/src/SimilarityMeasurement.java</strong></em> file should be compiled and run using the Eclipse IDE by including all the required <em><strong>.jar</strong></em> files in the <em><strong>Code/ATM/Similarity/lib</strong></em> directory as part of the classpath. A bash script is provided along with a pre-generated <em><strong>.jar</strong></em> file in the <em><strong>Code/ATM/Similarity/bin</strong></em> directory to run this step, as follows:</p>
<pre><code class="language-bash">cd Code/ATM/Similarity
bash measure_similarity.sh</code></pre>
<p>ASTs of each project in the <em><strong>Data/ATM/ASTs/all_test_cases</strong></em> and <em><strong>Data/ATM/ASTs/changed_test_cases</strong></em> directories are parsed to create pairs of ASTs containing one test case from the <em><strong>Data/ATM/ASTs/all_test_cases</strong></em> directory with another test case from the <em><strong>Data/ATM/ASTs/changed_test_cases</strong></em> directory (redundant pairs are discarded). Then, all similarity measurements are saved in the <em><strong>Data/ATM/similarity_measurements.zip</strong></em> file.<br>
__________________________________________</p>
<p><strong>Search-based Minimization Algorithms:</strong><br>
The source code for this step is in the <em><strong>Code/ATM/Search</strong></em> directory.<br>
<br>
<strong>Requirements:</strong><br>
To run this step, Python 3 is required (we used <em><strong>Python 3.10</strong></em>). Also, the libraries in the <strong>Code/AMT/Search/requirements.txt</strong> file should be installed, as follows:</p>
<pre><code class="language-bash">cd Code/ATM/Search
pip install -r requirements.txt</code></pre>
<p><strong>Input:</strong><br>
* Data/ATM/similarity_measurements<br>
<br>
<strong>Output:</strong><br>
* Results/ATM/minimization_results<br>
<br>
<strong>Running the experiment:</strong><br>
To minimize the test suites in our dataset, the following bash script should be executed:</p>
<pre><code class="language-bash">bash minimize.sh</code></pre>
<p>All similarity measurements are parsed for each version of the projects, independently. Each version is run 10 times using three minimization budgets (25%, 50%, and 75%). Genetic Algorithm (GA) is run using four similarity measures, namely top-down, bottom-up, combined, and tree edit distance. NSGA-II is run using two combinations of similarity measures: top-down & bottom-up and combined & tree edit distance. The minimization results are generated in the <em><strong>Results/ATM/minimization_results</strong></em> directory.<br>
__________________</p>
<p><strong>Evaluate results:</strong><br>
To evaluate and summarize the minimization results, run the following:</p>
<pre><code class="language-bash">cd Code/ATM/Evaluation
bash evaluate.sh</code></pre>
<p>This will generate summarized <em>FDR</em> and execution time results (per-project and per-version) for each minimization budget, which can all be found in <strong>Results/ATM</strong>. In this replication package, we provide the final, merged <em>FDR</em> with execution time results.</p>
<p><strong>_________________________________</strong></p>
<p><strong>Running FAST-R experiments</strong><br>
ATM was compared to <a href="https://github.com/ICSE19-FAST-R/FAST-R">FAST-R</a>, a state-of-the-art baseline, which is a set of test case minimization techniques called: <em>FAST++, FAST-CS, FAST-pw, and FAST-all</em>, which we adapted to our data and experimental setup.</p>
<p><strong>Requirements:</strong><br>
To run this step, Python 3.7 is required. Also, the libraries in the <em><strong>Code/FAST-R/requirements.txt</strong></em> file should be installed, as follows:</p>
<pre><code class="language-bash">cd Code/FAST-R
pip install -r requirements.txt</code></pre>
<p><strong>Input:</strong><br>
* Data/FAST-R/test_methods<br>
* Data/FAST-R/test_classes</p>
<p><strong>Output:</strong><br>
* Results/FAST-R/test_methods/FDR_and_Exec_Time_Results_[budget]%_budget.csv<br>
* Results/FAST-R/test_classes/FDR_and_Exec_Time_Results_[budget]%_budget.csv<br>
<br>
To run FAST-R experiments, the following bash script should be executed:</p>
<pre><code class="language-bash">bash fast_r.sh test_methods #method level
bash fast_r.sh test_classes #class level</code></pre>
<p>Results are generated in <em><strong>.csv</strong></em> files for each budget. For example, for the 50% budget, results are saved in <strong>FDR_and_Exec_Time_Results_50%_budget.csv</strong> in the <em><strong>Results/FAST-R/test_methods</strong></em> and <em><strong>Results/FAST-R/test_classes</strong></em> directories.</p>
<p><strong>_________________________________</strong></p>
<p><strong>Running the random minimization experiments</strong><br>
ATM was also compared to random minimization as a standard baseline.</p>
<p><strong>Requirements:</strong> To run this step, Python 3 is required (we used <em><strong>Python 3.10</strong></em>). Also, the libraries in the <em><strong>Code/RandomMinimization/requirements.txt</strong></em> file should be installed, as follows:</p>
<pre><code class="language-bash">cd Code/RandomMinimization
pip install -r requirements.txt</code></pre>
<p><strong>Input:</strong><br>
<em>N/A</em></p>
<p><strong>Output:</strong><br>
* Results/RandomMinimization/FDR_and_Exec_Time_Results_[budget]%_budget.csv<br>
<br>
To run the random selection experiments, the following bash script should be executed:</p>
<pre><code class="language-bash">bash random_minimization.sh</code></pre>
<p>Results are generated in <em><strong>.csv</strong></em> files for each budget. For example, for the 50% budget, results are saved in <em><strong>FDR_and_Exec_Time_Results_50%_budget.csv</strong></em> in the <em><strong>Results/RandomMinimization</strong></em> directory.</p>
opencc-by-4.0Dec 2022View details →
