zenodoopen

ATM: Black-box Test Case Minimization based on Test Code Similarity and Evolutionary Search – Replication Package

This is the replication package associated with the paper "ATM: Black-box Test Case Minimization based on Test Code Similarity and Evolutionary Search" accepted at the 45th IEEE/ACM International Conference on Software Engineering (ICSE 2023) – Technical Track. Cite this paper using the following: @inproceedings{pan2023atm,   title={ATM: Black-box Test Case Minimization based on Test Code Similarity and Evolutionary Search},   author={Pan, Rongqi and Ghaleb, Taher A. and Briand, Lionel},   booktitle={Proceedings of the 45th IEEE/ACM International Conference on Software Engineering},   year={2023},   pages={1--12} } Replication Package Contents: The replication package contains all the necessary data and code required to reproduce the results reported in the paper. We also provide the results for other minimization budgets, and detailed FDR, execution time, and statistical test results. In addition, we provide the data and code required to reproduce the results of baselines techniques: FAST-R and random minimization. Data: We provide in the Data directory the data used in our experiments, which is based on 16 projects from <a href="https://github.com/rjust/defects4j">Defects4J</a>, whose characteristics can be found in Data/subject_projects.csv. Code: We provide in the Code directory the code and scripts (Java, Python, and Bash) required to run the experiments and reproduce the results. Results: We provide in the Results directory the results for each technique independently, and also a summary of all results together for comparison purposes. The source code for this step is in the Code/ATM/CodeToAST directory. The source code for this step is in the Code/ATM/Similarity directory. _________________________________ ATM - Code to AST transformation: Requirements: * Eclipse IDE (we used 2021-12) * The libraries (the .jar files in the Code/ATM/CodeToAST/lib directory) Input: All zipped data files should be unzipped before running each step. * Data/test_suites/all_test_cases.zip → Data/test_suites/all_test_cases * Data/test_suites/changed_test_cases.zip → Data/test_suites/changed_test_cases * Data/test_suites/relevant_test_cases.zip → Data/test_suites/relevant_test_cases Output: * Data/ATM/ASTs/all_test_cases * Data/ATM/ASTs/changed_test_cases Running the experiment: To generate ASTS for all test cases in the project test suites, the Code/ATM/CodeToAST/src/CodeToAST.java file should be compiled and run using the Eclipse IDE by including all the required .jar files in the Code/ATM/CodeToAST/lib directory as part of the classpath. A bash script is provided along with a pre-generated .jar file in the Code/ATM/CodeToAST/bin directory to run this step, as follows: <pre><code class="language-bash">cd Code/ATM/CodeToAST bash transform_code_to_ast.sh</code></pre> Each test file in the Data/test_suites/all_test_cases and Data/test_suites/changed_test_cases directories is parsed to generate a corresponding AST for each test case method (saved in an XML format in Data/ATM/ASTs/all_test_cases and Data/ATM/ASTs/changed_test_cases for each project version) _________________________________ ATM - Similarity Measurement: Requirements: * Eclipse IDE (we used 2021-12) * The libraries (the .jar files in the Code/ATM/Similarity/lib directory) Input: * Data/test_suites/all_test_cases * Data/test_suites/changed_test_cases Output: * Data/ATM/similarity_measurements Running the experiment: To measure the similarity between each pair of test cases, the Code/ATM/Similarity/src/SimilarityMeasurement.java file should be compiled and run using the Eclipse IDE by including all the required .jar files in the Code/ATM/Similarity/lib directory as part of the classpath. A bash script is provided along with a pre-generated .jar file in the Code/ATM/Similarity/bin directory to run this step, as follows: <pre><code class="language-bash">cd Code/ATM/Similarity bash measure_similarity.sh</code></pre> ASTs of each project in the Data/ATM/ASTs/all_test_cases and Data/ATM/ASTs/changed_test_cases directories are parsed to create pairs of ASTs containing one test case from the Data/ATM/ASTs/all_test_cases directory with another test case from the Data/ATM/ASTs/changed_test_cases directory (redundant pairs are discarded). Then, all similarity measurements are saved in the Data/ATM/similarity_measurements.zip file. __________________________________________ Search-based Minimization Algorithms: The source code for this step is in the Code/ATM/Search directory. Requirements: To run this step, Python 3 is required (we used Python 3.10). Also, the libraries in the Code/AMT/Search/requirements.txt file should be installed, as follows: <pre><code class="language-bash">cd Code/ATM/Search pip install -r requirements.txt</code></pre> Input: * Data/ATM/similarity_measurements Output: * Results/ATM/minimization_results Running the experiment: To minimize the test suites in our dataset, the following bash script should be executed: <pre><code class="language-bash">bash minimize.sh</code></pre> All similarity measurements are parsed for each version of the projects, independently. Each version is run 10 times using three minimization budgets (25%, 50%, and 75%). Genetic Algorithm (GA) is run using four similarity measures, namely top-down, bottom-up, combined, and tree edit distance. NSGA-II is run using two combinations of similarity measures: top-down & bottom-up and combined & tree edit distance. The minimization results are generated in the Results/ATM/minimization_results directory. __________________ Evaluate results: To evaluate and summarize the minimization results, run the following: <pre><code class="language-bash">cd Code/ATM/Evaluation bash evaluate.sh</code></pre> This will generate summarized FDR and execution time results (per-project and per-version) for each minimization budget, which can all be found in Results/ATM. In this replication package, we provide the final, merged FDR with execution time results. _________________________________ Running FAST-R experiments ATM was compared to <a href="https://github.com/ICSE19-FAST-R/FAST-R">FAST-R</a>, a state-of-the-art baseline, which is a set of test case minimization techniques called: FAST++, FAST-CS, FAST-pw, and FAST-all, which we adapted to our data and experimental setup. Requirements: To run this step, Python 3.7 is required. Also, the libraries in the Code/FAST-R/requirements.txt file should be installed, as follows: <pre><code class="language-bash">cd Code/FAST-R pip install -r requirements.txt</code></pre> Input: * Data/FAST-R/test_methods * Data/FAST-R/test_classes Output: * Results/FAST-R/test_methods/FDR_and_Exec_Time_Results_[budget]%_budget.csv * Results/FAST-R/test_classes/FDR_and_Exec_Time_Results_[budget]%_budget.csv To run FAST-R experiments, the following bash script should be executed: <pre><code class="language-bash">bash fast_r.sh test_methods #method level bash fast_r.sh test_classes #class level</code></pre> Results are generated in .csv files for each budget. For example, for the 50% budget, results are saved in FDR_and_Exec_Time_Results_50%_budget.csv in the Results/FAST-R/test_methods and Results/FAST-R/test_classes directories. _________________________________ Running the random minimization experiments ATM was also compared to random minimization as a standard baseline. Requirements: To run this step, Python 3 is required (we used Python 3.10). Also, the libraries in the Code/RandomMinimization/requirements.txt file should be installed, as follows: <pre><code class="language-bash">cd Code/RandomMinimization pip install -r requirements.txt</code></pre> Input: N/A Output: * Results/RandomMinimization/FDR_and_Exec_Time_Results_[budget]%_budget.csv To run the random selection experiments, the following bash script should be executed: <pre><code class="language-bash">bash random_minimization.sh</code></pre> Results are generated in .csv files for each budget. For example, for the 50% budget, results are saved in FDR_and_Exec_Time_Results_50%_budget.csv in the Results/RandomMinimization directory.

ShareScore

36/100

Overall dataset sharing score

Score breakdown

These five areas show where the dataset supports — or may limit — practical reuse.

Stewardship: 8
Harmonization: 4
Access: 16
Reuse readiness: 8
Engagement: 0

Topics

Test case minimization Test suite reduction Tree-based similarity AST Genetic algorithm Black-box testing