Using White-Box Test Generation During Development: A Replicated Study

<p>This package contains the artifacts for our external, differentiated replication of an empirical study on using white-box test generation.</p> <p>The original experiment is described in the following paper.</p> <blockquote> <p>Jos&eacute; Miguel Rojas, Gordon Fraser, and Andrea Arcuri. 2015. Automated unit test generation during software development: a controlled experiment and think-aloud observations. In Proceedings of the 2015 International Symposium on Software Testing and Analysis (ISSTA 2015), 338&ndash;349. DOI: 10.1145/2771783.2771801</p> </blockquote> <p>30 developers and students participated in the experiment in January 2016.</p> <p>The main difference in the replication was using the IntelliTest test generator tool instead of EvoSuite.</p> <p>The contents of the package are the followings.</p> <ul> <li>/assignments <ul> <li>session_starts_endings.csv - Timestamps when each session was started</li> <li>assignments_with_projects.csv - Assigments of participants to projects, classes and sessions</li> </ul> </li> <li>/projects <ul> <li>/golden - Contains the golden implementation and test suites of all 4 classes</li> <li>/user - Contains the user implementation and test method stubs of all 4 classs</li> </ul> </li> <li>/results <ul> <li>time_merged.csv - File of timing analysis containing each test run of subjects</li> <li>results.csv - Contains the coverage and other required data of each participant tasks</li> <li>exit.csv - Answers of the exit survey</li> <li>background.csv - Answers of the background questionnaire</li> </ul> </li> <li>/scripts <ul> <li>/ps <ul> <li>Get-Results.ps1 - Gets the results of both tasks for one specified participant (invokes MSBuild, MSTest, VisualMutator, OpenCover)</li> <li>Get-Failures-Errors.ps1 - Gets failures and errors of both tasks for one specified participant (invokes MSBuild, MSTest)</li> <li>Get-TimeData.ps1 - Iterates over all test runs for each participant</li> </ul> </li> <li>/r <ul> <li>likert.R - Generates the Likert chart</li> <li>si_st.R - Generates table of subject implementation executed with subject test suites</li> <li>gi_st.R - Generates table of results where subject test suites was run on golden implementation</li> <li>time_spent_tests.R - Generates a table of time and activity of testing</li> <li>errors_failures.R - Generates table of errors and failures obtained using the golden test suite</li> <li>correlation_analysis.R - Generates table containing correlation between lack of correctness and IntelliTest usage</li> </ul> </li> </ul> </li> </ul>