Does Unit-Tested Code Crash? A Case Study of Eclipse: Replication Package

<p><strong>Does Unit-Tested Code Crash? A Case Study of Eclipse: Replication Package</strong></p> <p>This is a replication package associated with the paper titled &ldquo;Does Unit-Tested Code Crash? A Case Study of Eclipse&rdquo;. Below is a description of the package&rsquo;s contents.</p> <p><strong>Data</strong></p> <p>Data files associated with the paper are provided in the <code>data</code> directory.</p> <p><strong>Text file <code>tested-crashed.txt</code></strong></p> <p>Data specifying whether methods were tested and whether they crashed (according to the criteria adopted in the study). Extracted from <code>matches.xlsx</code>. The data are used as input for Fisher&rsquo;s test (RQ1).</p> <p><strong>Spreadsheet <code>matches.xlsx</code></strong></p> <p>Test coverage data and calculations associated with failed methods, class coverage, and matching method coverage results are provided in an Excel spreadsheet. Below is the description of the spreadsheet&rsquo;s contents.</p> <p>Worksheet <em>Test Coverage</em></p> <p>Contains the data regarding the JaCoCo test code coverage analysis.</p> <ul> <li>Class: The name of the class in which a method appears in JVM internal form notation</li> <li>Method: The method&rsquo;s name</li> <li>Parameters: The method&rsquo;s arguments in JVM parameter descriptor format; required to handle Java&rsquo;s {} polymporphism</li> <li>Class Has Unit Test: Whether the corresponding class has associated unit test code</li> <li>Class Unit-Test Line Density: The ratio of lines in class&rsquo;s test code over those in the class&rsquo;s implementation code</li> <li>Covered Instructions / Branches / Lines: As reported by JaCoCo</li> <li>Total Instructions / Branches / Lines: As reported by JaCoCo</li> <li>Covered Instructions / Branches / Lines ratio: The ratio between the two preceding values; 1 for methods without any branches</li> <li>Top-1 / Top-6 / Top-10 : In how many stack traces the method appears within; the top-10 / top-6 / the very first stack frame(s)</li> <li>Tested: TRUE if the method is considered tested by having a test code coverage above the median (0.966) and an associated test class</li> <li>Crashed: TRUE if the method has crashed as evidenced by its appearance on the topmost stack frame</li> <li>Stack trace file names: in which the method appeared</li> </ul> <p>Worksheet <em>Test Existence</em></p> <p>Contains the data of the analysis regarding the existence of test code.</p> <ul> <li>Class: Class containing implementation code</li> <li>TestClassNames: Classes that contain tests for the above</li> <li>Number of relevant tests</li> <li>Lines in class test code</li> <li>Lines of class</li> <li>Class Unit-Test Line Density: The ratio between the two above</li> </ul> <p>Worksheet <em>Metrics</em></p> <p>Contains the derivation of metrics reported in the paper. In the cases of tables these are formatted in LaTeX for direct incorporation into the text.</p> <p><strong>Spreadsheet <code>jacoco.xlsx</code></strong></p> <p>Complete test coverage data obtained from JaCoCo are provided in an Excel spreadsheet. Below is the description of the spreadsheet&rsquo;s contents.</p> <p>Worksheet <em>Data</em></p> <p>Contains the following method code coverage fields as reported by JaCoCo, as well as the calculated percentages.</p> <ul> <li>Class</li> <li>Method</li> <li>Parameters</li> <li>Covered Instructions</li> <li>Total Instructions</li> <li>% Covered Instructions</li> <li>Covered Branches</li> <li>Total Branches</li> <li>% Covered Branches</li> <li>Covered Lines</li> <li>Total Lines</li> <li>% Covered Lines</li> </ul> <p>Worksheet <em>Metrics</em></p> <p>Contains the derivation of numbers reported in the preliminary quantitative analysis and Figure 2.</p> <p>Compressed tar archive <code>eclipse-src.tar.gz</code></p> <p>Contains the Eclipse source code used for running the Eclipse tests with JaCoCo code coverage analysis. It was obtained from the Eclipse source code repositories as follows.</p> <ul> <li>Clone the Eclipse aggreagator repository into a directory named z by running: <code>git clone -b master --recursive git://git.eclipse.org/gitroot/platform/eclipse.platform.releng.aggregator.git z</code></li> <li>In the <code>z</code> directory, checking out the used release by running <code>cd z &amp;&amp; git submodule foreach git checkout M20160212-1500</code></li> <li>Checking out the release for the main repository by running: <code>git checkout M20160212-1500</code></li> <li>Applying the patch <code>eclipse-src.diff</code></li> </ul> <p><strong>Patch file <code>eclipse-src.diff</code></strong></p> <p>See above.</p> <p><strong>Zip file <code>incidents.zip</code></strong></p> <p>Contains the 126,026 incidents (crash report stack traces and meta-data) associated with <em>EclipseProduct</em> <code>org.eclipse.epp.package.java.product</code> and <em>BuildID</em> <code>4.5.2.M20160212-1500</code>. This is a subset from the two million incidents available as the <a href="http://software-data.org/datasets/aeri-stacktraces/downloads/incidents_full.tar.bz2">AERI stack traces data set</a>.</p> <p>The subset of incidents was extracted from the full AERI data set with the following command.</p> <pre><code class="language-bash">for f in *; do grep -q org.eclipse.epp.package.java.product $f &amp;&amp; grep -q 4.5.2.M20160212-1500 $f &amp;&amp; mv $f selected-files/ done</code></pre> <p><br> <strong>Compressed file <code>jacoco.xml.gz</code></strong></p> <p>Contains the results of the JaCoCo code coverage analysis over the Eclipse testing.</p> <p><strong>Code</strong></p> <p>The following scripts are provided in the <code>src</code> directory</p> <ul> <li><code>extract.py</code>: script for extracting crash (incidents) and coverage (JaCoCo) data</li> <li><code>unit-tested-classes.py</code>: script for finding the classes with associated unit test code</li> <li><code>merge.py</code>: script for matching crash (incidents) with coverage (JaCoCo) data</li> <li><code>fisher.r</code>: R script for running Fisher&rsquo;s test</li> </ul>