Identification of Performance Changes at Code Level (Jetty Evaluation Dataset)

<p><em>This is the anonymous reviewing version; the source code repository will be added after the review.</em></p> <p>This dataset provides the results of measuring jetty with the 1,000 artificial regressions with Peass and with JMH. The creation of the artificial regressions and the measurement is defined here: https://anonymous.4open.science/r/jetty-evaluation-6F58/ (Repository named jetty-evaluation, GitHub link will be provided after review) An example regression is contained in <a href="https://anonymous.4open.science/r/jetty-experiments-202D">https://anonymous.4open.science/r/jetty-experiments-202D</a> We obtained these data from measurement on Intel Xeon CPU E5-2620 v3 @ 2.40GHz.</p> <p>The dataset contains the following data:</p> <ul> <li>regression-results-peass-0.tar.xz (Results of the measurement with Peass, part 0)</li> <li>regression-results-peass-1.tar.xz (Results of the measurement with Peass, part 1)</li> <li>regression-results-peass-2.tar.xz (Results of the measurement with Peass, part 2)</li> <li>regression-results-peass-3.tar.xz (Results of the measurement with Peass, part 3)</li> <li>regression-results-jmh.tar.xz (Results of the measurement with JMH)</li> <li>tree-results.tar.xz (Metadata of the trees)</li> </ul> <p>To get the data in a usable format, extract the peass data to one folder (the folder will be named <strong>$PEASS_RESULT_FOLDER</strong>):</p> <pre><code class="language-bash">mkdir peass for file in *; do echo $file; tar -xf $file; done for i in {0..3}; do mv $i/* .; done</code></pre> <p>This will yield to a folder containing 1000 folders named regression-$i, where each consists of</p> <ul> <li>deps.tar.xz: The regression test selection results</li> <li>logs.tar.xz: The logs of the test executions</li> <li>results: The traces of the regression test selection and a file named changes_*testcase.json, which contains statistical details of the measured performance change (if present)</li> <li>jetty.project_peass: Detailed measurement data and logs of individual JVM starts</li> </ul> <p>To analyse the Peass results, run</p> <pre><code class="language-bash">cd scripts/peass ./analyzeChangeIdentification.sh $PEASS_RESULTS_FOLDER ./analyzeFrequency.sh $PEASS_RESULTS_FOLDER</code></pre> <p>This will take some time, since partial results need to be unpacked for analysis. The first script will create the following results:</p> <p>and the second will yield the following results:</p> <pre><code>Correct Measurement: 587 Not selected changes: 146 Wrong measurement result: 267 Wrong analysis (should be 0): 0 Overall: 1000 Share of changed method on correct measurements: 0.109571 0.11238 32 Method call count on correct measurement: 15638.4 32853.7 32 Average tree depth on correct measurements: 1.1022 2.6875 32 Share of changed method on wrong measurements: 0.17692 0.180365 968 Method call count on wrong measurement: 711415 180438 968 Average tree depth on wrong measurements: 1.23239 2.42252 968</code></pre> <p>To analyze the JMH data, first extract the metadata (the folder will be named <strong>$TREEFOLDER</strong>):</p> <pre><code class="language-bash">tar -xf tree-results.tar.xz</code></pre> <p>Afterwards extract the JMH results (the folder will be named $<strong>JMH_RESULTS_FOLDER</strong>):</p> <pre><code class="language-bash">tar -xvf regression-results-jmh.tar.xz</code></pre> <p>This will yield a folder containing a measurement for each regression with two files:</p> <ul> <li>basic.json: The performance measurement result of the basic version</li> <li>regression-$i.json: The performance measurement result of the version containing the regression</li> </ul> <p>Afterwards, run the analysis in the jetty-evaluation repository:</p> <pre><code class="language-bash">cd scripts/jmh ./analyzeFrequency.sh $JMH_RESULTS_FOLDER $TREEFOLDER</code></pre> <p>Since the regression are injected in the call tree of the benchmark, there are now unselected changes. The analysis will yield the following results:</p> <pre><code>Share of changed method on correct measurements: 0.184631 0.271968 587 Method call count on correct measurement: 14628.2 4979.6 587 Share of changed method on wrong measurements: 0.180981 0.235614 267 Method call count on wrong measurement: 14902.2 4333.58 267</code></pre> <p>&nbsp;</p>