Regression-Test History Data for Flaky Test-Research, Dataset
<p>The dataset comprises developer test results of Maven projects with flaky tests across a range of consecutive commits from the projects' git commit histories. The Maven projects are a subset of those investigated in an <a href="https://doi.org/10.1145/3428270">OOPSLA 2020 paper</a>. The commit range for this dataset has been chosen as the flakiness-introducing commit (FIC) and iDFlakies-commit (see the OOPSLA paper for details). The commit hashes have been obtained from the <a href="https://github.com/TestingResearchIllinois/idoft/blob/main/tic-fic-data.csv">IDoFT dataset</a>.</p>
<p>The dataset will be presented at the <a href="https://conf.researchr.org/home/icse-2024/ftw-2024">1st International Flaky Tests Workshop 2024 (FTW 2024)</a>. Please refer to <a href="https://doi.org/10.1145/3643656.3643901">our extended abstract</a> for more details about the motivation for and context of this dataset.</p>
<p>The following table provides a summary of the data.</p>
<table>
<tbody>
<tr>
<td><strong>Slug (Module)</strong></td>
<td><strong>FIC Hash</strong></td>
<td><strong>Tests</strong></td>
<td><strong>Commits</strong></td>
<td><strong>Av. Commits/Test</strong></td>
<td><strong>Flaky Tests</strong></td>
<td><strong>Tests w/ Consistent Failures</strong></td>
<td><strong>Total Distinct Histories</strong></td>
</tr>
<tr>
<td>TooTallNate/Java-WebSocket</td>
<td> 822d40</td>
<td>146</td>
<td> 75</td>
<td> 75</td>
<td>24</td>
<td> 1</td>
<td>2.6x10^9</td>
</tr>
<tr>
<td>apereo/java-cas-client (cas-client-core)</td>
<td> 5e3655</td>
<td>157</td>
<td> 65</td>
<td>61.7</td>
<td> 3</td>
<td> 2</td>
<td>1.0x10^7</td>
</tr>
<tr>
<td>eclipse-ee4j/tyrus (tests/e2e/standard-config)</td>
<td> ce3b8c</td>
<td>185</td>
<td> 16</td>
<td> 16</td>
<td>12</td>
<td> 0</td>
<td> 261</td>
</tr>
<tr>
<td>feroult/yawp (yawp-testing/yawp-testing-appengine)</td>
<td> abae17</td>
<td> 1</td>
<td>191</td>
<td>191</td>
<td> 1</td>
<td> 1</td>
<td> 8</td>
</tr>
<tr>
<td>fluent/fluent-logger-java</td>
<td> 5fd463</td>
<td> 19</td>
<td>131</td>
<td>105.6</td>
<td>11</td>
<td> 2</td>
<td>8.0x10^32</td>
</tr>
<tr>
<td>fluent/fluent-logger-java</td>
<td> 87e957</td>
<td> 19</td>
<td>160</td>
<td>122.4</td>
<td>11</td>
<td> 3</td>
<td>2.1x10^31</td>
</tr>
<tr>
<td>javadelight/delight-nashorn-sandbox</td>
<td> d0d651</td>
<td> 81</td>
<td>113</td>
<td>100.6</td>
<td> 2</td>
<td> 5</td>
<td>4.2x10^10</td>
</tr>
<tr>
<td>javadelight/delight-nashorn-sandbox</td>
<td> d19eee</td>
<td> 81</td>
<td> 93</td>
<td>83.5</td>
<td> 1</td>
<td> 5</td>
<td>2.6x10^9</td>
</tr>
<tr>
<td>sonatype-nexus-community/nexus-repository-helm</td>
<td> 5517c8</td>
<td> 18</td>
<td> 32</td>
<td> 32</td>
<td> 0</td>
<td> 0</td>
<td> 18</td>
</tr>
<tr>
<td>spotify/helios (helios-services)</td>
<td> 23260</td>
<td>190</td>
<td>448</td>
<td>448</td>
<td> 0</td>
<td> 37</td>
<td> 190</td>
</tr>
<tr>
<td>spotify/helios (helios-testing)</td>
<td> 78a864</td>
<td> 43</td>
<td>474</td>
<td>474</td>
<td> 0</td>
<td> 7</td>
<td> 43</td>
</tr>
</tbody>
</table>
<p> </p>
<p>The columns are composed of the following variables:</p>
<ul>
<li><strong>Slug (Module):</strong> The project's GitHub slug (i.e., the project's URL is https://github.com/{Slug}) and, if specified, the module for which tests have been executed.</li>
<li><strong>FIC Hash:</strong> The flakiness-introducing commit hash for a known flaky test as described in this OOPSLA 2020 paper. As different flaky tests have different FIC hashes, there may be multiple rows for the same slug/module with different FIC hashes. </li>
<li><strong>Tests:</strong> The number of distinct test class and method combinations over the entire considered commit range.</li>
<li><strong>Commits:</strong> The number of commits in the considered commit range</li>
<li><strong>Av. Commits/Test:</strong> The average number of commits per test class and method combination in the considered commit range. The number of commits may vary for each test class, as some tests may be added or removed within the considered commit range.</li>
<li><strong>Flaky Tests:</strong> The number of distinct test class and method combinations that have more than one test result (passed/skipped/error/failure + exception type, if any + assertion message, if any) across 30 repeated test suite executions on at least one commit in the considered commit range.</li>
<li><strong>Tests w/ Consistent Failures:</strong> The number of distinct test class and method combinations that have the same error or failure result (error/failure + exception type, if any + assertion message, if any) across all 30 repeated test suite executions on at least one commit in the considered commit range.</li>
<li><strong>Total Distinct Histories:</strong> The number of distinct test results (passed/skipped/error/failure + exception type, if any + assertion message, if any) for all test class and method combinations along all commits for that test in the considered commit range.</li>
</ul>
openother-openFeb 2024View details →
