On the Performance of Method-Level Bug Prediction: A Negative Result. Appendix

<p>Abstract Bug prediction is aimed at identifying software artifacts that are more likely to be defective. Most approaches defined so far target the prediction of bugs at class/file level. Nevertheless, past research has provided evidence that this granularity might be too coarse-grained, thus reducing the usability of bug prediction in practice. As a consequence, researchers have started proposing defect prediction models targeting a finer granularity, particularly targeting methods, providing promising evidence that it is possible to operate at this granularity. Particularly, models based on a mixture of product and process metrics provided the best results.&nbsp;</p> <p>In this paper, we first replicate previous research on method-level bug- prediction using different systems and timespans. Afterward, based on the limitations of existing research, we (1) re-evaluate method-level bug prediction models more realistically and (2) analyze the whether textual features&mdash; previously shown as valuable sources of information for the evaluation of software quality (yet surprisingly unexplored in this research field)&mdash;can be exploited to improve method-level bug prediction abilities. Key results of our study include that (1) the performance of the previously proposed models, tested using the same strategy but with different systems/timespans, is con- firmed. However, (2) when evaluated with a more realistic strategy all the models show a dramatic drop in performance exhibiting results close to that of a random classifier. In addition, we find that (3) the contribution of textual features within such models is limited and unable to improve the prediction capabilities significantly. As a consequence, our replication and negative results indicate that method-level bug prediction is still an open challenge.</p>