1,237 Annotated Developer Apologies from GitHub

<p><em><strong>Software Developer Apologies</strong></em></p> <p>This dataset contains 1,237 GitHub comments with apology annotations (apology vs not apology), released as part of the following publication:</p> <ul> <li>Benjamin S. Meyers. <a href="https://scholarworks.rit.edu/theses/11609/">Human Error Assessment in Software Engineering.</a> Rochester Institute of Technology. 2023.&nbsp;</li> </ul> <p><em><strong>Included Files</strong></em></p> <p>The "github_apologies.csv" file contains the full dataset of 1,237 GitHub comments with apology annotations. In total, there are 365 comments containing an apology (872 non apologies). The comments themselves are a subset of those included in <a href="../records/5603093">88.6 Million Developer Comments from GitHub</a>.</p> <p><em><strong>Annotation Details</strong></em></p> <p>Full details are provided in the above publication. We implemented a naive classifier (Precision: 41.7%, Recall: 99.7%, F1: 86.9%, Accuracy: 91.1%) using counts of apology lemmas. 91% of developer comments containing at least one apology lemma matched our manual annotations. Agreement between raters was almost perfect (Cohen's Kappa = 0.94).</p> <p><em><strong>CSV Fields</strong></em></p> <ul> <li><strong>ID</strong>: Unique identifier for the comment.</li> <li><strong>SOURCE</strong>: Whether this comment originates from a commit, issue, or pull request.</li> <li><strong>COMMENT_URL</strong>: The URL linking to the comment.</li> <li><strong>COMMENT_TEXT</strong>: The raw comment text.</li> <li><strong>NUM_APOLOGY_LEMMAS</strong>: The count of apology lemmas present in the comment.</li> <li><strong>CLASSIFIER_LABEL</strong>: The automatically assigned label ("Apology" or "Not Apology").</li> <li><strong>RATER_1_LABEL</strong>: The manually assigned label ("Apology" or "Not Apology") from Rater 1.</li> <li><strong>RATER_2_LABEL</strong>: The manually assigned label ("Apology" or "Not Apology") from Rater 2.</li> <li><strong>AGREED_LABEL</strong>: The agreed upon label ("Apology" or "Not Apology") after Rater 1 and Rater 12 resolved disagreements.</li> </ul> <p><em><strong>Contact</strong></em></p> <p>Please contact Benjamin S. Meyers (<a href="mailto:bsm9339@rit.edu">email</a>) with questions about this data and its collection.</p> <p><em><strong>Acknowledgments</strong></em></p> <p>Collection of this data has been sponsored in part by the National Science Foundation (grant 1922169), by the NSA Science of Security Lablet program (grant H98230-17-D-0080/2018-0438-02), and by a Department of Defense DARPA SBIR program (grant 140D63-19-C-0018).</p>