ASE2021 vulnerability fix dataset

<p>The dataset of &quot;Finding A Needle in a Haystack: Automated Mining of Silent Vulnerability Fixes&quot;, which was accepted in the 36th IEEE/ACM Automated Software Engineering (<em>ASE</em>) Conference.</p> <p>Followings are the descriptions of columns:</p> <ol> <li>commit_id: The commit ID/hash.</li> <li>repo: The Github Author and repository (e.g., &quot;apache/hive&quot;).</li> <li>filename: The name of the file changed in the commit.</li> <li>partition: Which dataset the commit information belongs to (i.e., &quot;train&quot;, &quot;val&quot;, or &quot;test&quot;).</li> <li>PL: Programming Language (PL) (i.e., &quot;java&quot; or &quot;py&quot;).</li> <li>label: Label of the commit, 0 for non-vulnerability fixing commit and 1 for vulnerability fixing commit.</li> <li>diff: The entire code change information of the file in this commit.</li> <li>committer_date: The date of the commit (e.g., 2015-03-02 13:48:25+13:00)</li> <li>msg: The commit message (NA if empty).</li> <li>MOD_DIFF: The code change of the file in this commit after preprocessing: filtering out lines that are not added lines or removed lines, and removing refactoring information and comments.</li> <li>BPE_MOD_DIFF: BPE processing applied to MOD_DIFF information (using codeprep Python package).</li> <li>ADD_DIFF: The added lines from the MOD_DIFF information (indicated as a line starting with &#39;+&#39; character).</li> <li>REM_DIFF: The removed lines from the MOD_DIFF information (indicated as a line starting with &#39;-&#39; character).</li> <li>LOC_ADD: Total lines of code added in this file change.</li> <li>LOC_REM: Total lines of code removed in this file change.</li> <li>LOC_MOD: Total lines of code modified in this file change (LOC_ADD + LOC_REM).</li> <li>commit_repo: The commit ID and repository concatenated.</li> <li>cve_list: A list of CVEs which the commit fixes (e.g., CVE-2015-5348, CVE-2016-8902).</li> </ol> <p>Following is the code snippet to reproduce Table 1.</p> <pre><code class="language-python">import pandas as pd all_commits = pd.read_csv('./ase_dataset_sept_19_2021.csv') #Separate by language, since the Java commits are missing some info which we will add later on. py = all_commits[all_commits.PL == 'python'] java = all_commits[all_commits.PL == 'java'] #Java first: partition into train/val/test and check # of commits print("Java VF vs NVF for train/val/test") java_train = java[java.partition =="train"] java_val = java[java.partition == "val"] java_test = java[java.partition == "test"] print(java_train.drop_duplicates(subset='commit_id').label.value_counts()) print(java_val.drop_duplicates(subset='commit_id').label.value_counts()) print(java_test.drop_duplicates(subset='commit_id').label.value_counts()) #Python: partition into train/val/test and check # of commits print("Py VF vs NVF for train/val/test") py_train = py[py.partition =="train"] py_val = py[py.partition == "val"] py_test = py[py.partition == "test"] print(py_train.drop_duplicates(subset='commit_id').label.value_counts()) print(py_val.drop_duplicates(subset='commit_id').label.value_counts()) print(py_test.drop_duplicates(subset='commit_id').label.value_counts())</code></pre> <p>&nbsp;</p>