NJR-1 Dataset

<p>NJR is a Normalized Java Resource.&nbsp;The <em>NJR-1</em> dataset consists of 293 Java programs that can be used with several analysis tools.</p> <p>&nbsp;</p> <p><strong>TOOLS THAT RUN ON&nbsp;NJR-1</strong></p> <p>Each program runs successfully with the following 14&nbsp;Java static analysis tools:</p> <ol> <li>SpotBugs (https://spotbugs.github.io)</li> <li>Wala (https://wala.github.io)</li> <li>Doop (https://bitbucket.org/yanniss/doop)</li> <li>Soot (https://github.com/soot-oss/soot)</li> <li>Petablox (https://github.com/petablox/petablox)</li> <li>Infer (https://fbinfer.com)</li> <li>Error-Prone (http://errorprone.info)</li> <li>Checker-Framework (https://checkerframework.org)</li> <li>Opium (Opal-framework) (https://www.opal-project.de)</li> <li>Spoon (https://spoon.gforge.inria.fr)</li> <li>PMD (https://pmd.github.io)</li> <li>CheckStyle (https://checkstyle.org)</li> <li>JavaParser (https://javaparser.org/)</li> <li>Codeguru*&nbsp;(https://aws.amazon.com/codeguru)</li> </ol> <p>In addition to these static analysis tools, the NJR dataset has also been tested with 9&nbsp;other tools that operate on Java bytecode.</p> <ol> <li>Jacoco (https://www.jacoco.org): Dynamic analysis tool</li> <li>Wiretap (https://github.com/ucla-pls/wiretap): Dynamic analysis tool</li> <li>JReduce (https://github.com/ucla-pls/jreduce): Bytecode reduction tool</li> <li>QueryMax (https://doi.org/10.5281/zenodo.5551128): Preprocessor for application code analysis</li> <li>Call-Graph Pruner (https://doi.org/10.5281/zenodo.5177161): Static call-graph pruning tool</li> <li>FootPatch (https://github.com/squaresLab/footpatch): Automated Repair Tool</li> <li>Procyon (https://github.com/ststeiger/procyon): Decompiler</li> <li>CFR (https://www.benf.org/other/cfr/): Decompiler</li> <li>Fernflower (https://github.com/fesh0r/fernflower): Decompiler</li> </ol> <p>&nbsp;</p> <p><strong>BENCHMARK PROGRAMS</strong></p> <p>The NJR&nbsp;programs are repositories picked from a set of Java-8 projects on Github that compile and run successfully.&nbsp;Each of these programs comes with a jar file, the compiled bytecode files, compiled library files, and the Java source code.&nbsp;The availability of the files in both jar-file form, as well as source code form (with the compiled library classes) is a major reason the dataset works&nbsp;with&nbsp;so many tools, without requiring any extra effort. These features make NJR-1 a great benchmark for any kind of Java static analysis, dynamic analysis, or tool building.</p> <p>Internally, each benchmark program has the following structure:</p> <ul> <li><em>src:</em>&nbsp;directory with source files.</li> <li><em>classes:</em>&nbsp;directory with class files.</li> <li><em>lib:</em>&nbsp;compiled third party library classes (source files not available, since libraries are distributed as class-files).</li> <li><em>jarfile:</em>&nbsp;jar&nbsp;file containing the compiled application classes and third-party library classes.</li> <li><em>info:</em>&nbsp;directory with information about the program. It includes the following files. <ul> <li><em>classes:</em> list of application classes (excludes third-party library classes).</li> <li><em>mainclasses:</em> list of main classes that can be run.</li> <li><em>sources:</em> list of source file names.</li> <li><em>declarations:</em>&nbsp;list of method declarations categorized&nbsp;by source file name.</li> </ul> </li> </ul> <p>The benchmarks already come with a compiled JAR file, but some tools need to compile and run the benchmarks.&nbsp;The following simple commands can be used for compilation and running (replace &lt;jarfilename&gt; with the file in the <em>jarfile</em>&nbsp;directory. replace &lt;mainclassname&gt; with any of the classes from info/mainclasses):</p> <p><em>javac -d compiled_classes -cp lib @info/sources</em></p> <p><em>java -cp&nbsp;jarfile/&lt;jarfilename&gt; &lt;mainclassname&gt;</em></p> <p>&nbsp;</p> <p><strong>FILES AVAILABLE FOR DOWNLOAD</strong></p> <p>There are 4&nbsp;files available for download: <em>njr-1_dataset.zip, scripts.zip, benchmark_stats.zip, and a Readme.</em></p> <p><em>njr-1_dataset.zip</em> has the actual dataset programs. <em>scripts.zip</em> contains&nbsp;Python3 scripts&nbsp;for each tool, to run it&nbsp;on the entire dataset. The Readme details&nbsp;the version number, download link and setup instructions for each tool. The <em>benchmark_stats.zip&nbsp;</em>file lists some statistics for the benchmark programs.</p> <p>&nbsp;</p> <p><strong>STATISTICS</strong></p> <p>Here are some summary statistics about the benchmark programs:</p> <ul> <li>The mean number of application classes: 97</li> <li>Each program&nbsp;executes at least 100 unique application methods at runtime.</li> <li>The mean lines of application source code: 9911</li> <li>The mean number of 3rd party library classes: 2608</li> <li>The mean (estimated) lines of 3rd party library source code: 250,000</li> <li>The mean number of static edges in the application call graph: 1404</li> <li>The mean number of dynamic edges in the application&nbsp;call graph: 469&nbsp;</li> </ul> <p>&nbsp;</p> <p><strong>NOTES</strong></p> <p>Note: Zenodo shows some&nbsp;changes for this repository. However, all the changes involve updating the scripts folder, as more tools get tested on the dataset. The programs in the dataset themselves remain unchanged.</p> <p>*Note 2: Codeguru Reviewer is a paid, proprietary tool by Amazon. Our experiments show that it runs successfully on all the benchmarks in this dataset. However, we don&#39;t include any scripts to replicate this run because of its paid nature.</p> <p>To cite this dataset, please cite the following paper:<br> Jens Palsberg and Cristina V. Lopes,&nbsp;NJR: a&nbsp;Normalized Java Resource.&nbsp;<br> In Proceedings of ACM SIGPLAN International Workshop&nbsp;on State Of the Art in Program Analysis (SOAP), 2018.</p>