Testing Scratch Programs Automatically

<p># Replication Package</p> <p>This is the replication package for our work on<br> &quot;Testing Scratch Programs Automatically&quot;.</p> <p>The package contains our raw results and scripts for generating<br> the plots of the paper from the raw data.</p> <p>## Abstract</p> <p>Block-based programming environments like Scratch foster engagement<br> with computer programming and are used by millions of young learners.<br> Scratch allows learners to quickly create entertaining programs and<br> games, while eliminating syntactical program errors that could<br> interfere with progress.</p> <p>However, functional programming errors may still lead to incorrect<br> programs, and learners and their teachers need to identify and<br> understand these errors. This is currently an entirely manual process.</p> <p>In this paper, we introduce a formal testing framework that describes<br> the problem of Scratch testing in detail. We instantiate this formal<br> framework with the Whisker tool, which provides automated and<br> property-based testing functionality for Scratch programs.</p> <p>Empirical evaluation on real student and teacher programs<br> demonstrates that Whisker can successfully test Scratch programs,<br> and automatically achieves an average of 95.25% code coverage.</p> <p>Although well-known testing problems such as test flakiness also<br> exist in the scenario of Scratch testing, we show that automated and<br> property-based testing can accurately reproduce and replace the<br> manually and laboriously produced grading efforts of a teacher, and<br> opens up new possibilities to support learners of programming in<br> their struggles.</p> <p>## Contents</p> <p>The replication package is structured into two main directories:</p> <p>* &#39;data/&#39;:<br> &nbsp;&nbsp;&nbsp; raw data and scripts that have been used for collecting the data</p> <p>* &#39;scripts/&#39;:<br> &nbsp;&nbsp;&nbsp; scripts for generating the plots that are presented in the paper</p> <p>### RAW data</p> <p>* &#39;data/teacher-data/&#39;<br> &nbsp;&nbsp;&nbsp; data from the scratch workshop: sample solution and scores for student solutions</p> <p>* &#39;data/code-club-stats/&#39;<br> &nbsp;&nbsp;&nbsp; block counts and input methods of the used Code Club projects</p> <p>* &#39;data/coverage`<br> &nbsp;&nbsp;&nbsp; code for measuring the coverage of automated input generation</p> <p>* &#39;data/coverage-results/&#39;<br> &nbsp;&nbsp;&nbsp; coverage measurements on the Code Club projects</p> <p>* &#39;data/test/&#39;<br> &nbsp;&nbsp;&nbsp; test suites for the projects of the Scratch workshop</p> <p>* &#39;data/test-results/&#39;<br> &nbsp;&nbsp;&nbsp; test results from the test suites in &#39;data/test/&#39;</p> <p>* &#39;data/time/&#39;<br> &nbsp;&nbsp;&nbsp; Scratch programs for time measurement (10x the sample solution from &#39;data/teacher-data/`)</p> <p>* &#39;data/time-results/&#39;<br> &nbsp;&nbsp;&nbsp; time measurements on the projects in &#39;data/time/&#39;</p> <p>## Reproducing the Plots</p> <p>### Prerequisites</p> <p>We describe the process based on:</p> <p>&nbsp;&nbsp;&nbsp; * the R statistics package in version 3.5<br> &nbsp;&nbsp;&nbsp; * an Unix environment (Linux or MacOSX)</p> <p>Following R packages are required:</p> <p>&nbsp;&nbsp;&nbsp; * ggplot2<br> &nbsp;&nbsp;&nbsp; * dplyr<br> &nbsp;&nbsp;&nbsp; * viridis</p> <p>The package can be installed with the R command &quot;install.packages&quot;.</p> <p>### Generating the Plots</p> <p>Coverage (Figure 10)</p> <p>&nbsp;&nbsp;&nbsp; ./scripts/coverage.R</p> <p>&nbsp;&nbsp;&nbsp; The result is a set of &quot;coverage-*.pdf&quot; files</p> <p>Inconsistency (Figure 9)</p> <p>&nbsp;&nbsp;&nbsp; ./scripts/consistency.R</p> <p>&nbsp;&nbsp;&nbsp; The result is a set of &quot;consistency-*.pdf&quot; files</p> <p>Scatter Plots (Figure 8, Figure 11)</p> <p>&nbsp;&nbsp;&nbsp; ./scripts/scatter.R</p> <p>&nbsp;&nbsp;&nbsp; The result is a set of &quot;scatter-*.pdf&quot; files</p>