Find research datasets worth reusing

Combining dynamic and static analysis for automated grading SQL statements

<p><strong>Introduction</strong></p> <p>Our experiment was conducted in an undergraduate Relational Database course at the Australian National University.&nbsp;The experiment was conducted on August 10th 2018 when students enrolled in the Relational Database course started to learn relational data model and SQL.&nbsp;The experiment was carried out fully online for three weeks and a total of 393 students were enrolled.&nbsp;The students were asked to login in an online assessment platform and complete 15 exercises.&nbsp;This platform provided an SQLite environment in students browsers by compiling the SQLite C code with Emscripten.</p> <p>Students were allowed to submit and execute their answers in the form of SQL statements.&nbsp;If the execution result of the statement submitted by the student is the same as that of the reference statement,&nbsp;the online assessment platforms will return a feedback message indicating that the execution result is correct.&nbsp;During the interaction with the assessment platform,&nbsp;statements submitted by students were recorded and archived.&nbsp;Overall,&nbsp;our experiment had collected 12,899 statements submitted by students.&nbsp;To create a benchmark dataset that can be used to evaluate different grading approaches,&nbsp;we randomly selected 45 SQL statements submitted by students for each exercise,&nbsp;and asked three teaching assistants to grade them manually.&nbsp;Finally,&nbsp;we average the scores provided by the three assistants and take it as the final score of each statement.&nbsp;The dataset collected in this experiment is ready for public release.</p> <p>All experimental data are stored in Submission.sqlite,&nbsp;which is an SQLite database file.&nbsp;It is recommended to use software such as DB browser or SQLite expert to explore the database.</p> <p>&nbsp;</p> <p><strong>Datatable description</strong></p> <p>&nbsp;</p> <p><em><strong>exercises_result</strong></em></p> <p>This datatable stores the statements submitted by students.&nbsp;Based on the execution result of statement,&nbsp;statements were divided into three categories.</p> <ul> <li>noninterpretable: the statement is non-executable.</li> <li>partially correct: the execution result of statement is different from the expected result.</li> <li>correct: the execution result of the SQL statement is the same as the expected result.</li> </ul> <p>After analyzing the correct statements,&nbsp;we found that the correct set contains some statements carefully constructed by students to deceive the examination system.</p> <p>Take exercise 1 as an example,&nbsp;the task is to answer the following questions using SQL statements.</p> <p>Question:&nbsp;Assume persons who were born in the same year are the same age and there is only one youngest person&nbsp;(with no ties/draws)&nbsp;in this database,&nbsp;who is/are the second youngest person(s)&nbsp;in the database?&nbsp;List the id(s)&nbsp;of the person(s).</p> <p>The reference statement to this exercise is:</p> <pre><code class="language-sql">SELECT p.id FROM person p WHERE p.year_born = (SELECT MAX(year_born) FROM person WHERE year_born &lt; (SELECT MAX(year_born) FROM person)); </code></pre> <p>By exploring the database or trying to execute different statements,&nbsp;some students found that the ID of the person who met the conditions was&nbsp;&#39;00000842&#39;,&nbsp;so the following statement was submitted.</p> <pre><code class="language-sql">select id from person where id ='00000842'; </code></pre> <p>The execution result of the above code was correct,&nbsp;but it was obviously not what the tutor expected.&nbsp;Therefore,&nbsp;we identified such statements as&nbsp;&#39;cheating&#39;.</p> <p>Table 1 Description of exercises_result table.</p> <table> <thead> <tr> <th> <p><strong>field</strong></p> </th> <th> <p><strong>desc</strong></p> </th> <th> <p><strong>datatype</strong></p> </th> </tr> </thead> <tbody> <tr> <td> <p>submission_id</p> </td> <td> <p>Submission ID</p> </td> <td> <p>INT</p> </td> </tr> <tr> <td> <p>submitted_answer</p> </td> <td> <p>statement submitted by student</p> </td> <td> <p>TEXT</p> </td> </tr> <tr> <td> <p>submission_time</p> </td> <td> <p>Submission time</p> </td> <td> <p>NUM</p> </td> </tr> <tr> <td> <p>exercise_id</p> </td> <td> <p>Exercise ID</p> </td> <td> <p>INT</p> </td> </tr> <tr> <td> <p>is_correct</p> </td> <td> <p>Mark whether the statement is correct</p> </td> <td> <p>INT</p> </td> </tr> <tr> <td> <p>student_id</p> </td> <td> <p>Student ID</p> </td> <td> <p>INT</p> </td> </tr> <tr> <td> <p>category</p> </td> <td> <p>categories of statement</p> </td> <td> <p>TEXT</p> </td> </tr> </tbody> </table> <p>&nbsp;</p> <p><em><strong>exercises_benchmark</strong></em></p> <p>This datatable stores the scores provided by different assistants.&nbsp;We randomly selected 45 SQL statements submitted by students for each exercise,&nbsp;and asked three teaching assistants to grade them manually.&nbsp;Finally,&nbsp;we averaged the scores provided by the three assistants as the final score of each statement.</p> <p>Table 2 Description of exercises_benchmark table.</p> <table> <thead> <tr> <th> <p><strong>Field</strong></p> </th> <th> <p><strong>comment</strong></p> </th> <th> <p><strong>datatype</strong></p> </th> </tr> </thead> <tbody> <tr> <td> <p>Submission_id</p> </td> <td> <p>Submission ID</p> </td> <td> <p>INT</p> </td> </tr> <tr> <td> <p>grade</p> </td> <td> <p>grade provided by tutor</p> </td> <td> <p>REAL</p> </td> </tr> <tr> <td> <p>tutor</p> </td> <td> <p>tutor</p> </td> <td> <p>TEXT</p> </td> </tr> </tbody> </table> <p>&nbsp;</p> <p><em><strong>exercises_exercise</strong></em></p> <p>This datatable stores the exercises provided by tutor.</p> <p>Table 3 Description of exercises_exercise table.</p> <table> <thead> <tr> <th> <p><strong>Field</strong></p> </th> <th> <p><strong>comment</strong></p> </th> <th> <p><strong>datatype</strong></p> </th> </tr> </thead> <tbody> <tr> <td> <p>id</p> </td> <td> <p>Exercise ID</p> </td> <td> <p>INT</p> </td> </tr> <tr> <td> <p>title</p> </td> <td> <p>Title of exercise</p> </td> <td> <p>TEXT</p> </td> </tr> <tr> <td> <p>preamble</p> </td> <td> <p>Description of exercise</p> </td> <td> <p>TEXT</p> </td> </tr> <tr> <td> <p>difficulty</p> </td> <td> <p>Coefficient of difficulty</p> </td> <td> <p>integer</p> </td> </tr> <tr> <td> <p>ref</p> </td> <td> <p>Reference statement</p> </td> <td> <p>integer</p> </td> </tr> </tbody> </table> <p>&nbsp;</p> <p><em><strong>database schema</strong></em></p> <p>Please refer to db_schema.pdf for the database schema used in the experiment.</p> <p>&nbsp;</p> <p><strong>BibTex</strong></p> <p>if you want to cite our paper:</p> <p>&nbsp;</p> <blockquote> <pre>@article{wang2020combining, title={Combining dynamic and static analysis for automated grading SQL statements}, author={Wang, Jinshui and Zhao, Yunpeng and Tang, Zhengyi and Xing, Zhenchang}, journal={J Netw Intell}, volume={5}, number={4}, pages={179--190}, year={2020} }</pre> </blockquote>