KGCW 2023 Challenge @ ESWC 2023

<h1><strong>Knowledge Graph Construction Workshop 2023: challenge</strong></h1> <p>Knowledge graph construction of heterogeneous data has seen a lot of uptake<br>in the last decade from compliance to performance optimizations with respect<br>to execution time. Besides execution time as a metric for comparing knowledge<br>graph construction, other metrics e.g. CPU or memory usage are not considered.<br>This challenge aims at benchmarking systems to find which RDF graph<br>construction system optimizes for metrics e.g. execution time, CPU,<br>memory usage, or a combination of these metrics.</p> <p><strong>Task description</strong></p> <p>The task is to reduce and report the execution time and computing resources<br>(CPU and memory usage) for the parameters listed in this challenge, compared<br>to the state-of-the-art of the existing tools and the baseline results provided<br>by this challenge. This challenge is not limited to execution times to create<br>the fastest pipeline, but also computing resources to achieve the most efficient<br>pipeline.</p> <p>We provide a tool which can execute such pipelines end-to-end. This tool also<br>collects and aggregates the metrics such as execution time, CPU and memory<br>usage, necessary for this challenge as CSV files. Moreover, the information<br>about the hardware used during the execution of the pipeline is available as<br>well to allow fairly comparing different pipelines. Your pipeline should consist<br>of Docker images which can be executed on Linux to run the tool. The tool is<br>already tested with existing systems, relational databases e.g. MySQL and<br>PostgreSQL, and triplestores e.g. Apache Jena Fuseki and OpenLink Virtuoso<br>which can be combined in any configuration. It is strongly encouraged to use<br>this tool for participating in this challenge. If you prefer to use a different<br>tool or our tool imposes technical requirements you cannot solve, please contact<br>us directly.</p> <p><strong>Part 1: Knowledge Graph Construction Parameters</strong></p> <p>These parameters are evaluated using synthetic generated data to have more<br>insights of their influence on the pipeline.</p> <p><em><strong>Data</strong></em></p> <ul> <li>Number of data records: scaling the data size vertically by the number of records with a fixed number of data properties (10K, 100K, 1M, 10M records).</li> <li>Number of data properties: scaling the data size horizontally by the number of data properties with a fixed number of data records (1, 10, 20, 30 columns).</li> <li>Number of duplicate values: scaling the number of duplicate values in the dataset (0%, 25%, 50%, 75%, 100%).</li> <li>Number of empty values: scaling the number of empty values in the dataset (0%, 25%, 50%, 75%, 100%).</li> <li>Number of input files: scaling the number of datasets (1, 5, 10, 15).</li> </ul> <p><em><strong>Mappings</strong></em></p> <ul> <li>Number of subjects: scaling the number of subjects with a fixed number of predicates and objects (1, 10, 20, 30 TMs).</li> <li>Number of predicates and objects: scaling the number of predicates and objects with a fixed number of subjects (1, 10, 20, 30 POMs).</li> <li>Number of and type of joins: scaling the number of joins and type of joins (1-1, N-1, 1-N, N-M)</li> </ul> <p><strong>Part 2: GTFS-Madrid-Bench</strong></p> <p>The GTFS-Madrid-Bench provides insights in the pipeline with real data from the<br>public transport domain in Madrid.</p> <p><em><strong>Scaling</strong></em></p> <ul> <li>GTFS-1 SQL</li> <li>GTFS-10 SQL</li> <li>GTFS-100 SQL</li> <li>GTFS-1000 SQL</li> </ul> <p><em><strong>Heterogeneity</strong></em></p> <ul> <li>GTFS-100 XML + JSON</li> <li>GTFS-100 CSV + XML</li> <li>GTFS-100 CSV + JSON</li> <li>GTFS-100 SQL + XML + JSON + CSV</li> </ul> <p><strong>Example pipeline</strong></p> <p>The ground truth dataset and baseline results are generated in different steps<br>for each parameter:</p> <ol> <li>The provided CSV files and SQL schema are loaded into a MySQL relational database.</li> <li>Mappings are executed by accessing the MySQL relational database to construct a knowledge graph in N-Triples as RDF format.</li> <li>The constructed knowledge graph is loaded into a Virtuoso triplestore, tuned according to the Virtuoso documentation.</li> <li>The provided SPARQL queries are executed on the SPARQL endpoint exposed by Virtuoso.</li> </ol> <p>The pipeline is executed 5 times from which the median execution time of each<br>step is calculated and reported. Each step with the median execution time is<br>then reported in the baseline results with all its measured metrics.<br>Query timeout is set to 1 hour and knowledge graph construction timeout<br>to 24 hours. The execution is performed with the following tool: <a href="https://github.com/kg-construct/challenge-tool">https://github.com/kg-construct/challenge-tool</a>,<br>you can adapt the execution plans for this example pipeline to your own needs.</p> <p>Each parameter has its own directory in the ground truth dataset with the<br>following files:</p> <ul> <li>Input dataset as CSV.</li> <li>Mapping file as RML.</li> <li>Queries as SPARQL.</li> <li>Execution plan for the pipeline in <code>metadata.json</code>.</li> </ul> <p><strong>Datasets</strong></p> <p><em><strong>Knowledge Graph Construction Parameters</strong></em></p> <p>The dataset consists of:</p> <ul> <li>Input dataset as CSV for each parameter.</li> <li>Mapping file as RML for each parameter.</li> <li>SPARQL queries to retrieve the results for each parameter.</li> <li>Baseline results for each parameter with the example pipeline.</li> <li>Ground truth dataset for each parameter generated with the example pipeline.</li> </ul> <p><em>Format</em></p> <p>All input datasets are provided as CSV, depending on the parameter that is being<br>evaluated, the number of rows and columns may differ. The first row is always<br>the header of the CSV.</p> <p><em><strong>GTFS-Madrid-Bench</strong></em></p> <p>The dataset consists of:</p> <ul> <li>Input dataset as CSV with SQL schema for the scaling and a combination of XML,</li> <li>CSV, and JSON is provided for the heterogeneity.</li> <li>Mapping file as RML for both scaling and heterogeneity.</li> <li>SPARQL queries to retrieve the results.</li> <li>Baseline results with the example pipeline.</li> <li>Ground truth dataset generated with the example pipeline.</li> </ul> <p><em>Format</em></p> <p>CSV datasets always have a header as their first row.<br>JSON and XML datasets have their own schema.</p> <p><strong>Evaluation criteria</strong></p> <p>Submissions must evaluate the following metrics:</p> <ul> <li>Execution time of all the steps in the pipeline. The execution time of a step is the difference between the begin and end time of a step.</li> <li>CPU time as the time spent in the CPU for all steps of the pipeline. The CPU time of a step is the difference between the begin and end CPU time of a step.</li> <li>Minimal and maximal memory consumption for each step of the pipeline. The minimal and maximal memory consumption of a step is the minimum and maximum calculated of the memory consumption during the execution of a step.</li> </ul> <p><strong>Expected output</strong></p> <p><em><strong>Duplicate values</strong></em></p> <table> <tbody><tr> <th>Scale</th> <th>Number of Triples</th> </tr> </tbody><tbody> <tr> <td>0 percent</td> <td>2000000 triples</td> </tr> <tr> <td>25 percent</td> <td>1500020 triples&nbsp;</td> </tr> <tr> <td>50 percent</td> <td>1000020 triples&nbsp;</td> </tr> <tr> <td>75 percent</td> <td>500020 triples</td> </tr> <tr> <td>100 percent</td> <td>20 triples</td> </tr> </tbody> </table> <p>&nbsp;</p> <p><em><strong>Empty values</strong></em></p> <p>&nbsp;</p> <table> <tbody><tr> <th>Scale</th> <th>Number of Triples</th> </tr> </tbody><tbody> <tr> <td>0 percent</td> <td>2000000 triples</td> </tr> <tr> <td>25 percent</td> <td>1500000 triples&nbsp;</td> </tr> <tr> <td>50 percent</td> <td>1000000 triples&nbsp;</td> </tr> <tr> <td>75 percent</td> <td>500000 triples</td> </tr> <tr> <td>100 percent</td> <td>0 triples</td> </tr> </tbody> </table> <p><em><strong>Mappings</strong></em></p> <table> <tbody><tr> <th>Scale</th> <th>Number of Triples</th> </tr> </tbody><tbody> <tr> <td>1TM + 15POM</td> <td>1500000 triples</td> </tr> <tr> <td>3TM + 5POM</td> <td>1500000 triples&nbsp;</td> </tr> <tr> <td>5TM + 3POM&nbsp;</td> <td>1500000 triples&nbsp;</td> </tr> <tr> <td>15TM + 1POM</td> <td>1500000 triples</td> </tr> </tbody> </table> <p><em><strong>Properties</strong></em></p> <table> <tbody><tr> <th>Scale</th> <th>Number of Triples</th> </tr> <tr> <td>1M rows 1 column</td> <td>1000000 triples</td> </tr> <tr> <td>1M rows 10 columns</td> <td>10000000 triples&nbsp;</td> </tr> <tr> <td>1M rows 20 columns</td> <td>20000000 triples&nbsp;</td> </tr> <tr> <td>1M rows 30 columns</td> <td>30000000 triples</td> </tr> </tbody></table> <p><em><strong>Records</strong></em></p> <table> <tbody><tr> <th>Scale</th> <th>Number of Triples</th> </tr> <tr> <td>10K rows 20 columns</td> <td>200000 triples</td> </tr> <tr> <td>100K rows 20 columns</td> <td>2000000 triples&nbsp;</td> </tr> <tr> <td>1M rows 20 columns</td> <td>20000000 triples&nbsp;</td> </tr> <tr> <td>10M rows 20 columns</td> <td>200000000 triples</td> </tr> </tbody></table> <p><em><strong>Joins</strong></em></p> <p><em>1-1 joins</em></p> <table> <tbody><tr> <th>Scale</th> <th>Number of Triples</th> </tr> <tr> <td>0 percent</td> <td>0 triples</td> </tr> <tr> <td>25 percent</td> <td>125000 triples&nbsp;</td> </tr> <tr> <td>50 percent</td> <td>250000 triples&nbsp;</td> </tr> <tr> <td>75 percent</td> <td>375000 triples</td> </tr> <tr> <td>100 percent</td> <td>500000 triples</td> </tr> </tbody></table> <p><em>1-N joins</em></p> <table> <tbody><tr> <th>Scale</th> <th>Number of Triples</th> </tr> <tr> <td>1-10 0 percent</td> <td>0 triples</td> </tr> <tr> <td>1-10 25 percent</td> <td>125000 triples&nbsp;</td> </tr> <tr> <td>1-10 50 percent</td> <td>250000 triples&nbsp;</td> </tr> <tr> <td>1-10 75 percent</td> <td>375000 triples</td> </tr> <tr> <td>1-10 100 percent</td> <td>500000 triples</td> </tr> <tr> <td>1-5 50 percent</td> <td>250000&nbsp;triples</td> </tr> <tr> <td>1-10 50 percent</td> <td>250000 triples&nbsp;</td> </tr> <tr> <td>1-15 50 percent</td> <td>250005 triples&nbsp;</td> </tr> <tr> <td>1-20 50 percent</td> <td>250000 triples</td> </tr> </tbody></table> <p><em>1-N joins</em></p> <table> <tbody><tr> <th>Scale</th> <th>Number of Triples</th> </tr> <tr> <td>10-1 0 percent</td> <td>0 triples</td> </tr> <tr> <td>10-1 25 percent</td> <td>125000 triples&nbsp;</td> </tr> <tr> <td>10-1 50 percent</td> <td>250000 triples&nbsp;</td> </tr> <tr> <td>10-1 75 percent</td> <td>375000 triples</td> </tr> <tr> <td>10-1 100 percent</td> <td>500000 triples</td> </tr> <tr> <td>5-1 50 percent</td> <td>250000&nbsp;triples</td> </tr> <tr> <td>10-1 50 percent</td> <td>250000 triples&nbsp;</td> </tr> <tr> <td>15-1 50 percent</td> <td>250005 triples&nbsp;</td> </tr> <tr> <td>20-1 50 percent</td> <td>250000 triples</td> </tr> </tbody></table> <p><em>N-M joins</em></p> <table> <tbody><tr> <th>Scale</th> <th>Number of Triples</th> </tr> <tr> <td>5-5 50 percent</td> <td>1374085 triples</td> </tr> <tr> <td>10-5 50 percent</td> <td>1375185 triples</td> </tr> <tr> <td>5-10 50 percent&nbsp;</td> <td>1375290 triples</td> </tr> <tr> <td>5-5 25 percent</td> <td>718785 triples</td> </tr> <tr> <td>5-5 50 percent</td> <td>1374085 triples</td> </tr> <tr> <td>5-5 75 percent&nbsp;</td> <td>1968100 triples</td> </tr> <tr> <td>5-5 100 percent&nbsp;</td> <td>2500000 triples&nbsp;</td> </tr> <tr> <td>5-10 25 percent&nbsp;</td> <td>719310 triples</td> </tr> <tr> <td>5-10 50 percent&nbsp;</td> <td>1375290 triples</td> </tr> <tr> <td>5-10 75 percent&nbsp;</td> <td>1967660 triples</td> </tr> <tr> <td>5-10 100 percent&nbsp;</td> <td>2500000 triples</td> </tr> <tr> <td>10-5 25 percent&nbsp;</td> <td>719370 triples&nbsp;</td> </tr> <tr> <td>10-5 50 percent&nbsp;</td> <td>1375185 triples</td> </tr> <tr> <td>10-5 75 percent&nbsp;</td> <td>1968235 triples</td> </tr> <tr> <td>10-5 100 percent&nbsp;</td> <td>2500000 triples</td> </tr> </tbody></table> <p><strong>GTFS Madrid Bench</strong></p> <p><em>Generated Knowledge Graph</em></p> <table> <tbody><tr> <th>Scale</th> <th>Number of Triples</th> </tr> <tr> <td>1</td> <td>395953 triples</td> </tr> <tr> <td>10</td> <td>3959530 triples&nbsp;</td> </tr> <tr> <td>100</td> <td>39595300 triples&nbsp;</td> </tr> <tr> <td>1000</td> <td>395953000 triples</td> </tr> </tbody></table> <p><em>Queries</em></p> <table> <tbody><tr> <th>Query</th> <th>Scale 1</th> <th>Scale 10</th> <th>Scale 100</th> <th>Scale 1000</th> </tr> <tr> <td>Q1</td> <td>58540 results</td> <td>585400 results</td> <td>No results available</td> <td>No results available</td> </tr> <tr> <td>Q2</td> <td>636 results</td> <td>11998 results&nbsp;&nbsp;</td> <td>125565 results</td> <td>1261368 results</td> </tr> <tr> <td>Q3</td> <td>421 results</td> <td>4207 results&nbsp;</td> <td>42067 results</td> <td>420667 results</td> </tr> <tr> <td>Q4</td> <td>13 results</td> <td>130 results</td> <td>1300 results</td> <td>13000 results</td> </tr> <tr> <td>Q5</td> <td>35 results</td> <td>350 results</td> <td>3500 results</td> <td>35000 results</td> </tr> <tr> <td>Q6</td> <td>1 result</td> <td>1 result</td> <td>1 result</td> <td>1 result</td> </tr> <tr> <td>Q7</td> <td>68 results</td> <td>67 results</td> <td>67 results</td> <td>53 results</td> </tr> <tr> <td>Q8</td> <td>35460 results</td> <td>354600 results</td> <td>No results available</td> <td>No results available</td> </tr> <tr> <td>Q9</td> <td>130 results</td> <td>1300 results</td> <td>13000 results</td> <td>130000 results</td> </tr> <tr> <td>Q10</td> <td>1 result</td> <td>1 result</td> <td>1 result</td> <td>1 result</td> </tr> <tr> <td>Q11</td> <td>130 results</td> <td>260 results&nbsp;</td> <td>260 results&nbsp;</td> <td>260 results&nbsp;</td> </tr> <tr> <td>Q12</td> <td>13 results</td> <td>130 results</td> <td>1300 results</td> <td>13000 results</td> </tr> <tr> <td>Q13</td> <td>265 results</td> <td>2650 results</td> <td>26500 results</td> <td>265000 results</td> </tr> <tr> <td>Q14</td> <td>2234 results</td> <td>22340 results</td> <td>223400 results</td> <td>No results available</td> </tr> <tr> <td>Q15</td> <td>592 results</td> <td>8684 results</td> <td>35502 results</td> <td>206628 results</td> </tr> <tr> <td>Q16</td> <td>390 results</td> <td>780 results</td> <td>260 results</td> <td>780 results</td> </tr> <tr> <td>Q17</td> <td>855 results</td> <td>8550 results&nbsp;</td> <td>85500 results&nbsp;</td> <td>855000 results&nbsp;</td> </tr> <tr> <td>Q18</td> <td>104 results</td> <td>1300 results</td> <td>13000 results</td> <td>130000 results</td> </tr> </tbody></table> <p>&nbsp;</p>