Web Data Commons (November 2019) Property and Datatype Usage Dataset

<p>This is a dataset about the usage of properties and datatypes in the <a href="http://webdatacommons.org/structureddata/2019-12/stats/stats.html">Web Data Commons RDFa, Microdata, Embedded JSON-LD, and Microformats Data Sets (November 2019)</a> based on the <a href="https://commoncrawl.org/2019/11/november-2019-crawl-archive-now-available/">Common Crawl November 2019 archive</a>. The dataset has been produced using the <a href="https://doi.org/10.5281/zenodo.6338129">RDF Property and Datatype Usage Scanner v2.1.1</a>, which is based on the <a href="https://jena.apache.org/">Apache Jena</a> framework. Only RDFa and embedded JSON-LD data were considered, as Microdata and Microformats do not incorporate explicit datatypes.</p> <p><strong>Dataset Properties</strong></p> <ul> <li><strong>Size</strong>: 37.0 MiB compressed, 785.7 MiB uncompressed, 3&nbsp;542&nbsp;700 rows plus 1 head line determined using <code class="language-bash">gunzip -c measurements.csv.gz | wc -l</code></li> <li><strong>Parsing Failures</strong>: The scanner failed to parse 5&nbsp;080&nbsp;473 triples (~0.053 %) of the source dataset (containing 9&nbsp;547&nbsp;365&nbsp;107 triples).</li> <li><strong>Content</strong>: <ul> <li>CATEGORY: The category (<em>html-embedded-jsonld</em> or <em>html-rdfa</em>) of the Web Data Commons file that has been measured.</li> <li>FILE_URL: The URL of the Web Data Commons file that has been measured.</li> <li>MEASUREMENT: The applied measurement with specific conditions, one of: <ul> <li><strong>UnpreciseRepresentableInDouble</strong>: The number of lexicals that are in the lexical space but not in the value space of <code>xsd:double</code>.</li> <li><strong>UnpreciseRepresentableInFloat</strong>: The number of lexicals that are in the lexical space but not in the value space of <code>xsd:float</code>.</li> <li><strong>UsedAsDatatype</strong>: The total number of literals with the datatype.</li> <li><strong>UsedAsPropertyRange</strong>: The number of statements that specify the datatype as range of the property.</li> <li><strong>ValidDateNotation</strong>: The number of lexicals that are in the lexical space of <code>xsd:date</code>.</li> <li><strong>ValidDateTimeNotation</strong>: The number of lexicals that are in the lexical space of <code>xsd:dateTime</code>.</li> <li><strong>ValidDecimalNotation</strong>: The number of lexicals that represent a number with decimal notation and whose lexical representation is thereby in the lexical space of <code>xsd:decimal</code>, <code>xsd:float</code>, and <code>xsd:double</code>.</li> <li><strong>ValidExponentialNotation</strong>: The number of lexicals that represent a number with exponential notation and whose lexical representation is thereby in the lexical space of <code>xsd:float</code>, and <code>xsd:double</code>.</li> <li><strong>ValidInfOrNaNNotation</strong>: The number of lexicals that equals either <code>INF</code>, <code>+INF</code>, <code>-INF</code> or <code>NaN</code> and whose lexical representation is thereby in the lexical space of <code>xsd:float</code>, and <code>xsd:double</code>.</li> <li><strong>ValidIntegerNotation</strong>: The number of lexicals that represent an integer number and whose lexical representation is thereby in the lexical space of <code>xsd:integer</code>, <code>xsd:decimal</code>, <code>xsd:float</code>, and <code>xsd:double</code>.</li> <li><strong>ValidTimeNotation</strong>: The number of lexicals that are in the lexical space of <code>xsd:time</code>.</li> <li><strong>ValidTrueOrFalseNotation</strong>: The number of lexicals that equal either <code>true</code> or <code>false</code> and whose lexical representation is thereby in the lexical space of <code>xsd:boolean</code>.</li> <li><strong>ValidZeroOrOneNotation</strong>: The number of lexicals that equal either <code>0</code> or <code>1</code> and whose lexical representation is thereby in the lexical space of <code>xsd:boolean</code>, and <code>xsd:integer</code>, <code>xsd:decimal</code>, <code>xsd:float</code>, and <code>xsd:double</code>.</li> </ul> <strong>Note</strong>: Lexical representation of <code>xsd:double</code> values in embedded JSON-LD got normalized to always use exponential notation with up to 16 fractional digits (see <a href="https://github.com/jsonld-java/jsonld-java/blob/v0.13.1/core/src/main/java/com/github/jsonldjava/core/RDFDataset.java#L648">related code</a>). Be careful by drawing conclusions from according <strong>Valid&hellip;</strong> and <strong>Unprecise&hellip;</strong> measures.</li> <li>PROPERTY: The property that has been measured.</li> <li>DATATYPE: The datatype that has been measured.</li> <li>QUANTITY: The count of statements that fulfill the condition specified by the measurement per file, property and datatype.</li> </ul> </li> </ul> <p><strong>Preview</strong></p> <pre><code>"CATEGORY","FILE_URL","MEASUREMENT","PROPERTY","DATATYPE","QUANTITY" "html-rdfa","http://data.dws.informatik.uni-mannheim.de/structureddata/2019-12/quads/dpef.html-rdfa.nq-00000.gz","UnpreciseRepresentableInDouble","https://www.w3.org/2006/vcard/ns#longitude","https://www.w3.org/2001/XMLSchema#float","1" "html-rdfa","http://data.dws.informatik.uni-mannheim.de/structureddata/2019-12/quads/dpef.html-rdfa.nq-00000.gz","UnpreciseRepresentableInDouble","https://www.w3.org/2006/vcard/ns#latitude","https://www.w3.org/2001/XMLSchema#float","1" "html-rdfa","http://data.dws.informatik.uni-mannheim.de/structureddata/2019-12/quads/dpef.html-rdfa.nq-00000.gz","UnpreciseRepresentableInDouble","http://purl.org/goodrelations/v1#hasCurrencyValue","http://www.w3.org/2001/XMLSchema#floatfloat","10" … "html-embedded-jsonld","http://data.dws.informatik.uni-mannheim.de/structureddata/2019-12/quads/dpef.html-embedded-jsonld.nq-01426.gz","ValidZeroOrOneNotation","http://schema.org/ratingValue","http://www.w3.org/2001/XMLSchema#integer","109" "html-embedded-jsonld","http://data.dws.informatik.uni-mannheim.de/structureddata/2019-12/quads/dpef.html-embedded-jsonld.nq-01426.gz","ValidZeroOrOneNotation","http://schema.org/width","http://www.w3.org/2001/XMLSchema#integer","60" "html-embedded-jsonld","http://data.dws.informatik.uni-mannheim.de/structureddata/2019-12/quads/dpef.html-embedded-jsonld.nq-01426.gz","ValidZeroOrOneNotation","http://schema.org/minValue","http://www.w3.org/2001/XMLSchema#integer","68"</code></pre> <p><strong>Note</strong>: The data contain malformed IRIs, like &quot;<code>xsd:dateTime</code>&quot; (instead of probably &quot;<code>http://www.w3.org/2001/XMLSchema#dateTime</code>&quot;), which are caused by missing namespace definitions in the original source website.</p> <p><strong>Reproduce</strong></p> <p>To reproduce this dataset checkout the <a href="https://doi.org/10.5281/zenodo.6337650">RDF Property and Datatype Usage Scanner v2.1.0</a> and execute:</p> <pre><code class="language-bash">mvn clean package java -jar target/Scanner.jar --category html-rdfa --list http://webdatacommons.org/structureddata/2019-12/files/html-rdfa.list November2019 java -jar target/Scanner.jar --category html-embedded-jsonld --list http://webdatacommons.org/structureddata/2019-12/files/html-embedded-jsonld.list November2019 ./measure.sh November2019 # Wait until the scan has completed. This will take a few days java -jar target/Scanner.jar --results ./November2019/measurements.csv.gz November2019 </code></pre>