Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
36
datasets available to search
ShareScore release 0.9.0
Dataset results
36 results for “benchmark graphs”
MUHAI Benchmark : Task 1 (Short story generation with Knowledge Graphs)
<p><strong>Meaning and Understanding in Human-Centric AI (MUHAI) Benchmark<br> Task 1 (Short story generation with Knowledge Graphs and Language Models)</strong> </p> <p>The dataset can be used to test understandability of text generated through the combination of knowledge graphs and language models without using knowledge graph embeddings.<br> <br> The task here is to generate 5-sentence stories from a set of <em>subject-predicate-object</em> triples that are extracted from a knowledge graph. Two steps need to be performed:</p> <p>1. Language model fine-tuning (SVO triple extraction + model fine-tuning)<br> 2. Story generation (knowledge enrichment + text generation) <br> <br> The submission includes the following data:</p> <ol> <li>Original ROC stories corpus (100 stories)</li> <li>ROC stories encoded with relevant triples (extracted through SpaCy, 2 versions, with and without coreference resolution)</li> <li>Stories generated by the pre-trained model (GPT2-simple)</li> <li>Stories generated by the fine-tuned model (DICE + ConceptNet + DBpedia )</li> <li>Stories generated by the fine-tuned model (DICE + ConceptNet + DBpedia + WordNet )</li> <li>Stories generated by the GPT-2-keyword-generation (an open-source software that uses GPT-2 to generate text pertaining to the specified keywords)</li> <li>Model results</li> <li>Evaluation metrics description</li> <li>User-evaluation questionnaire </li> </ol> <p>Code : https://github.com/kmitd/muhai-dice_story</p>
Lehigh University Benchmark (LUBM): Evolving Graph Simulation
<p>The Lehigh University Benchmark (LUBM) generates benchmark datasets containing people working at universities [1]. We use the Data Generator v1.7 to generate 10 versions of a graph containing 100 universities [2].<br> Thus, all versions are of similar size, but we emulate modifications by generating different vertex identifiers, i.e., each version is considered a timestamped graph. Each graph contains about 2.1 M vertices and 13 M edges.<br> Over all versions, the mean degree is 6.7 (+- 0.1), the mean in-degree is 6.8 (+- 0.1), and the mean out-degree is 5.1 (+- 0.1).</p> <p>1. <a href="https://dblp.uni-trier.de/pid/80/5390.html">Yuanbo Guo</a>, <a href="https://dblp.uni-trier.de/pid/48/6834.html">Zhengxiang Pan</a>, <a href="https://dblp.uni-trier.de/pid/94/1154.html">Jeff Heflin</a>: LUBM: A benchmark for OWL knowledge base systems. <a href="https://dblp.uni-trier.de/db/journals/ws/ws3.html#GuoPH05">J. Web Semant. 3(2-3)</a>: 158-182 (2005)</p> <p>2. <a href="https://dblp.uni-trier.de/pid/222/6353.html">Till Blume</a>, <a href="https://dblp.uni-trier.de/pid/r/DavidRicherby.html">David Richerby</a>, <a href="https://dblp.uni-trier.de/pid/06/2380.html">Ansgar Scherp</a>: Incremental and Parallel Computation of Structural Graph Summaries for Evolving Graphs. <a href="https://dblp.uni-trier.de/db/conf/cikm/cikm2020.html#BlumeRS20">CIKM 2020</a>: 75-84</p>
Berlin Sparql Benchmark (BSBM): Evolving Graph Simulation
<p>The Berlin SPARQL Benchmark (BSBM) is a suite of benchmarks built around an e-commerce use case [1]. We generated 21 versions of the dataset with different scale factors. The first dataset, with a scale factor of 100, contains about 7,000 vertices and 75,000 edges. We generated versions with scale factors between 2,000 and 40,000 in steps of 2,000. The largest dataset contains about 1.3 M vertices and 13 M edges. For our experiments in [2], we first use the different versions ordered from smallest to largest (version 0 to 20) to simulate a growing graph database. Subsequently, we reverse the order to emulate a shrinking graph database. Over all versions, the mean degree is 8.1 (+- 0.5), the mean in-degree is 4.6 (+- 0.3), and the mean out-degree is 9.8 (+- 0.2).</p> <p>1. <a href="https://dblp.uni-trier.de/pid/b/ChristianBizer.html">Christian Bizer</a>, <a href="https://dblp.uni-trier.de/pid/47/7466.html">Andreas Schultz</a>: The Berlin SPARQL Benchmark. <a href="https://dblp.uni-trier.de/db/journals/ijswis/ijswis5.html#BizerS09">Int. J. Semantic Web Inf. Syst. 5(2)</a>: 1-24 (2009)</p> <p>2. <a href="https://dblp.uni-trier.de/pid/222/6353.html">Till Blume</a>, <a href="https://dblp.uni-trier.de/pid/r/DavidRicherby.html">David Richerby</a>, <a href="https://dblp.uni-trier.de/pid/06/2380.html">Ansgar Scherp</a>: Incremental and Parallel Computation of Structural Graph Summaries for Evolving Graphs. <a href="https://dblp.uni-trier.de/db/conf/cikm/cikm2020.html#BlumeRS20">CIKM 2020</a>: 75-84</p>
SCG Dataset from Graph Neural Networks in Supply Chain Analytics and Optimization: Concepts, Perspectives, Dataset and Benchmarks
<p><strong>Abstract:</strong> Graph Neural Networks (GNNs) have recently gained traction in transportation, bioinformatics, language and image processing, but research on their application to supply chain management remains limited. Supply chains are inherently graph-like, making them ideal for GNN methodologies, which can optimize and solve complex problems. The barriers include a lack of proper conceptual foundations, familiarity with graph applications in SCM, and real-world benchmark datasets for GNN-based supply chain research. To address this, we discuss and connect supply chains with graph structures for effective GNN application, providing detailed formulations, examples, mathematical definitions, and task guidelines. Additionally, we present a multi-perspective real-world benchmark dataset from a leading FMCG company in Bangladesh, focusing on supply chain planning. We discuss various supply chain tasks using GNNs and benchmark several state-of-the-art models on homogeneous and heterogeneous graphs across six supply chain analytics tasks. Our analysis shows that GNN-based models consistently outperform statistical ML and other deep learning models by around 10-30% in regression, 10-30% in classification and detection tasks, and 15-40% in anomaly detection tasks on designated metrics. With this work, we lay the groundwork for solving supply chain problems using GNNs, supported by conceptual discussions, methodological insights, and a comprehensive dataset.</p>
hybrid-graph-benchmark
<p>This is a benchmark dataset for evaluating <strong>hybrid-graph</strong> (hypergraph and hierarchical graph) learning algorithms. It contains:</p> <ul> <li>21 real-world higer-order graphs from the domains of biology, social media, and wikipedia</li> </ul> <p>For accessing additional functionalities, please access through <a href="https://github.com/Zehui127/hybrid-graph-benchmark">Project Page</a></p> <ul> <li>Built-in functionalities for preprocessing hybrid-graphs</li> <li>A framework to easily train and evaluate Graph Neural Networks</li> </ul>
Results of KROWN: Knowledge Graph Construction Benchmark
<p>In this Zenodo repository we present the results of using KROWN to benchmark popular RDF Graph Materialization systems such as RMLMapper, RMLStreamer, Morph-KGC, SDM-RDFizer, and Ontop (in materialization mode). </p> <h1>What is KROWN 👑?</h1> <p>KROWN 👑 is a benchmark for materialization systems to construct Knowledge Graphs from (semi-)heterogeneous data sources using declarative mappings such as<a href="http://w3id.org/rml/portal"> RML</a>.</p> <p>Many benchmarks already exist for virtualization systems e.g.<a href="https://github.com/oeg-upm/gtfs-bench"> GTFS-Madrid-Bench</a>,<a href="https://ontop-vkg.org/npd-benchmark/"> NPD</a>,<a href="http://wbsg.informatik.uni-mannheim.de/bizer/berlinsparqlbenchmark/"> BSBM</a> which focus on complex queries with a single declarative mapping. However, materialization systems are unaffected by complex queries since their input is the dataset and the mappings to generate a Knowledge Graph. Some specialized datasets exist to benchmark specific limitations of materialization systems such as duplicated or empty values in datasets e.g.<a href="https://doi.org/10.57702/4c9ivpgs"> GENOMICS</a>, but they do not cover all aspects of materialization systems. Therefore, it is hard to compare materialization systems among each other in general which is where KROWN 👑 comes in! </p> <h1>Results</h1> <p>The raw results are available as ZIP archives, the analysis of the results are available in the spreadsheet <em>results.ods</em>.</p> <h2>Evaluation setup</h2> <p>We generated several scenarios using <a href="https://github.com/kg-construct/KROWN/tree/main/data-generator">KROWN’s data generator</a> and executed them 5 times with <a href="https://github.com/kg-construct/KROWN/tree/main/execution-framework">KROWN’s execution framework</a>. All experiments were performed on Ubuntu 22.04 LTS machines (Linux 5.15.0, x86_64) with each Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz, 48 GB RAM memory, and 2 GB swap memory. The output of each materialization system was set to N-Triples.</p> <h2>Materialization systems</h2> <p>We selected the most popular maintained materialization systems for constructing RDF graphs for performing our experiments with KROWN:</p> <ul> <li> <p>RMLMapper</p> </li> <li> <p>RMLStreamer</p> </li> <li> <p>Morph-KGC</p> </li> <li> <p>SDM-RDFizer</p> </li> <li> <p>OntopM (Ontop in materialization mode)</p> </li> </ul> <p><strong>Note</strong>: KROWN is flexible and allows adding any other materialization system, see <a href="https://github.com/kg-construct/KROWN/tree/main/execution-framework">KROWN’s execution framework</a> documentation for more information.</p> <h2>Scenarios</h2> <p>We consider the following scenarios:</p> <ul> <li> <p>Raw data: number of rows, columns and cell size</p> </li> <li> <p>Duplicates & empty values: percentage of the data containing duplicates or empty values</p> </li> <li> <p>Mappings: Triples Maps (TM), Predicate Object Maps (POM), Named Graph Maps (NG).</p> </li> <li> <p>Joins: relations (1-N, N-1, N-M), conditions, and duplicates during joins</p> </li> </ul> <p><strong>Note</strong>: KROWN is flexible and allows adding any other scenario, see <a href="https://github.com/kg-construct/KROWN/tree/main/data-generator">KROWN’s data generator documentation</a> for more information.</p> <p>In the table below we list all parameter values we used to configure our scenarios:</p> <div> <table> <tbody> <tr> <td> <p><strong>Scenario</strong></p> </td> <td> <p><strong>Parameter values</strong></p> </td> </tr> <tr> <td> <p>Raw data: rows</p> </td> <td> <p>10K, 100K, 1M, 10M</p> </td> </tr> <tr> <td> <p>Raw data: columns</p> </td> <td> <p>1, 10, 20, 30</p> </td> </tr> <tr> <td> <p>Raw data: cell size</p> </td> <td> <p>500, 1K, 5K, 10K </p> </td> </tr> <tr> <td> <p>Duplicates: percentage</p> </td> <td> <p>0%, 25%, 50%, 75%, 100%</p> </td> </tr> <tr> <td> <p>Empty values: percentage</p> </td> <td> <p>0%, 25%, 50%, 75%, 100%</p> </td> </tr> <tr> <td> <p>Mappings: TMs + 5POMs</p> </td> <td> <p>1, 10, 20, 30 TMs</p> </td> </tr> <tr> <td> <p>Mappings: 20TMs + POMs</p> </td> <td> <p>1, 3, 5, 10 POMs</p> </td> </tr> <tr> <td> <p>Mappings: NG in SM</p> </td> <td> <p>1, 5, 10, 15 NGs</p> </td> </tr> <tr> <td> <p>Mappings: NG in POM</p> </td> <td> <p>1, 5, 10, 15 NGs</p> </td> </tr> <tr> <td> <p>Mappings: NG in SM/POM</p> </td> <td> <p>1/1, 5/5, 10/10, 15/15 NGs</p> </td> </tr> <tr> <td> <p>Joins: 1-N relations</p> </td> <td> <p>1-1, 1-5, 1-10, 1-15</p> </td> </tr> <tr> <td> <p>Joins: N-1 relations</p> </td> <td> <p>1-1, 5-1, 10-1, 15-1</p> </td> </tr> <tr> <td> <p>Joins: N-M relations </p> </td> <td> <p>3-3, 3-5, 5-3, 10-5, 5-10</p> </td> </tr> <tr> <td> <p>Joins: join conditions</p> </td> <td> <p>1, 5, 10, 15</p> </td> </tr> <tr> <td> <p>Joins: join duplicates</p> </td> <td> <p>0, 5, 10, 15</p> </td> </tr> </tbody> </table> </div> <h1> </h1>
PheKnowLator Human Disease Knowledge Graph Benchmarks -- v1.0.0
<p><strong>PKT Human Disease Knowledge Graph Benchmark Builds (v1.0.0)</strong></p><p><strong>Build Date: September 03, 2019</strong></p><p>The KG Benchmark Builds can also be downloaded from Zenodo:<br>👉 <strong>KGs:</strong> <a href="https://doi.org/10.5281/zenodo.7030200">https://doi.org/10.5281/zenodo.7030200</a><br>👉 <strong>Embeddings:</strong> <a href="https://zenodo.org/record/7030189">https://zenodo.org/record/7030189</a></p><p> </p><p><strong>Required Input Documents</strong></p><ul><li>resource_info.txt</li><li>class_source_list.txt</li><li>instance_source_list.txt</li><li>ontology_source_list.txt</li></ul><p> </p><p><strong>Data</strong></p><p><strong>Data Download Date:</strong> November 30, 2018</p><p><i><strong>Ontologies</strong></i></p><ul><li><a href="http://purl.obolibrary.org/obo/go.owl">Gene Ontology</a></li><li><a href="http://purl.obolibrary.org/obo/hp.owl">Human Phenotype Ontology</a></li></ul><p><i><strong>Classes</strong></i></p><ul><li><a href="http://purl.obolibrary.org/obo/doid.owl">Human Disease Ontology</a></li><li><a href="http://geneontology.org/gene-associations/goa_human.gaf.gz">Gene Ontology: gene associations</a></li><li><a href="https://reactome.org/download/current/gene_association.reactome">Reactome: gene associations</a></li><li><a href="http://compbio.charite.de/jenkins/job/hpo.annotations.monthly/lastStableBuild/artifact/annotation/ALL_SOURCES_ALL_FREQUENCIES_genes_to_phenotype.txt">Human Phenotype Ontology: all source annotations - genes to phenotypes</a></li><li><a href="http://compbio.charite.de/jenkins/job/hpo.annotations.monthly/lastSt">Human Phenotype Ontology: all source annotations - diseases to genes to phenotypes</a></li></ul><p><i><strong>Instances</strong></i></p><ul><li><a href="http://ctdbase.org/reports/CTD_chem_gene_ixns.tsv.gz">CTD: chemicals-genes</a></li><li><a href="http://ctdbase.org/reports/CTD_chem_pathways_enriched.tsv.gz">CTD: chemicals-pathways</a></li><li><a href="http://ctdbase.org/reports/CTD_chemicals_diseases.tsv.gz">CTD: chemicals-diseases</a></li><li><a href="http://ctdbase.org/reports/CTD_genes_pathways.tsv.gz">CTD: genes-pathways</a></li><li><a href="http://ctdbase.org/reports/CTD_diseases_pathways.tsv.gz">CTD: diseases-pathways</a></li><li><a href="https://stringdb-static.org/download/protein.links.v10.5/9606.protein.links.v10.5.txt.gz">STRING DB: Proteins</a></li><li><a href="https://string-db.org/mapping_files/entrez_mappings/entrez_gene_id.vs.string.v10.28042015.tsv">String DB: entrez gene mappings</a></li></ul><p> </p><p><strong>Knowledge Graphs</strong></p><p><strong>Knowledge Representation</strong><br>We worked with a PhD-level biologist to develop a knowledge representation (see the figure below) that modeled mechanisms underlying human disease.</p><p> </p><p>To do this, we manually mapped all possible combinations of the following six node types:</p><ul><li>Humans Diseases</li><li>Human Phenotypes</li><li>Human Genes</li><li>Gene Ontology concepts</li><li>Reactome Pathways</li><li>Chemicals</li></ul><p>As shown in the figure above, the <a href="http://basic-formal-ontology.org/">Basic Formal Ontology</a> and <a href="https://github.com/oborel/obo-relations/">Relation Ontology</a> ontologies were then used to create edges between the node types.</p><p> </p><p>As shown in this figure, the following edge-types were created:</p><ul><li><strong>Phenotypes-Genes:</strong> The <a href="http://purl.obolibrary.org/obo/hp.owl">Human Phenotype Ontology (HP)</a> provides <a href="http://compbio.charite.de/jenkins/job/hpo.annotations.monthly/lastStableBuild/artifact/annotation/ALL_SOURCES_ALL_FREQUENCIES_genes_to_phenotype.txt">phenotype-Entrez gene annotations</a> that were used to map 6,651 HP classes to 120,288 Entrez genes.</li><li><strong>Phenotypes-Diseases:</strong> The <a href="http://purl.obolibrary.org/obo/hp.owl">HP</a> provides <a href="http://compbio.charite.de/jenkins/job/hpo.annotations.monthly/lastStableBuild/artifact/annotation/ALL_SOURCES_ALL_FREQUENCIES_diseases_to_genes_to_phenotypes.txt">HP-DOID-Gene annotations</a> that were used to map 5,438 HP concepts to 43,817 DOID concepts.</li><li><strong>Biological processes, Molecular Functions, and Cellular Locations-Genes:</strong> The <a href="http://purl.obolibrary.org/obo/go.owl">Gene Ontology (GO)</a> provides <a href="http://geneontology.org/gene-associations/goa_human.gaf.gz">GO-Gene annotations</a> that were used to map 17,505 GO concepts to 265,002 Entrez genes.</li><li><strong>Biological processes, Molecular Functions, and Cellular Locations-Pathways-Pathways:</strong> <a href="https://reactome.org/">Reactome</a> provides <a href="https://reactome.org/download/current/gene_association.reactome">GO-Gene links</a> that were used to map 17,906 pathways to 1,910 biological processes, molecular functions, and cellular locations.</li><li><strong>Chemicals-Pathways:</strong> The <a href="http://ctdbase.org/">Comparative Toxicogenomics Database (CTD)</a> provides <a href="http://ctdbase.org/reports/CTD_chem_pathways_enriched.tsv.gz">Chemical-pathway links</a> that were used to map 8,886 MESH concepts to 711,043 Reactome pathways.</li><li><strong>Chemicals-Genes:</strong> The <a href="http://ctdbase.org/">Comparative Toxicogenomics Database (CTD)</a> provides <a href="http://ctdbase.org/reports/CTD_chem_gene_ixns.tsv.gz">Chemical-Gene links</a> that were used to map 8,881 MESH concepts 410,379 Entrez genes.</li><li><strong>Chemicals-Diseases:</strong> The <a href="http://ctdbase.org/">Comparative Toxicogenomics Database (CTD)</a> provides <a href="http://ctdbase.org/reports/CTD_chemicals_diseases.tsv.gz">Chemical-Disease links</a> that were used to map 14,238 MESH concepts 1,216,900 DOID concepts.</li><li><strong>Genes-Genes:</strong> The<a href="https://string-db.org/">STRING Database</a> provides <a href="https://stringdb-static.org/download/protein.links.v10.5/9606.protein.links.v10.5.txt.gz">Gene-Gene links</a> that were used to create 594,100 gene-gene interactions. When generating these mappings, only the inferred protein-protein relationships considered to be high confidence were used (score of 700 or better).</li><li><strong>Genes-Disease:</strong> Mappings between genes and diseases were retrieved from <a href="http://www.disgenet.org/web/DisGeNET/menu">DisGeNet</a> via SPARQL endpoint and used to map 6,051 Entrez genes to 20,452 DOID concepts.</li><li><strong>Genes-Pathways:</strong> The <a href="http://ctdbase.org/">Comparative Toxicogenomics Database (CTD)</a> provides <a href="http://ctdbase.org/reports/CTD_genes_pathways.tsv.gz">Gene-Pathway links</a> that were used to map 110,370 Entrez genes to 107,029 Reactome pathways.</li><li><strong>Pathways-Disease:</strong> The <a href="http://ctdbase.org/">Comparative Toxicogenomics Database (CTD)</a> provides <a href="http://ctdbase.org/reports/CTD_diseases_pathways.tsv.gz">Pathway-Disease links</a> that were used to map 1,818 Reactome pathways to 106,727 DOID concepts.</li></ul><p> </p><p><strong>Knowledge Graph</strong><br>The knowledge graph represented above was built using the following steps: Merge Ontologies: Merge ontologies using the <a href="https://github.com/owlcollab/owltools/wiki">OWL Tools API</a><br>Express New Ontology Concept Annotations: Create new ontology annotations by asserting a relation between the instance and an instance of the ontology class. For example to assert the following relations:</p><blockquote><p><a href="https://www.ncbi.nlm.nih.gov/mesh/68009020">Morphine</a> --> <a href="https://www.ebi.ac.uk/ols/ontologies/ro/properties?iri=http%3A%2F%2Fpurl.obolibrary.org%2Fobo%2FRO_0002606">is substance that treats</a> --> <a href="https://hpo.jax.org/app/browse/term/HP:0002076">Migraine</a></p><p>We would need to create two axioms:</p><ul><li>isSubstanceThatTreats(Morphine, x1)</li><li>instanceOf(x1, Migraine)</li></ul></blockquote><p>While the instance of the HP class hemiplegic migraines can be treated as an anonymous node in the knowledge graph, we generate a new international resource identifier for each newly generated instance.</p><p><strong>Deductively Close Knowledge Graph:</strong> The knowledge graph is deductively closed by using the OWL 2 EL reasoner, ELK via Protégé v5.1.1. ELK is able to classify instances and supports inferences over class hierarchies and object properties. inference over disjointness, intersection, and existential quantification (ontology class hierarchies).</p><p><strong>Generate Edge List:</strong> The final step before exporting the edge list is to remove any nodes that are not biologically meaningful or would otherwise reduce the performance of machine learning algorithms and the algorithm used to generate embeddings.</p><p> </p><p>🚨 <strong>AVAILABLE FILES </strong>🚨Available KG benchmark files are zipped and listed below. For additional details on what each file contains, please see the associated Wiki page 👉 <a href="https://github.com/callahantiff/PheKnowLator/wiki/September-3,-2019">here</a>.</p>
Datasets for Paper "BenchTemp: A General Benchmark for Evaluating Temporal Graph Neural Networks"
<p>Datasets for Paper "BenchTemp: A General Benchmark for Evaluating Temporal Graph Neural Networks"<br> URL: https://github.com/qianghuangwhu/benchtemp</p> <p>Openreview: https://openreview.net/forum?id=rnZm2vQq31</p> <p><br> There are 19 (15+4) benchmark temporal graph datasets:<br> reddit,<br> wikipedia,<br> mooc,<br> lastfm,<br> enron,<br> SocialEvo,<br> uci,<br> CollegeMsg,<br> TaobaoSmall,<br> CanParl,<br> Contacts,<br> Flights,<br> UNtrade,<br> USLegis,<br> UNvote,</p> <p>DGraphFin,</p> <p>TaobaoLarge,</p> <p>YoutubeReddit,</p> <p>YoutubeRedditLarge</p> <p> </p> <p><br> Each dataset has three files:<br> 1. ml_{data_name}.csv - the csv file of the Temporal Graph.</p> <p>This file have five columns with properties:</p> <p>'u': the id of the user.<br> 'i': the id of the item.<br> 'ts': the timestamp of the interaction (edge) between the user and the item.<br> 'label': the label of the interaction (edge).<br> 'idx': the index of the interaction (edge).<br> For example:</p> <p>,u,i,ts,label,idx<br> 0,1,2,0.0,0.0,1<br> 1,1,3,0.0,0.0,2<br> 2,1,4,0.0,0.0,3<br> 2. ml_{data_name}.npy - the edge features corresponding to the interactions (edges) in the the Temporal Graph..</p> <p>3. ml_{data_name}_node.npy - the initialization node features of the Temporal Graph.</p>
CoDEx: A Comprehensive Knowledge Graph Completion Benchmark
<p>This repository hosts the <strong>relational-only part</strong> of the CoDEx benchmark, which was presented at the EMNLP 2020 conference. You can access the paper <a href="https://www.aclweb.org/anthology/2020.emnlp-main.669.pdf">here</a> and the full dataset, including text and pretrained models, <a href="https://bit.ly/2EPbrJs">on GitHub</a>.</p> <p>Abstract:</p> <p><em>We present CoDEx, a set of knowledge graph completion datasets extracted from Wikidata and Wikipedia that improve upon existing knowledge graph completion benchmarks in scope and level of difficulty. In terms of scope, CoDEx comprises three knowledge graphs varying in size and structure, multilingual descriptions of entities and relations, and tens of thousands of hard negative triples that are plausible but verified to be false. To characterize CoDEx, we contribute thorough empirical analyses and benchmarking experiments. First, we analyze each CoDEx dataset in terms of logical relation patterns. Next, we report baseline link prediction and triple classification results on CoDEx for five extensively tuned embedding models. Finally, we differentiate CoDEx from the popular FB15K-237 knowledge graph completion dataset by showing that CoDEx covers more diverse and interpretable content, and is a more difficult link prediction benchmark. Data, code, and pretrained models are available <a href="https://bit.ly/2EPbrJs">here</a>.</em></p>
PheKnowLator Human Disease Knowledge Graph Benchmarks Archive
<h2><strong>PKT Human Disease KG Benchmark Builds</strong></h2> <p>The PheKnowLator (PKT) Human Disease KG (PKT-KG) was built to model mechanisms of human disease, which includes the Central Dogma and represents multiple biological scales of organization including molecular, cellular, tissue, and organ. The knowledge representation was designed in collaboration with a PhD-level molecular biologist (<a href="https://user-images.githubusercontent.com/8030363/195469903-86598760-40b7-4126-857c-3d6368305a86.png">Figure</a>). </p> <p>The <strong>PKT Human Disease KG</strong> was constructed using 12 OBO Foundry ontologies, 31 Linked Open Data sets, and results from two large-scale experiments (<a href="https://doi.org/10.48550/arXiv.2307.05727">Supplementary Material</a>). The 12 OBO Foundry ontologies were selected to represent chemicals and vaccines (i.e., ChEBI and Vaccine Ontology), cells and cell lines (i.e., Cell Ontology, Cell Line Ontology), gene/gene product attributes (i.e., Gene Ontology), phenotypes and diseases (i.e., Human Phenotype Ontology, Mondo Disease Ontology), proteins, including complexes and isoforms (i.e., Protein Ontology), pathways (i.e., Pathway Ontology), types and attributes of biological sequences (i.e., Sequence Ontology), and anatomical entities (Uberon ontology). The RO is used to provide relationships between the core OBO Foundry ontologies and database entities.</p> <p>The <strong>PKT Human Disease KG</strong> contained 18 node types and 33 edge types. Note that the number of nodes and edge types reflects those that are explicitly added to the core set of OBO Foundry ontologies and does not take into account the node and edge types provided by the ontologies. These nodes and edge types were used to construct 12 different PKT Human Disease benchmark KGs by altering the Knowledge Model (i.e., class- vs. instance-based), Relation Strategy (i.e., standard vs. inverse relations), and Semantic Abstraction (i.e., OWL-NETS (yes/no) with and without Knowledge Model harmonization [OWL-NETS Only vs. OWL-NETS + Harmonization]) parameters. Benchmarks within the PheKnowLator ecosystem are different versions of a KG that can be built under alternative knowledge models, relation strategies, and with or without semantic abstraction. They provide users with the ability to evaluate different modeling decisions (based on the prior mentioned parameters) and to examine the impact of these decisions on different downstream tasks.</p> <p>The Figures and Tables explaining attributes in the builds can be found <a href="https://github.com/callahantiff/PheKnowLator/wiki/Archived-Builds">here</a>.</p> <p> </p> <h3><strong>Build Data Access</strong></h3> <h4><strong>Important Build Information</strong></h4> <p>The benchmarks were originally built and stored using Google Cloud Platform (GCP) resources. For details and a complete description of this process, can be found on GitHub (<a href="https://github.com/callahantiff/PheKnowLator/tree/master/builds#readme">here</a>). Note that we have developed this Zenodo-based archive for the builds. While the original GCP resources contained all of the resources needed to generate the builds, due to the file size upload limits associated with each archive, we have limited the uploaded files to the KGs, associated metadata, and log files. The list of resources, including their URLs, and date of download, can all be found in the logs associated with each build.</p> <p>🗂 For additional information on the KG file types please see the following <a href="https://github.com/callahantiff/PheKnowLator/wiki/KG-Construction#table-knowledge-graph-build-output">Wiki page</a>, which is also available as a download from this repository (PheKnowLator_HumanDiseaseKG_Output_FileInformation.xlsx). </p> <h4><strong>v1.0.0</strong></h4> <ul> <li>KGs: <a href="../doi/10.5281/zenodo.7030200">https://zenodo.org/doi/10.5281/zenodo.7030200</a></li> <li>Embeddings: <a href="../doi/10.5281/zenodo.7030188">https://zenodo.org/doi/10.5281/zenodo.7030188</a></li> </ul> <h4><strong>All Other Build Versions</strong></h4> <p><strong>Class-based Builds</strong></p> <p><em>Standard Relations</em></p> <ul> <li>OWL Build <ul> <li>v2.0.0: <a href="../doi/10.5281/zenodo.7029957">MAY2020</a><a href="../record/8178783">; </a><a href="../doi/10.5281/zenodo.8180239">JAN2021</a>; <a href="../doi/10.5281/zenodo.8180539">FEB2021</a></li> <li>v2.1.0: <a href="../doi/10.5281/zenodo.8180774">MAY2021</a>;<a href="../doi/10.5281/zenodo.8180825"> JUN2021</a>; <a href="../doi/10.5281/zenodo.8180972">JUL2021</a>; <a href="../doi/10.5281/zenodo.8183987">AUG2021</a>;<a href="../doi/10.5281/zenodo.8184090"> SEP2021</a></li> <li>v3.0.2: <a href="../doi/10.5281/zenodo.8184131">OCT2021</a>; <a href="../doi/10.5281/zenodo.8184205">NOV2021</a></li> </ul> </li> <li>OWL-NETS Build <ul> <li>v2.0.0: <a href="../doi/10.5281/zenodo.7029953">MAY2020</a><a href="../record/8178783">; </a><a href="../doi/10.5281/zenodo.8180255">JAN2021</a>; <a href="../doi/10.5281/zenodo.8180545">FEB2021</a></li> <li>v2.1.0: <a href="../doi/10.5281/zenodo.8180772">MAY2021</a>; <a href="../doi/10.5281/zenodo.8180827">JUN2021</a>; <a href="../doi/10.5281/zenodo.8180974">JUL2021</a>; <a href="../doi/10.5281/zenodo.8183989">AUG2021</a>; <a href="../doi/10.5281/zenodo.8184088">SEP2021</a></li> <li>v3.0.2: <a href="../doi/10.5281/zenodo.8184133">OCT2021</a>; <a href="../doi/10.5281/zenodo.8184208">NOV2021</a></li> </ul> </li> </ul> <p><em>Inverse Relations</em></p> <ul> <li>OWL Build <ul> <li>v2.0.0: <a href="../doi/10.5281/zenodo.7029893">MAY2020</a><a href="../record/8178783">; </a><a href="../doi/10.5281/zenodo.8180269">JAN2021</a>; <a href="../doi/10.5281/zenodo.8180550">FEB2021</a></li> <li>v2.1.0: <a href="../doi/10.5281/zenodo.8180766">MAY2021</a>; <a href="../doi/10.5281/zenodo.8180829">JUN2021</a>; <a href="../doi/10.5281/zenodo.8180976">JUL2021</a>;<a href="../doi/10.5281/zenodo.8183991"> AUG2021</a>; <a href="../doi/10.5281/zenodo.8184086">SEP2021</a></li> <li>v3.0.2: <a href="../doi/10.5281/zenodo.8184135">OCT2021</a>; <a href="../doi/10.5281/zenodo.8184210">NOV2021</a></li> </ul> </li> <li>OWL-NETS Build <ul> <li>v2.0.0: <a href="../doi/10.5281/zenodo.7029921">MAY2020</a><a href="../record/8178783">; </a><a href="../doi/10.5281/zenodo.8180279">JAN2021</a>; <a href="../doi/10.5281/zenodo.8180555">FEB2021</a></li> <li>v2.1.0: <a href="../doi/10.5281/zenodo.8180768">MAY2021</a>; <a href="../doi/10.5281/zenodo.8180833">JUN2021</a>; <a href="../doi/10.5281/zenodo.8180982">JUL2021</a>; <a href="../doi/10.5281/zenodo.8183993">AUG2021</a>; <a href="../doi/10.5281/zenodo.8184084">SEP2021</a></li> <li>v3.0.2: <a href="../doi/10.5281/zenodo.8184137">OCT2021</a>; <a href="../doi/10.5281/zenodo.8184212">NOV2021</a></li> </ul> </li> </ul> <p><strong>Instance-based Builds</strong></p> <p><em>Standard Relations</em></p> <ul> <li>OWL Build <ul> <li>v2.0.0: <a href="../doi/10.5281/zenodo.7029941">MAY2020</a><a href="../record/8178783">; </a><a href="../doi/10.5281/zenodo.8180333">JAN2021</a>;<a href="../doi/10.5281/zenodo.8180558"> FEB2021</a></li> <li>v2.1.0: <a href="../doi/10.5281/zenodo.8180764">MAY2021</a>; <a href="../doi/10.5281/zenodo.8180835">JUN2021</a>; <a href="../doi/10.5281/zenodo.8180984">JUL2021</a>; <a href="../doi/10.5281/zenodo.8183995">AUG2021</a>; <a href="../doi/10.5281/zenodo.8184082">SEP2021 </a></li> <li>v3.0.2: <a href="../doi/10.5281/zenodo.8184139">OCT2021</a>; <a href="../doi/10.5281/zenodo.8184216">NOV2021 </a></li> </ul> </li> <li>OWL-NETS Build <ul> <li>v2.0.0: <a href="../doi/10.5281/zenodo.7029939">MAY2020</a><a href="../record/8178783">; </a><a href="../doi/10.5281/zenodo.8180335">JAN2021</a>;<a href="../doi/10.5281/zenodo.8180564"> FEB2021</a></li> <li>v2.1.0: <a href="../doi/10.5281/zenodo.8180762">MAY2021</a>; <a href="../doi/10.5281/zenodo.8180837">JUN2021</a>; <a href="../doi/10.5281/zenodo.8180986">JUL2021</a>; <a href="../doi/10.5281/zenodo.8183997">AUG2021</a>; <a href="../doi/10.5281/zenodo.8184080">SEP2021</a></li> <li>v3.0.2: <a href="../doi/10.5281/zenodo.8184141">OCT2021</a>; <a href="../doi/10.5281/zenodo.8184218">NOV2021</a></li> </ul> </li> </ul> <p><em>Inverse Relations</em></p> <ul> <li>OWL Build <ul> <li>v2.0.0: <a href="../doi/10.5281/zenodo.7029945">MAY2020</a><a href="../record/8178783">; </a><a href="../doi/10.5281/zenodo.8180338">JAN2021</a>; <a href="../doi/10.5281/zenodo.8180588">FEB2021</a></li> <li>v2.1.0: <a href="../doi/10.5281/zenodo.8180758">MAY2021</a>; <a href="../doi/10.5281/zenodo.8180878">JUN2021</a>; <a href="../doi/10.5281/zenodo.8180992">JUL2021</a>; <a href="../doi/10.5281/zenodo.8184001">AUG2021</a>; <a href="../doi/10.5281/zenodo.8184078">SEP2021 </a></li> <li>v3.0.2: <a href="../doi/10.5281/zenodo.8184143">OCT2021</a>; <a href="../doi/10.5281/zenodo.8184220">NOV2021</a></li> </ul> </li> <li>OWL-NETS Build <ul> <li>v2.0.0: <a href="../doi/10.5281/zenodo.7029919">MAY2020</a><a href="../record/8178783">; </a><a href="../doi/10.5281/zenodo.8180340">JAN2021</a>; <a href="../doi/10.5281/zenodo.8180584">FEB2021</a></li> <li>v2.1.0: <a href="../doi/10.5281/zenodo.8180756">MAY2021</a>; <a href="../doi/10.5281/zenodo.8180823">JUN2021</a>; <a href="../doi/10.5281/zenodo.8180996">JUL2021</a>; <a href="../doi/10.5281/zenodo.8184003">AUG2021</a>; <a href="../doi/10.5281/zenodo.8184076">SEP2021 </a></li> <li>v3.0.2: <a href="../doi/10.5281/zenodo.8184145">OCT2021</a>; <a href="../doi/10.5281/zenodo.8184222">NOV2021</a></li> </ul> </li> </ul>
Kiez Benchmarking Knowledge Graph Embeddings
<p>This upload contains pre-calculated Knowledge Graph Embeddings produced by our study "<a href="https://dbs.uni-leipzig.de/file/KIEZ_KEOD_2021_Obraczka_Rahm.pdf">An Evaluation of Hubness Reduction Methods for Entity Alignment with Knowledge Graph Embeddings</a>"</p>
Benchmark Datasets for: EGR: Equivariant Graph Refinement and Assessment of 3D Protein Complex Structures
<p>This archive contains three benchmark datasets associated with the Equivariant Graph Refiner (EGR), two for protein complex structure refinement (PSR Test and Benchmark 2) and the other for protein complex structure assessment (M4S Test). The refinement datasets contain (1) a `pred` directory that contains decoy structure PDB files and (2) a `true` directory that contains native structure PDB files. The quality assessment dataset contains (1) `target_name` directories that each contain decoy structure PDB files for a given protein target and (2) a `label_info.csv` file listing each decoy structure's DockQ score and CAPRI class label.</p>
PGB: A PubMed Graph Benchmark for Heterogeneous Network Representation Learning
<p>PubMed Graph Benchmark (PGB) aggregates the metadata associated with the biomedical articles from PubMed into a unified source. The benchmark contains metadata including title, abstract, authors, in/out citations, MeSH terms, MeSH hierarchy, venue, publication type, and chemicals.</p>
LRGB: Long Range Graph Benchmark
<p>Graph Neural Networks (GNNs) that are based on the message passing (MP) paradigm exchange information between 1-hop neighbors to build node representations at each layer. In principle, such networks are not able to capture long-range interactions (LRI) that may be desired or necessary for learning a given task on graphs. Recently, there has been an increasing interest in development of Transformer-based methods for graphs that can consider full node connectivity beyond the original sparse structure, thus enabling the modeling of LRI. However, MP-GNNs that simply rely on 1-hop message passing often fare better in several existing graph benchmarks when combined with positional feature representations, among other innovations, hence limiting the perceived utility and ranking of Transformer-like architectures. Here, we present the Long Range Graph Benchmark (LRGB) with 5 graph learning datasets: PascalVOC-SP, COCO-SP, PCQM-Contact, Peptides-func and Peptides-struct that arguably require LRI reasoning to achieve strong performance in a given task. We benchmark both baseline GNNs and Graph Transformer networks to verify that the models which capture long-range dependencies perform significantly better on these tasks. Therefore, these datasets are suitable for benchmarking and exploration of MP-GNNs and Graph Transformer architectures that are intended to capture LRI.</p> <p><a href="https://arxiv.org/abs/2206.08164">arXiv</a>, <a href="https://paperswithcode.com/paper/long-range-graph-benchmark">Papers with Code</a></p>
AQL queries and benchmark results from PhD thesis "ANNIS: A graph-based query system for deeply annotated text corpora"
<p>These are the queries, the benchmark results and the evaluation scripts of the thesis "ANNIS: A graph-based query system for deeply annotated text corpora" (Thomas Krause 2018, Humboldt-Universität zu Berlin)</p> <p><strong>diss_2018-01-12_v0.5.0.csv </strong><br> Results of all configurations of executed benchmarks for graphANNIS and also the baseline times of relANNIS.</p> <p><strong>queries.zip</strong><br> Contains folders for each corpus containing all queries used for the benchmark. Each file-name begins with the ID of the query. The extension denotes the type, which can be one of the following:</p> <ul> <li><em>".</em>aql" contains the original AQL (ANNIS query language) query which was collected</li> <li>".json" is JSON representation of the parsed AQL query</li> <li>".count" is the number of matches a query should have</li> <li>".time" is the average time in milliseconds that was needed to execute the query in relANNIS on the benchmark system</li> <li>".corpora" contains the name of the corpus the query belongs to (should be only one corpus and the same as the folder name in the selection of queries in this data set)</li> <li>".relplan" contains the PostgreSQL plan for the query</li> <li>".graphplan" contains the graphANNIS plan for the query</li> </ul> <p><strong>evaluation-scripts.py/evaluation-scripts.ipynb</strong><br> Python scripts to perform the evaluation and generate the images. This are both a Python-file and the original notebook file that can be used with the Jupyter Notebook application.</p> <p><strong>relannis_benchmark_scripts.zip </strong><br> The files in this zip-file can be used to execute the benchmarks in the relANNIS system by piping the into the "annis.sh" command line tool of relANNIS</p>
Task graphs for benchmarking schedulers
<p><strong>Workflow Task Graph Dataset</strong></p> <p>This dataset contains three sets of task graphs representing different types of task workflows:</p> <ul> <li><em>Elementary</em> - contains trivial graph shapes, such as tasks with no dependencies or simple fork-join graphs. This set should test how the scheduler heuristics react to basic graph scenarios that frequently form parts of larger workflows.</li> <li><em>IRW</em> - is inspired by real-world workflows, such as machine learning cross-validation or map-reduce.</li> <li> <em>Pegasus</em> - is derived from graphs created by Pegasus Synthetic Workflow Generators (https://github.com/pegasus-isi/WorkflowGenerator)</li> </ul> <p>All of the provided task graphs are generated and compatible with ESTEE (https://github.com/It4innovations/estee) that allows to simulate their execution on a distributed system using various scheduling heuristics and environment conditions.</p> <p><strong>Data Format</strong></p> <p>Task graphs are stored in {elementary, irw, pegasus}.zip files that contain JSON representation of respective task graphs with the following fields:</p> <ul> <li>`graph_name` - Task graph name</li> <li>`graph_id` - Unique task graph identifier</li> <li> `graph` - Task graph representation - list of tasks where each task is represented as a dictionary with the following keys:</li> <li> `d`: Actual task duration in seconds (float value)</li> <li> `e_d`: User estimated task duration in seconds (float value)</li> <li> `cpus`: Task CPU core requirements (integer value)</li> <li> `outputs`: List of task outputs (list of integers indicating sizes of task outputs in MiB)</li> <li> `inputs`: List of task inputs in format of list [task\_id, output\_index]}. Output index is zero-based.</li> </ul> <p>For example this task graph:</p> <p>[{'d': 200, 'e_d': 180, 'cpus': 1, 'outputs': [100], 'inputs': []},</p> <p>{'d': 50, 'e_d': 60, 'cpus': 2, 'outputs': [], 'inputs': [[0, 0]]}]</p> <p>contains two tasks. One requiring no input, single CPU core with estimated duration 180s, actual duration 200s and producing a single output of 100 MiB. And another one requiring as an input task0's 0-th output, requiring 2 CPU cores, producing no output with estimated duration 60s and actual duration 50s.</p> <p> </p> <p><strong>Parsing the data</strong></p> <p>In Python, to load the elementary task graph set run the following snippet:</p> <pre><code class="language-python">import pandas as pd graphs = pd.read_json("./elementary.zip")</code></pre> <p> </p> <p>If you have Estee installed, you can use its provided `json_deserialize`</p> <p>function to parse the JSON encoded graphs into Estee TaskGraph data structure.</p> <p> </p> <pre><code class="language-python">from estee.serialization.dask_json import json_deserialize graph_json = graphs.loc[0, "graph"] graph = json_deserialize(graph)</code></pre> <p> </p>
A Benchmark Dataset with Knowledge Graph Generation for Industry 4.0 Production Lines
<p>A benchmark dataset for knowledge graph generation in Industry 4.0 production lines and to show the benefits of using ontologies and semantic annotations of data to showcase how I4.0 industry can benefit from KGs and semantic datasets. This work is a result of collaborations with the production line managers, supervisors, and engineers of a football industry to acquire realistic production line data. Knowledge Graphs (KGs) or a Knowledge Graph (KG) emerged as a significant technology to store the semantics of the domain entities. The data is mapped and populated with RGOM classes and relations using an automated solution based on JenaAPI, producing an I4.0 KG. <br> <br> Usage:<br> <br> Recently, we use this dataset to analyze the performance of the five state-of-the-art KG embedding models, namely ComplEx, DistMult,TransE, ConvKB, and ConvE. We evaluated the models using two key metrics: Mean Reciprocal Rank (MRR), and Hits@N (Hits@10, Hits@3, and Hits@1). We observed that the TransE model outperforms other models, followed by ComplEx and DistMult, with ConvE demonstrating the lowest performance. Similarly, the dataset can be used alternatively in other potential scenarios.</p>
Text2KGBench: A Benchmark for Ontology-Driven Knowledge Graph Generation from Text
<p>This is the repository for ISWC 2023 Resource Track submission for <code>Text2KGBench: Benchmark for Ontology-Driven Knowledge Graph Generation from Text</code>. Text2KGBench is a benchmark to evaluate the capabilities of language models to generate KGs from natural language text guided by an ontology. Given an input ontology and a set of sentences, the task is to extract facts from the text while complying with the given ontology (concepts, relations, domain/range constraints) and being faithful to the input sentences.</p> <p>It contains two datasets (i) Wikidata-TekGen with 10 ontologies and 13,474 sentences and (ii) DBpedia-WebNLG with 19 ontologies and 4,860 sentences.</p> <p><strong>An example</strong></p> <p>An example test sentence:</p> <pre><code>Test Sentence: {"id": "ont_music_test_n", "sent": "\"The Loco-Motion\" is a 1962 pop song written by American songwriters Gerry Goffin and Carole King."} </code></pre> <p>An example of ontology:</p> <p>Ontology: <a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/ontologies/owl/ont_2_music.ttl">Music Ontology</a></p> <p>Expected Output:</p> <pre><code>{ "id": "ont_k_music_test_n", "sent": "\"The Loco-Motion\" is a 1962 pop song written by American songwriters Gerry Goffin and Carole King.", "triples": [ { "sub": "The Loco-Motion", "rel": "publication date", "obj": "01 January 1962" },{ "sub": "The Loco-Motion", "rel": "lyrics by", "obj": "Gerry Goffin" },{ "sub": "The Loco-Motion", "rel": "lyrics by", "obj": "Carole King" },] } </code></pre> <p>The data is released under a Creative Commons Attribution-ShareAlike 4.0 International (CC BY 4.0) License.</p> <p>The structure of the repo is as the following.</p> <ul> <li>Text2KGBench <ul> <li>src: the source code used for generation and evaluation, and baseline <ul> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/src/benchmark"><code>benchmark</code></a> the code used to generate the benchmark</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/src/evaluation"><code>evaluation</code></a> evaluation scripts for calculating the results</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/src/evaluation">baseline</a> code for generating the baselines including prompts, sentence similarities, and LLM client.</li> </ul> </li> <li>data: the benchmark datasets and baseline data. There are two datasets: wikidata_tekgen and dbpedia_webnlg. <ul> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen">wikidata_tekgen</a> Wikidata-TekGen Dataset <ul> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/ontologies">ontologies</a> 10 ontologies used by this dataset</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/train">train</a> training data</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/test">test</a> test data</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/manually_verified_sentences">manually_verified_sentences</a> ids of a subset of test cases manually validated</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/unseen_sentences">unseen_sentences</a> new sentences that are added by the authors which are not part of Wikipedia <ul> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/unseen_sentences/test">test unseen</a> test unseen test sentences</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/unseen_sentences/ground_truth">ground_truth</a> ground truth for unseen test sentences.</li> </ul> </li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/ground_truth">ground_truth</a> ground truth for the test data</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/baselines">baselines</a> data related to running the baselines. <ul> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/baselines/test_train_sent_similarity">test_train_sent_similarity</a> for each test case, 5 most similar train sentences generated using SBERT T5-XXL model.</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/baselines/prompts">prompts</a> prompts corresponding to each test file <ul> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/baselines/prompts/unseen">unseen prompts</a> unseen prompts for the unseen test cases</li> </ul> </li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/baselines/Alpaca-LoRA-13B">Alpaca-LoRA-13B</a> data related to the Alpaca-LoRA model <ul> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/baselines/Alpaca-LoRA-13B/llm_responses">llm_responses</a> raw LLM responses and extracted triples</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/baselines/Alpaca-LoRA-13B/eval_metrics">eval_metrics</a> ontology-level and aggregated evaluation results</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/baselines/Alpaca-LoRA-13B/unseen">unseen results</a> results for the unseen test cases <ul> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/baselines/Alpaca-LoRA-13B/unseen/llm_responses">llm_responses</a> raw LLM responses and extracted triples</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/baselines/Alpaca-LoRA-13B/unseen/eval_metrics">eval_metrics</a> ontology-level and aggregated evaluation results</li> </ul> </li> </ul> </li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/baselines/Vicuna-13B">Vicuna-13B</a> data related to the Vicuna-13B model <ul> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/baselines/Vicuna-13B/llm_responses">llm_responses</a> raw LLM responses and extracted triples</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/baselines/Vicuna-13B/eval_metrics">eval_metrics</a> ontology-level and aggregated evaluation results</li> </ul> </li> </ul> </li> </ul> </li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/dbpedia_webnlg">dbpedia_webnlg</a> DBpedia Dataset <ul> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/dbpedia_webnlg/ontologies">ontologies</a> 19 ontologies used by this dataset</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/dbpedia_webnlg/train">train</a> training data</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/dbpedia_webnlg/test">test</a> test data</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/dbpedia_webnlg/ground_truth">ground_truth</a> ground truth for the test data</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/dbpedia_webnlg/baselines">baselines</a> data related to running the baselines. <ul> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/dbpedia_webnlg/baselines/test_train_sent_similarity">test_train_sent_similarity</a> for each test case, 5 most similar train sentences generated using SBERT T5-XXL model.</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/dbpedia_webnlg/baselines/prompts">prompts</a> prompts corresponding to each test file</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/dbpedia_webnlg/baselines/Alpaca-LoRA-13B">Alpaca-LoRA-13B</a> data related to the Alpaca-LoRA model <ul> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/dbpedia_webnlg/baselines/Alpaca-LoRA-13B/llm_responses">llm_responses</a> raw LLM responses and extracted triples</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/dbpedia_webnlg/baselines/Alpaca-LoRA-13B/eval_metrics">eval_metrics</a> ontology-level and aggregated evaluation results</li> </ul> </li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/dbpedia_webnlg/baselines/Vicuna-13B">Vicuna-13B</a> data related to the Vicuna-13B model <ul> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/dbpedia_webnlg/baselines/Vicuna-13B/llm_responses">llm_responses</a> raw LLM responses and extracted triples</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/dbpedia_webnlg/baselines/Vicuna-13B/eval_metrics">eval_metrics</a> ontology-level and aggregated evaluation results</li> </ul> </li> </ul> </li> </ul> </li> </ul> </li> </ul> </li> </ul> <p>This benchmark contains data derived from the TekGen corpus (part of the KELM corpus) [1] released under CC BY-SA 2.0 license and WebNLG 3.0 corpus [2] released under CC BY-NC-SA 4.0 license.</p> <p>[1] Oshin Agarwal, Heming Ge, Siamak Shakeri, and Rami Al-Rfou. 2021. Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3554–3565, Online. Association for Computational Linguistics.</p> <p>[2] Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. Creating Training Corpora for NLG Micro-Planners. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 179–188, Vancouver, Canada. Association for Computational Linguistics.</p>
Supplementary dataset for "Evaluating Graph Neural Networks for Link Prediction: Current Pitfalls and New Benchmarking"
<p>The supplementary dataset for the paper "Evaluating Graph Neural Networks for Link Prediction: Current Pitfalls and New Benchmarking". We include the splits for cora, citeseer, and pubmed, the hard negative samples, and the node2vec embeddings. We also include a jupyter file <em>read_data.ipynb</em> to show how to read the non-txt file.</p> <ul> <li>heart_test_samples.npy, heart_valid_samples.npy: the heard negative samples</li> <li>*-n2v-embedding.pt: node2vec embeddings</li> <li>test_samples_index.pt, valid_samples_index.pt: the node index of the selected samples in ogbl-ppa under HeaRT</li> <li>gnn_feature: the input feature of cora, citeseer, pubmed</li> </ul> <p>More details for our code and how to use the dataset are on the code repository: https://github.com/Juanhui28/HeaRT .</p>
Results and log of LLM-KG-Bench runs described in article "Developing a Scalable Benchmark for Assessing Large Language Models in Knowledge Graph Engineering", Meyer et al. 2023
<p>Results and logs of <a href="https://github.com/AKSW/LLM-KG-Bench">LLM-KG-Bench</a> runs described in article "Developing a Scalable Benchmark for Assessing Large Language Models in Knowledge Graph Engineering", Meyer et al., to appear in <a href="https://2023-eu.semantics.cc/page/accepted_posters">SEMANTICS 2023 poster track</a> proceedings.</p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.