Text2KGBench: A Benchmark for Ontology-Driven Knowledge Graph Generation from Text

<p>This is the repository for ISWC 2023 Resource Track submission for&nbsp;<code>Text2KGBench: Benchmark for Ontology-Driven Knowledge Graph Generation from Text</code>. Text2KGBench is a benchmark to evaluate the capabilities of language models to generate KGs from natural language text guided by an ontology. Given an input ontology and a set of sentences, the task is to extract facts from the text while complying with the given ontology (concepts, relations, domain/range constraints) and being faithful to the input sentences.</p> <p>It contains two datasets (i) Wikidata-TekGen with 10 ontologies and 13,474 sentences and (ii) DBpedia-WebNLG with 19 ontologies and 4,860 sentences.</p> <p><strong>An example</strong></p> <p>An example test sentence:</p> <pre><code>Test Sentence: {"id": "ont_music_test_n", "sent": "\"The Loco-Motion\" is a 1962 pop song written by American songwriters Gerry Goffin and Carole King."} </code></pre> <p>An example of ontology:</p> <p>Ontology:&nbsp;<a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/ontologies/owl/ont_2_music.ttl">Music Ontology</a></p> <p>Expected Output:</p> <pre><code>{ "id": "ont_k_music_test_n", "sent": "\"The Loco-Motion\" is a 1962 pop song written by American songwriters Gerry Goffin and Carole King.", "triples": [ { "sub": "The Loco-Motion", "rel": "publication date", "obj": "01 January 1962" },{ "sub": "The Loco-Motion", "rel": "lyrics by", "obj": "Gerry Goffin" },{ "sub": "The Loco-Motion", "rel": "lyrics by", "obj": "Carole King" },] } </code></pre> <p>The data is released under a Creative Commons Attribution-ShareAlike 4.0 International (CC BY 4.0) License.</p> <p>The structure of the repo is as the following.</p> <ul> <li>Text2KGBench <ul> <li>src: the source code used for generation and evaluation, and baseline <ul> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/src/benchmark"><code>benchmark</code></a>&nbsp;the code used to generate the benchmark</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/src/evaluation"><code>evaluation</code></a>&nbsp;evaluation scripts for calculating the results</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/src/evaluation">baseline</a>&nbsp;code for generating the baselines including prompts, sentence similarities, and LLM client.</li> </ul> </li> <li>data: the benchmark datasets and baseline data. There are two datasets: wikidata_tekgen and dbpedia_webnlg. <ul> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen">wikidata_tekgen</a>&nbsp;Wikidata-TekGen Dataset <ul> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/ontologies">ontologies</a>&nbsp;10 ontologies used by this dataset</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/train">train</a>&nbsp;training data</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/test">test</a>&nbsp;test data</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/manually_verified_sentences">manually_verified_sentences</a>&nbsp;ids of a subset of test cases manually validated</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/unseen_sentences">unseen_sentences</a>&nbsp;new sentences that are added by the authors which are not part of Wikipedia <ul> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/unseen_sentences/test">test unseen</a>&nbsp;test unseen test sentences</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/unseen_sentences/ground_truth">ground_truth</a>&nbsp;ground truth for unseen test sentences.</li> </ul> </li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/ground_truth">ground_truth</a>&nbsp;ground truth for the test data</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/baselines">baselines</a>&nbsp;data related to running the baselines. <ul> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/baselines/test_train_sent_similarity">test_train_sent_similarity</a>&nbsp;for each test case, 5 most similar train sentences generated using SBERT T5-XXL model.</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/baselines/prompts">prompts</a>&nbsp;prompts corresponding to each test file <ul> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/baselines/prompts/unseen">unseen prompts</a>&nbsp;unseen prompts for the unseen test cases</li> </ul> </li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/baselines/Alpaca-LoRA-13B">Alpaca-LoRA-13B</a>&nbsp;data related to the Alpaca-LoRA model <ul> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/baselines/Alpaca-LoRA-13B/llm_responses">llm_responses</a>&nbsp;raw LLM responses and extracted triples</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/baselines/Alpaca-LoRA-13B/eval_metrics">eval_metrics</a>&nbsp;ontology-level and aggregated evaluation results</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/baselines/Alpaca-LoRA-13B/unseen">unseen results</a>&nbsp;results for the unseen test cases <ul> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/baselines/Alpaca-LoRA-13B/unseen/llm_responses">llm_responses</a>&nbsp;raw LLM responses and extracted triples</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/baselines/Alpaca-LoRA-13B/unseen/eval_metrics">eval_metrics</a>&nbsp;ontology-level and aggregated evaluation results</li> </ul> </li> </ul> </li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/baselines/Vicuna-13B">Vicuna-13B</a>&nbsp;data related to the Vicuna-13B model <ul> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/baselines/Vicuna-13B/llm_responses">llm_responses</a>&nbsp;raw LLM responses and extracted triples</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/wikidata_tekgen/baselines/Vicuna-13B/eval_metrics">eval_metrics</a>&nbsp;ontology-level and aggregated evaluation results</li> </ul> </li> </ul> </li> </ul> </li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/dbpedia_webnlg">dbpedia_webnlg</a>&nbsp;DBpedia Dataset <ul> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/dbpedia_webnlg/ontologies">ontologies</a>&nbsp;19 ontologies used by this dataset</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/dbpedia_webnlg/train">train</a>&nbsp;training data</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/dbpedia_webnlg/test">test</a>&nbsp;test data</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/dbpedia_webnlg/ground_truth">ground_truth</a>&nbsp;ground truth for the test data</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/dbpedia_webnlg/baselines">baselines</a>&nbsp;data related to running the baselines. <ul> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/dbpedia_webnlg/baselines/test_train_sent_similarity">test_train_sent_similarity</a>&nbsp;for each test case, 5 most similar train sentences generated using SBERT T5-XXL model.</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/dbpedia_webnlg/baselines/prompts">prompts</a>&nbsp;prompts corresponding to each test file</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/dbpedia_webnlg/baselines/Alpaca-LoRA-13B">Alpaca-LoRA-13B</a>&nbsp;data related to the Alpaca-LoRA model <ul> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/dbpedia_webnlg/baselines/Alpaca-LoRA-13B/llm_responses">llm_responses</a>&nbsp;raw LLM responses and extracted triples</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/dbpedia_webnlg/baselines/Alpaca-LoRA-13B/eval_metrics">eval_metrics</a>&nbsp;ontology-level and aggregated evaluation results</li> </ul> </li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/dbpedia_webnlg/baselines/Vicuna-13B">Vicuna-13B</a>&nbsp;data related to the Vicuna-13B model <ul> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/dbpedia_webnlg/baselines/Vicuna-13B/llm_responses">llm_responses</a>&nbsp;raw LLM responses and extracted triples</li> <li><a href="https://github.com/cenguix/Text2KGBench/blob/main/data/dbpedia_webnlg/baselines/Vicuna-13B/eval_metrics">eval_metrics</a>&nbsp;ontology-level and aggregated evaluation results</li> </ul> </li> </ul> </li> </ul> </li> </ul> </li> </ul> </li> </ul> <p>This benchmark contains data derived from the TekGen corpus (part of the KELM corpus) [1] released under CC BY-SA 2.0 license and WebNLG 3.0 corpus [2] released under CC BY-NC-SA 4.0 license.</p> <p>[1] Oshin Agarwal, Heming Ge, Siamak Shakeri, and Rami Al-Rfou. 2021. Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3554&ndash;3565, Online. Association for Computational Linguistics.</p> <p>[2] Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. Creating Training Corpora for NLG Micro-Planners. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 179&ndash;188, Vancouver, Canada. Association for Computational Linguistics.</p>