GPT Capabilites for Extracting Tasks From Textual Process Descriptions

<p>This dataset provides multiple tables which evaluate the capabilities of GPT 1, GPT 2, GPT 3, GPT 3.5, and GPT 4 regarding the extraction of tasks from&nbsp;<a href="https://doi.org/10.5281/zenodo.7783492">https://doi.org/10.5281/zenodo.7783492</a> and PET datasets in model_completeness.zip, model_correctness.xslx and control_evaluation.xlsx.</p><p>The performance of the LLMs is measured by calculating a range of similarity metrics:</p><ul><li>Extracted number of tasks from text vs. extracted number of tasks from Model</li><li>Semantic Text Similarity: Contextual and Non-Contextual between extracted sets of tasks</li><li>Semantic Text Similarity: Contextual and Non-Contextual between extracted individual tasks</li><li>Similarities and Prevalence for length restricted extracted labels</li><li>Similarities and Prevalence for augmented texts (each text has been paraphrased by 9 different paraphrasing methods)</li><li>Recall and Precision for the above mentioned datasets</li><li>Jaccard Index for the above mentioned datasets</li></ul><p>This dataset also includes all data collected from a survey (survey_data.ods), which lets users (currently n=40) evaluate how good LLM created models are: <a href="https://forms.office.com/e/Y55jyNuPi2?origin=lprLink">https://forms.office.com/e/Y55jyNuPi2?origin=lprLink</a></p>