Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
111
datasets available to search
ShareScore release 0.9.0
Dataset results
111 results for “gold standard”
MEDDOCAN corpus: gold standard annotations for Medical Document Anonymization on Spanish clinical case reports
<p><strong>Intro:</strong></p> <p>Meddocan shared task dataset (divided in train, dev and test). In addition, we include here the Meddocan background set.</p> <p>It contains the training, development and test sets of the Meddocan shared task with Gold Standard annotations.</p> <p>In addition, it contains the documents of the background set, without annotations.</p> <p> </p> <p><strong>Annotation quality</strong></p> <p>Inter-annotator agreement: 98% </p> <p>For more information, see the <a href="http://ceur-ws.org/Vol-2421/MEDDOCAN_overview.pdf">paper</a>. </p> <p> </p> <p><strong>Format:</strong></p> <p>Annotations are distributed in Brat format. See <a href="https://brat.nlplab.org/standoff.html">Brat webpage</a> for more information.</p> <p>In addition, annotations are also distributed in XML format (based on i2b2 XML format).</p> <p>In the <a href="https://temu.bsc.es/meddocan/index.php/resources/">Meddocan webpage</a>, there is a script to convert between MEDDOCAN-Brat, MEDDOCAN-XML, and i2b2 formats.</p> <p> </p> <p><strong>Shared task goal:</strong></p> <p>In the three subtasks, the goal will be to predict the annotations given only the plain text files. </p> <p> </p> <p><strong>Resources:</strong></p> <ul> <li><strong><a href="https://temu.bsc.es/meddocan/">Web</a></strong></li> <li><strong>Citation: </strong>Montserrat Marimon et al. “Automatic De-identification of Medical Texts in Spanish: the MEDDOCAN Track, Corpus, Guidelines, Methods and Evaluation of Results.” In: IberLEF@ SEPLN. 2019, pp. 618–638.</li> <li><strong>Silver Standard corpus</strong></li> <li><a href="https://doi.org/10.5281/zenodo.4279337"><strong>Annotation guidelines</strong></a></li> </ul> <p> </p> <p>For further information, please visit <a href="https://temu.bsc.es/meddocan/">https://temu.bsc.es/meddocan/</a> or email us at encargo-pln-life@bsc.es</p> <p>Copyright (c) 2019 Secretaría de Estado para el Avance Digital (SEAD)</p>
Medical Relation Extraction Gold Standard with CrowdTruth
<p>The lack of annotated datasets for training and benchmarking is one of the main challenges of Clinical Natural Language Processing. In addition, current methods for collecting annotations attempt to minimize disagreement between annotators, and therefore fail to model the ambiguity inherent in language. We propose the <strong>CrowdTruth</strong> method for collecting medical ground truth through crowdsourcing, based on the observation that disagreement between annotators can signal ambiguity in the text, target semantics, or the worker's interpretation.</p> <p>This repository contains a dataset of 3,984 English sentences for medical relation extraction, centering on the cause and treat medical relations, that have been processed with CrowdTruth disagreement analytics to capture ambiguity. In addition, we provide the raw crowdsourcing data used to compile this ground truth, as well as the task templates used to collect the data on CrowdFlower.</p>
A rule based Tibetan part-of-speech (POS) tagger for the creation of gold standard training data
<p>This rule based Tibetan part-of-speech (POS) tagger was prepared in the course of the research project 'Tibetan in Digital Communication' (2012-2015) hosted at SOAS, University of London and funded by the UK's Arts and Humanities Research Council (grant code: AH/J00152X/1). For a description of the tag set see Garrett et al. 2014. and Garrett et al. 2015. For a description of the tagger itself see Garrett et al. 2014. Note that the tagger must be used together with a lexicon (for example Hill & Garrett 2017a). One must use one's own script to tag all words with all tags in the lexicon and then apply the tagger to remove incorrect tags.</p> <p>On the associated corpus of 318,230 words (Hill & Garrett 2017b) the lexical tagger (i.e. simply applying all available tags to all words) tags 141,911 words with the correct unique tag, achieves as accuracy of 1.000 (by definition getting the right tag among others for each word) with an ambiguity of 2.73111. In contrast, the Rule Tagger tags 241,256 words with the correct unique tag, achieves an accuracy of 0.99893 and an ambiguity of 1.38577.</p> <p>Because this tagger does not achieve ambiguity 1.000 it is not suitable for tagging large scale corpora, but instead is useful for the creation of gold standard training data.</p> <p>N.B. In some rare cases the tagger removes all POS-tags.</p>
Files and code for English dictionaries, gold and silver standard corpora for biomedical natural language processing related to SARS-CoV-2 and COVID-19
<p><span lang="EN-GB">Automated information extraction with natural language processing (NLP) tools is required to gain systematic insights from the large number of COVID-19 publications, reports and social media posts, which far exceed human processing capabilities. </span></p> <p><span lang="EN-GB">Here we present an NLP toolbox comprising COVID-19-related dictionaries and annotated corpora in English as well as useful code and workflows for their update and use. The dictionaries contain terms referring to the COVID-19 disease, the SARS-CoV-2 virus, its variants and common mutations, respectively. They were used together with the EasyNER NLP tool to extract and annotate all 764 398 abstracts in the CORD-19 dataset, creating a very large silver standard corpus (named Lund-Annotated-CORD-19 corpus). This was complemented with a small gold standard corpus consisting of PubMed abstracts manually annotated for key entity classes such as disease, virus, symptom, protein/gene, cell type, chemical and species terms. </span></p> <p><span lang="EN-GB">The toolbox can support various text analysis tasks related to COVID-19 such as named entity recognition and co-mention analysis. A preliminary version of the toolbox, which was released early in the pandemic, was</span><span lang="EN-GB"> for example already used to create a COVID-19 knowledge graph and study the evolution and variation of COVID-19-related terminology. In addition, the toolbox can be applied in the development of other NLP tools, for example to train and evaluate large language models.</span></p> <p><span lang="EN-GB">When using the toolbox, please cite this record and the associated article.</span></p> <p> </p> <p> </p>
GeoEDdA: A Gold Standard Dataset for Named Entity Recognition and Span Categorization Annotations of Diderot & d'Alembert's Encyclopédie
<p>This repository contains a gold standard dataset for named entity recognition and span categorization annotations from Diderot & d’Alembert’s Encyclopédie entries.</p> <p>The dataset is available in the following formats:</p> <ul> <li>JSONL format provided by <a href="https://prodi.gy/" rel="nofollow">Prodigy</a></li> <li>binary spaCy format (ready to use with the spaCy train pipeline)</li> </ul> <p>The Gold Standard dataset is composed of 2,200 paragraphs out of 2,001 Encyclopédie's entries randomly selected. All paragraphs were written in 19th-century French.</p> <p>The spans/entities were labeled by the project team along with using pre-labelling with early machine learning models to speed up the labelling process. A train/val/test split was used. Validation and test sets are composed of 200 paragraphs each: 100 classified under 'Géographie' and 100 from another knowledge domain. The datasets have the following breakdown of tokens and spans/entities.</p> <h2>Tagset</h2> <ul> <li><strong>NC-Spatial</strong>: a common noun that identifies a spatial entity (nominal spatial entity) including natural features, e.g. <code>ville</code>, <code>la rivière</code>, <code>royaume</code>.</li> <li><strong>NP-Spatial</strong>: a proper noun identifying the name of a place (spatial named entities), e.g. <code>France</code>, <code>Paris</code>, <code>la Chine</code>.</li> <li><strong>ENE-Spatial</strong>: nested spatial entity , e.g. <code>ville de France</code> , <code>royaume de Naples</code>, <code>la mer Baltique</code>.</li> <li><strong>Relation</strong>: spatial relation, e.g. <code>dans</code>, <code>sur</code>, <code>à 10 lieues de</code>.</li> <li><strong>Latlong</strong>: geographic coordinates, e.g. <code>Long. 19. 49. lat. 43. 55. 44.</code></li> <li><strong>NC-Person</strong>: a common noun that identifies a person (nominal spatial entity), e.g. <code>roi</code>, <code>l'empereur</code>, <code>les auteurs</code>.</li> <li><strong>NP-Person</strong>: a proper noun identifying the name of a person (person named entities), e.g. <code>Louis XIV</code>, <code>Pline</code>.</li> <li><strong>ENE-Person</strong>: nested people entity, e.g. <code>le czar Pierre</code>, <code>roi de Macédoine</code>.</li> <li><strong>NP-Misc</strong>: a proper noun identifying entities not classified as spatial or person, e.g. <code>l'Eglise</code>, <code>1702</code>, <code>Pélasgique</code></li> <li><strong>ENE-Misc</strong>: nested named entity not classified as spatial or person, e.g. <code>l'ordre de S. Jacques</code>, <code>la déclaration du 21 Mars 1671</code>.</li> <li><strong>Head</strong>: entry name</li> <li><strong>Domain-Mark</strong>: words indicating the knowledge domain (usually after the head and between parenthesis), e.g. <code>Géographie</code>, <code>Geog.</code>, <code>en Anatomie</code>.</li> </ul> <h2>HuggingFace</h2> <p>The GeoEDdA dataset is available on the HuggingFace Hub: <a href="https://huggingface.co/datasets/GEODE/GeoEDdA">https://huggingface.co/datasets/GEODE/GeoEDdA</a></p> <h2>spaCy Custom Spancat trained on Diderot & d’Alembert’s Encyclopédie entries</h2> <p>This dataset was used to train and evaluate a custom spancat model for French using <a href="https://spacy.io/" rel="nofollow">spaCy</a>. The model is available on HuggingFace's model hub: <a href="https://huggingface.co/GEODE/fr_spacy_custom_spancat_edda" rel="nofollow">https://huggingface.co/GEODE/fr_spacy_custom_spancat_edda</a>.</p> <h2>Acknowledgement</h2> <p>The authors are grateful to the <a href="https://aslan.universite-lyon.fr/" rel="nofollow">ASLAN project</a> (ANR-10-LABX-0081) of the Université de Lyon, for its financial support within the French program "Investments for the Future" operated by the National Research Agency (ANR). Data courtesy the <a href="https://artfl-project.uchicago.edu/" rel="nofollow">ARTFL Encyclopédie Project</a>, University of Chicago.</p>
DLCC Gold Standard
<p>Corresponding GitHub repository: <a href="https://github.com/janothan/DL-TC-Generator">DL-TC-Generator on GitHub</a></p> <p> </p> <p><strong>Abstract</strong></p> <p>Knowledge graph embedding is a representation learning technique which projects entities and relations in a knowledge graph to continuous vector spaces.<br> Embeddings have gained a lot of uptake and have been heavily used in link prediction and other downstream prediction tasks.<br> Most approaches are evaluated on a single task or a single group of tasks to determine their overall performance. The evaluation is then assessed in terms of how well the embedding approach performs on the task at hand, but it is hardly evaluated (and often not even deeply understood) what information the embedding approaches are <em>actually</em> learning to represent.</p> <p>To fill this gap, we present the DLCC (Description Logic Class Constructors) benchmark, a resource to analyze embedding approaches in terms of which kinds of classes they can represent. Two gold standards are presented, one based on the real world knowledge graph DBpedia, and one synthetic gold standard.</p>
SocialDisNER corpus: gold standard annotations for detection of disease mentions in Spanish tweets
<p><strong>If you use any data from this repository, please cite our scientific paper instead of the Zenodo repo: </strong></p> <p>Luis Gasco Sánchez, Darryl Estrada Zavala, Eulàlia Farré-Maduell, Salvador Lima-López, Antonio Miranda-Escalada, and Martin Krallinger. 2022. <a href="https://aclanthology.org/2022.smm4h-1.48">The SocialDisNER shared task on detection of disease mentions in health-relevant content from social media: methods, evaluation, guidelines and corpora</a>. In <em>Proceedings of The Seventh Workshop on Social Media Mining for Health Applications, Workshop & Shared Task</em>, pages 182–189, Gyeongju, Republic of Korea. Association for Computational Linguistics.</p> <pre><code class="language-json">@inproceedings{gasco2022socialdisner, title = "The {S}ocial{D}is{NER} shared task on detection of disease mentions in health-relevant content from social media: methods, evaluation, guidelines and corpora", author = "Gasco S{\'a}nchez, Luis and Estrada Zavala, Darryl and Farr{\'e}-Maduell, Eul{\`a}lia and Lima-L{\'o}pez, Salvador and Miranda-Escalada, Antonio and Krallinger, Martin", booktitle = "Proceedings of The Seventh Workshop on Social Media Mining for Health Applications, Workshop {\&} Shared Task", month = oct, year = "2022", address = "Gyeongju, Republic of Korea", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2022.smm4h-1.48", pages = "182--189" }</code></pre> <p> </p> <p><strong>Introduction:</strong><br> The <strong>SocialDisNER corpus</strong> of the SMM4H 2022 – Task 10 task focus on the recognition of disease mentions in tweets written in Spanish after selecting primarily<strong><em> first-hand experience of diseases</em></strong> and other health-relevant content (from patient associations, professional healthcare institutions, and through followers of patient association accounts of a <em>diversity of pathologies</em> including rare diseases, mental health, cancer, etc..).</p> <p><strong>SocialDisNER Gold Standard</strong></p> <p>The Gold Standard corpus was manually annotated by medical experts following the <a href="https://doi.org/10.5281/zenodo.6983041">SMM4H-SocialDisNER guidelines</a>. These guidelines were adapted from previous efforts used to annotate patient clinical records and medical literature. It covers rules for annotating <strong>mentions of diseases</strong> in health-related tweets in Spanish,</p> <p>The training set consists of 5000 tweets written in Spanish and the validation set consists of 2500 tweets written in Spanish. Both sets have been manually annotated by healthcare professionals. The test dataset contains 23430 tweets, although only 2000 will be used to evaluate the systems participating in the task (the rest is background set). We don't plan to publish the test set, but if you want you can test your system from <a href="https://codalab.lisn.upsaclay.fr/competitions/3531">SocialDisNER Codalab</a>.</p> <p><strong>SocialDisNER Large Scale Corpus</strong></p> <p>The large-scale data contains mentions automatically extracted from a set of 85000 tweets. Separate datasets are shown for each entity including diseases, drugs, symptoms, professions, procedures, species, morphology neoplasm, and persons.</p> <p><strong>SocialDisNER co-mention networks</strong></p> <p>We have computed a co-occurrence matrix of the extracted diseases, as well as several co-mention matrices between the disease mentions and the rest of the entities in the large-scale corpora.</p> <p> </p> <p><strong>File structure:</strong></p> <p>The structure of the corpus is: </p> <ul> <li><strong>SocialDisNER_Data:</strong> <ul> <li>training-validation-data folder <ul> <li><strong><em>train-valid-txt-files</em></strong>: folder with training and validation text files. One text file per tweet, the file name corresponds to the tweet id. One sub-directory per corpus split (train and valid). The files named <em>ids_dev_set.txt</em> and<em> ids_train_set.txt </em>contain the list of file identifiers for each of the data splits (validation and train).</li> <li><strong><em>mentions.tsv</em></strong>: This file contains the manually annotated disease mentions. The file has the following fields: <ul> <li><em>tweets_id</em>: This is the id of the tweet, using Twitter API you can query the content of the tweet.</li> <li><em>Begin</em>: This is the position in the tweet where the annotation was found.</li> <li><em>End</em>: This is the position of the last character of the annotation in the tweet.</li> <li><em>Type: </em>This is the type of entity found, in our case "ENFERMEDAD".</li> <li><em>Extraction</em>: This is the literal extraction, in other words, the fragment of text which refers to the annotation. </li> </ul> </li> </ul> </li> <li>test-data folder: <ul> <li><strong>test-data-txt-files</strong>: folder with test text files. One file per tweet, the file name corresponds to the tweet id. The folder contains 23430 tweets to be used as test set of the task. Of them, 2000 will be used to evaluate the participating systems.</li> </ul> </li> </ul> </li> </ul> <p> </p> <ul> <li><strong>SocialDisNER_LargeScale_additionaldata:</strong> <ul> <li>socialdisner_diseases: <ul> <li><strong>tweets_txt:</strong> Folder with large-scale tweet database. One text file per tweet, the file name corresponds to the tweet id.</li> <li><strong>diseases_mentions.tsv</strong>: This file contains the automatically annotated disease mentions from the large-scale SocialDisNER corpus (Silver Standard). The structure is the same than the Golden Standard annotations.</li> </ul> </li> <li>socialdisner_ENTITY: Each folder with this naming convention contains the following data structure. Corpora have been generated with mentions of diseases, drugs, symptoms, professions, procedures, species, morphology neoplasm and persons <ul> <li><strong>tweets_txt:</strong> Folder with large-scale tweet database. One text file per tweet, the file name corresponds to the tweet id.</li> <li><strong>ENTITY_mentions.tsv</strong>: This file contains the automatically annotated mentions of type “ENTITY” from the large-scale SocialDisNER corpus (Silver Standard). The structure is the same than the Golden Standard annotations.</li> </ul> </li> <li>socialdisner_networks: This folder contains tsv files containing the co-mention matrices between the diseases and the rest of the entities of the large-scale socialdisner data. Each file follows the following naming convention: <ul> <li><strong>socialdisner_disease-ENTITY_net.tsv</strong><em>: </em>The tsv file contains a series of columns and rows corresponding to the mentions used for building the matrix. Each column is separated by “;”. The type of each mention is identified by the label in parentheses of each title. The count represents the number of times that mention x and mention y were found in the same tweet of the large-scale dataset.</li> <li><strong>socialdiser_disease_net.tsv</strong>: This tsv file contains the array of socialdisner-disease large-scale corpus co-mentions separated by ";". This file can be loaded into NetworkX to perform disease co-morbidity analysis on the socialdisner-disease large-scale data.</li> </ul> </li> </ul> </li> </ul> <p><em>Note: In previous versions of the dataset the order of the columns in the mentions.tsv file was not in the correct order. From this version onwards the order is correct and adequate to send the predictions of the task.</em></p> <p> </p> <p>For further information, please visit <a href="https://temu.bsc.es/socialdisner/">https://temu.bsc.es/socialdisner/</a></p> <p><strong>Summary statistics:</strong></p> <table> <caption>Manually annotated data</caption> <thead> <tr> <th scope="row"> </th> <th scope="col">Training set</th> <th scope="col">Development set</th> </tr> </thead> <tbody> <tr> <th scope="row"># tweets</th> <td>5000</td> <td>2500</td> </tr> <tr> <th scope="row"># characters</th> <td>1253431</td> <td>516768</td> </tr> <tr> <th scope="row"># tokens</th> <td>211555</td> <td>84478</td> </tr> <tr> <th scope="row">Avg. char / tweet</th> <td>250.69</td> <td>206.71</td> </tr> <tr> <th scope="row">Avg. tok. / tweet</th> <td>42.31</td> <td>33.79</td> </tr> <tr> <th scope="row"># mentions</th> <td>15173</td> <td>4252</td> </tr> <tr> <th scope="row"># unique mentions</th> <td>4407</td> <td>1413</td> </tr> </tbody> </table> <p> </p> <table> <caption>Large-scale annotated data (Silver Standard)</caption> <tbody> <tr> <td> </td> <td><em>Socialdisner-diseases</em></td> <td><em>Socialdisner-pharma</em></td> <td><em>Socialdisner-morphology_neoplasms</em></td> <td><em>Socialdisner-symptoms</em></td> <td><em>Socialdisner-professions</em></td> <td><em>Socialdisner-Procedures</em></td> <td><em>Socialdisnerv-Person</em></td> <td><em>Socialdisner-Species</em></td> </tr> <tr> <td><strong># tweets</strong></td> <td>85077</td> <td>1759</td> <td>8518</td> <td>12624</td> <td>15831</td> <td>11462</td> <td>41033</td> <td>12118</td> </tr> <tr> <td><strong># characters</strong></td> <td>19920670</td> <td>435141</td> <td>2082574</td> <td>3023784</td> <td>4063114</td> <td>2873791</td> <td>10273278</td> <td>2933925</td> </tr> <tr> <td><strong># tokens</strong></td> <td>3236411</td> <td>68269</td> <td>332539</td> <td>521503</td> <td>660071</td> <td>467059</td> <td>1689479</td> <td>486249</td> </tr> <tr> <td><strong>Avg. char / tweet</strong></td> <td>234.15</td> <td>247.38</td> <td>244.49</td> <td>239.53</td> <td>256.66</td> <td>250.72</td> <td>250.37</td> <td>242.11</td> </tr> <tr> <td><strong>Avg. tok. / tweet</strong></td> <td>38.04</td> <td>38.81</td> <td>39.04</td> <td>41.31</td> <td>41.69</td> <td>40.75</td> <td>41.17</td> <td>40.13</td> </tr> <tr> <td><strong># mentions</strong></td> <td>116260</td> <td>1029</td> <td>8943</td> <td>12896</td> <td>18590</td> <td>10080</td> <td>58007</td> <td>14014</td> </tr> <tr> <td><strong># unique mentions</strong></td> <td>16034</td> <td>530</td> <td>541</td> <td>6991</td> <td>3667</td> <td>3841</td> <td>3446</td> <td>1676</td> </tr> </tbody> </table> <p> </p> <p> </p> <p> </p> <p>Do not share the data with other individuals/teams without permission from the task organizer. Tweets IDs are the primary source of information. Tweet texts are provided as support material. By downloading this resource, you agree to the Twitter <a href="https://twitter.com/en/tos">Terms of Service</a>, <a href="https://twitter.com/en/privacy">Privacy Policy</a>, <a href="https://developer.twitter.com/en/developer-terms/agreement">Developer Agreement</a>, and <a href="https://developer.twitter.com/en/developer-terms/policy">Developer Policy</a>.</p> <p> </p> <p> </p>
Research data supporting "Platinum Nanocatalyst Amplification: Redefining the Gold Standard for Lateral Flow Immunoassays with Ultra-Broad Dynamic Range"
<p>Research data supporting the publication: Loynachan C. N., et al., 2017, ACS Nano, DOI: http://dx.doi.org/10.1021/acsnano.7b06229.</p>
SSIX Finance Gold Standards
<p><strong>SSIX Finance Gold Standards </strong></p> <ol> <li>Microblogs - English</li> <li>News Headlines - English</li> </ol> <p>These resources were created in the context of the SSIX project. Only a sample is available here, to download the full files please follow the instruction at <a href="https://ssix-project.eu/knowledgebase/gold-standards/">https://ssix-project.eu/knowledgebase/gold-standards/</a></p>
Themed Evidence: Reading Experiences - Gold Standard
<p>Exploring digital resources in search of pieces of evidence relevant to a certain research theme is a difficult and important task for humanities research. The concept of evidence is a particularly difficult one, as it relates to a fact being reported in a text, which is relevant to a certain subject of enquiry. </p> <p>The UK Reading Experience Database Project (UK RED) is an open-access dataset containing records that document the history of reading in Britain from 1450 to 1945. Evidence of reading presented in UK RED is drawn from published and unpublished sources as diverse as diaries, commonplace books, memoirs, sociological surveys, and criminal court and prison records. See <a href="http://www.open.ac.uk/Arts/reading/UK">http://www.open.ac.uk/Arts/reading/UK</a>. </p> <p>This dataset is a gold standard for supporting the evaluation of competing methods for detecting themed evidence of "reading experience".</p>
Themed Evidence: Listening Experiences - Gold Standard
<p>Exploring digital resources in search of pieces of evidence relevant to a certain research theme is a difficult and important task for humanities research. The concept of evidence is a particularly difficult one, as it relates to a fact being reported in a text, which is relevant to a certain subject of enquiry. </p> <p>The Listening Experience Database Project (LED) - http://www.listeningexperience.org/ - is an initiative funded by the UK's Arts and Humanities Research Council (AHRC) aimed at collecting accounts of people’s private experiences of listening to music.</p> <p>This dataset is a gold standard for supporting the evaluation of competing methods for detecting themed evidence of "listening experience".</p>
All Computer Science Papers @ arXiv.org -- A High-Quality Gold Standard for Citation-based Tasks
<p>We propose a newly-created gold standard <strong>data set for citation-based tasks</strong>. This gold standard is based on <strong>all computer science papers in arXiv.org</strong>.</p> <p><strong>Abstract</strong>. Analyzing and recommending citations with their specific citation contexts have recently received much attention due to the growing number of available publications. Although data sets such as CiteSeerX have been created for evaluating approaches for such tasks, those data sets exhibit striking defects. This is understandable if one considers that both information extraction and entity linking as well as entity resolution need to be performed. In this paper, we propose a new evaluation data set for citation-dependent tasks based on arXiv.org publications. Our data set is characterized by the fact that it exhibits almost zero noise in the extracted content and that all citations are linked to their correct publications. Besides the pure content, available on a sentence-basis, cited publications are annotated directly in the text via global identifiers. As far as possible, referenced publications are further linked to DBLP. Our data set consists of over 15M sentences and is freely available for research purposes. It can be used for training and testing citation-based tasks, such as recommending citations, determining the functions or importance of citations, and summarizing documents based on their citations.</p> <p> </p> <p>More information can be found in our <strong>publication "<a href="http://www.lrec-conf.org/proceedings/lrec2018/pdf/283.pdf">A High-Quality Gold Standard for Citation-based Tasks</a>" (LREC'18)</strong>.</p> <p>You can cite the data set as follows:</p> <pre><code>@inproceedings{DBLP:conf/lrec/0001TJ18, author = {Michael F{\"{a}}rber and Alexander Thiemann and Adam Jatowt}, title = "{A High-Quality Gold Standard for Citation-based Tasks}", booktitle = "{Proceedings of the Eleventh International Conference on Language Resources and Evaluation}", series = "{LREC'18}", location = "{Miyazaki, Japan}", year = {2018}, url = {http://www.lrec-conf.org/proceedings/lrec2018/summaries/283.html} } </code></pre> <p> </p>
LAGOS-AND: A Large Gold Standard Dataset for MAG/OpenAlex Author Name Disambiguation
<p>We present a large gold standard dataset for author name disambiguation (AND) research (LAGOS-AND), which contains two sub-datasets, LAGOS-AND-BLOCK and LAGOS-AND-PAIRWISE. The datasets were automatically built by using the two authoritative sources, ORCID and DOI, based on the ORCID open database and an open literature database (MAG or OpenAlex).</p> <p>The currently available versions of the LAGOS-AND datasets are:</p> <ol> <li><strong>Version 1.0 (MAG+ORCID)</strong>: This is the initial version of the LAGOS-AND dataset, the evaluation results and quality control measures of this version dataset can be found in this paper <a href="https://arxiv.org/abs/2104.01821">https://arxiv.org/abs/2104.01821</a>.</li> <li><strong>Version 2.0 (OpenAlex+ORCID)</strong>: This version builds on OpenAlex instead of MAG because MAG was discontinued on 31 December 2021, and OpenAlex not only positions itself as a drop-in replacement for MAG but also keeps evolving by aggregating academic resources from other repositories. In addition, the pairwise-based sub-dataset (LAGOS-AND-PAIRWISE v2.0) improves the accuracy of labeled authorship (class label) as compared to LAGOS-AND-PAIRWISE v1.0.</li> </ol> <p>Note that there are other versions of the dataset, which we call "pre-release". We recommend users to use the normal version of the dataset. These pre-release versions were originally intended to be released as the normal versions. However, during the preparation of the research paper, we found an issue with the dataset and the reviewers also made some reasonable requests for the dataset. This led us to update the dataset. Unfortunately, the Zenodo platform does not allow updates to the same version of dataset, so we had to create new versions. The created pre-release datasets are as follows:</p> <ol> <li><strong>Version 2.0-alpha</strong>: For few samples in LAGOS-AND-PAIRWISE, the class labels are incorrect. We improve the accuracy of the class label in the normal Version 2.0 dataset by using a better random sampling approach.</li> <li><strong>Version 1.0-beta</strong>: We created this version because a sub-dataset of this version LAGOS-AND-PAIRWISE contains only ~500K author pairs, while it should contain ~1M author pairs, as described in our paper. We fixed the problem in the Version 1.0 dataset.</li> <li><strong>Version 1.0-alpha</strong>: The earliest dataset uploaded to Zenodo, corresponding to the original dataset before addressing the reviewers' comments and suggestions. In contrast, the dataset in Version 1.0 is the dataset after the reviewers' comments and suggestions have been addressed.</li> </ol>
Gollum: A Gold Standard for Large Scale\\Multi Source Knowledge Graph Matching
<p>The set of Knowledge Graphs (KGs) generated with automatic and manual approaches is constantly growing.<br> For an integrated view and usage, an alignment between these KGs is necessary on the schema as well as instance level.<br> There are already approaches which try to tackle this multi source knowledge graph matching problem,<br> but large gold standards are missing to evaluate their effectiveness and scalability.<br> In particular, most existing gold standards are fairly small and can be solved by matchers which match exactly two KGs (1:1), which are the majority of existing matching systems.<br> <br> We close this gap by presenting Gollum -- a gold standard for large-scale multi source knowledge graph matching with over 275,000 correspondences between 4,149 different KGs.<br> They originate from knowledge graphs derived by applying the DBpedia extraction framework to a large wiki farm.<br> <br> Three variations of the gold standard are made available:<br> (1) a version with all correspondences for evaluating unsupervised matching approaches, and two versions for evaluating supervised matching: (2) one where each KG is contained both in the train and test set, and (3) one where each KG is exclusively contained in the train or the test set.</p> <p><br> We plan to extend our KG track at the Ontology Alignment Evaluation Initiative (OAEI) to allow for matching systems <br> which are specifically designed to solve the multi KG matching problem.<br> As a first step towards this direction, we evaluate multi source matching approaches which reuse two-KG (1:1) matchers from the past OAEI.</p> <p> </p> <p>Due to the size of the KG files, they are hosted at the institute:</p> <p><a href="http://data.dws.informatik.uni-mannheim.de/dbkwik/gollum/40K.tar">http://data.dws.informatik.uni-mannheim.de/dbkwik/gollum/40K.tar</a> (50,3 GB)<br> <a href="http://data.dws.informatik.uni-mannheim.de/dbkwik/gollum/all.tar">http://data.dws.informatik.uni-mannheim.de/dbkwik/gollum/all.tar</a> (74,7 GB)<br> <a href="http://data.dws.informatik.uni-mannheim.de/dbkwik/gollum/gold.tar">http://data.dws.informatik.uni-mannheim.de/dbkwik/gollum/gold.tar</a> (25,3 GB)</p>
MEDDOPLACE Corpus: Gold Standard annotations for Medical Documents Place-related Content Extraction
<p><strong>MEDDOPLACE</strong> stands for MEDical DOcument PLAce-related Content Extraction. It is a shared task and set of resources focused on the detection, normalization (entity linking/toponym resolution) and classification of different kinds of places, as well as related types of information such as clinical departments, nationalities or patient movements, in medical documents in Spanish.</p> <p>This repository includes the corpus' <strong>train and test sets</strong> in multiple formats, as well as the <strong>SNOMED gazetteer</strong>, <strong>cross-mapping</strong> between SNOMED and MeSH and the <strong>multilingual silver standard in 8 languages </strong>(Catalan, English, French, Italian, Dutch, Portuguese, Romanian and Swedish). For more information, please check the attached README file.</p> <p>MEDDOPLACE was developed by the Barcelona Supercomputing Center's NLP for Biomedical Information Analysis and used as part of IberLEF 2023. For more information on the corpus, annotation scheme and task in general, please visit: <a href="https://temu.bsc.es/meddoplace">https://temu.bsc.es/meddoplace</a>.</p> <p> </p> <p><strong>Please cite if you use this resource:</strong></p> <p>Salvador Lima-López, Eulàlia Farré-Maduell, Antonio Miranda-Escalada, Vicent Brivá-Iglesias and Martin Krallinger. NLP applied to occupational health: MEDDOPROF shared task at IberLEF 2021 on automatic recognition, classification and normalization of professions and occupations from medical texts. In Procesamiento del Lenguaje Natural, 67. 2021.</p> <pre><code>@article{meddoplace, title={MEDDOPLACE Shared Task overview: recognition, normalization and classification of locations and patient movement in clinical texts}, author={Lima-López, Salvador and Farré-Maduell, Eulàlia and Brivá-Iglesias, Vicent and Gasco-Sanchez, Luis and Krallinger, Martin}, journal = {Procesamiento del Lenguaje Natural}, volume = {71}, year={2023}, issn = {1135-5948},<br>DOI = {10.26342/2023-71-23}, url = {http://journal.sepln.org/sepln/ojs/ojs/index.php/pln/article/view/6561/3961}, pages = {301--311} }</code></pre> <p><strong>Related Links:</strong></p> <p>- MEDDOPLACE website: <a href="https://temu.bsc.es/meddoplace">https://temu.bsc.es/meddoplace</a></p> <p>- MEDDOPLACE overview paper: <a href="http://journal.sepln.org/sepln/ojs/ojs/index.php/pln/article/view/6561">http://journal.sepln.org/sepln/ojs/ojs/index.php/pln/article/view/6561</a></p> <p>- Annotation Guidelines (Spanish): <a href="https://doi.org/10.5281/zenodo.7775234">https://doi.org/10.5281/zenodo.7775234</a></p> <p>- Annotation Guidelines (English): <a href="https://doi.org/10.5281/zenodo.7928145">https://doi.org/10.5281/zenodo.7928145</a></p> <p><strong>License</strong></p> <p>This work is licensed under a <a href="http://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution 4.0 International License</a>.</p> <p><strong>Contact</strong></p> <p>If you have any questions or suggestions, please contact us at:</p> <p>- Salvador Lima-López (<salvador [dot] limalopez [at] gmail [dot] com>)<br>- Martin Krallinger (<krallinger [dot] martin [at] gmail [dot] com>)</p>
XEN-45 Gel Stent Versus Trabeculectomy in Glaucoma: Gold-Standard Pathway Study (GPS)
ClinicalTrials.gov study NCT03654885. IPD Sharing: YES. Countries: 1. Publications: 1.
CrowdTruth/Crowdsourcing-NamedEntities-GoldStandard: Data release for crowdsourcing named entity gold standards
<p>This repository contains the experimental results of identifying and typing named entities in English Wikipedia sentences by using a hybrid Multi-NER - crowd-enhanced approach.</p>
BiodivNERE: Gold Standard Corpora for Named Entity Recognition and Relation Extraction in Biodiversity Domain
<p>BiodivNER+RE are two gold-standard manually annotated corpora that are meant to be for Named Entity Recognition (NER) and Relation Extraction (RE) tasks based on abstracts and metadata files from the biodiversity domain</p> <p>Such corpora are designed for machine learning techniques, for example, NER as TokenClassification technique, and RE as SequenceClassification technique.</p>
MEDDOPROF corpus: complete gold standard annotations for occupation detection in medical documents in Spanish
<p><strong>UPDATE 27/09/2022: </strong>A complete normalization of all mentions in the corpus to SNOMED CT has been added to the 'meddoprof-norm.tsv' file.</p> <p><strong>Description</strong></p> <p>This repository contains the complete MEDDOPROF Gold Standard, a collection of 1,844 clinical cases in Spanish with annotations for occupations, working statuses and activities. MEDDOPROF is a Shared Task celebrated in 2021 that explores the application of natural language processing to occupational health. If you'd like to learn more, please visit: <a href="https://temu.bsc.es/meddoprof">https://temu.bsc.es/meddoprof</a>.</p> <p><strong>Folder and File Structure</strong></p> <p>The corpus' files are presented in the format used by the annotation tool brat. That is, for each clinical case there is a .txt file with the text and a .ann file with its corresponding annotations.</p> <p><em>- meddoprof-ner/</em></p> <p>Clinical cases annotated with these labels: PROFESION (PROFESSION), SITUACION_LABORAL (WORKING_STATUS) or ACTIVIDAD (ACTIVIDAD).</p> <p><em>- meddoprof-class/</em></p> <p>Clinical cases with the same annotations as 'meddoprof-ner' but with these labels instead: PACIENTE (patient), FAMILIAR (family member), SANITARIO (health professional) or OTRO (other).</p> <p><em>- ner_class_joint/</em></p> <p>Clinical cases with both levels of annotation (ner and class) joint (that is, a mention classified as as PROFESOR in meddoprof-ner and as PACIENTE in meddoprof-class would be PROFESION-PACIENTE here).</p> <p><em>- meddoprof-norm.tsv</em></p> <p>Tab-separated file (.tsv) with the mapping of each mention in the corpus to ESCO and SNOMED CT. The file has five columns: filename, mention text, span, ESCO code and SNOMED code.</p> <p>Additionally, two files with the filenames of the train and test partitions are included.</p> <p> </p> <p><strong>Please cite if you use this resource:</strong></p> <p>Salvador Lima-López, Eulàlia Farré-Maduell, Antonio Miranda-Escalada, Vicent Brivá-Iglesias and Martin Krallinger. NLP applied to occupational health: MEDDOPROF shared task at IberLEF 2021 on automatic recognition, classification and normalization of professions and occupations from medical texts. In Procesamiento del Lenguaje Natural, 67. 2021.</p> <pre><code>@article{meddoprof, title={NLP applied to occupational health: MEDDOPROF shared task at IberLEF 2021 on automatic recognition, classification and normalization of professions and occupations from medical texts}, author={Lima-López, Salvador and Farré-Maduell, Eulàlia and Miranda-Escalada, Antonio and Brivá-Iglesias, Vicent and Krallinger, Martin}, journal = {Procesamiento del Lenguaje Natural}, volume = {67}, year={2021}, issn = {1989-7553}, url = {http://journal.sepln.org/sepln/ojs/ojs/index.php/pln/article/view/6393}, pages = {243--256} }</code></pre> <p><strong>Related Resources:</strong></p> <p>- <a href="http://temu.bsc.es/meddoprof">Web</a></p> <p>- <a href="https://doi.org/10.5281/zenodo.4694768">Training Data</a></p> <p>- <a href="https://doi.org/10.5281/zenodo.4889776">Test set</a></p> <p>- <a href="https://zenodo.org/record/4722741">Codes Reference List</a> (for MEDDOPROF-NORM)</p> <p>- <a href="https://zenodo.org/record/4720833">Annotation Guidelines</a></p> <p>- <a href="https://doi.org/10.5281/zenodo.4524658">Occupations Gazetteer</a></p> <p> </p> <blockquote> <p>MEDDOPROF is part of the IberLEF 2021 workshop, which is co-located with the SEPLN 2021 conference. For further information, please visit <a href="https://temu.bsc.es/meddoprof/">https://temu.bsc.es/meddoprof/</a> or email us at encargo-pln-life@bsc.es</p> <p>MEDDOPROF is promoted by the Plan de Impulso de las Tecnologías del Lenguaje de la Agenda Digital (Plan TL) and the Spanish government's 2020 Proyectos de I+D+i RTI Tipo A (AI4PROFHEALTH - DESCIFRANDO EL PAPEL DE LAS PROFESIONES EN LA SALUD DE LOS PACIENTES A TRAVES DE LA MINERIA DE TEXTOS (PID2020-119266RA-I00)).</p> </blockquote>
Manual Dependency Annotation of Three German Text Extracts from the Project hermA (Gold Standard Data)
<p>This dataset was created in the digital humanities project hermA (www.herma.uni-hamburg.de) and comprises annotated extracts of the following three texts:</p> <ul> <li>Modern literature (Lit2009): novel <em>Corpus Delicti: Ein Prozess </em>by German author Juli Zeh, published in Frankfurt/Main in 2009.</li> <li>Non-contemporary literature (Lit1850):<em> Eine Frauenfahrt um die Welt </em>('A woman’s journey around the world') by Austrian author Ida Pfeiffer (1850). Full text available at Deutsches Textarchiv: http://www.deutschestextarchiv.de/pfeiffer_frauenfahrt01_1850/6.</li> <li>Modern academic writing (Aca2009): <em>Stand, Möglichkeiten und Grenzen der Telemedizin in Deutschland</em> ('Telemedicine in Germany: status, chances and limits') by Rüdiger Klar and Ernst Pelikan, published in Bundesgesundheitsblatt ('Federal Health Gazette') in 2009. DOI 10.1007/s00103-009-0787-7.</li> </ul> <p>The texts are annotated for part-of-speech and dependency syntax and are made available in CoNLL file format. We describe the annotation process and report inter-annotator agreements in:</p> <p>Adelmann, Benedikt, Melanie Andresen, Wolfgang Menzel & Heike Zinsmeister. 2018. Evaluation of Out-Of Domain Dependency Parsing for its Application in a Digital Humanities Project. <em>Proceedings of the 14th Conference on Natural Language Processing (KONVENS 2018)</em>. Vienna, Austria.</p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.