Find research datasets worth reusing

opencc-by-4.0Dec 2023View details →

Named Entity Recognition Dataset for Dutch Biographical Texts

A dataset for Named Entity Recognition for Dutch biographies. The original data is available in the Biographical portal of the Netherlands (http://www.biografischportaal.nl/). The annotations are for 6 types of entities: PERSON, LOCATION, ORGANIZATION, DATE, ARTWORK, MISC. Additionally, the CoNLL formatted files were manually checked for tokenization and sentence splitting.

opencc-by-4.0Dec 2021View details →

Event Registry events associated to Wikidata entities

This is a set of labels that associate the cluster and events ids of the Event Registry dataset with entities in the Wikidata KG. This can be used in order to analyse the event description completeness in news articles, evaluate a query based document retrieval on Event Registry, etc.

Multi-head CRF classifier for biomedical multi-class Named Entity Recognition on Spanish clinical notes

This contains the merged dataset as described in the work "Multi-head CRF classifier for biomedical multi-class Named Entity Recognition on Spanish clinical notes". This dataset consists of 4 seperate datasets: <ul> <li><a href="../records/8224056" target="_blank" rel="noopener">MedProcNer</a></li> <li><a href="../records/7614764" target="_blank" rel="noopener">DisTEMIST</a></li> <li><a href="../records/4270158" target="_blank" rel="noopener">PharmaCoNER</a></li> <li><a href="../records/10635215" target="_blank" rel="noopener">SympTEMIST</a></li> </ul> The dataset contains two tasks: Task 1: This task is related to multi-class Named Entity Recognition. This dataset contains 5 possible classes: SYMPTOM, PROCEDURE, DISEASE, CHEMICAL and PROTEIN. Task 2: This task is related to Named Entity Linking, where each code corresponds to a code within the SNOMED-CT corpus. The exact corpus used can be obtained <a href="https://download.nlm.nih.gov/umls/kss/IHTSDO20190131/SnomedCT_SpanishRelease-es_PRODUCTION_20190430T120000Z.zip" target="_blank" rel="noopener">here</a>. Further for the MedProcNER, SympTEMIST and DisTEMIST datasets, a gazetteer is provided in the original datasets.  For more information on the construction of the dataset, aswell as dataloaders, we refer you to our <a href="https://github.com/ieeta-pt/Multi-Head-CRF" target="_blank" rel="noopener">GitHub repository</a>. Further this also contains the embeddings from the <a href="https://huggingface.co/cambridgeltl/SapBERT-UMLS-2020AB-all-lang-from-XLMR-large" target="_blank" rel="noopener">SapBERT</a> model. Please, cite: <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <blockquote> <div>@article{jonker2024a, title = {Multi-head {{CRF}} classifier for biomedical multi-class named entity recognition on {{Spanish}} clinical notes}, author = {Jonker, Richard A. A. and Almeida, Tiago and Antunes, Rui and Almeida, Jo{\~a}o R. and Matos, S{\'e}rgio}, year = {2024}, journal = {Database}, publisher = {Oxford University Press} }</div> </blockquote> <div>Jonker, R. A. A., Almeida, T., Antunes, R., Almeida, J. R., & Matos, S. (2024). Multi-head CRF classifier for biomedical multi-class named entity recognition on Spanish clinical notes. (Submitted.) </div> <div> </div> <div> License This work is licensed under a <a href="http://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution 4.0 International License</a>. </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div>

opencc-zeroFeb 2019View details →

Types, open citations, closed citations, publishers, and participation reports of Crossref entities

This publication contains several datasets that have been used in the paper "Crowdsourcing open citations with CROCI – An analysis of the current status of open citations, and a proposal" submitted to the <a href="https://www.issi2019.org/">17th International Conference on Scientometrics and Bibliometrics (ISSI 2019)</a>, available at <a href="https://opencitations.wordpress.com/2019/02/07/crowdsourcing-open-citations-with-croci/">https://opencitations.wordpress.com/2019/02/07/crowdsourcing-open-citations-with-croci/</a>. Additional information about the analyses described in the paper, including the code and the data we have used to compute all the figures, is available as a Jupyter notebook at <a href="https://github.com/sosgang/pushing-open-citations-issi2019/blob/master/script/croci_nb.ipynb">https://github.com/sosgang/pushing-open-citations-issi2019/blob/master/script/croci_nb.ipynb</a>. The datasets contain the following information. non_open.zip: it is a zipped (~5 GB unzipped) CSV file containing the numbers of open citations and closed citations received by the entities in the Crossref dump used in our computation, dated October 2018. All the entity types retrieved from Crossref were aligned to one of following five categories: journal, book, proceedings, dataset, other. The open CC0 citation data we used came from the CSV dump of <a href="https://doi.org/10.6084/m9.figshare.6741422.v3">most recent release of COCI dated 12 November 2018</a>. The number of closed citations was calculated by subtracting the number of open citations to each entity available within COCI from the value “is-referenced-by-count” available in the Crossref metadata for that particular cited entity, which reports all the DOI-to-DOI citation links that point to the cited entity from within the whole Crossref database (including those present in the Crossref ‘closed’ dataset). The columns of the CSV file are the following ones: <ul> <li>doi: the DOI of the publication in Crossref;</li> <li>type: the type of the publication as indicated in Crossref;</li> <li>cited_by: the number of open citations received by the publication according to COCI;</li> <li>non_open: the number of closed citations received by the publication according to Crossref + COCI.</li> </ul> croci_types.csv: it is a CSV file that contains the numbers of open citations and closed citations received by the entities in the Crossref dump used in our computation, as collected in the previous CSV file, alligned in five classes depening on the entity types retrieved from Crossref: journal (Crossref types: journal-article, journal-issue, journal-volume, journal), book (Crossref types: book, book-chapter, book-section, monograph, book track, book-part, book-set, reference-book, dissertation, book series, edited book), proceedings (Crossref types: proceedings-article, proceedings, proceedings-series), dataset (Crossref types: dataset), other (Crossref types: other, report, peer review, reference-entry, component, report-series, standard, posted-content, standard-series). The columns of the CSV file are the following ones: <ul> <li>type: the type publication between "journal", "book", "proceedings", "dataset", "other";</li> <li>label: the label assigned to the type for visualisation purposes;</li> <li>coci_open_cit: the number of open citations received by the publication type according to COCI;</li> <li>crossref_close_cit: the number of closed citations received by the publication according to Crossref + COCI.</li> </ul> publishers_cits.csv: it is a CSV file that contains the top twenty publishers that received the greatest number of open citations. The columns of the CSV file are the following ones: <ul> <li>publisher: the name of the publisher;</li> <li>doi_prefix: the list of DOI prefixes used assigned by the publisher;</li> <li>coci_open_cit: the number of open citations received by the publications of the publisher according to COCI;</li> <li>crossref_close_cit: the number of closed citations received by the publications of the publishers according to Crossref + COCI;</li> <li>total_cit: the total number of citations received by the publications of the publisher (= coci_open_cit + crossref_close_cit).</li> </ul> 20publishers_cr.csv: it is a CSV file that contains the numbers of the contributions to open citations made by the twenty publishers introduced in the previous CSV file as of 24 January 2018, according to the data available through the Crossref API. The counts listed in this file refers to the number of publications for which each publisher has submitted metadata to Crossref that include the publication’s reference list. The categories 'closed', 'limited' and 'open' refer to publications for which the reference lists are not visible to anyone outside the Crossref Cited-by membership, are visible only to them and to Crossref Metadata Plus members, or are visible to all, respectively. In addition, the file also record the total number of publications for which the publisher has submitted metadata to Crossref, whether or not those metadata include the reference lists of those publications. The columns of the CSV file are the following ones: <ul> <li>publisher: the name of the publisher;</li> <li>open: the number of publications in Crossref with an 'open' visibility for their reference lists;</li> <li>limited: the number of publications in Crossref with an 'limited' visibility for their reference lists;</li> <li>closed: the number of publications in Crossref with an 'closed' visibility for their reference lists;</li> <li>overall_deposited: the overall number of publications for which the publisher has submitted metadata to Crossref.</li> </ul>

Named Entity Corpus for Occupational Substance Exposure Assessment

This is a corpus consisting of selected sections (i.e., Abstract, Methods and Results) of scientific research articles concerning occupational exposures to two different types of substance, i.e.,  diesel exhaust (51 articles) and respirable crystalline silica (RCS)  (50 articles).  The article sections have been annotated by experts in the field with 6 categories of named entities (NEs) relevant to the assessment of occupational substance exposures, particularly in the context of Job Exposure Matrices (JEMs). The corpus is available in two different formats, <a href="https://brat.nlplab.org/standoff.html">brat standoff format</a> and JSON.  The corpus and associated NER models are described in more detail in the following atricle, which should be cited if you use the corpus:  Thompson, P., Ananiadou, S., Basinas I., Brinchmann, B. C., Cramer, C., Galea, K. S., Ge, C., Georgiadis, P., Kirkeleit, J., Kuijpers, E., Nguyen, N., Nuñez, R., Schlünssen, V., Stokholm, Z. A., Taher, E. A., Tinnerberg, H., Van Tongeren, M. and Xie, Q. (2024).<a href="https://doi.org/10.1371/journal.pone.0307844"> </a><a href="https://doi.org/10.1371/journal.pone.0307844">Supporting the working life exposome: annotating occupational exposure for enhanced literature search</a>. PLoS ONE 19(8): e0307844

opencc-by-4.0Mar 2023View details →

French Entity-Linking dataset between annotated tweets collected during major crises in France and French Wikipedia corpus

Most of the available datasets are not particularly adapted to our target application: geolocate natural disasters from social networks. First, social media posts are largely underrepresented in these datasets, and the only Twitter dataset lacks Entity-Linking annotations. Second, none of the datasets focuses on a crisis or natural disaster event. To mitigate these issues, we extracted a collection of French tweets written during earthquakes and major floods that have occurred in France in recent years. We set up Label-Studio in order to annotate these tweets. A total of 4617 tweets were annotated, including 1678 tweets posted during earthquakes and 2939 during floods. For each annotated tweet, mentions were annotated using the set of labels described earlier in the paper as well as, when possible, the target Wikipedia title. Named “RéSoCIO” in reference to the research project in which it was carried out, the dataset resulting from this work contains a total of 12 828 annotated mentions and 1 513 distinct Wikipedia entities. 85% of mentions were associated with a Wikipedia page and 94 % if we ignore the RISKNAT and DAMAGES labels, which are often difficult to map to an existing entity. <table> <tbody> <tr> <td>Labels</td> <td>#Mentions</td> <td>#Linked</td> <td>#Entities</td> </tr> <tr> <td>PERSON</td> <td>315</td> <td>263</td> <td>136</td> </tr> <tr> <td>ORG</td> <td>863</td> <td>790</td> <td>281</td> </tr> <tr> <td>GEOLOC</td> <td>4375</td> <td>4234</td> <td>701</td> </tr> <tr> <td>TRANSPORT</td> <td>250</td> <td>203</td> <td>101</td> </tr> <tr> <td>EVENT</td> <td>35</td> <td>21</td> <td>16</td> </tr> <tr> <td>FACILITY</td> <td>129</td> <td>94</td> <td>49</td> </tr> <tr> <td>RISKNAT</td> <td>5502</td> <td>4994</td> <td>128</td> </tr> <tr> <td>DAMAGES</td> <td>1136</td> <td>121</td> <td>56</td> </tr> <tr> <td>OTHER</td> <td>223</td> <td>200</td> <td>46</td> </tr> <tr> <td>Total</td> <td>12828</td> <td>1322</td> <td>1513</td> </tr> </tbody> </table> Overview of the mentions annotated in the Twitter dataset. #Mentions shows the total number of mentions per label, #Linked the number of mentions linked to an entity and #Entities the number of distinct entities per label present in the dataset. <table> <tbody> <tr> <td>Labels</td> <td>#Mentions</td> <td>#Linked</td> <td>#Entities</td> </tr> <tr> <td>PERSON</td> <td>1100102</td> <td>1098406</td> <td>557697</td> </tr> <tr> <td>ORG</td> <td>750925</td> <td>749504</td> <td>130394</td> </tr> <tr> <td>GEOLOC</td> <td>2729702</td> <td>2728296</td> <td>215924</td> </tr> <tr> <td>TRANSPORT</td> <td>161539</td> <td>160487</td> <td>53405</td> </tr> <tr> <td>EVENT</td> <td>798433</td> <td>798251</td> <td>86471</td> </tr> <tr> <td>FACILITY</td> <td>258835</td> <td>258513</td> <td>109867</td> </tr> <tr> <td>RISKNAT</td> <td>5502</td> <td>4994</td> <td>127</td> </tr> <tr> <td>DAMAGES</td> <td>1136</td> <td>121</td> <td>56</td> </tr> <tr> <td>OTHER</td> <td>4340621</td> <td>4339658</td> <td>682458</td> </tr> <tr> <td>Total</td> <td>10146795</td> <td>10138230</td> <td>1836399</td> </tr> </tbody> </table> Overview of the mentions annotated in the full dataset. #Mentions shows the total number of mentions per label, #Linked the number of mentions linked to an entity and #Entities the number of distinct entities per label present in the dataset.

opencc-by-4.0Mar 2024View details →

PPORTAL_ner: An Annotated Corpus of Portuguese Literary Entities

<h2><a href="https://marianaossilva.github.io/pportal_ner/" target="_blank" rel="noopener">PPORTAL_ner</a></h2> <h3>An Annotated Dataset of Portuguese Literary Entities</h3> The corpus is tailored to Brazilian and Portuguese literary texts, containing annotations for five entity categories, including PER, LOC, GPE, ORG, and DATE. Within a diverse collection of 25 literary works, it offers a total of 125,059 tokens and 5,266 annotated entities. This dataset contributes to the development of potentially more accurate and context-aware NER models, as well as to encourage further exploration within Portuguese literature. <h2>Corpus Statistics</h2> Our corpus is sourced from <a href="https://doi.org/10.5281/zenodo.5178063">PPORTAL</a>, an extensive repository of metadata containing over 80,000 public domain literary works in the Portuguese language, predominantly derived from Brazil and Portugal. <a href="https://doi.org/10.5281/zenodo.5178063">PPORTAL</a> aggregates data from three digital libraries: <a href="https://www.dominiopublico.gov.br/">Domínio Público</a>, <a href="https://projectoadamastor.org/">Projecto Adamastor</a>, and <a href="https://www.literaturabrasileira.ufsc.br/">Biblioteca Digital de Literatura dos Países Lusófonos (BLPL)</a>. To simplify referencing, this new dataset is called PPORTAL_ner. PPORTAL_ner selection process contains a diverse range of 25 individual literary works, spanning different authors and literary styles. All of these texts were published prior to 1953, adhering to the current criteria for public domain status in Brazil, with the majority falling within the timeframe spanning from 1554 to 1938.

opencc-by-nc-sa-4.0Oct 2024View details →

ZibaldonED: Silver annotations for Entity Disambiguation from Digitalzibaldone

The Zibaldoned dataset provides silver annotations for entity disambiguation, extracted from DigitalZibaldone, the HTML digital edition of Giacomo Leopardi’s Zibaldone di pensieri, curated by Silvia Stoyanova and Ben Johnston [1]. The dataset was collected through a simple web scraping algorithm and includes 2,941 references to people, places, and bibliographical sources across 957 paragraphs. These annotations offer a valuable resource for tasks such as Named Entity Recognition (NER) and Entity Linking (EL) in the context of literary and linguistic research. The dataset can assist in training AI models to automatically recognize and disambiguate entities in literary works. The GitHub repository containing the code used to extract the dataset and train NER and EL models is available here: <a href="https://github.com/sntcristian/zibaldoned" target="_blank" rel="noopener">GitHub - zibaldoned</a>. [1] Silvia Stoyanova and Ben Johnston (Eds.), Giacomo Leopardi's Zibaldone di pensieri: a digital research platform.

opencc-by-4.0Apr 2022View details →

MediCause Dataset of Causal Sentences with Annotated Entities

The MediCause dataset contains 1202 causal sentences from medical publications where the entities involved in the causal relations have been annotated according to the MediCause ontological model for causal relations. The entities are annotated according to the Inside-Outside-Beginning (IOB) format. The labels used for the annotation are B-C (Cause), B-VC (Causal Variable), B-CS (Beginning Causal Specifier), I-CS (Inside Causal Specifier), B-CON (Beginning Connective), I-CON (Inside Connective), B-EF (Effect), B-VE (Effect Variable), B-ES (Beginning Effect Specifier), I-ES (nside Effect Specifier), O (Outside).

opencc-byApr 2022View details →

DISTANT-CTO: A Zero Cost, Distantly Supervised Approach to Improve Low-Resource Entity Extraction Using Clinical Trials Literature

Datasets <ol> <li>DISTANT-CTO is a weakly-labelled dataset of 'Intervention' and 'Comparator' entity annotated sentences. The dataset was obtained using candidate generation the approach described in "DISTANT-CTO: A Zero Cost, Distantly Supervised Approach to Improve Low Resource Entity Extraction Using Clinical Trials Literature". <ol> <li>distantcto_high_conf.txt    - ds conf 1.0 (full dataset)</li> <li>extraction1_pos_posnegtrail_conf09.txt - ds conf 0.9 (partial dataset)</li> </ol> </li> <li>The physio test set is a dataset comprising 153 PICO annotated randomized controlled trial abstracts from Physiotherapy and Rehabilitation. This dataset was used as an additional benchmark to evaluate the generalization power of the weakly annotated dataset and NER model for this sub-domain.</li> </ol>   Utility The dataset could be used as an input for training 'Intervention' named-entity recognition (NER) models.   Availability This directory includes extraction1_pos_posnegtrail_conf09.txt - This text data file contains all the weak annotations (source intervention terms mapped onto target sentences) from clinicaltrials.org (CTO) with a confidence score of 0.9 and above. The directory also includes ‘physio_sent_annot2POS_posnegtrail.txt’ – This data file contains manually annotated (Intervention entity) data from the physiotherapy and rehabilitation domain. It follows a roughly similar structure as described in the ‘Description for long targets’ section. (‘Participant’ and ‘Outcome’ annotations are removed from this file)

Context-based Entity Recommendation on Real-Life Knowledge Work in Context (RLKWiC dataset)

<h2><a href="../records/11059573">RLKWiC</a> Add-on: Benchmarking Dataset for Entity Recommendation</h2> This benchmark, built on top of the Real-Life Knowledge Work in Context (<a href="../records/11059573">RLKWiC</a>) dataset, is designed to evaluate context-based entity recommendation by simulating a scenario where participants receive entities extracted from their activities across their defined contexts.  In total, 1850 entity recommendations were generated across 56 contexts. After deduplication, these entities were presented to participants for explicit relevance assessment on a 3-point scale: <ul> <li>0 [Irrelevant]: Signifying a lack of relevance between the recommended entity and the context.</li> <li>1 [Relevant] Denoting a connection between the entity and the context, although it may not fully represent it.</li> <li>2 [Representative]: The entity closely aligns with the context, indicating a high level of relevance where the context can be inferred to be about this entity.</li> </ul> Participants could also suggest additional relevant entities. The resulting dataset comprises 1067 entities with explicit relevance scores, offering a resource for benchmarking entity recommendation in real-life knowledge work. <h3>Paper: <a href="https://dl.acm.org/doi/10.1145/3640457.3688068" target="_blank" rel="noopener">Context-based Entity Recommendation for Knowledge Workers: Establishing a Benchmark on Real-life Data</a></h3>

opencc-by-4.0Jun 2024View details →

Web-based Editor for Entity-Relationship Modeling with SQL Transformation Algorithm

With the daily growth of data in all kinds of sectors, such as Information Technologies (IT), healthcare, education, commerce or telecommunication, it becomes important to use a high-performance system to manage all this data in the best possible way. Indeed, in the absence of good data management, it can be difficult for these sectors to prevent data loss, ensure safe maintenance or guarantee data security. For this reason, an effective data management system is the Entity-Relationship model. Indeed, thanks to this model, all kinds of sectors have the possibility of designing and organizing their data in relational databases, thus improving data security and better internal communication. Furthermore, it is interesting to modernize the classic approach of the Entity-Relationship model and its visual representations of data in the present day. The technologies of Augmented and Virtual Reality respond precisely to this challenge of innovation in the Entity-Relationship model. Therefore, the first objective of this thesis is to implement the Entity-Relationship model in a Meta-Modeling Platform for Augmented and Virtual Reality. The second objective is to implement an algorithm that transforms an Entity-Relationship model into SQL statements to create database tables. The Entity-Relationship model is used to design the logic of a database, and the implementation of the algorithm is used to create the database.

FloraNER: a Named Entity Recognition Dataset for Botanical French Text

FloraNER is a Named-Entity Recognition (NER) dataset for botanical french literature. The dataset covers plant species names and plant morphological terms for both plant organs/characteristics and their descriptors. the descriptors are annotated both in a coarse-grained manner as the named entity type "DESCRIPTOR" and in a fine-grained manner, categorized into the following named-entity types: Form, Measure, Surface, Color, Position, Disposition, Structure, and Development. FloraNER is distantly annotated using a specialized botanical corpus. Consequently, it's important to note that not all named entities within the text are captured and annotated for the coarse-grained and fine-grained datasets.

opencc-by-4.0Apr 2018View details →

Gold standard corpus, ontologies, and Entity-Quality ontology annotations for evolutionary phenotypes

This data set includes a gold-standard corpus of evolutionary phenotype descriptions (in the form of character state descriptions pulled from a variety of phylogenetic systematics studies), and their corresponding expert-curated annotations with ontology terms in the form of Entity-Quality (EQ) statements. EQ annotatons allow machine-reasoning (through the semantics encoded in the requisite ontologies from which the ontology terms are drawn), and machine-reasoning in turn enables computing metrics for quantifying the semantic similarity between different phenotype descriptions as represented by their EQ annotations. Also included are the ontologies, and the human expert-generated and Semantic Charaparser (i.e., machine) generated EQ annotations used to assess Semantic Charaparser performance relative to inter-curator variation and to the effect of having access to external knowledge. The ontologies include those used as input, the "augmented" ontologies created by human curators in each experiment round, and the merged ontology used to maximize Semantic Charaparser's performance. The production of the gold standard corpus, annotation experiments, and evaluation of the results are described in detail in the following manuscript: <blockquote> Dahdul et al (2018) Annotation of phenotypes using ontologies: a Gold Standard for the training and evaluation of natural language processing systems. BioRxiv https://doi.org/10.1101/322156. Submitted to Database. </blockquote> The analysis code for evaluating the gold standard corpus (and the input data and ontologies for that) are available separately from the following: <blockquote> Manda et al (2018) Code and data for analysis of evolutionary phenotype ontology annotations and gold standard corpus. Zenodo. https://doi.org/10.5281/zenodo.1218010 </blockquote> In comparison to the previous version (v1.0.0), this record includes a file of MD5 checksums of the Gold Standard data files. The data files themselves are unchanged.

opencc-zeroSep 2019View details →

Crossref metadata of COCI bibliographic resources, as of November 2018 and LCC categories of the ISBN entities in the dataset

The all.zip CSV file (zipped) contains citation counts obtained from the November 2018 dump of COCI (https://doi.org/10.6084/m9.figshare.6741422.v3) and some metadata (title, DOI, number of authors, ISBN, ISBN of the container, type of the bibliographic resource) of the related citing and cited entities obtained by using the Crossref dump downloaded in October 2018 – which is the same dump used to create the COCI data. In addition, it contains all the Library of Congress Classification (LCC) categories associated with each ISBN in the previous dataset (file isbn_cat_lcc.csv), according to the data retrieved using the services at <a href="http://classify.oclc.org/classify2/api_docs/index.html">http://classify.oclc.org/classify2/api_docs/index.html</a>. Two ancillary mapping files have been also added: one (ddc_to_lcc_mapping.csv) for converting a Dewey Decimal Classification (DDC) categories into LCC categories, in the case the service mentioned above returned only DDC categories for some ISBN; the other (lcc_to_wos_mapping.csv) to map each LCC category into the related <a href="https://images.webofknowledge.com/images/help/WOS/hp_research_areas_easca.html">Web of Science research area</a>.

opencc-zeroAug 2024View details →

Test data sets: Test data set for "Request Entity Too Large" exception

The Encyclopedia of Life (EOL, eol.org) aggregates biodiversity information from more than 400 sources and provides access to the data through taxon pages, visual query and application programming interfaces. Scientific names are essential elements of the data integration infrastructure, but their shortcomings as key identifiers are well documented (Patterson et al., 2016). Complex automated workflows and continuous manual curation are required to address idiosyncrasies of source taxonomies, variation in data quality, and conflicting taxonomic opinions. To achieve a harmonized taxonomic view of EOL content, names from data sources are mapped to a dynamic reference hierarchy ([see current version here](https://opendata.eol.org/dataset/tram-807-808-809-810-dh-v1-1/resource/00adb47b-57ed-4f6b-8f66-83bfdb5120e8)) using an algorithm that leverages canonical name strings, hierarchical information (ancestry, descendants), taxonomic ranks, synonym data, and author strings. Names that cannot be associated with a reference taxon are still accessible, but their unmapped status excludes them and any associated content from certain core EOL functions. For more information about the EOL taxonomy, see [EOL Dynamic Hierarchy](https://eol.org/docs/eol-dynamic-hierarchy)

opencc-by-sa-4.0Oct 2024View details →

Dataset for the paper "Entity Insertion in Multilingual Linked Corpora: The Case of Wikipedia"

Dataset for the EMNLP'24 Main conference paper titled "Entity Insertion in Multilingual Linked Corpora: The Case of Wikipedia".

opencc-by-4.0Aug 2021View details →

Named-Entity Recognition for Modern Tibetan Newspapers: Tagset, Guidelines and Training Data

This dataset, tagset and guidelines were the output of a six-month incubator project on the feasibility of developing Named-Entity Recognition (NER) for modern Tibetan, primarily for use with contemporary Tibetan-language newspapers and media published inside the PRC. The project was carried out by the Mongolian and Inner Asian Studies Unit at Cambridge University’s Department of Social Anthropology. It was funded by an incubator grant from Cambridge Language Sciences. The project title was “Named-Entity Recognition in Tibetan and Mongolian Newspapers.” The Project PI was Dr Hildegard Diemberger (Cambridge), the Coordinator and Lead Author was Dr Robert Barnett (SOAS), and Senior Advisers were Dr Nathan Hill (SOAS), Dr Marieke Meelen (Cambridge), and Dr Thomas White (Cambridge).  Although some forms of NER and other NLP procedures have been developed within China for modern Tibetan (see Liu, Nuo et al, 2011), the data underlying those initiatives have not been made publicly available and their findings cannot be tested or reproduced. Significant work on developing NLP for Tibetan has been carried out outside China, but has focused largely on classical Tibetan and religious texts (see Hill & Garrett, Edward, 2017).  The Cambridge incubator project therefore produced a tagset, guidelines and training data for developing NER for modern Tibetan, with a focus on historical and political analysis of contemporary newspapers, media and other public documents in Tibetan. We compiled 3.11m syllables of data in Tibetan extracted from articles downloaded from Chinese-language news aggregator sites within China, primarily tibet.cpc.people.com.cn and tibet.people.com.cn. From this data, we selected texts containing 280,000 syllables in Tibetan, grouped in 26,000 utterances/sentences (available on request). Using Lighttag, an online annotation site, we developed a tagset for NER consisting of 17 tags (and one for wrong segmentation if using segmented data). We annotated approximately 186,000 syllables, leading to 9,884 annotations. Of these, after discounting flawed data, we produced training data containing c.6,700 annotations.  We carried out the secondary, manual review offline (for our method of converting Lighttag data for offline review, see the attached report “Using Spreadsheets to Review Annotations Offline.pdf”), and found an error rate of 3.6%. The final total of reviewed annotations was 6,624.  The dataset, tagset, guidelines and reports were developed and documented by Robert Barnett, with assistance from Tsering Samdrup, Dr Hill and Dr Meelen. Primary annotation was by Tsering Samdrup, assisted by Dr Barnett. The datasets published here include:        <ol> <li>The tagseet guidelines and annotation manual, including the 17-tag tagset, guidelines, and recommendations ("NER for Modern Tibetan-tagset and guidelines.pdf").</li> <li>The tagged training data in .csv format ("Tibetan NER Training Data-tagged, reviewed wth context-v10-UTF-8.csv") and .xls format ("Tibetan NER Training Data-tagged with context-v10-UTF-8.xlsx"). This includes 6,624 reveiwed annotations, arranged according to the Tibetan alphabet together with the tags and context (utterance) for each annotation.</li> <li>The raw annotation results downloaded from Lighttag as .json files ("Raw Training Data for NER in Modern Tibetan -Jobs2-11-JSON.zip") and as .xls files ("Training Data for NER in Modern Tibetan -Jobs2-11-XLS.zip"). These include 10 "tasks" or datasets of articles scraped from Tibetan-language websites within Tibet.    </li> <li>A guide to preparing Lighttag annotation results for manual review offline (“Using Spreadsheets to Review Annotations Offline.pdf”).</li> </ol> The project's findings regarding the status of NER and NLP for vertical Mongolian are available at DOI: 10.5281/zenodo.5103499.

opencc-by-4.0Apr 2021View details →

French indirectly named entities

A set of indirectly entities in French. (indirectly named entities are named entities described by a periphrasis)