Find research datasets worth reusing

opencc-by-4.0Mar 2021View details →

Sentiment Analysis and Cross-lingual Word Embeddings for Endangered Languages

A sentiment analyzer and cross-lingual word embeddings for endangered languages (e.g., Erzya, Moksha, Skolt Sami, Komi-Zyrian).

opencc-by-nc-nd-4.0Feb 2021View details →

Pokémon Word Embeddings

Word vector models for Pokémon (poke2vec). Word2Vec, FastText and Meta4Meaning models trained on a big Pokémon corpus. code.zip files has examples of how to load and use the models. Please cite the following paper if you use the resources: Hämäläinen, M.,  Alnajjar, K. & Partanen, N. (2021). <a href="https://researchportal.helsinki.fi/en/publications/nettikorpuksen-avulla-tuotettuja-sanavektorimalleja-pok%C3%A9monien-om">Nettikorpuksen avulla tuotettuja sanavektorimalleja Pokémonien ominaisuuksien kuvaamiseksi</a>. In Saarikivi, T. & Saarikivi, J. (eds.) Turhan tiedon kirja — Tutkimuksista pois jätettyjä sivuja. p. 199-214. SKS Kirjat. <a href="https://www.researchgate.net/publication/354088508_How_Cute_is_Pikachu_Gathering_and_Ranking_Pokemon_Properties_from_Data_with_Pokemon_Word_Embeddings">English version of the paper</a>

opencc-by-4.0Nov 2022View details →

Spanish CBOW Word Embeddings in Floret

Spanish CBOW Word Embeddings in Floret The embeddings have been trained with the corpus from the National Library of Spain (<a href="http://www.bne.es/en/Inicio/index.html">Biblioteca Nacional de España</a> or BNE)  using <a href="https://arxiv.org/pdf/1607.04606.pdf">floret</a> with the following  hyperparameters: <blockquote> mode: str = "floret", model: str = "cbow", dim: int = 300, mincount: int = 10, minn: int = 5, maxn: int = 6, neg: int = 10, hashcount: int = 2, bucket: int = 50000, thread: int = 128, </blockquote>   Detailed information about the corpus can be found <a href="http://journal.sepln.org/sepln/ojs/ojs/index.php/pln/article/view/6405">here </a> The processing took place on an HPC <a href="https://www.bsc.es/innovation-and-services/technical-information-cte-amd">node</a> equipped with an AMD EPYC 7742 (@ 2.250GHz) processor with 128 threads. How to use First initialize the spacy vectors from the floret table (.floret file): <pre><code>spacy init vectors es floret_embeddings_bne_es.floret floret_embeddings_bne_es --mode floret</code></pre> <pre><code>import spacy # Load the floret vectors floret_embeddings = spacy.load("floret_embeddings_bne_es") # Get the embeddings of some words playa = floret_embeddings.vocab["playa"] frío = floret_embeddings.vocab["frío"] invierno = floret_embeddings.vocab["invierno"] verano = floret_embeddings.vocab["verano"] # Get some similarities print(frío.similarity(invierno)) print(frío.similarity(verano)) # frío should be more similar to invierno than verano. print(playa.similarity(invierno)) print(playa.similarity(verano)) # playa should be more similar to verano than invierno.</code></pre> Intended Uses and Limitations At the time of submission, no measures have been taken to estimate the bias and toxicity embedded in the model. However, we are well aware that our models may be biased since the corpora have been collected using crawling techniques on multiple web sources. We intend to conduct research in these areas in the future, and if completed, this card will be updated. Authors The Text Mining Unit from Barcelona Supercomputing Center. Contact Information For further information, send an email to <a href="http://plantl-gob-es@bsc.es/">plantl-gob-es@bsc.es</a> Funding This work was funded by the <a href="https://portal.mineco.gob.es/en-us/digitalizacionIA/Pages/sedia.aspx">Spanish State Secretariat for Digitalization and Artificial Intelligence (SEDIA)</a> within the framework of the Plan-TL. Copyright Copyright (c) 2021 Secretaría de Estado de Digitalización e Inteligencia Artificial

opencc-by-4.0Nov 2022View details →

Biomedical Spanish CBOW Word Embeddings in Floret

Biomedical Spanish CBOW Word Embeddings in Floret The embeddings have been trained with a biomedical Spanish corpus using <a href="https://arxiv.org/pdf/1607.04606.pdf">floret</a> with the following  hyperparameters: <blockquote> mode: str = "floret", model: str = "cbow", dim: int = 300, mincount: int = 10, minn: int = 5, maxn: int = 6, neg: int = 10, hashcount: int = 2, bucket: int = 50000, thread: int = 128, </blockquote> The embeddings were trained on the concatenation of all corpora from the Spanish biomedical corpus that includes Spanish data from various sources for a total of 1.1B tokens across 2,5M documents. <table> <thead> <tr> <th scope="col">Source</th> <th scope="col">No. tokens</th> </tr> </thead> <tbody> <tr> <td>Medical crawler</td> <td>903,558,136</td> </tr> <tr> <td>Clinical cases misc.</td> <td>102,855,267</td> </tr> <tr> <td>EHRs documents*</td> <td>95,267,204</td> </tr> <tr> <td>Scielo</td> <td>60,007,289</td> </tr> <tr> <td>BARR2 Background</td> <td>24,516,442</td> </tr> <tr> <td>Wikipedia (Life Sciences)</td> <td>13,890,501</td> </tr> <tr> <td>Patents</td> <td>13,463,387</td> </tr> <tr> <td>EMEA</td> <td>5,377,448</td> </tr> <tr> <td>Mespen (MedlinePlus)</td> <td>4,166,077</td> </tr> <tr> <td>PubMed</td> <td>1,858,966</td> </tr> </tbody> </table> More information about the corpus can be found here <a href="https://aclanthology.org/2022.bionlp-1.19/">https://aclanthology.org/2022.bionlp-1.19/</a> and here <a href="https://arxiv.org/abs/2109.07765">https://arxiv.org/abs/2109.07765</a> The processing took place on an HPC <a href="https://www.bsc.es/innovation-and-services/technical-information-cte-amd">node</a> equipped with an AMD EPYC 7742 (@ 2.250GHz) processor with 128 threads. How to use First initialize the spacy vectors from the floret table (.floret file): <pre><code class="language-bash">spacy init vectors es floret_embeddings_bio_es.floret floret_embeddings_bio_es --mode floret</code></pre> <pre><code>import spacy # Load the floret vectors floret_embeddings = spacy.load("floret_embeddings_bio_es") # Get the embeddings of some words diabetes = floret_embeddings.vocab["diabetes"] insulina = floret_embeddings.vocab["insulina"] radiografia = floret_embeddings.vocab["radiografia"] # Get some similarities print(diabetes.similarity(insulina)) print(diabetes.similarity(radiografia)) # diabetes should be more similar to insuline than radiografia </code></pre>   Intended Uses and Limitations At the time of submission, no measures have been taken to estimate the bias and toxicity embedded in the model. However, we are well aware that our models may be biased since the corpora have been collected using crawling techniques on multiple web sources. We intend to conduct research in these areas in the future, and if completed, this card will be updated. Authors The Text Mining Unit from Barcelona Supercomputing Center. Contact Information For further information, send an email to <a href="http://plantl-gob-es@bsc.es">plantl-gob-es@bsc.es</a> Funding This work was funded by the <a href="https://portal.mineco.gob.es/en-us/digitalizacionIA/Pages/sedia.aspx">Spanish State Secretariat for Digitalization and Artificial Intelligence (SEDIA)</a> within the framework of the Plan-TL. Copyright Copyright (c) 2022 Secretaría de Estado de Digitalización e Inteligencia Artificial

opencc-by-4.0Nov 2022View details →

Catalan CBOW Word Embeddings in Floret

Embeddings with the Catalan Textual Corpus The embeddings have been trained with a Catalan textual corpus of  more than 34GB of data using <a href="https://arxiv.org/pdf/1607.04606.pdf">floret</a> with the following  hyperparameters: <blockquote>     mode: str = "floret",     model: str = "cbow",     dim: int = 300,     mincount: int = 10,     minn: int = 5,     maxn: int = 6,     neg: int = 10,     hashcount: int = 2,     bucket: int = 50000,     thread: int = 128, </blockquote> The Catalan Textual Corpus used to train this embeddings, is the extended version of  the initial available corpora described in <a href="https://arxiv.org/pdf/2107.07903.pdf">Armengol-Estapé et al. (2021)</a>. This new version includes: <table> <caption> </caption> <thead> <tr> <th scope="col">Corpus</th> <th scope="col">Size in GB</th> </tr> </thead> <tbody> <tr> <td>CaCrawlat</td> <td>13.00</td> </tr> <tr> <td>Wikipedia</td> <td>1.10</td> </tr> <tr> <td>DOGC</td> <td>0.78</td> </tr> <tr> <td>Catalan Open Subtitles</td> <td>0.02</td> </tr> <tr> <td>Catalan Oscar</td> <td>4.00</td> </tr> <tr> <td>CaWaC</td> <td>3.60</td> </tr> <tr> <td>Cat. General Crawling</td> <td>2.50</td> </tr> <tr> <td>Cat. Goverment Crawling</td> <td>0.24</td> </tr> <tr> <td>ACN</td> <td>0.42</td> </tr> <tr> <td>Padicat</td> <td>0.63</td> </tr> <tr> <td>RacoCatalà</td> <td>8.10</td> </tr> <tr> <td>NacióDigital</td> <td>0.42</td> </tr> <tr> <td>VilaWeb</td> <td>0.06</td> </tr> </tbody> </table> From the new corpora, VilaWeb and NacióDigital come from digital newspapers, Padicat is composed of crawlings of the Biblioteca de Catalunya, and CaCrawlat comes from the Biblioteca Nacional de España (BNE). The processing took place on an HPC <a href="https://www.bsc.es/innovation-and-services/technical-information-cte-amd">node</a> equipped with an AMD EPYC 7742 (@ 2.250GHz) processor with 128 threads. How to use First initialize the spacy vectors from the floret table (.floret file): <pre><code class="language-bash">spacy init vectors ca floret_embeddings_ca.floret floret_embeddings_ca --mode floret</code></pre> <pre><code class="language-python">import spacy # Load the floret vectors floret_embeddings = spacy.load("floret_embeddings_ca") # Get the embeddings of some words castanyes = floret_embeddings.vocab["castanyes"] flors = floret_embeddings.vocab["flors"] primavera = floret_embeddings.vocab["primavera"] tardor = floret_embeddings.vocab["tardor"] # Get some similarities print(flors.similarity(tardor)) print(flors.similarity(primavera)) # flors should be more similar to primavera than tardor. print(castanyes.similarity(primavera)) print(castanyes.similarity(tardor)) # castanyes should be more similar to tardor than primavera.</code></pre> Intended Uses and Limitations At the time of submission, no measures have been taken to estimate the bias and toxicity embedded in the model. However, we are well aware that our models may be biased since the corpora have been collected using crawling techniques on multiple web sources. We intend to conduct research in these areas in the future, and if completed, this card will be updated. Authors The Text Mining Unit from Barcelona Supercomputing Center. Contact Information For further information, send an email to <a href="mailto:aina@bsc.es">aina@bsc.es</a>. Funding This work was funded by the <a href="https://politiquesdigitals.gencat.cat/ca/inici/index.html">Departament de la Vicepresidència i de Polítiques Digitals i Territori de la Generalitat de Catalunya</a>  within the framework of <a href="https://politiquesdigitals.gencat.cat/ca/economia/catalonia-ai/aina">Projecte AINA</a>. Copyright Copyright (c) 2022 Text Mining Unit  - Barcelona Supercomputing Center.

opencc-by-4.0Oct 2021View details →

Ancient Greek Fasttext Word Embeddings

Word embeddings generated with Fasttext and 1 GB of Ancient Greek texts. These embeddings were produced for the study of social networks and social semantics in ancient Greece by the Diogenet project at the University of San Diego, California.

opencc-by-4.0Mar 2020View details →

Supplementary Material for the paper: Automatic Document Screening of Medical Literature Using Word and Text Embeddings in an Active Learning Setting

This is the dataset used in the paper: Automatic Document Screening of Medical Literature Using Word and Text Embeddings in an Active Learning Setting.  It is composed of:  - Pre-trained models using active learning for document screening on HealthCLEF and Epistemonikos datasets.  - Epistemonikos and HealthCLEF datasets containing medical questions and relevant/non relevant articles.  - Embeddings and Document Representations used for experiments on both datasets.  Scripts to run experiments can be found at: <a href="https://github.com/afcarvallo/active_learning_document_screening">https://github.com/afcarvallo/active_learning_document_screening</a>   Paper abstract: Document screening is a fundamental task within Evidence-based Medicine (EBM), a practice that provides scientific evidence to support medical decisions. Several approaches have tried to reduce physicians' workload of screening and labeling vast amounts of documents to answer clinical questions. Previous works tried to semi-automate document screening, reporting promising results, but their evaluation was conducted on small datasets, which hinders generalization. Moreover, recent works in natural language processing have introduced neural language models, but none have compared their performance in EBM. In this paper, we evaluate the impact of several document representations such as TF-IDF along with neural language models (BioBERT, BERT, Word2vec, and GloVe) on an active learning-based setting for document screening in EBM. Our goal is to reduce the number of documents that physicians need to label to answer clinical questions. We evaluate these methods using both a small challenging dataset (HealthCLEF 2017) as well as a larger one but easier to rank (Epistemonikos). Our results indicate that word as well as textual neural embeddings always outperform the traditional TF-IDF representation. When comparing among neural and textual embeddings, in the HealthCLEF dataset the models BERT and BioBERT yielded the best results. On the larger dataset, Epistemonikos, Word2Vec and BERT were the most competitive, showing that BERT was the most consistent model across different corpuses. In term of active learning, an uncertainty sampling strategy combined with logistic regression achieved the best performance overall, above other methods under evaluation, and in fewer iterations.

opencc-by-4.0Mar 2017View details →

Russian Distributional Thesaurus (RDT): Word Embeddings

This resource is a part of the Russian Distributional Thesaurus (RDT): see http://russe.nlpub.ru/downloads and http://nlpub.ru/RDT. This dataset contains a large scale word embeddings model for Russian trained using the SGNS model (Mikolov et al., 2013) on a 12.9 billion word collection of books in Russian. According to the results of our participation in the shared task on Russian semantic similarity (Panchenko et al., 2015), this approach scored in the top 5 among 105 submissions (Arefyev et al., 2015). Following our prior experiments (Arefyev et al., 2015) we have selected the following parameters for the model: minimal word frequency – 5, number of dimensions in a word vector – 500, three or five iterations of the learning algorithm over the input corpus, context window size of 1, 2, 3, 5, 7 and 10 words. Parameters of the model are listed below: <ul> <li>Model: skip-gram</li> <li>Corpus: a 150Gb sample of the lib.rus.ec book collection.</li> <li>Context window size: 10 words</li> <li>Number of dimensions: 500</li> <li>Number of iterations: 3</li> <li>Minimal word frequency: 5</li> </ul> References: <ul> <li>Panchenko A., Ustalov D., Arefyev N., Paperno D., Konstantinova N., Loukachevitch N. and Biemann C. (2016): Human and Machine Judgements about Russian Semantic Relatedness. In Proceedings of the 5th Conference on Analysis of Images, Social Networks, and Texts (AIST'2016). Communications in Computer and Information Science (CCIS). Springer-Verlag Berlin Heidelberg</li> </ul> <ul> <li>Panchenko A., Loukachevitch N. V., Ustalov D., Paperno D., Meyer C. M., Konstantinova N. (2015): RUSSE: The First International Workshop on Russian Semantic Similarity. In Proceedings of the 21st International Conference on Computational Linguistics and Intellectual Technologies (Dialogue'2015). Moscow, Russia. RGGU</li> </ul> <ul> <li>Arefyev N., Panchenko A., Lukanin A., Lesota O., Romanov P. (2015): Evaluating Three Corpus-Based Semantic Similarity Systems for Russian. In Proceedings of the 21st International Conference on Computational Linguistics and Intellectual Technologies (Dialogue'2015). Moscow, Russia. RGGU</li> </ul>

opencc-by-4.0May 2017View details →

Word Embedding Data Sets Learned from Tweets and General Data

This includes 10 word embedding data sets learned from about 400 million tweets and 7 billion words from general data. They can be used in tasks involving social media data, especially tweets, and other types of textual data. Users can choose different embedding sets based on their use cases; they can also easily try all of them to see which one provides the best performance for their application. More details about the training data collection, word embedding generation, preprocessing steps, and how to use them can be found from the following paper: Quanzhi Li, Sameena Shah, Xiaomo Liu, Armineh Nourbakhsh, Data Set: Word Embeddings Learned from Tweets and General Data, The 11th International AAAI Conference on Web and Social Media (ICWSM-17). Montreal, Canada. May 16-18, 2017

opencc-by-4.0Nov 2018View details →

sentiment specific word embedding

sentiment specific word embedding  learned based on the approach described in the following paper: D. Tang, et al., Learning Sentiment-Specific Word Embedding for Twitter Sentiment Classification, ACL 2014

opencc-by-4.0Apr 2018View details →

An Empirical Study of Word Embedding Dimensionality Reduction

In order to analyze the impact on model quality while reducing the number of dimensions, strictly controlled trainings of word embedding are performed on Wikipedia corpora of 170 languages. The specially designed word embedding training tool makes use of processed corpus and intermediate results to accelerate the training, while keeping the consistency of negative sampling. Tests of semantic relatedness show that, except for some corpora of poor scale, the margin gain from extra dimensions significantly decreases above 200.

opencc-by-4.0Jun 2021View details →

Arabic Word Embedding Models

These are several Arabic Word Embedding Models for NLP tasks and it has been described in our paper titled "Leveraging Arabic Sentiment Classification Using an Enhanced CNN-LSTM Approach and Effective Arabic Text Preparation"

opencc-by-4.0Dec 2022View details →

Sentiment analysis data and word embeddings for Erzya, Komi-Zyrian, Moksha and Udmurt

The aligned sentiment annotated data is in setiment_eval_data.json, vectors.zip has the word embeddings in a textual Gensim format, code.zip has the code and models.zip the sentiment analysis model. Please cite the following paper: Alnajjar, K., Hämäläinen, M., & Rueter, J, (2023) Sentiment Analysis Using Aligned Word Embeddings for Uralic Languages. In Proceedings of the Second Workshop on Resources and Representations for Under-resourced Languages and Domains (RESOURCEFUL-2023)

opencc-by-4.0Apr 2023View details →

Spanish 3B words Word2Vec Embeddings [Update for gensim>4]

This is the KeyedVectors of the gensim model created by Aitor Almeida and Aritz Bilbao, but compatible with Gensim>4 versions.  The training and generation of these vectors are detailed in the <a href="https://github.com/aitoralmeida/spanish_word2vec">GitHub repository</a> and in the official <a href="https://zenodo.org/record/1410403">Zenodo</a>.  Please, any kind of citation or mention should be made to its <a href="https://zenodo.org/record/1410403">original repository</a> since I am not the research scientist who originally generate this model and all the merit is theirs.   To load this new version you should use: word_vectors = KeyedVectors.load_word2vec_format("complete.bin",binary=True)

opencc-by-4.0Oct 2023View details →

Early Slavic word embeddings

Word embeddings trained on the lemmatised TOROT Treebank, using Word2Vec and the following parameters: <pre><code>sg = True min_count = <1,3,5> window = <3,5> vector_size = <100,200,300> epochs = 5</code></pre> One model was trained for each combination of the parameters enclosed in angled brackets (< >).  The release contains both the full models (.model) and the plain vector files (_vectors.txt). The models are named according to the parameters they were trained with. Note that these are the result of very preliminary experiments and no systematic evaluation of their quality was carried out, so use with caution.

opencc-by-4.0Oct 2020View details →

Do Word Embeddings Capture Spelling Variation?

When using the data, please cite: "Do Word Embeddings Capture Spelling Variation?". Dong Nguyen and Jack Grieve, COLING 2020. The data contains: <ul> <li>the trained embeddings (embeddings-reddit.tgz and embeddings-twitter.tgz), 50-300 dimensions</li> <li>the analysis/output files (data.tgz).</li> </ul> See also the Github repository: https://github.com/dongpng/coling2020.

opencc-zeroJan 2015View details →

Phrase-Level Word Embeddings

Phrase-level word embeddings for sentiment analysis.

opencc-by-4.0Jan 2022View details →

Korean embedding files using the different morphological segmentation granularity of the word

Embedding files using the following segmentation: <ol> <li>wordUD</li> <li>morphUD</li> <li>+morphUD </li> </ol> Based on wordUD there are 9,692,938 sentences and 157,653,628 words (tokenized) including all articles published in The Hankyoreh during 2016 (1.2M sentences), Sejong morphologically analyzed corpus (3M), and Korean Wiki (20201101) (5.3M):   <blockquote> ./fasttext skipgram -input input -output embedding -dim 300 </blockquote>

dryad32/100

Data from: Wide range screening of algorithmic bias in word embedding models using large sentiment lexicons reveals underreported bias types

Concerns about gender bias in word embedding models have captured substantial attention in the algorithmic bias research literature. Other bias types however have received lesser amounts of scrutiny. This work describes a large-scale analysis of sentiment associations in popular word embedding models along the lines of gender and ethnicity but also along the less frequently studied dimensions of socioeconomic status, age, physical appearance, sexual orientation, religious sentiment and political leanings. Consistent with previous scholarly literature, this work has found systemic bias against given names popular among African-Americans in most embedding models examined. Gender bias in embedding models however appears to be multifaceted and often reversed in polarity to what has been regularly reported. Interestingly, using the common operationalization of the term bias in the fairness literature, novel types of so far unreported bias types in word embedding models have also been identified. Specifically, the popular embedding models analyzed here display negative biases against middle and working-class socioeconomic status, male children, senior citizens, plain physical appearance and intellectual phenomena such as Islamic religious faith, non-religiosity and conservative political orientation. Reasons for the paradoxical underreporting of these bias types in the relevant literature are probably manifold but widely held blind spots when searching for algorithmic bias and a lack of widespread technical jargon to unambiguously describe a variety of algorithmic associations could conceivably be playing a role. The causal origins for the multiplicity of loaded associations attached to distinct demographic groups within embedding models are often unclear but the heterogeneity of said associations and their potential multifactorial roots raises doubts about the validity of grouping them all under the umbrella term bias. Richer and more fine-grained terminology as well as a more comprehensive exploration of the bias landscape could help the fairness epistemic community to characterize and neutralize algorithmic discrimination more efficiently.

opencc-zeroApr 2020View details →