Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
27
datasets available to search
ShareScore release 0.7.1
Dataset results
27 results for “word2vec”
Lemmatized English Word2Vec data
<p># Lemmatized English Word2Vec data</p> <p>This is a version of the original GoogleNews-vectors-negative300 Word2Vec embeddings for English.<br> In addition, we provide the following modified files:</p> <p>- converted to conventional CSV format (and gzipped)<br> - subclassified:<br> for the most frequent 1.000.000 words:<br> subclassified according to WordNet parts of speech: ADJ, ADV, NOUN, VERB, OTHER<br> note that one embedding can be associated with multiple parts of speech<br> for the remaining words:<br> RARE: top 1.000.001 - 2.000.000 words<br> VERY_RARE: top 2.000.001 - 3.000.000 words<br> - WordNet lemmatization (via NLTK) in separate files<br> (first lemma only)</p> <p>Note that this is not a product of original research, but a derived work, deposited here as a point of permanent reference and as a building stone of subsequent research. For such application, a publication independent from Google is necessary to guarantee stability against changes in their data releases.</p> <p>The original Word2vec code and data was published via https://code.google.com/archive/p/word2vec/ under an Apache License 2.0. We obtained the Word2vec data from https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit?usp=sharing on Jun 3, 2020.</p> <p>The Word2vec documentation included the following references:</p> <p> [1] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space. In Proceedings of Workshop at ICLR, 2013.</p> <p> [2] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed Representations of Words and Phrases and their Compositionality. In Proceedings of NIPS, 2013.</p> <p> [3] Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic Regularities in Continuous Space Word Representations. In Proceedings of NAACL HLT, 2013.</p> <p>The derived data is made available under the same license (Apache License 2.0). However, note that the content derived from WordNet (lemmas) are subject to the Princeton Wordnet license as stated in LICENSE.wordnet.</p> <p>Data provided by the Applied Computational Linguistics Lab of the Goethe University Frankfurt, Germany. Original data developed by Mikolov et al.</p>
Decade-level Word2Vec models from automatically transcribed 19th-century newspapers digitised by the British Library (1800-1919)
<p>Word embeddings trained on a 4.2-billion-word corpus of 19th-century British newspapers using Word2Vec and the following parameters:</p> <pre><code>sg = True min_count = 5 window = 5 vector_size = 100 epochs = 5</code></pre> <p>The embeddings are divided into periods of ten years each. Unlike those in <a href="https://doi.org/10.5281/zenodo.7181682">this repository</a>, these were not aligned and OCR errors skimmed from the vocabulary. </p> <p>See related GitHub repository for the full documentation: <a href="https://github.com/Living-with-machines/DiachronicEmb-BigHistData">https://github.com/Living-with-machines/DiachronicEmb-BigHistData</a></p> <p>Project website (Living with Machines): <a href="https://livingwithmachines.ac.uk/">https://livingwithmachines.ac.uk/</a></p>
Word2Vec model - Czech legislation
<p>Word2Vec embedding model trained on Czech legislation (from April 2020) corpus using gensim implementation with the following parameters in addition to default settings:</p> <ul> <li>vector dimension = <span class="math-tex">\(400\)</span>,</li> <li>window size = <span class="math-tex">\(10\)</span>,</li> <li>word minimum count = <span class="math-tex">\(10\)</span>,</li> <li>sample = <span class="math-tex">\(10^{-5}\)</span>.</li> </ul>
Top topics from word2vec run of EFSA outputs
<p>List of neural word embeddings extracted from EFSAs outputs and their closest neighbours, which have a distance < 0.3 to the original word.</p> <p>These group of words form a congruent groups which for a human seems to represent a topic.</p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p>
A word2vec model file built from the French Wikipedia XML Dump using gensim.
<p>A word2vec model file built from the French Wikipedia XML dump using gensim. The data published here includes three model files (you need all three of them in the same folder) as well as the Python script used to build the model (for documentation). The Wikipedia dump was downloaded on October 7, 2016 from https://dumps.wikimedia.org/. Before building the model, plain text was extracted from the dump. The size of that dataset is about 500 million words or 3.6 GB of plain text. The principal parameters for building the model were the following: no lemmatization was performed, tokenization was done using the "\W" regular expression (any non-word character splits tokens), and the model was built with 500 dimensions.</p>
Thai Word Embeddings (word2vec) Trained on Oscar Corpus
<p>A large Thai word2vec model trained on Oscar corpus and tokenized and normalized with PyThaiNLP. The model can be loaded using gensim, it is saved in binary format. </p>
Word2Vec Models built from a Collection of French 20th-Century Novels
<p>The models were trained using the Gensim library for Python, developed by Radim Rehurek, in 2017. All models are based on the same collection of 20th century French novels that covers the period from 1900 to 2010, with a large range of authors and genres respresented. The collection contains approximately 1,200 novels and about 60 million tokens.</p> <p>The models were created using the SGNS (Skip-Gram with Negative Sampling) architecture, the context window was always of size of 6 + 6 around the target word, and the texts were lemmatised and POS-tagged beforehand. POS-Tags remain attached to each token (as in "souris_nom"). Other parameters vary by model: some have 200, some have 300 dimensional vectors; the minimum frequency of the words in the model varies with values of 50, 100 and 200, something which influences the size of the vocabulary and the size of the model. </p>
Spanish 3B words Word2Vec Embeddings
<p>Ready to use gensim Word2Vec embedding models for the Spanish language. Models are created using a window of +/- 5 words, discarding those words with less than 5 instances and creating a vector of 400 dimensions for each word. The text used to create the embeddings has been recovered from news, Wikipedia, the Spanish BOE, web crawling and open literary sources. The used text has a total of 3.257.329.900 words and 18.852.481.207 characters.</p> <p>We support two types of models: Gensim full models (complete_model.zip) and KeyedVectors (keyed_vectors.zip). You can check the differences between them in the following URL: <a href="https://radimrehurek.com/gensim/models/keyedvectors.html">https://radimrehurek.com/gensim/models/keyedvectors.html</a></p> <p>To load the full model use: model = Word2Vec.load("complete.model")<br> To load the KeyedVectors use: word_vectors = KeyedVectors.load('complete.kv', mmap='r')</p> <p>More info about the models can be found in: <a href="https://github.com/aitoralmeida/spanish_word2vec">https://github.com/aitoralmeida/spanish_word2vec</a></p>
word2vec model trained on lemmatized French Wikipedia 2018
<p>The files presented are trained <a href="https://en.wikipedia.org/wiki/Word2vec">word2vec</a> models for French.</p> <p><strong>Corpus</strong></p> <p>The base corpus used for training is a dump of the French Wikipedia performed on 20 October 2018. The corpus was then processed to remove, as much as possible, the Mediawiki syntax, links, etc... Note that this is not perfect, but hopefully has little consequence on the weights calculated.</p> <p>The corpus was then POS-tagged and lemmatized with <a href="http://www.cis.uni-muenchen.de/~schmid/tools/TreeTagger/">TreeTagger</a>. The list of tags can be found <a href="http://www.cis.uni-muenchen.de/~schmid/tools/TreeTagger/data/french-tagset.html">here</a>. During POS-tagging, each word is replaced with its lemma and is replaced with the syntax `[lemma]_[tag]`. For instance, the sentence "Il a sauté dans sa voiture" will produce this output:</p> <pre><code>il_PRO:PER avoir_VER:pres sauter_VER:pper dans_PRP son_DET:POS voiture_NOM</code></pre> <p><strong>Word2vec training</strong></p> <p>The training was performed using the <a href="https://pypi.org/project/gensim/">gensim</a> Python module (v3.5.0). The skip-grams method was used, with a size of 500, a window size of 5 and a minimum count of 5. This is how the model creation was invoked:</p> <pre><code class="language-python">model = Word2Vec(size=500, window=5, min_count=5, workers=workers, sg=1)</code></pre> <p>Two versions of the model were trained:</p> <ul> <li>frwiki-20181020.treetag.2__2019-01-24_10.41__.s500_w5_skip.word2vec.bin<br> No extra processing was performed on the corpus before training.</li> <li>frwiki-20181020.treetag.2.ngram-pass2__2019-04-08_09.02__.s500_w5_skip.word2vec.bin<br> Where two passes of the 2-gram detection were ran before training. This allows detection of 2-, 3- and 4-grams in the vocabulary.</li> </ul> <p>The files are in the word2vec binary format, so can be used either with the original <a href="https://github.com/tmikolov/word2vec">C implementation of word2vec</a>, or with the Python gensim version (and possibly other libraries that support that format).</p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p>
LTW2V: The Large Thai Word2Vec
<p>LTW2V is The large Thai Word2Vec. It trained from OSCAR Corpus (Open Super-large Crawled Aggregated coRpus).</p> <p>For Version 1.0, It trained from OSCAR Corpus (Open Super-large Crawled Aggregated coRpus) and use newmm in PyThaiNLP 4.0 for word segmentation. We cleand the dataset before training that customize the pre-processing script from <a href="https://github.com/cstorm125/thai2fit">thai2fit</a> and trained 5 window, 15 windows. It trained with Gensim (50 epochs), so you can use the model from Gensim.</p> <p><strong>About Word2Vec</strong></p> <ul> <li>vector dimension = 400</li> <li>window size = 5, 15</li> <li>word minimum count = 5</li> </ul> <p>Source code at GitHub: <a href="https://github.com/PyThaiNLP/large-thaiword2vec">https://github.com/PyThaiNLP/large-thaiword2vec</a></p> <p><strong>File</strong></p> <ul> <li>LTW2V_v1.0-window5.bin - 5 window with newmm in PyThaiNLP 4.0 for word segmentation and Gensim 4.0.</li> <li>LTW2V_v1.0-window15.bin - 15 window with newmm in PyThaiNLP 4.0 for word segmentation and Gensim 4.0.</li> </ul>
Data and trained word2vec model for ``Easy over Hard: A Case Study on Deep Learning''
<p>The data include: training and testing data pairs</p> <p>The word2vec model is pre-trained. </p> <p>More details, please refer to the paper</p>
Pre-trained word2vec models for ``Easy over Hard: A Case Study on Deep Learning''
<p>Since the whole stack overflow dump is so big, we can't easily handle well. Here, we provide 10 pre trained word2vec models with different seeds.</p> <p> </p> <p>More details about how to use it, please see paper </p>
Word2vec models trained on English Wikipedia
<p>This repository contains Word2Vec models trained on the full text of the English Wikipedia as downloaded in December 2021.</p> <p>Preprocessing:</p> <ul> <li>lowercasing</li> <li>n-grams up to 4-grams were computed using Bouma 2009 (https://svn.spraakdata.gu.se/repos/gerlof/pub/www/Docs/npmi-pfd.pdf), min freq threshold of 10</li> </ul> <p>Two models, trained with Gensim:</p> <ul> <li>wiki_300_5_word2vec --> dim 300, freq threshold 5</li> <li>wiki_300_50_word2vec --> dim 300, freq threshold 50</li> </ul> <p>Other hyperparameters set as follows: window=5, epochs=5, seed=1830, sg=1</p> <p>Note:<br> Machine learning models trained on uncurated data inevitably learn hidden or obvious biases and as a result, the models shared with here might contain characteristics including sexism, racism, antisemitism, homophobia, and other such types of unacceptable biases. I encourage whoever is using these models to make sure such biases are actually removed before using them in production settings (see eg https://aclanthology.org/N19-1061/)</p>
Data Vektor Embeddings Menggunakan IndoBERT dan Word2Vec Terkait Pemilu Pemilu Presiden Indonesia
Open the record for dataset details and reuse information.
Time shifting word2vec models from Times
<p>Time-shifting word2vec models based on Times news paper. These models were generated using the "Generate time shifting models" scripts found <a href="https://github.com/c-martinez/w2v-convergence/#generate-time-shifting-models">here</a>.</p> <p>In summary, these scripts generate a collection of sentences for every 2 years period, and trains a word2vec model on this period using <a href="https://radimrehurek.com/gensim/">gensim</a>. The original text from the Times news paper articles is processed as follows:</p> <ul> <li>Articles are divided into sentences using punctuation.</li> <li>Punctuation symbols are removed.</li> <li>Text is converted to lower case.</li> <li>Word are validated to ensure they are valid English non-stop words (using nltk).</li> </ul> <p>The two year time period was selected following the Measure convergence for a range described <a href="https://github.com/c-martinez/w2v-convergence/#measure-convergence-for-a-range">here</a>.</p> <p>This data publication was made possible thanks to collaboration with the <a href="https://dig.hum.uu.nl/">Utrecht Digital Humanities Lab</a>.</p> <p>Unfortunately original Times data set is not publicly available.</p>
Word2Vec Models Dutch Newspapers
<p>Word Embedding models trained on 6 national Dutch newspapers. </p> <p>We use the Gensim implementation of Word2Vec to train four embedding models per newspaper, each representing one decade between 1950 and 1990. The models were trained using C-BOW with hierarchical softmax, with a dimensionality of 300, a minimal word count and context of 5, and downsampling of 10<sup>-5</sup></p> <p>These models belong to the article: Using Word Embeddings to Examine Gender Bias in Dutch Newspapers, 1950-1990</p>
Training word vectors on text from The Physics Teacher using Word2Vec
<p>This notebook and dataset allows one to play around with word vectors trained on text from articles in the journal <a href="https://pubs.aip.org/aapt/pte">The Physics Teacher</a> published between 1963 (the start of publication) and 2020, around 15000 articles in total.</p> <p>The primary datafile, "TPT_word2vec_words_bigrams_V1.pkl", is a list of cleaned text from these articles. It contains a list, within which each paper is a sub-list. Each sentence in that paper is yet another sub-list which contains the words in that sentence in order. However, in the data cleaning process we have removed “stop words” (like if, and, but, etc.), punctuation, symbols, and numbers, as well as lowercased all words and combined words that frequently go together into one (like “high” and "school” to “high_school”). Here is an example of 3 sentences taken from a random paper: <br> <br>[['magnet', 'spin', 'tape', 'magnetize', 'strongly', 'time', 'pole', 'approach'],<br>['magnet', 'place', 'center', 'counterweight', 'period', 'magnetize', 'pulse', 'twice', 'long'],<br>['trial', 'tape', 'examine', 'sprinkle_iron', 'filing', 'length'], ... ]</p> <p>In the notebook, we create a set of word vectors from these sentences using the Word2Vec technique, first published by Mikolov et al. (2013):</p> <p>Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space (No. arXiv:1301.3781). arXiv. https://doi.org/10.48550/arXiv.1301.3781</p> <p>The notebook includes code for both loading in word vectors from a trained model (also included, "TPT_word2vec.model") and creating the same model from the TPT text dataset. Note that word vectors are randomly initialized, so we include a random seed to make this training replicable. Changing the seed will alter some of the results (although the changes seem fairly minor).</p> <p>With a trained model, we demonstrate some applications of word vectors: adding and subtracting meanings <br>(for example "experiment" - "uncertainty" = "demonstration") and visualizing low-dimensional representations of word vectors.</p> <p>In order to install the required packages, you can use the requirements.txt file. If using pip, run "pip install requirements.txt". Or, if using Anaconda (recommended), you can use "conda install --file requirements.txt". You will also need the software to run jupyter notebooks, which can be installed with Anaconda or pip.</p>
Spanish 3B words Word2Vec Embeddings [Update for gensim>4]
<p>This is the KeyedVectors of the gensim model created by Aitor Almeida and Aritz Bilbao, but compatible with Gensim>4 versions. </p> <p>The training and generation of these vectors are detailed in the <a href="https://github.com/aitoralmeida/spanish_word2vec">GitHub repository</a> and in the official <a href="https://zenodo.org/record/1410403">Zenodo</a>. </p> <p>Please, any kind of citation or mention should be made to its <a href="https://zenodo.org/record/1410403">original repository</a> since I am not the research scientist who originally generate this model and all the merit is theirs.</p> <p> </p> <p>To load this new version you should use:</p> <p><em>word_vectors = KeyedVectors.load_word2vec_format("complete.bin",binary=True)</em></p>
Italian Word2Vec models
<p>Italian Word2Vec models trained from scratch on a dataset composed of:</p> <p>- <strong>wiki</strong>: a dump of Italian Wikipedia (as of December 15, 2022), comprising 25,548,651 sentences and 526,640,982 words (3.2 GB of raw text);<br> - <strong>webz</strong>: a dataset of Italian news (159,226 documents) from the webz.io platform, crawled in October 2015, containing 44,041,823 sentences and 44,544,385 words (244 MB);<br> - a dataset of 5,510 Italian news articles from the newspaper ModenaToday (<strong>MT</strong>) or 15,115 documents from the Italian version of Reuters (<strong>RCV2</strong>).</p> <p><strong>w2v_wiki_wbz_mt_20_epochs.zip</strong>: Word2Vec model trained on the dataset consisting of wiki, webz, and MT for 20 epochs</p> <p><strong>w2v_wiki_wbz_mt_50_epochs.zip</strong>: Word2Vec model trained on the dataset consisting of wiki, webz, and MT for 50 epochs</p> <p><strong>w2v_wiki_wbz_reut_20_epochs.zip</strong>: Word2Vec model trained on the dataset consisting of wiki, webz, and RCV2 for 20 epochs</p> <p><strong>w2v_wiki_wbz_reut_50_epochs.zip</strong>: Word2Vec model trained on the dataset consisting of wiki, webz, and RCV2 for 50 epochs</p>
Word2Vec model - Czech wikipedia
<p>Word2Vec embedding model trained on Czech wikipedia (from April 2020) corpus using gensim implementation with the following parameters in addition to default settings:</p> <ul> <li>vector dimension = <span class="math-tex">\(400\)</span>,</li> <li>window size = <span class="math-tex">\(10\)</span>,</li> <li>word minimum count = <span class="math-tex">\(10\)</span>,</li> <li>sample = <span class="math-tex">\(10^{-5}\)</span>.</li> </ul>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.