Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
4
datasets available to search
ShareScore release 0.9.0
Dataset results
4 results for “lemmatization”
Lemmatized English Word2Vec data
<p># Lemmatized English Word2Vec data</p> <p>This is a version of the original GoogleNews-vectors-negative300 Word2Vec embeddings for English.<br> In addition, we provide the following modified files:</p> <p>- converted to conventional CSV format (and gzipped)<br> - subclassified:<br> for the most frequent 1.000.000 words:<br> subclassified according to WordNet parts of speech: ADJ, ADV, NOUN, VERB, OTHER<br> note that one embedding can be associated with multiple parts of speech<br> for the remaining words:<br> RARE: top 1.000.001 - 2.000.000 words<br> VERY_RARE: top 2.000.001 - 3.000.000 words<br> - WordNet lemmatization (via NLTK) in separate files<br> (first lemma only)</p> <p>Note that this is not a product of original research, but a derived work, deposited here as a point of permanent reference and as a building stone of subsequent research. For such application, a publication independent from Google is necessary to guarantee stability against changes in their data releases.</p> <p>The original Word2vec code and data was published via https://code.google.com/archive/p/word2vec/ under an Apache License 2.0. We obtained the Word2vec data from https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit?usp=sharing on Jun 3, 2020.</p> <p>The Word2vec documentation included the following references:</p> <p> [1] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space. In Proceedings of Workshop at ICLR, 2013.</p> <p> [2] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed Representations of Words and Phrases and their Compositionality. In Proceedings of NIPS, 2013.</p> <p> [3] Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic Regularities in Continuous Space Word Representations. In Proceedings of NAACL HLT, 2013.</p> <p>The derived data is made available under the same license (Apache License 2.0). However, note that the content derived from WordNet (lemmas) are subject to the Princeton Wordnet license as stated in LICENSE.wordnet.</p> <p>Data provided by the Applied Computational Linguistics Lab of the Goethe University Frankfurt, Germany. Original data developed by Mikolov et al.</p>
Pennsylvania German word list (lemmatized and POS-annotated)
<p>The file presents the words used in the Pennsylvania German part of the ENDE corpus (www.deitsch.eu). The list contains every lemma with its associated word forms documented in the corpus, comprised of 1761 lemmata and 2704 word forms.</p> <p>The ENDE corpus (“English-Deitsch translation corpus”) is the first POS-annotated and searchable text corpus in Pennsylvania German (= Deitsch; ISO language code: pdc), aligned to the English source texts. Despite many digital texts in Deitsch are available on the internet, there are, so far, no digital corpora for this language. This is due mainly to the lack of a generally recognized standard variety which could serve as a reference point for the linguistic analysis needed for lemmatization and annotation.</p> <p>Lemmatization was done with the help of different lexicographic resources (https://www.deitsch.eu/news/view/9) most of which follow other spelling conventions. A fair number of word forms, especially English loanwords of some sort, cannot be found in the dictionaries. Moreover, the variety used here is characterized by a high variability regarding not only the spelling but also other aspects of the language.</p> <p>Part-of-speech tags were assigned manually (see tagsets A and B below). These tagsets for part-of-speech annotation of Deitsch texts are based on the 2017 version of the STTS system created and widely used for German (https://ids-pub.bsz-bw.de/frontdoor/deliver/index/docId/6063/file/Westpfahl_Schmidt_Jonietz_Borlinghaus_STTS_2_0_2017.pdf), which has been slightly modified and adapted to the corpus texts written in the Plain Deitsch variety. Tagset A gives a broader view and refers to the lemma level, tagset B is more fine-grained and suitable for the single word forms documented in the corpus. Only those tags are listed which are actually employed for the annotation of the corpus texts. Foreign items not integrated in the Deitsch text flow (e.g. English quotations) have been omitted.</p> <p>For more details about the corpus and the project please refer to the above mentioned website.</p>
word2vec model trained on lemmatized French Wikipedia 2018
<p>The files presented are trained <a href="https://en.wikipedia.org/wiki/Word2vec">word2vec</a> models for French.</p> <p><strong>Corpus</strong></p> <p>The base corpus used for training is a dump of the French Wikipedia performed on 20 October 2018. The corpus was then processed to remove, as much as possible, the Mediawiki syntax, links, etc... Note that this is not perfect, but hopefully has little consequence on the weights calculated.</p> <p>The corpus was then POS-tagged and lemmatized with <a href="http://www.cis.uni-muenchen.de/~schmid/tools/TreeTagger/">TreeTagger</a>. The list of tags can be found <a href="http://www.cis.uni-muenchen.de/~schmid/tools/TreeTagger/data/french-tagset.html">here</a>. During POS-tagging, each word is replaced with its lemma and is replaced with the syntax `[lemma]_[tag]`. For instance, the sentence "Il a sauté dans sa voiture" will produce this output:</p> <pre><code>il_PRO:PER avoir_VER:pres sauter_VER:pper dans_PRP son_DET:POS voiture_NOM</code></pre> <p><strong>Word2vec training</strong></p> <p>The training was performed using the <a href="https://pypi.org/project/gensim/">gensim</a> Python module (v3.5.0). The skip-grams method was used, with a size of 500, a window size of 5 and a minimum count of 5. This is how the model creation was invoked:</p> <pre><code class="language-python">model = Word2Vec(size=500, window=5, min_count=5, workers=workers, sg=1)</code></pre> <p>Two versions of the model were trained:</p> <ul> <li>frwiki-20181020.treetag.2__2019-01-24_10.41__.s500_w5_skip.word2vec.bin<br> No extra processing was performed on the corpus before training.</li> <li>frwiki-20181020.treetag.2.ngram-pass2__2019-04-08_09.02__.s500_w5_skip.word2vec.bin<br> Where two passes of the 2-gram detection were ran before training. This allows detection of 2-, 3- and 4-grams in the vocabulary.</li> </ul> <p>The files are in the word2vec binary format, so can be used either with the original <a href="https://github.com/tmikolov/word2vec">C implementation of word2vec</a>, or with the Python gensim version (and possibly other libraries that support that format).</p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p>
Wikidata Lemmatization Dataset
<p>The Wikidata Lemmatization Dataset was collected using the following SPARQL query: https://w.wiki/9TwH</p> <p>Languages included in the dataset:</p> <ul> <li>Akkadian : AKK (Q35518)</li> <li>Arabic : AR (Q13955)</li> <li>Czech : CS (Q9056)</li> <li>German : DE (Q188)</li> <li>English : EN (Q1860)</li> <li>French : FR (Q150)</li> <li>Hebrew : HE (Q9288)</li> <li>Hittite : HIT (Q35668)</li> <li>Italian : IT (Q652)</li> <li>Russian : RU (Q7737)</li> <li>Sumerian : SUX (Q36790)</li> <li>Turkish : TR (Q256)</li> </ul> <p>The choice of languages to include have to do with a collection of primary and secondary source documents which we have digitized (OCR) and are using as references for the FactGrid Cuneiform project. The resulting lexemes for each language are shared in CSV with the file names references each language, their Wikidata Q-ids, the number of lexemes at that date, and the date of access (MM_YYYY).</p> <p>The format of each CSV includes the following fields:</p> <ol> <li>lexeme : the Wikidata lexeme id (L-id)</li> <li>lexemeLabel : the label assigned to the lexeme in Wikidata</li> <li>lexical_category : the Wikidata Q-item for the part of speech</li> <li>lexical_categoryLabel : the label assigned to the lexical category (e.g. noun, verb, adjective, etc.)</li> </ol> <p>This dataset will be updated periodically using standard version control.</p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.