Skip to main content
zenodoopen

Preprocessed Penn Tree Bank

<p><a href="https://catalog.ldc.upenn.edu/LDC99T42">Penn Tree Bank</a> is originally a corpus of English sentences with linguistic structure annotations. This is a variant originally distributed by Mikolov <a href="http://www.fit.vutbr.cz/~imikolov/rnnlm/simple-examples.tgz">here</a> and Zaremba <a href="https://github.com/wojzaremba/lstm">here</a> which omits the annotation and splits the dataset into raining, validation, and test.</p> <p>The dataset consists of <strong>929k</strong> <strong>training</strong> words, <strong>73k</strong> <strong>validation</strong> words, and <strong>82k</strong> test words. As part of the preprocessing, words were <strong>lower-cased</strong>, <strong>numbers</strong> were replaced with <strong>N</strong>, newlines were replaced with <strong>&lt;eos&gt;</strong>, and all other punctuation was removed. The vocabulary is the most frequent <strong>10k</strong> words with the rest of the tokens replaced by an <strong>&lt;unk&gt;</strong> token.</p> <p>&nbsp;</p>

ShareScore

32/100

Overall dataset sharing score

Score breakdown

These five areas show where the dataset supports — or may limit — practical reuse.

Stewardship
8
Harmonization
4
Access
12
Reuse readiness
8
Engagement
0

Topics

Dataset details

Repository
zenodo
Published
June 25, 2020
License
other-open
DOI
10.5281/zenodo.3910020

Scored with release 0.9.0

ScienceDex for faculty

Your academic CV and NIH biosketch

Make your CV in your school's format and your NIH biosketch from it, then download both as Word files.