Preprocessed Penn Tree Bank
<p><a href="https://catalog.ldc.upenn.edu/LDC99T42">Penn Tree Bank</a> is originally a corpus of English sentences with linguistic structure annotations. This is a variant originally distributed by Mikolov <a href="http://www.fit.vutbr.cz/~imikolov/rnnlm/simple-examples.tgz">here</a> and Zaremba <a href="https://github.com/wojzaremba/lstm">here</a> which omits the annotation and splits the dataset into raining, validation, and test.</p> <p>The dataset consists of <strong>929k</strong> <strong>training</strong> words, <strong>73k</strong> <strong>validation</strong> words, and <strong>82k</strong> test words. As part of the preprocessing, words were <strong>lower-cased</strong>, <strong>numbers</strong> were replaced with <strong>N</strong>, newlines were replaced with <strong><eos></strong>, and all other punctuation was removed. The vocabulary is the most frequent <strong>10k</strong> words with the rest of the tokens replaced by an <strong><unk></strong> token.</p> <p> </p>
ShareScore
32/100
Overall dataset sharing score
Score breakdown
These five areas show where the dataset supports — or may limit — practical reuse.
- Stewardship
- 8
- Harmonization
- 4
- Access
- 12
- Reuse readiness
- 8
- Engagement
- 0
Topics
Dataset details
- Repository
- zenodo
- Published
- June 25, 2020
- License
- other-open
- DOI
- 10.5281/zenodo.3910020
Scored with release 0.9.0
ScienceDex for faculty
Your academic CV and NIH biosketch
Make your CV in your school's format and your NIH biosketch from it, then download both as Word files.