Skip to main content
zenodoopen

The Unreasonable Ineffectiveness of Nucleus Sampling on Mitigating Text Memorization

<p>We present <strong>OpenMemText</strong> a diagnostic dataset with a known distribution of duplicates that gives us some control over the likelihood of memorization of certain parts of the training data. Given this diagnostic dataset, we analyse the text memorization behavior of large language models (LLMs) when subjected to nucleus sampling.</p> <p>Stochastic decoding methods like nucleus sampling are typically applied to overcome issues such as monotonous and repetitive text generation, which are often observed with maximization-based decoding techniques. We hypothesize that nucleus sampling might also reduce the occurrence of memorization patterns, because it could lead to the selection of tokens outside the memorized sequence.</p> <p>See https://github.com/lukaborec/memorization-nucleus-sampling for more information.</p>

ShareScore

32/100

Overall dataset sharing score

Score breakdown

These five areas show where the dataset supports — or may limit — practical reuse.

Stewardship
4
Harmonization
4
Access
16
Reuse readiness
8
Engagement
0

Dataset details

Repository
zenodo
Published
August 13, 2024
License
cc-by-4.0
DOI
10.5281/zenodo.13318541

Scored with release 0.9.0

ScienceDex for faculty

Your academic CV and NIH biosketch

Make your CV in your school's format and your NIH biosketch from it, then download both as Word files.