Blog-1K

<p>The Blog-1K corpus is a redistributable authorship identification testbed for contemporary English prose.&nbsp;It has 1,000 candidate authors, 16K+ posts, and a pre-defined data split (train/dev/test proportional to ca. 8:1:1).&nbsp;It is a subset of the <a href="http://www.kaggle.com/datasets/rtatman/blog-authorship-corpus">Blog Authorship Corpus</a>&nbsp;from Kaggle. The MD5 for Blog-1K is &#39;0a9e38740af9f921b6316b7f400acf06&#39;.</p> <p>1. Preprocessing</p> <p>We first filter out texts shorter than 1,000 characters. Then we select one thousand&nbsp;authors whose writings meet the following criteria:<br> - accumulatively at least 10,000 characters,<br> - accumulatively at most 49,410 characters,<br> - accumulatively at least 16 posts,<br> - accumulatively at most 40 posts, and&nbsp;<br> - each text has at least 50 function words found in the Koppel512 list (to filter out non-English prose).</p> <p>Blog-1K has three columns: &#39;id&#39;, &#39;text&#39;, and &#39;split&#39;, where &#39;id&#39;&nbsp;corresponds to its parent corpus.</p> <p>2. Statistics</p> <p>Its creation and statistics can be found in <a href="https://codeberg.org/haining/Blog-1K/src/branch/main/blog-1k_generation_and_stats.ipynb">the Jupyter Notebook</a>.</p> <table> <tbody> <tr> <td>Split</td> <td># Authors</td> <td># Posts</td> <td># Characters</td> <td>Avg. Characters Per Author (Std.)</td> <td>Avg. Characters Per Post (Std.)</td> </tr> <tr> <td>Train</td> <td>1,000</td> <td>16,132</td> <td>30,092,057</td> <td>30,092 (5,884)</td> <td>1,865 (1,007)</td> </tr> <tr> <td>Validation</td> <td>935</td> <td>2,017</td> <td>3,755,362</td> <td>4,016 (2,269)</td> <td>1,862 (999)</td> </tr> <tr> <td>Test</td> <td>924</td> <td>2,017</td> <td>3,732,448</td> <td>4,039 (2,188)</td> <td>1,850 (936)</td> </tr> </tbody> </table> <p><br> 3. Usage</p> <pre><code class="language-python">import pandas as pd df = pd.read_csv('blog1000.csv.gz', compression='infer') # read in training data train_text, train_label = zip(*df.loc[df.split=='train'][['text', 'id']].itertuples(index=False))</code></pre> <p>&nbsp;</p> <p>4. License<br> All the materials is licensed under the ISC License.</p> <p><br> 5. Contact<br> Please contact <a href="mailto:hw56@indiana.edu">its maintainer</a>&nbsp;for questions.</p>