Find research datasets worth reusing

Cross-mentions between 4chan and Reddit

<p>The included datasets are at the basis of the article "No space for Reddit spacing: Mapping the reflexive relationship between groups on 4chan and Reddit", published in Social Media + Society. They include cross-mentions between 4chan and Reddit, as well as various metrics associated to these cross-references.</p><p>The timeframe ranges from the earliest date available (for Reddit: June 2006; for 4chan/b/: April 2006; for 4chan/pol/: December 2013) and ends in January 2023 (except for the 4chan/b/ dataset, which ends in December 2008).</p><p>The datasets specifically entail the following:</p><h3><strong>1. Cross-mentions from Reddit to 4chan</strong></h3><p><i>reddit-mentions-to-4chan.csv</i>&nbsp;</p><p>I used the Pushshift API's search endpoint to fetch Reddit comments (so no opening posts) with the keyword "4chan" (note: this Pushshift functionality is now deprecated). I also used a rudimentary filter to remove posts by bots, specifically by 1) deleting posts from every account that had "bot" or "auto" in the username and 2) removing all posts by authors with 100 or more contributions and which I manually identified as automated accounts.</p><p>I removed URL-only cross-references, i.e. posts that only mentioned "://boards.4chan.org" or "://boards.4channel.org" without another 4chan-reference/</p><p>This resulted in 2,638,621 "4chan" references across Reddit.</p><h3><strong>2. Cross-mentions from 4chan/pol/ to Reddit</strong></h3><p><i>4chan-pol_mentions-of-reddit.csv</i></p><p>With a complete dataset of /pol/ collected through <a href="https://4cat.nl">4CAT</a>, I queried for "reddit" or the common synonym "plebbit", capital-insensitive, with post- and suffixes allowed (e.g. "Redditor").</p><p>I removed URL-only cross-references, i.e. posts that only mentioned "://reddit.com/", "www.reddit.com/", or "i.reddit.com/". without another Reddit-reference/</p><p>This resulted in 1,640,273 "Reddit" references on /pol/.</p><h3><strong>3. Cross-mentions from 4chan/b/ to Reddit</strong></h3><p><i>4chan-b_mentions-to-reddit.csv</i></p><p>I extracted five million posts from <a href="https://archive.org/details/4chan_threads_archive_10_billion">Jason Scott's 4chan/b/ dump</a>. I then queried for "reddit" or the common synonym "plebbit", capital-insensitive, with post- and suffixes allowed (e.g. "Redditor").</p><p>I removed URL-only cross-references, i.e. posts that only mentioned "://reddit.com/", "www.reddit.com/", or "i.reddit.com/". without another Reddit-reference/</p><p>This resulted in 1,287 "Reddit" references on /b/.</p><p>See <a href="https://oilab.eu/an-overview-of-4chan-b-archives-what-is-left-of-the-internets-cesspool">Hagen (2020)</a> for more information on the 4chan/b/ dataset.</p><h3><strong>4. Cross-mention metrics</strong></h3><p><i>cross-mention-metrics.xlsx</i></p><p>I extracted the following metrics from the datasets above:</p><p><i>4.1 The total number of cross-mentions, absolute and relative, per month</i><br>This simply used the monthly counts from datasets 1 and 2.</p><p><i>4.2 The most mentioned subreddits on /pol/, per year</i><br>Using the regular expression:<i> r\/[a-zA-Z_]</i></p><p><i>4.3 Subreddits that mention 4chan most often, per year</i></p><p><i>4.4 4chan boards mentioned across Reddit, per month</i></p><p><i>4.5 4chan boards mentioned by subreddits</i></p><p>I counted every subreddit- or board-mention <i>per post</i> instead of total occurrences.</p><p>For 4.4 and 4.5, I used the following regular expression to extract 4chan board names:</p><p><i>(\s|^|4chan)\/(a|b|c|d|e|f|g|gif|h|hr|k|m|o|p|t|v|vg|vm|vmg|vr|vrpg|vst|w|wg|i|ic|r9k|s4s|vip|qa|cm|hm|lgbt|y|3|aco|adv|an|bant|biz|cgl|ck|co|diy|fa|fit|gd|hc|his|int|jp|lit|mlp|mu|n|news|out|po|pol|pw|qst|sci|soc|sp|tg|toy|trv|tv|vp|vt|wsg|wsr|x|xs|new)\/(\s|$)</i></p><p>I also omitted 4chan's /r/, /u/, and /s/ boards; despite their small scale, they appeared as false positives due to their unrelated vernacular meaning on Reddit (e.g. /u/ as a username prefix).</p><p>4.5 was also transformed and included as a Gephi network file (<i>subreddit-board-mentions.gephi</i>).</p><p>Lastly, I also included:</p><p>4.6 The <i>total amount of posts</i> on 4chan and Reddit</p><p>This was used to calculate 4.1. It uses <a href="https://files.pushshift.io">Pushshift's database statistics</a> (which as of Nov. 2023 requires a login; see <a href="https://pastebin.com/McS2DSNz">this Pastebin</a> for an alternative) and metrics of total 4chan post counts from <a href="https://4stats.io">4stats.io</a>.</p><p>Each of these metrics has their own corresponding tab in the Excel file.</p><h3><strong>5. Co-words of "4chan" and "reddit" in the cross-mentions</strong></h3><p><i>co-words.xslx</i></p><p>Using datasets 1, 2, and 3, I extracted the top ten words appearing directly next to "4chan" on Reddit, and next to "Reddit" on 4chan, <i>per year</i>.</p><p>I first pre-processed the text, which involved tokenisation, filtering of unwanted text elements like URLs, stop word removal (I whitelisted <i>back</i>), and lemmatisation.</p><p>For the co-word extraction I used a window size of two. I excluded a range of semantically uninteresting words or commonly used hate speech terms prevalent throughout 4chan.</p><h3><strong>6. Annotated cross-mentions between Reddit and 4chan/pol/ in September 2014</strong></h3><p><i>annotations_4chanpol-2014.csv</i><br><i>annotations_reddit-2014-kotakuinaction-anonimised.csv</i><br><i>annotations_reddit-2014-tumblrinaction-anonimised.csv</i></p><p>I extracted cross-mentions from /pol/ to Reddit and from Reddit to 4chan in September 2014 for close-reading and annotation.</p><p>__&nbsp;</p><p>The author names are removed for all datasets.</p>