Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
3
datasets available to search
ShareScore release 0.9.0
Dataset results
3 results for “manosphere”
Datasets from the paper "Measuring the relationship between the use of typical Manosphere discourse and the engagement of a user with the pick-up artist community"
<p>Datasets from the paper "Measuring the relationship between the use of typical Manosphere discourse and the engagement of a user with the pick-up artist community", presented at the 24th International Conference on Intelligent Data Engineering and Automated Learning (IDEAL).</p> <p>The dataset consists in 2 files:</p> <ul> <li>tweets.csv: This file contains the ids of the tweets and the user that published the tweet.</li> <li>df_users_seeds.csv: This file contains a table were the first field is the user id; the other columns indicate the seed number and if the user follows that seed (1) or not (0)</li> </ul>
Dataset for: The Evolution of the Manosphere Across the Web
<p><strong>The Evolution of the Manosphere Across the Web</strong></p> <p>We make available data related to subreddit and standalone forums from the manosphere.</p> <p>We also make available Perspective API annotations for all posts.</p> <p>You can find the code in <a href="https://github.com/manoelhortaribeiro/manosphere_analysis">GitHub</a>.</p> <p>Please cite this paper if you use this data:</p> <pre><code>@article{ribeiroevolution2021, title={The Evolution of the Manosphere Across the Web}, author={Ribeiro, Manoel Horta and Blackburn, Jeremy and Bradlyn, Barry and De Cristofaro, Emiliano and Stringhini, Gianluca and Long, Summer and Greenberg, Stephanie and Zannettou, Savvas}, booktitle = {{Proceedings of the 15th International AAAI Conference on Weblogs and Social Media (ICWSM'21)}}, year={2021} } </code></pre> <p><strong>1. Reddit data</strong></p> <p>We make available data for forums and for relevant subreddits (56 of them, as described in <code>subreddit_descriptions.csv</code>).<br> These are available, 1 line per post in each subreddit Reddit in <code>/ndjson/reddit.ndjson</code>.<br> A sample for example is:</p> <pre><code>{ "author": "Handheld_Gaming", "date_post": 1546300852, "id_post": "abcusl", "number_post": 9.0, "subreddit": "Braincels", "text_post": "Its been 2019 for almost 1 hour And I am at a party with 120 people, half of them being foids. The last year had been the best in my life. I actually was happy living hope because I was redpilled to the death. \n\nNow that I am blackpilled I see that I am the shortest of all men and that I am the only one with a recessed jaw. \n\nIts over. Its only thanks to my age old friendship with chads and my social skills I had developed in the past year that a lot of men like me a lot as a friend.\n\nNo leg lengthening syrgery is gonna save me. Ignorance was a bliss. Its just horror now seeing that everyone can make out wirth some slin hoe at the party. \n\n I actually feel so unbelivably bad for turbomanlets. Life as an unattractive manlet is a pain, I cant imagine the hell being an ugly turbomanlet is like. I would have roped instsntly if I were one. Its so unfair. \n\nTallcels are fakecels and they all can (and should) suck my cock. \n\nIf I were 17cm taller my life would be a heaven and I would be the happiest man alive. \n\nJust cope and wait for affordable body tranpslants.", "thread": "t3_abcusl" } </code></pre> <p><strong>2. Forums</strong></p> <p>We here describe the <code>.sqlite</code> and <code>.ndjson</code> files that contain the data from the following forums.</p> <pre><code>(avfm) --- https://d2ec906f9aea-003845.vbulletin.net (incels) --- https://incels.co/ (love_shy) --- http://love-shy.com/lsbb/ (redpilltalk) --- https://redpilltalk.com/ (mgtow) --- https://www.mgtow.com/forums/ (rooshv) --- https://www.rooshvforum.com/ (pua_forum) --- https://www.pick-up-artist-forum.com/ (the_attraction) --- http://www.theattractionforums.com/ </code></pre> <p>The files are in folders <code>/sqlite/</code> and <code>/ndjson</code>.</p> <p><strong><code>2.1 .sqlite</code></strong></p> <p>All the tables in the <code>sqlite.</code> datasets follow a very simple <code>{key:value}</code> format.<br> Each <code>key</code> is a thread name (for example <code>/threads/housewife-is-like-a-job.123835/</code>) and each value is a python dictionary or a list.<br> This file contains three tables:</p> <ul> <li> <p><code>idx</code> each key is the relative address to a thread and maps to a post. Each post is represented by a dict:</p> <pre><code>"type": (list) in some forums you can add a descriptor such as [RageFuel] to each topic, and you may also have special types of posts, like sticked/pool/locked posts. "title": (str) title of the thread; "link": (str) link to the thread; "author_topic": (str) username that created the thread; "replies": (int) number of replies, may differ from number of posts due to difference in crawling date; "views": (int) number of views; "subforum": (str) name of the subforum; "collected": (bool) indicates if raw posts have been collected; "crawled_idx_at": (str) datetime of the collection. </code></pre> </li> <li> <p><code>processed_posts</code> each key is the relative address to a thread and maps to a list with posts (in order). Each post is represented by a dict:</p> <pre><code>"author": (str) author's username; "resume_author": (str) author's little description; "joined_author": (str) date author joined; "messages_author": (int) number of messages the author has; "text_post": (str) text of the main post; "number_post": (int) number of the post in the thread; "id_post": (str) unique post identifier (depends), for sure unique within thread; "id_post_interaction": (list) list with other posts ids this post quoted; "date_post": (str) datetime of the post, "links": (tuple) nice tuple with the url parsed, e.g. ('https', 'www.youtube.com', '/S5t6K9iwcdw'); "thread": (str) same as key; "crawled_at": (str) datetime of the collection. </code></pre> </li> <li> <p><code>raw_posts</code> each key is the relative address to a thread and maps to a list with unprocessed posts (in order). Each post is represented by a dict:</p> <pre><code>"post_raw": (binary) raw html binary; "crawled_at": (str) datetime of the collection. </code></pre> </li> </ul> <p><strong><code>2.2 .ndjson</code></strong></p> <p>Each line consists of a json object representing a different comment with the following fields:</p> <pre><code> "author": (str) author's username; "resume_author": (str) author's little description; "joined_author": (str) date author joined; "messages_author": (int) number of messages the author has; "text_post": (str) text of the main post; "number_post": (int) number of the post in the thread; "id_post": (str) unique post identifier (depends), for sure unique within thread; "id_post_interaction": (list) list with other posts ids this post quoted; "date_post": (str) datetime of the post, "links": (tuple) nice tuple with the url parsed, e.g. ('https', 'www.youtube.com', '/S5t6K9iwcdw'); "thread": (str) same as key; "crawled_at": (str) datetime of the collection. </code></pre> <p><strong>3. Perspective</strong></p> <p>We also run each post and reddit post through perspective, the files are located in the <code>/perspective/</code> folder.<br> They are compressed with gzip.<br> One example output</p> <pre><code>{ "id_post": 5200, "hate_output": { "text": "I still can\u2019t wrap my mind around both of those articles about these c\~\~\~s sleeping with poor Haitian Men. Where\u2019s the uproar?, where the hell is the outcry?, the \u201cpig\u201d comments or the \u201ccreeper comments\u201d. F\~\~\~ing hell, if roles were reversed and it was an article about Men going to Europe where under 18 sex in legal, you better believe they would crucify the writer of that article and DEMAND an apology by the paper that wrote it.. This is exactly what I try and explain to people about the double standards within our modern society. A bunch of older women, wanna get their kicks off by sleeping with poor Men, just before they either hit or are at menopause age. F~~~ing unreal, I\u2019ll never forget going to Sweden and Norway a few years ago with one of my buddies and his girlfriend who was from there, the legal age of consent in Norway is 16 and in Sweden it\u2019s 15. I couldn\u2019t believe it, but my friend told me \u201c hey, it\u2019s normal here\u201d . Not only that but the age wasn\u2019t a big different in other European countries as well. One thing i learned very quickly was how very Misandric Sweden as well as Denmark were.", "TOXICITY": 0.6079781, "SEVERE_TOXICITY": 0.53744453, "INFLAMMATORY": 0.7279288, "PROFANITY": 0.58842486, "INSULT": 0.5511079, "OBSCENE": 0.9830818, "SPAM": 0.17009115 } } </code></pre> <p><strong>4. Working with sqlite</strong></p> <p>A nice way to read some of the files of the dataset is using <code>SqliteDict</code>, for example:</p> <pre><code>from sqlitedict import SqliteDict processed_posts = SqliteDict("./data/forums/incels.sqlite", tablename="processed_posts") for key, posts in processed_posts.items(): for post in posts: # here you could do something with each post in the dataset pass </code></pre> <p><strong>5. Helpers</strong></p> <p>Additionally, we provide two <code>.sqlite</code> files that are helpers used in the analyses.<br> These are related to reddit, and not to the forums!<br> They are:</p> <ul> <li> <p><code>channel_dict.sqlite</code> a sqlite where each key corresponds to a subreddit and values are lists of dictionaries users who posted on it, along with timestamps.</p> </li> <li> <p><code>author_dict.sqlite</code> a sqlite where each key corresponds to an author and values are lists of dictionaries of the subreddits they posted on, along with timestamps.</p> </li> </ul> <p>These are used in the paper for the migration analyses.</p> <p><strong>6. Examples and particularities for forums</strong></p> <p>Although we did our best to clean the data and be consistent across forums, this is not always possible. In the following subsections we talk about the particularities of each forum, directions to improve the parsing which were not pursued as well as give some examples on how things work in each forum.</p> <p><strong>6.1 incels</strong></p> <p>Check out an archived version of the <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/screenshots/incels_front_page.png">front page</a>, the <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/screenshots/incels_thread_page.png">thread page</a> and a <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/screenshots/incels_post_page.png">post page</a>, as well as a dump of the data stored for a <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/examples/incels_thread_page_example.txt">thread page</a> and a <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/examples/incels_post_page_example.txt">post page</a>.</p> <ul> <li> <p><em>types</em>: for the incel forums the special types associated with each thread in the <code>idx</code> table are “Sticky”, “Pool”, “Closed”, and the custom types added by users, such as <code>[LifeFuel]</code>. These last ones are all in brackets. You can see some examples of these in the on the example thread page.</p> </li> <li> <p><em>quotes</em>: quotes in this forum were quite nice and thus, all quotations are deterministic.</p> </li> </ul> <p><strong>6.2 LoveShy</strong></p> <p>Check out an archived version of the <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/screenshots/love_shy_front_page.png">front page</a>, the <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/screenshots/love_shy_thread_page.png">thread page</a> and a <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/screenshots/love_shy_thread_page.png">post page</a>, as well as a dump of the data stored for a <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/examples/love_shy_thread_page_example.txt">thread page</a> and a <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/examples/love_shy_post_page_example.txt">post page</a>.</p> <ul> <li> <p><em>types</em>: no types were parsed. There are some rules in the forum, but not significant.</p> </li> <li> <p><em>quotes</em>: quotes were obtained from exact text+author match, or author match + a jaccard similarity of more than 0.9 in the tokens of the text split on spaces.</p> </li> <li> <p><em>other</em>: this forum apparently deleted a bunch of posts (or hid them from new users).</p> </li> </ul> <p><strong>6.3 MGTOW</strong></p> <p>Check out an archived version of the <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/screenshots/mgtow_front_page.png">front page</a>, the <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/screenshots/mgtow_thread_page.png">thread page</a> and a <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/screenshots/mgtow_thread_page.png">post page</a>, as well as a dump of the data stored for a <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/examples/mgtow_thread_page_example.txt">thread page</a> and a <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/examples/mgtow_post_page_example.txt">post page</a>.</p> <ul> <li> <p><em>types</em>: no types were parsed. There are some rules in the forum, but not significant.</p> </li> <li> <p><em>quotes</em>: quotes in this forum were quite nice and thus, all quotations are deterministic.</p> </li> <li> <p><em>other</em>: collecting users for this forum was hard because info like number of threads and join date do not show on the threads. Thus there was a 2-step collection. Some users may be missing because of this.</p> </li> </ul> <p><strong>6.4 A Voice for Men</strong></p> <p>Check out an archived version of the <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/screenshots/avfm_front_page.png">front page</a>, the <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/screenshots/avfm_thread_page.png">thread page</a> and a <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/screenshots/avfm_thread_page.png">post page</a>, as well as a dump of the data stored for a <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/examples/avfm_thread_page_example.txt">thread page</a> and a <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/examples/avfm_post_page_example.txt">post page</a>.</p> <ul> <li> <p><em>types</em>: “sticky” is the only existing type.</p> </li> <li> <p><em>quotes</em>: quotes in this forum were quite nice and thus, all quotations are deterministic.</p> </li> </ul> <p><strong>6.5 Red Pill Talk (formerly <a href="http://sluthate.com">sluthate.com</a>)</strong></p> <p>Check out an archived version of the <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/screenshots/avfm_front_page.png">front page</a>, the <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/screenshots/avfm_thread_page.png">thread page</a> and a <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/screenshots/avfm_thread_page.png">post page</a>, as well as a dump of the data stored for a <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/examples/avfm_thread_page_example.txt">thread page</a> and a <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/examples/avfm_post_page_example.txt">post page</a>.</p> <ul> <li> <p><em>types</em>: no types were parsed.</p> </li> <li> <p><em>quotes</em>: quotes were obtained from exact text+author match, or author match + a jaccard similarity of more than 0.9 in the tokens of the text split on spaces.</p> </li> </ul> <p><strong>6.6 The Attraction</strong></p> <p>Check out an archived version of the <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/screenshots/the_attraction_front_page.png">front page</a>, the <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/screenshots/the_attraction_thread_page.png">thread page</a> and a <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/screenshots/the_attraction_thread_page.png">post page</a>, as well as a dump of the data stored for a <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/examples/the_attraction_thread_page_example.txt">thread page</a> and a <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/examples/the_attraction_post_page_example.txt">post page</a>.</p> <ul> <li> <p><em>types</em>: “sticky” is the only existing type.</p> </li> <li> <p><em>quotes</em>: quotes in this forum were quite nice and thus, all quotations are deterministic.</p> </li> <li> <p><em>other</em>: there is the possibility to parse stuff like age and location, but this was not done.</p> </li> </ul> <p><strong>6.7 PUA Forum</strong></p> <p>Check out an archived version of the <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/screenshots/pua_forum_front_page.png">front page</a>, the <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/screenshots/pua_forum_thread_page.png">thread page</a> and a <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/screenshots/pua_forum_thread_page.png">post page</a>, as well as a dump of the data stored for a <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/examples/pua_forum_thread_page_example.txt">thread page</a> and a <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/examples/pua_forum_post_page_example.txt">post page</a>.</p> <ul> <li> <p><em>types</em>: There are several types parsed: Sticky, Announcement, General Announcement, Locked, to name a few.</p> </li> <li> <p><em>quotes</em>: quotes were obtained from exact text match, or a jaccard similarity of more than 0.9 in the tokens of the text split on spaces. There are actually some problems with quotes with other quotes inside (check example .txt).</p> </li> </ul> <p><strong>6.8 Rooshv</strong></p> <p>Check out an archived version of the <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/screenshots/rooshv_front_page.png">front page</a>, the <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/screenshots/rooshv_thread_page.png">thread page</a> and a <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/screenshots/pua_forum_thread_page.png">post page</a>, as well as a dump of the data stored for a <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/examples/rooshv_thread_page_example.txt">thread page</a> and a <a href="https://github.com/manoelhortaribeiro/manosphere_analysis/blob/master/data/forums/examples/rooshv_post_page_example.txt">post page</a>.</p> <ul> <li> <p><em>types</em>: Two types parsed: <code>sticky</code>, <code>announcement</code>.</p> </li> <li> <p><em>quotes</em>: quotes in this forum were quite nice and thus, all quotations are deterministic.</p> </li> <li> <p><em>other</em>: lots of posts have some problem with dates, which is quite annoying (~5%). Some threads could not be captured also, website behaves a bit weird. Some don’t exist.</p> </li> </ul>
Brazilian manosphere of Telegram
<p>Brazilian manosphere in Telegram. Data obtained using an snowball approach.</p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.