Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
4
datasets available to search
ShareScore release 0.9.0
Dataset results
4 results for “Wikipedia references”
Dataset of first appearances of the scholarly bibliographic references on English Wikipedia articles as of 1 March 2017 and as of 1 October 2021
<p><strong>Abstract</strong></p> <p>We developed a methodology to detect the oldest scholarly reference added to Wikipedia articles by which a certain paper is uniquely identifiable as the "first appearance of the scholarly reference." We identified the first appearances of 923,894 scholarly references (611,119 unique DOIs) in180,795 unique pages on English Wikipedia as of March 1, 2017, and stored them in the dataset. Moreover, we assessed the precision of the dataset, which was and it was a high precision regardless of the research field. In this version, it is available not only the dataset of English Wikipedia as of March 1, 2017, but also English Wikipedia as of October 1, 2021, generated by using the same methodology.</p> <p> </p> <p><strong>Data Records</strong></p> <p>The data format of the dataset is JSON lines, where each line is a single record. In this dataset, we detected the first appearance of each scholarly reference added to Wikipedia articles. If there are multiple references corresponding to the same paper on the same page, only the oldest one is collected. Sample of the record is the following.</p> <ul> <li>doi -- DOI corresponding to the paper (String), e.g., "10.1006/anbe.1996.0497"</li> <li>paper_type -- Document type of the paper (String), e.g., "journal-article"</li> <li>paper_container_title -- Journal title, book title, or proceedings title (Array of String), e.g., ["Animal Behaviour"]</li> <li>paper_publisher -- Publisher name (String), e.g., "Elsevier BV"</li> <li>paper_title -- Paper title (Array of String), e.g., ["Push or pull: an experimental study on imitation in marmosets"]</li> <li>paper_published_year -- Published year (String), e.g., "1997"</li> <li>paper_issue -- Issue number (String), e.g., "4"</li> <li>paper_volume -- Volume number (String), e.g., "54"</li> <li>paper_page -- Page numbers (String), e.g., "817-831"</li> <li>paper_author -- Authors information consisted of given and family names, sequences (order in author names), and affiliations (Array of JSON), e.g., [{"given":"THOMAS", "family":"BUGNYAR", "sequence":"first", "affiliation":[]}, {"given":"LUDWIG", "family":"HUBER", "sequence":"additional", "affiliation":[]}]</li> <li>issn -- ISSN related to the paper (Array of String), e.g., ["0003-3472"]</li> <li>research_field -- Research fields from ESI categories (Array of String), e.g., ["PLANT & ANIMAL SCIENCE"]</li> <li>page_id -- Page id (String), e.g., "577858"</li> <li>page_title -- Page title (String), e.g., "Imitation"</li> <li>revision_id -- Revision id (String), e.g., "203309031"</li> <li>revision_timestamp -- Revision timestamp (String), e.g., "2008-04-04 15:54:09 UTC"</li> <li>revision_comment -- Revision comment (edit summary) (String), e.g., "/* Animal Behaviour */"</li> <li>editor_name -- Wikipedia editor's name (String), e.g., "Nicemr"</li> <li>editor_type -- Type of the editor (String), e.g., "User"</li> </ul> <p><strong>References</strong></p> <ul> <li>Kikkawa, J., Takaku, M. & Yoshikane, F. "Dataset of first appearances of the scholarly bibliographic references on Wikipedia articles", Scientific Data, Vol. 9, Article number 85, pp. 1-11, 2022. <a href="https://doi.org/10.1038/s41597-022-01190-z">https://doi.org/10.1038/s41597-022-01190-z</a>.</li> </ul> <p><strong>FUNDING</strong></p> <ul> <li>JSPS KAKENHI Grant Number <a href="https://kaken.nii.ac.jp/en/grant/KAKENHI-PROJECT-20K12543">JP20K12543</a></li> <li>JSPS KAKENHI Grant Number <a href="https://kaken.nii.ac.jp/en/grant/KAKENHI-PROJECT-21K21303">JP21K21303</a></li> </ul>
英語版Wikipediaにおける学術文献の参照記述の初出時点データセット (Dataset of First Appears of Scholarly References on English Wikipedia)
<p>概要</p> <p>筑波大学大学院図書館情報メディア研究科に提出準備中の博士論文「Wikipediaにおける学術文献の参照記述に関する研究」の付録とする予定のデータセットです。作者は<a href="https://researchmap.jp/jir_o">吉川次郎</a>です。</p> <p>ごく簡単な説明</p> <ul> <li>博論本体の付録としてドキュメントを書きました (そのドラフトは<a href="https://www.dropbox.com/s/c8h0hkkbu2wb4jm/PhDThesis_Appendix.pdf?dl=0">こちら</a>)</li> <li>英語版Wikipedia上の学術文献の参照記述に関するデータセットです <ul> <li>dataset_doi_links はDOIリンクを対象として特定・抽出し、研究分野の紐付けを行ったものです (詳細は関連論文の1番)</li> <li>dataset_refs はdataset_doi_linksの参照記述群を対象に、誰が、いつ追加したのか? の特定を行ったものです (詳細は関連論文の2番)</li> </ul> </li> <li>いまのところ、ドキュメントは日本語のみです。英語への対応等は今後の課題です;</li> </ul> <p>関連論文</p> <ol> <li>吉川次郎; 高久雅生; 芳鐘冬樹: 「DOIリンクに基づくWikipedia上の参照記述における編集者の分析」, 情報知識学会誌, Vol. 30, No. 1, pp. 21--41, 2020. <a href="https://doi.org/10.2964/jsik_2020_004">https://doi.org/10.2964/jsik_2020_004</a></li> <li>吉川次郎; 高久雅生; 芳鐘冬樹: 「Wikipediaに学術文献の参照記述を追加する編集の特定手法」, 情報知識学会誌, Vol. 30, No. 3,2020 (全20ページ,採録決定).<a href="https://doi.org/10.2964/jsik_2020_033">https://doi.org/10.2964/jsik_2020_033</a></li> <li>吉川次郎; 高久雅生; 芳鐘冬樹: 「Wikipedia上の学術文献の参照記述の追加に関する時系列分析」(投稿中)</li> </ol>
External References of English Wikipedia (ref-wiki-en)
<p><strong>External References of English Wikipedia </strong>(<strong>ref-wiki-en</strong>) is a corpus of the plain-text content of 2,475,461 external webpages linked from the reference section of articles in English Wikipedia. Specifically:</p> <ul> <li>32,329,989 external reference URLs were extracted from a <a href="https://zenodo.org/record/3483254">2018 HTML dump of English Wikipedia</a>. Removing repeated and ill-formed URLs yielded 23,036,318 unique URLs.</li> <li>These URLs were filtered to remove file extensions for unsupported formats (videos, audio, etc.), yielding 17,781,974 downloadable URLs. The URLs were loaded into <a href="http://nutch.apache.org/">Apache Nutch</a> and continuously downloaded from August 2019 to December 2019, resulting in 2,475,461 successfully downloaded URLs. Not all URLs could be accessed. The order in which URLs were accessed was determined by Nutch, which partitions URLs by <a href="https://docs.oracle.com/javase/7/docs/api/java/net/URL.html#getHost()">host</a> and then randomly chooses amongst the URLs for each host.</li> <li>The content of these webpages were indexed in <a href="https://lucene.apache.org/solr/">Apache Solr</a> by Nutch. From Solr we extracted a JSON dump of the content.</li> <li>Many URLs offer a redirect; unfortunately Nutch does not index redirect information. This means that connecting the Wikipedia article (with the pre-direct link) to the downloaded webpage (at the post-redirect link) was complicated. However, by inspecting the order of download in the Nutch log files, we managed to recover links for 2,058,896 documents (83%) from their original Wikipedia article(s).</li> <li>We further managed to associate 3,899,953 unique Wikidata items with at least one external reference webpage in the corpus.</li> </ul> <p>The ref-en-wiki corpus is incomplete, i.e., we did not attempt to download all reference URLs for English Wikipedia. We thus also collect a smaller complete corpus for the external references of 5,000 Wikipedia articles (<strong>ref-wiki-en-5k</strong>). We sampled from 5 ranges of Wikidata items: Q1-10000, Q10001-100000, Q100001-1000000, Q1000001-10000000, and Q10000001-100000000. From each range we sampled 1000 items. We then scraped the external reference URLs for the Wikipedia article corresponding to these items and downloaded them. The resulting corpus contains 37,983 webpages.</p> <p>Each line of the corpus (ref-wiki-en, ref-wiki-en-5k) encodes the webpage of an external reference in JSON format. Specifically, we provide:</p> <ul> <li><em>tstamp</em>: When the webpage was accessed</li> <li><em>host</em>: The domain (FQDN post-redirect) from which the webpage was retrieved.</li> <li><em>title</em>: The title (meta) of the document</li> <li><em>url</em>: The URL (post-redirect) of the webpage</li> <li><em>Q</em>: The Q-code identifiers of the Wikidata items whose corresponding Wikipedia article is confirmed to link to this webpage.</li> <li><em>content</em>: A plain-text encoding of the content of the webpage.</li> </ul> <p>Below we provide an abbreviated example of a line from the corpus:</p> <pre><code>{"tstamp":"2019-09-26T01:22:43.621Z","host":"geology.isu.edu","title":"Digital Geology of Idaho - Basin And Range","url":"http://geology.isu.edu/Digital_Geology_Idaho/Module9/mod9.htm","Q":[810178],"content":"Digital Geology of Idaho - Basin And Range\n1 - Idaho Basement Rock\n2 - Belt Supergroup\n3 - Rifting & Passive Margin\n4 - Accreted Terranes\n5 - Thrust Belt\n6 - Idaho Batholith\n7 - North Idaho & Mining\n8 - Challis Volcanics\n9 - Basin and Range\n10 - Columbia River Basalts\n11 - SRP & Yellowstone\n12 - Pleistocene Glaciation\n13 - Palouse & Lake Missoula\n14 - Lake Bonneville Flood\n15 - Snake River Plain Aquifer\nBasin and Range Province - Teritiary Extension\nGeneral geology of the Basin and Range Province\nMechanisms of Basin and Range faulting\nIdaho Basin and Range south of the Snake River Plain\nIdaho Basin and Range north of the Snake River Plain\nLocal areas of active and recent Basin & Range faulting: Borah Peak\nPDF Slideshows: North of SRP , South of SRP , Borah Earthquake\nFlythroughs: Teton Valley , Henry's Fork , Big Lost River , Blackfoot , Portneuf , Raft River Valley , Bear River , Salmon Falls Creek , Snake River , Big Wood River\nVocabulary Words\nthrust fault\nBasin and Range\nSnake River Plain\nhalf-graben\ntransfer zone\n \n \n \n \nFly-throughs\nGeneral geology of the Basin and Range Province\nThe Basin and Range Province generally includes most of eastern California, eastern Oregon, eastern Washington, Nevada, western Utah, southern and western Arizona, and southeastern Idaho. ..."},</code></pre> <p>A summary of the files we make available:</p> <ul> <li><strong>ref-wiki-en.json.gz</strong>: 2,475,461 external reference webpages (JSON format)</li> <li><strong>ref-wiki-en_urls.txt.gz</strong>: 23,036,318 unique raw links to external references (plain-text format)</li> <li><strong>ref-wiki-en-5k.json.gz</strong>: 37,983 external reference webpages (JSON format)</li> <li><strong>ref-wiki-en-5k_urls.json.gz</strong>: 70,375 unique raw links to external references (plain-text format)</li> <li><strong>ref-wiki-en-5k_Q.txt.gz</strong>: 5,000 Wikidata Q identifiers forming the 5k dataset (plain-text format)</li> </ul> <p>Further details can be found in the publication:</p> <ul> <li><em>Suggesting References for Wikidata Claims based on Wikipedia's External References</em>. Paolo Curotto, Aidan Hogan. Wikidata Workshop @ISWC 2020.</li> </ul> <p>Further material relating to this publication (including code for a proof-of-concept interface) is <a href="http://aidanhogan.com/wikiref/">also available</a>.</p>
Individual Edit Histories of All References in the English Wikipedia
<p><strong>This dataset includes the historical versions of all individual references per article in the English Wikipedia. Each reference object also contains information about its original creating editor, editors implementing changes to it, and timestamps of all actions (creations, modifications, deletions, and reinsertions) that were applied to the reference. Each historical version of a reference is represented as a list of tokens </strong> <strong>(≈ words)</strong><strong>, where each token has an individual creator and change history.</strong></p> <p>The extraction process was meticulously vetted through crowdsourcing evaluations, assuring very high accuracy in contrast to standard textual difference algorithms. The dataset includes references that were created with the "<ref>" tag until June 2019. It contains 55,503,998 references with 164,530,374 actions. These references were found in 4,690,046 Wikipedia articles.</p> <p>The dataset consists of JSON files where each article's page ID (here: article_id) is used as a file name. Each file is represented as a list of “References”. Each reference is a dictionary with the following keys:</p> <ul> <li>"<strong>first_rev_id</strong>" type: Integer, first revision where the reference was inserted (the same value is represented in “ins” as the first element of the list and in "rev_id" of the first element in the "change_sequence"),</li> <li>"<strong>first_hash_id</strong>" type: String, the hash value of the first version of token_id (from WikiWho<a href="#wikiwholine"><sup>1</sup></a>, see below) list of the reference (the same value is represented as "hash_id" of the first element in the "change_sequence"),</li> <li>"<strong>first_editor_id</strong>" type: String, user_id or IP address of the first revision where the reference was inserted (the same value is represented as "editor_id" of the first element in the "change_sequence",</li> <li>"<strong>deleted</strong>" type: Boolean, an indicator if the reference exists in the last available revision,</li> <li>"<strong>ins</strong>" type: List of Integers, list of revisions where the reference was inserted (includes the first revision mentioned as "first_rev_id"),</li> <li>"<strong>ins_editor</strong>" type: List of Strings, list of user_id or IP addresses of editors where the reference was inserted (includes the first user mentioned as "first_editor_id"),</li> <li>"<strong>del</strong>" type: List of Integers, list of revisions where the reference was deleted from the article or reference was modified in a way that less than 25% of tokens remained,</li> <li>"<strong>del_editor</strong>“ type: List of Strings, list of user_id or IP addresses of editors where the reference was deleted or reference was modified in a way that less than 25% of tokens remained,</li> <li>"<strong>modif</strong>" type: List of Integers, list of revisions where the reference was modified, or reinserted with modification,</li> <li>"<strong>hashes</strong>": type: List of Strings, list of hash values of all versions of references,</li> <li>"<strong>first_rev_time</strong>": type: DateTime, the timestamp when the reference was created (the same value is represented in "ins_time” as the first element of the list and in "time" of the first element in the "change_sequence"),</li> <li>"<strong>ins_time</strong>" type: List of DateTime, list of timestamps when the reference was inserted or reinserted,</li> <li>"<strong>del_time</strong>" type: List of DateTime, list of timestamps when the reference was deleted,</li> <li>"<strong>change_sequence</strong>" type: List of dictionaries, with information about tokens, editors and revisions where the reference was modified (the first element representing the first revision where the reference was inserted), where: <ul> <li>"<strong>hash_id</strong>" type: String, the hash value of the token_id (WikiWho<a href="#wikiwholine"><sup>1</sup></a>) list of the reference version,</li> <li>"<strong>rev_id</strong>" type: Integer, the revision number of the particular version of the reference,</li> <li>"<strong>editor_id</strong>" type: String, user_id or IP address of the revision editor,</li> <li>"<strong>time</strong>" type: DateTime, the timestamp when of this particular version of the reference,</li> <li>"<strong>tokens</strong>" type: List of Strings, ordered list of tokens (created by WikiWho<a href="#wikiwholine"><sup>1</sup></a>) that represents the particular version of the reference (the list has the same length as "token_editors"),</li> <li>"<strong>token_editors</strong>" type: List of Strings, ordered list of user_ids or IP addresses of editors that were first who added the corresponding token (see "tokens") to Wikipedia article.</li> </ul> </li> </ul> <p><sup>1</sup> WikiWho is a text mining algorithm to extract changes to tokens from Wikipedia revisions. Each token is assigned a unique ID. More information: <a href="https://www.wikiwho.net/#technical_details">https://www.wikiwho.net/#technical_details</a></p> <p>GitHub Repository with Python example code on how to process data and extract document identifiers: <a href="https://github.com/gesiscss/wikipedia_references">https://github.com/gesiscss/wikipedia_references</a></p> <p>To run the code at GESIS Notebook follow the link: <a href="https://notebooks.gesis.org/binder/v2/gh/gesiscss/wikipedia_references/master">https://notebooks.gesis.org/binder/v2/gh/gesiscss/wikipedia_references/master</a></p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.