Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
6
datasets available to search
ShareScore release 0.9.0
Dataset results
6 results for “text reuse”
Webis Wikipedia Text Reuse Corpus 2018 (Webis-Wikipedia-Text-Reuse-18)
<p>The Wikipedia Text Reuse Corpus 2018 (Webis-Wikipedia-Text-Reuse-18) containing text reuse cases extracted from within Wikipedia and in between Wikipedia and a sample of the Common Crawl.</p> <p>The corpus has following structure:</p> <ul> <li>wikipedia.jsonl.bz2: Each line, representing a Wikipedia article, contains a json array of article_id, article_title, and article_body</li> <li>within-wikipedia-tr-01.jsonl.bz2: Each line, representing a text reuse case, contains a json array of s_id (source article id), t_id (target article id), s_text (source text), t_text (target text)</li> <li>within-wikipedia-tr-02.jsonl.bz2: Each line, representing a text reuse case, contains a json array of s_id (source article id), t_id (target article id), s_text (source text), t_text (target text)</li> <li>preprocessed-web-sample.jsonl.xz: Each line, representing a web page, contains a json object of d_id, d_url, and content</li> <li>without-wikipedia-tr.jsonl.bz2: Each line, representing a text reuse case, contains a json array of s_id (Wikipedia article id), d_id (web page id), s_text (article text), d_content (web page content)</li> </ul> <p>The datasets were extracted in the work by Alshomary et al. 2018 that aimed to study the text reuse phenomena related to Wikipedia at scale. A pipeline for large scale text reuse extraction was developed and used on Wikipedia and the CommonCrawl.</p>
KITAB Text Reuse Data
<p><a href="http://kitab-project.org/">KITAB</a> is funded by the <a href="https://erc.europa.eu/">European Research Council</a> under the European Union’s Horizon 2020 research and innovation programme, awarded to the <a href="http://kitab-project.org/">KITAB</a> project (Grant Agreement No. 772989, PI Sarah Bowen Savant), hosted at Aga Khan University, London. In addition, it has received funding from the <a href="https://www.qnl.qa/en">Qatar National Library</a> to aid in the adaptation of the passim algorithm for Arabic.</p> <p><a href="http://kitab-project.org/">KITAB</a>’s text reuse data is generated by running <a href="https://github.com/dasmiq/passim">passim</a> on the OpenITI corpus (DOI: <a href="https://doi.org/10.5281/zenodo.3082463">10.5281/zenodo.3082463</a>). Each version is the output of a separate run and the version number corresponds to the corpus releases.</p> <p>To prepare the corpus for a passim run, we normalize texts and remove most of the non-Arabic characters and then chunk the texts into passages of 300 words (using the non-Arabic characters, including white space) in length. The chunks, called milestones, are identified by unique ids. This dataset represents the reuse cases that have been identified among milestones. </p> <p>The text reuse dataset consists of folders for each book. Each folder includes CSV files of the text reuse cases (alignments) between the corresponding book and all other books with which passim has found instances of reuses. The files have the below naming convention, using the book ids:</p> <p> <bookVersionID1>_<bookVersionID2>.csv<br> (e.g., ‘Shamela000001.mARkdown_Shamela000002.csv’).</p> <p><br>The CSV files are not the immediate output of passim, rather the result of the post-processing step. The folder structure is as below (for a total of four books, for example).</p> <p> bookVersionID1<br> |- bookVersionID1_bookVersionID4.csv<br> |- bookVersionID1_bookVersionID3.csv <br> bookVersionID4<br> |-bookVersionID4_bookVersionID3.csv</p> <p><br>Where we do not have any CSV files in any of the folders, it means that the passim algorithm has not been able to find any text reuse cases for that specific book. In the above example, we can not find any folder or CSV files for bookVresionID2, that means no reuse cases are detected between book2 and of the other three books.</p> <p>To save computational resources, we generate text reuse data uni-directionally, which means a pair of documents is compared only once (document1 to document2, not document2 to document1). </p> <p>The alignments the CSV files are a list of records. Each record shows a pair of matched passages between two books together with statistics, such as the algorithm score, and contextual information, such as the start and end positions of aligned passages so that one can find those passages in the books. A description of the alignment fields is given in the release notes.</p> <p>For each dataset, we also generate statistical data on the alignments between the book pairs. The data is published in an application that facilitates search, filtering, and visualizations. The link to the corresponding application is given in the release notes.</p> <p><strong>Note on Release Numbering</strong>: Version <strong>2020.1.1</strong>—where <strong>2020</strong> is the year of the release, the first dotted number—<strong>.1</strong>—is the ordinal release number in 2020, and the second dotted number—<strong>.1</strong>—is the overall release number. The first dotted number will reset every year, while the second one will continue on increasing.</p> <p><strong>Note:</strong> The very first release of the KITAB text reuse data (2019.1.1) is published <a href="http://dev.kitab-project.org/passim01022019/">here</a> as it was too big to publish on Zenodo. To receive more information on the complete datasets please contact us via kitab-project@outlook.com (or other team members). </p> <p>Future releases may include part of the generated data if the size of whole data is too big to publish on Zenodo. However, the data is open access for anyone to use. We provide the detailed information on the datasets in the corresponding release notes.</p>
Evaluation Set for Historical Text Reuse Detection (HTRD) in 15th-Century German Prose Texts
<p>The evaluation set provides a pair of 15th-century German prose texts in which instances of assumed text reuse (TR) have been annotated, categorised and linked using the TEI standard. The annotated texts are intended to serve as evaluation set for computational methods and tools specialising on Historical Text Reuse Detection (HTRD). The texts featured in the evaluation set are (1) "Der Doernenkrantz van Collen", a late medieval lay didactic treatise on the estates of society, and (2) "Die Cronica van der hilliger Stat Coellen", one of the most important medieval chronicles of the city of Cologne. Both texts are written in Ripuarian, a west middle German dialect of the late middle ages and early modern period, and exhibit orthographic variance as well as transcription errors due to automatic Handwritten Text Recognition (HTR), which makes them challenging for HTRD.</p>
Webis Text Reuse Corpus 2012
<p>The Webis Text Reuse Corpus 2012 (Webis-TRC-12) compiles manually written documents obtained from a completely controlled, yet representative environment that emulates the web. Each document in the corpus is about one of the 150 topics used at the TREC Web Tracks 2009–2011, thus forming a strong connection with existing evaluation efforts. Writers, hired at the crowdsourcing platform oDesk, had to retrieve sources for a given topic and to reuse text from what they found. Part of the corpus are detailed interaction logs that consistently cover the search for sources as well as the creation of documents. This will allow for in-depth analyses of how text is composed if a writer is at liberty to reuse texts from a third party.</p> <p> </p>
Reception Reader: Shakespeare text reuse dataset
<p>A sample dataset to accompany an article on text reuse. The data is a subset of larger corpus of text reuses detected in all of Eighteenth Century Collections Online (ECCO) and Early English Books Online (EEBO), and includes all detected text reuse instances connected with works of Wllam Shakespeare.</p> <p>See the accompanying README.md for details on files.</p> <p>The data is available under Attribution 4.0 International (CC BY 4.0) license, and is free to use with proper citation to the article.</p>
Text Reuse in the Medieval Chinese Public Inscriptions (618-907)
<p>It is commonly known that Chinese premodern authors preferred using a number of allusions originating from classics in their own works, which is certainly a way of text reuse. They sometimes quoted original texts directly, while sometimes substituted some characters or inverted grammatical constructions without changing the original meaning. What I want to study in this paper is to find the network of text reuse covering all the public inscriptions, including those inscribed on steles, walls or bells to which the public had access, in the Tang Dynasty China (618-907). A database including more than 1,000 inscriptions with about 1,000,000 Chinese characters / words has been built as the base of this project. Metadata such as authors, dates and places, have been manually added to each inscription. Based on the metadata, temporal and spatial distribution of the inscriptions will be visually displayed. According to my pilot studies, Confucian canons, especially the Five Classics, were quoted most frequently in the inscriptions. Based on close reading experience, history books and religious canons had great influence on medieval Chinese literati’s writing as well. So I would like to discover the network which reflects how public inscriptions reused Chinese Confucian classics, prestigious history books, Buddhist and Daoist canons.<br> Beyond finding the network of text reuse, I also want to find out whether one author had preferences to some specific classical allusions, and which books were referred to the most frequently in different periods or places. Besides distant reading, I plan to do close reading of texts to show how genres of texts influenced authors’ choices of text reuse and how authors’ writing strategies were influenced by certain political contexts. Ideally, I want to further analyze the intellectual and political history behind the network of text reuse. This research is an experiment on applying digital humanities to Chinese history studies, in which methodological issues will be especially addressed.</p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.