Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
878
datasets available to search
ShareScore release 0.9.0
Dataset results
878 results for “Corpus”
French Entity-Linking dataset between annotated tweets collected during major crises in France and French Wikipedia corpus
<p>Most of the available datasets are not particularly adapted to our target application: geolocate natural disasters from social networks. First, social media posts are largely underrepresented in these datasets, and the only Twitter dataset lacks Entity-Linking annotations. Second, none of the datasets focuses on a crisis or natural disaster event.</p> <p>To mitigate these issues, we extracted a collection of French tweets written during earthquakes and major floods that have occurred in France in recent years. We set up Label-Studio in order to annotate these tweets. A total of 4617 tweets were annotated, including 1678 tweets posted during earthquakes and 2939 during floods. For each annotated tweet, mentions were annotated using the set of labels described earlier in the paper as well as, when possible, the target Wikipedia title.</p> <p>Named “RéSoCIO” in reference to the research project in which it was carried out, the dataset resulting from this work contains a total of 12 828 annotated mentions and 1 513 distinct Wikipedia entities. 85% of mentions were associated with a Wikipedia page and 94 % if we ignore the RISKNAT and DAMAGES labels, which are often difficult to map to an existing entity.</p> <table> <tbody> <tr> <td><strong>Labels</strong></td> <td><strong>#Mentions</strong></td> <td><strong>#Linked</strong></td> <td><strong>#Entities</strong></td> </tr> <tr> <td>PERSON</td> <td>315</td> <td>263</td> <td>136</td> </tr> <tr> <td>ORG</td> <td>863</td> <td>790</td> <td>281</td> </tr> <tr> <td>GEOLOC</td> <td>4375</td> <td>4234</td> <td>701</td> </tr> <tr> <td>TRANSPORT</td> <td>250</td> <td>203</td> <td>101</td> </tr> <tr> <td>EVENT</td> <td>35</td> <td>21</td> <td>16</td> </tr> <tr> <td>FACILITY</td> <td>129</td> <td>94</td> <td>49</td> </tr> <tr> <td>RISKNAT</td> <td>5502</td> <td>4994</td> <td>128</td> </tr> <tr> <td>DAMAGES</td> <td>1136</td> <td>121</td> <td>56</td> </tr> <tr> <td>OTHER</td> <td>223</td> <td>200</td> <td>46</td> </tr> <tr> <td><strong>Total</strong></td> <td><strong>12828</strong></td> <td><strong>1322</strong></td> <td><strong>1513</strong></td> </tr> </tbody> </table> <p>Overview of the mentions annotated in the Twitter dataset. #Mentions shows the total number of mentions per label, #Linked the number of mentions linked to an entity and #Entities the number of distinct entities per label present in the dataset.</p> <table> <tbody> <tr> <td><strong>Labels</strong></td> <td><strong>#Mentions</strong></td> <td><strong>#Linked</strong></td> <td><strong>#Entitie</strong>s</td> </tr> <tr> <td>PERSON</td> <td>1100102</td> <td>1098406</td> <td>557697</td> </tr> <tr> <td>ORG</td> <td>750925</td> <td>749504</td> <td>130394</td> </tr> <tr> <td>GEOLOC</td> <td>2729702</td> <td>2728296</td> <td>215924</td> </tr> <tr> <td>TRANSPORT</td> <td>161539</td> <td>160487</td> <td>53405</td> </tr> <tr> <td>EVENT</td> <td>798433</td> <td>798251</td> <td>86471</td> </tr> <tr> <td>FACILITY</td> <td>258835</td> <td>258513</td> <td>109867</td> </tr> <tr> <td>RISKNAT</td> <td>5502</td> <td>4994</td> <td>127</td> </tr> <tr> <td>DAMAGES</td> <td>1136</td> <td>121</td> <td>56</td> </tr> <tr> <td>OTHER</td> <td>4340621</td> <td>4339658</td> <td>682458</td> </tr> <tr> <td><strong>Total</strong></td> <td><strong>10146795</strong></td> <td><strong>10138230</strong></td> <td><strong>1836399</strong></td> </tr> </tbody> </table> <p>Overview of the mentions annotated in the full dataset. #Mentions shows the total number of mentions per label, #Linked the number of mentions linked to an entity and #Entities the number of distinct entities per label present in the dataset.</p>
OcWikiAnnot: Annotated Wikipedia Corpus of Occitan
<p>OcWikiAnnot is a corpus of Wikipedia content in Occitan that is tokenized, PoS-tagged and lemmatized. The corpus contains 100 000 sentences for a total of 2 037 723 tokens. It is based on the Wikipedia corpus in Occitan that is part of the <a href="https://corpora.uni-leipzig.de/en?corpusId=oci_wikipedia_2021">Leipzig Corpora Collection</a>.</p> <p> </p> <p> </p>
First Steps towards a Risk of Bias Corpus of Randomized Controlled Trials
<p><strong>Abstract</strong></p> <p>Risk of bias (RoB) assessment of randomized clinical trials (RCTs) is vital to conducting systematic reviews. Manual RoB assessment for hundreds of RCTs is a cognitively demanding, lengthy process and is prone to subjective judgment. Supervised machine learning (ML) can help to accelerate this process but requires a hand-labelled corpus. There are currently no RoB annotation guidelines for randomized clinical trials or annotated corpora. In this pilot project, we test the practicality of directly using the revised Cochrane RoB 2.0 guidelines for developing an RoB annotated corpus using a novel multi-level annotation scheme. We report inter-annotator agreement among four annotators who used Cochrane RoB 2.0 guidelines. The agreement ranges between 0% for some bias classes and 76% for others. Finally, we discuss the shortcomings of this direct translation of annotation guidelines and scheme and suggest approaches to improve them to obtain an RoB annotated corpus suitable for ML.</p> <p> </p> <p><strong>Methods</strong></p> <p>The upload contains two zip files and a .json file.</p> <ul> <li>plain.html.zip</li> </ul> <p>Original corpus (n = 10) in .html format. The corpus was generated using the methodology described in the paper. Each .html file could be opened in any default text editor in any operating system or browser. A .html contains full text divided into several annotatable text parts. </p> <p> </p> <ul> <li>ann.json.zip</li> </ul> <p>The .zip contains RoB annotations conducted by the authors (R.H., M.S., K.G., R.C.). The annotation files are in .json format. Each .json is divided into two JSON objects and three JSON arrays. </p> <ol> <li>annotatable (object): Parts from the full-text document corresponding to the text parts from the plain .html files. </li> <li>metas (object): full-text document label</li> <li>entities (array): contains labelled entities. Each entity is linked to which part of the full-text it is linked to.</li> <li>relations (array)</li> <li>sources (array)</li> </ol> <p> </p> <ul> <li>annotations-legend.json</li> </ul> <p>This .json file contains entity and entity labels encoded to text legends. For example, entity class label "1_2_Yes_Good" is encoded as "e_113".</p> <p> </p> <p><strong>Resources</strong></p> <p>The code to parse annotations can be found on <a href="http:// https://github.com/anjani-dhrangadhariya">GitHub</a>.</p> <p> </p> <p><strong>Funding</strong></p> <p>HES-SO Valais-Wallis, Sierre, Switzerland</p>
Archi text corpus
<p>Archi belongs to the Lezgic group of the Nakh-Daghestanian (North-East Caucasian) languages, being quite loosely related to the rest of the group. It has been long time surrounded by non-Lezgic languages and therefore has kept and/or acquired a number of peculiar features.</p> <p>Here is presented a sample of texts collected in the village of Archi in 2006 and 2007. In total, over 50 texts of various genres have been recorded, including stories, conversations, tales, legends and songs. Most of them were recorded in both video and audio.</p> <p>Two kinds of texts were recorded. First, some 30 previously published (1977) texts were re-recorded in video and audio, read by one of three speakers. (The original recordings do not exist anymore). Second, new texts have been collected, mostly dialogues and stories. Texts available in this version were all originally published in [Kibrik et al. 1977].</p> <p><em>Кибрик А. Е., Кодзасов С. В., Оловянникова И. П., Самедов Д. С.</em> Арчинский язык. Тексты и словари. — М.: МГУ, 1977.<br> [Kibrik, Aleksandr E.; Kodzasov, S. V.; Olovjannikova, I. P. & Samedov, D. S. (1977). <em>Arčinskij jazyk. Teksiy i slovari</em>. Moscow: Izdatel'stvo moskovskogo universiteta.]</p> <p>The project was generously supported by NSF grant #0553546 «Five languages of Eurasia» (PI under the Documenting Endangered Languages Program, and by RFBR grants № 05-06-80351 «Minority languages and cultures: On the verge of extinction» and № 08-06-00345 «Multimedia corpora for endangered languages».</p>
OpenITI: a Machine-Readable Corpus of Islamicate Texts
<p><strong>Co-PIs</strong>: Matthew Thomas Miller (University of Maryland, College Park), Maxim G. Romanov (University of Hamburg), Sarah Bowen Savant (Aga Khan University—ISMC, London).</p> <p><em>Open Islamicate Texts Initiative</em> (<strong>OpenITI</strong>, see <a href="https://openiti.org/">https://openiti.org/</a>) is a multi-institutional effort to construct the first machine-actionable scholarly corpus of premodern Islamicate texts. Led by researchers at the Aga Khan University, Institute for the Study of Muslim Civilisations (AKU-ISMC), University of Hamburg (UH), and the Roshan Institute for Persian Studies at the University of Maryland (College Park) and an interdisciplinary advisory board of leading digital humanists and Islamic, Persian, and Arabic studies scholars, <strong>OpenITI</strong> aims to provide the essential textual infrastructure in Arabic, Persian and other Islamicate languages for new forms of textual analysis and digital scholarship. In the process, OpenITI will enable new synergies between Digital Humanities and the inter-related Islamicate fields of Islamic, Persian, and Arabic Studies. In addition to support from the researchers’ home institutions, it is supported by funding from the <a href="https://erc.europa.eu/">European Research Council</a> under the European Union’s Horizon 2020 research and innovation programme, awarded to the <a href="http://kitab-project.org/">KITAB</a> project (Grant Agreement No. 772989, PI Sarah Bowen Savant) and the <a href="https://www.qnl.qa/en">Qatar National Library</a>.</p> <p>Currently, <strong>OpenITI</strong> contains almost exclusively Arabic texts, which were first assembled into a corpus within the <strong>OpenArabic</strong> project, developed first at Tufts University (at <em>The Perseus Project</em>, 2013–2015) and then at Leipzig University (at the Alexander von Humboldt Chair for Digital Humanities, 2015–2017)—in both cases with the support and under the patronage of Prof. Gregory Crane. The much more limited number of Persian texts were compiled during 2015–2016 in the Persian Digital Library (PDL) pilot (see <a href="https://persdigumd.github.io/PDL/">Persian Digital Library by PersDigUMD</a>) at Roshan Institute for Persian Studies at the University of Maryland. These texts have not been made fully compatible with OpenITI mARkdown yet and will be made fully available in next releases.</p> <p>This release contains all digital versions of the same text that are available in the OpenITI corpus . <strong>We also release a <a href="https://doi.org/10.5281/zenodo.7764025">'primary' version of the corpus</a></strong> that contains a single digital version for each text in the corpus that is marked as 'PRI' in the corpus metadata and may be more convenient for some use cases.</p> <p><strong>Note on Release Numbering</strong>: Version <strong>2019.1.1</strong>—where <strong>2019</strong> is the year of the release, the first dotted number—<strong>.1</strong>—is the ordinal release number in 2019, and the second dotted number—<strong>.1</strong>—is the overall release number; the first dotted number will reset every year, while the second one will continue on increasing.</p> <p>For more details: <a href="https://github.com/OpenITI/RELEASE">https://github.com/OpenITI/RELEASE</a></p> <p><strong>Note: </strong>In case of any issues with unzipping the files on Windows using built-in utilities, please use free softwares, such as WinRAR and 7zip.</p> <p> </p>
Webis Query Spelling Corpus 2017 (Webis-QSpell-17)
<p>The Webis Query Spelling Corpus 2017 (Webis-QSpell-17) contains 54,772 web queries that were manually spell-checked; for 9,171 queries alternative spelling variants are contained.</p> <p>As for segmentations of many of the queries (i.e., tagged concepts and phrases), please refer to the companion corpus <a href="https://webis.de/data/webis-qsec-10.html">Webis-QSeC-10</a>.</p>
Preprocessed Java Code Corpus
<p>A preprocessed code corpus for the Java programming language.<br> The corpus was used for the experiments in the paper Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code.<br> It contains preprocessed-tokenized files for training, validation, testing, and BPE encoding learning.<br> The BPE segmented versions of the above files are also included for three different encoding sizes i,e., 2000, 5000, and 10000 BPE merge operations as well as the learned BPE encodings.<br> Similar versions are also contained for splitting compound identifiers on camelCase and snake_case as in (Allamanis et al., 2015) as well as the corresponding subtoken maps.</p>
Hypernym-LIBre: A free Web-based corpus from Hypernym Detection [ Hearst Pattern extractions from Hypernym-LIBre]
<p>Hypernym-LIBre ( DOI: 10.5281/zenodo.3662204 ) is a free Web-based corpus for Hypernym detection.</p> <p>Its part-of-speech tagged and dependency annotated version is present at this: (DOI: 10.5281/zenodo.3689303)</p> <p>Here we provide the hypernym-hyponym pairs that were extracted from Hypernym-LIBre using Hearst patterns. This is to further the usage of these extractions with more techniques and methods. We also provide the counts of each pattern in a separate file.</p> <p> </p> <p>Format:</p> <p>hyponym \t hypernym</p> <p> </p> <p>Format for the counts file:</p> <p>pair \t frequency of extraction</p> <p> </p> <p>There are 2 files, one with the pairs, one with unique pair and its counts. Both total ~430MB.</p>
CodiEsp corpus: gold standard Spanish clinical cases coded in ICD10 (CIE10) - eHealth CLEF2020
<p><strong>Introduction</strong></p> <p>These are the train, development and test sets of the CodiEsp corpus. Train, development and test have gold standard annotations. In addition, the unannotated background set is also distributed. All documents are released in the context of the CodiEsp track for CLEF ehealth 2020 (<a href="http://temu.bsc.es/codiesp/">http://temu.bsc.es/codiesp/</a>).</p> <p>The CodiEsp corpus contains manually coded clinical cases. All documents are in Spanish language and CIE10 is the coding terminology (it is the Spanish version of ICD10-CM and ICD10-PCS). The CodiEsp corpus has been randomly sampled into three subsets: the train, the development, and the test set. The train set contains 500 clinical cases, and the development and test set 250 clinical cases each. CodiEsp participants must submit predictions for the test and background set, but they will only be evaluated on the test set.</p> <p> </p> <p><strong>Please cite if you use this dataset:</strong></p> <p>Antonio Miranda-Escalada, Aitor Gonzalez-Agirre, Jordi Armengol-Estapé and Martin Krallinger. Overview of automatic clinical coding: annotations, guidelines, and solutions for non-English clinical cases at CodiEsp track of CLEF eHealth 2020. In CLEF (Working Notes). 2020</p> <pre><code>@inproceedings{miranda2020overview, title={Overview of automatic clinical coding: annotations, guidelines, and solutions for non-english clinical cases at codiesp track of CLEF eHealth 2020}, author={Miranda-Escalada, Antonio and Gonzalez-Agirre, Aitor and Armengol-Estap{\'e}, Jordi and Krallinger, Martin}, booktitle={Working Notes of Conference and Labs of the Evaluation (CLEF) Forum. CEUR Workshop Proceedings}, year={2020} }</code></pre> <p> </p> <p><strong>Annotation quality</strong></p> <p>Inter-annotator agreement: 88.6% for diagnosis coding, 88.9% for procedure coding and 80.5% for the textual reference annotation. For more information, see the <a href="http://ceur-ws.org/Vol-2696/paper_263.pdf">paper</a>.</p> <p><br> <strong>Zip structure</strong><br> Four folders: train, dev, test and background. Each one of them contains the files for the train, development, test and background corpora, respectively.</p> <ul> <li><strong>train, dev and test</strong> folders have: <ul> <li>3 tab-separated files with the annotation information relevant for each of the 3 sub-tracks of CodiEsp. </li> <li>A subfolder named <em>text_files</em> with the plain text files of the clinical cases.</li> <li>A subfolder named <em>text_files_en</em> with the plain text files machine-translated to English. Due to the translation process, the text files are sentence-splitted.</li> </ul> </li> <li>The <strong>background</strong> folder has only <em>text_files</em> and <em>text_files_en</em> subfolders with the plain text files.</li> </ul> <p><br> <strong>Format</strong><br> The CodiEsp corpus is distributed in plain text in UTF8 encoding, where each clinical case is stored as a single file whose name is the clinical case identifier. Annotations are released in a tab-separated file. Since the CodiEsp track has 3 sub-tracks, every set of documents (train and test) has 3 tab-separated files associated with it. </p> <p>For the sub-tracks CodiEsp-D and CodiEsp-P, the file has the following fields:</p> <pre>articleID ICD10-code </pre> <p>Tab-separated files for the sub-track CodiEsp-X contain extra fields that provide the text-reference and its position:</p> <pre>articleID label ICD10-code text-reference reference-position</pre> <p><br> <strong>Corpus summary statistics</strong><br> The final collection of 1000 clinical cases that make up the corpus had a total of 16504 sentences, with an average of 16.5 sentences per clinical case. It contains a total of 396,988 words, with an average of 396.2 words per clinical case.</p> <p> </p> <p><strong>Resources:</strong></p> <ul> <li><strong><a href="https://temu.bsc.es/codiesp/">Web</a></strong></li> <li><strong><a href="http://ceur-ws.org/Vol-2696/paper_263.pdf">Citation</a>: </strong>Antonio Miranda-Escalada, Aitor Gonzalez-Agirre, Jordi Armengol-Estapé and Martin Krallinger. Overview of automatic clinical coding: annotations, guidelines, and solutions for non-English clinical cases at CodiEsp track of CLEF eHealth 2020. In CLEF (Working Notes). 2020</li> <li><strong><a href="https://doi.org/10.5281/zenodo.3859869">Silver Standard corpus</a></strong></li> <li><strong><a href="https://doi.org/10.5281/zenodo.3730566">Annotation guidelines</a></strong></li> <li><a href="https://www.youtube.com/playlist?list=PL5uSCzf1azhA0crlSVCYMPqMUWd4mXc4x"><strong>YouTube presentations</strong></a></li> <li><a href="https://temu.bsc.es/codiesp/index.php/participants-systems/"><strong>Participant codes</strong></a></li> </ul> <p> </p> <p>For more information, visit the track webpage: http://temu.bsc.es/codiesp/ or email us at encargo-pln-life@bsc.es</p> <p> </p> <p>Copyright (c) 2019 Secretaría de Estado para el Avance Digital</p>
Dataset do DH2020 [The Lusophone Digital Humanities and What they (we) are doing from the South: textual corpus analysis and FAIR principles to tackle Hegemony]
<p>Planilha de dados recuperados do Google Scholar utilizado na análise e apresentação da pesquisa empírica intitulada - <strong>The Lusophone Digital Humanities and What they (we) are doing from the South: textual corpus analysis and FAIR principles to tackle Hegemony </strong>- no evento <strong>DH2020 Ottawa</strong>: <a href="https://hcommons.org/deposits/item/hc:32051/">https://hcommons.org/deposits/item/hc:32051/</a></p>
Manually Annotated Instances of Ich ('I') from the German KoLas Corpus
<p>Dataset used in Andresen/Knorr (2020). The dataset comprises 360 instances of <em>ich</em> ('I') taken from the German learner corpus KoLaS (Andresen/Knorr 2017, see <a href="http://hdl.handle.net/11022/0000-0001-B732-8">http://hdl.handle.net/11022/0000-0001-B732-8</a> for full corpus access) and manually annotated with categories taken from Steinhoff (2007).</p> <p>Column descriptions:</p> <ul> <li>document: name of the document by which it can be found in the KoLaS corpus</li> <li>code_annotator1 - code_annotator4: Annotations by four annotators. Possible values: Verfasser-<em>Ich</em> (author <em>I</em>), Forscher-<em>Ich</em> (researcher <em>I</em>), Erzähler-<em>Ich</em> (narrator <em>I</em>)</li> <li>max_agreement_freq: Highest number of anntators that agreed on one label</li> <li>max_agreement_label: Label on which the highest number of annotators agreed</li> <li>context_before: 150 characters of context before the match</li> <li>match: the match itself (either <em>ich</em> or <em>Ich</em>)</li> <li>context_after: 150 characters of context after the match</li> </ul> <p><strong>References</strong></p> <p>Andresen M, Knorr D. KoLaS – Ein Lernendenkorpus in der Schreibberatungsausbildung einsetzen. <em>Zeitschrift Schreiben</em>. Published online July 5, 2017:10-17.</p> <p>Andresen M, Knorr D. Exploring the Use of the Pronoun I in German Academic Texts with Machine Learning. In: Burghardt M, Müller-Birn C, eds. <em>Methoden und Anwendungen der Computational Humanities</em>. Lecture Notes in Informatics (LNI). Gesellschaft für Informatik; 2020.</p> <p>Steinhoff T. Zum ich-Gebrauch in Wissenschaftstexten. <em>Zeitschrift für germanistische Linguistik</em>. 2007;35(1-2):1–26.</p>
GitHub Java Corpus - Function Identifiers
<p>This dataset contains function identifiers extracted from the GitHub Java Corpus (<a href="http://groups.inf.ed.ac.uk/cup/javaGithub/">http://groups.inf.ed.ac.uk/cup/javaGithub/</a>).</p> <p>Each line corresponds to a method declaration. A line contains the name of the method declaration followed by the function identifiers (i.e., function calls) contained within the method body. </p> <p>The file embeddings_train.json can be used to train a word/sentence embedding model using the code in the Github repository (link below).</p> <p>The corpus was used for the experiments in the paper <strong>Combining Code Embedding with Static Analysis for Function-Call Completion</strong>.</p> <p>Github repository to replicate the experiments: https://github.com/mweyssow/cse-saner</p>
ENGLISH-AKUAPEM TWI PARALLEL CORPUS
<p>This dataset <em><strong>(verified_data.csv)</strong></em> is bilingual machine translation training corpus for English and Akuapem Twi of 25,421 sentence pairs. <br> A transformer-based machine translator was used to generate initial translations in Akuapem Twi, which were later verified and corrected where necessary by native speakers. <br> The main idea of a typical use case for the dataset is for further training of machine translation models in Akuapem Twi.<br> The data can also be used for other downstream NLP tasks such as Named Entity Recognition and POS tagging, with appropriate additional annotations. <br> Another potential application is training unsupervised embeddings for the Akuapem Twi language.<br> In addition a higher quality 697 crowdsourced sentences <em><strong>(crowdsourced_data.csv) </strong></em>are provided for use as an evaluation set for the tasks highlighted above. It is recommended as a testing dataset for machine translation English to Twi and Twi to English models.</p> <p><strong>Acknowledgement</strong>: This project was supported by the <a href="https://www.k4all.org/project/language-dataset-fellowship/">AI4D language dataset fellowship</a> through K4all and Zindi Africa</p>
Webis-Simple-Sentences-17 Corpus
<p>A corpus of 471,085,690 English sentences extracted from the ClueWeb12 Web Crawl. The sentences were sampled from a larger corpus to achieve a level of sentence complexity similar to the one of sentences that humans make up as a memory aid for remembering passwords. Sentence complexity was determined by syllables per word.</p> <p>The corpus is split in training and test set as it is used in the associated publication. The test set is extracted from part 00 of the ClueWeb12, while the training set is extracted from the other parts.</p> <p>More information on the corpus can be found on the corpus web page at our university (listed under documented by).</p>
Qualitas Corpus
<p><strong>Overview of Data</strong></p> <p>Although the original Qualitas Corpus has provided a valuable contribution for experimentation in software engineering, there are several scenarios—e.g., experiments that rely on Abstract Syntax Tree (AST) or bytecode—in which researchers need to import and compile the source code. Since this task is not trivial in the case of systems with many external dependencies, our goal is to assist researchers by removing the compilation effort when conducting empirical studies.</p> <p><strong>Qualitas</strong></p> <p>The Qualitas Corpus is a curated collection of software systems intended to be used for empirical studies of code artefacts. The primary goal is to provide a resource that supports reproducible studies of software. The current release of the Corpus contains open-source Java software systems, often multiple versions.</p>
Toward a Comparable Corpus of Latvian, Russian and English Tweets
<p>Twitter has become a rich source for linguistic data. Here, a possibility of building a trilingual Latvian-Russian-English corpus of tweets from Riga, Latvia is investigated. Such a corpus, once constructed, might be of great use for multiple purposes such as training machine translation models, examining cross-lingual phenomena and studying the population of Riga. This pilot study shows that it is feasible to build such a resource by building and analysing a pilot corpus, which is made publicly available and can be used to construct a large comparable corpus.</p>
A Corpus of Biblical Names in the Greek New Testament to Study the Additions, Omissions, and Variations across Different Manuscripts
<p>The analysis of textual variants of verses in the Ancient Greek New Testament across different manuscripts has mainly been done by close reading with manual effort. With the increasing number of transcriptions of the different manuscripts, quantitative analyses (so-called distant reading) can be used to search for patterns of omission, addition, or other variations, to formulate novel hypotheses to be investigated by close reading. In this work, we present a corpus of biblical names including spelling variation and inflections and their mentions in the transcriptions of the Ancient Greek New Testament.</p>
Data Citation Corpus Data File
<p>Data file for the fourth release of the Data Citation Corpus, produced by DataCite and Make Data Count as part of an ongoing grant project funded by the Wellcome Trust. <a href="https://makedatacount.org/data-citation/">Read more about the project</a>.</p> <p>The data file includes 10,697,745 data citation records (of which 9,682,257 represent unique dataset-publication pairs) in JSON and CSV formats. The JSON file is the version of record.</p> <p>Data is provided in batches of approximately 1 million records each. The publication date and batch number are included in the file name, ex: 2025-08-15-data-citation-corpus-01-v4.1.json.</p> <p>The data citations in the file originate from the following sources:</p> <ul> <li>DataCite Event Data</li> <li>Chan Zuckerberg Initiative (CZI) Science Knowledge Graph</li> <li>Aligning Science Across Parkinson’s (ASAP)</li> <li>Europe PMC</li> </ul> <p>Each data citation record is comprised of:</p> <ul> <li> <p>A pair of identifiers: An identifier for the dataset (a DOI or an accession number) and the DOI of the publication (journal article or preprint) in which the dataset is cited </p> </li> <li> <p>Metadata for the cited dataset and for the citing publication </p> </li> </ul> <p>The data file includes the following fields:</p> <div> <table> <tbody> <tr> <td> <p>Field</p> </td> <td> <p>Description</p> </td> <td> <p>Required?</p> </td> </tr> <tr> <td> <p>id</p> </td> <td> <p>Internal identifier for the citation</p> </td> <td> <p>Yes</p> </td> </tr> <tr> <td> <p>created</p> </td> <td> <p>Date of item's incorporation into the corpus</p> </td> <td> <p>Yes</p> </td> </tr> <tr> <td> <p>updated</p> </td> <td> <p>Date of item's most recent update in corpus</p> </td> <td> <p>Yes</p> </td> </tr> <tr> <td> <p>repository</p> </td> <td> <p>Repository where cited data is stored</p> </td> <td> <p>No</p> </td> </tr> <tr> <td> <p>publisher</p> </td> <td> <p>Publisher for the article citing the data</p> </td> <td> <p>No</p> </td> </tr> <tr> <td> <p>journal</p> </td> <td> <p>Journal for the article citing the data</p> </td> <td> <p>No</p> </td> </tr> <tr> <td> <p>title</p> </td> <td> <p>Title of cited data</p> </td> <td> <p>No</p> </td> </tr> <tr> <td> <p>publication</p> </td> <td> <p>DOI of article where data is cited</p> </td> <td> <p>Yes</p> </td> </tr> <tr> <td> <p>dataset</p> </td> <td> <p>DOI or accession number of cited data</p> </td> <td> <p>Yes</p> </td> </tr> <tr> <td> <p>publishedDate</p> </td> <td> <p>Date when citing article was published</p> </td> <td> <p>No</p> </td> </tr> <tr> <td> <p>source</p> </td> <td> <p>Source where citation was harvested</p> </td> <td> <p>Yes</p> </td> </tr> <tr> <td> <p>subjects</p> </td> <td> <p>Subject information for cited data</p> </td> <td> <p>No</p> </td> </tr> <tr> <td> <p>affiliations</p> </td> <td> <p>Affiliation information for creator of cited data</p> </td> <td> <p>No</p> </td> </tr> <tr> <td> <p>funders</p> </td> <td> <p>Funding information for cited data</p> </td> <td> <p>No</p> </td> </tr> </tbody> </table> </div> <p><strong> </strong></p> <p>Additional documentation about the citations and metadata in the file is available on the <a href="https://makedatacount.org/find-a-tool/data-citation-corpus-documentation/">Make Data Count website</a>. </p> <p><strong>Notes on v4.1:</strong></p> <p>Version 4.1 of the Data Citation Corpus is a minor update to v4.0 that corrects (1) an error that occurred when a portion of DOI-DOI citations originating from Europe PMC were attributed to the wrong repository, and (2) a small number of DOI formatting errors in the "publication" field.</p> <p><strong>Notes on v4.0:</strong></p> <p>The fourth release of the Data Citation Corpus data file adds new citations from the following sources:</p> <ul> <li> <p dir="ltr">5.2 million data citations from <a href="https://europepmc.org/">Europe PMC</a> identified as "eupmc" in the source field. Ingest of these citations was performed 9 July 2025.</p> </li> <li> <p dir="ltr">139,647 data citations from DataCite Event Data for the period 1 January 2025 through 30 June 2025.</p> </li> </ul> <p dir="ltr">This release also includes the following new metadata enhancements:</p> <ul> <li> <p dir="ltr">Affiliation information for cited data from the Gene Expression Omnibus (GEO) repository, reonciled to Research Organization Registry (ROR) IDs where possible.</p> </li> <li> <p dir="ltr">Reconciliation of organization and funders names with the Research Organization Registry (ROR) for new citations from Event Data.</p> </li> <li> <p dir="ltr">Application of Field of Science subject terms to citation records originating from Europe PMC, based on disciplinary area of data repository.</p> </li> </ul> <p>Additional details about the above changes, including scripts used to perform the above tasks, are available in <a href="https://github.com/Make-Data-Count-Community/corpus-data-file" target="_blank" rel="noopener">GitHub</a>. </p> <p>Additional enhancements to the corpus are ongoing and will be addressed in the course of subsequent releases. Users are invited to submit feedback via <a href="https://github.com/Make-Data-Count-Community/data-citation-corpus-feedback">GitHub</a>. For general questions, email <a href="mailto:info@makedatacount.org">info@makedatacount.org</a>.</p>
Corpus der amtlichen Entscheidungssammlung des Bundesverfassungsgerichts (C-BVerfGE)
<p> </p> <p><strong>Überblick</strong></p> <p>Das <strong>Corpus der amtlichen Entscheidungssammlung des Bundesverfassungsgerichts (C-BVerfGE)</strong> ist eine digitale Zusammenstellung von möglichst vielen Entscheidungen, die in der amtlichen Entscheidungssammlung des Bundesverfassungsgerichts (BVerfGE) veröffentlicht sind. Der Datensatz enthält alle Entscheidungen, die auf der <a href="https://www.bundesverfassungsgericht.de">amtlichen Webseite des Bundesverfassungsgerichts</a> am jeweiligen Stichtag in der <a href="https://www.bundesverfassungsgericht.de/DE/Entscheidungen/Entscheidungen/Amtliche%20Sammlung%20BVerfGE.html">Auflistung der Entscheidungen der BVerfGE</a> verlinkt waren.</p> <p><em>Bitte beachten Sie das beiliegende Codebook!</em> Es enthält wichtige Informationen zur korrekten Nutzung des Datensatzes. Es hilft auch bei der Entscheidung, welche Variante für Sie am besten geeignet ist. In der Regel empfehle ich für quantitative Forschung die CSV-Dateien und für traditionelle juristische Arbeit die PDF-Sammlung.</p> <p>Alle die <em>Corona-Pandemie</em> betreffenden Entscheidungen des Bundesverfassungsgerichts finden Sie zusätzlich separat dokumentiert und analysiert im Datensatz <a href="https://doi.org/10.5281/zenodo.4459405"><strong>Corona-Rechtsprechung des Bundesverfassungsgerichts (BVerfG-Corona)</strong></a><strong>.</strong></p> <p>Das <em>C-BVerfGE</em> sollte nicht mit dem <strong><a href="http://doi.org/10.5281/zenodo.3902658">Corpus der Entscheidungen des Bundesverfassungsgerichts (CE-BVerfG)</a> </strong>verwechselt werden! Letzterer zielt auf eine Abbildung <em>aller</em> auf <a href="https://www.bundesverfassungsgericht.de">www.bundesverfassungsgericht.de</a> veröffentlichten Entscheidungen und ist mit über 8000 Entscheidungen mehr als zehnmal so groß.</p> <p> </p> <p><strong>Aktualisierung</strong></p> <p>Dieser Datensatz wird <em>1-2 mal im Jahr</em> aktualisiert. Benachrichtigungen über neue und aktualisierte Datensätze veröffentliche ich immer zeitnah auf Mastodon unter <a href="https://fediscience.org/@seanfobbe">@seanfobbe@fediscience.org</a></p> <p> </p> <p> </p> <p><strong>NEU in Version 2025-08-20</strong></p> <ul> <li>Vollständige Aktualisierung der Daten (bis einschließlich Band 169)</li> <li>LIZENZÄNDERUNG: Source Code jetzt unter GNU General Public License Version 3 (GPLv3) oder später lizenziert</li> <li>Neue Variablen: Tenor und Beschlussformel</li> <li>Variable entfernt: Kurzbeschreibung (wird vom BVerfG nicht mehr angeboten)</li> <li>R-Version auf 4.4.0 aktualisiert (wegen CVE-2024-27322)</li> <li>Expliziter R Package Version Lock für 2024-06-13 (CRAN Date)</li> <li>Paralellisierung von Quanteda repariert</li> <li>Überarbeitung des Dockerfiles</li> <li>Anpassung von Compose File an Debian 11</li> <li>Docker Zeitzone auf Berlin eingestellt</li> <li>Zusätzliches Lösch-Skript mit Docker-Integration</li> <li>Vereinfachung der Run-Skripte und stärkere Integration mit Docker Compose</li> <li>/tmp in Arbeitsspeicher ausgelagert</li> </ul> <p> </p> <p><strong>Features</strong></p> <ul> <li>Insgesamt bis zu 35 Variablen</li> <li>Fortlaufende Aktualisierung</li> <li>Urheberrechtsfreiheit</li> <li>Offene und plattformunabhängige Formate (PDF, TXT, CSV, HTML)</li> <li>Entscheidungsnamen und BVerfGE-Fundstelle</li> <li>Verknüpfung mit PräsidentIn/Vize-PräsidentIn</li> <li>Linguistische Kennzahlen</li> <li>Umfangreiches Codebook</li> <li>Compilation Report um den Erstellungs-Prozess zu erläutern</li> <li>Dutzende Diagramme und Tabellen für alle Zwecke (im ZIP-Archiv 'ANALYSE').</li> <li>Jedes Diagramm liegt in einem für den Druck (PDF) und das Web (PNG) optimierten Format vor. Tabellen sind im CSV-Format bereitgestellt und sind damit sowohl für Menschen als auch für Maschinen gut lesbar</li> <li>Kryptographisch signiert</li> <li><a href="../doi/10.5281/zenodo.4265933">Veröffentlichung des Source Codes</a></li> </ul> <p> </p> <p><strong>Eckdaten</strong></p> <p><em>Stichtag:</em> 20. August 2025</p> <p><em>Inhaltlicher Umfang</em>: 970 Entscheidungen</p> <p><em>Zeitlicher Umfang:</em> 1997 bis 2024, plus vereinzelte Entscheidungen aus anderen Jahren</p> <p><em>Formate:</em><strong> </strong>PDF, TXT, CSV und HTML</p> <p> </p> <p><strong>Source Code und Compilation Report</strong></p> <p>Der gesamte Erstellungs-Prozess ist ab Version 2021-01-03 vollautomatisiert und detailliert dokumentiert. Mit jeder Kompilierung des vollständigen Datensatzes wird auch ein umfangreicher Compilation Report in einem attraktiv designten PDF-Format erstellt (ähnlich dem Codebook).</p> <p>Der Compilation Report enthält den vollständigen Source Code, dokumentiert relevante Rechenergebnisse, gibt sekundengenaue Zeitstempel an und ist mit einem klickbaren Inhaltsverzeichnis versehen. Er ist zusammen mit dem Source Code hinterlegt. Wenn Sie sich für Details des Erstellungs-Prozesses interessieren, lesen Sie diesen bitte zuerst.</p> <p>Der <em>vollständige Source Code</em> — sowohl für die Erstellung des Datensatzes, als auch für das Codebook — ist <em>öffentlich einsehbar und dauerhaft erreichbar</em> im wissenschaftlichen Archiv des CERN unter diesem Link hinterlegt: <a href="../doi/10.5281/zenodo.4265933">https://zenodo.org/doi/10.5281/zenodo.4265933</a></p> <p> </p> <p><strong>Kryptographische Signaturen</strong></p> <p>Die Integrität und Echtheit der einzelnen Archive des Datensatzes sind durch eine <em>Zwei- Phasen-Signatur</em> sichergestellt.</p> <p>In <em>Phase I</em> werden während der Kompilierung für jedes ZIP-Archiv Hash-Werte in zwei verschiedenen Verfahren (SHA2-256 und SHA3-512) berechnet und in einer CSV-Datei dokumentiert.</p> <p>In <em>Phase II</em> wird diese CSV-Datei mit meinem persönlichen geheimen GPG-Schlüssel signiert. Dieses Verfahren stellt sicher, dass die Kompilierung von jedermann durchgeführt werden kann, insbesondere im Rahmen von Replikationen, die persönliche Gewähr für Ergebnisse aber dennoch vorhanden ist.</p> <p>Die während der Kompilierung des Datensatzes erstellte CSV-Datei mit den Hash-Prüfsummen ist mit meiner <em>persönlichen GPG-Signatur</em> versehen. Der mit dieser Version korrespondierende Public Key ist sowohl mit dem Datensatz als auch mit dem Source Code hinterlegt. Er hat folgende Kenndaten:</p> <p><em>Name:</em> Sean Fobbe (fobbe-data@posteo.de)</p> <p><em>Fingerabdruck:</em> FE6F B888 F0E5 656C 1D25 3B9A 50C4 1384 F44A 4E42</p> <p> </p> <p><strong>Kein Urheberrecht: Public Domain</strong></p> <p>An den Entscheidungstexten und amtlichen Leitsätzen besteht gem. § 5 Abs. 1 UrhG <em>kein </em>Urheberrecht, da sie amtliche Werke sind. § 5 UrhG ist auf amtliche Datenbanken analog anzuwenden (BGH, Beschluss vom 28.09.2006 - I ZR 261/03, "Sächsischer Ausschreibungsdienst"). Alle eigenen Beiträge (z.B. durch Zusammenstellung und Anpassung der Metadaten) und damit den gesamten Datensatz stelle ich gemäß einer <a href="https://creativecommons.org/publicdomain/zero/1.0/legalcode">CC0 1.0 Universal Public Domain License</a> vollständig urheberrechtsfrei.</p> <p> </p> <p><strong>Disclaimer</strong></p> <p>Dieser Datensatz ist eine private wissenschaftliche Initiative und steht weder mit dem Bundesverfassungsgericht noch mit den Herausgebern der BVerfGE in Verbindung.</p> <p> </p> <p><strong>Alternative Online-Quellen zur BVerfGE</strong></p> <p>Die Urteile im C-BVerfGE sind mit Randnummern versehen, enthalten aber keine der amtlichen BVerfGE entsprechende Paginierung, hier empfehle ich die DFR-Sammlung von Tschentscher. Für statistische Analysen von älteren Entscheidungen sollten Sie auf die Datensätze von Coupette oder Möllers/Shadrova/Wendel zurückgreifen. Diese Datensätze stehen unter teils sehr restriktiven Lizenzen.</p> <p> </p> <p><em>[BVerfGE 1951-2019]</em> Möllers, Christoph, Shadrova, Anna, & Wendel, Luisa. (2021). "BVerfGE-Korpus". <a href="https://doi.org/10.5281/zenodo.4551408">https://doi.org/10.5281/zenodo.4551408</a></p> <p>[Senatsentscheidungen 1972-2010] Engst, Benjamin G., Thomas Gschwend, Christoph Hönnige & Caroline E. Wittig. (2020).<em> “The Constitutional Court Database." </em><a href="https://ccdb.eu/">https://ccdb.eu/</a><em><br></em></p> <p><em>[BVerfGE 1951-2015 als TXT-Datensatz]</em> Coupette, Corinna: "Juristische Netzwerkforschung: Modellierung, Quantifizierung und Visualisierung relationaler Daten im Recht (Online-Appendix)". <a href="https://doi.org/10.1628/978-3-16-157012-4-appendix">https://doi.org/10.1628/978-3-16-157012-4-appendix</a></p> <p><em>[BVerfGE von 1951 bis heute, mit Paginierung nach gedruckter BVerfGE!]</em> Tschentscher, Axel (Hrsg): "Deutschsprachiges Fallrecht (DFR)". <a href="https://www.servat.unibe.ch/dfr/dfr_bverfg.html">https://www.servat.unibe.ch/dfr/dfr_bverfg.html</a></p> <p> </p> <p><strong>Weitere Open Access Veröffentlichungen (Fobbe)</strong></p> <p>Website<em> </em>—<em> </em><a href="https://www.seanfobbe.de">www.seanfobbe.de</a></p> <p>Open Data — <a href="../communities/sean-fobbe-data/">zenodo.org/communities/sean-fobbe-data/</a></p> <p>Source Code — <a href="../communities/sean-fobbe-code/">zenodo.org/communities/sean-fobbe-code/</a></p> <p>Volltexte regulärer Publikationen — <a href="../communities/sean-fobbe-publications/">zenodo.org/communities/sean-fobbe-publications/</a></p> <p> </p> <p><strong>Kontakt</strong></p> <p>Fehler gefunden? Anregungen? Kommentieren Sie gerne im Issue Tracker oder kontaktieren Sie mich über www.seanfobbe.de</p> <p> </p>
City-Data.com Corpus
<p>The City-Data.com Corpus consists of five discussion forum threads from the site, city-data.com scraped using the BeautifulSoup Python library (Richardson, 2015). City-Data.com aggregates geographic, demographic, and historical information about major cities across the United States and Canada from public and governmental sources. The site provides data visualizations that enable users to compare cities.The City-Data.com Corpus contains 15,008 across five forum topics from the US > Pennsylvania > Philadelphia forums. </p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.