Find research datasets worth reusing

Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.

datasets available to search

ShareScore release 0.9.0

Dataset results

7 results for “Historical Newspapers”

Learn how ShareScore rates datasets ↗

zenodo48/100

Models for "A data-driven approach to studying changing vocabularies in historical newspaper collections"

NOTE: This is a badly rendered version of the README within the archive. A data-driven approach to studying changing vocabularies in historical newspaper collections Simon Hengchen,* Ruben Ros,** Jani Marjanen,*** Mikko Tolonen*** *<a href="https://spraakbanken.gu.se/en/about/staff/simon">Språkbanken Text</a>, University of Gothenburg, Sweden and <a href="https://iguanodon.ai">iguanodon.ai</a>, Belgium: firstname.lastname@gu.se **<a href="https://www.c2dh.uni.lu/people/ruben-ros">Centre for Contemporary and Digital History (C2DH)</a>, University of Luxembourg: firstname.lastname@uni.lu ***<a href="https://www.helsinki.fi/en/researchgroups/computational-history">COMHIS</a>, University of Helsinki: <a href="mailto:firstname.lastname@helsinki.fi">firstname.lastname@helsinki.fi</a>; These are the supplementary materials for the DH2019 paper A data-driven approach to the changing vocabulary of the ‘nation’ in English, Dutch, Swedish and Finnish newspapers, 1750-1950, as well as the 2021 Digital Scholarship in the Humanities publication available in OpenAccess: <a href="https://academic.oup.com/dsh/article/36/Supplement_2/ii109/6421793">https://academic.oup.com/dsh/article/36/Supplement_2/ii109/6421793</a>. If you end up using whole or parts of this resource, please use the following citation(s): <ul> <li>Hengchen, S., Ros, R., and Marjanen, J. (2019). A data-driven approach to the changing vocabulary of the 'nation' in English, Dutch, Swedish and Finnish newspapers, 1750-1950. In Proceedings of the Digital Humanities (DH) conference 2019, Utrecht, The Netherlands</li> </ul> and/or: <ul> <li>Hengchen, S., Ros, R., Marjanen, J. and Tolonen, M., 2021. A data-driven approach to studying changing vocabularies in historical newspaper collections. Digital Scholarship in the Humanities, 36(Supplement_2), pp.ii109-ii126.</li> </ul> or alternatively use one of the following <code>bib</code>s: <pre><code>@inproceedings{hengchen2019nation, title="A data-driven approach to the changing vocabulary of the 'nation' in {E}nglish, {D}utch, {S}wedish and {F}innish newspapers, 1750-1950.", author={Hengchen, Simon and Ros, Ruben and Marjanen, Jani}, year={2019}, address = "Utrecht, The Netherlands", booktitle={Proceedings of the Digital Humanities (DH) conference 2019} }</code></pre> <pre><code>@article{hengchen2021data, title={A data-driven approach to studying changing vocabularies in historical newspaper collections}, author={Hengchen, Simon and Ros, Ruben and Marjanen, Jani and Tolonen, Mikko}, journal={Digital Scholarship in the Humanities}, volume={36}, number={Supplement\_2}, pages={ii109--ii126}, year={2021}, publisher={Oxford University Press} }</code></pre>   Files This archive contains two folders -- one per diachronic representation method -- as well as this README. The folders each contain four folders, which contain the models for their respective languages. As can be inferred from the small datasize, most of the earlier models are not reliable and should not be used, but are still made available. This work is licensed under a <a href="http://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>. Source material Finnish: The models were created with data from the Finnish Sub-corpus of the Newspaper and Periodical Corpus of the National Library of Finland (National Library of Finland, 2011). We used everything in the corpus. Filesizes: <pre><code>[simon@taito-login3 SGNS]$ du -h fi* 12M fi_1820_SGNS_corpus_file.gensim 89M fi_1840_SGNS_corpus_file.gensim 797M fi_1860_SGNS_corpus_file.gensim 7.0G fi_1880_SGNS_corpus_file.gensim 22G fi_1900_SGNS_corpus_file.gensim</code></pre> Swedish: The models were created with data from the Kubhist 2 corpus (Språkbanken) -- more precisely, the data dumps available at <a href="https://spraakbanken.gu.se/lb/resurser/meningsmangder/">https://spraakbanken.gu.se</a>. After a manual evaluation of Swedish embeddings trained without pre-processing seemed to show that the embeddings were of low quality, we retrained models, only keeping sentences that were at least 10 tokens long and were constituted of at least 50% of lemmas as per the KORP processing pipeline (Borin et al, 2012). Filesizes: <pre><code>[simon@taito-login3 SGNS]$ du -h sv* 1.6M sv_1740_SGNS_corpus_file.gensim 44M sv_1760_SGNS_corpus_file.gensim 124M sv_1780_SGNS_corpus_file.gensim 228M sv_1800_SGNS_corpus_file.gensim 678M sv_1820_SGNS_corpus_file.gensim 1.6G sv_1840_SGNS_corpus_file.gensim 4.5G sv_1860_SGNS_corpus_file.gensim 6.5G sv_1880_SGNS_corpus_file.gensim 113M sv_1900_SGNS_corpus_file.gensim</code></pre> Dutch: The models were created with data from the Delpher newspaper archive (Royal Dutch Library, 2017), through data dumps for newspapers until and including 1876, and through API hits for articles from 1877 to 1899 (included). <ul> <li>For anything pre-1877 we discarded full texts that had, in the metadata, anything else than exclusively <code>nl</code> or <code>NL</code> as a language tag.</li> <li>For the full texts between 1877 and 1899: we queried the API for all items in the “artikel” category that contained the determiner <code>de</code>.</li> </ul> Our assumption was that most articles should contain <code>de</code> at least once, and those that didn't were too short to be deemed interesting. A subsequent study showed that was not exactly the case, but we were reassured by the fact that left-out articles were probably "shipping or financial reports" (thanks go to Melvin Wevers). We also did not include the colonial newspapers for our embeddings. This is motivated by our research questions. A list of removed newspapers is available on request. Filesizes: <pre><code>[simon@taito-login3 SGNS]$ du -h nl* 6.8M nl_1620_SGNS_corpus_file.gensim 7.9M nl_1640_SGNS_corpus_file.gensim 43M nl_1660_SGNS_corpus_file.gensim 78M nl_1680_SGNS_corpus_file.gensim 138M nl_1700_SGNS_corpus_file.gensim 243M nl_1720_SGNS_corpus_file.gensim 287M nl_1740_SGNS_corpus_file.gensim 431M nl_1760_SGNS_corpus_file.gensim 825M nl_1780_SGNS_corpus_file.gensim 1.2G nl_1800_SGNS_corpus_file.gensim 1.8G nl_1820_SGNS_corpus_file.gensim 3.1G nl_1840_SGNS_corpus_file.gensim 5.2G nl_1860_SGNS_corpus_file.gensim 13G nl_1880_SGNS_corpus_file.gensim</code></pre> English: The models were created with data from the British Library Newspapers collection (<a href="https://www.gale.com/intl/primary-sources/british-library-newspapers%5D">link</a>), the Nichols collection (<a href="https://www.gale.com/intl/c/17th-and-18th-century-burney-newspapers-collection">link</a>), and the Burney collection (<a href="https://www.gale.com/intl/c/17th-and-18th-century-nichols-newspapers-collection">link</a>). We used everything in the corpora. For English, only SGNS_ALIGN models are available. We thank Gale Cengage for their help with this project. Filesizes: <pre><code>[simon@taito-login3 SGNS]$ du -h en* 4.3M en_1620_SGNS_corpus_file.gensim 11M en_1640_SGNS_corpus_file.gensim 11M en_1660_SGNS_corpus_file.gensim 106M en_1680_SGNS_corpus_file.gensim 409M en_1700_SGNS_corpus_file.gensim 1.7G en_1720_SGNS_corpus_file.gensim 834M en_1740_SGNS_corpus_file.gensim 2.4G en_1760_SGNS_corpus_file.gensim 5.3G en_1780_SGNS_corpus_file.gensim 5.5G en_1800_SGNS_corpus_file.gensim 15G en_1820_SGNS_corpus_file.gensim 42G en_1840_SGNS_corpus_file.gensim 65G en_1860_SGNS_corpus_file.gensim 88G en_1880_SGNS_corpus_file.gensim 26G en_1900_SGNS_corpus_file.gensim 21G en_1920_SGNS_corpus_file.gensim 6.3G en_1940_SGNS_corpus_file.gensim</code></pre> Word embeddings For every language, we train diachronic embeddings as follows. We divide the data in 20-year time bins. We train SGNS_UPDATE and SGNS_ALIGN models. Current research on German (Schlechtweg et al, 2019) and English (Shoemark et al, 2019) indicates you should use the SGNS_ALIGN models. For EN, FI, NL, no tokens (including punctuation) were removed nor altered, aside from lowercasing. For SV, see above. Parameters are as follows: SGNS architecture (Mikolov et al 2013), window size of 5, frequency threshold of 100, 5 epochs, 300 dimensions (or 100 for EN). <ul> <li>For SGNS_UPDATE: We first train a model for the first time bin <code>t</code>. To train the model for <code>t+1</code>, we use the <code>t</code> model to initialise the vectors for <code>t+1</code>, set the learning rate to correspond to the end learning rate of <code>t</code>, and continue training. This approach, closely following Kim et al (2014), has the advantage of avoiding the need for post-training vector space alignment.</li> </ul> The Python snippet below, which makes use of gensim (Rehurek and Sojka, 2010), illustrates the approach. Special thanks go to Sara Budts. <pre><code>## dict_files[key] is a dictionary with double decades as keys and a corresponding LineSentence object as value: https://radimrehurek.com/gensim/models/word2vec.html#gensim.models.word2vec.LineSentence count = 0 for key in sorted(list(dict_files.keys())): if count == 0: ## This is the first model. model = gensim.models.Word2Vec(corpus_file=dict_files[key], min_count=100, sg=1 ,size=300, workers=64, seed=1830, iter=5) model.save(os.path.join(data_path_final,"KIM",lang+"_"+str(timebin)+".w2v")) print("Model saved, on to the next\n") count += 1 if count > 0: ## this is for the subsequent models. print("model for double decade starting in",str(key)) model = gensim.models.Word2Vec.load(os.path.join(data_path_final,"KIM",lang+"_"+str(timebin-20)+".w2v")) print("previous model loaded") model.build_vocab(corpus_file=dict_files[key], update=True) model.train(corpus_file=dict_files[key], total_words = model.corpus_count, total_examples = model.corpus_count, start_alpha = model.alpha, end_alpha = model.min_alpha, epochs=model.epochs) model.save(os.path.join(data_path_final,"KIM",lang+"_"+str(timebin)+".w2v")) </code></pre> <ul> <li>For SGNS_ALIGN: We independently train models for all time bins. The models in this repository are NOT aligned, leaving you the choice of how to align them. For example, <a href="https://gist.github.com/quadrismegistus/09a93e219a6ffc4f216fb85235535faf">here</a> is a link to code by Ryan Heuser to do just that. Models were trained with the <code>count == 0</code> scenario in the snippet above.</li> </ul> Acknowledgments This work has been supported by the European Union's Horizon 2020 research and innovation programme under grant 770299 <a href="https://www.newseye.eu/">NewsEye</a>. Specials thanks go to the data providers/collection-holding institutions: the Finnish Language Bank, the Swedish Language Bank, the Royal Dutch Library, and Gale Cengage. The authors would like to thank the following persons and group, listed alphabetically: Antoine Doucet, Antti Kanner, Axel-Jean Caurant, Dominik Schlechtweg, Eetu Mäkelä, Elaine Zosa, Estelle Bunout, Haim Dubossarsky, Joris van Eijnatten, Krister Lindén, Lars Borin, Lidia Pivovarova, Melvin Wevers, Nina Tahmasebi, Sara Budts, Senka Drobac, Tanja Säily, the COMHIS group, and Steven Claeyssens. Computational resources were provided by CSC – IT Center for Science Ltd. References Borin, L., Forsberg, M., Roxendal, J. (2012). Korp-the corpus infrastructure of Spräkbanken,in: LREC. pp. 474–478. Kim, Y., Chiu, Y.I., Hanaki, K., Hegde, D. and Petrov, S. (2014). Temporal Analysis of Language through Neural Language Models. ACL 2014, p.61. Mikolov, T., Chen, K., Corrado, G. and Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. National Library of Finland (2011). The Finnish Sub-corpus of the Newspaper and Periodical Corpus of the National Library of Finland, Kielipankki Version [text corpus]. Kielipankki. Retrieved from <a href="http://urn.fi/urn:nbn:fi:lb-2016050302">http://urn.fi/urn:nbn:fi:lb-2016050302</a>. Rehurek, R. and Sojka, P. (2010). Software framework for topic modelling with large corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks. Royal Dutch Library (2017). Delpher open krantenarchief (1.0). Den Haag, 2017. Schlechtweg D., Hätty A, del Tredici M., and Schulte im Walde S. (2019). A Wind of Change: Detecting and Evaluating Lexical Semantic Change across Times and Domains. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Florence, Italy. ACL. Shoemark, P., Liza, F.F., Nguyen, D., Hale, S. and McGillivray, B. (2019). Room to Glo: A Systematic Comparison of Semantic Change Detection Approaches with Word Embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (pp. 66-76), Hong Kong. Språkbanken. The Kubhist Corpus. Department of Swedish, University of Gothenburg. <a href="https://spraakbanken.gu.se/korp/?mode=kubhist">https://spraakbanken.gu.se/korp/?mode=kubhist</a>.

opencc-by-4.0Dec 2019View details →

zenodo44/100

Datasets and Models for Historical Newspaper Article Segmentation

This record contains the datasets and models used and produced for the work reported in the paper "Combining Visual and Textual Features for Semantic Segmentation of Historical Newspapers" (<a href="https://infoscience.epfl.ch/record/282863?ln=en">link</a>). Please cite this paper if you are using the models/datasets or find it relevant to your research: <pre><code>@article{barman_combining_2020, title = {{Combining Visual and Textual Features for Semantic Segmentation of Historical Newspapers}}, author = {Raphaël Barman and Maud Ehrmann and Simon Clematide and Sofia Ares Oliveira and Frédéric Kaplan}, journal= {Journal of Data Mining \& Digital Humanities}, volume= {HistoInformatics} DOI = {10.5281/zenodo.4065271}, year = {2021}, url = {https://jdmdh.episciences.org/7097}, }</code></pre> Please note that this record contains data under different licenses. 1. DATA <ul> <li>Annotations (json files): JSON files contains image annotations, with one file per newspaper containing region annotations (label and coordinates) in VIA format. The following licenses apply: <ul> <li> luxwort.json: those annotations are under a <a href="https://creativecommons.org/publicdomain/zero/1.0/legalcode">CC0 1.0 license</a>. Please refer to the right statement specified for each image in the file.</li> <li>GDL.json, IMP.json and JDG.json: those annotations are under a <a href="https://creativecommons.org/licenses/by-sa/4.0/legalcode">CC BY-SA 4.0 license</a>.</li> </ul> </li> </ul>   <ul> <li>Image files: The archive images.zip contains the Swiss titles image files (GDL, IMP, JDG) used for the experiments described in the paper. Those images are under copyright (property of the journal Le Temps and of ArcInfo) and can be used for academic research or educational purposes only. Redistribution, publication or commercial use are not permitted. These terms of use are similar to the following right statement: <a href="http://rightsstatements.org/vocab/InC-EDU/1.0/">http://rightsstatements.org/vocab/InC-EDU/1.0/</a></li> </ul>   2. MODELS Some of the best models are released under a <a href="https://creativecommons.org/licenses/by-sa/4.0/">CC BY-SA 4.0</a> license (they are also available as assets of the current Github <a href="https://github.com/dhlab-epfl/dhSegment-text/releases/tag/0.1">release</a>). <ul> <li>JDG_flair-FT: this model was trained on JDG using french Flair and FastText embeddings. It is able to predict the four classes presented in the paper (<code>Serial</code>, <code>Weather</code>, <code>Death notice</code> and <code>Stocks</code>).</li> <li>Luxwort_obituary_flair-bpemb: this model was trained on Luxwort using multilingual Flair and Byte-pair embeddings. It is able to predict the <code>Death notice</code> class.</li> <li>Luxwort_obituary_flair-FT_indomain: this model was trained on Luxwort using in-domain Flair and FastText embeddings (trained on Luxwort data). It is also able to predict the <code>Death notice</code> class.</li> </ul> Those models can be used to predict probabilities on new images using the same code as in the original <a href="https://github.com/dhlab-epfl/dhSegment">dhSegment</a> repository. One needs to adjust three parameters to the <code>predict</code> function: 1) <code>embeddings_path</code> (the path to the embeddings list), 2) <code>embeddings_map_path</code>(the path to the compressed embedding map), and 3) <code>embeddings_dim</code> (the size of the embeddings). Please refer to the paper for further information or contact us.   3. CODE:  <a href="https://github.com/dhlab-epfl/dhSegment-text">https://github.com/dhlab-epfl/dhSegment-text</a> 4. ACKNOWLEDGEMENTS We warmly thank the journal <a href="https://letemps.ch">Le Temps</a> (owner of La Gazette de Lausanne and the Journal de Genève) and the group <a href="https://www.arcinfo.ch/">ArcInfo</a> (owner of L'Impartial) for accepting to share the related datasets for academic purposes. We also thank the <a href="https://bnl.public.lu/fr.html">National Library of Luxembourg</a> for its support with all steps related to the Luxemburger Wort annotation release. This work was realized in the context of the <a href="https://impresso-project.ch">impresso - Media Monitoring of the Past</a> project and supported by the Swiss National Science Foundation under grant CR- SII5_173719. 5. CONTACT Maud Ehrmann (EPFL-DHLAB) Simon Clematide (UZH)

openother-ncJan 2021View details →

zenodo40/100

Dataset for Logical-layout analysis on French historical newspapers

Dataset for Logical-layout analysis on French historical newspapers This dataset is intended for training and testing Logical Layout Analysis and recognition system on French historical documents published between 1900 and 1950. The original data is part of the "<a href="https://gallica.bnf.fr/services/engine/search/sru?operation=searchRetrieve&exactSearch=false&version=1.2&query=%28colnum%20adj%20%22Appartient%20%C3%A0%20l%27ensemble%20documentaire%20:%20FrancComt1%22%29">Fond régional: Franche-Comté</a>", which is curated by <a href="https://gallica.bnf.fr/accueil/fr/content/accueil-fr?mode=desktop">Gallica</a>, the digital portal of the Bibliothèque Nationale de France (BnF). This dataset has the following structure: ├── train   ├── 1c     ├── cb32836282t       ├── cb32836282t.xml       ├── bpt6k112325g         ├── bpt6k112325g.xml         ├── truelabels_block.csv         ├── truelabels_line.csv       ├── …     ├── …   ├── 2c   ├── 3c+ └── test   ├── 1c   ├── 2c   └── 3c+ The dataset is divided into a train and a test set. The train and test datasets have been designed to cover as much as possible the various possible layouts that exist in the "Fond régional: Franche-Comté" dataset. To do so, we have divided them into three layout types:   • 1c: documents where the text is displayed in one column, as in books;   • 2c: documents where the text is displayed into two columns;   • 3c+: documents where there are at least 3 columns of text, as in newspapers. Each of the 1c, 2c, and 3c+ folder contains subfolders prefixed by ‘cb’, which contain a collection of documents. For instance, « cb32836282t » is the identifier used in Gallica for « Le Petit écho du 21e Régiment d'infanterie », a French military periodical published during WWI. An XML file with the same name, for instance «cb32836282t.xml », contains metadata about the collection, such as its title, publisher, creator, number of issues, etc. This XML file serves only to describe the collection, and is not to be used for Logical-Layout analysis. The issues in each collection can be found in the subfolders prefixed with « bpt ». For instance, « bpt6k112325g » is the identifier used in Gallica for an issue published in September 1917 of « Le Petit écho du 21e Régiment d'infanterie ». The information about each issue is given in three files, which are described below: 1-bptXXXXXXXXXX.xml The original data, as collected from Gallica. The most important tags of this document and their values are described below:   • oai: metadata about the document, such as its author, title, publisher, original publication date, number of issues, …   • image_url: the url to the document’s scan (in high resolution)   • pagination: a description of each page in the document (size of the page, if it contains a table of content or not, …)   • num_pages: the total number of pages in the document   • ocr: the OCR representation of the document in the XML ALTO format The XML ALTO format provides the text content and physical layout of documents in the following manner. Lines of text are contained in TextLine tags, which in their turn contain String tags for words and SP tags for spaces. TextLine tags are grouped into blocks in TextBlock tags. Sometimes, TextBlock tags are also grouped into ComposedBlock tags. TextBlock and TextLine tags have the following attributes:   • Id : the tag’s identifier   • Height, Width : the text height and width   • Vpos : the vertical position of the text on the page. The higher the value, the lower the word is on the page   • Hpos : the horizontal position of the text on the page. The higher the value, the further on the right the text is on the page   • Language : the language of the text (only for TextBlock tags). Among the attributes listed above, some TextBlock tags also have a Type attribute. This attribute contains logical labels of the lines in the block. In this dataset it appears most often for tables or advertisements. Overall, TextBlock tags that have a Type attribute are rare in this dataset (about 4 % only). Note: The original scan of every document is accessible on the Gallica website, using the URL https://gallica.bnf.fr/ark:/12148/<IDENTIFIER>, where <IDENTIFIER> should be replaced by the id of the document (e.g.: bpt6k112325g) or the collection (e.g.: cb32836282t). 2-truelabels_block.csv A CSV file where each line corresponds to a TextBlock tag from the file bptXXXXXXXXXX.xml. This CSV file contains the following columns:   • page: the page on which the TextBlock tag is located   • block_id: the id of the TextBlock tag   • first_last_line: the text content of the first and last TextLine tags inside this TextBlock tag   • classes: the logical label(s) associated with this TextBlock tag The possible values in the column classes are : Text, Title, Header and Other. 3-truelabels_line.csv A CSV file where each line corresponds to a TextLine tag from the file bptXXXXXXXXXX.xml. This CSV file contains the following columns:   • page: the page where the TextLine tag is located   • block_id: the id of the TextBlock tag that contains this TextLine tag   • line_id: the id of this TextLine tag   • text_line: the text content of this TextLine tag   • classes: the logical label(s) associated with this TextLine tag The possible values in the column classes are : Text, Firstline, Title, Header and Other. Firstline indicates the « first line » of a paragraph.

opencc-by-4.0Oct 2021View details →

zenodo40/100

Dataset and Models for Detection of News Agency Releases in Historical Newspapers

This record contains the annotated datasets and models used and produced for the work reported in the Master Thesis "Where Did the News come from? Detection of News Agency Releases in Historical Newspapers " (<a href="https://infoscience.epfl.ch/record/305129?&ln=en">link</a>). Please cite this report if you are using the models/datasets or find it relevant to your research: <pre><code>@article{Marxen:305129, title = {Where Did the News Come From? Detection of News Agency Releases in Historical Newspapers}, author = {Marxen, Lea}, pages = {114p}, year = {2023}, url = {http://infoscience.epfl.ch/record/305129}, }</code></pre> 1. DATA The newsagency-dataset contains historical newspaper articles with annotations of news agency mentions. The articles are divided into French (fr) and German (de) subsets and a train, dev and test set respectively. The data is annotated at token-level in the CoNLL format with IOB tagging format. The distribution of articles in the different sets is as follows: <table> <caption>Dataset Statistics</caption> <thead> <tr> <th scope="row"> </th> <th scope="col">Lg.</th> <th scope="col">Docs</th> <th scope="col">Agency Mentions</th> </tr> </thead> <tbody> <tr> <th scope="row">Train</th> <td>de</td> <td>333</td> <td>493</td> </tr> <tr> <th scope="row"> </th> <td>fr</td> <td>903</td> <td>1,122</td> </tr> <tr> <th scope="row">Dev</th> <td>de</td> <td>32</td> <td>26</td> </tr> <tr> <th scope="row"> </th> <td>fr</td> <td>110</td> <td>114</td> </tr> <tr> <th scope="row">Test</th> <td>de</td> <td>32</td> <td>58</td> </tr> <tr> <th scope="row"> </th> <td>fr</td> <td>120</td> <td>163</td> </tr> </tbody> </table> Due to an error, there are seven duplicated articles in the French test set (article IDs: courriergdl-1847-10-02-a-i0002, courriergdl-1852-02-14-a-i0002, courriergdl-1860-10-31-a-i0016, courriergdl-1864-12-15-a-i0005, lunion-1860-11-27-a-i0004, lunion-1865-02-05-a-i0012, lunion-1866-02-16-a-i0009).   2. MODELS The two agency detection and classification models used for the inference on the <a href="https://impresso-project.ch/">impresso</a> Corpus are released as well: <ul> <li>newsagency-model-de: based on <a href="https://www.deepset.ai/german-bert">German BERT</a> (with maximum sequence length 128), fine-tuned with the German training set of the newsagency-dataset</li> <li>newsagency-model-fr: based on <a href="https://huggingface.co/dbmdz/bert-base-french-europeana-cased">French Europeana BERT</a> (with maximum sequence length 128), fine-tuned with the French training set of the newsagency-dataset</li> </ul> The models perform multitask classification with two prediction heads, one for token-level agency entity classification and one for sentence-level (has_agency: yes/no). They can be run with TorchServe, for details see the <a href="https://github.com/impresso/newsagency-classification/tree/main/lib/bert_classification">newsagency-classification</a> repository.   Please refer to the report for further information or contact us.   3. CODE <a href="https://github.com/impresso/newsagency-classification">https://github.com/impresso/newsagency-classification</a>   4. CONTACT Maud Ehrmann (EPFL-DHLAB) Emanuela Boros (EPFL-DHLAB)

opencc-by-nc-4.0Sep 2023View details →

zenodo36/100

Historical Newspapers Ground Truth

This dataset contains 50 pages of ground truth data for digitized historical newspapers from the Berlin State Library for training and evaluation of OCR/OLR systems as produced in the context of the EU ICT-PSP project Europeana Newspapers (http://www.europeana-newspapers.eu/). The dataset comprises of the following resources: <ul> <li>gt_page.zip Ground Truth files in PAGE-XML format (cf. https://github.com/PRImA-Research-Lab/PAGE-XML)</li> <li>img_full.zip Full resolution scanned images in TIF format</li> <li>img_bin.zip Binarized (using the Gatos method) images in TIF format</li> <li>ocr_full.zip OCR (FineReaderEngine11) results for full resolution images</li> <li>ocr_bin.zip OCR (FineReaderEngine11) results for binarized images</li> </ul>

opencc-zeroMar 2019View details →

zenodo36/100

Estonian Historical Newspaper Crowdsourced OCR Corrections

<div> <div>This dataset consists of newspaper articles from the National Library of Estonia's DIGAR archive and their respective crowdsourced corrections.</div> </div>

opencc-by-4.0Aug 2024View details →

zenodo12/100

Historic news comments from the G1 brazilian online newspaper

This dataset contains data of user comments about the news published in the brazilian online newspaper G1 from 2010 and before. It contains 54,634 comments distributed between 4,549 news, which includes news published date, comments texts (in brazilian portuguese) and comments published date. That dataset intentionally ommits the titles, links, usernames or other kind of data that can redirect back to someones identity.

restrictedOct 2021View details →

ScienceDex guides

Understand access before you commit

These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.

Compare curated datasets

Allen Brain Atlas

Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.

allen-brain-atlas

neuroscienceopenDocumentation, web resources, and API references are available online.

Last verified 2026-04-30Open record

Annotated Behaviour and Observability Dataset (ABODe)

ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.

abode-home-cage

behavioral-neuroscienceopenThe DataShare record exposes download links for annotations, documentation, license text, and the zipped per-snippet data directory.

Last verified 2026-04-30Open record

DANDI Archive for NWB datasets

DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.

dandi-nwb

electrophysiologyopenPublished Dandiset metadata and archive endpoints are available through the production DANDI API.

Last verified 2026-04-30Open record

International Brain Laboratory public data

The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.

ibl

behavioral-neuroscienceopenPublic sessions can be searched and loaded from the IBL public data server through ONE.

Last verified 2026-04-29Open record

OpenNeuro

OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.

openneuro

neuroscienceopenPublished datasets are available on demand over the internet.

Last verified 2026-04-29Open record