Find research datasets worth reusing

opencc-by-4.0Dec 2019View details →

Models for "A data-driven approach to studying changing vocabularies in historical newspaper collections"

NOTE: This is a badly rendered version of the README within the archive. A data-driven approach to studying changing vocabularies in historical newspaper collections Simon Hengchen,* Ruben Ros,** Jani Marjanen,*** Mikko Tolonen*** *<a href="https://spraakbanken.gu.se/en/about/staff/simon">Språkbanken Text</a>, University of Gothenburg, Sweden and <a href="https://iguanodon.ai">iguanodon.ai</a>, Belgium: firstname.lastname@gu.se **<a href="https://www.c2dh.uni.lu/people/ruben-ros">Centre for Contemporary and Digital History (C2DH)</a>, University of Luxembourg: firstname.lastname@uni.lu ***<a href="https://www.helsinki.fi/en/researchgroups/computational-history">COMHIS</a>, University of Helsinki: <a href="mailto:firstname.lastname@helsinki.fi">firstname.lastname@helsinki.fi</a>; These are the supplementary materials for the DH2019 paper A data-driven approach to the changing vocabulary of the ‘nation’ in English, Dutch, Swedish and Finnish newspapers, 1750-1950, as well as the 2021 Digital Scholarship in the Humanities publication available in OpenAccess: <a href="https://academic.oup.com/dsh/article/36/Supplement_2/ii109/6421793">https://academic.oup.com/dsh/article/36/Supplement_2/ii109/6421793</a>. If you end up using whole or parts of this resource, please use the following citation(s): <ul> <li>Hengchen, S., Ros, R., and Marjanen, J. (2019). A data-driven approach to the changing vocabulary of the 'nation' in English, Dutch, Swedish and Finnish newspapers, 1750-1950. In Proceedings of the Digital Humanities (DH) conference 2019, Utrecht, The Netherlands</li> </ul> and/or: <ul> <li>Hengchen, S., Ros, R., Marjanen, J. and Tolonen, M., 2021. A data-driven approach to studying changing vocabularies in historical newspaper collections. Digital Scholarship in the Humanities, 36(Supplement_2), pp.ii109-ii126.</li> </ul> or alternatively use one of the following <code>bib</code>s: <pre><code>@inproceedings{hengchen2019nation, title="A data-driven approach to the changing vocabulary of the 'nation' in {E}nglish, {D}utch, {S}wedish and {F}innish newspapers, 1750-1950.", author={Hengchen, Simon and Ros, Ruben and Marjanen, Jani}, year={2019}, address = "Utrecht, The Netherlands", booktitle={Proceedings of the Digital Humanities (DH) conference 2019} }</code></pre> <pre><code>@article{hengchen2021data, title={A data-driven approach to studying changing vocabularies in historical newspaper collections}, author={Hengchen, Simon and Ros, Ruben and Marjanen, Jani and Tolonen, Mikko}, journal={Digital Scholarship in the Humanities}, volume={36}, number={Supplement\_2}, pages={ii109--ii126}, year={2021}, publisher={Oxford University Press} }</code></pre>   Files This archive contains two folders -- one per diachronic representation method -- as well as this README. The folders each contain four folders, which contain the models for their respective languages. As can be inferred from the small datasize, most of the earlier models are not reliable and should not be used, but are still made available. This work is licensed under a <a href="http://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>. Source material Finnish: The models were created with data from the Finnish Sub-corpus of the Newspaper and Periodical Corpus of the National Library of Finland (National Library of Finland, 2011). We used everything in the corpus. Filesizes: <pre><code>[simon@taito-login3 SGNS]$ du -h fi* 12M fi_1820_SGNS_corpus_file.gensim 89M fi_1840_SGNS_corpus_file.gensim 797M fi_1860_SGNS_corpus_file.gensim 7.0G fi_1880_SGNS_corpus_file.gensim 22G fi_1900_SGNS_corpus_file.gensim</code></pre> Swedish: The models were created with data from the Kubhist 2 corpus (Språkbanken) -- more precisely, the data dumps available at <a href="https://spraakbanken.gu.se/lb/resurser/meningsmangder/">https://spraakbanken.gu.se</a>. After a manual evaluation of Swedish embeddings trained without pre-processing seemed to show that the embeddings were of low quality, we retrained models, only keeping sentences that were at least 10 tokens long and were constituted of at least 50% of lemmas as per the KORP processing pipeline (Borin et al, 2012). Filesizes: <pre><code>[simon@taito-login3 SGNS]$ du -h sv* 1.6M sv_1740_SGNS_corpus_file.gensim 44M sv_1760_SGNS_corpus_file.gensim 124M sv_1780_SGNS_corpus_file.gensim 228M sv_1800_SGNS_corpus_file.gensim 678M sv_1820_SGNS_corpus_file.gensim 1.6G sv_1840_SGNS_corpus_file.gensim 4.5G sv_1860_SGNS_corpus_file.gensim 6.5G sv_1880_SGNS_corpus_file.gensim 113M sv_1900_SGNS_corpus_file.gensim</code></pre> Dutch: The models were created with data from the Delpher newspaper archive (Royal Dutch Library, 2017), through data dumps for newspapers until and including 1876, and through API hits for articles from 1877 to 1899 (included). <ul> <li>For anything pre-1877 we discarded full texts that had, in the metadata, anything else than exclusively <code>nl</code> or <code>NL</code> as a language tag.</li> <li>For the full texts between 1877 and 1899: we queried the API for all items in the “artikel” category that contained the determiner <code>de</code>.</li> </ul> Our assumption was that most articles should contain <code>de</code> at least once, and those that didn't were too short to be deemed interesting. A subsequent study showed that was not exactly the case, but we were reassured by the fact that left-out articles were probably "shipping or financial reports" (thanks go to Melvin Wevers). We also did not include the colonial newspapers for our embeddings. This is motivated by our research questions. A list of removed newspapers is available on request. Filesizes: <pre><code>[simon@taito-login3 SGNS]$ du -h nl* 6.8M nl_1620_SGNS_corpus_file.gensim 7.9M nl_1640_SGNS_corpus_file.gensim 43M nl_1660_SGNS_corpus_file.gensim 78M nl_1680_SGNS_corpus_file.gensim 138M nl_1700_SGNS_corpus_file.gensim 243M nl_1720_SGNS_corpus_file.gensim 287M nl_1740_SGNS_corpus_file.gensim 431M nl_1760_SGNS_corpus_file.gensim 825M nl_1780_SGNS_corpus_file.gensim 1.2G nl_1800_SGNS_corpus_file.gensim 1.8G nl_1820_SGNS_corpus_file.gensim 3.1G nl_1840_SGNS_corpus_file.gensim 5.2G nl_1860_SGNS_corpus_file.gensim 13G nl_1880_SGNS_corpus_file.gensim</code></pre> English: The models were created with data from the British Library Newspapers collection (<a href="https://www.gale.com/intl/primary-sources/british-library-newspapers%5D">link</a>), the Nichols collection (<a href="https://www.gale.com/intl/c/17th-and-18th-century-burney-newspapers-collection">link</a>), and the Burney collection (<a href="https://www.gale.com/intl/c/17th-and-18th-century-nichols-newspapers-collection">link</a>). We used everything in the corpora. For English, only SGNS_ALIGN models are available. We thank Gale Cengage for their help with this project. Filesizes: <pre><code>[simon@taito-login3 SGNS]$ du -h en* 4.3M en_1620_SGNS_corpus_file.gensim 11M en_1640_SGNS_corpus_file.gensim 11M en_1660_SGNS_corpus_file.gensim 106M en_1680_SGNS_corpus_file.gensim 409M en_1700_SGNS_corpus_file.gensim 1.7G en_1720_SGNS_corpus_file.gensim 834M en_1740_SGNS_corpus_file.gensim 2.4G en_1760_SGNS_corpus_file.gensim 5.3G en_1780_SGNS_corpus_file.gensim 5.5G en_1800_SGNS_corpus_file.gensim 15G en_1820_SGNS_corpus_file.gensim 42G en_1840_SGNS_corpus_file.gensim 65G en_1860_SGNS_corpus_file.gensim 88G en_1880_SGNS_corpus_file.gensim 26G en_1900_SGNS_corpus_file.gensim 21G en_1920_SGNS_corpus_file.gensim 6.3G en_1940_SGNS_corpus_file.gensim</code></pre> Word embeddings For every language, we train diachronic embeddings as follows. We divide the data in 20-year time bins. We train SGNS_UPDATE and SGNS_ALIGN models. Current research on German (Schlechtweg et al, 2019) and English (Shoemark et al, 2019) indicates you should use the SGNS_ALIGN models. For EN, FI, NL, no tokens (including punctuation) were removed nor altered, aside from lowercasing. For SV, see above. Parameters are as follows: SGNS architecture (Mikolov et al 2013), window size of 5, frequency threshold of 100, 5 epochs, 300 dimensions (or 100 for EN). <ul> <li>For SGNS_UPDATE: We first train a model for the first time bin <code>t</code>. To train the model for <code>t+1</code>, we use the <code>t</code> model to initialise the vectors for <code>t+1</code>, set the learning rate to correspond to the end learning rate of <code>t</code>, and continue training. This approach, closely following Kim et al (2014), has the advantage of avoiding the need for post-training vector space alignment.</li> </ul> The Python snippet below, which makes use of gensim (Rehurek and Sojka, 2010), illustrates the approach. Special thanks go to Sara Budts. <pre><code>## dict_files[key] is a dictionary with double decades as keys and a corresponding LineSentence object as value: https://radimrehurek.com/gensim/models/word2vec.html#gensim.models.word2vec.LineSentence count = 0 for key in sorted(list(dict_files.keys())): if count == 0: ## This is the first model. model = gensim.models.Word2Vec(corpus_file=dict_files[key], min_count=100, sg=1 ,size=300, workers=64, seed=1830, iter=5) model.save(os.path.join(data_path_final,"KIM",lang+"_"+str(timebin)+".w2v")) print("Model saved, on to the next\n") count += 1 if count > 0: ## this is for the subsequent models. print("model for double decade starting in",str(key)) model = gensim.models.Word2Vec.load(os.path.join(data_path_final,"KIM",lang+"_"+str(timebin-20)+".w2v")) print("previous model loaded") model.build_vocab(corpus_file=dict_files[key], update=True) model.train(corpus_file=dict_files[key], total_words = model.corpus_count, total_examples = model.corpus_count, start_alpha = model.alpha, end_alpha = model.min_alpha, epochs=model.epochs) model.save(os.path.join(data_path_final,"KIM",lang+"_"+str(timebin)+".w2v")) </code></pre> <ul> <li>For SGNS_ALIGN: We independently train models for all time bins. The models in this repository are NOT aligned, leaving you the choice of how to align them. For example, <a href="https://gist.github.com/quadrismegistus/09a93e219a6ffc4f216fb85235535faf">here</a> is a link to code by Ryan Heuser to do just that. Models were trained with the <code>count == 0</code> scenario in the snippet above.</li> </ul> Acknowledgments This work has been supported by the European Union's Horizon 2020 research and innovation programme under grant 770299 <a href="https://www.newseye.eu/">NewsEye</a>. Specials thanks go to the data providers/collection-holding institutions: the Finnish Language Bank, the Swedish Language Bank, the Royal Dutch Library, and Gale Cengage. The authors would like to thank the following persons and group, listed alphabetically: Antoine Doucet, Antti Kanner, Axel-Jean Caurant, Dominik Schlechtweg, Eetu Mäkelä, Elaine Zosa, Estelle Bunout, Haim Dubossarsky, Joris van Eijnatten, Krister Lindén, Lars Borin, Lidia Pivovarova, Melvin Wevers, Nina Tahmasebi, Sara Budts, Senka Drobac, Tanja Säily, the COMHIS group, and Steven Claeyssens. Computational resources were provided by CSC – IT Center for Science Ltd. References Borin, L., Forsberg, M., Roxendal, J. (2012). Korp-the corpus infrastructure of Spräkbanken,in: LREC. pp. 474–478. Kim, Y., Chiu, Y.I., Hanaki, K., Hegde, D. and Petrov, S. (2014). Temporal Analysis of Language through Neural Language Models. ACL 2014, p.61. Mikolov, T., Chen, K., Corrado, G. and Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. National Library of Finland (2011). The Finnish Sub-corpus of the Newspaper and Periodical Corpus of the National Library of Finland, Kielipankki Version [text corpus]. Kielipankki. Retrieved from <a href="http://urn.fi/urn:nbn:fi:lb-2016050302">http://urn.fi/urn:nbn:fi:lb-2016050302</a>. Rehurek, R. and Sojka, P. (2010). Software framework for topic modelling with large corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks. Royal Dutch Library (2017). Delpher open krantenarchief (1.0). Den Haag, 2017. Schlechtweg D., Hätty A, del Tredici M., and Schulte im Walde S. (2019). A Wind of Change: Detecting and Evaluating Lexical Semantic Change across Times and Domains. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Florence, Italy. ACL. Shoemark, P., Liza, F.F., Nguyen, D., Hale, S. and McGillivray, B. (2019). Room to Glo: A Systematic Comparison of Semantic Change Detection Approaches with Word Embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (pp. 66-76), Hong Kong. Språkbanken. The Kubhist Corpus. Department of Swedish, University of Gothenburg. <a href="https://spraakbanken.gu.se/korp/?mode=kubhist">https://spraakbanken.gu.se/korp/?mode=kubhist</a>.

opencc-by-4.0Oct 2022View details →

Diachronic word embeddings from 19th-century newspapers digitised by the British Library (1800-1919)

Word vectors related to the paper Machines in the media: semantic change in the lexicon of mechanization in 19th-century British newspapers by Nilo Pedrazzini and Barbara McGillivray (2022). The embeddings were trained on a 4.2-billion-word corpus of 19th-century British newspapers using Word2Vec and the following parameters: <pre><code>sg = True min_count = 1 window = 3 vector_size = 200 epochs = 5</code></pre> The embeddings are divided into periods of ten years each, with the vectors from each decade aligned to the ones from the most recent decade (1910s) using Orthogonal Procrustes. See related GitHub repository for the full documentation: <a href="https://github.com/Living-with-machines/DiachronicEmb-BigHistData">https://github.com/Living-with-machines/DiachronicEmb-BigHistData</a> Project webpage (Living with Machines): <a href="https://livingwithmachines.ac.uk/">https://livingwithmachines.ac.uk/</a>

opencc-by-4.0Aug 2019View details →

Survey of digitized newspaper interfaces (dataset and notebooks)

This record contains the datasets and jupyter notebooks which support the analysis presented in the paper "Historical Newspaper User Interfaces: A Review". Please refer to the paper or the github repository for more information (see links below), or do not hesitate to contact us!

opencc-zeroOct 2021View details →

trove-newspaper-issues

This dataset contains information about the published issues of newspapers digitised and made available through Trove. The data was harvested from the Trove API, using <a href="https://glam-workbench.net/trove-newspapers/harvest_newspaper_issues/">this notebook in the GLAM Workbench</a>. There are two data files: <ul> <li><code>newspaper_issues_totals_by_year.csv</code> – the total number of newspaper issues per year for each digitised newspaper</li> <li><code>newspaper_issues.csv</code> – a complete list of newspaper issues available from Trove</li> </ul> <h2>newspaper_issues_totals_by_year.csv</h2> The dataset contains the following columns: <table> <tbody> <tr> <td>Column</td> <td>Contents</td> </tr> <tr> <td><code>title</code></td> <td>newspaper title</td> </tr> <tr> <td><code>title_id</code></td> <td>newspaper id</td> </tr> <tr> <td><code>state</code></td> <td>place of publication</td> </tr> <tr> <td><code>year</code></td> <td>year published</td> </tr> <tr> <td><code>issues</code></td> <td>number of issues</td> </tr> </tbody> </table> <h2>newspaper_issues.csv</h2> The dataset contains the following columns: <table> <tbody> <tr> <td>Column</td> <td>Contents</td> </tr> <tr> <td><code>title</code></td> <td>newspaper title</td> </tr> <tr> <td><code>title_id</code></td> <td>newspaper id</td> </tr> <tr> <td><code>state</code></td> <td>place of publication</td> </tr> <tr> <td><code>issue_id</code></td> <td>issue identifier</td> </tr> <tr> <td><code>issue_date</code></td> <td>date of publication (YYYY-MM-DD)</td> </tr> </tbody> </table> To keep the file size down, I haven't included an <code>issue_url</code> in this dataset, but these are easily generated from the <code>issue_id</code>. Just add the <code>issue_id</code> to the end of <code>http://nla.gov.au/nla.news-issue</code>. For example: <a href="http://nla.gov.au/nla.news-issue495426">http://nla.gov.au/nla.news-issue495426</a>. Note that when you follow an issue url, you actually get redirected to the url of the first page in the issue.

opencc-by-4.0May 2023View details →

Decade-level Word2Vec models from automatically transcribed 19th-century newspapers digitised by the British Library (1800-1919)

Word embeddings trained on a 4.2-billion-word corpus of 19th-century British newspapers using Word2Vec and the following parameters: <pre><code>sg = True min_count = 5 window = 5 vector_size = 100 epochs = 5</code></pre> The embeddings are divided into periods of ten years each. Unlike those in <a href="https://doi.org/10.5281/zenodo.7181682">this repository</a>, these were not aligned and OCR errors skimmed from the vocabulary.  See related GitHub repository for the full documentation: <a href="https://github.com/Living-with-machines/DiachronicEmb-BigHistData">https://github.com/Living-with-machines/DiachronicEmb-BigHistData</a> Project website (Living with Machines): <a href="https://livingwithmachines.ac.uk/">https://livingwithmachines.ac.uk/</a>

openother-ncJan 2021View details →

Datasets and Models for Historical Newspaper Article Segmentation

This record contains the datasets and models used and produced for the work reported in the paper "Combining Visual and Textual Features for Semantic Segmentation of Historical Newspapers" (<a href="https://infoscience.epfl.ch/record/282863?ln=en">link</a>). Please cite this paper if you are using the models/datasets or find it relevant to your research: <pre><code>@article{barman_combining_2020, title = {{Combining Visual and Textual Features for Semantic Segmentation of Historical Newspapers}}, author = {Raphaël Barman and Maud Ehrmann and Simon Clematide and Sofia Ares Oliveira and Frédéric Kaplan}, journal= {Journal of Data Mining \& Digital Humanities}, volume= {HistoInformatics} DOI = {10.5281/zenodo.4065271}, year = {2021}, url = {https://jdmdh.episciences.org/7097}, }</code></pre> Please note that this record contains data under different licenses. 1. DATA <ul> <li>Annotations (json files): JSON files contains image annotations, with one file per newspaper containing region annotations (label and coordinates) in VIA format. The following licenses apply: <ul> <li> luxwort.json: those annotations are under a <a href="https://creativecommons.org/publicdomain/zero/1.0/legalcode">CC0 1.0 license</a>. Please refer to the right statement specified for each image in the file.</li> <li>GDL.json, IMP.json and JDG.json: those annotations are under a <a href="https://creativecommons.org/licenses/by-sa/4.0/legalcode">CC BY-SA 4.0 license</a>.</li> </ul> </li> </ul>   <ul> <li>Image files: The archive images.zip contains the Swiss titles image files (GDL, IMP, JDG) used for the experiments described in the paper. Those images are under copyright (property of the journal Le Temps and of ArcInfo) and can be used for academic research or educational purposes only. Redistribution, publication or commercial use are not permitted. These terms of use are similar to the following right statement: <a href="http://rightsstatements.org/vocab/InC-EDU/1.0/">http://rightsstatements.org/vocab/InC-EDU/1.0/</a></li> </ul>   2. MODELS Some of the best models are released under a <a href="https://creativecommons.org/licenses/by-sa/4.0/">CC BY-SA 4.0</a> license (they are also available as assets of the current Github <a href="https://github.com/dhlab-epfl/dhSegment-text/releases/tag/0.1">release</a>). <ul> <li>JDG_flair-FT: this model was trained on JDG using french Flair and FastText embeddings. It is able to predict the four classes presented in the paper (<code>Serial</code>, <code>Weather</code>, <code>Death notice</code> and <code>Stocks</code>).</li> <li>Luxwort_obituary_flair-bpemb: this model was trained on Luxwort using multilingual Flair and Byte-pair embeddings. It is able to predict the <code>Death notice</code> class.</li> <li>Luxwort_obituary_flair-FT_indomain: this model was trained on Luxwort using in-domain Flair and FastText embeddings (trained on Luxwort data). It is also able to predict the <code>Death notice</code> class.</li> </ul> Those models can be used to predict probabilities on new images using the same code as in the original <a href="https://github.com/dhlab-epfl/dhSegment">dhSegment</a> repository. One needs to adjust three parameters to the <code>predict</code> function: 1) <code>embeddings_path</code> (the path to the embeddings list), 2) <code>embeddings_map_path</code>(the path to the compressed embedding map), and 3) <code>embeddings_dim</code> (the size of the embeddings). Please refer to the paper for further information or contact us.   3. CODE:  <a href="https://github.com/dhlab-epfl/dhSegment-text">https://github.com/dhlab-epfl/dhSegment-text</a> 4. ACKNOWLEDGEMENTS We warmly thank the journal <a href="https://letemps.ch">Le Temps</a> (owner of La Gazette de Lausanne and the Journal de Genève) and the group <a href="https://www.arcinfo.ch/">ArcInfo</a> (owner of L'Impartial) for accepting to share the related datasets for academic purposes. We also thank the <a href="https://bnl.public.lu/fr.html">National Library of Luxembourg</a> for its support with all steps related to the Luxemburger Wort annotation release. This work was realized in the context of the <a href="https://impresso-project.ch">impresso - Media Monitoring of the Past</a> project and supported by the Swiss National Science Foundation under grant CR- SII5_173719. 5. CONTACT Maud Ehrmann (EPFL-DHLAB) Simon Clematide (UZH)

opencc-by-4.0May 2021View details →

Gado2: multilingual newspapers from the Netherlands Indies

This Handwritten Text Recognition (HTR) xml-page file dataset contains the ground truths of the Gado2 named entity processing application for newspapers from the Netherlands Indies and Indonesia, see: https://github.com/KBNLresearch/gado2. Optical Character Recognition (OCR) resulted in high Character Error Rates (CER) due to the inferior quality of many scans. In contrast, HTR led to CERs below 0.5 percent thus increasing the efficiency of the NER engine. All uploaded files are free of errors and fully tagged. A relevant knowledge base of Indonesian persons, places and organisations is attached in json format for entity linking.

opencc-by-4.0Mar 2021View details →

NewsEye / READ AS training dataset from French Newspapers (19th, early 20th C.)

The dataset comprises French newspaper pages from 19th and early 20th century with annotated text. The page images were provided by the <a href="https://www.bnf.fr/en">French National Library</a> and comprise 183 pages (training set). The data are formed according to the PAGE format (cf. Cf. <a href="https://github.com/PRImA-Research-Lab/PAGE-XML/">https://github.com/PRImA-Research-Lab/PAGE-XML/</a>) and were produced with the <a href="http://read.transkribus.eu/">Transkribus </a>platform with support of the <a href="http://newseye.eu/">NewsEye</a> and the <a href="http://read.transkribus.eu/">READ </a>project. The guidelines with which the AS GT was created are uploaded here as well.

openother-openJan 2022View details →

19th Century United States Newspaper images predicted as Photographs with labels for "human", "animal", "human-structure" and "landscape"

The Dataset contains images derived from the Newspaper Navigator (news-navigator.labs.loc.gov/), a dataset of images drawn from the Library of Congress Chronicling America collection (<a href="https://chroniclingamerica.loc.gov/">chroniclingamerica.loc.gov/</a>).  <blockquote> [The Newspaper Navigator dataset] consists of extracted visual content for 16,358,041 historic newspaper pages in Chronicling America. The visual content was identified using an object detection model trained on annotations of World War 1-era Chronicling America pages, including annotations made by volunteers as part of the <a href="https://labs.loc.gov/work/experiments/beyond-words/">Beyond Words</a> crowdsourcing project. source:<a href="https://news-navigator.labs.loc.gov/"> https://news-navigator.labs.loc.gov/</a> </blockquote> One of these categories is 'photographs'. This dataset contains a sample of these images with additional labels indicating if the photograph has one or more of the following labels: "human", "animal", "human-structure" and "landscape" The data is organised as follows: <ul> <li>The images themselves can be found in `images.zip`</li> <li>`newspaper-navigator-sample-metadata.csv` contains metadata about each image drawn from the Newspaper Navigator Dataset.</li> <li>`multi_label.csv` contains the labels for the images as a CSV file</li> <li>`annotations.csv` conains the labels for the images with additional metadata</li> </ul> This dataset was created for use in an under-review Programming Historian tutorial (<a href="http://programminghistorian.github.io/ph-submissions/lessons/computer-vision-deep-learning-pt2">http://programminghistorian.github.io/ph-submissions/lessons/computer-vision-deep-learning-pt2</a>) The primary aim of the data was to provide a realistic example dataset for teaching computer vision for working with digitised heritage material. The data is shared here since it may be useful for others. This data documentation is a work in progress and will be updated when the Programming Historian tutorial is released publicly. The metadata CSV file contains the following columns: - filepath - pub_date - page_seq_num - edition_seq_num - batch - lccn - box - score - ocr - place_of_publication - geographic_coverage - name - publisher - url - page_url - month - year - iiif_url

opencc-by-4.0Jan 2022View details →

The ALPIN Sentiment Dictionary: Austrian Language Polarity in Newspapers

These datasets are part of the submitted paper for the LREC2022 conference entitled: "The ALPIN Sentiment Dictionary: Austrian Language Polarity in Newspapers" The various data sources, as well as the methodology, are explained in detail in the research paper which will be available soon. ALPIN stands for Austrian Language Polarity in Newspapers. The dictionary consists of three different parts which were merged together: <ul> <li>Austrian Media Corpus: AMC (AMC_v1.0.csv)</li> <li>STANDARD posts: STP (STP_v1.0.csv)</li> <li>Austriacisms: AUT (AUT_v1.0.csv)</li> </ul> Austrian Media Corpus (AMC) (Ransmayr et al., 2017) & STANDARD posts (STP) (Schabus et al., 2017) rely on the SPLM algorithm as used in SentiDraw (Sharma & Dutta 2021). Austriacisms (AUT) was generated by using the Best-Worst scaling (BWS) (Kiritchenko and Mohammad, 2017b). The AUT list was collected from the “Variantenwörterbuch des Deutschen” (Ammon et al., 2016) (thereby only selecting those words that only surface in Austrian German and in no other variety of German) and an austriacism list of Wikipedia (https://de.wikipedia.org/wiki/Liste_von_Austriazismen). The scores are scaled to the interval [-1, 1] using the min-max-abs scaling, ranging from negative to positive. References: Sharma, S. S., & Dutta, G. (2021). SentiDraw: Using star ratings of reviews to develop domain specific sentiment lexicon for polarity determination. Information Processing & Management, 58(1), 102412. Kiritchenko, S. and Mohammad, S. M. (2017b). Capturing reliable fine-grained sentiment associations by crowdsourcing and best-worst scaling. Schabus, D., Skowron, M., & Trapp, M. (2017). One Million Posts: A Data Set of German Online Discussions. Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, 1241–1244. https://doi.org/10.1145/3077136.3080711 Ransmayr, J., Mörth, K., & Ďurčo, M. (2017). AMC (Austrian Media Corpus). In Korpusbasierte Forschungen zum österreichischen Deutsch. In Digitale Methoden der Korpusforschung in Österreich (= Veröffentlichungen zur Linguistik und Kommunikationsforschung Nr. 30) (pp. 27–38). Verlag der Österreichischen Akademie der Wissenschaften. Ammon, U., Bickel, H., & Ebner, J. (2016). Variantenwörterbuch des Deutschen : die Standardsprache in Österreich, der Schweiz, Deutschland, Liechtenstein, Luxemburg, Ostbelgien und Südtirol sowie Rumänien, Namibia und Mennonitensiedlungen. Walter de Gruyter.

opencc-by-sa-4.0Sep 2022View details →

Newspapers_09/29/22

Documentation material from the Mastic pilot of the Mingei project

opencc-by-sa-4.0Mar 2018View details →

KB newspaper models for ShiCo

The word2vec models were created from the KB news paper archive collection (a collection of digitised newspaper, created by the National Library of the Netherlands). Each word2vec model is created using newspaper articles published within a 10 year time frame. These models were created using the Gensim word2vec implementation with default parameters: Skip-gram architecture, with hierarchical softmax (no negative sampling), vector dimensionality of 300, window size of 5, and minimum word frequency of 5. Models are saved in binary word2vec format.

opencc-by-4.0Aug 2021View details →

Named-Entity Recognition for Modern Tibetan Newspapers: Tagset, Guidelines and Training Data

This dataset, tagset and guidelines were the output of a six-month incubator project on the feasibility of developing Named-Entity Recognition (NER) for modern Tibetan, primarily for use with contemporary Tibetan-language newspapers and media published inside the PRC. The project was carried out by the Mongolian and Inner Asian Studies Unit at Cambridge University’s Department of Social Anthropology. It was funded by an incubator grant from Cambridge Language Sciences. The project title was “Named-Entity Recognition in Tibetan and Mongolian Newspapers.” The Project PI was Dr Hildegard Diemberger (Cambridge), the Coordinator and Lead Author was Dr Robert Barnett (SOAS), and Senior Advisers were Dr Nathan Hill (SOAS), Dr Marieke Meelen (Cambridge), and Dr Thomas White (Cambridge).  Although some forms of NER and other NLP procedures have been developed within China for modern Tibetan (see Liu, Nuo et al, 2011), the data underlying those initiatives have not been made publicly available and their findings cannot be tested or reproduced. Significant work on developing NLP for Tibetan has been carried out outside China, but has focused largely on classical Tibetan and religious texts (see Hill & Garrett, Edward, 2017).  The Cambridge incubator project therefore produced a tagset, guidelines and training data for developing NER for modern Tibetan, with a focus on historical and political analysis of contemporary newspapers, media and other public documents in Tibetan. We compiled 3.11m syllables of data in Tibetan extracted from articles downloaded from Chinese-language news aggregator sites within China, primarily tibet.cpc.people.com.cn and tibet.people.com.cn. From this data, we selected texts containing 280,000 syllables in Tibetan, grouped in 26,000 utterances/sentences (available on request). Using Lighttag, an online annotation site, we developed a tagset for NER consisting of 17 tags (and one for wrong segmentation if using segmented data). We annotated approximately 186,000 syllables, leading to 9,884 annotations. Of these, after discounting flawed data, we produced training data containing c.6,700 annotations.  We carried out the secondary, manual review offline (for our method of converting Lighttag data for offline review, see the attached report “Using Spreadsheets to Review Annotations Offline.pdf”), and found an error rate of 3.6%. The final total of reviewed annotations was 6,624.  The dataset, tagset, guidelines and reports were developed and documented by Robert Barnett, with assistance from Tsering Samdrup, Dr Hill and Dr Meelen. Primary annotation was by Tsering Samdrup, assisted by Dr Barnett. The datasets published here include:        <ol> <li>The tagseet guidelines and annotation manual, including the 17-tag tagset, guidelines, and recommendations ("NER for Modern Tibetan-tagset and guidelines.pdf").</li> <li>The tagged training data in .csv format ("Tibetan NER Training Data-tagged, reviewed wth context-v10-UTF-8.csv") and .xls format ("Tibetan NER Training Data-tagged with context-v10-UTF-8.xlsx"). This includes 6,624 reveiwed annotations, arranged according to the Tibetan alphabet together with the tags and context (utterance) for each annotation.</li> <li>The raw annotation results downloaded from Lighttag as .json files ("Raw Training Data for NER in Modern Tibetan -Jobs2-11-JSON.zip") and as .xls files ("Training Data for NER in Modern Tibetan -Jobs2-11-XLS.zip"). These include 10 "tasks" or datasets of articles scraped from Tibetan-language websites within Tibet.    </li> <li>A guide to preparing Lighttag annotation results for manual review offline (“Using Spreadsheets to Review Annotations Offline.pdf”).</li> </ol> The project's findings regarding the status of NER and NLP for vertical Mongolian are available at DOI: 10.5281/zenodo.5103499.

opencc-by-4.0May 2023View details →

Diachronic and diatopic word embeddings from newspapers digitised by the British Library (1830-1889): North and South England

Diachronic word embeddings (decade-level) trained with Word2Vec (via Gensim) on different geographic subcorpora of the Heritage Made Digital British and the Living with Machines historical newspaper collections: - North England (north.zip) - South England (south.zip) At the moment, for each subcorpus, Word2Vec models are available for each decade in the period 1830-1889. More models are on the way for the following: - each decade in the periods 1780-1829 and 1890-1920 for both North and South England. - diachronic models for the following regions: Scotland, Wales, and Midlands. The models were trained using the following parameters: <pre><code>sg = True min_count = 1 window = 5 vector_size = 200 epochs = 5</code></pre> Like the embeddings in <a href="https://doi.org/10.5281/zenodo.7181682">this repository</a>, the model for each decade was aligned to the most recent one with Orthogonal Procrustes. See related GitHub repository for the full documentation: <a href="https://github.com/Living-with-machines/DiachronicEmb-BigHistData">https://github.com/Living-with-machines/DiachronicEmb-BigHistData</a>. Project website (Living with Machines): <a href="https://livingwithmachines.ac.uk/">https://livingwithmachines.ac.uk/</a> Data related to: Nilo Pedrazzini & Barbara McGillivray, Diachronic and diatopic word embeddings from British historical newspapers, presented at AIUCD (Convegno dell’Associazione per l’Informatica Umanistica e la Cultura Digitale) in Siena (Italy), June 2023.

opencc-by-4.0May 2023View details →

Argentina news from Pagina12 Newspaper

This dataset contains information from the web page of the Argentinian newspaper "Pagina12" (https://www.pagina12.com.ar/). The dataset was developed with a Python code in the context of academical studies (Data Science master). The final purpose was only for academic research. The dataset includes different fields of an article link of the cited newspaper, specifically the code was implemented in Python using Srapy library. Hence, the code scrap the final link of each notice or new from the different sections. For each notice or new, the following fields have been extracted: title, date, author, section and url of the new.

opencc-by-4.0Oct 2023View details →

Multilingual Coarse Political Stance Classification of Media. The Editorial Line of a ChatGPT and Bard Newspaper

Corpora used in the publication: <ul> <li>Cristina España-Bonet. 2023. Multilingual Coarse Political Stance Classification of Media. The Editorial Line of a ChatGPT and Bard Newspaper. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore. Pages 11757–11777. Association for Computational Linguistics.</li> </ul> Three corpora are included: <ol> <li>Newspaper articles extracted from the OSCAR corpus in English, German, Spanish and Catalan automatically annotated for political stance (left vs right) and topic</li> <li>Newspaper-like article generations by different versions of ChatGPT for 101 topics in the 4 languages</li> <li>Newspaper-like article generations by Bard for 101 topics in the 4 languages</li> </ol> See the README file and the original article for further details.

opencc-by-4.0Nov 2020View details →

NewsEye / READ OCR training dataset from French Newspapers (18th, 19th, early 20th C.)

The dataset comprises French newspaper pages from 18th, 19th and early 20th century with carefully corrected text. The page images were provided by the <a href="https://www.bnf.fr/en">French National Library</a> and comprise 127 pages (training set) and 8 pages (validation set). The data are formed according to the PAGE format (cf. Cf. <a href="https://github.com/PRImA-Research-Lab/PAGE-XML/">https://github.com/PRImA-Research-Lab/PAGE-XML/</a>) and were produced with the <a href="http://read.transkribus.eu/">Transkribus </a>platform with support of the <a href="http://newseye.eu/">NewsEye</a> and the <a href="http://read.transkribus.eu/">READ </a>project.

opencc-by-4.0Apr 2024View details →

BioPropaPhenKG on Online Newspapers and Medical Articles

The coronavirus disease (COVID-19) spread rampantly around the world at the beginning of 2020 before the governments of each country could prevent it by making decisions based on medical data analysis. With proper formalization, the terabytes of new textual data available online every day could have been used for the early description and detection of cases of this virus. Since then, the number of Event-Based Surveillance (EBS) applications has increased exponentially. These applications aim to mine channels of unstructured information to detect signs of possible public health events' progression. However, one problem with such systems is the need for expert intervention to define which event will be captured, which relevant terms should be used in the search, and to analyze the events to modify the search procedure constantly. Another problem is that many of these applications do not consider both spatial and temporal characteristics. Addressing such limitations, this datasets presents a novel approach. We propose the use of BioPropaPhenKG to replace such systems. In this dataset, BioPropaPhen was enhanced with information comming from unstructured texts from online newspapers and medical articles. BioPropaPhenKG, its ontology and other useful information can be found in <a href="../records/10911980">https://zenodo.org/records/10911980</a>. The code used for this use case can be found in <a href="https://github.com/Gabriel382/DDPF-Health-Risks">https://github.com/Gabriel382/DDPF-Health-Risks</a> . Finally, the datasets used where UMLS MetamorphoSys, OpenStreetMaps, Wikidata, <a href="https://aylien.com/blog/free-coronavirus-news-dataset">Aylien</a> (data only from November of 2019) and <a href="https://allenai.org/data/cord-19">CORD-19</a> (data only from December of 2019).    To read, you just need to load it with Neo4j:4.4.3. Alternatively, you can open it with docker using the following command:  docker run --interactive --tty --rm \     --publish=7474:7474 --publish=7687:7687 \     --volume=/path-to-data-folder:/data --user="$(id -u):$(id -g)"\     neo4j:4.4.3 \ neo4j-admin load --from=/data/BioPropaPhenKG-Journal-Medical.dump --database "neo4j" --force

opencc-by-4.0Apr 2024View details →

BioPropaPhenKG on Multi-Relation Extraction Methods on Online Newspapers

The coronavirus disease (COVID-19) spread rampantly around the world at the beginning of 2020 before the governments of each country could prevent it by making decisions based on medical data analysis. With proper formalization, the terabytes of new textual data available online every day could have been used for the early description and detection of cases of this virus. Since then, the number of Event-Based Surveillance (EBS) applications has increased exponentially. These applications aim to mine channels of unstructured data to detect signs of possible public health events. However, one problem with such systems is the need for expert intervention to define which event will be captured, which relevant terms should be used in the search, and to analyze the events to modify the search procedure constantly. Another problem is that many of these applications do not consider both spatial and temporal characteristics. Addressing such limitations, this article presents a novel approach. We propose the biomedical domain specialization of the Core Propagation Phenomenon Ontology (PropaPhen) to capture spatiotemporal characteristics of the propagation of health-related phenomena. We also propose the Description-Detection-Framework (DDF), which leverages PropaPhen, UMLS, and OpenStreetMaps to detect new medical events automatically. Finally, we demonstrate a use case with experiments on extracts from online newspapers about COVID-19. The results show that DDF can be useful for detecting clusters of suspicious cases of possible emerging health-related phenomena. BioPropaPhenKG, its ontology and other useful information can be found in <a href="../records/10911980">https://zenodo.org/records/10911980</a>. The code used for this use case can be found in <a href="https://github.com/Gabriel382/DDPF-Health-Risks">https://github.com/Gabriel382/DDPF-Health-Risks</a> . Finally, the datasets used where UMLS MetamorphoSys, OpenStreetMaps, Wikidata, <a href="https://aylien.com/blog/free-coronavirus-news-dataset">Aylien</a> (data only from November of 2019).   To read, you just need to load it with Neo4j:4.4.3. Alternatively, you can open it with docker using the following command:  docker run --interactive --tty --rm \     --publish=7474:7474 --publish=7687:7687 \     --volume=/path-to-data-folder:/data --user="$(id -u):$(id -g)"\     neo4j:4.4.3 \ neo4j-admin load --from=/data/BioPropaPhenKG-Journal-MultiRE.dump --database "neo4j" --force