Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
34
datasets available to search
ShareScore release 0.9.0
Dataset results
34 results for “multilingual dataset”
Multilingual news article similarity dataset
<p>This dataset contains the extended version of the authors' earlier work: <a href="../records/6507872">https://zenodo.org/records/6507872,</a> where pairs of news articles drawn from the first half of 2020 are annotated for seven aspects of similarity in the original version as well as an additional FRAME aspect:</p> <ul> <li><strong>GEO</strong>: How similar is the geographic focus (places, cities, countries, etc.) of the two articles?</li> <li><strong>ENT:</strong> How similar are the named entities (e.g., people, companies, organizations, products, named living beings), excluding previously considered locations appearing in the two articles?</li> <li><strong>TIME</strong> Are the two articles relevant to similar time periods or describing similar time periods?</li> <li><strong>NAR</strong> How similar are the narrative schemas presented in the two articles?</li> <li><strong>OVERALL</strong> Overall, are the two articles covering the same substantive news story? (excluding style, framing, and tone)</li> <li><strong>STYLE</strong> Do the articles have similar writing styles?</li> <li><strong>TONE</strong> Do the articles have similar tones?</li> <li><strong>FRAME</strong> Do the articles have similar framing and express similar opinions?</li> </ul>
Curlie Enhanced with LLM Annotations: Two Datasets for Advancing Homepage2Vec's Multilingual Website Classification
<h3>Advancing Homepage2Vec with LLM-Generated Datasets for Multilingual Website Classification</h3> <p>This dataset contains two subsets of labeled website data, specifically created to enhance the performance of Homepage2Vec, a multi-label model for website classification. The datasets were generated using Large Language Models (LLMs) to provide more accurate and diverse topic annotations for websites, addressing a limitation of existing Homepage2Vec training data.</p> <p><strong>Key Features:</strong></p> <ul> <li><strong>LLM-generated annotations:</strong> Both datasets feature website topic labels generated using LLMs, a novel approach to creating high-quality training data for website classification models.</li> <li><strong>Improved multi-label classification:</strong> Fine-tuning Homepage2Vec with these datasets has been shown to improve its macro F1 score from 38% to 43% evaluated on a human-labeled dataset, demonstrating their effectiveness in capturing a broader range of website topics.</li> <li><strong>Multilingual applicability:</strong> The datasets facilitate classification of websites in multiple languages, reflecting the inherent multilingual nature of Homepage2Vec.</li> </ul> <p><strong>Dataset Composition:</strong></p> <ul> <li><strong>curlie-gpt3.5-10k:</strong> 10,000 websites labeled using GPT-3.5, context 2 and 1-shot</li> <li><strong>curlie-gpt4-10k:</strong> 10,000 websites labeled using GPT-4, context 2 and zero-shot</li> </ul> <p><strong>Intended Use:</strong></p> <ul> <li>Fine-tuning and advancing Homepage2Vec or similar website classification models</li> <li>Research on LLM-generated datasets for text classification tasks</li> <li>Exploration of multilingual website classification</li> </ul> <p><strong>Additional Information:</strong></p> <ul> <li><strong>Project and report repository:</strong> https://github.com/CS-433/ml-project-2-mlp</li> </ul> <p><strong>Acknowledgments:</strong></p> <p>This dataset was created as part of a project at EPFL's Data Science Lab (DLab) in collaboration with <a href="https://people.epfl.ch/robert.west">Prof. Robert West</a> and <a href="https://tizianopiccardi.github.io/" rel="nofollow">Tiziano Piccardi.</a></p>
Multilingual Wikidata Property Translation Flow Dataset
<p>Multilingual Wikidata property translation Flow dataset contains the translation flow of Wikidata properties as collected on July 7, 2019. It contains four columns: timestamp, property, language, type. Every line in the dataset corresponds to the <strong>first-time action</strong> related to a language, i.e., the time at which the first translation of a label, description or an alias was made in a given language for a given property.</p> <p>Taking an example line from this dataset,</p> <p><em>2013-09-10T22:43:54Z,P856,en,label </em></p> <p>corresponds to the action that an <em>English</em> <em>label</em> of Property <em>P856</em> was added for the first time at <em>2013-09-10T22:43:54Z</em>.</p> <p>Following is a description of each column.</p> <p>1. timestamp: the time at which an action was made. For example, <em>2013-09-10T22:43:54Z</em></p> <p>2. property: Wikidata property identifier. It uses the P-number, For example, <em>P856</em></p> <p>3. language: the language in which a label/description/alias was first translated</p> <p>4. type: It could be one of the following values: label, description and alias</p>
Wikipedia Multilingual Vandalism Detection Dataset
<p>This dataset accompanies a research paper that introduces a novel system designed to support the Wikipedia community in combating vandalism on the platform. The dataset has been prepared to enhance the accuracy and efficiency of Wikipedia patrolling in multiple languages.</p> <p>The release of this comprehensive dataset aims to encourage further research and development in vandalism detection techniques, fostering a safer and more inclusive environment for the Wikipedia community. Researchers and practitioners can utilize this dataset to train and validate their models for vandalism detection and contribute to improving online platforms' content moderation strategies.</p> <p><strong>Dataset Details:</strong></p> <ul> <li><strong>Number of Languages:</strong> 47</li> <li><strong>Observation period: </strong>6 months training, one week hold-out testing</li> <li><strong>Use Case:</strong> The dataset is primarily intended for training and evaluating vandalism detection systems.</li> <li><strong>Features:</strong> Each record characterizes the corresponding revision of the Wikipedia page, including revision metadata, user details, text inserted, removed, or changed, and corresponding MLMs-based features. </li> <li><strong>Data Filtering and Feature Engineering:</strong> Advanced filtering and feature engineering techniques were applied to ensure the dataset's quality and relevance for effectively training the vandalism detection system.</li> <li><strong>Files: </strong>Training and hold-out testing datasets of anonymous and all users. </li> </ul> <p> </p> <p><strong>Related paper citation:</strong></p> <pre><code>@inproceedings{10.1145/3580305.3599823, author = {Trokhymovych, Mykola and Aslam, Muniza and Chou, Ai-Jou and Baeza-Yates, Ricardo and Saez-Trumper, Diego}, title = {Fair Multilingual Vandalism Detection System for Wikipedia}, year = {2023}, isbn = {9798400701030}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3580305.3599823}, doi = {10.1145/3580305.3599823}, abstract = {This paper presents a novel design of the system aimed at supporting the Wikipedia community in addressing vandalism on the platform. To achieve this, we collected a massive dataset of 47 languages, and applied advanced filtering and feature engineering techniques, including multilingual masked language modeling to build the training dataset from human-generated data. The performance of the system was evaluated through comparison with the one used in production in Wikipedia, known as ORES. Our research results in a significant increase in the number of languages covered, making Wikipedia patrolling more efficient to a wider range of communities. Furthermore, our model outperforms ORES, ensuring that the results provided are not only more accurate but also less biased against certain groups of contributors.}, booktitle = {Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining}, pages = {4981–4990}, numpages = {10}, location = {Long Beach, CA, USA}, series = {KDD '23} }</code></pre> <p> </p>
CLDF dataset accompanying Chechuro et al.'s "Small-scale multilingualism through the prism of lexical borrowing" from 2021
<p>Cite the source of the dataset as:</p> <blockquote> <p>Chechuro et al. (2021) Small-scale multilingualism through the prism of lexical borrowing. International Journal of Bilingualism.</p> </blockquote>
A Multilingual Dataset of COVID-19 Vaccination Attitudes on Twitter
<p>This dataset consists of the IDs of 2,198,090 tweets collected from Western Europe, of which 17,934 are annotated with labels indicating the originators' affective vaccination stances, including Positive (PO), Negative (NG), Positive but dissatisfaction (PD), Neutral (NE) and Off-topic (OT).</p> <p>all_tweets.txt contains all the ids of the collected tweets, annotated_tweets.txt contains the ids of the annotated tweets and the categories they are annotated to.</p> <p> </p>
Common Phone: A Multilingual Dataset for Robust Acoustic Modelling
<p><em>Release Date: 17.01.22</em></p> <p><strong>Welcome to Common Phone 1.0</strong></p> <p><strong>Legal Information</strong></p> <p><em>Common Phone</em> is a subset of the <em>Common Voice</em> corpus collected by <em>Mozilla Corporation</em>. By using <em>Common Phone</em>, you agree to the <a href="https://commonvoice.mozilla.org/en/terms">Common Voice Legal Terms</a>. <em>Common Phone</em> is maintained and distributed by speech researchers at the <a href="https://lme.tf.fau.de/">Pattern Recognition Lab</a> of Friedrich-Alexander-University Erlangen-Nuremberg (<a href="https://www.fau.de/">FAU</a>) under the <a href="https://creativecommons.org/publicdomain/zero/1.0/">CC0 license</a>.</p> <p>Like for <em>Common Voice</em>, you must not make any attempt to identify speakers that contributed to <em>Common Phone</em>.</p> <p><strong>About <em>Common Phone</em></strong></p> <p>This corpus aims to provide a basis for Machine Learning (ML) researchers and enthusiasts to train and test their models against a wide variety of speakers, hardware/software ecosystems and acoustic conditions to improve generalization and availability of ML in real-world speech applications.<br> The current version of <em>Common Phone</em> comprises 116,5 hours of speech samples, collected from 11.246 speakers in 6 languages:</p> <table align="center"> <thead> <tr> <th> <p><strong>Language</strong></p> </th> <th> <p><strong>Speakers</strong></p> </th> <th> <p><strong>Hours</strong></p> </th> </tr> </thead> <tbody> <tr> <td> </td> <td> <p><code>train</code> / <code>dev</code> / <code>test</code></p> </td> <td> <p><code>train</code> / <code>dev</code> / <code>test</code></p> </td> </tr> <tr> <td> <p>English</p> </td> <td> <p>4716 / 771 / 774</p> </td> <td> <p>14.1 / 2.3 / 2.3</p> </td> </tr> <tr> <td> <p>French</p> </td> <td> <p>796 / 138 / 135</p> </td> <td> <p>13.6 / 2.3 / 2.2</p> </td> </tr> <tr> <td> <p>German</p> </td> <td> <p>1176 / 202 / 206</p> </td> <td> <p>14.5 / 2.5 / 2.6</p> </td> </tr> <tr> <td> <p>Italian</p> </td> <td> <p>1031 / 176 / 178</p> </td> <td> <p>14.6 / 2.5 / 2.5</p> </td> </tr> <tr> <td> <p>Spanish</p> </td> <td> <p>508 / 88 / 91</p> </td> <td> <p>16.5 / 3.0 / 3.1</p> </td> </tr> <tr> <td> <p>Russian</p> </td> <td> <p>190 / 34 / 36</p> </td> <td> <p>12.7 / 2.6 / 2.8</p> </td> </tr> <tr> <td> <p><strong>Total</strong></p> </td> <td> <p>8417 / 1409 / 1420</p> </td> <td> <p>85.8 / 15.2 / 15.5</p> </td> </tr> </tbody> </table> <p> </p> <p>Presented <code>train</code>, <code>dev</code> and <code>test</code> splits are <strong>not identical</strong> to those shipped with <em>Common Voice</em>. Speaker separation among splits was realized by only using those speakers that had provided age and gender information. This information can only be provided as a registered user on the website. When logged in, the session ID of contributed recordings is always linked to your user, thus we could easily link recordings to individual speakers. Keep in mind this would not be possible for unregistered users, as their session ID changes if they decide to contribute more than once.<br> During speaker selection, we considered that some speakers had contributed to more than one of the six <em>Common Voice</em> datasets (one for each language). In <em>Common Phone</em>, a speaker will only appear in one language.<br> The dataset is structured as follows:</p> <ul> <li>Six top-level directories, one for each language.</li> <li>Each language folder contains: <ul> <li>[train|dev|test].csv files listing audio files, respective speaker ID and plain text transcript.</li> <li>meta.csv provides speaker information: age group, gender, language, accent (if available) and which of the three splits this speaker was assigned to. File names match corresponding audio file names except their extension.</li> <li>/grids/ contains phonetic transcription for every audio file in Praat TextGrid format.</li> <li>/mp3/ contains audio files in mp3, identical to those of <em>Common Voice</em>, e.g., sampling rates have been preserved and may vary for different files.</li> <li>/wav/ contains raw audio files in 16 bits/sample, 16 kHz single channel. They had been created from the original mp3 audios. We provide them for convenience, keep in mind that their source had undergone MP3-compression.</li> </ul> </li> </ul> <p><strong>Where does the phonetic annotation come from?</strong></p> <p>Phonetic annotation was computed via <a href="https://clarin.phonetik.uni-muenchen.de/BASWebServices/interface/Pipeline">BAS Web Services</a>. We used the regular Pipeline (G2P-MAUS) without ASR to create an alignment of text transcripts with audio signals. We chose International Phonetic Alphabet (IPA) output symbols as they work well even in a multi-lingual setup. <em>Common Phone</em> annotation comprises 101 phonetic symbols, including silence.</p> <p><strong>Why <em>Common Phone</em>?</strong></p> <ul> <li>Large number of speakers and varying acoustic conditions to improve robustness of ML models</li> <li>Time-aligned IPA phonetic transcription for every audio sample</li> <li>Gender-balanced and age-group-matched (equal number of female/male speakers in every age group)</li> <li>Support for six different languages to leverage multi-lingual approaches</li> <li>Original MP3 files plus standard WAVE files</li> </ul> <p><strong>Is there any publication available?</strong></p> <p><em>Yes, a paper describing Common Phone in detail is currently under revision for LREC </em><em>2022. You can access a pre-print version on arXiv entitled “<a href="https://arxiv.org/abs/2201.05912">Common Phone: A Multilingual Dataset for Robust Acoustic Modelling</a>”.</em></p>
A Dataset of Multilingual Facebook Comments on Moros and Armed Conflict in the Southern Philippines
<p>This dataset is a collection of 12,478 social media comments found on the official Facebook pages of ten Philippine newspapers, T<em>he Philippine Daily Inquirer, Manila Bulletin, The Philippine Star, The Manila Times, Sunstar Cebu, Sunstar Davao, Cebu Daily News, The Freeman, Sunstar Davao, MindaNews, </em>and <em>The Mindanao Times</em>, spanning the years 2015, 2017 and 2019. The comments contain terms related to the Moro identity and the Mamasapano Clash, the Marawi Siege and the establishment of BARMM in the southern Philippines, allowing researchers to study semantic fields with regard to Muslims and the relationship between the texts and the source newspaper, their region of origin, and political administration, among other variables. All comments in the dataset were downloaded through Facebook's Graph API via Facepager (Jünger & Keyling, 2019).</p> <p>One CSV file (MMB151719SOCMED_v2.csv) is provided, along with a codebook that contains descriptions of the variables and codes used in the CSV file, and a Readme document with a changelog. </p> <p>Each social media comment is annotated with the following metadata: </p> <ol> <li><strong>object_id</strong>: identifier associated with the comment;</li> <li><strong>message</strong>: the textual string of the comment;</li> <li><strong>message_proc:</strong> the textual string of the comment after pre-processing;</li> <li><strong>lang_label:</strong> categorical value for the language of the comment (Tagalog (Filipino), Cebuano, English, Taglish, Bislog, Bislish, Trilingual or Other);</li> <li><strong>from_name</strong>: identifier of public pages (not profiles of individuals) leaving comments (NaN for profiles of individuals, 'NAME' for public pages besides the newspapers, otherwise, the page name of the newspaper);</li> <li><strong>created_time</strong>: Facebook Graph API's-generated string for the date and time the comment was posted; </li> <li><strong>month_year</strong>: categorical value in the form string+YY (e.g. Jun-15) of the month and year when the comment was posted;</li> <li><strong>year</strong>: numerical value in the form YY; </li> <li><strong>newspaper</strong>: categorical value for the newspaper Facebook page under which the comment was found;</li> <li><strong>corpus</strong>: categorical value for comments from the main corpus or the side (control) corpus;</li> <li><strong>administration</strong>: categorical value for political administration (pbsa = President Benigno Aquino III, prrd = President Rodrigo Roa Duterte);</li> <li><strong>count</strong>: numerical value referring to the number of string sequences without spaces;</li> </ol> <p>The dataset may only be used for non-commercial purposes and is licensed under the <a href="https://creativecommons.org/licenses/by-nc-sa/4.0/" target="_blank" rel="noopener">CC BY-NC-SA 4.0 DEED.</a></p> <p>_____________________________________________________________________________________</p> <p>V2 - 05/06/2024</p> <p>Corrections</p> <ul> <li>Corrections made to region to include Luzon, Visayas and Mindanao (as opposed to Mindanao, non-Mindanao);</li> <li>Corrections made to administration coding. </li> </ul> <p> </p> <p>This dataset is described by: </p> <p><span>Cruz, F. A. (2024). </span><span>A Multilingual Collection of Facebook Comments on the Moro Identity and Armed Conflict in the Southern Philippines<em>. Journal of Open Humanities Data, 10</em>(1), 41. </span><span>DOI: <a href="https://doi.org/10.5334/johd.219" target="_blank" rel="noopener">https://doi.org/10.5334/johd.219</a></span></p> <p> </p> <p><strong>Bibiliography</strong></p> <div> <div>Jünger, J., & Keyling, T. (2019). <em>Facepager: An application for automated data retrieval on the web</em> (4.5.3) [Computer software]. <a href="https://github.com/strohne/Facepager/">https://github.com/strohne/Facepager/</a></div> <div> </div> <div> </div> </div>
Dataset for the paper "Entity Insertion in Multilingual Linked Corpora: The Case of Wikipedia"
<p>Dataset for the EMNLP'24 Main conference paper titled "Entity Insertion in Multilingual Linked Corpora: The Case of Wikipedia".</p>
MultiSubs: A Large-scale Multimodal and Multilingual Dataset
<p>MultiSubs is a dataset of multilingual subtitles gathered from <a href="https://opus.nlpl.eu/OpenSubtitles.php">the OPUS OpenSubtitles dataset</a>, which in turn was sourced from <a href="http://www.opensubtitles.org/">opensubtitles.org</a>. We have supplemented some text fragments (visually salient nouns in this release) within the subtitles with web images, where the word sense of the fragment has been disambiguated using a cross-lingual approach. </p> <p>Please refer to our paper for a more detailed description of the dataset:</p> <p>Josiah Wang, Pranava Madhyastha, Josiel Figueiredo, Chiraag Lala, Lucia Specia (2021). <a href="https://arxiv.org/abs/2103.01910">MultiSubs: A Large-scale Multimodal and Multilingual Dataset</a>. CoRR, abs/2103.01910. Available at: <a href="https://arxiv.org/abs/2103.01910">https://arxiv.org/abs/2103.01910</a></p>
CLDF dataset accompanying List and Forkel's "Borrowing Detection in Multilingual Wordlists" from 2021
<p>Cite the source of the dataset as:</p> <blockquote> <p>List, Johann-Mattis and Forkel, Robert (2021): Automated identification of borrowings in multilingual wordlists. Leipzig: Max Planck Institute for Evolutionary Anthropology.</p> </blockquote>
GeoCoV19: A Dataset of Hundreds of Millions of Multilingual COVID-19 Tweets with Location Information
<p>We present GeoCoV19, a large-scale Twitter dataset related to the ongoing COVID-19 pandemic. The dataset has been collected over a period of 90 days from February 1 to May 1, 2020 and consists of more than 524 million multilingual tweets. As the geolocation information is essential for many tasks such as disease tracking and surveillance, we employed a gazetteer-based approach to extract toponyms from user location and tweet content to derive their geolocation information using the Nominatim (Open Street Maps) data at different geolocation granularity levels. In terms of geographical coverage, the dataset spans over 218 countries and 47K cities in the world. The tweets in the dataset are from more than 43 million Twitter users, including around 209K verified accounts. These users posted tweets in 62 different languages.</p>
Multilingual Persuasion Dataset
<p>This dataset contains dialogue lines from the games Knights of the Old Republic 1 & 2 and Neverwinter Nights 1. Some of the dialogue lines are marked as persuasive (which is when the player character is attempting a Persuade skill check.) </p> <p>If you use this data, please cite:</p> <p>Pöyhönen, T., Hämäläinen, M., Alnajjar, K. (2022) "Multilingual Persuasion Detection: Video Games as an Invaluable Data Source for NLP" DiGRA '22 - Proceedings of the 2022 DiGRA International Conference</p> <p>Code:</p> <p>https://github.com/Teemursu/multilingual_persuasion_dataset</p>
GLAMI-1M: A Multilingual Image-Text Fashion Dataset
<p>We introduce GLAMI-1M: the largest multilingual image-text classification dataset and benchmark. The dataset contains images of fashion products with item descriptions, each in 1 of 13 languages. Categorization into 191 classes has high-quality annotations: all 100k images in the test set and 75% of the 1M training set were human-labeled. The paper presents baselines for image-text classification showing that the dataset presents a challenging fine-grained classification problem: The best scoring EmbraceNet model using both visual and textual features achieves 69.7% accuracy. Experiments with a modified Imagen model show the dataset is also suitable for image generation conditioned on text. The dataset, source code and model checkpoints are published at: https://github.com/glami/glami-1m.</p>
GLAMI-1M: A Multilingual Image-Text Fashion Dataset - 800px
<p>We introduce GLAMI-1M: the largest multilingual image-text classification dataset and benchmark. The dataset contains images of fashion products with item descriptions, each in 1 of 13 languages. Categorization into 191 classes has high-quality annotations: all 100k images in the test set and 75% of the 1M training set were human-labeled. The paper presents baselines for image-text classification showing that the dataset presents a challenging fine-grained classification problem: The best scoring EmbraceNet model using both visual and textual features achieves 69.7% accuracy. Experiments with a modified Imagen model show the dataset is also suitable for image generation conditioned on text. The dataset, source code and model checkpoints are published at: https://github.com/glami/glami-1m</p>
FakeCovid- A Multilingual Cross domain Fact Check Dataset for COVID-19
<p>FakeCovid is the first multilingual cross-domain dataset of 7623 fact-checked news articles for COVID-19, collected from 04/01/2020 to 01/07/2020. We have collected the fact-checked articles from 92 fact-checking websites after obtaining references from Poynter and Snopes. We have manually annotated the collected articles into 11 categories of the fact-checked news according to their content. We ultimately generated dataset is in 40 languages from 105 countries. </p>
Dataset from the ISMIR2020 article "Multilingual Music Genre Embeddings for Effective Cross-Lingual Music Item Annotation"
<p>We release the data required to reproduce the cross-lingual music genre translation experiments from the article <strong><em>Multilingual Music Genre Embeddings for Effective Cross-Lingual Music Item Annotation</em></strong> presented at the <a href="https://ismir.github.io/ISMIR2020/">ISMIR 2020</a> conference.</p> <p>More information about this data and how it should be used in the experiments can be found in the GitHub repository <a href="http://github.com/deezer/MultilingualMusicGenreEmbedding">deezer/MultilingualMusicGenreEmbedding</a>.</p> <p>Please cite our paper if you use the code or data in your work.</p>
Data set for "Token-Level Multilingual Epidemic Dataset for Event Extraction"
<p>This is the data for the TPDL 2021 paper "<a href="https://zenodo.org/record/5780020">Token-Level Multilingual Epidemic Dataset for Event Extraction</a>". If you use this resource, please cite the paper:</p> <pre><code>@inproceedings{mutuvi2021dataset, title = "Token-level Multilingual Epidemic Dataset for Event Extraction", author = {Mutuvi, Stephen and Boros, Emanuela and Doucet, Antoine, and Lejeune, Gaël and Jatowt, Adam and Odeo, Moses}, booktitle = "Proceedings of the 25th International Conference on Theory and Practice of Digital Libraries, September 13–17, 2021, TPDL 2021", year = "2021", location = "Online" }</code></pre> <p> </p> <p>This work has been supported by the European Union Horizon 2020 research and innovation programme under grants 825153 (Embeddia) and 770299 (NewsEye).</p>
Multilingual named entity recognition for medieval charters. Datasets and models
<p>Annotated dataset for training named entities recognition models for medieval charters in Latin, French and Spanish.</p> <p> </p> <p>The original raw texts for all charters were collected from four charters collections</p> <p>- HOME-ALCAR corpus : <a href="https://zenodo.org/record/5600884">https://zenodo.org/record/5600884</a></p> <p>- CBMA : <a href="https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&ved=2ahUKEwisvNOa3qP3AhULyIUKHenpBoAQFnoECA0QAQ&url=http%3A%2F%2Fwww.cbma-project.eu%2F&usg=AOvVaw0blsASXzOKSNz_EkixwfJT">http://www.cbma-project.eu</a></p> <p>- Diplomata Belgica : <a href="https://www.diplomata-belgica.be">https://www.diplomata-belgica.be</a></p> <p>- CODEA corpus :<a href="http://https://corpuscodea.es/"> https://corpuscodea.es/</a></p> <p> </p> <p>We include (i) the annotated training datasets, (ii) the contextual and static embeddings trained on medieval multilingual texts and (iii) the named entity recognition models trained using two architectures: Bi-LSTM-CRF + stacked embeddings and fine-tuning on Bert-based models (mBert and RoBERTa)</p> <p>Codes, datasets and notebooks used to train models can be consulted in our gitlab repository: <a href="https://gitlab.com/magistermilitum/ner_medieval_multilingual">https://gitlab.com/magistermilitum/ner_medieval_multilingual</a></p> <p>Our best RoBERTa model is also available in the HuggingFace library: <a href="https://huggingface.co/magistermilitum/roberta-multilingual-medieval-ner">https://huggingface.co/magistermilitum/roberta-multilingual-medieval-ner</a></p>
CT-FAN: A Multilingual dataset for Fake News Detection
<p><strong>By downloading the data, you agree with the terms & conditions mentioned below:</strong></p> <p><strong>Data Access: </strong>The data in the research collection may only be used for research purposes. Portions of the data are copyrighted and have commercial value as data, so you must be careful to use them only for research purposes. </p> <p>Summaries, analyses and interpretations of the linguistic properties of the information may be derived and published, provided it is impossible to reconstruct the information from these summaries. You may not try identifying the individuals whose texts are included in this dataset. You may not try to identify the original entry on the fact-checking site. You are not permitted to publish any portion of the dataset besides summary statistics or share it with anyone else.</p> <p>We grant you the right to access the collection's content as described in this agreement. You may not otherwise make unauthorised commercial use of, reproduce, prepare derivative works, distribute copies, perform, or publicly display the collection or parts of it. You are responsible for keeping and storing the data in a way that others cannot access. The data is provided free of charge.</p> <p><strong>Citation</strong></p> <p>Please cite our work as</p> <pre>@InProceedings{clef-checkthat:2022:task3, author = {K{\"o}hler, Juliane and Shahi, Gautam Kishore and Stru{\ss}, Julia Maria and Wiegand, Michael and Siegel, Melanie and Mandl, Thomas}, title = "Overview of the {CLEF}-2022 {CheckThat}! Lab Task 3 on Fake News Detection", year = {2022}, booktitle = "Working Notes of CLEF 2022---Conference and Labs of the Evaluation Forum", series = {CLEF~'2022}, address = {Bologna, Italy},} </pre> <pre>@article{shahi2021overview, title={Overview of the CLEF-2021 CheckThat! lab task 3 on fake news detection}, author={Shahi, Gautam Kishore and Stru{\ss}, Julia Maria and Mandl, Thomas}, journal={Working Notes of CLEF}, year={2021} }</pre> <p><strong>Problem Definition:</strong> Given the text of a news article, determine whether the main claim made in the article is true, partially true, false, or other (e.g., claims in dispute) and detect the topical domain of the article. This task will run in <strong>English and German.</strong></p> <p><strong>Task 3:</strong> <strong>Multi-class fake news detection of news articles (English)</strong> Sub-task A would detect fake news designed as a four-class classification problem. Given the text of a news article, determine whether the main claim made in the article is true, partially true, false, or other. The training data will be released in batches and roughly about 1264 articles with the respective label in English language. Our definitions for the categories are as follows:</p> <ul> <li> <p>False - The main claim made in an article is untrue.</p> </li> <li> <p>Partially False - The main claim of an article is a mixture of true and false information. The article contains partially true and partially false information but cannot be considered 100% true. It includes all articles in categories like partially false, partially true, mostly true, miscaptioned, misleading etc., as defined by different fact-checking services.</p> </li> <li> <p>True - This rating indicates that the primary elements of the main claim are demonstrably true.</p> </li> <li> <p>Other- An article that cannot be categorised as true, false, or partially false due to a lack of evidence about its claims. This category includes articles in dispute and unproven articles.</p> </li> </ul> <p><strong>Cross-Lingual Task (German)</strong></p> <p>Along with the multi-class task for the English language, we have introduced a task for low-resourced language. We will provide the data for the test in the German language. The idea of the task is to use the English data and the concept of transfer to build a classification model for the German language.</p> <p><strong>Input Data</strong></p> <p>The data will be provided in the format of Id, title, text, rating, the domain; the description of the columns is as follows:</p> <ul> <li>ID- Unique identifier of the news article</li> <li>Title- Title of the news article</li> <li>text- Text mentioned inside the news article</li> <li>our rating - class of the news article as false, partially false, true, other</li> </ul> <p><strong>Output data format</strong></p> <ul> <li>public_id- Unique identifier of the news article</li> <li>predicted_rating- predicted class</li> </ul> <p>Sample File</p> <pre><code>public_id, predicted_rating 1, false 2, true</code></pre> <p><strong>IMPORTANT! </strong></p> <ol> <li>We have used the data from 2010 to 2022, and the content of fake news is mixed up with several topics like elections, COVID-19 etc.</li> </ol> <p><strong>Baseline:</strong> For this task, we have created a baseline system. The baseline system can be found at <a href="https://zenodo.org/record/6362498">https://zenodo.org/record/6362498</a></p> <p><strong>Related Work</strong></p> <ul> <li>Shahi GK. AMUSED: An Annotation Framework of Multi-modal Social Media Data. arXiv preprint arXiv:2010.00502. 2020 Oct 1.<a href="https://arxiv.org/pdf/2010.00502.pdf">https://arxiv.org/pdf/2010.00502.pdf</a></li> <li>G. K. Shahi and D. Nandini, “FakeCovid – a multilingual cross-domain fact check news dataset for covid-19,” in workshop Proceedings of the 14th International AAAI Conference on Web and Social Media, 2020. <a href="http://workshop-proceedings.icwsm.org/abstract?id=2020_14">http://workshop-proceedings.icwsm.org/abstract?id=2020_14</a></li> <li>Shahi, G. K., Dirkson, A., & Majchrzak, T. A. (2021). An exploratory study of covid-19 misinformation on twitter. <em>Online Social Networks and Media</em>, <em>22</em>, 100104. doi: <a href="https://dx.doi.org/10.1016%2Fj.osnem.2020.100104">10.1016/j.osnem.2020.100104</a></li> <li>Shahi, G. K., Struß, J. M., & Mandl, T. (2021). Overview of the CLEF-2021 CheckThat! lab task 3 on fake news detection. <em>Working Notes of CLEF</em>.</li> <li>Nakov, P., Da San Martino, G., Elsayed, T., Barrón-Cedeno, A., Míguez, R., Shaar, S., ... & Mandl, T. (2021, March). The CLEF-2021 CheckThat! lab on detecting check-worthy claims, previously fact-checked claims, and fake news. In <em>European Conference on Information Retrieval</em> (pp. 639-649). Springer, Cham.</li> <li>Nakov, P., Da San Martino, G., Elsayed, T., Barrón-Cedeño, A., Míguez, R., Shaar, S., ... & Kartal, Y. S. (2021, September). Overview of the CLEF–2021 CheckThat! Lab on Detecting Check-Worthy Claims, Previously Fact-Checked Claims, and Fake News. In <em>International Conference of the Cross-Language Evaluation Forum for European Languages</em> (pp. 264-291). Springer, Cham.</li> </ul>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.