Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
10
datasets available to search
ShareScore release 0.9.0
Dataset results
10 results for “document classification”
Data Cleaning, Translation & Split of the Dataset for the Automatic Classification of Documents for the Classification System for the Berliner Handreichungen zur Bibliotheks- und Informationswissenschaft
<ul> <li>Cleaned_Dataset.csv – The combined CSV files of all scraped documents from DABI, e-LiS, o-bib and Springer.</li> <li>Data_Cleaning.ipynb – The Jupyter Notebook with python code for the analysis and cleaning of the original dataset.</li> <li>ger_train.csv – The German training set as CSV file.</li> <li>ger_validation.csv – The German validation set as CSV file.</li> <li>en_test.csv – The English test set as CSV file.</li> <li>en_train.csv – The English training set as CSV file.</li> <li>en_validation.csv – The English validation set as CSV file.</li> <li>splitting.py – The python code for splitting a dataset into train, test and validation set.</li> <li>DataSetTrans_de.csv – The final German dataset as a CSV file.</li> <li>DataSetTrans_en.csv – The final English dataset as a CSV file.</li> <li>translation.py – The python code for translating the cleaned dataset.</li> </ul>
ICDAR 2021 Historical Document Classification Test Dataset for Task 1 - Font Groups
<p>Test set for the font group classification task of the ICDAR 2021 Competition on Historical Document Classification competition. The tar.gz file contains the images and a ground truth CSV file. The csv file indicates for each image from which document and page it corresponds to, as well as whether augmentations have been applied.</p>
ICDAR 2021 Historical Document Classification Dataset for Task 2 - Dating
<p>Dataset for the localization classification task of the ICDAR 2021 Competition on Historical Document Classification competition. The tar.gz file contains the images and ground truth file (CSV format). The metadata csv file contains information, such as where the image comes from.</p>
Pairwise Multi-Class Document Classification for Semantic Relations between Wikipedia Articles (Dataset, Models & Code)
<p>Many digital libraries recommend literature to their users considering the similarity between a query document and their repository. However, they often fail to distinguish what is the relationship that makes two documents alike. In this paper, we model the problem of finding the relationship between two documents as a pairwise document classification task. To find the semantic relation between documents, we apply a series of techniques, such as GloVe, Paragraph-Vectors, BERT, and XLNet under different configurations (e.g., sequence length, vector concatenation scheme), including a Siamese architecture for the Transformer-based systems. We perform our experiments on a newly proposed dataset of 32,168 Wikipedia article pairs and Wikidata properties that define the semantic document relations. Our results show vanilla BERT as the best performing system with an F1-score of 0.93,<br> which we manually examine to better understand its applicability to other domains. Our findings suggest that classifying semantic relations between documents is a solvable task and motivates the development of recommender systems based on the evaluated techniques. The discussions in this paper serve as first steps in the exploration of documents through SPARQL-like queries such that one could find documents that are similar in one aspect but dissimilar in another.</p> <p>Additional information can be found on <a href="https://github.com/malteos/semantic-document-relations/">GitHub</a>.</p> <p>The following data is supplemental to the experiments described in our research paper. The data consists of:</p> <ul> <li>Datasets (articles, class labels, cross-validation splits)</li> <li>Pretrained models (Transformers, GloVe, Doc2vec)</li> <li>Model output (prediction) for the best performing models</li> </ul> <p><strong>Dataset</strong></p> <p>The Wikipedia article corpus is available in <code>enwiki-20191101-pages-articles.weighted.10k.jsonl.bz2</code>. The original data have been downloaded as <a href="https://dumps.wikimedia.org/enwiki/">XML dump</a>, and the corresponding articles were extracted as plain-text with <a href="https://radimrehurek.com/gensim/scripts/segment_wiki.html">gensim.scripts.segment_wiki</a>. The archive contains only articles that are available in training or test data.</p> <p>The actual dataset is provided as used in the stratified k-fold with <code>k=4</code> in <code>train_testdata__4folds.tar.gz</code>.</p> <pre><code>├── 1 │ ├── test.csv │ └── train.csv ├── 2 │ ├── test.csv │ └── train.csv ├── 3 │ ├── test.csv │ └── train.csv └── 4 ├── test.csv └── train.csv 4 directories, 8 files </code></pre> <p>Pretrained models</p> <p>PyTorch: vanilla and Siamese BERT + XLNet</p> <p>Pretrained model for each fold is available in the corresponding model archives:</p> <pre><code># Vanilla model_wiki.bert_base__joint__seq512.tar.gz model_wiki.xlnet_base__joint__seq512.tar.gz # Siamese model_wiki.bert_base__siamese__seq512__4d.tar.gz model_wiki.xlnet_base__siamese__seq512__4d.tar.gz </code></pre>
Architecturally Significant Requirements and Software Architecture for AI-Based Systems: A Case Study with Document Classification
Open the record for dataset details and reuse information.
MultiEURLEX - A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer
<p>The dataset is published with:<br> <br> <em>MultiEURLEX - A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer. Ilias Chalkidis, Manos Fergadiotis, and Ion Androutsopoulos. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021. Punta Cana, Dominican Republic.</em><br> <br> <strong>Documents: </strong>MultiEURLEX comprises 65k EU in 23 official EU languages. Each EU law has been annotated with EUROVOC concepts (labels) by the Publication Office of EU. Each EUROVOC label ID is associated with a Label descriptor, e.g., [60, `agri-foodstuffs'], [6006, `plant product'], [1115, `fruit']. The descriptors are also available in 23 languages. Chalkidis et al. (2019) published a monolingual (English) version of this dataset, called EURLEX57K, comprising 57k EU laws with the originally assigned gold labels.</p> <p><strong>Languages: </strong>MultiEURLEX covers 23 languages from 7 families. EU laws are published in all official EU languages, except for Irish for resource-related reasons (Read more: https://europa.eu/european-union/about-eu/eu-languages_en). This wide coverage makes the dataset a valuable testbed for cross-lingual transfer. All languages use the Latin script, except for Bulgarian (Cyrillic script) and Greek.</p> <p><strong>Multi-granular Labeling: </strong>EUROVOC<strong> </strong>has eight levels of concepts. Each document is assigned one or more concepts (labels). If a document is assigned a concept, the ancestors and descendants of that concept are typically not assigned to the same document. The documents were originally annotated with concepts from levels 3 to 8. We created three alternative sets of labels per document, by replacing each assigned concept by its ancestor from levels 1, 2, or 3, respectively. Thus, we provide four sets of gold labels per document, one for each of the first three levels of the hierarchy, plus the original sparse label assignment.</p> <p><strong>Supported Tasks: </strong>Similarly to EURLEX (Chalkidis et al., 2019), MultiEURLEX can be used for legal topic classification, a multi-label classification task where legal documents need to be assigned concepts (in our case, from EUROVOC) reflecting their topics. Unlike EURLEX57K, however, MultiEURLEX supports labels from three different granularities (EUROVOC levels). More importantly, apart from monolingual (one-to-one) experiments, it can be used to study cross-lingual transfer scenarios, including one-to-many (systems trained in one language and used in other languages with no training data), and many-to-one or many-to-many (systems jointly trained in multiple languages and used in one or more other languages).</p> <p><strong>Data Split and Concept Drift: </strong>MultiEURLEX is chronologically split in training (55k, 1958-2010), development (5k, 2010-2012), test (5k, 2012-2016) subsets, using the English documents. The test subset contains the same 5k documents in all 23 languages. The development subset also contains the same 5k documents in 23 languages, except Croatian. Croatia is the most recent EU member (2013); older laws are gradually translated. For the official languages of the seven oldest member countries, the same 55k training documents are available; for the other languages, only a subset of the 55k training documents is available. Compared to EURLEX57K (Chalkidis et al., 2019), MultiEURLEX is not only larger (8k more documents) and multilingual; it is also more challenging, as the chronological split leads to temporal real-world concept drift across the training, development, test subsets, i.e., differences in label distribution and phrasing, representing a realistic temporal generalization problem (Huang and Paul, 2019; Lazaridou et al., 2021). Recently, Søgaard et al. (2021) showed this setup is more realistic, as it does not overestimate real performance, contrary to random splits (Gorman and Bedrick, 2019).</p>
ICDAR 2021 Historical Document Classification Test Dataset for Task 3 - Location
<p>Test set for the location classification task of the ICDAR 2021 Historical Document Classification.</p>
ICDAR 2021 Historical Document Classification Test Dataset for Task 1 - Scripts
<p>Test set for thescript classification task of the ICDAR 2021 Competition on Historical Document Classification competition. The tar.gz file contains the images and a ground truth CSV file. The csv file indicates for each image from which document and page it corresponds to, as well as whether augmentations have been applied.</p>
ICDAR 2021 Historical Document Classification Dataset for Task 3 - Location
<p>Test set for the localization classification task of the ICDAR 2021 Competition on Historical Document Classification competition. The tar.gz file contains the images and a ground truth CSV file. The metadata csv file contains information, such as where the image comes from.</p>
Classification of Aeronautics System Health and Safety Documents
Most complex aerospace systems have many text reports on safety, maintenance, and associated issues. The Aviation Safety Reporting System (ASRS) spans several decades and contains over 700 000 reports. The Aviation Safety Action Plan (ASAP) contains over 12 000 reports from various airlines. Problem categorizations have been developed for both ASRS and ASAP to enable identification of system problems. However, repository volume and complexity make human analysis difficult. Multiple experts are needed, and they often disagree on classifications. Even the same person has classified the same document differently at different times due to evolving experiences. Consistent classification is necessary to support tracking trends in problem categories over time. A decision support system that performs consistent document classification quickly and over large repositories would be useful. We discuss the results of two algorithms we have developed to classify ASRS and ASAP documents. The first is Mariana---a support vector machine (SVM) with simulated annealing, which is used to optimize hyperparameters for the model. The second method is classification built on top of nonnegative matrix factorization (NMF), which attempts to find a model that represents document features that add up in various combinations to form documents. We tested both methods on ASRS and ASAP documents with the latter categorized two different ways. We illustrate the potential of NMF to provide document features that are interpretable and indicative of topics. We also briefly discuss the tool that we have incorporated Mariana into in order to allow human experts to provide feedback on the document categorizations.
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.