Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
46
datasets available to search
ShareScore release 0.7.1
Dataset results
46 results for “multimodal learning”
Written and spoken digits database for multimodal learning
<p><strong>Database description:</strong></p> <p>The written and spoken digits database is not a new database but a constructed database from existing ones, in order to provide a ready-to-use database for multimodal fusion [1].</p> <p>The written digits database is the original MNIST handwritten digits database [2] with no additional processing. It consists of 70000 images (60000 for training and 10000 for test) of 28 x 28 = 784 dimensions.</p> <p>The spoken digits database was extracted from Google Speech Commands [3], an audio dataset of spoken words that was proposed to train and evaluate keyword spotting systems. It consists of 105829 utterances of 35 words, amongst which 38908 utterances of the ten digits (34801 for training and 4107 for test). A pre-processing was done via the extraction of the Mel Frequency Cepstral Coefficients (MFCC) with a framing window size of 50 ms and frame shift size of 25 ms. Since the speech samples are approximately 1 s long, we end up with 39 time slots. For each one, we extract 12 MFCC coefficients with an additional energy coefficient. Thus, we have a final vector of 39 x 13 = 507 dimensions. Standardization and normalization were applied on the MFCC features.</p> <p>To construct the multimodal digits dataset, we associated written and spoken digits of the same class respecting the initial partitioning in [2] and [3] for the training and test subsets. Since we have less samples for the spoken digits, we duplicated some random samples to match the number of written digits and have a multimodal digits database of 70000 samples (60000 for training and 10000 for test).</p> <p>The dataset is provided in six files as described below. Therefore, if a shuffle is performed on the training or test subsets, it must be performed in unison with the same order for the written digits, spoken digits and labels.</p> <p> </p> <p><strong>Files:</strong></p> <ul> <li>data_wr_train.npy: 60000 samples of 784-dimentional written digits for training;</li> <li>data_sp_train.npy: 60000 samples of 507-dimentional spoken digits for training;</li> <li>labels_train.npy: 60000 labels for the training subset;</li> <li>data_wr_test.npy: 10000 samples of 784-dimentional written digits for test;</li> <li>data_sp_test.npy: 10000 samples of 507-dimentional spoken digits for test;</li> <li>labels_test.npy: 10000 labels for the test subset.</li> </ul> <p> </p> <p><strong>References:</strong></p> <ol> <li>Khacef, L. et al. (2020), "Brain-Inspired Self-Organization with Cellular Neuromorphic Computing for Multimodal Unsupervised Learning".</li> <li>LeCun, Y. & Cortes, C. (1998), “MNIST handwritten digit database”.</li> <li>Warden, P. (2018), “Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition”.</li> </ol>
MEWL: Few-shot multimodal word learning with referential uncertainty
<p><strong>Dataset Release for <a href="https://arxiv.org/abs/2306.00503">MEWL: Few-shot multimodal word learning with referential uncertainty (ICML 2023) </a></strong></p> <p><strong>GitHub:</strong> <a href="https://github.com/jianggy/MEWL">https://github.com/jianggy/MEWL</a></p> <p><strong>Abstract: </strong>Without explicit feedback, humans can rapidly learn the meaning of words. Children can acquire a new word after just a few passive exposures, a process known as fast mapping. This word learning capability is believed to be the most fundamental building block of multimodal understanding and reasoning. Despite recent advancements in multimodal learning, a systematic and rigorous evaluation is still missing for human-like word learning in machines. To fill in this gap, we introduce the MachinE Word Learning (MEWL) benchmark to assess how machines learn word meaning in grounded visual scenes. MEWL covers human's core cognitive toolkits in word learning: cross-situational reasoning, bootstrapping, and pragmatic learning. Specifically, MEWL is a few-shot benchmark suite consisting of nine tasks for probing various word learning capabilities. These tasks are carefully designed to be aligned with the children's core abilities in word learning and echo the theories in the developmental literature. By evaluating multimodal and unimodal agents' performance with a comparative analysis of human performance, we notice a sharp divergence in human and machine word learning. We further discuss these differences between humans and machines and call for human-like few-shot word learning in machines.</p>
Multimodala Dataset for multimodal contrastive learning for crop classification
<p>We developed this dataset using an existing dataset name DENETHOR developed by TUM <a href="https://openreview.net/forum?id=uUa4jNMLjrL">https://openreview.net/forum?id=uUa4jNMLjrL</a> to conduct our multi-modal contrastive learning experiments.</p>
MultiFranceFences: A novel deep learning dataset for automated fence detection from multimodal aerial imagery
<p>The <strong>MultiFranceFences</strong> dataset is a large-scale, multimodal remote sensing benchmark for the semantic segmentation of fences across various landscapes in France. This dataset integrates high-resolution orthophotographs (RGB through BDOrtho) and Digital Surface Models (DSM) derived from LiDARHD data. </p> <p>MultiFranceFences is suitable for deep learning models in semantic segmentation, including state-of-the-art models like UNet, D-LinkNet, and the newly proposed H-IncepUNet, which integrates handcrafted features and multi-scale feature extraction modules for enhanced fence detection.</p> <p><strong>Dataset features:</strong></p> <ul> <li><strong>Multimodal imagery</strong>: Combines orthophotographs and DSM data from LiDARHD for fences semantic segmentation (folders <em>ortho</em> and <em>lidar</em>).</li> <li><strong>Buffer options</strong>: 2-meter and 3-meter buffer fence annotations to fit varying detection requirements (folders <em>fences_2m</em> and <em>fences_3m</em>).</li> <li><strong>Diverse landscapes</strong>: Covers rural, and natural environments across France.</li> <li><strong>Validated dataset</strong>: Manually cleaned and validated to remove erroneous fence labels under tree canopies or areas with limited visibility.</li> </ul> <p>Each patch is named according to the nomenclature of the original BDOrtho tile, followed by the specific x and y coordinates of the patch within that tile.</p>
Supplementary files for the article "A Systematic Literature Review on Multimodal Machine Learning"
<p>All included data was used for this review study. Data contains information from collected articles. Results of the analysis of each article are also available. All files are related to each other. A file named "All Articles Selected And Rejected Decision.xlsx" contains the name and ID of all articles, and that ID is used in other files as a reference for the analysis.</p>
Data for "A multimodal machine learning fused global 0.1° daily evapotranspiration dataset from 1950-2022" (2000-2024)
<p>The data contains simulation results from 2000-2024, 25 years total.</p> <p>You can access the remaining part of the dataset via Qingchen Xu and Lu Li (2025) using the following reference:</p> <p>Qingchen Xu, & Lu Li. (2025). Data for "A multimodal machine learning fused global 0.1° daily evapotranspiration dataset from 1950-2022" (1950-1974) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.15671245</p> <p>Qingchen Xu, & Lu Li. (2025). Data for "A multimodal machine learning fused global 0.1° daily evapotranspiration dataset from 1950-2022" (1975-1999) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.15671253</p>
Data for "A multimodal machine learning fused global 0.1° daily evapotranspiration dataset from 1950-2022" (1975-1999)
<p>The data contains simulation results from 1975-1999, 25 years total.</p> <p>You can access the remaining part of the dataset via Qingchen Xu and Lu Li (2025) using the following reference:</p> <p>Qingchen Xu, & Lu Li. (2025). Data for "A multimodal machine learning fused global 0.1° daily evapotranspiration dataset from 1950-2022" (1950-1974) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.15671245</p> <p>Qingchen Xu, & Lu Li. (2025). Data for "A multimodal machine learning fused global 0.1° daily evapotranspiration dataset from 1950-2022" (2000-2024) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.15671254</p>
Data for "A multimodal machine learning fused global 0.1° daily evapotranspiration dataset from 1950-2022" (1950-1974)
<p>The data contains simulation results from 1950-1974, 25 years total.</p> <p>You can access the remaining part of the dataset via Qingchen Xu and Lu Li (2025) using the following reference:<br>Qingchen Xu, & Lu Li. (2025). Data for "A multimodal machine learning fused global 0.1° daily evapotranspiration dataset from 1950-2022" (1975-1999) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.15671253</p> <p>Qingchen Xu, & Lu Li. (2025). Data for "A multimodal machine learning fused global 0.1° daily evapotranspiration dataset from 1950-2022" (2000-2024) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.15671254</p>
Transformer-based graphical neural network with expert experience multimodal learning (TGEML) framework: a nanocomposite performance predictor
<p>TGEML is a novel multimodal nanocomposite processing framework consists of a polymer multimodal featurizer called TGEML-polymer and a nanoparticle expert experience featurizer called TGEML-nano.</p>
Multimodal Sensory Learning for Object Manipulation
<p><strong>Multimodal Manipulation Learning Database</strong></p> <p>The dataset consists of data recordings for object manipulation with audio-tactile sensory feedback for object handover. It captures the auditory and tactile signals of a Kuka IIWA robot with an Allegro hand holding a plastic container containing different materials. The robot manipulates the container with vertical shaking and rotation motions. The data consists of force/pressure measurements on the Allegro hand using a Tekscan tactile skin sensor, auditory signals from a microphone, and the joints data of the IIWA robot and the Allegro hand joints. </p> <p><strong>Dataset</strong></p> <p>Each datafile is a rosbag file containing the data recording from one trial of a robot motion with one material, with rostopics on the following data:</p> <ul> <li>Kuka IIWA 7 Joint data: /iiwa/TorqueController/command /iiwa/eePose /iiwa/joint_states</li> <li>Allegro hand joint data: /allegro_hand_right/joint_states</li> <li>Tekscan sensor recording (tactile force/pressure sensor data on hand): /tekscan/frame</li> <li>Audio data (for microphone attached to hand): /audio/audio /audio/audio_info</li> <li>Experiment information: /trialInfo <ul> <li>which contains: <ul> <li>trial information (motion type, speed, etc.)</li> <li>start/stop of different phases of the trials</li> </ul> </li> </ul> </li> </ul> <p><strong>Motion Types</strong></p> <p>The database contains recordings for the robot executing two different motion types: vertical shaking of the object and rotation of the object.</p> <p><strong>Materials</strong></p> <p>The database contains recordings for 5 different material classes in the plastic container, as shown below: empty, vitamins, gummies, cornflakes, and rice. We used approximately the same volume of each material for each trial. We tested each material class and motion combination for a total of 10 different experimental conditions and collected 30 trials for each condition.</p> <p>The vertical motion dataset was entirely collected on 2021/08/25. The rotation dataset was split into two day. The empty, gummies and rice class data was collected on 2021/08/26. The vitamins and cornflakes classes were collected on 2021/09/13.</p> <p><strong>Database Setup</strong></p> <p>The database consists of the data in two formats: annotated ('annotated_bags_mml.zip') and unannotated/numbered filenames ('numbered_bags_mml.zip') datasets. The data in the two datasets are identical- the annotated filename dataset has the experimental descriptions in the filename directly (as described below).</p> <p>The annotated filenames dataset ('annotated_bags_mml.zip') consists of a single directory with all 300 rosbag datafiles (10 experimental conditions, 30 trials each). Each rosbag (<code>.bag</code>) is saved in the directory, with filename specified ('Date Recorded YYYYMMDD' + '_motion' + '_material' + '_trialID' + '.bag'). Motion Types are: {'vertical', 'rotation'}. Materials are: {'empty', 'cornflakes', 'gummies', 'rice', 'vitamins'}. For each experimental condition, there are 30 datafiles with trial IDs from 0-29.</p> <p>All data recordings for the vertical motion have filenames: '20210825_vertical_+ 'material' + 'trialID' +'.bag). For the rotation motion, the empty, gummy and rice classes have filenames: '20210826_rotation_+ 'material' + 'trialID' +'.bag). For cornflakes and vitamins classes, the filenames are: '20210913_rotation_+ 'material' + 'trialID' +'.bag).</p> <p>The numbered/unannotated file dataset ('numbered_bags_mml.zip') consists of the same 300 data files as in the annotated dataset except here the filenames are numbered '{000-299}.bag'. The directory contains a spreadsheet ('annotations.csv') listing the experimental descriptions for each file name. The columns of the xls spreadsheet are {'Bagfile name', 'Year', 'Month', 'Day', 'Motion/Movement (mvt_type)', 'Material', 'Trial ID'}, where {Year, Month, Day} refer to the date that trial data was collected (either 2021/08/25, 2021/08/26, or 2021/09/13). </p>
Multimodal learning of noncoding variant effects using genome sequence and chromatin structure
<p>ncVarPred-1D3D:</p> <p>The data used for testing the inconsistency among genome sequence, epigenetic profile, and later, to show its relation to 3D chromatin structure can be found in sanity_check_data.tar.gz.</p> <p>Some trained model for noncoding mutation effect prediction (mapping genome sequence to epigenetic profile) can be found in CNN_MLP, CNN_GCN, CNN_RNN_MLP, CNN_RNN_GCN.tar.gz.</p> <p>The trained model for pathogenic variants prediction can be found in fewshot_pathogenic_model.tar.gz. </p> <p>The training data can be found in training_data.tar.gz.</p> <p>Some noncoding variant effects prediction results, e.g. eQTL and pathogenic variants, can be replicated using the data shared in ncVar_data.tar.gz.</p>
Multimodal learning of noncoding variant effects using genome sequence and chromatin structure
<p>ncVarPred-1D3D: pretrained models of Sei (PMID: 35817977) + our 3D structure embedding models are shared. The models are trained and validated using DeepSEA (PMID: 26301843) selected 200 bp regions (we extended to 4K bp neighboring) to predict the epigenetic profile containing 21907 epigenetic events Sei processed.</p> <p>The pretrained DeepSEA (PMID: 26301843) and reproduced DanQ (PMID: 27084946) can be found in SOTA.tar.gz.</p>
Predicting Pathological Complete Response in Esophageal Squamous Cell Carcinoma Using a Multimodal Model Integrating Clinical, Radiomics, and Deep Learning Features
ClinicalTrials.gov study NCT07181850. IPD Sharing: NO. Countries: 1. Publications: 4.
Perceived Mental Workload Classification using Intermediate Fusion Multimodal Deep Learning
<p><em>This repository contains all code -from data collection to perceived mental workload classification- used in the "Perceived Mental Workload Classification using Intermediate Fusion Multimodal Deep Learning" research and serves as supplementary material. <strong>A clear README file is provided, please refer to that for information about the individual scripts and their usage. </strong></em></p> <p>Mental workload detection has been attempted using various bio-signals. Recently, deep learning has allowed for novel methods and results within the BCI community. However, studies currently often only use a single modality to classify mental workload, whereas a plethora of modalities have proven to be valuable in this task.</p> <p>A dataset on which these scripts was also made publicly available under the following DOI: 10.4121/12932801<br> This dataset contains data collected during research into mental workload (MWL) detection using deep learning. It is being made public as supplementary data for publications, as well as for reuse in research that seeks to classify MWL using multimodal physiological data. The goal of this repository and dataset is to serve as a testing ground for the creation of deep neural networks that can classify MWL using multimodal physiological data.</p> <p>The data in this dataset was collected in the Behavioural, Management, and Social Sciences Lab, University of Twente, Enschede, The Netherlands in June/July 2020.</p>
Multimodal contrastive learning for spatial gene expression prediction using histology images
<p>we employed two human breast cancer datasets and one human cutaneous squamous cell carcinoma (cSCC) dataset.</p>
Nondestructive Fatigue Life Prediction for Additively Manufactured Metal Parts through a Multimodal Transfer Learning Framework
Open the record for dataset details and reuse information.
The dataset of "Deep Learning-enabled 3D Multimodal Fusion of Cone-Beam CT and Intraoral Mesh Scans for Clinically Applicable Tooth-bone Reconstruction"
<p>The dataset used in the study "Deep Learning-enabled 3D Multimodal Fusion of Cone-Beam CT and Intraoral Mesh Scans for Clinically Applicable Tooth-bone Reconstruction" is available upon request. Please contact the authors of the study or the responsible institution for access to the dataset.</p>
Blast Furnace Raw Material Granularity Recognition Model Based on Deep Learning and Multimodal Fusion of 3D Point Cloud
<p>Provide data code</p>
A Multimodal Approach to Cervical Dystonia Treatment With Association of Botulinum Toxin and Motor Learning Techniques
ClinicalTrials.gov study NCT03247868. IPD Sharing: NO. Countries: 1. Publications: 12.
Effectiveness of a Multimodal Intervention With Simulation for Learning Home Health Nursing Care of Patients With Multimorbidity and Heart Failure
ClinicalTrials.gov study NCT06855719. IPD Sharing: YES. Countries: 1. Publications: 17.
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.