Find research datasets worth reusing

opencc-by-4.0May 2020View details →

Synthetic Datasets for mpEDM

Generated datasets are synthesized for benchmarking mpEDM. The datasets were generated with numpy. File Name: Generated_TimeSeries_{No. of time steps}_{No. of time series}.h5

opencc-by-4.0Apr 2020View details →

Afro-MNIST: Synthetic generation of MNIST-style datasets for low-resource languages

We present Afro-MNIST, a set of synthetic MNIST-style datasets for four orthographies used in Afro-Asiatic and Niger-Congo languages: Ge`ez (Ethiopic), Vai, Osmanya, and N'Ko. These datasets serve as ``drop-in'' replacements for MNIST. We hope that MNIST-style datasets will be developed for other numeral systems, and that these datasets vitalize machine learning education in underrepresented nations in the research community.

opencc-by-4.0Dec 2021View details →

Synthetic student dataset on levels of use of digital tools and frequency of personal activities with ICTs

Dataset artificially generated through the Generative Adversarial Networks of students in postgraduate degrees at the University of La Laguna on levels of use of digital tools and frequency of personal activities with ICT.

opencc-by-4.0Jul 2024View details →

Synth-Yard-MCMOT-1 – A synthetic multi camera multi object dataset in the context of yard logistics

The dataset "Synth-Yard-MCMOT-1" is a novel image dataset for multi-camera multi-object tracking (MCMOT). It is the first of its kind to be generated in a virtual environment with the main focus on the tracking of trucks in yard logistics environments. The dataset consists of a total of 12,008 images generated by eight different cameras. The images contain 44,232 bounding boxes and segmentation masks and 52 individual tracks. Additionally, we provide a ninth camera, which is used to generate unified ground-truth information for the whole scene from an orthographic, top-down perspective comparable to a bird’s eye or map-view. The purpose of this dataset is to provide yard management systems with relevant data, which can be employed when aiming to determine the exact position of a truck and specifically identifying which gateway or designated parking spot it is located in.  Additionally in this dataset we added the annotations in the COCO-style format as well as the raw image-data. Initial benchmarks for single-camera tracking demonstrate a mean IDF1 score of 0.96 and a mean MOTA score of 0.94, laying the baseline for computing world coordinates via MCMOT. This work is part of the project “Silicon Economy Logistics Ecosystem” which is funded by the German Federal Ministry of Transport and Digital Infrastructure. This research has partly been funded by the Federal Ministry of Education and Research of Germany and the state of North-Rhine Westphalia as part of the Lamarr-Institute  for Machine Learning and Artificial Intelligence. We would like to thank our colleagues Julian Hinxlage, Oleg Belov, Habib Kilic, Nicole Wagner-Hanl, Leona Cerimi, Antonia Ponikarov and Nico Lindemann for their support. The code to create this dataset can be found here: https://git.openlogisticsfoundation.org/silicon-economy/base/ml-toolbox/synth-data-mcmot <div> <div> </div> <div> </div> </div>

opencc-by-4.0Oct 2019View details →

Synthetic Speech Dataset

This contains the speech data synthesized for the following paper: "Using Speech Synthesis to Train End-to-End Spoken Language Understanding Models" by Loren Lugosch, Brett Meyer, Derek Nowrouzezahrai, and Mirco Ravanelli

opencc-by-4.0Oct 2024View details →

A large synthetic dataset for machine learning applications in power transmission grids

With the ongoing energy transition, power grids are evolving fast. They operate more and more often close to their technical limit, under more and more volatile conditions. Fast, essentially real-time computational approaches to evaluate their operational safety, stability and reliability are therefore highly desirable. Machine Learning methods have been advocated to solve this challenge, however they are heavy consumers of training and testing data, while historical operational data for real-world power grids are hard if not impossible to access.  This dataset contains long time series for production, consumption, and line flows, amounting to 20 years of data with a time resolution of one hour, for several thousands of loads and several hundreds of generators of various types representing the ultra-high-voltage transmission grid of continental Europe. The synthetic time series have been statistically validated agains real-world data. <h2>Data generation algorithm</h2> The algorithm is described in a <a href="https://doi.org/10.1038/s41597-025-04479-x">Nature Scientific Data paper</a>. It relies on <a href="https://zenodo.org/records/2642175" target="_blank" rel="noopener">the PanTaGruEl model of the European transmission network</a> -- the admittance of its lines as well as the location, type and capacity of its power generators -- and aggregated data gathered from <a href="https://transparency.entsoe.eu/" target="_blank" rel="noopener">the ENTSO-E transparency platform</a>, such as power consumption aggregated at the national level. <h2>Network</h2> The network information is encoded in the file <a href="https://zenodo.org/records/13378476/files/europe_network.json">europe_network.json</a>. It is given in <a href="https://lanl-ansi.github.io/PowerModels.jl/stable/" target="_blank" rel="noopener">PowerModels format</a>, which it itself derived from <a href="https://matpower.org/" target="_blank" rel="noopener">MatPower</a> and compatible with <a href="https://www.pandapower.org/" target="_blank" rel="noopener">PandaPower</a>. The network features 7822 power lines and 553 transformers connecting 4097 buses, to which are attached 815 generators of various types. <h2>Time series</h2> The time series forming the core of this dataset are given in CSV format. Each CSV file is a table with 8736 rows, one for each hourly time step of a 364-day year. All years are truncated to exactly 52 weeks of 7 days, and start on a Monday (the load profiles are typically different during weekdays and weekends). The number of columns depends on the type of table: there are 4097 columns in load files, 815 for generators, and 8375 for lines (including transformers). Each column is described by a header corresponding to the element identifier in the network file. All values are given in per-unit, both in the model file and in the tables, i.e. they are multiples of a base unit taken to be 100 MW. There are 20 tables of each type, labeled with a reference year (2016 to 2020) and an index (1 to 4), zipped into archive files arranged by year. This amount to a total of 20 years of synthetic data.  When using loads, generators, and lines profiles together, it is important to use the same label: for instance, the files loads_2020_1.csv, gens_2020_1.csv, and lines_2020_1.csv represent a same year of the dataset, whereas gens_2020_2.csv is unrelated (it actually shares some features, such as nuclear profiles, but it is based on a dispatch with distinct loads). <h2>Usage</h2> The time series can be used without a reference to the network file, simply using all or a selection of columns of the CSV files, depending on the needs. We show below how to select series from a particular country, or how to aggregate hourly time steps into days or weeks. These examples use Python and the data analyis library pandas, but other frameworks can be used as well (Matlab, Julia). Since all the yearly time series are periodic, it is always possible to define a coherent time window modulo the length of the series. <h3>Selecting a particular country</h3> This example illustrates how to select generation data for Switzerland in Python. This can be done without parsing the network file, but using instead <a href="https://zenodo.org/records/13378476/files/gens_by_country.csv">gens_by_country.csv</a>, which contains a list of all generators for any country in the network. We start by importing the pandas library, and read the column of the file corresponding to Switzerland (country code CH): <pre><code>import pandas as pd CH_gens = pd.read_csv('gens_by_country.csv', usecols=['CH'], dtype=str)</code></pre> The object created in this way is Dataframe with some null values (not all countries have the same number of generators). It can be turned into a list with: <pre><code>CH_gens_list = CH_gens.dropna().squeeze().to_list()</code></pre> Finally, we can import all the time series of Swiss generators from a given data table with <pre><code>pd.read_csv('gens_2016_1.csv', usecols=CH_gens_list)</code></pre> The same procedure can be applied to loads using the list contained in the file <a href="https://zenodo.org/records/13378476/files/loads_by_country.csv">loads_by_country.csv</a>. <h3>Averaging over time</h3> This second example shows how to change the time resolution of the series. Suppose that we are interested in all the loads from a given table, which are given by default with a one-hour resolution: <pre><code>hourly_loads = pd.read_csv('loads_2018_3.csv')</code></pre> To get a daily average of the loads, we can use:  <pre><code>daily_loads = hourly_loads.groupby([t // 24 for t in range(24 * 364)]).mean()</code></pre> This results in series of length 364. To average further over entire weeks and get series of length 52, we use:  <pre><code>weekly_loads = hourly_loads.groupby([t // (24 * 7) for t in range(24 * 364)]).mean()</code></pre> <h2>Source code</h2> The code used to generate the dataset is freely available at <a href="https://github.com/GeeeHesso/PowerData" target="_blank" rel="noopener">https://github.com/GeeeHesso/PowerData</a>. It consists in two packages and several documentation notebooks. The first package, written in Python, provides functions to handle the data and to generate synthetic series based on historical data. The second package, written in Julia, is used to perform the optimal power flow. The documentation in the form of Jupyter notebooks contains numerous examples on how to use both packages. The entire workflow used to create this dataset is also provided, starting from raw ENTSO-E data files and ending with the synthetic dataset given in the repository. <h2>Funding</h2> This work was supported by the <a href="https://www.cydcampus.admin.ch">Cyber-Defence Campus of armasuisse</a> and by an internal research grant of the Engineering and Architecture domain of <a href="https://www.hes-so.ch">HES-SO</a>.

opencc-by-4.0Oct 2024View details →

EmotionCaps: A Synthetic Emotion-Enriched Audio Captioning Dataset

Version 1.0, October 2024 <h2>Created by</h2> Mithun Manivannan (1), Vignesh Nethrapalli (1), Mark Cartwright (1) <ol> <li>Sound Interaction and Computer Lab, New Jersey Institute of Technology</li> </ol> <h2>Publication</h2> If using this data in an academic work, please reference the DOI and version, as well as cite the following paper, which presented the data collection procedure and the first version of the dataset: Manivannan, M., Nethrapalli, V., Cartwright, M. EmotionCaps: Enhancing Audio Captioning Through Emotion-Augmented Data Generation. arXiv preprint arXiv:2410.12028, 2024. <h2>Description</h2> EmotionCaps is a ChatGPT-assisted, weakly-labeled audio captioning dataset developed to bridge the gap between soundscape emotion recognition (SER) and automated audio captioning (AAC). Created through a three-stage pipeline, the dataset leverages ground-truth annotations from AudioSet SL, which are enhanced by ChatGPT using tailored prompts and emotions assigned via a soundscape emotion recognition model trained on Emo-Soundscapes Dataset. It comprises four subsets of captions for 120,071 audio clips, each reflecting a different prompt variation: WavCaps-like, Scene-Focused, Emotion Addon, and Emotion Rewrite. The average word counts for these subsets are: WavCaps-like (12.61), Scene-Focused (14.04), Emotion Addon (18.35), and Emotion Rewrite (18.65). The increase in word count for the emotion prompts illustrates the difference in sentence length when integrating emotion information into the captions. <h2>Audio Data</h2> The audio data is from AudioSet SL, the strongly-labled subset of 120,071 audio clips from the larger AudioSet dataset. <h2>Synthetic Captions</h2> The synthetic captions were generated using a three-stage pipeline, beginning with training a soundscape emotion recognition model. This model assesses the valence and arousal of each audio clip, mapping the resulting vector to an emotion identifier. Next, we leveraged the ground-truth annotations from AudioSet SL, and extracted the list of sound events. Using these sound events, we employed ChatGPT to create different variations of captions by applying distinct prompts. We first used the WavCaps prompt for AudioSet SL as a base, the output of which we call WavCaps-like. Building on this, we created three new prompt variations (1) scene-focused which is a modified WavCaps prompt that describes the scene, (2) emotion addon which is an extension of the scene-Focused prompt, where an emotion is appended to the list of sound events to guide the caption generation, and (3) emotion rewrite which consists of two-step prompt where ChatGPT first generates the scene-focused caption, then is instructed to rewrite it with a specific emotion in mind. Using these four prompt styles — WavCaps, Scene-Focused, Emotion Addon, and Emotion Rewrite — along with the AudioSet SL sound events and predicted emotions, we employed ChatGPT-3.5 Turbo to generate four corresponding caption variations for the dataset. Each caption variation has been organized into separate CSV files for clarity and accessibility. All files correspond to the same set of audio clips from AudioSet SL, with the key distinction being the caption variation associated with each clip. The different subsets are designed to be used independently, as they each fulfill specific roles in understanding the impact of emotion in audio captions. <ul> <li> wavcaps-like.csv: Contains captions generated using the WavCaps prompt, serving as the baseline before emotion is introduced. </li> <li> scene-focused.csv: Provides captions focused on describing the scene or environment of the audio clip, without emotion integration. </li> <li> emotion-addon.csv: Captions where emotion data is appended to the scene-focused base caption. </li> <li> emotion-rewrite.csv: Captions that are completely rewritten based on the scene-focused base caption and the assigned emotion. </li> </ul> This structure allows users to explore how emotional content influences captioning models by comparing the variations both with and without emotional enrichment. <h2>Columns in CSV files</h2> segment_id : The ID of the audio recording in AudioSet SL. These are in the form <YouTube ID>_<start time in ms>_<end time in ms> caption : The caption generated for each audio clip, corresponding to the specific subset (e.g., WavCaps, Scene-Focused, Emotion Addon, or Emotion Rewrite) as indicated by the file name. <h2>Conditions of use</h2> Dataset created by Mithun Manivannan, Vignesh Nethrapalli, Mark Cartwright The EmotionCaps dataset is offered free of charge under the terms of the Creative Commons Attribution 4.0 International (CC BY 4.0) license: <a href="https://creativecommons.org/licenses/by/4.0/">https://creativecommons.org/licenses/by/4.0/</a> The dataset and its contents are made available on an “as is” basis and without warranties of any kind, including without limitation satisfactory quality and conformity, merchantability, fitness for a particular purpose, accuracy or completeness, or absence of errors. Subject to any liability that may not be excluded or limited by law, New Jersey Institute of Technology is not liable for, and expressly excludes all liability for, loss or damage however and whenever caused to anyone by any use of the EmotionCaps dataset or any part of it. <h2>Feedback</h2> Please help us improve EmotionCaps by sending your feedback to: <ul> <li>Mithun Manivannan: <a href="mailto:mithun.mani01@gmail.com">mithun.mani01@gmail.com</a></li> <li>Mark Cartwright: <a href="mailto:mcartwright@gmail.com">mcartwright@gmail.com</a></li> </ul> In case of a problem, please include as many details as possible. <h2>Acknowledgments</h2> This work was partially supported by the New Jersey Institute of Technology Honors Summer Research Institute (HSRI).

opencc-by-4.0Dec 2022View details →

RAFT synthetic tropical cyclones dataset for Balaguru et al. 2022 - Science Advances

This is the RAFT synthetic tropical cyclone (TC) dataset generated for the paper "Increased US coastal hurricane risk under climate change" submitted to the journal Science Advances in 2022. Each file contains 50,000 synthetic TCs from RAFT either for the historical period (1980-2014) or the future period (2066-2100) under “SSP585”, and from a CMIP6 global climate model.       intensity_model_output_corrVMPI_11vars_alltcs_cutoff15_CMIP6_{PERIOD} _{MODEL}.mat To read a .mat file in Python, one can use “scipy.io.loadmat”. There are several variables included in each file, and all have the same dimension [number of storms, number of timesteps]. Here are a list of variable names and what they represent:       ‘lat’: Storm latitude;       ‘lon’: Storm longitude;       ‘year’: year;       ‘jday_syn’: Julian day in the year;       ‘vs0_syn’: maximum surface wind (knot). Please note that this version of synthetic TC dataset is only intended for assessing the large-scale change of hurricane risk under climate change (through statistical-dynamical downscaling of CMIP6 GCMs), which is addressed in the above mentioned paper. Due to the model biases in CMIP6 and the low temporal resolution (monthly) used for RAFT inputs, the synthetic TCs' life-time maximum intensity is underestimated. Therefore, the synthetic TCs here should not be treated directly as "example TCs of current or future climate" without bias correction on the TC intensity. The authors plan to release a separate version of RAFT simulated synthetic TCs with proper bias correction for localized TC impact assessment. Please email authors if you have questions.

openAug 2023View details →

Synthetic dataset for dual-perspective self-supervised learning

The synthetic Ca datasets for training and testing, including training dataset with bidirectional collinear scan (Ny = 2Nx) for MP-SSL, training dataset with normal scan (Ny = Nx) for TP-SSL, and testing data (Ny = Nx) If you use these data simulated using our modified <a href="https://doi.org/10.1016/j.jneumeth.2021.109173">NAOMi</a> model, please cite the corresponding work: <a href="https://doi.org/10.1186/s43074-023-00117-0">B. Shen, C. Luo, W. Pang, Y. Jiang, W. Wu, R. Hu, J. Qu, B. Gu, L. Liu. Surmounting photon limits and motion artifacts for biological dynamics imaging via dual-perspective self-supervised learning. PhotoniX 5, 1 (2024). </a>

opencc-by-nc-sa-4.0Mar 2024View details →

Synthetic Training Dataset for Real-World Terminal Strip Object Detection

This dataset provides synthetic training data for the real-world industrial application of terminal strip object detection to investigate the sim-to-real generalization performance of modern object detectors based on state-of-the-art image synthesis methods. It consists of 30.000 randomly generated synthetic images of terminal strips covering 36 different terminal blocks in five colors and additional accessories such as plug-in bridges, test adapters, end covers and markings. Except from the markings and the DIN rail all objects of the terminal strips are labeled with a bounding box and the respective object class for supervised learning. Additionally, 300 real images of terminal strips were taken and manually labeled for the real-world test. If you use this datset for your research, please consider citing this: <a href="https://arxiv.org/abs/2403.04809">Investigation of the Impact of Synthetic Training Data in the Industrial Application of Terminal Strip Object Detection</a>

Synthetic Datasets from the Article titled Privacy-preserving Ground-truth Data for Evaluating Additive Feature Attribution in Regression Models with Additive CBR and CQV

Synthetic datasets were generated as benchmarks capturing the intrinsic characteristics of original data to investigate the performance of additive feature attribution methods for regression tasks. The synthetic datasets were generated based on 2, 6 and 8 clusters formed with the original data. The 6-cluster dataset was used for primary analysis and the other two were used for sensitivity analysis.The synthetic dataset was generated from the original data acquired from <a href="https://www.eurocontrol.int/dashboard/rnd-data-archive">Aviation Data for Research Repository</a>, which was collected and processed by <a href="https://www.eurocontrol.int/">EUROCONTROL</a> from the Enhanced Tactical Flow Management System (ETFMS) flight data messages containing all flights in Europe throughout the year 2019, from May to October. The original dataset consisted of fundamental details of the flights, flight status, preceding flight legs, ATFM regulations, weather conditions, calendar information, etc. A brief description of the columns in the synthetic data files is presented in the file 'data_description.pdf' and a more detailed discussion on features can be found in the works of Koolen and Coliban [1] and Dalmau et al. [2]. References [1]  H. Koolen and I. Coliban, <a href="https://www.eurocontrol.int/sites/default/files/2020-06/flight-progress-msg-update-230620.pdf">Flight Progress Messages Document</a>, EUROCONTROL, Brussels, Belgium, Tech. Rep., 2020. [2]  R. Dalmau, F. Ballerini, H. Naessens, S. Belkoura, and S. Wangnick, <a href="https://www.sciencedirect.com/science/article/pii/S0969699721000739">An Explainable Machine Learning Approach to Improve Take-off Time Predictions</a>, Journal of Air Transport Management, vol. 95, p. 102 090, Aug. 2021. doi: 10.1016/j.jairtraman.2021.102090.

Small spots DeepMIB project, synthetic dataset for testing 2D semantic segmentation

A complete DeepMIB project with a synthetic dataset generated for quick tests of semantic segmentation approaches. The dataset includes a trained U-net network for detection of small random spots of 2 colors on a black background.The network can be opened by loading "2D_SmallSpots_3cl_Unet.mibCfg" file by<ul><li>MIB->Menu->Tools->Deep learning segmentation->Options tab->Config files->Load </li><li>Drag and drop of the config file into DeepMIB window</li></ul>Microscopy Image Browser: <a href="https://mib.helsinki.fi">https://mib.helsinki.fi</a>

Large spots DeepMIB project, synthetic dataset for testing 2D patch-wise segmentation

A complete DeepMIB project with a synthetic dataset generated for quick tests of the patch-wise segmentation (classification) approaches The dataset includes a trained Resnet18 network for detection of patches that belong to large white spots on black background. The network can be opened by loading "2D_LargeSpots_Patchwise_Resnet18.mibCfg" by<ul><li>MIB->Menu->Tools->Deep learning segmentation->Options tab->Config files->Load </li><li>Drag and drop of the config file into DeepMIB window</li></ul>Microscopy Image Browser: <a href="https://mib.helsinki.fi">https://mib.helsinki.fi</a>

Large Spots DeepMIB project, synthetic dataset for testing 2.5D semantic segmentation

A complete DeepMIB project with a synthetic dataset generated for quick tests of 2.5D semantic segmentation approaches. The dataset includes trained<ul><li>2.5D DeepLabV3-Resnet18 depth2color (Spots_25D_DLv3RN18_Z2C_xy200z5)</li><li>2.5D U-net depth2color (Spots_25D_Unet_Z2C_xy200z5)</li></ul>networkd for detection of large white 3D spots on a black background. The spots that are present on a single slice only are considered as background.The network can be opened by loading the config files "*.mibCfg" by<ul><li>MIB->Menu->Tools->Deep learning segmentation->Options tab->Config files->Load </li><li>Drag and drop of the config file into DeepMIB window</li></ul>Microscopy Image Browser: <a href="https://mib.helsinki.fi/">https://mib.helsinki.fi</a>

opencc-by-4.0Dec 2023View details →

Synthetic GW dataset with latent diffusion models

SyntheticHTR: Handwritten Text Image Synthesis based on Latent Diffusion Models

opencc-by-4.0Dec 2023View details →

Synthetic basement depth, gravity anomalies, density and observation points training dataset to train deep learning model

Contains 200000 training data for our DNN model.