Find research datasets worth reusing

opencc-by-4.0Apr 2020View details →

Dataset for "Remapping of Greenland ice sheet surface mass balance anomalies for large ensemble sea-level change projections"

This dataset is used to reproduce the results presented in the following publication: Goelzer, H., Noel, B. P. Y., Edwards, T. L., Fettweis, X., Gregory, J. M., Lipscomb, W. H., van de Wal, R. S. W., and van den Broeke, M. R.: Remapping of Greenland ice sheet surface mass balance anomalies for large ensemble sea-level change projections, The Cryosphere Discuss., https://doi.org/10.5194/tc-2019-188, in review, 2019.

opencc-by-4.0Nov 2024View details →

Dataset for the paper "Anomaly Detection in Large-Scale Cloud Systems: An Industry Case and Dataset"

We present a large-scale anomaly detection dataset collected from IBM Cloud's Console over approximately 4.5 months. This high-dimensional dataset captures telemetry data from multiple data centers, specifically designed to aid researchers in developing and benchmarking anomaly detection methods in large-scale cloud environments. It contains 39,365 entries, each representing a 5-minute interval, with 117,448 features/attributes, as interval_start is used as the index. The dataset includes detailed information on request counts, HTTP response codes, and various aggregated statistics. The dataset also includes labeled anomaly events identified through IBM's internal monitoring tools, providing a comprehensive resource for real-world anomaly detection research and evaluation. File Descriptions <ul> <li><code>location_downtime.csv</code> - Details planned and unplanned downtimes for IBM Cloud data centers, including start and end times in ISO 8601 format.</li> <li><code>unpivoted_data.parquet</code> - Contains raw telemetry data with 413 million+ rows, covering details like location, HTTP status codes, request types, and aggregated statistics (min, max, median response times).</li> <li><code>anomaly_windows.csv</code> - Ground truth for anomalies, listing start and end times of recorded anomalies, categorized by source (Issue Tracker, Instant Messenger, Test Log).</li> <li><code>pivoted_data_all.parquet</code> - Pivoted version of the telemetry dataset with 39,365 rows and 117,449 columns, including aggregated statistics across multiple metrics and intervals.</li> <li><code>demo/demo.[ipynb|html]</code>: This demo file provides examples of how to access data in the Parquet files, available in Jupyter Notebook (<code>.ipynb</code>) and HTML (<code>.html</code>) formats, respectively.</li> </ul> Further details of the dataset can be found in Appendix B: Dataset Characteristics of the <a href="https://arxiv.org/abs/2411.09047">paper</a> titled "Anomaly Detection in Large-Scale Cloud Systems: An Industry Case and Dataset." Sample code for training anomaly detectors using this data is provided in <a href="https://doi.org/10.5281/zenodo.14598119" target="_blank" rel="noopener">this package</a>.   When using the dataset, please cite it as follows: <pre><code>@misc{islam2024anomaly,</code> <code>      title={Anomaly Detection in Large-Scale Cloud Systems: An Industry Case and Dataset}, </code> <code>      author={Mohammad Saiful Islam and Mohamed Sami Rakha and William Pourmajidi and Janakan Sivaloganathan and John Steinbacher and Andriy Miranskyy},</code> <code>      year={2024},</code> <code>      eprint={2411.09047},</code> <code>     archivePrefix={arXiv},</code> <code>     url={https://arxiv.org/abs/2411.09047}</code> <code>}</code></pre>

opencc-by-4.0Jun 2024View details →

OPSSAT-AD - anomaly detection dataset for satellite telemetry

This is the AI-ready benchmark dataset (OPSSAT-AD) containing the telemetry data acquired on board OPS-SAT---a CubeSat mission that has been operated by the European Space Agency. It is accompanied by the paper with baseline results obtained using 30 supervised and unsupervised classic and deep machine learning algorithms for anomaly detection. They were trained and validated using the training-test dataset split introduced in this work, and we present a suggested set of quality metrics that should always be calculated to confront the new algorithms for anomaly detection while exploiting OPSSAT-AD. We believe that this work may become an important step toward building a fair, reproducible, and objective validation procedure that can be used to quantify the capabilities of the emerging anomaly detection techniques in an unbiased and fully transparent way. The included files are: <ul> <li><code>segments.csv</code> with the acquired telemetry signals from ESA OPS-SAT aircraft,</li> <li><code>dataset.csv</code> with the extracted, synthetic features are computed for each manually split and labeled telemetry segment.</li> <li>code files for data processing and example modeliing (<code>dataset_generator.ipynb</code> for data processing, <code>modeling_examples.ipynb</code> with simple examples,  <code>requirements.txt</code>- with details on Python configuration, and the <code>LICENSE</code> file)</li> </ul>   Please have a look at our two papers commenting on this dataset: <ul> <li>The benchmark paper with results of 30 supervised and unsupervised anomaly detection models for this collection: Ruszczak, B., Kotowski. K., Nalepa, J., Evans, D.: The OPS-SAT benchmark for detecting anomalies in satellite telemetry, 2024, <a href="https://arxiv.org/abs/2407.04730" target="_blank" rel="noopener">preprint arxiv: 2407.04730</a>,</li> <li>the conference paper in which we presented some preliminary results for this dataset: Ruszczak, B., Kotowski. K., Andrzejewski, J., et al.: (2023). Machine Learning Detects Anomalies in OPS-SAT Telemetry. Computational Science – ICCS 2023. LNCS, vol 14073. Springer, Cham, <a href="https://doi.org/10.1007/978-3-031-35995-8_21">DOI:10.1007/978-3-031-35995-8_21</a>.</li> </ul>

opencc-by-4.0Dec 2021View details →

Packaging Industry Anomaly DEtection (PIADE) Dataset

PIADE dataset contains data from five industrial packaging machines: <ul> <li>Machine s_1: from 2020-01-01 14:00:00 to 2021-12-31 13:00:00</li> <li>Machine s_2: from 2020-06-17 08:00:00 to 2021-12-31 07:00:00</li> <li>Machine s_3: from 2020-10-07 12:00:00 to 2022-01-01 23:00:00</li> <li>Machine s_4: from 2020-01-01 01:00:00 to 2022-01-01 23:00:00</li> <li>Machine s_5: from 2020-01-20 08:00:00 to 2022-01-01 12:00:00</li> </ul> ## Raw Data Each row represents a production interval, with the following schema: <ul> <li>interval_start: start of the production interval    </li> <li>equipment_ID: equipment identifier    </li> <li>alarm: alarm code of the active stop reason, if it occurred     </li> <li>type: idle, production, downtime, performance_loss or scheduled_downtime    </li> <li>start: start of the production interval    </li> <li>end: end of the production interval    </li> <li>elapsed: duration of the production interval    </li> <li>pi: input packages    </li> <li>po: output packages    </li> <li>speed: speed (packages per hour)</li> </ul> There are 133 different types of alerts, and 429394 rows.   ## Sequences (1h) data For each piece of equipment, we define sequences of length = 1 hour and we aggregate raw interval data as follows: <ul> <li>'equipment_ID': machine identifier</li> <li>'#changes': changes in machine state</li> <li>'%downtime': time spent in 'downtime' state</li> <li>'%idle': time spent in 'idle' state</li> <li>'%performance_loss': time spent in 'performance loss' state</li> <li>'%production': time spent in production</li> <li>'%scheduled_downtime': time spent in scheduled downtime</li> <li>'count_sum': sum of all alarm occurrences</li> <li>'A_<XXX>': counter of alarm <XXX> occurrences</li> <li>'<state1>/<state2>': number of transitions from <state1> to <state2></li> </ul>

opencc-by-4.0Aug 2024View details →

Three Annotated Anomaly Detection Datasets for Line-Scan Algorithms

<h1>Summary</h1> This dataset contains two hyperspectral and one multispectral anomaly detection images, and their corresponding binary pixel masks. They were initially used for real-time anomaly detection in line-scanning, but they can be used for any anomaly detection task. They are in .npy file format (will add tiff or geotiff variants in the future), with the image datasets being in the order of (height, width, channels). The SNP dataset was collected using sentinelhub, and the Synthetic dataset was collected from AVIRIS. The Python code used to analyse these datasets can be found at: https://github.com/WiseGamgee/HyperAD <h1>How to Get Started</h1> All that is needed to load these datasets is Python (preferably 3.8+) and the NumPy package. Example code for loading the Beach Dataset if you put it in a folder called "data" with the python script is: <pre><code>import numpy as np # Load image file hsi_array = np.load("data/beach_hsi.npy") n_pixels, n_lines, n_bands = hsi_array.shape print(f"This dataset has {n_pixels} pixels, {n_lines} lines, and {n_bands}.") # Load image mask mask_array = np.load("data/beach_mask.npy") m_pixels, m_lines = mask_array.shape print(f"The corresponding anomaly mask is {m_pixels} pixels by {m_lines} lines.")</code></pre> <h1>Citing the Datasets</h1> If you use any of these datasets, please cite the following paper: <pre><code>@article{garske2024erx,</code> <code>  title={ERX - a Fast Real-Time Anomaly Detection Algorithm for Hyperspectral Line-Scanning},</code> <code>  author={Garske, Samuel and Evans, Bradley and Artlett, Christopher and Wong, KC},</code> <code>  journal={arXiv preprint arXiv:2408.14947},</code> <code>  year={2024},</code> <code>}</code></pre> <div> <pre>If you use the beach dataset please cite the following paper as well (original source):</pre> </div> <pre><code>@article{mao2022openhsi, title={OpenHSI: A complete open-source hyperspectral imaging solution for everyone}, author={Mao, Yiwei and Betters, Christopher H and Evans, Bradley and Artlett, Christopher P and Leon-Saval, Sergio G and Garske, Samuel and Cairns, Iver H and Cocks, Terry and Winter, Robert and Dell, Timothy}, journal={Remote Sensing}, volume={14}, number={9}, pages={2244}, year={2022}, publisher={MDPI} }</code></pre>

opencc-by-4.0Sep 2020View details →

A global dataset of SST anomaly evolving processes retrieved from remote sensing products (GDSSTAEP V1.0)

The GDSSTAEP includes three datasets and two relationship files with a time range from January 1982 to December 2009. Three datasets formatted in SHP are a dataset of process object-oriented SSTA, named DSPOSSTA, storing SSTA process objects, a dataset of sequence object-oriented SSTA, named DSSOSSTA, storing SSTA sequence objects, and a dataset of variation object-oriented SSTA, named DSVOSSTA, storing SSTA variation objects, respectively. And two relationship files formatted in CSV store the evolving behaviors among sequence objects of SSTA and variation objects of SSTA, respectively.

opencc-by-4.0Nov 2020View details →

Precipitation and Temperature Anomalies from MERRA-2 dataset

Anomalies in Precipitation and Temperature for tiles around the globe. Data derived from the reanalysis MERRA-2 project [1]. The dataset covers the period from 1980 to 2018, between 80ºN and 80ºS. The spatial resolution is 1.0º x 1.0º resulting in 45,792 tiles and a time resolution of 7 days (by averaging the values over each week). Precipitation time-series were re-scaled by applying a logarithmic function to all values. To discount seasonality effects, we averaged temperature and precipitation values for each of the 365 calendar days. We considered the interval from 1 January 1980 through 28 February 2018 as the climatic period for which the long-term averages were computed. The anomalies are then obtained by subtracting for each day the respective average temperature or precipitation from the climatic period (for example, the 1 January 1998 anomaly is computed as the value for that day minus the average of all January 1 values between 1980 and 2018). To process the data, use the code available in: <a href="https://github.com/filipinascimento/teleconnectionsgranger/">https://github.com/filipinascimento/teleconnectionsgranger/</a> <a href="http://arxiv.org/abs/2012.03848">http://arxiv.org/abs/2012.03848</a> [1] A. Molod, L. Takacs, M. Suarez, and J. Bacmeister, “Development of the geos-5 atmospheric general circulation model: evolution from merra to merra2,” Geoscientific Model Development 8, 1339–1356 (2015).

opencc-by-4.0Apr 2024View details →

Dataset for the paper "Historical model biases in monthly high temperature anomalies indicate under-projection of future temperature extremes"

<div> <div>This repository holds data and scripts related to the revision of the paper entitled: "Historical model biases in monthly high temperature anomalies indicate under-projection of future temperature extremes" by Lei Duan, Lyssa M. Freese, Govindasamy Bala, and Ken Caldeira. The paper is currently submitted for peer review. Any questions regarding the data and paper could be sent to the corresponding author: Lei Duan (leiduan@carnegiescience.edu). </div> </div>

opencc-by-4.0Nov 2024View details →

Geospatial Dataset of GNSS Anomalies and Political Violence Events

Geospatial Dataset of GNSS Anomalies and Political Violence Events Overview The Geospatial Dataset of GNSS Anomalies and Political Violence Events is a collection of data that integrates aircraft flight information, GNSS (Global Navigation Satellite System) anomalies, and political violence events from the ACLED (Armed Conflict Location & Event Data Project) database. Dataset Files The dataset consists of three CSV files: <ol> <li>Daily_GNSS_Anomalies_and_ACLED-2023-V1.csv</li> <ul> <li>Description: Contains all grids and dates that had aircraft traffic during 2023.</li> <li>Number of Records: 6,777,228</li> <li>Purpose: Provides a complete view of aircraft movements and associated data, including grids without any GNSS anomalies.</li> </ul> <li>Daily_GNSS_Anomalies_and_ACLED-2023-V2.csv</li> <ul> <li>Description: A filtered version of V1, including only the grids and dates where GNSS anomalies (jumps or gaps) were reported.</li> <li>Number of Records: 718,237</li> <li>Purpose: Focuses on areas and times with GNSS anomalies for targeted analysis.</li> </ul> <li>Monthly_GNSS_Anomalies_and_ACLED-2023-V9.csv</li> <ul> <li>Description: Contains aggregated monthly data for each grid cell, combining GNSS anomalies and ACLED political violence events. Summarizes aircraft traffic, anomaly counts, and conflict activity at a monthly resolution.</li> <li>Number of Records: 25,770</li> <li>Purpose: Enables temporal trend analysis and spatial correlation studies between GNSS interference and political violence, using reduced data volume suitable for modeling and visualization.</li> </ul> </ol> Data Fields:    Daily_GNSS_Anomalies_and_ACLED-2023-V1.csv and Daily_GNSS_Anomalies_and_ACLED-2023-V2.csv <ol> <li>grid_id</li> <ul> <li>Description: Unique identifier for a grid cell on Earth measuring 0.5 degrees latitude by 0.5 degrees longitude.</li> <li>Format: String combining latitude and longitude (e.g., -10.0_-36.0).</li> </ul> <li>day</li> <ul> <li>Description: Date of the recorded data.</li> <li>Format: YYYY-MM-DD (e.g., 2023-03-28).</li> </ul> <li>geometry</li> <ul> <li>Description: Polygon coordinates of the grid cell in Well-Known Text (WKT) format.</li> <li>Format: POLYGON((longitude latitude, ...)) (e.g., POLYGON((-36.0 -10.0, -35.5 -10.0, -35.5 -9.5, -36.0 -9.5, -36.0 -10.0))).</li> </ul> <li>flights</li> <ul> <li>Description: Number of aircraft flights that passed through the grid on that day.</li> <li>Format: Integer (e.g., 28).</li> </ul> <li>GPS_jumps</li> <ul> <li>Description: Number of reported GNSS "jump" anomalies (possible spoofing incidents) in the grid on that day.</li> <li>Format: Integer (e.g., 1).</li> </ul> <li>GPS_gaps</li> <ul> <li>Description: Number of reported GNSS "gap" anomalies, indicating gaps in aircraft routes, in the grid on that day.</li> <li>Format: Integer (e.g., 0).</li> </ul> <li>gaps_density</li> <ul> <li>Description: Density of GNSS gaps, calculated as the number of gaps divided by the number of flights.</li> <li>Format: Decimal (e.g., 0).</li> </ul> <li>jumps_density</li> <ul> <li>Description: Density of GNSS jumps, calculated as the number of jumps divided by the number of flights.</li> <li>Format: Decimal (e.g., 0.035714286).</li> </ul> <li>event_id_cnty</li> <ul> <li>Description: ACLED event ID corresponding to political violence events in the grid on that day.</li> <li>Format: String (e.g., BRA69267).</li> </ul> <li>disorder_type</li> <ul> <li>Description: Type of disorder as classified by ACLED (e.g., "Political violence").</li> <li>Format: String.</li> </ul> <li>event_type</li> <ul> <li>Description: General category of the event according to ACLED (e.g., "Violence against civilians").</li> <li>Format: String.</li> </ul> <li>sub_event_type</li> <ul> <li>Description: Specific subtype of the event as per ACLED classification (e.g., "Attack").</li> <li>Format: String.</li> </ul> <li>acled_count</li> <ul> <li>Description: Number of ACLED events in the grid on that day.</li> <li>Format: Integer (e.g., 1).</li> </ul> <li>acled_flag</li> <ul> <li>Description: Indicator of ACLED event presence in the grid on that day (0 for no events, 1 for one or more events).</li> <li>Format: Integer (0 or 1).</li> </ul> </ol>   Data Fields: Monthly_GNSS_Anomalies_and_ACLED-2023-V9.csv The file contains monthly aggregated GNSS anomaly and ACLED event data per grid cell. The structure and meaning of each field are detailed below: <ol> <li>grid_id</li> <ul> <li>Description: Unique identifier for a grid cell on Earth measuring 0.5° latitude by 0.5° longitude.</li> <li>Format: String combining latitude and longitude (e.g., -0.5_-79.0).</li> </ul> <li>year_month</li> <ul> <li>Description: Month and year of the aggregated data.</li> <li>Format: String in Mon-YY format (e.g., Jan-23).</li> </ul> <li>geometry</li> <ul> <li>Description: Polygon coordinates of the grid cell in Well-Known Text (WKT) format.</li> <li>Format: POLYGON((longitude latitude, ...)) (e.g., POLYGON((-79.0 -0.5, -78.5 -0.5, -78.5 0.0, -79.0 0.0, -79.0 -0.5))).</li> </ul> <li>flights</li> <ul> <li>Description: Total number of aircraft flights that passed through the grid cell during the month.</li> <li>Format: Integer (e.g., 1230).</li> </ul> <li>GPS_jumps</li> <ul> <li>Description: Total number of GNSS "jump" anomalies (possible spoofing events) in the grid cell during the month.</li> <li>Format: Integer (e.g., 13).</li> </ul> <li>GPS_gaps</li> <ul> <li>Description: Total number of GNSS "gap" anomalies, indicating interruptions in aircraft routes, during the month.</li> <li>Format: Integer (e.g., 0).</li> </ul> <li>event_id_cnty</li> <ul> <li>Description: Semicolon-separated list of ACLED event IDs associated with the grid cell during the month.</li> <li>Format: String (e.g., ECU3151;ECU3158;ECU3150).</li> </ul> <li>disorder_type</li> <ul> <li>Description: Semicolon-separated list of disorder types (e.g., "Political violence", "Demonstrations") reported by ACLED in that grid cell during the month.</li> <li>Format: String.</li> </ul> <li>event_type</li> <ul> <li>Description: Semicolon-separated list of high-level ACLED event types (e.g., "Riots", "Protests").</li> <li>Format: String.</li> </ul> <li>sub_event_type</li> </ol> <ul> <li>Description: Semicolon-separated list of detailed subtypes of ACLED events (e.g., "Mob violence", "Armed clash").</li> <li>Format: String.</li> </ul> <ol> <li>acled_count</li> </ol> <ul> <li>Description: Total number of ACLED conflict events in the grid cell during the month.</li> <li>Format: Integer (e.g., 2).</li> </ul> <ol> <li>acled_flag</li> </ol> <ul> <li>Description: Conflict presence indicator: 1 if any ACLED event occurred in the grid cell during the month, otherwise 0.</li> <li>Format: Integer (0 or 1).</li> </ul> <ol> <li>gaps_density</li> </ol> <ul> <li>Description: Monthly density of GNSS gaps, calculated as GPS_gaps / flights.</li> <li>Format: Decimal (e.g., 0.0).</li> </ul> <ol> <li>jumps_density</li> </ol> <ul> <li>Description: Monthly density of GNSS jumps, calculated as GPS_jumps / flights.</li> <li>Format: Decimal (e.g., 0.0106).</li> </ul>   Data Sources <ul> <li>GNSS Anomalies Data:</li> <ul> <li>Calculated from ADS-B (Automatic Dependent Surveillance-Broadcast) messages obtained via the OpenSky Network's Trino database.</li> <li>GNSS anomalies include "jumps" (potential spoofing incidents) and "gaps" (interruptions in aircraft route data).</li> </ul> <li>Political Violence Events Data:</li> <ul> <li>Sourced from the ACLED database, which provides detailed information on political violence and protest events worldwide.</li> </ul> </ul> Temporal and Spatial Coverage <ul> <li>Temporal Coverage:</li> <ul> <li>From January 1, 2023, to December 31, 2023.</li> <li>Daily records provide temporal granularity for time-series analysis.</li> </ul> <li>Spatial Coverage:</li> <ul> <li>Global coverage with grid cells measuring 0.5 degrees latitude by 0.5 degrees longitude.</li> <li>Each grid cell represents an area on Earth's surface, facilitating spatial analysis.</li> </ul> </ul> Usage and Applications <ul> <li>Security Analysis:</li> <ul> <li>Assess potential correlations between GNSS anomalies and political violence events.</li> <li>Identify regions with increased risk of GNSS spoofing or signal disruption.</li> </ul> <li>Research and Development:</li> <ul> <li>Develop models to predict socio-political events based on GNSS anomalies.</li> <li>Study the impact of political instability on aviation safety.</li> </ul> <li>Policy and Decision Making:</li> <ul> <li>Inform aviation authorities and policymakers about regions requiring enhanced navigation security measures.</li> <li>Support conflict analysis and monitoring efforts.</li> </ul> </ul>

opencc-by-4.0Oct 2023View details →

Dataset for Investigating Anomalies in Compute Clusters

AbstractThe dataset was collected for 332 compute nodes throughout May 19 - 23, 2023. May 19 - 22 characterizes normal compute cluster behavior, while May 23 includes an anomalous event. The dataset includes eight CPU, 11 disk, 47 memory, and 22 Slurm metrics. It represents five distinct hardware configurations and contains over one million records, totaling more than 180GB of raw data.BackgroundMotivated by the goal to develop a digital twin of a compute cluster, the dataset was collected using a Prometheus server (1) scraping the Thomas Jefferson National Accelerator Facility (JLab) batch cluster used to run an assortment of physics analysis and simulation jobs, where analysis workloads leverage data generated from the laboratory's electron accelerator, and simulation workloads generate large amounts of flat data that is then carved to verify amplitudes. Metrics were scraped from the cluster throughout May 19 - 23, 2023. Data from May 19 to May 22 primarily reflected normal system behavior, while May 23, 2023, recorded a notable anomaly. This anomaly was severe enough to necessitate intervention by JLab IT Operations staff.The metrics were collected from CPU, disk, memory, and Slurm. Metrics related to CPU, disk, and memory provide insights into the status of individual compute nodes. Furthermore, Slurm metrics collected from the network have the capability to detect anomalies that may propagate to compute nodes executing the same job.Usage NotesWhile the data from May 19 - 22 characterizes normal compute cluster behavior, and May 23 includes anomalous observations, the dataset cannot be considered labeled data. The set of nodes and the exact start and end time affected nodes demonstrate abnormal effects are unclear. Thus, the dataset could be used to develop unsupervised machine-learning algorithms to detect anomalous events in a batch cluster.<a href="https://doi.org/10.48550/arXiv.2311.16129">https://doi.org/10.48550/arXiv.2311.16129</a>

opencc-by-nc-sa-4.0Feb 2024View details →

The Automotive Visual Inspection Dataset (AutoVI): A Genuine Industrial Production Dataset for Unsupervised Anomaly Detection

See the official website: <a href="https://autovi.utc.fr">https://autovi.utc.fr</a> Modern industrial production lines must be set up with robust defect inspection modules that are able to withstand high product variability. This means that in a context of industrial production, new defects that are not yet known may appear, and must therefore be identified. On industrial production lines, the typology of potential defects is vast (texture, part failure, logical defects, etc.). Inspection systems must therefore be able to detect non-listed defects, i.e. not-yet-observed defects upon the development of the inspection system. To solve this problem, research and development of unsupervised AI algorithms on real-world data is required. Renault Group and the Université de technologie de Compiègne (Roberval and Heudiasyc Laboratories) have jointly developed the Automotive Visual Inspection Dataset (AutoVI), the purpose of which is to be used as a scientific benchmark to compare and develop advanced unsupervised anomaly detection algorithms under real production conditions. The images were acquired on Renault Group's automotive production lines, in a genuine industrial production line environment, with variations in brightness and lighting on constantly moving components. This dataset is representative of actual data acquisition conditions on automotive production lines. The dataset contains 3950 images, split into 1530 training images and 2420 testing images. The evaluation code can be found at <a href="https://github.com/phcarval/autovi_evaluation_code">https://github.com/phcarval/autovi_evaluation_code</a>. Disclaimer All defects shown were intentionally created on Renault Group's production lines for the purpose of producing this dataset. The images were examined and labeled by Renault Group experts, and all defects were corrected after shooting. License Copyright © 2023-2024 Renault Group This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. To view a copy of the license, visit <a href="https://creativecommons.org/licenses/by-nc-sa/4.0/">https://creativecommons.org/licenses/by-nc-sa/4.0/</a>. For using the data in a way that falls under the commercial use clause of the license, please contact us. Attribution Please use the following for citing the dataset in scientific work: Carvalho, P., Lafou, M., Durupt, A., Leblanc, A., & Grandvalet, Y. (2024). The Automotive Visual Inspection Dataset (AutoVI): A Genuine Industrial Production Dataset for Unsupervised Anomaly Detection [Dataset]. <a href="https://doi.org/10.5281/zenodo.10459003">https://doi.org/10.5281/zenodo.10459003</a> Contact If you have any questions or remarks about this dataset, please contact us at philippe.carvalho@utc.fr, meriem.lafou@renault.com, alexandre.durupt@utc.fr, antoine.leblanc@renault.com, yves.grandvalet@utc.fr. Changelog <ul> <li>v1.0.0 <ul> <li>Cropped engine_wiring, pipe_clip and pipe_staple images</li> <li>Reduced tank_screw, underbody_pipes and underbody_screw image sizes</li> </ul> </li> <li>v0.1.1 <ul> <li>Added ground truth segmentation maps</li> <li>Fixed categorization of some images</li> <li>Added new defect categories</li> <li>Removed tube_fastening and kitting_cart</li> <li>Removed duplicates in pipe_clip</li> </ul> </li> </ul>

opencc-by-4.0Sep 2024View details →

CESNET-TimeSeries24: Time Series Dataset for Network Traffic Anomaly Detection and Forecasting

<h2>CESNET-TimeSeries24: The dataset for network traffic forecasting and anomaly detection</h2> The dataset called CESNET-TimeSeries24 was collected by long-term monitoring of selected statistical metrics for 40 weeks for each IP address on the ISP network CESNET3 (Czech Education and Science Network). The dataset encompasses network traffic from more than 275,000 active IP addresses, assigned to a wide variety of devices, including office computers, NATs, servers, WiFi routers, honeypots, and video-game consoles found in dormitories. Moreover, the dataset is also rich in network anomaly types since it contains all types of anomalies, ensuring a comprehensive evaluation of anomaly detection methods. Last but not least, the CESNET-TimeSeries24 dataset provides traffic time series on institutional and IP subnet levels to cover all possible anomaly detection or forecasting scopes. Overall, the time series dataset was created from the 66 billion IP flows that contain 4 trillion packets that carry approximately 3.7 petabytes of data. The CESNET-TimeSeries24 dataset is a complex real-world dataset that will finally bring insights into the evaluation of forecasting models in real-world environments. Please cite the usage of our dataset as: <blockquote> Koumar, J., Hynek, K., Čejka, T. et al. CESNET-TimeSeries24: Time Series Dataset for Network Traffic Anomaly Detection and Forecasting. Sci Data 12, 338 (2025). https://doi.org/10.1038/s41597-025-04603-x @Article{cesnettimeseries24,     author={Koumar, Josef and Hynek, Karel and {\v{C}}ejka, Tom{\'a}{\v{s}} and {\v{S}}i{\v{s}}ka, Pavel},     title={CESNET-TimeSeries24: Time Series Dataset for Network Traffic Anomaly Detection and Forecasting},     journal={Scientific Data},     year={2025},     month={Feb},     day={26},     volume={12},     number={1},     pages={338},     issn={2052-4463},     doi={10.1038/s41597-025-04603-x},     url={https://doi.org/10.1038/s41597-025-04603-x} } </blockquote>   <h3>Time series</h3> We create evenly spaced time series for each IP address by aggregating IP flow records into time series datapoints. The created datapoints represent the behavior of IP addresses within a defined time window of 10 minutes. The vector of time-series metrics v_{ip, i} describes the IP address ip in the i-th time window. Thus, IP flows for vector v_{ip, i} are captured in time windows starting at t_i and ending at t_{i+1}. The time series are built from these datapoints.   Datapoints created by the aggregation of IP flows contain the following time-series metrics: <ul> <li>Simple volumetric metrics: the number of IP flows, the number of packets, and the transmitted data size (i.e. number of bytes)</li> <li>Unique volumetric metrics: the number of unique destination IP addresses, the number of unique destination Autonomous System Numbers (ASNs), and the number of unique destination transport layer ports. The aggregation of \textit{Unique volumetric metrics} is memory intensive since all unique values must be stored in an array. We used a server with 41 GB of RAM, which was enough for 10-minute aggregation on the ISP network.   </li> <li>Ratios metrics: the ratio of UDP/TCP packets, the ratio of UDP/TCP transmitted data size, the direction ratio of packets, and the direction ratio of transmitted data size</li> <li>Average metrics: the average flow duration, and the average Time To Live (TTL)</li> </ul>   Multiple time aggregation:  The original datapoints in the dataset are aggregated by 10 minutes of network traffic. The size of the aggregation interval influences anomaly detection procedures, mainly the training speed of the detection model. However, the 10-minute intervals can be too short for longitudinal anomaly detection methods. Therefore, we added two more aggregation intervals to the datasets--1 hour and 1 day. Time series of institutions:  We identify 283 institutions inside the CESNET3 network. These time series aggregated per each institution ID provide a view of the institution's data.  Time series of institutional subnets: We identify 548 institution subnets inside the CESNET3 network. These time series aggregated per each institution ID provide a view of the institution subnet's data.    <h3>Data Records</h3> The file hierarchy is described below: <blockquote> cesnet-timeseries24/      |- institution_subnets/      |     |- agg_10_minutes/<id_institution>.csv      |     |- agg_1_hour/<id_institution>.csv      |     |- agg_1_day/<id_institution>.csv      |     |- identifiers.csv      |- institutions/      |     |- agg_10_minutes/<id_institution_subnet>.csv      |     |- agg_1_hour/<id_institution_subnet>.csv      |     |- agg_1_day/<id_institution_subnet>.csv      |     |- identifiers.csv      |- ip_addresses_full/      |     |- agg_10_minutes/<id_ip_folder>/<id_ip>.csv      |     |- agg_1_hour/<id_ip_folder>/<id_ip>.csv      |     |- agg_1_day/<id_ip_folder>/<id_ip>.csv      |     |- identifiers.csv      |- ip_addresses_sample/      |      |- agg_10_minutes/<id_ip>.csv      |      |- agg_1_hour/<id_ip>.csv      |      |- agg_1_day/<id_ip>.csv      |      |- identifiers.csv      |- times/      |      |- times_10_minutes.csv      |      |- times_1_hour.csv      |      |- times_1_day.csv      |- ids_relationship.csv      |- weekends_and_holidays.csv </blockquote> The following list describes time series data fields in CSV files: <ul> <li>id_time:  Unique identifier for each aggregation interval within the time series, used to segment the dataset into specific time periods for analysis.</li> <li>n_flows: Total number of flows observed in the aggregation interval, indicating the volume of distinct sessions or connections for the IP address.</li> <li>n_packets: Total number of packets transmitted during the aggregation interval, reflecting the packet-level traffic volume for the IP address.</li> <li>n_bytes: Total number of bytes transmitted during the aggregation interval, representing the data volume for the IP address.</li> <li>n_dest_ip: Number of unique destination IP addresses contacted by the IP address during the aggregation interval, showing the diversity of endpoints reached.</li> <li>n_dest_asn: Number of unique destination Autonomous System Numbers (ASNs) contacted by the IP address during the aggregation interval, indicating the diversity of networks reached.</li> <li>n_dest_port: Number of unique destination transport layer ports contacted by the IP address during the aggregation interval, representing the variety of services accessed.</li> <li>tcp_udp_ratio_packets: Ratio of packets sent using TCP versus UDP by the IP address during the aggregation interval, providing insight into the transport protocol usage pattern. This metric belongs to the interval <0, 1> where 1 is when all packets are sent over TCP, and 0 is when all packets are sent over UDP.</li> <li>tcp_udp_ratio_bytes: Ratio of bytes sent using TCP versus UDP by the IP address during the aggregation interval, highlighting the data volume distribution between protocols. This metric belongs to the interval <0, 1>  with same rule as tcp_udp_ratio_packets.</li> <li>dir_ratio_packets: Ratio of packet directions (inbound versus outbound) for the IP address during the aggregation interval, indicating the balance of traffic flow directions. This metric belongs to the interval <0, 1>, where 1 is when all packets are sent in the outgoing direction from the monitored IP address, and 0 is when all packets are sent in the incoming direction to the monitored IP address.</li> <li>dir_ratio_bytes: Ratio of byte directions (inbound versus outbound) for the IP address during the aggregation interval, showing the data volume distribution in traffic flows. This metric belongs to the interval <0, 1> with the same rule as dir_ratio_packets.</li> <li>avg_duration: Average duration of IP flows for the IP address during the aggregation interval, measuring the typical session length.</li> <li>avg_ttl: Average Time To Live (TTL) of IP flows for the IP address during the aggregation interval, providing insight into the lifespan of packets.</li> </ul> Moreover, the time series created by re-aggregation contains following time series metrics instead of n_dest_ip, n_dest_asn, and n_dest_port: <ul> <li>sum_n_dest_ip: Sum of numbers of unique destination IP addresses.</li> <li>avg_n_dest_ip: The average number of unique destination IP addresses.</li> <li>std_n_dest_ip: Standard deviation of numbers of unique destination IP addresses.</li> <li>sum_n_dest_asn: Sum of numbers of unique destination ASNs.</li> <li>avg_n_dest_asn: The average number of unique destination ASNs.</li> <li>std_n_dest_asn: Standard deviation of numbers of unique destination ASNs)</li> <li>sum_n_dest_port: Sum of numbers of unique destination transport layer ports.</li> <li>avg_n_dest_port:  The average number of unique destination transport layer ports.</li> <li>std_n_dest_port: Standard deviation of numbers of unique destination transport layer ports.</li> </ul>   Moreover, files  identifiers.csv in each dataset type contain IDs of time series that are present in the dataset. Furthermore, the ids_relationship.csv file contains a relationship between IP addresses, Institutions, and institution subnets. The weekends_and_holidays.csv contains information about the non-working days in the Czech Republic.

opencc-by-4.0Jan 2019View details →

Tri-hourly dataset of wind and wave anomalies of the GFS and WAVEWATCH III models in the entire tropic region (TROPWA).

This dataset contains the anomalies of the total height and peak period of the waves and of the zonal and meridional components of the wind at 10 m above the sea surface obtained from the outputs of the GFS and WAVEWATCH III coupled global models in the entire tropical region (180°W to 178.75°E longitude/30°S to 30°N latitude), with a spatial resolution of 1.25°x1°. It is made up of two files in NetCDF format, where the data for wind anomalies (speed module and its zonal and meridional components) and wave anomalies (total wave height and peak period) are contained separately. This dataset was created in order to study all types of tropical storms, cold fronts and other physical processes that influence significant changes in wave parameters, as well as for the design and construction of coastal engineering works. The authors thanks to Tropical Atlantic Interdisciplinary Laboratory on physical, biogeochemical, ecological and human dynamics (IJL TAPIOCA).

opencc-by-4.0Mar 2022View details →

Hazards&Robots: A Dataset for Visual Anomaly Detection in Robotics

This is the final version of our dataset; we further expand the Corridor scenario. This new version of Corridor includes 20 anomalies and the total frames are 324,408. In this version, we release feature embeddings extracted using a CLIP ViT-B/32 model. This dataset is part of a Data in Brief paper submission. For more information check https://github.com/idsia-robotics/hazard-detection

opencc-by-4.0Feb 2023View details →

Controlled Anomalies Time Series (CATS) Dataset

The Controlled Anomalies Time Series (CATS) Dataset consists of commands, external stimuli, and telemetry readings of a simulated complex dynamical system with 200 injected anomalies. The CATS Dataset exhibits a set of desirable properties that make it very suitable for benchmarking Anomaly Detection Algorithms in Multivariate Time Series [1]: <ul> <li>Multivariate (17 variables) including sensors reading and control signals. It simulates the operational behaviour of an arbitrary complex system including: <ul> <li>4 Deliberate Actuations / Control Commands sent by a simulated operator / controller, for instance, commands of an operator to turn ON/OFF some equipment.</li> <li>3 Environmental Stimuli / External Forces acting on the system and affecting its behaviour, for instance, the wind affecting the orientation of a large ground antenna.</li> <li>10 Telemetry Readings representing the observable states of the complex system by means of sensors, for instance, a position, a temperature, a pressure, a voltage, current, humidity, velocity, acceleration, etc.</li> </ul> </li> <li>5 million timestamps. Sensors readings are at 1Hz sampling frequency. <ul> <li>1 million nominal observations (the first 1 million datapoints). This is suitable to start learning the "normal" behaviour.</li> <li>4 million observations that include both nominal and anomalous segments. This is suitable to evaluate both semi-supervised approaches (novelty detection) as well as unsupervised approaches (outlier detection).</li> </ul> </li> <li>200 anomalous segments. One anomalous segment may contain several successive anomalous observations / timestamps. Only the last 4 million observations contain anomalous segments.</li> <li>Different types of anomalies to understand what anomaly types can be detected by different approaches. The categories are available in the dataset and in the metadata.</li> <li>Fine control over ground truth. As this is a simulated system with deliberate anomaly injection, the start and end time of the anomalous behaviour is known very precisely. In contrast to real world datasets, there is no risk that the ground truth contains mislabelled segments which is often the case for real data.</li> <li>Suitable for root cause analysis. In addition to the anomaly category, the time series channel in which the anomaly first developed itself is recorded and made available as part of the metadata. This can be useful to evaluate the performance of algorithm to trace back anomalies to the right root cause channel.</li> <li>Affected channels. In addition to the knowledge of the root cause channel in which the anomaly first developed itself, we provide information of channels possibly affected by the anomaly. This can also be useful to evaluate the explainability of anomaly detection systems which may point out to the anomalous channels (root cause and affected).</li> <li>Obvious anomalies. The simulated anomalies have been designed to be "easy" to be detected for human eyes (i.e., there are very large spikes or oscillations), hence also detectable for most algorithms. It makes this synthetic dataset useful for screening tasks (i.e., to eliminate algorithms that are not capable to detect those obvious anomalies). However, during our initial experiments, the dataset turned out to be challenging enough even for state-of-the-art anomaly detection approaches, making it suitable also for regular benchmark studies.</li> <li>Context provided. Some variables can only be considered anomalous in relation to other behaviours. A typical example consists of a light and switch pair. The light being either on or off is nominal, the same goes for the switch, but having the switch on and the light off shall be considered anomalous. In the CATS dataset, users can choose (or not) to use the available context, and external stimuli, to test the usefulness of the context for detecting anomalies in this simulation.</li> <li>Pure signal ideal for robustness-to-noise analysis. The simulated signals are provided without noise: while this may seem unrealistic at first, it is an advantage since users of the dataset can decide to add on top of the provided series any type of noise and choose an amplitude. This makes it well suited to test how sensitive and robust detection algorithms are against various levels of noise.</li> <li>No missing data. You can drop whatever data you want to assess the impact of missing values on your detector with respect to a clean baseline.</li> </ul> Change Log Version 2 <ul> <li>Metadata: we include a metadata.csv with information about: <ul> <li>Anomaly categories</li> <li>Root cause channel (signal in which the anomaly is first visible)</li> <li>Affected channel (signal in which the anomaly might propagate) through coupled system dynamics</li> </ul> </li> <li>Removal of anomaly overlaps: version 1 contained anomalies which overlapped with each other resulting in only 190 distinct anomalous segments. Now, there are no more anomaly overlaps.</li> <li>Two data files: CSV and parquet for convenience.</li> </ul> [1] Example Benchmark of Anomaly Detection in Time Series: “Sebastian Schmidl, Phillip Wenig, and Thorsten Papenbrock. Anomaly Detection in Time Series: A Comprehensive Evaluation. PVLDB, 15(9): 1779 - 1797, 2022. doi:10.14778/3538598.3538602” About Solenix Solenix is an international company providing software engineering, consulting services and software products for the space market. Solenix is a dynamic company that brings innovative technologies and concepts to the aerospace market, keeping up to date with technical advancements and actively promoting spin-in and spin-out technology activities. We combine modern solutions which complement conventional practices. We aspire to achieve maximum customer satisfaction by fostering collaboration, constructivism, and flexibility.

opencc-by-4.0Dec 2023View details →

Dataset of Hyperspectral Melt Pool Signatures and Thermal Anomalies in DED of 316L steel

Description of the dataset The dataset includes in-situ melt pool signatures (hyperspectral NIR images) during the Directed Energy Deposition of 316L steel for several classes of thermal anomalies. Thermal anomalies were created during the process by varying the scanning speed. Samples were printed on the MiCLAD machine at the Vrije Universiteit Brussel (Belgium). Process and acquisition parameters: <ul> <li>Hardware: <ul> <li>Machine: MiCLAD (Vrije Universiteit Brussel)</li> <li>Laser: High-YAG BIMO 1064nm, 2.55mm fibre, flat-top</li> <li>Nozzle: Harald-Dickler HighNo 4.0</li> </ul> </li> <li>Process parameters: <ul> <li>Laser power: 600 W</li> <li>Scanning speed: 500/700/900/1100/1300 mm/min</li> <li>Powder: 316L 45-105 um</li> <li>Powder flow rate: 3.5 g/m</li> <li>Layer thickness: 0.2 mm</li> </ul> </li> <li>Image characteristics: <ul> <li>Camera: 3D-One Avior AX-M25NIR</li> <li>Hyperspectral filter layout: 5x5 (25 wavelengths per image)</li> </ul> </li> </ul> Description of the files <ul> <li>CSV dataset (hyperspectral_nir_meltpool_dataset.csv): List of filename, sample, label, time (ms), X and Z position (mm) and local scanning speed (mm/min) for all melt pool signatures. Thermal anomalies are labelled accordingly: <ul> <li>0 : baseline</li> <li>1 : edge</li> <li>2 : underheat</li> <li>3 : strong underheat</li> <li>4 : overheat</li> <li>5 : strong overheat</li> </ul> </li> <li>Melt pool signatures (hyperspectral_nir_meltpool_images_*.zip): Raw .tif thermal images of the melt pool taken in-situ. The raw images must debayered to retrieve the spectral information, see the Python function and example script. </li> <li>Python debayer function (debayer.py): Debayering function to retrieve the spectral information from the raw images. </li> </ul>

opencc-by-4.0Oct 2021View details →

A Global Dataset of Standardized Moisture Anomaly Index Incorporating Snow Dynamics (SZIsnow) from 1948 to 2010

The SZIsnow dataset was calculated based on systematic physical fields from the Global Land Data Assimilation System Version 2 (GLDAS-2) with the Noah land surface model. This SZIsnow dataset considers different physical water-energy processes, especially snow processes. The evaluation shows the dataset is capable of investigating different types of droughts across different timescales. The assessment also indicates that the dataset has an adequate performance to capture droughts across different spatial scales. The consideration of snow processes improved the capability of SZIsnow, and the improvement is evident over snow-covered areas (e.g., Arctic region) and high-altitude areas (e.g., Tibet Plateau). Moreover, the analysis also implies that SZIsnow dataset is able to well capture large-scale drought events across the world. This drought dataset has high application potential for monitoring, assessing, and supplying information on drought, and also can serve as a valuable resource for drought studies.

opencc-by-nc-4.0Jan 2022View details →

Dataset for: An experimental comparison of anomaly detection methods for collaborative robot manipulators

The dataset contains data recordings from a UR5e robot during normal and anomalous operation and is recorded to support the authors Master thesis project and the associated Paper: "An Experimental Comparison of Anomaly Detection Methods for Collaborative Robot Manipulators" (inProceeding). An in-depth description of the dataset can be found in the pdf uploaded with the dataset and an example of a data loader is also provided.