Find research datasets worth reusing

opencc-by-4.0Dec 2020View details →

Anomaly detection in the Zwicky Transient Facility DR3

The feature data set extracted from <a href="https://www.ztf.caltech.edu/page/dr3">ZTF DR3</a> light curves. It was used in <a href="https://arxiv.org/abs/2012.01419">Malanchev et al. 2020</a> to detect anomalous astrophysical sources in ZTF data.  "feature_XXX.dat" files contain object-ordered light curve feature data, every object is built on 42 feature values, which are encoded as little endian single precision IEEE-754 float (32bit float) numbers. Feature code-names are the same for all three data sets and are listed in plain text files "feature_XXX.name", one code-name per line. "oid_XXX.dat" files contain ZTF DR object identifiers encoded as little endian 64-bit unsigned integer numbers. "oid_XXX.dat" and "feature_XXX.dat" have same object order, for example the first 8 bytes of "oid_m31.dat" files contain the OID of the ZTF DR3 light curve which feature are presented in the first 168 bytes of "feature_m31.dat" file. "m31", "deep" and "disk" denote different ZTF fields and contain 57 546, 406 611, 1 790 565 objects. Note that observations between 58194 ≤ MJD ≤ 58483 are used, see <a href="https://doi.org/10.1093/mnras/stab316">the paper</a> for field and features details. The sample Python code to access the data as Numpy arrays: <pre><code class="language-python">import numpy as np oid = np.memmap('oid_m31.dat', mode='r', dtype=np.uint64) with open('feature_m31.name') as f: names = f.read().split() dtype = [(name, np.float32) for name in names] feature = np.memmap('feature_m31.dat', mode='r', dtype=dtype, shape=oid.shape) idx = np.argmax(feature['amplitude']) print('Object {} has maximum amplitude {:.3f}'.format(oid[idx], feature['amplitude'][idx]))</code></pre>

opencc-by-4.0Nov 2024View details →

Dataset for the paper "Anomaly Detection in Large-Scale Cloud Systems: An Industry Case and Dataset"

We present a large-scale anomaly detection dataset collected from IBM Cloud's Console over approximately 4.5 months. This high-dimensional dataset captures telemetry data from multiple data centers, specifically designed to aid researchers in developing and benchmarking anomaly detection methods in large-scale cloud environments. It contains 39,365 entries, each representing a 5-minute interval, with 117,448 features/attributes, as interval_start is used as the index. The dataset includes detailed information on request counts, HTTP response codes, and various aggregated statistics. The dataset also includes labeled anomaly events identified through IBM's internal monitoring tools, providing a comprehensive resource for real-world anomaly detection research and evaluation. File Descriptions <ul> <li><code>location_downtime.csv</code> - Details planned and unplanned downtimes for IBM Cloud data centers, including start and end times in ISO 8601 format.</li> <li><code>unpivoted_data.parquet</code> - Contains raw telemetry data with 413 million+ rows, covering details like location, HTTP status codes, request types, and aggregated statistics (min, max, median response times).</li> <li><code>anomaly_windows.csv</code> - Ground truth for anomalies, listing start and end times of recorded anomalies, categorized by source (Issue Tracker, Instant Messenger, Test Log).</li> <li><code>pivoted_data_all.parquet</code> - Pivoted version of the telemetry dataset with 39,365 rows and 117,449 columns, including aggregated statistics across multiple metrics and intervals.</li> <li><code>demo/demo.[ipynb|html]</code>: This demo file provides examples of how to access data in the Parquet files, available in Jupyter Notebook (<code>.ipynb</code>) and HTML (<code>.html</code>) formats, respectively.</li> </ul> Further details of the dataset can be found in Appendix B: Dataset Characteristics of the <a href="https://arxiv.org/abs/2411.09047">paper</a> titled "Anomaly Detection in Large-Scale Cloud Systems: An Industry Case and Dataset." Sample code for training anomaly detectors using this data is provided in <a href="https://doi.org/10.5281/zenodo.14598119" target="_blank" rel="noopener">this package</a>.   When using the dataset, please cite it as follows: <pre><code>@misc{islam2024anomaly,</code> <code>      title={Anomaly Detection in Large-Scale Cloud Systems: An Industry Case and Dataset}, </code> <code>      author={Mohammad Saiful Islam and Mohamed Sami Rakha and William Pourmajidi and Janakan Sivaloganathan and John Steinbacher and Andriy Miranskyy},</code> <code>      year={2024},</code> <code>      eprint={2411.09047},</code> <code>     archivePrefix={arXiv},</code> <code>     url={https://arxiv.org/abs/2411.09047}</code> <code>}</code></pre>

opencc-by-4.0Jun 2024View details →

OPSSAT-AD - anomaly detection dataset for satellite telemetry

This is the AI-ready benchmark dataset (OPSSAT-AD) containing the telemetry data acquired on board OPS-SAT---a CubeSat mission that has been operated by the European Space Agency. It is accompanied by the paper with baseline results obtained using 30 supervised and unsupervised classic and deep machine learning algorithms for anomaly detection. They were trained and validated using the training-test dataset split introduced in this work, and we present a suggested set of quality metrics that should always be calculated to confront the new algorithms for anomaly detection while exploiting OPSSAT-AD. We believe that this work may become an important step toward building a fair, reproducible, and objective validation procedure that can be used to quantify the capabilities of the emerging anomaly detection techniques in an unbiased and fully transparent way. The included files are: <ul> <li><code>segments.csv</code> with the acquired telemetry signals from ESA OPS-SAT aircraft,</li> <li><code>dataset.csv</code> with the extracted, synthetic features are computed for each manually split and labeled telemetry segment.</li> <li>code files for data processing and example modeliing (<code>dataset_generator.ipynb</code> for data processing, <code>modeling_examples.ipynb</code> with simple examples,  <code>requirements.txt</code>- with details on Python configuration, and the <code>LICENSE</code> file)</li> </ul>   Please have a look at our two papers commenting on this dataset: <ul> <li>The benchmark paper with results of 30 supervised and unsupervised anomaly detection models for this collection: Ruszczak, B., Kotowski. K., Nalepa, J., Evans, D.: The OPS-SAT benchmark for detecting anomalies in satellite telemetry, 2024, <a href="https://arxiv.org/abs/2407.04730" target="_blank" rel="noopener">preprint arxiv: 2407.04730</a>,</li> <li>the conference paper in which we presented some preliminary results for this dataset: Ruszczak, B., Kotowski. K., Andrzejewski, J., et al.: (2023). Machine Learning Detects Anomalies in OPS-SAT Telemetry. Computational Science – ICCS 2023. LNCS, vol 14073. Springer, Cham, <a href="https://doi.org/10.1007/978-3-031-35995-8_21">DOI:10.1007/978-3-031-35995-8_21</a>.</li> </ul>

opencc-by-4.0Dec 2021View details →

Packaging Industry Anomaly DEtection (PIADE) Dataset

PIADE dataset contains data from five industrial packaging machines: <ul> <li>Machine s_1: from 2020-01-01 14:00:00 to 2021-12-31 13:00:00</li> <li>Machine s_2: from 2020-06-17 08:00:00 to 2021-12-31 07:00:00</li> <li>Machine s_3: from 2020-10-07 12:00:00 to 2022-01-01 23:00:00</li> <li>Machine s_4: from 2020-01-01 01:00:00 to 2022-01-01 23:00:00</li> <li>Machine s_5: from 2020-01-20 08:00:00 to 2022-01-01 12:00:00</li> </ul> ## Raw Data Each row represents a production interval, with the following schema: <ul> <li>interval_start: start of the production interval    </li> <li>equipment_ID: equipment identifier    </li> <li>alarm: alarm code of the active stop reason, if it occurred     </li> <li>type: idle, production, downtime, performance_loss or scheduled_downtime    </li> <li>start: start of the production interval    </li> <li>end: end of the production interval    </li> <li>elapsed: duration of the production interval    </li> <li>pi: input packages    </li> <li>po: output packages    </li> <li>speed: speed (packages per hour)</li> </ul> There are 133 different types of alerts, and 429394 rows.   ## Sequences (1h) data For each piece of equipment, we define sequences of length = 1 hour and we aggregate raw interval data as follows: <ul> <li>'equipment_ID': machine identifier</li> <li>'#changes': changes in machine state</li> <li>'%downtime': time spent in 'downtime' state</li> <li>'%idle': time spent in 'idle' state</li> <li>'%performance_loss': time spent in 'performance loss' state</li> <li>'%production': time spent in production</li> <li>'%scheduled_downtime': time spent in scheduled downtime</li> <li>'count_sum': sum of all alarm occurrences</li> <li>'A_<XXX>': counter of alarm <XXX> occurrences</li> <li>'<state1>/<state2>': number of transitions from <state1> to <state2></li> </ul>

opencc-by-4.0Jun 2019View details →

Data set for anomaly detection on a HPC system

This data set contains the data collected on the DAVIDE HPC system (CINECA & E4 & University of Bologna, Bologna, Italy) in the period March-May 2018. The data set has been used to train a autoencoder-based model to automatically detect anomalies in a semi-supervised fashion, on a real HPC system. This work is described in: 1) "Anomaly Detection using Autoencoders in High Performance Computing Systems", <a href="https://arxiv.org/search/cs?searchtype=author&query=Borghesi%2C+A">Andrea Borghesi</a>, <a href="https://arxiv.org/search/cs?searchtype=author&query=Bartolini%2C+A">Andrea Bartolini</a>, <a href="https://arxiv.org/search/cs?searchtype=author&query=Lombardi%2C+M">Michele Lombardi</a>, <a href="https://arxiv.org/search/cs?searchtype=author&query=Milano%2C+M">Michela Milano</a>, <a href="https://arxiv.org/search/cs?searchtype=author&query=Benini%2C+L">Luca Benini,</a> IAAI19 (proceedings in process) -- https://arxiv.org/abs/1902.08447 2) "Online Anomaly Detection in HPC Systems", <a href="https://arxiv.org/search/cs?searchtype=author&query=Borghesi%2C+A">Andrea Borghesi</a>, <a href="https://arxiv.org/search/cs?searchtype=author&query=Libri%2C+A">Antonio Libri</a>, <a href="https://arxiv.org/search/cs?searchtype=author&query=Benini%2C+L">Luca Benini</a>, <a href="https://arxiv.org/search/cs?searchtype=author&query=Bartolini%2C+A">Andrea Bartolini, </a>AICAS19 (proceedings in process) -- https://arxiv.org/abs/1811.05269 See the git repository for usage examples & details --> https://github.com/AndreaBorghesi/anomaly_detection_HPC

opencc-by-4.0Aug 2024View details →

Three Annotated Anomaly Detection Datasets for Line-Scan Algorithms

<h1>Summary</h1> This dataset contains two hyperspectral and one multispectral anomaly detection images, and their corresponding binary pixel masks. They were initially used for real-time anomaly detection in line-scanning, but they can be used for any anomaly detection task. They are in .npy file format (will add tiff or geotiff variants in the future), with the image datasets being in the order of (height, width, channels). The SNP dataset was collected using sentinelhub, and the Synthetic dataset was collected from AVIRIS. The Python code used to analyse these datasets can be found at: https://github.com/WiseGamgee/HyperAD <h1>How to Get Started</h1> All that is needed to load these datasets is Python (preferably 3.8+) and the NumPy package. Example code for loading the Beach Dataset if you put it in a folder called "data" with the python script is: <pre><code>import numpy as np # Load image file hsi_array = np.load("data/beach_hsi.npy") n_pixels, n_lines, n_bands = hsi_array.shape print(f"This dataset has {n_pixels} pixels, {n_lines} lines, and {n_bands}.") # Load image mask mask_array = np.load("data/beach_mask.npy") m_pixels, m_lines = mask_array.shape print(f"The corresponding anomaly mask is {m_pixels} pixels by {m_lines} lines.")</code></pre> <h1>Citing the Datasets</h1> If you use any of these datasets, please cite the following paper: <pre><code>@article{garske2024erx,</code> <code>  title={ERX - a Fast Real-Time Anomaly Detection Algorithm for Hyperspectral Line-Scanning},</code> <code>  author={Garske, Samuel and Evans, Bradley and Artlett, Christopher and Wong, KC},</code> <code>  journal={arXiv preprint arXiv:2408.14947},</code> <code>  year={2024},</code> <code>}</code></pre> <div> <pre>If you use the beach dataset please cite the following paper as well (original source):</pre> </div> <pre><code>@article{mao2022openhsi, title={OpenHSI: A complete open-source hyperspectral imaging solution for everyone}, author={Mao, Yiwei and Betters, Christopher H and Evans, Bradley and Artlett, Christopher P and Leon-Saval, Sergio G and Garske, Samuel and Cairns, Iver H and Cocks, Terry and Winter, Robert and Dell, Timothy}, journal={Remote Sensing}, volume={14}, number={9}, pages={2244}, year={2022}, publisher={MDPI} }</code></pre>

opencc-by-4.0Jan 2023View details →

Bibliographic Data from the Digital Twin Anomaly Detection Decision-Making for Bridge Management Systematic Review

This database contains all the bibliographic information about the 8673 records found after applying the Search Strategy used for the Digital Twin Anomaly Detection Decision-Making for Bridge Management Systematic Review. Such strategy consisted on using seven initial keywords and similar terms of interest (namely: bridge and bridges, etc.):  <ul> <li>Bridge.</li> <li>Digital twin.</li> <li>Bridge information modelling.</li> <li>Finite elements.</li> <li>Bridge health monitoring.</li> <li>Anomaly detection algorithm.</li> <li>Cultural heritage.</li> </ul> Six initial queries were done combining the first keyword with the rest of them: <ul> <li>bridge* AND "digital twin*"</li> <li>bridge* AND (BrIM OR "bridge information model*")</li> <li>bridge* AND (FEM OR FEA OR "finite element method*" OR "finite element analy*")</li> <li>bridge* AND ("bridge health monitoring" OR "structural health monitoring")</li> <li>bridge* AND (ADA OR "anomaly detection algorithm*")</li> <li>bridge* AND ("cultural heritage" OR "monument* bridge*" OR "old bridge*" OR "ancient bridge*" OR "historic* bridge*")</li> </ul> As a first screening step, the combination of these 6 initial searches was done to obtain relevant works containing at least three of the main keywords of interest: <ul> <li>#1 AND #2</li> <li>#1 AND #3</li> <li>#1 AND #4</li> <li>#1 AND #5</li> <li>#1 AND #6</li> <li>#2 AND #3</li> <li>#2 AND #4</li> <li>#2 AND #5</li> <li>#2 AND #6</li> <li>#3 AND #4</li> <li>#3 AND #5</li> <li>#3 AND #6</li> <li>#4 AND #5</li> <li>#4 AND #6</li> <li>#5 AND #6</li> </ul> All records found in Scopus where downloaded both in .ris and .csv format and are included in this database. The search was conducted on 10/12/2022. Note: Searches 10, 14, 17 and 21 did not return any records.

opencc-by-nc-sa-4.0Feb 2024View details →

The Automotive Visual Inspection Dataset (AutoVI): A Genuine Industrial Production Dataset for Unsupervised Anomaly Detection

See the official website: <a href="https://autovi.utc.fr">https://autovi.utc.fr</a> Modern industrial production lines must be set up with robust defect inspection modules that are able to withstand high product variability. This means that in a context of industrial production, new defects that are not yet known may appear, and must therefore be identified. On industrial production lines, the typology of potential defects is vast (texture, part failure, logical defects, etc.). Inspection systems must therefore be able to detect non-listed defects, i.e. not-yet-observed defects upon the development of the inspection system. To solve this problem, research and development of unsupervised AI algorithms on real-world data is required. Renault Group and the Université de technologie de Compiègne (Roberval and Heudiasyc Laboratories) have jointly developed the Automotive Visual Inspection Dataset (AutoVI), the purpose of which is to be used as a scientific benchmark to compare and develop advanced unsupervised anomaly detection algorithms under real production conditions. The images were acquired on Renault Group's automotive production lines, in a genuine industrial production line environment, with variations in brightness and lighting on constantly moving components. This dataset is representative of actual data acquisition conditions on automotive production lines. The dataset contains 3950 images, split into 1530 training images and 2420 testing images. The evaluation code can be found at <a href="https://github.com/phcarval/autovi_evaluation_code">https://github.com/phcarval/autovi_evaluation_code</a>. Disclaimer All defects shown were intentionally created on Renault Group's production lines for the purpose of producing this dataset. The images were examined and labeled by Renault Group experts, and all defects were corrected after shooting. License Copyright © 2023-2024 Renault Group This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. To view a copy of the license, visit <a href="https://creativecommons.org/licenses/by-nc-sa/4.0/">https://creativecommons.org/licenses/by-nc-sa/4.0/</a>. For using the data in a way that falls under the commercial use clause of the license, please contact us. Attribution Please use the following for citing the dataset in scientific work: Carvalho, P., Lafou, M., Durupt, A., Leblanc, A., & Grandvalet, Y. (2024). The Automotive Visual Inspection Dataset (AutoVI): A Genuine Industrial Production Dataset for Unsupervised Anomaly Detection [Dataset]. <a href="https://doi.org/10.5281/zenodo.10459003">https://doi.org/10.5281/zenodo.10459003</a> Contact If you have any questions or remarks about this dataset, please contact us at philippe.carvalho@utc.fr, meriem.lafou@renault.com, alexandre.durupt@utc.fr, antoine.leblanc@renault.com, yves.grandvalet@utc.fr. Changelog <ul> <li>v1.0.0 <ul> <li>Cropped engine_wiring, pipe_clip and pipe_staple images</li> <li>Reduced tank_screw, underbody_pipes and underbody_screw image sizes</li> </ul> </li> <li>v0.1.1 <ul> <li>Added ground truth segmentation maps</li> <li>Fixed categorization of some images</li> <li>Added new defect categories</li> <li>Removed tube_fastening and kitting_cart</li> <li>Removed duplicates in pipe_clip</li> </ul> </li> </ul>

opencc-by-4.0Sep 2024View details →

CESNET-TimeSeries24: Time Series Dataset for Network Traffic Anomaly Detection and Forecasting

<h2>CESNET-TimeSeries24: The dataset for network traffic forecasting and anomaly detection</h2> The dataset called CESNET-TimeSeries24 was collected by long-term monitoring of selected statistical metrics for 40 weeks for each IP address on the ISP network CESNET3 (Czech Education and Science Network). The dataset encompasses network traffic from more than 275,000 active IP addresses, assigned to a wide variety of devices, including office computers, NATs, servers, WiFi routers, honeypots, and video-game consoles found in dormitories. Moreover, the dataset is also rich in network anomaly types since it contains all types of anomalies, ensuring a comprehensive evaluation of anomaly detection methods. Last but not least, the CESNET-TimeSeries24 dataset provides traffic time series on institutional and IP subnet levels to cover all possible anomaly detection or forecasting scopes. Overall, the time series dataset was created from the 66 billion IP flows that contain 4 trillion packets that carry approximately 3.7 petabytes of data. The CESNET-TimeSeries24 dataset is a complex real-world dataset that will finally bring insights into the evaluation of forecasting models in real-world environments. Please cite the usage of our dataset as: <blockquote> Koumar, J., Hynek, K., Čejka, T. et al. CESNET-TimeSeries24: Time Series Dataset for Network Traffic Anomaly Detection and Forecasting. Sci Data 12, 338 (2025). https://doi.org/10.1038/s41597-025-04603-x @Article{cesnettimeseries24,     author={Koumar, Josef and Hynek, Karel and {\v{C}}ejka, Tom{\'a}{\v{s}} and {\v{S}}i{\v{s}}ka, Pavel},     title={CESNET-TimeSeries24: Time Series Dataset for Network Traffic Anomaly Detection and Forecasting},     journal={Scientific Data},     year={2025},     month={Feb},     day={26},     volume={12},     number={1},     pages={338},     issn={2052-4463},     doi={10.1038/s41597-025-04603-x},     url={https://doi.org/10.1038/s41597-025-04603-x} } </blockquote>   <h3>Time series</h3> We create evenly spaced time series for each IP address by aggregating IP flow records into time series datapoints. The created datapoints represent the behavior of IP addresses within a defined time window of 10 minutes. The vector of time-series metrics v_{ip, i} describes the IP address ip in the i-th time window. Thus, IP flows for vector v_{ip, i} are captured in time windows starting at t_i and ending at t_{i+1}. The time series are built from these datapoints.   Datapoints created by the aggregation of IP flows contain the following time-series metrics: <ul> <li>Simple volumetric metrics: the number of IP flows, the number of packets, and the transmitted data size (i.e. number of bytes)</li> <li>Unique volumetric metrics: the number of unique destination IP addresses, the number of unique destination Autonomous System Numbers (ASNs), and the number of unique destination transport layer ports. The aggregation of \textit{Unique volumetric metrics} is memory intensive since all unique values must be stored in an array. We used a server with 41 GB of RAM, which was enough for 10-minute aggregation on the ISP network.   </li> <li>Ratios metrics: the ratio of UDP/TCP packets, the ratio of UDP/TCP transmitted data size, the direction ratio of packets, and the direction ratio of transmitted data size</li> <li>Average metrics: the average flow duration, and the average Time To Live (TTL)</li> </ul>   Multiple time aggregation:  The original datapoints in the dataset are aggregated by 10 minutes of network traffic. The size of the aggregation interval influences anomaly detection procedures, mainly the training speed of the detection model. However, the 10-minute intervals can be too short for longitudinal anomaly detection methods. Therefore, we added two more aggregation intervals to the datasets--1 hour and 1 day. Time series of institutions:  We identify 283 institutions inside the CESNET3 network. These time series aggregated per each institution ID provide a view of the institution's data.  Time series of institutional subnets: We identify 548 institution subnets inside the CESNET3 network. These time series aggregated per each institution ID provide a view of the institution subnet's data.    <h3>Data Records</h3> The file hierarchy is described below: <blockquote> cesnet-timeseries24/      |- institution_subnets/      |     |- agg_10_minutes/<id_institution>.csv      |     |- agg_1_hour/<id_institution>.csv      |     |- agg_1_day/<id_institution>.csv      |     |- identifiers.csv      |- institutions/      |     |- agg_10_minutes/<id_institution_subnet>.csv      |     |- agg_1_hour/<id_institution_subnet>.csv      |     |- agg_1_day/<id_institution_subnet>.csv      |     |- identifiers.csv      |- ip_addresses_full/      |     |- agg_10_minutes/<id_ip_folder>/<id_ip>.csv      |     |- agg_1_hour/<id_ip_folder>/<id_ip>.csv      |     |- agg_1_day/<id_ip_folder>/<id_ip>.csv      |     |- identifiers.csv      |- ip_addresses_sample/      |      |- agg_10_minutes/<id_ip>.csv      |      |- agg_1_hour/<id_ip>.csv      |      |- agg_1_day/<id_ip>.csv      |      |- identifiers.csv      |- times/      |      |- times_10_minutes.csv      |      |- times_1_hour.csv      |      |- times_1_day.csv      |- ids_relationship.csv      |- weekends_and_holidays.csv </blockquote> The following list describes time series data fields in CSV files: <ul> <li>id_time:  Unique identifier for each aggregation interval within the time series, used to segment the dataset into specific time periods for analysis.</li> <li>n_flows: Total number of flows observed in the aggregation interval, indicating the volume of distinct sessions or connections for the IP address.</li> <li>n_packets: Total number of packets transmitted during the aggregation interval, reflecting the packet-level traffic volume for the IP address.</li> <li>n_bytes: Total number of bytes transmitted during the aggregation interval, representing the data volume for the IP address.</li> <li>n_dest_ip: Number of unique destination IP addresses contacted by the IP address during the aggregation interval, showing the diversity of endpoints reached.</li> <li>n_dest_asn: Number of unique destination Autonomous System Numbers (ASNs) contacted by the IP address during the aggregation interval, indicating the diversity of networks reached.</li> <li>n_dest_port: Number of unique destination transport layer ports contacted by the IP address during the aggregation interval, representing the variety of services accessed.</li> <li>tcp_udp_ratio_packets: Ratio of packets sent using TCP versus UDP by the IP address during the aggregation interval, providing insight into the transport protocol usage pattern. This metric belongs to the interval <0, 1> where 1 is when all packets are sent over TCP, and 0 is when all packets are sent over UDP.</li> <li>tcp_udp_ratio_bytes: Ratio of bytes sent using TCP versus UDP by the IP address during the aggregation interval, highlighting the data volume distribution between protocols. This metric belongs to the interval <0, 1>  with same rule as tcp_udp_ratio_packets.</li> <li>dir_ratio_packets: Ratio of packet directions (inbound versus outbound) for the IP address during the aggregation interval, indicating the balance of traffic flow directions. This metric belongs to the interval <0, 1>, where 1 is when all packets are sent in the outgoing direction from the monitored IP address, and 0 is when all packets are sent in the incoming direction to the monitored IP address.</li> <li>dir_ratio_bytes: Ratio of byte directions (inbound versus outbound) for the IP address during the aggregation interval, showing the data volume distribution in traffic flows. This metric belongs to the interval <0, 1> with the same rule as dir_ratio_packets.</li> <li>avg_duration: Average duration of IP flows for the IP address during the aggregation interval, measuring the typical session length.</li> <li>avg_ttl: Average Time To Live (TTL) of IP flows for the IP address during the aggregation interval, providing insight into the lifespan of packets.</li> </ul> Moreover, the time series created by re-aggregation contains following time series metrics instead of n_dest_ip, n_dest_asn, and n_dest_port: <ul> <li>sum_n_dest_ip: Sum of numbers of unique destination IP addresses.</li> <li>avg_n_dest_ip: The average number of unique destination IP addresses.</li> <li>std_n_dest_ip: Standard deviation of numbers of unique destination IP addresses.</li> <li>sum_n_dest_asn: Sum of numbers of unique destination ASNs.</li> <li>avg_n_dest_asn: The average number of unique destination ASNs.</li> <li>std_n_dest_asn: Standard deviation of numbers of unique destination ASNs)</li> <li>sum_n_dest_port: Sum of numbers of unique destination transport layer ports.</li> <li>avg_n_dest_port:  The average number of unique destination transport layer ports.</li> <li>std_n_dest_port: Standard deviation of numbers of unique destination transport layer ports.</li> </ul>   Moreover, files  identifiers.csv in each dataset type contain IDs of time series that are present in the dataset. Furthermore, the ids_relationship.csv file contains a relationship between IP addresses, Institutions, and institution subnets. The weekends_and_holidays.csv contains information about the non-working days in the Czech Republic.

opencc-by-4.0Feb 2023View details →

DeepAstroUDA: Semi-Supervised Universal Domain Adaptation for Cross-Survey Galaxy Morphology Classification and Anomaly Detection

We present the data used in "DeepAstroUDA: Semi-Supervised Universal Domain Adaptation for Cross-Survey Galaxy Morphology Classification and Anomaly Detection". It was also used in the conference paper presented in Machine Learning and the Physical Sciences workshop at NeurIPS 2022: "Semi-Supervised Domain Adaptation for Cross-Survey Galaxy Morphology Classification and Anomaly Detection". A plethora of AI methods, has already shown huge promise in increasing quality and speed of work with astronomical datasets, but high complexity of AI methods leads to extraction of dataset-specific non-robust features, which leads to models that cannot work on multiple datasets at the same time. We develop a Universal Domain Adaptation method DeepAstroUDA, capable of performing semi-supervised domain adaptation, that can be applied to datasets with different data distributions and class overlap. Extra classes can be present in any of the two datasets, and the method can even be used in the presence of unknown classes. We apply our model to three examples of galaxy morphology classification tasks of different complexities (3-class and 10-class problems), with anomaly detection i.e. in all our experiments we have one extra class in the unlabeled target dataset, which represents our anomaly class.   DATA: 1) DA across two different data releases of the same survey (LSST 1 and 10 years of observation): We use data from Ciprijanovic et al. 2022. which can also be found on Zenodoo: <a href="https://zenodo.org/record/5514180#.Y6SM7y-B2_w">https://zenodo.org/record/5514180</a> . Data contains three classes: spiral (0), elliptical (1) and merging galaxies (3, anomaly class). 2) DA across two surveys (SDSS and DeCALS): We create datasets using data and labels from the Galaxy Zoo project. Datasets contain 10 classes (9 known classes present in both SDSS and DeCALS data, and one unknown anomaly class present only in DeCALS data): disturbed (0), merging (1), round smooth (2), cigar shaped smooth (3), barred spiral (4), unbarred tight spiral (5), unbarred loose spiral (6), edge-on without bulge (7), edge-on with bulge (8), lenses (9, unknown anomaly class). SDSS (wide filed): datasets is split into two files  - sdss_1.h5, sdss_2.h5 DeCALS:  decals.zip 3) DA between wide and deep observing fields of the same survey (SDSS): We create datasets using data and labels from the Galaxy Zoo project. Datasets contain same 10 classes as in 2), with the final lens anomaly class being only present in the SDSS deep field. SDSS (wide filed): the same data as in 2) SDSS (Strip 82 deep field): sdss_stripe82.zip All SDSS and DECaLS files contain full datasets (train, validation and test). Exact split that we performed (0.6 : 0.2 : 0.2) can be done using the code that accompanies this publication: <a href="https://github.com/deepskies/DeepAstroUDA">https://github.com/deepskies/DeepAstroUDA</a> .

opencc-by-4.0Mar 2022View details →

Hazards&Robots: A Dataset for Visual Anomaly Detection in Robotics

This is the final version of our dataset; we further expand the Corridor scenario. This new version of Corridor includes 20 anomalies and the total frames are 324,408. In this version, we release feature embeddings extracted using a CLIP ViT-B/32 model. This dataset is part of a Data in Brief paper submission. For more information check https://github.com/idsia-robotics/hazard-detection

opencc-by-4.0Oct 2020View details →

Data for the MLCS 2020 paper "A Year of Automated Anomaly Detection in a Datacenter"

This contains the data used for the paper by Ahmed et. al in the MLCS 2020 paper "A Year of Automated Anomaly Detection in a Datacenter". Each of the four CSV files corresponds to one of the quarters discussed in the paper, and each has a metadata file containing information about the query that produced them. The CSV files contain the 'raw' log messages, and an eventID that identifies which pattern the log entry matched; the eventID is used to group together log messages of the same type. These logfiles were collected on the CloudLab facility (https://cloudlab.us/) from Jan 1 - Dec 30, 2019. The violated_unviolated_sessions_*.txt files each contain 20 randomly-selected sessions: half of the sessions were labeled by the invariant miner as being 'normal', and the other half 'anomalous'. CloudLab developers and system administrators were asked to label these sessions manually (and were not given the invariant miner's labels). The corresponding *_manual_labels.txt contain the labels that the administrators assigned, and in some cases additional correspondence with the administrators and information about which manual labels matched the invariant miner and which did not.

opencc-by-4.0Mar 2024View details →

Geodetic anomaly detection and analysis in the Campi Flegrei caldera (Italy) deformation pattern of the 2021-2023 escalating unrest phase

Data used within the manuscript: "First evidence of a geodetic anomaly in the Campi Flegrei caldera (Italy) ground deformation pattern revealed by DInSAR and GNSS measurements during the 2021-2023 escalating unrest phase"   Archive content: <ul> <li><code>DTSLOS_CNRIREA_20150325_20231021_FB9K</code>: Line of Sight displacement time series retrieved by applying the P-SBAS algorithm to Sentinel-1 data set acquired from ascending orbits (Track 44) over Campi Flegrei caldera in the 20150325 - 20231021 interval. Data format is according to the <a href="https://gitlab.com/epos-tcs-satdata/doc/-/blob/main/TCS_SATD_Product_Description.md#los-displacement-time-series-dtslos" target="_blank" rel="noopener noreferrer">EPOS specification</a>.</li> <li><code>DTSLOS_CNRIREA_20150324_20231020_UJBI</code>: Line of Sight displacement time series retrieved by applying the P-SBAS algorithm to Sentinel-1 data set acquired from descending orbits (Track 22) over Campi Flegrei caldera in the 20150324 - 20231020 interval. Data format is according to the <a href="https://gitlab.com/epos-tcs-satdata/doc/-/blob/main/TCS_SATD_Product_Description.md#los-displacement-time-series-dtslos" target="_blank" rel="noopener noreferrer">EPOS specification</a>.</li> <li><code>Campi_Flegrei_GNSS_Weekly_Timeseries</code>: Weekly displacement time series of Campi Flegrei caldera GNSS network from 2016 to 2023.</li> </ul>

opencc-by-4.0Nov 2024View details →

Real-bogus scores for active anomaly detection

Data description for <a href="https://arxiv.org/abs/2409.10256">Semenikhin et al., 2024</a> The dataset consists of the following files: feature_snad4_r_100.dat contains light curve feature data for objects, where each object is represented by 54 feature values. These values are encoded as little-endian single-precision IEEE-754 floating-point numbers (32-bit floats). Feature names are listed in the plain text file feature_snad4_r_100.name, with one name per line. sid_snad4_r_100.dat contains ZTF DR object identifiers, encoded as little-endian 64-bit unsigned integers. exp_feature_snad4_r_100.dat contains the same features as feature_snad4_r_100.dat, but with an additional column representing the real-bogus classifier prediction. Each object in this file corresponds to 55 features: the original 54 features plus 1 additional feature. Feature names for this file are provided in exp_feature_snad4_r_100.name. The files sid_snad4_r_100.dat, feature_snad4_r_100.dat, and exp_feature_snad4_r_100.dat share the same object order. Below is a sample Python script for accessing the data using NumPy: <code>import numpy as np</code> <code># Load object IDs</code> <code>oid     =  np.memmap('sid_snad4_r_100.dat', mode='c', dtype=np.uint64)</code> <code># Load features and reshape</code> <code>feature = np.memmap('feature_snad4_r_100.dat', mode='c', dtype=np.float32).reshape(oid.shape[0], -1)</code> <code># Print dataset information</code> <code>print(f'Number of objects: {len(oid)}')</code> <code>print(f'Features shape: {feature.shape}')</code>

opencc-by-nc-4.0Jan 2022View details →

Dataset for: An experimental comparison of anomaly detection methods for collaborative robot manipulators

The dataset contains data recordings from a UR5e robot during normal and anomalous operation and is recorded to support the authors Master thesis project and the associated Paper: "An Experimental Comparison of Anomaly Detection Methods for Collaborative Robot Manipulators" (inProceeding). An in-depth description of the dataset can be found in the pdf uploaded with the dataset and an example of a data loader is also provided.

opencc-by-4.0Feb 2022View details →

Dataset for Anomaly Detection in a Production Wireless Mesh Community Network

CSV dataset generated gathering data from a production wireless mesh community network. Data is gathered every 5 minutes during the interval 2021-04-13 00:00:00 to 2021-04-16 00:00:00. During the interval 2021-04-14 02:00:00 2021-04-14 17:50:00 (both included) there is the failure of a gateway in the mesh (nodeid 24).  Live mesh network monitoring link: <a href="http://dsg.ac.upc.edu/qmpsu">http://dsg.ac.upc.edu/qmpsu</a> The dataset consists of single gzip compressed CSV file. The first line of the file is a header describing the features. The first column is a GMT timestamp of the sample in the format as "2021-03-16 00:00:00".  The rest of the columns provide the comma-separated values of the features collected from each node in the corresponding capture. A suffix with the nodeid is added to each feature. For instance, the feature having the number of processes of node with nodeid 24 is named as "processes-24". In total, 63 different nodes showed up during the samples, each being assigned a different nodeid. Features are of two types: (i) absolute values, for instance, the CPU 1-minute load average, and (ii) counters that are monotonically increased, for instance the number of transmitted packets. We have converted counter-type kernel variables to rates, by dividing the difference between two consecutive samples, over the difference of the corresponding timestamps in seconds, as shown in the following pseudo-code:   feature.rate are columns computed from feature as     feature.rate <- (feature[2:n]-feature[1:(n-1)])/(epoch[2:n]-epoch[1:(n-1)])     feature.rate <- feature.rate[feature.rate >= 0] # discard samples where the counter is restarted    where n is the number of samples features - processes     number of processes - loadavg.m1     1 minute load average - softirq.rate     servicing softirqs - iowait.rate     waiting for I/O to complete - intr.rate       - system.rate     processes executing in kernel mode - idle.rate     twiddling thumbs - user.rate     normal processes executing in user mode - irq.rate     servicing interrupts - ctxt.rate     total number of context switches across all CPUs - nice.rate     niced processes executing in user mode - nr_slab_unreclaimable     The part of the Slab that can't be reclaimed under memory pressure - nr_anon_pages     anonymous memory pages - swap_cache     Memory that once was swapped out, is swapped back in but still also is in the swapfile - page_tables     Memory used to map between virtual and physical memory addresses - swap       - eth.txe.rate     tx errors over all ethernet interfaces - eth.rxe.rate     rx errors over all ethernet interfaces - eth.txb.rate     tx bytes over all ethernet interfaces - eth.rxb.rate     rx bytes over all ethernet interfaces - eth.txp.rate     tx packets over all ethernet interfaces - eth.rxp.rate     rx packets over all ethernet interfaces - wifi.txe.rate     tx errors over all wireless interfaces - wifi.rxe.rate     rx errors over all wireless interfaces - wifi.txb.rate     tx bytes over all wireless interfaces - wifi.rxb.rate     rx bytes over all wireless interfaces - wifi.txp.rate     tx packets over all wireless interfaces - wifi.rxp.rate     rx packets over all wireless interfaces - txb.rate        tx bytes over all ethernet and wifi interfaces - txp.rate        tx packets over all ethernet and wifi interfaces - rxb.rate        rx bytes over all ethernet and wifi interfaces - rxp.rate        rx packets over all ethernet and wifi interfaces - sum.xb.rate    tx+rx bytes over all ethernet and wifi interfaces - sum.xp.rate    tx+rx packets over all ethernet and wifi interfaces - diff.xb.rate      tx-rx bytes over all ethernet and wifi interfaces - diff.xp.rate      tx-rx packets over all ethernet and wifi interfaces

opencc-by-4.0Apr 2019View details →

R&D Dataset for LHC Olympics 2020 Anomaly Detection Challenge

This is the first R&D dataset for the LHC Olympics 2020 Anomaly Detection Challenge. It consists of 1M QCD dijet events and 100k W'->XY events, with X->qq and Y->qq. The W', X, and Y masses are 3.5 TeV, 500 GeV and 100 GeV respectively. The events are produced using Pythia8 and Delphes 3.4.1, with no pileup or MPI included. They are selected using a single fat-jet (R=1) trigger with pT threshold of 1.2 TeV.  The events are randomly shuffled together, but for the purposes of testing and development, we provide the user with a signal/background truth bit for each event. Obviously, the truth bit will not be included in the actual challenge. These events are stored as pandas dataframes saved to compressed h5 format. For each event, all Delphes reconstructed particles in the event are assumed to be massless and are recorded in detector coordinates (pT, eta, phi). More detailed information such as particle charge is not included. Events are zero padded to constant size arrays of 700 particles, with the truth bit appended at the end. The array format is therefore (Nevents=1.1M, 2101). For more information, including an example Jupyter notebook illustrating how to read and process the events, see the official LHC Olympics 2020 webpage. <a href="https://lhco2020.github.io/homepage/">https://lhco2020.github.io/homepage/</a> UPDATE May 18 2020 We have uploaded a second signal dataset for R&D, consisting of 100k W'->XY with X,Y->qqq (i.e. 3-prong substructure). Everything else about this signal dataset (particle masses, trigger, Pythia configuration, detector simulation) is the same as the previous one described above.  UPDATE November 23 2020 We now include high-level feature files for the background and 2-prong signal (events_anomalydetection_v2.features.h5) and for the 3-prong signal (events_anomalydetection_Z_XY_qqq.features.h5). To produce the features, we have clustered every event into R=1 jets using the anti-kT algorithm. The features (calculated using fastjet plugins) are the 3-momenta, invariant masses, and n-jettiness variables tau1, tau2 and tau3 for the highest pT jet (j1) and the second highest pT jet (j2): 'pxj1', 'pyj1', 'pzj1', 'mj1', 'tau1j1', 'tau2j1', 'tau3j1', 'pxj2', 'pyj2', 'pzj2', 'mj2', 'tau1j2', 'tau2j2', 'tau3j2' The rows (events) in each feature file should be ordered exactly the same as in their corresponding raw event file. For convenience, we have also included the label (1 for signal and 0 for background) as an additional column in the first feature file (events_anomalydetection_v2.features.h5). UPDATE February 11 2021 We have included the Delphes detector card and the Pythia8 command files used to produce the R&D datasets. UPDATE April 17 2022 It was brought to our attention that somehow the raw events file events_anomalydetection.h5 was never updated to v2, which had a lower generator-level pT threshold (PhaseSpace:pTHatMin = 500) for QCD events to minimize artificial trigger sculpting. This v2 is the version that the features file (events_anomalydetection_v2.features.h5) corresponds to, as well as the Pythia cmnd file (pythia_RnD_qcd.cmnd). Now the raw events file has been brought up to date as well.