Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
625
datasets available to search
ShareScore release 0.9.0
Dataset results
625 results for “Anomaly”
DeepAstroUDA: Semi-Supervised Universal Domain Adaptation for Cross-Survey Galaxy Morphology Classification and Anomaly Detection
<p>We present the data used in "DeepAstroUDA: Semi-Supervised Universal Domain Adaptation for Cross-Survey Galaxy Morphology Classification and Anomaly Detection". It was also used in the conference paper presented in Machine Learning and the Physical Sciences workshop at NeurIPS 2022: "Semi-Supervised Domain Adaptation for Cross-Survey Galaxy Morphology Classification and Anomaly Detection".</p> <p>A plethora of AI methods, has already shown huge promise in increasing quality and speed of work with astronomical datasets, but high complexity of AI methods leads to extraction of dataset-specific non-robust features, which leads to models that cannot work on multiple datasets at the same time. We develop a Universal Domain Adaptation method <em><strong>DeepAstroUDA</strong></em>, capable of performing <strong>semi-supervised domain adaptation, that can be applied to datasets with different data distributions and class overlap</strong>. Extra classes can be present in any of the two datasets, and the method can even be used in the presence of unknown classes. We apply our model to three examples of galaxy morphology classification tasks of different complexities (3-class and 10-class problems), with anomaly detection i.e. in all our experiments we have one extra class in the unlabeled target dataset, which represents our anomaly class.</p> <p> </p> <p><strong>DATA:</strong></p> <p><strong>1) DA across two different data releases of the same survey (LSST 1 and 10 years of observation):</strong> We use data from Ciprijanovic et al. 2022. which can also be found on Zenodoo: <a href="https://zenodo.org/record/5514180#.Y6SM7y-B2_w">https://zenodo.org/record/5514180</a> . Data contains three classes: spiral (0), elliptical (1) and merging galaxies (3, anomaly class).</p> <p><strong>2) DA across two surveys (SDSS and DeCALS): </strong>We create datasets using data and labels from the Galaxy Zoo project. Datasets contain 10 classes (9 known classes present in both SDSS and DeCALS data, and one unknown anomaly class present only in DeCALS data): disturbed (0), merging (1), round smooth (2), cigar shaped smooth (3), barred spiral (4), unbarred tight spiral (5), unbarred loose spiral (6), edge-on without bulge (7), edge-on with bulge (8), lenses (9, unknown anomaly class).</p> <p>SDSS (wide filed): datasets is split into two files - sdss_1.h5, sdss_2.h5</p> <p>DeCALS: decals.zip</p> <p><strong>3) DA between wide and deep observing fields of the same survey (SDSS):</strong> We create datasets using data and labels from the Galaxy Zoo project. Datasets contain same 10 classes as in 2), with the final lens anomaly class being only present in the SDSS deep field.</p> <p>SDSS (wide filed): the same data as in 2)</p> <p>SDSS (Strip 82 deep field): sdss_stripe82.zip</p> <p>All SDSS and DECaLS files contain full datasets (train, validation and test). Exact split that we performed (0.6 : 0.2 : 0.2) can be done using the code that accompanies this publication: <a href="https://github.com/deepskies/DeepAstroUDA">https://github.com/deepskies/DeepAstroUDA</a> .</p>
Hazards&Robots: A Dataset for Visual Anomaly Detection in Robotics
<p>This is the final version of our dataset; we further expand the Corridor scenario.</p> <p>This new version of Corridor includes 20 anomalies and the total frames are 324,408.</p> <p>In this version, we release feature embeddings extracted using a CLIP ViT-B/32 model.</p> <p>This dataset is part of a Data in Brief paper submission.</p> <p>For more information check https://github.com/idsia-robotics/hazard-detection</p> <p> </p>
Global Urban Precipitation Anomalies
<p>This research reports the global urban precipitation anomalies for over one thousand cities worldwide. We provide the shapefiles of one urban domain and three rural domains (of different distances from the urban edges) for each city. The precipitation data include the mean daily precipitation, extreme precipitation magnitude, and extreme precipitation frequency in the urban and rural domains between 2001 and 2019 based on the IMERG precipitation dataset. Besides, data about mean elevation, wind, land surface temperature, aerosol optical thickness, and urbanization are also provided.</p>
Controlled Anomalies Time Series (CATS) Dataset
<p>The Controlled Anomalies Time Series (CATS) Dataset consists of commands, external stimuli, and telemetry readings of a simulated complex dynamical system with 200 injected anomalies.</p> <p>The CATS Dataset exhibits a set of desirable properties that make it very suitable for benchmarking<strong> Anomaly Detection Algorithms in Multivariate Time Series </strong>[1]:</p> <ul> <li><strong>Multivariate (17 variables) </strong>including sensors reading and control signals. It simulates the operational behaviour of an arbitrary complex system including: <ul> <li><strong>4 Deliberate Actuations / Control Commands sent by a simulated operator / controller</strong>, for instance, commands of an operator to turn ON/OFF some equipment.</li> <li><strong>3 Environmental Stimuli / External Forces</strong> acting on the system and affecting its behaviour, for instance, the wind affecting the orientation of a large ground antenna.</li> <li><strong>10 Telemetry Readings</strong> representing the observable states of the complex system by means of sensors, for instance, a position, a temperature, a pressure, a voltage, current, humidity, velocity, acceleration, etc.</li> </ul> </li> <li><strong>5 million timestamps</strong>. Sensors readings are at 1Hz sampling frequency. <ul> <li><strong>1 million nominal </strong>observations (the first 1 million datapoints). This is suitable to start learning the "normal" behaviour.</li> <li><strong>4 million</strong> observations that include both <strong>nominal and anomalous segments</strong>. This is suitable to evaluate both semi-supervised approaches (novelty detection) as well as unsupervised approaches (outlier detection).</li> </ul> </li> <li><strong>200 anomalous segments. </strong>One anomalous segment may contain several successive anomalous observations / timestamps. Only the last 4 million observations contain anomalous segments.</li> <li><strong>Different types of anomalies </strong>to understand what anomaly types can be detected by different approaches. The categories are available in the dataset and in the metadata.</li> <li><strong>Fine control over ground truth.</strong> As this is a simulated system with deliberate anomaly injection, the start and end time of the anomalous behaviour is known very precisely. In contrast to real world datasets, there is no risk that the ground truth contains mislabelled segments which is often the case for real data.</li> <li><strong>Suitable for root cause analysis.</strong> In addition to the anomaly category, the time series channel in which the anomaly first developed itself is recorded and made available as part of the metadata. This can be useful to evaluate the performance of algorithm to trace back anomalies to the right root cause channel.</li> <li><strong>Affected channels.</strong> In addition to the knowledge of the root cause channel in which the anomaly first developed itself, we provide information of channels possibly affected by the anomaly. This can also be useful to evaluate the explainability of anomaly detection systems which may point out to the anomalous channels (root cause and affected).</li> <li><strong>Obvious anomalies.</strong> The simulated anomalies have been designed to be "easy" to be detected for human eyes (i.e., there are very large spikes or oscillations), hence also detectable for most algorithms. It makes this synthetic dataset useful for screening tasks (i.e., to eliminate algorithms that are not capable to detect those obvious anomalies). However, during our initial experiments, the dataset turned out to be challenging enough even for state-of-the-art anomaly detection approaches, making it suitable also for regular benchmark studies.</li> <li><strong>Context provided. </strong>Some variables can only be considered anomalous in relation to other behaviours. A typical example consists of a light and switch pair. The light being either on or off is nominal, the same goes for the switch, but having the switch on and the light off shall be considered anomalous. In the CATS dataset, users can choose (or not) to use the available context, and external stimuli, to test the usefulness of the context for detecting anomalies in this simulation.</li> <li><strong>Pure signal ideal for robustness-to-noise analysis.</strong> The simulated signals are provided without noise: while this may seem unrealistic at first, it is an advantage since users of the dataset can decide to add on top of the provided series any type of noise and choose an amplitude. This makes it well suited to test how sensitive and robust detection algorithms are against various levels of noise.</li> <li><strong>No missing data.</strong> You can drop whatever data you want to assess the impact of missing values on your detector with respect to a clean baseline.</li> </ul> <p><strong>Change Log</strong></p> <p>Version 2</p> <ul> <li><strong>Metadata:</strong> we include a metadata.csv with information about: <ul> <li>Anomaly categories</li> <li>Root cause channel (signal in which the anomaly is first visible)</li> <li>Affected channel (signal in which the anomaly might propagate) through coupled system dynamics</li> </ul> </li> <li><strong>Removal of anomaly overlaps:</strong> version 1 contained anomalies which overlapped with each other resulting in only 190 distinct anomalous segments. Now, there are no more anomaly overlaps.</li> <li><strong>Two data files: </strong>CSV and parquet for convenience.</li> </ul> <p>[1] Example Benchmark of Anomaly Detection in Time Series: “Sebastian Schmidl, Phillip Wenig, and Thorsten Papenbrock. Anomaly Detection in Time Series: A Comprehensive Evaluation. PVLDB, 15(9): 1779 - 1797, 2022. doi:10.14778/3538598.3538602”</p> <p><strong>About Solenix</strong></p> <p>Solenix is an international company providing software engineering, consulting services and software products for the space market. Solenix is a dynamic company that brings innovative technologies and concepts to the aerospace market, keeping up to date with technical advancements and actively promoting spin-in and spin-out technology activities. We combine modern solutions which complement conventional practices. We aspire to achieve maximum customer satisfaction by fostering collaboration, constructivism, and flexibility.</p>
Data for the MLCS 2020 paper "A Year of Automated Anomaly Detection in a Datacenter"
<p>This contains the data used for the paper by Ahmed et. al in the MLCS 2020 paper "A Year of Automated Anomaly Detection in a Datacenter". Each of the four CSV files corresponds to one of the quarters discussed in the paper, and each has a metadata file containing information about the query that produced them. The CSV files contain the 'raw' log messages, and an eventID that identifies which pattern the log entry matched; the eventID is used to group together log messages of the same type. These logfiles were collected on the CloudLab facility (https://cloudlab.us/) from Jan 1 - Dec 30, 2019.</p> <p>The violated_unviolated_sessions_*.txt files each contain 20 randomly-selected sessions: half of the sessions were labeled by the invariant miner as being 'normal', and the other half 'anomalous'. CloudLab developers and system administrators were asked to label these sessions manually (and were not given the invariant miner's labels). The corresponding *_manual_labels.txt contain the labels that the administrators assigned, and in some cases additional correspondence with the administrators and information about which manual labels matched the invariant miner and which did not.</p>
Validation of ESA CCI SM combined v05.2 vs ISMN 20191211 global - Anomalies and no ISMN flags
QA4SM validation of soil moisture data: ESA CCI SM combined v05.2 vs ISMN 20191211 global. URL: https://qa4sm.eu/result/5f6ae4c5-5115-4022-b489-99f4dda1089f/. Produced on QA4SM (https://qa4sm.eu)
Validation of ESA CCI SM combined v05.2 vs ISMN 20191211 global - Anomalies and ISMN flagged
QA4SM validation of soil moisture data: ESA CCI SM combined v05.2 vs ISMN 20191211 global. URL: https://qa4sm.eu/result/d8b5f409-2bb3-4580-be0b-0c9c2d71c968/. Produced on QA4SM (https://qa4sm.eu)
Validation of ESA CCI SM combined v05.2 vs ISMN 20191211 global - without Anomalies and ISMN flagged
QA4SM validation of soil moisture data: ESA CCI SM combined v05.2 vs ISMN 20191211 global. URL: https://qa4sm.eu/result/73ed1e31-eaa5-469a-ab9b-4451e4e4d4df/. Produced on QA4SM (https://qa4sm.eu)
Validation of ESA CCI SM combined v05.2 vs ISMN 20191211 global - without anomalies and without ISMN flags
QA4SM validation of soil moisture data: ESA CCI SM combined v05.2 vs ISMN 20191211 global. URL: https://qa4sm.eu/result/bbf7f693-74ff-4b03-8f5d-792b7b846b40/. Produced on QA4SM (https://qa4sm.eu)
GFDL CM2.1 Partially-Coupled Simulations Data for "Understanding Lead Times of Warm-Water-Volumes to ENSO Sea Surface Temperature Anomalies"
<p>GFDL CM2.1 partially-coupled idealized simulations:</p> <p>Two sets of idealized experiments with prescribed EP and CP ENSO SST anomaly patterns. Each set of experiments has a prescribed idealized sinusoidal ENSO oscillation with periodicities of 48, 36, and 24 months, respectively.</p> <p>For the details please refer to our paper;<br> Zhao, S., Jin, F.-F., & Stuecker, M. F. (2021). Understanding Lead Times of Warm Water Volumes to ENSO Sea Surface Temperature Anomalies. <em>Geophysical Research Letters</em>, <em>48</em>(19), e2021GL094366. <a href="https://doi.org/10.1029/2021GL094366">https://doi.org/10.1029/2021GL094366</a></p> <p> </p> <p> </p> <p> </p>
Gravity, Free-Air and Bouguer Anomaly Data in the Ivrea-Verbano Zone (Western Alps, Italy)
<p>Gravity dataset collected in the Ivrea-Verbano Zone (IVZ, Western Alps, Italy). </p><p>The data was collected in the frame of a gravity-based investigation and modelling of the Ivrea Geophysical Body. </p><p>For citation and further details on the work see Scarponi et al. (2020, GJI): <a href="https://doi.org/10.1093/gji/ggaa263">https://doi.org/10.1093/gji/ggaa263</a></p><p>The file contains the gravity data collected in the IVZ region, including free-air anomaly and Bouguer gravity anomaly (in mGal).</p><p>Longitude, Latitude coordinates are in degrees, elevation in meters.</p><p>Uncertainty on the final gravity data products and gravity data is 1 mGal.</p><p>---</p><p>Data collection, as well as the associated research, were supported by the Swiss National Science Foundation (SNF) (grant numbers PP00P2_157627 and PP00P2_187199).</p>
Dataset of Hyperspectral Melt Pool Signatures and Thermal Anomalies in DED of 316L steel
<p><strong>Description of the dataset</strong><br>The dataset includes in-situ melt pool signatures (hyperspectral NIR images) during the Directed Energy Deposition of 316L steel for several classes of thermal anomalies. Thermal anomalies were created during the process by varying the scanning speed.</p> <p>Samples were printed on the MiCLAD machine at the Vrije Universiteit Brussel (Belgium).</p> <p>Process and acquisition parameters:</p> <ul> <li>Hardware: <ul> <li>Machine: MiCLAD (Vrije Universiteit Brussel)</li> <li>Laser: High-YAG BIMO 1064nm, 2.55mm fibre, flat-top</li> <li>Nozzle: Harald-Dickler HighNo 4.0</li> </ul> </li> <li>Process parameters: <ul> <li>Laser power: 600 W</li> <li>Scanning speed: 500/700/900/1100/1300 mm/min</li> <li>Powder: 316L 45-105 um</li> <li>Powder flow rate: 3.5 g/m</li> <li>Layer thickness: 0.2 mm</li> </ul> </li> <li>Image characteristics: <ul> <li>Camera: 3D-One Avior AX-M25NIR</li> <li>Hyperspectral filter layout: 5x5 (25 wavelengths per image)</li> </ul> </li> </ul> <p><strong>Description of the files</strong></p> <ul> <li>CSV dataset (hyperspectral_nir_meltpool_dataset.csv): List of filename, sample, label, time (ms), X and Z position (mm) and local scanning speed (mm/min) for all melt pool signatures. Thermal anomalies are labelled accordingly: <ul> <li>0 : baseline</li> <li>1 : edge</li> <li>2 : underheat</li> <li>3 : strong underheat</li> <li>4 : overheat</li> <li>5 : strong overheat</li> </ul> </li> <li>Melt pool signatures (hyperspectral_nir_meltpool_images_*.zip): Raw .tif thermal images of the melt pool taken in-situ. The raw images must debayered to retrieve the spectral information, see the Python function and example script. </li> <li>Python debayer function (debayer.py): Debayering function to retrieve the spectral information from the raw images. </li> </ul>
Geodetic anomaly detection and analysis in the Campi Flegrei caldera (Italy) deformation pattern of the 2021-2023 escalating unrest phase
<p>Data used within the manuscript: "<strong><span>First evidence of a geodetic anomaly in the Campi Flegrei caldera (Italy) ground deformation pattern revealed by DInSAR and GNSS measurements during the 2021-2023 escalating unrest phase</span>"</strong></p> <p> </p> <p>Archive content:</p> <ul> <li><code>DTSLOS_CNRIREA_20150325_20231021_FB9K</code>: Line of Sight displacement time series retrieved by applying the P-SBAS algorithm to Sentinel-1 data set acquired from ascending orbits (Track 44) over Campi Flegrei caldera in the 20150325 - 20231021 interval. Data format is according to the <a href="https://gitlab.com/epos-tcs-satdata/doc/-/blob/main/TCS_SATD_Product_Description.md#los-displacement-time-series-dtslos" target="_blank" rel="noopener noreferrer">EPOS specification</a>.</li> <li><code>DTSLOS_CNRIREA_20150324_20231020_UJBI</code>: Line of Sight displacement time series retrieved by applying the P-SBAS algorithm to Sentinel-1 data set acquired from descending orbits (Track 22) over Campi Flegrei caldera in the 20150324 - 20231020 interval. Data format is according to the <a href="https://gitlab.com/epos-tcs-satdata/doc/-/blob/main/TCS_SATD_Product_Description.md#los-displacement-time-series-dtslos" target="_blank" rel="noopener noreferrer">EPOS specification</a>.</li> <li><code>Campi_Flegrei_GNSS_Weekly_Timeseries</code>: Weekly displacement time series of Campi Flegrei caldera GNSS network from 2016 to 2023.</li> </ul>
Evaluation of Upper Tropospheric Geopotential Height Anomalies over the Tropical and Subtropical Oceans in CMIP6 Models Using GNSS Radio Occultation Observations
<p>The set-up of CESM2-CAM6 sensitivity experiments for winter season (Dec-Jan-Feb: DJF), with prognostic falling ice radiative effects on (SON) and off (NOS), is an updated two-moment stratiform cloud scheme (MG2, Gettelman & Morrison, 2015) in the CESM2 atmospheric component of CAM6. CESM2-CAM6 participated in CMIP6. Both the NOS and SON simulations were configured following the same approach as the CMIP6 "historical" run spanning from 1980 to 2014.</p> <p> </p> <p>The data are:</p> <p> </p> <p>TS: skin temperature (K)</p> <p>TAUX: zonal surface wind stress</p> <p>TAUY: meridinal surface wind stress</p> <p>DTCOND: moist condensation heating rate</p> <p>QRL: long wave heating rate</p> <p>OMEGA: vertical motion</p> <p>Z3: geopotential height</p> <p> </p>
Real-bogus scores for active anomaly detection
<p>Data description for <a href="https://arxiv.org/abs/2409.10256">Semenikhin et al., 2024</a></p> <p>The dataset consists of the following files:</p> <p><strong>feature_snad4_r_100.dat</strong> contains light curve feature data for objects, where each object is represented by 54 feature values. These values are encoded as little-endian single-precision IEEE-754 floating-point numbers (32-bit floats). Feature names are listed in the plain text file <strong>feature_snad4_r_100.name</strong>, with one name per line.<br><strong>sid_snad4_r_100.dat</strong> contains ZTF DR object identifiers, encoded as little-endian 64-bit unsigned integers.</p> <p><strong>exp_feature_snad4_r_100.dat</strong> contains the same features as <strong>feature_snad4_r_100.dat</strong>, but with an additional column representing the real-bogus classifier prediction. Each object in this file corresponds to 55 features: the original 54 features plus 1 additional feature. Feature names for this file are provided in <strong>exp_feature_snad4_r_100.name</strong>.</p> <p>The files <strong>sid_snad4_r_100.dat</strong>, <strong>feature_snad4_r_100.dat</strong>, and <strong>exp_feature_snad4_r_100.dat</strong> share the same object order.</p> <p><br>Below is a sample Python script for accessing the data using NumPy:</p> <p><code>import numpy as np</code></p> <p><code># Load object IDs</code><br><code>oid = np.memmap('sid_snad4_r_100.dat', mode='c', dtype=np.uint64)</code></p> <p><code># Load features and reshape</code><br><code>feature = np.memmap('feature_snad4_r_100.dat', mode='c', dtype=np.float32).reshape(oid.shape[0], -1)</code></p> <p><code># Print dataset information</code><br><code>print(f'Number of objects: {len(oid)}')</code><br><code>print(f'Features shape: {feature.shape}')</code></p>
Testing the Fidelity of Paleopole Determinations from Multidirectionally Magnetized Lunar Crustal Anomaly Source Bodies
<p>Code and data for "Testing the Fidelity of Paleopole Determinations from Multidirectionally Magnetized Lunar Crustal Anomaly Source Bodies ". See readme.txt</p>
A Global Dataset of Standardized Moisture Anomaly Index Incorporating Snow Dynamics (SZIsnow) from 1948 to 2010
<p>The SZI<sub>snow</sub> dataset was calculated based on systematic physical fields from the Global Land Data Assimilation System Version 2 (GLDAS-2) with the Noah land surface model. This SZI<sub>snow</sub> dataset considers different physical water-energy processes, especially snow processes. The evaluation shows the dataset is capable of investigating different types of droughts across different timescales. The assessment also indicates that the dataset has an adequate performance to capture droughts across different spatial scales. The consideration of snow processes improved the capability of SZI<sub>snow</sub>, and the improvement is evident over snow-covered areas (e.g., Arctic region) and high-altitude areas (e.g., Tibet Plateau). Moreover, the analysis also implies that SZI<sub>snow</sub> dataset is able to well capture large-scale drought events across the world. This drought dataset has high application potential for monitoring, assessing, and supplying information on drought, and also can serve as a valuable resource for drought studies.</p>
Figures 1–3 in A rare telson anomaly in Parabuthus liosoma (Ehrenberg, 1828) (Scorpiones: Buthidae)
Figures 1–3: Parabuthus liosoma, female. Figure 1. Habitus. Figures 2–3: Telson in lateral views; arrow shows dorsal aculeus.
Figures 1–6 in An anomaly of chelicera in Scorpio kruglovi Birula, 1910 (Scorpiones: Scorpionidae)
Figures 1–6: Scorpio kruglovi, female. Figures 1–2. Dorsal (1) and ventral (2) views. Figures 3–6. Carapace (3), coxosternal area (4), and chelicerae in dorsal (5) and ventral (6) views. Scale bar: 10 mm (1–2).
A novel method for objective identification of 3-D potential vorticity anomalies - Visualizations using Met.3D
<p>This video provides 3-D visualizations of the outputs of the Potential Vorticity anomaly identification technique regarding an investigated case study. An in-depth description of the algorithm, as well as evaluations of the visible results are performed in the study.</p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.