Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
9,302
datasets available to search
ShareScore release 0.9.0
Dataset results
9,302 results for “DETECT”
XMM-Newton 2XMMi-DR3 Selected Source Detections Catalog
The authors have carried out a classification of 4,330 X-ray sources in the 2XMMi-DR3 catalog. The sources were selected under the requirement of being a point source with multiple XMM-Newton observations and at least one detection with a signal-to-noise ratio larger than 20. For about one-third of the sources, the authors are able to obtain reliable source types from the literature. They mostly correspond to various types of stars (611), active galactic nuclei (AGNs, 753), and compact object systems (138) containing white dwarfs, neutron stars, and stellar-mass black holes. The authors find that about 99% of stars can be separated from other source types based on their low X-ray-to-IR flux ratios and frequent X-ray flares. AGNs have remarkably similar X-ray spectra, with the power-law photon index centered around 1.91 +/- 0.31, and their 0.2-4.5 keV flux long-term variation factors have a median of 1.48, with 98.5% being less than 10. In contrast, 70% of compact object systems can be very soft or hard, highly variable in X-rays, and/or have very large X-ray-to-IR flux ratios, separating them from AGNs. Using these results, the authors derive a source type classification scheme to classify the other sources and find 644 candidate stars, 1,376 candidate AGNs, and 202 candidate compact object systems, whose false identification probabilities are estimated to be about 1%, 3%, and 18%, respectively. There are still 320 sources associated with nearby galaxies and 151 in the Galactic plane, which the authors expect to be mostly compact object systems or background AGNs. There are also 100 candidate ultraluminous X-ray sources. They are found to be much less variable than other accreting compact objects. This table contains the list of 19,637 detections of the 4,330 unique X-ray sources which comprise the authors' sample. The list of 4,330 unique X-ray sources and their classifications is also available as the HEASARC XMMSSCLWBS table. This table was created by the HEASARC in October 2012 based on an electronic version of Table 3 from the reference paper which was obtained from the ApJ web site. This is a service provided by NASA HEASARC .
A knowledge-based system approach for sensor fault modeling, detection and mitigation
Sensors are vital components for control and advanced health management techniques. However, sensors continue to be considered the weak link in many engineering applications since often they are less reli- able than the system they are observing. This is in part due to the sensors’ operating principles and their susceptibility to interference from the environment. Detecting and mitigating sensor failure modes are becoming increasingly important in more complex and safety-critical applications. This paper reports on different techniques for sensor fault detection, disambiguation, and mitigation. It presents an expert system that uses a combination of object-oriented modeling, rules, and semantic networks to deal with the most common sensor faults, such as bias, drift, scaling, and dropout, as well as system faults. The paper also describes a sensor correction module that is based on fault parameters extraction (for bias, drift, and scaling fault modes) as well as utilizing partial redundancy for dropout sensor fault modes). The knowledge-based system was derived from the results obtained in a previously deployed Neural Network (NN) application for fault detection and disambiguation. Results are illustrated on an electromechanical actuator application where the system faults are jam and spalling. In addition to the functions implemented in the previous work, system fault detection under sensor failure was also modeled. The paper includes a sensitivity analysis that compares the results previously obtained with the NN. It concludes with a discussion of similarities and differences between the two approaches and how the knowledge based system provides additional functionality compared to the NN implementation.
ULYSSES DUST DETECTION SYSTEM V3.1
This data set contains the data from the Ulysses dust detector system (UDDS) from start of mission through the end of mission, 1990-2007. (As the dust detector was turned off after Nov. 30, 2007, this is the last date for which UDDS data is recorded.) Included are the dust impact data, noise data, laboratory calibration data, and location and orientation of the spacecraft and instrument.
Comparative Analysis of Data-Driven Anomaly Detection Methods
This paper provides a review of three different advanced machine learning algorithms for anomaly detection in continuous data streams from a ground-test firing of a subscale Solid Rocket Motor (SRM). This study compares Orca, one-class support vector machines, and the Inductive Monitoring System (IMS) for anomaly detection on the data streams. We measure the performance of the algorithm with respect to the detection horizon for situations where fault information is available. These algorithms have been also studied by the present authors (and other co-authors) as applied to liquid propulsion systems. The trade space will be explored between these algorithms for both types of propulsion systems.
Algorithms for Speeding up Distance-Based Outlier Detection
The problem of distance-based outlier detection is difficult to solve efficiently in very large datasets because of potential quadratic time complexity. We address this problem and develop sequential and distributed algorithms that are significantly more efficient than state-of-the-art methods while still guaranteeing the same outliers. By combining simple but effective indexing and disk block accessing techniques, we have developed a sequential algorithm iOrca that is up to an order-of-magnitude faster than the state-of-the-art. The indexing scheme is based on sorting the data points in order of increasing distance from a fixed reference point and then accessing those points based on this sorted order. To speed up the basic outlier detection technique, we develop two distributed algorithms (DOoR and iDOoR) for modern distributed multi-core clusters of machines, connected on a ring topology. The first algorithm passes data blocks from each machine around the ring, incrementally updating the nearest neighbors of the points passed. By maintaining a cutoff threshold, it is able to prune a large number of points in a distributed fashion. The second distributed algorithm extends this basic idea with the indexing scheme discussed earlier. In our experiments, both distributed algorithms exhibit significant improvements compared to the state-of-the-art distributed methods.
Comparison of Unsupervised Anomaly Detection Methods
Several different unsupervised anomaly detection algorithms have been applied to Space Shuttle Main Engine (SSME) data to serve the purpose of developing a comprehensive suite of Integrated Systems Health Management (ISHM) tools. As the theoretical bases for these methods vary considerably, it is reasonable to conjecture that the resulting anomalies detected by them may differ quite significantly as well. As such, it would be useful to apply a common metric with which to compare the results. However, for such a quantitative analysis to be statistically significant, a sufficient number of examples of both nominally categorized and anomalous data must be available. Due to the lack of sufficient examples of anomalous data, use of any statistics that rely upon a statistically significant sample of anomalous data is infeasible. Therefore, the main focus of this paper will be to compare actual examples of anomalies detected by the algorithms via the sensors in which they appear, as well the times at which they appear. We find that there is enough overlap in detection of the anomalies among all of the different algorithms tested in order for them to corroborate the severity of these anomalies. In certain cases, the severity of these anomalies is supported by their categorization as failures by experts, with realistic physical explanations. For those anomalies that can not be corroborated by at least one other method, this overlap says less about the severity of the anomaly, and more about their technical nuances, which will also be discussed.
Anomaly Detection in a Fleet of Systems
A fleet is a group of systems (e.g., cars, aircraft) that are designed and manufactured the same way and are intended to be used the same way. For example, a fleet of delivery trucks may consist of one hundred instances of a particular model of truck, each of which is intended for the same type of service—almost the same amount of time and distance driven every day, approximately the same total weight carried, etc. For this reason, one may imagine that data mining for fleet monitoring may merely involve collecting operating data from the multiple systems in the fleet and developing some sort of model, such as a model of normal operation that can be used for anomaly detection. However, one then may realize that each member of the fleet will be unique in some ways—there will be minor variations in manufacturing, quality of parts, and usage. For this reason, the typical machine learning and statis- tics algorithm’s assumption that all the data are independent and identically distributed is not correct. One may realize that data from each system in the fleet must be treated as unique so that one can notice significant changes in the operation of that system.
Near-UV Detected Bright Early-M Dwarf Star Catalog
Planets orbiting within the close-in habitable zones of M dwarf stars will be exposed to elevated high-energy radiation driven by strong magnetohydrodynamic dynamos during stellar youth. Near-ultraviolet (NUV) irradiation can erode and alter the chemistry of planetary atmospheres, and a quantitative description of the evolution of NUV emission from M dwarfs is needed when modeling these effects. The authors investigated the NUV luminosity evolution of early M-type dwarfs by cross-correlating the Lepine & Gaidos (LG11: 2011, AJ, 142, 138) catalog of bright M dwarfs (available at the HEASARC as the MDWARFASC table) with the Galaxy Evolution Explorer (GALEX) catalog of NUV (1771-2831 Angstrom) sources. Of the 4,805 sources with GALEX counterparts, 797 have NUV emission significantly (> 2.5 sigma) in excess of an empirical basal level. The authors inspected these candidate active stars using visible-wavelength spectra, high-resolution adaptive optics imaging, time-series photometry, and literature searches to identify cases where the elevated NUV emission is due to unresolved background sources or stellar companions; they estimated the overall occurrence of these "false positives" (FPs) as ~ 16%. The authors constructed an NUV luminosity function that accounted for FPs, detection biases of the source catalogs, and GALEX upper limits. They found the NUV luminosity function to be inconsistent with predictions from a constant star-formation rate and simplified age-activity relation defined by a two-parameter power law. This table was created by the HEASARC in March 2015 based on an electronic versions of Table 4 from the reference paper which was obtained from the ApJ web site. This is a service provided by NASA HEASARC .
ν-Anomica: A Fast Support Vector based Novelty Detection Technique
In this paper we propose ν-Anomica, a novel anomaly detection technique that can be trained on huge data sets with much reduced running time compared to the benchmark one-class Support Vector Machines algorithm. In ν-Anomica, the idea is to train the machine such that it can provide a close approximation to the exact decision plane using fewer training points and without losing much of the generalization performance of the classical approach. We have tested the proposed algorithm on a variety of continuous data sets under different conditions. We show that under all test conditions the developed procedure closely preserves the accuracy of standard oneclass Support Vector Machines while reducing both the training time and the test time by 5 − 20 times.
Anomica: Fast Support Vector Based Novelty Detection
In this paper we propose ν-Anomica, a novel anomaly detection technique that can be trained on huge data sets with much reduced running time compared to the benchmark one-class Support Vector Machines algorithm. In ν-Anomica, the idea is to train the machine such that it can provide a close approximation to the exact decision plane using fewer training points and without losing much of the generalization performance of the classical approach. We have tested the proposed algorithm on a variety of continuous data sets under different conditions. We show that under all test conditions the developed procedure closely preserves the accuracy of standard oneclass Support Vector Machines while reducing both the training time and the test time by 5 − 20 times.
Multiple Kernel Learning for Heterogeneous Anomaly Detection: Algorithm and Aviation Safety Case Study
The world-wide aviation system is one of the most complex dynamical systems ever developed and is generating data at an extremely rapid rate. Most modern commercial aircraft record several hundred flight parameters including information from the guidance, navigation, and control systems, the avionics and propulsion systems, and the pilot inputs into the aircraft. These parameters may be continuous measurements or binary or categorical measurements recorded in one second intervals for the duration of the flight. Currently, most approaches to aviation safety are reactive, meaning that they are designed to react to an aviation safety incident or accident. In this paper, we discuss a novel approach based on the theory of multiple kernel learning to detect potential safety anomalies in very large data bases of discrete and continuous data from world-wide operations of commercial fleets. We pose a general anomaly detection problem which includes both discrete and continuous data streams, where we assume that the discrete streams have a causal influence on the continuous streams. We also assume that atypical sequences of events in the discrete streams can lead to off-nominal system performance. We discuss the application domain, novel algorithms, and also discuss results on real-world data sets. Our algorithm uncovers operationally significant events in high dimensional data streams in the aviation industry which are not detectable using state of the art methods.
Modeling, Detection, and Disambiguation of Sensor Faults for Aerospace Applications
Sensor faults continue to be a major hurdle for sys- tems health management to reach its full potential. At the same time, few recorded instances of sensor faults exist. It is equally dif- ficult to seed particular sensor faults. Therefore, research is un- derway to better understand the different fault modes seen in sen- sors and to model the faults. The fault models can then be used in simulated sensor fault scenarios to ensure that algorithms can distinguish between sensor faults and system faults. The paper il- lustrates the work with data collected from an electromechanical actuator in an aerospace setting, equipped with temperature, vi- bration, current, and position sensors. The most common sensor faults, such as bias, drift, scaling, and dropout were simulated and injected into the experimental data, with the goal of making these simulations as realistic as feasible. A neural network-based classi- fier was then created and tested on both experimental data and the more challenging randomized data sequences. Additional studies were also conducted to determine sensitivity of detection and dis- ambiguation efficacy with respect to severity of fault conditions.
Scalable Distributed Change Detection from Astronomy Data Streams using Local, Asynchronous Eigen Monitoring Algorithms
This paper considers the problem of change detection using local distributed eigen monitoring algorithms for next generation of astronomy petascale data pipelines such as the Large Synoptic Survey Telescopes (LSST). This telescope will take repeat images of the night sky every 20 seconds, thereby generating 30 terabytes of calibrated imagery every night that will need to be coanalyzed with other astronomical data stored at different locations around the world. Change point detection and event classification in such data sets may provide useful insights to unique astronomical phenomenon displaying astrophysically significant variations: quasars, supernovae, variable stars, and potentially hazardous asteroids. However, performing such data mining tasks is a challenging problem for such high-throughput distributed data streams. In this paper we propose a highly scalable and distributed asynchronous algorithm for monitoring the principal components (PC) of such dynamic data streams. We demonstrate the algorithm on a large set of distributed astronomical data to accomplish well-known astronomy tasks such as measuring variations in the fundamental plane of galaxy parameters. The proposed algorithm is provably correct (i.e. converges to the correct PCs without centralizing any data) and can seamlessly handle changes to the data or the network. Real experiments performed on Sloan Digital Sky Survey (SDSS) catalogue data show the effectiveness of the algorithm.
Privacy Preserving Outlier Detection through Random Nonlinear Data Distortion
Consider a scenario in which the data owner has some private/sensitive data and wants a data miner to access it for studying important patterns without revealing the sensitive information. Privacy preserving data mining aims to solve this problem by randomly transforming the data prior to its release to data miners. Previous work only considered the case of linear data perturbations — additive, multiplicative or a combination of both for studying the usefulness of the perturbed output. In this paper, we discuss nonlinear data distortion using potentially nonlinear random data transformation and show how it can be useful for privacy preserving anomaly detection from sensitive datasets. We develop bounds on the expected accuracy of the nonlinear distortion and also quantify privacy by using standard definitions. The highlight of this approach is to allow a user to control the amount of privacy by varying the degree of nonlinearity. We show how our general transformation can be used for anomaly detection in practice for two specific problem instances: a linear model and a popular nonlinear model using the sigmoid function. We also analyze the proposed nonlinear transformation in full generality and then show that for specific cases it is distance preserving. A main contribution of this paper is the discussion between the invertibility of a transformation and privacy preservation and the application of these techniques to outlier detection. Experiments conducted on real-life datasets demonstrate the effectiveness of the approach.
Anomaly Detection from ASRS Databases of Textual Reports
Our primary goal is to automatically analyze textual reports from the Aviation Safety Reporting System (ASRS) database to detect/discover the anomaly categories reported by the pilots, and to assign each report to the appropriate category/categories. We have used two state-of-the-art models for text analysis: (i) mixture of von Mises Fisher (movMF) distributions, and (ii) latent Dirichlet allocation (LDA) on a subset of all ASRS reports. The models achieve a reasonably high performance in discovering anomaly categories and clustering reports. Each category is represented by the most representative words with the highest probability in this category. In addition, since the inference algorithm for LDA was somewhat slow, we have developed a new fast LDA algorithm which is 5-10 times more efficient than the original one, therefore more applicable for the practical use. Further, we have developed a simple visualization tool based on non-linear manifold embedding (ISOMAP) to generate a 2-d visual representation of each report based on its content/topics, which gives a direct view of the structure of the whole dataset as well as the outliers.
Anomaly Detection in Sequences
We present a set of novel algorithms which we call sequenceMiner, that detect and characterize anomalies in large sets of high-dimensional symbol sequences that arise from recordings of switch sensors in the cockpits of commercial airliners. While the algorithms we present are general and domain-independent, we focus on a specific problem that is critical to determining system-wide health of a fleet of aircraft. The approach taken uses unsupervised clustering of sequences using the normalized length of he longest common subsequence (nLCS) as a similarity measure, followed by a detailed analysis of outliers to detect anomalies. In this method, an outlier sequence is defined as a sequence that is far away from a cluster. We present new algorithms for outlier analysis that provide comprehensible indicators as to why a particular sequence is deemed to be an outlier. The algorithm provides a coherent description to an analyst of the anomalies in the sequence when compared to more normal sequences. The final section of the paper demonstrates the effectiveness of sequenceMiner for anomaly detection on a real set of discrete sequence data from a fleet of commercial airliners. We show that sequenceMiner discovers actionable and operationally significant safety events. We also compare our innovations with standard HiddenMarkov Models, and show that our methods are superior
ULYSSES DUST DETECTION SYSTEM V3.0
This data set contains the data from the Ulysses dust detector system (UDDS) from start of mission through the end of mission, 1990-2007. (As the dust detector was turned off after Nov. 30, 2007, this is the last date for which UDDS data is recorded.) Included are the dust impact data, noise data, laboratory calibration data, and location and orientation of the spacecraft and instrument.
GALILEO DUST DETECTION SYSTEM V3.0
Detector responses and derived quantities from the Galileo dust detector as well as spacecraft geometry information for reliable impacts from launch through 2001. See Gruen et al. (Plan. Sp. Sci. 43, 953-969, 1995) and Krueger et al. (Plan. Sp. Sci. 47, 85-106, 1999; Plan. Sp. Sci. 49, 1285-1301, 2001) for more information.
Onboard Detection of Snow, Ice, Clouds, and Other Processes
The detection of clouds within a satellite image is essential for retrieving surface geophysical parameters from optical and thermal imagery. Even a small percentage of cloud cover within a radiometer pixel can adversely affect the determination of surface variables such as albedo and temperature. Thus, onboard processing of satellite data requires reliable automated cloud detection algorithms that are applicable to a wide range of surface types. Unfortunately cloud-detection, particularly over snow- and ice-covered surfaces, is a problem that plagues the field of remote sensing because of the lack of spectral contrast. This paper discusses preliminary results based on kernel methods for unsupervised discovery of snow, ice, clouds, and other geophysical processes based on data from the MODIS instrument and discusses implementation in computationally constrained environments such as those found on satellites.
Rotor health monitoring combining spin tests and data-driven anomaly detection methods
Health monitoring is highly dependent on sensor systems that are capable of performing in various engine environmental conditions and able to transmit a signal upon a predetermined crack length, while acting in a neutral form upon the overall performance of the engine system. Efforts are under way at NASA Glenn Research Center through support of the Intelligent Vehicle Health Management Project (IVHM) to develop and implement such sensor technology for a wide variety of applications. These efforts are focused on developing high temperature, wireless, low cost, and durable products. In an effort to address technical issues concerning health monitoring, this article considers data collected from an experimental study using high frequency capacitive sensor technology to capture blade tip clearance and tip timing measurements in a rotating turbine engine-like-disk to detect the disk faults and assess its structural integrity. The experimental results composed at a range of rotational speeds from tests conducted at the NASA Glenn Research Center’s Rotordynamics Laboratory are evaluated and integrated into multiple data-driven anomaly detection techniques to identify faults and anomalies in the disk. In summary, this study presents a select evaluation of online health monitoring of a rotating disk using high caliber capacitive sensors and demonstrates the capability of the in-house spin system.
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.