Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
1,782
datasets available to search
ShareScore release 0.9.0
Dataset results
1,782 results for “Algorithm”
Evaluating Algorithm Performance Metrics Tailored for Prognostics
Prognostics has taken center stage in Condition Based Maintenance (CBM) where it is desired to estimate Remaining Useful Life (RUL) of a system so that remedial measures may be taken in advance to avoid catastrophic events or unwanted downtimes. Validation of such predictions is an important but difficult proposition and a lack of appropriate evaluation methods renders prognostics meaningless. Evaluation methods currently used in the research community are not standardized and in many cases do not sufficiently assess key performance aspects expected out of a prognostics algorithm. In this paper we introduce several new evaluation metrics tailored for prognostics and show that they can effectively evaluate various algorithms as compared to other conventional metrics. Four prognostic algorithms, Relevance Vector Machine (RVM), Gaussian Process Regression (GPR), Artificial Neural Network (ANN), and Polynomial Regression (PR), are compared. These algorithms vary in complexity and their ability to manage uncertainty around predicted estimates. Results show that the new metrics rank these algorithms in a different manner; depending on the requirements and constraints suitable metrics may be chosen. Beyond these results, this paper offers ideas about how metrics suitable to prognostics may be designed so that the evaluation procedure can be standardized.
A Generic Local Algorithm for Mining Data Streams in Large Distributed Systems
In a large network of computers or wireless sensors, each of the components (henceforth, peers) has some data about the global state of the system. Much of the system's functionality such as message routing, information retrieval and load sharing relies on modeling the global state. We refer to the outcome of the function (e.g., the load experienced by each peer) as the emph{model} of the system. Since the state of the system is constantly changing, it is necessary to keep the models up-to-date. Computing global data mining models e.g. decision trees, k-means clustering in large distributed systems may be very costly due to the scale of the system and due to communication cost, which may be high. The cost further increases in a dynamic scenario when the data changes rapidly. In this paper we describe a two step approach for dealing with these costs. First, we describe a highly efficient emph{local} algorithm which can be used to monitor a wide class of data mining models. Then, we use this algorithm as a feedback loop for the monitoring of complex functions of the data such as its k-means clustering. The theoretical claims are corroborated with a thorough experimental analysis.
nu-Anomica algorithm
One-class nu-Support Vector machine (SVMs) learning technique maps the input data into a much higher dimensional space and then uses a small portion of the training data (support vectors) to parametrize the decision surface that can linearly separate nu fraction of training points (labeled as anomalies) from the rest. The exact solution of standard one-class nu SVMs assigns (at least) nu fraction of training points as support vectors. However some of these support vectors may be unnecessary or redundant. Hence the computational issue turns alarming especially when SVMs based novelty detectors with nonlinear kernels are trained on data sets of huge size. The proposed nu-Anomica algorithm can solve this problem. The idea is to train the machine such that it can provide a close approximation to the exact decision plane using far less number of training points and without loosing much of the generalization performance of the classical approach. The developed procedure closely preserves the accuracy of standard One-class nu-SVMs while reducing both training time and test time by several factors.
Highly Scalable Matching Pursuit Signal Decomposition Algorithm
In this research, we propose a variant of the classical Matching Pursuit Decomposition (MPD) algorithm with significantly improved scalability and computational performance. MPD is a powerful iterative algorithm that decomposes a signal into linear combinations of its dictionary elements or “atoms”. A best fit atom from an arbitrarily defined dictionary is determined through cross-correlation. The selected atom is subtracted from the signal and this procedure is repeated on the residual in the subsequent iterations until a stopping criteria is met. A sufficiently large dictionary is required for an accurate reconstruction; this in return increases the computational burden of the algorithm, thus limiting its applicability and level of adoption. Our main contribution lies in improving the computational efficiency of the algorithm to allow faster decomposition while maintaining a similar level of accuracy. The Correlation Thresholding and Multiple Atom Extractions techniques were proposed to decrease the computational burden of the algorithm. Correlation thresholds prune insignificant atoms from the dictionary. The ability to extract multiple atoms within a single iteration enhanced the effectiveness and efficiency of each iteration. The proposed algorithm, entitled MPD++, was demonstrated using real world data set.
AMSR-E/Aqua L2B Global Swath Surface Precipitation GSFC Profiling Algorithm V003
The AMSR-E/Aqua Level-2B precipitation product includes instantaneous surface precipitation rate and type over ice-free/snow-free land and ocean between 89.24 degrees north and south latitudes at at 10 km spatial resolution along the track and 5 km spatial resolution along the scan. The data are generated by the GPROF 2010 Version 2 algorithm using Version 3 of the AMSR-E Level-2A Brightness Temperatures.
Distilling the Verification Process for Prognostics Algorithms
The goal of prognostics and health management (PHM) systems is to ensure system safety, and reduce downtime and maintenance costs. It is important that a PHM system is verified and validated before it can be successfully deployed. Prognostics algorithms are integral parts of PHM systems. This paper investigates a systematic process of verification of such prognostics algorithms. To this end, first, this paper distinguishes between technology maturation and product development. Then, the paper describes the verification process for a prognostics algorithm as it moves up to higher maturity levels. This process is shown to be an iterative process where verification activities are interleaved with validation activities at each maturation level. In this work, we adopt the concept of technology readiness levels (TRLs) to represent the different maturity levels of a prognostics algorithm. It is shown that at each TRL, the verification of a prognostics algorithm depends on verifying the different components of the algorithm according to the requirements laid out by the PHM system that adopts this prognostics algorithm. Finally, using simplified examples, the systematic process for verifying a prognostics algorithm is demonstrated as the prognostics algorithm moves up TRLs.
A Local Asynchronous Distributed Privacy Preserving Feature Selection Algorithm for Large Peer-to-Peer Networks
In this paper we develop a local distributed privacy preserving algorithm for feature selection in a large peer-to-peer environment. Feature selection is often used in machine learning for data compaction and efficient learning by eliminating the curse of dimensionality. There exist many solutions for feature selection when the data is located at a central location. However, it becomes extremely challenging to perform the same when the data is distributed across a large number of peers or machines. Centralizing the entire dataset or portions of it can be very costly and impractical because of the large number of data sources, the asynchronous nature of the peer-to-peer networks, dynamic nature of the data/network and privacy concerns. The solution proposed in this paper allows us to perform feature selection in an asynchronous fashion with a low communication overhead where each peer can specify its own privacy constraints. The algorithm works based on local interactions among participating nodes. We present results on real-world datasets in order to performance of the proposed algorithm.
AMSR-E/Aqua L2B Global Swath Surface Precipitation GSFC Profiling Algorithm V004
The AMSR-E/Aqua Level-2B precipitation product includes instantaneous surface precipitation rate and type over ice-free/snow-free land and ocean between 89.24 degrees north and south latitudes at at 10 km spatial resolution along the track and 5 km spatial resolution along the scan. The data are generated by the GPROF 2017 algorithm using Version 4 of the AMSR-E Level-2A Brightness Temperatures.
Development of a Mobile Robot Test Platform and Methods for Validation of Prognostics-Enabled Decision Making Algorithms
As fault diagnosis and prognosis systems in aerospace applications become more capable, the ability to utilize information supplied by them becomes increasingly important. While certain types of vehicle health data can be effectively processed and acted upon by crew or support personnel, others, due to their complexity or time constraints, require either automated or semi-automated reasoning. Prognostics-enabled Decision Making (PDM) is an emerging research area that aims to integrate prognostic health information and knowledge about the future operating conditions into the process of selecting subsequent actions for the system. The newly developed PDM algorithms require suitable software and hardware platforms for testing under realistic fault scenarios. The paper describes the development of such a platform, based on the K11 planetary rover prototype. A variety of injectable fault modes are being investigated for electrical, mechanical, and power subsystems of the testbed, along with methods for data collection and processing. In addition to the hardware platform, a software simulator with matching capabilities has been developed. The simulator allows for prototyping and initial validation of the algorithms prior to their deployment on the K11. The simulator is also available to the PDM algorithms to assist with the reasoning process. A reference set of diagnostic, prognostic, and decision making algorithms is also described, followed by an overview of the current test scenarios and the results of their execution on the simulator.
AMSR-E/Aqua L2B Global Swath Surface Precipitation GSFC Profiling Algorithm V004
The AMSR-E/Aqua Level-2B precipitation product includes instantaneous surface precipitation rate and type over ice-free/snow-free land and ocean between 89.24 degrees north and south latitudes at at 10 km spatial resolution along the track and 5 km spatial resolution along the scan. The data are generated by the GPROF 2017 algorithm using Version 4 of the AMSR-E Level-2A Brightness Temperatures.
AMSR-E/Aqua L2B Global Swath Ocean Products derived from Wentz Algorithm V002
This daily Level-2B swath data set includes Sea Surface Temperature (SST), Near-Surface Wind Speed, Columnar Water Vapor, and Cloud liquid Water data arrays, and was used as input to generate the following daily, weekly, and monthly Level-3 gridded ocean products; AE_DyOcn, AE_WkOcn, and AE_MoOcn.
A Discussion on Uncertainty Representation and Interpretation in Model-based Prognostics Algorithms based on Kalman Filter Estimation Applied to Prognostics of Electronics Components
This article presented a discussion on uncertainty representation and management for model-based prog- nostics methodologies based on the Bayesian tracking framework and specifically for a Kalman filter appli- cation to electronics components. In particular, it explores the implication of modeling remaining useful life prediction as a stochastic process and how it relates to remaining useful life computation by statistical models, to uncertainty representation and management, and to the role of prognostics in decision-making. A discussion on how uncertainty propagates from the health state estimation process through the health state forecasting process is provided. Remaining useful life computation steps under uncertainty are pre- sented and analytical results on uncertainty quantification are provided under a simplified scenario. A proper propagation of uncertainty through the RUL prediction step as well as its correct interpretation are key to developing decision-making methodologies that make use of the remaining useful life prediction estimates and their corresponding uncertainties in order to make actionable choices that will optimize reliability, operations or safety in view of the prognostics information.
sequenceMiner algorithm
Detecting and describing anomalies in large repositories of discrete symbol sequences. **sequenceMiner has been open-sourced! Download the file below to try it out.** sequenceMiner was developed to address the problem of detecting and describing anomalies in large sets of high-dimensional symbol sequences. sequenceMiner works by performing unsupervised clustering (grouping) of sequences using the normalized longest common subsequence (LCS) as a similarity measure, followed by a detailed analysis of outliers to detect anomalies. sequenceMiner utilizes a new hybrid algorithm for computing the LCS that has been shown to outperform existing algorithms by a factor of five. sequenceMiner also includes new algorithms for outlier analysis that provide comprehensible indicators as to why a particular sequence was deemed to be an outlier. This provides analysts with a coherent description of the anomalies identified in the sequence, and why they differ from more “normal” sequences. sequenceMiner was developed with funding from the NASA Aviation Safety Program. In the commercial aviation domain, sequenceMiner can be used to discover atypical behavior in airline performance data that may have possible operational significance for safety analysts. But because the sequenceMiner approach is general and not restricted in any way to a domain, and these algorithms can be applied in other fields where anomaly detection and event mining would be useful.
A Scalable Local Algorithm for Distributed Multivariate Regression
This paper offers a local distributed algorithm for multivariate regression in large peer-to-peer environments. The algorithm can be used for distributed inferencing, data compaction, data modeling and classification tasks in many emerging peer-to-peer applications for bioinformatics, astronomy, social networking, sensor networks and web mining. Computing a global regression model from data available at the different peer-nodes using a traditional centralized algorithm for regression can be very costly and impractical because of the large number of data sources, the asynchronous nature of the peer-to-peer networks, and dynamic nature of the data/network. This paper proposes a two-step approach to deal with this problem. First, it offers an efficient local distributed algorithm that monitors the quality of the current regression model. If the model is outdated, it uses this algorithm as a feedback mechanism for rebuilding the model. The local nature of the monitoring algorithm guarantees low monitoring cost. Experimental results presented in this paper strongly support the theoretical claims.
Algorithms for Speeding up Distance-Based Outlier Detection
The problem of distance-based outlier detection is difficult to solve efficiently in very large datasets because of potential quadratic time complexity. We address this problem and develop sequential and distributed algorithms that are significantly more efficient than state-of-the-art methods while still guaranteeing the same outliers. By combining simple but effective indexing and disk block accessing techniques, we have developed a sequential algorithm iOrca that is up to an order-of-magnitude faster than the state-of-the-art. The indexing scheme is based on sorting the data points in order of increasing distance from a fixed reference point and then accessing those points based on this sorted order. To speed up the basic outlier detection technique, we develop two distributed algorithms (DOoR and iDOoR) for modern distributed multi-core clusters of machines, connected on a ring topology. The first algorithm passes data blocks from each machine around the ring, incrementally updating the nearest neighbors of the points passed. By maintaining a cutoff threshold, it is able to prune a large number of points in a distributed fashion. The second distributed algorithm extends this basic idea with the indexing scheme discussed earlier. In our experiments, both distributed algorithms exhibit significant improvements compared to the state-of-the-art distributed methods.
Entropy-based Probabilistic Fatigue Damage Prognosis and Algorithmic Performance Comparison
In this paper, a maximum entropy-based general framework for probabilistic fatigue damage prognosis is investigated. The proposed methodology is based on an underlying physics-based crack growth model. V arious uncertainties from measurements, modeling, and parameter estimations are considered to describe the stochastic process of fatigue damage accumulation. A probabilistic prognosis updating procedure based on the maximum relative entropy concept is proposed to incorporate measurement data. Markov Chain Monte Carlo (MCMC) technique is used to provide the posterior samples for model updating in the maximum entropy approach. Experimental data are used to demonstrate the operation of the proposed probabilistic prognosis methodology. A set of prognostics-based metrics are employed to quantitatively evaluate the prognosis performance and compare the proposed method with the classical Bayesian updating algorithm. In particular, model accuracy, precision and convergence are rigorously evaluated in* addition to the qualitative visual comparison. It is shown that the proposed maximum relative entropy methodology has narrower confidence bounds of the remaining life prediction than classical Bayesian updating algorithm.
AMSR-E/Aqua L2B Global Swath Surface Precipitation GSFC Profiling Algorithm V003
The AMSR-E/Aqua Level-2B precipitation product includes instantaneous surface precipitation rate and type over ice-free/snow-free land and ocean between 89.24 degrees north and south latitudes at at 10 km spatial resolution along the track and 5 km spatial resolution along the scan. The data are generated by the GPROF 2010 Version 2 algorithm using Version 3 of the AMSR-E Level-2A Brightness Temperatures.
Multiple Kernel Learning for Heterogeneous Anomaly Detection: Algorithm and Aviation Safety Case Study
The world-wide aviation system is one of the most complex dynamical systems ever developed and is generating data at an extremely rapid rate. Most modern commercial aircraft record several hundred flight parameters including information from the guidance, navigation, and control systems, the avionics and propulsion systems, and the pilot inputs into the aircraft. These parameters may be continuous measurements or binary or categorical measurements recorded in one second intervals for the duration of the flight. Currently, most approaches to aviation safety are reactive, meaning that they are designed to react to an aviation safety incident or accident. In this paper, we discuss a novel approach based on the theory of multiple kernel learning to detect potential safety anomalies in very large data bases of discrete and continuous data from world-wide operations of commercial fleets. We pose a general anomaly detection problem which includes both discrete and continuous data streams, where we assume that the discrete streams have a causal influence on the continuous streams. We also assume that atypical sequences of events in the discrete streams can lead to off-nominal system performance. We discuss the application domain, novel algorithms, and also discuss results on real-world data sets. Our algorithm uncovers operationally significant events in high dimensional data streams in the aviation industry which are not detectable using state of the art methods.
Impact of Input Uncertainty on Failure Prognostic Algorithms: Extending the Remaining Useful Life of Nonlinear Systems
This paper presents a novel set of uncertainty measures to quantify the impact of input uncertainty on nonlinear prognosis systems. A Particle Filtering-based method is also presented that uses this set of uncertainty measures to quantify, in real time, the impact of load, environmen- tal, and other stresses for long-term prediction. Further- more, this work shows how these measures can be used to implement a novel feedback correction loop aimed to suggest modifications, at a system input level, with the purpose of extending the remaining useful life of a faulty nonlinear, non-Gaussian system. The correction scheme is tested and illustrated using real vibration feature data from a fatigue-driven fault in a critical aircraft compo- nent.
A Local Distributed Peer-to-Peer Algorithm Using Multi-Party Optimization Based Privacy Preservation for Data Mining Primitive Computation
This paper proposes a scalable, local privacy-preserving algorithm for distributed peer-to-peer (P2P) data aggregation useful for many advanced data mining/analysis tasks such as average/sum computation, decision tree induction, feature selection, and more. Unlike most multi-party privacy-preserving data mining algorithms, this approach works in an asynchronous manner through local interactions and therefore, is highly scalable. It particularly deals with the distributed computation of the sum of a set of numbers stored at different peers in a P2P network in the context of a P2P web mining application. The proposed optimization-based privacy-preserving technique for computing the sum allows different peers to specify different privacy requirements without having to adhere to a global set of parameters for the chosen privacy model. Since distributed sum computation is a frequently used primitive, the proposed approach is likely to have significant impact on many data mining tasks such as multi-party privacypreserving clustering, frequent itemset mining, and statistical aggregate computation.
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.