Find research datasets worth reusing

opencc-by-4.0Apr 2023View details →

Mapping forests with different levels of naturalness using machine learning and landscape data mining - GRASS GIS DB

The GRASS GIS database containing the input raster layers needed to reproduce the results from the manuscript entitled: "Mapping forests with different levels of naturalness using machine learning and landscape data mining" (under review) Abstract: To conserve biodiversity, it is imperative to maintain and restore sufficient amounts of functional habitat networks. Hence, locating remaining forests with natural structures and processes over landscapes and large regions is a key task. We integrated machine learning (Random Forest) and wall-to-wall open landscape data to scan all forest landscapes in Sweden with a 1 ha spatial resolution with respect to the relative likelihood of hosting High Conservation Value Forests (HCVF). Using independent spatial stand- and plot-level validation data we confirmed that our predictions (ROC AUC in the range of 0.89 - 0.90) correctly represent forests with different levels of naturalness, from deteriorated to those with high and associated biodiversity conservation values. Given ambitious national and international conservation objectives, and increasingly intensive forestry, our model and the resulting wall-to-wall mapping fills an urgent gap for assessing fulfilment of evidence-based conservation targets, spatial planning, and designing forest landscape restoration. This database was compiled from the following sources: 1. HCVF. A database of High Conservation Value Forests in Sweden. Swedish Environmental Protection Agency. source: <a href="https://geodata.naturvardsverket.se/nedladdning/skogliga_vardekarnor_2016.zip">https://geodata.naturvardsverket.se/nedladdning/skogliga_vardekarnor_2016.zip</a> 2. NMD. National Land Cover Data. Swedish Environmental Protection Agency. source: <a href="https://www.naturvardsverket.se/en/services-and-permits/maps-and-map-services/national-land-cover-database/">https://www.naturvardsverket.se/en/services-and-permits/maps-and-map-services/national-land-cover-database/</a> 3. DEM. Terrain Model Download, grid 50+. Lantmateriet, Swedish Ministry of Finance. source: <a href="https://www.lantmateriet.se/en/geodata/geodata-products/product-list/terrain-model-download-grid-50/">https://www.lantmateriet.se/en/geodata/geodata-products/product-list/terrain-model-download-grid-50/</a> 4. GFC. Global Forest Change. Global Land Analysis and Discovery, University of Maryland. source: <a href="https://glad.earthengine.app">https://glad.earthengine.app</a> 5. LIGHTS. A harmonized global nighttime light dataset 1992–2018. Land pollution with night-time lights expressed as calibrated digital numbers (DN). source: <a href="https://doi.org/10.6084/m9.figshare.9828827.v2">https://doi.org/10.6084/m9.figshare.9828827.v2</a> 6. POPULATION. Total Population in Sweden. Statistics Sweden. source: <a href="https://www.scb.se/en/services/open-data-api/open-geodata/grid-statistics/">https://www.scb.se/en/services/open-data-api/open-geodata/grid-statistics/</a>   To learn more about the GRASS GIS database structure, see: <a href="https://grass.osgeo.org/grass82/manuals/grass_database.html">https://grass.osgeo.org/grass82/manuals/grass_database.html</a>

FIG. 5 in Ultrasound-based prediction of preoperative core biopsy categories in solid breast tumor using machine learning.

FIG. 5. Influence of parasite infection on total snail abundance. Number of original and juvenile snails observed in each parasite treatment (absent, present) on experimental day 65. Plotted values are means ± 1 SE. The y-axis is log transformed.

FIG. 2 in Ultrasound-based prediction of preoperative core biopsy categories in solid breast tumor using machine learning.

FIG. 2. Example Northern Leopard Frog metamorphs observed with limb deformities. (Top left) Extra limbs. (Top right) Extra limb and boney triangle. (Bottom left) Boney triangle. (Bottom right) Absent limbs.

FIG. 4 in Ultrasound-based prediction of preoperative core biopsy categories in solid breast tumor using machine learning.

FIG. 4. Influence of treatments on metamorph responses. (A) Proportion of Northern Leopard Frog tadpoles exposed to pesticide treatments (control, atrazine, Bti) and parasite treatments (absent, present) that survived to metamorphosis. (B) Larval period of Northern Leopard Frogs exposed to pesticide treatments (control, atrazine, Bti) and parasite treatments (absent, present). (C) Mass at metamorphosis of Northern Leopard Frogs exposed to pesticide treatments (control, atrazine, Bti) and parasite treatments (absent, present). Plotted values are means ± 1 SE.

FIG. 3 in Ultrasound-based prediction of preoperative core biopsy categories in solid breast tumor using machine learning.

FIG. 3. Influence of pesticide treatment on the proportion of metamorphs with deformities. Proportion of parasitized Northern Leopard Frog metamorphs with deformities exposed to pesticide treatments (control, atrazine, Bti). Plotted values are means ± 1 SE. The gray points indicate proportion of metamorphs deformed within each mesocosm.

FIG. 1 in Ultrasound-based prediction of preoperative core biopsy categories in solid breast tumor using machine learning.

FIG. 1. Effects of parasites on tadpole responses. Mass and Gosner developmental stage for Northern Leopard Frog tadpoles exposed to different parasite treatments (absent, present) for 7 wk. Plotted values are means ± 1 SE.

opencc-by-4.0Jan 2020View details →

Machine learning meets pKa - Datasets

Datesets belonging to the publication: Baltruschat M and Czodrowski P. Machine learning meets pKa [version 2; peer review: 2 approved]. F1000Research 2020, 9(Chem Inf Sci):113 (<a href="https://doi.org/10.12688/f1000research.22090.2">https://doi.org/10.12688/f1000research.22090.2</a>) Corresponding GitHub repository: <a href="https://github.com/czodrowskilab/Machine-learning-meets-pKa">https://github.com/czodrowskilab/Machine-learning-meets-pKa</a>

Mechanism of building a machine learning model and its main features

This figure is a flowchart summarizing the steps required to build a machine learning model in general way.

opencc-zeroMay 2023View details →

dryad32/100

Data from: Integrating machine learning with otolith isoscapes: reconstructing connectivity of a marine fish over four decades

Stable isotopes are an important tool to uncover animal migration. Geographic natal assignments often require categorizing the spatial domain through a nominal approach, which can introduce bias given the continuous nature of these tracers. Stable isotopes predicted over a spatial gradient (i.e., isoscapes) allow a probabilistic and continuous assignment of origin across space, although applications to marine organisms remain limited. We present a new framework that integrates nominal and continuous assignment approaches by (1) developing a machine-learning multi-model ensemble classifier using Bayesian model averaging (nominal); and (2) integrating nominal predictions with continuous isoscapes to estimate the probability of origin across the spatial domain (continuous). We applied this integrated framework to predict the geographic origin of the Northwest Atlantic mackerel (Scomber scombrus), a migratory pelagic fish comprised of northern and southern components that have distinct spawning sites off Canada (northern contingent) and the US (southern contingent), and seasonally overlap in US fished regions. The nominal approach based on otolith carbon and oxygen stable isotopes (δ13C/δ18O) yielded high contingent classification accuracy (84.9%). Contingent assignment of unknown-origin samples revealed prevalent, yet highly varied contingent mixing levels (12.5–83.7%) within the US waters over four decades (1975–2019). Nominal predictions were integrated into mackerel-specific otolith oxygen isoscapes developed independently for Canadian and US waters. The combined approach identified geographic nursery hotspots in known spawning sites, but also detected geographic shifts over multi-decadal time scales. This framework can be applied to other marine species to understand migration and connectivity at high spatial resolution, relevant to management of unit stocks in fisheries and other conservation assessments.

opencc-by-4.0Feb 2023View details →

Supplementary materials for "Machine Learning Electronic Structure Methods Based On The One-Electron Reduced Density Matrix"

Supplementary materials for "Machine Learning Electronic Structure Methods Based On The One-Electron Reduced Density Matrix" by X. Shao, L. Paetow, M. E. Tuckerman and M. Pavanello The QMLearn software A current snapshot of the QMLearn software is available on GitLab at https://gitlab.com/pavanello-research-group/qmlearn. Video tutorials and other examples (including Jupyter Notebooks) are available at http://qmlearn.rutgers.edu Data availability We share all training/test sets and notebooks needed to reproduce Table II and Figures 1-7. For all figures,  we provide Jupyter notebooks and the needed data to exactly reproduce the figures. Trajectory files for all IR spectra are shared. The collection of all materials is available in this dataset. The QMLearn version used for this work is 0.0.1. Specific QMLearn dependencies and their version used for all calculations are listed as follows:  - ase           (3.22.1)  - h5py          (3.7.0)  - numpy         (1.23.1)  - pyscf         (2.0.1)  - scikit-learn  (1.1.1)  - scipy         (1.8.1)

dryad32/100

Predicting species richness and diversity using satellite remote sensing and random forest machine learning algorithm

Aims: Remote sensing approaches could be beneficial for monitoring and compiling essential biodiversity data because it is cost-effective and allows for coverage of large areas over a short period. This study investigated the relationship between multispectral remote sensing data from Landsat 8 and Sentinel 2 and species richness and diversity in mountainous and protected grasslands. Locations: Golden Gate Highlands National Park, Free State, South Africa. Methods: In-situ data of plant species composition and cover from 142 plots with 16 releves each were distributed across the study site and used to calculate species richness and Shannon-wiener species diversity index (species diversity. We used a machine-learning random forest algorithm to optimise the prediction of species richness and diversity. The algorithm was used to identify the optimal spectral bands and vegetation indices for estimating species richness and diversity. Subsequently, the selected bands and vegetation indices were used to estimate species richness through random forest regression. Results: This research found weak relationships between remote sensing vegetation indices and the diversity metrics, but significant relationships were found between some spectral bands and diversity metrics. Moreover, using machine learning random forest, the multispectral datasets exhibited strong predictive powers. In this investigation, for both sensors, near-infrared (NIR) seemed to be the most selected band to explain species diversity in mountainous grasslands. Main conclusions: This finding further ascertains the efficiency of using NIR in vegetation mapping. This research shows that NIR, SAVI and EVI are the most adequate for predicting species richness and diversity in mountainous grasslands with relatively good accuracies.

opencc-zeroMay 2023View details →

Dataset of Flow Velocity Prediction in Vegetated Alluvial Channels Comparing Empirical and State-of-the-art Hybrid Machine Learning Models

We compiled 447 datasets from different sources and lab- and field-based measurements. These datasets included Einstein and Banks (1950), Fenzl (1962), Kouwen et al. (1969), Ree and Crow (1977), Murota (1984), Tsujimoto and Kitamura (1990), Tsujimoto (1991), Tsujimoto (1993), Shimizu (1994), Dunn et al. (1996), Ikeda and Kanazawa (1996), Meijer (1998), Jarvela (2002), Rowinski and Kubrak (2002), Stone and Shen (2002), Poggi et al. (2004), Carollo et al. (2005), and Murphy et al. (2007).

opencc-by-4.0Feb 2023View details →

Minimal dataset for master's dissertation: Updating a conceptual rainfall-runoff model based on radar observation and machine learning

The minimal dataset needed of data which can not be retrieved from the internet by APIs in the preprocessing. Consists of shape, land use and rivers for the Zwalm catchment.   Code related to this dataset can be found here: https://github.com/olivierbonte/master_thesis

Preprocessing output for master's dissertation: Updating a conceptual rainfall-runoff model based on radar observation and machine learning

Outputs of the local preprocessing of the OpenEO data (see <a href="https://doi.org/10.5281/zenodo.7691342">here</a>) and pywaterinfo data (see <a href="https://doi.org/10.5281/zenodo.7689200">here</a>). Code related to this dataset can be found <a href="http://github.com/olivierbonte/master_thesis">here</a>

Inverse observation operator parameters/models for master's dissertation: Updating a conceptual rainfall-runoff model based on radar observation and machine learning

Both saved models and results of hyperparameter tuning are given.  Code related to this dataset can be found <a href="http://github.com/olivierbonte/master_thesis">here</a>

Artifacts for the ISSTA 2023 Paper: An Empirical Study on the Effects of Obfuscation on Static Machine Learning-based Malicious JavaScript Detectors

An Empirical Study on the Effects of Obfuscation on Static Machine Learning-Based Malicious JavaScript Detectors This repository contains the evaluation script and the corresponding data of the ISSTA'23 paper "An Empirical Study on the Effects of Obfuscation on Static Machine Learning-Based Malicious JavaScript Detectors". Abstract Machine learning is increasingly being applied to malicious JavaScript detection in response to the growing number of Web attacks and the attendant costly manual identification. In practice, to hide their malicious behaviors or protect intellectual copyrights, both malicious and benign scripts tend to obfuscate their own code before uploading. While obfuscation is beneficial, it also introduces some additional code features (e.g., dead code) into the code. When machine learning is employed to learn a malicious JavaScript detector, these additional features can affect the model to make it less effective. However, there is still a lack of clear understanding of how robust existing machine learning-based detectors are on different obfuscators. In this paper, we conduct the first empirical study to figure out how obfuscation affects machine learning detectors based on static features. Through the results, we observe several findings: 1) Obfuscation has a significant impact on the effectiveness of detectors, causing an increase both in false negative rate (FNR) and false positive rate (FPR), and the bias of obfuscation in the training set induces detectors to detect obfuscation rather than malicious behaviors. 2) The common measures such as improving the quality of the training set by adding relevant obfuscated samples and leveraging state-of-the-art deep learning models can not work well. 3) The root cause of obfuscation effects on these detectors is that feature spaces they use can only reflect shallow differences in code, not about the nature of benign and malicious, which can be easily affected by the differences brought by obfuscation. 4) Obfuscation has a similar effect on realistic detectors in VirusTotal, indicating that this is a common real-world problem. Getting Started Requirements <pre>install python3 version 3.9.12 pip3 install -r requirements.txt install nodejs install npm npm install escodegen npm install esprima</pre> Step 1: Generating PDGs for JStap <code>cd detectors/jstap/pdg_generation</code> <code>python generate_PDGs.py</code> Step 2: Getting the results for RQ1: What Impact Does Obfuscation Have on Static Machine Learning Malicious JavaScript Detectors? <code>cd RQ1/</code> 1. Detectors Performance on Obfuscated Samples. To train the models: <code>python RQ1_1_train.py</code> To get the results: <code>python RQ1_1_test.py</code> 2. Different Machine Learning Algorithms. To train the models: <code>python RQ1_2_train.py</code> To get the results: <code>python RQ1_2_test.py</code> 3. Biased Training Sets To train the models: <code>python RQ1_3_train.py</code> To get the results: <code>python RQ1_3_test.py</code> All the trained models will be stored in RQ1/models/. All the results will be stored in RQ2/results/. Step 3: Getting the results for RQ2: Are the Common Measures to Mitigate the Impact of Obfuscation Effective? <code>cd RQ2/</code> 1. Training and Testing Detectors on Samples with Same Types of Obfuscation. To train the models: <code>python RQ2_1_train.py</code> To get the results: <code>python RQ2_1_test.py</code> 2. Training and Testing Detectors on Samples with Different Types of Obfuscation. If you follow the steps, the models is already trained. To get the results: <code>python RQ2_2_test.py</code> 3. BERT Variants. To get the results: <code>python RQ2_3.py</code> All the trained models will be stored in RQ2/models/. All the results will be stored in RQ2/results/.   Step 4: Getting the results for RQ3: What Is the Root Cause of Obfuscation Affecting Static Machine Learning Malicious JavaScript Detectors? To get the results of vectors visualization, top ten features, and distances between vectors sets: <code>cd RQ3</code> <code>python visulization.py</code> The figures of vectors visualization will be stored in RQ3/results/. Other results will be shown in the console.   Step 5: Getting the results for RQ4: How Does Obfuscation Affect Real-world Static Malicious JavaScript Detectors? To get the results, submit the sample under the folder samples/ to <a href="https://www.virustotal.com/gui/home/upload">VirusTotal</a> .   Detailed Instructions detectors The detectors under the folder <code>detectors/</code> are the main projects to be evaluated in our paper, which are CUJO, ZOZZLE, JAST, and JSTAP. Detailed setup and usage instructions are described in <code>README.md</code> in the corresponding folder. samples The files under the folder <code>samples/</code> are the samples from a random tenth of our dataset used in our paper. Results can be obtained quickly using these samples. These results will not be exactly the same as in the paper, but they are similar. RQ1 The code under folder <code>RQ1/</code> is to figure out how obfuscation affects these detectors. <code>RQ1_1_train.py</code> is to train four detectors with unobfuscated samples. <code>RQ1_1_test.py</code> tests these trained detectors with unobfuscated and obfuscated samples. <code>RQ1_2_train.py</code> is to train the detector ZOZZLE that uses different machine learning algorithms. <code>RQ1_2_test.py</code> tests these trained models with unobfuscated and obfuscated samples. <code>RQ1_3_train.py</code> uses a training set with all unobfuscated benign samples and all obfuscated malicious samples, and a training set with all obfuscated benign samples and all unobfuscated malicious samples to train the detectors. <code>RQ1_3_test.py</code> uses these detectors to detect unobfuscated benign samples, obfuscated benign samples, unobfuscated malicious samples, and obfuscated malicious samples, respectively. RQ2 The code under folder <code>RQ2/</code> is to study the two measures to mitigate the impact of obfuscation effective or not. <code>RQ2_1_train.py</code> uses obfuscated samples to train four detectors. <code>RQ2_1_test.py</code> tests these detectors on the same type of obfuscated samples. <code>RQ2_2_test.py</code> tests thest detectors on the different type of obfuscated samples. <code>RQ2_3.py</code> uses the BERT variants to generate code representation of unobfuscated samples, trains the detector with these code representations, and tests the trained detectors with code representations of obfuscated samples. RQ3 The code unser fodler <code>RQ3/</code> visualizes the vectors, extracts the ten most important features, and calculates the distance between different sets of vectors. RQ4 There is no code related to RQ4 here because the actual operation of RQ4 is to submit the samples to <a href="https://www.virustotal.com/gui/home/upload">VirusTotal</a> .   The whole dataset is available at <a href="https://drive.google.com/file/d/1a7pNUwzikiJyY9L7dIu53I6_MR0oDpgi/view?usp=sharing." target="_blank" rel="noopener">https://drive.google.com/file/d/1a7pNUwzikiJyY9L7dIu53I6_MR0oDpgi/view?usp=sharing.</a>   Cite this work <pre>@inproceedings{staticanalysis, author   = {Kunlun Ren, Qiang Weizhong, Yueming Wu, Yi Zhou, Deqing Zou, Hai Jin}, title     = {An Empirical Study on the Effects of Obfuscation on Static Machine Learning-Based Malicious JavaScript Detectors}, booktitle = {Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA'23)}, year     = {2023} }</pre>

Img2brain: Predicting the neural responses to visual stimuli of naturalistic scenes using machine learning

The data for this project is part of the <a href="https://doi.org/10.1038/s41593-021-00962-x">Natural Scenes Dataset</a> (NSD), a massive dataset of 7T fMRI responses to images of natural scenes coming from the <a href="https://cocodataset.org/#home">COCO dataset</a>. The training dataset consists of brain responses measured at 10.000 brain locations (voxels) to 8857 images (in jpg format) for one subject. The 10.000 voxels are distributed around the visual pathway and may encode perceptual and semantic features in different proportions. The test dataset comprises 984 images (in jpg format), and the goal is to predict the brain responses to these images. The zip file contains the following folders: 1. trainingIMG: contains the training images (8857) in jpg format. The numbering corresponds to the order of the rows in the brain response matrix. 2. testIMG: contains test images (984) in jpg format. 3. trainingfMRI: contains a npy file with the fMRI responses measured at 10000 brain locations (voxels) to the training images. The matrix has 8857 rows (one for each image) and 10000 columns (one for each voxel).

opencc-by-4.0Jun 2023View details →

Datasets used in "Machine Learning and VIIRS Satellite Retrievals for Skillful Fuel Moisture Content Monitoring in Wildfire Management"

Data sets used to train, validate, and test machine learning models for the prediction of 10-hour dead fuel moisture content. The data sets are for 375 m and 1 km resolutions. MADIS climatography data is also included. The datasets may be processed using the code supplied at https://github.com/NCAR/fmc_viirs.