Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
1,481
datasets available to search
ShareScore release 0.9.0
Dataset results
1,481 results for “data processing”
Data for "Bulk and fracture process zone contribution to the rate-dependent adhesion amplification in viscoelastic broad-band materials"
<p>This dataset contains all the data generated for the publication</p> <p>[1] A. Maghami, Q. Wang, M. Tricarico et al., Bulk and fracture process zone contribution to the rate-dependent adhesion amplification in viscoelastic broad-band materials. Journal of the Mechanics and Physics of Solids (2024), doi:<br>https://doi.org/10.1016/j.jmps.2024.105844.</p> <p><br>The provided data are those which appear in the figures of Ref. [1]. Data are stored using informative named structures in a ".mat" file. The data are easily accessible through the Commercial Software MATLAB (© 1994-2023 The MathWorks, Inc.) or by using the free software GNU Octave. An exemplary code for loading and plotting the data contained in "data_Sphere_Viscoelastic_BroadBand.mat" is the following:</p> <p>%%%%%%%%%%%%%</p> <div> <div> <div> <div>clc</div> <div>clearvars</div> <div> </div> <div>close all</div> <div> </div> <div> </div> <div>% Create a new figure</div> <div>load('data_Sphere_Viscoelastic_BroadBand.mat');</div> <div> </div> <div> </div> <div>figure</div> <div> </div> <div>% Choose one of the following names:</div> <div>% Fig_3_a, Fig_3_b, Fig_3_c,</div> <div>% Fig_4, Fig_5_a, Fig_5_b, Fig_5_b_inset,</div> <div>% Fig_6, Fig_6_inset, Fig_7, Fig_7_inset,</div> <div>% Fig_8_a, Fig_8_b, Fig_10_a, Fig_10_b</div> <div> </div> <div>name=Fig_3_a;</div> <div>% Note: some plots in the manuscript are in log space.</div> <div> </div> <div>% Plot the data from the structure</div> <div> </div> <div> </div> <div>for i = 1:numel(name.x) hold on</div> <div> </div> <div>x_cell = name.x(i);</div> <div>y_cell = name.y(i);</div> <div>x = cell2mat(x_cell);</div> <div>y = cell2mat(y_cell);</div> <div> </div> <div>plot(x, y);</div> <div>end</div> <div> </div> <div> </div> <div> </div> </div> %%%%%%%%%%%%%</div> </div> <p>The code can be easily adapted for plotting the curves of all the figures contained in Ref. [1]. </p>
Exploring deep learning models for 4D-STEM-DPC data processing
<p>This repository contains scanning transmission electron microscopy data and processing files used in the journal publication <strong>"Exploring deep learning models for 4D-STEM-DPC data processing"</strong>. DOI: <a href="https://doi.org/10.1016/j.ultramic.2024.114058">10.1016/j.ultramic.2024.114058</a></p> <p><strong>Prerequisites</strong></p> <p>The scripts presented below require certain open-source Python packages to run. Library versions used to run the scripts are:</p> <ul> <li>hyperspy 1.7.1</li> <li>pyxem 0.14.2</li> <li>fpd 0.2.5</li> <li>pytorch 1.12.1 (cudatoolkit 11.6.0)</li> <li>jupyterlab 4.0.7</li> </ul> <p><strong>Data files</strong></p> <p>Three zipped folders are included. Two of them contain the training- and inference data for the neural networks, aptly named <em>training_data.zip</em> and <em>inference_data.zip</em>. PyTorch state dictionaries for trained models are included in the <em>models.zip</em> folder.</p> <p><strong>Processing scripts</strong></p> <p>All scripts are included in an IPython notebook format (.ipynb extension). The notebooks <em>Segmentation.ipynb</em> and <em>Regression.ipynb</em> contain the code for training and inference of the segmentation and regression models, respectively. The <em>Training_data_creation.ipynb<strong> </strong></em>notebook contains the code to preprocess the training data for both neural network models. The <em>Standard_algorithms.ipynb</em> notebook has the code for doing center of mass and edge filtering/disc detection algorithms for STEM-DPC processing.</p>
Data Access Made Easy: flexible, on the fly data standardization and processing (for research automatic weather stations)
<pre>Automatic Weather Stations (AWS) deployed in the context of research projects provide very valuable data thanks to the flexibility they offer in term of measured meteorological parameters, choice of sensors and quick deployment and redeployment. However this flexibility is a challenge in terms of metadata and data management. Traditional approaches based on networks of standard stations can not accommodate these needs and often no tools are available to manage these research AWS, leading to wasted data periods because of difficult data reuse, low reactivity in identifying potential measurement problems, and lack of metadata to document what happened. The Data Access Made Easy (DAME) effort is our answer to these challenges. At its core, it relies on the mature and flexible open source MeteoIO meteorological pre-processing library. It was originally developed as a flexible data processing engine for the needs of numerical models consuming meteorological data and further developed as a data standardization engine for the Global Cryosphere Watch (GCW) of the World Meteorological Organization (WMO). For each AWS, a single configuration file describes how to read and parse the data, defines a mapping between the available fields and a set of standardized names and provides relevant Attribute Conventions Dataset Discovery (ACDD) metadata fields, if necessary on a per input file basis. Low level data editing is also available, such as excluding a given sensor, swapping sensors or merging data from another AWS, for any given time period. Moreover an arbitrary number of filters can be applied on each meteorological parameter, restricted to specific time periods if required. This allows to describe the whole history of an AWS within a single configuration file and to deliver a single, consistent, standardized output file possibly spanning many years, many input data files and many changes both in format and available sensors. Finally, all configuration files are versionned in order to document their history. A web interface has been developed that allows data owners to manage the configuration files for their stations, refresh their data at regular intervals, inspect the data QA log files and allow on-demand data generation. The same interface allows other users to request data on-demand for any time period.<br><br>This presentation and software has received funding from the World Meteorological Organization under grant agreement No. 29539/2022-1.9 as well as the European Union’s Horizon 2020 research and innovation program under grant agreement No. 101003472 (Arctic Passion). It has also been supported by the WSL/SLF over many years and projects.</pre>
The data for Radar Circular Polarization Ratio of Near-Earth Asteroids: Links to Spectral Taxonomy and Surface Processes
<p>README</p> <p>% =========================================================================<br>% Project: "Radar Circular Polarization Ratio of Near-Earth Asteroids: Links to <br>% Spectral Taxonomy and Surface Processes"<br>% Author: Edgard G. Rivera-Valentín<br>% Institution: Johns Hopkins University Applied Physics Laboratory<br>% Email: edgard.rivera-valentin@jhuapl.edu<br>% ORCID: 0000-0002-0786-7307<br>% Date: 18 September 2024<br>% =========================================================================</p> <p>% =========================================================================<br>% Licenses:<br>% Any software provided in this repository is licenced under the MIT License, detailed below and within <br>% this archive. <br>% Any data provided in this repository is licenced under Creative Commons Attribution 4.0 International, <br>% detailed within this archive. <br>%<br>% MIT License<br>%<br>% Copyright (c) 2024 The Johns Hopkins University Applied Physics Laboratory LLC<br>%<br>% Permission is hereby granted, free of charge, to any person obtaining a copy<br>% of this software, data, and associated documentation files (the "Software"), to deal<br>% in the Software without restriction, including without limitation the rights<br>% to use, copy, modify, merge, publish, distribute, sublicense, and/or sell<br>% copies of the Software, and to permit persons to whom the Software is<br>% furnished to do so, subject to the following conditions:<br>%<br>% The above copyright notice and this permission notice shall be included in<br>% all copies or substantial portions of the Software.<br>%<br>% THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR<br>% IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,<br>% FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE<br>% AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER<br>% LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,<br>% OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN<br>% THE SOFTWARE.<br>%<br>% ADDITIONAL LICENSING INFORMATION:<br>% Any reuse of the figures and data provided in this repository must abide by <br>% the Copyright policy of the American Astronomical Society Journals:<br>% https://journals.aas.org/article-charges-and-copyright/#AAS_material<br>% =========================================================================</p> <p>% =========================================================================<br>% Description:<br>% This is a data repository for the manuscript published in the Planetary<br>% Science Journal:<br>% Title = Radar Circular Polarization Ratio of Near-Earth Asteroids: Links <br>% to Spectral Taxonomy and Surface Processes<br>% Please see the Zenodo metadata for detailed publicatoin information.<br>% This repository includes:<br>% (1) A csv file that has the table of data used in the published work. The<br>% table includes the data for each near-Earth asteroid used in the<br>% anlaysis. The header information includes: Asteroid Number, Designation<br>% or Name, Taxonomic Type (Following the Bus-DeMeo classification system),<br>% CPR (where CPR is circular polarization ratio), CPR uncertainty (where<br>% the uncertainty is the 1-sigma value), a (where a is semi-major axis in<br>% au), q (where q is the perihelion distance in au), Q (where Q is the<br>% aphelion distance in au), P (where P is the rotational period in hours),<br>% Hmag (where Hmag is the absolute magnitude), and Orbital Class. <br>% (2) A .mrt file that contains the same data as the csv file, but formatted<br>% according to the specifications of the machine-readable table format<br>% used by the AAS Journals. The .mrt version will also be published with<br>% the final PSJ article.<br>% (3) .mat files containing the output from the statistical modeling<br>% presented in the paper. These files are required to run the .m file<br>% included in this repository.<br>% (4) Make Figures.m, which is a Matlab code that remakes all the figures<br>% presented in the paper given the data in each of the .mat files. <br>% (5) The .png files for each of the figures presented in the paper. <br>% =========================================================================</p> <p>% =========================================================================<br>% File Formats:<br>% This archive includes various file formats. <br>% .csv file is a text file format that uses commas to separate values, and<br>% newlines to separate records. <br>% .mrt file is an ASCII byte-by-byte format that is documented here:<br>% https://journals.aas.org/mrt-overview/<br>% and readable by tools such as astropy, TOPCACT, etc. <br>% .m file is a simple text file used by Matlab, it can be opened by any<br>% text editor and executed by Matlab. <br>% .mat file is the file format used by MATLAB for saving data. It can be<br>% ready by other software, such as python, e.g., <br>% https://docs.scipy.org/doc/scipy/reference/generated/scipy.io.loadmat.html<br>% .png file is short for Portable Network Graphic, which is a type of<br>% raster image file. <br>% =========================================================================</p>
Data for "Using physics-informed neural networks to predict the lifetime of laser powder bed fusion processed 316L stainless steel under multiaxial low-cycle fatigue loading"
<p>Title of dataset: Data for "Using physics-informed neural networks to predict the lifetime of laser powder bed fusion processed 316L stainless steel under multiaxial low-cycle fatigue loading".</p> <p>Name/institution/contact information: Dr. Michal Bartošák, Czech Technical University in Prague - Faculty of Mechanical Engineering, email: michal.bartosak@fs.cvut.cz.</p> <p>Date of data collection: The data were collected between 2021 and 2024.</p> <p>File name structure: The data consists of two files: "316L_fatigue_and_defects.xls," which contains fatigue lifetime data and defect characteristics, and an associated description file, "read_me.txt."</p> <p>See "https://doi.org/10.1016/j.ijfatigue.2024.108608" for the associated article and a detailed description of the methods.</p>
Data and Processing from "Carbon-centric dynamics of Earth's marine phytoplankton"
<div><strong>Brief Summary:</strong></div> <div>This documentation is for associated data and code for: </div> <div>A. Stoer, K. Fennel, Carbon-centric dynamics of Earth's marine phytoplankton. Proceedings of the National Academy of Sciences (2024).</div> <div> </div> <div>To cite this software and data, please use:</div> <div> <div>A. Stoer, K. Fennel, Data and processing from "Carbon-centric dynamics of Earth's marine phytoplankton". Zenodo. <a href="https://doi.org/10.5281/zenodo.10949682" target="_blank" rel="noopener">https://doi.org/10.5281/zenodo.10949682</a>. Deposited 1 October 2024.</div> </div> <div> </div> <div><strong>List of folders and subfolders and what they contain:</strong></div> <div> <ol> <li>raw data: Contains raw data used in the analysis. This folder does not contain the satellite imagery, which will need to be downloaded from the NASA Ocean Color website (https://oceancolor.gsfc.nasa.gov/). <ol> <li>bgc-argo float data (subfolder): Includes Argo data from its original source or put into a similar Argo format</li> <li>global region data (subfolder): Includes data used to subset the Argo profiles into each 10deg lat region and basin.</li> <li>graff et al 2015 data (subfolder): Include the data digitized from Graff et al.'s Fig. 2.</li> </ol> </li> <li>processed data: data processing by this study (Stoer and Fennel, 2024) <ol> <li>processed bgc-argo data (subfolder): A binned processed file is present for each Argo float used in the analysis. Note these files include those describe in Table S1 (these are later processed in "3_stock_bloom_calc.py")</li> <li>processed satellite data (subfolder): includes a 10-deg latitude averaged for each satellite image processed (called "chl_sat_df_merged.csv"). This is later used to calculate a satellite chlorophyll-a climatology in "3_stock_bloom_calc.py".</li> <li>processed chla-irrad data (subfolder): includes the quality-controlled light diffuse attenuation data coupled with the chlorophyll-a fluorescence data to calculate slope factor corrections (the file is called "processed chla-irrad data.csv").</li> <li>processed topography data (subfolder): includes smoothed topography data (file named "ETOPO_2022_v1_60s_N90W180_surface_mod.tiff").</li> </ol> </li> <li>software: <ol> <li>0_ftp_argo_data_download.py: This program downloads the Argo data from the Global Data Assembly Center's FTP. Running this program will provide new Argo float profiles. However, there will be new floats and profiles present if downloaded. This will not match the historical record of Argo floats used in this analysis but could be useful for replicating this analysis when more data becomes available. The historical record of BGC-Argo floats are present in "/raw data/bgc-argo float data/" path. If you wish to downloaded other float data, see Gordon et al. (2020), Hamilton and Leidos (2017) and the data from the misclab website (https://misclab.umeoce.maine.edu/floats/).</li> <li>1_argo_data_processing.py: This program quality-controls and bins the biogeochemical data into a consistent format. This includes corrections and checks, like the spike/noise test or the non-photochemical quenching correction.</li> <li>2_sat_data_processing.py: this program processes the satellite data downloaded from the NASA Ocean Color website.</li> <li>3_stock_bloom_calc.py: this is the main program used to described the results of the study. The program takes the processed Argo data and groups it into regions and calculates slope factors, phytoplankton carbon & chlorophyll-a, global stocks, and bloom metrics.</li> <li>4_stock_calc_longhurst_province.py: This program repeats the global stocks calculations performed in "3_stock_bloom_calc.py" but bases the grouping on Longhurst Biogeochemical Provinces.</li> </ol> </li> </ol> </div> <div><strong>How to Replicate this Analysis:</strong></div> <div>Each program should be run in the order listed above. Path names where the data files have been downloaded will need to be updated in the code.</div> <div> </div> <div>To use the exact same Sprof files as used in the paper, skip running "0_ftp_argo_data_download.py" and start with "1_argo_data_processing.py" instead. Use the float data from the folder "bgc-argo float data". The program "0_ftp_argo_data_download.py" downloads the latest data from Argo database, so it is useful for updating the analysis. The program "1_argo_data_processing.py" may also be skipped to save time and the processed BGC-Argo float data may be used instead (see folder named "processed bgc-argo data"). </div> <div> </div> <div>Similarly, the program "2_sat_data_processing.py" may also be skipped, which otherwise can take multiple hours to process. The raw data is available from the NASA Ocean Color website (<a href="https://oceancolor.gsfc.nasa.gov/">https://oceancolor.gsfc.nasa.gov/</a>). The processed data from "2_sat_data_processing.py" is available so this step may be skipped to save time as well.</div> <div> </div> <div>The program "3_stock_bloom_calc.py" will require running "ocean_toolbox.py" (see below) in another tab. The portion of the program that involves QC for the irradiance profiles has been commented out to save processing time, and the pre-processed data used in the study has been linked instead (see folder "processed light data"). Similarly, pre-processed topography data is present in this repository. The original Earth Topography data can be accessed at <a href="https://www.ncei.noaa.gov/products/etopo-global-relief-model">https://www.ncei.noaa.gov/products/etopo-global-relief-model.</a></div> <p> </p> <p>A version of "3_stock_bloom_calc.py" using Longhurst provinces is available for exploring alternative groupings and their effects on stock calculations. See the program named "4_stock_calc_longhurst_province.py". You will need to download the Longhurst biogeochemical provinces from <a href="https://www.marineregions.org/">https://www.marineregions.org/</a>.</p> <p>To explore the effects of different slope factors, averaging methods, bbp spectral slopes, etc, the user will likely want to make changes to "3_stock_bloom_calc.py". Please do not hesitate to contact the correponding author (Adam Stoer) for guidance or questions.</p> <p><strong>ocean_toolbox.py:</strong></p> <p>import statsmodels.formula.api as smf<br>import os<br>import matplotlib.pyplot as plt<br>import numpy as np<br>from uncertainties import unumpy as unp<br>from scipy import stats</p> <p>def file_grab(root,find,start): #grabs files by file extensions and location<br> filelst = []<br> for subdir, dirs, files in os.walk(root):<br> for file in files:<br> filepath = subdir + os.sep + file<br> if filepath.endswith(find):<br> if filepath.startswith(start):<br> filelst.append(filepath)<br> return filelst</p> <p>def sep_bbp(data, name_z, name_chla, name_bbp):<br> <br> '''<br> data: Pandas Dataframe containing the profile data<br> name_z: name of the depth variable in data<br> name_chla: name of the chlorophyll-a variable in data<br> name_bbp: name of the particle backscattering variable in data <br> <br> returns: the data variable with particle backscattering partitioned into <br> phytoplankton (bbpphy) and non-algal particle components (bbpnap).<br> '''<br> #name_chla = 'chla'<br> #name_z = 'depth'<br> #name_bbp = 'bbp470'<br> dcm = data[data.loc[:,name_chla]==data.loc[:,name_chla].max()][name_z].values[0] # Find depth of deep chla maximum<br> part_prof = data[(data.loc[:,name_bbp]<np.median(data.loc[:,name_bbp]))] # find median bbp of profile<br> <br> mod = smf.quantreg('bbp470 ~ ' + str(name_z), <br> part_prof).fit(q=0.01) # Find model to 1 percentile<br> y_pred = mod.predict(part_prof.loc[:,name_z]) # Create predicted bbp_nap<br> <br> part_prof.loc[:,'bbp_back'] = y_pred.values # Predicted bbp NAP from linear trend<br> z_lim = part_prof.loc[(part_prof.loc[:,'bbp_back'].div(part_prof.loc[:,name_bbp])>=1), name_z].min() <br> <br> # Find depth where bbp NAP and bbp intersect<br> data.loc[data[name_z]>=z_lim, 'bbp_back'] = data.loc[data[name_z]>=z_lim, name_bbp].tolist()<br> data.loc[data[name_z]<z_lim,'bbp_back'] = data.loc[data[name_z]==z_lim, name_bbp].values[0] #data.loc[data[name_z]<z_lim, name_z].mul(lr.slope).add(lr.intercept)<br> <br> <br> data.loc[:,'bbpphy'] = data.loc[:, name_bbp].sub(data.loc[:,'bbp_back']) # Subtract bbp NAP from bbp for bbp from phytoplankton<br> data.loc[(data['bbpphy']<0)|(data['depth']>z_lim),'bbpphy'] = 0 # Subtract bbp NAP from bbp for bbp from phytoplankton</p> <p> return data['bbpphy'], z_lim</p> <p>def bbp_to_cphy(bbp_data, sf):<br> <br> '''<br> data: Pandas Dataframe containing the profile data<br> name_bbp: name of the particulate backscattering variable in data<br> name_bbp_err: name of particulate backscattering error variable in data<br> <br> returns: the data variable with particle backscattering [/m] converted into<br> phytoplankton carbon [mg/m^3].<br> '''<br> <br> cphy_data = bbp_data.mul(sf) </p> <p> return cphy_data</p>
Raw and processed data, gating strategy, and photographs of instrumental setup of NAVETTA
<p>Supplementary Information and Raw Data for Weiss et al., Comp Struct Biotechn J: Nanosci Adv Mat, 2024</p> <p>1. xls sheet of raw and processed data for all figures</p> <p>2.-3. gating strategies for flow cytometry experiments</p> <p>4.-8. photographs of instrumental setups</p>
Processed data for "Characterising the evolutionary dynamics of cancer proliferation in single-cell clones with SPRINTER"
<p>This dataset contains the processed data for the figures and analyses performed in the publication "Characterising the evolutionary dynamics of cancer proliferation in single-cell clones with SPRINTER" from Lucas O., Ward S., Zaidi R., Bunkum A., ..., Zaccaria S. Nature genetics, in press, 2024.</p> <p>The processed data are separated into three respective folders:</p> <ul> <li>GT contains all the data related to the analysis of the generated ground truth datasets;</li> <li>NSCLC contains all the data related to the analysis of the NSCLC dataset;</li> <li>TNBC_HGSC contains all the data related to the analysis of the TNBC and HGSC datasets. </li> </ul>
Questionnaires answers and data processing for a mixed-presence user study with two wall-sized displays
<p>Questionnaire answers and data processing tabs that was part of a mixed-presence experiment with two wall-sized displays.<br>Was used for a study in Q4 2023. Accompanies a paper.<br>Complements the protocol for that study that can be found at https://zenodo.org/doi/10.5281/zenodo.12663837 and contains answers for the questionnaires that can be found at https://zenodo.org/doi/10.5281/zenodo.12664007</p>
Processed data used for JGR publication "Role of Midwater Mixed Waves in the Loop Current Separation Events from A Coupled Ocean-Atmosphere Regional Model and In-Situ Observations"
<p>This is the processed dataset used in the JGR publication "Role of Midwater Mixed Waves in the Loop Current Separation Events from A Coupled Ocean-Atmosphere Regional Model and In-Situ Observations" by Xiao Ge.</p> <p>Please contact the author (gexiao@tamu.edu) for all the original/processed outputs of R-CESM, and use the following original papers as citations.</p> <p>The dataset used in this research includes:</p> <p>1. Loop Current Dynamics 2009-2011: LC_*.nc is the processed (reorganized) data for each in-situ station, * represents their station ID</p> <ul> <li>https://digital.library.unt.edu/ark:/67531/metadc955416/</li> <li>https://www.sciencedirect.com/science/article/pii/S0377026516301348?via%3Dihub</li> <li>https://search.dataone.org/view/%7BBD2513E6-3B34-4B7C-BCB9-3C4ED5E8D0FB%7D</li> </ul> <p>2. Regional Community Earth System Model, R-CESM: <a href="https://zenodo.org/api/records/13932074/draft/files/h.nc/content" target="_blank" rel="noopener noreferrer">h.nc</a> is the bathymetry data of R-CESM; cmpr_*.nc files are provided as examples of the original R-CESM outputs; pvsf_prho_*.nc are the processed (subsampled at the target region and interpolated on potential density layers, derived stream function, potential vorticity, and relative vorticity) R-CESM outputs used in this research; and <a href="https://zenodo.org/uploads/13932074" target="_blank" rel="noopener noreferrer">LC_pv_40hlp_2013.nc</a> is the example of organized processed R-CESM (pvsf_prho_*.nc files) containing potential vorticity and relative vorticity for figures plotting</p> <ul> <li>https://journals.ametsoc.org/view/journals/bams/102/9/BAMS-D-20-0024.1.xml?tab_body=fulltext-display</li> </ul> <p> </p> <p> </p> <p> </p> <p> </p>
Low-Range FMCW Insect Radar - Lab Experiments Data, Results, and Data Processing Software
<p>This research was funded by the Hessian State Ministry for Higher Education, Research and the Arts, Germany, as part of the LOEWE priority project Nature 4.0—Sensing Biodiversity.</p>
Supplementary data for "Deep learning for industrial processes: Forecasting amine emissions from a carbon capture plant"
<p>A preliminary analysis of the data already has been discussed in <a href="https://dx.doi.org/10.2139/ssrn.3812299">10.2139/ssrn.3812299</a>.</p> <p><strong>Raw data</strong></p> <p>Raw measurement data is in the Excel files `day*_raw.xlsx`.</p> <p><strong>Model</strong></p> <p>Covariate and label scaler objects are serialized in joblib format in the following files:</p> <ul> <li>20210812_y_transformer_co2_ammonia_reduced_feature_set</li> <li>20210812_y_transformer__reduced_feature_set</li> <li>20210812_x_scaler_reduced_feature_set</li> </ul> <p>Checkpoints of the models are in the `*.pth.tar` files. An example for loading the models is:</p> <pre><code class="language-python">from pyprocessta.model.tcn import TCNModelDropout model_cov = TCNModelDropout( input_chunk_length=8, output_chunk_length=1, num_layers=5, num_filters=16, kernel_size=6, dropout=0.3, weight_norm=True, batch_size=32, n_epochs=100, log_tensorboard=True, optimizer_kwargs={"lr": 2e-4}, ) model_cov.load_from_checkpoint('20210814_2amp_pip_model_reduced_feature_set_darts')</code></pre> <p>which assumes that the checkpoints are placed as `model_best.pth.tar` in a folder called `20210812_2amp_pip_model_reduced_feature_set_darts`.</p> <p> </p>
Data from: Fractal triads efficiently sample ecological diversity and processes across spatial scales
<p>The relative influence of ecological assembly processes, such as environmental filtering, competition, and dispersal, vary across spatial scales. Changes in phylogenetic and taxonomic diversity across environments provide insight into these processes, however, it is challenging to assess the effect of spatial scale on these metrics. Here, we outline a nested sampling design that fractally spaces sampling locations to concentrate statistical power across spatial scales in a study area. We test this design in northeast Utah, at a study site with distinct vegetation types (including sagebrush steppe and mixed conifer forest), that vary across environmental gradients. We demonstrate the power of this design to detect changes in community phylogenetic diversity across environmental gradients and assess the spatial scale at which the sampling design captures the most variation in empirical data. We find clear evidence of broad-scale changes in multiple features of phylogenetic and taxonomic diversity across aspect. At finer scales, we find additional variation in phylodiversity, highlighting the power of our fractal sampling design to efficiently detect patterns across multiple spatial scales. Thus, our fractal sampling design and analysis effectively identify important environmental gradients and spatial scales that drive community phylogenetic structure. We discuss the insights this gives us into the ecological assembly processes that differentiate plant communities found in northeast Utah.</p>
Binary Classification as a Phase Separation Process (data repository)
<p><strong>For version 0.0.2 (from 2021) see below:</strong></p> <p>This is a data repository for the paper "Binary classification as a phase separation process", by Rafael Monteiro.</p> <ul> <li>Website with description of this project: <a href="https://rafael-a-monteiro-math.github.io/Binary_classification_phase_separation/index.html">https://rafael-a-monteiro-math.github.io/Binary_classification_phase_separation/index.html</a></li> <li>Github: <a href="https://github.com/rafael-a-monteiro-math/Binary_classification_phase_separation">https://github.com/rafael-a-monteiro-math/Binary_classification_phase_separation</a></li> </ul> <p>This is a second version, which I wrote using tensorflow. It is much smaller (5 Gb when decompressed), a remarkable improvement when compared to the more than 100 Gb of the previous version).</p> <p>The new files are </p> <ul> <li>PSBC_BCs.tar.gz</li> <li>PSBC_classifier_PCA.tar.gz</li> <li>PSBC_dataset.tar.gz</li> <li>PSBC_libs_grids_statistics.tar.gz</li> <li> PSBC_notebooks.tar.gz</li> </ul> <p>Their content is explained in the file README_v2.pdf</p> <p><strong>UPDATE: <a href="https://drive.google.com/drive/folders/18l_92HuHDWJDkZnvXRuyGedcyC_3YZ2M?usp=sharing">a Google Colab folder is also available</a>. You can also find all the data and libraries there, unpacked.</strong></p> <p>For usage, see the Git-hub. </p> <blockquote> <p><strong>NOTE)</strong> I will keep the content for the previous version available in my Github as well. It is still a "nice exercise" to do all that is done in this new version in numpy, as done there. <strong><em>(Or, I should say, they should be studied as a cautionary tale of what to avoid.)</em></strong></p> </blockquote> <p> </p> <p><strong>For version 0.0.1 (from 2020) see below:</strong></p> <p>This is a data repository for the paper "Binary classification as a phase separation process", by Rafael Monteiro.</p> <ul> <li>Website with description of this project: <a href="https://rafael-a-monteiro-math.github.io/Binary_classification_phase_separation/index.html">https://rafael-a-monteiro-math.github.io/Binary_classification_phase_separation/index.html</a></li> <li>Github: <a href="https://github.com/rafael-a-monteiro-math/Binary_classification_phase_separation">https://github.com/rafael-a-monteiro-math/Binary_classification_phase_separation</a></li> </ul> <p>Therein you will find</p> <ul> <li>Examples</li> <li>1D toy model examples</li> <li>Computational statistics</li> <li>Several trained PSBC on MNIST dataset, with different parameter configurations</li> <li>Extra simulations, investigating normalization properties, low dimensional models that fail due to "too much" model compression, and comparison among ANNs, KNNs, and the PSBC in 1D</li> </ul> <p>If you want to know</p> <ol> <li>how to read the data</li> <li>how to access computational statistics, raw data, and examples</li> <li>how to use the data stored in this data repository</li> </ol> <p>see the guide README.pdf on GitHub page at <a href="https://github.com/rafael-a-monteiro-math/Binary_Classification_Phase_Separation">Binary_Classification_Phase_Separation</a>, where a script that downloads (and organizes) all this data is also available ("download_PSBC.sh).</p> <p>I did not include a copy of the train-test set (0-1dubset of the MNIST database) in every folder with simulations. But you can find a copy of the normalized dataset in the tar ball "PSBC_Examples.tar.gz" as</p> <p>data_test_normalized_MNIST.csv and data_train_normalized_MNIST.csv.</p> <p> </p>
HYPERCOG_Process data_Welding log_2020-07-30
<p>The HyperCOG project addresses the full digital transformation of process industry through an innovative Industrial Cyber-Physical System and Data Analytics. It is based on advanced technologies that enable the development of a hyperconnected network of digital nodes. The nodes can catch outstanding streams of data in real-time, which together with the high computing capabilities, provide sensing, knowledge and cognitive reasoning, making companies robust in the face of variant scenarios. The breaking-edge system proposed in this work is validated on productivity, environmental and replicability aspects on three use cases of three di_erent sectors: steel, cement and chemical.</p> <p>Participating entities: LORTEK. The data was gathered and used in the proof of concept of the architecture introduced in the paper in the way that is described in it.</p> <p><strong>Dataset 1: Welding process data (xlsx files)</strong></p> <p>Real time data of the process of a welding cell on Excel sheets. The data is obtained at a frequency of 100 Hz and variables of voltage, current, temperature, gas flux, etc. are registered in the Excel file by rows.</p> <p><strong>Dataset 2: Temperature and movement of the piece constructed (zip files)</strong></p> <p>Data of temperatures obtained by thermocouple sensors and distortion of the structure measured by a laser sensor. The zip files contain coma separated values of 8 thermocouples welded to the substrate of the piece constructed by the welding cell. The reading of a laser sensor is also recorded along the x coordinate of the movement of the robot for synchronization purposes.</p> <p>The article corresponding to these datasets are available in open access in </p> <pre><a href="https://doi.org/10.5281/zenodo.5533904">https://doi.org/10.5281/zenodo.5533904</a></pre>
Data for: Analysis of Conformational Exchange Processes using Methyl-TROSY-Based Hahn Echo Measurements of Quadruple-Quantum Relaxation
<p>Raw experimental data used in associated publication. A full list of experiments is provided in the README.md file.</p>
Datasets and OpenLCA foreground data processes for the article: Understanding environmental trade-offs and resource demand of direct air capture technologies through comparative life-cycle assessment
<p>This data set contains the supplementary data sets (1-3) and exported foreground data processes from OpenLCA for the manuscript “Understanding environmental trade-offs and resource demand of direct air capture technologies through comparative life-cycle assessment”, submitted to Nature Energy.</p> <p>This repository contains:</p> <ul> <li>Supplementary data set 1: Ancillary calculations and numerical values for HT-Aq DAC</li> <li>Supplementary data set 2: Ancillary calculations and numerical values for TSA DAC</li> <li>Supplementary data set 3: Ancillary calculations and numerical values shown in plots and table 3</li> <li>Foreground data from OpenLCA. OpenLCA process model for different cases of HT-Aq DAC and TSA DAC. To re-run the LCA calculations, OpenLCA (freeware) and the Ecoinvent 3.5 database (license required) need to be installed on a standard desktop computer or laptop with at least 8 GB RAM.</li> </ul>
Data from: Cryogenic land surface processes shape vegetation biomass patterns in northern European tundra
<p>Tundra ecosystems have experienced changes in vegetation composition, distribution, and productivity over the past century due to climate warming. However, the increase in above-ground biomass (AGB) may be constrained by cryogenic land surface processes (LSP) that cause topsoil disturbance and variable microsite conditions. These effects have remained unaccounted for in tundra biomass models, although they can impact multiple opposing feedbacks between the biosphere and atmosphere, ecosystem functioning and biodiversity. Here, by using field-quantified data from northern Europe, remote sensing, and machine learning, we show that LSP substantially constrain AGB in tundra. The three surveyed LSP (cryoturbation, solifluction and nivation) collectively reduced AGB by an average of 123.0 g m<sup>-2 </sup>(-30.0%). This effect was significant over landscape positions and was especially pronounced in snowbed environments, where the mean reduction in AGB was 57.3%. Our results imply that LSP are pivotal in shaping future patterns of tundra biomass, as long as cryogenic ground activity is retained by climate warming.</p> <p>These are the key data and codes related from Aalto et al., (2021).</p> <p> </p>
Escherichia coli DNA replication study: processed alignment data
<p>Genomes are replicated by large protein complexes called replisomes. In bacterial DNA replication, two replisomes replicate the DNA starting from the same origin site and proceeding in opposite directions. Understanding their movement in vivo has been challenging. We used quantitative genome sequencing to characterize the dynamics of bacterial replisomes at 5 different temperatures (17, 22, 27, 32 and 37 °C) in exponential growth (3 replicates) or in stationary phase (one experiment at 17, 27 and 37 °C).</p> <p>The data deposited here give the coordinates of the sequence reads (deposited under the BioProject PRJNA772106) covering the Escherichia coli str. K-12 substr. MG1655 complete genome (accession number U00096.3).</p> <p>The file archive contains data files for each sample, at nucleotide resolution and binned in intervals of 10,000 base pairs. It also contains a C program to perform the binning and a README summarising how the alignment was done. <em>Please note that once uncompressed, the data will take 5 Gb of disks space in total.</em></p>
OnSSET processed GIS input data for 20 Sub Sahara Africa countries
<p>This repository contains the processed geospatial data files needed to run the Open-Source Spatial Electrification Tool (OnSSET), for 20 countries in Sub Sahara Africa. These data files were created by the KTH team for the Global Electrification Platform (GEP) model (https://electrifynow.energydata.info/). To access result files and geospatial population clusters, please go to https://energydata.info/dataset/?q=gep.</p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.