Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
27
datasets available to search
ShareScore release 0.7.1
Dataset results
27 results for “Machine Observation”
Gravity Spy Machine Learning Classifications of LIGO Glitches from Observing Runs O1, O2, O3a, and O3b
<p>This data set contains all classifications that the Gravity Spy Machine Learning model for LIGO glitches from the first three observing runs (<a href="https://doi.org/10.7935/K57P8W9D">O1</a>, <a href="https://doi.org/10.7935/CA75-FM95">O2</a> and O3, where O3 is split into <a href="https://doi.org/10.7935/nfnt-hm34">O3a</a> and <a href="https://doi.org/10.7935/pr1e-j706">O3b</a>). Gravity Spy classified all noise events identified by the <a href="https://doi.org/10.1016/j.softx.2020.100620">Omicron trigger pipeline</a> in which Omicron identified that the signal-to-noise ratio was above 7.5 and the peak frequency of the noise event was between 10 Hz and 2048 Hz. To classify noise events, Gravity Spy made <a href="https://en.wikipedia.org/wiki/Constant-Q_transform">Omega scans</a> of every glitch consisting of 4 different durations, which helps capture the morphology of noise events that are both short and long in duration.</p> <p>There are <a href="https://doi.org/10.1088/1361-6382/aa5cea">22 classes</a> used for O1 and O2 data (including No_Glitch and None_of_the_Above), while there are <a href="https://doi.org/10.1088/1361-6382/ac1ccb">two additional classes</a> used to classify O3 data (while None_of_the_Above was removed).</p> <p>For O1 and O2, the glitch classes were: 1080Lines, 1400Ripples, Air_Compressor, Blip, Chirp, Extremely_Loud, Helix, Koi_Fish, Light_Modulation, Low_Frequency_Burst, Low_Frequency_Lines, No_Glitch, None_of_the_Above, Paired_Doves, Power_Line, Repeating_Blips, Scattered_Light, Scratchy, Tomte, Violin_Mode, Wandering_Line, Whistle</p> <p>For O3, the glitch classes were: 1080Lines, 1400Ripples, Air_Compressor, Blip, <strong>Blip_Low_Frequency</strong>, Chirp, Extremely_Loud, <strong>Fast_Scattering</strong>, Helix, Koi_Fish, Light_Modulation, Low_Frequency_Burst, Low_Frequency_Lines, No_Glitch, None_of_the_Above, Paired_Doves, Power_Line, Repeating_Blips, Scattered_Light, Scratchy, Tomte, Violin_Mode, Wandering_Line, Whistle</p> <p>The data set is described in <a href="https://doi.org/10.1088/1361-6382/acb633"><strong>Glanzer </strong><em>et al</em><strong>. (2023)</strong></a>, which we ask to be cited in any publications using this data release. Example code using the data can be found in this <a href="https://colab.research.google.com/drive/19q_lItODPk7qw_sohlHyWPnAbY0FZyt8?usp=sharing"><strong>Colab notebook</strong></a>.</p> <p>If you would like to download the Omega scans associated with each glitch, then you can use the gravitational-wave data-analysis tool <a href="https://gwpy.github.io/docs/stable/">GWpy</a>. If you would like to use this tool, please install anaconda if you have not already and create a virtual environment using the following command</p> <pre><code class="language-bash">conda create --name gravityspy-py38 -c conda-forge python=3.8 gwpy pandas psycopg2 sqlalchemy</code></pre> <p>After downloading one of the CSV files for a specific era and interferometer, please run the following Python script if you would like to download the data associated with the metadata in the CSV file. We recommend not trying to download too many images at one time. For example, the script below will read data on Hanford glitches from O2 that were classified by Gravity Spy and filter for only glitches that were labelled as Blips with 90% confidence or higher, and then download the first 4 rows of the filtered table.</p> <pre><code class="language-python">from gwpy.table import GravitySpyTable H1_O2 = GravitySpyTable.read('H1_O2.csv') H1_O2[(H1_O2["ml_label"] == "Blip") & (H1_O2["ml_confidence"] > 0.9)] H1_O2[0:4].download(nproc=1)</code></pre> <p>Each of the columns in the CSV files are taken from various different inputs: </p> <p>[‘event_time’, ‘ifo’, ‘peak_time’, ‘peak_time_ns’, ‘start_time’, ‘start_time_ns’, ‘duration’, ‘peak_frequency’, ‘central_freq’, ‘bandwidth’, ‘channel’, ‘amplitude’, ‘snr’, ‘q_value’] contain metadata about the signal from the <a href="https://virgo.docs.ligo.org/virgoapp/Omicron/">Omicron pipeline</a>. </p> <p>[‘gravityspy_id’] is the unique identifier for each glitch in the dataset. </p> <p>[‘1400Ripples’, ‘1080Lines’, ‘Air_Compressor’, ‘Blip’, ‘Chirp’, ‘Extremely_Loud’, ‘Helix’, ‘Koi_Fish’, ‘Light_Modulation’, ‘Low_Frequency_Burst’, ‘Low_Frequency_Lines’, ‘No_Glitch’, ‘None_of_the_Above’, ‘Paired_Doves’, ‘Power_Line’, ‘Repeating_Blips’, ‘Scattered_Light’, ‘Scratchy’, ‘Tomte’, ‘Violin_Mode’, ‘Wandering_Line’, ‘Whistle’] contain the machine learning confidence for a glitch being in a particular Gravity Spy class (the confidence in all these columns should sum to unity). These use the original 22 classes in all cases.</p> <p>[‘ml_label’, ‘ml_confidence’] provide the machine-learning predicted label for each glitch, and the machine learning confidence in its classification. </p> <p>[‘url1’, ‘url2’, ‘url3’, ‘url4’] are the links to the publicly-available <a href="https://gwdetchar.readthedocs.io/en/stable/omega/">Omega scans</a> for each glitch. ‘url1’ shows the glitch for a duration of 0.5 seconds, ‘url2’ for 1 seconds, ‘url3’ for 2 seconds, and ‘url4’ for 4 seconds.</p> <p>For the most recently uploaded training set used in Gravity Spy machine learning algorithms, please see <a href="https://zenodo.org/record/1486046#.YZfcar3MJqs">Gravity Spy Training Set</a> on Zenodo. </p> <p><br> For detailed information on the training set used for the original Gravity Spy machine learning paper, please see <a href="https://zenodo.org/record/1476156#.YZfchL3MJqs">Machine learning for Gravity Spy: Glitch classification and dataset</a> on Zenodo.</p>
Dataset for: Importance of satellite observations for high-resolution mapping of near-surface NO2 by machine learning
<p>Dataset for: Importance of satellite observations for high-resolution mapping of near-surface NO<sub>2 </sub>by machine learning</p> <p>This dataset is uploaded as a part of the article by Kim et al. (2021). The dataset is the hourly maps of near-surface nitrogen dioxide (NO<sub>2</sub>) concentrations at 100 m resolution for an Alpine domain (Switzerland and northern Italy, 6-12 °E, 42-48 °N). The dataset is provided per day (24 hours) in a netcdf (*.nc ~550MB). In this work, we have generated NO<sub>2 </sub>hourly maps for Feb. 2019 to May 2020 and, here, we upload for March 2019 only (~16 GB). If you need data for another period of time, please contact Gerrit Kuhlmann (gerrit.kuhlmann@empa.ch) or Minsu Kim (minsu.kim@empa.ch). </p>
EO4WildFires: An Earth Observation multi-sensor, time-series machine-learning-ready benchmark dataset for wildfire impact prediction
<p>This paper presents a benchmark dataset called EO4WildFires; a multi-sensor (multi spectral; Sentinel-2, Synthetic-Aperture Radar - SAR; Sentinel-1, meteorological parameters; NASA Power) time-series dataset that spans 45 countries, which can be used for developing machine learning and deep learning methods targeted for the estimation of the area that a forest wildfire might cover.</p> <p>This novel EO4WildFires dataset is annotated using EFFIS (European Forest Fire Information System) as forest fire detection and size estimation data source. A total of 31,742 wildfire events are gathered from 2018 to 2022. For each event, Sentinel-2 (multispectral), Sentinel-1 (SAR) and meteorological data are assembled into a single data cube. The meteorological parameters that are included in the data cube are: ratio of actual partial pressure of water vapor to the partial pressure at saturation, average temperature, bias corrected average total precipitation, average wind speed, fraction of land covered by snowfall, percent of root zone soil wetness, snow depth, snow precipitation, as well as percent of soil moisture.</p> <p>The main problem that this dataset is designed to address, is the severity forecasting before wildfires occur. The dataset is not used to predict wildfire events, but rather to predict the severity (size of area damaged by fire) of a wildfire event, if that happens in a specific place under the current and historical forest status, as recorded from multispectral and SAR images, and meteorological data.</p> <p>Using the data cube for the collected wildfire events, the EO4WildFires dataset is used to realize three (3) different preliminary experiments, in order to evaluate the contributing factors for wildfire severity prediction. The first experiment evaluates wildfire size using only the meteorological parameters, the second one utilizes both the multispectral and SAR parts of the dataset, while the third exploits all dataset parts. In each experiment, machine learning models are developed, and their accuracy is evaluated.</p>
A dataset of Earth Observation Data for Lithological Mapping using Machine Learning
<p><strong>Dataset Information</strong></p> <p>Machine Learning (ML) algorithms had successfully contributed in the creation of automated methods of recognizing patterns in high-dimensional data. Remote sensing data covers wide geographical areas and could be used to solve the problem of the demand of various in-situ data. Lithologicall mapping using remotely sensed data is one of the most challenging applications of ML algorithms. In the framework of the “AI for Geoapplications” project , ML and especially Deep Learning (DL) methodologies are investigated for the identification and characterization of the lithology based on remote sensing data in various pilot areas in Greece. In order to train and test the various ML algorithms, a dataset consisting of 30 ROIs selected mainly from low -vegetated areas, that cover 2% of the total area of Greece was created</p> <p><strong>Dataset Preprocessing</strong></p> <p>Dataset preprocessing was executed using a combination of SNAP, QGIS and ENVI tools.</p> <p>Preprocessing steps:</p> <p>Defining areas with the following properties:</p> <ul> <li> <p>Zero cloud and snow coverage</p> </li> <li> <p>No water bodies</p> </li> <li> <p>Minimum vegetation</p> </li> </ul> <p>For the Aster Images:</p> <ul> <li> <p>Subset on defined areas</p> </li> <li> <p>Mosaic images when needed</p> </li> <li> <p>Digitising clouds</p> </li> </ul> <p>For the Labels:</p> <ul> <li> <p>We got the Soil map from YPEN (<a href="https://ypen.gov.gr/">https://ypen.gov.gr/</a>)</p> </li> <li> <p>Subset on defined areas</p> </li> <li> <p>All categories are represented with good analogies</p> </li> <li> <p>Clip label files with digitised clouds</p> </li> <li> <p>Rasterize</p> </li> </ul> <p> </p> <p>For the Labels we have eighteen categories for the twenty-eight areas that we collected data. We use the following coding for the Labels of our <strong>Dataset</strong>:</p> <table> <tbody> <tr> <td> <p><strong>Alluvial deposits</strong></p> </td> <td> <p><strong>0</strong></p> </td> </tr> <tr> <td> <p><strong>Limestone colluvial deposits</strong></p> </td> <td> <p><strong>1</strong></p> </td> </tr> <tr> <td> <p><strong>Limestones</strong></p> </td> <td> <p><strong>2</strong></p> </td> </tr> <tr> <td> <p><strong>Schists</strong></p> </td> <td> <p><strong>3</strong></p> </td> </tr> <tr> <td> <p><strong>Quaternary sediments</strong></p> </td> <td> <p><strong>4</strong></p> </td> </tr> <tr> <td> <p><strong>Gneiss</strong></p> </td> <td> <p><strong>5</strong></p> </td> </tr> <tr> <td> <p><strong>Slope fan debris</strong></p> </td> <td> <p><strong>6</strong></p> </td> </tr> <tr> <td> <p><strong>Mixed flysch</strong></p> </td> <td> <p><strong>7</strong></p> </td> </tr> <tr> <td> <p><strong>Flysch shale and cherts</strong></p> </td> <td> <p><strong>8</strong></p> </td> </tr> <tr> <td> <p><strong>Dolomites</strong></p> </td> <td> <p><strong>9</strong></p> </td> </tr> <tr> <td> <p><strong>Granite</strong></p> </td> <td> <p><strong>10</strong></p> </td> </tr> <tr> <td> <p><strong>Sandstone flysch</strong></p> </td> <td> <p><strong>11</strong></p> </td> </tr> <tr> <td> <p><strong>Flysch colluvial deposits</strong></p> </td> <td> <p><strong>12</strong></p> </td> </tr> <tr> <td> <p><strong>Peridotite and Gabbro</strong></p> </td> <td> <p><strong>13</strong></p> </td> </tr> <tr> <td> <p><strong>River bed deposits</strong></p> </td> <td> <p><strong>14</strong></p> </td> </tr> <tr> <td> <p><strong>Gneiss colluvial deposits</strong></p> </td> <td> <p><strong>15</strong></p> </td> </tr> <tr> <td> <p><strong>Not available</strong></p> </td> <td> <p><strong>-100</strong></p> </td> </tr> <tr> <td> <p><strong>cloud coverage</strong></p> </td> <td> <p><strong>-999</strong></p> </td> </tr> </tbody> </table> <p>The following table lists the available <strong>areas </strong>and the <strong>categories </strong>that each contains<strong>: <a href="https://docs.google.com/spreadsheets/d/17q0L5Ltz7V4uBY9i6DhULJsJCtf7BOY1nbblB-hf3Pw/edit?usp=share_link">Lithology_Dataset</a> </strong></p> <p> </p> <p>For the <strong>Sentinel-2 images</strong>, we made the following process:</p> <ul> <li> <p><strong>Resampling 10m</strong></p> </li> <li> <p><strong>Subset on defined areas</strong></p> </li> </ul> <p>The Sentinel-2 map contains: Sentinel 2 false colour composite 11/8/4 with OSM background</p> <p>The Final step is the collocation of the previous into a datacube i.e a multidimensional array with 25 bands (datacube dimensions differentiate for every area) using the Aster image as base (15m spatial resolution). </p> <ul> <li> <p>Bands 1-14: Aster</p> </li> <li> <p>Bands 15-24: S2</p> </li> <li> <p>Band 25: Label</p> </li> </ul> <p>The code for preprocessing the dataset in order to be used for machine learning algorithms can be found in the following link: </p> <p><a href="https://github.com/georgegiannop/Lithology">https://github.com/georgegiannop/Lithology</a></p> <p><strong>Citation</strong></p> <p>If you use this dataset in your work, please cite our paper:</p> <p>Vernikos, I., Giannopoulos, G., Christopoulou, A., Begaj, A., Stefouli, M., Bratsolis, E., and Charou, E.: A dataset of Earth Observation Data for Lithological Mapping using Machine Learning, EGU General Assembly 2023, Vienna, Austria, 24–28 Apr 2023, EGU23-17570, <a href="https://doi.org/10.5194/egusphere-egu23-17570">https://doi.org/10.5194/egusphere-egu23-17570</a>, 2023.</p> <p> </p> <p> </p>
Gauging Size Resolved Ambient Particulate Matter Concentration Solely Using Biometric Observations: A Machine Learning and Causal Approach
<p>Notebook and data to accompany the (unpublished) paper titled "Gauging Size Resolved Ambient Particulate Matter Concentration Solely Using Biometric Observations: A Machine Learning and Causal Approach". This work expands a previous study, relating particulate matter concentrations and short-term biometric features across multiple participants. </p><p>Github link: https://github.com/mi3nts/DUEDARE_multiple_participants</p>
Global estimates of marine gross primary production based on machine‐learning upscaling of field observations
<p>4 variables (excluding dimension variables):</p> <p>double GPP_LD_MLD_RF[Lon,Lat,Month] <br> units: mmol O2 m-2 d-1<br> fill value: NaN<br> long_name: Monthly mixed-layer integration of gross primary production trained from the<br> dataset determined by the light-dark bottle incubation using Random Forest<br> algorithm<br> coordinates: [Longitude, Latitude Month]</p> <p>double GPP_LD_ZEU_RF[Lon,Lat,Month] </p> <p> units: mmol O2 m-2 d-1<br> fill value: NaN<br> long_name: Monthly euphotic-zone integration of gross primary production trained from<br> the dataset determined by the light-dark bottle incubation using Random<br> Forest algorithm<br> coordinates: [Longitude, Latitude Month]</p> <p>double GPP_Triple_MLD_RF[Lon,Lat,Month] <br> units: mmol mmol O2 m-2 d-1<br> fillvalue: NaN<br> long_name: Monthly mixed-layer integration of gross primary production trained from<br> the dataset determined by the triple isotopes of dissolved oxygen using<br> Random Forest algorithm<br> coordinates: [Longitude, Latitude Month]<br> <br> double GPP_Triple_ZEU_RF[Lon,Lat,Month] <br> units: mmol O2 m-2 d-1<br> fill value: NaN<br> long_name: Monthly euphotic-zone integration of gross primary production trained from<br> the dataset determined by the triple isotopes of dissolved oxygen using<br> Random Forest algorithm</p> <p>3 dimensions:</p> <p> Lon Size:181<br> units: degree_north<br> long_name: Longitude</p> <p> Lat Size:91<br> units: degree_east<br> long_name: Latitude</p> <p> Month Size:13<br> units: Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct, Nov, Dec, Annuual_mean<br> long_name: Month</p> <p><br> Author: Yibin Huang & Nicolas Cassar<br> Correspond: nicolas.cassar@duke.edu<br> <br> Request_for_citation: If you use these data in publications or presentations, please cite: Huang,<br> Y., Nicholson, D., Huang, B., & Cassar, N. (2021). Global estimates of<br> marine gross primary production based on machine‐learning upscaling of<br> field observations. Global Biogeochemical Cycles, 35, e2020GB006718.<br> https://doi.org/10.1029/2020GB006718<br> <br> Creation date: Dec/6th/2021</p>
Weather data (forecast and observation) at three locations in France over 2021 for Machine Learning Training
<p>The data provided data are historical weather measurement and forecast at three location in France.</p> <p>Measurements are inside files named OBS_xxx</p> <p>Forecasts are inside files names YYY_xxx, with YYY is the name of the forecast simultion (GFS0.25, WRF12km or WRF3KM).</p> <p>In the two cases, xxx is the name of the site (Site 1, Site2 or Site3).</p> <p><br> <strong>Description of OBS_xxx files:</strong><br> - One line per measurement with hourly resolution<br> - columns are: Date(TU),Temperature2m_degC,WindSpeed10m_m/s,WindDirection10m_m/s<br> Date = date of measurement in TU and format DD/MM/YYYY HH:MM<br> Temperature2m_degC = air temperature at 2m height in °Celsius<br> WindSpeed10m_m/s = wind speed at 10m height in m/s<br> WindDirection10m_deg = wind direction at 10m height in deg. (0 or 360 = wind from north to south, 45°=wind from east to east, ....)<br> If measurement is not available for a specific hour for one parameter, the value "-999" is used.</p> <p>The observation data go:<br> from 16/04/2021 00H <br> to 31/01/2022 23H</p> <p><br> <strong>Description of YYY_xxx files:</strong><br> - One line per forecast with hourly resolution<br> - columns are: First date run (TU),forecast hour,Temperature2m_degC,WindSpeed10m_m/s,WindDirection10m_m/s<br> First date run (TU) = date of start of the forecast in TU and format DD/MM/YYYY HH:MM. HH could be 00 and 12 according to the cycle of forecast start.<br> forecast hour = forecast hour from the start of the forecast date. 00 = forecast for "first date run". 01 = forecast for "First date run" + 1 hour. .... 95 = forecast for "First date run" + 95 hours.<br> For GFS0.25, forecast hour go from 00 to 95<br> For WRF12km, forecast hour go from 00 to 95<br> For WRF3m, forecast hour go from 00 to 95<br> Temperature2m_degC = air temperature at 2m height in °Celsius<br> WindSpeed10m_m/s = wind speed at 10m height in m/s<br> WindDirection10m_deg = wind direction at 10m height in deg. (0 or 360 = wind from north to south, 45°=wind from east to east, ....)<br> If measurement is not available for a specific hour for one parameter, the value "-999" is used.</p> <p>The forecast data go:<br> from 13/04/2021 00H + 72H = first forecast for the 16/04/2021 00H<br> to 31/01/2022 12H + 11H = last forecast for the 31/01/2022 23H</p>
Gauging Ambient Environmental Carbon Dioxide Concentration Solely Using Biometric Observations: A Machine Learning Approach
<p>Data and code in form of Jupyter Notebook to accompany an unpublished paper with the title " Gauging Ambient Environmental Carbon Dioxide Concentration Solely Using Biometric Observations: A Machine Learning Approach". This work makes use of biometric variables of a participant to estimate the inhaled carbon dioxide in microenvironments and understand various physiological and cognitive responses. </p><p>Github link: <a href="https://github.com/mi3nts/Estimate-CO2">mi3nts/Estimate-CO2: Data and code to estimate inhaled CO2 (github.com)</a></p>
Data for: A comprehensive dataset of forest above-ground biomass from field observations, machine learning and topographically augmented allometric models over the Kashmir Himalaya
<p>The repository contains observed Above Ground Biomass (AGB) estimates at about 275 sample plots chosen for AGB assessment in the forests of Kashmir Himalaya. The AGB is assessed as a fucntion of dbh using various allometric equations developed specifically for the region. It also contains the AGB for years 1978, 1990, 2000, 2010 and 2021 predicted using topographcally augmeneted multivariate regression model. The extent of forest, delineated using on-screen digitization using Landsat and Sentinel image collection at decadal scale is also provided for the years 1978, 1990, 2000, 2010 and 2021.</p>
On the application of an observations-based machine learning parameterization of surface layer fluxes within an atmospheric large-eddy simulation model: Trained Models
<p>Trained machine learning models and scaling values used in the paper "On the application of an observations-based machine learning parameterization of surface layer fluxes within an atmospheric large-eddy simulation model."</p>
Data for NHESS manuscript by Biass et al. (2022): Insights into the vulnerability of vegetation to tephra fallouts from interpretable machine learning and big Earth observation data
<p>This repository contains the data produced in the context of the following paper:</p> <blockquote> <p>Biass S, Jenkins SF, Aberhard WH, Delmelle P, Wilson T (2022): Insights into the vulnerability of vegetation to tephra fallouts from interpretable machine learning and big Earth observation data, Accepted in NHESS</p> </blockquote> <p>Naming convention is: `run_date`_`landcover`_`impact_metrics`_`VI`_`anomaly`_test.pkl, where:</p> <ul> <li>Landcover is either crops, shrubs, herbaceous vegetation (grass), forests (trees) or all together</li> <li>Impact metrics is either minV (impact magnitude) or minT (impact duration)</li> <li>VI is the vegetation index (here, EVI)</li> <li>Anomaly is the impact indicator (here, cumulative difference index)</li> </ul> <p>Refer to the associated paper for more information on the methodology.</p> <p>Files are saved as .pkl and were generated by the <a href="https://explainerdashboard.readthedocs.io/en/latest/">explainerdashboard</a> library. They are the result of <a href="https://xgboost.readthedocs.io/en/stable/">XGBoost</a> runs that were optimised with <a href="https://optuna.org">Optuna</a> and analysed with the <a href="https://shap.readthedocs.io/en/latest/">SHAP</a> library. They contain:</p> <ol> <li>The explanatory variables and observed and computed target variables for all features</li> <li>The SHAP values</li> </ol> <p>To load the files, use <a href="https://explainerdashboard.readthedocs.io/en/latest/cli.html?highlight=load#explainerdashboard.explainers.BaseExplainer.from_file">this method</a>.</p> <p> </p>
Observing flow of He II with unsupervised machine learning
<p>Data repository for observing flow in He II with unsupervised machine learning.</p>
Weather data (forecast and observation) at 48 locations in France for beginning of 2024 for Machine Learning Training
<p>The data provided data are historical weather measurement and forecast at 48 locations in France and its boundary.</p> <p>Measurements are inside files named MES_YYYY.csv with YYYY is the id code of the station.</p> <p>The file "Station_list.csv" contains the list of the 45 locations with the id code, the name and then the latitude and longitude.</p> <p><br>Forecasts are inside files named XXX_YYYY.csv with YYYY the id code corresponding of the location of the grid ouput close to the associated observation location.<br>XXX is the id of the numerical forecast:<br> "GFS0.25-Complet" for GFS file at 0.25° resolution<br> "LEXIS" for WRF produced by EVEREST project using the LEXIS chain<br> "WRF3KM-Complet" for WRF at 3km resolution produced by NUMTECH<br> "WRF12KM-Complet" for WRF at 12km resolution produced by NUMTECH</p> <p><br>Description of MES-YYYY files:<br>- One line per measurement with hourly resolution<br>- columns are: Date(TU),Temperature2m_degC,WindSpeed10m_m/s,WindDirection10m_m/s<br> Date = date of measurement in TU and format DD/MM/YYYY HH:MM<br> Temperature2m_degC = air temperature at 2m height in °Celsius<br> WindSpeed10m_m/s = wind speed at 10m height in m/s<br> WindDirection10m_deg = wind direction at 10m height in deg. (0 or 360 = wind from north to south, 45°=wind from east to east, ....)<br>If measurement is not available for a specific hour for one parameter, the value "-999" is used.</p> <p>The observation data gocfrom 28/01/2024 00HTU to 17/03/2024 23HTU</p> <p><br>Description of XXX_YYYY forecast files:<br>- One line per forecast with hourly resolution<br>- columns are: First date run (TU),Forecast date,Temperature2m_degC,WindSpeed10m_m/s,WindDirection10m_m/s<br> First date run (TU) = date of start of the forecast in TU and format DD/MM/YYYY HH:MM. HH could be 00 and 12 according to the cycle of forecast start.<br> Forecast date = date of the forecast in TU and format DD/MM/YYYY HH:MM. HH go from 00 to 23. <br> Temperature2m_degC = air temperature at 2m height in °Celsius<br> WindSpeed10m_m/s = wind speed at 10m height in m/s<br> WindDirection10m_deg = wind direction at 10m height in deg. (0 or 360 = wind from north to south, 45°=wind from east to east, ....)<br>If forecast is not available for a specific hour for one parameter, the value "-999" is used.</p> <p>The forecast data go from 28/01/2024 00HTU to 17/03/2024 23HTU</p>
Addressing Observational Gaps in Aerosol Parameters using Machine Learning: Implications to Aerosol Radiative Forcing
<p>This dataset represents Aerosol Optical Depth (AOD), Single Scattering Albedo (SSA), and Absorption Parameter (AP) data over Kanpur, India, sourced from AERONET with initial data gaps of approximately 37%, 62%, and 58% respectively. To reduce these gaps, XGBoost, a machine learning model trained with reanalysis and satellite datasets, was employed with optimized hyperparameter tuning. Using AERONET data for training, XGBoost effectively addressed gaps, improving AOD by 10%, SSA by 23%, and AP by 21%.</p>
Probabilistic Machine Learning Estimation of Ocean Mixed Layer Depth from Dense Satellite and Sparse In-Situ Observations: Preprocessed Satellite and In-situ observation datasets
<p>This record includes all of the prepared data used in the manuscript, "Probabilistic Machine Learning Estimation of Ocean Mixed Layer Depth from Dense Satellite and Sparse In-Situ Observations" (citation information forthcoming). As a part of this manuscript, we analyzed the ability for machine learning models to extract sea surface information (from salinity, temperature, sea height anomaly) to predict mixed layer depth. In this manuscript there are two experimental datasets: (1) info derived from CESM POP2 ocean model dataset (1989-1998), and (2) info derived from a combination of satellite sources and MLD from Argo profiles. More details below. </p> <p>All of these data files are preprocessed and organized to be used with the ml-ocean-bl github code found at https://github.com/NCAR/ml-ocean-bl/mloceanbl/.</p> <ul> <li><strong>CESM POP2 Ocean model dataset</strong></li> </ul> <p>Preprocessed sea surface salinity (SSS), temperature (SST), sea surface height anomalies (SSH), and ocean mixed layer depth (MLD, or HMXL) derived from the CESM POP2 Ocean model. Specifically, CESM POP2 model in a hindcast forced by JRA55do atmospheric reanalysis from 1958 to present and initialized with an oceanic climatology as in e.g. <a href="https://journals.ametsoc.org/view/journals/phoc/aop/JPO-D-20-0217.1/JPO-D-20-0217.1.xml">Deppenmeier et al. (2021)</a>. The model outputs include the ocean mixed layer depth (MLD), sea surface salinity (SSS), sea surface temperature (SST), and sea height anomaly (SSH) at a temporal frequency of 5-days and an approximate latitude and longitude resolution of 0.1 degrees.</p> <p>Relevant files:</p> <ol> <li>full_EPO.nc, full_SIO.nc <ul> <li>NetCDF4 containing SSS, SST, SSH, MLD for the equatorial Pacific Ocean (EPO) and southern Indian Ocean (SIO) (see manuscript for details). Data is regridded onto a 1/2 degree lat/lon 5 day grid to correspond with data used for Argo datasets (see below).</li> </ul> </li> <li>clim_EPO.nc, clim_SIO.nc, clim_std_EPO.nc, std_clim_EPO.nc, std_clim_SIO.nc <ul> <li>NetCDF4 containing mean and standard deviation climatologies of SSS, SST, SSH, and MLD for EPO and SIO.</li> </ul> </li> <li>std_anomalies_EPO.nc, std_anomalies_SIO.nc <ul> <li>NetCDF4 containing SSS, SST, SSH, and MLD standardized anomalies for EPO and SIO. This is the dataset directly used for training in aforementioned manuscript. Use with ml-ocean-bl/ml-ocean-test/data. </li> </ul> </li> </ol> <ul> <li><strong>Satellite and Argo datasets</strong></li> </ul> <p>Preprocessed satellite sea surface salinity (SSS), temperature (SST), and sea surface height anomalies (SSH) and Argo-based mixed layer depth (MLD) profiles. Original data can be found at:</p> <p>(SST): Remote Sensing Systems. 2017. MW optimum interpolated SST data set. Ver. 5.0. PO.DAAC, CA, USA. Further information available at at <a href="https://doi.org/10.5067/GHMWO-4FR05">https://doi.org/10.5067/GHMWO-4FR05</a>. Data can be accessed at https://podaac-tools.jpl.nasa.gov/drive/files/allData/ghrsst/data/GDS2/L4/GLOB/REMSS/mw_OI/v5.0/.</p> <p>(SSS): Oleg Melnichenko. 2018. Aquarius L4 Optimally Interpolated Sea Surface Salinity. Ver. 5.0. PO.DAAC, CA, USA. Further information at <a href="https://doi.org/10.5067/AQR50-4U7CS">https://doi.org/10.5067/AQR50-4U7CS</a>. Data can be accessed at https://podaac-tools.jpl.nasa.gov/drive/files/SalinityDensity/aquarius/L4/IPRC/v5/7day. </p> <p>(SSH): Zlotnicki, Victor; Qu, Zheng; Willis, Joshua. 2019. SEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL1609. Ver. 1812. PO.DAAC, CA, USA. Information available at <a href="https://doi.org/10.5067/SLREF-CDRV2">https://doi.org/10.5067/SLREF-CDRV2</a>. Data can be accessed at https://podaac-tools.jpl.nasa.gov/drive/files/SeaSurfaceTopography/merged_alt/L4/cdr_grid</p> <p>(MLD) Argo-based ocean surface mixed layer depths using the buoyancy gradient definition of Whitt Nicholson and Carranza (2019) processed dataset available at https://doi.org/10.5281/zenodo.4291175.</p> <p>Relevant files:</p> <ol> <li>https://github.com/NCAR/ml-ocean-bl/mloceanbl/preprocess_mld.py and .../preprocess_sss_sst_ssh.py. <ul> <li>Preprocessing code</li> </ul> </li> <li>sss_sst_ssh_anomalies.nc. <ul> <li>Regridded and resampled SSS, SST, SSH onto a 1/2 degree lat/lon 7day grid. Contains preprocessed seasonal data along with anomalies.</li> </ul> </li> <li> mldb_climatology_climatologystd_binned.nc <ul> <li>Smoothed argo-based mixed layer depths are used to calculate climatologies and standardized climatologies. 4 degree lat/lon gridded climatologies.</li> </ul> </li> <li>mldb_full_anomalies_stdanomalies_climatology_stdclimatology.nc <ul> <li>Contains the Argo profile-derived MLD, anomalies, standard anomalies, climatologies, and standardized climatologies with corresponding argo locations, times, and corresponding weeks. </li> </ul> </li> <li>equatorial_pacific_model_oi_re.nc, southern_indian_model_oi_re.nc <ul> <li>Model outputs for the Equatorial Pacific Ocean and Southern Indian Ocean. These gridded files contain the model outputs (vlcnn, vlcnn variance, OI, OI variance, reanalysis, and reanalysis variance - see manuscript for nomenclature details) at each of the 200 weeks available. It should be noted that, in the equatorial Pacific Ocean, the lat/lon location of (-138.75, -9.75) is masked during the training and filled with a NaN in the .nc files. </li> </ul> </li> </ol> <p> </p> <p> </p> <p>Contact D. Foster with any questions.</p> <p> </p>
Data from: Observation definitions and their implications in machine learning-based predictions of excessive rainfall
Open the record for dataset details and reuse information.
On the application of an observations-based machine learning parameterization of surface layer fluxes within an atmospheric large-eddy simulation model: Article Data
<p>NetCDF datatset of presented results from the publication titled "On the application of an observations-based machine learning parameterization of surface layer fluxes within an atmospheric large-eddy simulation model" in the Journal of Geophysical Research - Atmospheres, Paper #2021JD036214R.</p>
pywaterinfo dataset for master's dissertation: Updating a conceptual rainfall-runoff model based on radar observation and machine learning
<p>This forcings dataset is the output of the pywaterinfo (https://fluves.github.io/pywaterinfo/) read in of forcing data (rain and potential evapotranspiration).</p> <p>Code related to this dataset can be found here: https://github.com/olivierbonte/master_thesis</p>
OpenEO dataset for master's dissertation: Updating a conceptual rainfall-runoff model based on radar observation and machine learning
<p>This dataset is the output of the <a href="https://openeo.org/">OpenEO</a> processing of satellite data (SAR backscatter and LAI). </p> <p>Code related to this dataset can be found <a href="https://github.com/olivierbonte/master_thesis">here</a></p>
Minimal dataset for master's dissertation: Updating a conceptual rainfall-runoff model based on radar observation and machine learning
<p>The minimal dataset needed of data which can not be retrieved from the internet by APIs in the preprocessing. Consists of shape, land use and rivers for the Zwalm catchment. </p> <p>Code related to this dataset can be found here: https://github.com/olivierbonte/master_thesis </p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.