Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
5
datasets available to search
ShareScore release 0.9.0
Dataset results
5 results for “forecast benchmarking”
A comprehensive dataset for the accelerated development and benchmarking of solar forecasting methods
<p><strong>Description</strong><br> This repository contains a comprehensive solar irradiance, imaging, and forecasting dataset. <br> The goal with this release is to provide standardized solar and meteorological datasets to the research community for the accelerated development and benchmarking of forecasting methods. <br> The data consist of three years (2014–2016) of quality-controlled, 1-min resolution global horizontal irradiance and direct normal irradiance ground measurements in California. <br> In addition, we provide overlapping data from commonly used exogenous variables, including sky images, satellite imagery, Numerical Weather Prediction forecasts, and weather data. <br> We also include sample codes of baseline models for benchmarking of more elaborated models.</p> <p><strong>Data usage</strong><br> The usage of the datasets and sample codes presented here is intended for research and development purposes only and implies explicit reference to the paper:<br> <em>Pedro, H.T.C., Larson, D.P., Coimbra, C.F.M., 2019. A comprehensive dataset for the accelerated development and benchmarking of solar forecasting methods. Journal of Renewable and Sustainable Energy 11, 036102. https://doi.org/10.1063/1.5094494</em></p> <p>Although every effort was made to ensure the quality of the data, no guarantees or liabilities are implied by the authors or publishers of the data.</p> <p><strong>Sample code</strong><br> As part of the data release, we are also including the sample code written in Python 3. <br> The preprocessed data used in the scripts are also provided. <br> The code can be used to reproduce the results presented in this work and as a starting point for future studies. <br> Besides the standard scientific Python packages (numpy, scipy, and matplotlib), the code depends on pandas for time-series operations, pvlib for common solar-related tasks, and scikit-learn for Machine Learning models. <br> All required Python packages are readily available on Mac, Linux, and Windows and can be installed via, e.g., pip. </p> <p><strong>Units</strong><br> All time stamps are in UTC (YYYY-MM-DD HH:MM:SS).<br> All irradiance and weather data are in SI units.<br> Sky image features are derived from 8-bit RGB (256 color levels) data.<br> Satellite images are derived from 8-bit gray-scale (256 color levels) data.</p> <p><strong>Missing data</strong><br> The string "NAN" indicates missing data</p> <p><strong>File formats</strong><br> All time series data files as in CSV (comma separated values)<br> Images are given in tar.bz2 files</p> <p><strong>Files </strong></p> <ul> <li><em>Folsom_irradiance.csv</em> Primary One-minute GHI, DNI, and DHI data.</li> <li><em>Folsom_weather.csv </em> Primary One-minute weather data.</li> <li><em>Folsom_sky_images_{YEAR}.tar.bz2</em> Primary Tar archives with daytime sky images captured at 1-min intervals for the years 2014, 2015, and 2016, compressed with bz2.</li> <li><em>Folsom_NAM_lat{LAT}_lon{LON}.csv </em> Primary NAM forecasts for the four nodes nearest the target location. {LAT} and {LON} are replaced by the node’s coordinates listed in Table I in the paper. </li> <li><em>Folsom_sky_image_features.csv </em> Secondary Features derived from the sky images.</li> <li><em>Folsom_satellite.csv </em> Secondary 10 pixel by 10 pixel GOES-15 images centered in the target location. </li> <li><em>Irradiance_features_{horizon}.csv</em> Secondary Irradiance features for the different forecasting horizons ({horizon} 1⁄4 {intra-hour, intra-day, day-ahead}). </li> <li><em>Sky_image_features_intra-hour.csv</em> Secondary Sky image features for the intra-hour forecasting issuing times. </li> <li><em>Sat_image_features_intra-day.csv</em> Secondary Satellite image features for the intra-day forecasting issuing times. </li> <li><em>NAM_nearest_node_day-ahead.csv </em> Secondary NAM forecasts (GHI, DNI computed with the DISC algorithm, and total cloud cover) for the nearest node to the target location prepared for day-ahead forecasting.</li> <li><em>Target_{horizon}.csv</em> Secondary Target data for the different forecasting horizons.</li> <li>F<em>orecast_{horizon}.py </em> Code Python script used to create the forecasts for the different horizons. </li> <li><em>Postprocess.py</em> Code Python script used to compute the error metric for all the forecasts.</li> </ul> <p> </p>
Time Series used in the Forecasting Benchmark
<p>This data set contains the time series used in Libra (GitHub: <a href="https://github.com/DescartesResearch/ForecastBenchmark">https://github.com/DescartesResearch/ForecastBenchmark</a> ; CodeOcean: <a href="https://doi.org/10.24433/CO.3240518.v1">https://doi.org/10.24433/CO.3240518.v1</a>). Libra is a forecasting benchmark that automatically evaluates and ranks forecasting methods based on their performance in a diverse set of evaluation scenarios. The benchmark comprises four different use cases, each covering 100 heterogeneous time series taken from different domains.</p>
Supplementary Material for 'Benchmarking Explanatory Models for Inertia Forecasting using Public Data of the Nordic Area'
<p>This data set is supplementary material for the paper 'Benchmarking Explanatory Models for Inertia Forecasting using Public Data of the Nordic Area' by Jemima Sophie Graham, Evelyn Heylen, and Fei Teng.</p> <p>This data set is intended for day-ahead inertia forecasting in the Nordic (Eastern Denmark, Finland, Norway, Sweden). It contains hourly data for the inertial energy (MVAs), day-ahead national demand forecast (MW), day-ahead wind power forecast (MW), day-ahead solar power forecast (MW), and interconnection flow (MW) in the Nordic between January 2016 and August 2020. </p>
WaterBench-Iowa: A Large-scale Benchmark Dataset for Data-Driven Streamflow Forecasting
<p>WaterBench-Iowa is a comprehensive benchmark dataset for streamflow forecasting. It follows FAIR data principles that are prepared with a focus on convenience for utilizing in data-driven and machine learning studies and provides benchmark performance for state-of-art deep learning architectures on the dataset for comparative analysis. By aggregating the datasets of streamflow, precipitation, watershed area, slope, soil types, and evapotranspiration from federal agencies and state organizations (i.e., NASA, NOAA, USGS, and Iowa Flood Center), we provided the WaterBench for hourly streamflow forecast studies. This dataset has a high temporal and spatial resolution with rich metadata and relational information, which can be used for varieties of deep learning and machine learning research. To some extent, WaterBench makes up for the lack of a unified benchmark in earth science research. We highly encourage researchers to use the WaterBench for deep learning research in hydrology.</p>
CY-Bench: A comprehensive benchmark dataset for subnational crop yield forecasting
<h1>CY-Bench: A comprehensive benchmark dataset for sub-national crop yield forecasting</h1> <h2><br>Overview</h2> <p>CY-Bench is a dataset and benchmark for subnational crop yield forecasting, with coverage of major crop growing countries of the world for maize and wheat. By subnational, we mean the administrative level where yield statistics are published. When statistics are available for multiple levels, we pick the highest resolution. The dataset combines sub-national yield statistics with relevant predictors, such as growing-season weather indicators, remote sensing indicators, evapotranspiration, soil moisture indicators, and static soil properties. CY-Bench has been designed and curated by agricultural experts, climate scientists, and machine learning researchers from the <a href="https://www.agml.org/">AgML Community</a>, with the aim of facilitating model intercomparison across the diverse agricultural systems around the globe in conditions as close as possible to real-world operationalization. Ultimately, by lowering the barrier to entry for ML researchers in this crucial application area, CY-Bench will facilitate the development of improved crop forecasting tools that can be used to support decision-makers in food security planning worldwide.</p> <p>* Crops : Wheat & Maize<br>* Spatial Coverage : Wheat (29 countries), Maize (38).<br> See <a href="https://github.com/WUR-AI/AgML-CY-Bench/blob/main/notebooks/cybench_summary.ipynb">CY-Bench Summary</a> for the list of countries.<br>* Temporal Coverage : Varies. See <a href="https://github.com/WUR-AI/AgML-CY-Bench/blob/main/notebooks/cybench_summary.ipynb">CY-Bench Summary</a>.</p> <h2>Data </h2> <h3>Data format</h3> <p><br>The benchmark data is organized as a collection of CSV files (with the exception of location information, see below), with each file representing a specific category of variable for a particular country. Each CSV file is named according to the category and the country it pertains to, facilitating easy identification and retrieval. The data within each CSV file is structured in tabular format, where rows represent observations and columns represent different predictors related to a category of variable.</p> <h3>Data content</h3> <p>All data files are provided as .csv.</p> <table style="width: 100%;"> <tbody> <tr> <td style="width: 10.3122%;">Data</td> <td style="width: 21.9489%;">Description</td> <td style="width: 18.069%;">Variables (units)</td> <td style="width: 13.8137%;">Temporal Resolution</td> <td style="width: 35.8562%;">Data Source (Reference)</td> </tr> <tr> <td style="width: 10.3122%;">crop_calendar</td> <td style="width: 21.9489%;">start and end of growing season</td> <td style="width: 18.069%;">sos (day of the year),<br>eos (day of the year)</td> <td style="width: 13.8137%;">static</td> <td style="width: 35.8562%;">World Cereal (Franch et al, 2022)</td> </tr> <tr> <td style="width: 10.3122%;">crop_mask</td> <td style="width: 21.9489%;">crop area fraction</td> <td style="width: 18.069%;">crop_area (km2), crop_area_percentage (%)</td> <td style="width: 13.8137%;">static</td> <td style="width: 35.8562%;">WorldCereal (Van Tricht et al., 2023; EC-JRC, 2024)</td> </tr> <tr> <td style="width: 10.3122%;">fpar</td> <td style="width: 21.9489%;">fraction of absorbed photosynthetically active radiation</td> <td style="width: 18.069%;">fpar (%)</td> <td style="width: 13.8137%;">Dekadal (3 times a month; 1-10, 11-20, 21-31)</td> <td style="width: 35.8562%;">European Commission's Joint Research Centre (EC-JRC, 2024)</td> </tr> <tr> <td style="width: 10.3122%;">ndvi</td> <td style="width: 21.9489%;">normalized difference vegetation index</td> <td style="width: 18.069%;">-</td> <td style="width: 13.8137%;">approximately weekly</td> <td style="width: 35.8562%;">MOD09CMG (Vermote, 2015)</td> </tr> <tr> <td style="width: 10.3122%;">meteo</td> <td style="width: 21.9489%;">temperature, precipitation (prec), radiation, potential evapotranspiration (et0), climatic water balance (= prec - et0) </td> <td style="width: 18.069%;">tmin (C), tmax (C), tavg (C), prec (mm0, et0 (mm), cwb (mm), rad (J m-2 day-1)</td> <td style="width: 13.8137%;">daily</td> <td style="width: 35.8562%;">AgERA5 (Boogaard et al, 2022)</td> </tr> <tr> <td style="width: 10.3122%;">soil_moisture</td> <td style="width: 21.9489%;">surface soil moisture, rootzone soil moisture</td> <td style="width: 18.069%;">ssm (kg m-2), rsm (kg m-2)</td> <td style="width: 13.8137%;">daily</td> <td style="width: 35.8562%;">GLDAS (Rodell et al, 2004)</td> </tr> <tr> <td style="width: 10.3122%;">soil</td> <td style="width: 21.9489%;">available water capacity, bulk density, drainage class</td> <td style="width: 18.069%;">awc (c m-1), bulk_density (kg dm-3), drainage class (category)</td> <td style="width: 13.8137%;">static</td> <td style="width: 35.8562%;">WISE Soil database (Batjes, 2016)</td> </tr> <tr> <td style="width: 10.3122%;">location</td> <td style="width: 21.9489%;">centroid</td> <td style="width: 18.069%;">latitude, logitude, region_area (km2)</td> <td style="width: 13.8137%;">static</td> <td style="width: 35.8562%;"> </td> </tr> <tr> <td style="width: 10.3122%;">yield</td> <td style="width: 21.9489%;">end-of-season yield</td> <td style="width: 18.069%;">yield (t ha-1)</td> <td style="width: 13.8137%;">yearly</td> <td style="width: 35.8562%;">Various country or region specific sources (see crop_statistics_... in https://github.com/WUR-AI/AgML-CY-Bench/tree/main/data_preparation)</td> </tr> </tbody> </table> <h3> </h3> <h3>Folder structure</h3> <ol> <li>cybench-data: The CY-Bench dataset has been structure at first level by crop type and subsequently by country. For each country, the folder name follows the ISO 3166-1 alpha-2 two-character code. A separate .csv is available for each predictor data and crop calendar as shown below. The csv files are named to reflect the corresponding country and crop type e.g. **variable_croptype_country.csv**.<br>```<br>CY-Bench<br>│<br>└─── maize<br>│ │<br>│ └─── AO<br>│ │ -- crop_calendar_maize_AO.csv<br>│ │ -- crop_mask_maize_AO.csv<br>│ │ -- fpar_maize_AO.cs<br>│ │ -- location_maize_AO.csv<br>│ │ -- meteo_maize_AO.csv<br>│ │ -- ndvi_maize_AO.csv<br>│ │ -- soil_maize_AO.csv<br>│ │ -- soil_moisture_maize_AO.csv<br>│ │ -- yield_maize_AO.csv<br>│ │ <br>│ └─── AR<br>│ -- crop_calendar_maize_AR.csv<br>│ -- crop_mask_maize_AR.csv<br>│ -- fpar_maize_AR.csv<br>│ -- ...<br>│ <br>└─── wheat<br>│ │<br>│ └─── AR<br>│ │ -- crop_calendar_wheat_AR.csv<br>│ │ -- crop_mask_wheat_AR.csv<br>│ │ -- fpar_wheat_AR.csv<br>│ │ ...<br>``` <h3>Example : CSV data content for maize in country X</h3> <p>```<br>X<br>└─── crop_calendar_maize_X.csv<br>│ -- crop_name (name of the crop)<br>│ -- adm_id (unique identifier for a subnational unit)<br>│ -- sos (start of crop season)<br>│ -- eos (end of crop season)<br>│ <br>└─── crop_mask_maize_X.csv<br>│ -- crop_name<br>│ -- adm_id <br>│ -- crop_area<br>│ -- crop_area_percentage<br>│ <br>└─── fpar_maize_X.csv<br>│ -- crop_name<br>│ -- adm_id <br>│ -- date (in the format YYYYMMdd)<br>│ -- fpar<br>│<br>└─── location_maize_X.csv<br>│ -- crop_name<br>│ -- adm_id <br>│ -- latitude<br>│ -- longitude<br>│ -- region_area<br>│<br>└─── meteo_maize_X.csv<br>│ -- crop_name<br>│ -- adm_id <br>│ -- date (in the format YYYYMMdd)</p> <p>│ -- tmin (minimum temperature)<br>│ -- tmax (maximum temperature)<br>│ -- prec (precipitation)<br>│ -- rad (radiation)<br>│ -- tavg (average temperature)<br>│ -- et0 (evapotranspiration)<br>│ -- vpd (vapor pressure deficit)<br>│ -- cwb (crop water balance)<br>│ <br>└─── ndvi_maize_X.csv<br>│ -- crop_name<br>│ -- adm_id<br>│ -- date (in the format YYYYMMdd)<br>│ -- ndvi <br>│ <br>└─── soil_maize_X.csv<br>│ -- crop_name<br>│ -- adm_id<br>│ -- awc (available water capacity)<br>│ -- bulk_density<br>│ -- drainage_class<br>│ <br>└─── soil_moisture_maize_X.csv<br>│ -- crop_name<br>│ -- adm_id<br>│ -- date (in the format YYYYMMdd)<br>│ -- ssm (surface soil moisture)<br>│ -- rsm ()<br>│ <br>└─── yield_maize_X.csv<br>│ -- crop_name<br>│ -- country_code<br>│ -- adm_id<br>│ -- harvest_year<br>│ -- yield<br>│ -- harvest_area<br>│ -- production</p> </li> <li>centroids.zip and polygons.zip include shapes or geometries as centroids ( x and y coordinates) and polygons (multipolygons) of administrative regions respectively. They are organized as follows: <p>centroids</p> <p>│ └─── AO<br>│ │ -- AO.cpg<br>│ │ -- AO.dbf<br>│ │ -- AO.prj<br>│ │ -- AO.shp<br>│ │ -- AO.shx<br>│ └─── AR<br>│ │ -- AR.cpg<br>│ │ -- AR.dbf<br>│ │ -- AR.prj<br>│ │ -- AR.shp<br>│ │ -- AR.shx</p> ... <p>polygons</p> <p>│ └─── AO<br>│ │ -- AO.cpg<br>│ │ -- AO.dbf<br>│ │ -- AO.prj<br>│ │ -- AO.shp<br>│ │ -- AO.shx<br>│ └─── AR<br>│ │ -- AR.cpg<br>│ │ -- AR.dbf<br>│ │ -- AR.prj<br>│ │ -- AR.shp<br>│ │ -- AR.shx</p> ...</li> </ol> <h3>Data access</h3> <p>The full dataset can be downloaded directly from Zenodo or using the ```zenodo_get``` library</p> <h2><br>License and citation</h2> <p><br>We kindly ask all users of CY-Bench to properly respect licensing and citation conditions of the datasets included.</p> <p> </p> <h2>Version Notes</h2> <p>1.0 is the dataset submitted to NeurIPS Datasets and Benchmarks Track. The paper and discussions are here: https://openreview.net/forum?id=jkJDNG468g#discussion</p> <p>1.1 and 1.2 fix some issues with column names and mismatches in adm_id between yield data and input data.</p> <p>1.3 includes location information in the form of centroids and polygons of admin regions.</p> <p>1.4 updates the fpar data for 2023. fpar data was incomplete for 2023 in earlier versions (due to unavailability in the data source itself).</p> <p>1.5 fixes an issue in crop calendar</p> <p>1.6 fixes an issue in ndvi time series</p> <p>1.7 updates storage precision to 3 decimal places to reduce data size</p> <p>1.8 filter out invalid yield values</p> <p>1.9 Add vpd. Add location. ET0 obtained from AgERA5 (was AQUASTAT-FAO ). Use AgERA5 2.0 (was AgERA5 1.1)</p> <p>1.10 Add region_are to location*.csv. Add crop_mask_*.csv. Fix error in yield Australia. </p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.