SDO 2H Machine Learning Dataset

<p>This dataset provides a compact Machine Learning ready dataset of SDO EUV and HMI medium-resolution&nbsp; (1024x1024 pixels) images, for a total of 56,664 samples from May 14, 2010, to April 18, 2023, with a temporal cadence of 2 hours.</p> <p>EUV images are provided at the following wavelength : 1600A, 304A, 211A, 193A, 171A and 94A.<br>They are processed from the level 1.5 AIA-synoptic dataset (http://jsoc.stanford.edu/data/aia/synoptic/) and are successively:</p> <ul> <li>corrected for instrument degradation</li> <li>normalised by exposure time</li> <li>log-transformed (x-&gt;log(1+x)), symetrically on positive and negative values</li> <li>saturated to the 99.9 percentile maximum pixel value of the dataset, up to 2020*, for each channel</li> <li>linearly scaled between 0 and 255, converted to 8bit integers and compressed as jpegs</li> </ul> <p>The HMI's line-of-sight magnetograms (blos.zip) are retrieved from JSOC from the level 1.5 45-second line-of-sight serie and are successively :</p> <ul> <li>downscaled to 1024x1024 pixels</li> <li>standardized to a 2.4 arcec-to-pixel resolution (equal to the EUV images)</li> <li>aligned with the EUV images</li> <li>log-transformed (x-&gt;log(1+x))</li> <li>saturated to the 99.9 percentile maximum pixel value of the dataset, up to 2020*&nbsp;</li> <li>linearly scaled between 0 and 255, with 127 representig original null values, 0 and 255 respecivelly the negative and positive saturation value (approximately 4644G before log-transformation)</li> <li>converted to 8bit integers and compressed as jpegs</li> </ul> <p>Downscaled and cropped images (224x448 pixels) used in&nbsp;<a title="Francisco et al., 2023" href="https://doi.org/10.22541/essoar.170688972.24631782/v1" target="_blank" rel="noopener">Francisco et al., 2023</a> are aso provided in pcnn_images.zip</p> <p>An outlier study is also provided in anomalies.zip, from which '{wavelength}_anomalies_grades.csv' files can be used to exclude the dates where abnormal samples of a given type ('anomalies_grade_scale.txt') are identified.</p> <p>*The percentile values are computed on the pixels joint distribution using all sample from 2010 to 2019-12 included, so that the period starting from 2020-01 can be used as a completely independant test set.<br>Original exposure and instrument degradation corrected values can be retrieved using the saturation values provided belows. Althought the JPEG encoding results in the loss of small scale information, the dataset processing preserve the physical intensity of the original inputs so that the provided compressed images can efficiently be used to estimate large and medium-scale Active Regions physical features.</p> <table> <tbody> <tr> <td>HMI / BLOS</td> <td>&plusmn;4644 G</td> </tr> <tr> <td>1600</td> <td>9,360 DN</td> </tr> <tr> <td>304</td> <td>44,488 DN</td> </tr> <tr> <td>171</td> <td>29,599 DN</td> </tr> <tr> <td>193</td> <td>81,139 DN</td> </tr> <tr> <td>211</td> <td>8,179 DN</td> </tr> <tr> <td>94</td> <td>6,099 DN</td> </tr> </tbody> </table> <p>&nbsp;</p>