Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
22
datasets available to search
ShareScore release 0.9.0
Dataset results
22 results for “magnetograms”
Solar flare forecasting based on magnetogram sequences learning with MViT and data augmentation
<p><strong>Source codes and dataset of the research "Solar flare forecasting based on magnetogram sequences learning with MViT and data augmentation".</strong></p><p>Our work employed PyTorch, a framework for training Deep Learning models with GPU support and automatic back-propagation, to load the MViTv2 s models with Kinetics-400 weights. To simplify the code implementation, eliminating the need for an explicit loop to train and the automation of some hyperparameters, we use the PyTorch Lightning module. The inputs were batches of 10 samples with 16 sequenced images in 3-channel resized to 224 × 224 pixels and normalized from 0 to 1.</p><p>Most of the papers in our literature survey split the original dataset chronologically. Some authors also apply k-fold cross-validation to emphasize the evaluation of the model stability. However, we adopt a hybrid split taking the first 50,000 to apply the 5-fold cross-validation between the training and validation sets (known data), with 40,000 samples for training and 10,000 for validation. Thus, we can evaluate performance and stability by analyzing the mean and standard deviation of all trained models in the test set, composed of the last 9,834 samples, preserving the chronological order (simulating unknown data).</p><p>We develop three distinct models to evaluate the impact of oversampling magnetogram sequences through the dataset. The first model, Solar Flare MViT (SF MViT), has trained only with the original data from our base dataset without using oversampling. In the second model, Solar Flare MViT over Train (SF MViT oT), we only apply oversampling on training data, maintaining the original validation dataset. In the third model, Solar Flare MViT over Train and Validation (SF MViT oTV), we apply oversampling in both training and validation sets.</p><p>We also trained a model oversampling the entire dataset. We called it the "SF_MViT_oTV Test" to verify how resampling or adopting a test set with unreal data may bias the results positively.</p><p><strong>GitHub version</strong></p><p>The .zip hosted here contains all files from the project, including the checkpoint and the output files generated by the codes. We have a clean version hosted on GitHub (<a href="https://github.com/lfgrim/SFF_MagSeq_MViTs">https://github.com/lfgrim/SFF_MagSeq_MViTs</a>), without the magnetogram_jpg folder (which can be downloaded directly on <a href="https://tianchi-competition.oss-cn-hangzhou.aliyuncs.com/531804/dataset_ss2sff.zip">https://tianchi-competition.oss-cn-hangzhou.aliyuncs.com/531804/dataset_ss2sff.zip)</a> and the output and checkpoint files. Most code files hosted here also contain comments on the Portuguese language, which are being updated to English in the GitHub version.</p><p><strong>Folders Structure</strong></p><p>In the Root directory of the project, we have two folders: </p><ul><li>magnetogram_jpg: holds the source images provided by Space Environment Artificial Intelligence Early Warning Innovation Workshop through the link <a href="https://tianchi-competition.oss-cn-hangzhou.aliyuncs.com/531804/dataset_ss2sff.zip">https://tianchi-competition.oss-cn-hangzhou.aliyuncs.com/531804/dataset_ss2sff.zip. </a>It comprises 73,810 samples of high-quality magnetograms captured by HMI/SDO from 2010 May 4 to 2019 January 26. The HMI instrument provides these data (stored in hmi.sharp_720s dataset), making new samples available every 12 minutes. However, the images from this dataset were collected every 96 minutes. Each image has an associated magnetogram comprising a ready-made snippet of one or most solar ARs. It is essential to notice that the magnetograms cropped by SHARP can contain one or more solar ARs classified by the National Oceanic and Atmospheric Administration (NOAA).</li><li>Seq_Magnetogram: contains the references for source images with the corresponding labels in the next 24 h. and 48 h. in the respectively M24 and M48 sub-folders.<ul><li>M24/M48: both present the following sub-folders structure:<ul><li>Seqs16;</li><li>SF_MViT;</li><li>SF_MViT_oT;</li><li>SF_MViT_oTV;</li><li>SF_MViT_oTV_Test.</li></ul></li></ul></li></ul><p>There are also two files in root:</p><ul><li>inst_packages.sh: install the packages and dependencies to run the models.</li><li>download_MViTS.py: download the pre-trained MViTv2_S from PyTorch and store it in the cache.</li></ul><p>M24 and M48 folders hold reference text files (flare_Mclass...) linking the images in the magnetogram_jpg folders or the sequences (Seq16_flare_Mclass...) in the Seqs16 folders with their respective labels. They also hold "cria_seqs.py" which was responsible for creating the sequences and "test_pandas.py" to verify head info and check the number of samples categorized by the label of the text files. All the text files with the prefix "Seq16" and inside the Seqs16 folder were created by "criaseqs.py" code based on the correspondent "flare_Mclass" prefixed text files.</p><p>Seqs16 folder holds reference text files, in which each file contains a sequence of images that was pointed to the magnetogram_jpg folders.</p><p>All SF_MViT... folders hold the model training codes itself (SF_MViT...py) and the corresponding job submission (jobMViT...), temporary input (Seq16_flare...), output (saida_MVIT... and MViT_S...), error (err_MViT...) and checkpoint files (sample-FLARE...ckpt). Executed model training codes generate output, error, and checkpoint files. There is also a folder called "lightning_logs" that stores logs of trained models.</p><p><strong>Naming pattern for the files:</strong></p><ul><li>magnetogram_jpg: follows the format<i> </i>"hmi.sharp_720s.<SHARP-ID>.<date>.magnetogram.fits.jpg" and</li><li>Seqs16: follows the format "hmi.sharp_720s.<i><</i>SHARP-ID<i>></i>.<init-date>.to.<end-date>", where:<ul><li>hmi: is the instrument that captured the image</li><li>sharp_720s: is the database source of SDO/HMI.</li><li><SHARP-ID>: is the identification of SHARP region, and can contain one or more solar ARs classified by the (NOAA).</li><li><date>: is the date-time the instrument captured the image in the format yyyymmdd_hhnnss_TAI (y:year, m:month, d:day, h:hours, n:minutes, s:seconds).</li><li><init-date>: is the date-time when the sequence starts, and follow the same format of <date>.</li><li><end-date>: is the date-time when the sequence ends, and follow the same format of <date>.</li></ul></li><li>Reference text files in M24 and M48 or inside SF_MViT... folders follows the format "<prefix>flare_Mclass_<forecasting-horizon>_<dataset>.txt<over>", where:<ul><li><prefix>: is Seq16 if refers to a sequence, or void if refers direct to images.</li><li><forecasting-horizon>: "24h" or "48h".</li><li><dataset>: is "TrainVal<n>" or "Test". The <n> refers to the split of Train/Val.</li><li><over>: void or "_over" after the extension (...txt_over): means temporary input reference that was over-sampled by a training model.</li></ul></li><li>All SF_MViT...folders:<ul><li>Model training codes: "SF_MViT_<oversampling-type>_M+_<forecasting-horizon>_<split-type><gpu-type>", where:<ul><li><oversampling -type>: void or "oT" (over Train) or "oTV" (over Train and Val) or "oTV_Test" (over Train, Val and Test);</li><li><forecasting-horizon>: "24h" or "48h";</li><li><split-type>: "oneSplit" for a specific split or "allSplits" if run all splits.</li><li><gpu-type>: void is default to run 1 GPU or "2gpu" to run into 2 gpus systems;</li></ul></li><li>Job submission files: "jobMViT_<queue>", where:<ul><li><queue>: point the queue in Lovelace environment hosted on CENAPAD-SP (<a href="https://www.cenapad.unicamp.br/parque/jobsLovelace">https://www.cenapad.unicamp.br/parque/jobsLovelace</a>)</li></ul></li><li>Temporary inputs: "Seq16_flare_Mclass_<forecasting-horizon>_<dataset>.txt<over>:<ul><li><dataset>: train or val;</li><li><over>: void or "_over" after the extension (...txt_over): means temporary input reference that was over-sampled by a training model.</li></ul></li><li>Outputs: "saida_MViT_Adam_10-7<split>", where:<ul><li><split>: k0 to k4, means the correlated split of the output, or void if the output is from all splits.</li></ul></li><li>Error files: "err_MViT_Adam_10-7<split>", where:<ul><li><split>: k0 to k4, means the correlated split of the error log file, or void if the error file is from all splits.</li></ul></li><li>Checkpoint files: "sample-FLARE_MViT_S_10-7-epoch=<n-epoch>-valid_loss=<loss-value>-Wloss_k=<n-split>.ckpt", where:<ul><li><n-opoch>: epoch number of the checkpoint;</li><li><loss-value>: corresponding valid loss;</li><li><n-split>: 0 to 4.</li></ul></li></ul></li></ul>
Active Region Magnetograms for Solar Flare Prediction: Reduced Resolution Dataset Images
<p>This dataset is the images associated with Dryad dataset https://doi.org/10.5061/dryad.jq2bvq898. These images are consistently sized images of active region magnetograms from the National Aeronautics and Space Administration's (NASA's) Solar Dynamics Observatory (SDO). These data are the reduced resolution (224x224 pixels) images in .png format.</p>
Active region magnetograms for solar flare prediction: Reduced resolution dataset
<p>In this dataset, we provide a comprehensive collection of magnetograms from the National Aeronautics and Space Administration's (NASA's) Solar Dynamics Observatory (SDO). The dataset incorporates data from three sources and provides SDO Helioseismic and Magnetic Imager (HMI) magnetograms of solar active regions as well as labels of corresponding flaring activity. This dataset will be useful for image analysis or solar physics research related to magnetic structure, its evolution over time, and its relation to solar flares. The dataset will be of interest to those researchers investigating automated solar flare prediction methods, including supervised and unsupervised machine learning (classical and deep), binary and multi-class classification, and regression. This dataset is a minimally processed, user configurable dataset of consistently sized images of solar active regions that can serve as a benchmark dataset for solar flare prediction research. This dataset consists of reduced resolution images (see usage notes below).</p>
Active Region Magnetograms for Solar Flare Prediction: Full Resolution Dataset Images for ARs 1307 through 1505
<p>This dataset is the images associated with Dryad dataset https://doi.org/10.5061/dryad.dv41ns23n. These images are consistently sized images of active region magnetograms from the National Aeronautics and Space Administration's (NASA's) Solar Dynamics Observatory (SDO). These data are the full sized images (600x600 pixels) for active regions (ARs) 1307 through 1505 in .fits format.</p>
Active Region Magnetograms for Solar Flare Prediction: Full Resolution Dataset Images for ARs 1064 through 1306
<p>This dataset is the images associated with Dryad dataset https://doi.org/10.5061/dryad.dv41ns23n. These images are consistently sized images of active region magnetograms from the National Aeronautics and Space Administration's (NASA's) Solar Dynamics Observatory (SDO). These data are the full sized images (600x600 pixels) for active regions (ARs) 1064 through 1306 in .fits format.</p>
Active Region Magnetograms for Solar Flare Prediction: Full Resolution Dataset Images for ARs 1506 through 1707
<p>This dataset is the images associated with Dryad dataset https://doi.org/10.5061/dryad.dv41ns23n. These images are consistently sized images of active region magnetograms from the National Aeronautics and Space Administration's (NASA's) Solar Dynamics Observatory (SDO). These data are the full sized images (600x600 pixels) for active regions (ARs) 1506 through 1707 in .fits format.</p>
Active Region Magnetograms for Solar Flare Prediction: Full Resolution Dataset Images for ARs 1708 through 1918
<p>This dataset is the images associated with Dryad dataset https://doi.org/10.5061/dryad.dv41ns23n. These images are consistently sized images of active region magnetograms from the National Aeronautics and Space Administration's (NASA's) Solar Dynamics Observatory (SDO). These data are the full sized images (600x600 pixels) for active regions (ARs) 1708 through 1918 in .fits format.</p>
Active Region Magnetograms for Solar Flare Prediction: Full Resolution Dataset Images for ARs 1919 through 2103
<p>This dataset is the images associated with Dryad dataset https://doi.org/10.5061/dryad.dv41ns23n. These images are consistently sized images of active region magnetograms from the National Aeronautics and Space Administration's (NASA's) Solar Dynamics Observatory (SDO). These data are the full sized images (600x600 pixels) for active regions (ARs) 1919 through 2103 in .fits format.</p>
Active Region Magnetograms for Solar Flare Prediction: Full Resolution Dataset Images for ARs 2284 through 2488
<p>This dataset is the images associated with Dryad dataset https://doi.org/10.5061/dryad.dv41ns23n. These images are consistently sized images of active region magnetograms from the National Aeronautics and Space Administration's (NASA's) Solar Dynamics Observatory (SDO). These data are the full sized images (600x600 pixels) for active regions (ARs) 2284 through 2488 in .fits format.</p>
Active Region Magnetograms for Solar Flare Prediction: Full Resolution Dataset Images for ARs 2104 through 2283
<p>This dataset is the images associated with Dryad dataset https://doi.org/10.5061/dryad.dv41ns23n. These images are consistently sized images of active region magnetograms from the National Aeronautics and Space Administration's (NASA's) Solar Dynamics Observatory (SDO). These data are the full sized images (600x600 pixels) for active regions (ARs) 2104 through 2283 in .fits format.</p>
Active Region Magnetograms for Solar Flare Prediction: Full Resolution Dataset Images for ARs 2489 through 2731
<p>This dataset is the images associated with Dryad dataset https://doi.org/10.5061/dryad.dv41ns23n. These images are consistently sized images of active region magnetograms from the National Aeronautics and Space Administration's (NASA's) Solar Dynamics Observatory (SDO). These data are the full sized images (600x600 pixels) for active regions (ARs) 2489 through 2731 in .fits format.</p>
Active Region Magnetograms for Solar Flare Prediction: Extra Dataset Images for ARs 1064 through 1527
<p>This dataset is the extra images associated with Dryad dataset <a href="https://doi.org/10.5061/dryad.qjq2bvqmj">https://doi.org/10.5061/dryad.qjq2bvqmj</a>. These images are consistently sized images of active region magnetograms from the National Aeronautics and Space Administration's (NASA's) Solar Dynamics Observatory (SDO). These data are the full sized images (600x600 pixels) for active regions (ARs) 1064 through 1527 in .fits format. These are images that were removed from the preconfigured dataset https://doi.org/10.5061/dryad.jq2bvq898.</p>
Active Region Magnetograms for Solar Flare Prediction: Extra Dataset Images for ARs 2470 through 2731
<p>This dataset is the extra images associated with Dryad dataset <a href="https://doi.org/10.5061/dryad.qjq2bvqmj">https://doi.org/10.5061/dryad.qjq2bvqmj</a>. These images are consistently sized images of active region magnetograms from the National Aeronautics and Space Administration's (NASA's) Solar Dynamics Observatory (SDO). These data are the full sized images (600x600 pixels) for active regions (ARs) 2470 through 2731 in .fits format. These are images that were removed from the preconfigured dataset https://doi.org/10.5061/dryad.jq2bvq898.</p>
Active Region Magnetograms for Solar Flare Prediction: Extra Dataset Images for ARs 1981 through 2469
<p>This dataset is the extra images associated with Dryad dataset <a href="https://doi.org/10.5061/dryad.qjq2bvqmj">https://doi.org/10.5061/dryad.qjq2bvqmj</a>. These images are consistently sized images of active region magnetograms from the National Aeronautics and Space Administration's (NASA's) Solar Dynamics Observatory (SDO). These data are the full sized images (600x600 pixels) for active regions (ARs) 1981 through 2469 in .fits format. These are images that were removed from the preconfigured dataset https://doi.org/10.5061/dryad.jq2bvq898.</p>
Active Region Magnetograms for Solar Flare Prediction: Extra Dataset Images for ARs 1528 through 1980
<p>This dataset is the extra images associated with Dryad dataset <a href="https://doi.org/10.5061/dryad.qjq2bvqmj">https://doi.org/10.5061/dryad.qjq2bvqmj</a>. These images are consistently sized images of active region magnetograms from the National Aeronautics and Space Administration's (NASA's) Solar Dynamics Observatory (SDO). These data are the full sized images (600x600 pixels) for active regions (ARs) 1528 through 1980 in .fits format. These are images that were removed from the preconfigured dataset https://doi.org/10.5061/dryad.jq2bvq898.</p>
Active region magnetograms for solar flare prediction: Extra images dataset
<p>In this dataset, we provide a comprehensive collection of magnetograms from the National Aeronautics and Space Administration's (NASA's) Solar Dynamics Observatory (SDO). The dataset incorporates data from three sources and provides SDO Helioseismic and Magnetic Imager (HMI) magnetograms of solar active regions as well as labels of corresponding flaring activity. This dataset will be useful for image analysis or solar physics research related to magnetic structure, its evolution over time, and its relation to solar flares. The dataset will be of interest to those researchers investigating automated solar flare prediction methods, including supervised and unsupervised machine learning (classical and deep), binary and multi-class classification, and regression. This dataset contains those images that were removed from the preconfigured datasets (see usage notes below).</p>
Active region magnetograms for solar flare prediction: Full resolution dataset
<p>In this dataset, we provide a comprehensive collection of magnetograms from the National Aeronautics and Space Administration's (NASA's) Solar Dynamics Observatory (SDO). The dataset incorporates data from three sources and provides SDO Helioseismic and Magnetic Imager (HMI) magnetograms of solar active regions as well as labels of corresponding flaring activity. This dataset will be useful for image analysis or solar physics research related to magnetic structure, its evolution over time, and its relation to solar flares. The dataset will be of interest to those researchers investigating automated solar flare prediction methods, including supervised and unsupervised machine learning (classical and deep), binary and multi-class classification, and regression. This dataset is a minimally processed, user configurable dataset of consistently sized images of solar active regions that can serve as a benchmark dataset for solar flare prediction research. This dataset consists of full resolution images (see usage notes below).</p>
Active region magnetograms for solar flare prediction: Full resolution dataset
Open the record for dataset details and reuse information.
Active region magnetograms for solar flare prediction: Reduced resolution dataset
Open the record for dataset details and reuse information.
Active region magnetograms for solar flare prediction: Extra images dataset
Open the record for dataset details and reuse information.
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.