Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
4
datasets available to search
ShareScore release 0.9.0
Dataset results
4 results for “Gravity Spy”
Gravity Spy Machine Learning Classifications of LIGO Glitches from Observing Runs O1, O2, O3a, and O3b
<p>This data set contains all classifications that the Gravity Spy Machine Learning model for LIGO glitches from the first three observing runs (<a href="https://doi.org/10.7935/K57P8W9D">O1</a>, <a href="https://doi.org/10.7935/CA75-FM95">O2</a> and O3, where O3 is split into <a href="https://doi.org/10.7935/nfnt-hm34">O3a</a> and <a href="https://doi.org/10.7935/pr1e-j706">O3b</a>). Gravity Spy classified all noise events identified by the <a href="https://doi.org/10.1016/j.softx.2020.100620">Omicron trigger pipeline</a> in which Omicron identified that the signal-to-noise ratio was above 7.5 and the peak frequency of the noise event was between 10 Hz and 2048 Hz. To classify noise events, Gravity Spy made <a href="https://en.wikipedia.org/wiki/Constant-Q_transform">Omega scans</a> of every glitch consisting of 4 different durations, which helps capture the morphology of noise events that are both short and long in duration.</p> <p>There are <a href="https://doi.org/10.1088/1361-6382/aa5cea">22 classes</a> used for O1 and O2 data (including No_Glitch and None_of_the_Above), while there are <a href="https://doi.org/10.1088/1361-6382/ac1ccb">two additional classes</a> used to classify O3 data (while None_of_the_Above was removed).</p> <p>For O1 and O2, the glitch classes were: 1080Lines, 1400Ripples, Air_Compressor, Blip, Chirp, Extremely_Loud, Helix, Koi_Fish, Light_Modulation, Low_Frequency_Burst, Low_Frequency_Lines, No_Glitch, None_of_the_Above, Paired_Doves, Power_Line, Repeating_Blips, Scattered_Light, Scratchy, Tomte, Violin_Mode, Wandering_Line, Whistle</p> <p>For O3, the glitch classes were: 1080Lines, 1400Ripples, Air_Compressor, Blip, <strong>Blip_Low_Frequency</strong>, Chirp, Extremely_Loud, <strong>Fast_Scattering</strong>, Helix, Koi_Fish, Light_Modulation, Low_Frequency_Burst, Low_Frequency_Lines, No_Glitch, None_of_the_Above, Paired_Doves, Power_Line, Repeating_Blips, Scattered_Light, Scratchy, Tomte, Violin_Mode, Wandering_Line, Whistle</p> <p>The data set is described in <a href="https://doi.org/10.1088/1361-6382/acb633"><strong>Glanzer </strong><em>et al</em><strong>. (2023)</strong></a>, which we ask to be cited in any publications using this data release. Example code using the data can be found in this <a href="https://colab.research.google.com/drive/19q_lItODPk7qw_sohlHyWPnAbY0FZyt8?usp=sharing"><strong>Colab notebook</strong></a>.</p> <p>If you would like to download the Omega scans associated with each glitch, then you can use the gravitational-wave data-analysis tool <a href="https://gwpy.github.io/docs/stable/">GWpy</a>. If you would like to use this tool, please install anaconda if you have not already and create a virtual environment using the following command</p> <pre><code class="language-bash">conda create --name gravityspy-py38 -c conda-forge python=3.8 gwpy pandas psycopg2 sqlalchemy</code></pre> <p>After downloading one of the CSV files for a specific era and interferometer, please run the following Python script if you would like to download the data associated with the metadata in the CSV file. We recommend not trying to download too many images at one time. For example, the script below will read data on Hanford glitches from O2 that were classified by Gravity Spy and filter for only glitches that were labelled as Blips with 90% confidence or higher, and then download the first 4 rows of the filtered table.</p> <pre><code class="language-python">from gwpy.table import GravitySpyTable H1_O2 = GravitySpyTable.read('H1_O2.csv') H1_O2[(H1_O2["ml_label"] == "Blip") & (H1_O2["ml_confidence"] > 0.9)] H1_O2[0:4].download(nproc=1)</code></pre> <p>Each of the columns in the CSV files are taken from various different inputs: </p> <p>[‘event_time’, ‘ifo’, ‘peak_time’, ‘peak_time_ns’, ‘start_time’, ‘start_time_ns’, ‘duration’, ‘peak_frequency’, ‘central_freq’, ‘bandwidth’, ‘channel’, ‘amplitude’, ‘snr’, ‘q_value’] contain metadata about the signal from the <a href="https://virgo.docs.ligo.org/virgoapp/Omicron/">Omicron pipeline</a>. </p> <p>[‘gravityspy_id’] is the unique identifier for each glitch in the dataset. </p> <p>[‘1400Ripples’, ‘1080Lines’, ‘Air_Compressor’, ‘Blip’, ‘Chirp’, ‘Extremely_Loud’, ‘Helix’, ‘Koi_Fish’, ‘Light_Modulation’, ‘Low_Frequency_Burst’, ‘Low_Frequency_Lines’, ‘No_Glitch’, ‘None_of_the_Above’, ‘Paired_Doves’, ‘Power_Line’, ‘Repeating_Blips’, ‘Scattered_Light’, ‘Scratchy’, ‘Tomte’, ‘Violin_Mode’, ‘Wandering_Line’, ‘Whistle’] contain the machine learning confidence for a glitch being in a particular Gravity Spy class (the confidence in all these columns should sum to unity). These use the original 22 classes in all cases.</p> <p>[‘ml_label’, ‘ml_confidence’] provide the machine-learning predicted label for each glitch, and the machine learning confidence in its classification. </p> <p>[‘url1’, ‘url2’, ‘url3’, ‘url4’] are the links to the publicly-available <a href="https://gwdetchar.readthedocs.io/en/stable/omega/">Omega scans</a> for each glitch. ‘url1’ shows the glitch for a duration of 0.5 seconds, ‘url2’ for 1 seconds, ‘url3’ for 2 seconds, and ‘url4’ for 4 seconds.</p> <p>For the most recently uploaded training set used in Gravity Spy machine learning algorithms, please see <a href="https://zenodo.org/record/1486046#.YZfcar3MJqs">Gravity Spy Training Set</a> on Zenodo. </p> <p><br> For detailed information on the training set used for the original Gravity Spy machine learning paper, please see <a href="https://zenodo.org/record/1476156#.YZfchL3MJqs">Machine learning for Gravity Spy: Glitch classification and dataset</a> on Zenodo.</p>
Machine learning for Gravity Spy: Glitch classification and dataset
<p>We present the first version of the training set used in the Gravity Spy citizen science project. This training set, discussed in detail <a href="https://www.sciencedirect.com/science/article/pii/S0020025518301634">here</a>, was utilized to train the convolutional neural network employed in the Gravity Spy project. We anticipate moving forward to release more labelled Gravity Spy data sets, including a refined version of this training set which can be found here <a href="https://doi.org/10.5281/zenodo.1476551">10.5281/zenodo.1476551</a>, and data sets containing the annotations provided by our citizen science volunteers.</p> <p><strong>Data Set Information</strong></p> <p>There are three files provided in this data set</p> <ul> <li><strong>trainingset_v1d0_metadata.csv</strong> <ul> <li>This file has three columns, <em>gravityspy_id, label, </em>and <em>sample_type.</em><em> gravityspy_id </em>is the unique 10 character hash given to every Gravity Spy sample. <em>label</em> is the string label of the sample. <em>sample_type </em>indicates whether this sample was used in the paper for testing training or validating the models. This is provided for those who would like to do direct comparisons to the network described in the paper.</li> </ul> </li> <li><strong>trainingsetv1d0.h5</strong> <ul> <li>This file contains the exact arrays used in the paper for every Gravity Spy sample. Each Gravity Spy sample is defined by four different images with varying temporal duration, <em>0.5, 1.0, 2.0, and 4.0</em> second, respectively. This also determines the naming conventions of the PNGs: <em>interferometer_gravityspyid_spectrogram_duration.png (e.g. H1_Fv3p6eROvA_spectrogram_0.5.png, H1_Fv3p6eROvA_spectrogram_1.0.png, H1_Fv3p6eROvA_spectrogram_2.0.png, H1_Fv3p6eROvA_spectrogram_4.0.png</em>).</li> <li>This file contains all the information needed for each sample in the Gravity Spy dataset (i.e. the label, the sample type of the sample, the unique id of the sample, and the image data for that sample. <ul> <li>/1080Lines/validation/xUEyaWr34c Group<br> /1080Lines/validation/xUEyaWr34c/0.5.png Dataset {1, 140, 170}<br> /1080Lines/validation/xUEyaWr34c/1.0.png Dataset {1, 140, 170}<br> /1080Lines/validation/xUEyaWr34c/2.0.png Dataset {1, 140, 170}<br> /1080Lines/validation/xUEyaWr34c/4.0.png Dataset {1, 140, 170}</li> </ul> </li> </ul> </li> <li><strong>trainingsetv1d0.tar.gz</strong> <ul> <li>Contains the raw PNGs of the Gravity Spy training set.</li> <li>The structure of the folder is <em>/"label"/"sample_type"/"pngs"</em></li> </ul> </li> </ul> <p><strong>Data Set Parsing Information</strong></p> <p>To read and crop out the plot axis and labels of the provided PNGs, the following small python code using scikit-image should work.</p> <p>from skimage import io</p> <p>image_data = io.imread("filename_of_image")</p> <p>x=[66, 532]; y=[105, 671]</p> <p>image_data = image_data[x[0]:x[1], y[0]:y[1], :3]</p>
Gravity Spy Training Set
<p>We present an updated version of the training set used in the Gravity Spy citizen science project. This training set, curated further from the one discussed in detail <a href="https://www.sciencedirect.com/science/article/pii/S0020025518301634">here</a> and available here <a href="https://doi.org/10.5281/zenodo.1476156">10.5281/zenodo.1476156</a>, was utilized to train the convolutional neural network employed in the Gravity Spy project. We anticipate moving forward to release more labelled Gravity Spy data sets, including a data set containing the annotations provided by our citizen science volunteers.</p> <p><strong>Data Set Information</strong></p> <p>There are three files provided in this data set</p> <ul> <li><strong>trainingset_v1d1_metadata.csv</strong> <ul> <li>This file has many columns, <em>gravityspy_id, label, </em>and <em>sample_type.</em><em> gravityspy_id </em>is the unique 10 character hash given to every Gravity Spy sample. <em>label</em> is the string label of the sample. <em>sample_type </em>indicates whether this sample was used in the paper for testing training or validating the models. This is provided for those who would like to do direct comparisons to the network described in the paper.</li> <li>Additional columns contain some metadata information about the glitch <ul> <li><em>event_time,ifo,peak_time,peak_time_ns,start_time,start_time_ns,duration,search,process_id,event_id,peak_frequency,central_freq,bandwidth,channel,amplitude,snr,confidence,chisq,chisq_dof,param_one_name,param_one_value,</em></li> </ul> </li> </ul> </li> <li><em><strong>trainingsetv1d1.h5</strong></em> <ul> <li>This file contains the exact arrays used in the paper for every Gravity Spy sample. Each Gravity Spy sample is defined by four different images with varying temporal duration, <em>0.5, 1.0, 2.0, and 4.0</em> second, respectively. This also determines the naming conventions of the PNGs: <em>interferometer_gravityspyid_spectrogram_duration.png (e.g. H1_Fv3p6eROvA_spectrogram_0.5.png, H1_Fv3p6eROvA_spectrogram_1.0.png, H1_Fv3p6eROvA_spectrogram_2.0.png, H1_Fv3p6eROvA_spectrogram_4.0.png</em>).</li> <li>This file contains all the information needed for each sample in the Gravity Spy dataset (i.e. the label, the sample type of the sample, the unique id of the sample, and the image data for that sample. <ul> <li>/1080Lines/validation/xUEyaWr34c Group<br> /1080Lines/validation/xUEyaWr34c/0.5.png Dataset {1, 140, 170}<br> /1080Lines/validation/xUEyaWr34c/1.0.png Dataset {1, 140, 170}<br> /1080Lines/validation/xUEyaWr34c/2.0.png Dataset {1, 140, 170}<br> /1080Lines/validation/xUEyaWr34c/4.0.png Dataset {1, 140, 170}</li> </ul> </li> </ul> </li> <li><strong>trainingsetv1d1.tar.gz</strong> <ul> <li>Contains the raw PNGs of the Gravity Spy training set.</li> <li>The structure of the folder is <em>/"label"/"sample_type"/"pngs"</em></li> </ul> </li> </ul> <p><strong>Data Set Parsing Information</strong></p> <p>To read and crop out the plot axis and labels of the provided PNGs, the following small python code using scikit-image should work.</p> <p>from skimage import io</p> <p>image_data = io.imread("filename_of_image")</p> <p>x=[66, 532]; y=[105, 671]</p> <p>image_data = image_data[x[0]:x[1], y[0]:y[1], :3]</p>
Gravity Spy Volunteer Classifications of LIGO Glitches from Observing Runs O1, O2, O3a, and O3b
<p>This dataset contains machine learning and volunteer classifications from the <a href="https://www.zooniverse.org/projects/zooniverse/gravity-spy">Gravity Spy project</a>. It includes glitches from observing runs <a href="https://doi.org/10.7935/K57P8W9D">O1</a>, <a href="https://doi.org/10.7935/CA75-FM95">O2</a>, <a href="https://doi.org/10.7935/nfnt-hm34">O3a</a> and <a href="https://doi.org/10.7935/pr1e-j706">O3b</a> that received at least one classification from a registered volunteer in the project. It also indicates glitches that are nominally retired from the project using our default set of retirement parameters, which are described below. See more details in the <a href="https://ui.adsabs.harvard.edu/abs/2017CQGra..34f4003Z/abstract">Gravity Spy Methods paper</a>. </p> <p>When a particular subject in a citizen science project (in this case, glitches from the LIGO datastream) is deemed to be classified sufficiently it is "retired" from the project. For the Gravity Spy project, retirement depends on a combination of both volunteer and machine learning classifications, and a number of parameterizations affect how quickly glitches get retired. For this dataset, we use a default set of retirement parameters, the most important of which are: </p> <ol> <li>A glitches must be classified by at least 2 registered volunteers</li> <li>Based on both the initial machine learning classification and volunteer classifications, the glitch has more than a 90% probability of residing in a particular class</li> <li>Each volunteer classification (weighted by that volunteer's confusion matrix) contains a weight equal to the initial machine learning score when determining the final probability</li> </ol> <p>The choice of these and other parameterization will affect the accuracy of the retired dataset as well as the number of glitches that are retired, and will be explored in detail in an upcoming publication (Zevin et al. in prep). </p> <p>The dataset can be read in using e.g. Pandas: <br> ```<br> import pandas as pd<br> dataset = pd.read_hdf('<a href="https://zenodo.org/api/files/512bfa79-dfbc-4af0-b563-9fdd06edcb16/retired_fulldata_min2_max50_ret0p9.hdf5?versionId=7f568823-76d9-4452-8553-c1eee5993f81">retired_fulldata_min2_max50_ret0p9.hdf5</a>', key='image_db')<br> ```<br> Each row in the dataframe contains information about a particular glitch in the Gravity Spy dataset. </p> <p><strong>Description of series in dataframe</strong></p> <ul> <li>['1080Lines', '1400Ripples', 'Air_Compressor', 'Blip', 'Chirp', 'Extremely_Loud', 'Helix', 'Koi_Fish', 'Light_Modulation', 'Low_Frequency_Burst', 'Low_Frequency_Lines', 'No_Glitch', 'None_of_the_Above', 'Paired_Doves', 'Power_Line', 'Repeating_Blips', 'Scattered_Light', 'Scratchy', 'Tomte', 'Violin_Mode', 'Wandering_Line', 'Whistle'] <ul> <li>Machine learning scores for each glitch class in the trained model, which for a particular glitch will sum to unity</li> </ul> </li> <li>['ml_confidence', 'ml_label'] <ul> <li>Highest machine learning confidence score across all classes for a particular glitch, and the class associated with this score</li> </ul> </li> <li>['gravityspy_id', 'id'] <ul> <li>Unique identified for each glitch on the Zooniverse platform ('gravityspy_id') and in the Gravity Spy project ('id'), which can be used to link a particular glitch to the <a href="https://zenodo.org/record/5649212#.YfLNjVjMLzc">full Gravity Spy dataset</a> (which contains GPS times among many other descriptors)</li> </ul> </li> <li>['retired'] <ul> <li>Marks whether the glitch is retired using our default set of retirement parameters (1=retired, 0=not retired)</li> </ul> </li> <li>['Nclassifications'] <ul> <li>The total number of classifications performed by registered volunteers on this glitch</li> </ul> </li> <li>['final_score', 'final_label'] <ul> <li>The final score (weighted combination of machine learning and volunteer classifications) and the most probable type of glitch</li> </ul> </li> <li>['tracks'] <ul> <li>Array of classification weights that were added to each glitch category due to each volunteer's classification</li> </ul> </li> </ul> <p> </p> <p>```<br> For machine learning classifications on all glitches in O1, O2, O3a, and O3b, please see <a href="https://zenodo.org/record/5649212#.YfLNjVjMLzc">Gravity Spy Machine Learning Classifications</a> on Zenodo</p> <p>For the most recently uploaded training set used in Gravity Spy machine learning algorithms, please see <a href="https://zenodo.org/record/1486046#.YZfcar3MJqs">Gravity Spy Training Set</a> on Zenodo.</p> <p>For detailed information on the training set used for the original Gravity Spy machine learning paper, please see <a href="https://zenodo.org/record/1476156#.YZfchL3MJqs">Machine learning for Gravity Spy: Glitch classification and dataset</a> on Zenodo. </p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.