Find research datasets worth reusing

opencc-by-4.0Oct 2024View details →

Data and code for: "Global Sampling Decline Erodes Science Potential of Natural History Collections"

# GBIF Specimen Data Analysis and Forecasting ## Version 2 - modified date ranges for figures 1 and 2 in response to reviewer comments This repository contains the code and data for analysing and forecasting trends in Global Biodiversity Information Facility (GBIF) specimen records across three major taxonomic groups: Chordata, Arthropoda, and Plantae.  The analysis pipeline includes data cleaning, anomaly detection, primary analyses, and forecasting based on historical database snapshots. These scripts and data correspond to analyses in the following manuscript: Global Sampling Decline Erodes Science Potential of Natural History Collections Authors: Owen Forbes Andrew G. Young Peter H. Thrall ## Repository Structure The repository consists of three main Quarto (.qmd) scripts and associated data files: 1. `1_DataCleaning_Forbes-et-al_2025.qmd`: Data cleaning and anomaly detection 2. `2_PrimaryAnalyses_Forbes-et-al_2025.qmd`: Primary analyses and visualisation 3. `3_SnapshotsForecasting_Forbes-et-al_2025.qmd`: Historical snapshot analysis and forecasting ## Requirements - R (version 4.3.2 or later) - Required R packages:   - tidyverse (v2.0.0) - for data manipulation and visualization   - readr (v2.1.5) - for reading CSV/TSV files   - ggplot2 (v3.4.0 or v3.5.0) - for creating visualizations   - rnaturalearth (v1.0.1) - for accessing natural earth map data   - dplyr (v1.1.0 or v1.1.4) - for data manipulation   - countrycode (v1.6.0) - for converting country names and codes   - spdep (v1.3-3) - for spatial dependence modeling   - sp (v1.6-0 or v2.1-3) - for spatial data manipulation   - sf (v1.0-15 or v1.0-16) - for simple features access   - data.table (v1.14.8) - for fast aggregation of large data   - lubridate (v1.9.2) - for date-time manipulation   - viridis (v0.6.3) - for color palettes   - gridExtra (v2.3) - for arranging multiple plots   - ggpubr (v0.6.0) - for creating publication-ready plots   - zoo (v1.8-12) - for time series, including moving averages   - scales (v1.3.0) - for graphical scales   - forecast (v8.22.0) - for ARIMA forecast models   - purrr (v1.0.2) - for mapping custom forecast function onto each dataset   - arrow - for working with parquet files Install these packages before running the scripts. ## How to Use 1. Download this repository to your local machine. 2. Set your working directory to the location of the scripts. 3. Download raw datasets from GBIF (as required) 4. Ensure all required R packages are installed. 5. Run the scripts in RStudio or your preferred R environment. ### Data Cleaning (`1_DataCleaning_Forbes-et-al_2025.qmd`) This script cleans the raw GBIF data and identifies anomalies. It produces files containing indexes of dataset records to be removed, which are used in subsequent analyses. **Note**: The raw GBIF exported datasets for contemporary records are not included in this repository due to file size constraints. Download them from the GBIF links provided in the script and place them in the `data/` directory. ### Primary Analyses (`2_PrimaryAnalyses_Forbes-et-al_2025.qmd`) This script performs the main analyses and generates visualisations. It uses the outputs from the data cleaning script to filter anomalous records. To reproduce all analysis stages from the original raw .csv files: - Start at the chunks labelled "DATA LOAD AND FILTERING". - Run the pipeline for non-spatial analyses before spatial analyses. - Due to memory constraints, it's recommended to run analyses for one taxonomic group and one analysis stream at a time. To skip to plot generation: - Navigate to sections tagged as "@! SKIP TO PLOTTING !@". - Ensure all required analysis output files are in the `data/` directory. ### Forecasting (`3_SnapshotsForecasting_Forbes-et-al_2025.qmd`) This script analyses historical GBIF database snapshots and forecasts future growth. It uses the cleaned snapshot data produced by the data cleaning script. ## Data Files ### GBIF Exports - Raw Data (not included on Zenodo due to file size, please download directly from GBIF) - `0016915-240425142415019.csv` for Chordata -  https://www.gbif.org/occurrence/download/0016915-240425142415019 - `0016914-240425142415019.csv` for Plantae - https://www.gbif.org/occurrence/download/0016914-240425142415019  - `0016913-240425142415019.csv` for Arthropoda - https://www.gbif.org/occurrence/download/0016913-240425142415019 ### Included Data Files #### Raw Data - `GBIF_snapshots.parquet` # Historical snapshots RAW dataset (arrow/parquet format) - `GBIF_integer_to_datasetKey.tsv` # Mapping old dataset IDs onto new datasetKey field #### Contemporary Datasets - data cleaning outputs - `chordata_counts_to_highlight_030724` # List of anomalous Chordata dataset + year indexes to filter - `arthropoda_counts_to_highlight_OG_030724` # List of anomalous Arthropoda dataset + year indexes to filter - `plantae_counts_to_highlight_030724` # List of anomalous Plantae dataset + year indexes to filter #### Cleaned Snapshots - `plantae_snapshots_filter_threshold_IN_040924` # Cleaned Plantae snapshots - `arthropoda_snapshots_filter_threshold_IN_040924` # Cleaned Arthropoda snapshots - `chordata_snapshots_filter_threshold_IN_040924` # Cleaned Chordata snapshots - `gbif_dates_df_anomaly_filtered_090724` # Anomaly-filtered snapshots (combined dataset) - `gbif_dates_df_anomalies_highlighted_090724` # Anomalies highlighted snapshots (combined dataset) #### Analysis Outputs - for skipping straight to plot/figure generation - `arthropoda_specimens_per_year_080724` # Arthropoda specimen counts per year - `arthropoda_unique_species_per_year_080724` # Arthropoda unique species counts per year - `arthropoda_grid_counts_080724` # Arthropoda grid counts - `chordata_specimens_per_year_080724` # Chordata specimen counts per year - `chordata_unique_species_per_year_080724` # Chordata unique species counts per year - `chordata_grid_counts_080724` # Chordata grid counts - `plantae_specimens_per_year_080724` # Plantae specimen counts per year - `plantae_unique_species_per_year_080724` # Plantae unique species counts per year - `plantae_grid_counts_080724` # Plantae grid counts - `chordata_continent_count_080724` # Chordata continent-specific counts - `arthropoda_continent_count_080724` # Arthropoda continent-specific counts - `plantae_continent_count_080724` # Plantae continent-specific counts

opencc-by-4.0Oct 2021View details →

Challenges of sampling and how phylogenetic comparative methods help: Supplementary data

Supplementary data and results files for the paper: Macklin-Cordes, Jayden L. & Erich R. Round (2022). Challenges of sampling and how phylogenetic comparative methods help: With a case study of the Pama-Nyungan laminal contrast. Linguistic Typology (advance online publication). <a href="https://doi.org/10.1515/lingty-2021-0025">https://doi.org/10.1515/lingty-2021-0025</a>

opencc-by-4.0Nov 2021View details →

Supporting data (structural data, sample descriptions, microprobe data)

Supporting data for the manuscript entitled "Tectonic evolution of the Nevado-Filábride complex (Sierra de los Filábres, Southeastern Spain): insights from new structural and geochronological data", submitted to Tectonics.

opencc-by-4.0Nov 2021View details →

Sample data FRPPIXEL

Sample data containing some 8080 fires from September 1st, 2020 extracted from EUMESAT Surface Analysis, Satellite Application Facility on Land Surface Analysis (<a href="https://www.eumetsat.int/lsa-saf">LSA SAF</a>) fire database.

opencc-by-4.0Nov 2021View details →

Audio samples from generative models trained on the TIMIT speech data.

This is a posting of audio snippets to accompany the paper "Benchmarking Generative Latent Variable Models for Speech". The snippets include samples and reconstructions. All samples are completely unconditional and utilise only the prior internal representations learned by the model. Reconstructions are computed from a given test audio snippet by first encoding it to a learned representation and then decoding that to a reconstruction of the audio. All models are trained on the TIMIT speech dataset (<a href="https://catalog.ldc.upenn.edu/LDC93s1">https://catalog.ldc.upenn.edu/LDC93s1</a>). Some snippets are from models trained at different temporal resolutions denoted by `s1` and `s64`. We refer to the paper for details. The files include: <ul> <li>`clockwork-vae-s64-reconstruction-*` <ul> <li>Four reconstructions using a two-layered Clockwork VAE trained with temporal resolution s=64.</li> </ul> </li> <li>`clockwork-vae-s64-sample-*` <ul> <li>Four samples from the prior of a Clockwork VAE trained with temporal resolution s=64.</li> </ul> </li> <li>`original-*` <ul> <li>Four original samples from TIMIT corresponding in pairs to the reconstructions.</li> </ul> </li> <li>`vrnn-s64-sample-*` <ul> <li>Two samples from the prior of a VRNN trained with temporal resolution s=64.</li> </ul> </li> <li>`vrnn-s1-sample-*` <ul> <li>Two samples from the prior of a VRNN trained with temporal resolution s=1.</li> </ul> </li> <li>`srnn-s64-sample-*` <ul> <li>Two samples from the prior of a SRNN trained with temporal resolution s=64.</li> </ul> </li> <li>`srnn-s1-sample-*` <ul> <li>Two samples from the prior of a SRNN trained with temporal resolution s=1.</li> </ul> </li> <li>`wavenet-s64-sample-*` <ul> <li>Four samples from a WaveNet trained with temporal resolution s=1.</li> </ul> </li> <li>`wavenet-s1-sample-*` <ul> <li>Two samples from a WaveNet trained with temporal resolution s=64.</li> </ul> </li> </ul>

Sample based prevalence data complementing the European Union One Health 2020 Zoonoses Report - Norway

This dataset contains monitoring data on zoonoses and zoonotic agents under the Directive 2003/99/EC. This Directive requires Member Sates (MSs) to collect, evaluate and report data on zoonoses and zoonotic agents. MSs can also report monitoring data and information on some other pathogenic microbiological agents in foodstuffs. Relevant EU legislation: Commission Regulation (EC) No 2073/2005,Commission Regulation (EC) No 1441/2007, Commission Regulation (EU) No 1086/2011, Commission Regulation (EU) No 209/2013, Commission Regulation(EU) No 217/2014.

Sample based prevalence data complementing the European Union One Health 2020 Zoonoses Report - Luxembourg

Sample based prevalence data complementing the European Union One Health 2020 Zoonoses Report - the United kingdom

Sample based prevalence data complementing the European Union One Health 2020 Zoonoses Report - Finland

Sample based prevalence data complementing the European Union One Health 2020 Zoonoses Report - Sweden

Supplementary Data for "Sequencing the Pandemic: Rapid and High-Throughput Processing and Analysis of COVID-19 Clinical Samples for 21st Century Public Health"

Supplementary material for F1000 methods manuscript. Includes raw sequencing metrics for two COVID sequencing methodologies, as well as a complete cost breakdown for each methodology.

Netflow data with sampling 500 for training (D2)

NetFlow traffic generated using DOROTHEA (DOcker-based fRamework fOr gaTHering nEtflow trAffic) NetFlow is a network protocol developed by Cisco for the collection and monitoring of network traffic flow data generated. A flow is defined as a unidirectional sequence of packets with some common properties that pass through a network device. NetFlow flows have been captured with sampling 500 at the packet level. A sampling means that 1 out of every X packets is selected to be flow while the rest of the packets are not valued. The version of NetFlow used to build the datasets is 5.

Netflow data with sampling 250 for test (D4)

NetFlow traffic generated using DOROTHEA (DOcker-based fRamework fOr gaTHering nEtflow trAffic) NetFlow is a network protocol developed by Cisco for the collection and monitoring of network traffic flow data generated. A flow is defined as a unidirectional sequence of packets with some common properties that pass through a network device. NetFlow flows have been captured with sampling 250 at the packet level. A sampling means that 1 out of every X packets is selected to be flow while the rest of the packets are not valued. The version of NetFlow used to build the datasets is 5.

Netflow data with sampling 500 for test (D5)

Netflow data with sampling 250 for training (D1)

NetFlow traffic generated using DOROTHEA (DOcker-based fRamework fOr gaTHering nEtflow trAffic) NetFlow is a network protocol developed by Cisco for the collection and monitoring of network traffic flow data generated. A flow is defined as a unidirectional sequence of packets with some common properties that pass through a network device. NetFlow flows have been captured with sampling 250 at the packet level. A sampling means that 1 out of every X packets is selected to be flow while the rest of the packets are not valued. The version of NetFlow used to build the datasets is 5.

Netflow data with sampling 1000 for training (D3)

NetFlow traffic generated using DOROTHEA (DOcker-based fRamework fOr gaTHering nEtflow trAffic) NetFlow is a network protocol developed by Cisco for the collection and monitoring of network traffic flow data generated. A flow is defined as a unidirectional sequence of packets with some common properties that pass through a network device. NetFlow flows have been captured with sampling 1000 at the packet level. A sampling means that 1 out of every X packets is selected to be flow while the rest of the packets are not valued. The version of NetFlow used to build the datasets is 5.

Netflow data with sampling collected from RedCAYLE (D7)

NetFlow traffic generated using DOROTHEA (DOcker-based fRamework fOr gaTHering nEtflow trAffic) NetFlow is a network protocol developed by Cisco for the collection and monitoring of network traffic flow data generated. A flow is defined as a unidirectional sequence of packets with some common properties that pass through a network device. NetFlow flows have been captured with sampling 1000 at the packet level. A sampling means that 1 out of every X packets is selected to be flow while the rest of the packets are not valued. The version of NetFlow used to build the datasets is 5.

Sampled ΔH/Δλ and ΔH data from ABFE calculations of 19 ligands bound to MCL-1

Supplementary Information: "Evaluating the use of absolute binding free energy in the fragment optimization process" Included are the ABFE raw free energy samples for multiple replicates (labelled by `run` number) of 19 ligands to bound MCL-1. These ligands are originally detailed by Friberg et al. (https://doi.org/10.1021/jm301448p). All samples are provided as a set of `.xvg` files as generated by GROMACS 2021 (https://doi.org/10.5281/zenodo.5849961). The `.xvg` files are labelled as dhdl.N.xvg where N represents the λ state the free energy values were sampled from. The `.xvg` files contain both ΔH/Δλ and ΔH values, please see the header of each files for more information. Samples detailing the partial decoupling of the ligand from the protein-ligand complex are contained within the `complex` folder. These consist of an orientational restraint addition step (found within the `restraints-xvg` folders), charge annihilation step (found within the `coul-xvg` folders), and Van der Waals decoupling step (found within the `vdw-xvg` folders). Samples detailing the partial decoupling of the ligand from solvent are contained within the `ligand` folder and consist of a charge annihilation step (found within the individual `coul-xvg` folders) and a Van der Waals decoupling step (found within the individual `vdw-xvg` folders).

Sampled ΔH/Δλ and ΔH data from ABFE calculations (using standard atomic masses) of 10 ligands bound to Cyclophilin D

Supplementary Information: "Evaluating the use of absolute binding free energy in the fragment optimization process" Included are the ABFE raw free energy samples for multiple replicates (labelled by `run` number) of 10 ligands to bound Cyclophilin D. These ligands are originally detailed by Grädler et al. (https://doi.org/10.1016/j.bmcl.2019.126717). Unlike other datasets in this work, which employed hydrogen mass repartitioning, the ligands here were calculated using standard atomic masses. All samples are provided as a set of `.xvg` files as generated by GROMACS 2021 (https://doi.org/10.5281/zenodo.5849961). The `.xvg` files are labelled as dhdl.N.xvg where N represents the λ state the free energy values were sampled from. The `.xvg` files contain both ΔH/Δλ and ΔH values, please see the header of each files for more information. Samples detailing the partial decoupling of the ligand from the protein-ligand complex are contained within the `complex` folder. These consist of an orientational restraint addition step (found within the `restraints-xvg` folders), charge annihilation step (found within the `coul-xvg` folders), and Van der Waals decoupling step (found within the `vdw-xvg` folders). Samples detailing the partial decoupling of the ligand from solvent are contained within the `ligand` folder and consist of a charge annihilation step (found within the individual `coul-xvg` folders) and a Van der Waals decoupling step (found within the individual `vdw-xvg` folders).

Sampled ΔH/Δλ and ΔH data from ABFE calculations of 12 ligands bound to PWWP1

Supplementary Information: "Evaluating the use of absolute binding free energy in the fragment optimization process" Included are the ABFE raw free energy samples for multiple replicates (labelled by `run` number) of 12 ligands to bound PWWP1. These ligands are originally detailed by Böttcher et al. (https://doi.org/10.1038/s41589-019-0310-x). All samples are provided as a set of `.xvg` files as generated by GROMACS 2021 (https://doi.org/10.5281/zenodo.5849961). The `.xvg` files are labelled as dhdl.N.xvg where N represents the λ state the free energy values were sampled from. The `.xvg` files contain both ΔH/Δλ and ΔH values, please see the header of each files for more information. Samples detailing the partial decoupling of the ligand from the protein-ligand complex are contained within the `complex` folder. These consist of an orientational restraint addition step (found within the `restraints-xvg` folders), charge annihilation step (found within the `coul-xvg` folders), and Van der Waals decoupling step (found within the `vdw-xvg` folders). Samples detailing the partial decoupling of the ligand from solvent are contained within the `ligand` folder and consist of a charge annihilation step (found within the individual `coul-xvg` folders) and a Van der Waals decoupling step (found within the individual `vdw-xvg` folders).