Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
1,454
datasets available to search
ShareScore release 0.9.0
Dataset results
1,454 results for “large scale”
TBPos: Dataset for Large-Scale Precision Visual Localization (database files)
<p>Large-scale dataset for visual localization, provided in the format of the well-known InLoc dataset (Taira et al, 2018). Contains co-registered RGB point clouds and a script for generating the rest of the 'database' files for visual localization by the InLoc algorithm. Note: query images are provided in a separate repository.</p>
EMBERSim: A Large-Scale Databank for Boosting Similarity Search in Malware Analysis
<p>In recent years there has been a shift from heuristics-based malware detection towards machine learning, which proves to be more robust in the current heavily adversarial threat landscape. While we acknowledge machine learning to be better equipped to mine for patterns in the increasingly high amounts of similar-looking files, we also note a remarkable scarcity of the data available for similarity-targeted research. Moreover, we observe that the focus in the few related works falls on quantifying similarity in malware, often overlooking the clean data. This one-sided quantification is especially dangerous in the context of detection bypass. We propose to address the deficiencies in the space of similarity research on binary files, starting from EMBER — one of the largest malware classification data sets. We enhance EMBER with similarity information as well as malware class tags, to enable further research in the similarity space. Our contribution is threefold: (1) we publish EMBERSim, an augmented version of EMBER, that includes similarity-informed tags; (2) we enrich EMBERSim with automatically determined malware class tags using the open-source tool AVClass on VirusTotal data and (3) we describe and share the implementation for our class scoring technique and leaf similarity method.</p>
Public Dataset for "Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior"
<p>Dataset for the "Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior" paper, published in ICWSM 2018. The full text of the paper can be found <a href="https://arxiv.org/pdf/1802.00393.pdf">here</a>. </p> <p>The dataset provided here includes an updated version of the original dataset, with ~100k tweets annotated using the CrowdFlower platform: </p> <ul> <li> <p>hatespeech_id_label_PUBLIC_100K.csv: contains ~100K rows, where every row consists of a unique Tweet ID. </p> </li> <li> <p>hatespeech_text_label_vote_RESTRICTED_100K.csv: contains ~100K rows, where every row consists of the tweet text, its label according to majority annotation and the number of majority annotators. Available only <a href="https://zenodo.org/record/3706866#.Xmkh6i97FQI">here</a>.</p> </li> <li> <p>retweets.csv: contains ~2K rows, where every row consists of the row number in the hatespeech_text_label_vote_RESTRICTED_100K.csv file which is the first occurrence of a Tweet text followed by comma-separated row numbers of all other occurrences of the same Tweet text in the same file. There are ~8K other occurrences due to retweets. Available only <a href="https://zenodo.org/record/3706866#.Xmkh6i97FQI">here</a>.</p> </li> </ul> <p> </p> <p>UPDATE: It has come to our understanding that a number of the tweets are not available anymore for download on Twitter. Therefore, we provide <a href="https://zenodo.org/record/3706866#.YYLG6S8RqjQ">here </a>the hatespeech_text_label_vote_RESTRICTED_100K file with the full ~100K tweet texts, their associated majority label, and the number of votes for the majority label. The tweets are shuffled so that there is no connection between tweet IDs and texts (in order to be in line with the T&C of Twitter). </p> <p>Please cite the paper in any published work that uses any of these resources. </p> <p>@inproceedings{founta2018large, <br> title={Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior}, <br> author={Founta, Antigoni-Maria and Djouvas, Constantinos and Chatzakou, Despoina and Leontiadis, Ilias and Blackburn, Jeremy and Stringhini, Gianluca and Vakali, Athena and Sirivianos, Michael and Kourtellis, Nicolas}, <br> booktitle={11th International Conference on Web and Social Media, ICWSM 2018}, <br> year={2018}, <br> organization={AAAI Press} <br> } </p> <p>For any further questions contact a.m.founta at gmail dot com AND markos.charalambous at eecei dot cut dot ac dot cy </p>
Data from a Large-Scale Experiment to Evaluate the Effects of Trapping to Control Muskrats (Ondatra zibethicus) in The Netherlands
<p>This data set supports the publication 'A Large-Scale Experiment to Evaluate the Effects of Trapping to Control Muskrats (Ondatra zibethicus) in The Netherlands' by Daan Bos, Emiel van Loon, Erik Klop and Ron Ydenberg. (the paper was accepted for publication in Wildlife Society Bulletin in 2020)</p> <p>The Muskrat is an invasive species in Europe and in the Netherlands muskrat burrowing can compromise the integrity of dykes and hence poses a public safety threat. For that reason a control programme has been in effect since the arrival of the species in 1941. To investigate the relation between catch and effort and enhance prediction models, a large randomized controlled experiment was designed and conducted from 2013 till 2016. The publication by Bos et al. (2020) analyses the experimental results and here we present and document the experimental data. See the readme.md file for further information.</p>
Dataset for "Large scale patterns and drivers of the diving behavior of gill-breathing large pelagic predators"
<p>This dataset includes all supporting data and scritps to generate figure panels in the paper "Large scale patterns and drivers of the diving behavior of gill-breathing large pelagic predators" (A. Nuno, J. Guiet, B. Baranek and D. Bianchi)</p>
Replication Data & Code - Large-scale land acquisitions exacerbate local land inequalities in Tanzania
<h3><strong>Reference</strong></h3><p>Sullivan J.A., Samii, C., Brown, D., Moyo, F., Agrawal, A. 2023. Large-scale land acquisitions exacerbate local farmland inequalities in Tanzania. Proceedings of the National Academy of Sciences 120, e2207398120. <a href="https://doi.org/10.1073/pnas.2207398120">https://doi.org/10.1073/pnas.2207398120</a> </p><h3><strong>Abstract</strong></h3><p>Land inequality stalls economic development, entrenches poverty, and is associated with environmental degradation. Yet, rigorous assessments of land-use interventions attend to inequality only rarely. A land inequality lens is especially important to understand how recent large-scale land acquisitions (LSLAs) affect smallholder and indigenous communities across as much as 100 million hectares around the world. This paper studies inequalities in land assets, specifically landholdings and farm size, to derive insights into the distributional outcomes of LSLAs. Using a household survey covering four pairs of land acquisition and control sites in Tanzania, we use a quasi-experimental design to characterize changes in land inequality and subsequent impacts on well-being. We find convincing evidence that LSLAs in Tanzania lead to both reduced landholdings and greater farmland inequality among smallholders. Households in proximity to LSLAs are associated with 21.1% (<i>P</i> = 0.02) smaller landholdings while evidence, although insignificant, is suggestive that farm sizes are also declining. Aggregate estimates, however, hide that households in the bottom quartiles of farm size suffer the brunt of landlessness and land loss induced by LSLAs that combine to generate greater farmland inequality. Additional analyses find that land inequality is not offset by improvements in other livelihood dimensions, rather farm size decreases among households near LSLAs are associated with no income improvements, lower wealth, increased poverty, and higher food insecurity. The results demonstrate that without explicit consideration of distributional outcomes, land-use policies can systematically reinforce existing inequalities.</p><h3><strong>Replication Data</strong></h3><p>We include anonymized household survey data from our analysis to support open and reproducible science. In particular, we provide i) an anoymized household dataset collected in 2018 (n=994) for households nearby (treatment) and far-away from (control) LSLAs and ii) a household dataset collected in 2019 (n=165) within the same sites. For the 2018 surveys, several anonymized extracts are provided including an imputed (n=10) dataset to fill in missing data that was used for the main analysis. This data can be found in the <i>hh_data</i> folder and includes:</p><ul><li><i>hh_imputed10_2018:</i> anonymized household dataset for 2018 with variables used for the main analysis where missing data was imputed 10 times</li><li><i>hh_compensation_2018:</i> anonymized household extract for 2018 representing household benefits and compensation directly received from LSLAs</li><li><i>hh_migration_2018:</i> anonymized household extract for 2018 representing household migration behavior following LSLAs</li><li><i>hh_rsdata_2018:</i> extracted remote sensing data at the household geo-location for 2018</li><li><i>hh_land_<strong>2019</strong>:</i><strong> </strong> anonymized household extract for <strong>2019 </strong>of land variables</li></ul><p>Our analysis also incorporates data from the Living Standards Measurement Survey (LSMS) collected by the World Bank (found in <i>lsms_data</i> folder). We've provide sub-modules from the LSMS dataset relevant to our analysis but the full datasets can be access through the World Bank's Microdata Library (https://microdata.worldbank.org/index.php/home). </p><p>Across several analyses we use the LSLA boundaries for our four selected sites. We provide a shapefile for the LSLA boundaries in the <i>gis_data</i> folder.</p><p>Finally, our data replication includes several model outputs (found in <i>mod_outputs)</i>, particularly those that are lengthy to run in R. These datasets can optionally be loaded into R rather than re-running analysis using our <i>main_analysis.Rmd</i> script. </p><h3><strong>Replication Code</strong></h3><p>We provide replication code in the form of R Markdown (.Rmd) or R (.R) files. Alongside the replication data, this can be used to reproduce main figures, table, supplementary materials, and results reported in our article. Scripts include:</p><ul><li><i>main_analysis.Rmd:</i> main analysis supporting the finding, graphs, and tables reported in our main manuscript</li><li><i>compensation.R:</i> analysis of benefits and compensation received directly by households from LSLAs</li><li><i>landvalue.R:</i> analysis of household land values as a function of distance from LSLAs</li><li><i>migration.R:</i> analysis of migration behavior following LSLAs</li><li><i>selection_bias.R:</i> analysis of LSLA selection bias between control and treatment enumeration areas</li></ul>
A Large Scale Side-Scan Sonar Dataset of Seafloor Sediments for Self-Supervised Pretraining
<p>This dataset serves as an extension to the dataset part of "A convolutional vision transformer for semantic segmentation of side-scan sonar data" published in Ocean Engineering, Volume 86, part 2, 15 October 2023,<strong> </strong>DOI: <a href="https://www.sciencedirect.com/science/article/pii/S0029801823020310">10.1016/j.oceaneng.2023.115647</a> for self-supervised pretraining.</p><p>This dataset consists of patches of side-scan sonar waterfalls collected along the coast of Catalunya during an extensive survey. The waterfalls were partitioned in batches of 384 lines to generate images of size 384 × 384 with a 192 pixel-overlap along-track and across-track. This resulted in a total of 434,164 images capturing various seafloor types including rocky bottoms, sand ripples, detrital funds, posidonia, cymocea, mud, corals, artificial reefs etc.</p><p>Additional tools for using the data for self-supervised pretraining can be found under <a href="https://github.com/DeeperSense/deepersense-seafloorscan">https://github.com/DeeperSense/deepersense-seafloorscan</a></p><p> </p><p><strong>Acknowledgements</strong></p><p>The data in this repository were collected by Tecnoambiente SL as part of the project DeeperSense that received funding from the European Commission. Program H2020-ICT-2020-2 ICT-47-2020. Project Number: 101016958.</p>
Large-scale attributed graph & hypergraph datasets: TWeibo, Amazon2M, Amazon, MAG-PM
<p>Here we provide additional large-scale datasets used in our work "A Versatile Framework for Attributed Network Clustering via K-Nearest Neighbor Augmentation", along with the index files for constructing KNN graphs using ScaNN and Faiss.</p> <p>Usage:</p> <p>cd ANCKA/</p> <p>unzip ~/Download_path/ANCKA_data.zip -d data/</p>
The Femern-project: a large-scale excavation of a Stone Age landscape - supplementary data
<p>This dataset contains all radiocarbon dates from the Femern project.</p> <p>Please cite the dataset as: </p> <p>Måge, B.T., Groß, D., Kanstrup, M. 2023. The Femern-project: a large-scale excavation of a Stone Age landscape. In: Groß, D. and Rothstein, M.: Changing Identity in a Changing World. Archaeological Studies on Human Interaction in Northern Europe around 4000 cal BC. Leiden: Sidestone, supplementary material.</p> <p>19.02.2024: Dataset updated: Wrong species ID in original dataset for AAR-27426</p>
Dataset - Papyrus 05.4 - A large scale curated dataset aimed at bioactivity predictions
<div> <p>This repository contains the version 05.5 of the Papyrus dataset, an aggregated dataset of small molecule bioactivities, as described in the article "Papyrus - A large scale curated dataset aimed at bioactivity predictions" <a href="https://doi.org/10.26434/chemrxiv-2021-1rxhk">https://doi.org/10.26434/chemrxiv-2021-1rxhk</a>.</p> <p> </p> <p>With the ongoing rapid growth of publicly available ligand-protein bioactivity data, there is a trove of valuable data that can be used to train a plethora of machine learning algorithms. However, not all data is equal in terms of size and quality and a significant portion of researchers’ time is needed to adapt the data to their needs. On top of that, finding the right data for a research question can often be a challenge on its own. To meet these challenges we have constructed the Papyrus dataset. Papyrus is comprised of around 60 million datapoints. This dataset contains multiple large publicly available datasets such as ChEMBL and ExCAPE-DB combined with several smaller datasets containing high-quality data. The aggregated data has been standardised and normalised in a manner that is suitable for machine learning. We show how data can be filtered in a variety of ways and also perform some example quantitative structure-activity relationship analyses and proteochemometric modelling. Our ambition is that this pruned data collection constitutes a benchmark set that can be used for constructing predictive models, while also providing a solid baseline for related research.</p> </div>
Accompanying data - Papyrus - A large scale curated dataset aimed at bioactivity predictions
<p><strong>Addition of supporting files:<br>- </strong>LICENSE.txt<strong><br>- </strong>data_types.json<strong><br>- </strong>data_size.json</p> <p> </p> <p><strong>Fixed version of Papyrus++ 05.5:<br>- In the previous 05.5 version </strong>data was incorrectly duplicated based on assay type. This resulted in unintended data augmentation.<br><strong>- In this fixed 05.5 version</strong> the duplicates have been eliminated, now reporting the correct amount of data per assay type.</p> <p> </p> <p>This repository contains the version 05.5 of the Papyrus dataset, an aggregated dataset of small molecule bioactivities, as described in the article "Papyrus - A large scale curated dataset aimed at bioactivity predictions" <a href="http://doi.org/10.1186/s13321-022-00672-x">http://doi.org/10.1186/s13321-022-00672-x</a>.</p> <p> </p> <p>With the ongoing rapid growth of publicly available ligand-protein bioactivity data, there is a trove of valuable data that can be used to train a plethora of machine learning algorithms. However, not all data is equal in terms of size and quality and a significant portion of researchers’ time is needed to adapt the data to their needs. On top of that, finding the right data for a research question can often be a challenge on its own. To meet these challenges we have constructed the Papyrus dataset. Papyrus is comprised of around 60 million datapoints. This dataset contains multiple large publicly available datasets such as ChEMBL and ExCAPE-DB combined with several smaller datasets containing high-quality data. The aggregated data has been standardised and normalised in a manner that is suitable for machine learning. We show how data can be filtered in a variety of ways and also perform some example quantitative structure-activity relationship analyses and proteochemometric modelling. Our ambition is that this pruned data collection constitutes a benchmark set that can be used for constructing predictive models, while also providing a solid baseline for related research.</p>
Efficient and accurate framework for genome-wide gene-environment interaction analysis in large-scale biobanks
<p>Gene-environment interaction (GxE) analysis elucidates the interplay between genetic predispositions and environmental influences, offering significant potential for precision medicine. With the increasing use of electronic health records (EHR) linked to genetic data in large-scale biobanks, genome-wide association studies (GWAS) have expanded to encompass complex traits with intricate structures, such as time-to-event and ordinal categorical traits. Although these complex traits convey more phenotypic information, most existing scalable genome-wide GxE analysis approaches only focus on quantitative or binary traits. In this work, we propose a scalable and accurate analysis framework, SPAGxE<sub>CCT</sub>, that is applicable to a wide variety of trait types. We extend SPAGxE to SPAGxE+, which can account for sample relatedness. In addition, we extend SPAGxE<sub>CCT</sub> to SPAGxEmix<sub>CCT</sub>, which accounts for population stratification and is applicable to include individuals from multiple ancestries or admixed populations. We applied SPAGxE<sub>CCT</sub>, SPAGxE+, and SPAGxEmix<sub>CCT</sub> to analyze time-to-event traits in UK Biobank. For the SPAGxE<sub>CCT</sub> analyses, 281,149 White British individuals were included. For the SPAGxE+ analyses, 337,367 WB individuals with sample relatedness were included. For the SPAGxEmix<sub>CCT</sub> analyses, 338,044 individuals from all ancestries were included. SPAGxE<sub>CCT</sub>, SPAGxE+, and SPAGxEmix<sub>CCT</sub> are computationally efficient to analyze large datasets with hundreds of thousands of individuals, can accurately control type I error rates while remaining powerful to identify novel GxE findings.</p>
Large-scale 3D building and tree datasets constructed from airborne LiDAR point clouds in Glasgow, UK
<p>This is the updated version of building 3D model data. The revision includes appending attributes to the lod1 and lod2 shapefile and creating cityjson file for each 3D building model. All 3D building models are available in mesh (.obj), multipath shapefile, and cityjson (.json) now.</p> <p><strong>IMPORTANT NOTE: We suggest using the building footprint, lod1, and lod2 data of this version (Version v4).</strong></p> <p>Urban Big Data Centre of the University of Glasgow generates 3D city models via the airborne LiDAR point clouds acquired between 2020-2021 on behalf of Glasgow City Council. It is a large-scale 3D city model containing 3D information on terrain, trees, and buildings in Glasgow City. This dataset comprises terrain, tree canopy, and building products derived from high-density airborne LiDAR point clouds. </p> <p>The terrain products include Digital Terrain Model (DTM), Digital Surface Model (DSM), and normalized Digital Surface Model (nDSM) in 0.5 m spatial resolution. The DTM and DSM rasters were provided by the vendor and nDSM rasters were obtained by subtracting DTM from DSM. Terrain products are provided in 5 km by 5 km GeoTIF format raster.</p> <p>The tree canopy products are composed of canopy height models (CHM) and tree top locations. Classified tree point clouds were applied with pit-free algorithm to generate CHM in 0.5 m grid raster in GeoTIF format [1]-[2]. Treetop locations were identified by using Local Maximum Filter based on CHM and are recorded as points in Shapefile format. The tree canopy products are provided in 5 km by 5 km tiles.</p> <p>Building 3D model products include footprint polygons with building height attributes and 3D mesh of building models in LoD1 and LoD2 levels. A series of processes such as converting building point clouds to building height models (BHM), converting BHM to polygons, and polygon regularization were conducted to obtain the building footprint polygons. Building height attributes were calculated from BHM for each footprint. The building footprint data are provided in Shapefile format. LoD1 models were generated based on the footprint and average height of the building. LoD2 models were constructed based on footprint and building point cloud with City3D tool[3]. LoD1 and LoD2 models are provided in OBJ and shapefile format. Building 3D model products are provided in 5 km by 5 km tiles. The RMSE of Euclidean distances between each point in the point cloud to the reconstructed model was calculated to evaluate the LoD2 model construction. A table of RMSE and a note for a few problematic models are provided.</p>
WikiMed and PubMedDS: Two large-scale datasets for medical concept extraction and normalization research
<p>Two large-scale, automatically-created datasets of medical concept mentions, linked to the <a href="https://uts.nlm.nih.gov/uts/umls/home">Unified Medical Language System (UMLS)</a>.</p> <p><strong>WikiMed</strong></p> <p>Derived from Wikipedia data. Mappings of Wikipedia page identifiers to UMLS Concept Unique Identifiers (CUIs) was extracted by crosswalking Wikipedia, Wikidata, Freebase, and the NCBI Taxonomy to reach existing mappings to UMLS CUIs. This created a 1:1 mapping of approximately 60,500 Wikipedia pages to UMLS CUIs. Links to these pages were then extracted as mentions of the corresponding UMLS CUIs.</p> <p>WikiMed contains:</p> <ul> <li>393,618 Wikipedia page texts</li> <li>1,067,083 mentions of medical concepts</li> <li>57,739 unique UMLS CUIs</li> </ul> <p>Manual evaluation of 100 random samples of WikiMed found 91% accuracy in the automatic annotations at the level of UMLS CUIs, and 95% accuracy in terms of semantic type.</p> <p><strong>PubMedDS</strong></p> <p>Derived from biomedical literature abstracts from <a href="https://pubmed.ncbi.nlm.nih.gov/">PubMed</a>. Mentions were automatically identified using distant supervision based on Medical Subject Heading (MeSH) headers assigned to the papers in PubMed, and recognition of medical concept mentions using the high-performance <a href="https://allenai.github.io/scispacy/">scispaCy</a> model. MeSH header codes are included as well as their mappings to UMLS CUIs.</p> <p>PubMedDS contains:</p> <ul> <li>13,197,430 abstract texts</li> <li>57,943,354 medical concept mentions</li> <li>44,881 unique UMLS CUIs</li> </ul> <p>Comparison with existing manually-annotated datasets (NCBI Disease Corpus, BioCDR, and MedMentions) found 75-90% precision in automatic annotations. Please note this dataset is <em>not </em>a comprehensive annotation of medical concept mentions in these abstracts (only mentions located through distant supervision from MeSH headers were included), but is intended as data for <em>concept n</em><em>ormalization</em> research.</p> <p>Due to its size, PubMedDS is distributed as 30 individual files of approximately 1.5 million mentions each.</p> <p><strong>Data format</strong></p> <p>Both datasets use JSON format with one document per line. Each document has the following structure:</p> <pre><code class="language-json">{ "_id": "A unique identifier of each document", "text": "Contains text over which mentions are ", "title": "Title of Wikipedia/PubMed Article", "split": "[Not in PubMedDS] Dataset split: <train/test/valid>", "mentions": [ { "mention": "Surface form of the mention", "start_offset": "Character offset indicating start of the mention", "end_offset": "Character offset indicating end of the mention", "link_id": "UMLS CUI. In case of multiple CUIs, they are concatenated using '|', i.e., CUI1|CUI2|..." }, {} ] }</code></pre> <p><strong>Version history</strong></p> <table align="left"> <thead> <tr> <th scope="col">Version</th> <th scope="col">Notes</th> </tr> </thead> <tbody> <tr> <td>1.0.0</td> <td>Initial release</td> </tr> <tr> <td>1.0.1</td> <td>Corrected duplication error in WikiMed.zip file</td> </tr> </tbody> </table> <p> </p> <p> </p> <p> </p> <p> </p>
BirdVox-296h: a large-scale dataset for detection and classification of flight calls
<p>BirdVox 296 hours dataset (BirdVox-296h)<br> ====================================</p> <p>Version 2.1, May 2022.</p> <p><br> Created By<br> ----------</p> <p>Andrew Farnsworth (1), Benjamin Mark Van Doren (1), Steve Kelling (1), Vincent Lostanlen (2), Justin Salamon (3), Aurora Cramer (4), Juan Pablo Bello (4)</p> <p>(1): Cornell Lab of Ornithology (CLO)<br> (2): Laboratoire des Sciences du Numérique de Nantes (LS2N), CNRS<br> (3): Adobe Research<br> (4): New York University</p> <p>https://wp.nyu.edu/birdvox<br> <br> </p> <p>Description<br> ---------------</p> <p>The BirdVox-296h dataset contains 148 audio recordings, each two hours in duration. These recordings come from ROBIN autonomous recording units, placed near Ithaca, NY, USA during the fall 2015. They were captured by nine different sensors, originally numbered 1, 2, 3, 4, 5, 6, 7, 8, and 10.<br> <br> Ornithologist Andrew Farnsworth used the Raven software to pinpoint and label every avian flight call in time and frequency. He found 26138 sound events, of which 21546 are flight calls from Passeriformes. Of those, 13385 are identifiable in terms of family, and 8669 are identifiable in terms of both family and species. The annotation process took over 600 hours.</p> <p>The dataset can be used, among other things, for the research, development and testing of machine listening models for bird migration monitoring.</p> <p> </p> <p>Data Files<br> ------------</p> <p>The BirdVox-296h_wav folder contains 148 recordings as WAV files, sampled at 24 kHz, with a single channel (mono). Each recording lasts exactly two hours and is named according to the following format:</p> <p>YYYY-MM-DD_hh-mm-ss_unitUU.wav</p> <p>Where Y means Year, M means Month, D means Day, h means hour, m means minute, and s means second. This date format corresponds to the start time of the recording file, expressed in Coordinated Universal Time (UTC).</p> <p>The field UU contains two digits corresponding to the identifier of the autonomous recording unit (i.e., bioacoustic sensor). UU is either equal to 01, 02, 03, 04, 05, 06, 07, 08, or 10. Note that 09 is absent from the list because sensor 09 failed during the acquisition campaign.</p> <p> </p> <p>Metadata Files<br> -------------------</p> <p>The BirdVox-296h_csv-annotations folder contains CSV files, one for each audio file. The columns of each CSV file are:</p> <p>ID,Time (s),Frequency (Hz),Taxonomy Code,Fine Label,Medium Label,Coarse Label</p> <p><br> "Taxonomy Code" is compliant with the BirdVoxClassify software: github.com/BirdVox/BirdVoxClassify</p> <p>"Fine Label", "Medium Label", and "Coarse Label" most often correspond to species, family and order respectively.</p> <p> </p> <p>The BirdVox-296h_gps-coordinates.csv file contains the approximate GPS coordinates of the sensors (latitudes and longitudes rounded to 2 decimal points) of all nine sensors.</p> <p> </p> <p> </p> <p>Conditions of Use<br> -----------------</p> <p>Dataset created by Andrew Farnsworth, Steve Kelling, Vincent Lostanlen, Justin Salamon, Aurora Cramer, and Juan Pablo Bello.</p> <p>The BirdVox-full-night dataset is offered free of charge under the terms of the Creative Commons Attribution 4.0 International (CC BY 4.0) license:<br> https://creativecommons.org/licenses/by/4.0/</p> <p>The dataset and its contents are made available on an "as is" basis and without warranties of any kind, including without limitation satisfactory quality and conformity, merchantability, fitness for a particular purpose, accuracy or completeness, or absence of errors. Subject to any liability that may not be excluded or limited by law, Cornell Lab of Ornithology is not liable for, and expressly excludes all liability for, loss or damage however and whenever caused to anyone by any use of the BirdVox-full-night dataset or any part of it.</p> <p> </p> <p>Feedback<br> -------------</p> <p>Please help us improve BirdVox-296h by sending your feedback to:<br> vincent.lostanlen@ls2n.fr and af27@cornell.edu</p> <p>In case of a problem, please include as many details as possible.</p> <p> </p> <p>Acknowledgements<br> --------------------------</p> <p>Jessie Barry, Ian Davies, Tom Fredericks, Jeff Gerbracht, Sara Keen, Holger Klinck, Anne Klingensmith, Ray Mack, Peter Marchetto, Ed Moore, Matt Robbins, Ken Rosenberg, and Chris Tessaglia-Hymes.</p> <p>We acknowledge that the land on which the data was collected is the unceded territory of the Cayuga nation, which is part of the Haudenosaunee (Iroquois) confederacy.</p>
TBGA: A Large-Scale Gene-Disease Association Dataset for Biomedical Relation Extraction
<p>This repository contains the TBGA dataset. TBGA is a large-scale, semi-automatically annotated dataset for Gene-Disease Association (GDA) extraction. The dataset consists of three text files, corresponding to train, validation, and test sets, plus an additional JSON file containing the mapping between relation names and IDs. Each record in train, validation, or test files corresponds to a single GDA extracted from a sentence. Records are represented as JSON objects with the following structure:</p> <ul> <li><strong>text:</strong> sentence from which the GDA was extracted.</li> <li><strong>relation:</strong> relation name associated with the given GDA.</li> <li><strong>h: </strong>JSON object representing the gene entity, composed of: <ul> <li><strong>id: </strong>NCBI Entrez ID associated with the gene entity.</li> <li><strong>name:</strong> NCBI official gene symbol associated with the gene entity.</li> <li><strong>pos: </strong>list consisting of starting position and length of the gene mention within text.</li> </ul> </li> <li><strong>t:</strong> JSON object representing the disease entity, composed of: <ul> <li><strong>id: </strong>UMLS CUI associated with the disease entity.</li> <li><strong>name:</strong> UMLS preferred term associated with the disease entity.</li> <li><strong>pos:</strong> list consisting of starting position and length of the disease mention within text.</li> </ul> </li> </ul> <p>TBGA contains over 200,000 instances and 100,000 bags.<br> The zip file consists of one folder, named TBGA, containing the files corresponding to the dataset.</p> <p>If you use or extend our work, please cite the following: https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-022-04646-6#citeas<br> TBGA paper can be found at: <a href="https://rdcu.be/cKkY2">https://rdcu.be/cKkY2</a><br> TBGA code is available at: https://github.com/GDAMining/gda-extraction</p>
MOBDrone: a large-scale drone-view dataset for man overboard detection
<p><strong>Dataset</strong></p> <p>The <em>Man OverBoard Drone (MOBDrone)</em> dataset is a large-scale collection of aerial footage images. It contains 126,170 frames extracted from 66 video clips gathered from one UAV flying at an altitude of 10 to 60 meters above the mean sea level. Images are manually annotated with more than 180K bounding boxes localizing objects belonging to 5 categories --- <em>person, boat, lifebuoy, surfboard, wood</em>. More than 113K of these bounding boxes belong to the person category and localize people in the water simulating the need to be rescued.</p> <p>In this repository, we provide:</p> <ul> <li> <p>66 Full HD video clips (total size: 5.5 GB) </p> </li> <li> <p>126,170 images extracted from the videos at a rate of 30 FPS (total size: 243 GB)</p> </li> <li> <p>3 annotation files for the extracted images that follow the MS COCO data format (for more info see <a href="https://cocodataset.org/#format-data">https://cocodataset.org/#format-data</a>):</p> <ul> <li> <p><em>annotations_5_custom_classes.json</em>: this file contains annotations concerning all five categories; please note that class ids do not correspond with the ones provided by the MS COCO standard since we account for two new classes not previously considered in the MS COCO dataset --- <em>lifebuoy </em>and <em>wood</em></p> </li> <li> <p><em>annotations_3_coco_classes.json:</em> this file contains annotations concerning the three classes also accounted by the MS COCO dataset --- <em>person, boat, surfboard</em>. Class ids correspond with the ones provided by the MS COCO standard.</p> </li> <li> <p><em>annotations_person_coco_classes.json</em>: this file contains annotations concerning only the '<em>person</em>' class. Class id corresponds to the one provided by the MS COCO standard.</p> </li> </ul> </li> </ul> <p>The MOBDrone dataset is intended as a test data benchmark. However, for researchers interested in using our data also for training purposes, we provide training and test splits:</p> <ul> <li><em>Test set: </em>All the images whose filename starts with "DJI_0804" (total: 37,604 images)</li> <li><em>Training set:</em> All the images whose filename starts with "DJI_0915" (total: 88,568 images)</li> </ul> <p>More details about data generation and the evaluation protocol can be found at our MOBDrone paper: <a href="https://arxiv.org/abs/2203.07973">https://arxiv.org/abs/2203.07973</a><br> The code to reproduce our results is available at this GitHub Repository: <a href="https://github.com/ciampluca/MOBDrone_eval">https://github.com/ciampluca/MOBDrone_eval</a><br> See also <strong> </strong><a href="http://aimh.isti.cnr.it/dataset/MOBDrone">http://aimh.isti.cnr.it/dataset/MOBDrone</a></p> <p><strong>Citing the MOBDrone</strong></p> <p>The MOBDrone is released under a Creative Commons Attribution license, so please cite the MOBDrone if it is used in your work in any form.<br> Published academic papers should use the academic paper citation for our MOBDrone paper, where we evaluated several pre-trained state-of-the-art object detectors focusing on the detection of the overboard people</p> <blockquote> <pre>@inproceedings{MOBDrone2021, title={MOBDrone: a Drone Video Dataset for Man OverBoard Rescue}, author={Donato Cafarelli and Luca Ciampi and Lucia Vadicamo and Claudio Gennaro and Andrea Berton and Marco Paterni and Chiara Benvenuti and Mirko Passera and Fabrizio Falchi}, booktitle={ICIAP2021: 21th International Conference on Image Analysis and Processing}, year={2021} } </pre> </blockquote> <p>and this Zenodo Dataset</p> <blockquote> <pre>@dataset{donato_cafarelli_2022_5996890, author={Donato Cafarelli and Luca Ciampi and Lucia Vadicamo and Claudio Gennaro and Andrea Berton and Marco Paterni and Chiara Benvenuti and Mirko Passera and Fabrizio Falchi}, title = {{MOBDrone: a large-scale drone-view dataset for man overboard detection}}, month = feb, year = 2022, publisher = {Zenodo}, version = {1.0.0}, doi = {10.5281/zenodo.5996890}, url = {https://doi.org/10.5281/zenodo.5996890} }</pre> </blockquote> <p>Personal works, such as machine learning projects/blog posts, should provide a URL to the MOBDrone Zenodo page (<a href="https://doi.org/10.5281/zenodo.5996890">https://doi.org/10.5281/zenodo.5996890</a>), though a reference to our MOBDrone paper would also be appreciated.</p> <p> </p> <p><strong>Contact Information</strong></p> <p>If you would like further information about the MOBDrone or if you experience any issues downloading files, please contact us at <a href="mailto:mobdrone@isti.cnr.it">mobdrone[at]isti.cnr.it</a></p> <p> </p> <p><strong>Acknowledgements</strong></p> <p>This work was partially supported by NAUSICAA - "NAUtical Safety by means of Integrated Computer-Assistance Appliances 4.0" project funded by the Tuscany region (CUP D44E20003410009). The data collection was carried out with the collaboration of the Fly&Sense Service of the CNR of Pisa - for the flight operations of remotely piloted aerial systems - and of the Institute of Clinical Physiology (IFC) of the CNR - for the water immersion operations. </p>
A Large-scale Dataset of (Open Source) License Text Variants
<p>We introduce a large-scale dataset of the complete texts of free/open source software (FOSS) license variants. To assemble it we have collected from the Software Heritage archive—the largest publicly available archive of FOSS source code with accompanying development history—all versions of files whose names are commonly used to convey licensing terms to software users and developers.<br> The dataset consists of 6.5 million unique license files that can be used to conduct empirical studies on open source licensing, training of automated license classifiers, natural language processing (NLP) analyses of legal texts, as well as historical and phylogenetic studies on FOSS licensing.<br> Additional metadata about shipped license files are also provided, making the dataset ready to use in various contexts; they include: file length measures, detected MIME type, detected SPDX license (using ScanCode), example origin (e.g., GitHub repository), oldest public commit in which the license appeared.<br> The dataset is released as open data as an archive file containing all deduplicated license blobs, plus several portable CSV files for metadata, referencing blobs via cryptographic checksums.</p> <p>For more details see the included README file and companion paper:</p> <ul> <li>Stefano Zacchiroli. <a href="https://doi.org/10.1145/3524842.3528491"><em>A Large-scale Dataset of (Open Source) License Text Variants</em></a>. In proceedings of the <a href="https://conf.researchr.org/home/msr-2022">2022 Mining Software Repositories Conference (MSR 2022)</a>. 23-24 May 2022 Pittsburgh, Pennsylvania, United States. ACM 2022.</li> </ul> <p>If you use this dataset for research purposes, please acknowledge its use by citing the above paper.</p> <ul> </ul>
LARD: Large-scale Artificial Disfluency Generation
<p>This dataset contains 95,992 examples of utterances with 71,994 artificial inserted disfluencies using the LARD method. We use the <a href="https://arxiv.org/pdf/1801.04871.pdf">Schema-Guided Dialogue (SGD) </a>dataset as a base to construct the synthetic disfluencies. The LARD dataset contains three different types of disfluencies: repetitions, replacements, and restarts. </p>
HISTORIAN: a large-scale HISTORIcal film dataset with cinematographic ANnotation
<p>Developing automated tools for sustainable film preservation of extensive historical film collections assumes an understanding of fundamental cinematographic settings. In order to be able to investigate new approaches to detect and classify cinematographic settings, this paper proposes a novel large-scale historical film dataset with cinematographic annotations (HISTORIAN), i.e., shot boundaries, shot types, camera movements. The dataset consists of 98 digitized original analog film reels related to the Second World War and 10593 film shots manually annotated by human film experts. Moreover, annotations for overscan areas such as sprocket holes are included. A baseline film analysis pipeline is introduced and evaluated. To the best of our knowledge, HISTORIAN is the first dataset that covers the challenges and characteristics of historical film documentaries and provides novel possibilities for exploring automatic film analysis tools.</p> <p>This repository presents a tiny set including a few examples for demonstration.</p> <p>A link to the Github repository (including helper scripts and readme) can be found <a href="https://github.com/dahe-cvl/historian_dataset">here</a></p> <p> </p> <p> </p> <p> </p> <p> </p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.