Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
1,454
datasets available to search
ShareScore release 0.9.0
Dataset results
1,454 results for “large scale”
A Large-scale Synthetic Pathological Dataset for Deep Learning-enabled Segmentation of Breast Cancer
<p>Dataset access for the paper: A Large-scale Synthetic Pathological Dataset for Deep Learning-enabled Segmentation of Breast Cancer</p>
Data for manuscript "rMATS-turbo: an efficient and flexible computational tool for alternative splicing analysis of large-scale RNA-seq data"
<p>Output files generated by rMATS-turbo for the two example datasets described in the manuscript titled "rMATS-turbo: an efficient and flexible computational tool for alternative splicing analysis of large-scale RNA-seq data".</p> <table> <tbody> <tr> <td>File</td> <td>Description</td> <td>Cell lines</td> <td>BioProject</td> </tr> <tr> <td>PC3E-GS689.tar.gz</td> <td>Compressed folder containing all 36 rMATS-turbo output files for Example 1 described in the manuscript</td> <td>PC3E and GS689 cell lines</td> <td>PRJNA438990</td> </tr> <tr> <td>CCLE.tar.gz</td> <td>Compressed folder containing all 36 rMATS-turbo output files for Example 2 described in the manuscript</td> <td>1,019 CCLE human cancer cell lines</td> <td>PRJNA523380</td> </tr> </tbody> </table> <p>A detailed description of the output files is available in the manuscript and the rMATS-turbo software GitHub repository (https://github.com/Xinglab/rmats-turbo).</p>
The Virtual Macaque Brain: A macaque connectome for large-scale network simulations in TheVirtualBrain
<p>A whole-cortex macaque structural connectome constructed from a combination of axonal tract-tracing and diffusion-weighted imaging data. Created for modeling brain dynamics using TheVirtualBrain platform. Website: thevirtualbrain.org</p>
DongTing: A Large-scale Dataset for Anomaly Detection of the Linux Kernel
<p>DongTing is the first large-scale dataset dedicated to Linux kernel anomaly detection. The dataset covers Linux kernels released in the last five years and includes a total of 18,966 well-labeled normal and attack sequences. The entire dataset is 85 GB in size (after decompression). The attack data covers 26 major kernel releases and contains a total of 12,116 system call sequences collected from running 17,855 bug-triggering programs. The normal data comes from 6,850 normal programs in four kernel regression test suites. We maintain the dataset and source code in Zenodo and Github, respectively, and back up the dataset and code in Baidu netdisk.</p> <h3><strong>Dataset</strong></h3> <p>The dataset is stored at <a href="http://doi.org/10.5281/zenodo.6627050">http://doi.org/10.5281/zenodo.6627050</a></p> <ul> <li>The data includes `abnormal_data`, `normal_data`, `models`, `npz` and baseline data, with a total volume of nearly 87 GB (including 85 GB for abnormal data and normal data, it's after decompression files size).</li> <li>The `Abnormal_data` directory contains 12,116 files containing system call sequence for 26 kernel releases, and the `Normal_data` directory contains 6,850 files containing system call sequences collected from four regression test suites. All of which are raw sequences.</li> <li>CNN/RNN, LSTM, and Wavenet (three sets of hyperparameters per model) machine learning models are selected, the ECOD model (without hyperparameters) was also chosen for the evaluation of DT. DT_abnormal, DT_normal, ADFA-LD, and PLAID are used for training respectively. The results of DT training models are stored in the directory `Models-DongTing`, and the results of ADFA-LD and PLAID training models are stored in the directory `Models-Comparison`.</li> <li>The directory `npz `stores the encoded dataset of DongTing, ADFA-LD, and PLAID (sequence length varies from 8 to 4495), according to syscall_64.tbl in Linux kernel 5.17, including the training set, validation set, and test set.</li> <li>The file `Baseline.xlsx` contains all the information about DongTing dataset, which can be used in training machine learning models. For example, the whole dataset is randomly divided into three sets with the ratio of 80%:10%:10% (training: validation: test). The implementation of dataset division can be found in the source code.</li> </ul> <h3><strong>Source Code</strong></h3> <p><br>The source code for dataset development is stored at <a href="https://github.com/HNUSystemsLab/DongTing">https://github.com/HNUSystemsLab/DongTing</a> and the following is a brief introduction.</p> <ul> <li>The source code contains three folders, i.e., `Source Code Files`, `Documents` and `DB`, where `Documents `stores the detailed documents related to development, `DB` stores samples data, and `Source Code Files` stores the source code related to the development of our dataset.</li> <li>The detailed description about the source code can be found in `Documents/Documentation.pdf`. The document consists of four parts: environment requirements, database, program structure and working steps, model training and evaluation (including training and evaluation). It details the preparation of the environment, data import method, functional description of each file in the source code directory, how model training and evaluation work and other related contents.</li> </ul> <p>We additionally maintain the dataset and source code on Baidu.com <a href="https://pan.baidu.com/s/1vu1WGZpf2DqMIoyGayNu3w?pwd=dtds">https://pan.baidu.com/s/1vu1WGZpf2DqMIoyGayNu3w?pwd=dtds</a> to facilitate the access from China.</p> <p> </p> <h3>Tips: </h3> <p>If you find DongTing useful for your research, please cite the article as "DongTing: A large-scale dataset for anomaly detection of the Linux kernel".</p> <blockquote> <p><br>@article{DUAN2023111745,<br>title = {DongTing: A large-scale dataset for anomaly detection of the Linux kernel},<br>journal = {Journal of Systems and Software},<br>volume = {203},<br>pages = {111745},<br>year = {2023},<br>issn = {0164-1212},<br>doi = {https://doi.org/10.1016/j.jss.2023.111745},<br>url = {https://www.sciencedirect.com/science/article/pii/S0164121223001401},<br>author = {Guoyun Duan and Yuanzhi Fu and Minjie Cai and Hao Chen and Jianhua Sun}<br>}<br><br></p> </blockquote>
A Large-Scale Empirical Study of Android Sports Apps in the Google Play Store
<p>This repository contains the dataset for our study "A Large-Scale Empirical Study of Android Sports Apps in the Google Play Store" and this will help to replicate our study, also the <a href="https://github.com/mooselab/Sports-Apps-Analysis">replication package</a> to direct you to help replicate it for your dataset too. </p> <p>Note: The dataset given are protected with password, and the password is available in our published paper</p>
WaterBench-Iowa: A Large-scale Benchmark Dataset for Data-Driven Streamflow Forecasting
<p>WaterBench-Iowa is a comprehensive benchmark dataset for streamflow forecasting. It follows FAIR data principles that are prepared with a focus on convenience for utilizing in data-driven and machine learning studies and provides benchmark performance for state-of-art deep learning architectures on the dataset for comparative analysis. By aggregating the datasets of streamflow, precipitation, watershed area, slope, soil types, and evapotranspiration from federal agencies and state organizations (i.e., NASA, NOAA, USGS, and Iowa Flood Center), we provided the WaterBench for hourly streamflow forecast studies. This dataset has a high temporal and spatial resolution with rich metadata and relational information, which can be used for varieties of deep learning and machine learning research. To some extent, WaterBench makes up for the lack of a unified benchmark in earth science research. We highly encourage researchers to use the WaterBench for deep learning research in hydrology.</p>
Flickr Africa: Examining Geo-Diversity in Large-Scale, Human-Centric Visual Data
<p>This dataset is provided for the paper "Flickr Africa: Examining Geo-Diversity in Large-Scale, Human-Centric Visual Data".</p> <p>Please refer to the readme in the zipped file for additional documentation.</p> <p>The zipped file contains two CSV files for every country in Africa obtained by queries "[country name]" and "[country name + people]".</p>
Text-fig. 12. Paramblypterus cf. rohani. Arrows indicate directio cranialis. a, b: photograph and drawing of four ridge scales in front of the dorsal fin base, locality Otovice "Chmelnice", P 80178, scale bars 5 mm; c: scale rows from the area between the pectoral and pelvic fins, outer surfaces of the scales bear fine ridges terminating as denticles on the posterior edge of the scales, locality Otovice "Chmelnice", NM-M 4916, scale bar 5 mm; d: isolated scale, from anterior area of the lateral side of the body, with denticulated posterior edge, locality Otovice "Chmelnice", P 30945, scale bar 2 mm; e: isolated scale from the pelvic area of the body, locality Otovice "Chmelnice", P 30945, scale bar 2 mm; f, g: drawing and photograph of the postcleithrum and the scales behind the pectoral girdle (the scales bear conspicuous ridges on their outer surface; well preserved large postcleithrum is without ridges.), locality Otovice "Chmelnice", NM-M 4915, scale bars 5 mm. Abbreviations: Cl – cleithrum, Pcl – postcleithrum, Scl – supracleithrum. in Actinopterygians Of The Broumov Formation (Permian) In The Czech Part Of The Intra-Sudetic Basin (The Czech Republic)
Text-fig. 12. Paramblypterus cf. rohani. Arrows indicate directio cranialis. a, b: photograph and drawing of four ridge scales in front of the dorsal fin base, locality Otovice "Chmelnice", P 80178, scale bars 5 mm; c: scale rows from the area between the pectoral and pelvic fins, outer surfaces of the scales bear fine ridges terminating as denticles on the posterior edge of the scales, locality Otovice "Chmelnice", NM-M 4916, scale bar 5 mm; d: isolated scale, from anterior area of the lateral side of the body, with denticulated posterior edge, locality Otovice "Chmelnice", P 30945, scale bar 2 mm; e: isolated scale from the pelvic area of the body, locality Otovice "Chmelnice", P 30945, scale bar 2 mm; f, g: drawing and photograph of the postcleithrum and the scales behind the pectoral girdle (the scales bear conspicuous ridges on their outer surface; well preserved large postcleithrum is without ridges.), locality Otovice "Chmelnice", NM-M 4915, scale bars 5 mm. Abbreviations: Cl – cleithrum, Pcl – postcleithrum, Scl – supracleithrum.
Text-fig. 17. Aeduellidae. Scale bars 5 mm. a: the scales of oblong shape on the lateral side of the body, locality Otovice "Stěnava", DP 4307, whitened; b: drawing of the scales with fine denticles on their posterior edge, locality Otovice "Stěnava", DP 4307; c, d: drawing and photograph (whitened) of the scales of lateral side of the body, two times large scales occur in the rows 14, 16, 17, 19 (they are marked with arrows), locality Otovice "Černý potok", NM-M 4912; e: lepidotrichia of the anal fin with sigmoidal sutures between the segments (marked by arrows), locality Otovice, NM-M 4931, whitened; f: anterior edge of the dorsal fin and lepidotrichia with sigmoidal sutures between the segments, locality Otovice "Stěnava", DP 4307, whitened; g: anterior edge of the ventral lobe of the caudal fin, locality Otovice, NM-M 4931, whitened; h: the caudal peduncle with begin of bifurcation of the dorsal and ventral lobes of the caudal fin, locality Otovice, NM-M 4931. in Actinopterygians Of The Broumov Formation (Permian) In The Czech Part Of The Intra-Sudetic Basin (The Czech Republic)
Text-fig. 17. Aeduellidae. Scale bars 5 mm. a: the scales of oblong shape on the lateral side of the body, locality Otovice "Stěnava", DP 4307, whitened; b: drawing of the scales with fine denticles on their posterior edge, locality Otovice "Stěnava", DP 4307; c, d: drawing and photograph (whitened) of the scales of lateral side of the body, two times large scales occur in the rows 14, 16, 17, 19 (they are marked with arrows), locality Otovice "Černý potok", NM-M 4912; e: lepidotrichia of the anal fin with sigmoidal sutures between the segments (marked by arrows), locality Otovice, NM-M 4931, whitened; f: anterior edge of the dorsal fin and lepidotrichia with sigmoidal sutures between the segments, locality Otovice "Stěnava", DP 4307, whitened; g: anterior edge of the ventral lobe of the caudal fin, locality Otovice, NM-M 4931, whitened; h: the caudal peduncle with begin of bifurcation of the dorsal and ventral lobes of the caudal fin, locality Otovice, NM-M 4931.
Text-fig. 1. SEM images of flowers of Miranthus elegans gen. et sp. nov.; Mira locality, Portugal. a, b: Flowers in lateral view showing elongated pedicel, narrowly triangular sepals and elongated protruding style (a); note the large openings in the floral tissue and pedicel (asterisks) interpreted as schizogenous secretory cavities. c: Flower in lateral view with portion of the calyx missing exposing the ovary wall and slightly raised nectariferous ring with probable stomata-like secretory structures (arrow). d: Flower in lateral view showing long pedicel and three of the five tepals; note the elongated narrowly triangular form of the sepals. e: Flower in oblique lateral view with portion of the calyx missing exposing the ovary and elongated style. f, g: Flowers in apical view showing the bases of five sepals (f) and apex of the five-parted ovary; note larger openings in the floral tissue (asterisk) interpreted as schizogenous secretory cavities. Specimens, Mira 100-S170155 (a, holotype), Mira 100-S153145 (b, c, g), Mira 100- S101267 (d), Mira 105-S100732 (e), Mira 100-S101268 (f). Scale bars = 600 µm (a–g). in Early Flowers Of Primuloid Ericales From The Late Cretaceous Of Portugal And Their Ecological And Phytogeographic Implications
Text-fig. 1. SEM images of flowers of Miranthus elegans gen. et sp. nov.; Mira locality, Portugal. a, b: Flowers in lateral view showing elongated pedicel, narrowly triangular sepals and elongated protruding style (a); note the large openings in the floral tissue and pedicel (asterisks) interpreted as schizogenous secretory cavities. c: Flower in lateral view with portion of the calyx missing exposing the ovary wall and slightly raised nectariferous ring with probable stomata-like secretory structures (arrow). d: Flower in lateral view showing long pedicel and three of the five tepals; note the elongated narrowly triangular form of the sepals. e: Flower in oblique lateral view with portion of the calyx missing exposing the ovary and elongated style. f, g: Flowers in apical view showing the bases of five sepals (f) and apex of the five-parted ovary; note larger openings in the floral tissue (asterisk) interpreted as schizogenous secretory cavities. Specimens, Mira 100-S170155 (a, holotype), Mira 100-S153145 (b, c, g), Mira 100- S101267 (d), Mira 105-S100732 (e), Mira 100-S101268 (f). Scale bars = 600 µm (a–g).
Text-fig. 10. Progyrolepis heyleri POPLIN, 1999. a: dorsal lobe of the caudal fin with the fulcral scales along the dorsal edge of the lobe, GMC 55, whitened, scale bar 5 mm; b: basal fulcral scales from the dorsal edge of the caudal peduncle, G 123, whitened, scale bar 5 mm; c: fragment of the body of juvenile specimen with dorsal and anal fins, GMC 11, whitened, scale bar 5 mm; d: isolated scales from lateral side of the body, G 123, whitened, scale bar 5 mm; e: ridges on the scale surface, the frame delineates the area illustrated in (f) at higher magnification, G 123, scale bar 500 µm; f: details of the surface with microtubercles, scale bar 50 µm; g: isolated lepidotrichium of an adult specimen with very short and wide segments and with unsegmented basal part, GMC 101, whitened, scale bar 5 mm; h: large conical teeth from the internal row of the maxilla, G 123, scale bar 2 mm; i: microsculpture formed by elliptical proximo-distally elongated protuberances on the large conical tooth, G 123, scale bar 100 µm; j: large conical tooth from the internal row of the maxilla, G 123, scale bar 2 mm; k: microsculpture formed by elliptical proximo-distally elongated protuberances on the large conical tooth, G 123, scale bar 100 µm. in New Actinopterygians From The Permian Of The Brive Basin, And The Ichthyofaunas Of The French Massif Central
Text-fig. 10. Progyrolepis heyleri POPLIN, 1999. a: dorsal lobe of the caudal fin with the fulcral scales along the dorsal edge of the lobe, GMC 55, whitened, scale bar 5 mm; b: basal fulcral scales from the dorsal edge of the caudal peduncle, G 123, whitened, scale bar 5 mm; c: fragment of the body of juvenile specimen with dorsal and anal fins, GMC 11, whitened, scale bar 5 mm; d: isolated scales from lateral side of the body, G 123, whitened, scale bar 5 mm; e: ridges on the scale surface, the frame delineates the area illustrated in (f) at higher magnification, G 123, scale bar 500 µm; f: details of the surface with microtubercles, scale bar 50 µm; g: isolated lepidotrichium of an adult specimen with very short and wide segments and with unsegmented basal part, GMC 101, whitened, scale bar 5 mm; h: large conical teeth from the internal row of the maxilla, G 123, scale bar 2 mm; i: microsculpture formed by elliptical proximo-distally elongated protuberances on the large conical tooth, G 123, scale bar 100 µm; j: large conical tooth from the internal row of the maxilla, G 123, scale bar 2 mm; k: microsculpture formed by elliptical proximo-distally elongated protuberances on the large conical tooth, G 123, scale bar 100 µm.
Text-fig. 4. Briveichthys chantepieorum gen. et sp. nov. a, b: photograph and drawing of the parasphenoid in dorsal view, GMC 15, whitened, scale bars 5 mm; c: maxillary plate in medial view with horizontal lamina along the ventral edge of the bone, segment of the lower jaw with assembly of large slender teeth of the inner row and coronoids with small teeth, GMC 15, whitened, scale bar 5 mm; d: detail of the sculpture on the maxilla and dentalosplenial, GMC 126, whitened, scale bar 5 mm; e: detail of the teeth of the inner and outer row and coronoids on the lower jaw, the frame delineates the area illustrated in (f) at higher magnification, GMC 15, scale bar 2 mm; f: microsculpture formed by elliptical proximo-distally elongated protuberances on the large conical teeth, GMC 15, scale bar 100 µm; g: small fringing fulcra tightly attached to the anterior edge of a lepidotrichium, individual fulcral scales are indicated by arrows, GMC 18, scale bar 2 mm. Abbreviation: bhf – bucco-hypophysial foramen, Cor – coronoids, cp – corpus parasphenoidis, De – dentalosplenial, hl – horizontal lamina, mc – pores of the mandibular sensory canal, Mx – maxilla, paa – processus ascendens anterior, pap – processus ascendens posterior. in New Actinopterygians From The Permian Of The Brive Basin, And The Ichthyofaunas Of The French Massif Central
Text-fig. 4. Briveichthys chantepieorum gen. et sp. nov. a, b: photograph and drawing of the parasphenoid in dorsal view, GMC 15, whitened, scale bars 5 mm; c: maxillary plate in medial view with horizontal lamina along the ventral edge of the bone, segment of the lower jaw with assembly of large slender teeth of the inner row and coronoids with small teeth, GMC 15, whitened, scale bar 5 mm; d: detail of the sculpture on the maxilla and dentalosplenial, GMC 126, whitened, scale bar 5 mm; e: detail of the teeth of the inner and outer row and coronoids on the lower jaw, the frame delineates the area illustrated in (f) at higher magnification, GMC 15, scale bar 2 mm; f: microsculpture formed by elliptical proximo-distally elongated protuberances on the large conical teeth, GMC 15, scale bar 100 µm; g: small fringing fulcra tightly attached to the anterior edge of a lepidotrichium, individual fulcral scales are indicated by arrows, GMC 18, scale bar 2 mm. Abbreviation: bhf – bucco-hypophysial foramen, Cor – coronoids, cp – corpus parasphenoidis, De – dentalosplenial, hl – horizontal lamina, mc – pores of the mandibular sensory canal, Mx – maxilla, paa – processus ascendens anterior, pap – processus ascendens posterior.
Text-fig. 1. a: Po Plain and foothills of the Northern Apennine in Northern Italy (inset) with the location of Oriolo (black star) and other Early and Middle Pleistocene plant localities, Enza and Stirone. Red lines indicate the frontal thrust arcs (modified from Martinetto et al. 2015). b: The "La Salita" section, Oriolo and chronology of the two "Sabbie gialle" cycles based on large mammals and palaeomagnetic correlation (modified from Toniato et al. 2017; IMMS 2020* [Italian Mediterranean Marine Stages] updated from Cohen and Gibbars 2020; GTS 2021* [Global Time Scale] updated from Head et al. 2021). c: Quarry "La Salita", Oriolo, in 1987. Main unconformities (U) separating the two "Sabbie gialle" cycles and terrestrial deposits on top are shown. Leaf symbols indicate the positions of some of the layers rich in fossil leaves (photo by G. B. Vai, modified). d: Surroundings of Faenza with the location of Oriolo and adjacent coeval sites yielding plant macrofossils. in The Late Early Pleistocene Flora Of Oriolo, Faenza (Italy): Assembly Of The Modern Forest Biome
Text-fig. 1. a: Po Plain and foothills of the Northern Apennine in Northern Italy (inset) with the location of Oriolo (black star) and other Early and Middle Pleistocene plant localities, Enza and Stirone. Red lines indicate the frontal thrust arcs (modified from Martinetto et al. 2015). b: The "La Salita" section, Oriolo and chronology of the two "Sabbie gialle" cycles based on large mammals and palaeomagnetic correlation (modified from Toniato et al. 2017; IMMS 2020* [Italian Mediterranean Marine Stages] updated from Cohen and Gibbars 2020; GTS 2021* [Global Time Scale] updated from Head et al. 2021). c: Quarry "La Salita", Oriolo, in 1987. Main unconformities (U) separating the two "Sabbie gialle" cycles and terrestrial deposits on top are shown. Leaf symbols indicate the positions of some of the layers rich in fossil leaves (photo by G. B. Vai, modified). d: Surroundings of Faenza with the location of Oriolo and adjacent coeval sites yielding plant macrofossils.
Text-fig. 2. Charred fern from diatomite of Saint-Bauzile. a: Overview of diatomite slab with large fragment of charred fern; SM.B 22258; scale bar = 1 cm. b: SEM overview image of charred fern pinnule. c: Enlargement of (b) showing details of undulating anticlinal walls of the epidermis. in Evidence For Wildfires During Deposition Of The Late Miocene Diatomites Of The Konservat-Lagerstätte Lake Saint-Bauzile (Ardèche, France) - Preliminary Results
Text-fig. 2. Charred fern from diatomite of Saint-Bauzile. a: Overview of diatomite slab with large fragment of charred fern; SM.B 22258; scale bar = 1 cm. b: SEM overview image of charred fern pinnule. c: Enlargement of (b) showing details of undulating anticlinal walls of the epidermis.
Data from: The State of OA: A large-scale analysis of the prevalence and impact of Open Access articles
<p>This is the raw data behind the publication: </p> <p><strong>The State of OA: A large-scale analysis of the prevalence and impact of Open Access articles.</strong></p> <p>Despite growing interest in Open Access (OA) to scholarly literature, there is an unmet need for large-scale, up-to-date, and reproducible studies assessing the prevalence and characteristics of OA. We address this need using oaDOI, an open online service that determines OA status for 67 million articles. We use three samples, each of 100,000 articles, to investigate OA in three populations: 1) all journal articles assigned a Crossref DOI, 2) recent journal articles indexed in Web of Science, and 3) articles viewed by users of Unpaywall, an open-source browser extension that lets users find OA articles using oaDOI. We estimate that at least 28% of the scholarly literature is OA (19M in total) and that this proportion is growing, driven particularly by growth in Gold and Hybrid. The most recent year analyzed (2015) also has the highest percentage of OA (45%). Because of this growth, and the fact that readers disproportionately access newer articles, we find that Unpaywall users encounter OA quite frequently: 47% of articles they view are OA. Notably, the most common mechanism for OA is not Gold, Green, or Hybrid OA, but rather an under-discussed category we dub Bronze: articles made free-to-read on the publisher website, without an explicit Open license. We also examine the citation impact of OA articles, corroborating the so-called open-access citation advantage: accounting for age and discipline, OA articles receive 18% more citations than average, an effect driven primarily by Green and Hybrid OA. We encourage further research using the free oaDOI service, as a way to inform OA policy and practice.</p>
Widespread Error Detection in Large Scale Continuous Integration Systems
<p>Dataset of 5000 json documents describing verification process of <a href="https://github.com/facebook/react">React</a> project collected between 2023 and 2024. Errors recorded in this dataset were used in the presentation at CCIW workshop.</p> <p><strong>Abstract: </strong>Continuous Integration systems are widely used in the software industry to validate and integrate code changes into central repositories. Their effectiveness can be impacted by non-deterministic tests which can fail in the absence of any regression. Integration tests which depend on external services are particularly prone to this problem. We present a system which allows us to reduce the impact of non-deterministic failures by detecting widespread errors. The key assumption, which works well in practice, is that developers tend not to make identical mistakes simultaneously. If we observe a widespread error, it strongly suggests there is a problem with upstream services and not with the code change being evaluated. The detection algorithm consists of three main phases. First, the error text gets extracted from logs using predefined heuristics or automated methods. Then, this text gets fuzzy matched against a database of recently observed errors. Finally, statistics get checked to determine if they meet the criteria for a widespread error. When an error meets the criteria it either gets demoted to a warning or it gets enriched with information about an ongoing incident.</p>
Adaptive Behavior of Farmers Under Consecutive Droughts Results In More Vulnerable Farmers: A Large-Scale Agent-Based Modeling Analysis in the Bhima Basin, India
Open the record for dataset details and reuse information.
CompanyKG Dataset V2.0: A Large-Scale Heterogeneous Graph for Company Similarity Quantification
<p><strong>CompanyKG</strong> is a heterogeneous graph consisting of 1,169,931 nodes and 50,815,503 undirected edges, with each node representing a real-world company and each edge signifying a relationship between the connected pair of companies.</p> <p><strong>Edges</strong>: We model 15 different inter-company relations as undirected edges, each of which corresponds to a unique edge type. These edge types capture various forms of similarity between connected company pairs. Associated with each edge of a certain type, we calculate a real-numbered weight as an approximation of the similarity level of that type. It is important to note that the constructed edges do not represent an exhaustive list of all possible edges due to incomplete information. Consequently, this leads to a sparse and occasionally skewed distribution of edges for individual relation/edge types. Such characteristics pose additional challenges for downstream learning tasks. Please refer to our paper for a detailed definition of edge types and weight calculations.</p> <p><strong>Nodes</strong>: The graph includes all companies connected by edges defined previously. Each node represents a company and is associated with a descriptive text, such as "<em>Klarna is a fintech company that provides support for direct and post-purchase payments</em> ...". To comply with privacy and confidentiality requirements, we encoded the text into numerical embeddings using four different pre-trained text embedding models: <a href="https://huggingface.co/sentence-transformers/distiluse-base-multilingual-cased-v2">mSBERT</a> (multilingual Sentence BERT), <a href="https://platform.openai.com/docs/guides/embeddings/what-are-embeddings">ADA2</a>, <a href="https://github.com/princeton-nlp/SimCSE">SimCSE</a> (fine-tuned on the raw company descriptions) and <a href="https://github.com/EQTPartners/pause">PAUSE</a>.</p> <p><strong>Evaluation Tasks</strong>. The primary goal of CompanyKG is to develop algorithms and models for quantifying the similarity between pairs of companies. In order to evaluate the effectiveness of these methods, we have carefully curated three evaluation tasks:</p> <ul> <li><strong>Similarity Prediction (SP)</strong>. To assess the accuracy of pairwise company similarity, we constructed the SP evaluation set comprising 3,219 pairs of companies that are labeled either as positive (similar, denoted by "1") or negative (dissimilar, denoted by "0"). Of these pairs, 1,522 are positive and 1,697 are negative.</li> <li><strong>Competitor Retrieval (CR)</strong>. Each sample contains one <em>target company</em> and one of its direct competitors. It contains 76 distinct target companies, each of which has 5.3 competitors annotated in average. For a given target company A with <em>N</em> direct competitors in this CR evaluation set, we expect a competent method to retrieve all <em>N</em> competitors when searching for similar companies to A. </li> <li><strong>Similarity Ranking (SR)</strong> is designed to assess the ability of any method to rank <em>candidate companies</em> (numbered 0 and 1) based on their similarity to a <em>query company</em>. Paid human annotators, with backgrounds in engineering, science, and investment, were tasked with determining which candidate company is more similar to the query company. It resulted in an evaluation set comprising 1,856 rigorously labeled ranking questions. We retained 20% (368 samples) of this set as a validation set for model development. </li> <li><strong>Edge Prediction (EP)</strong> evaluates a model's ability to predict future or missing relationships between companies, providing forward-looking insights for investment professionals. The EP dataset, derived (and sampled) from new edges collected between April 6, 2023, and May 25, 2024, includes 40,000 samples, with edges not present in the pre-existing CompanyKG (a snapshot up until April 5, 2023).</li> </ul> <p><strong>Background and Motivation</strong></p> <p>In the investment industry, it is often essential to identify similar companies for a variety of purposes, such as market/competitor mapping and Mergers & Acquisitions (M&A). Identifying comparable companies is a critical task, as it can inform investment decisions, help identify potential synergies, and reveal areas for growth and improvement. The accurate quantification of inter-company similarity, also referred to as <strong>company similarity quantification</strong>, is the cornerstone to successfully executing such tasks. However, company similarity quantification is often a challenging and time-consuming process, given the vast amount of data available on each company, and the complex and diversified relationships among them.</p> <p>While there is no universally agreed definition of company similarity, researchers and practitioners in PE industry have adopted various criteria to measure similarity, typically reflecting the companies' operations and relationships. These criteria can embody one or more dimensions such as industry sectors, employee profiles, keywords/tags, customers' review, financial performance, co-appearance in news, and so on. Investment professionals usually begin with a limited number of companies of interest (a.k.a. seed companies) and require an algorithmic approach to expand their search to a larger list of companies for potential investment. </p> <p>In recent years, transformer-based Language Models (LMs) have become the preferred method for encoding textual company descriptions into vector-space embeddings. Then companies that are similar to the seed companies can be searched in the embedding space using distance metrics like cosine similarity. The rapid advancements in Large LMs (LLMs), such as GPT-3/4 and LLaMA, have significantly enhanced the performance of general-purpose conversational models. These models, such as ChatGPT, can be employed to answer questions related to similar company discovery and quantification in a Q&A format.</p> <p>However, graph is still the most natural choice for representing and learning diverse company relations due to its ability to model complex relationships between a large number of entities. By representing companies as nodes and their relationships as edges, we can form a <strong>Knowledge Graph (KG)</strong>. Utilizing this KG allows us to efficiently capture and analyze the network structure of the business landscape. Moreover, KG-based approaches allow us to leverage powerful tools from network science, graph theory, and graph-based machine learning, such as Graph Neural Networks (GNNs), to extract insights and patterns to facilitate similar company analysis. While there are various company datasets (mostly commercial/proprietary and non-relational) and graph datasets available (mostly for single link/node/graph-level predictions), there is a scarcity of datasets and benchmarks that combine both to create a large-scale KG dataset expressing rich pairwise company relations.</p> <p><strong>Source Code and Tutorial:<br></strong><a href="https://github.com/llcresearch/CompanyKG2"><strong>https://github.com/llcresearch/CompanyKG2</strong></a></p> <p><strong>Paper: to be published<br></strong></p>
Data and code from: Learning a deep language model for microbiomes: The power of large scale unlabeled microbiome data
<p>We use open source human gut microbiome data to learn a microbial "language" model by adapting techniques from Natural Language Processing (NLP). Our microbial "language" model is trained in a self-supervised fashion (i.e., without additional external labels) to capture the interactions among different microbial species and the common compositional patterns in microbial communities. The learned model produces contextualized taxa representations that allow a single bacteria species to be represented differently according to the specific microbial environment it appears in. The model further provides a sample representation by collectively interpreting different bacteria species in the sample and their interactions as a whole. We show that, compared to baseline representations, our sample representation consistently leads to improved performance for multiple prediction tasks including predicting Irritable Bowel Disease (IBD) and diet patterns. Coupled with a simple ensemble strategy, it produces a highly robust IBD prediction model that generalizes well to microbiome data independently collected from different populations with substantial distribution shift.</p> <p>We visualize the contextualized taxa representations and find that they exhibit meaningful phylum-level structure, despite never exposing the model to such a signal. Finally, we apply an interpretation method to highlight bacterial species that are particularly influential in driving our model's predictions for IBD.</p>
Results from Interpreting Cis-Regulatory Interactions from Large-Scale Deep Neural Networks for Genomics
Open the record for dataset details and reuse information.
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.