Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
419
datasets available to search
ShareScore release 0.9.0
Dataset results
419 results for “large dataset”
Large-scale annotated dataset for cochlear hair cell detection and classification
<p>Our sense of hearing is mediated by cochlear hair cells, of which there are two types organized in one row of inner hair cells and three rows of outer hair cells. Each cochlea contains 5 - 15 thousand terminally differentiated hair cells, and their survival is essential for hearing as they do not regenerate after insult. It is often desirable in hearing research to quantify the number of hair cells within cochlear samples, in both pathological conditions, and in response to treatment. Machine learning can be used to automate the quantification process but requires a vast and diverse dataset for effective training. In this study, we present a large collection of annotated cochlear hair-cell datasets, labeled with commonly used hair-cell markers and imaged using various fluorescence microscopy techniques. The collection includes samples from mouse, rat, guinea pig, pig, primate, and human cochlear tissue, from normal conditions and following <i>in-vivo</i> and <i>in-vitro</i>ototoxic drug application. The dataset includes over 107,000 hair cells which have been manually identified and annotated as either inner or outer hair cells. This dataset is the result of a collaborative effort from multiple laboratories and has been carefully curated to represent a variety of imaging techniques. With suggested usage parameters and a well-described annotation procedure, this collection can facilitate the development of generalizable cochlear hair-cell detection models or serve as a starting point for fine-tuning models for other analysis tasks. By providing this dataset, we aim to give other hearing research groups the opportunity to develop their own tools with which to analyze cochlear imaging data more fully, accurately, and with greater ease. </p><p>Associated code is provided here: https://github.com/indzhykulianlab/hcat-data</p>
CESNET-TLS22: A large dataset for fine-grained classification of TLS services
<p><strong>Please refer to the original article for further data description:</strong> Jan Luxemburk et al. Fine-grained TLS services classification with reject option, Computer Networks, 2023, 109467, ISSN 1389-1286, <a href="https://doi.org/10.1016/j.comnet.2022.109467">https://doi.org/10.1016/j.comnet.2022.109467</a></p> <p><strong>We recommend using the</strong> <strong>CESNET DataZoo python library, which facilitates the work with large network traffic datasets. </strong>More information about the DataZoo project can be found in the GitHub repository <a href="https://github.com/CESNET/cesnet-datazoo">https://github.com/CESNET/cesnet-datazoo</a>.</p> <p>The recent success and proliferation of machine learning and deep learning have provided powerful tools, which are also utilized for encrypted traffic analysis, classification, and threat detection. These methods, neural networks in particular, are often complex and require a huge corpus of training data. Moreover, because most of the network traffic is being encrypted, the traditional deep-packet-inspecting (DPI) solutions are becoming obsolete, and there is an urgent need for modern classification methods capable of analyzing encrypted traffic. These methods have to forgo the packet's opaque payload and focus on flow statistics and packet metadata sequences like packet sizes, directions, and inter-arrival times. The classification can be further extended with the task of "rejecting" unknown traffic, i.e., the traffic not seen during the training phase. This makes the problem more challenging, and neural networks offer superior performance for tackling this problem.<strong> When the factors of (1) the hardness of classification of encrypted traffic with unknown traffic detection and (2) the neural networks' inherent need for large datasets are combined, the requirement for a rich, large, and up-to-date dataset is even stronger.</strong></p> <p>Therefore, we created a large dataset spanning two weeks, consisting of 141 million network flows, and having 191 fine-grained service labels. The dataset is intended as a benchmark for the task of identification of services in encrypted traffic with the detection of unknown services.</p> <p><strong>Data capture</strong> The data was captured in the flow monitoring infrastructure of the <a href="https://www.cesnet.cz">CESNET2</a> network. The capturing was done for two weeks between 4.10.2021 and 17.10.2021. The following table provides per-week flow count, capture period, and uncompressed size:</p> <ul> <li><strong>W-2021-40</strong> <ul> <li>Uncompressed Size: 22 GB</li> <li>Capture Period: 4.10.2021 - 10.10.2021</li> <li>Flows: 73.2M</li> </ul> </li> <li><strong>W-2021-41</strong> <ul> <li>Uncompressed Size: 20 GB</li> <li>Capture Period: 11.10.2021 - 17.10.2021</li> <li>Flows: 68.5M</li> </ul> </li> <li><strong>CESNET-TLS22</strong> <ul> <li>Uncompressed Size: 42 GB</li> <li>Capture Period: 4.10.2021 - 17.10.2021</li> <li>Flows: 141.7M</li> </ul> </li> </ul> <p><strong>Dataset structure</strong> The dataset flows are delivered in compressed CSV files, which contain one flow per row. For each flow data file, there is a JSON file with the number of saved flows per service. There is also the <em>stats-week.json</em> file aggregating flow counts of a whole week and the <em>stats-dataset.json</em> file aggregating flow counts for the entire dataset. The mapping between services and service providers is provided in the <em>servicemap.csv</em> file, which also includes SNI domains used for ground truth labeling. The following table describes flow data fields in CSV files:</p> <ul> <li><strong>ID:</strong> Unique identifier</li> <li><strong>BYTES:</strong> Number of transmitted bytes from client to server</li> <li><strong>BYTES_REV:</strong> Number of transmitted bytes from server to client</li> <li><strong>PACKETS:</strong> Number of packets transmitted from client to server</li> <li><strong>PACKETS_REV:</strong> Number of packets transmitted from server to client</li> <li><strong>DURATION:</strong> Duration of the flow in seconds</li> <li><strong>PPI:</strong> Packet metadata sequence in the format: [[inter-packet times], [packet directions], [packet sizes]]</li> <li><strong>PPI_LEN:</strong> Number of packets in the PPI sequence</li> <li><strong>PPI_DURATION:</strong> Duration of the PPI sequence in seconds</li> <li><strong>PPI_ROUNDTRIPS:</strong> Number of roundtrips in the PPI sequence</li> <li><strong>APP:</strong> Web service label</li> <li><strong>CATEGORY:</strong> Service category</li> <li><strong>TCP_FLAGS:</strong> TCP flags sent from client to server</li> <li><strong>TCP_FLAGS_REV:</strong> TCP flags sent from server to client</li> <li><strong>FLAG_CWR:</strong> Presence of the CWR flag</li> <li><strong>FLAG_CWR_REV:</strong> Presence of the CWR flag in the reverse direction</li> <li><strong>FLAG_ECE:</strong> Presence of the ECE flag</li> <li><strong>FLAG_ECE_REV:</strong> Presence of the ECE flag in the reverse direction</li> <li><strong>FLAG_URG:</strong> Presence of the URG flag</li> <li><strong>FLAG_URG_REV:</strong> Presence of the URG flag in the reverse direction</li> <li><strong>FLAG_ACK:</strong> Presence of the ACK flag</li> <li><strong>FLAG_ACK_REV:</strong> Presence of the ACK flag in the reverse direction</li> <li><strong>FLAG_PSH:</strong> Presence of the PSH flag</li> <li><strong>FLAG_PSH_REV:</strong> Presence of the PSH flag in the reverse direction</li> <li><strong>FLAG_RST:</strong> Presence of the RST flag</li> <li><strong>FLAG_RST_REV:</strong> Presence of the RST flag in the reverse direction</li> <li><strong>FLAG_SYN:</strong> Presence of the SYN flag</li> <li><strong>FLAG_SYN_REV:</strong> Presence of the SYN flag in the reverse direction</li> <li><strong>FLAG_FIN:</strong> Presence of the FIN flag</li> <li><strong>FLAG_FIN_REV:</strong> Presence of the FIN flag in the reverse direction</li> </ul> <p><strong>Link to other CESNET datasets</strong></p> <ul> <li><a href="https://www.liberouter.org/technology-v2/tools-services-datasets/datasets/">https://www.liberouter.org/technology-v2/tools-services-datasets/datasets/</a></li> <li><a href="https://github.com/CESNET/cesnet-datazoo">https://github.com/CESNET/cesnet-datazoo</a></li> </ul> <p><strong>Please cite the original article:</strong></p> <blockquote> <p>@article{luxemburk_fine-grained-tls_2023, author = {Jan Luxemburk and Tomáš Čejka}, title = {Fine-grained TLS services classification with reject option}, journal = {Computer Networks}, volume = {220}, pages = {109467}, year = {2023}, issn = {1389-1286}, doi = {https://doi.org/10.1016/j.comnet.2022.109467}, url = {https://www.sciencedirect.com/science/article/pii/S1389128622005011} }</p> </blockquote>
CESNET-QUIC22: A large one-month QUIC network traffic dataset from backbone lines
<p><strong>Please refer to the original data article for further data description: </strong>Jan Luxemburk et al. CESNET-QUIC22: A large one-month QUIC network traffic dataset from backbone lines, Data in Brief, 2023, 108888, ISSN 2352-3409, <a href="https://doi.org/10.1016/j.dib.2023.108888">https://doi.org/10.1016/j.dib.2023.108888</a>. </p><p><strong>We recommend using the</strong> <strong>CESNET DataZoo python library, which facilitates the work with large network traffic datasets. </strong>More information about the DataZoo project can be found in the GitHub repository <a href="https://github.com/CESNET/cesnet-datazoo">https://github.com/CESNET/cesnet-datazoo</a>.</p><p>The QUIC (Quick UDP Internet Connection) protocol has the potential to replace TLS over TCP, which is the standard choice for reliable and secure Internet communication. Due to its design that makes the inspection of QUIC handshakes challenging and its usage in HTTP/3, there is an increasing demand for research in QUIC traffic analysis. This dataset contains one month of QUIC traffic collected in an ISP backbone network, which connects 500 large institutions and serves around half a million people. The data are delivered as enriched flows that can be useful for various network monitoring tasks. The provided server names and packet-level information allow research in the encrypted traffic classification area. Moreover, included QUIC versions and user agents (smartphone, web browser, and operating system identifiers) provide information for large-scale QUIC deployment studies.</p><p><strong>Data capture</strong> The data was captured in the flow monitoring infrastructure of the <a href="https://www.cesnet.cz">CESNET2</a> network. The capturing was done for four weeks between 31.10.2022 and 27.11.2022. The following list provides per-week flow count, capture period, and uncompressed size:</p><ul><li><strong>W-2022-44</strong><ul><li>Uncompressed Size: 19 GB</li><li>Capture Period: 31.10.2022 - 6.11.2022</li><li>Number of flows: 32.6M</li></ul></li><li><strong>W-2022-45</strong><ul><li>Uncompressed Size: 25 GB</li><li>Capture Period: 7.11.2022 - 13.11.2022</li><li>Number of flows: 42.6M</li></ul></li><li><strong>W-2022-46</strong><ul><li>Uncompressed Size: 20 GB</li><li>Capture Period: 14.11.2022 - 20.11.2022</li><li>Number of flows: 33.7M</li></ul></li><li><strong>W-2022-47</strong><ul><li>Uncompressed Size: 25 GB</li><li>Capture Period: 21.11.2022 - 27.11.2022</li><li>Number of flows: 44.1M</li></ul></li><li><strong>CESNET-QUIC22 </strong><ul><li>Uncompressed Size: 89 GB</li><li>Capture Period: 31.10.2022 - 27.11.2022</li><li>Number of flows: 153M</li></ul></li></ul><p> </p><p><strong>Data description</strong> The dataset consists of network flows describing encrypted QUIC communications. Flows were created using <a href="https://github.com/CESNET/ipfixprobe">ipfixprobe</a> flow exporter and are extended with packet metadata sequences, packet histograms, and with fields extracted from the QUIC Initial Packet, which is the first packet of the QUIC connection handshake. The extracted handshake fields are the Server Name Indication (SNI) domain, the used version of the QUIC protocol, and the user agent string that is available in a subset of QUIC communications.</p><p><strong>Packet Sequences</strong> Flows in the dataset are extended with sequences of packet sizes, directions, and inter-packet times. For the packet sizes, we consider payload size after transport headers (UDP headers for the QUIC case). Packet directions are encoded as ±1, <i>+1</i> meaning a packet sent from client to server, and <i>-1</i> a packet from server to client. Inter-packet times depend on the location of communicating hosts, their distance, and on the network conditions on the path. However, it is still possible to extract relevant information that correlates with user interactions and, for example, with the time required for an API/server/database to process the received data and generate the response to be sent in the next packet. Packet metadata sequences have a length of 30, which is the default setting of the used flow exporter. We also derive three fields from each packet sequence: its length, time duration, and the number of roundtrips. The roundtrips are counted as the number of changes in the communication direction (from packet directions data); in other words, each client request and server response pair counts as one roundtrip.</p><p><strong>Flow statistics</strong> Flows also include standard flow statistics, which represent aggregated information about the entire bidirectional flow. The fields are: the number of transmitted bytes and packets in both directions, the duration of flow, and packet histograms. Packet histograms include binned counts of packet sizes and inter-packet times of the entire flow in both directions (more information in the <a href="https://github.com/CESNET/ipfixprobe/tree/master#phists">PHISTS plugin documentation</a> There are eight bins with a logarithmic scale; the intervals are 0-15, 16-31, 32-63, 64-127, 128-255, 256-511, 512-1024, >1024 [ms or B]. The units are milliseconds for inter-packet times and bytes for packet sizes. Moreover, each flow has its end reason - either it was idle, reached the active timeout, or ended due to other reasons. This corresponds with the official <a href="https://www.iana.org/assignments/ipfix/ipfix.xhtml#ipfix-flow-end-reason">IANA IPFIX-specified values</a>. The <i>FLOW_ENDREASON_OTHER</i> field represents the <i>forced end</i> and <i>lack of resources</i> reasons. The <i>end of flow detected</i> reason is not considered because it is not relevant for UDP connections.</p><p><strong>Dataset structure</strong> The dataset flows are delivered in compressed CSV files. CSV files contain one flow per row; data columns are summarized in the provided list below. For each flow data file, there is a JSON file with the number of saved and seen (before sampling) flows per service and total counts of all received (observed on the CESNET2 network), service (belonging to one of the dataset's services), and saved (provided in the dataset) flows. There is also the <i>stats-week.json</i> file aggregating flow counts of a whole week and the <i>stats-dataset.json</i> file aggregating flow counts for the entire dataset. Flow counts before sampling can be used to compute sampling ratios of individual services and to resample the dataset back to the original service distribution. Moreover, various dataset statistics, such as feature distributions and value counts of QUIC versions and user agents, are provided in the <i>dataset-statistics</i> folder. The mapping between services and service providers is provided in the <i>servicemap.csv</i> file, which also includes SNI domains used for ground truth labeling. The following list describes flow data fields in CSV files:</p><ul><li><strong>ID:</strong> Unique identifier</li><li><strong>SRC_IP:</strong> Source IP address</li><li><strong>DST_IP:</strong> Destination IP address</li><li><strong>DST_ASN:</strong> Destination Autonomous System number</li><li><strong>SRC_PORT:</strong> Source port</li><li><strong>DST_PORT:</strong> Destination port</li><li><strong>PROTOCOL:</strong> Transport protocol</li><li><strong>QUIC_VERSION QUIC:</strong> protocol version</li><li><strong>QUIC_SNI:</strong> Server Name Indication domain</li><li><strong>QUIC_USER_AGENT:</strong> User agent string, if available in the QUIC Initial Packet</li><li><strong>TIME_FIRST:</strong> Timestamp of the first packet in format YYYY-MM-DDTHH-MM-SS.ffffff</li><li><strong>TIME_LAST:</strong> Timestamp of the last packet in format YYYY-MM-DDTHH-MM-SS.ffffff</li><li><strong>DURATION:</strong> Duration of the flow in seconds</li><li><strong>BYTES:</strong> Number of transmitted bytes from client to server</li><li><strong>BYTES_REV:</strong> Number of transmitted bytes from server to client</li><li><strong>PACKETS:</strong> Number of packets transmitted from client to server</li><li><strong>PACKETS_REV:</strong> Number of packets transmitted from server to client</li><li><strong>PPI:</strong> Packet metadata sequence in the format: [[inter-packet times], [packet directions], [packet sizes]]</li><li><strong>PPI_LEN:</strong> Number of packets in the PPI sequence</li><li><strong>PPI_DURATION:</strong> Duration of the PPI sequence in seconds</li><li><strong>PPI_ROUNDTRIPS:</strong> Number of roundtrips in the PPI sequence</li><li><strong>PHIST_SRC_SIZES:</strong> Histogram of packet sizes from client to server</li><li><strong>PHIST_DST_SIZES: </strong>Histogram of packet sizes from server to client</li><li><strong>PHIST_SRC_IPT: </strong>Histogram of inter-packet times from client to server</li><li><strong>PHIST_DST_IPT:</strong> Histogram of inter-packet times from server to client</li><li><strong>APP:</strong> Web service label</li><li><strong>CATEGORY:</strong> Service category</li><li><strong>FLOW_ENDREASON_IDLE:</strong> Flow was terminated because it was idle</li><li><strong>FLOW_ENDREASON_ACTIVE:</strong> Flow was terminated because it reached the active timeout</li><li><strong>FLOW_ENDREASON_OTHER:</strong> Flow was terminated for other reasons</li></ul><p> </p><p><strong>Link to other CESNET datasets</strong></p><ul><li><a href="https://www.liberouter.org/technology-v2/tools-services-datasets/datasets/">https://www.liberouter.org/technology-v2/tools-services-datasets/datasets/</a></li><li><a href="https://github.com/CESNET/cesnet-datazoo">https://github.com/CESNET/cesnet-datazoo</a></li></ul><p><strong>Please cite the original data article:</strong></p><blockquote><p>@article{CESNETQUIC22, author = {Jan Luxemburk and Karel Hynek and Tomáš Čejka and Andrej Lukačovič and Pavel Šiška}, title = {CESNET-QUIC22: a large one-month QUIC network traffic dataset from backbone lines}, journal = {Data in Brief}, pages = {108888}, year = {2023}, issn = {2352-3409}, doi = {https://doi.org/10.1016/j.dib.2023.108888}, url = {https://www.sciencedirect.com/science/article/pii/S2352340923000069} }</p></blockquote>
Dataset for article "W18O49 Nanowhiskers Decorating SiO2 Nanofibers: Lessons from In Situ SEM/TEM Growth to Large Scale Synthesis and Fundamental Structural Understanding" in Crystal Growth & Design, Des. 2024, 24, 1, 378–390
<p>Supported dataset for publication "W18O49 Nanowhiskers Decorating SiO2 Nanofibers: Lessons from In Situ SEM/TEM Growth to Large Scale Synthesis and Fundamental Structural Understanding" in Crystal Growth & Design Des. 2024, 24, 1, 378–390, https://doi.org/10.1021/acs.cgd.3c01094.</p> <p>The directory Data contains the figures (png, jpg), and vido (AVI) source data files for the manuscript.</p>
Synthetic dataset from - Bar et al., Sifting through the haystack - efficiently finding rare behaviors in large-scale datasets, WACV 2025
<p>This is a synthetic dataset emulating pose estimation data, introduced in the associated paper. </p> <p>Briefly, each sample in the dataset is a sequence of 5-keypoints with 9 timesteps. Movement of each keypoint in time is determined by a sinus with some amplitude A and some frequency f, this is loosely inspired by larval zebrafish swimming movement. <br>There are two types of behaviors - a common behavior (aka forming the majority of the samples in the dataset) where the frequency of the sinus is larger than the amplitude, and a rare one where the amplitude is larger than the frequency. <br>We vary the similarity between the rare and common behaviors by relaxing the standard deviation of the gaussian from which we draw these movement parameters (behavior similarity, sd= [0.5, 1.5, 2.5, 5]). We also test different levels of data imbalance, varying the frequency of the rare behavior (rarity=[1.5%,5%,12%,24%]). </p> <p>Thus we created 16 datasets with all possible combinations.</p> <p>The data generation code will become available in our code repository: https://github.com/shir3bar/SiftingTheHaystack</p> <p>The data was used to create a controlled experimental sandbox in which we could test our pipeline for detecting rare behaviors. Sounds interesting? Read our paper and check out the code :)</p> <p> </p> <p> </p>
USPTO-LLM: A Large Language Model-Assisted Information-enriched Chemical Reaction Dataset
<p>USPTO-LLM is an <strong>information-enriched chemical reaction dataset</strong> that provides more side information (reaction conditions and reaction steps division) for developing new reaction prediction and retrosynthesis methods and inspires new problems, such as reaction condition prediction. It comprises over <strong>247K chemical reactions</strong> extracted from the patent documents of USPTO (United States Patent and Trademark Office), encompassing abundant information on reaction conditions. </p> <p>We employ large language models to expedite the data collection procedures automatically with a reliable quality control process. The extracted chemical reactions are organized as <strong>heterogeneous directed graphs</strong>, allowing us to formulate a series of prediction tasks, such as reaction prediction, retrosynthesis, and reaction condition prediction, in a unified graph-filling framework.</p>
A Dataset of Information (DNS, IP, WHOIS/RDAP, TLS, GeoIP) for a Large Corpus of Benign, Phishing, and Malware Domain Names 2024
<p>The dataset contains DNS records, IP-related features, WHOIS/RDAP information, information from TLS handshakes and certificates, and GeoIP information for 368,956 benign domains from Cisco Umbrella, 461,338 benign domains from the actual CESNET network traffic, 164,425 phishing domains from PhishTank and OpenPhish services, and 100,809 malware domains from various sources like ThreatFox, The Firebog, MISP threat intelligence platform, and other sources. The ground truth for the phishing dataset was double-check with the VirusTotal (VT) service. Domain names not considered malicious by VT have been removed from phishing and malware datasets. Similarly, benign domain names that were considered risky by VT have been removed from the benign datasets. The data was collected between March 2023 and July 2024. The final assessment of the data was conducted in August 2024.</p> <p>The dataset is useful for cybersecurity research, e.g. statistical analysis of domain data or feature extraction for training machine learning-based classifiers, e.g. for phishing and malware website detection.</p> <p>The dataset was created using software available in the associated GitHub repository <a href="https://github.com/nesfit/domainradar-dib" target="_blank" rel="noopener">nesfit/domainradar-dib</a>.</p> <h1>Data Files</h1> <ul> <li> <p>The data is located in the following individual files:</p> <ul> <li><strong>benign_umbrella.json</strong> - data for 368,956 benign domains from Cisco Umbrella,</li> <li><strong>benign_cesnet.json </strong>- data for 461,338 benign domains from the CESNET network,</li> <li><strong>phishing.json</strong> - data for 164,425 phishing domains, and</li> <li><strong>malware.json</strong> - data for 100,809 malware domains.</li> </ul> </li> <li>The <strong>schema.json</strong> file contains a JSON Schema with detailed description of the data entries.</li> </ul> <h1>Data Structure</h1> <div> <p>Both files contain a JSON array of records generated using mongoexport (in the MongoDB Extended JSON (v2) format in Relaxed Mode). The following table documents the structure of a record. Please note that:</p> <ul> <li>some fields may be missing (they should be interpreted as nulls), </li> <li>extra fields may be present (they should be ignored).</li> </ul> </div> <table> <tbody> <tr> <td> <div> <div> <p><strong>Field name </strong></p> </div> </div> </td> <td> <div> <div> <p><strong>Field type </strong></p> </div> </div> </td> <td> <div> <div> <p><strong>Nullable </strong></p> </div> </div> </td> <td> <div> <div> <p><strong>Description </strong></p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>domain_name </p> </div> </div> </td> <td> <div> <div> <p>String </p> </div> </div> </td> <td> <div> <div> <p>No </p> </div> </div> </td> <td> <div> <div> <p>The evaluated domain name </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>url </p> </div> </div> </td> <td> <div> <div> <p>String </p> </div> </div> </td> <td> <div> <div> <p>No </p> </div> </div> </td> <td> <div> <div> <p>The source URL for the domain name </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>evaluated_on </p> </div> </div> </td> <td> <div> <div> <p>Date </p> </div> </div> </td> <td> <div> <div> <p>No </p> </div> </div> </td> <td> <div> <div> <p>Date of last collection attempt </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>source </p> </div> </div> </td> <td> <div> <div> <p>String </p> </div> </div> </td> <td> <div> <div> <p>No </p> </div> </div> </td> <td> <div> <div> <p>An identifier of the source </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>sourced_on </p> </div> </div> </td> <td> <div> <div> <p>Date </p> </div> </div> </td> <td> <div> <div> <p>No </p> </div> </div> </td> <td> <div> <div> <p>Date of ingestion of the domain name </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>dns </p> </div> </div> </td> <td> <div> <div> <p>Object </p> </div> </div> </td> <td> <div> <div> <p>Yes </p> </div> </div> </td> <td> <div> <div> <p>Data from DNS scan </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>rdap </p> </div> </div> </td> <td> <div> <div> <p>Object </p> </div> </div> </td> <td> <div> <div> <p>Yes </p> </div> </div> </td> <td> <div> <div> <p>Data from RDAP or WHOIS </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>tls </p> </div> </div> </td> <td> <div> <div> <p>Object </p> </div> </div> </td> <td> <div> <div> <p>Yes </p> </div> </div> </td> <td> <div> <div> <p>Data from TLS handshake </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>ip_data </p> </div> </div> </td> <td> <div> <div> <p>Array of Objects </p> </div> </div> </td> <td> <div> <div> <p>Yes </p> </div> </div> </td> <td> <div> <div> <p>Array of data objects capturing the IP addresses related to the domain name </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>malware_type</p> </div> </div> </td> <td> <div> <div> <p>String</p> </div> </div> </td> <td> <div> <div> <p>No</p> </div> </div> </td> <td> <div> <div> <p>The malware type/family or “unknown” (only present in malware.json) </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p><strong>DNS data (dns field) </strong></p> </div> </div> </td> <td> </td> <td> </td> <td> </td> </tr> <tr> <td> <div> <div> <p>A </p> </div> </div> </td> <td> <div> <div> <p>Array of Strings </p> </div> </div> </td> <td> <div> <div> <p>No </p> </div> </div> </td> <td> <div> <div> <p>Array of IPv4 addresses </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>AAAA </p> </div> </div> </td> <td> <div> <div> <p>Array of Strings </p> </div> </div> </td> <td> <div> <div> <p>No </p> </div> </div> </td> <td> <div> <div> <p>Array of IPv6 addresses </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>TXT </p> </div> </div> </td> <td> <div> <div> <p>Array of Strings </p> </div> </div> </td> <td> <div> <div> <p>No </p> </div> </div> </td> <td> <div> <div> <p>Array of raw TXT values </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>CNAME </p> </div> </div> </td> <td> <div> <div> <p>Object </p> </div> </div> </td> <td> <div> <div> <p>No </p> </div> </div> </td> <td> <div> <div> <p>The CNAME target and related IPs </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>MX </p> </div> </div> </td> <td> <div> <div> <p>Array of Objects </p> </div> </div> </td> <td> <div> <div> <p>No </p> </div> </div> </td> <td> <div> <div> <p>Array of objects with the MX target hostname, priority and related IPs </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>NS </p> </div> </div> </td> <td> <div> <div> <p>Array of Objects </p> </div> </div> </td> <td> <div> <div> <p>No </p> </div> </div> </td> <td> <div> <div> <p>Array of objects with the NS target hostname and related IPs </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>SOA </p> </div> </div> </td> <td> <div> <div> <p>Object </p> </div> </div> </td> <td> <div> <div> <p>No </p> </div> </div> </td> <td> <div> <div> <p>All the SOA fields, present if found at the target domain name </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>zone_SOA </p> </div> </div> </td> <td> <div> <div> <p>Object </p> </div> </div> </td> <td> <div> <div> <p>No </p> </div> </div> </td> <td> <div> <div> <p>The SOA fields of the target’s zone (closest point of delegation), present if found and not a record in the target domain directly </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>dnssec </p> </div> </div> </td> <td> <div> <div> <p>Object </p> </div> </div> </td> <td> <div> <div> <p>No </p> </div> </div> </td> <td> <div> <div> <p>Flags describing the DNSSEC validation result for each record type </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>ttls </p> </div> </div> </td> <td> <div> <div> <p>Object </p> </div> </div> </td> <td> <div> <div> <p>No </p> </div> </div> </td> <td> <div> <div> <p>The TTL values for each record type </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>remarks </p> </div> </div> </td> <td> <div> <div> <p>Object </p> </div> </div> </td> <td> <div> <div> <p>No </p> </div> </div> </td> <td> <div> <div> <p>The zone domain name and DNSSEC flags </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p><strong>RDAP data (rdap field) </strong></p> </div> </div> </td> <td> </td> <td> </td> <td> </td> </tr> <tr> <td> <div> <div> <p>copyright_notice </p> </div> </div> </td> <td> <div> <div> <p>String </p> </div> </div> </td> <td> <div> <div> <p>No </p> </div> </div> </td> <td> <div> <div> <p>RDAP/WHOIS data usage copyright notice </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>dnssec </p> </div> </div> </td> <td> <div> <div> <p>Bool </p> </div> </div> </td> <td> <div> <div> <p>No </p> </div> </div> </td> <td> <div> <div> <p>DNSSEC presence flag </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>entitites </p> </div> </div> </td> <td> <div> <div> <p>Object </p> </div> </div> </td> <td> <div> <div> <p>No </p> </div> </div> </td> <td> <div> <div> <p>An object with various arrays representing the found related entity types (e.g. abuse, admin, registrant). The arrays contain objects describing the individual entities. </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>expiration_date </p> </div> </div> </td> <td> <div> <div> <p>Date </p> </div> </div> </td> <td> <div> <div> <p>Yes </p> </div> </div> </td> <td> <div> <div> <p>The current date of expiration </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>handle </p> </div> </div> </td> <td> <div> <div> <p>String </p> </div> </div> </td> <td> <div> <div> <p>No </p> </div> </div> </td> <td> <div> <div> <p>RDAP handle </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>last_changed_date </p> </div> </div> </td> <td> <div> <div> <p>Date </p> </div> </div> </td> <td> <div> <div> <p>Yes </p> </div> </div> </td> <td> <div> <div> <p>The date when the domain was last changed </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>name </p> </div> </div> </td> <td> <div> <div> <p>String </p> </div> </div> </td> <td> <div> <div> <p>No </p> </div> </div> </td> <td> <div> <div> <p>The target domain name for which the data in this object are stored </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>nameservers </p> </div> </div> </td> <td> <div> <div> <p>Array of Strings </p> </div> </div> </td> <td> <div> <div> <p>No </p> </div> </div> </td> <td> <div> <div> <p>Nameserver hostnames provided by RDAP or WHOIS </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>registration_date </p> </div> </div> </td> <td> <div> <div> <p>Date </p> </div> </div> </td> <td> <div> <div> <p>Yes </p> </div> </div> </td> <td> <div> <div> <p>First registration date </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>status </p> </div> </div> </td> <td> <div> <div> <p>Array of Strings </p> </div> </div> </td> <td> <div> <div> <p>No </p> </div> </div> </td> <td> <div> <div> <p>The state of the registered object (see RFC 7483, section 10.2.2)</p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>terms_of_service_url </p> </div> </div> </td> <td> <div> <div> <p>String </p> </div> </div> </td> <td> <div> <div> <p>No </p> </div> </div> </td> <td> <div> <div> <p>URL of the RDAP usage ToS </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>url </p> </div> </div> </td> <td> <div> <div> <p>String </p> </div> </div> </td> <td> <div> <div> <p>No </p> </div> </div> </td> <td> <div> <div> <p>URL of the RDAP entity </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>whois_server </p> </div> </div> </td> <td> <div> <div> <p>String </p> </div> </div> </td> <td> <div> <div> <p>No </p> </div> </div> </td> <td> <div> <div> <p>WHOIS server address </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p><strong>TLS data (tls field) </strong></p> </div> </div> </td> <td> </td> <td> </td> <td> </td> </tr> <tr> <td> <div> <div> <p>cipher </p> </div> </div> </td> <td> <div> <div> <p>String </p> </div> </div> </td> <td> <div> <div> <p>No </p> </div> </div> </td> <td> <div> <div> <p>TLS cipher suite description according to <a href="https://www.iana.org/assignments/tls-parameters/tls-parameters.xhtml#tls-parameters-4" target="_blank" rel="noopener">IANA</a></p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>protocol </p> </div> </div> </td> <td> <div> <div> <p>String </p> </div> </div> </td> <td> <div> <div> <p>No </p> </div> </div> </td> <td> <div> <div> <p>One of “TLS”, ”TLSv1.2”, ”TLSv1.3” </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>certificates </p> </div> </div> </td> <td> <div> <div> <p>Array of Objects </p> </div> </div> </td> <td> <div> <div> <p>No </p> </div> </div> </td> <td> <div> <div> <p>Array of objects representing the certificate chain, the first element is the root certificate </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p><strong>IP data (elements in the ip_data array) </strong></p> </div> </div> </td> <td> </td> <td> </td> <td> </td> </tr> <tr> <td> <div> <div> <p>ip </p> </div> </div> </td> <td> <div> <div> <p>String </p> </div> </div> </td> <td> <div> <div> <p>No </p> </div> </div> </td> <td> <div> <div> <p>The IP address </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>from_record </p> </div> </div> </td> <td> <div> <div> <p>String </p> </div> </div> </td> <td> <div> <div> <p>No </p> </div> </div> </td> <td> <div> <div> <p>The type of the DNS record the address was captured from </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>remarks </p> </div> </div> </td> <td> <div> <div> <p>Object </p> </div> </div> </td> <td> <div> <div> <p>No </p> </div> </div> </td> <td> <div> <div> <p>Ping round-trip time, “is alive” flag and rdap/geo/asn evaluation dates </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>rdap </p> </div> </div> </td> <td> <div> <div> <p>Object </p> </div> </div> </td> <td> <div> <div> <p>Yes </p> </div> </div> </td> <td> <div> <div> <p>RDAP data, similar to DNS RDAP, see the JSON Schema for details </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>geo </p> </div> </div> </td> <td> <div> <div> <p>Object </p> </div> </div> </td> <td> <div> <div> <p>Yes </p> </div> </div> </td> <td> <div> <div> <p>Geolocation data from the GeoLite2 City database (e.g. latitude, longitude, city, country, etc.) </p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>asn </p> </div> </div> </td> <td> <div> <div> <p>Object </p> </div> </div> </td> <td> <div> <div> <p>Yes </p> </div> </div> </td> <td> <div> <div> <p>Autonomous system data from the GeoLite2 ASN database (ASN, organization, network) </p> </div> </div> </td> </tr> </tbody> </table> <h1>Acknowledgements</h1> <p>We would like to thank the OpenPhish Team for grating permission to use and publish their dataset. We also thank VirusTotal for providing us access to the API for research purposes.</p> <p>This dataset includes GeoLite2 data created by MaxMind, available from <a href="https://www.maxmind.com" target="_blank" rel="noopener">https://www.maxmind.com</a>.</p> <p>The research has been supported by the <strong>Flow-based Encrypted Traffic Analysis</strong> project, no. VJ02010024, granted by the Ministry of the Interior of the Czech Republic and the <strong>Smart Information Technology for a Resilient Society </strong>project, no. FIT-S-23-8209, granted by Brno University of Technology.</p>
Caravan-DE: Caravan extension Germany - German dataset for large-sample hydrology
<p><em>Caravan</em> is an open community dataset of meteorological forcing data, catchment attributes, and discharge data for catchments around the world:<a href="../record/6578598" target="_blank" rel="noopener"> https://zenodo.org/record/6578598</a>. <br>We have employed the published code to derive meteorological forcing data and catchment attributes from global data sources to extend Caravan with data for <strong>1887 catchments</strong> in Germany. The time series data are in <strong>daily resolution and span up to 70 years, from January 1951 to December 2020</strong>.</p> <p>Most of the catchments in Caravan-DE are also part of the CAMELS-DE dataset (<a href="https://doi.org/10.5281/zenodo.13837553" target="_blank" rel="noopener">10.5281/zenodo.13837553</a>, 1582 catchments). As CAMELS-DE relies on meteorological forcing data that is only available within the borders of Germany, catchments going beyond the German national borders had to be discarded. Caravan uses global data products for the meteorological forcing data and catchment attributes, which is why Caravan-DE includes these catchments that are partly located outside of Germany. As catchments in Caravan-DE and CAMELS-DE are identified by the same ID, the datasets can be used together.<br>Please refer to the CAMELS-DE paper (<a href="https://doi.org/10.5194/essd-2024-318">https://doi.org/10.5194/essd-2024-318</a>) for information about discharge data and the catchment geometries used for both Caravan-DE and CAMELS-DE.</p> <p>For the processing of the data the following guide was followed step by step: <a href="https://github.com/kratzert/Caravan/wiki/Extending-Caravan-with-new-basins">https://github.com/kratzert/Caravan/wiki/Extending-Caravan-with-new-basins</a></p> <h3> </h3> <h3>Disclaimer for discharge and water level data provided by the German federal state agencies:</h3> <p>english:<em><br>The state agencies do not guarantee the accuracy or completeness of the discharge or water level data provided. In addition, all hydrological data may be subject to future revisions, including adjustments to the rating curves or corrections of errors. Therefore, it is necessary to obtain the most recent discharge time series directly from the federal state authorities for projects that require water law permits. Additionally, the regulations of the respective federal state apply and specific enquiries should be made as needed. It is also important to note that the state agencies explicitly disclaim any warranty as to the accuracy or completeness of the data and therefore any liability claims against any of the federal states are also excluded.</em></p> <p>german:<em><br>Die Ländesämter gewährleisten nicht die Genauigkeit oder Vollständigkeit der bereitgestellten Abfluss oder Wasserstandsdaten. Zudem können alle hydrologischen Daten zukünftigen Überarbeitungen unterliegen, einschließlich Anpassungen der Wasserstands-Abflussbeziehung oder der Korrektur von Fehlern. Daher ist es notwendig, die aktuellsten Abflusszeitreihen direkt bei den Landesbehörden zu beziehen, falls Wasserrechtsgenehmigungen erforderlich sind. Zusätzlich gelten die Vorschriften des jeweiligen Bundeslandes, und spezifische Anfragen sollten bei Bedarf gestellt werden. Es ist ebenfalls wichtig zu beachten, dass die staatlichen Behörden ausdrücklich jegliche Gewährleistung hinsichtlich der Genauigkeit oder Vollständigkeit der Daten ausschließen und somit auch jegliche Haftungsansprüche gegenüber einem der Bundesländer ausgeschlossen sind.</em></p> <p> </p> <h3>Changelog</h3> <ul> <li>v1.0.1 <ul> <li>Small changes to be consistent with other Caravan extensions (see <a href="https://github.com/kratzert/Caravan/issues/35" target="_blank" rel="noopener">https://github.com/kratzert/Caravan/issues/35</a>): <ul> <li>camelsde_basin_shapes.shp: removed column gauge_name</li> <li> <div>attributes_other_camelsde.csv: added river name to gauge_name column</div> </li> </ul> </li> </ul> </li> <li>v1.1.0 <ul> <li>Added FAO Penman-Monteith PET (potential_evaporation_sum_FAO_PENMAN_MONTEITH) and renamed the ERA5-LAND potential_evaporation band to potential_evaporation_sum_ERA5_LAND. Also added all PET-related climate indices derived with the Penman-Monteith PET band (suffix "_FAO_PM") and renamed the old PET-related indices accordingly (suffix "_ERA5_LAND"). <br>This was conducted to be consistent with <a href="https://doi.org/10.5281/zenodo.14673536" target="_blank" rel="noopener">Caravan version 1.5</a>.</li> </ul> </li> <li>v1.1.1 <ul> <li>By mistake, the new variables for v1.1.0 were missing in the climatic indices, these have now been added.</li> </ul> </li> </ul>
Large EBSD Dataset of Forged Ti64
<p>An extremely large EBSD dataset - obtained from an alpha + beta deformed forging measuring approximately 7 x 5 cm with 45’268’880 data points and 68% coverage. Sample prep conducted by Nicholas Byres, EBSD microscopy conducted by Alec Davis.</p>
The Large Labelled Logo Dataset (L3D)
<p>Public release of the dataset from the paper: "The Large Labelled Logo Dataset (L3D): A Multipurpose and Hand-Labelled Continuously Growing Dataset"</p> <p>More information: <a href="https://github.com/lhf-labs/tm-dataset">https://github.com/lhf-labs/tm-dataset</a></p> <p>Citing:</p> <pre><code>@misc{gutierrezfandino2021L3D, title={The Large Labelled Logo Dataset (L3D): A Multipurpose and Hand-Labelled Continuously Growing Dataset}, author={Asier Gutiérrez-Fandiño and David Pérez-Fernández and Jordi Armengol-Estapé}, year={2021}, eprint={2112.05404}, archivePrefix={arXiv}, primaryClass={cs.CV} }</code></pre> <p> </p>
The Piraeus AIS Dataset for Large-scale Maritime Data Analytics
<p><strong>AIS data collected by the University of Piraeus' AIS receiver</strong></p> <p> </p> <p><strong>Abstract</strong></p> <p>The advent of Big Data and streaming technologies has resulted in a swarm of voluminous, heterogeneous information, especially in the domains of Internet of Things (IoT) and transportation. Focusing on the maritime field, we present a dataset that contains vessel position information transmitted by vessels of different types and collected via the Automatic Identification System (AIS). The AIS dataset comes along with spatially and temporally correlated data about the vessels and the area of interest, including weather information. It covers a time span of over 2.5 years, from May 9<sup>th</sup>, 2017 to December 26<sup>th</sup>, 2019 and provides anonymised vessel positions within the wider area of the port of Piraeus (Greece), one of the busiest ports in Europe and worldwide. The dataset consists of over 244 million AIS records, an average of more than 10,000 records per hour, which makes it an ideal input for large-scale mobility data processing and analytics purposes.</p> <p> </p> <p><strong>Dataset related to the following publication</strong></p> <blockquote> <p>Andreas Tritsarolis, Yannis Kontoulis, Yannis Theodoridis, The Piraeus AIS dataset for large-scale maritime data analytics, Data in Brief, Volume 40, 2022, 107782, ISSN 2352-3409, <a href="https://doi.org/10.1016/j.dib.2021.107782">https://doi.org/10.1016/j.dib.2021.107782</a>.</p> </blockquote> <p> </p> <p><strong>Files Description</strong></p> <ul> </ul> <ul> <li><strong>ais_static</strong>: CSV flat files containing vessels' static information and their corresponding types</li> </ul> <ul> <li><strong>geodata</strong>: ESRI Shapefiles containing several geographic-related data (e.g. harbours, islands, etc.)</li> </ul> <ul> <li><strong>noaa_weather</strong>: ESRI Shapefiles containing weather forecast from GRIB files (as provided by NOAA)</li> </ul> <ul> <li><strong>unipi_ais_dynamic</strong>: CSV flat files containing AIS kinematic information </li> </ul> <ul> <li><strong>unipi_ais_dynamic_synopses</strong>: CSV flat files containing metadata (i.e. synopses) regarding vessels' AIS positions</li> </ul> <p> </p> <p><strong>Privacy Statement</strong></p> <p><strong>For privacy-related queries, please contact the authors</strong></p>
Source data to publication "Benchmarking of Analysis Strategies for Data-Independent Acquisition Proteomics Using a Large-Scale Dataset Comprising Inter-Patient Heterogeneity"
<p>Source data to publication "Benchmarking of Analysis Strategies for Data-Independent Acquisition Proteomics Using a Large-Scale Dataset Comprising Inter-Patient Heterogeneity".</p> <p>Data and further information at GitHub repository https://github.com/kreutz-lab/dia-benchmarking (DOI: 10.5281/zenodo.6371925)</p>
Dataset of Jupyter Notebooks from the paper "A Large-Scale Comparison of Python Code in Jupyter Notebooks and Scripts"
<pre>This archive contains the dataset of properly-licensed Jupyter notebooks from the MSR'22 paper "A Large-Scale Comparison of Python Code in Jupyter Notebooks and Scripts". The dataset contains 847,881 notebooks stored in the PostgreSQL dump file. You can find the details about the database in the README file. To transform the notebooks into this convenient format and to calcuate the structural metrics, we used our library called Matroskin, which can be found here: <a href="https://github.com/JetBrains-Research/Matroskin">https://github.com/JetBrains-Research/Matroskin</a>. </pre>
EvaNIL: silver standard dataset for large-scale NIL entity linking evaluation
<p>The EvaNIL dataset can be used to train or evaluate approaches developed for NIL entity linking. It was built from several Biomedical and Life Sciences corpora:</p> <ul> <li>PubMed DS</li> <li>CRAFT corpus</li> <li>MedMentions</li> </ul> <p>These corpora contain entities associated with knowledge base concepts. To build the EvaNIL dataset, we assumed that those knowledge base concepts did not exist in the respective knowledge bases, so each entity is associated instead with the direct ancestors of those original concepts.</p> <p>The EvaNIL dataset is divided into 6 partitions including annotations from several knowledge bases:</p> <ul> <li>"medic" (CTD-MEDIC)</li> <li>"ctd_anatomy" (CTD-Anatomy)</li> <li>"ctd_chemicals" (CTD-Chemicals)</li> <li>"chebi" (ChEBI)</li> <li>"go_bp" (GO-Biological Process)</li> <li>"hp" (HPO)</li> </ul> <p> </p>
GTAV-NightRain: Photometric Realistic Large-scale Dataset for Night-time Rain Streak Removal
<p>Existing synthetic deraining datasets ignored the photometry property of rain streaks and superimposed 2D rain layer onto clean images without any interactions between rain and environment, making rain streaks unrealistic. The appearance of rain can change drastically with space due to environmental illumination, especially in night scenes where lights are not parallel and uniform. Considering photometry and three dimensional space, rain streaks rendered in GTA V look much more realistic. With proper modifications, it can turn into a good platform for collecting paired data on deraining.</p> <p>GTAV-NightRain dataset is a large-scale photometric realistic dataset for night-time rain streak removal. This dataset contains 12860 rainy images together with 1286 rain-free ground truths collected in GTA V. Two rain shape and different rain density are included.</p> <p>We also provide a small version of the dataset on <a href="https://drive.google.com/drive/folders/1Tsoh_9iCfYc2rtMCHnJACnKW5W_SOheb?usp=sharing">Google Drive</a>. The dataset will be continuously maintained and information of updates will be in informed at <a href="https://github.com/zkawfanx/GTAV-NightRain">Github</a>. Please check it for more information and stay tuned if you are interested in our dataset.</p>
JetClass: A Large-Scale Dataset for Deep Learning in Jet Physics
<p>JetClass is a new large-scale dataset to facilitate deep learning research in jet physics. It consists of 100M jets for training, 5M for validation and 20M for testing. The dataset contains 10 classes of jets, simulated with MadGraph + Pythia + Delphes. <br> <br> A detailed description of the JetClass dataset is presented in the paper <a href="https://arxiv.org/abs/2202.03772">Particle Transformer for Jet Tagging</a>. An interface to use the dataset is provided in <a href="https://github.com/jet-universe/particle_transformer">https://github.com/jet-universe/particle_transformer</a>.</p>
A Large Scale Fabrication of Graphene Based Nano-electromechanical Contact Switches With Ultra-low Pull-in Voltage [Dataset]
<p>Original data set for the 'A Large Scale Fabrication of Graphene Based Nano-electromechanical Contact<br> Switches With Ultra-low Pull-in Voltage' is uploaded.</p>
Modelling assumptions and input dataset for the case study of the paper "Societal Effects of Large-Scale Energy Storage in the Current and Future Day-Ahead Market: A Belgian Case Study"
<p>This data package includes the modelling assumptions and input data to replicate the results of the case study included in the paper "Societal Effects of Large-Scale Energy Storage in the Current and Future Day-Ahead Market: A Belgian Case Study". This paper is part of the 18th International Conference on the European Energy Market (EEM22).</p> <p>The case study models the Belgian day-ahead electricity market, in which the existing storage is considered, in addition to large-scale battery energy storage systems of different sizes for varying renewable energy shares. A detailed description of the case study is provided in the readme file. </p> <p>This supplementary data package includes the following files: </p> <p> --Belgium Model Input Data.xlsx: Dataset used as input in the case study of the mentioned paper<br> --Modelling Assumptions.pdf: Modelling assumptions considered in the case study<br> --readme.txt (this file): Includes a detailed description of the data package</p> <p> </p> <p>The data included in this dataset was collected from public open sources [1]-[2]. Please notice that this dataset does not replace the original open access information. For accessing the data, please visit the following websites:</p> <p>[1] “ENTSO-E Transparency Platform.” [Online]. Available: https://transparency.entsoe.eu/dashboard/show. [Accessed: 06-Jul-2022].<br> [2] “Grid data.” [Online]. Available: https://www.elia.be/en/grid-data. [Accessed: 06-Jul-2022].</p> <p><br> </p> <p> </p>
Hunting for vampires and other unlikely forms of parity violation at the Large Hadron Collider: truth-jet and reco-jet datasets
<p>All truth-jet and reco-jet datasets used in the <a href="https://arxiv.org/abs/2205.09876">paper</a>.</p> <p><strong>Main</strong>: fragments are named `truth-jet_${MODEL}.tar.gz`, where MODEL is either `pv_msme_${lambdaPV}` (PV-mSME and lambdaPV is a floating point number with "." replaced with "p"), or `sm` for the Standard Model (lambdaPV = 0).</p> <p>Within each are three directories for the independent splits:</p> <ul> <li>train,</li> <li>test, and</li> <li>private_test.</li> </ul> <p> In the paper, we describe "test" as the validation set and "private_test" as the test set. Data in "private_test" were not used until models were finalized for the paper.<br> Within each of those are the processed results from simulations with different random seeds. They are independent shards which can be trivially combined.</p> <p>Each data file is in <a href="https://www.h5py.org/">h5</a> format. Its data are under the key "events" as an array with shape (n, 20). That last axis contains the reconstructed four-momenta of the hardest five jets in the order [Px, Py, Pz, E] * 5, with missing jets filled with zeros.</p> <p>Truth-jet files have additional keys "flavors" and "helicities". These contain truth-level flavour and helicity information, respectively. Their shape is (n, 5), where the last axis contains the zero-padded results. Flavours are encoded in the PDG ID scheme.</p> <p><strong>Rotated</strong>: Rotated PV-mSME data are in archives named `truth-jet-rot_${HOUR}.tar.gz`.<br> These have similar (train, test, private_test) structures as the others, but comprise parts of the same mixed model and should be combined by subsampling to a weighted average.</p> <p>HOUR is an integer rotation in [0, 23] for which the detector is rotated by an angle given in radians as HOUR * 2 pi / 24. This rotated dataset has lambdaPV=1. We demonstrate merging of these datasets in the code sharing (<a href="https://github.com/Rupt/paper-hunting-vampires">git</a>, <a href="https://doi.org/10.5281/zenodo.6827723">Zenodo</a>).</p>
Hunting for vampires and other unlikely forms of parity violation at the Large Hadron Collider: calo-image datasets for the standard model
<p>An example calo-image dataset used in the <a href="https://arxiv.org/abs/2205.09876">paper</a>: standard model.</p> <p>Each shard is named `calo-image_sm_${DATASET}_${INDEX}.tar.gz`. Each contains one data file. DATASET is in {train,test,private_test} to label the three independent splits for training, validation, and testing respectively. INDEX labels separate batches which should be trivially combined.</p> <p>Each data file is in <a href="https://www.h5py.org/">h5</a> format. Its data are under the key "entries" as an array with shape (n, 32, 32) corresponding to (event_index, eta, phi) for the n calorimeter images in an unrolled eta--phi surface.</p> <p> </p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.