Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
13
datasets available to search
ShareScore release 0.7.1
Dataset results
13 results for “netflow”
AIT Netflow Data Set
<p><strong>AIT Netflow Data Sets</strong></p> <p>This repository contains labeled synthetic netflows suitable for evaluation of intrusion detection systems, federated learning, and alert aggregation. The netflows are generated from the packet captures contained in the <a href="http://doi.org/10.5281/zenodo.5789064">AIT-LDS-v2.0</a>. A detailed description of that dataset is available in [1]. The packet captures were collected from eight testbeds that were built at the Austrian Institute of Technology (AIT) following the approach by [2]. Please cite these papers if the data is used for academic publications.</p> <p>In brief, each of the datasets corresponds to a testbed representing a small enterprise network including mail server, file share, WordPress server, VPN, firewall, etc. Normal user behavior is simulated to generate background noise over a time span of 4-6 days. At some point, a sequence of attack steps is launched against the network. The following attacks are launched in the network:</p> <ul> <li>Scans (nmap, WPScan, dirb)</li> <li>Webshell upload (CVE-2020-24186)</li> <li>Password cracking (John the Ripper)</li> <li>Privilege escalation</li> <li>Remote command execution</li> <li>Data exfiltration (DNSteal)</li> </ul> <p>This repository contains the following files:</p> <ul> <li><em><testbed>_netflows.zip</em>: CSV files of labeled TCP and UDP netflows for each testbed.</li> <li><em>label_info.txt</em>: File describing which labels in TCP and UDP are benign and which ones are malicious.</li> <li><em>README.md</em>: Instructions on how to reproduce the generation and labeling of the netflows from the <a href="http://doi.org/10.5281/zenodo.5789064">AIT-LDS-v2.0</a>. Note that it is only necessary to run the python scripts if you want to extend or change the labeling procedure.</li> <li><em>1_format_dataset_info.ipynb</em>: Generates the tables necessary for labeling (see README.md).</li> <li><em>2_label_logs.ipynb</em>: Labels the netflows (see README.md).</li> </ul> <p>Acknowledgements: Partially funded by the FFG projects INDICAETING (868306) and DECEPT (873980), and the EU projects GUARD (833456) and PANDORA (SI2.835928).</p> <p><strong>If you use the dataset, please cite the following publications:</strong></p> <p>[1] M. Landauer, F. Skopik, M. Frank, W. Hotwagner, M. Wurzenberger, and A. Rauber. <a href="https://ieeexplore.ieee.org/abstract/document/9866880">"Maintainable Log Datasets for Evaluation of Intrusion Detection Systems"</a>. IEEE Transactions on Dependable and Secure Computing, vol. 20, no. 4, pp. 3466-3482. [<a href="https://arxiv.org/pdf/2203.08580.pdf">PDF</a>]</p> <p>[2] M. Landauer, F. Skopik, M. Wurzenberger, W. Hotwagner and A. Rauber, <a href="https://ieeexplore.ieee.org/document/9262078">"Have it Your Way: Generating Customized Log Datasets With a Model-Driven Simulation Testbed,"</a> in IEEE Transactions on Reliability, vol. 70, no. 1, pp. 402-415, March 2021, doi: 10.1109/TR.2020.3031317. [<a href="https://www.skopik.at/ait/2020_trel.pdf">PDF</a>]</p>
Netflow data without sampling for test (D2)
<p>NetFlow traffic generated using <strong>DOROTHEA</strong> (<strong>DO</strong>cker-based f<strong>R</strong>amework f<strong>O</strong>r ga<strong>TH</strong>ering n<strong>E</strong>tflow tr<strong>A</strong>ffic)</p> <p>NetFlow is a network protocol developed by Cisco for the collection and monitoring of network traffic flow data generated. A flow is defined as a unidirectional sequence of packets with some common properties that pass through a network device.</p> <p>NetFlow flows have been captured without sampling at the packet level. A sampling means that 1 out of every X packets is selected to be flow while the rest of the packets are not valued.</p> <p>The version of NetFlow used to build the datasets is 5.</p> <p>In the construction of the datasets, different percentages of flows considered attacks and flows considered normal traffic have been used.</p> <p>These datasets have been used to test machine learning models.</p>
Netflow data without sampling for training (D1)
<p>NetFlow traffic generated using <strong>DOROTHEA</strong> (<strong>DO</strong>cker-based f<strong>R</strong>amework f<strong>O</strong>r ga<strong>TH</strong>ering n<strong>E</strong>tflow tr<strong>A</strong>ffic)</p> <p>NetFlow is a network protocol developed by Cisco for the collection and monitoring of network traffic flow data generated. A flow is defined as a unidirectional sequence of packets with some common properties that pass through a network device.</p> <p>NetFlow flows have been captured without sampling at the packet level. A sampling means that 1 out of every X packets is selected to be flow while the rest of the packets are not valued.</p> <p>The version of NetFlow used to build the datasets is 5.</p> <p>In the construction of the datasets, different percentages of flows considered attacks and flows considered normal traffic have been used.</p> <p>These datasets have been used to train machine learning models.</p>
NetFlow
<p>This is a preprocessed dataset in numpy format.</p> <p>The dataset consists of multi-variate time series contained in an array of shape [n_samples, time_steps, n_variables] associated with class labels contained in an array of shape [n_samples, 1]. It can be loaded as follows:</p> <pre><code>loaded_data = np.load("NET.npz") Xtr = loaded_data['Xtr'] # Training data of shape (803, 997, 4) Ytr = loaded_data['Ytr'] # Training labels of shape (803, 1) Xte = loaded_data['Xte'] # Test data of shape (534, 997, 4) Yte = loaded_data['Yte'] # Test labels of shape (534, 1)</code></pre>
Netflow data with sampling 500 for training (D2)
<p>NetFlow traffic generated using DOROTHEA (DOcker-based fRamework fOr gaTHering nEtflow trAffic) NetFlow is a network protocol developed by Cisco for the collection and monitoring of network traffic flow data generated. A flow is defined as a unidirectional sequence of packets with some common properties that pass through a network device.</p> <p>NetFlow flows have been captured with sampling 500 at the packet level. A sampling means that 1 out of every X packets is selected to be flow while the rest of the packets are not valued.</p> <p>The version of NetFlow used to build the datasets is 5.</p>
Netflow data with sampling 250 for test (D4)
<p>NetFlow traffic generated using DOROTHEA (DOcker-based fRamework fOr gaTHering nEtflow trAffic) NetFlow is a network protocol developed by Cisco for the collection and monitoring of network traffic flow data generated. A flow is defined as a unidirectional sequence of packets with some common properties that pass through a network device.</p> <p>NetFlow flows have been captured with sampling 250 at the packet level. A sampling means that 1 out of every X packets is selected to be flow while the rest of the packets are not valued.</p> <p>The version of NetFlow used to build the datasets is 5.</p>
Netflow data with sampling 500 for test (D5)
<p>NetFlow traffic generated using DOROTHEA (DOcker-based fRamework fOr gaTHering nEtflow trAffic) NetFlow is a network protocol developed by Cisco for the collection and monitoring of network traffic flow data generated. A flow is defined as a unidirectional sequence of packets with some common properties that pass through a network device.</p> <p>NetFlow flows have been captured with sampling 500 at the packet level. A sampling means that 1 out of every X packets is selected to be flow while the rest of the packets are not valued.</p> <p>The version of NetFlow used to build the datasets is 5.</p>
Netflow data with sampling 250 for training (D1)
<p>NetFlow traffic generated using DOROTHEA (DOcker-based fRamework fOr gaTHering nEtflow trAffic) NetFlow is a network protocol developed by Cisco for the collection and monitoring of network traffic flow data generated. A flow is defined as a unidirectional sequence of packets with some common properties that pass through a network device.</p> <p>NetFlow flows have been captured with sampling 250 at the packet level. A sampling means that 1 out of every X packets is selected to be flow while the rest of the packets are not valued.</p> <p>The version of NetFlow used to build the datasets is 5.</p>
Netflow data with sampling 1000 for training (D3)
<p>NetFlow traffic generated using DOROTHEA (DOcker-based fRamework fOr gaTHering nEtflow trAffic) NetFlow is a network protocol developed by Cisco for the collection and monitoring of network traffic flow data generated. A flow is defined as a unidirectional sequence of packets with some common properties that pass through a network device.</p> <p>NetFlow flows have been captured with sampling 1000 at the packet level. A sampling means that 1 out of every X packets is selected to be flow while the rest of the packets are not valued.</p> <p>The version of NetFlow used to build the datasets is 5.</p>
Netflow data with sampling collected from RedCAYLE (D7)
<p>NetFlow traffic generated using DOROTHEA (DOcker-based fRamework fOr gaTHering nEtflow trAffic) NetFlow is a network protocol developed by Cisco for the collection and monitoring of network traffic flow data generated. A flow is defined as a unidirectional sequence of packets with some common properties that pass through a network device.</p> <p>NetFlow flows have been captured with sampling 1000 at the packet level. A sampling means that 1 out of every X packets is selected to be flow while the rest of the packets are not valued.</p> <p>The version of NetFlow used to build the datasets is 5.</p>
SQL Injection Attack Netflow
<p><strong>Introduction</strong></p> <p>This datasets have SQL injection attacks (SLQIA) as malicious Netflow data. The attacks carried out are SQL injection for Union Query and Blind SQL injection. To perform the attacks, the SQLMAP tool has been used.</p> <p>NetFlow traffic has generated using DOROTHEA (DOcker-based fRamework fOr gaTHering nEtflow trAffic). NetFlow is a network protocol developed by Cisco for the collection and monitoring of network traffic flow data generated. A flow is defined as a unidirectional sequence of packets with some common properties that pass through a network device.</p> <p><strong>Datasets</strong></p> <p>The firts dataset was colleted to train the detection models (<strong>D1</strong>) and other collected using different attacks than those used in training to test the models and ensure their generalization (<strong>D2</strong>).</p> <p>The datasets contain both benign and malicious traffic. All collected datasets are balanced.</p> <p>The version of NetFlow used to build the datasets is 5.</p> <table> <thead> <tr> <th scope="col">Dataset</th> <th scope="col">Aim</th> <th scope="col">Samples</th> <th scope="col">Benign-malicious<br> traffic ratio</th> </tr> </thead> <tbody> <tr> <td>D1</td> <td>Training</td> <td>400,003</td> <td>50%</td> </tr> <tr> <td>D2</td> <td>Test</td> <td>57,239</td> <td>50%</td> </tr> </tbody> </table> <p> </p> <p><strong>Infrastructure and implementation</strong></p> <p>Two sets of flow data were collected with <a href="https://niebla.unileon.es/cybersecurity/dorothea">DOROTHEA</a>. DOROTHEA is a Docker-based framework for NetFlow data collection. It allows you to build interconnected virtual networks to generate and collect flow data using the NetFlow protocol. In DOROTHEA, network traffic packets are sent to a NetFlow generator that has a sensor<a href="https://github.com/aabc/ipt-netflow"><em> ipt_netflow</em></a> installed. The sensor consists of a module for the Linux kernel using Iptables, which processes the packets and converts them to NetFlow flows.</p> <p>DOROTHEA is configured to use Netflow V5 and export the flow after it is inactive for 15 seconds or after the flow is active for 1800 seconds (30 minutes)</p> <p>Benign traffic generation nodes simulate network traffic generated by real users, performing tasks such as searching in web browsers, sending emails, or establishing Secure Shell (SSH) connections. Such tasks run as Python scripts. Users may customize them or even incorporate their own. The network traffic is managed by a gateway that performs two main tasks. On the one hand, it routes packets to the Internet. On the other hand, it sends it to a NetFlow data generation node (this process is carried out similarly to packets received from the Internet).</p> <p>The malicious traffic collected (SQLI attacks) was performed using <a href="https://sqlmap.org/">SQLMAP</a>. SQLMAP is a penetration tool used to automate the process of detecting and exploiting SQL injection vulnerabilities.</p> <p>The attacks were executed on 16 nodes and launch SQLMAP with the parameters of the following table.</p> <table> <thead> <tr> <th scope="col">Parameters</th> <th scope="col">Description</th> </tr> </thead> <tbody> <tr> <td>'--banner','--current-user','--current-db','--hostname','--is-dba','--users','--passwords','--privileges','--roles','--dbs','--tables','--columns','--schema','--count','--dump','--comments', --schema'</td> <td>Enumerate users, password hashes, privileges, roles, databases, tables and columns</td> </tr> <tr> <td>--level=5</td> <td>Increase the probability of a false positive identification</td> </tr> <tr> <td>--risk=3</td> <td>Increase the probability of extracting data</td> </tr> <tr> <td>--random-agent</td> <td>Select the User-Agent randomly</td> </tr> <tr> <td>--batch</td> <td>Never ask for user input, use the default behavior</td> </tr> <tr> <td>--answers="follow=Y"</td> <td>Predefined answers to yes</td> </tr> </tbody> </table> <p>Every node executed SQLIA on 200 victim nodes. The victim nodes had deployed a web form vulnerable to Union-type injection attacks, which was connected to the <a href="https://www.mysql.com/">MYSQL </a>or <a href="https://www.microsoft.com/es-es/sql-server/sql-server-2019">SQLServer </a>database engines (50% of the victim nodes deployed MySQL and the other 50% deployed SQLServer).</p> <p>The web service was accessible from ports 443 and 80, which are the ports typically used to deploy web services. The IP address space was 182.168.1.1/24 for the benign and malicious traffic-generating nodes. For victim nodes, the address space was 126.52.30.0/24.<br> The malicious traffic in the test sets was collected under different conditions. For <strong>D1</strong>, SQLIA was performed using Union attacks on the MySQL and SQLServer databases.</p> <p>However, for <strong>D2</strong>, BlindSQL SQLIAs were performed against the web form connected to a <a href="https://www.postgresql.org/">PostgreSQL </a>database. The IP address spaces of the networks were also different from those of <strong>D1</strong>. In <strong>D2</strong>, the IP address space was 152.148.48.1/24 for benign and malicious traffic generating nodes and 140.30.20.1/24 for victim nodes.</p> <p>To run the MySQL server we ran <a href="https://mariadb.org/">MariaDB</a> version 10.4.12.<br> Microsoft SQL Server 2017 Express and PostgreSQL version 13 were used.</p>
Netflow data with sampling 1000 for test (D6)
<p>NetFlow traffic generated using DOROTHEA (DOcker-based fRamework fOr gaTHering nEtflow trAffic) NetFlow is a network protocol developed by Cisco for the collection and monitoring of network traffic flow data generated. A flow is defined as a unidirectional sequence of packets with some common properties that pass through a network device.</p> <p>NetFlow flows have been captured with sampling 1000 at the packet level. A sampling means that 1 out of every X packets is selected to be flow while the rest of the packets are not valued.</p> <p>The version of NetFlow used to build the datasets is 5.</p>
NetFlow data collected with different packet sampling rates
<p>NetFlow traffic generated using DOROTHEA (DOcker-based fRamework fOr gaTHering nEtflow trAffic) NetFlow is a network protocol developed by Cisco for the collection and monitoring of network traffic flow data generated. A flow is defined as a unidirectional sequence of packets with some common properties that pass through a network device.</p> <p>NetFlow flows have been captured with different sampling at the packet level. A sampling means that 1 out of every X packets is selected to be flow while the rest of the packets are not valued.</p> <p>The version of NetFlow used to build the datasets is 5.</p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.