Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
6
datasets available to search
ShareScore release 0.9.0
Dataset results
6 results for “HTTPS”
Dataset used for fingerprinting of DNS over HTTPS responses.
<p><strong> </strong>The dataset consists of multiple different data sources:</p> <ol> <li>DoH enabled Firefox on Linux OS</li> <li>DoH enabled Firefox on Windows 10 OS</li> <li>DoH enabled Chrome on Windows 10 OS</li> </ol> <p> </p> <p>We captured the traffic from the DoH enabled web-browsers using tcpdump. To automate the process of traffic generation, we installed Google Chrome and Mozilla Firefox into separate virtual machines and controlled them with the Selenium framework shows detailed information about used browsers and environments). Selenium simulates a user's browsing according to the predefined script and a list of domain names (i.e., URLs from Alexa's top websites list in our case). The selenium was configured to visit pages in random order multiple times. For capturing the traffic, we used the default settings of each browser. We did not disable the DNS cache of the browser, and the random order of visiting webpages secures that the dataset contains traces influenced by DNS caching mechanisms. Each virtual machine was configured to export TLS cryptographic keys, that was used for decrypting the traffic using WireShark application. </p> <p>The WireShark text output of the decrypted traffic is provided in the dataset files. The detailed information about each file is provided in dataset README.</p> <p> </p> <p> </p> <p> </p>
HTTPS Brute-force dataset with extended network flows
<p>We are publishing a dataset we created for designing a brute-force detector of attacks in HTTPS. The dataset consists of extended network flows that we captured with flow exporter <a href="https://github.com/CESNET/ipfixprobe">Ipifixprobe</a>. Apart from traditional fields like source and destination IP addresses and ports, each flow contains information (size, direction, inter-packet time, TCP flags) about up to the first 100 packets. The sizes of packets are taken from the transport layer (TCP, UPD); packets with zero payload (e.g., TCP ACKs) are ignored.</p> <p>We publish three files:</p> <ul> <li><em>flows.csv</em>, which contains raw flow data.</li> <li><em>aggregated_flows.csv</em>, which contains aggregated flows</li> <li><em>samples.csv</em>, which contains samples with extracted features. This data can be used for training a machine-learning classification model.</li> </ul> <p> </p> <p>All IP addresses, source ports, TLS SNIs are sha256-hashed. Column <em>CLASS</em> is 0 for benign samples and 1 for brute-force samples.</p> <p><br> <strong>Brute-force data</strong><br> The brute-force data were generated with three popular attack tools - Ncrack, Thc-hydra, and Patator. Attacks were performed against these applications:</p> <ul> <li> WordPress</li> <li> Joomla </li> <li> MediaWiki</li> <li> Ghost</li> <li> Grafana</li> <li> Discourse</li> <li> PhpBB</li> <li> OpenCart</li> <li> Redmine</li> <li> Nginx</li> <li> Apache</li> </ul> <p>The <em>SCENARIO</em> columns indicate which tool and application were used to generate the sample.</p> <p><strong>Benign data</strong><br> Bening data consists of eight captures from a backbone network. The <em>SCENARIO</em> column indicates individual captures.</p> <p> </p> <p> </p> <p> </p>
List of DNS over HTTPS resolvers on the internet
<p>The DoH Internet Servers dataset comprises a verified list of Internet servers offering DNS over HTTPS (DoH) service. The list was created by active scanning of the IPv4 address space. The scanning was done two times in April 2021 and in January 2022. Together, the list contains 1987 different IP addresses with their reverse DNS record (if available), supported DoH method, and TLS 1.3 support. </p> <p>The scanning was done in three phases:</p> <ol> <li>We scanned the IPv4 address space for opened port 443 using masscan. </li> <li>IP addresses found in the previous step were scanned for DoH support using a custom Nmap-NSE script.</li> <li>The IP addresses found in previous steps were reached by a slower python script that also validated responses, obtained domain names.</li> </ol> <p> </p> <p><strong>Scanning limitation: </strong>The main limitation of our scanning was that it could not find DoH resolvers hosted on infrastructures hosting multiple services behind a single IP address. In such cases, an SNI, or HTTP Host header, or HTTP/2 <em>:authority</em> header is needed for a successful request. Since we did not have the SNI, it was impossible to provide it. </p> <p><strong>Used Scripts:</strong></p> <p>NMap: https://github.com/cejkato2/dns-doh.nse<br> Python: https://github.com/hynekkar/DoH-Checker</p> <p> </p> <p> </p> <p> </p>
Dataset of DNS over HTTPS (DoH) Internet Servers
<p><strong>Description</strong></p> <p>The DoH Internet Servers dataset comprises a verified list of Internet servers offering DNS over HTTPS (DoH). This is an updated <a href="http://doi.org/10.17632/ny4m53g6bw.1">10.17632/ny4m53g6bw.1</a> The list was created through the aggregation of a previously existing, but incomplete, list of DoH servers. The servers in this dataset went through a verification phase where it was confirmed they were active and working as advertised. The verification was done between May 1st, 2022, and May 4th, 2022. The dataset contains a total of 254 unique DoH servers, out of which 136 are over IPv4 and 118 over IPv6. The DoH servers belong to 59 unique Autonomous Systems and are associated with a total of 106 unique domain names.</p> <p>The following public lists of existing DoH servers were used to create this dataset:</p> <ul> <li>https://developers.google.com/speed/public-dns/docs/doh/json</li> <li>https://blog.nightly.mozilla.org/2018/06/01/improving-dns-privacy-in-firefox/</li> <li>https://github.com/curl/curl/wiki/DNS-over-HTTPS</li> <li>https://help.keenetic.com/hc/en-us/articles/360007687159-DNS-over-TLS-and-DNS-over-HTTPS-proxy-servers-for-DNS-requests-encryption</li> <li>https://dnsprivacy.org/wiki/display/DP/DNS+Privacy+Public+Resolvers</li> <li>https://kb.adguard.com/en/general/dns-providers</li> <li>https://applied-privacy.net/services/dns/</li> <li>https://www.pacnog.org/pacnog24/presentations/DoT-DoH-DNS-Privacy.pdf</li> <li>https://www.privacytools.io/providers/dns/</li> </ul> <p> </p> <p>The verification of the DoH servers was performed using a custom-made python script. The script is available at: https://github.com/stratosphereips/DoH-Research/tree/main/validation-script</p>
Supplementary files for Collection of Datasets with DNS over HTTPS Traffic
<p>The DNS over HTTPS (DoH) is becoming a default option for domain resolution in modern privacy-aware software. Therefore, research has already focused on various aspects; however, a comprehensive dataset from an actual production network is still missing. In this paper, we present a novel dataset, which comprises multiple PCAP files of DoH traffic. The captured traffic is generated towards various DoH providers to cover differences of various DoH server implementations and configurations. In addition to generated traffic, we also provide real network traffic captured on high-speed backbone lines of a large Internet Service Provider with around half a million users. Network identifiers (excluding network identifiers of DoH resolvers) in the real network traffic (e.g., IP addresses and transmitted content) were anonymized, but still, the important characteristics of the traffic can still be obtained from the data that can be used, e.g., for network traffic classification research. The real network traffic dataset contains DoH and also non-DoH HTTPS traffic as observed at the collection points in the network.</p> <p> </p> <p>This repository provides supplementary files for the "Collection of Datasets with DNS over HTTPS Traffic" :</p> <pre><code>─── supplementary_files | - Directory with supplementary files (scripts, DoH resolver list) used for dataset creation ├── chrome | - Generation scripts for Chrome browser and visited websites during generation ├── doh_resolvers | - The list of DoH resolvers used for filter creation during ISP backbone capture ├── firefox | - Generation scripts for Firefox browser and visited websites during generation └── pcap-anonymizer | - Anonymization script of real backbone captures</code></pre> <p>Collection of datasets:</p> <ul> <li><strong>DoH-Gen-F-AABBC</strong> --- <a href="https://doi.org/10.5281/zenodo.5957277">https://doi.org/10.5281/zenodo.5957277</a></li> <li><strong>DoH-Gen-F-FGHOQS</strong> --- <a href="https://doi.org/10.5281/zenodo.5957121">https://doi.org/10.5281/zenodo.5957121</a></li> <li><strong>DoH-Gen-F-CCDDD</strong> --- <a href="https://doi.org/10.5281/zenodo.5957420">https://doi.org/10.5281/zenodo.5957420</a></li> <li><strong>DoH-Gen-C-AABBCC </strong>--- <a href="https://doi.org/10.5281/zenodo.5957465">https://doi.org/10.5281/zenodo.5957465</a></li> <li><strong>DoH-Gen-C-DDD</strong> --<a href="https://doi.org/10.5281/zenodo.5957676"> https://doi.org/10.5281/zenodo.5957676</a></li> <li><strong>DoH-Gen-C-CFGHOQS</strong> --- <a href="https://doi.org/10.5281/zenodo.5957659">https://doi.org/10.5281/zenodo.5957659</a></li> <li><strong>DoH-Real-world</strong> --- <a href="https://doi.org/10.5281/zenodo.5956043">https://doi.org/10.5281/zenodo.5956043</a></li> </ul> <p> </p> <p> </p>
Dataset used for detecting DNS over HTTPS by Machine Learning.
<p><strong> </strong>The dataset consists of three different data sources:</p> <ol> <li> DoH enabled Firefox</li> <li>DoH enabled Google Chrome</li> <li>Cloudflared DoH proxy</li> </ol> <p>The capture of web browser data was made using the Selenium framework, which simulated classical user browsing. The browsers received command for visiting domains taken from Alexa's top 10K most visited websites. The capturing was performed on the host by listening to the network interface of the virtual machine. Overall the dataset contains almost 5,000 web-page visits by Mozilla and 1,000 pages visited by Chrome.</p> <p>The Cloudflared DoH proxy was installed in Raspberry PI, and the IP address of the Raspberry was set as the default DNS resolver in two separate offices in our university. It was continuously capturing the DNS/DoH traffic created up to 20 devices for around three months.</p> <p>The dataset contains 1,128,904 flows from which is around 33,000 labeled as DoH. We provide raw pcap data, CSV with flow data, and CSV file with extracted features.</p> <p>The CSV with extracted features has the following data fields:</p> <p>- Label (1 - Doh, 0 - regular HTTPS)<br> - Data source<br> - Duration<br> - Minimal Inter-Packet Delay<br> - Maximal Inter-Packet Delay<br> - Average Inter-Packet Delay<br> - A variance of Incoming Packet Sizes<br> - A variance of Outgoing Packet Sizes<br> - A ratio of the number of Incoming and outgoing bytes<br> - A ration of the number of Incoming and outgoing packets<br> - Average of Incoming Packet sizes<br> - Average of Outgoing Packet sizes<br> - The median value of Incoming Packet sizes<br> - The median value of outgoing Packet sizes<br> - The ratio of bursts and pauses<br> - Number of bursts<br> - Number of pauses<br> - Autocorrelation<br> - Transmission symmetry in the 1st third of connection<br> - Transmission symmetry in the 2nd third of connection<br> - Transmission symmetry in the last third of connection</p> <p>The observed network traffic does not contain privacy-sensitive information. </p> <p>The zip file structure is:</p> <pre><code>|-- data | |-- extracted-features...extracted features used in ML for DoH recognition | | |-- chrome | | |-- cloudflared | | `-- firefox | |-- flows...............................................exported flow data | | |-- chrome | | |-- cloudflared | | `-- firefox | `-- pcaps....................................................raw PCAP data | |-- chrome | |-- cloudflared | `-- firefox |-- LICENSE `-- README.md</code></pre> <p><br> When using this dataset, please cite the original work as follows:</p> <pre><code>@inproceedings{vekshin2020, author = {Vekshin, Dmitrii and Hynek, Karel and Cejka, Tomas}, title = {DoH Insight: Detecting DNS over HTTPS by Machine Learning}, year = {2020}, isbn = {9781450388337}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3407023.3409192}, doi = {10.1145/3407023.3409192}, booktitle = {Proceedings of the 15th International Conference on Availability, Reliability and Security}, articleno = {87}, numpages = {8}, keywords = {classification, DoH, DNS over HTTPS, machine learning, detection, datasets}, location = {Virtual Event, Ireland}, series = {ARES '20} } </code></pre> <p> </p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.