Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
251
datasets available to search
ShareScore release 0.9.0
Dataset results
251 results for “log data”
PIE LTER marsh water table height, logging data from the railroad Spartina marsh site on the Parker River, Newbury, MA, for April-November 2016.
Measurements of water table height in the Parker River marsh located downstream of the railroad bridge. Measurements were taken every 5 minutes at each logger along a transect of water level loggers running perpendicular to the Parker River bank at the railroad site, MAR-PR-Wtable-RR for April-November 2016.
PIE LTER marsh water table height, logging data from the Typha marsh site on the upper Parker River, Newbury, MA, for April-November 2016.
Measurements of water table height in the upper Parker River Typha sp. marsh. Measurements were taken every 5 minutes at each logger along a transect of water level loggers running perpendicular to the Parker River bank at the Typha site, MAR-PR-Wtable-T, for April - November 2016
Marsh water table height, logging data from the Typha site on the upper Parker River for September-October 2001
Measurements of water table height in the upper Parker River Typha sp. marsh. Measurements were taken every 4 minutes at each logger along a transect of water level loggers running perpendicular to the Parker River bank at the Typha site, MAR-PR-Wtable-T, for September - October 2001.
Marsh water table height, logging data from the Typha site on the upper Parker River for September-December 2002
Measurements of water table height in the upper Parker River Typha sp. marsh. Measurements were taken every 4 minutes at each logger along a transect of water level loggers running perpendicular to the Parker River bank at the Typha site, MAR-PR-Wtable-T, September - December 2002.
Marsh water table height, logging data from the Shad Creek Spartina marsh site for May-October 2017, Rowley, MA, PIE LTER.
Measurements of water table height in the Shad Creek marsh, Rowley, MA, located near the Shad Creek eddy flux tower. Measurements were taken every 5 minutes at each logger along a transect of water level loggers running perpendicular to the Shad creek stream bank, MAR-RO-Wtable-Shad for May - October 2017.
Marsh water table height, logging data from the Shad Creek Spartina marsh site for April-November 2018, Rowley, MA, PIE LTER.
Measurements of water table height in the Shad Creek marsh, Rowley, MA, located near the Shad Creek eddy flux tower. Measurements were taken every 5 minutes at each logger along a transect of water level loggers running perpendicular to the Shad creek stream bank, MAR-RO-Wtable-Shad for April-November 2018.
PIE LTER marsh water table height, logging data from the Nelson Island Spartina marsh site, Rowley, MA for May-October 2017.
Measurements of water table height in the Nelson Island marsh located near Nelson Island eddy flux tower. Measurements were taken every 5 minutes at each logger along a transect of water level loggers running perpendicular to the Nelson stream bank at Nelson Island, Rowley, MA, MAR-RO-Wtable-Nel for May-October 2017.
PIE LTER marsh water table height, logging data from the Nelson Island Spartina marsh site, Rowley, MA for April-November 2018.
Measurements of water table height in the Nelson Island marsh located near Nelson Island eddy flux tower. Measurements were taken every 5 minutes at each logger along a transect of water level loggers running perpendicular to the Nelson stream bank at Nelson Island, MAR-RO-Wtable-Nel for April-November 2018.
PIE LTER marsh water table height, logging data from the Nelson Island Spartina marsh site for April-November 2019, Rowley, MA.
Measurements of water table height in the Nelson Island marsh located near Nelson Island eddy flux tower, Rowley MA. Measurements were taken every 5 minutes at each logger along a transect of water level loggers running perpendicular to the Nelson stream bank at Nelson Island, MAR-PR-Wtable-Nel for April-November 2019.
Marsh water table height, logging data from the Shad Creek Spartina marsh site for April-November 2019, Rowley, MA, PIE LTER.
Measurements of water table height in the Shad Creek marsh located near the Shad Creek eddy flux tower, Rowley, MA. Measurements were taken every 5 minutes at each logger along a transect of water level loggers running perpendicular to the Nelson stream bank at Nelson Island, MAR-PR-Wtable-Shad for April-November 2019.
Data and code related to the article "Characterization of the Log-normal Model for Received Signal Strength Measurements in Real Wireless Sensor Networks"
<p>This upload contains the data and code related to the article "Characterization of the Log-normal Model for Received Signal Strength Measurements in Real Wireless Sensor Networks", (D.O.I: <a href="https://doi.org/10.3390/jsan9010012">10.3390/jsan9010012</a>) published in the the special issue on "Localization in Wireless Sensor Networks" of the <a href="https://www.mdpi.com/journal/jsan"><em>Journal of Sensor and Actuator Networks</em></a> (ISSN 2224-2708).</p> <p>The data and code included allows to replicate the results of the article.</p>
Traffic and Log Data Captured During a Cyber Defense Exercise
<p>This dataset was acquired during Cyber Czech – a hands-on cyber defense exercise (Red Team/Blue Team) held in March 2019 at Masaryk University, Brno, Czech Republic. Network traffic flows and a high variety of event logs were captured in an <em>exercise network</em> deployed in the KYPO Cyber Range Platform.</p> <p><strong>Contents</strong></p> <p>The dataset covers two distinct time intervals, which correspond to the official schedule of the exercise. The timestamps provided below are in the ISO 8601 date format. </p> <ul> <li>Day 1, March 19, 2019 <ul> <li>Start: 2019-03-19T11:00:00.000000+01:00 </li> <li>End: 2019-03-19T18:00:00.000000+01:00 </li> </ul> </li> <li>Day 2, March 20, 2019 <ul> <li>Start: 2019-03-20T08:00:00.000000+01:00 </li> <li>End: 2019-03-20T15:30:00.000000+01:00 </li> </ul> </li> </ul> <p>The captured and collected data were normalized into three distinct event types and they are stored as structured JSON. The data are sorted by a timestamp, which represents the time they were observed. Each event type includes a raw payload ready for further processing and analysis. The description of the respective event types and the corresponding data files follows. </p> <ul> <li><em>cz.muni.csirt.IpfixEntry.tgz</em> – an archive of IPFIX traffic flows enriched with an additional payload of parsed application protocols in raw JSON. </li> <li><em>cz.muni.csirt.SyslogEntry.tgz</em> – an archive of Linux Syslog entries with the payload of corresponding text-based log messages. </li> <li><em>cz.muni.csirt.WinlogEntry.tgz</em> – an archive of Windows Event Log entries with the payload of original events in raw XML. </li> </ul> <p>Each archive listed above includes a directory of the same name with the following four files, ready to be processed. </p> <ul> <li><em>data.json.gz </em>– the actual data entries in a single gzipped JSON file. </li> <li><em>dictionary.yml</em> – data dictionary for the entries. </li> <li><em>schema.ddl</em> – data schema for Apache Spark analytics engine. </li> <li><em>schema.jsch</em> – JSON schema for the entries. </li> </ul> <p>Finally, the exercise network topology is described in a machine-readable NetJSON format and it is a part of a set of auxiliary files archive – <em>auxiliary-material.tgz</em> – which includes the following. </p> <ul> <li><em>global-gateway-config.json</em> – the network configuration of the global gateway in the NetJSON format. </li> <li><em>global-gateway-routing.json</em> – the routing configuration of the global gateway in the NetJSON format. </li> <li><em>redteam-attack-schedule.{csv,odt}</em> – the schedule of the Red Team attacks in CSV and ODT format. Source for Table 2. </li> <li><em>redteam-reserved-ip-ranges.{csv,odt}</em> – the list of IP segments reserved for the Red Team in CSV and ODT format. Source for Table 1. </li> <li><em>topology.{json,pdf,png}</em> – the topology of the complete Cyber Czech exercise network in the NetJSON, PDF and PNG format. </li> <li><em>topology-small.{pdf,png}</em> – simplified topology in the PDF and PNG format. Source for Figure 1. </li> </ul> <p> </p>
AIT Log Data Set V1.1
<p><strong>AIT Log Data Sets</strong></p> <p>This repository contains synthetic log data suitable for evaluation of intrusion detection systems. The logs were collected from four independent testbeds that were built at the Austrian Institute of Technology (AIT) following the approach by Landauer et al. (2020) [1]. Please refer to the paper for more detailed information on automatic testbed generation and cite it if the data is used for academic publications. In brief, each testbed simulates user accesses to a webserver that runs Horde Webmail and OkayCMS. The duration of the simulation is six days. On the fifth day (2020-03-04) two attacks are launched against each web server.</p> <p>The archive AIT-LDS-v1_0.zip contains the directories "data" and "labels".</p> <p>The data directory is structured as follows. Each directory mail.<name>.com contains the logs of one web server. Each directory user-<ID> contains the logs of one user host machine, where one or more users are simulated. Each file log<UID>.log in the user-<ID> directories contains the activity logs of one particular user.</p> <p>Setup details of the web servers:</p> <ul> <li>OS: Debian Stretch 9.11.6</li> <li>Services: <ul> <li>Apache2</li> <li>PHP7</li> <li>Exim 4.89</li> <li>Horde 5.2.22</li> <li>OkayCMS 2.3.4</li> <li>Suricata</li> <li>ClamAV</li> <li>MariaDB</li> </ul> </li> </ul> <p>Setup details of user machines:</p> <ul> <li>OS: Ubuntu Bionic</li> <li>Services: <ul> <li>Chromium</li> <li>Firefox</li> </ul> </li> </ul> <p>User host machines are assigned to web servers in the following way:</p> <ul> <li>mail.cup.com is accessed by users from host machines user-{0, 1, 2, 6}</li> <li>mail.spiral.com is accessed by users from host machines user-{3, 5, 8}</li> <li>mail.insect.com is accessed by users from host machines user-{4, 9}</li> <li>mail.onion.com is accessed by users from host machines user-{7, 10}</li> </ul> <p>The following attacks are launched against the web servers (different starting times for each web server, please check the labels for exact attack times):</p> <ul> <li>Attack 1: multi-step attack with sequential execution of the following attacks: <ul> <li>nmap scan</li> <li>nikto scan</li> <li>smtp-user-enum tool for account enumeration</li> <li>hydra brute force login</li> <li>webshell upload through Horde exploit (CVE-2019-9858)</li> <li>privilege escalation through Exim exploit (CVE-2019-10149)</li> </ul> </li> <li>Attack 2: webshell injection through malicious cookie (CVE-2019-16885)</li> </ul> <p>Attacks are launched from the following user host machines. In each of the corresponding directories user-<ID>, logs of the attack execution are found in the file attackLog.txt:</p> <ul> <li>user-6 attacks mail.cup.com</li> <li>user-5 attacks mail.spiral.com</li> <li>user-4 attacks mail.insect.com</li> <li>user-7 attacks mail.onion.com</li> </ul> <p>The log data collected from the web servers includes</p> <ul> <li> Apache access and error logs</li> <li> syscall logs collected with the Linux audit daemon</li> <li> suricata logs</li> <li> exim logs</li> <li> auth logs</li> <li> daemon logs</li> <li> mail logs</li> <li> syslogs</li> <li> user logs</li> </ul> <p> <br> Note that due to their large size, the audit/audit.log files of each server were compressed in a .zip-archive. In case that these logs are needed for analysis, they must first be unzipped.<br> <br> Labels are organized in the same directory structure as logs. Each file contains two labels for each log line separated by a comma, the first one based on the occurrence time, the second one based on similarity and ordering. Note that this does not guarantee correct labeling for all lines and that no manual corrections were conducted.</p> <p>Version history and related data sets:</p> <ul> <li><a href="https://doi.org/10.5281/zenodo.3723083">AIT-LDS-v1.0</a>: Four datasets, logs from single host, fine-granular audit logs, mail/CMS. <ul> <li><a href="https://doi.org/10.5281/zenodo.4264796">AIT-LDS-v1.1</a>: Removed carriage return of line endings in audit.log files.</li> </ul> </li> <li><a href="http://doi.org/10.5281/zenodo.5789064">AIT-LDS-v2.0</a>: Eight datasets, logs from all hosts, system logs and network traffic, mail/CMS/cloud/web.</li> </ul> <p>Acknowledgements: Partially funded by the FFG projects INDICAETING (868306) and DECEPT (873980), and the EU project GUARD (833456).</p> <p><strong>If you use the dataset, please cite the following publication:</strong></p> <p>[1] M. Landauer, F. Skopik, M. Wurzenberger, W. Hotwagner and A. Rauber, <a href="https://ieeexplore.ieee.org/document/9262078">"Have it Your Way: Generating Customized Log Datasets With a Model-Driven Simulation Testbed,"</a> in IEEE Transactions on Reliability, vol. 70, no. 1, pp. 402-415, March 2021, doi: 10.1109/TR.2020.3031317. [<a href="https://www.skopik.at/ait/2020_trel.pdf">PDF</a>]</p>
solar_home_system_data_log: Initial release of data sets and script
<p>In this first release, this repository includes three sets of data (date/time, temperature, current, and voltage) of over 6 months of electricity consumption of three households in an off-grid area in the state of Jharkhand, India. The goal of this data set is to be made open so that the community working in off-grid electricity access can get a sense of electricity consumption patterns and apply various analytical and visualization techniques.</p>
Data for: Temporal variations in female moose responses to roads and logging in the absence of wolves
<p>Animal movements, needed to acquire food resources, avoid predation risk, and find breeding partners, are influenced by annual and circadian cycles. Decisions related to movement reflect a quest to maximize benefits while limiting costs, especially in heterogeneous landscapes. Predation by wolves (<em>Canis lupus</em>) has been identified as the major driver of moose (<em>Alces alces</em>) habitat selection patterns, and linear features have been shown to increase wolf efficiency to travel, hunt and kill prey. However, few studies have described moose behavioral response to roads and logging in Canada in the absence of wolves. We thus characterized temporal changes (i.e., day phases and biological periods) in eastern moose (<em>Alces alces americana</em>) habitat selection and space use patterns near a road network in a wolf-free area located south of the St. Lawrence River (eastern Canada). We used telemetry data collected on 18 females between 2017 and 2019 to build resource selection functions and mixed linear regressions to explain variations in habitat selection patterns, home-range size and movement rates. Female moose selected forest stands providing forage when movement was not impeded by snow cover (i.e., spring/green-up, summer/rearing, fall/rut) and stands offering protection against incidental predation during calving. In winter, home-range size decreased with an increasing proportion of stands providing food and shelter against harsh weather, limiting the energetic costs associated with movement. Our results reaffirmed the year-round aversive effect of roads, even in the absence of wolves, but the magnitude of this avoidance differed between day phases, being lower during the "dusk-night-dawn" phase, perhaps due to a lower level of human activity on and near roads. Female moose behavior in our study area was similar to what was observed in landscapes where moose and wolves cohabit, suggesting that the risk associated with humans, perceived as another type of predator, and with incidental predators (coyote <em>Canis latrans</em>,<em> </em>black bear <em>Ursus americanus</em>), equates that of wolf predation in heavily managed landscapes.</p>
Processed Datasets - Imputation in Well Log Data: A Benchmark
<p>Imputation of well log data is a common task in the field. However a quick review of the literature reveals a lack of padronization when evaluating methods for the problem. The goal of the benchmark is to introduce a standard evaluation protocol to any imputation method for well log data. </p> <p>In the proposed benchmark, three public datasets are used:</p> <ul> <li><strong>Geolink:</strong> The Geolink Dataset is another public dataset of wells in the Norwegian offshore. The data is provided by the company of the same name, <a href="https://www.geolink-s2.com/" target="_blank" rel="noopener">GEOLINK</a> and follows the NOLD 2.0 license. <br>This dataset contains a total of 223 wells. It also has lithology labels for the wells with a total of 36 lithology classes. [<a href="https://drive.google.com/drive/folders/1EgDN57LDuvlZAwr5-eHWB5CTJ7K9HpDP" target="_blank" rel="noopener">download original</a>]</li> <li><strong>Taranaki Basin:</strong> The Taranaki Basin Dataset is a curated set of wells and a convenient option for experimentation especially due to it is ease of accessibility and use.<br>This collection, under the CDLA-Sharing-1.0 license, contains well logs extracted from the <a href="https://geodata.nzpam.govt.nz/" target="_blank" rel="noopener">New Zealand Petroleum & Minerals Online Exploration Database</a> and <a href="http://pet.gns.cri.nz/" target="_blank" rel="noopener">Petlab</a>.<br>There are a total of 407 wells, of which 289 are onshore and 118 are offshore exploration and production wells. [<a href="https://developer.ibm.com/exchanges/data/all/taranaki-basin-curated-well-logs/" target="_blank" rel="noopener">download original</a>]</li> <li><strong>Teapot Dome:</strong> The Teapot Dome dataset is provided by the Rocky Mountain Oilfield Testing Center (RMOTC) and the US Department of Energy.<br>It contains different types of data related to the Teapot Dome oil field, such as 2D and 3D seismic data, well logs, and GIS data. The data is licensed under the Creative Commons 4.0 license. <br>In total, the dataset has 1,179 wells with available logs. The number of available logs varies across wells. There are only 91 wells with the gamma ray, bulk density, and neutron porosity logs, while only three wells have the complete basic suite. [<a href="http://s3.amazonaws.com/open.source.geoscience/open_data/teapot/rmotc.tar" target="_blank" rel="noopener">direct download</a>]</li> </ul> <p>Here you can download all three datasets already preprocessed to be used with our implementation, found <a href="https://github.com/uai-ufmg/well-log-imputation" target="_blank" rel="noopener">here</a>.</p> <p> </p> <h3>File Description:</h3> <p>There are six files for each fold partition for each dataset.</p> <ul> <li><code><em>datasetname_fold_k_well_log_metadata_train.json </em></code>: JSON file with general information of the slices of <strong>training </strong>partition of the fold <strong>k</strong>. Contains total number of slices and the number of slices per well.<em> </em></li> <li><em><code>datasetname_fold_k_well_log_metadata_val.json</code> </em>: JSON file with general information of the slices of <strong>validation </strong>partition of the fold <strong>k</strong>. Contains total number of slices and the number of slices per well. </li> <li><em><code>datasetname_fold_k_well_log_slices_train.npy</code>: </em>.npy (numpy) file ready to be loaded with the slices for <strong>training </strong>of the fold <strong>k </strong>already processed. When loaded<em> </em>should have shape of<em> (total_slices, 256, number_of_logs)</em></li> <li><em><code>datasetname_fold_k_well_log_slices_val.npy</code> </em>: .npy (numpy) file ready to be loaded with the slices for <strong>validation </strong>of the fold <strong>k </strong>already processed.</li> <li><em><code>datasetname_fold_k_well_log_slices_meta_train.json</code> : </em>JSON file with the slices info for all slices in the <strong>training </strong>partition of the fold <strong>k</strong>. For each slice, 7 data points are provided, the last four are discarded (it would contain other information that was not used). The first three are in order the: origin well name, the starting position in that well, and the end position of the slice in that well.</li> <li><em><code>datasetname_fold_k_well_log_slices_meta_val.json</code> </em>: JSON file with the slices info for all slices in the <strong>validation </strong>partition of the fold <strong>k</strong>.</li> </ul>
Kyoushi Log Data Set
<p>This repository contains synthetic log data suitable for evaluation of intrusion detection systems. The logs were collected from a testbed that was built at the Austrian Institute of Technology (AIT) following the approaches by [1], [2], and [3]. Please refer to these papers for more detailed information on the dataset and cite them if the data is used for academic publications. Other than the related <a href="https://zenodo.org/record/4264796">AIT-LDSv1.1</a>, this dataset involves a more complex network structure, makes use of a different attack scenario, and collects log data from multiple hosts in the network. In brief, the testbed simulates a small enterprise network including mail server, file share, WordPress server, VPN, firewall, etc. Normal user behavior is simulated to generate background noise. After some days, two attack scenarios are launched against the network. Note that the <a href="https://zenodo.org/record/5789064">AIT-LDSv2.0</a> extends this dataset with additional attack cases and variations of attack parameters.</p> <p>The archives have the following structure. The <em>gather </em>directory contains the raw log data from each host in the network, as well as their system configurations. The <em>labels </em>directory contains the ground truth for those log files that are labeled. The <em>processing</em> directory contains configurations for the labeling procedure and the <em>rules </em>directory contains the labeling rules. Labeling of events that are related to the attacks is carried out with the <a href="https://github.com/ait-aecid/kyoushi-dataset">Kyoushi Labeling Framework</a>.</p> <p>Each dataset contains traces of a specific attack scenario:</p> <ul> <li>Scenario 1 (see <em>gather/attacker_0/logs/sm.log</em> for detailed attack log): <ul> <li>nmap scan</li> <li>WPScan</li> <li>dirb scan</li> <li>webshell upload through wpDiscuz exploit (CVE-2020-24186)</li> <li>privilege escalation</li> </ul> </li> <li>Scenario 2 (see <em>gather/attacker_0/logs/dnsteal.log</em> for detailed attack log): <ul> <li>DNSteal data exfiltration</li> </ul> </li> </ul> <p>The log data collected from the servers includes</p> <ul> <li>Apache access and error logs (labeled)</li> <li>audit logs (labeled)</li> <li>auth logs (labeled)</li> <li>VPN logs (labeled)</li> <li>DNS logs (labeled)</li> <li>syslog</li> <li>suricata logs</li> <li>exim logs</li> <li>horde logs</li> <li>mail logs</li> </ul> <p>Note that only log files from affected servers are labeled. Label files and the directories in which they are located have the same name as their corresponding log file in the <em>gather </em>directory. Labels are in JSON format and comprise the following attributes: line (number of line in corresponding log file), labels (list of labels assigned to that log line), rules (names of labeling rules matching that log line). Note that not all attack traces are labeled in all log files; please refer to the labeling rules in case that some labels are not clear.</p> <p>Acknowledgements: Partially funded by the FFG projects INDICAETING (868306) and DECEPT (873980), and the EU project GUARD (833456).</p> <p><strong>If you use the dataset, please cite the following publications:</strong></p> <p>[1] <a href="https://ieeexplore.ieee.org/document/9262078">M. Landauer, F. Skopik, M. Wurzenberger, W. Hotwagner and A. Rauber, "Have it Your Way: Generating Customized Log Datasets With a Model-Driven Simulation Testbed," in IEEE Transactions on Reliability, vol. 70, no. 1, pp. 402-415, March 2021, doi: 10.1109/TR.2020.3031317.</a></p> <p>[2] <a href="https://dl.acm.org/doi/10.1145/3510547.3517924">M. Landauer, M. Frank, F. Skopik, W. Hotwagner, M. Wurzenberger, and A. Rauber, "A Framework for Automatic Labeling of Log Datasets from Model-driven Testbeds for HIDS Evaluation". ACM Workshop on Secure and Trustworthy Cyber-Physical Systems (ACM SaT-CPS 2022), April 27, 2022, Baltimore, MD, USA. ACM.</a></p> <p>[3] <a href="https://repositum.tuwien.at/handle/20.500.12708/17804">M. Frank, "Quality improvement of labels for model-driven benchmark data generation for intrusion detection systems", Master's Thesis, Vienna University of Technology, 2021. </a></p>
Compositional discovery of architecture-aware and sound process models from event logs of multi-agent systems: experimental data.
<p>This repository contains the experimental data used for the evaluation of the compositional approach to the discovery of process models from event logs of multi-agent systems, where agents interact according to specific patterns of synchronous and asynchronous interactions.</p> <p>According to the experiment plan, there is the folder for each interface pattern containing:</p> <ol> <li>The reference model (Petri net encoded in PNML-file)</li> <li>The event log obtained by simulating the behavior of the reference model (XES-file)</li> <li>The model discovered directly from the generated event log (Petri net encoded in PNML-file)</li> <li>The model discovered by composing the agent model w.r.t. the interface pattern (Petri net encoded in PNML-file)</li> </ol>
AIT Log Data Set V2.0
<p><strong>AIT Log Data Sets</strong></p> <p>This repository contains synthetic log data suitable for evaluation of intrusion detection systems, federated learning, and alert aggregation. A detailed description of the dataset is available in [1]. The logs were collected from eight testbeds that were built at the Austrian Institute of Technology (AIT) following the approach by [2]. Please cite these papers if the data is used for academic publications.</p> <p>In brief, each of the datasets corresponds to a testbed representing a small enterprise network including mail server, file share, WordPress server, VPN, firewall, etc. Normal user behavior is simulated to generate background noise over a time span of 4-6 days. At some point, a sequence of attack steps is launched against the network. Log data is collected from all hosts and includes Apache access and error logs, authentication logs, DNS logs, VPN logs, audit logs, Suricata logs, network traffic packet captures, horde logs, exim logs, syslog, and system monitoring logs. Separate ground truth files are used to label events that are related to the attacks. Compared to the <a href="../record/4264796">AIT-LDSv1.1</a>, a more complex network and diverse user behavior is simulated, and logs are collected from all hosts in the network. If you are only interested in network traffic analysis, we also provide the <a href="../record/6610489">AIT-NDS</a> containing the labeled netflows of the testbed networks. We also provide the <a href="../records/8263181">AIT-ADS</a>, an alert data set derived by forensically applying open-source intrusion detection systems on the log data.</p> <p>The datasets in this repository have the following structure:</p> <ul> <li>The <em>gather </em>directory contains all logs collected from the testbed. Logs collected from each host are located in <em>gather/<host_name>/logs/</em>.</li> <li>The <em>labels </em>directory contains the ground truth of the dataset that indicates which events are related to attacks. The directory mirrors the structure of the gather directory so that each label files is located at the same path and has the same name as the corresponding log file. Each line in the label files references the log event corresponding to an attack by the line number counted from the beginning of the file ("line"), the labels assigned to the line that state the respective attack step ("labels"), and the labeling rules that assigned the labels ("rules"). An example is provided below.</li> <li>The <em>processing </em>directory contains the source code that was used to generate the labels.</li> <li>The <em>rules </em>directory contains the labeling rules.</li> <li>The <em>environment </em>directory contains the source code that was used to deploy the testbed and run the simulation using the <a href="https://github.com/ait-aecid/kyoushi-environment">Kyoushi Testbed Environment</a>.</li> <li>The <em>dataset.yml</em> file specifies the start and end time of the simulation.</li> </ul> <p>The following table summarizes relevant properties of the datasets:</p> <ul> <li>fox <ul> <li>Simulation time: 2022-01-15 00:00 - 2022-01-20 00:00</li> <li>Attack time: 2022-01-18 11:59 - 2022-01-18 13:15</li> <li>Scan volume: High</li> <li>Unpacked size: 26 GB</li> </ul> </li> <li>harrison <ul> <li>Simulation time: 2022-02-04 00:00 - 2022-02-09 00:00</li> <li>Attack time: 2022-02-08 07:07 - 2022-02-08 08:38</li> <li>Scan volume: High</li> <li>Unpacked size: 27 GB</li> </ul> </li> <li>russellmitchell <ul> <li>Simulation time: 2022-01-21 00:00 - 2022-01-25 00:00</li> <li>Attack time: 2022-01-24 03:01 - 2022-01-24 04:39</li> <li>Scan volume: Low</li> <li>Unpacked size: 14 GB</li> </ul> </li> <li>santos <ul> <li>Simulation time: 2022-01-14 00:00 - 2022-01-18 00:00</li> <li>Attack time: 2022-01-17 11:15 - 2022-01-17 11:59</li> <li>Scan volume: Low</li> <li>Unpacked size: 17 GB</li> </ul> </li> <li>shaw <ul> <li>Simulation time: 2022-01-25 00:00 - 2022-01-31 00:00</li> <li>Attack time: 2022-01-29 14:37 - 2022-01-29 15:21</li> <li>Scan volume: Low</li> <li>Data exfiltration is not visible in DNS logs</li> <li>Unpacked size: 27 GB</li> </ul> </li> <li>wardbeck <ul> <li>Simulation time: 2022-01-19 00:00 - 2022-01-24 00:00</li> <li>Attack time: 2022-01-23 12:10 - 2022-01-23 12:56</li> <li>Scan volume: Low</li> <li>Unpacked size: 26 GB</li> </ul> </li> <li>wheeler <ul> <li>Simulation time: 2022-01-26 00:00 - 2022-01-31 00:00</li> <li>Attack time: 2022-01-30 07:35 - 2022-01-30 17:53</li> <li>Scan volume: High</li> <li>No password cracking in attack chain</li> <li>Unpacked size: 30 GB</li> </ul> </li> <li>wilson <ul> <li>Simulation time: 2022-02-03 00:00 - 2022-02-09 00:00</li> <li>Attack time: 2022-02-07 10:57 - 2022-02-07 11:49</li> <li>Scan volume: High</li> <li>Unpacked size: 39 GB</li> </ul> </li> </ul> <p>The following attacks are launched in the network:</p> <ul> <li>Scans (nmap, WPScan, dirb)</li> <li>Webshell upload (CVE-2020-24186)</li> <li>Password cracking (John the Ripper)</li> <li>Privilege escalation</li> <li>Remote command execution</li> <li>Data exfiltration (DNSteal)</li> </ul> <p>Note that attack parameters and their execution orders vary in each dataset. Labeled log files are trimmed to the simulation time to ensure that their labels (which reference the related event by the line number in the file) are not misleading. Other log files, however, also contain log events generated before or after the simulation time and may therefore be affected by testbed setup or data collection. It is therefore recommended to only consider logs with timestamps within the simulation time for analysis.</p> <p>The structure of labels is explained using the audit logs from the intranet server in the russellmitchell data set as an example in the following. The first four labels in the <em>labels/intranet_server/logs/audit/audit.log</em> file are as follows:</p> <blockquote> <p>{"line": 1860, "labels": ["attacker_change_user", "escalate"], "rules": {"attacker_change_user": ["attacker.escalate.audit.su.login"], "escalate": ["attacker.escalate.audit.su.login"]}}</p> <p>{"line": 1861, "labels": ["attacker_change_user", "escalate"], "rules": {"attacker_change_user": ["attacker.escalate.audit.su.login"], "escalate": ["attacker.escalate.audit.su.login"]}}</p> <p>{"line": 1862, "labels": ["attacker_change_user", "escalate"], "rules": {"attacker_change_user": ["attacker.escalate.audit.su.login"], "escalate": ["attacker.escalate.audit.su.login"]}}</p> <p>{"line": 1863, "labels": ["attacker_change_user", "escalate"], "rules": {"attacker_change_user": ["attacker.escalate.audit.su.login"], "escalate": ["attacker.escalate.audit.su.login"]}}</p> </blockquote> <p>Each JSON object in this file assigns a label to one specific log line in the corresponding log file located at <em>gather/intranet_server/logs/audit/audit.log</em>. The field "line" in the JSON objects specify the line number of the respective event in the original log file, while the field "labels" comprise the corresponding labels. For example, the lines in the sample above provide the information that lines 1860-1863 in the <em>gather/intranet_server/logs/audit/audit.log</em> file are labeled with "attacker_change_user" and "escalate" corresponding to the attack step where the attacker receives escalated privileges. Inspecting these lines shows that they indeed correspond to the user authenticating as root:</p> <blockquote> <p>type=USER_AUTH msg=audit(1642999060.603:2226): pid=27950 uid=33 auid=4294967295 ses=4294967295 msg='op=PAM:authentication acct="jhall" exe="/bin/su" hostname=? addr=? terminal=/dev/pts/1 res=success'</p> <p>type=USER_ACCT msg=audit(1642999060.603:2227): pid=27950 uid=33 auid=4294967295 ses=4294967295 msg='op=PAM:accounting acct="jhall" exe="/bin/su" hostname=? addr=? terminal=/dev/pts/1 res=success'</p> <p>type=CRED_ACQ msg=audit(1642999060.615:2228): pid=27950 uid=33 auid=4294967295 ses=4294967295 msg='op=PAM:setcred acct="jhall" exe="/bin/su" hostname=? addr=? terminal=/dev/pts/1 res=success'</p> <p>type=USER_START msg=audit(1642999060.627:2229): pid=27950 uid=33 auid=4294967295 ses=4294967295 msg='op=PAM:session_open acct="jhall" exe="/bin/su" hostname=? addr=? terminal=/dev/pts/1 res=success'</p> </blockquote> <p>The same applies to all other labels for this log file and all other log files. There are no labels for logs generated by "normal" (i.e., non-attack) behavior; instead, all log events that have no corresponding JSON object in one of the files from the <em>labels </em>directory, such as the lines 1-1859 in the example above, can be considered to be labeled as "normal". This means that in order to figure out the labels for the log data it is necessary to store the line numbers when processing the original logs from the <em>gather </em>directory and see if these line numbers also appear in the corresponding file in the <em>labels </em>directory.</p> <p>Beside the attack labels, a general overview of the exact times when specific attack steps are launched are available in <em>gather/attacker_0/logs/attacks.log</em>. An enumeration of all hosts and their IP addresses is stated in processing/config/servers.yml. Moreover, configurations of each host are provided in <em>gather/<host_name>/configs/</em> and <em>gather/<host_name>/facts.json</em>.</p> <p>Version history:</p> <ul> <li><a href="https://doi.org/10.5281/zenodo.3723082">AIT-LDS-v1.x</a>: Four datasets, logs from single host, fine-granular audit logs, mail/CMS.</li> <li><a href="http://doi.org/10.5281/zenodo.5789064">AIT-LDS-v2.0</a>: Eight datasets, logs from all hosts, system logs and network traffic, mail/CMS/cloud/web.</li> </ul> <p>Acknowledgements: Partially funded by the FFG projects INDICAETING (868306) and DECEPT (873980), and the EU projects GUARD (833456) and PANDORA (SI2.835928).</p> <p><strong>If you use the dataset, please cite the following publications:</strong></p> <p>[1] M. Landauer, F. Skopik, M. Frank, W. Hotwagner, M. Wurzenberger, and A. Rauber. <a href="https://ieeexplore.ieee.org/abstract/document/9866880">"Maintainable Log Datasets for Evaluation of Intrusion Detection Systems"</a>. IEEE Transactions on Dependable and Secure Computing, vol. 20, no. 4, pp. 3466-3482, doi: 10.1109/TDSC.2022.3201582. [<a href="https://arxiv.org/pdf/2203.08580.pdf">PDF</a>]</p> <p>[2] M. Landauer, F. Skopik, M. Wurzenberger, W. Hotwagner and A. Rauber, <a href="https://ieeexplore.ieee.org/document/9262078">"Have it Your Way: Generating Customized Log Datasets With a Model-Driven Simulation Testbed,"</a> in IEEE Transactions on Reliability, vol. 70, no. 1, pp. 402-415, March 2021, doi: 10.1109/TR.2020.3031317. [<a href="https://www.skopik.at/ait/2020_trel.pdf">PDF</a>]</p>
Data from: eDNA metabarcoding of log hollow sediments and soils highlights the importance of substrate type, frequency of sampling and animal size, for vertebrate species detection
<p>Fauna monitoring often relies on visual monitoring techniques such as camera trappings, which have biases leading to underestimates of vertebrate species diversity. Environmental DNA (eDNA) has emerged as a new source of biodiversity data that may improve biomonitoring; however, eDNA based assessments of species richness remain relatively untested in terrestrial environments. We investigated the suitability of fallen log hollow sediment as a source of vertebrate eDNA, across two sites in south-western Australia - one with a Mediterranean climate and the other semi-arid. We compared two different approaches (camera trapping and eDNA metabarcoding) for monitoring of vertebrate species, and investigated the effect of other factors (frequency of species, timing of visits, frequency of sampling, body size) on vertebrate species detectability. Metabarcoding of hollow sediments resulted in the detection of higher species richness in comparison Hollow sediment detected higher species richness (29 taxa: six birds, three reptiles and 20 mammals) to metabarcoding of soil at the entrance of the hollow (13 taxa: three birds, two reptiles and eight mammals). We detected 31 taxa in total with eDNA metabarcoding and 47 with camera traps, with 14 taxa detected by both (12 mammals and two birds). By comparing camera trap data with eDNA read abundance, we were able to detect vertebrates through eDNA metabarcoding that had visited the area up to two months prior to sample collection. Larger animals were more likely to be detected, and so were vertebrates that were identified multiple times in the camera traps. These findings demonstrate the importance of substrate selection, frequency of sampling, and animal size, on eDNA based monitoring. Future eDNA experimental design should consider all these factors as they affect detection of target taxa. </p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.