Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
18
datasets available to search
ShareScore release 0.9.0
Dataset results
18 results for “malicious”
MaDroid: A Maliciousness-aware Multifeatured Dataset for Detecting Android Malware
<p>MaDroid is a maliciousness-aware multifeatured dataset of system calls focused on APK anomaly detection. The dataset includes 50,429 well-marked normal and abnormal system call sequences, with 24,789 and 25,640 sets of normal and abnormal sequences, respectively, for a total of 1.1 billion system call feature information. Each APK is labeled with the latest VT checksum information, and the sequence data includes 81 groups of system calls, system call parameters, and return values. The size of the whole dataset is 457 GB (19 GB after compression), including 236 GB of malicious system call sequence data. The APKs from which the system call feature sequences are derived cover mobile apps of different types released at different times in the past 14 years (2010-2023), covering 10 mainstream app markets, including Google Play, PlayDrone, Anzhi, etc. The APKs are also used as the source of the system call feature sequences, and the system call sequence data is used as the source of the APKs. anzhi, etc. We store the source code and dataset in two open platforms, GitHub and Zenodo, respectively.</p><h2>DataSet</h2><p>Release address: <a href="http://doi.org/10.5281/zenodo.7997398">http://doi.org/10.5281/zenodo.7997398</a></p><ul><li>The dataset consists of two classifications, Normal and Malware, with a total of 21 zip files. The installation files of each sequence come from 10 application markets such as Google Play, PlayDrone, Anzhi, etc. The Malware classification contains information about the running system call sequences of some APKs in the Drebin dataset.</li><li>RF, MLP and GBDT models were used to establish benchmarks for the dataset, the use of the models can be found in the source code.</li><li>The file `merge_all_csv_count_online_check_replenish.csv` is the dataset APK information. We provide APK name (SHA256 name for APK only), classification, APK capacity, number of sequences, log capacity, CVT, OVT value, check time, etc.</li></ul><h2>Source Code</h2><p>Release address: <a href="https://github.com/HNUSystemsLab/MaDroid">https://github.com/HNUSystemsLab/MaDroid</a></p><ul><li>The released source code contains two folders, `Source_Code` and `ml_metadata`. Where `Source_Code` is the automated framework for data collection, the tool chain and some notes on the structure of the source files. `ml_metadata` contains the metadata used for machine learning, the partitioned data on which the article builds its benchmark.</li><li>The automation framework is described in detail in the `Readme.md` document in the `Source_Code` directory. It consists of four main parts: environment requirements, program structure, quick start (working steps), model training and evaluation (including training and evaluation). It describes in detail the preparation of the environment, the data import method, the functional description of each file in the source code directory, the working principle of model training and evaluation, and other related contents.</li></ul><h2>Tips: </h2><ul><li>MAS is another name of MaDroid, the content shown here is the final version of "Readme.md".</li><li>A Large-scale Multi-feature Dataset for Anomaly Detection of Mobile Applications, which is the name of the document during our experiment.</li></ul>
Network Digital Twin-Generated Dataset for Machine Learning-based Detection of Benign and Malicious Heavy Hitter Flows
<h3>Overview</h3> <p>This record provides a dataset created as part of the study presented in the following publication and is made <strong>publicly available for research purposes</strong>. The associated article provides a comprehensive description of the dataset, its structure, and the methodology used in its creation. If you use this dataset, please <strong>cite the following article </strong>published in the journal <strong>IEEE Communications Magazine</strong>:</p> <blockquote> <p><strong>A. Karamchandani, J. Nunez, L. de-la-Cal, Y. Moreno, A. Mozo, and A. Pastor, “On the Applicability of Network Digital Twins in Generating Synthetic Data for Heavy Hitter Discrimination,” IEEE Communications Magazine, pp. 2–8, 2025, DOI: 10.1109/MCOM.003.2400648.</strong></p> </blockquote> <p>More specifically, the record contains several synthetic datasets generated to differentiate between benign and malicious heavy hitter flows within a realistic virtualized network environment. Heavy Hitter flows, which include high-volume data transfers, can significantly impact network performance, leading to congestion and degraded quality of service. Distinguishing legitimate heavy hitter activity from malicious Distributed Denial-of-Service traffic is critical for network management and security, yet existing datasets lack the granularity needed for training machine learning models to effectively make this distinction.</p> <p>To address this, a Network Digital Twin (NDT) approach was utilized to emulate realistic network conditions and traffic patterns, enabling automated generation of labeled data for both benign and malicious HH flows alongside regular traffic.</p> <h3>Feature Set:</h3> <p>The feature set includes the following flow statistics commonly used in the literature on network traffic classification:</p> <ul> <li>The protocol used for the connection, identifying whether it is TCP, UDP, ICMP, or OSPF.</li> <li>The time (relative to the connection start) of the most recent packet sent from source to destination at the time of each snapshot.</li> <li>The time (relative to the connection start) of the most recent packet sent from destination to source at the time of each snapshot.</li> <li>The cumulative count of data packets sent from source to destination at the time of each snapshot.</li> <li>The cumulative count of data packets sent from destination to source at the time of each snapshot.</li> <li>The cumulative bytes sent from source to destination at the time of each snapshot.</li> <li>The cumulative bytes sent from destination to source at the time of each snapshot.</li> <li>The time difference between the first packet sent from source to destination and the first packet sent from destination to source.</li> </ul> <h3>Dataset Variations:</h3> <p>To accommodate diverse research needs and scenarios, the dataset is provided in the following variations:</p> <ol> <li> <p><strong><code>All at Once</code></strong>:</p> <ol> <li>Contains a synthetic dataset where all traffic types, including benign, normal, and malicious DDoS heavy hitter (HH) flows, are combined into a single dataset.</li> <li>This version represents a holistic view of the traffic environment, simulating real-world scenarios where all traffic occurs simultaneously.</li> </ol> </li> <li> <p><strong><code>Balanced Traffic Generation</code></strong>:</p> <ol> <li>Represents a balanced traffic dataset with an equal proportion of benign, normal, and malicious DDoS traffic.</li> <li>Designed for scenarios where a balanced dataset is needed for fair training and evaluation of machine learning models.</li> </ol> </li> <li> <p><strong><code>DDoS at Intervals</code></strong>:</p> <ol> <li>Contains traffic data where malicious DDoS HH traffic occurs at specific time intervals, mimicking real-world attack patterns.</li> <li>Useful for studying the impact and detection of intermittent malicious activities.</li> </ol> </li> <li> <p><strong><code>Only Benign HH Traffic</code></strong>:</p> <ol> <li>Includes only benign HH traffic flows.</li> <li>Suitable for training and evaluating models to identify and differentiate benign heavy hitter traffic patterns.</li> </ol> </li> <li> <p><strong><code>Only DDoS Traffic</code></strong>:</p> <ol> <li>Contains only malicious DDoS HH traffic.</li> <li>Helps in isolating and analyzing attack characteristics for targeted threat detection.</li> </ol> </li> <li> <p><strong><code>Only Normal Traffic</code></strong>:</p> <ol> <li>Comprises only regular, non-HH traffic flows.</li> <li>Useful for understanding baseline network behavior in the absence of heavy hitters.</li> </ol> </li> <li> <p><strong><code>Unbalanced Traffic Generation</code></strong>:</p> <ol> <li>Features an unbalanced dataset with varying proportions of benign, normal, and malicious traffic.</li> <li>Simulates real-world scenarios where certain types of traffic dominate, providing insights into model performance in unbalanced conditions.</li> </ol> </li> </ol> <p>For each variation, the output of the different packet aggregators is provided separated in its respective folder.</p> <p>Each variation was generated using the NDT approach to demonstrate its flexibility and ensure the reproducibility of our study's experiments, while also contributing to future research on network traffic patterns and the detection and classification of heavy hitter traffic flows. The dataset is designed to support research in network security, machine learning model development, and applications of digital twin technology.</p>
A Comprehensive Dataset for Webpage Classification (Part 1: Adult & Malicious)
<p>This dataset, split across three parts due to Zenodo's size constraints, serves as a fundamental resource for enhancing webpage classification techniques. It encompasses 1,069,715 URLs, each annotated with labels to signify their categorization into Malicious, Benign, or Adult content, and further into 20 detailed sublabels for granular analysis. The dataset is designed to facilitate the evaluation and benchmarking of machine learning models, notably Stochastic Gradient Descent (SGD) and Support Vector Classifier (SVC), across a variety of tokenization methods and input types, including URLs, raw HTML, and parsed HTML content.</p> <p>The primary objective of assembling this dataset is to support research into effective webpage classification, thereby improving content prioritization and filtering in web crawling applications. It has been meticulously curated to provide a robust framework for studying the impact of different feature representation techniques on classification accuracy.</p> <p>The dataset is structured as JSON lines (jsonl) files, with each entry detailing a URL's label, sublabel, source, status code, and HTML content. This comprehensive dataset is divided into three parts due to size constraints on Zenodo, each targeting specific content categories to ensure ease of use and accessibility for researchers:</p> <ul> <li><strong>Part 1: Adult & Malicious</strong> encompasses URLs classified under Adult and Malicious categories, offering insights into content that requires stringent filtering.</li> <li><strong>Part 2: Benign 1</strong> and <strong>Part 3: Benign 2</strong> cover benign URLs, facilitating the study of safe web content and its classification nuances.</li> </ul> <p>We also created a .csv file without the HTML content so it is easier to work with URLs only, this .csv file contains the next columns `['uid', 'url', 'label', 'sublabel']`</p> <p>By providing this dataset, we aim to contribute significantly to the field of webpage classification, offering a valuable asset for researchers and practitioners looking to advance the state of web crawling technology and its applications.</p> <p>JSON line format for each line:</p> <p><code>{"url": "<URL>", "label": "<Label>", "sublabel": "<Sublabel>", "source": "<Source of this URL>", "status_code": <Status Code>, "html": "<HTML Textual Content>"}</code><br><br><strong>Other parts of this dataset:</strong></p> <ul> <li><a title="A Comprehensive Dataset for Webpage Classification (Part 2: Benign 1)" href="../records/10795435">A Comprehensive Dataset for Webpage Classification (Part 2: Benign 1)</a></li> <li><a title="A Comprehensive Dataset for Webpage Classification (Part 3: Benign 2)" href="../records/10795437">A Comprehensive Dataset for Webpage Classification (Part 3: Benign 2)</a></li> </ul> <p> </p> <p><strong>Citation</strong></p> <p>if you use this dataset, please cite us:</p> <p><code>Al-Maamari, M., Istaiti, M., Zerhoudi, S., Dinzinger, M., Granitzer, M. and Mitrovic, J., A COMPREHENSIVE DATASET FOR WEBPAGE CLASSIFICATION.</code></p> <p>https://ca-roll.github.io/downloads/A_Comprehensive_Dataset_for_Webpage_Classification.pdf</p> <p> </p> <p><code>Granitzer, M., Voigt, S., Fathima, N.A., Golasowski, M., Guetl, C., Hecking, T., Hendriksen, G., Hiemstra, D., Martinovič, J., Mitrović, J. and Mlakar, I., 2023. Impact and development of an Open Web Index for open web search. <em>Journal of the Association for Information Science and Technology.</em></code></p> <p>https://doi.org/10.1002/asi.24818</p>
MAD (MAlicious Traffic Dataset) in home and commercial environments - Home environment
<p>For the home environment we have: 01 Wifi Modem Router, 03 Smartphones, 01 server, 01 desktop, 01 Multifunction Printer, 01 network extender, 01 SmartTV, 01 Cable TV decoder and 01 firewall. This environment is a local network. The server has the Monitoring Environment and a network card, which provides connectivity and receives all network traffic for analysis.</p> <p>The results were obtained from Suricata and Telegraf collections from the TICK stack. All evidence was performed by queries via EveBox, which received data from Suricata, Grafana or graphics with information extracted from the InfluxDB (Grafana) and PostgreSQL (EveBox) databases.</p> <p>events.csv.gz - Suricata / Evebox collections</p> <p>net.csv.gz - Telegraf collections from the TICK stack</p> <p>netstat.csv.gz - Telegraf collections from the TICK stack</p> <p>For correlation purposes, use the events.csv.gz file as a basis. The key to correlation is the 'timestamp' column events.csv.gz with the 'time' column in the net.csv.gz and netstat.csv.gz files.</p> <p>The interval between collections, non-consecutive, was from 2018-09-15 to 2019-02-04</p>
MAD (MAlicious Traffic Dataset) in home and commercial environments - Internal environment
<p>In this environment we have: 01 Wifi Router, 01 Smartphone, 01 server and 01 desktop with virtual machines. This environment, called Internal, is a local network. One of the servers has the Security and Performance Monitoring Environment installed. In addition, 05 virtual machines were instantiated via QEMU on the same network. In this server, a network card provides connectivity to the environment and the other network card receives all network traffic for analysis by the Monitoring Environment. Getting traffic to Suricata is done by Ettercap. The desktop has two virtual machines instantiated via Oracle VirtualBox, on the same network and acts on the network as a client as well.</p> <p>The results were obtained from Suricata and Telegraf collections from the TICK stack. All evidence was performed by queries via EveBox, which received data from Suricata, Grafana or graphics with information extracted from the InfluxDB (Grafana) and PostgreSQL (EveBox) databases.</p> <p>events.csv.gz - Suricata / Evebox collections</p> <p>net.csv.gz - Telegraf collections from the TICK stack</p> <p>netstat.csv.gz - Telegraf collections from the TICK stack</p> <p>For correlation purposes, use the events.csv.gz file as a basis. The key to correlation is the 'timestamp' column events.csv.gz with the 'time' column in the net.csv.gz and netstat.csv.gz files.</p> <p>The interval between collections, non-consecutive, was from 2018-06-06 to 2019-01-31</p> <p> </p>
MAD (MAlicious Traffic Dataset) in home and commercial environments - Internet environment
<p>We have for the Internet environment: 01 Switch, 01 IP camera, 01 server for monitoring, 01 server for honeypot and no firewall. This environment is directly connected to the Internet. We installed a server, functioning as a Monitoring Environment. The network traffic was obtained via Port Mirroring on the switch to the Monitoring Environment server.</p> <p>The results were obtained from Suricata and Telegraf collections from the TICK stack. All evidence was performed by queries via EveBox, which received data from Suricata, Grafana or graphics with information extracted from the InfluxDB (Grafana) and PostgreSQL (EveBox) databases.</p> <p>events.csv.gz - Suricata / Evebox collections</p> <p>net.csv.gz - Telegraf collections from the TICK stack</p> <p>netstat.csv.gz - Telegraf collections from the TICK stack</p> <p>For correlation purposes, use the events.csv.gz file as a basis. The key to correlation is the 'timestamp' column events.csv.gz with the 'time' column in the net.csv.gz and netstat.csv.gz files.</p> <p>The interval between collections, non-consecutive, was from 2018-08-28 to 2019-11-14</p>
MAD (MAlicious Traffic Dataset) in home and commercial environments - Environment with scalability
<p>We have used the Internet environment: 01 Switch, 01 IP camera, 01 server for monitoring, 01 server for honeypot and no firewall. This environment is directly connected to the Internet. We installed a server, functioning as a Monitoring Environment. The network traffic was obtained via Port Mirroring on the switch to the Monitoring Environment server.</p> <p>We added 08 virtual machines and performed the following test with a denial of service DoS attack:</p> <p>01 virtual machine from 04:00 pm to 23:55 pm on 2019-12-04 with an interval every 01 hour;<br> 02 virtual machines from 23:55 am on 2019-12-04 to 08:50 am on 2019-12-05 with an interval every 01 hour;<br> 04 virtual machines as of 08:55 am on 2019-12-05 to 05:25 pm on 2019-12-06 with an interval every 5 minutes;<br> 08 virtual machines from 05:30 pm on 2019-12-06 to 23:59 on 2019-12-06 with an interval every 5 minutes;<br> End of tests with shutdown of virtual machines at 23:59 on 2019-12-06.</p> <p>The results were obtained from Suricata and Telegraf collections from the TICK stack. All evidence was performed by queries via EveBox, which received data from Suricata, Grafana or graphics with information extracted from the InfluxDB (Grafana) and PostgreSQL (EveBox) databases.</p> <p>events.csv.gz - Suricata / Evebox collections</p> <p>net.csv.gz - Telegraf collections from the TICK stack</p> <p>netstat.csv.gz - Telegraf collections from the TICK stack</p> <p>For correlation purposes, use the events.csv.gz file as a basis. The key to correlation is the 'timestamp' column events.csv.gz with the 'time' column in the net.csv.gz and netstat.csv.gz files.</p> <p>The interval between collections, non-consecutive, was from 2019-12-04 to 2019-12-06</p>
CTU-SME-11: a labeled dataset with real benign and malicious network traffic mimicking a small medium-size enterprise environment
<p>As technology advances, the number and complexity of cyber-attacks increase, forcing defense techniques to be updated and improved. To help develop effective tools for detecting security threats it is essential to have reliable and representative security datasets. Many existing security datasets have limitations that make them unsuitable for research, including lack of labels, unbalanced traffic, and outdated threats.</p> <p>CTU-SME-11 is a labeled network dataset designed to address the limitations of previous datasets. The dataset was captured in a real network that mimics a small-medium enterprise setting. Raw network traffic (packets) was captured from 11 devices using tcpdump for a duration of 7 days, from 20th to 26th of February, 2023 in Prague, Czech Republic. The devices were chosen based on the enterprise setting and consists of IoT, desktop and mobile devices, both bare metal and virtualized. The devices were infected with malware or exposed to Internet attacks, and factory reset to restore benign behavior. </p> <p>The raw data was processed to generate network flows (Zeek logs) which were analyzed and labeled. The dataset contains two types of levels, a high level label and a descriptive label, which were put by experts. The former can take three values, benign, malicious or background. The latter contains detailed information about the specific behavior observed in the network flows. The dataset contains 99 million labeled network flows. The overall compressed size of the dataset is 80GB and the uncompressed size is 170GB.</p>
Malicious Forensic Texts Corpus
<p>The Malicious Forensic Texts (MFT) corpus is a corpus of authentic malicious forensic texts that has been compiled in order to study their register variation.</p>
Seen Time Dataset of Malicious DNS IPs (October 2021)
<p><span>This collection consists of two complementary datasets focused on the seen time records of IP addresses involved in DNS-related attacks targeting port 53, identified in October 2021. The datasets were constructed using open-source intelligence (OSINT) sources and enriched with historical data from AbuseIPDB.</span></p> <p><strong><span>Raw Data</span></strong></p> <p><span>Description: This dataset includes the full historical seen time records for all IP addresses linked to DNS-related attacks on port 53, as identified by OSINT in October 2021. The data, sourced from AbuseIPDB, encompasses 173 IP addresses before preprocessing, providing a comprehensive view of the observed malicious activities.</span></p> <p><strong><span>Preprocessed Data</span></strong></p> <p><span>Description: This dataset provides the preprocessed seen time records for IP addresses involved in DNS-related attacks on port 53, sourced from OSINT and enriched with data from AbuseIPDB. The preprocessed dataset narrows the focus to 111 IP addresses, each with at least 20 seen time records, offering a high-quality subset for detailed cybersecurity analysis and research into DNS attack trends.</span></p>
Datasets used in the paper "The Face of Deception: The Impact of AI-Generated Photos on Malicious Social Bots"
<p>Datasets used in the paper "The Face of Deception: The Impact of AI-Generated Photos on Malicious Social Bots"</p> <p>We changed the datasets' titles and omitted authors' names for the blind review process. After the review, we will upload it to GitHub in an unanonymised form.</p> <p>Check README.md for details.</p>
Towards Fine-granularity Malicious Component Detection for Android Apps
<p>These files are the experiment data of AMCDroid.</p> <p> (1) is the dataset used in the experiment (we do not use the VirusShare dataset in 2019 since we failed to download it);</p> <p> (2) is the malicious component detection result of the running example;</p> <p> (3) to (7) are the experiment data in RQ1 to RQ5;</p> <p> (8) is the malicious component detection result of Roaming Mantis; and</p> <p> (9) is the malicious component detection result in internal validity.</p> <p>We will open source the code of AMCDroid after our paper is accepted.</p>
Artifacts for the ISSTA 2023 Paper: An Empirical Study on the Effects of Obfuscation on Static Machine Learning-based Malicious JavaScript Detectors
<p>An Empirical Study on the Effects of Obfuscation on Static Machine Learning-Based Malicious JavaScript Detectors</p> <p>This repository contains the evaluation script and the corresponding data of the ISSTA'23 paper "An Empirical Study on the Effects of Obfuscation on Static Machine Learning-Based Malicious JavaScript Detectors".</p> <p>Abstract</p> <p>Machine learning is increasingly being applied to malicious JavaScript detection in response to the growing number of Web attacks and the attendant costly manual identification. In practice, to hide their malicious behaviors or protect intellectual copyrights, both malicious and benign scripts tend to obfuscate their own code before uploading. While obfuscation is beneficial, it also introduces some additional code features (e.g., dead code) into the code. When machine learning is employed to learn a malicious JavaScript detector, these additional features can affect the model to make it less effective. However, there is still a lack of clear understanding of how robust existing machine learning-based detectors are on different obfuscators.</p> <p>In this paper, we conduct the first empirical study to figure out how obfuscation affects machine learning detectors based on static features. Through the results, we observe several findings: 1) Obfuscation has a significant impact on the effectiveness of detectors, causing an increase both in false negative rate (FNR) and false positive rate (FPR), and the bias of obfuscation in the training set induces detectors to detect obfuscation rather than malicious behaviors. 2) The common measures such as improving the quality of the training set by adding relevant obfuscated samples and leveraging state-of-the-art deep learning models can not work well. 3) The root cause of obfuscation effects on these detectors is that feature spaces they use can only reflect shallow differences in code, not about the nature of benign and malicious, which can be easily affected by the differences brought by obfuscation. 4) Obfuscation has a similar effect on realistic detectors in VirusTotal, indicating<br>that this is a common real-world problem.</p> <p>Getting Started</p> <p>Requirements</p> <pre>install python3 version 3.9.12 pip3 install -r requirements.txt install nodejs install npm npm install escodegen npm install esprima</pre> <p>Step 1: Generating PDGs for JStap</p> <p><code>cd detectors/jstap/pdg_generation</code></p> <p><code>python generate_PDGs.py</code></p> <p>Step 2: Getting the results for RQ1: What Impact Does Obfuscation Have on Static Machine Learning Malicious JavaScript Detectors?</p> <p><code>cd RQ1/</code></p> <p>1. Detectors Performance on Obfuscated Samples.</p> <p>To train the models:</p> <p><code>python RQ1_1_train.py</code></p> <p>To get the results:</p> <p><code>python RQ1_1_test.py</code></p> <p>2. Different Machine Learning Algorithms.</p> <p>To train the models:</p> <p><code>python RQ1_2_train.py</code></p> <p>To get the results:</p> <p><code>python RQ1_2_test.py</code></p> <p>3. Biased Training Sets</p> <p>To train the models:</p> <p><code>python RQ1_3_train.py</code></p> <p>To get the results:</p> <p><code>python RQ1_3_test.py</code></p> <p>All the trained models will be stored in RQ1/models/.</p> <p>All the results will be stored in RQ2/results/.</p> <p>Step 3: Getting the results for RQ2: Are the Common Measures to Mitigate the Impact of Obfuscation Effective?</p> <p><code>cd RQ2/</code></p> <p>1. Training and Testing Detectors on Samples with Same Types of Obfuscation.</p> <p>To train the models:</p> <p><code>python RQ2_1_train.py</code></p> <p>To get the results:</p> <p><code>python RQ2_1_test.py</code></p> <p>2. Training and Testing Detectors on Samples with Different Types of Obfuscation.</p> <p>If you follow the steps, the models is already trained.</p> <p>To get the results:</p> <p><code>python RQ2_2_test.py</code></p> <p>3. BERT Variants.</p> <p>To get the results:</p> <p><code>python RQ2_3.py</code></p> <p>All the trained models will be stored in RQ2/models/.</p> <p>All the results will be stored in RQ2/results/.</p> <p> </p> <p>Step 4: Getting the results for RQ3: What Is the Root Cause of Obfuscation Affecting Static Machine Learning Malicious JavaScript Detectors?</p> <p>To get the results of vectors visualization, top ten features, and distances between vectors sets:</p> <p><code>cd RQ3</code></p> <p><code>python visulization.py</code></p> <p>The figures of vectors visualization will be stored in RQ3/results/.</p> <p>Other results will be shown in the console.</p> <p> </p> <p>Step 5: Getting the results for RQ4: How Does Obfuscation Affect Real-world Static Malicious JavaScript Detectors?</p> <p>To get the results, submit the sample under the folder samples/ to <a href="https://www.virustotal.com/gui/home/upload">VirusTotal</a> .</p> <p> </p> <p>Detailed Instructions</p> <p>detectors</p> <p>The detectors under the folder <code>detectors/</code> are the main projects to be evaluated in our paper, which are <strong>CUJO</strong>, <strong>ZOZZLE</strong>, <strong>JAST</strong>, and <strong>JSTAP</strong>.</p> <p>Detailed setup and usage instructions are described in <code>README.md</code> in the corresponding folder.</p> <p>samples</p> <p>The files under the folder <code>samples/</code> are the samples from a random tenth of our dataset used in our paper.</p> <p>Results can be obtained quickly using these samples. These results will not be exactly the same as in the paper, but they are similar.</p> <p>RQ1</p> <p>The code under folder <code>RQ1/</code> is to figure out how obfuscation affects these detectors.</p> <p><code>RQ1_1_train.py</code> is to train four detectors with unobfuscated samples.</p> <p><code>RQ1_1_test.py</code> tests these trained detectors with unobfuscated and obfuscated samples.</p> <p><code>RQ1_2_train.py</code> is to train the detector <strong>ZOZZLE</strong> that uses different machine learning algorithms.</p> <p><code>RQ1_2_test.py</code> tests these trained models with unobfuscated and obfuscated samples.</p> <p><code>RQ1_3_train.py</code> uses a training set with all unobfuscated benign samples and all obfuscated malicious samples, and a training set with all obfuscated benign samples and all unobfuscated malicious samples to train the detectors.</p> <p><code>RQ1_3_test.py</code> uses these detectors to detect unobfuscated benign samples, obfuscated benign samples, unobfuscated malicious samples, and obfuscated malicious samples, respectively.</p> <p>RQ2</p> <p>The code under folder <code>RQ2/</code> is to study the two measures to mitigate the impact of obfuscation effective or not.</p> <p><code>RQ2_1_train.py</code> uses obfuscated samples to train four detectors.</p> <p><code>RQ2_1_test.py</code> tests these detectors on the same type of obfuscated samples.</p> <p><code>RQ2_2_test.py</code> tests thest detectors on the different type of obfuscated samples.</p> <p><code>RQ2_3.py</code> uses the BERT variants to generate code representation of unobfuscated samples, trains the detector with these code representations, and tests the trained detectors with code representations of obfuscated samples.</p> <p>RQ3</p> <p>The code unser fodler <code>RQ3/</code> visualizes the vectors, extracts the ten most important features, and calculates the distance between different sets of vectors.</p> <p>RQ4</p> <p>There is no code related to RQ4 here because the actual operation of RQ4 is to submit the samples to <a href="https://www.virustotal.com/gui/home/upload">VirusTotal</a> .</p> <p> </p> <p>The whole dataset is available at <a href="https://drive.google.com/file/d/1a7pNUwzikiJyY9L7dIu53I6_MR0oDpgi/view?usp=sharing." target="_blank" rel="noopener">https://drive.google.com/file/d/1a7pNUwzikiJyY9L7dIu53I6_MR0oDpgi/view?usp=sharing.</a></p> <p> </p> <p>Cite this work</p> <pre>@inproceedings{staticanalysis, author = {Kunlun Ren, Qiang Weizhong, Yueming Wu, Yi Zhou, Deqing Zou, Hai Jin}, title = {An Empirical Study on the Effects of Obfuscation on Static Machine Learning-Based Malicious JavaScript Detectors}, booktitle = {Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA'23)}, year = {2023} }</pre> <p> </p>
IoT Emulated Dataset for ICMP/Ping Normal and Malicious Traffic
<p>These datasets are related to Intrusion Detection System, Computer Network Traffic and IoT.</p> <p>These datasets are generated for the purpose of differentiating <strong>ICMP/Ping normal and malicious traffic </strong>that are generated from an embedded device (IoT). The differentiation analysis is done using machine learning.</p> <p>There are three types of files that depend on each module of our research framework. The data generation sequence is as follows:</p> <p>The <strong>pcap files </strong>(network traffic) are generated first, the device used to generate the data is an ESP-01s. Afterwards, the pcap files are transformed into <strong>log files </strong>using the Zeek tool, the log files are then extracted and placed into <strong>CSV files</strong>.</p> <p>The CSV files are labeled and ready for the Machine Learning process.</p> <p> </p> <p>The publication reference for this work is here : https://doi.org/10.1109/ACCESS.2023.3327061</p> <p>The code link: <a href="../badge/latestdoi/619245496">https://zenodo.org/badge/latestdoi/619245496</a></p> <p><strong>This version of the release (0.2.0) is for ping flood with spoofed IPs, however, the previous version (</strong>0.1.0)<strong> is for static IP</strong></p> <p> </p> <p> </p> <p> </p>
In vitro cross-talk between metastasis-competent circulating tumor cells and platelets in colon cancer: a malicious association during a harsh journey in the blood.
GEO Series GSE230557. Homo sapiens. 20 samples. Type: Expression profiling by high throughput sequencing.
Malicious Large Language Models Detection using Metadata Information
<p># Introduction<br># This is the replication package for the paper "Malicious Large Language Models Detection using Metadata Information".</p> <p><br>## Task Definition<br>Given the information of LLM, the task is to identify whether it is a malicious LLM that may attack software systems. We treat the task as binary classification (0/1), where 1 stands for malicious LLMs and 0 for malicious-free LLMs.</p> <p><br>## The static of dataset<br>### Data Format<br>Before preprocessing dataset, each line in the uncompressed file represents multiple metadata of one large language model (LLM). One row is illustrated below.<br> - **idx:** the index of example<br> - **repo_id:** the id of LLM (e.g., microsoft/codebert-base)<br> - **tags:** the tags of LLM<br> - **pipeline_tag:** the pipeline_tag of LLM<br> - **downloads:** the number of downloads<br> - **created_time:** the created time of LLM<br> - **modelCard:** the text content of LLM<br> - **num_discussion:** the number of discussions<br> - **discussion:** the discussions of LLM<br> - **para_size:** the size of LLM<br> - **tensor_type:** the type of LLM<br> - **num_commit:** the number of commits<br> - **commit:** the commit of LLM</p> <p>After preprocessing dataset, you can obtain three .csv files, i.e. train.csv, valid.csv, test.csv<br> - **idx:** the index of example.<br> - **repo_id:** the id of LLM (e.g., microsoft/codebert-base).<br> - **tags:** the tags of LLM.<br> - **pipeline_tags:** the pipeline_tag of LLM.<br> - **created_time:** the created time of LLM.<br> - **model_size:** the size of LLM.<br> - **Tensor_type:** the type of LLM.<br> - **is_model_card:** Whether to include model card. If the model has model card, the value is 1, otherwise, the value is 0.<br> - **malicious_model_card:** Whether to include keywords describing the malicious model in model card. If the model card has malicious keywords, the value is 1, otherwise, the value is 0<br> - **repository_link:** Whether to include repository link: GitHub link, Arxiv link, homepage link, bugs link and issues link in model card. If the model card has link, the value is 1, otherwise, the value is 0.<br> - **dataset_info:** Whether to include the adopted dataset information in model card. If the model card has dataset information, the value is 1, otherwise, the value is 0.<br> - **metrics_info:** Whether to include the evaluated metrics information in model card. If the model card has evaluation metrics information, the value is 1, otherwise, the value is 0.<br> ?- **script_info:** Whether to include script information. If the model has script information, the value is 1, otherwise, the value is 0.<br> - **config_content:** The content of the configuration script file. This value is string type.<br> - **stakeholder_name:** The name of authors, contributors, and maintainers. This value is string type.<br> - **number_discussion:** The number of discussion. <br> - **num_pr:** The number of pull request.<br> - **malicious_discussion:** Whether the discussion contains malicious behavior keywords. If the discussion has malicious behavior keywords, the value is 1, otherwise, the value is 0.<br> - **number_commit:** The number of commit.<br> - **malicious_commit:** Whether the title and message of commits contain malicious behavior keywords. If the commit has malicious behavior keywords, the value is 1, otherwise, the value is 0.<br> - **z_download:** The z-score of number of download.<br> - **z_like:** The z-score of number of likes.</p> <p><br>### Data Statistics<br>Data statistics of the dataset are shown in the below table:</p> <p>| #File Names | #Examples |<br>| ------------------- | :------------------------: |<br>| dataset_feature.csv | 578,502 (560,257/18,245) | <br>| train_imbalance.csv | 462,801 (448205/14596) |<br>| valid_imbalance.csv | 57,849 (56025/1824) |<br>| test_imbalance.csv | 57,852 (56027/1825) |<br>| train_balance.csv | 29192 (14596/14596) |<br>| valid_balance.csv | 3648 (1824/1824) |<br>| test_balance.csv | 3650 (1825/1825) |<br>| train_imbalance_50.csv | 289252 (280129/9123) |<br>| train_imbalance_60.csv | 347101 (336154/10947) |<br>| train_imbalance_70.csv | 404952 (392180/12772) |</p> <p>29646 models contain github link<br>all_dataset.csv 596383<br>all_dataset_information.csv 589140 (safe_dataset.csv(570549), unsafe_dataset.csv(18591))<br>safe_dataset_information.csv (559582), unsafe_dataset_information.csv (18212)</p> <p>Description Feature: 'malicious_model_card', 'repository_link', 'dataset_info', 'metrics_info', 'config_conteng'<br>Stakeholder Feature: 'stakeholder_name'<br>Event Feature: 'num_pr', 'number_commit', 'malicious_commit',<br>Context Feature: 'z_download', 'z_like'</p> <p><br>## Pipeline-MPTMHunter<br>We also provide a pipeline that fine-tunes [MPTMHunter](https://doi.org/10.5281/zenodo.12578531) on this task.</p> <p>### Experimental environment configuration<br>```bash<br>huggingface_hub 0.23.1<br>libxgboost 2.0.3<br>lightgbm 4.3.0<br>networkx 3.2.1<br>nltk 3.8.1<br>numpy 1.26.3<br>openssl 3.0.13<br>pandas 2.2.1<br>pillow 10.2.0<br>scikit-learn 1.4.2<br>scipy 1.13.0<br>torch 2.3.0+cu118<br>torchaudio 2.3.0+cu118<br>torchvision 0.18.0+cu118<br>tqdm 4.66.2<br>transformers 4.37.2<br>xgboost 2.0.3<br>```</p> <p>### Dataset Collection Script<br>```bash<br>python ./script/DataExtraction.ipynb<br>python ./script/dataset_spider.py<br>python ./script/config_crawl.py<br>```</p> <p>### Dataset Preprocess Script<br>```bash<br>python feature_generation.py --input_file='../dataset/dataset_information.csv' --output_file='../dataset/dataset_feature.csv'</p> <p>python feature_generation.py --input_file='../dataset/real_world_dataset_information_0701.csv' --output_file='../dataset/real_world_dataset_feature_0701.csv'<br>```</p> <p>### Model Training Script<br>```bash<br>python run_codet5_lstm.py --output_dir='../saved_models/codet5_lstm_imbalance_final' --model_type=codet5 --tokenizer_name='../models/codet5' --model_name_or_path='../models/codet5' --do_train --train_data_file='../dataset/train_imbalance_70.csv' --eval_data_file='../dataset/valid_imbalance_70.csv' --test_data_file='../dataset/test_imbalance.csv' --epoch=3 --block_size=510 --train_batch_size=64 --eval_batch_size=64 --learning_rate=2e-5 --max_grad_norm=1.0 --evaluate_during_training --seed=123456<br>```</p> <p>### Model Inference Script<br>```bash<br>python run_codet5_lstm.py --output_dir='../saved_models/codet5_lstm_imbalance' --model_type=codet5 --tokenizer_name='../models/codet5' --model_name_or_path='../models/codet5' --do_eval --do_test --train_data_file='../dataset/train_imbalance_70.csv' --eval_data_file='../dataset/valid_imbalance_70.csv' --test_data_file='../dataset/test_imbalance.csv' --epoch=3 --block_size=510 --train_batch_size=64 --eval_batch_size=64 --learning_rate=2e-5 --max_grad_norm=1.0 --evaluate_during_training --seed=123456<br>```</p> <p>### Evaluation Script<br>```bash<br>python ../evaluation/evaluation.py -a ../dataset/test_balance.csv -p ../saved_models/codebert_imbalance_all/predictions.txt<br>python ../evaluation/evaluation.py -a ../dataset/test_imbalance.csv -p ../dataset/predictions.txt<br>```</p> <p>## Result<br>The results on the test set are shown as below (We use the OpenTextClassification as the baseline):</p> <p>| Methods | ACC | Precision | Recall | F1-Score |<br>| Random Forest | 97.18% | 83.80% | 13.04% | 22.57% |<br>| LR | 96.80% | 43.46% | 4.55% | 8.23% |<br>| LightGBM | 97.62% | 95.18% | 25.97% | 40.81% |<br>| TextRNN | 96.86% | 73.33% | 0.60% | 1.20% |<br>| TextCNN | 98.89% | 95.61% | 68.00% | 79.47% |<br>| TextRCNN | 98.87% | 95.36% | 67.62% | 79.13% |<br>| TextRNN_Att | 98.94% | 95.29% | 69.86% | 80.62% |<br>| MPTMHunter | ** 99.99% ** | ** 99.95% ** | ** 99.78% ** | ** 99.86% ** |</p>
Enhancing Malware Detection for Android Apps: Detecting Fine-granularity Malicious Components
<p>Please refer to the version v1.</p>
One Model to Rule Them All: Cross-Language Detection of Malicious Packages for NPM and PyPI - Dataset, Models, Malicious Packages List
<p>This artifact complements the paper "One Model to Rule Them All: Language-Independent Detection of Malicious Packages for PyPI and NPM" submitted at ESEC/FSE 2023.</p> <p>In particular:</p> <ul> <li>"Labelled_Dataset.csv " contains the data used for the features evaluation, models evaluation and models training</li> <li>The .pkl files are the best-performing classifiers obtained after the models evaluation</li> <li>"Malicious_Packages_Discovered.csv" contain the list of malicious packages discovered during our real-world experiment</li> </ul>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.