Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
1,034
datasets available to search
ShareScore release 0.9.0
Dataset results
1,034 results for “Empirical”
PEASANT EVERYDAY LIFE IN THE RUSSIAN EMPIRE IN THE XIX CENTURY
<p>The article examines the daily life of peasants in the XIX century. It is shown that the peasantry made up about 9/10 of the productive population of Russia. There were noticeable changes in the social and legal status of the peasants during this period. The nature of their relations with the nobles and the state changed, the relations within the peasantry itself underwent changes. The economic differentiation of various categories of peasants intensified. Special attention in the article is paid to the peasant economy. Peasant farming increasingly lost its natural character and strengthened ties with the market. Important indicators of the penetration of bourgeois relations into agriculture were the purchase and lease of land, the use of wage labor. Wage labor was widely used in the southern steppe provinces, in the Volga region and the Baltic States.</p>
An Empirical Analysis of Rebalancing Methods for Security Issue Report Identification - Supplementary Material
<p>Bug report data set for the Ubuntu issue tracker on Launchpad and accompanying test python scripts.</p>
Dataset of Flow Velocity Prediction in Vegetated Alluvial Channels Comparing Empirical and State-of-the-art Hybrid Machine Learning Models
<p>We compiled 447 datasets from different sources and lab- and field-based measurements. These datasets included Einstein and Banks (1950), Fenzl (1962), Kouwen et al. (1969), Ree and Crow (1977), Murota (1984), Tsujimoto and Kitamura (1990), Tsujimoto (1991), Tsujimoto (1993), Shimizu (1994), Dunn et al. (1996), Ikeda and Kanazawa (1996), Meijer (1998), Jarvela (2002), Rowinski and Kubrak (2002), Stone and Shen (2002), Poggi et al. (2004), Carollo et al. (2005), and Murphy et al. (2007).</p>
Artifacts for the ISSTA 2023 Paper: An Empirical Study on the Effects of Obfuscation on Static Machine Learning-based Malicious JavaScript Detectors
<p>An Empirical Study on the Effects of Obfuscation on Static Machine Learning-Based Malicious JavaScript Detectors</p> <p>This repository contains the evaluation script and the corresponding data of the ISSTA'23 paper "An Empirical Study on the Effects of Obfuscation on Static Machine Learning-Based Malicious JavaScript Detectors".</p> <p>Abstract</p> <p>Machine learning is increasingly being applied to malicious JavaScript detection in response to the growing number of Web attacks and the attendant costly manual identification. In practice, to hide their malicious behaviors or protect intellectual copyrights, both malicious and benign scripts tend to obfuscate their own code before uploading. While obfuscation is beneficial, it also introduces some additional code features (e.g., dead code) into the code. When machine learning is employed to learn a malicious JavaScript detector, these additional features can affect the model to make it less effective. However, there is still a lack of clear understanding of how robust existing machine learning-based detectors are on different obfuscators.</p> <p>In this paper, we conduct the first empirical study to figure out how obfuscation affects machine learning detectors based on static features. Through the results, we observe several findings: 1) Obfuscation has a significant impact on the effectiveness of detectors, causing an increase both in false negative rate (FNR) and false positive rate (FPR), and the bias of obfuscation in the training set induces detectors to detect obfuscation rather than malicious behaviors. 2) The common measures such as improving the quality of the training set by adding relevant obfuscated samples and leveraging state-of-the-art deep learning models can not work well. 3) The root cause of obfuscation effects on these detectors is that feature spaces they use can only reflect shallow differences in code, not about the nature of benign and malicious, which can be easily affected by the differences brought by obfuscation. 4) Obfuscation has a similar effect on realistic detectors in VirusTotal, indicating<br>that this is a common real-world problem.</p> <p>Getting Started</p> <p>Requirements</p> <pre>install python3 version 3.9.12 pip3 install -r requirements.txt install nodejs install npm npm install escodegen npm install esprima</pre> <p>Step 1: Generating PDGs for JStap</p> <p><code>cd detectors/jstap/pdg_generation</code></p> <p><code>python generate_PDGs.py</code></p> <p>Step 2: Getting the results for RQ1: What Impact Does Obfuscation Have on Static Machine Learning Malicious JavaScript Detectors?</p> <p><code>cd RQ1/</code></p> <p>1. Detectors Performance on Obfuscated Samples.</p> <p>To train the models:</p> <p><code>python RQ1_1_train.py</code></p> <p>To get the results:</p> <p><code>python RQ1_1_test.py</code></p> <p>2. Different Machine Learning Algorithms.</p> <p>To train the models:</p> <p><code>python RQ1_2_train.py</code></p> <p>To get the results:</p> <p><code>python RQ1_2_test.py</code></p> <p>3. Biased Training Sets</p> <p>To train the models:</p> <p><code>python RQ1_3_train.py</code></p> <p>To get the results:</p> <p><code>python RQ1_3_test.py</code></p> <p>All the trained models will be stored in RQ1/models/.</p> <p>All the results will be stored in RQ2/results/.</p> <p>Step 3: Getting the results for RQ2: Are the Common Measures to Mitigate the Impact of Obfuscation Effective?</p> <p><code>cd RQ2/</code></p> <p>1. Training and Testing Detectors on Samples with Same Types of Obfuscation.</p> <p>To train the models:</p> <p><code>python RQ2_1_train.py</code></p> <p>To get the results:</p> <p><code>python RQ2_1_test.py</code></p> <p>2. Training and Testing Detectors on Samples with Different Types of Obfuscation.</p> <p>If you follow the steps, the models is already trained.</p> <p>To get the results:</p> <p><code>python RQ2_2_test.py</code></p> <p>3. BERT Variants.</p> <p>To get the results:</p> <p><code>python RQ2_3.py</code></p> <p>All the trained models will be stored in RQ2/models/.</p> <p>All the results will be stored in RQ2/results/.</p> <p> </p> <p>Step 4: Getting the results for RQ3: What Is the Root Cause of Obfuscation Affecting Static Machine Learning Malicious JavaScript Detectors?</p> <p>To get the results of vectors visualization, top ten features, and distances between vectors sets:</p> <p><code>cd RQ3</code></p> <p><code>python visulization.py</code></p> <p>The figures of vectors visualization will be stored in RQ3/results/.</p> <p>Other results will be shown in the console.</p> <p> </p> <p>Step 5: Getting the results for RQ4: How Does Obfuscation Affect Real-world Static Malicious JavaScript Detectors?</p> <p>To get the results, submit the sample under the folder samples/ to <a href="https://www.virustotal.com/gui/home/upload">VirusTotal</a> .</p> <p> </p> <p>Detailed Instructions</p> <p>detectors</p> <p>The detectors under the folder <code>detectors/</code> are the main projects to be evaluated in our paper, which are <strong>CUJO</strong>, <strong>ZOZZLE</strong>, <strong>JAST</strong>, and <strong>JSTAP</strong>.</p> <p>Detailed setup and usage instructions are described in <code>README.md</code> in the corresponding folder.</p> <p>samples</p> <p>The files under the folder <code>samples/</code> are the samples from a random tenth of our dataset used in our paper.</p> <p>Results can be obtained quickly using these samples. These results will not be exactly the same as in the paper, but they are similar.</p> <p>RQ1</p> <p>The code under folder <code>RQ1/</code> is to figure out how obfuscation affects these detectors.</p> <p><code>RQ1_1_train.py</code> is to train four detectors with unobfuscated samples.</p> <p><code>RQ1_1_test.py</code> tests these trained detectors with unobfuscated and obfuscated samples.</p> <p><code>RQ1_2_train.py</code> is to train the detector <strong>ZOZZLE</strong> that uses different machine learning algorithms.</p> <p><code>RQ1_2_test.py</code> tests these trained models with unobfuscated and obfuscated samples.</p> <p><code>RQ1_3_train.py</code> uses a training set with all unobfuscated benign samples and all obfuscated malicious samples, and a training set with all obfuscated benign samples and all unobfuscated malicious samples to train the detectors.</p> <p><code>RQ1_3_test.py</code> uses these detectors to detect unobfuscated benign samples, obfuscated benign samples, unobfuscated malicious samples, and obfuscated malicious samples, respectively.</p> <p>RQ2</p> <p>The code under folder <code>RQ2/</code> is to study the two measures to mitigate the impact of obfuscation effective or not.</p> <p><code>RQ2_1_train.py</code> uses obfuscated samples to train four detectors.</p> <p><code>RQ2_1_test.py</code> tests these detectors on the same type of obfuscated samples.</p> <p><code>RQ2_2_test.py</code> tests thest detectors on the different type of obfuscated samples.</p> <p><code>RQ2_3.py</code> uses the BERT variants to generate code representation of unobfuscated samples, trains the detector with these code representations, and tests the trained detectors with code representations of obfuscated samples.</p> <p>RQ3</p> <p>The code unser fodler <code>RQ3/</code> visualizes the vectors, extracts the ten most important features, and calculates the distance between different sets of vectors.</p> <p>RQ4</p> <p>There is no code related to RQ4 here because the actual operation of RQ4 is to submit the samples to <a href="https://www.virustotal.com/gui/home/upload">VirusTotal</a> .</p> <p> </p> <p>The whole dataset is available at <a href="https://drive.google.com/file/d/1a7pNUwzikiJyY9L7dIu53I6_MR0oDpgi/view?usp=sharing." target="_blank" rel="noopener">https://drive.google.com/file/d/1a7pNUwzikiJyY9L7dIu53I6_MR0oDpgi/view?usp=sharing.</a></p> <p> </p> <p>Cite this work</p> <pre>@inproceedings{staticanalysis, author = {Kunlun Ren, Qiang Weizhong, Yueming Wu, Yi Zhou, Deqing Zou, Hai Jin}, title = {An Empirical Study on the Effects of Obfuscation on Static Machine Learning-Based Malicious JavaScript Detectors}, booktitle = {Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA'23)}, year = {2023} }</pre> <p> </p>
Two Class Dependency Cycle Empirical Study
<p>This dataset includes 3 folders.</p> <ul> <li>Folder <strong>recurring-patterns</strong> contains successful untangling cases we manually inspect and classify into each pattern. In one instance folder, it contains src code before and after a commit (src/), cyclic files presented in dependency graph before and after a commit (cycle-dependency-change/), all modified files presented in dependency graph before and after a commit (all-modified-files-dependency-change), detailed code diff (diff.txt), and mined refactoring operations from RefactoringMiner 2.0.0 (refactor.json).</li> <li>Folder<strong> common-mistakes</strong> contains cases where dependency cycles are not correctly resolved. The file structure of each instance is the same as above.</li> <li>Folder <strong>design-characteristics </strong>contains a csv file, which list the internal dependency relations of cyclic files and their design context.</li> </ul>
Raw data for the article 'Revisiting power-law distributions in empirical outage data of power systems'
<p>Raw data for the article 'Revisiting power-law distributions in empirical outage data of power systems' (see the manuscript preprint https://arxiv.org/abs/2303.12714 for more details).</p>
Empirical Study Data for Test Program-Based Generative Fuzzing for Differential Testing of the Kotlin Compiler
<p>Empirical Study Dataset for MSc. Thesis titled "Test Program-Based Generative Fuzzing for Differential Testing of the Kotlin Compiler". The data contains automatically generated Kotlin files, the results of differentially testing the generated files, and aggregated data containing file information, and compilation features.</p>
Empirical models of "Magnetosheath Jets Over Solar Cycle 24: An Empirical Model"
<p>The empirical models of jet observations rates as a function of solar wind parameters used in the article "Magnetosheath Jets Over Solar Cycle 24: An Empirical Model" by L.Vuorinen, A. T. LaMoury, H. Hietala, and F. Koller. Please contact L. Vuorinen (lakavu@utu.fi) for additional information.</p>
Named Entity Recognition in the Regesta of Emperor Frederick III. of the Holy Roman Empire
<p>This dataset contains Named Entity Recognition and Entity Linking across 8883 Regesta of Emperor Frederick III. of the Holy Roman Empire. The Regesta are from the Regesta Imperii Edition project: http://www.regesta-imperii.de/en/the-project.html.</p> <p>There are 4554 different names normalized to 2849 distinct entities, identified by the index entry identifier of the yet to be fully published index of the Regesta of Frederick III. All in all, 17258 instances of names are identified.</p> <p><strong>Not all named entities are identified!</strong></p> <p>This dataset is intended as a step stone for recording already identified entities and training NER classifiers for the Regesta Imperii. Some named entities were deliberately omitted.</p> <p>Only those that could be identified with reasonable certainty via the index were included, for example: "Mgf. Albrecht von Brandenburg" is included, while "Albrecht", even when refering to the same entity as the former name, is not. Even of this subset, some entities may have been missed. A newer version may rectify that in the future.</p> <p>The JSONL format is modeled after the <a href="https://github.com/doccano/doccano">Doccano</a> format for entity linking.</p>
Empirical Exploration of Open-Source Issues for Predicting Privacy Compliance
<p>The dataset for the above publication RQ1</p>
Do Agile Scaling Approaches Make A Difference? An Empirical Comparison of Team Effectiveness Across Popular Scaling Approaches
<p>This bundle contains supplementary materials for an upcoming academic publication <em>Do Agile Scaling Approaches Make A Difference? An Empirical Comparison of Team Effectiveness Across Popular Scaling Approaches?</em>, by Christiaan Verwijs and Daniel Russo. Included in the bundle are the dataset and SPSS syntaxes. This replication package is made available by C. Verwijs under a "Creative Commons Attribution Non-Commercial Share-Alike 4.0 International"-license (CC-BY-NC-SA 4.0).</p> <p><strong>About the dataset</strong></p> <p>The dataset (SPSS) contains anonymized response data from 15,078 team members aggregated into 4,013 Agile teams that participated from <a href="https://scrumteamsurvey.org">scrumteamsurvey.org</a>. Stakeholder evaluations of 1,841 stakeholders were also collected for 529 of those teams. Data was gathered between September 2021, and September 2023. We cleaned the individual response data from careless responses and removed all data that could potentially identify teams, individuals, or their parent organizations. Because we wanted to analyze our measures at the team level, we calculated a team-level mean for each item in the survey. Such aggregation is only justified when at least 10% of the variance exists at the team level (Hair, 2019), which was the case (ICC = 35-50%). No data was missing at the team level.</p> <p>Question labels and option labels are provided separately in Questions.csv. To conform to the privacy statement of <a href="https://scrumteamsurvey.org">scrumteamsurvey.org</a>, the bundle does not include response data from before the team-level aggregation.</p> <p><strong>About the SPSS syntaxes</strong></p> <p>The bundle includes the syntaxes we used to prepare the dataset from the raw import, as well as the syntax we used to generate descriptives. This is mostly there for other researchers to verify our procedure.</p>
Fig. 17 in Empirical mode decomposition applied to acoustic detection of a cicadid pest
Fig. 17. The number of times each kernel returned the best result in the experiment performed for the BWC, RFC, OC and VDS criteria.
Fig. 10 in Empirical mode decomposition applied to acoustic detection of a cicadid pest
Fig. 10. Representation in PP for all analyzed series of the sets of vectors formed by the SD, SSC, TTC and EDT criteria.
Fig. 13 in Empirical mode decomposition applied to acoustic detection of a cicadid pest
Fig. 13. Box plots representing the accuracy of the vectors given by the SD, SSC, TTC and EDT criteria. The cross symbol represents the arithmetic mean.
Fig. 18 in Empirical mode decomposition applied to acoustic detection of a cicadid pest
Fig. 18. CM for the vectors given by the SD, SSC, TTC and EDT criteria. The parameters of the SVM were considered according to Table 5.
Fig. 7 in Empirical mode decomposition applied to acoustic detection of a cicadid pest
Fig. 7. Example of IMFs and the residue obtained by EMD with SD criterion applied to an audio signal of one second.
Fig. 6 in Empirical mode decomposition applied to acoustic detection of a cicadid pest
Fig. 6. SVM structure used in the proposed approach. Layer A is the input layer, with m passive elements; layer B is the hidden layer, with n(Xm) active sc elements; and layer C is the output layer, with one active linear element. In layer B, φ 2 = φn(Xm) 1, φ 1 = φn(Xm), w 2 = wn(Xm) 1 and w 1 = wn(Xm).
Fig. 15 in Empirical mode decomposition applied to acoustic detection of a cicadid pest
Fig. 15. Accuracy as a function of the number of training samples for the best series using minimum features and best overall series, respectively, both provided by the vectors generated by EMD with the EDT criterion.
Fig. 11 in Empirical mode decomposition applied to acoustic detection of a cicadid pest
Fig. 11. Representation in PP for all analyzed series of the sets of vectors formed by the BWC, RFC, OC and VDS criteria.
Fig. 5 in Empirical mode decomposition applied to acoustic detection of a cicadid pest
Fig. 5. Representation of the sets P and Q in the Cartesian plane and in PP, both with two classes. (a) The set P of vectors is linearly separable; (b) The set Q is not linearly separable. (c) PP representing the sets P and Q through PFE, suggesting a simple and a robust classifier for P and Q, respectively. The example is only illustrative and not related to the sets analyzed in this paper.
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.