syslrn: Learning What to Monitor for Efficient Anomaly Detection [Dataset]

<p>This repository includes the dataset for the paper:</p> <p><em><a href="http://doi.org/10.1145/3517207.3526979">D. Sanvito, G. Siracusano, S. Santhanam, R. Gonzalez, R. Bifulco</a></em><br> <strong><em><a href="http://doi.org/10.1145/3517207.3526979">syslrn: Learning What to Monitor for Efficient Anomaly Detection </a></em></strong><br> <em><a href="http://doi.org/10.1145/3517207.3526979">ACM EuroMLSys 2022</a></em></p> <p>The dataset contains two directories at the root level:</p> <ul> <li><em><strong>raw_dataset</strong></em></li> <li><strong><em>processed_dataset</em></strong></li> </ul> <p>Each folder in the <strong><em>raw_dataset</em> </strong>directory contains the raw monitoring data used to generate the graph associated to a single experiment together with additional metadata files.<br> Each folder in the <strong><em>processed_dataset</em> </strong>directory contains the graph associated to a single experiment as a set of three CSV files: two for the graph edges (<em>pid_childof_pid_df.csv</em> and <em>pid_speakswith_pid_df.csv</em>) and one for the graph nodes (<em>proc_df.csv</em>).<br> We provide below a code snippet to parse a graph from <strong><em>processed_dataset</em> </strong>directory.</p> <p>In both folders the name of each sub-folder is based on the following schema: <strong><em>[SCENARIO]_[W]wl/test_[TEST_ID]</em></strong> where:</p> <ul> <li><em>[SCENARIO]</em> reports the target component for the failure injection (<em>cinder_failure</em>, <em>neutron_failure</em>, <em>nova_failure</em>). <em>ff</em>&nbsp; indicates instead a failure-free execution</li> <li><em>[W]</em> reports the number of concurrent workloads</li> <li><em>[TEST_ID] </em>reports the ID of the specific failure scenario injected (same ID selected by the OpenStack failure injection framework [1] )</li> </ul> <p>Each experiment includes the following data in the <strong><em>raw_dataset</em></strong> sub-folders:</p> <ul> <li><em>audit_raw_logs_[TEST_ID]/</em>: raw audit monitoring data</li> <li><em>bpf_tools_[TEST_ID]/</em>: raw ebpf tools monitoring data</li> <li><em>instance-[INSTANCE_ID]/</em>: workload-specific metadata files, e.g. stdout/stderr (generated by the OpenStack failure injection framework [1] )</li> <li><em>logs_workload_[TEST_ID]/:</em> OpenStack application logs</li> <li><em>perf_tools_[TEST_ID]/</em>: raw perf tools monitoring data</li> <li><em>audit_filtered_[TEST_ID].log:</em> audit data pre-processed by <em>ausearch</em> (e.g. numerical entities are resolved to symbols)</li> <li><em>failure_[TEST_ID].info</em>: metadata information about the specific failure scenario (generated by the OpenStack failure injection framework [1] )</li> <li><em>timestamps_[TEST_ID]:</em> timing information</li> </ul> <p><em>[1] D. Cotroneo, L. De Simone, P. Liguori, R. Natella, N. Bidokhti - How Bad Can a Bug Get? An Empirical Analysis of Software Failures in the OpenStack Cloud Computing Platform [ACM ESEC/FSE 2019]</em></p> <p>&nbsp;</p> <p>Example: parsing a graph from <strong><em>processed_dataset</em> </strong>directory</p> <pre><code class="language-python">import pandas as pd import networkx as nx def parse_csv(path): processes_df = pd.read_csv('%sproc_df.csv' % path, index_col=0).reset_index(drop=True) speakswith_edges_df = pd.read_csv('%spid_speakswith_pid_df.csv' % path, index_col=0) speakswith_edges_df['type'] = 'speaksWith' childof_edges_df = pd.read_csv('%spid_childof_pid_df.csv' % path, index_col=0) childof_edges_df['type'] = 'childOf' return processes_df, pd.concat([speakswith_edges_df, childof_edges_df], ignore_index=True) def make_graph(nodes_df, edges_df): G = nx.MultiGraph() for _, node in nodes_df.iterrows(): G.add_node(node.pid, **node) for _, edge in edges_df.iterrows(): G.add_edge(edge.pid1, edge.pid2, type=edge.type) return G PATH = 'processed_dataset/ff_1wl/test_1/' nodes_df, edges_df = parse_csv(PATH) G = make_graph(nodes_df, edges_df) nx.draw_networkx(G, node_size=10, with_labels=False)</code></pre> <p>&nbsp;</p> <p>If you use this dataset for your research, please cite the following paper:</p> <pre><code>@inproceedings{sanvito2022syslrn, title={syslrn: Learning What to Monitor for Efficient Anomaly Detection}, author={Sanvito, Davide and Siracusano, Giuseppe and Santhanam, Sharan and Gonzalez, Roberto and Bifulco, Roberto}, booktitle={2nd European Workshop on Machine Learning and Systems (EuroMLSys '22)}, year={2022}, address = {Rennes, France}, publisher = {ACM}, month = apr, } </code></pre>