Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
11
datasets available to search
ShareScore release 0.7.1
Dataset results
11 results for “Datacenter”
A Systematic Survey of Datacenter Scheduling: Data Artifacts
<p>This release contains the raw search results of the survey conducted in the paper <em>A Systematic Survey of Datacenter Scheduling</em>. </p>
Traces for studying Datacenter Scheduler Programming Abstractions
<p>Traces for the experiments for the research work that investigates the performance impact of various datacenter scheduler programming abstractions.</p>
Beneath the SURFace: An MRI-like View into the Life of a 21st Century Datacenter
<p>This is a trace archive of metrics collected from the Lisa cluster at SURFsara associated with the article that will be published in USENIX;login: in July 2020.</p> <p>Github repository which contains documentation as well as scripts required to replicate the work from the login paper: <a href="https://github.com/sara-nl/SURFace">https://github.com/sara-nl/SURFace</a></p> <p>Real-world data can be instrumental in answering detailed questions: How do we know which assumptions regarding large-scale systems are realistic? How do we know that the systems we build are practical? How do we know which metrics are important to assess when analyzing performance? To answer such questions, we need to collect and share operational traces containing real-world, detailed data. Not only is the presence of low-level metrics significant, but they also help avoid biases through their variety. To address variety, there exist several types of archives, such as the Parallel Workloads Archive, the Grid Workloads Archive, and the Google or Microsoft logs (the Appendix gives a multi-decade overview). However, such traces mostly focus on higher-level scheduling decisions and high-level, job-based resource utilization (e.g., consumed CPU and memory). Thus, they do not provide vital information to system administrators or researchers analyzing the full-stack or the OS-level operation of datacenters. </p> <p><br> The traces we are sharing have the finest granularity of all other open-source traces published so far. In addition to scheduler-level logs, they contain over <em>100 low-level, server-based metrics, going to the granularity of page-faults or bytes transferred through a NIC</em>.</p> <p> </p> <p><strong>The SURF archive</strong></p> <p>Datacenters already exhibit unprecedented scale and are becoming increasingly more complex. Moreover, such computer systems have begun having a significant impact on the environment, for example, training some machine learning models has sizable carbon footprints. As our recent work on modern datacenter networks shows, low-level data is key to understanding full-stack operation, including high-level application behavior. We advocate it is time to start using such data more systematically, unlocking its potential in helping us understand how to make (datacenter) systems more efficient. We advocate that our data can contribute to a more holistic approach, looking at how the multitude of these systems work together in a large-scale datacenter. </p> <p> </p> <p>This archive contains data from the Dutch National Infrastructure, Lisa.</p> <p> </p> <p><a href="https://userinfo.surfsara.nl/systems/lisa/description">Description of the Lisa system</a></p> <p> </p> <p><a href="https://userinfo.surfsara.nl/systems/cartesius/description">Description of the Cartesius system</a></p> <p> </p> <p>We gather metrics, at 15-second intervals, from several data sources:</p> <ul> <li> <p><strong>Slurm:</strong> all job, task, and scheduler related data, such as running time, queueing time, failures, servers involved in the execution, organization in partitions, and scheduling policies.</p> </li> <li> <p><strong>NVIDIA Management Library (NVML): </strong>per GPU, data such as power metrics, temperature, fan speed, or used memory.</p> </li> <li> <p><strong>IPMI: </strong>per server, data such as power metrics and temperature.</p> </li> <li> <p><strong>OS-level: </strong>from either <em>procfs</em>, <em>sockstat,</em> or <em>netstat</em> data: low-level OS metrics, regarding the state of each server, including CPU, disk, memory, network utilization, context switches, and interrupts. </p> </li> </ul> <p> </p> <p>We also release other kinds of novel information, related to datacenter topology and organization.</p> <p> </p> <p>The audience we envision using these metrics is composed of systems researchers, infrastructure developers and designers, system administrators, and software developers for large-scale infrastructure. The frequency of collecting data is uniquely high for open-source data, which could allow these experts unprecedented views into the operation of a real datacenter.</p> <p><br> * Note: For the GPU metrics a number of nodes were introduced to the system in late Feb/start of March and as such these specific nodes have no data available in January and February which may cause irregularities. The github will contain code snippets that will show how to filter this data such that this is not a problem and how to graph the parquet data (this is pending update in the next few days.</p>
Data for the MLCS 2020 paper "A Year of Automated Anomaly Detection in a Datacenter"
<p>This contains the data used for the paper by Ahmed et. al in the MLCS 2020 paper "A Year of Automated Anomaly Detection in a Datacenter". Each of the four CSV files corresponds to one of the quarters discussed in the paper, and each has a metadata file containing information about the query that produced them. The CSV files contain the 'raw' log messages, and an eventID that identifies which pattern the log entry matched; the eventID is used to group together log messages of the same type. These logfiles were collected on the CloudLab facility (https://cloudlab.us/) from Jan 1 - Dec 30, 2019.</p> <p>The violated_unviolated_sessions_*.txt files each contain 20 randomly-selected sessions: half of the sessions were labeled by the invariant miner as being 'normal', and the other half 'anomalous'. CloudLab developers and system administrators were asked to label these sessions manually (and were not given the invariant miner's labels). The corresponding *_manual_labels.txt contain the labels that the administrators assigned, and in some cases additional correspondence with the administrators and information about which manual labels matched the invariant miner and which did not.</p>
Dataset of the HazardNet: A Thermal Hazard Prediction Framework for Datacenters
<p>This dataset entry showcases a comprehensive collection obtained from the Tier-0 supercomputer, Marconi A2, hosted at CINECA (<a href="https://www.hpc.cineca.it/">https://www.hpc.cineca.it/</a>). The dataset records inlet and outlet temperatures along with power consumption data from 3312 computing nodes, spanning from January 14, 2019, to December 31, 2019. The data is generated through ExaMon, a sophisticated monitoring datacenter infrastructure. The primary objective of this dataset is to support the research and development of HazardNet, an innovative thermal hazard prediction framework tailored specifically for datacenters. HazardNet integrates a comprehensive pipeline of machine-learning models. Researchers and enthusiasts interested in exploring our work further can find the complete set of codes and machine-learning models at our GitHub repository: <a href="https://github.com/MSKazemi/HazardNet">https://github.com/MSKazemi/HazardNet</a></p>
[opendc-sc18-dataset] A Reference Architecture for Datacenter Scheduling: Data Artifacts
<p>This release contains the data artifacts of the paper A Reference Architecture for Datacenter Scheduling presented at <a href="https://sc18.supercomputing.org/">Supercomputing 2018</a></p> <p>For the paper, experiments have been run on the following traces:</p> <ul> <li><strong>Askalon (W-Eng)</strong> - <code>askalon_workload_ee</code></li> <li><strong>Chronos (W-Ind)</strong> - <code>chronos_exp_noscaler_ca</code></li> </ul> <p>Each of the directories for the traces have the following structure:</p> <ul> <li><strong>/setup.txt</strong><br> This text file describes the trace used for the experiment in addition to the amount of times the experiment was repeated and the amount of warm-up experiments.</li> <li><strong>/setup.json</strong><br> This JSON file describes the topology of the datacenter used in the experiments. Each item represents the identifiers of the resource (here, CPU type) to use in the machine. The available CPU types are (1) Intel i7 (4 cores, 4100 MHz) and (2) Intel i5 (2 cores, 3500 MHz).</li> <li><strong>/trace</strong><br> This directory contains the trace used in the simulation. The trace is stored in the Grid Workload Format. See the <a href="http://gwa.ewi.tudelft.nl/">Grid Workload Archive</a> for more information.</li> <li><strong>/data/experiments.csv</strong><br> A CSV file containing information of all simulations that have been run on the OpenDC platform for this experiment.</li> <li><strong>/data/job_metrics.csv</strong><br> A CSV file containing metrics (NSL, JMS, etc.) for each job that ran during the simulations.</li> <li><strong>/data/stage_measurements.csv</strong><br> A CSV file containing timing measurements for the scheduling stages that ran during the simulations.</li> <li><strong>/data/task_metrics.csv</strong><br> A CSV file containing metrics for each task that ran during the simulations.</li> <li> <p><strong>/data/tasks.csv</strong><br> A CSV file containing information about the tasks (submit time, runtime, etc.) that ran during the simulations as extracted from the traces.</p> <p>Additionally, we describe the format of each data file in the associated metadata file.</p> </li> </ul> <p><strong>Hardware</strong></p> <p>The hardware used for running the experiments is a MacBook Pro with a 2,9 GHz Intel Core i7 processor and 16 GB 2133 MHz LPDDR3 internal memory.</p> <p><strong>Reproduction</strong></p> <p>This section describes the instructions for reproducing the paper results using a provided Docker image. Please make sure you have <a href="https://www.docker.com/">Docker</a> installed and running.</p> <p>For reproduction, you will run the following experiments:</p> <ul> <li><code>askalon_workload_ee</code><br> This is the large experiment of the paper and will take approximately 4 hours to complete similar hardware.</li> <li><code>chronos_exp_noscaler_ca</code><br> This is the smaller experiment of the paper and will take approximately 5 minutes to complete on similar hardware.</li> </ul> <p>The Docker image <a href="https://hub.docker.com/r/atlargeresearch/sc18-experiment-runner/"><code>atlargeresearch/sc18-experiment-runner</code></a> can be used for running the experiments. A volume can be attached to the directory <code>/home/gradle/simulator/data</code> to capture the results of the experiments.</p> <p>Make sure you have, in your current working directory, the following files:</p> <ul> <li><strong>/setup.json</strong><br> This JSON file describes the topology of the datacenter and can be found in this archive at <code>askalon_workload_ee/setup.json</code>.</li> <li><strong>/askalon_workload_ee.gwf</strong><br> This file contains the trace for the Askalon workload. This file can be found in the archive at <code>askalon_workload_ee/trace/askalon_workload_ee.gwf</code>.</li> <li><strong>/chronos_exp_noscaler_ca.gwf</strong><br> This file contains the trace for the Chronos workload. This file can be found in the archive at <code>chronos_exp_noscaler_ca/trace/chronos_exp_noscaler_ca.gwf</code>.</li> </ul> <p>Then, you can start the Askalon experiments as follows:</p> <pre><code>$ docker run -it --rm -v $(pwd):/home/gradle/simulator/data atlargeresearch/sc18-experiment-runner -r 32 -w 4 -s data/setup.json data/askalon_workload_ee.gwf </code></pre> <p>The experiment runner can be configured with the following options</p> <ul> <li><strong>-r</strong>, <strong>--repeat</strong><br> The amount of times to repeat an experiment for each scheduler.</li> <li><strong>-w</strong>, <strong>--warm-up</strong><br> The amount of times to warm-up the simulator for each scheduler.</li> <li><strong>-p</strong>, <strong>--parallelism</strong><br> The number of experiments to run in parallel.</li> <li><strong>--schedulers</strong><br> The list of schedulers to test, separated by spaces. The following schedulers are available: <code>SRTF-BESTFIT</code>, <code>SRTF-FIRSTFIT</code>, <code>SRTF-WORSTFIT</code>, <code>FIFO-BESTFIT</code>, <code>FIFO-FIRSTFIT</code>, <code>FIFO-WORSTFIT</code>, <code>RANDOM-BESTFIT</code>, <code>RANDOM-FIRSTFIT</code>, <code>RANDOM-WORSTFIT</code>.</li> </ul> <p>After the Askalon experiments have been finished, you can start the Chronos experiments. <strong>Make sure</strong> you have a copy of the result files in your directory as the result files will be overwritten.</p> <pre><code>$ docker run -it --rm -v $(pwd):/home/gradle/simulator/data atlargeresearch/sc18-experiment-runner -r 32 -w 4 -s data/setup.json data/chronos_exp_noscaler_ca.gwf </code></pre>
Generic and ML Workloads in an HPC Datacenter
<p>Updated Version of the <a title="previous upload" href="../records/13625495">previous upload</a>, adjusts node timestamps lacking behind at the beginning of the data collection.</p> <p>This archive contains hardware and workload traces from SURF Lisa, a Dutch datacenter consisting of 338 nodes, used by universities and researchers for various jobs. Around 85% of the nodes are equipped only with CPUs, handling generic compute-heavy workloads, the other 15% come with additional GPUs, serving as accelerators for Machine Learning (ML) jobs. Individual node hardware configurations are listed in `node_hardware_info.parquet`.</p> <p>Jobs within Lisa are submitted over the SLURM scheduler, where we logged job start and end time, resource allocation, and exit state for roughly 10 months (December 2021 to November 2022). This data saved in `slurm_table_cleaned.parquet`.</p> <p>Addidionally, we provide detailed Prometheus monitoring logs from all nodes over a timespan of 5 months (June 2022 to November 2022) in `prom_table_cleaned.parquet`. These logs contain over 90 attributes, including CPU/GPU power and temperatures, network I/O, memory and storage usage, and many more. These metrics are sampled at 30s intervals, resulting in a total of almost 130 million records across all nodes.</p> <p>Finally, job and node data are provided as a joined dataset in `prom_slurm_joined.parquet` for their 4 months of overlapping timespan. This combined data can provide more insights into the resource consumption and performance patterns of jobs.</p> <p>We conducted detailed analysis of this data where we specifically looked at the different characteristics of generic vs. ML workloads in a heterogeneous HPC environment. The pre-print of our analysis work can be found on <a href="https://arxiv.org/abs/2409.08949">arXiv</a>. Our code used for evaluation can be found on <a href="https://github.com/atlarge-research/2024-icpads-hpc-workload-characterization">GitHub</a>.<br><br></p> <table> <tbody> <tr> <th>Dataset Name</th> <th>Explanation</th> </tr> <tr> <td>slurm_table_cleaned.parquet</td> <td>Job data collected by SLURM</td> </tr> <tr> <td>prom_table_cleaned.parquet</td> <td>Node data collected by Prometheus</td> </tr> <tr> <td>prom_slurm_joined.parquet</td> <td>Joined Job and Node dataset</td> </tr> <tr> <td>node_hardware_info.parquet</td> <td>Hardware configurations of each node</td> </tr> </tbody> </table>
Traces for studying Datacenter Scheduler Programming Abstractions
<p>Traces for the experiments for the research work that investigates the performance impact of various datacenter scheduler programming abstractions.</p>
Artifacts of Paper "Understanding and Discovering Software Configuration Dependencies in Cloud and Datacenter Systems"
<p>This package contains all the artifacts (i.e. codes & datasets) we use in our paper "Understanding and Discovering Software Configuration Dependencies in Cloud and Datacenter Systems" accepted to FSE 2020.</p>
Artifacts of Paper "Understanding and Discovering Software Configuration Dependencies in Cloud and Datacenter Systems"
<p>This package contains all the artifacts (i.e. codes & datasets) we use in our paper "Understanding and Discovering Software Configuration Dependencies in Cloud and Datacenter Systems" accepted to FSE 2020.</p>
Artifacts of Paper "Understanding and Discovering Software Configuration Dependencies in Cloud and Datacenter Systems"
<p>This package contains all the artifacts (i.e. codes & datasets) we use in our paper "Understanding and Discovering Software Configuration Dependencies in Cloud and Datacenter Systems" accepted to FSE 2020.</p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.