Understanding the Energy Consumption of Cloud-native Software Systems

<p>These artifacts contain the dataset generated in the paper "Understanding the Energy Consumption of Cloud-native Software Systems". The dataset contains resource utilisation, power consumption, estimated power consumption and load metrics for a cloud-native software system. The test setup is consists of 6 machines running an OpenStack cluster. This OpenStack cluster is running 12 virtual machines that are in turn hosting a Kubernetes cluster. Performance and (estimated) power consumption are measured on all levels of the system.</p> <p>The Bare-Metal (BM) and Virtual Machines (VM) are identified by their IP addresses. For BM the IP mapping is as follows:</p> <ul> <li>192.168.1.109 - i3-04</li> <li>192.168.1.110 - i3-02</li> <li>192.168.1.111 - i5-02</li> <li>192.168.1.112 - i3-01</li> <li>192.168.1.113 - i5-01</li> <li>192.168.1.114 - i5-04</li> </ul> <p>For VM the IP mapping is:</p> <ul> <li>192.168.1.190 - kubernetes-agent-7</li> <li>192.168.1.191 - kubernetes-agent-6</li> <li>192.168.1.192 - kubernetes-agent-10</li> <li>192.168.1.194 - kubernetes-master</li> <li>192.168.1.196 - kubernetes-agent-3</li> <li>192.168.1.197 - kubernetes-agent-1</li> <li>192.168.1.198 - kubernetes-agent-2</li> <li>192.168.1.201 - kubernetes-agent-4</li> <li>192.168.1.203 - kubernetes-agent-9</li> <li>192.168.1.204 - kubernetes-agent-5</li> <li>192.168.1.207 - kubernetes-agent-8</li> <li>192.168.1.209 - kubernetes-agent-11</li> </ul> <p>The VMs are deployed on the BMs as follows:</p> <ul> <li>i3-01 <ul> <li>kubernetes-agent-2</li> <li>kubernetes-agent-8</li> </ul> </li> <li>i3-02 <ul> <li>kubernetes-agent-1</li> <li>kubernetes-agent-6</li> </ul> </li> <li>i3-04 <ul> <li>kubernetes-agent-10</li> <li>kubernetes-agent-11</li> </ul> </li> <li>i5-01 <ul> <li>kubernetes-agent-3</li> <li>kubernetes-agent-5</li> </ul> </li> <li>i5-02 <ul> <li>kubernetes-master</li> <li>kubernetes-agent-7</li> </ul> </li> <li>i5-04 <ul> <li>kubernetes-agent-4</li> <li>kubernetes-agent-9</li> </ul> </li> </ul> <p>Note that the following BMs are excluded from the experiments as they have other roles in the cluster and do not run workloads:</p> <ul> <li>i3-03 (Juju - deploying OpenStack on BM nodes)</li> <li>i3-05 (ProxMox - external observability tools that do not run in the cluster)</li> <li>i5-03 (MAAS - provisioning of BM nodes)</li> </ul> <h1>Data Sets</h1> <p>The artifacts consists of 3 separate data sets:&nbsp;<code>constant</code>,&nbsp;<code>direct</code>&nbsp;and&nbsp;<code>linear</code>, corresponding to the respective load profile applied to the SUT as discussed in the paper. Each data set contains the same metrics, but the system is put under a different load.</p> <p>Every experiment is repeated 3 times. All 3 repetitions are included in the data set. The data is collected using Prometheus and stored in JSON files.</p> <p>For the&nbsp;<code>constant</code>&nbsp;and&nbsp;<code>linear</code>&nbsp;data sets, load is applied by deploying the "OpenTelemetry demo application", and sending automated user requests to the application. For the&nbsp;<code>direct</code>&nbsp;dataset, this application is not used and instead Kubernetes pods are created that apply a constant load to the cluster.</p> <p>The timestamps of the experiments are as follows:</p> <table> <tbody> <tr> <th>Run</th> <th>Iteration</th> <th>Start</th> <th>End</th> </tr> </tbody> <tbody> <tr> <td>Constant - 0 users</td> <td>1</td> <td>02-05-2024 10:08</td> <td>02-05-2024 10:52</td> </tr> <tr> <td>&nbsp;</td> <td>2</td> <td>06-05-2024 13:20</td> <td>06-05-2024 13:58</td> </tr> <tr> <td>&nbsp;</td> <td>3</td> <td>07-05-2024 09:20</td> <td>07-05-2024 09:54</td> </tr> <tr> <td>Constant - 50 users</td> <td>1</td> <td>02-05-2024 13:26</td> <td>02-05-2024 13:57</td> </tr> <tr> <td>&nbsp;</td> <td>2</td> <td>15-05-2024 14:06</td> <td>15-05-2024 14:38</td> </tr> <tr> <td>&nbsp;</td> <td>3</td> <td>07-05-2024 09:56</td> <td>07-05-2024 10:39</td> </tr> <tr> <td>Constant - 100 users</td> <td>1</td> <td>02-05-2024 14:00</td> <td>02-05-2024 14:40</td> </tr> <tr> <td>&nbsp;</td> <td>2</td> <td>06-05-2024 14:40</td> <td>06-05-2024 15:30</td> </tr> <tr> <td>&nbsp;</td> <td>3</td> <td>07-05-2024 10:42</td> <td>07-05-2024 11:33</td> </tr> <tr> <td>Constant - 200 users</td> <td>1</td> <td>16-05-2024 09:37</td> <td>16-05-2024 10:09</td> </tr> <tr> <td>&nbsp;</td> <td>2</td> <td>06-05-2024 15:32</td> <td>06-05-2024 16:13</td> </tr> <tr> <td>&nbsp;</td> <td>3</td> <td>07-05-2024 11:34</td> <td>07-05-2024 12:11</td> </tr> <tr> <td>Constant - 400 users</td> <td>1</td> <td>02-05-2024 15:24</td> <td>02-05-2024 15:55</td> </tr> <tr> <td>&nbsp;</td> <td>2</td> <td>06-05-2024 16:14</td> <td>06-05-2024 16:49</td> </tr> <tr> <td>&nbsp;</td> <td>3</td> <td>07-05-2024 12:13</td> <td>07-05-2024 13:08</td> </tr> <tr> <td>Linear</td> <td>1</td> <td>22-05-2024 15:13</td> <td>22-05-2024 15:56</td> </tr> <tr> <td>&nbsp;</td> <td>2</td> <td>22-05-2024 16:05</td> <td>22-05-2024 16:48</td> </tr> <tr> <td>&nbsp;</td> <td>3</td> <td>23-05-2024 09:58</td> <td>23-05-2024 10:41</td> </tr> <tr> <td>Direct</td> <td>1</td> <td>2024-05-23 13:01:28</td> <td>2024-05-23 14:00:05</td> </tr> <tr> <td>&nbsp;</td> <td>2</td> <td>2024-05-23 14:33:23</td> <td>2024-05-23 15:32:01</td> </tr> <tr> <td>&nbsp;</td> <td>3</td> <td>2024-05-23 15:39:36</td> <td>2024-05-23 16:38:12</td> </tr> </tbody> </table> <p>A complete log of the experiments can be found in&nbsp;<code>experiments_log.txt</code>.</p> <h2>Constant</h2> <p>The constant data set contains the metrics for the system under a constant load. The load is generated by a Locust script that sends requests to the system. Data collection starts 1 minute after the desired number of concurrent users is reached and requests have stabilised. This experiment is performed for the following constant number of concurrent users:</p> <ul> <li>0 users</li> <li>50 users</li> <li>100 users</li> <li>200 users</li> <li>400 users</li> </ul> <p>Note that the&nbsp;<code>0 users</code>&nbsp;data does not contain the&nbsp;<code>report_*.html</code>&nbsp;and&nbsp;<code>request_*.csv</code>&nbsp;files, as these are generated by Locust and Locust is not run for the&nbsp;<code>0 users</code>&nbsp;scenario.</p> <p>Furthermore, note that Horizontal Pod Autoscaling is&nbsp;<em>not</em>&nbsp;enabled for this data set.</p> <h2>Linear</h2> <p>The linear dataset contains the metrics for the system under a linearly scaling load. The load is generated through Locust. It starts at 0 users, and scales up to 100 users at a rate of 1 user per 20 seconds. After the load reaches 100 users, another 10 minutes of data is recorded. Every dataset is 45 minutes long, with 1.7 minutes of no load, 33.3 minutes of scaling up, and 10 minutes of max load.</p> <p>Note that Horizontal Pod Autoscaling&nbsp;<em>is</em>&nbsp;enabled for this data set.</p> <h2>Direct</h2> <p>The direct data set compliments the linear dataset. While the linear dataset provides a realistic load with things like networking factors being taken into account, the linear dataset can bottleneck on things like networking and the request client, so CPU usage is not maxed out. The direct dataset scales up linearly by applying direct CPU load to the Kubernetes pods without any application simulating a real usecase. The dataset works by deploying Kubernetes pods that max out immediately on exactly 200mCPU of load. The experiment starts with 1 pod, and 2 pods are added every 90 seconds up till 77 pods (the maximum number of pods the cluster allows to be scheduled). The first 1.5 minutes is no load, then 57 minutes to scale up, and then another 1.5 minutes at max load.</p> <p>Note that the&nbsp;<code>direct</code>&nbsp;data set does not contain the&nbsp;<code>app_*.json</code>&nbsp;files, as this data set does not deploy an application and therefore no application specific metrics are collected. Instead, a&nbsp;<code>script_log.txt</code>&nbsp;is provided that explains when and how the direct load was scaled up.</p> <h1>Files</h1> <p>A table for each file in the data set can be found below, including the unit of the metric and a description of what data is collected in that file. The format of all JSON files is the API response format used by Prometheus. More information on this topic can be found here:&nbsp;<a title="https://prometheus.io/docs/prometheus/latest/querying/api/" href="https://prometheus.io/docs/prometheus/latest/querying/api/">https://prometheus.io/docs/prometheus/latest/querying/api/</a>. All timestamps are in the CEST timezone.</p> <table> <tbody> <tr> <th>Filename</th> <th>Unit</th> <th>Description</th> </tr> </tbody> <tbody> <tr> <td>app_ads_ad_requests_total.json</td> <td>Total Count</td> <td>Total requests received by the ad microservice</td> </tr> <tr> <td>app_currency_counter_total.json</td> <td>Total Count</td> <td>Total currency that circulated through the system</td> </tr> <tr> <td>app_frontend_requests_total.json</td> <td>Total Count</td> <td>Total requests received by the frontend service</td> </tr> <tr> <td>app_payment_transactions_total.json</td> <td>Total Count</td> <td>Total transactions made to the transaction microservice</td> </tr> <tr> <td>app_recommendations_counter_total.json</td> <td>Total Count</td> <td>Total recommendations made by the recommendation microservice</td> </tr> <tr> <td>container_blkio_device_usage_total.json</td> <td>Total Bytes</td> <td>Total bytes used by blkio devices for pods</td> </tr> <tr> <td>container_cpu_usage_seconds_total.json</td> <td>Total Seconds</td> <td>Cumulative cpu time consumed by the pod</td> </tr> <tr> <td>container_cpu_user_seconds_total.json</td> <td>Total Seconds</td> <td>Cumulative user cpu time consumed by the pod</td> </tr> <tr> <td>container_fs_reads_bytes_total.json</td> <td>Total Bytes</td> <td>Cumulative count of bytes read by the pod</td> </tr> <tr> <td>container_fs_writes_bytes_total.json</td> <td>Total Bytes</td> <td>Cumulative count of bytes written by the pod</td> </tr> <tr> <td>container_memory_rss.json</td> <td>Bytes</td> <td>Resident Set Size of the pod</td> </tr> <tr> <td>kepler_container_bpf_cpu_time_ms_total.json</td> <td>Milliseconds</td> <td>CPU time for the pod as measured through a Kepler BPF program</td> </tr> <tr> <td>kepler_container_core_joules_total.json</td> <td>Total Joules</td> <td>Total energy consumption of CPU cores used by a pod</td> </tr> <tr> <td>kepler_container_dram_joules_total.json</td> <td>Total Joules</td> <td>Total energy consumption of DRAM used by a pod</td> </tr> <tr> <td>kepler_container_joules_total.json</td> <td>Total Joules</td> <td>Aggregated total energy consumption of a pod</td> </tr> <tr> <td>kepler_container_package_joules_total.json</td> <td>Total Joules</td> <td>Cumulative energy consumed by all cores and uncore components of a pod</td> </tr> <tr> <td>kepler_node_core_joules_total.json</td> <td>Total Joules</td> <td>Aggregation of core_joules of all pods running on a Kubernetes node</td> </tr> <tr> <td>kepler_node_dram_joules_total.json</td> <td>Total Joules</td> <td>Aggregation of dram_joules of all pods running on a Kubernetes node</td> </tr> <tr> <td>kepler_node_package_joules_total.json</td> <td>Total Joules</td> <td>Aggregation of package_joules of all pods running on a Kubernetes node</td> </tr> <tr> <td>node_cpu_scaling_frequency_hertz.json</td> <td>Hertz</td> <td>Current scaled cpu thread frequency of a machine (BM or VM)</td> </tr> <tr> <td>node_cpu_seconds_total.json</td> <td>Total Seconds</td> <td>Total number of seconds the CPU worked on a machine (BM or VM)</td> </tr> <tr> <td>node_disk_read_time_seconds_total.json</td> <td>Total Seconds</td> <td>Total number of seconds spent reading disk on a machine (BM or VM)</td> </tr> <tr> <td>node_disk_write_time_seconds_total.json</td> <td>Total Seconds</td> <td>Total number of seconds spent writing disk on a machine (BM or VM)</td> </tr> <tr> <td>node_hwmon_temp_celsius.json</td> <td>Celsius</td> <td>Temperature of the machine (BM) as reported by its monitoring hardware</td> </tr> <tr> <td>node_load1.json</td> <td>Load Average</td> <td>Load on the machine (BM or VM) averaged over 1 minute</td> </tr> <tr> <td>node_load5.json</td> <td>Load Average</td> <td>Load on the machine (BM or VM) averaged over 5 minutes</td> </tr> <tr> <td>node_load15.json</td> <td>Load Average</td> <td>Load on the machine (BM or VM) averaged over 15 minutes</td> </tr> <tr> <td>node_memory_Active_bytes.json</td> <td>Bytes</td> <td>Active number of bytes in memory on the machine (BM or VM)</td> </tr> <tr> <td>node_memory_Committed_AS_bytes.json</td> <td>Bytes</td> <td>Committed number of bytes in memory on the machine (BM or VM)</td> </tr> <tr> <td>node_rapl_core_joules_total.json</td> <td>Total Joules</td> <td>Total energy consumption of CPU cores by a machine, estimated by RAPL</td> </tr> <tr> <td>node_rapl_dram_joules_total.json</td> <td>Total Joules</td> <td>Total energy consumption of DRAM by a machine, estimated by RAPL</td> </tr> <tr> <td>node_rapl_package_joules_total.json</td> <td>Total Joules</td> <td>Total energy consumption of the machine package, estimated by RAPL</td> </tr> <tr> <td>node_rapl_psys_joules_total.json</td> <td>Total Joules</td> <td>Total energy consumption of the machine psys, estimated by RAPL</td> </tr> <tr> <td>power_consumption.json</td> <td>Watt</td> <td>Energy consumption as measured by the physical power plugs</td> </tr> <tr> <td>report_*.html</td> <td>-</td> <td>HTML report describing details of locust actions during the experiment</td> </tr> <tr> <td>requests_*.csv</td> <td>-</td> <td>Request summary per endpoint generated by locust</td> </tr> <tr> <td>scaph_host_power_microwatts.json</td> <td>Microwatt</td> <td>Power consumption of the whole machine as estimated by Scaphandre</td> </tr> <tr> <td>scaph_process_cpu_usage_percentage.json</td> <td>Percentage</td> <td>Per-process CPU usage as a percentage of total machine CPU</td> </tr> <tr> <td>scaph_process_memory_bytes.json</td> <td>Bytes</td> <td>Per-process memory usage</td> </tr> <tr> <td>scaph_process_power_consumption_microwatts.json</td> <td>Microwatt</td> <td>Per-process energy consumption as estimated by Scaphandre</td> </tr> <tr> <td>script.log</td> <td>-</td> <td>Log for the direct experiments for scaling up the pods</td> </tr> </tbody> </table> <h1>Scripts</h1> <div> <div>All scripts used to query this data from the prometheus endpoint and to generate the results in the associated paper are included in the scripts directory. To run a script, the script must be placed in the same directory as the data it is operated on (e.g. /scripts/constant/power_estimation.ipynb has to be in /constant).</div> </div>