Replication Package for: Theodolite: Scalability Benchmarking of Distributed Stream Processing Engines in Microservice Architectures

<p>This repository contains a replication package and experimental results for our study <em>Theodolite: Scalability Benchmarking of Distributed Stream Processing Engines in Microservice Architectures</em>.</p> <p>The following description can also be found in the README.md file.</p> <p><strong>Repeating Benchmark Execution</strong></p> <p><em><strong>The following introduction describes how to repeat our scalability experiments. If you plan to conduct your own studies, we suggest to use the latest version of <a href="https://github.com/cau-se/theodolite">Theodolite</a> with significantly enhanced usability.</strong></em></p> <p>The Apache Kafka Streams scalability experiments of our study were executed with <a href="https://github.com/cau-se/theodolite/tree/v0.1.2">Theodolite v0.1.2</a>. To repeat our Kafka Streams experiments:</p> <ol> <li>Clone and install <a href="https://github.com/cau-se/theodolite/tree/v0.1.2">Theodolite v0.1.2</a> according to the official documentation located in <code>execution</code>.</li> <li>Copy the file <code>repeat-kstream.sh</code> into Theodolite&#39;s <code>execution</code> directory.</li> <li>Run the repetition file with <code>./repeat-kstream.sh</code> from within the <code>execution</code> directory.</li> </ol> <p>Our Apache Flink benchmark implementations are currently migrated to the latest version of Theodolite. <a href="https://github.com/cau-se/theodolite/tree/apache-flink">Theodolite&#39;s <code>apache-flink</code> Branch</a> provides the basis for our Flink scalability experiments. To repeat them:</p> <ol> <li>Clone <a href="https://github.com/cau-se/theodolite/tree/apache-flink">Theodolite&#39;s <code>apache-flink</code> Branch</a> and install Theodolite according to the official documentation located in <code>execution</code> (should be identical to the installation for Kafka Streams (see above)).</li> <li>Copy the files <code>repeat-flink-without-checkpointing.sh</code> and <code>repeat-flink-with-checkpointing.sh</code> into Theodolite&#39;s <code>execution</code> directory.</li> <li>Switch to the <code>execution</code> directory.</li> <li>Run the first repetition file with <code>./repeat-flink-with-checkpointing.sh</code>.</li> <li>Disable checkpointing by reconfiguring the Kubernetes resources <code>jobmanager-job.yaml</code> and <code>taskmanager-job-deployment.yaml</code> for each benchmark (<code>uc{1,2,3,4}-application</code>) by setting the environment variable <code>CHECKPOINTING</code> to <code>&quot;false&quot;</code>.</li> <li>Run the second repetition file with <code>./repeat-flink-without-checkpointing.sh</code>.</li> </ol> <p><em>Please note that the naming of our benchmarks recently changed. While our publication already uses the new naming, the corresponding Theodolite versions are is still using the old one. Specifically, this means that UC1 in the publication is UC1 in Theodolite, UC2 in the publication is UC3 in Theodolite, UC3 in the publication is UC4 in Theodolite, and UC4 in the publication is UC2 in Theodolite.</em></p> <p><strong>Raw Measurements</strong></p> <p>The results of above benchmark execution can be found in the <code>measurements</code> directory. These are CSV files, containing the measured lag trend over time for a certain subexperiment. Theodolite creates a bunch of additional files, which serve for debugging and preliminary interpretation. As these files are not required for replication, we do not included them in this package.</p> <p>The CSV files are named according to the schema <code>exp{id}_{uc}_{load}_{inst}_totallag.csv</code>, where <code>{id}</code> represents the experiment ID, assigned by Theodolite, <code>{uc}</code> the benchmark name, <code>{load}</code> the generated load, and <code>{inst}</code> the number of evaluated instances.</p> <p>The CSV table <code>experiments.csv</code> provides an overview about the configurations used in each experiment.</p> <p><strong>Reproducing Scalability Analysis</strong></p> <p><em><strong>The following introduction describes how to repeat our scalability analysis, either with our measurements or with your own. If you plan to conduct your own studies, we suggest to use the latest version of <a href="https://github.com/cau-se/theodolite">Theodolite</a> with significantly enhanced usability.</strong></em></p> <p>Analyzing the Theodolite&#39;s measurements is done using two Jupyter notebooks. In general, these notebooks should be runnable by any Jupyter server. Python 3.7 or 3.8 is required (e.g., in a virtual environment) as well as some Python libraries, which can be installed via: <code>pip install -r requirements.txt</code>. See the <a href="https://github.com/cau-se/theodolite/tree/master/analysis">Theodolite documentation</a> for additional installation guidance.</p> <p><strong>Obtaining a Scalability Graph as a CSV File</strong></p> <p>The <code>scalability-graph.ipynb</code> notebook combines the measurements (i.e., the <code>totallag.csv</code> files) of one experiment. It produces a CSV file, which provides a mapping of load intensities to minimum required resources for that load (i.e., the scalability graph). The CSV files are named according to the schema <code>exp{id}_min-suitable-instances.csv</code>, where <code>{id}</code> represents the experiment ID. Additional guidance is provided in the notebook.</p> <p><strong>Resulting Scalability Graph CSV Files</strong></p> <p>The <code>results</code> directory provides the scalability graphs for all our executed experiments.</p> <p><strong>Visualization of the Scalability Graph</strong></p> <p>The <code>scalability-graph-plotter.ipynb</code> notebook creates PDF plots of a scalability graph and allows to combine multiple scalability graphs in one plot. It can be adjusted to match the desired visualization.</p> <p><strong>Acknowledgments</strong></p> <p>This research is funded by the German Federal Ministry of Education and Research (BMBF) under grant no. 01IS17084 and is part of the <a href="https://www.industrial-devops.org">Titan project</a>.</p>