Ghost Echoes Revealed: Benchmarking Maintainability Metrics and Machine Learning Predictions Against Human Assessments

<p>This repo contains a complete replication package, including raw data and scripts for the statistical analysis, for the paper "Ghost Echoes Revealed: Benchmarking Maintainability Metrics and Machine Learning Predictions Against Human Assessments" submitted to the industry track of the <a href="https://conf.researchr.org/home/icsme-2024" rel="nofollow">40th International Conference on Software Maintenance and Evolution (ICSME)</a>, Flagstaff, AZ, USA, Oct 6-11, 2024.</p> <div> <h2>Authors</h2> <a href="https://github.com/codescene-research/maintainability-benchmarking-icsme-2024#authors"></a>Markus Borg, Marwa Ezzouhri, and Adam Tornhill</div> <div> <h2>Abstract</h2> <a href="https://github.com/codescene-research/maintainability-benchmarking-icsme-2024#abstract"></a>As generative AI is expected to increase global code volumes, the importance of maintainability from a human perspective will become even greater. Various methods have been developed to identify the most important maintainability issues, including aggregated metrics and advanced Machine Learning (ML) models. This study benchmarks several maintainability prediction approaches, including State-of-the-Art (SotA) ML, SonarQube's Maintainability Rating, CodeScene's Code Health, and Microsoft's Maintainability Index. Our results indicate that CodeScene matches the accuracy of SotA ML and outperforms the average human expert. Importantly, unlike SotA ML, CodeScene also provides end users with actionable code smell details to remedy identified issues. Finally, caution is advised with SonarQube due to its tendency to generate many false positives. Unfortunately, our findings call into question the validity of previous studies that solely relied on SonarQube output for establishing ground truth labels. To improve reliability in future maintainability and technical debt studies, we recommend employing more accurate metrics. Moreover, reevaluating previous findings with Code Health would mitigate this revealed validity threat.</div> <div> <h2>Repository Content</h2> Two Jupyter Notebooks.</div> <ul> <li> <ul> <li>uc1_maintainability_prediction.ipynb: A Notebook for Use Case 1 - Maintainability Prediction.</li> <li>uc2_liability_prediction.ipynb: A Notebook for Use Case 2 - Liability Prediction.</li> </ul> </li> <li>maintainability_data.csv: The dataset covering the 404 open-source files from the Maintainability Dataset (Bertrand <em>et al.</em>, 2020) <ul> <li>Majority vote ground truth labels from Schnappinger <em>et al.</em> (2020)</li> <li>Low-level code metrics from Bertrand <em>et al.</em> (2023)</li> <li>Code Health</li> <li>SonarQube output, i.e., TD Ratio and TD Time</li> <li>Microsoft Maintainability index provided by <a href="https://plugins.jetbrains.com/plugin/13959-metricstree" rel="nofollow">MetricsTree</a></li> </ul> </li> </ul> <div> <h2>References</h2> </div> <ul> <li>Schnappinger <em>et al.</em>, <a href="https://figshare.com/articles/dataset/A_Software_Maintainability_Dataset/12801215" rel="nofollow">A Software Maintainability Dataset</a>, 10.6084/m9.figshare.12801215, 2020.</li> <li>Schnappinger <em>et al.</em>, Defining a Software Maintainability Dataset: Collecting, Aggregating and Analysing Expert Evaluations of Software Maintainability, in <em>Proc. of the 36th International Conference on Software Maintenance and Evolution</em>, pp. 278&ndash;289, 2020.</li> <li>S. Bertrand <em>et al.</em>, Replication and Extension of Schnappinger&rsquo;s Study on Human-level Ordinal Maintainability Prediction Based on Static Code Metrics, in <em>Proc. of the 27th International Conference on Evaluation and Assessment in Software Engineering</em>, pp. 241&ndash;246, 2023.</li> </ul>