Skip to main content
zenodoopen

Does Co-Development with AI Assistants Lead to More Maintainable Code? Replication Package

<p>This is the replication package for the study "Echoes of AI: Investigating the Downstream Effects of AI Assistants on Software Maintainability" preregistered aat ICSME 2024 as&nbsp;&ldquo;Does Co-Development with AI Assistants Lead to More Maintainable Code?&rdquo;</p> <p>Abstract from the registered report:</p> <p>[Background/Context] AI assistants like GitHub Copilot are transforming software engineering, with several studies highlighting productivity improvements. However, their impact on code quality, particularly in terms of maintainability, requires further investigation.<br>[Objective/Aim] This study aims to examine the influence of AI assistants on software maintainability, specifically assessing how these tools affect the ability of developers to evolve code.<br>[Method] We will conduct a two-phased controlled experiment involving professional developers. In Phase 1, developers will add a new feature to a Java project, with or without the aid of an AI assistant. Phase 2, a randomized controlled trial, will involve a different set of developers evolving random Phase 1 projects - working without AI assistants. We will employ Bayesian analysis to evaluate differences in completion time, perceived productivity, code quality, and test coverage.</p> <p>Note: To maintain the integrity of the study, i.e., preventing any leakage to AI assistants' training data, we choose not to host the code in a public git repository. Instead, all relevant documents and code are shared through a replication package on Zenodo, available as PDF documents generated by repo2pdf (https://github.com/BankkRoll/repo2pdf). We have deliberately used settings to obfuscate the code (e.g., line numbers) to ensure it will not be scraped by any large language models before the study has been completed.</p> <p>Contents:</p> <ul> <li>Task 1 instructions.</li> <li>Task 2 instructions.</li> <li>The source code that the participants received.</li> <li>A causal graph with analysis details.</li> <li>Archives containing anonymized experimental data and analysis scripts (in .zip and .tar.gz for convenience).&nbsp;</li> </ul>

ShareScore

32/100

Overall dataset sharing score

Score breakdown

These five areas show where the dataset supports — or may limit — practical reuse.

Stewardship
4
Harmonization
4
Access
16
Reuse readiness
8
Engagement
0

Dataset details

Repository
zenodo
Published
July 24, 2024
License
cc-by-4.0
DOI
10.5281/zenodo.11243196

Scored with release 0.9.0

ScienceDex for faculty

Your academic CV and NIH biosketch

Make your CV in your school's format and your NIH biosketch from it, then download both as Word files.