Replication Package for 'How do Machine Learning Models Change?'

<h1>Replication Package: How Do Machine Learning Models Change?</h1> <p>&nbsp;</p> <h2>Overview</h2> <div>This replication package accompanies the paper "<strong>How Do Machine Learning Models Change?</strong>". In this study, we conducted a large-scale analysis of&nbsp;<strong>over 680,000 commits from 100,000 models</strong>&nbsp;and <strong>2,251 releases from 202 of these models</strong>&nbsp;on the Hugging Face (HF) platform. Our goal was to understand how machine learning (ML) models evolve by classifying commit types using a detailed ML change taxonomy and analyzing temporal patterns in their activities using Bayesian networks.</div> <p>&nbsp;</p> <div>Our research addresses three main aspects:</div> <div><strong>1. Categorization of Commit Changes:</strong> We classified over 960,000 commits on HF, providing a detailed breakdown of change types and their distribution.</div> <div><strong>2. Analysis of Commit Sequences</strong>: We examined the sequence and dependencies of commit types using Bayesian networks to identify temporal patterns.</div> <div><strong>3. Release Analysis**</strong>: We investigated the distribution and evolution of release types, analyzing how model attributes and metadata change across successive releases.</div> <p>&nbsp;</p> <div>This package provides all the necessary code, data, and documentation to reproduce the results presented in our paper.</div> <p>&nbsp;</p> <h2>Data Collection and Preprocessing</h2> <h3>Data Collection</h3> <div>We collected data from the Hugging Face platform using the Hugging Face Hub API. The data extraction was performed up to&nbsp;<strong>May 2025,</strong> capturing details from over 1 million models available at that time.</div> <p>&nbsp;</p> <div><strong>- Model &amp; Release Information</strong>: We collected model metadata, commit histories, and release information (Git tags) for our sampled models.</div> <div><strong>- Detailed Commit Changes</strong>: To get a detailed list of files modified in each commit, we implemented a direct Git processing approach. For each model, its repository was temporarily cloned to programmatically extract the list of changed files for every commit SHA.</div> <p>&nbsp;</p> <h3>Data Preprocessing</h3> <div><strong>Commit Diffs</strong></div> <div>We computed the differences for key JSON configuration files (e.g., `config.json`) between commits to identify added, deleted, and updated keys, which served as input for classification.</div> <div>&nbsp;</div> <div><strong>Commit Classification</strong></div> <div>We classified each commit according to Bhatia et al.'s ML change taxonomy using the&nbsp;<strong>Gemini 2.5 Flash LLM</strong>. To ensure the reliability of this process, we implemented a rigorous&nbsp;<strong>two-phase validation</strong>:</div> <div><strong>1. Prompt Refinement (Training): </strong>The prompt was iteratively refined over 6 cycles using a curated training set of 143 commits. The process was guided by comparing LLM classifications against a gold standard created by two human annotators (Human-Human IRR on a subset: 𝜅 = 0.7798). The final refined prompt achieved a Kappa of 0.9068 against the training gold standard.</div> <div><strong>2. Final Validation (Testing): </strong>The validated prompt was tested on an independent, statistically significant sample of 384 commits. The LLM's classifications achieved a Cohen's Kappa of 0.8568 when compared against the test set's gold standard, which itself was validated with a human-human IRR of 𝜅 = 0.8150.</div> <p>&nbsp;</p> <div>We also classified commits into Swanson's categories using a fine-tuned DistilBERT model, as detailed in the paper.</div> <p>&nbsp;</p> <h2>Folder Structure</h2> <div>The replication package is organized as follows. The structure has been designed to separate code, data, and metadata for clarity.</div> <p>&nbsp;</p> <ul> <li>`code/`: Contains all Jupyter notebooks for the study. <ul> <li>`Collection/`: Scripts for data extraction from Hugging Face.</li> <li>`HFExtraction.ipynb`: Collects primary model and commit information.</li> <li>`HFReleasesExtraction.ipynb`: Collects release (tag) specific information.</li> </ul> </li> <li>`Preprocessing/`: Scripts for data cleaning, processing, and classification. <ul> <li>`HFCommitsPreprocessing.ipynb`: Processes commits, computes diffs, and prepares data for classification and analysis.</li> <li>`HFReleasesPreprocessing.ipynb`: Processes and classifies release data.</li> </ul> </li> <li>`Analysis/`: Notebooks for reproducing the analysis for each research question. <ul> <li>`HFFileChanges.ipynb`: Contains the preliminary analysis of file change patterns.</li> <li>`RQ1_Analysis.ipynb`: Analysis for Research Question 1.</li> <li>`RQ2_Analysis.ipynb`: Analysis for Research Question 2.</li> <li>`RQ3_Analysis.ipynb`: Analysis for Research Question 3.</li> </ul> </li> <li>`datasets/`: Contains the key final datasets used in the analysis notebooks. <ul> <li>`commits_datasets/`: Contains the main classified commit dataset. <ul> <li>`HFCommitsClassification_final.csv`: The final dataset with over 960,000 classified commits for RQ1 and RQ2.</li> </ul> </li> <li>`releases_datasets/`: Contains the datasets related to releases. <ul> <li>`HFReleasesClassification.csv`: The final dataset of 2,251 classified releases for RQ3.</li> </ul> </li> <li>`model_metadata.csv`: The extracted internal metadata from model files for RQ3.4.</li> </ul> </li> <li>`metadata/`: Contains configuration files and the data used for the validation process. <ul> <li>`validation_data/`: A sub-folder containing the gold standard data. <ul> <li>`Agreement TOSEM Commit Changes.xlsx`: Excel containing details of the classication and validation processes.</li> <li>`prompt_refinement.txt`: The final, validated prompt used for the LLM classification along its previous iterations.</li> <li>`training_set_ground_truth.json`: Gold standard for the 143-commit training set.</li> <li>`training_set_first_classification.json`: First annotator's labels for the training IRR subset.</li> <li>`training_set_second_classification.json`: Second annotator's labels for the training IRR subset.</li> <li>`test_set_ground_truth.json`: Gold standard for the 384-commit test set.</li> <li>`test_first_classification.json`: First annotator's labels for the training IRR subset.</li> </ul> </li> <li>`test_set_second_classification.json`: Second annotator's labels for the test IRR subset.</li> <li>`tags_metadata.yaml`: Auxiliary metadata file used during preprocessing.</li> </ul> </li> <li>`README.md`: This file.</li> <li>`requirements.txt`: Lists the required Python packages.</li> </ul> <p>*Note: Other intermediate CSV files are provided to facilitate re-running specific parts of the analysis without starting from scratch.*</p> <p>&nbsp;</p> <h2>How to Use This Package</h2> <p>&nbsp;</p> <h3>Setup</h3> <div><strong>1. Create and activate a virtual environment</strong>&nbsp;(recommended).</div> <div>```bash</div> <div>python -m venv venv</div> <div>source venv/bin/activate # On Windows: venv\Scripts\activate</div> <div>```</div> <div><strong>2. Install required packages.</strong></div> <div>```bash</div> <div>pip install -r requirements.txt</div> <div>```</div> <p>&nbsp;</p> <h3>Running the Analysis</h3> <div>The Jupyter notebooks in the `code/` directory are numbered and named to be run in a logical sequence:&nbsp;<strong>Collection -&gt; Preprocessing -&gt; Analysis.</strong>&nbsp;We recommend following this order.</div> <div>&nbsp;</div> <div>-&nbsp;<strong>To reproduce our findings directly,</strong> you can start with the notebooks in `code/Analysis/`. They are configured to load the final, processed datasets provided in the `datasets/` folder.</div> <div>-&nbsp;<strong>To re-run the entire pipeline</strong>, start with the notebooks in `code/Collection/`, followed by `code/Preprocessing/`. Please note that running the full data collection and classification pipeline is time-consuming and may require significant computational resources and appropriate API keys for the LLM.</div> <p>&nbsp;</p> <h3>Key Datasets Provided</h3> <div><strong>- For RQ1 &amp; RQ2:</strong>&nbsp;`datasets/commits_datasets/HFCommitsClassification_final.csv` (100,000 models for RQ1; filtered to 14,343 models for RQ2).</div> <div><strong>- For RQ3.1-3.3:</strong>`datasets/releases_datasets/HFReleasesClassification.csv` (2,251 releases from 202 models).</div> <div><strong>- For RQ3.4: </strong>`datasets/releases_datasets/model_metadata.csv` (from 28 models).</div> <p>&nbsp;</p> <h2>Contact</h2> <div>If you have any questions or encounter issues with this package, please contact the corresponding author. If you find our work useful, please consider citing our paper.</div>