Source Code Archiving to the Rescue of Reproducible Deployment — Replication Package

<p>Replication package for the paper:</p> <p>Ludovic Court&egrave;s, Timothy Sample, Simon Tournier, Stefano Zacchiroli.<br><em>Source Code Archiving to the Rescue of Reproducible Deployment</em><br><a href="https://acm-rep.github.io/2024/">ACM REP'24</a>, June 18-20, 2024, Rennes, France<br><a href="https://doi.org/10.1145/3641525.3663622">https://doi.org/10.1145/3641525.3663622</a></p> <h2>Generating the paper</h2> <p>The paper can be generated using the following command:</p> <pre><code>guix time-machine -C channels.scm \ -- shell -C -m manifest.scm \ -- make </code></pre> <p>This uses GNU Guix to run&nbsp;<code>make</code>&nbsp;in the exact same computational environment used when preparing the paper. The computational environment is described by two files. The&nbsp;<code>channels.scm</code>&nbsp;file specifies the exact version of the Guix package collection to use. The&nbsp;<code>manifest.scm</code>&nbsp;file selects a subset of those packages to include in the environment.</p> <p>It may be possible to generate the paper without Guix. To do so, you will need the following software (on top of a Unix-like environment):</p> <ul> <li>GNU Make</li> <li>SQLite 3</li> <li>GNU AWK</li> <li>Rubber</li> <li>Graphviz</li> <li>TeXLive</li> </ul> <h2>Structure</h2> <ul> <li><code>data/</code>&nbsp;contains the data examined in the paper</li> <li><code>scripts/</code>&nbsp;contains dedicated code for the paper</li> <li><code>logs/</code>&nbsp;contains logs generated during certain computations</li> </ul> <h2>Preservation of Guix</h2> <p>Some of the claims in the paper come from analyzing the Preservation of Guix (PoG) database as published on January 26, 2024. This database is the result of years of monitoring the extent to which the source code referenced by Guix packages is archived. This monitoring has been carried out by Timothy Sample who occasionally publishes reports on his personal website:&nbsp;<a href="https://ngyro.com/pog-reports/latest/">https://ngyro.com/pog-reports/latest/</a>. The database included in this package (<code>data/pog.sql</code>) was downloaded from&nbsp;<a href="https://ngyro.com/pog-reports/2024-01-26/pog.db">https://ngyro.com/pog-reports/2024-01-26/pog.db</a>&nbsp;and then exported to SQL format. In addition to the SQL file, the database schema is also included in this package as&nbsp;<code>data/schema.sql</code>.</p> <p>The database itself is largely the result of scripts, but also of manual adjustments (where necessary or convenient). The scripts are available at&nbsp;<a href="https://git.ngyro.com/preservation-of-guix/">https://git.ngyro.com/preservation-of-guix/</a>, which is preserved in the Software Heritage archive as well:&nbsp;<a href="https://archive.softwareheritage.org/swh:1:snp:efba3456a4aff0bc25b271e128aa8340ae2bc816;origin=https://git.ngyro.com/preservation-of-guix">https://archive.softwareheritage.org/swh:1:snp:efba3456a4aff0bc25b271e128aa8340ae2bc816;origin=https://git.ngyro.com/preservation-of-guix</a>. These scripts rely on the availability of source code in certain locations on the Internet, and therefore will not yield exactly the same result when run again.</p> <h3>Analysis</h3> <p>Here is an overview of how we use the PoG database in the paper. The exact way it is queried to produce graphs and tables for the paper is laid out in the Makefile.</p> <p>The&nbsp;<code>pog-types.sql</code>&nbsp;query gives the counts of each source type (e.g. &ldquo;git&rdquo; or &ldquo;tar-gz&rdquo;) for each commit covered by the database.</p> <p>The&nbsp;<code>pog-status.sql</code>&nbsp;query gives the archival status of the sources by commit. For each commit, it produces a count of how many sources are&nbsp;<em>stored</em>&nbsp;in the Software Heritage archive,&nbsp;<em>missing</em>&nbsp;from it, or&nbsp;<em>unknown</em>&nbsp;if stored or missing. The&nbsp;<code>pog-status-total.sql</code>&nbsp;query does the same thing but over all sources without sorting them into individual commits.</p> <p>The&nbsp;<code>disarchive-ratio.sql</code>&nbsp;query estimates the success rate of Disarchive disassembly.</p> <p>Finally, the&nbsp;<code>swhid-ratio.sql</code>&nbsp;query gives the proportion of sources for which the PoG database has an SWHID.</p> <h3>Estimating missing sources</h3> <p>The Preservation of Guix database only covers sources from a sample of commits to the Guix repository. This greatly simplifies the process of collecting the sources at the risk of missing a few. We estimate how many are missed by searching Guix&rsquo;s Git history for Nix-style base-32 hashes. The result of this search is compared to the hashes in the PoG database.</p> <p>A na&iuml;ve search of Git history results in an over estimate due to Guix&rsquo;s branch development model. We find hashes that were never exposed to users of &lsquo;guix pull&rsquo;. To work around this, we also approximate the history of commits available to &lsquo;guix pull&rsquo;. We do this by scraping push events from the guix-commits mailing list archives (<code>data/guix-commits.mbox</code>). Unfortunately, those archives are not quite complete. Missing history is reconstructed in the&nbsp;<code>data/missing-links.txt</code>&nbsp;file.</p> <p>This estimate requires a copy of the Guix Git repository (not included in this package). The repository can be obtained from GNU at&nbsp;<a href="https://git.savannah.gnu.org/git/guix.git">https://git.savannah.gnu.org/git/guix.git</a>&nbsp;or from the Software Heritage archive:&nbsp;<a href="https://archive.softwareheritage.org/swh:1:snp:9d7b8dcf5625c17e42d51357848baa226b70e4bb;origin=https://git.savannah.gnu.org/git/guix.git">https://archive.softwareheritage.org/swh:1:snp:9d7b8dcf5625c17e42d51357848baa226b70e4bb;origin=https://git.savannah.gnu.org/git/guix.git</a>. Once obtained, its location must be specified in the Makefile.</p> <p>To generate the estimate, use:</p> <pre><code>guix time-machine -C channels.scm \ -- shell -C -m manifest.scm \ -- make data/missing-sources.txt </code></pre> <p>If not using Guix, you will need additional software beyond what is used to generate the paper:</p> <ul> <li>GNU Guile</li> <li>GNU Bash</li> <li>GNU Mailutils</li> <li>GNU Parallel</li> </ul> <h2>Measuring link rot</h2> <p>In order to measure link rot, we ran Guix Scheme scripts, i.e., scripts that exploit Guix as a Scheme library. The scripts depend on the state of world at the very specific moment when they ran. Hence, it is not possible to reproduce the exact same outputs. However, their tendency over the passing of time should be very similar. For running them, you need an installation of&nbsp;<a href="https://guix.gnu.org/manual/deve/en/html_node/Installation.html">Guix</a>. For instance,</p> <pre><code>guix repl -q scripts/table-per-origin.scm </code></pre> <p>When running these scripts for the paper, we tracked their output and saved it inside the&nbsp;<code>logs</code> directory.</p>