Geographic Diversity in Public Code Contributions — Replication Package

<p>Geographic Diversity in Public Code Contributions - Replication Package</p> <p>This document describes how to replicate the findings of the paper: Davide Rossi and Stefano Zacchiroli, 2022,&nbsp;<em>Geographic Diversity in Public Code Contributions - An Exploratory Large-Scale Study Over 50 Years</em>. In 19th International Conference on Mining Software Repositories (MSR &rsquo;22), May 23-24, Pittsburgh, PA, USA. ACM, New York, NY, USA, 5 pages.&nbsp;<a href="https://doi.org/10.1145/3524842.3528471">https://doi.org/10.1145/3524842.3528471</a></p> <p>This document comes with the software needed to mine and analyze the data presented in the paper.</p> <p>Prerequisites</p> <p>These instructions assume the use of the&nbsp;<a href="https://www.gnu.org/software/bash/">bash</a>&nbsp;shell, the&nbsp;<a href="https://www.python.org/">Python</a>&nbsp;programming language, the&nbsp;<a href="https://www.postgresql.org/">PosgreSQL</a>&nbsp;DBMS (version 11 or later), the&nbsp;<a href="https://facebook.github.io/zstd/">zstd</a>&nbsp;compression utility and various usual *nix shell utilities (cat, pv, &hellip;), all of which are available for multiple architectures and OSs.<br> It is advisable to create a&nbsp;<a href="https://docs.python.org/3/tutorial/venv.html">Python virtual environment</a>&nbsp;and install the following PyPI packages:</p> <pre><code>click==8.0.4 cycler==0.11.0 fonttools==4.31.2 kiwisolver==1.4.0 matplotlib==3.5.1 numpy==1.22.3 packaging==21.3 pandas==1.4.1 patsy==0.5.2 Pillow==9.0.1 pyparsing==3.0.7 python-dateutil==2.8.2 pytz==2022.1 scipy==1.8.0 six==1.16.0 statsmodels==0.13.2</code></pre> <p>Initial data</p> <ul> <li><code>swh-replica</code>, a PostgreSQL database containing a copy of Software Heritage data. The schema for the database is available at&nbsp;<a href="https://forge.softwareheritage.org/source/swh-storage/browse/master/swh/storage/sql/">https://forge.softwareheritage.org/source/swh-storage/browse/master/swh/storage/sql/</a>.<br> We retrieved these data from&nbsp;<a href="https://www.softwareheritage.org">Software Heritage</a>, in collaboration with the archive operators, taking an archive snapshot as of 2021-07-07. We cannot make these data available in full as part of the replication package due to both its volume and the presence in it of personal information such as user email addresses. However, equivalent data (stripped of email addresses) can be obtained from the Software Heritage archive dataset, as documented in the article: Antoine Pietri, Diomidis Spinellis, Stefano Zacchiroli,&nbsp;<em>The Software Heritage Graph Dataset: Public software development under one roof</em>. In proceedings of MSR 2019: The 16th International Conference on Mining Software Repositories, May 2019, Montreal, Canada. Pages 138-142, IEEE 2019.&nbsp;<a href="http://dx.doi.org/10.1109/MSR.2019.00030">http://dx.doi.org/10.1109/MSR.2019.00030</a>.<br> Once retrieved, the data can be loaded in PostgreSQL to populate&nbsp;<code>swh-replica</code>.</li> <li><code>names.tab</code>&nbsp;- forenames and surnames per country with their frequency</li> <li><code>zones.acc.tab</code>&nbsp;- countries/territories, timezones, population and world zones</li> <li><code>c_c.tab</code>&nbsp;- ccTDL entities - world zones matches</li> </ul> <p>Data preparation</p> <ul> <li> <p>Export data from the&nbsp;<code>swh-replica</code>&nbsp;database to create&nbsp;<code>commits.csv.zst</code>&nbsp;and&nbsp;<code>authors.csv.zst</code></p> <pre><code>sh&gt; ./export.sh</code></pre> </li> <li> <p>Run the authors cleanup script to create&nbsp;<code>authors--clean.csv.zst</code></p> <pre><code>sh&gt; ./cleanup.sh authors.csv.zst</code></pre> </li> <li> <p>Filter out implausible names and create&nbsp;<code>authors--plausible.csv.zst</code></p> <pre><code>sh&gt; pv authors--clean.csv.zst | unzstd | ./filter_names.py 2&gt; authors--plausible.csv.log | zstdmt &gt; authors--plausible.csv.zst</code></pre> </li> </ul> <p>Zone detection by email</p> <ul> <li> <p>Run the email detection script to create&nbsp;<code>author-country-by-email.tab.zst</code></p> <pre><code>sh&gt; pv authors--plausible.csv.zst | zstdcat | ./guess_country_by_email.py -f 3 2&gt; author-country-by-email.csv.log | zstdmt &gt; author-country-by-email.tab.zst</code></pre> </li> </ul> <p>Database creation and initial data ingestion</p> <ul> <li> <p>Create the PostgreSQL DB</p> <pre><code>sh&gt; createdb zones-commit</code></pre> <p>Notice that from now on when prepending the&nbsp;<code>psql&gt;</code>&nbsp;prompt we assume the execution of psql on the&nbsp;<code>zones-commit</code>&nbsp;database.</p> </li> <li> <p>Import data into PostgreSQL DB</p> <pre><code>sh&gt; ./import_data.sh</code></pre> </li> </ul> <p>Zone detection by name</p> <ul> <li> <p>Extract commits data from the DB and create&nbsp;<code>commits.tab</code>, that is used as input for the zone detection script</p> <pre><code>sh&gt; psql -f extract_commits.sql zones-commit</code></pre> </li> <li> <p>Run the world zone detection script to create&nbsp;<code>commit_zones.tab.zst</code></p> <pre><code>sh&gt; pv commits.tab | ./assign_world_zone.py -a -n names.tab -p zones.acc.tab -x -w 8 | zstdmt &gt; commit_zones.tab.zst</code></pre> Use&nbsp;<code>./assign_world_zone.py --help</code>&nbsp;if you are interested in changing the script parameters.</li> <li> <p>Ingest zones assignment data into the DB</p> <pre><code>psql&gt; \copy commit_zone from program 'zstdcat commit_zones.tab.zst | cut -f1,6 | grep -Ev ''\s$'''</code></pre> </li> </ul> <p>Extraction and graphs</p> <ul> <li> <p>Run the script to execute the queries to extract the data to plot from the DB. This creates&nbsp;<code>commit_zones_7120.tab</code>,&nbsp;<code>author_zones_7120_t5.tab</code>,&nbsp;<code>commit_zones_7120.grid</code>&nbsp;and&nbsp;<code>author_zones_7120_t5.grid</code>.<br> Edit&nbsp;<code>extract_data.sql</code>&nbsp;if you whish to modify extraction parameters (start/end year, sampling, &hellip;).</p> <pre><code>sh&gt; ./extract_data.sh</code></pre> </li> <li> <p>Run the script to create the graphs from all the previously extracted tabfiles.</p> <pre><code>sh&gt; ./create_stackedbar_chart.py -w 20 -s 1971 -f commit_zones_7120.grid -f author_zones_7120_t5.grid -o chart.pdf</code></pre> </li> </ul>