ArXiv OAI-PMH arXivRaw publication metadata

<p>This dataset contains&nbsp;<a href="https://www.openarchives.org/pmh/">OAI-PMH</a> metadata for all <a href="https://arxiv.org/">ArXiv</a> publications up until 2024-04-23 in the <a href="http://arxiv.org/OAI/arXivRaw.xsd"><em>arXivRaw</em> XML format</a>.</p> <p>The metadata has been harvested using the <a href="https://github.com/miku/metha/releases/tag/v0.3.3"><em>metha</em> Go package v0.3.3</a> [1] on <a href="https://go.dev/"><em>go1.18</em></a>.&nbsp;Specifically, harvesting was run on a small HPC cluster using the following <a href="https://slurm.schedmd.com">SLURM</a> script. The script had to be scheduled twice due to the connection being reset by the peer (see <em>combined-slurm.out</em>). <em>metha</em> caters for these situations and is able to pick up where it left off with cumulative harvesting.</p> <pre><code>#!/bin/bash #SBATCH --job-name=metha #SBATCH --nodes=1 #SBATCH --cpus-per-task=1 #SBATCH --ntasks=1 #SBATCH --time=10-20:00:00 module purge echo "Installing Go module." module add go/go-1.18/go-1.18-gcc-9.4.0-okbjyoy echo "Installed Go module: $(go version)." echo "Installing metha." go install -v github.com/miku/metha/cmd/...@latest echo "Installed metha: $(&lt;retracted&gt;/go/bin/metha-sync -v)" echo "Harvesting ArXiv OAI-PMH metadata in format 'arXivRaw' from http://export.arxiv.org/oai2." &lt;retracted&gt;/go/bin/metha-sync -T 5m -base-dir /scratch/&lt;retracted&gt;/arxiv -format "arXivRaw" http://export.arxiv.org/oai2 # For the second run, '-from' was specified to pick up the harvest where it was left off. # &lt;retracted&gt;/go/bin/metha-sync -from 2020-09-29 -T 5m -base-dir /scratch/&lt;retracted&gt;/arxiv -format "arXivRaw" http://export.arxiv.org/oai2 echo "Done." exit 0</code></pre> <h2>Dataset contents</h2> <p>This deposit of the dataset contains the following files:</p> <ul> <li><strong>metha-output-OAI-PMH-arXivRaw-until-2024-03-24.tar.gz</strong>: an archive file containing the archive files (gzipped, <em>*.xml.gz</em>) produced by&nbsp;<em>metha</em>, which in turn contain the XML metadata files. The gzipped files contained in the archive are named following the pattern <em>YYYY-MM-DD-&lt;8-digit zero-padded 0-index file count&gt;.xml.gz</em>, e.g., <em>2024-03-24-00000001.xml.gz</em>.</li> <li><strong>README.md</strong>: This file, containing basic information about the dataset and deposit.</li> <li><strong>combined-slurm.out</strong>: The combined SLURM log for the two consecutive SLURM runs that have produced the dataset. Run-specific information has been retracted.</li> </ul> <h2>Reproducibility</h2> <p>As the OAI-PMH metadata is not static but may change at any time, this dataset isn't fully reproducible. However, running the same <em>metha</em> version on the same&nbsp;<em>go</em> version with the same commands should yield very similar results, but will contain newer metadata.</p> <h2>Licenses</h2> <ul> <li>All ArXiv OAI-PMH metadata is <a href="https://info.arxiv.org/help/license/index.html#metadata-license">licensed</a> under <a href="https://creativecommons.org/publicdomain/zero/1.0/deed">CC0-1.0</a>.</li> <li>combined-slurm.out is licensed under <a href="https://creativecommons.org/publicdomain/zero/1.0/deed">CC0-1.0</a>.</li> <li>README.md is licensed under&nbsp;<a href="https://creativecommons.org/publicdomain/zero/1.0/deed">CC0-1.0</a>.</li> </ul> <p>Licenses are documented&nbsp;in a machine-readble manner following the <a href="https://reuse.software/spec/">REUSE 3.0 Specification</a>. License deeds are included in this deposit as&nbsp;<em>.txt</em> files named using the respective SPDX license identifiers.</p> <p>[1] Martin Czygan, Thomas Gersch, ACz-UniBi, Justin Kelly, Gunnar &THORN;&oacute;r Magn&uacute;sson, dvglc, &amp; Natanael Arndt. (2024). miku/metha: v0.3.3 (v0.3.3). Zenodo. doi:<a href="https://doi.org/10.5281/zenodo.10940212">10.5281/zenodo.10940212</a>.</p>