Materials for 2d representation of the HathiTrust Library

<p>Materials to create the LargeVis visualization online at&nbsp;http://creatingdata.us/datasets/hathi-features/, and described in&nbsp;<em>Benjamin Schmidt, &quot;Stable random projection: lightweight, general-purpose dimensionality reduction for digitized libraries,&quot; Journal of Cultural Analytics. October 3, 2018.</em></p> <p>Two items. First, `hathi_pca.bin`: a binary file with 100-dimensional representations of the complete Hathi Trust Extended Features set. These began as 1280-dimensional SRP features, and were reduced to 100 dimensions using a PCA transformation matrix derived using a random sample of the full 13 million book set. Vectors were reduced to unit length before PCA, but not afterwords; this means that in general, their length gives some sense of much information was lost in the PCA representation. This can be read using the code at https://github.com/bmschmidt/pySRP, or anything that reads word2vec formatted vectors. Includes HathiTrust identifiers.</p> <p>Second, `hathi.tsv.gz`: a row oriented set containing a variety of metadata fields for each set, including (as &#39;x&#39; and &#39;y&#39;) the coordinates of a 2-d LargeVis visualization. This is the immediate input to the visualization at&nbsp;ttp://creatingdata.us/datasets/hathi-features/. Columns should be relatively straightforward; they are derived from the HathiTrust MARC records, which can be accessed through Hathi&#39;s public API. Classification codes (&#39;lc1&#39;) are using the Library of Congress classification; they represent the subclass (generally two characters, though it can be one or three). The first character alone represents the LC class and can be useful for coloring high-level overviews.</p> <p>These two files can be merged through the Hathi Trust identifier present in both.</p> <p>&nbsp;</p>