Harvest Quebec Government Websites from December 2006 web archive collection derivatives

<p>Web archive derivatives of the Sites of the Harvest Quebec Government Websites from December 2006 collection from the <a href="https://www.banq.qc.ca/accueil/">Biblioth&egrave;que et Archives nationales du Qu&eacute;bec</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a>. Merci beaucoup BAnQ!</p> <p>These&nbsp;derivatives&nbsp;are&nbsp;in&nbsp;the <a href="https://parquet.apache.org/">Apache&nbsp;Parquet format</a>,&nbsp;which&nbsp;is&nbsp;a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar&nbsp;storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/parquet_pandas_example.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web&nbsp;Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web&nbsp;Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image&nbsp;Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary&nbsp;Analysis</strong></a></p> <ul> <li>Audio</li> <li>Images</li> <li>PDFs</li> <li>Presentation&nbsp;program&nbsp;files</li> <li>Spreadsheets</li> <li>Text&nbsp;files</li> <li>Videos</li> <li>Word&nbsp;processor&nbsp;files</li> </ul>