Skip to main content
Powered by ShareScore

Find research datasets worth reusing

Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.

12,346

datasets available to search

ShareScore release 0.9.0

Reset

Dataset results

12,346 results for “derivatives”

Learn how ShareScore rates datasets ↗
zenodo36/100

Elucidation of the Atroposelectivity in the Synthesis of Axially Chiral Thiohydantoin Derivatives

<p>Recently, Sarigul and Dogan&nbsp;have synthesized a number of enantiomerically enriched axially chiral atropoisomeric 2-thiohydantoins by the reaction of L-amino acid ester salts and <em>o</em>-aryl isothiocyanates in the presence of triethyl amine (TEA) in dichloromethane. The nonaxially chiral derivative 5-methyl-3-phenyl-2-thiohydantoin gave a racemic product whereas the axially chiral 5-methyl-3-<em>o</em>-bromophenyl-2-thiohydantoin was less prone to racemize at C<sub>5</sub> of the heterocyclic ring. In this study, we present a computational study (M06-2X/6-311+G(d,p) for C, H, O, N and S; M06-2X/6-311++G(3df,3pd) for Br) in order to propose plausible mechanisms for the racemization and cyclization steps for &nbsp;2-thiohydantoin derivatives. The study includes rationalization based on steric as well as the electrostatic effects to elucidate the epimerization differences at C<sub>5</sub>.</p>

opencc-by-4.0Jan 2020View details →
zenodo36/100

Web Archive of Independent News Sites on Turkish Affairs derivatives

<p>Derivatives of the&nbsp;<a href="https://archive-it.org/collections/12911">Web Archive of Independent News Sites on Turkish Affairs</a> collection from the <a href="https://archive-it.org/home/IvyPlus">Ivy Plus Libraries Confederation</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a> and <a href="https://cloud.archivesunleashed.org/">Archives Unleashed Cloud</a>.</p> <p>The <strong>ivy-12911-parquet.tar.gz</strong> derivatives&nbsp;are&nbsp;in&nbsp;the <a href="https://parquet.apache.org/">Apache&nbsp;Parquet format</a>,&nbsp;which&nbsp;is&nbsp;a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar&nbsp;storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/datathon-nyc/parquet_pandas_stonewall.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web&nbsp;Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web&nbsp;Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image&nbsp;Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary&nbsp;Analysis</strong></a></p> <ul> <li>Audio</li> <li>Images</li> <li>PDFs</li> <li>Presentation&nbsp;program&nbsp;files</li> <li>Spreadsheets</li> <li>Text&nbsp;files</li> <li>Word&nbsp;processor&nbsp;files<br> &nbsp;</li> </ul> <p>The <strong>ivy-12911-auk.tar.gz </strong>derivatives<strong> </strong>are the <a href="https://cloud.archivesunleashed.org/derivatives">standard set of web archive derivatives</a> produced by the Archives Unleashed Cloud.</p> <ul> <li><strong>Gephi </strong>file, which can be loaded into <a href="https://gephi.org/">Gephi</a>. It will have basic characteristics already computed and a basic layout.</li> <li><strong>Raw Network</strong> file, which can also be loaded into <a href="https://gephi.org/">Gephi</a>. You will have to use that network program to lay it out yourself.</li> <li><strong>Full text</strong> file. In it, each website within the web archive collection will have its full text presented on one line, along with information around when it was crawled, the name of the domain, and the full URL of the content.</li> <li><strong>Domains count</strong> file. A text file containing the frequency count of domains captured within your web archive.</li> </ul>

opencc-by-4.0Jan 2020View details →
zenodo36/100

Popline and K4Health Web Archive collection derivatives

<p>Web archive derivatives of the&nbsp;<a href="https://archive-it.org/collections/12006">Popline and K4Health Web Archive</a> collection from the <a href="https://archive-it.org/home/IvyPlus">Ivy Plus Libraries Confederation</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a> and <a href="https://cloud.archivesunleashed.org/">Archives Unleashed Cloud</a>.</p> <p>The <strong>ivy-12006-parquet.tar.gz</strong> derivatives&nbsp;are&nbsp;in&nbsp;the <a href="https://parquet.apache.org/">Apache&nbsp;Parquet format</a>,&nbsp;which&nbsp;is&nbsp;a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar&nbsp;storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/datathon-nyc/parquet_pandas_stonewall.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web&nbsp;Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web&nbsp;Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image&nbsp;Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary&nbsp;Analysis</strong></a></p> <ul> <li>Audio</li> <li>Images</li> <li>PDFs</li> <li>Presentation&nbsp;program&nbsp;files</li> <li>Spreadsheets</li> <li>Text&nbsp;files</li> <li>Word&nbsp;processor&nbsp;files<br> &nbsp;</li> </ul> <p>The <strong>ivy-12006-auk.tar.gz </strong>derivatives<strong> </strong>are the <a href="https://cloud.archivesunleashed.org/derivatives">standard set of web archive derivatives</a> produced by the Archives Unleashed Cloud.</p> <ul> <li><strong>Gephi </strong>file, which can be loaded into <a href="https://gephi.org/">Gephi</a>. It will have basic characteristics already computed and a basic layout.</li> <li><strong>Raw Network</strong> file, which can also be loaded into <a href="https://gephi.org/">Gephi</a>. You will have to use that network program to lay it out yourself.</li> <li><strong>Full text</strong> file. In it, each website within the web archive collection will have its full text presented on one line, along with information around when it was crawled, the name of the domain, and the full URL of the content.</li> <li><strong>Domains count</strong> file. A text file containing the frequency count of domains captured within your web archive.</li> </ul>

opencc-by-4.0Jan 2020View details →
zenodo36/100

Neurogliaform cortical interneurons derive from cells in the preoptic area

<p>Excel document containing the raw dataset used in each figures and links to morphology reconstructions</p>

opencc-by-4.0Mar 2018View details →
zenodo36/100

Independent Documentary Filmmakers from China, Hong Kong, and Taiwan Web Archive collection derivatives

<p>Web archive derivatives of the <a href="https://archive-it.org/collections/12172">Literary Authors from Europe and Eurasia Web Archive</a> collection from the <a href="https://archive-it.org/home/IvyPlus">Ivy Plus Libraries Confederation</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a> and <a href="https://cloud.archivesunleashed.org/">Archives Unleashed Cloud</a>.</p> <p>The <strong>ivy-12126-parquet.tar.gz</strong> derivatives&nbsp;are&nbsp;in&nbsp;the <a href="https://parquet.apache.org/">Apache&nbsp;Parquet format</a>,&nbsp;which&nbsp;is&nbsp;a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar&nbsp;storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/datathon-nyc/parquet_pandas_stonewall.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web&nbsp;Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web&nbsp;Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image&nbsp;Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary&nbsp;Analysis</strong></a></p> <ul> <li>Audio</li> <li>Images</li> <li>PDFs</li> <li>Presentation&nbsp;program&nbsp;files</li> <li>Spreadsheets</li> <li>Text&nbsp;files</li> <li>Word&nbsp;processor&nbsp;files<br> &nbsp;</li> </ul> <p>The <strong>ivy-12126-auk.tar.gz </strong>derivatives<strong> </strong>are the <a href="https://cloud.archivesunleashed.org/derivatives">standard set of web archive derivatives</a> produced by the Archives Unleashed Cloud.</p> <ul> <li><strong>Gephi </strong>file, which can be loaded into <a href="https://gephi.org/">Gephi</a>. It will have basic characteristics already computed and a basic layout.</li> <li><strong>Raw Network</strong> file, which can also be loaded into <a href="https://gephi.org/">Gephi</a>. You will have to use that network program to lay it out yourself.</li> <li><strong>Full text</strong> file. In it, each website within the web archive collection will have its full text presented on one line, along with information around when it was crawled, the name of the domain, and the full URL of the content.</li> <li><strong>Domains count</strong> file. A text file containing the frequency count of domains captured within your web archive.</li> </ul>

opencc-by-4.0Jan 2020View details →
zenodo36/100

Literary Authors from Europe and Eurasia Web Archive collection derivatives

<p>Web archive derivatives of the <a href="https://archive-it.org/collections/12172">Literary Authors from Europe and Eurasia Web Archive</a> collection from the <a href="https://archive-it.org/home/IvyPlus">Ivy Plus Libraries Confederation</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a> and <a href="https://cloud.archivesunleashed.org/">Archives Unleashed Cloud</a>.</p> <p>The <strong>ivy-12172-parquet.tar.gz</strong> derivatives&nbsp;are&nbsp;in&nbsp;the <a href="https://parquet.apache.org/">Apache&nbsp;Parquet format</a>,&nbsp;which&nbsp;is&nbsp;a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar&nbsp;storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/datathon-nyc/parquet_pandas_stonewall.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web&nbsp;Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web&nbsp;Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image&nbsp;Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary&nbsp;Analysis</strong></a></p> <ul> <li>Audio</li> <li>Images</li> <li>PDFs</li> <li>Presentation&nbsp;program&nbsp;files</li> <li>Spreadsheets</li> <li>Text&nbsp;files</li> <li>Word&nbsp;processor&nbsp;files<br> &nbsp;</li> </ul> <p>The <strong>ivy-12172-auk.tar.gz </strong>derivatives<strong> </strong>are the <a href="https://cloud.archivesunleashed.org/derivatives">standard set of web archive derivatives</a> produced by the Archives Unleashed Cloud.</p> <ul> <li><strong>Gephi </strong>file, which can be loaded into <a href="https://gephi.org/">Gephi</a>. It will have basic characteristics already computed and a basic layout.</li> <li><strong>Raw Network</strong> file, which can also be loaded into <a href="https://gephi.org/">Gephi</a>. You will have to use that network program to lay it out yourself.</li> <li><strong>Full text</strong> file. In it, each website within the web archive collection will have its full text presented on one line, along with information around when it was crawled, the name of the domain, and the full URL of the content.</li> <li><strong>Domains count</strong> file. A text file containing the frequency count of domains captured within your web archive.</li> </ul>

opencc-by-4.0Jan 2020View details →
zenodo36/100

Queer Japan Web Archive collection derivatives

<p>Web archive derivatives of the&nbsp;<a href="https://archive-it.org/collections/11854">Queer Japan Web Archive</a> collection from the <a href="https://archive-it.org/home/IvyPlus">Ivy Plus Libraries Confederation</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a> and <a href="https://cloud.archivesunleashed.org/">Archives Unleashed Cloud</a>.</p> <p>The <strong>ivy-12172-parquet.tar.gz</strong> derivatives&nbsp;are&nbsp;in&nbsp;the <a href="https://parquet.apache.org/">Apache&nbsp;Parquet format</a>,&nbsp;which&nbsp;is&nbsp;a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar&nbsp;storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/datathon-nyc/parquet_pandas_stonewall.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web&nbsp;Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web&nbsp;Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image&nbsp;Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary&nbsp;Analysis</strong></a></p> <ul> <li>Audio</li> <li>Images</li> <li>PDFs</li> <li>Presentation&nbsp;program&nbsp;files</li> <li>Spreadsheets</li> <li>Text&nbsp;files</li> <li>Videos</li> <li>Word&nbsp;processor&nbsp;files<br> &nbsp;</li> </ul> <p>The <strong>ivy-11854-auk.tar.gz </strong>derivatives<strong> </strong>are the <a href="https://cloud.archivesunleashed.org/derivatives">standard set of web archive derivatives</a> produced by the Archives Unleashed Cloud.</p> <ul> <li><strong>Gephi </strong>file, which can be loaded into <a href="https://gephi.org/">Gephi</a>. It will have basic characteristics already computed and a basic layout.</li> <li><strong>Raw Network</strong> file, which can also be loaded into <a href="https://gephi.org/">Gephi</a>. You will have to use that network program to lay it out yourself.</li> <li><strong>Full text</strong> file. In it, each website within the web archive collection will have its full text presented on one line, along with information around when it was crawled, the name of the domain, and the full URL of the content.</li> <li><strong>Domains count</strong> file. A text file containing the frequency count of domains captured within your web archive.</li> </ul>

opencc-by-4.0Jan 2020View details →
zenodo36/100

General web archive collection derivatives

<p>Web archive derivatives of the <a href="https://archive-it.org/collections/1716">General</a> collection from <a href="https://archive-it.org/home/Columbia">Columbia University Libraries</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a> and <a href="https://cloud.archivesunleashed.org/">Archives Unleashed Cloud</a>.</p> <p>The&nbsp;<strong>cul-1716-parquet.tar.gz</strong> derivatives&nbsp;are&nbsp;in&nbsp;the <a href="https://parquet.apache.org/">Apache&nbsp;Parquet format</a>,&nbsp;which&nbsp;is&nbsp;a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar&nbsp;storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/datathon-nyc/parquet_pandas_stonewall.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web&nbsp;Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web&nbsp;Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image&nbsp;Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary&nbsp;Analysis</strong></a></p> <ul> <li>Audio</li> <li>Images</li> <li>PDFs</li> <li>Presentation&nbsp;program&nbsp;files</li> <li>Spreadsheets</li> <li>Text&nbsp;files</li> <li>Word&nbsp;processor&nbsp;files<br> &nbsp;</li> </ul> <p>The <strong>cul-1716-auk.tar.gz </strong>derivatives<strong> </strong>are the <a href="https://cloud.archivesunleashed.org/derivatives">standard set of web archive derivatives</a> produced by the Archives Unleashed Cloud.</p> <ul> <li><strong>Gephi </strong>file, which can be loaded into <a href="https://gephi.org/">Gephi</a>. It will have basic characteristics already computed and a basic layout.</li> <li><strong>Raw Network</strong> file, which can also be loaded into <a href="https://gephi.org/">Gephi</a>. You will have to use that network program to lay it out yourself.</li> <li><strong>Full text</strong> file. In it, each website within the web archive collection will have its full text presented on one line, along with information around when it was crawled, the name of the domain, and the full URL of the content.</li> <li><strong>Domains count</strong> file. A text file containing the frequency count of domains captured within your web archive.</li> </ul>

opencc-by-4.0Jan 2020View details →
zenodo36/100

Global Webcomics Web Archive collection derivatives

<p>Web archive derivatives of the&nbsp;<a href="https://archive-it.org/collections/10181">Global Webcomics Web Archive</a> collection from the <a href="https://archive-it.org/home/IvyPlus">Ivy Plus Libraries Confederation</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a> and <a href="https://cloud.archivesunleashed.org/">Archives Unleashed Cloud</a>.</p> <p>The <strong>ivy-10181-parquet.tar.gz</strong> derivatives&nbsp;are&nbsp;in&nbsp;the <a href="https://parquet.apache.org/">Apache&nbsp;Parquet format</a>,&nbsp;which&nbsp;is&nbsp;a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar&nbsp;storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/datathon-nyc/parquet_pandas_stonewall.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web&nbsp;Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web&nbsp;Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image&nbsp;Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary&nbsp;Analysis</strong></a></p> <ul> <li>Audio</li> <li>Images</li> <li>PDFs</li> <li>Presentation&nbsp;program&nbsp;files</li> <li>Spreadsheets</li> <li>Text&nbsp;files</li> <li>Word&nbsp;processor&nbsp;files<br> &nbsp;</li> </ul> <p>The <strong>ivy-10181-auk.tar.gz </strong>derivatives<strong> </strong>are the <a href="https://cloud.archivesunleashed.org/derivatives">standard set of web archive derivatives</a> produced by the Archives Unleashed Cloud.</p> <ul> <li><strong>Gephi </strong>file, which can be loaded into <a href="https://gephi.org/">Gephi</a>. It will have basic characteristics already computed and a basic layout.</li> <li><strong>Raw Network</strong> file, which can also be loaded into <a href="https://gephi.org/">Gephi</a>. You will have to use that network program to lay it out yourself.</li> <li><strong>Full text</strong> file. In it, each website within the web archive collection will have its full text presented on one line, along with information around when it was crawled, the name of the domain, and the full URL of the content.</li> <li><strong>Domains count</strong> file. A text file containing the frequency count of domains captured within your web archive.</li> </ul>

opencc-by-4.0Jan 2020View details →
zenodo36/100

Brazilian Presidential Transition (2018) Web Archive collection derivatives

<p>Web archive derivatives of the&nbsp;<a href="https://archive-it.org/collections/11549">Brazilian Presidential Transition (2018) Web Archive collection</a> collection from the <a href="https://archive-it.org/home/IvyPlus">Ivy Plus Libraries Confederation</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a> and <a href="https://cloud.archivesunleashed.org/">Archives Unleashed Cloud</a>.</p> <p>The <strong>ivy-11549-parquet.tar.gz</strong> derivatives&nbsp;are&nbsp;in&nbsp;the <a href="https://parquet.apache.org/">Apache&nbsp;Parquet format</a>,&nbsp;which&nbsp;is&nbsp;a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar&nbsp;storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/datathon-nyc/parquet_pandas_stonewall.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web&nbsp;Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web&nbsp;Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image&nbsp;Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary&nbsp;Analysis</strong></a></p> <ul> <li>Audio</li> <li>Images</li> <li>PDFs</li> <li>Presentation&nbsp;program&nbsp;files</li> <li>Spreadsheets</li> <li>Text&nbsp;files</li> <li>Word&nbsp;processor&nbsp;files<br> &nbsp;</li> </ul> <p>The <strong>ivy-11549-auk.tar.gz </strong>derivatives<strong> </strong>are the <a href="https://cloud.archivesunleashed.org/derivatives">standard set of web archive derivatives</a> produced by the Archives Unleashed Cloud.</p> <ul> <li><strong>Gephi </strong>file, which can be loaded into <a href="https://gephi.org/">Gephi</a>. It will have basic characteristics already computed and a basic layout.</li> <li><strong>Raw Network</strong> file, which can also be loaded into <a href="https://gephi.org/">Gephi</a>. You will have to use that network program to lay it out yourself.</li> <li><strong>Full text</strong> file. In it, each website within the web archive collection will have its full text presented on one line, along with information around when it was crawled, the name of the domain, and the full URL of the content.</li> <li><strong>Domains count</strong> file. A text file containing the frequency count of domains captured within your web archive.</li> </ul>

opencc-by-4.0Feb 2020View details →
zenodo36/100

Collaborative Architecture, Urbanism, and Sustainability Web Archive (CAUSEWAY) collection derivatives

<p>Web archive derivatives of the&nbsp;<a href="https://archive-it.org/collections/4638">Collaborative Architecture, Urbanism, and Sustainability Web Archive (CAUSEWAY)</a> collection from the <a href="https://archive-it.org/home/IvyPlus">Ivy Plus Libraries Confederation</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a> and <a href="https://cloud.archivesunleashed.org/">Archives Unleashed Cloud</a>.</p> <p>The <strong>ivy-4638-parquet.tar.gz</strong> derivatives&nbsp;are&nbsp;in&nbsp;the <a href="https://parquet.apache.org/">Apache&nbsp;Parquet format</a>,&nbsp;which&nbsp;is&nbsp;a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar&nbsp;storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/datathon-nyc/parquet_pandas_stonewall.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web&nbsp;Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web&nbsp;Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image&nbsp;Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary&nbsp;Analysis</strong></a></p> <ul> <li>Audio</li> <li>Images</li> <li>Presentation&nbsp;program&nbsp;files</li> <li>Spreadsheets</li> <li>Text&nbsp;files</li> <li>Word&nbsp;processor&nbsp;files<br> &nbsp;</li> </ul> <p>The <strong>ivy-4638-auk.tar.gz </strong>derivatives<strong> </strong>are the <a href="https://cloud.archivesunleashed.org/derivatives">standard set of web archive derivatives</a> produced by the Archives Unleashed Cloud.</p> <ul> <li><strong>Gephi </strong>file, which can be loaded into <a href="https://gephi.org/">Gephi</a>. It will have basic characteristics already computed and a basic layout.</li> <li><strong>Raw Network</strong> file, which can also be loaded into <a href="https://gephi.org/">Gephi</a>. You will have to use that network program to lay it out yourself.</li> <li><strong>Full text</strong> file. In it, each website within the web archive collection will have its full text presented on one line, along with information around when it was crawled, the name of the domain, and the full URL of the content.</li> <li><strong>Domains count</strong> file. A text file containing the frequency count of domains captured within your web archive.</li> </ul>

opencc-by-4.0Feb 2020View details →
zenodo36/100

Quebec Ministry of Agriculture, Fisheries and Food from 2012-2018 web archive collection derivatives

<p>Web archive derivatives of the Quebec Ministry of Agriculture, Fisheries and Food from 2012-2018 collection from the <a href="https://www.banq.qc.ca/accueil/">Biblioth&egrave;que et Archives nationales du Qu&eacute;bec</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a>. Merci beaucoup BAnQ!</p> <p>These&nbsp;derivatives&nbsp;are&nbsp;in&nbsp;the <a href="https://parquet.apache.org/">Apache&nbsp;Parquet format</a>,&nbsp;which&nbsp;is&nbsp;a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar&nbsp;storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/parquet_pandas_example.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web&nbsp;Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web&nbsp;Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image&nbsp;Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary&nbsp;Analysis</strong></a></p> <ul> <li>Audio</li> <li>Images</li> <li>PDFs</li> <li>Presentation&nbsp;program&nbsp;files</li> <li>Spreadsheets</li> <li>Text&nbsp;files</li> <li>Videos</li> <li>Word&nbsp;processor&nbsp;files</li> </ul>

opencc-by-4.0Feb 2020View details →
zenodo36/100

Sites of the Quebec Ministry of Immigration from 2012 to 2018 web archive collection derivatives

<p>Web archive derivatives of the Sites of the Quebec Ministry of Immigration from 2012 to 2018 collection from the <a href="https://www.banq.qc.ca/accueil/">Biblioth&egrave;que et Archives nationales du Qu&eacute;bec</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a>. Merci beaucoup BAnQ!</p> <p>These&nbsp;derivatives&nbsp;are&nbsp;in&nbsp;the <a href="https://parquet.apache.org/">Apache&nbsp;Parquet format</a>,&nbsp;which&nbsp;is&nbsp;a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar&nbsp;storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/parquet_pandas_example.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web&nbsp;Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web&nbsp;Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image&nbsp;Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary&nbsp;Analysis</strong></a></p> <ul> <li>Audio</li> <li>Images</li> <li>PDFs</li> <li>Presentation&nbsp;program&nbsp;files</li> <li>Spreadsheets</li> <li>Text&nbsp;files</li> <li>Videos</li> <li>Word&nbsp;processor&nbsp;files</li> </ul>

opencc-by-4.0Feb 2020View details →
zenodo36/100

Contemporary Composers Web Archive (CCWA) web archive collection derivatives

<p>Web archive derivatives of the&nbsp;<a href="https://archive-it.org/collections/4019">Contemporary Composers Web Archive (CCWA)</a> collection from the <a href="https://archive-it.org/home/IvyPlus">Ivy Plus Libraries Confederation</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a> and <a href="https://cloud.archivesunleashed.org/">Archives Unleashed Cloud</a>.</p> <p>The <strong>ivy-4019-parquet.tar.gz</strong> derivatives&nbsp;are&nbsp;in&nbsp;the <a href="https://parquet.apache.org/">Apache&nbsp;Parquet format</a>,&nbsp;which&nbsp;is&nbsp;a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar&nbsp;storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/datathon-nyc/parquet_pandas_stonewall.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web&nbsp;Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web&nbsp;Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image&nbsp;Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary&nbsp;Analysis</strong></a></p> <ul> <li>Audio</li> <li>Images</li> <li>PDFs</li> <li>Presentation&nbsp;program&nbsp;files</li> <li>Spreadsheets</li> <li>Text&nbsp;files</li> <li>Word&nbsp;processor&nbsp;files<br> &nbsp;</li> </ul> <p>The <strong>ivy-4019-auk.tar.gz </strong>derivatives<strong> </strong>are the <a href="https://cloud.archivesunleashed.org/derivatives">standard set of web archive derivatives</a> produced by the Archives Unleashed Cloud.</p> <ul> <li><strong>Gephi </strong>file, which can be loaded into <a href="https://gephi.org/">Gephi</a>. It will have basic characteristics already computed and a basic layout.</li> <li><strong>Raw Network</strong> file, which can also be loaded into <a href="https://gephi.org/">Gephi</a>. You will have to use that network program to lay it out yourself.</li> <li><strong>Full text</strong> file. In it, each website within the web archive collection will have its full text presented on one line, along with information around when it was crawled, the name of the domain, and the full URL of the content.</li> <li><strong>Domains count</strong> file. A text file containing the frequency count of domains captured within your web archive.</li> </ul>

opencc-by-4.0Feb 2020View details →
zenodo36/100

Parti Québécois (PQ) web archive collection derivatives

<p>Web archive derivatives of the Parti Qu&eacute;b&eacute;cois (PQ) collection from the <a href="https://www.banq.qc.ca/accueil/">Biblioth&egrave;que et Archives nationales du Qu&eacute;bec</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a>. Merci beaucoup BAnQ!</p> <p>These&nbsp;derivatives&nbsp;are&nbsp;in&nbsp;the <a href="https://parquet.apache.org/">Apache&nbsp;Parquet format</a>,&nbsp;which&nbsp;is&nbsp;a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar&nbsp;storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/parquet_pandas_example.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web&nbsp;Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web&nbsp;Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image&nbsp;Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary&nbsp;Analysis</strong></a></p> <ul> <li>Audio</li> <li>Images</li> <li>PDFs</li> <li>Presentation&nbsp;program&nbsp;files</li> <li>Spreadsheets</li> <li>Text&nbsp;files</li> <li>Word&nbsp;processor&nbsp;files</li> </ul>

opencc-by-4.0Mar 2020View details →
zenodo36/100

Rare Book and Manuscript Library web archive collection derivatives

<p>Web archive derivatives of the&nbsp;<a href="https://archive-it.org/collections/2766">Rare Book and Manuscript Library</a> collection from <a href="https://archive-it.org/home/Columbia">Columbia University Libraries</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a> and <a href="https://cloud.archivesunleashed.org/">Archives Unleashed Cloud</a>.</p> <p>The&nbsp;<strong>cul-2766-parquet.tar.gz</strong> derivatives&nbsp;are&nbsp;in&nbsp;the <a href="https://parquet.apache.org/">Apache&nbsp;Parquet format</a>,&nbsp;which&nbsp;is&nbsp;a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar&nbsp;storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/datathon-nyc/parquet_pandas_stonewall.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web&nbsp;Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web&nbsp;Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image&nbsp;Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary&nbsp;Analysis</strong></a></p> <ul> <li>Images</li> <li>PDFs</li> <li>Presentation&nbsp;program&nbsp;files</li> <li>Spreadsheets</li> <li>Word&nbsp;processor&nbsp;files<br> &nbsp;</li> </ul> <p>The <strong>cul-2766-auk.tar.gz </strong>derivatives<strong> </strong>are the <a href="https://cloud.archivesunleashed.org/derivatives">standard set of web archive derivatives</a> produced by the Archives Unleashed Cloud.</p> <ul> <li><strong>Gephi </strong>file, which can be loaded into <a href="https://gephi.org/">Gephi</a>. It will have basic characteristics already computed and a basic layout.</li> <li><strong>Raw Network</strong> file, which can also be loaded into <a href="https://gephi.org/">Gephi</a>. You will have to use that network program to lay it out yourself.</li> <li><strong>Full text</strong> file. In it, each website within the web archive collection will have its full text presented on one line, along with information around when it was crawled, the name of the domain, and the full URL of the content.</li> <li><strong>Domains count</strong> file. A text file containing the frequency count of domains captured within your web archive.</li> </ul>

opencc-by-4.0Mar 2020View details →
zenodo36/100

Burke Library New York City Religions web archive collection derivatives

<p>Web archive derivatives of the&nbsp;<a href="https://archive-it.org/collections/1945">Burke Library New York City Religions</a> collection from <a href="https://archive-it.org/home/Columbia">Columbia University Libraries</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a> and <a href="https://cloud.archivesunleashed.org/">Archives Unleashed Cloud</a>.</p> <p>The&nbsp;<strong>cul-1945-parquet.tar.gz</strong> derivatives&nbsp;are&nbsp;in&nbsp;the <a href="https://parquet.apache.org/">Apache&nbsp;Parquet format</a>,&nbsp;which&nbsp;is&nbsp;a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar&nbsp;storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/datathon-nyc/parquet_pandas_stonewall.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web&nbsp;Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web&nbsp;Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image&nbsp;Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces&nbsp;a&nbsp;DataFrame&nbsp;with&nbsp;the&nbsp;following&nbsp;columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary&nbsp;Analysis</strong></a></p> <ul> <li>Images</li> <li>PDFs</li> <li>Presentation&nbsp;program&nbsp;files</li> <li>Spreadsheets</li> <li>Text&nbsp;files</li> <li>Word&nbsp;processor&nbsp;files<br> &nbsp;</li> </ul> <p>The <strong>cul-1945-auk.tar.gz </strong>derivatives<strong> </strong>are the <a href="https://cloud.archivesunleashed.org/derivatives">standard set of web archive derivatives</a> produced by the Archives Unleashed Cloud.</p> <ul> <li><strong>Gephi </strong>file, which can be loaded into <a href="https://gephi.org/">Gephi</a>. It will have basic characteristics already computed and a basic layout.</li> <li><strong>Raw Network</strong> file, which can also be loaded into <a href="https://gephi.org/">Gephi</a>. You will have to use that network program to lay it out yourself.</li> <li><strong>Full text</strong> file. In it, each website within the web archive collection will have its full text presented on one line, along with information around when it was crawled, the name of the domain, and the full URL of the content.</li> <li><strong>Domains count</strong> file. A text file containing the frequency count of domains captured within your web archive.</li> </ul>

opencc-by-4.0Mar 2020View details →
zenodo36/100

Adaptation time to magnified flood hazards underestimated when derived from tide gauge records [dataset]

<p>This dataset contains the data supporting the manuscript Lambert et al., Adaptation time to magnified flood hazards underestimated when derived from tide gauge records, submitted to Environmental Research Letters.</p> <p>All files contain two main variables:<br> AF is the amplification factor as a function of [station, time, scenario]<br> DT is the doubling time as a function of [station, scenario]</p> <p>Note that these files contain data of all 299 available stations, rather than the subset of 130 presented in the manuscript.</p> <p>Filenames:<br> diss_AWL.nc&nbsp;&nbsp;&nbsp;&nbsp; Average Water Level at dissipative beaches<br> diss_IWL.nc&nbsp;&nbsp;&nbsp;&nbsp; Instantaneous Water Level at dissipative beaches<br> grd_AWL.nc&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Average Water Level at gentle rocky dikes<br> grd_IWL.nc&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Instantaneous Water Level at gentle rocky dikes<br> harbour.nc&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Average and Instantaneous Water Level at harbours, excluding wave contributions<br> srd_AWL.nc&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Average Water Level at steep rocky dikes<br> srd_IWL.nc&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Instantaneous Water Level at steep rocky dikes<br> ssb_AWL.nc&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Average Water Level at steep sandy beaches<br> ssb_IWL.nc&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Instantaneous Water Level at steep sandy beaches<br> tidegauge.nc&nbsp;&nbsp;&nbsp; Tide gauge-based analysis</p>

opencc-by-4.0Mar 2020View details →
zenodo36/100

Derived data and code for Deines et al. 2020, Agricultural Water Management

<p>This codebase accompanies the paper:</p> <p>Deines, J.M., M.E. Schipanski, B. Golden, S.C. Zipper, S. Nozari, C. Rottler, B. Guerrero, &amp; V. Sharda. 2020. Transitions from irrigated to dryland agriculture in the Ogallala Aquifer: Land use suitability and regional economic impacts. Agricultural Water Management 233:106061. DOI:&nbsp;<a href="https://doi.org/10.1016/j.agwat.2020.106061">https://doi.org/10.1016/j.agwat.2020.106061</a></p> <p>Data needed to reproduce the figures from Deines et al. 2020 can be found in the&nbsp;<code>data</code>&nbsp;folder.</p> <p>See the Readme.md or&nbsp;<a href="https://github.com/jdeines/Deines_etal_2020_agwat_OgallalaTransistions">https://github.com/jdeines/Deines_etal_2020_agwat_OgallalaTransistions</a>&nbsp;for more information.</p>

opencc-by-4.0Feb 2020View details →
zenodo36/100

soilmap_simple: a simplified and standardized derivative of the digital soil map of the Flemish Region

<p>The data source <code>soilmap_simple</code> is a simplified and standardized derived form of the &#39;<a href="https://www.dov.vlaanderen.be/geonetwork/srv/dut/catalog.search#/metadata/5c129f2d-4498-4bc3-8860-01cb2d513f8f">digital soil map of the Flemish Region</a>&#39; (the shapefile of which we named <code>soilmap</code>, for analytical workflows in R) published by &#39;Databank Ondergrond Vlaanderen&rsquo; (DOV). It is a GeoPackage that contains a&nbsp;<strong>spatial polygon layer</strong> &lsquo;<code>soilmap_simple</code>&rsquo; in the Belgian Lambert 72 coordinate reference system (EPSG-code <a href="https://epsg.io/31370">31370</a>), plus a <strong>non-spatial table</strong> &lsquo;<code>explanations</code>&rsquo; with the meaning&nbsp;of category codes that occur in&nbsp;the spatial layer. Further documentation about the digital soil map of the Flemish Region is available in Van Ranst &amp; Sys (2000) and Dudal et al. (2005).</p> <p>This version of <code>soilmap_simple</code> was derived from version &#39;<code>soilmap_2017-06-20</code>&#39; (<a href="https://doi.org/10.5281/zenodo.3387008">Zenodo DOI</a>) as follows:</p> <ul> <li>all attribute variables received English names (purpose of standardization), starting with prefix <code>bsm_</code> (referring to the &#39;Belgian soil map&#39;);</li> <li>attribute variables were reordered;</li> <li>the values of the morphogenetic substrate, texture and drainage variables (<code>bsm_mo_substr</code>, <code>bsm_mo_tex</code> and <code>bsm_mo_drain</code> + their <code>_explan</code> counterparts) were filled for most&nbsp;features in the &#39;coastal plain&#39; area. <ul> <li>To derive morphogenetic texture and drainage levels from the geomorphological soil types, a conversion table by Bruno De Vos &amp; Carole Ampe was applied (for earlier work on this, see Ampe 2013).</li> <li>Substrate classes were copied over from <code>bsm_ge_substr</code> into <code>bsm_mo_substr</code> (<code>bsm_ge_substr</code> already followed the categories of <code>bsm_mo_substr</code>).</li> </ul> These steps coincide with the approach that had been taken to construct the <code>Unitype</code> variable in the <code>soilmap</code> data source;</li> <li>only a minimal number of variables were selected: those that are most useful for analytical work.</li> </ul> <p>See R-code in the GitHub repository <a href="https://github.com/inbo/n2khab-preprocessing/tree/b3c6696/src/generate_soilmap_simple">&#39;n2khab-preprocessing&#39; at commit b3c6696</a> for the creation from the <code>soilmap</code> data source.</p> <p>A reading function to return <code>soilmap_simple</code> (this data source) or <code>soilmap</code> in a standardized way into the R environment is provided by the R-package <a href="https://inbo.github.io/n2khab/">n2khab</a>.</p> <p><strong>The attributes</strong> of the spatial polygon layer&nbsp;<code>soilmap_simple</code> can have <code>mo_</code> in their name to refer&nbsp;to the <em>Belgian Morphogenetic System</em>:</p> <ul> <li><code>bsm_poly_id</code>: unique polygon ID (numeric)</li> <li><code>bsm_region</code>: name of the region</li> <li><code>bsm_converted</code>: boolean.&nbsp;Were morphogenetic texture and drainage variables (<code>bsm_mo_tex</code>&nbsp;and&nbsp;<code>bsm_mo_drain</code>) derived from a conversion table (see above)? Value&nbsp;<code>TRUE</code>&nbsp;is largely confined to the &#39;coastal plain&#39; areas.</li> <li><code>bsm_mo_soilunitype</code>: code of the soil type (applying morphogenetic codes within the coastal plain areas when possible, just as for the following three variables)</li> <li><code>bsm_mo_substr</code>: code of the soil substrate</li> <li><code>bsm_mo_tex</code>: code of the soil texture category</li> <li><code>bsm_mo_drain</code>: code of the soil drainage category</li> <li><code>bsm_mo_prof</code>: code of the soil profile category</li> <li><code>bsm_mo_parentmat</code>: code of a variant regarding the parent material</li> <li><code>bsm_mo_profvar</code>: code of a variant regarding the soil profile</li> </ul> <p>The <strong>non-spatial table</strong>&nbsp;<code>explanations</code>&nbsp;has following variables:</p> <ul> <li><code>subject</code>:&nbsp;attribute name of the spatial layer: either <code>bsm_mo_substr</code>, <code>bsm_mo_tex</code>, <code>bsm_mo_drain</code>, <code>bsm_mo_prof</code>, <code>bsm_mo_parentmat</code>&nbsp;or&nbsp;<code>bsm_mo_profvar</code></li> <li><code>code</code>: category code that occurs as value for the corresponding attribute in the spatial layer</li> <li><code>name</code>: explanation of the value of&nbsp;<code>code</code></li> </ul>

opencc-by-4.0Mar 2020View details →

ScienceDex guides

Understand access before you commit

These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.

Compare curated datasets

Allen Brain Atlas

Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.

allen-brain-atlas
neuroscienceopenDocumentation, web resources, and API references are available online.
Last verified 2026-04-30Open record

Annotated Behaviour and Observability Dataset (ABODe)

ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.

abode-home-cage
behavioral-neuroscienceopenThe DataShare record exposes download links for annotations, documentation, license text, and the zipped per-snippet data directory.
Last verified 2026-04-30Open record

DANDI Archive for NWB datasets

DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.

dandi-nwb
electrophysiologyopenPublished Dandiset metadata and archive endpoints are available through the production DANDI API.
Last verified 2026-04-30Open record

International Brain Laboratory public data

The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.

ibl
behavioral-neuroscienceopenPublic sessions can be searched and loaded from the IBL public data server through ONE.
Last verified 2026-04-29Open record

OpenNeuro

OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.

openneuro
neuroscienceopenPublished datasets are available on demand over the internet.
Last verified 2026-04-29Open record

ScienceDex for faculty

Your academic CV and NIH biosketch

Make your CV in your school's format and your NIH biosketch from it, then download both as Word files.