Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
61
datasets available to search
ShareScore release 0.9.0
Dataset results
61 results for “web archive”
Parti Québécois (PQ) web archive collection derivatives
<p>Web archive derivatives of the Parti Québécois (PQ) collection from the <a href="https://www.banq.qc.ca/accueil/">Bibliothèque et Archives nationales du Québec</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a>. Merci beaucoup BAnQ!</p> <p>These derivatives are in the <a href="https://parquet.apache.org/">Apache Parquet format</a>, which is a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/parquet_pandas_example.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary Analysis</strong></a></p> <ul> <li>Audio</li> <li>Images</li> <li>PDFs</li> <li>Presentation program files</li> <li>Spreadsheets</li> <li>Text files</li> <li>Word processor files</li> </ul>
Rare Book and Manuscript Library web archive collection derivatives
<p>Web archive derivatives of the <a href="https://archive-it.org/collections/2766">Rare Book and Manuscript Library</a> collection from <a href="https://archive-it.org/home/Columbia">Columbia University Libraries</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a> and <a href="https://cloud.archivesunleashed.org/">Archives Unleashed Cloud</a>.</p> <p>The <strong>cul-2766-parquet.tar.gz</strong> derivatives are in the <a href="https://parquet.apache.org/">Apache Parquet format</a>, which is a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/datathon-nyc/parquet_pandas_stonewall.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary Analysis</strong></a></p> <ul> <li>Images</li> <li>PDFs</li> <li>Presentation program files</li> <li>Spreadsheets</li> <li>Word processor files<br> </li> </ul> <p>The <strong>cul-2766-auk.tar.gz </strong>derivatives<strong> </strong>are the <a href="https://cloud.archivesunleashed.org/derivatives">standard set of web archive derivatives</a> produced by the Archives Unleashed Cloud.</p> <ul> <li><strong>Gephi </strong>file, which can be loaded into <a href="https://gephi.org/">Gephi</a>. It will have basic characteristics already computed and a basic layout.</li> <li><strong>Raw Network</strong> file, which can also be loaded into <a href="https://gephi.org/">Gephi</a>. You will have to use that network program to lay it out yourself.</li> <li><strong>Full text</strong> file. In it, each website within the web archive collection will have its full text presented on one line, along with information around when it was crawled, the name of the domain, and the full URL of the content.</li> <li><strong>Domains count</strong> file. A text file containing the frequency count of domains captured within your web archive.</li> </ul>
Burke Library New York City Religions web archive collection derivatives
<p>Web archive derivatives of the <a href="https://archive-it.org/collections/1945">Burke Library New York City Religions</a> collection from <a href="https://archive-it.org/home/Columbia">Columbia University Libraries</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a> and <a href="https://cloud.archivesunleashed.org/">Archives Unleashed Cloud</a>.</p> <p>The <strong>cul-1945-parquet.tar.gz</strong> derivatives are in the <a href="https://parquet.apache.org/">Apache Parquet format</a>, which is a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/datathon-nyc/parquet_pandas_stonewall.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary Analysis</strong></a></p> <ul> <li>Images</li> <li>PDFs</li> <li>Presentation program files</li> <li>Spreadsheets</li> <li>Text files</li> <li>Word processor files<br> </li> </ul> <p>The <strong>cul-1945-auk.tar.gz </strong>derivatives<strong> </strong>are the <a href="https://cloud.archivesunleashed.org/derivatives">standard set of web archive derivatives</a> produced by the Archives Unleashed Cloud.</p> <ul> <li><strong>Gephi </strong>file, which can be loaded into <a href="https://gephi.org/">Gephi</a>. It will have basic characteristics already computed and a basic layout.</li> <li><strong>Raw Network</strong> file, which can also be loaded into <a href="https://gephi.org/">Gephi</a>. You will have to use that network program to lay it out yourself.</li> <li><strong>Full text</strong> file. In it, each website within the web archive collection will have its full text presented on one line, along with information around when it was crawled, the name of the domain, and the full URL of the content.</li> <li><strong>Domains count</strong> file. A text file containing the frequency count of domains captured within your web archive.</li> </ul>
Interviews on Current Practices for Describing and Providing Access to UK Public Sector Web Archives
<p>This dataset contains qualitative interview data which investigated current practice for describing and providing access to UK Public Sector Web Archives. Participants included staff responsible for the management and curation of the following web archives:</p><ul><li>UK Web Archive (four of the six Legal Deposit libraries: the British Library, Bodleian Libraries, Cambridge University Library, and the National Library of Scotland)</li><li>UK Government Web Archive (The National Archives)</li><li>UK Parliament Web Archive (Parliamentary Archives)</li><li>NRS Web Archive (National Records of Scotland) and</li><li>PRONI Web Archive (Public Record Office Northern Ireland).</li></ul><p>Available to the public are the University of Dundee (UoD) ethics application for this study, including the research data management plan and information provided to organisations before participating in the study. The report of interview codes and code groups (the 'Codebook') demonstrates the connections made across responses. This is supplemented by a redacted report of quotations by code, organised by code group and document.</p><p>This qualitative interview data, and subsequent analysis, forms the basis of the Masters thesis 'Web Archives for All? Towards Equitable Access to UK Public Sector Web Archives' submitted as part of the MLitt Archives and Records Management at the University of Dundee. </p>
Figures. Exploring the Archived Web in a Highly Transformative Age. Proceedings. dir. S.Gebeil & J.-C. Peyssard
<p>Given recent global crises, the imperative to preserve and analyze online content has never been more vital to enhancing our comprehension of contemporary changes. This book, the outcome of an 5th international RESAW conference that convened experts from 50 disciplines across 17 countries in Marseille in June 2023, tackles the multifaceted challenges of web archiving. It underscores the dual roles of web archiving, as a cultural heritage and as essential source material for researchers delving into contemporary events and the evolution of digital culture. Through 20 chapters, it explores the development of web archiving and examines how technical, cultural, geopolitical, societal and environmental shifts impact its conception, study and dissemination. </p> <p> </p>
Mining the UK Web Archive for Semantic Change Detection (Dataset)
<p>The dataset that was used and released with the RANLP 2019 paper, titled "Mining the UK Web Archive for Semantic Change Detection" (see <a href="https://github.com/adtsakal/Semantic_Change">https://github.com/adtsakal/Semantic_Change</a>). It contains annual word2vec representations of more than 47K words over the period 2000-2013, along with a list of 65 words with known semantic change over the same time period. </p>
Aesthetic Trends and Semantic Web Adoption of Media Outlets Identified through Automated Archival Data Extraction
<p>This dataset includes a variety of structured data gathered via various Web data extraction techniques which were employed in order to collect current and archival data from almost a thousand news websites that are popular in Greece, for the purpose of monitoring and recording their progress through time. The collected information, that took the form of a website’s source code and an impression of their homepage in different time instances of the last decade, has been used to identify trends concerning Semantic Web integration, DOM structure complexity, number of graphics, color usage and more. In total more than ten thousands impressions (including screenshots and source code) were analyzed which resulted to conclusions regarding the evolution of aesthetics and the adoption of new technologies.</p>
Figures - Semantic analysis of web archive historical data 1983 "Marche pour l'égalité et contre le racisme"
Open the record for dataset details and reuse information.
Ditte Laursen: Developing a legal agreement for research-access to a web archive
<p>Ditte Laursen: Developing a legal agreement for research-access to a web archive</p> <p>WARcnet Luxembourg meeting Thursday 5 November 2020</p>
Monday 4 May: Web archives as a critical method for the future of digital research, Matthew Weber
<p>Monday 4 May: Web archives as a critical method for the future of digital research, Matthew Weber</p> <p>Matthew S. Weber, Associate Professor in the Hubbard School of Journalism and Mass Communication at the University of Minnesota</p>
Tuesday 5 May: The promise of web archiving in Belgium, Eveline Vlassenroot, imec-mict-UGent Belgium
<p>Tuesday 5 May: The promise of web archiving in Belgium, Eveline Vlassenroot, imec-mict-UGent Belgium</p>
Tuesday 5 May: You shouldn't need to be a web historian to use web archives, Ian Milligan U Waterloo
<p>Keynote: You shouldn’t need to be a web historian to use web archives: Lowering barriers to access through community and infrastructure, Ian Milligan, Associate Professor, Department of History at University of Waterloo</p>
Wednesday 6 May: Engaging Students in web archival research, Max Odsbjerg Pedersen, Royal Danish Lib
<p>Wednesday 6 May: Engaging Students in web archival research, Max Odsbjerg Pedersen, Royal Danish Lib</p>
Ministry of Environment of Québec (2011-2014) web archive collection derivatives
<p>Web archive derivatives of the Ministry of Environment of Québec (2011-2014) collection from the <a href="https://www.banq.qc.ca/accueil/">Bibliothèque et Archives nationales du Québec</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a>. Merci beaucoup BAnQ!</p> <p>These derivatives are in the <a href="https://parquet.apache.org/">Apache Parquet format</a>, which is a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/parquet_pandas_example.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary Analysis</strong></a></p> <ul> <li>Audio</li> <li>Images</li> <li>PDFs</li> <li>Presentation program files</li> <li>Spreadsheets</li> <li>Text files</li> <li>Videos</li> <li>Word processor files</li> </ul>
Stonewall 50 Commemoration web archive collection derivatives
<p>Web archive derivatives of the <a href="https://archive-it.org/collections/12143">Stonewall 50 Commemoration</a> collection from <a href="https://archive-it.org/home/Columbia">Columbia University Libraries</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a> and <a href="https://cloud.archivesunleashed.org/">Archives Unleashed Cloud</a>.</p> <p>The <strong>cul-12143-parquet.tar.gz</strong> derivatives are in the <a href="https://parquet.apache.org/">Apache Parquet format</a>, which is a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/datathon-nyc/parquet_pandas_stonewall.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary Analysis</strong></a></p> <ul> <li>Audio</li> <li>Images</li> <li>PDFs</li> <li>Presentation program files</li> <li>Spreadsheets</li> <li>Text files</li> <li>Word processor files<br> </li> </ul> <p>The <strong>cul-12143-auk.tar.gz </strong>derivatives<strong> </strong>are the <a href="https://cloud.archivesunleashed.org/derivatives">standard set of web archive derivatives</a> produced by the Archives Unleashed Cloud.</p> <ul> <li><strong>Gephi </strong>file, which can be loaded into <a href="https://gephi.org/">Gephi</a>. It will have basic characteristics already computed and a basic layout.</li> <li><strong>Raw Network</strong> file, which can also be loaded into <a href="https://gephi.org/">Gephi</a>. You will have to use that network program to lay it out yourself.</li> <li><strong>Full text</strong> file. In it, each website within the web archive collection will have its full text presented on one line, along with information around when it was crawled, the name of the domain, and the full URL of the content.</li> <li><strong>Domains count</strong> file. A text file containing the frequency count of domains captured within your web archive.</li> </ul>
Latin American and Caribbean Contemporary Art Web Archive collection derivatives
<p>Web archive derivatives of the <a href="https://archive-it.org/collections/11576">Latin American and Caribbean Contemporary Art Web Archive collection</a> collection from the <a href="https://archive-it.org/home/IvyPlus">Ivy Plus Libraries Confederation</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a> and <a href="https://cloud.archivesunleashed.org/">Archives Unleashed Cloud</a>.</p> <p>The <strong>ivy-11576-parquet.tar.gz</strong> derivatives are in the <a href="https://parquet.apache.org/">Apache Parquet format</a>, which is a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/datathon-nyc/parquet_pandas_stonewall.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary Analysis</strong></a></p> <ul> <li>Audio</li> <li>Images</li> <li>PDFs</li> <li>Presentation program files</li> <li>Spreadsheets</li> <li>Text files</li> <li>Word processor files<br> </li> </ul> <p>The <strong>ivy-11576-auk.tar.gz </strong>derivatives<strong> </strong>are the <a href="https://cloud.archivesunleashed.org/derivatives">standard set of web archive derivatives</a> produced by the Archives Unleashed Cloud.</p> <ul> <li><strong>Gephi </strong>file, which can be loaded into <a href="https://gephi.org/">Gephi</a>. It will have basic characteristics already computed and a basic layout.</li> <li><strong>Raw Network</strong> file, which can also be loaded into <a href="https://gephi.org/">Gephi</a>. You will have to use that network program to lay it out yourself.</li> <li><strong>Full text</strong> file. In it, each website within the web archive collection will have its full text presented on one line, along with information around when it was crawled, the name of the domain, and the full URL of the content.</li> <li><strong>Domains count</strong> file. A text file containing the frequency count of domains captured within your web archive.</li> </ul>
Eastern Europe and Former Soviet Union Web Archive collection derivatives
<p>Web archive derivatives of the <a href="https://archive-it.org/collections/11360">Eastern Europe and Former Soviet Union Web Archive</a> collection from the <a href="https://archive-it.org/home/IvyPlus">Ivy Plus Libraries Confederation</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a> and <a href="https://cloud.archivesunleashed.org/">Archives Unleashed Cloud</a>.</p> <p>The <strong>ivy-11360-parquet.tar.gz</strong> derivatives are in the <a href="https://parquet.apache.org/">Apache Parquet format</a>, which is a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/datathon-nyc/parquet_pandas_stonewall.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary Analysis</strong></a></p> <ul> <li>Audio</li> <li>Images</li> <li>PDFs</li> <li>Presentation program files</li> <li>Spreadsheets</li> <li>Text files</li> <li>Word processor files<br> </li> </ul> <p>The <strong>ivy-11360-auk.tar.gz </strong>derivatives<strong> </strong>are the <a href="https://cloud.archivesunleashed.org/derivatives">standard set of web archive derivatives</a> produced by the Archives Unleashed Cloud.</p> <ul> <li><strong>Gephi </strong>file, which can be loaded into <a href="https://gephi.org/">Gephi</a>. It will have basic characteristics already computed and a basic layout.</li> <li><strong>Raw Network</strong> file, which can also be loaded into <a href="https://gephi.org/">Gephi</a>. You will have to use that network program to lay it out yourself.</li> <li><strong>Full text</strong> file. In it, each website within the web archive collection will have its full text presented on one line, along with information around when it was crawled, the name of the domain, and the full URL of the content.</li> <li><strong>Domains count</strong> file. A text file containing the frequency count of domains captured within your web archive.</li> </ul>
Extreme Right Movements in Europe web archive collection derivatives
<p>Web archive derivatives of the <a href="https://archive-it.org/collections/12172">Literary Authors from Europe and Eurasia Web Archive</a> collection from the <a href="https://archive-it.org/home/IvyPlus">Ivy Plus Libraries Confederation</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a> and <a href="https://cloud.archivesunleashed.org/">Archives Unleashed Cloud</a>.</p> <p>The <strong>ivy-11670-parquet.tar.gz</strong> derivatives are in the <a href="https://parquet.apache.org/">Apache Parquet format</a>, which is a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/datathon-nyc/parquet_pandas_stonewall.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary Analysis</strong></a></p> <ul> <li>Audio</li> <li>Images</li> <li>PDFs</li> <li>Presentation program files</li> <li>Spreadsheets</li> <li>Text files</li> <li>Word processor files<br> </li> </ul> <p>The <strong>ivy-11670-auk.tar.gz </strong>derivatives<strong> </strong>are the <a href="https://cloud.archivesunleashed.org/derivatives">standard set of web archive derivatives</a> produced by the Archives Unleashed Cloud.</p> <ul> <li><strong>Gephi </strong>file, which can be loaded into <a href="https://gephi.org/">Gephi</a>. It will have basic characteristics already computed and a basic layout.</li> <li><strong>Raw Network</strong> file, which can also be loaded into <a href="https://gephi.org/">Gephi</a>. You will have to use that network program to lay it out yourself.</li> <li><strong>Full text</strong> file. In it, each website within the web archive collection will have its full text presented on one line, along with information around when it was crawled, the name of the domain, and the full URL of the content.</li> <li><strong>Domains count</strong> file. A text file containing the frequency count of domains captured within your web archive.</li> </ul>
National Statistical Offices and Central Banks Web Archive collection derivatives
<p>Web archive derivatives of the <a href="https://archive-it.org/collections/10637">National Statistical Offices and Central Banks Web Archive</a> collection from the <a href="https://archive-it.org/home/IvyPlus">Ivy Plus Libraries Confederation</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a> and <a href="https://cloud.archivesunleashed.org/">Archives Unleashed Cloud</a>.</p> <p>The <strong>ivy-10637-parquet.tar.gz</strong> derivatives are in the <a href="https://parquet.apache.org/">Apache Parquet format</a>, which is a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/datathon-nyc/parquet_pandas_stonewall.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary Analysis</strong></a></p> <ul> <li>Audio</li> <li>Images</li> <li>PDFs</li> <li>Presentation program files</li> <li>Spreadsheets</li> <li>Text files</li> <li>Word processor files<br> </li> </ul> <p>The <strong>ivy-10637-auk.tar.gz </strong>derivatives<strong> </strong>are the <a href="https://cloud.archivesunleashed.org/derivatives">standard set of web archive derivatives</a> produced by the Archives Unleashed Cloud.</p> <ul> <li><strong>Gephi </strong>file, which can be loaded into <a href="https://gephi.org/">Gephi</a>. It will have basic characteristics already computed and a basic layout.</li> <li><strong>Raw Network</strong> file, which can also be loaded into <a href="https://gephi.org/">Gephi</a>. You will have to use that network program to lay it out yourself.</li> <li><strong>Full text</strong> file. In it, each website within the web archive collection will have its full text presented on one line, along with information around when it was crawled, the name of the domain, and the full URL of the content.</li> <li><strong>Domains count</strong> file. A text file containing the frequency count of domains captured within your web archive.</li> </ul>
#MeToo and the Women's Rights Movement in China Web Archive collection derivatives
<p>Web archive derivatives of the <a href="https://archive-it.org/collections/12172">Literary Authors from Europe and Eurasia Web Archive</a> collection from the <a href="https://archive-it.org/home/IvyPlus">Ivy Plus Libraries Confederation</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a> and <a href="https://cloud.archivesunleashed.org/">Archives Unleashed Cloud</a>.</p> <p>The <strong>ivy-12145-parquet.tar.gz</strong> derivatives are in the <a href="https://parquet.apache.org/">Apache Parquet format</a>, which is a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/datathon-nyc/parquet_pandas_stonewall.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary Analysis</strong></a></p> <ul> <li>Audio</li> <li>Images</li> <li>PDFs</li> <li>Presentation program files</li> <li>Spreadsheets</li> <li>Text files</li> <li>Word processor files<br> </li> </ul> <p>The <strong>ivy-12145-auk.tar.gz </strong>derivatives<strong> </strong>are the <a href="https://cloud.archivesunleashed.org/derivatives">standard set of web archive derivatives</a> produced by the Archives Unleashed Cloud.</p> <ul> <li><strong>Gephi </strong>file, which can be loaded into <a href="https://gephi.org/">Gephi</a>. It will have basic characteristics already computed and a basic layout.</li> <li><strong>Raw Network</strong> file, which can also be loaded into <a href="https://gephi.org/">Gephi</a>. You will have to use that network program to lay it out yourself.</li> <li><strong>Full text</strong> file. In it, each website within the web archive collection will have its full text presented on one line, along with information around when it was crawled, the name of the domain, and the full URL of the content.</li> <li><strong>Domains count</strong> file. A text file containing the frequency count of domains captured within your web archive.</li> </ul>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.