Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
1,298
datasets available to search
ShareScore release 0.9.0
Dataset results
1,298 results for “Archive”
LTER Epibenthos Sampling Data for North Inlet Estuary, Georgetown, South Carolina from 1981 to 1992, North Inlet LTER (Reformatted to a Darwin Core Archive)
This data package is formatted as a Darwin Core Archive (DwC-A, event core). For more information on Darwin Core see https://www.tdwg.org/standards/dwc/. This Level 2 data package was derived from the Level 1 data package found here: https://pasta.lternet.edu/package/metadata/eml/edi/354/2, which was derived from the Level 0 data package found here: https://pasta.lternet.edu/package/metadata/eml/knb-lter-nin/7/1. The abstract below was extracted from the Level 0 data package and is included for context: This data package consists of Epibenthos Sampling for North Inlet Stations Bread and Butter Creek, from 1981 to 1992, and Debidue Creek from 1981 to 1984, The purpose of the long term monitoring of Epibenthos was to determine seasonal and inter-annual changes in the taxonomic/life stage composition and abundance of small motile epibenthic invertebrates and fishes (1-20 mm in length) in the major sub- tidal habitats of North Inlet estuary.
Macrobenthos Sampling data for the North Inlet Estuary, Georgetown,South Carolina, from 1981 to 1992 North Inlet LTER (Reformatted to a Darwin Core Archive)
This data package is formatted as a Darwin Core Archive (DwC-A, event core). For more information on Darwin Core see https://www.tdwg.org/standards/dwc/. This Level 2 data package was derived from the Level 1 data package found here: https://pasta.lternet.edu/package/metadata/eml/edi/353/2, which was derived from the Level 0 data package found here: https://pasta.lternet.edu/package/metadata/eml/knb-lter-nin/9/1. The abstract below was extracted from the Level 0 data package and is included for context: This data package consists of Macrobenthos Sampling Data for North Inlet Stations Bread and Butter Creek from 1981 to 1992, and Debidue Creek from 1981 to 1984, North Inlet LTER. The purpose of this study was to document the composition and abundance of macrobenthic subtidal populations over time at one mud and one sand site. Macrobenthos was defined here as those animals retained on a 0.5 mm mesh screen.
Long Term Mammal Data from Powdermill Biological Station 1979-1999 (Reformatted to a Darwin Core Archive)
This data package is formatted as a Darwin Core Archive (DwC-A, event core). For more information on Darwin Core see https://www.tdwg.org/standards/dwc/. This Level 2 data package was derived from the Level 1 data package found here: https://pasta.lternet.edu/package/metadata/eml/edi/356/2, which was derived from the Level 0 data package found here: https://pasta.lternet.edu/package/metadata/eml/knb-lter-vcr/67/20. The abstract below was extracted from the Level 0 data package and is included for context: This is a 20-year record of small mammal trapping from the Powdermill Biological Station, Rector, PA 15677 collected by Joseph F. Merritt. It is included here as a comparative source of small mammal data.
Archival records of fire history, 1910-1977, central western Cascades, Oregon
Historical wild fire records for the central western Cascades of Oregon are summarized here for the period from 1910 to 1977. Data records are obtained and summarized from historical statistical reports that were generated by the U.S. Forest Service and exist in various forms including fire maps, summary tables, and individual fire reports. The location, cause of the fire, its size class, and the source of information regarding each fire are included.
VCRLTER-Northampton County GIS data archive, 1995.
This data archive is a collection of GIS files and FGDC metadata prepared in 1995 for the Northampton County Planning Office by the Virginia Coast Reserve LTER project at the University of Virginia with support from the Virginia Department of Environmental Quality (DEQ) and the National Science Foundation (NSF). Original data sources include:  1:100,000-scale USGS digital line graph (DLG) hydrography and transportation data;  1:6,000-scale boundary, road, and railroad data for the town of Cape Charles from VDOT; 1:190,000-scale county-wide general soil map data and 1:15,540-scale detailed soil data for the Cape Charles area digitized from printed USDA soil survey maps;  a land use and vegetation cover dataset (30 m. resolution) created by the VCRLTER derived from a 1993 Landsat Thematic Mapper satellite image;  1:20,000-scale plant association maps for 10 seaside barrier and marsh islands between Hog and Smith Islands, inclusive, prepared by Cheryl McCaffrey for TNC in 1975 and published in the Virginia Journal of Science in 1990; and 1993 colonial bird nesting site data collected by The Center for Conservation Biology (with partners The Nature Conservancy, College of William and Mary, University of Virginia, USFWS, VA-DCR, and VA-DGIF). Contents: HYDROGRAPHY Based on USGS 1:100,000 Digital Line Graph (DLG) data. Files: h100k_arc_u84 (streams, shorelines, etc.) and h100k_poly_u84 (marshes, mudflats, etc.). Note that the hydrographic data has been superseded by the more recent and more detailed USGS National Hydrography Dataset, available for the entire state of Virginia at "ftp://nhdftp.usgs.gov/DataSets/Staged/States/FileGDB/HighResolution/NHDH_VA_931v210.zip" (see http://nhd.usgs.gov/data.html for more information). A static 2013 version of the NHD data that includes shapefiles extracted from the original ESRI geodatabase format data and covering just the watersheds of the Eastern Shore of VA can also be found in the VCRLTER Data Catalog (dataset VCR1422
FEW-meter project data archive 07 Jan 2020
<p>Data collected in the FEW-meter project for the first growing season (2019) from all case studies. The state as of January 07, 2020.</p>
Metadata for the archive of JPG image files of Herbarium specimens from Columbia used in the BRAVO project.
<p>An excel spreadsheet of the metadata to accompany the images in the file "Archive of JPG image files of Herbarium specimens from Columbia used in the BRAVO project".</p> <p>The metadata is information like (Family, genus, names, collector, coll_num, plot number, & location, identification of the material...)</p> <p>A majority of the metadata is also included on the "label" on the specimens in the images in the archive.</p>
An archive of data from Resonant Column and Cyclic Torsional Shear Tests performed on Italian Clays
<p>A large data-set of index and dynamic parameters measured from resonant column (RC) and cyclic torsional shear(CTS) tests on 170 undisturbed isotropically consolidated fine-grained specimens deriving from 90 sites in Central and Northern Italy is made available. Tests were all performed over the past 20 years at the Geotechnical Laboratory of the Civil and Environmental Engineering Department of the Florence University using the same apparatus and following the same standardized procedures.</p> <p>The experimental data are organized in an excel file (named as “Italian_Clays_Archive.xlsx”). For each tested sample, the main physical, index and dynamic properties measured are archived with the code number of the sample (No) in the sheet named as “Dataset” as well as any information available about the borehole from which the sample has been taken. The list and the meaning of the symbols used can be found in the sheet named as “Legend”. Other sheets containing borehole stratigraphy are named as “XX-ST” (where “XX” stands as the bore-hole code, BH) and they can be recalled directly from the “Dataset” sheet. Note that stratigraphy is given in its original format, when available. However, depth and thickness of each layer can be easily deduced by the figure provided and the soil lithology is well represented by the symbol used that are those generally adopted internationally. Finally, the sheets named as "YY-CTS-STEPZ" (where “YY” and “Z” stand as the sample code, No, and the step number, respectively) contain the shear stress and strain values measured after CTS tests at different steps (i.e. amplitudes of the cyclic dynamic torsional loading applied) during the 1st, 5th, 15th, 20th and 25<sup>th</sup>.and/or and/or the corresponding shear modulus and damping ratio calculated from the same cycles.</p> <p>The selected samples were taken mostly in Holocene and Pleistocene fluvio-lacustrine soil deposits at depths ranging from 1 m to 75 m below ground level and they mainly consist of normally and over-consolidated clayey silts or clays (1 < OCR < 9.4) of medium-to-high plasticity (4 < PI < 84), with very low-to high consistency (-1< Ic < 1.9) and initial void ratio, e<sub>0</sub>, ranging between 0.175 and 2.456. The database also includes some samples of organic clays of low consistency, very high water content and void ratio and low unit weight. The initial (small strain) values of shear modulus, G<sub>0</sub>, and damping ratio, D<sub>0</sub>, range between 21 MPa and 292 MPa and between 0.8% and 5.1%, respectively. The smallest and the largest shear strain values induced by RC and CTS tests are 1.9x10<sup>-5</sup> % and 6.3x10<sup>-1</sup>%, respectively.</p>
Web Archive of Independent News Sites on Turkish Affairs derivatives
<p>Derivatives of the <a href="https://archive-it.org/collections/12911">Web Archive of Independent News Sites on Turkish Affairs</a> collection from the <a href="https://archive-it.org/home/IvyPlus">Ivy Plus Libraries Confederation</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a> and <a href="https://cloud.archivesunleashed.org/">Archives Unleashed Cloud</a>.</p> <p>The <strong>ivy-12911-parquet.tar.gz</strong> derivatives are in the <a href="https://parquet.apache.org/">Apache Parquet format</a>, which is a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/datathon-nyc/parquet_pandas_stonewall.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary Analysis</strong></a></p> <ul> <li>Audio</li> <li>Images</li> <li>PDFs</li> <li>Presentation program files</li> <li>Spreadsheets</li> <li>Text files</li> <li>Word processor files<br> </li> </ul> <p>The <strong>ivy-12911-auk.tar.gz </strong>derivatives<strong> </strong>are the <a href="https://cloud.archivesunleashed.org/derivatives">standard set of web archive derivatives</a> produced by the Archives Unleashed Cloud.</p> <ul> <li><strong>Gephi </strong>file, which can be loaded into <a href="https://gephi.org/">Gephi</a>. It will have basic characteristics already computed and a basic layout.</li> <li><strong>Raw Network</strong> file, which can also be loaded into <a href="https://gephi.org/">Gephi</a>. You will have to use that network program to lay it out yourself.</li> <li><strong>Full text</strong> file. In it, each website within the web archive collection will have its full text presented on one line, along with information around when it was crawled, the name of the domain, and the full URL of the content.</li> <li><strong>Domains count</strong> file. A text file containing the frequency count of domains captured within your web archive.</li> </ul>
Popline and K4Health Web Archive collection derivatives
<p>Web archive derivatives of the <a href="https://archive-it.org/collections/12006">Popline and K4Health Web Archive</a> collection from the <a href="https://archive-it.org/home/IvyPlus">Ivy Plus Libraries Confederation</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a> and <a href="https://cloud.archivesunleashed.org/">Archives Unleashed Cloud</a>.</p> <p>The <strong>ivy-12006-parquet.tar.gz</strong> derivatives are in the <a href="https://parquet.apache.org/">Apache Parquet format</a>, which is a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/datathon-nyc/parquet_pandas_stonewall.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary Analysis</strong></a></p> <ul> <li>Audio</li> <li>Images</li> <li>PDFs</li> <li>Presentation program files</li> <li>Spreadsheets</li> <li>Text files</li> <li>Word processor files<br> </li> </ul> <p>The <strong>ivy-12006-auk.tar.gz </strong>derivatives<strong> </strong>are the <a href="https://cloud.archivesunleashed.org/derivatives">standard set of web archive derivatives</a> produced by the Archives Unleashed Cloud.</p> <ul> <li><strong>Gephi </strong>file, which can be loaded into <a href="https://gephi.org/">Gephi</a>. It will have basic characteristics already computed and a basic layout.</li> <li><strong>Raw Network</strong> file, which can also be loaded into <a href="https://gephi.org/">Gephi</a>. You will have to use that network program to lay it out yourself.</li> <li><strong>Full text</strong> file. In it, each website within the web archive collection will have its full text presented on one line, along with information around when it was crawled, the name of the domain, and the full URL of the content.</li> <li><strong>Domains count</strong> file. A text file containing the frequency count of domains captured within your web archive.</li> </ul>
Independent Documentary Filmmakers from China, Hong Kong, and Taiwan Web Archive collection derivatives
<p>Web archive derivatives of the <a href="https://archive-it.org/collections/12172">Literary Authors from Europe and Eurasia Web Archive</a> collection from the <a href="https://archive-it.org/home/IvyPlus">Ivy Plus Libraries Confederation</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a> and <a href="https://cloud.archivesunleashed.org/">Archives Unleashed Cloud</a>.</p> <p>The <strong>ivy-12126-parquet.tar.gz</strong> derivatives are in the <a href="https://parquet.apache.org/">Apache Parquet format</a>, which is a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/datathon-nyc/parquet_pandas_stonewall.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary Analysis</strong></a></p> <ul> <li>Audio</li> <li>Images</li> <li>PDFs</li> <li>Presentation program files</li> <li>Spreadsheets</li> <li>Text files</li> <li>Word processor files<br> </li> </ul> <p>The <strong>ivy-12126-auk.tar.gz </strong>derivatives<strong> </strong>are the <a href="https://cloud.archivesunleashed.org/derivatives">standard set of web archive derivatives</a> produced by the Archives Unleashed Cloud.</p> <ul> <li><strong>Gephi </strong>file, which can be loaded into <a href="https://gephi.org/">Gephi</a>. It will have basic characteristics already computed and a basic layout.</li> <li><strong>Raw Network</strong> file, which can also be loaded into <a href="https://gephi.org/">Gephi</a>. You will have to use that network program to lay it out yourself.</li> <li><strong>Full text</strong> file. In it, each website within the web archive collection will have its full text presented on one line, along with information around when it was crawled, the name of the domain, and the full URL of the content.</li> <li><strong>Domains count</strong> file. A text file containing the frequency count of domains captured within your web archive.</li> </ul>
Literary Authors from Europe and Eurasia Web Archive collection derivatives
<p>Web archive derivatives of the <a href="https://archive-it.org/collections/12172">Literary Authors from Europe and Eurasia Web Archive</a> collection from the <a href="https://archive-it.org/home/IvyPlus">Ivy Plus Libraries Confederation</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a> and <a href="https://cloud.archivesunleashed.org/">Archives Unleashed Cloud</a>.</p> <p>The <strong>ivy-12172-parquet.tar.gz</strong> derivatives are in the <a href="https://parquet.apache.org/">Apache Parquet format</a>, which is a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/datathon-nyc/parquet_pandas_stonewall.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary Analysis</strong></a></p> <ul> <li>Audio</li> <li>Images</li> <li>PDFs</li> <li>Presentation program files</li> <li>Spreadsheets</li> <li>Text files</li> <li>Word processor files<br> </li> </ul> <p>The <strong>ivy-12172-auk.tar.gz </strong>derivatives<strong> </strong>are the <a href="https://cloud.archivesunleashed.org/derivatives">standard set of web archive derivatives</a> produced by the Archives Unleashed Cloud.</p> <ul> <li><strong>Gephi </strong>file, which can be loaded into <a href="https://gephi.org/">Gephi</a>. It will have basic characteristics already computed and a basic layout.</li> <li><strong>Raw Network</strong> file, which can also be loaded into <a href="https://gephi.org/">Gephi</a>. You will have to use that network program to lay it out yourself.</li> <li><strong>Full text</strong> file. In it, each website within the web archive collection will have its full text presented on one line, along with information around when it was crawled, the name of the domain, and the full URL of the content.</li> <li><strong>Domains count</strong> file. A text file containing the frequency count of domains captured within your web archive.</li> </ul>
Queer Japan Web Archive collection derivatives
<p>Web archive derivatives of the <a href="https://archive-it.org/collections/11854">Queer Japan Web Archive</a> collection from the <a href="https://archive-it.org/home/IvyPlus">Ivy Plus Libraries Confederation</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a> and <a href="https://cloud.archivesunleashed.org/">Archives Unleashed Cloud</a>.</p> <p>The <strong>ivy-12172-parquet.tar.gz</strong> derivatives are in the <a href="https://parquet.apache.org/">Apache Parquet format</a>, which is a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/datathon-nyc/parquet_pandas_stonewall.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary Analysis</strong></a></p> <ul> <li>Audio</li> <li>Images</li> <li>PDFs</li> <li>Presentation program files</li> <li>Spreadsheets</li> <li>Text files</li> <li>Videos</li> <li>Word processor files<br> </li> </ul> <p>The <strong>ivy-11854-auk.tar.gz </strong>derivatives<strong> </strong>are the <a href="https://cloud.archivesunleashed.org/derivatives">standard set of web archive derivatives</a> produced by the Archives Unleashed Cloud.</p> <ul> <li><strong>Gephi </strong>file, which can be loaded into <a href="https://gephi.org/">Gephi</a>. It will have basic characteristics already computed and a basic layout.</li> <li><strong>Raw Network</strong> file, which can also be loaded into <a href="https://gephi.org/">Gephi</a>. You will have to use that network program to lay it out yourself.</li> <li><strong>Full text</strong> file. In it, each website within the web archive collection will have its full text presented on one line, along with information around when it was crawled, the name of the domain, and the full URL of the content.</li> <li><strong>Domains count</strong> file. A text file containing the frequency count of domains captured within your web archive.</li> </ul>
General web archive collection derivatives
<p>Web archive derivatives of the <a href="https://archive-it.org/collections/1716">General</a> collection from <a href="https://archive-it.org/home/Columbia">Columbia University Libraries</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a> and <a href="https://cloud.archivesunleashed.org/">Archives Unleashed Cloud</a>.</p> <p>The <strong>cul-1716-parquet.tar.gz</strong> derivatives are in the <a href="https://parquet.apache.org/">Apache Parquet format</a>, which is a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/datathon-nyc/parquet_pandas_stonewall.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary Analysis</strong></a></p> <ul> <li>Audio</li> <li>Images</li> <li>PDFs</li> <li>Presentation program files</li> <li>Spreadsheets</li> <li>Text files</li> <li>Word processor files<br> </li> </ul> <p>The <strong>cul-1716-auk.tar.gz </strong>derivatives<strong> </strong>are the <a href="https://cloud.archivesunleashed.org/derivatives">standard set of web archive derivatives</a> produced by the Archives Unleashed Cloud.</p> <ul> <li><strong>Gephi </strong>file, which can be loaded into <a href="https://gephi.org/">Gephi</a>. It will have basic characteristics already computed and a basic layout.</li> <li><strong>Raw Network</strong> file, which can also be loaded into <a href="https://gephi.org/">Gephi</a>. You will have to use that network program to lay it out yourself.</li> <li><strong>Full text</strong> file. In it, each website within the web archive collection will have its full text presented on one line, along with information around when it was crawled, the name of the domain, and the full URL of the content.</li> <li><strong>Domains count</strong> file. A text file containing the frequency count of domains captured within your web archive.</li> </ul>
Global Webcomics Web Archive collection derivatives
<p>Web archive derivatives of the <a href="https://archive-it.org/collections/10181">Global Webcomics Web Archive</a> collection from the <a href="https://archive-it.org/home/IvyPlus">Ivy Plus Libraries Confederation</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a> and <a href="https://cloud.archivesunleashed.org/">Archives Unleashed Cloud</a>.</p> <p>The <strong>ivy-10181-parquet.tar.gz</strong> derivatives are in the <a href="https://parquet.apache.org/">Apache Parquet format</a>, which is a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/datathon-nyc/parquet_pandas_stonewall.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary Analysis</strong></a></p> <ul> <li>Audio</li> <li>Images</li> <li>PDFs</li> <li>Presentation program files</li> <li>Spreadsheets</li> <li>Text files</li> <li>Word processor files<br> </li> </ul> <p>The <strong>ivy-10181-auk.tar.gz </strong>derivatives<strong> </strong>are the <a href="https://cloud.archivesunleashed.org/derivatives">standard set of web archive derivatives</a> produced by the Archives Unleashed Cloud.</p> <ul> <li><strong>Gephi </strong>file, which can be loaded into <a href="https://gephi.org/">Gephi</a>. It will have basic characteristics already computed and a basic layout.</li> <li><strong>Raw Network</strong> file, which can also be loaded into <a href="https://gephi.org/">Gephi</a>. You will have to use that network program to lay it out yourself.</li> <li><strong>Full text</strong> file. In it, each website within the web archive collection will have its full text presented on one line, along with information around when it was crawled, the name of the domain, and the full URL of the content.</li> <li><strong>Domains count</strong> file. A text file containing the frequency count of domains captured within your web archive.</li> </ul>
Brazilian Presidential Transition (2018) Web Archive collection derivatives
<p>Web archive derivatives of the <a href="https://archive-it.org/collections/11549">Brazilian Presidential Transition (2018) Web Archive collection</a> collection from the <a href="https://archive-it.org/home/IvyPlus">Ivy Plus Libraries Confederation</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a> and <a href="https://cloud.archivesunleashed.org/">Archives Unleashed Cloud</a>.</p> <p>The <strong>ivy-11549-parquet.tar.gz</strong> derivatives are in the <a href="https://parquet.apache.org/">Apache Parquet format</a>, which is a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/datathon-nyc/parquet_pandas_stonewall.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary Analysis</strong></a></p> <ul> <li>Audio</li> <li>Images</li> <li>PDFs</li> <li>Presentation program files</li> <li>Spreadsheets</li> <li>Text files</li> <li>Word processor files<br> </li> </ul> <p>The <strong>ivy-11549-auk.tar.gz </strong>derivatives<strong> </strong>are the <a href="https://cloud.archivesunleashed.org/derivatives">standard set of web archive derivatives</a> produced by the Archives Unleashed Cloud.</p> <ul> <li><strong>Gephi </strong>file, which can be loaded into <a href="https://gephi.org/">Gephi</a>. It will have basic characteristics already computed and a basic layout.</li> <li><strong>Raw Network</strong> file, which can also be loaded into <a href="https://gephi.org/">Gephi</a>. You will have to use that network program to lay it out yourself.</li> <li><strong>Full text</strong> file. In it, each website within the web archive collection will have its full text presented on one line, along with information around when it was crawled, the name of the domain, and the full URL of the content.</li> <li><strong>Domains count</strong> file. A text file containing the frequency count of domains captured within your web archive.</li> </ul>
Supporting information for IUCr survey on raw data archival and reuse in chemical crystallography
<p>Supporting information i.e. questions, responses and raw data, relating to a study into raw data management and availability in small molecule crystallography conducted under the auspices of the International Union of Crystallography Committee on Data.</p> <p>It is now common to deposit structure factors when publishing, which means that the small molecule crystallography community caters very well for routine structures. However this is generally only the case if everything in a raw image is fully and/or properly accounted for and the model is correct or appropriate. So for example, in some cases raw data may no longer be required, while in others it may be necessary to validate or ‘do better’ in the future. Moreover there are increasing pressures from bodies e.g. funders to make the data relating to research outputs Findable, Accessible, Interoperable and Reusable (FAIR). In acknowledgement of this situation and in order to begin addressing it, IUCr Journals now facilitate access to and citation of large raw diffraction datasets in its articles. Therefore it is important for our community understand and define how we manage our raw data in this respect.</p> <p>As Members of the IUCr Committee on Data we see the need to conduct this survey about exploring raw data archival<br> practice and gathering opinions as to if/how raw data could/should be used if it were to be made more widely<br> available.</p>
Collaborative Architecture, Urbanism, and Sustainability Web Archive (CAUSEWAY) collection derivatives
<p>Web archive derivatives of the <a href="https://archive-it.org/collections/4638">Collaborative Architecture, Urbanism, and Sustainability Web Archive (CAUSEWAY)</a> collection from the <a href="https://archive-it.org/home/IvyPlus">Ivy Plus Libraries Confederation</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a> and <a href="https://cloud.archivesunleashed.org/">Archives Unleashed Cloud</a>.</p> <p>The <strong>ivy-4638-parquet.tar.gz</strong> derivatives are in the <a href="https://parquet.apache.org/">Apache Parquet format</a>, which is a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/datathon-nyc/parquet_pandas_stonewall.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary Analysis</strong></a></p> <ul> <li>Audio</li> <li>Images</li> <li>Presentation program files</li> <li>Spreadsheets</li> <li>Text files</li> <li>Word processor files<br> </li> </ul> <p>The <strong>ivy-4638-auk.tar.gz </strong>derivatives<strong> </strong>are the <a href="https://cloud.archivesunleashed.org/derivatives">standard set of web archive derivatives</a> produced by the Archives Unleashed Cloud.</p> <ul> <li><strong>Gephi </strong>file, which can be loaded into <a href="https://gephi.org/">Gephi</a>. It will have basic characteristics already computed and a basic layout.</li> <li><strong>Raw Network</strong> file, which can also be loaded into <a href="https://gephi.org/">Gephi</a>. You will have to use that network program to lay it out yourself.</li> <li><strong>Full text</strong> file. In it, each website within the web archive collection will have its full text presented on one line, along with information around when it was crawled, the name of the domain, and the full URL of the content.</li> <li><strong>Domains count</strong> file. A text file containing the frequency count of domains captured within your web archive.</li> </ul>
Quebec Ministry of Agriculture, Fisheries and Food from 2012-2018 web archive collection derivatives
<p>Web archive derivatives of the Quebec Ministry of Agriculture, Fisheries and Food from 2012-2018 collection from the <a href="https://www.banq.qc.ca/accueil/">Bibliothèque et Archives nationales du Québec</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a>. Merci beaucoup BAnQ!</p> <p>These derivatives are in the <a href="https://parquet.apache.org/">Apache Parquet format</a>, which is a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/parquet_pandas_example.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary Analysis</strong></a></p> <ul> <li>Audio</li> <li>Images</li> <li>PDFs</li> <li>Presentation program files</li> <li>Spreadsheets</li> <li>Text files</li> <li>Videos</li> <li>Word processor files</li> </ul>
Sites of the Quebec Ministry of Immigration from 2012 to 2018 web archive collection derivatives
<p>Web archive derivatives of the Sites of the Quebec Ministry of Immigration from 2012 to 2018 collection from the <a href="https://www.banq.qc.ca/accueil/">Bibliothèque et Archives nationales du Québec</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a>. Merci beaucoup BAnQ!</p> <p>These derivatives are in the <a href="https://parquet.apache.org/">Apache Parquet format</a>, which is a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/parquet_pandas_example.ipynb">this</a> notebook for examples.</p> <p><strong>Domains</strong></p> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>domain</li> <li>count</li> </ul> <p><strong>Web Pages</strong></p> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>url</li> <li>mime_type_web_server</li> <li>mime_type_tika</li> <li>content</li> </ul> <p><strong>Web Graph</strong></p> <pre><code class="language-java">.webgraph()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>crawl_date</li> <li>src</li> <li>dest</li> <li>anchor</li> </ul> <p><strong>Image Links</strong></p> <pre><code class="language-java">.imageLinks()</code></pre> <p>Produces a DataFrame with the following columns:</p> <ul> <li>src</li> <li>image_url</li> </ul> <p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary Analysis</strong></a></p> <ul> <li>Audio</li> <li>Images</li> <li>PDFs</li> <li>Presentation program files</li> <li>Spreadsheets</li> <li>Text files</li> <li>Videos</li> <li>Word processor files</li> </ul>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.