Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
104
datasets available to search
ShareScore release 0.7.1
Dataset results
104 results for “Scraping”
Complete Rxivist dataset of scraped biology preprint data
<p><a href="https://rxivist.org">rxivist.org</a> allowed readers to sort and filter the tens of thousands of preprints posted to <a href="https://www.biorxiv.org">bioRxiv</a> and <a href="https://www.medrxiv.org">medRxiv</a>. Rxivist used a custom web crawler to index all papers posted to those two websites; this is a snapshot of Rxivist the production database. The version number indicates the date on which the snapshot was taken. See the included "README.md" file for instructions on how to use the "rxivist.backup" file to import data into a PostgreSQL database server.</p> <p>Please note this is a different repository than the one used for <a href="https://www.biorxiv.org/content/early/2019/01/13/515643">the Rxivist manuscript</a>—that is in <a href="https://doi.org/10.5281/zenodo.2465689">a separate Zenodo repository</a>. You're welcome (and encouraged!) to use this data in your research, but <strong>please cite our paper, now published <a href="https://doi.org/10.7554/eLife.45133">in <em>eLife</em></a>.</strong></p> <p>Previous versions are also available pre-loaded into Docker images, available at <a href="https://hub.docker.com/r/blekhmanlab/rxivist_data">blekhmanlab/rxivist_data</a>.</p> <p><strong>Version notes:</strong></p> <ul> <li><strong>2023-03-01</strong> <ul> <li>The final Rxivist data upload, more than four years after the first and encompassing 223,541 preprints posted to bioRxiv and medRxiv through the end of February 2023.</li> </ul> </li> <li><em><strong>2020-12-07***</strong></em> <ul> <li>In addition to bioRxiv preprints, <em><strong>the database now includes all medRxiv preprints as well</strong></em>. <ul> <li>The website where a preprint was posted is now recorded in a <strong>new field</strong> in the "articles" table, called "<strong>repo</strong>".</li> </ul> </li> <li>We've significantly refactored the web crawler to take advantage of developments with the bioRxiv API. <ul> <li>The main difference is that preprints flagged as "published" by bioRxiv are no longer recorded on the same schedule that download metrics are updated: The Rxivist database should now record published DOI entries the same day bioRxiv detects them.</li> </ul> </li> <li>Twitter metrics have returned, for the most part. Improvements with the Crossref Event Data API mean we can once again tally daily Twitter counts for all bioRxiv DOIs. <ul> <li>The "crossref_daily" table remains where these are recorded, and daily numbers are now up to date.</li> <li>Historical daily counts have also been re-crawled to fill in the empty space that started in October 2019.</li> <li>There are still several gaps that are more than a week long due to missing data from Crossref.</li> <li>We have recorded available Crossref Twitter data for all papers with DOI numbers starting with "10.1101," which includes all medRxiv preprints. However, <strong>there appears to be almost no Twitter data available for medRxiv preprints</strong>.</li> </ul> </li> <li>The download metrics for article id 72514 (DOI 10.1101/2020.01.30.927871) were found to be out of date for February 2020 and are now correct. This is notable because article 72514 is the most downloaded preprint of all time; we're still looking into why this wasn't updated after the month ended.</li> </ul> </li> <li><strong>2020-11-18</strong> <ul> <li>Publication checks should be back on schedule.</li> </ul> </li> <li><strong>2020-10-26</strong> <ul> <li>This snapshot fixes most of the data issues found in the previous version. Indexed papers are now up to date, and download metrics are back on schedule. <em>The check for publication status remains behind schedule</em>, however, and the database may not include published DOIs for papers that have been flagged on bioRxiv as "published" over the last two months. Another snapshot will be posted in the next few weeks with updated publication information.</li> </ul> </li> <li><strong>2020-09-15</strong> <ul> <li>A crawler error caused this snapshot to exclude all papers posted after about August 29, with some papers having download metrics that were more out of date than usual. The "last_crawled" field is accurate.</li> </ul> </li> <li><strong>2020-09-08</strong> <ul> <li>This snapshot is misconfigured and will not work without modification; it has been replaced with version 2020-09-15.</li> </ul> </li> <li><strong>2019-12-27</strong> <ul> <li>Several dozen papers did not have dates associated with them; that has been fixed.</li> <li>Some authors have had two entries in the "authors" table for portions of 2019, one profile that was linked to their ORCID and one that was not, occasionally with almost identical "name" strings. This happened after bioRxiv began changing author names to reflect the names in the PDFs, rather than the ones manually entered into their system. These database records are mostly consolidated now, but some may remain.</li> </ul> </li> <li><strong>2019-11-29</strong> <ul> <li>The Crossref Event Data API remains down; Twitter data is unavailable for dates after early October.</li> </ul> </li> <li><strong>2019-10-31</strong> <ul> <li>The Crossref Event Data API is still <a href="https://status.crossref.org/">experiencing problems</a>; the Twitter data for October is incomplete in this snapshot.</li> <li>The README file has been modified to reflect changes in the process for creating your own DB snapshots if using the newly released PostgreSQL 12.</li> </ul> </li> <li><strong>2019-10-01</strong> <ul> <li>The Crossref API is back online, and the "crossref_daily" table should now include up-to-date tweet information for July through September.</li> <li>About 40,000 authors were removed from the author table because the name had been removed from all preprints they had previously been associated with, likely because their name changed slightly on the bioRxiv website ("John Smith" to "J Smith" or "John M Smith"). The "author_emails" table was also modified to remove entries referring to the deleted authors. The web crawler is being updated to clean these orphaned entries more frequently.</li> </ul> </li> <li><strong>2019-08-30</strong> <ul> <li>The Crossref Event Data API, which provides the data used to populate the table of tweet counts, has not been fully functional since early July. While we are optimistic that accurate tweet counts will be available at some point, the sparse values currently in the "crossref_daily" table for July and August should not be considered reliable.</li> </ul> </li> <li><strong>2019-07-01</strong> <ul> <li>A new "institution" field has been <a href="https://github.com/blekhmanlab/rxivist/commit/1cb570703085841e80cc3073af445bc86f0cbb63#diff-a04b1c1a66d16f9b3acfed9b9d2128c5">added</a> to the "article_authors" table that stores each author's institutional affiliation <em>as listed on that paper</em>. The "authors" table still has each author's most recently observed institution. <ul> <li>We began collecting this data in the middle of May, but it has not been applied to older papers yet.</li> </ul> </li> </ul> </li> <li><strong>2019-05-11</strong> <ul> <li>The README was updated to correct a link to the Docker repository used for the pre-built images.</li> </ul> </li> <li><strong>2019-03-21</strong> <ul> <li>The license for this dataset has been changed to <a href="https://creativecommons.org/licenses/by/4.0/">CC-BY</a>, which allows use for any purpose and requires only attribution.</li> <li>A new table, "publication_dates," has been added and will be continually updated. This table will include an entry for each preprint that has been published externally for which we can determine a date of publication, based on data from Crossref. (This table was previously included in the "paper" schema but was not updated after early December 2018.)</li> <li>Foreign key constraints have been added to almost every table in the database. This should not impact any read behavior, but anyone writing to these tables will encounter constraints on existing fields that refer to other tables. Most frequently, this means the "article" field in a table will need to refer to an ID that actually exists in the "articles" table.</li> <li>The "author_translations" table has been removed. This was used to redirect incoming requests for outdated author profile pages and was likely not of any functional use to others.</li> <li>The "README.md" file has been renamed "1README.md" because Zenodo only displays a preview for the file that appears first in the list alphabetically.</li> <li>The "article_ranks" and "article_ranks_working" tables have been removed as well; they were unused.</li> </ul> </li> <li><strong>2019-02-13.1</strong> <ul> <li>After consultation with bioRxiv, the "fulltext" table will not be included in further snapshots until (and if) concerns about licensing and copyright can be resolved.</li> <li>The "docker-compose.yml" file was added, with corresponding instructions in the README to streamline deployment of a local copy of this database.</li> </ul> </li> <li><strong>2019-02-13</strong> <ul> <li>The redundant "paper" schema has been removed.</li> <li>BioRxiv has begun making the full text of preprints available online. Beginning with this version, a new table ("fulltext") is available that contains the text of preprints that have been processed already. <strong>The format in which this information is stored may change in the future</strong>; any digression will be noted here.</li> <li>This is the first version that has <a href="https://cloud.docker.com/u/blekhmanlab/repository/docker/blekhmanlab/rxivist_data">a corresponding Docker image</a>.</li> </ul> </li> </ul>
Germination and early establishment of dryland grasses and shrubs on wind-eroded soils from the Jornada Basin LTER Scrape site, 2018-2019
In this dataset, we report germination and seedling growth of contrasting perennial grass (Bouteloua eriopoda, Sporobolus airoides, and Aristida purpurea) and shrub (Prosopis glandulosa, Atriplex canescens, and Larrea tridentata) functional groups grown on non-winnowed and winnowed soils collected from the Jornada Basin LTER Scrape Site at the Jornada Experimental Range (JER) in southern New Mexico, U.S.A. The soil physical and chemical properties of winnowed and non-winnowed soils were evaluated, and soil water retention curves were obtained for the two soil types. A controlled pot experiment was conducted under the well-watered greenhouse conditions at the University of Arizona campus, Tucson, AZ in 2018 and 2019 to test if topsoil "winnowing" by wind erosion would differentially affect grass and shrub seedling establishment to promote shrub recruitment over that of grass. Data include soil water retention curves, soil chemistry, nutrients, and texture, seedling germination, and seedling growth and biomass. This study is complete and the data are published in the article below. Niu, F., Pierce, N. A., Archer, S. R., & Okin, G. S. (2021). Germination and early establishment of dryland grasses and shrubs on intact and wind-eroded soils under greenhouse conditions. Plant and Soil, 1-16. DOI: 10.1007/s11104-021-05005-9
Práctica web scraping motorflash_v1
<p>Esta práctica se ha realizado en el contexto de la asignatura de 'tipología y ciclo de vida de los datos' del Master en Ciencia de Datos de la Universitat Oberta de Catalunya. En ella, se aplican técnicas de web scraping utilizando el lenguaje de programación Python y la librería scrapy. Los datos se han extraído de la página web de anuncios de coches de segunda mano '<a href="https://www.motorflash.com/">https://www.motorflash.com/</a>', de la que se obtienen datos generales y características del vehículo anunciado. </p> <p>Para conocer más a fondo el proceso de extracción puede visitar el repositorio del proyecto <a href="https://github.com/CarlosRea/MotorflashScraper">https://github.com/CarlosRea/MotorflashScraper</a> </p>
Iphones publicados en la página web de Back Market el 25/03/2022 extraídos con Web Scraping
<p>El dataset recoge todos los Iphone en venta a la página web de Back Market el día 25/03/2022 a las 19:56, correspondiendo a una extracción con Web Scraping de esta plataforma online, que se dedica a vender productos reacondicionados.</p> <p>El conjunto de datos incluye todos los Iphone disponibles en la tienda con sus características principales: nombre del producto (<em>Producte</em>), su capacidad en Gigas (<em>Capacitat</em>), su color (<em>Color</em>), si es libre de operador (<em>Operador</em>), el precio del producto (<em>Preu</em>), la puntuación del producto (<em>Puntuacio</em>), la empresa que ha reacondicionado el producto (<em>Reacondicionador</em>), desde donde se envía el producto (<em>Origen_enviament</em>), los meses de garantía (<em>Garantia</em>) y la url de cada producto (<em>Url</em>). </p> <p> </p>
115th U.S. Congress Member Website (Full JavaScript-enabled Scrape) Collection
<p>This data set represents a point-in-time full JavaScript-enabled scrape of all available 115th U.S. Congress member web sites. The data collection originated and completed on 2018-04-13 and the results are in ndjson/jsonlines/streaming JSON format. File format information is in the enclosed README.md file.</p> <p>The data was used to evaluate the privacy profiles of each U.S. Congress members' official (.gov hosted) websites for the discussion in <https://rud.is/b/2018/04/13/does-congress-really-care-about-your-privacy/>.</p> <p>ScrapingHub's "Splash" platform (<https://github.com/scrapinghub/splash>) was used along with the "splashr" R package (<https://github.com/hrbrmstr/splashr>) to retrieve the content.</p>
Books to scrape dataset
<p>This dataset contains the data from the books in the website https://books.toscrape.com/, which is a public website for webscraping. The columns in this dataset are:</p> <ul> <li> <p>titulo – Title of the book.</p> </li> <li> <p>categoria – Category of the book. </p> </li> <li> <p>rating – Rating given to the book.</p> </li> <li> <p>precio - Price of the book.</p> </li> <li> <p>cantidad_stock – Number of copies available in stock of the book at the time of creation of the dataset.</p> </li> </ul>
Scraped tweets about women in STEM from April 2022 to May 2023
<p>The data comprises one csv file named tweets. It has 168,677 tweets scraped with the help of snscrape spanning April 2022 to May 2023. Each entry contains metadata regarding the tweet and it's author. The tweets are curated to be representative of discourse regarding women in STEM. The search queries used while scraping are "womeninSTEM", "womeninTech", etc.</p> <p> </p>
Surface abrasion and crust evolution following vegetation removal at the Jornada Basin LTER Scrape Site, 1995 to 2019
This data package contains measurements of soil surface abrasion and the evolution of soil crust shear strength following a soil and vegetation removal (scrape) treatment at the Jornada Basin LTER site in southern New Mexico, USA. Soil erosion and soil crust shear strength is measured at 3 locations along the prevailing wind direction across a site initially scraped in 1995 to remove the A soil horizon in a 100 meter radius semicircle. There are three monitoring stations (East, Middle, West) on the Scrape Site. The West site is the windward site, the East site is the leeward site, and the middle site is halfway between the other two. At each location, measurements are taken of the distance of the sand surface and soil crust to a crossbar set into the soil. Three "Torvane" measurements that measure the torque (shear strength) needed to break the crust are also made at each location. These measurements were made monthly until 2014, and are made annually since 2015. The Scrape Site has now become the location for the GROWES study (JRN study IDs 511 and 523). Data collection for this study is ongoing.
Imágenes de Satélite del canal IR-GOES/SENAMHI Obtenidas por Scraping.
<p>Imágenes de Satélite del canal IR GOES-16/SENAMHI obtenidas por técnica de scraping.</p>
Practica 1 Web Scraping Oil Price Data
<p>Dataset about the prices of the oil and its products with and without taxes across the years1.0</p>
Web Scraping Universidad de Sevilla
<p>Proyecto de Web Scraping con Beautiful Soup (Python) realizado para la asignatura Tipología y ciclo de vida de los datos, del Master en Ciencia de Datos de la UOC.<br>Se aplican las técnicas necesarias para extraer información de la web de la Universidad de Sevilla y generar un dataset con información sobre los grados que ofrece la misma.<br>El dataset generado contiene las siguientes variables:<br> * grado: nombre del título en cuestión<br> * rama: rama del conocimiento al que pertenece (Ciencias Sociales, Ciencias de la Salud, Ingeniería..., etc.)<br> * facultad: centro de la Universidad de Sevilla responsable del título<br> * duracion: número en años en los que se plantea el título<br> * creditos: número de créditos necesarios para obtener el título<br> * oferta: número total de plazas ofertadas por la Universidad de Sevilla el curso anterior<br> * demanda: número total de veces que se ha solicitado una plaza durante las fases de inscripción</p>
Tipologia de Datos UOC | Práctica 1 | Web Scraping de Booking.com
<p>El presente dataset ha sido obtenido bajo el marco de la asignatura "Tipologia y ciclo de vida de los datos". Concretamente se trata del resultado obtenido tras realizar web scraping del metabuscador de hoteles "Booking.com" utilizando el lenguaje Python y el paquete Selenium.</p>
Web scraping UOC, wines from Vinissimus
<p>Dataset with 1320 wines obtained from scraping the <a href="https://www.vinissimus.com/es/">Vinissimus</a> website. The code used to obtain it can be found here: <a href="https://github.com/PepIngla/webScrapingWines">Github</a>.</p> <p>It contains the following fields:</p> <p>Type: String, indicates the type of wine. It can take three values “Vino tinto” or red wine, “Vino blanco” or white wine, “Vinos rosados y rosé” or rosé wine because we have restricted our search to these types.</p> <p>Name: String, contains the name of the wine.</p> <p>Year: Integer, indicates the crop year. If it was not specified on the website its value in the database is -1.</p> <p>Cellar: String, wine producer.</p> <p>Region: String, the region where the wine was produced.</p> <p>Country: String, the country where the wine was produced.</p> <p>Varieties: String, grape varieties contained in the wine. If there is no variety specified on the website, we have set it to "".</p> <p>Eco: String, if the wine has the "eco" label it takes the value "ECO", otherwise, it is nan.</p> <p>Rating: Float, wine rating according to the vinissimus.com customers.</p> <p>Stars: Integer, tells us the number of stars of the wine according to the vinissimus.com customers.</p> <p>Opinions: Integer, number of opinions published on the website about the wine.</p> <p>Likes: Integer, number of likes by the vinissimus.com customers.</p> <p>Parker: String, Parker rating. If it is not available, the value is "".</p> <p>Penin: String, Peñín rating. If it is not available, the value is "".</p> <p>Suckling: String, Suckling rating. If it is not available, the value is "".</p> <p>Tim_atkin: String, Tim Atkin rating. If it is not available, the value is "".</p> <p>Price: String, the price of the wine.</p> <p>Old_price: String, if the wine is on discount, it indicates the old price.</p> <p>Offer: Boolean, True if the wine is on discount, False otherwise.</p> <p>Volume: String, indicates the volume of the bottle.indica el volum de l’ampolla. The default value in case it is not on the website is: “ / bot. 0,75 L “.</p> <p>Image: bytes, an image of the wine bottle.</p>
Figure 3 in Capoeta raghazensis, a new species of algae-scraping cyprinid from the Raghaz Canyon in Hormuz basin, southern Iran (Teleostei: Cyprinidae)
Figure 3. Ventral view of Head. Capoeta raghazensis sp. n. (A: IMNRF-UT-1105-4, 114 mm SL, male; B: IMNRF-UT-1105-7, 231 mm SL, female; C: IMNRF-UT-1105-6, 200 mm SL, female; D: IMNRF-UT-1105-4, 111.8 mm SL, male).
Figure 2 in Capoeta raghazensis, a new species of algae-scraping cyprinid from the Raghaz Canyon in Hormuz basin, southern Iran (Teleostei: Cyprinidae)
Figure 2. Capoeta raghazensis sp. n., paratypes; A, IMNRF-UT-1105-7; 231 mm SL; B, IMNRF-UT-1105-6; 200 mm SL; C, IMNRF-UT- 1105-4; 114 mm SL.
Figure 2 in Capoeta raghazensis, a new species of algae-scraping cyprinid from the Raghaz Canyon in Hormuz basin, southern Iran (Teleostei: Cyprinidae)
Figure 2. Capoeta raghazensis sp. n., IMNRF-UT-1105-4, holotype, 112 mm SL. Iran: Hormuzgan prov., Darab City, Raghaz Canyon, Hormuz basin.
Figure 4 in Capoeta raghazensis, a new species of algae-scraping cyprinid from the Raghaz Canyon in Hormuz basin, southern Iran (Teleostei: Cyprinidae)
Figure 4. Raghaz Canyon, near Darab City, Hormuz basin, type locality of Capoeta raghazensis, sp. n..
Practica1 Web Scraping
<p>En el marco de la asignatura "Tipología y ciclo de vida de los datos" del Máster en Ciencia de Datos de la Universitat Oberta de Catalunya, se ha llevado a cabo un proyecto basado en el enunciado de la Práctica 1. Este proyecto consiste en utilizar diversas técnicas de web scraping y código en Python para obtener datos que posteriormente serán analizados. El objetivo principal es extraer el catálogo completo del sitio web https://www.filmin.es/ 🎬 y analizar la cantidad de contenido disponible en catalán. Este proyecto tiene el potencial de ser escalable y aplicable a otras plataformas, lo que permitiría realizar análisis comparativos entre ellas</p>
Technoableism & Social Media: TikTok Python Scrape
<p>This Notebook uses Deen Freelon's module "Pyktok" to scrape metadata from videos that mention terms related to disability and technology. Search terms include "wearable tech," "disability," "technology," and "biohack." </p>
Bitcointalk.org scrape
<p>Bitcoin scrape (April-may 2023) or board 1 (https://bitcointalk.org/index.php?board=1.0) with within this board/list </p> <p>"PostID": post_id,</p> <p>"Username": username_element.text,</p> <p>"PostDate": post_date_element.text,</p> <p>"PostContent": post_content_element.text,</p> <p>"PostTitle": post_title_element.text,</p> <p>"PostURL": post_url</p> <p>for all pages per post/topic</p> <p>File name : output_file_name = output_folder + url.split('=')[1].replace('.', '_') + ".csv" (from the list of url's per topic)</p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.