Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
51
datasets available to search
ShareScore release 0.9.0
Dataset results
51 results for “jupyter”
Python scripts / Jupyter Notebooks and data for training segmentation models on slide scans of diatom preparations from river Menne
<p>This archive contains the Jupyter Notebooks and data used for the deep learning experiments published in Kloster et al. 2022: Improving deep learning-based segmentation of diatoms in gigapixel-sized virtual slides by object-based tile positioning and object integrity constraint.</p> <p>The notebooks are numbered according to the order in which they are to execute. Please refer to the comments and documentation within the notebooks as well as to the manuscript for details. The data (image data, mask data & segmentation ground truth in COCO format for several different tiling strategies) is stored in separate subfolders corresponding with data usage (model training, validation, test) and tiling strategy. Please refer to the "readme" files for detailed information.</p> <p> </p> <p> </p>
Data for "Interactive maps in the Jupyter notebook"
<p>Dataset used for the lesson "<a href="https://annefou.github.io/jupyter_maps/index.html">Interactive maps in the Jupyter notebook</a>" </p> <p> </p> <p>Taught at CarpentryConnect, Manchester 2019. </p>
FAIR Jupyter Knowledge Graph
<p><a href="https://w3id.org/fairjupyter/">FAIR Jupyter</a> is a knowledge graph representation of <a href="https://doi.org/10.5281/zenodo.8226725">Dataset of a Study of Computational reproducibility of Jupyter notebooks from biomedical publications</a>. <br>This repository contains the data, mapping, and the RDF files for each entity represented in the FAIR Jupyter KG.</p> <p>Folder structure:</p> <p>data: contains the CSV files that are exported for each entity type from the <a href="https://doi.org/10.5281/zenodo.8226725">original sqlite database</a>.</p> <p>mapping: contains the RML and YARRML mapping files for each entity type of FAIR Jupyter KG.</p> <p>kg: contains the RDF files created in N-Triples format for each entity type of FAIR Jupyter KG.</p> <p>The code for creating the FAIR Jupyter Knowledge graph is available here: https://github.com/fusion-jena/fairjupyter</p>
Small-angle Scattering Data Analysis Round Robin: anonymized results, figures and Jupyter notebook
<p>The intent of this round robin was to find out how comparable results from different researchers are, who analyse exactly the same processed, corrected dataset.</p> <p>This zip file contains the anonymized results and the jupyter notebook used to do the data processing, analysis and visualisation. Additionally, TEM images of the samples are included. </p>
Dataset of a Study of Computational reproducibility of Jupyter notebooks from biomedical publications
<p>This repository contains the dataset for the study of <a href="https://doi.org/10.1093/gigascience/giad113">computational reproducibility of Jupyter notebooks from biomedical publications</a>. Our focus lies in evaluating the extent of reproducibility of Jupyter notebooks derived from GitHub repositories linked to publications present in the biomedical literature repository, PubMed Central. We analyzed the reproducibility of Jupyter notebooks from GitHub repositories associated with publications indexed in the biomedical literature repository PubMed Central. The dataset includes the metadata information of the journals, publications, the Github repositories mentioned in the publications and the notebooks present in the Github repositories.</p> <p><strong>Data Collection and Analysis</strong></p> <p>We use the code for reproducibility of Jupyter notebooks from the study done by <a href="../record/2592524">Pimentel et al., 2019</a> and adapted the code from <a href="https://github.com/fusion-jena/ReproduceMeGit">ReproduceMeGit</a>. We provide code for collecting the publication metadata from PubMed Central using <a href="https://biopython.org/docs/1.76/api/Bio.Entrez.html">NCBI Entrez utilities via Biopython</a>.</p> <p>Our approach involves searching PMC using the esearch function for Jupyter notebooks using the query: ``(ipynb OR jupyter OR ipython) AND github''. We meticulously retrieve data in XML format, capturing essential details about journals and articles. By systematically scanning the entire article, encompassing the abstract, body, data availability statement, and supplementary materials, we extract GitHub links. Additionally, we mine repositories for key information such as dependency declarations found in files like requirements.txt, setup.py, and pipfile. Leveraging the GitHub API, we enrich our data by incorporating repository creation dates, update histories, pushes, and programming languages.</p> <p>All the extracted information is stored in a SQLite database. After collecting and creating the database tables, we ran a pipeline to collect the Jupyter notebooks contained in the GitHub repositories based on the code from Pimentel et al., 2019.</p> <p>Our reproducibility pipeline was started on 27 March 2023.</p> <p><strong>Repository Structure</strong></p> <p>Our repository is organized into two main folders:</p> <ul> <li><strong>archaeology</strong>: This directory hosts scripts designed to download, parse, and extract metadata from PubMed Central publications and associated repositories. There are 24 database tables created which store the information on articles, journals, authors, repositories, notebooks, cells, modules, executions, etc. in the db.sqlite database file.</li> <li><strong>analyses</strong>: Here, you will find notebooks instrumental in the in-depth analysis of data related to our study. The db.sqlite file generated by running the archaelogy folder is stored in the analyses folder for further analysis. The path can however be configured in the config.py file. There are two sets of notebooks: one set (naming pattern N[0-9]*.ipynb) is focused on examining data pertaining to repositories and notebooks, while the other set (PMC[0-9]*.ipynb) is for analyzing data associated with publications in PubMed Central, i.e.\ for plots involving data about articles, journals, publication dates or research fields. The resultant figures from the these notebooks are stored in the 'outputs' folder.</li> <li><strong>MethodsWorkflow</strong>: The MethodsWorkflow file provides a conceptual overview of the workflow used in this study.</li> </ul> <p><strong>Accessing Data and Resources:</strong></p> <ul> <li>All the data generated during the initial study can be accessed at https://doi.org/10.5281/zenodo.6802158</li> <li>For the latest results and re-run data, refer to this link.</li> <li>The comprehensive SQLite database that encapsulates all the study's extracted data is stored in the db.sqlite file.</li> <li>The metadata in xml format extracted from PubMed Central which contains the information about the articles and journal can be accessed in pmc.xml file.</li> </ul> <p><strong>System Requirements:</strong></p> <ul> <li>Centos 7 (Documentation: https://www.centos.org/)</li> <li>Conda 4.9.4 (Installation Guide: https://docs.anaconda.com/anaconda/install/linux/)</li> <li>Python 3.7.6 (Download Link: https://www.python.org/downloads/)</li> <li>GitHub account (Get Started: https://github.com/, Requires GitHub Username and Token)</li> <li>gcc 7.3.0 (Installation Guide: https://gcc.gnu.org/install/)</li> <li>lbzip2 (Command: `conda install -c conda-forge lbzip2')</li> </ul> <p><strong>Running the pipeline:</strong></p> <ul> <li>Clone the computational-reproducibility-pmc repository using Git:<br>git clone https://github.com/fusion-jena/computational-reproducibility-pmc.git<br> </li> <li>Navigate to the computational-reproducibility-pmc directory:<br>cd computational-reproducibility-pmc/computational-reproducibility-pmc</li> <li>Configure environment variables in the config.py file:<br>GITHUB_USERNAME = os.environ.get("JUP_GITHUB_USERNAME", "add your github username here")<br>GITHUB_TOKEN = os.environ.get("JUP_GITHUB_PASSWORD", "add your github token here")</li> <li>Other environment variables can also be set in the config.py file.<br>BASE_DIR = Path(os.environ.get("JUP_BASE_DIR", "./")).expanduser() # Add the path of directory where the GitHub repositories will be saved<br>DB_CONNECTION = os.environ.get("JUP_DB_CONNECTION", "sqlite:///db.sqlite") # Add the path where the database is stored.</li> <li>To set up conda environments for each python versions, upgrade pip, install pipenv, and install the archaeology package in each environment, execute:<br>source conda-setup.sh</li> <li>Change to the archaeology directory<br>cd archaeology</li> <li>Activate conda environment. We used py36 to run the pipeline.<br>conda activate py36</li> <li>Execute the main pipeline script (r0_main.py):<br>python r0_main.py</li> </ul> <p><strong>Running the analysis:</strong></p> <ul> <li>Navigate to the analysis directory.<br>cd analyses</li> <li>Activate conda environment. We use raw38 for the analysis of the metadata collected in the study.<br>conda activate raw38</li> <li>Install the required packages using the requirements.txt file.<br>pip install -r requirements.txt</li> <li>Launch Jupyterlab<br>jupyter lab</li> <li>Refer to the Index.ipynb notebook for the execution order and guidance.</li> </ul> <p><strong>References:</strong></p> <ul> <li>Sheeba Samuel, Daniel Mietchen. (2024). Computational reproducibility of Jupyter notebooks from biomedical publications, https://doi.org/10.1093/gigascience/giad113, GigaScience</li> <li>Sheeba Samuel, Daniel Mietchen. (2022). Computational reproducibility of Jupyter notebooks from biomedical publications, https://arxiv.org/pdf/2209.04308.pdf, CoRR abs/2209.04308</li> <li>Sheeba Samuel, & Daniel Mietchen. (2022). Dataset of a Study of Computational reproducibility of Jupyter notebooks from biomedical publications [Data set]. Zenodo. https://doi.org/10.5281/zenodo.6802158</li> </ul> <p> </p>
Outputs of the Jupyter Notebook - Deep learning and variational inversion to quantify and attribute climate change (CIRC23)
<p>The dataset contains the outputs of the notebook "Deep learning and variational inversion to quantify and attribute climate change (CIRC23)" published in The Environmental Data Science Book.</p>
Outputs of the Jupyter Notebook - Variational data assimilation with deep prior (CIRC23)
<p>The repository contains the outputs of the notebook "Variational data assimilation with deep prior (CIRC23)" published in The Environmental Data Science Book.</p>
Jupyter Notebook and comprising data for GRL2023GL106264R: Understanding the Cascade: Removing GCM biases improves dynamically downscaled climate projections
<p>This notebook and attendant files allows users to interface with a small subset of the data used to create the data in GRL2023GL106264R. Also feel free to check out the overall description of the non-bias corrected dynamically downscaled GCMs in WUS-D3 here: https://zenodo.org/records/10635867. This DOI also contains version of WRF 4.1.3 allowing for yearly CH4, CO2, and N2O updates, as well as a 360-day calendar version.</p>
Datasets for the paper "ReSplit: Improving the Structure of Jupyter Notebooks by Re-Splitting Their Cells"
<p>In this archive, you can find all the data used in the paper "ReSplit: Improving the Structure of Jupyter Notebooks by Re-Splitting Their Cells".</p> <p><strong>sklearn_full_cells.csv</strong> is the dataset from the paper of Pimentel et al. filtered with only Data Science notebooks.<br> <strong>complete.csv</strong> is the dataset obtained after the full run of ReSplit on the dataset: both merging and splitting.<br> <strong>split.csv</strong> is the dataset obtained after running only the splitting part of our dataset.<br> <strong>merged.csv</strong> is the dataset obtained after running only the merging part of our dataset.<br> <strong>duplicates_id.csv</strong> contains the IDs of the duplicate notebooks for deduplication.<br> <strong>changes.csv</strong> contains the IDs of the datasets, as well as their length before and after running ReSplit.<br> <strong>survey.csv</strong> is the table with the results of the survey.</p> <p>In the dataset CSVs, each line is a cell that has a unique identifier and an identifier of the corresonding notebook.</p>
Jupyter Notebooks for "Evaluating CephFS Performance vs. Cost on High-Density Commodity Disk Servers" 10.1007/s41781-021-00071-1
<p>Jupyter notebooks used to create plots in article DOI 10.1007/s41781-021-00071-1</p> <p>Title "Evaluating CephFS Performance vs. Cost on High-Density Commodity Disk Servers"</p> <p>Journal "Computing and Software for Big Science"</p>
Biotechnology data analysis training with Jupyter Notebooks
<p>Biotechnology has experienced innovations in analytics and data processing. As the volume of data and its complexity grows, new computational procedures for extracting information are developed. However, the rate of change outpaces the adaptation of biotechnology curricula, necessitating new teaching methodologies to equip biotechnologists with data analysis abilities. To simulate experimental data, we created a virtual organism simulator (<em>silvio</em>) by combining diverse cellular and sub-cellular microbial models. With the <em>silvio </em>Python package, we constructed a computer-based instructional workflow to teach growth curve data analysis, promoter sequence design, and expression rate measurement. The instructional workflow is a Jupyter Notebook with background explanations and Python-based experiment simulations combined. The data analysis is either conducted within the Notebook in Python or externally with Excel. This instructional workflow was separately implemented in two distance courses for Master's students in biology and biotechnology with assessment of the pedagogic efficiency. The concept of using virtual organism simulations that generate coherent results across different experiments can be used to construct consistent and motivating case studies for biotechnological data literacy.</p> <p>Here, the supplementary material is provided.</p> <table> <tbody> <tr> <td>2207_BLS-RecExpSim.mbz</td> <td>Moodle backup file for import as new moodle function.</td> </tr> <tr> <td>BLS_RecExpSim_PerformanceEvaluation Rubric.docx</td> <td>Expected learning outcomes with associated performance levels.</td> </tr> <tr> <td>BLS_SurveryQuestions.docx</td> <td>Survey questions to evaluate the educational approach.</td> </tr> <tr> <td>RecExpSim.html</td> <td>Html-Export of the Jupyter Notebook to teach biotechnology data analysis. This only serves as visual impression of the course because the dynamic Python-evaluations are not functioning.</td> </tr> <tr> <td>RecExpSim_Lecture.pdf</td> <td>Static pdf of preparatory lecture to cover the theoretical aspects in the simulations and to get student on comparable level.</td> </tr> <tr> <td>RecExpSim_Lecture.pptx</td> <td>Adjustable pptx of preparatory lecture to cover the theoretical aspects in the simulations and to get student on comparable level.</td> </tr> </tbody> </table> <p> </p> <p> </p>
Outputs of the Jupyter Notebook - Sea ice forecasting using the IceNet Library
<p>The dataset contains the outputs of the notebook "Sea ice forecasting using the IceNet library" published in The Environmental Data Science Book.</p> <p><strong>Contributions</strong></p> <p><em>Notebook</em></p> <ul> <li>James Byrne (author), British Antarctic Survey, <a href="https://github.com/JimCircadian">@JimCircadian</a></li> <li>Bryn Noel Ubald (author), British Antarctic Survey, <a href="https://github.com/tom-andersson">@tom-andersson</a></li> <li>Wei Ji (reviewer), Development Seed, <a href="https://github.com/weiji14">@weiji14</a></li> <li>William Gregory (reviewer), Princeton University, <a href="https://github.com/William-gregory">@William-gregory</a></li> <li>Anne Fouilloux (editor), Simula Research Laboratory, <a href="https://github.com/annefou">@annefou</a></li> </ul> <p><em>Modelling codebase</em></p> <ul> <li>James Byrne (Code author)</li> <li>Tom Andersson (Science author)</li> <li>Bryn Noel Ubald (Code maintainer and contributor)</li> </ul>
Reproducibility by Default: Jupyter on Chameleon (screencast)
<p>The Jupyter notebook is an interactive environment that allows users to tell the story of an experiment by combining explanations in text, capturing the process in code, and representing results as images or graphs.</p> <p>The Notebook a particularly useful tool for expressing reproducible experiments because it allows you to record and share the experimental process – as well as the reasoning that went with it -- rather than just the results. Sharing a notebook allows others to easily repeat – and potentially also modify your experiment. This makes “standing on the shoulders of giants” a much easier proposition than ever before.</p>
Vibrational-EELS Dataset and data processing routine (Jupyter Notebook) (Laforet et al.)
<p>Contains all the vibrational-EELS data presented in the article, accompanied with the python HyperSpy processing routine used (Jupyter Notebook). </p>
Python and Jupyter Notebook for Medical Image Analysis - OpenMRBenelux 2020
<p>Dataset for the workshop "Python and Jupyter Notebook for Medical Image Analysis" at OpenMRBenelux - January 22, 2020 - Nijmegen (The Netherlands)</p>
Complete set of raw and processed datasets, as well as associated Jupyter notebooks for analysis, associated with manuscript entitled: "The MOUSE project: a practical approach for obtaining traceable, wide-range X-ray scattering information"
<p>This dataset is a complete set of raw, processed and analyzed data, complete with Jupiter notebooks, associated with the manuscript mentioned in the title. </p> <p>In the manuscript, we provide a ``systems architecture''-like overview and detailed discussions of the methodological and instrumental components that, together, comprise the "MOUSE" project (<strong>M</strong>ethodology <strong>O</strong>ptimization for <strong>U</strong>ltrafine <strong>S</strong>tructure <strong>E</strong>xploration). Through this project, we aim to provide a comprehensive methodology for obtaining the highest quality X-ray scattering information (at small and wide angles) from measurements on materials science samples. </p>
Jupyter Notebook Activity Dataset (rsds-20241113)
<h2>List of data</h2> <ul> <li>rsds-20241113.zip: Collection of SQLite database files</li> <li>image.tar.gz: Docker image provided in our data collection experiment</li> <li>redspot-341ffa5.zip: Redspot source code (<a href="https://github.com/tomokinakamaru/redspot/tree/341ffa56cb941b6f1ad74bd50d23fcf0ee96b270" target="_blank" rel="noopener">redspot@341ffa5</a>)</li> </ul> <div> <h2>Extended version of Section 2D of our paper</h2> Redspot is a Jupyter extension (i.e., Python package) that records activity signals. However, it also offers interfaces to read recorded signals. The following shows the most basic usage of its command-line interface:<br> <div> </div> <div><code>redspot replay <path-to-db></code></div> <br> <div>This command generates snapshots (.ipynb files) restored from the signal records. Note that this command does not produce a snapshot for every signal. Since the change represented by a single signal is typically minimal (e.g., one keystroke), generating a snapshot for each signal results in a meaninglessly large number of snapshots. <em><strong>However, we want to obtain signal-level snapshots for some analyses. In such cases, one can analyze them using the application programming interfaces:</strong></em></div> <br> <div><code>from redspot import database</code></div> <div><code>from redspot.notebook import Notebook</code></div> <div><code>nbk = Notebook()</code></div> <div><code>for signal in database.get("path-to-db"):</code></div> <div><code> time, panel, kind, args = signal</code></div> <div><code> nbk.apply(kind, args) # apply change</code></div> <div><code> print(nbk) # print notebook</code></div> <br> <div>To record activities, one needs to run the Redspot command in the recording mode as follows:</div> <br> <div><code>redspot record</code></div> <br> <div>This command launches Jupyter Notebook with Redspot enabled. Activities made in the launched environment are stored in an SQLite file named ``redspot.db'' under the current path.</div> <br> <div>To launch the environment we provided to the participants, one first needs to download and import the image (image.tar.gz). One can then run the image with the following command:</div> <br> <div><code>docker run --rm -it -p8888:8888 <image-name></code></div> <br> <div>Note that the SQLite file is generated in the running container. The file can be downloaded into the host machine via the file viewer of Jupyter Notebook.</div> </div>
Dataset and code for "ReSplit: Improving the Structure of Jupyter Notebooks by Re-Splitting Their Cells"
<pre>This package contains the data and code for the paper "ReSplit: Improving the Structure of Jupyter Notebooks by Re-Splitting Their Cells".</pre> <pre> </pre> <pre>`code.zip` contains the source code of our approach, as well as the details about the conducted survey.</pre> <pre>`data.zip` contains all the data used in our work, both before and after running ReSplit, as well as the results of the survey.</pre> <pre> </pre> <pre>You can find more details in READMEs in each archive. Please note that running the tool requires unzipping the data and placing it into the correct folder, according to the inner README.</pre>
Outputs of the Jupyter Notebook - MODIS MOD021KM and FIRMS
<p>The dataset contains the outputs of the notebook "MODIS MOD021KM and FIRMS" published in The Environmental Data Science Book.</p> <p><strong>Contributions</strong></p> <p><em>Notebook</em></p> <ul> <li> <p>Samuel Jackson (author), Science & Technology Facilities Council, <a href="https://github.com/samueljackson92">@samueljackson92</a></p> </li> <li> <p>Alejandro Coca-Castro (reviewer), The Alan Turing Institute, <a href="https://github.com/acocac">@acocac</a></p> </li> </ul> <p><em>Dataset originator/creator</em></p> <p>MOD021KM</p> <ul> <li> <p>MODIS Characterization Support Team (MCST)</p> </li> <li> <p>MODIS Adaptive Processing System (MODAPS)</p> </li> </ul> <p>Firms</p> <ul> <li> <p>University of Maryland</p> </li> </ul> <p><em>Dataset authors</em></p> <p>MOD021KM</p> <ul> <li> <p>MODIS Science Data Support Team (SDST)</p> </li> </ul> <p>Firms</p> <ul> <li> <p>NASA’s Applied Sciences Program</p> </li> </ul> <p><em>Dataset documentation</em></p> <ul> <li> <p>Louis Giglio, Wilfrid Schroeder, Joanne V. Hall, and Christopher O. Justice. MODIS Collection 6 Active Fire Product User’s Guide Revision B. Technical Report, NASA, 2018. URL: <a href="https://modis-fire.umd.edu/files/MODIS_C6_Fire_User_Guide_B.pdf">https://modis-fire.umd.edu/files/MODIS_C6_Fire_User_Guide_B.pdf</a>.</p> </li> </ul> <p> </p>
Additional 1000 notebooks for the paper "A Large-Scale Comparison of Python Code in Jupyter Notebooks and Scripts"
<p>Additional 1000 notebooks for the review of the paper "A Large-Scale Comparison of Python Code in Jupyter Notebooks and Scripts".</p> <p>The notebooks can be processed using Matroskin tool from the supplementary materials. To do that, place:</p> <p>- the folder "1k_notebooks_dataset" into the folder ".../databases/datasets"</p> <p>- the file "mapping_of_1k_notebooks.json" into the folder ".../databases/mappings"</p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.