Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
11
datasets available to search
ShareScore release 0.9.0
Dataset results
11 results for “pypi”
Dependency networks of PyPI, npm, CRAN and Bioconductor repositories
<p>A set of datasets with a list of dependencies from the software repositories PyPI, npm, CRAN and Bioconductor.</p>
PyPI Packages Annotated
<p>PyPI Package data, as a directed graph, in .gexf format. Directed edges are software dependencies. Package metadata includes information about release dates and number of downloads.</p>
Ecosystem-Level Determinants of Sustained Activity in Open-Source Projects: A Case Study of the PyPI Ecosystem
<pre><em>Replication pack, FSE2018 submission #164: </em><em>------------------------------------------ </em></pre> <pre><strong>**</strong>Working title:<strong>** </strong>Ecosystem-Level Factors Affecting the Survival of Open-Source Projects: A Case Study of the PyPI Ecosystem <strong>**</strong>Note:<strong>** </strong>link to data artifacts is already included in the paper. Link to the code will be included in the Camera Ready version as well. <em>Content description </em><em>=================== </em> <strong>- **</strong>ghd-0.1.0.zip<strong>** </strong>- the code archive. This code produces the dataset files described below <strong>- **</strong>settings.py<strong>** </strong>- settings template for the code archive. <strong>- **</strong>dataset_minimal_Jan_2018.zip<strong>** </strong>- the minimally sufficient version of the dataset. This dataset only includes stats aggregated by the ecosystem (PyPI) <strong>- **</strong>dataset_full_Jan_2018.tgz<strong>** </strong>- full version of the dataset, including project-level statistics. It is ~34Gb unpacked. This dataset still doesn't include PyPI packages themselves, which take around 2TB. <strong>- **</strong>build_model.r, helpers.r<strong>** </strong>- R files to process the survival data (`survival_data.csv` in <strong>**</strong>dataset_minimal_Jan_2018.zip<strong>**</strong>, `common.cache/survival_data.pypi_2008_2017-12_6.csv` in <strong>**</strong>dataset_full_Jan_2018.tgz<strong>**</strong>) <strong>- **</strong>Interview protocol.pdf<strong>** </strong>- approximate protocol used for semistructured interviews. <strong>- </strong>LICENSE - text of GPL v3, under which this dataset is published <strong>- </strong>INSTALL.md - replication guide (~2 pages)</pre> <pre><em>Replication guide </em><em>================= </em> <em>Step 0 - prerequisites </em><em>---------------------- </em> <strong>- </strong>Unix-compatible OS (Linux or OS X) <strong>- </strong>Python interpreter (2.7 was used; Python 3 compatibility is highly likely) <strong>- </strong>R 3.4 or higher (3.4.4 was used, 3.2 is known to be incompatible) Depending on detalization level (see Step 2 for more details): <strong>- </strong>up to 2Tb of disk space (see Step 2 detalization levels) <strong>- </strong>at least 16Gb of RAM (64 preferable) <strong>- </strong>few hours to few month of processing time <em>Step 1 - software </em><em>---------------- </em> <strong>- </strong>unpack <strong>**</strong>ghd-0.1.0.zip<strong>**</strong>, or clone from gitlab: git clone https://gitlab.com/user2589/ghd.git git checkout 0.1.0 `cd` into the extracted folder. All commands below assume it as a current directory. <strong>- </strong>copy `settings.py` into the extracted folder. Edit the file: <strong> * </strong>set `DATASET_PATH` to some newly created folder path <strong> * </strong>add at least one GitHub API token to `SCRAPER_GITHUB_API_TOKENS` <strong>- </strong>install docker. For Ubuntu Linux, the command is `sudo apt-get install docker-compose` <strong>- </strong>install libarchive and headers: `sudo apt-get install libarchive-dev` <strong>- </strong>(optional) to replicate on NPM, install yajl: `sudo apt-get install yajl-tools` Without this dependency, you might get an error on the next step, but it's safe to ignore. <strong>- </strong>install Python libraries: `pip install --user -r requirements.txt` . <strong>- </strong>disable all APIs except GitHub (Bitbucket and Gitlab support were not yet implemented when this study was in progress): edit `scraper/init.py`, comment out everything except GitHub support in `PROVIDERS`. <em>Step 2 - obtaining the dataset </em><em>----------------------------- </em> The ultimate goal of this step is to get output of the Python function `common.utils.survival_data()` and save it into a CSV file: # copy and paste into a Python console from common import utils survival_data = utils.survival_data('pypi', '2008', smoothing=6) survival_data.to_csv('survival_data.csv') Since full replication will take several months, here are some ways to speedup the process: <em>####Option 2.a, difficulty level: easiest </em> Just use the precomputed data. Step 1 is not necessary under this scenario. <strong>- </strong>extract <strong>**</strong>dataset_minimal_Jan_2018.zip<strong>** </strong><strong>- </strong>get `survival_data.csv`, go to the next step <em>####Option 2.b, difficulty level: easy </em> Use precomputed longitudinal feature values to build the final table. The whole process will take 15..30 minutes. <strong>- </strong>create a folder `<DATASET_PATH>/common.cache`, where `<DATASET_PATH>` is the value of the variable `DATASET_PATH` in `settings.py` <strong>- </strong>extract <strong>**</strong>dataset_minimal_Jan_2018<strong>** </strong>to the newly created folder <strong>- </strong>rename files: mv backporting.csv monthly_data.pypi_backporting.csv mv cc_degree.csv monthly_data.pypi_cc_degree.csv mv commercial.csv monthly_data.pypi_commercial.csv mv commits.csv monthly_data.pypi_commits.csv mv contributors.csv monthly_data.pypi_contributors.csv mv dc_katz.csv monthly_data.pypi_dc_katz.csv mv downstreams.csv monthly_data.pypi_downstreams.csv mv d_upstreams.csv monthly_data.pypi_d_upstreams.csv mv github_user_info.csv user_info.pypi.csv mv issues.csv monthly_data.pypi_issues.csv mv non_dev_issues.csv monthly_data.pypi_non_dev_issues.csv mv non_dev_submitters.csv monthly_data.pypi_non_dev_submitters mv package_urls.csv package_urls.pypi.csv mv q90.csv monthly_data.pypi_q90.csv # raw_dependencies.csv is not required # raw_packages_info.csv is not required # Feel free to read README.md for more details about the data mv submitters.csv monthly_data.pypi_submitters.csv # In this scenario we'll generate a new survival_data.csv mv university.csv monthly_data.pypi_university.csv mv upstreams.csv monthly_data.pypi_upstreams.csv <strong>- </strong>edit `common/decorators.py`, set `DEFAULT_EXPIRY` to some higher value, e.g. `DEFAULT_EXPIRY = float('inf') # cache never expires` Then, use the Python code above to obtain `survival_data.csv`. <em>####Option 2.c, difficulty level: medium </em> Use pre-downloaded raw data to build longitudinal feature values, and then the dataset. Despite most of the data is cached, some functions will pull up updates which might take anywhere from days to couple weeks to run. <strong>- </strong>Download <strong>**</strong>dataset_full_Jan_2018.tgz<strong>** </strong>(5.4Gb compressed, 34Gb unpacked). <strong>- </strong>edit `common/decorators.py`, set `DEFAULT_EXPIRY` to some higher value, e.g. `DEFAULT_EXPIRY = float('inf') # cache never expires` <strong>- </strong>extract the content of this archive into `<DATASET_PATH>`. <strong>- </strong>clean up `<DATASET_PATH>/common.cache` (otherwise you'll get Step 2.a. You can reproduce Step 2.b by deleting only `survival_data.pypi_2008_2017-12_6.csv`) Run the Python code above to obtain `survival_data.csv`. <em>####Option 2.d, difficulty level: hard </em> Build the dataset from scratch. Although most of the processing is parallelized, it will take at least couple months on a reasonably powerful server (32 cores, 512G of RAM, 2Tb+ of HDD space in our setup). <strong>- </strong>ensure the `<DATASET_PATH>` is empty <strong>- </strong>add more GitHub tokens (borrow from your coworkers) to `settings.py`. Run the Python code above to obtain `survival_data.csv`. <em>Step 3 - run the regression </em><em>--------------------------- </em> install R libraries: install.packages(c("htmlTable", "OIsurv", "survival", "car", "survminer", "ggplot2", "sqldf", "pscl", "texreg", "xtable")) Use `build_model.r` (e.g. in RStudio) and produced `survival_data.csv` to build the regressions used in the paper. This process takes at least 16Gb of RAM and takes few hours to run due to the gigantic size of the dataset. </pre>
An Analysis of Dependency Network Evolution in PyPI
<p>Retrieved from Libraries.io.</p>
Replication Package for ICSE 2023 submission "How Deep Learning Packages Form Supply Chain in PyPI: Types, Clusters, and Detachment"
<p>This is the replication package for our ICSE 2023 submission <em><strong>How Deep Learning Packages Form Supply Chain in PyPI: Types, Clusters, and Detachment</strong></em>. </p>
Ecosystem-Level Factors Affecting the Survival of Open-Source Projects: A Case Study of the PyPI Ecosystem - the dataset
<pre><em>Replication pack, FSE2018 submission #164: </em><em>------------------------------------------ </em></pre> <pre><strong>**</strong>Working title:<strong>** </strong>Ecosystem-Level Factors Affecting the Survival of Open-Source Projects: A Case Study of the PyPI Ecosystem <strong>**</strong>Note:<strong>** </strong>link to data artifacts is already included in the paper. Link to the code will be included in the Camera Ready version as well. <em>Content description </em><em>=================== </em> <strong>- **</strong>ghd-0.1.0.zip<strong>** </strong>- the code archive. This code produces the dataset files described below <strong>- **</strong>settings.py<strong>** </strong>- settings template for the code archive. <strong>- **</strong>dataset_minimal_Jan_2018.zip<strong>** </strong>- the minimally sufficient version of the dataset. This dataset only includes stats aggregated by the ecosystem (PyPI) <strong>- **</strong>dataset_full_Jan_2018.tgz<strong>** </strong>- full version of the dataset, including project-level statistics. It is ~34Gb unpacked. This dataset still doesn't include PyPI packages themselves, which take around 2TB. <strong>- **</strong>build_model.r, helpers.r<strong>** </strong>- R files to process the survival data (`survival_data.csv` in <strong>**</strong>dataset_minimal_Jan_2018.zip<strong>**</strong>, `common.cache/survival_data.pypi_2008_2017-12_6.csv` in <strong>**</strong>dataset_full_Jan_2018.tgz<strong>**</strong>) <strong>- **</strong>Interview protocol.pdf<strong>** </strong>- approximate protocol used for semistructured interviews. <strong>- </strong>LICENSE - text of GPL v3, under which this dataset is published <strong>- </strong>INSTALL.md - replication guide (~2 pages)</pre> <pre><em>Replication guide </em><em>================= </em> <em>Step 0 - prerequisites </em><em>---------------------- </em> <strong>- </strong>Unix-compatible OS (Linux or OS X) <strong>- </strong>Python interpreter (2.7 was used; Python 3 compatibility is highly likely) <strong>- </strong>R 3.4 or higher (3.4.4 was used, 3.2 is known to be incompatible) Depending on detalization level (see Step 2 for more details): <strong>- </strong>up to 2Tb of disk space (see Step 2 detalization levels) <strong>- </strong>at least 16Gb of RAM (64 preferable) <strong>- </strong>few hours to few month of processing time <em>Step 1 - software </em><em>---------------- </em> <strong>- </strong>unpack <strong>**</strong>ghd-0.1.0.zip<strong>**</strong>, or clone from gitlab: git clone https://gitlab.com/user2589/ghd.git git checkout 0.1.0 `cd` into the extracted folder. All commands below assume it as a current directory. <strong>- </strong>copy `settings.py` into the extracted folder. Edit the file: <strong> * </strong>set `DATASET_PATH` to some newly created folder path <strong> * </strong>add at least one GitHub API token to `SCRAPER_GITHUB_API_TOKENS` <strong>- </strong>install docker. For Ubuntu Linux, the command is `sudo apt-get install docker-compose` <strong>- </strong>install libarchive and headers: `sudo apt-get install libarchive-dev` <strong>- </strong>(optional) to replicate on NPM, install yajl: `sudo apt-get install yajl-tools` Without this dependency, you might get an error on the next step, but it's safe to ignore. <strong>- </strong>install Python libraries: `pip install --user -r requirements.txt` . <strong>- </strong>disable all APIs except GitHub (Bitbucket and Gitlab support were not yet implemented when this study was in progress): edit `scraper/init.py`, comment out everything except GitHub support in `PROVIDERS`. <em>Step 2 - obtaining the dataset </em><em>----------------------------- </em> The ultimate goal of this step is to get output of the Python function `common.utils.survival_data()` and save it into a CSV file: # copy and paste into a Python console from common import utils survival_data = utils.survival_data('pypi', '2008', smoothing=6) survival_data.to_csv('survival_data.csv') Since full replication will take several months, here are some ways to speedup the process: <em>####Option 2.a, difficulty level: easiest </em> Just use the precomputed data. Step 1 is not necessary under this scenario. <strong>- </strong>extract <strong>**</strong>dataset_minimal_Jan_2018.zip<strong>** </strong><strong>- </strong>get `survival_data.csv`, go to the next step <em>####Option 2.b, difficulty level: easy </em> Use precomputed longitudinal feature values to build the final table. The whole process will take 15..30 minutes. <strong>- </strong>create a folder `<DATASET_PATH>/common.cache`, where `<DATASET_PATH>` is the value of the variable `DATASET_PATH` in `settings.py` <strong>- </strong>extract <strong>**</strong>dataset_minimal_Jan_2018<strong>** </strong>to the newly created folder <strong>- </strong>rename files: mv backporting.csv monthly_data.pypi_backporting.csv mv cc_degree.csv monthly_data.pypi_cc_degree.csv mv commercial.csv monthly_data.pypi_commercial.csv mv commits.csv monthly_data.pypi_commits.csv mv contributors.csv monthly_data.pypi_contributors.csv mv dc_katz.csv monthly_data.pypi_dc_katz.csv mv downstreams.csv monthly_data.pypi_downstreams.csv mv d_upstreams.csv monthly_data.pypi_d_upstreams.csv mv github_user_info.csv user_info.pypi.csv mv issues.csv monthly_data.pypi_issues.csv mv non_dev_issues.csv monthly_data.pypi_non_dev_issues.csv mv non_dev_submitters.csv monthly_data.pypi_non_dev_submitters mv package_urls.csv package_urls.pypi.csv mv q90.csv monthly_data.pypi_q90.csv # raw_dependencies.csv is not required # raw_packages_info.csv is not required # Feel free to read README.md for more details about the data mv submitters.csv monthly_data.pypi_submitters.csv # In this scenario we'll generate a new survival_data.csv mv university.csv monthly_data.pypi_university.csv mv upstreams.csv monthly_data.pypi_upstreams.csv <strong>- </strong>edit `common/decorators.py`, set `DEFAULT_EXPIRY` to some higher value, e.g. `DEFAULT_EXPIRY = float('inf') # cache never expires` Then, use the Python code above to obtain `survival_data.csv`. <em>####Option 2.c, difficulty level: medium </em> Use predownloaded raw data to build longitudinal feature values, and then the dataset. Despite most of the data is cached, some functions will pull up updates which might take anywhere from days to couple weeks to run. <strong>- </strong>Download <strong>**</strong>dataset_full_Jan_2018.tgz<strong>** </strong>from http://k.soberi.us/dataset_full_Jan_2018.tgz . This file is not included in this archive because of its size (5.4Gb compressed, 34Gb unpacked). <strong>- </strong>edit `common/decorators.py`, set `DEFAULT_EXPIRY` to some higher value, e.g. `DEFAULT_EXPIRY = float('inf') # cache never expires` <strong>- </strong>extract the content of this archive into `<DATASET_PATH>`. <strong>- </strong>clean up `<DATASET_PATH>/common.cache` (otherwise you'll get Step 2.a. You can reproduce Step 2.b by deleting only `survival_data.pypi_2008_2017-12_6.csv`) Run the Python code above to obtain `survival_data.csv`. <em>####Option 2.d, difficulty level: hard </em> Build the dataset from scratch. Although most of the processing is parallelized, it will take at least couple months on a reasonably powerful server (32 cores, 512G of RAM, 2Tb+ of HDD space in our setup). <strong>- </strong>ensure the `<DATASET_PATH>` is empty <strong>- </strong>add more GitHub tokens (borrow from your coworkers) to `settings.py`. Run the Python code above to obtain `survival_data.csv`. <em>Step 3 - run the regression </em><em>--------------------------- </em> install R libraries: install.packages(c("htmlTable", "OIsurv", "survival", "car", "survminer", "ggplot2", "sqldf", "pscl", "texreg", "xtable")) Use `build_model.r` (e.g. in RStudio) and produced `survival_data.csv` to build the regressions used in the paper. This process takes at least 16Gb of RAM and takes few hours to run due to the gigantic size of the dataset. </pre>
Do Developers Present Proficient Code Snippets in Their README Files? An Analysis of PyPI Libraries in GitHub
<p><strong>Abstract:</strong><br> A README file plays an essential role as the face of a software project and the initial point of contact for developers in Open Source Software (OSS) projects. The code snippet is one of the most important contents in the README file for demonstrating the usage of software and APIs. Presenting easy to comprehend code snippets are preferred by developers to quickly understand the usage and features of the software. However, README files may contain proficient code snippets due to readability and performance reasons. In this paper, we first investigate the prevalence of each competency level of Python code snippets in the README files. Then, we analyze the relationships between the usage of proficient code snippets and topics of libraries. From our empirical study on 1,620 README files of PyPI libraries, we find that developers mainly present 92% of basic elements and 7% of independent elements in code snippets of the README files. However, developers may present proficient elements in code snippets for Application Framework, Quality Assurance, and User Interface libraries. Thus, we encourage developers to (i) write basic README files to attract more newcomers or (ii) write proficient README files to introduce effective usage of PyPI libraries.</p>
Supporting dataset for phuEGO PyPI package
<p>This upload contain support dataset for the Python package phuEGO, required for version 1.2.0 onwards and compatible with older version.</p> <p>It is labelled as the "slim" version as it only contains sematic similarity of type "gic".</p> <p>Github: https://github.com/haoqichen20/phuego</p> <p>PyPI: https://pypi.org/project/phuego/</p> <p>Documentation: https://phuego.readthedocs.io/en/latest/</p>
Mongodb Database dump for TOSEM submission "Characterizing Deep Learning Package Supply Chains in PyPI: Domains, Clusters, and Disengagement"
<p>The Mongodb Database dump for TOSEM submission "Characterizing Deep Learning Package Supply Chains in PyPI: Domains, Clusters, and Disengagement"</p>
[DEPRECATED] A manually-curated categorisation of Java Maven libraries along Python PyPI Topics (dataset)
<p>This dataset has been superseded by <strong><a title="https://zenodo.org/records/10480832" href="../records/10480832">https://zenodo.org/records/10480832</a></strong></p> <p> </p> <p>-- Test edit (new version?)</p>
One Model to Rule Them All: Cross-Language Detection of Malicious Packages for NPM and PyPI - Dataset, Models, Malicious Packages List
<p>This artifact complements the paper "One Model to Rule Them All: Language-Independent Detection of Malicious Packages for PyPI and NPM" submitted at ESEC/FSE 2023.</p> <p>In particular:</p> <ul> <li>"Labelled_Dataset.csv " contains the data used for the features evaluation, models evaluation and models training</li> <li>The .pkl files are the best-performing classifiers obtained after the models evaluation</li> <li>"Malicious_Packages_Discovered.csv" contain the list of malicious packages discovered during our real-world experiment</li> </ul>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.