DistilKaggle: a distilled dataset of Kaggle Jupyter notebooks

<h2><strong>Overview</strong></h2> <p>DistilKaggle is a curated dataset extracted from Kaggle Jupyter notebooks spanning from September 2015 to October 2023. This dataset is a distilled version derived from the download of over 300GB of Kaggle kernels, focusing on essential data for research purposes. The dataset exclusively comprises publicly available Python Jupyter notebooks from Kaggle. The essential information for retrieving the data needed to download the dataset is obtained from the MetaKaggle dataset provided by Kaggle.</p> <h2><strong>Contents</strong></h2> <p>The DistilKaggle dataset consists of three main CSV files:</p> <p><strong>code.csv:</strong> Contains over 12 million rows of code cells extracted from the Kaggle kernels. Each row is identified by the kernel's ID and cell index for reproducibility.</p> <p><strong>markdown.csv:</strong> Includes over 5 million rows of markdown cells extracted from Kaggle kernels. Similar to <strong>code.csv</strong>, each row is identified by the kernel's ID and cell index.</p> <p><strong>notebook_metrics.csv:</strong> This file provides notebook features described in the accompanying paper released with this dataset. It includes metrics for over 517,000 Python notebooks.</p> <h2><strong>Directory Structure</strong></h2> <p>The <strong>kernels</strong> directory is organized based on Kaggle's Performance Tiers (PTs), a ranking system in Kaggle that classifies users. The structure includes PT-specific directories, each containing user ids that belong to this PT, download logs, and the essential data needed for downloading the notebooks.</p> <p>The <strong>utility</strong> directory contains two important files:</p> <p><strong>aggregate_data.py:</strong> A Python script for aggregating data from different PTs into the mentioned CSV files.</p> <p><strong>application.ipynb:</strong> A Jupyter notebook serving as a simple example application using the metrics dataframe. It demonstrates predicting the PT of the author based on notebook metrics.</p> <p><strong>DistilKaggle.tar.gz: </strong>It is just the compressed version of the whole dataset. If you downloaded all of the other files independently already, there is no need to download this file.</p> <h2><strong>Usage</strong></h2> <p>Researchers can leverage this distilled dataset for various analyses without dealing with the bulk of the original 300GB dataset. For access to the raw, unprocessed Kaggle kernels, researchers can request the dataset directly.</p> <h2><strong>Note</strong></h2> <p>The original dataset of Kaggle kernels is substantial, exceeding 300GB, making it impractical for direct upload to Zenodo. Researchers interested in the full dataset can contact the dataset maintainers for access.</p> <h2><strong>Citation</strong></h2> <p>If you use this dataset in your research, please cite the accompanying paper or provide appropriate acknowledgment as outlined in the documentation.</p> <p>If you have any questions regarding the dataset, don't hesitate to contact me at <a href="mailto:mohammad.abolnejadian@gmail.com">mohammad.abolnejadian@gmail.com</a></p> <p>Thank you for using DistilKaggle!</p>