Find research datasets worth reusing

opencc-by-4.0May 2022View details →

ImageNet-Patch: A Dataset for Benchmarking Machine Learning Robustness against Adversarial Patches

Adversarial patches are optimized contiguous pixel blocks in an input image that cause a machine-learning model to misclassify it. However, their optimization is computationally demanding and requires careful hyperparameter tuning. To overcome these issues, we propose ImageNet-Patch, a dataset to benchmark machine-learning models against adversarial patches. It consists of a set of patches optimized to generalize across different models and applied to ImageNet data after preprocessing them with affine transformations. This process enables an approximate yet faster robustness evaluation, leveraging the transferability of adversarial perturbations. We release our dataset as a set of folders indicating the patch target label (e.g., `banana`), each containing 1000 subfolders as the ImageNet output classes. An example showing how to use the dataset is shown below. <pre><code class="language-python"># code for testing robustness of a model import os.path from torchvision import datasets, transforms, models import torch.utils.data class ImageFolderWithEmptyDirs(datasets.ImageFolder): """ This is required for handling empty folders from the ImageFolder Class. """ def find_classes(self, directory): classes = sorted(entry.name for entry in os.scandir(directory) if entry.is_dir()) if not classes: raise FileNotFoundError(f"Couldn't find any class folder in {directory}.") class_to_idx = {cls_name: i for i, cls_name in enumerate(classes) if len(os.listdir(os.path.join(directory, cls_name))) > 0} return classes, class_to_idx # extract and unzip the dataset, then write top folder here dataset_folder = 'data/ImageNet-Patch' available_labels = { 487: 'cellular telephone', 513: 'cornet', 546: 'electric guitar', 585: 'hair spray', 804: 'soap dispenser', 806: 'sock', 878: 'typewriter keyboard', 923: 'plate', 954: 'banana', 968: 'cup' } # select folder with specific target target_label = 954 dataset_folder = os.path.join(dataset_folder, str(target_label)) normalizer = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]) transforms = transforms.Compose([ transforms.ToTensor(), normalizer ]) dataset = ImageFolderWithEmptyDirs(dataset_folder, transform=transforms) model = models.resnet50(pretrained=True) loader = torch.utils.data.DataLoader(dataset, shuffle=True, batch_size=5) model.eval() batches = 10 correct, attack_success, total = 0, 0, 0 for batch_idx, (images, labels) in enumerate(loader): if batch_idx == batches: break pred = model(images).argmax(dim=1) correct += (pred == labels).sum() attack_success += sum(pred == target_label) total += pred.shape[0] accuracy = correct / total attack_sr = attack_success / total print("Robust Accuracy: ", accuracy) print("Attack Success: ", attack_sr) </code></pre>

opencc-by-4.0Jul 2022View details →

ImageNet-Cartoon and ImageNet-Drawing: two domain shift datasets for ImageNet

Benchmarking the robustness to distribution shifts traditionally relies on dataset collection which is typically laborious and expensive, in particular for datasets with a large number of classes like ImageNet. An exception to this procedure is ImageNet-C (Hendrycks & Dietterich, 2019), a dataset created by applying common real-world corruptions at different levels of intensity to the (clean) ImageNet images. Inspired by this work, we introduce ImageNet-Cartoon and ImageNet-Drawing, two datasets constructed by converting ImageNet images into cartoons and colored pencil drawings, using a GAN framework (Wang & Yu, 2020) and simple image processing (Lu et al., 2012), respectively. This repository contains ImageNet-Cartoon and ImageNet-Drawing. Checkout the <a href="https://github.com/oberman-lab/imagenet-shift">official GitHub Repo</a> for the code on how to reproduce the datasets. If you find this useful in your research, please consider citing:     @inproceedings{imagenetshift,       title={ImageNet-Cartoon and ImageNet-Drawing: two domain shift datasets for ImageNet},       author={Tiago Salvador and Adam M. Oberman},       booktitle={ICML Workshop on Shift happens: Crowdsourcing metrics and test datasets beyond ImageNet.},       year={2022}     }

opencc-by-4.0Aug 2022View details →

Model Zoo: A Dataset of Diverse Populations of Resnet-18 Models - Tiny ImageNet

Abstract In the last years, neural networks have evolved from laboratory environments to the state-of-the-art for many real-world problems. Our hypothesis is that neural network models (i.e., their weights and biases) evolve on unique, smooth trajectories in weight space during training. Following, a population of such neural network models (refereed to as “model zoo”) would form topological structures in weight space. We think that the geometry, curvature and smoothness of these structures contain information about the state of training and can be reveal latent properties of individual models. With such zoos, one could investigate novel approaches for (i) model analysis, (ii) discover unknown learning dynamics, (iii) learn rich representations of such populations, or (iv) exploit the model zoos for generative modelling of neural network weights and biases. Unfortunately, the lack of standardized model zoos and available benchmarks significantly increases the friction for further research about populations of neural networks. With this work, we publish a novel dataset of model zoos containing systematically generated and diverse populations of neural network models for further research. In total the proposed model zoo dataset is based on six image datasets, consist of 27 model zoos with varying hyperparameter combinations are generated and includes 50’360 unique neural network models resulting in over 2’585’360 collected model states. Additionally, to the model zoo data we provide an in-depth analysis of the zoos and provide benchmarks for multiple downstream tasks as mentioned before. Dataset This dataset is part of a larger collection of model zoos and contains the zoo of 1000 ResNet18 models trained on Tiny Imagenet. All zoos with extensive information and code can be found at www.modelzoos.cc. The complete zoo is 2.6TB large. Due to the size, this repository contains the checkpoints of the first 115 models at their last epoch 60. For a link to the full dataset as well as more information on the zoos and code to access and use the zoos, please see www.modelzoos.cc.

opencc-by-4.0Jul 2023View details →

ImageNet Mechanistic Interpretability

To enable research on automated alignment/interpretability evaluations, we release the experimental results of our paper "Scale Alone Does not Improve Mechanistic Interpretability in Vision Models" as a separate dataset. Note that this is the first dataset containing interpretability measurements obtained through psychophysical experiments for multiple explanation methods and models. The dataset contains >120'000 anonymized human responses, each consisting of the final choice, a confidence score, and a reaction time. Out of these >120'000 responses, > 69'000 passed all our quality assertions - this is the main data (see responses_main.csv). The other responses failed (some) quality assertions and might be of lower quality - they should be used with care (see responses_lower_quality.csv). We consider the former the main dataset and provide the latter as data for development/debugging purposes. Furthermore, the dataset contains the used query images as well as the generated explanations for >760 units across nine models.  The dataset itself is a collection of labels and metainformation without the presence of fixed features that should be predictive of a unit's interpretability. Moreover, finding and constructing features that are predictive of the recorded labels will be one of the open challenges posed by this line of research.

zenodo36/100

GTDA-based ReebNet for full ImageNet data with ResNet, VOLO, and AlexNet

This folder is used to produce the experiments of ImageNet reported in paper Topological structure of complex predictions. A preprint version of the paper is available on arXiv: https://arxiv.org/abs/2207.14358Unzip the three .tar.gz files. At the end, you should have a directory that looks like:<ul><li>exp_comparison_plotting_resnet_alexnet.py</li><li>exp_comparison_plotting_volo_resnet.py</li><li>GTDA</li><li>GTDA.tar.gz</li><li>GTDA.yml</li><li>ImageNet</li><li>ImageNet-details.txt</li><li>meta.mat</li><li>README.txt</li><li>resnet_alexnet</li><li>resnet_alexnet.tar.gz</li><li>volo_resnet</li><li>volo_resnet.tar.gz</li></ul>The ImageNet files are too large to include and are available from many other sources. You can verify you have the right setup by checking the ImageNet-details.txt file, which has the sha sum for each file we used along with the directory structure. There are additional instructions inside the REAMDE.txt file on how to setup the python environment to reproduce the figures.

opencc-by-4.0Oct 2023View details →

zenodo36/100

NINCO (Out-Of-Distribution detection dataset for ImageNet)

The NINCO (No ImageNet Class Objects) dataset is introduced in the ICML 2023 paper <a href="https://arxiv.org/abs/2306.00826">In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluation</a>. The images in this dataset are free from objects that belong to any of the 1000 classes of ImageNet-1K (ILSVRC2012), which makes NINCO suitable for evaluating out-of-distribution detection on ImageNet-1K . The NINCO main dataset consists of 64 OOD classes with a total of 5879 samples. These OOD classes were selected to have no categorical overlap with any classes of ImageNet-1K. Each sample was inspected individually by the authors to not contain ID objects. Besides NINCO, included are (in the same .tar.gz file) truly OOD versions of 11 popular OOD datasets with in total 2715 OOD samples. Further included are 17 OOD unit-tests, with 400 samples each. Code for loading and evaluating on each of the three datasets is provided at <a href="https://github.com/j-cb/NINCO">https://github.com/j-cb/NINCO</a>.   When using NINCO, please consider citing (besides the bibtex given below) the following data sources that were used to create NINCO: <ul> <li>Hendrycks et al.: ”Scaling out-of-distribution detection for real-world settings”, ICML, 2022.  </li> <li>Bossard et al.: ”Food-101 – mining discriminative components with random forests”, ECCV 2014.  </li> <li>Zhou et al.: ”Places: A 10 million image database for scene recognition”, IEEE PAMI 2017.  </li> <li>Huang et al.: ”Mos: Towards scaling out-of-distribution detection for large semantic space”, CVPR 2021.  </li> <li>Li et al.: ”Caltech 101 (1.0)”, 2022.</li> <li>Ismail et al.: ”MYNursingHome: A fully-labelled image dataset for indoor object classification.”, Data in Brief (V. 32) 2020.</li> <li>The iNaturalist project: https://www.inaturalist.org/</li> </ul> When using NINCO_popular_datasets_subsamples, additionally to the above, please consider citing: <ul> <li>Cimpoi et al.: ”Describing textures in the wild”, CVPR 2014.  </li> <li>Hendrycks et al.: ”Natural adversarial examples”, CVPR 2021.  </li> <li>Wang et al.: ”Vim: Out-of-distribution with virtual-logit matching”, CVPR 2022.  </li> <li>Bendale et al.: ”Towards Open Set Deep Networks”, CVPR 2016.  </li> <li>Vaze et al.: ”Open-set Recognition: a Good Closed-set Classifier is All You Need?”, ICLR 2022.  </li> <li>Wang et al.: ”Partial and Asymmetric Contrastive Learning for Out-of-Distribution Detection in Long-Tailed Recognition.” ICML, 2022.  </li> <li>Galil et al.: “A framework for benchmarking Class-out-of-distribution detection and its application to ImageNet”, ICLR 2023.  </li> </ul> For citing our paper, we would appreciate using the following bibtex entry (this will be updated once the ICML 2023 proceedings are public): @inproceedings{     bitterwolf2023ninco,     title={In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluation},     author={Julian Bitterwolf and Maximilian Mueller and Matthias Hein},     booktitle={ICML},     year={2023},     url={https://proceedings.mlr.press/v202/bitterwolf23a.html} }

opencc-by-4.0Jun 2023View details →

opencc-by-4.0Dec 2014View details →

Tiny ImageNet

TinyImageNet200 dataset created for Tiny-ImageNet Challenge at Stanford CS231N (<a href="http://cs231n.stanford.edu/reports/2015/pdfs/yle_project.pdf">http://cs231n.stanford.edu/reports/2015/pdfs/yle_project.pdf</a>)

opencc-by-4.0Jun 2021View details →

PASS: An ImageNet replacement for self-supervised pretraining without humans

Computer vision has long relied on ImageNet and other large datasets of images sampled from the Internet for pretraining models. However, these datasets have ethical and technical shortcomings, such as containing personal information taken without consent, unclear license usage, biases, and, in some cases, even problematic image content. On the other hand, state-of-the-art pretraining is nowadays obtained with unsupervised methods, meaning that labelled datasets such as ImageNet may not be necessary, or perhaps not even optimal, for model pretraining. We thus propose an unlabelled dataset PASS: Pictures without humAns for Self-Supervision. PASS only contains images with CC-BY license and complete attribution metadata, addressing the copyright issue. Most importantly, it contains no images of people at all, and also avoids other types of images that are problematic for data protection or ethics. We show that PASS can be used for pretraining with methods such as MoCo-v2, SwAV and DINO. In the transfer learning setting, it yields similar downstream performances to ImageNet pretraining even on tasks that involve humans, such as human pose estimation. PASS does not make existing datasets obsolete, as for instance it is insufficient for benchmarking. However, it shows that model pretraining is often possible while using safer data, and it also provides the basis for a more robust evaluation of pretraining methods. A simple download script is here: <a href="https://github.com/yukimasano/PASS/blob/main/download.sh">https://github.com/yukimasano/PASS/blob/main/download.sh</a> Visit our webpage here: <a href="https://www.robots.ox.ac.uk/~vgg/research/pass/">https://www.robots.ox.ac.uk/~vgg/research/pass/</a>

opencc-by-4.0Jul 2024View details →

Imagenet Features Extracted with VGG-19

This dataset contains features extracted from the Imagenet dataset using a pre-trained VGG-19 neural network. The network was configured with an input layer of (200, 200, 3). Feature extraction was performed using the Python package <a href="https://github.com/jgoodman8/py-image-features-extractor" target="_blank" rel="nofollow noopener noreferrer">Py Image Feature Extractor</a>.

openapache2.0Jul 2023View details →

Tiny ImageNet-C (Fixed)

Upload of the corrupted version of Tiny ImageNet (also known as ImageNet-200) with fixed frost-corrupted samples (see https://github.com/hendrycks/robustness/issues/60). This version can be downloaded on a server (for instance using <a href="https://github.com/ENSTA-U2IS/torch-uncertainty">TorchUncertainty</a>) and is safer than the original mirror on berkeley connect (that may soon be deleted). Original work by Dan Hendrycks & Thomas Dietterich under the <a href="https://github.com/hendrycks/robustness/blob/master/LICENSE">Apache-2.0 license</a>. If you consider this dataset useful, please cite: <pre><code>@article{hendrycks2019robustness, title={Benchmarking Neural Network Robustness to Common Corruptions and Perturbations}, author={Dan Hendrycks and Thomas Dietterich}, journal={Proceedings of the International Conference on Learning Representations}, year={2019} }</code></pre>

opencc-by-4.0Jun 2020View details →

Investigating Distributional Robustness: Semantic Perturbations Using Generative Models (ImageNet Examples)

This dataset contains examples of semantically-perturbed images, for NeurIPS 2020 submission #4915. There are four top-level folders, each containing results for semantic perturbations restricted to adjust the activation values at only certain layers of the BigGAN generative network: the first six layers, the middle six layers, the last six layers, and all layers. Within each top-level folder, there are a further four folders, each corresponding to a classifier neural network whose evaluation is being evaluated. These are EfficientNet-B4 with NoisyStudent training [1], the standard ResNet50 [2], a pixel-perturbation-robust ResNet50 trained by Engstrom et al. [3] and another trained by Wong et al. [4], using their "Fast is better than free" technique. Within each of these, there are many folders, named 'version_$N'. Each one of these contains three images: the unperturbed generated image, named unpert_generated_x_grid_0.png; the semantically-perturbed generated image, named generated_x_grid_0.png; and an image named semantic_pert_diffs_grid_0.png showing the pixel-space effect of the semantic perturbation, that is, the diff between the perturbed and unperturbed images. Note that if the perturbed and unperturbed images are identical, and the classifier misclassifies the unperturbed images, and so we skip this example. Along with the 'version_$N' folders containing the images, there exists a file for each classifier named results.json. Each top-level item in this JSON file corresponds to one 'version_$N' example. There are 5 attributes: 'label', indicating the target label of the unperturbed image; 'magnitude', which gives the magnitude of the semantic perturbation found; 'skipped_cla', which is 1 if the example is skipped because the classifier did not correctly classify the unperturbed image; 'skipped_judge', which is 1 if the human judged that the unperturbed image did not match its label, so this example is skipped; and 'pert_judgement', which is 1 if the semantically-perturbed image is judged by the human to be of the same class as the unperturbed image. These judgements on these images were used to construct the main graphs in the paper.   [1] Qizhe Xie, Eduard H. Hovy, Minh-Thang Luong, and Quoc V. Le. Self-training with Noisy Student improves ImageNet classification. CoRR, abs/1911.04252, 2019. URL http://arxiv.org/abs/1911.04252. [2] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770–778. IEEE Computer Society, 2016. doi: 10.1109/CVPR.2016.90. URL https://doi.org/10.1109/CVPR.2016.90. [3] Logan Engstrom, Andrew Ilyas, Shibani Santurkar, and Dimitris Tsipras. Robustness (Python library), 2019. URL ttps://github.com/MadryLab/robustness. [4] Eric Wong, Leslie Rice, and J. Zico Kolter. Fast is better than free: Revisiting adversarial training. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://openreview.net/forum?id=BJx040EFvH.

SAMPLER representations of TCGA-BRCA DX WSIs using an InceptionV3 backbone pretrained on imagenet

Open the record for dataset details and reuse information.

SAMPLER representations of CPTAC-LSCC WSIs using an InceptionV3 backbone pretrained on imagenet

Open the record for dataset details and reuse information.

SAMPLER representations of frozen and FFPE TCGA-KICH WSIs using an InceptionV3 backbone pretrained on imagenet

Open the record for dataset details and reuse information.

SAMPLER representations of TCGA-BRCA frozen WSIs using an InceptionV3 backbone pretrained on imagenet

Open the record for dataset details and reuse information.

SAMPLER representations of CPTAC-LUAD WSIs using an InceptionV3 backbone pretrained on imagenet

Open the record for dataset details and reuse information.

SAMPLER representations of frozen and FFPE TCGA-KIRC WSIs using an InceptionV3 backbone pretrained on imagenet

Open the record for dataset details and reuse information.

SAMPLER representations of CPTAC-BRCA WSIs using an InceptionV3 backbone pretrained on imagenet

Open the record for dataset details and reuse information.

SAMPLER representations of frozen and FFPE TCGA-LUSC WSIs using an InceptionV3 backbone pretrained on imagenet

Open the record for dataset details and reuse information.