Skip to content
/ deic Public

Benchmark for Data-Efficient Image Classification

Notifications You must be signed in to change notification settings

cvjena/deic

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

3 Commits
 
 
 
 
 
 

Repository files navigation

Data-Efficient Deep Learning Benchmark (DEIC)

This repository contains descriptions, download instructions, and canonical train/validation/test splits for the six datasets used as a benchmark for data-efficient image classification in the following paper:

Lorenzo Brigato, Björn Barz, Luca Iocchi, and Joachim Denzler.
Tune It or Don't Use It: Benchmarking Data-Efficient Image Classification.
ICCV 2021 Workshop on Visual Inductive Priors for Data-Efficient Deep Learning (VIPriors).

⚠️ For the newest version of the benchmark, and the full PyTorch implementations of several data-efficient methods, consider visiting:

https://github.com/lorenzobrigato/gem

Datasets

Dataset Classes Imgs/Class Trainval Test Problem Domain Data Type
ImageNet 1,000 50 50,000 50,000 Natural Images RGB
ciFAIR-10 10 50 500 10,000 Natural Images RGB (32x32)
CUB 200 30 5,994 5,794 Fine-Grained RGB
EuroSAT 10 50 500 19,500 Remote Sensing Multispectral
ISIC 2018 7 80 560 1,944 Medical RGB
CLaMM 12 50 600 2,000 Handwriting Grayscale

For all datasets except CUB, we use subsampled versions of the original datasets with fewer images per class.

Canonical Splits

The datasets directory contains one sub-directory for each dataset in our benchmark. These directories contain the files train.txt, val.txt, and test.txt specifying the three subsets. Additionally, the file trainval.txt is simply the concatenation of train.txt and val.txt. This subset can be used for the final training before evaluating a method on the test set. Development and hyper-parameter optimization, however, should only be conducted using the training and validation sets.

The aforementioned files list all images contained in the respective subset, one per line, along with their class labels. Each line contains the filename of an image followed by a space and the numeric index of its label.

The only exception from this common format is ciFAIR-10, since it does not have filenames. A description of the split can be found in the README.md of the respective directory.