How to prepare the dataset for nncf?

[jeck00119]

Hello! I have a dataset with the following structure:

valid(folder)->|
               |---> images(.jpg files)
               | 
               |---> labels(.txt files)

How to use the dataset for: val_dataset = datasets.ImageFolder("/patch", transform=transforms.Compose([transforms.ToTensor()]))
Also i want to quantise a onnx model.

[alexsu52]

Hi @jeck00119,

You can use any iterable Python object as dataset, not required to use datasets.ImageFolder(...) from torchvision. For example, nncf.quantize(...) does not require labels, thus you can use only images from your dataset and use python list as dataset. I mean the following:

# load images to the list from your valid(folder)
images_list = []
for file_name in Path(valid_dir).glob('*.jpg'):
    image = load_image(file_name)
    image = preprocess(image) # convert, resize, normalize and etc.
    images_list.append(image)

# create calibration dataset
calibration_dataset = nncf.Dataset(images_list)

If you want to use datasets.ImageFolder(...) from torchvision please align your dataset with required format: https://pytorch.org/vision/stable/generated/torchvision.datasets.ImageFolder.html.