Image preprocessor default when loading checkpoints

[jn2clark]

Hi,

I noticed something when loading checkpoints other than the pretrained ones and wanted to understand what the intended behavior was. For example, loading and saving a B32 pretrained checkpoint like the following;

import open_clip

architecture1 = 'ViT-B-32'
pretrained1 = 'laion2b_s34b_b79k'
checkpoint1 = "B32.pt"
model, _, preprocess = open_clip.create_model_and_transforms(architecture1, pretrained=pretrained1)
print(preprocess)

torch.save(model.state_dict(), checkpoint1)

model, _, preprocess = open_clip.create_model_and_transforms(architecture1, pretrained=checkpoint1)
print(preprocess)

results in the prepocess having the same configurations across both methods. It looks like

Compose(
    Resize(size=224, interpolation=bicubic, max_size=None, antialias=True)
    CenterCrop(size=(224, 224))
    <function _convert_to_rgb at 0x7f25cb351900>
    ToTensor()
    Normalize(mean=(0.48145466, 0.4578275, 0.40821073), std=(0.26862954, 0.26130258, 0.27577711))
)
Compose(
    Resize(size=224, interpolation=bicubic, max_size=None, antialias=True)
    CenterCrop(size=(224, 224))
    <function _convert_to_rgb at 0x7f25cb351900>
    ToTensor()
    Normalize(mean=(0.48145466, 0.4578275, 0.40821073), std=(0.26862954, 0.26130258, 0.27577711))
)

However, when using an architecture that does not share the same defaults results in different behavior. For example, running the above for a SigLip model;

architecture2 = 'ViT-B-16-SigLIP'
pretrained2 = 'webli'
checkpoint2 = "SigLip.pt"
model, _, preprocess = open_clip.create_model_and_transforms(architecture2, pretrained=pretrained2)
print(preprocess)
torch.save(model.state_dict(), checkpoint2)

model, _, preprocess = open_clip.create_model_and_transforms(architecture2, pretrained=checkpoint2)
print(preprocess)

results in the preprocess for loading the saved checkpoint to use default settings rather than the settings for the SigLip architecture.

Compose(
    Resize(size=(224, 224), interpolation=bicubic, max_size=None, antialias=True)
    <function _convert_to_rgb at 0x7f25cb351900>
    ToTensor()
    Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5))
)
Compose(
    Resize(size=224, interpolation=bicubic, max_size=None, antialias=True)
    CenterCrop(size=(224, 224))
    <function _convert_to_rgb at 0x7f25cb351900>
    ToTensor()
    Normalize(mean=(0.48145466, 0.4578275, 0.40821073), std=(0.26862954, 0.26130258, 0.27577711))

Is this intended behavior? I understand you can pass through the parameters for the preprocess explicitly but I had expected the preprocess to inherit defaults from the architecture since these seem to be already used for loading pretrained. For example -

open_clip/src/open_clip/pretrained.py

Line 42 in fc5a37b

def _slpcfg(url='', hf_hub='', **kwargs):

Thanks!

[rwightman]

@jn2clark there are no defaults for the architecture, the arch config covers only the model. The preprocess cfg (mean/std) are part of the pretrained mappings and there is one per pretrained weight instance, which could have many for one arch, they can also be specified in the HF hub config which includes both the preproces and arch config.

So, without adding functionality to pass a file containing preprocess config (or the full HF hub like config with both) separately or in a folder with the checkpoint, you can't easily pass those around locally without using the args.

[rwightman]

I have thought about this a little be in context of #883 (that solution doesn't work) but could add support for saving/loading folder w/ the full config + checkpoint.

[jn2clark]

Thanks @rwightman . The HF method seems to be the best given it fully specifies everything