Training Flux transformer with no baked-in VAE/text encoder

[PCanavelli]

Hi guys,

I'd like to train a Flux adapter on checkpoints stored as a Civitai-style pruned checkpoint, with no baked-in VAE or text encoder.

Does SimpleTuner provide a way to directly use this checkpoint as pretrained_model_name_or_path (or equivalent), and use Flux-dev for missing model parts?

The reason behind this is that converting checkpoints to a diffusers-like full model takes up a lot of time and compute, so I'd rather skip this step.

Cheers

[bghira]

any way that ST would load these files will do exactly the same conversion process as just converting it externally.

people who upload models for continued finetuning should ideally include a Diffusers layout version of the model.

if they didn't include one, i would say it is safe to assume they do not intend for users to tune it.

[PCanavelli]

Thanks for the rapid answer @bghira

Makes sense - expected as much to be honest, but was worth giving it a shot

Only point of disagreement: I'm pretty sure most fine-tuners don't upload diffusers-like weights because the majority of them aren't that technical and only know about libraries with a GUI

[bghira]

if there is a copy of it already on the huggingface hub (a lot of times people do this) i would just use that version as the pretrained model name or path.

if you just have the transformer folder, use --pretrained_transformer_model_name_or_path