Improve SD35 LoRA support to cover most popular LoRA formats

[vladmandic]

SD3.x pipeline does implement SD3LoraLoaderMixin and as such load_lora_weights on SD3.x does "work".

However, attempting to load any of the most popular LoRAs results in silent failure:
load is successful without any warnings, but loads ZERO keys.

Looking at implementation at:

diffusers/src/diffusers/loaders/lora_pipeline.py

Line 1214 in 345907f

def load_lora_weights(

Shows that if there are no matching keys in text_encoder_state_dict, text_encoder_state_dict, text_encoder_2_state_dict, there is no warning raised, the method simply does NOTHING.

When looking at state_dict of loaded LoRA, it shows that keys are NOT in the expected format - they need remapping.
For example:
dict_keys(['lora_unet_joint_blocks_0_context_block_adaLN_modulation_1.alpha', 'lora_unet_joint_blocks_0_context_block_adaLN_modulation_1.lora_down.weight', 'lora_unet_joint_blocks_0_context_block_adaLN_modulation_1.lora_up.weight', 'lora_unet_joint_blocks_0_context_block_attn_proj.alpha' ...])

Below are download links for some of the popular LoRAs for SD35, all experience same behavior:

• https://civitai.com/models/943882/tokino-sd35-ambient-scene?modelVersionId=1056722
• https://civitai.com/models/539031/lineart-sd35fluxsdxlpony?modelVersionId=1052194
• https://civitai.com/models/948259/brian-sum-mech-lora-sd35-large?modelVersionId=1061663

CC @yiyixuxu @sayakpaul @DN6 @asomoza
CC @AI-Casanova

[sayakpaul]

Ah I know why this is happening.

It's because when we supported SD3 LoRA, there weren't popular non-diffusers LoRAs. But clearly, things have changed now. I will look into it.