[Bug]: cannot load SD2 checkpoint after performance update in dev branch

[light-and-ray]

After the huge performance update in dev branch it is not possible to load sd2 model, while on master branch it works:

RuntimeError: mat1 and mat2 must have the same dtype, but got Half and Float

Loading weights [3f067a1b94] from /home/user/ai-apps/stable-diffusion-webui/models/Stable-diffusion/sd_v2-1_turbo.safetensors
loading stable diffusion model: RuntimeError
Traceback (most recent call last):
  File "/usr/lib/python3.11/threading.py", line 1002, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.11/threading.py", line 982, in run
    self._target(*self._args, **self._kwargs)
  File "/home/user/ai-apps/stable-diffusion-webui/modules/initialize.py", line 149, in load_model
    shared.sd_model  # noqa: B018
  File "/home/user/ai-apps/stable-diffusion-webui/modules/shared_items.py", line 175, in sd_model
    return modules.sd_models.model_data.get_sd_model()
  File "/home/user/ai-apps/stable-diffusion-webui/modules/sd_models.py", line 648, in get_sd_model
    load_model()
  File "/home/user/ai-apps/stable-diffusion-webui/modules/sd_models.py", line 736, in load_model
    checkpoint_config = sd_models_config.find_checkpoint_config(state_dict, checkpoint_info)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/ai-apps/stable-diffusion-webui/modules/sd_models_config.py", line 119, in find_checkpoint_config
    return guess_model_config_from_state_dict(state_dict, info.filename)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/ai-apps/stable-diffusion-webui/modules/sd_models_config.py", line 91, in guess_model_config_from_state_dict
    elif is_using_v_parameterization_for_sd2(sd):
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/ai-apps/stable-diffusion-webui/modules/sd_models_config.py", line 64, in is_using_v_parameterization_for_sd2
    out = (unet(x_test, torch.asarray([999], device=device), context=test_cond) - x_test).mean().item()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/ai-apps/stable-diffusion-webui/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/ai-apps/stable-diffusion-webui/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/ai-apps/stable-diffusion-webui/modules/sd_unet.py", line 91, in UNetModel_forward
    return original_forward(self, x, timesteps, context, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/ai-apps/stable-diffusion-webui/repositories/stable-diffusion-stability-ai/ldm/modules/diffusionmodules/openaimodel.py", line 789, in forward
    emb = self.time_embed(t_emb)
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/ai-apps/stable-diffusion-webui/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/ai-apps/stable-diffusion-webui/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/ai-apps/stable-diffusion-webui/venv/lib/python3.11/site-packages/torch/nn/modules/container.py", line 215, in forward
    input = module(input)
            ^^^^^^^^^^^^^
  File "/home/user/ai-apps/stable-diffusion-webui/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/ai-apps/stable-diffusion-webui/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/ai-apps/stable-diffusion-webui/extensions-builtin/Lora/networks.py", line 527, in network_Linear_forward
    return originals.Linear_forward(self, input)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/ai-apps/stable-diffusion-webui/venv/lib/python3.11/site-packages/torch/nn/modules/linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: mat1 and mat2 must have the same dtype, but got Half and Float


Stable diffusion model failed to load

[MrCheeze]

Any arguments being used?

[light-and-ray]

--medvram-sdxl --xformers --disable-model-loading-ram-optimization

I remember it worked on my rtx 3060 either with --xformers or --no-half. But now in dev it doesn't work with both. In master - okay

[huchenlei]

Suspected culprit PR: #15820

[huchenlei]

Culprit verified. master + #15820 can repro the issue.

[huchenlei]

--medvram-sdxl --xformers --disable-model-loading-ram-optimization

I remember it worked on my rtx 3060 either with --xformers or --no-half. But now in dev it doesn't work with both. In master - okay

Can you verify #16078 fixes the issue?