Error Importing Package After Upgrading 0.0.12->0.0.16

[luisfrentzen-cc]

im trying to install exllama v0.0.16+cu121+cp310 as Im looking for support for cohere models, but after installing:

pip install https://github.com/turboderp/exllamav2/releases/download/v0.0.16/exllamav2-0.0.16+cu121-cp310-cp310-linux_x86_64.whl

I got this error message:

ImportError                               Traceback (most recent call last)

      1 import sys
----> 2 from exllamav2 import(
      3     ExLlamaV2,
      4     ExLlamaV2Config,
      5     ExLlamaV2Cache,
      6     ExLlamaV2Cache_8bit,
      7     ExLlamaV2Tokenizer,
      8     model_init,
      9 )
     10 import argparse
     11 import torch

File ~/.cache/pypoetry/virtualenvs/llm-api-LqunwCJV-py3.10/lib/python3.10/site-packages/exllamav2/__init__.py:3
      1 from exllamav2.version import __version__
----> 3 from exllamav2.model import ExLlamaV2
      4 from exllamav2.cache import ExLlamaV2CacheBase
      5 from exllamav2.cache import ExLlamaV2Cache

File ~/.cache/pypoetry/virtualenvs/llm-api-LqunwCJV-py3.10/lib/python3.10/site-packages/exllamav2/model.py:23
...
---> 15     import exllamav2_ext
     16 except ModuleNotFoundError:
     17     build_jit = True

ImportError: /home/username/.cache/pypoetry/virtualenvs/llm-api-LqunwCJV-py3.10/lib/python3.10/site-packages/exllamav2_ext.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c107WarningC1ESt7variantIJNS0_11UserWarningENS0_18DeprecationWarningEEERKNS_14SourceLocationESsb

is there any way to fix this? what causes this error? I was using version 0.0.12+cu121-cp310 and it worked perfectly fine before, thankyou beforehand

[turboderp]

I would try uninstalling exllamav2 and making sure the exllamav2_ext package is uninstalled in the process, then reinstalling from that wheel (or better yet a 0.0.17 wheel).

Also clear out ~/.cache/torch_extensions just in case Torch's build system has become confused somehow.

It's also possible there's a PyTorch version mismatch. They keep breaking extension compatibility with every new release and while you can build from source using pretty much any version, it's really hard to keep track of which versions are compatible with the prebuilt wheels. They're currently built for torch==2.2.0, so possibly matching that version could help.

[luisfrentzen-cc]

how can I made sure that exllamav2_ext is uninstalled? I am using poetry and I don't see it present in poetry run pip list. But I just tried clearing ~/.cache/torch_extensions and also downgrading to torch==2.2.0 (was using torch==2.2.2), but now I got a different error:

  File "<stdin>", line 1, in <module>
  File "/home/luis_frentzen/.cache/pypoetry/virtualenvs/exltest-7ParZrLj-py3.10/lib/python3.10/site-packages/exllamav2/__init__.py", line 3, in <module>
    from exllamav2.model import ExLlamaV2
  File "/home/luis_frentzen/.cache/pypoetry/virtualenvs/exltest-7ParZrLj-py3.10/lib/python3.10/site-packages/exllamav2/model.py", line 29, in <module>
    from exllamav2.attn import ExLlamaV2Attention
  File "/home/luis_frentzen/.cache/pypoetry/virtualenvs/exltest-7ParZrLj-py3.10/lib/python3.10/site-packages/exllamav2/attn.py", line 25, in <module>
    import flash_attn
  File "/home/luis_frentzen/.cache/pypoetry/virtualenvs/exltest-7ParZrLj-py3.10/lib/python3.10/site-packages/flash_attn/__init__.py", line 3, in <module>
    from flash_attn.flash_attn_interface import (
  File "/home/luis_frentzen/.cache/pypoetry/virtualenvs/exltest-7ParZrLj-py3.10/lib/python3.10/site-packages/flash_attn/flash_attn_interface.py", line 10, in <module>
    import flash_attn_2_cuda as flash_attn_cuda
ImportError: /home/luis_frentzen/.cache/pypoetry/virtualenvs/exltest-7ParZrLj-py3.10/lib/python3.10/site-packages/flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN2at4_ops5zeros4callEN3c108ArrayRefINS2_6SymIntEEENS2_8optionalINS2_10ScalarTypeEEENS6_INS2_6LayoutEEENS6_INS2_6DeviceEEENS6_IbEE```

I am using `flash-attn==2.5.6` currently, is there any problem with flash-attn somehow? I worked with a fresh new poetry environment to test this out

[luisfrentzen-cc]

building the package from source seems to work fine, thanks

[luisfrentzen-cc]

Hi @turboderp sorry for asking again. I tried to build my own wheel for the package that I installed from source as I need to install it elsewhere. But then I noticed that my newly created wheel doesn't seem to contain the extension library built in it, as it took quite a while to import the library on the first run. How do I build the extension library into the wheel to avoid this? sorry for the noob questions