Addressing a skill issue getting CUDA 12.1 to support the latest MSVC under Windows when building current exllamav2

[jim-plus]

Just add a line to the installation steps, and building exllamav2 will work again.

pip install -r requirements.txt
set NVCC_APPEND_FLAGS=-allow-unsupported-compiler
pip install .

[turboderp]

I'm not sure about any potential side effects from that. It might be fine, but it could break things, so definitely be careful with it.

For the github runners (that also updated to the broken version of VS Build Tools), bdashore3 fixed it with this hack to roll back the version. I think that's also what I would prefer on Windows, just don't update MSVC, or maybe update CUDA to a version that's validated against the newer version of MSVC.

[jim-plus]

I've been able to quant exl2 8bpw and run the result in ooba, though I consider this to be a stopgap measure. I see that technically Pytorch itself is still mid-migration to CUDA 12.4, and text-generation-webui uses flash attention wheels compiled against 12.2 or 12.3 despite using exllamav2 wheels against 12.1.