Multi-GPU Support

[mattwebbio]

Hi all 👋

I see CUDA support was added a few months back. Super exciting!

Is there anything preventing me from using multiple GPUs as was enabled in llama.cpp by this PR? I deploy LocalAI by Docker (currently CPU-only), would anything special need to change in the example GPU Docker Compose in the docs for multiple GPUs other than appending the second card to the device reservations?

[lunamidori5]

I think just change the number of GPU in the docker compose @mattwebbio

[BradKML]

Are there anything worth double-checking when doing multi-GPU?

[TwinFinz]

look up documentation on how to setup a custom llama model and setup the "TheBloke
/
Mistral-7B-OpenOrca-GGUF" model (use the Q4_K_M version) with the custom files (both "mistral.yaml" and "mistral-chat.tmpl") edit the docker compose to include the gpu options(shown in documentation) and the "rebuild=true" portion in the ".env" file. Then just "docker compose up -d" wget the Mistral model to the "models" folder along with the Mistral.yaml and mistral-chat.tmpl (you edit the examples to make these 2 files). bobs your uncle. it works on my 3080ti, 1070 ti and 2070 all fine. it does also work split between the 1070 ti and 2070 both using half the vram. but it is overall slower (idk why) and the accuracy seems to drop when split. i can run this specific model with 4k context fine on the 8gb cards but on my 3080 ti i can run 16k context (32k causes extreme slowdowns due to ram->vram transfers i assume).