✨ Text generation web UI #2647

MurkleMan · 2024-03-02T23:55:41Z

MurkleMan
Mar 2, 2024

A Gradio web UI for Large Language Models. Supports transformers, GPTQ, AWQ, EXL2, llama.cpp (GGUF), Llama models.

3 interface modes: default (two columns), notebook, and chat.
Multiple model backends: Transformers, llama.cpp (through llama-cpp-python), ExLlamaV2, AutoGPTQ, AutoAWQ, GPTQ-for-LLaMa, CTransformers, QuIP#.
Dropdown menu for quickly switching between different models.
Large number of extensions (built-in and user-contributed), including Coqui TTS for realistic voice outputs, Whisper STT for voice inputs, translation, multimodal pipelines, vector databases, Stable Diffusion integration, and a lot more. See the wiki and the extensions directory for details.
Chat with custom characters.
Precise chat templates for instruction-following models, including Llama-2-chat, Alpaca, Vicuna, Mistral.
LoRA: train new LoRAs with your own data, load/unload LoRAs on the fly for generation.
Transformers library integration: load models in 4-bit or 8-bit precision through bitsandbytes, use llama.cpp with transformers samplers (llamacpp_HF loader), CPU inference in 32-bit precision using PyTorch.
OpenAI-compatible API server with Chat and Completions endpoints -- see the examples.

snapshot-2024-02-25

AGPL-3.0 license

No response

I confirm that I've put the app name in the title of the discussion
I confirm that the application is actively maintained.
I confirm that the docker image of the application is actively maintained.
I confirm that I have indeed checked that no similar request for this application already exists

Hey folks!

Please upvote ⬆️ this discussion to show your interest in this request!

Thanks ⛺