Good front end and any way to quickly load models without using terminal?

[Kaszebe]

I'm relatively new to AI. I recently completed a 5x 3090 build (and will be adding more as budget permits). I use it for work and learning (I have an interest in fine tuning).

I have two questions:

I am growing bored of Oobabooga and read that Aphrodite works great for multi-gpu rigs like mine. However, it has been a challenge to set up (due to my relative inexperience) and I had to resort to having ChatGPT help me. We got it set up correctly but I'm struggling to find a halfway decent GUI front end. Silly Tavern looks like role play stuff and that's of no interest to me. OpenWeb UI looked very nice but was unable to get it working with Aphrodite.

Is there a recommended front end for Aphrodite?

Question 2:

Is there a quicker way to switch between LLM models instead of using Terminal? Or do you guys just create script that you double click instead of having to type all that out into terminal when loading a model? If I'm not mistaken you have to load the model with all the parameters/etc?

And does anyone know if there is a halfway decent guide to optimizing Aphrodite? I mostly use GGUF and EXL2 models. I am also in the process of setting up my first fine-tune project.

[AlpinDale]

Hi and thanks for your interest!

It depends on what you're using Aphrodite for. Aphrodite creates an OpenAI-compatible API, meaning any frontend that works with an OpenAI API (and allows configuring the base OpenAI url to http://localhost), you should be fine. If you want a notebook style UI, I recommend mikupad, otherwise Open WebUI and LobeChat are good alternatives.

As for the second question, you can create a shell script and make it an executable. For example, save this in a start.sh file:

#/bin/bash
aphrodite run NousResearch/Meta-Llama-3.1-8B-Instruct -tp 4 --enable-chunked-prefill [other args go here as needed]

Depending on your desktop environment and your file manager, there should be an option for making it an executable when you right click on it. You can also do it from the terminal by running chmod +x start.sh. That should let you double-click the file to launch aphrodite.

On a somewhat related note, we will be getting support for asymmetric multi-GPU soon, e.g. allowing you to launch aphrodite on 5 GPUs using Tensor Parallelism.

[Kaszebe]

Ok, thanks. Will give Open WebUI another shot and see if I can get both connected. And dumb question...the model first gets loaded into Aphrodite and all OpenWeb UI does is connect via API and act as the pretty front end? I think that's where my initial misunderstanding was (and it was like at 3am when I tried to get everything connected...so I was exhausted).

I have like 3 favorite LLMs and I can easily set up a script for the desktop for each one.

On a somewhat related note, we will be getting support for asymmetric multi-GPU soon, e.g. allowing you to launch aphrodite on 5 GPUs using Tensor Parallelism.

Really looking forward to this! For right now, will I get better performance if I disconnect one of the 3090s? My models run ~70GB.

Thank you!

[AlpinDale]

will I get better performance if I disconnect one of the 3090s?

No, you don't need to do that. The -tp flag is for specifying the number of GPUs to use, it won't touch your 5th one for now if you pass 4. 4x3090 should give you 96GB which is plenty, I suppose. Let me know if you have more questions, happy to help out someone new to the AI community! :)

[Kaszebe]

Much appreciated! I do have one other question. I was an English major in college. I never saw a use for math until now...in my early 50s. I stopped at basic algebra in ~1995.

Do you think I will eventually reach a point where I cannot learn anymore due to my lack of math skills? I just started to learn Python on the advice of others. I have some minor coding/HTML/server experience.

My goal is to continue self-educating because I find this field incredibly fascinating. And if one day AI should ever "take my jerb," I would be able to present myself as a qualified candidate for some AI position. I'm a copywriter, and based on what I have seen so far—I have job security for the time being. But I'm betting that will eventually change in 5? 10? years from now.

[StefanGliga]

Do you think I will eventually reach a point where I cannot learn anymore due to my lack of math skills?

Using AI luckily doesn't require any math whatsoever, and there is plenty to do in the sphere of integrating AI into bigger systems. Even model finetuning doesn't require math(but it is a separate skill one has to learn). Math is only needed for fundamental research, and even then probably not as much math as you would expect, and vast majority of people won't be doing research anyway.

[Kaszebe]

Ah, good to know!