Installation of fabric in a python3 virtualenvironment and then installing ollama with local LLM:s

[robscurity]

Hi everybody, I just want to show how I got ollama working with fabric.

Installation of fabric in a python3 virtualenvironment and then installing ollama with local LLM:s

Virtualenvironments to prevent problems with other python projects

Install fabric:

cd /where/you/keep/code
git clone https://github.com/danielmiessler/fabric.git # Clone Fabric to your computer
python3 -m venv fabric #create virtualenvironment
cd fabric
source bin/activate #activate virtualenv
chmod +x setup.sh
curl -sSL https://install.python-poetry.org | python3 -
source ~/.zshrc #reload .zshrc
poetry --version #check if poetry is working
./setup.sh #start installation of fabric
source ~/.zshrc #reload .zshrc
vi ~/.config/fabric/.env #create .env file. below is your personal OpenAI key from openai.com
# OPENAI_API_KEY=llama2
OPENAI_API_KEY=
# OPENAI_BASE_URL=http://localhost:114341

Istalling ollama:
Download client from ollama.com and install it.
In terminal:

ollama pull llama2 #download llama2:latest. Wait until download finished.
ollama list

Check if fabric also can "see" the local models

fabric --listmodels

Try it by copy some text and then

pbpaste | fabric -m llama2:latest -p summarize

[andreabolandrina]

Thanks for this. It all worked up to running the "listmodels" command.

$ fabric --listmodels Error: Client error '404 Not Found' for url 'http://localhost:11434/models' For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404

I have the models:
$ ollama list NAME ID SIZE MODIFIED codellama:latest 8fdf8f752f6e 3.8 GB 7 hours ago falcon:latest 4280f7257e73 4.2 GB 6 hours ago gemma:latest 430ed3535049 5.2 GB 7 hours ago llama2:13b d475bf4c50bc 7.4 GB 6 hours ago llama2:latest 78e26419b446 3.8 GB 8 hours ago llama2-uncensored:latest 44040b922233 3.8 GB 6 hours ago mistral:latest 61e88e884507 4.1 GB 7 hours ago openchat:latest 537a4e03b649 4.1 GB 6 hours ago phi:latest e2fd6321a5fe 1.6 GB 6 hours ago vicuna:latest 370739dc897b 3.8 GB 6 hours ago

Ollama is listening locally:
sudo netstat -tunlp|grep -i lama tcp 0 0 127.0.0.1:11434 0.0.0.0:* LISTEN 4112467/ollama

NB: in your code you have an extra "1" at the end of the port.

and this is my config:
$ cat ~/.config/fabric/.env OPENAI_API_KEY=llama2 OPENAI_BASE_URL=http://localhost:11434

Any clue on what I might be missing would be highly appreciated, thanks!

[andreabolandrina]

for some reason, commenting out everything I had in the .env file made it work.
It uses my local ollama by default...

[bkartel123]

Hi, i have the same problem. i have followed the instructions on osx to configure it with ollama but i am getting the error: Error: Client error '404 Not Found' for url 'http://localhost:11434/v1/models'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404

every other application i have adopted to ollama has worked fine with http://localhost:11434/v1/ . i think the problem is that somewhere /models is hard coded in to the application and when it look up http://localhost:11434/v1/models it will not get the proper repose from ollama . this is the responce of ollama to that request: [GIN] 2024/05/19 - 17:01:22 | 404 | 13.834µs | 127.0.0.1 | GET "/v1/models"

it would be great this could work with ollama ...