How to get local models working on Windows?

[ksylvan]

First Try: Using Fabric in WSL

I have a Windows 11 machine with an NVIDIA GeoForce RTX 4090 installed. Fabric is updated to the latest from Git.

I read about the Ollama Networking issue issue here: ollama/ollama#703

So, in PowerShell, I do this:

PS C:\Users\kayvan> $env:OLLAMA_HOST="0.0.0.0"

PS C:\Users\kayvan> ollama serve
time=2024-03-18T06:57:00.504-07:00 level=INFO source=images.go:806 msg="total blobs: 14"
time=2024-03-18T06:57:00.504-07:00 level=INFO source=images.go:813 msg="total unused blobs removed: 0"
time=2024-03-18T06:57:00.505-07:00 level=INFO source=routes.go:1110 msg="Listening on [::]:11434 (version 0.1.29)"
time=2024-03-18T06:57:00.505-07:00 level=INFO source=payload_common.go:112 msg="Extracting dynamic libraries to C:\\Users\\kayvan\\AppData\\Local\\Temp\\ollama3941994260\\runners ..."
time=2024-03-18T06:57:00.561-07:00 level=INFO source=payload_common.go:139 msg="Dynamic LLM libraries [rocm_v5.7 cpu_avx2 cuda_v11.3 cpu_avx cpu]"

Windows firewall then asks me to give permission to OLLAMA.EXE so it can serve the other interfaces.

Now in the WSL Ubuntu environment, I can see this:

kayvan@SHAKTI:/mnt/c/Users/kayvan/src/fabric$ ollama.exe list
NAME                    ID              SIZE    MODIFIED     
llama2:latest           78e26419b446    3.8 GB  2 weeks ago 
mistral:latest          61e88e884507    4.1 GB  13 hours ago
mixtral:instruct        7708c059a8bb    26 GB   13 hours ago
mixtral:latest          7708c059a8bb    26 GB   2 weeks ago

Now, using the default gateway as the address for the Ollama server URL:

kayvan@SHAKTI:/mnt/c/Users/kayvan/src/fabric$ cat /etc/resolv.conf 
# This file was automatically generated by WSL. To stop automatic generation of this file, add the following entry to /etc/wsl.conf:
# [network]
# generateResolvConf = false
nameserver 172.23.112.1

kayvan@SHAKTI:/mnt/c/Users/kayvan/src/fabric$ curl http://172.23.112.1:11434; echo ""
Ollama is running

I'm still not seeing the local models in fabric:

$ fabric --remoteOllamaServer http://172.23.112.1:11434 --listmodels
GPT Models:
gpt-3.5-turbo
gpt-3.5-turbo-0125
gpt-3.5-turbo-0301
gpt-3.5-turbo-0613
gpt-3.5-turbo-1106
gpt-3.5-turbo-16k
gpt-3.5-turbo-16k-0613
gpt-3.5-turbo-instruct
gpt-3.5-turbo-instruct-0914
gpt-4
gpt-4-0125-preview
gpt-4-0613
gpt-4-1106-preview
gpt-4-turbo-preview
gpt-4-vision-preview

Local Models:

Claude Models:
claude-3-opus-20240229
claude-3-sonnet-20240229
claude-3-haiku-20240307
claude-2.1

Any ideas for how to get local models working in Windows/WSL?

[ksylvan]

Still using WSL

To show that OLLAMA is working on my Windows box, here is the output from inside the WSL Ubuntu session:

kayvan@SHAKTI:/mnt/c/Users/kayvan/src/fabric$ curl -s -X POST -H "Content-Type: application/json" -d '{"model":"mistral", "prompt":"Why is the sky blue?", "stream": false}' http://172.29.112.1:11434/api/generate | jq 'del(.context)'
{
  "model": "mistral",
  "created_at": "2024-03-19T04:35:26.9660117Z",
  "response": " The color of the sky appears blue due to a process called Rayleigh scattering. When sunlight interacts with
      the Earth's atmosphere, shorter wavelengths of light such as blue and violet are more likely to be scattered in all
      directions than longer wavelengths like red, orange, and yellow. This scattering of blue light is what makes the sky
      appear blue during a clear day. However, at sunrise and sunset, when the sunlight passes through more of the Earth's
      atmosphere, the red and orange colors are also strongly scattered, giving us the beautiful hues we associate with
      those times of day.",
  "done": true,
  "total_duration": 5152356100,
  "load_duration": 3520299800,
  "prompt_eval_count": 15,
  "prompt_eval_duration": 92803000,
  "eval_count": 126,
  "eval_duration": 1538614000
}

And in Windows Powershell, the only real difference is using localhost:

PS C:\Users\kayvan\src\fabric> (Invoke-WebRequest -method POST -Body '{"model":"mistral", "prompt":"Why is the sky blue?", "stream": false}' -uri http://localhost:11434/api/generate ).Content | ConvertFrom-json
                                                                                                                        
model                : mistral
created_at           : 3/19/2024 4:45:45 AM
response             :  The color of the sky appears blue due to a phenomenon called Rayleigh scattering. When the sun's rays
                               enter Earth's atmosphere, they are scattered in all directions by the gases and particles in the air. 
                               Blue light has a shorter wavelength and gets scattered more easily than other colors such as red or yellow.
                               As a result, when we look up at the sky, we primarily see the blue light that has been scattered,
                               making the sky appear blue during a clear day. However, at sunrise and sunset, the sky can take on various
                               shades of red, pink, and orange due to the scattering of sunlight in the Earth's atmosphere at specific angles.
done                 : True
context              : {733, 16289, 28793, 28705…}
total_duration       : 5433303800
load_duration        : 3608535800
prompt_eval_count    : 15
prompt_eval_duration : 102676000
eval_count           : 141
eval_duration        : 1721152000

[ksylvan]

@danielmiessler Any ideas?

[ksylvan]

Fabric Running Natively on Windows 11

Still no luck with Local models 😢

I installed fabric (latest from git commit c3df1e7 ) from scratch on Windows 11, natively, using the new pipx setup. I basically used the python 3.12.2 setup on windows, then pip install pipx and then use pipx and follow the rest of the install instructions.

I also have ollama 0.1.29 on the same machine.

PS C:\Users\kayvan\src\fabric> ollama -v  
ollama version is 0.1.29
PS C:\Users\kayvan\src\fabric> ollama list
NAME                    ID              SIZE    MODIFIED    
llama2:latest           78e26419b446    3.8 GB  3 weeks ago
llava:latest            8dd30f6b0cb1    4.7 GB  2 days ago
mistral:latest          61e88e884507    4.1 GB  3 days ago
mixtral:instruct        7708c059a8bb    26 GB   3 days ago
mixtral:latest          7708c059a8bb    26 GB   3 weeks ago

And it's working correctly:

PS C:\Users\kayvan\src\fabric> Invoke-WebRequest -Uri http://localhost:11434
                                                                                                                        
StatusCode        : 200
StatusDescription : OK
Content           : Ollama is running
RawContent        : HTTP/1.1 200 OK
                    Date: Thu, 21 Mar 2024 22:10:17 GMT
                    Content-Type: text/plain; charset=utf-8
                    Content-Length: 17

                    Ollama is running
Headers           : {[Date, System.String[]], [Content-Type, System.String[]], [Content-Length, System.String[]]}
Images            : {}
InputFields       : {}
Links             : {}
RawContentLength  : 17
RelationLink      : {}

And here is the local model respnding:

PS C:\Users\kayvan\src\fabric> (Invoke-WebRequest -method POST -Body '{"model":"mistral", "prompt":"Why is the sky blue?", "stream": false}' -uri http://localhost:11434/api/generate ).Content | ConvertFrom-json
                                                                                                                        
model                : mistral
created_at           : 3/21/2024 10:10:53 PM
response             :  The color of the sky appears blue due to a process called scattering. When the Sun emits light, it sends out electromagnetic waves in all parts of the spectrum. However, Earth's atmosphere scatters
                       short-wavelength light (blue and violet) more effectively than longer wavelengths (yellow, orange, and red). As a result, when we look up at the sky, we predominantly see the blue light that has been
                       scattered in all directions. Additionally, some of the violet light gets absorbed by the ozone layer in the stratosphere, leaving us with more blue than violet.

                       However, it's important to note that the color of the sky isn't always blue - during sunrise or sunset, for instance, the sky can take on hues of pink, orange, and red due to the presence of more scattered    
                       longer wavelengths.
done                 : True
context              : {733, 16289, 28793, 28705…}
total_duration       : 4894802400
load_duration        : 3076050200
prompt_eval_count    : 15
prompt_eval_duration : 16438000
eval_count           : 187
eval_duration        : 1802114000

However, fabric still does not see the local models:

PS C:\Users\kayvan\src\fabric> fabric --listmodels
GPT Models:
gpt-3.5-turbo
gpt-3.5-turbo-0125
gpt-3.5-turbo-0301
gpt-3.5-turbo-0613
gpt-3.5-turbo-1106
gpt-3.5-turbo-16k
gpt-3.5-turbo-16k-0613
gpt-3.5-turbo-instruct
gpt-3.5-turbo-instruct-0914
gpt-4
gpt-4-0125-preview
gpt-4-0613
gpt-4-1106-preview
gpt-4-turbo-preview
gpt-4-vision-preview

Local Models:

Claude Models:
claude-3-opus-20240229
claude-3-sonnet-20240229
claude-3-haiku-20240307
claude-2.1

Anyone know what I can do to debug this?

Respond here or in this issue: #178

[ksylvan]

This is starting to get interesting. This is running in Windows 11 using Powershell. Activated the pipx venv environment, then did this:

(fabric) PS C:\Users\kayvan\src\fabric> python.exe
Python 3.12.2 (tags/v3.12.2:6abddd9, Feb  6 2024, 21:26:36) [MSC v.1937 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import ollama
>>> c = ollama.Client()
>>> l = c.list()
Traceback (most recent call last):
  File "C:\Users\kayvan\AppData\Local\pipx\pipx\venvs\fabric\Lib\site-packages\httpx\_transports\default.py", line 66, in map_httpcore_exceptions
    yield
  File "C:\Users\kayvan\AppData\Local\pipx\pipx\venvs\fabric\Lib\site-packages\httpx\_transports\default.py", line 228, in handle_request
    resp = self._pool.handle_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\kayvan\AppData\Local\pipx\pipx\venvs\fabric\Lib\site-packages\httpcore\_sync\connection_pool.py", line 216, in handle_request
    raise exc from None
  File "C:\Users\kayvan\AppData\Local\pipx\pipx\venvs\fabric\Lib\site-packages\httpcore\_sync\connection_pool.py", line 196, in handle_request
    response = connection.handle_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\kayvan\AppData\Local\pipx\pipx\venvs\fabric\Lib\site-packages\httpcore\_sync\connection.py", line 99, in handle_request
    raise exc
  File "C:\Users\kayvan\AppData\Local\pipx\pipx\venvs\fabric\Lib\site-packages\httpcore\_sync\connection.py", line 76, in handle_request
    stream = self._connect(request)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\kayvan\AppData\Local\pipx\pipx\venvs\fabric\Lib\site-packages\httpcore\_sync\connection.py", line 122, in _connect
    stream = self._network_backend.connect_tcp(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\kayvan\AppData\Local\pipx\pipx\venvs\fabric\Lib\site-packages\httpcore\_backends\sync.py", line 205, in connect_tcp
    with map_exceptions(exc_map):
  File "C:\Users\kayvan\scoop\apps\python\3.12.2\Lib\contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "C:\Users\kayvan\AppData\Local\pipx\pipx\venvs\fabric\Lib\site-packages\httpcore\_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectError: [WinError 10049] The requested address is not valid in its context

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\kayvan\AppData\Local\pipx\pipx\venvs\fabric\Lib\site-packages\ollama\_client.py", line 328, in list
    return self._request('GET', '/api/tags').json()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\kayvan\AppData\Local\pipx\pipx\venvs\fabric\Lib\site-packages\ollama\_client.py", line 68, in _request
    response = self._client.request(method, url, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\kayvan\AppData\Local\pipx\pipx\venvs\fabric\Lib\site-packages\httpx\_client.py", line 814, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\kayvan\AppData\Local\pipx\pipx\venvs\fabric\Lib\site-packages\httpx\_client.py", line 901, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\kayvan\AppData\Local\pipx\pipx\venvs\fabric\Lib\site-packages\httpx\_client.py", line 929, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\kayvan\AppData\Local\pipx\pipx\venvs\fabric\Lib\site-packages\httpx\_client.py", line 966, in _send_handling_redirects
    response = self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\kayvan\AppData\Local\pipx\pipx\venvs\fabric\Lib\site-packages\httpx\_client.py", line 1002, in _send_single_request
    response = transport.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\kayvan\AppData\Local\pipx\pipx\venvs\fabric\Lib\site-packages\httpx\_transports\default.py", line 227, in handle_request
    with map_httpcore_exceptions():
  File "C:\Users\kayvan\scoop\apps\python\3.12.2\Lib\contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "C:\Users\kayvan\AppData\Local\pipx\pipx\venvs\fabric\Lib\site-packages\httpx\_transports\default.py", line 83, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: [WinError 10049] The requested address is not valid in its context

Next, the same code with an explicit server address.

>>> c = ollama.Client(host="http://localhost:11434") 
>>> l = c.list()
>>> print(l)
{'models': [{'name': 'llama2:latest', 'model': 'llama2:latest', 'modified_at': '2024-03-30T16:44:31.5326054-07:00', 'size': 3826793677, 'digest': '78e26419b4469263f75331927a00a0284ef6544c1975b826b15abdaef17bb962', 'details': {'parent_model': '', 'format': 'gguf', 'family': 'llama', 'families': ['llama'], 'parameter_size': '7B', 'quantization_level': 'Q4_0'}}]}
>>>

[ksylvan]

Similarly in my WSL installed fabric:

kayvan@SHAKTI:/mnt/c/Users/kayvan/src/fabric$ . ~/.local/pipx/venvs/fabric/bin/activate
(fabric) kayvan@SHAKTI:/mnt/c/Users/kayvan/src/fabric$ python3
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ollama
>>> l = ollama.list()
Traceback (most recent call last):
  File "/home/kayvan/.local/pipx/venvs/fabric/lib/python3.10/site-packages/httpx/_transports/default.py", line 66, in map_httpcore_exceptions
    yield
  File "/home/kayvan/.local/pipx/venvs/fabric/lib/python3.10/site-packages/httpx/_transports/default.py", line 228, in handle_request
    resp = self._pool.handle_request(req)
  File "/home/kayvan/.local/pipx/venvs/fabric/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 216, in handle_request
    raise exc from None
  File "/home/kayvan/.local/pipx/venvs/fabric/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 196, in handle_request
    response = connection.handle_request(
  File "/home/kayvan/.local/pipx/venvs/fabric/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 99, in handle_request
    raise exc
  File "/home/kayvan/.local/pipx/venvs/fabric/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 76, in handle_request
    stream = self._connect(request)
  File "/home/kayvan/.local/pipx/venvs/fabric/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 122, in _connect
    stream = self._network_backend.connect_tcp(**kwargs)
  File "/home/kayvan/.local/pipx/venvs/fabric/lib/python3.10/site-packages/httpcore/_backends/sync.py", line 205, in connect_tcp
    with map_exceptions(exc_map):
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/kayvan/.local/pipx/venvs/fabric/lib/python3.10/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/kayvan/.local/pipx/venvs/fabric/lib/python3.10/site-packages/ollama/_client.py", line 328, in list
    return self._request('GET', '/api/tags').json()
  File "/home/kayvan/.local/pipx/venvs/fabric/lib/python3.10/site-packages/ollama/_client.py", line 68, in _request
    response = self._client.request(method, url, **kwargs)
  File "/home/kayvan/.local/pipx/venvs/fabric/lib/python3.10/site-packages/httpx/_client.py", line 814, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/home/kayvan/.local/pipx/venvs/fabric/lib/python3.10/site-packages/httpx/_client.py", line 901, in send
    response = self._send_handling_auth(
  File "/home/kayvan/.local/pipx/venvs/fabric/lib/python3.10/site-packages/httpx/_client.py", line 929, in _send_handling_auth
    response = self._send_handling_redirects(
  File "/home/kayvan/.local/pipx/venvs/fabric/lib/python3.10/site-packages/httpx/_client.py", line 966, in _send_handling_redirects
    response = self._send_single_request(request)
  File "/home/kayvan/.local/pipx/venvs/fabric/lib/python3.10/site-packages/httpx/_client.py", line 1002, in _send_single_request
    response = transport.handle_request(request)
  File "/home/kayvan/.local/pipx/venvs/fabric/lib/python3.10/site-packages/httpx/_transports/default.py", line 227, in handle_request
    with map_httpcore_exceptions():
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/kayvan/.local/pipx/venvs/fabric/lib/python3.10/site-packages/httpx/_transports/default.py", line 83, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: [Errno 111] Connection refused
>>> c = ollama.Client("http://172.23.128.1:11434")
>>> l = c.list()
>>> print(l)
{'models': [{'name': 'llama2:latest', 'model': 'llama2:latest', 'modified_at': '2024-03-30T16:44:31.5326054-07:00', 'size': 3826793677, 'digest': '78e26419b4469263f75331927a00a0284ef6544c1975b826b15abdaef17bb962', 'details': {'parent_model': '', 'format': 'gguf', 'family': 'llama', 'families': ['llama'], 'parameter_size': '7B', 'quantization_level': 'Q4_0'}}]}
>>>

[ksylvan]

Answer

After a few fixes in this PR: #315

Install Ollama on Windows (not in WSL)

Once it's installed, set the OLLAMA_HOST environment variable.

Set OLLAMA_HOST to 0.0.0.0 and once it is set, restart ollama.

This will make sure that ollama serves all interfaces, including the WSL network interface.

Install Fabric on Windows

Install git and Python on Windows

Once these are installed, clone the repository and install pipx. In PowerShell or Command prompt:

git clone https://github.com/danielmiessler/fabric.git
python.exe -m  pip install pipx

Then follow the usual process to install fabric:

cd fabric
pipx install .
fabric --setup

During the fabric setup process, add your API keys.

Additionally, Install Fabric on Windows WSL

In the WSL linux container, go to the directory on Windows where you cloned fabric and do this:

cd /mnt/c/Users/YourUser/src/fabric
sudo apt install pipx
pipx install .
fabric --setup

You can skip the steps of adding your API keys here since you can
simply copy the .env file from Windows from /mnt/c/Users/YourUser/.config/fabric/.env to ~/.config/fabric so you can run fabric
in the WSL partition.

Add the following snippet to your ~/.bashrc File

export OLLAMA_HOST=$(ip route | grep default | awk "{print \$3}")

This will make fabric in the WSL container talk to the ollama server running on the Windows host.

Demonstration

From Windows PowerShell

Listing models:

PS C:\Users\kayvan> fabric --listmodels
GPT Models:
gpt-3.5-turbo
gpt-3.5-turbo-0125
gpt-3.5-turbo-0301
gpt-3.5-turbo-0613
gpt-3.5-turbo-1106
gpt-3.5-turbo-16k
gpt-3.5-turbo-16k-0613
gpt-3.5-turbo-instruct
gpt-3.5-turbo-instruct-0914
gpt-4
gpt-4-0125-preview
gpt-4-0613
gpt-4-1106-preview
gpt-4-1106-vision-preview
gpt-4-turbo-preview
gpt-4-vision-preview

Local Models:
codellama:13b
codellama:latest
llama2:13b
llama2:latest
mistral:latest

Claude Models:
claude-3-opus-20240229
claude-3-sonnet-20240229
claude-3-haiku-20240307
claude-2.1

Running a local model:

PS C:\Users\kayvan> fabric --pattern ai --model llama2:13b --text 'What is the tallest tree in the world?'

🌲 The tallest tree in the world is a coast redwood (Sequoia sempervirens) named Hyperion, located in Redwood National Park, California, USA.

💪 Hyperion stands at an impressive 115.55 meters (379.1 feet) tall and has a diameter of 4.88 meters (16.0 feet) at breast height.

🌿 The tree's age is estimated to be around 700-800 years old, making it one of the oldest living things on Earth.

In WSL

List models:

kayvan@SHAKTI:/mnt/c/Users/kayvan$ fabric --listmodels
GPT Models:
gpt-3.5-turbo
gpt-3.5-turbo-0125
gpt-3.5-turbo-0301
gpt-3.5-turbo-0613
gpt-3.5-turbo-1106
gpt-3.5-turbo-16k
gpt-3.5-turbo-16k-0613
gpt-3.5-turbo-instruct
gpt-3.5-turbo-instruct-0914
gpt-4
gpt-4-0125-preview
gpt-4-0613
gpt-4-1106-preview
gpt-4-1106-vision-preview
gpt-4-turbo-preview
gpt-4-vision-preview

Local Models:
codellama:13b
codellama:latest
llama2:13b
llama2:latest
mistral:latest

Claude Models:
claude-3-opus-20240229
claude-3-sonnet-20240229
claude-3-haiku-20240307
claude-2.1

Running a local model:

kayvan@SHAKTI:/mnt/c/Users/kayvan$ fabric --pattern ai --model mistral:latest --text 'What is the tallest tree in the wo
rld?'
 The tallest tree in the world is the Hyperion Tree, measuring approximately 379.7 feet (115.7 m) tall.

[Samitch1]

Thank you. I have the same problem, and I tried all these steps on two machines, but I still can't see any local models from Ollama. Any help would be appreciated!

[ZA5542]

A few things to keep in mind:

1. WSL 2 connects to the host via a special DNS name when using Docker: host.docker.internal
2. Ollama does not listen on all interfaces by default
3. Windows firewall rules do not allow connections to port 11434 by default

So, here's how I got it working:

Set a System-level environment variable in Windows for OLLAMA_HOST to 0.0.0.0:11434. Then configure a firewall rule to allow inbound connections on port 11434. You can restrict this rule's scope to your private network or the host's IP address or whatever you decide. You can verify that Ollama is now listening on the correct interfaces by typing this command in you PowerShell:

netstat -ano | findstr :11434

Next, you need to configure your .env file in fabric .env file by adding this line:

OLLAMA_HOST=host.docker.internal

Again, this is the workaround that Docker in WSL uses to connect to services on the host. You can verify that you can connect to the Ollama port we opened in the firewall rule by running this command in WSL:

nc -zv host.docker.internal 11434

And to check that Ollama is running from WSL:

curl host.docker.internal:11434; echo ""

Once you set this up, your local models should show in fabric:

fabric --listmodels