Skip to content

Commit

Permalink
Adding docs about embeddings settings + adding the embedding.mode: lo…
Browse files Browse the repository at this point in the history
…cal in mock profile (zylon-ai#1316)
  • Loading branch information
gianniacquisto authored Nov 26, 2023
1 parent bafdd3b commit baf29f0
Show file tree
Hide file tree
Showing 2 changed files with 10 additions and 3 deletions.
10 changes: 7 additions & 3 deletions fern/docs/pages/manual/ingestion.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -33,16 +33,20 @@ Are you running out of memory when ingesting files?
To do not run out of memory, you should ingest your documents without the LLM loaded in your (video) memory.
To do so, you should change your configuration to set `llm.mode: mock`.

In other words, you should update your `settings.yaml` (or your custom configuration file) to set the
following **before** ingesting your documents:
You can also use the existing `PGPT_PROFILES=mock` that will set the following configuration for you:

```yaml
llm:
mode: mock
embedding:
mode: local
```
This configuration allows you to use hardware acceleration for creating embeddings while avoiding loading the full LLM into (video) memory.
Once your documents are ingested, you can set the `llm.mode` value back to `local` (or your previous custom value).

You can also use the existing `PGPT_PROFILES=mock` that will set the `llm.mode` to `mock` for you.


## Supported file formats

Expand Down
3 changes: 3 additions & 0 deletions settings-mock.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
server:
env_name: ${APP_ENV:mock}

# This configuration allows you to use GPU for creating embeddings while avoiding loading LLM into vRAM
llm:
mode: mock
embedding:
mode: local

0 comments on commit baf29f0

Please sign in to comment.