Must have a huggingface token to get default tokenizer for local setups.

[BradHutchings]

This changed recently. You now have to provide a token and sign up on Hugging Face to get the default tokenizer for local setups.

1. Make a HuggingFace account. https://huggingface.co
2. Create a read token. https://huggingface.co/settings/tokens
3. Go to https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2 and agree to give them your info.
4. Edit scripts/setup in your downloaded copy of the repo.
5. Find AutoTokenizer near bottom, add a line after cache_dir: token=settings().huggingface.access_token,
6. From shell command line: export HUGGINGFACE_TOKEN="[your-hf-token-here-no-braces]"
7. Then you can run: poetry run python scripts/setup

It looks like someone is in the middle of tracking down all the places to use the hf token. Please finish and test ASAP. Please update the documentation for local installs.

[ykanfi]

Apologies for the newb question here but what am I doing wrong? Is the export meant to be called outside of AutoTokenizer (I know to replace with my token)?

AutoTokenizer.from_pretrained(
    pretrained_model_name_or_path=settings().llm.tokenizer,
    cache_dir=models_cache_path,
    token=settings().huggingface.access_token,
    export HUGGINGFACE_TOKEN="[your-hf-token-here-no-braces]",
)

[BradHutchings]

I'm sorry I wasn't clear. Call this from shell command line:

export HUGGINGFACE_TOKEN="[your-hf-token-here-no-braces]"

I edited my original post above to clear this up.

[KansaiTraining]

I would like to ask these questions regarding this

• What exactly is a tokenizer and what does it do?
• Why- in a local setup- do we need to access an external resource? Is this a one time only? or every time?
• What happens if we deactivate the tokenizer?

[BradHutchings]

I Googled that for you: "Tokenization is a fundamental step in LLMs. It is the process of breaking down text into smaller subword units, known as tokens. We recently open-sourced our tokenizer at Mistral AI."

The local setup has to have all the things running... guess where... locally. So that's why it needs to be installed.

If you deactivate the tokenizer, then it won't be able to turn your prompts into tokens to send to the LLM.

HTH.

-Brad

[KansaiTraining]

The local setup has to have all the things running... guess where... locally. So that's why it needs to be installed.

So that means the access in one time only? and then I dont need this access anymore, since it is local, right?

The question is not trivial. In my development group we are planning to adopt privateGPT and use it locally and everyone from the project manager down is asking, why a "local" set up needs external access

[BradHutchings]

After you install the local stuff, correct. You don't need to install anything from HF after that.

You should be able to zip up your privateGPT folder after installing everything and pass that around. To get all the python dependencies, others would just install up to the point where it fails. Then unzip you privateGPT folder and go from there.

Also, it's paid per user, but I offer this whole thing packaged in a virtual machine for Windows 11 Pro. Plus a launcher that keeps you out of Hyper-V Manager. https://brad-gpt.com/privategpt/