Allow AdapterModels to have custom tokens

[mapmeld]

PEFT has a feature for adapters to add tokens to a model: https://github.com/huggingface/peft/blob/main/examples/causal_language_modeling/peft_lora_clm_with_additional_tokens.ipynb

When using an AdapterModel with new tokens in LightEval, the script fails because:

• AdapterModel._create_auto_tokenizer always uses the base model path, without checking config.tokenizer
• the new tokenizer and base model will have a mismatched number of embeddings

Notebook with error: https://colab.research.google.com/drive/1AMJ6_MiZGFTBf8KdRn-zj7soKyZrzpbf?usp=sharing

This PR would create the tokenizer from config.tokenizer or config.base_model
and run base.resize_token_embeddings(...) before PeftModel.from_pretrained(base, adapter_weights)
This is based on my fix for llm-evaluation-harness: EleutherAI/lm-evaluation-harness#1828

Notes:

• +1 please merge Fixes bug: You can't create a model without either a list of model_args or a model_config_path when model_config_path was submited. #298 (cherry-picked here), their one-line fix passed model_config_path to AdapterModel and this is necessary for most advanced models
• opinion: I think that adapter_weights and delta_weights should be optional so I do not need delta_weights: false in my config

[clefourrier]

Hi!
That works for me, but I would add a more detailed hlog message at the resize step: if you're adding tokens, we need to know how many, since they are likely to behave weirdly, and if you're removing tokens from the vocab, we would want to know which ones.
So a log like "You're using the adapter model's tokenizer, which has less/more tokens than its base: adding/removing X tokens" + in case of removing specifying which, would be useful.

You'll also need to run ruff to fix the style.

[clefourrier]

Re your comment, adapter_weights and delta_weights should be skippable by default, what's the error message you're getting when not specifying them?