inference

[yahya-mdarhri]

hello, for inference, is there a way to read a peft model directly from local repository, or we're obligated to pushtohub, and pss by hugingface, i would like to build a model that works locally without network

[alexflint]

You can pass a path to a local directory like this:

    adapter_path = "path/to/dir/containing/adapter"   # this dir must contain files adapter_config.json and adapter_model.bin

    # Load the config file
    config = PeftConfig.from_pretrained(adapter_path)

    # Setup the bitsandbytes config
    bnb_config = BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_use_double_quant=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.bfloat16,
    )

    # Load the base model
    base_model = AutoModelForCausalLM.from_pretrained(
        config.base_model_name_or_path,
        return_dict=True,
        quantization_config=bnb_config,
        device_map="auto",
        trust_remote_code=True,
    )

    # Load the tokenizer
    tokenizer = AutoTokenizer.from_pretrained(adapter_path)
    tokenizer.pad_token = tokenizer.eos_token

    # Load the adapter
    begin = timer()
    adapted_model = PeftModel.from_pretrained(base_model, adapter_path)

[yahya-mdarhri]

great!! thank you for your response