Skip to content

Commit

Permalink
Feat (example/llm): Specify LLMs to use SDPA for their attn implement…
Browse files Browse the repository at this point in the history
…ation
  • Loading branch information
nickfraser committed Nov 18, 2024
1 parent 87c4d70 commit d54c832
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion src/brevitas_examples/llm/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -183,7 +183,7 @@ def main(args):
kwargs['torchscript'] = True

print("Model loading...")
model = AutoModelForCausalLM.from_pretrained(args.model, **kwargs)
model = AutoModelForCausalLM.from_pretrained(args.model, attn_implementation="sdpa", **kwargs)
print("Model loaded.")
model.eval()
tokenizer = AutoTokenizer.from_pretrained(args.model)
Expand Down

0 comments on commit d54c832

Please sign in to comment.