Fix LLM tests #1088
Fix LLM tests #1088
Conversation
tests/brevitas_examples/test_llm.py
Outdated
| "act_equalization": "layerwise", | ||
| "gptq": True, | ||
| "float_ppl": 31056.0, # 31274.05078125 | ||
| "quant_ppl": 33056.0},]) # 33139.23046875},]) |
There was a problem hiding this comment.
What do the numbers in the comments mean?
There was a problem hiding this comment.
Those were the numbers for transformers < 4.46.0. They need to be removed, but I wanted to get confirmation on the preferred way to go regarding the transformers version.
There was a problem hiding this comment.
Thanks - can you add that to the comment?
There was a problem hiding this comment.
Maybe we can have a dict that pulls out different reference accuracy based on transformer version? It might get annoying in the long run to keep track of everything, but it could also help catch regression that we might have initially attributed to "numbers have changed"
|
I think only @Giuseppe5's comment needs to be addressed, otherwise this can be merged IMO. |
|
There's a bunch of test failing on the extended test suite @pablomlago |
pablomlago
force-pushed
the
fix-llm-tests
branch
from
c9c8206
to
5e4149c
Compare
Reason for this PR
See #1076.
Changes Made in this PR
Since transformers 4.46.0 (https://github.com/huggingface/transformers/releases/tag/v4.46.0), OPT attention mechanism relies, by default, on OPTSdpaAttention, thus resulting in the method replace_mha_with_quantizable_layers (https://github.com/pablomlago/brevitas/blob/master/src/brevitas_examples/llm/llm_quant/prepare_for_quantize.py#L11) not injecting the Brevitas layers.
Additionally, the signature of the forward function in OPTAttention was updated in transformers 4.46.0, thus requiring to update the signature of QuantMultiHeadAttention analogously, even if this attribute is not actually used.
Testing Summary
TODO
Risk Highlight
Checklist
devbranch.