Breaking Changes
- Remove ONNX QOp export (#917)
- QuantTensor cannot have empty metadata fields (e.g., scale, bitwidth, etc.) (#819)
- Bias quantization now requires the specification of bit-width (#839)
- QuantLayers do not expose quant_metadata directly. This is delegated to the proxies (#883)
- QuantDropout has been removed (#861)
- QuantMaxPool has been removed (#858)
Highlights
-
Support for OCP/FNUZ FP8 quantization
- Compatibility with QAT/PTQ, including all current PTQ algorithms implemented (GPTQ, LearnedRound, GPFQ, etc.)
- Possibility to fully customize the minifloat configuration (i.e., select mantissa/exponent bit-width, exponent bias, etc.)
- Support for ONNX QDQ export
-
Support for OCP MX Quantization
- Compatibility with QAT/PTQ, including all current PTQ algorithms implemented (GPTQ, LearnedRound, GPFQ, etc.)
- Possibility to fully customize the minifloat configuration (i.e., select mantissa/exponent bit-width, exponent bias, group size, etc.)
-
New QuantTensor supports:
- FloatQuantTensor: supports OCP FP formats and general minifloat quantization
- GroupwiseQuantTensor: supports for OCP MX formats and general groupwise int/minifloat quantization
-
Support for Channel splitting
-
Support for HQO optimization for zero point
-
Support for HQO optimization for scale (prototype)
-
Improved SDXL entrypoint under brevitas_examples
-
Improved LLM entrypoint under brevitas_examples
- Compatibility with accelerate
-
Prototype support for
torch.compile
:- Check PR #1006 for an example on how to use it
What's Changed
For a more comprehensive list of changes and fix, check the list below:
- Enhance: Importing quantized models after bias correction by @costigt-dev in #868
- Fix QCDQDecoupledWeightQuantProxyHandlerMixin return args by @costigt-dev in #870
- Fix - Speech to text: Create an empty json file by @costigt-dev in #871
- Feat (scaling/standalone): flag to retrieve full state dict by @Giuseppe5 in #874
- Notebooks: makes notebooks deterministic and prints output of asserts by @fabianandresgrob in #847
- Fix (proxy): revert value tracer change by @Giuseppe5 in #888
- Fix (proxy): fix for attributes retrieval by @Giuseppe5 in #880
- Feat (notebook): add example for dynamic quantization to ONNX export by @fabianandresgrob in #877
- Fix (gpxq): handling empty tensors with GPxQ and adding unit tests by @i-colbert in #892
- Fix (ptq): expose uint_sym_act flag and fix issue with minifloat sign by @fabianandresgrob in #898
- Feat (minifloat): add support for user specified minifloat format by @fabianandresgrob in #821
- Feat: Add QuantConv3d and QuantConv3dTranspose by @costigt-dev in #805
- Add tutorial examples of per-channel quantization by @OscarSavolainenDR in #867
- Fix (tests): revert pytest pin by @Giuseppe5 in #903
- Remove: Remove original_cat workaround by @costigt-dev in #902
- Infra: Update issue template by @nickfraser in #893
- Pull Request Template by @capnramses in #885
- Fix (core): add return in state_dict by @Giuseppe5 in #910
- Fix (quant_tensor): fix typing and remove unused checks by @Giuseppe5 in #913
- Fix (nn): removed unused caching in adaptive avgpool2d by @Giuseppe5 in #911
- Fix (quant_tensor): remove unused checks by @Giuseppe5 in #918
- Setup: pin ONNX to 1.15 due to ORT incompatibility by @Giuseppe5 in #924
- Feat (examples): add support for Stable Diffusion XL by @Giuseppe5 in #909
- Assert all ptq-common bit widths are positive integers by @OscarSavolainenDR in #931
- Enhance: Quant Tensor Test by @costigt-dev in #894
- Fix (examples/stable_diffusion): README formatting and clarification by @Giuseppe5 in #932
- Fix (examples/ptq): fix for bitwidth check by @Giuseppe5 in #934
- Feat: functionalize QuantTensor by @Giuseppe5 in #878
- Feat (minifloat): cleanup minifloat impl by @Giuseppe5 in #922
- Fix tests in dev by @Giuseppe5 in #939
- Feat (proxy): scale computation delegated to bias proxy by @Giuseppe5 in #938
- Fix (gpxq): adding input quant to process input by @i-colbert in #943
- Fix (quant): propagate device and dtype in subinjector by @Giuseppe5 in #942
- Fix (gpxq): correct variable name by @Giuseppe5 in #944
- Fix (quant_tensor): fix AvgPool functional implementation by @Giuseppe5 in #945
- Feat (quant_tensor): support for dim() and ndim by @Giuseppe5 in #947
- Fix (graph/standardize): correct check for Mean to AvgPool by @Giuseppe5 in #948
- Feat (graph/standardize): default keepdim value by @Giuseppe5 in #950
- Fix bullet formatting in getting started guide by @timkpaine in #952
- Fix (quant/float): correct scaling_impl and float_scaling_impl by @Giuseppe5 in #953
- Fix/remove-numel - Remove numel is zero check from context manager exit method by @costigt-dev in #920
- Feat (examples/ptq): support for dynamic act quant by @Giuseppe5 in #935
- Feat (quant_tensor): support for FloatQuantTensor by @Giuseppe5 in #919
- Fix (examples/llm): Add all rewriters to the list by @nickfraser in #956
- Fix (core/quant/float): use eps to avoid log(0) by @Giuseppe5 in #957
- Fix (test/actions): Excluded
torch==1.9.1
,platform=macos-latest
tests by @nickfraser in #960 - Adding FP8 weight export by @costigt-dev in #907
- Fix (llm): fix device issue for eval when not using default device by @fabianandresgrob in #949
- Fix (GPFQ): using random projection for speed up/less memory usage by @fabianandresgrob in #964
- Fix (calibrate/minifloat): fix for act calibration by @Giuseppe5 in #966
- Fix (quant/float): restore fix for log(0) by @Giuseppe5 in #968
- Setup: pin numpy version by @Giuseppe5 in #974
- Feat (minifloat): support for FNUZ variants by @Giuseppe5 in #973
- Fix (core/float): add default for float_scaling_impl by @Giuseppe5 in #972
- Feat (graph/equalize): upcast during equalization computation by @Giuseppe5 in #970
- Generative improv by @Giuseppe5 in #965
- Fix (requirements/setuptools): Set maximum requirement for
setuptools
by @nickfraser in #963 - Fix: Typo fix on SDXL command line args by @nickfraser in #976
- Fix (graph/bias_correction): Fix when layer parameters are offloaded to
accelerate
by @nickfraser in #962 - Fix (ptq/bias_correction): remove unnecessary forward pass by @Giuseppe5 in #980
- Fix (export/qonnx): Fixed symbolic kwargs order. by @nickfraser in #988
- Various SDXL quantization fixes by @nickfraser in #977
- Fix (brevitas_examples/sdxl): Various fixes by @Giuseppe5 in #991
- Feat (proxy/parameter_quant): cache quant weights by @Giuseppe5 in #990
- Docs: Added 0.10.3 release note to README. by @nickfraser in #993
- Added some preliminary unit tests to the CNNs 'quantize_model' by @OscarSavolainenDR in #927
- Feat (tests): extended minifloat unit tests by @alexredd99 in #979
- Fix (proxy/runtime_quant): correct handling of mixed type quantization by @Giuseppe5 in #985
- docs (readme): Fixed GH actions badges by @nickfraser in #996
- Feat: Update LLM entry-point by @nickfraser in #987
- Feat: Support for Groupwise (MX) quantization by @Giuseppe5 in #971
- Feat(graph): better exclusion mechanism by @Giuseppe5 in #1003
- Fix (mx): input view during quantization by @Giuseppe5 in #1005
- Feat (mx): adding padding and transposed support by @Giuseppe5 in #1007
- fix (nn/avg_pool): Fix for trunc quant not being applied by @nickfraser in #1014
- Fix (nn/conv): Fixed conversion of convolutions when
padding_mode='same'
by @nickfraser in #1017 - Fix (proxy): clean-up by @Giuseppe5 in #1011
- Feat (mx): automatic group_dim in layerwise quant by @Giuseppe5 in #1012
- fix (nn/conv): Fix regression introduced in #1017 by @nickfraser in #1019
- Feat (mx): gptq compatibility and quant tests by @Giuseppe5 in #1013
- Feat (mx): PTQ MX + Float support by @Giuseppe5 in #1010
- Fix (graph/quant): Bugfix in blacklist matching in
find_module
by @nickfraser in #1021 - Test (graph/quantize) Added extra prefix test to
layerwise_quantize
by @nickfraser in #1022 - Test (example/llm): Refactor and add basic tests for the LLM entry-point by @nickfraser in #1002
- Feat (examples/sdxl): Updates to SDXL entry-point by @nickfraser in #1020
- Feat (gptq): optimizing CPU to GPU memory transfer by @i-colbert in #1009
- notebooks: rerun notebooks. by @nickfraser in #1026
- HQO for scale/zero point by @Giuseppe5 in #937
- Test calibration reference by @Giuseppe5 in #1031
- Fix (sdxl): avoid suppressing checkpoint errors by @Giuseppe5 in #1034
- Setup: update checkout version to v3 by @Giuseppe5 in #1039
- Fix (gpxq): correct index for groupwise GPxQ by @Giuseppe5 in #1040
- Fix (llm): small fixes to LLM by @Giuseppe5 in #1035
- Feat (activation_calibration): speed-up by skipping quantization by @Giuseppe5 in #1029
- Fix (proxy): fix for float quant properties is_ocp and is_fnuz by @alexredd99 in #1028
- Decoupled PerChannel/PerTensor quantization by @Giuseppe5 in #1025
- Fix (examples/llm): Fix infinite loop in LLM entrypoint with WikiText2 by @pablomlago in #1044
- Fix po2 for float quant by @Giuseppe5 in #1033
- Feat (gpfq): adding memory-efficient formulation by @i-colbert in #1043
- Fix weights mse by @Giuseppe5 in #1047
New Contributors
- @OscarSavolainenDR made their first contribution in #861
- @costigt-dev made their first contribution in #868
- @timkpaine made their first contribution in #952
- @alexredd99 made their first contribution in #979
- @pablomlago made their first contribution in #1044
Full Changelog: v0.10.2...v0.11.0