Skip to content
/ DSQ Public

pytorch implementation of "Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks"

Notifications You must be signed in to change notification settings

ricky40403/DSQ

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

24 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

DSQ

pytorch unofficial implementation of "Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks"


The Origin Paper : https://arxiv.org/abs/1908.05033


This repository follow the Algorithm 1 in the paper.

This repository uses the max value of int32 as the initial value.
It should not affect the value range (because the parameter of the deep model should not too large), and most of the edge device range is up to int32.


Training

Training with quantization. Scrip modified from https://github.com/pytorch/examples/tree/master/imagenet

Now support uniform/DSQ quantization
adding argments
-q : quantization type, default is None
--quantize_input : quantize input or not
--quan_bit : quantization bit num
--log_path : tensorboard log path to write, default folder is ./log.

Examples
Training DSQ with 8 bit (no quantiza input)

python train.py -a resnet18  -q DSQ --quan_bit 8 {Path to data}

Training DSQ with 8 bit ( quantiza input)

python train.py -a resnet18  -q DSQ --quan_bit 8 --quantize_input {Path to data}

Evaluating (directly use evaluation and resume from model_best.pth.tar)

python train.py -a resnet18 -q DSQ --quan_bit 8 --quantize_input --resume {path to model_best.pth.tar} -- evaluate {Path to data}

Experiments

The results is base on fake-quantization.(only quantized convolution). As the mentioned in the paper, not to quantize the final Linear Layer.

model QuanType W/A bit top1 top5
resnet18 UniformQuan 4/32 69.486 89.004
DSQ 4/32 69.328 88.872
UniformQuan 4/4 69.306 88.780
DSQ 4/4 69.542 88.884

learned alpha for 4 bit DSQ (quantize weight and input)

layer weight activation
layer1.0.conv1 0.4832 0.5661
layer1.0.conv2 0.3730 0.2953
layer1.1.conv1 0.4405 0.2975
layer1.1.conv2 0.3427 0.1959
layer2.0.conv1 0.3966 0.1653
layer2.0.conv2 0.4140 0.2014
layer2.downsample 0.3275 0.1779
layer2.1.conv1 0.4303 0.1675
layer2.1.conv2 0.4207 0.1570
layer3.0.conv1 0.4590 0.2774
layer3.0.conv2 0.4838 0.2569
layer3.downsample 0.2305 0.1073
layer3.1.conv1 0.4523 0.1775
layer3.1.conv2 0.4382 0.1792

Resutls:

As the table2 in the paper, it indeed show that

Second, different layers show different sensitivity to the quantization.  
For example, the downsampling convolution layers can be quantized much (a small α),
while some layers such as layer3.0.conv2 are not suitable for  quantization (a large α).  

Issue:

It seems that α of weights is bigger than that of activations.
Maybe the un-quantize batchnorm restricts the activation and cause the difference to the paper. (or someone can tell why)

Update Note

20191218: Update uniform quantization results. It seems that the sgn function still need STE backward or the loss will becomes Nan.
20191231: Update Experiments.

About

pytorch implementation of "Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks"

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages