Quantization

the idea is to compare the difference between two probability distribution when scaling, for example from int16 to int8

Does my operation support fp16?

• CPU does support saving fp16 weights, but computations are done in fp32

Does my operation sensitive to fp16?

For example epsilon in LayerNormalization usually is very small $1e^{-12}$, but smallest value in fp16 is $\approx 6e^{-5}$, which cause NaN issues.

Consider a float x in [a, b], such that affine quantization scheme:

where:

• $x_q$ is the quantized int8 associated with x
• $S$ and $Z$ are scaling and zero-point parameters
  • $S$ is the scale, positive float32
  • $Z$ is the zero-point, or the int8 value corresponding to value 0 in fp32

Thus quantized value $x_q$ is: $x_q = \text{round}(x / S + Z)$

And fp32 value outside of [a, b] is clipped to closest representable value.

See also: paper

• Post-training dynamic quantization: range of each activation is computed on the fly at runtime
• Post-training static quantization: range of each activation is computed offline before runtime
  • Observers are put on activations to collect their value
  • certain number of forward passes on calibration datasets
  • range of each computation are computed according to some calibration technique
• Quantization aware training: range of each activation is computed during training
  • fake_quantize operations are inserted in the computation graph
  • fake_quantize is a no-op during inference, but during training, it simulates the effect of quantization

bitsandbytes and GPTQ