← Back to the chapter · Next detail →

Technical detail · Model Optimization for Edge AI

Why 0.236 becomes int8 code 60

12.7 Small Quantization Example

For weights in [-0.50, 0.50], symmetric signed int8 can use s=0.50/127=0.0039370s=0.50/127=0.0039370. A weight of 0.236 becomes code 60 and reconstructs as approximately 0.236220, an error of 0.000220. The half-step error bound is approximately ±0.001969\pm0.001969, and the quantization-noise estimate is σq=s/12=0.0011365\sigma_q=s/\sqrt{12}=0.0011365.

Using the coarsely rounded scale 0.00394 instead gives round(0.236 / 0.00394) = 60 and 60 x 0.00394 = 0.2364. The different reconstructed value comes from rounding the scale. Preserve sufficient precision when comparing results.

The affine rule also explains a simple exact example: with scale 0.01 and zero point 0, weight 0.53 maps to code 53 and reconstructs as 0.53. A small change in one weight never replaces calibration or task-level validation: many small changes can alter a borderline decision.

← Back to the chapter · Next detail →