← Back to the chapter · Previous detail

Technical detail · Model Optimization for Edge AI

Close the storage and compute trade

12.13 Deep Dive: Proving the Optimization Trade

A model with 2.4 million FP32 weights uses about 9.6 MB for weights alone. INT8 storage for the same weights is about 2.4 MB, saving 7.2 MB before activations, tensor arenas, preprocessing buffers, or update staging. With a 3 MB model-flash reservation and at most a 2 percentage point accuracy drop allowed, the INT8 candidate remains provisional until both limits are measured on the converted artifact.

For a 24 million multiply–accumulate (MAC) inference, reducing dense work by 25 percent gives about 18 million MACs. At 5 inferences per second, that is 90 million MACs/s instead of 120 million MACs/s. Removing 25 percent of channels alone does not establish that exact work reduction; verify the exported graph. Unstructured zeros in dense tensors do not automatically save latency.

The proof record differs by technique. For quantization, include the baseline metric, calibration set, converted artifact, operator list, false accepts, false rejects, latency, peak memory, and rollback version. For pruning, identify structured or unstructured removal, changed dense dimensions, sparse kernels, and whether fine-tuning recovered the metric. For distillation, name the teacher version, student architecture, training and validation data, and budget reason for replacing the original. For runtime changes, record delegate coverage, CPU fallbacks, tensor arena size, preprocessing, postprocessing, power, thermal behavior, and update path.

← Back to the chapter · Previous detail