← Back to the chapter · Next detail →
Technical detail · Hardware Accelerators for Edge AI
Why INT8 neural accelerators can be efficient
13.4.1 Why INT8 NPUs Can Be Efficient
Neural-network inference is dominated by multiply-accumulate (MAC) work: inputs are multiplied by weights and the products are summed. A general-purpose CPU core executes a limited number of MACs per cycle. An NPU or TPU packs a large array of MAC units, often arranged so data streams through many multipliers with high reuse. Moving from 32-bit floating point (FP32) to 8-bit integers (INT8) makes arithmetic units smaller and cheaper, allowing more units on the chip with less energy per unit.
Raw MAC throughput helps only while the array stays fed. Memory bandwidth—moving weights and activations to those units—is a frequent bottleneck. On-chip SRAM, weight reuse, quantized weights and clean operator mapping matter alongside TOPS. An unsupported resize, activation or custom layer can fall back to the CPU, forcing tensors between memory regions and letting CPU work gate the entire path.
13.5 Phoebe’s Field Notes: The Other Quantization – From FP32 to INT8
Over , INT8 has , and an ideal 49.9 dB quantisation ceiling; INT16 reaches 98.1 dB. Independently, a 10 inferences/s cadence can track event changes only up to the 5 Hz Nyquist bound. A denser MAC array fixes neither lost numeric margin nor a slow decision clock.