Edge & Fog Computing · Study deck
Model Optimization for Edge AI
Picture a camera near a factory belt that must spot a cracked part before it moves on.
Edge Eddie is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- Explain: If the representative data misses night images, vibration spikes, clipped audio windows, or rare sensor poses, the activation range can be too narrow and real inputs may saturate after conversion.
- Explain: Storing the same weights as INT8 uses about 2.4 MB, saving 7.2 MB before activation memory, tensor arenas, preprocessing buffers, or update staging are counted.
- Explain: The mathematical gist.: For weights in $[-0.50,0.50]$, symmetric int8 uses $s=0.50/127=0.0039370$ and $\sigma_q=s/\sqrt{12}=0.0011365$.
- Explain post-training quantization, quantization-aware training, pruning, distillation, and operator fusion.
Major section
In 60 Seconds
The original program works on a large office computer.
- The camera has less memory, less power, and only a moment to decide.
- The engineer should first save a trusted set of test images and record the large program's results.
- Include poor light, dust, motion, and parts near the decision limit.
Major section
In 60 Seconds (continued)
A faster result is not a win if it hides more dangerous defects or cannot be supported after an update.
- A single average can hide that the new version misses the rare crack the system was built to find.
- Another engineer should be able to repeat the release decision without a private notebook.
- If the new version is unsure, it may stop the belt, request a closer look, or mark the part for manual review.
Major section
In 60 Seconds (continued)
If the error, heat, or response time crosses that line, return to the known version and keep the failed record.
- This simple test does not explain every way a learned model can be reduced.
- Under the Hood explains number size, removed connections, teaching from a larger model, and hardware-specific changes.
- Edge AI model optimization is a measured conversion process, not a promise that every model becomes tiny and fast.
Major section
Phoebe's Field Notes: Why "0.236 to 60" Is Not a Rounding Accident
The mathematical gist.: For weights in $[-0.50,0.50]$, symmetric int8 uses $s=0.50/127=0.0039370$ and $\sigma_q=s/\sqrt{12}=0.0011365$.
- That one pass does not replace calibration and task-level validation.
Major section
Start With the Deployment Budget
Accuracy gate: task metric, validation split, edge cases, confidence threshold, false accept cost, and false reject cost.
- Runtime verifies operators and delegates before the final Gate says measure, then accept or reject.
- This order protects task quality while locating the specific device limit each optimization is meant to relieve.
Major section
Quantization
where x is the floating value, s is the scale, and z is the zero point.
- Those depend on activations, operators, runtime kernels, delegates, memory layout, and calibration data.
- Weights are quantized, while some activations may remain floating point.
- Weights and activations use integer representations.
- Weights use half precision.
Major section
Pruning
Pruning removes model capacity that appears unnecessary for the task.
- Removes channels, filters, layers, attention heads, or blocks.
- The result can be exported as a smaller dense model, which is usually easier for MCUs and edge runtimes.
- Sets individual weights to zero.
- The model may look sparse, but runtime speed improves only if the sparse representation is preserved and supported.
Major section
Knowledge Distillation
Knowledge distillation uses a larger teacher model to train a smaller student model.
- Target conversion fixes the exported artefact, runtime, and delegates before Hardware measure tests the device and sensor path.
- At Meets the gates?, a yes branch can Accept and stage rollout, whereas no returns to Revise data, preprocessing, model, or thresholds.
Major section
Knowledge Distillation (continued)
Distillation is useful only when the smaller student survives that same measured route.
- The teacher can be a larger cloud model, ensemble, or previous production model.
- It provides richer output distributions than a single hard label.
- The student is chosen for the target device class.
- Its architecture should fit the memory, operator, and latency budget before training starts.
Major section
Compiler and Runtime Optimization
Optimization is not complete when the model converter produces a file.
- Edge runtimes can change performance more than the model transformation itself.
- Unsupported operators may fall back to CPU and dominate latency.
- Reuses activation buffers when lifetimes do not overlap.
- Freezes training-only behavior, removes unused nodes, locks input shapes when possible, and checks that preprocessing matches training.
Major section
Deep Dive: Proving the Optimization Trade
Structured pruning needs the same proof.
- An optimization is a trade only after the numbers close.
- A model with 2.4 million FP32 weights uses about 9.6 MB for weights alone.
- If a 24 million-MAC inference removes 25 percent of channels, the dense work target falls to about 18 million MACs.
Major section
Deep Dive: Proving the Optimization Trade (continued)
Storing the same weights as INT8 uses about 2.4 MB, saving 7.2 MB before activation memory, tensor arenas, preprocessing buffers, or update staging are counted.
- At 5 inferences per second, that is 90 million MACs/s instead of 120 million MACs/s.
- If the representative data misses night images, vibration spikes, clipped audio windows, or rare sensor poses, the activation range can be too narrow and real inputs may saturate after conversion.
- If the range is too wide, the step size becomes coarse and borderline classes can drift.
Major section
Common Pitfalls
Calibration that misses real sensor conditions can produce a model that works in a notebook and fails in the field.
- Unstructured zeros do not help if the exported artifact and runtime still execute dense kernels.
- Weights may fit in flash while intermediate activations exceed RAM.
- Image resize, feature extraction, filtering, or normalization may dominate the latency path.
Deck summary
Key takeaways
The original program works on a large office computer.
- A faster result is not a win if it hides more dangerous defects or cannot be supported after an update.
- If the error, heat, or response time crosses that line, return to the known version and keep the failed record.
- The mathematical gist.: For weights in $[-0.50,0.50]$, symmetric int8 uses $s=0.50/127=0.0039370$ and $\sigma_q=s/\sqrt{12}=0.0011365$.
- Accuracy gate: task metric, validation split, edge cases, confidence threshold, false accept cost, and false reject cost.
Retrieval practice
Recall check 1 of 4
Edge Eddie says: answer from memory, then check your reasoning.
Q1A team wants to quantize a model before defining the target memory, latency, operator, and accuracy budgets. What is the main risk?
Show answer
Answer: C Optimization has to be judged against deployment constraints, not against a generic file-size reduction.
Retrieval practice
Recall check 2 of 4
Edge Eddie says: answer from memory, then check your reasoning.
Q2A post-training int8 model is smaller, but its false rejects increase near the confidence threshold. What is the best next step?
Show answer
Answer: B The issue is numeric behavior near the decision boundary, so calibration and QAT are appropriate next checks.
Retrieval practice
Recall check 3 of 4
Edge Eddie says: answer from memory, then check your reasoning.
Q3In post-training INT8 quantization, what does representative calibration data set?
Show answer
Answer: B Representative calibration data sets activation ranges so scale and zero point can map float values into the int8 range before deployment validation.
Retrieval practice
Recall check 4 of 4
Edge Eddie says: answer from memory, then check your reasoning.
Q4Place each edge ai optimization concept where it lives so you can compare compression gains with device limits before approving a release.
Show answer
Answer: A These boundaries keep measure baseline, fit the device, gate deployment distinct so you can compare compression gains with device limits before approving a release.
Q5Complete a TensorFlow Lite post-training quantization setup that uses representative data.
Show answer
Answer: A Post-training integer quantization uses a TFLite converter, default optimization, and representative data so activation ranges are calibrated before conversion.
Print reference
Answers 1 of 2
Answer key.
- C · Optimization has to be judged against deployment constraints, not against a generic file-size reduction.
- B · The issue is numeric behavior near the decision boundary, so calibration and QAT are appropriate next checks.
- B · Representative calibration data sets activation ranges so scale and zero point can map float values into the int8 range before deployment validation.
- A · These boundaries keep measure baseline, fit the device, gate deployment distinct so you can compare compression gains with device limits before approving a release.
Print reference
Answers 2 of 2
Answer key.
- A · Post-training integer quantization uses a TFLite converter, default optimization, and representative data so activation ranges are calibrated before conversion.