Edge & Fog Computing · Study deck

Model Optimization for Edge AI

Picture a camera near a factory belt that must spot a cracked part before it moves on.

Edge Eddie is your guide for this deck.

optimization
Edge Eddie, the module guide, in a scene from this chapter.
iotclass.org

After studying this chapter

Learning objectives

You will be able to:

  • Explain: If the representative data misses night images, vibration spikes, clipped audio windows, or rare sensor poses, the activation range can be too narrow and real inputs may saturate after conversion.
  • Explain: Storing the same weights as INT8 uses about 2.4 MB, saving 7.2 MB before activation memory, tensor arenas, preprocessing buffers, or update staging are counted.
  • Explain: The mathematical gist: For weights in [−0.50,0.50][-0.50,0.50][−0.50,0.50], symmetric int8 uses s=0.50/127=0.0039370s=0.50/127=0.0039370s=0.50/127=0.0039370 and σq=s/12=0.0011365\sigma_q=s/\sqrt{12}=0.0011365σq​=s/12​=0.0011365.
  • Explain post-training quantization, quantization-aware training, pruning, distillation, and operator fusion.
iotclass.org

Major section

In 60 Seconds

The original program works on a large office computer.

  • The camera has less memory, less power, and only a moment to decide.
  • The engineer should first save a trusted set of test images and record the large program's results.
  • Include poor light, dust, motion, and parts near the decision limit.

Key terms

If the new version
If the new version is unsure, it may stop the belt, request a closer look, or mark the part for manual review.
Edge AI model optimization
Edge AI model optimization is a measured conversion process, not a promise that every model becomes tiny and fast.
iotclass.org

Major section

In 60 Seconds (continued)

A faster result is not a win if it hides more dangerous defects or cannot be supported after an update.

  • A single average can hide that the new version misses the rare crack the system was built to find.
  • Another engineer should be able to repeat the release decision without a private notebook.
  • If the new version is unsure, it may stop the belt, request a closer look, or mark the part for manual review.
iotclass.org

Major section

In 60 Seconds (continued)

If the error, heat, or response time crosses that line, return to the known version and keep the failed record.

  • This simple test does not explain every way a learned model can be reduced.
  • Under the Hood explains number size, removed connections, teaching from a larger model, and hardware-specific changes.
  • Edge AI model optimization is a measured conversion process, not a promise that every model becomes tiny and fast.
iotclass.org

Major section

Phoebe's Field Notes: Why "0.236 to 60" Is Not a Rounding Accident · Start Simple

The mathematical gist: For weights in [−0.50,0.50][-0.50,0.50][−0.50,0.50], symmetric int8 uses s=0.50/127=0.0039370s=0.50/127=0.0039370s=0.50/127=0.0039370 and σq=s/12=0.0011365\sigma_q=s/\sqrt{12}=0.0011365σq​=s/12​=0.0011365.

  • That one pass does not replace calibration and task-level validation.

Let Edge Eddie name the scale, rounding move, reconstruction, and error boundary.

iotclass.org

Major section

Start With the Deployment Budget · Optimization Techniques

Accuracy gate: task metric, validation split, edge cases, confidence threshold, false accept cost, and false reject cost.

  • Runtime verifies operators and delegates before the final Gate says measure, then accept or reject.
  • This order protects task quality while locating the specific device limit each optimization is meant to relieve.
  • Removes individual weights.
A model optimization decision map that starts with deployment budget, then checks model fit, accuracy tolerance, runtime support, and target validation
A model optimization decision map that starts with deployment budget, then checks model fit, accuracy tolerance, runtime support, and target validation
iotclass.org

Major section

Quantization · Small Quantization Example

where x is the floating value, s is the scale, and z is the zero point.

  • Those depend on activations, operators, runtime kernels, delegates, memory layout, and calibration data.
  • Weights are quantized, while some activations may remain floating point.
  • Weights and activations use integer representations.
  • Weights use half precision.

Why it matters

This single weight changed only slightly, but the whole model still needs validation because many small changes can alter borderline decisions.

iotclass.org

Major section

Pruning · Pruning Trap

Pruning removes model capacity that appears unnecessary for the task.

  • Removes channels, filters, layers, attention heads, or blocks.
  • The result can be exported as a smaller dense model, which is usually easier for MCUs and edge runtimes.
  • Sets individual weights to zero.
  • Baseline: Train or select the model that already meets task quality.

Why it matters

If the runtime stores zeros in a dense tensor or lacks sparse kernels, pruning may reduce neither latency nor memory.

iotclass.org

Major section

Knowledge Distillation · Distillation Is Not Weight Copying

Knowledge distillation uses a larger teacher model to train a smaller student model.

  • Eddie records the original board result before swapping one model part.
  • The candidate then goes through the target converter and runs beside the real sensor and power meter.
  • The lights keep quality, speed, memory, and energy as separate checks.

Key terms

Distillation
Distillation is useful only when the smaller student survives that same measured route.
Optimization validation route from baseline through candidate, target conversion, hardware measurements, and a release gate for accuracy, latency, memory, operators, power, and monitoring.
Optimization validation route from baseline through candidate, target conversion, hardware measurements, and a release gate for accuracy, latency, memory, operators, power, and monitoring.
iotclass.org

Major section

Compiler and Runtime Optimization · Unsupported Operator Trap

Freezes training-only behavior, removes unused nodes, locks input shapes when possible, and checks that preprocessing matches training.

  • Optimization is not complete when the model converter produces a file.
  • Edge runtimes can change performance more than the model transformation itself.
  • Unsupported operators may fall back to CPU and dominate latency.
iotclass.org

Major section

Deep Dive: Proving the Optimization Trade

Structured pruning needs the same proof.

  • An optimization is a trade only after the numbers close.
  • A model with 2.4 million FP32 weights uses about 9.6 MB for weights alone.
  • If a 24 million-MAC inference removes 25 percent of channels, the dense work target falls to about 18 million MACs.

Key terms

If the range
If the range is too wide, the step size becomes coarse and borderline classes can drift.

Why it matters

The saving matters only if the exported model and runtime execute the smaller dense graph; unstructured zeros stored in dense tensors do not automatically reduce latency.

iotclass.org

Major section

Deep Dive: Proving the Optimization Trade (continued)

Storing the same weights as INT8 uses about 2.4 MB, saving 7.2 MB before activation memory, tensor arenas, preprocessing buffers, or update staging are counted.

  • At 5 inferences per second, that is 90 million MACs/s instead of 120 million MACs/s.
  • If the representative data misses night images, vibration spikes, clipped audio windows, or rare sensor poses, the activation range can be too narrow and real inputs may saturate after conversion.
  • If the range is too wide, the step size becomes coarse and borderline classes can drift.
iotclass.org

Major section

Common Pitfalls · Reference Path

Calibration that misses real sensor conditions can produce a model that works in a notebook and fails in the field.

  • Unstructured zeros do not help if the exported artifact and runtime still execute dense kernels.
  • Weights may fit in flash while intermediate activations exceed RAM.
  • Image resize, feature extraction, filtering, or normalization may dominate the latency path.
iotclass.org

Deck summary

Key takeaways

The original program works on a large office computer.

  • A faster result is not a win if it hides more dangerous defects or cannot be supported after an update.
  • If the error, heat, or response time crosses that line, return to the known version and keep the failed record.
  • The mathematical gist: For weights in [−0.50,0.50][-0.50,0.50][−0.50,0.50], symmetric int8 uses s=0.50/127=0.0039370s=0.50/127=0.0039370s=0.50/127=0.0039370 and σq=s/12=0.0011365\sigma_q=s/\sqrt{12}=0.0011365σq​=s/12​=0.0011365.
  • Accuracy gate: task metric, validation split, edge cases, confidence threshold, false accept cost, and false reject cost.
iotclass.org

Retrieval practice

Recall check 1 of 5

Edge Eddie says: answer from memory, then check your reasoning.

Q1A team wants to quantize a model before defining the target memory, latency, operator, and accuracy budgets. What is the main risk?

AQuantization will always make the model too accurate, so validation can wait until after release
BThey may optimize a model that still fails the real deployment constraints or uses unsupported runtime operators
CThe model will no longer need calibration or validation data once the converted file is smaller
DThe device will automatically switch to cloud inference whenever an optimized edge model misses its budget
Show answer

Answer: B Optimization has to be judged against deployment constraints, not against a generic file-size reduction.

iotclass.org

Retrieval practice

Recall check 2 of 5

Edge Eddie says: answer from memory, then check your reasoning.

Q2A model fits memory but the sensor transfer dominates response time. Which optimization should be tried first?

ARemove validation to shorten the test cycle.
BPrune model weights because smaller weights also reduce transfer costs proportionally.
CProfile and reduce the sensor transfer path.
DShrink the model file solely because it is easy.
Show answer

Answer: C The measured bottleneck is outside model computation.

iotclass.org

Retrieval practice

Recall check 3 of 5

Edge Eddie says: answer from memory, then check your reasoning.

Q3How should the smaller model be assessed after the threshold errors increase?

AImprove calibration data, check the real validation set, and consider quantization-aware training if the behavior still fails
BAccept the smaller model because its reduced file size preserves threshold decisions.
CDelete the validation set and lower the threshold until the model passes the smallest-file requirement
DPrune to recover lost confidence because reduced weights improve false-accept and false-reject rates.
Show answer

Answer: A The issue is numeric behavior near the decision boundary, so calibration and QAT are appropriate next checks.

iotclass.org

Retrieval practice

Recall check 4 of 5

Edge Eddie says: answer from memory, then check your reasoning.

Q4In post-training INT8 quantization, what does representative calibration data set?

AThe legal license for the model file before it is deployed to the edge
BThe class names added during conversion after training has already finished
CThe number of epochs used to retrain all weights from random values
DThe activation ranges used to choose scale and zero point for the converted model
Show answer

Answer: D Representative calibration data sets activation ranges so scale and zero point can map float values into the int8 range before deployment validation.

Q5What anchors accuracy, latency and model-size comparisons before TinyML optimization?

AOptimization Candidate
BTarget Runtime
CBaseline Model
DDeployment Gate
Show answer

Answer: C It anchors every accuracy, latency, and size comparison.

iotclass.org

Retrieval practice

Recall check 5 of 5

Edge Eddie says: answer from memory, then check your reasoning.

Q6Complete a TensorFlow Lite post-training quantization setup that uses representative data.

Aconverter = tf.lite.TFLiteConverter.from_saved_model('saved_model')
Bconverter = tf.lite.Converter.load('saved_model')
Cconverter = tf.convert_to_tflite('saved_model')
Dconverter = tf.lite.Model('saved_model')
Show answer

Answer: A Post-training integer quantization uses a TFLite converter, default optimization, and representative data so activation ranges are calibrated before conversion.

iotclass.org

Print reference

Answers 1 of 2

Answer key.

  1. B · Optimization has to be judged against deployment constraints, not against a generic file-size reduction.
  2. C · The measured bottleneck is outside model computation.
  3. A · The issue is numeric behavior near the decision boundary, so calibration and QAT are appropriate next checks.
  4. D · Representative calibration data sets activation ranges so scale and zero point can map float values into the int8 range before deployment validation.
  5. C · It anchors every accuracy, latency, and size comparison.
iotclass.org

Print reference

Answers 2 of 2

Answer key.

  1. A · Post-training integer quantization uses a TFLite converter, default optimization, and representative data so activation ranges are calibrated before conversion.
iotclass.org