stateDiagram-v2 [*] --> Capture Capture --> Preprocess Preprocess --> Infer Infer --> LowConfidence: confidence below policy Infer --> LocalAction: approved local label Infer --> ReportOnly: observation label LowConfidence --> StoreEvidence StoreEvidence --> SafeFallback LocalAction --> ReportEvent ReportOnly --> ReportEvent SafeFallback --> Monitor ReportEvent --> Monitor Monitor --> [*]
16 Edge AI Fit and Deployment
edge-fog
ai
ml
Phoebe’s Field Notes: Two Different Things Called “Quantization”
Phoebe’s Why
This chapter’s “quantize” means shrinking a trained model’s weights from 32-bit floats to 8-bit integers so it fits and runs faster on the target device – that is a software compression choice, made after training. There is an older, physical “quantization” underneath it: every sensor reading that ever reached that model first passed through an analog-to-digital converter, which rounds a continuous voltage to the nearest of \(2^N\) discrete steps. That rounding is not a design choice; it is a noise floor baked into the signal before a single frame reaches the model. And before quantization noise even applies, the camera’s frame rate already decided which frequencies in the physical scene it was allowed to see at all.
The Derivation
Nyquist sampling criterion for a camera or sensor running at frame rate \(f_s\):
\[f_s \ge 2 f_{max}\]
Temporal aliasing folds an under-sampled frequency \(f\) to an apparent frequency:
\[f_{alias} = \left| n f_s - f \right|, \quad n = \mathrm{round}(f/f_s)\]
ADC quantization step over \(N\) bits, full-scale range normalized to 1:
\[q = \frac{1}{2^N} \qquad e_{RMS} = \frac{q}{\sqrt{12}}\]
Signal-to-noise ratio for a full-scale signal:
\[\mathrm{SNR} = 6.02N + 1.76\ \text{dB}\]
Worked Numbers: This Chapter’s Own 30 fps Line
- This chapter’s own numbers: the packing-line camera runs at \(f_s=30\) fps, so the Nyquist limit is \(f_{max}=15.0\) Hz – any real motion in the scene faster than 15 Hz cannot be captured correctly at this frame rate.
- Aliasing example: a machine vibration at \(f=22.0\) Hz (catalog-typical bearing-fault frequency range) sampled at 30 fps aliases to \(|1\times30-22|=8.00\) Hz – the video would show a slow 8 Hz wobble that is not the real 22 Hz vibration, a classic trap for vision-based condition monitoring.
- Quantization, catalog-typical 10-bit pixel ADC (this chapter does not fix a sensor bit depth, so this is stated as typical for industrial machine-vision parts): \(q=1/1024=0.0977\%\) of full scale, RMS noise \(e_{RMS}=0.0977\%/\sqrt{12}=0.0282\%\) FS, and \(\mathrm{SNR}=6.02(10)+1.76=62.0\) dB.
- Keep the two “quantize” words apart in the record: the 62.0 dB ADC ceiling is fixed by the sensor before any frame is captured; the model’s 8-bit weight quantization is a separate, later choice that trades accuracy for footprint and never appears in the same units.
16.1 Start Simple
Choose one operational decision where a nearby prediction changes what the system can do. The core idea is that edge AI is valuable only when local action, privacy, bandwidth, offline continuity, or site autonomy improves the workflow. Everyday IoT application design starts with the failure cost and the human review path, then adds data, model, conversion, deployment, rollback, and drift monitoring. Build one application story before claiming broad value.
16.2 Learning Objectives
By the end of this chapter, you will be able to:
- Select edge AI application candidates from latency, privacy, bandwidth, autonomy, and operational-value evidence.
- Compare visual inspection, predictive maintenance, audio/event detection, and privacy-preserving sensing patterns.
- Design an edge AI deployment loop from data collection through monitoring and retraining.
- Identify target-device validation requirements for model size, memory, latency, power, and fallback behavior.
- Explain how model updates, rollback, drift monitoring, and human review fit into a production edge AI system.
- Build a short edge AI application decision record that avoids generic financial claims.
Chapter Roadmap
Read this chapter as an application review in four passes:
- First decide whether edge inference is justified by local latency, offline continuity, privacy, bandwidth, human attention, or guarded local control.
- Then compare the four application families: visual inspection, predictive maintenance, audio and event detection, and privacy-preserving sensing.
- Next walk the deployment loop from decision definition through dataset, baseline, compression, target validation, rollout, and monitoring.
- Finally turn the runtime evidence into an application record that names fallback behavior, review signals, rollback, and retraining triggers.
The checkpoints below mark each pass so the final record stays tied to measured application evidence rather than a generic “AI at the edge” claim.
Most Valuable Understanding
Edge AI should not be justified by “AI on device” as a novelty. It is justified when a local decision changes the outcome: a defect is rejected before it ships, a motor is inspected before it fails, private media stays local, a wake event is filtered before the main processor wakes, or a site keeps operating during poor connectivity.
16.3 Prerequisites
- Edge AI Fundamentals: Review why inference sometimes belongs near the sensor.
- Edge AI Hardware: Understand target classes from microcontrollers to accelerator-equipped gateways.
- Edge AI Optimization: Review quantization, pruning, and conversion constraints.
- TinyML on Microcontrollers: Understand small-model deployment on constrained devices.
- SOA Resilience Patterns: Connect local inference with fallback, monitoring, and safe degradation.
16.4 Application Map
Edge AI applications are easier to reason about when you separate the sensing path, local inference path, action path, and cloud evidence path.
16.4.1 Local Decision
The device or gateway classifies, detects, scores, or filters data before sending anything upstream.
16.4.2 Local Action
The edge system raises an alert, rejects an item, wakes a larger processor, changes a control mode, or stores evidence for human review.
16.4.3 Cloud Evidence
The cloud receives compact results, examples for review, model health, fleet metadata, and enough evidence to improve the model.
16.4.4 Update Loop
Models, thresholds, labels, and fallback policy are updated through a controlled deployment process, not copied manually to devices.
16.5 When Edge AI Is a Good Fit
Use edge AI when at least one of these conditions is true and measurable.
16.5.1 Latency Is Operational
The decision must happen before a conveyor, vehicle, operator, process, or device state changes.
16.5.2 Connectivity Is Not Guaranteed
The site must keep filtering, detecting, or protecting equipment when cloud connectivity is delayed or unavailable.
16.5.3 Raw Data Is Sensitive
Images, audio, health signals, location, or occupancy data should be minimized before it leaves the device or site.
16.5.4 Bandwidth Is a Constraint
The raw stream is large, frequent, or expensive to transmit, but the useful result is small.
16.5.5 Human Attention Is Scarce
The model can triage events, rank evidence, or reduce the volume of manual review without hiding uncertainty.
16.5.6 Local Control Needs Guardrails
The edge device can apply a low-risk decision locally and escalate high-risk cases to cloud or human review.
Avoid Generic Value Claims
Do not copy payback periods, labor-replacement numbers, or savings estimates from another setting. Edge AI value depends on the specific process, baseline cost, defect cost, false positive cost, missed detection cost, installation work, retraining cost, and review workflow.
Checkpoint: Fit Evidence
You now know:
- Edge AI needs at least one measurable fit condition: latency, connectivity, raw-data sensitivity, bandwidth, human attention, or guarded local control.
- A good candidate turns nearby sensor data into a local decision, local action, compact cloud evidence, and a controlled update loop.
- Generic value claims are not evidence; the application record must name the local action, false-positive cost, missed-detection cost, and review path.
With fit evidence bounded, the next question is which application family carries the strongest operational case.
16.6 Application Family 1: Visual Inspection
Visual inspection is a strong edge AI candidate when image volume is high and the action needs to happen near the line, station, vehicle, or gate.
16.6.1 Useful Decisions
Presence or absence, product orientation, surface defect, packaging mismatch, count mismatch, object class, safety-zone occupancy, and visible quality drift.
16.6.2 Target Evidence
Representative lighting, camera placement, lens choice, motion blur, acceptable defect definitions, rejection workflow, and examples of near-miss cases.
16.6.3 Failure Cost
False rejects can waste product or slow production. Missed defects can reach customers or downstream processes. Both costs must be included.
16.6.4 Deployment Pattern
Run inference near the camera, trigger a local indicator or reject action, and upload sampled evidence for review and retraining.
For visual inspection, benchmark the full pipeline, not only model inference. Camera exposure, preprocessing, data transfer, postprocessing, actuator timing, and human review can dominate the real decision time.
16.7 Application Family 2: Predictive Maintenance
Predictive maintenance works best when the edge model catches a local pattern that is costly to stream continuously and meaningful to maintenance teams.
16.7.1 Useful Signals
Vibration, acoustic emission, current draw, temperature, pressure, flow, torque, duty cycle, and event counters.
16.7.2 Useful Decisions
Normal, watch, inspect soon, stop safely, schedule service, collect high-resolution evidence, or escalate to expert review.
16.7.3 Data Risk
Failure examples are often rare. A model trained only on normal operation may need anomaly detection, expert labels, and careful alert thresholds.
16.7.4 Deployment Pattern
Extract features locally, run a compact model, store short evidence windows around alerts, and send trend summaries to the cloud.
16.8 Application Family 3: Audio and Event Detection
Audio edge AI often uses a staged pipeline: a small always-on detector decides whether to wake a larger model, processor, radio, or application.
16.8.1 Useful Decisions
Wake word, alarm sound, machine sound change, glass break, occupancy cue, animal call, safety event, or abnormal operating noise.
16.8.2 Privacy Benefit
The device can avoid streaming raw audio by sending event metadata, confidence, short redacted snippets, or no audio at all.
16.8.3 Review Trap
Wake-word examples, accents, background noise, room acoustics, and microphone placement can change performance after deployment.
16.8.4 Deployment Pattern
Use a low-power first stage, a stronger verification stage when needed, and a cloud loop for false-positive and missed-event review.
16.9 Application Family 4: Privacy-Preserving Sensing
Some applications are primarily about reducing raw data exposure rather than controlling a machine.
16.9.1 Examples
Occupancy without identity, activity recognition without raw video upload, local face redaction, local license-plate redaction, and health signal summarization.
16.9.2 Design Question
What is the minimum information the cloud needs to deliver the service, diagnose failures, and improve the model?
16.9.3 Governance Risk
Local inference does not automatically make a system privacy-preserving. Logs, samples, labels, and update data can still leak sensitive information.
16.9.4 Deployment Pattern
Keep raw data local by default, send compact results, and document what evidence may be uploaded for review.
Checkpoint: Application Family
You now know:
- The chapter compares four families: visual inspection, predictive maintenance, audio and event detection, and privacy-preserving sensing.
- Each family needs its own evidence: lighting and rejection workflow, rare failure labels, wake-stage behavior, or upload and retention policy.
- Local inference reduces raw streaming, but it does not remove the need for sampled evidence, operator review, governance, and retraining data.
Once the family is clear, the work shifts from use-case selection to the release loop that keeps the model controlled after export.
16.10 Deployment Pipeline
Edge AI deployment is a loop, not a one-time model export.
1. Define the decision State the local action, user impact, false-positive cost, missed-detection cost, fallback behavior, and human review path.
2. Build the dataset Collect data from the real sensor, placement, firmware, environment, and operating modes. Track labels and uncertain examples.
3. Train a baseline Create a model that proves the signal is learnable before optimizing for the target.
4. Compress and convert Quantize, prune, distill, or choose a smaller architecture only after the baseline is meaningful.
5. Validate on target Measure accuracy, latency, memory, power, thermal behavior, startup time, and failure handling on representative hardware.
6. Roll out and monitor Use staged deployment, versioning, rollback, drift monitoring, alert review, and retraining triggers.
16.11 Interactive: Trace the Edge AI Pipeline
Use the animation to walk a sensor sample through buffering, preprocessing, inference, thresholding, local action, and cloud evidence before treating the deployment loop as release-ready.
16.12 Target-Device Validation
An edge AI project is not production-ready until it passes target-device tests.
Model footprint: Flash, RAM, tensor arena, activation buffers, runtime libraries, and model metadata fit with headroom.
Latency path: Sensor acquisition, preprocessing, inference, postprocessing, action, and logging meet the workflow deadline.
Power and thermal behavior: The inference schedule is compatible with battery, heat, enclosure, duty cycle, and site conditions.
Input robustness: Lighting, vibration, noise, temperature, placement, calibration, and sensor aging are represented in validation.
Fallback behavior: The device has an explicit behavior when confidence is low, the model is missing, memory is exhausted, or the runtime fails.
Update safety: Model versions are signed or controlled, staged, observable, and rollback-capable.
Notebook Accuracy Is Not Enough
A model can look good in a notebook and fail on the device because preprocessing differs, quantization changes scores, memory pressure changes timing, or live sensor data differs from training data.
Checkpoint: Deployment Loop
You now know:
- The deployment loop has six stages: define the decision, build the dataset, train a baseline, compress and convert, validate on target, and roll out and monitor.
- Target-device validation must measure model footprint, latency path, power and thermal behavior, input robustness, fallback behavior, and update safety.
- Optimization is a release step, not a substitute for representative data, baseline validation, and target-device measurements.
After the release loop is explicit, define the runtime boundary so the local device, cloud control plane, and review workflow each own the right evidence.
16.13 Runtime Architecture
Edge AI systems usually combine local inference with cloud operations.
16.13.1 Device or Gateway Runtime
Owns sensor acquisition, preprocessing, model execution, postprocessing, local action, local buffering, and watchdog behavior.
16.13.2 Fleet Control Plane
Owns model version inventory, deployment rings, device eligibility, rollback, evidence upload policy, and audit history.
16.13.3 Monitoring and Review
Tracks confidence, class distribution, alert volume, rejected samples, latency, memory, power, and examples for human review.
16.13.4 Training and Retraining
Uses reviewed evidence, representative datasets, test suites, and deployment metadata to improve future model versions.
def handle_sensor_window(window, model, policy, device):
features = device.preprocess(window)
result = model.predict(features)
if result.confidence < policy.minimum_confidence:
device.store_evidence(window, reason="low_confidence")
return device.safe_fallback("needs_review")
if result.label in policy.local_action_labels:
device.act(result.label)
device.report_event(result.summary())
return "acted_locally"
device.report_event(result.summary())
return "observed"16.14 Monitoring and Drift
Edge models drift when the world changes. The model is local, but the monitoring plan must still be fleet-wide.
16.14.1 Model Health
Track confidence distribution, class mix, unknown rate, low-confidence rate, model version, and runtime errors.
16.14.2 Device Health
Track latency, memory, reset reason, thermal state, battery or power budget, sensor quality, and local storage pressure.
16.14.3 Data Drift
Watch for changes in lighting, soundscape, vibration baseline, materials, site layout, firmware, or user behavior.
16.14.4 Review Signals
Collect false positives, missed events, operator overrides, discarded detections, and uncertain examples for retraining.
Checkpoint: Operations Evidence
You now know:
- Runtime architecture separates device or gateway inference, fleet control, monitoring and review, and training or retraining.
- Monitoring must watch model health, device health, data drift, and review signals even when inference stays local.
- A production record should name the model version, preprocessing version, confidence policy, fallback path, evidence upload rule, rollback trigger, and retraining trigger.
Use those operational signals to spot the common ways edge AI projects fail after the first demo.
16.15 Common Pitfalls
16.15.1 Starting With Hardware
Buying an accelerator before defining the local decision often leads to mismatched cost, power, memory, and update requirements.
16.15.2 Reusing Business Claims
Borrowed payback periods and savings estimates hide the real economics of false alarms, installation, review, maintenance, and retraining.
16.15.3 Ignoring Preprocessing
Training and deployment must use the same preprocessing. Small image, audio, or feature differences can dominate model behavior.
16.15.4 No Fallback Path
The device must define what happens when confidence is low, the model fails to load, the sensor is unhealthy, or storage is full.
16.15.5 No Evidence Loop
Edge inference without sampled evidence and operator feedback becomes hard to improve after release.
16.15.6 One-Shot Deployment
Production edge AI needs versioning, staged rollout, rollback, monitoring, and retraining triggers.
16.16 Deep Dive: Operating the Edge AI Lifecycle
An edge AI application is more than a trained network. In the field it is a running pipeline: capture, preprocess, infer, act or report, monitor, and update. The model is one controlled artifact inside that pipeline, alongside the preprocessing version, confidence policy, local action, evidence rules, fleet inventory, rollback path, and review workflow.
16.16.1 Pipeline Evidence
For example, a packing line may start with a defect classifier that works well in the lab. In production, the application also needs camera exposure control, a preprocessing version, a confidence threshold, a reject-lane action, a hold-for-review path, and evidence upload rules. If the model sees a low-confidence image, the safe behavior may be to hold the item and store the frame, not to force a reject. If alert volume jumps after a lens is cleaned or a light is replaced, monitoring should surface that drift before operators lose trust.
A small latency budget makes the operating boundary concrete. Suppose a camera line runs at 30 frames per second, so each frame arrives about every 33 ms. If capture takes 4 ms, preprocessing 6 ms, inference 12 ms, postprocessing 3 ms, reject-lane signaling 3 ms, and local evidence logging 2 ms, the edge path is 30 ms end to end and still has 3 ms of margin. Adding an 80 ms cloud round trip would miss the physical action window even if model accuracy were identical.
16.16.2 Release Evidence and Rollback
Treat a model version like firmware. A release record should name the training dataset, model hash, quantization settings, preprocessing version, minimum confidence, fallback policy, rollback trigger, and operator who can pause or resume the rollout.
| Stage | What Happens | Evidence To Keep |
|---|---|---|
| Train in cloud | Train and validate the full-precision model on representative data. | Dataset version, metrics, and exact model artifact. |
| Convert and optimize | Export to TensorFlow Lite/LiteRT or ONNX and quantize for the target. | On-target accuracy after conversion, not only server accuracy. |
| Deploy over the air | Roll a versioned model to a canary group first. | Device-to-model inventory, rollout ring, and rollback result. |
| Monitor | Track confidence, input statistics, alerts, sampled hard cases, and device health. | Drift signals and field cases for retraining. |
A practical rollout might start with ten devices on one production cell. During the canary, operators compare reject rate, low-confidence rate, latency, memory headroom, and sampled false-positive reviews against the previous model. If the reject rate doubles without a process change, the rollout pauses and the fleet control plane keeps the older model everywhere else. If signals stay inside limits, the release widens by site or region.
16.16.3 Drift Feedback Loop
Two kinds of drift erode a deployed model. Data drift is a change in the input distribution, such as new lighting, a replaced sensor, a new season, or a changed product mix. Concept drift is a change in the relationship itself, so the mapping the model learned is no longer correct. Both can appear as a quiet accuracy decline because the model still returns confident answers; they are just increasingly wrong.
The fix is a loop, not a patch. Sample hard cases from the field, send them to the cloud, relabel representative examples, retrain, re-optimize the candidate, test it against both the old validation set and the new field slice, and roll it out in stages. Without that evidence loop, drift becomes an unexplained local complaint instead of a measurable lifecycle event.
16.17 Summary
- Edge AI applications should be selected from measurable latency, privacy, bandwidth, autonomy, and operational-value evidence.
- Visual inspection, predictive maintenance, audio/event detection, and privacy-preserving sensing are common but different application families.
- The deployment loop includes use-case definition, data collection, baseline modeling, optimization, target-device validation, staged rollout, monitoring, and retraining.
- Target-device validation must measure the whole path, including preprocessing, inference, action, memory, power, thermal behavior, and fallback.
- Production systems need model versioning, rollback, cloud monitoring, evidence collection, and drift review.
16.18 Knowledge Check
16.19 Try It Yourself: Edge AI Application Record
Choose one application and complete a design record before choosing hardware.
application: visual-inspection-station
local_decision: detect missing label before package leaves station
why_edge:
- local action must happen before item moves downstream
- raw images should not be stored by default
sensor_context:
sensor: fixed camera
environment_risks: [lighting_change, motion_blur, label_variants]
model_plan:
baseline: train cloud model first
target_runtime: validate converted model on device class
optimization: quantize only after baseline is acceptable
validation:
measure: [accuracy, latency, memory, thermal, false_rejects, missed_defects]
fallback:
low_confidence: hold for human review
model_missing: route to manual inspection
monitoring:
collect: [confidence_distribution, class_mix, low_confidence_samples, model_version]
rollout:
strategy: staged deployment with rollbackReview the record with operations, privacy, and maintenance owners. A good edge AI application plan is a deployment and review plan, not just a model architecture.
16.20 References
16.21 What’s Next
16.21.1 Edge AI Fundamentals
Review the core reasons edge inference is useful and when cloud-only inference remains appropriate.
16.21.2 Edge AI Hardware
Compare microcontrollers, gateways, accelerators, memory budgets, and power constraints.
16.21.3 Edge AI Optimization
Learn how quantization, pruning, distillation, and conversion affect deployability.
16.21.4 Edge AI Lab
Practice a small end-to-end deployment workflow with target-device thinking.
16.23 Key Takeaway
Edge ML is most valuable when local inference changes the outcome: lower latency, lower bandwidth, better privacy, or continued operation during cloud loss. A smaller monitored model at the right tier often beats a larger model that cannot run reliably in the field.