Edge & Fog Computing · Study deck

Hardware Accelerators for Edge AI

Picture a camera beside a conveyor that must spot a blocked guard and stop a feed.

Edge Eddie is your guide for this deck.

hardware

Edge Eddie, the module guide, in a scene from this chapter.

After studying this chapter

Learning objectives

You will be able to:

Compare microcontrollers, neural accelerators, edge GPUs, and FPGAs by workload fit rather than headline specifications.
Identify the hardware requirements created by model size, operators, input rate, memory, latency, power, and thermal limits.
Explain why TOPS, GFLOPS, frame-rate demos, and vendor benchmarks cannot be compared without workload context.
Design a benchmark plan that measures the complete edge inference path.

Major section

In 60 Seconds · Start Simple

Latency means the time from the physical event to the result that matters.

A fast model alone is not enough; the camera, data preparation, model, rule, output, and final motion all use part of that time.
A large speed number from one model step is not system proof.
This opening does not rank every chip.

Major section

Minimum Viable Understanding · Most Valuable Understanding

The full path must be benchmarked.: Sensor capture, preprocessing, model invocation, postprocessing, local action, and logging all count.

Hardware is a constraint system.: Compute is only one constraint; memory, I/O, power, heat, runtime support, and updates can dominate.
Accelerator metrics are not interchangeable.: Neural-operation metrics, floating-point metrics, and vendor benchmark charts measure different workloads.
Deployment support matters.: Model registry, versioning, observability, security updates, and rollback are part of hardware selection.

Major section

Start With Requirements

Deployment scale: supply availability, manufacturability, certification, maintenance, replacement, and support lifetime.

A tiny model can point toward MCU; standard inference may suit an NPU or TPU; a flexible pipeline can justify an Edge GPU; and custom timing may lead to FPGA.

A hardware selection map showing MCU, neural accelerator, edge GPU, and FPGA choices from workload constraints

Major section

Hardware Families · Avoid Universal Hardware Rankings

The main constraints are memory, input size, and energy budget.

Best for flexible pipelines, multiple models, custom preprocessing, larger input tensors, development velocity, and frameworks that need general parallel compute.
Best for custom signal processing, deterministic timing, unusual I/O, and hardware pipelines where software scheduling jitter is unacceptable.
A wearable event detector, a warehouse camera, a vibration gateway, and a robotic safety controller need different tradeoffs.

Major section

Operator and Runtime Fit · CPU Fallback Can Hide Failure

A model that performs well in training may fail to map cleanly to hardware.

Runtime fit is often the deciding factor.
Some accelerators require a model compiler.
Unsupported layers may fall back to CPU, fail conversion, or change latency unpredictably.
The hardware may favor integer, floating-point, or mixed-precision inference.

Major section

Benchmark the Whole Path

Hardware evaluation should be a repeatable experiment.

The benchmark must use representative input and measure the complete deployed path.
Benchmark measures the real pipeline; Thermal repeats the work in the power case; and Update path checks version and rollback support.

A hardware validation loop showing requirements, candidate hardware, real benchmark, thermal and power test, update path, and decision

Major section

Power, Thermal, and Enclosure Reality · Software Ecosystem and Lifecycle

Peak inference power may be acceptable if the duty cycle is low.

Continuous inference needs sustained thermal testing.
Outdoor boxes, sealed industrial enclosures, wearable housings, and drones have different cooling paths and ambient conditions.
A board that meets latency when cool can miss deadlines after sustained load or high ambient temperature.

Major section

Common Pitfalls · Implementation Sketch

Evaluation kits are cheaper than a fleet mistake.

Peak compute numbers do not include memory bottlenecks, unsupported operators, preprocessing, thermals, or I/O.
Unsupported operations may run on the CPU and quietly break latency or power assumptions.
Desk performance can fail in heat, dust, vibration, sealed housings, or battery-powered duty cycles.

Major section

Deep Dive: Sizing Hardware From Workload Evidence

A board that can only sustain 20 million MACs per second on the real graph will miss the deadline even if its sleep current looks attractive.

Memory starts with weights, activations, tensor arena, input buffers, runtime libraries, and whether the accelerator can keep data close enough to its arithmetic units.
If one MAC is counted as a multiply plus an add, that is roughly 160 million operations per second.
Memory can be the harder constraint.

Major section

Deep Dive: Sizing Hardware From Workload Evidence (continued)

If a converted model needs 8 million MACs per inference and the application needs 10 inferences per second, the sustained demand is 80 million MACs per second.

A neural accelerator with ample TOPS may meet the compute target, but only if the model's operators and tensors stay on the accelerated path.
A small model may fit in flash but still fail if activation buffers, input frames, tensor arena, and runtime memory exceed available RAM.
A vision target must also move camera frames through resize, normalization, inference, postprocessing, action, and logging.

Major section

Deep Dive: Sizing Hardware From Workload Evidence (continued)

The hardware decision should therefore record both compute demand and memory headroom for the final converted artifact, not the training model.

The winning board is the one that meets the full sensor-to-action deadline under the target duty cycle and enclosure conditions.
A general-purpose CPU core executes only a limited number of MACs per cycle.
The catch is that raw MAC throughput is only useful if the array stays fed.

Major section

Phoebe's Field Notes: The Other Quantization -- From FP32 to INT8 · Summary

The mathematical gist.: Over $[-1,1]$, INT8 has $q=2/256=7.81\times10^{-3}$, $q_{rms}=2.26\times10^{-3}$, and an ideal 49.9 dB quantisation ceiling; INT16 reaches 98.1 dB.

Edge AI hardware selection starts with workload and deployment requirements, not a board name.

Deck summary

Key takeaways

Latency means the time from the physical event to the result that matters.

The full path must be benchmarked.: Sensor capture, preprocessing, model invocation, postprocessing, local action, and logging all count.
Deployment scale: supply availability, manufacturability, certification, maintenance, replacement, and support lifetime.
The main constraints are memory, input size, and energy budget.
A model that performs well in training may fail to map cleanly to hardware.

Retrieval practice

Recall check 1 of 6

Edge Eddie says: answer from memory, then check your reasoning.

Q1A team wants to choose an accelerator before deciding the model, preprocessing path, sensor rate, or enclosure. What is the main risk?

AThe accelerator will automatically make every model faster, even when operators, memory, and data movement do not fit

BThe cloud will no longer be needed for model registry, monitoring, updates, or rollback once the board is selected

CThe selected hardware may not support the real model, latency, memory, power, thermal, or update requirements

DThe sensor rate no longer matters once a board has an accelerator and a published benchmark

Show answer

Answer: C A hardware decision is defensible only when it is tied to the actual model, pipeline, runtime, power, thermal, and lifecycle requirements.

Retrieval practice

Recall check 2 of 6

Edge Eddie says: answer from memory, then check your reasoning.

Q2Why is it risky to compare one board's TOPS number directly with another board's GFLOPS number?

ABecause TOPS counts operations per inference while GFLOPS measures how quickly an inference finishes

BBecause converting GFLOPS to TOPS by unit scale is enough to rank the boards for the model

CThey measure different kinds of operations and may apply to different precisions, runtimes, and model graphs

DBecause frame rate is the only metric that matters, even when latency, power, and thermals fail

Show answer

Answer: C TOPS, GFLOPS, latency, and frame-rate numbers are workload-specific.

Retrieval practice

Recall check 3 of 6

Edge Eddie says: answer from memory, then check your reasoning.

Q3A board meets latency targets on a desk but misses them inside a sealed outdoor enclosure after running for an hour. What likely changed?

AThe model became untrained because it ran locally for too long in the enclosure

BThe benchmark was too complete, so it should ignore sustained power and heat

CThermal conditions caused throttling or changed sustained performance

DThe cloud should choose the board automatically without thermal validation

Show answer

Answer: C Thermal and enclosure testing are part of hardware validation because sustained performance may differ from short desk benchmarks.

Retrieval practice

Recall check 4 of 6

Edge Eddie says: answer from memory, then check your reasoning.

Q4Place each hardware or operations target where it lives so you can shortlist an edge AI platform from workload evidence.

AMCU

BCloud database

CTraining cluster

DSearch index

Show answer

Answer: A Separate constrained devices, throughput engines, and fleet operations so you can match compute evidence to the deployed workload without forgetting lifecycle support.

Retrieval practice

Recall check 5 of 6

Edge Eddie says: answer from memory, then check your reasoning.

Q5A camera pipeline uses standard image classification, has a stable input shape, and the converted model is fully supported by a neural accelerator. What should still be measured before deployment?

AOnly the accelerator's peak TOPS value, because supported operators prove the full pipeline

BMeasure the complete camera-to-action path.

COnly whether the model runs once on a desk, without thermal, logging, or action timing

DMeasure inference latency for the converted model.

Show answer

Answer: B Even when a model maps cleanly to an accelerator, the full sensor-to-action path and deployment environment must be measured.

Retrieval practice

Recall check 6 of 6

Edge Eddie says: answer from memory, then check your reasoning.

Q6A team has narrowed hardware to two candidates. Candidate A is slightly faster in isolated inference. Candidate B has better driver support, update tooling, rollback, and monitoring hooks. What should drive the final choice?

AAlways pick isolated inference speed, even if updates are manual and rollback is unclear

BMeasured deployment readiness, including lifecycle support, not isolated speed alone

CIgnore software support because hardware is separate from field monitoring and maintenance

DPick neither, because edge hardware cannot be updated safely after installation

Show answer

Answer: B Final hardware selection should weigh the complete deployment lifecycle: performance, software support, observability, updates, and rollback.

Print reference

Answers 1 of 2

Answer key.

C · A hardware decision is defensible only when it is tied to the actual model, pipeline, runtime, power, thermal, and lifecycle requirements.
C · TOPS, GFLOPS, latency, and frame-rate numbers are workload-specific.
C · Thermal and enclosure testing are part of hardware validation because sustained performance may differ from short desk benchmarks.
A · Separate constrained devices, throughput engines, and fleet operations so you can match compute evidence to the deployed workload without forgetting lifecycle support.

Print reference

Answers 2 of 2

Answer key.

B · Even when a model maps cleanly to an accelerator, the full sensor-to-action path and deployment environment must be measured.
B · Final hardware selection should weigh the complete deployment lifecycle: performance, software support, observability, updates, and rollback.