Edge & Fog Computing · Study deck
Hardware Accelerators for Edge AI
Picture a camera beside a conveyor that must spot a blocked guard and stop a feed.
Edge Eddie is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- Compare microcontrollers, neural accelerators, edge GPUs, and FPGAs by workload fit rather than headline specifications.
- Identify the hardware requirements created by model size, operators, input rate, memory, latency, power, and thermal limits.
- Explain why TOPS, GFLOPS, frame-rate demos, and vendor benchmarks cannot be compared without workload context.
- Design a benchmark plan that measures the complete edge inference path.
Major section
In 60 Seconds
Latency means the time from the physical event to the result that matters.
- A fast model alone is not enough; the camera, data preparation, model, rule, output, and final motion all use part of that time.
- A large speed number from one model step is not system proof.
- This opening does not rank every chip.
Major section
In 60 Seconds (continued)
The wider service can retain evidence and improve models, but it must not hide a missed deadline.
- Edge AI hardware selection starts with the workload, not the most impressive accelerator.
- A good choice fits the model artifact, supported operators, sensor pipeline, latency deadline, memory footprint, power budget, thermal envelope, software stack, update process, and deployment scale.
- The right answer is proven by benchmarking the real model and preprocessing path on the target class of hardware, then validating power, thermal behavior, rollback, and monitoring before rollout.
Major section
Minimum Viable Understanding · Most Valuable Understanding
The full path must be benchmarked: Sensor capture, preprocessing, model invocation, postprocessing, local action, and logging all count.
- Hardware is a constraint system: Compute is only one constraint; memory, I/O, power, heat, runtime support, and updates can dominate.
- Accelerator metrics are not interchangeable: Neural-operation metrics, floating-point metrics, and vendor benchmark charts measure different workloads.
- Deployment support matters: Model registry, versioning, observability, security updates, and rollback are part of hardware selection.
Major section
Start With Requirements
Deployment scale: supply availability, manufacturability, certification, maintenance, replacement, and support lifetime.
- A tiny model can point toward MCU; standard inference may suit an NPU or TPU; a flexible pipeline can justify an Edge GPU; and custom timing may lead to FPGA.
Major section
Hardware Families · Avoid Universal Hardware Rankings
The main constraints are memory, input size, and energy budget.
- Best for flexible pipelines, multiple models, custom preprocessing, larger input tensors, development velocity, and frameworks that need general parallel compute.
- Best for custom signal processing, deterministic timing, unusual I/O, and hardware pipelines where software scheduling jitter is unacceptable.
- A wearable event detector, a warehouse camera, a vibration gateway, and a robotic safety controller need different tradeoffs.
Major section
Operator and Runtime Fit · CPU Fallback Can Hide Failure
A model that performs well in training may fail to map cleanly to hardware.
- Runtime fit is often the deciding factor.
- Some accelerators require a model compiler.
- Unsupported layers may fall back to CPU, fail conversion, or change latency unpredictably.
- The hardware may favor integer, floating-point, or mixed-precision inference.
Major section
Benchmark the Whole Path
Hardware evaluation should be a repeatable experiment.
- The benchmark must use representative input and measure the complete deployed path.
- Benchmark measures the real pipeline; Thermal repeats the work in the power case; and Update path checks version and rollback support.
Major section
Power, Thermal, and Enclosure Reality · Software Ecosystem and Lifecycle
Peak inference power may be acceptable if the duty cycle is low.
- Continuous inference needs sustained thermal testing.
- Outdoor boxes, sealed industrial enclosures, wearable housings, and drones have different cooling paths and ambient conditions.
- A board that meets latency when cool can miss deadlines after sustained load or high ambient temperature.
Major section
Common Pitfalls
Evaluation kits are cheaper than a fleet mistake.
- Peak compute numbers do not include memory bottlenecks, unsupported operators, preprocessing, thermals, or I/O.
- Unsupported operations may run on the CPU and quietly break latency or power assumptions.
- Desk performance can fail in heat, dust, vibration, sealed housings, or battery-powered duty cycles.
Major section
Deep Dive: Sizing Hardware From Workload Evidence
A board that can only sustain 20 million MACs per second on the real graph will miss the deadline even if its sleep current looks attractive.
- Memory starts with weights, activations, tensor arena, input buffers, runtime libraries, and whether the accelerator can keep data close enough to its arithmetic units.
- If one MAC is counted as a multiply plus an add, that is roughly 160 million operations per second.
- Memory can be the harder constraint.
Major section
Deep Dive: Sizing Hardware From Workload Evidence (continued)
If a converted model needs 8 million MACs per inference and the application needs 10 inferences per second, the sustained demand is 80 million MACs per second.
- A neural accelerator with ample TOPS may meet the compute target, but only if the model's operators and tensors stay on the accelerated path.
- A small model may fit in flash but still fail if activation buffers, input frames, tensor arena, and runtime memory exceed available RAM.
- A vision target must also move camera frames through resize, normalization, inference, postprocessing, action, and logging.
Major section
Deep Dive: Sizing Hardware From Workload Evidence (continued)
The hardware decision should therefore record both compute demand and memory headroom for the final converted artifact, not the training model.
- The winning board is the one that meets the full sensor-to-action deadline under the target duty cycle and enclosure conditions.
- A general-purpose CPU core executes only a limited number of MACs per cycle.
- The catch is that raw MAC throughput is only useful if the array stays fed.
Major section
Phoebe's Field Notes: The Other Quantization -- From FP32 to INT8 · Summary
The mathematical gist: Over [−1,1], INT8 has q=2/256=7.81×10−3, qrms=2.26×10−3, and an ideal 49.9 dB quantisation ceiling; INT16 reaches 98.1 dB.
- Edge AI hardware selection starts with workload and deployment requirements, not a board name.
Deck summary
Key takeaways
Latency means the time from the physical event to the result that matters.
- The wider service can retain evidence and improve models, but it must not hide a missed deadline.
- The full path must be benchmarked: Sensor capture, preprocessing, model invocation, postprocessing, local action, and logging all count.
- Deployment scale: supply availability, manufacturability, certification, maintenance, replacement, and support lifetime.
- The main constraints are memory, input size, and energy budget.
Retrieval practice
Recall check 1 of 5
Edge Eddie says: answer from memory, then check your reasoning.
Q1A team wants to choose an accelerator before deciding the model, preprocessing path, sensor rate, or enclosure. What is the main risk?
Show answer
Answer: B A hardware decision is defensible only when it is tied to the actual model, pipeline, runtime, power, thermal, and lifecycle requirements.
Retrieval practice
Recall check 2 of 5
Edge Eddie says: answer from memory, then check your reasoning.
Q2Why is it risky to compare one board's TOPS number directly with another board's GFLOPS number?
Show answer
Answer: A TOPS, GFLOPS, latency, and frame-rate numbers are workload-specific.
Retrieval practice
Recall check 3 of 5
Edge Eddie says: answer from memory, then check your reasoning.
Q3A board meets latency targets on a desk but misses them inside a sealed outdoor enclosure after running for an hour. What likely changed?
Show answer
Answer: A Thermal and enclosure testing are part of hardware validation because sustained performance may differ from short desk benchmarks.
Q4What constrained computing device can run a tiny model under tight memory and power limits?
Show answer
Answer: D Tiny models can run within tight memory and power budgets.
Retrieval practice
Recall check 4 of 5
Edge Eddie says: answer from memory, then check your reasoning.
Q5A camera pipeline uses standard image classification, has a stable input shape, and the converted model is fully supported by a neural accelerator. What should still be measured before deployment?
Show answer
Answer: C Even when a model maps cleanly to an accelerator, the full sensor-to-action path and deployment environment must be measured.
Retrieval practice
Recall check 5 of 5
Edge Eddie says: answer from memory, then check your reasoning.
Q6A team has narrowed hardware to two candidates. Candidate A is slightly faster in isolated inference. Candidate B has better driver support, update tooling, rollback, and monitoring hooks. What should drive the final choice?
Show answer
Answer: D Final hardware selection should weigh the complete deployment lifecycle: performance, software support, observability, updates, and rollback.
Print reference
Answers 1 of 2
Answer key.
- B · A hardware decision is defensible only when it is tied to the actual model, pipeline, runtime, power, thermal, and lifecycle requirements.
- A · TOPS, GFLOPS, latency, and frame-rate numbers are workload-specific.
- A · Thermal and enclosure testing are part of hardware validation because sustained performance may differ from short desk benchmarks.
- D · Tiny models can run within tight memory and power budgets.
Print reference
Answers 2 of 2
Answer key.
- C · Even when a model maps cleanly to an accelerator, the full sensor-to-action path and deployment environment must be measured.
- D · Final hardware selection should weigh the complete deployment lifecycle: performance, software support, observability, updates, and rollback.