21 Sensor Fusion Applications
21.1 Start With the Story
Picture an IoT team using the ideas in Sensor Fusion Applications during a live operations review. A device has produced messy evidence, an analytic step is about to change an alert or control decision, and someone has to explain why the result should be trusted.
Read this page as that path from sensor evidence to accountable action. Start with what the system observes, keep the model or data treatment visible, and finish with the check that would convince an operator, maintainer, or auditor to act.
21.2 Applications Define Fusion Contracts
Sensor fusion applications start with a decision, not with a fashionable algorithm. A phone needs a stable orientation for display and interaction. A wearable needs activity or fall evidence. A building system needs a trustworthy air-quality or occupancy state. A mobile robot needs an obstacle, pose, or track estimate with known uncertainty. Each application chooses sensors, latency, validation, and fallback rules around that decision.
The same raw sensor can play different roles in different applications. An accelerometer can supply gravity for roll and pitch, step features for activity recognition, vibration evidence for equipment monitoring, or a fault signal when it saturates. A good application design records the state being estimated, the sensors that can observe it, the failure modes each sensor introduces, and the evidence that downstream users need before acting.
Daily-life context recognition often uses indirect traces rather than one obvious sensor. A workshop task might be inferred from wrist vibration plus sound, a home scene from camera regions or thermal patches, appliance use from power-line, plumbing-pressure, or acoustic signatures, a vacuum surface from its spectrum, and attention or interface state from eye-movement or gaze-path sensors. A faucet, toilet, or tub event may appear as a pressure drop and recovery shape; a gaze trace may show attention only after it is tied to scene context. Those are all activity or object claims, so the fusion contract should state the proxy feature, placement, privacy boundary, and validated labels instead of treating the sensor name as proof.
Motion capture is a useful stress test for that contract. Optical rigs, marker suits, depth cameras, and body-worn IMUs can all produce a plausible moving body, but they carry different infrastructure and failure boundaries. An Orient-style wearable IMU design is attractive because it can be fully wireless and interactive, but the application still has to choose what travels over the radio. A low-rate Bluetooth path may be enough for a 200 Hz quaternion stream, while a faster 2.4 GHz path may be needed for roughly 2500 Hz quaternions or higher-rate gyroscope and accelerometer evidence. The record should therefore name the body placement, sample rates, radio mode, latency budget, wireless charging state, calibration pose, dropped-frame policy, and whether the display is a demonstration view or validated motion evidence.
Camera-based 3-D motion capture is a complementary fusion case, not just a bigger camera array. Each camera or marker view sees only part of the body and can be blocked by occlusion, lighting, reflective clothing, or calibration drift. A defensible capture record names the camera layout, calibration volume, marker set, synchronization method, reconstruction error, missing-marker policy, and the conditions where a depth-camera or wearable-IMU fallback is allowed to contribute.
Performance and game cases make the same contract stricter because the fused pose may drive an actuator immediately. In a music-from-motion design, a dancer’s 3-D orientation can be estimated from accelerometer, gyroscope, and magnetometer data and mapped into sound sequences; the review should record the raw sensor rate, quaternion update rate, sound-response latency, and the radio capacity that keeps multiple performers synchronized. A concrete record might include 512 Hz sensor data, 256 Hz quaternion updates, a 2.4 GHz Zigbee/TDMA link near 150 kbit/s, and whether the design still meets its timing budget at 128 Hz updates for 12 devices or 64 Hz updates for 24 devices. For avatar, game, or sports demos, the screenshot is only a demonstration artifact; the validation record still has to prove latency, synchronization, dropped frames, and realistic movement.
Robot imitation and two-person interaction add another boundary. A wearable demo may show a MotionViewer axis trace on a phone, a human arm driving a humanoid robot, or Speckled Tango Dancers with chest, waist, and foot landmarks. Treat these as relational motion claims: record the body segment placement, human-to-robot mapping, gesture replay alignment, per-person clock synchronization, and the metric being inferred. Chest-parallel angle difference, chest-foot motion correlation, and chest-bend angle are not just pictures; they are derived features that need timing and calibration evidence before they can support a robot-learning or social-interaction claim.
Environmental deployments show the difference between redundant and cooperative fusion. Five co-located temperature and relative-humidity sensors can be redundant witnesses for the same local air state, so the useful output is not only an average but also the spread, outlier status, sensor id, calibration age, and failure flag for each channel. An inertial-magnetic unit plus a barometric altimeter is different: accelerometer, gyroscope, magnetometer, and pressure signals jointly support heading and relative-height claims that no single channel can publish safely without axis, magnetic-quality, pressure-baseline, and weather-drift evidence.
The application should say what “good enough” means: maximum age, minimum sensor set, uncertainty limit, degraded-mode behavior, and validation dataset. Without those limits, a fused value can look precise while being stale, unobservable, or driven by a failed input.
State
The thing the application needs, such as pose, orientation, occupancy, air quality, activity, or machine condition.
Sensor Mix
The inputs that observe the state from different angles, with different noise, rate, and failure modes.
Decision Boundary
The threshold for display, alerting, control, or actuation, including confidence and freshness limits.
Fallback
The behavior when one sensor is stale, disagrees, saturates, or becomes unavailable.
Overview Knowledge Check
21.3 Match Sensors by State and Rate
Application fusion is usually more than averaging values. Measurements must describe the same state at compatible times, use the same coordinate frame and units, and include uncertainty or quality labels. For tracking applications, a data-association step decides whether camera, lidar, radar, radio, or inertial evidence belongs to the same object or person before any update is applied.
A simple inverse-variance fusion is a useful design check when several sensors report the same scalar state with independent errors. It is not a replacement for a full tracker, but it shows the core idea: lower-variance sensors receive more weight, and the fused output should publish its remaining uncertainty.
Worked example: obstacle range from three sensors state being estimated: pedestrian range ahead of a vehicle camera measurement: 44.8 m, sigma = 1.0 m lidar measurement: 45.1 m, sigma = 0.3 m radar measurement: 45.5 m, sigma = 0.5 m precision weight = 1 / sigma^2 camera precision = 1 / 1.0^2 = 1.00 lidar precision = 1 / 0.3^2 = 11.11 radar precision = 1 / 0.5^2 = 4.00 total precision = 16.11 normalized weights: camera = 1.00 / 16.11 = 0.062 lidar = 11.11 / 16.11 = 0.690 radar = 4.00 / 16.11 = 0.248 fused range: 44.8*0.062 + 45.1*0.690 + 45.5*0.248 = about 45.18 m fused standard deviation: sqrt(1 / total precision) = sqrt(1 / 16.11) = about 0.25 m Interpretation: The output is pulled mostly by lidar because its stated uncertainty is smallest. The application should still keep camera classification evidence and radar velocity evidence if those are needed for the decision.
Practitioner Knowledge Check
Phoebe’s Why
The inverse-variance table above treats camera, lidar, and radar sigma as three numbers to plug into a formula, but each sigma is the fingerprint of a completely different governing equation. Lidar and radar both invert a timing measurement: a pulse or wave leaves, bounces, returns, and range falls straight out of the round-trip time, so their precision is only as good as the clock or bandwidth that times the echo. A camera inverts a completely different equation – triangulation from a pixel-level disparity – and that equation is nonlinear in range: the same one-pixel wobble that barely matters up close turns into meters of range error far away, because the geometry that recovers distance from angle gets shallower and shallower the farther out you look. None of the three sigmas in this chapter’s own worked example is an opinion; each is what its own physics can and cannot resolve at 45 metres.
The Derivation
Time-of-flight ranging (lidar, radar) inverts a round-trip travel time:
\[r = \frac{c\,t}{2} \quad\Longrightarrow\quad \sigma_r = \frac{c\,\sigma_t}{2}\]
An FMCW radar’s native range resolution follows from its chirp sweep bandwidth \(B_{sweep}\):
\[\Delta R = \frac{c}{2\,B_{sweep}}\]
Triangulation ranging (stereo camera) inverts a disparity measured in pixels, with baseline \(B\) and focal length \(f\):
\[r = \frac{fB}{d} \quad\Longrightarrow\quad \left|\frac{dr}{dd}\right| = \frac{r^2}{fB}\]
so a fixed pixel-level disparity uncertainty produces a range error that grows with the square of range.
Worked Numbers: This Chapter’s Own Camera / Lidar / Radar Sigmas
- Lidar, \(\sigma_r=0.300\) m: implies a timing precision of \(\sigma_t = 2(0.300)/(3.00\times10^8) = 2.00\) ns – an ordinary figure for pulsed time-of-flight lidar, and the reason lidar earned the largest precision weight (11.1) in this chapter’s own fusion table.
- Radar, \(\sigma_r=0.500\) m: matches \(\Delta R = c/(2B_{sweep})\) for a catalog-typical automotive FMCW sweep bandwidth of \(300\) MHz exactly: \((3.00\times10^8)/(2\times3.00\times10^8) = 0.500\) m – radar’s coarser range resolution traces directly to its narrower sweep bandwidth versus lidar’s optical pulse, not to any inherent unreliability.
- Camera, \(\sigma_r=1.00\) m at this chapter’s own \(r=44.8\) m: for a catalog-typical stereo rig (\(f=1000\) px, \(B=0.300\) m), the implied disparity precision is \(dd = \sigma_r fB/r^2 = 1.00\times1000\times0.300/44.8^2 = 0.150\) px – a realistic sub-pixel matching precision, confirming the camera is not “worse,” it is solving a harder, quadratically range-dependent equation.
- Range check on the growth law: at twice the distance (\(r=89.6\) m) the same \(0.150\) px disparity precision would give \(\sigma_r = 0.150\times89.6^2/(1000\times0.300) = 4.02\) m – four times worse from only doubling range, the \(r^2\) term at work.
The fusion weights in this chapter’s own example are therefore not a black box: lidar dominates because time-of-flight physics is linear in its timing precision, while camera range noise is baked in by a quadratic geometric law that no amount of fusion math removes – only shorter range, wider baseline, or better optics can.
21.4 Timing Failures in Fusion
Many fusion failures are not caused by the filter equation. They come from stale evidence, mismatched coordinate frames, duplicate detections, missing context, weak validation labels, or a downstream decision that assumes normal confidence during degraded mode. These are application-level failures, so the application must define how evidence enters, ages, conflicts, and exits the fused state.
Latency deserves special attention. A measurement that was accurate when captured can be unsafe when used too late. Tracking systems therefore need event timestamps, prediction to the fusion time, stale thresholds, and a rule for publishing lower confidence when evidence is old or out of sequence.
Worked example: stale position evidence tracked object speed: 1.6 m/s camera event time: 120 ms before the fusion update lidar event time: 40 ms before the fusion update uncompensated motion during camera age: 1.6 m/s * 0.120 s = 0.192 m uncompensated motion during lidar age: 1.6 m/s * 0.040 s = 0.064 m application rule: If stale error budget is 0.10 m, the 120 ms camera measurement needs prediction, downweighting, or rejection before range fusion. The 40 ms lidar measurement is inside the budget if its other quality gates pass.
Event Time
Use the physical capture time for fusion. Arrival time can hide network, buffering, or processing delay.
Coordinate Frame
Record transforms between sensor, device, vehicle, room, map, and global frames before comparing values.
Ground Truth
Validate against labeled walks, surveyed anchors, reference instruments, or reviewed incidents that match the use case.
Operating Envelope
Define the environment, speed, load, lighting, weather, placement, and sensor health where the fused output is valid.
Under-the-Hood Knowledge Check
21.5 Summary
Sensor fusion applications connect algorithms to decisions. The application defines the estimated state, sensor mix, timing budget, uncertainty limit, minimum evidence set, validation data, and fallback behavior. Phone orientation, body motion capture, activity recognition, indoor tracking, environmental control, and object tracking all use different evidence contracts even when they share sensors. Reliable application fusion keeps data association, event time, coordinate frames, uncertainty, sensor health, degraded modes, and retest triggers visible with the fused output.
A sensor-fusion application is trustworthy when it publishes not just the fused state, but also the evidence boundary that says which sensors contributed, how fresh they were, how uncertain the result is, and what degraded mode is active.
21.6 See Also
Fusion Architectures
Choose centralised, hierarchical, distributed, or decision-level placement for an application.
Fusion Best Practices
Apply calibration, event-time alignment, gates, degraded modes, and retest triggers.
Kalman Filters
Use covariance and innovation evidence when tracking state with model and measurement updates.
Particle Filters
Use weighted hypotheses when an application has nonlinear, constrained, or multimodal state beliefs.