Analytics & ML · Study deck
Sensor Fusion Applications
Imagine a shuttle enters a busy depot and several sensors disagree about an object near its path.
Data Dora is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- Explain: A good application design records the state being estimated, the sensors that can observe it, the failure modes each sensor introduces, and the evidence that downstream users need before acting.
- Explain: A wearable demo may show a MotionViewer axis trace on a phone, a human arm driving a humanoid robot, or Speckled Tango Dancers with chest, waist, and foot landmarks.
- Explain: A faucet, toilet, or tub event may appear as a pressure drop and recovery shape; a gaze trace may show attention only after it is tied to scene context.
- match sensor mixes to state and rate
Major section
Decide Whether the Shuttle Can Move
The safety choice must include how uncertain the position is; a missing or drifting sensor can change whether moving is allowed.
- The team must decide whether the shuttle may move, slow down, or stop.
- Latency means the wait from a real event to the system's response.
- Say what state the team needs.
Major section
Decide Whether the Shuttle Can Move (continued)
At the depot, the object estimate is usable only if the shuttle's camera, lidar, and radar readings refer to the same object at compatible times.
- If the readings cannot be matched to one object, record that mismatch and use the stop or slow response chosen for weak evidence.
- A precise-looking distance cannot rescue a bad object match or a drifting clock.
- This short path is the core of fusion.
Major section
Applications Define Fusion Contracts
A phone needs a stable orientation for display and interaction.
- A wearable needs activity or fall evidence.
- A building system needs a trustworthy air-quality or occupancy state.
- A mobile robot needs an obstacle, pose, or track estimate with known uncertainty.
- The same raw sensor can play different roles in different applications.
Major section
Applications Define Fusion Contracts (continued)
Optical rigs, marker suits, depth cameras, and body-worn IMUs can all produce a plausible moving body, but they carry different infrastructure and failure boundaries.
- An accelerometer can supply gravity for roll and pitch, step features for activity recognition, vibration evidence for equipment monitoring, or a fault signal when it saturates.
- Daily-life context recognition often uses indirect traces rather than one obvious sensor.
- Motion capture is a useful stress test for that contract.
Major section
Applications Define Fusion Contracts (continued)
A good application design records the state being estimated, the sensors that can observe it, the failure modes each sensor introduces, and the evidence that downstream users need before acting.
- A faucet, toilet, or tub event may appear as a pressure drop and recovery shape; a gaze trace may show attention only after it is tied to scene context.
- An Orient-style wearable IMU design is attractive because it can be fully wireless and interactive, but the application still has to choose what travels over the radio.
- Camera-based 3-D motion capture is a complementary fusion case, not just a bigger camera array.
Major section
Applications Define Fusion Contracts (continued)
Performance and game cases make the same contract stricter because the fused pose may drive an actuator immediately.
- A low-rate Bluetooth path may be enough for a 200 Hz quaternion stream, while a faster 2.4 GHz path may be needed for roughly 2500 Hz quaternions or higher-rate gyroscope and accelerometer evidence.
- The record should therefore name the body placement, sample rates, radio mode, latency budget, wireless charging state, calibration pose, dropped-frame policy, and whether the display is a demonstration view or validated motion evidence.
- Decision BoundaryThe threshold for display, alerting, control, or actuation, including confidence and freshness limits.
Major section
Applications Define Fusion Contracts (continued)
Each camera or marker view sees only part of the body and can be blocked by occlusion, lighting, reflective clothing, or calibration drift.
- A defensible capture record names the camera layout, calibration volume, marker set, synchronization method, reconstruction error, missing-marker policy, and the conditions where a depth-camera or wearable-IMU fallback is allowed to contribute.
- For avatar, game, or sports demos, the screenshot is only a demonstration artifact; the validation record still has to prove latency, synchronization, dropped frames, and realistic movement.
- Without those limits, a fused value can look precise while being stale, unobservable, or driven by a failed input.
Major section
Applications Define Fusion Contracts (continued)
A wearable demo may show a MotionViewer axis trace on a phone, a human arm driving a humanoid robot, or Speckled Tango Dancers with chest, waist, and foot landmarks.
- The application should say what "good enough" means: maximum age, minimum sensor set, uncertainty limit, degraded-mode behavior, and validation dataset.
- The first stage of Figure: A fusion application should name the decision pipeline separates Temperature: Humidity: Motion: Light, and: Door readings.
- FallbackThe behavior when one sensor is stale, disagrees, saturates, or becomes unavailable.
- Object use, appliance event, room activity, surface, attention, or daily-activity label with confidence and unknown state.
Major section
Match Sensors by State and Rate
Application fusion is usually more than averaging values.
- Measurements must describe the same state at compatible times, use the same coordinate frame and units, and include uncertainty or quality labels.
- A simple inverse-variance fusion is a useful design check when several sensors report the same scalar state with independent errors.
- For tracking applications, a data-association step decides whether camera, lidar, radar, radio, or inertial evidence belongs to the same object or person before any update is applied.
Major section
Timing Failures in Fusion
application rule: If stale error budget is 0.10 m, the 120 ms camera measurement needs prediction, downweighting, or rejection before range fusion.
- Many fusion failures are not caused by the filter equation.
- A measurement that was accurate when captured can be unsafe when used too late.
- Event TimeUse the physical capture time for fusion.
Major section
Timing Failures in Fusion (continued)
Tracking systems therefore need event timestamps, prediction to the fusion time, stale thresholds, and a rule for publishing lower confidence when evidence is old or out of sequence.
- The 40 ms lidar measurement is inside the budget if its other quality gates pass.
- Arrival time can hide network, buffering, or processing delay.
- Ground TruthValidate against labeled walks, surveyed anchors, reference instruments, or reviewed incidents that match the use case.
Major section
Timing Failures in Fusion (continued)
Operating EnvelopeDefine the environment, speed, load, lighting, weather, placement, and sensor health where the fused output is valid.
- The system reports a precise state after losing a key sensor.
- Fallback did not update uncertainty or mode labels.
- Activity, occupancy, or condition labels degrade after deployment changes.
- Training labels, sensor placement, or environment no longer match production.
Deck summary
Key takeaways
The safety choice must include how uncertain the position is; a missing or drifting sensor can change whether moving is allowed.
- At the depot, the object estimate is usable only if the shuttle's camera, lidar, and radar readings refer to the same object at compatible times.
- A phone needs a stable orientation for display and interaction.
- Optical rigs, marker suits, depth cameras, and body-worn IMUs can all produce a plausible moving body, but they carry different infrastructure and failure boundaries.
- Performance and game cases make the same contract stricter because the fused pose may drive an actuator immediately.
Retrieval practice
Recall check 1 of 3
Data Dora says: answer from memory, then check your reasoning.
Q1What should a sensor-fusion application define before choosing the algorithm?
Show answer
Answer: A Application requirements define the fusion output, evidence, and fallback rules.
Retrieval practice
Recall check 2 of 3
Data Dora says: answer from memory, then check your reasoning.
Q2In the obstacle-range example, why does lidar receive the largest normalized weight?
Show answer
Answer: C Inverse-variance fusion assigns larger weights to measurements with lower stated variance.
Retrieval practice
Recall check 3 of 3
Data Dora says: answer from memory, then check your reasoning.
Q3Why can a sensor measurement become unsafe even if it was accurate when captured?
Show answer
Answer: B Application-level fusion must account for event time, state motion, and stale evidence.
Print reference
Answers
Answer key.
- A · Application requirements define the fusion output, evidence, and fallback rules.
- C · Inverse-variance fusion assigns larger weights to measurements with lower stated variance.
- B · Application-level fusion must account for event time, state motion, and stale evidence.