UX Design · Study deck
Multimodal Interfaces: Channel Contracts
A voice prompt, screen, and vibration can disagree during one alert.
UX Uma is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- Design Multimodal Interactions: Create interfaces that support voice, touch, physical, and gesture modalities appropriately
- Apply Modality Selection Frameworks: Match interface modality to user context and task complexity
- Implement Graceful Degradation: Design systems that continue functioning when components fail
- Balance Tradeoffs: Make informed decisions between touch vs. voice, visual vs. audio, and cloud vs. local architectures
Major section
Start Simple
A worker wearing gloves must stop a noisy pump.
- A small touch target may fail through the glove.
- A spoken warning may vanish under machine noise.
- A red light may not help someone facing away or unable to distinguish its colour.
- The design must start with the important action, not with a list of fashionable input modes.
Major section
Start Simple (continued)
The worker should know whether the system heard the action and whether the machine actually reached the safe state.
- More modes do not automatically mean better access or safer control.
- The deeper sections compare voice, touch, physical controls, visual and touch-based feedback, redundancy, timing, and the field checks that prove the chosen combination supports the named action.
- Multimodal design is not about adding channels for novelty; it is about giving people reliable ways to perceive and control important states.
Major section
Phoebe's Field Notes: Why a Capacitive Button Goes Deaf Under a Glove
The mathematical gist.: A 1.00 mm glass overlay gives about 3.54 pF of finger coupling.
- Recovering a nine-times smaller signal by averaging alone needs about 81 samples and raises this teaching scan from 0.0100 mA to 0.810 mA.
Major section
MVU: Multimodal Interaction Patterns
Core Concept: IoT interfaces must provide feedback through multiple simultaneous channels (visual, audio, haptic) because users interact in varied contexts where any single modality may be unavailable or inappropriate.
- If feedback requires focused attention on a single channel (reading text, counting LED blinks), users will miss critical information and lose trust in the system.
Major section
Multimodal Interface Basics
IoT devices are used in many different contexts -- a kitchen with greasy hands, a noisy factory floor, a dark bedroom.
- No single interaction method works everywhere.
- Multimodal design means supporting multiple ways to interact: voice ("turn off the lights"), touch (tapping an app), physical controls (pressing a button), and gesture (waving to activate).
Major section
Multimodal Redundant Meaning
Multimodal IoT design is not about adding every possible input.
- The channels must agree about the same state.
- If the app says locked while the device is still pending, redundant channels spread confusion faster because each channel is repeating a different version of the truth.
- Each channel has a narrower job than the whole product.
Major section
Multimodal Redundant Meaning (continued)
Channel choice should begin with the user's situation, not the device's feature list. The linked figure in Part 2 pairs contexts with modalities so each channel can be assigned a job and a fallback.
- On the right,: Voice can serve hands-free input,: Touch Screen can carry complex configuration,: Physical Controls preserve local action, and a: Wearable can provide discreet cues.
- The best-practice strip requires two or more modalities per task, an offline fallback, and accessibility across diverse needs.
- A phone app can explain settings, but it should not be the only control for heating, access, alarms, or other local functions.
Major section
Multimodal Redundant Meaning (continued)
A voice command can start a simple action, but it should not be the only path for a visitor, caregiver, technician, or child who lacks account access.
- A privacy indicator can help, but for a camera or microphone it may need a physical shutter or disconnect if the user must verify the claim without trusting software.
- That contract keeps multimodal design from becoming a feature list.
- It turns the design into a set of promises that can be tested with users, prototypes, logs, and failure drills.
- A strong design chooses modalities because each one covers a specific user need, attention state, risk level, or failure mode.
Major section
Practitioner: Specify Channel Contracts
Each modality needs a contract that designers and engineers can test.
- Voice may depend on wake-word detection, on-device speech recognition, cloud ASR, NLU intent matching, and TTS or pre-recorded confirmation.
- Touch may depend on app state, target size, focus order, screen-reader labels, and network freshness.
- These choices affect privacy, latency, accessibility, and support.
Major section
Practitioner: Specify Channel Contracts (continued)
Physical controls may depend on switch type, debounce, press duration, tactile shape, LED placement, and enclosure affordance.
- For physical controls, test tactile identification, long-press versus short-press ambiguity, debounce, accidental activation, and whether the enclosure hides the control.
- A Figma prototype can test wording and flow, but not radio latency or buzzer volume.
- A Wokwi or bench prototype can test button, LED, OLED, serial, and firmware state behavior, but not household notification fatigue.
Major section
Sync Modalities Around State
Multimodal interfaces fail when channels race each other.
- A mobile app may optimistically show unlocked, while the physical lock rejected the command.
- A push notification may arrive after the local LED already shows recovery.
- The design needs a state model shared across channels.
- Authority: separate requested, accepted, pending, completed, rejected, stale, offline, unsafe, and overridden states.
Major section
Sync Modalities Around State (continued)
Under-the-hood review should name the source of truth and timing budget for each state.
- The core implementation pattern is a small state machine rather than separate scripts for each channel.
- A command can move through requested, locally accepted, sent, queued, delivered, applied, verified, rejected, timed out, stale, and overridden.
- That model also needs ownership rules.
Major section
Sync Modalities Around State (continued)
Not every product needs every state, but the chosen states should be named consistently across firmware, mobile UI, hub UI, cloud API, analytics, and support tools.
- A local physical switch may override a cloud schedule.
- A hardware shutter should override camera streaming even if an app toggle says enabled.
- Engineers can make the contract testable with instrumentation.
Major section
Sync Modalities Around State (continued)
A hub may keep cached control during internet loss, but should label data freshness when cloud dashboards reconnect.
- A BLE direct-control session may be authoritative for setup, while a cloud device shadow is authoritative for remote monitoring.
- The interface should expose the ownership boundary in plain language instead of showing a generic spinner or success toast.
- For Matter or Zigbee devices, distinguish attribute reports from command requests.
Major section
Sync Modalities Around State (continued)
For mobile notifications, record whether APNs or FCM delivery is only awareness, not proof that the device acted.
- Failure drills should exercise the real split between channels: cloud outage, hub restart, phone offline, muted speaker, display failure, low battery, privacy switch closed, delayed actuator acknowledgement, duplicate command, and role conflict.
- A multimodal interface is production-ready only when those drills produce understandable feedback and a safe next action.
- Otherwise, one surface may show success while another still has only a request.
Major section
Talk, Touch, or Tap?
Today we're learning about the different ways you can talk to your smart devices!
- Voice (talking): "Hey lamp, turn blue!": Great when your hands are full with pizza!
- Motion Marley adds: "And some devices can even see you wave your hand!
Deck summary
Key takeaways
A worker wearing gloves must stop a noisy pump.
- The worker should know whether the system heard the action and whether the machine actually reached the safe state.
- The mathematical gist.: A 1.00 mm glass overlay gives about 3.54 pF of finger coupling.
- Core Concept: IoT interfaces must provide feedback through multiple simultaneous channels (visual, audio, haptic) because users interact in varied contexts where any single modality may be unavailable or inappropriate.
- IoT devices are used in many different contexts -- a kitchen with greasy hands, a noisy factory floor, a dark bedroom.
Retrieval practice
Recall check 1 of 2
UX Uma says: answer from memory, then check your reasoning.
Q1A gloved worker must stop a noisy pump. What should the channel design prioritize?
Show answer
Answer: C The example tests gloves, noise, view, and the distinction between request and confirmation.
Retrieval practice
Recall check 2 of 2
UX Uma says: answer from memory, then check your reasoning.
Q2A kitchen user has greasy hands and looks away from a device. What feedback design fits the chapter’s principle?
Show answer
Answer: B The chapter pairs channels so feedback remains available across different attention and ability conditions.
Print reference
Answers
Answer key.
- C · The example tests gloves, noise, view, and the distinction between request and confirmation.
- B · The chapter pairs channels so feedback remains available across different attention and ability conditions.