UX Design · Study deck

Multimodal Interfaces: Cases and Practice

A doorbell can show video, speak an alert, and vibrate a watch, yet still hide a failed command.

UX Uma is your guide for this deck.

interfacemultimodal

UX Uma, the module guide, in a scene from this chapter.

After studying this chapter

Learning objectives

You will be able to:

Explain: Lesson learned: The Echo Show's physical camera shutter (a sliding plastic cover) demonstrates an important principle -- some privacy controls must be physical, not software-based, because users need absolute certainty that the camera is off.
Explain: Mother is in the kitchen (hands wet, can't touch phone), father is in basement workshop (noisy, can't hear audio), teenager is upstairs with headphones (can't hear doorbell).
Explain: Graceful Degradation: Design five levels of degradation from full cloud through local hub, direct control, conservation, and emergency mode -- core functions must never depend on internet.

Major section

Real-World Case Study: Amazon Echo Show

Lesson learned: The Echo Show's physical camera shutter (a sliding plastic cover) demonstrates an important principle -- some privacy controls must be physical, not software-based, because users need absolute certainty that the camera is off.

No amount of on-screen indicators can match the trust of a physical barrier.

Major section

Smart Doorbell Multimodal Design

Active ring draw while audio, LED, radio, processor, and video preview are running.

Scenario:: A family has a smart doorbell.
Mother is in the kitchen (hands wet, can't touch phone), father is in basement workshop (noisy, can't hear audio), teenager is upstairs with headphones (can't hear doorbell).
Cloud notification, video relay, storage, and support costs.

Major section

Voice-First Needs Fallbacks

The mistake:: Designing smart devices that ONLY work via voice assistant, with no physical controls or app backup.

Voice assistant offline: internet outage means no fan control unless local controls exist.
Multiple users: guests may not know the command syntax or have account access.
Network dependency:: Voice requires internet + cloud + assistant service—3 failure points.

Major section

Voice-First Needs Fallbacks (continued)

Privacy moments: users may need silent control without speaking in front of roommates or guests.

Setup failure: account linking, Wi-Fi onboarding, or cloud consent can fail before the product is usable.
Guests/children:: Can't use device without setup.
Secondary:: Physical wall switch for reliability (works offline, guests can use it).

Major section

Voice-First Needs Fallbacks (continued)

Speech mismatch: accents, language settings, speech impairments, and background noise can prevent recognition.

Design principle:: Every modality should enable 100% of CORE functionality.
Advanced features can be modality-specific, but turning on/off must work via all three methods.
The "magic" of voice wears off fast when it fails 10% of the time.

Major section

Summary

Graceful Degradation: Design five levels of degradation from full cloud through local hub, direct control, conservation, and emergency mode -- core functions must never depend on internet.

Accessibility as Default: The curb cut effect means designing for edge cases (motor impairments, vision loss, noisy environments) improves the experience for all users.
Physical Privacy Controls: For high-stakes privacy features like cameras, physical mechanisms (shutters, hardware switches) provide trust that software indicators cannot match.
Redundant Modalities: Every critical function should be accessible through at least two different modalities so that failure of one channel does not block the user.

Deck summary

Key takeaways

Active ring draw while audio, LED, radio, processor, and video preview are running.
The mistake:: Designing smart devices that ONLY work via voice assistant, with no physical controls or app backup.
Privacy moments: users may need silent control without speaking in front of roommates or guests.
Speech mismatch: accents, language settings, speech impairments, and background noise can prevent recognition.

Retrieval practice

Recall check 1 of 6

UX Uma says: answer from memory, then check your reasoning.

Q1A smart speaker responds to commands, but a deaf user cannot interact with it. According to accessibility principles, what multi-modal interaction should be added?

AIncrease speaker volume and add louder alert tones, assuming residual hearing is enough

BAdd flashing status lights so users know the device is speaking, but not what it said

CKeep the product voice-only because smart speakers are not meant for text interaction

DProvide a display or app with captions, visual responses, and text or touch input

Show answer

Answer: D Accessible IoT design provides equivalent functionality through multiple modalities.

Retrieval practice

Recall check 2 of 6

UX Uma says: answer from memory, then check your reasoning.

Q2A smart thermostat loses internet connectivity. According to graceful degradation principles, what should happen?

AShow a cloud outage screen and block temperature changes until reconnection

BPower down the thermostat so cloud and device state cannot diverge

CKeep local temperature control on the device, show offline status, and sync later

DTreat Wi-Fi loss as an emergency and prioritize cellular alerts over local control

Show answer

Answer: C Graceful degradation ensures core functionality remains available during failures.

Retrieval practice

Recall check 3 of 6

UX Uma says: answer from memory, then check your reasoning.

Q3You're designing a smart doorbell camera. Which modality combination best balances privacy, accessibility, and reliability?

AUse only smart-speaker voice control, keeping visitor response dependent on cloud recognition

BUse only the phone app, making video rich but requiring a charged phone nearby

CUse only an indoor button, preserving local control but omitting video and remote access

DCombine indoor physical control, app video, voice handling, and optional display or haptic alerts

Show answer

Answer: D Security devices require maximum reliability, making redundant modalities essential.

Retrieval practice

Recall check 4 of 6

UX Uma says: answer from memory, then check your reasoning.

Q4The voice interface latency budget targets <1 second total. Wake word detection should complete in under 100ms. Why is this critical?

AUsers expect immediate response; delays >150ms feel sluggish and cause users to repeat commands

BCloud processing requires the full 900ms budget, leaving no room for local delays

C100ms is the maximum time voice recordings can be buffered on device

DWake word detection is computationally expensive and cannot run faster

Show answer

Answer: A Research shows users perceive response times >150ms as delayed.

Retrieval practice

Recall check 5 of 6

UX Uma says: answer from memory, then check your reasoning.

Q5An IoT camera has a software-controlled "privacy mode" toggle in its app. A user advocacy group criticizes this approach. What is their most likely concern, and what is the recommended solution?

AThe toggle is too difficult to find in the app; the solution is to make it more prominent

BSoftware controls can be overridden remotely; the solution is a physical camera shutter or hardware disconnect switch

CThe privacy mode reduces video quality; the solution is to maintain full resolution in privacy mode

DThe toggle creates latency; the solution is to process the privacy mode locally

Show answer

Answer: B Software-only privacy controls can theoretically be bypassed by firmware updates, cloud-side changes, or security vulnerabilities.

Retrieval practice

Recall check 6 of 6

UX Uma says: answer from memory, then check your reasoning.

Q6Complete the form validation for IoT device configuration:

Aconst deviceName = form.querySelector('[name="device-name"]').value;

Bconst deviceName = form.getValue('device-name');

Cconst deviceName = form['device-name'];

Dconst deviceName = form.find('device-name').text;

Show answer

Answer: A querySelector() with attribute selectors finds form fields.

Print reference

Answers

Answer key.

D · Accessible IoT design provides equivalent functionality through multiple modalities.
C · Graceful degradation ensures core functionality remains available during failures.
D · Security devices require maximum reliability, making redundant modalities essential.
A · Research shows users perceive response times >150ms as delayed.
B · Software-only privacy controls can theoretically be bypassed by firmware updates, cloud-side changes, or security vulnerabilities.
A · querySelector() with attribute selectors finds form fields.