UX Design · Study deck
Interface Examples: Gesture and Multimodal UX
A gesture can feel quick while light, clothing, or motion makes it fail.
UX Uma is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- Explain: This visualization demonstrates effective dashboard design: prominent placement of frequently-used controls, clear visual hierarchy indicating device states, and progressive disclosure that hides complexity until needed.
- Explain: But Grandma had some special needs: her hearing wasn't as sharp as it used to be, and sometimes she forgot the exact words for things.
- Explain: The key challenge is maintaining consistency across modalities - the same action should be possible through voice, app, or physical control with predictable results.
- Explain: Users encountering 2+ consecutive failures within 5 minutes → 78% abandon voice, revert to physical controls (from exit survey data).
Major section
Gesture Control Interactions
This visualization shows common gesture vocabularies and the feedback loop between user action and system response.
- Effective gesture interfaces provide clear affordances about available gestures and immediate visual or haptic feedback confirming recognition.
Major section
Multimodal Interaction Framework
IoT systems typically support multiple interaction modalities to accommodate diverse contexts and user preferences.
- This framework shows how different input channels (voice, touch, physical, automated) can be combined to create flexible, accessible interfaces.
- The key challenge is maintaining consistency across modalities - the same action should be possible through voice, app, or physical control with predictable results.
Major section
Smart Home Interface Design
Smart home interfaces must balance comprehensive control with simplicity.
- This visualization demonstrates effective dashboard design: prominent placement of frequently-used controls, clear visual hierarchy indicating device states, and progressive disclosure that hides complexity until needed.
Major section
Voice Interface Design Flow
Voice interfaces require careful orchestration of multiple components.
- The user speaks a wake word to activate the system, which then captures and processes speech in real-time.
- Natural language processing extracts intent and entities from the utterance, enabling the system to execute commands and provide appropriate audio feedback.
- Effective voice UI design must handle errors gracefully and maintain conversational context across multi-turn interactions.
Major section
Accessibility in IoT Design
Visual accessibility includes high-contrast modes and screen reader support.
- Motor accessibility provides voice control and large touch targets for users with limited dexterity.
- Cognitive accessibility emphasizes simple language, consistent patterns, and error prevention to reduce mental load.
Major section
For Kids: Meet the Sensor Squad!
Designing for everyone means making sure your device works for young AND old, with or without perfect hearing or vision!
- The Sensor Squad was helping Grandma Rose set up her smart home.
- But Grandma had some special needs: her hearing wasn't as sharp as it used to be, and sometimes she forgot the exact words for things.
- "We need to think like GRANDMA, not like engineers!" said Temperature Terry.
Major section
Putting Numbers to It
Users encountering 2+ consecutive failures within 5 minutes → 78% abandon voice, revert to physical controls (from exit survey data).
- <80% success → 2.4/5 satisfaction ("I just use the switches instead").
- Key insight: 85% success rate is the MINIMUM for adoption.
- Below that, users abandon the voice system entirely and revert to physical controls.
Major section
When to Use Voice Interfaces
3-4 factors favor voice → Secondary option (provide physical fallback).
- Must deliver value: if <20% of users use voice, ROI is negative.
- Design principle:: Voice is convenience, not a replacement.
- Guest users (don't have account/app).
- High-reliability scenarios (can't tolerate 10-15% failure rate).
Major section
Avoid Overbuilt Voice NLU
Result: system over-interprets and makes wrong guesses.
- Real example:: A smart home voice system for elderly users tried to be "too smart": User: "It's cold in here".
- Another example: User: "Turn on the lights. Wait, no".
- Elderly users with slower speech need longer timeout before command execution.
Major section
Avoid Overbuilt Voice NLU (continued)
Cognitive mismatch:: Sophisticated AI makes assumptions elderly users didn't intend.
- Loss of control:: Users feel the system is "doing its own thing".
- Slower speech patterns:: Elderly users speak with more pauses—system should wait longer.
- The fix: Simpler is better for elderly users.
- System: [Silently sets to 75°F based on historical data].
Deck summary
Key takeaways
This visualization shows common gesture vocabularies and the feedback loop between user action and system response.
- IoT systems typically support multiple interaction modalities to accommodate diverse contexts and user preferences.
- Smart home interfaces must balance comprehensive control with simplicity.
- Voice interfaces require careful orchestration of multiple components.
- Visual accessibility includes high-contrast modes and screen reader support.
Retrieval practice
Recall check 1 of 3
UX Uma says: answer from memory, then check your reasoning.
Q1You're designing a voice-controlled smart home for elderly users. When the system doesn't understand a command, what is the best error response?
Show answer
Answer: B The system should take blame for the misunderstanding ("I didn't catch that" not "Invalid command") and offer exactly 2 clear choices to reduce cognitive load.
Retrieval practice
Recall check 2 of 3
UX Uma says: answer from memory, then check your reasoning.
Q2An elderly user says "I'm cold" to the smart home system. What should the system do?
Show answer
Answer: C Intent-based understanding recognizes that "I'm cold" expresses a desire for warmth without using command syntax.
Retrieval practice
Recall check 3 of 3
UX Uma says: answer from memory, then check your reasoning.
Q3Place each worked-example responsibility where it lives so you can audit whether a voice or smart-home interaction preserves context, recovers safely, and confirms the result accessibly.
Show answer
Answer: A A worked interface example is complete only when it carries context into a bounded state change and returns accessible evidence of success, failure, and recovery.
Q4Complete the responsive IoT dashboard layout handler:
Show answer
Answer: A window.innerWidth returns the viewport width. Changing gridTemplateColumns via style creates responsive layouts.
Print reference
Answers 1 of 2
Answer key.
- B · The system should take blame for the misunderstanding ("I didn't catch that" not "Invalid command") and offer exactly 2 clear choices to reduce cognitive load.
- C · Intent-based understanding recognizes that "I'm cold" expresses a desire for warmth without using command syntax.
- A · A worked interface example is complete only when it carries context into a bounded state change and returns accessible evidence of success, failure, and recovery.
Print reference
Answers 2 of 2
Answer key.
- A · window.innerWidth returns the viewport width. Changing gridTemplateColumns via style creates responsive layouts.