7 IoT Research Methods: Evidence and Selection

7.1 Start With the Decision

This section: Start With the Decision — A team cannot learn field routines from a survey alone.

A team cannot learn field routines from a survey alone. The research question should drive the method and the evidence it can yield.

7.2 Route Overview

This section: Route Overview — This is part 1 of 2.

This is part 1 of 2. Continue with IoT Research Methods: Fieldwork and Synthesis.

7.3 Part Objectives

This section: Part Objectives

• Match IoT research questions to suitable methods.
• Plan evidence that covers people, devices, and context.

7.4 Chapter Roadmap

This section: Chapter Roadmap

• Begin With the Gap in What You Know
• Start Simple
• In 60 Seconds
• Concept Check: Match Method to Question
• Prerequisites
• Pick Methods for Uncertainty
• Field Tests, Prototypes, Logs
• Research Plans Need Instrumentation
• How It Works: Match Method to Evidence
• Method Selection Map

7.5 Begin With the Gap in What You Know

This section: Begin With the Gap in What You Know — Telemetry means records sent from a system so its state can be watched.

Telemetry means records sent from a system so its state can be watched. Picture a team that sees many failed door-lock setups. The logs show where people stop, but not why. The first choice is the question that blocks the design choice. Then pick the lightest method that can answer it.

Watch real work when place and tools matter. Ask people when reasons and words matter. Use a small model when touch and flow matter. Use a survey when a known question needs wider reach. Use logs for real use patterns, but join them with care and limits. Run a field pilot when time, support, and upkeep may change the result.

Each method has a blind spot. An interview can miss real habit. A log can miss intent. A neat demo can hide long use. A large sample can still ask the wrong thing. Record the decision, people, method, tool, data, keep rule, gap, and next act. For a research method, build the mixed-method plan and field checks; check sampling, instrument links, bias, privacy, and overclaim before accepting the result.

Turn the first question into a small plan. “Why does setup fail?” is still broad. Split it. Where does the person stop? What did they expect? What did the device show? What happened on the network? A short task test can show the stop. A talk after the task can show the reason. A log can tie the step to device state. One method checks the next.

Choose people who face the real task. Include a new user and a repeat user. Include a weak link or old phone if those are common. Do not recruit only the people close to the team. Write down who is missing. That gap limits the claim.

Before the study, decide what will change if each answer appears. This keeps the work tied to a choice. After the study, compare the sources. Note where they agree. Note where they clash. Do not smooth away a clash. It may reveal a hidden group, place, or failure. End with the next action and the fact that would make the team study again.

Keep the result modest. “Five people failed this step” is clear. “All users hate it” is not. Show the task, group, place, and date. Keep a short quote only when consent allows it. Link each design change to the evidence that caused it. Mark open questions. A useful plan tells the team what it still does not know.

7.6 Start Simple

This section: Start Simple — Choose the research method by the uncertainty you need to reduce.

Choose the research method by the uncertainty you need to reduce. If the team does not understand context, go observe; if wording is risky, interview and prototype; if reliability is doubtful, combine field pilots, telemetry, support logs, and task tests so the method matches the decision.

7.7 In 60 Seconds

This section: In 60 Seconds — Research methods are ways to reduce uncertainty before a design decision.

Research methods are ways to reduce uncertainty before a design decision. In IoT, the right method depends on what the team needs to learn about people, context, device behavior, service touchpoints, and long-term use.

No single method is enough for every question:

interviews reveal goals, explanations, language, and mental models. contextual inquiry shows behavior in the real environment. diary studies show repeated use across days or weeks. usability tests expose interaction and comprehension failures. prototype tests compare design alternatives before full implementation. surveys estimate how common a pattern may be after the pattern is understood. telemetry and support logs reveal field signals, but need interpretation. field pilots show how product, context, service, and maintenance interact over time.

The method should match the decision. A team should not use a quick survey to replace observation when the real question is physical placement, shared use, failure recovery, or trust.

7.8 Learning Objectives

This section: Learning Objectives

By the end of this chapter, you will be able to:

• choose a research method based on the design decision and evidence gap
• distinguish behavior evidence, explanation evidence, scale evidence, and field evidence
• plan contextual inquiry, interviews, diary studies, prototype tests, surveys, telemetry review, and pilots
• combine methods without turning research into an unfocused activity list
• document method limits, affected roles, validation needs, and conditions that would reopen research
• avoid common method defects such as convenience sampling, leading questions, and overclaiming survey results

7.9 Prerequisites

This section: Prerequisites

Before reading this chapter, review:

• User Research Fundamentals, which explains why observed behavior matters.
• Context-of-Use Analysis, which explains how environment and constraints shape design.
• Personas and Journey Maps for IoT, which explains how research evidence becomes synthesis.
• Pitfalls and Ethics for IoT User Research, which explains consent, privacy, sampling, and automation risks.

7.10 Pick Methods for Uncertainty

This section: Pick Methods for Uncertainty — An IoT research method is useful only when it answers the uncertainty behind a design decision.

An IoT research method is useful only when it answers the uncertainty behind a design decision. A team choosing setup copy for a leak sensor needs different evidence from a team deciding whether a building dashboard should aggregate occupancy by room, floor, or shift. Interviews, observation, logs, prototype tests, surveys, and pilots each answer a different kind of question.

Start by naming what is unknown: behavior, explanation, frequency, comprehension, physical fit, signal reliability, support burden, maintenance, privacy expectation, or long-term trust. Then choose the smallest method mix that can answer that unknown without pretending it proves more than it does.

Use Figure 7.1 to choose a method for the question still unanswered.

+

In the meeting room comfort sensor case, the first uncertainty is why staff ignore a prompt. An interview after an observed incident can reveal goals, trust, language, and responsibility that a count alone cannot explain. Once the pattern is understood, a survey or telemetry can estimate how often it occurs in the studied setting. The figure gives these methods different evidence jobs. Use the result to decide what to change, and keep the participant and site limits visible when making that choice.

For a connected product, the evidence type is often mixed. A setup abandonment question needs observation of physical placement, a prototype check of state messages, logs for Wi-Fi join or BLE provisioning failures, and a follow-up interview about what the user believed was happening. A survey can help later, but only after the team knows which setup conditions and failure states to ask about.

For shared spaces, method choice also needs affected-role coverage. A meeting-room comfort prompt involves the person who receives the prompt, other people in the room, facilities staff, support staff, and the team responsible for sensor placement or HVAC rules. A method plan that only asks one role whether they like automation will miss social pressure, privacy expectations, signal trust, and maintenance ownership.

Behavior unknown: observe setup, placement, workaround, handoff, alert response, or maintenance in the real environment. Meaning unknown: interview after an observed event so participants can explain goals, trust, language, responsibility, and expectations. Scale unknown: use surveys or telemetry after qualitative work defines the pattern that needs counting.

7.11 Field Tests, Prototypes, Logs

This section: Field Tests, Prototypes, Logs — For connected products, one method rarely covers the whole decision.

For connected products, one method rarely covers the whole decision. A first-time smart-lock setup study might combine contextual inquiry at the door, a prototype test of invite and permission screens, support-log review of failed access cases, and a short interview after the task. The observation shows what people do, the prototype test checks comprehension, the logs show which failures happen in the field, and the interview explains why a user trusted or distrusted the state.

Choose participants around roles, not only market segments. A shared-access product may need account owners, guests, installers, property managers, support agents, and bystanders. A comfort-sensor study may need workers from different shifts, facilities staff, people in shared spaces, and people with accessibility needs. A diary study can then capture intermittent events such as muted alerts, low-battery warnings, stale readings, missed notifications, and manual overrides.

Build the plan around decision checkpoints. Before a pilot, observation and prototype tests can decide whether the setup flow distinguishes permission denial, offline state, account conflict, bad placement, and low battery. During a pilot, diary entries and telemetry can show whether the same states recur across days. After support tickets arrive, interviews with users and support staff can explain whether the recovery path, diagnostic fields, or escalation script needs to change.

Keep the method mix small enough to run well. A useful plan names who will be observed, which task or state will be tested, which logs will be joined, how consent is recorded, what data is excluded, and what design decision will change if the evidence is strong. That prevents research from becoming a generic list of interviews, surveys, analytics, and workshops that no one can trace to a release decision.

1. Write the decision: state the feature, state model, permission flow, alert, dashboard, or support path the research will change.
2. Map method to evidence: use observation for action, interviews for explanation, prototype tests for comprehension, telemetry for field signals, and pilots for long-term interaction.
3. State the boundary: record which roles, contexts, devices, firmware versions, connectivity conditions, and service paths the findings do and do not cover.

7.12 Research Plans Need Instrumentation

This section: Research Plans Need Instrumentation — When a research method uses logs, telemetry, or a prototype, name the signals that will support interpretation.

When a research method uses logs, telemetry, or a prototype, name the signals that will support interpretation. Setup research may need Wi-Fi join error codes, BLE provisioning state, Matter commissioning step, QR or NFC scan result, OAuth invite token status, Android or iOS permission state, APNs or FCM notification delivery, MQTT publish/ack timing, device-shadow version, firmware version, battery voltage, RSSI/SNR, and support ticket category.

Instrumentation should match the research question. A diary entry that says “the alert was late” becomes easier to interpret when it can be compared with event timestamp, sensor sampling interval, gateway queue depth, retry count, notification quiet-hours state, push delivery receipt, app foreground/background state, and acknowledgement time. The goal is not to collect everything; it is to collect the minimum signals that let the team distinguish user confusion, device failure, cloud delay, permission denial, and support-process breakdown.

Plan joins before the study starts. A prototype test can log the screen, variant, selected action, error state, and recovery path. A field pilot can join anonymized participant id, device id, firmware version, gateway id, role, event timestamp, support ticket tag, and study note without exposing unnecessary household or workplace detail. If the join key is missing, teams often end up with interviews that cannot be connected to logs or logs that cannot explain user behavior.

Also name retention and access rules. Research data may include location, occupancy, access events, health-adjacent routines, household patterns, or workplace behavior. The method plan should say who can inspect raw events, what is aggregated, what is redacted before design review, when data expires, and which claim the evidence can support. Those controls make telemetry and support review usable without turning research into uncontrolled surveillance.

• Prototype handle: name the screen, state, error copy, permission prompt, alert channel, or recovery path being tested.
• Telemetry handle: name the event, field, timestamp, version, role, and retention limit before collecting data.
• Support handle: name the ticket tag, escalation path, resolution code, and handoff point that connect field trouble to design change.

7.13 How It Works: Match Method to Evidence

This section: How It Works: Match Method to Evidence — A research plan should begin with the decision the team needs to make.

A research plan should begin with the decision the team needs to make.

Weak research question:

“What do users want?”.

Stronger research question:

“Which setup state does a first-time user need to see when a shared entry device fails: phone permission denied, invite pending, credential inactive, device offline, or account ownership conflict?”.

The stronger question makes the method easier to choose. It points toward field setup observation, prototype testing of recovery messages, support-log review, and role-specific interviews.

Use this method-selection rule:

If the question is about what people actually do, observe behavior in context. If the question is about why people act that way, interview after observation. If the question is about repeated use over time, use diaries, logs, or field pilots. If the question is about whether an interface is understandable, test the prototype. If the question is about how common a known pattern is, use a survey or telemetry review. If the question affects safety, privacy, access, or shared spaces, include safeguards and affected roles.

7.14 Method Selection Map

This section: Method Selection Map — The same product question can produce weak evidence when the method does not match the learning goal.

The same product question can produce weak evidence when the method does not match the learning goal. Figure 7.2 makes that choice explicit before recruitment or data collection begins.

+

Across the top of Figure 7.2, understand asks what people do and where they do it. Explore asks what they think and need. Measure adds size and the number of people in the sample. Prove asks whether one change caused a result. Follow the questions below each goal. A site visit gives rich detail in the place of use. Talks give deep views from a small group. Forms compare views across a much larger group. Choose the method from the claim to be backed, and join sources when one method leaves a gap.

Method selection is only the start; findings must remain traceable to the situation in which they were observed. Figure 7.3 shows the minimum chain needed to turn research into a reviewable requirement.

+

Start Figure 7.3 with Observation, then attach the Role affected and the Context in which it happened; that preserves who experienced what under which conditions. Translate that evidence into a named Risk before writing the Requirement, so the design response has a reason. The final Evidence and Retest Trigger fields state how the requirement will be proved and what future change should reopen it. This record carries the chosen research method into the running design decision.

7.15 Practice the Decision

This section: Practice the Decision

7.16 Continue to the Next Part

This section: Continue to the Next Part — Carry this evidence into IoT Research Methods: Fieldwork and Synthesis, which begins with Contextual Inquiry.

Carry this evidence into IoT Research Methods: Fieldwork and Synthesis, which begins with Contextual Inquiry.