7 IoT Research Methods: Evidence and Selection

7.1 Start With the Decision

A team cannot learn field routines from a survey alone. The research question should drive the method and the evidence it can yield.

7.2 Route Overview

This is part 1 of 2. Continue with IoT Research Methods: Fieldwork and Synthesis.

7.3 Part Objectives

• Match IoT research questions to suitable methods.
• Plan evidence that covers people, devices, and context.

7.4 Chapter Roadmap

• Begin With the Gap in What You Know
• Start Simple
• In 60 Seconds
• Concept Check: Match Method to Question
• Prerequisites
• Pick Methods for Uncertainty
• Field Tests, Prototypes, Logs
• Research Plans Need Instrumentation
• How It Works: Match Method to Evidence
• Method Selection Map

7.5 Begin With the Gap in What You Know

Telemetry means records sent from a system so its state can be watched. Picture a team that sees many failed door-lock setups. The logs show where people stop, but not why. The first choice is the question that blocks the design choice. Then pick the lightest method that can answer it.

Watch real work when place and tools matter. Ask people when reasons and words matter. Use a small model when touch and flow matter. Use a survey when a known question needs wider reach. Use logs for real use patterns, but join them with care and limits. Run a field pilot when time, support, and upkeep may change the result.

Each method has a blind spot. An interview can miss real habit. A log can miss intent. A neat demo can hide long use. A large sample can still ask the wrong thing. Record the decision, people, method, tool, data, keep rule, gap, and next act. Use the Practitioner layer to build the mixed-method plan and field checks. Use the Under the Hood layer to inspect sampling, instrument links, bias, privacy, and overclaim. Those routes keep the method tied to the uncertainty it must reduce.

Turn the first question into a small plan. “Why does setup fail?” is still broad. Split it. Where does the person stop? What did they expect? What did the device show? What happened on the network? A short task test can show the stop. A talk after the task can show the reason. A log can tie the step to device state. One method checks the next.

Choose people who face the real task. Include a new user and a repeat user. Include a weak link or old phone if those are common. Do not recruit only the people close to the team. Write down who is missing. That gap limits the claim.

Before the study, decide what will change if each answer appears. This keeps the work tied to a choice. After the study, compare the sources. Note where they agree. Note where they clash. Do not smooth away a clash. It may reveal a hidden group, place, or failure. End with the next action and the fact that would make the team study again.

Keep the result modest. “Five people failed this step” is clear. “All users hate it” is not. Show the task, group, place, and date. Keep a short quote only when consent allows it. Link each design change to the evidence that caused it. Mark open questions. A useful plan tells the team what it still does not know.

7.6 Start Simple

Choose the research method by the uncertainty you need to reduce. If the team does not understand context, go observe; if wording is risky, interview and prototype; if reliability is doubtful, combine field pilots, telemetry, support logs, and task tests so the method matches the decision.

7.7 In 60 Seconds

Research methods are ways to reduce uncertainty before a design decision. In IoT, the right method depends on what the team needs to learn about people, context, device behavior, service touchpoints, and long-term use.

No single method is enough for every question:

interviews reveal goals, explanations, language, and mental models. contextual inquiry shows behavior in the real environment. diary studies show repeated use across days or weeks. usability tests expose interaction and comprehension failures. prototype tests compare design alternatives before full implementation. surveys estimate how common a pattern may be after the pattern is understood. telemetry and support logs reveal field signals, but need interpretation. field pilots show how product, context, service, and maintenance interact over time.

The method should match the decision. A team should not use a quick survey to replace observation when the real question is physical placement, shared use, failure recovery, or trust.

7.8 Learning Objectives

By the end of this chapter, you will be able to:

• choose a research method based on the design decision and evidence gap
• distinguish behavior evidence, explanation evidence, scale evidence, and field evidence
• plan contextual inquiry, interviews, diary studies, prototype tests, surveys, telemetry review, and pilots
• combine methods without turning research into an unfocused activity list
• document method limits, affected roles, validation needs, and conditions that would reopen research
• avoid common method defects such as convenience sampling, leading questions, and overclaiming survey results

7.9 Prerequisites

Before reading this chapter, review:

• User Research Fundamentals, which explains why observed behavior matters.
• Context-of-Use Analysis, which explains how environment and constraints shape design.
• Personas and Journey Maps for IoT, which explains how research evidence becomes synthesis.
• Pitfalls and Ethics for IoT User Research, which explains consent, privacy, sampling, and automation risks.

7.10 Pick Methods for Uncertainty

An IoT research method is useful only when it answers the uncertainty behind a design decision. A team choosing setup copy for a leak sensor needs different evidence from a team deciding whether a building dashboard should aggregate occupancy by room, floor, or shift. Interviews, observation, logs, prototype tests, surveys, and pilots each answer a different kind of question.

Start by naming what is unknown: behavior, explanation, frequency, comprehension, physical fit, signal reliability, support burden, maintenance, privacy expectation, or long-term trust. Then choose the smallest method mix that can answer that unknown without pretending it proves more than it does.

Before deciding how over time shapes pick methods for uncertainty, inspect Figure 7.1 beside & metrics. Together, over time and & metrics frame the pick methods for uncertainty claim: method choice starts by separating qualitative evidence that explains behavior from quantitative evidence that estimates scale after the pattern is understood.

Read over time alongside & metrics in Figure 7.1; their named relationship makes method choice starts by separating qualitative evidence that explains behavior from quantitative evidence that estimates scale after the pattern is understood concrete. For pick methods for uncertainty, over time supplies visible evidence; & metrics constrains the decision. In Figure 7.1, retain over time beside & metrics so pick methods for uncertainty remains explicit.

For a connected product, the evidence type is often mixed. A setup abandonment question needs observation of physical placement, a prototype check of state messages, logs for Wi-Fi join or BLE provisioning failures, and a follow-up interview about what the user believed was happening. A survey can help later, but only after the team knows which setup conditions and failure states to ask about.

For shared spaces, method choice also needs affected-role coverage. A meeting-room comfort prompt involves the person who receives the prompt, other people in the room, facilities staff, support staff, and the team responsible for sensor placement or HVAC rules. A method plan that only asks one role whether they like automation will miss social pressure, privacy expectations, signal trust, and maintenance ownership.

Behavior unknown: observe setup, placement, workaround, handoff, alert response, or maintenance in the real environment. Meaning unknown: interview after an observed event so participants can explain goals, trust, language, responsibility, and expectations. Scale unknown: use surveys or telemetry after qualitative work defines the pattern that needs counting.

7.11 Field Tests, Prototypes, Logs

For connected products, one method rarely covers the whole decision. A first-time smart-lock setup study might combine contextual inquiry at the door, a prototype test of invite and permission screens, support-log review of failed access cases, and a short interview after the task. The observation shows what people do, the prototype test checks comprehension, the logs show which failures happen in the field, and the interview explains why a user trusted or distrusted the state.

Choose participants around roles, not only market segments. A shared-access product may need account owners, guests, installers, property managers, support agents, and bystanders. A comfort-sensor study may need workers from different shifts, facilities staff, people in shared spaces, and people with accessibility needs. A diary study can then capture intermittent events such as muted alerts, low-battery warnings, stale readings, missed notifications, and manual overrides.

Build the plan around decision checkpoints. Before a pilot, observation and prototype tests can decide whether the setup flow distinguishes permission denial, offline state, account conflict, bad placement, and low battery. During a pilot, diary entries and telemetry can show whether the same states recur across days. After support tickets arrive, interviews with users and support staff can explain whether the recovery path, diagnostic fields, or escalation script needs to change.

Keep the method mix small enough to run well. A useful plan names who will be observed, which task or state will be tested, which logs will be joined, how consent is recorded, what data is excluded, and what design decision will change if the evidence is strong. That prevents research from becoming a generic list of interviews, surveys, analytics, and workshops that no one can trace to a release decision.

1. Write the decision: state the feature, state model, permission flow, alert, dashboard, or support path the research will change.
2. Map method to evidence: use observation for action, interviews for explanation, prototype tests for comprehension, telemetry for field signals, and pilots for long-term interaction.
3. State the boundary: record which roles, contexts, devices, firmware versions, connectivity conditions, and service paths the findings do and do not cover.

7.12 Research Plans Need Instrumentation

When a research method uses logs, telemetry, or a prototype, name the signals that will support interpretation. Setup research may need Wi-Fi join error codes, BLE provisioning state, Matter commissioning step, QR or NFC scan result, OAuth invite token status, Android or iOS permission state, APNs or FCM notification delivery, MQTT publish/ack timing, device-shadow version, firmware version, battery voltage, RSSI/SNR, and support ticket category.

Instrumentation should match the research question. A diary entry that says “the alert was late” becomes easier to interpret when it can be compared with event timestamp, sensor sampling interval, gateway queue depth, retry count, notification quiet-hours state, push delivery receipt, app foreground/background state, and acknowledgement time. The goal is not to collect everything; it is to collect the minimum signals that let the team distinguish user confusion, device failure, cloud delay, permission denial, and support-process breakdown.

Plan joins before the study starts. A prototype test can log the screen, variant, selected action, error state, and recovery path. A field pilot can join anonymized participant id, device id, firmware version, gateway id, role, event timestamp, support ticket tag, and study note without exposing unnecessary household or workplace detail. If the join key is missing, teams often end up with interviews that cannot be connected to logs or logs that cannot explain user behavior.

Also name retention and access rules. Research data may include location, occupancy, access events, health-adjacent routines, household patterns, or workplace behavior. The method plan should say who can inspect raw events, what is aggregated, what is redacted before design review, when data expires, and which claim the evidence can support. Those controls make telemetry and support review usable without turning research into uncontrolled surveillance.

• Prototype handle: name the screen, state, error copy, permission prompt, alert channel, or recovery path being tested.
• Telemetry handle: name the event, field, timestamp, version, role, and retention limit before collecting data.
• Support handle: name the ticket tag, escalation path, resolution code, and handoff point that connect field trouble to design change.

7.13 How It Works: Match Method to Evidence

A research plan should begin with the decision the team needs to make.

Weak research question:

“What do users want?”.

Stronger research question:

“Which setup state does a first-time user need to see when a shared entry device fails: phone permission denied, invite pending, credential inactive, device offline, or account ownership conflict?”.

The stronger question makes the method easier to choose. It points toward field setup observation, prototype testing of recovery messages, support-log review, and role-specific interviews.

Use this method-selection rule:

If the question is about what people actually do, observe behavior in context. If the question is about why people act that way, interview after observation. If the question is about repeated use over time, use diaries, logs, or field pilots. If the question is about whether an interface is understandable, test the prototype. If the question is about how common a known pattern is, use a survey or telemetry review. If the question affects safety, privacy, access, or shared spaces, include safeguards and affected roles.

7.14 Method Selection Map

The same product question can produce weak evidence when the method does not match the learning goal. Figure 7.2 makes that choice explicit before recruitment or data collection begins.

Across the top of Figure 7.2, UNDERSTAND asks about context and behaviour, whereas EXPLORE asks about opinions and needs. The MEASURE branch introduces scale and sample size; PROVE demands a causal relationship rather than a plausible story. Follow the questions below each goal: Contextual Inquiry supplies depth in the user’s environment, Interviews provide rich accounts from a small sample, and Surveys can compare opinions across a much larger group. The method is therefore chosen by the claim the team needs to support.

Method selection is only the start; findings must remain traceable to the situation in which they were observed. Figure 7.3 shows the minimum chain needed to turn research into a reviewable requirement.

Start Figure 7.3 with Observation, then attach the Role affected and the Context in which it happened; that preserves who experienced what under which conditions. Translate that evidence into a named Risk before writing the Requirement, so the design response has a reason. The final Evidence and Retest Trigger fields state how the requirement will be proved and what future change should reopen it. This record carries the chosen research method into the running design decision.

7.15 Continue to the Next Part

Carry this evidence into IoT Research Methods: Fieldwork and Synthesis, which begins with Contextual Inquiry.