24 Validation and Outlier Detection
24.1 Start With the Story
Reject One Plausible-Looking Bad Reading
Picture a greenhouse report that says humidity is 108 percent. The message arrived on time and looks tidy, yet using it could trigger the wrong control action.
JavaScript Object Notation, or JSON, is a text format for named data fields. A payload means the useful data carried inside a message. Valid structure does not prove that the values make sense.
Pass one normal record, then try a missing unit, future time, frozen value, impossible range, wrong type, and old device version. Record the rule, input, pass, reject or quarantine state, reason, and downstream action.
This runway does not make every unusual value false. The deeper sections separate schema, physics, timing, context, cleaning, anomaly review, and the evidence needed to reuse or reject data.
Picture an IoT team using the ideas in Validation and Outlier Detection during a live operations review. A device has produced messy evidence, an analytic step is about to change an alert or control decision, and someone has to explain why the result should be trusted.
Read this page as that path from sensor evidence to accountable action. Start with what the system observes, keep the model or data treatment visible, and finish with the check that would convince an operator, maintainer, or auditor to act.
24.2 Validation Stops Bad Data
IoT systems produce data continuously, but continuous data is not automatically usable data. A payload may be well-formed JSON while still carrying a humidity value of 108 percent, a pressure reading from the future, a frozen value repeated for six hours, or a temperature that jumps 20 C in one second. Validation is the entry gate that decides whether each reading is complete, plausible, timely, and safe enough to use.
Validation is different from later cleaning or anomaly analysis. Validation applies known rules from schemas, physics, device configuration, deployment context, and timing. Cleaning repairs or imputes selected values after validation. Anomaly detection searches for unusual but possible behaviour in data that has already passed the basic trust checks.
A good validation layer does not merely reject data. It records why a reading passed, failed, or was quarantined so that data quality becomes observable instead of invisible.
Schema
Required fields exist, data types match, schema version is known, and payload size is within expected limits.
Physics
Values obey sensor and environment limits such as humidity 0-100 percent or indoor temperature within an operational band.
Time
Event time is monotonic enough, clock skew is bounded, arrival delay is acceptable, and reporting cadence is plausible.
Context
Related fields agree: pump flow matches pump state, dew point does not exceed temperature, and peer sensors are not wildly inconsistent.
Overview Knowledge Check
24.3 Rules Plus Outlier Bounds
Validation should run in layers. Hard rules reject impossible data: malformed payloads, values beyond sensor limits, invalid timestamps, and impossible rates of change. Soft rules flag suspicious but possible data: a value outside the usual operating envelope, a peer-sensor disagreement, or a statistical outlier. Treating every warning as a hard reject loses useful evidence; treating every warning as valid poisons downstream analytics.
Use deployment-specific thresholds. A greenhouse, a data center, and an outdoor weather station should not share the same temperature band. The rule file should name the sensor type, placement, units, expected cadence, physical range, operational range, maximum rate, peer checks, and the action taken on each failure.
Worked example: validating an office HVAC sensor incoming samples, one second apart: 22.0 C, 22.1 C, 22.2 C, 45.0 C, 22.3 C rules: physical range: -10 C to 60 C office operating range: 18 C to 26 C maximum rate: 0.5 C per second peer sensor tolerance: 3 C sample at 45.0 C: schema: pass physical range: pass operating range: fail rate of change from 22.2 C: abs(45.0 - 22.2) / 1 second = 22.8 C/s 22.8 C/s is greater than 0.5 C/s peer check: nearby zone sensor reads 22.4 C difference = 22.6 C, greater than 3 C decision: quarantine the 45.0 C sample keep the previous valid value for control logic store the original value, canonical unit, failed rule ids, and peer comparison alert only if the failure repeats or affects multiple sensors
Hard validation protects control decisions. Soft validation protects analysis by adding quality flags without pretending every suspicious value is certainly wrong.
Z-score
Compares distance from the mean in standard deviations. Useful for near-normal signals, but sensitive to previous outliers.
IQR
Uses Q1, Q3, and the interquartile range. Better for skewed or bounded data such as battery voltage and occupancy counts.
MAD
Uses median absolute deviation. Robust when the baseline should not be pulled around by rare extreme values.
Peer Check
Compares related sensors or derived physics, such as dew point versus temperature and humidity.
Worked example: IQR outlier bound window values: 20, 21, 21, 22, 22, 23, 23, 24, 25, 85 quartiles: Q1 = 21 Q3 = 24 IQR = Q3 - Q1 = 3 fences: lower = Q1 - 1.5 * IQR = 21 - 4.5 = 16.5 upper = Q3 + 1.5 * IQR = 24 + 4.5 = 28.5 decision: 85 is outside [16.5, 28.5], so flag it as an outlier. Do not replace it silently. Store the flag, the rule version, and the handling decision used by each downstream job.
24.4 Qualitative Evidence Has a Validation Workflow
Telemetry can show that an alert was acknowledged late, but it may not explain whether the operator could understand it, whether a maintenance procedure conflicted with production, or why a workaround became normal practice. Interviews, observation notes, incident narratives, audio, and video preserve that context. They are not unstructured leftovers: they need a question, a sampling boundary, a repeatable interpretation method, and an audit trail just as numeric data does.
Begin with a decision-focused question such as “What prevents technicians from acting on this alarm?” Select participants and situations that can illuminate that question, record consent and access limits, then transcribe or index the material without erasing time, role, or setting. Code short passages with descriptive labels, compare codes across records, and group recurring relationships into candidate themes. Actively search for negative cases that contradict a theme, distinguish direct observation from interpretation, and keep enough excerpts and coding history for another reviewer to challenge the conclusion.
The trade-off is depth against representativeness. A small set of field interviews can expose mechanisms that a dashboard never records, but it cannot support a population percentage merely because a theme recurs. Triangulate claims against maintenance tickets, device logs, workflow observations, and participants with different roles. For example, repeated “false alarm” comments may turn out to describe three different mechanisms: a drifting sensor, an alert without asset context, and an approval rule that prevents timely action. Those mechanisms require different engineering responses even though the original complaint used the same words.
Minimal qualitative evidence record: state the decision question and sample boundary; identify each source with its time, role, and setting; version the codebook and retain coded excerpts with interpretation notes; test each theme against supporting cases, negative cases, and competing explanations; record the telemetry, ticket, procedure, or second-role evidence used for triangulation; finish with the decision and the access, redaction, retention, and consent boundary.
24.5 Compare Sensor Agreement
When a deployment compares places, people, or measurement methods, validation needs more than a pass/fail rule. A mobile air-quality study might group 0.38-0.42 um particle counts by library, park walk, traffic street, car, and indoor context. The box plot is not decoration: the median, IQR, whiskers, and outliers show whether traffic exposure, enclosure placement, or sampling route changes the distribution enough to affect the claim.
Confidence intervals belong in the same evidence record. A sample mean by itself says what happened in the sampled window; a confidence interval says how wide the plausible population effect remains. If a 95 percent interval for a treatment, calibration change, or firmware filter crosses the no-change value, the honest result is "not enough evidence of a reliable effect," not a forced improvement label.
Distribution
Compare medians, IQR, tails, and outliers before reducing a route, room, or cohort to one average.
Confidence
Report the estimate and interval together, especially when the interval is wide or crosses the no-change value.
Equality
A regression line can show correlation while the points sit away from the line of equality. That is bias, not agreement.
Agreement
Use ICC for continuous ratings and kappa for ordered categories when independent observers or devices must agree.
Correlation is not enough when the question is whether two methods are interchangeable. Two blood-pressure cuffs, gait scorers, or particulate counters can move together and still disagree by a consistent offset. A line-of-equality plot shows whether method A and method B produce the same value, while ICC or kappa summarizes agreement after chance agreement and category structure are considered. For ordered clinical or maintenance categories, a low kappa means the reviewers may be consistent with themselves but not with each other.
A Bland-Altman record makes method comparison explicit. Plot the mean of the two measurements on the x-axis and their difference on the y-axis, then record the mean bias, standard deviation, and whether the difference grows with the measurement level. If a scale change leaves correlation high but produces large differences, downstream logic should not treat the methods as drop-in replacements.
Do the extra arithmetic before accepting a method comparison. A regression line between method A and method B can look strong even when the differences are too wide for release. The Bland-Altman check should name the mean difference d, the standard deviation of the differences s, and the limits of agreement d +/- 1.96s after checking that the differences are roughly normal. If d = -27.2 and s = 34.8, the practical agreement band is about -95.4 to 41.1: method A may read nearly 95 units below method B or 41 units above it. That is a release decision, not just a plot annotation.
Normality and significance also need careful wording. A histogram with a normal curve or a Shapiro-Wilk, D'Agostino-Pearson, or Kolmogorov-Smirnov test can support the limits-of-agreement assumption, but it does not prove the devices are interchangeable. A p-value should be read against a stated null hypothesis such as "no difference between methods." Smaller values mean the observed difference would be surprising under that null; they do not by themselves measure engineering importance, safety margin, or field fitness.
Pearson's product-moment coefficient belongs in that same caution box. It measures linear correlation between two numeric variables: positive values mean the variables tend to rise together, negative values mean one tends to fall as the other rises, and values near zero mean there is little linear dependence. The population form is rho_XY = E[(X - mu_X)(Y - mu_Y)] / (sigma_X sigma_Y), so the statistic depends on means, standard deviations, and the expected cross-product of deviations. It is useful for summarising association, but it is not a release argument unless the unit error, bias, and decision threshold also pass.
ANOVA is the group-comparison partner to that idea. It tests a null hypothesis that two or more groups come from distributions with the same mean, then reports an F-statistic that compares variation between group means with variation within the samples. A p-value can be derived from that F-statistic, but the field decision still needs effect size, group definition, sampling conditions, and engineering consequence. For IoT experiments, ANOVA can help compare firmware filters, sensor placements, calibration recipes, or operating modes, but it cannot by itself say which option is safe to ship.
For binary outcomes, keep odds and risk separate. Odds divide events by non-events; risk divides events by all people or devices at risk. An odds ratio compares exposed odds with control odds, while a risk ratio compares exposed risk with control risk. In both cases, a ratio of 1 means no difference, values above 1 indicate a higher event rate in the exposed group, and values below 1 indicate a lower event rate. Report the 95 percent confidence interval with the ratio; if the interval crosses 1, do not call the result statistically significant.
24.5.1 Cross-Tabulate Categorical Outcomes
When both variables are categories, start with a contingency table rather than forcing category codes into a correlation. Rows might represent firmware version and columns might represent outcome: normal, recovered fault, or unrecovered fault. Each cell is an observed count. Add row percentages when the question is “How do outcomes differ within each firmware group?” and column percentages when the question is “Which firmware groups make up each outcome?” Raw counts remain visible so a striking percentage based on very few devices cannot masquerade as strong evidence.
To test association, calculate the count expected in each cell if the row and column variables were independent:
The chi-square statistic sums across cells. A large departure can be evidence against independence, but the result does not identify causation or say which difference matters operationally. Inspect cell contributions, report an effect size such as Cramer’s V, preserve the sampling and exposure denominators, and combine sparse categories or use an appropriate exact method when expected counts are too small for the approximation. In an IoT fleet, site, device age, workload, or rollout timing can confound an apparent firmware-fault association.
24.5.2 Precision, Standard Error, and Error Limits
Precision has two related meanings that must not be mixed. Repeated sensor readings can be tightly grouped, which is measurement repeatability, while an estimate such as a sample mean can vary little across hypothetical repeated samples, which is statistical precision. For an independent sample mean, the estimated standard error is
and a confidence-interval margin of error has the general form
The interval is the estimate plus or minus that margin. More independent observations usually reduce sampling error, but they do not correct calibration bias, missing sites, selective device survival, or a flawed measurement procedure. Time-series readings from one sensor and readings clustered within one gateway are correlated; counting them as independent makes the standard error look artificially small. Use a design-aware standard error, block or cluster resampling, or a model that represents the dependence.
Suppose a team estimates mean temperature error after a firmware change. The evidence record should state the estimator, sampling unit, number of independent units, variability estimate, confidence level, critical-value method, and resulting interval. If thousands of readings came from only a few devices, the device—not each timestamp—is often the relevant independent unit. This matters because a narrow interval is only a precise statement about the sampled design; it is not proof that the sensor is accurate or that the result generalizes to the fleet.
Minimal method-agreement record comparison_id: "pm-counter-field-check-2026-07" method_a: "reference optical particle counter" method_b: "low-cost mobile node" unit: "particles in 0.38-0.42 um bin" groups: ["library", "park walk", "traffic street", "car", "indoors"] distribution_summary: "median, IQR, whiskers, outlier count by group" confidence_interval: "estimate plus 95 percent interval for each claim" agreement_plot: "method A vs method B with line of equality" bland_altman: "mean(A,B), A-B, percent difference, mean bias, SD, normality check, limits of agreement" binary_effects: "odds ratio or risk ratio, denominator, 95 percent CI, null value = 1" decision: "use for exposure trends; do not use as a reference-grade substitute without bias correction"
Worked example: ratio interpretation cases: 40 events, 60 non-events controls: 20 events, 80 non-events odds in cases = 40 / 60 = 0.66 odds in controls = 20 / 80 = 0.25 odds ratio = 0.66 / 0.25 = 2.64 95 percent CI = 1.41 to 5.02 decision: the interval does not include 1, so the exposed group has a statistically significant higher odds of the event in this sample. Do not translate that into risk, causation, or deployment policy without checking study design, confounders, and the IoT decision cost.
Practitioner Knowledge Check
24.6 Policy for Failed Data
The important engineering decision is not only how to detect invalid data, but what to do after detection. A physically impossible value can be rejected for control logic, but the rejected record is still valuable diagnostic evidence. A suspicious but possible value may need a quality flag, a quarantine workflow, or a reviewer decision. A missing reading may be imputable for a trend chart but unacceptable for billing, safety, or compliance.
Make those decisions explicit in data contracts and job configurations. Each validation result should include the source id, event time, ingestion time, schema version, canonical unit, rule version, pass/fail state, handling action, and reason. Downstream jobs should declare whether they use only valid records, valid plus flagged records, or a curated training subset.
Minimal validation record for audit
{
"sensor_id": "hvac-17",
"event_time": "2026-07-03T07:00:00Z",
"ingested_at": "2026-07-03T07:00:02Z",
"schema_version": "hvac-temp-v4",
"raw_value": 45.0,
"canonical_unit": "C",
"validation_rule_version": "office-temp-rules-2026-07",
"quality_state": "quarantined",
"failed_rules": ["operating_range", "rate_of_change", "peer_check"],
"handling": "exclude_from_control; retain_for_review",
"lineage_ref": "raw/topic/hvac-17/offset/884201"
}
This record lets a dashboard warn users, a model-training job exclude the
sample, and a maintenance team investigate the sensor without losing evidence.
The safest default is not "drop bad data"; it is "make quality state explicit, preserve evidence, and choose use-case-specific handling."
Under-the-Hood Knowledge Check
24.7 Summary
Validation protects IoT analytics before data enters dashboards, models, alerts, and control loops. Start with schema and type checks, then apply physical range, rate-of-change, freshness, completeness, and cross-sensor plausibility rules. Use statistical outlier methods such as Z-score, IQR, and MAD only after basic validity is established. Hard failures can be rejected for automation, but the original payload, failed rule, quality state, and handling decision should remain available for audit.
The practical goal is traceable quality state. Each reading should become valid, flagged, quarantined, or rejected for a named reason. Downstream consumers then know whether the value is safe for control, acceptable for a warning dashboard, excluded from model training, or awaiting review.
24.8 Key Takeaway
Validate first, clean second, analyze third. A data-quality pipeline should catch impossible readings with explicit rules, flag suspicious readings with evidence, and preserve enough validation metadata to reproduce every downstream decision.
24.9 See Also
- Cloud Data: Quality and Security - How validation evidence connects to governance, access, and audit controls.
- Data Quality and Preprocessing - The full data-quality workflow after validation.
- Missing Value Imputation and Noise Filtering - How to handle gaps and noise after invalid data is identified.
- Data Quality Normalization Lab - Hands-on scaling and preparation for validated records.
- Anomaly Detection - Finding unusual but possible behaviour after basic validity checks pass.