Analytics & ML · Study deck
Validation and Outlier Detection
Picture a greenhouse report that says humidity is 108 percent.
Data Dora is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- Explain: If a 95 percent interval for a treatment, calibration change, or firmware filter crosses the no-change value, the honest result is "not enough evidence of a reliable effect," not a forced improvement label.
- Explain: The Bland-Altman check should name the mean difference d, the standard deviation of the differences s, and the limits of agreement d +/- 1.96s after checking that the differences are roughly normal.
- Explain: Each validation result should include the source id, event time, ingestion time, schema version, canonical unit, rule version, pass/fail state, handling action, and reason.
Major section
Start With the Story · Validation Stops Bad Data
The message arrived on time and looks tidy, yet using it could trigger the wrong control action.
- JavaScript Object Notation, or JSON, is a text format for named data fields.
- A payload means the useful data carried inside a message.
- Valid structure does not prove that the values make sense.
Major section
Rules Plus Outlier Bounds · Qualitative Evidence Has a Validation Workflow
Validation should run in layers.
- Hard rules reject impossible data: malformed payloads, values beyond sensor limits, invalid timestamps, and impossible rates of change.
- Treating every warning as a hard reject loses useful evidence; treating every warning as valid poisons downstream analytics.
- Hard validation protects control decisions.
- MAD Uses median absolute deviation.
Major section
Compare Sensor Agreement
A Bland-Altman record makes method comparison explicit.
- When a deployment compares places, people, or measurement methods, validation needs more than a pass/fail rule.
- A mobile air-quality study might group 0.38-0.42 um particle counts by library, park walk, traffic street, car, and indoor context.
- Normality and significance also need careful wording.
Major section
Compare Sensor Agreement (continued)
Correlation is not enough when the question is whether two methods are interchangeable.
- The box plot is not decoration: the median, IQR, whiskers, and outliers show whether traffic exposure, enclosure placement, or sampling route changes the distribution enough to affect the claim.
- ANOVA is the group-comparison partner to that idea.
- Each cell is an observed count.
Major section
Compare Sensor Agreement (continued)
For ordered clinical or maintenance categories, a low kappa means the reviewers may be consistent with themselves but not with each other.
- A sample mean by itself says what happened in the sampled window; a confidence interval says how wide the plausible population effect remains.
- Confidence Report the estimate and interval together, especially when the interval is wide or crosses the no-change value.
- For binary outcomes, keep odds and risk separate.
Major section
Compare Sensor Agreement (continued)
Precision has two related meanings that must not be mixed.
- If a 95 percent interval for a treatment, calibration change, or firmware filter crosses the no-change value, the honest result is "not enough evidence of a reliable effect," not a forced improvement label.
- Equality A regression line can show correlation while the points sit away from the line of equality.
- and a confidence-interval margin of error has the general form.
Major section
Compare Sensor Agreement (continued)
Agreement Use ICC for continuous ratings and kappa for ordered categories when independent observers or devices must agree.
- Two blood-pressure cuffs, gait scorers, or particulate counters can move together and still disagree by a consistent offset.
- A line-of-equality plot shows whether method A and method B produce the same value, while ICC or kappa summarizes agreement after chance agreement and category structure are considered.
- The interval is the estimate plus or minus that margin.
Major section
Compare Sensor Agreement (continued)
When both variables are categories, start with a contingency table rather than forcing category codes into a correlation.
- If a scale change leaves correlation high but produces large differences, downstream logic should not treat the methods as drop-in replacements.
- A regression line between method A and method B can look strong even when the differences are too wide for release.
- Rows might represent firmware version and columns might represent outcome: normal, recovered fault, or unrecovered fault.
Major section
Compare Sensor Agreement (continued)
A large departure can be evidence against independence, but the result does not identify causation or say which difference matters operationally.
- The Bland-Altman check should name the mean difference d, the standard deviation of the differences s, and the limits of agreement d +/- 1.96s after checking that the differences are roughly normal.
- A histogram with a normal curve or a Shapiro-Wilk, D'Agostino-Pearson, or Kolmogorov-Smirnov test can support the limits-of-agreement assumption, but it does not prove the devices are interchangeable.
- In an IoT fleet, site, device age, workload, or rollout timing can confound an apparent firmware-fault association.
Major section
Compare Sensor Agreement (continued)
The population form is rho_XY = E[(X - mu_X)(Y - mu_Y)] / (sigma_X sigma_Y), so the statistic depends on means, standard deviations, and the expected cross-product of deviations.
- It tests a null hypothesis that two or more groups come from distributions with the same mean, then reports an F-statistic that compares variation between group means with variation within the samples.
- A p-value can be derived from that F-statistic, but the field decision still needs effect size, group definition, sampling conditions, and engineering consequence.
- If thousands of readings came from only a few devices, the device—not each timestamp—is often the relevant independent unit.
Major section
Compare Sensor Agreement (continued)
For IoT experiments, ANOVA can help compare firmware filters, sensor placements, calibration recipes, or operating modes, but it cannot by itself say which option is safe to ship.
- In both cases, a ratio of 1 means no difference, values above 1 indicate a higher event rate in the exposed group, and values below 1 indicate a lower event rate.
- Repeated sensor readings can be tightly grouped, which is measurement repeatability, while an estimate such as a sample mean can vary little across hypothetical repeated samples, which is statistical precision.
- Worked example: ratio interpretation cases: 40 events, 60 non-events controls: 20 events, 80 non-events.
Major section
Compare Sensor Agreement (continued)
More independent observations usually reduce sampling error, but they do not correct calibration bias, missing sites, selective device survival, or a flawed measurement procedure.
- The evidence record should state the estimator, sampling unit, number of independent units, variability estimate, confidence level, critical-value method, and resulting interval.
- odds in cases = 40 / 60 = 0.66 odds in controls = 20 / 80 = 0.25 odds ratio = 0.66 / 0.25 = 2.64 95 percent CI = 1.41 to 5.02.
- decision: the interval does not include 1, so the exposed group has a statistically significant higher odds of the event in this sample.
Major section
Policy for Failed Data
Clamping hides sensor faults and makes root-cause analysis impossible.
- The important engineering decision is not only how to detect invalid data, but what to do after detection.
- A physically impossible value can be rejected for control logic, but the rejected record is still valuable diagnostic evidence.
- Version validation rules and rerun or compare results when thresholds change.
Major section
Policy for Failed Data (continued)
Rare real events are often the events analysts and operators care about most.
- A suspicious but possible value may need a quality flag, a quarantine workflow, or a reviewer decision.
- A missing reading may be imputable for a trend chart but unacceptable for billing, safety, or compliance.
- Without rule versions, old decisions cannot be explained or reproduced.
Major section
Policy for Failed Data (continued)
Each validation result should include the source id, event time, ingestion time, schema version, canonical unit, rule version, pass/fail state, handling action, and reason.
- Downstream jobs should declare whether they use only valid records, valid plus flagged records, or a curated training subset.
- Silent fills can make dead devices look healthy and create false confidence.
- This record lets a dashboard warn users, a model-training job exclude the sample, and a maintenance team investigate the sensor without losing evidence.
Major section
Summary · Key Takeaway
Validation protects IoT analytics before data enters dashboards, models, alerts, and control loops.
- Hard failures can be rejected for automation, but the original payload, failed rule, quality state, and handling decision should remain available for audit.
- The practical goal is traceable quality state.
- Each reading should become valid, flagged, quarantined, or rejected for a named reason.
Deck summary
Key takeaways
The message arrived on time and looks tidy, yet using it could trigger the wrong control action.
- Validation should run in layers.
- A Bland-Altman record makes method comparison explicit.
- Correlation is not enough when the question is whether two methods are interchangeable.
- For ordered clinical or maintenance categories, a low kappa means the reviewers may be consistent with themselves but not with each other.
Retrieval practice
Recall check 1 of 2
Data Dora says: answer from memory, then check your reasoning.
Q1Which validation rule most directly catches a soil-moisture reading of 980 percent?
Show answer
Answer: A Physically impossible values are best caught by simple range validation.
Retrieval practice
Recall check 2 of 2
Data Dora says: answer from memory, then check your reasoning.
Q2Why is silently deleting every reading that fails validation a poor policy?
Show answer
Answer: A Silent deletion hides quality evidence and can bias downstream analytics.
Print reference
Answers
Answer key.
- A · Physically impossible values are best caught by simple range validation.
- A · Silent deletion hides quality evidence and can bias downstream analytics.