20  Calibration Span Error and Validation

Reference Span, Residuals, and Release Evidence

Keywords

sensor calibration, calibration span, residuals, reference uncertainty, range guard, two-point calibration

20.1 Start With the Measurement Story

A sensor can look correct at one point and fail across the working range. The story here is span evidence: checking low, middle, and high values so the calibration works where the IoT system actually operates.

Phoebe’s Field Notes: The Two Floors Under Every Raw Count

Phoebe’s Why

Every raw_low and raw_high this chapter fits a line through is an ADC count, and an ADC count carries two separate physical limits that no amount of clever calibration can remove. The first is quantization: a continuous voltage gets rounded into one of a fixed number of discrete codes, so there is an irreducible “last digit” floor under the fitted line, independent of how good the two reference points are. The second is sampling: a reference reading is not instantaneous truth, it is one sample (or an average of a few) of a physical process that may itself be fluctuating – ice slowly melting, or boiling water roiling with bubbles. If that process has real frequency content and the reference is captured too slowly, the fast fluctuation folds down and contaminates exactly the “steady” reference value the calibration span depends on.

The Derivation

An \(N\)-bit ADC with reference voltage \(V_{ref}\) divides the input into \(2^N\) codes, so the step size is

\[q = \frac{V_{ref}}{2^N}\]

Treating the rounding error as uniformly distributed across one code gives a mean-square error of \(q^2/12\), so the noise floor’s RMS value and the resulting signal-to-noise ceiling are

\[\sigma_q = \frac{q}{\sqrt{12}}, \qquad \mathrm{SNR}_{dB} = 6.02\,N + 1.76\]

Separately, a signal sampled at rate \(f_s\) cannot distinguish a true frequency \(f\) from any alias

\[f_{alias} = \left| f - f_s \cdot \mathrm{round}\!\left(\frac{f}{f_s}\right) \right|\]

unless \(f_s \geq 2f_{max}\) – the Nyquist criterion. Below that rate, fast content folds into a slower apparent frequency and cannot be told apart from a real slow signal.

Worked Numbers: This Chapter’s Own Ice/Boiling Calibration

• This chapter’s own two-point code example uses a 12-bit ADC at ATTN_11DB (catalog-typical \(V_{ref}\approx3.3\) V full scale), with raw_low, known_low = 400, 0.0 (ice water) and raw_high, known_high = 3200, 100.0 (boiling): \(q = 3.3/2^{12} = 0.806\) mV, noise floor \(\sigma_q = 0.233\) mV, and \(\mathrm{SNR} = 6.02(12)+1.76 = 74.0\) dB.
• Span floor: the fit covers 3200 - 400 = 2800 codes over 100.0 - 0.0 = 100.0 degrees, so the quantization step alone sets a resolution of \(100.0/2800 = 0.0357\,^{\circ}\mathrm{C}\) per code – no reference precision below this can be recovered from the raw counts, regardless of how narrow or wide the span error discussion above is.
• Nyquist risk in the reference itself: nucleate boiling produces real surface-temperature fluctuation content on the order of a few hertz (catalog-typical order of magnitude, not this chapter’s own figure). If raw_high is captured by averaging repeated adc.read() calls at, say, 3 Hz while the boiling surface fluctuates near 5 Hz, that 5 Hz component aliases to \(|5 - 3\cdot\mathrm{round}(5/3)| = 1\) Hz – a slow-looking wobble buried inside the very average meant to be a steady 100 °C reference.
• The fix is the same discipline the chapter already applies to reference spacing: a boiling or ice reference should be over-sampled well above its own fluctuation bandwidth (comfortably above 10 Hz here) and averaged, not captured as a single adc.read(), or the “known_high = 100.0” reference itself carries hidden, aliased error before the span-amplification story even starts.

20.2 Learning Objectives

After this page, you should be able to:

• Explain why reference points that are too close together amplify calibration error.
• Choose calibration references that bracket the operating range instead of extrapolating from the middle.
• Use residual checks to separate random noise from systematic nonlinearity.
• Decide what evidence belongs in a production calibration release record.

20.3 Why This Follows the Lab

The Sensor Calibration Lab builds the two-point equation and stores it in firmware. This page tests the equation as an engineering model: the reference span, independent verification points, residual limits, and range guards decide whether the model is trustworthy enough to ship.

A calibration coefficient is not proof by itself. A useful record includes the two fit references, the calculated slope and offset, at least one holdout verification point, the accepted residual limit, and the range where the correction is allowed to operate.

20.4 Reference Points Too Close

The mistake: An engineer performs two-point calibration on a soil moisture sensor using reference points at 45% and 55% moisture (only 10% apart). After deployment, the sensor shows large errors at 10% and 90% moisture because small inaccuracies in the closely-spaced reference measurements were amplified across the full range.

Why it happens: Two-point calibration creates a linear equation (y = mx + b) using two measurements. When those measurements are close together, any small error in the reference measurement or sensor reading gets amplified when extrapolating to values far from the calibration range.

Real numbers: A 10 ADC count error (1% of full scale) in your reference point, when calibration range is only 100 counts, produces up to 5% error at the measurement extremes. The narrow calibration range acts as a lever arm amplifying tiny reference errors into significant measurement errors.

The fix: Use calibration points that bracket your expected measurement range with margin:

• Bad practice: Greenhouse needs 20-80% moisture, but calibration points are only 45% and 55%. The 10% calibration span covers only 17% of the operating range.
• Good practice: Greenhouse needs 20-80% moisture, and calibration points are 10% and 90%. The 80% span brackets the full operating range with margin.

Industry rule of thumb: Calibration span should be at least 1.2x your measurement range, ideally centered on your operating range. For a 20-80% application, calibrate at 10% and 90%.

Why wide span matters:

1. Error division: 10 ADC count error / 800 ADC span = 1.25% relative error in gain, versus 10% relative error with a 100-count span.
2. Interpolation not extrapolation: Measuring within calibration range is accurate; measuring outside it magnifies errors.
3. Sensor nonlinearity: Most sensors are most linear in their mid-range; calibrating near the working endpoints exposes curvature sooner.

Verification test: After calibration, check several points across the full range:

• 0% moisture: Read dry sensor in air.
• 25% moisture: Use calibrated reference soil.
• 50% moisture: Half-saturated reference.
• 75% moisture: Nearly saturated reference.
• 100% moisture: Fully submerged in water.

If errors at 0% and 100% exceed +/-2%, your calibration points were too close together.

20.5 Close Reference Points Add Error

• Calibration points: 45% and 55%.
• Correct raw readings: 450 and 550 on a 0-1000 ADC scale.
• Correct gain: (55 - 45) / (550 - 450) = 0.1.
• If the high point is off by only 10 ADC counts, the gain becomes (55 - 45) / (560 - 450) = 0.0909.
• At the low extreme, 0% becomes 4.1%.
• At the high extreme, 100% becomes 95.0%.

The tiny reference error becomes a large endpoint error because the calibration span was too narrow.

20.6 Calibration Span Error

Explore how the width of your calibration span affects error amplification. Adjust the calibration points and reference error to see the impact on measurement accuracy at the extremes.

Low calibration point:

10

% High calibration point:

90

% Reference error:

10

ADC counts Full scale:

1000

ADC counts

Calibration span:

Reference error:

Gain relative error:

Error at 0% extrapolated:

Error at 100% extrapolated:

Try narrowing the calibration span to 45%-55% and watch the endpoint errors grow.

20.7 Knowledge Check: Calibration Span

20.8 Concept Relationships

This Concept	Relates To	Relationship Type
Two-Point Calibration	Linear Algebra	Uses y=mx+b equation for correction
Gain Correction	Sensitivity Error	Multiplier that fixes slope/sensitivity issues
Offset Correction	Zero-Point Error	Additive constant that fixes bias
EEPROM Storage	Persistence	Non-volatile memory preserves calibration across power cycles
Moving Average Filter	Noise Reduction	Pre-filters data before calibration capture

20.9 Label the Diagram

20.10 Code Challenge

20.11 Calibration Linearity Limits

A two-point calibration produces a straight line, so it silently assumes the sensor is linear between and beyond those points. That assumption is where calibrations quietly go wrong.

Use residuals to see the limit. Suppose the 0 kPa and 100 kPa references both calibrate exactly after the two-point fit, but a 50 kPa check reference reports 54 kPa. The residual is +4 kPa at mid-range. Repeating the 50 kPa measurement ten times may reduce random scatter around that 54 kPa value, but it will not move the average to 50 kPa because the remaining error is systematic curvature. A production calibration record should therefore keep the reference points, the fitted coefficients, and at least one independent verification residual, not just the final slope and offset.

Extrapolation is even riskier. The same line may be acceptable from 0 to 100 kPa but untrustworthy at 120 kPa, because the sensor diaphragm, bridge circuit, or ADC front end may stop behaving linearly outside the bracketed range. If the device must operate across a wider range, add reference points and use a piecewise-linear table or a fitted curve, then document the maximum allowed residual. The important engineering habit is to treat calibration as a model with a valid range, not as a permanent magic correction.

Validation treats calibration as a model: fit points build the correction, holdout points test it, residuals expose where it fails, and range guards stop extrapolated readings being trusted blindly.

One point vs two points

A single reference can only correct offset; it trusts the datasheet slope. Two references correct offset and gain. Neither can correct curvature — that needs more points.

Interpolation vs extrapolation

Readings between the two references are interpolated and usually solid. Readings outside them are extrapolated, where unmodelled nonlinearity grows fastest. Bracket your working range.

Residual nonlinearity

If the true curve bows away from the line, a two-point fit leaves a bow-shaped error, largest near the middle. A three-point or polynomial fit is the fix, not a better two points.

Only as good as the reference

The calibration inherits the uncertainty of your reference standard. An uncertain reference gives an uncertain line — traceability matters, and drift means the line expires and must be redone.

It is also worth separating two jobs that are easy to confuse. Calibration removes systematic error (offset, gain, and with more points, curvature). Averaging many samples removes random error (noise). Averaging a badly calibrated sensor just gives you a very repeatable wrong answer.

20.11.1 Calibration Validation Limits

20.12 Release Checklist

Before accepting a two-point sensor calibration, record this evidence:

1. The low and high reference values, including their uncertainty and traceability.
2. The raw ADC counts or electrical readings captured at each reference.
3. The calculated slope and offset, with units.
4. At least one independent holdout point that was not used to fit the line.
5. The maximum allowed residual and the actual residuals observed.
6. The operating range where interpolation is allowed and the rule for rejecting extrapolated readings.

20.13 See Also

• Sensor Calibration Lab - hands-on Wokwi calibration workflow
• Sensor Calibration Techniques - broader calibration method reference
• Sensor Data Processing - filtering and signal conditioning before calibration capture

20.14 What’s Next

Return to Sensor Calibration Lab for the lab summary, or continue to Sensor Power Management to connect measurement quality with field energy constraints.