Calibration Span Error
Calibration Span Error
Technical boundaries. The model assumes a positive count error in the span denominator rather than independent errors at both references. It rounds the calibration points to ADC counts and compares two straight-line mappings, without random noise or sensor curvature.
Calibration Span Error
Explore how the width of your calibration span affects error amplification. Adjust the calibration points and reference error to see the impact on measurement accuracy at the extremes.
An ADC (analog-to-digital converter) reports a sensor reading as a count. Here, the normalized index expresses the calibrated reading on a 0–100 scale; it is not a temperature or a voltage. Full scale is the count assigned to the top of that scale. The low and high controls select reference positions, which are rounded to whole ADC counts before calculating the gain. Gain means the number of index points assigned to one additional count.
Follow one calculation
With Low at 10%, High at 90%, Reference error at 10 counts and Full scale at 1000 counts, the two reference counts are 100 and 900. Their difference is 800 counts. The correct mapping is 10 + (count − 100) × 0.1. The altered mapping is 10 + (count − 100) × (80 / 810). At zero counts, the altered line reads 0.123% instead of 0%; at 1000 counts, it reads 98.889% instead of 100%. Those are absolute errors of 0.123 and 1.111 index points. The relative gain error compares the slopes: (0.1 − 80 / 810) / 0.1 × 100 = 1.235%, displayed as 1.23%.
Read across the live table before changing a slider. At the low reference, both mappings give 10%. At the high reference, the altered mapping gives 89.012%, so adding the error to the denominator does not keep the high reference anchored. Change only Reference error to 20 counts: the low row remains 10%, while the altered high row becomes 88.049%. This is a span error anchored at the low point, rather than a constant offset applied equally to every reading.
When this goes wrong
Move Low to 45% and High to 55%, leaving the other opening settings unchanged. The count span shrinks to 100, so the 10-count error now occupies one tenth of the reference separation. The altered gain is 10 / 110 = 0.09091 instead of 0.1. Gain relative error rises to 9.09%; the bottom reads 4.091% and the top reads 95.000%. A check made only at the anchored low reference would miss both endpoint errors. Use the two endpoint rows to inspect the extrapolation outside the calibrated interval.
Keep that 45%–55% interval and reduce Full scale to 100 counts. The reference separation is now just 10 counts, equal to the 10-count error. The gain halves, and the endpoint errors become 22.50 and 27.50 points. A narrow interval and a small count range can therefore make the same reference error dominate the calibration. Increasing Full scale in this model changes the available counts; it does not simulate buying a more accurate sensor.
These calculations assume a straight sensor response and a positive error added only to the span denominator. They do not test curvature, clipping, drift or independent uncertainty in both references. Rounding can also make the correct endpoint index differ slightly from 0 or 100 for some full-scale settings; the table compares the two computed lines at counts 0 and Full scale, rather than silently forcing those values. Green, orange and red metric text use illustrative cutoffs at 2 and 5; read the units and set a tolerance for the actual measurement before deciding whether an error is acceptable.