Calibration Span Error

Calibration Span Error

animation
sensors
calibration
Try Start with low and high calibration points at 10% and 90%, reference error at 10 counts and full scale at 1000 counts. Bring the two calibration points closer together.
Observe The opening span is 80%, or 800 ADC counts, with 1.23% gain relative error. The endpoint errors are 0.12 and 1.11 percentage points of the normalized index. At 45% and 55%, the same 10-count error gives 9.09% gain error and endpoint errors of 4.09 and 5.00 points.
Explain At the opening settings, the correct gain is (90 − 10) / (900 − 100) = 0.10000 index points per count. Adding the 10-count reference error to the denominator gives 80 / 810 = 0.09877. Anchoring both lines at 100 counts and 10% index makes their separation grow with distance from that low reference.

Technical boundaries. The model assumes a positive count error in the span denominator rather than independent errors at both references. It rounds the calibration points to ADC counts and compares two straight-line mappings, without random noise or sensor curvature.

Calibration Span Error

Explore how the width of your calibration span affects error amplification. Adjust the calibration points and reference error to see the impact on measurement accuracy at the extremes.

An ADC (analog-to-digital converter) reports a sensor reading as a count. Here, the normalized index expresses the calibrated reading on a 0–100 scale; it is not a temperature or a voltage. Full scale is the count assigned to the top of that scale. The low and high controls select reference positions, which are rounded to whole ADC counts before calculating the gain. Gain means the number of index points assigned to one additional count.

Calibration span (normalized index):

Reference error:

Gain relative error:

Error at 0% index extrapolated (index points):

Error at 100% index extrapolated (index points):

Try narrowing the calibration span to 45%-55% and watch the endpoint errors grow.

Live mapping comparison at the current control settings
Reading position ADC counts Correct index (%) Altered index (%) Signed difference (points)

The table uses the same gains and offsets as the metrics above. Signed difference means altered minus correct: a positive value reads too high, a negative value reads too low. The endpoint metrics show the absolute size of that difference. The low reference stays fixed even when the span error changes.

Follow one calculation

With Low at 10%, High at 90%, Reference error at 10 counts and Full scale at 1000 counts, the two reference counts are 100 and 900. Their difference is 800 counts. The correct mapping is 10 + (count − 100) × 0.1. The altered mapping is 10 + (count − 100) × (80 / 810). At zero counts, the altered line reads 0.123% instead of 0%; at 1000 counts, it reads 98.889% instead of 100%. Those are absolute errors of 0.123 and 1.111 index points. The relative gain error compares the slopes: (0.1 − 80 / 810) / 0.1 × 100 = 1.235%, displayed as 1.23%.

Read across the live table before changing a slider. At the low reference, both mappings give 10%. At the high reference, the altered mapping gives 89.012%, so adding the error to the denominator does not keep the high reference anchored. Change only Reference error to 20 counts: the low row remains 10%, while the altered high row becomes 88.049%. This is a span error anchored at the low point, rather than a constant offset applied equally to every reading.

When this goes wrong

Move Low to 45% and High to 55%, leaving the other opening settings unchanged. The count span shrinks to 100, so the 10-count error now occupies one tenth of the reference separation. The altered gain is 10 / 110 = 0.09091 instead of 0.1. Gain relative error rises to 9.09%; the bottom reads 4.091% and the top reads 95.000%. A check made only at the anchored low reference would miss both endpoint errors. Use the two endpoint rows to inspect the extrapolation outside the calibrated interval.

Keep that 45%–55% interval and reduce Full scale to 100 counts. The reference separation is now just 10 counts, equal to the 10-count error. The gain halves, and the endpoint errors become 22.50 and 27.50 points. A narrow interval and a small count range can therefore make the same reference error dominate the calibration. Increasing Full scale in this model changes the available counts; it does not simulate buying a more accurate sensor.

These calculations assume a straight sensor response and a positive error added only to the span denominator. They do not test curvature, clipping, drift or independent uncertainty in both references. Rounding can also make the correct endpoint index differ slightly from 0 or 100 for some full-scale settings; the table compares the two computed lines at counts 0 and Full scale, rather than silently forcing those values. Green, orange and red metric text use illustrative cutoffs at 2 and 5; read the units and set a tolerance for the actual measurement before deciding whether an error is acceptable.

Back to the chapter